Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(cfn): apply AWS::IoT::Policy updates instead of rolling the stack… (#1406)
maximooseminedc652dfcf101
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f7324e55..1d96d2d0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).### Added- **AWS IoT Jobs — control plane and device data plane** — `CreateJob` fell through to `Unsupported IoT path` and the `iot-jobs-data` endpoint did not exist, so device OTA loops could not be rehearsed locally. The `iot` service now serves the nine job operations and a new `iot-jobs-data` service the four device ones (incl. the `$next` sentinel), sharing one store and the AWS execution state machine. A `documentSource` job gets a placeholder, not the S3 object. Contributed by @iot-rocket.### Fixed+- **CloudFormation — `AWS::IoT::Policy` updates apply instead of rolling the stack back** — the type had a create and a delete but no update handler, so a stack update fell through to create, hit `ResourceAlreadyExistsException` on the policy it had provisioned itself, and rolled back with the document unchanged: an IoT policy could be deployed once and never edited again. A changed `PolicyDocument` is now a no-interruption update stored as a new policy version and made the default, so `Ref` keeps naming the same policy and `GetPolicy` returns the new document; a changed `PolicyName` is a replacement, creating the policy under the new name and removing the old one. Because IoT caps a policy at five versions, the update prunes the oldest non-default versions before storing a new document, as the CloudFormation handler does on AWS.- **EC2 — instance public IP and DNS reach the SDKs, and generated addresses are addresses** — `DescribeInstances` and `RunInstances` emitted the public address under `publicIpAddress` / `publicDnsName`, which are not the tags the EC2 wire schema defines (`ipAddress` / `dnsName`, per botocore's `ec2-2016-11-15` model), so every SDK dropped both fields silently and `PublicIpAddress` came back absent on instances that had one. They now ride the real tags. Fixing that exposed a second one: `_random_ip` appended two octets whatever it was given, so a one-octet prefix produced `52.55.218` — `AllocateAddress` has been handing back that shape as `PublicIp` all along, and no address parser accepts it. The generator now completes any prefix to four octets. Contributed by @iot-rocket.- **S3 — `CRC64NVME` checksums are computed instead of refused** — CRC-64/NVME is the algorithm current AWS SDKs and the CLI checksum uploads with by default, and MiniStack answered it with `InvalidRequest` ("requires optional native dependencies"), so a stock `aws s3 cp` or `aws s3api put-object` failed before it began unless the caller knew to set `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`. The algorithm is plain arithmetic — the reflected form of polynomial `0xAD93D23594C93659` with all-ones init and xorout — so it is now computed from a byte-at-a-time table in the stdlib, adding no dependency: `PutObject` and `CopyObject` validate a client-supplied `x-amz-checksum-crc64nvme` (`BadDigest` on mismatch) and `GetObject` / `HeadObject` return it under `x-amz-checksum-mode: ENABLED`, typed `FULL_OBJECT`. The tests pin the implementation to the algorithm's published check value (`b"123456789"` → `0xAE8B14860A799888`) and to a bit-at-a-time reference rather than to itself. `CRC32C` still requires the native `google-crc32c` and is still refused.diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.pyindex ca36aa99..7068d9db 100644--- a/ministack/services/cloudformation/provisioners.py+++ b/ministack/services/cloudformation/provisioners.py@@ -5614,17 +5614,69 @@ def _iot_thing_type_delete(physical_id, props):_iot._thing_types.pop(physical_id, None)+def _iot_policy_document(props):+ doc = props.get("PolicyDocument")+ return json.dumps(doc) if isinstance(doc, (dict, list)) else doc+++_IOT_POLICY_VERSION_LIMIT = 5+++def _iot_policy_prune_versions(name):+ """Make room for one more version, the way the CloudFormation handler does.++ IoT caps a policy at five versions and ``CreatePolicyVersion`` answers+ ``VersionsLimitExceeded`` once that is reached, so CloudFormation deletes+ the oldest non-default versions before storing a new document. MiniStack+ does not enforce the cap today; pruning here keeps the version list the+ same shape it has on AWS either way.+ """+ resp = _iot._list_policy_versions(name)+ if resp[0] >= 400:+ return+ versions = json.loads(resp[2])["policyVersions"]+ surplus = len(versions) - _IOT_POLICY_VERSION_LIMIT + 1+ if surplus <= 0:+ return+ prunable = sorted(+ (v["versionId"] for v in versions if not v["isDefaultVersion"]), key=int+ )+ for version_id in prunable[:surplus]:+ _iot._delete_policy_version(name, version_id)++def _iot_policy_create(logical_id, props, stack_name):name = props.get("PolicyName") or _physical_name(stack_name, logical_id)- doc = props.get("PolicyDocument")- if isinstance(doc, (dict, list)):- doc = json.dumps(doc)- resp = _iot._create_policy(name, {"policyDocument": doc})+ resp = _iot._create_policy(name, {"policyDocument": _iot_policy_document(props)})if resp[0] >= 400:raise ValueError(f"AWS::IoT::Policy create failed: {resp[2]!r}")return name, {"Arn": _iot._policy_arn(name), "Id": name}+def _iot_policy_update(physical_id, old_props, new_props, stack_name, logical_id=None):+ """Apply a policy change in place, the way CloudFormation does.++ A new ``PolicyDocument`` is a no-interruption update: IoT stores it as a new+ version and makes it the default, so ``Ref`` keeps naming the same policy.+ Renaming is a replacement — the new policy is created and the old one+ removed, since a policy no template still declares is not left behind.+ """+ name = new_props.get("PolicyName") or _physical_name(+ stack_name, logical_id or physical_id+ )+ if name != physical_id:+ created = _iot_policy_create(logical_id or physical_id, new_props, stack_name)+ _iot_policy_delete(physical_id, old_props)+ return created+ _iot_policy_prune_versions(name)+ resp = _iot._create_policy_version(+ name, {"policyDocument": _iot_policy_document(new_props)}, {"setAsDefault": "true"}+ )+ if resp[0] >= 400:+ raise ValueError(f"AWS::IoT::Policy update failed: {resp[2]!r}")+ return name, {"Arn": _iot._policy_arn(name), "Id": name}++def _iot_policy_delete(physical_id, props):_iot._policies.pop(physical_id, None)@@ -5862,7 +5914,12 @@ _RESOURCE_HANDLERS = {"AWS::RDS::DBInstance": {"create": _rds_db_instance_create, "delete": _rds_db_instance_delete},"AWS::IoT::TopicRule": {"create": _iot_topic_rule_create, "delete": _iot_topic_rule_delete},"AWS::IoT::ThingType": {"create": _iot_thing_type_create, "delete": _iot_thing_type_delete},- "AWS::IoT::Policy": {"create": _iot_policy_create, "delete": _iot_policy_delete},+ "AWS::IoT::Policy": {+ "create": _iot_policy_create,+ "update": _iot_policy_update,+ "update_with_logical_id": True,+ "delete": _iot_policy_delete,+ },"AWS::Cognito::IdentityPoolRoleAttachment": {"create": _cognito_identity_pool_role_attachment_create,"delete": _cognito_identity_pool_role_attachment_delete,diff --git a/tests/test_cfn.py b/tests/test_cfn.pyindex 9cb9f0bc..2bc7bb62 100644--- a/tests/test_cfn.py+++ b/tests/test_cfn.py@@ -552,6 +552,104 @@ def test_cfn_iot_and_cognito_role_attachment(cfn, iot_client, cognito_identity):_wait_stack(cfn, "cfn-iot-cog")+def test_cfn_iot_policy_document_update_applies_in_place(cfn, iot_client):+ """A changed PolicyDocument updates the policy instead of rolling the stack+ back: IoT stores a new default version and Ref keeps the same name."""+ def template(actions):+ return json.dumps({+ "Resources": {"Pol": {"Type": "AWS::IoT::Policy", "Properties": {+ "PolicyName": "cfn-pol-upd",+ "PolicyDocument": {"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": actions, "Resource": "*"}]}}}},+ "Outputs": {"Name": {"Value": {"Ref": "Pol"}}},+ })++ cfn.create_stack(StackName="cfn-iot-pol-upd", TemplateBody=template(["iot:Connect"]))+ assert _wait_stack(cfn, "cfn-iot-pol-upd")["StackStatus"] == "CREATE_COMPLETE"++ cfn.update_stack(+ StackName="cfn-iot-pol-upd",+ TemplateBody=template(["iot:Connect", "iot:Publish"]),+ )+ stack = _wait_stack(cfn, "cfn-iot-pol-upd")+ assert stack["StackStatus"] == "UPDATE_COMPLETE"+ assert next(+ o["OutputValue"] for o in stack["Outputs"] if o["OutputKey"] == "Name"+ ) == "cfn-pol-upd"++ policy = iot_client.get_policy(policyName="cfn-pol-upd")+ assert json.loads(policy["policyDocument"])["Statement"][0]["Action"] == [+ "iot:Connect", "iot:Publish",+ ]+ assert policy["defaultVersionId"] == "2"+ versions = iot_client.list_policy_versions(policyName="cfn-pol-upd")["policyVersions"]+ assert {v["versionId"] for v in versions} == {"1", "2"}++ cfn.delete_stack(StackName="cfn-iot-pol-upd")+ _wait_stack(cfn, "cfn-iot-pol-upd")+++def test_cfn_iot_policy_updates_stay_under_the_version_cap(cfn, iot_client):+ """IoT keeps at most five versions of a policy, so repeated updates prune the+ oldest non-default version rather than growing the list without bound."""+ def template(count):+ return json.dumps({+ "Resources": {"Pol": {"Type": "AWS::IoT::Policy", "Properties": {+ "PolicyName": "cfn-pol-cap",+ "PolicyDocument": {"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "iot:Connect",+ "Resource": [f"arn:aws:iot:*:*:client/c{i}" for i in range(count)]}]}}}},+ })++ cfn.create_stack(StackName="cfn-iot-pol-cap", TemplateBody=template(1))+ assert _wait_stack(cfn, "cfn-iot-pol-cap")["StackStatus"] == "CREATE_COMPLETE"++ for count in range(2, 9):+ cfn.update_stack(StackName="cfn-iot-pol-cap", TemplateBody=template(count))+ assert _wait_stack(cfn, "cfn-iot-pol-cap")["StackStatus"] == "UPDATE_COMPLETE"++ versions = iot_client.list_policy_versions(policyName="cfn-pol-cap")["policyVersions"]+ assert len(versions) == 5+ assert {v["versionId"] for v in versions} == {"4", "5", "6", "7", "8"}++ policy = iot_client.get_policy(policyName="cfn-pol-cap")+ assert policy["defaultVersionId"] == "8"+ assert len(json.loads(policy["policyDocument"])["Statement"][0]["Resource"]) == 8++ cfn.delete_stack(StackName="cfn-iot-pol-cap")+ _wait_stack(cfn, "cfn-iot-pol-cap")+++def test_cfn_iot_policy_rename_replaces_the_policy(cfn, iot_client):+ """Renaming is a replacement — the new policy exists under the new name and+ the old one does not survive the update."""+ def template(name):+ return json.dumps({+ "Resources": {"Pol": {"Type": "AWS::IoT::Policy", "Properties": {+ "PolicyName": name,+ "PolicyDocument": {"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "iot:Connect", "Resource": "*"}]}}}},+ "Outputs": {"Name": {"Value": {"Ref": "Pol"}}},+ })++ cfn.create_stack(StackName="cfn-iot-pol-ren", TemplateBody=template("cfn-pol-before"))+ assert _wait_stack(cfn, "cfn-iot-pol-ren")["StackStatus"] == "CREATE_COMPLETE"++ cfn.update_stack(StackName="cfn-iot-pol-ren", TemplateBody=template("cfn-pol-after"))+ stack = _wait_stack(cfn, "cfn-iot-pol-ren")+ assert stack["StackStatus"] == "UPDATE_COMPLETE"+ assert next(+ o["OutputValue"] for o in stack["Outputs"] if o["OutputKey"] == "Name"+ ) == "cfn-pol-after"++ assert iot_client.get_policy(policyName="cfn-pol-after")["policyName"] == "cfn-pol-after"+ with pytest.raises(iot_client.exceptions.ResourceNotFoundException):+ iot_client.get_policy(policyName="cfn-pol-before")++ cfn.delete_stack(StackName="cfn-iot-pol-ren")+ _wait_stack(cfn, "cfn-iot-pol-ren")++def test_cfn_lambda_layer_version_permission(cfn, lam):"""AWS::Lambda::LayerVersionPermission attaches a statement to the reallayer version's policy, readable via GetLayerVersionPolicy. (#1345, item 5)"""