Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(iot): stack tags on IoT resources and provisioning template checks (#2191)
iot-rocketda1ba805b857
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 56152223..3b4c9e89 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]+### Fixed+- **IoT / CloudFormation — stack tags and provisioning templates** — stack-level tags never reached IoT resources; they now reach thing types, thing groups, policies, topic rules, CA certificates and provisioning templates without the `aws:cloudformation:` tags (job templates get none, as on AWS), and an update that only changes tags no longer adds a policy version. An `AWS::IoT::ProvisioningTemplate` `Enabled` of `"false"` from a parameter or the template enabled the template; it now disables it. `CreateProvisioningTemplate` and the CloudFormation type accepted any certificate resource; they now refuse one without a `Status` of `Active` or `Inactive` (any case) or whose `CertificateId` is not a reference to `AWS::IoT::Certificate::Id`, a second certificate resource and a resource without `Properties`, with AWS's messages. Contributed by @iot-rocket.### Added- **WAF v2 — regex pattern sets** — `CreateRegexPatternSet`, `GetRegexPatternSet`, `UpdateRegexPatternSet`, `DeleteRegexPatternSet` and `ListRegexPatternSets` now manage regional and CloudFront-scoped regex pattern sets, with their ARN (`regexpatternset`), tags and `LockToken` checks on update and delete (`WAFOptimisticLockException` for a stale token), as AWS does. Web ACL and rule group rules are stored as sent, so a `RegexPatternSetReferenceStatement` pointing at a pattern set ARN round-trips; the statement is not evaluated, as MiniStack does not inspect requests.### Fixeddiff --git a/README.md b/README.mdindex 4fdc2596..b07d87a7 100644--- a/README.md+++ b/README.md@@ -467,7 +467,7 @@ subnet = ec2.create_subnet(| **Intrinsic Functions** | Ref, Fn::GetAtt, Fn::Join, Fn::Sub (both forms), Fn::Select, Fn::Split, Fn::If, Fn::Equals, Fn::And, Fn::Or, Fn::Not, Fn::Base64, Fn::FindInMap, Fn::ImportValue, Fn::GetAZs, Fn::Cidr || **Pseudo-Parameters** | AWS::StackName, AWS::StackId, AWS::Region, AWS::AccountId, AWS::URLSuffix, AWS::Partition, AWS::NoValue || **Parameters** | Default values, AllowedValues validation, NoEcho masking, String/Number/CommaDelimitedList types |-| **Stack Tags** | The `Tags` of CreateStack, UpdateStack and change sets reach every resource whose type carries a tag property, together with `aws:cloudformation:stack-name`, `stack-id` and `logical-id`; a key the template sets wins |+| **Stack Tags** | The `Tags` of CreateStack, UpdateStack and change sets reach every resource whose type carries a tag property, together with `aws:cloudformation:stack-name`, `stack-id` and `logical-id` (IoT types get the stack's tags without these three, IoT job templates get none); a key the template sets wins || **Conditions** | Fn::Equals, Fn::And, Fn::Or, Fn::Not — conditional resource creation || **Rules** | The `Rules` section runs after the parameters resolve and before any resource is touched (CreateStack, UpdateStack, CreateChangeSet): `RuleCondition`, `Assertions` with `AssertDescription`, Fn::And/Or/Not/Equals/If/Contains/EachMemberEquals/EachMemberIn and Ref; Fn::RefAll, Fn::ValueOf and Fn::ValueOfAll read the EC2 store for VPC, subnet and security-group ids || **Transforms** | `AWS::Serverless-2016-10-31` (SAM, in the `full` image), `AWS::LanguageExtensions` (`Fn::ForEach` over a literal list, a `CommaDelimitedList` parameter, `Fn::FindInMap` or `Fn::Split`, nested and inside `Properties`; `${Id}` / `&{Id}` in the fragment keys, `${Id}` in `Fn::Sub` and a `Ref` to the identifier; `Fn::Length`, `Fn::ToJsonString`) and the embedded `AWS::Include` transform (`Fn::Transform` with an `s3://` `Location`, JSON or YAML snippets) |diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.pyindex 61fbaa6f..a6123768 100644--- a/ministack/services/cloudformation/provisioners.py+++ b/ministack/services/cloudformation/provisioners.py@@ -1140,6 +1140,12 @@ _STACK_TAG_PROPERTY: dict[str, tuple[str, str]] = {"AWS::Events::EventBus": ("Tags", "list"),"AWS::IAM::Role": ("Tags", "list"),"AWS::IAM::User": ("Tags", "list"),+ "AWS::IoT::CACertificate": ("Tags", "list"),+ "AWS::IoT::Policy": ("Tags", "list"),+ "AWS::IoT::ProvisioningTemplate": ("Tags", "list"),+ "AWS::IoT::ThingGroup": ("Tags", "list"),+ "AWS::IoT::ThingType": ("Tags", "list"),+ "AWS::IoT::TopicRule": ("Tags", "list"),"AWS::KMS::Key": ("Tags", "list"),"AWS::Kinesis::Stream": ("Tags", "list"),"AWS::Lambda::Function": ("Tags", "list"),@@ -1182,9 +1188,11 @@ def _with_stack_tags(resource_type: str, props: dict, stack_tags: list,return propsprop, shape = specextra = _tag_map(stack_tags)- extra["aws:cloudformation:stack-name"] = stack_name- extra["aws:cloudformation:stack-id"] = stack_id- extra["aws:cloudformation:logical-id"] = logical_id+ # IoT refuses aws: tag keys and lists none of these on its resources.+ if not resource_type.startswith("AWS::IoT::"):+ extra["aws:cloudformation:stack-name"] = stack_name+ extra["aws:cloudformation:stack-id"] = stack_id+ extra["aws:cloudformation:logical-id"] = logical_idown = props.get(prop)if shape == "map":merged = {**extra, **(own if isinstance(own, dict) else _tag_map(own))}@@ -13616,12 +13624,13 @@ def _iot_policy_update(physical_id, old_props, new_props, stack_name, logical_idcreated = _iot_policy_create(logical_id or physical_id, new_props, stack_name)_delete_predecessor(_iot_policy_delete, physical_id, old_props)return created- _iot_policy_prune_versions(name)- resp = _iot._create_policy_version(- name, {"policyDocument": _iot_policy_document(new_props)}, {"setAsDefault": "true"}- )- if resp[0] >= 400:- raise ValueError(f"AWS::IoT::Policy update failed: {resp[2]!r}")+ if _iot_policy_document(new_props) != _iot_policy_document(old_props):+ _iot_policy_prune_versions(name)+ resp = _iot._create_policy_version(+ name, {"policyDocument": _iot_policy_document(new_props)}, {"setAsDefault": "true"}+ )+ if resp[0] >= 400:+ raise ValueError(f"AWS::IoT::Policy update failed: {resp[2]!r}")_iot_update_tags(_iot._policy_arn(name), old_props, new_props)return name, {"Arn": _iot._policy_arn(name), "Id": name}@@ -13661,7 +13670,7 @@ def _iot_provisioning_template_payload(name, props):if "Description" in props:payload["description"] = props["Description"]if "Enabled" in props:- payload["enabled"] = props["Enabled"]+ payload["enabled"] = _cfn_bool(props["Enabled"])if "TemplateType" in props:payload["type"] = props["TemplateType"]if props.get("PreProvisioningHook"):diff --git a/ministack/services/iot.py b/ministack/services/iot.pyindex d97822e8..45715e25 100644--- a/ministack/services/iot.py+++ b/ministack/services/iot.py@@ -1926,9 +1926,25 @@ def _validate_provisioning_template_name(name: str | None) -> tuple | None:return None+def _provisioning_template_resource_error(resource, certificates: int) -> str | None:+ """Why the service refuses one resource of a template body, or None."""+ props = resource.get("Properties")+ if props is None:+ return "Properties of a resource cannot be null."+ if resource.get("Type") != "AWS::IoT::Certificate":+ return None+ if certificates:+ return "Only one certificate resource can be defined"+ if not isinstance(props, dict) or props.get("CertificateId") != {"Ref": "AWS::IoT::Certificate::Id"}:+ return "CertificateId must contain a reference to AWS::IoT::Certificate::Id"+ status = props.get("Status")+ if not isinstance(status, str) or status.lower() not in ("active", "inactive"):+ return 'A certificate resource must specify a Status of either "Active" or "Inactive"'+ return None++def _validate_provisioning_template_body(body) -> tuple | None:- """The live service refuses a body without an AWS::IoT::Certificate- resource; its message is quoted verbatim below."""+ """Refuse a body the live service refuses, with its messages verbatim."""doc = bodyif not isinstance(doc, dict):try:@@ -1943,10 +1959,17 @@ def _validate_provisioning_template_body(body) -> tuple | None:)resources = doc.get("Resources")values = resources.values() if isinstance(resources, dict) else ()- if not any(- isinstance(r, dict) and r.get("Type") == "AWS::IoT::Certificate"- for r in values- ):+ certificates = 0+ for resource in values:+ if not isinstance(resource, dict):+ continue+ reason = _provisioning_template_resource_error(resource, certificates)+ if reason:+ return error_response_json(+ "InvalidRequestException", f"The template body is invalid: {reason}", 400+ )+ certificates += resource.get("Type") == "AWS::IoT::Certificate"+ if not certificates:return error_response_json("InvalidRequestException","The template body is invalid: Template must have certificate resource.",diff --git a/tests/test_cfn.py b/tests/test_cfn.pyindex 4d80e47d..ea4bdeb7 100644--- a/tests/test_cfn.py+++ b/tests/test_cfn.py@@ -1413,7 +1413,8 @@ def _cfn_provisioning_template_body(marker="one"):"Resources": {"certificate": {"Type": "AWS::IoT::Certificate","Properties": {"CertificateId":- {"Ref": "AWS::IoT::Certificate::Id"}}},+ {"Ref": "AWS::IoT::Certificate::Id"},+ "Status": "ACTIVE"}},"thing": {"Type": "AWS::IoT::Thing","Properties": {"ThingName": {"Ref": "SerialNumber"},"AttributePayload": {"Attributes":@@ -1475,6 +1476,60 @@ def test_cfn_iot_provisioning_template(cfn, iot_client):iot_client.describe_provisioning_template(templateName="cfn-provtemplate")+def test_cfn_iot_provisioning_template_enabled_from_a_string(cfn, iot_client):+ """Enabled as the string 'false' disables the template; a parameter update to 'true' enables it in place."""+ uid = _uuid_mod.uuid4().hex[:8]+ stack_name, by_ref, literal = f"cfn-pt-enabled-{uid}", f"cfn-pt-ref-{uid}", f"cfn-pt-str-{uid}"++ def template(name, enabled):+ return {"Type": "AWS::IoT::ProvisioningTemplate", "Properties": {+ "TemplateName": name, "Enabled": enabled,+ "ProvisioningRoleArn": "arn:aws:iam::000000000000:role/provisioning",+ "TemplateBody": _cfn_provisioning_template_body()}}++ body = json.dumps({+ "Parameters": {"EnabledParam": {"Type": "String", "Default": "false"}},+ "Resources": {"ByRef": template(by_ref, {"Ref": "EnabledParam"}),+ "Literal": template(literal, "false")},+ })++ def enabled(name):+ return iot_client.describe_provisioning_template(templateName=name)["enabled"]++ cfn.create_stack(StackName=stack_name, TemplateBody=body)+ try:+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "CREATE_COMPLETE", stack.get("StackStatusReason")+ assert (enabled(by_ref), enabled(literal)) == (False, False)+ cfn.update_stack(StackName=stack_name, TemplateBody=body, Parameters=[+ {"ParameterKey": "EnabledParam", "ParameterValue": "true"}])+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")+ assert (enabled(by_ref), enabled(literal)) == (True, False)+ finally:+ _delete_cfn_test_stack(cfn, stack_name)+++def test_cfn_iot_provisioning_template_certificate_without_status_fails_the_stack(cfn, iot_client):+ """A certificate resource without Status fails the create and rolls the stack back with no template left."""+ uid = _uuid_mod.uuid4().hex[:8]+ stack_name, name = f"cfn-pt-nostatus-{uid}", f"cfn-pt-nostatus-{uid}"+ body = json.loads(_cfn_provisioning_template_body())+ del body["Resources"]["certificate"]["Properties"]["Status"]+ cfn.create_stack(StackName=stack_name, TemplateBody=json.dumps({"Resources": {"PT": {+ "Type": "AWS::IoT::ProvisioningTemplate", "Properties": {+ "TemplateName": name, "TemplateBody": json.dumps(body),+ "ProvisioningRoleArn": "arn:aws:iam::000000000000:role/provisioning"}}}}))+ try:+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "ROLLBACK_COMPLETE"+ assert ("The template body is invalid: A certificate resource must specify a Status"+ in _stack_event_reasons(cfn, stack_name))+ with pytest.raises(iot_client.exceptions.ResourceNotFoundException):+ iot_client.describe_provisioning_template(templateName=name)+ finally:+ _delete_cfn_test_stack(cfn, stack_name)++def test_cfn_iot_provisioning_template_rename_replaces(cfn, iot_client):"""A TemplateName change replaces the template the way AWS::IoT::Policy'srename does: new physical id, old template gone."""@@ -27718,6 +27773,65 @@ def test_cfn_iot_tags_follow_the_template(cfn, iot_client):assert listed() == [{}] * 3+def test_cfn_iot_stack_tags_reach_the_tagged_types(cfn, iot_client):+ """Stack tags reach every tagged IoT type but JobTemplate, yield to a same-key template tag, and update in place."""+ pytest.importorskip("cryptography")+ from ministack.core.x509_utils import generate_ca++ uid = _uuid_mod.uuid4().hex[:8]+ stack_name = f"cfn-iot-stacktags-{uid}"+ ca_pem, _key = generate_ca(common_name=f"cfn-stacktags-{uid}")+ own = [{"Key": "own", "Value": "1"}]+ body = json.dumps({"Resources": {+ "Type": {"Type": "AWS::IoT::ThingType", "Properties": {"ThingTypeName": f"cfn-st-type-{uid}", "Tags": own}},+ "Group": {"Type": "AWS::IoT::ThingGroup", "Properties": {+ "ThingGroupName": f"cfn-st-group-{uid}", "Tags": own + [{"Key": "shared", "Value": "own"}]}},+ "Policy": {"Type": "AWS::IoT::Policy", "Properties": {+ "PolicyName": f"cfn-st-policy-{uid}", "Tags": own, "PolicyDocument": {+ "Version": "2012-10-17",+ "Statement": [{"Effect": "Deny", "Action": "iot:Publish", "Resource": "*"}]}}},+ "Rule": {"Type": "AWS::IoT::TopicRule", "Properties": {+ "RuleName": f"cfn_st_rule_{uid}", "Tags": own,+ "TopicRulePayload": {"Sql": "SELECT * FROM 't'", "Actions": [], "RuleDisabled": True}}},+ "Ca": {"Type": "AWS::IoT::CACertificate", "Properties": {+ "CACertificatePem": ca_pem, "CertificateMode": "SNI_ONLY", "Status": "INACTIVE", "Tags": own}},+ "Pt": {"Type": "AWS::IoT::ProvisioningTemplate", "Properties": {+ "TemplateName": f"cfn-st-pt-{uid}", "TemplateBody": _cfn_provisioning_template_body(),+ "ProvisioningRoleArn": "arn:aws:iam::000000000000:role/provisioning"}},+ "Jt": {"Type": "AWS::IoT::JobTemplate", "Properties": {+ "JobTemplateId": f"cfn-st-jt-{uid}", "Description": "d", "Document": "{}", "Tags": own}},+ }, "Outputs": {"Ca": {"Value": {"Ref": "Ca"}}}})++ def tags_by_resource():+ return [{t["Key"]: t["Value"] for t in iot_client.list_tags_for_resource(resourceArn=arn)["tags"]}+ for arn in arns]++ cfn.create_stack(StackName=stack_name, TemplateBody=body,+ Tags=[{"Key": "stacktag", "Value": "s"}, {"Key": "shared", "Value": "stack"}])+ try:+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "CREATE_COMPLETE", stack.get("StackStatusReason")+ prefix = "arn:aws:iot:us-east-1:000000000000:"+ arns = [f"{prefix}thingtype/cfn-st-type-{uid}", f"{prefix}thinggroup/cfn-st-group-{uid}",+ f"{prefix}policy/cfn-st-policy-{uid}", f"{prefix}rule/cfn_st_rule_{uid}",+ f"{prefix}cacert/{_output(stack, 'Ca')}", f"{prefix}provisioningtemplate/cfn-st-pt-{uid}",+ f"{prefix}jobtemplate/cfn-st-jt-{uid}"]+ stacked = {"own": "1", "shared": "stack", "stacktag": "s"}+ assert tags_by_resource() == [stacked, {**stacked, "shared": "own"}, stacked, stacked, stacked,+ {"shared": "stack", "stacktag": "s"}, {"own": "1"}]++ cfn.update_stack(StackName=stack_name, TemplateBody=body,+ Tags=[{"Key": "stacktag", "Value": "t"}, {"Key": "added", "Value": "1"}])+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")+ stacked = {"own": "1", "stacktag": "t", "added": "1"}+ assert tags_by_resource() == [stacked, {**stacked, "shared": "own"}, stacked, stacked, stacked,+ {"stacktag": "t", "added": "1"}, {"own": "1"}]+ assert len(iot_client.list_policy_versions(policyName=f"cfn-st-policy-{uid}")["policyVersions"]) == 1+ finally:+ _delete_cfn_test_stack(cfn, stack_name)++def test_cfn_kms_alias_target_change_in_place_and_rename_replaces(cfn, kms_client):"""A TargetKeyId change re-points the alias under its name (Ref keeps itsvalue), an AliasName change replaces the alias — the new name exists, thediff --git a/tests/test_iot.py b/tests/test_iot.pyindex df98b9ea..c2c4ec30 100644--- a/tests/test_iot.py+++ b/tests/test_iot.py@@ -1903,7 +1903,8 @@ def _provisioning_template_body() -> str:"Resources": {"certificate": {"Type": "AWS::IoT::Certificate",- "Properties": {"CertificateId": {"Ref": "AWS::IoT::Certificate::Id"}},+ "Properties": {"CertificateId": {"Ref": "AWS::IoT::Certificate::Id"},+ "Status": "ACTIVE"},},"thing": {"Type": "AWS::IoT::Thing",@@ -2009,6 +2010,56 @@ def test_iot_provisioning_template_body_must_carry_certificate_resource(iot_clieiot_client.describe_provisioning_template(templateName=name)+def test_iot_provisioning_template_body_checks_the_certificate_resource(iot_client):+ """The template body's certificate resource needs a CertificateId reference and an Active or Inactive Status."""+ role = "arn:aws:iam::000000000000:role/provisioning"+ cert_id = {"CertificateId": {"Ref": "AWS::IoT::Certificate::Id"}}+ status_msg = 'A certificate resource must specify a Status of either "Active" or "Inactive"'+ id_msg = "CertificateId must contain a reference to AWS::IoT::Certificate::Id"+ thing = {"Type": "AWS::IoT::Thing", "Properties": {"ThingName": "t"}}++ def cert(**props):+ return {"Type": "AWS::IoT::Certificate", "Properties": props}++ cases = [+ ("FLEET_PROVISIONING", {"c": cert(**cert_id)}, status_msg),+ ("JITP", {"c": cert(**cert_id)}, status_msg),+ ("FLEET_PROVISIONING", {"c": cert(**cert_id, Status="PENDING_ACTIVATION")}, status_msg),+ ("FLEET_PROVISIONING", {"c": cert(**cert_id, Status="")}, status_msg),+ ("FLEET_PROVISIONING", {"c": cert(**cert_id, Status=None)}, status_msg),+ ("FLEET_PROVISIONING", {"c": cert(**cert_id, Status={"Ref": "S"})}, status_msg),+ ("JITP", {"c": cert()}, id_msg),+ ("FLEET_PROVISIONING", {"c": cert(CertificateId="abc", Status="ACTIVE")}, id_msg),+ ("FLEET_PROVISIONING", {"c": cert(CertificateSigningRequest={"Ref": "Csr"}, Status="ACTIVE")}, id_msg),+ ("FLEET_PROVISIONING", {"c": {"Type": "AWS::IoT::Certificate"}},+ "Properties of a resource cannot be null."),+ ("FLEET_PROVISIONING", {"c": cert(**cert_id, Status="ACTIVE"),+ "t": {"Type": "AWS::IoT::Thing", "Properties": None}},+ "Properties of a resource cannot be null."),+ ("FLEET_PROVISIONING", {"a": cert(**cert_id, Status="ACTIVE"), "b": cert(**cert_id)},+ "Only one certificate resource can be defined"),+ ("FLEET_PROVISIONING", {"a": cert(**cert_id), "b": cert(**cert_id, Status="ACTIVE")}, status_msg),+ ]+ for kind, resources, message in cases:+ name = _unique("provtemplate")+ with pytest.raises(ClientError) as ei:+ iot_client.create_provisioning_template(+ templateName=name, templateBody=json.dumps({"Resources": resources}),+ provisioningRoleArn=role, type=kind)+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException"+ assert ei.value.response["Error"]["Message"] == f"The template body is invalid: {message}", resources+ with pytest.raises(ClientError):+ iot_client.describe_provisioning_template(templateName=name)++ for kind, status in (("FLEET_PROVISIONING", "active"), ("JITP", "Inactive"),+ ("FLEET_PROVISIONING", "AcTiVe")):+ name = _unique("provtemplate")+ iot_client.create_provisioning_template(+ templateName=name, provisioningRoleArn=role, type=kind,+ templateBody=json.dumps({"Resources": {"c": cert(**cert_id, Status=status), "t": thing}}))+ iot_client.delete_provisioning_template(templateName=name)++def test_iot_provisioning_template_name_validation(iot_client):role = "arn:aws:iam::000000000000:role/provisioning"for bad in ("x" * 37, "no:colons"):