Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(cloudformation): a failed cleanup delete stays visible, DescribeStackResource carries Metadata and ResourceStatusReason (#1653)
iot-rocketbc4081acb1aa
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 364cdc9f..f06d6ec2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,9 +9,12 @@ Versioning follows [Semantic Versioning](https://semver.org/).### Fixed- **CloudFormation — `Fn::Cidr`, `Fn::GetAZs`, `Fn::FindInMap` and the condition functions follow AWS** — `Fn::Cidr` answered `10.0.{i}.0/…` whatever block it was given (now the block is split: `192.168.0.0/16, 2, 8` is `192.168.0.0/24,192.168.1.0/24`, measured; IPv6 included), `Fn::GetAZs` answered a fabricated `a/b/c` for any region (now the stack's own zones, and an empty list for another region, measured), `Fn::FindInMap` answered `""` for a missing key and had no `DefaultValue` (now the default when given, otherwise `Template error: Unable to get mapping for M::x::y` before the stack exists, measured), and `Fn::And`/`Fn::Or`/`Fn::Not` in a value position leaked the unresolved call (now a boolean; as an Output value they are refused with `The Value field of every Outputs member must evaluate to a String.`, measured). Contributed by @iot-rocket.+- **CloudFormation — `DescribeStackResource` and `DescribeStackResources` carry `ResourceStatusReason`** — a resource in `DELETE_FAILED` (or any other failed state) kept its reason in the stack events only; the two Describe responses now carry the `ResourceStatusReason` element the API reference defines, absent for a healthy resource; `DescribeStackResource` also names its timestamp `LastUpdatedTimestamp`, the field `StackResourceDetail` defines (it carried a `Timestamp` element that clients drop, so the detail had no timestamp at all). Contributed by @iot-rocket.+- **CloudFormation — a removed resource whose delete fails stays in the stack as `DELETE_FAILED`** — when a resource dropped from the template could not be deleted in the cleanup after an update, the stack ended `UPDATE_COMPLETE` (as on AWS) but the resource was dropped from the stack record while it kept existing in the service, and `DeleteStack` never tried again. It now stays listed with `DELETE_FAILED` and the reason (in a nested stack too), the next update with a changed template retries the delete, and `DeleteStack` also reaches the resources the template no longer declares. Contributed by @iot-rocket.- **S3 — the Multi-Region Access Point alias starts with a letter and follows the documented pattern** — the alias was the first 13 hex characters of a uuid, so it could come out as digits only (`1240729989764.mrap`), which is not what S3 mints (`mfzwi23gnjvgw.mrap`) and made `test_cfn_s3_multi_region_access_point` fail whenever no letter was drawn (`str.islower()` is False without a cased character). The alias now follows the documented pattern `^[a-z][a-z0-9]*[.]mrap$`: a letter, then twelve lowercase letters or digits; the test checks that pattern. Contributed by @iot-rocket.### Added- **CloudFormation — `ListImports`, `UpdateTerminationProtection`, `SetStackPolicy`, `GetStackPolicy`, `CancelUpdateStack` and `ContinueUpdateRollback`** — the first four were routed to the service and answered `InvalidAction`, the last two did not exist, so a stack in `UPDATE_ROLLBACK_FAILED` had no way out short of a reset. `ListImports` names the stacks importing an export (a `ValidationError` when none does); termination protection is set on create or later and makes `DeleteStack` refuse with the stack unchanged (`DescribeStacks` reports `EnableTerminationProtection`); a stack policy is stored and returned (it is not enforced); `CancelUpdateStack` stops an `UPDATE_IN_PROGRESS` update before its next resource and rolls it back ("User Initiated"); `ContinueUpdateRollback` retries the deletes that failed the rollback, skips the `ResourcesToSkip` (their status becomes `UPDATE_COMPLETE`, the resource stays) and lands the stack in `UPDATE_ROLLBACK_COMPLETE`. Contributed by @iot-rocket.+- **CloudFormation — `DescribeStackResource` returns the resource `Metadata`** — the `Metadata` attribute of a resource was accepted and never returned; `StackResourceDetail` now carries it as a JSON string with intrinsics interpreted, as the API reference defines it (a CDK stack's `aws:cdk:path` is the common case); a `Metadata` that resolves away or cannot be resolved comes back as declared. Contributed by @iot-rocket.### Fixed- **CloudFormation — parameter constraints are enforced** — `AllowedPattern`, `MinLength`, `MaxLength`, `MinValue` and `MaxValue` were ignored (only `AllowedValues` was checked), so a value a real account refuses deployed locally. They are checked before a stack exists, with CloudFormation's message (`Parameter 'P' must match pattern ^[a-z]+$`, measured) and a `ConstraintDescription` in place of the reason; a `CommaDelimitedList` is matched per member and a `List<Number>` is validated as numbers. Contributed by @iot-rocket.- **CloudFormation — an unregistered `AWS::CloudFormation::*` type is unrecognized, not a silent no-op** — `AWS::CloudFormation::Macro`, `HookVersion`, `ModuleVersion` and any typo under that prefix deployed as a placeholder with a fabricated physical id, logged at INFO, while the README promises that unsupported types do not silently succeed. The pre-flight now treats them like every other unrecognized type (`Template format error: Unrecognized resource types`); the registered ones — `WaitCondition`, `WaitConditionHandle`, `Stack`, `CustomResource` — are unaffected. Contributed by @iot-rocket.diff --git a/ministack/services/cloudformation/handlers.py b/ministack/services/cloudformation/handlers.pyindex 84063882..227925de 100644--- a/ministack/services/cloudformation/handlers.py+++ b/ministack/services/cloudformation/handlers.py@@ -18,6 +18,7 @@ from .changesets import (_list_change_sets,)from .engine import (+ _NO_VALUE,_apply_sam_transform_if_applicable,_evaluate_conditions,_has_dynamic_references,@@ -339,6 +340,40 @@ def _describe_stack_events(params):# --- DescribeStackResource ---+def _resource_status_reason_xml(res):+ """The ``ResourceStatusReason`` element of a stack resource, empty when+ the record carries none (a healthy resource has no reason on AWS)."""+ reason = res.get("ResourceStatusReason")+ if not reason:+ return ""+ return f"<ResourceStatusReason>{_esc(reason)}</ResourceStatusReason>"+++def _resource_metadata_xml(stack, logical_id):+ """The ``Metadata`` element of ``StackResourceDetail``: the resource's+ ``Metadata`` attribute as a JSON string, intrinsics resolved the way a+ property is (AWS interprets ``Ref``/``Fn::GetAtt`` inside it), empty when+ the template declares none."""+ template = stack.get("_template") or {}+ res_def = (template.get("Resources") or {}).get(logical_id) or {}+ metadata = res_def.get("Metadata")+ if metadata is None:+ return ""+ try:+ resolved = _resolve_refs(+ copy.deepcopy(metadata), stack.get("_resources", {}),+ stack.get("_resolved_params", {}), stack.get("_conditions", {}),+ template.get("Mappings", {}), stack.get("StackName", ""),+ stack.get("StackId", ""))+ # ``Metadata: {"Ref": "AWS::NoValue"}`` resolves the whole attribute+ # away; the literal is what the template declared.+ body = json.dumps(metadata if resolved is _NO_VALUE else resolved)+ except Exception as exc: # the literal is still better than nothing+ logger.warning("Metadata of %s left unresolved: %s", logical_id, exc)+ body = json.dumps(metadata)+ return f"<Metadata>{_esc(body)}</Metadata>"++def _describe_stack_resource(params):from ministack.services.cloudformation import _stacksstack_name = _p(params, "StackName")@@ -361,9 +396,11 @@ def _describe_stack_resource(params):f"<PhysicalResourceId>{_esc(res.get('PhysicalResourceId', ''))}</PhysicalResourceId>"f"<ResourceType>{_esc(res.get('ResourceType', ''))}</ResourceType>"f"<ResourceStatus>{res.get('ResourceStatus', '')}</ResourceStatus>"- f"<Timestamp>{res.get('Timestamp', '')}</Timestamp>"+ f"{_resource_status_reason_xml(res)}"+ f"<LastUpdatedTimestamp>{res.get('Timestamp', '')}</LastUpdatedTimestamp>"f"<StackName>{_esc(stack_name)}</StackName>"f"<StackId>{_esc(stack['StackId'])}</StackId>"+ f"{_resource_metadata_xml(stack, logical_id)}")return _xml(200, "DescribeStackResourceResponse",@@ -403,6 +440,7 @@ def _describe_stack_resources(params):f"<PhysicalResourceId>{_esc(res.get('PhysicalResourceId', ''))}</PhysicalResourceId>"f"<ResourceType>{_esc(res.get('ResourceType', ''))}</ResourceType>"f"<ResourceStatus>{res.get('ResourceStatus', '')}</ResourceStatus>"+ f"{_resource_status_reason_xml(res)}"f"<Timestamp>{res.get('Timestamp', '')}</Timestamp>"f"<StackName>{_esc(stack_name)}</StackName>"f"<StackId>{_esc(stack['StackId'])}</StackId>"diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.pyindex 09645a3b..7f679abe 100644--- a/ministack/services/cloudformation/provisioners.py+++ b/ministack/services/cloudformation/provisioners.py@@ -3007,8 +3007,14 @@ def _cfn_nested_stack_deploy(logical_id, props, parent_stack_name, *,old.get("Properties", {}),child_name, stale_id)except Exception as exc:- logger.warning("Nested-stack %s: failed to delete pruned %s: %s",- child_name, stale_id, exc)+ # As for a top-level stack: the resource stays in the child+ # stack as DELETE_FAILED so a later delete retries it.+ logger.error("Nested-stack %s: failed to delete pruned %s: %s",+ child_name, stale_id, exc)+ stale = provisioned.setdefault(stale_id, dict(old))+ stale["ResourceStatus"] = "DELETE_FAILED"+ stale["ResourceStatusReason"] = str(exc)+ stale["Timestamp"] = now_iso()resolved_outputs = []output_attrs: dict[str, str] = {}diff --git a/ministack/services/cloudformation/stacks.py b/ministack/services/cloudformation/stacks.pyindex e3c029b9..d03f46e4 100644--- a/ministack/services/cloudformation/stacks.py+++ b/ministack/services/cloudformation/stacks.py@@ -448,11 +448,19 @@ async def _deploy_stack_async(stack_name: str, stack_id: str, template: dict,# A cleanup miss doesn't fail the update — real CloudFormation# reports the resource DELETE_FAILED during the# UPDATE_COMPLETE_CLEANUP phase and still lands the stack in- # UPDATE_COMPLETE — but it must be visible, not a warning.+ # UPDATE_COMPLETE. The resource stays in the stack with that+ # status (it still exists in the service), so the next update+ # or the stack delete tries the delete again.logger.error("Failed to delete old resource %s: %s",logical_id, exc)_add_event(stack_id, stack_name, logical_id, rtype,"DELETE_FAILED", str(exc), pid)+ leftover = provisioned_resources.get(logical_id)+ if leftover is not None:+ leftover["ResourceStatus"] = "DELETE_FAILED"+ leftover["ResourceStatusReason"] = str(exc)+ leftover["Timestamp"] = now_iso()+ continueprovisioned_resources.pop(logical_id, None)await asyncio.sleep(0)@@ -627,6 +635,10 @@ async def _delete_stack_async(stack_name: str, stack_id: str,ordered = _topological_sort(res_defs, conditions) if res_defs else list(resources.keys())except ValueError:ordered = list(resources.keys())+ # A resource the template no longer declares but the stack still holds+ # (its cleanup delete failed on an earlier update) has no dependents left;+ # it goes first, so a retried delete reaches it.+ ordered += [lid for lid in resources if lid not in ordered]delete_failures = []for logical_id in reversed(ordered):diff --git a/tests/test_cfn.py b/tests/test_cfn.pyindex 1ec7c448..8cc108c9 100644--- a/tests/test_cfn.py+++ b/tests/test_cfn.py@@ -2362,6 +2362,75 @@ def test_cfn_describe_stack_resources_logical_id_filter(cfn, s3, sqs):assert exc_info.value.response["Error"]["Code"] == "ValidationError"+def test_cfn_describe_stack_resource_returns_the_metadata(cfn):+ """DescribeStackResource carries the resource's Metadata attribute as a+ JSON string with intrinsics interpreted, and follows the template after an+ update; a Metadata that resolves away or cannot be resolved is returned as+ declared; a resource without Metadata has no such field; a healthy resource+ has no ResourceStatusReason."""+ stack_name = f"cfn-resource-metadata-{_uuid_mod.uuid4().hex[:8]}"++ def template(path, rev):+ return {+ "AWSTemplateFormatVersion": "2010-09-09",+ "Parameters": {"Owner": {"Type": "String", "Default": "platform"}},+ "Resources": {+ "Queue": {+ "Type": "AWS::SQS::Queue",+ "Metadata": {+ "aws:cdk:path": path,+ "Owner": {"Ref": "Owner"},+ "Region": {"Ref": "AWS::Region"},+ "Nested": {"Flag": True, "List": [1, rev]},+ },+ },+ "Plain": {"Type": "AWS::SQS::Queue"},+ "Gone": {+ "Type": "AWS::SQS::Queue",+ "Metadata": {"Ref": "AWS::NoValue"},+ },+ "Broken": {+ "Type": "AWS::SQS::Queue",+ "Metadata": {"Fn::GetAtt": ["Plain", "NoSuchAttr"]},+ },+ },+ }++ def detail(logical_id):+ return cfn.describe_stack_resource(+ StackName=stack_name, LogicalResourceId=logical_id)["StackResourceDetail"]++ cfn.create_stack(StackName=stack_name,+ TemplateBody=json.dumps(template("ExampleStack/Queue/Resource", 2)))+ try:+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "CREATE_COMPLETE"+ queue = detail("Queue")+ assert json.loads(queue["Metadata"]) == {+ "aws:cdk:path": "ExampleStack/Queue/Resource",+ "Owner": "platform",+ "Region": "us-east-1",+ "Nested": {"Flag": True, "List": [1, 2]},+ }+ assert "ResourceStatusReason" not in queue+ assert queue["LastUpdatedTimestamp"]+ assert "Metadata" not in detail("Plain")+ # A Metadata that resolves away entirely, or one that cannot be+ # resolved, is returned as declared, not a failed request.+ assert json.loads(detail("Gone")["Metadata"]) == {"Ref": "AWS::NoValue"}+ assert json.loads(detail("Broken")["Metadata"]) == {+ "Fn::GetAtt": ["Plain", "NoSuchAttr"]}++ # After an update the new template's Metadata is what comes back.+ cfn.update_stack(StackName=stack_name,+ TemplateBody=json.dumps(template("ExampleStack/Queue/Resource/v2", 3)))+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "UPDATE_COMPLETE"+ updated = json.loads(detail("Queue")["Metadata"])+ assert updated["aws:cdk:path"] == "ExampleStack/Queue/Resource/v2"+ assert updated["Nested"] == {"Flag": True, "List": [1, 3]}+ finally:+ _delete_cfn_test_stack(cfn, stack_name)++def test_cfn_stack_id_addresses_every_read_action(cfn, sqs):"""Every action that takes a StackName accepts the stack id (what the CDKsends after its first DescribeStacks), and the responses carry the stack's@@ -10707,6 +10776,12 @@ def test_cfn_stack_delete_failure_lands_delete_failed(cfn, lam, sns):retained = cfn.describe_stack_resources(StackName=stack_name)["StackResources"]assert [r["LogicalResourceId"] for r in retained] == ["CR"]assert retained[0]["ResourceStatus"] == "DELETE_FAILED"+ assert fn in retained[0]["ResourceStatusReason"]+ detail = cfn.describe_stack_resource(+ StackName=stack_name, LogicalResourceId="CR")["StackResourceDetail"]+ assert detail["ResourceStatus"] == "DELETE_FAILED"+ assert fn in detail["ResourceStatusReason"]+ assert detail["LastUpdatedTimestamp"]# And the failure is visible as a stack-level event.events = cfn.describe_stack_events(StackName=stack_name)["StackEvents"]@@ -10790,6 +10865,153 @@ def test_cfn_stack_delete_failed_keeps_exports(cfn, lam):pass+def _stack_with_a_failed_cleanup_delete(cfn, lam, fn, stack_name, queue_name):+ """Create a stack (queue, SSM marker, custom resource whose handler+ refuses the Delete), then drop the custom resource from the template.+ Returns the template without it; ``Rev`` changes the marker's value."""+ lam.create_function(+ FunctionName=fn,+ Runtime="python3.12",+ Role=_CR_LAMBDA_ROLE,+ Handler="index.handler",+ Code={"ZipFile": _cr_make_zip(_CR_HANDLER_DELETE_FAILS)},+ )+ resources = {+ "Queue": {"Type": "AWS::SQS::Queue", "Properties": {"QueueName": queue_name}},+ "Marker": {+ "Type": "AWS::SSM::Parameter",+ "Properties": {"Name": f"/{stack_name}/rev", "Type": "String",+ "Value": {"Ref": "Rev"}},+ },+ }+ cr = {+ "Type": "Custom::Tester",+ "Properties": {+ "ServiceToken": f"arn:aws:lambda:us-east-1:000000000000:function:{fn}",+ },+ }+ without_cr = {+ "AWSTemplateFormatVersion": "2010-09-09",+ "Parameters": {"Rev": {"Type": "String", "Default": "1"}},+ "Resources": resources,+ }+ with_cr = json.loads(json.dumps(without_cr))+ with_cr["Resources"]["CR"] = cr+ cfn.create_stack(StackName=stack_name, TemplateBody=json.dumps(with_cr))+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "CREATE_COMPLETE", stack.get("StackStatusReason")+ cfn.update_stack(StackName=stack_name, TemplateBody=json.dumps(without_cr))+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")+ return without_cr+++def _swap_cr_handler(lam, fn, code):+ lam.delete_function(FunctionName=fn)+ lam.create_function(+ FunctionName=fn,+ Runtime="python3.12",+ Role=_CR_LAMBDA_ROLE,+ Handler="index.handler",+ Code={"ZipFile": _cr_make_zip(code)},+ )+++def test_cfn_update_cleanup_delete_failure_keeps_the_resource_visible(cfn, lam, ssm):+ """A resource dropped from the template whose delete fails during the+ cleanup phase stays in the stack as DELETE_FAILED with its reason: the+ update still ends UPDATE_COMPLETE (the new template is in effect), and the+ next update with a changed template retries the delete, which removes the+ resource once the delete works."""+ suffix = _uuid_mod.uuid4().hex[:8]+ fn = f"cr-cleanup-fail-{suffix}"+ stack_name = f"cfn-cleanup-failed-{suffix}"+ queue_name = f"cfn-cleanup-failed-q-{suffix}"+ try:+ without_cr = _stack_with_a_failed_cleanup_delete(cfn, lam, fn, stack_name, queue_name)++ # The failed cleanup is an event and a visible resource, not a log line.+ events = cfn.describe_stack_events(StackName=stack_name)["StackEvents"]+ assert any(e["LogicalResourceId"] == "CR" and e["ResourceStatus"] == "DELETE_FAILED"+ for e in events)+ by_id = {r["LogicalResourceId"]: r+ for r in cfn.describe_stack_resources(StackName=stack_name)["StackResources"]}+ assert set(by_id) == {"Queue", "Marker", "CR"}+ assert by_id["CR"]["ResourceStatus"] == "DELETE_FAILED"+ assert "delete refused for testing" in by_id["CR"]["ResourceStatusReason"]+ assert by_id["Queue"]["ResourceStatus"] == "UPDATE_COMPLETE"+ assert "ResourceStatusReason" not in by_id["Queue"]+ detail = cfn.describe_stack_resource(+ StackName=stack_name, LogicalResourceId="CR")["StackResourceDetail"]+ assert detail["ResourceStatus"] == "DELETE_FAILED"+ assert "delete refused for testing" in detail["ResourceStatusReason"]+ assert detail["LastUpdatedTimestamp"]+ # The new template is what the stack runs.+ assert "CR" not in cfn.get_template(StackName=stack_name)["TemplateBody"]["Resources"]++ # A changed template retries the delete; still refused, the resource+ # stays DELETE_FAILED and the update itself succeeds.+ cfn.update_stack(StackName=stack_name, TemplateBody=json.dumps(without_cr),+ Parameters=[{"ParameterKey": "Rev", "ParameterValue": "2"}])+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")+ assert ssm.get_parameter(Name=f"/{stack_name}/rev")["Parameter"]["Value"] == "2"+ by_id = {r["LogicalResourceId"]: r+ for r in cfn.describe_stack_resources(StackName=stack_name)["StackResources"]}+ assert by_id["CR"]["ResourceStatus"] == "DELETE_FAILED"+ assert "delete refused for testing" in by_id["CR"]["ResourceStatusReason"]++ # With a handler that accepts the delete, the next update removes it.+ _swap_cr_handler(lam, fn, _CR_HANDLER_SUCCESS)+ cfn.update_stack(StackName=stack_name, TemplateBody=json.dumps(without_cr),+ Parameters=[{"ParameterKey": "Rev", "ParameterValue": "3"}])+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")+ listed = {r["LogicalResourceId"]+ for r in cfn.describe_stack_resources(StackName=stack_name)["StackResources"]}+ assert listed == {"Queue", "Marker"}+ cfn.delete_stack(StackName=stack_name)+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "DELETE_COMPLETE"+ finally:+ _delete_cfn_test_stack(cfn, stack_name)+ try:+ lam.delete_function(FunctionName=fn)+ except ClientError:+ pass+++def test_cfn_delete_stack_retries_a_failed_cleanup_delete(cfn, lam, sqs):+ """DeleteStack reaches a resource the template no longer declares: the+ retry is refused again, the stack lands DELETE_FAILED naming it while the+ declared resources are gone, and a delete that works completes the stack."""+ suffix = _uuid_mod.uuid4().hex[:8]+ fn = f"cr-cleanup-del-{suffix}"+ stack_name = f"cfn-cleanup-delete-{suffix}"+ queue_name = f"cfn-cleanup-delete-q-{suffix}"+ try:+ _stack_with_a_failed_cleanup_delete(cfn, lam, fn, stack_name, queue_name)++ cfn.delete_stack(StackName=stack_name)+ stack = _wait_stack(cfn, stack_name)+ assert stack["StackStatus"] == "DELETE_FAILED"+ assert "CR" in stack.get("StackStatusReason", "")+ remaining = cfn.describe_stack_resources(StackName=stack_name)["StackResources"]+ assert [r["LogicalResourceId"] for r in remaining] == ["CR"]+ assert "delete refused for testing" in remaining[0]["ResourceStatusReason"]+ with pytest.raises(ClientError):+ sqs.get_queue_url(QueueName=queue_name)++ _swap_cr_handler(lam, fn, _CR_HANDLER_SUCCESS)+ cfn.delete_stack(StackName=stack_name)+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "DELETE_COMPLETE"+ finally:+ _delete_cfn_test_stack(cfn, stack_name)+ try:+ lam.delete_function(FunctionName=fn)+ except ClientError:+ pass++def test_cfn_create_rollback_delete_failure_lands_rollback_failed(cfn, lam):"""A rollback that cannot undo what it created reports ROLLBACK_FAILEDinstead of pretending the rollback completed."""