Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(cloudformation): a failed cleanup delete stays visible, DescribeStackResource carries Metadata and ResourceStatusReason (#1653)

iot-rocketbc4081acb1aa
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 364cdc9f..f06d6ec2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,9 +9,12 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Fixed
- **CloudFormation — `Fn::Cidr`, `Fn::GetAZs`, `Fn::FindInMap` and the condition functions follow AWS** — `Fn::Cidr` answered `10.0.{i}.0/…` whatever block it was given (now the block is split: `192.168.0.0/16, 2, 8` is `192.168.0.0/24,192.168.1.0/24`, measured; IPv6 included), `Fn::GetAZs` answered a fabricated `a/b/c` for any region (now the stack's own zones, and an empty list for another region, measured), `Fn::FindInMap` answered `""` for a missing key and had no `DefaultValue` (now the default when given, otherwise `Template error: Unable to get mapping for M::x::y` before the stack exists, measured), and `Fn::And`/`Fn::Or`/`Fn::Not` in a value position leaked the unresolved call (now a boolean; as an Output value they are refused with `The Value field of every Outputs member must evaluate to a String.`, measured). Contributed by @iot-rocket.
+- **CloudFormation — `DescribeStackResource` and `DescribeStackResources` carry `ResourceStatusReason`** — a resource in `DELETE_FAILED` (or any other failed state) kept its reason in the stack events only; the two Describe responses now carry the `ResourceStatusReason` element the API reference defines, absent for a healthy resource; `DescribeStackResource` also names its timestamp `LastUpdatedTimestamp`, the field `StackResourceDetail` defines (it carried a `Timestamp` element that clients drop, so the detail had no timestamp at all). Contributed by @iot-rocket.
+- **CloudFormation — a removed resource whose delete fails stays in the stack as `DELETE_FAILED`** — when a resource dropped from the template could not be deleted in the cleanup after an update, the stack ended `UPDATE_COMPLETE` (as on AWS) but the resource was dropped from the stack record while it kept existing in the service, and `DeleteStack` never tried again. It now stays listed with `DELETE_FAILED` and the reason (in a nested stack too), the next update with a changed template retries the delete, and `DeleteStack` also reaches the resources the template no longer declares. Contributed by @iot-rocket.
- **S3 — the Multi-Region Access Point alias starts with a letter and follows the documented pattern** — the alias was the first 13 hex characters of a uuid, so it could come out as digits only (`1240729989764.mrap`), which is not what S3 mints (`mfzwi23gnjvgw.mrap`) and made `test_cfn_s3_multi_region_access_point` fail whenever no letter was drawn (`str.islower()` is False without a cased character). The alias now follows the documented pattern `^[a-z][a-z0-9]*[.]mrap$`: a letter, then twelve lowercase letters or digits; the test checks that pattern. Contributed by @iot-rocket.
### Added
- **CloudFormation — `ListImports`, `UpdateTerminationProtection`, `SetStackPolicy`, `GetStackPolicy`, `CancelUpdateStack` and `ContinueUpdateRollback`** — the first four were routed to the service and answered `InvalidAction`, the last two did not exist, so a stack in `UPDATE_ROLLBACK_FAILED` had no way out short of a reset. `ListImports` names the stacks importing an export (a `ValidationError` when none does); termination protection is set on create or later and makes `DeleteStack` refuse with the stack unchanged (`DescribeStacks` reports `EnableTerminationProtection`); a stack policy is stored and returned (it is not enforced); `CancelUpdateStack` stops an `UPDATE_IN_PROGRESS` update before its next resource and rolls it back ("User Initiated"); `ContinueUpdateRollback` retries the deletes that failed the rollback, skips the `ResourcesToSkip` (their status becomes `UPDATE_COMPLETE`, the resource stays) and lands the stack in `UPDATE_ROLLBACK_COMPLETE`. Contributed by @iot-rocket.
+- **CloudFormation — `DescribeStackResource` returns the resource `Metadata`** — the `Metadata` attribute of a resource was accepted and never returned; `StackResourceDetail` now carries it as a JSON string with intrinsics interpreted, as the API reference defines it (a CDK stack's `aws:cdk:path` is the common case); a `Metadata` that resolves away or cannot be resolved comes back as declared. Contributed by @iot-rocket.
### Fixed
- **CloudFormation — parameter constraints are enforced** — `AllowedPattern`, `MinLength`, `MaxLength`, `MinValue` and `MaxValue` were ignored (only `AllowedValues` was checked), so a value a real account refuses deployed locally. They are checked before a stack exists, with CloudFormation's message (`Parameter 'P' must match pattern ^[a-z]+$`, measured) and a `ConstraintDescription` in place of the reason; a `CommaDelimitedList` is matched per member and a `List<Number>` is validated as numbers. Contributed by @iot-rocket.
- **CloudFormation — an unregistered `AWS::CloudFormation::*` type is unrecognized, not a silent no-op** — `AWS::CloudFormation::Macro`, `HookVersion`, `ModuleVersion` and any typo under that prefix deployed as a placeholder with a fabricated physical id, logged at INFO, while the README promises that unsupported types do not silently succeed. The pre-flight now treats them like every other unrecognized type (`Template format error: Unrecognized resource types`); the registered ones — `WaitCondition`, `WaitConditionHandle`, `Stack`, `CustomResource` — are unaffected. Contributed by @iot-rocket.
diff --git a/ministack/services/cloudformation/handlers.py b/ministack/services/cloudformation/handlers.py
index 84063882..227925de 100644
--- a/ministack/services/cloudformation/handlers.py
+++ b/ministack/services/cloudformation/handlers.py
@@ -18,6 +18,7 @@ from .changesets import (
_list_change_sets,
)
from .engine import (
+ _NO_VALUE,
_apply_sam_transform_if_applicable,
_evaluate_conditions,
_has_dynamic_references,
@@ -339,6 +340,40 @@ def _describe_stack_events(params):
# --- DescribeStackResource ---
+def _resource_status_reason_xml(res):
+ """The ``ResourceStatusReason`` element of a stack resource, empty when
+ the record carries none (a healthy resource has no reason on AWS)."""
+ reason = res.get("ResourceStatusReason")
+ if not reason:
+ return ""
+ return f"<ResourceStatusReason>{_esc(reason)}</ResourceStatusReason>"
+
+
+def _resource_metadata_xml(stack, logical_id):
+ """The ``Metadata`` element of ``StackResourceDetail``: the resource's
+ ``Metadata`` attribute as a JSON string, intrinsics resolved the way a
+ property is (AWS interprets ``Ref``/``Fn::GetAtt`` inside it), empty when
+ the template declares none."""
+ template = stack.get("_template") or {}
+ res_def = (template.get("Resources") or {}).get(logical_id) or {}
+ metadata = res_def.get("Metadata")
+ if metadata is None:
+ return ""
+ try:
+ resolved = _resolve_refs(
+ copy.deepcopy(metadata), stack.get("_resources", {}),
+ stack.get("_resolved_params", {}), stack.get("_conditions", {}),
+ template.get("Mappings", {}), stack.get("StackName", ""),
+ stack.get("StackId", ""))
+ # ``Metadata: {"Ref": "AWS::NoValue"}`` resolves the whole attribute
+ # away; the literal is what the template declared.
+ body = json.dumps(metadata if resolved is _NO_VALUE else resolved)
+ except Exception as exc: # the literal is still better than nothing
+ logger.warning("Metadata of %s left unresolved: %s", logical_id, exc)
+ body = json.dumps(metadata)
+ return f"<Metadata>{_esc(body)}</Metadata>"
+
+
def _describe_stack_resource(params):
from ministack.services.cloudformation import _stacks
stack_name = _p(params, "StackName")
@@ -361,9 +396,11 @@ def _describe_stack_resource(params):
f"<PhysicalResourceId>{_esc(res.get('PhysicalResourceId', ''))}</PhysicalResourceId>"
f"<ResourceType>{_esc(res.get('ResourceType', ''))}</ResourceType>"
f"<ResourceStatus>{res.get('ResourceStatus', '')}</ResourceStatus>"
- f"<Timestamp>{res.get('Timestamp', '')}</Timestamp>"
+ f"{_resource_status_reason_xml(res)}"
+ f"<LastUpdatedTimestamp>{res.get('Timestamp', '')}</LastUpdatedTimestamp>"
f"<StackName>{_esc(stack_name)}</StackName>"
f"<StackId>{_esc(stack['StackId'])}</StackId>"
+ f"{_resource_metadata_xml(stack, logical_id)}"
)
return _xml(200, "DescribeStackResourceResponse",
@@ -403,6 +440,7 @@ def _describe_stack_resources(params):
f"<PhysicalResourceId>{_esc(res.get('PhysicalResourceId', ''))}</PhysicalResourceId>"
f"<ResourceType>{_esc(res.get('ResourceType', ''))}</ResourceType>"
f"<ResourceStatus>{res.get('ResourceStatus', '')}</ResourceStatus>"
+ f"{_resource_status_reason_xml(res)}"
f"<Timestamp>{res.get('Timestamp', '')}</Timestamp>"
f"<StackName>{_esc(stack_name)}</StackName>"
f"<StackId>{_esc(stack['StackId'])}</StackId>"
diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.py
index 09645a3b..7f679abe 100644
--- a/ministack/services/cloudformation/provisioners.py
+++ b/ministack/services/cloudformation/provisioners.py
@@ -3007,8 +3007,14 @@ def _cfn_nested_stack_deploy(logical_id, props, parent_stack_name, *,
old.get("Properties", {}),
child_name, stale_id)
except Exception as exc:
- logger.warning("Nested-stack %s: failed to delete pruned %s: %s",
- child_name, stale_id, exc)
+ # As for a top-level stack: the resource stays in the child
+ # stack as DELETE_FAILED so a later delete retries it.
+ logger.error("Nested-stack %s: failed to delete pruned %s: %s",
+ child_name, stale_id, exc)
+ stale = provisioned.setdefault(stale_id, dict(old))
+ stale["ResourceStatus"] = "DELETE_FAILED"
+ stale["ResourceStatusReason"] = str(exc)
+ stale["Timestamp"] = now_iso()
resolved_outputs = []
output_attrs: dict[str, str] = {}
diff --git a/ministack/services/cloudformation/stacks.py b/ministack/services/cloudformation/stacks.py
index e3c029b9..d03f46e4 100644
--- a/ministack/services/cloudformation/stacks.py
+++ b/ministack/services/cloudformation/stacks.py
@@ -448,11 +448,19 @@ async def _deploy_stack_async(stack_name: str, stack_id: str, template: dict,
# A cleanup miss doesn't fail the update — real CloudFormation
# reports the resource DELETE_FAILED during the
# UPDATE_COMPLETE_CLEANUP phase and still lands the stack in
- # UPDATE_COMPLETE — but it must be visible, not a warning.
+ # UPDATE_COMPLETE. The resource stays in the stack with that
+ # status (it still exists in the service), so the next update
+ # or the stack delete tries the delete again.
logger.error("Failed to delete old resource %s: %s",
logical_id, exc)
_add_event(stack_id, stack_name, logical_id, rtype,
"DELETE_FAILED", str(exc), pid)
+ leftover = provisioned_resources.get(logical_id)
+ if leftover is not None:
+ leftover["ResourceStatus"] = "DELETE_FAILED"
+ leftover["ResourceStatusReason"] = str(exc)
+ leftover["Timestamp"] = now_iso()
+ continue
provisioned_resources.pop(logical_id, None)
await asyncio.sleep(0)
@@ -627,6 +635,10 @@ async def _delete_stack_async(stack_name: str, stack_id: str,
ordered = _topological_sort(res_defs, conditions) if res_defs else list(resources.keys())
except ValueError:
ordered = list(resources.keys())
+ # A resource the template no longer declares but the stack still holds
+ # (its cleanup delete failed on an earlier update) has no dependents left;
+ # it goes first, so a retried delete reaches it.
+ ordered += [lid for lid in resources if lid not in ordered]
delete_failures = []
for logical_id in reversed(ordered):
diff --git a/tests/test_cfn.py b/tests/test_cfn.py
index 1ec7c448..8cc108c9 100644
--- a/tests/test_cfn.py
+++ b/tests/test_cfn.py
@@ -2362,6 +2362,75 @@ def test_cfn_describe_stack_resources_logical_id_filter(cfn, s3, sqs):
assert exc_info.value.response["Error"]["Code"] == "ValidationError"
+def test_cfn_describe_stack_resource_returns_the_metadata(cfn):
+ """DescribeStackResource carries the resource's Metadata attribute as a
+ JSON string with intrinsics interpreted, and follows the template after an
+ update; a Metadata that resolves away or cannot be resolved is returned as
+ declared; a resource without Metadata has no such field; a healthy resource
+ has no ResourceStatusReason."""
+ stack_name = f"cfn-resource-metadata-{_uuid_mod.uuid4().hex[:8]}"
+
+ def template(path, rev):
+ return {
+ "AWSTemplateFormatVersion": "2010-09-09",
+ "Parameters": {"Owner": {"Type": "String", "Default": "platform"}},
+ "Resources": {
+ "Queue": {
+ "Type": "AWS::SQS::Queue",
+ "Metadata": {
+ "aws:cdk:path": path,
+ "Owner": {"Ref": "Owner"},
+ "Region": {"Ref": "AWS::Region"},
+ "Nested": {"Flag": True, "List": [1, rev]},
+ },
+ },
+ "Plain": {"Type": "AWS::SQS::Queue"},
+ "Gone": {
+ "Type": "AWS::SQS::Queue",
+ "Metadata": {"Ref": "AWS::NoValue"},
+ },
+ "Broken": {
+ "Type": "AWS::SQS::Queue",
+ "Metadata": {"Fn::GetAtt": ["Plain", "NoSuchAttr"]},
+ },
+ },
+ }
+
+ def detail(logical_id):
+ return cfn.describe_stack_resource(
+ StackName=stack_name, LogicalResourceId=logical_id)["StackResourceDetail"]
+
+ cfn.create_stack(StackName=stack_name,
+ TemplateBody=json.dumps(template("ExampleStack/Queue/Resource", 2)))
+ try:
+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "CREATE_COMPLETE"
+ queue = detail("Queue")
+ assert json.loads(queue["Metadata"]) == {
+ "aws:cdk:path": "ExampleStack/Queue/Resource",
+ "Owner": "platform",
+ "Region": "us-east-1",
+ "Nested": {"Flag": True, "List": [1, 2]},
+ }
+ assert "ResourceStatusReason" not in queue
+ assert queue["LastUpdatedTimestamp"]
+ assert "Metadata" not in detail("Plain")
+ # A Metadata that resolves away entirely, or one that cannot be
+ # resolved, is returned as declared, not a failed request.
+ assert json.loads(detail("Gone")["Metadata"]) == {"Ref": "AWS::NoValue"}
+ assert json.loads(detail("Broken")["Metadata"]) == {
+ "Fn::GetAtt": ["Plain", "NoSuchAttr"]}
+
+ # After an update the new template's Metadata is what comes back.
+ cfn.update_stack(StackName=stack_name,
+ TemplateBody=json.dumps(template("ExampleStack/Queue/Resource/v2", 3)))
+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "UPDATE_COMPLETE"
+ updated = json.loads(detail("Queue")["Metadata"])
+ assert updated["aws:cdk:path"] == "ExampleStack/Queue/Resource/v2"
+ assert updated["Nested"] == {"Flag": True, "List": [1, 3]}
+ finally:
+ _delete_cfn_test_stack(cfn, stack_name)
+
+
def test_cfn_stack_id_addresses_every_read_action(cfn, sqs):
"""Every action that takes a StackName accepts the stack id (what the CDK
sends after its first DescribeStacks), and the responses carry the stack's
@@ -10707,6 +10776,12 @@ def test_cfn_stack_delete_failure_lands_delete_failed(cfn, lam, sns):
retained = cfn.describe_stack_resources(StackName=stack_name)["StackResources"]
assert [r["LogicalResourceId"] for r in retained] == ["CR"]
assert retained[0]["ResourceStatus"] == "DELETE_FAILED"
+ assert fn in retained[0]["ResourceStatusReason"]
+ detail = cfn.describe_stack_resource(
+ StackName=stack_name, LogicalResourceId="CR")["StackResourceDetail"]
+ assert detail["ResourceStatus"] == "DELETE_FAILED"
+ assert fn in detail["ResourceStatusReason"]
+ assert detail["LastUpdatedTimestamp"]
# And the failure is visible as a stack-level event.
events = cfn.describe_stack_events(StackName=stack_name)["StackEvents"]
@@ -10790,6 +10865,153 @@ def test_cfn_stack_delete_failed_keeps_exports(cfn, lam):
pass
+def _stack_with_a_failed_cleanup_delete(cfn, lam, fn, stack_name, queue_name):
+ """Create a stack (queue, SSM marker, custom resource whose handler
+ refuses the Delete), then drop the custom resource from the template.
+ Returns the template without it; ``Rev`` changes the marker's value."""
+ lam.create_function(
+ FunctionName=fn,
+ Runtime="python3.12",
+ Role=_CR_LAMBDA_ROLE,
+ Handler="index.handler",
+ Code={"ZipFile": _cr_make_zip(_CR_HANDLER_DELETE_FAILS)},
+ )
+ resources = {
+ "Queue": {"Type": "AWS::SQS::Queue", "Properties": {"QueueName": queue_name}},
+ "Marker": {
+ "Type": "AWS::SSM::Parameter",
+ "Properties": {"Name": f"/{stack_name}/rev", "Type": "String",
+ "Value": {"Ref": "Rev"}},
+ },
+ }
+ cr = {
+ "Type": "Custom::Tester",
+ "Properties": {
+ "ServiceToken": f"arn:aws:lambda:us-east-1:000000000000:function:{fn}",
+ },
+ }
+ without_cr = {
+ "AWSTemplateFormatVersion": "2010-09-09",
+ "Parameters": {"Rev": {"Type": "String", "Default": "1"}},
+ "Resources": resources,
+ }
+ with_cr = json.loads(json.dumps(without_cr))
+ with_cr["Resources"]["CR"] = cr
+ cfn.create_stack(StackName=stack_name, TemplateBody=json.dumps(with_cr))
+ stack = _wait_stack(cfn, stack_name)
+ assert stack["StackStatus"] == "CREATE_COMPLETE", stack.get("StackStatusReason")
+ cfn.update_stack(StackName=stack_name, TemplateBody=json.dumps(without_cr))
+ stack = _wait_stack(cfn, stack_name)
+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")
+ return without_cr
+
+
+def _swap_cr_handler(lam, fn, code):
+ lam.delete_function(FunctionName=fn)
+ lam.create_function(
+ FunctionName=fn,
+ Runtime="python3.12",
+ Role=_CR_LAMBDA_ROLE,
+ Handler="index.handler",
+ Code={"ZipFile": _cr_make_zip(code)},
+ )
+
+
+def test_cfn_update_cleanup_delete_failure_keeps_the_resource_visible(cfn, lam, ssm):
+ """A resource dropped from the template whose delete fails during the
+ cleanup phase stays in the stack as DELETE_FAILED with its reason: the
+ update still ends UPDATE_COMPLETE (the new template is in effect), and the
+ next update with a changed template retries the delete, which removes the
+ resource once the delete works."""
+ suffix = _uuid_mod.uuid4().hex[:8]
+ fn = f"cr-cleanup-fail-{suffix}"
+ stack_name = f"cfn-cleanup-failed-{suffix}"
+ queue_name = f"cfn-cleanup-failed-q-{suffix}"
+ try:
+ without_cr = _stack_with_a_failed_cleanup_delete(cfn, lam, fn, stack_name, queue_name)
+
+ # The failed cleanup is an event and a visible resource, not a log line.
+ events = cfn.describe_stack_events(StackName=stack_name)["StackEvents"]
+ assert any(e["LogicalResourceId"] == "CR" and e["ResourceStatus"] == "DELETE_FAILED"
+ for e in events)
+ by_id = {r["LogicalResourceId"]: r
+ for r in cfn.describe_stack_resources(StackName=stack_name)["StackResources"]}
+ assert set(by_id) == {"Queue", "Marker", "CR"}
+ assert by_id["CR"]["ResourceStatus"] == "DELETE_FAILED"
+ assert "delete refused for testing" in by_id["CR"]["ResourceStatusReason"]
+ assert by_id["Queue"]["ResourceStatus"] == "UPDATE_COMPLETE"
+ assert "ResourceStatusReason" not in by_id["Queue"]
+ detail = cfn.describe_stack_resource(
+ StackName=stack_name, LogicalResourceId="CR")["StackResourceDetail"]
+ assert detail["ResourceStatus"] == "DELETE_FAILED"
+ assert "delete refused for testing" in detail["ResourceStatusReason"]
+ assert detail["LastUpdatedTimestamp"]
+ # The new template is what the stack runs.
+ assert "CR" not in cfn.get_template(StackName=stack_name)["TemplateBody"]["Resources"]
+
+ # A changed template retries the delete; still refused, the resource
+ # stays DELETE_FAILED and the update itself succeeds.
+ cfn.update_stack(StackName=stack_name, TemplateBody=json.dumps(without_cr),
+ Parameters=[{"ParameterKey": "Rev", "ParameterValue": "2"}])
+ stack = _wait_stack(cfn, stack_name)
+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")
+ assert ssm.get_parameter(Name=f"/{stack_name}/rev")["Parameter"]["Value"] == "2"
+ by_id = {r["LogicalResourceId"]: r
+ for r in cfn.describe_stack_resources(StackName=stack_name)["StackResources"]}
+ assert by_id["CR"]["ResourceStatus"] == "DELETE_FAILED"
+ assert "delete refused for testing" in by_id["CR"]["ResourceStatusReason"]
+
+ # With a handler that accepts the delete, the next update removes it.
+ _swap_cr_handler(lam, fn, _CR_HANDLER_SUCCESS)
+ cfn.update_stack(StackName=stack_name, TemplateBody=json.dumps(without_cr),
+ Parameters=[{"ParameterKey": "Rev", "ParameterValue": "3"}])
+ stack = _wait_stack(cfn, stack_name)
+ assert stack["StackStatus"] == "UPDATE_COMPLETE", stack.get("StackStatusReason")
+ listed = {r["LogicalResourceId"]
+ for r in cfn.describe_stack_resources(StackName=stack_name)["StackResources"]}
+ assert listed == {"Queue", "Marker"}
+ cfn.delete_stack(StackName=stack_name)
+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "DELETE_COMPLETE"
+ finally:
+ _delete_cfn_test_stack(cfn, stack_name)
+ try:
+ lam.delete_function(FunctionName=fn)
+ except ClientError:
+ pass
+
+
+def test_cfn_delete_stack_retries_a_failed_cleanup_delete(cfn, lam, sqs):
+ """DeleteStack reaches a resource the template no longer declares: the
+ retry is refused again, the stack lands DELETE_FAILED naming it while the
+ declared resources are gone, and a delete that works completes the stack."""
+ suffix = _uuid_mod.uuid4().hex[:8]
+ fn = f"cr-cleanup-del-{suffix}"
+ stack_name = f"cfn-cleanup-delete-{suffix}"
+ queue_name = f"cfn-cleanup-delete-q-{suffix}"
+ try:
+ _stack_with_a_failed_cleanup_delete(cfn, lam, fn, stack_name, queue_name)
+
+ cfn.delete_stack(StackName=stack_name)
+ stack = _wait_stack(cfn, stack_name)
+ assert stack["StackStatus"] == "DELETE_FAILED"
+ assert "CR" in stack.get("StackStatusReason", "")
+ remaining = cfn.describe_stack_resources(StackName=stack_name)["StackResources"]
+ assert [r["LogicalResourceId"] for r in remaining] == ["CR"]
+ assert "delete refused for testing" in remaining[0]["ResourceStatusReason"]
+ with pytest.raises(ClientError):
+ sqs.get_queue_url(QueueName=queue_name)
+
+ _swap_cr_handler(lam, fn, _CR_HANDLER_SUCCESS)
+ cfn.delete_stack(StackName=stack_name)
+ assert _wait_stack(cfn, stack_name)["StackStatus"] == "DELETE_COMPLETE"
+ finally:
+ _delete_cfn_test_stack(cfn, stack_name)
+ try:
+ lam.delete_function(FunctionName=fn)
+ except ClientError:
+ pass
+
+
def test_cfn_create_rollback_delete_failure_lands_rollback_failed(cfn, lam):
"""A rollback that cannot undo what it created reports ROLLBACK_FAILED
instead of pretending the rollback completed."""