Light, fluffy, and always free - The AWS Local Emulator alternative
feat(cloudformation): refuse updates after a failed rollback and add ContinueUpdateRollback (#4949)
Mario Toffiac4a809f47f81
diff --git a/docs/services/cloudformation.md b/docs/services/cloudformation.mdindex faffdf1914..5828867463 100644--- a/docs/services/cloudformation.md+++ b/docs/services/cloudformation.md@@ -12,6 +12,7 @@| `CreateStack` | Deploy a CloudFormation template || `UpdateStack` | Update an existing stack || `DeleteStack` | Delete a stack and its resources |+| `ContinueUpdateRollback` | Retry the rollback of a stack in `UPDATE_ROLLBACK_FAILED`, optionally skipping resources or forcing past every failed one (see [Failed update rollbacks](#failed-update-rollbacks)) || `UpdateTerminationProtection` | - || `CreateChangeSet` | Create a change set || `DescribeChangeSet` | Get change set details with a computed Add/Modify/Remove resource diff |@@ -127,10 +128,8 @@ owed cleanup while their retry budget remains. Exhausted historical records arelater updates and stack-delete retries. Historical cleanup is address-based and does not providean atomic ownership check against another stack or a direct Cognito API writer reusing the address.-Failed in-place restoration preserves the original name and scopes for a later update retry.-Deleting the current server keeps this snapshot if deletion fails. An update that skips a server-with pending restoration can remain in `UPDATE_COMPLETE_CLEANUP_IN_PROGRESS`, blocking another-update. `ContinueUpdateRollback` remains unsupported.+Failed in-place restoration preserves the original name and scopes for a `ContinueUpdateRollback`+retry. Deleting the current server keeps this snapshot if deletion fails.## Supported Resource Types@@ -494,6 +493,55 @@ was sent gets a `Delete` straight away for the same reason. An update that changreaches no handler: it fails with `Modifying service token is not allowed.`, as on AWS, and itsrollback sends nothing.+## Failed update rollbacks++When an update fails, Floci rolls back the resources it changed. A resource whose rollback fails+is left `UPDATE_FAILED`, and the stack settles in `UPDATE_ROLLBACK_FAILED`. As on AWS, `UpdateStack`+and an `UPDATE` change set are then refused with `ValidationError` (`Stack:<arn> is in+UPDATE_ROLLBACK_FAILED state and can not be updated.`). Two ways out remain:++- `DeleteStack` deletes the stack and its resources, the `UPDATE_FAILED` ones included (see+ [Deletion Policies](#deletion-policies)).+- `ContinueUpdateRollback` runs the rollback again for the resources still owed one, starting with+ an `UPDATE_ROLLBACK_IN_PROGRESS` stack event whose reason is `User Initiated`. A retry re-runs+ the restore of each resource still owed one, including the delete of a replacement whose earlier+ delete failed; a resource whose restore cannot be redone fails again, leaving the stack in+ `UPDATE_ROLLBACK_FAILED`, and can be passed in `ResourcesToSkip`. `ResourcesToSkip` names+ `UPDATE_FAILED` resources to report as `UPDATE_COMPLETE` with the reason `Resource skipped+ during UpdateRollback`, without rolling them back; a skipped resource keeps the state the failed+ update left it in. Once every other resource rolled back, the stack reaches+ `UPDATE_ROLLBACK_COMPLETE` with the template, parameters and outputs from before the failed+ update, and takes updates again.++`ForceRollback=true` skips every `UPDATE_FAILED` resource of the stack, each reported as a+`ResourcesToSkip` skip is. When a forced rollback skipped a resource and completes, the stack's+`StackStatusReason` and the reason of its `UPDATE_ROLLBACK_COMPLETE` event are `One or more+resources were skipped during UpdateRollback. These resources were not rolled back and may have+drifted from the template. Run drift detection (DetectStackDrift) to review their current state+before your next update.` A rollback completed with `ResourcesToSkip` has no stack reason, as on+AWS.++As on AWS, a malformed `ContinueUpdateRollback` is refused with `ValidationError` before the stack+is looked up, listing every violation: a `StackName` that is missing, empty, or does not match+`[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*`, and a `ResourcesToSkip` member that does not+match `[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\.[a-zA-Z0-9]+`. `ForceRollback=true` with a non-empty+`ResourcesToSkip` is then refused with `ForceRollback and ResourcesToSkip are mutually exclusive`,+a stack that does not exist with `Stack: <StackName> does not exist`, and a stack in any status+other than `UPDATE_ROLLBACK_FAILED` with `RollbackUpdatedStack cannot be called from current stack+status`. `ResourcesToSkip` is checked against the stack after the call is accepted: a skip that is+not a resource of the stack, or not `UPDATE_FAILED`, returns the stack to `UPDATE_ROLLBACK_FAILED`+with the reason, touching no resource, and a later call can still finish the rollback. `RoleARN`+and `ClientRequestToken` are accepted and have no effect.++Known difference: `ResourcesToSkip` does not support a nested stack's resource+(`NestedStackName.ResourceLogicalID`). When the prefix names a nested stack of the stack, the+rollback fails again with a reason saying Floci does not support it, where AWS would skip that+resource.++A stack that reached `UPDATE_ROLLBACK_FAILED` under an earlier Floci version kept no record of its+state before the update. `ContinueUpdateRollback` then retries every `UPDATE_FAILED` resource and+keeps the template the stack holds.+## Deleted StacksA deleted stack is kept for a short window afterwards rather than being forgotten atdiff --git a/docs/services/index.md b/docs/services/index.mdindex 9e6ff94391..f714e64613 100644--- a/docs/services/index.md+++ b/docs/services/index.md@@ -36,7 +36,7 @@ Operation counts are exact. For dispatch-table services (Query and JSON 1.1) eac| [Secrets Manager](secrets-manager.md) | `POST /` + `X-Amz-Target: secretsmanager.*` | JSON 1.1 | 23 || [Step Functions](step-functions.md) | `POST /` + `X-Amz-Target: AmazonStatesService.*` | JSON 1.1 | 32 || [SWF](swf.md) | `POST /` + `X-Amz-Target: SimpleWorkflowService.*` | JSON 1.0 | 39 |-| [CloudFormation](cloudformation.md) | `POST /` with `Action=` param | Query | 35 |+| [CloudFormation](cloudformation.md) | `POST /` with `Action=` param | Query | 36 || [Cloud Control API](cloudcontrol.md) | `POST /` + `X-Amz-Target: CloudApiService.*` | JSON 1.1 | 5 || [EventBridge](eventbridge.md) | `POST /` + `X-Amz-Target: AmazonEventBridge.*` | JSON 1.1 | 40 || [EventBridge Scheduler](scheduler.md) | `/schedules/*`, `/schedule-groups/*`, `/tags/*` | REST JSON | 12 |diff --git a/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java b/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.javaindex 93c690ca58..2c51c20fb1 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java@@ -581,6 +581,7 @@ public class AwsQueryController {private static final Set<String> CLOUDFORMATION_ACTIONS = Set.of("CreateStack", "DeleteStack", "UpdateStack", "DescribeStacks", "UpdateTerminationProtection",+ "ContinueUpdateRollback","ListStacks", "ListExports", "GetTemplate", "GetTemplateSummary", "ValidateTemplate","CreateChangeSet", "DeleteChangeSet", "DescribeChangeSet", "ExecuteChangeSet", "ListChangeSets","DescribeStackEvents", "DescribeStackResources", "ListStackResources", "DescribeStackResource",diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.javaindex eac7e9e14c..d42df99e6e 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java@@ -51,6 +51,7 @@ public class CloudFormationQueryHandler {case "CreateStack" -> createStack(params, region);case "UpdateStack" -> updateStack(params, region);case "DeleteStack" -> deleteStack(params, region);+ case "ContinueUpdateRollback" -> continueUpdateRollback(params, region);case "UpdateTerminationProtection" -> updateTerminationProtection(params, region);case "CreateChangeSet" -> createChangeSet(params, region);case "DescribeChangeSet" -> describeChangeSet(params, region);@@ -189,6 +190,28 @@ public class CloudFormationQueryHandler {}}+ // ── ContinueUpdateRollback ───────────────────────────────────────────────++ private Response continueUpdateRollback(MultivaluedMap<String, String> params, String region) {+ String stackName = params.getFirst("StackName");+ List<String> resourcesToSkip = extractList(params, "ResourcesToSkip.member.");+ boolean forceRollback = Boolean.parseBoolean(params.getFirst("ForceRollback"));+ // RoleARN and ClientRequestToken are accepted and have no effect: Floci assumes no role+ // for stack operations and does not deduplicate retried requests.+ try {+ cfnService.continueUpdateRollback(stackName, resourcesToSkip, forceRollback, region);+ } catch (AwsException e) {+ return xmlError(e.getErrorCode(), e.getMessage(), e.getHttpStatus());+ }+ String xml = new XmlBuilder()+ .start("ContinueUpdateRollbackResponse", CF_NS)+ .raw("<ContinueUpdateRollbackResult/>")+ .raw(AwsQueryResponse.responseMetadata())+ .end("ContinueUpdateRollbackResponse")+ .build();+ return Response.ok(xml).type("text/xml").build();+ }+// ── UpdateTerminationProtection ─────────────────────────────────────────────private Response updateTerminationProtection(MultivaluedMap<String, String> params, String region) {@@ -450,6 +473,7 @@ public class CloudFormationQueryHandler {.elem("PhysicalResourceId", res.getPhysicalId()).elem("ResourceType", res.getResourceType()).elem("ResourceStatus", res.getStatus())+ .elem("ResourceStatusReason", res.getStatusReason()).elem("LastUpdatedTimestamp", ISO.format(res.getTimestamp())).end("StackResourceDetail").end("DescribeStackResourceResult")diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.javaindex 6b9602cbe9..1c87a5e2a4 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java@@ -21,6 +21,7 @@ import io.github.hectorvent.floci.services.cloudformation.model.ChangeSet;import io.github.hectorvent.floci.services.cloudformation.model.Stack;import io.github.hectorvent.floci.services.cloudformation.model.StackEvent;import io.github.hectorvent.floci.services.cloudformation.model.StackResource;+import io.github.hectorvent.floci.services.cloudformation.model.StackUpdateSnapshot;import io.github.hectorvent.floci.services.cloudformation.model.TemplateSummary;import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnDynamicReferences;import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnResourceDispatcher;@@ -66,6 +67,30 @@ public class CloudFormationService implements ResourceProvider {private static final int MAX_OPERATION_THREADS = 16;private static final int MAX_QUEUED_OPERATIONS = 128;+ // Texts a client may match on, kept here so each has one place to change. All but+ // SKIP_NESTED_UNSUPPORTED are the texts real CloudFormation emits, measured against a real+ // account; that one is Floci's own, for a skip AWS accepts and Floci does not support.+ private static final String UPDATE_REFUSED = "Stack:%s is in %s state and can not be updated.";+ private static final String CONTINUE_ROLLBACK_REFUSED =+ "RollbackUpdatedStack cannot be called from current stack status";+ private static final String SKIP_NOT_UPDATE_FAILED =+ "Only the resources in UPDATE_FAILED state are allowed to be skipped";+ private static final String SKIP_NOT_IN_STACK = "Resources with logicalIds [%s] do not belong to stack %s";+ private static final String SKIP_NESTED_STACK_MISSING = "Stack [%s] does not exist";+ private static final String SKIP_NESTED_UNSUPPORTED =+ "Skipping a resource of a nested stack is not supported by Floci: %s";+ private static final String SKIPPED_RESOURCE_REASON = "Resource skipped during UpdateRollback";+ private static final String FORCED_ROLLBACK_COMPLETE_REASON =+ "One or more resources were skipped during UpdateRollback. These resources were not rolled"+ + " back and may have drifted from the template. Run drift detection (DetectStackDrift)"+ + " to review their current state before your next update.";+ private static final String FORCE_WITH_SKIPS = "ForceRollback and ResourcesToSkip are mutually exclusive";+ private static final String CONTINUE_STACK_MISSING = "Stack: %s does not exist";+ private static final Pattern CONTINUE_STACK_NAME =+ Pattern.compile("[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*");+ private static final Pattern RESOURCE_TO_SKIP =+ Pattern.compile("[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\\.[a-zA-Z0-9]+");+private final ConcurrentHashMap<String, Stack> stacks = new ConcurrentHashMap<>();private final ConcurrentHashMap<String, DeletedStackEntry> deletedStacks = new ConcurrentHashMap<>();// Account-scoped exports registry: account:region:exportName -> exportValue@@ -235,10 +260,15 @@ public class CloudFormationService implements ResourceProvider {* rather than by suffix, because {@code UPDATE_ROLLBACK_COMPLETE} ends the same way and is a* perfectly updatable stack: it is where an update that failed and rolled back settles, and* retrying the update is how it is repaired.+ *+ * <p>{@code UPDATE_ROLLBACK_FAILED} is refused because the rollback of the last update left+ * resources in an unknown state, and an update would build on them. ContinueUpdateRollback+ * finishes that rollback, and DeleteStack removes the stack; both stay allowed.*/static boolean refusesUpdate(String status) {return status != null- && (status.endsWith("_IN_PROGRESS") || "ROLLBACK_COMPLETE".equals(status));+ && (status.endsWith("_IN_PROGRESS") || "ROLLBACK_COMPLETE".equals(status)+ || "UPDATE_ROLLBACK_FAILED".equals(status));}/**@@ -470,8 +500,7 @@ public class CloudFormationService implements ResourceProvider {}private static AwsException cannotBeUpdated(String stackId, String status) {- return new AwsException("ValidationError",- "Stack:" + stackId + " is in " + status + " state and can not be updated.", 400);+ return new AwsException("ValidationError", UPDATE_REFUSED.formatted(stackId, status), 400);}// AWS refuses a deleted stack's ID by its state, while a name, or an ID it never issued,@@ -1050,6 +1079,91 @@ public class CloudFormationService implements ResourceProvider {}}+ // ── ContinueUpdateRollback ────────────────────────────────────────────────++ /**+ * Retries the rollback of a stack in {@code UPDATE_ROLLBACK_FAILED}, the only status AWS accepts+ * it from; any other status is refused here, before anything changes, as is a malformed request+ * or a forced rollback that also names skips. The status check and the move to+ * {@code UPDATE_ROLLBACK_IN_PROGRESS} are one step under the stack's lock, so a racing update or+ * second retry is refused. Everything else, the {@code resourcesToSkip} checks against the stack+ * included, runs on the operation executor, because AWS accepts the call and reports a bad skip+ * as the reason the rollback failed again.+ */+ public Future<?> continueUpdateRollback(+ String stackNameOrArn, List<String> resourcesToSkip, boolean forceRollback, String region) {+ validateContinueUpdateRollback(stackNameOrArn, resourcesToSkip);+ if (forceRollback && !resourcesToSkip.isEmpty()) {+ throw new AwsException("ValidationError", FORCE_WITH_SKIPS, 400);+ }+ String accountId = currentAccount();+ Stack found = resolveStack(stackNameOrArn, region, accountId);+ if (found == null) {+ throw new AwsException("ValidationError", CONTINUE_STACK_MISSING.formatted(stackNameOrArn), 400);+ }+ String stackName = found.getStackName();+ List<String> skips = List.copyOf(resourcesToSkip);+ StackMutationSnapshot[] snapshot = new StackMutationSnapshot[1];+ Stack stack = stacks.compute(stackKey(accountId, stackName, region), (k, existing) -> {+ if (existing == null) {+ throw new AwsException("ValidationError", CONTINUE_STACK_MISSING.formatted(stackNameOrArn), 400);+ }+ if (!"UPDATE_ROLLBACK_FAILED".equals(existing.getStatus())) {+ throw new AwsException("ValidationError", CONTINUE_ROLLBACK_REFUSED, 400);+ }+ snapshot[0] = snapshot(existing);+ existing.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ addEvent(existing, existing.getStackName(), existing.getStackId(),+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_IN_PROGRESS", "User Initiated");+ return existing;+ });+ try {+ persistStack(stack);+ return submitOperation(() -> RequestScopes.runAs(accountId, region,+ () -> resumeUpdateRollback(stack, skips, forceRollback, region)));+ } catch (AwsException e) {+ if ("LimitExceededException".equals(e.getErrorCode())) {+ Stack restored = restoreStack(accountId, stackName, region, snapshot[0]);+ if (restored != null) {+ persistStack(restored);+ }+ }+ throw e;+ }+ }++ /**+ * Refuses a malformed ContinueUpdateRollback request as AWS does, before the stack is looked up:+ * every constraint violation of {@code StackName} and {@code ResourcesToSkip} in one+ * {@code ValidationError}, with AWS's texts. A skip has no length limit.+ */+ private static void validateContinueUpdateRollback(String stackNameOrArn, List<String> resourcesToSkip) {+ List<String> violations = new ArrayList<>();+ if (stackNameOrArn == null) {+ violations.add("Value null at 'stackName' failed to satisfy constraint: Member must not be null");+ } else {+ if (stackNameOrArn.isEmpty()) {+ violations.add("Value '' at 'stackName' failed to satisfy constraint:"+ + " Member must have length greater than or equal to 1");+ }+ if (!CONTINUE_STACK_NAME.matcher(stackNameOrArn).matches()) {+ violations.add("Value '" + stackNameOrArn + "' at 'stackName' failed to satisfy constraint:"+ + " Member must satisfy regular expression pattern: " + CONTINUE_STACK_NAME.pattern());+ }+ }+ if (resourcesToSkip.stream().anyMatch(skip -> !RESOURCE_TO_SKIP.matcher(skip).matches())) {+ violations.add("Value '" + resourcesToSkip + "' at 'resourcesToSkip' failed to satisfy constraint:"+ + " Member must satisfy constraint: [Member must satisfy regular expression pattern: "+ + RESOURCE_TO_SKIP.pattern() + "]");+ }+ if (!violations.isEmpty()) {+ String prefix = violations.size() == 1+ ? "1 validation error detected: "+ : violations.size() + " validation errors detected: ";+ throw new AwsException("ValidationError", prefix + String.join("; ", violations), 400);+ }+ }+private StackMutationSnapshot snapshot(Stack stack) {Map<ChangeSet, ChangeSetState> states = new IdentityHashMap<>();for (ChangeSet changeSet : stack.changeSetsSnapshot().values()) {@@ -2115,6 +2229,109 @@ public class CloudFormationService implements ResourceProvider {}}+ /**+ * The body of ContinueUpdateRollback, on a stack already moved to UPDATE_ROLLBACK_IN_PROGRESS:+ * checks {@code resourcesToSkip}, reports each skipped resource UPDATE_COMPLETE, runs the+ * rollback walker again over the resources still owed a rollback, and finishes as the rollback+ * of the failed update does. A bad skip touches no resource: the stack returns to+ * UPDATE_ROLLBACK_FAILED with the reason, keeping what a later retry needs. A forced rollback+ * skips every UPDATE_FAILED resource of the stack instead, and says so when it completes.+ */+ void resumeUpdateRollback(Stack stack, List<String> resourcesToSkip, boolean forceRollback, String region) {+ try {+ List<String> skips = forceRollback+ ? stack.resourcesSnapshot().values().stream()+ .filter(resource -> "UPDATE_FAILED".equals(resource.getStatus()))+ .map(StackResource::getLogicalId)+ .toList()+ : resourcesToSkip;+ String invalidSkip = invalidSkipReason(stack, skips);+ if (invalidSkip != null) {+ // Reason and event before the status, which is the volatile publishing write.+ stack.setStatusReason(invalidSkip);+ addEvent(stack, stack.getStackName(), stack.getStackId(),+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_FAILED", invalidSkip);+ stack.setStatus("UPDATE_ROLLBACK_FAILED");+ persistStack(stack);+ LOG.warnv("Stack {0} update rollback not continued: {1}", stack.getStackName(), invalidSkip);+ return;+ }+ StackUpdateSnapshot previousState = stack.getUpdateRollbackSnapshot();+ List<String> owedResourceIds = stack.getUpdateRollbackResourceIds();+ if (previousState == null || owedResourceIds == null) {+ // A stack stored before the snapshot was kept: every UPDATE_FAILED resource is owed a+ // rollback, and the template the stack holds now is the one it keeps.+ previousState = snapshotForUpdate(stack);+ owedResourceIds = new ArrayList<>();+ for (StackResource previous : previousState.resources().values()) {+ if ("UPDATE_FAILED".equals(previous.getStatus())) {+ owedResourceIds.add(previous.getLogicalId());+ previous.setStatus("UPDATE_COMPLETE");+ previous.setStatusReason(null);+ }+ }+ }+ Set<String> owed = new LinkedHashSet<>(owedResourceIds);+ for (String logicalId : skips) {+ StackResource resource = stack.getResources().get(logicalId);+ resource.setStatus("UPDATE_COMPLETE");+ resource.setStatusReason(SKIPPED_RESOURCE_REASON);+ addEvent(stack, logicalId, resource.getPhysicalId(), resource.getResourceType(),+ "UPDATE_COMPLETE", SKIPPED_RESOURCE_REASON);+ owed.remove(logicalId);+ }+ // Recorded before and after the walker, so an unexpected failure never re-runs a skip or a done rollback.+ stack.setUpdateRollbackSnapshot(previousState);+ stack.setUpdateRollbackResourceIds(List.copyOf(owed));+ List<String> rollbackFailures = rollbackUpdatedResources(+ stack, previousState.resources(), owed, region);+ stack.setUpdateRollbackResourceIds(List.copyOf(rollbackFailures));+ finishUpdateRollback(stack, region, previousState, rollbackFailures,+ forceRollback && !skips.isEmpty() ? FORCED_ROLLBACK_COMPLETE_REASON : null);+ } catch (Exception e) {+ // Nobody reads this worker's future: a stack left UPDATE_ROLLBACK_IN_PROGRESS here could+ // never be continued. The snapshot and the resources still owed a rollback are kept.+ LOG.errorv("Stack {0} update rollback could not continue: {1}",+ stack.getStackName(), e.getMessage());+ stack.setStatusReason(e.getMessage());+ addEvent(stack, stack.getStackName(), stack.getStackId(),+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_FAILED", e.getMessage());+ stack.setStatus("UPDATE_ROLLBACK_FAILED");+ persistStack(stack);+ }+ }++ /**+ * Why AWS would fail the rollback over {@code resourcesToSkip}, or {@code null} when every+ * skip is an UPDATE_FAILED resource of this stack. AWS names a nested stack's resource+ * {@code NestedStackName.ResourceLogicalID}; Floci does not support skipping one.+ */+ private static String invalidSkipReason(Stack stack, List<String> resourcesToSkip) {+ List<String> notInStack = new ArrayList<>();+ boolean notUpdateFailed = false;+ for (String logicalId : resourcesToSkip) {+ int dot = logicalId.indexOf('.');+ if (dot >= 0) {+ String nestedStackId = logicalId.substring(0, dot);+ StackResource nested = stack.getResources().get(nestedStackId);+ if (nested == null || !"AWS::CloudFormation::Stack".equals(nested.getResourceType())) {+ return SKIP_NESTED_STACK_MISSING.formatted(nestedStackId);+ }+ return SKIP_NESTED_UNSUPPORTED.formatted(logicalId);+ }+ StackResource resource = stack.getResources().get(logicalId);+ if (resource == null) {+ notInStack.add(logicalId);+ } else if (!"UPDATE_FAILED".equals(resource.getStatus())) {+ notUpdateFailed = true;+ }+ }+ if (!notInStack.isEmpty()) {+ return SKIP_NOT_IN_STACK.formatted(String.join(", ", notInStack), stack.getStackName());+ }+ return notUpdateFailed ? SKIP_NOT_UPDATE_FAILED : null;+ }+private void rollbackFailedUpdate(Stack stack,String region,@@ -2128,6 +2345,23 @@ public class CloudFormationService implements ResourceProvider {List<String> rollbackFailures = rollbackUpdatedResources(stack, previousState.resources(), attemptedResourceIds, region);+ finishUpdateRollback(stack, region, previousState, rollbackFailures, null);+ }++ /**+ * Ends an update rollback, the first one and every ContinueUpdateRollback retry alike: restores+ * the parameters, outputs and exports, and the template once every resource rolled back, then+ * settles the stack in UPDATE_ROLLBACK_COMPLETE, with {@code completionReason} as its reason, or+ * UPDATE_ROLLBACK_FAILED. A rollback that fails keeps the snapshot and the resources still owed+ * a rollback on the stack for a retry.+ */+ private void finishUpdateRollback(+ Stack stack,+ String region,+ StackUpdateSnapshot previousState,+ List<String> rollbackFailures,+ String completionReason) {+ List<String> owedResourceIds = List.copyOf(rollbackFailures);// Parameters are independent of resource-rollback outcome - always restore them to the last// successfully deployed values, even when resource rollback itself fails and the stack lands// in UPDATE_ROLLBACK_FAILED, so DescribeStacks and later change-set previews don't keep@@ -2152,19 +2386,24 @@ public class CloudFormationService implements ResourceProvider {}stack.setLastUpdatedTime(now());if (rollbackFailures.isEmpty()) {- stack.setStatus("UPDATE_ROLLBACK_COMPLETE");- stack.setStatusReason(null);+ stack.setUpdateRollbackSnapshot(null);+ stack.setUpdateRollbackResourceIds(null);+ // Reason and event before the status, which is the volatile publishing write.+ stack.setStatusReason(completionReason);addEvent(stack, stack.getStackName(), stack.getStackId(),- "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_COMPLETE", null);+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_COMPLETE", completionReason);+ stack.setStatus("UPDATE_ROLLBACK_COMPLETE");LOG.infov("Stack {0} update rolled back (UPDATE_ROLLBACK_COMPLETE)",stack.getStackName());} else {String reason = "The following resource(s) failed to roll back: ["+ String.join(", ", rollbackFailures) + "].";- stack.setStatus("UPDATE_ROLLBACK_FAILED");+ stack.setUpdateRollbackSnapshot(previousState);+ stack.setUpdateRollbackResourceIds(owedResourceIds);stack.setStatusReason(reason);addEvent(stack, stack.getStackName(), stack.getStackId(),"AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_FAILED", reason);+ stack.setStatus("UPDATE_ROLLBACK_FAILED");LOG.errorv("Stack {0} update rollback failed: {1}", stack.getStackName(), reason);}persistStack(stack);@@ -2382,17 +2621,6 @@ public class CloudFormationService implements ResourceProvider {return copy;}- private record StackUpdateSnapshot(- String templateBody,- String originalTemplateBody,- Map<String, String> parameters,- Map<String, String> resolvedParameters,- Map<String, String> outputs,- Map<String, String> exports,- Map<String, String> outputExportNames,- Map<String, StackResource> resources) {- }-private record UpdateCleanupFailure(String logicalId, String physicalId, String failureReason) {}@@ -2593,6 +2821,8 @@ public class CloudFormationService implements ResourceProvider {throw new IllegalStateException(reason);}+ stack.setUpdateRollbackSnapshot(null);+ stack.setUpdateRollbackResourceIds(null);// Before the status, never after. Status is the volatile publishing write, so a reader// that observes DELETE_COMPLETE is guaranteed to observe every write that preceded it.// Assigned afterwards, a concurrent DescribeStacks can report the terminal status withdiff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.javaindex b8c2b0517a..14f0e4d0d9 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java@@ -64,6 +64,14 @@ public class Stack {private Map<String, ChangeSet> changeSets = orderedMap();private Map<String, String> tags = orderedMap();private boolean enableTerminationProtection = false;+ /**+ * Set when an update rollback fails, and kept until a rollback completes or the stack is+ * deleted: the state from before the failed update, and the logical ids whose rollback has not+ * succeeded yet. A ContinueUpdateRollback retry reads them in UPDATE_ROLLBACK_IN_PROGRESS, so+ * that transition must keep them. Absent on a stack stored before they were kept.+ */+ private StackUpdateSnapshot updateRollbackSnapshot;+ private List<String> updateRollbackResourceIds;public String getStackId() { return stackId; }public void setStackId(String stackId) { this.stackId = stackId; }@@ -187,4 +195,8 @@ public class Stack {}public boolean isEnableTerminationProtection() { return enableTerminationProtection; }public void setEnableTerminationProtection(boolean enableTerminationProtection) { this.enableTerminationProtection = enableTerminationProtection; }+ public StackUpdateSnapshot getUpdateRollbackSnapshot() { return updateRollbackSnapshot; }+ public void setUpdateRollbackSnapshot(StackUpdateSnapshot updateRollbackSnapshot) { this.updateRollbackSnapshot = updateRollbackSnapshot; }+ public List<String> getUpdateRollbackResourceIds() { return updateRollbackResourceIds; }+ public void setUpdateRollbackResourceIds(List<String> updateRollbackResourceIds) { this.updateRollbackResourceIds = updateRollbackResourceIds; }}diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.javanew file mode 100644index 0000000000..cce509008f--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.java@@ -0,0 +1,24 @@+package io.github.hectorvent.floci.services.cloudformation.model;++import com.fasterxml.jackson.annotation.JsonIgnoreProperties;+import io.quarkus.runtime.annotations.RegisterForReflection;++import java.util.Map;++/**+ * A stack's state from before an update, which the rollback of that update restores. A rollback+ * that fails keeps it on the stack, so ContinueUpdateRollback can run the rollback again after the+ * update that took it has returned, or after a restart.+ */+@RegisterForReflection+@JsonIgnoreProperties(ignoreUnknown = true)+public record StackUpdateSnapshot(+ String templateBody,+ String originalTemplateBody,+ Map<String, String> parameters,+ Map<String, String> resolvedParameters,+ Map<String, String> outputs,+ Map<String, String> exports,+ Map<String, String> outputExportNames,+ Map<String, StackResource> resources) {+}diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.javaindex 101785cc10..2437f92f07 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java@@ -23,9 +23,10 @@ import java.util.Set;* or put back when a later resource fails the update and it rolls back.** <p>The record on the resource lists every entity owed a delete, not only the one this update- * displaced: a replacement whose rollback could not delete it stays listed, so the next committed- * update or the stack delete removes it instead of it being forgotten. The prior entity's id and- * attributes are kept beside the list only while this update's replacement can still be rolled back.+ * displaced: a replacement whose rollback could not delete it stays listed, so a retried rollback,+ * the next committed update or the stack delete removes it instead of it being forgotten. The prior+ * entity's id and attributes are kept beside the list only while this update's replacement can+ * still be rolled back.** <p>{@code PipesCfnProvisioner} carries its own richer record because it also snapshots the* entity's configuration for in-place rollback; this is the plain version for types without one.@@ -70,6 +71,7 @@ public final class ReplacementCleanup {boolean replaced) {ObjectNode cleanup = readOrEmpty(r);cleanup.remove("priorPhysicalId");+ cleanup.remove("undeletedReplacement");cleanup.remove("priorAttributes");cleanup.put("region", ctx.region());String prior = ctx.priorPhysicalId();@@ -99,11 +101,28 @@ public final class ReplacementCleanup {* standing again, and a resource still owing its delete would put it on the next cleanup's list.* A replacement whose delete fails is listed in its place, never retained (a failed update* created it), so the next committed update or the stack delete removes it, and the failure- * propagates so the stack reports it. Returns false when this update replaced nothing, which+ * propagates so the stack reports it. A retry (CloudFormation {@code ContinueUpdateRollback})+ * re-attempts that failed delete. Returns false when this update replaced nothing, which* leaves the engine's handling of in-place updates unchanged.*/static boolean rollback(StackResource r, Deleter deleter) {ObjectNode cleanup = read(r);+ String undeleted = cleanup == null ? null : cleanup.path("undeletedReplacement").asText(null);+ if (undeleted != null) {+ JsonNode owed = null;+ for (JsonNode entry : cleanup.path("displaced")) {+ if (undeleted.equals(entry.path("physicalId").asText(null))) {+ owed = entry;+ }+ }+ if (owed == null) {+ cleanup.remove("undeletedReplacement");+ write(r, cleanup);+ return true;+ }+ return deleteReplacement(r, cleanup, deleter, owed.path("resourceType").asText(r.getResourceType()),+ undeleted);+ }if (cleanup == null || cleanup.path("priorPhysicalId").asText(null) == null) {return false;}@@ -125,13 +144,23 @@ public final class ReplacementCleanup {if (prior.equals(replacement) && !CfnResourceDispatcher.isStub(prior, r.getAttributes())) {return true;}+ return deleteReplacement(r, cleanup, deleter, r.getResourceType(), replacement);+ }++ /** Deletes a rolled-back replacement; one whose delete throws stays owed and named for the retry. */+ private static boolean deleteReplacement(StackResource r, ObjectNode cleanup, Deleter deleter,+ String resourceType, String replacement) {try {- deleter.delete(r.getResourceType(), replacement, region(cleanup));+ deleter.delete(resourceType, replacement, region(cleanup));} catch (RuntimeException deleteFailure) {- addDisplaced(cleanup, replacement, r.getResourceType(), region(cleanup), false);+ addDisplaced(cleanup, replacement, resourceType, region(cleanup), false);+ cleanup.put("undeletedReplacement", replacement);write(r, cleanup);throw deleteFailure;}+ removeDisplaced(cleanup, replacement);+ cleanup.remove("undeletedReplacement");+ write(r, cleanup);return true;}diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.javanew file mode 100644index 0000000000..93cb00cdb3--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.java@@ -0,0 +1,585 @@+package io.github.hectorvent.floci.services.cloudformation;++import io.github.hectorvent.floci.core.common.XmlParser;+import io.github.hectorvent.floci.services.cloudwatch.dashboards.CloudWatchDashboardsService;+import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.mockito.InjectSpy;+import io.restassured.RestAssured;+import io.restassured.config.EncoderConfig;+import io.restassured.http.ContentType;+import io.restassured.response.Response;+import io.restassured.specification.RequestSpecification;+import org.junit.jupiter.api.Test;+import org.mockito.ArgumentMatcher;+import org.mockito.Mockito;++import java.util.List;+import java.util.Map;++import static io.restassured.RestAssured.given;+import static org.hamcrest.Matchers.containsString;+import static org.hamcrest.Matchers.not;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.anyMap;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.argThat;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;++/**+ * A stack whose update rollback failed sits in UPDATE_ROLLBACK_FAILED. AWS refuses to update it;+ * the ways out are ContinueUpdateRollback, optionally skipping the resources that cannot roll+ * back, and DeleteStack. Each test reaches that state as a real deploy does: an update changes a+ * log group, which has no update rollback, then adds a second log group whose name collides with+ * one made outside the stack.+ *+ * <p>The test module has no AWS SDK CloudFormation client, so the Query protocol is driven with+ * form POSTs, as the other CloudFormation integration tests do.+ */+@QuarkusTest+class CloudFormationContinueUpdateRollbackIntegrationTest {++ private static final String SKIPPED = "Resource skipped during UpdateRollback";+ private static final String USER_INITIATED = "User Initiated";+ private static final String FORCED = "One or more resources were skipped during UpdateRollback."+ + " These resources were not rolled back and may have drifted from the template."+ + " Run drift detection (DetectStackDrift) to review their current state before your next update.";+ private static final String STACK_NAME_PATTERN = "[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*";+ private static final String SKIP_PATTERN = "[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\\.[a-zA-Z0-9]+";+ private static final String NULL_STACK_NAME =+ "Value null at 'stackName' failed to satisfy constraint: Member must not be null";+ private static final String CW_AUTH =+ "AWS4-HMAC-SHA256 Credential=test/20260908/us-east-1/monitoring/aws4_request";++ @InjectSpy+ CloudWatchDashboardsService dashboardsService;++ @Test+ void updateAndChangeSetAreRefusedWhileDeleteStillWorks() {+ RollbackFailedStack stack = rollbackFailedStack("cur-refuse-");+ try {+ cfn("UpdateStack", stack.name())+ .formParam("TemplateBody", template(stack, 14, false))+ .when()+ .post("/")+ .then()+ .statusCode(400)+ .body(containsString("<Code>ValidationError</Code>"))+ .body(containsString("<Message>Stack:" + stack.id()+ + " is in UPDATE_ROLLBACK_FAILED state and can not be updated.</Message>"));++ cfn("CreateChangeSet", stack.name())+ .formParam("ChangeSetName", "cur-refuse-cs")+ .formParam("ChangeSetType", "UPDATE")+ .formParam("TemplateBody", template(stack, 14, false))+ .when()+ .post("/")+ .then()+ .statusCode(400)+ .body(containsString("<Code>ValidationError</Code>"))+ .body(containsString("<Message>Stack:" + stack.id()+ + " is in UPDATE_ROLLBACK_FAILED state and can not be updated.</Message>"));++ assertEquals("UPDATE_ROLLBACK_FAILED", CfnStackWaits.awaitTerminal(stack.name()).status());++ cfn("DeleteStack", stack.name()).when().post("/").then().statusCode(200);+ CfnStackWaits.awaitStackDeleted(stack.name());+ } finally {+ cleanUp(stack);+ }+ }++ /** A log group cannot roll an update back, so retrying the rollback fails the same way again. */+ @Test+ void continueUpdateRollbackWithoutSkipsFailsAgain() {+ RollbackFailedStack stack = rollbackFailedStack("cur-retry-");+ try {+ cfn("ContinueUpdateRollback", stack.name())+ .when()+ .post("/")+ .then()+ .statusCode(200)+ .body(containsString("<ContinueUpdateRollbackResponse"+ + " xmlns=\"http://cloudformation.amazonaws.com/doc/2010-05-15/\">"+ + "<ContinueUpdateRollbackResult/><ResponseMetadata><RequestId>"));++ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());+ assertTrue(state.reason().contains("LogGroup"), state.reason());+ assertEquals("UPDATE_FAILED", resourceStatuses(stack.name()).get("LogGroup"));++ List<Map<String, String>> events = events(stack.id());+ List<Map<String, String>> rollbacks = events.stream()+ .filter(event -> stack.name().equals(event.get("LogicalResourceId"))+ && "UPDATE_ROLLBACK_IN_PROGRESS".equals(event.get("ResourceStatus")))+ .toList();+ assertEquals(2, rollbacks.size(), events.toString());+ assertEquals(USER_INITIATED, rollbacks.get(0).get("ResourceStatusReason"), events.toString());++ // The retry restores nothing it could not restore the first time, so the failed+ // template stays.+ getTemplate(stack.name()).then().statusCode(200).body(containsString(stack.outsideGroup()));+ } finally {+ cleanUp(stack);+ }+ }++ @Test+ void skippingTheLogGroupCompletesTheRollbackAndUnblocksUpdates() {+ RollbackFailedStack stack = rollbackFailedStack("cur-skip-");+ try {+ cfn("ContinueUpdateRollback", stack.id())+ .formParam("ResourcesToSkip.member.1", "LogGroup")+ .formParam("RoleARN", "arn:aws:iam::000000000000:role/ignored")+ .formParam("ClientRequestToken", "cur-skip-token")+ .when()+ .post("/")+ .then()+ .statusCode(200);++ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());+ cfn("DescribeStackResource", stack.name())+ .formParam("LogicalResourceId", "LogGroup")+ .when()+ .post("/")+ .then()+ .statusCode(200)+ .body(containsString("<ResourceStatus>UPDATE_COMPLETE</ResourceStatus>"))+ .body(containsString("<ResourceStatusReason>" + SKIPPED + "</ResourceStatusReason>"));++ // Newest first: the skipped resource's UPDATE_COMPLETE follows the second rollback start.+ List<Map<String, String>> events = events(stack.id());+ boolean skipped = false;+ for (Map<String, String> event : events) {+ if (stack.name().equals(event.get("LogicalResourceId"))+ && "UPDATE_ROLLBACK_IN_PROGRESS".equals(event.get("ResourceStatus"))) {+ assertEquals(USER_INITIATED, event.get("ResourceStatusReason"), event.toString());+ break;+ }+ if ("LogGroup".equals(event.get("LogicalResourceId"))+ && "UPDATE_COMPLETE".equals(event.get("ResourceStatus"))+ && SKIPPED.equals(event.get("ResourceStatusReason"))+ && stack.logGroup().equals(event.get("PhysicalResourceId"))) {+ skipped = true;+ }+ }+ assertTrue(skipped, "no skipped UPDATE_COMPLETE event for LogGroup in " + events);++ // The completed rollback restores the template from before the failed update.+ getTemplate(stack.name()).then().statusCode(200).body(not(containsString(stack.outsideGroup())));++ // The stack is no longer UPDATE_ROLLBACK_FAILED, so a second call is refused.+ assertContinueRefused(stack.name());++ cfn("UpdateStack", stack.name())+ .formParam("TemplateBody", template(stack, 14, false))+ .when()+ .post("/")+ .then()+ .statusCode(200);+ assertEquals("UPDATE_COMPLETE", CfnStackWaits.awaitTerminal(stack.name()).status());+ } finally {+ cleanUp(stack);+ }+ }++ @Test+ void continueUpdateRollbackIsRefusedOnAStackInAnotherState() {+ String suffix = Long.toString(System.nanoTime(), 36);+ String stackName = "cur-state-" + suffix;+ String group = "/cur-state/" + suffix;+ try {+ cfn("CreateStack", stackName)+ .formParam("TemplateBody", "{\"Resources\":{\"LogGroup\":{\"Type\":\"AWS::Logs::LogGroup\","+ + "\"Properties\":{\"LogGroupName\":\"" + group + "\"}}}}")+ .when().post("/").then().statusCode(200);+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());++ assertContinueRefused(stackName);++ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());+ } finally {+ cfn("DeleteStack", stackName).when().post("/");+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + group + "\"}");+ }+ }++ /**+ * AWS accepts a call with a bad skip and fails the rollback again with the reason, touching no+ * resource. What the retry needs is kept, so a later call with a good skip still finishes it.+ */+ @Test+ void aBadSkipFailsTheRollbackAgainWithoutTouchingAResource() {+ RollbackFailedStack stack = rollbackFailedStack("cur-badskip-");+ try {+ assertSkipFails(stack, "Missing",+ "Resources with logicalIds [Missing] do not belong to stack " + stack.name());+ assertSkipFails(stack, "Stable", "Only the resources in UPDATE_FAILED state are allowed to be skipped");+ assertSkipFails(stack, "Nested.LogGroup", "Stack [Nested] does not exist");++ Map<String, String> statuses = resourceStatuses(stack.name());+ assertEquals("UPDATE_FAILED", statuses.get("LogGroup"), statuses.toString());+ assertEquals("CREATE_COMPLETE", statuses.get("Stable"), statuses.toString());++ cfn("ContinueUpdateRollback", stack.name())+ .formParam("ResourcesToSkip.member.1", "LogGroup")+ .when()+ .post("/")+ .then()+ .statusCode(200);+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());+ getTemplate(stack.name()).then().statusCode(200).body(not(containsString(stack.outsideGroup())));+ } finally {+ cleanUp(stack);+ }+ }++ /**+ * ContinueUpdateRollback is a real retry: it runs the restore of a resource still owed one+ * again. A dashboard keeps its update snapshot until its restore succeeds, so a restore that+ * fails once is put right by the retry.+ */+ @Test+ void continueUpdateRollbackRetriesARestorationThatThrew() {+ String suffix = Long.toString(System.nanoTime(), 36);+ String stackName = "cur-dashboard-retry-" + suffix;+ String dashboardName = "cur-dashboard-retry-" + suffix;+ try {+ cfn("CreateStack", stackName)+ .formParam("TemplateBody", dashboardTemplate(dashboardName, "Before", false))+ .when().post("/").then().statusCode(200);+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());++ // The create wrote the pre-update body too; only the restores are counted below.+ Mockito.clearInvocations(dashboardsService);+ ArgumentMatcher<String> preUpdateBody = body -> body != null && body.contains("Before");+ Mockito.doThrow(new IllegalStateException("simulated restore failure"))+ .doCallRealMethod()+ .when(dashboardsService)+ .putDashboard(eq(dashboardName), argThat(preUpdateBody), anyMap(), anyString());++ cfn("UpdateStack", stackName)+ .formParam("TemplateBody", dashboardTemplate(dashboardName, "After", true))+ .when().post("/").then().statusCode(200);+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stackName);+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());+ assertEquals("UPDATE_FAILED", resourceStatuses(stackName).get("Dashboard"));++ cfn("ContinueUpdateRollback", stackName).when().post("/").then().statusCode(200);+ state = CfnStackWaits.awaitTerminal(stackName);+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());+ assertEquals("CREATE_COMPLETE", resourceStatuses(stackName).get("Dashboard"));+ verify(dashboardsService, times(2))+ .putDashboard(eq(dashboardName), argThat(preUpdateBody), anyMap(), anyString());++ String body = given()+ .contentType("application/x-www-form-urlencoded")+ .header("Authorization", CW_AUTH)+ .formParam("Action", "GetDashboard")+ .formParam("DashboardName", dashboardName)+ .when()+ .post("/")+ .then()+ .statusCode(200)+ .extract().path("GetDashboardResponse.GetDashboardResult.DashboardBody");+ assertTrue(body.contains("Before") && !body.contains("After"), body);+ } finally {+ cfn("DeleteStack", stackName).when().post("/");+ }+ }++ @Test+ void aMissingStackNameIsAValidationError() {+ assertValidationError(cfn("ContinueUpdateRollback", null),+ "1 validation error detected: " + NULL_STACK_NAME);+ }++ @Test+ void anEmptyStackNameFailsBothItsLengthAndItsPattern() {+ assertValidationError(cfn("ContinueUpdateRollback", ""),+ "2 validation errors detected: Value '' at 'stackName' failed to satisfy constraint:"+ + " Member must have length greater than or equal to 1; Value '' at 'stackName' failed"+ + " to satisfy constraint: Member must satisfy regular expression pattern: "+ + STACK_NAME_PATTERN);+ }++ @Test+ void aStackNameOutsideThePatternIsAValidationError() {+ assertValidationError(cfn("ContinueUpdateRollback", "bad_stack!"),+ "1 validation error detected: Value 'bad_stack!' at 'stackName' failed to satisfy constraint:"+ + " Member must satisfy regular expression pattern: " + STACK_NAME_PATTERN);+ }++ /** One bad member rejects the whole list, before the stack, which does not exist, is looked up. */+ @Test+ void aSkipOutsideThePatternRejectsTheWholeList() {+ assertValidationError(cfn("ContinueUpdateRollback", "cur-no-such-stack")+ .formParam("ResourcesToSkip.member.1", "CrA")+ .formParam("ResourcesToSkip.member.2", "bad_id"),+ "1 validation error detected: " + badSkips("[CrA, bad_id]"));+ }++ @Test+ void aMissingStackNameAndABadSkipAreBothReportedStackNameFirst() {+ assertValidationError(cfn("ContinueUpdateRollback", null)+ .formParam("ResourcesToSkip.member.1", "bad_id"),+ "2 validation errors detected: " + NULL_STACK_NAME + "; " + badSkips("[bad_id]"));+ }++ @Test+ void forceRollbackWithSkipsIsRefusedWithoutTouchingTheStack() {+ RollbackFailedStack stack = rollbackFailedStack("cur-force-skip-");+ try {+ List<Map<String, String>> eventsBefore = events(stack.id());++ assertValidationError(cfn("ContinueUpdateRollback", stack.name())+ .formParam("ForceRollback", "true")+ .formParam("ResourcesToSkip.member.1", "LogGroup"),+ "ForceRollback and ResourcesToSkip are mutually exclusive");++ assertEquals("UPDATE_ROLLBACK_FAILED", CfnStackWaits.awaitTerminal(stack.name()).status());+ assertEquals(eventsBefore, events(stack.id()));+ } finally {+ cleanUp(stack);+ }+ }++ @Test+ void aStackThatDoesNotExistIsNamedAsGiven() {+ String stackName = "cur-missing-" + Long.toString(System.nanoTime(), 36);+ assertValidationError(cfn("ContinueUpdateRollback", stackName), "Stack: " + stackName + " does not exist");+ }++ /**+ * ForceRollback skips every UPDATE_FAILED resource, reporting each as a ResourcesToSkip skip+ * is, and the completed rollback says in the stack's reason that resources were skipped.+ */+ @Test+ void forceRollbackSkipsTheFailedResourcesAndUnblocksUpdates() {+ RollbackFailedStack stack = rollbackFailedStack("cur-force-");+ try {+ cfn("ContinueUpdateRollback", stack.name())+ .formParam("ForceRollback", "true")+ .when()+ .post("/")+ .then()+ .statusCode(200);++ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());+ assertEquals(FORCED, state.reason());+ assertEquals("UPDATE_COMPLETE", resourceStatuses(stack.name()).get("LogGroup"));+ cfn("DescribeStackResource", stack.name())+ .formParam("LogicalResourceId", "LogGroup")+ .when()+ .post("/")+ .then()+ .statusCode(200)+ .body(containsString("<ResourceStatus>UPDATE_COMPLETE</ResourceStatus>"))+ .body(containsString("<ResourceStatusReason>" + SKIPPED + "</ResourceStatusReason>"));++ // Newest first: the completion, then the skipped log group.+ List<Map<String, String>> events = events(stack.id());+ Map<String, String> completed = events.get(0);+ assertEquals(stack.name(), completed.get("LogicalResourceId"), events.toString());+ assertEquals("UPDATE_ROLLBACK_COMPLETE", completed.get("ResourceStatus"), events.toString());+ assertEquals(FORCED, completed.get("ResourceStatusReason"), events.toString());+ Map<String, String> skipped = events.get(1);+ assertEquals("LogGroup", skipped.get("LogicalResourceId"), events.toString());+ assertEquals("UPDATE_COMPLETE", skipped.get("ResourceStatus"), events.toString());+ assertEquals(SKIPPED, skipped.get("ResourceStatusReason"), events.toString());++ getTemplate(stack.name()).then().statusCode(200).body(not(containsString(stack.outsideGroup())));++ cfn("UpdateStack", stack.name())+ .formParam("TemplateBody", template(stack, 14, false))+ .when()+ .post("/")+ .then()+ .statusCode(200);+ assertEquals("UPDATE_COMPLETE", CfnStackWaits.awaitTerminal(stack.name()).status());+ } finally {+ cleanUp(stack);+ }+ }++ // ── Helpers ───────────────────────────────────────────────────────────────++ private static String badSkips(String list) {+ return "Value '" + list + "' at 'resourcesToSkip' failed to satisfy constraint: Member must satisfy"+ + " constraint: [Member must satisfy regular expression pattern: " + SKIP_PATTERN + "]";+ }++ /** AWS rejects a malformed request with ValidationError before it looks the stack up. */+ private static void assertValidationError(RequestSpecification request, String message) {+ String body = request.when().post("/").then().statusCode(400).extract().asString();+ assertEquals("ValidationError", XmlParser.extractFirst(body, "Code", null), body);+ assertEquals(message, XmlParser.extractFirst(body, "Message", null), body);+ }++ private record RollbackFailedStack(String name, String id, String logGroup, String outsideGroup) {+ }++ /**+ * Creates a stack, then updates it so the log group changes and an added log group collides+ * with one created outside the stack. The update fails, the log group cannot roll back, and the+ * stack settles in UPDATE_ROLLBACK_FAILED with the log group UPDATE_FAILED.+ */+ private static RollbackFailedStack rollbackFailedStack(String prefix) {+ String suffix = Long.toString(System.nanoTime(), 36);+ String name = prefix + suffix;+ String logGroup = "/" + prefix + suffix + "/lg";+ String outsideGroup = "/" + prefix + suffix + "/outside";+ logs("CreateLogGroup", "{\"logGroupName\":\"" + outsideGroup + "\"}").then().statusCode(200);++ RollbackFailedStack stack = new RollbackFailedStack(name, null, logGroup, outsideGroup);+ String id = XmlParser.extractFirst(+ cfn("CreateStack", name)+ .formParam("TemplateBody", template(stack, null, false))+ .when().post("/").then().statusCode(200).extract().asString(),+ "StackId", null);+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(name).status());++ cfn("UpdateStack", name)+ .formParam("TemplateBody", template(stack, 7, true))+ .when()+ .post("/")+ .then()+ .statusCode(200);+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(name);+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());+ Map<String, String> statuses = resourceStatuses(name);+ assertEquals("UPDATE_FAILED", statuses.get("LogGroup"), statuses.toString());+ assertEquals("CREATE_COMPLETE", statuses.get("Stable"), statuses.toString());+ return new RollbackFailedStack(name, id, logGroup, outsideGroup);+ }++ /**+ * {@code Stable} never changes. {@code LogGroup} carries {@code retention} when it is set, and+ * {@code Dup}, which depends on it, claims the name of the group made outside the stack.+ */+ private static String template(RollbackFailedStack stack, Integer retention, boolean collide) {+ String retentionProperty = retention == null ? "" : ", \"RetentionInDays\": " + retention;+ String dup = !collide ? "" : """+ ,+ "Dup": {+ "Type": "AWS::Logs::LogGroup",+ "DependsOn": "LogGroup",+ "Properties": {"LogGroupName": "%s"}+ }""".formatted(stack.outsideGroup());+ return """+ {+ "Resources": {+ "Stable": {+ "Type": "AWS::Logs::LogGroup",+ "Properties": {"LogGroupName": "%s/stable"}+ },+ "LogGroup": {+ "Type": "AWS::Logs::LogGroup",+ "Properties": {"LogGroupName": "%s"%s}+ }%s+ }+ }+ """.formatted(stack.logGroup(), stack.logGroup(), retentionProperty, dup);+ }++ /** AWS validates the skips after accepting the call, and reports a bad one as the failure. */+ private static void assertSkipFails(RollbackFailedStack stack, String skip, String reason) {+ cfn("ContinueUpdateRollback", stack.name())+ .formParam("ResourcesToSkip.member.1", skip)+ .when()+ .post("/")+ .then()+ .statusCode(200);+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());+ assertEquals(reason, state.reason());++ List<Map<String, String>> events = events(stack.id());+ Map<String, String> failed = events.get(0);+ Map<String, String> started = events.get(1);+ assertEquals(stack.name(), failed.get("LogicalResourceId"), events.toString());+ assertEquals("UPDATE_ROLLBACK_FAILED", failed.get("ResourceStatus"), events.toString());+ assertEquals(reason, failed.get("ResourceStatusReason"), events.toString());+ assertEquals("UPDATE_ROLLBACK_IN_PROGRESS", started.get("ResourceStatus"), events.toString());+ assertEquals(USER_INITIATED, started.get("ResourceStatusReason"), events.toString());+ }++ private static void assertContinueRefused(String stackName) {+ cfn("ContinueUpdateRollback", stackName)+ .when()+ .post("/")+ .then()+ .statusCode(400)+ .body(containsString("<Code>ValidationError</Code>"))+ .body(containsString(+ "<Message>RollbackUpdatedStack cannot be called from current stack status</Message>"));+ }++ /** A text dashboard titled {@code title}, joined when {@code failing} by a resource that cannot provision. */+ private static String dashboardTemplate(String dashboardName, String title, boolean failing) {+ String fail = !failing ? "" : """+ ,+ "ZFail": {"Type": "AWS::CloudFormation::Stack", "DependsOn": "Dashboard", "Properties": {}}""";+ return """+ {+ "Resources": {+ "Dashboard": {+ "Type": "AWS::CloudWatch::Dashboard",+ "Properties": {+ "DashboardName": "%s",+ "DashboardBody": "{\\"widgets\\":[{\\"type\\":\\"text\\",\\"properties\\":{\\"markdown\\":\\"%s\\"}}]}"+ }+ }%s+ }+ }+ """.formatted(dashboardName, title, fail);+ }++ private static Map<String, String> resourceStatuses(String stackName) {+ String body = cfn("DescribeStackResources", stackName).when().post("/")+ .then().statusCode(200).extract().asString();+ return XmlParser.extractPairs(body, "StackResources", "LogicalResourceId", "ResourceStatus");+ }++ private static List<Map<String, String>> events(String stackId) {+ return XmlParser.extractGroups(+ cfn("DescribeStackEvents", stackId).when().post("/").then().statusCode(200).extract().asString(),+ "member");+ }++ private static Response getTemplate(String stackName) {+ return cfn("GetTemplate", stackName).when().post("/");+ }++ /** Best effort and unchecked, so a failed assertion above is not masked. */+ private static void cleanUp(RollbackFailedStack stack) {+ cfn("DeleteStack", stack.name()).when().post("/");+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + stack.logGroup() + "\"}");+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + stack.logGroup() + "/stable\"}");+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + stack.outsideGroup() + "\"}");+ }++ /** A Query request for {@code action}, naming the stack unless {@code stackName} is null. */+ private static RequestSpecification cfn(String action, String stackName) {+ RequestSpecification request = given()+ .contentType("application/x-www-form-urlencoded")+ .formParam("Action", action);+ return stackName == null ? request : request.formParam("StackName", stackName);+ }++ private static Response logs(String target, String body) {+ return given()+ .config(RestAssured.config().encoderConfig(EncoderConfig.encoderConfig()+ .encodeContentTypeAs("application/x-amz-json-1.1", ContentType.TEXT)))+ .header("X-Amz-Target", "Logs_20140328." + target)+ .contentType("application/x-amz-json-1.1")+ .body(body)+ .when()+ .post("/");+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.javaindex 4b8e74655c..7e2c2ff6d3 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java@@ -13,6 +13,7 @@ import static org.hamcrest.Matchers.containsString;import static org.hamcrest.Matchers.equalTo;import static org.hamcrest.Matchers.not;import static org.hamcrest.Matchers.nullValue;+import static org.junit.jupiter.api.Assertions.assertEquals;import static org.mockito.ArgumentMatchers.anyMap;import static org.mockito.ArgumentMatchers.anyString;import static org.mockito.ArgumentMatchers.eq;@@ -338,7 +339,8 @@ class CloudFormationPipesCleanupIntegrationTest {* can fail. The pipe the rollback exists to preserve is the prior one, so the resource names it* and the rename cleanup is spent before the delete is attempted. The stack still reports* UPDATE_ROLLBACK_FAILED naming the resource, the replacement is what stays orphaned, and the- * next update finds nothing on the resource that would delete the prior pipe.+ * next update, once ContinueUpdateRollback has skipped the resource, finds nothing on it that+ * would delete the prior pipe.*/@Testvoid aRollbackThatCannotDeleteTheReplacementStillKeepsThePriorPipe() {@@ -393,8 +395,22 @@ class CloudFormationPipesCleanupIntegrationTest {assertPipe(priorName);assertPipe(replacementName);+ // AWS refuses an update over UPDATE_ROLLBACK_FAILED. The pipe's rollback already restored+ // the prior pipe and only failed to delete the replacement, so skipping it finishes the+ // rollback and the stack takes updates again.+ given()+ .contentType("application/x-www-form-urlencoded")+ .formParam("Action", "ContinueUpdateRollback")+ .formParam("StackName", stackName)+ .formParam("ResourcesToSkip.member.1", "MyPipe")+ .when()+ .post("/")+ .then()+ .statusCode(200);+ assertEquals("UPDATE_ROLLBACK_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());+Mockito.doCallRealMethod().when(pipesService).deletePipe(eq(replacementName), anyString());- updateStack(stackName, renameRollbackTemplate(replacementName, ""));+ updateStack(stackName, renameRollbackTemplate(priorName, ""));given().contentType("application/json")@@ -407,8 +423,8 @@ class CloudFormationPipesCleanupIntegrationTest {verify(pipesService, never()).deletePipe(eq(priorName), anyString());deleteStack(stackName);- // The rollback left the replacement orphaned and the resource UPDATE_FAILED, so neither pipe- // is one DeleteStack removes and this test removes them itself.+ // The replacement the rollback orphaned is no resource of the stack, so this test removes+ // it itself, and the prior pipe too should DeleteStack have left it.given().when().delete("/v1/pipes/" + priorName);given().when().delete("/v1/pipes/" + replacementName);}diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.javaindex a62d1c1fe1..2fdbeb6b89 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java@@ -2,13 +2,17 @@ package io.github.hectorvent.floci.services.cloudformation;import com.fasterxml.jackson.core.type.TypeReference;import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.databind.SerializationFeature;+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.cloudformation.model.Stack;+import io.github.hectorvent.floci.services.cloudformation.model.StackEvent;import io.github.hectorvent.floci.services.cloudformation.model.StackResource;+import io.github.hectorvent.floci.services.cloudformation.model.StackUpdateSnapshot;import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnDynamicReferences;import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnResourceDispatcher;import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnRollback;@@ -20,13 +24,19 @@ import org.junit.jupiter.api.Test;import org.mockito.InOrder;import java.time.Clock;+import java.time.Instant;+import java.time.ZoneId;+import java.time.ZoneOffset;+import java.util.LinkedHashSet;import java.util.List;import java.util.Map;import java.util.Set;+import java.util.concurrent.TimeUnit;import java.util.concurrent.atomic.AtomicInteger;import java.util.concurrent.atomic.AtomicReference;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;import static org.junit.jupiter.api.Assertions.assertNull;import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;@@ -40,18 +50,23 @@ import static org.mockito.Mockito.doThrow;import static org.mockito.Mockito.inOrder;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.never;+import static org.mockito.Mockito.times;import static org.mockito.Mockito.verify;import static org.mockito.Mockito.when;/**- * Covers rollback cleanup when another actor removes a resource after its create succeeded, and- * which resources a stack delete walks after a failed operation left them in a failed status, and- * that a stack which finished deleting is no longer updatable.+ * Covers rollback cleanup when another actor removes a resource after its create succeeded, which+ * resources a stack delete walks after a failed operation left them in a failed status, what a+ * failed update rollback keeps so ContinueUpdateRollback can retry it, and that a stack which+ * finished deleting is no longer updatable.*/class CloudFormationServiceRollbackTest {private static final String ACCOUNT = "000000000000";private static final String REGION = "us-east-1";+ private static final String FORCED_ROLLBACK_REASON = "One or more resources were skipped during UpdateRollback."+ + " These resources were not rolled back and may have drifted from the template."+ + " Run drift detection (DetectStackDrift) to review their current state before your next update.";private CfnResourceDispatcher provisioner;private CloudFormationService service;@@ -424,6 +439,293 @@ class CloudFormationServiceRollbackTest {assertNull(resource.getAttributes().get("rollbackSnapshot"));}+ @Test+ void refusesUpdate_afterUpdateRollbackFailed_refusesButStillAllowsAfterUpdateRollbackComplete() {+ assertTrue(CloudFormationService.refusesUpdate("UPDATE_ROLLBACK_FAILED"));+ assertFalse(CloudFormationService.refusesUpdate("UPDATE_ROLLBACK_COMPLETE"));+ }++ /**+ * The walker's inputs live only for the update that failed. What it needs to run again is kept+ * on the stack, and must survive the stack's persisted JSON: a ContinueUpdateRollback after a+ * restart works from the stored copy alone.+ */+ @Test+ void updateRollbackFailed_persistsWhatContinueUpdateRollbackNeeds() throws Exception {+ Stack stack = new Stack();+ stack.setStackName("update-rollback-failed-snapshot");+ stack.setStackId("stack-id");+ stack.setRegion(REGION);+ stack.setStatus("UPDATE_IN_PROGRESS");+ stack.setTemplateBody("failed-template");+ stack.setOriginalTemplateBody("failed-original");++ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_COMPLETE");+ StackResource added = resource("Dup", null, "AWS::Logs::LogGroup", "CREATE_FAILED");+ added.setStatusReason("already exists");+ stack.getResources().put(logGroup.getLogicalId(), logGroup);+ stack.getResources().put(added.getLogicalId(), added);+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(+ "previous-template", "previous-original", Map.of("P", "1"), Map.of("P", "1"),+ Map.of("Out", "v"), Map.of(), Map.of(),+ Map.of("LogGroup", resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "CREATE_COMPLETE")));++ service.rollbackFailedExecution(stack, REGION, false, added, previousState,+ new LinkedHashSet<>(List.of("LogGroup", "Dup")));++ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());+ assertEquals("failed-template", stack.getTemplateBody());++ Stack restored = persistedCopy(stack);+ assertEquals(List.of("LogGroup"), restored.getUpdateRollbackResourceIds());+ assertEquals("previous-template", restored.getUpdateRollbackSnapshot().templateBody());+ assertEquals("CREATE_COMPLETE",+ restored.getUpdateRollbackSnapshot().resources().get("LogGroup").getStatus());++ when(provisioner.rollbackUpdate(any(StackResource.class), any())).thenReturn(true);+ restored.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ service.resumeUpdateRollback(restored, List.of(), false, REGION);++ assertEquals("UPDATE_ROLLBACK_COMPLETE", restored.getStatus());+ assertEquals("previous-template", restored.getTemplateBody());+ assertEquals("previous-original", restored.getOriginalTemplateBody());+ assertEquals("v", restored.getOutputs().get("Out"));+ assertEquals("CREATE_COMPLETE", restored.getResources().get("LogGroup").getStatus());+ assertNull(restored.getUpdateRollbackSnapshot());+ assertNull(restored.getUpdateRollbackResourceIds());+ }++ /**+ * A stack stored before the snapshot was kept still has a way out: every UPDATE_FAILED+ * resource is owed a rollback, and the template it already holds is kept.+ */+ @Test+ void resumeUpdateRollback_withoutAStoredSnapshot_owesEveryUpdateFailedResourceAndKeepsTheTemplate() {+ Stack stack = new Stack();+ stack.setStackName("legacy-update-rollback-failed");+ stack.setStackId("stack-id");+ stack.setRegion(REGION);+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ stack.setTemplateBody("current-template");++ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_FAILED");+ StackResource stable = resource("Stable", "/stable", "AWS::Logs::LogGroup", "UPDATE_COMPLETE");+ stack.getResources().put(logGroup.getLogicalId(), logGroup);+ stack.getResources().put(stable.getLogicalId(), stable);+ when(provisioner.rollbackUpdate(any(StackResource.class), any())).thenReturn(true);++ service.resumeUpdateRollback(stack, List.of(), false, REGION);++ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());+ assertEquals("current-template", stack.getTemplateBody());+ assertEquals("UPDATE_COMPLETE", logGroup.getStatus());+ assertEquals("UPDATE_COMPLETE", stable.getStatus());+ verify(provisioner).rollbackUpdate(eq(logGroup), any());+ verify(provisioner, never()).rollbackUpdate(eq(stable), any());+ }++ /**+ * A skip naming a resource of a nested stack Floci holds is one AWS would act on and Floci does+ * not support: the rollback fails again with Floci's reason, touching no resource and keeping+ * what a later retry needs.+ */+ @Test+ void resumeUpdateRollback_skippingANestedStackResource_failsAgainWithoutTouchingAResource() {+ Stack stack = new Stack();+ stack.setStackName("nested-skip");+ stack.setStackId("stack-id");+ stack.setRegion(REGION);+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ StackResource nested = resource("Nested", "nested-stack-id", "AWS::CloudFormation::Stack", "UPDATE_FAILED");+ stack.getResources().put(nested.getLogicalId(), nested);+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),+ Map.of("Nested", resource("Nested", "nested-stack-id", "AWS::CloudFormation::Stack",+ "CREATE_COMPLETE")));+ stack.setUpdateRollbackSnapshot(previousState);+ stack.setUpdateRollbackResourceIds(List.of("Nested"));++ service.resumeUpdateRollback(stack, List.of("Nested.Child"), false, REGION);++ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());+ assertEquals("Skipping a resource of a nested stack is not supported by Floci: Nested.Child",+ stack.getStatusReason());+ assertEquals("UPDATE_FAILED", nested.getStatus());+ assertEquals(previousState, stack.getUpdateRollbackSnapshot());+ assertEquals(List.of("Nested"), stack.getUpdateRollbackResourceIds());+ verify(provisioner, never()).rollbackUpdate(any(StackResource.class), any());+ }++ /**+ * A resume runs the restore of a resource still owed one again, through the same provisioner+ * hook the first rollback used: here one whose first restore threw.+ */+ @Test+ void resumeUpdateRollback_afterARestoreThatThrew_runsTheRestoreAgain() {+ Stack stack = new Stack();+ stack.setStackName("retry-thrown-restore");+ stack.setStackId("stack-id");+ stack.setRegion(REGION);+ stack.setStatus("UPDATE_IN_PROGRESS");+ StackResource dashboard = resource("Dashboard", "ops", "AWS::CloudWatch::Dashboard", "UPDATE_COMPLETE");+ StackResource added = resource("ZFail", null, "AWS::CloudFormation::Stack", "CREATE_FAILED");+ stack.getResources().put(dashboard.getLogicalId(), dashboard);+ stack.getResources().put(added.getLogicalId(), added);+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),+ Map.of("Dashboard", resource("Dashboard", "ops", "AWS::CloudWatch::Dashboard", "CREATE_COMPLETE")));+ when(provisioner.rollbackUpdate(eq(dashboard), any()))+ .thenThrow(new IllegalStateException("simulated restore failure"))+ .thenReturn(true);++ service.rollbackFailedExecution(stack, REGION, false, added, previousState,+ new LinkedHashSet<>(List.of("Dashboard", "ZFail")));++ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());+ assertEquals("UPDATE_FAILED", dashboard.getStatus());++ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ service.resumeUpdateRollback(stack, List.of(), false, REGION);++ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());+ assertEquals("CREATE_COMPLETE", dashboard.getStatus());+ verify(provisioner, times(2)).rollbackUpdate(eq(dashboard), any());+ }++ /**+ * The resume runs on the operation executor, whose future nobody reads. An unexpected failure+ * must still settle the stack in UPDATE_ROLLBACK_FAILED, keeping what a retry needs, instead+ * of stranding it in UPDATE_ROLLBACK_IN_PROGRESS, where ContinueUpdateRollback refuses it.+ * A resource the failed pass already rolled back is no longer owed, so the retry does not roll it back again.+ */+ @Test+ void resumeUpdateRollback_whenTheBodyThrows_settlesInUpdateRollbackFailedAndCanBeContinued()+ throws Exception {+ FailingClock clock = new FailingClock();+ CloudFormationService resumeService = serviceReading(clock);+ resumeService.createChangeSet("resume-throws", "create", "CREATE",+ "{\"Resources\":{\"LogGroup\":{\"Type\":\"AWS::Logs::LogGroup\"}}}", null,+ Map.of(), List.of(), Map.of(), REGION, ACCOUNT);+ Stack stack = resumeService.describeStacks("resume-throws", REGION, ACCOUNT).getFirst();+ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_FAILED");+ stack.getResources().put(logGroup.getLogicalId(), logGroup);+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),+ Map.of("LogGroup", resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "CREATE_COMPLETE")));+ stack.setUpdateRollbackSnapshot(previousState);+ stack.setUpdateRollbackResourceIds(List.of("LogGroup"));+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ when(provisioner.rollbackUpdate(eq(logGroup), any())).thenReturn(true, false);+ clock.failNextRead("simulated clock failure");++ resumeService.resumeUpdateRollback(stack, List.of(), false, REGION);++ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());+ assertEquals("simulated clock failure", stack.getStatusReason());+ StackEvent last = stack.getEvents().getLast();+ assertEquals("resume-throws", last.getLogicalResourceId());+ assertEquals("AWS::CloudFormation::Stack", last.getResourceType());+ assertEquals("UPDATE_ROLLBACK_FAILED", last.getResourceStatus());+ assertEquals("simulated clock failure", last.getResourceStatusReason());+ assertEquals(previousState, stack.getUpdateRollbackSnapshot());+ assertEquals(List.of(), stack.getUpdateRollbackResourceIds());++ resumeService.continueUpdateRollback("resume-throws", List.of(), false, REGION).get(10, TimeUnit.SECONDS);++ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());+ assertNull(stack.getUpdateRollbackSnapshot());+ verify(provisioner).rollbackUpdate(eq(logGroup), any());+ }++ /**+ * A skip already reported is no longer owed a rollback when an unexpected failure settles the+ * stack, or a later retry would roll back a resource the caller skipped.+ */+ @Test+ void resumeUpdateRollback_whenTheBodyThrowsAfterASkip_noLongerOwesTheSkippedResource() throws Exception {+ FailingClock clock = new FailingClock();+ CloudFormationService resumeService = serviceReading(clock);+ resumeService.createChangeSet("resume-throws-after-skip", "create", "CREATE",+ "{\"Resources\":{\"LogGroup\":{\"Type\":\"AWS::Logs::LogGroup\"}}}", null,+ Map.of(), List.of(), Map.of(), REGION, ACCOUNT);+ Stack stack = resumeService.describeStacks("resume-throws-after-skip", REGION, ACCOUNT).getFirst();+ StackResource skipped = resource("Skipped", "/skipped", "AWS::Logs::LogGroup", "UPDATE_FAILED");+ stack.getResources().put(skipped.getLogicalId(), skipped);+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),+ Map.of("Skipped", resource("Skipped", "/skipped", "AWS::Logs::LogGroup", "CREATE_COMPLETE")));+ stack.setUpdateRollbackSnapshot(previousState);+ stack.setUpdateRollbackResourceIds(List.of("Skipped"));+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ clock.failNextRead("simulated clock failure");++ resumeService.resumeUpdateRollback(stack, List.of("Skipped"), false, REGION);++ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());+ assertEquals("simulated clock failure", stack.getStatusReason());+ assertEquals("UPDATE_COMPLETE", skipped.getStatus());+ assertEquals("Resource skipped during UpdateRollback", skipped.getStatusReason());+ assertEquals(previousState, stack.getUpdateRollbackSnapshot());+ assertEquals(List.of(), stack.getUpdateRollbackResourceIds());++ resumeService.continueUpdateRollback("resume-throws-after-skip", List.of(), false, REGION)+ .get(10, TimeUnit.SECONDS);++ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());+ assertEquals("UPDATE_COMPLETE", skipped.getStatus());+ verify(provisioner, never()).rollbackUpdate(any(StackResource.class), any());+ }++ /**+ * ForceRollback skips every UPDATE_FAILED resource, reporting each as a ResourcesToSkip skip+ * is, and a rollback it completes says in the stack's reason that resources were skipped.+ */+ @Test+ void resumeUpdateRollback_forced_skipsEveryUpdateFailedResourceAndExplainsTheCompletion() {+ Stack stack = new Stack();+ stack.setStackName("forced-rollback");+ stack.setStackId("stack-id");+ stack.setRegion(REGION);+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");+ stack.setTemplateBody("failed-template");+ StackResource role = resource("Role", "role", "AWS::IAM::Role", "UPDATE_FAILED");+ StackResource stable = resource("Stable", "/stable", "AWS::Logs::LogGroup", "CREATE_COMPLETE");+ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_FAILED");+ for (StackResource resource : new StackResource[] {role, stable, logGroup}) {+ stack.getResources().put(resource.getLogicalId(), resource);+ }+ stack.setUpdateRollbackSnapshot(new StackUpdateSnapshot(+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of("Out", "v"), Map.of(),+ Map.of(), Map.of(+ "Role", resource("Role", "role", "AWS::IAM::Role", "CREATE_COMPLETE"),+ "Stable", resource("Stable", "/stable", "AWS::Logs::LogGroup", "CREATE_COMPLETE"),+ "LogGroup", resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "CREATE_COMPLETE"))));+ stack.setUpdateRollbackResourceIds(List.of("Role", "LogGroup"));++ service.resumeUpdateRollback(stack, List.of(), true, REGION);++ for (StackResource skipped : List.of(role, logGroup)) {+ assertEquals("UPDATE_COMPLETE", skipped.getStatus(), skipped.getLogicalId());+ assertEquals("Resource skipped during UpdateRollback", skipped.getStatusReason(), skipped.getLogicalId());+ }+ verify(provisioner, never()).rollbackUpdate(any(StackResource.class), any());+ assertEquals(List.of(+ "Role UPDATE_COMPLETE Resource skipped during UpdateRollback",+ "LogGroup UPDATE_COMPLETE Resource skipped during UpdateRollback",+ "forced-rollback UPDATE_ROLLBACK_COMPLETE " + FORCED_ROLLBACK_REASON),+ stack.getEvents().stream()+ .map(event -> event.getLogicalResourceId() + " " + event.getResourceStatus() + " "+ + event.getResourceStatusReason())+ .toList());+ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());+ assertEquals(FORCED_ROLLBACK_REASON, stack.getStatusReason());+ assertEquals(FORCED_ROLLBACK_REASON, stack.getEvents().getLast().getResourceStatusReason());+ assertEquals("previous-template", stack.getTemplateBody());+ assertEquals("v", stack.getOutputs().get("Out"));+ assertNull(stack.getUpdateRollbackSnapshot());+ assertNull(stack.getUpdateRollbackResourceIds());+ }+private static StackResource resource(String logicalId, String physicalId, String resourceType, String status) {StackResource resource = new StackResource();resource.setLogicalId(logicalId);@@ -433,6 +735,53 @@ class CloudFormationServiceRollbackTest {return resource;}+ /** A service over the shared provisioner that reads the time from {@code clock}. */+ private CloudFormationService serviceReading(Clock clock) {+ RegionResolver regionResolver = mock(RegionResolver.class);+ when(regionResolver.getAccountId()).thenReturn(ACCOUNT);+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.defaultAccountId()).thenReturn(ACCOUNT);+ return new CloudFormationService(provisioner, mock(S3Service.class), mock(SsmService.class),+ mock(CfnDynamicReferences.class), new ObjectMapper(), config, regionResolver, clock,+ new InMemoryStorageFactory());+ }++ /** The stack as {@code PersistentStorage} writes it and reads it back. */+ private static Stack persistedCopy(Stack stack) throws Exception {+ ObjectMapper mapper = new ObjectMapper()+ .registerModule(new JavaTimeModule())+ .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);+ return mapper.readValue(mapper.writeValueAsString(stack), Stack.class);+ }++ /** The system clock, except that the read after {@link #failNextRead} throws. */+ private static final class FailingClock extends Clock {+ private final AtomicReference<String> failure = new AtomicReference<>();++ void failNextRead(String message) {+ failure.set(message);+ }++ @Override+ public ZoneId getZone() {+ return ZoneOffset.UTC;+ }++ @Override+ public Clock withZone(ZoneId zone) {+ throw new UnsupportedOperationException();+ }++ @Override+ public Instant instant() {+ String message = failure.getAndSet(null);+ if (message != null) {+ throw new IllegalStateException(message);+ }+ return Instant.now();+ }+ }+private static final class InMemoryStorageFactory extends StorageFactory {private InMemoryStorageFactory() {super(null, null);diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.javaindex 2ee80e31e2..d719840692 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java@@ -19,7 +19,6 @@ import org.junit.jupiter.api.Test;import org.junit.jupiter.params.ParameterizedTest;import org.junit.jupiter.params.provider.ValueSource;-import java.time.Duration;import java.util.ArrayList;import java.util.List;import java.util.Map;@@ -27,7 +26,6 @@ import java.util.Map;import static io.github.hectorvent.floci.services.cognito.CognitoRestAssuredUtils.cognitoAction;import static io.github.hectorvent.floci.services.cognito.CognitoRestAssuredUtils.cognitoJson;import static io.restassured.RestAssured.given;-import static org.awaitility.Awaitility.await;import static org.hamcrest.Matchers.equalTo;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertFalse;@@ -155,6 +153,10 @@ class CognitoResourceServerCfnIntegrationTest {}clearInvocations(cognitoService);+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE");+ assertServerGone(failedPool, IDENTIFIER);+ assertServer(originalPool, IDENTIFIER, "Original API", "read");+updateStack(template(finalPool, IDENTIFIER, "Final API", "admin"), "UPDATE_COMPLETE");assertEquals(IDENTIFIER, output("ServerRef"));@@ -433,10 +435,7 @@ class CognitoResourceServerCfnIntegrationTest {deleteServer(pool);createServer(pool, "Out of band API", "admin");- ObjectNode nextAttempt = template(pool, IDENTIFIER, "Next attempt", "write");- addFailingDependentResource(nextAttempt);- nextAttempt.withObject("/Resources/Broken/Properties").put("ClientName", "second-broken");- updateStack(nextAttempt, "UPDATE_ROLLBACK_COMPLETE");+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE");assertServer(pool, IDENTIFIER, "Original API", "read");cloudFormation("DeleteStack", null);CfnStackWaits.awaitStackDeleted(stack);@@ -459,6 +458,7 @@ class CognitoResourceServerCfnIntegrationTest {} finally {doCallRealMethod().when(cognitoService).deleteResourceServer(replacementPool, IDENTIFIER);}+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE", "Server");deleteServer(replacementPool);doAnswer(invocation -> {cognitoService.describeResourceServer(replacementPool, IDENTIFIER);@@ -495,6 +495,7 @@ class CognitoResourceServerCfnIntegrationTest {} finally {doCallRealMethod().when(cognitoService).deleteResourceServer(historicalPool, IDENTIFIER);}+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE", "Server");assertServer(historicalPool, IDENTIFIER, "Failed replacement", "write");deleteServer(originalPool);clearInvocations(cognitoService);@@ -521,7 +522,7 @@ class CognitoResourceServerCfnIntegrationTest {}@Test- void aPendingRollbackSurvivesFailedUpdatesUntilItCanBeRestored() throws Exception {+ void aPendingRollbackSurvivesFailedRetriesUntilItCanBeRestored() throws Exception {String pool = createPool();createStack(template(pool, IDENTIFIER, "Original API", "read"));doThrow(new AwsException("InternalErrorException", "temporary restore failure", 500))@@ -533,18 +534,12 @@ class CognitoResourceServerCfnIntegrationTest {updateStack(firstAttempt, "UPDATE_ROLLBACK_FAILED");assertServer(pool, IDENTIFIER, "First attempt", "write");- ObjectNode nextAttempt = template(pool, IDENTIFIER, "Next attempt", "admin");- addFailingDependentResource(nextAttempt);- nextAttempt.withObject("/Resources/Broken/Properties").put("ClientName", "broken-next");- updateStack(nextAttempt, "UPDATE_ROLLBACK_FAILED");+ continueUpdateRollback("UPDATE_ROLLBACK_FAILED");assertServer(pool, IDENTIFIER, "First attempt", "write");doCallRealMethod().when(cognitoService).updateResourceServer(eq(pool), eq(IDENTIFIER), eq("Original API"), any());- ObjectNode recoveryAttempt = template(pool, IDENTIFIER, "Recovery attempt", "write");- addFailingDependentResource(recoveryAttempt);- recoveryAttempt.withObject("/Resources/Broken/Properties").put("ClientName", "broken-recovery");- updateStack(recoveryAttempt, "UPDATE_ROLLBACK_COMPLETE");+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE");assertServer(pool, IDENTIFIER, "Original API", "read");assertEquals(IDENTIFIER, output("ServerRef"));} finally {@@ -554,7 +549,7 @@ class CognitoResourceServerCfnIntegrationTest {}@Test- void skippingAPendingRollbackPreservesRestorationAndBlocksAnotherUpdate() throws Exception {+ void anUpdateThatLeavesAPendingRollbackAloneIsRefused() throws Exception {String pool = createPool();createStack(template(pool, IDENTIFIER, "Original API", "read"));doThrow(new AwsException("InternalErrorException", "temporary restore failure", 500))@@ -569,23 +564,16 @@ class CognitoResourceServerCfnIntegrationTest {ObjectNode outputsOnly = firstAttempt.deepCopy();outputsOnly.withObject("/Resources").remove("Broken");outputsOnly.withObject("/Outputs").putObject("Marker").put("Value", "unchanged-resource");- cloudFormation("UpdateStack", outputsOnly);- await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->- assertEquals("UPDATE_COMPLETE_CLEANUP_IN_PROGRESS", stackStatus(),- "Skipping the resource must not discard its pending restoration"));- assertServer(pool, IDENTIFIER, "First attempt", "write");- assertEquals(IDENTIFIER, output("ServerRef"));-- ObjectNode nextAttempt = template(pool, IDENTIFIER, "Next attempt", "admin");String error = given().contentType("application/x-www-form-urlencoded").header("Authorization", CFN_AUTH).formParam("Action", "UpdateStack")- .formParam("StackName", stack).formParam("TemplateBody", nextAttempt.toString())+ .formParam("StackName", stack).formParam("TemplateBody", outputsOnly.toString()).post("/").then().statusCode(400).extract().asString();assertEquals("ValidationError", XmlParser.extractFirst(error, "Code", null));assertTrue(XmlParser.extractFirst(error, "Message", "")- .contains("UPDATE_COMPLETE_CLEANUP_IN_PROGRESS"));- assertEquals("UPDATE_COMPLETE_CLEANUP_IN_PROGRESS", stackStatus());+ .contains("UPDATE_ROLLBACK_FAILED"));+ assertEquals("UPDATE_ROLLBACK_FAILED", stackStatus());assertServer(pool, IDENTIFIER, "First attempt", "write");+ assertEquals(IDENTIFIER, output("ServerRef"));cloudFormation("DeleteStack", null);CfnStackWaits.awaitStackDeleted(stack);@@ -729,6 +717,19 @@ class CognitoResourceServerCfnIntegrationTest {assertEquals(expectedStatus, CfnStackWaits.awaitTerminal(stack).status());}+ private void continueUpdateRollback(String expectedStatus, String... resourcesToSkip) {+ RequestSpecification request = given()+ .contentType("application/x-www-form-urlencoded")+ .header("Authorization", CFN_AUTH)+ .formParam("Action", "ContinueUpdateRollback")+ .formParam("StackName", stack);+ for (int i = 0; i < resourcesToSkip.length; i++) {+ request.formParam("ResourcesToSkip.member." + (i + 1), resourcesToSkip[i]);+ }+ request.when().post("/").then().statusCode(200);+ assertEquals(expectedStatus, CfnStackWaits.awaitTerminal(stack).status());+ }+private void cloudFormation(String action, ObjectNode template) {RequestSpecification request = given().contentType("application/x-www-form-urlencoded")diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.javaindex b06ae36251..d181a8bb29 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java@@ -635,8 +635,8 @@ class EcsCfnProvisionerTest {assertThrows(AwsException.class, () -> provisioner.rollbackUpdate(r));assertEquals(TASK_DEF_ARN, r.getPhysicalId());- // A rollback with no replacement of its own deletes nothing and keeps the orphan owed.- assertFalse(provisioner.rollbackUpdate(r));+ // A retried rollback (ContinueUpdateRollback) re-attempts the failed delete; failing again, the orphan stays owed.+ assertThrows(AwsException.class, () -> provisioner.rollbackUpdate(r));assertTrue(provisioner.hasReplacementUpdate(r));// The next update registers revision 5: revision 3 (the prior) joins the orphan on the listdiff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.javaindex a38d16c372..14c9faae8b 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java@@ -447,12 +447,9 @@ class ElbV2CfnProvisionerTest {assertTrue(provisioner.hasReplacementUpdate(r));assertEquals(renamed.getLoadBalancerArn(), provisioner.updateCleanupPhysicalId(r));org.mockito.Mockito.reset(elb);- // A second rollback has no replacement of this update to undo: it deletes nothing and keeps- // the orphan owed (the load balancer itself rolls back as complete, nothing being modified).- provisioner.rollbackUpdate(r);- verify(elb, never()).deleteLoadBalancer(anyString(), anyString());- assertTrue(provisioner.hasReplacementUpdate(r));- assertEquals(new UpdateCleanupResult(true, true, renamed.getLoadBalancerArn(), 0, null), provisioner.completeUpdate(r));+ // A retried rollback (ContinueUpdateRollback) re-attempts the delete that failed; succeeding,+ // it owes nothing more and never touches the restored prior.+ assertTrue(provisioner.rollbackUpdate(r));verify(elb).deleteLoadBalancer(REGION, renamed.getLoadBalancerArn());verify(elb, never()).deleteLoadBalancer(REGION, LB_ARN);assertFalse(provisioner.hasReplacementUpdate(r));diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.javaindex 648029b6c4..0db5388a26 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java@@ -254,6 +254,62 @@ class ReplacementCleanupTest {assertEquals(List.of("subnet-2"), deleted);}+ /** A retried rollback (ContinueUpdateRollback) re-attempts the delete that failed, then has nothing left to do. */+ @Test+ void rollback_afterTheReplacementDeleteFailed_retriesThatDeleteOnTheNextRollback() {+ StackResource r = failedReplacementRollback();+ List<String> deleted = new ArrayList<>();++ assertTrue(ReplacementCleanup.rollback(r, (type, id, region) -> deleted.add(type + ":" + id + ":" + region)));++ assertEquals(List.of("AWS::EC2::Subnet:subnet-2:us-east-1"), deleted);+ assertEquals("subnet-1", r.getPhysicalId());+ assertFalse(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));+ assertFalse(ReplacementCleanup.hasReplacement(r));+ List<String> deletedAgain = new ArrayList<>();+ assertFalse(ReplacementCleanup.rollback(r, (type, id, region) -> deletedAgain.add(id)));+ assertEquals(List.of(), deletedAgain);+ }++ @Test+ void rollback_retryThatFailsAgain_keepsTheReplacementOwed() {+ StackResource r = failedReplacementRollback();++ assertThrows(IllegalStateException.class, () -> ReplacementCleanup.rollback(r, (type, id, region) -> {+ throw new IllegalStateException("DependencyViolation");+ }));++ assertTrue(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));+ List<String> deleted = new ArrayList<>();+ assertTrue(ReplacementCleanup.rollback(r, (type, id, region) -> deleted.add(id)));+ assertEquals(List.of("subnet-2"), deleted);+ }++ /** A new provision supersedes the failed rollback; the replacement stays owed to the next cleanup instead. */+ @Test+ void record_afterAFailedRollbackDelete_doesNotRetryThatDeleteOnTheNextRollback() {+ StackResource r = failedReplacementRollback();+ List<String> deleted = new ArrayList<>();++ ReplacementCleanup.record(r, update("subnet-1"), Map.of("VpcId", "vpc-old"));++ assertFalse(ReplacementCleanup.rollback(r, (type, id, region) -> deleted.add(id)));+ assertEquals(List.of(), deleted);+ assertTrue(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));+ }++ /** Replaces subnet-1 with subnet-2, then rolls back with a delete of subnet-2 that fails. */+ private static StackResource failedReplacementRollback() {+ StackResource r = resource("subnet-2");+ ReplacementCleanup.record(r, update("subnet-1"), Map.of("VpcId", "vpc-old"));+ assertThrows(IllegalStateException.class, () -> ReplacementCleanup.rollback(r, (type, id, region) -> {+ throw new IllegalStateException("DependencyViolation");+ }));+ assertEquals("subnet-1", r.getPhysicalId());+ assertTrue(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));+ return r;+ }+private static ProvisionContext update(String priorPhysicalId) {return new ProvisionContext(null, "us-east-1", "000000000000", "my-stack", priorPhysicalId);}