Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(cloudformation): refuse updates after a failed rollback and add ContinueUpdateRollback (#4949)

Mario Toffiac4a809f47f81
diff --git a/docs/services/cloudformation.md b/docs/services/cloudformation.md
index faffdf1914..5828867463 100644
--- a/docs/services/cloudformation.md
+++ b/docs/services/cloudformation.md
@@ -12,6 +12,7 @@
| `CreateStack` | Deploy a CloudFormation template |
| `UpdateStack` | Update an existing stack |
| `DeleteStack` | Delete a stack and its resources |
+| `ContinueUpdateRollback` | Retry the rollback of a stack in `UPDATE_ROLLBACK_FAILED`, optionally skipping resources or forcing past every failed one (see [Failed update rollbacks](#failed-update-rollbacks)) |
| `UpdateTerminationProtection` | - |
| `CreateChangeSet` | Create a change set |
| `DescribeChangeSet` | Get change set details with a computed Add/Modify/Remove resource diff |
@@ -127,10 +128,8 @@ owed cleanup while their retry budget remains. Exhausted historical records are
later updates and stack-delete retries. Historical cleanup is address-based and does not provide
an atomic ownership check against another stack or a direct Cognito API writer reusing the address.
-Failed in-place restoration preserves the original name and scopes for a later update retry.
-Deleting the current server keeps this snapshot if deletion fails. An update that skips a server
-with pending restoration can remain in `UPDATE_COMPLETE_CLEANUP_IN_PROGRESS`, blocking another
-update. `ContinueUpdateRollback` remains unsupported.
+Failed in-place restoration preserves the original name and scopes for a `ContinueUpdateRollback`
+retry. Deleting the current server keeps this snapshot if deletion fails.
## Supported Resource Types
@@ -494,6 +493,55 @@ was sent gets a `Delete` straight away for the same reason. An update that chang
reaches no handler: it fails with `Modifying service token is not allowed.`, as on AWS, and its
rollback sends nothing.
+## Failed update rollbacks
+
+When an update fails, Floci rolls back the resources it changed. A resource whose rollback fails
+is left `UPDATE_FAILED`, and the stack settles in `UPDATE_ROLLBACK_FAILED`. As on AWS, `UpdateStack`
+and an `UPDATE` change set are then refused with `ValidationError` (`Stack:<arn> is in
+UPDATE_ROLLBACK_FAILED state and can not be updated.`). Two ways out remain:
+
+- `DeleteStack` deletes the stack and its resources, the `UPDATE_FAILED` ones included (see
+ [Deletion Policies](#deletion-policies)).
+- `ContinueUpdateRollback` runs the rollback again for the resources still owed one, starting with
+ an `UPDATE_ROLLBACK_IN_PROGRESS` stack event whose reason is `User Initiated`. A retry re-runs
+ the restore of each resource still owed one, including the delete of a replacement whose earlier
+ delete failed; a resource whose restore cannot be redone fails again, leaving the stack in
+ `UPDATE_ROLLBACK_FAILED`, and can be passed in `ResourcesToSkip`. `ResourcesToSkip` names
+ `UPDATE_FAILED` resources to report as `UPDATE_COMPLETE` with the reason `Resource skipped
+ during UpdateRollback`, without rolling them back; a skipped resource keeps the state the failed
+ update left it in. Once every other resource rolled back, the stack reaches
+ `UPDATE_ROLLBACK_COMPLETE` with the template, parameters and outputs from before the failed
+ update, and takes updates again.
+
+`ForceRollback=true` skips every `UPDATE_FAILED` resource of the stack, each reported as a
+`ResourcesToSkip` skip is. When a forced rollback skipped a resource and completes, the stack's
+`StackStatusReason` and the reason of its `UPDATE_ROLLBACK_COMPLETE` event are `One or more
+resources were skipped during UpdateRollback. These resources were not rolled back and may have
+drifted from the template. Run drift detection (DetectStackDrift) to review their current state
+before your next update.` A rollback completed with `ResourcesToSkip` has no stack reason, as on
+AWS.
+
+As on AWS, a malformed `ContinueUpdateRollback` is refused with `ValidationError` before the stack
+is looked up, listing every violation: a `StackName` that is missing, empty, or does not match
+`[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*`, and a `ResourcesToSkip` member that does not
+match `[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\.[a-zA-Z0-9]+`. `ForceRollback=true` with a non-empty
+`ResourcesToSkip` is then refused with `ForceRollback and ResourcesToSkip are mutually exclusive`,
+a stack that does not exist with `Stack: <StackName> does not exist`, and a stack in any status
+other than `UPDATE_ROLLBACK_FAILED` with `RollbackUpdatedStack cannot be called from current stack
+status`. `ResourcesToSkip` is checked against the stack after the call is accepted: a skip that is
+not a resource of the stack, or not `UPDATE_FAILED`, returns the stack to `UPDATE_ROLLBACK_FAILED`
+with the reason, touching no resource, and a later call can still finish the rollback. `RoleARN`
+and `ClientRequestToken` are accepted and have no effect.
+
+Known difference: `ResourcesToSkip` does not support a nested stack's resource
+(`NestedStackName.ResourceLogicalID`). When the prefix names a nested stack of the stack, the
+rollback fails again with a reason saying Floci does not support it, where AWS would skip that
+resource.
+
+A stack that reached `UPDATE_ROLLBACK_FAILED` under an earlier Floci version kept no record of its
+state before the update. `ContinueUpdateRollback` then retries every `UPDATE_FAILED` resource and
+keeps the template the stack holds.
+
## Deleted Stacks
A deleted stack is kept for a short window afterwards rather than being forgotten at
diff --git a/docs/services/index.md b/docs/services/index.md
index 9e6ff94391..f714e64613 100644
--- a/docs/services/index.md
+++ b/docs/services/index.md
@@ -36,7 +36,7 @@ Operation counts are exact. For dispatch-table services (Query and JSON 1.1) eac
| [Secrets Manager](secrets-manager.md) | `POST /` + `X-Amz-Target: secretsmanager.*` | JSON 1.1 | 23 |
| [Step Functions](step-functions.md) | `POST /` + `X-Amz-Target: AmazonStatesService.*` | JSON 1.1 | 32 |
| [SWF](swf.md) | `POST /` + `X-Amz-Target: SimpleWorkflowService.*` | JSON 1.0 | 39 |
-| [CloudFormation](cloudformation.md) | `POST /` with `Action=` param | Query | 35 |
+| [CloudFormation](cloudformation.md) | `POST /` with `Action=` param | Query | 36 |
| [Cloud Control API](cloudcontrol.md) | `POST /` + `X-Amz-Target: CloudApiService.*` | JSON 1.1 | 5 |
| [EventBridge](eventbridge.md) | `POST /` + `X-Amz-Target: AmazonEventBridge.*` | JSON 1.1 | 40 |
| [EventBridge Scheduler](scheduler.md) | `/schedules/*`, `/schedule-groups/*`, `/tags/*` | REST JSON | 12 |
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java b/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java
index 93c690ca58..2c51c20fb1 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java
@@ -581,6 +581,7 @@ public class AwsQueryController {
private static final Set<String> CLOUDFORMATION_ACTIONS = Set.of(
"CreateStack", "DeleteStack", "UpdateStack", "DescribeStacks", "UpdateTerminationProtection",
+ "ContinueUpdateRollback",
"ListStacks", "ListExports", "GetTemplate", "GetTemplateSummary", "ValidateTemplate",
"CreateChangeSet", "DeleteChangeSet", "DescribeChangeSet", "ExecuteChangeSet", "ListChangeSets",
"DescribeStackEvents", "DescribeStackResources", "ListStackResources", "DescribeStackResource",
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java
index eac7e9e14c..d42df99e6e 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java
@@ -51,6 +51,7 @@ public class CloudFormationQueryHandler {
case "CreateStack" -> createStack(params, region);
case "UpdateStack" -> updateStack(params, region);
case "DeleteStack" -> deleteStack(params, region);
+ case "ContinueUpdateRollback" -> continueUpdateRollback(params, region);
case "UpdateTerminationProtection" -> updateTerminationProtection(params, region);
case "CreateChangeSet" -> createChangeSet(params, region);
case "DescribeChangeSet" -> describeChangeSet(params, region);
@@ -189,6 +190,28 @@ public class CloudFormationQueryHandler {
}
}
+ // ── ContinueUpdateRollback ───────────────────────────────────────────────
+
+ private Response continueUpdateRollback(MultivaluedMap<String, String> params, String region) {
+ String stackName = params.getFirst("StackName");
+ List<String> resourcesToSkip = extractList(params, "ResourcesToSkip.member.");
+ boolean forceRollback = Boolean.parseBoolean(params.getFirst("ForceRollback"));
+ // RoleARN and ClientRequestToken are accepted and have no effect: Floci assumes no role
+ // for stack operations and does not deduplicate retried requests.
+ try {
+ cfnService.continueUpdateRollback(stackName, resourcesToSkip, forceRollback, region);
+ } catch (AwsException e) {
+ return xmlError(e.getErrorCode(), e.getMessage(), e.getHttpStatus());
+ }
+ String xml = new XmlBuilder()
+ .start("ContinueUpdateRollbackResponse", CF_NS)
+ .raw("<ContinueUpdateRollbackResult/>")
+ .raw(AwsQueryResponse.responseMetadata())
+ .end("ContinueUpdateRollbackResponse")
+ .build();
+ return Response.ok(xml).type("text/xml").build();
+ }
+
// ── UpdateTerminationProtection ─────────────────────────────────────────────
private Response updateTerminationProtection(MultivaluedMap<String, String> params, String region) {
@@ -450,6 +473,7 @@ public class CloudFormationQueryHandler {
.elem("PhysicalResourceId", res.getPhysicalId())
.elem("ResourceType", res.getResourceType())
.elem("ResourceStatus", res.getStatus())
+ .elem("ResourceStatusReason", res.getStatusReason())
.elem("LastUpdatedTimestamp", ISO.format(res.getTimestamp()))
.end("StackResourceDetail")
.end("DescribeStackResourceResult")
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java
index 6b9602cbe9..1c87a5e2a4 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java
@@ -21,6 +21,7 @@ import io.github.hectorvent.floci.services.cloudformation.model.ChangeSet;
import io.github.hectorvent.floci.services.cloudformation.model.Stack;
import io.github.hectorvent.floci.services.cloudformation.model.StackEvent;
import io.github.hectorvent.floci.services.cloudformation.model.StackResource;
+import io.github.hectorvent.floci.services.cloudformation.model.StackUpdateSnapshot;
import io.github.hectorvent.floci.services.cloudformation.model.TemplateSummary;
import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnDynamicReferences;
import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnResourceDispatcher;
@@ -66,6 +67,30 @@ public class CloudFormationService implements ResourceProvider {
private static final int MAX_OPERATION_THREADS = 16;
private static final int MAX_QUEUED_OPERATIONS = 128;
+ // Texts a client may match on, kept here so each has one place to change. All but
+ // SKIP_NESTED_UNSUPPORTED are the texts real CloudFormation emits, measured against a real
+ // account; that one is Floci's own, for a skip AWS accepts and Floci does not support.
+ private static final String UPDATE_REFUSED = "Stack:%s is in %s state and can not be updated.";
+ private static final String CONTINUE_ROLLBACK_REFUSED =
+ "RollbackUpdatedStack cannot be called from current stack status";
+ private static final String SKIP_NOT_UPDATE_FAILED =
+ "Only the resources in UPDATE_FAILED state are allowed to be skipped";
+ private static final String SKIP_NOT_IN_STACK = "Resources with logicalIds [%s] do not belong to stack %s";
+ private static final String SKIP_NESTED_STACK_MISSING = "Stack [%s] does not exist";
+ private static final String SKIP_NESTED_UNSUPPORTED =
+ "Skipping a resource of a nested stack is not supported by Floci: %s";
+ private static final String SKIPPED_RESOURCE_REASON = "Resource skipped during UpdateRollback";
+ private static final String FORCED_ROLLBACK_COMPLETE_REASON =
+ "One or more resources were skipped during UpdateRollback. These resources were not rolled"
+ + " back and may have drifted from the template. Run drift detection (DetectStackDrift)"
+ + " to review their current state before your next update.";
+ private static final String FORCE_WITH_SKIPS = "ForceRollback and ResourcesToSkip are mutually exclusive";
+ private static final String CONTINUE_STACK_MISSING = "Stack: %s does not exist";
+ private static final Pattern CONTINUE_STACK_NAME =
+ Pattern.compile("[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*");
+ private static final Pattern RESOURCE_TO_SKIP =
+ Pattern.compile("[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\\.[a-zA-Z0-9]+");
+
private final ConcurrentHashMap<String, Stack> stacks = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, DeletedStackEntry> deletedStacks = new ConcurrentHashMap<>();
// Account-scoped exports registry: account:region:exportName -> exportValue
@@ -235,10 +260,15 @@ public class CloudFormationService implements ResourceProvider {
* rather than by suffix, because {@code UPDATE_ROLLBACK_COMPLETE} ends the same way and is a
* perfectly updatable stack: it is where an update that failed and rolled back settles, and
* retrying the update is how it is repaired.
+ *
+ * <p>{@code UPDATE_ROLLBACK_FAILED} is refused because the rollback of the last update left
+ * resources in an unknown state, and an update would build on them. ContinueUpdateRollback
+ * finishes that rollback, and DeleteStack removes the stack; both stay allowed.
*/
static boolean refusesUpdate(String status) {
return status != null
- && (status.endsWith("_IN_PROGRESS") || "ROLLBACK_COMPLETE".equals(status));
+ && (status.endsWith("_IN_PROGRESS") || "ROLLBACK_COMPLETE".equals(status)
+ || "UPDATE_ROLLBACK_FAILED".equals(status));
}
/**
@@ -470,8 +500,7 @@ public class CloudFormationService implements ResourceProvider {
}
private static AwsException cannotBeUpdated(String stackId, String status) {
- return new AwsException("ValidationError",
- "Stack:" + stackId + " is in " + status + " state and can not be updated.", 400);
+ return new AwsException("ValidationError", UPDATE_REFUSED.formatted(stackId, status), 400);
}
// AWS refuses a deleted stack's ID by its state, while a name, or an ID it never issued,
@@ -1050,6 +1079,91 @@ public class CloudFormationService implements ResourceProvider {
}
}
+ // ── ContinueUpdateRollback ────────────────────────────────────────────────
+
+ /**
+ * Retries the rollback of a stack in {@code UPDATE_ROLLBACK_FAILED}, the only status AWS accepts
+ * it from; any other status is refused here, before anything changes, as is a malformed request
+ * or a forced rollback that also names skips. The status check and the move to
+ * {@code UPDATE_ROLLBACK_IN_PROGRESS} are one step under the stack's lock, so a racing update or
+ * second retry is refused. Everything else, the {@code resourcesToSkip} checks against the stack
+ * included, runs on the operation executor, because AWS accepts the call and reports a bad skip
+ * as the reason the rollback failed again.
+ */
+ public Future<?> continueUpdateRollback(
+ String stackNameOrArn, List<String> resourcesToSkip, boolean forceRollback, String region) {
+ validateContinueUpdateRollback(stackNameOrArn, resourcesToSkip);
+ if (forceRollback && !resourcesToSkip.isEmpty()) {
+ throw new AwsException("ValidationError", FORCE_WITH_SKIPS, 400);
+ }
+ String accountId = currentAccount();
+ Stack found = resolveStack(stackNameOrArn, region, accountId);
+ if (found == null) {
+ throw new AwsException("ValidationError", CONTINUE_STACK_MISSING.formatted(stackNameOrArn), 400);
+ }
+ String stackName = found.getStackName();
+ List<String> skips = List.copyOf(resourcesToSkip);
+ StackMutationSnapshot[] snapshot = new StackMutationSnapshot[1];
+ Stack stack = stacks.compute(stackKey(accountId, stackName, region), (k, existing) -> {
+ if (existing == null) {
+ throw new AwsException("ValidationError", CONTINUE_STACK_MISSING.formatted(stackNameOrArn), 400);
+ }
+ if (!"UPDATE_ROLLBACK_FAILED".equals(existing.getStatus())) {
+ throw new AwsException("ValidationError", CONTINUE_ROLLBACK_REFUSED, 400);
+ }
+ snapshot[0] = snapshot(existing);
+ existing.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ addEvent(existing, existing.getStackName(), existing.getStackId(),
+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_IN_PROGRESS", "User Initiated");
+ return existing;
+ });
+ try {
+ persistStack(stack);
+ return submitOperation(() -> RequestScopes.runAs(accountId, region,
+ () -> resumeUpdateRollback(stack, skips, forceRollback, region)));
+ } catch (AwsException e) {
+ if ("LimitExceededException".equals(e.getErrorCode())) {
+ Stack restored = restoreStack(accountId, stackName, region, snapshot[0]);
+ if (restored != null) {
+ persistStack(restored);
+ }
+ }
+ throw e;
+ }
+ }
+
+ /**
+ * Refuses a malformed ContinueUpdateRollback request as AWS does, before the stack is looked up:
+ * every constraint violation of {@code StackName} and {@code ResourcesToSkip} in one
+ * {@code ValidationError}, with AWS's texts. A skip has no length limit.
+ */
+ private static void validateContinueUpdateRollback(String stackNameOrArn, List<String> resourcesToSkip) {
+ List<String> violations = new ArrayList<>();
+ if (stackNameOrArn == null) {
+ violations.add("Value null at 'stackName' failed to satisfy constraint: Member must not be null");
+ } else {
+ if (stackNameOrArn.isEmpty()) {
+ violations.add("Value '' at 'stackName' failed to satisfy constraint:"
+ + " Member must have length greater than or equal to 1");
+ }
+ if (!CONTINUE_STACK_NAME.matcher(stackNameOrArn).matches()) {
+ violations.add("Value '" + stackNameOrArn + "' at 'stackName' failed to satisfy constraint:"
+ + " Member must satisfy regular expression pattern: " + CONTINUE_STACK_NAME.pattern());
+ }
+ }
+ if (resourcesToSkip.stream().anyMatch(skip -> !RESOURCE_TO_SKIP.matcher(skip).matches())) {
+ violations.add("Value '" + resourcesToSkip + "' at 'resourcesToSkip' failed to satisfy constraint:"
+ + " Member must satisfy constraint: [Member must satisfy regular expression pattern: "
+ + RESOURCE_TO_SKIP.pattern() + "]");
+ }
+ if (!violations.isEmpty()) {
+ String prefix = violations.size() == 1
+ ? "1 validation error detected: "
+ : violations.size() + " validation errors detected: ";
+ throw new AwsException("ValidationError", prefix + String.join("; ", violations), 400);
+ }
+ }
+
private StackMutationSnapshot snapshot(Stack stack) {
Map<ChangeSet, ChangeSetState> states = new IdentityHashMap<>();
for (ChangeSet changeSet : stack.changeSetsSnapshot().values()) {
@@ -2115,6 +2229,109 @@ public class CloudFormationService implements ResourceProvider {
}
}
+ /**
+ * The body of ContinueUpdateRollback, on a stack already moved to UPDATE_ROLLBACK_IN_PROGRESS:
+ * checks {@code resourcesToSkip}, reports each skipped resource UPDATE_COMPLETE, runs the
+ * rollback walker again over the resources still owed a rollback, and finishes as the rollback
+ * of the failed update does. A bad skip touches no resource: the stack returns to
+ * UPDATE_ROLLBACK_FAILED with the reason, keeping what a later retry needs. A forced rollback
+ * skips every UPDATE_FAILED resource of the stack instead, and says so when it completes.
+ */
+ void resumeUpdateRollback(Stack stack, List<String> resourcesToSkip, boolean forceRollback, String region) {
+ try {
+ List<String> skips = forceRollback
+ ? stack.resourcesSnapshot().values().stream()
+ .filter(resource -> "UPDATE_FAILED".equals(resource.getStatus()))
+ .map(StackResource::getLogicalId)
+ .toList()
+ : resourcesToSkip;
+ String invalidSkip = invalidSkipReason(stack, skips);
+ if (invalidSkip != null) {
+ // Reason and event before the status, which is the volatile publishing write.
+ stack.setStatusReason(invalidSkip);
+ addEvent(stack, stack.getStackName(), stack.getStackId(),
+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_FAILED", invalidSkip);
+ stack.setStatus("UPDATE_ROLLBACK_FAILED");
+ persistStack(stack);
+ LOG.warnv("Stack {0} update rollback not continued: {1}", stack.getStackName(), invalidSkip);
+ return;
+ }
+ StackUpdateSnapshot previousState = stack.getUpdateRollbackSnapshot();
+ List<String> owedResourceIds = stack.getUpdateRollbackResourceIds();
+ if (previousState == null || owedResourceIds == null) {
+ // A stack stored before the snapshot was kept: every UPDATE_FAILED resource is owed a
+ // rollback, and the template the stack holds now is the one it keeps.
+ previousState = snapshotForUpdate(stack);
+ owedResourceIds = new ArrayList<>();
+ for (StackResource previous : previousState.resources().values()) {
+ if ("UPDATE_FAILED".equals(previous.getStatus())) {
+ owedResourceIds.add(previous.getLogicalId());
+ previous.setStatus("UPDATE_COMPLETE");
+ previous.setStatusReason(null);
+ }
+ }
+ }
+ Set<String> owed = new LinkedHashSet<>(owedResourceIds);
+ for (String logicalId : skips) {
+ StackResource resource = stack.getResources().get(logicalId);
+ resource.setStatus("UPDATE_COMPLETE");
+ resource.setStatusReason(SKIPPED_RESOURCE_REASON);
+ addEvent(stack, logicalId, resource.getPhysicalId(), resource.getResourceType(),
+ "UPDATE_COMPLETE", SKIPPED_RESOURCE_REASON);
+ owed.remove(logicalId);
+ }
+ // Recorded before and after the walker, so an unexpected failure never re-runs a skip or a done rollback.
+ stack.setUpdateRollbackSnapshot(previousState);
+ stack.setUpdateRollbackResourceIds(List.copyOf(owed));
+ List<String> rollbackFailures = rollbackUpdatedResources(
+ stack, previousState.resources(), owed, region);
+ stack.setUpdateRollbackResourceIds(List.copyOf(rollbackFailures));
+ finishUpdateRollback(stack, region, previousState, rollbackFailures,
+ forceRollback && !skips.isEmpty() ? FORCED_ROLLBACK_COMPLETE_REASON : null);
+ } catch (Exception e) {
+ // Nobody reads this worker's future: a stack left UPDATE_ROLLBACK_IN_PROGRESS here could
+ // never be continued. The snapshot and the resources still owed a rollback are kept.
+ LOG.errorv("Stack {0} update rollback could not continue: {1}",
+ stack.getStackName(), e.getMessage());
+ stack.setStatusReason(e.getMessage());
+ addEvent(stack, stack.getStackName(), stack.getStackId(),
+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_FAILED", e.getMessage());
+ stack.setStatus("UPDATE_ROLLBACK_FAILED");
+ persistStack(stack);
+ }
+ }
+
+ /**
+ * Why AWS would fail the rollback over {@code resourcesToSkip}, or {@code null} when every
+ * skip is an UPDATE_FAILED resource of this stack. AWS names a nested stack's resource
+ * {@code NestedStackName.ResourceLogicalID}; Floci does not support skipping one.
+ */
+ private static String invalidSkipReason(Stack stack, List<String> resourcesToSkip) {
+ List<String> notInStack = new ArrayList<>();
+ boolean notUpdateFailed = false;
+ for (String logicalId : resourcesToSkip) {
+ int dot = logicalId.indexOf('.');
+ if (dot >= 0) {
+ String nestedStackId = logicalId.substring(0, dot);
+ StackResource nested = stack.getResources().get(nestedStackId);
+ if (nested == null || !"AWS::CloudFormation::Stack".equals(nested.getResourceType())) {
+ return SKIP_NESTED_STACK_MISSING.formatted(nestedStackId);
+ }
+ return SKIP_NESTED_UNSUPPORTED.formatted(logicalId);
+ }
+ StackResource resource = stack.getResources().get(logicalId);
+ if (resource == null) {
+ notInStack.add(logicalId);
+ } else if (!"UPDATE_FAILED".equals(resource.getStatus())) {
+ notUpdateFailed = true;
+ }
+ }
+ if (!notInStack.isEmpty()) {
+ return SKIP_NOT_IN_STACK.formatted(String.join(", ", notInStack), stack.getStackName());
+ }
+ return notUpdateFailed ? SKIP_NOT_UPDATE_FAILED : null;
+ }
+
private void rollbackFailedUpdate(
Stack stack,
String region,
@@ -2128,6 +2345,23 @@ public class CloudFormationService implements ResourceProvider {
List<String> rollbackFailures = rollbackUpdatedResources(
stack, previousState.resources(), attemptedResourceIds, region);
+ finishUpdateRollback(stack, region, previousState, rollbackFailures, null);
+ }
+
+ /**
+ * Ends an update rollback, the first one and every ContinueUpdateRollback retry alike: restores
+ * the parameters, outputs and exports, and the template once every resource rolled back, then
+ * settles the stack in UPDATE_ROLLBACK_COMPLETE, with {@code completionReason} as its reason, or
+ * UPDATE_ROLLBACK_FAILED. A rollback that fails keeps the snapshot and the resources still owed
+ * a rollback on the stack for a retry.
+ */
+ private void finishUpdateRollback(
+ Stack stack,
+ String region,
+ StackUpdateSnapshot previousState,
+ List<String> rollbackFailures,
+ String completionReason) {
+ List<String> owedResourceIds = List.copyOf(rollbackFailures);
// Parameters are independent of resource-rollback outcome - always restore them to the last
// successfully deployed values, even when resource rollback itself fails and the stack lands
// in UPDATE_ROLLBACK_FAILED, so DescribeStacks and later change-set previews don't keep
@@ -2152,19 +2386,24 @@ public class CloudFormationService implements ResourceProvider {
}
stack.setLastUpdatedTime(now());
if (rollbackFailures.isEmpty()) {
- stack.setStatus("UPDATE_ROLLBACK_COMPLETE");
- stack.setStatusReason(null);
+ stack.setUpdateRollbackSnapshot(null);
+ stack.setUpdateRollbackResourceIds(null);
+ // Reason and event before the status, which is the volatile publishing write.
+ stack.setStatusReason(completionReason);
addEvent(stack, stack.getStackName(), stack.getStackId(),
- "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_COMPLETE", null);
+ "AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_COMPLETE", completionReason);
+ stack.setStatus("UPDATE_ROLLBACK_COMPLETE");
LOG.infov("Stack {0} update rolled back (UPDATE_ROLLBACK_COMPLETE)",
stack.getStackName());
} else {
String reason = "The following resource(s) failed to roll back: ["
+ String.join(", ", rollbackFailures) + "].";
- stack.setStatus("UPDATE_ROLLBACK_FAILED");
+ stack.setUpdateRollbackSnapshot(previousState);
+ stack.setUpdateRollbackResourceIds(owedResourceIds);
stack.setStatusReason(reason);
addEvent(stack, stack.getStackName(), stack.getStackId(),
"AWS::CloudFormation::Stack", "UPDATE_ROLLBACK_FAILED", reason);
+ stack.setStatus("UPDATE_ROLLBACK_FAILED");
LOG.errorv("Stack {0} update rollback failed: {1}", stack.getStackName(), reason);
}
persistStack(stack);
@@ -2382,17 +2621,6 @@ public class CloudFormationService implements ResourceProvider {
return copy;
}
- private record StackUpdateSnapshot(
- String templateBody,
- String originalTemplateBody,
- Map<String, String> parameters,
- Map<String, String> resolvedParameters,
- Map<String, String> outputs,
- Map<String, String> exports,
- Map<String, String> outputExportNames,
- Map<String, StackResource> resources) {
- }
-
private record UpdateCleanupFailure(
String logicalId, String physicalId, String failureReason) {
}
@@ -2593,6 +2821,8 @@ public class CloudFormationService implements ResourceProvider {
throw new IllegalStateException(reason);
}
+ stack.setUpdateRollbackSnapshot(null);
+ stack.setUpdateRollbackResourceIds(null);
// Before the status, never after. Status is the volatile publishing write, so a reader
// that observes DELETE_COMPLETE is guaranteed to observe every write that preceded it.
// Assigned afterwards, a concurrent DescribeStacks can report the terminal status with
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java
index b8c2b0517a..14f0e4d0d9 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java
@@ -64,6 +64,14 @@ public class Stack {
private Map<String, ChangeSet> changeSets = orderedMap();
private Map<String, String> tags = orderedMap();
private boolean enableTerminationProtection = false;
+ /**
+ * Set when an update rollback fails, and kept until a rollback completes or the stack is
+ * deleted: the state from before the failed update, and the logical ids whose rollback has not
+ * succeeded yet. A ContinueUpdateRollback retry reads them in UPDATE_ROLLBACK_IN_PROGRESS, so
+ * that transition must keep them. Absent on a stack stored before they were kept.
+ */
+ private StackUpdateSnapshot updateRollbackSnapshot;
+ private List<String> updateRollbackResourceIds;
public String getStackId() { return stackId; }
public void setStackId(String stackId) { this.stackId = stackId; }
@@ -187,4 +195,8 @@ public class Stack {
}
public boolean isEnableTerminationProtection() { return enableTerminationProtection; }
public void setEnableTerminationProtection(boolean enableTerminationProtection) { this.enableTerminationProtection = enableTerminationProtection; }
+ public StackUpdateSnapshot getUpdateRollbackSnapshot() { return updateRollbackSnapshot; }
+ public void setUpdateRollbackSnapshot(StackUpdateSnapshot updateRollbackSnapshot) { this.updateRollbackSnapshot = updateRollbackSnapshot; }
+ public List<String> getUpdateRollbackResourceIds() { return updateRollbackResourceIds; }
+ public void setUpdateRollbackResourceIds(List<String> updateRollbackResourceIds) { this.updateRollbackResourceIds = updateRollbackResourceIds; }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.java
new file mode 100644
index 0000000000..cce509008f
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.java
@@ -0,0 +1,24 @@
+package io.github.hectorvent.floci.services.cloudformation.model;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import io.quarkus.runtime.annotations.RegisterForReflection;
+
+import java.util.Map;
+
+/**
+ * A stack's state from before an update, which the rollback of that update restores. A rollback
+ * that fails keeps it on the stack, so ContinueUpdateRollback can run the rollback again after the
+ * update that took it has returned, or after a restart.
+ */
+@RegisterForReflection
+@JsonIgnoreProperties(ignoreUnknown = true)
+public record StackUpdateSnapshot(
+ String templateBody,
+ String originalTemplateBody,
+ Map<String, String> parameters,
+ Map<String, String> resolvedParameters,
+ Map<String, String> outputs,
+ Map<String, String> exports,
+ Map<String, String> outputExportNames,
+ Map<String, StackResource> resources) {
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java
index 101785cc10..2437f92f07 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java
@@ -23,9 +23,10 @@ import java.util.Set;
* or put back when a later resource fails the update and it rolls back.
*
* <p>The record on the resource lists every entity owed a delete, not only the one this update
- * displaced: a replacement whose rollback could not delete it stays listed, so the next committed
- * update or the stack delete removes it instead of it being forgotten. The prior entity's id and
- * attributes are kept beside the list only while this update's replacement can still be rolled back.
+ * displaced: a replacement whose rollback could not delete it stays listed, so a retried rollback,
+ * the next committed update or the stack delete removes it instead of it being forgotten. The prior
+ * entity's id and attributes are kept beside the list only while this update's replacement can
+ * still be rolled back.
*
* <p>{@code PipesCfnProvisioner} carries its own richer record because it also snapshots the
* entity's configuration for in-place rollback; this is the plain version for types without one.
@@ -70,6 +71,7 @@ public final class ReplacementCleanup {
boolean replaced) {
ObjectNode cleanup = readOrEmpty(r);
cleanup.remove("priorPhysicalId");
+ cleanup.remove("undeletedReplacement");
cleanup.remove("priorAttributes");
cleanup.put("region", ctx.region());
String prior = ctx.priorPhysicalId();
@@ -99,11 +101,28 @@ public final class ReplacementCleanup {
* standing again, and a resource still owing its delete would put it on the next cleanup's list.
* A replacement whose delete fails is listed in its place, never retained (a failed update
* created it), so the next committed update or the stack delete removes it, and the failure
- * propagates so the stack reports it. Returns false when this update replaced nothing, which
+ * propagates so the stack reports it. A retry (CloudFormation {@code ContinueUpdateRollback})
+ * re-attempts that failed delete. Returns false when this update replaced nothing, which
* leaves the engine's handling of in-place updates unchanged.
*/
static boolean rollback(StackResource r, Deleter deleter) {
ObjectNode cleanup = read(r);
+ String undeleted = cleanup == null ? null : cleanup.path("undeletedReplacement").asText(null);
+ if (undeleted != null) {
+ JsonNode owed = null;
+ for (JsonNode entry : cleanup.path("displaced")) {
+ if (undeleted.equals(entry.path("physicalId").asText(null))) {
+ owed = entry;
+ }
+ }
+ if (owed == null) {
+ cleanup.remove("undeletedReplacement");
+ write(r, cleanup);
+ return true;
+ }
+ return deleteReplacement(r, cleanup, deleter, owed.path("resourceType").asText(r.getResourceType()),
+ undeleted);
+ }
if (cleanup == null || cleanup.path("priorPhysicalId").asText(null) == null) {
return false;
}
@@ -125,13 +144,23 @@ public final class ReplacementCleanup {
if (prior.equals(replacement) && !CfnResourceDispatcher.isStub(prior, r.getAttributes())) {
return true;
}
+ return deleteReplacement(r, cleanup, deleter, r.getResourceType(), replacement);
+ }
+
+ /** Deletes a rolled-back replacement; one whose delete throws stays owed and named for the retry. */
+ private static boolean deleteReplacement(StackResource r, ObjectNode cleanup, Deleter deleter,
+ String resourceType, String replacement) {
try {
- deleter.delete(r.getResourceType(), replacement, region(cleanup));
+ deleter.delete(resourceType, replacement, region(cleanup));
} catch (RuntimeException deleteFailure) {
- addDisplaced(cleanup, replacement, r.getResourceType(), region(cleanup), false);
+ addDisplaced(cleanup, replacement, resourceType, region(cleanup), false);
+ cleanup.put("undeletedReplacement", replacement);
write(r, cleanup);
throw deleteFailure;
}
+ removeDisplaced(cleanup, replacement);
+ cleanup.remove("undeletedReplacement");
+ write(r, cleanup);
return true;
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.java
new file mode 100644
index 0000000000..93cb00cdb3
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.java
@@ -0,0 +1,585 @@
+package io.github.hectorvent.floci.services.cloudformation;
+
+import io.github.hectorvent.floci.core.common.XmlParser;
+import io.github.hectorvent.floci.services.cloudwatch.dashboards.CloudWatchDashboardsService;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.mockito.InjectSpy;
+import io.restassured.RestAssured;
+import io.restassured.config.EncoderConfig;
+import io.restassured.http.ContentType;
+import io.restassured.response.Response;
+import io.restassured.specification.RequestSpecification;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentMatcher;
+import org.mockito.Mockito;
+
+import java.util.List;
+import java.util.Map;
+
+import static io.restassured.RestAssured.given;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.anyMap;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.argThat;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+/**
+ * A stack whose update rollback failed sits in UPDATE_ROLLBACK_FAILED. AWS refuses to update it;
+ * the ways out are ContinueUpdateRollback, optionally skipping the resources that cannot roll
+ * back, and DeleteStack. Each test reaches that state as a real deploy does: an update changes a
+ * log group, which has no update rollback, then adds a second log group whose name collides with
+ * one made outside the stack.
+ *
+ * <p>The test module has no AWS SDK CloudFormation client, so the Query protocol is driven with
+ * form POSTs, as the other CloudFormation integration tests do.
+ */
+@QuarkusTest
+class CloudFormationContinueUpdateRollbackIntegrationTest {
+
+ private static final String SKIPPED = "Resource skipped during UpdateRollback";
+ private static final String USER_INITIATED = "User Initiated";
+ private static final String FORCED = "One or more resources were skipped during UpdateRollback."
+ + " These resources were not rolled back and may have drifted from the template."
+ + " Run drift detection (DetectStackDrift) to review their current state before your next update.";
+ private static final String STACK_NAME_PATTERN = "[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*";
+ private static final String SKIP_PATTERN = "[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\\.[a-zA-Z0-9]+";
+ private static final String NULL_STACK_NAME =
+ "Value null at 'stackName' failed to satisfy constraint: Member must not be null";
+ private static final String CW_AUTH =
+ "AWS4-HMAC-SHA256 Credential=test/20260908/us-east-1/monitoring/aws4_request";
+
+ @InjectSpy
+ CloudWatchDashboardsService dashboardsService;
+
+ @Test
+ void updateAndChangeSetAreRefusedWhileDeleteStillWorks() {
+ RollbackFailedStack stack = rollbackFailedStack("cur-refuse-");
+ try {
+ cfn("UpdateStack", stack.name())
+ .formParam("TemplateBody", template(stack, 14, false))
+ .when()
+ .post("/")
+ .then()
+ .statusCode(400)
+ .body(containsString("<Code>ValidationError</Code>"))
+ .body(containsString("<Message>Stack:" + stack.id()
+ + " is in UPDATE_ROLLBACK_FAILED state and can not be updated.</Message>"));
+
+ cfn("CreateChangeSet", stack.name())
+ .formParam("ChangeSetName", "cur-refuse-cs")
+ .formParam("ChangeSetType", "UPDATE")
+ .formParam("TemplateBody", template(stack, 14, false))
+ .when()
+ .post("/")
+ .then()
+ .statusCode(400)
+ .body(containsString("<Code>ValidationError</Code>"))
+ .body(containsString("<Message>Stack:" + stack.id()
+ + " is in UPDATE_ROLLBACK_FAILED state and can not be updated.</Message>"));
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", CfnStackWaits.awaitTerminal(stack.name()).status());
+
+ cfn("DeleteStack", stack.name()).when().post("/").then().statusCode(200);
+ CfnStackWaits.awaitStackDeleted(stack.name());
+ } finally {
+ cleanUp(stack);
+ }
+ }
+
+ /** A log group cannot roll an update back, so retrying the rollback fails the same way again. */
+ @Test
+ void continueUpdateRollbackWithoutSkipsFailsAgain() {
+ RollbackFailedStack stack = rollbackFailedStack("cur-retry-");
+ try {
+ cfn("ContinueUpdateRollback", stack.name())
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200)
+ .body(containsString("<ContinueUpdateRollbackResponse"
+ + " xmlns=\"http://cloudformation.amazonaws.com/doc/2010-05-15/\">"
+ + "<ContinueUpdateRollbackResult/><ResponseMetadata><RequestId>"));
+
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());
+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());
+ assertTrue(state.reason().contains("LogGroup"), state.reason());
+ assertEquals("UPDATE_FAILED", resourceStatuses(stack.name()).get("LogGroup"));
+
+ List<Map<String, String>> events = events(stack.id());
+ List<Map<String, String>> rollbacks = events.stream()
+ .filter(event -> stack.name().equals(event.get("LogicalResourceId"))
+ && "UPDATE_ROLLBACK_IN_PROGRESS".equals(event.get("ResourceStatus")))
+ .toList();
+ assertEquals(2, rollbacks.size(), events.toString());
+ assertEquals(USER_INITIATED, rollbacks.get(0).get("ResourceStatusReason"), events.toString());
+
+ // The retry restores nothing it could not restore the first time, so the failed
+ // template stays.
+ getTemplate(stack.name()).then().statusCode(200).body(containsString(stack.outsideGroup()));
+ } finally {
+ cleanUp(stack);
+ }
+ }
+
+ @Test
+ void skippingTheLogGroupCompletesTheRollbackAndUnblocksUpdates() {
+ RollbackFailedStack stack = rollbackFailedStack("cur-skip-");
+ try {
+ cfn("ContinueUpdateRollback", stack.id())
+ .formParam("ResourcesToSkip.member.1", "LogGroup")
+ .formParam("RoleARN", "arn:aws:iam::000000000000:role/ignored")
+ .formParam("ClientRequestToken", "cur-skip-token")
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());
+ cfn("DescribeStackResource", stack.name())
+ .formParam("LogicalResourceId", "LogGroup")
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200)
+ .body(containsString("<ResourceStatus>UPDATE_COMPLETE</ResourceStatus>"))
+ .body(containsString("<ResourceStatusReason>" + SKIPPED + "</ResourceStatusReason>"));
+
+ // Newest first: the skipped resource's UPDATE_COMPLETE follows the second rollback start.
+ List<Map<String, String>> events = events(stack.id());
+ boolean skipped = false;
+ for (Map<String, String> event : events) {
+ if (stack.name().equals(event.get("LogicalResourceId"))
+ && "UPDATE_ROLLBACK_IN_PROGRESS".equals(event.get("ResourceStatus"))) {
+ assertEquals(USER_INITIATED, event.get("ResourceStatusReason"), event.toString());
+ break;
+ }
+ if ("LogGroup".equals(event.get("LogicalResourceId"))
+ && "UPDATE_COMPLETE".equals(event.get("ResourceStatus"))
+ && SKIPPED.equals(event.get("ResourceStatusReason"))
+ && stack.logGroup().equals(event.get("PhysicalResourceId"))) {
+ skipped = true;
+ }
+ }
+ assertTrue(skipped, "no skipped UPDATE_COMPLETE event for LogGroup in " + events);
+
+ // The completed rollback restores the template from before the failed update.
+ getTemplate(stack.name()).then().statusCode(200).body(not(containsString(stack.outsideGroup())));
+
+ // The stack is no longer UPDATE_ROLLBACK_FAILED, so a second call is refused.
+ assertContinueRefused(stack.name());
+
+ cfn("UpdateStack", stack.name())
+ .formParam("TemplateBody", template(stack, 14, false))
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+ assertEquals("UPDATE_COMPLETE", CfnStackWaits.awaitTerminal(stack.name()).status());
+ } finally {
+ cleanUp(stack);
+ }
+ }
+
+ @Test
+ void continueUpdateRollbackIsRefusedOnAStackInAnotherState() {
+ String suffix = Long.toString(System.nanoTime(), 36);
+ String stackName = "cur-state-" + suffix;
+ String group = "/cur-state/" + suffix;
+ try {
+ cfn("CreateStack", stackName)
+ .formParam("TemplateBody", "{\"Resources\":{\"LogGroup\":{\"Type\":\"AWS::Logs::LogGroup\","
+ + "\"Properties\":{\"LogGroupName\":\"" + group + "\"}}}}")
+ .when().post("/").then().statusCode(200);
+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());
+
+ assertContinueRefused(stackName);
+
+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());
+ } finally {
+ cfn("DeleteStack", stackName).when().post("/");
+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + group + "\"}");
+ }
+ }
+
+ /**
+ * AWS accepts a call with a bad skip and fails the rollback again with the reason, touching no
+ * resource. What the retry needs is kept, so a later call with a good skip still finishes it.
+ */
+ @Test
+ void aBadSkipFailsTheRollbackAgainWithoutTouchingAResource() {
+ RollbackFailedStack stack = rollbackFailedStack("cur-badskip-");
+ try {
+ assertSkipFails(stack, "Missing",
+ "Resources with logicalIds [Missing] do not belong to stack " + stack.name());
+ assertSkipFails(stack, "Stable", "Only the resources in UPDATE_FAILED state are allowed to be skipped");
+ assertSkipFails(stack, "Nested.LogGroup", "Stack [Nested] does not exist");
+
+ Map<String, String> statuses = resourceStatuses(stack.name());
+ assertEquals("UPDATE_FAILED", statuses.get("LogGroup"), statuses.toString());
+ assertEquals("CREATE_COMPLETE", statuses.get("Stable"), statuses.toString());
+
+ cfn("ContinueUpdateRollback", stack.name())
+ .formParam("ResourcesToSkip.member.1", "LogGroup")
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());
+ getTemplate(stack.name()).then().statusCode(200).body(not(containsString(stack.outsideGroup())));
+ } finally {
+ cleanUp(stack);
+ }
+ }
+
+ /**
+ * ContinueUpdateRollback is a real retry: it runs the restore of a resource still owed one
+ * again. A dashboard keeps its update snapshot until its restore succeeds, so a restore that
+ * fails once is put right by the retry.
+ */
+ @Test
+ void continueUpdateRollbackRetriesARestorationThatThrew() {
+ String suffix = Long.toString(System.nanoTime(), 36);
+ String stackName = "cur-dashboard-retry-" + suffix;
+ String dashboardName = "cur-dashboard-retry-" + suffix;
+ try {
+ cfn("CreateStack", stackName)
+ .formParam("TemplateBody", dashboardTemplate(dashboardName, "Before", false))
+ .when().post("/").then().statusCode(200);
+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());
+
+ // The create wrote the pre-update body too; only the restores are counted below.
+ Mockito.clearInvocations(dashboardsService);
+ ArgumentMatcher<String> preUpdateBody = body -> body != null && body.contains("Before");
+ Mockito.doThrow(new IllegalStateException("simulated restore failure"))
+ .doCallRealMethod()
+ .when(dashboardsService)
+ .putDashboard(eq(dashboardName), argThat(preUpdateBody), anyMap(), anyString());
+
+ cfn("UpdateStack", stackName)
+ .formParam("TemplateBody", dashboardTemplate(dashboardName, "After", true))
+ .when().post("/").then().statusCode(200);
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stackName);
+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());
+ assertEquals("UPDATE_FAILED", resourceStatuses(stackName).get("Dashboard"));
+
+ cfn("ContinueUpdateRollback", stackName).when().post("/").then().statusCode(200);
+ state = CfnStackWaits.awaitTerminal(stackName);
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());
+ assertEquals("CREATE_COMPLETE", resourceStatuses(stackName).get("Dashboard"));
+ verify(dashboardsService, times(2))
+ .putDashboard(eq(dashboardName), argThat(preUpdateBody), anyMap(), anyString());
+
+ String body = given()
+ .contentType("application/x-www-form-urlencoded")
+ .header("Authorization", CW_AUTH)
+ .formParam("Action", "GetDashboard")
+ .formParam("DashboardName", dashboardName)
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200)
+ .extract().path("GetDashboardResponse.GetDashboardResult.DashboardBody");
+ assertTrue(body.contains("Before") && !body.contains("After"), body);
+ } finally {
+ cfn("DeleteStack", stackName).when().post("/");
+ }
+ }
+
+ @Test
+ void aMissingStackNameIsAValidationError() {
+ assertValidationError(cfn("ContinueUpdateRollback", null),
+ "1 validation error detected: " + NULL_STACK_NAME);
+ }
+
+ @Test
+ void anEmptyStackNameFailsBothItsLengthAndItsPattern() {
+ assertValidationError(cfn("ContinueUpdateRollback", ""),
+ "2 validation errors detected: Value '' at 'stackName' failed to satisfy constraint:"
+ + " Member must have length greater than or equal to 1; Value '' at 'stackName' failed"
+ + " to satisfy constraint: Member must satisfy regular expression pattern: "
+ + STACK_NAME_PATTERN);
+ }
+
+ @Test
+ void aStackNameOutsideThePatternIsAValidationError() {
+ assertValidationError(cfn("ContinueUpdateRollback", "bad_stack!"),
+ "1 validation error detected: Value 'bad_stack!' at 'stackName' failed to satisfy constraint:"
+ + " Member must satisfy regular expression pattern: " + STACK_NAME_PATTERN);
+ }
+
+ /** One bad member rejects the whole list, before the stack, which does not exist, is looked up. */
+ @Test
+ void aSkipOutsideThePatternRejectsTheWholeList() {
+ assertValidationError(cfn("ContinueUpdateRollback", "cur-no-such-stack")
+ .formParam("ResourcesToSkip.member.1", "CrA")
+ .formParam("ResourcesToSkip.member.2", "bad_id"),
+ "1 validation error detected: " + badSkips("[CrA, bad_id]"));
+ }
+
+ @Test
+ void aMissingStackNameAndABadSkipAreBothReportedStackNameFirst() {
+ assertValidationError(cfn("ContinueUpdateRollback", null)
+ .formParam("ResourcesToSkip.member.1", "bad_id"),
+ "2 validation errors detected: " + NULL_STACK_NAME + "; " + badSkips("[bad_id]"));
+ }
+
+ @Test
+ void forceRollbackWithSkipsIsRefusedWithoutTouchingTheStack() {
+ RollbackFailedStack stack = rollbackFailedStack("cur-force-skip-");
+ try {
+ List<Map<String, String>> eventsBefore = events(stack.id());
+
+ assertValidationError(cfn("ContinueUpdateRollback", stack.name())
+ .formParam("ForceRollback", "true")
+ .formParam("ResourcesToSkip.member.1", "LogGroup"),
+ "ForceRollback and ResourcesToSkip are mutually exclusive");
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", CfnStackWaits.awaitTerminal(stack.name()).status());
+ assertEquals(eventsBefore, events(stack.id()));
+ } finally {
+ cleanUp(stack);
+ }
+ }
+
+ @Test
+ void aStackThatDoesNotExistIsNamedAsGiven() {
+ String stackName = "cur-missing-" + Long.toString(System.nanoTime(), 36);
+ assertValidationError(cfn("ContinueUpdateRollback", stackName), "Stack: " + stackName + " does not exist");
+ }
+
+ /**
+ * ForceRollback skips every UPDATE_FAILED resource, reporting each as a ResourcesToSkip skip
+ * is, and the completed rollback says in the stack's reason that resources were skipped.
+ */
+ @Test
+ void forceRollbackSkipsTheFailedResourcesAndUnblocksUpdates() {
+ RollbackFailedStack stack = rollbackFailedStack("cur-force-");
+ try {
+ cfn("ContinueUpdateRollback", stack.name())
+ .formParam("ForceRollback", "true")
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", state.status(), state.reason());
+ assertEquals(FORCED, state.reason());
+ assertEquals("UPDATE_COMPLETE", resourceStatuses(stack.name()).get("LogGroup"));
+ cfn("DescribeStackResource", stack.name())
+ .formParam("LogicalResourceId", "LogGroup")
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200)
+ .body(containsString("<ResourceStatus>UPDATE_COMPLETE</ResourceStatus>"))
+ .body(containsString("<ResourceStatusReason>" + SKIPPED + "</ResourceStatusReason>"));
+
+ // Newest first: the completion, then the skipped log group.
+ List<Map<String, String>> events = events(stack.id());
+ Map<String, String> completed = events.get(0);
+ assertEquals(stack.name(), completed.get("LogicalResourceId"), events.toString());
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", completed.get("ResourceStatus"), events.toString());
+ assertEquals(FORCED, completed.get("ResourceStatusReason"), events.toString());
+ Map<String, String> skipped = events.get(1);
+ assertEquals("LogGroup", skipped.get("LogicalResourceId"), events.toString());
+ assertEquals("UPDATE_COMPLETE", skipped.get("ResourceStatus"), events.toString());
+ assertEquals(SKIPPED, skipped.get("ResourceStatusReason"), events.toString());
+
+ getTemplate(stack.name()).then().statusCode(200).body(not(containsString(stack.outsideGroup())));
+
+ cfn("UpdateStack", stack.name())
+ .formParam("TemplateBody", template(stack, 14, false))
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+ assertEquals("UPDATE_COMPLETE", CfnStackWaits.awaitTerminal(stack.name()).status());
+ } finally {
+ cleanUp(stack);
+ }
+ }
+
+ // ── Helpers ───────────────────────────────────────────────────────────────
+
+ private static String badSkips(String list) {
+ return "Value '" + list + "' at 'resourcesToSkip' failed to satisfy constraint: Member must satisfy"
+ + " constraint: [Member must satisfy regular expression pattern: " + SKIP_PATTERN + "]";
+ }
+
+ /** AWS rejects a malformed request with ValidationError before it looks the stack up. */
+ private static void assertValidationError(RequestSpecification request, String message) {
+ String body = request.when().post("/").then().statusCode(400).extract().asString();
+ assertEquals("ValidationError", XmlParser.extractFirst(body, "Code", null), body);
+ assertEquals(message, XmlParser.extractFirst(body, "Message", null), body);
+ }
+
+ private record RollbackFailedStack(String name, String id, String logGroup, String outsideGroup) {
+ }
+
+ /**
+ * Creates a stack, then updates it so the log group changes and an added log group collides
+ * with one created outside the stack. The update fails, the log group cannot roll back, and the
+ * stack settles in UPDATE_ROLLBACK_FAILED with the log group UPDATE_FAILED.
+ */
+ private static RollbackFailedStack rollbackFailedStack(String prefix) {
+ String suffix = Long.toString(System.nanoTime(), 36);
+ String name = prefix + suffix;
+ String logGroup = "/" + prefix + suffix + "/lg";
+ String outsideGroup = "/" + prefix + suffix + "/outside";
+ logs("CreateLogGroup", "{\"logGroupName\":\"" + outsideGroup + "\"}").then().statusCode(200);
+
+ RollbackFailedStack stack = new RollbackFailedStack(name, null, logGroup, outsideGroup);
+ String id = XmlParser.extractFirst(
+ cfn("CreateStack", name)
+ .formParam("TemplateBody", template(stack, null, false))
+ .when().post("/").then().statusCode(200).extract().asString(),
+ "StackId", null);
+ assertEquals("CREATE_COMPLETE", CfnStackWaits.awaitTerminal(name).status());
+
+ cfn("UpdateStack", name)
+ .formParam("TemplateBody", template(stack, 7, true))
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(name);
+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());
+ Map<String, String> statuses = resourceStatuses(name);
+ assertEquals("UPDATE_FAILED", statuses.get("LogGroup"), statuses.toString());
+ assertEquals("CREATE_COMPLETE", statuses.get("Stable"), statuses.toString());
+ return new RollbackFailedStack(name, id, logGroup, outsideGroup);
+ }
+
+ /**
+ * {@code Stable} never changes. {@code LogGroup} carries {@code retention} when it is set, and
+ * {@code Dup}, which depends on it, claims the name of the group made outside the stack.
+ */
+ private static String template(RollbackFailedStack stack, Integer retention, boolean collide) {
+ String retentionProperty = retention == null ? "" : ", \"RetentionInDays\": " + retention;
+ String dup = !collide ? "" : """
+ ,
+ "Dup": {
+ "Type": "AWS::Logs::LogGroup",
+ "DependsOn": "LogGroup",
+ "Properties": {"LogGroupName": "%s"}
+ }""".formatted(stack.outsideGroup());
+ return """
+ {
+ "Resources": {
+ "Stable": {
+ "Type": "AWS::Logs::LogGroup",
+ "Properties": {"LogGroupName": "%s/stable"}
+ },
+ "LogGroup": {
+ "Type": "AWS::Logs::LogGroup",
+ "Properties": {"LogGroupName": "%s"%s}
+ }%s
+ }
+ }
+ """.formatted(stack.logGroup(), stack.logGroup(), retentionProperty, dup);
+ }
+
+ /** AWS validates the skips after accepting the call, and reports a bad one as the failure. */
+ private static void assertSkipFails(RollbackFailedStack stack, String skip, String reason) {
+ cfn("ContinueUpdateRollback", stack.name())
+ .formParam("ResourcesToSkip.member.1", skip)
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+ CfnStackWaits.StackState state = CfnStackWaits.awaitTerminal(stack.name());
+ assertEquals("UPDATE_ROLLBACK_FAILED", state.status(), state.reason());
+ assertEquals(reason, state.reason());
+
+ List<Map<String, String>> events = events(stack.id());
+ Map<String, String> failed = events.get(0);
+ Map<String, String> started = events.get(1);
+ assertEquals(stack.name(), failed.get("LogicalResourceId"), events.toString());
+ assertEquals("UPDATE_ROLLBACK_FAILED", failed.get("ResourceStatus"), events.toString());
+ assertEquals(reason, failed.get("ResourceStatusReason"), events.toString());
+ assertEquals("UPDATE_ROLLBACK_IN_PROGRESS", started.get("ResourceStatus"), events.toString());
+ assertEquals(USER_INITIATED, started.get("ResourceStatusReason"), events.toString());
+ }
+
+ private static void assertContinueRefused(String stackName) {
+ cfn("ContinueUpdateRollback", stackName)
+ .when()
+ .post("/")
+ .then()
+ .statusCode(400)
+ .body(containsString("<Code>ValidationError</Code>"))
+ .body(containsString(
+ "<Message>RollbackUpdatedStack cannot be called from current stack status</Message>"));
+ }
+
+ /** A text dashboard titled {@code title}, joined when {@code failing} by a resource that cannot provision. */
+ private static String dashboardTemplate(String dashboardName, String title, boolean failing) {
+ String fail = !failing ? "" : """
+ ,
+ "ZFail": {"Type": "AWS::CloudFormation::Stack", "DependsOn": "Dashboard", "Properties": {}}""";
+ return """
+ {
+ "Resources": {
+ "Dashboard": {
+ "Type": "AWS::CloudWatch::Dashboard",
+ "Properties": {
+ "DashboardName": "%s",
+ "DashboardBody": "{\\"widgets\\":[{\\"type\\":\\"text\\",\\"properties\\":{\\"markdown\\":\\"%s\\"}}]}"
+ }
+ }%s
+ }
+ }
+ """.formatted(dashboardName, title, fail);
+ }
+
+ private static Map<String, String> resourceStatuses(String stackName) {
+ String body = cfn("DescribeStackResources", stackName).when().post("/")
+ .then().statusCode(200).extract().asString();
+ return XmlParser.extractPairs(body, "StackResources", "LogicalResourceId", "ResourceStatus");
+ }
+
+ private static List<Map<String, String>> events(String stackId) {
+ return XmlParser.extractGroups(
+ cfn("DescribeStackEvents", stackId).when().post("/").then().statusCode(200).extract().asString(),
+ "member");
+ }
+
+ private static Response getTemplate(String stackName) {
+ return cfn("GetTemplate", stackName).when().post("/");
+ }
+
+ /** Best effort and unchecked, so a failed assertion above is not masked. */
+ private static void cleanUp(RollbackFailedStack stack) {
+ cfn("DeleteStack", stack.name()).when().post("/");
+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + stack.logGroup() + "\"}");
+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + stack.logGroup() + "/stable\"}");
+ logs("DeleteLogGroup", "{\"logGroupName\":\"" + stack.outsideGroup() + "\"}");
+ }
+
+ /** A Query request for {@code action}, naming the stack unless {@code stackName} is null. */
+ private static RequestSpecification cfn(String action, String stackName) {
+ RequestSpecification request = given()
+ .contentType("application/x-www-form-urlencoded")
+ .formParam("Action", action);
+ return stackName == null ? request : request.formParam("StackName", stackName);
+ }
+
+ private static Response logs(String target, String body) {
+ return given()
+ .config(RestAssured.config().encoderConfig(EncoderConfig.encoderConfig()
+ .encodeContentTypeAs("application/x-amz-json-1.1", ContentType.TEXT)))
+ .header("X-Amz-Target", "Logs_20140328." + target)
+ .contentType("application/x-amz-json-1.1")
+ .body(body)
+ .when()
+ .post("/");
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java
index 4b8e74655c..7e2c2ff6d3 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java
@@ -13,6 +13,7 @@ import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.anyMap;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
@@ -338,7 +339,8 @@ class CloudFormationPipesCleanupIntegrationTest {
* can fail. The pipe the rollback exists to preserve is the prior one, so the resource names it
* and the rename cleanup is spent before the delete is attempted. The stack still reports
* UPDATE_ROLLBACK_FAILED naming the resource, the replacement is what stays orphaned, and the
- * next update finds nothing on the resource that would delete the prior pipe.
+ * next update, once ContinueUpdateRollback has skipped the resource, finds nothing on it that
+ * would delete the prior pipe.
*/
@Test
void aRollbackThatCannotDeleteTheReplacementStillKeepsThePriorPipe() {
@@ -393,8 +395,22 @@ class CloudFormationPipesCleanupIntegrationTest {
assertPipe(priorName);
assertPipe(replacementName);
+ // AWS refuses an update over UPDATE_ROLLBACK_FAILED. The pipe's rollback already restored
+ // the prior pipe and only failed to delete the replacement, so skipping it finishes the
+ // rollback and the stack takes updates again.
+ given()
+ .contentType("application/x-www-form-urlencoded")
+ .formParam("Action", "ContinueUpdateRollback")
+ .formParam("StackName", stackName)
+ .formParam("ResourcesToSkip.member.1", "MyPipe")
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200);
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", CfnStackWaits.awaitTerminal(stackName).status());
+
Mockito.doCallRealMethod().when(pipesService).deletePipe(eq(replacementName), anyString());
- updateStack(stackName, renameRollbackTemplate(replacementName, ""));
+ updateStack(stackName, renameRollbackTemplate(priorName, ""));
given()
.contentType("application/json")
@@ -407,8 +423,8 @@ class CloudFormationPipesCleanupIntegrationTest {
verify(pipesService, never()).deletePipe(eq(priorName), anyString());
deleteStack(stackName);
- // The rollback left the replacement orphaned and the resource UPDATE_FAILED, so neither pipe
- // is one DeleteStack removes and this test removes them itself.
+ // The replacement the rollback orphaned is no resource of the stack, so this test removes
+ // it itself, and the prior pipe too should DeleteStack have left it.
given().when().delete("/v1/pipes/" + priorName);
given().when().delete("/v1/pipes/" + replacementName);
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java
index a62d1c1fe1..2fdbeb6b89 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java
@@ -2,13 +2,17 @@ package io.github.hectorvent.floci.services.cloudformation;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.cloudformation.model.Stack;
+import io.github.hectorvent.floci.services.cloudformation.model.StackEvent;
import io.github.hectorvent.floci.services.cloudformation.model.StackResource;
+import io.github.hectorvent.floci.services.cloudformation.model.StackUpdateSnapshot;
import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnDynamicReferences;
import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnResourceDispatcher;
import io.github.hectorvent.floci.services.cloudformation.provisioners.CfnRollback;
@@ -20,13 +24,19 @@ import org.junit.jupiter.api.Test;
import org.mockito.InOrder;
import java.time.Clock;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -40,18 +50,23 @@ import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
- * Covers rollback cleanup when another actor removes a resource after its create succeeded, and
- * which resources a stack delete walks after a failed operation left them in a failed status, and
- * that a stack which finished deleting is no longer updatable.
+ * Covers rollback cleanup when another actor removes a resource after its create succeeded, which
+ * resources a stack delete walks after a failed operation left them in a failed status, what a
+ * failed update rollback keeps so ContinueUpdateRollback can retry it, and that a stack which
+ * finished deleting is no longer updatable.
*/
class CloudFormationServiceRollbackTest {
private static final String ACCOUNT = "000000000000";
private static final String REGION = "us-east-1";
+ private static final String FORCED_ROLLBACK_REASON = "One or more resources were skipped during UpdateRollback."
+ + " These resources were not rolled back and may have drifted from the template."
+ + " Run drift detection (DetectStackDrift) to review their current state before your next update.";
private CfnResourceDispatcher provisioner;
private CloudFormationService service;
@@ -424,6 +439,293 @@ class CloudFormationServiceRollbackTest {
assertNull(resource.getAttributes().get("rollbackSnapshot"));
}
+ @Test
+ void refusesUpdate_afterUpdateRollbackFailed_refusesButStillAllowsAfterUpdateRollbackComplete() {
+ assertTrue(CloudFormationService.refusesUpdate("UPDATE_ROLLBACK_FAILED"));
+ assertFalse(CloudFormationService.refusesUpdate("UPDATE_ROLLBACK_COMPLETE"));
+ }
+
+ /**
+ * The walker's inputs live only for the update that failed. What it needs to run again is kept
+ * on the stack, and must survive the stack's persisted JSON: a ContinueUpdateRollback after a
+ * restart works from the stored copy alone.
+ */
+ @Test
+ void updateRollbackFailed_persistsWhatContinueUpdateRollbackNeeds() throws Exception {
+ Stack stack = new Stack();
+ stack.setStackName("update-rollback-failed-snapshot");
+ stack.setStackId("stack-id");
+ stack.setRegion(REGION);
+ stack.setStatus("UPDATE_IN_PROGRESS");
+ stack.setTemplateBody("failed-template");
+ stack.setOriginalTemplateBody("failed-original");
+
+ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_COMPLETE");
+ StackResource added = resource("Dup", null, "AWS::Logs::LogGroup", "CREATE_FAILED");
+ added.setStatusReason("already exists");
+ stack.getResources().put(logGroup.getLogicalId(), logGroup);
+ stack.getResources().put(added.getLogicalId(), added);
+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(
+ "previous-template", "previous-original", Map.of("P", "1"), Map.of("P", "1"),
+ Map.of("Out", "v"), Map.of(), Map.of(),
+ Map.of("LogGroup", resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "CREATE_COMPLETE")));
+
+ service.rollbackFailedExecution(stack, REGION, false, added, previousState,
+ new LinkedHashSet<>(List.of("LogGroup", "Dup")));
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());
+ assertEquals("failed-template", stack.getTemplateBody());
+
+ Stack restored = persistedCopy(stack);
+ assertEquals(List.of("LogGroup"), restored.getUpdateRollbackResourceIds());
+ assertEquals("previous-template", restored.getUpdateRollbackSnapshot().templateBody());
+ assertEquals("CREATE_COMPLETE",
+ restored.getUpdateRollbackSnapshot().resources().get("LogGroup").getStatus());
+
+ when(provisioner.rollbackUpdate(any(StackResource.class), any())).thenReturn(true);
+ restored.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ service.resumeUpdateRollback(restored, List.of(), false, REGION);
+
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", restored.getStatus());
+ assertEquals("previous-template", restored.getTemplateBody());
+ assertEquals("previous-original", restored.getOriginalTemplateBody());
+ assertEquals("v", restored.getOutputs().get("Out"));
+ assertEquals("CREATE_COMPLETE", restored.getResources().get("LogGroup").getStatus());
+ assertNull(restored.getUpdateRollbackSnapshot());
+ assertNull(restored.getUpdateRollbackResourceIds());
+ }
+
+ /**
+ * A stack stored before the snapshot was kept still has a way out: every UPDATE_FAILED
+ * resource is owed a rollback, and the template it already holds is kept.
+ */
+ @Test
+ void resumeUpdateRollback_withoutAStoredSnapshot_owesEveryUpdateFailedResourceAndKeepsTheTemplate() {
+ Stack stack = new Stack();
+ stack.setStackName("legacy-update-rollback-failed");
+ stack.setStackId("stack-id");
+ stack.setRegion(REGION);
+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ stack.setTemplateBody("current-template");
+
+ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_FAILED");
+ StackResource stable = resource("Stable", "/stable", "AWS::Logs::LogGroup", "UPDATE_COMPLETE");
+ stack.getResources().put(logGroup.getLogicalId(), logGroup);
+ stack.getResources().put(stable.getLogicalId(), stable);
+ when(provisioner.rollbackUpdate(any(StackResource.class), any())).thenReturn(true);
+
+ service.resumeUpdateRollback(stack, List.of(), false, REGION);
+
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());
+ assertEquals("current-template", stack.getTemplateBody());
+ assertEquals("UPDATE_COMPLETE", logGroup.getStatus());
+ assertEquals("UPDATE_COMPLETE", stable.getStatus());
+ verify(provisioner).rollbackUpdate(eq(logGroup), any());
+ verify(provisioner, never()).rollbackUpdate(eq(stable), any());
+ }
+
+ /**
+ * A skip naming a resource of a nested stack Floci holds is one AWS would act on and Floci does
+ * not support: the rollback fails again with Floci's reason, touching no resource and keeping
+ * what a later retry needs.
+ */
+ @Test
+ void resumeUpdateRollback_skippingANestedStackResource_failsAgainWithoutTouchingAResource() {
+ Stack stack = new Stack();
+ stack.setStackName("nested-skip");
+ stack.setStackId("stack-id");
+ stack.setRegion(REGION);
+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ StackResource nested = resource("Nested", "nested-stack-id", "AWS::CloudFormation::Stack", "UPDATE_FAILED");
+ stack.getResources().put(nested.getLogicalId(), nested);
+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(
+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),
+ Map.of("Nested", resource("Nested", "nested-stack-id", "AWS::CloudFormation::Stack",
+ "CREATE_COMPLETE")));
+ stack.setUpdateRollbackSnapshot(previousState);
+ stack.setUpdateRollbackResourceIds(List.of("Nested"));
+
+ service.resumeUpdateRollback(stack, List.of("Nested.Child"), false, REGION);
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());
+ assertEquals("Skipping a resource of a nested stack is not supported by Floci: Nested.Child",
+ stack.getStatusReason());
+ assertEquals("UPDATE_FAILED", nested.getStatus());
+ assertEquals(previousState, stack.getUpdateRollbackSnapshot());
+ assertEquals(List.of("Nested"), stack.getUpdateRollbackResourceIds());
+ verify(provisioner, never()).rollbackUpdate(any(StackResource.class), any());
+ }
+
+ /**
+ * A resume runs the restore of a resource still owed one again, through the same provisioner
+ * hook the first rollback used: here one whose first restore threw.
+ */
+ @Test
+ void resumeUpdateRollback_afterARestoreThatThrew_runsTheRestoreAgain() {
+ Stack stack = new Stack();
+ stack.setStackName("retry-thrown-restore");
+ stack.setStackId("stack-id");
+ stack.setRegion(REGION);
+ stack.setStatus("UPDATE_IN_PROGRESS");
+ StackResource dashboard = resource("Dashboard", "ops", "AWS::CloudWatch::Dashboard", "UPDATE_COMPLETE");
+ StackResource added = resource("ZFail", null, "AWS::CloudFormation::Stack", "CREATE_FAILED");
+ stack.getResources().put(dashboard.getLogicalId(), dashboard);
+ stack.getResources().put(added.getLogicalId(), added);
+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(
+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),
+ Map.of("Dashboard", resource("Dashboard", "ops", "AWS::CloudWatch::Dashboard", "CREATE_COMPLETE")));
+ when(provisioner.rollbackUpdate(eq(dashboard), any()))
+ .thenThrow(new IllegalStateException("simulated restore failure"))
+ .thenReturn(true);
+
+ service.rollbackFailedExecution(stack, REGION, false, added, previousState,
+ new LinkedHashSet<>(List.of("Dashboard", "ZFail")));
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());
+ assertEquals("UPDATE_FAILED", dashboard.getStatus());
+
+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ service.resumeUpdateRollback(stack, List.of(), false, REGION);
+
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());
+ assertEquals("CREATE_COMPLETE", dashboard.getStatus());
+ verify(provisioner, times(2)).rollbackUpdate(eq(dashboard), any());
+ }
+
+ /**
+ * The resume runs on the operation executor, whose future nobody reads. An unexpected failure
+ * must still settle the stack in UPDATE_ROLLBACK_FAILED, keeping what a retry needs, instead
+ * of stranding it in UPDATE_ROLLBACK_IN_PROGRESS, where ContinueUpdateRollback refuses it.
+ * A resource the failed pass already rolled back is no longer owed, so the retry does not roll it back again.
+ */
+ @Test
+ void resumeUpdateRollback_whenTheBodyThrows_settlesInUpdateRollbackFailedAndCanBeContinued()
+ throws Exception {
+ FailingClock clock = new FailingClock();
+ CloudFormationService resumeService = serviceReading(clock);
+ resumeService.createChangeSet("resume-throws", "create", "CREATE",
+ "{\"Resources\":{\"LogGroup\":{\"Type\":\"AWS::Logs::LogGroup\"}}}", null,
+ Map.of(), List.of(), Map.of(), REGION, ACCOUNT);
+ Stack stack = resumeService.describeStacks("resume-throws", REGION, ACCOUNT).getFirst();
+ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_FAILED");
+ stack.getResources().put(logGroup.getLogicalId(), logGroup);
+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(
+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),
+ Map.of("LogGroup", resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "CREATE_COMPLETE")));
+ stack.setUpdateRollbackSnapshot(previousState);
+ stack.setUpdateRollbackResourceIds(List.of("LogGroup"));
+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ when(provisioner.rollbackUpdate(eq(logGroup), any())).thenReturn(true, false);
+ clock.failNextRead("simulated clock failure");
+
+ resumeService.resumeUpdateRollback(stack, List.of(), false, REGION);
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());
+ assertEquals("simulated clock failure", stack.getStatusReason());
+ StackEvent last = stack.getEvents().getLast();
+ assertEquals("resume-throws", last.getLogicalResourceId());
+ assertEquals("AWS::CloudFormation::Stack", last.getResourceType());
+ assertEquals("UPDATE_ROLLBACK_FAILED", last.getResourceStatus());
+ assertEquals("simulated clock failure", last.getResourceStatusReason());
+ assertEquals(previousState, stack.getUpdateRollbackSnapshot());
+ assertEquals(List.of(), stack.getUpdateRollbackResourceIds());
+
+ resumeService.continueUpdateRollback("resume-throws", List.of(), false, REGION).get(10, TimeUnit.SECONDS);
+
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());
+ assertNull(stack.getUpdateRollbackSnapshot());
+ verify(provisioner).rollbackUpdate(eq(logGroup), any());
+ }
+
+ /**
+ * A skip already reported is no longer owed a rollback when an unexpected failure settles the
+ * stack, or a later retry would roll back a resource the caller skipped.
+ */
+ @Test
+ void resumeUpdateRollback_whenTheBodyThrowsAfterASkip_noLongerOwesTheSkippedResource() throws Exception {
+ FailingClock clock = new FailingClock();
+ CloudFormationService resumeService = serviceReading(clock);
+ resumeService.createChangeSet("resume-throws-after-skip", "create", "CREATE",
+ "{\"Resources\":{\"LogGroup\":{\"Type\":\"AWS::Logs::LogGroup\"}}}", null,
+ Map.of(), List.of(), Map.of(), REGION, ACCOUNT);
+ Stack stack = resumeService.describeStacks("resume-throws-after-skip", REGION, ACCOUNT).getFirst();
+ StackResource skipped = resource("Skipped", "/skipped", "AWS::Logs::LogGroup", "UPDATE_FAILED");
+ stack.getResources().put(skipped.getLogicalId(), skipped);
+ StackUpdateSnapshot previousState = new StackUpdateSnapshot(
+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of(), Map.of(), Map.of(),
+ Map.of("Skipped", resource("Skipped", "/skipped", "AWS::Logs::LogGroup", "CREATE_COMPLETE")));
+ stack.setUpdateRollbackSnapshot(previousState);
+ stack.setUpdateRollbackResourceIds(List.of("Skipped"));
+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ clock.failNextRead("simulated clock failure");
+
+ resumeService.resumeUpdateRollback(stack, List.of("Skipped"), false, REGION);
+
+ assertEquals("UPDATE_ROLLBACK_FAILED", stack.getStatus());
+ assertEquals("simulated clock failure", stack.getStatusReason());
+ assertEquals("UPDATE_COMPLETE", skipped.getStatus());
+ assertEquals("Resource skipped during UpdateRollback", skipped.getStatusReason());
+ assertEquals(previousState, stack.getUpdateRollbackSnapshot());
+ assertEquals(List.of(), stack.getUpdateRollbackResourceIds());
+
+ resumeService.continueUpdateRollback("resume-throws-after-skip", List.of(), false, REGION)
+ .get(10, TimeUnit.SECONDS);
+
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());
+ assertEquals("UPDATE_COMPLETE", skipped.getStatus());
+ verify(provisioner, never()).rollbackUpdate(any(StackResource.class), any());
+ }
+
+ /**
+ * ForceRollback skips every UPDATE_FAILED resource, reporting each as a ResourcesToSkip skip
+ * is, and a rollback it completes says in the stack's reason that resources were skipped.
+ */
+ @Test
+ void resumeUpdateRollback_forced_skipsEveryUpdateFailedResourceAndExplainsTheCompletion() {
+ Stack stack = new Stack();
+ stack.setStackName("forced-rollback");
+ stack.setStackId("stack-id");
+ stack.setRegion(REGION);
+ stack.setStatus("UPDATE_ROLLBACK_IN_PROGRESS");
+ stack.setTemplateBody("failed-template");
+ StackResource role = resource("Role", "role", "AWS::IAM::Role", "UPDATE_FAILED");
+ StackResource stable = resource("Stable", "/stable", "AWS::Logs::LogGroup", "CREATE_COMPLETE");
+ StackResource logGroup = resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "UPDATE_FAILED");
+ for (StackResource resource : new StackResource[] {role, stable, logGroup}) {
+ stack.getResources().put(resource.getLogicalId(), resource);
+ }
+ stack.setUpdateRollbackSnapshot(new StackUpdateSnapshot(
+ "previous-template", "previous-original", Map.of(), Map.of(), Map.of("Out", "v"), Map.of(),
+ Map.of(), Map.of(
+ "Role", resource("Role", "role", "AWS::IAM::Role", "CREATE_COMPLETE"),
+ "Stable", resource("Stable", "/stable", "AWS::Logs::LogGroup", "CREATE_COMPLETE"),
+ "LogGroup", resource("LogGroup", "/lg", "AWS::Logs::LogGroup", "CREATE_COMPLETE"))));
+ stack.setUpdateRollbackResourceIds(List.of("Role", "LogGroup"));
+
+ service.resumeUpdateRollback(stack, List.of(), true, REGION);
+
+ for (StackResource skipped : List.of(role, logGroup)) {
+ assertEquals("UPDATE_COMPLETE", skipped.getStatus(), skipped.getLogicalId());
+ assertEquals("Resource skipped during UpdateRollback", skipped.getStatusReason(), skipped.getLogicalId());
+ }
+ verify(provisioner, never()).rollbackUpdate(any(StackResource.class), any());
+ assertEquals(List.of(
+ "Role UPDATE_COMPLETE Resource skipped during UpdateRollback",
+ "LogGroup UPDATE_COMPLETE Resource skipped during UpdateRollback",
+ "forced-rollback UPDATE_ROLLBACK_COMPLETE " + FORCED_ROLLBACK_REASON),
+ stack.getEvents().stream()
+ .map(event -> event.getLogicalResourceId() + " " + event.getResourceStatus() + " "
+ + event.getResourceStatusReason())
+ .toList());
+ assertEquals("UPDATE_ROLLBACK_COMPLETE", stack.getStatus());
+ assertEquals(FORCED_ROLLBACK_REASON, stack.getStatusReason());
+ assertEquals(FORCED_ROLLBACK_REASON, stack.getEvents().getLast().getResourceStatusReason());
+ assertEquals("previous-template", stack.getTemplateBody());
+ assertEquals("v", stack.getOutputs().get("Out"));
+ assertNull(stack.getUpdateRollbackSnapshot());
+ assertNull(stack.getUpdateRollbackResourceIds());
+ }
+
private static StackResource resource(String logicalId, String physicalId, String resourceType, String status) {
StackResource resource = new StackResource();
resource.setLogicalId(logicalId);
@@ -433,6 +735,53 @@ class CloudFormationServiceRollbackTest {
return resource;
}
+ /** A service over the shared provisioner that reads the time from {@code clock}. */
+ private CloudFormationService serviceReading(Clock clock) {
+ RegionResolver regionResolver = mock(RegionResolver.class);
+ when(regionResolver.getAccountId()).thenReturn(ACCOUNT);
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.defaultAccountId()).thenReturn(ACCOUNT);
+ return new CloudFormationService(provisioner, mock(S3Service.class), mock(SsmService.class),
+ mock(CfnDynamicReferences.class), new ObjectMapper(), config, regionResolver, clock,
+ new InMemoryStorageFactory());
+ }
+
+ /** The stack as {@code PersistentStorage} writes it and reads it back. */
+ private static Stack persistedCopy(Stack stack) throws Exception {
+ ObjectMapper mapper = new ObjectMapper()
+ .registerModule(new JavaTimeModule())
+ .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
+ return mapper.readValue(mapper.writeValueAsString(stack), Stack.class);
+ }
+
+ /** The system clock, except that the read after {@link #failNextRead} throws. */
+ private static final class FailingClock extends Clock {
+ private final AtomicReference<String> failure = new AtomicReference<>();
+
+ void failNextRead(String message) {
+ failure.set(message);
+ }
+
+ @Override
+ public ZoneId getZone() {
+ return ZoneOffset.UTC;
+ }
+
+ @Override
+ public Clock withZone(ZoneId zone) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Instant instant() {
+ String message = failure.getAndSet(null);
+ if (message != null) {
+ throw new IllegalStateException(message);
+ }
+ return Instant.now();
+ }
+ }
+
private static final class InMemoryStorageFactory extends StorageFactory {
private InMemoryStorageFactory() {
super(null, null);
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java
index 2ee80e31e2..d719840692 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java
@@ -19,7 +19,6 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
-import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@@ -27,7 +26,6 @@ import java.util.Map;
import static io.github.hectorvent.floci.services.cognito.CognitoRestAssuredUtils.cognitoAction;
import static io.github.hectorvent.floci.services.cognito.CognitoRestAssuredUtils.cognitoJson;
import static io.restassured.RestAssured.given;
-import static org.awaitility.Awaitility.await;
import static org.hamcrest.Matchers.equalTo;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
@@ -155,6 +153,10 @@ class CognitoResourceServerCfnIntegrationTest {
}
clearInvocations(cognitoService);
+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE");
+ assertServerGone(failedPool, IDENTIFIER);
+ assertServer(originalPool, IDENTIFIER, "Original API", "read");
+
updateStack(template(finalPool, IDENTIFIER, "Final API", "admin"), "UPDATE_COMPLETE");
assertEquals(IDENTIFIER, output("ServerRef"));
@@ -433,10 +435,7 @@ class CognitoResourceServerCfnIntegrationTest {
deleteServer(pool);
createServer(pool, "Out of band API", "admin");
- ObjectNode nextAttempt = template(pool, IDENTIFIER, "Next attempt", "write");
- addFailingDependentResource(nextAttempt);
- nextAttempt.withObject("/Resources/Broken/Properties").put("ClientName", "second-broken");
- updateStack(nextAttempt, "UPDATE_ROLLBACK_COMPLETE");
+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE");
assertServer(pool, IDENTIFIER, "Original API", "read");
cloudFormation("DeleteStack", null);
CfnStackWaits.awaitStackDeleted(stack);
@@ -459,6 +458,7 @@ class CognitoResourceServerCfnIntegrationTest {
} finally {
doCallRealMethod().when(cognitoService).deleteResourceServer(replacementPool, IDENTIFIER);
}
+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE", "Server");
deleteServer(replacementPool);
doAnswer(invocation -> {
cognitoService.describeResourceServer(replacementPool, IDENTIFIER);
@@ -495,6 +495,7 @@ class CognitoResourceServerCfnIntegrationTest {
} finally {
doCallRealMethod().when(cognitoService).deleteResourceServer(historicalPool, IDENTIFIER);
}
+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE", "Server");
assertServer(historicalPool, IDENTIFIER, "Failed replacement", "write");
deleteServer(originalPool);
clearInvocations(cognitoService);
@@ -521,7 +522,7 @@ class CognitoResourceServerCfnIntegrationTest {
}
@Test
- void aPendingRollbackSurvivesFailedUpdatesUntilItCanBeRestored() throws Exception {
+ void aPendingRollbackSurvivesFailedRetriesUntilItCanBeRestored() throws Exception {
String pool = createPool();
createStack(template(pool, IDENTIFIER, "Original API", "read"));
doThrow(new AwsException("InternalErrorException", "temporary restore failure", 500))
@@ -533,18 +534,12 @@ class CognitoResourceServerCfnIntegrationTest {
updateStack(firstAttempt, "UPDATE_ROLLBACK_FAILED");
assertServer(pool, IDENTIFIER, "First attempt", "write");
- ObjectNode nextAttempt = template(pool, IDENTIFIER, "Next attempt", "admin");
- addFailingDependentResource(nextAttempt);
- nextAttempt.withObject("/Resources/Broken/Properties").put("ClientName", "broken-next");
- updateStack(nextAttempt, "UPDATE_ROLLBACK_FAILED");
+ continueUpdateRollback("UPDATE_ROLLBACK_FAILED");
assertServer(pool, IDENTIFIER, "First attempt", "write");
doCallRealMethod().when(cognitoService)
.updateResourceServer(eq(pool), eq(IDENTIFIER), eq("Original API"), any());
- ObjectNode recoveryAttempt = template(pool, IDENTIFIER, "Recovery attempt", "write");
- addFailingDependentResource(recoveryAttempt);
- recoveryAttempt.withObject("/Resources/Broken/Properties").put("ClientName", "broken-recovery");
- updateStack(recoveryAttempt, "UPDATE_ROLLBACK_COMPLETE");
+ continueUpdateRollback("UPDATE_ROLLBACK_COMPLETE");
assertServer(pool, IDENTIFIER, "Original API", "read");
assertEquals(IDENTIFIER, output("ServerRef"));
} finally {
@@ -554,7 +549,7 @@ class CognitoResourceServerCfnIntegrationTest {
}
@Test
- void skippingAPendingRollbackPreservesRestorationAndBlocksAnotherUpdate() throws Exception {
+ void anUpdateThatLeavesAPendingRollbackAloneIsRefused() throws Exception {
String pool = createPool();
createStack(template(pool, IDENTIFIER, "Original API", "read"));
doThrow(new AwsException("InternalErrorException", "temporary restore failure", 500))
@@ -569,23 +564,16 @@ class CognitoResourceServerCfnIntegrationTest {
ObjectNode outputsOnly = firstAttempt.deepCopy();
outputsOnly.withObject("/Resources").remove("Broken");
outputsOnly.withObject("/Outputs").putObject("Marker").put("Value", "unchanged-resource");
- cloudFormation("UpdateStack", outputsOnly);
- await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
- assertEquals("UPDATE_COMPLETE_CLEANUP_IN_PROGRESS", stackStatus(),
- "Skipping the resource must not discard its pending restoration"));
- assertServer(pool, IDENTIFIER, "First attempt", "write");
- assertEquals(IDENTIFIER, output("ServerRef"));
-
- ObjectNode nextAttempt = template(pool, IDENTIFIER, "Next attempt", "admin");
String error = given().contentType("application/x-www-form-urlencoded")
.header("Authorization", CFN_AUTH).formParam("Action", "UpdateStack")
- .formParam("StackName", stack).formParam("TemplateBody", nextAttempt.toString())
+ .formParam("StackName", stack).formParam("TemplateBody", outputsOnly.toString())
.post("/").then().statusCode(400).extract().asString();
assertEquals("ValidationError", XmlParser.extractFirst(error, "Code", null));
assertTrue(XmlParser.extractFirst(error, "Message", "")
- .contains("UPDATE_COMPLETE_CLEANUP_IN_PROGRESS"));
- assertEquals("UPDATE_COMPLETE_CLEANUP_IN_PROGRESS", stackStatus());
+ .contains("UPDATE_ROLLBACK_FAILED"));
+ assertEquals("UPDATE_ROLLBACK_FAILED", stackStatus());
assertServer(pool, IDENTIFIER, "First attempt", "write");
+ assertEquals(IDENTIFIER, output("ServerRef"));
cloudFormation("DeleteStack", null);
CfnStackWaits.awaitStackDeleted(stack);
@@ -729,6 +717,19 @@ class CognitoResourceServerCfnIntegrationTest {
assertEquals(expectedStatus, CfnStackWaits.awaitTerminal(stack).status());
}
+ private void continueUpdateRollback(String expectedStatus, String... resourcesToSkip) {
+ RequestSpecification request = given()
+ .contentType("application/x-www-form-urlencoded")
+ .header("Authorization", CFN_AUTH)
+ .formParam("Action", "ContinueUpdateRollback")
+ .formParam("StackName", stack);
+ for (int i = 0; i < resourcesToSkip.length; i++) {
+ request.formParam("ResourcesToSkip.member." + (i + 1), resourcesToSkip[i]);
+ }
+ request.when().post("/").then().statusCode(200);
+ assertEquals(expectedStatus, CfnStackWaits.awaitTerminal(stack).status());
+ }
+
private void cloudFormation(String action, ObjectNode template) {
RequestSpecification request = given()
.contentType("application/x-www-form-urlencoded")
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java
index b06ae36251..d181a8bb29 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java
@@ -635,8 +635,8 @@ class EcsCfnProvisionerTest {
assertThrows(AwsException.class, () -> provisioner.rollbackUpdate(r));
assertEquals(TASK_DEF_ARN, r.getPhysicalId());
- // A rollback with no replacement of its own deletes nothing and keeps the orphan owed.
- assertFalse(provisioner.rollbackUpdate(r));
+ // A retried rollback (ContinueUpdateRollback) re-attempts the failed delete; failing again, the orphan stays owed.
+ assertThrows(AwsException.class, () -> provisioner.rollbackUpdate(r));
assertTrue(provisioner.hasReplacementUpdate(r));
// The next update registers revision 5: revision 3 (the prior) joins the orphan on the list
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java
index a38d16c372..14c9faae8b 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java
@@ -447,12 +447,9 @@ class ElbV2CfnProvisionerTest {
assertTrue(provisioner.hasReplacementUpdate(r));
assertEquals(renamed.getLoadBalancerArn(), provisioner.updateCleanupPhysicalId(r));
org.mockito.Mockito.reset(elb);
- // A second rollback has no replacement of this update to undo: it deletes nothing and keeps
- // the orphan owed (the load balancer itself rolls back as complete, nothing being modified).
- provisioner.rollbackUpdate(r);
- verify(elb, never()).deleteLoadBalancer(anyString(), anyString());
- assertTrue(provisioner.hasReplacementUpdate(r));
- assertEquals(new UpdateCleanupResult(true, true, renamed.getLoadBalancerArn(), 0, null), provisioner.completeUpdate(r));
+ // A retried rollback (ContinueUpdateRollback) re-attempts the delete that failed; succeeding,
+ // it owes nothing more and never touches the restored prior.
+ assertTrue(provisioner.rollbackUpdate(r));
verify(elb).deleteLoadBalancer(REGION, renamed.getLoadBalancerArn());
verify(elb, never()).deleteLoadBalancer(REGION, LB_ARN);
assertFalse(provisioner.hasReplacementUpdate(r));
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java
index 648029b6c4..0db5388a26 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java
@@ -254,6 +254,62 @@ class ReplacementCleanupTest {
assertEquals(List.of("subnet-2"), deleted);
}
+ /** A retried rollback (ContinueUpdateRollback) re-attempts the delete that failed, then has nothing left to do. */
+ @Test
+ void rollback_afterTheReplacementDeleteFailed_retriesThatDeleteOnTheNextRollback() {
+ StackResource r = failedReplacementRollback();
+ List<String> deleted = new ArrayList<>();
+
+ assertTrue(ReplacementCleanup.rollback(r, (type, id, region) -> deleted.add(type + ":" + id + ":" + region)));
+
+ assertEquals(List.of("AWS::EC2::Subnet:subnet-2:us-east-1"), deleted);
+ assertEquals("subnet-1", r.getPhysicalId());
+ assertFalse(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));
+ assertFalse(ReplacementCleanup.hasReplacement(r));
+ List<String> deletedAgain = new ArrayList<>();
+ assertFalse(ReplacementCleanup.rollback(r, (type, id, region) -> deletedAgain.add(id)));
+ assertEquals(List.of(), deletedAgain);
+ }
+
+ @Test
+ void rollback_retryThatFailsAgain_keepsTheReplacementOwed() {
+ StackResource r = failedReplacementRollback();
+
+ assertThrows(IllegalStateException.class, () -> ReplacementCleanup.rollback(r, (type, id, region) -> {
+ throw new IllegalStateException("DependencyViolation");
+ }));
+
+ assertTrue(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));
+ List<String> deleted = new ArrayList<>();
+ assertTrue(ReplacementCleanup.rollback(r, (type, id, region) -> deleted.add(id)));
+ assertEquals(List.of("subnet-2"), deleted);
+ }
+
+ /** A new provision supersedes the failed rollback; the replacement stays owed to the next cleanup instead. */
+ @Test
+ void record_afterAFailedRollbackDelete_doesNotRetryThatDeleteOnTheNextRollback() {
+ StackResource r = failedReplacementRollback();
+ List<String> deleted = new ArrayList<>();
+
+ ReplacementCleanup.record(r, update("subnet-1"), Map.of("VpcId", "vpc-old"));
+
+ assertFalse(ReplacementCleanup.rollback(r, (type, id, region) -> deleted.add(id)));
+ assertEquals(List.of(), deleted);
+ assertTrue(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));
+ }
+
+ /** Replaces subnet-1 with subnet-2, then rolls back with a delete of subnet-2 that fails. */
+ private static StackResource failedReplacementRollback() {
+ StackResource r = resource("subnet-2");
+ ReplacementCleanup.record(r, update("subnet-1"), Map.of("VpcId", "vpc-old"));
+ assertThrows(IllegalStateException.class, () -> ReplacementCleanup.rollback(r, (type, id, region) -> {
+ throw new IllegalStateException("DependencyViolation");
+ }));
+ assertEquals("subnet-1", r.getPhysicalId());
+ assertTrue(ReplacementCleanup.owedPhysicalIds(r).contains("subnet-2"));
+ return r;
+ }
+
private static ProvisionContext update(String priorPhysicalId) {
return new ProvisionContext(null, "us-east-1", "000000000000", "my-stack", priorPhysicalId);
}