Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(redshift-serverless): execute scheduled actions on cron and at schedules (#6851)

Vũ Thế Huye820414f2257
diff --git a/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java b/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java
index 6a2aad9a5d..c64760048f 100644
--- a/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java
+++ b/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java
@@ -504,6 +504,15 @@ class RedshiftServerlessTest {
.build())).scheduledAction();
assertThat(action.state().toString()).isEqualTo("ACTIVE");
assertThat(action.schedule().cron()).isEqualTo("cron(0 3 * * ? *)");
+ assertThat(action.nextInvocations()).hasSize(1);
+ assertThatThrownBy(() -> client.createScheduledAction(request -> request
+ .scheduledActionName("floci-compat-too-often").namespaceName(namespaceName)
+ .roleArn("arn:aws:iam::000000000000:role/r")
+ .schedule(Schedule.builder().cron("cron(0,30 * * * ? *)").build())
+ .targetAction(TargetAction.builder().createSnapshot(CreateSnapshotScheduleActionParameters
+ .builder().namespaceName(namespaceName).snapshotNamePrefix("nightly").build())
+ .build())))
+ .isInstanceOf(ValidationException.class);
client.deleteScheduledAction(request -> request.scheduledActionName("floci-compat-sched"));
} finally {
deleteBestEffort(client, namespaceName);
diff --git a/docs/services/redshift-serverless.md b/docs/services/redshift-serverless.md
index 59fb4b1cbd..2eaaaf7e26 100644
--- a/docs/services/redshift-serverless.md
+++ b/docs/services/redshift-serverless.md
@@ -62,7 +62,7 @@ For the upstream API shape, see the [Amazon Redshift Serverless API Reference](h
| `ListSnapshotCopyConfigurations` | Page through the snapshot copy configurations, filtered by `namespaceName` |
| `UpdateSnapshotCopyConfiguration` | Change the `snapshotRetentionPeriod` of a snapshot copy configuration |
| `DeleteSnapshotCopyConfiguration` | Remove a snapshot copy configuration and return it |
-| `CreateScheduledAction` | Record a scheduled `createSnapshot` action; it is stored, never run |
+| `CreateScheduledAction` | Create a scheduled `createSnapshot` action that Floci runs on its schedule |
| `GetScheduledAction` | Return a scheduled action by name |
| `ListScheduledActions` | Page through the scheduled actions, filtered by `namespaceName` |
| `UpdateScheduledAction` | Apply the supplied members to a scheduled action; `schedule` and `targetAction` replace the stored ones |
@@ -108,6 +108,8 @@ AWS also models `AccessDeniedException`, `InternalServerException`, and `Throttl
| Variable | Default | Description |
|---|---|---|
| `FLOCI_SERVICES_REDSHIFT_SERVERLESS_ENABLED` | `true` | Enable or disable Redshift Serverless |
+| `FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTIONS_ENABLED` | `true` | Run the background worker that executes scheduled actions |
+| `FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTION_SWEEP_INTERVAL_SECONDS` | `60` | Delay between scheduled action sweeps |
Workgroup containers and proxies use the `floci.services.redshift.*` settings: `image-version`, `proxy-base-port` and `proxy-max-port`, `endpoint-host`, `docker-network` and the proxy connection limits.
@@ -145,4 +147,4 @@ aws redshift-serverless delete-namespace --namespace-name analytics
- **Endpoint access is recorded, not acted on.** An endpoint is stored per account and Region, echoed in the shape of the API model, and dropped on read once its workgroup is gone; a workgroup recreated under the same name is a different workgroup and does not inherit it. `CreateEndpointAccess` creates no VPC endpoint. `address` and `port` are the workgroup's own, so a client can connect; `vpcEndpoint` carries a generated id and one network interface per subnet, without a `vpcId`, and `ListEndpointAccess` with a `vpcId` returns nothing. `vpcSecurityGroupIds` is optional. Endpoint names are 1 to 30 letters, digits or hyphens, start with a letter, and contain no consecutive or trailing hyphens. Security groups are the only member `UpdateEndpointAccess` changes.
- **Custom domain associations are recorded, not acted on.** An association is stored per account and Region and dropped on read once its workgroup is gone; a workgroup recreated under the same name does not inherit it. A workgroup has at most one custom domain association. The certificate expiry time comes from ACM when the certificate exists in the emulator.
- **Snapshot copy configurations are recorded, not acted on.** `CreateSnapshotCopyConfiguration` copies no snapshot. A configuration is stored per account and Region and dropped on read once its namespace is gone; a namespace recreated under the same name does not inherit it. The destination Region must differ from the namespace's, and a namespace has only one configuration at a time (`ConflictException`). Delete the existing configuration before choosing another destination Region.
-- **Scheduled actions are recorded, never run.** An action is stored per account and Region and dropped on read once its namespace is gone; a namespace recreated under the same name does not inherit it. The owning `namespaceName` and the `targetAction.createSnapshot.namespaceName` must each exist but are not required to match, as the API reference does not say they must. `schedule` holds exactly one of `at` and `cron`; a one-off `at` in the future is the only `nextInvocations` reported, because cron expressions are not evaluated. `startTime`, `endTime` and `at` are epoch seconds.
+- **Scheduled actions run.** A background sweep (every `FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTION_SWEEP_INTERVAL_SECONDS`, 60 by default) takes the `createSnapshot` of each enabled action whose `at` or six-field `cron` schedule came due, in UTC and inside the `startTime` to `endTime` window. A cron expression may be wrapped as `cron(...)` and must fire at most once an hour; AWS day-of-week numbering applies (1 is Sunday). The snapshot is named `<snapshotNamePrefix>-yyyy-MM-dd-HH-mm-ss` after the scheduled time (lowercased) and keeps the action's `retentionPeriod` and tags. `nextInvocations` reports the next eligible time for both schedule types, and is empty once none remains. Floci records the invocation checkpoint with a normal atomic write after attempting the snapshot. A failure (an existing snapshot of that name, a missing namespace, a refused role) is logged and skipped while that checkpoint is retained. A crash before the checkpoint write, or before a hybrid flush, can cause the invocation to be retried; the scheduled-time snapshot name and the existing-name check prevent a duplicate snapshot. This is not exactly-once delivery. After downtime only the latest due time runs, and nothing before the action was created, rescheduled or re-enabled is replayed. That is a Floci choice, not a statement about AWS. Persistent and WAL storage commit checkpoints synchronously; hybrid storage persists them on its normal periodic flush. In memory mode actions are lost on restart. With IAM enforcement on, `roleArn` must be a role of the action's account that trusts `scheduler.redshift.amazonaws.com` and allows `redshift-serverless:CreateSnapshot` on the target namespace; otherwise nothing is created. An action is stored per account and Region and dropped on read once its namespace is gone; a namespace recreated under the same name does not inherit it. The owning `namespaceName` and the `targetAction.createSnapshot.namespaceName` must each exist but are not required to match. `startTime`, `endTime` and `at` are epoch seconds. Recovery points, snapshot expiration and cross-Region copy are not part of a scheduled action.
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index b3cad3fec4..6ed7923c09 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -1908,6 +1908,12 @@ public interface EmulatorConfig {
interface RedshiftServerlessServiceConfig {
@WithDefault("true")
boolean enabled();
+
+ @WithDefault("true")
+ boolean scheduledActionsEnabled();
+
+ @WithDefault("60")
+ int scheduledActionSweepIntervalSeconds();
}
interface NeptuneServiceConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java
index 8f9b4a82e5..69765a0e00 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java
@@ -344,7 +344,7 @@ public class RedshiftServerlessAdminJsonHandler {
return Response.ok(response).build();
}
- /** A one-off schedule reports its own time while it is ahead and within its active window on an enabled action; a cron schedule reports none, as Floci never computes them. */
+ /** Reports the next invocation inside the active window of an enabled action, for both schedule types. */
private ObjectNode scheduledActionNode(ScheduledAction action) {
ObjectNode node = objectMapper.createObjectNode();
if (action.getEndTime() != null) {
@@ -352,13 +352,7 @@ public class RedshiftServerlessAdminJsonHandler {
}
node.put("namespaceName", action.getNamespaceName());
ArrayNode invocations = node.putArray("nextInvocations");
- if (action.getScheduleAt() != null
- && !Boolean.FALSE.equals(action.getEnabled())
- && action.getScheduleAt().isAfter(Instant.now())
- && (action.getStartTime() == null || !action.getScheduleAt().isBefore(action.getStartTime()))
- && (action.getEndTime() == null || !action.getScheduleAt().isAfter(action.getEndTime()))) {
- invocations.add(epoch(action.getScheduleAt()));
- }
+ RedshiftServerlessSchedule.next(action, Instant.now()).ifPresent(time -> invocations.add(epoch(time)));
node.put("roleArn", action.getRoleArn());
ObjectNode schedule = node.putObject("schedule");
if (action.getScheduleAt() != null) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessSchedule.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessSchedule.java
new file mode 100644
index 0000000000..d2de256992
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessSchedule.java
@@ -0,0 +1,111 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import com.cronutils.model.Cron;
+import com.cronutils.model.time.ExecutionTime;
+import io.github.hectorvent.floci.core.common.AwsCronDefinitions;
+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.util.Optional;
+
+/** UTC schedule rules of a Serverless scheduled action: one-off {@code at} or six-field AWS {@code cron}. */
+final class RedshiftServerlessSchedule {
+ private static final Duration MINIMUM_INTERVAL = Duration.ofHours(1);
+ private static final Duration SPACING_HORIZON = Duration.ofDays(400);
+ private static final int MAX_SPACING_CHECKS = 1000;
+
+ private RedshiftServerlessSchedule() { }
+
+ /** Rejects a cron that does not parse or fires more often than hourly; an {@code at} schedule always passes. */
+ static void validate(ScheduledAction action) {
+ if (action.getScheduleCron() == null || action.getScheduleCron().isBlank()) {
+ return;
+ }
+ ExecutionTime cron = parse(action.getScheduleCron());
+ Optional<Instant> first = nextCron(cron, Instant.EPOCH);
+ if (first.isEmpty()) {
+ return;
+ }
+ Instant cursor = first.get().minusSeconds(1);
+ Instant limit = cursor.plus(SPACING_HORIZON);
+ Instant previous = null;
+ for (int i = 0; i < MAX_SPACING_CHECKS; i++) {
+ Optional<Instant> next = nextCron(cron, cursor);
+ if (next.isEmpty() || next.get().isAfter(limit)) {
+ return;
+ }
+ if (previous != null && Duration.between(previous, next.get()).compareTo(MINIMUM_INTERVAL) < 0) {
+ throw ServerlessSupport.validation("The cron schedule must be at least one hour apart.");
+ }
+ previous = next.get();
+ cursor = next.get();
+ }
+ }
+
+ /** The first invocation strictly after {@code after} that is enabled and inside the active window. */
+ static Optional<Instant> next(ScheduledAction action, Instant after) {
+ if (Boolean.FALSE.equals(action.getEnabled())) {
+ return Optional.empty();
+ }
+ Instant candidate;
+ if (action.getScheduleAt() != null) {
+ candidate = action.getScheduleAt().isAfter(after) ? action.getScheduleAt() : null;
+ } else {
+ Instant from = action.getStartTime() != null && action.getStartTime().isAfter(after)
+ ? action.getStartTime().minusNanos(1) : after;
+ candidate = nextCron(parse(action.getScheduleCron()), from).orElse(null);
+ }
+ return Optional.ofNullable(candidate).filter(time -> inWindow(action, time));
+ }
+
+ /** The latest invocation in {@code (anchor, now]} that is inside the window; older ones are coalesced away. */
+ static Optional<Instant> latestDue(ScheduledAction action, Instant anchor, Instant now) {
+ if (Boolean.FALSE.equals(action.getEnabled())) {
+ return Optional.empty();
+ }
+ Instant upper = action.getEndTime() != null && action.getEndTime().isBefore(now) ? action.getEndTime() : now;
+ Optional<Instant> latest;
+ if (action.getScheduleAt() != null) {
+ latest = Optional.of(action.getScheduleAt()).filter(time -> !time.isAfter(upper));
+ } else {
+ latest = parse(action.getScheduleCron()).lastExecution(upper.atZone(ZoneOffset.UTC).plusNanos(1))
+ .map(value -> value.toInstant());
+ }
+ return latest.filter(time -> time.isAfter(anchor) && inWindow(action, time));
+ }
+
+ private static boolean inWindow(ScheduledAction action, Instant time) {
+ return (action.getStartTime() == null || !time.isBefore(action.getStartTime()))
+ && (action.getEndTime() == null || !time.isAfter(action.getEndTime()));
+ }
+
+ private static Optional<Instant> nextCron(ExecutionTime cron, Instant after) {
+ return cron.nextExecution(after.atZone(ZoneOffset.UTC)).map(value -> value.toInstant());
+ }
+
+ private static ExecutionTime parse(String expression) {
+ String body = expression.trim();
+ if (body.startsWith("cron")) {
+ body = body.substring(4).trim();
+ }
+ if (body.startsWith("(") && body.endsWith(")")) {
+ body = body.substring(1, body.length() - 1).trim();
+ }
+ String[] fields = body.split("\\s+");
+ if (fields.length != 6) {
+ throw ServerlessSupport.validation("The cron schedule needs six fields.");
+ }
+ if ("?".equals(fields[2]) == "?".equals(fields[4])) {
+ throw ServerlessSupport.validation("Use ? in exactly one of day-of-month and day-of-week.");
+ }
+ try {
+ Cron cron = AwsCronDefinitions.newParser().parse("0 " + String.join(" ", fields));
+ cron.validate();
+ return ExecutionTime.forCron(cron);
+ } catch (IllegalArgumentException e) {
+ throw ServerlessSupport.validation("The cron schedule is invalid: " + e.getMessage());
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizer.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizer.java
new file mode 100644
index 0000000000..781c71a272
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizer.java
@@ -0,0 +1,103 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
+import io.github.hectorvent.floci.core.common.IamConditionContextResolver;
+import io.github.hectorvent.floci.core.common.ServicePrincipals;
+import io.github.hectorvent.floci.services.iam.AssumeRolePolicyEvaluator;
+import io.github.hectorvent.floci.services.iam.IamPolicyEvaluator;
+import io.github.hectorvent.floci.services.iam.IamPolicyEvaluator.Decision;
+import io.github.hectorvent.floci.services.iam.IamService;
+import io.github.hectorvent.floci.services.iam.ScpProvider;
+import io.github.hectorvent.floci.services.iam.model.CallerContext;
+import io.github.hectorvent.floci.services.iam.model.IamRole;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.enterprise.inject.Instance;
+import jakarta.inject.Inject;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Decides whether the execution role of a scheduled action may take its snapshot. The role has to
+ * trust the Redshift scheduler principal and its identity policies, boundary and SCPs have to allow
+ * {@code redshift-serverless:CreateSnapshot} on the target. Only active with IAM enforcement on, as
+ * for every other API call. A role of another account is refused, because the snapshot would
+ * otherwise be taken on the strength of a role the action's account does not own.
+ */
+@ApplicationScoped
+public class RedshiftServerlessScheduledActionAuthorizer {
+
+ static final String CREATE_SNAPSHOT = "redshift-serverless:CreateSnapshot";
+ private static final String SCHEDULER_PRINCIPAL = ServicePrincipals.of("scheduler.redshift");
+
+ private final EmulatorConfig config;
+ private final IamService iamService;
+ private final IamPolicyEvaluator evaluator;
+ private final AssumeRolePolicyEvaluator trustEvaluator;
+ private final Instance<ScpProvider> scpProvider;
+
+ @Inject
+ public RedshiftServerlessScheduledActionAuthorizer(EmulatorConfig config, IamService iamService,
+ IamPolicyEvaluator evaluator,
+ AssumeRolePolicyEvaluator trustEvaluator,
+ Instance<ScpProvider> scpProvider) {
+ this.config = config;
+ this.iamService = iamService;
+ this.evaluator = evaluator;
+ this.trustEvaluator = trustEvaluator;
+ this.scpProvider = scpProvider;
+ }
+
+ /**
+ * Returns why {@code roleArn} may not create a snapshot of {@code targetArn}, or nothing when it may.
+ *
+ * @param accountId the account that owns the action
+ */
+ public Optional<String> refusal(String roleArn, String accountId, String targetArn) {
+ if (!config.services().iam().enforcementEnabled()) {
+ return Optional.empty();
+ }
+ AwsArnUtils.Arn parsed;
+ try {
+ parsed = AwsArnUtils.parse(roleArn);
+ } catch (IllegalArgumentException e) {
+ return Optional.of("The execution role " + roleArn + " is not a valid ARN.");
+ }
+ if (!"iam".equals(parsed.service()) || !parsed.resource().startsWith("role/")) {
+ return Optional.of("The execution role " + roleArn + " is not an IAM role.");
+ }
+ if (!accountId.equals(parsed.accountId())) {
+ return Optional.of("The execution role " + roleArn + " belongs to another account.");
+ }
+ String roleName = parsed.resource().substring(parsed.resource().lastIndexOf('/') + 1);
+ Optional<IamRole> role = iamService.findRole(accountId, roleName)
+ .filter(found -> IamService.roleArnMatches(roleArn, found));
+ if (role.isEmpty()) {
+ return Optional.of("The execution role " + roleArn + " does not exist.");
+ }
+ if (!trustEvaluator.allowsService(role.get().getAssumeRolePolicyDocument(), SCHEDULER_PRINCIPAL,
+ null, accountId)) {
+ return Optional.of("The execution role " + roleArn + " does not trust " + SCHEDULER_PRINCIPAL + ".");
+ }
+ CallerContext caller = iamService.resolvePrincipalContext(role.get().getArn());
+ List<List<String>> scpLevels = scpProvider.isResolvable()
+ ? scpProvider.get().effectiveScpLevels(accountId) : null;
+ if (scpLevels != null) {
+ caller = caller.withScpLevels(scpLevels);
+ }
+ AwsArnUtils.Arn target = AwsArnUtils.isArn(targetArn) ? AwsArnUtils.parse(targetArn) : null;
+ String targetRegion = target != null ? target.region() : null;
+ String targetAccount = target != null ? target.accountId() : accountId;
+ Map<String, List<String>> context = IamConditionContextResolver.withGlobalContext(
+ Map.of("aws:PrincipalArn", List.of(role.get().getArn())),
+ targetArn, targetRegion, accountId, targetAccount);
+ Decision decision = evaluator.evaluate(caller, List.of(), CREATE_SNAPSHOT, targetArn, context);
+ if (decision != Decision.ALLOW) {
+ return Optional.of("The execution role " + roleArn + " is not authorized to perform "
+ + CREATE_SNAPSHOT + " on " + targetArn + ".");
+ }
+ return Optional.empty();
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionScheduler.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionScheduler.java
new file mode 100644
index 0000000000..5bccc5bfa5
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionScheduler.java
@@ -0,0 +1,103 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.Resettable;
+import io.quarkus.runtime.ShutdownEvent;
+import io.quarkus.runtime.StartupEvent;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.enterprise.event.Observes;
+import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+
+import java.time.Instant;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+/** Drives Serverless scheduled actions outside AWS request threads, one sweep at a time. */
+@ApplicationScoped
+public class RedshiftServerlessScheduledActionScheduler implements Resettable {
+ private static final Logger LOG = Logger.getLogger(RedshiftServerlessScheduledActionScheduler.class);
+ private final RedshiftServerlessScheduledActionService service;
+ private final ScheduledExecutorService executor;
+ private final long intervalSeconds;
+ private final ReentrantLock sweepLock = new ReentrantLock();
+ private volatile boolean paused;
+
+ @Inject
+ public RedshiftServerlessScheduledActionScheduler(RedshiftServerlessScheduledActionService service,
+ EmulatorConfig config) {
+ EmulatorConfig.RedshiftServerlessServiceConfig serverless = config.services().redshiftServerless();
+ this.service = service;
+ this.intervalSeconds = Math.max(1, serverless.scheduledActionSweepIntervalSeconds());
+ this.executor = serverless.enabled() && serverless.scheduledActionsEnabled()
+ ? Executors.newSingleThreadScheduledExecutor(task -> {
+ Thread thread = new Thread(task, "redshift-serverless-scheduled-actions");
+ thread.setDaemon(true);
+ return thread;
+ }) : null;
+ }
+
+ void onStart(@Observes StartupEvent ignored) {
+ if (executor != null) {
+ executor.scheduleWithFixedDelay(this::tickSafely, intervalSeconds, intervalSeconds, TimeUnit.SECONDS);
+ }
+ }
+
+ void onStop(@Observes ShutdownEvent ignored) {
+ if (executor != null) {
+ executor.shutdownNow();
+ try {
+ if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {
+ LOG.warn("The Redshift Serverless scheduled action worker is still finishing a snapshot during shutdown");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.debugv(e, "Interrupted while stopping the Redshift Serverless scheduled action worker");
+ }
+ }
+ }
+
+ /** Whether a background worker exists, which the two configuration switches decide. */
+ boolean active() {
+ return executor != null;
+ }
+
+ public void tick(Instant now) {
+ sweepLock.lock();
+ try {
+ if (!paused) {
+ service.runScheduledActions(now);
+ }
+ } finally {
+ sweepLock.unlock();
+ }
+ }
+
+ /** Drains a running sweep so it cannot write a claim back into storage that reset is about to clear. */
+ @Override
+ public void beforeReset() {
+ paused = true;
+ sweepLock.lock();
+ sweepLock.unlock();
+ }
+
+ @Override
+ public void clear() {
+ // Claims live with the actions in RedshiftServerlessScheduledActionService storage, which reset wipes.
+ }
+
+ @Override
+ public void afterReset() {
+ paused = false;
+ }
+
+ private void tickSafely() {
+ try {
+ tick(Instant.now());
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "The Redshift Serverless scheduled action sweep failed");
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java
index bf96877a2b..c484821008 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java
@@ -3,23 +3,36 @@ package io.github.hectorvent.floci.services.redshiftserverless;
import com.fasterxml.jackson.core.type.TypeReference;
import io.github.hectorvent.floci.core.common.PaginatedResult;
import io.github.hectorvent.floci.core.common.Pagination;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.RequestScopes;
import io.github.hectorvent.floci.core.common.Resettable;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend.AccountEntry;
import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.redshiftserverless.model.Namespace;
import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+import java.time.Clock;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
import java.util.Comparator;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Objects;
+import java.util.Optional;
import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Supplier;
/**
- * Redshift Serverless scheduled actions. They are stored and echoed, not run: Floci never takes
- * the snapshot on the schedule, and a cron schedule reports no upcoming invocations.
+ * Redshift Serverless scheduled actions. A sweep ({@link #runScheduledActions}) takes the snapshot of
+ * every action whose schedule came due since it last looked.
*
* <p>Required members, name patterns and lengths come from the AWS model through
* {@link RedshiftServerlessRequestShapes}; what the model cannot express (the schedule union, the time
@@ -27,44 +40,78 @@ import java.util.UUID;
*
* <p>An action belongs to a namespace but lives in its own store, so deleting the namespace needs
* no hook here. Each read checks that the namespace is still there and drops the action otherwise.
+ *
+ * <p>The sweep records its checkpoint with a normal atomic write after attempting the snapshot.
+ * A crash before that write, or before a hybrid flush, can cause the time to be retried. Snapshot names
+ * derive from the scheduled time, so an existing snapshot prevents a duplicate on retry. After downtime
+ * only the latest due time is attempted, and nothing before an action was created, scheduled again or
+ * enabled again is replayed. Memory storage forgets actions on restart.
+ *
+ * <p>Lock order: the action's lock, then this service's monitor, then the Serverless service's. A
+ * mutation and a sweep of the same action hold its lock, so neither sees the other half done; the
+ * monitor is never held while a snapshot is dumped, so reads stay available during one.
*/
@ApplicationScoped
public class RedshiftServerlessScheduledActionService implements Resettable {
+ private static final Logger LOG = Logger.getLogger(RedshiftServerlessScheduledActionService.class);
private static final int MAX_RETENTION_DAYS = 3653;
+ private static final int MAX_SNAPSHOT_NAME_LENGTH = 255;
+ private static final DateTimeFormatter INVOCATION_SUFFIX =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd-HH-mm-ss", Locale.ROOT).withZone(ZoneOffset.UTC);
private final AccountAwareStorageBackend<ScheduledAction> actions;
private final RedshiftServerlessService serverless;
+ private final RedshiftServerlessScheduledActionAuthorizer authorizer;
+ private final RegionResolver regionResolver;
+ private final Clock clock;
+ private final ConcurrentHashMap<String, ReentrantLock> actionLocks = new ConcurrentHashMap<>();
@Inject
public RedshiftServerlessScheduledActionService(StorageFactory storageFactory,
- RedshiftServerlessService serverless) {
+ RedshiftServerlessService serverless,
+ RedshiftServerlessScheduledActionAuthorizer authorizer,
+ RegionResolver regionResolver,
+ Clock clock) {
this.actions = storageFactory.create("redshiftserverless", "redshiftserverless-scheduledactions.json",
new TypeReference<Map<String, ScheduledAction>>() {});
this.serverless = serverless;
+ this.authorizer = authorizer;
+ this.regionResolver = regionResolver;
+ this.clock = clock;
+ }
+
+ RedshiftServerlessScheduledActionService(StorageFactory storageFactory,
+ RedshiftServerlessService serverless,
+ RedshiftServerlessScheduledActionAuthorizer authorizer,
+ RegionResolver regionResolver) {
+ this(storageFactory, serverless, authorizer, regionResolver, Clock.systemUTC());
}
/**
* Creates the action described by {@code request}. {@code enabled} defaults to true. The target
* namespace of the snapshot and the owning namespace must both exist.
*/
- public synchronized ScheduledAction createScheduledAction(ScheduledAction request, String region) {
- requireNamespace(request.getNamespaceName(), region);
- validateSchedule(request);
- validateTarget(request, region);
- validateWindow(request);
- if (live(region).stream().anyMatch(existing ->
- existing.getScheduledActionName().equals(request.getScheduledActionName()))) {
- throw ServerlessSupport.conflict("The scheduled action " + request.getScheduledActionName()
- + " already exists.");
- }
+ public ScheduledAction createScheduledAction(ScheduledAction request, String region) {
+ return mutate(region, request.getScheduledActionName(), () -> {
+ requireNamespace(request.getNamespaceName(), region);
+ validateSchedule(request);
+ validateTarget(request, region);
+ validateWindow(request);
+ if (live(region).stream().anyMatch(existing ->
+ existing.getScheduledActionName().equals(request.getScheduledActionName()))) {
+ throw ServerlessSupport.conflict("The scheduled action " + request.getScheduledActionName()
+ + " already exists.");
+ }
- ScheduledAction action = new ScheduledAction(request);
- action.setOwnerId(namespaceIdOf(request.getNamespaceName(), region));
- action.setScheduledActionUuid(UUID.randomUUID().toString());
- action.setEnabled(request.getEnabled() == null || request.getEnabled());
- actions.put(storageKey(region, action.getScheduledActionName()), action);
- return new ScheduledAction(action);
+ ScheduledAction action = new ScheduledAction(request);
+ action.setOwnerId(namespaceIdOf(request.getNamespaceName(), region));
+ action.setScheduledActionUuid(UUID.randomUUID().toString());
+ action.setEnabled(request.getEnabled() == null || request.getEnabled());
+ action.setActivatedAt(clock.instant());
+ actions.put(storageKey(region, action.getScheduledActionName()), action);
+ return new ScheduledAction(action);
+ });
}
public synchronized ScheduledAction getScheduledAction(String scheduledActionName, String region) {
@@ -86,55 +133,191 @@ public class RedshiftServerlessScheduledActionService implements Resettable {
/**
* Applies the members {@code changes} carries. The schedule and the target replace the stored
- * ones as a whole, because each is a union of which exactly one member may be set.
+ * ones as a whole, because each is a union of which exactly one member may be set. A new schedule,
+ * or enabling a disabled action, restarts the count from now so nothing earlier is replayed.
*/
- public synchronized ScheduledAction updateScheduledAction(ScheduledAction changes, String region) {
- ScheduledAction current = require(changes.getScheduledActionName(), region);
- ScheduledAction updated = new ScheduledAction(current);
- if (changes.getRoleArn() != null) {
- updated.setRoleArn(changes.getRoleArn());
+ public ScheduledAction updateScheduledAction(ScheduledAction changes, String region) {
+ return mutate(region, changes.getScheduledActionName(), () -> {
+ ScheduledAction current = require(changes.getScheduledActionName(), region);
+ ScheduledAction updated = new ScheduledAction(current);
+ if (changes.getRoleArn() != null) {
+ updated.setRoleArn(changes.getRoleArn());
+ }
+ if (changes.getDescription() != null) {
+ updated.setDescription(changes.getDescription());
+ }
+ if (changes.getEnabled() != null) {
+ if (changes.getEnabled() && Boolean.FALSE.equals(current.getEnabled())) {
+ updated.setActivatedAt(clock.instant());
+ }
+ updated.setEnabled(changes.getEnabled());
+ }
+ if (changes.getStartTime() != null) {
+ updated.setStartTime(changes.getStartTime());
+ }
+ if (changes.getEndTime() != null) {
+ updated.setEndTime(changes.getEndTime());
+ }
+ if (changes.isScheduleSpecified()) {
+ validateSchedule(changes);
+ updated.setScheduleAt(changes.getScheduleAt());
+ updated.setScheduleCron(changes.getScheduleCron());
+ updated.setActivatedAt(clock.instant());
+ updated.setLastClaimedInvocation(null);
+ }
+ boolean changesTarget = changes.isTargetSpecified()
+ || changes.getTargetNamespaceName() != null || changes.getSnapshotNamePrefix() != null
+ || changes.getRetentionPeriod() != null || changes.getSnapshotTags() != null;
+ if (changesTarget) {
+ validateTarget(changes, region);
+ updated.setTargetNamespaceName(changes.getTargetNamespaceName());
+ updated.setSnapshotNamePrefix(changes.getSnapshotNamePrefix());
+ updated.setRetentionPeriod(changes.getRetentionPeriod());
+ updated.setSnapshotTags(changes.getSnapshotTags());
+ }
+ validateWindow(updated);
+ actions.put(storageKey(region, updated.getScheduledActionName()), updated);
+ return new ScheduledAction(updated);
+ });
+ }
+
+ public ScheduledAction deleteScheduledAction(String scheduledActionName, String region) {
+ return mutate(region, scheduledActionName, () -> {
+ ScheduledAction existing = new ScheduledAction(require(scheduledActionName, region));
+ actions.delete(storageKey(region, scheduledActionName));
+ return existing;
+ });
+ }
+
+ /**
+ * Takes the snapshot of every action whose latest due time falls in {@code (checkpoint, now]}. One
+ * action's failure never stops the others. A deterministic entry point: the lifecycle scheduler
+ * supplies the clock and tests supply a fixed instant.
+ */
+ public void runScheduledActions(Instant now) {
+ for (AccountEntry<ScheduledAction> entry : actions.scanAllAccountEntries(key -> true)) {
+ if (Thread.currentThread().isInterrupted()) {
+ return;
+ }
+ String region = regionOf(entry.key());
+ if (region == null) {
+ continue;
+ }
+ ReentrantLock lock = lockOf(entry.accountId(), entry.key());
+ lock.lock();
+ try {
+ RequestScopes.runAs(entry.accountId(), region, () -> runScheduledAction(entry, region, now));
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Scheduled action {0} in account {1} could not be processed",
+ entry.key(), entry.accountId());
+ } finally {
+ lock.unlock();
+ }
}
- if (changes.getDescription() != null) {
- updated.setDescription(changes.getDescription());
+ }
+
+ /** Runs under the action's lock, with the action's account and Region as the request scope. */
+ private void runScheduledAction(AccountEntry<ScheduledAction> listed, String region, Instant now) {
+ String accountId = listed.accountId();
+ String key = listed.key();
+ ScheduledAction action = actions.getForAccount(accountId, key).orElse(null);
+ if (action == null) {
+ return;
}
- if (changes.getEnabled() != null) {
- updated.setEnabled(changes.getEnabled());
+ if (!isOnSameNamespace(action, region)) {
+ actions.deleteForAccount(accountId, key);
+ return;
}
- if (changes.getStartTime() != null) {
- updated.setStartTime(changes.getStartTime());
+ if (action.getActivatedAt() == null) {
+ // A stored action from before scheduling existed starts counting now, without replaying history.
+ ScheduledAction activated = new ScheduledAction(action);
+ activated.setActivatedAt(now);
+ actions.putAllAtomicallyForAccount(accountId, Map.of(key, activated));
+ return;
}
- if (changes.getEndTime() != null) {
- updated.setEndTime(changes.getEndTime());
+ Instant checkpoint = action.getLastClaimedInvocation() != null
+ && action.getLastClaimedInvocation().isAfter(action.getActivatedAt())
+ ? action.getLastClaimedInvocation() : action.getActivatedAt();
+ Optional<Instant> due = RedshiftServerlessSchedule.latestDue(action, checkpoint, now);
+ if (due.isEmpty()) {
+ return;
}
- if (changes.isScheduleSpecified()) {
- validateSchedule(changes);
- updated.setScheduleAt(changes.getScheduleAt());
- updated.setScheduleCron(changes.getScheduleCron());
+ Instant scheduled = due.get();
+ ScheduledAction outcome = new ScheduledAction(action);
+ outcome.setLastClaimedInvocation(scheduled);
+ outcome.setLastFailure(null);
+ try {
+ takeSnapshot(action, accountId, region, scheduled);
+ outcome.setLastCompletedInvocation(scheduled);
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Scheduled action {0} in account {1} failed at {2}",
+ action.getScheduledActionName(), accountId, scheduled);
+ outcome.setLastFailure(String.valueOf(e.getMessage()));
}
- boolean changesTarget = changes.isTargetSpecified()
- || changes.getTargetNamespaceName() != null || changes.getSnapshotNamePrefix() != null
- || changes.getRetentionPeriod() != null || changes.getSnapshotTags() != null;
- if (changesTarget) {
- validateTarget(changes, region);
- updated.setTargetNamespaceName(changes.getTargetNamespaceName());
- updated.setSnapshotNamePrefix(changes.getSnapshotNamePrefix());
- updated.setRetentionPeriod(changes.getRetentionPeriod());
- updated.setSnapshotTags(changes.getSnapshotTags());
+ recordOutcome(accountId, key, action.getScheduledActionUuid(), outcome);
+ }
+
+ private void takeSnapshot(ScheduledAction action, String accountId, String region, Instant scheduled) {
+ Namespace target = serverless.findNamespace(action.getTargetNamespaceName(), region).orElseThrow(() ->
+ ServerlessSupport.notFound("The namespace " + action.getTargetNamespaceName() + " was not found."));
+ Optional<String> refusal = authorizer.refusal(action.getRoleArn(), accountId, target.getNamespaceArn());
+ if (refusal.isPresent()) {
+ throw ServerlessSupport.validation(refusal.get());
}
- validateWindow(updated);
- actions.put(storageKey(region, updated.getScheduledActionName()), updated);
- return new ScheduledAction(updated);
+ serverless.createSnapshot(snapshotName(action.getSnapshotNamePrefix(), scheduled),
+ action.getTargetNamespaceName(), action.getRetentionPeriod(), action.getSnapshotTags(), region);
}
- public synchronized ScheduledAction deleteScheduledAction(String scheduledActionName, String region) {
- ScheduledAction existing = new ScheduledAction(require(scheduledActionName, region));
- actions.delete(storageKey(region, scheduledActionName));
- return existing;
+ /** Stores the checkpoint and outcome only for the action that was run, not for a recreated one. */
+ private void recordOutcome(String accountId, String key, String uuid, ScheduledAction outcome) {
+ try {
+ ScheduledAction current = actions.getForAccount(accountId, key).orElse(null);
+ if (current != null && Objects.equals(current.getScheduledActionUuid(), uuid)) {
+ ScheduledAction recorded = new ScheduledAction(current);
+ recorded.setLastClaimedInvocation(outcome.getLastClaimedInvocation());
+ recorded.setLastCompletedInvocation(outcome.getLastCompletedInvocation());
+ recorded.setLastFailure(outcome.getLastFailure());
+ actions.putAllAtomicallyForAccount(accountId, Map.of(key, recorded));
+ }
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "The outcome of scheduled action {0} in account {1} could not be stored", key, accountId);
+ }
+ }
+
+ static String snapshotName(String prefix, Instant scheduled) {
+ String suffix = "-" + INVOCATION_SUFFIX.format(scheduled);
+ String base = prefix == null ? "" : prefix.toLowerCase(Locale.ROOT);
+ if (base.length() + suffix.length() > MAX_SNAPSHOT_NAME_LENGTH) {
+ base = base.substring(0, MAX_SNAPSHOT_NAME_LENGTH - suffix.length());
+ }
+ return base + suffix;
}
@Override
public void clear() {
actions.clear();
+ actionLocks.clear();
+ }
+
+ private <T> T mutate(String region, String name, Supplier<T> body) {
+ ReentrantLock lock = lockOf(regionResolver.getAccountId(), storageKey(region, name));
+ lock.lock();
+ try {
+ synchronized (this) {
+ return body.get();
+ }
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ private ReentrantLock lockOf(String accountId, String key) {
+ return actionLocks.computeIfAbsent(accountId + "|" + key, ignored -> new ReentrantLock());
+ }
+
+ private static String regionOf(String key) {
+ int separator = key.indexOf("::");
+ return separator < 0 ? null : key.substring(0, separator);
}
private ScheduledAction require(String scheduledActionName, String region) {
@@ -180,6 +363,7 @@ public class RedshiftServerlessScheduledActionService implements Resettable {
if (hasAt == hasCron) {
throw ServerlessSupport.validation("schedule must set exactly one of at and cron.");
}
+ RedshiftServerlessSchedule.validate(action);
}
private void validateTarget(ScheduledAction action, String region) {
@@ -187,6 +371,9 @@ public class RedshiftServerlessScheduledActionService implements Resettable {
throw ServerlessSupport.validation("targetAction must set createSnapshot.");
}
requireNamespace(action.getTargetNamespaceName(), region);
+ if (action.getSnapshotNamePrefix() != null) {
+ RedshiftServerlessService.validateSnapshotName(snapshotName(action.getSnapshotNamePrefix(), Instant.EPOCH));
+ }
Integer retention = action.getRetentionPeriod();
if (retention != null && retention != -1 && (retention < 1 || retention > MAX_RETENTION_DAYS)) {
throw ServerlessSupport.validation("targetAction.createSnapshot.retentionPeriod must be -1 "
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java
index eae435af76..1066cdbafa 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java
@@ -1166,7 +1166,7 @@ public class RedshiftServerlessService implements Resettable {
}
}
- private static void validateSnapshotName(String snapshotName) {
+ static void validateSnapshotName(String snapshotName) {
requireSnapshotName(snapshotName);
if (snapshotName.length() < 3 || snapshotName.length() > 255 || !SNAPSHOT_NAME.matcher(snapshotName).matches()) {
throw ServerlessSupport.validation("snapshotName must be 3-255 characters of lowercase letters, numbers, and hyphens.");
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java
index ae9d69eb1d..9a5889acba 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java
@@ -8,8 +8,8 @@ import java.util.LinkedHashMap;
import java.util.Map;
/**
- * A Redshift Serverless scheduled action. Floci stores and echoes it but never runs it, so no
- * snapshot is taken on the schedule. The same type carries an update request, in which a null member
+ * A Redshift Serverless scheduled action. The activation and invocation members are internal: the
+ * scheduler keeps them, and no AWS response shows them. The same type carries an update request, in which a null member
* means the caller left it out.
*/
@RegisterForReflection
@@ -33,12 +33,20 @@ public class ScheduledAction {
private String ownerId;
private boolean scheduleSpecified;
private boolean targetSpecified;
+ private Instant activatedAt;
+ private Instant lastClaimedInvocation;
+ private Instant lastCompletedInvocation;
+ private String lastFailure;
public ScheduledAction() {
}
public ScheduledAction(ScheduledAction other) {
this.ownerId = other.ownerId;
+ this.activatedAt = other.activatedAt;
+ this.lastClaimedInvocation = other.lastClaimedInvocation;
+ this.lastCompletedInvocation = other.lastCompletedInvocation;
+ this.lastFailure = other.lastFailure;
this.scheduleSpecified = other.scheduleSpecified;
this.targetSpecified = other.targetSpecified;
this.scheduledActionName = other.scheduledActionName;
@@ -192,4 +200,38 @@ public class ScheduledAction {
public void setTargetSpecified(boolean targetSpecified) {
this.targetSpecified = targetSpecified;
}
+
+ /** When the current schedule began counting, so no invocation before it is replayed. */
+ public Instant getActivatedAt() {
+ return activatedAt;
+ }
+
+ public void setActivatedAt(Instant activatedAt) {
+ this.activatedAt = activatedAt;
+ }
+
+ /** The latest scheduled time whose snapshot was started; it is never attempted twice. */
+ public Instant getLastClaimedInvocation() {
+ return lastClaimedInvocation;
+ }
+
+ public void setLastClaimedInvocation(Instant lastClaimedInvocation) {
+ this.lastClaimedInvocation = lastClaimedInvocation;
+ }
+
+ public Instant getLastCompletedInvocation() {
+ return lastCompletedInvocation;
+ }
+
+ public void setLastCompletedInvocation(Instant lastCompletedInvocation) {
+ this.lastCompletedInvocation = lastCompletedInvocation;
+ }
+
+ public String getLastFailure() {
+ return lastFailure;
+ }
+
+ public void setLastFailure(String lastFailure) {
+ this.lastFailure = lastFailure;
+ }
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 44363c5f3f..d58274e381 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -583,6 +583,8 @@ floci:
shutdown-timeout-seconds: ${FLOCI_SERVICES_REDSHIFT_DATA_SHUTDOWN_TIMEOUT_SECONDS:10}
redshift-serverless:
enabled: true
+ scheduled-actions-enabled: ${FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTIONS_ENABLED:true}
+ scheduled-action-sweep-interval-seconds: ${FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTION_SWEEP_INTERVAL_SECONDS:60}
neptune:
enabled: true
proxy-base-port: 8182
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java
index f55934d9a0..eb95e9e4a5 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java
@@ -93,7 +93,8 @@ class RedshiftServerlessAdministrationHandlerTest {
new RedshiftServerlessEndpointAccessService(storageFactory, regionResolver, serverlessService),
customDomains,
new RedshiftServerlessSnapshotCopyService(storageFactory, regionResolver, serverlessService),
- new RedshiftServerlessScheduledActionService(storageFactory, serverlessService)),
+ new RedshiftServerlessScheduledActionService(storageFactory, serverlessService,
+ mock(RedshiftServerlessScheduledActionAuthorizer.class), regionResolver)),
customDomains);
}
@@ -524,7 +525,7 @@ class RedshiftServerlessAdministrationHandlerTest {
}
@Test
- void scheduledActionIsStoredAndNeverRun() {
+ void scheduledActionIsStoredWithItsNextCronInvocation() {
namespace("sched-ns");
long future = Instant.now().plusSeconds(3600).getEpochSecond();
String target = "\"targetAction\":{\"createSnapshot\":{\"namespaceName\":\"sched-ns\","
@@ -541,7 +542,7 @@ class RedshiftServerlessAdministrationHandlerTest {
assertEquals(7, created.get("targetAction").get("createSnapshot").get("retentionPeriod").intValue());
assertEquals("env", created.get("targetAction").get("createSnapshot").get("tags").get(0).get("key").textValue());
assertEquals("nightly", created.get("scheduledActionDescription").textValue());
- assertEquals(0, created.get("nextInvocations").size());
+ assertEquals(1, created.get("nextInvocations").size());
assertTrue(created.get("scheduledActionUuid").textValue().length() > 8);
assertEquals("ConflictException", errorType("CreateScheduledAction", "{\"scheduledActionName\":\"nightly-snap\","
@@ -696,7 +697,7 @@ class RedshiftServerlessAdministrationHandlerTest {
void aNamespaceRecreatedUnderTheSameNameDoesNotInheritActions() {
namespace("recreate-ns");
ok("CreateScheduledAction", "{\"scheduledActionName\":\"old-action\",\"namespaceName\":\"recreate-ns\","
- + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"x\"},\"targetAction\":{\"createSnapshot\":"
+ + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{\"createSnapshot\":"
+ "{\"namespaceName\":\"recreate-ns\",\"snapshotNamePrefix\":\"p\"}}}");
ok("DeleteNamespace", "{\"namespaceName\":\"recreate-ns\"}");
@@ -705,7 +706,7 @@ class RedshiftServerlessAdministrationHandlerTest {
assertEquals("ResourceNotFoundException", errorType("GetScheduledAction",
"{\"scheduledActionName\":\"old-action\"}"));
ok("CreateScheduledAction", "{\"scheduledActionName\":\"old-action\",\"namespaceName\":\"recreate-ns\","
- + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"x\"},\"targetAction\":{\"createSnapshot\":"
+ + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{\"createSnapshot\":"
+ "{\"namespaceName\":\"recreate-ns\",\"snapshotNamePrefix\":\"p\"}}}");
}
@@ -724,7 +725,7 @@ class RedshiftServerlessAdministrationHandlerTest {
void scheduledActionUpdateRejectsAPartialTargetAndATagWithoutAKey() {
namespace("partial-ns");
String create = "{\"scheduledActionName\":\"partial-action\",\"namespaceName\":\"partial-ns\",\"roleArn\":\"r\","
- + "\"schedule\":{\"cron\":\"x\"},\"targetAction\":{\"createSnapshot\":"
+ + "\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{\"createSnapshot\":"
+ "{\"namespaceName\":\"partial-ns\",\"snapshotNamePrefix\":\"p\"";
ok("CreateScheduledAction", create + "}}}");
@@ -737,12 +738,12 @@ class RedshiftServerlessAdministrationHandlerTest {
"{\"scheduledActionName\":\"partial-action\",\"schedule\":{}}"));
JsonNode unchanged = ok("UpdateScheduledAction",
"{\"scheduledActionName\":\"partial-action\",\"enabled\":false}").get("scheduledAction");
- assertEquals("x", unchanged.get("schedule").get("cron").textValue());
+ assertEquals("0 3 * * ? *", unchanged.get("schedule").get("cron").textValue());
assertEquals("DISABLED", unchanged.get("state").textValue());
assertEquals("ValidationException", errorType("CreateScheduledAction",
"{\"scheduledActionName\":\"empty-target\",\"namespaceName\":\"partial-ns\",\"roleArn\":\"r\","
- + "\"schedule\":{\"cron\":\"x\"},\"targetAction\":{}}"));
+ + "\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{}}"));
assertEquals("ValidationException", errorType("CreateScheduledAction",
create.replace("partial-action", "tag-action") + ",\"tags\":[{\"value\":\"v\"}]}}}"));
assertEquals("ValidationException", errorType("CreateScheduledAction",
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java
index c7cf734c0b..97a5c5acaf 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java
@@ -10,6 +10,7 @@ import io.quarkus.test.common.http.TestHTTPResource;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.response.ValidatableResponse;
import io.restassured.specification.RequestSpecification;
+import jakarta.inject.Inject;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -21,6 +22,7 @@ import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
+import java.time.temporal.ChronoUnit;
import java.util.List;
import java.util.Map;
import java.util.UUID;
@@ -785,6 +787,43 @@ class RedshiftServerlessIntegrationTest {
call("DeleteScheduledAction", "{\"scheduledActionName\":\"admin-sched\"}").statusCode(200);
}
+ @Inject
+ RedshiftServerlessScheduledActionService scheduledActions;
+
+ @Test
+ void scheduledActionTakesItsSnapshotOnASweepAndReportsTheNextInvocation() {
+ String namespace = "sweep-ns";
+ call("CreateNamespace", "{\"namespaceName\":\"" + namespace + "\",\"adminUsername\":\"admin\"}")
+ .statusCode(200);
+ cleanup.register(() -> call("DeleteNamespace", "{\"namespaceName\":\"" + namespace + "\"}"));
+ call("CreateScheduledAction", "{\"scheduledActionName\":\"sweep-sched\",\"namespaceName\":\"" + namespace + "\","
+ + "\"roleArn\":\"arn:aws:iam::000000000000:role/r\",\"schedule\":{\"cron\":\"0 * * * ? *\"},"
+ + "\"targetAction\":{\"createSnapshot\":{\"namespaceName\":\"" + namespace + "\","
+ + "\"snapshotNamePrefix\":\"sweep\",\"retentionPeriod\":3,"
+ + "\"tags\":[{\"key\":\"env\",\"value\":\"it\"}]}}}")
+ .statusCode(200)
+ .body("scheduledAction.nextInvocations", hasSize(1));
+ cleanup.register(() -> call("DeleteScheduledAction", "{\"scheduledActionName\":\"sweep-sched\"}"));
+ call("CreateScheduledAction", "{\"scheduledActionName\":\"too-often\",\"namespaceName\":\"" + namespace + "\","
+ + "\"roleArn\":\"arn:aws:iam::000000000000:role/r\",\"schedule\":{\"cron\":\"0,30 * * * ? *\"},"
+ + "\"targetAction\":{\"createSnapshot\":{\"namespaceName\":\"" + namespace + "\","
+ + "\"snapshotNamePrefix\":\"p\"}}}")
+ .statusCode(400)
+ .body("__type", equalTo("ValidationException"));
+
+ Instant tick = Instant.now().plusSeconds(3 * 3600);
+ scheduledActions.runScheduledActions(tick);
+ String snapshot = RedshiftServerlessScheduledActionService.snapshotName("sweep",
+ tick.truncatedTo(ChronoUnit.HOURS));
+ cleanup.register(() -> call("DeleteSnapshot", "{\"snapshotName\":\"" + snapshot + "\"}"));
+
+ call("GetSnapshot", "{\"snapshotName\":\"" + snapshot + "\"}")
+ .statusCode(200)
+ .body("snapshot.namespaceName", equalTo(namespace))
+ .body("snapshot.status", equalTo("AVAILABLE"))
+ .body("snapshot.snapshotRetentionPeriod", equalTo(3));
+ }
+
private static ValidatableResponse call(String action, String body) {
return given()
.header("X-Amz-Target", TARGET_PREFIX + action)
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java
index 78f544dba6..f27de5a7f3 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java
@@ -132,7 +132,8 @@ class RedshiftServerlessJsonHandlerTest {
new RedshiftServerlessEndpointAccessService(storageFactory, regionResolver, service),
customDomains,
new RedshiftServerlessSnapshotCopyService(storageFactory, regionResolver, service),
- new RedshiftServerlessScheduledActionService(storageFactory, service)),
+ new RedshiftServerlessScheduledActionService(storageFactory, service,
+ mock(RedshiftServerlessScheduledActionAuthorizer.class), regionResolver)),
customDomains);
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java
index 29bfef67bf..1880d9c79f 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.redshiftserverless;
+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.response.ExtractableResponse;
@@ -20,6 +21,8 @@ import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
import java.util.Objects;
import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -107,9 +110,13 @@ class RedshiftServerlessRuntimeIntegrationTest {
}
private String executeAndWait(String sql) {
+ return executeAndWait(sql, WORKGROUP);
+ }
+
+ private String executeAndWait(String sql, String workgroup) {
String id = RestAssuredJsonUtils.awsAction("RedshiftData", "ExecuteStatement", """
{"Sql": "%s", "WorkgroupName": "%s", "Database": "dev"}
- """.formatted(sql, WORKGROUP)).then().statusCode(200)
+ """.formatted(sql, workgroup)).then().statusCode(200)
.extract().path("Id");
Awaitility.await().atMost(Duration.ofSeconds(30)).pollDelay(Duration.ZERO)
.pollInterval(Duration.ofMillis(250)).until(() -> {
@@ -163,6 +170,42 @@ class RedshiftServerlessRuntimeIntegrationTest {
""".formatted(workgroupArn)).then().statusCode(200);
}
+ @Inject
+ RedshiftServerlessScheduledActionService scheduledActions;
+
+ @Test
+ void aScheduledSnapshotCapturesRowsAndRestoresThem() {
+ executeAndWait("CREATE TABLE sl_scheduled (id int)");
+ executeAndWait("INSERT INTO sl_scheduled VALUES (1), (2), (3)");
+ ScheduledAction request = new ScheduledAction();
+ request.setScheduledActionName("rt-it-sched");
+ request.setNamespaceName(NAMESPACE);
+ request.setRoleArn("arn:aws:iam::000000000000:role/r");
+ request.setScheduleCron("0 * * * ? *");
+ request.setTargetNamespaceName(NAMESPACE);
+ request.setSnapshotNamePrefix("rt-sched");
+ scheduledActions.createScheduledAction(request, REGION);
+ Instant tick = Instant.now().truncatedTo(ChronoUnit.HOURS).plus(Duration.ofHours(3)).plus(Duration.ofMinutes(15));
+ String snapshot = RedshiftServerlessScheduledActionService.snapshotName("rt-sched",
+ tick.truncatedTo(ChronoUnit.HOURS));
+ try {
+ scheduledActions.runScheduledActions(tick);
+ executeAndWait("DROP TABLE sl_scheduled");
+
+ RestAssuredJsonUtils.awsAction("RedshiftServerless", "RestoreFromSnapshot", """
+ {"namespaceName":"%s","workgroupName":"%s","snapshotName":"%s"}
+ """.formatted(NAMESPACE, WORKGROUP, snapshot)).then().statusCode(200);
+ String id = executeAndWait("SELECT count(*) FROM sl_scheduled", WORKGROUP);
+ ExtractableResponse<Response> result = RestAssuredJsonUtils.awsAction("RedshiftData", "GetStatementResult",
+ "{\"Id\":\"" + id + "\"}").then().statusCode(200).extract();
+ assertEquals(3, ((Number) result.path("Records[0][0].longValue")).intValue());
+ } finally {
+ scheduledActions.deleteScheduledAction("rt-it-sched", REGION);
+ service.deleteSnapshot(snapshot, REGION);
+ executeAndWait("DROP TABLE IF EXISTS sl_scheduled");
+ }
+ }
+
@Test
void theDataApiRejectsAWrongDatabaseAndBothIdentifiers() {
RestAssuredJsonUtils.awsAction("RedshiftData", "ExecuteStatement", """
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduleTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduleTest.java
new file mode 100644
index 0000000000..6fcb0bbc29
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduleTest.java
@@ -0,0 +1,75 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+class RedshiftServerlessScheduleTest {
+ private static final Instant NOW = Instant.parse("2030-01-07T02:00:00Z");
+
+ @Test
+ void cronComputesNextAndLatestInUtc() {
+ ScheduledAction action = cron("(0 3 ? * MON *)");
+ RedshiftServerlessSchedule.validate(action);
+ assertEquals(NOW.plusSeconds(3600), RedshiftServerlessSchedule.next(action, NOW).orElseThrow());
+ assertEquals(NOW.plusSeconds(3600),
+ RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(7200)).orElseThrow());
+ assertTrue(RedshiftServerlessSchedule.latestDue(action, NOW, NOW).isEmpty());
+ }
+
+ @Test
+ void numericWeekdayUsesAwsNumbering() {
+ assertEquals(NOW.plusSeconds(3600),
+ RedshiftServerlessSchedule.next(cron("0 3 ? * 2 *"), NOW).orElseThrow());
+ }
+
+ @Test
+ void supportsExistingCronWrapper() {
+ assertEquals(NOW.plusSeconds(3600),
+ RedshiftServerlessSchedule.next(cron("cron(0 3 ? * MON *)"), NOW).orElseThrow());
+ }
+
+ @Test
+ void rejectsInvalidAndSubHourlyCron() {
+ for (String expression : new String[]{"x", "* * *", "0 3 * * * *", "0,30 * * * ? *"}) {
+ assertThrows(AwsException.class, () -> RedshiftServerlessSchedule.validate(cron(expression)), expression);
+ }
+ }
+
+ @Test
+ void activeWindowBoundsNextAndDue() {
+ ScheduledAction action = cron("0 * * * ? *");
+ action.setStartTime(NOW.plusSeconds(7200));
+ action.setEndTime(NOW.plusSeconds(10800));
+ assertEquals(action.getStartTime(), RedshiftServerlessSchedule.next(action, NOW).orElseThrow());
+ assertEquals(action.getEndTime(),
+ RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(14400)).orElseThrow());
+ assertTrue(RedshiftServerlessSchedule.next(action, action.getEndTime()).isEmpty());
+ action.setEnabled(false);
+ assertTrue(RedshiftServerlessSchedule.next(action, NOW).isEmpty());
+ assertTrue(RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(14400)).isEmpty());
+ }
+
+ @Test
+ void oneOffIsDueOnlyOnceAndWithinWindow() {
+ ScheduledAction action = new ScheduledAction();
+ action.setScheduleAt(NOW.plusSeconds(10));
+ RedshiftServerlessSchedule.validate(action);
+ assertEquals(action.getScheduleAt(), RedshiftServerlessSchedule.next(action, NOW).orElseThrow());
+ assertEquals(action.getScheduleAt(),
+ RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(10)).orElseThrow());
+ assertTrue(RedshiftServerlessSchedule.latestDue(action, action.getScheduleAt(), NOW.plusSeconds(20)).isEmpty());
+ action.setEndTime(NOW.plusSeconds(9));
+ assertTrue(RedshiftServerlessSchedule.next(action, NOW).isEmpty());
+ }
+
+ private ScheduledAction cron(String expression) {
+ ScheduledAction action = new ScheduledAction();
+ action.setScheduleCron(expression);
+ return action;
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizerTest.java
new file mode 100644
index 0000000000..a96b51c10d
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizerTest.java
@@ -0,0 +1,177 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.services.iam.AssumeRolePolicyEvaluator;
+import io.github.hectorvent.floci.services.iam.IamPolicyEvaluator;
+import io.github.hectorvent.floci.services.iam.IamService;
+import io.github.hectorvent.floci.services.iam.ScpProvider;
+import io.github.hectorvent.floci.services.iam.model.CallerContext;
+import io.github.hectorvent.floci.services.iam.model.IamRole;
+import jakarta.enterprise.inject.Instance;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class RedshiftServerlessScheduledActionAuthorizerTest {
+ private static final String ACCOUNT_ID = "123456789012";
+ private static final String ROLE_ARN = "arn:aws:iam::" + ACCOUNT_ID + ":role/scheduler";
+ private static final String TARGET = "arn:aws:redshift-serverless:us-east-1:" + ACCOUNT_ID + ":namespace/ns-id";
+ private static final String TRUST = """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",
+ "Principal":{"Service":"scheduler.redshift.amazonaws.com"},"Action":"sts:AssumeRole"}]}""";
+ private static final String ALLOW = """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",
+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*"}]}""";
+
+ private EmulatorConfig config;
+ private IamService iamService;
+ private Instance<ScpProvider> scpProvider;
+ private ScpProvider scps;
+ private RedshiftServerlessScheduledActionAuthorizer authorizer;
+
+ @BeforeEach
+ @SuppressWarnings("unchecked")
+ void setUp() {
+ config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().iam().enforcementEnabled()).thenReturn(true);
+ iamService = mock(IamService.class);
+ IamPolicyEvaluator evaluator = new IamPolicyEvaluator(new ObjectMapper());
+ scpProvider = mock(Instance.class);
+ scps = mock(ScpProvider.class);
+ when(scpProvider.isResolvable()).thenReturn(true);
+ when(scpProvider.get()).thenReturn(scps);
+ authorizer = new RedshiftServerlessScheduledActionAuthorizer(config, iamService, evaluator,
+ new AssumeRolePolicyEvaluator(new ObjectMapper(), evaluator), scpProvider);
+ }
+
+ private void role(String arn, String trust, String policy) {
+ IamRole role = new IamRole("id", "scheduler", "/", arn, trust);
+ when(iamService.findRole(ACCOUNT_ID, "scheduler")).thenReturn(Optional.of(role));
+ when(iamService.resolvePrincipalContext(arn)).thenReturn(CallerContext.of(List.of(policy)));
+ }
+
+ @Test
+ void trustedRoleWithThePermissionIsAllowed() {
+ role(ROLE_ARN, TRUST, ALLOW);
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());
+ }
+
+ @Test
+ void roleInAnotherPartitionOfTheSameAccountIsEvaluatedAsItsOwn() {
+ String chinaRole = "arn:aws-cn:iam::" + ACCOUNT_ID + ":role/scheduler";
+ role(chinaRole, TRUST, ALLOW);
+ assertTrue(authorizer.refusal(chinaRole, ACCOUNT_ID,
+ "arn:aws-cn:redshift-serverless:cn-north-1:" + ACCOUNT_ID + ":namespace/ns-id").isEmpty());
+ }
+
+
+ @Test
+ void serviceControlPolicyDenyIsRefused() {
+ role(ROLE_ARN, TRUST, ALLOW);
+ when(scps.effectiveScpLevels(ACCOUNT_ID)).thenReturn(List.of(List.of("""
+ {"Version":"2012-10-17","Statement":[{"Effect":"Deny",
+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*"}]}""")));
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void serviceControlPolicyThatAllowsLeavesTheRoleAllowed() {
+ role(ROLE_ARN, TRUST, ALLOW);
+ when(scps.effectiveScpLevels(ACCOUNT_ID)).thenReturn(List.of(List.of("""
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"*","Resource":"*"}]}""")));
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());
+ }
+ @Test
+ void trustPolicyWithMatchingSourceAccountConditionIsAllowed() {
+ String conditionalTrust = """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",
+ "Principal":{"Service":"scheduler.redshift.amazonaws.com"},"Action":"sts:AssumeRole",
+ "Condition":{"StringEquals":{"aws:SourceAccount":"%s"}}}]}""".formatted(ACCOUNT_ID);
+ role(ROLE_ARN, conditionalTrust, ALLOW);
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());
+ }
+
+ @Test
+ void trustPolicyWithMismatchedSourceAccountConditionIsRefused() {
+ String conditionalTrust = """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",
+ "Principal":{"Service":"scheduler.redshift.amazonaws.com"},"Action":"sts:AssumeRole",
+ "Condition":{"StringEquals":{"aws:SourceAccount":"999999999999"}}}]}""";
+ role(ROLE_ARN, conditionalTrust, ALLOW);
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void regionalConditionMatchingTargetRegionIsAllowed() {
+ String regionalPolicy = """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",
+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*",
+ "Condition":{"StringEquals":{"aws:RequestedRegion":"us-east-1"}}}]}""";
+ role(ROLE_ARN, TRUST, regionalPolicy);
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());
+ }
+
+ @Test
+ void regionalConditionForDifferentRegionIsRefused() {
+ String regionalPolicy = """
+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",
+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*",
+ "Condition":{"StringEquals":{"aws:RequestedRegion":"us-west-2"}}}]}""";
+ role(ROLE_ARN, TRUST, regionalPolicy);
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void missingPermissionIsRefused() {
+ role(ROLE_ARN, TRUST, ALLOW.replace("CreateSnapshot", "GetSnapshot"));
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void explicitDenyIsRefused() {
+ role(ROLE_ARN, TRUST, ALLOW.replace("\"Allow\"", "\"Deny\""));
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void roleThatDoesNotTrustTheSchedulerIsRefused() {
+ role(ROLE_ARN, TRUST.replace("scheduler.redshift", "lambda"), ALLOW);
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void missingRoleIsRefused() {
+ when(iamService.findRole(ACCOUNT_ID, "scheduler")).thenReturn(Optional.empty());
+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void roleOfAnotherAccountIsRefusedWithoutLookingItUp() {
+ Optional<String> refusal = authorizer.refusal("arn:aws:iam::210987654321:role/scheduler", ACCOUNT_ID, TARGET);
+ assertTrue(refusal.isPresent());
+ verify(iamService, never()).findRole(anyString(), anyString());
+ }
+
+ @Test
+ void nonRoleArnIsRefused() {
+ assertTrue(authorizer.refusal("arn:aws:iam::" + ACCOUNT_ID + ":user/someone", ACCOUNT_ID, TARGET).isPresent());
+ assertTrue(authorizer.refusal("not-an-arn", ACCOUNT_ID, TARGET).isPresent());
+ }
+
+ @Test
+ void enforcementOffAllowsEverything() {
+ when(config.services().iam().enforcementEnabled()).thenReturn(false);
+ assertTrue(authorizer.refusal("not-an-arn", ACCOUNT_ID, TARGET).isEmpty());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionRecoveryTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionRecoveryTest.java
new file mode 100644
index 0000000000..f06f4b25e8
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionRecoveryTest.java
@@ -0,0 +1,148 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.HybridStorage;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.redshiftserverless.model.Namespace;
+import io.github.hectorvent.floci.services.redshiftserverless.model.RedshiftServerlessSnapshot;
+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.nio.file.Path;
+import java.time.Clock;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.*;
+import static org.mockito.Mockito.*;
+
+class RedshiftServerlessScheduledActionRecoveryTest {
+
+ private static final String ACCOUNT = "123456789012";
+ private static final String REGION = "us-east-1";
+ private static final Instant CREATED = Instant.parse("2026-10-28T12:15:00Z");
+ private static final Instant DUE = Instant.parse("2026-10-28T13:00:00Z");
+ private static final String SNAPSHOT_NAME = "nightly-2026-10-28-13-00-00";
+
+ @TempDir
+ Path directory;
+
+ private final List<HybridStorage<String, ScheduledAction>> backends = new ArrayList<>();
+ private StorageFactory factory;
+ private RegionResolver regions;
+ private RedshiftServerlessService serverless;
+ private AccountAwareStorageBackend<RedshiftServerlessSnapshot> snapshots;
+ private RedshiftServerlessScheduledActionService service;
+
+ @BeforeEach
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ void setUp() {
+ factory = mock(StorageFactory.class);
+ when(factory.create(anyString(), anyString(), any(TypeReference.class)))
+ .thenAnswer(invocation -> AccountAwareStorageBackend.inMemory(ACCOUNT));
+ AccountAwareStorageBackend<Namespace> namespaces = AccountAwareStorageBackend.inMemory(ACCOUNT);
+ Namespace namespace = new Namespace();
+ namespace.setNamespaceName("source-ns");
+ namespace.setNamespaceId("ns-id");
+ namespace.setNamespaceArn("arn:aws:redshift-serverless:us-east-1:" + ACCOUNT + ":namespace/ns-id");
+ namespaces.put(REGION + "::source-ns", namespace);
+ when(factory.create(eq("redshiftserverless"), eq("redshiftserverless-namespaces.json"), any(TypeReference.class)))
+ .thenReturn((AccountAwareStorageBackend) namespaces);
+ snapshots = AccountAwareStorageBackend.inMemory(ACCOUNT);
+ when(factory.create(eq("redshiftserverless"), eq("redshiftserverless-snapshots.json"), any(TypeReference.class)))
+ .thenReturn((AccountAwareStorageBackend) snapshots);
+ regions = mock(RegionResolver.class);
+ when(regions.getAccountId()).thenReturn(ACCOUNT);
+ serverless = spy(new RedshiftServerlessService(factory, regions, mock(RedshiftServerlessEndpoints.class),
+ mock(RedshiftServerlessRuntime.class), mock(EmulatorConfig.class)));
+ service = reload();
+ ScheduledAction action = new ScheduledAction();
+ action.setScheduledActionName("hourly");
+ action.setNamespaceName("source-ns");
+ action.setRoleArn("arn:aws:iam::" + ACCOUNT + ":role/scheduler");
+ action.setScheduleCron("0 * * * ? *");
+ action.setTargetNamespaceName("source-ns");
+ action.setSnapshotNamePrefix("nightly");
+ service.createScheduledAction(action, REGION);
+ backends.getFirst().flush();
+ }
+
+ @AfterEach
+ void stopBackends() {
+ backends.forEach(HybridStorage::shutdown);
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private RedshiftServerlessScheduledActionService reload() {
+ HybridStorage<String, ScheduledAction> backend = new HybridStorage<>(directory.resolve("actions.json"),
+ new TypeReference<Map<String, ScheduledAction>>() {}, 600_000);
+ backends.add(backend);
+ backend.load();
+ AccountAwareStorageBackend<ScheduledAction> actions = new AccountAwareStorageBackend<>(backend, null, ACCOUNT);
+ when(factory.create(eq("redshiftserverless"), eq("redshiftserverless-scheduledactions.json"), any(TypeReference.class)))
+ .thenReturn((AccountAwareStorageBackend) actions);
+ return new RedshiftServerlessScheduledActionService(factory, serverless,
+ mock(RedshiftServerlessScheduledActionAuthorizer.class), regions, Clock.fixed(CREATED, ZoneOffset.UTC));
+ }
+
+ @Test
+ void crashBeforeSnapshotRetriesAfterHybridReloadEvenIfAFlushRanDuringTheAttempt() {
+ doAnswer(invocation -> {
+ backends.getFirst().flush();
+ throw new AssertionError("simulated process crash before snapshot creation");
+ }).when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+
+ assertThrows(AssertionError.class, () -> service.runScheduledActions(DUE));
+ RedshiftServerlessScheduledActionService restarted = reload();
+ assertNull(restarted.getScheduledAction("hourly", REGION).getLastClaimedInvocation());
+ doCallRealMethod().when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ restarted.runScheduledActions(DUE.plusSeconds(60));
+
+ assertEquals(1, snapshots.scan(key -> true).size());
+ assertEquals(SNAPSHOT_NAME, snapshots.scan(key -> true).getFirst().getSnapshotName());
+ assertEquals(DUE, restarted.getScheduledAction("hourly", REGION).getLastCompletedInvocation());
+ }
+
+ @Test
+ void crashAfterSnapshotRetriesTheSameNameWithoutCreatingADuplicate() {
+ doAnswer(invocation -> {
+ invocation.callRealMethod();
+ throw new AssertionError("simulated process crash after snapshot creation");
+ }).when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+
+ assertThrows(AssertionError.class, () -> service.runScheduledActions(DUE));
+ RedshiftServerlessScheduledActionService restarted = reload();
+ assertNull(restarted.getScheduledAction("hourly", REGION).getLastClaimedInvocation());
+ doCallRealMethod().when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ restarted.runScheduledActions(DUE.plusSeconds(60));
+ restarted.runScheduledActions(DUE.plusSeconds(120));
+
+ assertEquals(1, snapshots.scan(key -> true).size());
+ verify(serverless, times(2)).createSnapshot(eq(SNAPSHOT_NAME), eq("source-ns"), any(), any(), eq(REGION));
+ assertEquals(DUE, restarted.getScheduledAction("hourly", REGION).getLastClaimedInvocation());
+ assertTrue(restarted.getScheduledAction("hourly", REGION).getLastFailure().contains("already exists"));
+ }
+
+ @Test
+ void completedCheckpointSurvivesHybridFlushAndReload() {
+ service.runScheduledActions(DUE);
+ backends.getFirst().flush();
+
+ RedshiftServerlessScheduledActionService restarted = reload();
+ restarted.runScheduledActions(DUE.plusSeconds(60));
+
+ assertEquals(DUE, restarted.getScheduledAction("hourly", REGION).getLastCompletedInvocation());
+ verify(serverless).createSnapshot(eq(SNAPSHOT_NAME), eq("source-ns"), any(), any(), eq(REGION));
+ assertEquals(1, snapshots.scan(key -> true).size());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionSchedulerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionSchedulerTest.java
new file mode 100644
index 0000000000..5cb8a88abe
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionSchedulerTest.java
@@ -0,0 +1,115 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class RedshiftServerlessScheduledActionSchedulerTest {
+ private static final Instant NOW = Instant.parse("2030-01-07T02:00:00Z");
+
+ private final RedshiftServerlessScheduledActionService service = mock(RedshiftServerlessScheduledActionService.class);
+
+ private RedshiftServerlessScheduledActionScheduler scheduler(boolean serviceEnabled, boolean actionsEnabled) {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().redshiftServerless().enabled()).thenReturn(serviceEnabled);
+ when(config.services().redshiftServerless().scheduledActionsEnabled()).thenReturn(actionsEnabled);
+ when(config.services().redshiftServerless().scheduledActionSweepIntervalSeconds()).thenReturn(60);
+ return new RedshiftServerlessScheduledActionScheduler(service, config);
+ }
+
+ @Test
+ void workerExistsOnlyWhenTheServiceAndTheSwitchAreBothOn() {
+ assertTrue(scheduler(true, true).active());
+ assertFalse(scheduler(true, false).active());
+ assertFalse(scheduler(false, true).active());
+ }
+
+ @Test
+ void tickRunsTheSweepAtTheGivenInstant() {
+ scheduler(true, false).tick(NOW);
+ verify(service).runScheduledActions(NOW);
+ }
+
+ @Test
+ void resetDrainsARunningSweepAndPausesUntilItEnds() throws Exception {
+ RedshiftServerlessScheduledActionScheduler scheduler = scheduler(true, false);
+ CountDownLatch running = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ doAnswer(invocation -> {
+ running.countDown();
+ release.await(10, TimeUnit.SECONDS);
+ return null;
+ }).when(service).runScheduledActions(NOW);
+ ExecutorService executor = Executors.newFixedThreadPool(2);
+ try {
+ Future<?> sweep = executor.submit(() -> scheduler.tick(NOW));
+ assertTrue(running.await(5, TimeUnit.SECONDS));
+ Future<?> reset = executor.submit(scheduler::beforeReset);
+ assertThrows(TimeoutException.class, () -> reset.get(300, TimeUnit.MILLISECONDS));
+
+ release.countDown();
+ reset.get(5, TimeUnit.SECONDS);
+ sweep.get(5, TimeUnit.SECONDS);
+ scheduler.tick(NOW.plusSeconds(60));
+ verify(service, never()).runScheduledActions(NOW.plusSeconds(60));
+
+ scheduler.afterReset();
+ scheduler.tick(NOW.plusSeconds(120));
+ verify(service).runScheduledActions(NOW.plusSeconds(120));
+ } finally {
+ release.countDown();
+ executor.shutdownNow();
+ }
+ }
+
+ @Test
+ void overlappingTicksNeverRunTheSweepConcurrently() throws Exception {
+ RedshiftServerlessScheduledActionScheduler scheduler = scheduler(true, false);
+ AtomicInteger concurrent = new AtomicInteger();
+ AtomicInteger peak = new AtomicInteger();
+ doAnswer(invocation -> {
+ peak.accumulateAndGet(concurrent.incrementAndGet(), Math::max);
+ Thread.sleep(100);
+ concurrent.decrementAndGet();
+ return null;
+ }).when(service).runScheduledActions(NOW);
+ ExecutorService executor = Executors.newFixedThreadPool(3);
+ try {
+ Future<?> first = executor.submit(() -> scheduler.tick(NOW));
+ Future<?> second = executor.submit(() -> scheduler.tick(NOW));
+ Future<?> third = executor.submit(() -> scheduler.tick(NOW));
+ first.get(5, TimeUnit.SECONDS);
+ second.get(5, TimeUnit.SECONDS);
+ third.get(5, TimeUnit.SECONDS);
+ } finally {
+ executor.shutdownNow();
+ }
+ assertEquals(1, peak.get());
+ }
+
+ @Test
+ void aFailedSweepReleasesTheLock() {
+ RedshiftServerlessScheduledActionScheduler scheduler = scheduler(true, false);
+ doThrow(new IllegalStateException("boom")).when(service).runScheduledActions(NOW);
+ assertThrows(IllegalStateException.class, () -> scheduler.tick(NOW));
+ scheduler.tick(NOW.plusSeconds(60));
+ verify(service).runScheduledActions(NOW.plusSeconds(60));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionServiceTest.java
new file mode 100644
index 0000000000..2e1ad39e83
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionServiceTest.java
@@ -0,0 +1,354 @@
+package io.github.hectorvent.floci.services.redshiftserverless;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.InMemoryStorage;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.redshiftserverless.model.Namespace;
+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.time.temporal.ChronoUnit;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.ArgumentMatchers.startsWith;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class RedshiftServerlessScheduledActionServiceTest {
+ private static final String REGION = "us-east-1";
+ private static final String ACCOUNT_ID = "123456789012";
+ private static final String ROLE_ARN = "arn:aws:iam::" + ACCOUNT_ID + ":role/scheduler";
+ private static final String NAMESPACE_ARN = "arn:aws:redshift-serverless:us-east-1:" + ACCOUNT_ID + ":namespace/ns-id";
+ private static final Instant BASE_TIME = Instant.parse("2026-10-28T12:15:00Z");
+
+ private final AtomicBoolean failCheckpointWrites = new AtomicBoolean();
+ private AccountAwareStorageBackend<ScheduledAction> store;
+ private StorageFactory storageFactory;
+ private RedshiftServerlessService serverless;
+ private RedshiftServerlessScheduledActionAuthorizer authorizer;
+ private RegionResolver regionResolver;
+ private String namespaceId;
+ private Clock clock;
+ private RedshiftServerlessScheduledActionService service;
+
+ @BeforeEach
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ void setUp() {
+ namespaceId = "ns-id";
+ clock = Clock.fixed(BASE_TIME, ZoneOffset.UTC);
+ store = new AccountAwareStorageBackend<>(new InMemoryStorage<String, ScheduledAction>() {
+ @Override
+ public synchronized void putAllAtomically(Map<String, ScheduledAction> entries) {
+ if (failCheckpointWrites.get()) {
+ throw new UncheckedIOException(new IOException("disk full"));
+ }
+ super.putAllAtomically(entries);
+ }
+ }, null, ACCOUNT_ID);
+ storageFactory = mock(StorageFactory.class);
+ when(storageFactory.create(eq("redshiftserverless"), eq("redshiftserverless-scheduledactions.json"),
+ any(TypeReference.class))).thenReturn((AccountAwareStorageBackend) store);
+ serverless = mock(RedshiftServerlessService.class);
+ when(serverless.namespaceExists(anyString(), eq(REGION))).thenReturn(true);
+ when(serverless.findNamespace(anyString(), eq(REGION))).thenAnswer(invocation -> Optional.of(namespace()));
+ authorizer = mock(RedshiftServerlessScheduledActionAuthorizer.class);
+ regionResolver = mock(RegionResolver.class);
+ when(regionResolver.getAccountId()).thenReturn(ACCOUNT_ID);
+ service = newService();
+ }
+
+ private RedshiftServerlessScheduledActionService newService() {
+ return new RedshiftServerlessScheduledActionService(storageFactory, serverless, authorizer, regionResolver, clock);
+ }
+
+ private Namespace namespace() {
+ Namespace namespace = new Namespace();
+ namespace.setNamespaceId(namespaceId);
+ namespace.setNamespaceArn(NAMESPACE_ARN);
+ return namespace;
+ }
+
+ private ScheduledAction request(String name) {
+ ScheduledAction request = new ScheduledAction();
+ request.setScheduledActionName(name);
+ request.setNamespaceName("source-ns");
+ request.setRoleArn(ROLE_ARN);
+ request.setScheduleCron("0 * * * ? *");
+ request.setTargetNamespaceName("source-ns");
+ request.setSnapshotNamePrefix("Nightly");
+ request.setRetentionPeriod(7);
+ Map<String, String> tags = new LinkedHashMap<>();
+ tags.put("env", "dev");
+ request.setSnapshotTags(tags);
+ return request;
+ }
+
+ private static Instant later() {
+ return BASE_TIME.plus(Duration.ofHours(3));
+ }
+
+ private static String expectedName(Instant tick) {
+ return RedshiftServerlessScheduledActionService.snapshotName("nightly", tick.truncatedTo(ChronoUnit.HOURS));
+ }
+
+ @Test
+ void cronActionTakesTheLatestDueSnapshotOnceWithItsArguments() {
+ service.createScheduledAction(request("hourly"), REGION);
+ Instant tick = later();
+
+ service.runScheduledActions(tick);
+ service.runScheduledActions(tick.plusSeconds(60));
+
+ Map<String, String> tags = new LinkedHashMap<>();
+ tags.put("env", "dev");
+ verify(serverless, times(1)).createSnapshot(expectedName(tick), "source-ns", 7, tags, REGION);
+ assertEquals(tick.truncatedTo(ChronoUnit.HOURS),
+ store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastCompletedInvocation());
+ }
+
+ @Test
+ void missedInvocationsAreCoalescedIntoTheLatestOne() {
+ service.createScheduledAction(request("hourly"), REGION);
+
+ service.runScheduledActions(BASE_TIME.plus(Duration.ofHours(30)));
+
+ verify(serverless, times(1)).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ }
+
+ @Test
+ void oneOffActionRunsOnceAtItsTime() {
+ ScheduledAction oneOff = request("once");
+ oneOff.setScheduleCron(null);
+ Instant at = BASE_TIME.plusSeconds(600);
+ oneOff.setScheduleAt(at);
+ service.createScheduledAction(oneOff, REGION);
+
+ service.runScheduledActions(at.minusSeconds(1));
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ service.runScheduledActions(at.plusSeconds(5));
+ service.runScheduledActions(at.plusSeconds(3600));
+
+ verify(serverless, times(1)).createSnapshot(eq(RedshiftServerlessScheduledActionService.snapshotName("nightly", at)),
+ eq("source-ns"), eq(7), any(), eq(REGION));
+ }
+
+ @Test
+ void activeWindowAndDisabledActionsDoNotRun() {
+ ScheduledAction early = request("windowed");
+ early.setStartTime(BASE_TIME.plus(Duration.ofHours(10)));
+ service.createScheduledAction(early, REGION);
+ ScheduledAction disabled = request("off");
+ disabled.setEnabled(false);
+ service.createScheduledAction(disabled, REGION);
+
+ service.runScheduledActions(later());
+
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ }
+
+ @Test
+ void enablingDoesNotReplayWhatWasMissedWhileDisabled() {
+ ScheduledAction disabled = request("off");
+ disabled.setEnabled(false);
+ service.createScheduledAction(disabled, REGION);
+ ScheduledAction enable = new ScheduledAction();
+ enable.setScheduledActionName("off");
+ enable.setEnabled(true);
+ service.updateScheduledAction(enable, REGION);
+
+ service.runScheduledActions(BASE_TIME.plusSeconds(60));
+
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ }
+
+ @Test
+ void storedActionWithoutACheckpointStartsCountingWithoutReplay() {
+ ScheduledAction legacy = new ScheduledAction(request("legacy"));
+ legacy.setScheduledActionUuid("u-1");
+ legacy.setOwnerId(namespaceId);
+ legacy.setEnabled(true);
+ store.put("us-east-1::legacy", legacy);
+ Instant tick = later();
+
+ service.runScheduledActions(tick);
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ service.runScheduledActions(tick.plus(Duration.ofHours(2)));
+
+ verify(serverless, times(1)).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ }
+
+ @Test
+ void oneFailingActionDoesNotStopOthersAndIsNotRetried() {
+ service.createScheduledAction(request("a-failing"), REGION);
+ ScheduledAction other = request("b-healthy");
+ other.setSnapshotNamePrefix("healthy");
+ service.createScheduledAction(other, REGION);
+ Instant tick = later();
+ when(serverless.createSnapshot(anyString(), anyString(), any(), any(), anyString())).thenAnswer(invocation -> {
+ if (invocation.getArgument(0, String.class).startsWith("nightly")) {
+ throw ServerlessSupport.conflict("exists");
+ }
+ return null;
+ });
+
+ service.runScheduledActions(tick);
+ service.runScheduledActions(tick.plusSeconds(30));
+
+ verify(serverless, times(1)).createSnapshot(startsWith("nightly"), anyString(), any(), any(), anyString());
+ verify(serverless, times(1)).createSnapshot(startsWith("healthy"), anyString(), any(), any(), anyString());
+ assertNotNull(store.getForAccount(ACCOUNT_ID, "us-east-1::a-failing").orElseThrow().getLastFailure());
+ }
+
+ @Test
+ void failedCheckpointLeavesTheInvocationEligibleForRetry() {
+ service.createScheduledAction(request("hourly"), REGION);
+ failCheckpointWrites.set(true);
+
+ service.runScheduledActions(later());
+
+ assertNull(store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastClaimedInvocation());
+ failCheckpointWrites.set(false);
+ service.runScheduledActions(later().plusSeconds(60));
+
+ verify(serverless, times(2)).createSnapshot(eq(expectedName(later())), anyString(), any(), any(), eq(REGION));
+ assertEquals(later().truncatedTo(ChronoUnit.HOURS),
+ store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastCompletedInvocation());
+ }
+
+ @Test
+ void claimedInvocationIsNotRepeatedAfterRestart() {
+ service.createScheduledAction(request("hourly"), REGION);
+ Instant tick = later();
+ service.runScheduledActions(tick);
+
+ RedshiftServerlessScheduledActionService restarted = newService();
+ restarted.runScheduledActions(tick.plusSeconds(60));
+
+ verify(serverless, times(1)).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ }
+
+ @Test
+ void recreatedNamespaceInvalidatesTheOldAction() {
+ service.createScheduledAction(request("hourly"), REGION);
+ namespaceId = "ns-id-2";
+
+ service.runScheduledActions(later());
+
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ assertTrue(store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").isEmpty());
+ }
+
+ @Test
+ void deniedRoleTakesNoSnapshotAndIsNotRetried() {
+ service.createScheduledAction(request("hourly"), REGION);
+ when(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, NAMESPACE_ARN)).thenReturn(Optional.of("denied"));
+ Instant tick = later();
+
+ service.runScheduledActions(tick);
+ service.runScheduledActions(tick.plusSeconds(60));
+
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ verify(authorizer, times(1)).refusal(ROLE_ARN, ACCOUNT_ID, NAMESPACE_ARN);
+ assertEquals("denied", store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastFailure());
+ }
+
+ @Test
+ void anotherAccountsActionIsAuthorizedAsItsOwnAccount() {
+ ScheduledAction foreign = new ScheduledAction(request("foreign"));
+ foreign.setScheduledActionUuid("u-2");
+ foreign.setOwnerId(namespaceId);
+ foreign.setEnabled(true);
+ foreign.setRoleArn("arn:aws:iam::210987654321:role/scheduler");
+ foreign.setActivatedAt(BASE_TIME);
+ store.putAllAtomicallyForAccount("210987654321", Map.of("us-east-1::foreign", foreign));
+ when(authorizer.refusal(anyString(), eq("210987654321"), anyString())).thenReturn(Optional.of("denied"));
+
+ service.runScheduledActions(later());
+
+ verify(authorizer).refusal(eq("arn:aws:iam::210987654321:role/scheduler"), eq("210987654321"), anyString());
+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());
+ }
+
+ @Test
+ void readsAndMutationsOfOtherActionsAreNotBlockedByARunningDump() throws Exception {
+ service.createScheduledAction(request("hourly"), REGION);
+ ScheduledAction other = request("other");
+ other.setScheduleCron("0 0 1 1 ? *");
+ service.createScheduledAction(other, REGION);
+ CountDownLatch dumping = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ when(serverless.createSnapshot(anyString(), anyString(), any(), any(), anyString())).thenAnswer(invocation -> {
+ dumping.countDown();
+ release.await(10, TimeUnit.SECONDS);
+ return null;
+ });
+ ExecutorService executor = Executors.newFixedThreadPool(2);
+ try {
+ Future<?> sweep = executor.submit(() -> service.runScheduledActions(later()));
+ assertTrue(dumping.await(5, TimeUnit.SECONDS));
+
+ Future<ScheduledAction> read = executor.submit(() -> service.getScheduledAction("hourly", REGION));
+ Future<ScheduledAction> mutation = executor.submit(() -> {
+ ScheduledAction change = new ScheduledAction();
+ change.setScheduledActionName("other");
+ change.setDescription("changed");
+ return service.updateScheduledAction(change, REGION);
+ });
+ assertEquals("hourly", read.get(5, TimeUnit.SECONDS).getScheduledActionName());
+ assertEquals("changed", mutation.get(5, TimeUnit.SECONDS).getDescription());
+ release.countDown();
+ sweep.get(5, TimeUnit.SECONDS);
+ } finally {
+ release.countDown();
+ executor.shutdownNow();
+ }
+ }
+
+ @Test
+ void snapshotNameStaysWithinTheNamingRules() {
+ String name = RedshiftServerlessScheduledActionService.snapshotName("Nightly", Instant.parse("2030-01-07T03:00:00Z"));
+ assertEquals("nightly-2030-01-07-03-00-00", name);
+ assertTrue(RedshiftServerlessScheduledActionService.snapshotName("a".repeat(400), BASE_TIME).length() <= 255);
+ }
+
+ @Test
+ void unusableSnapshotNamePrefixIsRejectedOnCreate() {
+ ScheduledAction invalid = request("invalid-prefix");
+ invalid.setSnapshotNamePrefix("nightly_backup");
+ assertThrows(AwsException.class, () -> service.createScheduledAction(invalid, REGION));
+ }
+
+ @Test
+ void unusableSnapshotNamePrefixIsRejectedOnUpdate() {
+ service.createScheduledAction(request("valid-prefix"), REGION);
+ ScheduledAction update = new ScheduledAction();
+ update.setScheduledActionName("valid-prefix");
+ update.setSnapshotNamePrefix("nightly_backup");
+ assertThrows(AwsException.class, () -> service.updateScheduledAction(update, REGION));
+ }
+}
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index 2e56a97e07..198f0f66c9 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -251,6 +251,8 @@ floci:
shutdown-timeout-seconds: 10
redshift-serverless:
enabled: true
+ scheduled-actions-enabled: false
+ scheduled-action-sweep-interval-seconds: 60
neptune:
enabled: true
proxy-base-port: 8182