Light, fluffy, and always free - The AWS Local Emulator alternative
feat(redshift-serverless): execute scheduled actions on cron and at schedules (#6851)
Vũ Thế Huye820414f2257
diff --git a/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java b/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.javaindex 6a2aad9a5d..c64760048f 100644--- a/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java+++ b/compatibility-tests/sdk-test-java/src/test/java/com/floci/test/RedshiftServerlessTest.java@@ -504,6 +504,15 @@ class RedshiftServerlessTest {.build())).scheduledAction();assertThat(action.state().toString()).isEqualTo("ACTIVE");assertThat(action.schedule().cron()).isEqualTo("cron(0 3 * * ? *)");+ assertThat(action.nextInvocations()).hasSize(1);+ assertThatThrownBy(() -> client.createScheduledAction(request -> request+ .scheduledActionName("floci-compat-too-often").namespaceName(namespaceName)+ .roleArn("arn:aws:iam::000000000000:role/r")+ .schedule(Schedule.builder().cron("cron(0,30 * * * ? *)").build())+ .targetAction(TargetAction.builder().createSnapshot(CreateSnapshotScheduleActionParameters+ .builder().namespaceName(namespaceName).snapshotNamePrefix("nightly").build())+ .build())))+ .isInstanceOf(ValidationException.class);client.deleteScheduledAction(request -> request.scheduledActionName("floci-compat-sched"));} finally {deleteBestEffort(client, namespaceName);diff --git a/docs/services/redshift-serverless.md b/docs/services/redshift-serverless.mdindex 59fb4b1cbd..2eaaaf7e26 100644--- a/docs/services/redshift-serverless.md+++ b/docs/services/redshift-serverless.md@@ -62,7 +62,7 @@ For the upstream API shape, see the [Amazon Redshift Serverless API Reference](h| `ListSnapshotCopyConfigurations` | Page through the snapshot copy configurations, filtered by `namespaceName` || `UpdateSnapshotCopyConfiguration` | Change the `snapshotRetentionPeriod` of a snapshot copy configuration || `DeleteSnapshotCopyConfiguration` | Remove a snapshot copy configuration and return it |-| `CreateScheduledAction` | Record a scheduled `createSnapshot` action; it is stored, never run |+| `CreateScheduledAction` | Create a scheduled `createSnapshot` action that Floci runs on its schedule || `GetScheduledAction` | Return a scheduled action by name || `ListScheduledActions` | Page through the scheduled actions, filtered by `namespaceName` || `UpdateScheduledAction` | Apply the supplied members to a scheduled action; `schedule` and `targetAction` replace the stored ones |@@ -108,6 +108,8 @@ AWS also models `AccessDeniedException`, `InternalServerException`, and `Throttl| Variable | Default | Description ||---|---|---|| `FLOCI_SERVICES_REDSHIFT_SERVERLESS_ENABLED` | `true` | Enable or disable Redshift Serverless |+| `FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTIONS_ENABLED` | `true` | Run the background worker that executes scheduled actions |+| `FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTION_SWEEP_INTERVAL_SECONDS` | `60` | Delay between scheduled action sweeps |Workgroup containers and proxies use the `floci.services.redshift.*` settings: `image-version`, `proxy-base-port` and `proxy-max-port`, `endpoint-host`, `docker-network` and the proxy connection limits.@@ -145,4 +147,4 @@ aws redshift-serverless delete-namespace --namespace-name analytics- **Endpoint access is recorded, not acted on.** An endpoint is stored per account and Region, echoed in the shape of the API model, and dropped on read once its workgroup is gone; a workgroup recreated under the same name is a different workgroup and does not inherit it. `CreateEndpointAccess` creates no VPC endpoint. `address` and `port` are the workgroup's own, so a client can connect; `vpcEndpoint` carries a generated id and one network interface per subnet, without a `vpcId`, and `ListEndpointAccess` with a `vpcId` returns nothing. `vpcSecurityGroupIds` is optional. Endpoint names are 1 to 30 letters, digits or hyphens, start with a letter, and contain no consecutive or trailing hyphens. Security groups are the only member `UpdateEndpointAccess` changes.- **Custom domain associations are recorded, not acted on.** An association is stored per account and Region and dropped on read once its workgroup is gone; a workgroup recreated under the same name does not inherit it. A workgroup has at most one custom domain association. The certificate expiry time comes from ACM when the certificate exists in the emulator.- **Snapshot copy configurations are recorded, not acted on.** `CreateSnapshotCopyConfiguration` copies no snapshot. A configuration is stored per account and Region and dropped on read once its namespace is gone; a namespace recreated under the same name does not inherit it. The destination Region must differ from the namespace's, and a namespace has only one configuration at a time (`ConflictException`). Delete the existing configuration before choosing another destination Region.-- **Scheduled actions are recorded, never run.** An action is stored per account and Region and dropped on read once its namespace is gone; a namespace recreated under the same name does not inherit it. The owning `namespaceName` and the `targetAction.createSnapshot.namespaceName` must each exist but are not required to match, as the API reference does not say they must. `schedule` holds exactly one of `at` and `cron`; a one-off `at` in the future is the only `nextInvocations` reported, because cron expressions are not evaluated. `startTime`, `endTime` and `at` are epoch seconds.+- **Scheduled actions run.** A background sweep (every `FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTION_SWEEP_INTERVAL_SECONDS`, 60 by default) takes the `createSnapshot` of each enabled action whose `at` or six-field `cron` schedule came due, in UTC and inside the `startTime` to `endTime` window. A cron expression may be wrapped as `cron(...)` and must fire at most once an hour; AWS day-of-week numbering applies (1 is Sunday). The snapshot is named `<snapshotNamePrefix>-yyyy-MM-dd-HH-mm-ss` after the scheduled time (lowercased) and keeps the action's `retentionPeriod` and tags. `nextInvocations` reports the next eligible time for both schedule types, and is empty once none remains. Floci records the invocation checkpoint with a normal atomic write after attempting the snapshot. A failure (an existing snapshot of that name, a missing namespace, a refused role) is logged and skipped while that checkpoint is retained. A crash before the checkpoint write, or before a hybrid flush, can cause the invocation to be retried; the scheduled-time snapshot name and the existing-name check prevent a duplicate snapshot. This is not exactly-once delivery. After downtime only the latest due time runs, and nothing before the action was created, rescheduled or re-enabled is replayed. That is a Floci choice, not a statement about AWS. Persistent and WAL storage commit checkpoints synchronously; hybrid storage persists them on its normal periodic flush. In memory mode actions are lost on restart. With IAM enforcement on, `roleArn` must be a role of the action's account that trusts `scheduler.redshift.amazonaws.com` and allows `redshift-serverless:CreateSnapshot` on the target namespace; otherwise nothing is created. An action is stored per account and Region and dropped on read once its namespace is gone; a namespace recreated under the same name does not inherit it. The owning `namespaceName` and the `targetAction.createSnapshot.namespaceName` must each exist but are not required to match. `startTime`, `endTime` and `at` are epoch seconds. Recovery points, snapshot expiration and cross-Region copy are not part of a scheduled action.diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex b3cad3fec4..6ed7923c09 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1908,6 +1908,12 @@ public interface EmulatorConfig {interface RedshiftServerlessServiceConfig {@WithDefault("true")boolean enabled();++ @WithDefault("true")+ boolean scheduledActionsEnabled();++ @WithDefault("60")+ int scheduledActionSweepIntervalSeconds();}interface NeptuneServiceConfig {diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.javaindex 8f9b4a82e5..69765a0e00 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdminJsonHandler.java@@ -344,7 +344,7 @@ public class RedshiftServerlessAdminJsonHandler {return Response.ok(response).build();}- /** A one-off schedule reports its own time while it is ahead and within its active window on an enabled action; a cron schedule reports none, as Floci never computes them. */+ /** Reports the next invocation inside the active window of an enabled action, for both schedule types. */private ObjectNode scheduledActionNode(ScheduledAction action) {ObjectNode node = objectMapper.createObjectNode();if (action.getEndTime() != null) {@@ -352,13 +352,7 @@ public class RedshiftServerlessAdminJsonHandler {}node.put("namespaceName", action.getNamespaceName());ArrayNode invocations = node.putArray("nextInvocations");- if (action.getScheduleAt() != null- && !Boolean.FALSE.equals(action.getEnabled())- && action.getScheduleAt().isAfter(Instant.now())- && (action.getStartTime() == null || !action.getScheduleAt().isBefore(action.getStartTime()))- && (action.getEndTime() == null || !action.getScheduleAt().isAfter(action.getEndTime()))) {- invocations.add(epoch(action.getScheduleAt()));- }+ RedshiftServerlessSchedule.next(action, Instant.now()).ifPresent(time -> invocations.add(epoch(time)));node.put("roleArn", action.getRoleArn());ObjectNode schedule = node.putObject("schedule");if (action.getScheduleAt() != null) {diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessSchedule.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessSchedule.javanew file mode 100644index 0000000000..d2de256992--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessSchedule.java@@ -0,0 +1,111 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import com.cronutils.model.Cron;+import com.cronutils.model.time.ExecutionTime;+import io.github.hectorvent.floci.core.common.AwsCronDefinitions;+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;++import java.time.Duration;+import java.time.Instant;+import java.time.ZoneOffset;+import java.util.Optional;++/** UTC schedule rules of a Serverless scheduled action: one-off {@code at} or six-field AWS {@code cron}. */+final class RedshiftServerlessSchedule {+ private static final Duration MINIMUM_INTERVAL = Duration.ofHours(1);+ private static final Duration SPACING_HORIZON = Duration.ofDays(400);+ private static final int MAX_SPACING_CHECKS = 1000;++ private RedshiftServerlessSchedule() { }++ /** Rejects a cron that does not parse or fires more often than hourly; an {@code at} schedule always passes. */+ static void validate(ScheduledAction action) {+ if (action.getScheduleCron() == null || action.getScheduleCron().isBlank()) {+ return;+ }+ ExecutionTime cron = parse(action.getScheduleCron());+ Optional<Instant> first = nextCron(cron, Instant.EPOCH);+ if (first.isEmpty()) {+ return;+ }+ Instant cursor = first.get().minusSeconds(1);+ Instant limit = cursor.plus(SPACING_HORIZON);+ Instant previous = null;+ for (int i = 0; i < MAX_SPACING_CHECKS; i++) {+ Optional<Instant> next = nextCron(cron, cursor);+ if (next.isEmpty() || next.get().isAfter(limit)) {+ return;+ }+ if (previous != null && Duration.between(previous, next.get()).compareTo(MINIMUM_INTERVAL) < 0) {+ throw ServerlessSupport.validation("The cron schedule must be at least one hour apart.");+ }+ previous = next.get();+ cursor = next.get();+ }+ }++ /** The first invocation strictly after {@code after} that is enabled and inside the active window. */+ static Optional<Instant> next(ScheduledAction action, Instant after) {+ if (Boolean.FALSE.equals(action.getEnabled())) {+ return Optional.empty();+ }+ Instant candidate;+ if (action.getScheduleAt() != null) {+ candidate = action.getScheduleAt().isAfter(after) ? action.getScheduleAt() : null;+ } else {+ Instant from = action.getStartTime() != null && action.getStartTime().isAfter(after)+ ? action.getStartTime().minusNanos(1) : after;+ candidate = nextCron(parse(action.getScheduleCron()), from).orElse(null);+ }+ return Optional.ofNullable(candidate).filter(time -> inWindow(action, time));+ }++ /** The latest invocation in {@code (anchor, now]} that is inside the window; older ones are coalesced away. */+ static Optional<Instant> latestDue(ScheduledAction action, Instant anchor, Instant now) {+ if (Boolean.FALSE.equals(action.getEnabled())) {+ return Optional.empty();+ }+ Instant upper = action.getEndTime() != null && action.getEndTime().isBefore(now) ? action.getEndTime() : now;+ Optional<Instant> latest;+ if (action.getScheduleAt() != null) {+ latest = Optional.of(action.getScheduleAt()).filter(time -> !time.isAfter(upper));+ } else {+ latest = parse(action.getScheduleCron()).lastExecution(upper.atZone(ZoneOffset.UTC).plusNanos(1))+ .map(value -> value.toInstant());+ }+ return latest.filter(time -> time.isAfter(anchor) && inWindow(action, time));+ }++ private static boolean inWindow(ScheduledAction action, Instant time) {+ return (action.getStartTime() == null || !time.isBefore(action.getStartTime()))+ && (action.getEndTime() == null || !time.isAfter(action.getEndTime()));+ }++ private static Optional<Instant> nextCron(ExecutionTime cron, Instant after) {+ return cron.nextExecution(after.atZone(ZoneOffset.UTC)).map(value -> value.toInstant());+ }++ private static ExecutionTime parse(String expression) {+ String body = expression.trim();+ if (body.startsWith("cron")) {+ body = body.substring(4).trim();+ }+ if (body.startsWith("(") && body.endsWith(")")) {+ body = body.substring(1, body.length() - 1).trim();+ }+ String[] fields = body.split("\\s+");+ if (fields.length != 6) {+ throw ServerlessSupport.validation("The cron schedule needs six fields.");+ }+ if ("?".equals(fields[2]) == "?".equals(fields[4])) {+ throw ServerlessSupport.validation("Use ? in exactly one of day-of-month and day-of-week.");+ }+ try {+ Cron cron = AwsCronDefinitions.newParser().parse("0 " + String.join(" ", fields));+ cron.validate();+ return ExecutionTime.forCron(cron);+ } catch (IllegalArgumentException e) {+ throw ServerlessSupport.validation("The cron schedule is invalid: " + e.getMessage());+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizer.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizer.javanew file mode 100644index 0000000000..781c71a272--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizer.java@@ -0,0 +1,103 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsArnUtils;+import io.github.hectorvent.floci.core.common.IamConditionContextResolver;+import io.github.hectorvent.floci.core.common.ServicePrincipals;+import io.github.hectorvent.floci.services.iam.AssumeRolePolicyEvaluator;+import io.github.hectorvent.floci.services.iam.IamPolicyEvaluator;+import io.github.hectorvent.floci.services.iam.IamPolicyEvaluator.Decision;+import io.github.hectorvent.floci.services.iam.IamService;+import io.github.hectorvent.floci.services.iam.ScpProvider;+import io.github.hectorvent.floci.services.iam.model.CallerContext;+import io.github.hectorvent.floci.services.iam.model.IamRole;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.inject.Instance;+import jakarta.inject.Inject;++import java.util.List;+import java.util.Map;+import java.util.Optional;++/**+ * Decides whether the execution role of a scheduled action may take its snapshot. The role has to+ * trust the Redshift scheduler principal and its identity policies, boundary and SCPs have to allow+ * {@code redshift-serverless:CreateSnapshot} on the target. Only active with IAM enforcement on, as+ * for every other API call. A role of another account is refused, because the snapshot would+ * otherwise be taken on the strength of a role the action's account does not own.+ */+@ApplicationScoped+public class RedshiftServerlessScheduledActionAuthorizer {++ static final String CREATE_SNAPSHOT = "redshift-serverless:CreateSnapshot";+ private static final String SCHEDULER_PRINCIPAL = ServicePrincipals.of("scheduler.redshift");++ private final EmulatorConfig config;+ private final IamService iamService;+ private final IamPolicyEvaluator evaluator;+ private final AssumeRolePolicyEvaluator trustEvaluator;+ private final Instance<ScpProvider> scpProvider;++ @Inject+ public RedshiftServerlessScheduledActionAuthorizer(EmulatorConfig config, IamService iamService,+ IamPolicyEvaluator evaluator,+ AssumeRolePolicyEvaluator trustEvaluator,+ Instance<ScpProvider> scpProvider) {+ this.config = config;+ this.iamService = iamService;+ this.evaluator = evaluator;+ this.trustEvaluator = trustEvaluator;+ this.scpProvider = scpProvider;+ }++ /**+ * Returns why {@code roleArn} may not create a snapshot of {@code targetArn}, or nothing when it may.+ *+ * @param accountId the account that owns the action+ */+ public Optional<String> refusal(String roleArn, String accountId, String targetArn) {+ if (!config.services().iam().enforcementEnabled()) {+ return Optional.empty();+ }+ AwsArnUtils.Arn parsed;+ try {+ parsed = AwsArnUtils.parse(roleArn);+ } catch (IllegalArgumentException e) {+ return Optional.of("The execution role " + roleArn + " is not a valid ARN.");+ }+ if (!"iam".equals(parsed.service()) || !parsed.resource().startsWith("role/")) {+ return Optional.of("The execution role " + roleArn + " is not an IAM role.");+ }+ if (!accountId.equals(parsed.accountId())) {+ return Optional.of("The execution role " + roleArn + " belongs to another account.");+ }+ String roleName = parsed.resource().substring(parsed.resource().lastIndexOf('/') + 1);+ Optional<IamRole> role = iamService.findRole(accountId, roleName)+ .filter(found -> IamService.roleArnMatches(roleArn, found));+ if (role.isEmpty()) {+ return Optional.of("The execution role " + roleArn + " does not exist.");+ }+ if (!trustEvaluator.allowsService(role.get().getAssumeRolePolicyDocument(), SCHEDULER_PRINCIPAL,+ null, accountId)) {+ return Optional.of("The execution role " + roleArn + " does not trust " + SCHEDULER_PRINCIPAL + ".");+ }+ CallerContext caller = iamService.resolvePrincipalContext(role.get().getArn());+ List<List<String>> scpLevels = scpProvider.isResolvable()+ ? scpProvider.get().effectiveScpLevels(accountId) : null;+ if (scpLevels != null) {+ caller = caller.withScpLevels(scpLevels);+ }+ AwsArnUtils.Arn target = AwsArnUtils.isArn(targetArn) ? AwsArnUtils.parse(targetArn) : null;+ String targetRegion = target != null ? target.region() : null;+ String targetAccount = target != null ? target.accountId() : accountId;+ Map<String, List<String>> context = IamConditionContextResolver.withGlobalContext(+ Map.of("aws:PrincipalArn", List.of(role.get().getArn())),+ targetArn, targetRegion, accountId, targetAccount);+ Decision decision = evaluator.evaluate(caller, List.of(), CREATE_SNAPSHOT, targetArn, context);+ if (decision != Decision.ALLOW) {+ return Optional.of("The execution role " + roleArn + " is not authorized to perform "+ + CREATE_SNAPSHOT + " on " + targetArn + ".");+ }+ return Optional.empty();+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionScheduler.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionScheduler.javanew file mode 100644index 0000000000..5bccc5bfa5--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionScheduler.java@@ -0,0 +1,103 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.Resettable;+import io.quarkus.runtime.ShutdownEvent;+import io.quarkus.runtime.StartupEvent;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.event.Observes;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.time.Instant;+import java.util.concurrent.Executors;+import java.util.concurrent.ScheduledExecutorService;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.locks.ReentrantLock;++/** Drives Serverless scheduled actions outside AWS request threads, one sweep at a time. */+@ApplicationScoped+public class RedshiftServerlessScheduledActionScheduler implements Resettable {+ private static final Logger LOG = Logger.getLogger(RedshiftServerlessScheduledActionScheduler.class);+ private final RedshiftServerlessScheduledActionService service;+ private final ScheduledExecutorService executor;+ private final long intervalSeconds;+ private final ReentrantLock sweepLock = new ReentrantLock();+ private volatile boolean paused;++ @Inject+ public RedshiftServerlessScheduledActionScheduler(RedshiftServerlessScheduledActionService service,+ EmulatorConfig config) {+ EmulatorConfig.RedshiftServerlessServiceConfig serverless = config.services().redshiftServerless();+ this.service = service;+ this.intervalSeconds = Math.max(1, serverless.scheduledActionSweepIntervalSeconds());+ this.executor = serverless.enabled() && serverless.scheduledActionsEnabled()+ ? Executors.newSingleThreadScheduledExecutor(task -> {+ Thread thread = new Thread(task, "redshift-serverless-scheduled-actions");+ thread.setDaemon(true);+ return thread;+ }) : null;+ }++ void onStart(@Observes StartupEvent ignored) {+ if (executor != null) {+ executor.scheduleWithFixedDelay(this::tickSafely, intervalSeconds, intervalSeconds, TimeUnit.SECONDS);+ }+ }++ void onStop(@Observes ShutdownEvent ignored) {+ if (executor != null) {+ executor.shutdownNow();+ try {+ if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {+ LOG.warn("The Redshift Serverless scheduled action worker is still finishing a snapshot during shutdown");+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ LOG.debugv(e, "Interrupted while stopping the Redshift Serverless scheduled action worker");+ }+ }+ }++ /** Whether a background worker exists, which the two configuration switches decide. */+ boolean active() {+ return executor != null;+ }++ public void tick(Instant now) {+ sweepLock.lock();+ try {+ if (!paused) {+ service.runScheduledActions(now);+ }+ } finally {+ sweepLock.unlock();+ }+ }++ /** Drains a running sweep so it cannot write a claim back into storage that reset is about to clear. */+ @Override+ public void beforeReset() {+ paused = true;+ sweepLock.lock();+ sweepLock.unlock();+ }++ @Override+ public void clear() {+ // Claims live with the actions in RedshiftServerlessScheduledActionService storage, which reset wipes.+ }++ @Override+ public void afterReset() {+ paused = false;+ }++ private void tickSafely() {+ try {+ tick(Instant.now());+ } catch (RuntimeException e) {+ LOG.warnv(e, "The Redshift Serverless scheduled action sweep failed");+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.javaindex bf96877a2b..c484821008 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionService.java@@ -3,23 +3,36 @@ package io.github.hectorvent.floci.services.redshiftserverless;import com.fasterxml.jackson.core.type.TypeReference;import io.github.hectorvent.floci.core.common.PaginatedResult;import io.github.hectorvent.floci.core.common.Pagination;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.RequestScopes;import io.github.hectorvent.floci.core.common.Resettable;import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend.AccountEntry;import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.redshiftserverless.model.Namespace;import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;+import org.jboss.logging.Logger;+import java.time.Clock;+import java.time.Instant;+import java.time.ZoneOffset;+import java.time.format.DateTimeFormatter;import java.util.Comparator;import java.util.List;+import java.util.Locale;import java.util.Map;import java.util.Objects;+import java.util.Optional;import java.util.UUID;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.locks.ReentrantLock;+import java.util.function.Supplier;/**- * Redshift Serverless scheduled actions. They are stored and echoed, not run: Floci never takes- * the snapshot on the schedule, and a cron schedule reports no upcoming invocations.+ * Redshift Serverless scheduled actions. A sweep ({@link #runScheduledActions}) takes the snapshot of+ * every action whose schedule came due since it last looked.** <p>Required members, name patterns and lengths come from the AWS model through* {@link RedshiftServerlessRequestShapes}; what the model cannot express (the schedule union, the time@@ -27,44 +40,78 @@ import java.util.UUID;** <p>An action belongs to a namespace but lives in its own store, so deleting the namespace needs* no hook here. Each read checks that the namespace is still there and drops the action otherwise.+ *+ * <p>The sweep records its checkpoint with a normal atomic write after attempting the snapshot.+ * A crash before that write, or before a hybrid flush, can cause the time to be retried. Snapshot names+ * derive from the scheduled time, so an existing snapshot prevents a duplicate on retry. After downtime+ * only the latest due time is attempted, and nothing before an action was created, scheduled again or+ * enabled again is replayed. Memory storage forgets actions on restart.+ *+ * <p>Lock order: the action's lock, then this service's monitor, then the Serverless service's. A+ * mutation and a sweep of the same action hold its lock, so neither sees the other half done; the+ * monitor is never held while a snapshot is dumped, so reads stay available during one.*/@ApplicationScopedpublic class RedshiftServerlessScheduledActionService implements Resettable {+ private static final Logger LOG = Logger.getLogger(RedshiftServerlessScheduledActionService.class);private static final int MAX_RETENTION_DAYS = 3653;+ private static final int MAX_SNAPSHOT_NAME_LENGTH = 255;+ private static final DateTimeFormatter INVOCATION_SUFFIX =+ DateTimeFormatter.ofPattern("yyyy-MM-dd-HH-mm-ss", Locale.ROOT).withZone(ZoneOffset.UTC);private final AccountAwareStorageBackend<ScheduledAction> actions;private final RedshiftServerlessService serverless;+ private final RedshiftServerlessScheduledActionAuthorizer authorizer;+ private final RegionResolver regionResolver;+ private final Clock clock;+ private final ConcurrentHashMap<String, ReentrantLock> actionLocks = new ConcurrentHashMap<>();@Injectpublic RedshiftServerlessScheduledActionService(StorageFactory storageFactory,- RedshiftServerlessService serverless) {+ RedshiftServerlessService serverless,+ RedshiftServerlessScheduledActionAuthorizer authorizer,+ RegionResolver regionResolver,+ Clock clock) {this.actions = storageFactory.create("redshiftserverless", "redshiftserverless-scheduledactions.json",new TypeReference<Map<String, ScheduledAction>>() {});this.serverless = serverless;+ this.authorizer = authorizer;+ this.regionResolver = regionResolver;+ this.clock = clock;+ }++ RedshiftServerlessScheduledActionService(StorageFactory storageFactory,+ RedshiftServerlessService serverless,+ RedshiftServerlessScheduledActionAuthorizer authorizer,+ RegionResolver regionResolver) {+ this(storageFactory, serverless, authorizer, regionResolver, Clock.systemUTC());}/*** Creates the action described by {@code request}. {@code enabled} defaults to true. The target* namespace of the snapshot and the owning namespace must both exist.*/- public synchronized ScheduledAction createScheduledAction(ScheduledAction request, String region) {- requireNamespace(request.getNamespaceName(), region);- validateSchedule(request);- validateTarget(request, region);- validateWindow(request);- if (live(region).stream().anyMatch(existing ->- existing.getScheduledActionName().equals(request.getScheduledActionName()))) {- throw ServerlessSupport.conflict("The scheduled action " + request.getScheduledActionName()- + " already exists.");- }+ public ScheduledAction createScheduledAction(ScheduledAction request, String region) {+ return mutate(region, request.getScheduledActionName(), () -> {+ requireNamespace(request.getNamespaceName(), region);+ validateSchedule(request);+ validateTarget(request, region);+ validateWindow(request);+ if (live(region).stream().anyMatch(existing ->+ existing.getScheduledActionName().equals(request.getScheduledActionName()))) {+ throw ServerlessSupport.conflict("The scheduled action " + request.getScheduledActionName()+ + " already exists.");+ }- ScheduledAction action = new ScheduledAction(request);- action.setOwnerId(namespaceIdOf(request.getNamespaceName(), region));- action.setScheduledActionUuid(UUID.randomUUID().toString());- action.setEnabled(request.getEnabled() == null || request.getEnabled());- actions.put(storageKey(region, action.getScheduledActionName()), action);- return new ScheduledAction(action);+ ScheduledAction action = new ScheduledAction(request);+ action.setOwnerId(namespaceIdOf(request.getNamespaceName(), region));+ action.setScheduledActionUuid(UUID.randomUUID().toString());+ action.setEnabled(request.getEnabled() == null || request.getEnabled());+ action.setActivatedAt(clock.instant());+ actions.put(storageKey(region, action.getScheduledActionName()), action);+ return new ScheduledAction(action);+ });}public synchronized ScheduledAction getScheduledAction(String scheduledActionName, String region) {@@ -86,55 +133,191 @@ public class RedshiftServerlessScheduledActionService implements Resettable {/*** Applies the members {@code changes} carries. The schedule and the target replace the stored- * ones as a whole, because each is a union of which exactly one member may be set.+ * ones as a whole, because each is a union of which exactly one member may be set. A new schedule,+ * or enabling a disabled action, restarts the count from now so nothing earlier is replayed.*/- public synchronized ScheduledAction updateScheduledAction(ScheduledAction changes, String region) {- ScheduledAction current = require(changes.getScheduledActionName(), region);- ScheduledAction updated = new ScheduledAction(current);- if (changes.getRoleArn() != null) {- updated.setRoleArn(changes.getRoleArn());+ public ScheduledAction updateScheduledAction(ScheduledAction changes, String region) {+ return mutate(region, changes.getScheduledActionName(), () -> {+ ScheduledAction current = require(changes.getScheduledActionName(), region);+ ScheduledAction updated = new ScheduledAction(current);+ if (changes.getRoleArn() != null) {+ updated.setRoleArn(changes.getRoleArn());+ }+ if (changes.getDescription() != null) {+ updated.setDescription(changes.getDescription());+ }+ if (changes.getEnabled() != null) {+ if (changes.getEnabled() && Boolean.FALSE.equals(current.getEnabled())) {+ updated.setActivatedAt(clock.instant());+ }+ updated.setEnabled(changes.getEnabled());+ }+ if (changes.getStartTime() != null) {+ updated.setStartTime(changes.getStartTime());+ }+ if (changes.getEndTime() != null) {+ updated.setEndTime(changes.getEndTime());+ }+ if (changes.isScheduleSpecified()) {+ validateSchedule(changes);+ updated.setScheduleAt(changes.getScheduleAt());+ updated.setScheduleCron(changes.getScheduleCron());+ updated.setActivatedAt(clock.instant());+ updated.setLastClaimedInvocation(null);+ }+ boolean changesTarget = changes.isTargetSpecified()+ || changes.getTargetNamespaceName() != null || changes.getSnapshotNamePrefix() != null+ || changes.getRetentionPeriod() != null || changes.getSnapshotTags() != null;+ if (changesTarget) {+ validateTarget(changes, region);+ updated.setTargetNamespaceName(changes.getTargetNamespaceName());+ updated.setSnapshotNamePrefix(changes.getSnapshotNamePrefix());+ updated.setRetentionPeriod(changes.getRetentionPeriod());+ updated.setSnapshotTags(changes.getSnapshotTags());+ }+ validateWindow(updated);+ actions.put(storageKey(region, updated.getScheduledActionName()), updated);+ return new ScheduledAction(updated);+ });+ }++ public ScheduledAction deleteScheduledAction(String scheduledActionName, String region) {+ return mutate(region, scheduledActionName, () -> {+ ScheduledAction existing = new ScheduledAction(require(scheduledActionName, region));+ actions.delete(storageKey(region, scheduledActionName));+ return existing;+ });+ }++ /**+ * Takes the snapshot of every action whose latest due time falls in {@code (checkpoint, now]}. One+ * action's failure never stops the others. A deterministic entry point: the lifecycle scheduler+ * supplies the clock and tests supply a fixed instant.+ */+ public void runScheduledActions(Instant now) {+ for (AccountEntry<ScheduledAction> entry : actions.scanAllAccountEntries(key -> true)) {+ if (Thread.currentThread().isInterrupted()) {+ return;+ }+ String region = regionOf(entry.key());+ if (region == null) {+ continue;+ }+ ReentrantLock lock = lockOf(entry.accountId(), entry.key());+ lock.lock();+ try {+ RequestScopes.runAs(entry.accountId(), region, () -> runScheduledAction(entry, region, now));+ } catch (RuntimeException e) {+ LOG.warnv(e, "Scheduled action {0} in account {1} could not be processed",+ entry.key(), entry.accountId());+ } finally {+ lock.unlock();+ }}- if (changes.getDescription() != null) {- updated.setDescription(changes.getDescription());+ }++ /** Runs under the action's lock, with the action's account and Region as the request scope. */+ private void runScheduledAction(AccountEntry<ScheduledAction> listed, String region, Instant now) {+ String accountId = listed.accountId();+ String key = listed.key();+ ScheduledAction action = actions.getForAccount(accountId, key).orElse(null);+ if (action == null) {+ return;}- if (changes.getEnabled() != null) {- updated.setEnabled(changes.getEnabled());+ if (!isOnSameNamespace(action, region)) {+ actions.deleteForAccount(accountId, key);+ return;}- if (changes.getStartTime() != null) {- updated.setStartTime(changes.getStartTime());+ if (action.getActivatedAt() == null) {+ // A stored action from before scheduling existed starts counting now, without replaying history.+ ScheduledAction activated = new ScheduledAction(action);+ activated.setActivatedAt(now);+ actions.putAllAtomicallyForAccount(accountId, Map.of(key, activated));+ return;}- if (changes.getEndTime() != null) {- updated.setEndTime(changes.getEndTime());+ Instant checkpoint = action.getLastClaimedInvocation() != null+ && action.getLastClaimedInvocation().isAfter(action.getActivatedAt())+ ? action.getLastClaimedInvocation() : action.getActivatedAt();+ Optional<Instant> due = RedshiftServerlessSchedule.latestDue(action, checkpoint, now);+ if (due.isEmpty()) {+ return;}- if (changes.isScheduleSpecified()) {- validateSchedule(changes);- updated.setScheduleAt(changes.getScheduleAt());- updated.setScheduleCron(changes.getScheduleCron());+ Instant scheduled = due.get();+ ScheduledAction outcome = new ScheduledAction(action);+ outcome.setLastClaimedInvocation(scheduled);+ outcome.setLastFailure(null);+ try {+ takeSnapshot(action, accountId, region, scheduled);+ outcome.setLastCompletedInvocation(scheduled);+ } catch (RuntimeException e) {+ LOG.warnv(e, "Scheduled action {0} in account {1} failed at {2}",+ action.getScheduledActionName(), accountId, scheduled);+ outcome.setLastFailure(String.valueOf(e.getMessage()));}- boolean changesTarget = changes.isTargetSpecified()- || changes.getTargetNamespaceName() != null || changes.getSnapshotNamePrefix() != null- || changes.getRetentionPeriod() != null || changes.getSnapshotTags() != null;- if (changesTarget) {- validateTarget(changes, region);- updated.setTargetNamespaceName(changes.getTargetNamespaceName());- updated.setSnapshotNamePrefix(changes.getSnapshotNamePrefix());- updated.setRetentionPeriod(changes.getRetentionPeriod());- updated.setSnapshotTags(changes.getSnapshotTags());+ recordOutcome(accountId, key, action.getScheduledActionUuid(), outcome);+ }++ private void takeSnapshot(ScheduledAction action, String accountId, String region, Instant scheduled) {+ Namespace target = serverless.findNamespace(action.getTargetNamespaceName(), region).orElseThrow(() ->+ ServerlessSupport.notFound("The namespace " + action.getTargetNamespaceName() + " was not found."));+ Optional<String> refusal = authorizer.refusal(action.getRoleArn(), accountId, target.getNamespaceArn());+ if (refusal.isPresent()) {+ throw ServerlessSupport.validation(refusal.get());}- validateWindow(updated);- actions.put(storageKey(region, updated.getScheduledActionName()), updated);- return new ScheduledAction(updated);+ serverless.createSnapshot(snapshotName(action.getSnapshotNamePrefix(), scheduled),+ action.getTargetNamespaceName(), action.getRetentionPeriod(), action.getSnapshotTags(), region);}- public synchronized ScheduledAction deleteScheduledAction(String scheduledActionName, String region) {- ScheduledAction existing = new ScheduledAction(require(scheduledActionName, region));- actions.delete(storageKey(region, scheduledActionName));- return existing;+ /** Stores the checkpoint and outcome only for the action that was run, not for a recreated one. */+ private void recordOutcome(String accountId, String key, String uuid, ScheduledAction outcome) {+ try {+ ScheduledAction current = actions.getForAccount(accountId, key).orElse(null);+ if (current != null && Objects.equals(current.getScheduledActionUuid(), uuid)) {+ ScheduledAction recorded = new ScheduledAction(current);+ recorded.setLastClaimedInvocation(outcome.getLastClaimedInvocation());+ recorded.setLastCompletedInvocation(outcome.getLastCompletedInvocation());+ recorded.setLastFailure(outcome.getLastFailure());+ actions.putAllAtomicallyForAccount(accountId, Map.of(key, recorded));+ }+ } catch (RuntimeException e) {+ LOG.warnv(e, "The outcome of scheduled action {0} in account {1} could not be stored", key, accountId);+ }+ }++ static String snapshotName(String prefix, Instant scheduled) {+ String suffix = "-" + INVOCATION_SUFFIX.format(scheduled);+ String base = prefix == null ? "" : prefix.toLowerCase(Locale.ROOT);+ if (base.length() + suffix.length() > MAX_SNAPSHOT_NAME_LENGTH) {+ base = base.substring(0, MAX_SNAPSHOT_NAME_LENGTH - suffix.length());+ }+ return base + suffix;}@Overridepublic void clear() {actions.clear();+ actionLocks.clear();+ }++ private <T> T mutate(String region, String name, Supplier<T> body) {+ ReentrantLock lock = lockOf(regionResolver.getAccountId(), storageKey(region, name));+ lock.lock();+ try {+ synchronized (this) {+ return body.get();+ }+ } finally {+ lock.unlock();+ }+ }++ private ReentrantLock lockOf(String accountId, String key) {+ return actionLocks.computeIfAbsent(accountId + "|" + key, ignored -> new ReentrantLock());+ }++ private static String regionOf(String key) {+ int separator = key.indexOf("::");+ return separator < 0 ? null : key.substring(0, separator);}private ScheduledAction require(String scheduledActionName, String region) {@@ -180,6 +363,7 @@ public class RedshiftServerlessScheduledActionService implements Resettable {if (hasAt == hasCron) {throw ServerlessSupport.validation("schedule must set exactly one of at and cron.");}+ RedshiftServerlessSchedule.validate(action);}private void validateTarget(ScheduledAction action, String region) {@@ -187,6 +371,9 @@ public class RedshiftServerlessScheduledActionService implements Resettable {throw ServerlessSupport.validation("targetAction must set createSnapshot.");}requireNamespace(action.getTargetNamespaceName(), region);+ if (action.getSnapshotNamePrefix() != null) {+ RedshiftServerlessService.validateSnapshotName(snapshotName(action.getSnapshotNamePrefix(), Instant.EPOCH));+ }Integer retention = action.getRetentionPeriod();if (retention != null && retention != -1 && (retention < 1 || retention > MAX_RETENTION_DAYS)) {throw ServerlessSupport.validation("targetAction.createSnapshot.retentionPeriod must be -1 "diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.javaindex eae435af76..1066cdbafa 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessService.java@@ -1166,7 +1166,7 @@ public class RedshiftServerlessService implements Resettable {}}- private static void validateSnapshotName(String snapshotName) {+ static void validateSnapshotName(String snapshotName) {requireSnapshotName(snapshotName);if (snapshotName.length() < 3 || snapshotName.length() > 255 || !SNAPSHOT_NAME.matcher(snapshotName).matches()) {throw ServerlessSupport.validation("snapshotName must be 3-255 characters of lowercase letters, numbers, and hyphens.");diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.javaindex ae9d69eb1d..9a5889acba 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftserverless/model/ScheduledAction.java@@ -8,8 +8,8 @@ import java.util.LinkedHashMap;import java.util.Map;/**- * A Redshift Serverless scheduled action. Floci stores and echoes it but never runs it, so no- * snapshot is taken on the schedule. The same type carries an update request, in which a null member+ * A Redshift Serverless scheduled action. The activation and invocation members are internal: the+ * scheduler keeps them, and no AWS response shows them. The same type carries an update request, in which a null member* means the caller left it out.*/@RegisterForReflection@@ -33,12 +33,20 @@ public class ScheduledAction {private String ownerId;private boolean scheduleSpecified;private boolean targetSpecified;+ private Instant activatedAt;+ private Instant lastClaimedInvocation;+ private Instant lastCompletedInvocation;+ private String lastFailure;public ScheduledAction() {}public ScheduledAction(ScheduledAction other) {this.ownerId = other.ownerId;+ this.activatedAt = other.activatedAt;+ this.lastClaimedInvocation = other.lastClaimedInvocation;+ this.lastCompletedInvocation = other.lastCompletedInvocation;+ this.lastFailure = other.lastFailure;this.scheduleSpecified = other.scheduleSpecified;this.targetSpecified = other.targetSpecified;this.scheduledActionName = other.scheduledActionName;@@ -192,4 +200,38 @@ public class ScheduledAction {public void setTargetSpecified(boolean targetSpecified) {this.targetSpecified = targetSpecified;}++ /** When the current schedule began counting, so no invocation before it is replayed. */+ public Instant getActivatedAt() {+ return activatedAt;+ }++ public void setActivatedAt(Instant activatedAt) {+ this.activatedAt = activatedAt;+ }++ /** The latest scheduled time whose snapshot was started; it is never attempted twice. */+ public Instant getLastClaimedInvocation() {+ return lastClaimedInvocation;+ }++ public void setLastClaimedInvocation(Instant lastClaimedInvocation) {+ this.lastClaimedInvocation = lastClaimedInvocation;+ }++ public Instant getLastCompletedInvocation() {+ return lastCompletedInvocation;+ }++ public void setLastCompletedInvocation(Instant lastCompletedInvocation) {+ this.lastCompletedInvocation = lastCompletedInvocation;+ }++ public String getLastFailure() {+ return lastFailure;+ }++ public void setLastFailure(String lastFailure) {+ this.lastFailure = lastFailure;+ }}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 44363c5f3f..d58274e381 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -583,6 +583,8 @@ floci:shutdown-timeout-seconds: ${FLOCI_SERVICES_REDSHIFT_DATA_SHUTDOWN_TIMEOUT_SECONDS:10}redshift-serverless:enabled: true+ scheduled-actions-enabled: ${FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTIONS_ENABLED:true}+ scheduled-action-sweep-interval-seconds: ${FLOCI_SERVICES_REDSHIFT_SERVERLESS_SCHEDULED_ACTION_SWEEP_INTERVAL_SECONDS:60}neptune:enabled: trueproxy-base-port: 8182diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.javaindex f55934d9a0..eb95e9e4a5 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessAdministrationHandlerTest.java@@ -93,7 +93,8 @@ class RedshiftServerlessAdministrationHandlerTest {new RedshiftServerlessEndpointAccessService(storageFactory, regionResolver, serverlessService),customDomains,new RedshiftServerlessSnapshotCopyService(storageFactory, regionResolver, serverlessService),- new RedshiftServerlessScheduledActionService(storageFactory, serverlessService)),+ new RedshiftServerlessScheduledActionService(storageFactory, serverlessService,+ mock(RedshiftServerlessScheduledActionAuthorizer.class), regionResolver)),customDomains);}@@ -524,7 +525,7 @@ class RedshiftServerlessAdministrationHandlerTest {}@Test- void scheduledActionIsStoredAndNeverRun() {+ void scheduledActionIsStoredWithItsNextCronInvocation() {namespace("sched-ns");long future = Instant.now().plusSeconds(3600).getEpochSecond();String target = "\"targetAction\":{\"createSnapshot\":{\"namespaceName\":\"sched-ns\","@@ -541,7 +542,7 @@ class RedshiftServerlessAdministrationHandlerTest {assertEquals(7, created.get("targetAction").get("createSnapshot").get("retentionPeriod").intValue());assertEquals("env", created.get("targetAction").get("createSnapshot").get("tags").get(0).get("key").textValue());assertEquals("nightly", created.get("scheduledActionDescription").textValue());- assertEquals(0, created.get("nextInvocations").size());+ assertEquals(1, created.get("nextInvocations").size());assertTrue(created.get("scheduledActionUuid").textValue().length() > 8);assertEquals("ConflictException", errorType("CreateScheduledAction", "{\"scheduledActionName\":\"nightly-snap\","@@ -696,7 +697,7 @@ class RedshiftServerlessAdministrationHandlerTest {void aNamespaceRecreatedUnderTheSameNameDoesNotInheritActions() {namespace("recreate-ns");ok("CreateScheduledAction", "{\"scheduledActionName\":\"old-action\",\"namespaceName\":\"recreate-ns\","- + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"x\"},\"targetAction\":{\"createSnapshot\":"+ + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{\"createSnapshot\":"+ "{\"namespaceName\":\"recreate-ns\",\"snapshotNamePrefix\":\"p\"}}}");ok("DeleteNamespace", "{\"namespaceName\":\"recreate-ns\"}");@@ -705,7 +706,7 @@ class RedshiftServerlessAdministrationHandlerTest {assertEquals("ResourceNotFoundException", errorType("GetScheduledAction","{\"scheduledActionName\":\"old-action\"}"));ok("CreateScheduledAction", "{\"scheduledActionName\":\"old-action\",\"namespaceName\":\"recreate-ns\","- + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"x\"},\"targetAction\":{\"createSnapshot\":"+ + "\"roleArn\":\"r\",\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{\"createSnapshot\":"+ "{\"namespaceName\":\"recreate-ns\",\"snapshotNamePrefix\":\"p\"}}}");}@@ -724,7 +725,7 @@ class RedshiftServerlessAdministrationHandlerTest {void scheduledActionUpdateRejectsAPartialTargetAndATagWithoutAKey() {namespace("partial-ns");String create = "{\"scheduledActionName\":\"partial-action\",\"namespaceName\":\"partial-ns\",\"roleArn\":\"r\","- + "\"schedule\":{\"cron\":\"x\"},\"targetAction\":{\"createSnapshot\":"+ + "\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{\"createSnapshot\":"+ "{\"namespaceName\":\"partial-ns\",\"snapshotNamePrefix\":\"p\"";ok("CreateScheduledAction", create + "}}}");@@ -737,12 +738,12 @@ class RedshiftServerlessAdministrationHandlerTest {"{\"scheduledActionName\":\"partial-action\",\"schedule\":{}}"));JsonNode unchanged = ok("UpdateScheduledAction","{\"scheduledActionName\":\"partial-action\",\"enabled\":false}").get("scheduledAction");- assertEquals("x", unchanged.get("schedule").get("cron").textValue());+ assertEquals("0 3 * * ? *", unchanged.get("schedule").get("cron").textValue());assertEquals("DISABLED", unchanged.get("state").textValue());assertEquals("ValidationException", errorType("CreateScheduledAction","{\"scheduledActionName\":\"empty-target\",\"namespaceName\":\"partial-ns\",\"roleArn\":\"r\","- + "\"schedule\":{\"cron\":\"x\"},\"targetAction\":{}}"));+ + "\"schedule\":{\"cron\":\"0 3 * * ? *\"},\"targetAction\":{}}"));assertEquals("ValidationException", errorType("CreateScheduledAction",create.replace("partial-action", "tag-action") + ",\"tags\":[{\"value\":\"v\"}]}}}"));assertEquals("ValidationException", errorType("CreateScheduledAction",diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.javaindex c7cf734c0b..97a5c5acaf 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessIntegrationTest.java@@ -10,6 +10,7 @@ import io.quarkus.test.common.http.TestHTTPResource;import io.quarkus.test.junit.QuarkusTest;import io.restassured.response.ValidatableResponse;import io.restassured.specification.RequestSpecification;+import jakarta.inject.Inject;import org.junit.jupiter.api.BeforeAll;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;@@ -21,6 +22,7 @@ import java.net.URI;import java.nio.file.Files;import java.nio.file.Path;import java.time.Instant;+import java.time.temporal.ChronoUnit;import java.util.List;import java.util.Map;import java.util.UUID;@@ -785,6 +787,43 @@ class RedshiftServerlessIntegrationTest {call("DeleteScheduledAction", "{\"scheduledActionName\":\"admin-sched\"}").statusCode(200);}+ @Inject+ RedshiftServerlessScheduledActionService scheduledActions;++ @Test+ void scheduledActionTakesItsSnapshotOnASweepAndReportsTheNextInvocation() {+ String namespace = "sweep-ns";+ call("CreateNamespace", "{\"namespaceName\":\"" + namespace + "\",\"adminUsername\":\"admin\"}")+ .statusCode(200);+ cleanup.register(() -> call("DeleteNamespace", "{\"namespaceName\":\"" + namespace + "\"}"));+ call("CreateScheduledAction", "{\"scheduledActionName\":\"sweep-sched\",\"namespaceName\":\"" + namespace + "\","+ + "\"roleArn\":\"arn:aws:iam::000000000000:role/r\",\"schedule\":{\"cron\":\"0 * * * ? *\"},"+ + "\"targetAction\":{\"createSnapshot\":{\"namespaceName\":\"" + namespace + "\","+ + "\"snapshotNamePrefix\":\"sweep\",\"retentionPeriod\":3,"+ + "\"tags\":[{\"key\":\"env\",\"value\":\"it\"}]}}}")+ .statusCode(200)+ .body("scheduledAction.nextInvocations", hasSize(1));+ cleanup.register(() -> call("DeleteScheduledAction", "{\"scheduledActionName\":\"sweep-sched\"}"));+ call("CreateScheduledAction", "{\"scheduledActionName\":\"too-often\",\"namespaceName\":\"" + namespace + "\","+ + "\"roleArn\":\"arn:aws:iam::000000000000:role/r\",\"schedule\":{\"cron\":\"0,30 * * * ? *\"},"+ + "\"targetAction\":{\"createSnapshot\":{\"namespaceName\":\"" + namespace + "\","+ + "\"snapshotNamePrefix\":\"p\"}}}")+ .statusCode(400)+ .body("__type", equalTo("ValidationException"));++ Instant tick = Instant.now().plusSeconds(3 * 3600);+ scheduledActions.runScheduledActions(tick);+ String snapshot = RedshiftServerlessScheduledActionService.snapshotName("sweep",+ tick.truncatedTo(ChronoUnit.HOURS));+ cleanup.register(() -> call("DeleteSnapshot", "{\"snapshotName\":\"" + snapshot + "\"}"));++ call("GetSnapshot", "{\"snapshotName\":\"" + snapshot + "\"}")+ .statusCode(200)+ .body("snapshot.namespaceName", equalTo(namespace))+ .body("snapshot.status", equalTo("AVAILABLE"))+ .body("snapshot.snapshotRetentionPeriod", equalTo(3));+ }+private static ValidatableResponse call(String action, String body) {return given().header("X-Amz-Target", TARGET_PREFIX + action)diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.javaindex 78f544dba6..f27de5a7f3 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessJsonHandlerTest.java@@ -132,7 +132,8 @@ class RedshiftServerlessJsonHandlerTest {new RedshiftServerlessEndpointAccessService(storageFactory, regionResolver, service),customDomains,new RedshiftServerlessSnapshotCopyService(storageFactory, regionResolver, service),- new RedshiftServerlessScheduledActionService(storageFactory, service)),+ new RedshiftServerlessScheduledActionService(storageFactory, service,+ mock(RedshiftServerlessScheduledActionAuthorizer.class), regionResolver)),customDomains);}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.javaindex 29bfef67bf..1880d9c79f 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessRuntimeIntegrationTest.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.services.redshiftserverless;+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;import io.quarkus.test.junit.QuarkusTest;import io.restassured.response.ExtractableResponse;@@ -20,6 +21,8 @@ import java.sql.ResultSet;import java.sql.SQLException;import java.sql.Statement;import java.time.Duration;+import java.time.Instant;+import java.time.temporal.ChronoUnit;import java.util.Objects;import static org.junit.jupiter.api.Assertions.assertEquals;@@ -107,9 +110,13 @@ class RedshiftServerlessRuntimeIntegrationTest {}private String executeAndWait(String sql) {+ return executeAndWait(sql, WORKGROUP);+ }++ private String executeAndWait(String sql, String workgroup) {String id = RestAssuredJsonUtils.awsAction("RedshiftData", "ExecuteStatement", """{"Sql": "%s", "WorkgroupName": "%s", "Database": "dev"}- """.formatted(sql, WORKGROUP)).then().statusCode(200)+ """.formatted(sql, workgroup)).then().statusCode(200).extract().path("Id");Awaitility.await().atMost(Duration.ofSeconds(30)).pollDelay(Duration.ZERO).pollInterval(Duration.ofMillis(250)).until(() -> {@@ -163,6 +170,42 @@ class RedshiftServerlessRuntimeIntegrationTest {""".formatted(workgroupArn)).then().statusCode(200);}+ @Inject+ RedshiftServerlessScheduledActionService scheduledActions;++ @Test+ void aScheduledSnapshotCapturesRowsAndRestoresThem() {+ executeAndWait("CREATE TABLE sl_scheduled (id int)");+ executeAndWait("INSERT INTO sl_scheduled VALUES (1), (2), (3)");+ ScheduledAction request = new ScheduledAction();+ request.setScheduledActionName("rt-it-sched");+ request.setNamespaceName(NAMESPACE);+ request.setRoleArn("arn:aws:iam::000000000000:role/r");+ request.setScheduleCron("0 * * * ? *");+ request.setTargetNamespaceName(NAMESPACE);+ request.setSnapshotNamePrefix("rt-sched");+ scheduledActions.createScheduledAction(request, REGION);+ Instant tick = Instant.now().truncatedTo(ChronoUnit.HOURS).plus(Duration.ofHours(3)).plus(Duration.ofMinutes(15));+ String snapshot = RedshiftServerlessScheduledActionService.snapshotName("rt-sched",+ tick.truncatedTo(ChronoUnit.HOURS));+ try {+ scheduledActions.runScheduledActions(tick);+ executeAndWait("DROP TABLE sl_scheduled");++ RestAssuredJsonUtils.awsAction("RedshiftServerless", "RestoreFromSnapshot", """+ {"namespaceName":"%s","workgroupName":"%s","snapshotName":"%s"}+ """.formatted(NAMESPACE, WORKGROUP, snapshot)).then().statusCode(200);+ String id = executeAndWait("SELECT count(*) FROM sl_scheduled", WORKGROUP);+ ExtractableResponse<Response> result = RestAssuredJsonUtils.awsAction("RedshiftData", "GetStatementResult",+ "{\"Id\":\"" + id + "\"}").then().statusCode(200).extract();+ assertEquals(3, ((Number) result.path("Records[0][0].longValue")).intValue());+ } finally {+ scheduledActions.deleteScheduledAction("rt-it-sched", REGION);+ service.deleteSnapshot(snapshot, REGION);+ executeAndWait("DROP TABLE IF EXISTS sl_scheduled");+ }+ }+@Testvoid theDataApiRejectsAWrongDatabaseAndBothIdentifiers() {RestAssuredJsonUtils.awsAction("RedshiftData", "ExecuteStatement", """diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduleTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduleTest.javanew file mode 100644index 0000000000..6fcb0bbc29--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduleTest.java@@ -0,0 +1,75 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;+import org.junit.jupiter.api.Test;++import java.time.Instant;++import static org.junit.jupiter.api.Assertions.*;++class RedshiftServerlessScheduleTest {+ private static final Instant NOW = Instant.parse("2030-01-07T02:00:00Z");++ @Test+ void cronComputesNextAndLatestInUtc() {+ ScheduledAction action = cron("(0 3 ? * MON *)");+ RedshiftServerlessSchedule.validate(action);+ assertEquals(NOW.plusSeconds(3600), RedshiftServerlessSchedule.next(action, NOW).orElseThrow());+ assertEquals(NOW.plusSeconds(3600),+ RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(7200)).orElseThrow());+ assertTrue(RedshiftServerlessSchedule.latestDue(action, NOW, NOW).isEmpty());+ }++ @Test+ void numericWeekdayUsesAwsNumbering() {+ assertEquals(NOW.plusSeconds(3600),+ RedshiftServerlessSchedule.next(cron("0 3 ? * 2 *"), NOW).orElseThrow());+ }++ @Test+ void supportsExistingCronWrapper() {+ assertEquals(NOW.plusSeconds(3600),+ RedshiftServerlessSchedule.next(cron("cron(0 3 ? * MON *)"), NOW).orElseThrow());+ }++ @Test+ void rejectsInvalidAndSubHourlyCron() {+ for (String expression : new String[]{"x", "* * *", "0 3 * * * *", "0,30 * * * ? *"}) {+ assertThrows(AwsException.class, () -> RedshiftServerlessSchedule.validate(cron(expression)), expression);+ }+ }++ @Test+ void activeWindowBoundsNextAndDue() {+ ScheduledAction action = cron("0 * * * ? *");+ action.setStartTime(NOW.plusSeconds(7200));+ action.setEndTime(NOW.plusSeconds(10800));+ assertEquals(action.getStartTime(), RedshiftServerlessSchedule.next(action, NOW).orElseThrow());+ assertEquals(action.getEndTime(),+ RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(14400)).orElseThrow());+ assertTrue(RedshiftServerlessSchedule.next(action, action.getEndTime()).isEmpty());+ action.setEnabled(false);+ assertTrue(RedshiftServerlessSchedule.next(action, NOW).isEmpty());+ assertTrue(RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(14400)).isEmpty());+ }++ @Test+ void oneOffIsDueOnlyOnceAndWithinWindow() {+ ScheduledAction action = new ScheduledAction();+ action.setScheduleAt(NOW.plusSeconds(10));+ RedshiftServerlessSchedule.validate(action);+ assertEquals(action.getScheduleAt(), RedshiftServerlessSchedule.next(action, NOW).orElseThrow());+ assertEquals(action.getScheduleAt(),+ RedshiftServerlessSchedule.latestDue(action, NOW, NOW.plusSeconds(10)).orElseThrow());+ assertTrue(RedshiftServerlessSchedule.latestDue(action, action.getScheduleAt(), NOW.plusSeconds(20)).isEmpty());+ action.setEndTime(NOW.plusSeconds(9));+ assertTrue(RedshiftServerlessSchedule.next(action, NOW).isEmpty());+ }++ private ScheduledAction cron(String expression) {+ ScheduledAction action = new ScheduledAction();+ action.setScheduleCron(expression);+ return action;+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizerTest.javanew file mode 100644index 0000000000..a96b51c10d--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionAuthorizerTest.java@@ -0,0 +1,177 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import com.fasterxml.jackson.databind.ObjectMapper;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.services.iam.AssumeRolePolicyEvaluator;+import io.github.hectorvent.floci.services.iam.IamPolicyEvaluator;+import io.github.hectorvent.floci.services.iam.IamService;+import io.github.hectorvent.floci.services.iam.ScpProvider;+import io.github.hectorvent.floci.services.iam.model.CallerContext;+import io.github.hectorvent.floci.services.iam.model.IamRole;+import jakarta.enterprise.inject.Instance;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Optional;++import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++class RedshiftServerlessScheduledActionAuthorizerTest {+ private static final String ACCOUNT_ID = "123456789012";+ private static final String ROLE_ARN = "arn:aws:iam::" + ACCOUNT_ID + ":role/scheduler";+ private static final String TARGET = "arn:aws:redshift-serverless:us-east-1:" + ACCOUNT_ID + ":namespace/ns-id";+ private static final String TRUST = """+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",+ "Principal":{"Service":"scheduler.redshift.amazonaws.com"},"Action":"sts:AssumeRole"}]}""";+ private static final String ALLOW = """+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*"}]}""";++ private EmulatorConfig config;+ private IamService iamService;+ private Instance<ScpProvider> scpProvider;+ private ScpProvider scps;+ private RedshiftServerlessScheduledActionAuthorizer authorizer;++ @BeforeEach+ @SuppressWarnings("unchecked")+ void setUp() {+ config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().iam().enforcementEnabled()).thenReturn(true);+ iamService = mock(IamService.class);+ IamPolicyEvaluator evaluator = new IamPolicyEvaluator(new ObjectMapper());+ scpProvider = mock(Instance.class);+ scps = mock(ScpProvider.class);+ when(scpProvider.isResolvable()).thenReturn(true);+ when(scpProvider.get()).thenReturn(scps);+ authorizer = new RedshiftServerlessScheduledActionAuthorizer(config, iamService, evaluator,+ new AssumeRolePolicyEvaluator(new ObjectMapper(), evaluator), scpProvider);+ }++ private void role(String arn, String trust, String policy) {+ IamRole role = new IamRole("id", "scheduler", "/", arn, trust);+ when(iamService.findRole(ACCOUNT_ID, "scheduler")).thenReturn(Optional.of(role));+ when(iamService.resolvePrincipalContext(arn)).thenReturn(CallerContext.of(List.of(policy)));+ }++ @Test+ void trustedRoleWithThePermissionIsAllowed() {+ role(ROLE_ARN, TRUST, ALLOW);+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());+ }++ @Test+ void roleInAnotherPartitionOfTheSameAccountIsEvaluatedAsItsOwn() {+ String chinaRole = "arn:aws-cn:iam::" + ACCOUNT_ID + ":role/scheduler";+ role(chinaRole, TRUST, ALLOW);+ assertTrue(authorizer.refusal(chinaRole, ACCOUNT_ID,+ "arn:aws-cn:redshift-serverless:cn-north-1:" + ACCOUNT_ID + ":namespace/ns-id").isEmpty());+ }+++ @Test+ void serviceControlPolicyDenyIsRefused() {+ role(ROLE_ARN, TRUST, ALLOW);+ when(scps.effectiveScpLevels(ACCOUNT_ID)).thenReturn(List.of(List.of("""+ {"Version":"2012-10-17","Statement":[{"Effect":"Deny",+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*"}]}""")));+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void serviceControlPolicyThatAllowsLeavesTheRoleAllowed() {+ role(ROLE_ARN, TRUST, ALLOW);+ when(scps.effectiveScpLevels(ACCOUNT_ID)).thenReturn(List.of(List.of("""+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"*","Resource":"*"}]}""")));+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());+ }+ @Test+ void trustPolicyWithMatchingSourceAccountConditionIsAllowed() {+ String conditionalTrust = """+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",+ "Principal":{"Service":"scheduler.redshift.amazonaws.com"},"Action":"sts:AssumeRole",+ "Condition":{"StringEquals":{"aws:SourceAccount":"%s"}}}]}""".formatted(ACCOUNT_ID);+ role(ROLE_ARN, conditionalTrust, ALLOW);+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());+ }++ @Test+ void trustPolicyWithMismatchedSourceAccountConditionIsRefused() {+ String conditionalTrust = """+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",+ "Principal":{"Service":"scheduler.redshift.amazonaws.com"},"Action":"sts:AssumeRole",+ "Condition":{"StringEquals":{"aws:SourceAccount":"999999999999"}}}]}""";+ role(ROLE_ARN, conditionalTrust, ALLOW);+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void regionalConditionMatchingTargetRegionIsAllowed() {+ String regionalPolicy = """+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*",+ "Condition":{"StringEquals":{"aws:RequestedRegion":"us-east-1"}}}]}""";+ role(ROLE_ARN, TRUST, regionalPolicy);+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isEmpty());+ }++ @Test+ void regionalConditionForDifferentRegionIsRefused() {+ String regionalPolicy = """+ {"Version":"2012-10-17","Statement":[{"Effect":"Allow",+ "Action":"redshift-serverless:CreateSnapshot","Resource":"*",+ "Condition":{"StringEquals":{"aws:RequestedRegion":"us-west-2"}}}]}""";+ role(ROLE_ARN, TRUST, regionalPolicy);+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void missingPermissionIsRefused() {+ role(ROLE_ARN, TRUST, ALLOW.replace("CreateSnapshot", "GetSnapshot"));+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void explicitDenyIsRefused() {+ role(ROLE_ARN, TRUST, ALLOW.replace("\"Allow\"", "\"Deny\""));+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void roleThatDoesNotTrustTheSchedulerIsRefused() {+ role(ROLE_ARN, TRUST.replace("scheduler.redshift", "lambda"), ALLOW);+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void missingRoleIsRefused() {+ when(iamService.findRole(ACCOUNT_ID, "scheduler")).thenReturn(Optional.empty());+ assertTrue(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void roleOfAnotherAccountIsRefusedWithoutLookingItUp() {+ Optional<String> refusal = authorizer.refusal("arn:aws:iam::210987654321:role/scheduler", ACCOUNT_ID, TARGET);+ assertTrue(refusal.isPresent());+ verify(iamService, never()).findRole(anyString(), anyString());+ }++ @Test+ void nonRoleArnIsRefused() {+ assertTrue(authorizer.refusal("arn:aws:iam::" + ACCOUNT_ID + ":user/someone", ACCOUNT_ID, TARGET).isPresent());+ assertTrue(authorizer.refusal("not-an-arn", ACCOUNT_ID, TARGET).isPresent());+ }++ @Test+ void enforcementOffAllowsEverything() {+ when(config.services().iam().enforcementEnabled()).thenReturn(false);+ assertTrue(authorizer.refusal("not-an-arn", ACCOUNT_ID, TARGET).isEmpty());+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionRecoveryTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionRecoveryTest.javanew file mode 100644index 0000000000..f06f4b25e8--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionRecoveryTest.java@@ -0,0 +1,148 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.HybridStorage;+import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.redshiftserverless.model.Namespace;+import io.github.hectorvent.floci.services.redshiftserverless.model.RedshiftServerlessSnapshot;+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;+import org.junit.jupiter.api.AfterEach;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.io.TempDir;++import java.nio.file.Path;+import java.time.Clock;+import java.time.Instant;+import java.time.ZoneOffset;+import java.util.ArrayList;+import java.util.List;+import java.util.Map;++import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.ArgumentMatchers.*;+import static org.mockito.Mockito.*;++class RedshiftServerlessScheduledActionRecoveryTest {++ private static final String ACCOUNT = "123456789012";+ private static final String REGION = "us-east-1";+ private static final Instant CREATED = Instant.parse("2026-10-28T12:15:00Z");+ private static final Instant DUE = Instant.parse("2026-10-28T13:00:00Z");+ private static final String SNAPSHOT_NAME = "nightly-2026-10-28-13-00-00";++ @TempDir+ Path directory;++ private final List<HybridStorage<String, ScheduledAction>> backends = new ArrayList<>();+ private StorageFactory factory;+ private RegionResolver regions;+ private RedshiftServerlessService serverless;+ private AccountAwareStorageBackend<RedshiftServerlessSnapshot> snapshots;+ private RedshiftServerlessScheduledActionService service;++ @BeforeEach+ @SuppressWarnings({"unchecked", "rawtypes"})+ void setUp() {+ factory = mock(StorageFactory.class);+ when(factory.create(anyString(), anyString(), any(TypeReference.class)))+ .thenAnswer(invocation -> AccountAwareStorageBackend.inMemory(ACCOUNT));+ AccountAwareStorageBackend<Namespace> namespaces = AccountAwareStorageBackend.inMemory(ACCOUNT);+ Namespace namespace = new Namespace();+ namespace.setNamespaceName("source-ns");+ namespace.setNamespaceId("ns-id");+ namespace.setNamespaceArn("arn:aws:redshift-serverless:us-east-1:" + ACCOUNT + ":namespace/ns-id");+ namespaces.put(REGION + "::source-ns", namespace);+ when(factory.create(eq("redshiftserverless"), eq("redshiftserverless-namespaces.json"), any(TypeReference.class)))+ .thenReturn((AccountAwareStorageBackend) namespaces);+ snapshots = AccountAwareStorageBackend.inMemory(ACCOUNT);+ when(factory.create(eq("redshiftserverless"), eq("redshiftserverless-snapshots.json"), any(TypeReference.class)))+ .thenReturn((AccountAwareStorageBackend) snapshots);+ regions = mock(RegionResolver.class);+ when(regions.getAccountId()).thenReturn(ACCOUNT);+ serverless = spy(new RedshiftServerlessService(factory, regions, mock(RedshiftServerlessEndpoints.class),+ mock(RedshiftServerlessRuntime.class), mock(EmulatorConfig.class)));+ service = reload();+ ScheduledAction action = new ScheduledAction();+ action.setScheduledActionName("hourly");+ action.setNamespaceName("source-ns");+ action.setRoleArn("arn:aws:iam::" + ACCOUNT + ":role/scheduler");+ action.setScheduleCron("0 * * * ? *");+ action.setTargetNamespaceName("source-ns");+ action.setSnapshotNamePrefix("nightly");+ service.createScheduledAction(action, REGION);+ backends.getFirst().flush();+ }++ @AfterEach+ void stopBackends() {+ backends.forEach(HybridStorage::shutdown);+ }++ @SuppressWarnings({"unchecked", "rawtypes"})+ private RedshiftServerlessScheduledActionService reload() {+ HybridStorage<String, ScheduledAction> backend = new HybridStorage<>(directory.resolve("actions.json"),+ new TypeReference<Map<String, ScheduledAction>>() {}, 600_000);+ backends.add(backend);+ backend.load();+ AccountAwareStorageBackend<ScheduledAction> actions = new AccountAwareStorageBackend<>(backend, null, ACCOUNT);+ when(factory.create(eq("redshiftserverless"), eq("redshiftserverless-scheduledactions.json"), any(TypeReference.class)))+ .thenReturn((AccountAwareStorageBackend) actions);+ return new RedshiftServerlessScheduledActionService(factory, serverless,+ mock(RedshiftServerlessScheduledActionAuthorizer.class), regions, Clock.fixed(CREATED, ZoneOffset.UTC));+ }++ @Test+ void crashBeforeSnapshotRetriesAfterHybridReloadEvenIfAFlushRanDuringTheAttempt() {+ doAnswer(invocation -> {+ backends.getFirst().flush();+ throw new AssertionError("simulated process crash before snapshot creation");+ }).when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());++ assertThrows(AssertionError.class, () -> service.runScheduledActions(DUE));+ RedshiftServerlessScheduledActionService restarted = reload();+ assertNull(restarted.getScheduledAction("hourly", REGION).getLastClaimedInvocation());+ doCallRealMethod().when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ restarted.runScheduledActions(DUE.plusSeconds(60));++ assertEquals(1, snapshots.scan(key -> true).size());+ assertEquals(SNAPSHOT_NAME, snapshots.scan(key -> true).getFirst().getSnapshotName());+ assertEquals(DUE, restarted.getScheduledAction("hourly", REGION).getLastCompletedInvocation());+ }++ @Test+ void crashAfterSnapshotRetriesTheSameNameWithoutCreatingADuplicate() {+ doAnswer(invocation -> {+ invocation.callRealMethod();+ throw new AssertionError("simulated process crash after snapshot creation");+ }).when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());++ assertThrows(AssertionError.class, () -> service.runScheduledActions(DUE));+ RedshiftServerlessScheduledActionService restarted = reload();+ assertNull(restarted.getScheduledAction("hourly", REGION).getLastClaimedInvocation());+ doCallRealMethod().when(serverless).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ restarted.runScheduledActions(DUE.plusSeconds(60));+ restarted.runScheduledActions(DUE.plusSeconds(120));++ assertEquals(1, snapshots.scan(key -> true).size());+ verify(serverless, times(2)).createSnapshot(eq(SNAPSHOT_NAME), eq("source-ns"), any(), any(), eq(REGION));+ assertEquals(DUE, restarted.getScheduledAction("hourly", REGION).getLastClaimedInvocation());+ assertTrue(restarted.getScheduledAction("hourly", REGION).getLastFailure().contains("already exists"));+ }++ @Test+ void completedCheckpointSurvivesHybridFlushAndReload() {+ service.runScheduledActions(DUE);+ backends.getFirst().flush();++ RedshiftServerlessScheduledActionService restarted = reload();+ restarted.runScheduledActions(DUE.plusSeconds(60));++ assertEquals(DUE, restarted.getScheduledAction("hourly", REGION).getLastCompletedInvocation());+ verify(serverless).createSnapshot(eq(SNAPSHOT_NAME), eq("source-ns"), any(), any(), eq(REGION));+ assertEquals(1, snapshots.scan(key -> true).size());+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionSchedulerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionSchedulerTest.javanew file mode 100644index 0000000000..5cb8a88abe--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionSchedulerTest.java@@ -0,0 +1,115 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import io.github.hectorvent.floci.config.EmulatorConfig;+import org.junit.jupiter.api.Test;++import java.time.Instant;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.TimeoutException;+import java.util.concurrent.atomic.AtomicInteger;++import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.doAnswer;+import static org.mockito.Mockito.doThrow;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++class RedshiftServerlessScheduledActionSchedulerTest {+ private static final Instant NOW = Instant.parse("2030-01-07T02:00:00Z");++ private final RedshiftServerlessScheduledActionService service = mock(RedshiftServerlessScheduledActionService.class);++ private RedshiftServerlessScheduledActionScheduler scheduler(boolean serviceEnabled, boolean actionsEnabled) {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().redshiftServerless().enabled()).thenReturn(serviceEnabled);+ when(config.services().redshiftServerless().scheduledActionsEnabled()).thenReturn(actionsEnabled);+ when(config.services().redshiftServerless().scheduledActionSweepIntervalSeconds()).thenReturn(60);+ return new RedshiftServerlessScheduledActionScheduler(service, config);+ }++ @Test+ void workerExistsOnlyWhenTheServiceAndTheSwitchAreBothOn() {+ assertTrue(scheduler(true, true).active());+ assertFalse(scheduler(true, false).active());+ assertFalse(scheduler(false, true).active());+ }++ @Test+ void tickRunsTheSweepAtTheGivenInstant() {+ scheduler(true, false).tick(NOW);+ verify(service).runScheduledActions(NOW);+ }++ @Test+ void resetDrainsARunningSweepAndPausesUntilItEnds() throws Exception {+ RedshiftServerlessScheduledActionScheduler scheduler = scheduler(true, false);+ CountDownLatch running = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ doAnswer(invocation -> {+ running.countDown();+ release.await(10, TimeUnit.SECONDS);+ return null;+ }).when(service).runScheduledActions(NOW);+ ExecutorService executor = Executors.newFixedThreadPool(2);+ try {+ Future<?> sweep = executor.submit(() -> scheduler.tick(NOW));+ assertTrue(running.await(5, TimeUnit.SECONDS));+ Future<?> reset = executor.submit(scheduler::beforeReset);+ assertThrows(TimeoutException.class, () -> reset.get(300, TimeUnit.MILLISECONDS));++ release.countDown();+ reset.get(5, TimeUnit.SECONDS);+ sweep.get(5, TimeUnit.SECONDS);+ scheduler.tick(NOW.plusSeconds(60));+ verify(service, never()).runScheduledActions(NOW.plusSeconds(60));++ scheduler.afterReset();+ scheduler.tick(NOW.plusSeconds(120));+ verify(service).runScheduledActions(NOW.plusSeconds(120));+ } finally {+ release.countDown();+ executor.shutdownNow();+ }+ }++ @Test+ void overlappingTicksNeverRunTheSweepConcurrently() throws Exception {+ RedshiftServerlessScheduledActionScheduler scheduler = scheduler(true, false);+ AtomicInteger concurrent = new AtomicInteger();+ AtomicInteger peak = new AtomicInteger();+ doAnswer(invocation -> {+ peak.accumulateAndGet(concurrent.incrementAndGet(), Math::max);+ Thread.sleep(100);+ concurrent.decrementAndGet();+ return null;+ }).when(service).runScheduledActions(NOW);+ ExecutorService executor = Executors.newFixedThreadPool(3);+ try {+ Future<?> first = executor.submit(() -> scheduler.tick(NOW));+ Future<?> second = executor.submit(() -> scheduler.tick(NOW));+ Future<?> third = executor.submit(() -> scheduler.tick(NOW));+ first.get(5, TimeUnit.SECONDS);+ second.get(5, TimeUnit.SECONDS);+ third.get(5, TimeUnit.SECONDS);+ } finally {+ executor.shutdownNow();+ }+ assertEquals(1, peak.get());+ }++ @Test+ void aFailedSweepReleasesTheLock() {+ RedshiftServerlessScheduledActionScheduler scheduler = scheduler(true, false);+ doThrow(new IllegalStateException("boom")).when(service).runScheduledActions(NOW);+ assertThrows(IllegalStateException.class, () -> scheduler.tick(NOW));+ scheduler.tick(NOW.plusSeconds(60));+ verify(service).runScheduledActions(NOW.plusSeconds(60));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionServiceTest.javanew file mode 100644index 0000000000..2e1ad39e83--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftserverless/RedshiftServerlessScheduledActionServiceTest.java@@ -0,0 +1,354 @@+package io.github.hectorvent.floci.services.redshiftserverless;++import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.InMemoryStorage;+import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.redshiftserverless.model.Namespace;+import io.github.hectorvent.floci.services.redshiftserverless.model.ScheduledAction;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.io.IOException;+import java.io.UncheckedIOException;+import java.time.Clock;+import java.time.Duration;+import java.time.Instant;+import java.time.ZoneOffset;+import java.time.temporal.ChronoUnit;+import java.util.LinkedHashMap;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicBoolean;++import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.ArgumentMatchers.startsWith;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++class RedshiftServerlessScheduledActionServiceTest {+ private static final String REGION = "us-east-1";+ private static final String ACCOUNT_ID = "123456789012";+ private static final String ROLE_ARN = "arn:aws:iam::" + ACCOUNT_ID + ":role/scheduler";+ private static final String NAMESPACE_ARN = "arn:aws:redshift-serverless:us-east-1:" + ACCOUNT_ID + ":namespace/ns-id";+ private static final Instant BASE_TIME = Instant.parse("2026-10-28T12:15:00Z");++ private final AtomicBoolean failCheckpointWrites = new AtomicBoolean();+ private AccountAwareStorageBackend<ScheduledAction> store;+ private StorageFactory storageFactory;+ private RedshiftServerlessService serverless;+ private RedshiftServerlessScheduledActionAuthorizer authorizer;+ private RegionResolver regionResolver;+ private String namespaceId;+ private Clock clock;+ private RedshiftServerlessScheduledActionService service;++ @BeforeEach+ @SuppressWarnings({"unchecked", "rawtypes"})+ void setUp() {+ namespaceId = "ns-id";+ clock = Clock.fixed(BASE_TIME, ZoneOffset.UTC);+ store = new AccountAwareStorageBackend<>(new InMemoryStorage<String, ScheduledAction>() {+ @Override+ public synchronized void putAllAtomically(Map<String, ScheduledAction> entries) {+ if (failCheckpointWrites.get()) {+ throw new UncheckedIOException(new IOException("disk full"));+ }+ super.putAllAtomically(entries);+ }+ }, null, ACCOUNT_ID);+ storageFactory = mock(StorageFactory.class);+ when(storageFactory.create(eq("redshiftserverless"), eq("redshiftserverless-scheduledactions.json"),+ any(TypeReference.class))).thenReturn((AccountAwareStorageBackend) store);+ serverless = mock(RedshiftServerlessService.class);+ when(serverless.namespaceExists(anyString(), eq(REGION))).thenReturn(true);+ when(serverless.findNamespace(anyString(), eq(REGION))).thenAnswer(invocation -> Optional.of(namespace()));+ authorizer = mock(RedshiftServerlessScheduledActionAuthorizer.class);+ regionResolver = mock(RegionResolver.class);+ when(regionResolver.getAccountId()).thenReturn(ACCOUNT_ID);+ service = newService();+ }++ private RedshiftServerlessScheduledActionService newService() {+ return new RedshiftServerlessScheduledActionService(storageFactory, serverless, authorizer, regionResolver, clock);+ }++ private Namespace namespace() {+ Namespace namespace = new Namespace();+ namespace.setNamespaceId(namespaceId);+ namespace.setNamespaceArn(NAMESPACE_ARN);+ return namespace;+ }++ private ScheduledAction request(String name) {+ ScheduledAction request = new ScheduledAction();+ request.setScheduledActionName(name);+ request.setNamespaceName("source-ns");+ request.setRoleArn(ROLE_ARN);+ request.setScheduleCron("0 * * * ? *");+ request.setTargetNamespaceName("source-ns");+ request.setSnapshotNamePrefix("Nightly");+ request.setRetentionPeriod(7);+ Map<String, String> tags = new LinkedHashMap<>();+ tags.put("env", "dev");+ request.setSnapshotTags(tags);+ return request;+ }++ private static Instant later() {+ return BASE_TIME.plus(Duration.ofHours(3));+ }++ private static String expectedName(Instant tick) {+ return RedshiftServerlessScheduledActionService.snapshotName("nightly", tick.truncatedTo(ChronoUnit.HOURS));+ }++ @Test+ void cronActionTakesTheLatestDueSnapshotOnceWithItsArguments() {+ service.createScheduledAction(request("hourly"), REGION);+ Instant tick = later();++ service.runScheduledActions(tick);+ service.runScheduledActions(tick.plusSeconds(60));++ Map<String, String> tags = new LinkedHashMap<>();+ tags.put("env", "dev");+ verify(serverless, times(1)).createSnapshot(expectedName(tick), "source-ns", 7, tags, REGION);+ assertEquals(tick.truncatedTo(ChronoUnit.HOURS),+ store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastCompletedInvocation());+ }++ @Test+ void missedInvocationsAreCoalescedIntoTheLatestOne() {+ service.createScheduledAction(request("hourly"), REGION);++ service.runScheduledActions(BASE_TIME.plus(Duration.ofHours(30)));++ verify(serverless, times(1)).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ }++ @Test+ void oneOffActionRunsOnceAtItsTime() {+ ScheduledAction oneOff = request("once");+ oneOff.setScheduleCron(null);+ Instant at = BASE_TIME.plusSeconds(600);+ oneOff.setScheduleAt(at);+ service.createScheduledAction(oneOff, REGION);++ service.runScheduledActions(at.minusSeconds(1));+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ service.runScheduledActions(at.plusSeconds(5));+ service.runScheduledActions(at.plusSeconds(3600));++ verify(serverless, times(1)).createSnapshot(eq(RedshiftServerlessScheduledActionService.snapshotName("nightly", at)),+ eq("source-ns"), eq(7), any(), eq(REGION));+ }++ @Test+ void activeWindowAndDisabledActionsDoNotRun() {+ ScheduledAction early = request("windowed");+ early.setStartTime(BASE_TIME.plus(Duration.ofHours(10)));+ service.createScheduledAction(early, REGION);+ ScheduledAction disabled = request("off");+ disabled.setEnabled(false);+ service.createScheduledAction(disabled, REGION);++ service.runScheduledActions(later());++ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ }++ @Test+ void enablingDoesNotReplayWhatWasMissedWhileDisabled() {+ ScheduledAction disabled = request("off");+ disabled.setEnabled(false);+ service.createScheduledAction(disabled, REGION);+ ScheduledAction enable = new ScheduledAction();+ enable.setScheduledActionName("off");+ enable.setEnabled(true);+ service.updateScheduledAction(enable, REGION);++ service.runScheduledActions(BASE_TIME.plusSeconds(60));++ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ }++ @Test+ void storedActionWithoutACheckpointStartsCountingWithoutReplay() {+ ScheduledAction legacy = new ScheduledAction(request("legacy"));+ legacy.setScheduledActionUuid("u-1");+ legacy.setOwnerId(namespaceId);+ legacy.setEnabled(true);+ store.put("us-east-1::legacy", legacy);+ Instant tick = later();++ service.runScheduledActions(tick);+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ service.runScheduledActions(tick.plus(Duration.ofHours(2)));++ verify(serverless, times(1)).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ }++ @Test+ void oneFailingActionDoesNotStopOthersAndIsNotRetried() {+ service.createScheduledAction(request("a-failing"), REGION);+ ScheduledAction other = request("b-healthy");+ other.setSnapshotNamePrefix("healthy");+ service.createScheduledAction(other, REGION);+ Instant tick = later();+ when(serverless.createSnapshot(anyString(), anyString(), any(), any(), anyString())).thenAnswer(invocation -> {+ if (invocation.getArgument(0, String.class).startsWith("nightly")) {+ throw ServerlessSupport.conflict("exists");+ }+ return null;+ });++ service.runScheduledActions(tick);+ service.runScheduledActions(tick.plusSeconds(30));++ verify(serverless, times(1)).createSnapshot(startsWith("nightly"), anyString(), any(), any(), anyString());+ verify(serverless, times(1)).createSnapshot(startsWith("healthy"), anyString(), any(), any(), anyString());+ assertNotNull(store.getForAccount(ACCOUNT_ID, "us-east-1::a-failing").orElseThrow().getLastFailure());+ }++ @Test+ void failedCheckpointLeavesTheInvocationEligibleForRetry() {+ service.createScheduledAction(request("hourly"), REGION);+ failCheckpointWrites.set(true);++ service.runScheduledActions(later());++ assertNull(store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastClaimedInvocation());+ failCheckpointWrites.set(false);+ service.runScheduledActions(later().plusSeconds(60));++ verify(serverless, times(2)).createSnapshot(eq(expectedName(later())), anyString(), any(), any(), eq(REGION));+ assertEquals(later().truncatedTo(ChronoUnit.HOURS),+ store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastCompletedInvocation());+ }++ @Test+ void claimedInvocationIsNotRepeatedAfterRestart() {+ service.createScheduledAction(request("hourly"), REGION);+ Instant tick = later();+ service.runScheduledActions(tick);++ RedshiftServerlessScheduledActionService restarted = newService();+ restarted.runScheduledActions(tick.plusSeconds(60));++ verify(serverless, times(1)).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ }++ @Test+ void recreatedNamespaceInvalidatesTheOldAction() {+ service.createScheduledAction(request("hourly"), REGION);+ namespaceId = "ns-id-2";++ service.runScheduledActions(later());++ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ assertTrue(store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").isEmpty());+ }++ @Test+ void deniedRoleTakesNoSnapshotAndIsNotRetried() {+ service.createScheduledAction(request("hourly"), REGION);+ when(authorizer.refusal(ROLE_ARN, ACCOUNT_ID, NAMESPACE_ARN)).thenReturn(Optional.of("denied"));+ Instant tick = later();++ service.runScheduledActions(tick);+ service.runScheduledActions(tick.plusSeconds(60));++ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ verify(authorizer, times(1)).refusal(ROLE_ARN, ACCOUNT_ID, NAMESPACE_ARN);+ assertEquals("denied", store.getForAccount(ACCOUNT_ID, "us-east-1::hourly").orElseThrow().getLastFailure());+ }++ @Test+ void anotherAccountsActionIsAuthorizedAsItsOwnAccount() {+ ScheduledAction foreign = new ScheduledAction(request("foreign"));+ foreign.setScheduledActionUuid("u-2");+ foreign.setOwnerId(namespaceId);+ foreign.setEnabled(true);+ foreign.setRoleArn("arn:aws:iam::210987654321:role/scheduler");+ foreign.setActivatedAt(BASE_TIME);+ store.putAllAtomicallyForAccount("210987654321", Map.of("us-east-1::foreign", foreign));+ when(authorizer.refusal(anyString(), eq("210987654321"), anyString())).thenReturn(Optional.of("denied"));++ service.runScheduledActions(later());++ verify(authorizer).refusal(eq("arn:aws:iam::210987654321:role/scheduler"), eq("210987654321"), anyString());+ verify(serverless, never()).createSnapshot(anyString(), anyString(), any(), any(), anyString());+ }++ @Test+ void readsAndMutationsOfOtherActionsAreNotBlockedByARunningDump() throws Exception {+ service.createScheduledAction(request("hourly"), REGION);+ ScheduledAction other = request("other");+ other.setScheduleCron("0 0 1 1 ? *");+ service.createScheduledAction(other, REGION);+ CountDownLatch dumping = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ when(serverless.createSnapshot(anyString(), anyString(), any(), any(), anyString())).thenAnswer(invocation -> {+ dumping.countDown();+ release.await(10, TimeUnit.SECONDS);+ return null;+ });+ ExecutorService executor = Executors.newFixedThreadPool(2);+ try {+ Future<?> sweep = executor.submit(() -> service.runScheduledActions(later()));+ assertTrue(dumping.await(5, TimeUnit.SECONDS));++ Future<ScheduledAction> read = executor.submit(() -> service.getScheduledAction("hourly", REGION));+ Future<ScheduledAction> mutation = executor.submit(() -> {+ ScheduledAction change = new ScheduledAction();+ change.setScheduledActionName("other");+ change.setDescription("changed");+ return service.updateScheduledAction(change, REGION);+ });+ assertEquals("hourly", read.get(5, TimeUnit.SECONDS).getScheduledActionName());+ assertEquals("changed", mutation.get(5, TimeUnit.SECONDS).getDescription());+ release.countDown();+ sweep.get(5, TimeUnit.SECONDS);+ } finally {+ release.countDown();+ executor.shutdownNow();+ }+ }++ @Test+ void snapshotNameStaysWithinTheNamingRules() {+ String name = RedshiftServerlessScheduledActionService.snapshotName("Nightly", Instant.parse("2030-01-07T03:00:00Z"));+ assertEquals("nightly-2030-01-07-03-00-00", name);+ assertTrue(RedshiftServerlessScheduledActionService.snapshotName("a".repeat(400), BASE_TIME).length() <= 255);+ }++ @Test+ void unusableSnapshotNamePrefixIsRejectedOnCreate() {+ ScheduledAction invalid = request("invalid-prefix");+ invalid.setSnapshotNamePrefix("nightly_backup");+ assertThrows(AwsException.class, () -> service.createScheduledAction(invalid, REGION));+ }++ @Test+ void unusableSnapshotNamePrefixIsRejectedOnUpdate() {+ service.createScheduledAction(request("valid-prefix"), REGION);+ ScheduledAction update = new ScheduledAction();+ update.setScheduledActionName("valid-prefix");+ update.setSnapshotNamePrefix("nightly_backup");+ assertThrows(AwsException.class, () -> service.updateScheduledAction(update, REGION));+ }+}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex 2e56a97e07..198f0f66c9 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -251,6 +251,8 @@ floci:shutdown-timeout-seconds: 10redshift-serverless:enabled: true+ scheduled-actions-enabled: false+ scheduled-action-sweep-interval-seconds: 60neptune:enabled: trueproxy-base-port: 8182