Light, fluffy, and always free - The AWS Local Emulator alternative
feat(redshift): execute automated snapshot schedules (#6536)
Vũ Thế Huyf184cce726d7
diff --git a/docs/services/redshift.md b/docs/services/redshift.mdindex 0c93f73745..923cfea112 100644--- a/docs/services/redshift.md+++ b/docs/services/redshift.md@@ -23,7 +23,7 @@ For running SQL without a PostgreSQL wire connection (the way Lambda and Step Fu| `DescribeClusters` | List clusters and their connection details || `DeleteCluster` | Create the requested final snapshot, then remove the cluster; accepts `SkipFinalClusterSnapshot` to delete without a backup || `CreateClusterSnapshot` | Back up a cluster to a SQL dump via `pg_dump` |-| `DescribeClusterSnapshots` | List snapshots, optionally filtered by snapshot or cluster identifier |+| `DescribeClusterSnapshots` | List snapshots, optionally filtered by snapshot or cluster identifier and SnapshotType (manual or automated) || `DeleteClusterSnapshot` | Remove a snapshot and its stored dump || `RestoreFromClusterSnapshot` | Create a new cluster and load a snapshot's dump into it via `psql`; accepts `SnapshotIdentifier` or a same-account, same-region `SnapshotArn` || `CreateClusterParameterGroup` | Register a parameter group (metadata only) |@@ -48,7 +48,7 @@ For running SQL without a PostgreSQL wire connection (the way Lambda and Step Fu| `EnableSnapshotCopy` | Record cross-region snapshot copy on a cluster (`RetentionPeriod` defaults to 7, `ManualSnapshotRetentionPeriod` to -1); `DescribeClusters` then reports `ClusterSnapshotCopyStatus`. A `DestinationRegion` that is not an AWS region is refused with `UnknownSnapshotCopyRegionFault`. No snapshot is copied || `DisableSnapshotCopy` | Clear a cluster's snapshot copy configuration || `ModifySnapshotCopyRetentionPeriod` | Change the automated (or, with `Manual`, the manual) snapshot copy retention period |-| `CreateSnapshotSchedule` | Store a snapshot schedule of `rate(...)` / `cron(...)` definitions with optional tags; a `ScheduleIdentifier` other than letters, digits and hyphens is refused with `InvalidParameterValue`. No snapshots are taken on it |+| `CreateSnapshotSchedule` | Create a UTC rate or cron schedule that takes automated snapshots on associated clusters, with optional tags || `ModifySnapshotSchedule` | Replace a schedule's definitions || `DescribeSnapshotSchedules` | List schedules filtered by `ScheduleIdentifier`, `ClusterIdentifier`, `TagKeys` and `TagValues`, paged with `MaxRecords` and `Marker`, with their associated clusters || `DeleteSnapshotSchedule` | Remove a schedule; refused with `InvalidClusterSnapshotScheduleState` while a cluster uses it |@@ -194,6 +194,37 @@ and [COPY authorization parameters](https://docs.aws.amazon.com/redshift/latest/`CreateTags`/`DeleteTags`/`DescribeTags` do not list that fault at all and return`ResourceNotFoundFault` (HTTP 404) for any missing resource, grants included.+## Automated snapshots++Clusters take automated SQL-dump snapshots using their associated snapshot schedule. Use+`CreateSnapshotSchedule` followed by `ModifyClusterSnapshotSchedule` to attach a schedule.+Without an attached schedule, a cluster uses `rate(8 hours)`. Schedules run in UTC and accept+`rate(1 hour)` through `rate(24 hours)`, `rate(1 day)`, Redshift's three-field shorthand such+as `cron(0 9,17 MON-FRI)`, and full six-field cron expressions. Field ranges and near-term+execution spacing are validated; overlapping executions less than one hour apart are rejected+within a 400-day lookahead. The worker also enforces at least one hour between actual backups.++`CreateCluster` and `ModifyCluster` accept `AutomatedSnapshotRetentionPeriod` from 0 to 35 days,+with a default of 1. Zero disables automation and removes automated snapshots; RA3 and RG node+families require 1 to 35 days. Decreasing retention cleans up expired automated snapshots.+The newest automated snapshot is kept even after its retention period, until automation is+disabled or the cluster is deleted. Manual snapshots are unaffected. `DescribeClusterSnapshots`+returns `SnapshotType` and accepts `SnapshotType=automated` or `manual`. Automated snapshots+can be restored with `RestoreFromClusterSnapshot`; `DeleteClusterSnapshot` refuses their+manual deletion with `InvalidClusterSnapshotState`.++The background sweep checks all owning accounts, takes dumps only for available clusters, and+removes automated backups left by deleted clusters. Checkpoints use the Redshift storage mode+and survive restart. A failed dump leaves its occurrence retryable and removes partial output.+A committed dump is reused if saving its checkpoint failed. Cluster operations and concurrent+sweeps share resource locks. Changing or reattaching a schedule starts a new scheduling anchor.++These backups are full PostgreSQL dumps using the existing Docker backend. Floci does not+implement AWS's data-change-based trigger or incremental snapshot storage. Missed occurrences+are coalesced to one current backup, which contains the data at dump time, rather than replayed+as historical backups. Older persisted clusters acquire their initial scheduling anchor on the+first sweep. Cleanup after deleting a cluster runs on the next sweep.+## DynamoDB zero-ETLThe supported zero-ETL path is DynamoDB Streams to a provisioned Redshift cluster. The source ARN@@ -285,6 +316,8 @@ For `AWS::Redshift::ClusterParameterGroup`, `Parameters` is applied via `ModifyC| Variable | Default | Description ||----------|---------|-------------|| `FLOCI_SERVICES_REDSHIFT_ENABLED` | `true` | Enable or disable Redshift |+| `FLOCI_SERVICES_REDSHIFT_AUTOMATED_SNAPSHOTS_ENABLED` | `true` | Run background automated snapshot and retention sweeps |+| `FLOCI_SERVICES_REDSHIFT_SNAPSHOT_SWEEP_INTERVAL_SECONDS` | `60` | Delay between background snapshot sweeps || `FLOCI_SERVICES_REDSHIFT_IMAGE_VERSION` | `postgres:15-alpine` | PostgreSQL Docker image backing each cluster || `FLOCI_SERVICES_REDSHIFT_DEFAULT_PORT` | `5439` | Reported Redshift port hint (the real host port is dynamic and comes from `DescribeClusters`) || `FLOCI_SERVICES_REDSHIFT_PROXY_BASE_PORT` | `7100` | Lowest host port the per-cluster auth proxies bind |@@ -696,7 +729,7 @@ These views expose the documented Redshift column names, types, and ordering map- Multi-node clusters: `NodeType` and `NumberOfNodes` are stored as metadata; every cluster is a single PostgreSQL container.- Parameter groups apply no real engine settings; values are stored and echoed back only.- Subnet groups, VPC routing, and security groups are metadata only.-- Resize, pause/resume, and IAM authentication. Snapshot schedules and cross-region snapshot copy are stored and reported, but no snapshot is taken on a schedule or copied to another region.+- Resize, pause/resume, and IAM authentication. Cross-region snapshot copy is stored and reported, but no snapshot is copied to another region.- Cross-account role ARNs are not supported for COPY or UNLOAD.- The auth proxy validates the master user's password and any live `GetClusterCredentials` credential. Other non-master users pass straight through to PostgreSQL, which remains the authority for their credentials.- A cluster using `ManageMasterPassword` keeps its generated password in sync with thediff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex c1bdfe27d1..dd8a508eb7 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1723,6 +1723,10 @@ public interface EmulatorConfig {}interface RedshiftServiceConfig {+ @WithDefault("true")+ boolean automatedSnapshotsEnabled();+ @WithDefault("60")+ long snapshotSweepIntervalSeconds();@WithDefault("true")boolean enabled();@WithDefault("5439")diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.javaindex 468aecac0e..e51ff3b44d 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java@@ -92,12 +92,23 @@ public class RedshiftQueryHandler {String region = regionResolver.resolveRegionFromAuth(authorizationHeader);String defaultIamRoleArn = params.getFirst("DefaultIamRoleArn");- Cluster cluster = manageMasterPassword- ? service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,- params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn)- : service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn);+ Integer automatedRetention = parseOptionalInteger(params, "AutomatedSnapshotRetentionPeriod");+ Cluster cluster;+ if (manageMasterPassword) {+ cluster = automatedRetention == null+ ? service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,+ params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn)+ : service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,+ params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn, automatedRetention);+ } else {+ cluster = automatedRetention == null+ ? service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn)+ : service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, automatedRetention);+ }String parameterGroupName = params.getFirst("ClusterParameterGroupName");String multiAz = params.getFirst("MultiAZ");boolean hasParameterGroup = parameterGroupName != null && !parameterGroupName.isBlank();@@ -177,6 +188,13 @@ public class RedshiftQueryHandler {String snapshotIdentifier = params.getFirst("SnapshotIdentifier");String clusterIdentifier = params.getFirst("ClusterIdentifier");List<Snapshot> snapshots = service.describeSnapshots(snapshotIdentifier, clusterIdentifier);+ String snapshotType = params.getFirst("SnapshotType");+ if (snapshotType != null) {+ if (!List.of("manual", "automated").contains(snapshotType)) {+ throw new AwsException("InvalidParameterValue", "SnapshotType must be manual or automated", 400);+ }+ snapshots = snapshots.stream().filter(snapshot -> snapshotType.equals(snapshot.getSnapshotType())).toList();+ }XmlBuilder xmlBuilder = new XmlBuilder().start("DescribeClusterSnapshotsResponse").start("DescribeClusterSnapshotsResult")@@ -680,9 +698,14 @@ public class RedshiftQueryHandler {String clusterParameterGroupName = params.getFirst("ClusterParameterGroupName");List<String> vpcSecurityGroupIds = memberList(params, "VpcSecurityGroupIds");String multiAz = params.getFirst("MultiAZ");- Cluster cluster = service.modifyCluster(clusterIdentifier, nodeType, numberOfNodes,- masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds,- multiAz == null ? null : Boolean.parseBoolean(multiAz));+ Integer automatedRetention = parseOptionalInteger(params, "AutomatedSnapshotRetentionPeriod");+ Cluster cluster = automatedRetention == null+ ? service.modifyCluster(clusterIdentifier, nodeType, numberOfNodes,+ masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds,+ multiAz == null ? null : Boolean.parseBoolean(multiAz))+ : service.modifyCluster(clusterIdentifier, nodeType, numberOfNodes,+ masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds,+ multiAz == null ? null : Boolean.parseBoolean(multiAz), automatedRetention);String xml = new XmlBuilder().start("ModifyClusterResponse").start("ModifyClusterResult")@@ -926,6 +949,7 @@ public class RedshiftQueryHandler {.elem("NodeType", cluster.getNodeType()).elem("MasterUsername", cluster.getMasterUsername()).elem("ClusterStatus", cluster.getClusterStatus())+ .elem("AutomatedSnapshotRetentionPeriod", String.valueOf(cluster.getAutomatedSnapshotRetentionPeriod())).elem("ClusterAvailabilityStatus", availabilityStatus(cluster.getClusterStatus())).elem("AvailabilityZoneRelocationStatus", "disabled")// AWS always returns "enabled" or "disabled" here, never blank.@@ -1063,6 +1087,7 @@ public class RedshiftQueryHandler {.elem("ClusterIdentifier", snapshot.getClusterIdentifier()).elem("SnapshotArn", snapshot.getSnapshotArn()).elem("SnapshotCreateTime", createTime)+ .elem("SnapshotType", snapshot.getSnapshotType()).elem("ManualSnapshotRetentionPeriod", snapshot.getManualSnapshotRetentionPeriod().toString()).elem("Status", snapshot.getStatus()).elem("Port", String.valueOf(snapshot.getPort()))diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.javaindex f0134bbaa3..2d132d28d9 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java@@ -16,6 +16,7 @@ import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReaderimport io.github.hectorvent.floci.services.rds.proxy.PasswordValidator;import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;+import io.github.hectorvent.floci.services.redshift.model.AutomatedSnapshotState;import io.github.hectorvent.floci.services.redshift.model.Cluster;import io.github.hectorvent.floci.services.redshift.model.ClusterParameterGroup;import io.github.hectorvent.floci.services.redshift.model.ClusterSubnetGroup;@@ -37,9 +38,11 @@ import jakarta.inject.Inject;import org.jboss.logging.Logger;import java.io.IOException;+import java.nio.charset.StandardCharsets;import java.nio.file.Files;import java.nio.file.Path;import java.nio.file.Paths;+import java.time.Duration;import java.time.Instant;import java.time.format.DateTimeFormatter;import java.util.ArrayList;@@ -49,6 +52,7 @@ import java.util.LinkedHashMap;import java.util.LinkedHashSet;import java.util.List;import java.util.Map;+import java.util.Objects;import java.util.Optional;import java.util.Set;import java.util.UUID;@@ -56,7 +60,6 @@ import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.locks.ReentrantLock;import java.util.function.Consumer;import java.util.function.Supplier;-import java.util.regex.Matcher;import java.util.regex.Pattern;@ApplicationScoped@@ -83,6 +86,7 @@ public class RedshiftService implements ManagedSecretRotator {private final AccountAwareStorageBackend<Integration> integrations;private final AccountAwareStorageBackend<SnapshotCopyGrant> snapshotCopyGrants;private final AccountAwareStorageBackend<SnapshotSchedule> snapshotSchedules;+ private final AccountAwareStorageBackend<AutomatedSnapshotState> automatedSnapshotStates;private final RedshiftContainerManager containerManager;private final EmulatorConfig config;private final RegionResolver regionResolver;@@ -159,6 +163,8 @@ public class RedshiftService implements ManagedSecretRotator {this.integrations = storageFactory.create("redshift", "redshift-integrations.json", new TypeReference<Map<String, Integration>>() {});this.snapshotCopyGrants = storageFactory.create("redshift", "redshift-snapshot-copy-grants.json", new TypeReference<Map<String, SnapshotCopyGrant>>() {});this.snapshotSchedules = storageFactory.create("redshift", "redshift-snapshot-schedules.json", new TypeReference<Map<String, SnapshotSchedule>>() {});+ this.automatedSnapshotStates = storageFactory.create("redshift", "redshift-automated-snapshot-state.json",+ new TypeReference<Map<String, AutomatedSnapshotState>>() {});this.containerManager = containerManager;this.config = config;this.regionResolver = regionResolver;@@ -265,13 +271,22 @@ public class RedshiftService implements ManagedSecretRotator {public Cluster createCluster(String identifier, String nodeType, String username, String password,String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,List<String> iamRoleArns, String defaultIamRoleArn) {+ return createCluster(identifier, nodeType, username, password, clusterSubnetGroupName,+ vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, null);+ }++ public Cluster createCluster(String identifier, String nodeType, String username, String password,+ String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,+ List<String> iamRoleArns, String defaultIamRoleArn, Integer automatedRetention) {+ validateAutomatedRetention(nodeType, automatedRetention == null ? 1 : automatedRetention);return mutateCluster(identifier, () -> createClusterLocked(identifier, nodeType, username, password,- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn));+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, automatedRetention));}private Cluster createClusterLocked(String identifier, String nodeType, String username, String password,String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,- List<String> iamRoleArns, String defaultIamRoleArn) {+ List<String> iamRoleArns, String defaultIamRoleArn,+ Integer automatedRetention) {if (clusters.get(identifier).isPresent()) {throw new AwsException("ClusterAlreadyExists", "Cluster " + identifier + " already exists", 400);}@@ -284,6 +299,10 @@ public class RedshiftService implements ManagedSecretRotator {Cluster cluster = new Cluster();cluster.setClusterIdentifier(identifier);cluster.setNodeType(nodeType);+ cluster.setAutomatedSnapshotRetentionPeriod(automatedRetention);+ cluster.setAutomatedSnapshotAnchor(Instant.now());+ cluster.setSnapshotGeneration(UUID.randomUUID().toString());+ cluster.setSnapshotRegion(regionResolver.getRegion());cluster.setMasterUsername(username);cluster.setMasterPassword(password);cluster.setClusterSubnetGroupName(clusterSubnetGroupName);@@ -359,18 +378,31 @@ public class RedshiftService implements ManagedSecretRotator {String identifier, String nodeType, String username, String clusterSubnetGroupName,List<String> vpcSecurityGroupIds, List<String> iamRoleArns,String kmsKeyId, String region, String defaultIamRoleArn) {+ return createClusterWithManagedMasterPassword(identifier, nodeType, username, clusterSubnetGroupName,+ vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region, defaultIamRoleArn, null);+ }++ public Cluster createClusterWithManagedMasterPassword(+ String identifier, String nodeType, String username, String clusterSubnetGroupName,+ List<String> vpcSecurityGroupIds, List<String> iamRoleArns,+ String kmsKeyId, String region, String defaultIamRoleArn, Integer automatedRetention) {+ validateAutomatedRetention(nodeType, automatedRetention == null ? 1 : automatedRetention);return mutateCluster(identifier, () -> createClusterWithManagedMasterPasswordLocked(identifier,- nodeType, username, clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region, defaultIamRoleArn));+ nodeType, username, clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region,+ defaultIamRoleArn, automatedRetention));}private Cluster createClusterWithManagedMasterPasswordLocked(String identifier, String nodeType, String username, String clusterSubnetGroupName,List<String> vpcSecurityGroupIds, List<String> iamRoleArns,- String kmsKeyId, String region, String defaultIamRoleArn) {+ String kmsKeyId, String region, String defaultIamRoleArn, Integer automatedRetention) {requireSecretsManager();String password = RandomPasswordGenerator.generate(objectMapper.createObjectNode());Cluster cluster = createCluster(identifier, nodeType, username, password,- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn);+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, automatedRetention);+ if (region != null) {+ cluster.setSnapshotRegion(region);+ }try {return createManagedMasterSecret(cluster, kmsKeyId, region);} catch (RuntimeException e) {@@ -798,16 +830,32 @@ public class RedshiftService implements ManagedSecretRotator {public Cluster modifyCluster(String clusterIdentifier, String nodeType, Integer numberOfNodes,String masterUserPassword, String clusterParameterGroupName,List<String> vpcSecurityGroupIds, Boolean multiAZ) {- return mutateCluster(clusterIdentifier, () -> modifyClusterLocked(clusterIdentifier, nodeType,- numberOfNodes, masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds, multiAZ));+ return modifyCluster(clusterIdentifier, nodeType, numberOfNodes, masterUserPassword, clusterParameterGroupName,+ vpcSecurityGroupIds, multiAZ, null);+ }++ public Cluster modifyCluster(String clusterIdentifier, String nodeType, Integer numberOfNodes,+ String masterUserPassword, String clusterParameterGroupName,+ List<String> vpcSecurityGroupIds, Boolean multiAZ, Integer automatedRetention) {+ return withResourceLock("cluster", clusterIdentifier, () -> {+ Cluster cluster = mutate(() -> modifyClusterLocked(clusterIdentifier, nodeType,+ numberOfNodes, masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds, multiAZ,+ automatedRetention));+ if (automatedRetention != null) {+ expireAutomatedSnapshots(clusters.accountId(), clusterIdentifier, cluster, Instant.now());+ }+ return cluster;+ });}private Cluster modifyClusterLocked(String clusterIdentifier, String nodeType, Integer numberOfNodes,String masterUserPassword, String clusterParameterGroupName,- List<String> vpcSecurityGroupIds, Boolean multiAZ) {+ List<String> vpcSecurityGroupIds, Boolean multiAZ,+ Integer automatedRetention) {Cluster cluster = clusters.get(clusterIdentifier).orElseThrow(() -> new AwsException("ClusterNotFound", "Cluster " + clusterIdentifier + " not found", 404));-+ validateAutomatedRetention(nodeType == null ? cluster.getNodeType() : nodeType,+ automatedRetention == null ? cluster.getAutomatedSnapshotRetentionPeriod() : automatedRetention);// alterUserPassword runs before any mutation of the cluster object (a live reference// from HybridStorage, not a copy); if it throws, no metadata has been changed yet.if (masterUserPassword != null && !masterUserPassword.isBlank()) {@@ -836,6 +884,12 @@ public class RedshiftService implements ManagedSecretRotator {if (multiAZ != null) {cluster.setMultiAZ(multiAZ);}+ if (automatedRetention != null) {+ if (cluster.getAutomatedSnapshotRetentionPeriod() == 0 && automatedRetention > 0) {+ cluster.setAutomatedSnapshotAnchor(Instant.now());+ }+ cluster.setAutomatedSnapshotRetentionPeriod(automatedRetention);+ }clusters.put(clusterIdentifier, cluster);clusters.flush();@@ -1208,9 +1262,13 @@ public class RedshiftService implements ManagedSecretRotator {/** Absolute, normalised {@code <persistentPath>/redshift-dumps/<accountId>} for this request's account. */private Path accountDumpDir() {+ return accountDumpDir(clusters.accountId());+ }++ private Path accountDumpDir(String accountId) {return Paths.get(config.storage().persistentPath()).resolve("redshift-dumps")- .resolve(clusters.accountId())+ .resolve(accountId).toAbsolutePath().normalize();}@@ -1239,27 +1297,43 @@ public class RedshiftService implements ManagedSecretRotator {if (clusterOpt.isEmpty()) {throw new AwsException("ClusterNotFound", "Cluster " + clusterIdentifier + " not found", 404);}- if (snapshots.get(snapshotIdentifier).isPresent()) {+ return createSnapshot(clusters.accountId(), snapshotIdentifier, clusterOpt.get(), retentionPeriod,+ Instant.now(), null);+ }++ private Snapshot createSnapshot(String accountId, String snapshotIdentifier, Cluster cluster,+ Integer retentionPeriod, Instant now, Instant scheduledTime) {+ validateSnapshotIdentifier(snapshotIdentifier);+ String clusterIdentifier = cluster.getClusterIdentifier();+ if ((scheduledTime == null ? snapshots.get(snapshotIdentifier)+ : snapshots.getForAccount(accountId, snapshotIdentifier)).isPresent()) {throw new AwsException("ClusterSnapshotAlreadyExists", "Snapshot " + snapshotIdentifier + " already exists", 400);}- Cluster cluster = clusterOpt.get();Snapshot snapshot = new Snapshot();snapshot.setSnapshotIdentifier(snapshotIdentifier);snapshot.setClusterIdentifier(clusterIdentifier);snapshot.setStatus("available");snapshot.setMasterUsername(cluster.getMasterUsername());snapshot.setMasterPassword(cluster.getMasterPassword());- snapshot.setSnapshotArn(snapshotArn(clusterIdentifier, snapshotIdentifier));- snapshot.setSnapshotCreateTime(Instant.now());+ snapshot.setSnapshotArn(scheduledTime == null ? snapshotArn(clusterIdentifier, snapshotIdentifier)+ : AwsArnUtils.Arn.of("redshift", snapshotRegion(cluster), accountId,+ "snapshot:" + clusterIdentifier + "/" + snapshotIdentifier).toString());+ snapshot.setSnapshotCreateTime(now);snapshot.setManualSnapshotRetentionPeriod(retentionPeriod);+ snapshot.setSnapshotType(scheduledTime == null ? "manual" : "automated");+ snapshot.setScheduledTime(scheduledTime);+ snapshot.setClusterGeneration(cluster.getSnapshotGeneration());+ if (scheduledTime != null) {+ snapshot.setAutomatedSnapshotRetentionPeriod(cluster.getAutomatedSnapshotRetentionPeriod());+ }if (cluster.getEndpoint() != null) {snapshot.setPort(cluster.getEndpoint().getPort());} else {snapshot.setPort(5439);}- Path dumpDir = accountDumpDir();+ Path dumpDir = accountDumpDir(accountId);// Defence in depth: validateSnapshotIdentifier already rejects path separators,// but keep the containment check so the dump can never land outside the account dir.Path dumpFile = dumpDir.resolve(snapshotIdentifier + ".sql").normalize();@@ -1269,9 +1343,13 @@ public class RedshiftService implements ManagedSecretRotator {boolean committed = false;try {Files.createDirectories(dumpDir);- containerManager.takeSnapshot(clusters.accountId(), clusterIdentifier, cluster.getMasterUsername(), dumpFile);+ containerManager.takeSnapshot(accountId, clusterIdentifier, cluster.getMasterUsername(), dumpFile);snapshot.setSqlDump(dumpFile.toString());- snapshots.putAllAtomically(Map.of(snapshotIdentifier, snapshot));+ if (scheduledTime == null) {+ snapshots.putAllAtomically(Map.of(snapshotIdentifier, snapshot));+ } else {+ snapshots.putAllAtomicallyForAccount(accountId, Map.of(snapshotIdentifier, snapshot));+ }committed = true;snapshots.flush();} catch (Exception e) {@@ -1328,6 +1406,9 @@ public class RedshiftService implements ManagedSecretRotator {throw new AwsException("ClusterSnapshotNotFound", "Snapshot " + snapshotIdentifier + " not found", 404);}Snapshot snapshot = snapshotOpt.get();+ if ("automated".equals(snapshot.getSnapshotType())) {+ throw new AwsException("InvalidClusterSnapshotState", "Automated snapshots cannot be deleted manually", 400);+ }snapshots.delete(snapshotIdentifier);snapshots.flush();if (isTrustedDumpPath(snapshot.getSqlDump())) {@@ -1439,6 +1520,9 @@ public class RedshiftService implements ManagedSecretRotator {Cluster cluster = new Cluster();cluster.setClusterIdentifier(clusterIdentifier);cluster.setNodeType(effectiveNodeType);+ cluster.setAutomatedSnapshotAnchor(Instant.now());+ cluster.setSnapshotGeneration(UUID.randomUUID().toString());+ cluster.setSnapshotRegion(regionResolver.getRegion());cluster.setMasterUsername(username);cluster.setMasterPassword(password);cluster.setClusterParameterGroupName(DEFAULT_PARAMETER_GROUP_NAME);@@ -1941,41 +2025,182 @@ public class RedshiftService implements ManagedSecretRotator {private static final int SNAPSHOT_SCHEDULE_PAGE_MAX = 100;- private static final Pattern RATE_DEFINITION =- Pattern.compile("rate\\(\\s*(\\d+)\\s+(hours?|days?)\\s*\\)");- private static final Pattern CRON_DEFINITION =- Pattern.compile("cron\\(\\s*([A-Za-z0-9,\\-*/?#]+(?:\\s+[A-Za-z0-9,\\-*/?#]+){2,5})\\s*\\)");- // The API reference allows only alphanumerics; the management guide's own example uses a hyphen.- private static final Pattern SCHEDULE_IDENTIFIER = Pattern.compile("[A-Za-z0-9-]+");+ private static void validateAutomatedRetention(String nodeType, int retention) {+ boolean required = nodeType != null && (nodeType.startsWith("ra3.") || nodeType.startsWith("rg."));+ if (retention < 0 || retention > 35 || required && retention == 0) {+ throw new AwsException("InvalidParameterValue", "AutomatedSnapshotRetentionPeriod must be "+ + (required ? "1" : "0") + " to 35 days", 400);+ }+ }- /**- * Shape check against the Redshift snapshot-schedule documentation: {@code rate(<n> hours|days)}- * with a frequency between once an hour and once a day, or {@code cron(...)} with three to six- * whitespace-separated fields drawn from the documented cron alphabet. Per-field value ranges- * are not checked.- */- private static boolean isValidScheduleDefinition(String definition) {- if (definition == null) {- return false;+ /** A deterministic sweep; the lifecycle scheduler supplies now, tests supply a fixed instant. */+ void runSnapshotSchedules(Instant now) {+ for (AccountAwareStorageBackend.AccountEntry<Cluster> entry : clusters.scanAllAccountEntries(key -> true)) {+ if (Thread.currentThread().isInterrupted()) {+ return;+ }+ try {+ withResourceLock(entry.accountId(), "cluster", entry.key(), () -> {+ Cluster cluster = clusters.getForAccount(entry.accountId(), entry.key()).orElse(null);+ if (cluster != null) {+ try {+ runSnapshotSchedule(entry.accountId(), cluster, now);+ } finally {+ // Retention must not depend on the next dump succeeding.+ expireAutomatedSnapshots(entry.accountId(), cluster.getClusterIdentifier(), cluster, now);+ }+ }+ return null;+ });+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not process automated snapshots for account {0}, cluster {1}",+ entry.accountId(), entry.key());+ }}- Matcher rate = RATE_DEFINITION.matcher(definition);- if (rate.matches()) {- long n = Long.parseLong(rate.group(1).length() > 9 ? "0" : rate.group(1));- return rate.group(2).startsWith("hour") ? n >= 1 && n <= 24 : n == 1;+ for (AccountAwareStorageBackend.AccountEntry<Snapshot> entry : snapshots.scanAllAccountEntries(key -> true)) {+ if (!"automated".equals(entry.value().getSnapshotType())) {+ continue;+ }+ String clusterId = entry.value().getClusterIdentifier();+ try {+ withResourceLock(entry.accountId(), "cluster", clusterId, () -> {+ if (clusters.getForAccount(entry.accountId(), clusterId).isEmpty()) {+ removeAutomatedSnapshot(entry.accountId(), entry.key());+ }+ return null;+ });+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not clean up automated snapshot {0} in account {1}", entry.key(), entry.accountId());+ }+ }+ for (AccountAwareStorageBackend.AccountEntry<AutomatedSnapshotState> entry+ : automatedSnapshotStates.scanAllAccountEntries(key -> true)) {+ try {+ withResourceLock(entry.accountId(), "cluster", entry.key(), () -> {+ if (clusters.getForAccount(entry.accountId(), entry.key()).isEmpty()) {+ automatedSnapshotStates.deleteForAccount(entry.accountId(), entry.key());+ }+ return null;+ });+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not remove the automated snapshot checkpoint of cluster {0} in account {1}",+ entry.key(), entry.accountId());+ }}- return CRON_DEFINITION.matcher(definition).matches();}- private static void validateScheduleDefinitions(List<String> definitions) {- if (definitions == null || definitions.isEmpty()) {- throw new AwsException("InvalidSchedule", "ScheduleDefinitions must contain at least one definition.", 400);++ private void runSnapshotSchedule(String accountId, Cluster cluster, Instant now) {+ if (!"available".equals(cluster.getClusterStatus()) || cluster.getAutomatedSnapshotRetentionPeriod() == 0) {+ return;+ }+ SnapshotSchedule schedule = cluster.getSnapshotScheduleIdentifier() == null ? null+ : snapshotSchedules.getForAccount(accountId, cluster.getSnapshotScheduleIdentifier()).orElseThrow(+ () -> new AwsException("SnapshotScheduleNotFound", "The cluster's snapshot schedule is missing", 400));+ List<String> definitions = schedule == null ? List.of("rate(8 hours)")+ : List.copyOf(schedule.getScheduleDefinitions());+ Instant activation = cluster.getAutomatedSnapshotAnchor();+ if (schedule != null && schedule.getUpdatedAt() != null+ && (activation == null || schedule.getUpdatedAt().isAfter(activation))) {+ activation = schedule.getUpdatedAt();+ }+ String scheduleKey = cluster.getSnapshotGeneration() + "|" + cluster.getSnapshotScheduleIdentifier()+ + "|" + definitions + "|" + activation;+ String clusterId = cluster.getClusterIdentifier();+ AutomatedSnapshotState state = automatedSnapshotStates.getForAccount(accountId, clusterId).orElse(null);+ if (state == null || !scheduleKey.equals(state.scheduleKey())) {+ state = new AutomatedSnapshotState(scheduleKey, activation == null ? now : activation, null);+ automatedSnapshotStates.putAllAtomicallyForAccount(accountId, Map.of(clusterId, state));+ }+ Optional<Instant> due = RedshiftSnapshotSchedule.latestDue(definitions, state.anchor(), now);+ if (due.isEmpty()) {+ return;+ }+ Instant scheduled = due.get();+ String identity = clusterId + "|" + state.anchor() + "|" + scheduled;+ String snapshotId = "rs-auto-" + UUID.nameUUIDFromBytes(identity.getBytes(StandardCharsets.UTF_8));+ Instant last = state.lastScheduledTime();+ // The checkpoint lives in a separate store, so it can outlive a snapshot whose metadata never reached disk.+ if (last != null && (scheduled.isBefore(last)+ || scheduled.equals(last) && snapshots.getForAccount(accountId, snapshotId).isPresent())) {+ return;+ }+ boolean tooSoon = snapshots.scanForAccount(accountId, key -> true).stream()+ .filter(snapshot -> "automated".equals(snapshot.getSnapshotType()))+ .filter(snapshot -> clusterId.equals(snapshot.getClusterIdentifier()))+ .map(Snapshot::getSnapshotCreateTime).filter(time -> time != null)+ .anyMatch(time -> time.plus(Duration.ofHours(1)).isAfter(now));+ boolean completed = withResourceLock(accountId, "snapshot", snapshotId, () -> {+ Snapshot existing = snapshots.getForAccount(accountId, snapshotId).orElse(null);+ if (existing == null) {+ if (tooSoon) {+ return false;+ }+ createSnapshot(accountId, snapshotId, cluster, null, now, scheduled);+ } else if (!"automated".equals(existing.getSnapshotType())+ || !clusterId.equals(existing.getClusterIdentifier()) || !scheduled.equals(existing.getScheduledTime())) {+ throw new AwsException("ClusterSnapshotAlreadyExists", "Snapshot identifier is already in use", 400);+ }+ return true;+ });+ if (!completed) {+ return;}- for (String definition : definitions) {- if (!isValidScheduleDefinition(definition)) {- throw new AwsException("InvalidSchedule", "Invalid schedule definition: " + definition, 400);+ automatedSnapshotStates.putAllAtomicallyForAccount(accountId, Map.of(clusterId,+ new AutomatedSnapshotState(scheduleKey, state.anchor(), scheduled)));+ }++ private String snapshotRegion(Cluster cluster) {+ return cluster.getSnapshotRegion() == null ? regionResolver.getDefaultRegion() : cluster.getSnapshotRegion();+ }++ private void expireAutomatedSnapshots(String accountId, String clusterId, Cluster cluster, Instant now) {+ List<Snapshot> automated = snapshots.scanForAccount(accountId, key -> true).stream()+ .filter(snapshot -> "automated".equals(snapshot.getSnapshotType()))+ .filter(snapshot -> clusterId.equals(snapshot.getClusterIdentifier()))+ .filter(snapshot -> snapshot.getSnapshotCreateTime() != null).toList();+ Snapshot latest = automated.stream().max(Comparator.comparing(Snapshot::getSnapshotCreateTime)+ .thenComparing(Snapshot::getSnapshotIdentifier)).orElse(null);+ for (Snapshot snapshot : automated) {+ boolean replacedCluster = cluster.getSnapshotGeneration() != null+ && !Objects.equals(cluster.getSnapshotGeneration(), snapshot.getClusterGeneration());+ if (replacedCluster || cluster.getAutomatedSnapshotRetentionPeriod() == 0 || snapshot != latest+ && !snapshot.getSnapshotCreateTime().plus(Duration.ofDays(cluster.getAutomatedSnapshotRetentionPeriod()))+ .isAfter(now)) {+ removeAutomatedSnapshot(accountId, snapshot.getSnapshotIdentifier());}}}+ private void removeAutomatedSnapshot(String accountId, String snapshotId) {+ withResourceLock(accountId, "snapshot", snapshotId, () -> mutate(() -> {+ Snapshot current = snapshots.getForAccount(accountId, snapshotId).orElse(null);+ if (current == null || !"automated".equals(current.getSnapshotType())) {+ return null;+ }+ snapshots.deleteAllAtomicallyForAccounts(Map.of(accountId, Set.of(snapshotId)));+ // Hybrid storage flushes lazily; a crash after the dump is gone would resurrect a dump-less snapshot.+ snapshots.flush();+ if (current.getSqlDump() != null) {+ Path file = Paths.get(current.getSqlDump()).toAbsolutePath().normalize();+ if (file.startsWith(accountDumpDir(accountId))) {+ try {+ Files.deleteIfExists(file);+ } catch (IOException e) {+ LOG.warnv(e, "Could not delete the dump of automated snapshot {0}", snapshotId);+ }+ }+ }+ return null;+ }));+ }++ // The API reference allows only alphanumerics; the management guide's own example uses a hyphen.+ private static final Pattern SCHEDULE_IDENTIFIER = Pattern.compile("[A-Za-z0-9-]+");++ private static void validateScheduleDefinitions(List<String> definitions) {+ RedshiftSnapshotSchedule.validate(definitions);+ }+private SnapshotSchedule requireSnapshotSchedule(String scheduleIdentifier) {return snapshotSchedules.get(scheduleIdentifier).orElseThrow(() -> new AwsException("SnapshotScheduleNotFound",@@ -1996,6 +2221,7 @@ public class RedshiftService implements ManagedSecretRotator {}validateScheduleDefinitions(definitions);SnapshotSchedule schedule = new SnapshotSchedule(id, description, definitions);+ schedule.setUpdatedAt(Instant.now());if (tags != null && !tags.isEmpty()) {schedule.setTags(new LinkedHashMap<>(tags));}@@ -2043,9 +2269,11 @@ public class RedshiftService implements ManagedSecretRotator {}public synchronized SnapshotSchedule modifySnapshotSchedule(String scheduleIdentifier, List<String> definitions) {- SnapshotSchedule schedule = requireSnapshotSchedule(scheduleIdentifier);+ SnapshotSchedule current = requireSnapshotSchedule(scheduleIdentifier);validateScheduleDefinitions(definitions);- schedule.setScheduleDefinitions(new ArrayList<>(definitions));+ SnapshotSchedule schedule = new SnapshotSchedule(scheduleIdentifier, current.getScheduleDescription(), definitions);+ schedule.setTags(new LinkedHashMap<>(current.getTags()));+ schedule.setUpdatedAt(Instant.now());snapshotSchedules.put(scheduleIdentifier, schedule);snapshotSchedules.flush();return schedule;@@ -2080,6 +2308,7 @@ public class RedshiftService implements ManagedSecretRotator {requireSnapshotSchedule(scheduleIdentifier);cluster.setSnapshotScheduleIdentifier(scheduleIdentifier);}+ cluster.setAutomatedSnapshotAnchor(Instant.now());clusters.put(clusterIdentifier, cluster);clusters.flush();return cluster;diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedule.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedule.javanew file mode 100644index 0000000000..b00b42bf0a--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedule.java@@ -0,0 +1,116 @@+package io.github.hectorvent.floci.services.redshift;++import com.cronutils.model.Cron;+import com.cronutils.model.time.ExecutionTime;+import io.github.hectorvent.floci.core.common.AwsCronDefinitions;+import io.github.hectorvent.floci.core.common.AwsException;++import java.time.Duration;+import java.time.Instant;+import java.time.ZoneOffset;+import java.time.temporal.ChronoUnit;+import java.util.ArrayList;+import java.util.List;+import java.util.Optional;+import java.util.regex.Matcher;+import java.util.regex.Pattern;++/** Redshift's UTC snapshot rules, including its minute/hour/day-of-week shorthand. */+final class RedshiftSnapshotSchedule {+ private static final Pattern RATE = Pattern.compile("rate\\(\\s*(\\d+)\\s+(hours?|days?)\\s*\\)");+ private static final Duration MINIMUM_INTERVAL = Duration.ofHours(1);++ private RedshiftSnapshotSchedule() { }++ static void validate(List<String> definitions) {+ try {+ if (definitions == null || definitions.isEmpty()) {+ throw new IllegalArgumentException("ScheduleDefinitions must contain at least one definition");+ }+ Instant anchor = Instant.now().truncatedTo(ChronoUnit.DAYS);+ List<Rule> rules = definitions.stream().map(RedshiftSnapshotSchedule::parse).toList();+ List<Instant> next = new ArrayList<>();+ for (Rule rule : rules) {+ next.add(rule.next(anchor, anchor).orElse(null));+ }+ Instant previous = null;+ Instant limit = null;+ while (true) {+ Instant earliest = next.stream().filter(value -> value != null).min(Instant::compareTo).orElse(null);+ if (earliest == null || limit != null && earliest.isAfter(limit)) {+ break;+ }+ if (previous != null && Duration.between(previous, earliest).compareTo(MINIMUM_INTERVAL) < 0) {+ throw new IllegalArgumentException("Snapshot schedules must be at least one hour apart");+ }+ if (limit == null) {+ limit = earliest.plus(Duration.ofDays(400));+ }+ previous = earliest;+ for (int i = 0; i < rules.size(); i++) {+ if (earliest.equals(next.get(i))) {+ next.set(i, rules.get(i).next(anchor, earliest).orElse(null));+ }+ }+ }+ } catch (IllegalArgumentException e) {+ throw new AwsException("InvalidSchedule", "Invalid snapshot schedule: " + e.getMessage(), 400);+ }+ }++ static Optional<Instant> latestDue(List<String> definitions, Instant anchor, Instant now) {+ return definitions.stream().map(RedshiftSnapshotSchedule::parse)+ .map(rule -> rule.latest(anchor, now)).flatMap(Optional::stream)+ .filter(time -> time.isAfter(anchor) && !time.isAfter(now)).max(Instant::compareTo);+ }++ private static Rule parse(String definition) {+ if (definition == null) {+ throw new IllegalArgumentException("A schedule definition cannot be null");+ }+ Matcher rate = RATE.matcher(definition);+ if (rate.matches()) {+ long value = Long.parseLong(rate.group(1));+ boolean days = rate.group(2).startsWith("day");+ if (value < 1 || days && value != 1 || !days && value > 24) {+ throw new IllegalArgumentException("Snapshot rates must be between one hour and one day");+ }+ long hours = days ? 24 : value;+ return new Rule(Duration.ofHours(hours), null);+ }+ if (!definition.startsWith("cron(") || !definition.endsWith(")")) {+ throw new IllegalArgumentException("Expected rate(...) or cron(...)");+ }+ String[] fields = definition.substring(5, definition.length() - 1).trim().split("\\s+");+ String normalized = switch (fields.length) {+ case 3 -> fields[0] + " " + fields[1] + " ? * " + fields[2] + " *";+ case 6 -> String.join(" ", fields);+ default -> throw new IllegalArgumentException("Redshift cron needs three or six fields");+ };+ String[] full = normalized.split(" ");+ if ("?".equals(full[2]) == "?".equals(full[4])) {+ throw new IllegalArgumentException("Use ? in exactly one of day-of-month and day-of-week");+ }+ Cron cron = AwsCronDefinitions.newParser().parse("0 " + normalized);+ cron.validate();+ return new Rule(null, ExecutionTime.forCron(cron));+ }++ private record Rule(Duration rate, ExecutionTime cron) {+ Optional<Instant> latest(Instant anchor, Instant now) {+ if (rate != null) {+ long count = Duration.between(anchor, now).getSeconds() / rate.getSeconds();+ return count < 1 ? Optional.empty() : Optional.of(anchor.plus(rate.multipliedBy(count)));+ }+ return cron.lastExecution(now.atZone(ZoneOffset.UTC).plusNanos(1)).map(value -> value.toInstant());+ }++ Optional<Instant> next(Instant anchor, Instant after) {+ if (rate != null) {+ long count = Math.max(0, Duration.between(anchor, after).getSeconds() / rate.getSeconds()) + 1;+ return Optional.of(anchor.plus(rate.multipliedBy(count)));+ }+ return cron.nextExecution(after.atZone(ZoneOffset.UTC)).map(value -> value.toInstant());+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduler.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduler.javanew file mode 100644index 0000000000..f3c590cc08--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduler.java@@ -0,0 +1,96 @@+package io.github.hectorvent.floci.services.redshift;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.Resettable;+import io.quarkus.runtime.ShutdownEvent;+import io.quarkus.runtime.StartupEvent;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.event.Observes;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.time.Instant;+import java.util.concurrent.Executors;+import java.util.concurrent.ScheduledExecutorService;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.locks.ReentrantLock;++/** Drives automated snapshots outside AWS request threads. */+@ApplicationScoped+public class RedshiftSnapshotScheduler implements Resettable {+ private static final Logger LOG = Logger.getLogger(RedshiftSnapshotScheduler.class);+ private final RedshiftService service;+ private final ScheduledExecutorService executor;+ private final long intervalSeconds;+ private final ReentrantLock sweepLock = new ReentrantLock();+ private volatile boolean paused;++ @Inject+ public RedshiftSnapshotScheduler(RedshiftService service, EmulatorConfig config) {+ this.service = service;+ this.intervalSeconds = Math.max(1, config.services().redshift().snapshotSweepIntervalSeconds());+ this.executor = config.services().redshift().enabled() && config.services().redshift().automatedSnapshotsEnabled()+ ? Executors.newSingleThreadScheduledExecutor(task -> {+ Thread thread = new Thread(task, "redshift-snapshot-scheduler");+ thread.setDaemon(true);+ return thread;+ }) : null;+ }++ void onStart(@Observes StartupEvent ignored) {+ if (executor != null) {+ executor.scheduleWithFixedDelay(this::tickSafely, intervalSeconds, intervalSeconds, TimeUnit.SECONDS);+ }+ }++ void onStop(@Observes ShutdownEvent ignored) {+ if (executor != null) {+ executor.shutdownNow();+ try {+ if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {+ LOG.warn("Redshift snapshot scheduler is still finishing a dump during shutdown");+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ LOG.debugv(e, "Interrupted while stopping the Redshift snapshot scheduler");+ }+ }+ }++ public void tick(Instant now) {+ sweepLock.lock();+ try {+ if (!paused) {+ service.runSnapshotSchedules(now);+ }+ } finally {+ sweepLock.unlock();+ }+ }++ /** Drains a running sweep so it cannot write old snapshots back into storage that reset is about to clear. */+ @Override+ public void beforeReset() {+ paused = true;+ sweepLock.lock();+ sweepLock.unlock();+ }++ @Override+ public void clear() {+ // Snapshot state lives in RedshiftService storage, which reset wipes.+ }++ @Override+ public void afterReset() {+ paused = false;+ }++ private void tickSafely() {+ try {+ tick(Instant.now());+ } catch (RuntimeException e) {+ LOG.warnv(e, "Redshift automated snapshot sweep failed");+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/AutomatedSnapshotState.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/AutomatedSnapshotState.javanew file mode 100644index 0000000000..3962bf7574--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/AutomatedSnapshotState.java@@ -0,0 +1,9 @@+package io.github.hectorvent.floci.services.redshift.model;++import io.quarkus.runtime.annotations.RegisterForReflection;++import java.time.Instant;++/** Restart checkpoint for a cluster's currently effective snapshot rules. */+@RegisterForReflection+public record AutomatedSnapshotState(String scheduleKey, Instant anchor, Instant lastScheduledTime) { }diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.javaindex 427e56fee4..cdca5ea596 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.redshift.model;import io.quarkus.runtime.annotations.RegisterForReflection;+import java.time.Instant;import java.util.ArrayList;import java.util.LinkedHashMap;import java.util.List;@@ -90,6 +91,22 @@ public class Cluster {private Integer snapshotCopyManualRetentionPeriod;private String snapshotCopyGrantName;private String snapshotScheduleIdentifier;+ private Instant automatedSnapshotAnchor;+ private String snapshotRegion;+ private Integer automatedSnapshotRetentionPeriod;+ private String snapshotGeneration;++ public String getSnapshotGeneration() { return snapshotGeneration; }+ public void setSnapshotGeneration(String value) { snapshotGeneration = value; }++ public Instant getAutomatedSnapshotAnchor() { return automatedSnapshotAnchor; }+ public void setAutomatedSnapshotAnchor(Instant value) { automatedSnapshotAnchor = value; }+ public String getSnapshotRegion() { return snapshotRegion; }+ public void setSnapshotRegion(String value) { snapshotRegion = value; }+ public int getAutomatedSnapshotRetentionPeriod() {+ return automatedSnapshotRetentionPeriod == null ? 1 : automatedSnapshotRetentionPeriod;+ }+ public void setAutomatedSnapshotRetentionPeriod(Integer value) { automatedSnapshotRetentionPeriod = value; }public String getSnapshotCopyDestinationRegion() { return snapshotCopyDestinationRegion; }public void setSnapshotCopyDestinationRegion(String snapshotCopyDestinationRegion) { this.snapshotCopyDestinationRegion = snapshotCopyDestinationRegion; }diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.javaindex ec95d52197..43eca6efe0 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java@@ -18,6 +18,20 @@ public class Snapshot {private String snapshotArn;private Instant snapshotCreateTime;private Integer manualSnapshotRetentionPeriod;+ private String snapshotType;+ private Integer automatedSnapshotRetentionPeriod;+ private Instant scheduledTime;+ private String clusterGeneration;++ public String getClusterGeneration() { return clusterGeneration; }+ public void setClusterGeneration(String value) { clusterGeneration = value; }++ public String getSnapshotType() { return snapshotType == null ? "manual" : snapshotType; }+ public void setSnapshotType(String value) { snapshotType = value; }+ public Integer getAutomatedSnapshotRetentionPeriod() { return automatedSnapshotRetentionPeriod; }+ public void setAutomatedSnapshotRetentionPeriod(Integer value) { automatedSnapshotRetentionPeriod = value; }+ public Instant getScheduledTime() { return scheduledTime; }+ public void setScheduledTime(Instant value) { scheduledTime = value; }public Snapshot() {}diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.javaindex d746013c17..3a6df88421 100644--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.redshift.model;import io.quarkus.runtime.annotations.RegisterForReflection;+import java.time.Instant;import java.util.ArrayList;import java.util.LinkedHashMap;import java.util.List;@@ -17,6 +18,10 @@ public class SnapshotSchedule {private String scheduleDescription;private List<String> scheduleDefinitions = new ArrayList<>();private Map<String, String> tags = new LinkedHashMap<>();+ private Instant updatedAt;++ public Instant getUpdatedAt() { return updatedAt; }+ public void setUpdatedAt(Instant value) { updatedAt = value; }public SnapshotSchedule() {}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex b20001e314..8b30358898 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -533,6 +533,8 @@ floci:default-image: "valkey/valkey:8"redshift:enabled: true+ automated-snapshots-enabled: true+ snapshot-sweep-interval-seconds: 60default-port: 5439image-version: postgres:15-alpinepoll-interval-ms: 1000diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotIntegrationTest.javanew file mode 100644index 0000000000..dd22ef447f--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotIntegrationTest.java@@ -0,0 +1,120 @@+package io.github.hectorvent.floci.services.redshift;++import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;+import io.github.hectorvent.floci.services.redshift.model.Cluster;+import io.github.hectorvent.floci.services.redshift.proxy.RedshiftProxyManager;+import io.quarkus.test.InjectMock;+import io.quarkus.test.common.http.TestHTTPResource;+import io.quarkus.test.junit.QuarkusTest;+import jakarta.inject.Inject;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;+import software.amazon.awssdk.regions.Region;+import software.amazon.awssdk.services.redshift.RedshiftClient;+import software.amazon.awssdk.services.redshift.model.RedshiftException;+import software.amazon.awssdk.services.redshift.model.Snapshot;++import java.net.URI;+import java.nio.file.Files;+import java.nio.file.Path;+import java.time.Instant;+import java.util.List;+import java.util.Map;++import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.ArgumentMatchers.*;+import static org.mockito.Mockito.*;++@QuarkusTest+class RedshiftAutomatedSnapshotIntegrationTest {+ private static final String ACCOUNT = "333333333333";++ @InjectMock+ RedshiftContainerManager manager;+ @InjectMock+ RedshiftProxyManager proxy;+ @Inject+ RedshiftSnapshotScheduler scheduler;+ @Inject+ StorageFactory factory;+ @TestHTTPResource+ URI endpoint;++ @BeforeEach+ void setup() {+ when(manager.start(anyString(), anyString(), anyString(), anyString(), any(), any()))+ .thenReturn(new RedshiftContainerHandle("mock", "warehouse", "localhost", 5432));+ doAnswer(invocation -> {+ Files.writeString(invocation.getArgument(3, Path.class), "backup contents");+ return null;+ }).when(manager).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));+ }++ @Test+ void sdkReportsAutomatedSnapshotsAndControlsTheirRetention() {+ String clusterId = "sdk-automated-warehouse";+ String scheduleId = "sdk-automated-hourly";+ try (RedshiftClient client = client()) {+ client.createSnapshotSchedule(request -> request.scheduleIdentifier(scheduleId).scheduleDefinitions("rate(1 hour)"));+ client.createCluster(request -> request.clusterIdentifier(clusterId).nodeType("dc2.large")+ .masterUsername("admin").masterUserPassword("Secret123").automatedSnapshotRetentionPeriod(1));+ try {+ client.modifyClusterSnapshotSchedule(request -> request.clusterIdentifier(clusterId).scheduleIdentifier(scheduleId));+ AccountAwareStorageBackend<Cluster> clusters = factory.create("redshift", "redshift-clusters.json",+ new TypeReference<Map<String, Cluster>>() { });+ Instant due = clusters.getForAccount(ACCOUNT, clusterId).orElseThrow().getAutomatedSnapshotAnchor().plusSeconds(3600);+ scheduler.tick(due);+ List<Snapshot> automated = client.describeClusterSnapshots(request ->+ request.clusterIdentifier(clusterId).snapshotType("automated")).snapshots();+ assertEquals(1, automated.size());+ Snapshot snapshot = automated.getFirst();+ assertEquals("automated", snapshot.snapshotType());+ assertEquals(due, snapshot.snapshotCreateTime());+ assertTrue(snapshot.snapshotArn().contains(":" + ACCOUNT + ":snapshot:"));+ assertTrue(client.describeClusterSnapshots(request ->+ request.clusterIdentifier(clusterId).snapshotType("manual")).snapshots().isEmpty());+ RedshiftException refused = assertThrows(RedshiftException.class, () ->+ client.deleteClusterSnapshot(request -> request.snapshotIdentifier(snapshot.snapshotIdentifier())));+ assertEquals("InvalidClusterSnapshotState", refused.awsErrorDetails().errorCode());+ client.createClusterSnapshot(request -> request.clusterIdentifier(clusterId).snapshotIdentifier("sdk-manual-keep"));+ assertEquals(0, client.modifyCluster(request -> request.clusterIdentifier(clusterId)+ .automatedSnapshotRetentionPeriod(0)).cluster().automatedSnapshotRetentionPeriod());+ scheduler.tick(due.plusSeconds(60));+ assertTrue(client.describeClusterSnapshots(request ->+ request.clusterIdentifier(clusterId).snapshotType("automated")).snapshots().isEmpty());+ assertEquals("manual", client.describeClusterSnapshots(request -> request.snapshotIdentifier("sdk-manual-keep"))+ .snapshots().getFirst().snapshotType());+ client.deleteClusterSnapshot(request -> request.snapshotIdentifier("sdk-manual-keep"));+ } finally {+ client.deleteCluster(request -> request.clusterIdentifier(clusterId).skipFinalClusterSnapshot(true));+ client.deleteSnapshotSchedule(request -> request.scheduleIdentifier(scheduleId));+ scheduler.tick(Instant.now().plusSeconds(7200));+ }+ }+ }++ @Test+ void sdkRejectsInvalidRetentionAndTooFrequentCron() {+ try (RedshiftClient client = client()) {+ RedshiftException retention = assertThrows(RedshiftException.class, () -> client.createCluster(request ->+ request.clusterIdentifier("invalid-auto-retention").nodeType("ra3.xlplus")+ .masterUsername("admin").masterUserPassword("Secret123").automatedSnapshotRetentionPeriod(0)));+ assertEquals("InvalidParameterValue", retention.awsErrorDetails().errorCode());+ RedshiftException cron = assertThrows(RedshiftException.class, () -> client.createSnapshotSchedule(request ->+ request.scheduleIdentifier("invalid-auto-cron").scheduleDefinitions("cron(* * *)")));+ assertEquals("InvalidSchedule", cron.awsErrorDetails().errorCode());+ verify(manager, never()).start(anyString(), anyString(), anyString(), anyString(), any(), any());+ }+ }++ private RedshiftClient client() {+ return RedshiftClient.builder().endpointOverride(endpoint).region(Region.US_EAST_1)+ .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCOUNT, "test"))).build();+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotServiceTest.javanew file mode 100644index 0000000000..97d77155ae--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotServiceTest.java@@ -0,0 +1,420 @@+package io.github.hectorvent.floci.services.redshift;++import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;+import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;+import io.github.hectorvent.floci.services.redshift.model.AutomatedSnapshotState;+import io.github.hectorvent.floci.services.redshift.model.Cluster;+import io.github.hectorvent.floci.services.redshift.model.Snapshot;+import io.github.hectorvent.floci.services.redshift.model.SnapshotSchedule;+import io.github.hectorvent.floci.services.redshift.proxy.RedshiftProxyManager;+import org.awaitility.Awaitility;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.io.TempDir;++import java.nio.file.Files;+import java.nio.file.Path;+import java.time.Duration;+import java.time.Instant;+import java.util.List;+import java.util.Map;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicInteger;+import java.util.stream.Stream;++import static org.junit.jupiter.api.Assertions.*;+import static org.mockito.ArgumentMatchers.*;+import static org.mockito.Mockito.*;++class RedshiftAutomatedSnapshotServiceTest {+ private static final String ACCOUNT = "111111111111";+ private static final Instant ANCHOR = Instant.parse("2026-10-25T00:00:00Z");++ @TempDir+ Path directory;++ @Test+ void rateSnapshotsAndCheckpointsSurviveRestartWithoutDuplicateDumps() throws Exception {+ Fixture first = fixture();+ seed(first, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ first.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ first.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ Snapshot snapshot = first.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();+ assertEquals("automated", snapshot.getSnapshotType());+ assertEquals("backup contents", Files.readString(Path.of(snapshot.getSqlDump())));+ verify(first.manager()).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));++ Fixture restarted = fixture();+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ verifyNoInteractions(restarted.manager());+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(6 * 3600));+ assertEquals(2, restarted.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ verify(restarted.manager()).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));+ }++ @Test+ void delayedTicksDoNotCreateTwoBackupsWithinOneHour() {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7199));+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(10799));+ assertEquals(2, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ }++ @Test+ void backgroundSweepKeepsAccountsRegionsAndPartitionsSeparate() {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "same-name", "cn-north-1", "rate(1 hour)");+ seed(fixture, "222222222222", "same-name", "us-gov-west-1", "rate(1 hour)");++ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));++ Snapshot china = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();+ Snapshot gov = fixture.snapshots().scanForAccount("222222222222", key -> true).getFirst();+ assertTrue(china.getSnapshotArn().startsWith("arn:aws-cn:redshift:cn-north-1:" + ACCOUNT + ":"));+ assertTrue(gov.getSnapshotArn().startsWith("arn:aws-us-gov:redshift:us-gov-west-1:222222222222:"));+ assertTrue(Path.of(china.getSqlDump()).startsWith(directory.resolve("redshift-dumps").resolve(ACCOUNT)));+ assertTrue(Path.of(gov.getSqlDump()).startsWith(directory.resolve("redshift-dumps").resolve("222222222222")));+ assertNotEquals(china.getSqlDump(), gov.getSqlDump());+ }++ @Test+ void retentionPreservesTheLatestAutomatedSnapshotAndEveryManualSnapshot() throws Exception {+ Fixture fixture = fixture();+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ Snapshot first = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));+ Snapshot latest = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).stream()+ .filter(snapshot -> !snapshot.getSnapshotIdentifier().equals(first.getSnapshotIdentifier()))+ .findFirst().orElseThrow();+ Snapshot manual = fixture.service().createSnapshot("manual-backup", "warehouse");+ cluster.setClusterStatus("paused");+ fixture.clusters().putForAccount(ACCOUNT, "warehouse", cluster);++ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));++ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, first.getSnapshotIdentifier()).isEmpty());+ assertFalse(Files.exists(Path.of(first.getSqlDump())));+ assertTrue(Files.exists(Path.of(latest.getSqlDump())));+ assertTrue(Files.exists(Path.of(manual.getSqlDump())));+ assertEquals(2, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ assertEquals("InvalidClusterSnapshotState", assertThrows(AwsException.class,+ () -> fixture.service().deleteSnapshot(latest.getSnapshotIdentifier())).getErrorCode());++ cluster.setAutomatedSnapshotRetentionPeriod(0);+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(4)));+ assertEquals(List.of(manual.getSnapshotIdentifier()), fixture.snapshots().scanForAccount(ACCOUNT, key -> true)+ .stream().map(Snapshot::getSnapshotIdentifier).toList());+ }++ @Test+ void missingClustersLoseAutomatedSnapshotsButKeepManualBackups() {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ fixture.service().createSnapshot("manual-backup", "warehouse");+ fixture.clusters().deleteForAccount(ACCOUNT, "warehouse");++ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));++ assertEquals(List.of("manual-backup"), fixture.snapshots().scanForAccount(ACCOUNT, key -> true).stream()+ .map(Snapshot::getSnapshotIdentifier).toList());+ assertTrue(fixture.states().getForAccount(ACCOUNT, "warehouse").isEmpty());+ }++ @Test+ void occurrenceIsRetakenWhenItsSnapshotMetadataWasLostButTheCheckpointSurvived() {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ Snapshot lost = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();+ fixture.snapshots().deleteForAccount(ACCOUNT, lost.getSnapshotIdentifier());++ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));++ assertEquals(List.of(lost.getSnapshotIdentifier()), fixture.snapshots().scanForAccount(ACCOUNT, key -> true)+ .stream().map(Snapshot::getSnapshotIdentifier).toList());+ verify(fixture.manager(), times(2)).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));+ }++ @Test+ void retentionStillRunsWhenTheNextDumpFails() throws Exception {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ Snapshot first = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));+ Snapshot latest = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).stream()+ .filter(snapshot -> !snapshot.getSnapshotIdentifier().equals(first.getSnapshotIdentifier()))+ .findFirst().orElseThrow();+ doThrow(new IllegalStateException("dump failed")).when(fixture.manager())+ .takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));++ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));++ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, first.getSnapshotIdentifier()).isEmpty());+ assertFalse(Files.exists(Path.of(first.getSqlDump())));+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, latest.getSnapshotIdentifier()).isPresent());+ }++ @Test+ void recreatedClusterDoesNotKeepThePreviousInstancesAutomatedBackups() {+ Fixture fixture = fixture();+ Cluster original = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ original.setSnapshotGeneration("first-instance");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ Cluster replacement = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ replacement.setSnapshotGeneration("replacement-instance");+ replacement.setClusterStatus("creating");++ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3601));++ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());+ }++ @Test+ void defaultScheduleAndLegacyAnchorAreStableAfterRestart() {+ Fixture fixture = fixture();+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", null);+ cluster.setAutomatedSnapshotAnchor(null);+ fixture.clusters().putForAccount(ACCOUNT, "warehouse", cluster);+ fixture.service().runSnapshotSchedules(ANCHOR);+ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());++ Fixture restarted = fixture();+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(8 * 3600));++ assertEquals(1, restarted.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ }++ @Test+ void failedDumpIsCleanedUpAndTheSameOccurrenceCanRetry() throws Exception {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ AtomicInteger attempts = new AtomicInteger();+ doAnswer(invocation -> {+ Files.writeString(invocation.getArgument(3, Path.class), "partial dump");+ if (attempts.incrementAndGet() == 1) {+ throw new IllegalStateException("dump failed");+ }+ return null;+ }).when(fixture.manager()).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));++ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));++ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());+ try (Stream<Path> files = Files.list(directory.resolve("redshift-dumps").resolve(ACCOUNT))) {+ assertEquals(0, files.count());+ }+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ assertEquals(2, attempts.get());+ }++ @Test+ void committedDumpIsReusedWhenCheckpointWriteFailed() {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ doAnswer(invocation -> {+ Map<String, AutomatedSnapshotState> checkpoint = invocation.getArgument(1);+ AutomatedSnapshotState state = checkpoint.get("warehouse");+ if (state.lastScheduledTime() != null) {+ throw new IllegalStateException("checkpoint write failed");+ }+ return invocation.callRealMethod();+ }).when(fixture.states()).putAllAtomicallyForAccount(eq(ACCOUNT), anyMap());+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());++ Fixture restarted = fixture();+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));++ verifyNoInteractions(restarted.manager());+ assertEquals(ANCHOR.plusSeconds(3600), restarted.states().getForAccount(ACCOUNT, "warehouse")+ .orElseThrow().lastScheduledTime());+ }++ @Test+ void scheduleChangesResetTheAnchorAndDoNotReplayOldOccurrences() {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ SnapshotSchedule changed = fixture.service().modifySnapshotSchedule("schedule-warehouse", List.of("rate(2 hours)"));+ assertNotNull(changed.getUpdatedAt());+ Instant activation = ANCHOR.plusSeconds(7200);+ changed.setUpdatedAt(activation);+ fixture.schedules().putForAccount(ACCOUNT, changed.getScheduleIdentifier(), changed);++ fixture.service().runSnapshotSchedules(activation.plusSeconds(7199));+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ fixture.service().runSnapshotSchedules(activation.plusSeconds(7200));+ assertEquals(2, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ }++ @Test+ void failedAssociationPreservesTheExistingAnchorAndCheckpoint() {+ Fixture fixture = fixture();+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ AutomatedSnapshotState checkpoint = fixture.states().getForAccount(ACCOUNT, "warehouse").orElseThrow();++ for (String scheduleId : List.of("missing", "")) {+ assertThrows(AwsException.class,+ () -> fixture.service().modifyClusterSnapshotSchedule("warehouse", scheduleId, false));+ assertEquals(ANCHOR, cluster.getAutomatedSnapshotAnchor());+ assertEquals("schedule-warehouse", cluster.getSnapshotScheduleIdentifier());+ assertEquals(checkpoint, fixture.states().getForAccount(ACCOUNT, "warehouse").orElseThrow());+ }+ }++ @Test+ void sweepCannotReadNewRulesWithTheOldActivationTime() throws Exception {+ Fixture fixture = fixture();+ Instant activation = Instant.now().minusSeconds(7200);+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(24 hours)");+ cluster.setAutomatedSnapshotAnchor(activation);+ SnapshotSchedule schedule = spy(fixture.schedules().getForAccount(ACCOUNT, "schedule-warehouse").orElseThrow());+ schedule.setUpdatedAt(activation);+ fixture.schedules().putForAccount(ACCOUNT, "schedule-warehouse", schedule);+ CountDownLatch betweenSetters = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ doAnswer(invocation -> {+ betweenSetters.countDown();+ assertTrue(release.await(5, TimeUnit.SECONDS));+ return invocation.callRealMethod();+ }).when(schedule).setUpdatedAt(any());+ try (ExecutorService executor = Executors.newSingleThreadExecutor()) {+ Future<?> updating = executor.submit(() -> fixture.service().modifySnapshotSchedule(+ "schedule-warehouse", List.of("rate(1 hour)")));+ try {+ Awaitility.await().atMost(Duration.ofSeconds(5))+ .until(() -> betweenSetters.getCount() == 0 || updating.isDone());+ fixture.service().runSnapshotSchedules(Instant.now().plusSeconds(1800));+ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());+ } finally {+ release.countDown();+ }+ updating.get(5, TimeUnit.SECONDS);+ }+ }++ @Test+ void failedRetentionCommitPreservesMetadataAndDumpForRetry() throws Exception {+ Fixture fixture = fixture();+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));+ Snapshot old = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));+ cluster.setClusterStatus("paused");+ Path metadata = directory.resolve("redshift-snapshots.json");+ Path saved = directory.resolve("snapshots-before-failure.json");+ Files.move(metadata, saved);+ Files.createDirectory(metadata);+ Files.writeString(metadata.resolve("block-replacement"), "keep");+ try {+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));++ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, old.getSnapshotIdentifier()).isPresent());+ assertTrue(Files.exists(Path.of(old.getSqlDump())));+ } finally {+ Files.delete(metadata.resolve("block-replacement"));+ Files.delete(metadata);+ Files.move(saved, metadata);+ }+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, old.getSnapshotIdentifier()).isEmpty());+ assertFalse(Files.exists(Path.of(old.getSqlDump())));+ assertEquals(1, fixture().snapshots().scanForAccount(ACCOUNT, key -> true).size());+ }++ @Test+ void concurrentSweepsTakeOnlyOneDumpForTheSameOccurrence() throws Exception {+ Fixture fixture = fixture();+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");+ CountDownLatch dumping = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ doAnswer(invocation -> {+ dumping.countDown();+ assertTrue(release.await(5, TimeUnit.SECONDS));+ Files.writeString(invocation.getArgument(3, Path.class), "backup contents");+ return null;+ }).when(fixture.manager()).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));+ try (ExecutorService executor = Executors.newFixedThreadPool(2)) {+ Future<?> first = executor.submit(() -> fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600)));+ try {+ assertTrue(dumping.await(5, TimeUnit.SECONDS));+ Future<?> second = executor.submit(() -> fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600)));+ release.countDown();+ first.get(5, TimeUnit.SECONDS);+ second.get(5, TimeUnit.SECONDS);+ } finally {+ release.countDown();+ }+ }+ verify(fixture.manager()).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());+ }++ private Fixture fixture() {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.storage().persistentPath()).thenReturn(directory.toString());+ when(config.defaultAccountId()).thenReturn(ACCOUNT);+ ServiceConfigAccess access = mock(ServiceConfigAccess.class);+ when(access.storageMode("redshift")).thenReturn("persistent");+ StorageFactory factory = spy(new StorageFactory(config, access));+ AccountAwareStorageBackend<AutomatedSnapshotState> states = spy(factory.create("redshift",+ "redshift-automated-snapshot-state.json", new TypeReference<Map<String, AutomatedSnapshotState>>() { }));+ doReturn(states).when(factory).create(eq("redshift"), eq("redshift-automated-snapshot-state.json"), any());+ RedshiftContainerManager manager = mock(RedshiftContainerManager.class);+ doAnswer(invocation -> {+ Files.writeString(invocation.getArgument(3, Path.class), "backup contents");+ return null;+ }).when(manager).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));+ RedshiftService service = new RedshiftService(factory, manager, config,+ new RegionResolver("us-east-1", ACCOUNT), mock(RedshiftProxyManager.class),+ mock(DockerHostResolver.class), new RedshiftCredentialBroker());+ return new Fixture(service, manager, states,+ factory.create("redshift", "redshift-clusters.json", new TypeReference<Map<String, Cluster>>() { }),+ factory.create("redshift", "redshift-snapshots.json", new TypeReference<Map<String, Snapshot>>() { }),+ factory.create("redshift", "redshift-snapshot-schedules.json", new TypeReference<Map<String, SnapshotSchedule>>() { }));+ }++ private static Cluster seed(Fixture fixture, String account, String id, String region, String definition) {+ Cluster cluster = new Cluster();+ cluster.setClusterIdentifier(id);+ cluster.setNodeType("dc2.large");+ cluster.setMasterUsername("admin");+ cluster.setMasterPassword("Secret123!");+ cluster.setClusterStatus("available");+ cluster.setSnapshotRegion(region);+ cluster.setAutomatedSnapshotAnchor(ANCHOR);+ if (definition != null) {+ SnapshotSchedule schedule = new SnapshotSchedule("schedule-" + id, null, List.of(definition));+ schedule.setUpdatedAt(ANCHOR);+ fixture.schedules().putForAccount(account, schedule.getScheduleIdentifier(), schedule);+ cluster.setSnapshotScheduleIdentifier(schedule.getScheduleIdentifier());+ }+ fixture.clusters().putForAccount(account, id, cluster);+ return cluster;+ }++ private record Fixture(RedshiftService service, RedshiftContainerManager manager,+ AccountAwareStorageBackend<AutomatedSnapshotState> states,+ AccountAwareStorageBackend<Cluster> clusters,+ AccountAwareStorageBackend<Snapshot> snapshots,+ AccountAwareStorageBackend<SnapshotSchedule> schedules) { }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.javaindex 9ffeaadc7d..5a166cd918 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.redshift;import io.github.hectorvent.floci.services.iam.IamService;import io.github.hectorvent.floci.services.redshift.model.Cluster;+import io.github.hectorvent.floci.services.redshift.model.Snapshot;import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.s3.model.S3Object;import io.quarkus.test.junit.QuarkusTest;@@ -25,10 +26,12 @@ import java.sql.ResultSet;import java.sql.SQLException;import java.sql.Statement;import java.time.Duration;+import java.time.Instant;import java.util.Comparator;import java.util.List;import java.util.Map;import java.util.Objects;+import java.util.UUID;import java.util.zip.GZIPInputStream;import java.util.zip.GZIPOutputStream;@@ -52,6 +55,9 @@ class RedshiftInterceptorIntegrationTest {@InjectRedshiftService service;+ @Inject+ RedshiftSnapshotScheduler snapshotScheduler;+@InjectS3Service s3;@@ -72,6 +78,8 @@ class RedshiftInterceptorIntegrationTest {void deleteSharedCluster() {if (sharedClusterId != null) {service.deleteCluster(sharedClusterId);+ // Drops the automated snapshot dumps taken from the deleted cluster.+ snapshotScheduler.tick(Instant.now());}}@@ -694,5 +702,42 @@ class RedshiftInterceptorIntegrationTest {assertTrue(ex.getMessage().toLowerCase().contains("does not exist") || ex.getMessage().toLowerCase().contains("missing.csv"), ex.getMessage());}}-}+ @Test+ void automatedDumpRestoresTheRowsPresentAtSnapshotTime() throws Exception {+ String suffix = UUID.randomUUID().toString().substring(0, 8);+ String table = "snapshot_rows_" + suffix;+ String restoredId = "it-auto-restored-" + suffix;+ String scheduleId = "auto-hourly-" + suffix;+ service.createSnapshotSchedule(scheduleId, null, List.of("rate(1 hour)"), Map.of());+ boolean restored = false;+ try {+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");+ Statement statement = connection.createStatement()) {+ statement.execute("CREATE TABLE " + table + " (id integer)");+ statement.execute("INSERT INTO " + table + " VALUES (42)");+ Cluster associated = service.modifyClusterSnapshotSchedule(sharedClusterId, scheduleId, false);+ snapshotScheduler.tick(associated.getAutomatedSnapshotAnchor().plusSeconds(3600));+ statement.execute("INSERT INTO " + table + " VALUES (99)");+ }+ Snapshot snapshot = service.describeSnapshots(null, sharedClusterId).stream()+ .filter(value -> "automated".equals(value.getSnapshotType())).findFirst().orElseThrow();+ Cluster target = service.restoreFromClusterSnapshot(restoredId, snapshot.getSnapshotIdentifier());+ restored = true;+ try (Connection connection = waitForConnection(target, "admin", "Secret123");+ Statement statement = connection.createStatement();+ ResultSet rows = statement.executeQuery("SELECT count(*), min(id), max(id) FROM " + table)) {+ assertTrue(rows.next());+ assertEquals(1, rows.getInt(1));+ assertEquals(42, rows.getInt(2));+ assertEquals(42, rows.getInt(3));+ }+ } finally {+ if (restored) {+ service.deleteCluster(restoredId);+ }+ service.modifyClusterSnapshotSchedule(sharedClusterId, null, true);+ service.deleteSnapshotSchedule(scheduleId);+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.javaindex 6d81f89ae8..26a77be54f 100644--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java@@ -2653,6 +2653,16 @@ class RedshiftServiceTest {when(snapshotScheduleBackend.get("res")).thenReturn(Optional.of(schedule));}+ @Test+ void snapshotScheduleRejectsOutOfRangeCronFields() {+ when(snapshotScheduleBackend.get("invalid-cron")).thenReturn(Optional.empty());++ AwsException failure = assertThrows(AwsException.class, () -> service.createSnapshotSchedule(+ "invalid-cron", null, List.of("cron(60 12 *)"), Map.of()));++ assertEquals("InvalidSchedule", failure.getErrorCode());+ }+private static String extractResourceId(String arn) {String resource = arn.substring(arn.lastIndexOf(':') + 1);return resource.contains("/") ? resource.substring(resource.lastIndexOf('/') + 1) : resource;diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduleTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduleTest.javanew file mode 100644index 0000000000..1ea463f3c6--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduleTest.java@@ -0,0 +1,63 @@+package io.github.hectorvent.floci.services.redshift;++import io.github.hectorvent.floci.core.common.AwsException;+import org.junit.jupiter.api.Test;++import java.time.Instant;+import java.util.List;++import static org.junit.jupiter.api.Assertions.*;++class RedshiftSnapshotScheduleTest {+ private static final Instant ANCHOR = Instant.parse("2026-10-25T00:00:00Z");++ @Test+ void ratesUseTheAnchorAndCoalesceMissedOccurrences() {+ assertTrue(RedshiftSnapshotSchedule.latestDue(List.of("rate(2 hours)"), ANCHOR,+ ANCHOR.plusSeconds(7199)).isEmpty());+ assertEquals(ANCHOR.plusSeconds(7200), RedshiftSnapshotSchedule.latestDue(List.of("rate(2 hours)"),+ ANCHOR, ANCHOR.plusSeconds(7200)).orElseThrow());+ assertEquals(ANCHOR.plusSeconds(21600), RedshiftSnapshotSchedule.latestDue(List.of("rate(2 hours)"),+ ANCHOR, ANCHOR.plusSeconds(25000)).orElseThrow());+ }++ @Test+ void shorthandAndFullCronUseUtcAndAwsWeekdayNumbers() {+ Instant expected = Instant.parse("2026-10-26T09:30:00Z");+ for (String expression : List.of("cron(30 9 MON-FRI)", "cron(30 9 ? * 2-6 *)")) {+ assertEquals(expected, RedshiftSnapshotSchedule.latestDue(List.of(expression), ANCHOR,+ expected).orElseThrow());+ }+ Instant saturday = Instant.parse("2026-10-31T09:00:00Z");+ assertEquals(saturday, RedshiftSnapshotSchedule.latestDue(List.of("cron(0 9 7)"), ANCHOR,+ saturday).orElseThrow());+ }++ @Test+ void cronRangesStepsAndMultipleDefinitionsChooseLatestOccurrence() {+ Instant now = Instant.parse("2026-10-27T18:15:00Z");+ assertEquals(Instant.parse("2026-10-27T18:00:00Z"), RedshiftSnapshotSchedule.latestDue(+ List.of("cron(0 14-20/1 TUE)", "cron(15 15/2 *)"), ANCHOR, now).orElseThrow());+ }++ @Test+ void fullCronSupportsThePublishedYearRange() {+ Instant future = Instant.parse("2199-01-01T00:00:00Z");+ assertEquals(future, RedshiftSnapshotSchedule.latestDue(List.of("cron(0 0 1 JAN ? 2199)"), ANCHOR,+ future).orElseThrow());+ }++ @Test+ void validatesFrequencyAndFieldRanges() {+ for (String expression : List.of("rate(0 hours)", "rate(25 hours)", "rate(2 days)",+ "cron(60 12 *)", "cron(* * *)", "cron(0 25 *)", "cron(0 12 BAD)",+ "cron(0 12 * * * *)", "rate(99999999999999999999999 days)")) {+ assertEquals("InvalidSchedule", assertThrows(AwsException.class,+ () -> RedshiftSnapshotSchedule.validate(List.of(expression)), expression).getErrorCode());+ }+ assertThrows(AwsException.class, () -> RedshiftSnapshotSchedule.validate(+ List.of("cron(0 12 *)", "cron(30 12 *)")));+ RedshiftSnapshotSchedule.validate(List.of("cron(0 12 *)", "cron(0 12 *)"));+ RedshiftSnapshotSchedule.validate(List.of("rate(1 day)"));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedulerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedulerTest.javanew file mode 100644index 0000000000..49e1a693e1--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedulerTest.java@@ -0,0 +1,33 @@+package io.github.hectorvent.floci.services.redshift;++import io.github.hectorvent.floci.config.EmulatorConfig;+import org.junit.jupiter.api.Test;++import java.time.Instant;++import static org.mockito.ArgumentMatchers.any;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++class RedshiftSnapshotSchedulerTest {+ private static final Instant NOW = Instant.parse("2026-10-25T00:00:00Z");++ @Test+ void tickIsSkippedBetweenBeforeResetAndAfterReset() {+ RedshiftService service = mock(RedshiftService.class);+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().redshift().snapshotSweepIntervalSeconds()).thenReturn(60L);+ RedshiftSnapshotScheduler scheduler = new RedshiftSnapshotScheduler(service, config);++ scheduler.beforeReset();+ scheduler.tick(NOW);+ verify(service, never()).runSnapshotSchedules(any());++ scheduler.afterReset();+ scheduler.tick(NOW);+ verify(service).runSnapshotSchedules(NOW);+ }+}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex cba3a9c43a..58c0e43fe0 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -211,6 +211,8 @@ floci:default-image: "valkey/valkey:8"redshift:enabled: true+ automated-snapshots-enabled: false+ snapshot-sweep-interval-seconds: 60default-port: 5439image-version: postgres:15-alpinepoll-interval-ms: 1000