Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(redshift): execute automated snapshot schedules (#6536)

Vũ Thế Huyf184cce726d7
diff --git a/docs/services/redshift.md b/docs/services/redshift.md
index 0c93f73745..923cfea112 100644
--- a/docs/services/redshift.md
+++ b/docs/services/redshift.md
@@ -23,7 +23,7 @@ For running SQL without a PostgreSQL wire connection (the way Lambda and Step Fu
| `DescribeClusters` | List clusters and their connection details |
| `DeleteCluster` | Create the requested final snapshot, then remove the cluster; accepts `SkipFinalClusterSnapshot` to delete without a backup |
| `CreateClusterSnapshot` | Back up a cluster to a SQL dump via `pg_dump` |
-| `DescribeClusterSnapshots` | List snapshots, optionally filtered by snapshot or cluster identifier |
+| `DescribeClusterSnapshots` | List snapshots, optionally filtered by snapshot or cluster identifier and SnapshotType (manual or automated) |
| `DeleteClusterSnapshot` | Remove a snapshot and its stored dump |
| `RestoreFromClusterSnapshot` | Create a new cluster and load a snapshot's dump into it via `psql`; accepts `SnapshotIdentifier` or a same-account, same-region `SnapshotArn` |
| `CreateClusterParameterGroup` | Register a parameter group (metadata only) |
@@ -48,7 +48,7 @@ For running SQL without a PostgreSQL wire connection (the way Lambda and Step Fu
| `EnableSnapshotCopy` | Record cross-region snapshot copy on a cluster (`RetentionPeriod` defaults to 7, `ManualSnapshotRetentionPeriod` to -1); `DescribeClusters` then reports `ClusterSnapshotCopyStatus`. A `DestinationRegion` that is not an AWS region is refused with `UnknownSnapshotCopyRegionFault`. No snapshot is copied |
| `DisableSnapshotCopy` | Clear a cluster's snapshot copy configuration |
| `ModifySnapshotCopyRetentionPeriod` | Change the automated (or, with `Manual`, the manual) snapshot copy retention period |
-| `CreateSnapshotSchedule` | Store a snapshot schedule of `rate(...)` / `cron(...)` definitions with optional tags; a `ScheduleIdentifier` other than letters, digits and hyphens is refused with `InvalidParameterValue`. No snapshots are taken on it |
+| `CreateSnapshotSchedule` | Create a UTC rate or cron schedule that takes automated snapshots on associated clusters, with optional tags |
| `ModifySnapshotSchedule` | Replace a schedule's definitions |
| `DescribeSnapshotSchedules` | List schedules filtered by `ScheduleIdentifier`, `ClusterIdentifier`, `TagKeys` and `TagValues`, paged with `MaxRecords` and `Marker`, with their associated clusters |
| `DeleteSnapshotSchedule` | Remove a schedule; refused with `InvalidClusterSnapshotScheduleState` while a cluster uses it |
@@ -194,6 +194,37 @@ and [COPY authorization parameters](https://docs.aws.amazon.com/redshift/latest/
`CreateTags`/`DeleteTags`/`DescribeTags` do not list that fault at all and return
`ResourceNotFoundFault` (HTTP 404) for any missing resource, grants included.
+## Automated snapshots
+
+Clusters take automated SQL-dump snapshots using their associated snapshot schedule. Use
+`CreateSnapshotSchedule` followed by `ModifyClusterSnapshotSchedule` to attach a schedule.
+Without an attached schedule, a cluster uses `rate(8 hours)`. Schedules run in UTC and accept
+`rate(1 hour)` through `rate(24 hours)`, `rate(1 day)`, Redshift's three-field shorthand such
+as `cron(0 9,17 MON-FRI)`, and full six-field cron expressions. Field ranges and near-term
+execution spacing are validated; overlapping executions less than one hour apart are rejected
+within a 400-day lookahead. The worker also enforces at least one hour between actual backups.
+
+`CreateCluster` and `ModifyCluster` accept `AutomatedSnapshotRetentionPeriod` from 0 to 35 days,
+with a default of 1. Zero disables automation and removes automated snapshots; RA3 and RG node
+families require 1 to 35 days. Decreasing retention cleans up expired automated snapshots.
+The newest automated snapshot is kept even after its retention period, until automation is
+disabled or the cluster is deleted. Manual snapshots are unaffected. `DescribeClusterSnapshots`
+returns `SnapshotType` and accepts `SnapshotType=automated` or `manual`. Automated snapshots
+can be restored with `RestoreFromClusterSnapshot`; `DeleteClusterSnapshot` refuses their
+manual deletion with `InvalidClusterSnapshotState`.
+
+The background sweep checks all owning accounts, takes dumps only for available clusters, and
+removes automated backups left by deleted clusters. Checkpoints use the Redshift storage mode
+and survive restart. A failed dump leaves its occurrence retryable and removes partial output.
+A committed dump is reused if saving its checkpoint failed. Cluster operations and concurrent
+sweeps share resource locks. Changing or reattaching a schedule starts a new scheduling anchor.
+
+These backups are full PostgreSQL dumps using the existing Docker backend. Floci does not
+implement AWS's data-change-based trigger or incremental snapshot storage. Missed occurrences
+are coalesced to one current backup, which contains the data at dump time, rather than replayed
+as historical backups. Older persisted clusters acquire their initial scheduling anchor on the
+first sweep. Cleanup after deleting a cluster runs on the next sweep.
+
## DynamoDB zero-ETL
The supported zero-ETL path is DynamoDB Streams to a provisioned Redshift cluster. The source ARN
@@ -285,6 +316,8 @@ For `AWS::Redshift::ClusterParameterGroup`, `Parameters` is applied via `ModifyC
| Variable | Default | Description |
|----------|---------|-------------|
| `FLOCI_SERVICES_REDSHIFT_ENABLED` | `true` | Enable or disable Redshift |
+| `FLOCI_SERVICES_REDSHIFT_AUTOMATED_SNAPSHOTS_ENABLED` | `true` | Run background automated snapshot and retention sweeps |
+| `FLOCI_SERVICES_REDSHIFT_SNAPSHOT_SWEEP_INTERVAL_SECONDS` | `60` | Delay between background snapshot sweeps |
| `FLOCI_SERVICES_REDSHIFT_IMAGE_VERSION` | `postgres:15-alpine` | PostgreSQL Docker image backing each cluster |
| `FLOCI_SERVICES_REDSHIFT_DEFAULT_PORT` | `5439` | Reported Redshift port hint (the real host port is dynamic and comes from `DescribeClusters`) |
| `FLOCI_SERVICES_REDSHIFT_PROXY_BASE_PORT` | `7100` | Lowest host port the per-cluster auth proxies bind |
@@ -696,7 +729,7 @@ These views expose the documented Redshift column names, types, and ordering map
- Multi-node clusters: `NodeType` and `NumberOfNodes` are stored as metadata; every cluster is a single PostgreSQL container.
- Parameter groups apply no real engine settings; values are stored and echoed back only.
- Subnet groups, VPC routing, and security groups are metadata only.
-- Resize, pause/resume, and IAM authentication. Snapshot schedules and cross-region snapshot copy are stored and reported, but no snapshot is taken on a schedule or copied to another region.
+- Resize, pause/resume, and IAM authentication. Cross-region snapshot copy is stored and reported, but no snapshot is copied to another region.
- Cross-account role ARNs are not supported for COPY or UNLOAD.
- The auth proxy validates the master user's password and any live `GetClusterCredentials` credential. Other non-master users pass straight through to PostgreSQL, which remains the authority for their credentials.
- A cluster using `ManageMasterPassword` keeps its generated password in sync with the
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index c1bdfe27d1..dd8a508eb7 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -1723,6 +1723,10 @@ public interface EmulatorConfig {
}
interface RedshiftServiceConfig {
+ @WithDefault("true")
+ boolean automatedSnapshotsEnabled();
+ @WithDefault("60")
+ long snapshotSweepIntervalSeconds();
@WithDefault("true")
boolean enabled();
@WithDefault("5439")
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java
index 468aecac0e..e51ff3b44d 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftQueryHandler.java
@@ -92,12 +92,23 @@ public class RedshiftQueryHandler {
String region = regionResolver.resolveRegionFromAuth(authorizationHeader);
String defaultIamRoleArn = params.getFirst("DefaultIamRoleArn");
- Cluster cluster = manageMasterPassword
- ? service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,
- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,
- params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn)
- : service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,
- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn);
+ Integer automatedRetention = parseOptionalInteger(params, "AutomatedSnapshotRetentionPeriod");
+ Cluster cluster;
+ if (manageMasterPassword) {
+ cluster = automatedRetention == null
+ ? service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,
+ params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn)
+ : service.createClusterWithManagedMasterPassword(identifier, nodeType, masterUsername,
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns,
+ params.getFirst("MasterPasswordSecretKmsKeyId"), region, defaultIamRoleArn, automatedRetention);
+ } else {
+ cluster = automatedRetention == null
+ ? service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn)
+ : service.createCluster(identifier, nodeType, masterUsername, masterUserPassword,
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, automatedRetention);
+ }
String parameterGroupName = params.getFirst("ClusterParameterGroupName");
String multiAz = params.getFirst("MultiAZ");
boolean hasParameterGroup = parameterGroupName != null && !parameterGroupName.isBlank();
@@ -177,6 +188,13 @@ public class RedshiftQueryHandler {
String snapshotIdentifier = params.getFirst("SnapshotIdentifier");
String clusterIdentifier = params.getFirst("ClusterIdentifier");
List<Snapshot> snapshots = service.describeSnapshots(snapshotIdentifier, clusterIdentifier);
+ String snapshotType = params.getFirst("SnapshotType");
+ if (snapshotType != null) {
+ if (!List.of("manual", "automated").contains(snapshotType)) {
+ throw new AwsException("InvalidParameterValue", "SnapshotType must be manual or automated", 400);
+ }
+ snapshots = snapshots.stream().filter(snapshot -> snapshotType.equals(snapshot.getSnapshotType())).toList();
+ }
XmlBuilder xmlBuilder = new XmlBuilder()
.start("DescribeClusterSnapshotsResponse")
.start("DescribeClusterSnapshotsResult")
@@ -680,9 +698,14 @@ public class RedshiftQueryHandler {
String clusterParameterGroupName = params.getFirst("ClusterParameterGroupName");
List<String> vpcSecurityGroupIds = memberList(params, "VpcSecurityGroupIds");
String multiAz = params.getFirst("MultiAZ");
- Cluster cluster = service.modifyCluster(clusterIdentifier, nodeType, numberOfNodes,
- masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds,
- multiAz == null ? null : Boolean.parseBoolean(multiAz));
+ Integer automatedRetention = parseOptionalInteger(params, "AutomatedSnapshotRetentionPeriod");
+ Cluster cluster = automatedRetention == null
+ ? service.modifyCluster(clusterIdentifier, nodeType, numberOfNodes,
+ masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds,
+ multiAz == null ? null : Boolean.parseBoolean(multiAz))
+ : service.modifyCluster(clusterIdentifier, nodeType, numberOfNodes,
+ masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds,
+ multiAz == null ? null : Boolean.parseBoolean(multiAz), automatedRetention);
String xml = new XmlBuilder()
.start("ModifyClusterResponse")
.start("ModifyClusterResult")
@@ -926,6 +949,7 @@ public class RedshiftQueryHandler {
.elem("NodeType", cluster.getNodeType())
.elem("MasterUsername", cluster.getMasterUsername())
.elem("ClusterStatus", cluster.getClusterStatus())
+ .elem("AutomatedSnapshotRetentionPeriod", String.valueOf(cluster.getAutomatedSnapshotRetentionPeriod()))
.elem("ClusterAvailabilityStatus", availabilityStatus(cluster.getClusterStatus()))
.elem("AvailabilityZoneRelocationStatus", "disabled")
// AWS always returns "enabled" or "disabled" here, never blank.
@@ -1063,6 +1087,7 @@ public class RedshiftQueryHandler {
.elem("ClusterIdentifier", snapshot.getClusterIdentifier())
.elem("SnapshotArn", snapshot.getSnapshotArn())
.elem("SnapshotCreateTime", createTime)
+ .elem("SnapshotType", snapshot.getSnapshotType())
.elem("ManualSnapshotRetentionPeriod", snapshot.getManualSnapshotRetentionPeriod().toString())
.elem("Status", snapshot.getStatus())
.elem("Port", String.valueOf(snapshot.getPort()))
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
index f0134bbaa3..2d132d28d9 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
@@ -16,6 +16,7 @@ import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader
import io.github.hectorvent.floci.services.rds.proxy.PasswordValidator;
import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;
import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;
+import io.github.hectorvent.floci.services.redshift.model.AutomatedSnapshotState;
import io.github.hectorvent.floci.services.redshift.model.Cluster;
import io.github.hectorvent.floci.services.redshift.model.ClusterParameterGroup;
import io.github.hectorvent.floci.services.redshift.model.ClusterSubnetGroup;
@@ -37,9 +38,11 @@ import jakarta.inject.Inject;
import org.jboss.logging.Logger;
import java.io.IOException;
+import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
+import java.time.Duration;
import java.time.Instant;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
@@ -49,6 +52,7 @@ import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;
@@ -56,7 +60,6 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Consumer;
import java.util.function.Supplier;
-import java.util.regex.Matcher;
import java.util.regex.Pattern;
@ApplicationScoped
@@ -83,6 +86,7 @@ public class RedshiftService implements ManagedSecretRotator {
private final AccountAwareStorageBackend<Integration> integrations;
private final AccountAwareStorageBackend<SnapshotCopyGrant> snapshotCopyGrants;
private final AccountAwareStorageBackend<SnapshotSchedule> snapshotSchedules;
+ private final AccountAwareStorageBackend<AutomatedSnapshotState> automatedSnapshotStates;
private final RedshiftContainerManager containerManager;
private final EmulatorConfig config;
private final RegionResolver regionResolver;
@@ -159,6 +163,8 @@ public class RedshiftService implements ManagedSecretRotator {
this.integrations = storageFactory.create("redshift", "redshift-integrations.json", new TypeReference<Map<String, Integration>>() {});
this.snapshotCopyGrants = storageFactory.create("redshift", "redshift-snapshot-copy-grants.json", new TypeReference<Map<String, SnapshotCopyGrant>>() {});
this.snapshotSchedules = storageFactory.create("redshift", "redshift-snapshot-schedules.json", new TypeReference<Map<String, SnapshotSchedule>>() {});
+ this.automatedSnapshotStates = storageFactory.create("redshift", "redshift-automated-snapshot-state.json",
+ new TypeReference<Map<String, AutomatedSnapshotState>>() {});
this.containerManager = containerManager;
this.config = config;
this.regionResolver = regionResolver;
@@ -265,13 +271,22 @@ public class RedshiftService implements ManagedSecretRotator {
public Cluster createCluster(String identifier, String nodeType, String username, String password,
String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,
List<String> iamRoleArns, String defaultIamRoleArn) {
+ return createCluster(identifier, nodeType, username, password, clusterSubnetGroupName,
+ vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, null);
+ }
+
+ public Cluster createCluster(String identifier, String nodeType, String username, String password,
+ String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,
+ List<String> iamRoleArns, String defaultIamRoleArn, Integer automatedRetention) {
+ validateAutomatedRetention(nodeType, automatedRetention == null ? 1 : automatedRetention);
return mutateCluster(identifier, () -> createClusterLocked(identifier, nodeType, username, password,
- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn));
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, automatedRetention));
}
private Cluster createClusterLocked(String identifier, String nodeType, String username, String password,
String clusterSubnetGroupName, List<String> vpcSecurityGroupIds,
- List<String> iamRoleArns, String defaultIamRoleArn) {
+ List<String> iamRoleArns, String defaultIamRoleArn,
+ Integer automatedRetention) {
if (clusters.get(identifier).isPresent()) {
throw new AwsException("ClusterAlreadyExists", "Cluster " + identifier + " already exists", 400);
}
@@ -284,6 +299,10 @@ public class RedshiftService implements ManagedSecretRotator {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier(identifier);
cluster.setNodeType(nodeType);
+ cluster.setAutomatedSnapshotRetentionPeriod(automatedRetention);
+ cluster.setAutomatedSnapshotAnchor(Instant.now());
+ cluster.setSnapshotGeneration(UUID.randomUUID().toString());
+ cluster.setSnapshotRegion(regionResolver.getRegion());
cluster.setMasterUsername(username);
cluster.setMasterPassword(password);
cluster.setClusterSubnetGroupName(clusterSubnetGroupName);
@@ -359,18 +378,31 @@ public class RedshiftService implements ManagedSecretRotator {
String identifier, String nodeType, String username, String clusterSubnetGroupName,
List<String> vpcSecurityGroupIds, List<String> iamRoleArns,
String kmsKeyId, String region, String defaultIamRoleArn) {
+ return createClusterWithManagedMasterPassword(identifier, nodeType, username, clusterSubnetGroupName,
+ vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region, defaultIamRoleArn, null);
+ }
+
+ public Cluster createClusterWithManagedMasterPassword(
+ String identifier, String nodeType, String username, String clusterSubnetGroupName,
+ List<String> vpcSecurityGroupIds, List<String> iamRoleArns,
+ String kmsKeyId, String region, String defaultIamRoleArn, Integer automatedRetention) {
+ validateAutomatedRetention(nodeType, automatedRetention == null ? 1 : automatedRetention);
return mutateCluster(identifier, () -> createClusterWithManagedMasterPasswordLocked(identifier,
- nodeType, username, clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region, defaultIamRoleArn));
+ nodeType, username, clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, kmsKeyId, region,
+ defaultIamRoleArn, automatedRetention));
}
private Cluster createClusterWithManagedMasterPasswordLocked(
String identifier, String nodeType, String username, String clusterSubnetGroupName,
List<String> vpcSecurityGroupIds, List<String> iamRoleArns,
- String kmsKeyId, String region, String defaultIamRoleArn) {
+ String kmsKeyId, String region, String defaultIamRoleArn, Integer automatedRetention) {
requireSecretsManager();
String password = RandomPasswordGenerator.generate(objectMapper.createObjectNode());
Cluster cluster = createCluster(identifier, nodeType, username, password,
- clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn);
+ clusterSubnetGroupName, vpcSecurityGroupIds, iamRoleArns, defaultIamRoleArn, automatedRetention);
+ if (region != null) {
+ cluster.setSnapshotRegion(region);
+ }
try {
return createManagedMasterSecret(cluster, kmsKeyId, region);
} catch (RuntimeException e) {
@@ -798,16 +830,32 @@ public class RedshiftService implements ManagedSecretRotator {
public Cluster modifyCluster(String clusterIdentifier, String nodeType, Integer numberOfNodes,
String masterUserPassword, String clusterParameterGroupName,
List<String> vpcSecurityGroupIds, Boolean multiAZ) {
- return mutateCluster(clusterIdentifier, () -> modifyClusterLocked(clusterIdentifier, nodeType,
- numberOfNodes, masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds, multiAZ));
+ return modifyCluster(clusterIdentifier, nodeType, numberOfNodes, masterUserPassword, clusterParameterGroupName,
+ vpcSecurityGroupIds, multiAZ, null);
+ }
+
+ public Cluster modifyCluster(String clusterIdentifier, String nodeType, Integer numberOfNodes,
+ String masterUserPassword, String clusterParameterGroupName,
+ List<String> vpcSecurityGroupIds, Boolean multiAZ, Integer automatedRetention) {
+ return withResourceLock("cluster", clusterIdentifier, () -> {
+ Cluster cluster = mutate(() -> modifyClusterLocked(clusterIdentifier, nodeType,
+ numberOfNodes, masterUserPassword, clusterParameterGroupName, vpcSecurityGroupIds, multiAZ,
+ automatedRetention));
+ if (automatedRetention != null) {
+ expireAutomatedSnapshots(clusters.accountId(), clusterIdentifier, cluster, Instant.now());
+ }
+ return cluster;
+ });
}
private Cluster modifyClusterLocked(String clusterIdentifier, String nodeType, Integer numberOfNodes,
String masterUserPassword, String clusterParameterGroupName,
- List<String> vpcSecurityGroupIds, Boolean multiAZ) {
+ List<String> vpcSecurityGroupIds, Boolean multiAZ,
+ Integer automatedRetention) {
Cluster cluster = clusters.get(clusterIdentifier)
.orElseThrow(() -> new AwsException("ClusterNotFound", "Cluster " + clusterIdentifier + " not found", 404));
-
+ validateAutomatedRetention(nodeType == null ? cluster.getNodeType() : nodeType,
+ automatedRetention == null ? cluster.getAutomatedSnapshotRetentionPeriod() : automatedRetention);
// alterUserPassword runs before any mutation of the cluster object (a live reference
// from HybridStorage, not a copy); if it throws, no metadata has been changed yet.
if (masterUserPassword != null && !masterUserPassword.isBlank()) {
@@ -836,6 +884,12 @@ public class RedshiftService implements ManagedSecretRotator {
if (multiAZ != null) {
cluster.setMultiAZ(multiAZ);
}
+ if (automatedRetention != null) {
+ if (cluster.getAutomatedSnapshotRetentionPeriod() == 0 && automatedRetention > 0) {
+ cluster.setAutomatedSnapshotAnchor(Instant.now());
+ }
+ cluster.setAutomatedSnapshotRetentionPeriod(automatedRetention);
+ }
clusters.put(clusterIdentifier, cluster);
clusters.flush();
@@ -1208,9 +1262,13 @@ public class RedshiftService implements ManagedSecretRotator {
/** Absolute, normalised {@code <persistentPath>/redshift-dumps/<accountId>} for this request's account. */
private Path accountDumpDir() {
+ return accountDumpDir(clusters.accountId());
+ }
+
+ private Path accountDumpDir(String accountId) {
return Paths.get(config.storage().persistentPath())
.resolve("redshift-dumps")
- .resolve(clusters.accountId())
+ .resolve(accountId)
.toAbsolutePath()
.normalize();
}
@@ -1239,27 +1297,43 @@ public class RedshiftService implements ManagedSecretRotator {
if (clusterOpt.isEmpty()) {
throw new AwsException("ClusterNotFound", "Cluster " + clusterIdentifier + " not found", 404);
}
- if (snapshots.get(snapshotIdentifier).isPresent()) {
+ return createSnapshot(clusters.accountId(), snapshotIdentifier, clusterOpt.get(), retentionPeriod,
+ Instant.now(), null);
+ }
+
+ private Snapshot createSnapshot(String accountId, String snapshotIdentifier, Cluster cluster,
+ Integer retentionPeriod, Instant now, Instant scheduledTime) {
+ validateSnapshotIdentifier(snapshotIdentifier);
+ String clusterIdentifier = cluster.getClusterIdentifier();
+ if ((scheduledTime == null ? snapshots.get(snapshotIdentifier)
+ : snapshots.getForAccount(accountId, snapshotIdentifier)).isPresent()) {
throw new AwsException("ClusterSnapshotAlreadyExists", "Snapshot " + snapshotIdentifier + " already exists", 400);
}
- Cluster cluster = clusterOpt.get();
Snapshot snapshot = new Snapshot();
snapshot.setSnapshotIdentifier(snapshotIdentifier);
snapshot.setClusterIdentifier(clusterIdentifier);
snapshot.setStatus("available");
snapshot.setMasterUsername(cluster.getMasterUsername());
snapshot.setMasterPassword(cluster.getMasterPassword());
- snapshot.setSnapshotArn(snapshotArn(clusterIdentifier, snapshotIdentifier));
- snapshot.setSnapshotCreateTime(Instant.now());
+ snapshot.setSnapshotArn(scheduledTime == null ? snapshotArn(clusterIdentifier, snapshotIdentifier)
+ : AwsArnUtils.Arn.of("redshift", snapshotRegion(cluster), accountId,
+ "snapshot:" + clusterIdentifier + "/" + snapshotIdentifier).toString());
+ snapshot.setSnapshotCreateTime(now);
snapshot.setManualSnapshotRetentionPeriod(retentionPeriod);
+ snapshot.setSnapshotType(scheduledTime == null ? "manual" : "automated");
+ snapshot.setScheduledTime(scheduledTime);
+ snapshot.setClusterGeneration(cluster.getSnapshotGeneration());
+ if (scheduledTime != null) {
+ snapshot.setAutomatedSnapshotRetentionPeriod(cluster.getAutomatedSnapshotRetentionPeriod());
+ }
if (cluster.getEndpoint() != null) {
snapshot.setPort(cluster.getEndpoint().getPort());
} else {
snapshot.setPort(5439);
}
- Path dumpDir = accountDumpDir();
+ Path dumpDir = accountDumpDir(accountId);
// Defence in depth: validateSnapshotIdentifier already rejects path separators,
// but keep the containment check so the dump can never land outside the account dir.
Path dumpFile = dumpDir.resolve(snapshotIdentifier + ".sql").normalize();
@@ -1269,9 +1343,13 @@ public class RedshiftService implements ManagedSecretRotator {
boolean committed = false;
try {
Files.createDirectories(dumpDir);
- containerManager.takeSnapshot(clusters.accountId(), clusterIdentifier, cluster.getMasterUsername(), dumpFile);
+ containerManager.takeSnapshot(accountId, clusterIdentifier, cluster.getMasterUsername(), dumpFile);
snapshot.setSqlDump(dumpFile.toString());
- snapshots.putAllAtomically(Map.of(snapshotIdentifier, snapshot));
+ if (scheduledTime == null) {
+ snapshots.putAllAtomically(Map.of(snapshotIdentifier, snapshot));
+ } else {
+ snapshots.putAllAtomicallyForAccount(accountId, Map.of(snapshotIdentifier, snapshot));
+ }
committed = true;
snapshots.flush();
} catch (Exception e) {
@@ -1328,6 +1406,9 @@ public class RedshiftService implements ManagedSecretRotator {
throw new AwsException("ClusterSnapshotNotFound", "Snapshot " + snapshotIdentifier + " not found", 404);
}
Snapshot snapshot = snapshotOpt.get();
+ if ("automated".equals(snapshot.getSnapshotType())) {
+ throw new AwsException("InvalidClusterSnapshotState", "Automated snapshots cannot be deleted manually", 400);
+ }
snapshots.delete(snapshotIdentifier);
snapshots.flush();
if (isTrustedDumpPath(snapshot.getSqlDump())) {
@@ -1439,6 +1520,9 @@ public class RedshiftService implements ManagedSecretRotator {
Cluster cluster = new Cluster();
cluster.setClusterIdentifier(clusterIdentifier);
cluster.setNodeType(effectiveNodeType);
+ cluster.setAutomatedSnapshotAnchor(Instant.now());
+ cluster.setSnapshotGeneration(UUID.randomUUID().toString());
+ cluster.setSnapshotRegion(regionResolver.getRegion());
cluster.setMasterUsername(username);
cluster.setMasterPassword(password);
cluster.setClusterParameterGroupName(DEFAULT_PARAMETER_GROUP_NAME);
@@ -1941,41 +2025,182 @@ public class RedshiftService implements ManagedSecretRotator {
private static final int SNAPSHOT_SCHEDULE_PAGE_MAX = 100;
- private static final Pattern RATE_DEFINITION =
- Pattern.compile("rate\\(\\s*(\\d+)\\s+(hours?|days?)\\s*\\)");
- private static final Pattern CRON_DEFINITION =
- Pattern.compile("cron\\(\\s*([A-Za-z0-9,\\-*/?#]+(?:\\s+[A-Za-z0-9,\\-*/?#]+){2,5})\\s*\\)");
- // The API reference allows only alphanumerics; the management guide's own example uses a hyphen.
- private static final Pattern SCHEDULE_IDENTIFIER = Pattern.compile("[A-Za-z0-9-]+");
+ private static void validateAutomatedRetention(String nodeType, int retention) {
+ boolean required = nodeType != null && (nodeType.startsWith("ra3.") || nodeType.startsWith("rg."));
+ if (retention < 0 || retention > 35 || required && retention == 0) {
+ throw new AwsException("InvalidParameterValue", "AutomatedSnapshotRetentionPeriod must be "
+ + (required ? "1" : "0") + " to 35 days", 400);
+ }
+ }
- /**
- * Shape check against the Redshift snapshot-schedule documentation: {@code rate(<n> hours|days)}
- * with a frequency between once an hour and once a day, or {@code cron(...)} with three to six
- * whitespace-separated fields drawn from the documented cron alphabet. Per-field value ranges
- * are not checked.
- */
- private static boolean isValidScheduleDefinition(String definition) {
- if (definition == null) {
- return false;
+ /** A deterministic sweep; the lifecycle scheduler supplies now, tests supply a fixed instant. */
+ void runSnapshotSchedules(Instant now) {
+ for (AccountAwareStorageBackend.AccountEntry<Cluster> entry : clusters.scanAllAccountEntries(key -> true)) {
+ if (Thread.currentThread().isInterrupted()) {
+ return;
+ }
+ try {
+ withResourceLock(entry.accountId(), "cluster", entry.key(), () -> {
+ Cluster cluster = clusters.getForAccount(entry.accountId(), entry.key()).orElse(null);
+ if (cluster != null) {
+ try {
+ runSnapshotSchedule(entry.accountId(), cluster, now);
+ } finally {
+ // Retention must not depend on the next dump succeeding.
+ expireAutomatedSnapshots(entry.accountId(), cluster.getClusterIdentifier(), cluster, now);
+ }
+ }
+ return null;
+ });
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not process automated snapshots for account {0}, cluster {1}",
+ entry.accountId(), entry.key());
+ }
}
- Matcher rate = RATE_DEFINITION.matcher(definition);
- if (rate.matches()) {
- long n = Long.parseLong(rate.group(1).length() > 9 ? "0" : rate.group(1));
- return rate.group(2).startsWith("hour") ? n >= 1 && n <= 24 : n == 1;
+ for (AccountAwareStorageBackend.AccountEntry<Snapshot> entry : snapshots.scanAllAccountEntries(key -> true)) {
+ if (!"automated".equals(entry.value().getSnapshotType())) {
+ continue;
+ }
+ String clusterId = entry.value().getClusterIdentifier();
+ try {
+ withResourceLock(entry.accountId(), "cluster", clusterId, () -> {
+ if (clusters.getForAccount(entry.accountId(), clusterId).isEmpty()) {
+ removeAutomatedSnapshot(entry.accountId(), entry.key());
+ }
+ return null;
+ });
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not clean up automated snapshot {0} in account {1}", entry.key(), entry.accountId());
+ }
+ }
+ for (AccountAwareStorageBackend.AccountEntry<AutomatedSnapshotState> entry
+ : automatedSnapshotStates.scanAllAccountEntries(key -> true)) {
+ try {
+ withResourceLock(entry.accountId(), "cluster", entry.key(), () -> {
+ if (clusters.getForAccount(entry.accountId(), entry.key()).isEmpty()) {
+ automatedSnapshotStates.deleteForAccount(entry.accountId(), entry.key());
+ }
+ return null;
+ });
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not remove the automated snapshot checkpoint of cluster {0} in account {1}",
+ entry.key(), entry.accountId());
+ }
}
- return CRON_DEFINITION.matcher(definition).matches();
}
- private static void validateScheduleDefinitions(List<String> definitions) {
- if (definitions == null || definitions.isEmpty()) {
- throw new AwsException("InvalidSchedule", "ScheduleDefinitions must contain at least one definition.", 400);
+
+ private void runSnapshotSchedule(String accountId, Cluster cluster, Instant now) {
+ if (!"available".equals(cluster.getClusterStatus()) || cluster.getAutomatedSnapshotRetentionPeriod() == 0) {
+ return;
+ }
+ SnapshotSchedule schedule = cluster.getSnapshotScheduleIdentifier() == null ? null
+ : snapshotSchedules.getForAccount(accountId, cluster.getSnapshotScheduleIdentifier()).orElseThrow(
+ () -> new AwsException("SnapshotScheduleNotFound", "The cluster's snapshot schedule is missing", 400));
+ List<String> definitions = schedule == null ? List.of("rate(8 hours)")
+ : List.copyOf(schedule.getScheduleDefinitions());
+ Instant activation = cluster.getAutomatedSnapshotAnchor();
+ if (schedule != null && schedule.getUpdatedAt() != null
+ && (activation == null || schedule.getUpdatedAt().isAfter(activation))) {
+ activation = schedule.getUpdatedAt();
+ }
+ String scheduleKey = cluster.getSnapshotGeneration() + "|" + cluster.getSnapshotScheduleIdentifier()
+ + "|" + definitions + "|" + activation;
+ String clusterId = cluster.getClusterIdentifier();
+ AutomatedSnapshotState state = automatedSnapshotStates.getForAccount(accountId, clusterId).orElse(null);
+ if (state == null || !scheduleKey.equals(state.scheduleKey())) {
+ state = new AutomatedSnapshotState(scheduleKey, activation == null ? now : activation, null);
+ automatedSnapshotStates.putAllAtomicallyForAccount(accountId, Map.of(clusterId, state));
+ }
+ Optional<Instant> due = RedshiftSnapshotSchedule.latestDue(definitions, state.anchor(), now);
+ if (due.isEmpty()) {
+ return;
+ }
+ Instant scheduled = due.get();
+ String identity = clusterId + "|" + state.anchor() + "|" + scheduled;
+ String snapshotId = "rs-auto-" + UUID.nameUUIDFromBytes(identity.getBytes(StandardCharsets.UTF_8));
+ Instant last = state.lastScheduledTime();
+ // The checkpoint lives in a separate store, so it can outlive a snapshot whose metadata never reached disk.
+ if (last != null && (scheduled.isBefore(last)
+ || scheduled.equals(last) && snapshots.getForAccount(accountId, snapshotId).isPresent())) {
+ return;
+ }
+ boolean tooSoon = snapshots.scanForAccount(accountId, key -> true).stream()
+ .filter(snapshot -> "automated".equals(snapshot.getSnapshotType()))
+ .filter(snapshot -> clusterId.equals(snapshot.getClusterIdentifier()))
+ .map(Snapshot::getSnapshotCreateTime).filter(time -> time != null)
+ .anyMatch(time -> time.plus(Duration.ofHours(1)).isAfter(now));
+ boolean completed = withResourceLock(accountId, "snapshot", snapshotId, () -> {
+ Snapshot existing = snapshots.getForAccount(accountId, snapshotId).orElse(null);
+ if (existing == null) {
+ if (tooSoon) {
+ return false;
+ }
+ createSnapshot(accountId, snapshotId, cluster, null, now, scheduled);
+ } else if (!"automated".equals(existing.getSnapshotType())
+ || !clusterId.equals(existing.getClusterIdentifier()) || !scheduled.equals(existing.getScheduledTime())) {
+ throw new AwsException("ClusterSnapshotAlreadyExists", "Snapshot identifier is already in use", 400);
+ }
+ return true;
+ });
+ if (!completed) {
+ return;
}
- for (String definition : definitions) {
- if (!isValidScheduleDefinition(definition)) {
- throw new AwsException("InvalidSchedule", "Invalid schedule definition: " + definition, 400);
+ automatedSnapshotStates.putAllAtomicallyForAccount(accountId, Map.of(clusterId,
+ new AutomatedSnapshotState(scheduleKey, state.anchor(), scheduled)));
+ }
+
+ private String snapshotRegion(Cluster cluster) {
+ return cluster.getSnapshotRegion() == null ? regionResolver.getDefaultRegion() : cluster.getSnapshotRegion();
+ }
+
+ private void expireAutomatedSnapshots(String accountId, String clusterId, Cluster cluster, Instant now) {
+ List<Snapshot> automated = snapshots.scanForAccount(accountId, key -> true).stream()
+ .filter(snapshot -> "automated".equals(snapshot.getSnapshotType()))
+ .filter(snapshot -> clusterId.equals(snapshot.getClusterIdentifier()))
+ .filter(snapshot -> snapshot.getSnapshotCreateTime() != null).toList();
+ Snapshot latest = automated.stream().max(Comparator.comparing(Snapshot::getSnapshotCreateTime)
+ .thenComparing(Snapshot::getSnapshotIdentifier)).orElse(null);
+ for (Snapshot snapshot : automated) {
+ boolean replacedCluster = cluster.getSnapshotGeneration() != null
+ && !Objects.equals(cluster.getSnapshotGeneration(), snapshot.getClusterGeneration());
+ if (replacedCluster || cluster.getAutomatedSnapshotRetentionPeriod() == 0 || snapshot != latest
+ && !snapshot.getSnapshotCreateTime().plus(Duration.ofDays(cluster.getAutomatedSnapshotRetentionPeriod()))
+ .isAfter(now)) {
+ removeAutomatedSnapshot(accountId, snapshot.getSnapshotIdentifier());
}
}
}
+ private void removeAutomatedSnapshot(String accountId, String snapshotId) {
+ withResourceLock(accountId, "snapshot", snapshotId, () -> mutate(() -> {
+ Snapshot current = snapshots.getForAccount(accountId, snapshotId).orElse(null);
+ if (current == null || !"automated".equals(current.getSnapshotType())) {
+ return null;
+ }
+ snapshots.deleteAllAtomicallyForAccounts(Map.of(accountId, Set.of(snapshotId)));
+ // Hybrid storage flushes lazily; a crash after the dump is gone would resurrect a dump-less snapshot.
+ snapshots.flush();
+ if (current.getSqlDump() != null) {
+ Path file = Paths.get(current.getSqlDump()).toAbsolutePath().normalize();
+ if (file.startsWith(accountDumpDir(accountId))) {
+ try {
+ Files.deleteIfExists(file);
+ } catch (IOException e) {
+ LOG.warnv(e, "Could not delete the dump of automated snapshot {0}", snapshotId);
+ }
+ }
+ }
+ return null;
+ }));
+ }
+
+ // The API reference allows only alphanumerics; the management guide's own example uses a hyphen.
+ private static final Pattern SCHEDULE_IDENTIFIER = Pattern.compile("[A-Za-z0-9-]+");
+
+ private static void validateScheduleDefinitions(List<String> definitions) {
+ RedshiftSnapshotSchedule.validate(definitions);
+ }
+
private SnapshotSchedule requireSnapshotSchedule(String scheduleIdentifier) {
return snapshotSchedules.get(scheduleIdentifier)
.orElseThrow(() -> new AwsException("SnapshotScheduleNotFound",
@@ -1996,6 +2221,7 @@ public class RedshiftService implements ManagedSecretRotator {
}
validateScheduleDefinitions(definitions);
SnapshotSchedule schedule = new SnapshotSchedule(id, description, definitions);
+ schedule.setUpdatedAt(Instant.now());
if (tags != null && !tags.isEmpty()) {
schedule.setTags(new LinkedHashMap<>(tags));
}
@@ -2043,9 +2269,11 @@ public class RedshiftService implements ManagedSecretRotator {
}
public synchronized SnapshotSchedule modifySnapshotSchedule(String scheduleIdentifier, List<String> definitions) {
- SnapshotSchedule schedule = requireSnapshotSchedule(scheduleIdentifier);
+ SnapshotSchedule current = requireSnapshotSchedule(scheduleIdentifier);
validateScheduleDefinitions(definitions);
- schedule.setScheduleDefinitions(new ArrayList<>(definitions));
+ SnapshotSchedule schedule = new SnapshotSchedule(scheduleIdentifier, current.getScheduleDescription(), definitions);
+ schedule.setTags(new LinkedHashMap<>(current.getTags()));
+ schedule.setUpdatedAt(Instant.now());
snapshotSchedules.put(scheduleIdentifier, schedule);
snapshotSchedules.flush();
return schedule;
@@ -2080,6 +2308,7 @@ public class RedshiftService implements ManagedSecretRotator {
requireSnapshotSchedule(scheduleIdentifier);
cluster.setSnapshotScheduleIdentifier(scheduleIdentifier);
}
+ cluster.setAutomatedSnapshotAnchor(Instant.now());
clusters.put(clusterIdentifier, cluster);
clusters.flush();
return cluster;
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedule.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedule.java
new file mode 100644
index 0000000000..b00b42bf0a
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedule.java
@@ -0,0 +1,116 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import com.cronutils.model.Cron;
+import com.cronutils.model.time.ExecutionTime;
+import io.github.hectorvent.floci.core.common.AwsCronDefinitions;
+import io.github.hectorvent.floci.core.common.AwsException;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.time.temporal.ChronoUnit;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Redshift's UTC snapshot rules, including its minute/hour/day-of-week shorthand. */
+final class RedshiftSnapshotSchedule {
+ private static final Pattern RATE = Pattern.compile("rate\\(\\s*(\\d+)\\s+(hours?|days?)\\s*\\)");
+ private static final Duration MINIMUM_INTERVAL = Duration.ofHours(1);
+
+ private RedshiftSnapshotSchedule() { }
+
+ static void validate(List<String> definitions) {
+ try {
+ if (definitions == null || definitions.isEmpty()) {
+ throw new IllegalArgumentException("ScheduleDefinitions must contain at least one definition");
+ }
+ Instant anchor = Instant.now().truncatedTo(ChronoUnit.DAYS);
+ List<Rule> rules = definitions.stream().map(RedshiftSnapshotSchedule::parse).toList();
+ List<Instant> next = new ArrayList<>();
+ for (Rule rule : rules) {
+ next.add(rule.next(anchor, anchor).orElse(null));
+ }
+ Instant previous = null;
+ Instant limit = null;
+ while (true) {
+ Instant earliest = next.stream().filter(value -> value != null).min(Instant::compareTo).orElse(null);
+ if (earliest == null || limit != null && earliest.isAfter(limit)) {
+ break;
+ }
+ if (previous != null && Duration.between(previous, earliest).compareTo(MINIMUM_INTERVAL) < 0) {
+ throw new IllegalArgumentException("Snapshot schedules must be at least one hour apart");
+ }
+ if (limit == null) {
+ limit = earliest.plus(Duration.ofDays(400));
+ }
+ previous = earliest;
+ for (int i = 0; i < rules.size(); i++) {
+ if (earliest.equals(next.get(i))) {
+ next.set(i, rules.get(i).next(anchor, earliest).orElse(null));
+ }
+ }
+ }
+ } catch (IllegalArgumentException e) {
+ throw new AwsException("InvalidSchedule", "Invalid snapshot schedule: " + e.getMessage(), 400);
+ }
+ }
+
+ static Optional<Instant> latestDue(List<String> definitions, Instant anchor, Instant now) {
+ return definitions.stream().map(RedshiftSnapshotSchedule::parse)
+ .map(rule -> rule.latest(anchor, now)).flatMap(Optional::stream)
+ .filter(time -> time.isAfter(anchor) && !time.isAfter(now)).max(Instant::compareTo);
+ }
+
+ private static Rule parse(String definition) {
+ if (definition == null) {
+ throw new IllegalArgumentException("A schedule definition cannot be null");
+ }
+ Matcher rate = RATE.matcher(definition);
+ if (rate.matches()) {
+ long value = Long.parseLong(rate.group(1));
+ boolean days = rate.group(2).startsWith("day");
+ if (value < 1 || days && value != 1 || !days && value > 24) {
+ throw new IllegalArgumentException("Snapshot rates must be between one hour and one day");
+ }
+ long hours = days ? 24 : value;
+ return new Rule(Duration.ofHours(hours), null);
+ }
+ if (!definition.startsWith("cron(") || !definition.endsWith(")")) {
+ throw new IllegalArgumentException("Expected rate(...) or cron(...)");
+ }
+ String[] fields = definition.substring(5, definition.length() - 1).trim().split("\\s+");
+ String normalized = switch (fields.length) {
+ case 3 -> fields[0] + " " + fields[1] + " ? * " + fields[2] + " *";
+ case 6 -> String.join(" ", fields);
+ default -> throw new IllegalArgumentException("Redshift cron needs three or six fields");
+ };
+ String[] full = normalized.split(" ");
+ if ("?".equals(full[2]) == "?".equals(full[4])) {
+ throw new IllegalArgumentException("Use ? in exactly one of day-of-month and day-of-week");
+ }
+ Cron cron = AwsCronDefinitions.newParser().parse("0 " + normalized);
+ cron.validate();
+ return new Rule(null, ExecutionTime.forCron(cron));
+ }
+
+ private record Rule(Duration rate, ExecutionTime cron) {
+ Optional<Instant> latest(Instant anchor, Instant now) {
+ if (rate != null) {
+ long count = Duration.between(anchor, now).getSeconds() / rate.getSeconds();
+ return count < 1 ? Optional.empty() : Optional.of(anchor.plus(rate.multipliedBy(count)));
+ }
+ return cron.lastExecution(now.atZone(ZoneOffset.UTC).plusNanos(1)).map(value -> value.toInstant());
+ }
+
+ Optional<Instant> next(Instant anchor, Instant after) {
+ if (rate != null) {
+ long count = Math.max(0, Duration.between(anchor, after).getSeconds() / rate.getSeconds()) + 1;
+ return Optional.of(anchor.plus(rate.multipliedBy(count)));
+ }
+ return cron.nextExecution(after.atZone(ZoneOffset.UTC)).map(value -> value.toInstant());
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduler.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduler.java
new file mode 100644
index 0000000000..f3c590cc08
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduler.java
@@ -0,0 +1,96 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.Resettable;
+import io.quarkus.runtime.ShutdownEvent;
+import io.quarkus.runtime.StartupEvent;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.enterprise.event.Observes;
+import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+
+import java.time.Instant;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+/** Drives automated snapshots outside AWS request threads. */
+@ApplicationScoped
+public class RedshiftSnapshotScheduler implements Resettable {
+ private static final Logger LOG = Logger.getLogger(RedshiftSnapshotScheduler.class);
+ private final RedshiftService service;
+ private final ScheduledExecutorService executor;
+ private final long intervalSeconds;
+ private final ReentrantLock sweepLock = new ReentrantLock();
+ private volatile boolean paused;
+
+ @Inject
+ public RedshiftSnapshotScheduler(RedshiftService service, EmulatorConfig config) {
+ this.service = service;
+ this.intervalSeconds = Math.max(1, config.services().redshift().snapshotSweepIntervalSeconds());
+ this.executor = config.services().redshift().enabled() && config.services().redshift().automatedSnapshotsEnabled()
+ ? Executors.newSingleThreadScheduledExecutor(task -> {
+ Thread thread = new Thread(task, "redshift-snapshot-scheduler");
+ thread.setDaemon(true);
+ return thread;
+ }) : null;
+ }
+
+ void onStart(@Observes StartupEvent ignored) {
+ if (executor != null) {
+ executor.scheduleWithFixedDelay(this::tickSafely, intervalSeconds, intervalSeconds, TimeUnit.SECONDS);
+ }
+ }
+
+ void onStop(@Observes ShutdownEvent ignored) {
+ if (executor != null) {
+ executor.shutdownNow();
+ try {
+ if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {
+ LOG.warn("Redshift snapshot scheduler is still finishing a dump during shutdown");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.debugv(e, "Interrupted while stopping the Redshift snapshot scheduler");
+ }
+ }
+ }
+
+ public void tick(Instant now) {
+ sweepLock.lock();
+ try {
+ if (!paused) {
+ service.runSnapshotSchedules(now);
+ }
+ } finally {
+ sweepLock.unlock();
+ }
+ }
+
+ /** Drains a running sweep so it cannot write old snapshots back into storage that reset is about to clear. */
+ @Override
+ public void beforeReset() {
+ paused = true;
+ sweepLock.lock();
+ sweepLock.unlock();
+ }
+
+ @Override
+ public void clear() {
+ // Snapshot state lives in RedshiftService storage, which reset wipes.
+ }
+
+ @Override
+ public void afterReset() {
+ paused = false;
+ }
+
+ private void tickSafely() {
+ try {
+ tick(Instant.now());
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Redshift automated snapshot sweep failed");
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/AutomatedSnapshotState.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/AutomatedSnapshotState.java
new file mode 100644
index 0000000000..3962bf7574
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/AutomatedSnapshotState.java
@@ -0,0 +1,9 @@
+package io.github.hectorvent.floci.services.redshift.model;
+
+import io.quarkus.runtime.annotations.RegisterForReflection;
+
+import java.time.Instant;
+
+/** Restart checkpoint for a cluster's currently effective snapshot rules. */
+@RegisterForReflection
+public record AutomatedSnapshotState(String scheduleKey, Instant anchor, Instant lastScheduledTime) { }
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java
index 427e56fee4..cdca5ea596 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Cluster.java
@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.redshift.model;
import io.quarkus.runtime.annotations.RegisterForReflection;
+import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
@@ -90,6 +91,22 @@ public class Cluster {
private Integer snapshotCopyManualRetentionPeriod;
private String snapshotCopyGrantName;
private String snapshotScheduleIdentifier;
+ private Instant automatedSnapshotAnchor;
+ private String snapshotRegion;
+ private Integer automatedSnapshotRetentionPeriod;
+ private String snapshotGeneration;
+
+ public String getSnapshotGeneration() { return snapshotGeneration; }
+ public void setSnapshotGeneration(String value) { snapshotGeneration = value; }
+
+ public Instant getAutomatedSnapshotAnchor() { return automatedSnapshotAnchor; }
+ public void setAutomatedSnapshotAnchor(Instant value) { automatedSnapshotAnchor = value; }
+ public String getSnapshotRegion() { return snapshotRegion; }
+ public void setSnapshotRegion(String value) { snapshotRegion = value; }
+ public int getAutomatedSnapshotRetentionPeriod() {
+ return automatedSnapshotRetentionPeriod == null ? 1 : automatedSnapshotRetentionPeriod;
+ }
+ public void setAutomatedSnapshotRetentionPeriod(Integer value) { automatedSnapshotRetentionPeriod = value; }
public String getSnapshotCopyDestinationRegion() { return snapshotCopyDestinationRegion; }
public void setSnapshotCopyDestinationRegion(String snapshotCopyDestinationRegion) { this.snapshotCopyDestinationRegion = snapshotCopyDestinationRegion; }
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java
index ec95d52197..43eca6efe0 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Snapshot.java
@@ -18,6 +18,20 @@ public class Snapshot {
private String snapshotArn;
private Instant snapshotCreateTime;
private Integer manualSnapshotRetentionPeriod;
+ private String snapshotType;
+ private Integer automatedSnapshotRetentionPeriod;
+ private Instant scheduledTime;
+ private String clusterGeneration;
+
+ public String getClusterGeneration() { return clusterGeneration; }
+ public void setClusterGeneration(String value) { clusterGeneration = value; }
+
+ public String getSnapshotType() { return snapshotType == null ? "manual" : snapshotType; }
+ public void setSnapshotType(String value) { snapshotType = value; }
+ public Integer getAutomatedSnapshotRetentionPeriod() { return automatedSnapshotRetentionPeriod; }
+ public void setAutomatedSnapshotRetentionPeriod(Integer value) { automatedSnapshotRetentionPeriod = value; }
+ public Instant getScheduledTime() { return scheduledTime; }
+ public void setScheduledTime(Instant value) { scheduledTime = value; }
public Snapshot() {}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java
index d746013c17..3a6df88421 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/SnapshotSchedule.java
@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.redshift.model;
import io.quarkus.runtime.annotations.RegisterForReflection;
+import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
@@ -17,6 +18,10 @@ public class SnapshotSchedule {
private String scheduleDescription;
private List<String> scheduleDefinitions = new ArrayList<>();
private Map<String, String> tags = new LinkedHashMap<>();
+ private Instant updatedAt;
+
+ public Instant getUpdatedAt() { return updatedAt; }
+ public void setUpdatedAt(Instant value) { updatedAt = value; }
public SnapshotSchedule() {}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index b20001e314..8b30358898 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -533,6 +533,8 @@ floci:
default-image: "valkey/valkey:8"
redshift:
enabled: true
+ automated-snapshots-enabled: true
+ snapshot-sweep-interval-seconds: 60
default-port: 5439
image-version: postgres:15-alpine
poll-interval-ms: 1000
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotIntegrationTest.java
new file mode 100644
index 0000000000..dd22ef447f
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotIntegrationTest.java
@@ -0,0 +1,120 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;
+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;
+import io.github.hectorvent.floci.services.redshift.model.Cluster;
+import io.github.hectorvent.floci.services.redshift.proxy.RedshiftProxyManager;
+import io.quarkus.test.InjectMock;
+import io.quarkus.test.common.http.TestHTTPResource;
+import io.quarkus.test.junit.QuarkusTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.redshift.RedshiftClient;
+import software.amazon.awssdk.services.redshift.model.RedshiftException;
+import software.amazon.awssdk.services.redshift.model.Snapshot;
+
+import java.net.URI;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.*;
+import static org.mockito.Mockito.*;
+
+@QuarkusTest
+class RedshiftAutomatedSnapshotIntegrationTest {
+ private static final String ACCOUNT = "333333333333";
+
+ @InjectMock
+ RedshiftContainerManager manager;
+ @InjectMock
+ RedshiftProxyManager proxy;
+ @Inject
+ RedshiftSnapshotScheduler scheduler;
+ @Inject
+ StorageFactory factory;
+ @TestHTTPResource
+ URI endpoint;
+
+ @BeforeEach
+ void setup() {
+ when(manager.start(anyString(), anyString(), anyString(), anyString(), any(), any()))
+ .thenReturn(new RedshiftContainerHandle("mock", "warehouse", "localhost", 5432));
+ doAnswer(invocation -> {
+ Files.writeString(invocation.getArgument(3, Path.class), "backup contents");
+ return null;
+ }).when(manager).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));
+ }
+
+ @Test
+ void sdkReportsAutomatedSnapshotsAndControlsTheirRetention() {
+ String clusterId = "sdk-automated-warehouse";
+ String scheduleId = "sdk-automated-hourly";
+ try (RedshiftClient client = client()) {
+ client.createSnapshotSchedule(request -> request.scheduleIdentifier(scheduleId).scheduleDefinitions("rate(1 hour)"));
+ client.createCluster(request -> request.clusterIdentifier(clusterId).nodeType("dc2.large")
+ .masterUsername("admin").masterUserPassword("Secret123").automatedSnapshotRetentionPeriod(1));
+ try {
+ client.modifyClusterSnapshotSchedule(request -> request.clusterIdentifier(clusterId).scheduleIdentifier(scheduleId));
+ AccountAwareStorageBackend<Cluster> clusters = factory.create("redshift", "redshift-clusters.json",
+ new TypeReference<Map<String, Cluster>>() { });
+ Instant due = clusters.getForAccount(ACCOUNT, clusterId).orElseThrow().getAutomatedSnapshotAnchor().plusSeconds(3600);
+ scheduler.tick(due);
+ List<Snapshot> automated = client.describeClusterSnapshots(request ->
+ request.clusterIdentifier(clusterId).snapshotType("automated")).snapshots();
+ assertEquals(1, automated.size());
+ Snapshot snapshot = automated.getFirst();
+ assertEquals("automated", snapshot.snapshotType());
+ assertEquals(due, snapshot.snapshotCreateTime());
+ assertTrue(snapshot.snapshotArn().contains(":" + ACCOUNT + ":snapshot:"));
+ assertTrue(client.describeClusterSnapshots(request ->
+ request.clusterIdentifier(clusterId).snapshotType("manual")).snapshots().isEmpty());
+ RedshiftException refused = assertThrows(RedshiftException.class, () ->
+ client.deleteClusterSnapshot(request -> request.snapshotIdentifier(snapshot.snapshotIdentifier())));
+ assertEquals("InvalidClusterSnapshotState", refused.awsErrorDetails().errorCode());
+ client.createClusterSnapshot(request -> request.clusterIdentifier(clusterId).snapshotIdentifier("sdk-manual-keep"));
+ assertEquals(0, client.modifyCluster(request -> request.clusterIdentifier(clusterId)
+ .automatedSnapshotRetentionPeriod(0)).cluster().automatedSnapshotRetentionPeriod());
+ scheduler.tick(due.plusSeconds(60));
+ assertTrue(client.describeClusterSnapshots(request ->
+ request.clusterIdentifier(clusterId).snapshotType("automated")).snapshots().isEmpty());
+ assertEquals("manual", client.describeClusterSnapshots(request -> request.snapshotIdentifier("sdk-manual-keep"))
+ .snapshots().getFirst().snapshotType());
+ client.deleteClusterSnapshot(request -> request.snapshotIdentifier("sdk-manual-keep"));
+ } finally {
+ client.deleteCluster(request -> request.clusterIdentifier(clusterId).skipFinalClusterSnapshot(true));
+ client.deleteSnapshotSchedule(request -> request.scheduleIdentifier(scheduleId));
+ scheduler.tick(Instant.now().plusSeconds(7200));
+ }
+ }
+ }
+
+ @Test
+ void sdkRejectsInvalidRetentionAndTooFrequentCron() {
+ try (RedshiftClient client = client()) {
+ RedshiftException retention = assertThrows(RedshiftException.class, () -> client.createCluster(request ->
+ request.clusterIdentifier("invalid-auto-retention").nodeType("ra3.xlplus")
+ .masterUsername("admin").masterUserPassword("Secret123").automatedSnapshotRetentionPeriod(0)));
+ assertEquals("InvalidParameterValue", retention.awsErrorDetails().errorCode());
+ RedshiftException cron = assertThrows(RedshiftException.class, () -> client.createSnapshotSchedule(request ->
+ request.scheduleIdentifier("invalid-auto-cron").scheduleDefinitions("cron(* * *)")));
+ assertEquals("InvalidSchedule", cron.awsErrorDetails().errorCode());
+ verify(manager, never()).start(anyString(), anyString(), anyString(), anyString(), any(), any());
+ }
+ }
+
+ private RedshiftClient client() {
+ return RedshiftClient.builder().endpointOverride(endpoint).region(Region.US_EAST_1)
+ .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCOUNT, "test"))).build();
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotServiceTest.java
new file mode 100644
index 0000000000..97d77155ae
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftAutomatedSnapshotServiceTest.java
@@ -0,0 +1,420 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
+import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;
+import io.github.hectorvent.floci.services.redshift.model.AutomatedSnapshotState;
+import io.github.hectorvent.floci.services.redshift.model.Cluster;
+import io.github.hectorvent.floci.services.redshift.model.Snapshot;
+import io.github.hectorvent.floci.services.redshift.model.SnapshotSchedule;
+import io.github.hectorvent.floci.services.redshift.proxy.RedshiftProxyManager;
+import org.awaitility.Awaitility;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.*;
+import static org.mockito.Mockito.*;
+
+class RedshiftAutomatedSnapshotServiceTest {
+ private static final String ACCOUNT = "111111111111";
+ private static final Instant ANCHOR = Instant.parse("2026-10-25T00:00:00Z");
+
+ @TempDir
+ Path directory;
+
+ @Test
+ void rateSnapshotsAndCheckpointsSurviveRestartWithoutDuplicateDumps() throws Exception {
+ Fixture first = fixture();
+ seed(first, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ first.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ first.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ Snapshot snapshot = first.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();
+ assertEquals("automated", snapshot.getSnapshotType());
+ assertEquals("backup contents", Files.readString(Path.of(snapshot.getSqlDump())));
+ verify(first.manager()).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));
+
+ Fixture restarted = fixture();
+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ verifyNoInteractions(restarted.manager());
+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(6 * 3600));
+ assertEquals(2, restarted.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ verify(restarted.manager()).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));
+ }
+
+ @Test
+ void delayedTicksDoNotCreateTwoBackupsWithinOneHour() {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7199));
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));
+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(10799));
+ assertEquals(2, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ }
+
+ @Test
+ void backgroundSweepKeepsAccountsRegionsAndPartitionsSeparate() {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "same-name", "cn-north-1", "rate(1 hour)");
+ seed(fixture, "222222222222", "same-name", "us-gov-west-1", "rate(1 hour)");
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+
+ Snapshot china = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();
+ Snapshot gov = fixture.snapshots().scanForAccount("222222222222", key -> true).getFirst();
+ assertTrue(china.getSnapshotArn().startsWith("arn:aws-cn:redshift:cn-north-1:" + ACCOUNT + ":"));
+ assertTrue(gov.getSnapshotArn().startsWith("arn:aws-us-gov:redshift:us-gov-west-1:222222222222:"));
+ assertTrue(Path.of(china.getSqlDump()).startsWith(directory.resolve("redshift-dumps").resolve(ACCOUNT)));
+ assertTrue(Path.of(gov.getSqlDump()).startsWith(directory.resolve("redshift-dumps").resolve("222222222222")));
+ assertNotEquals(china.getSqlDump(), gov.getSqlDump());
+ }
+
+ @Test
+ void retentionPreservesTheLatestAutomatedSnapshotAndEveryManualSnapshot() throws Exception {
+ Fixture fixture = fixture();
+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ Snapshot first = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));
+ Snapshot latest = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).stream()
+ .filter(snapshot -> !snapshot.getSnapshotIdentifier().equals(first.getSnapshotIdentifier()))
+ .findFirst().orElseThrow();
+ Snapshot manual = fixture.service().createSnapshot("manual-backup", "warehouse");
+ cluster.setClusterStatus("paused");
+ fixture.clusters().putForAccount(ACCOUNT, "warehouse", cluster);
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));
+
+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, first.getSnapshotIdentifier()).isEmpty());
+ assertFalse(Files.exists(Path.of(first.getSqlDump())));
+ assertTrue(Files.exists(Path.of(latest.getSqlDump())));
+ assertTrue(Files.exists(Path.of(manual.getSqlDump())));
+ assertEquals(2, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ assertEquals("InvalidClusterSnapshotState", assertThrows(AwsException.class,
+ () -> fixture.service().deleteSnapshot(latest.getSnapshotIdentifier())).getErrorCode());
+
+ cluster.setAutomatedSnapshotRetentionPeriod(0);
+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(4)));
+ assertEquals(List.of(manual.getSnapshotIdentifier()), fixture.snapshots().scanForAccount(ACCOUNT, key -> true)
+ .stream().map(Snapshot::getSnapshotIdentifier).toList());
+ }
+
+ @Test
+ void missingClustersLoseAutomatedSnapshotsButKeepManualBackups() {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ fixture.service().createSnapshot("manual-backup", "warehouse");
+ fixture.clusters().deleteForAccount(ACCOUNT, "warehouse");
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));
+
+ assertEquals(List.of("manual-backup"), fixture.snapshots().scanForAccount(ACCOUNT, key -> true).stream()
+ .map(Snapshot::getSnapshotIdentifier).toList());
+ assertTrue(fixture.states().getForAccount(ACCOUNT, "warehouse").isEmpty());
+ }
+
+ @Test
+ void occurrenceIsRetakenWhenItsSnapshotMetadataWasLostButTheCheckpointSurvived() {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ Snapshot lost = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();
+ fixture.snapshots().deleteForAccount(ACCOUNT, lost.getSnapshotIdentifier());
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+
+ assertEquals(List.of(lost.getSnapshotIdentifier()), fixture.snapshots().scanForAccount(ACCOUNT, key -> true)
+ .stream().map(Snapshot::getSnapshotIdentifier).toList());
+ verify(fixture.manager(), times(2)).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));
+ }
+
+ @Test
+ void retentionStillRunsWhenTheNextDumpFails() throws Exception {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ Snapshot first = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));
+ Snapshot latest = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).stream()
+ .filter(snapshot -> !snapshot.getSnapshotIdentifier().equals(first.getSnapshotIdentifier()))
+ .findFirst().orElseThrow();
+ doThrow(new IllegalStateException("dump failed")).when(fixture.manager())
+ .takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));
+
+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, first.getSnapshotIdentifier()).isEmpty());
+ assertFalse(Files.exists(Path.of(first.getSqlDump())));
+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, latest.getSnapshotIdentifier()).isPresent());
+ }
+
+ @Test
+ void recreatedClusterDoesNotKeepThePreviousInstancesAutomatedBackups() {
+ Fixture fixture = fixture();
+ Cluster original = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ original.setSnapshotGeneration("first-instance");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ Cluster replacement = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ replacement.setSnapshotGeneration("replacement-instance");
+ replacement.setClusterStatus("creating");
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3601));
+
+ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());
+ }
+
+ @Test
+ void defaultScheduleAndLegacyAnchorAreStableAfterRestart() {
+ Fixture fixture = fixture();
+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", null);
+ cluster.setAutomatedSnapshotAnchor(null);
+ fixture.clusters().putForAccount(ACCOUNT, "warehouse", cluster);
+ fixture.service().runSnapshotSchedules(ANCHOR);
+ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());
+
+ Fixture restarted = fixture();
+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(8 * 3600));
+
+ assertEquals(1, restarted.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ }
+
+ @Test
+ void failedDumpIsCleanedUpAndTheSameOccurrenceCanRetry() throws Exception {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ AtomicInteger attempts = new AtomicInteger();
+ doAnswer(invocation -> {
+ Files.writeString(invocation.getArgument(3, Path.class), "partial dump");
+ if (attempts.incrementAndGet() == 1) {
+ throw new IllegalStateException("dump failed");
+ }
+ return null;
+ }).when(fixture.manager()).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));
+
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+
+ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());
+ try (Stream<Path> files = Files.list(directory.resolve("redshift-dumps").resolve(ACCOUNT))) {
+ assertEquals(0, files.count());
+ }
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ assertEquals(2, attempts.get());
+ }
+
+ @Test
+ void committedDumpIsReusedWhenCheckpointWriteFailed() {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ doAnswer(invocation -> {
+ Map<String, AutomatedSnapshotState> checkpoint = invocation.getArgument(1);
+ AutomatedSnapshotState state = checkpoint.get("warehouse");
+ if (state.lastScheduledTime() != null) {
+ throw new IllegalStateException("checkpoint write failed");
+ }
+ return invocation.callRealMethod();
+ }).when(fixture.states()).putAllAtomicallyForAccount(eq(ACCOUNT), anyMap());
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+
+ Fixture restarted = fixture();
+ restarted.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+
+ verifyNoInteractions(restarted.manager());
+ assertEquals(ANCHOR.plusSeconds(3600), restarted.states().getForAccount(ACCOUNT, "warehouse")
+ .orElseThrow().lastScheduledTime());
+ }
+
+ @Test
+ void scheduleChangesResetTheAnchorAndDoNotReplayOldOccurrences() {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ SnapshotSchedule changed = fixture.service().modifySnapshotSchedule("schedule-warehouse", List.of("rate(2 hours)"));
+ assertNotNull(changed.getUpdatedAt());
+ Instant activation = ANCHOR.plusSeconds(7200);
+ changed.setUpdatedAt(activation);
+ fixture.schedules().putForAccount(ACCOUNT, changed.getScheduleIdentifier(), changed);
+
+ fixture.service().runSnapshotSchedules(activation.plusSeconds(7199));
+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ fixture.service().runSnapshotSchedules(activation.plusSeconds(7200));
+ assertEquals(2, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ }
+
+ @Test
+ void failedAssociationPreservesTheExistingAnchorAndCheckpoint() {
+ Fixture fixture = fixture();
+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ AutomatedSnapshotState checkpoint = fixture.states().getForAccount(ACCOUNT, "warehouse").orElseThrow();
+
+ for (String scheduleId : List.of("missing", "")) {
+ assertThrows(AwsException.class,
+ () -> fixture.service().modifyClusterSnapshotSchedule("warehouse", scheduleId, false));
+ assertEquals(ANCHOR, cluster.getAutomatedSnapshotAnchor());
+ assertEquals("schedule-warehouse", cluster.getSnapshotScheduleIdentifier());
+ assertEquals(checkpoint, fixture.states().getForAccount(ACCOUNT, "warehouse").orElseThrow());
+ }
+ }
+
+ @Test
+ void sweepCannotReadNewRulesWithTheOldActivationTime() throws Exception {
+ Fixture fixture = fixture();
+ Instant activation = Instant.now().minusSeconds(7200);
+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(24 hours)");
+ cluster.setAutomatedSnapshotAnchor(activation);
+ SnapshotSchedule schedule = spy(fixture.schedules().getForAccount(ACCOUNT, "schedule-warehouse").orElseThrow());
+ schedule.setUpdatedAt(activation);
+ fixture.schedules().putForAccount(ACCOUNT, "schedule-warehouse", schedule);
+ CountDownLatch betweenSetters = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ doAnswer(invocation -> {
+ betweenSetters.countDown();
+ assertTrue(release.await(5, TimeUnit.SECONDS));
+ return invocation.callRealMethod();
+ }).when(schedule).setUpdatedAt(any());
+ try (ExecutorService executor = Executors.newSingleThreadExecutor()) {
+ Future<?> updating = executor.submit(() -> fixture.service().modifySnapshotSchedule(
+ "schedule-warehouse", List.of("rate(1 hour)")));
+ try {
+ Awaitility.await().atMost(Duration.ofSeconds(5))
+ .until(() -> betweenSetters.getCount() == 0 || updating.isDone());
+ fixture.service().runSnapshotSchedules(Instant.now().plusSeconds(1800));
+ assertTrue(fixture.snapshots().scanForAccount(ACCOUNT, key -> true).isEmpty());
+ } finally {
+ release.countDown();
+ }
+ updating.get(5, TimeUnit.SECONDS);
+ }
+ }
+
+ @Test
+ void failedRetentionCommitPreservesMetadataAndDumpForRetry() throws Exception {
+ Fixture fixture = fixture();
+ Cluster cluster = seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600));
+ Snapshot old = fixture.snapshots().scanForAccount(ACCOUNT, key -> true).getFirst();
+ fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(7200));
+ cluster.setClusterStatus("paused");
+ Path metadata = directory.resolve("redshift-snapshots.json");
+ Path saved = directory.resolve("snapshots-before-failure.json");
+ Files.move(metadata, saved);
+ Files.createDirectory(metadata);
+ Files.writeString(metadata.resolve("block-replacement"), "keep");
+ try {
+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));
+
+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, old.getSnapshotIdentifier()).isPresent());
+ assertTrue(Files.exists(Path.of(old.getSqlDump())));
+ } finally {
+ Files.delete(metadata.resolve("block-replacement"));
+ Files.delete(metadata);
+ Files.move(saved, metadata);
+ }
+ fixture.service().runSnapshotSchedules(ANCHOR.plus(Duration.ofDays(3)));
+ assertTrue(fixture.snapshots().getForAccount(ACCOUNT, old.getSnapshotIdentifier()).isEmpty());
+ assertFalse(Files.exists(Path.of(old.getSqlDump())));
+ assertEquals(1, fixture().snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ }
+
+ @Test
+ void concurrentSweepsTakeOnlyOneDumpForTheSameOccurrence() throws Exception {
+ Fixture fixture = fixture();
+ seed(fixture, ACCOUNT, "warehouse", "us-east-1", "rate(1 hour)");
+ CountDownLatch dumping = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ doAnswer(invocation -> {
+ dumping.countDown();
+ assertTrue(release.await(5, TimeUnit.SECONDS));
+ Files.writeString(invocation.getArgument(3, Path.class), "backup contents");
+ return null;
+ }).when(fixture.manager()).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));
+ try (ExecutorService executor = Executors.newFixedThreadPool(2)) {
+ Future<?> first = executor.submit(() -> fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600)));
+ try {
+ assertTrue(dumping.await(5, TimeUnit.SECONDS));
+ Future<?> second = executor.submit(() -> fixture.service().runSnapshotSchedules(ANCHOR.plusSeconds(3600)));
+ release.countDown();
+ first.get(5, TimeUnit.SECONDS);
+ second.get(5, TimeUnit.SECONDS);
+ } finally {
+ release.countDown();
+ }
+ }
+ verify(fixture.manager()).takeSnapshot(eq(ACCOUNT), eq("warehouse"), eq("admin"), any(Path.class));
+ assertEquals(1, fixture.snapshots().scanForAccount(ACCOUNT, key -> true).size());
+ }
+
+ private Fixture fixture() {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.storage().persistentPath()).thenReturn(directory.toString());
+ when(config.defaultAccountId()).thenReturn(ACCOUNT);
+ ServiceConfigAccess access = mock(ServiceConfigAccess.class);
+ when(access.storageMode("redshift")).thenReturn("persistent");
+ StorageFactory factory = spy(new StorageFactory(config, access));
+ AccountAwareStorageBackend<AutomatedSnapshotState> states = spy(factory.create("redshift",
+ "redshift-automated-snapshot-state.json", new TypeReference<Map<String, AutomatedSnapshotState>>() { }));
+ doReturn(states).when(factory).create(eq("redshift"), eq("redshift-automated-snapshot-state.json"), any());
+ RedshiftContainerManager manager = mock(RedshiftContainerManager.class);
+ doAnswer(invocation -> {
+ Files.writeString(invocation.getArgument(3, Path.class), "backup contents");
+ return null;
+ }).when(manager).takeSnapshot(anyString(), anyString(), anyString(), any(Path.class));
+ RedshiftService service = new RedshiftService(factory, manager, config,
+ new RegionResolver("us-east-1", ACCOUNT), mock(RedshiftProxyManager.class),
+ mock(DockerHostResolver.class), new RedshiftCredentialBroker());
+ return new Fixture(service, manager, states,
+ factory.create("redshift", "redshift-clusters.json", new TypeReference<Map<String, Cluster>>() { }),
+ factory.create("redshift", "redshift-snapshots.json", new TypeReference<Map<String, Snapshot>>() { }),
+ factory.create("redshift", "redshift-snapshot-schedules.json", new TypeReference<Map<String, SnapshotSchedule>>() { }));
+ }
+
+ private static Cluster seed(Fixture fixture, String account, String id, String region, String definition) {
+ Cluster cluster = new Cluster();
+ cluster.setClusterIdentifier(id);
+ cluster.setNodeType("dc2.large");
+ cluster.setMasterUsername("admin");
+ cluster.setMasterPassword("Secret123!");
+ cluster.setClusterStatus("available");
+ cluster.setSnapshotRegion(region);
+ cluster.setAutomatedSnapshotAnchor(ANCHOR);
+ if (definition != null) {
+ SnapshotSchedule schedule = new SnapshotSchedule("schedule-" + id, null, List.of(definition));
+ schedule.setUpdatedAt(ANCHOR);
+ fixture.schedules().putForAccount(account, schedule.getScheduleIdentifier(), schedule);
+ cluster.setSnapshotScheduleIdentifier(schedule.getScheduleIdentifier());
+ }
+ fixture.clusters().putForAccount(account, id, cluster);
+ return cluster;
+ }
+
+ private record Fixture(RedshiftService service, RedshiftContainerManager manager,
+ AccountAwareStorageBackend<AutomatedSnapshotState> states,
+ AccountAwareStorageBackend<Cluster> clusters,
+ AccountAwareStorageBackend<Snapshot> snapshots,
+ AccountAwareStorageBackend<SnapshotSchedule> schedules) { }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
index 9ffeaadc7d..5a166cd918 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftInterceptorIntegrationTest.java
@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.redshift;
import io.github.hectorvent.floci.services.iam.IamService;
import io.github.hectorvent.floci.services.redshift.model.Cluster;
+import io.github.hectorvent.floci.services.redshift.model.Snapshot;
import io.github.hectorvent.floci.services.s3.S3Service;
import io.github.hectorvent.floci.services.s3.model.S3Object;
import io.quarkus.test.junit.QuarkusTest;
@@ -25,10 +26,12 @@ import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
+import java.time.Instant;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import java.util.UUID;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
@@ -52,6 +55,9 @@ class RedshiftInterceptorIntegrationTest {
@Inject
RedshiftService service;
+ @Inject
+ RedshiftSnapshotScheduler snapshotScheduler;
+
@Inject
S3Service s3;
@@ -72,6 +78,8 @@ class RedshiftInterceptorIntegrationTest {
void deleteSharedCluster() {
if (sharedClusterId != null) {
service.deleteCluster(sharedClusterId);
+ // Drops the automated snapshot dumps taken from the deleted cluster.
+ snapshotScheduler.tick(Instant.now());
}
}
@@ -694,5 +702,42 @@ class RedshiftInterceptorIntegrationTest {
assertTrue(ex.getMessage().toLowerCase().contains("does not exist") || ex.getMessage().toLowerCase().contains("missing.csv"), ex.getMessage());
}
}
-}
+ @Test
+ void automatedDumpRestoresTheRowsPresentAtSnapshotTime() throws Exception {
+ String suffix = UUID.randomUUID().toString().substring(0, 8);
+ String table = "snapshot_rows_" + suffix;
+ String restoredId = "it-auto-restored-" + suffix;
+ String scheduleId = "auto-hourly-" + suffix;
+ service.createSnapshotSchedule(scheduleId, null, List.of("rate(1 hour)"), Map.of());
+ boolean restored = false;
+ try {
+ try (Connection connection = waitForConnection(sharedCluster, "admin", "Secret123");
+ Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE " + table + " (id integer)");
+ statement.execute("INSERT INTO " + table + " VALUES (42)");
+ Cluster associated = service.modifyClusterSnapshotSchedule(sharedClusterId, scheduleId, false);
+ snapshotScheduler.tick(associated.getAutomatedSnapshotAnchor().plusSeconds(3600));
+ statement.execute("INSERT INTO " + table + " VALUES (99)");
+ }
+ Snapshot snapshot = service.describeSnapshots(null, sharedClusterId).stream()
+ .filter(value -> "automated".equals(value.getSnapshotType())).findFirst().orElseThrow();
+ Cluster target = service.restoreFromClusterSnapshot(restoredId, snapshot.getSnapshotIdentifier());
+ restored = true;
+ try (Connection connection = waitForConnection(target, "admin", "Secret123");
+ Statement statement = connection.createStatement();
+ ResultSet rows = statement.executeQuery("SELECT count(*), min(id), max(id) FROM " + table)) {
+ assertTrue(rows.next());
+ assertEquals(1, rows.getInt(1));
+ assertEquals(42, rows.getInt(2));
+ assertEquals(42, rows.getInt(3));
+ }
+ } finally {
+ if (restored) {
+ service.deleteCluster(restoredId);
+ }
+ service.modifyClusterSnapshotSchedule(sharedClusterId, null, true);
+ service.deleteSnapshotSchedule(scheduleId);
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
index 6d81f89ae8..26a77be54f 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
@@ -2653,6 +2653,16 @@ class RedshiftServiceTest {
when(snapshotScheduleBackend.get("res")).thenReturn(Optional.of(schedule));
}
+ @Test
+ void snapshotScheduleRejectsOutOfRangeCronFields() {
+ when(snapshotScheduleBackend.get("invalid-cron")).thenReturn(Optional.empty());
+
+ AwsException failure = assertThrows(AwsException.class, () -> service.createSnapshotSchedule(
+ "invalid-cron", null, List.of("cron(60 12 *)"), Map.of()));
+
+ assertEquals("InvalidSchedule", failure.getErrorCode());
+ }
+
private static String extractResourceId(String arn) {
String resource = arn.substring(arn.lastIndexOf(':') + 1);
return resource.contains("/") ? resource.substring(resource.lastIndexOf('/') + 1) : resource;
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduleTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduleTest.java
new file mode 100644
index 0000000000..1ea463f3c6
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotScheduleTest.java
@@ -0,0 +1,63 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+class RedshiftSnapshotScheduleTest {
+ private static final Instant ANCHOR = Instant.parse("2026-10-25T00:00:00Z");
+
+ @Test
+ void ratesUseTheAnchorAndCoalesceMissedOccurrences() {
+ assertTrue(RedshiftSnapshotSchedule.latestDue(List.of("rate(2 hours)"), ANCHOR,
+ ANCHOR.plusSeconds(7199)).isEmpty());
+ assertEquals(ANCHOR.plusSeconds(7200), RedshiftSnapshotSchedule.latestDue(List.of("rate(2 hours)"),
+ ANCHOR, ANCHOR.plusSeconds(7200)).orElseThrow());
+ assertEquals(ANCHOR.plusSeconds(21600), RedshiftSnapshotSchedule.latestDue(List.of("rate(2 hours)"),
+ ANCHOR, ANCHOR.plusSeconds(25000)).orElseThrow());
+ }
+
+ @Test
+ void shorthandAndFullCronUseUtcAndAwsWeekdayNumbers() {
+ Instant expected = Instant.parse("2026-10-26T09:30:00Z");
+ for (String expression : List.of("cron(30 9 MON-FRI)", "cron(30 9 ? * 2-6 *)")) {
+ assertEquals(expected, RedshiftSnapshotSchedule.latestDue(List.of(expression), ANCHOR,
+ expected).orElseThrow());
+ }
+ Instant saturday = Instant.parse("2026-10-31T09:00:00Z");
+ assertEquals(saturday, RedshiftSnapshotSchedule.latestDue(List.of("cron(0 9 7)"), ANCHOR,
+ saturday).orElseThrow());
+ }
+
+ @Test
+ void cronRangesStepsAndMultipleDefinitionsChooseLatestOccurrence() {
+ Instant now = Instant.parse("2026-10-27T18:15:00Z");
+ assertEquals(Instant.parse("2026-10-27T18:00:00Z"), RedshiftSnapshotSchedule.latestDue(
+ List.of("cron(0 14-20/1 TUE)", "cron(15 15/2 *)"), ANCHOR, now).orElseThrow());
+ }
+
+ @Test
+ void fullCronSupportsThePublishedYearRange() {
+ Instant future = Instant.parse("2199-01-01T00:00:00Z");
+ assertEquals(future, RedshiftSnapshotSchedule.latestDue(List.of("cron(0 0 1 JAN ? 2199)"), ANCHOR,
+ future).orElseThrow());
+ }
+
+ @Test
+ void validatesFrequencyAndFieldRanges() {
+ for (String expression : List.of("rate(0 hours)", "rate(25 hours)", "rate(2 days)",
+ "cron(60 12 *)", "cron(* * *)", "cron(0 25 *)", "cron(0 12 BAD)",
+ "cron(0 12 * * * *)", "rate(99999999999999999999999 days)")) {
+ assertEquals("InvalidSchedule", assertThrows(AwsException.class,
+ () -> RedshiftSnapshotSchedule.validate(List.of(expression)), expression).getErrorCode());
+ }
+ assertThrows(AwsException.class, () -> RedshiftSnapshotSchedule.validate(
+ List.of("cron(0 12 *)", "cron(30 12 *)")));
+ RedshiftSnapshotSchedule.validate(List.of("cron(0 12 *)", "cron(0 12 *)"));
+ RedshiftSnapshotSchedule.validate(List.of("rate(1 day)"));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedulerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedulerTest.java
new file mode 100644
index 0000000000..49e1a693e1
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftSnapshotSchedulerTest.java
@@ -0,0 +1,33 @@
+package io.github.hectorvent.floci.services.redshift;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class RedshiftSnapshotSchedulerTest {
+ private static final Instant NOW = Instant.parse("2026-10-25T00:00:00Z");
+
+ @Test
+ void tickIsSkippedBetweenBeforeResetAndAfterReset() {
+ RedshiftService service = mock(RedshiftService.class);
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().redshift().snapshotSweepIntervalSeconds()).thenReturn(60L);
+ RedshiftSnapshotScheduler scheduler = new RedshiftSnapshotScheduler(service, config);
+
+ scheduler.beforeReset();
+ scheduler.tick(NOW);
+ verify(service, never()).runSnapshotSchedules(any());
+
+ scheduler.afterReset();
+ scheduler.tick(NOW);
+ verify(service).runSnapshotSchedules(NOW);
+ }
+}
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index cba3a9c43a..58c0e43fe0 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -211,6 +211,8 @@ floci:
default-image: "valkey/valkey:8"
redshift:
enabled: true
+ automated-snapshots-enabled: false
+ snapshot-sweep-interval-seconds: 60
default-port: 5439
image-version: postgres:15-alpine
poll-interval-ms: 1000