Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(rds): pause and resume Aurora Serverless v2 clusters at zero capacity (#4232)

jwoa957e4d4dafae9d
diff --git a/docs/services/rds.md b/docs/services/rds.md
index 11f03c463e..357335851c 100644
--- a/docs/services/rds.md
+++ b/docs/services/rds.md
@@ -152,6 +152,8 @@ checked against the instance's other window. Modifications apply immediately —
| `FLOCI_SERVICES_RDS_PROXY_HANDSHAKE_TIMEOUT_MILLIS` | `10000` | Max time a client has to complete the startup/auth handshake before the proxy drops it |
| `FLOCI_SERVICES_RDS_PROXY_BACKEND_CONNECT_TIMEOUT_MILLIS` | `5000` | Max time the proxy waits for the backend TCP connect |
| `FLOCI_SERVICES_RDS_PROXY_MAX_CONNECTIONS` | `100` | Max concurrent connections per proxy before new ones are refused |
+| `FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED` | `true` | Pause an Aurora Serverless v2 cluster whose `MinCapacity` is 0 after `SecondsUntilAutoPause` without connections; see [Automatic pause and resume](#automatic-pause-and-resume) |
+| `FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS` | `0` | How long the first connection to a paused cluster is held while it resumes; Aurora takes about `15000` |
### Docker Compose
@@ -258,8 +260,47 @@ values. `AWS::RDS::DBCluster` creation also maps the equivalent CloudFormation p
If a persisted cluster record does not contain its original AWS engine identifier, Floci rejects a
new scaling configuration instead of assuming that the cluster is Aurora.
-This is control-plane compatibility: Floci persists and returns the scaling configuration, but it
-does not resize or automatically pause the backing Docker container.
+Floci persists and returns the scaling configuration but does not resize the backing Docker
+container: capacity between `MinCapacity` and `MaxCapacity` is not modeled. A `MinCapacity` of 0
+does take effect, as described next.
+
+### Automatic pause and resume
+
+A cluster whose `MinCapacity` is 0 pauses the way Aurora Serverless v2 does. Once nothing has used
+it for `SecondsUntilAutoPause`, Floci freezes its database container with `docker pause`, which
+keeps its data and its endpoint. The next connection resumes it: the client is held, not refused,
+until the database runs again, and is then served as usual. As on AWS, a connection attempt with
+wrong credentials also resumes the cluster.
+
+- **What keeps a cluster awake.** Every open connection through the cluster endpoint, the reader
+ endpoint, an instance endpoint or an RDS Proxy, every RDS Data API call and every open Data API
+ transaction. The idle clock starts when the last of them ends.
+- **Which clusters pause.** Aurora PostgreSQL and Aurora MySQL clusters with at least one instance,
+ all of them `db.serverless` and available. As on AWS, a cluster with a provisioned instance, a
+ cluster in a global database and a cluster registered with an RDS Proxy stay running. Floci runs
+ a cluster's instances in one container, so they pause and resume together, like Aurora's writer
+ and its tier 0 and 1 readers; a reader in tiers 2 to 15 does not pause on its own.
+- **What you can observe.** The cluster `Status` and each `DBInstanceStatus` stay `available`.
+ `DescribeEvents` reports RDS-EVENT-0370 to 0374 for each instance, for example "Successfully
+ paused the DB instance.". While the cluster is paused, `ServerlessDatabaseCapacity` reports 0
+ and `ACUUtilization` and `CPUUtilization` report 0 percent in the `AWS/RDS` namespace once a
+ minute, by `DBClusterIdentifier` and by `DBInstanceIdentifier`. Floci publishes none of them
+ while the cluster runs.
+- **Resume delay.** Aurora takes about 15 seconds to resume. Floci resumes at once unless
+ `FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS` is set; `15000` exercises client connect timeouts,
+ retries and connection-pool validation against a realistic cold start.
+- **Configuration changes.** A `ModifyDBCluster` that raises `MinCapacity` above 0 or changes
+ `SecondsUntilAutoPause` resumes a paused cluster and restarts its idle clock.
+- **Commands Floci runs in the container.** `CreateDBClusterSnapshot` and a master password change
+ run inside the database container, so they resume a paused cluster first. Aurora takes snapshots
+ from storage without resuming it.
+- Stopping, rebooting or deleting a cluster, restarting Floci and resetting its state never leave a
+ container frozen. A cluster that was paused when Floci stopped comes back running.
+- `FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED=false` keeps every cluster running.
+
+Not emulated: Data API requests to a paused cluster wait for the resume, as the Aurora user guide
+describes, and never fail with `DatabaseResumingException`; logical or binlog replication does not
+keep a cluster awake; and parameter group changes do not resume one.
## Modifying a DB instance
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 1996668dcc..4d9828839e 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -1560,6 +1560,18 @@ public interface EmulatorConfig {
int proxyBackendConnectTimeoutMillis();
@WithDefault("100")
int proxyMaxConnections();
+
+ /** Whether an Aurora Serverless v2 cluster with MinCapacity 0 pauses its container
+ * after SecondsUntilAutoPause without connections, as Aurora does.
+ * Env: FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED */
+ @WithDefault("true")
+ boolean auroraAutoPauseEnabled();
+
+ /** How long the first connection to an auto-paused cluster is held while it resumes.
+ * Aurora takes about 15 seconds; 0 resumes at once.
+ * Env: FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS */
+ @WithDefault("0")
+ int auroraResumeDelayMillis();
}
interface RdsDataServiceConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java b/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java
index 250b4ebd43..4fd139b6ac 100644
--- a/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java
@@ -20,9 +20,13 @@ import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.InMemoryStorage;
import io.github.hectorvent.floci.core.storage.StorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.cloudwatch.metrics.CloudWatchMetricsService;
+import io.github.hectorvent.floci.services.cloudwatch.metrics.model.Dimension;
+import io.github.hectorvent.floci.services.cloudwatch.metrics.model.MetricDatum;
import io.github.hectorvent.floci.services.ec2.Ec2Service;
import io.github.hectorvent.floci.services.ec2.model.Subnet;
import io.github.hectorvent.floci.services.ec2.model.Vpc;
+import io.github.hectorvent.floci.services.rds.container.AutoPauseListener;
import io.github.hectorvent.floci.services.rds.container.RdsContainerHandle;
import io.github.hectorvent.floci.services.rds.container.RdsContainerManager;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
@@ -180,11 +184,16 @@ public class RdsService implements Resettable, ResourceProvider {
private final DockerHostResolver dockerHostResolver;
private final CurrentContainerNetworkResolver currentContainerNetworkResolver;
private final ResourceGroupsTaggingService taggingService;
+ // Null when a test constructs the service without CloudWatch; auto-pause then reports no metrics.
+ private final CloudWatchMetricsService metricsService;
private final Set<Integer> usedPorts = ConcurrentHashMap.newKeySet();
private static final Pattern IMAGE_TAG_VERSION_PATTERN = Pattern.compile("^(\\d+(?:\\.\\d+)*)(.*)$");
private static final Pattern SAFE_IMAGE_TAG_PATTERN = Pattern.compile("[A-Za-z0-9._-]+");
private static final int SERVERLESS_V2_DEFAULT_AUTO_PAUSE_SECONDS = 300;
private static final int SERVERLESS_V2_MAX_AUTO_PAUSE_SECONDS = 86_400;
+ /** Only Aurora Serverless v2 instances pause; a provisioned one keeps its whole cluster awake. */
+ private static final String SERVERLESS_V2_INSTANCE_CLASS = "db.serverless";
+ private static final List<String> AUTO_PAUSE_EVENT_CATEGORIES = List.of("notification", "serverless");
private static final Pattern DB_PROXY_NAME_PATTERN =
Pattern.compile("[a-zA-Z](?:-?[a-zA-Z0-9]+)*");
/**
@@ -213,7 +222,8 @@ public class RdsService implements Resettable, ResourceProvider {
DockerHostResolver dockerHostResolver,
CurrentContainerNetworkResolver currentContainerNetworkResolver,
ResourceGroupsTaggingService taggingService,
- KmsService kmsService) {
+ KmsService kmsService,
+ CloudWatchMetricsService metricsService) {
this.containerManager = containerManager;
this.proxyManager = proxyManager;
this.ec2Service = ec2Service;
@@ -224,6 +234,7 @@ public class RdsService implements Resettable, ResourceProvider {
this.dockerHostResolver = dockerHostResolver;
this.currentContainerNetworkResolver = currentContainerNetworkResolver;
this.taggingService = taggingService;
+ this.metricsService = metricsService;
this.instances = storageFactory.create("rds", "rds-instances.json",
new TypeReference<Map<String, DbInstance>>() {});
this.clusters = storageFactory.create("rds", "rds-clusters.json",
@@ -381,6 +392,7 @@ public class RdsService implements Resettable, ResourceProvider {
this.proxies = proxies;
this.proxyTargetGroups = proxyTargetGroups;
this.taggingService = taggingService;
+ this.metricsService = null;
this.globalClusters = new io.github.hectorvent.floci.core.storage.InMemoryStorage<>();
this.snapshots = new io.github.hectorvent.floci.core.storage.InMemoryStorage<>();
this.clusterSnapshots = new InMemoryStorage<>();
@@ -2954,6 +2966,7 @@ public class RdsService implements Resettable, ResourceProvider {
(user, pw) -> validateDbClusterPasswordForScope(accountId, clusterRegion, id, user, pw),
proxyBinding(cluster.getEngine(), cluster.getEndpoint().address(), cluster.getProxyPort(),
clusterRegion, accountId, cluster.getDbClusterResourceId()));
+ applyAutoPause(cluster);
}
}
@@ -3183,6 +3196,7 @@ public class RdsService implements Resettable, ResourceProvider {
cluster.setContainerHost(started.getHost());
cluster.setContainerPort(started.getPort());
putClusterForScope(currentAccountId(), effectiveRegion, id, cluster);
+ applyAutoPause(cluster);
LOG.infov("Backing database container for DB cluster {0} started on retry", id);
return cluster;
}
@@ -3469,6 +3483,7 @@ public class RdsService implements Resettable, ResourceProvider {
}
throw e;
}
+ applyAutoPause(cluster);
LOG.infov("DB cluster {0} created (mock={1}), engine={2}, endpoint={3}:{4}",
id, String.valueOf(mock), engine, endpoint.address(), String.valueOf(endpoint.port()));
return cluster;
@@ -3547,6 +3562,175 @@ public class RdsService implements Resettable, ResourceProvider {
|| "aurora-postgresql".equalsIgnoreCase(engineIdentifier));
}
+ // ── Aurora Serverless v2 auto-pause ───────────────────────────────────────
+
+ /**
+ * Hands a cluster's auto-pause interval to its container: SecondsUntilAutoPause for an Aurora
+ * cluster whose MinCapacity is 0, otherwise none, which keeps the container running.
+ */
+ private void applyAutoPause(DbCluster cluster) {
+ if (config.services().rds().mock() || cluster.getDbClusterArn() == null) {
+ return;
+ }
+ Integer secondsUntilAutoPause = isAuroraEngine(cluster.getEngineIdentifier())
+ ? cluster.getServerlessV2SecondsUntilAutoPause()
+ : null;
+ containerManager.configureAutoPause(cluster.getDbClusterArn(), secondsUntilAutoPause,
+ new ClusterAutoPause(accountIdFromArn(cluster.getDbClusterArn()),
+ regionFromArn(cluster.getDbClusterArn()), cluster.getDbClusterIdentifier()));
+ }
+
+ /**
+ * Aurora's conditions for pausing an idle cluster besides its zero MinCapacity: the cluster
+ * and its instances are available, every instance is Aurora Serverless v2 (a provisioned one
+ * keeps the writer awake), and the cluster is neither in a global database nor the target of
+ * an RDS Proxy, which holds connections open to it.
+ */
+ private boolean clusterMayAutoPause(String accountId, String region, String clusterId) {
+ DbCluster cluster = findClusterForScope(accountId, region, clusterId);
+ if (cluster == null
+ || (cluster.getStatus() != null && cluster.getStatus() != DbInstanceStatus.AVAILABLE)
+ || cluster.getServerlessV2SecondsUntilAutoPause() == null
+ || cluster.getGlobalClusterIdentifier() != null) {
+ return false;
+ }
+ List<String> memberIds = List.copyOf(cluster.getDbClusterMembers());
+ if (memberIds.isEmpty()) {
+ return false;
+ }
+ for (String memberId : memberIds) {
+ DbInstance member = findInstanceForScope(accountId, region, memberId);
+ if (member == null
+ || (member.getStatus() != null && member.getStatus() != DbInstanceStatus.AVAILABLE)
+ || !SERVERLESS_V2_INSTANCE_CLASS.equalsIgnoreCase(member.getDbInstanceClass())) {
+ return false;
+ }
+ }
+ return !isDbProxyTargetOfAccount(accountId, region, clusterId, memberIds);
+ }
+
+ /** {@link #isRegisteredProxyTarget} for an explicit account, since auto-pause runs outside a request. */
+ private boolean isDbProxyTargetOfAccount(String accountId, String region, String clusterId,
+ List<String> memberIds) {
+ List<DbProxyTargetGroup> targetGroups =
+ proxyTargetGroups instanceof AccountAwareStorageBackend<DbProxyTargetGroup> aware
+ ? aware.scanForAccount(accountId, key -> true)
+ : proxyTargetGroups.scan(key -> true);
+ return targetGroups.stream()
+ .filter(targetGroup -> targetGroupBelongsTo(targetGroup, accountId, region))
+ .flatMap(targetGroup -> targetGroup.getTargets().stream())
+ .anyMatch(target -> ("TRACKED_CLUSTER".equals(target.getType())
+ && clusterId.equals(target.getRdsResourceId()))
+ || ("RDS_INSTANCE".equals(target.getType())
+ && memberIds.contains(target.getRdsResourceId())));
+ }
+
+ private List<DbInstance> serverlessMembers(String accountId, String region, String clusterId) {
+ DbCluster cluster = findClusterForScope(accountId, region, clusterId);
+ if (cluster == null) {
+ return List.of();
+ }
+ List<DbInstance> members = new ArrayList<>();
+ for (String memberId : List.copyOf(cluster.getDbClusterMembers())) {
+ DbInstance member = findInstanceForScope(accountId, region, memberId);
+ if (member != null && SERVERLESS_V2_INSTANCE_CLASS.equalsIgnoreCase(member.getDbInstanceClass())) {
+ members.add(member);
+ }
+ }
+ return members;
+ }
+
+ /**
+ * Records an auto-pause step as the RDS-EVENT-0370 to 0374 event of each Serverless v2 instance,
+ * under the cluster's own account, since auto-pause runs outside a request.
+ */
+ private void recordAutoPauseEvents(String accountId, List<DbInstance> members,
+ AutoPauseListener.Event event, Instant at) {
+ String message = switch (event) {
+ case PAUSE_INITIATED -> "Initiated pause for the DB instance.";
+ case PAUSE_CANCELED -> "Pause was canceled for the DB instance.";
+ case PAUSED -> "Successfully paused the DB instance.";
+ case RESUME_INITIATED -> "Initiated resume for the DB instance.";
+ case RESUMED -> "Successfully resumed the DB instance.";
+ // Aurora emits no event while an instance stays paused; that only shows in metrics.
+ case STILL_PAUSED -> null;
+ };
+ if (message == null) {
+ return;
+ }
+ for (DbInstance member : members) {
+ String eventId = "auto-pause:" + member.getDbInstanceArn() + ":" + event + ":" + at;
+ putEventForAccount(accountId, eventId, new RdsEvent(eventId, member.getDbInstanceIdentifier(),
+ "db-instance", message, AUTO_PAUSE_EVENT_CATEGORIES, at, member.getDbInstanceArn()));
+ }
+ }
+
+ private void putEventForAccount(String accountId, String eventId, RdsEvent event) {
+ if (events instanceof AccountAwareStorageBackend<RdsEvent> aware) {
+ aware.putForAccount(accountId, eventId, event);
+ } else {
+ events.put(eventId, event);
+ }
+ }
+
+ /**
+ * Publishes what a paused Aurora Serverless v2 instance still sends CloudWatch: a zero
+ * ServerlessDatabaseCapacity, ACUUtilization and CPUUtilization, for the cluster and for each
+ * instance.
+ */
+ private void publishPausedCapacity(String accountId, String region, String clusterId,
+ List<DbInstance> members, Instant at) {
+ if (metricsService == null) {
+ return;
+ }
+ publishZeroCapacity(accountId, region, new Dimension("DBClusterIdentifier", clusterId), at);
+ for (DbInstance member : members) {
+ publishZeroCapacity(accountId, region,
+ new Dimension("DBInstanceIdentifier", member.getDbInstanceIdentifier()), at);
+ }
+ }
+
+ private void publishZeroCapacity(String accountId, String region, Dimension dimension, Instant at) {
+ for (String metricName : List.of("ServerlessDatabaseCapacity", "ACUUtilization", "CPUUtilization")) {
+ MetricDatum datum = new MetricDatum();
+ datum.setMetricName(metricName);
+ datum.setUnit("ServerlessDatabaseCapacity".equals(metricName) ? "Count" : "Percent");
+ datum.setDimensions(List.of(dimension));
+ datum.setTimestamp(at.getEpochSecond());
+ datum.setValue(0.0);
+ metricsService.publishMetricForAccount(accountId, "AWS/RDS", datum, region,
+ "rds-auto-pause:" + dimension.name() + "=" + dimension.value() + ":" + metricName + ":" + at);
+ }
+ }
+
+ /** The auto-pause of one cluster's container: Aurora's pause rules, its events and its metrics. */
+ private final class ClusterAutoPause implements AutoPauseListener {
+
+ private final String accountId;
+ private final String region;
+ private final String clusterId;
+
+ private ClusterAutoPause(String accountId, String region, String clusterId) {
+ this.accountId = accountId;
+ this.region = region;
+ this.clusterId = clusterId;
+ }
+
+ @Override
+ public boolean mayPause() {
+ return clusterMayAutoPause(accountId, region, clusterId);
+ }
+
+ @Override
+ public void onAutoPause(Event event, Instant at) {
+ List<DbInstance> members = serverlessMembers(accountId, region, clusterId);
+ recordAutoPauseEvents(accountId, members, event, at);
+ if (event == Event.PAUSED || event == Event.STILL_PAUSED) {
+ publishPausedCapacity(accountId, region, clusterId, members, at);
+ }
+ }
+ }
+
/**
* Whether RDS holds a cluster or instance under this identifier.
*
@@ -3700,6 +3884,9 @@ public class RdsService implements Resettable, ResourceProvider {
cluster.setServerlessV2SecondsUntilAutoPause(effectiveAutoPauseSeconds);
}
putClusterForScope(currentAccountId(), effectiveRegion, id, cluster);
+ if (modifiesServerlessV2Scaling) {
+ applyAutoPause(cluster);
+ }
// A cluster rotation applies to every endpoint: the cluster's own proxy and each member
// instance's proxy hold start-time password snapshots, and member endpoints validate
@@ -6838,6 +7025,7 @@ public class RdsService implements Resettable, ResourceProvider {
cluster.setStatus(DbInstanceStatus.AVAILABLE);
putClusterForScope(accountId, clusterRegion,
cluster.getDbClusterIdentifier(), cluster);
+ applyAutoPause(cluster);
} catch (Exception e) {
if (!config.services().rds().mock()) {
try {
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseController.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseController.java
new file mode 100644
index 0000000000..02a5dc1c10
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseController.java
@@ -0,0 +1,493 @@
+package io.github.hectorvent.floci.services.rds.container;
+
+import org.jboss.logging.Logger;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.Executor;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.LongConsumer;
+
+/**
+ * Aurora Serverless v2 auto-pause for one RDS backend container.
+ *
+ * <p>Every client of the container, a proxied connection, a Data API call or a command Floci runs
+ * inside it, holds a {@link RdsBackendGate.Lease} while it uses the container. Once no lease has
+ * been held for {@code SecondsUntilAutoPause}, the container is frozen with {@code docker pause},
+ * which keeps its data and its published port. The next lease request thaws it and waits until
+ * it runs again.
+ *
+ * <p>The state moves from RUNNING to PAUSING, PAUSED and RESUMING, and back to RUNNING. Every
+ * transition happens under {@link #lock}; the Docker calls and the listener run without it, so a
+ * slow daemon or a listener waiting for the RDS service never holds up a thread that needs the
+ * lock. A lease request that finds the container PAUSING cancels the pause, and one that finds it
+ * RESUMING waits for the resume, so no client reaches a frozen container and none is turned away.
+ */
+final class AutoPauseController {
+
+ private static final Logger LOG = Logger.getLogger(AutoPauseController.class);
+
+ /** How often a paused container reports zero capacity, as a paused Aurora instance does. */
+ private static final Duration PAUSED_REPORT_INTERVAL = Duration.ofMinutes(1);
+
+ enum State {
+ RUNNING,
+ PAUSING,
+ PAUSED,
+ RESUMING
+ }
+
+ /** Runs a task after a delay; cancelling the returned future stops a task that has not started. */
+ @FunctionalInterface
+ interface Scheduler {
+ Future<?> schedule(Runnable task, Duration delay);
+ }
+
+ /** The Docker freezer. Both calls succeed when the container is already in the requested state. */
+ interface Freezer {
+ void pause(String containerId);
+
+ void unpause(String containerId);
+ }
+
+ private final String containerId;
+ private final String host;
+ private final int port;
+ private final Freezer freezer;
+ private final Scheduler scheduler;
+ private final Executor notifier;
+ private final Clock clock;
+ private final Duration resumeDelay;
+
+ private final ReentrantLock lock = new ReentrantLock();
+ private final Condition settled = lock.newCondition();
+
+ // Everything below is guarded by lock.
+ private final List<Signal> pendingSignals = new ArrayList<>();
+ private State state = State.RUNNING;
+ private int leases;
+ private Integer secondsUntilAutoPause;
+ private AutoPauseListener listener = AutoPauseListener.NONE;
+ // The one timer of the current state: the idle timer while RUNNING, the zero-capacity report
+ // while PAUSED, the resume delay while RESUMING. A task whose token is not the current one
+ // was cancelled or belongs to an earlier state, and does nothing.
+ private Future<?> timer;
+ private long timerToken;
+ private boolean pauseCanceled;
+ private boolean closed;
+ private Instant lastSignal = Instant.EPOCH;
+
+ AutoPauseController(String containerId, String host, int port, Freezer freezer,
+ Scheduler scheduler, Executor notifier, Clock clock, Duration resumeDelay) {
+ this.containerId = containerId;
+ this.host = host;
+ this.port = port;
+ this.freezer = freezer;
+ this.scheduler = scheduler;
+ this.notifier = notifier;
+ this.clock = clock;
+ this.resumeDelay = resumeDelay;
+ }
+
+ boolean serves(String backendHost, int backendPort) {
+ return port == backendPort && Objects.equals(host, backendHost);
+ }
+
+ boolean runs(String candidateContainerId) {
+ return containerId.equals(candidateContainerId);
+ }
+
+ State state() {
+ lock.lock();
+ try {
+ return state;
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ int leaseCount() {
+ lock.lock();
+ try {
+ return leases;
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Sets the idle interval after which the container pauses; null keeps it running. The idle
+ * clock restarts when the interval changes, and a container paused or pausing under the old
+ * interval is resumed, as Aurora resumes an instance whose scaling range changes.
+ */
+ void configure(Integer seconds, AutoPauseListener newListener) {
+ lock.lock();
+ try {
+ if (closed) {
+ return;
+ }
+ listener = newListener != null ? newListener : AutoPauseListener.NONE;
+ if (Objects.equals(secondsUntilAutoPause, seconds)) {
+ return;
+ }
+ secondsUntilAutoPause = seconds;
+ switch (state) {
+ case RUNNING -> {
+ cancelTimer();
+ armIdleTimer();
+ }
+ case PAUSING -> pauseCanceled = true;
+ case PAUSED -> startTimer(this::resumeInBackground, Duration.ZERO);
+ case RESUMING -> {
+ // Finishing the resume arms the idle timer with the new interval.
+ }
+ }
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ /**
+ * Returns a lease that keeps the container awake, resuming it first when it is paused. Waits
+ * while a pause or a resume is under way; a pause still in progress is cancelled.
+ *
+ * @throws InterruptedException when interrupted while waiting
+ * @throws IllegalStateException when Docker cannot resume the container, which stays paused
+ */
+ RdsBackendGate.Lease acquire() throws InterruptedException {
+ lock.lock();
+ try {
+ while (true) {
+ if (closed && state != State.PAUSING && state != State.RESUMING) {
+ return RdsBackendGate.Lease.NONE;
+ }
+ if (state == State.RUNNING) {
+ leases++;
+ cancelTimer();
+ return new ControllerLease();
+ }
+ if (state == State.PAUSED) {
+ resume();
+ } else {
+ if (state == State.PAUSING) {
+ pauseCanceled = true;
+ }
+ settled.await();
+ }
+ }
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ /**
+ * Stops auto-pausing for good, thawing the container when it is paused. A pause or a resume
+ * in progress is not waited for: the thread running it thaws the container once it sees the
+ * controller closed.
+ */
+ void close() {
+ lock.lock();
+ try {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ // A resume whose timer is set has already thawed the container and only waits out
+ // the resume delay; cancelling that timer must not leave its waiters behind.
+ boolean resumeDelayPending = state == State.RESUMING && timer != null;
+ cancelTimer();
+ if (resumeDelayPending) {
+ finishResume();
+ } else if (state == State.PAUSED) {
+ state = State.RESUMING;
+ RuntimeException failure = withoutLock(() -> freezer.unpause(containerId));
+ if (failure != null) {
+ LOG.warnv(failure, "Could not resume auto-paused RDS container {0} before releasing it",
+ containerId);
+ state = State.PAUSED;
+ } else {
+ state = State.RUNNING;
+ }
+ }
+ settled.signalAll();
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ /**
+ * Runs the idle check now rather than when the idle timer fires, for a caller that cannot wait
+ * SecondsUntilAutoPause. Returns whether the container paused.
+ */
+ boolean pauseIfIdle() {
+ long token;
+ lock.lock();
+ try {
+ if (!idleTimerArmed(timerToken)) {
+ return false;
+ }
+ token = timerToken;
+ } finally {
+ unlockAndDispatch();
+ }
+ idleTimerFired(token);
+ return state() == State.PAUSED;
+ }
+
+ private void release() {
+ lock.lock();
+ try {
+ if (leases > 0) {
+ leases--;
+ }
+ armIdleTimer();
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ private void idleTimerFired(long token) {
+ AutoPauseListener current;
+ lock.lock();
+ try {
+ if (!idleTimerArmed(token)) {
+ return;
+ }
+ current = listener;
+ } finally {
+ unlockAndDispatch();
+ }
+ // Asked without the lock: the answer comes from the RDS service, which may be busy.
+ boolean allowed = mayPause(current);
+ lock.lock();
+ try {
+ if (!idleTimerArmed(token)) {
+ return;
+ }
+ if (!allowed) {
+ armIdleTimer();
+ return;
+ }
+ cancelTimer();
+ state = State.PAUSING;
+ pauseCanceled = false;
+ signal(AutoPauseListener.Event.PAUSE_INITIATED);
+ pause();
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ private boolean idleTimerArmed(long token) {
+ return !closed && state == State.RUNNING && leases == 0 && secondsUntilAutoPause != null
+ && timer != null && token == timerToken;
+ }
+
+ /** Freezes the container. Entered and left holding the lock, in PAUSING. */
+ private void pause() {
+ RuntimeException pauseFailure = withoutLock(() -> freezer.pause(containerId));
+ if (pauseFailure == null && !pauseCanceled && !closed) {
+ state = State.PAUSED;
+ signal(AutoPauseListener.Event.PAUSED);
+ armPausedReport();
+ settled.signalAll();
+ LOG.infov("Auto-paused idle RDS container {0} (SecondsUntilAutoPause {1})",
+ containerId, String.valueOf(secondsUntilAutoPause));
+ return;
+ }
+ if (pauseFailure != null) {
+ LOG.warnv(pauseFailure, "Could not auto-pause RDS container {0}; it keeps running", containerId);
+ } else {
+ // A client arrived, the interval changed or the container is being released while it
+ // was freezing: thaw it again rather than let anyone reach it frozen.
+ RuntimeException thawFailure = withoutLock(() -> freezer.unpause(containerId));
+ if (thawFailure != null) {
+ LOG.warnv(thawFailure, "Could not undo the auto-pause of RDS container {0}", containerId);
+ state = State.PAUSED;
+ armPausedReport();
+ settled.signalAll();
+ return;
+ }
+ }
+ state = State.RUNNING;
+ signal(AutoPauseListener.Event.PAUSE_CANCELED);
+ settled.signalAll();
+ armIdleTimer();
+ }
+
+ /**
+ * Thaws the container. Entered and left holding the lock, in PAUSED. Waiters see RESUMING
+ * until the resume delay has passed.
+ */
+ private void resume() {
+ cancelTimer();
+ state = State.RESUMING;
+ signal(AutoPauseListener.Event.RESUME_INITIATED);
+ RuntimeException failure = withoutLock(() -> freezer.unpause(containerId));
+ if (failure != null) {
+ state = State.PAUSED;
+ armPausedReport();
+ settled.signalAll();
+ throw new IllegalStateException(
+ "Could not resume auto-paused RDS container " + containerId, failure);
+ }
+ if (closed || resumeDelay.isZero()) {
+ finishResume();
+ } else {
+ startTimer(this::resumeDelayElapsed, resumeDelay);
+ }
+ }
+
+ private void resumeDelayElapsed(long token) {
+ lock.lock();
+ try {
+ if (state == State.RESUMING && token == timerToken) {
+ finishResume();
+ }
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ private void finishResume() {
+ cancelTimer();
+ state = State.RUNNING;
+ signal(AutoPauseListener.Event.RESUMED);
+ settled.signalAll();
+ armIdleTimer();
+ LOG.infov("Resumed auto-paused RDS container {0}", containerId);
+ }
+
+ private void resumeInBackground(long token) {
+ lock.lock();
+ try {
+ if (!closed && state == State.PAUSED && token == timerToken) {
+ resume();
+ }
+ } catch (IllegalStateException e) {
+ LOG.warnv(e, "Could not resume auto-paused RDS container {0}", containerId);
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ private void armIdleTimer() {
+ if (closed || state != State.RUNNING || leases > 0 || secondsUntilAutoPause == null) {
+ return;
+ }
+ startTimer(this::idleTimerFired, Duration.ofSeconds(secondsUntilAutoPause));
+ }
+
+ private void armPausedReport() {
+ if (!closed) {
+ startTimer(this::pausedReportDue, PAUSED_REPORT_INTERVAL);
+ }
+ }
+
+ private void pausedReportDue(long token) {
+ lock.lock();
+ try {
+ if (!closed && state == State.PAUSED && token == timerToken) {
+ signal(AutoPauseListener.Event.STILL_PAUSED);
+ armPausedReport();
+ }
+ } finally {
+ unlockAndDispatch();
+ }
+ }
+
+ private void startTimer(LongConsumer task, Duration delay) {
+ cancelTimer();
+ long token = timerToken;
+ timer = scheduler.schedule(() -> task.accept(token), delay);
+ }
+
+ private void cancelTimer() {
+ if (timer != null) {
+ timer.cancel(false);
+ timer = null;
+ }
+ timerToken++;
+ }
+
+ /** Runs a Docker call with the lock released, and returns its failure instead of throwing it. */
+ private RuntimeException withoutLock(Runnable dockerCall) {
+ if (lock.getHoldCount() != 1) {
+ throw new IllegalStateException("Auto-pause lock must be held exactly once around a Docker call");
+ }
+ unlockAndDispatch();
+ try {
+ dockerCall.run();
+ return null;
+ } catch (RuntimeException e) {
+ return e;
+ } finally {
+ lock.lock();
+ }
+ }
+
+ private boolean mayPause(AutoPauseListener current) {
+ try {
+ return current.mayPause();
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not tell whether RDS container {0} may auto-pause; it keeps running",
+ containerId);
+ return false;
+ }
+ }
+
+ /** Queues an event for the listener, stamped later than the previous one so they keep their order. */
+ private void signal(AutoPauseListener.Event event) {
+ Instant now = clock.instant();
+ Instant at = now.isAfter(lastSignal) ? now : lastSignal.plusMillis(1);
+ lastSignal = at;
+ pendingSignals.add(new Signal(listener, event, at));
+ }
+
+ /** Releases the lock, then hands the events queued under it to the notifier, in order. */
+ private void unlockAndDispatch() {
+ List<Signal> ready = pendingSignals.isEmpty() ? List.of() : List.copyOf(pendingSignals);
+ pendingSignals.clear();
+ lock.unlock();
+ for (Signal signal : ready) {
+ try {
+ notifier.execute(() -> deliver(signal));
+ } catch (RejectedExecutionException e) {
+ LOG.debugv("Dropped RDS auto-pause event {0} for container {1}: {2}",
+ signal.event(), containerId, e.getMessage());
+ }
+ }
+ }
+
+ private void deliver(Signal signal) {
+ try {
+ signal.listener().onAutoPause(signal.event(), signal.at());
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "Could not report RDS auto-pause event {0} for container {1}",
+ signal.event(), containerId);
+ }
+ }
+
+ private record Signal(AutoPauseListener listener, AutoPauseListener.Event event, Instant at) {
+ }
+
+ private final class ControllerLease implements RdsBackendGate.Lease {
+
+ private final AtomicBoolean released = new AtomicBoolean();
+
+ @Override
+ public void close() {
+ if (released.compareAndSet(false, true)) {
+ release();
+ }
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseListener.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseListener.java
new file mode 100644
index 0000000000..e6eb17e2ca
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseListener.java
@@ -0,0 +1,40 @@
+package io.github.hectorvent.floci.services.rds.container;
+
+import java.time.Instant;
+
+/**
+ * What the auto-pause of an RDS container asks of the cluster it belongs to, and what it reports
+ * back. Both calls run on a background thread, never while the auto-pause holds its lock, so an
+ * implementation may take the RDS service's lock.
+ */
+public interface AutoPauseListener {
+
+ AutoPauseListener NONE = new AutoPauseListener() {
+ @Override
+ public boolean mayPause() {
+ return false;
+ }
+
+ @Override
+ public void onAutoPause(Event event, Instant at) {
+ }
+ };
+
+ /**
+ * The auto-pause steps Aurora reports as RDS-EVENT-0370 to 0374, plus {@code STILL_PAUSED},
+ * repeated once a minute while the container stays paused.
+ */
+ enum Event {
+ PAUSE_INITIATED,
+ PAUSE_CANCELED,
+ PAUSED,
+ STILL_PAUSED,
+ RESUME_INITIATED,
+ RESUMED
+ }
+
+ /** Whether the cluster may pause now that it has been idle for SecondsUntilAutoPause. */
+ boolean mayPause();
+
+ void onAutoPause(Event event, Instant at);
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsBackendGate.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsBackendGate.java
new file mode 100644
index 0000000000..2104e52191
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsBackendGate.java
@@ -0,0 +1,31 @@
+package io.github.hectorvent.floci.services.rds.container;
+
+/**
+ * Lets a client into an RDS backend container. An Aurora Serverless v2 cluster that auto-paused is
+ * resumed first, and it does not pause again while the client holds the returned lease.
+ */
+@FunctionalInterface
+public interface RdsBackendGate {
+
+ /** A gate for backends that never pause. */
+ RdsBackendGate OPEN = (host, port) -> Lease.NONE;
+
+ /**
+ * Waits until the container serving {@code host:port} can take a client, resuming it when it
+ * is auto-paused, and keeps it awake until the lease is closed. A backend with no auto-pause
+ * gets {@link Lease#NONE}.
+ *
+ * @throws InterruptedException when interrupted while a pause or resume is under way
+ */
+ Lease enter(String host, int port) throws InterruptedException;
+
+ /** Held for as long as a client uses the backend; closing it more than once is harmless. */
+ interface Lease extends AutoCloseable {
+
+ Lease NONE = () -> {
+ };
+
+ @Override
+ void close();
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java
index 967fd80fc0..99518b223c 100644
--- a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java
@@ -1,10 +1,12 @@
package io.github.hectorvent.floci.services.rds.container;
import com.github.dockerjava.api.async.ResultCallback;
+import com.github.dockerjava.api.command.InspectContainerResponse;
import com.github.dockerjava.api.model.Frame;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.Resettable;
import io.github.hectorvent.floci.core.common.ServiceConfigAccess;
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
import io.github.hectorvent.floci.core.common.docker.ContainerDetector;
@@ -15,6 +17,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.E
import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
+import jakarta.annotation.PreDestroy;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
@@ -27,22 +30,32 @@ import java.io.Closeable;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.charset.StandardCharsets;
+import java.time.Clock;
+import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.regex.Pattern;
/**
* Manages backend Docker container lifecycle for RDS DB instances and clusters.
* Starts postgres/mysql/mariadb containers and resolves the backend host:port for the auth proxy.
+ * Every client of a container enters it through {@link #enter}, which is how an Aurora Serverless
+ * v2 cluster with MinCapacity 0 knows when it is idle enough to pause and when to resume.
*/
@ApplicationScoped
-public class RdsContainerManager {
+public class RdsContainerManager implements RdsBackendGate, Resettable {
private static final Logger LOG = Logger.getLogger(RdsContainerManager.class);
private static final Pattern SAFE_STORAGE_COMPONENT = Pattern.compile("[A-Za-z0-9._-]+");
@@ -59,6 +72,29 @@ public class RdsContainerManager {
private final Set<String> claimedRuntimes = ConcurrentHashMap.newKeySet();
private final Set<String> cleanupPending = ConcurrentHashMap.newKeySet();
private volatile boolean dockerUnavailableLogged;
+ // Auto-pause state of every running container, by runtime id. The timer thread only keeps
+ // time: each task it fires runs on a virtual thread of its own, since a pause first asks the
+ // RDS service whether the cluster may pause. Events reach the listener one at a time, in order.
+ private final Map<String, AutoPauseController> autoPauses = new ConcurrentHashMap<>();
+ private final ScheduledExecutorService autoPauseTimer = Executors.newSingleThreadScheduledExecutor(
+ runnable -> {
+ Thread thread = new Thread(runnable, "rds-auto-pause-timer");
+ thread.setDaemon(true);
+ return thread;
+ });
+ private final ExecutorService autoPauseEvents = Executors.newSingleThreadExecutor(
+ Thread.ofVirtual().name("rds-auto-pause-events").factory());
+ private final AutoPauseController.Freezer dockerFreezer = new AutoPauseController.Freezer() {
+ @Override
+ public void pause(String containerId) {
+ pauseContainer(containerId);
+ }
+
+ @Override
+ public void unpause(String containerId) {
+ unpauseContainer(containerId);
+ }
+ };
@Inject
public RdsContainerManager(ContainerBuilder containerBuilder,
@@ -281,6 +317,9 @@ public class RdsContainerManager {
info.containerId(), logGroup, logStream, runtimeIdentity.region(),
"rds:" + effectiveRuntimeId);
handle.setLogStream(logHandle);
+ // Before the handle is returned, so that the connections of every proxy started on it
+ // are counted from the first one.
+ registerAutoPause(effectiveRuntimeId, handle);
activeContainers.put(effectiveRuntimeId, handle);
return handle;
@@ -349,6 +388,7 @@ public class RdsContainerManager {
}
RdsContainerHandle active = activeContainers.get(handle.getRuntimeId());
RdsContainerHandle effectiveHandle = active != null ? active : handle;
+ releaseAutoPause(effectiveHandle.getRuntimeId());
try {
lifecycleManager.stopAndRemoveStrict(
effectiveHandle.getContainerId(), effectiveHandle.getLogStream());
@@ -411,6 +451,156 @@ public class RdsContainerManager {
}
}
+ /**
+ * Applies an Aurora Serverless v2 auto-pause interval to a running container: it pauses once
+ * no client has used it for that many seconds, and a null interval, for MinCapacity above 0
+ * or with auto-pause turned off, keeps it running. The listener decides whether the cluster
+ * may pause at that moment and is told each step.
+ */
+ public void configureAutoPause(String runtimeId, Integer secondsUntilAutoPause,
+ AutoPauseListener listener) {
+ AutoPauseController controller = runtimeId != null ? autoPauses.get(runtimeId) : null;
+ if (controller == null) {
+ return;
+ }
+ boolean enabled = config.services().rds().auroraAutoPauseEnabled();
+ controller.configure(enabled ? secondsUntilAutoPause : null, listener);
+ }
+
+ @Override
+ public Lease enter(String host, int port) throws InterruptedException {
+ if (host == null) {
+ return Lease.NONE;
+ }
+ for (AutoPauseController controller : autoPauses.values()) {
+ if (controller.serves(host, port)) {
+ return controller.acquire();
+ }
+ }
+ return Lease.NONE;
+ }
+
+ /**
+ * Runs a runtime's auto-pause idle check now instead of after SecondsUntilAutoPause, which
+ * the API does not let drop below 300 seconds. Returns whether the container paused.
+ */
+ boolean pauseIfIdle(String runtimeId) {
+ AutoPauseController controller = autoPauses.get(runtimeId);
+ return controller != null && controller.pauseIfIdle();
+ }
+
+ /**
+ * A reset wipes the RDS records but leaves their containers running, as it always has. It
+ * releases their auto-pause, which thaws a paused container, so none stays frozen with no
+ * cluster left to resume it.
+ */
+ @Override
+ public void clear() {
+ for (String runtimeId : List.copyOf(autoPauses.keySet())) {
+ releaseAutoPause(runtimeId);
+ }
+ }
+
+ @PreDestroy
+ void shutdownAutoPause() {
+ autoPauseTimer.shutdownNow();
+ autoPauseEvents.shutdownNow();
+ }
+
+ private void registerAutoPause(String runtimeId, RdsContainerHandle handle) {
+ Duration resumeDelay = Duration.ofMillis(
+ Math.max(0, config.services().rds().auroraResumeDelayMillis()));
+ AutoPauseController controller = new AutoPauseController(
+ handle.getContainerId(), handle.getHost(), handle.getPort(), dockerFreezer,
+ this::scheduleAutoPauseTask, autoPauseEvents, Clock.systemUTC(), resumeDelay);
+ // Clients find a container by its address, so the new container owns it from now on,
+ // even if Docker removed the previous owner without Floci stopping it.
+ for (Map.Entry<String, AutoPauseController> entry : List.copyOf(autoPauses.entrySet())) {
+ if (entry.getValue().serves(handle.getHost(), handle.getPort())
+ && autoPauses.remove(entry.getKey(), entry.getValue())) {
+ entry.getValue().close();
+ }
+ }
+ AutoPauseController previous = autoPauses.put(runtimeId, controller);
+ if (previous != null) {
+ previous.close();
+ }
+ }
+
+ /** Stops a runtime's auto-pause before its container goes away, thawing it when paused. */
+ private void releaseAutoPause(String runtimeId) {
+ AutoPauseController controller = autoPauses.remove(runtimeId);
+ if (controller != null) {
+ controller.close();
+ }
+ }
+
+ /** Keeps a container awake, resuming it first, while Floci runs a command inside it. */
+ private Lease holdContainer(String containerId) {
+ if (containerId == null) {
+ return Lease.NONE;
+ }
+ for (AutoPauseController controller : autoPauses.values()) {
+ if (controller.runs(containerId)) {
+ try {
+ return controller.acquire();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException(
+ "Interrupted while resuming RDS container " + containerId, e);
+ }
+ }
+ }
+ return Lease.NONE;
+ }
+
+ private Future<?> scheduleAutoPauseTask(Runnable task, Duration delay) {
+ try {
+ return autoPauseTimer.schedule(
+ () -> Thread.ofVirtual().name("rds-auto-pause").start(task),
+ delay.toMillis(), TimeUnit.MILLISECONDS);
+ } catch (RejectedExecutionException e) {
+ LOG.debugv("RDS auto-pause timer is shut down; not scheduling: {0}", e.getMessage());
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+
+ private void pauseContainer(String containerId) {
+ try {
+ lifecycleManager.getDockerClient().pauseContainerCmd(containerId).exec();
+ } catch (RuntimeException e) {
+ // Docker refuses to pause a paused container; that one is where it should be.
+ if (!isRunning(containerId, true)) {
+ throw e;
+ }
+ }
+ }
+
+ private void unpauseContainer(String containerId) {
+ try {
+ lifecycleManager.getDockerClient().unpauseContainerCmd(containerId).exec();
+ } catch (RuntimeException e) {
+ // Docker refuses to unpause a container that is not paused, as after a manual
+ // `docker unpause`; one still running is where it should be.
+ if (!isRunning(containerId, false)) {
+ throw e;
+ }
+ }
+ }
+
+ /** Whether the container runs, paused or not as asked; false when Docker cannot say. */
+ private boolean isRunning(String containerId, boolean paused) {
+ try {
+ InspectContainerResponse.ContainerState state =
+ lifecycleManager.getDockerClient().inspectContainerCmd(containerId).exec().getState();
+ return state != null && Boolean.TRUE.equals(state.getRunning())
+ && paused == Boolean.TRUE.equals(state.getPaused());
+ } catch (RuntimeException e) {
+ LOG.debugv("Could not inspect RDS container {0}: {1}", containerId, e.getMessage());
+ return false;
+ }
+ }
+
private void addPersistenceMounts(
ContainerBuilder.Builder specBuilder, String storageResourceId,
String exactVolumeName, DatabaseEngine engine, String image) {
@@ -555,9 +745,11 @@ public class RdsContainerManager {
*/
public void rotateMasterPassword(String containerName, String containerId, DatabaseEngine engine,
String masterUsername, String oldPassword, String newPassword) {
- execUntilSuccess(containerName, containerId,
- passwordRotationCommand(engine, masterUsername, oldPassword, newPassword),
- "master-password rotation");
+ try (Lease _ = holdContainer(containerId)) {
+ execUntilSuccess(containerName, containerId,
+ passwordRotationCommand(engine, masterUsername, oldPassword, newPassword),
+ "master-password rotation");
+ }
}
static String[] passwordRotationCommand(DatabaseEngine engine, String masterUsername,
@@ -679,7 +871,7 @@ public class RdsContainerManager {
"pg_dumpall",
"-U", effectiveUser
};
- try {
+ try (Lease _ = holdContainer(containerId)) {
ContainerExecResult result = execInContainer(containerId, cmd, 120);
if (result.exitCode() != 0) {
throw new RuntimeException("pg_dumpall failed with exit code " + result.exitCode() + ": " + result.stderr());
@@ -693,7 +885,7 @@ public class RdsContainerManager {
public void restorePostgresSnapshot(String containerId, String masterUsername, String sqlDump) {
String effectiveUser = (masterUsername != null && !masterUsername.isBlank()) ? masterUsername : "postgres";
- try {
+ try (Lease _ = holdContainer(containerId)) {
String restoreScript = postgresRestoreScript();
java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java
index c50cd9b866..dbe302b37e 100644
--- a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.rds.proxy;
+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
import org.jboss.logging.Logger;
@@ -34,6 +35,7 @@ public class RdsAuthProxy {
private final int handshakeTimeoutMillis;
private final int backendConnectTimeoutMillis;
private final Semaphore connectionPermits;
+ private final RdsBackendGate backendGate;
private volatile boolean running;
private ServerSocket serverSocket;
@@ -47,7 +49,8 @@ public class RdsAuthProxy {
int maxConnections) {
this(instanceId, backendHost, backendPort, engine, iamEnabled, masterUsername, masterPassword,
dbName, sigV4, tlsCertificates, passwordValidator, handshakeTimeoutMillis,
- backendConnectTimeoutMillis, maxConnections, null, username -> false);
+ backendConnectTimeoutMillis, maxConnections, null, username -> false,
+ RdsBackendGate.OPEN);
}
public RdsAuthProxy(String instanceId, String backendHost, int backendPort,
@@ -56,11 +59,11 @@ public class RdsAuthProxy {
RdsSigV4Validator sigV4, RdsProxyTlsCertificates tlsCertificates,
MasterPasswordCheck passwordValidator,
int handshakeTimeoutMillis, int backendConnectTimeoutMillis,
- int maxConnections, RdsProxyBinding binding) {
+ int maxConnections, RdsProxyBinding binding, RdsBackendGate backendGate) {
this(instanceId, backendHost, backendPort, engine, iamEnabled, masterUsername, masterPassword,
dbName, sigV4, tlsCertificates, passwordValidator, handshakeTimeoutMillis,
backendConnectTimeoutMillis, maxConnections, binding,
- username -> binding != null);
+ username -> binding != null, backendGate);
}
public RdsAuthProxy(String instanceId, String backendHost, int backendPort,
@@ -70,7 +73,8 @@ public class RdsAuthProxy {
MasterPasswordCheck passwordValidator,
int handshakeTimeoutMillis, int backendConnectTimeoutMillis,
int maxConnections, RdsProxyBinding binding,
- MySqlProtocolHandler.IamUserChecker iamUserChecker) {
+ MySqlProtocolHandler.IamUserChecker iamUserChecker,
+ RdsBackendGate backendGate) {
this.instanceId = instanceId;
this.backendHost = backendHost;
this.backendPort = backendPort;
@@ -87,6 +91,7 @@ public class RdsAuthProxy {
this.handshakeTimeoutMillis = handshakeTimeoutMillis;
this.backendConnectTimeoutMillis = backendConnectTimeoutMillis;
this.connectionPermits = new Semaphore(Math.max(1, maxConnections));
+ this.backendGate = backendGate;
}
public void start(int proxyPort) throws IOException {
@@ -151,7 +156,11 @@ public class RdsAuthProxy {
private void handleConnection(Socket client) {
Socket backend = null;
PostgresProtocolHandler.AuthenticatedSession session = null;
+ RdsBackendGate.Lease lease = null;
try {
+ // An auto-paused Aurora backend resumes before the client is served: the client is
+ // held, not refused, and the backend stays awake for as long as the connection lasts.
+ lease = backendGate.enter(backendHost, backendPort);
client.setTcpNoDelay(true);
// RDS only proxy-validates the master user; a non-master user passes through so the
@@ -196,6 +205,9 @@ public class RdsAuthProxy {
TcpStreamBridge.relay(client, backend);
}
}
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.debugv("RDS connection for instance {0} interrupted while its backend resumed", instanceId);
} catch (Exception e) {
LOG.debugv("RDS connection error for instance {0}: {1}", instanceId, e.getMessage());
} finally {
@@ -209,6 +221,9 @@ public class RdsAuthProxy {
if (session != null) {
closeQuietly(session.backend());
}
+ if (lease != null) {
+ lease.close();
+ }
}
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java
index 0695c0ce3f..ce65954e1d 100644
--- a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java
@@ -1,6 +1,7 @@
package io.github.hectorvent.floci.services.rds.proxy;
import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
@@ -20,14 +21,21 @@ public class RdsProxyManager {
private final RdsSigV4Validator sigV4Validator;
private final RdsProxyTlsCertificates tlsCertificates;
private final EmulatorConfig config;
+ private final RdsBackendGate backendGate;
private final ConcurrentHashMap<String, RdsAuthProxy> proxies = new ConcurrentHashMap<>();
@Inject
public RdsProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates,
- EmulatorConfig config) {
+ EmulatorConfig config, RdsBackendGate backendGate) {
this.sigV4Validator = sigV4Validator;
this.tlsCertificates = tlsCertificates;
this.config = config;
+ this.backendGate = backendGate;
+ }
+
+ RdsProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates,
+ EmulatorConfig config) {
+ this(sigV4Validator, tlsCertificates, config, RdsBackendGate.OPEN);
}
public synchronized void startProxy(String instanceId, DatabaseEngine engine, boolean iamEnabled,
@@ -42,7 +50,7 @@ public class RdsProxyManager {
instanceId, backendHost, backendPort, engine, iamEnabled,
masterUsername, masterPassword, dbName, sigV4Validator, tlsCertificates, passwordValidator,
rdsConfig.proxyHandshakeTimeoutMillis(), rdsConfig.proxyBackendConnectTimeoutMillis(),
- rdsConfig.proxyMaxConnections(), binding);
+ rdsConfig.proxyMaxConnections(), binding, backendGate);
try {
proxy.start(proxyPort);
} catch (IOException e) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java b/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java
index 2f1e09a7b1..5f913afe31 100644
--- a/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java
@@ -8,6 +8,7 @@ import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.Resettable;
import io.github.hectorvent.floci.core.common.SqlParameterParser.ParsedSql;
+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;
import io.github.hectorvent.floci.services.secretsmanager.model.SecretVersion;
@@ -53,15 +54,17 @@ public class RdsDataService implements Resettable {
private final Duration transactionTtl;
private final ConcurrentMap<String, TransactionContext> transactions = new ConcurrentHashMap<>();
private final ScheduledExecutorService transactionCleanupExecutor;
+ private final RdsBackendGate backendGate;
@Inject
public RdsDataService(RdsDataResourceResolver resourceResolver,
SecretsManagerService secretsManagerService,
ObjectMapper objectMapper,
RdsDataConnectionFactory connectionFactory,
- EmulatorConfig config) {
+ EmulatorConfig config,
+ RdsBackendGate backendGate) {
this(resourceResolver, secretsManagerService, objectMapper, connectionFactory,
- Duration.ofSeconds(config.services().rdsData().transactionTtlSeconds()));
+ Duration.ofSeconds(config.services().rdsData().transactionTtlSeconds()), backendGate);
}
RdsDataService(RdsDataResourceResolver resourceResolver,
@@ -69,11 +72,22 @@ public class RdsDataService implements Resettable {
ObjectMapper objectMapper,
RdsDataConnectionFactory connectionFactory,
Duration transactionTtl) {
+ this(resourceResolver, secretsManagerService, objectMapper, connectionFactory, transactionTtl,
+ RdsBackendGate.OPEN);
+ }
+
+ RdsDataService(RdsDataResourceResolver resourceResolver,
+ SecretsManagerService secretsManagerService,
+ ObjectMapper objectMapper,
+ RdsDataConnectionFactory connectionFactory,
+ Duration transactionTtl,
+ RdsBackendGate backendGate) {
this.resourceResolver = resourceResolver;
this.secretsManagerService = secretsManagerService;
this.objectMapper = objectMapper;
this.connectionFactory = connectionFactory;
this.transactionTtl = transactionTtl;
+ this.backendGate = backendGate;
this.transactionCleanupExecutor = Executors.newSingleThreadScheduledExecutor(r -> {
Thread thread = new Thread(r, "rds-data-transaction-cleanup");
thread.setDaemon(true);
@@ -95,7 +109,7 @@ public class RdsDataService implements Resettable {
synchronized (tx) {
if (transactions.remove(id, tx)) {
rollbackQuietly(tx.connection);
- closeQuietly(tx.connection);
+ tx.close();
}
}
});
@@ -106,7 +120,7 @@ public class RdsDataService implements Resettable {
synchronized (tx) {
if (transactions.remove(id, tx)) {
rollbackQuietly(tx.connection);
- closeQuietly(tx.connection);
+ tx.close();
}
}
});
@@ -149,13 +163,18 @@ public class RdsDataService implements Resettable {
Credentials credentials = credentials(request, target, region);
String database = databaseName(request, target);
+ // The transaction's connection keeps the cluster awake until it commits, rolls back or expires.
+ RdsBackendGate.Lease lease = enterBackend(target);
+ boolean leaseHandedOver = false;
Connection connection = null;
try {
connection = connectionFactory.open(target, credentials.username(), credentials.password(), database);
connection.setAutoCommit(false);
String transactionId = UUID.randomUUID().toString();
transactions.put(transactionId, new TransactionContext(
- transactionId, connection, target.engine(), target.arn(), database, region, transactionTtl));
+ transactionId, connection, target.engine(), target.arn(), database, region, transactionTtl,
+ lease));
+ leaseHandedOver = true;
ObjectNode response = objectMapper.createObjectNode();
response.put("transactionId", transactionId);
@@ -165,6 +184,10 @@ public class RdsDataService implements Resettable {
closeQuietly(connection);
}
throw databaseError(e);
+ } finally {
+ if (!leaseHandedOver) {
+ lease.close();
+ }
}
}
@@ -184,7 +207,7 @@ public class RdsDataService implements Resettable {
} catch (SQLException e) {
throw databaseError(e);
} finally {
- closeQuietly(tx.connection);
+ tx.close();
}
}
ObjectNode response = objectMapper.createObjectNode();
@@ -208,7 +231,7 @@ public class RdsDataService implements Resettable {
} catch (SQLException e) {
throw databaseError(e);
} finally {
- closeQuietly(tx.connection);
+ tx.close();
}
}
ObjectNode response = objectMapper.createObjectNode();
@@ -236,11 +259,28 @@ public class RdsDataService implements Resettable {
resourceResolver.resolve(resourceArn, region);
Credentials credentials = credentials(request, target, region);
String database = databaseName(request, target);
- try (Connection connection = connectionFactory.open(target, credentials.username(), credentials.password(), database)) {
+ try (RdsBackendGate.Lease _ = enterBackend(target);
+ Connection connection = connectionFactory.open(target, credentials.username(), credentials.password(), database)) {
return work.execute(connection, target.engine());
}
}
+ /**
+ * Resumes an auto-paused Aurora cluster before the Data API uses its database, as Aurora
+ * does, and keeps it from pausing again until the returned lease is closed.
+ */
+ private RdsBackendGate.Lease enterBackend(RdsDataResourceResolver.DatabaseTarget target) {
+ try {
+ return backendGate.enter(target.host(), target.port());
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new AwsException("InternalServerErrorException",
+ "Interrupted while the DB cluster was resuming.", 500);
+ } catch (IllegalStateException e) {
+ throw new AwsException("InternalServerErrorException", e.getMessage(), 500);
+ }
+ }
+
private ObjectNode executeOnConnection(Connection connection, DatabaseEngine engine, String sql,
Map<String, JsonNode> parameters, boolean includeMetadata)
throws SQLException {
@@ -583,7 +623,7 @@ public class RdsDataService implements Resettable {
synchronized (tx) {
if (tx.expiresAt.isBefore(now) && transactions.remove(id, tx)) {
rollbackQuietly(tx.connection);
- closeQuietly(tx.connection);
+ tx.close();
}
}
}
@@ -737,23 +777,33 @@ public class RdsDataService implements Resettable {
private final String resourceArn;
private final String database;
private final String region;
+ private final RdsBackendGate.Lease lease;
private volatile Instant expiresAt;
private TransactionContext(
String id, Connection connection, DatabaseEngine engine, String resourceArn,
- String database, String region, Duration ttl) {
+ String database, String region, Duration ttl, RdsBackendGate.Lease lease) {
this.id = id;
this.connection = connection;
this.engine = engine;
this.resourceArn = resourceArn;
this.database = database;
this.region = region;
+ this.lease = lease;
refresh(ttl);
}
private void refresh(Duration ttl) {
this.expiresAt = Instant.now().plus(ttl);
}
+
+ private void close() {
+ try {
+ closeQuietly(connection);
+ } finally {
+ lease.close();
+ }
+ }
}
private record Credentials(String username, String password) {
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 5ce38f14a1..327725e64f 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -418,6 +418,8 @@ floci:
proxy-handshake-timeout-millis: 10000 # Max time a client has to complete the startup/auth handshake
proxy-backend-connect-timeout-millis: 5000 # Max time to wait for the backend TCP connect
proxy-max-connections: 100 # Max concurrent proxy connections before new ones are refused
+ aurora-auto-pause-enabled: true # FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED; pause a MinCapacity=0 Aurora cluster after SecondsUntilAutoPause without connections
+ aurora-resume-delay-millis: 0 # FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS; hold the first connection this long while a paused cluster resumes (Aurora takes about 15000)
rds-data:
enabled: true
transaction-ttl-seconds: 180
diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java
index 769465e5bd..6af033a7ab 100644
--- a/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java
@@ -16,6 +16,7 @@ import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
import io.github.hectorvent.floci.services.rds.model.DbCluster;
import io.github.hectorvent.floci.services.rds.model.DbClusterSnapshot;
import io.github.hectorvent.floci.services.rds.model.DbClusterParameterGroup;
+import io.github.hectorvent.floci.services.rds.container.AutoPauseListener;
import io.github.hectorvent.floci.services.rds.container.RdsContainerHandle;
import io.github.hectorvent.floci.services.rds.container.RdsContainerManager;
import io.github.hectorvent.floci.services.rds.model.DbEndpoint;
@@ -73,6 +74,7 @@ import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.ArgumentMatchers.isNull;
+import static org.mockito.Mockito.atLeastOnce;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
@@ -1758,6 +1760,102 @@ class RdsServiceTest {
assertEquals(300, passwordOnlyChange.getServerlessV2SecondsUntilAutoPause());
}
+ @Test
+ void auroraClusterHandsItsAutoPauseIntervalToItsContainer() {
+ DbCluster cluster = createAutoPauseCluster(rdsService, "paused", 600);
+
+ verify(containerManager).configureAutoPause(eq(cluster.getDbClusterArn()), eq(600), any());
+
+ rdsService.modifyDbCluster("paused", null, null, 0.5, null, null);
+ verify(containerManager).configureAutoPause(eq(cluster.getDbClusterArn()), isNull(), any());
+ }
+
+ @Test
+ void auroraClusterMayAutoPauseOnlyWithServerlessInstancesAndNothingHoldingItAwake() {
+ DbCluster cluster = createAutoPauseCluster(rdsService, "paused", 300);
+ AutoPauseListener listener = capturedAutoPauseListener(cluster);
+ assertFalse(listener.mayPause(), "a cluster without instances has nothing to pause");
+
+ createMember(rdsService, "paused-1", "db.serverless", "paused");
+ assertTrue(listener.mayPause());
+
+ createMember(rdsService, "paused-2", "db.r6g.large", "paused");
+ assertFalse(listener.mayPause(), "a provisioned instance keeps the cluster awake");
+ rdsService.deleteDbInstance("paused-2");
+ assertTrue(listener.mayPause());
+
+ rdsService.modifyDbCluster("paused", null, null, 0.5, null, null);
+ assertFalse(listener.mayPause(), "a nonzero MinCapacity never pauses");
+ rdsService.modifyDbCluster("paused", null, null, 0.0, null, null);
+ assertTrue(listener.mayPause());
+
+ rdsService.createGlobalCluster("global-1", "paused", null, null, null, null, null,
+ Map.of(), "us-east-1");
+ assertFalse(listener.mayPause(), "a global database keeps its primary awake");
+ }
+
+ @Test
+ void auroraClusterBehindAnRdsProxyDoesNotAutoPause() {
+ InMemoryStorage<String, DbProxyTargetGroup> targetGroups = new InMemoryStorage<>();
+ RdsService service = proxyStoreService(regionResolver, config, new InMemoryStorage<>(),
+ targetGroups, new InMemoryStorage<>(), new InMemoryStorage<>());
+ DbCluster cluster = createAutoPauseCluster(service, "paused", 300);
+ createMember(service, "paused-1", "db.serverless", "paused");
+ AutoPauseListener listener = capturedAutoPauseListener(cluster);
+ assertTrue(listener.mayPause());
+
+ DbProxyTargetGroup targetGroup = persistedTargetGroup("app-proxy", "us-east-1", "123456789012", "abc");
+ targetGroup.setTargets(List.of(new DbProxyTarget("TRACKED_CLUSTER", "paused",
+ cluster.getDbClusterArn(), "localhost", 5432)));
+ targetGroups.put("us-east-1::app-proxy", targetGroup);
+
+ assertFalse(listener.mayPause(), "an RDS Proxy holds connections open to its targets");
+ }
+
+ @Test
+ void autoPauseStepsAreRecordedAsEventsOfEachServerlessInstance() {
+ DbCluster cluster = createAutoPauseCluster(rdsService, "paused", 300);
+ DbInstance member = createMember(rdsService, "paused-1", "db.serverless", "paused");
+ AutoPauseListener listener = capturedAutoPauseListener(cluster);
+ Instant start = Instant.now();
+
+ listener.onAutoPause(AutoPauseListener.Event.PAUSE_INITIATED, start);
+ listener.onAutoPause(AutoPauseListener.Event.PAUSED, start.plusSeconds(1));
+ listener.onAutoPause(AutoPauseListener.Event.STILL_PAUSED, start.plusSeconds(61));
+ listener.onAutoPause(AutoPauseListener.Event.RESUME_INITIATED, start.plusSeconds(90));
+ listener.onAutoPause(AutoPauseListener.Event.RESUMED, start.plusSeconds(91));
+
+ List<RdsEvent> events = rdsService.describeEvents(
+ "paused-1", "db-instance", start.minusSeconds(1), start.plusSeconds(120), null);
+ assertEquals(List.of(
+ "Initiated pause for the DB instance.",
+ "Successfully paused the DB instance.",
+ "Initiated resume for the DB instance.",
+ "Successfully resumed the DB instance."),
+ events.stream().map(RdsEvent::message).toList());
+ RdsEvent paused = events.get(1);
+ assertEquals(List.of("notification", "serverless"), paused.eventCategories());
+ assertEquals(member.getDbInstanceArn(), paused.sourceArn());
+ assertEquals(start.plusSeconds(1), paused.date());
+ }
+
+ private static DbCluster createAutoPauseCluster(RdsService service, String id, int secondsUntilAutoPause) {
+ return service.createDbCluster(id, "aurora-postgresql", "16.3", "admin", "password", "dbname",
+ false, null, null, null, false, "us-east-1", 0.0, 4.0, secondsUntilAutoPause);
+ }
+
+ private static DbInstance createMember(RdsService service, String id, String instanceClass, String clusterId) {
+ return service.createDbInstance(id, "aurora-postgresql", "16.3", "admin", "password", "dbname",
+ instanceClass, 20, false, null, null, clusterId);
+ }
+
+ private AutoPauseListener capturedAutoPauseListener(DbCluster cluster) {
+ ArgumentCaptor<AutoPauseListener> listener = ArgumentCaptor.forClass(AutoPauseListener.class);
+ verify(containerManager, atLeastOnce())
+ .configureAutoPause(eq(cluster.getDbClusterArn()), any(), listener.capture());
+ return listener.getValue();
+ }
+
@Test
void modifyDbClusterRejectsServerlessV2ScalingForNonAuroraCluster() {
when(config.services().rds().mock()).thenReturn(true);
diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/container/AutoPauseControllerTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/container/AutoPauseControllerTest.java
new file mode 100644
index 0000000000..2fa57ce614
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/rds/container/AutoPauseControllerTest.java
@@ -0,0 +1,567 @@
+package io.github.hectorvent.floci.services.rds.container;
+
+import io.github.hectorvent.floci.services.rds.container.AutoPauseController.State;
+import io.github.hectorvent.floci.services.rds.container.AutoPauseListener.Event;
+import org.junit.jupiter.api.Test;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Future;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BooleanSupplier;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+
+class AutoPauseControllerTest {
+
+ private static final Duration IDLE = Duration.ofSeconds(300);
+
+ private final ManualScheduler scheduler = new ManualScheduler();
+ private final FakeFreezer freezer = new FakeFreezer();
+ private final RecordingListener listener = new RecordingListener();
+
+ @Test
+ void pausesOnceIdleForSecondsUntilAutoPause() {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+
+ scheduler.advance(Duration.ofSeconds(299));
+ assertEquals(State.RUNNING, controller.state());
+ assertEquals(0, freezer.pauses.get());
+
+ scheduler.advance(Duration.ofSeconds(1));
+ assertEquals(State.PAUSED, controller.state());
+ assertTrue(freezer.paused);
+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSED), listener.events);
+ assertTrue(listener.times.get(1).isAfter(listener.times.get(0)),
+ "events raised within one instant still keep their order");
+ }
+
+ @Test
+ void openConnectionsKeepItAwakeAndTheIdleClockStartsWhenTheLastOneCloses() throws Exception {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ RdsBackendGate.Lease first = controller.acquire();
+ RdsBackendGate.Lease second = controller.acquire();
+
+ scheduler.advance(Duration.ofHours(1));
+ first.close();
+ scheduler.advance(Duration.ofHours(1));
+ assertEquals(State.RUNNING, controller.state(), "one connection is still open");
+
+ second.close();
+ scheduler.advance(Duration.ofSeconds(299));
+ assertEquals(State.RUNNING, controller.state());
+ scheduler.advance(Duration.ofSeconds(1));
+ assertEquals(State.PAUSED, controller.state());
+ }
+
+ @Test
+ void connectionResumesAPausedContainerBeforeItIsServed() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ZERO);
+
+ RdsBackendGate.Lease lease = controller.acquire();
+
+ assertEquals(State.RUNNING, controller.state());
+ assertFalse(freezer.paused);
+ assertEquals(1, freezer.unpauses.get());
+ assertEquals(1, controller.leaseCount());
+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSED, Event.RESUME_INITIATED, Event.RESUMED),
+ listener.events);
+
+ lease.close();
+ scheduler.advance(IDLE);
+ assertEquals(State.PAUSED, controller.state(), "idle again after the connection closed");
+ }
+
+ @Test
+ void resumeDelayHoldsConnectionsIncludingOnesThatArriveWhileResuming() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ofSeconds(15));
+
+ Acquirer first = new Acquirer(controller, freezer);
+ awaitCondition(() -> controller.state() == State.RESUMING && first.isWaiting(), "first waits");
+ Acquirer second = new Acquirer(controller, freezer);
+ second.awaitWaiting();
+
+ scheduler.advance(Duration.ofSeconds(14));
+ assertFalse(first.isDone());
+ assertFalse(second.isDone());
+ assertEquals(State.RESUMING, controller.state());
+
+ scheduler.advance(Duration.ofSeconds(1));
+ first.lease();
+ second.lease();
+ assertFalse(first.sawFrozenContainer);
+ assertFalse(second.sawFrozenContainer);
+ assertEquals(1, freezer.unpauses.get());
+ assertEquals(2, controller.leaseCount());
+ }
+
+ @Test
+ void connectionDuringPausingCancelsThePause() throws Exception {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ freezer.blockPause();
+ Thread pausing = Thread.ofPlatform().daemon().start(() -> scheduler.advance(IDLE));
+ freezer.awaitPauseEntered();
+ assertEquals(State.PAUSING, controller.state());
+
+ Acquirer client = new Acquirer(controller, freezer);
+ client.awaitWaiting();
+ freezer.releasePause();
+ RdsBackendGate.Lease lease = client.lease();
+ pausing.join(TimeUnit.SECONDS.toMillis(5));
+
+ assertFalse(client.sawFrozenContainer, "the client must not reach the frozen container");
+ assertEquals(State.RUNNING, controller.state());
+ assertFalse(freezer.paused);
+ assertEquals(1, freezer.unpauses.get());
+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSE_CANCELED), listener.events);
+ assertEquals(1, controller.leaseCount());
+
+ lease.close();
+ scheduler.advance(IDLE);
+ assertEquals(State.PAUSED, controller.state());
+ }
+
+ @Test
+ void concurrentConnectionsResumeTheContainerOnce() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ZERO);
+ freezer.blockUnpause();
+
+ List<Acquirer> clients = new ArrayList<>();
+ for (int i = 0; i < 16; i++) {
+ clients.add(new Acquirer(controller, freezer));
+ }
+ freezer.awaitUnpauseEntered();
+ awaitCondition(() -> clients.stream().allMatch(Acquirer::isWaiting), "every client waits");
+ freezer.releaseUnpause();
+ for (Acquirer client : clients) {
+ client.lease();
+ assertFalse(client.sawFrozenContainer);
+ }
+
+ assertEquals(1, freezer.unpauses.get());
+ assertEquals(16, controller.leaseCount());
+ assertEquals(1, listener.events.stream().filter(event -> event == Event.RESUMED).count());
+ }
+
+ @Test
+ void turningAutoPauseOffResumesAPausedContainerAndKeepsItRunning() {
+ AutoPauseController controller = pausedController(Duration.ZERO);
+
+ controller.configure(null, listener);
+ scheduler.advance(Duration.ZERO);
+
+ assertEquals(State.RUNNING, controller.state());
+ assertFalse(freezer.paused);
+ scheduler.advance(Duration.ofDays(1));
+ assertEquals(State.RUNNING, controller.state());
+ assertEquals(1, freezer.pauses.get());
+ }
+
+ @Test
+ void turningAutoPauseOffWhilePausingCancelsThePause() throws Exception {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ freezer.blockPause();
+ Thread pausing = Thread.ofPlatform().daemon().start(() -> scheduler.advance(IDLE));
+ freezer.awaitPauseEntered();
+
+ controller.configure(null, listener);
+ freezer.releasePause();
+ pausing.join(TimeUnit.SECONDS.toMillis(5));
+
+ assertEquals(State.RUNNING, controller.state());
+ assertFalse(freezer.paused);
+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSE_CANCELED), listener.events);
+ scheduler.advance(Duration.ofDays(1));
+ assertEquals(1, freezer.pauses.get());
+ }
+
+ @Test
+ void turningAutoPauseOffBeforeTheIntervalEndsPreventsThePause() {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ scheduler.advance(Duration.ofSeconds(200));
+
+ controller.configure(null, listener);
+ scheduler.advance(Duration.ofDays(1));
+
+ assertEquals(State.RUNNING, controller.state());
+ assertEquals(0, freezer.pauses.get());
+ assertEquals(0, listener.checks.get());
+ }
+
+ @Test
+ void changingTheIntervalRestartsTheIdleClock() {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ scheduler.advance(Duration.ofSeconds(200));
+
+ controller.configure(600, listener);
+ scheduler.advance(Duration.ofSeconds(599));
+ assertEquals(State.RUNNING, controller.state());
+ scheduler.advance(Duration.ofSeconds(1));
+ assertEquals(State.PAUSED, controller.state());
+ }
+
+ @Test
+ void clusterThatMayNotPauseIsAskedAgainAfterAnotherInterval() {
+ AutoPauseController controller = controller(Duration.ZERO);
+ listener.mayPause = false;
+ controller.configure(300, listener);
+
+ scheduler.advance(IDLE);
+ assertEquals(State.RUNNING, controller.state());
+ assertEquals(1, listener.checks.get());
+ assertEquals(0, freezer.pauses.get());
+
+ listener.mayPause = true;
+ scheduler.advance(IDLE);
+ assertEquals(State.PAUSED, controller.state());
+ assertEquals(2, listener.checks.get());
+ }
+
+ @Test
+ void failedPauseLeavesTheContainerRunningAndTriesAgainLater() {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ freezer.pauseFailure = new IllegalStateException("daemon unavailable");
+
+ scheduler.advance(IDLE);
+ assertEquals(State.RUNNING, controller.state());
+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSE_CANCELED), listener.events);
+
+ freezer.pauseFailure = null;
+ scheduler.advance(IDLE);
+ assertEquals(State.PAUSED, controller.state());
+ }
+
+ @Test
+ void failedResumeFailsTheConnectionAndLeavesTheContainerPaused() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ZERO);
+ freezer.unpauseFailure = new IllegalStateException("daemon unavailable");
+
+ assertThrows(IllegalStateException.class, controller::acquire);
+ assertEquals(State.PAUSED, controller.state());
+
+ freezer.unpauseFailure = null;
+ controller.acquire();
+ assertEquals(State.RUNNING, controller.state());
+ assertFalse(freezer.paused);
+ }
+
+ @Test
+ void closeThawsAPausedContainerAndStopsTracking() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ZERO);
+
+ controller.close();
+
+ assertFalse(freezer.paused);
+ assertSame(RdsBackendGate.Lease.NONE, controller.acquire());
+ assertEquals(0, controller.leaseCount());
+ scheduler.advance(Duration.ofDays(1));
+ assertEquals(1, freezer.pauses.get());
+ }
+
+ @Test
+ void closeDuringTheResumeDelayReleasesTheWaitingConnection() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ofSeconds(15));
+ Acquirer client = new Acquirer(controller, freezer);
+ awaitCondition(() -> controller.state() == State.RESUMING && client.isWaiting(), "client waits");
+
+ controller.close();
+
+ assertSame(RdsBackendGate.Lease.NONE, client.lease());
+ assertFalse(freezer.paused);
+ }
+
+ @Test
+ void closeWhilePausingThawsTheContainerOnceTheFreezeReturns() throws Exception {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ freezer.blockPause();
+ Thread pausing = Thread.ofPlatform().daemon().start(() -> scheduler.advance(IDLE));
+ freezer.awaitPauseEntered();
+
+ controller.close();
+ freezer.releasePause();
+ pausing.join(TimeUnit.SECONDS.toMillis(5));
+
+ assertFalse(freezer.paused, "a released container must not stay frozen");
+ assertEquals(State.RUNNING, controller.state());
+ }
+
+ @Test
+ void pausedContainerReportsEveryMinuteUntilItResumes() throws Exception {
+ AutoPauseController controller = pausedController(Duration.ZERO);
+
+ scheduler.advance(Duration.ofMinutes(2));
+ assertEquals(2, listener.events.stream().filter(event -> event == Event.STILL_PAUSED).count());
+
+ controller.acquire();
+ scheduler.advance(Duration.ofMinutes(10));
+ assertEquals(2, listener.events.stream().filter(event -> event == Event.STILL_PAUSED).count());
+ }
+
+ @Test
+ void closingALeaseTwiceCountsOnce() throws Exception {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ RdsBackendGate.Lease first = controller.acquire();
+ controller.acquire();
+
+ first.close();
+ first.close();
+
+ assertEquals(1, controller.leaseCount());
+ }
+
+ @Test
+ void pauseIfIdleRunsTheIdleCheckWithoutWaiting() throws Exception {
+ AutoPauseController controller = controller(Duration.ZERO);
+ controller.configure(300, listener);
+ RdsBackendGate.Lease lease = controller.acquire();
+ assertFalse(controller.pauseIfIdle(), "a connection is open");
+
+ lease.close();
+ assertTrue(controller.pauseIfIdle());
+ assertEquals(State.PAUSED, controller.state());
+ }
+
+ private AutoPauseController controller(Duration resumeDelay) {
+ return new AutoPauseController("container-1", "localhost", 5432, freezer, scheduler,
+ Runnable::run, scheduler.clock(), resumeDelay);
+ }
+
+ private AutoPauseController pausedController(Duration resumeDelay) {
+ AutoPauseController controller = controller(resumeDelay);
+ controller.configure(300, listener);
+ scheduler.advance(IDLE);
+ assertEquals(State.PAUSED, controller.state());
+ return controller;
+ }
+
+ private static void awaitCondition(BooleanSupplier condition, String description) {
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
+ while (!condition.getAsBoolean()) {
+ if (System.nanoTime() > deadline) {
+ fail("Timed out waiting until " + description);
+ }
+ Thread.onSpinWait();
+ }
+ }
+
+ /** Runs scheduled tasks on whichever thread advances its clock, in the order they fall due. */
+ private static final class ManualScheduler implements AutoPauseController.Scheduler {
+
+ private Instant now = Instant.parse("2026-01-01T00:00:00Z");
+ private long sequence;
+ private final List<Scheduled> tasks = new ArrayList<>();
+
+ @Override
+ public synchronized Future<?> schedule(Runnable task, Duration delay) {
+ FutureTask<Void> future = new FutureTask<>(task, null);
+ tasks.add(new Scheduled(now.plus(delay), sequence++, future));
+ return future;
+ }
+
+ void advance(Duration duration) {
+ Instant target;
+ synchronized (this) {
+ target = now.plus(duration);
+ }
+ while (true) {
+ Scheduled next;
+ synchronized (this) {
+ next = tasks.stream()
+ .filter(task -> !task.due().isAfter(target))
+ .min(Comparator.comparing(Scheduled::due).thenComparingLong(Scheduled::sequence))
+ .orElse(null);
+ if (next == null) {
+ now = target;
+ return;
+ }
+ tasks.remove(next);
+ if (next.due().isAfter(now)) {
+ now = next.due();
+ }
+ }
+ // Outside the monitor: the task schedules its successors.
+ next.future().run();
+ }
+ }
+
+ Clock clock() {
+ return new Clock() {
+ @Override
+ public ZoneId getZone() {
+ return ZoneOffset.UTC;
+ }
+
+ @Override
+ public Clock withZone(ZoneId zone) {
+ return this;
+ }
+
+ @Override
+ public Instant instant() {
+ synchronized (ManualScheduler.this) {
+ return now;
+ }
+ }
+ };
+ }
+
+ private record Scheduled(Instant due, long sequence, FutureTask<Void> future) {
+ }
+ }
+
+ private static final class FakeFreezer implements AutoPauseController.Freezer {
+
+ final AtomicInteger pauses = new AtomicInteger();
+ final AtomicInteger unpauses = new AtomicInteger();
+ volatile boolean paused;
+ volatile RuntimeException pauseFailure;
+ volatile RuntimeException unpauseFailure;
+ private volatile CountDownLatch pauseEntered;
+ private volatile CountDownLatch pauseRelease;
+ private volatile CountDownLatch unpauseEntered;
+ private volatile CountDownLatch unpauseRelease;
+
+ @Override
+ public void pause(String containerId) {
+ pauses.incrementAndGet();
+ block(pauseEntered, pauseRelease);
+ if (pauseFailure != null) {
+ throw pauseFailure;
+ }
+ paused = true;
+ }
+
+ @Override
+ public void unpause(String containerId) {
+ unpauses.incrementAndGet();
+ block(unpauseEntered, unpauseRelease);
+ if (unpauseFailure != null) {
+ throw unpauseFailure;
+ }
+ paused = false;
+ }
+
+ void blockPause() {
+ pauseEntered = new CountDownLatch(1);
+ pauseRelease = new CountDownLatch(1);
+ }
+
+ void awaitPauseEntered() throws InterruptedException {
+ assertTrue(pauseEntered.await(5, TimeUnit.SECONDS), "docker pause was not called");
+ }
+
+ void releasePause() {
+ pauseRelease.countDown();
+ }
+
+ void blockUnpause() {
+ unpauseEntered = new CountDownLatch(1);
+ unpauseRelease = new CountDownLatch(1);
+ }
+
+ void awaitUnpauseEntered() throws InterruptedException {
+ assertTrue(unpauseEntered.await(5, TimeUnit.SECONDS), "docker unpause was not called");
+ }
+
+ void releaseUnpause() {
+ unpauseRelease.countDown();
+ }
+
+ private static void block(CountDownLatch entered, CountDownLatch release) {
+ if (entered == null) {
+ return;
+ }
+ entered.countDown();
+ try {
+ if (!release.await(5, TimeUnit.SECONDS)) {
+ throw new IllegalStateException("The test never released the Docker call");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException(e);
+ }
+ }
+ }
+
+ private static final class RecordingListener implements AutoPauseListener {
+
+ final List<Event> events = new CopyOnWriteArrayList<>();
+ final List<Instant> times = new CopyOnWriteArrayList<>();
+ final AtomicInteger checks = new AtomicInteger();
+ volatile boolean mayPause = true;
+
+ @Override
+ public boolean mayPause() {
+ checks.incrementAndGet();
+ return mayPause;
+ }
+
+ @Override
+ public void onAutoPause(Event event, Instant at) {
+ events.add(event);
+ times.add(at);
+ }
+ }
+
+ /** Calls acquire() on a platform thread of its own, so that a test can watch it wait. */
+ private static final class Acquirer {
+
+ private final CompletableFuture<RdsBackendGate.Lease> result = new CompletableFuture<>();
+ private final Thread thread;
+ private volatile boolean sawFrozenContainer;
+
+ Acquirer(AutoPauseController controller, FakeFreezer freezer) {
+ thread = Thread.ofPlatform().daemon().start(() -> {
+ try {
+ RdsBackendGate.Lease lease = controller.acquire();
+ sawFrozenContainer = freezer.paused;
+ result.complete(lease);
+ } catch (InterruptedException | RuntimeException e) {
+ result.completeExceptionally(e);
+ }
+ });
+ }
+
+ boolean isWaiting() {
+ Thread.State state = thread.getState();
+ return state == Thread.State.WAITING || state == Thread.State.TIMED_WAITING;
+ }
+
+ void awaitWaiting() {
+ awaitCondition(this::isWaiting, "the client waits");
+ }
+
+ boolean isDone() {
+ return result.isDone();
+ }
+
+ RdsBackendGate.Lease lease() throws Exception {
+ return result.get(5, TimeUnit.SECONDS);
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsAuroraAutoPauseIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsAuroraAutoPauseIntegrationTest.java
new file mode 100644
index 0000000000..9143d6df22
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsAuroraAutoPauseIntegrationTest.java
@@ -0,0 +1,259 @@
+package io.github.hectorvent.floci.services.rds.container;
+
+import com.github.dockerjava.api.DockerClient;
+import io.github.hectorvent.floci.services.rds.RdsService;
+import io.github.hectorvent.floci.services.rds.model.DbCluster;
+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
+import io.quarkus.test.junit.QuarkusTest;
+import io.restassured.path.xml.XmlPath;
+import io.restassured.response.Response;
+import io.restassured.specification.RequestSpecification;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.List;
+
+import static io.restassured.RestAssured.given;
+import static io.restassured.http.ContentType.URLENC;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.not;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Aurora Serverless v2 auto-pause against a real PostgreSQL container: an idle cluster with
+ * MinCapacity 0 is frozen with docker pause while it reports "available", zero capacity and the
+ * pause events, and the next connection through its endpoint resumes it and finds its data. The
+ * API does not accept a SecondsUntilAutoPause below 300, so the test runs the idle check directly
+ * instead of waiting for it.
+ */
+@QuarkusTest
+@Tag("docker")
+class RdsAuroraAutoPauseIntegrationTest {
+
+ private static final String RDS_AUTH =
+ "AWS4-HMAC-SHA256 Credential=test/20260923/us-east-1/rds/aws4_request, "
+ + "SignedHeaders=content-type;host, Signature=test";
+ private static final String OWNER_RDS_AUTH =
+ "AWS4-HMAC-SHA256 Credential=222222222222/20260923/us-east-1/rds/aws4_request, "
+ + "SignedHeaders=content-type;host, Signature=test";
+ private static final String CLOUDWATCH_AUTH =
+ "AWS4-HMAC-SHA256 Credential=test/20260923/us-east-1/monitoring/aws4_request";
+ private static final String MASTER_USER = "master";
+ private static final String MASTER_PASSWORD = "Master123!";
+
+ @Inject
+ DockerClient dockerClient;
+
+ @Inject
+ RdsService rdsService;
+
+ @Inject
+ RdsContainerManager containerManager;
+
+ @BeforeAll
+ static void configureRestAssured() {
+ RestAssuredJsonUtils.configureAwsContentTypes();
+ }
+
+ @BeforeEach
+ void checkDocker() {
+ try {
+ dockerClient.pingCmd().exec();
+ } catch (Exception e) {
+ assumeTrue(false, "Docker is not available: " + e.getMessage());
+ }
+ }
+
+ @Test
+ void idleClusterPausesAndItsNextConnectionResumesIt() throws Exception {
+ String clusterId = "auto-pause-" + Long.toString(System.nanoTime(), 36);
+ String instanceId = clusterId + "-1";
+ Instant start = Instant.now();
+ rds("CreateDBCluster")
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("Engine", "aurora-postgresql")
+ .formParam("MasterUsername", MASTER_USER)
+ .formParam("MasterUserPassword", MASTER_PASSWORD)
+ .formParam("DatabaseName", "appdb")
+ .formParam("ServerlessV2ScalingConfiguration.MinCapacity", "0")
+ .formParam("ServerlessV2ScalingConfiguration.MaxCapacity", "2")
+ .formParam("ServerlessV2ScalingConfiguration.SecondsUntilAutoPause", "300")
+ .when().post("/").then().statusCode(200);
+ try {
+ rds("CreateDBInstance")
+ .formParam("DBInstanceIdentifier", instanceId)
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("Engine", "aurora-postgresql")
+ .formParam("DBInstanceClass", "db.serverless")
+ .when().post("/").then().statusCode(200);
+ DbCluster cluster = rdsService.getDbCluster(clusterId);
+ String containerId = cluster.getContainerId();
+ String jdbcUrl = "jdbc:postgresql://127.0.0.1:" + cluster.getEndpoint().port()
+ + "/appdb?sslmode=disable";
+ try (Connection connection = DriverManager.getConnection(jdbcUrl, MASTER_USER, MASTER_PASSWORD);
+ Statement statement = connection.createStatement()) {
+ statement.execute("CREATE TABLE marker(v text); INSERT INTO marker VALUES ('before-pause');");
+ }
+
+ // The proxy lets go of the closed connection on its own thread.
+ await().atMost(Duration.ofSeconds(10))
+ .until(() -> containerManager.pauseIfIdle(cluster.getDbClusterArn()));
+ assertTrue(isPaused(containerId), "the idle cluster's container is frozen");
+ rds("DescribeDBClusters").formParam("DBClusterIdentifier", clusterId)
+ .when().post("/").then().statusCode(200)
+ .body(containsString("<Status>available</Status>"));
+ rds("DescribeDBInstances").formParam("DBInstanceIdentifier", instanceId)
+ .when().post("/").then().statusCode(200)
+ .body(containsString("<DBInstanceStatus>available</DBInstanceStatus>"));
+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
+ events(instanceId).then().statusCode(200)
+ .body(containsString("<Message>Initiated pause for the DB instance.</Message>"))
+ .body(containsString("<Message>Successfully paused the DB instance.</Message>")));
+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
+ for (String metricName : List.of("ServerlessDatabaseCapacity", "CPUUtilization")) {
+ List<Float> minimums = metricMinimums(metricName, clusterId, start);
+ assertFalse(minimums.isEmpty(), "a paused cluster reports its " + metricName);
+ assertEquals(0.0f, minimums.get(minimums.size() - 1), metricName);
+ }
+ });
+
+ try (Connection connection = DriverManager.getConnection(jdbcUrl, MASTER_USER, MASTER_PASSWORD);
+ Statement statement = connection.createStatement();
+ ResultSet rows = statement.executeQuery("SELECT v FROM marker")) {
+ assertTrue(rows.next(), "the data written before the pause is still there");
+ assertEquals("before-pause", rows.getString(1));
+ }
+ assertFalse(isPaused(containerId), "the connection resumed the container");
+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
+ events(instanceId).then().statusCode(200)
+ .body(containsString("<Message>Initiated resume for the DB instance.</Message>"))
+ .body(containsString("<Message>Successfully resumed the DB instance.</Message>")));
+
+ await().atMost(Duration.ofSeconds(10))
+ .until(() -> containerManager.pauseIfIdle(cluster.getDbClusterArn()));
+ rds("ModifyDBCluster")
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("ServerlessV2ScalingConfiguration.MinCapacity", "0.5")
+ .formParam("ServerlessV2ScalingConfiguration.MaxCapacity", "2")
+ .when().post("/").then().statusCode(200);
+ await().atMost(Duration.ofSeconds(10)).until(() -> !isPaused(containerId));
+ assertFalse(containerManager.pauseIfIdle(cluster.getDbClusterArn()),
+ "a nonzero MinCapacity keeps the cluster running");
+ } finally {
+ rds("DeleteDBInstance")
+ .formParam("DBInstanceIdentifier", instanceId)
+ .formParam("SkipFinalSnapshot", "true")
+ .when().post("/");
+ rds("DeleteDBCluster")
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("SkipFinalSnapshot", "true")
+ .when().post("/");
+ }
+ }
+
+ @Test
+ void pauseEventsBelongToTheAccountThatOwnsTheCluster() throws Exception {
+ String clusterId = "auto-pause-owner-" + Long.toString(System.nanoTime(), 36);
+ String instanceId = clusterId + "-1";
+ XmlPath created = rds(OWNER_RDS_AUTH, "CreateDBCluster")
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("Engine", "aurora-postgresql")
+ .formParam("MasterUsername", MASTER_USER)
+ .formParam("MasterUserPassword", MASTER_PASSWORD)
+ .formParam("DatabaseName", "appdb")
+ .formParam("ServerlessV2ScalingConfiguration.MinCapacity", "0")
+ .formParam("ServerlessV2ScalingConfiguration.MaxCapacity", "2")
+ .formParam("ServerlessV2ScalingConfiguration.SecondsUntilAutoPause", "300")
+ .when().post("/").then().statusCode(200).extract().xmlPath();
+ String clusterArn = created.getString("CreateDBClusterResponse.CreateDBClusterResult.DBCluster.DBClusterArn");
+ int port = created.getInt("CreateDBClusterResponse.CreateDBClusterResult.DBCluster.Port");
+ try {
+ rds(OWNER_RDS_AUTH, "CreateDBInstance")
+ .formParam("DBInstanceIdentifier", instanceId)
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("Engine", "aurora-postgresql")
+ .formParam("DBInstanceClass", "db.serverless")
+ .when().post("/").then().statusCode(200);
+
+ // The pause and resume events reach RDS on a background thread, outside any request.
+ await().atMost(Duration.ofSeconds(10)).until(() -> containerManager.pauseIfIdle(clusterArn));
+ try (Connection connection = DriverManager.getConnection(
+ "jdbc:postgresql://127.0.0.1:" + port + "/appdb?sslmode=disable", MASTER_USER, MASTER_PASSWORD)) {
+ assertTrue(connection.isValid(5), "the connection resumed the cluster");
+ }
+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
+ events(OWNER_RDS_AUTH, instanceId).then().statusCode(200)
+ .body(containsString("<Message>Initiated pause for the DB instance.</Message>"))
+ .body(containsString("<Message>Successfully paused the DB instance.</Message>"))
+ .body(containsString("<Message>Initiated resume for the DB instance.</Message>"))
+ .body(containsString("<Message>Successfully resumed the DB instance.</Message>")));
+ events(RDS_AUTH, instanceId).then().statusCode(200)
+ .body(not(containsString("<Event>")));
+ } finally {
+ rds(OWNER_RDS_AUTH, "DeleteDBInstance")
+ .formParam("DBInstanceIdentifier", instanceId)
+ .formParam("SkipFinalSnapshot", "true")
+ .when().post("/");
+ rds(OWNER_RDS_AUTH, "DeleteDBCluster")
+ .formParam("DBClusterIdentifier", clusterId)
+ .formParam("SkipFinalSnapshot", "true")
+ .when().post("/");
+ }
+ }
+
+ private static RequestSpecification rds(String action) {
+ return rds(RDS_AUTH, action);
+ }
+
+ private static RequestSpecification rds(String authorization, String action) {
+ return given().header("Authorization", authorization)
+ .contentType(URLENC)
+ .formParam("Action", action)
+ .formParam("Version", "2014-10-31");
+ }
+
+ private static Response events(String instanceId) {
+ return events(RDS_AUTH, instanceId);
+ }
+
+ private static Response events(String authorization, String instanceId) {
+ return rds(authorization, "DescribeEvents")
+ .formParam("SourceType", "db-instance")
+ .formParam("SourceIdentifier", instanceId)
+ .formParam("Duration", "60")
+ .when().post("/");
+ }
+
+ private static List<Float> metricMinimums(String metricName, String clusterId, Instant from) {
+ String body = """
+ {"Namespace":"AWS/RDS","MetricName":"%s",
+ "Dimensions":[{"Name":"DBClusterIdentifier","Value":"%s"}],
+ "StartTime":%d,"EndTime":%d,"Period":60,"Statistics":["Minimum"]}
+ """.formatted(metricName, clusterId, from.minusSeconds(60).getEpochSecond(),
+ Instant.now().plusSeconds(60).getEpochSecond());
+ return given().contentType("application/x-amz-json-1.0")
+ .header("Authorization", CLOUDWATCH_AUTH)
+ .header("X-Amz-Target", "GraniteServiceVersion20100801.GetMetricStatistics")
+ .body(body)
+ .post("/")
+ .then().statusCode(200)
+ .extract().jsonPath().getList("Datapoints.Minimum", Float.class);
+ }
+
+ private boolean isPaused(String containerId) {
+ return Boolean.TRUE.equals(dockerClient.inspectContainerCmd(containerId).exec().getState().getPaused());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java
index 65bcca44a0..7f5d9a7175 100644
--- a/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java
@@ -12,9 +12,11 @@ import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.mockito.InOrder;
import java.nio.file.Files;
import java.nio.file.Path;
+import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -24,6 +26,8 @@ import java.util.concurrent.TimeUnit;
import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.command.CopyArchiveToContainerCmd;
+import com.github.dockerjava.api.command.PauseContainerCmd;
+import com.github.dockerjava.api.command.UnpauseContainerCmd;
import com.github.dockerjava.api.model.Bind;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
@@ -31,14 +35,17 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -1155,6 +1162,118 @@ class RdsContainerManagerTest {
"Script must assign the first argument to the USER variable");
}
+ @Test
+ void autoPausedContainerResumesForItsNextClientAndIsThawedBeforeItStops() throws Exception {
+ EmulatorConfig config = config(tempDir.resolve("auto-pause-root"));
+ when(config.services().rds().auroraAutoPauseEnabled()).thenReturn(true);
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ stubStarts(lifecycleManager, new ContainerLifecycleManager.ContainerInfo(
+ "cluster-container", Map.of(3306, new ContainerLifecycleManager.EndpointInfo("db1", 3306))));
+ DockerClient dockerClient = mock(DockerClient.class);
+ PauseContainerCmd pause = mock(PauseContainerCmd.class);
+ UnpauseContainerCmd unpause = mock(UnpauseContainerCmd.class);
+ when(dockerClient.pauseContainerCmd("cluster-container")).thenReturn(pause);
+ when(dockerClient.unpauseContainerCmd("cluster-container")).thenReturn(unpause);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+ RdsContainerManager manager = autoPauseManager(config, lifecycleManager);
+ String runtimeId = "arn:aws:rds:us-east-1:000000000000:cluster:paused";
+ RdsContainerHandle handle = manager.start(runtimeId, "paused", null,
+ DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");
+ manager.configureAutoPause(runtimeId, 300, alwaysMayPause());
+
+ try (RdsBackendGate.Lease client = manager.enter("db1", 3306)) {
+ assertFalse(manager.pauseIfIdle(runtimeId), "a client is connected");
+ }
+ assertTrue(manager.pauseIfIdle(runtimeId));
+ verify(pause).exec();
+
+ manager.enter("db1", 3306).close();
+ verify(unpause).exec();
+
+ assertTrue(manager.pauseIfIdle(runtimeId));
+ manager.stop(handle);
+ InOrder order = inOrder(unpause, lifecycleManager);
+ order.verify(unpause, times(2)).exec();
+ order.verify(lifecycleManager).stopAndRemoveStrict(eq("cluster-container"), any());
+ assertSame(RdsBackendGate.Lease.NONE, manager.enter("db1", 3306));
+ }
+
+ @Test
+ void resetThawsAPausedContainerAndEndsItsAutoPause() throws Exception {
+ EmulatorConfig config = config(tempDir.resolve("auto-pause-root"));
+ when(config.services().rds().auroraAutoPauseEnabled()).thenReturn(true);
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ stubStarts(lifecycleManager, new ContainerLifecycleManager.ContainerInfo(
+ "cluster-container", Map.of(3306, new ContainerLifecycleManager.EndpointInfo("db1", 3306))));
+ DockerClient dockerClient = mock(DockerClient.class);
+ UnpauseContainerCmd unpause = mock(UnpauseContainerCmd.class);
+ when(dockerClient.pauseContainerCmd("cluster-container")).thenReturn(mock(PauseContainerCmd.class));
+ when(dockerClient.unpauseContainerCmd("cluster-container")).thenReturn(unpause);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+ RdsContainerManager manager = autoPauseManager(config, lifecycleManager);
+ String runtimeId = "arn:aws:rds:us-east-1:000000000000:cluster:paused";
+ manager.start(runtimeId, "paused", null, DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");
+ manager.configureAutoPause(runtimeId, 300, alwaysMayPause());
+ assertTrue(manager.pauseIfIdle(runtimeId));
+
+ manager.clear();
+
+ verify(unpause).exec();
+ assertSame(RdsBackendGate.Lease.NONE, manager.enter("db1", 3306));
+ assertFalse(manager.pauseIfIdle(runtimeId));
+ }
+
+ @Test
+ void containerStartedOnAnAddressTakesItOverFromOneDockerRemoved() throws Exception {
+ EmulatorConfig config = config(tempDir.resolve("auto-pause-root"));
+ when(config.services().rds().auroraAutoPauseEnabled()).thenReturn(true);
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ ContainerLifecycleManager.EndpointInfo address = new ContainerLifecycleManager.EndpointInfo("db1", 3306);
+ stubStarts(lifecycleManager,
+ new ContainerLifecycleManager.ContainerInfo("removed-container", Map.of(3306, address)),
+ new ContainerLifecycleManager.ContainerInfo("new-container", Map.of(3306, address)));
+ DockerClient dockerClient = mock(DockerClient.class);
+ when(dockerClient.pauseContainerCmd(any())).thenReturn(mock(PauseContainerCmd.class));
+ when(dockerClient.unpauseContainerCmd(any())).thenReturn(mock(UnpauseContainerCmd.class));
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+ RdsContainerManager manager = autoPauseManager(config, lifecycleManager);
+ String removedRuntime = "arn:aws:rds:us-east-1:000000000000:cluster:removed";
+ manager.start(removedRuntime, "removed", null,
+ DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");
+ manager.configureAutoPause(removedRuntime, 300, alwaysMayPause());
+ assertTrue(manager.pauseIfIdle(removedRuntime));
+
+ manager.start("arn:aws:rds:us-east-1:000000000000:cluster:new", "new", null,
+ DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");
+ manager.enter("db1", 3306).close();
+
+ verify(dockerClient, never()).unpauseContainerCmd("new-container");
+ assertFalse(manager.pauseIfIdle(removedRuntime), "the removed container's auto-pause is gone");
+ }
+
+ private RdsContainerManager autoPauseManager(EmulatorConfig config, ContainerLifecycleManager lifecycleManager) {
+ ContainerLogStreamer logStreamer = mock(ContainerLogStreamer.class);
+ lenient().when(logStreamer.generateLogStreamName(any())).thenReturn("log-stream");
+ return new RdsContainerManager(
+ new ContainerBuilder(config, mock(DockerHostResolver.class), mock(EmbeddedDnsServer.class)),
+ lifecycleManager, logStreamer, mock(ContainerDetector.class), config,
+ new RegionResolver("us-east-1", "000000000000"),
+ mock(ServiceConfigAccess.class));
+ }
+
+ private static AutoPauseListener alwaysMayPause() {
+ return new AutoPauseListener() {
+ @Override
+ public boolean mayPause() {
+ return true;
+ }
+
+ @Override
+ public void onAutoPause(Event event, Instant at) {
+ }
+ };
+ }
+
private static EmulatorConfig config(Path hostRoot) {
EmulatorConfig config = mock(EmulatorConfig.class);
EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java
index ed758a7fde..9823ad7acf 100644
--- a/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java
@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.rds.proxy;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.services.acm.CertificateGenerator;
+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
@@ -13,7 +14,9 @@ import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketTimeoutException;
import java.nio.file.Path;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -100,6 +103,37 @@ class RdsAuthProxyBoundsTest {
}
}
+ @Test
+ void holdsTheClientUntilTheBackendGateAdmitsItAndReleasesTheLeaseWhenItLeaves() throws Exception {
+ fakeBackend = new ServerSocket(0);
+ fakeBackend.setSoTimeout(300);
+ CountDownLatch admit = new CountDownLatch(1);
+ CountDownLatch released = new CountDownLatch(1);
+ RdsBackendGate gate = (host, port) -> {
+ admit.await();
+ return released::countDown;
+ };
+ int proxyPort = freePort();
+ proxy = new RdsAuthProxy("db-1", "localhost", fakeBackend.getLocalPort(),
+ DatabaseEngine.SQLSERVER, false, "sa", "Secret123", null,
+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,
+ 5000, 5000, 100, null, gate);
+ proxy.start(proxyPort);
+
+ try (Socket client = new Socket("localhost", proxyPort)) {
+ client.setSoTimeout(5000);
+ assertThrows(SocketTimeoutException.class, fakeBackend::accept,
+ "the proxy reached the backend before the gate admitted the client");
+ admit.countDown();
+ fakeBackend.setSoTimeout(5000);
+ try (Socket backendSide = fakeBackend.accept()) {
+ backendSide.getOutputStream().write('x');
+ assertEquals('x', client.getInputStream().read(), "the held client is served once admitted");
+ }
+ }
+ assertTrue(released.await(5, TimeUnit.SECONDS), "the lease is released when the connection ends");
+ }
+
private RdsAuthProxy newProxy(int handshakeTimeoutMillis, int backendConnectTimeoutMillis,
int maxConnections) {
return new RdsAuthProxy("db-1", "localhost", fakeBackend.getLocalPort(),
diff --git a/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java
index b2798d1486..9421ca6b2a 100644
--- a/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java
@@ -5,6 +5,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;
import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;
import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;
import io.github.hectorvent.floci.services.secretsmanager.model.SecretVersion;
@@ -18,9 +19,12 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Types;
import java.time.Duration;
+import java.util.List;
import java.util.Properties;
import java.util.UUID;
+import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
import org.h2.Driver;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
@@ -773,6 +777,66 @@ class RdsDataServiceTest {
assertEquals(0L, harness.countById("shutdown"));
}
+ @Test
+ void statementOutsideATransactionHoldsTheClusterAwakeOnlyWhileItRuns() throws Exception {
+ TestHarness harness = new TestHarness();
+ harness.createTables();
+
+ harness.service.executeStatement(harness.request("select 1"), REGION);
+
+ assertEquals(List.of("127.0.0.1:3306"), harness.gate.entered);
+ assertEquals(0, harness.gate.held.get());
+ }
+
+ @Test
+ void transactionHoldsTheClusterAwakeUntilItEnds() throws Exception {
+ TestHarness harness = new TestHarness();
+ harness.createTables();
+
+ String committed = harness.service.beginTransaction(harness.beginRequest(), REGION)
+ .get("transactionId").asText();
+ ObjectNode insert = harness.request("insert into data_api_items(id, title, score) values ('held', 'Held', 1)");
+ insert.put("transactionId", committed);
+ harness.service.executeStatement(insert, REGION);
+ assertEquals(1, harness.gate.held.get(), "an open transaction keeps the cluster from pausing");
+ harness.service.commitTransaction(harness.transactionRequest(committed), REGION);
+ assertEquals(0, harness.gate.held.get());
+
+ String rolledBack = harness.service.beginTransaction(harness.beginRequest(), REGION)
+ .get("transactionId").asText();
+ assertEquals(1, harness.gate.held.get());
+ harness.service.rollbackTransaction(harness.transactionRequest(rolledBack), REGION);
+ assertEquals(0, harness.gate.held.get());
+ assertEquals(2, harness.gate.entered.size(), "statements in a transaction reuse its hold");
+ }
+
+ @Test
+ void expiredTransactionLetsTheClusterPause() throws Exception {
+ TestHarness harness = new TestHarness(Duration.ofMillis(50));
+ String tx = harness.service.beginTransaction(harness.beginRequest(), REGION)
+ .get("transactionId").asText();
+ Thread.sleep(200);
+
+ ObjectNode next = harness.request("select 1");
+ next.put("transactionId", tx);
+ assertThrows(AwsException.class, () -> harness.service.executeStatement(next, REGION));
+
+ assertEquals(0, harness.gate.held.get());
+ }
+
+ @Test
+ void clusterThatCannotResumeFailsTheRequestAsAnInternalError() {
+ TestHarness harness = new TestHarness();
+ harness.gate.failure = new IllegalStateException("Could not resume auto-paused RDS container");
+
+ AwsException error = assertThrows(AwsException.class,
+ () -> harness.service.executeStatement(harness.request("select 1"), REGION));
+
+ assertEquals("InternalServerErrorException", error.getErrorCode());
+ assertEquals(500, error.getHttpStatus());
+ assertEquals(0, harness.gate.held.get());
+ }
+
@Test
void closesConnectionWhenTransactionSetupFails() {
RdsDataResourceResolver resolver = mock(RdsDataResourceResolver.class);
@@ -921,10 +985,33 @@ class RdsDataServiceTest {
});
}
+ /** Counts the backends the Data API enters and the holds still open on them. */
+ private static final class CountingGate implements RdsBackendGate {
+ private final List<String> entered = new CopyOnWriteArrayList<>();
+ private final AtomicInteger held = new AtomicInteger();
+ private volatile IllegalStateException failure;
+
+ @Override
+ public Lease enter(String host, int port) {
+ if (failure != null) {
+ throw failure;
+ }
+ entered.add(host + ":" + port);
+ held.incrementAndGet();
+ AtomicBoolean closed = new AtomicBoolean();
+ return () -> {
+ if (closed.compareAndSet(false, true)) {
+ held.decrementAndGet();
+ }
+ };
+ }
+ }
+
private final class TestHarness {
private final String jdbcUrl;
private final RdsDataResourceResolver resolver;
private final RdsDataResourceResolver.DatabaseTarget target;
+ private final CountingGate gate = new CountingGate();
private final RdsDataService service;
private TestHarness() {
@@ -966,7 +1053,7 @@ class RdsDataServiceTest {
return getConnection();
}
};
- service = new RdsDataService(resolver, secrets, objectMapper, connectionFactory, transactionTtl);
+ service = new RdsDataService(resolver, secrets, objectMapper, connectionFactory, transactionTtl, gate);
}
private Connection getConnection() throws SQLException {
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index bb48878108..5db99bc4ba 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -206,6 +206,8 @@ floci:
proxy-handshake-timeout-millis: 10000
proxy-backend-connect-timeout-millis: 5000
proxy-max-connections: 100
+ aurora-auto-pause-enabled: true
+ aurora-resume-delay-millis: 0
rds-data:
enabled: true
transaction-ttl-seconds: 180