Light, fluffy, and always free - The AWS Local Emulator alternative
feat(rds): pause and resume Aurora Serverless v2 clusters at zero capacity (#4232)
jwoa957e4d4dafae9d
diff --git a/docs/services/rds.md b/docs/services/rds.mdindex 11f03c463e..357335851c 100644--- a/docs/services/rds.md+++ b/docs/services/rds.md@@ -152,6 +152,8 @@ checked against the instance's other window. Modifications apply immediately —| `FLOCI_SERVICES_RDS_PROXY_HANDSHAKE_TIMEOUT_MILLIS` | `10000` | Max time a client has to complete the startup/auth handshake before the proxy drops it || `FLOCI_SERVICES_RDS_PROXY_BACKEND_CONNECT_TIMEOUT_MILLIS` | `5000` | Max time the proxy waits for the backend TCP connect || `FLOCI_SERVICES_RDS_PROXY_MAX_CONNECTIONS` | `100` | Max concurrent connections per proxy before new ones are refused |+| `FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED` | `true` | Pause an Aurora Serverless v2 cluster whose `MinCapacity` is 0 after `SecondsUntilAutoPause` without connections; see [Automatic pause and resume](#automatic-pause-and-resume) |+| `FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS` | `0` | How long the first connection to a paused cluster is held while it resumes; Aurora takes about `15000` |### Docker Compose@@ -258,8 +260,47 @@ values. `AWS::RDS::DBCluster` creation also maps the equivalent CloudFormation pIf a persisted cluster record does not contain its original AWS engine identifier, Floci rejects anew scaling configuration instead of assuming that the cluster is Aurora.-This is control-plane compatibility: Floci persists and returns the scaling configuration, but it-does not resize or automatically pause the backing Docker container.+Floci persists and returns the scaling configuration but does not resize the backing Docker+container: capacity between `MinCapacity` and `MaxCapacity` is not modeled. A `MinCapacity` of 0+does take effect, as described next.++### Automatic pause and resume++A cluster whose `MinCapacity` is 0 pauses the way Aurora Serverless v2 does. Once nothing has used+it for `SecondsUntilAutoPause`, Floci freezes its database container with `docker pause`, which+keeps its data and its endpoint. The next connection resumes it: the client is held, not refused,+until the database runs again, and is then served as usual. As on AWS, a connection attempt with+wrong credentials also resumes the cluster.++- **What keeps a cluster awake.** Every open connection through the cluster endpoint, the reader+ endpoint, an instance endpoint or an RDS Proxy, every RDS Data API call and every open Data API+ transaction. The idle clock starts when the last of them ends.+- **Which clusters pause.** Aurora PostgreSQL and Aurora MySQL clusters with at least one instance,+ all of them `db.serverless` and available. As on AWS, a cluster with a provisioned instance, a+ cluster in a global database and a cluster registered with an RDS Proxy stay running. Floci runs+ a cluster's instances in one container, so they pause and resume together, like Aurora's writer+ and its tier 0 and 1 readers; a reader in tiers 2 to 15 does not pause on its own.+- **What you can observe.** The cluster `Status` and each `DBInstanceStatus` stay `available`.+ `DescribeEvents` reports RDS-EVENT-0370 to 0374 for each instance, for example "Successfully+ paused the DB instance.". While the cluster is paused, `ServerlessDatabaseCapacity` reports 0+ and `ACUUtilization` and `CPUUtilization` report 0 percent in the `AWS/RDS` namespace once a+ minute, by `DBClusterIdentifier` and by `DBInstanceIdentifier`. Floci publishes none of them+ while the cluster runs.+- **Resume delay.** Aurora takes about 15 seconds to resume. Floci resumes at once unless+ `FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS` is set; `15000` exercises client connect timeouts,+ retries and connection-pool validation against a realistic cold start.+- **Configuration changes.** A `ModifyDBCluster` that raises `MinCapacity` above 0 or changes+ `SecondsUntilAutoPause` resumes a paused cluster and restarts its idle clock.+- **Commands Floci runs in the container.** `CreateDBClusterSnapshot` and a master password change+ run inside the database container, so they resume a paused cluster first. Aurora takes snapshots+ from storage without resuming it.+- Stopping, rebooting or deleting a cluster, restarting Floci and resetting its state never leave a+ container frozen. A cluster that was paused when Floci stopped comes back running.+- `FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED=false` keeps every cluster running.++Not emulated: Data API requests to a paused cluster wait for the resume, as the Aurora user guide+describes, and never fail with `DatabaseResumingException`; logical or binlog replication does not+keep a cluster awake; and parameter group changes do not resume one.## Modifying a DB instancediff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 1996668dcc..4d9828839e 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1560,6 +1560,18 @@ public interface EmulatorConfig {int proxyBackendConnectTimeoutMillis();@WithDefault("100")int proxyMaxConnections();++ /** Whether an Aurora Serverless v2 cluster with MinCapacity 0 pauses its container+ * after SecondsUntilAutoPause without connections, as Aurora does.+ * Env: FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED */+ @WithDefault("true")+ boolean auroraAutoPauseEnabled();++ /** How long the first connection to an auto-paused cluster is held while it resumes.+ * Aurora takes about 15 seconds; 0 resumes at once.+ * Env: FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS */+ @WithDefault("0")+ int auroraResumeDelayMillis();}interface RdsDataServiceConfig {diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java b/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.javaindex 250b4ebd43..4fd139b6ac 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/RdsService.java@@ -20,9 +20,13 @@ import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;import io.github.hectorvent.floci.core.storage.InMemoryStorage;import io.github.hectorvent.floci.core.storage.StorageBackend;import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.cloudwatch.metrics.CloudWatchMetricsService;+import io.github.hectorvent.floci.services.cloudwatch.metrics.model.Dimension;+import io.github.hectorvent.floci.services.cloudwatch.metrics.model.MetricDatum;import io.github.hectorvent.floci.services.ec2.Ec2Service;import io.github.hectorvent.floci.services.ec2.model.Subnet;import io.github.hectorvent.floci.services.ec2.model.Vpc;+import io.github.hectorvent.floci.services.rds.container.AutoPauseListener;import io.github.hectorvent.floci.services.rds.container.RdsContainerHandle;import io.github.hectorvent.floci.services.rds.container.RdsContainerManager;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;@@ -180,11 +184,16 @@ public class RdsService implements Resettable, ResourceProvider {private final DockerHostResolver dockerHostResolver;private final CurrentContainerNetworkResolver currentContainerNetworkResolver;private final ResourceGroupsTaggingService taggingService;+ // Null when a test constructs the service without CloudWatch; auto-pause then reports no metrics.+ private final CloudWatchMetricsService metricsService;private final Set<Integer> usedPorts = ConcurrentHashMap.newKeySet();private static final Pattern IMAGE_TAG_VERSION_PATTERN = Pattern.compile("^(\\d+(?:\\.\\d+)*)(.*)$");private static final Pattern SAFE_IMAGE_TAG_PATTERN = Pattern.compile("[A-Za-z0-9._-]+");private static final int SERVERLESS_V2_DEFAULT_AUTO_PAUSE_SECONDS = 300;private static final int SERVERLESS_V2_MAX_AUTO_PAUSE_SECONDS = 86_400;+ /** Only Aurora Serverless v2 instances pause; a provisioned one keeps its whole cluster awake. */+ private static final String SERVERLESS_V2_INSTANCE_CLASS = "db.serverless";+ private static final List<String> AUTO_PAUSE_EVENT_CATEGORIES = List.of("notification", "serverless");private static final Pattern DB_PROXY_NAME_PATTERN =Pattern.compile("[a-zA-Z](?:-?[a-zA-Z0-9]+)*");/**@@ -213,7 +222,8 @@ public class RdsService implements Resettable, ResourceProvider {DockerHostResolver dockerHostResolver,CurrentContainerNetworkResolver currentContainerNetworkResolver,ResourceGroupsTaggingService taggingService,- KmsService kmsService) {+ KmsService kmsService,+ CloudWatchMetricsService metricsService) {this.containerManager = containerManager;this.proxyManager = proxyManager;this.ec2Service = ec2Service;@@ -224,6 +234,7 @@ public class RdsService implements Resettable, ResourceProvider {this.dockerHostResolver = dockerHostResolver;this.currentContainerNetworkResolver = currentContainerNetworkResolver;this.taggingService = taggingService;+ this.metricsService = metricsService;this.instances = storageFactory.create("rds", "rds-instances.json",new TypeReference<Map<String, DbInstance>>() {});this.clusters = storageFactory.create("rds", "rds-clusters.json",@@ -381,6 +392,7 @@ public class RdsService implements Resettable, ResourceProvider {this.proxies = proxies;this.proxyTargetGroups = proxyTargetGroups;this.taggingService = taggingService;+ this.metricsService = null;this.globalClusters = new io.github.hectorvent.floci.core.storage.InMemoryStorage<>();this.snapshots = new io.github.hectorvent.floci.core.storage.InMemoryStorage<>();this.clusterSnapshots = new InMemoryStorage<>();@@ -2954,6 +2966,7 @@ public class RdsService implements Resettable, ResourceProvider {(user, pw) -> validateDbClusterPasswordForScope(accountId, clusterRegion, id, user, pw),proxyBinding(cluster.getEngine(), cluster.getEndpoint().address(), cluster.getProxyPort(),clusterRegion, accountId, cluster.getDbClusterResourceId()));+ applyAutoPause(cluster);}}@@ -3183,6 +3196,7 @@ public class RdsService implements Resettable, ResourceProvider {cluster.setContainerHost(started.getHost());cluster.setContainerPort(started.getPort());putClusterForScope(currentAccountId(), effectiveRegion, id, cluster);+ applyAutoPause(cluster);LOG.infov("Backing database container for DB cluster {0} started on retry", id);return cluster;}@@ -3469,6 +3483,7 @@ public class RdsService implements Resettable, ResourceProvider {}throw e;}+ applyAutoPause(cluster);LOG.infov("DB cluster {0} created (mock={1}), engine={2}, endpoint={3}:{4}",id, String.valueOf(mock), engine, endpoint.address(), String.valueOf(endpoint.port()));return cluster;@@ -3547,6 +3562,175 @@ public class RdsService implements Resettable, ResourceProvider {|| "aurora-postgresql".equalsIgnoreCase(engineIdentifier));}+ // ── Aurora Serverless v2 auto-pause ───────────────────────────────────────++ /**+ * Hands a cluster's auto-pause interval to its container: SecondsUntilAutoPause for an Aurora+ * cluster whose MinCapacity is 0, otherwise none, which keeps the container running.+ */+ private void applyAutoPause(DbCluster cluster) {+ if (config.services().rds().mock() || cluster.getDbClusterArn() == null) {+ return;+ }+ Integer secondsUntilAutoPause = isAuroraEngine(cluster.getEngineIdentifier())+ ? cluster.getServerlessV2SecondsUntilAutoPause()+ : null;+ containerManager.configureAutoPause(cluster.getDbClusterArn(), secondsUntilAutoPause,+ new ClusterAutoPause(accountIdFromArn(cluster.getDbClusterArn()),+ regionFromArn(cluster.getDbClusterArn()), cluster.getDbClusterIdentifier()));+ }++ /**+ * Aurora's conditions for pausing an idle cluster besides its zero MinCapacity: the cluster+ * and its instances are available, every instance is Aurora Serverless v2 (a provisioned one+ * keeps the writer awake), and the cluster is neither in a global database nor the target of+ * an RDS Proxy, which holds connections open to it.+ */+ private boolean clusterMayAutoPause(String accountId, String region, String clusterId) {+ DbCluster cluster = findClusterForScope(accountId, region, clusterId);+ if (cluster == null+ || (cluster.getStatus() != null && cluster.getStatus() != DbInstanceStatus.AVAILABLE)+ || cluster.getServerlessV2SecondsUntilAutoPause() == null+ || cluster.getGlobalClusterIdentifier() != null) {+ return false;+ }+ List<String> memberIds = List.copyOf(cluster.getDbClusterMembers());+ if (memberIds.isEmpty()) {+ return false;+ }+ for (String memberId : memberIds) {+ DbInstance member = findInstanceForScope(accountId, region, memberId);+ if (member == null+ || (member.getStatus() != null && member.getStatus() != DbInstanceStatus.AVAILABLE)+ || !SERVERLESS_V2_INSTANCE_CLASS.equalsIgnoreCase(member.getDbInstanceClass())) {+ return false;+ }+ }+ return !isDbProxyTargetOfAccount(accountId, region, clusterId, memberIds);+ }++ /** {@link #isRegisteredProxyTarget} for an explicit account, since auto-pause runs outside a request. */+ private boolean isDbProxyTargetOfAccount(String accountId, String region, String clusterId,+ List<String> memberIds) {+ List<DbProxyTargetGroup> targetGroups =+ proxyTargetGroups instanceof AccountAwareStorageBackend<DbProxyTargetGroup> aware+ ? aware.scanForAccount(accountId, key -> true)+ : proxyTargetGroups.scan(key -> true);+ return targetGroups.stream()+ .filter(targetGroup -> targetGroupBelongsTo(targetGroup, accountId, region))+ .flatMap(targetGroup -> targetGroup.getTargets().stream())+ .anyMatch(target -> ("TRACKED_CLUSTER".equals(target.getType())+ && clusterId.equals(target.getRdsResourceId()))+ || ("RDS_INSTANCE".equals(target.getType())+ && memberIds.contains(target.getRdsResourceId())));+ }++ private List<DbInstance> serverlessMembers(String accountId, String region, String clusterId) {+ DbCluster cluster = findClusterForScope(accountId, region, clusterId);+ if (cluster == null) {+ return List.of();+ }+ List<DbInstance> members = new ArrayList<>();+ for (String memberId : List.copyOf(cluster.getDbClusterMembers())) {+ DbInstance member = findInstanceForScope(accountId, region, memberId);+ if (member != null && SERVERLESS_V2_INSTANCE_CLASS.equalsIgnoreCase(member.getDbInstanceClass())) {+ members.add(member);+ }+ }+ return members;+ }++ /**+ * Records an auto-pause step as the RDS-EVENT-0370 to 0374 event of each Serverless v2 instance,+ * under the cluster's own account, since auto-pause runs outside a request.+ */+ private void recordAutoPauseEvents(String accountId, List<DbInstance> members,+ AutoPauseListener.Event event, Instant at) {+ String message = switch (event) {+ case PAUSE_INITIATED -> "Initiated pause for the DB instance.";+ case PAUSE_CANCELED -> "Pause was canceled for the DB instance.";+ case PAUSED -> "Successfully paused the DB instance.";+ case RESUME_INITIATED -> "Initiated resume for the DB instance.";+ case RESUMED -> "Successfully resumed the DB instance.";+ // Aurora emits no event while an instance stays paused; that only shows in metrics.+ case STILL_PAUSED -> null;+ };+ if (message == null) {+ return;+ }+ for (DbInstance member : members) {+ String eventId = "auto-pause:" + member.getDbInstanceArn() + ":" + event + ":" + at;+ putEventForAccount(accountId, eventId, new RdsEvent(eventId, member.getDbInstanceIdentifier(),+ "db-instance", message, AUTO_PAUSE_EVENT_CATEGORIES, at, member.getDbInstanceArn()));+ }+ }++ private void putEventForAccount(String accountId, String eventId, RdsEvent event) {+ if (events instanceof AccountAwareStorageBackend<RdsEvent> aware) {+ aware.putForAccount(accountId, eventId, event);+ } else {+ events.put(eventId, event);+ }+ }++ /**+ * Publishes what a paused Aurora Serverless v2 instance still sends CloudWatch: a zero+ * ServerlessDatabaseCapacity, ACUUtilization and CPUUtilization, for the cluster and for each+ * instance.+ */+ private void publishPausedCapacity(String accountId, String region, String clusterId,+ List<DbInstance> members, Instant at) {+ if (metricsService == null) {+ return;+ }+ publishZeroCapacity(accountId, region, new Dimension("DBClusterIdentifier", clusterId), at);+ for (DbInstance member : members) {+ publishZeroCapacity(accountId, region,+ new Dimension("DBInstanceIdentifier", member.getDbInstanceIdentifier()), at);+ }+ }++ private void publishZeroCapacity(String accountId, String region, Dimension dimension, Instant at) {+ for (String metricName : List.of("ServerlessDatabaseCapacity", "ACUUtilization", "CPUUtilization")) {+ MetricDatum datum = new MetricDatum();+ datum.setMetricName(metricName);+ datum.setUnit("ServerlessDatabaseCapacity".equals(metricName) ? "Count" : "Percent");+ datum.setDimensions(List.of(dimension));+ datum.setTimestamp(at.getEpochSecond());+ datum.setValue(0.0);+ metricsService.publishMetricForAccount(accountId, "AWS/RDS", datum, region,+ "rds-auto-pause:" + dimension.name() + "=" + dimension.value() + ":" + metricName + ":" + at);+ }+ }++ /** The auto-pause of one cluster's container: Aurora's pause rules, its events and its metrics. */+ private final class ClusterAutoPause implements AutoPauseListener {++ private final String accountId;+ private final String region;+ private final String clusterId;++ private ClusterAutoPause(String accountId, String region, String clusterId) {+ this.accountId = accountId;+ this.region = region;+ this.clusterId = clusterId;+ }++ @Override+ public boolean mayPause() {+ return clusterMayAutoPause(accountId, region, clusterId);+ }++ @Override+ public void onAutoPause(Event event, Instant at) {+ List<DbInstance> members = serverlessMembers(accountId, region, clusterId);+ recordAutoPauseEvents(accountId, members, event, at);+ if (event == Event.PAUSED || event == Event.STILL_PAUSED) {+ publishPausedCapacity(accountId, region, clusterId, members, at);+ }+ }+ }+/*** Whether RDS holds a cluster or instance under this identifier.*@@ -3700,6 +3884,9 @@ public class RdsService implements Resettable, ResourceProvider {cluster.setServerlessV2SecondsUntilAutoPause(effectiveAutoPauseSeconds);}putClusterForScope(currentAccountId(), effectiveRegion, id, cluster);+ if (modifiesServerlessV2Scaling) {+ applyAutoPause(cluster);+ }// A cluster rotation applies to every endpoint: the cluster's own proxy and each member// instance's proxy hold start-time password snapshots, and member endpoints validate@@ -6838,6 +7025,7 @@ public class RdsService implements Resettable, ResourceProvider {cluster.setStatus(DbInstanceStatus.AVAILABLE);putClusterForScope(accountId, clusterRegion,cluster.getDbClusterIdentifier(), cluster);+ applyAutoPause(cluster);} catch (Exception e) {if (!config.services().rds().mock()) {try {diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseController.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseController.javanew file mode 100644index 0000000000..02a5dc1c10--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseController.java@@ -0,0 +1,493 @@+package io.github.hectorvent.floci.services.rds.container;++import org.jboss.logging.Logger;++import java.time.Clock;+import java.time.Duration;+import java.time.Instant;+import java.util.ArrayList;+import java.util.List;+import java.util.Objects;+import java.util.concurrent.Executor;+import java.util.concurrent.Future;+import java.util.concurrent.RejectedExecutionException;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.locks.Condition;+import java.util.concurrent.locks.ReentrantLock;+import java.util.function.LongConsumer;++/**+ * Aurora Serverless v2 auto-pause for one RDS backend container.+ *+ * <p>Every client of the container, a proxied connection, a Data API call or a command Floci runs+ * inside it, holds a {@link RdsBackendGate.Lease} while it uses the container. Once no lease has+ * been held for {@code SecondsUntilAutoPause}, the container is frozen with {@code docker pause},+ * which keeps its data and its published port. The next lease request thaws it and waits until+ * it runs again.+ *+ * <p>The state moves from RUNNING to PAUSING, PAUSED and RESUMING, and back to RUNNING. Every+ * transition happens under {@link #lock}; the Docker calls and the listener run without it, so a+ * slow daemon or a listener waiting for the RDS service never holds up a thread that needs the+ * lock. A lease request that finds the container PAUSING cancels the pause, and one that finds it+ * RESUMING waits for the resume, so no client reaches a frozen container and none is turned away.+ */+final class AutoPauseController {++ private static final Logger LOG = Logger.getLogger(AutoPauseController.class);++ /** How often a paused container reports zero capacity, as a paused Aurora instance does. */+ private static final Duration PAUSED_REPORT_INTERVAL = Duration.ofMinutes(1);++ enum State {+ RUNNING,+ PAUSING,+ PAUSED,+ RESUMING+ }++ /** Runs a task after a delay; cancelling the returned future stops a task that has not started. */+ @FunctionalInterface+ interface Scheduler {+ Future<?> schedule(Runnable task, Duration delay);+ }++ /** The Docker freezer. Both calls succeed when the container is already in the requested state. */+ interface Freezer {+ void pause(String containerId);++ void unpause(String containerId);+ }++ private final String containerId;+ private final String host;+ private final int port;+ private final Freezer freezer;+ private final Scheduler scheduler;+ private final Executor notifier;+ private final Clock clock;+ private final Duration resumeDelay;++ private final ReentrantLock lock = new ReentrantLock();+ private final Condition settled = lock.newCondition();++ // Everything below is guarded by lock.+ private final List<Signal> pendingSignals = new ArrayList<>();+ private State state = State.RUNNING;+ private int leases;+ private Integer secondsUntilAutoPause;+ private AutoPauseListener listener = AutoPauseListener.NONE;+ // The one timer of the current state: the idle timer while RUNNING, the zero-capacity report+ // while PAUSED, the resume delay while RESUMING. A task whose token is not the current one+ // was cancelled or belongs to an earlier state, and does nothing.+ private Future<?> timer;+ private long timerToken;+ private boolean pauseCanceled;+ private boolean closed;+ private Instant lastSignal = Instant.EPOCH;++ AutoPauseController(String containerId, String host, int port, Freezer freezer,+ Scheduler scheduler, Executor notifier, Clock clock, Duration resumeDelay) {+ this.containerId = containerId;+ this.host = host;+ this.port = port;+ this.freezer = freezer;+ this.scheduler = scheduler;+ this.notifier = notifier;+ this.clock = clock;+ this.resumeDelay = resumeDelay;+ }++ boolean serves(String backendHost, int backendPort) {+ return port == backendPort && Objects.equals(host, backendHost);+ }++ boolean runs(String candidateContainerId) {+ return containerId.equals(candidateContainerId);+ }++ State state() {+ lock.lock();+ try {+ return state;+ } finally {+ lock.unlock();+ }+ }++ int leaseCount() {+ lock.lock();+ try {+ return leases;+ } finally {+ lock.unlock();+ }+ }++ /**+ * Sets the idle interval after which the container pauses; null keeps it running. The idle+ * clock restarts when the interval changes, and a container paused or pausing under the old+ * interval is resumed, as Aurora resumes an instance whose scaling range changes.+ */+ void configure(Integer seconds, AutoPauseListener newListener) {+ lock.lock();+ try {+ if (closed) {+ return;+ }+ listener = newListener != null ? newListener : AutoPauseListener.NONE;+ if (Objects.equals(secondsUntilAutoPause, seconds)) {+ return;+ }+ secondsUntilAutoPause = seconds;+ switch (state) {+ case RUNNING -> {+ cancelTimer();+ armIdleTimer();+ }+ case PAUSING -> pauseCanceled = true;+ case PAUSED -> startTimer(this::resumeInBackground, Duration.ZERO);+ case RESUMING -> {+ // Finishing the resume arms the idle timer with the new interval.+ }+ }+ } finally {+ unlockAndDispatch();+ }+ }++ /**+ * Returns a lease that keeps the container awake, resuming it first when it is paused. Waits+ * while a pause or a resume is under way; a pause still in progress is cancelled.+ *+ * @throws InterruptedException when interrupted while waiting+ * @throws IllegalStateException when Docker cannot resume the container, which stays paused+ */+ RdsBackendGate.Lease acquire() throws InterruptedException {+ lock.lock();+ try {+ while (true) {+ if (closed && state != State.PAUSING && state != State.RESUMING) {+ return RdsBackendGate.Lease.NONE;+ }+ if (state == State.RUNNING) {+ leases++;+ cancelTimer();+ return new ControllerLease();+ }+ if (state == State.PAUSED) {+ resume();+ } else {+ if (state == State.PAUSING) {+ pauseCanceled = true;+ }+ settled.await();+ }+ }+ } finally {+ unlockAndDispatch();+ }+ }++ /**+ * Stops auto-pausing for good, thawing the container when it is paused. A pause or a resume+ * in progress is not waited for: the thread running it thaws the container once it sees the+ * controller closed.+ */+ void close() {+ lock.lock();+ try {+ if (closed) {+ return;+ }+ closed = true;+ // A resume whose timer is set has already thawed the container and only waits out+ // the resume delay; cancelling that timer must not leave its waiters behind.+ boolean resumeDelayPending = state == State.RESUMING && timer != null;+ cancelTimer();+ if (resumeDelayPending) {+ finishResume();+ } else if (state == State.PAUSED) {+ state = State.RESUMING;+ RuntimeException failure = withoutLock(() -> freezer.unpause(containerId));+ if (failure != null) {+ LOG.warnv(failure, "Could not resume auto-paused RDS container {0} before releasing it",+ containerId);+ state = State.PAUSED;+ } else {+ state = State.RUNNING;+ }+ }+ settled.signalAll();+ } finally {+ unlockAndDispatch();+ }+ }++ /**+ * Runs the idle check now rather than when the idle timer fires, for a caller that cannot wait+ * SecondsUntilAutoPause. Returns whether the container paused.+ */+ boolean pauseIfIdle() {+ long token;+ lock.lock();+ try {+ if (!idleTimerArmed(timerToken)) {+ return false;+ }+ token = timerToken;+ } finally {+ unlockAndDispatch();+ }+ idleTimerFired(token);+ return state() == State.PAUSED;+ }++ private void release() {+ lock.lock();+ try {+ if (leases > 0) {+ leases--;+ }+ armIdleTimer();+ } finally {+ unlockAndDispatch();+ }+ }++ private void idleTimerFired(long token) {+ AutoPauseListener current;+ lock.lock();+ try {+ if (!idleTimerArmed(token)) {+ return;+ }+ current = listener;+ } finally {+ unlockAndDispatch();+ }+ // Asked without the lock: the answer comes from the RDS service, which may be busy.+ boolean allowed = mayPause(current);+ lock.lock();+ try {+ if (!idleTimerArmed(token)) {+ return;+ }+ if (!allowed) {+ armIdleTimer();+ return;+ }+ cancelTimer();+ state = State.PAUSING;+ pauseCanceled = false;+ signal(AutoPauseListener.Event.PAUSE_INITIATED);+ pause();+ } finally {+ unlockAndDispatch();+ }+ }++ private boolean idleTimerArmed(long token) {+ return !closed && state == State.RUNNING && leases == 0 && secondsUntilAutoPause != null+ && timer != null && token == timerToken;+ }++ /** Freezes the container. Entered and left holding the lock, in PAUSING. */+ private void pause() {+ RuntimeException pauseFailure = withoutLock(() -> freezer.pause(containerId));+ if (pauseFailure == null && !pauseCanceled && !closed) {+ state = State.PAUSED;+ signal(AutoPauseListener.Event.PAUSED);+ armPausedReport();+ settled.signalAll();+ LOG.infov("Auto-paused idle RDS container {0} (SecondsUntilAutoPause {1})",+ containerId, String.valueOf(secondsUntilAutoPause));+ return;+ }+ if (pauseFailure != null) {+ LOG.warnv(pauseFailure, "Could not auto-pause RDS container {0}; it keeps running", containerId);+ } else {+ // A client arrived, the interval changed or the container is being released while it+ // was freezing: thaw it again rather than let anyone reach it frozen.+ RuntimeException thawFailure = withoutLock(() -> freezer.unpause(containerId));+ if (thawFailure != null) {+ LOG.warnv(thawFailure, "Could not undo the auto-pause of RDS container {0}", containerId);+ state = State.PAUSED;+ armPausedReport();+ settled.signalAll();+ return;+ }+ }+ state = State.RUNNING;+ signal(AutoPauseListener.Event.PAUSE_CANCELED);+ settled.signalAll();+ armIdleTimer();+ }++ /**+ * Thaws the container. Entered and left holding the lock, in PAUSED. Waiters see RESUMING+ * until the resume delay has passed.+ */+ private void resume() {+ cancelTimer();+ state = State.RESUMING;+ signal(AutoPauseListener.Event.RESUME_INITIATED);+ RuntimeException failure = withoutLock(() -> freezer.unpause(containerId));+ if (failure != null) {+ state = State.PAUSED;+ armPausedReport();+ settled.signalAll();+ throw new IllegalStateException(+ "Could not resume auto-paused RDS container " + containerId, failure);+ }+ if (closed || resumeDelay.isZero()) {+ finishResume();+ } else {+ startTimer(this::resumeDelayElapsed, resumeDelay);+ }+ }++ private void resumeDelayElapsed(long token) {+ lock.lock();+ try {+ if (state == State.RESUMING && token == timerToken) {+ finishResume();+ }+ } finally {+ unlockAndDispatch();+ }+ }++ private void finishResume() {+ cancelTimer();+ state = State.RUNNING;+ signal(AutoPauseListener.Event.RESUMED);+ settled.signalAll();+ armIdleTimer();+ LOG.infov("Resumed auto-paused RDS container {0}", containerId);+ }++ private void resumeInBackground(long token) {+ lock.lock();+ try {+ if (!closed && state == State.PAUSED && token == timerToken) {+ resume();+ }+ } catch (IllegalStateException e) {+ LOG.warnv(e, "Could not resume auto-paused RDS container {0}", containerId);+ } finally {+ unlockAndDispatch();+ }+ }++ private void armIdleTimer() {+ if (closed || state != State.RUNNING || leases > 0 || secondsUntilAutoPause == null) {+ return;+ }+ startTimer(this::idleTimerFired, Duration.ofSeconds(secondsUntilAutoPause));+ }++ private void armPausedReport() {+ if (!closed) {+ startTimer(this::pausedReportDue, PAUSED_REPORT_INTERVAL);+ }+ }++ private void pausedReportDue(long token) {+ lock.lock();+ try {+ if (!closed && state == State.PAUSED && token == timerToken) {+ signal(AutoPauseListener.Event.STILL_PAUSED);+ armPausedReport();+ }+ } finally {+ unlockAndDispatch();+ }+ }++ private void startTimer(LongConsumer task, Duration delay) {+ cancelTimer();+ long token = timerToken;+ timer = scheduler.schedule(() -> task.accept(token), delay);+ }++ private void cancelTimer() {+ if (timer != null) {+ timer.cancel(false);+ timer = null;+ }+ timerToken++;+ }++ /** Runs a Docker call with the lock released, and returns its failure instead of throwing it. */+ private RuntimeException withoutLock(Runnable dockerCall) {+ if (lock.getHoldCount() != 1) {+ throw new IllegalStateException("Auto-pause lock must be held exactly once around a Docker call");+ }+ unlockAndDispatch();+ try {+ dockerCall.run();+ return null;+ } catch (RuntimeException e) {+ return e;+ } finally {+ lock.lock();+ }+ }++ private boolean mayPause(AutoPauseListener current) {+ try {+ return current.mayPause();+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not tell whether RDS container {0} may auto-pause; it keeps running",+ containerId);+ return false;+ }+ }++ /** Queues an event for the listener, stamped later than the previous one so they keep their order. */+ private void signal(AutoPauseListener.Event event) {+ Instant now = clock.instant();+ Instant at = now.isAfter(lastSignal) ? now : lastSignal.plusMillis(1);+ lastSignal = at;+ pendingSignals.add(new Signal(listener, event, at));+ }++ /** Releases the lock, then hands the events queued under it to the notifier, in order. */+ private void unlockAndDispatch() {+ List<Signal> ready = pendingSignals.isEmpty() ? List.of() : List.copyOf(pendingSignals);+ pendingSignals.clear();+ lock.unlock();+ for (Signal signal : ready) {+ try {+ notifier.execute(() -> deliver(signal));+ } catch (RejectedExecutionException e) {+ LOG.debugv("Dropped RDS auto-pause event {0} for container {1}: {2}",+ signal.event(), containerId, e.getMessage());+ }+ }+ }++ private void deliver(Signal signal) {+ try {+ signal.listener().onAutoPause(signal.event(), signal.at());+ } catch (RuntimeException e) {+ LOG.warnv(e, "Could not report RDS auto-pause event {0} for container {1}",+ signal.event(), containerId);+ }+ }++ private record Signal(AutoPauseListener listener, AutoPauseListener.Event event, Instant at) {+ }++ private final class ControllerLease implements RdsBackendGate.Lease {++ private final AtomicBoolean released = new AtomicBoolean();++ @Override+ public void close() {+ if (released.compareAndSet(false, true)) {+ release();+ }+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseListener.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseListener.javanew file mode 100644index 0000000000..e6eb17e2ca--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/AutoPauseListener.java@@ -0,0 +1,40 @@+package io.github.hectorvent.floci.services.rds.container;++import java.time.Instant;++/**+ * What the auto-pause of an RDS container asks of the cluster it belongs to, and what it reports+ * back. Both calls run on a background thread, never while the auto-pause holds its lock, so an+ * implementation may take the RDS service's lock.+ */+public interface AutoPauseListener {++ AutoPauseListener NONE = new AutoPauseListener() {+ @Override+ public boolean mayPause() {+ return false;+ }++ @Override+ public void onAutoPause(Event event, Instant at) {+ }+ };++ /**+ * The auto-pause steps Aurora reports as RDS-EVENT-0370 to 0374, plus {@code STILL_PAUSED},+ * repeated once a minute while the container stays paused.+ */+ enum Event {+ PAUSE_INITIATED,+ PAUSE_CANCELED,+ PAUSED,+ STILL_PAUSED,+ RESUME_INITIATED,+ RESUMED+ }++ /** Whether the cluster may pause now that it has been idle for SecondsUntilAutoPause. */+ boolean mayPause();++ void onAutoPause(Event event, Instant at);+}diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsBackendGate.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsBackendGate.javanew file mode 100644index 0000000000..2104e52191--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsBackendGate.java@@ -0,0 +1,31 @@+package io.github.hectorvent.floci.services.rds.container;++/**+ * Lets a client into an RDS backend container. An Aurora Serverless v2 cluster that auto-paused is+ * resumed first, and it does not pause again while the client holds the returned lease.+ */+@FunctionalInterface+public interface RdsBackendGate {++ /** A gate for backends that never pause. */+ RdsBackendGate OPEN = (host, port) -> Lease.NONE;++ /**+ * Waits until the container serving {@code host:port} can take a client, resuming it when it+ * is auto-paused, and keeps it awake until the lease is closed. A backend with no auto-pause+ * gets {@link Lease#NONE}.+ *+ * @throws InterruptedException when interrupted while a pause or resume is under way+ */+ Lease enter(String host, int port) throws InterruptedException;++ /** Held for as long as a client uses the backend; closing it more than once is harmless. */+ interface Lease extends AutoCloseable {++ Lease NONE = () -> {+ };++ @Override+ void close();+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.javaindex 967fd80fc0..99518b223c 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManager.java@@ -1,10 +1,12 @@package io.github.hectorvent.floci.services.rds.container;import com.github.dockerjava.api.async.ResultCallback;+import com.github.dockerjava.api.command.InspectContainerResponse;import com.github.dockerjava.api.model.Frame;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsArnUtils;import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.Resettable;import io.github.hectorvent.floci.core.common.ServiceConfigAccess;import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;import io.github.hectorvent.floci.core.common.docker.ContainerDetector;@@ -15,6 +17,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.Eimport io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;import io.github.hectorvent.floci.core.common.docker.ContainerSpec;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;+import jakarta.annotation.PreDestroy;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.apache.commons.compress.archivers.tar.TarArchiveEntry;@@ -27,22 +30,32 @@ import java.io.Closeable;import java.io.IOException;import java.nio.file.Path;import java.nio.charset.StandardCharsets;+import java.time.Clock;+import java.time.Duration;import java.util.ArrayList;import java.util.List;import java.util.Map;import java.util.Optional;import java.util.Set;+import java.util.concurrent.CompletableFuture;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.CountDownLatch;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.RejectedExecutionException;+import java.util.concurrent.ScheduledExecutorService;import java.util.concurrent.TimeUnit;import java.util.regex.Pattern;/*** Manages backend Docker container lifecycle for RDS DB instances and clusters.* Starts postgres/mysql/mariadb containers and resolves the backend host:port for the auth proxy.+ * Every client of a container enters it through {@link #enter}, which is how an Aurora Serverless+ * v2 cluster with MinCapacity 0 knows when it is idle enough to pause and when to resume.*/@ApplicationScoped-public class RdsContainerManager {+public class RdsContainerManager implements RdsBackendGate, Resettable {private static final Logger LOG = Logger.getLogger(RdsContainerManager.class);private static final Pattern SAFE_STORAGE_COMPONENT = Pattern.compile("[A-Za-z0-9._-]+");@@ -59,6 +72,29 @@ public class RdsContainerManager {private final Set<String> claimedRuntimes = ConcurrentHashMap.newKeySet();private final Set<String> cleanupPending = ConcurrentHashMap.newKeySet();private volatile boolean dockerUnavailableLogged;+ // Auto-pause state of every running container, by runtime id. The timer thread only keeps+ // time: each task it fires runs on a virtual thread of its own, since a pause first asks the+ // RDS service whether the cluster may pause. Events reach the listener one at a time, in order.+ private final Map<String, AutoPauseController> autoPauses = new ConcurrentHashMap<>();+ private final ScheduledExecutorService autoPauseTimer = Executors.newSingleThreadScheduledExecutor(+ runnable -> {+ Thread thread = new Thread(runnable, "rds-auto-pause-timer");+ thread.setDaemon(true);+ return thread;+ });+ private final ExecutorService autoPauseEvents = Executors.newSingleThreadExecutor(+ Thread.ofVirtual().name("rds-auto-pause-events").factory());+ private final AutoPauseController.Freezer dockerFreezer = new AutoPauseController.Freezer() {+ @Override+ public void pause(String containerId) {+ pauseContainer(containerId);+ }++ @Override+ public void unpause(String containerId) {+ unpauseContainer(containerId);+ }+ };@Injectpublic RdsContainerManager(ContainerBuilder containerBuilder,@@ -281,6 +317,9 @@ public class RdsContainerManager {info.containerId(), logGroup, logStream, runtimeIdentity.region(),"rds:" + effectiveRuntimeId);handle.setLogStream(logHandle);+ // Before the handle is returned, so that the connections of every proxy started on it+ // are counted from the first one.+ registerAutoPause(effectiveRuntimeId, handle);activeContainers.put(effectiveRuntimeId, handle);return handle;@@ -349,6 +388,7 @@ public class RdsContainerManager {}RdsContainerHandle active = activeContainers.get(handle.getRuntimeId());RdsContainerHandle effectiveHandle = active != null ? active : handle;+ releaseAutoPause(effectiveHandle.getRuntimeId());try {lifecycleManager.stopAndRemoveStrict(effectiveHandle.getContainerId(), effectiveHandle.getLogStream());@@ -411,6 +451,156 @@ public class RdsContainerManager {}}+ /**+ * Applies an Aurora Serverless v2 auto-pause interval to a running container: it pauses once+ * no client has used it for that many seconds, and a null interval, for MinCapacity above 0+ * or with auto-pause turned off, keeps it running. The listener decides whether the cluster+ * may pause at that moment and is told each step.+ */+ public void configureAutoPause(String runtimeId, Integer secondsUntilAutoPause,+ AutoPauseListener listener) {+ AutoPauseController controller = runtimeId != null ? autoPauses.get(runtimeId) : null;+ if (controller == null) {+ return;+ }+ boolean enabled = config.services().rds().auroraAutoPauseEnabled();+ controller.configure(enabled ? secondsUntilAutoPause : null, listener);+ }++ @Override+ public Lease enter(String host, int port) throws InterruptedException {+ if (host == null) {+ return Lease.NONE;+ }+ for (AutoPauseController controller : autoPauses.values()) {+ if (controller.serves(host, port)) {+ return controller.acquire();+ }+ }+ return Lease.NONE;+ }++ /**+ * Runs a runtime's auto-pause idle check now instead of after SecondsUntilAutoPause, which+ * the API does not let drop below 300 seconds. Returns whether the container paused.+ */+ boolean pauseIfIdle(String runtimeId) {+ AutoPauseController controller = autoPauses.get(runtimeId);+ return controller != null && controller.pauseIfIdle();+ }++ /**+ * A reset wipes the RDS records but leaves their containers running, as it always has. It+ * releases their auto-pause, which thaws a paused container, so none stays frozen with no+ * cluster left to resume it.+ */+ @Override+ public void clear() {+ for (String runtimeId : List.copyOf(autoPauses.keySet())) {+ releaseAutoPause(runtimeId);+ }+ }++ @PreDestroy+ void shutdownAutoPause() {+ autoPauseTimer.shutdownNow();+ autoPauseEvents.shutdownNow();+ }++ private void registerAutoPause(String runtimeId, RdsContainerHandle handle) {+ Duration resumeDelay = Duration.ofMillis(+ Math.max(0, config.services().rds().auroraResumeDelayMillis()));+ AutoPauseController controller = new AutoPauseController(+ handle.getContainerId(), handle.getHost(), handle.getPort(), dockerFreezer,+ this::scheduleAutoPauseTask, autoPauseEvents, Clock.systemUTC(), resumeDelay);+ // Clients find a container by its address, so the new container owns it from now on,+ // even if Docker removed the previous owner without Floci stopping it.+ for (Map.Entry<String, AutoPauseController> entry : List.copyOf(autoPauses.entrySet())) {+ if (entry.getValue().serves(handle.getHost(), handle.getPort())+ && autoPauses.remove(entry.getKey(), entry.getValue())) {+ entry.getValue().close();+ }+ }+ AutoPauseController previous = autoPauses.put(runtimeId, controller);+ if (previous != null) {+ previous.close();+ }+ }++ /** Stops a runtime's auto-pause before its container goes away, thawing it when paused. */+ private void releaseAutoPause(String runtimeId) {+ AutoPauseController controller = autoPauses.remove(runtimeId);+ if (controller != null) {+ controller.close();+ }+ }++ /** Keeps a container awake, resuming it first, while Floci runs a command inside it. */+ private Lease holdContainer(String containerId) {+ if (containerId == null) {+ return Lease.NONE;+ }+ for (AutoPauseController controller : autoPauses.values()) {+ if (controller.runs(containerId)) {+ try {+ return controller.acquire();+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ throw new IllegalStateException(+ "Interrupted while resuming RDS container " + containerId, e);+ }+ }+ }+ return Lease.NONE;+ }++ private Future<?> scheduleAutoPauseTask(Runnable task, Duration delay) {+ try {+ return autoPauseTimer.schedule(+ () -> Thread.ofVirtual().name("rds-auto-pause").start(task),+ delay.toMillis(), TimeUnit.MILLISECONDS);+ } catch (RejectedExecutionException e) {+ LOG.debugv("RDS auto-pause timer is shut down; not scheduling: {0}", e.getMessage());+ return CompletableFuture.completedFuture(null);+ }+ }++ private void pauseContainer(String containerId) {+ try {+ lifecycleManager.getDockerClient().pauseContainerCmd(containerId).exec();+ } catch (RuntimeException e) {+ // Docker refuses to pause a paused container; that one is where it should be.+ if (!isRunning(containerId, true)) {+ throw e;+ }+ }+ }++ private void unpauseContainer(String containerId) {+ try {+ lifecycleManager.getDockerClient().unpauseContainerCmd(containerId).exec();+ } catch (RuntimeException e) {+ // Docker refuses to unpause a container that is not paused, as after a manual+ // `docker unpause`; one still running is where it should be.+ if (!isRunning(containerId, false)) {+ throw e;+ }+ }+ }++ /** Whether the container runs, paused or not as asked; false when Docker cannot say. */+ private boolean isRunning(String containerId, boolean paused) {+ try {+ InspectContainerResponse.ContainerState state =+ lifecycleManager.getDockerClient().inspectContainerCmd(containerId).exec().getState();+ return state != null && Boolean.TRUE.equals(state.getRunning())+ && paused == Boolean.TRUE.equals(state.getPaused());+ } catch (RuntimeException e) {+ LOG.debugv("Could not inspect RDS container {0}: {1}", containerId, e.getMessage());+ return false;+ }+ }+private void addPersistenceMounts(ContainerBuilder.Builder specBuilder, String storageResourceId,String exactVolumeName, DatabaseEngine engine, String image) {@@ -555,9 +745,11 @@ public class RdsContainerManager {*/public void rotateMasterPassword(String containerName, String containerId, DatabaseEngine engine,String masterUsername, String oldPassword, String newPassword) {- execUntilSuccess(containerName, containerId,- passwordRotationCommand(engine, masterUsername, oldPassword, newPassword),- "master-password rotation");+ try (Lease _ = holdContainer(containerId)) {+ execUntilSuccess(containerName, containerId,+ passwordRotationCommand(engine, masterUsername, oldPassword, newPassword),+ "master-password rotation");+ }}static String[] passwordRotationCommand(DatabaseEngine engine, String masterUsername,@@ -679,7 +871,7 @@ public class RdsContainerManager {"pg_dumpall","-U", effectiveUser};- try {+ try (Lease _ = holdContainer(containerId)) {ContainerExecResult result = execInContainer(containerId, cmd, 120);if (result.exitCode() != 0) {throw new RuntimeException("pg_dumpall failed with exit code " + result.exitCode() + ": " + result.stderr());@@ -693,7 +885,7 @@ public class RdsContainerManager {public void restorePostgresSnapshot(String containerId, String masterUsername, String sqlDump) {String effectiveUser = (masterUsername != null && !masterUsername.isBlank()) ? masterUsername : "postgres";- try {+ try (Lease _ = holdContainer(containerId)) {String restoreScript = postgresRestoreScript();java.io.ByteArrayOutputStream bos = new java.io.ByteArrayOutputStream();diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.javaindex c50cd9b866..dbe302b37e 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxy.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.services.rds.proxy;+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import org.jboss.logging.Logger;@@ -34,6 +35,7 @@ public class RdsAuthProxy {private final int handshakeTimeoutMillis;private final int backendConnectTimeoutMillis;private final Semaphore connectionPermits;+ private final RdsBackendGate backendGate;private volatile boolean running;private ServerSocket serverSocket;@@ -47,7 +49,8 @@ public class RdsAuthProxy {int maxConnections) {this(instanceId, backendHost, backendPort, engine, iamEnabled, masterUsername, masterPassword,dbName, sigV4, tlsCertificates, passwordValidator, handshakeTimeoutMillis,- backendConnectTimeoutMillis, maxConnections, null, username -> false);+ backendConnectTimeoutMillis, maxConnections, null, username -> false,+ RdsBackendGate.OPEN);}public RdsAuthProxy(String instanceId, String backendHost, int backendPort,@@ -56,11 +59,11 @@ public class RdsAuthProxy {RdsSigV4Validator sigV4, RdsProxyTlsCertificates tlsCertificates,MasterPasswordCheck passwordValidator,int handshakeTimeoutMillis, int backendConnectTimeoutMillis,- int maxConnections, RdsProxyBinding binding) {+ int maxConnections, RdsProxyBinding binding, RdsBackendGate backendGate) {this(instanceId, backendHost, backendPort, engine, iamEnabled, masterUsername, masterPassword,dbName, sigV4, tlsCertificates, passwordValidator, handshakeTimeoutMillis,backendConnectTimeoutMillis, maxConnections, binding,- username -> binding != null);+ username -> binding != null, backendGate);}public RdsAuthProxy(String instanceId, String backendHost, int backendPort,@@ -70,7 +73,8 @@ public class RdsAuthProxy {MasterPasswordCheck passwordValidator,int handshakeTimeoutMillis, int backendConnectTimeoutMillis,int maxConnections, RdsProxyBinding binding,- MySqlProtocolHandler.IamUserChecker iamUserChecker) {+ MySqlProtocolHandler.IamUserChecker iamUserChecker,+ RdsBackendGate backendGate) {this.instanceId = instanceId;this.backendHost = backendHost;this.backendPort = backendPort;@@ -87,6 +91,7 @@ public class RdsAuthProxy {this.handshakeTimeoutMillis = handshakeTimeoutMillis;this.backendConnectTimeoutMillis = backendConnectTimeoutMillis;this.connectionPermits = new Semaphore(Math.max(1, maxConnections));+ this.backendGate = backendGate;}public void start(int proxyPort) throws IOException {@@ -151,7 +156,11 @@ public class RdsAuthProxy {private void handleConnection(Socket client) {Socket backend = null;PostgresProtocolHandler.AuthenticatedSession session = null;+ RdsBackendGate.Lease lease = null;try {+ // An auto-paused Aurora backend resumes before the client is served: the client is+ // held, not refused, and the backend stays awake for as long as the connection lasts.+ lease = backendGate.enter(backendHost, backendPort);client.setTcpNoDelay(true);// RDS only proxy-validates the master user; a non-master user passes through so the@@ -196,6 +205,9 @@ public class RdsAuthProxy {TcpStreamBridge.relay(client, backend);}}+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ LOG.debugv("RDS connection for instance {0} interrupted while its backend resumed", instanceId);} catch (Exception e) {LOG.debugv("RDS connection error for instance {0}: {1}", instanceId, e.getMessage());} finally {@@ -209,6 +221,9 @@ public class RdsAuthProxy {if (session != null) {closeQuietly(session.backend());}+ if (lease != null) {+ lease.close();+ }}}diff --git a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.javaindex 0695c0ce3f..ce65954e1d 100644--- a/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/rds/proxy/RdsProxyManager.java@@ -1,6 +1,7 @@package io.github.hectorvent.floci.services.rds.proxy;import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;@@ -20,14 +21,21 @@ public class RdsProxyManager {private final RdsSigV4Validator sigV4Validator;private final RdsProxyTlsCertificates tlsCertificates;private final EmulatorConfig config;+ private final RdsBackendGate backendGate;private final ConcurrentHashMap<String, RdsAuthProxy> proxies = new ConcurrentHashMap<>();@Injectpublic RdsProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates,- EmulatorConfig config) {+ EmulatorConfig config, RdsBackendGate backendGate) {this.sigV4Validator = sigV4Validator;this.tlsCertificates = tlsCertificates;this.config = config;+ this.backendGate = backendGate;+ }++ RdsProxyManager(RdsSigV4Validator sigV4Validator, RdsProxyTlsCertificates tlsCertificates,+ EmulatorConfig config) {+ this(sigV4Validator, tlsCertificates, config, RdsBackendGate.OPEN);}public synchronized void startProxy(String instanceId, DatabaseEngine engine, boolean iamEnabled,@@ -42,7 +50,7 @@ public class RdsProxyManager {instanceId, backendHost, backendPort, engine, iamEnabled,masterUsername, masterPassword, dbName, sigV4Validator, tlsCertificates, passwordValidator,rdsConfig.proxyHandshakeTimeoutMillis(), rdsConfig.proxyBackendConnectTimeoutMillis(),- rdsConfig.proxyMaxConnections(), binding);+ rdsConfig.proxyMaxConnections(), binding, backendGate);try {proxy.start(proxyPort);} catch (IOException e) {diff --git a/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java b/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.javaindex 2f1e09a7b1..5f913afe31 100644--- a/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java+++ b/src/main/java/io/github/hectorvent/floci/services/rdsdata/RdsDataService.java@@ -8,6 +8,7 @@ import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.Resettable;import io.github.hectorvent.floci.core.common.SqlParameterParser.ParsedSql;+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;import io.github.hectorvent.floci.services.secretsmanager.model.SecretVersion;@@ -53,15 +54,17 @@ public class RdsDataService implements Resettable {private final Duration transactionTtl;private final ConcurrentMap<String, TransactionContext> transactions = new ConcurrentHashMap<>();private final ScheduledExecutorService transactionCleanupExecutor;+ private final RdsBackendGate backendGate;@Injectpublic RdsDataService(RdsDataResourceResolver resourceResolver,SecretsManagerService secretsManagerService,ObjectMapper objectMapper,RdsDataConnectionFactory connectionFactory,- EmulatorConfig config) {+ EmulatorConfig config,+ RdsBackendGate backendGate) {this(resourceResolver, secretsManagerService, objectMapper, connectionFactory,- Duration.ofSeconds(config.services().rdsData().transactionTtlSeconds()));+ Duration.ofSeconds(config.services().rdsData().transactionTtlSeconds()), backendGate);}RdsDataService(RdsDataResourceResolver resourceResolver,@@ -69,11 +72,22 @@ public class RdsDataService implements Resettable {ObjectMapper objectMapper,RdsDataConnectionFactory connectionFactory,Duration transactionTtl) {+ this(resourceResolver, secretsManagerService, objectMapper, connectionFactory, transactionTtl,+ RdsBackendGate.OPEN);+ }++ RdsDataService(RdsDataResourceResolver resourceResolver,+ SecretsManagerService secretsManagerService,+ ObjectMapper objectMapper,+ RdsDataConnectionFactory connectionFactory,+ Duration transactionTtl,+ RdsBackendGate backendGate) {this.resourceResolver = resourceResolver;this.secretsManagerService = secretsManagerService;this.objectMapper = objectMapper;this.connectionFactory = connectionFactory;this.transactionTtl = transactionTtl;+ this.backendGate = backendGate;this.transactionCleanupExecutor = Executors.newSingleThreadScheduledExecutor(r -> {Thread thread = new Thread(r, "rds-data-transaction-cleanup");thread.setDaemon(true);@@ -95,7 +109,7 @@ public class RdsDataService implements Resettable {synchronized (tx) {if (transactions.remove(id, tx)) {rollbackQuietly(tx.connection);- closeQuietly(tx.connection);+ tx.close();}}});@@ -106,7 +120,7 @@ public class RdsDataService implements Resettable {synchronized (tx) {if (transactions.remove(id, tx)) {rollbackQuietly(tx.connection);- closeQuietly(tx.connection);+ tx.close();}}});@@ -149,13 +163,18 @@ public class RdsDataService implements Resettable {Credentials credentials = credentials(request, target, region);String database = databaseName(request, target);+ // The transaction's connection keeps the cluster awake until it commits, rolls back or expires.+ RdsBackendGate.Lease lease = enterBackend(target);+ boolean leaseHandedOver = false;Connection connection = null;try {connection = connectionFactory.open(target, credentials.username(), credentials.password(), database);connection.setAutoCommit(false);String transactionId = UUID.randomUUID().toString();transactions.put(transactionId, new TransactionContext(- transactionId, connection, target.engine(), target.arn(), database, region, transactionTtl));+ transactionId, connection, target.engine(), target.arn(), database, region, transactionTtl,+ lease));+ leaseHandedOver = true;ObjectNode response = objectMapper.createObjectNode();response.put("transactionId", transactionId);@@ -165,6 +184,10 @@ public class RdsDataService implements Resettable {closeQuietly(connection);}throw databaseError(e);+ } finally {+ if (!leaseHandedOver) {+ lease.close();+ }}}@@ -184,7 +207,7 @@ public class RdsDataService implements Resettable {} catch (SQLException e) {throw databaseError(e);} finally {- closeQuietly(tx.connection);+ tx.close();}}ObjectNode response = objectMapper.createObjectNode();@@ -208,7 +231,7 @@ public class RdsDataService implements Resettable {} catch (SQLException e) {throw databaseError(e);} finally {- closeQuietly(tx.connection);+ tx.close();}}ObjectNode response = objectMapper.createObjectNode();@@ -236,11 +259,28 @@ public class RdsDataService implements Resettable {resourceResolver.resolve(resourceArn, region);Credentials credentials = credentials(request, target, region);String database = databaseName(request, target);- try (Connection connection = connectionFactory.open(target, credentials.username(), credentials.password(), database)) {+ try (RdsBackendGate.Lease _ = enterBackend(target);+ Connection connection = connectionFactory.open(target, credentials.username(), credentials.password(), database)) {return work.execute(connection, target.engine());}}+ /**+ * Resumes an auto-paused Aurora cluster before the Data API uses its database, as Aurora+ * does, and keeps it from pausing again until the returned lease is closed.+ */+ private RdsBackendGate.Lease enterBackend(RdsDataResourceResolver.DatabaseTarget target) {+ try {+ return backendGate.enter(target.host(), target.port());+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ throw new AwsException("InternalServerErrorException",+ "Interrupted while the DB cluster was resuming.", 500);+ } catch (IllegalStateException e) {+ throw new AwsException("InternalServerErrorException", e.getMessage(), 500);+ }+ }+private ObjectNode executeOnConnection(Connection connection, DatabaseEngine engine, String sql,Map<String, JsonNode> parameters, boolean includeMetadata)throws SQLException {@@ -583,7 +623,7 @@ public class RdsDataService implements Resettable {synchronized (tx) {if (tx.expiresAt.isBefore(now) && transactions.remove(id, tx)) {rollbackQuietly(tx.connection);- closeQuietly(tx.connection);+ tx.close();}}}@@ -737,23 +777,33 @@ public class RdsDataService implements Resettable {private final String resourceArn;private final String database;private final String region;+ private final RdsBackendGate.Lease lease;private volatile Instant expiresAt;private TransactionContext(String id, Connection connection, DatabaseEngine engine, String resourceArn,- String database, String region, Duration ttl) {+ String database, String region, Duration ttl, RdsBackendGate.Lease lease) {this.id = id;this.connection = connection;this.engine = engine;this.resourceArn = resourceArn;this.database = database;this.region = region;+ this.lease = lease;refresh(ttl);}private void refresh(Duration ttl) {this.expiresAt = Instant.now().plus(ttl);}++ private void close() {+ try {+ closeQuietly(connection);+ } finally {+ lease.close();+ }+ }}private record Credentials(String username, String password) {diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 5ce38f14a1..327725e64f 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -418,6 +418,8 @@ floci:proxy-handshake-timeout-millis: 10000 # Max time a client has to complete the startup/auth handshakeproxy-backend-connect-timeout-millis: 5000 # Max time to wait for the backend TCP connectproxy-max-connections: 100 # Max concurrent proxy connections before new ones are refused+ aurora-auto-pause-enabled: true # FLOCI_SERVICES_RDS_AURORA_AUTO_PAUSE_ENABLED; pause a MinCapacity=0 Aurora cluster after SecondsUntilAutoPause without connections+ aurora-resume-delay-millis: 0 # FLOCI_SERVICES_RDS_AURORA_RESUME_DELAY_MILLIS; hold the first connection this long while a paused cluster resumes (Aurora takes about 15000)rds-data:enabled: truetransaction-ttl-seconds: 180diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.javaindex 769465e5bd..6af033a7ab 100644--- a/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/rds/RdsServiceTest.java@@ -16,6 +16,7 @@ import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import io.github.hectorvent.floci.services.rds.model.DbCluster;import io.github.hectorvent.floci.services.rds.model.DbClusterSnapshot;import io.github.hectorvent.floci.services.rds.model.DbClusterParameterGroup;+import io.github.hectorvent.floci.services.rds.container.AutoPauseListener;import io.github.hectorvent.floci.services.rds.container.RdsContainerHandle;import io.github.hectorvent.floci.services.rds.container.RdsContainerManager;import io.github.hectorvent.floci.services.rds.model.DbEndpoint;@@ -73,6 +74,7 @@ import static org.mockito.ArgumentMatchers.anyList;import static org.mockito.ArgumentMatchers.anyString;import static org.mockito.ArgumentMatchers.eq;import static org.mockito.ArgumentMatchers.isNull;+import static org.mockito.Mockito.atLeastOnce;import static org.mockito.Mockito.doReturn;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.never;@@ -1758,6 +1760,102 @@ class RdsServiceTest {assertEquals(300, passwordOnlyChange.getServerlessV2SecondsUntilAutoPause());}+ @Test+ void auroraClusterHandsItsAutoPauseIntervalToItsContainer() {+ DbCluster cluster = createAutoPauseCluster(rdsService, "paused", 600);++ verify(containerManager).configureAutoPause(eq(cluster.getDbClusterArn()), eq(600), any());++ rdsService.modifyDbCluster("paused", null, null, 0.5, null, null);+ verify(containerManager).configureAutoPause(eq(cluster.getDbClusterArn()), isNull(), any());+ }++ @Test+ void auroraClusterMayAutoPauseOnlyWithServerlessInstancesAndNothingHoldingItAwake() {+ DbCluster cluster = createAutoPauseCluster(rdsService, "paused", 300);+ AutoPauseListener listener = capturedAutoPauseListener(cluster);+ assertFalse(listener.mayPause(), "a cluster without instances has nothing to pause");++ createMember(rdsService, "paused-1", "db.serverless", "paused");+ assertTrue(listener.mayPause());++ createMember(rdsService, "paused-2", "db.r6g.large", "paused");+ assertFalse(listener.mayPause(), "a provisioned instance keeps the cluster awake");+ rdsService.deleteDbInstance("paused-2");+ assertTrue(listener.mayPause());++ rdsService.modifyDbCluster("paused", null, null, 0.5, null, null);+ assertFalse(listener.mayPause(), "a nonzero MinCapacity never pauses");+ rdsService.modifyDbCluster("paused", null, null, 0.0, null, null);+ assertTrue(listener.mayPause());++ rdsService.createGlobalCluster("global-1", "paused", null, null, null, null, null,+ Map.of(), "us-east-1");+ assertFalse(listener.mayPause(), "a global database keeps its primary awake");+ }++ @Test+ void auroraClusterBehindAnRdsProxyDoesNotAutoPause() {+ InMemoryStorage<String, DbProxyTargetGroup> targetGroups = new InMemoryStorage<>();+ RdsService service = proxyStoreService(regionResolver, config, new InMemoryStorage<>(),+ targetGroups, new InMemoryStorage<>(), new InMemoryStorage<>());+ DbCluster cluster = createAutoPauseCluster(service, "paused", 300);+ createMember(service, "paused-1", "db.serverless", "paused");+ AutoPauseListener listener = capturedAutoPauseListener(cluster);+ assertTrue(listener.mayPause());++ DbProxyTargetGroup targetGroup = persistedTargetGroup("app-proxy", "us-east-1", "123456789012", "abc");+ targetGroup.setTargets(List.of(new DbProxyTarget("TRACKED_CLUSTER", "paused",+ cluster.getDbClusterArn(), "localhost", 5432)));+ targetGroups.put("us-east-1::app-proxy", targetGroup);++ assertFalse(listener.mayPause(), "an RDS Proxy holds connections open to its targets");+ }++ @Test+ void autoPauseStepsAreRecordedAsEventsOfEachServerlessInstance() {+ DbCluster cluster = createAutoPauseCluster(rdsService, "paused", 300);+ DbInstance member = createMember(rdsService, "paused-1", "db.serverless", "paused");+ AutoPauseListener listener = capturedAutoPauseListener(cluster);+ Instant start = Instant.now();++ listener.onAutoPause(AutoPauseListener.Event.PAUSE_INITIATED, start);+ listener.onAutoPause(AutoPauseListener.Event.PAUSED, start.plusSeconds(1));+ listener.onAutoPause(AutoPauseListener.Event.STILL_PAUSED, start.plusSeconds(61));+ listener.onAutoPause(AutoPauseListener.Event.RESUME_INITIATED, start.plusSeconds(90));+ listener.onAutoPause(AutoPauseListener.Event.RESUMED, start.plusSeconds(91));++ List<RdsEvent> events = rdsService.describeEvents(+ "paused-1", "db-instance", start.minusSeconds(1), start.plusSeconds(120), null);+ assertEquals(List.of(+ "Initiated pause for the DB instance.",+ "Successfully paused the DB instance.",+ "Initiated resume for the DB instance.",+ "Successfully resumed the DB instance."),+ events.stream().map(RdsEvent::message).toList());+ RdsEvent paused = events.get(1);+ assertEquals(List.of("notification", "serverless"), paused.eventCategories());+ assertEquals(member.getDbInstanceArn(), paused.sourceArn());+ assertEquals(start.plusSeconds(1), paused.date());+ }++ private static DbCluster createAutoPauseCluster(RdsService service, String id, int secondsUntilAutoPause) {+ return service.createDbCluster(id, "aurora-postgresql", "16.3", "admin", "password", "dbname",+ false, null, null, null, false, "us-east-1", 0.0, 4.0, secondsUntilAutoPause);+ }++ private static DbInstance createMember(RdsService service, String id, String instanceClass, String clusterId) {+ return service.createDbInstance(id, "aurora-postgresql", "16.3", "admin", "password", "dbname",+ instanceClass, 20, false, null, null, clusterId);+ }++ private AutoPauseListener capturedAutoPauseListener(DbCluster cluster) {+ ArgumentCaptor<AutoPauseListener> listener = ArgumentCaptor.forClass(AutoPauseListener.class);+ verify(containerManager, atLeastOnce())+ .configureAutoPause(eq(cluster.getDbClusterArn()), any(), listener.capture());+ return listener.getValue();+ }+@Testvoid modifyDbClusterRejectsServerlessV2ScalingForNonAuroraCluster() {when(config.services().rds().mock()).thenReturn(true);diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/container/AutoPauseControllerTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/container/AutoPauseControllerTest.javanew file mode 100644index 0000000000..2fa57ce614--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/rds/container/AutoPauseControllerTest.java@@ -0,0 +1,567 @@+package io.github.hectorvent.floci.services.rds.container;++import io.github.hectorvent.floci.services.rds.container.AutoPauseController.State;+import io.github.hectorvent.floci.services.rds.container.AutoPauseListener.Event;+import org.junit.jupiter.api.Test;++import java.time.Clock;+import java.time.Duration;+import java.time.Instant;+import java.time.ZoneId;+import java.time.ZoneOffset;+import java.util.ArrayList;+import java.util.Comparator;+import java.util.List;+import java.util.concurrent.CompletableFuture;+import java.util.concurrent.CopyOnWriteArrayList;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.Future;+import java.util.concurrent.FutureTask;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicInteger;+import java.util.function.BooleanSupplier;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertSame;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.junit.jupiter.api.Assertions.fail;++class AutoPauseControllerTest {++ private static final Duration IDLE = Duration.ofSeconds(300);++ private final ManualScheduler scheduler = new ManualScheduler();+ private final FakeFreezer freezer = new FakeFreezer();+ private final RecordingListener listener = new RecordingListener();++ @Test+ void pausesOnceIdleForSecondsUntilAutoPause() {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);++ scheduler.advance(Duration.ofSeconds(299));+ assertEquals(State.RUNNING, controller.state());+ assertEquals(0, freezer.pauses.get());++ scheduler.advance(Duration.ofSeconds(1));+ assertEquals(State.PAUSED, controller.state());+ assertTrue(freezer.paused);+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSED), listener.events);+ assertTrue(listener.times.get(1).isAfter(listener.times.get(0)),+ "events raised within one instant still keep their order");+ }++ @Test+ void openConnectionsKeepItAwakeAndTheIdleClockStartsWhenTheLastOneCloses() throws Exception {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ RdsBackendGate.Lease first = controller.acquire();+ RdsBackendGate.Lease second = controller.acquire();++ scheduler.advance(Duration.ofHours(1));+ first.close();+ scheduler.advance(Duration.ofHours(1));+ assertEquals(State.RUNNING, controller.state(), "one connection is still open");++ second.close();+ scheduler.advance(Duration.ofSeconds(299));+ assertEquals(State.RUNNING, controller.state());+ scheduler.advance(Duration.ofSeconds(1));+ assertEquals(State.PAUSED, controller.state());+ }++ @Test+ void connectionResumesAPausedContainerBeforeItIsServed() throws Exception {+ AutoPauseController controller = pausedController(Duration.ZERO);++ RdsBackendGate.Lease lease = controller.acquire();++ assertEquals(State.RUNNING, controller.state());+ assertFalse(freezer.paused);+ assertEquals(1, freezer.unpauses.get());+ assertEquals(1, controller.leaseCount());+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSED, Event.RESUME_INITIATED, Event.RESUMED),+ listener.events);++ lease.close();+ scheduler.advance(IDLE);+ assertEquals(State.PAUSED, controller.state(), "idle again after the connection closed");+ }++ @Test+ void resumeDelayHoldsConnectionsIncludingOnesThatArriveWhileResuming() throws Exception {+ AutoPauseController controller = pausedController(Duration.ofSeconds(15));++ Acquirer first = new Acquirer(controller, freezer);+ awaitCondition(() -> controller.state() == State.RESUMING && first.isWaiting(), "first waits");+ Acquirer second = new Acquirer(controller, freezer);+ second.awaitWaiting();++ scheduler.advance(Duration.ofSeconds(14));+ assertFalse(first.isDone());+ assertFalse(second.isDone());+ assertEquals(State.RESUMING, controller.state());++ scheduler.advance(Duration.ofSeconds(1));+ first.lease();+ second.lease();+ assertFalse(first.sawFrozenContainer);+ assertFalse(second.sawFrozenContainer);+ assertEquals(1, freezer.unpauses.get());+ assertEquals(2, controller.leaseCount());+ }++ @Test+ void connectionDuringPausingCancelsThePause() throws Exception {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ freezer.blockPause();+ Thread pausing = Thread.ofPlatform().daemon().start(() -> scheduler.advance(IDLE));+ freezer.awaitPauseEntered();+ assertEquals(State.PAUSING, controller.state());++ Acquirer client = new Acquirer(controller, freezer);+ client.awaitWaiting();+ freezer.releasePause();+ RdsBackendGate.Lease lease = client.lease();+ pausing.join(TimeUnit.SECONDS.toMillis(5));++ assertFalse(client.sawFrozenContainer, "the client must not reach the frozen container");+ assertEquals(State.RUNNING, controller.state());+ assertFalse(freezer.paused);+ assertEquals(1, freezer.unpauses.get());+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSE_CANCELED), listener.events);+ assertEquals(1, controller.leaseCount());++ lease.close();+ scheduler.advance(IDLE);+ assertEquals(State.PAUSED, controller.state());+ }++ @Test+ void concurrentConnectionsResumeTheContainerOnce() throws Exception {+ AutoPauseController controller = pausedController(Duration.ZERO);+ freezer.blockUnpause();++ List<Acquirer> clients = new ArrayList<>();+ for (int i = 0; i < 16; i++) {+ clients.add(new Acquirer(controller, freezer));+ }+ freezer.awaitUnpauseEntered();+ awaitCondition(() -> clients.stream().allMatch(Acquirer::isWaiting), "every client waits");+ freezer.releaseUnpause();+ for (Acquirer client : clients) {+ client.lease();+ assertFalse(client.sawFrozenContainer);+ }++ assertEquals(1, freezer.unpauses.get());+ assertEquals(16, controller.leaseCount());+ assertEquals(1, listener.events.stream().filter(event -> event == Event.RESUMED).count());+ }++ @Test+ void turningAutoPauseOffResumesAPausedContainerAndKeepsItRunning() {+ AutoPauseController controller = pausedController(Duration.ZERO);++ controller.configure(null, listener);+ scheduler.advance(Duration.ZERO);++ assertEquals(State.RUNNING, controller.state());+ assertFalse(freezer.paused);+ scheduler.advance(Duration.ofDays(1));+ assertEquals(State.RUNNING, controller.state());+ assertEquals(1, freezer.pauses.get());+ }++ @Test+ void turningAutoPauseOffWhilePausingCancelsThePause() throws Exception {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ freezer.blockPause();+ Thread pausing = Thread.ofPlatform().daemon().start(() -> scheduler.advance(IDLE));+ freezer.awaitPauseEntered();++ controller.configure(null, listener);+ freezer.releasePause();+ pausing.join(TimeUnit.SECONDS.toMillis(5));++ assertEquals(State.RUNNING, controller.state());+ assertFalse(freezer.paused);+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSE_CANCELED), listener.events);+ scheduler.advance(Duration.ofDays(1));+ assertEquals(1, freezer.pauses.get());+ }++ @Test+ void turningAutoPauseOffBeforeTheIntervalEndsPreventsThePause() {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ scheduler.advance(Duration.ofSeconds(200));++ controller.configure(null, listener);+ scheduler.advance(Duration.ofDays(1));++ assertEquals(State.RUNNING, controller.state());+ assertEquals(0, freezer.pauses.get());+ assertEquals(0, listener.checks.get());+ }++ @Test+ void changingTheIntervalRestartsTheIdleClock() {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ scheduler.advance(Duration.ofSeconds(200));++ controller.configure(600, listener);+ scheduler.advance(Duration.ofSeconds(599));+ assertEquals(State.RUNNING, controller.state());+ scheduler.advance(Duration.ofSeconds(1));+ assertEquals(State.PAUSED, controller.state());+ }++ @Test+ void clusterThatMayNotPauseIsAskedAgainAfterAnotherInterval() {+ AutoPauseController controller = controller(Duration.ZERO);+ listener.mayPause = false;+ controller.configure(300, listener);++ scheduler.advance(IDLE);+ assertEquals(State.RUNNING, controller.state());+ assertEquals(1, listener.checks.get());+ assertEquals(0, freezer.pauses.get());++ listener.mayPause = true;+ scheduler.advance(IDLE);+ assertEquals(State.PAUSED, controller.state());+ assertEquals(2, listener.checks.get());+ }++ @Test+ void failedPauseLeavesTheContainerRunningAndTriesAgainLater() {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ freezer.pauseFailure = new IllegalStateException("daemon unavailable");++ scheduler.advance(IDLE);+ assertEquals(State.RUNNING, controller.state());+ assertEquals(List.of(Event.PAUSE_INITIATED, Event.PAUSE_CANCELED), listener.events);++ freezer.pauseFailure = null;+ scheduler.advance(IDLE);+ assertEquals(State.PAUSED, controller.state());+ }++ @Test+ void failedResumeFailsTheConnectionAndLeavesTheContainerPaused() throws Exception {+ AutoPauseController controller = pausedController(Duration.ZERO);+ freezer.unpauseFailure = new IllegalStateException("daemon unavailable");++ assertThrows(IllegalStateException.class, controller::acquire);+ assertEquals(State.PAUSED, controller.state());++ freezer.unpauseFailure = null;+ controller.acquire();+ assertEquals(State.RUNNING, controller.state());+ assertFalse(freezer.paused);+ }++ @Test+ void closeThawsAPausedContainerAndStopsTracking() throws Exception {+ AutoPauseController controller = pausedController(Duration.ZERO);++ controller.close();++ assertFalse(freezer.paused);+ assertSame(RdsBackendGate.Lease.NONE, controller.acquire());+ assertEquals(0, controller.leaseCount());+ scheduler.advance(Duration.ofDays(1));+ assertEquals(1, freezer.pauses.get());+ }++ @Test+ void closeDuringTheResumeDelayReleasesTheWaitingConnection() throws Exception {+ AutoPauseController controller = pausedController(Duration.ofSeconds(15));+ Acquirer client = new Acquirer(controller, freezer);+ awaitCondition(() -> controller.state() == State.RESUMING && client.isWaiting(), "client waits");++ controller.close();++ assertSame(RdsBackendGate.Lease.NONE, client.lease());+ assertFalse(freezer.paused);+ }++ @Test+ void closeWhilePausingThawsTheContainerOnceTheFreezeReturns() throws Exception {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ freezer.blockPause();+ Thread pausing = Thread.ofPlatform().daemon().start(() -> scheduler.advance(IDLE));+ freezer.awaitPauseEntered();++ controller.close();+ freezer.releasePause();+ pausing.join(TimeUnit.SECONDS.toMillis(5));++ assertFalse(freezer.paused, "a released container must not stay frozen");+ assertEquals(State.RUNNING, controller.state());+ }++ @Test+ void pausedContainerReportsEveryMinuteUntilItResumes() throws Exception {+ AutoPauseController controller = pausedController(Duration.ZERO);++ scheduler.advance(Duration.ofMinutes(2));+ assertEquals(2, listener.events.stream().filter(event -> event == Event.STILL_PAUSED).count());++ controller.acquire();+ scheduler.advance(Duration.ofMinutes(10));+ assertEquals(2, listener.events.stream().filter(event -> event == Event.STILL_PAUSED).count());+ }++ @Test+ void closingALeaseTwiceCountsOnce() throws Exception {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ RdsBackendGate.Lease first = controller.acquire();+ controller.acquire();++ first.close();+ first.close();++ assertEquals(1, controller.leaseCount());+ }++ @Test+ void pauseIfIdleRunsTheIdleCheckWithoutWaiting() throws Exception {+ AutoPauseController controller = controller(Duration.ZERO);+ controller.configure(300, listener);+ RdsBackendGate.Lease lease = controller.acquire();+ assertFalse(controller.pauseIfIdle(), "a connection is open");++ lease.close();+ assertTrue(controller.pauseIfIdle());+ assertEquals(State.PAUSED, controller.state());+ }++ private AutoPauseController controller(Duration resumeDelay) {+ return new AutoPauseController("container-1", "localhost", 5432, freezer, scheduler,+ Runnable::run, scheduler.clock(), resumeDelay);+ }++ private AutoPauseController pausedController(Duration resumeDelay) {+ AutoPauseController controller = controller(resumeDelay);+ controller.configure(300, listener);+ scheduler.advance(IDLE);+ assertEquals(State.PAUSED, controller.state());+ return controller;+ }++ private static void awaitCondition(BooleanSupplier condition, String description) {+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);+ while (!condition.getAsBoolean()) {+ if (System.nanoTime() > deadline) {+ fail("Timed out waiting until " + description);+ }+ Thread.onSpinWait();+ }+ }++ /** Runs scheduled tasks on whichever thread advances its clock, in the order they fall due. */+ private static final class ManualScheduler implements AutoPauseController.Scheduler {++ private Instant now = Instant.parse("2026-01-01T00:00:00Z");+ private long sequence;+ private final List<Scheduled> tasks = new ArrayList<>();++ @Override+ public synchronized Future<?> schedule(Runnable task, Duration delay) {+ FutureTask<Void> future = new FutureTask<>(task, null);+ tasks.add(new Scheduled(now.plus(delay), sequence++, future));+ return future;+ }++ void advance(Duration duration) {+ Instant target;+ synchronized (this) {+ target = now.plus(duration);+ }+ while (true) {+ Scheduled next;+ synchronized (this) {+ next = tasks.stream()+ .filter(task -> !task.due().isAfter(target))+ .min(Comparator.comparing(Scheduled::due).thenComparingLong(Scheduled::sequence))+ .orElse(null);+ if (next == null) {+ now = target;+ return;+ }+ tasks.remove(next);+ if (next.due().isAfter(now)) {+ now = next.due();+ }+ }+ // Outside the monitor: the task schedules its successors.+ next.future().run();+ }+ }++ Clock clock() {+ return new Clock() {+ @Override+ public ZoneId getZone() {+ return ZoneOffset.UTC;+ }++ @Override+ public Clock withZone(ZoneId zone) {+ return this;+ }++ @Override+ public Instant instant() {+ synchronized (ManualScheduler.this) {+ return now;+ }+ }+ };+ }++ private record Scheduled(Instant due, long sequence, FutureTask<Void> future) {+ }+ }++ private static final class FakeFreezer implements AutoPauseController.Freezer {++ final AtomicInteger pauses = new AtomicInteger();+ final AtomicInteger unpauses = new AtomicInteger();+ volatile boolean paused;+ volatile RuntimeException pauseFailure;+ volatile RuntimeException unpauseFailure;+ private volatile CountDownLatch pauseEntered;+ private volatile CountDownLatch pauseRelease;+ private volatile CountDownLatch unpauseEntered;+ private volatile CountDownLatch unpauseRelease;++ @Override+ public void pause(String containerId) {+ pauses.incrementAndGet();+ block(pauseEntered, pauseRelease);+ if (pauseFailure != null) {+ throw pauseFailure;+ }+ paused = true;+ }++ @Override+ public void unpause(String containerId) {+ unpauses.incrementAndGet();+ block(unpauseEntered, unpauseRelease);+ if (unpauseFailure != null) {+ throw unpauseFailure;+ }+ paused = false;+ }++ void blockPause() {+ pauseEntered = new CountDownLatch(1);+ pauseRelease = new CountDownLatch(1);+ }++ void awaitPauseEntered() throws InterruptedException {+ assertTrue(pauseEntered.await(5, TimeUnit.SECONDS), "docker pause was not called");+ }++ void releasePause() {+ pauseRelease.countDown();+ }++ void blockUnpause() {+ unpauseEntered = new CountDownLatch(1);+ unpauseRelease = new CountDownLatch(1);+ }++ void awaitUnpauseEntered() throws InterruptedException {+ assertTrue(unpauseEntered.await(5, TimeUnit.SECONDS), "docker unpause was not called");+ }++ void releaseUnpause() {+ unpauseRelease.countDown();+ }++ private static void block(CountDownLatch entered, CountDownLatch release) {+ if (entered == null) {+ return;+ }+ entered.countDown();+ try {+ if (!release.await(5, TimeUnit.SECONDS)) {+ throw new IllegalStateException("The test never released the Docker call");+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ throw new IllegalStateException(e);+ }+ }+ }++ private static final class RecordingListener implements AutoPauseListener {++ final List<Event> events = new CopyOnWriteArrayList<>();+ final List<Instant> times = new CopyOnWriteArrayList<>();+ final AtomicInteger checks = new AtomicInteger();+ volatile boolean mayPause = true;++ @Override+ public boolean mayPause() {+ checks.incrementAndGet();+ return mayPause;+ }++ @Override+ public void onAutoPause(Event event, Instant at) {+ events.add(event);+ times.add(at);+ }+ }++ /** Calls acquire() on a platform thread of its own, so that a test can watch it wait. */+ private static final class Acquirer {++ private final CompletableFuture<RdsBackendGate.Lease> result = new CompletableFuture<>();+ private final Thread thread;+ private volatile boolean sawFrozenContainer;++ Acquirer(AutoPauseController controller, FakeFreezer freezer) {+ thread = Thread.ofPlatform().daemon().start(() -> {+ try {+ RdsBackendGate.Lease lease = controller.acquire();+ sawFrozenContainer = freezer.paused;+ result.complete(lease);+ } catch (InterruptedException | RuntimeException e) {+ result.completeExceptionally(e);+ }+ });+ }++ boolean isWaiting() {+ Thread.State state = thread.getState();+ return state == Thread.State.WAITING || state == Thread.State.TIMED_WAITING;+ }++ void awaitWaiting() {+ awaitCondition(this::isWaiting, "the client waits");+ }++ boolean isDone() {+ return result.isDone();+ }++ RdsBackendGate.Lease lease() throws Exception {+ return result.get(5, TimeUnit.SECONDS);+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsAuroraAutoPauseIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsAuroraAutoPauseIntegrationTest.javanew file mode 100644index 0000000000..9143d6df22--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsAuroraAutoPauseIntegrationTest.java@@ -0,0 +1,259 @@+package io.github.hectorvent.floci.services.rds.container;++import com.github.dockerjava.api.DockerClient;+import io.github.hectorvent.floci.services.rds.RdsService;+import io.github.hectorvent.floci.services.rds.model.DbCluster;+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;+import io.quarkus.test.junit.QuarkusTest;+import io.restassured.path.xml.XmlPath;+import io.restassured.response.Response;+import io.restassured.specification.RequestSpecification;+import jakarta.inject.Inject;+import org.junit.jupiter.api.BeforeAll;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Tag;+import org.junit.jupiter.api.Test;++import java.sql.Connection;+import java.sql.DriverManager;+import java.sql.ResultSet;+import java.sql.Statement;+import java.time.Duration;+import java.time.Instant;+import java.util.List;++import static io.restassured.RestAssured.given;+import static io.restassured.http.ContentType.URLENC;+import static org.awaitility.Awaitility.await;+import static org.hamcrest.Matchers.containsString;+import static org.hamcrest.Matchers.not;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.junit.jupiter.api.Assumptions.assumeTrue;++/**+ * Aurora Serverless v2 auto-pause against a real PostgreSQL container: an idle cluster with+ * MinCapacity 0 is frozen with docker pause while it reports "available", zero capacity and the+ * pause events, and the next connection through its endpoint resumes it and finds its data. The+ * API does not accept a SecondsUntilAutoPause below 300, so the test runs the idle check directly+ * instead of waiting for it.+ */+@QuarkusTest+@Tag("docker")+class RdsAuroraAutoPauseIntegrationTest {++ private static final String RDS_AUTH =+ "AWS4-HMAC-SHA256 Credential=test/20260923/us-east-1/rds/aws4_request, "+ + "SignedHeaders=content-type;host, Signature=test";+ private static final String OWNER_RDS_AUTH =+ "AWS4-HMAC-SHA256 Credential=222222222222/20260923/us-east-1/rds/aws4_request, "+ + "SignedHeaders=content-type;host, Signature=test";+ private static final String CLOUDWATCH_AUTH =+ "AWS4-HMAC-SHA256 Credential=test/20260923/us-east-1/monitoring/aws4_request";+ private static final String MASTER_USER = "master";+ private static final String MASTER_PASSWORD = "Master123!";++ @Inject+ DockerClient dockerClient;++ @Inject+ RdsService rdsService;++ @Inject+ RdsContainerManager containerManager;++ @BeforeAll+ static void configureRestAssured() {+ RestAssuredJsonUtils.configureAwsContentTypes();+ }++ @BeforeEach+ void checkDocker() {+ try {+ dockerClient.pingCmd().exec();+ } catch (Exception e) {+ assumeTrue(false, "Docker is not available: " + e.getMessage());+ }+ }++ @Test+ void idleClusterPausesAndItsNextConnectionResumesIt() throws Exception {+ String clusterId = "auto-pause-" + Long.toString(System.nanoTime(), 36);+ String instanceId = clusterId + "-1";+ Instant start = Instant.now();+ rds("CreateDBCluster")+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("Engine", "aurora-postgresql")+ .formParam("MasterUsername", MASTER_USER)+ .formParam("MasterUserPassword", MASTER_PASSWORD)+ .formParam("DatabaseName", "appdb")+ .formParam("ServerlessV2ScalingConfiguration.MinCapacity", "0")+ .formParam("ServerlessV2ScalingConfiguration.MaxCapacity", "2")+ .formParam("ServerlessV2ScalingConfiguration.SecondsUntilAutoPause", "300")+ .when().post("/").then().statusCode(200);+ try {+ rds("CreateDBInstance")+ .formParam("DBInstanceIdentifier", instanceId)+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("Engine", "aurora-postgresql")+ .formParam("DBInstanceClass", "db.serverless")+ .when().post("/").then().statusCode(200);+ DbCluster cluster = rdsService.getDbCluster(clusterId);+ String containerId = cluster.getContainerId();+ String jdbcUrl = "jdbc:postgresql://127.0.0.1:" + cluster.getEndpoint().port()+ + "/appdb?sslmode=disable";+ try (Connection connection = DriverManager.getConnection(jdbcUrl, MASTER_USER, MASTER_PASSWORD);+ Statement statement = connection.createStatement()) {+ statement.execute("CREATE TABLE marker(v text); INSERT INTO marker VALUES ('before-pause');");+ }++ // The proxy lets go of the closed connection on its own thread.+ await().atMost(Duration.ofSeconds(10))+ .until(() -> containerManager.pauseIfIdle(cluster.getDbClusterArn()));+ assertTrue(isPaused(containerId), "the idle cluster's container is frozen");+ rds("DescribeDBClusters").formParam("DBClusterIdentifier", clusterId)+ .when().post("/").then().statusCode(200)+ .body(containsString("<Status>available</Status>"));+ rds("DescribeDBInstances").formParam("DBInstanceIdentifier", instanceId)+ .when().post("/").then().statusCode(200)+ .body(containsString("<DBInstanceStatus>available</DBInstanceStatus>"));+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->+ events(instanceId).then().statusCode(200)+ .body(containsString("<Message>Initiated pause for the DB instance.</Message>"))+ .body(containsString("<Message>Successfully paused the DB instance.</Message>")));+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {+ for (String metricName : List.of("ServerlessDatabaseCapacity", "CPUUtilization")) {+ List<Float> minimums = metricMinimums(metricName, clusterId, start);+ assertFalse(minimums.isEmpty(), "a paused cluster reports its " + metricName);+ assertEquals(0.0f, minimums.get(minimums.size() - 1), metricName);+ }+ });++ try (Connection connection = DriverManager.getConnection(jdbcUrl, MASTER_USER, MASTER_PASSWORD);+ Statement statement = connection.createStatement();+ ResultSet rows = statement.executeQuery("SELECT v FROM marker")) {+ assertTrue(rows.next(), "the data written before the pause is still there");+ assertEquals("before-pause", rows.getString(1));+ }+ assertFalse(isPaused(containerId), "the connection resumed the container");+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->+ events(instanceId).then().statusCode(200)+ .body(containsString("<Message>Initiated resume for the DB instance.</Message>"))+ .body(containsString("<Message>Successfully resumed the DB instance.</Message>")));++ await().atMost(Duration.ofSeconds(10))+ .until(() -> containerManager.pauseIfIdle(cluster.getDbClusterArn()));+ rds("ModifyDBCluster")+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("ServerlessV2ScalingConfiguration.MinCapacity", "0.5")+ .formParam("ServerlessV2ScalingConfiguration.MaxCapacity", "2")+ .when().post("/").then().statusCode(200);+ await().atMost(Duration.ofSeconds(10)).until(() -> !isPaused(containerId));+ assertFalse(containerManager.pauseIfIdle(cluster.getDbClusterArn()),+ "a nonzero MinCapacity keeps the cluster running");+ } finally {+ rds("DeleteDBInstance")+ .formParam("DBInstanceIdentifier", instanceId)+ .formParam("SkipFinalSnapshot", "true")+ .when().post("/");+ rds("DeleteDBCluster")+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("SkipFinalSnapshot", "true")+ .when().post("/");+ }+ }++ @Test+ void pauseEventsBelongToTheAccountThatOwnsTheCluster() throws Exception {+ String clusterId = "auto-pause-owner-" + Long.toString(System.nanoTime(), 36);+ String instanceId = clusterId + "-1";+ XmlPath created = rds(OWNER_RDS_AUTH, "CreateDBCluster")+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("Engine", "aurora-postgresql")+ .formParam("MasterUsername", MASTER_USER)+ .formParam("MasterUserPassword", MASTER_PASSWORD)+ .formParam("DatabaseName", "appdb")+ .formParam("ServerlessV2ScalingConfiguration.MinCapacity", "0")+ .formParam("ServerlessV2ScalingConfiguration.MaxCapacity", "2")+ .formParam("ServerlessV2ScalingConfiguration.SecondsUntilAutoPause", "300")+ .when().post("/").then().statusCode(200).extract().xmlPath();+ String clusterArn = created.getString("CreateDBClusterResponse.CreateDBClusterResult.DBCluster.DBClusterArn");+ int port = created.getInt("CreateDBClusterResponse.CreateDBClusterResult.DBCluster.Port");+ try {+ rds(OWNER_RDS_AUTH, "CreateDBInstance")+ .formParam("DBInstanceIdentifier", instanceId)+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("Engine", "aurora-postgresql")+ .formParam("DBInstanceClass", "db.serverless")+ .when().post("/").then().statusCode(200);++ // The pause and resume events reach RDS on a background thread, outside any request.+ await().atMost(Duration.ofSeconds(10)).until(() -> containerManager.pauseIfIdle(clusterArn));+ try (Connection connection = DriverManager.getConnection(+ "jdbc:postgresql://127.0.0.1:" + port + "/appdb?sslmode=disable", MASTER_USER, MASTER_PASSWORD)) {+ assertTrue(connection.isValid(5), "the connection resumed the cluster");+ }+ await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->+ events(OWNER_RDS_AUTH, instanceId).then().statusCode(200)+ .body(containsString("<Message>Initiated pause for the DB instance.</Message>"))+ .body(containsString("<Message>Successfully paused the DB instance.</Message>"))+ .body(containsString("<Message>Initiated resume for the DB instance.</Message>"))+ .body(containsString("<Message>Successfully resumed the DB instance.</Message>")));+ events(RDS_AUTH, instanceId).then().statusCode(200)+ .body(not(containsString("<Event>")));+ } finally {+ rds(OWNER_RDS_AUTH, "DeleteDBInstance")+ .formParam("DBInstanceIdentifier", instanceId)+ .formParam("SkipFinalSnapshot", "true")+ .when().post("/");+ rds(OWNER_RDS_AUTH, "DeleteDBCluster")+ .formParam("DBClusterIdentifier", clusterId)+ .formParam("SkipFinalSnapshot", "true")+ .when().post("/");+ }+ }++ private static RequestSpecification rds(String action) {+ return rds(RDS_AUTH, action);+ }++ private static RequestSpecification rds(String authorization, String action) {+ return given().header("Authorization", authorization)+ .contentType(URLENC)+ .formParam("Action", action)+ .formParam("Version", "2014-10-31");+ }++ private static Response events(String instanceId) {+ return events(RDS_AUTH, instanceId);+ }++ private static Response events(String authorization, String instanceId) {+ return rds(authorization, "DescribeEvents")+ .formParam("SourceType", "db-instance")+ .formParam("SourceIdentifier", instanceId)+ .formParam("Duration", "60")+ .when().post("/");+ }++ private static List<Float> metricMinimums(String metricName, String clusterId, Instant from) {+ String body = """+ {"Namespace":"AWS/RDS","MetricName":"%s",+ "Dimensions":[{"Name":"DBClusterIdentifier","Value":"%s"}],+ "StartTime":%d,"EndTime":%d,"Period":60,"Statistics":["Minimum"]}+ """.formatted(metricName, clusterId, from.minusSeconds(60).getEpochSecond(),+ Instant.now().plusSeconds(60).getEpochSecond());+ return given().contentType("application/x-amz-json-1.0")+ .header("Authorization", CLOUDWATCH_AUTH)+ .header("X-Amz-Target", "GraniteServiceVersion20100801.GetMetricStatistics")+ .body(body)+ .post("/")+ .then().statusCode(200)+ .extract().jsonPath().getList("Datapoints.Minimum", Float.class);+ }++ private boolean isPaused(String containerId) {+ return Boolean.TRUE.equals(dockerClient.inspectContainerCmd(containerId).exec().getState().getPaused());+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.javaindex 65bcca44a0..7f5d9a7175 100644--- a/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/rds/container/RdsContainerManagerTest.java@@ -12,9 +12,11 @@ import io.github.hectorvent.floci.core.common.docker.ContainerSpec;import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;import org.junit.jupiter.api.Test;import org.junit.jupiter.api.io.TempDir;+import org.mockito.InOrder;import java.nio.file.Files;import java.nio.file.Path;+import java.time.Instant;import java.util.List;import java.util.Map;import java.util.Optional;@@ -24,6 +26,8 @@ import java.util.concurrent.TimeUnit;import com.github.dockerjava.api.DockerClient;import com.github.dockerjava.api.command.CopyArchiveToContainerCmd;+import com.github.dockerjava.api.command.PauseContainerCmd;+import com.github.dockerjava.api.command.UnpauseContainerCmd;import com.github.dockerjava.api.model.Bind;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;@@ -31,14 +35,17 @@ import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;import static org.junit.jupiter.api.Assertions.assertFalse;import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertSame;import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.eq;import static org.mockito.Mockito.lenient;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.never;import static org.mockito.Mockito.doThrow;import static org.mockito.Mockito.doAnswer;+import static org.mockito.Mockito.inOrder;import static org.mockito.Mockito.times;import static org.mockito.Mockito.verify;import static org.mockito.Mockito.when;@@ -1155,6 +1162,118 @@ class RdsContainerManagerTest {"Script must assign the first argument to the USER variable");}+ @Test+ void autoPausedContainerResumesForItsNextClientAndIsThawedBeforeItStops() throws Exception {+ EmulatorConfig config = config(tempDir.resolve("auto-pause-root"));+ when(config.services().rds().auroraAutoPauseEnabled()).thenReturn(true);+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ stubStarts(lifecycleManager, new ContainerLifecycleManager.ContainerInfo(+ "cluster-container", Map.of(3306, new ContainerLifecycleManager.EndpointInfo("db1", 3306))));+ DockerClient dockerClient = mock(DockerClient.class);+ PauseContainerCmd pause = mock(PauseContainerCmd.class);+ UnpauseContainerCmd unpause = mock(UnpauseContainerCmd.class);+ when(dockerClient.pauseContainerCmd("cluster-container")).thenReturn(pause);+ when(dockerClient.unpauseContainerCmd("cluster-container")).thenReturn(unpause);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ RdsContainerManager manager = autoPauseManager(config, lifecycleManager);+ String runtimeId = "arn:aws:rds:us-east-1:000000000000:cluster:paused";+ RdsContainerHandle handle = manager.start(runtimeId, "paused", null,+ DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");+ manager.configureAutoPause(runtimeId, 300, alwaysMayPause());++ try (RdsBackendGate.Lease client = manager.enter("db1", 3306)) {+ assertFalse(manager.pauseIfIdle(runtimeId), "a client is connected");+ }+ assertTrue(manager.pauseIfIdle(runtimeId));+ verify(pause).exec();++ manager.enter("db1", 3306).close();+ verify(unpause).exec();++ assertTrue(manager.pauseIfIdle(runtimeId));+ manager.stop(handle);+ InOrder order = inOrder(unpause, lifecycleManager);+ order.verify(unpause, times(2)).exec();+ order.verify(lifecycleManager).stopAndRemoveStrict(eq("cluster-container"), any());+ assertSame(RdsBackendGate.Lease.NONE, manager.enter("db1", 3306));+ }++ @Test+ void resetThawsAPausedContainerAndEndsItsAutoPause() throws Exception {+ EmulatorConfig config = config(tempDir.resolve("auto-pause-root"));+ when(config.services().rds().auroraAutoPauseEnabled()).thenReturn(true);+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ stubStarts(lifecycleManager, new ContainerLifecycleManager.ContainerInfo(+ "cluster-container", Map.of(3306, new ContainerLifecycleManager.EndpointInfo("db1", 3306))));+ DockerClient dockerClient = mock(DockerClient.class);+ UnpauseContainerCmd unpause = mock(UnpauseContainerCmd.class);+ when(dockerClient.pauseContainerCmd("cluster-container")).thenReturn(mock(PauseContainerCmd.class));+ when(dockerClient.unpauseContainerCmd("cluster-container")).thenReturn(unpause);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ RdsContainerManager manager = autoPauseManager(config, lifecycleManager);+ String runtimeId = "arn:aws:rds:us-east-1:000000000000:cluster:paused";+ manager.start(runtimeId, "paused", null, DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");+ manager.configureAutoPause(runtimeId, 300, alwaysMayPause());+ assertTrue(manager.pauseIfIdle(runtimeId));++ manager.clear();++ verify(unpause).exec();+ assertSame(RdsBackendGate.Lease.NONE, manager.enter("db1", 3306));+ assertFalse(manager.pauseIfIdle(runtimeId));+ }++ @Test+ void containerStartedOnAnAddressTakesItOverFromOneDockerRemoved() throws Exception {+ EmulatorConfig config = config(tempDir.resolve("auto-pause-root"));+ when(config.services().rds().auroraAutoPauseEnabled()).thenReturn(true);+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ ContainerLifecycleManager.EndpointInfo address = new ContainerLifecycleManager.EndpointInfo("db1", 3306);+ stubStarts(lifecycleManager,+ new ContainerLifecycleManager.ContainerInfo("removed-container", Map.of(3306, address)),+ new ContainerLifecycleManager.ContainerInfo("new-container", Map.of(3306, address)));+ DockerClient dockerClient = mock(DockerClient.class);+ when(dockerClient.pauseContainerCmd(any())).thenReturn(mock(PauseContainerCmd.class));+ when(dockerClient.unpauseContainerCmd(any())).thenReturn(mock(UnpauseContainerCmd.class));+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ RdsContainerManager manager = autoPauseManager(config, lifecycleManager);+ String removedRuntime = "arn:aws:rds:us-east-1:000000000000:cluster:removed";+ manager.start(removedRuntime, "removed", null,+ DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");+ manager.configureAutoPause(removedRuntime, 300, alwaysMayPause());+ assertTrue(manager.pauseIfIdle(removedRuntime));++ manager.start("arn:aws:rds:us-east-1:000000000000:cluster:new", "new", null,+ DatabaseEngine.MYSQL, "mysql:8.0", "root", "password", "db");+ manager.enter("db1", 3306).close();++ verify(dockerClient, never()).unpauseContainerCmd("new-container");+ assertFalse(manager.pauseIfIdle(removedRuntime), "the removed container's auto-pause is gone");+ }++ private RdsContainerManager autoPauseManager(EmulatorConfig config, ContainerLifecycleManager lifecycleManager) {+ ContainerLogStreamer logStreamer = mock(ContainerLogStreamer.class);+ lenient().when(logStreamer.generateLogStreamName(any())).thenReturn("log-stream");+ return new RdsContainerManager(+ new ContainerBuilder(config, mock(DockerHostResolver.class), mock(EmbeddedDnsServer.class)),+ lifecycleManager, logStreamer, mock(ContainerDetector.class), config,+ new RegionResolver("us-east-1", "000000000000"),+ mock(ServiceConfigAccess.class));+ }++ private static AutoPauseListener alwaysMayPause() {+ return new AutoPauseListener() {+ @Override+ public boolean mayPause() {+ return true;+ }++ @Override+ public void onAutoPause(Event event, Instant at) {+ }+ };+ }+private static EmulatorConfig config(Path hostRoot) {EmulatorConfig config = mock(EmulatorConfig.class);EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);diff --git a/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java b/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.javaindex ed758a7fde..9823ad7acf 100644--- a/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/rds/proxy/RdsAuthProxyBoundsTest.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.rds.proxy;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.services.acm.CertificateGenerator;+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import org.junit.jupiter.api.AfterEach;import org.junit.jupiter.api.Test;@@ -13,7 +14,9 @@ import java.net.ServerSocket;import java.net.Socket;import java.net.SocketTimeoutException;import java.nio.file.Path;+import java.util.concurrent.CountDownLatch;import java.util.concurrent.Semaphore;+import java.util.concurrent.TimeUnit;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertThrows;@@ -100,6 +103,37 @@ class RdsAuthProxyBoundsTest {}}+ @Test+ void holdsTheClientUntilTheBackendGateAdmitsItAndReleasesTheLeaseWhenItLeaves() throws Exception {+ fakeBackend = new ServerSocket(0);+ fakeBackend.setSoTimeout(300);+ CountDownLatch admit = new CountDownLatch(1);+ CountDownLatch released = new CountDownLatch(1);+ RdsBackendGate gate = (host, port) -> {+ admit.await();+ return released::countDown;+ };+ int proxyPort = freePort();+ proxy = new RdsAuthProxy("db-1", "localhost", fakeBackend.getLocalPort(),+ DatabaseEngine.SQLSERVER, false, "sa", "Secret123", null,+ mock(RdsSigV4Validator.class), realTls(), (user, pw) -> true,+ 5000, 5000, 100, null, gate);+ proxy.start(proxyPort);++ try (Socket client = new Socket("localhost", proxyPort)) {+ client.setSoTimeout(5000);+ assertThrows(SocketTimeoutException.class, fakeBackend::accept,+ "the proxy reached the backend before the gate admitted the client");+ admit.countDown();+ fakeBackend.setSoTimeout(5000);+ try (Socket backendSide = fakeBackend.accept()) {+ backendSide.getOutputStream().write('x');+ assertEquals('x', client.getInputStream().read(), "the held client is served once admitted");+ }+ }+ assertTrue(released.await(5, TimeUnit.SECONDS), "the lease is released when the connection ends");+ }+private RdsAuthProxy newProxy(int handshakeTimeoutMillis, int backendConnectTimeoutMillis,int maxConnections) {return new RdsAuthProxy("db-1", "localhost", fakeBackend.getLocalPort(),diff --git a/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.javaindex b2798d1486..9421ca6b2a 100644--- a/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/rdsdata/RdsDataServiceTest.java@@ -5,6 +5,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;import com.fasterxml.jackson.databind.node.ArrayNode;import com.fasterxml.jackson.databind.node.ObjectNode;import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.rds.container.RdsBackendGate;import io.github.hectorvent.floci.services.rds.model.DatabaseEngine;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;import io.github.hectorvent.floci.services.secretsmanager.model.SecretVersion;@@ -18,9 +19,12 @@ import java.sql.SQLException;import java.sql.Statement;import java.sql.Types;import java.time.Duration;+import java.util.List;import java.util.Properties;import java.util.UUID;+import java.util.concurrent.CopyOnWriteArrayList;import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicInteger;import org.h2.Driver;import static org.junit.jupiter.api.Assertions.assertArrayEquals;@@ -773,6 +777,66 @@ class RdsDataServiceTest {assertEquals(0L, harness.countById("shutdown"));}+ @Test+ void statementOutsideATransactionHoldsTheClusterAwakeOnlyWhileItRuns() throws Exception {+ TestHarness harness = new TestHarness();+ harness.createTables();++ harness.service.executeStatement(harness.request("select 1"), REGION);++ assertEquals(List.of("127.0.0.1:3306"), harness.gate.entered);+ assertEquals(0, harness.gate.held.get());+ }++ @Test+ void transactionHoldsTheClusterAwakeUntilItEnds() throws Exception {+ TestHarness harness = new TestHarness();+ harness.createTables();++ String committed = harness.service.beginTransaction(harness.beginRequest(), REGION)+ .get("transactionId").asText();+ ObjectNode insert = harness.request("insert into data_api_items(id, title, score) values ('held', 'Held', 1)");+ insert.put("transactionId", committed);+ harness.service.executeStatement(insert, REGION);+ assertEquals(1, harness.gate.held.get(), "an open transaction keeps the cluster from pausing");+ harness.service.commitTransaction(harness.transactionRequest(committed), REGION);+ assertEquals(0, harness.gate.held.get());++ String rolledBack = harness.service.beginTransaction(harness.beginRequest(), REGION)+ .get("transactionId").asText();+ assertEquals(1, harness.gate.held.get());+ harness.service.rollbackTransaction(harness.transactionRequest(rolledBack), REGION);+ assertEquals(0, harness.gate.held.get());+ assertEquals(2, harness.gate.entered.size(), "statements in a transaction reuse its hold");+ }++ @Test+ void expiredTransactionLetsTheClusterPause() throws Exception {+ TestHarness harness = new TestHarness(Duration.ofMillis(50));+ String tx = harness.service.beginTransaction(harness.beginRequest(), REGION)+ .get("transactionId").asText();+ Thread.sleep(200);++ ObjectNode next = harness.request("select 1");+ next.put("transactionId", tx);+ assertThrows(AwsException.class, () -> harness.service.executeStatement(next, REGION));++ assertEquals(0, harness.gate.held.get());+ }++ @Test+ void clusterThatCannotResumeFailsTheRequestAsAnInternalError() {+ TestHarness harness = new TestHarness();+ harness.gate.failure = new IllegalStateException("Could not resume auto-paused RDS container");++ AwsException error = assertThrows(AwsException.class,+ () -> harness.service.executeStatement(harness.request("select 1"), REGION));++ assertEquals("InternalServerErrorException", error.getErrorCode());+ assertEquals(500, error.getHttpStatus());+ assertEquals(0, harness.gate.held.get());+ }+@Testvoid closesConnectionWhenTransactionSetupFails() {RdsDataResourceResolver resolver = mock(RdsDataResourceResolver.class);@@ -921,10 +985,33 @@ class RdsDataServiceTest {});}+ /** Counts the backends the Data API enters and the holds still open on them. */+ private static final class CountingGate implements RdsBackendGate {+ private final List<String> entered = new CopyOnWriteArrayList<>();+ private final AtomicInteger held = new AtomicInteger();+ private volatile IllegalStateException failure;++ @Override+ public Lease enter(String host, int port) {+ if (failure != null) {+ throw failure;+ }+ entered.add(host + ":" + port);+ held.incrementAndGet();+ AtomicBoolean closed = new AtomicBoolean();+ return () -> {+ if (closed.compareAndSet(false, true)) {+ held.decrementAndGet();+ }+ };+ }+ }+private final class TestHarness {private final String jdbcUrl;private final RdsDataResourceResolver resolver;private final RdsDataResourceResolver.DatabaseTarget target;+ private final CountingGate gate = new CountingGate();private final RdsDataService service;private TestHarness() {@@ -966,7 +1053,7 @@ class RdsDataServiceTest {return getConnection();}};- service = new RdsDataService(resolver, secrets, objectMapper, connectionFactory, transactionTtl);+ service = new RdsDataService(resolver, secrets, objectMapper, connectionFactory, transactionTtl, gate);}private Connection getConnection() throws SQLException {diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex bb48878108..5db99bc4ba 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -206,6 +206,8 @@ floci:proxy-handshake-timeout-millis: 10000proxy-backend-connect-timeout-millis: 5000proxy-max-connections: 100+ aurora-auto-pause-enabled: true+ aurora-resume-delay-millis: 0rds-data:enabled: truetransaction-ttl-seconds: 180