Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(rds): make StopDBCluster/StartDBCluster stop and start Aurora compute (#1301)

kiran01bme364d2ed3f3b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9f0987e0..5873b185 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
### Fixed
+- **RDS — `StopDBCluster` / `StartDBCluster` now stop and start Aurora compute** — both operations only flipped metadata: a "stopped" cluster's backing container kept running and accepting SQL connections, and its members never left `available`. `StopDBCluster` now stops the cluster's shared container — preserving the container, volume, and data, exactly like Aurora keeps the cluster volume — and marks the cluster and every member `stopped`. `StartDBCluster` restarts the preserved container (recreating compute from the persistent named volume when the container is gone or unrestartable) and, like `CreateDBInstance`, returns immediately with a transitional status while a readiness worker flips the cluster and members to `available` once the database accepts authenticated connections. Invalid transitions return the AWS-exact `InvalidDBClusterStateFault` messages — including `CreateDBInstance`/`DeleteDBInstance` against a stopped cluster — and a warm boot keeps an intentionally stopped cluster stopped instead of reviving its compute. A start whose compute genuinely fails to come back lands the cluster back on `stopped` (retryable) instead of reporting a dead endpoint as `available`, and a stop that cannot stop the container surfaces an error instead of publishing a false `stopped`. Contributed by @kiran01bm.
- **RDS — Aurora PostgreSQL engine versions are validated at create time** — `CreateDBCluster` / `CreateDBInstance` accepted any `EngineVersion` string for `aurora-postgresql`: because the Docker image is derived from the major alone, a bogus minor like `16.99` silently ran PostgreSQL 16 while claiming to be `16.99`, and only an unknown major produced a visible image-pull failure. An unknown version — anything `DescribeDBEngineVersions` doesn't advertise and that isn't a dot-boundary prefix of an advertised version (bare majors like `16` stay accepted, since real AWS resolves them server-side) — now fails immediately with `InvalidParameterCombination` / `Cannot find version {version} for aurora-postgresql`, matching real AWS and the existing aurora-mysql behavior; `DescribeOrderableDBInstanceOptions` rejects unknown versions the same way instead of echoing a fabricated option. The advertised catalog is refreshed to the full creatable set real AWS returns (41 versions, 11.9 through 18.4, including `-limitless` variants, as of 2026-08-06) and is a single shared constant used by both the catalog and the validator; the default `aurora-postgresql` version is now `17.7` (AWS's current default) and the CloudFormation `AWS::RDS::DBCluster` provisioner uses that shared default instead of a hard-coded `15.4`. Contributed by @kiran01bm.
- **RDS — duplicate DB instance wire code matches AWS** — a duplicate `CreateDBInstance`, `CreateDBInstanceReadReplica`, or `RestoreDBInstanceFromDBSnapshot` target returned wire code `DBInstanceAlreadyExistsFault`, but real AWS omits the `Fault` suffix for instance-level error codes (cluster-level codes such as `DBClusterAlreadyExistsFault` keep it). SDKs match the exact string to produce their typed error — aws-sdk-go-v2, for example, deserialized the response as a generic API error instead of the typed `DBInstanceAlreadyExistsFault`. The wire code is now `DBInstanceAlreadyExists`. This deliberately reverts the v1.1.18 change, which mistook the SDK exception *shape name* for the *wire code* — the model's `error.code` field (`DBInstanceAlreadyExists`) is what SDKs match, and exact-string tests now pin the correct direction. Contributed by @kiran01bm.
### Added
diff --git a/ministack/services/rds.py b/ministack/services/rds.py
index 8476040e..2a314685 100644
--- a/ministack/services/rds.py
+++ b/ministack/services/rds.py
@@ -409,6 +409,16 @@ def restore_state(data):
for member in members:
member["DBInstanceStatus"] = "failed"
return
+ if cluster.get("Status") == "stopped":
+ # A stopped cluster stays stopped across host restarts;
+ # StartDBCluster brings compute back. (Real AWS also
+ # auto-starts a cluster stopped for seven days — out of
+ # scope for an emulator.)
+ cluster.pop("_shared_legacy_migration_in_progress", None)
+ cluster["_shared_container_ready"] = False
+ for member in members:
+ member["DBInstanceStatus"] = "stopped"
+ return
restore_epoch = int(cluster.get("_shared_container_epoch", 0))
# State from before cluster-owned storage has one container and
@@ -983,13 +993,19 @@ def _start_cluster_shared_container(cluster_id, cluster, remove_stale=False):
}
-def _stop_empty_cluster_shared_container(cluster_id, cluster):
- """Stop, but do not remove, an empty Aurora cluster's database.
+def _stop_cluster_shared_container(cluster_id, cluster):
+ """Stop, but do not remove, an Aurora cluster's database compute.
- Aurora keeps the cluster volume after its final DB instance is deleted,
- but no SQL endpoint is reachable until another instance is attached. A
- stopped Docker container models that split: the cluster still owns the
- container and its data, while only DeleteDBCluster removes either one.
+ Aurora keeps the cluster volume when compute goes away — after the final
+ DB instance is deleted, and while the cluster is stopped via
+ ``StopDBCluster`` — but no SQL endpoint is reachable until compute
+ returns. A stopped Docker container models that split: the cluster still
+ owns its data, and the cluster-owned volume is only removed by
+ DeleteDBCluster (the container itself may be replaced if a later
+ StartDBCluster has to fall back to recreating compute).
+
+ Returns True when compute is stopped — including when there was nothing
+ to stop — and False when a running container could not be stopped.
"""
with _shared_container_lock:
# Invalidate every readiness worker before the potentially slow Docker
@@ -1002,19 +1018,26 @@ def _stop_empty_cluster_shared_container(cluster_id, cluster):
container_id = cluster.get("_shared_container_id")
docker_client = _get_docker()
if not docker_client or not container_id:
- return
+ return True
try:
container = docker_client.containers.get(container_id)
+ except Exception:
+ # A container that no longer exists is not running compute; the
+ # goal state is already met.
+ return True
+ try:
container.reload()
if container.status not in ("created", "exited", "dead", "removing"):
container.stop(timeout=5)
- logger.info("RDS: stopped shared container for empty cluster %s", cluster_id)
+ logger.info("RDS: stopped shared container for cluster %s", cluster_id)
except Exception as e:
logger.warning(
- "RDS: failed to stop shared container for empty cluster %s: %s",
+ "RDS: failed to stop shared container for cluster %s: %s",
cluster_id,
e,
)
+ return False
+ return True
def _remove_cluster_shared_resources(
@@ -2465,6 +2488,17 @@ def _create_db_instance(p):
"migration is blocked.",
400,
)
+ if parent.get("Status") == "stopped":
+ # Real AWS rejects instance additions to a stopped cluster; the
+ # cluster must be started first. Without this guard the new
+ # member would land as ``creating`` with no readiness worker to
+ # ever complete it.
+ return _error(
+ "InvalidDBClusterStateFault",
+ f"DbCluster {parent['DBClusterIdentifier']} is in stopped "
+ "state but expected it to be one of available.",
+ 400,
+ )
cluster_id_param = parent["DBClusterIdentifier"]
engine = parent.get("Engine", engine)
engine_version = parent.get("EngineVersion", engine_version)
@@ -2973,6 +3007,25 @@ def _delete_db_instance(p):
return _error("InvalidParameterCombination",
"Cannot delete a DB instance when DeletionProtection is enabled.", 400)
+ parent_cluster_id = (
+ instance.get("_shared_cluster_id")
+ or instance.get("DBClusterIdentifier")
+ )
+ if parent_cluster_id:
+ parent = _resolve_cluster_in_request_region(parent_cluster_id)
+ if parent and parent.get("Status") == "stopped":
+ # Real AWS rejects member deletion from a stopped cluster; the
+ # cluster must be started first. Without this guard, deleting
+ # the last member of a stopped cluster would let a later
+ # CreateDBInstance restart compute behind the ``stopped``
+ # status.
+ return _error(
+ "InvalidDBClusterStateFault",
+ f"DbCluster {parent['DBClusterIdentifier']} is in stopped "
+ "state but expected it to be one of available.",
+ 400,
+ )
+
_unregister_instance_from_clusters(instance_id)
shared_cluster_id = (
@@ -2989,7 +3042,7 @@ def _delete_db_instance(p):
# secondary is detached, deleted, or gains compute again.
cluster["_mysql_headless_applier_required"] = True
else:
- _stop_empty_cluster_shared_container(shared_cluster_id, cluster)
+ _stop_cluster_shared_container(shared_cluster_id, cluster)
docker_client = _get_docker()
if (
@@ -4390,6 +4443,18 @@ def _modify_subnet_group(p):
# StartDBCluster / StopDBCluster
# ---------------------------------------------------------------------------
+def _cluster_member_instances(cluster):
+ """Resolve a cluster's member records to their instance dicts."""
+ return [
+ inst
+ for inst in (
+ _instances.get(member.get("DBInstanceIdentifier"))
+ for member in cluster.get("DBClusterMembers", [])
+ )
+ if inst is not None
+ ]
+
+
def _start_db_cluster(p):
cluster_id = _p(p, "DBClusterIdentifier")
cluster = _resolve_cluster_in_request_region(cluster_id)
@@ -4398,7 +4463,215 @@ def _start_db_cluster(p):
if wrong_region:
return wrong_region
return _error("DBClusterNotFoundFault", f"DBCluster {cluster_id} not found.", 404)
- cluster["Status"] = "available"
+ # The request may identify the cluster by ARN; every downstream consumer
+ # (the readiness worker's re-lookup, Docker container naming) needs the
+ # canonical identifier.
+ cluster_id = cluster["DBClusterIdentifier"]
+ status = cluster.get("Status")
+ if status != "stopped":
+ # Real AWS message shape, verified against live transcripts:
+ # "DbCluster <id> is in <state> state but expected it to be one of
+ # stopped,inaccessible-encryption-credentials-recoverable."
+ return _error(
+ "InvalidDBClusterStateFault",
+ f"DbCluster {cluster_id} is "
+ f"in {status} state but expected it to be one of "
+ "stopped,inaccessible-encryption-credentials-recoverable.",
+ 400,
+ )
+
+ member_instances = _cluster_member_instances(cluster)
+ docker_client = _get_docker()
+ had_compute = bool(cluster.get("_shared_container_id"))
+ if not docker_client or not (
+ had_compute or cluster.get("_shared_storage_initialized")
+ ):
+ # Control-plane only, or the cluster never had real compute:
+ # flipping the statuses is the entire start.
+ cluster["_shared_container_ready"] = True
+ cluster["Status"] = "available"
+ for inst in member_instances:
+ inst["DBInstanceStatus"] = "available"
+ return _xml(200, "StartDBClusterResponse",
+ f"<StartDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StartDBClusterResult>")
+
+ if had_compute:
+ start_result = _restart_cluster_shared_container(cluster_id, cluster)
+ if start_result.get("failed"):
+ # A preserved container can become unrestartable (for example,
+ # its old host port may have been claimed). Recreate only the
+ # compute layer; the cluster-owned named volume retains data.
+ start_result = _start_cluster_shared_container(
+ cluster_id, cluster, remove_stale=True,
+ )
+ else:
+ # Initialized storage without a container id happens after a host
+ # restart: persistence drops the container id but the stable-named
+ # container and volume survive. Recreate compute from them.
+ start_result = _start_cluster_shared_container(
+ cluster_id, cluster, remove_stale=True,
+ )
+
+ if not start_result.get("started"):
+ if start_result.get("failed"):
+ # Both the restart and the recreate fallback genuinely failed
+ # (Docker error, missing image, port exhaustion). Compute did
+ # not come back: keep the cluster stopped so StartDBCluster can
+ # be retried, and surface the failure instead of reporting a
+ # dead endpoint as available. The cluster-owned volume is
+ # untouched, so data survives for the retry.
+ cluster["Status"] = "stopped"
+ for inst in member_instances:
+ inst["DBInstanceStatus"] = "stopped"
+ return _error(
+ "InternalFailure",
+ f"Failed to start compute for DB cluster {cluster_id}.",
+ 500,
+ )
+ # Benign control-plane-only outcome: no Docker in this environment.
+ cluster["_shared_container_ready"] = True
+ cluster["Status"] = "available"
+ for inst in member_instances:
+ inst["DBInstanceStatus"] = "available"
+ return _xml(200, "StartDBClusterResponse",
+ f"<StartDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StartDBClusterResult>")
+
+ cluster["Status"] = "starting"
+ for inst in member_instances:
+ inst["DBInstanceStatus"] = "starting"
+ _sync_cluster_endpoints(cluster)
+
+ # Real AWS StartDBCluster returns immediately with a transitional status
+ # and the caller polls until the cluster is available. Finalize readiness
+ # on a daemon thread, exactly like CreateDBInstance does for first-member
+ # compute. contextvars.copy_context() carries the request's account and
+ # region into the daemon.
+ ctx = contextvars.copy_context()
+
+ def _bg_start_ready(
+ cluster_id=cluster_id,
+ container_id=cluster.get("_shared_container_id"),
+ container_epoch=start_result.get("container_epoch"),
+ ready_host=start_result.get("readiness_host"),
+ ready_port=start_result.get("readiness_port"),
+ ):
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ return
+
+ def _container_alive():
+ client = _get_docker()
+ if not client or not container_id:
+ return True
+ try:
+ container = client.containers.get(container_id)
+ container.reload()
+ return container.status not in ("exited", "dead", "removing")
+ except Exception:
+ return False
+
+ engine = cluster.get("Engine", "aurora-postgresql")
+ master_user = cluster.get("MasterUsername", "admin")
+ master_pass = cluster.get("_MasterUserPassword", "password")
+ readiness_user = master_user
+ readiness_pass = master_pass
+ readiness_db = cluster.get("DatabaseName") or "mydb"
+ pending_rotation = cluster.get("_pending_master_password_rotation")
+ if pending_rotation:
+ readiness_pass = pending_rotation["old_password"]
+ if _mysql_replication_secondary(cluster):
+ readiness_user = "root"
+ if cluster.get("_mysql_control_user_ready"):
+ readiness_user = _MYSQL_CONTROL_USER
+ readiness_pass = _MYSQL_CONTROL_PASSWORD
+ readiness_db = None
+ database_ready = _wait_for_database_ready(
+ ready_host, ready_port, engine, readiness_user,
+ readiness_pass, readiness_db, _container_alive,
+ )
+ with _shared_container_lock:
+ if (
+ cluster.get("_shared_container_epoch") != container_epoch
+ or cluster.get("_shared_container_id") != container_id
+ ):
+ logger.info(
+ "RDS: ignoring stale start readiness result for cluster "
+ "%s epoch %s container %s",
+ cluster_id, container_epoch, container_id,
+ )
+ return
+
+ # Real AWS lands a failed start back on ``stopped`` — never on a
+ # transitional status — so StartDBCluster can simply be retried.
+ def _fail_start():
+ # Stop the failed container first (a failed password rotation
+ # leaves it running and reachable; for an already-exited
+ # container this is a no-op) so the ``stopped`` status is
+ # honest the moment it becomes visible and a retried
+ # StartDBCluster begins from cleanly stopped compute. Holding
+ # _shared_container_lock across the Docker stop matches
+ # _stop_cluster_shared_container and keeps a concurrent retry
+ # from racing the stop.
+ client = _get_docker()
+ if client and container_id:
+ try:
+ failed = client.containers.get(container_id)
+ failed.reload()
+ if failed.status not in (
+ "created", "exited", "dead", "removing",
+ ):
+ failed.stop(timeout=5)
+ except Exception:
+ pass
+ cluster["_shared_container_ready"] = False
+ cluster["Status"] = "stopped"
+ for inst in _cluster_member_instances(cluster):
+ inst["DBInstanceStatus"] = "stopped"
+
+ start_failed = False
+ pending_rotation = cluster.get("_pending_master_password_rotation")
+ if not database_ready:
+ logger.warning(
+ "RDS: cluster %s container exited before becoming "
+ "reachable after StartDBCluster", cluster_id,
+ )
+ _fail_start()
+ start_failed = True
+ elif pending_rotation and not _rotate_real_password(
+ cluster,
+ pending_rotation["old_password"],
+ pending_rotation["new_password"],
+ ):
+ _fail_start()
+ start_failed = True
+ if not start_failed:
+ if pending_rotation:
+ cluster.pop("_pending_master_password_rotation", None)
+ _sync_global_mysql_credentials(cluster)
+ if _is_mysql_engine(engine) and not _mysql_replication_secondary(
+ cluster,
+ ):
+ _grant_mysql_master_user_privileges(
+ ready_host, ready_port, master_user,
+ cluster.get("_MasterUserPassword", master_pass),
+ cluster_id,
+ )
+ cluster["_shared_container_ready"] = True
+ for inst in _cluster_member_instances(cluster):
+ _attach_instance_to_shared_cluster(inst, cluster)
+ inst["DBInstanceStatus"] = "available"
+ _sync_cluster_endpoints(cluster)
+ _refresh_cluster_status(cluster_id)
+ if start_failed:
+ logger.warning(
+ "RDS: cluster %s failed to start; returned to stopped",
+ cluster_id,
+ )
+ return
+ logger.info("RDS: cluster %s started and ready", cluster_id)
+
+ threading.Thread(target=ctx.run, args=(_bg_start_ready,), daemon=True).start()
+
return _xml(200, "StartDBClusterResponse",
f"<StartDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StartDBClusterResult>")
@@ -4411,7 +4684,34 @@ def _stop_db_cluster(p):
if wrong_region:
return wrong_region
return _error("DBClusterNotFoundFault", f"DBCluster {cluster_id} not found.", 404)
+ status = cluster.get("Status")
+ if status != "available":
+ # Real AWS message shape, verified against live transcripts:
+ # "DbCluster <id> is in <state> state but expected it to be one of
+ # available."
+ return _error(
+ "InvalidDBClusterStateFault",
+ f"DbCluster {cluster.get('DBClusterIdentifier', cluster_id)} is "
+ f"in {status} state but expected it to be one of available.",
+ 400,
+ )
+ if not _stop_cluster_shared_container(
+ cluster.get("DBClusterIdentifier", cluster_id), cluster,
+ ):
+ # Compute is still running; publishing ``stopped`` here would be the
+ # reachable-endpoint lie this state machine exists to prevent.
+ # Restore readiness (the container is still serving) and surface the
+ # failure so StopDBCluster can be retried.
+ cluster["_shared_container_ready"] = True
+ return _error(
+ "InternalFailure",
+ "Failed to stop compute for DB cluster "
+ f"{cluster.get('DBClusterIdentifier', cluster_id)}.",
+ 500,
+ )
cluster["Status"] = "stopped"
+ for inst in _cluster_member_instances(cluster):
+ inst["DBInstanceStatus"] = "stopped"
return _xml(200, "StopDBClusterResponse",
f"<StopDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StopDBClusterResult>")
@@ -4785,7 +5085,7 @@ def _remove_from_global_cluster(p):
if not member.get("IsWriter"):
_clear_mysql_replication_metadata(cluster)
if not cluster.get("DBClusterMembers"):
- _stop_empty_cluster_shared_container(
+ _stop_cluster_shared_container(
cluster.get("DBClusterIdentifier", db_cluster_id),
cluster,
)
diff --git a/tests/test_rds.py b/tests/test_rds.py
index da5b0c0b..76e87b50 100644
--- a/tests/test_rds.py
+++ b/tests/test_rds.py
@@ -496,9 +496,23 @@ def test_rds_start_stop_cluster(rds):
MasterUsername="admin",
MasterUserPassword="password123",
)
+ with pytest.raises(ClientError) as exc_info:
+ rds.start_db_cluster(DBClusterIdentifier="ss-cl")
+ assert exc_info.value.response["Error"]["Code"] == "InvalidDBClusterStateFault"
+ assert exc_info.value.response["Error"]["Message"] == (
+ "DbCluster ss-cl is in available state but expected it to be one of "
+ "stopped,inaccessible-encryption-credentials-recoverable."
+ )
rds.stop_db_cluster(DBClusterIdentifier="ss-cl")
resp = rds.describe_db_clusters(DBClusterIdentifier="ss-cl")
assert resp["DBClusters"][0]["Status"] == "stopped"
+ with pytest.raises(ClientError) as exc_info:
+ rds.stop_db_cluster(DBClusterIdentifier="ss-cl")
+ assert exc_info.value.response["Error"]["Code"] == "InvalidDBClusterStateFault"
+ assert exc_info.value.response["Error"]["Message"] == (
+ "DbCluster ss-cl is in stopped state but expected it to be one of "
+ "available."
+ )
rds.start_db_cluster(DBClusterIdentifier="ss-cl")
resp2 = rds.describe_db_clusters(DBClusterIdentifier="ss-cl")
assert resp2["DBClusters"][0]["Status"] == "available"
@@ -2083,14 +2097,14 @@ def test_rds_last_member_delete_wins_after_readiness_epoch_check(monkeypatch):
"SkipFinalSnapshot": "true",
})
- original_stop = m._stop_empty_cluster_shared_container
+ original_stop = m._stop_cluster_shared_container
def _observed_stop(cluster_id, cluster):
stop_entered.set()
return original_stop(cluster_id, cluster)
monkeypatch.setattr(
- m, "_stop_empty_cluster_shared_container", _observed_stop,
+ m, "_stop_cluster_shared_container", _observed_stop,
)
m._create_db_instance({
"DBInstanceIdentifier": "readiness-race-replacement",
@@ -2125,6 +2139,509 @@ def test_rds_last_member_delete_wins_after_readiness_epoch_check(monkeypatch):
m._clusters.clear()
+def test_rds_stop_start_cluster_compute_lifecycle(monkeypatch):
+ """StopDBCluster stops the shared container; StartDBCluster restarts it."""
+ from ministack.services import rds as m
+
+ class FakeContainer:
+ id = "stopstart-shared-container"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def __init__(self):
+ self.status = "running"
+
+ def reload(self):
+ pass
+
+ def start(self):
+ self.status = "running"
+
+ def stop(self, timeout=5):
+ self.status = "exited"
+
+ container = FakeContainer()
+
+ class FakeContainers:
+ def run(self, **_kwargs):
+ container.status = "running"
+ return container
+
+ def get(self, identifier):
+ if identifier in (
+ container.id,
+ m._rds_cluster_docker_name("stopstart-cluster"),
+ ):
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16070)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+ monkeypatch.setattr(
+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,
+ )
+
+ def _wait_for_member_status(db_id, expected, timeout=2):
+ deadline = time.time() + timeout
+ while time.time() < deadline:
+ if m._instances.get(db_id, {}).get("DBInstanceStatus") == expected:
+ return
+ time.sleep(0.01)
+ pytest.fail(
+ f"instance {db_id} never reached {expected}; last status: "
+ f"{m._instances.get(db_id, {}).get('DBInstanceStatus')}",
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "stopstart-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "stopstart-writer",
+ "DBClusterIdentifier": "stopstart-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ _wait_for_member_status("stopstart-writer", "available")
+ cluster = m._clusters.get("stopstart-cluster")
+ assert cluster["Status"] == "available"
+
+ m._stop_db_cluster({"DBClusterIdentifier": "stopstart-cluster"})
+ assert container.status == "exited"
+ assert cluster["Status"] == "stopped"
+ assert cluster["_shared_container_ready"] is False
+ assert m._instances.get("stopstart-writer")[
+ "DBInstanceStatus"
+ ] == "stopped"
+
+ m._start_db_cluster({"DBClusterIdentifier": "stopstart-cluster"})
+ assert container.status == "running"
+ _wait_for_member_status("stopstart-writer", "available")
+ # The worker flips members available before its final
+ # _refresh_cluster_status, so poll the cluster status too.
+ deadline = time.time() + 2
+ while time.time() < deadline and cluster["Status"] != "available":
+ time.sleep(0.01)
+ assert cluster["Status"] == "available"
+ assert cluster["_shared_container_ready"] is True
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_restore_keeps_stopped_cluster_compute_stopped(monkeypatch):
+ """Warm boot must not revive compute for a cluster stopped via the API."""
+ from ministack.services import rds as m
+
+ started_calls = []
+
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+ monkeypatch.setattr(
+ m,
+ "_start_cluster_shared_container",
+ lambda *args, **kwargs: started_calls.append(args)
+ or {"started": False, "failed": False},
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "stopped-restore-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "stopped-restore-writer",
+ "DBClusterIdentifier": "stopped-restore-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ m._stop_db_cluster({"DBClusterIdentifier": "stopped-restore-cluster"})
+
+ persisted = m.get_state()
+ m._instances.clear()
+ m._clusters.clear()
+ started_calls.clear()
+ m.restore_state(persisted)
+
+ # restore_state marks every instance ``creating`` before spawning the
+ # cluster runner; the runner's stopped branch flips members back to
+ # ``stopped``. Polling for that transition observes runner completion
+ # without a fixed sleep, and the runner sets statuses before any
+ # start call could happen, so the started_calls assertion below
+ # cannot race it.
+ deadline = time.time() + 5
+ while time.time() < deadline:
+ inst = m._instances.get("stopped-restore-writer")
+ if inst and inst.get("DBInstanceStatus") == "stopped":
+ break
+ time.sleep(0.02)
+ cluster = m._clusters.get("stopped-restore-cluster")
+ assert cluster["Status"] == "stopped"
+ assert cluster["_shared_container_ready"] is False
+ assert m._instances.get("stopped-restore-writer")[
+ "DBInstanceStatus"
+ ] == "stopped"
+ assert started_calls == []
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def _stop_start_fake_docker(cluster_docker_name):
+ """Fake Docker client with switchable failures for stop/start tests."""
+ state = {"fail_start": False, "fail_run": False, "fail_stop": False}
+
+ class FakeContainer:
+ id = f"{cluster_docker_name}-container"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def __init__(self):
+ self.status = "running"
+
+ def reload(self):
+ pass
+
+ def start(self):
+ if state["fail_start"]:
+ raise Exception("injected docker start failure")
+ self.status = "running"
+
+ def stop(self, timeout=5):
+ if state["fail_stop"]:
+ raise Exception("injected docker stop failure")
+ self.status = "exited"
+
+ container = FakeContainer()
+
+ class FakeContainers:
+ def run(self, **_kwargs):
+ if state["fail_run"]:
+ raise Exception("injected docker run failure")
+ container.status = "running"
+ return container
+
+ def get(self, identifier):
+ if identifier in (container.id, cluster_docker_name):
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ return FakeDocker(), container, state
+
+
+def _poll_until(predicate, timeout=5):
+ deadline = time.time() + timeout
+ while time.time() < deadline:
+ if predicate():
+ return True
+ time.sleep(0.01)
+ return predicate()
+
+
+def test_rds_start_db_cluster_accepts_cluster_arn(monkeypatch):
+ """StartDBCluster by ARN must reach available, not wedge in starting."""
+ from ministack.services import rds as m
+
+ docker, container, _state = _stop_start_fake_docker(
+ m._rds_cluster_docker_name("arn-start-cluster"),
+ )
+ monkeypatch.setattr(m, "_get_docker", lambda: docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16072)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+ monkeypatch.setattr(
+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "arn-start-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "arn-start-writer",
+ "DBClusterIdentifier": "arn-start-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ cluster = m._clusters.get("arn-start-cluster")
+ assert _poll_until(lambda: cluster["Status"] == "available")
+
+ m._stop_db_cluster({"DBClusterIdentifier": "arn-start-cluster"})
+ assert cluster["Status"] == "stopped"
+
+ status, _, _body = m._start_db_cluster({
+ "DBClusterIdentifier": cluster["DBClusterArn"],
+ })
+ assert status == 200
+ assert _poll_until(lambda: cluster["Status"] == "available")
+ assert cluster["_shared_container_ready"] is True
+ assert container.status == "running"
+ assert m._instances.get("arn-start-writer")[
+ "DBInstanceStatus"
+ ] == "available"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_start_db_cluster_failure_stays_stopped_and_retryable(monkeypatch):
+ """A failed restart AND recreate surfaces an error, not 'available'."""
+ from ministack.services import rds as m
+
+ docker, container, state = _stop_start_fake_docker(
+ m._rds_cluster_docker_name("failed-start-cluster"),
+ )
+ monkeypatch.setattr(m, "_get_docker", lambda: docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16073)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+ monkeypatch.setattr(
+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "failed-start-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "failed-start-writer",
+ "DBClusterIdentifier": "failed-start-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ cluster = m._clusters.get("failed-start-cluster")
+ assert _poll_until(lambda: cluster["Status"] == "available")
+ m._stop_db_cluster({"DBClusterIdentifier": "failed-start-cluster"})
+
+ # Both the restart and the recreate fallback genuinely fail.
+ state["fail_start"] = True
+ state["fail_run"] = True
+ status, _, body = m._start_db_cluster({
+ "DBClusterIdentifier": "failed-start-cluster",
+ })
+ assert status == 500
+ assert b"InternalFailure" in body
+ assert cluster["Status"] == "stopped"
+ assert cluster["_shared_container_ready"] is False
+ assert m._instances.get("failed-start-writer")[
+ "DBInstanceStatus"
+ ] == "stopped"
+
+ # The failure is transient: a retry must succeed.
+ state["fail_start"] = False
+ state["fail_run"] = False
+ status, _, _body = m._start_db_cluster({
+ "DBClusterIdentifier": "failed-start-cluster",
+ })
+ assert status == 200
+ assert _poll_until(lambda: cluster["Status"] == "available")
+ assert cluster["_shared_container_ready"] is True
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_start_readiness_failure_returns_cluster_to_stopped(monkeypatch):
+ """A container that never becomes reachable lands back on stopped."""
+ from ministack.services import rds as m
+
+ docker, container, _state = _stop_start_fake_docker(
+ m._rds_cluster_docker_name("unready-start-cluster"),
+ )
+ ready_ok = [True]
+ monkeypatch.setattr(m, "_get_docker", lambda: docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16074)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(
+ m, "_wait_for_database_ready", lambda *_args: ready_ok[0],
+ )
+ monkeypatch.setattr(
+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "unready-start-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "unready-start-writer",
+ "DBClusterIdentifier": "unready-start-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ cluster = m._clusters.get("unready-start-cluster")
+ assert _poll_until(lambda: cluster["Status"] == "available")
+ m._stop_db_cluster({"DBClusterIdentifier": "unready-start-cluster"})
+
+ # The container starts but the database never answers.
+ ready_ok[0] = False
+ status, _, _body = m._start_db_cluster({
+ "DBClusterIdentifier": "unready-start-cluster",
+ })
+ assert status == 200
+ assert _poll_until(lambda: cluster["Status"] == "stopped")
+ assert cluster["_shared_container_ready"] is False
+ assert container.status == "exited"
+ assert m._instances.get("unready-start-writer")[
+ "DBInstanceStatus"
+ ] == "stopped"
+
+ # Once the database can answer, a retried start succeeds.
+ ready_ok[0] = True
+ status, _, _body = m._start_db_cluster({
+ "DBClusterIdentifier": "unready-start-cluster",
+ })
+ assert status == 200
+ assert _poll_until(lambda: cluster["Status"] == "available")
+ assert cluster["_shared_container_ready"] is True
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_instance_ops_rejected_while_cluster_stopped(monkeypatch):
+ """Create/DeleteDBInstance against a stopped cluster must be rejected."""
+ from ministack.services import rds as m
+
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "stopped-guard-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "stopped-guard-writer",
+ "DBClusterIdentifier": "stopped-guard-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ cluster = m._clusters.get("stopped-guard-cluster")
+ assert _poll_until(lambda: cluster["Status"] == "available")
+ m._stop_db_cluster({"DBClusterIdentifier": "stopped-guard-cluster"})
+ assert cluster["Status"] == "stopped"
+
+ status, _, body = m._create_db_instance({
+ "DBInstanceIdentifier": "stopped-guard-reader",
+ "DBClusterIdentifier": "stopped-guard-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ assert status == 400
+ assert b"InvalidDBClusterStateFault" in body
+ assert m._instances.get("stopped-guard-reader") is None
+
+ status, _, body = m._delete_db_instance({
+ "DBInstanceIdentifier": "stopped-guard-writer",
+ })
+ assert status == 400
+ assert b"InvalidDBClusterStateFault" in body
+ assert m._instances.get("stopped-guard-writer") is not None
+ assert cluster.get("DBClusterMembers")
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_stop_db_cluster_docker_failure_surfaces_error(monkeypatch):
+ """A failed Docker stop must not report the cluster as stopped."""
+ from ministack.services import rds as m
+
+ docker, container, state = _stop_start_fake_docker(
+ m._rds_cluster_docker_name("failed-stop-cluster"),
+ )
+ monkeypatch.setattr(m, "_get_docker", lambda: docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16075)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+ monkeypatch.setattr(
+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "failed-stop-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "failed-stop-writer",
+ "DBClusterIdentifier": "failed-stop-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ cluster = m._clusters.get("failed-stop-cluster")
+ assert _poll_until(lambda: cluster["Status"] == "available")
+
+ state["fail_stop"] = True
+ status, _, body = m._stop_db_cluster({
+ "DBClusterIdentifier": "failed-stop-cluster",
+ })
+ assert status == 500
+ assert b"InternalFailure" in body
+ assert cluster["Status"] == "available"
+ assert cluster["_shared_container_ready"] is True
+ assert container.status == "running"
+ assert m._instances.get("failed-stop-writer")[
+ "DBInstanceStatus"
+ ] == "available"
+
+ # The failure is transient: a retried stop succeeds.
+ state["fail_stop"] = False
+ status, _, _body = m._stop_db_cluster({
+ "DBClusterIdentifier": "failed-stop-cluster",
+ })
+ assert status == 200
+ assert cluster["Status"] == "stopped"
+ assert container.status == "exited"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
def test_rds_stale_readiness_ignores_same_id_cluster_recreation(monkeypatch):
"""Epoch reuse cannot let an old container worker mutate a new cluster."""
from ministack.services import rds as m
@@ -3681,7 +4198,7 @@ def test_rds_restore_state_respawns_one_container_per_cluster(
del m._instances[db_id]
m._unregister_instance_from_clusters(db_id)
restored_cluster = m._clusters["restored-shared-cluster"]
- m._stop_empty_cluster_shared_container(
+ m._stop_cluster_shared_container(
"restored-shared-cluster", restored_cluster,
)
return database_ready
@@ -3813,7 +4330,7 @@ def test_rds_restore_state_respawns_one_container_per_cluster(
del m._instances[db_id]
m._unregister_instance_from_clusters(db_id)
restored_cluster = m._clusters[cluster_id]
- m._stop_empty_cluster_shared_container(
+ m._stop_cluster_shared_container(
cluster_id,
restored_cluster,
)
@@ -6430,6 +6947,63 @@ def test_aurora_delete_member_keeps_shared_data(rds, rds_data):
assert containers == []
[email protected](not os.environ.get("DOCKER_NETWORK"), reason="DOCKER_NETWORK not set -- live Aurora")
+def test_aurora_stop_start_cluster_preserves_data(rds):
+ import docker
+ import pymysql
+
+ with _live_cluster(rds) as (
+ cluster_id,
+ writer_id,
+ reader_id,
+ writer,
+ _reader,
+ _cluster,
+ ):
+ with _aurora_connect(writer["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("CREATE TABLE stop_start_rows (id INT PRIMARY KEY)")
+ cursor.execute("INSERT INTO stop_start_rows VALUES (42)")
+
+ containers = docker.from_env().containers.list(
+ all=True,
+ filters={"label": ["ministack=rds", f"cluster_id={cluster_id}"]},
+ )
+ assert len(containers) == 1
+ container = containers[0]
+ original_container_id = container.id
+
+ rds.stop_db_cluster(DBClusterIdentifier=cluster_id)
+ stopped = rds.describe_db_clusters(
+ DBClusterIdentifier=cluster_id,
+ )["DBClusters"][0]
+ assert stopped["Status"] == "stopped"
+ for db_id in (writer_id, reader_id):
+ instance = rds.describe_db_instances(
+ DBInstanceIdentifier=db_id,
+ )["DBInstances"][0]
+ assert instance["DBInstanceStatus"] == "stopped"
+ container.reload()
+ assert container.status == "exited"
+ with pytest.raises((pymysql.err.OperationalError, OSError)):
+ _aurora_connect(writer["Endpoint"])
+
+ rds.start_db_cluster(DBClusterIdentifier=cluster_id)
+ writer = _wait_for_instance(rds, writer_id)
+ _wait_for_instance(rds, reader_id)
+ started = rds.describe_db_clusters(
+ DBClusterIdentifier=cluster_id,
+ )["DBClusters"][0]
+ assert started["Status"] == "available"
+ container.reload()
+ assert container.status == "running"
+ assert container.id == original_container_id
+ with _aurora_connect(writer["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("SELECT id FROM stop_start_rows")
+ assert cursor.fetchone() == (42,)
+
+
def test_rds_aurora_mysql_8_replication_command_is_stable_and_scoped(monkeypatch):
from ministack.core.responses import (
get_account_id,
@@ -7201,7 +7775,7 @@ def test_rds_deleting_last_global_secondary_instance_preserves_headless_applier(
set_request_region("us-west-2")
monkeypatch.setattr(
m,
- "_stop_empty_cluster_shared_container",
+ "_stop_cluster_shared_container",
lambda cluster_id, cluster: stopped.append((cluster_id, cluster)),
)
@@ -7661,7 +8235,7 @@ def test_rds_remove_headless_global_secondary_commits_detach_and_stops_applier(
monkeypatch.setattr(m, "_detach_mysql_replication", lambda *_args: True)
monkeypatch.setattr(
m,
- "_stop_empty_cluster_shared_container",
+ "_stop_cluster_shared_container",
lambda cluster_id, cluster: stopped.append((cluster_id, cluster)),
)