Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(rds): make StopDBCluster/StartDBCluster stop and start Aurora compute (#1301)
kiran01bme364d2ed3f3b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9f0987e0..5873b185 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]### Fixed+- **RDS — `StopDBCluster` / `StartDBCluster` now stop and start Aurora compute** — both operations only flipped metadata: a "stopped" cluster's backing container kept running and accepting SQL connections, and its members never left `available`. `StopDBCluster` now stops the cluster's shared container — preserving the container, volume, and data, exactly like Aurora keeps the cluster volume — and marks the cluster and every member `stopped`. `StartDBCluster` restarts the preserved container (recreating compute from the persistent named volume when the container is gone or unrestartable) and, like `CreateDBInstance`, returns immediately with a transitional status while a readiness worker flips the cluster and members to `available` once the database accepts authenticated connections. Invalid transitions return the AWS-exact `InvalidDBClusterStateFault` messages — including `CreateDBInstance`/`DeleteDBInstance` against a stopped cluster — and a warm boot keeps an intentionally stopped cluster stopped instead of reviving its compute. A start whose compute genuinely fails to come back lands the cluster back on `stopped` (retryable) instead of reporting a dead endpoint as `available`, and a stop that cannot stop the container surfaces an error instead of publishing a false `stopped`. Contributed by @kiran01bm.- **RDS — Aurora PostgreSQL engine versions are validated at create time** — `CreateDBCluster` / `CreateDBInstance` accepted any `EngineVersion` string for `aurora-postgresql`: because the Docker image is derived from the major alone, a bogus minor like `16.99` silently ran PostgreSQL 16 while claiming to be `16.99`, and only an unknown major produced a visible image-pull failure. An unknown version — anything `DescribeDBEngineVersions` doesn't advertise and that isn't a dot-boundary prefix of an advertised version (bare majors like `16` stay accepted, since real AWS resolves them server-side) — now fails immediately with `InvalidParameterCombination` / `Cannot find version {version} for aurora-postgresql`, matching real AWS and the existing aurora-mysql behavior; `DescribeOrderableDBInstanceOptions` rejects unknown versions the same way instead of echoing a fabricated option. The advertised catalog is refreshed to the full creatable set real AWS returns (41 versions, 11.9 through 18.4, including `-limitless` variants, as of 2026-08-06) and is a single shared constant used by both the catalog and the validator; the default `aurora-postgresql` version is now `17.7` (AWS's current default) and the CloudFormation `AWS::RDS::DBCluster` provisioner uses that shared default instead of a hard-coded `15.4`. Contributed by @kiran01bm.- **RDS — duplicate DB instance wire code matches AWS** — a duplicate `CreateDBInstance`, `CreateDBInstanceReadReplica`, or `RestoreDBInstanceFromDBSnapshot` target returned wire code `DBInstanceAlreadyExistsFault`, but real AWS omits the `Fault` suffix for instance-level error codes (cluster-level codes such as `DBClusterAlreadyExistsFault` keep it). SDKs match the exact string to produce their typed error — aws-sdk-go-v2, for example, deserialized the response as a generic API error instead of the typed `DBInstanceAlreadyExistsFault`. The wire code is now `DBInstanceAlreadyExists`. This deliberately reverts the v1.1.18 change, which mistook the SDK exception *shape name* for the *wire code* — the model's `error.code` field (`DBInstanceAlreadyExists`) is what SDKs match, and exact-string tests now pin the correct direction. Contributed by @kiran01bm.### Addeddiff --git a/ministack/services/rds.py b/ministack/services/rds.pyindex 8476040e..2a314685 100644--- a/ministack/services/rds.py+++ b/ministack/services/rds.py@@ -409,6 +409,16 @@ def restore_state(data):for member in members:member["DBInstanceStatus"] = "failed"return+ if cluster.get("Status") == "stopped":+ # A stopped cluster stays stopped across host restarts;+ # StartDBCluster brings compute back. (Real AWS also+ # auto-starts a cluster stopped for seven days — out of+ # scope for an emulator.)+ cluster.pop("_shared_legacy_migration_in_progress", None)+ cluster["_shared_container_ready"] = False+ for member in members:+ member["DBInstanceStatus"] = "stopped"+ returnrestore_epoch = int(cluster.get("_shared_container_epoch", 0))# State from before cluster-owned storage has one container and@@ -983,13 +993,19 @@ def _start_cluster_shared_container(cluster_id, cluster, remove_stale=False):}-def _stop_empty_cluster_shared_container(cluster_id, cluster):- """Stop, but do not remove, an empty Aurora cluster's database.+def _stop_cluster_shared_container(cluster_id, cluster):+ """Stop, but do not remove, an Aurora cluster's database compute.- Aurora keeps the cluster volume after its final DB instance is deleted,- but no SQL endpoint is reachable until another instance is attached. A- stopped Docker container models that split: the cluster still owns the- container and its data, while only DeleteDBCluster removes either one.+ Aurora keeps the cluster volume when compute goes away — after the final+ DB instance is deleted, and while the cluster is stopped via+ ``StopDBCluster`` — but no SQL endpoint is reachable until compute+ returns. A stopped Docker container models that split: the cluster still+ owns its data, and the cluster-owned volume is only removed by+ DeleteDBCluster (the container itself may be replaced if a later+ StartDBCluster has to fall back to recreating compute).++ Returns True when compute is stopped — including when there was nothing+ to stop — and False when a running container could not be stopped."""with _shared_container_lock:# Invalidate every readiness worker before the potentially slow Docker@@ -1002,19 +1018,26 @@ def _stop_empty_cluster_shared_container(cluster_id, cluster):container_id = cluster.get("_shared_container_id")docker_client = _get_docker()if not docker_client or not container_id:- return+ return Truetry:container = docker_client.containers.get(container_id)+ except Exception:+ # A container that no longer exists is not running compute; the+ # goal state is already met.+ return True+ try:container.reload()if container.status not in ("created", "exited", "dead", "removing"):container.stop(timeout=5)- logger.info("RDS: stopped shared container for empty cluster %s", cluster_id)+ logger.info("RDS: stopped shared container for cluster %s", cluster_id)except Exception as e:logger.warning(- "RDS: failed to stop shared container for empty cluster %s: %s",+ "RDS: failed to stop shared container for cluster %s: %s",cluster_id,e,)+ return False+ return Truedef _remove_cluster_shared_resources(@@ -2465,6 +2488,17 @@ def _create_db_instance(p):"migration is blocked.",400,)+ if parent.get("Status") == "stopped":+ # Real AWS rejects instance additions to a stopped cluster; the+ # cluster must be started first. Without this guard the new+ # member would land as ``creating`` with no readiness worker to+ # ever complete it.+ return _error(+ "InvalidDBClusterStateFault",+ f"DbCluster {parent['DBClusterIdentifier']} is in stopped "+ "state but expected it to be one of available.",+ 400,+ )cluster_id_param = parent["DBClusterIdentifier"]engine = parent.get("Engine", engine)engine_version = parent.get("EngineVersion", engine_version)@@ -2973,6 +3007,25 @@ def _delete_db_instance(p):return _error("InvalidParameterCombination","Cannot delete a DB instance when DeletionProtection is enabled.", 400)+ parent_cluster_id = (+ instance.get("_shared_cluster_id")+ or instance.get("DBClusterIdentifier")+ )+ if parent_cluster_id:+ parent = _resolve_cluster_in_request_region(parent_cluster_id)+ if parent and parent.get("Status") == "stopped":+ # Real AWS rejects member deletion from a stopped cluster; the+ # cluster must be started first. Without this guard, deleting+ # the last member of a stopped cluster would let a later+ # CreateDBInstance restart compute behind the ``stopped``+ # status.+ return _error(+ "InvalidDBClusterStateFault",+ f"DbCluster {parent['DBClusterIdentifier']} is in stopped "+ "state but expected it to be one of available.",+ 400,+ )+_unregister_instance_from_clusters(instance_id)shared_cluster_id = (@@ -2989,7 +3042,7 @@ def _delete_db_instance(p):# secondary is detached, deleted, or gains compute again.cluster["_mysql_headless_applier_required"] = Trueelse:- _stop_empty_cluster_shared_container(shared_cluster_id, cluster)+ _stop_cluster_shared_container(shared_cluster_id, cluster)docker_client = _get_docker()if (@@ -4390,6 +4443,18 @@ def _modify_subnet_group(p):# StartDBCluster / StopDBCluster# ---------------------------------------------------------------------------+def _cluster_member_instances(cluster):+ """Resolve a cluster's member records to their instance dicts."""+ return [+ inst+ for inst in (+ _instances.get(member.get("DBInstanceIdentifier"))+ for member in cluster.get("DBClusterMembers", [])+ )+ if inst is not None+ ]++def _start_db_cluster(p):cluster_id = _p(p, "DBClusterIdentifier")cluster = _resolve_cluster_in_request_region(cluster_id)@@ -4398,7 +4463,215 @@ def _start_db_cluster(p):if wrong_region:return wrong_regionreturn _error("DBClusterNotFoundFault", f"DBCluster {cluster_id} not found.", 404)- cluster["Status"] = "available"+ # The request may identify the cluster by ARN; every downstream consumer+ # (the readiness worker's re-lookup, Docker container naming) needs the+ # canonical identifier.+ cluster_id = cluster["DBClusterIdentifier"]+ status = cluster.get("Status")+ if status != "stopped":+ # Real AWS message shape, verified against live transcripts:+ # "DbCluster <id> is in <state> state but expected it to be one of+ # stopped,inaccessible-encryption-credentials-recoverable."+ return _error(+ "InvalidDBClusterStateFault",+ f"DbCluster {cluster_id} is "+ f"in {status} state but expected it to be one of "+ "stopped,inaccessible-encryption-credentials-recoverable.",+ 400,+ )++ member_instances = _cluster_member_instances(cluster)+ docker_client = _get_docker()+ had_compute = bool(cluster.get("_shared_container_id"))+ if not docker_client or not (+ had_compute or cluster.get("_shared_storage_initialized")+ ):+ # Control-plane only, or the cluster never had real compute:+ # flipping the statuses is the entire start.+ cluster["_shared_container_ready"] = True+ cluster["Status"] = "available"+ for inst in member_instances:+ inst["DBInstanceStatus"] = "available"+ return _xml(200, "StartDBClusterResponse",+ f"<StartDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StartDBClusterResult>")++ if had_compute:+ start_result = _restart_cluster_shared_container(cluster_id, cluster)+ if start_result.get("failed"):+ # A preserved container can become unrestartable (for example,+ # its old host port may have been claimed). Recreate only the+ # compute layer; the cluster-owned named volume retains data.+ start_result = _start_cluster_shared_container(+ cluster_id, cluster, remove_stale=True,+ )+ else:+ # Initialized storage without a container id happens after a host+ # restart: persistence drops the container id but the stable-named+ # container and volume survive. Recreate compute from them.+ start_result = _start_cluster_shared_container(+ cluster_id, cluster, remove_stale=True,+ )++ if not start_result.get("started"):+ if start_result.get("failed"):+ # Both the restart and the recreate fallback genuinely failed+ # (Docker error, missing image, port exhaustion). Compute did+ # not come back: keep the cluster stopped so StartDBCluster can+ # be retried, and surface the failure instead of reporting a+ # dead endpoint as available. The cluster-owned volume is+ # untouched, so data survives for the retry.+ cluster["Status"] = "stopped"+ for inst in member_instances:+ inst["DBInstanceStatus"] = "stopped"+ return _error(+ "InternalFailure",+ f"Failed to start compute for DB cluster {cluster_id}.",+ 500,+ )+ # Benign control-plane-only outcome: no Docker in this environment.+ cluster["_shared_container_ready"] = True+ cluster["Status"] = "available"+ for inst in member_instances:+ inst["DBInstanceStatus"] = "available"+ return _xml(200, "StartDBClusterResponse",+ f"<StartDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StartDBClusterResult>")++ cluster["Status"] = "starting"+ for inst in member_instances:+ inst["DBInstanceStatus"] = "starting"+ _sync_cluster_endpoints(cluster)++ # Real AWS StartDBCluster returns immediately with a transitional status+ # and the caller polls until the cluster is available. Finalize readiness+ # on a daemon thread, exactly like CreateDBInstance does for first-member+ # compute. contextvars.copy_context() carries the request's account and+ # region into the daemon.+ ctx = contextvars.copy_context()++ def _bg_start_ready(+ cluster_id=cluster_id,+ container_id=cluster.get("_shared_container_id"),+ container_epoch=start_result.get("container_epoch"),+ ready_host=start_result.get("readiness_host"),+ ready_port=start_result.get("readiness_port"),+ ):+ cluster = _clusters.get(cluster_id)+ if not cluster:+ return++ def _container_alive():+ client = _get_docker()+ if not client or not container_id:+ return True+ try:+ container = client.containers.get(container_id)+ container.reload()+ return container.status not in ("exited", "dead", "removing")+ except Exception:+ return False++ engine = cluster.get("Engine", "aurora-postgresql")+ master_user = cluster.get("MasterUsername", "admin")+ master_pass = cluster.get("_MasterUserPassword", "password")+ readiness_user = master_user+ readiness_pass = master_pass+ readiness_db = cluster.get("DatabaseName") or "mydb"+ pending_rotation = cluster.get("_pending_master_password_rotation")+ if pending_rotation:+ readiness_pass = pending_rotation["old_password"]+ if _mysql_replication_secondary(cluster):+ readiness_user = "root"+ if cluster.get("_mysql_control_user_ready"):+ readiness_user = _MYSQL_CONTROL_USER+ readiness_pass = _MYSQL_CONTROL_PASSWORD+ readiness_db = None+ database_ready = _wait_for_database_ready(+ ready_host, ready_port, engine, readiness_user,+ readiness_pass, readiness_db, _container_alive,+ )+ with _shared_container_lock:+ if (+ cluster.get("_shared_container_epoch") != container_epoch+ or cluster.get("_shared_container_id") != container_id+ ):+ logger.info(+ "RDS: ignoring stale start readiness result for cluster "+ "%s epoch %s container %s",+ cluster_id, container_epoch, container_id,+ )+ return++ # Real AWS lands a failed start back on ``stopped`` — never on a+ # transitional status — so StartDBCluster can simply be retried.+ def _fail_start():+ # Stop the failed container first (a failed password rotation+ # leaves it running and reachable; for an already-exited+ # container this is a no-op) so the ``stopped`` status is+ # honest the moment it becomes visible and a retried+ # StartDBCluster begins from cleanly stopped compute. Holding+ # _shared_container_lock across the Docker stop matches+ # _stop_cluster_shared_container and keeps a concurrent retry+ # from racing the stop.+ client = _get_docker()+ if client and container_id:+ try:+ failed = client.containers.get(container_id)+ failed.reload()+ if failed.status not in (+ "created", "exited", "dead", "removing",+ ):+ failed.stop(timeout=5)+ except Exception:+ pass+ cluster["_shared_container_ready"] = False+ cluster["Status"] = "stopped"+ for inst in _cluster_member_instances(cluster):+ inst["DBInstanceStatus"] = "stopped"++ start_failed = False+ pending_rotation = cluster.get("_pending_master_password_rotation")+ if not database_ready:+ logger.warning(+ "RDS: cluster %s container exited before becoming "+ "reachable after StartDBCluster", cluster_id,+ )+ _fail_start()+ start_failed = True+ elif pending_rotation and not _rotate_real_password(+ cluster,+ pending_rotation["old_password"],+ pending_rotation["new_password"],+ ):+ _fail_start()+ start_failed = True+ if not start_failed:+ if pending_rotation:+ cluster.pop("_pending_master_password_rotation", None)+ _sync_global_mysql_credentials(cluster)+ if _is_mysql_engine(engine) and not _mysql_replication_secondary(+ cluster,+ ):+ _grant_mysql_master_user_privileges(+ ready_host, ready_port, master_user,+ cluster.get("_MasterUserPassword", master_pass),+ cluster_id,+ )+ cluster["_shared_container_ready"] = True+ for inst in _cluster_member_instances(cluster):+ _attach_instance_to_shared_cluster(inst, cluster)+ inst["DBInstanceStatus"] = "available"+ _sync_cluster_endpoints(cluster)+ _refresh_cluster_status(cluster_id)+ if start_failed:+ logger.warning(+ "RDS: cluster %s failed to start; returned to stopped",+ cluster_id,+ )+ return+ logger.info("RDS: cluster %s started and ready", cluster_id)++ threading.Thread(target=ctx.run, args=(_bg_start_ready,), daemon=True).start()+return _xml(200, "StartDBClusterResponse",f"<StartDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StartDBClusterResult>")@@ -4411,7 +4684,34 @@ def _stop_db_cluster(p):if wrong_region:return wrong_regionreturn _error("DBClusterNotFoundFault", f"DBCluster {cluster_id} not found.", 404)+ status = cluster.get("Status")+ if status != "available":+ # Real AWS message shape, verified against live transcripts:+ # "DbCluster <id> is in <state> state but expected it to be one of+ # available."+ return _error(+ "InvalidDBClusterStateFault",+ f"DbCluster {cluster.get('DBClusterIdentifier', cluster_id)} is "+ f"in {status} state but expected it to be one of available.",+ 400,+ )+ if not _stop_cluster_shared_container(+ cluster.get("DBClusterIdentifier", cluster_id), cluster,+ ):+ # Compute is still running; publishing ``stopped`` here would be the+ # reachable-endpoint lie this state machine exists to prevent.+ # Restore readiness (the container is still serving) and surface the+ # failure so StopDBCluster can be retried.+ cluster["_shared_container_ready"] = True+ return _error(+ "InternalFailure",+ "Failed to stop compute for DB cluster "+ f"{cluster.get('DBClusterIdentifier', cluster_id)}.",+ 500,+ )cluster["Status"] = "stopped"+ for inst in _cluster_member_instances(cluster):+ inst["DBInstanceStatus"] = "stopped"return _xml(200, "StopDBClusterResponse",f"<StopDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></StopDBClusterResult>")@@ -4785,7 +5085,7 @@ def _remove_from_global_cluster(p):if not member.get("IsWriter"):_clear_mysql_replication_metadata(cluster)if not cluster.get("DBClusterMembers"):- _stop_empty_cluster_shared_container(+ _stop_cluster_shared_container(cluster.get("DBClusterIdentifier", db_cluster_id),cluster,)diff --git a/tests/test_rds.py b/tests/test_rds.pyindex da5b0c0b..76e87b50 100644--- a/tests/test_rds.py+++ b/tests/test_rds.py@@ -496,9 +496,23 @@ def test_rds_start_stop_cluster(rds):MasterUsername="admin",MasterUserPassword="password123",)+ with pytest.raises(ClientError) as exc_info:+ rds.start_db_cluster(DBClusterIdentifier="ss-cl")+ assert exc_info.value.response["Error"]["Code"] == "InvalidDBClusterStateFault"+ assert exc_info.value.response["Error"]["Message"] == (+ "DbCluster ss-cl is in available state but expected it to be one of "+ "stopped,inaccessible-encryption-credentials-recoverable."+ )rds.stop_db_cluster(DBClusterIdentifier="ss-cl")resp = rds.describe_db_clusters(DBClusterIdentifier="ss-cl")assert resp["DBClusters"][0]["Status"] == "stopped"+ with pytest.raises(ClientError) as exc_info:+ rds.stop_db_cluster(DBClusterIdentifier="ss-cl")+ assert exc_info.value.response["Error"]["Code"] == "InvalidDBClusterStateFault"+ assert exc_info.value.response["Error"]["Message"] == (+ "DbCluster ss-cl is in stopped state but expected it to be one of "+ "available."+ )rds.start_db_cluster(DBClusterIdentifier="ss-cl")resp2 = rds.describe_db_clusters(DBClusterIdentifier="ss-cl")assert resp2["DBClusters"][0]["Status"] == "available"@@ -2083,14 +2097,14 @@ def test_rds_last_member_delete_wins_after_readiness_epoch_check(monkeypatch):"SkipFinalSnapshot": "true",})- original_stop = m._stop_empty_cluster_shared_container+ original_stop = m._stop_cluster_shared_containerdef _observed_stop(cluster_id, cluster):stop_entered.set()return original_stop(cluster_id, cluster)monkeypatch.setattr(- m, "_stop_empty_cluster_shared_container", _observed_stop,+ m, "_stop_cluster_shared_container", _observed_stop,)m._create_db_instance({"DBInstanceIdentifier": "readiness-race-replacement",@@ -2125,6 +2139,509 @@ def test_rds_last_member_delete_wins_after_readiness_epoch_check(monkeypatch):m._clusters.clear()+def test_rds_stop_start_cluster_compute_lifecycle(monkeypatch):+ """StopDBCluster stops the shared container; StartDBCluster restarts it."""+ from ministack.services import rds as m++ class FakeContainer:+ id = "stopstart-shared-container"+ attrs = {"NetworkSettings": {"Networks": {}}}++ def __init__(self):+ self.status = "running"++ def reload(self):+ pass++ def start(self):+ self.status = "running"++ def stop(self, timeout=5):+ self.status = "exited"++ container = FakeContainer()++ class FakeContainers:+ def run(self, **_kwargs):+ container.status = "running"+ return container++ def get(self, identifier):+ if identifier in (+ container.id,+ m._rds_cluster_docker_name("stopstart-cluster"),+ ):+ return container+ raise Exception("not found")++ class FakeDocker:+ def __init__(self):+ self.containers = FakeContainers()++ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)+ monkeypatch.setattr(m, "_next_port", lambda: 16070)+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)+ monkeypatch.setattr(+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,+ )++ def _wait_for_member_status(db_id, expected, timeout=2):+ deadline = time.time() + timeout+ while time.time() < deadline:+ if m._instances.get(db_id, {}).get("DBInstanceStatus") == expected:+ return+ time.sleep(0.01)+ pytest.fail(+ f"instance {db_id} never reached {expected}; last status: "+ f"{m._instances.get(db_id, {}).get('DBInstanceStatus')}",+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "stopstart-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "stopstart-writer",+ "DBClusterIdentifier": "stopstart-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ _wait_for_member_status("stopstart-writer", "available")+ cluster = m._clusters.get("stopstart-cluster")+ assert cluster["Status"] == "available"++ m._stop_db_cluster({"DBClusterIdentifier": "stopstart-cluster"})+ assert container.status == "exited"+ assert cluster["Status"] == "stopped"+ assert cluster["_shared_container_ready"] is False+ assert m._instances.get("stopstart-writer")[+ "DBInstanceStatus"+ ] == "stopped"++ m._start_db_cluster({"DBClusterIdentifier": "stopstart-cluster"})+ assert container.status == "running"+ _wait_for_member_status("stopstart-writer", "available")+ # The worker flips members available before its final+ # _refresh_cluster_status, so poll the cluster status too.+ deadline = time.time() + 2+ while time.time() < deadline and cluster["Status"] != "available":+ time.sleep(0.01)+ assert cluster["Status"] == "available"+ assert cluster["_shared_container_ready"] is True+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_restore_keeps_stopped_cluster_compute_stopped(monkeypatch):+ """Warm boot must not revive compute for a cluster stopped via the API."""+ from ministack.services import rds as m++ started_calls = []++ monkeypatch.setattr(m, "_get_docker", lambda: None)+ monkeypatch.setattr(+ m,+ "_start_cluster_shared_container",+ lambda *args, **kwargs: started_calls.append(args)+ or {"started": False, "failed": False},+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "stopped-restore-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "stopped-restore-writer",+ "DBClusterIdentifier": "stopped-restore-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ m._stop_db_cluster({"DBClusterIdentifier": "stopped-restore-cluster"})++ persisted = m.get_state()+ m._instances.clear()+ m._clusters.clear()+ started_calls.clear()+ m.restore_state(persisted)++ # restore_state marks every instance ``creating`` before spawning the+ # cluster runner; the runner's stopped branch flips members back to+ # ``stopped``. Polling for that transition observes runner completion+ # without a fixed sleep, and the runner sets statuses before any+ # start call could happen, so the started_calls assertion below+ # cannot race it.+ deadline = time.time() + 5+ while time.time() < deadline:+ inst = m._instances.get("stopped-restore-writer")+ if inst and inst.get("DBInstanceStatus") == "stopped":+ break+ time.sleep(0.02)+ cluster = m._clusters.get("stopped-restore-cluster")+ assert cluster["Status"] == "stopped"+ assert cluster["_shared_container_ready"] is False+ assert m._instances.get("stopped-restore-writer")[+ "DBInstanceStatus"+ ] == "stopped"+ assert started_calls == []+ finally:+ m._instances.clear()+ m._clusters.clear()+++def _stop_start_fake_docker(cluster_docker_name):+ """Fake Docker client with switchable failures for stop/start tests."""+ state = {"fail_start": False, "fail_run": False, "fail_stop": False}++ class FakeContainer:+ id = f"{cluster_docker_name}-container"+ attrs = {"NetworkSettings": {"Networks": {}}}++ def __init__(self):+ self.status = "running"++ def reload(self):+ pass++ def start(self):+ if state["fail_start"]:+ raise Exception("injected docker start failure")+ self.status = "running"++ def stop(self, timeout=5):+ if state["fail_stop"]:+ raise Exception("injected docker stop failure")+ self.status = "exited"++ container = FakeContainer()++ class FakeContainers:+ def run(self, **_kwargs):+ if state["fail_run"]:+ raise Exception("injected docker run failure")+ container.status = "running"+ return container++ def get(self, identifier):+ if identifier in (container.id, cluster_docker_name):+ return container+ raise Exception("not found")++ class FakeDocker:+ def __init__(self):+ self.containers = FakeContainers()++ return FakeDocker(), container, state+++def _poll_until(predicate, timeout=5):+ deadline = time.time() + timeout+ while time.time() < deadline:+ if predicate():+ return True+ time.sleep(0.01)+ return predicate()+++def test_rds_start_db_cluster_accepts_cluster_arn(monkeypatch):+ """StartDBCluster by ARN must reach available, not wedge in starting."""+ from ministack.services import rds as m++ docker, container, _state = _stop_start_fake_docker(+ m._rds_cluster_docker_name("arn-start-cluster"),+ )+ monkeypatch.setattr(m, "_get_docker", lambda: docker)+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)+ monkeypatch.setattr(m, "_next_port", lambda: 16072)+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)+ monkeypatch.setattr(+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "arn-start-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "arn-start-writer",+ "DBClusterIdentifier": "arn-start-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ cluster = m._clusters.get("arn-start-cluster")+ assert _poll_until(lambda: cluster["Status"] == "available")++ m._stop_db_cluster({"DBClusterIdentifier": "arn-start-cluster"})+ assert cluster["Status"] == "stopped"++ status, _, _body = m._start_db_cluster({+ "DBClusterIdentifier": cluster["DBClusterArn"],+ })+ assert status == 200+ assert _poll_until(lambda: cluster["Status"] == "available")+ assert cluster["_shared_container_ready"] is True+ assert container.status == "running"+ assert m._instances.get("arn-start-writer")[+ "DBInstanceStatus"+ ] == "available"+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_start_db_cluster_failure_stays_stopped_and_retryable(monkeypatch):+ """A failed restart AND recreate surfaces an error, not 'available'."""+ from ministack.services import rds as m++ docker, container, state = _stop_start_fake_docker(+ m._rds_cluster_docker_name("failed-start-cluster"),+ )+ monkeypatch.setattr(m, "_get_docker", lambda: docker)+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)+ monkeypatch.setattr(m, "_next_port", lambda: 16073)+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)+ monkeypatch.setattr(+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "failed-start-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "failed-start-writer",+ "DBClusterIdentifier": "failed-start-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ cluster = m._clusters.get("failed-start-cluster")+ assert _poll_until(lambda: cluster["Status"] == "available")+ m._stop_db_cluster({"DBClusterIdentifier": "failed-start-cluster"})++ # Both the restart and the recreate fallback genuinely fail.+ state["fail_start"] = True+ state["fail_run"] = True+ status, _, body = m._start_db_cluster({+ "DBClusterIdentifier": "failed-start-cluster",+ })+ assert status == 500+ assert b"InternalFailure" in body+ assert cluster["Status"] == "stopped"+ assert cluster["_shared_container_ready"] is False+ assert m._instances.get("failed-start-writer")[+ "DBInstanceStatus"+ ] == "stopped"++ # The failure is transient: a retry must succeed.+ state["fail_start"] = False+ state["fail_run"] = False+ status, _, _body = m._start_db_cluster({+ "DBClusterIdentifier": "failed-start-cluster",+ })+ assert status == 200+ assert _poll_until(lambda: cluster["Status"] == "available")+ assert cluster["_shared_container_ready"] is True+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_start_readiness_failure_returns_cluster_to_stopped(monkeypatch):+ """A container that never becomes reachable lands back on stopped."""+ from ministack.services import rds as m++ docker, container, _state = _stop_start_fake_docker(+ m._rds_cluster_docker_name("unready-start-cluster"),+ )+ ready_ok = [True]+ monkeypatch.setattr(m, "_get_docker", lambda: docker)+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)+ monkeypatch.setattr(m, "_next_port", lambda: 16074)+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+ monkeypatch.setattr(+ m, "_wait_for_database_ready", lambda *_args: ready_ok[0],+ )+ monkeypatch.setattr(+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "unready-start-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "unready-start-writer",+ "DBClusterIdentifier": "unready-start-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ cluster = m._clusters.get("unready-start-cluster")+ assert _poll_until(lambda: cluster["Status"] == "available")+ m._stop_db_cluster({"DBClusterIdentifier": "unready-start-cluster"})++ # The container starts but the database never answers.+ ready_ok[0] = False+ status, _, _body = m._start_db_cluster({+ "DBClusterIdentifier": "unready-start-cluster",+ })+ assert status == 200+ assert _poll_until(lambda: cluster["Status"] == "stopped")+ assert cluster["_shared_container_ready"] is False+ assert container.status == "exited"+ assert m._instances.get("unready-start-writer")[+ "DBInstanceStatus"+ ] == "stopped"++ # Once the database can answer, a retried start succeeds.+ ready_ok[0] = True+ status, _, _body = m._start_db_cluster({+ "DBClusterIdentifier": "unready-start-cluster",+ })+ assert status == 200+ assert _poll_until(lambda: cluster["Status"] == "available")+ assert cluster["_shared_container_ready"] is True+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_instance_ops_rejected_while_cluster_stopped(monkeypatch):+ """Create/DeleteDBInstance against a stopped cluster must be rejected."""+ from ministack.services import rds as m++ monkeypatch.setattr(m, "_get_docker", lambda: None)++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "stopped-guard-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "stopped-guard-writer",+ "DBClusterIdentifier": "stopped-guard-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ cluster = m._clusters.get("stopped-guard-cluster")+ assert _poll_until(lambda: cluster["Status"] == "available")+ m._stop_db_cluster({"DBClusterIdentifier": "stopped-guard-cluster"})+ assert cluster["Status"] == "stopped"++ status, _, body = m._create_db_instance({+ "DBInstanceIdentifier": "stopped-guard-reader",+ "DBClusterIdentifier": "stopped-guard-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ assert status == 400+ assert b"InvalidDBClusterStateFault" in body+ assert m._instances.get("stopped-guard-reader") is None++ status, _, body = m._delete_db_instance({+ "DBInstanceIdentifier": "stopped-guard-writer",+ })+ assert status == 400+ assert b"InvalidDBClusterStateFault" in body+ assert m._instances.get("stopped-guard-writer") is not None+ assert cluster.get("DBClusterMembers")+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_stop_db_cluster_docker_failure_surfaces_error(monkeypatch):+ """A failed Docker stop must not report the cluster as stopped."""+ from ministack.services import rds as m++ docker, container, state = _stop_start_fake_docker(+ m._rds_cluster_docker_name("failed-stop-cluster"),+ )+ monkeypatch.setattr(m, "_get_docker", lambda: docker)+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)+ monkeypatch.setattr(m, "_next_port", lambda: 16075)+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)+ monkeypatch.setattr(+ m, "_grant_mysql_master_user_privileges", lambda *_args: None,+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "failed-stop-cluster",+ "Engine": "aurora-mysql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "failed-stop-writer",+ "DBClusterIdentifier": "failed-stop-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-mysql",+ })+ cluster = m._clusters.get("failed-stop-cluster")+ assert _poll_until(lambda: cluster["Status"] == "available")++ state["fail_stop"] = True+ status, _, body = m._stop_db_cluster({+ "DBClusterIdentifier": "failed-stop-cluster",+ })+ assert status == 500+ assert b"InternalFailure" in body+ assert cluster["Status"] == "available"+ assert cluster["_shared_container_ready"] is True+ assert container.status == "running"+ assert m._instances.get("failed-stop-writer")[+ "DBInstanceStatus"+ ] == "available"++ # The failure is transient: a retried stop succeeds.+ state["fail_stop"] = False+ status, _, _body = m._stop_db_cluster({+ "DBClusterIdentifier": "failed-stop-cluster",+ })+ assert status == 200+ assert cluster["Status"] == "stopped"+ assert container.status == "exited"+ finally:+ m._instances.clear()+ m._clusters.clear()++def test_rds_stale_readiness_ignores_same_id_cluster_recreation(monkeypatch):"""Epoch reuse cannot let an old container worker mutate a new cluster."""from ministack.services import rds as m@@ -3681,7 +4198,7 @@ def test_rds_restore_state_respawns_one_container_per_cluster(del m._instances[db_id]m._unregister_instance_from_clusters(db_id)restored_cluster = m._clusters["restored-shared-cluster"]- m._stop_empty_cluster_shared_container(+ m._stop_cluster_shared_container("restored-shared-cluster", restored_cluster,)return database_ready@@ -3813,7 +4330,7 @@ def test_rds_restore_state_respawns_one_container_per_cluster(del m._instances[db_id]m._unregister_instance_from_clusters(db_id)restored_cluster = m._clusters[cluster_id]- m._stop_empty_cluster_shared_container(+ m._stop_cluster_shared_container(cluster_id,restored_cluster,)@@ -6430,6 +6947,63 @@ def test_aurora_delete_member_keeps_shared_data(rds, rds_data):assert containers == [][email protected](not os.environ.get("DOCKER_NETWORK"), reason="DOCKER_NETWORK not set -- live Aurora")+def test_aurora_stop_start_cluster_preserves_data(rds):+ import docker+ import pymysql++ with _live_cluster(rds) as (+ cluster_id,+ writer_id,+ reader_id,+ writer,+ _reader,+ _cluster,+ ):+ with _aurora_connect(writer["Endpoint"]) as conn:+ with conn.cursor() as cursor:+ cursor.execute("CREATE TABLE stop_start_rows (id INT PRIMARY KEY)")+ cursor.execute("INSERT INTO stop_start_rows VALUES (42)")++ containers = docker.from_env().containers.list(+ all=True,+ filters={"label": ["ministack=rds", f"cluster_id={cluster_id}"]},+ )+ assert len(containers) == 1+ container = containers[0]+ original_container_id = container.id++ rds.stop_db_cluster(DBClusterIdentifier=cluster_id)+ stopped = rds.describe_db_clusters(+ DBClusterIdentifier=cluster_id,+ )["DBClusters"][0]+ assert stopped["Status"] == "stopped"+ for db_id in (writer_id, reader_id):+ instance = rds.describe_db_instances(+ DBInstanceIdentifier=db_id,+ )["DBInstances"][0]+ assert instance["DBInstanceStatus"] == "stopped"+ container.reload()+ assert container.status == "exited"+ with pytest.raises((pymysql.err.OperationalError, OSError)):+ _aurora_connect(writer["Endpoint"])++ rds.start_db_cluster(DBClusterIdentifier=cluster_id)+ writer = _wait_for_instance(rds, writer_id)+ _wait_for_instance(rds, reader_id)+ started = rds.describe_db_clusters(+ DBClusterIdentifier=cluster_id,+ )["DBClusters"][0]+ assert started["Status"] == "available"+ container.reload()+ assert container.status == "running"+ assert container.id == original_container_id+ with _aurora_connect(writer["Endpoint"]) as conn:+ with conn.cursor() as cursor:+ cursor.execute("SELECT id FROM stop_start_rows")+ assert cursor.fetchone() == (42,)++def test_rds_aurora_mysql_8_replication_command_is_stable_and_scoped(monkeypatch):from ministack.core.responses import (get_account_id,@@ -7201,7 +7775,7 @@ def test_rds_deleting_last_global_secondary_instance_preserves_headless_applier(set_request_region("us-west-2")monkeypatch.setattr(m,- "_stop_empty_cluster_shared_container",+ "_stop_cluster_shared_container",lambda cluster_id, cluster: stopped.append((cluster_id, cluster)),)@@ -7661,7 +8235,7 @@ def test_rds_remove_headless_global_secondary_commits_detach_and_stops_applier(monkeypatch.setattr(m, "_detach_mysql_replication", lambda *_args: True)monkeypatch.setattr(m,- "_stop_empty_cluster_shared_container",+ "_stop_cluster_shared_container",lambda cluster_id, cluster: stopped.append((cluster_id, cluster)),)