Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

Add Aurora shared storage parity (#1104)

Ian Oberst9ba656646f25
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a7d97532..3642c9cb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **IoT — device shadow operations** — `GetThingShadow`, `UpdateThingShadow`, and `DeleteThingShadow` on the `iot-data` endpoint (classic and named shadows via `?name=`) previously returned `InvalidRequestException: Unsupported iot-data path: .../shadow`. Shadows now store `desired`/`reported` state: updates deep-merge into the stored document (a `null` value removes the field), bump the version, and stamp per-attribute `metadata` timestamps (only the attributes an update touches are re-stamped); the `/accepted` response echoes only the sections sent, while `GetThingShadow` returns the full document including the computed `state.delta` and its `metadata.delta`. A stale `version` in an update is rejected with a `409`, deleting a shadow keeps its version (as on real AWS, it is not reset), and reading or deleting a nonexistent shadow returns `ResourceNotFoundException`. Shadows are account-scoped and persisted with the rest of the IoT state. Contributed by @maximoosemine.
- **IoT — MQTT publishes are routed through topic rules to Lambda** — an MQTT/`iot-data` publish is now matched against each account's topic rules by the rule's `FROM '<topic filter>'` clause (with `+`/`#` wildcards), and every matching enabled rule's `lambda` actions are invoked asynchronously with the message payload as the event (`SELECT *`). Basic Ingest is supported: a publish to `$aws/rules/<ruleName>` is delivered straight to that rule's actions and bypasses pub/sub. Disabled rules are skipped. Contributed by @maximoosemine.
- **API Gateway v2 — `AWS::ApiGatewayV2::Authorizer` CloudFormation support** — deploying a stack with a JWT authorizer on an HTTP API previously failed with `Unsupported resource type: AWS::ApiGatewayV2::Authorizer` and rolled the stack back, even though the same authorizer already worked via the raw API and Terraform. A provisioner now maps the CFN properties onto the existing authorizer store (`Ref` returns the authorizer id, `Fn::GetAtt AuthorizerId` supported, per the CFN resource reference), and `AWS::ApiGatewayV2::Route` carries `AuthorizerId`/`AuthorizationScopes` through (previously dropped) so the route is actually enforced. Contributed by @ryan-bennett.
+- **RDS — Aurora cluster shared-storage model** — Aurora MySQL and Aurora PostgreSQL members now share one cluster-owned database container instead of provisioning divergent per-instance databases. Writer and reader endpoints, direct connections, and the RDS Data API resolve to the same live data, so tables, users, and grants created through one member are visible through every member. Deleting the final provisioned member stops but preserves the shared container, leaving the cluster available while direct SQL and the Data API remain unavailable; attaching a new first member restarts the same container and data. Cluster deletion owns container and volume cleanup, and persisted clusters respawn one shared container on warm boot. The local reader endpoint intentionally remains read/write because it targets the same database process; engine-enforced read-only behavior would require a separate replicating process. Contributed by @Areson.
### Changed
- **EventBridge, ECS, and Firehose — region-isolated state** — continuing the multi-region work, three more services move their state to account+region scope: **EventBridge** (event buses, rules, targets, archives, replays, connections, API destinations, and endpoints; replays run under the region they were started in, and S3→EventBridge notifications dispatch in the bucket's region), **ECS** (clusters, task definitions and their revisions, services, tasks, capacity providers, account settings, and attributes), and **Firehose** (delivery streams, with Kinesis-source ingest kept within the stream's account and region). Same-name resources in different regions no longer collide, cross-region lookups return the same errors real AWS returns, and legacy persisted state migrates by recovering each record's region from its stored ARNs (ECS bumps its on-disk state format to v3). Contributed by @Areson.
diff --git a/README.md b/README.md
index b3173fd1..b1637122 100644
--- a/README.md
+++ b/README.md
@@ -160,7 +160,7 @@ Since 1.4.0 state is additionally isolated per **region** — see [Multi-Region]
| `111111111111` | `111111111111` |
| `048408301323` | `048408301323` |
| `test` | `000000000000` (default) |
-| `AKIAIOSFODNN7EXAMPLE` | `000000000000` (default) |
+| `test-key-do-not-use` | `000000000000` (default) |
**Terraform** — set `access_key` in your provider block:
```hcl
@@ -616,6 +616,17 @@ conn = psycopg2.connect(
Supported engines: `postgres`, `mysql`, `mariadb`, `aurora-postgresql`, `aurora-mysql`
+Aurora cluster members share one cluster-owned database container, so SQL,
+users, and grants created through one member are visible through every member
+and through both cluster endpoints. The local reader endpoint intentionally
+resolves to the same read/write database process as the writer; writes sent to
+the reader are therefore accepted. Emulating an engine-enforced read-only
+reader would require a separate replicating database process. Deleting the
+final provisioned member leaves the cluster `available` but stops its shared
+container, so direct SQL and the Data API remain unavailable until a new member
+is created. That first new member restarts the preserved container and data;
+only `DeleteDBCluster` removes the container and its storage.
+
---
## Real Redis Endpoints (ElastiCache)
diff --git a/ministack/services/rds.py b/ministack/services/rds.py
index e3413ed7..a7472b9b 100644
--- a/ministack/services/rds.py
+++ b/ministack/services/rds.py
@@ -84,6 +84,7 @@ _port_counter = [BASE_PORT]
_docker = None
_ministack_network = None
+_shared_container_lock = threading.RLock()
# Aurora MySQL versions are the creatable set returned by AWS RDS as of
# 2026-07-09. Refresh with:
@@ -141,12 +142,23 @@ DEFAULT_AURORA_MYSQL_IMAGE = "mysql:8.4"
def get_state():
instances = copy.deepcopy(_instances)
+ clusters = copy.deepcopy(_clusters)
# Strip Docker container IDs (not restorable across restarts)
for key in list(instances._data):
instances._data[key].pop("_docker_container_id", None)
+ for key in list(clusters._data):
+ cluster = clusters._data[key]
+ cluster.setdefault(
+ "_shared_storage_initialized",
+ bool(
+ cluster.get("_shared_container_id")
+ and cluster.get("_shared_container_ready")
+ ),
+ )
+ cluster.pop("_shared_container_id", None)
state = {
"instances": instances,
- "clusters": copy.deepcopy(_clusters),
+ "clusters": clusters,
"subnet_groups": copy.deepcopy(_subnet_groups),
"param_groups": copy.deepcopy(_param_groups),
"snapshots": copy.deepcopy(_snapshots),
@@ -164,6 +176,11 @@ def restore_state(data):
if not data:
return
_clusters.update(data.get("clusters", {}))
+ for key in list(_clusters._data):
+ cluster = _clusters._data[key]
+ cluster["_shared_container_id"] = None
+ if cluster.get("_shared_endpoint"):
+ cluster["_shared_container_ready"] = False
_subnet_groups.update(data.get("subnet_groups", {}))
_param_groups.update(data.get("param_groups", {}))
_snapshots.update(data.get("snapshots", {}))
@@ -182,7 +199,12 @@ def restore_state(data):
inst["_docker_container_id"] = None
inst["DBInstanceStatus"] = "creating"
if RDS_PERSIST:
- inst.setdefault("_docker_volume_name", _rds_docker_volume_name(instance_id, account_id, region))
+ inst.setdefault(
+ "_docker_volume_name",
+ _legacy_scoped_rds_docker_volume_name(
+ instance_id, account_id, region,
+ ),
+ )
_instances._data[key] = inst
to_respawn.append((account_id, region, instance_id, inst))
elif isinstance(instances_data, AccountScopedDict):
@@ -216,17 +238,374 @@ def restore_state(data):
# "available" with no running container, and StartDBInstance is
# metadata-only so it can't recover them either.
from ministack.core.responses import _request_account_id, _request_region
+
+ shared_groups = {}
+ standalone = []
for account_id, region, db_id, inst in to_respawn:
+ # `_shared_cluster_id` is new with the shared-storage model. Fall back
+ # to the public cluster identifier so state saved by earlier releases
+ # migrates to one cluster-owned container on its first warm boot.
+ cluster_id = inst.get("_shared_cluster_id") or inst.get("DBClusterIdentifier")
+ if cluster_id:
+ shared_groups.setdefault((account_id, region, cluster_id), []).append(inst)
+ else:
+ standalone.append((account_id, region, db_id, inst))
+
+ # A pre-namespace standalone named ``cluster-<id>`` used the same Docker
+ # name now reserved for cluster ``<id>``. Remove those legacy containers
+ # before any asynchronous cluster runner can claim the name, and mark the
+ # standalone runner so it does not later mistake the new cluster container
+ # for its own stale resource. Persistent data remains in the standalone's
+ # recorded volume and is mounted under its new ``instance-`` name.
+ shared_container_owners = {
+ _rds_cluster_docker_name(cluster_id, account_id, region): cluster_id
+ for account_id, region, cluster_id in shared_groups
+ }
+ docker_client = _get_docker()
+ if docker_client:
+ for account_id, region, db_id, inst in standalone:
+ legacy_name = _legacy_scoped_rds_docker_name(
+ db_id, account_id, region,
+ )
+ cluster_id = shared_container_owners.get(legacy_name)
+ if not cluster_id:
+ continue
+ try:
+ legacy_container = docker_client.containers.get(legacy_name)
+ except Exception:
+ legacy_container = None
+ if legacy_container and _rds_container_is_owned_by(
+ legacy_container,
+ expected_db_ids={db_id},
+ expected_cluster_ids={cluster_id},
+ account_id=account_id,
+ region=region,
+ ):
+ try:
+ legacy_container.remove(force=True, v=False)
+ except Exception:
+ pass
+ inst["_legacy_scoped_container_migrated"] = True
+
+ for account_id, region, cluster_id in shared_groups:
+ members = shared_groups[(account_id, region, cluster_id)]
+ cluster = _clusters.get_scoped(
+ account_id or get_account_id(), region, cluster_id,
+ )
+ if cluster:
+ # Publish the restore/migration gate before the daemon starts. A
+ # create arriving immediately after restore_state() must not start
+ # fresh cluster storage before the writer volume is adopted.
+ cluster["_shared_legacy_migration_in_progress"] = True
+ cluster.pop("_shared_legacy_migration_blocked", None)
+ ctx = contextvars.copy_context()
+
+ def _cluster_runner(
+ account_id=account_id,
+ region=region,
+ cluster_id=cluster_id,
+ members=members,
+ ):
+ if account_id is not None:
+ _request_account_id.set(account_id)
+ if region is not None:
+ _request_region.set(region)
+ cluster = _clusters.get(cluster_id)
+ if not cluster:
+ for member in members:
+ member["DBInstanceStatus"] = "failed"
+ return
+ restore_epoch = int(cluster.get("_shared_container_epoch", 0))
+
+ # State from before cluster-owned storage has one container and
+ # volume per member. Preserve the writer's volume as the
+ # authoritative cluster volume, remove the superseded member
+ # containers without deleting their volumes, and reap only the
+ # non-writer volumes after the shared container starts.
+ member_by_id = {
+ member.get("DBInstanceIdentifier"): member
+ for member in members
+ }
+ writer_id = next(
+ (
+ item.get("DBInstanceIdentifier")
+ for item in cluster.get("DBClusterMembers", [])
+ if item.get("IsClusterWriter")
+ ),
+ None,
+ )
+ writer = member_by_id.get(writer_id) or members[0]
+ legacy_volumes = {
+ member.get("_docker_volume_name")
+ for member in members
+ if member.get("_docker_volume_name")
+ }
+ adopted_volume = cluster.get("_shared_volume_name")
+ if not adopted_volume:
+ adopted_volume = writer.get("_docker_volume_name")
+ if adopted_volume:
+ cluster["_shared_volume_name"] = adopted_volume
+ cluster["_shared_storage_initialized"] = True
+ # The initialized writer volume is authoritative. Legacy
+ # releases allowed member connection settings to differ
+ # from the parent, and image environment variables do not
+ # rewrite users or databases on an existing volume.
+ cluster["MasterUsername"] = writer.get(
+ "MasterUsername",
+ cluster.get("MasterUsername", "admin"),
+ )
+ pending_rotation = cluster.get(
+ "_pending_master_password_rotation",
+ )
+ if pending_rotation:
+ cluster["_MasterUserPassword"] = pending_rotation[
+ "new_password"
+ ]
+ else:
+ cluster["_MasterUserPassword"] = writer.get(
+ "_MasterUserPassword",
+ cluster.get("_MasterUserPassword", "password"),
+ )
+ cluster["DatabaseName"] = (
+ writer.get("DBName")
+ or cluster.get("DatabaseName")
+ or "mydb"
+ )
+
+ docker_client = _get_docker()
+ if docker_client:
+ legacy_container_owners = {}
+ for member in members:
+ member_id = member.get("DBInstanceIdentifier")
+ if not member_id:
+ continue
+ scoped_legacy_name = _legacy_scoped_rds_docker_name(
+ member_id, account_id, region,
+ )
+ legacy_container_owners.setdefault(
+ scoped_legacy_name, set(),
+ ).add(member_id)
+ explicit_legacy_name = member.get(
+ "_legacy_docker_container_name",
+ )
+ if explicit_legacy_name:
+ legacy_container_owners.setdefault(
+ explicit_legacy_name, set(),
+ ).add(member_id)
+ legacy_container_remains = False
+ for legacy_name, expected_member_ids in (
+ legacy_container_owners.items()
+ ):
+ try:
+ legacy_container = docker_client.containers.get(
+ legacy_name,
+ )
+ except Exception:
+ continue
+ if _rds_container_is_owned_by(
+ legacy_container,
+ expected_db_ids=expected_member_ids,
+ expected_cluster_ids={cluster_id},
+ account_id=account_id,
+ region=region,
+ ):
+ try:
+ legacy_container.remove(
+ force=True, v=False,
+ )
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to remove legacy Aurora member "
+ "container %s: %s", legacy_name, e,
+ )
+ else:
+ logger.warning(
+ "RDS: refusing to remove container %s while "
+ "migrating Aurora members %s because its labels "
+ "do not prove ownership",
+ legacy_name,
+ sorted(expected_member_ids),
+ )
+ try:
+ docker_client.containers.get(legacy_name)
+ except Exception:
+ continue
+ logger.warning(
+ "RDS: legacy Aurora member container %s remains "
+ "after migration removal", legacy_name,
+ )
+ legacy_container_remains = True
+
+ if legacy_container_remains:
+ cluster.pop("_shared_legacy_migration_in_progress", None)
+ cluster["_shared_legacy_migration_blocked"] = True
+ cluster["_shared_container_ready"] = False
+ for member in members:
+ member["DBInstanceStatus"] = "failed"
+ return
+ cluster.pop("_shared_legacy_migration_blocked", None)
+ for member in members:
+ member.pop("_legacy_docker_container_name", None)
+
+ # The persisted member list is only a restore-time snapshot. Requests
+ # can delete those members (or the entire cluster) while legacy
+ # resources are being inspected above. Serialize the final check with
+ # cluster teardown and start compute only for this same cluster
+ # incarnation with at least one original member still attached.
+ original_member_ids = set(member_by_id)
+ with _shared_container_lock:
+ current_cluster = _clusters.get(cluster_id)
+ current_member_ids = {
+ item.get("DBInstanceIdentifier")
+ for item in cluster.get("DBClusterMembers", [])
+ if item.get("DBInstanceIdentifier")
+ }
+ original_member_still_attached = any(
+ member_id in current_member_ids
+ and _instances.get(member_id) is member_by_id[member_id]
+ for member_id in original_member_ids
+ )
+ if (
+ current_cluster is not cluster
+ or int(cluster.get("_shared_container_epoch", 0))
+ != restore_epoch
+ or not original_member_still_attached
+ ):
+ if current_cluster is cluster:
+ cluster.pop(
+ "_shared_legacy_migration_in_progress", None,
+ )
+ return
+ result = _start_cluster_shared_container(
+ cluster_id,
+ cluster,
+ remove_stale=True,
+ )
+ cluster.pop("_shared_legacy_migration_in_progress", None)
+ status = "failed" if result.get("failed") else "available"
+ authenticated_ready = False
+ if result.get("started"):
+ status = "creating"
+ for member in members:
+ _attach_instance_to_shared_cluster(member, cluster)
+ member["DBInstanceStatus"] = status
+ _sync_cluster_endpoints(cluster)
+
+ if result.get("started"):
+ container_id = cluster.get("_shared_container_id")
+ container_epoch = result.get("container_epoch")
+
+ def _container_alive():
+ try:
+ container = docker_client.containers.get(container_id)
+ container.reload()
+ return container.status not in (
+ "exited", "dead", "removing",
+ )
+ except Exception:
+ return False
+
+ with _shared_container_lock:
+ pending_rotation = cluster.get(
+ "_pending_master_password_rotation",
+ )
+ readiness_password = (
+ pending_rotation["old_password"]
+ if pending_rotation
+ else cluster.get("_MasterUserPassword", "password")
+ )
+ authenticated_ready = _wait_for_database_ready(
+ result.get("readiness_host")
+ or cluster["_shared_endpoint"]["Address"],
+ result.get("readiness_port")
+ or cluster["_shared_endpoint"]["Port"],
+ cluster.get("Engine", "aurora-postgresql"),
+ cluster.get("MasterUsername", "admin"),
+ readiness_password,
+ cluster.get("DatabaseName") or "mydb",
+ _container_alive,
+ )
+ with _shared_container_lock:
+ current_cluster = _clusters.get(cluster_id)
+ if (
+ current_cluster is not cluster
+ or (
+ container_epoch is not None
+ and cluster.get("_shared_container_epoch")
+ != container_epoch
+ )
+ or cluster.get("_shared_container_id") != container_id
+ ):
+ return
+ pending_rotation = cluster.get(
+ "_pending_master_password_rotation",
+ )
+ if authenticated_ready and pending_rotation:
+ authenticated_ready = _rotate_real_password(
+ cluster,
+ pending_rotation["old_password"],
+ pending_rotation["new_password"],
+ )
+ if authenticated_ready:
+ cluster.pop(
+ "_pending_master_password_rotation", None,
+ )
+ if authenticated_ready:
+ cluster["_shared_storage_initialized"] = True
+ if authenticated_ready and _is_mysql_engine(
+ cluster.get("Engine", ""),
+ ):
+ _grant_mysql_master_user_privileges(
+ result.get("readiness_host")
+ or cluster["_shared_endpoint"]["Address"],
+ result.get("readiness_port")
+ or cluster["_shared_endpoint"]["Port"],
+ cluster.get("MasterUsername", "admin"),
+ cluster.get("_MasterUserPassword", "password"),
+ cluster_id,
+ )
+ cluster["_shared_container_ready"] = authenticated_ready
+ status = "available" if authenticated_ready else "failed"
+ for current_member in cluster.get(
+ "DBClusterMembers", [],
+ ):
+ member = _instances.get(
+ current_member.get("DBInstanceIdentifier"),
+ )
+ if member is not None:
+ _attach_instance_to_shared_cluster(member, cluster)
+ member["DBInstanceStatus"] = status
+ _sync_cluster_endpoints(cluster)
+ _refresh_cluster_status(cluster_id)
+ else:
+ cluster["_shared_container_ready"] = status == "available"
+
+ # Superseded member volumes are recovery copies until the adopted
+ # writer volume has passed an authenticated database readiness
+ # check. Docker accepting containers.run() is not sufficient.
+ if docker_client and authenticated_ready:
+ for volume_name in legacy_volumes - {adopted_volume}:
+ try:
+ docker_client.volumes.get(volume_name).remove()
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to remove superseded Aurora member "
+ "volume %s: %s", volume_name, e,
+ )
+
+ threading.Thread(target=ctx.run, args=(_cluster_runner,), daemon=True).start()
+
+ for account_id, region, db_id, inst in standalone:
ctx = contextvars.copy_context()
- def _runner(account_id=account_id, region=region, db_id=db_id, inst=inst):
+ def _instance_runner(account_id=account_id, region=region, db_id=db_id, inst=inst):
if account_id is not None:
_request_account_id.set(account_id)
if region is not None:
_request_region.set(region)
_start_rds_container_for_instance(db_id, inst)
- threading.Thread(target=ctx.run, args=(_runner,), daemon=True).start()
+ threading.Thread(target=ctx.run, args=(_instance_runner,), daemon=True).start()
def _best_effort_region_from_record_arn(record, field):
@@ -250,11 +629,35 @@ def _rds_docker_scope(account_id=None, region=None):
def _rds_docker_name(db_id, account_id=None, region=None):
- return f"ministack-rds-{_rds_docker_scope(account_id, region)}-{db_id}"
+ return f"ministack-rds-{_rds_docker_scope(account_id, region)}-instance-{db_id}"
def _rds_docker_volume_name(db_id, account_id=None, region=None):
- return f"ministack-rds-{_rds_docker_scope(account_id, region)}-{db_id}-data"
+ return f"{_rds_docker_name(db_id, account_id, region)}-data"
+
+
+def _legacy_scoped_rds_docker_name(db_id, account_id=None, region=None):
+ """Return the pre-namespace standalone container name."""
+ return f"ministack-rds-{_rds_docker_scope(account_id, region)}-{db_id}"
+
+
+def _legacy_scoped_rds_docker_volume_name(db_id, account_id=None, region=None):
+ return f"{_legacy_scoped_rds_docker_name(db_id, account_id, region)}-data"
+
+
+def _rds_cluster_docker_name(cluster_id, account_id=None, region=None):
+ return f"ministack-rds-{_rds_docker_scope(account_id, region)}-cluster-{cluster_id}"
+
+
+def _rds_cluster_docker_volume_name(cluster_id, account_id=None, region=None):
+ # Put the resource type before the scope hash. The legacy standalone shape
+ # is ``ministack-rds-<scope>-<db-id>-data``, so merely inserting
+ # ``cluster-`` after the scope still collides with a legacy instance named
+ # ``cluster-<cluster-id>``.
+ return (
+ f"ministack-rds-cluster-{_rds_docker_scope(account_id, region)}-"
+ f"{cluster_id}-data"
+ )
def _legacy_rds_docker_name(db_id):
@@ -265,6 +668,336 @@ def _legacy_rds_docker_volume_name(db_id):
return f"ministack-rds-{db_id}-data"
+def _rds_container_is_owned_by(
+ container, expected_db_ids=(), expected_cluster_ids=(),
+ account_id=None, region=None,
+):
+ """Return whether RDS labels prove a container belongs to an owner."""
+ labels = getattr(container, "labels", None)
+ if not isinstance(labels, dict):
+ labels = (
+ getattr(container, "attrs", {})
+ .get("Config", {})
+ .get("Labels", {})
+ )
+ if not isinstance(labels, dict):
+ return False
+ if labels.get("ministack") != "rds":
+ return False
+ owns_expected_db = labels.get("db_id") in set(expected_db_ids)
+ owns_expected_cluster = labels.get("cluster_id") in set(
+ expected_cluster_ids,
+ )
+ if not owns_expected_db and not owns_expected_cluster:
+ return False
+ if labels.get("account_id") not in (None, account_id or get_account_id()):
+ return False
+ return labels.get("region") in (None, region or get_region())
+
+
+def _start_cluster_shared_container(cluster_id, cluster, remove_stale=False):
+ """Start the single backing container owned by an Aurora cluster.
+
+ Cluster members are control-plane records that all point at this endpoint.
+ The helper is shared by first-member creation and persisted-state restore.
+ """
+ engine = cluster.get("Engine", "aurora-postgresql")
+ engine_version = cluster.get("EngineVersion") or _default_engine_version(engine)
+ master_user = cluster.get("MasterUsername", "admin")
+ master_pass = cluster.get("_MasterUserPassword", "password")
+ db_name = cluster.get("DatabaseName") or "mydb"
+ endpoint = {
+ "Address": _MINISTACK_HOST,
+ "Port": int(cluster.get("Port") or _default_port(engine)),
+ "HostedZoneId": cluster.get("HostedZoneId", "Z2R2ITUGPM61AM"),
+ }
+ cluster.update({
+ "_shared_container_id": None,
+ "_shared_endpoint": endpoint,
+ "_shared_internal_address": None,
+ "_shared_internal_port": None,
+ "_shared_container_ready": True,
+ })
+
+ docker_client = _get_docker()
+ if not docker_client:
+ return {"started": False, "failed": False, "readiness_host": None, "readiness_port": None}
+
+ image, env, container_port, data_path = _docker_image_for_engine(
+ engine, engine_version, master_user, master_pass, db_name,
+ )
+ if not image:
+ return {"started": False, "failed": False, "readiness_host": None, "readiness_port": None}
+
+ container_name = _rds_cluster_docker_name(cluster_id)
+ if remove_stale:
+ try:
+ stale_container = docker_client.containers.get(container_name)
+ except Exception:
+ stale_container = None
+ if stale_container:
+ if _rds_container_is_owned_by(
+ stale_container,
+ expected_cluster_ids={cluster_id},
+ ):
+ try:
+ stale_container.remove(force=True, v=False)
+ except Exception:
+ pass
+ else:
+ logger.warning(
+ "RDS: refusing to remove container %s before starting "
+ "cluster %s because its labels do not prove ownership",
+ container_name,
+ cluster_id,
+ )
+
+ host_port = cluster.get("_shared_host_port") or _next_port()
+ if not _is_host_port_free(host_port):
+ logger.info(
+ "RDS: persisted shared host port %d for cluster %s is in use; allocating fresh free port",
+ host_port, cluster_id,
+ )
+ host_port = _next_port()
+
+ ms_network = _get_ministack_network(docker_client)
+ container_kwargs = dict(
+ image=image,
+ detach=True,
+ environment=env,
+ ports={f"{container_port}/tcp": host_port},
+ name=container_name,
+ labels={
+ "ministack": "rds",
+ "cluster_id": cluster_id,
+ "account_id": get_account_id(),
+ "region": get_region(),
+ },
+ )
+ if ms_network:
+ container_kwargs["network"] = ms_network
+ # Aurora storage belongs to the cluster, not to any member instance. Use a
+ # named volume even when standalone RDS persistence is disabled so stopping
+ # an empty cluster's compute and restarting it later cannot erase its data.
+ volume_name = (
+ cluster.get("_shared_volume_name")
+ or _rds_cluster_docker_volume_name(cluster_id)
+ )
+ cluster["_shared_volume_name"] = volume_name
+ container_kwargs["volumes"] = {
+ volume_name: {"bind": data_path, "mode": "rw"},
+ }
+
+ try:
+ container = docker_client.containers.run(**container_kwargs)
+ except Exception as e:
+ cluster["_shared_container_ready"] = False
+ logger.warning("RDS: failed to start shared container for cluster %s: %s", cluster_id, e)
+ return {"started": False, "failed": True, "readiness_host": None, "readiness_port": None}
+
+ endpoint_host = _MINISTACK_HOST
+ endpoint_port = host_port
+ internal_host = None
+ internal_port = None
+ readiness_host = "127.0.0.1"
+ readiness_port = host_port
+ if ms_network:
+ try:
+ container.reload()
+ networks = container.attrs.get("NetworkSettings", {}).get("Networks", {})
+ container_ip = networks.get(ms_network, {}).get("IPAddress", "")
+ if container_ip:
+ endpoint_host = container_ip
+ endpoint_port = container_port
+ internal_host = container_ip
+ internal_port = container_port
+ readiness_host = container_ip
+ readiness_port = container_port
+ except Exception:
+ pass
+
+ container_epoch = int(cluster.get("_shared_container_epoch", 0)) + 1
+ cluster.update({
+ "_shared_container_id": container.id,
+ "_shared_host_port": host_port,
+ "_shared_endpoint": {
+ "Address": endpoint_host,
+ "Port": endpoint_port,
+ "HostedZoneId": cluster.get("HostedZoneId", "Z2R2ITUGPM61AM"),
+ },
+ "_shared_internal_address": internal_host,
+ "_shared_internal_port": internal_port,
+ "_shared_container_ready": False,
+ "_shared_container_epoch": container_epoch,
+ })
+ return {
+ "started": True,
+ "failed": False,
+ "readiness_host": readiness_host,
+ "readiness_port": readiness_port,
+ "network": ms_network,
+ "container_port": container_port,
+ "container_epoch": container_epoch,
+ }
+
+
+def _stop_empty_cluster_shared_container(cluster_id, cluster):
+ """Stop, but do not remove, an empty Aurora cluster's database.
+
+ Aurora keeps the cluster volume after its final DB instance is deleted,
+ but no SQL endpoint is reachable until another instance is attached. A
+ stopped Docker container models that split: the cluster still owns the
+ container and its data, while only DeleteDBCluster removes either one.
+ """
+ with _shared_container_lock:
+ # Invalidate every readiness worker before the potentially slow Docker
+ # stop. The worker takes the same lock for its final epoch check and
+ # state transition, so it cannot publish ready=True after this point.
+ cluster["_shared_container_epoch"] = int(
+ cluster.get("_shared_container_epoch", 0),
+ ) + 1
+ cluster["_shared_container_ready"] = False
+ container_id = cluster.get("_shared_container_id")
+ docker_client = _get_docker()
+ if not docker_client or not container_id:
+ return
+ try:
+ container = docker_client.containers.get(container_id)
+ container.reload()
+ if container.status not in ("created", "exited", "dead", "removing"):
+ container.stop(timeout=5)
+ logger.info("RDS: stopped shared container for empty cluster %s", cluster_id)
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to stop shared container for empty cluster %s: %s",
+ cluster_id,
+ e,
+ )
+
+
+def _remove_cluster_shared_resources(
+ cluster_id, cluster, timeout=5, account_id=None, region=None,
+):
+ """Remove a cluster-owned container and volume by ID or stable name."""
+ docker_client = _get_docker()
+ if not docker_client:
+ return
+
+ container_identifiers = []
+ if cluster.get("_shared_container_id"):
+ container_identifiers.append(cluster["_shared_container_id"])
+ container_name = _rds_cluster_docker_name(
+ cluster_id, account_id, region,
+ )
+ if container_name not in container_identifiers:
+ container_identifiers.append(container_name)
+
+ for identifier in container_identifiers:
+ try:
+ container = docker_client.containers.get(identifier)
+ container.stop(timeout=timeout)
+ container.remove(v=True)
+ logger.info("RDS: removed shared container for cluster %s", cluster_id)
+ break
+ except Exception:
+ continue
+
+ volume_name = (
+ cluster.get("_shared_volume_name")
+ or _rds_cluster_docker_volume_name(
+ cluster_id, account_id, region,
+ )
+ )
+ try:
+ docker_client.volumes.get(volume_name).remove()
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to remove shared volume for cluster %s: %s",
+ cluster_id, e,
+ )
+
+
+def _restart_cluster_shared_container(cluster_id, cluster):
+ """Restart the preserved shared container when an empty cluster grows."""
+ docker_client = _get_docker()
+ container_id = cluster.get("_shared_container_id")
+ if not docker_client or not container_id:
+ return {
+ "started": False,
+ "failed": False,
+ "readiness_host": None,
+ "readiness_port": None,
+ }
+
+ try:
+ container = docker_client.containers.get(container_id)
+ container.start()
+ container.reload()
+ except Exception as e:
+ cluster["_shared_container_ready"] = False
+ logger.warning(
+ "RDS: failed to restart shared container for cluster %s: %s",
+ cluster_id,
+ e,
+ )
+ return {
+ "started": False,
+ "failed": True,
+ "readiness_host": None,
+ "readiness_port": None,
+ }
+
+ engine = cluster.get("Engine", "aurora-postgresql")
+ container_port = int(
+ cluster.get("_shared_internal_port") or _default_port(engine),
+ )
+ host_port = int(
+ cluster.get("_shared_host_port")
+ or (cluster.get("_shared_endpoint") or {}).get("Port")
+ or container_port,
+ )
+ ms_network = _get_ministack_network(docker_client)
+ endpoint_host = _MINISTACK_HOST
+ endpoint_port = host_port
+ internal_host = None
+ internal_port = None
+ readiness_host = "127.0.0.1"
+ readiness_port = host_port
+ if ms_network:
+ networks = container.attrs.get("NetworkSettings", {}).get("Networks", {})
+ container_ip = networks.get(ms_network, {}).get("IPAddress", "")
+ if container_ip:
+ endpoint_host = container_ip
+ endpoint_port = container_port
+ internal_host = container_ip
+ internal_port = container_port
+ readiness_host = container_ip
+ readiness_port = container_port
+
+ container_epoch = int(cluster.get("_shared_container_epoch", 0)) + 1
+ cluster.update({
+ "_shared_endpoint": {
+ "Address": endpoint_host,
+ "Port": endpoint_port,
+ "HostedZoneId": cluster.get("HostedZoneId", "Z2R2ITUGPM61AM"),
+ },
+ "_shared_internal_address": internal_host,
+ "_shared_internal_port": internal_port,
+ "_shared_container_ready": False,
+ "_shared_container_epoch": container_epoch,
+ })
+ logger.info("RDS: restarted shared container for cluster %s", cluster_id)
+ return {
+ "started": True,
+ "failed": False,
+ "readiness_host": readiness_host,
+ "readiness_port": readiness_port,
+ "network": ms_network,
+ "container_port": container_port,
+ "container_epoch": container_epoch,
+ }
+
+
def _start_rds_container_for_instance(db_id, instance):
"""Re-spin (or re-attach to) the Docker container for a restored instance.
@@ -314,6 +1047,8 @@ def _start_rds_container_for_instance(db_id, instance):
container_name = _rds_docker_name(db_id)
stale_names = [container_name]
+ if not instance.pop("_legacy_scoped_container_migrated", False):
+ stale_names.append(_legacy_scoped_rds_docker_name(db_id))
legacy_container_name = instance.pop("_legacy_docker_container_name", None)
if legacy_container_name and legacy_container_name != container_name:
stale_names.append(legacy_container_name)
@@ -622,7 +1357,6 @@ def _is_host_port_free(port: int) -> bool:
user-process bindings."""
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
s.bind(("0.0.0.0", port))
return True
@@ -960,24 +1694,47 @@ def _resolve_instance(db_id):
return None
-def _sync_cluster_endpoints(cluster):
- """Point Aurora cluster endpoints at reachable local DB instance endpoints."""
- members = cluster.get("DBClusterMembers") or []
- if not members:
+def _attach_instance_to_shared_cluster(instance, cluster):
+ endpoint = cluster.get("_shared_endpoint")
+ if not endpoint:
return
+ instance["Endpoint"] = copy.deepcopy(endpoint)
+ instance["_HostPort"] = cluster.get("_shared_host_port")
+ instance["_docker_container_id"] = cluster.get("_shared_container_id")
+ instance["_docker_volume_name"] = None
+ instance["_internal_address"] = cluster.get("_shared_internal_address")
+ instance["_internal_port"] = cluster.get("_shared_internal_port")
+ instance["_shared_cluster_id"] = cluster["DBClusterIdentifier"]
+ instance["MasterUsername"] = cluster.get(
+ "MasterUsername",
+ instance.get("MasterUsername", "admin"),
+ )
+ instance["_MasterUserPassword"] = cluster.get(
+ "_MasterUserPassword",
+ instance.get("_MasterUserPassword", "password"),
+ )
+ instance["DBName"] = (
+ cluster.get("DatabaseName")
+ or instance.get("DBName")
+ or "mydb"
+ )
+
- writer = next((m for m in members if m.get("IsClusterWriter")), members[0])
- writer_inst = _instances.get(writer.get("DBInstanceIdentifier"))
- reader_member = next((m for m in members if not m.get("IsClusterWriter")), writer)
- reader_inst = _instances.get(reader_member.get("DBInstanceIdentifier")) or writer_inst
+def _sync_cluster_endpoints(cluster):
+ """Point both Aurora endpoints at the cluster-owned shared container.
- if writer_inst and writer_inst.get("Endpoint"):
- writer_ep = writer_inst["Endpoint"]
- cluster["Endpoint"] = writer_ep.get("Address", cluster.get("Endpoint", ""))
- cluster["Port"] = int(writer_ep.get("Port", cluster.get("Port", 0)))
- if reader_inst and reader_inst.get("Endpoint"):
- cluster["ReaderEndpoint"] = reader_inst["Endpoint"].get(
- "Address", cluster.get("ReaderEndpoint", ""))
+ The local reader endpoint is intentionally read/write because it resolves
+ to the same MySQL process as the writer. Genuine read-only behavior would
+ require a separate replicating process.
+ """
+ endpoint = cluster.get("_shared_endpoint")
+ if not endpoint:
+ return
+ cluster["Endpoint"] = endpoint.get("Address", cluster.get("Endpoint", ""))
+ cluster["ReaderEndpoint"] = endpoint.get(
+ "Address", cluster.get("ReaderEndpoint", ""),
+ )
+ cluster["Port"] = int(endpoint.get("Port", cluster.get("Port", 0)))
def _register_instance_in_cluster(instance):
@@ -1038,13 +1795,30 @@ def _create_db_instance(p):
cluster_id_param = _p(p, "DBClusterIdentifier")
parent = _resolve_cluster_in_request_region(cluster_id_param) if cluster_id_param else None
if parent:
+ if parent.get("_shared_legacy_migration_in_progress"):
+ return _error(
+ "InvalidDBClusterStateFault",
+ "Cannot add a DB instance while legacy shared-storage "
+ "migration is in progress.",
+ 400,
+ )
+ if parent.get("_shared_legacy_migration_blocked"):
+ return _error(
+ "InvalidDBClusterStateFault",
+ "Cannot add a DB instance while legacy member storage "
+ "migration is blocked.",
+ 400,
+ )
cluster_id_param = parent["DBClusterIdentifier"]
- if not _p(p, "MasterUsername"):
- master_user = parent.get("MasterUsername", master_user)
- if not _p(p, "MasterUserPassword"):
- master_pass = parent.get("_MasterUserPassword", master_pass)
- if not db_name:
- db_name = parent.get("DatabaseName", "")
+ engine = parent.get("Engine", engine)
+ engine_version = parent.get("EngineVersion", engine_version)
+ port = int(parent.get("Port") or _default_port(engine))
+ # Aurora credentials and the initial database belong to the cluster,
+ # not individual members. Ignore conflicting member request fields so
+ # readiness always authenticates against the shared process.
+ master_user = parent.get("MasterUsername", master_user)
+ master_pass = parent.get("_MasterUserPassword", master_pass)
+ db_name = parent.get("DatabaseName") or "mydb"
elif _parse_rds_arn(cluster_id_param):
wrong_region = _invalid_cluster_identifier_error(cluster_id_param)
if wrong_region:
@@ -1055,6 +1829,24 @@ def _create_db_instance(p):
allocated_storage = int(_p(p, "AllocatedStorage") or "20")
storage_type = _p(p, "StorageType") or "gp2"
subnet_group_name = _p(p, "DBSubnetGroupName") or "default"
+ explicit_pg = _p(p, "DBParameterGroupName")
+ # Validate every request field before shared compute can start or restart.
+ # Otherwise an invalid member request can make an empty cluster reachable
+ # even though no member record is ultimately created.
+ if (
+ explicit_pg
+ and not explicit_pg.startswith("default.")
+ and explicit_pg not in _param_groups
+ ):
+ return _error(
+ "DBParameterGroupNotFound",
+ f"DBParameterGroup {explicit_pg} not found.",
+ 404,
+ )
+ param_group_name = (
+ explicit_pg
+ or f"default.{engine}{engine_version.split('.')[0]}"
+ )
arn = f"arn:aws:rds:{get_region()}:{get_account_id()}:db:{db_id}"
dbi_resource_id = f"db-{new_uuid().replace('-', '')[:20].upper()}"
@@ -1068,9 +1860,80 @@ def _create_db_instance(p):
real_container_started = False
readiness_host = None
readiness_port = None
+ readiness_master_pass = master_pass
+ ms_network = None
docker_client = _get_docker()
- if docker_client:
+ if parent:
+ pending_rotation = parent.get("_pending_master_password_rotation")
+ if pending_rotation:
+ readiness_master_pass = pending_rotation["old_password"]
+ # RDS action dispatch is synchronous and await-free, so this check and
+ # the shared metadata update cannot interleave with another create.
+ resume_control_plane_only = (
+ not docker_client
+ and not parent.get("DBClusterMembers")
+ and not parent.get("_shared_container_ready", True)
+ )
+ has_failed_shared_member = any(
+ (_instances.get(member.get("DBInstanceIdentifier")) or {}).get(
+ "DBInstanceStatus",
+ ) == "failed"
+ for member in parent.get("DBClusterMembers", [])
+ if member.get("DBInstanceIdentifier")
+ )
+ restart_unhealthy_container = (
+ docker_client
+ and parent.get("_shared_container_id")
+ and not parent.get("_shared_container_ready", True)
+ and (
+ not parent.get("DBClusterMembers")
+ or has_failed_shared_member
+ )
+ )
+ if resume_control_plane_only:
+ parent["_shared_container_ready"] = True
+ start_result = {"started": False, "failed": False}
+ elif restart_unhealthy_container:
+ start_result = _restart_cluster_shared_container(
+ cluster_id_param,
+ parent,
+ )
+ if start_result.get("failed"):
+ # A preserved container can become unrestartable (for example,
+ # its old host port may have been claimed). Recreate only the
+ # compute layer; the cluster-owned named volume retains data.
+ start_result = _start_cluster_shared_container(
+ cluster_id_param,
+ parent,
+ remove_stale=True,
+ )
+ else:
+ needs_shared_start = (
+ not parent.get("_shared_endpoint")
+ or (docker_client and not parent.get("_shared_container_id"))
+ )
+ start_result = (
+ _start_cluster_shared_container(
+ cluster_id_param,
+ parent,
+ remove_stale=bool(parent.get("_shared_endpoint")),
+ )
+ if needs_shared_start
+ else {"started": False, "failed": False}
+ )
+ shared_endpoint = parent.get("_shared_endpoint") or {}
+ endpoint_host = shared_endpoint.get("Address", endpoint_host)
+ endpoint_port = int(shared_endpoint.get("Port", endpoint_port))
+ host_port = parent.get("_shared_host_port")
+ docker_container_id = parent.get("_shared_container_id")
+ internal_host = parent.get("_shared_internal_address")
+ internal_port = parent.get("_shared_internal_port")
+ real_container_started = bool(start_result.get("started"))
+ readiness_host = start_result.get("readiness_host")
+ readiness_port = start_result.get("readiness_port")
+ ms_network = start_result.get("network")
+ elif docker_client:
ms_network = _get_ministack_network(docker_client)
image, env, container_port, data_path = _docker_image_for_engine(
engine, engine_version, master_user, master_pass, db_name
@@ -1137,14 +2000,6 @@ def _create_db_instance(p):
logger.warning("RDS: Docker failed for %s: %s", db_id, e)
cluster_id = cluster_id_param
- explicit_pg = _p(p, "DBParameterGroupName")
- # AWS rejects a reference to a parameter group that doesn't exist. Validate
- # custom names against the store; AWS-managed `default.*` groups are implicit
- # and not tracked here, so they're accepted as-is.
- if explicit_pg and not explicit_pg.startswith("default.") and explicit_pg not in _param_groups:
- return _error("DBParameterGroupNotFound",
- f"DBParameterGroup {explicit_pg} not found.", 404)
- param_group_name = explicit_pg or f"default.{engine}{engine_version.split('.')[0]}"
now_ts = time.time()
vpc_sgs = _parse_member_list(p, "VpcSecurityGroupIds")
@@ -1158,13 +2013,18 @@ def _create_db_instance(p):
"VpcId": "vpc-00000000",
"DBSubnetGroupArn": f"arn:aws:rds:{get_region()}:{get_account_id()}:subgrp:{subnet_group_name}",
})
+ instance_status = "creating" if real_container_started else "available"
+ if parent and not parent.get("_shared_container_ready", True):
+ instance_status = "creating"
+ if parent and start_result.get("failed"):
+ instance_status = "failed"
instance = {
"DBInstanceIdentifier": db_id,
"DBInstanceClass": db_class,
"Engine": engine,
"EngineVersion": engine_version,
- "DBInstanceStatus": "creating" if real_container_started else "available",
+ "DBInstanceStatus": instance_status,
"MasterUsername": master_user,
"DBName": db_name,
"Endpoint": {
@@ -1252,6 +2112,8 @@ def _create_db_instance(p):
"_internal_port": internal_port,
"_MasterUserPassword": master_pass,
}
+ if parent:
+ _attach_instance_to_shared_cluster(instance, parent)
_instances[db_id] = instance
_register_instance_in_cluster(instance)
@@ -1272,10 +2134,14 @@ def _create_db_instance(p):
def _bg_finalize_ready(
db_id=db_id, cluster_id=cluster_id, engine=engine,
master_user=master_user, master_pass=master_pass,
+ readiness_master_pass=readiness_master_pass,
db_name=db_name, ready_host=ready_host, ready_port=ready_port,
ms_network=ms_network, internal_host=internal_host,
internal_port=internal_port, endpoint_port=endpoint_port,
container_id=docker_container_id,
+ container_epoch=(
+ start_result.get("container_epoch") if parent else None
+ ),
):
# Tie readiness to backing-container liveness rather than a wall
# clock: real RDS `CreateDBInstance` has no caller-visible timeout
@@ -1293,10 +2159,92 @@ def _create_db_instance(p):
return c.status not in ("exited", "dead", "removing")
except Exception:
return False
- if not _wait_for_database_ready(
+ database_ready = _wait_for_database_ready(
ready_host, ready_port, engine, master_user,
- master_pass, db_name, _container_alive,
- ):
+ readiness_master_pass, db_name, _container_alive,
+ )
+ cluster = _clusters.get(cluster_id) if cluster_id else None
+ if cluster:
+ with _shared_container_lock:
+ if (
+ (
+ container_epoch is not None
+ and cluster.get("_shared_container_epoch")
+ != container_epoch
+ )
+ or cluster.get("_shared_container_id") != container_id
+ ):
+ logger.info(
+ "RDS: ignoring stale readiness result for cluster %s "
+ "epoch %s container %s",
+ cluster_id,
+ container_epoch,
+ container_id,
+ )
+ return
+ if not database_ready:
+ logger.warning(
+ "RDS: %s container for %s at %s:%s exited before "
+ "becoming reachable", engine, db_id,
+ ready_host, ready_port,
+ )
+ cluster["_shared_container_ready"] = False
+ for member in cluster.get("DBClusterMembers", []):
+ inst = _instances.get(
+ member.get("DBInstanceIdentifier"),
+ )
+ if inst is not None:
+ inst["DBInstanceStatus"] = "failed"
+ _refresh_cluster_status(cluster_id)
+ return
+
+ cluster["_shared_storage_initialized"] = True
+ pending_rotation = cluster.get(
+ "_pending_master_password_rotation",
+ )
+ if pending_rotation and not _rotate_real_password(
+ cluster,
+ pending_rotation["old_password"],
+ pending_rotation["new_password"],
+ ):
+ cluster["_shared_container_ready"] = False
+ for member in cluster.get("DBClusterMembers", []):
+ inst = _instances.get(
+ member.get("DBInstanceIdentifier"),
+ )
+ if inst is not None:
+ inst["DBInstanceStatus"] = "failed"
+ _refresh_cluster_status(cluster_id)
+ return
+ if pending_rotation:
+ cluster.pop("_pending_master_password_rotation", None)
+
+ if _is_mysql_engine(engine):
+ _grant_mysql_master_user_privileges(
+ ready_host, ready_port, master_user,
+ cluster.get("_MasterUserPassword", master_pass),
+ cluster_id,
+ )
+ cluster["_shared_container_ready"] = True
+ for member in cluster.get("DBClusterMembers", []):
+ inst = _instances.get(member.get("DBInstanceIdentifier"))
+ if inst is not None:
+ _attach_instance_to_shared_cluster(inst, cluster)
+ inst["DBInstanceStatus"] = "available"
+ _sync_cluster_endpoints(cluster)
+ _refresh_cluster_status(cluster_id)
+ if ms_network and internal_host:
+ logger.info(
+ "RDS: %s container for %s ready at %s:%s (network %s)",
+ engine, db_id, internal_host, internal_port, ms_network,
+ )
+ else:
+ logger.info(
+ "RDS: %s container for %s ready on port %s",
+ engine, db_id, endpoint_port,
+ )
+ return
+ if not database_ready:
logger.warning(
"RDS: %s container for %s at %s:%s exited before becoming reachable",
engine, db_id, ready_host, ready_port,
@@ -1308,15 +2256,12 @@ def _create_db_instance(p):
return
if _is_mysql_engine(engine):
_grant_mysql_master_user_privileges(
- ready_host, ready_port, master_user, master_pass, db_id,
+ ready_host, ready_port, master_user, master_pass,
+ cluster_id or db_id,
)
inst = _instances.get(db_id)
if inst is not None:
inst["DBInstanceStatus"] = "available"
- if cluster_id:
- cluster = _clusters.get(cluster_id)
- if cluster:
- _sync_cluster_endpoints(cluster)
_refresh_cluster_status(cluster_id)
if ms_network and internal_host:
logger.info(
@@ -1357,8 +2302,21 @@ def _delete_db_instance(p):
_unregister_instance_from_clusters(instance_id)
+ shared_cluster_id = (
+ instance.get("_shared_cluster_id")
+ or instance.get("DBClusterIdentifier")
+ )
+ if shared_cluster_id:
+ cluster = _resolve_cluster_in_request_region(shared_cluster_id)
+ if cluster and not cluster.get("DBClusterMembers"):
+ _stop_empty_cluster_shared_container(shared_cluster_id, cluster)
+
docker_client = _get_docker()
- if docker_client and instance.get("_docker_container_id"):
+ if (
+ docker_client
+ and instance.get("_docker_container_id")
+ and not instance.get("_shared_cluster_id")
+ ):
try:
c = docker_client.containers.get(instance["_docker_container_id"])
c.stop(timeout=5)
@@ -1463,6 +2421,18 @@ def _modify_db_instance(p):
return invalid_arn
return _error("DBInstanceNotFound", f"DBInstance {db_id} not found.", 404)
+ new_pass = _p(p, "MasterUserPassword")
+ if new_pass and (
+ instance.get("_shared_cluster_id")
+ or instance.get("DBClusterIdentifier")
+ ):
+ return _error(
+ "InvalidParameterCombination",
+ "MasterUserPassword cannot be modified on a DB instance that is "
+ "a member of a DB cluster. Use ModifyDBCluster instead.",
+ 400,
+ )
+
apply_immediately = _p(p, "ApplyImmediately") == "true"
field_map = {
@@ -1505,7 +2475,6 @@ def _modify_db_instance(p):
else:
pending[instance_key] = val
- new_pass = _p(p, "MasterUserPassword")
if new_pass:
old_pass = instance.get("_MasterUserPassword", "password")
instance["_MasterUserPassword"] = new_pass
@@ -1805,6 +2774,15 @@ def _create_db_cluster(p):
"AllocatedStorage": 1,
"Capacity": 0,
"ClusterScalabilityType": "standard",
+ "_shared_container_id": None,
+ "_shared_host_port": None,
+ "_shared_endpoint": None,
+ "_shared_volume_name": None,
+ "_shared_internal_address": None,
+ "_shared_internal_port": None,
+ "_shared_container_ready": True,
+ "_shared_container_epoch": 0,
+ "_shared_storage_initialized": False,
}
_clusters[cluster_id] = cluster
if global_cluster:
@@ -1837,14 +2815,50 @@ def _delete_db_cluster(p):
return _error("InvalidDBClusterStateFault",
"Cannot delete a DB cluster while it is a member of a global cluster.", 400)
+ if cluster.get("DBClusterMembers"):
+ return _error(
+ "InvalidDBClusterStateFault",
+ "Cannot delete a DB cluster while it contains DB instances.",
+ 400,
+ )
+
skip_snapshot = _p(p, "SkipFinalSnapshot") == "true"
final_snap_id = _p(p, "FinalDBSnapshotIdentifier")
if not skip_snapshot and final_snap_id:
pass
- cluster["Status"] = "deleting"
- _tags.pop(cluster["DBClusterArn"], None)
- del _clusters[cluster["DBClusterIdentifier"]]
+ # Serialize cluster identity and resource teardown with warm-boot startup.
+ # The membership can become empty between the initial check and this point
+ # when the last member is being deleted concurrently.
+ with _shared_container_lock:
+ current_cluster = _resolve_cluster_in_request_region(cluster_id)
+ if current_cluster is not cluster:
+ return _error(
+ "DBClusterNotFoundFault",
+ f"DBCluster {cluster_id} not found.",
+ 404,
+ )
+ if cluster.get("DBClusterMembers"):
+ return _error(
+ "InvalidDBClusterStateFault",
+ "Cannot delete a DB cluster while it contains DB instances.",
+ 400,
+ )
+ cluster["Status"] = "deleting"
+ if any(
+ cluster.get(field)
+ for field in (
+ "_shared_container_id",
+ "_shared_endpoint",
+ "_shared_volume_name",
+ )
+ ):
+ _remove_cluster_shared_resources(
+ cluster["DBClusterIdentifier"],
+ cluster,
+ )
+ _tags.pop(cluster["DBClusterArn"], None)
+ del _clusters[cluster["DBClusterIdentifier"]]
return _xml(200, "DeleteDBClusterResponse",
f"<DeleteDBClusterResult><DBCluster>{_cluster_xml(cluster)}</DBCluster></DeleteDBClusterResult>")
@@ -1871,58 +2885,100 @@ def _describe_db_clusters(p):
def _rotate_real_password(cluster, old_pass, new_pass):
- """Alter the root password on the real MySQL/MariaDB container."""
+ """Rotate the real cluster master login before publishing new metadata."""
cluster_id = cluster.get("DBClusterIdentifier", "")
+ engine = cluster.get("Engine", "")
+ master_user = cluster.get("MasterUsername", "admin")
+ db_name = cluster.get("DatabaseName") or "mydb"
+ host = cluster.get("_shared_internal_address")
+ port = cluster.get("_shared_internal_port")
for inst in _instances.values():
if inst.get("DBClusterIdentifier") != cluster_id:
continue
- engine = inst.get("Engine", "")
- if not any(e in engine for e in ("mysql", "aurora-mysql", "mariadb")):
- continue
- host = inst.get("_internal_address")
- port = inst.get("_internal_port")
+ engine = engine or inst.get("Engine", "")
+ host = host or inst.get("_internal_address")
+ port = port or inst.get("_internal_port")
if not host or not port:
endpoint = inst.get("Endpoint", {})
if not isinstance(endpoint, dict) or not endpoint.get("Port"):
continue
host = endpoint.get("Address", "localhost")
- port = int(endpoint.get("Port", 3306))
- try:
+ port = int(endpoint["Port"])
+ break
+ if not host or not port:
+ logger.warning(
+ "RDS: password rotation failed on %s: no reachable endpoint",
+ cluster_id,
+ )
+ return False
+
+ conn = None
+ cur = None
+ try:
+ if _is_mysql_engine(engine):
import pymysql
- except Exception as e:
- logger.warning("RDS: password rotation failed on %s: %s",
- cluster_id, e)
- return False
- conn = None
- cur = None
- try:
+
conn = pymysql.connect(
- host=host, port=port, user="root",
- password=old_pass, autocommit=True)
+ host=host,
+ port=int(port),
+ user="root",
+ password=old_pass,
+ autocommit=True,
+ )
cur = conn.cursor()
+ if master_user != "root":
+ cur.execute(
+ "ALTER USER %s@'%%' IDENTIFIED BY %s",
+ (master_user, new_pass),
+ )
cur.execute(
- "ALTER USER 'root'@'%%' IDENTIFIED BY %s", (new_pass,))
- cur.close()
- cur = None
- conn.close()
- conn = None
- logger.info("RDS: rotated root password on %s", cluster_id)
+ "ALTER USER 'root'@'%%' IDENTIFIED BY %s",
+ (new_pass,),
+ )
+ elif _is_postgres_engine(engine):
+ import psycopg2
+ from psycopg2 import sql as _pgsql
+
+ conn = psycopg2.connect(
+ host=host,
+ port=int(port),
+ user=master_user,
+ password=old_pass,
+ dbname=db_name,
+ )
+ conn.autocommit = True
+ cur = conn.cursor()
+ cur.execute(
+ _pgsql.SQL("ALTER USER {role} WITH PASSWORD %s").format(
+ role=_pgsql.Identifier(master_user),
+ ),
+ (new_pass,),
+ )
+ else:
return True
- except Exception as e:
- if cur:
- try:
- cur.close()
- except Exception:
- pass
- if conn:
- try:
- conn.close()
- except Exception:
- pass
- logger.warning("RDS: password rotation failed on %s: %s",
- cluster_id, e)
- return False
- return True
+
+ cur.close()
+ cur = None
+ conn.close()
+ conn = None
+ logger.info("RDS: rotated master password on %s", cluster_id)
+ return True
+ except Exception as e:
+ logger.warning(
+ "RDS: password rotation failed on %s: %s", cluster_id, e,
+ )
+ return False
+ finally:
+ if cur:
+ try:
+ cur.close()
+ except Exception:
+ pass
+ if conn:
+ try:
+ conn.close()
+ except Exception:
+ pass
def _modify_db_cluster(p):
@@ -1938,9 +2994,37 @@ def _modify_db_cluster(p):
cluster["EngineVersion"] = _p(p, "EngineVersion")
if _p(p, "MasterUserPassword"):
new_pass = _p(p, "MasterUserPassword")
- old_pass = cluster.get("_MasterUserPassword", "password")
- cluster["_MasterUserPassword"] = new_pass
- _rotate_real_password(cluster, old_pass, new_pass)
+ with _shared_container_lock:
+ old_pass = cluster.get("_MasterUserPassword", "password")
+ cluster["_MasterUserPassword"] = new_pass
+ pending_rotation = cluster.get(
+ "_pending_master_password_rotation",
+ )
+ rotation_old_pass = (
+ pending_rotation["old_password"]
+ if pending_rotation
+ else old_pass
+ )
+ has_available_compute = bool(
+ cluster.get("DBClusterMembers"),
+ ) and bool(
+ cluster.get("_shared_container_id"),
+ ) and cluster.get("_shared_container_ready", True)
+ if has_available_compute and _rotate_real_password(
+ cluster, rotation_old_pass, new_pass,
+ ):
+ cluster.pop("_pending_master_password_rotation", None)
+ elif (
+ cluster.get("_shared_container_id")
+ or cluster.get("_shared_storage_initialized")
+ ):
+ # The stopped preserved container still has rotation_old_pass.
+ # The first replacement member authenticates with it, applies
+ # the pending rotation, then publishes readiness using new_pass.
+ cluster["_pending_master_password_rotation"] = {
+ "old_password": rotation_old_pass,
+ "new_password": new_pass,
+ }
if _p(p, "Port"):
cluster["Port"] = int(_p(p, "Port"))
if _p(p, "BackupRetentionPeriod"):
@@ -3903,28 +4987,65 @@ _ACTION_MAP = {
def reset():
- docker_client = _get_docker()
- if docker_client:
- for instance in _instances.all_values():
- cid = instance.get("_docker_container_id")
- if cid:
- try:
- c = docker_client.containers.get(cid)
- c.stop(timeout=2)
- c.remove(v=True)
- except Exception as e:
- logger.warning("reset: failed to stop/remove container %s: %s", cid, e)
- _instances.clear()
- _clusters.clear()
- _subnet_groups.clear()
- _param_groups.clear()
- _snapshots.clear()
- _db_cluster_param_groups.clear()
- _db_cluster_snapshots.clear()
- _option_groups.clear()
- _global_clusters.clear()
- _tags.clear()
- _port_counter[0] = BASE_PORT
+ # Serialize teardown with warm-boot shared-container startup. Otherwise a
+ # restore worker can pass its membership check after reset has enumerated
+ # resources, then create a container after the stores are cleared.
+ with _shared_container_lock:
+ docker_client = _get_docker()
+ if docker_client:
+ # Shared containers are cluster-owned. Reap them once from the cluster
+ # records before considering standalone instances; otherwise every
+ # member would try to remove the same container and an empty cluster
+ # would leak its backing database.
+ shared_container_ids = {
+ cluster.get("_shared_container_id")
+ for cluster in _clusters.all_values()
+ if cluster.get("_shared_container_id")
+ }
+ for (account_id, region, cluster_id), cluster in _clusters.all_items():
+ if any(
+ cluster.get(field)
+ for field in (
+ "_shared_container_id",
+ "_shared_endpoint",
+ "_shared_volume_name",
+ )
+ ):
+ _remove_cluster_shared_resources(
+ cluster_id,
+ cluster,
+ timeout=2,
+ account_id=account_id,
+ region=region,
+ )
+ for instance in _instances.all_values():
+ cid = instance.get("_docker_container_id")
+ if (
+ cid
+ and cid not in shared_container_ids
+ and not instance.get("_shared_cluster_id")
+ ):
+ try:
+ c = docker_client.containers.get(cid)
+ c.stop(timeout=2)
+ c.remove(v=True)
+ except Exception as e:
+ logger.warning(
+ "reset: failed to stop/remove container %s: %s",
+ cid,
+ e,
+ )
+ _instances.clear()
+ _clusters.clear()
+ _subnet_groups.clear()
+ _param_groups.clear()
+ _snapshots.clear()
+ _db_cluster_param_groups.clear()
+ _db_cluster_snapshots.clear()
+ _option_groups.clear()
+ _global_clusters.clear()
+ _tags.clear()
+ _port_counter[0] = BASE_PORT
# Load persisted state at module import. Must run AFTER every helper this
diff --git a/ministack/services/rds_data.py b/ministack/services/rds_data.py
index 7508ba7c..a0622790 100644
--- a/ministack/services/rds_data.py
+++ b/ministack/services/rds_data.py
@@ -249,13 +249,50 @@ def _resolve_cluster(resource_arn):
engine = cluster.get("Engine", "postgres")
cluster_id = cluster["DBClusterIdentifier"]
+ # Aurora Serverless has no customer-managed DB instances and retains its
+ # intentional in-memory Data API stub. Provisioned clusters, however, must
+ # have an available member before SQL can run.
+ if cluster.get("EngineMode") == "serverless":
+ return cluster, engine
+
+ member_ids = {
+ member.get("DBInstanceIdentifier")
+ for member in cluster.get("DBClusterMembers", [])
+ if member.get("DBInstanceIdentifier")
+ }
+ if not member_ids or not cluster.get("_shared_container_ready", True):
+ return None, engine
+
# Find an instance belonging to this cluster
for inst in rds._instances.values_scoped(spec.account_id, spec.region):
- if inst.get("DBClusterIdentifier") == cluster_id:
+ if (
+ inst.get("DBClusterIdentifier") == cluster_id
+ and inst.get("DBInstanceIdentifier") in member_ids
+ and inst.get("DBInstanceStatus") == "available"
+ ):
+ # Aurora members intentionally share one backing container. During
+ # creation, recover its endpoint from the cluster if this member's
+ # control-plane record has not been stamped yet.
+ if not inst.get("Endpoint") and cluster.get("_shared_endpoint"):
+ inst["Endpoint"] = dict(cluster["_shared_endpoint"])
+ inst["_docker_container_id"] = cluster.get("_shared_container_id")
+ inst["_internal_address"] = cluster.get("_shared_internal_address")
+ inst["_internal_port"] = cluster.get("_shared_internal_port")
return inst, engine
- # No instance found — return cluster info but no connectable instance
- return cluster, engine
+ # The cluster exists, but it has no available compute to accept SQL.
+ return None, engine
+
+
+def _cluster_resolution_error(resource_arn, engine):
+ if engine:
+ return _transient_database_error(
+ f"Database cluster has no available DB instances: {resource_arn}",
+ )
+ return _error(
+ "BadRequestException",
+ f"Database cluster not found for ARN: {resource_arn}",
+ )
def _get_secret_credentials(secret_arn):
@@ -490,8 +527,7 @@ def _execute_statement(data):
instance, engine = _resolve_cluster(resource_arn)
if not instance:
- return _error("BadRequestException",
- f"Database cluster not found for ARN: {resource_arn}")
+ return _cluster_resolution_error(resource_arn, engine)
# Keep stub mode for intentional mock environments only. Once RDS has a
# real container-backed endpoint, connection failures must surface as
@@ -570,8 +606,7 @@ def _begin_transaction(data):
instance, engine = _resolve_cluster(resource_arn)
if not instance:
- return _error("BadRequestException",
- f"Database cluster not found for ARN: {resource_arn}")
+ return _cluster_resolution_error(resource_arn, engine)
secret_user, password = _get_secret_credentials(secret_arn)
@@ -656,8 +691,7 @@ def _batch_execute_statement(data):
instance, engine = _resolve_cluster(resource_arn)
if not instance:
- return _error("BadRequestException",
- f"Database cluster not found for ARN: {resource_arn}")
+ return _cluster_resolution_error(resource_arn, engine)
secret_user, password = _get_secret_credentials(secret_arn)
diff --git a/tests/test_rds.py b/tests/test_rds.py
index 444e6479..48469f6b 100644
--- a/tests/test_rds.py
+++ b/tests/test_rds.py
@@ -2,7 +2,9 @@ import asyncio
import io
import json
import os
+import sys
import time
+import types
import uuid as _uuid_mod
import zipfile
from urllib.parse import urlparse
@@ -1117,6 +1119,1279 @@ def test_rds_aurora_cluster_endpoints_follow_backing_instance(rds):
assert cluster["Port"] == inst["Endpoint"]["Port"]
+def test_rds_aurora_cluster_members_share_one_container(monkeypatch):
+ """Cluster members attach to one cluster-owned container and member delete keeps it alive."""
+ import threading
+
+ from ministack.services import rds as m
+
+ runs = []
+ containers = {}
+ next_ports = []
+ removed_volumes = []
+ wait_calls = []
+ stale_wait_started = threading.Event()
+ release_stale_wait = threading.Event()
+
+ class FakeContainer:
+ def __init__(self, name):
+ self.id = "cluster-container-id"
+ self.name = name
+ self.status = "running"
+ self.attrs = {"NetworkSettings": {"Networks": {}}}
+ self.stop_calls = 0
+ self.start_calls = 0
+ self.remove_calls = 0
+
+ def reload(self):
+ pass
+
+ def stop(self, timeout=5):
+ self.stop_calls += 1
+ self.status = "exited"
+
+ def start(self):
+ self.start_calls += 1
+ self.status = "running"
+
+ def remove(self, v=False, force=False):
+ self.remove_calls += 1
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ runs.append(kwargs)
+ container = FakeContainer(kwargs["name"])
+ containers[container.id] = container
+ containers[container.name] = container
+ return container
+
+ def get(self, identifier):
+ if identifier not in containers:
+ raise Exception("not found")
+ return containers[identifier]
+
+ class FakeVolume:
+ def __init__(self, name):
+ self.name = name
+
+ def remove(self):
+ removed_volumes.append(self.name)
+
+ class FakeVolumes:
+ def get(self, name):
+ return FakeVolume(name)
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+ self.volumes = FakeVolumes()
+
+ fake_docker = FakeDocker()
+
+ def _next_port():
+ port = 16000 + len(next_ports)
+ next_ports.append(port)
+ return port
+
+ def _wait_for_database_ready(*_args):
+ wait_calls.append(len(wait_calls) + 1)
+ if len(wait_calls) == 2:
+ stale_wait_started.set()
+ release_stale_wait.wait(timeout=2)
+ return False
+ return True
+
+ monkeypatch.setattr(m, "_get_docker", lambda: fake_docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", _next_port)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", _wait_for_database_ready)
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", lambda *_args: None)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "shared-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ for db_id in ("shared-writer", "shared-reader"):
+ m._create_db_instance({
+ "DBInstanceIdentifier": db_id,
+ "DBClusterIdentifier": "shared-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if all(
+ m._instances.get(db_id, {}).get("DBInstanceStatus") == "available"
+ for db_id in ("shared-writer", "shared-reader")
+ ):
+ break
+ time.sleep(0.01)
+
+ cluster = m._clusters.get("shared-cluster")
+ writer = m._instances.get("shared-writer")
+ reader = m._instances.get("shared-reader")
+ assert len(runs) == 1
+ assert next_ports == [16000]
+ assert runs[0]["name"] == m._rds_cluster_docker_name("shared-cluster")
+ assert runs[0]["labels"]["cluster_id"] == "shared-cluster"
+ volume_name = m._rds_cluster_docker_volume_name("shared-cluster")
+ assert runs[0]["volumes"] == {
+ volume_name: {"bind": "/var/lib/mysql", "mode": "rw"},
+ }
+ assert writer["_docker_container_id"] == reader["_docker_container_id"]
+ assert writer["Endpoint"] == reader["Endpoint"] == cluster["_shared_endpoint"]
+ assert cluster["Endpoint"] == cluster["ReaderEndpoint"] == writer["Endpoint"]["Address"]
+
+ persisted = m.get_state()
+ assert "_shared_container_id" not in persisted["clusters"].get("shared-cluster")
+ assert "_docker_container_id" not in persisted["instances"].get("shared-writer")
+
+ container = containers[writer["_docker_container_id"]]
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "shared-reader",
+ "SkipFinalSnapshot": "true",
+ })
+ assert container.stop_calls == 0
+ assert container.remove_calls == 0
+
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "shared-writer",
+ "SkipFinalSnapshot": "true",
+ })
+ assert cluster["Status"] == "available"
+ assert cluster["DBClusterMembers"] == []
+ assert cluster["_shared_container_ready"] is False
+ assert container.stop_calls == 1
+ assert container.remove_calls == 0
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "shared-stale-restart",
+ "DBClusterIdentifier": "shared-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ assert stale_wait_started.wait(timeout=1)
+ stale_epoch = cluster["_shared_container_epoch"]
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "shared-stale-restart",
+ "SkipFinalSnapshot": "true",
+ })
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "shared-replacement",
+ "DBClusterIdentifier": "shared-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ replacement = m._instances.get("shared-replacement")
+ if replacement and replacement.get("DBInstanceStatus") == "available":
+ break
+ time.sleep(0.01)
+ replacement = m._instances.get("shared-replacement")
+ assert replacement["DBInstanceStatus"] == "available"
+ assert replacement["_docker_container_id"] == container.id
+ assert cluster["_shared_container_ready"] is True
+ assert cluster["_shared_container_epoch"] > stale_epoch
+ release_stale_wait.set()
+ time.sleep(0.05)
+ assert replacement["DBInstanceStatus"] == "available"
+ assert cluster["_shared_container_ready"] is True
+ assert container.start_calls == 2
+ assert len(runs) == 1
+
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "shared-replacement",
+ "SkipFinalSnapshot": "true",
+ })
+ assert container.stop_calls == 3
+ assert container.remove_calls == 0
+
+ m._delete_db_cluster({
+ "DBClusterIdentifier": "shared-cluster",
+ "SkipFinalSnapshot": "true",
+ })
+ assert container.stop_calls == 4
+ assert container.remove_calls == 1
+ assert removed_volumes == [volume_name]
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_new_member_retries_failed_shared_container(monkeypatch):
+ """A new member restarts shared compute after its first boot fails."""
+ from ministack.services import rds as m
+
+ runs = []
+ readiness_calls = []
+
+ class FakeContainer:
+ def __init__(self, name):
+ self.id = "failed-shared-container"
+ self.name = name
+ self.status = "running"
+ self.attrs = {"NetworkSettings": {"Networks": {}}}
+ self.start_calls = 0
+
+ def reload(self):
+ pass
+
+ def start(self):
+ self.start_calls += 1
+ self.status = "running"
+
+ container = FakeContainer(
+ m._rds_cluster_docker_name("failed-retry-cluster"),
+ )
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ runs.append(kwargs)
+ return container
+
+ def get(self, identifier):
+ if identifier in (container.id, container.name):
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ def _wait_for_database_ready(*_args):
+ readiness_calls.append(len(readiness_calls) + 1)
+ if len(readiness_calls) == 1:
+ container.status = "exited"
+ return False
+ return True
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16041)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", _wait_for_database_ready)
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", lambda *_args: None)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "failed-retry-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "failed-original",
+ "DBClusterIdentifier": "failed-retry-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if m._instances["failed-original"]["DBInstanceStatus"] == "failed":
+ break
+ time.sleep(0.01)
+
+ cluster = m._clusters["failed-retry-cluster"]
+ assert m._instances["failed-original"]["DBInstanceStatus"] == "failed"
+ assert cluster["_shared_container_ready"] is False
+ assert cluster["_shared_container_id"] == container.id
+ assert container.status == "exited"
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "failed-replacement",
+ "DBClusterIdentifier": "failed-retry-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if all(
+ m._instances[db_id]["DBInstanceStatus"] == "available"
+ for db_id in ("failed-original", "failed-replacement")
+ ):
+ break
+ time.sleep(0.01)
+
+ assert readiness_calls == [1, 2]
+ assert container.start_calls == 1
+ assert len(runs) == 1
+ assert cluster["_shared_container_ready"] is True
+ assert all(
+ m._instances[db_id]["DBInstanceStatus"] == "available"
+ for db_id in ("failed-original", "failed-replacement")
+ )
+ assert all(
+ m._instances[db_id]["_docker_container_id"] == container.id
+ for db_id in ("failed-original", "failed-replacement")
+ )
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_password_change_after_failed_shared_start_uses_new_password(
+ monkeypatch,
+):
+ """A fresh retry must not rotate credentials on compute that never started."""
+ from ministack.services import rds as m
+
+ runs = []
+ readiness_passwords = []
+ rotation_calls = []
+
+ class FakeContainer:
+ id = "password-retry-container"
+ status = "running"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def reload(self):
+ pass
+
+ container = FakeContainer()
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ runs.append(kwargs)
+ if len(runs) == 1:
+ raise RuntimeError("initial start failed")
+ return container
+
+ def get(self, identifier):
+ if identifier == container.id:
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ def _wait_for_database_ready(_host, _port, _engine, _user, password, *_args):
+ readiness_passwords.append(password)
+ return True
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16042)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", _wait_for_database_ready)
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", lambda *_args: None)
+ monkeypatch.setattr(
+ m,
+ "_rotate_real_password",
+ lambda *_args: rotation_calls.append(True) or False,
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "password-retry-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "old-password",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "password-retry-original",
+ "DBClusterIdentifier": "password-retry-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+
+ cluster = m._clusters["password-retry-cluster"]
+ original = m._instances["password-retry-original"]
+ assert original["DBInstanceStatus"] == "failed"
+ assert cluster["_shared_container_id"] is None
+ assert cluster["_shared_container_ready"] is False
+
+ m._modify_db_cluster({
+ "DBClusterIdentifier": "password-retry-cluster",
+ "MasterUserPassword": "new-password",
+ })
+ assert rotation_calls == []
+ assert "_pending_master_password_rotation" not in cluster
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "password-retry-replacement",
+ "DBClusterIdentifier": "password-retry-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if all(
+ m._instances[db_id]["DBInstanceStatus"] == "available"
+ for db_id in (
+ "password-retry-original",
+ "password-retry-replacement",
+ )
+ ):
+ break
+ time.sleep(0.01)
+
+ assert len(runs) == 2
+ assert runs[1]["environment"]["MYSQL_ROOT_PASSWORD"] == "new-password"
+ assert readiness_passwords == ["new-password"]
+ assert all(
+ m._instances[db_id]["DBInstanceStatus"] == "available"
+ for db_id in (
+ "password-retry-original",
+ "password-retry-replacement",
+ )
+ )
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_restored_initialized_storage_defers_password_rotation(monkeypatch):
+ """Persisted initialized storage still contains the previous password."""
+ from ministack.services import rds as m
+
+ rotation_calls = []
+ monkeypatch.setattr(
+ m,
+ "_rotate_real_password",
+ lambda *_args: rotation_calls.append(True) or False,
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "restored-password-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "old-password",
+ })
+ cluster = m._clusters["restored-password-cluster"]
+ cluster.update({
+ "_shared_container_id": "stopped-shared-container",
+ "_shared_storage_initialized": True,
+ "_shared_volume_name": "restored-password-volume",
+ "_shared_container_ready": False,
+ })
+
+ persisted = m.get_state()
+ persisted_cluster = persisted["clusters"]["restored-password-cluster"]
+ assert "_shared_container_id" not in persisted_cluster
+ assert persisted_cluster["_shared_storage_initialized"] is True
+
+ m._clusters.clear()
+ m.restore_state(persisted)
+ restored = m._clusters["restored-password-cluster"]
+ assert restored["_shared_container_id"] is None
+
+ m._modify_db_cluster({
+ "DBClusterIdentifier": "restored-password-cluster",
+ "MasterUserPassword": "new-password",
+ })
+
+ assert rotation_calls == []
+ assert restored["_pending_master_password_rotation"] == {
+ "old_password": "old-password",
+ "new_password": "new-password",
+ }
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_last_member_delete_wins_after_readiness_epoch_check(monkeypatch):
+ """A worker already finalizing readiness cannot revive stopped compute."""
+ import threading
+
+ from ministack.services import rds as m
+
+ grant_entered = threading.Event()
+ release_grant = threading.Event()
+ stop_entered = threading.Event()
+ delete_done = threading.Event()
+ grant_calls = []
+
+ class FakeContainer:
+ id = "readiness-race-container"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def __init__(self):
+ self.status = "running"
+
+ def reload(self):
+ pass
+
+ def start(self):
+ self.status = "running"
+
+ def stop(self, timeout=5):
+ self.status = "exited"
+
+ container = FakeContainer()
+
+ class FakeContainers:
+ def run(self, **_kwargs):
+ return container
+
+ def get(self, identifier):
+ if identifier in (
+ container.id,
+ m._rds_cluster_docker_name("readiness-race-cluster"),
+ ):
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16020)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+
+ def _grant(*_args):
+ grant_calls.append(True)
+ if len(grant_calls) == 2:
+ grant_entered.set()
+ release_grant.wait(timeout=2)
+
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", _grant)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "readiness-race-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "readiness-race-original",
+ "DBClusterIdentifier": "readiness-race-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if m._instances.get("readiness-race-original", {}).get(
+ "DBInstanceStatus",
+ ) == "available":
+ break
+ time.sleep(0.01)
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "readiness-race-original",
+ "SkipFinalSnapshot": "true",
+ })
+
+ original_stop = m._stop_empty_cluster_shared_container
+
+ def _observed_stop(cluster_id, cluster):
+ stop_entered.set()
+ return original_stop(cluster_id, cluster)
+
+ monkeypatch.setattr(
+ m, "_stop_empty_cluster_shared_container", _observed_stop,
+ )
+ m._create_db_instance({
+ "DBInstanceIdentifier": "readiness-race-replacement",
+ "DBClusterIdentifier": "readiness-race-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ assert grant_entered.wait(timeout=1)
+
+ def _delete_replacement():
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "readiness-race-replacement",
+ "SkipFinalSnapshot": "true",
+ })
+ delete_done.set()
+
+ delete_thread = threading.Thread(target=_delete_replacement)
+ delete_thread.start()
+ assert stop_entered.wait(timeout=1)
+ assert not delete_done.is_set()
+ release_grant.set()
+ delete_thread.join(timeout=2)
+
+ cluster = m._clusters.get("readiness-race-cluster")
+ assert delete_done.is_set()
+ assert cluster["DBClusterMembers"] == []
+ assert cluster["_shared_container_ready"] is False
+ assert container.status == "exited"
+ finally:
+ release_grant.set()
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_stale_readiness_ignores_same_id_cluster_recreation(monkeypatch):
+ """Epoch reuse cannot let an old container worker mutate a new cluster."""
+ from ministack.services import rds as m
+
+ workers = []
+ containers = {}
+ ports = iter((16050, 16051))
+ run_count = [0]
+
+ class DeferredThread:
+ def __init__(self, target, args=(), **_kwargs):
+ self.target = target
+ self.args = args
+
+ def start(self):
+ workers.append(self)
+
+ class FakeContainer:
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def __init__(self, name):
+ run_count[0] += 1
+ self.id = f"recreated-container-{run_count[0]}"
+ self.name = name
+ self.status = "running"
+
+ def reload(self):
+ pass
+
+ def stop(self, timeout=5):
+ self.status = "exited"
+
+ def remove(self, **_kwargs):
+ containers.pop(self.id, None)
+ containers.pop(self.name, None)
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ container = FakeContainer(kwargs["name"])
+ containers[container.id] = container
+ containers[container.name] = container
+ return container
+
+ def get(self, identifier):
+ if identifier not in containers:
+ raise Exception("not found")
+ return containers[identifier]
+
+ class FakeVolume:
+ def remove(self):
+ pass
+
+ class FakeVolumes:
+ def get(self, _name):
+ return FakeVolume()
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+ self.volumes = FakeVolumes()
+
+ monkeypatch.setattr(m.threading, "Thread", DeferredThread)
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: next(ports))
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(
+ m,
+ "_wait_for_database_ready",
+ lambda _host, port, *_args: port == 16051,
+ )
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", lambda *_args: None)
+
+ cluster_id = "recreated-readiness-cluster"
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ for member_id in ("old-member", "new-member"):
+ m._create_db_cluster({
+ "DBClusterIdentifier": cluster_id,
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": member_id,
+ "DBClusterIdentifier": cluster_id,
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ if member_id == "old-member":
+ old_container_id = m._clusters[cluster_id][
+ "_shared_container_id"
+ ]
+ m._delete_db_instance({
+ "DBInstanceIdentifier": member_id,
+ "SkipFinalSnapshot": "true",
+ })
+ m._delete_db_cluster({
+ "DBClusterIdentifier": cluster_id,
+ "SkipFinalSnapshot": "true",
+ })
+
+ cluster = m._clusters[cluster_id]
+ new_container_id = cluster["_shared_container_id"]
+ assert old_container_id != new_container_id
+ assert len(workers) == 2
+
+ # Finalize the replacement first. Both cluster incarnations use epoch
+ # 1, so epoch-only validation cannot distinguish the old worker.
+ workers[1].target(*workers[1].args)
+ assert cluster["_shared_container_epoch"] == 1
+ assert cluster["_shared_container_ready"] is True
+ assert m._instances["new-member"]["DBInstanceStatus"] == "available"
+
+ workers[0].target(*workers[0].args)
+ assert cluster["_shared_container_ready"] is True
+ assert m._instances["new-member"]["DBInstanceStatus"] == "available"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_empty_control_plane_cluster_accepts_new_member(monkeypatch):
+ """A no-Docker cluster becomes connectable again when compute returns."""
+ from ministack.services import rds as m
+
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "control-plane-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "control-plane-original",
+ "DBClusterIdentifier": "control-plane-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "control-plane-original",
+ "SkipFinalSnapshot": "true",
+ })
+
+ cluster = m._clusters.get("control-plane-cluster")
+ assert cluster["DBClusterMembers"] == []
+ assert cluster["_shared_container_ready"] is False
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "control-plane-replacement",
+ "DBClusterIdentifier": "control-plane-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ replacement = m._instances.get("control-plane-replacement")
+ assert replacement["DBInstanceStatus"] == "available"
+ assert cluster["_shared_container_ready"] is True
+ assert [
+ member["DBInstanceIdentifier"]
+ for member in cluster["DBClusterMembers"]
+ ] == ["control-plane-replacement"]
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_invalid_member_does_not_restart_empty_cluster(monkeypatch):
+ """Request validation happens before a stopped cluster becomes reachable."""
+ from ministack.services import rds as m
+
+ start_calls = []
+
+ class FakeContainer:
+ status = "exited"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def start(self):
+ start_calls.append(True)
+
+ def reload(self):
+ pass
+
+ class FakeContainers:
+ def get(self, identifier):
+ assert identifier == "stopped-shared-container"
+ return FakeContainer()
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ m._instances.clear()
+ m._clusters.clear()
+ m._param_groups.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "invalid-member-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ cluster = m._clusters["invalid-member-cluster"]
+ cluster.update({
+ "_shared_container_id": "stopped-shared-container",
+ "_shared_host_port": 16042,
+ "_shared_endpoint": {
+ "Address": "localhost",
+ "Port": 16042,
+ "HostedZoneId": "Z2R2ITUGPM61AM",
+ },
+ "_shared_container_ready": False,
+ })
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "invalid-member",
+ "DBClusterIdentifier": "invalid-member-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ "DBParameterGroupName": "missing-custom-group",
+ })
+
+ assert start_calls == []
+ assert "invalid-member" not in m._instances
+ assert cluster["DBClusterMembers"] == []
+ assert cluster["_shared_container_ready"] is False
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+ m._param_groups.clear()
+
+
+def test_rds_cluster_member_connection_fields_are_normalized(monkeypatch):
+ from ministack.services import rds as m
+
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "credential-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "cluster_admin",
+ "MasterUserPassword": "cluster-password",
+ "DatabaseName": "cluster_db",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "credential-member",
+ "DBClusterIdentifier": "credential-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-postgresql",
+ "MasterUsername": "wrong_admin",
+ "MasterUserPassword": "wrong-password",
+ "DBName": "wrong_db",
+ })
+
+ member = m._instances.get("credential-member")
+ assert member["Engine"] == "aurora-mysql"
+ assert member["MasterUsername"] == "cluster_admin"
+ assert member["_MasterUserPassword"] == "cluster-password"
+ assert member["DBName"] == "cluster_db"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
[email protected]("rotation_succeeds", [True, False])
+def test_rds_empty_cluster_applies_pending_password_on_restart(
+ monkeypatch, rotation_succeeds,
+):
+ from ministack.services import rds as m
+
+ readiness_passwords = []
+ rotations = []
+ grants = []
+
+ class FakeContainer:
+ id = "pending-password-container"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+
+ def __init__(self):
+ self.status = "exited"
+
+ def start(self):
+ self.status = "running"
+
+ def reload(self):
+ pass
+
+ container = FakeContainer()
+
+ class FakeContainers:
+ def get(self, identifier):
+ assert identifier == container.id
+ return container
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ def _wait_for_database_ready(_host, _port, _engine, _user, password, *_args):
+ readiness_passwords.append(password)
+ return True
+
+ def _rotate(_cluster, old_password, new_password):
+ rotations.append((old_password, new_password))
+ return rotation_succeeds
+
+ def _grant(_host, _port, user, password, db_id):
+ grants.append((user, password, db_id))
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_wait_for_database_ready", _wait_for_database_ready)
+ monkeypatch.setattr(m, "_rotate_real_password", _rotate)
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", _grant)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "pending-password-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "old-password",
+ "DatabaseName": "appdb",
+ })
+ cluster = m._clusters.get("pending-password-cluster")
+ cluster.update({
+ "_shared_container_id": container.id,
+ "_shared_host_port": 16021,
+ "_shared_endpoint": {
+ "Address": "localhost",
+ "Port": 16021,
+ "HostedZoneId": "Z2R2ITUGPM61AM",
+ },
+ "_shared_container_ready": False,
+ "_shared_container_epoch": 1,
+ })
+
+ m._modify_db_cluster({
+ "DBClusterIdentifier": "pending-password-cluster",
+ "MasterUserPassword": "new-password",
+ })
+ assert cluster["_pending_master_password_rotation"] == {
+ "old_password": "old-password",
+ "new_password": "new-password",
+ }
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "pending-password-writer",
+ "DBClusterIdentifier": "pending-password-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ "MasterUserPassword": "member-password-must-be-ignored",
+ })
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if m._instances.get("pending-password-writer", {}).get(
+ "DBInstanceStatus",
+ ) in {"available", "failed"}:
+ break
+ time.sleep(0.01)
+
+ member = m._instances.get("pending-password-writer")
+ assert readiness_passwords == ["old-password"]
+ assert rotations == [("old-password", "new-password")]
+ assert member["_MasterUserPassword"] == "new-password"
+ if rotation_succeeds:
+ assert member["DBInstanceStatus"] == "available"
+ assert cluster["_shared_container_ready"] is True
+ assert "_pending_master_password_rotation" not in cluster
+ assert grants == [
+ ("admin", "new-password", "pending-password-cluster"),
+ ]
+ else:
+ assert member["DBInstanceStatus"] == "failed"
+ assert cluster["_shared_container_ready"] is False
+ assert cluster["_pending_master_password_rotation"] == {
+ "old_password": "old-password",
+ "new_password": "new-password",
+ }
+ assert grants == []
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_rotate_real_mysql_master_password(monkeypatch):
+ from ministack.services import rds as m
+
+ connects = []
+ executions = []
+
+ class FakeCursor:
+ def execute(self, query, params):
+ executions.append((query, params))
+
+ def close(self):
+ pass
+
+ class FakeConnection:
+ def cursor(self):
+ return FakeCursor()
+
+ def close(self):
+ pass
+
+ def _connect(**kwargs):
+ connects.append(kwargs)
+ return FakeConnection()
+
+ fake_pymysql = types.ModuleType("pymysql")
+ fake_pymysql.connect = _connect
+ monkeypatch.setitem(sys.modules, "pymysql", fake_pymysql)
+ m._instances.clear()
+ try:
+ m._instances["rotation-member"] = {
+ "DBInstanceIdentifier": "rotation-member",
+ "DBClusterIdentifier": "rotation-cluster",
+ "Endpoint": {"Address": "localhost", "Port": 16031},
+ }
+ assert m._rotate_real_password({
+ "DBClusterIdentifier": "rotation-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "app_admin",
+ }, "old-password", "new-password") is True
+
+ assert connects == [{
+ "host": "localhost",
+ "port": 16031,
+ "user": "root",
+ "password": "old-password",
+ "autocommit": True,
+ }]
+ assert executions == [
+ (
+ "ALTER USER %s@'%%' IDENTIFIED BY %s",
+ ("app_admin", "new-password"),
+ ),
+ ("ALTER USER 'root'@'%%' IDENTIFIED BY %s", ("new-password",)),
+ ]
+ finally:
+ m._instances.clear()
+
+
+def test_rds_password_change_serializes_with_readiness_finalization(monkeypatch):
+ """A rotation racing final readiness is applied exactly once."""
+ import threading
+
+ from ministack.services import rds as m
+
+ grant_entered = threading.Event()
+ release_grant = threading.Event()
+ modify_done = threading.Event()
+ rotations = []
+
+ class FakeContainer:
+ id = "password-race-container"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+ status = "running"
+
+ def reload(self):
+ pass
+
+ class FakeContainers:
+ def run(self, **_kwargs):
+ return FakeContainer()
+
+ def get(self, identifier):
+ if identifier == FakeContainer.id:
+ return FakeContainer()
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ def _grant(*_args):
+ grant_entered.set()
+ release_grant.wait(timeout=2)
+
+ def _rotate(_cluster, old_password, new_password):
+ rotations.append((old_password, new_password))
+ return True
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_next_port", lambda: 16033)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", _grant)
+ monkeypatch.setattr(m, "_rotate_real_password", _rotate)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "password-race-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "old-password",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "password-race-writer",
+ "DBClusterIdentifier": "password-race-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ assert grant_entered.wait(timeout=1)
+
+ def _modify_password():
+ m._modify_db_cluster({
+ "DBClusterIdentifier": "password-race-cluster",
+ "MasterUserPassword": "new-password",
+ })
+ modify_done.set()
+
+ modify_thread = threading.Thread(target=_modify_password)
+ modify_thread.start()
+ time.sleep(0.05)
+ assert not modify_done.is_set()
+
+ release_grant.set()
+ modify_thread.join(timeout=2)
+
+ cluster = m._clusters.get("password-race-cluster")
+ member = m._instances.get("password-race-writer")
+ assert modify_done.is_set()
+ assert rotations == [("old-password", "new-password")]
+ assert cluster["_MasterUserPassword"] == "new-password"
+ assert "_pending_master_password_rotation" not in cluster
+ assert cluster["_shared_container_ready"] is True
+ assert member["DBInstanceStatus"] == "available"
+ finally:
+ release_grant.set()
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_rotate_real_postgres_master_password(monkeypatch):
+ from ministack.services import rds as m
+
+ connects = []
+ executions = []
+
+ class FakeCursor:
+ def execute(self, query, params):
+ executions.append((query, params))
+
+ def close(self):
+ pass
+
+ class FakeConnection:
+ autocommit = False
+
+ def cursor(self):
+ return FakeCursor()
+
+ def close(self):
+ pass
+
+ def _connect(**kwargs):
+ connects.append(kwargs)
+ return FakeConnection()
+
+ class FakeIdentifier:
+ def __init__(self, value):
+ self.value = value
+
+ def __repr__(self):
+ return f"Identifier({self.value!r})"
+
+ class FakeSQL:
+ def __init__(self, value):
+ self.value = value
+
+ def format(self, **kwargs):
+ return self.value, kwargs
+
+ fake_psycopg2 = types.ModuleType("psycopg2")
+ fake_psycopg2.connect = _connect
+ fake_psycopg2.sql = types.SimpleNamespace(
+ Identifier=FakeIdentifier,
+ SQL=FakeSQL,
+ )
+ monkeypatch.setitem(sys.modules, "psycopg2", fake_psycopg2)
+ m._instances.clear()
+ try:
+ m._instances["rotation-member"] = {
+ "DBInstanceIdentifier": "rotation-member",
+ "DBClusterIdentifier": "rotation-cluster",
+ "Endpoint": {"Address": "localhost", "Port": 16032},
+ }
+ assert m._rotate_real_password({
+ "DBClusterIdentifier": "rotation-cluster",
+ "Engine": "aurora-postgresql",
+ "MasterUsername": "app_admin",
+ "DatabaseName": "appdb",
+ }, "old-password", "new-password") is True
+
+ assert connects == [{
+ "host": "localhost",
+ "port": 16032,
+ "user": "app_admin",
+ "password": "old-password",
+ "dbname": "appdb",
+ }]
+ assert len(executions) == 1
+ assert "Identifier('app_admin')" in repr(executions[0][0])
+ assert executions[0][1] == ("new-password",)
+ finally:
+ m._instances.clear()
+
+
+def test_rds_delete_cluster_rejects_attached_members(monkeypatch):
+ from ministack.services import rds as m
+
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "member-owned-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "member-owned-writer",
+ "DBClusterIdentifier": "member-owned-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+
+ status, _, body = m._delete_db_cluster({
+ "DBClusterIdentifier": "member-owned-cluster",
+ "SkipFinalSnapshot": "true",
+ })
+
+ assert status == 400
+ assert b"InvalidDBClusterStateFault" in body
+ assert m._clusters.get("member-owned-cluster") is not None
+ assert m._instances.get("member-owned-writer")["DBInstanceStatus"] == "available"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
def test_rds_mysql_master_user_privilege_grants(monkeypatch):
"""MySQL master users get admin grants, with dynamic grants best-effort."""
import sys
@@ -1259,6 +2534,52 @@ def test_rds_modify_instance_password(rds):
assert inst["DBInstanceStatus"] == "available"
+def test_rds_modify_cluster_member_password_is_rejected(monkeypatch):
+ from ministack.services import rds as m
+
+ rotations = []
+ monkeypatch.setattr(
+ m,
+ "_rotate_instance_password",
+ lambda *_args: rotations.append(_args),
+ )
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "member-password-cluster",
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "cluster-password",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "member-password-writer",
+ "DBClusterIdentifier": "member-password-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ member = m._instances.get("member-password-writer")
+
+ status, _, body = m._modify_db_instance({
+ "DBInstanceIdentifier": "member-password-writer",
+ "MasterUserPassword": "member-password",
+ "ApplyImmediately": "true",
+ })
+
+ assert status == 400
+ assert b"InvalidParameterCombination" in body
+ assert b"Use ModifyDBCluster instead" in body
+ assert member["_MasterUserPassword"] == "cluster-password"
+ assert m._clusters.get("member-password-cluster")[
+ "_MasterUserPassword"
+ ] == "cluster-password"
+ assert rotations == []
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
# ---------------------------------------------------------------------------
# Tests for the 8 previously-untested operations
# ---------------------------------------------------------------------------
@@ -1787,6 +3108,834 @@ def test_rds_restore_state_respawns_docker_container(monkeypatch):
m._instances.clear()
+ "scenario",
+ [
+ "ready",
+ "not-ready",
+ "writer-removal-fails",
+ "reader-removal-fails",
+ "ownership-mismatch",
+ "member-added-during-readiness",
+ "member-added-during-migration",
+ "pending-password-rotation",
+ "last-member-deleted-during-readiness",
+ "last-members-deleted-before-start",
+ ],
+)
+def test_rds_restore_state_respawns_one_container_per_cluster(
+ monkeypatch, scenario,
+):
+ """Legacy volumes are reaped only after the adopted writer is ready."""
+ import threading
+
+ from ministack.core.responses import AccountRegionScopedDict, get_account_id, get_region
+ from ministack.services import rds as m
+
+ runs = []
+ removed_containers = []
+ removed_volumes = []
+ readiness_credentials = []
+ rotations = []
+ grants = []
+ remaining_legacy_names = set()
+ legacy_owner_by_name = {}
+ writer_legacy_container_name = [None]
+ reader_legacy_container_name = [None]
+ callback_action_done = [False]
+ migration_pause_done = [False]
+ migration_remove_started = threading.Event()
+ release_migration_remove = threading.Event()
+ database_ready = scenario != "not-ready"
+ writer_removal_succeeds = scenario != "writer-removal-fails"
+ reader_removal_succeeds = scenario != "reader-removal-fails"
+
+ class FakeContainer:
+ id = "restored-shared-container"
+ attrs = {"NetworkSettings": {"Networks": {}}}
+ status = "running"
+
+ def reload(self):
+ pass
+
+ class FakeLegacyContainer:
+ def __init__(self, name):
+ self.name = name
+ self.labels = {
+ "ministack": "rds",
+ "db_id": legacy_owner_by_name[name],
+ "account_id": get_account_id(),
+ "region": get_region(),
+ }
+ if (
+ scenario == "ownership-mismatch"
+ and name == reader_legacy_container_name[0]
+ ):
+ self.labels.pop("db_id")
+ self.labels["cluster_id"] = "different-current-cluster"
+ self.attrs = {"Config": {"Labels": self.labels}}
+
+ def remove(self, force=False, v=False):
+ assert force is True
+ assert v is False
+ if (
+ scenario in (
+ "last-members-deleted-before-start",
+ "member-added-during-migration",
+ )
+ and not migration_pause_done[0]
+ ):
+ migration_pause_done[0] = True
+ migration_remove_started.set()
+ release_migration_remove.wait(timeout=2)
+ if (
+ self.name == writer_legacy_container_name[0]
+ and not writer_removal_succeeds
+ ):
+ raise Exception("writer container remains running")
+ if (
+ self.name == reader_legacy_container_name[0]
+ and not reader_removal_succeeds
+ ):
+ raise Exception("reader container remains running")
+ if scenario == "ownership-mismatch" and self.name == (
+ reader_legacy_container_name[0]
+ ):
+ raise AssertionError("unowned current container must not be removed")
+ removed_containers.append(self.name)
+ remaining_legacy_names.discard(self.name)
+
+ class FakeContainers:
+ def get(self, identifier):
+ if identifier in remaining_legacy_names:
+ return FakeLegacyContainer(identifier)
+ raise Exception("not found")
+
+ def run(self, **kwargs):
+ runs.append(kwargs)
+ return FakeContainer()
+
+ class FakeVolume:
+ def __init__(self, name):
+ self.name = name
+
+ def remove(self):
+ removed_volumes.append(self.name)
+
+ class FakeVolumes:
+ def get(self, name):
+ return FakeVolume(name)
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+ self.volumes = FakeVolumes()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ def _wait_for_database_ready(
+ _host, _port, _engine, user, password, db_name, *_args,
+ ):
+ readiness_credentials.append((user, password, db_name))
+ if (
+ scenario == "member-added-during-readiness"
+ and not callback_action_done[0]
+ ):
+ callback_action_done[0] = True
+ m._create_db_instance({
+ "DBInstanceIdentifier": "restored-late-reader",
+ "DBClusterIdentifier": "restored-shared-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ elif (
+ scenario == "last-member-deleted-during-readiness"
+ and not callback_action_done[0]
+ ):
+ callback_action_done[0] = True
+ for db_id in ("restored-reader", "restored-writer"):
+ del m._instances[db_id]
+ m._unregister_instance_from_clusters(db_id)
+ restored_cluster = m._clusters["restored-shared-cluster"]
+ m._stop_empty_cluster_shared_container(
+ "restored-shared-cluster", restored_cluster,
+ )
+ return database_ready
+
+ def _rotate(_cluster, old_password, new_password):
+ rotations.append((old_password, new_password))
+ return True
+
+ def _grant(_host, _port, user, password, db_id):
+ grants.append((user, password, db_id))
+
+ monkeypatch.setattr(m, "_wait_for_database_ready", _wait_for_database_ready)
+ monkeypatch.setattr(m, "_rotate_real_password", _rotate)
+ monkeypatch.setattr(m, "_grant_mysql_master_user_privileges", _grant)
+
+ account_id = get_account_id()
+ region = get_region()
+ cluster_id = "restored-shared-cluster"
+ legacy_container_names = {
+ m._legacy_scoped_rds_docker_name(
+ db_id, account_id, region,
+ )
+ for db_id in ("restored-writer", "restored-reader")
+ }
+ remaining_legacy_names.update(legacy_container_names)
+ legacy_owner_by_name.update({
+ m._legacy_scoped_rds_docker_name(db_id, account_id, region): db_id
+ for db_id in ("restored-writer", "restored-reader")
+ })
+ writer_legacy_container_name[0] = m._legacy_scoped_rds_docker_name(
+ "restored-writer", account_id, region,
+ )
+ reader_legacy_container_name[0] = m._legacy_scoped_rds_docker_name(
+ "restored-reader", account_id, region,
+ )
+ clusters = AccountRegionScopedDict()
+ cluster_record = {
+ "DBClusterIdentifier": cluster_id,
+ "Engine": "aurora-mysql",
+ "EngineVersion": DEFAULT_AURORA_MYSQL_ENGINE_VERSION,
+ "MasterUsername": "cluster-admin",
+ "_MasterUserPassword": "cluster-password",
+ "DatabaseName": "cluster-db",
+ "Port": 3306,
+ "HostedZoneId": "Z2R2ITUGPM61AM",
+ "DBClusterMembers": [
+ {"DBInstanceIdentifier": "restored-writer", "IsClusterWriter": True},
+ {"DBInstanceIdentifier": "restored-reader", "IsClusterWriter": False},
+ ],
+ "_shared_container_id": "stale-container-id",
+ "_shared_host_port": 16010,
+ "_shared_endpoint": {
+ "Address": "localhost",
+ "Port": 16010,
+ "HostedZoneId": "Z2R2ITUGPM61AM",
+ },
+ }
+ if scenario == "pending-password-rotation":
+ cluster_record["_MasterUserPassword"] = "rotated-password"
+ cluster_record["_pending_master_password_rotation"] = {
+ "old_password": "writer-password",
+ "new_password": "rotated-password",
+ }
+ clusters.set_scoped(account_id, region, cluster_id, cluster_record)
+ instances = AccountRegionScopedDict()
+ for db_id in ("restored-writer", "restored-reader"):
+ instances.set_scoped(account_id, region, db_id, {
+ "DBInstanceIdentifier": db_id,
+ "DBClusterIdentifier": cluster_id,
+ "Engine": "aurora-mysql",
+ "EngineVersion": DEFAULT_AURORA_MYSQL_ENGINE_VERSION,
+ "MasterUsername": (
+ "writer-admin" if db_id == "restored-writer" else "reader-admin"
+ ),
+ "_MasterUserPassword": (
+ "writer-password" if db_id == "restored-writer" else "reader-password"
+ ),
+ "DBName": "writer-db" if db_id == "restored-writer" else "reader-db",
+ "DBInstanceStatus": "available",
+ "Endpoint": {"Address": "localhost", "Port": 16010},
+ "_docker_container_id": "stale-container-id",
+ "_docker_volume_name": f"legacy-{db_id}-volume",
+ })
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m.restore_state({
+ "instances": instances,
+ "clusters": clusters,
+ "subnet_groups": {},
+ "param_groups": {},
+ "snapshots": {},
+ "db_cluster_param_groups": {},
+ "db_cluster_snapshots": {},
+ "option_groups": {},
+ "global_clusters": {},
+ "tags": {},
+ "port_counter": 16010,
+ })
+
+ if scenario == "last-members-deleted-before-start":
+ assert migration_remove_started.wait(timeout=1)
+ for db_id in ("restored-reader", "restored-writer"):
+ del m._instances[db_id]
+ m._unregister_instance_from_clusters(db_id)
+ restored_cluster = m._clusters[cluster_id]
+ m._stop_empty_cluster_shared_container(
+ cluster_id,
+ restored_cluster,
+ )
+ release_migration_remove.set()
+ elif scenario == "member-added-during-migration":
+ assert migration_remove_started.wait(timeout=1)
+ restored_cluster = m._clusters[cluster_id]
+ assert restored_cluster["_shared_legacy_migration_in_progress"] is True
+ response = m._create_db_instance({
+ "DBInstanceIdentifier": "racing-migration-member",
+ "DBClusterIdentifier": cluster_id,
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ assert response[0] == 400
+ assert "racing-migration-member" not in m._instances
+ assert runs == []
+ release_migration_remove.set()
+
+ deadline = time.time() + 2
+ while time.time() < deadline:
+ if scenario in (
+ "last-member-deleted-during-readiness",
+ "last-members-deleted-before-start",
+ ):
+ if m._clusters.get(cluster_id, {}).get("DBClusterMembers") == []:
+ break
+ time.sleep(0.01)
+ continue
+ statuses = {
+ m._instances.get(db_id, {}).get("DBInstanceStatus")
+ for db_id in ("restored-writer", "restored-reader")
+ }
+ if statuses <= {"available", "failed"}:
+ break
+ time.sleep(0.01)
+
+ if not writer_removal_succeeds or not reader_removal_succeeds or (
+ scenario == "ownership-mismatch"
+ ):
+ assert runs == []
+ assert readiness_credentials == []
+ expected_remaining_name = (
+ writer_legacy_container_name[0]
+ if not writer_removal_succeeds
+ else reader_legacy_container_name[0]
+ )
+ assert expected_remaining_name in remaining_legacy_names
+ assert removed_volumes == []
+ restored_cluster = m._clusters.get(cluster_id)
+ assert restored_cluster[
+ "_shared_legacy_migration_blocked"
+ ] is True
+ m._create_db_instance({
+ "DBInstanceIdentifier": "blocked-migration-member",
+ "DBClusterIdentifier": cluster_id,
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-mysql",
+ })
+ assert "blocked-migration-member" not in m._instances
+ assert runs == []
+ assert all(
+ m._instances.get(db_id)["DBInstanceStatus"] == "failed"
+ for db_id in ("restored-writer", "restored-reader")
+ )
+ return
+
+ if scenario == "last-members-deleted-before-start":
+ restored_cluster = m._clusters.get(cluster_id)
+ assert runs == []
+ assert readiness_credentials == []
+ assert restored_cluster["DBClusterMembers"] == []
+ assert restored_cluster["_shared_container_ready"] is False
+ assert restored_cluster["_shared_container_epoch"] > 0
+ assert rotations == []
+ assert grants == []
+ assert removed_volumes == []
+ return
+
+ if scenario == "last-member-deleted-during-readiness":
+ restored_cluster = m._clusters.get(cluster_id)
+ assert restored_cluster["DBClusterMembers"] == []
+ assert restored_cluster["_shared_container_ready"] is False
+ assert restored_cluster["_shared_container_epoch"] > 1
+ assert rotations == []
+ assert grants == []
+ assert removed_volumes == []
+ return
+
+ assert len(runs) == 1
+ assert runs[0]["name"] == m._rds_cluster_docker_name(cluster_id)
+ assert runs[0]["environment"]["MYSQL_USER"] == "writer-admin"
+ expected_password = (
+ "rotated-password"
+ if scenario == "pending-password-rotation"
+ else "writer-password"
+ )
+ assert runs[0]["environment"]["MYSQL_ROOT_PASSWORD"] == expected_password
+ assert runs[0]["environment"]["MYSQL_DATABASE"] == "writer-db"
+ assert readiness_credentials == [
+ ("writer-admin", "writer-password", "writer-db"),
+ ]
+ assert runs[0]["volumes"] == {
+ "legacy-restored-writer-volume": {
+ "bind": "/var/lib/mysql",
+ "mode": "rw",
+ },
+ }
+ assert set(removed_containers) == legacy_container_names
+ assert removed_volumes == (
+ ["legacy-restored-reader-volume"] if database_ready else []
+ )
+ restored_cluster = m._clusters.get(cluster_id)
+ assert restored_cluster["_shared_container_id"] == "restored-shared-container"
+ assert restored_cluster["_shared_volume_name"] == "legacy-restored-writer-volume"
+ assert restored_cluster["MasterUsername"] == "writer-admin"
+ assert restored_cluster["_MasterUserPassword"] == expected_password
+ assert restored_cluster["DatabaseName"] == "writer-db"
+ assert restored_cluster["_shared_container_ready"] is database_ready
+ assert restored_cluster["Endpoint"] == restored_cluster["ReaderEndpoint"]
+ assert rotations == (
+ [("writer-password", "rotated-password")]
+ if scenario == "pending-password-rotation"
+ else []
+ )
+ assert grants == (
+ [("writer-admin", expected_password, cluster_id)]
+ if database_ready
+ else []
+ )
+ for db_id in ("restored-writer", "restored-reader"):
+ instance = m._instances.get(db_id)
+ assert instance["MasterUsername"] == "writer-admin"
+ assert instance["_MasterUserPassword"] == expected_password
+ assert instance["DBName"] == "writer-db"
+ assert instance["_docker_container_id"] == "restored-shared-container"
+ assert instance["_shared_cluster_id"] == cluster_id
+ assert instance["Endpoint"] == restored_cluster["_shared_endpoint"]
+ assert instance["DBInstanceStatus"] == (
+ "available" if database_ready else "failed"
+ )
+ if scenario == "member-added-during-readiness":
+ late_member = m._instances.get("restored-late-reader")
+ assert late_member["DBInstanceStatus"] == "available"
+ assert late_member["_docker_container_id"] == (
+ "restored-shared-container"
+ )
+ assert restored_cluster["Status"] == "available"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
[email protected]("cleanup_action", ["delete", "delete-arn", "reset"])
+def test_rds_restored_empty_cluster_cleanup_recovers_container_by_name(
+ monkeypatch, cleanup_action,
+):
+ from ministack.services import rds as m
+
+ stopped = []
+ removed_containers = []
+ removed_volumes = []
+ cluster_id = f"restored-empty-{cleanup_action}"
+ container_name = m._rds_cluster_docker_name(cluster_id)
+ volume_name = m._rds_cluster_docker_volume_name(cluster_id)
+
+ class FakeContainer:
+ def stop(self, timeout=5):
+ stopped.append(timeout)
+
+ def remove(self, v=False):
+ removed_containers.append(v)
+
+ class FakeContainers:
+ def get(self, identifier):
+ if identifier == container_name:
+ return FakeContainer()
+ raise Exception("not found")
+
+ class FakeVolume:
+ def remove(self):
+ removed_volumes.append(volume_name)
+
+ class FakeVolumes:
+ def get(self, name):
+ assert name == volume_name
+ return FakeVolume()
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+ self.volumes = FakeVolumes()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": cluster_id,
+ "Engine": "aurora-mysql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ cluster = m._clusters.get(cluster_id)
+ cluster.update({
+ "_shared_container_id": "unrestorable-container-id",
+ "_shared_endpoint": {
+ "Address": "localhost",
+ "Port": 16022,
+ "HostedZoneId": "Z2R2ITUGPM61AM",
+ },
+ "_shared_volume_name": volume_name,
+ "_shared_container_ready": False,
+ })
+ persisted = m.get_state()
+ m._clusters.clear()
+ m.restore_state(persisted)
+
+ restored = m._clusters.get(cluster_id)
+ assert restored["DBClusterMembers"] == []
+ assert restored["_shared_container_id"] is None
+ if cleanup_action in ("delete", "delete-arn"):
+ cluster_identifier = (
+ restored["DBClusterArn"]
+ if cleanup_action == "delete-arn"
+ else cluster_id
+ )
+ status, _, _ = m._delete_db_cluster({
+ "DBClusterIdentifier": cluster_identifier,
+ "SkipFinalSnapshot": "true",
+ })
+ assert status == 200
+ else:
+ m.reset()
+
+ assert stopped == [5 if cleanup_action in ("delete", "delete-arn") else 2]
+ assert removed_containers == [True]
+ assert removed_volumes == [volume_name]
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_reset_removes_shared_container_once(monkeypatch):
+ """Reset reaps cluster-owned containers and volumes once."""
+ from ministack.services import rds as m
+
+ stop_calls = []
+ remove_calls = []
+ removed_volumes = []
+
+ class FakeContainer:
+ def stop(self, timeout=2):
+ stop_calls.append(timeout)
+
+ def remove(self, v=False):
+ remove_calls.append(v)
+
+ class FakeContainers:
+ def get(self, identifier):
+ assert identifier == "shared-reset-container"
+ return FakeContainer()
+
+ class FakeVolume:
+ def remove(self):
+ removed_volumes.append("shared-reset-volume")
+
+ class FakeVolumes:
+ def get(self, name):
+ assert name == "shared-reset-volume"
+ return FakeVolume()
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+ self.volumes = FakeVolumes()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._clusters["reset-cluster"] = {
+ "DBClusterIdentifier": "reset-cluster",
+ "_shared_container_id": "shared-reset-container",
+ "_shared_volume_name": "shared-reset-volume",
+ }
+ for db_id in ("reset-writer", "reset-reader"):
+ m._instances[db_id] = {
+ "DBInstanceIdentifier": db_id,
+ "_docker_container_id": "shared-reset-container",
+ "_shared_cluster_id": "reset-cluster",
+ }
+
+ m.reset()
+
+ assert stop_calls == [2]
+ assert remove_calls == [True]
+ assert removed_volumes == ["shared-reset-volume"]
+ assert not m._instances
+ assert not m._clusters
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_reset_uses_each_clusters_account_and_region(monkeypatch):
+ from ministack.services import rds as m
+
+ foreign_account = "111122223333"
+ foreign_region = "us-west-2"
+ cluster_id = "foreign-empty-cluster"
+ container_name = m._rds_cluster_docker_name(
+ cluster_id, foreign_account, foreign_region,
+ )
+ volume_name = m._rds_cluster_docker_volume_name(
+ cluster_id, foreign_account, foreign_region,
+ )
+ container_lookups = []
+ volume_lookups = []
+ stop_calls = []
+ remove_calls = []
+
+ class FakeContainer:
+ def stop(self, timeout=2):
+ stop_calls.append(timeout)
+
+ def remove(self, v=False):
+ remove_calls.append(v)
+
+ class FakeContainers:
+ def get(self, identifier):
+ container_lookups.append(identifier)
+ if identifier == container_name:
+ return FakeContainer()
+ raise Exception("not found")
+
+ class FakeVolume:
+ def remove(self):
+ pass
+
+ class FakeVolumes:
+ def get(self, name):
+ volume_lookups.append(name)
+ if name == volume_name:
+ return FakeVolume()
+ raise Exception("not found")
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+ self.volumes = FakeVolumes()
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._clusters.set_scoped(
+ foreign_account,
+ foreign_region,
+ cluster_id,
+ {
+ "DBClusterIdentifier": cluster_id,
+ "DBClusterMembers": [],
+ "_shared_container_id": None,
+ "_shared_endpoint": {
+ "Address": "localhost",
+ "Port": 16040,
+ },
+ "_shared_volume_name": None,
+ },
+ )
+
+ m.reset()
+
+ assert container_lookups == [container_name]
+ assert volume_lookups == [volume_name]
+ assert stop_calls == [2]
+ assert remove_calls == [True]
+ assert not m._clusters.has_any()
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_host_port_probe_rejects_loopback_listener():
+ """A loopback listener must not be mistaken for a reusable Docker port."""
+ import socket
+
+ from ministack.services import rds as m
+
+ listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+ listener.bind(("127.0.0.1", 0))
+ listener.listen()
+ try:
+ assert not m._is_host_port_free(listener.getsockname()[1])
+ finally:
+ listener.close()
+
+
+def test_rds_container_names_separate_instances_from_clusters():
+ from ministack.services import rds as m
+
+ assert m._rds_docker_name("cluster-orders") != m._rds_cluster_docker_name(
+ "orders",
+ )
+ assert "-instance-cluster-orders" in m._rds_docker_name("cluster-orders")
+ assert "-cluster-orders" in m._rds_cluster_docker_name("orders")
+
+
+def test_rds_cluster_volume_name_cannot_match_legacy_instance():
+ from ministack.services import rds as m
+
+ assert m._rds_cluster_docker_volume_name(
+ "orders",
+ ) != m._legacy_scoped_rds_docker_volume_name("cluster-orders")
+ assert "ministack-rds-cluster-" in m._rds_cluster_docker_volume_name(
+ "orders",
+ )
+
+
+def test_rds_restore_migrates_legacy_instance_name_before_cluster_claims_it(
+ monkeypatch,
+):
+ from ministack.core.responses import AccountRegionScopedDict, get_account_id, get_region
+ from ministack.services import rds as m
+
+ account_id = get_account_id()
+ region = get_region()
+ cluster_id = "orders"
+ standalone_id = "cluster-orders"
+ collision_name = m._legacy_scoped_rds_docker_name(
+ standalone_id, account_id, region,
+ )
+ assert collision_name == m._rds_cluster_docker_name(
+ cluster_id, account_id, region,
+ )
+ containers = {}
+ removed = []
+ runs = []
+
+ class FakeContainer:
+ def __init__(self, name, container_id, labels=None):
+ self.name = name
+ self.id = container_id
+ self.labels = labels or {}
+ self.attrs = {
+ "Config": {"Labels": self.labels},
+ "NetworkSettings": {"Networks": {}},
+ }
+
+ def reload(self):
+ pass
+
+ def remove(self, force=False, v=False):
+ removed.append((self.name, self.id, force, v))
+ containers.pop(self.name, None)
+ containers.pop(self.id, None)
+
+ legacy = FakeContainer(
+ collision_name,
+ "legacy-standalone-container",
+ labels={
+ "ministack": "rds",
+ "db_id": standalone_id,
+ "account_id": account_id,
+ "region": region,
+ },
+ )
+ containers[legacy.name] = legacy
+ containers[legacy.id] = legacy
+
+ class FakeContainers:
+ def get(self, identifier):
+ if identifier not in containers:
+ raise Exception("not found")
+ return containers[identifier]
+
+ def run(self, **kwargs):
+ container = FakeContainer(
+ kwargs["name"], f"new-container-{len(runs)}",
+ )
+ runs.append(kwargs)
+ containers[container.name] = container
+ containers[container.id] = container
+ return container
+
+ class FakeDocker:
+ def __init__(self):
+ self.containers = FakeContainers()
+
+ class ImmediateThread:
+ def __init__(self, target, args=(), **_kwargs):
+ self.target = target
+ self.args = args
+
+ def start(self):
+ self.target(*self.args)
+
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m.threading, "Thread", ImmediateThread)
+
+ clusters = AccountRegionScopedDict()
+ clusters.set_scoped(account_id, region, cluster_id, {
+ "DBClusterIdentifier": cluster_id,
+ "DBClusterArn": f"arn:aws:rds:{region}:{account_id}:cluster:{cluster_id}",
+ "Engine": "aurora-mysql",
+ "EngineVersion": DEFAULT_AURORA_MYSQL_ENGINE_VERSION,
+ "MasterUsername": "admin",
+ "_MasterUserPassword": "password123",
+ "DatabaseName": "mydb",
+ "Port": 3306,
+ "DBClusterMembers": [
+ {"DBInstanceIdentifier": "orders-writer", "IsClusterWriter": True},
+ ],
+ })
+ instances = AccountRegionScopedDict()
+ for db_id, parent_id in (
+ ("orders-writer", cluster_id),
+ (standalone_id, ""),
+ ):
+ instances.set_scoped(account_id, region, db_id, {
+ "DBInstanceIdentifier": db_id,
+ "DBClusterIdentifier": parent_id,
+ "DBInstanceArn": f"arn:aws:rds:{region}:{account_id}:db:{db_id}",
+ "Engine": "aurora-mysql" if parent_id else "mysql",
+ "EngineVersion": "8.0",
+ "MasterUsername": "admin",
+ "_MasterUserPassword": "password123",
+ "DBName": "mydb",
+ "DBInstanceStatus": "available",
+ "Endpoint": {"Address": "localhost", "Port": 16030},
+ })
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m.restore_state({
+ "instances": instances,
+ "clusters": clusters,
+ "subnet_groups": {},
+ "param_groups": {},
+ "snapshots": {},
+ "db_cluster_param_groups": {},
+ "db_cluster_snapshots": {},
+ "option_groups": {},
+ "global_clusters": {},
+ "tags": {},
+ "port_counter": 16030,
+ })
+
+ assert removed == [
+ (collision_name, "legacy-standalone-container", True, False),
+ ]
+ assert {run["name"] for run in runs} == {
+ m._rds_cluster_docker_name(cluster_id),
+ m._rds_docker_name(standalone_id),
+ }
+ assert containers[collision_name].id != "legacy-standalone-container"
+ assert containers[collision_name].name == m._rds_cluster_docker_name(
+ cluster_id,
+ )
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
def test_rds_restore_state_removes_stale_container_before_respawn(monkeypatch):
"""If a container with the deterministic name already exists, restore
must remove the stale one before re-creating, otherwise containers.run
@@ -1808,7 +3957,7 @@ def test_rds_restore_state_removes_stale_container_before_respawn(monkeypatch):
def remove(self, **kwargs):
removed.append(self.name)
- stale_name = m._rds_docker_name("stale-db")
+ stale_name = m._legacy_scoped_rds_docker_name("stale-db")
stale = FakeContainer(name=stale_name, container_id="cid-stale")
class FakeContainers:
@@ -1858,7 +4007,7 @@ def test_rds_restore_state_removes_stale_container_before_respawn(monkeypatch):
assert stale_name in removed, "stale container not removed"
assert runs, "fresh container not spawned after removing stale one"
- assert runs[0]["name"] == stale_name
+ assert runs[0]["name"] == m._rds_docker_name("stale-db")
m._instances.clear()
diff --git a/tests/test_rds_aurora_cluster_integration.py b/tests/test_rds_aurora_cluster_integration.py
new file mode 100644
index 00000000..ac29e73d
--- /dev/null
+++ b/tests/test_rds_aurora_cluster_integration.py
@@ -0,0 +1,230 @@
+"""Live Aurora MySQL shared-container coverage.
+
+These tests require MiniStack to have Docker socket access and its
+``DOCKER_NETWORK`` configured so the returned MySQL endpoint is reachable.
+"""
+
+import contextlib
+import os
+import time
+import uuid
+
+import pytest
+from botocore.exceptions import ClientError
+
+pytestmark = pytest.mark.skipif(
+ not os.environ.get("DOCKER_NETWORK"),
+ reason="DOCKER_NETWORK not set -- skipping live Aurora MySQL tests",
+)
+
+PASSWORD = "SharedStorage123!"
+DATABASE = "paritydb"
+
+
+def _wait_for_instance(rds, db_id, timeout=120):
+ deadline = time.time() + timeout
+ while time.time() < deadline:
+ instance = rds.describe_db_instances(
+ DBInstanceIdentifier=db_id,
+ )["DBInstances"][0]
+ if instance["DBInstanceStatus"] == "available":
+ return instance
+ if instance["DBInstanceStatus"] == "failed":
+ pytest.fail(f"RDS instance {db_id} failed while starting")
+ time.sleep(2)
+ raise TimeoutError(f"RDS instance {db_id} not available after {timeout}s")
+
+
+def _connect(endpoint, user="admin", password=PASSWORD, database=DATABASE):
+ import pymysql
+
+ return pymysql.connect(
+ host=endpoint["Address"],
+ port=int(endpoint["Port"]),
+ user=user,
+ password=password,
+ database=database,
+ autocommit=True,
+ connect_timeout=5,
+ )
+
+
+def _live_cluster(rds):
+ suffix = uuid.uuid4().hex[:10]
+ cluster_id = f"shared-{suffix}"
+ writer_id = f"{cluster_id}-writer"
+ reader_id = f"{cluster_id}-reader"
+ try:
+ rds.create_db_cluster(
+ DBClusterIdentifier=cluster_id,
+ Engine="aurora-mysql",
+ MasterUsername="admin",
+ MasterUserPassword=PASSWORD,
+ DatabaseName=DATABASE,
+ )
+ for db_id in (writer_id, reader_id):
+ rds.create_db_instance(
+ DBInstanceIdentifier=db_id,
+ DBClusterIdentifier=cluster_id,
+ DBInstanceClass="db.r6g.large",
+ Engine="aurora-mysql",
+ )
+ writer = _wait_for_instance(rds, writer_id)
+ reader = _wait_for_instance(rds, reader_id)
+ cluster = rds.describe_db_clusters(
+ DBClusterIdentifier=cluster_id,
+ )["DBClusters"][0]
+ yield cluster_id, writer_id, reader_id, writer, reader, cluster
+ finally:
+ for db_id in (reader_id, writer_id):
+ try:
+ rds.delete_db_instance(
+ DBInstanceIdentifier=db_id,
+ SkipFinalSnapshot=True,
+ )
+ except ClientError as e:
+ if e.response["Error"]["Code"] != "DBInstanceNotFound":
+ raise
+ try:
+ rds.delete_db_cluster(
+ DBClusterIdentifier=cluster_id,
+ SkipFinalSnapshot=True,
+ )
+ except ClientError as e:
+ if e.response["Error"]["Code"] != "DBClusterNotFoundFault":
+ raise
+
+
+def test_aurora_writer_data_is_visible_through_reader(rds):
+ with _live_cluster(rds) as (_cid, _wid, _rid, writer, reader, _cluster):
+ with _connect(writer["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("CREATE TABLE shared_rows (id INT PRIMARY KEY, value VARCHAR(32))")
+ cursor.execute("INSERT INTO shared_rows VALUES (1, 'writer-data')")
+ with _connect(reader["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("SELECT id, value FROM shared_rows")
+ assert cursor.fetchall() == ((1, "writer-data"),)
+
+
+def test_aurora_user_and_grant_are_visible_through_reader(rds):
+ with _live_cluster(rds) as (_cid, _wid, _rid, writer, reader, _cluster):
+ app_user = f"app_{uuid.uuid4().hex[:8]}"
+ app_password = "AppPassword123!"
+ with _connect(writer["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("CREATE TABLE granted_rows (id INT PRIMARY KEY)")
+ cursor.execute(f"CREATE USER `{app_user}`@'%%' IDENTIFIED BY %s", (app_password,))
+ cursor.execute(f"GRANT SELECT ON `{DATABASE}`.* TO `{app_user}`@'%'")
+ with _connect(reader["Endpoint"], app_user, app_password) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("SELECT COUNT(*) FROM granted_rows")
+ assert cursor.fetchone() == (0,)
+
+
+def test_aurora_cluster_uses_one_backing_container(rds):
+ import docker
+
+ with _live_cluster(rds) as (cluster_id, _wid, _rid, writer, reader, cluster):
+ containers = docker.from_env().containers.list(
+ all=True,
+ filters={"label": ["ministack=rds", f"cluster_id={cluster_id}"]},
+ )
+ assert len(containers) == 1
+ assert writer["Endpoint"] == reader["Endpoint"]
+ assert cluster["Endpoint"] == cluster["ReaderEndpoint"]
+ assert cluster["Endpoint"] == writer["Endpoint"]["Address"]
+ assert cluster["Port"] == writer["Endpoint"]["Port"]
+
+
+def test_aurora_delete_member_keeps_shared_data(rds, rds_data):
+ import docker
+ import pymysql
+
+ with _live_cluster(rds) as (
+ cluster_id,
+ writer_id,
+ reader_id,
+ writer,
+ _reader,
+ cluster,
+ ):
+ with _connect(writer["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("CREATE TABLE durable_rows (id INT PRIMARY KEY)")
+ cursor.execute("INSERT INTO durable_rows VALUES (7)")
+ rds.delete_db_instance(
+ DBInstanceIdentifier=reader_id,
+ SkipFinalSnapshot=True,
+ )
+ with _connect(writer["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("SELECT id FROM durable_rows")
+ assert cursor.fetchone() == (7,)
+
+ containers = docker.from_env().containers.list(
+ all=True,
+ filters={"label": ["ministack=rds", f"cluster_id={cluster_id}"]},
+ )
+ assert len(containers) == 1
+ container = containers[0]
+ original_container_id = container.id
+
+ rds.delete_db_instance(
+ DBInstanceIdentifier=writer_id,
+ SkipFinalSnapshot=True,
+ )
+ empty_cluster = rds.describe_db_clusters(
+ DBClusterIdentifier=cluster_id,
+ )["DBClusters"][0]
+ assert empty_cluster["Status"] == "available"
+ assert empty_cluster["DBClusterMembers"] == []
+
+ container.reload()
+ assert container.status == "exited"
+ with pytest.raises((pymysql.err.OperationalError, OSError)):
+ _connect(writer["Endpoint"])
+ with pytest.raises(ClientError) as exc_info:
+ rds_data.execute_statement(
+ resourceArn=cluster["DBClusterArn"],
+ secretArn=(
+ "arn:aws:secretsmanager:us-east-1:000000000000:secret:unused"
+ ),
+ sql="SELECT 1",
+ )
+ assert exc_info.value.response["Error"]["Code"] == "DatabaseUnavailableException"
+
+ replacement_id = f"{cluster_id}-replacement"
+ try:
+ rds.create_db_instance(
+ DBInstanceIdentifier=replacement_id,
+ DBClusterIdentifier=cluster_id,
+ DBInstanceClass="db.r6g.large",
+ Engine="aurora-mysql",
+ )
+ replacement = _wait_for_instance(rds, replacement_id)
+ container.reload()
+ assert container.status == "running"
+ assert container.id == original_container_id
+ with _connect(replacement["Endpoint"]) as conn:
+ with conn.cursor() as cursor:
+ cursor.execute("SELECT id FROM durable_rows")
+ assert cursor.fetchone() == (7,)
+ finally:
+ try:
+ rds.delete_db_instance(
+ DBInstanceIdentifier=replacement_id,
+ SkipFinalSnapshot=True,
+ )
+ except ClientError as e:
+ if e.response["Error"]["Code"] != "DBInstanceNotFound":
+ raise
+ containers = docker.from_env().containers.list(
+ all=True,
+ filters={"label": ["ministack=rds", f"cluster_id={cluster_id}"]},
+ )
+ assert containers == []
+
+
+# P2 adds FailoverDBCluster coverage here: role flip plus post-failover write.
diff --git a/tests/test_rds_data.py b/tests/test_rds_data.py
index 1a3dd499..a52b6112 100644
--- a/tests/test_rds_data.py
+++ b/tests/test_rds_data.py
@@ -157,12 +157,14 @@ def test_rds_data_resolves_cluster_arn_by_arn_region(rds, sm):
east_rds.create_db_cluster(
DBClusterIdentifier=cluster_id,
Engine="aurora-mysql",
+ EngineMode="serverless",
MasterUsername="admin",
MasterUserPassword="testpass123",
)
west_rds.create_db_cluster(
DBClusterIdentifier=cluster_id,
Engine="aurora-mysql",
+ EngineMode="serverless",
MasterUsername="admin",
MasterUserPassword="testpass123",
)
@@ -254,14 +256,15 @@ def test_rds_data_member_lookup_uses_resource_arn_region():
east_instance = {
"DBInstanceIdentifier": f"{cluster_id}-east",
"DBClusterIdentifier": cluster_id,
+ "DBInstanceStatus": "available",
"Engine": "aurora-mysql",
"Endpoint": {"Address": "east.example", "Port": 3306},
}
west_instance = {
"DBInstanceIdentifier": f"{cluster_id}-west",
"DBClusterIdentifier": cluster_id,
+ "DBInstanceStatus": "available",
"Engine": "aurora-mysql",
- "Endpoint": {"Address": "west.example", "Port": 3306},
}
try:
@@ -277,7 +280,21 @@ def test_rds_data_member_lookup_uses_resource_arn_region():
ACCOUNT_ID,
"us-west-2",
cluster_id,
- {"DBClusterIdentifier": cluster_id, "DBClusterArn": west_arn, "Engine": "aurora-mysql"},
+ {
+ "DBClusterIdentifier": cluster_id,
+ "DBClusterArn": west_arn,
+ "Engine": "aurora-mysql",
+ "EngineMode": "provisioned",
+ "DBClusterMembers": [{
+ "DBInstanceIdentifier": west_instance["DBInstanceIdentifier"],
+ "IsClusterWriter": True,
+ }],
+ "_shared_container_ready": True,
+ "_shared_endpoint": {"Address": "west.example", "Port": 3306},
+ "_shared_container_id": "west-shared-container",
+ "_shared_internal_address": "172.20.0.8",
+ "_shared_internal_port": 3306,
+ },
)
rds._instances.set_scoped(ACCOUNT_ID, "us-east-1", east_instance["DBInstanceIdentifier"], east_instance)
rds._instances.set_scoped(ACCOUNT_ID, "us-west-2", west_instance["DBInstanceIdentifier"], west_instance)
@@ -287,6 +304,10 @@ def test_rds_data_member_lookup_uses_resource_arn_region():
assert instance is west_instance
assert engine == "aurora-mysql"
+ assert instance["Endpoint"] == {"Address": "west.example", "Port": 3306}
+ assert instance["_docker_container_id"] == "west-shared-container"
+ assert instance["_internal_address"] == "172.20.0.8"
+ assert instance["_internal_port"] == 3306
finally:
rds.reset()
set_request_account_id(original_account)
@@ -475,12 +496,13 @@ def test_substitute_named_params_preserves_cast_and_unknowns():
# ── Stub mode tests ────────────────────────────────────────
def _setup_stub_cluster(rds, sm):
- """Create an RDS cluster (no real DB container) and a secret for stub testing."""
+ """Create an Aurora Serverless cluster and secret for stub testing."""
import uuid as _uuid
cluster_id = f"stub-test-{_uuid.uuid4().hex[:8]}"
rds.create_db_cluster(
DBClusterIdentifier=cluster_id,
Engine="aurora-mysql",
+ EngineMode="serverless",
MasterUsername="admin",
MasterUserPassword="testpass123",
)
@@ -617,6 +639,44 @@ def test_rds_data_real_endpoint_connection_failure_is_transient(monkeypatch):
assert "lost_write" not in rds_data._stub_users.get("real-cluster", set())
+def test_rds_data_provisioned_cluster_without_members_is_unavailable():
+ """An empty provisioned cluster exists but has no SQL compute."""
+ from ministack.services import rds, rds_data
+
+ cluster_id = f"empty-provisioned-{uuid.uuid4().hex[:8]}"
+ resource_arn = f"arn:aws:rds:{REGION}:{ACCOUNT_ID}:cluster:{cluster_id}"
+ rds._clusters[cluster_id] = {
+ "DBClusterIdentifier": cluster_id,
+ "DBClusterArn": resource_arn,
+ "Engine": "aurora-mysql",
+ "EngineMode": "provisioned",
+ "Status": "available",
+ "DBClusterMembers": [],
+ "_shared_container_ready": False,
+ }
+
+ try:
+ assert rds_data._resolve_cluster(resource_arn) == (None, "aurora-mysql")
+ common = {
+ "resourceArn": resource_arn,
+ "secretArn": FAKE_SECRET_ARN,
+ }
+ cases = (
+ (rds_data._execute_statement, {"sql": "SELECT 1"}),
+ (rds_data._begin_transaction, {}),
+ (rds_data._batch_execute_statement, {"sql": "SELECT 1"}),
+ )
+ for handler, extra in cases:
+ status, headers, body = handler({**common, **extra})
+ payload = json.loads(body)
+ assert status == 504
+ assert headers["x-amzn-errortype"] == "DatabaseUnavailableException"
+ assert payload["__type"] == "DatabaseUnavailableException"
+ assert "no available DB instances" in payload["message"]
+ finally:
+ rds._clusters.pop(cluster_id, None)
+
+
def test_rds_data_non_container_endpoint_keeps_stub_mode(monkeypatch):
"""Control-plane-only instances still use the lightweight SQL stub."""
from ministack.services import rds_data
@@ -684,8 +744,8 @@ def test_rds_cluster_status_tracks_member_readiness():
rds._clusters.pop(cluster_id, None)
-def test_rds_cluster_endpoints_sync_to_backing_instance():
- """Aurora cluster endpoints track the registered local writer instance."""
+def test_rds_cluster_endpoints_sync_to_shared_container():
+ """Aurora cluster endpoints track the cluster-owned shared container."""
from ministack.services import rds
cluster_id = "endpoint-sync-cluster"
@@ -697,6 +757,10 @@ def test_rds_cluster_endpoints_sync_to_backing_instance():
"ReaderEndpoint": "original-ro.cluster.local",
"Port": 3306,
"DBClusterMembers": [],
+ "_shared_endpoint": {
+ "Address": "10.0.0.15",
+ "Port": 3307,
+ },
}
rds._instances[instance_id] = {
"DBInstanceIdentifier": instance_id,
@@ -713,9 +777,9 @@ def test_rds_cluster_endpoints_sync_to_backing_instance():
rds._register_instance_in_cluster(rds._instances[instance_id])
cluster = rds._clusters[cluster_id]
- assert cluster["Endpoint"] == "10.0.0.12"
- assert cluster["ReaderEndpoint"] == "10.0.0.12"
- assert cluster["Port"] == 3306
+ assert cluster["Endpoint"] == "10.0.0.15"
+ assert cluster["ReaderEndpoint"] == "10.0.0.15"
+ assert cluster["Port"] == 3307
finally:
rds._instances.pop(instance_id, None)
rds._clusters.pop(cluster_id, None)
diff --git a/tests/test_stepfunctions.py b/tests/test_stepfunctions.py
index c2819a34..a20a5f27 100644
--- a/tests/test_stepfunctions.py
+++ b/tests/test_stepfunctions.py
@@ -4366,6 +4366,7 @@ def test_sfn_aws_sdk_rdsdata_execute_statement(sfn, sfn_sync, rds, sm):
rds.create_db_cluster(
DBClusterIdentifier=cluster_id,
Engine="aurora-mysql",
+ EngineMode="serverless",
MasterUsername="admin",
MasterUserPassword="testpass123",
)
@@ -4447,6 +4448,7 @@ def test_sfn_aws_sdk_rdsdata_output_uses_sfn_key_convention(sfn, sfn_sync, rds,
rds.create_db_cluster(
DBClusterIdentifier=cluster_id,
Engine="aurora-mysql",
+ EngineMode="serverless",
MasterUsername="admin",
MasterUserPassword="testpass123",
)
@@ -4728,6 +4730,7 @@ def test_sfn_rest_json_pascal_to_camel_conversion(sfn, sfn_sync, rds, sm):
rds.create_db_cluster(
DBClusterIdentifier=cluster_id,
Engine="aurora-mysql",
+ EngineMode="serverless",
MasterUsername="admin",
MasterUserPassword="testpass123",
)