Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

rds: opt-in Aurora PostgreSQL readers with real streaming replication (#1428)

kiran01bm20bd95099cb4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6d75df8f..c6a9e1dd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
### Added
+- **RDS — opt-in Aurora PostgreSQL reader instances backed by real streaming replication** — with `MINISTACK_RDS_PG_CLUSTER_REPLICATION=1`, the second and later members of an Aurora PostgreSQL cluster launch their own PostgreSQL containers, cloned from the writer with `pg_basebackup` and running as hot standbys that stream WAL from it. A reader is genuinely read-only (`pg_is_in_recovery()` is true, writes fail with `25006`), sees data written to the writer, and the cluster `ReaderEndpoint` resolves to an available reader instead of the writer — so read/write connection routing, replica-lag handling, and reader-endpoint pooling can be rehearsed locally. Off by default: without the flag (and for Aurora MySQL) members keep aliasing the writer's single shared container. Requires the shared Docker network (`DOCKER_NETWORK`); deleting a reader removes only its own container. Not yet modeled (follow-up slices of #1325): stopping a cluster with replicating readers is rejected with `InvalidDBClusterStateFault`, and a persisted reader demotes to a shared-container alias on warm boot. Contributed by @kiran01bm.
+
- **CloudFormation — API Gateway (v1) API keys and usage plans** — `AWS::ApiGateway::ApiKey`, `AWS::ApiGateway::UsagePlan`, and `AWS::ApiGateway::UsagePlanKey` failed stack creation with `Unsupported resource type`, even though the runtime `CreateApiKey` / `CreateUsagePlan` APIs were already served. The three resource types now provision through those same stores: an `ApiKey` supports a pinned `Value`, `Enabled`, and `StageKeys`, updates `Name` / `Description` / `Enabled` in place, and exposes both `Ref` and `Fn::GetAtt APIKeyId` as the key id; a `UsagePlan` maps `Throttle` / `Quota` / `ApiStages` and returns its id via `Ref` and `Fn::GetAtt Id`; a `UsagePlanKey` associates a key with a plan and is torn down from the plan on delete. This unblocks CDK `RestApi` / `ApiKey` constructs and Terraform `aws_api_gateway_api_key` deployed through CloudFormation. Contributed by @ryan-bennett.
- **AWS IoT Jobs — control plane and device data plane** — `CreateJob` fell through to `Unsupported IoT path` and the `iot-jobs-data` endpoint did not exist, so device OTA loops could not be rehearsed locally. The `iot` service now serves the nine job operations and a new `iot-jobs-data` service the four device ones (incl. the `$next` sentinel), sharing one store and the AWS execution state machine. A `documentSource` job gets a placeholder, not the S3 object. Contributed by @iot-rocket.
### Fixed
diff --git a/README.md b/README.md
index 9952755e..b4c0b2d7 100644
--- a/README.md
+++ b/README.md
@@ -803,6 +803,7 @@ end-to-end without any client config.
| `GLUE_DOCKER_IMAGE` | (auto by `GlueVersion`) | Override the `amazon/aws-glue-libs` PySpark image used for Spark Glue jobs. Defaults: `glue_libs_4.0.0_image_01` (GlueVersion 4.0), `glue_libs_3.0.0_image_01` (GlueVersion 3.0) |
| `RDS_PERSIST` | `0` | Set `1` to use Docker named volumes for RDS containers instead of tmpfs. Storage grows dynamically with no fixed cap |
| `MINISTACK_RDS_PUBLIC_ENDPOINT` | `0` | Set `1` when MiniStack itself runs in a Docker container but RDS clients connect from outside that network (remote MiniStack host, host-side clients, CI runners). `DescribeDBInstances` then returns `{MINISTACK_HOST, host_port}` — the externally-reachable host-published port — instead of the container-internal address. Set `MINISTACK_HOST` to the host clients will use |
+| `MINISTACK_RDS_PG_CLUSTER_REPLICATION` | `0` | Set `1` to give the second and later members of an Aurora PostgreSQL cluster their own hot-standby containers streaming from the writer (real read-only replicas; the cluster `ReaderEndpoint` resolves to one when available). Requires `DOCKER_NETWORK`. Off: all members alias the writer's single shared container |
| `DSQL_BASE_PORT` | `25432` | Starting host port for Aurora DSQL cluster proxies (one port per cluster) |
| `DSQL_STRICT` | `0` | Set `1` to run a real Postgres container per DSQL cluster (requires Docker; capped by a fixed port window off `DSQL_BASE_PORT`, default 30 — further clusters are metadata-only stubs). Default `0` is control-plane-only |
| `DSQL_PG_IMAGE` | `postgres:16-alpine` | Image used for per-cluster DSQL backend containers |
diff --git a/ministack/services/rds.py b/ministack/services/rds.py
index a5c2fb54..78e0e6ee 100644
--- a/ministack/services/rds.py
+++ b/ministack/services/rds.py
@@ -79,6 +79,14 @@ DOCKER_NETWORK = os.environ.get("DOCKER_NETWORK", "")
# clients of a containerised ministack). Off by default: existing in-network
# behavior unchanged.
RDS_PUBLIC_ENDPOINT = os.environ.get("MINISTACK_RDS_PUBLIC_ENDPOINT", "0").lower() in ("1", "true", "yes")
+# Opt-in: per-instance Aurora PostgreSQL reader containers backed by real
+# streaming replication (#1325). Scoped to aurora-postgresql by name and by
+# gate: Aurora MySQL readers keep aliasing the single shared container, and
+# the physical-replication bootstrap below is PostgreSQL-specific
+# (pg_basebackup + hot standby). Off by default: existing behavior unchanged.
+RDS_PG_CLUSTER_REPLICATION = os.environ.get(
+ "MINISTACK_RDS_PG_CLUSTER_REPLICATION", "0",
+).lower() in ("1", "true", "yes")
_instances = AccountRegionScopedDict()
_clusters = AccountRegionScopedDict()
@@ -108,6 +116,66 @@ _MYSQL_REPLICATION_RETRY_INTERVAL = 1
# not an environment variable.
_MYSQL_BINLOG_RETENTION_SECONDS = 604800 # 7 days
+# Streaming-replication credentials for per-instance Aurora PostgreSQL reader
+# containers (#1325). ``rdsrepladmin`` mirrors the reserved role name real RDS
+# PostgreSQL provisions for replication. Fixed local-only credentials, same
+# convention as the MySQL replication user above.
+_PG_REPLICATION_USER = "rdsrepladmin"
+_PG_REPLICATION_PASSWORD = "rdsrepladmin-local-password"
+
+# Runs on the writer container (docker exec). Idempotently creates the
+# replication role and opens pg_hba.conf for remote replication connections:
+# the stock postgres image only allows replication from localhost, so
+# without the extra rule a reader's pg_basebackup is rejected. The role is
+# created with an explicit scram-sha-256 verifier so the pg_hba method below
+# authenticates it on every supported major (password_encryption defaulted
+# to md5 before PostgreSQL 14).
+_PG_REPLICATION_SOURCE_SCRIPT = f"""set -eu
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<'SQL'
+SET password_encryption = 'scram-sha-256';
+DO $do$
+BEGIN
+ IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '{_PG_REPLICATION_USER}') THEN
+ CREATE ROLE {_PG_REPLICATION_USER} WITH REPLICATION LOGIN
+ PASSWORD '{_PG_REPLICATION_PASSWORD}';
+ END IF;
+END
+$do$;
+SQL
+if ! grep -q "host replication {_PG_REPLICATION_USER}" "$PGDATA/pg_hba.conf"; then
+ printf 'host replication {_PG_REPLICATION_USER} all scram-sha-256\\n' \\
+ >> "$PGDATA/pg_hba.conf"
+fi
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \\
+ -c "SELECT pg_reload_conf()" >/dev/null
+"""
+
+# Container command for a reader: bypass the image entrypoint (which would
+# initdb a brand-new independent database) and instead clone the writer with
+# pg_basebackup, then start PostgreSQL as a hot standby. The base backup is
+# retried until the writer is ready and the replication role exists, so
+# reader creation may safely overlap writer creation. --write-recovery-conf
+# records the primary connection (including the password, because it is
+# passed inside --dbname) so the standby streams WAL from the writer.
+_PG_READER_BOOTSTRAP_SCRIPT = """set -eu
+until pg_isready -h "$MINISTACK_PG_PRIMARY_HOST" \\
+ -p "$MINISTACK_PG_PRIMARY_PORT" >/dev/null 2>&1; do
+ sleep 1
+done
+mkdir -p "$PGDATA"
+chown -R postgres:postgres "$PGDATA"
+chmod 700 "$PGDATA"
+until gosu postgres pg_basebackup \\
+ --dbname="host=$MINISTACK_PG_PRIMARY_HOST port=$MINISTACK_PG_PRIMARY_PORT \\
+user=$MINISTACK_PG_REPLICATION_USER password=$MINISTACK_PG_REPLICATION_PASSWORD" \\
+ --pgdata="$PGDATA" --write-recovery-conf --wal-method=stream \\
+ --checkpoint=fast; do
+ find "$PGDATA" -mindepth 1 -delete
+ sleep 1
+done
+exec gosu postgres postgres
+"""
+
# Aurora MySQL versions are the creatable set returned by AWS RDS as of
# 2026-07-09. Refresh with:
# aws rds describe-db-engine-versions --engine aurora-mysql \
@@ -260,6 +328,11 @@ def restore_state(data):
cluster = _clusters._data[key]
cluster["_shared_container_id"] = None
cluster.pop("_mysql_replication_retry_marker", None)
+ # Re-verify replication provisioning against the respawned writer
+ # (#1325): the role and pg_hba rule live in the cluster volume and
+ # normally survive, but the flag must not outlive a volume that
+ # did not.
+ cluster.pop("_pg_replication_source_ready", None)
if cluster.get("_mysql_replication_source_arn"):
# A respawned MySQL container gets a new hostname, while the
# persisted replica repository and relay-log names belong to the
@@ -293,6 +366,12 @@ def restore_state(data):
account_id, region, instance_id = key
inst["_docker_container_id"] = None
inst["DBInstanceStatus"] = "creating"
+ # Warm boot of per-instance reader containers is not implemented
+ # yet (#1325 slice 3). Demote a persisted replicating reader to a
+ # shared-container alias member so the restored cluster is fully
+ # usable instead of leaving a reader that points at a container
+ # the restart killed.
+ inst.pop("_pg_standby", None)
if RDS_PERSIST:
inst.setdefault(
"_docker_volume_name",
@@ -1056,6 +1135,303 @@ def _cluster_owned_container_ids(cluster):
return ids
+def _pg_cluster_replication_enabled(cluster):
+ """Whether this cluster launches per-instance reader containers (#1325).
+
+ Requires the ``MINISTACK_RDS_PG_CLUSTER_REPLICATION`` opt-in and an
+ Aurora PostgreSQL cluster: the reader bootstrap is physical streaming
+ replication (pg_basebackup + hot standby), which has no MySQL analog
+ here — Aurora MySQL members keep aliasing the shared container.
+ """
+ return (
+ RDS_PG_CLUSTER_REPLICATION
+ and cluster.get("Engine") == "aurora-postgresql"
+ )
+
+
+def _ensure_pg_replication_source(cluster_id, cluster):
+ """Provision streaming-replication access on the writer's container.
+
+ Idempotent: creates the replication role and appends the pg_hba.conf
+ rule that allows remote replication connections, then reloads
+ configuration. Returns True once the writer can serve pg_basebackup
+ for reader containers; False when provisioning could not run (no
+ Docker, no writer container, or the exec failed) — callers retry.
+
+ The check-then-provision runs under ``_shared_container_lock`` so
+ concurrent reader workers on the same cluster provision once instead
+ of racing the flag (the script tolerates a rerun, but two interleaved
+ executions against the same writer are pointless work).
+ """
+ with _shared_container_lock:
+ if cluster.get("_pg_replication_source_ready"):
+ return True
+ docker_client = _get_docker()
+ container_id = cluster.get("_shared_container_id")
+ if not docker_client or not container_id:
+ return False
+ try:
+ container = docker_client.containers.get(container_id)
+ exit_code, output = container.exec_run(
+ ["sh", "-c", _PG_REPLICATION_SOURCE_SCRIPT],
+ )
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to provision replication source for cluster %s: %s",
+ cluster_id, e,
+ )
+ return False
+ if exit_code != 0:
+ logger.warning(
+ "RDS: replication source provisioning for cluster %s exited %s: %s",
+ cluster_id,
+ exit_code,
+ output.decode(errors="replace") if isinstance(output, bytes) else output,
+ )
+ return False
+ cluster["_pg_replication_source_ready"] = True
+ return True
+
+
+def _start_pg_reader_container(db_id, cluster):
+ """Launch a member-owned hot-standby container for an Aurora PostgreSQL
+ cluster (#1325).
+
+ Returns a start-result dict ({"started", "failed", endpoint fields}),
+ or None when per-instance replication cannot run here — no Docker, no
+ shared internal Docker network, or no known writer address — in which
+ case the caller falls back to shared-container aliasing, the flag-off
+ behavior.
+ """
+ docker_client = _get_docker()
+ if not docker_client:
+ return None
+ cluster_id = cluster["DBClusterIdentifier"]
+ ms_network = _get_ministack_network(docker_client)
+ writer_host = cluster.get("_shared_internal_address")
+ writer_port = cluster.get("_shared_internal_port")
+ if not ms_network or not writer_host:
+ # pg_basebackup runs inside the reader container and must reach the
+ # writer container; without a shared Docker network there is no
+ # address that works from both the host and the reader.
+ logger.warning(
+ "RDS: MINISTACK_RDS_PG_CLUSTER_REPLICATION is on but cluster %s "
+ "has no internal Docker network address; reader %s falls back "
+ "to the shared container",
+ cluster_id, db_id,
+ )
+ return None
+ engine = cluster.get("Engine", "aurora-postgresql")
+ engine_version = cluster.get("EngineVersion") or _default_engine_version(engine)
+ master_user = cluster.get("MasterUsername", "admin")
+ master_pass = cluster.get("_MasterUserPassword", "password")
+ db_name = cluster.get("DatabaseName") or "mydb"
+ image, env, container_port, data_path = _docker_image_for_engine(
+ engine, engine_version, master_user, master_pass, db_name,
+ )
+ if not image:
+ return None
+ env = dict(env)
+ env.update({
+ "MINISTACK_PG_PRIMARY_HOST": str(writer_host),
+ "MINISTACK_PG_PRIMARY_PORT": str(writer_port or container_port),
+ "MINISTACK_PG_REPLICATION_USER": _PG_REPLICATION_USER,
+ "MINISTACK_PG_REPLICATION_PASSWORD": _PG_REPLICATION_PASSWORD,
+ })
+ host_port = _next_port()
+ container_kwargs = dict(
+ image=image,
+ detach=True,
+ environment=env,
+ command=["sh", "-c", _PG_READER_BOOTSTRAP_SCRIPT],
+ ports={f"{container_port}/tcp": host_port},
+ name=_rds_docker_name(db_id),
+ network=ms_network,
+ labels={
+ "ministack": "rds",
+ "db_id": db_id,
+ "cluster_id": cluster_id,
+ "account_id": get_account_id(),
+ "region": get_region(),
+ },
+ )
+ volume_name = None
+ if RDS_PERSIST:
+ volume_name = _rds_docker_volume_name(db_id)
+ container_kwargs["volumes"] = {
+ volume_name: {"bind": data_path, "mode": "rw"},
+ }
+ else:
+ container_kwargs["tmpfs"] = {
+ data_path: f"rw,noexec,nosuid,size={RDS_TMPFS_SIZE}",
+ }
+ try:
+ container = docker_client.containers.run(**container_kwargs)
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to start reader container for %s: %s", db_id, e,
+ )
+ if volume_name:
+ # Docker may have auto-created the named volume before the run
+ # failed; a leftover (possibly partially written) volume would
+ # poison a retried CreateDBInstance under the same identifier.
+ try:
+ docker_client.volumes.get(volume_name).remove()
+ except Exception:
+ pass
+ return {"started": False, "failed": True}
+ endpoint_host = _MINISTACK_HOST
+ endpoint_port = host_port
+ internal_host = None
+ internal_port = None
+ try:
+ container.reload()
+ networks = container.attrs.get("NetworkSettings", {}).get("Networks", {})
+ container_ip = networks.get(ms_network, {}).get("IPAddress", "")
+ if container_ip:
+ endpoint_host = container_ip
+ endpoint_port = container_port
+ internal_host = container_ip
+ internal_port = container_port
+ except Exception:
+ pass
+ return {
+ "started": True,
+ "failed": False,
+ "container_id": container.id,
+ "volume_name": volume_name,
+ "host_port": host_port,
+ "endpoint_host": endpoint_host,
+ "endpoint_port": endpoint_port,
+ "internal_host": internal_host,
+ "internal_port": internal_port,
+ "readiness_host": internal_host or "127.0.0.1",
+ "readiness_port": internal_port or host_port,
+ }
+
+
+def _remove_failed_pg_reader_compute(db_id, container_id, volume_name):
+ """Remove a failed replicating reader's container and named volume.
+
+ A standby that never became reachable leaves nothing worth keeping:
+ its data directory holds at most a partial base backup, and the volume
+ name is derived from the instance identifier, so leaving either behind
+ would leak a host port and poison a retried CreateDBInstance under the
+ same identifier. Cleanup for healthy readers stays in
+ ``_delete_db_instance``; this runs only for bootstrap failures.
+ """
+ docker_client = _get_docker()
+ if not docker_client:
+ return
+ try:
+ container = docker_client.containers.get(container_id)
+ container.stop(timeout=5)
+ container.remove(v=True)
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to remove container of failed reader %s: %s",
+ db_id, e,
+ )
+ if volume_name:
+ try:
+ docker_client.volumes.get(volume_name).remove()
+ except Exception as e:
+ logger.warning(
+ "RDS: failed to remove volume of failed reader %s: %s",
+ db_id, e,
+ )
+
+
+def _bg_finalize_pg_reader(
+ db_id, cluster_id, engine, master_user, master_pass, db_name,
+ ready_host, ready_port, container_id,
+):
+ """Readiness worker for a member-owned replicating reader (#1325).
+
+ Waits for the writer to become ready, provisions the replication
+ source on it, then waits for the standby to accept authenticated
+ connections (the reader container retries pg_basebackup internally
+ until the source is provisioned). Same liveness contract as the
+ generic worker: no wall clock — the instance stays ``creating`` while
+ its container is up and booting, and flips to ``failed`` if the
+ container dies first. Same staleness contract as the shared-container
+ worker: if the instance record no longer points at this worker's
+ container (deleted and recreated under the same identifier), the
+ worker is superseded and must not touch the new record. A bootstrap
+ failure removes the reader's own container and volume.
+ """
+ def _container_alive():
+ client = _get_docker()
+ if not client:
+ return True
+ try:
+ c = client.containers.get(container_id)
+ c.reload()
+ return c.status not in ("exited", "dead", "removing")
+ except Exception:
+ return False
+
+ def _stale(instance):
+ return instance.get("_docker_container_id") != container_id
+
+ while True:
+ instance = _instances.get(db_id)
+ cluster = _clusters.get(cluster_id)
+ if instance is None or cluster is None:
+ return # deleted while bootstrapping
+ if _stale(instance):
+ return # superseded: the recreated instance has its own worker
+ if instance.get("DBInstanceStatus") == "failed":
+ # Writer compute failed during creation; its readiness worker
+ # already published the failure for every member. The standby
+ # container would retry pg_basebackup against the dead writer
+ # forever, so remove it.
+ _remove_failed_pg_reader_compute(
+ db_id, container_id, instance.get("_docker_volume_name"),
+ )
+ return
+ if not _container_alive():
+ instance["DBInstanceStatus"] = "failed"
+ _remove_failed_pg_reader_compute(
+ db_id, container_id, instance.get("_docker_volume_name"),
+ )
+ _refresh_cluster_status(cluster_id)
+ return
+ if cluster.get(
+ "_shared_container_ready",
+ ) and _ensure_pg_replication_source(cluster_id, cluster):
+ break
+ time.sleep(1)
+
+ database_ready = _wait_for_database_ready(
+ ready_host, ready_port, engine, master_user, master_pass, db_name,
+ _container_alive,
+ )
+ instance = _instances.get(db_id)
+ cluster = _clusters.get(cluster_id)
+ if instance is None or _stale(instance):
+ return
+ if not database_ready:
+ logger.warning(
+ "RDS: reader container for %s at %s:%s exited before becoming "
+ "reachable", db_id, ready_host, ready_port,
+ )
+ instance["DBInstanceStatus"] = "failed"
+ _remove_failed_pg_reader_compute(
+ db_id, container_id, instance.get("_docker_volume_name"),
+ )
+ _refresh_cluster_status(cluster_id)
+ return
+ instance["DBInstanceStatus"] = "available"
+ if cluster:
+ _sync_cluster_endpoints(cluster)
+ _refresh_cluster_status(cluster_id)
+ logger.info(
+ "RDS: replicating reader %s ready at %s:%s", db_id,
+ ready_host, ready_port,
+ )
+
+
def _stop_cluster_shared_container(cluster_id, cluster):
"""Stop, but do not remove, an Aurora cluster's database compute.
@@ -2550,6 +2926,10 @@ def _set_cluster_members_status(cluster, status):
def _attach_instance_to_shared_cluster(instance, cluster):
+ if instance.get("_pg_standby"):
+ # A replicating reader owns its container and endpoint (#1325);
+ # aliasing would clobber them with the writer's.
+ return
endpoint = cluster.get("_shared_endpoint")
if not endpoint:
return
@@ -2578,13 +2958,31 @@ def _attach_instance_to_shared_cluster(instance, cluster):
def _cluster_reader_endpoint(cluster):
"""The endpoint a cluster's ``ReaderEndpoint`` should resolve to.
- Today this is the writer's shared endpoint, so the local reader endpoint
- is read/write: it resolves to the same database process as the writer.
- Genuine read-only behavior requires a separate replicating process —
- per-instance reader containers (#1325) will resolve here instead once a
- cluster has one.
+ When a member owns a replicating reader container (#1325) and it is
+ available, the reader endpoint resolves there and is genuinely
+ read-only. Otherwise it falls back to the writer's shared endpoint —
+ matching real Aurora, where the reader endpoint follows the writer in
+ a single-instance cluster, but read/write because it resolves to the
+ same database process as the writer.
+
+ A DBCluster advertises a single ``Port`` (the writer's, applied by
+ ``_sync_cluster_endpoints``), so a standby only reachable on a
+ different port — the host-port fallback when its network IP lookup
+ failed — must not win the ReaderEndpoint address: pairing its Address
+ with the writer's Port would be unreachable.
"""
- return cluster.get("_shared_endpoint")
+ shared = cluster.get("_shared_endpoint")
+ for member in _cluster_member_instances(cluster):
+ if (
+ member.get("_pg_standby")
+ and member.get("DBInstanceStatus") == "available"
+ and member.get("Endpoint")
+ ):
+ endpoint = member["Endpoint"]
+ if shared and endpoint.get("Port") != shared.get("Port"):
+ continue
+ return endpoint
+ return shared
def _sync_cluster_endpoints(cluster):
@@ -2748,7 +3146,31 @@ def _create_db_instance_impl(p):
ms_network = None
docker_client = _get_docker()
- if parent:
+ reader_launch = None
+ if (
+ parent
+ and parent.get("DBClusterMembers")
+ and _pg_cluster_replication_enabled(parent)
+ ):
+ # Second and later members of a flag-enabled Aurora PostgreSQL
+ # cluster get their own hot-standby container instead of aliasing
+ # the writer's (#1325). Returns None when that cannot run here,
+ # falling back to the aliasing branch below.
+ reader_launch = _start_pg_reader_container(db_id, parent)
+ if parent and reader_launch:
+ start_result = reader_launch
+ endpoint_host = reader_launch.get("endpoint_host", endpoint_host)
+ endpoint_port = int(reader_launch.get("endpoint_port", endpoint_port))
+ host_port = reader_launch.get("host_port")
+ docker_container_id = reader_launch.get("container_id")
+ docker_volume_name = reader_launch.get("volume_name")
+ internal_host = reader_launch.get("internal_host")
+ internal_port = reader_launch.get("internal_port")
+ real_container_started = bool(reader_launch.get("started"))
+ readiness_host = reader_launch.get("readiness_host")
+ readiness_port = reader_launch.get("readiness_port")
+ ms_network = _get_ministack_network(docker_client)
+ elif parent:
pending_rotation = parent.get("_pending_master_password_rotation")
if pending_rotation:
readiness_master_pass = pending_rotation["old_password"]
@@ -2996,12 +3418,33 @@ def _create_db_instance_impl(p):
"_internal_port": internal_port,
"_MasterUserPassword": master_pass,
}
- if parent:
+ if parent and reader_launch:
+ # A replicating reader keeps its own endpoint and container fields;
+ # the marker excludes it from shared-container aliasing and makes
+ # the cluster ReaderEndpoint resolve to it once available.
+ instance["_pg_standby"] = True
+ elif parent:
_attach_instance_to_shared_cluster(instance, parent)
_instances[db_id] = instance
_register_instance_in_cluster(instance)
- if real_container_started:
+ if real_container_started and reader_launch:
+ # Replicating readers have their own readiness contract (wait for
+ # the writer, provision replication, wait for the standby), so they
+ # get a dedicated worker instead of the generic one below.
+ reader_ctx = contextvars.copy_context()
+ threading.Thread(
+ target=reader_ctx.run,
+ args=(
+ _bg_finalize_pg_reader, db_id, cluster_id, engine,
+ master_user, master_pass, db_name,
+ readiness_host or endpoint_host,
+ readiness_port or endpoint_port,
+ docker_container_id,
+ ),
+ daemon=True,
+ ).start()
+ elif real_container_started:
# Real AWS CreateDBInstance returns immediately with status="creating"
# and the caller polls (or uses get_waiter('db_instance_available'))
# until the database becomes reachable. Do the same: run the readiness
@@ -3131,9 +3574,14 @@ def _create_db_instance_impl(p):
_configure_or_defer_mysql_replication(cluster_id, cluster)
for member in cluster.get("DBClusterMembers", []):
inst = _instances.get(member.get("DBInstanceIdentifier"))
- if inst is not None:
- _attach_instance_to_shared_cluster(inst, cluster)
- inst["DBInstanceStatus"] = "available"
+ if inst is None or inst.get("_pg_standby"):
+ # Replicating readers publish their own
+ # readiness (#1325); the writer becoming ready
+ # does not make a still-bootstrapping standby
+ # available.
+ continue
+ _attach_instance_to_shared_cluster(inst, cluster)
+ inst["DBInstanceStatus"] = "available"
_sync_cluster_endpoints(cluster)
_refresh_cluster_status(cluster_id)
if ms_network and internal_host:
@@ -5006,6 +5454,22 @@ def _stop_db_cluster(p):
f"in {status} state but expected it to be one of available.",
400,
)
+ if any(
+ inst.get("_pg_standby") for inst in _cluster_member_instances(cluster)
+ ):
+ # Restarting per-instance reader containers after a stop is not
+ # implemented yet (#1325 slice 3). Refusing here beats the two bad
+ # alternatives: an ``available`` reader whose container is exited,
+ # or a cluster wedged in ``creating`` because a member never
+ # becomes available again.
+ return _error(
+ "InvalidDBClusterStateFault",
+ "StopDBCluster is not supported for clusters with replicating "
+ "reader instances yet; delete the reader instances first "
+ "(MINISTACK_RDS_PG_CLUSTER_REPLICATION, "
+ "https://github.com/ministackorg/ministack/issues/1325).",
+ 400,
+ )
if not _stop_cluster_shared_container(
cluster.get("DBClusterIdentifier", cluster_id), cluster,
):
diff --git a/tests/test_rds.py b/tests/test_rds.py
index daa84e86..747fb7b6 100644
--- a/tests/test_rds.py
+++ b/tests/test_rds.py
@@ -9486,3 +9486,732 @@ def test_rds_cluster_reader_endpoint_resolves_shared_endpoint():
assert m._cluster_reader_endpoint(bare) is None
m._sync_cluster_endpoints(bare)
assert "ReaderEndpoint" not in bare
+
+
+# ---------------------------------------------------------------------------
+# Aurora PostgreSQL per-instance replicating readers (#1325 slice 2)
+# ---------------------------------------------------------------------------
+
+
+def test_rds_pg_cluster_replication_gate(monkeypatch):
+ """Replicating readers are opt-in and scoped to Aurora PostgreSQL."""
+ from ministack.services import rds as m
+
+ pg_cluster = {"Engine": "aurora-postgresql"}
+ mysql_cluster = {"Engine": "aurora-mysql"}
+
+ # Flag off (the default): no cluster qualifies.
+ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", False)
+ assert not m._pg_cluster_replication_enabled(pg_cluster)
+ assert not m._pg_cluster_replication_enabled(mysql_cluster)
+
+ # Flag on: Aurora PostgreSQL only. Aurora MySQL members keep aliasing
+ # the shared container even with the flag set.
+ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", True)
+ assert m._pg_cluster_replication_enabled(pg_cluster)
+ assert not m._pg_cluster_replication_enabled(mysql_cluster)
+
+
+def test_rds_cluster_reader_endpoint_prefers_available_standby():
+ """ReaderEndpoint resolves to an available replicating reader (#1325)."""
+ from ministack.core.responses import get_account_id, set_request_account_id
+ from ministack.services import rds as m
+
+ shared = {"Address": "10.0.0.5", "Port": 5432, "HostedZoneId": "Z2R2ITUGPM61AM"}
+ reader_ep = {"Address": "10.0.0.7", "Port": 5432, "HostedZoneId": "Z2R2ITUGPM61AM"}
+ original_account = get_account_id()
+ reader_id = "standby-ep-reader-1"
+ try:
+ set_request_account_id("111111111111")
+ cluster = {
+ "DBClusterIdentifier": "standby-ep",
+ "_shared_endpoint": shared,
+ "DBClusterMembers": [
+ {"DBInstanceIdentifier": "standby-ep-writer", "IsClusterWriter": True},
+ {"DBInstanceIdentifier": reader_id},
+ ],
+ }
+ # Reader still bootstrapping: fall back to the writer's endpoint.
+ m._instances[reader_id] = {
+ "DBInstanceIdentifier": reader_id,
+ "_pg_standby": True,
+ "DBInstanceStatus": "creating",
+ "Endpoint": reader_ep,
+ }
+ assert m._cluster_reader_endpoint(cluster) == shared
+
+ # Reader available: the cluster ReaderEndpoint follows the standby
+ # while the writer endpoint stays on the shared container.
+ m._instances[reader_id]["DBInstanceStatus"] = "available"
+ assert m._cluster_reader_endpoint(cluster) == reader_ep
+ m._sync_cluster_endpoints(cluster)
+ assert cluster["Endpoint"] == "10.0.0.5"
+ assert cluster["ReaderEndpoint"] == "10.0.0.7"
+ finally:
+ m._instances.pop(reader_id, None)
+ set_request_account_id(original_account)
+
+
+def test_rds_attach_instance_skips_pg_standby():
+ """Shared-cluster aliasing must not clobber a replicating reader (#1325)."""
+ from ministack.services import rds as m
+
+ cluster = {
+ "DBClusterIdentifier": "attach-guard",
+ "_shared_endpoint": {"Address": "10.0.0.5", "Port": 5432},
+ "_shared_container_id": "shared-c",
+ }
+ standby_endpoint = {"Address": "10.0.0.7", "Port": 5432}
+ standby = {
+ "DBInstanceIdentifier": "attach-guard-reader",
+ "_pg_standby": True,
+ "Endpoint": standby_endpoint,
+ "_docker_container_id": "reader-c",
+ }
+ m._attach_instance_to_shared_cluster(standby, cluster)
+ assert standby["Endpoint"] == standby_endpoint
+ assert standby["_docker_container_id"] == "reader-c"
+ assert "_shared_cluster_id" not in standby
+ # The reader keeps owning its container (delete removes it, #1339).
+ assert m._instance_owns_container(standby)
+
+
+def _pg_reader_fake_docker():
+ """Fake Docker client for _start_pg_reader_container unit tests."""
+ class FakeContainer:
+ id = "reader-container-1"
+ status = "running"
+ attrs = {
+ "NetworkSettings": {
+ "Networks": {"ms_net": {"IPAddress": "10.0.0.7"}},
+ },
+ }
+
+ def reload(self):
+ pass
+
+ container = FakeContainer()
+ calls = {}
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ calls["run"] = kwargs
+ return container
+
+ def get(self, identifier):
+ if identifier == container.id:
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ containers = FakeContainers()
+
+ return FakeDocker(), container, calls
+
+
+def test_rds_start_pg_reader_container_requires_internal_network(monkeypatch):
+ """No shared Docker network: no reader container, caller falls back."""
+ from ministack.services import rds as m
+
+ docker, _container, calls = _pg_reader_fake_docker()
+ monkeypatch.setattr(m, "_get_docker", lambda: docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)
+ cluster = {
+ "DBClusterIdentifier": "no-net",
+ "Engine": "aurora-postgresql",
+ "_shared_internal_address": None,
+ }
+ assert m._start_pg_reader_container("no-net-1", cluster) is None
+ assert "run" not in calls
+
+
+def test_rds_start_pg_reader_container_launches_owned_container(monkeypatch):
+ """A reader container bootstraps as a hot standby cloned from the writer."""
+ from ministack.services import rds as m
+
+ docker, container, calls = _pg_reader_fake_docker()
+ monkeypatch.setattr(m, "_get_docker", lambda: docker)
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: "ms_net")
+ monkeypatch.setattr(m, "_next_port", lambda: 16091)
+ cluster = {
+ "DBClusterIdentifier": "reader-launch",
+ "Engine": "aurora-postgresql",
+ "EngineVersion": "17.7",
+ "MasterUsername": "admin",
+ "_MasterUserPassword": "pw",
+ "DatabaseName": "appdb",
+ "_shared_internal_address": "10.0.0.5",
+ "_shared_internal_port": 5432,
+ }
+ result = m._start_pg_reader_container("reader-launch-1", cluster)
+ assert result["started"] is True
+ assert result["container_id"] == container.id
+ # Container-internal address on the shared network, like other
+ # in-network RDS endpoints.
+ assert result["endpoint_host"] == "10.0.0.7"
+ assert result["endpoint_port"] == 5432
+
+ kwargs = calls["run"]
+ assert kwargs["name"] == m._rds_docker_name("reader-launch-1")
+ assert kwargs["network"] == "ms_net"
+ assert kwargs["labels"]["db_id"] == "reader-launch-1"
+ assert kwargs["labels"]["cluster_id"] == "reader-launch"
+ # The image entrypoint (which would initdb an independent database) is
+ # bypassed: the reader clones the writer with pg_basebackup and starts
+ # as a hot standby.
+ assert kwargs["command"][:2] == ["sh", "-c"]
+ script = kwargs["command"][2]
+ assert "pg_basebackup" in script
+ assert "--write-recovery-conf" in script
+ env = kwargs["environment"]
+ assert env["MINISTACK_PG_PRIMARY_HOST"] == "10.0.0.5"
+ assert env["MINISTACK_PG_PRIMARY_PORT"] == "5432"
+ # The launched reader is member-owned compute (#1339 semantics).
+ assert m._instance_owns_container(
+ {"_docker_container_id": result["container_id"]},
+ )
+
+
+def test_rds_pg_replicating_reader_lifecycle(monkeypatch):
+ """Flag on: the second member owns a standby container end to end.
+
+ Fake-Docker walk of the control plane: the writer aliases the shared
+ cluster container, the reader launches its own, the cluster
+ ReaderEndpoint follows the reader once available, StopDBCluster is
+ refused while a replicating reader exists (slice 3), and deleting the
+ reader removes only reader-owned compute and falls the ReaderEndpoint
+ back to the writer.
+ """
+ from ministack.services import rds as m
+
+ containers = {}
+ removed = []
+
+ class FakeContainer:
+ def __init__(self, name, kwargs):
+ self.id = f"{name}-container"
+ self.name = name
+ self.kwargs = kwargs
+ self.status = "running"
+ net = kwargs.get("network", "ms_net")
+ is_member = "db_id" in kwargs.get("labels", {})
+ ip = "10.0.0.7" if is_member else "10.0.0.5"
+ self.attrs = {
+ "NetworkSettings": {"Networks": {net: {"IPAddress": ip}}},
+ }
+
+ def reload(self):
+ pass
+
+ def exec_run(self, _cmd):
+ return 0, b""
+
+ def stop(self, timeout=5):
+ self.status = "exited"
+
+ def remove(self, force=False, v=False):
+ removed.append(self.id)
+ containers.pop(self.name, None)
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ container = FakeContainer(kwargs["name"], kwargs)
+ containers[kwargs["name"]] = container
+ return container
+
+ def get(self, identifier):
+ for container in containers.values():
+ if identifier in (container.id, container.name):
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ containers = FakeContainers()
+
+ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", True)
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: "ms_net")
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "pgrepl-cluster",
+ "Engine": "aurora-postgresql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ m._create_db_instance({
+ "DBInstanceIdentifier": "pgrepl-writer",
+ "DBClusterIdentifier": "pgrepl-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-postgresql",
+ })
+ cluster = m._clusters.get("pgrepl-cluster")
+ writer = m._instances.get("pgrepl-writer")
+ assert _poll_until(
+ lambda: writer["DBInstanceStatus"] == "available",
+ )
+ # Single-member cluster: ReaderEndpoint follows the writer.
+ assert cluster["ReaderEndpoint"] == "10.0.0.5"
+
+ m._create_db_instance({
+ "DBInstanceIdentifier": "pgrepl-reader",
+ "DBClusterIdentifier": "pgrepl-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-postgresql",
+ })
+ reader = m._instances.get("pgrepl-reader")
+ assert reader["_pg_standby"] is True
+ assert _poll_until(
+ lambda: reader["DBInstanceStatus"] == "available",
+ )
+ # The reader owns a container distinct from the shared one.
+ assert reader["_docker_container_id"] != cluster["_shared_container_id"]
+ assert m._instance_owns_container(reader)
+ # Replication access was provisioned on the writer.
+ assert cluster["_pg_replication_source_ready"] is True
+ # ReaderEndpoint now resolves to the standby; the writer endpoint
+ # stays on the shared container; the writer stays the only writer.
+ assert cluster["ReaderEndpoint"] == "10.0.0.7"
+ assert cluster["Endpoint"] == "10.0.0.5"
+ writers = [
+ member for member in cluster["DBClusterMembers"]
+ if member.get("IsClusterWriter")
+ ]
+ assert [w["DBInstanceIdentifier"] for w in writers] == ["pgrepl-writer"]
+ assert _poll_until(lambda: cluster["Status"] == "available")
+
+ # Stop/start with a replicating reader is slice 3: refused, not
+ # published wrongly.
+ status, _, body = m._stop_db_cluster({
+ "DBClusterIdentifier": "pgrepl-cluster",
+ })
+ assert status == 400
+ assert "InvalidDBClusterStateFault" in str(body)
+
+ # Deleting the reader removes only its own compute; the
+ # ReaderEndpoint falls back to the writer.
+ reader_container_id = reader["_docker_container_id"]
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "pgrepl-reader",
+ "SkipFinalSnapshot": "true",
+ })
+ assert reader_container_id in removed
+ assert cluster["_shared_container_id"] not in removed
+ assert cluster["ReaderEndpoint"] == "10.0.0.5"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def _pg_repl_fake_docker(m, monkeypatch, member_ips, exec_calls, removed):
+ """Fake Docker for replicating-reader tests.
+
+ ``member_ips`` maps a member ``db_id`` label to the container IP the
+ fake reports; containers without a ``db_id`` label (the cluster-owned
+ shared container) get 10.0.0.5. ``exec_calls`` collects the names of
+ containers that received ``exec_run``; ``removed`` collects removed
+ container ids.
+ """
+ containers = {}
+
+ class FakeContainer:
+ def __init__(self, name, kwargs):
+ self.id = f"{name}-container"
+ self.name = name
+ self.kwargs = kwargs
+ self.status = "running"
+ net = kwargs.get("network", "ms_net")
+ db_id = kwargs.get("labels", {}).get("db_id")
+ ip = member_ips.get(db_id, "10.0.0.5")
+ self.attrs = {
+ "NetworkSettings": {"Networks": {net: {"IPAddress": ip}}},
+ }
+
+ def reload(self):
+ pass
+
+ def exec_run(self, _cmd):
+ exec_calls.append(self.name)
+ return 0, b""
+
+ def stop(self, timeout=5):
+ self.status = "exited"
+
+ def remove(self, force=False, v=False):
+ removed.append(self.id)
+ containers.pop(self.name, None)
+
+ class FakeContainers:
+ def run(self, **kwargs):
+ container = FakeContainer(kwargs["name"], kwargs)
+ containers[kwargs["name"]] = container
+ return container
+
+ def get(self, identifier):
+ for container in containers.values():
+ if identifier in (container.id, container.name):
+ return container
+ raise Exception("not found")
+
+ class FakeDocker:
+ containers = FakeContainers()
+
+ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", True)
+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())
+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: "ms_net")
+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)
+
+
+def test_rds_pg_two_replicating_readers_provision_source_once(monkeypatch):
+ """Two readers on one cluster: the writer is provisioned exactly once,
+ ReaderEndpoint selection is deterministic (member order), and a standby
+ whose port cannot pair with the cluster Port never wins the address.
+ """
+ from ministack.services import rds as m
+
+ exec_calls = []
+ removed = []
+ _pg_repl_fake_docker(
+ m, monkeypatch,
+ {"pgrepl2-reader1": "10.0.0.7", "pgrepl2-reader2": "10.0.0.8"},
+ exec_calls, removed,
+ )
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "pgrepl2-cluster",
+ "Engine": "aurora-postgresql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ for db_id in ("pgrepl2-writer", "pgrepl2-reader1", "pgrepl2-reader2"):
+ m._create_db_instance({
+ "DBInstanceIdentifier": db_id,
+ "DBClusterIdentifier": "pgrepl2-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-postgresql",
+ })
+ cluster = m._clusters.get("pgrepl2-cluster")
+ reader1 = m._instances.get("pgrepl2-reader1")
+ reader2 = m._instances.get("pgrepl2-reader2")
+ assert _poll_until(
+ lambda: reader1["DBInstanceStatus"] == "available"
+ and reader2["DBInstanceStatus"] == "available",
+ )
+ assert cluster["_pg_replication_source_ready"] is True
+ # Replication provisioning ran exactly once on the shared (writer)
+ # container even with two reader workers racing the flag.
+ shared_execs = [
+ name for name in exec_calls
+ if name == m._rds_cluster_docker_name("pgrepl2-cluster")
+ ]
+ assert len(shared_execs) == 1
+ assert exec_calls == shared_execs # nothing exec'd on the readers
+
+ # Member order (reader1 first) decides the ReaderEndpoint.
+ assert cluster["ReaderEndpoint"] == "10.0.0.7"
+
+ # Deleting the first reader moves the endpoint to the second.
+ m._delete_db_instance({
+ "DBInstanceIdentifier": "pgrepl2-reader1",
+ "SkipFinalSnapshot": "true",
+ })
+ assert cluster["ReaderEndpoint"] == "10.0.0.8"
+
+ # A standby only reachable on a different port than the cluster
+ # Port must not win the address: fall back to the writer.
+ reader2["Endpoint"]["Port"] = 5999
+ m._sync_cluster_endpoints(cluster)
+ assert cluster["ReaderEndpoint"] == "10.0.0.5"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_pg_reader_bootstrap_failure_removes_reader_compute(monkeypatch):
+ """A standby that never becomes reachable is failed AND its container
+ is removed — no leaked compute waiting for a manual DeleteDBInstance.
+ """
+ from ministack.services import rds as m
+
+ exec_calls = []
+ removed = []
+ _pg_repl_fake_docker(
+ m, monkeypatch, {"pgreplf-reader": "10.0.0.7"}, exec_calls, removed,
+ )
+ # The writer (10.0.0.5) becomes ready; the standby never does.
+ monkeypatch.setattr(
+ m, "_wait_for_database_ready",
+ lambda host, *_args: host != "10.0.0.7",
+ )
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "pgreplf-cluster",
+ "Engine": "aurora-postgresql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ for db_id in ("pgreplf-writer", "pgreplf-reader"):
+ m._create_db_instance({
+ "DBInstanceIdentifier": db_id,
+ "DBClusterIdentifier": "pgreplf-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-postgresql",
+ })
+ cluster = m._clusters.get("pgreplf-cluster")
+ reader = m._instances.get("pgreplf-reader")
+ reader_container_id = reader["_docker_container_id"]
+ assert _poll_until(
+ lambda: reader["DBInstanceStatus"] == "failed",
+ )
+ assert reader_container_id in removed
+ assert cluster["_shared_container_id"] not in removed
+ # The ReaderEndpoint never advertises the failed standby.
+ assert cluster["ReaderEndpoint"] == "10.0.0.5"
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_pg_stale_reader_worker_is_inert(monkeypatch):
+ """A superseded reader worker (captured container id no longer the
+ instance's) must not touch the recreated instance's status or compute.
+ """
+ from ministack.services import rds as m
+
+ exec_calls = []
+ removed = []
+ _pg_repl_fake_docker(
+ m, monkeypatch, {"pgrepls-reader": "10.0.0.7"}, exec_calls, removed,
+ )
+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)
+
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ m._create_db_cluster({
+ "DBClusterIdentifier": "pgrepls-cluster",
+ "Engine": "aurora-postgresql",
+ "MasterUsername": "admin",
+ "MasterUserPassword": "password123",
+ })
+ for db_id in ("pgrepls-writer", "pgrepls-reader"):
+ m._create_db_instance({
+ "DBInstanceIdentifier": db_id,
+ "DBClusterIdentifier": "pgrepls-cluster",
+ "DBInstanceClass": "db.r6g.large",
+ "Engine": "aurora-postgresql",
+ })
+ reader = m._instances.get("pgrepls-reader")
+ assert _poll_until(
+ lambda: reader["DBInstanceStatus"] == "available",
+ )
+
+ # A worker whose captured container id was superseded (delete +
+ # recreate under the same identifier) returns without mutating.
+ m._bg_finalize_pg_reader(
+ "pgrepls-reader", "pgrepls-cluster", "aurora-postgresql",
+ "admin", "password123", "mydb", "10.0.0.7", 5432,
+ "stale-container-id",
+ )
+ assert reader["DBInstanceStatus"] == "available"
+ assert removed == []
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+def test_rds_restore_state_demotes_pg_standby(monkeypatch):
+ """Warm boot demotes a persisted replicating reader to an alias member.
+
+ Restarting reader containers across host restarts is #1325 slice 3;
+ until then a restored reader must not stay marked ``_pg_standby``
+ pointing at a container the restart killed, and the cluster must
+ re-verify replication provisioning against the respawned writer.
+ """
+ from ministack.core.responses import (
+ AccountRegionScopedDict,
+ get_account_id,
+ get_region,
+ )
+ from ministack.services import rds as m
+
+ monkeypatch.setattr(m, "_get_docker", lambda: None)
+ key_scope = (get_account_id(), get_region())
+ m._instances.clear()
+ m._clusters.clear()
+ try:
+ clusters = AccountRegionScopedDict()
+ instances = AccountRegionScopedDict()
+ clusters._data[(*key_scope, "warm-pg")] = {
+ "DBClusterIdentifier": "warm-pg",
+ "Engine": "aurora-postgresql",
+ "Status": "available",
+ "_pg_replication_source_ready": True,
+ "DBClusterMembers": [
+ {"DBInstanceIdentifier": "warm-pg-writer", "IsClusterWriter": True},
+ {"DBInstanceIdentifier": "warm-pg-reader", "IsClusterWriter": False},
+ ],
+ }
+ instances._data[(*key_scope, "warm-pg-writer")] = {
+ "DBInstanceIdentifier": "warm-pg-writer",
+ "DBClusterIdentifier": "warm-pg",
+ "_shared_cluster_id": "warm-pg",
+ }
+ instances._data[(*key_scope, "warm-pg-reader")] = {
+ "DBInstanceIdentifier": "warm-pg-reader",
+ "DBClusterIdentifier": "warm-pg",
+ "_pg_standby": True,
+ "_docker_container_id": "dead-reader-container",
+ }
+ m.restore_state({"clusters": clusters, "instances": instances})
+
+ reader = m._instances.get("warm-pg-reader")
+ assert reader is not None
+ assert "_pg_standby" not in reader
+ cluster = m._clusters.get("warm-pg")
+ assert "_pg_replication_source_ready" not in cluster
+ # The demoted reader re-attaches to the shared cluster endpoint like
+ # any alias member (no Docker in this test: restore publishes the
+ # endpoint without real compute).
+ assert _poll_until(
+ lambda: reader.get("DBInstanceStatus") == "available",
+ )
+ assert reader.get("_shared_cluster_id") == "warm-pg"
+ assert m._cluster_reader_endpoint(cluster) == cluster["_shared_endpoint"]
+ finally:
+ m._instances.clear()
+ m._clusters.clear()
+
+
+_PG_REPLICATION_LIVE = (
+ os.environ.get("DOCKER_NETWORK")
+ and os.environ.get(
+ "MINISTACK_RDS_PG_CLUSTER_REPLICATION", "0",
+ ).lower() in ("1", "true", "yes")
+)
+
+
+def _pg_connect(endpoint, user="admin", password=PASSWORD, database=DATABASE):
+ import psycopg2
+
+ return psycopg2.connect(
+ host=endpoint["Address"],
+ port=int(endpoint["Port"]),
+ user=user,
+ password=password,
+ dbname=database,
+ connect_timeout=5,
+ )
+
+
+def _live_pg_cluster(rds):
+ suffix = uuid.uuid4().hex[:10]
+ cluster_id = f"pgrepl-{suffix}"
+ writer_id = f"{cluster_id}-writer"
+ reader_id = f"{cluster_id}-reader"
+ try:
+ rds.create_db_cluster(
+ DBClusterIdentifier=cluster_id,
+ Engine="aurora-postgresql",
+ MasterUsername="admin",
+ MasterUserPassword=PASSWORD,
+ DatabaseName=DATABASE,
+ )
+ for db_id in (writer_id, reader_id):
+ rds.create_db_instance(
+ DBInstanceIdentifier=db_id,
+ DBClusterIdentifier=cluster_id,
+ DBInstanceClass="db.r6g.large",
+ Engine="aurora-postgresql",
+ )
+ writer = _wait_for_instance(rds, writer_id, timeout=180)
+ reader = _wait_for_instance(rds, reader_id, timeout=180)
+ cluster = rds.describe_db_clusters(
+ DBClusterIdentifier=cluster_id,
+ )["DBClusters"][0]
+ yield cluster_id, writer_id, reader_id, writer, reader, cluster
+ finally:
+ for db_id in (reader_id, writer_id):
+ try:
+ rds.delete_db_instance(
+ DBInstanceIdentifier=db_id,
+ SkipFinalSnapshot=True,
+ )
+ except ClientError as e:
+ if e.response["Error"]["Code"] != "DBInstanceNotFound":
+ raise
+ try:
+ rds.delete_db_cluster(
+ DBClusterIdentifier=cluster_id,
+ SkipFinalSnapshot=True,
+ )
+ except ClientError as e:
+ if e.response["Error"]["Code"] != "DBClusterNotFoundFault":
+ raise
+
+
+ not _PG_REPLICATION_LIVE,
+ reason="DOCKER_NETWORK and MINISTACK_RDS_PG_CLUSTER_REPLICATION not set "
+ "-- live Aurora PostgreSQL replication",
+)
+def test_aurora_pg_replicating_reader_live(rds):
+ """A flag-enabled reader is a genuine hot standby streaming from the writer."""
+ with _live_pg_cluster(rds) as (_cid, _wid, _rid, writer, reader, cluster):
+ # Distinct endpoints: the reader is not an alias of the writer.
+ assert (
+ (reader["Endpoint"]["Address"], reader["Endpoint"]["Port"])
+ != (writer["Endpoint"]["Address"], writer["Endpoint"]["Port"])
+ )
+ # The cluster reader endpoint resolves to the reader.
+ assert cluster["ReaderEndpoint"] == reader["Endpoint"]["Address"]
+
+ with _pg_connect(reader["Endpoint"]) as conn, conn.cursor() as cursor:
+ cursor.execute("SELECT pg_is_in_recovery()")
+ assert cursor.fetchone() == (True,)
+
+ with _pg_connect(writer["Endpoint"]) as conn:
+ conn.autocommit = True
+ with conn.cursor() as cursor:
+ cursor.execute("SELECT pg_is_in_recovery()")
+ assert cursor.fetchone() == (False,)
+ cursor.execute(
+ "CREATE TABLE repl_rows (id INT PRIMARY KEY, value TEXT)",
+ )
+ cursor.execute("INSERT INTO repl_rows VALUES (1, 'writer-data')")
+
+ # The write streams to the standby (bounded poll for replica lag).
+ def _reader_sees_row():
+ with _pg_connect(reader["Endpoint"]) as conn, conn.cursor() as cursor:
+ try:
+ cursor.execute("SELECT id, value FROM repl_rows")
+ except Exception:
+ return False
+ return cursor.fetchall() == [(1, "writer-data")]
+
+ deadline = time.time() + 60
+ while time.time() < deadline and not _reader_sees_row():
+ time.sleep(1)
+ assert _reader_sees_row()
+
+ # Writes against the standby fail read-only (SQLSTATE 25006).
+ import psycopg2
+
+ with _pg_connect(reader["Endpoint"]) as conn:
+ conn.autocommit = True
+ with conn.cursor() as cursor:
+ with pytest.raises(psycopg2.Error) as excinfo:
+ cursor.execute("INSERT INTO repl_rows VALUES (2, 'nope')")
+ assert excinfo.value.pgcode == "25006"