Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
rds: opt-in Aurora PostgreSQL readers with real streaming replication (#1428)
kiran01bm20bd95099cb4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6d75df8f..c6a9e1dd 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]### Added+- **RDS — opt-in Aurora PostgreSQL reader instances backed by real streaming replication** — with `MINISTACK_RDS_PG_CLUSTER_REPLICATION=1`, the second and later members of an Aurora PostgreSQL cluster launch their own PostgreSQL containers, cloned from the writer with `pg_basebackup` and running as hot standbys that stream WAL from it. A reader is genuinely read-only (`pg_is_in_recovery()` is true, writes fail with `25006`), sees data written to the writer, and the cluster `ReaderEndpoint` resolves to an available reader instead of the writer — so read/write connection routing, replica-lag handling, and reader-endpoint pooling can be rehearsed locally. Off by default: without the flag (and for Aurora MySQL) members keep aliasing the writer's single shared container. Requires the shared Docker network (`DOCKER_NETWORK`); deleting a reader removes only its own container. Not yet modeled (follow-up slices of #1325): stopping a cluster with replicating readers is rejected with `InvalidDBClusterStateFault`, and a persisted reader demotes to a shared-container alias on warm boot. Contributed by @kiran01bm.+- **CloudFormation — API Gateway (v1) API keys and usage plans** — `AWS::ApiGateway::ApiKey`, `AWS::ApiGateway::UsagePlan`, and `AWS::ApiGateway::UsagePlanKey` failed stack creation with `Unsupported resource type`, even though the runtime `CreateApiKey` / `CreateUsagePlan` APIs were already served. The three resource types now provision through those same stores: an `ApiKey` supports a pinned `Value`, `Enabled`, and `StageKeys`, updates `Name` / `Description` / `Enabled` in place, and exposes both `Ref` and `Fn::GetAtt APIKeyId` as the key id; a `UsagePlan` maps `Throttle` / `Quota` / `ApiStages` and returns its id via `Ref` and `Fn::GetAtt Id`; a `UsagePlanKey` associates a key with a plan and is torn down from the plan on delete. This unblocks CDK `RestApi` / `ApiKey` constructs and Terraform `aws_api_gateway_api_key` deployed through CloudFormation. Contributed by @ryan-bennett.- **AWS IoT Jobs — control plane and device data plane** — `CreateJob` fell through to `Unsupported IoT path` and the `iot-jobs-data` endpoint did not exist, so device OTA loops could not be rehearsed locally. The `iot` service now serves the nine job operations and a new `iot-jobs-data` service the four device ones (incl. the `$next` sentinel), sharing one store and the AWS execution state machine. A `documentSource` job gets a placeholder, not the S3 object. Contributed by @iot-rocket.### Fixeddiff --git a/README.md b/README.mdindex 9952755e..b4c0b2d7 100644--- a/README.md+++ b/README.md@@ -803,6 +803,7 @@ end-to-end without any client config.| `GLUE_DOCKER_IMAGE` | (auto by `GlueVersion`) | Override the `amazon/aws-glue-libs` PySpark image used for Spark Glue jobs. Defaults: `glue_libs_4.0.0_image_01` (GlueVersion 4.0), `glue_libs_3.0.0_image_01` (GlueVersion 3.0) || `RDS_PERSIST` | `0` | Set `1` to use Docker named volumes for RDS containers instead of tmpfs. Storage grows dynamically with no fixed cap || `MINISTACK_RDS_PUBLIC_ENDPOINT` | `0` | Set `1` when MiniStack itself runs in a Docker container but RDS clients connect from outside that network (remote MiniStack host, host-side clients, CI runners). `DescribeDBInstances` then returns `{MINISTACK_HOST, host_port}` — the externally-reachable host-published port — instead of the container-internal address. Set `MINISTACK_HOST` to the host clients will use |+| `MINISTACK_RDS_PG_CLUSTER_REPLICATION` | `0` | Set `1` to give the second and later members of an Aurora PostgreSQL cluster their own hot-standby containers streaming from the writer (real read-only replicas; the cluster `ReaderEndpoint` resolves to one when available). Requires `DOCKER_NETWORK`. Off: all members alias the writer's single shared container || `DSQL_BASE_PORT` | `25432` | Starting host port for Aurora DSQL cluster proxies (one port per cluster) || `DSQL_STRICT` | `0` | Set `1` to run a real Postgres container per DSQL cluster (requires Docker; capped by a fixed port window off `DSQL_BASE_PORT`, default 30 — further clusters are metadata-only stubs). Default `0` is control-plane-only || `DSQL_PG_IMAGE` | `postgres:16-alpine` | Image used for per-cluster DSQL backend containers |diff --git a/ministack/services/rds.py b/ministack/services/rds.pyindex a5c2fb54..78e0e6ee 100644--- a/ministack/services/rds.py+++ b/ministack/services/rds.py@@ -79,6 +79,14 @@ DOCKER_NETWORK = os.environ.get("DOCKER_NETWORK", "")# clients of a containerised ministack). Off by default: existing in-network# behavior unchanged.RDS_PUBLIC_ENDPOINT = os.environ.get("MINISTACK_RDS_PUBLIC_ENDPOINT", "0").lower() in ("1", "true", "yes")+# Opt-in: per-instance Aurora PostgreSQL reader containers backed by real+# streaming replication (#1325). Scoped to aurora-postgresql by name and by+# gate: Aurora MySQL readers keep aliasing the single shared container, and+# the physical-replication bootstrap below is PostgreSQL-specific+# (pg_basebackup + hot standby). Off by default: existing behavior unchanged.+RDS_PG_CLUSTER_REPLICATION = os.environ.get(+ "MINISTACK_RDS_PG_CLUSTER_REPLICATION", "0",+).lower() in ("1", "true", "yes")_instances = AccountRegionScopedDict()_clusters = AccountRegionScopedDict()@@ -108,6 +116,66 @@ _MYSQL_REPLICATION_RETRY_INTERVAL = 1# not an environment variable._MYSQL_BINLOG_RETENTION_SECONDS = 604800 # 7 days+# Streaming-replication credentials for per-instance Aurora PostgreSQL reader+# containers (#1325). ``rdsrepladmin`` mirrors the reserved role name real RDS+# PostgreSQL provisions for replication. Fixed local-only credentials, same+# convention as the MySQL replication user above.+_PG_REPLICATION_USER = "rdsrepladmin"+_PG_REPLICATION_PASSWORD = "rdsrepladmin-local-password"++# Runs on the writer container (docker exec). Idempotently creates the+# replication role and opens pg_hba.conf for remote replication connections:+# the stock postgres image only allows replication from localhost, so+# without the extra rule a reader's pg_basebackup is rejected. The role is+# created with an explicit scram-sha-256 verifier so the pg_hba method below+# authenticates it on every supported major (password_encryption defaulted+# to md5 before PostgreSQL 14).+_PG_REPLICATION_SOURCE_SCRIPT = f"""set -eu+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<'SQL'+SET password_encryption = 'scram-sha-256';+DO $do$+BEGIN+ IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '{_PG_REPLICATION_USER}') THEN+ CREATE ROLE {_PG_REPLICATION_USER} WITH REPLICATION LOGIN+ PASSWORD '{_PG_REPLICATION_PASSWORD}';+ END IF;+END+$do$;+SQL+if ! grep -q "host replication {_PG_REPLICATION_USER}" "$PGDATA/pg_hba.conf"; then+ printf 'host replication {_PG_REPLICATION_USER} all scram-sha-256\\n' \\+ >> "$PGDATA/pg_hba.conf"+fi+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \\+ -c "SELECT pg_reload_conf()" >/dev/null+"""++# Container command for a reader: bypass the image entrypoint (which would+# initdb a brand-new independent database) and instead clone the writer with+# pg_basebackup, then start PostgreSQL as a hot standby. The base backup is+# retried until the writer is ready and the replication role exists, so+# reader creation may safely overlap writer creation. --write-recovery-conf+# records the primary connection (including the password, because it is+# passed inside --dbname) so the standby streams WAL from the writer.+_PG_READER_BOOTSTRAP_SCRIPT = """set -eu+until pg_isready -h "$MINISTACK_PG_PRIMARY_HOST" \\+ -p "$MINISTACK_PG_PRIMARY_PORT" >/dev/null 2>&1; do+ sleep 1+done+mkdir -p "$PGDATA"+chown -R postgres:postgres "$PGDATA"+chmod 700 "$PGDATA"+until gosu postgres pg_basebackup \\+ --dbname="host=$MINISTACK_PG_PRIMARY_HOST port=$MINISTACK_PG_PRIMARY_PORT \\+user=$MINISTACK_PG_REPLICATION_USER password=$MINISTACK_PG_REPLICATION_PASSWORD" \\+ --pgdata="$PGDATA" --write-recovery-conf --wal-method=stream \\+ --checkpoint=fast; do+ find "$PGDATA" -mindepth 1 -delete+ sleep 1+done+exec gosu postgres postgres+"""+# Aurora MySQL versions are the creatable set returned by AWS RDS as of# 2026-07-09. Refresh with:# aws rds describe-db-engine-versions --engine aurora-mysql \@@ -260,6 +328,11 @@ def restore_state(data):cluster = _clusters._data[key]cluster["_shared_container_id"] = Nonecluster.pop("_mysql_replication_retry_marker", None)+ # Re-verify replication provisioning against the respawned writer+ # (#1325): the role and pg_hba rule live in the cluster volume and+ # normally survive, but the flag must not outlive a volume that+ # did not.+ cluster.pop("_pg_replication_source_ready", None)if cluster.get("_mysql_replication_source_arn"):# A respawned MySQL container gets a new hostname, while the# persisted replica repository and relay-log names belong to the@@ -293,6 +366,12 @@ def restore_state(data):account_id, region, instance_id = keyinst["_docker_container_id"] = Noneinst["DBInstanceStatus"] = "creating"+ # Warm boot of per-instance reader containers is not implemented+ # yet (#1325 slice 3). Demote a persisted replicating reader to a+ # shared-container alias member so the restored cluster is fully+ # usable instead of leaving a reader that points at a container+ # the restart killed.+ inst.pop("_pg_standby", None)if RDS_PERSIST:inst.setdefault("_docker_volume_name",@@ -1056,6 +1135,303 @@ def _cluster_owned_container_ids(cluster):return ids+def _pg_cluster_replication_enabled(cluster):+ """Whether this cluster launches per-instance reader containers (#1325).++ Requires the ``MINISTACK_RDS_PG_CLUSTER_REPLICATION`` opt-in and an+ Aurora PostgreSQL cluster: the reader bootstrap is physical streaming+ replication (pg_basebackup + hot standby), which has no MySQL analog+ here — Aurora MySQL members keep aliasing the shared container.+ """+ return (+ RDS_PG_CLUSTER_REPLICATION+ and cluster.get("Engine") == "aurora-postgresql"+ )+++def _ensure_pg_replication_source(cluster_id, cluster):+ """Provision streaming-replication access on the writer's container.++ Idempotent: creates the replication role and appends the pg_hba.conf+ rule that allows remote replication connections, then reloads+ configuration. Returns True once the writer can serve pg_basebackup+ for reader containers; False when provisioning could not run (no+ Docker, no writer container, or the exec failed) — callers retry.++ The check-then-provision runs under ``_shared_container_lock`` so+ concurrent reader workers on the same cluster provision once instead+ of racing the flag (the script tolerates a rerun, but two interleaved+ executions against the same writer are pointless work).+ """+ with _shared_container_lock:+ if cluster.get("_pg_replication_source_ready"):+ return True+ docker_client = _get_docker()+ container_id = cluster.get("_shared_container_id")+ if not docker_client or not container_id:+ return False+ try:+ container = docker_client.containers.get(container_id)+ exit_code, output = container.exec_run(+ ["sh", "-c", _PG_REPLICATION_SOURCE_SCRIPT],+ )+ except Exception as e:+ logger.warning(+ "RDS: failed to provision replication source for cluster %s: %s",+ cluster_id, e,+ )+ return False+ if exit_code != 0:+ logger.warning(+ "RDS: replication source provisioning for cluster %s exited %s: %s",+ cluster_id,+ exit_code,+ output.decode(errors="replace") if isinstance(output, bytes) else output,+ )+ return False+ cluster["_pg_replication_source_ready"] = True+ return True+++def _start_pg_reader_container(db_id, cluster):+ """Launch a member-owned hot-standby container for an Aurora PostgreSQL+ cluster (#1325).++ Returns a start-result dict ({"started", "failed", endpoint fields}),+ or None when per-instance replication cannot run here — no Docker, no+ shared internal Docker network, or no known writer address — in which+ case the caller falls back to shared-container aliasing, the flag-off+ behavior.+ """+ docker_client = _get_docker()+ if not docker_client:+ return None+ cluster_id = cluster["DBClusterIdentifier"]+ ms_network = _get_ministack_network(docker_client)+ writer_host = cluster.get("_shared_internal_address")+ writer_port = cluster.get("_shared_internal_port")+ if not ms_network or not writer_host:+ # pg_basebackup runs inside the reader container and must reach the+ # writer container; without a shared Docker network there is no+ # address that works from both the host and the reader.+ logger.warning(+ "RDS: MINISTACK_RDS_PG_CLUSTER_REPLICATION is on but cluster %s "+ "has no internal Docker network address; reader %s falls back "+ "to the shared container",+ cluster_id, db_id,+ )+ return None+ engine = cluster.get("Engine", "aurora-postgresql")+ engine_version = cluster.get("EngineVersion") or _default_engine_version(engine)+ master_user = cluster.get("MasterUsername", "admin")+ master_pass = cluster.get("_MasterUserPassword", "password")+ db_name = cluster.get("DatabaseName") or "mydb"+ image, env, container_port, data_path = _docker_image_for_engine(+ engine, engine_version, master_user, master_pass, db_name,+ )+ if not image:+ return None+ env = dict(env)+ env.update({+ "MINISTACK_PG_PRIMARY_HOST": str(writer_host),+ "MINISTACK_PG_PRIMARY_PORT": str(writer_port or container_port),+ "MINISTACK_PG_REPLICATION_USER": _PG_REPLICATION_USER,+ "MINISTACK_PG_REPLICATION_PASSWORD": _PG_REPLICATION_PASSWORD,+ })+ host_port = _next_port()+ container_kwargs = dict(+ image=image,+ detach=True,+ environment=env,+ command=["sh", "-c", _PG_READER_BOOTSTRAP_SCRIPT],+ ports={f"{container_port}/tcp": host_port},+ name=_rds_docker_name(db_id),+ network=ms_network,+ labels={+ "ministack": "rds",+ "db_id": db_id,+ "cluster_id": cluster_id,+ "account_id": get_account_id(),+ "region": get_region(),+ },+ )+ volume_name = None+ if RDS_PERSIST:+ volume_name = _rds_docker_volume_name(db_id)+ container_kwargs["volumes"] = {+ volume_name: {"bind": data_path, "mode": "rw"},+ }+ else:+ container_kwargs["tmpfs"] = {+ data_path: f"rw,noexec,nosuid,size={RDS_TMPFS_SIZE}",+ }+ try:+ container = docker_client.containers.run(**container_kwargs)+ except Exception as e:+ logger.warning(+ "RDS: failed to start reader container for %s: %s", db_id, e,+ )+ if volume_name:+ # Docker may have auto-created the named volume before the run+ # failed; a leftover (possibly partially written) volume would+ # poison a retried CreateDBInstance under the same identifier.+ try:+ docker_client.volumes.get(volume_name).remove()+ except Exception:+ pass+ return {"started": False, "failed": True}+ endpoint_host = _MINISTACK_HOST+ endpoint_port = host_port+ internal_host = None+ internal_port = None+ try:+ container.reload()+ networks = container.attrs.get("NetworkSettings", {}).get("Networks", {})+ container_ip = networks.get(ms_network, {}).get("IPAddress", "")+ if container_ip:+ endpoint_host = container_ip+ endpoint_port = container_port+ internal_host = container_ip+ internal_port = container_port+ except Exception:+ pass+ return {+ "started": True,+ "failed": False,+ "container_id": container.id,+ "volume_name": volume_name,+ "host_port": host_port,+ "endpoint_host": endpoint_host,+ "endpoint_port": endpoint_port,+ "internal_host": internal_host,+ "internal_port": internal_port,+ "readiness_host": internal_host or "127.0.0.1",+ "readiness_port": internal_port or host_port,+ }+++def _remove_failed_pg_reader_compute(db_id, container_id, volume_name):+ """Remove a failed replicating reader's container and named volume.++ A standby that never became reachable leaves nothing worth keeping:+ its data directory holds at most a partial base backup, and the volume+ name is derived from the instance identifier, so leaving either behind+ would leak a host port and poison a retried CreateDBInstance under the+ same identifier. Cleanup for healthy readers stays in+ ``_delete_db_instance``; this runs only for bootstrap failures.+ """+ docker_client = _get_docker()+ if not docker_client:+ return+ try:+ container = docker_client.containers.get(container_id)+ container.stop(timeout=5)+ container.remove(v=True)+ except Exception as e:+ logger.warning(+ "RDS: failed to remove container of failed reader %s: %s",+ db_id, e,+ )+ if volume_name:+ try:+ docker_client.volumes.get(volume_name).remove()+ except Exception as e:+ logger.warning(+ "RDS: failed to remove volume of failed reader %s: %s",+ db_id, e,+ )+++def _bg_finalize_pg_reader(+ db_id, cluster_id, engine, master_user, master_pass, db_name,+ ready_host, ready_port, container_id,+):+ """Readiness worker for a member-owned replicating reader (#1325).++ Waits for the writer to become ready, provisions the replication+ source on it, then waits for the standby to accept authenticated+ connections (the reader container retries pg_basebackup internally+ until the source is provisioned). Same liveness contract as the+ generic worker: no wall clock — the instance stays ``creating`` while+ its container is up and booting, and flips to ``failed`` if the+ container dies first. Same staleness contract as the shared-container+ worker: if the instance record no longer points at this worker's+ container (deleted and recreated under the same identifier), the+ worker is superseded and must not touch the new record. A bootstrap+ failure removes the reader's own container and volume.+ """+ def _container_alive():+ client = _get_docker()+ if not client:+ return True+ try:+ c = client.containers.get(container_id)+ c.reload()+ return c.status not in ("exited", "dead", "removing")+ except Exception:+ return False++ def _stale(instance):+ return instance.get("_docker_container_id") != container_id++ while True:+ instance = _instances.get(db_id)+ cluster = _clusters.get(cluster_id)+ if instance is None or cluster is None:+ return # deleted while bootstrapping+ if _stale(instance):+ return # superseded: the recreated instance has its own worker+ if instance.get("DBInstanceStatus") == "failed":+ # Writer compute failed during creation; its readiness worker+ # already published the failure for every member. The standby+ # container would retry pg_basebackup against the dead writer+ # forever, so remove it.+ _remove_failed_pg_reader_compute(+ db_id, container_id, instance.get("_docker_volume_name"),+ )+ return+ if not _container_alive():+ instance["DBInstanceStatus"] = "failed"+ _remove_failed_pg_reader_compute(+ db_id, container_id, instance.get("_docker_volume_name"),+ )+ _refresh_cluster_status(cluster_id)+ return+ if cluster.get(+ "_shared_container_ready",+ ) and _ensure_pg_replication_source(cluster_id, cluster):+ break+ time.sleep(1)++ database_ready = _wait_for_database_ready(+ ready_host, ready_port, engine, master_user, master_pass, db_name,+ _container_alive,+ )+ instance = _instances.get(db_id)+ cluster = _clusters.get(cluster_id)+ if instance is None or _stale(instance):+ return+ if not database_ready:+ logger.warning(+ "RDS: reader container for %s at %s:%s exited before becoming "+ "reachable", db_id, ready_host, ready_port,+ )+ instance["DBInstanceStatus"] = "failed"+ _remove_failed_pg_reader_compute(+ db_id, container_id, instance.get("_docker_volume_name"),+ )+ _refresh_cluster_status(cluster_id)+ return+ instance["DBInstanceStatus"] = "available"+ if cluster:+ _sync_cluster_endpoints(cluster)+ _refresh_cluster_status(cluster_id)+ logger.info(+ "RDS: replicating reader %s ready at %s:%s", db_id,+ ready_host, ready_port,+ )++def _stop_cluster_shared_container(cluster_id, cluster):"""Stop, but do not remove, an Aurora cluster's database compute.@@ -2550,6 +2926,10 @@ def _set_cluster_members_status(cluster, status):def _attach_instance_to_shared_cluster(instance, cluster):+ if instance.get("_pg_standby"):+ # A replicating reader owns its container and endpoint (#1325);+ # aliasing would clobber them with the writer's.+ returnendpoint = cluster.get("_shared_endpoint")if not endpoint:return@@ -2578,13 +2958,31 @@ def _attach_instance_to_shared_cluster(instance, cluster):def _cluster_reader_endpoint(cluster):"""The endpoint a cluster's ``ReaderEndpoint`` should resolve to.- Today this is the writer's shared endpoint, so the local reader endpoint- is read/write: it resolves to the same database process as the writer.- Genuine read-only behavior requires a separate replicating process —- per-instance reader containers (#1325) will resolve here instead once a- cluster has one.+ When a member owns a replicating reader container (#1325) and it is+ available, the reader endpoint resolves there and is genuinely+ read-only. Otherwise it falls back to the writer's shared endpoint —+ matching real Aurora, where the reader endpoint follows the writer in+ a single-instance cluster, but read/write because it resolves to the+ same database process as the writer.++ A DBCluster advertises a single ``Port`` (the writer's, applied by+ ``_sync_cluster_endpoints``), so a standby only reachable on a+ different port — the host-port fallback when its network IP lookup+ failed — must not win the ReaderEndpoint address: pairing its Address+ with the writer's Port would be unreachable."""- return cluster.get("_shared_endpoint")+ shared = cluster.get("_shared_endpoint")+ for member in _cluster_member_instances(cluster):+ if (+ member.get("_pg_standby")+ and member.get("DBInstanceStatus") == "available"+ and member.get("Endpoint")+ ):+ endpoint = member["Endpoint"]+ if shared and endpoint.get("Port") != shared.get("Port"):+ continue+ return endpoint+ return shareddef _sync_cluster_endpoints(cluster):@@ -2748,7 +3146,31 @@ def _create_db_instance_impl(p):ms_network = Nonedocker_client = _get_docker()- if parent:+ reader_launch = None+ if (+ parent+ and parent.get("DBClusterMembers")+ and _pg_cluster_replication_enabled(parent)+ ):+ # Second and later members of a flag-enabled Aurora PostgreSQL+ # cluster get their own hot-standby container instead of aliasing+ # the writer's (#1325). Returns None when that cannot run here,+ # falling back to the aliasing branch below.+ reader_launch = _start_pg_reader_container(db_id, parent)+ if parent and reader_launch:+ start_result = reader_launch+ endpoint_host = reader_launch.get("endpoint_host", endpoint_host)+ endpoint_port = int(reader_launch.get("endpoint_port", endpoint_port))+ host_port = reader_launch.get("host_port")+ docker_container_id = reader_launch.get("container_id")+ docker_volume_name = reader_launch.get("volume_name")+ internal_host = reader_launch.get("internal_host")+ internal_port = reader_launch.get("internal_port")+ real_container_started = bool(reader_launch.get("started"))+ readiness_host = reader_launch.get("readiness_host")+ readiness_port = reader_launch.get("readiness_port")+ ms_network = _get_ministack_network(docker_client)+ elif parent:pending_rotation = parent.get("_pending_master_password_rotation")if pending_rotation:readiness_master_pass = pending_rotation["old_password"]@@ -2996,12 +3418,33 @@ def _create_db_instance_impl(p):"_internal_port": internal_port,"_MasterUserPassword": master_pass,}- if parent:+ if parent and reader_launch:+ # A replicating reader keeps its own endpoint and container fields;+ # the marker excludes it from shared-container aliasing and makes+ # the cluster ReaderEndpoint resolve to it once available.+ instance["_pg_standby"] = True+ elif parent:_attach_instance_to_shared_cluster(instance, parent)_instances[db_id] = instance_register_instance_in_cluster(instance)- if real_container_started:+ if real_container_started and reader_launch:+ # Replicating readers have their own readiness contract (wait for+ # the writer, provision replication, wait for the standby), so they+ # get a dedicated worker instead of the generic one below.+ reader_ctx = contextvars.copy_context()+ threading.Thread(+ target=reader_ctx.run,+ args=(+ _bg_finalize_pg_reader, db_id, cluster_id, engine,+ master_user, master_pass, db_name,+ readiness_host or endpoint_host,+ readiness_port or endpoint_port,+ docker_container_id,+ ),+ daemon=True,+ ).start()+ elif real_container_started:# Real AWS CreateDBInstance returns immediately with status="creating"# and the caller polls (or uses get_waiter('db_instance_available'))# until the database becomes reachable. Do the same: run the readiness@@ -3131,9 +3574,14 @@ def _create_db_instance_impl(p):_configure_or_defer_mysql_replication(cluster_id, cluster)for member in cluster.get("DBClusterMembers", []):inst = _instances.get(member.get("DBInstanceIdentifier"))- if inst is not None:- _attach_instance_to_shared_cluster(inst, cluster)- inst["DBInstanceStatus"] = "available"+ if inst is None or inst.get("_pg_standby"):+ # Replicating readers publish their own+ # readiness (#1325); the writer becoming ready+ # does not make a still-bootstrapping standby+ # available.+ continue+ _attach_instance_to_shared_cluster(inst, cluster)+ inst["DBInstanceStatus"] = "available"_sync_cluster_endpoints(cluster)_refresh_cluster_status(cluster_id)if ms_network and internal_host:@@ -5006,6 +5454,22 @@ def _stop_db_cluster(p):f"in {status} state but expected it to be one of available.",400,)+ if any(+ inst.get("_pg_standby") for inst in _cluster_member_instances(cluster)+ ):+ # Restarting per-instance reader containers after a stop is not+ # implemented yet (#1325 slice 3). Refusing here beats the two bad+ # alternatives: an ``available`` reader whose container is exited,+ # or a cluster wedged in ``creating`` because a member never+ # becomes available again.+ return _error(+ "InvalidDBClusterStateFault",+ "StopDBCluster is not supported for clusters with replicating "+ "reader instances yet; delete the reader instances first "+ "(MINISTACK_RDS_PG_CLUSTER_REPLICATION, "+ "https://github.com/ministackorg/ministack/issues/1325).",+ 400,+ )if not _stop_cluster_shared_container(cluster.get("DBClusterIdentifier", cluster_id), cluster,):diff --git a/tests/test_rds.py b/tests/test_rds.pyindex daa84e86..747fb7b6 100644--- a/tests/test_rds.py+++ b/tests/test_rds.py@@ -9486,3 +9486,732 @@ def test_rds_cluster_reader_endpoint_resolves_shared_endpoint():assert m._cluster_reader_endpoint(bare) is Nonem._sync_cluster_endpoints(bare)assert "ReaderEndpoint" not in bare+++# ---------------------------------------------------------------------------+# Aurora PostgreSQL per-instance replicating readers (#1325 slice 2)+# ---------------------------------------------------------------------------+++def test_rds_pg_cluster_replication_gate(monkeypatch):+ """Replicating readers are opt-in and scoped to Aurora PostgreSQL."""+ from ministack.services import rds as m++ pg_cluster = {"Engine": "aurora-postgresql"}+ mysql_cluster = {"Engine": "aurora-mysql"}++ # Flag off (the default): no cluster qualifies.+ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", False)+ assert not m._pg_cluster_replication_enabled(pg_cluster)+ assert not m._pg_cluster_replication_enabled(mysql_cluster)++ # Flag on: Aurora PostgreSQL only. Aurora MySQL members keep aliasing+ # the shared container even with the flag set.+ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", True)+ assert m._pg_cluster_replication_enabled(pg_cluster)+ assert not m._pg_cluster_replication_enabled(mysql_cluster)+++def test_rds_cluster_reader_endpoint_prefers_available_standby():+ """ReaderEndpoint resolves to an available replicating reader (#1325)."""+ from ministack.core.responses import get_account_id, set_request_account_id+ from ministack.services import rds as m++ shared = {"Address": "10.0.0.5", "Port": 5432, "HostedZoneId": "Z2R2ITUGPM61AM"}+ reader_ep = {"Address": "10.0.0.7", "Port": 5432, "HostedZoneId": "Z2R2ITUGPM61AM"}+ original_account = get_account_id()+ reader_id = "standby-ep-reader-1"+ try:+ set_request_account_id("111111111111")+ cluster = {+ "DBClusterIdentifier": "standby-ep",+ "_shared_endpoint": shared,+ "DBClusterMembers": [+ {"DBInstanceIdentifier": "standby-ep-writer", "IsClusterWriter": True},+ {"DBInstanceIdentifier": reader_id},+ ],+ }+ # Reader still bootstrapping: fall back to the writer's endpoint.+ m._instances[reader_id] = {+ "DBInstanceIdentifier": reader_id,+ "_pg_standby": True,+ "DBInstanceStatus": "creating",+ "Endpoint": reader_ep,+ }+ assert m._cluster_reader_endpoint(cluster) == shared++ # Reader available: the cluster ReaderEndpoint follows the standby+ # while the writer endpoint stays on the shared container.+ m._instances[reader_id]["DBInstanceStatus"] = "available"+ assert m._cluster_reader_endpoint(cluster) == reader_ep+ m._sync_cluster_endpoints(cluster)+ assert cluster["Endpoint"] == "10.0.0.5"+ assert cluster["ReaderEndpoint"] == "10.0.0.7"+ finally:+ m._instances.pop(reader_id, None)+ set_request_account_id(original_account)+++def test_rds_attach_instance_skips_pg_standby():+ """Shared-cluster aliasing must not clobber a replicating reader (#1325)."""+ from ministack.services import rds as m++ cluster = {+ "DBClusterIdentifier": "attach-guard",+ "_shared_endpoint": {"Address": "10.0.0.5", "Port": 5432},+ "_shared_container_id": "shared-c",+ }+ standby_endpoint = {"Address": "10.0.0.7", "Port": 5432}+ standby = {+ "DBInstanceIdentifier": "attach-guard-reader",+ "_pg_standby": True,+ "Endpoint": standby_endpoint,+ "_docker_container_id": "reader-c",+ }+ m._attach_instance_to_shared_cluster(standby, cluster)+ assert standby["Endpoint"] == standby_endpoint+ assert standby["_docker_container_id"] == "reader-c"+ assert "_shared_cluster_id" not in standby+ # The reader keeps owning its container (delete removes it, #1339).+ assert m._instance_owns_container(standby)+++def _pg_reader_fake_docker():+ """Fake Docker client for _start_pg_reader_container unit tests."""+ class FakeContainer:+ id = "reader-container-1"+ status = "running"+ attrs = {+ "NetworkSettings": {+ "Networks": {"ms_net": {"IPAddress": "10.0.0.7"}},+ },+ }++ def reload(self):+ pass++ container = FakeContainer()+ calls = {}++ class FakeContainers:+ def run(self, **kwargs):+ calls["run"] = kwargs+ return container++ def get(self, identifier):+ if identifier == container.id:+ return container+ raise Exception("not found")++ class FakeDocker:+ containers = FakeContainers()++ return FakeDocker(), container, calls+++def test_rds_start_pg_reader_container_requires_internal_network(monkeypatch):+ """No shared Docker network: no reader container, caller falls back."""+ from ministack.services import rds as m++ docker, _container, calls = _pg_reader_fake_docker()+ monkeypatch.setattr(m, "_get_docker", lambda: docker)+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: None)+ cluster = {+ "DBClusterIdentifier": "no-net",+ "Engine": "aurora-postgresql",+ "_shared_internal_address": None,+ }+ assert m._start_pg_reader_container("no-net-1", cluster) is None+ assert "run" not in calls+++def test_rds_start_pg_reader_container_launches_owned_container(monkeypatch):+ """A reader container bootstraps as a hot standby cloned from the writer."""+ from ministack.services import rds as m++ docker, container, calls = _pg_reader_fake_docker()+ monkeypatch.setattr(m, "_get_docker", lambda: docker)+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: "ms_net")+ monkeypatch.setattr(m, "_next_port", lambda: 16091)+ cluster = {+ "DBClusterIdentifier": "reader-launch",+ "Engine": "aurora-postgresql",+ "EngineVersion": "17.7",+ "MasterUsername": "admin",+ "_MasterUserPassword": "pw",+ "DatabaseName": "appdb",+ "_shared_internal_address": "10.0.0.5",+ "_shared_internal_port": 5432,+ }+ result = m._start_pg_reader_container("reader-launch-1", cluster)+ assert result["started"] is True+ assert result["container_id"] == container.id+ # Container-internal address on the shared network, like other+ # in-network RDS endpoints.+ assert result["endpoint_host"] == "10.0.0.7"+ assert result["endpoint_port"] == 5432++ kwargs = calls["run"]+ assert kwargs["name"] == m._rds_docker_name("reader-launch-1")+ assert kwargs["network"] == "ms_net"+ assert kwargs["labels"]["db_id"] == "reader-launch-1"+ assert kwargs["labels"]["cluster_id"] == "reader-launch"+ # The image entrypoint (which would initdb an independent database) is+ # bypassed: the reader clones the writer with pg_basebackup and starts+ # as a hot standby.+ assert kwargs["command"][:2] == ["sh", "-c"]+ script = kwargs["command"][2]+ assert "pg_basebackup" in script+ assert "--write-recovery-conf" in script+ env = kwargs["environment"]+ assert env["MINISTACK_PG_PRIMARY_HOST"] == "10.0.0.5"+ assert env["MINISTACK_PG_PRIMARY_PORT"] == "5432"+ # The launched reader is member-owned compute (#1339 semantics).+ assert m._instance_owns_container(+ {"_docker_container_id": result["container_id"]},+ )+++def test_rds_pg_replicating_reader_lifecycle(monkeypatch):+ """Flag on: the second member owns a standby container end to end.++ Fake-Docker walk of the control plane: the writer aliases the shared+ cluster container, the reader launches its own, the cluster+ ReaderEndpoint follows the reader once available, StopDBCluster is+ refused while a replicating reader exists (slice 3), and deleting the+ reader removes only reader-owned compute and falls the ReaderEndpoint+ back to the writer.+ """+ from ministack.services import rds as m++ containers = {}+ removed = []++ class FakeContainer:+ def __init__(self, name, kwargs):+ self.id = f"{name}-container"+ self.name = name+ self.kwargs = kwargs+ self.status = "running"+ net = kwargs.get("network", "ms_net")+ is_member = "db_id" in kwargs.get("labels", {})+ ip = "10.0.0.7" if is_member else "10.0.0.5"+ self.attrs = {+ "NetworkSettings": {"Networks": {net: {"IPAddress": ip}}},+ }++ def reload(self):+ pass++ def exec_run(self, _cmd):+ return 0, b""++ def stop(self, timeout=5):+ self.status = "exited"++ def remove(self, force=False, v=False):+ removed.append(self.id)+ containers.pop(self.name, None)++ class FakeContainers:+ def run(self, **kwargs):+ container = FakeContainer(kwargs["name"], kwargs)+ containers[kwargs["name"]] = container+ return container++ def get(self, identifier):+ for container in containers.values():+ if identifier in (container.id, container.name):+ return container+ raise Exception("not found")++ class FakeDocker:+ containers = FakeContainers()++ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", True)+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: "ms_net")+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "pgrepl-cluster",+ "Engine": "aurora-postgresql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ m._create_db_instance({+ "DBInstanceIdentifier": "pgrepl-writer",+ "DBClusterIdentifier": "pgrepl-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-postgresql",+ })+ cluster = m._clusters.get("pgrepl-cluster")+ writer = m._instances.get("pgrepl-writer")+ assert _poll_until(+ lambda: writer["DBInstanceStatus"] == "available",+ )+ # Single-member cluster: ReaderEndpoint follows the writer.+ assert cluster["ReaderEndpoint"] == "10.0.0.5"++ m._create_db_instance({+ "DBInstanceIdentifier": "pgrepl-reader",+ "DBClusterIdentifier": "pgrepl-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-postgresql",+ })+ reader = m._instances.get("pgrepl-reader")+ assert reader["_pg_standby"] is True+ assert _poll_until(+ lambda: reader["DBInstanceStatus"] == "available",+ )+ # The reader owns a container distinct from the shared one.+ assert reader["_docker_container_id"] != cluster["_shared_container_id"]+ assert m._instance_owns_container(reader)+ # Replication access was provisioned on the writer.+ assert cluster["_pg_replication_source_ready"] is True+ # ReaderEndpoint now resolves to the standby; the writer endpoint+ # stays on the shared container; the writer stays the only writer.+ assert cluster["ReaderEndpoint"] == "10.0.0.7"+ assert cluster["Endpoint"] == "10.0.0.5"+ writers = [+ member for member in cluster["DBClusterMembers"]+ if member.get("IsClusterWriter")+ ]+ assert [w["DBInstanceIdentifier"] for w in writers] == ["pgrepl-writer"]+ assert _poll_until(lambda: cluster["Status"] == "available")++ # Stop/start with a replicating reader is slice 3: refused, not+ # published wrongly.+ status, _, body = m._stop_db_cluster({+ "DBClusterIdentifier": "pgrepl-cluster",+ })+ assert status == 400+ assert "InvalidDBClusterStateFault" in str(body)++ # Deleting the reader removes only its own compute; the+ # ReaderEndpoint falls back to the writer.+ reader_container_id = reader["_docker_container_id"]+ m._delete_db_instance({+ "DBInstanceIdentifier": "pgrepl-reader",+ "SkipFinalSnapshot": "true",+ })+ assert reader_container_id in removed+ assert cluster["_shared_container_id"] not in removed+ assert cluster["ReaderEndpoint"] == "10.0.0.5"+ finally:+ m._instances.clear()+ m._clusters.clear()+++def _pg_repl_fake_docker(m, monkeypatch, member_ips, exec_calls, removed):+ """Fake Docker for replicating-reader tests.++ ``member_ips`` maps a member ``db_id`` label to the container IP the+ fake reports; containers without a ``db_id`` label (the cluster-owned+ shared container) get 10.0.0.5. ``exec_calls`` collects the names of+ containers that received ``exec_run``; ``removed`` collects removed+ container ids.+ """+ containers = {}++ class FakeContainer:+ def __init__(self, name, kwargs):+ self.id = f"{name}-container"+ self.name = name+ self.kwargs = kwargs+ self.status = "running"+ net = kwargs.get("network", "ms_net")+ db_id = kwargs.get("labels", {}).get("db_id")+ ip = member_ips.get(db_id, "10.0.0.5")+ self.attrs = {+ "NetworkSettings": {"Networks": {net: {"IPAddress": ip}}},+ }++ def reload(self):+ pass++ def exec_run(self, _cmd):+ exec_calls.append(self.name)+ return 0, b""++ def stop(self, timeout=5):+ self.status = "exited"++ def remove(self, force=False, v=False):+ removed.append(self.id)+ containers.pop(self.name, None)++ class FakeContainers:+ def run(self, **kwargs):+ container = FakeContainer(kwargs["name"], kwargs)+ containers[kwargs["name"]] = container+ return container++ def get(self, identifier):+ for container in containers.values():+ if identifier in (container.id, container.name):+ return container+ raise Exception("not found")++ class FakeDocker:+ containers = FakeContainers()++ monkeypatch.setattr(m, "RDS_PG_CLUSTER_REPLICATION", True)+ monkeypatch.setattr(m, "_get_docker", lambda: FakeDocker())+ monkeypatch.setattr(m, "_get_ministack_network", lambda _client: "ms_net")+ monkeypatch.setattr(m, "_is_host_port_free", lambda _port: True)+++def test_rds_pg_two_replicating_readers_provision_source_once(monkeypatch):+ """Two readers on one cluster: the writer is provisioned exactly once,+ ReaderEndpoint selection is deterministic (member order), and a standby+ whose port cannot pair with the cluster Port never wins the address.+ """+ from ministack.services import rds as m++ exec_calls = []+ removed = []+ _pg_repl_fake_docker(+ m, monkeypatch,+ {"pgrepl2-reader1": "10.0.0.7", "pgrepl2-reader2": "10.0.0.8"},+ exec_calls, removed,+ )+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "pgrepl2-cluster",+ "Engine": "aurora-postgresql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ for db_id in ("pgrepl2-writer", "pgrepl2-reader1", "pgrepl2-reader2"):+ m._create_db_instance({+ "DBInstanceIdentifier": db_id,+ "DBClusterIdentifier": "pgrepl2-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-postgresql",+ })+ cluster = m._clusters.get("pgrepl2-cluster")+ reader1 = m._instances.get("pgrepl2-reader1")+ reader2 = m._instances.get("pgrepl2-reader2")+ assert _poll_until(+ lambda: reader1["DBInstanceStatus"] == "available"+ and reader2["DBInstanceStatus"] == "available",+ )+ assert cluster["_pg_replication_source_ready"] is True+ # Replication provisioning ran exactly once on the shared (writer)+ # container even with two reader workers racing the flag.+ shared_execs = [+ name for name in exec_calls+ if name == m._rds_cluster_docker_name("pgrepl2-cluster")+ ]+ assert len(shared_execs) == 1+ assert exec_calls == shared_execs # nothing exec'd on the readers++ # Member order (reader1 first) decides the ReaderEndpoint.+ assert cluster["ReaderEndpoint"] == "10.0.0.7"++ # Deleting the first reader moves the endpoint to the second.+ m._delete_db_instance({+ "DBInstanceIdentifier": "pgrepl2-reader1",+ "SkipFinalSnapshot": "true",+ })+ assert cluster["ReaderEndpoint"] == "10.0.0.8"++ # A standby only reachable on a different port than the cluster+ # Port must not win the address: fall back to the writer.+ reader2["Endpoint"]["Port"] = 5999+ m._sync_cluster_endpoints(cluster)+ assert cluster["ReaderEndpoint"] == "10.0.0.5"+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_pg_reader_bootstrap_failure_removes_reader_compute(monkeypatch):+ """A standby that never becomes reachable is failed AND its container+ is removed — no leaked compute waiting for a manual DeleteDBInstance.+ """+ from ministack.services import rds as m++ exec_calls = []+ removed = []+ _pg_repl_fake_docker(+ m, monkeypatch, {"pgreplf-reader": "10.0.0.7"}, exec_calls, removed,+ )+ # The writer (10.0.0.5) becomes ready; the standby never does.+ monkeypatch.setattr(+ m, "_wait_for_database_ready",+ lambda host, *_args: host != "10.0.0.7",+ )++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "pgreplf-cluster",+ "Engine": "aurora-postgresql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ for db_id in ("pgreplf-writer", "pgreplf-reader"):+ m._create_db_instance({+ "DBInstanceIdentifier": db_id,+ "DBClusterIdentifier": "pgreplf-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-postgresql",+ })+ cluster = m._clusters.get("pgreplf-cluster")+ reader = m._instances.get("pgreplf-reader")+ reader_container_id = reader["_docker_container_id"]+ assert _poll_until(+ lambda: reader["DBInstanceStatus"] == "failed",+ )+ assert reader_container_id in removed+ assert cluster["_shared_container_id"] not in removed+ # The ReaderEndpoint never advertises the failed standby.+ assert cluster["ReaderEndpoint"] == "10.0.0.5"+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_pg_stale_reader_worker_is_inert(monkeypatch):+ """A superseded reader worker (captured container id no longer the+ instance's) must not touch the recreated instance's status or compute.+ """+ from ministack.services import rds as m++ exec_calls = []+ removed = []+ _pg_repl_fake_docker(+ m, monkeypatch, {"pgrepls-reader": "10.0.0.7"}, exec_calls, removed,+ )+ monkeypatch.setattr(m, "_wait_for_database_ready", lambda *_args: True)++ m._instances.clear()+ m._clusters.clear()+ try:+ m._create_db_cluster({+ "DBClusterIdentifier": "pgrepls-cluster",+ "Engine": "aurora-postgresql",+ "MasterUsername": "admin",+ "MasterUserPassword": "password123",+ })+ for db_id in ("pgrepls-writer", "pgrepls-reader"):+ m._create_db_instance({+ "DBInstanceIdentifier": db_id,+ "DBClusterIdentifier": "pgrepls-cluster",+ "DBInstanceClass": "db.r6g.large",+ "Engine": "aurora-postgresql",+ })+ reader = m._instances.get("pgrepls-reader")+ assert _poll_until(+ lambda: reader["DBInstanceStatus"] == "available",+ )++ # A worker whose captured container id was superseded (delete ++ # recreate under the same identifier) returns without mutating.+ m._bg_finalize_pg_reader(+ "pgrepls-reader", "pgrepls-cluster", "aurora-postgresql",+ "admin", "password123", "mydb", "10.0.0.7", 5432,+ "stale-container-id",+ )+ assert reader["DBInstanceStatus"] == "available"+ assert removed == []+ finally:+ m._instances.clear()+ m._clusters.clear()+++def test_rds_restore_state_demotes_pg_standby(monkeypatch):+ """Warm boot demotes a persisted replicating reader to an alias member.++ Restarting reader containers across host restarts is #1325 slice 3;+ until then a restored reader must not stay marked ``_pg_standby``+ pointing at a container the restart killed, and the cluster must+ re-verify replication provisioning against the respawned writer.+ """+ from ministack.core.responses import (+ AccountRegionScopedDict,+ get_account_id,+ get_region,+ )+ from ministack.services import rds as m++ monkeypatch.setattr(m, "_get_docker", lambda: None)+ key_scope = (get_account_id(), get_region())+ m._instances.clear()+ m._clusters.clear()+ try:+ clusters = AccountRegionScopedDict()+ instances = AccountRegionScopedDict()+ clusters._data[(*key_scope, "warm-pg")] = {+ "DBClusterIdentifier": "warm-pg",+ "Engine": "aurora-postgresql",+ "Status": "available",+ "_pg_replication_source_ready": True,+ "DBClusterMembers": [+ {"DBInstanceIdentifier": "warm-pg-writer", "IsClusterWriter": True},+ {"DBInstanceIdentifier": "warm-pg-reader", "IsClusterWriter": False},+ ],+ }+ instances._data[(*key_scope, "warm-pg-writer")] = {+ "DBInstanceIdentifier": "warm-pg-writer",+ "DBClusterIdentifier": "warm-pg",+ "_shared_cluster_id": "warm-pg",+ }+ instances._data[(*key_scope, "warm-pg-reader")] = {+ "DBInstanceIdentifier": "warm-pg-reader",+ "DBClusterIdentifier": "warm-pg",+ "_pg_standby": True,+ "_docker_container_id": "dead-reader-container",+ }+ m.restore_state({"clusters": clusters, "instances": instances})++ reader = m._instances.get("warm-pg-reader")+ assert reader is not None+ assert "_pg_standby" not in reader+ cluster = m._clusters.get("warm-pg")+ assert "_pg_replication_source_ready" not in cluster+ # The demoted reader re-attaches to the shared cluster endpoint like+ # any alias member (no Docker in this test: restore publishes the+ # endpoint without real compute).+ assert _poll_until(+ lambda: reader.get("DBInstanceStatus") == "available",+ )+ assert reader.get("_shared_cluster_id") == "warm-pg"+ assert m._cluster_reader_endpoint(cluster) == cluster["_shared_endpoint"]+ finally:+ m._instances.clear()+ m._clusters.clear()+++_PG_REPLICATION_LIVE = (+ os.environ.get("DOCKER_NETWORK")+ and os.environ.get(+ "MINISTACK_RDS_PG_CLUSTER_REPLICATION", "0",+ ).lower() in ("1", "true", "yes")+)+++def _pg_connect(endpoint, user="admin", password=PASSWORD, database=DATABASE):+ import psycopg2++ return psycopg2.connect(+ host=endpoint["Address"],+ port=int(endpoint["Port"]),+ user=user,+ password=password,+ dbname=database,+ connect_timeout=5,+ )+++def _live_pg_cluster(rds):+ suffix = uuid.uuid4().hex[:10]+ cluster_id = f"pgrepl-{suffix}"+ writer_id = f"{cluster_id}-writer"+ reader_id = f"{cluster_id}-reader"+ try:+ rds.create_db_cluster(+ DBClusterIdentifier=cluster_id,+ Engine="aurora-postgresql",+ MasterUsername="admin",+ MasterUserPassword=PASSWORD,+ DatabaseName=DATABASE,+ )+ for db_id in (writer_id, reader_id):+ rds.create_db_instance(+ DBInstanceIdentifier=db_id,+ DBClusterIdentifier=cluster_id,+ DBInstanceClass="db.r6g.large",+ Engine="aurora-postgresql",+ )+ writer = _wait_for_instance(rds, writer_id, timeout=180)+ reader = _wait_for_instance(rds, reader_id, timeout=180)+ cluster = rds.describe_db_clusters(+ DBClusterIdentifier=cluster_id,+ )["DBClusters"][0]+ yield cluster_id, writer_id, reader_id, writer, reader, cluster+ finally:+ for db_id in (reader_id, writer_id):+ try:+ rds.delete_db_instance(+ DBInstanceIdentifier=db_id,+ SkipFinalSnapshot=True,+ )+ except ClientError as e:+ if e.response["Error"]["Code"] != "DBInstanceNotFound":+ raise+ try:+ rds.delete_db_cluster(+ DBClusterIdentifier=cluster_id,+ SkipFinalSnapshot=True,+ )+ except ClientError as e:+ if e.response["Error"]["Code"] != "DBClusterNotFoundFault":+ raise+++ not _PG_REPLICATION_LIVE,+ reason="DOCKER_NETWORK and MINISTACK_RDS_PG_CLUSTER_REPLICATION not set "+ "-- live Aurora PostgreSQL replication",+)+def test_aurora_pg_replicating_reader_live(rds):+ """A flag-enabled reader is a genuine hot standby streaming from the writer."""+ with _live_pg_cluster(rds) as (_cid, _wid, _rid, writer, reader, cluster):+ # Distinct endpoints: the reader is not an alias of the writer.+ assert (+ (reader["Endpoint"]["Address"], reader["Endpoint"]["Port"])+ != (writer["Endpoint"]["Address"], writer["Endpoint"]["Port"])+ )+ # The cluster reader endpoint resolves to the reader.+ assert cluster["ReaderEndpoint"] == reader["Endpoint"]["Address"]++ with _pg_connect(reader["Endpoint"]) as conn, conn.cursor() as cursor:+ cursor.execute("SELECT pg_is_in_recovery()")+ assert cursor.fetchone() == (True,)++ with _pg_connect(writer["Endpoint"]) as conn:+ conn.autocommit = True+ with conn.cursor() as cursor:+ cursor.execute("SELECT pg_is_in_recovery()")+ assert cursor.fetchone() == (False,)+ cursor.execute(+ "CREATE TABLE repl_rows (id INT PRIMARY KEY, value TEXT)",+ )+ cursor.execute("INSERT INTO repl_rows VALUES (1, 'writer-data')")++ # The write streams to the standby (bounded poll for replica lag).+ def _reader_sees_row():+ with _pg_connect(reader["Endpoint"]) as conn, conn.cursor() as cursor:+ try:+ cursor.execute("SELECT id, value FROM repl_rows")+ except Exception:+ return False+ return cursor.fetchall() == [(1, "writer-data")]++ deadline = time.time() + 60+ while time.time() < deadline and not _reader_sees_row():+ time.sleep(1)+ assert _reader_sees_row()++ # Writes against the standby fail read-only (SQLSTATE 25006).+ import psycopg2++ with _pg_connect(reader["Endpoint"]) as conn:+ conn.autocommit = True+ with conn.cursor() as cursor:+ with pytest.raises(psycopg2.Error) as excinfo:+ cursor.execute("INSERT INTO repl_rows VALUES (2, 'nope')")+ assert excinfo.value.pgcode == "25006"