Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat(dsql): Aurora DSQL emulator with control plane and SQL-validating PG proxy (#1320)
Ryan834b8768feac
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 369c4dc9..5cf0675b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -62,7 +62,7 @@ jobs:- name: Start MiniStackrun: |- REDIS_HOST=localhost python -m hypercorn ministack.app:app --bind 0.0.0.0:4566 --keep-alive 75 > ministack.log 2>&1 &+ REDIS_HOST=localhost DSQL_CONTAINERS=1 python -m hypercorn ministack.app:app --bind 0.0.0.0:4566 --keep-alive 75 > ministack.log 2>&1 &for i in $(seq 1 15); docurl -sf http://localhost:4566/_ministack/health && breakecho "Waiting for MiniStack... ($i/15)"diff --git a/Dockerfile b/Dockerfileindex 1c7d64b9..a4ae4172 100644--- a/Dockerfile+++ b/Dockerfile@@ -60,6 +60,9 @@ ENV MINISTACK_VERSION=${MINISTACK_VERSION} \REDIS_PORT=6379 \RDS_BASE_PORT=15432 \RDS_PERSIST=0 \+ DSQL_BASE_PORT=25432 \+ DSQL_PERSIST=0 \+ DSQL_CONTAINERS=0 \ELASTICACHE_BASE_PORT=16379 \LAMBDA_EXECUTOR=local \USE_SSL=0 \diff --git a/Dockerfile.full b/Dockerfile.fullindex 52335f39..ae9a3182 100644--- a/Dockerfile.full+++ b/Dockerfile.full@@ -77,6 +77,9 @@ ENV MINISTACK_VERSION=${MINISTACK_VERSION} \REDIS_PORT=6379 \RDS_BASE_PORT=15432 \RDS_PERSIST=0 \+ DSQL_BASE_PORT=25432 \+ DSQL_PERSIST=0 \+ DSQL_CONTAINERS=0 \ELASTICACHE_BASE_PORT=16379 \LAMBDA_EXECUTOR=local \USE_SSL=0 \diff --git a/README.md b/README.mdindex b919e167..79ed5bed 100644--- a/README.md+++ b/README.md@@ -561,6 +561,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i|---------|-----------|-------|| **ECS** | CreateCluster, DeleteCluster, DescribeClusters, ListClusters, UpdateCluster, UpdateClusterSettings, PutClusterCapacityProviders, RegisterTaskDefinition, DeregisterTaskDefinition, DescribeTaskDefinition, ListTaskDefinitions, ListTaskDefinitionFamilies, DeleteTaskDefinitions, CreateService, DeleteService, DescribeServices, UpdateService, ListServices, ListServicesByNamespace, RunTask, StopTask, DescribeTasks, ListTasks, ExecuteCommand, UpdateTaskProtection, GetTaskProtection, CreateCapacityProvider, UpdateCapacityProvider, DeleteCapacityProvider, DescribeCapacityProviders, TagResource, UntagResource, ListTagsForResource, ListAccountSettings, PutAccountSetting, PutAccountSettingDefault, DeleteAccountSetting, PutAttributes, DeleteAttributes, ListAttributes, DescribeServiceDeployments, ListServiceDeployments, DescribeServiceRevisions, SubmitTaskStateChange, SubmitContainerStateChange, SubmitAttachmentStateChanges, DiscoverPollEndpoint | 47 actions; `RunTask` starts real Docker containers via Docker socket; full Terraform ECS coverage || **RDS** | CreateDBInstance, DeleteDBInstance, DescribeDBInstances, ModifyDBInstance, StartDBInstance, StopDBInstance, RebootDBInstance, CreateDBInstanceReadReplica, RestoreDBInstanceFromDBSnapshot, CreateDBCluster, DeleteDBCluster, DescribeDBClusters, ModifyDBCluster, StartDBCluster, StopDBCluster, CreateDBSnapshot, DeleteDBSnapshot, DescribeDBSnapshots, CreateDBClusterSnapshot, DescribeDBClusterSnapshots, DeleteDBClusterSnapshot, CreateDBSubnetGroup, DeleteDBSubnetGroup, DescribeDBSubnetGroups, ModifyDBSubnetGroup, CreateDBParameterGroup, DeleteDBParameterGroup, DescribeDBParameterGroups, DescribeDBParameters, ModifyDBParameterGroup, CreateDBClusterParameterGroup, DescribeDBClusterParameterGroups, DeleteDBClusterParameterGroup, DescribeDBClusterParameters, ModifyDBClusterParameterGroup, CreateOptionGroup, DeleteOptionGroup, DescribeOptionGroups, DescribeOptionGroupOptions, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, DescribeDBEngineVersions, DescribeOrderableDBInstanceOptions, CreateGlobalCluster, DescribeGlobalClusters, DeleteGlobalCluster, RemoveFromGlobalCluster, ModifyGlobalCluster | `CreateDBInstance` spins up real Postgres/MySQL Docker container, returns actual `host:port` endpoint |+| **Aurora DSQL** | CreateCluster, GetCluster, UpdateCluster, DeleteCluster, ListClusters, PutClusterPolicy, GetClusterPolicy, DeleteClusterPolicy, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spins up a real `postgres:16-alpine` Docker container fronted by a validating PG wire-protocol proxy that **fails like real DSQL** — rejects unsupported SQL (`serial`, foreign keys, extensions, ...), accepts `CREATE INDEX ASYNC` + `sys.jobs`; endpoint is `localhost:<port>`; any password accepted || **ElastiCache** | CreateCacheCluster, DeleteCacheCluster, DescribeCacheClusters, ModifyCacheCluster, RebootCacheCluster, CreateReplicationGroup, DeleteReplicationGroup, DescribeReplicationGroups, ModifyReplicationGroup, IncreaseReplicaCount, DecreaseReplicaCount, CreateCacheSubnetGroup, DescribeCacheSubnetGroups, ModifyCacheSubnetGroup, DeleteCacheSubnetGroup, CreateCacheParameterGroup, DescribeCacheParameterGroups, ModifyCacheParameterGroup, ResetCacheParameterGroup, DeleteCacheParameterGroup, DescribeCacheParameters, DescribeCacheEngineVersions, CreateUser, DescribeUsers, DeleteUser, ModifyUser, CreateUserGroup, DescribeUserGroups, DeleteUserGroup, ModifyUserGroup, ListTagsForResource, AddTagsToResource, RemoveTagsFromResource, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, DescribeEvents | `CreateCacheCluster` spins up a real Redis, Valkey, or Memcached Docker container (`Engine=redis`/`valkey`/`memcached`) || **Glue** | CreateDatabase, DeleteDatabase, GetDatabase, GetDatabases, UpdateDatabase, CreateTable, DeleteTable, GetTable, GetTables, UpdateTable, BatchDeleteTable, CreatePartition, DeletePartition, GetPartition, GetPartitions, BatchCreatePartition, BatchGetPartition, CreatePartitionIndex, GetPartitionIndexes, GetColumnStatisticsForTable, UpdateColumnStatisticsForTable, DeleteColumnStatisticsForTable, GetColumnStatisticsForPartition, UpdateColumnStatisticsForPartition, DeleteColumnStatisticsForPartition, CreateConnection, DeleteConnection, GetConnection, GetConnections, CreateCrawler, DeleteCrawler, GetCrawler, GetCrawlers, UpdateCrawler, StartCrawler, StopCrawler, GetCrawlerMetrics, CreateJob, DeleteJob, GetJob, GetJobs, UpdateJob, StartJobRun, GetJobRun, GetJobRuns, BatchStopJobRun, CreateTrigger, GetTrigger, DeleteTrigger, UpdateTrigger, StartTrigger, StopTrigger, ListTriggers, BatchGetTriggers, GetTriggers, CreateWorkflow, GetWorkflow, DeleteWorkflow, UpdateWorkflow, StartWorkflowRun, CreateUserDefinedFunction, UpdateUserDefinedFunction, DeleteUserDefinedFunction, GetUserDefinedFunction, GetUserDefinedFunctions, CreateSecurityConfiguration, DeleteSecurityConfiguration, GetSecurityConfiguration, GetSecurityConfigurations, CreateClassifier, GetClassifier, GetClassifiers, DeleteClassifier, TagResource, UntagResource, GetTags | Python shell jobs actually execute via subprocess; Spark jobs run on the official `amazon/aws-glue-libs` PySpark image (`GlueVersion: 4.0` / `3.0`) on MiniStack's Docker network; Glue-backed Iceberg REST catalog at `/iceberg` (signed with the `glue` credential scope, prefix `catalogs/{catalog}`) — read path for tables whose `metadata_location` parameter points at a metadata.json on MiniStack S3, e.g. DuckDB `ATTACH ... (TYPE iceberg, ENDPOINT 'localhost:4566/iceberg', AUTHORIZATION_TYPE 'sigv4')` with `USE_SSL=1` || **S3 Tables** | CreateTableBucket, ListTableBuckets, GetTableBucket, DeleteTableBucket, CreateNamespace, ListNamespaces, GetNamespace, DeleteNamespace, CreateTable, ListTables, GetTable, DeleteTable, GetTableMetadataLocation, UpdateTableMetadataLocation | Iceberg-format tables; embedded Iceberg REST catalog at `/iceberg` so Spark jobs (`spark.sql.catalog.*.type=rest`) can create / load / commit without an external catalog; data files in S3, metadata in memory |@@ -639,6 +640,20 @@ only `DeleteDBCluster` removes the container and its storage.---+## Aurora DSQL++Each `CreateCluster` returns an `ACTIVE` cluster with an endpoint. Set `DSQL_CONTAINERS=1` to additionally spin up a real `postgres:16-alpine` container per cluster (capped by `DSQL_MAX_CONTAINERS`, default 30) behind an in-process Postgres wire-protocol proxy.++**The proxy fails like real DSQL**: SQL that DSQL rejects fails here too (`serial` columns, foreign keys, `CREATE EXTENSION`, `TRUNCATE`, temp tables, `ALTER COLUMN TYPE`, `SET NOT NULL`, materialized views, partitioned tables, non-btree/partial/expression indexes, ...), plain `CREATE INDEX` on a non-empty table is refused (`use CREATE INDEX ASYNC instead`), `CREATE INDEX ASYNC` and `ALTER TABLE ASYNC ... VALIDATE CONSTRAINT` return a `job_id`, `sys.jobs` / `sys.wait_for_job` are emulated, CHECK constraints via ALTER TABLE require `NOT VALID`, `ALTER TABLE ... DROP COLUMN` works (several columns in one statement included) except on a primary key column, identity columns are bigint-only (CACHE 1 or ≥ 65536), index limits are enforced (8 columns/index, 24/table), transactions allow only one DDL statement with no DDL+DML mixing, `SELECT ... FOR UPDATE` requires a single table with equality predicates on the key, a transaction that outlives another session's DDL fails once with `40001` (OC001) and succeeds on retry, and the transaction limits (3,000 rows / 10 MiB / 5 min) are enforced. `DSQL_STRICT=0` disables validation.++`GetCluster` returns an `endpoint` of the form `localhost:<port>` (ports allocated from `DSQL_BASE_PORT`, default `25432`). Any user/password is accepted — IAM auth tokens from `generate_db_connect_auth_token` work but aren't validated. Connect with e.g. `psql "postgresql://admin:anything@localhost:25432/postgres"` (no TLS — use `sslmode=disable`/`prefer`; with psycopg2 use `autocommit=True`).++**Both wire protocols are validated**: the simple protocol (psql, psycopg2) and the extended `Parse`/`Bind`/`Execute` protocol used by pgjdbc, pgx, asyncpg, psycopg3 and every ORM built on prepared statements. Transaction state is read from the backend's `ReadyForQuery` status byte rather than inferred from statement text, so `BEGIN`/`COMMIT` sent over either protocol keeps the transaction rules accurate. A statement the proxy rejects inside a transaction aborts that block exactly as a real error would — later statements fail with `25P02` and `COMMIT` reports `ROLLBACK`.++Limitations: SQL analysis is heuristic regex, not a real parser (statement splitting, comment stripping and parenthesis matching are lexer-aware, but clause-level rules are pattern matches); transaction row counting is static (VALUES tuples only — `INSERT ... SELECT`/UPDATE/DELETE row counts aren't tracked); OC001 emulation is optimistic (catalog version bumps on forwarded DDL, not on successful commit); the 1 KiB PK/index key-size limit is not enforced; and in a pipelined extended-protocol batch (several statements before one `Sync`) the rules that need to inspect backend state — the `CREATE INDEX` non-empty-table probe, the 24-index limit and the `FOR UPDATE` key check — are skipped, while `CREATE INDEX ASYNC` is rejected as it is in multi-statement `Q` batches. Error messages match AWS-documented text where published, otherwise Postgres-style approximations (`0A000`, `25006`).++---+## Real Redis Endpoints (ElastiCache)```python@@ -786,7 +801,13 @@ end-to-end without any client config.| `GLUE_DOCKER_IMAGE` | (auto by `GlueVersion`) | Override the `amazon/aws-glue-libs` PySpark image used for Spark Glue jobs. Defaults: `glue_libs_4.0.0_image_01` (GlueVersion 4.0), `glue_libs_3.0.0_image_01` (GlueVersion 3.0) || `RDS_PERSIST` | `0` | Set `1` to use Docker named volumes for RDS containers instead of tmpfs. Storage grows dynamically with no fixed cap || `MINISTACK_RDS_PUBLIC_ENDPOINT` | `0` | Set `1` when MiniStack itself runs in a Docker container but RDS clients connect from outside that network (remote MiniStack host, host-side clients, CI runners). `DescribeDBInstances` then returns `{MINISTACK_HOST, host_port}` — the externally-reachable host-published port — instead of the container-internal address. Set `MINISTACK_HOST` to the host clients will use |-| `DOCKER_NETWORK` | _(unset)_ | Docker network for all container-backed services (RDS, EKS, ElastiCache, Lambda). Set to your Docker Compose network name so containers can reach each other. Replaces `LAMBDA_DOCKER_NETWORK` |+| `DSQL_BASE_PORT` | `25432` | Starting host port for Aurora DSQL cluster proxies (one port per cluster) |+| `DSQL_CONTAINERS` | `0` | Set `1` to run a real Postgres container per DSQL cluster (requires Docker; capped by `DSQL_MAX_CONTAINERS` — further clusters are metadata-only stubs). Default `0` is control-plane-only |+| `DSQL_MAX_CONTAINERS` | `30` | Maximum number of real DSQL backend containers |+| `DSQL_PG_IMAGE` | `postgres:16-alpine` | Image used for per-cluster DSQL backend containers |+| `DSQL_PERSIST` | `0` | Set `1` to use Docker named volumes for DSQL backend containers instead of tmpfs |+| `DSQL_STRICT` | `1` | Set `0` to make the DSQL wire proxy a transparent passthrough (no SQL-subset validation) |+| `DOCKER_NETWORK` | _(unset)_ | Docker network for all container-backed services (RDS, DSQL, EKS, ElastiCache, Lambda). Set to your Docker Compose network name so containers can reach each other. Replaces `LAMBDA_DOCKER_NETWORK` || `ELASTICACHE_BASE_PORT` | `16379` | Starting host port for ElastiCache containers || `ELASTICACHE_CLUSTER_MODE_REAL` | `0` | Set `1` (requires `DOCKER_NETWORK`) to provision real Redis Cluster replication groups: `NumNodeGroups × (1+ReplicasPerNodeGroup)` cluster-enabled nodes wired with `redis-cli --cluster create`, serving real `CLUSTER SLOTS` / `MOVED` redirects for cluster-aware clients || `OPENSEARCH_DATAPLANE` | `0` | Set `1` to spawn a real `opensearchproject/opensearch` container per `CreateDomain`. Default `0` returns a stub endpoint (`<domain>.ministack.local:9200`) — management plane only |@@ -891,7 +912,7 @@ Install DuckDB for full Athena SQL compatibility: `pip install ministack[full]`.When `PERSIST_STATE=1`, MiniStack saves service state to `STATE_DIR` on shutdown and reloads it on startup. Writes are atomic (write-to-tmp then rename) to prevent corruption on crash.-Services currently supporting persistence: **All services** — API Gateway v1/v2, ALB, ACM, AppConfig, AppSync, Athena, Bedrock, Cloud Map, CloudFront, CloudWatch, CloudWatch Logs, CodeBuild, Cognito, DynamoDB, EC2, ECR, ECS, EFS, EKS, ElastiCache, EMR, EventBridge, EventBridge Scheduler, Firehose, Glue, IAM/STS, Kinesis, KMS, Lambda, MSK, RDS, Route 53, S3, Secrets Manager, SES, SES v2, SNS, SQS, SSM, Step Functions, Transfer Family, WAF v2+Services currently supporting persistence: **All services** — API Gateway v1/v2, ALB, ACM, AppConfig, AppSync, Athena, Aurora DSQL, Bedrock, Cloud Map, CloudFront, CloudWatch, CloudWatch Logs, CodeBuild, Cognito, DynamoDB, EC2, ECR, ECS, EFS, EKS, ElastiCache, EMR, EventBridge, EventBridge Scheduler, Firehose, Glue, IAM/STS, Kinesis, KMS, Lambda, MSK, RDS, Route 53, S3, Secrets Manager, SES, SES v2, SNS, SQS, SSM, Step Functions, Transfer Family, WAF v2```bashdocker run -p 4566:4566 \@@ -919,7 +940,7 @@ Containers are named `lambda-<random-hex-16>` and pooled across invocations. Idl**Docker-in-Docker:** when MiniStack itself runs inside a container (with the host Docker socket mounted), it detects that automatically and copies each Lambda's code into the sibling runtime container over the socket. No shared volume or extra configuration is required. The legacy `LAMBDA_REMOTE_DOCKER_VOLUME_MOUNT` env var is deprecated and ignored.-**Networking:** when MiniStack runs in Docker Compose, set `DOCKER_NETWORK` to the Compose network name. All container-backed services (Lambda, RDS, EKS, ElastiCache) then attach to that network so Lambda code can reach MiniStack at `http://<ministack-service-name>:4566`. The legacy `LAMBDA_DOCKER_NETWORK` is still accepted (Lambda only) as a fallback.+**Networking:** when MiniStack runs in Docker Compose, set `DOCKER_NETWORK` to the Compose network name. All container-backed services (Lambda, RDS, DSQL, EKS, ElastiCache) then attach to that network so Lambda code can reach MiniStack at `http://<ministack-service-name>:4566`. The legacy `LAMBDA_DOCKER_NETWORK` is still accepted (Lambda only) as a fallback.Example `docker-compose.yml`:@@ -1315,6 +1336,7 @@ See [`Testcontainers/java-testcontainers`](Testcontainers/java-testcontainers),| SES | ✅ | ✅ | ✅ || Step Functions | ✅ | ✅ | ✅ || **RDS (real DB containers)** | ✅ | ❌ | ✅ |+| **Aurora DSQL (real PG + SQL-subset enforcement)** | ✅ | ❌ | ✅ (permissive — no dialect enforcement) || **ElastiCache (real Redis)** | ✅ | ❌ | ✅ || **ECS (real Docker containers)** | ✅ | ❌ | ✅ || **Athena (real SQL via DuckDB)** | ✅ | ❌ | ✅ |diff --git a/docker-compose.yml b/docker-compose.ymlindex 1221d19e..45fa005d 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -5,6 +5,7 @@ services:container_name: ministackports:- "4566:4566"+ # - "25432-25499:25432-25499" # Aurora DSQL cluster endpoints (DSQL data plane)environment:- GATEWAY_PORT=4566- LOG_LEVEL=INFO@@ -12,7 +13,9 @@ services:- REDIS_HOST=redis- REDIS_PORT=6379- RDS_BASE_PORT=15432- # - DOCKER_NETWORK=ministack_default # Docker network for RDS/EKS/ElastiCache/Lambda containers+ # - DSQL_BASE_PORT=25432 # first host port for DSQL cluster proxies+ # - DSQL_CONTAINERS=1 # run a real Postgres container per DSQL cluster (default: control-plane-only)+ # - DOCKER_NETWORK=ministack_default # Docker network for RDS/DSQL/EKS/ElastiCache/Lambda containers- ELASTICACHE_BASE_PORT=16379volumes:- ./data/s3:/tmp/ministack-data/s3diff --git a/ministack/app.py b/ministack/app.pyindex 328cf630..5bee83e6 100644--- a/ministack/app.py+++ b/ministack/app.py@@ -158,7 +158,7 @@ _NON_S3_VHOST_NAMES = frozenset({"apigateway", "cloudformation", "autoscaling", "codebuild", "transfer", "cur","cloudfront-kvs","appsync-api", "appsync-realtime-api",- "inspector2",+ "inspector2", "dsql",})from ministack.core.hypercorn_compat import install as _install_hypercorn_compat@@ -272,6 +272,7 @@ SERVICE_REGISTRY = {"config": {"module": "config"},"dynamodb": {"module": "dynamodb"},"dynamodbstreams": {"module": "dynamodb_streams"},+ "dsql": {"module": "dsql"},"ec2": {"module": "ec2"},"ecr": {"module": "ecr"},"ecs": {"module": "ecs"},@@ -366,7 +367,7 @@ _state_map = {"cloudfront_keyvaluestore": "cloudfront_keyvaluestore","resource_groups": "resource_groups","cloudtrail": "cloudtrail", "iot": "iot",- "inspector2": "inspector2",+ "inspector2": "inspector2", "dsql": "dsql","mediaconnect": "mediaconnect","mq": "mq","opensearch": "opensearch",@@ -422,7 +423,7 @@ BANNER = r"""ECS, RDS, ElastiCache, Glue, Athena, API Gateway, Firehose, Route53,Cognito, EC2, EMR, EBS, EFS, ALB/ELBv2, CloudFormation, KMS, ECR, CloudFront,AppSync, Cloud Map, S3 Files, RDS Data API, CodeBuild, AppConfig, Transfer, EKS,- Inspector2, IoT Core+ Inspector2, IoT Core, Aurora DSQL"""@@ -1999,6 +2000,15 @@ async def _handle_lifespan(scope, receive, send):await transfer.sftp_start()except Exception as e:logger.warning("Transfer SFTP startup failed: %s", e)+ # Start DSQL wire proxies for clusters restored from persistence.+ # Guarded on the module already being loaded (i.e. it had state+ # or was used this boot) so we never import dsql just for this.+ _dsql_mod = _loaded_modules.get("dsql")+ if _dsql_mod is not None:+ try:+ await _dsql_mod.start_restored_proxies()+ except Exception as e:+ logger.warning("DSQL proxy startup failed: %s", e)# Start the EventBridge scheduler daemon explicitly. Module-import# autostart is gated by MINISTACK_TEST_NO_AUTOSTART so unit tests# don't race; lifespan.startup is the canonical place to spin it up.@@ -2055,7 +2065,7 @@ def _stop_docker_containers():client = docker.from_env()except Exception:return- for label in ("ministack=rds", "ministack=ecs", "ministack=elasticache", "ministack=eks", "ministack=lambda"):+ for label in ("ministack=rds", "ministack=ecs", "ministack=elasticache", "ministack=eks", "ministack=lambda", "ministack=dsql"):try:# all=True so exited-but-not-removed orphans get cleaned at boot.for c in client.containers.list(all=True, filters={"label": label}):diff --git a/ministack/core/pgproxy.py b/ministack/core/pgproxy.pynew file mode 100644index 00000000..c4526f30--- /dev/null+++ b/ministack/core/pgproxy.py@@ -0,0 +1,1947 @@+"""+Asyncio Postgres wire-protocol proxy with Aurora DSQL validation.++One proxy per DSQL cluster: clients connect to the proxy on the cluster's+localhost port; the proxy forwards frames to a real Postgres backend after+enforcing DSQL's PostgreSQL-compatibility subset.++Validator behavior is gated by ``DSQL_STRICT`` (default "1"); "0" turns the+proxy into a transparent passthrough.++Both wire protocols are validated: the simple ``Q`` protocol (psql, psycopg2)+and the extended ``Parse``/``Bind``/``Execute`` protocol (pgjdbc, pgx, asyncpg,+psycopg3, and every ORM that uses prepared statements).++Transaction state is taken from the backend's ``ReadyForQuery`` status byte+rather than inferred from statement text, so it stays correct no matter which+protocol a client uses to send ``BEGIN``/``COMMIT``.++Documented limitations:+- ``CREATE INDEX ASYNC`` / ``ALTER TABLE ASYNC`` executed over the extended+ protocol run their rewritten DDL at ``Execute`` time on the proxy's own+ backend connection; a client that ``Parse``s such a statement and never+ ``Execute``s it registers no job.+- Type/name parsing is heuristic regex work, not a real SQL parser. Statement+ splitting, comment stripping and parenthesis matching are lexer-aware+ (string literals, dollar quotes, quoted identifiers and nested comments),+ but clause-level analysis is still pattern matching.+- Transaction row counting is static (``VALUES`` tuples only); ``INSERT ...+ SELECT`` / ``UPDATE`` / ``DELETE`` affected-row counts are not tracked.+- OC001 emulation is optimistic: the catalog version bumps on forwarded DDL,+ not on successful commit.+"""++import asyncio+import logging+import os+import re+import secrets+import string+import struct+import time+from datetime import datetime, timezone++logger = logging.getLogger("pgproxy")++STRICT = os.environ.get("DSQL_STRICT", "1").strip().lower() not in ("0", "false", "no", "off")++# ---------------------------------------------------------------------------+# Wire-protocol constants and frame helpers+# ---------------------------------------------------------------------------++_SSL_REQUEST = 80877103+_GSSENC_REQUEST = 80877104+_CANCEL_REQUEST = 80877102+_PROTOCOL_3 = 196608++_STARTUP_PARAMS = (+ ("server_version", "16.4"),+ ("server_encoding", "UTF8"),+ ("client_encoding", "UTF8"),+ ("DateStyle", "ISO"),+ ("integer_datetimes", "on"),+ ("standard_conforming_strings", "on"),+)++_TEXT_OID = 25+++def _frame(type_byte, payload):+ return type_byte + struct.pack("!I", len(payload) + 4) + payload+++def _error_response(sqlstate, message):+ payload = (+ b"SERROR\0VERROR\0"+ + b"C" + sqlstate.encode() + b"\0"+ + b"M" + message.encode() + b"\0"+ + b"\0"+ )+ return _frame(b"E", payload)+++def _ready(status=b"I"):+ return _frame(b"Z", status)+++def _command_complete(tag):+ return _frame(b"C", tag.encode() + b"\0")+++def _row_description(cols):+ """cols: names, or (name, type_oid) tuples."""+ out = struct.pack("!H", len(cols))+ for col in cols:+ name, oid = col if isinstance(col, tuple) else (col, _TEXT_OID)+ # table oid, column attnum, type oid, typlen, typmod, format+ out += name.encode() + b"\0" + struct.pack("!IhIhih", 0, 0, oid, -1, -1, 0)+ return _frame(b"T", out)+++def _data_row(values):+ out = struct.pack("!H", len(values))+ for v in values:+ if v is None:+ out += struct.pack("!i", -1)+ else:+ b = str(v).encode()+ out += struct.pack("!I", len(b)) + b+ return _frame(b"D", out)+++def _client_greeting():+ out = _frame(b"R", struct.pack("!I", 0)) # AuthenticationOk+ for key, value in _STARTUP_PARAMS:+ out += _frame(b"S", key.encode() + b"\0" + value.encode() + b"\0")+ out += _frame(b"K", struct.pack("!II", secrets.randbelow(2**31), secrets.randbelow(2**31)))+ out += _ready(b"I")+ return out+++async def _read_frame(reader):+ type_byte = await reader.readexactly(1)+ (length,) = struct.unpack("!I", await reader.readexactly(4))+ payload = await reader.readexactly(length - 4)+ return type_byte, payload+++# ---------------------------------------------------------------------------+# Job registry (CREATE INDEX ASYNC / sys.jobs emulation)+# ---------------------------------------------------------------------------++_proxies = {} # cluster_id -> asyncio.Server+_jobs = {} # cluster_id -> [job dict]+_catalog_versions = {} # cluster_id -> int, bumped on every DDL statement++_ID_ALPHABET = string.ascii_lowercase + string.digits++_JOB_COLUMNS = (+ "job_id", "status", "details", "job_type", "class_id", "object_id",+ "object_name", "start_time", "update_time",+)+++def get_jobs(cluster_id):+ return list(_jobs.get(cluster_id, []))+++def clear_jobs(cluster_id=None):+ """Drop job registry entries — one cluster, or all when no id is given."""+ if cluster_id is None:+ _jobs.clear()+ _catalog_versions.clear()+ else:+ _jobs.pop(cluster_id, None)+ _catalog_versions.pop(cluster_id, None)+++def _bump_catalog(cluster_id):+ """Record a catalog change (any DDL) for a cluster."""+ _catalog_versions[cluster_id] = _catalog_versions.get(cluster_id, 0) + 1+ return _catalog_versions[cluster_id]+++def _register_job(cluster_id, object_name, job_type="INDEX_BUILD"):+ now = datetime.now(timezone.utc).isoformat()+ job = {+ "job_id": "".join(secrets.choice(_ID_ALPHABET) for _ in range(26)),+ "status": "completed",+ "details": "",+ "job_type": job_type,+ "class_id": "1259",+ "object_id": str(16384 + secrets.randbelow(100000)),+ "object_name": object_name,+ "start_time": now,+ "update_time": now,+ }+ _jobs.setdefault(cluster_id, []).append(job)+ return job+++# ---------------------------------------------------------------------------+# SQL lexing primitives+#+# Everything that scans SQL text (statement splitting, comment stripping,+# parenthesis matching) goes through _skip_noise so that string literals,+# dollar-quoted bodies, quoted identifiers and comments can never be mistaken+# for structure. Without this a ';' or ')' inside a literal silently truncates+# the text a validation rule sees, and the rule passes on the fragment.+# ---------------------------------------------------------------------------++_DOLLAR_TAG_RE = re.compile(r"\$(?:[A-Za-z_]\w*)?\$")+++def _skip_noise(sql, i):+ """If ``sql[i]`` opens a literal/identifier/comment, return the index just+ past it; otherwise return ``i`` unchanged."""+ n = len(sql)+ ch = sql[i]+ if ch == "'":+ # E'...' uses backslash escapes; ordinary literals only double the quote+ # (standard_conforming_strings is on).+ escapes = i > 0 and sql[i - 1] in "Ee" and (+ i == 1 or not (sql[i - 2].isalnum() or sql[i - 2] == "_")+ )+ j = i + 1+ while j < n:+ if escapes and sql[j] == "\\":+ j += 2+ continue+ if sql[j] == "'":+ if j + 1 < n and sql[j + 1] == "'":+ j += 2+ continue+ return j + 1+ j += 1+ return n+ if ch == '"':+ j = i + 1+ while j < n:+ if sql[j] == '"':+ if j + 1 < n and sql[j + 1] == '"':+ j += 2+ continue+ return j + 1+ j += 1+ return n+ if ch == "$":+ m = _DOLLAR_TAG_RE.match(sql, i)+ if m:+ tag = m.group(0)+ close = sql.find(tag, m.end())+ return close + len(tag) if close >= 0 else n+ return i+ if sql.startswith("--", i):+ nl = sql.find("\n", i)+ return n if nl < 0 else nl + 1+ if sql.startswith("/*", i):+ depth, j = 0, i+ while j < n:+ if sql.startswith("/*", j):+ depth += 1+ j += 2+ elif sql.startswith("*/", j):+ depth -= 1+ j += 2+ if depth == 0:+ return j+ else:+ j += 1+ return n+ return i+++def strip_leading_comments(sql):+ """Drop leading whitespace and comments.++ Every validation rule anchors on the first keyword, so a leading ``--`` or+ ``/* */`` comment would otherwise disable the validator wholesale. Migration+ tools routinely prefix their SQL with comments.+ """+ i, n = 0, len(sql)+ while i < n:+ if sql[i].isspace():+ i += 1+ continue+ j = _skip_noise(sql, i) if sql.startswith(("--", "/*"), i) else i+ if j == i:+ break+ i = j+ return sql[i:]+++def _match_paren(sql, start):+ """Index of the ``)`` matching the ``(`` at ``start``, or -1."""+ depth, i, n = 0, start, len(sql)+ while i < n:+ j = _skip_noise(sql, i)+ if j != i:+ i = j+ continue+ if sql[i] == "(":+ depth += 1+ elif sql[i] == ")":+ depth -= 1+ if depth == 0:+ return i+ i += 1+ return -1+++def _find_char(sql, i, chars):+ """Index of the next character in ``chars`` outside literals/comments."""+ n = len(sql)+ while i < n:+ j = _skip_noise(sql, i)+ if j != i:+ i = j+ continue+ if sql[i] in chars:+ return i+ i += 1+ return -1+++# ---------------------------------------------------------------------------+# DSQL validator (pure, unit-testable)+# ---------------------------------------------------------------------------+++class DsqlError:+ """Validator rejection: rendered as a PG ErrorResponse."""++ def __init__(self, sqlstate, message):+ self.sqlstate = sqlstate+ self.message = message++ def __repr__(self):+ return f"DsqlError({self.sqlstate!r}, {self.message!r})"+++class Rewrite:+ """Statement must be rewritten before forwarding to the backend."""++ def __init__(self, sql, kind, object_name="", table="", job_type="INDEX_BUILD"):+ self.sql = sql+ self.kind = kind+ self.object_name = object_name+ self.table = table+ self.job_type = job_type++ def __repr__(self):+ return f"Rewrite({self.sql!r}, kind={self.kind!r})"+++class TxnState:+ """Per-connection transaction discipline tracking.++ ``in_txn`` is authoritative from the backend's ReadyForQuery status byte+ (see ``note_backend_status``) — inferring it from statement text alone goes+ wrong the moment a client sends BEGIN or COMMIT over the extended protocol,+ and a stranded ``in_txn`` makes the validator reject perfectly valid+ autocommit statements. The per-transaction counters below still come from+ statement text, since the wire protocol does not report them.+ """++ def __init__(self):+ self.in_txn = False+ self.ddl_seen = False+ self.dml_seen = False+ self.started_at = None # monotonic ts at BEGIN (duration limit)+ self.rows_est = 0 # static row estimate (VALUES tuples) in this txn+ self.bytes_est = 0 # cumulative DML payload bytes in this txn+ self.aborted = False # backend reported a failed transaction block+ self.synthetic_abort = False # *we* rejected a statement mid-transaction++ def copy(self):+ other = TxnState()+ other.in_txn = self.in_txn+ other.ddl_seen = self.ddl_seen+ other.dml_seen = self.dml_seen+ other.started_at = self.started_at+ other.rows_est = self.rows_est+ other.bytes_est = self.bytes_est+ other.aborted = self.aborted+ other.synthetic_abort = self.synthetic_abort+ return other++ def begin(self):+ self.in_txn = True+ self.ddl_seen = self.dml_seen = False+ self.started_at = time.monotonic()+ self.rows_est = self.bytes_est = 0+ self.aborted = self.synthetic_abort = False++ def reset(self):+ self.in_txn = False+ self.ddl_seen = self.dml_seen = False+ self.started_at = None+ self.rows_est = self.bytes_est = 0+ self.aborted = self.synthetic_abort = False++ def note_backend_status(self, status):+ """Reconcile with a backend ReadyForQuery status byte (I / T / E)."""+ if status == b"I":+ self.reset()+ elif status == b"T":+ if not self.in_txn:+ self.begin()+ self.aborted = False+ elif status == b"E":+ self.in_txn = True+ self.aborted = True++ def apply(self, sql):+ cls = classify_statement(sql)+ if cls == "tcl_begin":+ self.begin()+ elif cls == "tcl_end":+ self.reset()+ elif cls == "ddl":+ self.ddl_seen = True+ elif cls == "dml":+ self.dml_seen = True+ if self.in_txn:+ self.rows_est += _values_tuple_count(sql)+ self.bytes_est += len(sql.encode("utf-8", "replace"))+++_DDL_KEYWORDS = {"CREATE", "ALTER", "DROP"}+_DML_KEYWORDS = {"INSERT", "UPDATE", "DELETE"}+_TCL_BEGIN = {"BEGIN", "START"}+_TCL_END = {"COMMIT", "END", "ROLLBACK", "ABORT"}+++def _cte_main_keyword(sql):+ """Leading keyword of the statement a ``WITH`` clause feeds.++ ``WITH x AS (...) INSERT INTO ...`` is an INSERT, not an unclassified+ statement — otherwise CTE-leading writes escape every DML rule.+ """+ m = re.match(r"\s*WITH\s+(?:RECURSIVE\s+)?", sql, re.I)+ if not m:+ return None+ i = m.end()+ while True:+ opening = _find_char(sql, i, "(")+ if opening < 0:+ return None+ close = _match_paren(sql, opening)+ if close < 0:+ return None+ tail = sql[close + 1:].lstrip()+ if tail.startswith(","):+ i = close + 1+ continue+ km = re.match(r"([A-Za-z]+)", tail)+ if not km:+ return None+ word = km.group(1).upper()+ # Column list or AS [NOT] MATERIALIZED — the body is still ahead.+ if word in ("AS", "NOT", "MATERIALIZED"):+ i = close + 1+ continue+ return word+++def _explain_inner(sql):+ """Statement executed by ``EXPLAIN ANALYZE``, or None.++ Plain EXPLAIN only plans, so it stays unclassified; EXPLAIN ANALYZE really+ does run the underlying DML.+ """+ m = re.match(r"\s*EXPLAIN\s*", sql, re.I)+ if not m:+ return None+ i, analyze = m.end(), False+ if i < len(sql) and sql[i] == "(":+ close = _match_paren(sql, i)+ if close < 0:+ return None+ opts = sql[i + 1:close]+ analyze = bool(+ re.search(r"\bANALYZ?[ES]E?\b(?!\s+(?:false|off|0))", opts, re.I)+ )+ i = close + 1+ else:+ while True:+ wm = re.match(r"\s*(ANALYZE|ANALYSE|VERBOSE)\b", sql[i:], re.I)+ if not wm:+ break+ if wm.group(1).upper() in ("ANALYZE", "ANALYSE"):+ analyze = True+ i += wm.end()+ return sql[i:] if analyze else None+++def classify_statement(sql):+ """Classify one statement: ddl | dml | tcl_begin | tcl_end | other."""+ s = strip_leading_comments(sql)+ m = re.match(r"\s*([A-Za-z]+)", s)+ if not m:+ return "other"+ kw = m.group(1).upper()+ if kw == "WITH":+ kw = _cte_main_keyword(s) or kw+ elif kw == "EXPLAIN":+ inner = _explain_inner(s)+ return classify_statement(inner) if inner is not None else "other"+ if kw in _DDL_KEYWORDS:+ return "ddl"+ if kw in _DML_KEYWORDS:+ return "dml"+ if kw in _TCL_BEGIN:+ return "tcl_begin"+ if kw in _TCL_END:+ return "tcl_end"+ return "other"+++def split_statements(sql):+ """Split a ``Q`` batch on ';' outside literals, comments and quoted names."""+ out, last, i, n = [], 0, 0, len(sql)+ while i < n:+ j = _skip_noise(sql, i)+ if j != i:+ i = j+ continue+ if sql[i] == ";":+ out.append(sql[last:i])+ last = i + 1+ i += 1+ out.append(sql[last:])+ return [s.strip() for s in out if s.strip()]+++def _values_tuple_count(sql):+ """Count row tuples in an INSERT ... VALUES (...), (...), ... statement.++ Static estimate for the 3,000-rows-per-transaction limit; INSERT ...+ SELECT / UPDATE / DELETE row counts can't be known without executing.+ """+ m = re.search(r"\bVALUES\b", sql, re.I)+ if not m or not re.match(r"\s*INSERT\b", sql, re.I):+ return 0+ start = sql.find("(", m.end())+ if start < 0:+ return 0+ count, depth = 0, 0+ in_str = False+ for i in range(start, len(sql)):+ ch = sql[i]+ if ch == "'":+ in_str = not in_str+ elif in_str:+ continue+ elif ch == "(":+ if depth == 0:+ count += 1+ depth += 1+ elif ch == ")":+ depth -= 1+ if depth == 0 and i + 1 < len(sql) and sql[i + 1:].lstrip()[:1] not in (",",):+ break+ return count+++# --- Denylist (rule 1) ------------------------------------------------------++_DENYLIST = (+ (r"CREATE\s+DATABASE\b", "CREATE DATABASE"),+ (r"CREATE\s+TYPE\b", "CREATE TYPE"),+ (r"CREATE\s+EXTENSION\b", "CREATE EXTENSION"),+ (r"CREATE\s+TRIGGER\b", "CREATE TRIGGER"),+ (r"CREATE\s+(OR\s+REPLACE\s+)?PROCEDURE\b", "CREATE PROCEDURE"),+ (r"CREATE\s+TABLESPACE\b", "CREATE TABLESPACE"),+ (r"CREATE\s+MATERIALIZED\s+VIEW\b", "materialized views"),+ (r"CREATE\s+(TEMP|TEMPORARY|UNLOGGED)\s+TABLE\b", "temporary/unlogged tables"),+ (r"COPY\b", "COPY"),+ (r"LISTEN\b", "LISTEN"),+ (r"NOTIFY\b", "NOTIFY"),+ (r"UNLISTEN\b", "UNLISTEN"),+ (r"TRUNCATE\b", "TRUNCATE"),+ (r"DO\b", "DO"),+ (r"SAVEPOINT\b", "SAVEPOINT"),+ (r"LOCK\b", "LOCK TABLE"),+ (r"CREATE\s+RULE\b", "CREATE RULE"),+ (r"CREATE\s+(OR\s+REPLACE\s+)?AGGREGATE\b", "CREATE AGGREGATE"),+ (r"CREATE\s+DOMAIN\b", "CREATE DOMAIN"),+ (r"CREATE\s+CAST\b", "CREATE CAST"),+ (r"CREATE\s+COLLATION\b", "CREATE COLLATION"),+ (r"CREATE\s+(OR\s+REPLACE\s+)?OPERATOR\b", "CREATE OPERATOR"),+ (r"CREATE\s+PUBLICATION\b", "CREATE PUBLICATION"),+ (r"CREATE\s+SUBSCRIPTION\b", "CREATE SUBSCRIPTION"),+ (r"CREATE\s+(FOREIGN\s+TABLE|SERVER|FOREIGN\s+DATA\s+WRAPPER)\b",+ "foreign data wrappers"),+ (r"VACUUM\b", "VACUUM"),+ (r"CLUSTER\b", "CLUSTER"),+ (r"REINDEX\b", "REINDEX"),+ (r"ALTER\s+SYSTEM\b", "ALTER SYSTEM"),+ (r"PREPARE\s+TRANSACTION\b", "prepared transactions"),+ (r"(COMMIT|ROLLBACK)\s+PREPARED\b", "prepared transactions"),+)+++def _check_denylist(sql):+ for pattern, thing in _DENYLIST:+ if re.match(r"\s*" + pattern, sql, re.I):+ return DsqlError("0A000", f"{thing} is not supported")+ m = re.match(r"\s*CREATE\s+(OR\s+REPLACE\s+)?FUNCTION\b", sql, re.I)+ if m:+ lang = re.search(r"\bLANGUAGE\s+['\"]?([A-Za-z0-9_]+)", sql, re.I)+ if lang and lang.group(1).lower() != "sql":+ return DsqlError("0A000", f"LANGUAGE {lang.group(1)} is not supported")+ if re.match(r"\s*(CREATE|ALTER)\s+TABLE\b", sql, re.I) or re.match(+ r"\s*CREATE\s+(TEMP|TEMPORARY|UNLOGGED)\s+TABLE\b", sql, re.I+ ):+ if re.search(r"\bFOREIGN\s+KEY\b|\bREFERENCES\b", sql, re.I):+ return DsqlError("0A000", "foreign key constraints are not supported")+ if re.search(r"\bPARTITION\s+BY\b|\bPARTITION\s+OF\b|\bINHERITS\b", sql, re.I):+ return DsqlError("0A000", "table partitioning is not supported")+ if re.search(r"\bEXCLUDE\b", sql, re.I):+ return DsqlError("0A000", "the EXCLUDE constraint is not supported")+ if re.match(r"\s*ALTER\s+TABLE\b", sql, re.I):+ add = re.search(r"\bADD\s+COLUMN\b", sql, re.I)+ # STORAGE DEFAULT is a storage option and BY DEFAULT an identity+ # option — neither is an inline column default. Generated columns+ # (GENERATED ALWAYS AS ... STORED / identity) are allowed.+ if add and re.search(+ r"(?<!STORAGE )(?<!BY )\bDEFAULT\b|\bNOT\s+NULL\b", sql[add.end():], re.I+ ):+ return DsqlError(+ "0A000",+ "ALTER TABLE ADD COLUMN with DEFAULT or NOT NULL is not supported",+ )+ return None+++# --- Column types (rule 2) --------------------------------------------------++# Normalized allowlist of DSQL-supported column types.+_TYPE_ALIASES = {+ "smallint": "smallint", "int2": "smallint",+ "integer": "integer", "int": "integer", "int4": "integer",+ "bigint": "bigint", "int8": "bigint",+ "real": "real", "float4": "real",+ "double precision": "double precision", "float8": "double precision",+ "numeric": "numeric", "decimal": "numeric", "dec": "numeric",+ "character": "character", "char": "character", "bpchar": "character",+ "character varying": "character varying", "varchar": "character varying",+ "text": "text",+ "date": "date",+ "time": "time", "time without time zone": "time",+ "time with time zone": "time with time zone", "timetz": "time with time zone",+ "timestamp": "timestamp", "timestamp without time zone": "timestamp",+ "timestamp with time zone": "timestamp with time zone",+ "timestamptz": "timestamp with time zone",+ "interval": "interval",+ "boolean": "boolean", "bool": "boolean",+ "bytea": "bytea", "uuid": "uuid", "json": "json", "jsonb": "jsonb",+}++# Keywords that terminate the type portion of a column definition.+_COL_CONSTRAINT_KEYWORDS = {+ "not", "null", "default", "primary", "unique", "check", "references",+ "collate", "generated", "constraint", "storage",+}++# Field-qualified interval types: interval year, interval day to second, ...+_INTERVAL_RE = re.compile(+ r"interval(\s+(year|month|day|hour|minute|second)"+ r"(\s+to\s+(year|month|day|hour|minute|second))?)?$"+)++# Table-level constraint starters inside CREATE TABLE (...).+_TABLE_CONSTRAINT_STARTERS = {+ "primary", "unique", "check", "constraint", "foreign", "like", "exclude",+}+++def _split_top_level(body):+ """Split a parenthesised body on commas at paren depth 0."""+ parts, depth, last, i, n = [], 0, 0, 0, len(body)+ while i < n:+ j = _skip_noise(body, i)+ if j != i:+ i = j+ continue+ ch = body[i]+ if ch == "(":+ depth += 1+ elif ch == ")":+ depth -= 1+ elif ch == "," and depth == 0:+ parts.append(body[last:i])+ last = i + 1+ i += 1+ if body[last:]:+ parts.append(body[last:])+ return parts+++def _normalize_type(raw):+ """Return (normalized, display) for a raw type string, or None."""+ raw = raw.strip()+ if not raw:+ return None+ is_array = False+ while raw.rstrip().endswith("[]"):+ is_array = True+ raw = raw.rstrip()[:-2]+ display = re.sub(r"\s+", " ", raw.strip())+ # strip precision/scale and trailing modifiers: numeric(18,6), varchar(10)+ base = re.sub(r"\(.*?\)", "", display)+ base = re.sub(r"\s+", " ", base).strip().lower()+ if is_array:+ return None, display + "[]"+ if _INTERVAL_RE.match(base):+ return "interval", display+ normalized = _TYPE_ALIASES.get(base)+ return (normalized, display) if normalized else (None, display)+++def _type_from_tokens(tokens):+ """Given column-def tokens after the column name, extract the type."""+ type_tokens = []+ for tok in tokens:+ word = re.sub(r"\(.*", "", tok).lower()+ if word in _COL_CONSTRAINT_KEYWORDS:+ break+ type_tokens.append(tok)+ return " ".join(type_tokens)+++def _check_type(raw_type):+ """Return a DsqlError if the raw type string is not DSQL-supported."""+ result = _normalize_type(raw_type)+ if result is None:+ return None+ normalized, display = result+ if normalized is None:+ return DsqlError("0A000", f'type "{display}" is not supported')+ return None+++def _check_identity(col_def, raw_type):+ """DSQL supports identity columns on bigint columns only."""+ if re.search(r"\bGENERATED\b", col_def, re.I) and re.search(+ r"\bAS\s+IDENTITY\b", col_def, re.I+ ):+ result = _normalize_type(raw_type)+ if result and result[0] is not None and result[0] != "bigint":+ return DsqlError(+ "0A000", "identity columns are only supported on bigint columns"+ )+ return None+++def _check_column_types(sql):+ m = re.match(+ r"\s*CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?[\w\".]+\s*", sql, re.I+ )+ if m:+ rest = sql[m.end():]+ if not rest.lstrip().startswith("("):+ return None # CREATE TABLE ... AS / PARTITION OF / LIKE-only+ start = sql.index("(", m.end())+ end = _match_paren(sql, start)+ if end < 0:+ return None+ for part in _split_top_level(sql[start + 1 : end]):+ tokens = part.strip().split()+ if not tokens:+ continue+ if tokens[0].strip('"').lower() in _TABLE_CONSTRAINT_STARTERS:+ continue+ raw_type = _type_from_tokens(tokens[1:])+ err = _check_type(raw_type)+ if err:+ return err+ err = _check_identity(part, raw_type)+ if err:+ return err+ return None+ m = re.match(r"\s*ALTER\s+TABLE\b", sql, re.I)+ if m:+ add = re.search(r"\bADD\s+COLUMN\s+(?:IF\s+NOT\s+EXISTS\s+)?([\w\"]+)\s+", sql, re.I)+ if add:+ tail = sql[add.end():]+ tokens = tail.split()+ raw_type = _type_from_tokens(tokens)+ err = _check_type(raw_type)+ if err:+ return err+ err = _check_identity(tail, raw_type)+ if err:+ return err+ return None+++# --- ALTER TABLE subset (AWS alter-table-syntax-support) ---------------------++# DSQL supports only a subset of ALTER TABLE actions: ADD COLUMN (no inline+# DEFAULT/NOT NULL — handled in _check_denylist), DROP COLUMN (one or more+# columns, but not a primary key column — needs a catalog probe, so it lives in+# _check_drop_column rather than here), SET/DROP+# DEFAULT, DROP NOT NULL, DROP EXPRESSION, identity actions, SET STORAGE,+# ADD CONSTRAINT ... CHECK ... NOT VALID, ADD CONSTRAINT ... UNIQUE USING+# INDEX, DROP CONSTRAINT, RENAME, SET SCHEMA, OWNER TO, and the async+# VALIDATE CONSTRAINT form (handled as a Rewrite in validate()).+def _check_alter_table(sql):+ if not re.match(r"\s*ALTER\s+TABLE\b", sql, re.I):+ return None+ if re.search(+ r"\bALTER\s+COLUMN\s+[\w\"]+\s+(?:SET\s+DATA\s+)?TYPE\b", sql, re.I+ ):+ return DsqlError("0A000", "ALTER COLUMN TYPE is not supported")+ if re.search(r"\bSET\s+NOT\s+NULL\b", sql, re.I):+ return DsqlError(+ "0A000", "SET NOT NULL is not supported (only DROP NOT NULL)"+ )+ add = re.search(r"\bADD\s+(?:CONSTRAINT\s+[\w\"]+\s+)?(\w+)", sql, re.I)+ if add and add.group(1).upper() != "COLUMN":+ what = add.group(1).upper()+ if what == "PRIMARY":+ return DsqlError(+ "0A000", "ADD CONSTRAINT PRIMARY KEY is not supported"+ )+ if what == "UNIQUE" and not re.search(+ r"\bUNIQUE\s+USING\s+INDEX\b", sql, re.I+ ):+ return DsqlError(+ "0A000",+ "ADD CONSTRAINT UNIQUE (...) is not supported — use "+ "CREATE UNIQUE INDEX ASYNC, then ADD CONSTRAINT ... "+ "UNIQUE USING INDEX",+ )+ if what == "CHECK" and not re.search(r"\bNOT\s+VALID\b", sql, re.I):+ return DsqlError(+ "0A000",+ "CHECK constraints added via ALTER TABLE must use NOT VALID",+ )+ # ADD GENERATED ... AS IDENTITY requires an explicit CACHE value.+ if re.search(r"\bADD\s+GENERATED\b", sql, re.I) and re.search(+ r"\bIDENTITY\b", sql, re.I+ ):+ if not re.search(r"\bCACHE\s+\d+", sql, re.I):+ return DsqlError(+ "0A000",+ "ADD GENERATED AS IDENTITY requires an explicit CACHE value",+ )+ return None+++# IDENTITY columns and sequences: CACHE must be 1 or >= 65536.+def _check_cache_value(sql):+ if not re.match(r"\s*(CREATE|ALTER)\s+(TABLE|SEQUENCE)\b", sql, re.I):+ return None+ m = re.search(r"\bCACHE\s+(\d+)", sql, re.I)+ if m:+ n = int(m.group(1))+ if n != 1 and n < 65536:+ return DsqlError("0A000", "CACHE must be 1 or at least 65536")+ return None+++# DSQL index rules: btree only (no USING gin/gist/brin/hash/spgist), no+# CONCURRENTLY (use ASYNC), no partial (WHERE) or expression indexes, and at+# most 8 columns per index (24 indexes per table is enforced at runtime).+def _check_index_rules(sql):+ m = _INDEX_RE.match(sql) or _INDEX_ASYNC_RE.match(sql)+ if not m:+ return None+ using = re.search(r"\bUSING\s+(\w+)", sql, re.I)+ if using and using.group(1).lower() not in ("btree", "btree_index"):+ return DsqlError(+ "0A000", f"USING {using.group(1)} is not supported (btree only)"+ )+ if re.search(r"\bCONCURRENTLY\b", sql, re.I):+ return DsqlError(+ "0A000", "CREATE INDEX CONCURRENTLY is not supported — use CREATE INDEX ASYNC"+ )+ if re.search(r"\bWHERE\b", sql, re.I):+ return DsqlError("0A000", "partial indexes (WHERE) are not supported")+ start = sql.find("(", m.end())+ if start < 0:+ return None+ end = _match_paren(sql, start)+ if end < 0:+ return None+ parts = _split_top_level(sql[start + 1 : end])+ if any("(" in part for part in parts):+ return DsqlError("0A000", "expression indexes are not supported")+ if len(parts) > 8:+ return DsqlError(+ "0A000", f"indexes cannot have more than 8 columns ({len(parts)} given)"+ )+ return None+++_ALTER_ASYNC_VALIDATE_RE = re.compile(+ r"\s*ALTER\s+TABLE\s+ASYNC\s+(?:IF\s+EXISTS\s+)?(?:ONLY\s+)?"+ r"([\w\".]+)\s*\*?\s*VALIDATE\s+CONSTRAINT\s+([\w\"]+)",+ re.I,+)+++def rewrite_alter_async(sql):+ """Strip the DSQL-only ASYNC keyword from ALTER TABLE ASYNC."""+ return re.sub(r"(ALTER\s+TABLE)\s+ASYNC", r"\1", sql, count=1, flags=re.I)+++# --- CREATE INDEX ASYNC (rules 3 & 4) ---------------------------------------++_INDEX_ASYNC_RE = re.compile(+ r"\s*CREATE\s+(?:UNIQUE\s+)?INDEX\s+ASYNC\s+(?:IF\s+NOT\s+EXISTS\s+)?"+ r"(?:([\w\".]+)\s+)?ON\s+(?:ONLY\s+)?(?:USING\s+\w+\s+)?([\w\".]+)",+ re.I,+)++_INDEX_RE = re.compile(+ r"\s*CREATE\s+(?:UNIQUE\s+)?INDEX\s+(?:CONCURRENTLY\s+)?"+ r"(?:IF\s+NOT\s+EXISTS\s+)?(?:([\w\".]+)\s+)?ON\s+(?:ONLY\s+)?"+ r"(?:USING\s+\w+\s+)?([\w\".]+)",+ re.I,+)+++def rewrite_index_async(sql):+ """Strip the DSQL-only ASYNC keyword from CREATE [UNIQUE] INDEX ASYNC."""+ return re.sub(+ r"(CREATE\s+(?:UNIQUE\s+)?INDEX)\s+ASYNC", r"\1", sql, count=1, flags=re.I+ )+++def is_plain_create_index(sql):+ return bool(_INDEX_RE.match(sql)) and not _INDEX_ASYNC_RE.match(sql)+++def create_index_table(sql):+ m = _INDEX_RE.match(sql) or _INDEX_ASYNC_RE.match(sql)+ return m.group(2) if m else None+++def _index_object_name(sql, on_end, index_name, table):+ """Schema-qualified index name for the sys.jobs object_name field.++ Unnamed indexes get DSQL's documented auto-name ``<table>_<col>_..._idx``+ in the table's schema (default ``public``).+ """+ schema, _, bare_table = table.rpartition(".")+ schema = schema.strip('"') or "public"+ bare_table = bare_table.strip('"')+ if index_name:+ if "." in index_name:+ return index_name+ return f"{schema}.{index_name.strip(chr(34))}"+ cols = []+ start = sql.find("(", on_end)+ if start >= 0:+ end = _match_paren(sql, start)+ if end > 0:+ for part in _split_top_level(sql[start + 1 : end]):+ tokens = part.strip().split()+ if tokens:+ ident = re.match(r'[A-Za-z_"][\w"]*', tokens[0])+ if ident:+ cols.append(ident.group(0).strip('"'))+ suffix = "_" + "_".join(cols) if cols else ""+ return f"{schema}.{bare_table}{suffix}_idx"+++# --- sys.jobs / sys.wait_for_job (rules 5 & 6) ------------------------------++_SYS_JOBS_RE = re.compile(r"\s*SELECT\s+(.+?)\s+FROM\s+sys\.jobs\b", re.I | re.S)+_JOB_ID_FILTER_RE = re.compile(r"job_id\s*=\s*'([^']*)'", re.I)+_WAIT_JOB_RE = re.compile(+ r"\s*(?:SELECT|CALL)\s+sys\.wait_for_job\s*\(\s*"+ r"(?:job_id\s*\)\s*'([^']*)'|'([^']*)'\s*\))",+ re.I,+)+++def match_sys_jobs(sql):+ """Return (columns, job_id_filter) if this queries sys.jobs, else False.++ ``columns`` is None for ``SELECT *``; ``job_id_filter`` is None when+ there is no ``WHERE job_id = '<id>'`` clause.+ """+ m = _SYS_JOBS_RE.match(sql)+ if not m:+ return False+ select_list = m.group(1).strip()+ columns = None+ if select_list != "*":+ columns = [c.strip().strip('"') for c in select_list.split(",")]+ f = _JOB_ID_FILTER_RE.search(sql)+ return columns, (f.group(1) if f else None)+++def match_wait_for_job(sql):+ """Match both documented call forms: ``sys.wait_for_job(job_id) '<id>'``+ and the conventional ``sys.wait_for_job('<id>')``."""+ m = _WAIT_JOB_RE.match(sql)+ if not m:+ return None+ return m.group(1) or m.group(2)+++# --- Top-level validator -----------------------------------------------------+++def validate(sql, txn_state=None):+ """Validate one statement. Returns DsqlError | Rewrite | None.++ Read-only with respect to ``txn_state`` — the caller applies state+ changes (``TxnState.apply``) once the statement is actually forwarded.+ """+ if not STRICT:+ return None+ s = strip_leading_comments(sql).strip()+ if not s:+ return None++ if txn_state is not None and txn_state.in_txn:+ cls = classify_statement(s)+ if cls == "ddl":+ if txn_state.ddl_seen:+ return DsqlError(+ "25006", "only one DDL statement can be run in a transaction"+ )+ if txn_state.dml_seen:+ return DsqlError(+ "25006", "DDL and DML statements cannot be mixed in a transaction"+ )+ elif cls == "dml" and txn_state.ddl_seen:+ return DsqlError(+ "25006", "DDL and DML statements cannot be mixed in a transaction"+ )+ # Documented DSQL transaction limits (messages are approximations).+ if (+ txn_state.started_at is not None+ and time.monotonic() - txn_state.started_at > 300+ ):+ return DsqlError(+ "25006", "transaction exceeded the 5 minute duration limit"+ )++ # Row/size limits apply to any transaction, including an implicit+ # single-statement one. Row counting is static (VALUES tuples) —+ # INSERT ... SELECT and UPDATE/DELETE affected-row counts are not+ # enforced.+ if txn_state is not None and classify_statement(s) == "dml":+ rows_est = txn_state.rows_est if txn_state.in_txn else 0+ bytes_est = txn_state.bytes_est if txn_state.in_txn else 0+ if rows_est + _values_tuple_count(s) > 3000:+ return DsqlError(+ "25006", "transaction exceeds the 3,000 row limit"+ )+ if bytes_est + len(s.encode("utf-8", "replace")) > 10 * 1024 * 1024:+ return DsqlError(+ "25006", "transaction exceeds the 10 MiB size limit"+ )++ err = _check_denylist(s)+ if err:+ return err+ err = _check_alter_table(s)+ if err:+ return err+ err = _check_cache_value(s)+ if err:+ return err+ err = _check_column_types(s)+ if err:+ return err+ err = _check_index_rules(s)+ if err:+ return err++ m = _ALTER_ASYNC_VALIDATE_RE.match(s)+ if m:+ table = m.group(1)+ schema, _, bare = table.rpartition(".")+ object_name = (schema.strip('"') or "public") + "." + bare.strip('"')+ return Rewrite(+ rewrite_alter_async(s),+ kind="index_async",+ object_name=object_name,+ table=table,+ job_type="VALIDATE_CONSTRAINT",+ )++ m = _INDEX_ASYNC_RE.match(s)+ if m:+ index_name, table = m.group(1), m.group(2)+ return Rewrite(+ rewrite_index_async(s),+ kind="index_async",+ object_name=_index_object_name(s, m.end(), index_name, table),+ table=table,+ )+ return None+++# ---------------------------------------------------------------------------+# Proxy connection handling+# ---------------------------------------------------------------------------+++class _Conn:+ def __init__(self, cluster_id):+ self.cluster_id = cluster_id+ self.txn = TxnState()+ self.capture = None # asyncio.Queue while swallowing backend frames+ self.catalog_version = None # last catalog version this conn has seen+ # Extended-protocol state, reset at each Sync.+ self.ext_skip = False # dropping messages until Sync after an error+ self.ext_forwarded = False # any message reached the backend this cycle+ self.ext_stmts = {} # prepared statement name -> {"sql", "synth"}+ self.ext_portals = {} # portal name -> the same entry+++async def _read_startup(reader):+ (length,) = struct.unpack("!I", await reader.readexactly(4))+ body = await reader.readexactly(length - 4)+ (code,) = struct.unpack("!I", body[:4])+ return code, body[4:]+++def _parse_startup_params(payload):+ parts = payload.split(b"\0")+ params = {}+ for i in range(0, len(parts) - 1, 2):+ if parts[i]:+ params[parts[i].decode("utf-8", "replace")] = parts[i + 1].decode(+ "utf-8", "replace"+ )+ return params+++async def _connect_backend(host, port, database, attempts=20, delay=0.5):+ """Open a backend connection with trust auth and complete its startup.++ Retries briefly: docker-proxy accepts (then drops) connections while the+ postgres entrypoint is still initialising, and containers can bounce.+ """+ last_err = None+ for _ in range(attempts):+ try:+ return await _connect_backend_once(host, port, database)+ except Exception as e:+ last_err = e+ await asyncio.sleep(delay)+ raise RuntimeError(f"backend {host}:{port} not reachable: {last_err}")+++async def _connect_backend_once(host, port, database):+ reader, writer = await asyncio.open_connection(host, port)+ params = b"user\0postgres\0database\0" + database.encode() + b"\0\0"+ payload = struct.pack("!I", _PROTOCOL_3) + params+ writer.write(struct.pack("!I", len(payload) + 4) + payload)+ await writer.drain()+ try:+ while True:+ type_byte, payload = await _read_frame(reader)+ if type_byte == b"E":+ raise RuntimeError(f"backend startup failed: {payload!r}")+ if type_byte == b"R":+ (code,) = struct.unpack("!I", payload[:4])+ if code != 0:+ raise RuntimeError(f"backend requested unsupported auth method {code}")+ elif type_byte == b"Z":+ return reader, writer+ except Exception:+ writer.close()+ raise+++async def _backend_relay(conn, b_reader, c_writer):+ """Forward backend frames to the client (or to the capture queue).++ ReadyForQuery carries the backend's own view of the transaction block, so+ it is the authoritative source for ``in_txn`` regardless of which protocol+ the client used to open or close the transaction.+ """+ while True:+ type_byte, payload = await _read_frame(b_reader)+ if type_byte == b"Z" and payload:+ conn.txn.note_backend_status(payload[:1])+ if conn.capture is not None:+ conn.capture.put_nowait((type_byte, payload))+ else:+ c_writer.write(_frame(type_byte, payload))+ await c_writer.drain()+++async def _run_backend_capture(conn, b_writer, sql):+ """Send a query to the backend and swallow its frames until ReadyForQuery."""+ queue = asyncio.Queue()+ conn.capture = queue+ try:+ b_writer.write(_frame(b"Q", sql.encode() + b"\0"))+ await b_writer.drain()+ frames = []+ while True:+ type_byte, payload = await queue.get()+ frames.append((type_byte, payload))+ if type_byte == b"Z":+ return frames+ finally:+ conn.capture = None+++async def _table_has_rows(conn, b_writer, table):+ """Probe SELECT 1 FROM <table> LIMIT 1 on the client's backend connection."""+ try:+ frames = await _run_backend_capture(+ conn, b_writer, f"SELECT 1 FROM {table} LIMIT 1"+ )+ except Exception:+ return False+ for type_byte, _ in frames:+ if type_byte == b"E":+ return False # probe failed — forward original; backend errors+ if type_byte == b"D":+ return True+ return False+++async def _index_count(conn, b_writer, table):+ """Count the table's existing indexes via pg_indexes. None on probe error."""+ schema, _, bare = table.rpartition(".")+ schema = schema.strip('"') or "public"+ bare = bare.strip('"')+ try:+ frames = await _run_backend_capture(+ conn,+ b_writer,+ "SELECT count(*) FROM pg_indexes "+ f"WHERE schemaname = '{schema}' AND tablename = '{bare}'",+ )+ except Exception:+ return None+ for type_byte, payload in frames:+ if type_byte == b"E":+ return None+ if type_byte == b"D":+ try:+ (nfields,) = struct.unpack("!H", payload[:2])+ if nfields != 1:+ return None+ (vlen,) = struct.unpack("!i", payload[2:6])+ return int(payload[6 : 6 + vlen])+ except Exception:+ return None+ return None+++def _status(conn):+ """ReadyForQuery status byte for the client's current transaction block."""+ if conn.txn.aborted or conn.txn.synthetic_abort:+ return b"E"+ return b"T" if conn.txn.in_txn else b"I"+++def _reject(conn, c_writer, err):+ """Send an ErrorResponse for a statement the backend never saw.++ Inside an explicit transaction this has to poison the block the way a real+ error would: Postgres (and DSQL) refuse every later statement with 25P02+ until the transaction ends. The backend is still in a clean transaction, so+ the proxy tracks the abort itself.+ """+ if conn.txn.in_txn:+ conn.txn.synthetic_abort = True+ c_writer.write(_error_response(err.sqlstate, err.message) + _ready(_status(conn)))+++_ABORTED_ERR = DsqlError(+ "25P02",+ "current transaction is aborted, commands ignored until end of "+ "transaction block",+)+++def _abort_gate(conn, sql):+ """After a synthetic abort, only the statement that ends the block runs.++ Returns (error, forward_sql). ``forward_sql`` replaces COMMIT with ROLLBACK+ so the aborted work is discarded and the client sees the ``ROLLBACK``+ command tag, exactly as Postgres reports a committed-but-failed block.+ """+ if not conn.txn.synthetic_abort:+ return None, sql+ if classify_statement(sql) != "tcl_end":+ return _ABORTED_ERR, sql+ conn.txn.synthetic_abort = False+ if re.match(r"\s*(COMMIT|END)\b", strip_leading_comments(sql), re.I):+ return None, "ROLLBACK"+ return None, sql+++async def _primary_key_columns(conn, b_writer, table):+ """Primary key column names for ``table``; None if the probe failed."""+ try:+ frames = await _run_backend_capture(+ conn,+ b_writer,+ "SELECT a.attname FROM pg_index i JOIN pg_attribute a "+ "ON a.attrelid = i.indrelid AND a.attnum = ANY(i.indkey) "+ f"WHERE i.indrelid = '{table}'::regclass AND i.indisprimary",+ )+ except Exception:+ return None+ cols = []+ for type_byte, payload in frames:+ if type_byte == b"E":+ return None+ if type_byte == b"D":+ (nfields,) = struct.unpack("!H", payload[:2])+ if nfields == 1:+ (vlen,) = struct.unpack("!i", payload[2:6])+ cols.append(payload[6 : 6 + vlen].decode("utf-8", "replace"))+ return cols+++# ALTER TABLE ... DROP COLUMN is supported (multiple columns in one statement+# are fine), but dropping a primary key column is not. The COLUMN keyword is+# optional in Postgres, so the action keywords that merely start with DROP —+# DROP CONSTRAINT / DEFAULT / NOT NULL / EXPRESSION / IDENTITY — have to be+# told apart from a bare column name.+_DROP_COLUMN_RE = re.compile(+ r"\bDROP\s+(?:COLUMN\s+)?(?:IF\s+EXISTS\s+)?([\w\".]+)", re.I+)+_DROP_NON_COLUMN_ACTIONS = frozenset(+ ("constraint", "default", "not", "expression", "identity")+)+_ALTER_TABLE_TARGET_RE = re.compile(+ r"\s*ALTER\s+TABLE\s+(?:ASYNC\s+)?(?:IF\s+EXISTS\s+)?(?:ONLY\s+)?([\w\".]+)", re.I+)+++def dropped_columns(sql):+ """Column names an ALTER TABLE statement drops (may be several)."""+ if not re.match(r"\s*ALTER\s+TABLE\b", sql, re.I):+ return []+ cols = []+ for m in _DROP_COLUMN_RE.finditer(sql):+ raw = m.group(1)+ quoted = raw.startswith('"')+ name = raw.strip('"')+ if not quoted and name.lower() in _DROP_NON_COLUMN_ACTIONS:+ continue+ cols.append(name)+ return cols+++async def _check_drop_column(conn, b_writer, sql):+ """Refuse to drop a primary key column (AWS DSQL, 2026-08-03)."""+ cols = dropped_columns(sql)+ if not cols:+ return None+ m = _ALTER_TABLE_TARGET_RE.match(sql)+ if not m:+ return None+ pk_cols = await _primary_key_columns(conn, b_writer, m.group(1).strip('"'))+ if not pk_cols:+ return None # probe failed or no primary key — let the backend answer+ lowered = {c.lower() for c in pk_cols}+ for col in cols:+ if col.lower() in lowered:+ return DsqlError(+ "0A000",+ f'dropping primary key column "{col}" is not supported',+ )+ return None+++_FOR_UPDATE_MULTI = (+ "locking clause such as FOR UPDATE can be applied on a single table"+)+_FOR_UPDATE_EQ = (+ "locking clause such as FOR UPDATE can be applied only on tables "+ "with equality predicates on the key"+)+_FOR_UPDATE_RE = re.compile(r"\bFOR\s+(?:NO\s+KEY\s+)?UPDATE\b", re.I)+++async def _check_for_update(conn, b_writer, sql):+ """DSQL locking-clause rules (exact messages from the AWS features doc).++ FOR UPDATE needs a single table and equality predicates on all primary+ key columns. Returns a DsqlError, or None to forward.+ """+ body = _FOR_UPDATE_RE.split(sql)[0]+ m = re.search(+ r"\bFROM\b(.+?)(?:\bWHERE\b|\bGROUP\b|\bORDER\b|\bLIMIT\b|$)",+ body, re.I | re.S,+ )+ if not m:+ return DsqlError("0A000", _FOR_UPDATE_EQ)+ from_part = m.group(1)+ if "," in from_part or re.search(r"\bJOIN\b", from_part, re.I):+ return DsqlError("0A000", _FOR_UPDATE_MULTI)+ table = from_part.strip().split()[0].strip('"')+ pk_cols = await _primary_key_columns(conn, b_writer, table)+ if pk_cols is None:+ return None # probe failed — let the backend answer+ wm = re.search(r"\bWHERE\b(.+)$", body, re.I | re.S)+ if not pk_cols or not wm:+ return DsqlError("0A000", _FOR_UPDATE_EQ)+ where = wm.group(1)+ if re.search(+ r"<>|!=|<=|>=|<|>|\bOR\b|\bIN\s*\(|\bBETWEEN\b|\bLIKE\b", where, re.I+ ):+ return DsqlError("0A000", _FOR_UPDATE_EQ)+ for col in pk_cols:+ if not (+ re.search(rf"\b{re.escape(col)}\b\s*=[^=]", where, re.I)+ or re.search(rf"=[^=][^;]*\b{re.escape(col)}\b", where, re.I)+ ):+ return DsqlError("0A000", _FOR_UPDATE_EQ)+ return None+++_ASYNC_BATCH_ERR = DsqlError(+ "0A000", "asynchronous DDL is not supported in multi-statement queries"+)+++async def _plan_statement(conn, s, b_writer, allow_probe=True):+ """Decide what to do with one statement, for either wire protocol.++ Returns ``(kind, payload)`` where kind is one of:+ "error" — payload is a DsqlError to report+ "rows" — payload describes a synthetic result set (sys.jobs etc.)+ "rewrite" — payload is a Rewrite to run on the backend as a job+ "forward" — payload is None; relay the statement untouched++ ``allow_probe`` is False when the backend is mid extended-protocol+ sequence, where injecting a probe query would break the protocol. The+ text-only rules still apply; only the rules that need to look at backend+ state are skipped.+ """+ matched = match_sys_jobs(s)+ if matched is not False:+ columns, job_filter = matched+ cols = list(columns) if columns else list(_JOB_COLUMNS)+ bad = [c for c in cols if c not in _JOB_COLUMNS]+ if bad:+ return "error", DsqlError("42703", f'column "{bad[0]}" does not exist')+ jobs = get_jobs(conn.cluster_id)+ if job_filter:+ jobs = [j for j in jobs if j["job_id"] == job_filter]+ return "rows", {+ "cols": cols,+ "rows": [[job[c] for c in cols] for job in jobs],+ "tag": f"SELECT {len(jobs)}",+ }++ job_id = match_wait_for_job(s)+ if job_id is not None:+ known = any(+ j["job_id"] == job_id and j["status"] == "completed"+ for j in get_jobs(conn.cluster_id)+ )+ return "rows", {+ "cols": [("wait_for_job", 16)], # bool oid+ "rows": [["t" if known else "f"]],+ "tag": "SELECT 1",+ }++ result = validate(s, conn.txn)+ if isinstance(result, DsqlError):+ return "error", result+ if isinstance(result, Rewrite) and result.kind == "index_async":+ if not allow_probe:+ return "error", _ASYNC_BATCH_ERR+ if result.job_type == "INDEX_BUILD" and result.table:+ count = await _index_count(conn, b_writer, result.table)+ if count is not None and count >= 24:+ return "error", DsqlError(+ "0A000",+ f"table {result.table} already has the maximum of 24 indexes",+ )+ return "rewrite", result++ if is_plain_create_index(s):+ table = create_index_table(s)+ if table and allow_probe:+ count = await _index_count(conn, b_writer, table)+ if count is not None and count >= 24:+ return "error", DsqlError(+ "0A000", f"table {table} already has the maximum of 24 indexes"+ )+ if await _table_has_rows(conn, b_writer, table):+ return "error", DsqlError("0A000", "use CREATE INDEX ASYNC instead")+ return "forward", None++ if allow_probe:+ err = await _check_drop_column(conn, b_writer, s)+ if err:+ return "error", err++ if _FOR_UPDATE_RE.search(s) and allow_probe:+ err = await _check_for_update(conn, b_writer, s)+ if err:+ return "error", err+ return "forward", None+++def _note_catalog(conn, s):+ if classify_statement(s) == "ddl":+ conn.catalog_version = _bump_catalog(conn.cluster_id)+ else:+ conn.catalog_version = _catalog_versions.get(conn.cluster_id, 0)+++def _stale_catalog(conn):+ """OC001: this connection's cached catalog is behind another session's DDL."""+ return (+ conn.txn.in_txn+ and conn.catalog_version is not None+ and conn.catalog_version < _catalog_versions.get(conn.cluster_id, 0)+ )+++_OC001_ERR = DsqlError(+ "40001", "schema has been updated by another transaction (OC001)"+)+++async def _rollback_stale(conn, b_writer):+ await _run_backend_capture(conn, b_writer, "ROLLBACK")+ conn.txn.apply("ROLLBACK")+ conn.catalog_version = _catalog_versions.get(conn.cluster_id, 0)+++def _synth_columns(kind, result):+ return result["cols"] if kind == "rows" else ["job_id"]+++async def _run_rewrite(conn, b_writer, rewrite):+ """Execute a rewritten ASYNC statement; return (job, backend_error)."""+ frames = await _run_backend_capture(conn, b_writer, rewrite.sql)+ backend_err = next((p for t, p in frames if t == b"E"), None)+ if backend_err is not None:+ return None, backend_err+ conn.catalog_version = _bump_catalog(conn.cluster_id)+ job = _register_job(conn.cluster_id, rewrite.object_name, rewrite.job_type)+ return job, None+++async def _handle_query(conn, sql, b_writer, c_writer):+ txn = conn.txn+ stmts = split_statements(sql)++ if len(stmts) == 1 and STRICT:+ s = strip_leading_comments(stmts[0])++ # A statement rejected earlier in this transaction poisoned the block.+ err, forward_sql = _abort_gate(conn, s)+ if err:+ c_writer.write(+ _error_response(err.sqlstate, err.message) + _ready(_status(conn))+ )+ await c_writer.drain()+ return+ if forward_sql != s: # COMMIT of an aborted block -> ROLLBACK+ txn.apply(forward_sql)+ b_writer.write(_frame(b"Q", forward_sql.encode() + b"\0"))+ await b_writer.drain()+ return++ # OC001: abort like real DSQL (40001); the next attempt sees the fresh+ # catalog and succeeds.+ if _stale_catalog(conn):+ await _rollback_stale(conn, b_writer)+ _reject(conn, c_writer, _OC001_ERR)+ await c_writer.drain()+ return++ kind, result = await _plan_statement(conn, s, b_writer)+ if kind == "error":+ _reject(conn, c_writer, result)+ await c_writer.drain()+ return+ if kind == "rows":+ out = _row_description(result["cols"])+ for row in result["rows"]:+ out += _data_row(row)+ out += _command_complete(result["tag"])+ c_writer.write(out + _ready(_status(conn)))+ await c_writer.drain()+ return+ if kind == "rewrite":+ txn.apply(s)+ # Run the rewritten statement on the backend, swallow its+ # response, and synthesize the DSQL job_id result set.+ job, backend_err = await _run_rewrite(conn, b_writer, result)+ if backend_err is not None:+ c_writer.write(_frame(b"E", backend_err) + _ready(_status(conn)))+ await c_writer.drain()+ return+ out = _row_description(["job_id"])+ out += _data_row([job["job_id"]])+ out += _command_complete("SELECT 1")+ c_writer.write(out + _ready(_status(conn)))+ await c_writer.drain()+ return++ _note_catalog(conn, s)+ txn.apply(s)+ b_writer.write(_frame(b"Q", sql.encode() + b"\0"))+ await b_writer.drain()+ return++ # Multi-statement batch (implicit transaction) or non-strict mode:+ # validate every statement up front, then forward the batch verbatim.+ if STRICT and stmts:+ # An aborted block only accepts an explicit rollback; forwarding a+ # batch that opened with COMMIT would commit work the client was told+ # had failed.+ if conn.txn.synthetic_abort:+ if re.match(+ r"\s*(ROLLBACK|ABORT)\b", strip_leading_comments(stmts[0]), re.I+ ):+ conn.txn.synthetic_abort = False+ else:+ c_writer.write(+ _error_response(_ABORTED_ERR.sqlstate, _ABORTED_ERR.message)+ + _ready(_status(conn))+ )+ await c_writer.drain()+ return+ sim = txn.copy()+ explicit = False+ if not sim.in_txn and len(stmts) > 1:+ sim.in_txn = True # implicit transaction+ for s in stmts:+ if classify_statement(s) == "tcl_begin":+ explicit = True+ result = validate(s, sim)+ if isinstance(result, (DsqlError, Rewrite)):+ if isinstance(result, Rewrite):+ # ASYNC rewrites only supported for single statements.+ result = DsqlError(+ "0A000",+ "asynchronous DDL is not supported in multi-statement queries",+ )+ _reject(conn, c_writer, result)+ await c_writer.drain()+ return+ sim.apply(s)+ if txn.in_txn or explicit:+ txn.in_txn, txn.ddl_seen, txn.dml_seen = (+ sim.in_txn,+ sim.ddl_seen,+ sim.dml_seen,+ )+ txn.started_at, txn.rows_est, txn.bytes_est = (+ sim.started_at,+ sim.rows_est,+ sim.bytes_est,+ )+ # else: implicit batch committed — connection stays idle.+ if any(classify_statement(st) == "ddl" for st in stmts):+ conn.catalog_version = _bump_catalog(conn.cluster_id)+ else:+ conn.catalog_version = _catalog_versions.get(conn.cluster_id, 0)++ b_writer.write(_frame(b"Q", sql.encode() + b"\0"))+ await b_writer.drain()+++# ---------------------------------------------------------------------------+# Extended query protocol (Parse / Bind / Describe / Execute / Close / Sync)+#+# pgjdbc, pgx, asyncpg, psycopg3 and every ORM built on prepared statements+# send SQL this way. Validating only the simple 'Q' protocol would let all of+# them past the DSQL subset entirely.+# ---------------------------------------------------------------------------+++def _two_strings(payload):+ """The two leading NUL-terminated strings of a Parse or Bind payload."""+ first = payload.index(b"\0")+ second = payload.index(b"\0", first + 1)+ return (+ payload[:first].decode("utf-8", "replace"),+ payload[first + 1 : second].decode("utf-8", "replace"),+ )+++def _target_name(payload):+ """('S'|'P', name) of a Describe or Close payload."""+ return payload[:1], payload[1:].split(b"\0")[0].decode("utf-8", "replace")+++def _parse_payload(name, sql):+ return name.encode() + b"\0" + sql.encode() + b"\0" + struct.pack("!H", 0)+++async def _ext_forward(conn, b_writer, type_byte, payload):+ conn.ext_forwarded = True+ b_writer.write(_frame(type_byte, payload))+ await b_writer.drain()+++async def _ext_error(conn, c_writer, err):+ """Report an error and enter skip-until-Sync, per the protocol spec."""+ if conn.txn.in_txn:+ conn.txn.synthetic_abort = True+ conn.ext_skip = True+ c_writer.write(_error_response(err.sqlstate, err.message))+ await c_writer.drain()+++async def _ext_parse(conn, payload, b_writer, c_writer):+ try:+ name, sql = _two_strings(payload)+ except ValueError:+ await _ext_forward(conn, b_writer, b"P", payload)+ return+ s = strip_leading_comments(sql).strip()+ entry = {"sql": s, "synth": None}+ if not STRICT or not s:+ conn.ext_stmts[name] = entry+ await _ext_forward(conn, b_writer, b"P", payload)+ return++ err, forward_sql = _abort_gate(conn, s)+ if err:+ await _ext_error(conn, c_writer, err)+ return+ if forward_sql != s: # COMMIT of an aborted block -> ROLLBACK+ entry["sql"] = forward_sql+ conn.ext_stmts[name] = entry+ await _ext_forward(conn, b_writer, b"P", _parse_payload(name, forward_sql))+ return++ # Probes and rewrites need the backend to be idle; it is not once anything+ # in this pipelined batch has been forwarded.+ allow_probe = not conn.ext_forwarded+ if allow_probe and _stale_catalog(conn):+ await _rollback_stale(conn, b_writer)+ await _ext_error(conn, c_writer, _OC001_ERR)+ return++ kind, result = await _plan_statement(conn, s, b_writer, allow_probe=allow_probe)+ if kind == "error":+ await _ext_error(conn, c_writer, result)+ return+ if kind in ("rows", "rewrite"):+ entry["synth"] = (kind, result)+ conn.ext_stmts[name] = entry+ c_writer.write(_frame(b"1", b"")) # ParseComplete+ await c_writer.drain()+ return+ conn.ext_stmts[name] = entry+ await _ext_forward(conn, b_writer, b"P", payload)+++async def _ext_bind(conn, payload, b_writer, c_writer):+ try:+ portal, stmt = _two_strings(payload)+ except ValueError:+ await _ext_forward(conn, b_writer, b"B", payload)+ return+ entry = conn.ext_stmts.get(stmt)+ if entry is None:+ await _ext_forward(conn, b_writer, b"B", payload)+ return+ conn.ext_portals[portal] = entry+ if entry["synth"] is None:+ await _ext_forward(conn, b_writer, b"B", payload)+ return+ c_writer.write(_frame(b"2", b"")) # BindComplete+ await c_writer.drain()+++async def _ext_describe(conn, payload, b_writer, c_writer):+ target, name = _target_name(payload)+ store = conn.ext_stmts if target == b"S" else conn.ext_portals+ entry = store.get(name)+ if entry is None or entry["synth"] is None:+ await _ext_forward(conn, b_writer, b"D", payload)+ return+ out = b""+ if target == b"S":+ out += _frame(b"t", struct.pack("!H", 0)) # ParameterDescription+ out += _row_description(_synth_columns(*entry["synth"]))+ c_writer.write(out)+ await c_writer.drain()+++async def _ext_execute(conn, payload, b_writer, c_writer):+ name = payload.split(b"\0")[0].decode("utf-8", "replace")+ entry = conn.ext_portals.get(name)+ if entry is None or entry["synth"] is None:+ if entry is not None and entry["sql"]:+ _note_catalog(conn, entry["sql"])+ conn.txn.apply(entry["sql"])+ await _ext_forward(conn, b_writer, b"E", payload)+ return++ kind, result = entry["synth"]+ if kind == "rows":+ out = b""+ for row in result["rows"]:+ out += _data_row(row)+ c_writer.write(out + _command_complete(result["tag"]))+ await c_writer.drain()+ return++ conn.txn.apply(entry["sql"])+ job, backend_err = await _run_rewrite(conn, b_writer, result)+ if backend_err is not None:+ conn.ext_skip = True+ c_writer.write(_frame(b"E", backend_err))+ await c_writer.drain()+ return+ c_writer.write(_data_row([job["job_id"]]) + _command_complete("SELECT 1"))+ await c_writer.drain()+++async def _ext_close(conn, payload, b_writer, c_writer):+ target, name = _target_name(payload)+ store = conn.ext_stmts if target == b"S" else conn.ext_portals+ entry = store.pop(name, None)+ if entry is None or entry["synth"] is None:+ await _ext_forward(conn, b_writer, b"C", payload)+ return+ c_writer.write(_frame(b"3", b"")) # CloseComplete+ await c_writer.drain()+++async def _ext_sync(conn, b_writer, c_writer):+ forwarded = conn.ext_forwarded+ conn.ext_skip = False+ conn.ext_forwarded = False+ conn.ext_portals.pop("", None) # the unnamed portal dies at Sync+ if forwarded:+ # The backend answers with its own ReadyForQuery, which also refreshes+ # our transaction state via _backend_relay.+ b_writer.write(_frame(b"S", b""))+ await b_writer.drain()+ return+ c_writer.write(_ready(_status(conn)))+ await c_writer.drain()+++async def _handle_extended(conn, type_byte, payload, b_writer, c_writer):+ if conn.ext_skip and type_byte != b"S":+ return # after an error, everything is ignored until Sync+ if type_byte == b"P":+ await _ext_parse(conn, payload, b_writer, c_writer)+ elif type_byte == b"B":+ await _ext_bind(conn, payload, b_writer, c_writer)+ elif type_byte == b"D":+ await _ext_describe(conn, payload, b_writer, c_writer)+ elif type_byte == b"E":+ await _ext_execute(conn, payload, b_writer, c_writer)+ elif type_byte == b"C":+ await _ext_close(conn, payload, b_writer, c_writer)+ elif type_byte == b"S":+ await _ext_sync(conn, b_writer, c_writer)+ elif type_byte == b"H": # Flush+ if conn.ext_forwarded:+ await _ext_forward(conn, b_writer, b"H", payload)+ else:+ await c_writer.drain()+++_EXTENDED_TYPES = frozenset((b"P", b"B", b"D", b"E", b"C", b"S", b"H"))+++async def _handle_client(cluster_id, backend_host, backend_port, c_reader, c_writer):+ conn = _Conn(cluster_id)+ b_writer = None+ relay = None+ try:+ # Startup phase (no type byte): SSL/GSS negotiation, then StartupMessage.+ while True:+ code, payload = await _read_startup(c_reader)+ if code in (_SSL_REQUEST, _GSSENC_REQUEST):+ c_writer.write(b"N")+ await c_writer.drain()+ continue+ if code == _CANCEL_REQUEST:+ return+ if code == _PROTOCOL_3:+ params = _parse_startup_params(payload)+ break+ return # unknown startup packet — close++ # Authenticate the client ourselves: any user/password is accepted+ # (IAM auth tokens are client-side SigV4 presigning).+ c_writer.write(_client_greeting())+ await c_writer.drain()++ database = params.get("database") or "postgres"+ b_reader, b_writer = await _connect_backend(+ backend_host, backend_port, database+ )+ relay = asyncio.create_task(_backend_relay(conn, b_reader, c_writer))++ while True:+ type_byte, payload = await _read_frame(c_reader)+ if type_byte == b"X": # Terminate+ break+ if type_byte == b"Q":+ sql = payload[:-1].decode("utf-8", "replace") if payload.endswith(b"\0") else payload.decode("utf-8", "replace")+ await _handle_query(conn, sql, b_writer, c_writer)+ elif type_byte in _EXTENDED_TYPES:+ await _handle_extended(conn, type_byte, payload, b_writer, c_writer)+ else:+ # Everything else relays verbatim (d/c/f COPY frames, F...).+ b_writer.write(_frame(type_byte, payload))+ await b_writer.drain()+ except (asyncio.IncompleteReadError, ConnectionResetError, BrokenPipeError, OSError):+ pass+ except Exception:+ logger.exception("pgproxy: error in client connection (cluster %s)", cluster_id)+ finally:+ if relay is not None:+ relay.cancel()+ for w in (b_writer, c_writer):+ if w is not None:+ try:+ w.close()+ except Exception:+ pass+++# ---------------------------------------------------------------------------+# Public proxy lifecycle+# ---------------------------------------------------------------------------+++async def start_proxy(cluster_id, listen_port, backend_host, backend_port):+ """Bind one proxy listener for a cluster. Idempotent per cluster."""+ if cluster_id in _proxies:+ return++ async def handler(reader, writer):+ await _handle_client(cluster_id, backend_host, backend_port, reader, writer)++ server = await asyncio.start_server(handler, "0.0.0.0", listen_port)+ _proxies[cluster_id] = server+ logger.info(+ "pgproxy: cluster %s listening on port %d -> %s:%d",+ cluster_id, listen_port, backend_host, backend_port,+ )+++async def stop_proxy(cluster_id):+ server = _proxies.pop(cluster_id, None)+ if server is not None:+ server.close()+ await server.wait_closed()+ _jobs.pop(cluster_id, None)+++async def stop_all_proxies():+ for cluster_id in list(_proxies):+ await stop_proxy(cluster_id)diff --git a/ministack/core/router.py b/ministack/core/router.pyindex 14ec3972..f123243b 100644--- a/ministack/core/router.py+++ b/ministack/core/router.py@@ -422,6 +422,10 @@ SERVICE_PATTERNS = {"host_patterns": [r"inspector2\."],"credential_scope": "inspector2",},+ "dsql": {+ "host_patterns": [r"dsql\."],+ "credential_scope": "dsql",+ },"s3tables": {"host_patterns": [r"s3tables\."],"credential_scope": "s3tables",@@ -602,6 +606,7 @@ def detect_service(method: str, path: str, headers: dict, query_params: dict) ->"cloudtrail": "cloudtrail","cur": "cur","inspector2": "inspector2",+ "dsql": "dsql","s3tables": "s3tables","kafka": "kafka",}diff --git a/ministack/services/dsql.py b/ministack/services/dsql.pynew file mode 100644index 00000000..70af6ca8--- /dev/null+++ b/ministack/services/dsql.py@@ -0,0 +1,900 @@+"""+Amazon Aurora DSQL Emulator (control plane + data plane wiring).++REST-JSON API (no X-Amz-Target; routed on method + path). Covers cluster+lifecycle, tags, and cluster policies.++Data plane: with ``DSQL_CONTAINERS=1`` and a Docker daemon available, each+cluster gets a real Postgres container (RDS idiom, capped by+``DSQL_MAX_CONTAINERS``, default 30) fronted by an in-process wire-protocol+proxy (`ministack.core.pgproxy`) that enforces DSQL's SQL subset. Otherwise+the cluster goes ACTIVE metadata-only — nothing listens on its port.+"""++import asyncio+import copy+import json+import logging+import os+import secrets+import string+import threading+import time++from ministack.core import pgproxy+from ministack.core.persistence import load_state+from ministack.core.responses import (+ AccountScopedDict,+ get_account_id,+ get_region,+ json_response,+)++logger = logging.getLogger("dsql")++BASE_PORT = int(os.environ.get("DSQL_BASE_PORT", "25432"))+MAX_CONTAINERS = int(os.environ.get("DSQL_MAX_CONTAINERS", "30")) # cap on real Postgres backend containers (one per cluster)+PG_IMAGE = os.environ.get("DSQL_PG_IMAGE", "postgres:16-alpine")+DSQL_PERSIST = os.environ.get("DSQL_PERSIST", "0").lower() in ("1", "true", "yes")+# Backend containers are opt-in: default is control-plane-only (metadata stubs).+DSQL_CONTAINERS = os.environ.get("DSQL_CONTAINERS", "0").lower() in ("1", "true", "yes")++_clusters = AccountScopedDict() # identifier -> cluster dict+_client_tokens = AccountScopedDict() # clientToken -> identifier (create idempotency)+_tags = AccountScopedDict() # cluster arn -> {key: value}++_port_counter = [BASE_PORT]+_port_lock = threading.Lock()++_main_loop = None # captured app event loop for proxy startup from threads++_ID_ALPHABET = string.ascii_lowercase + string.digits+++def _next_port():+ """Allocate a unique localhost port for a cluster endpoint.++ The Postgres wire proxy binds this port per cluster.+ """+ with _port_lock:+ port = _port_counter[0]+ _port_counter[0] += 1+ return port+++def _new_identifier():+ return "".join(secrets.choice(_ID_ALPHABET) for _ in range(26))+++def _cluster_arn(identifier, account_id=None, region=None):+ account_id = account_id or get_account_id()+ region = region or get_region()+ return f"arn:aws:dsql:{region}:{account_id}:cluster/{identifier}"+++def _identifier_from_arn(arn):+ # arn:aws:dsql:{region}:{account}:cluster/{identifier}+ if ":cluster/" in arn:+ return arn.rsplit(":cluster/", 1)[1]+ return ""+++def _kms_key_error(kms_key):+ """ValidationException if a customer-managed key isn't in the KMS store.++ Accepts key id, key ARN, alias name, or alias ARN (same as KMS).+ AWS-managed aliases (``alias/aws/...``) always exist, so they pass.+ """+ if not kms_key:+ return None+ alias = kms_key.split(":alias/")[-1] if ":alias/" in kms_key else kms_key+ if alias.startswith("alias/aws/"):+ return None+ from ministack.services import kms # lazy: only needed when a key is given++ if kms._resolve_key(kms_key) is None:+ return _validation(+ f"KMS key '{kms_key}' does not exist",+ reason="fieldValidationFailed", field="kmsEncryptionKey",+ )+ return None+++def _encryption_details(kms_key, account_id, region):+ if kms_key:+ key_arn = kms_key if kms_key.startswith("arn:") else (+ f"arn:aws:kms:{region}:{account_id}:key/{kms_key}"+ )+ return {+ "encryptionType": "CUSTOMER_MANAGED_KMS_KEY",+ "kmsKeyArn": key_arn,+ "encryptionStatus": "ENABLED",+ }+ return {+ "encryptionType": "AWS_OWNED_KMS_KEY",+ "kmsKeyArn": f"arn:aws:kms:{region}:{account_id}:key/aws/dsql",+ "encryptionStatus": "ENABLED",+ }+++def _error(code, message, status, **extra):+ """AWS-style JSON error with the model's structured members included."""+ data = {"__type": code, "message": message, **extra}+ body = json.dumps(data, ensure_ascii=False).encode("utf-8")+ return status, {+ "Content-Type": "application/x-amz-json-1.0",+ "x-amzn-errortype": code,+ }, body+++def _not_found(resource_id, resource_type="cluster"):+ return _error(+ "ResourceNotFoundException",+ f"The {resource_type} with id '{resource_id}' does not exist",+ 404,+ resourceId=resource_id,+ resourceType=resource_type,+ )+++def _validation(message, reason="other", field=None):+ extra = {"reason": reason}+ if field:+ extra["fieldList"] = [{"name": field, "message": message}]+ return _error("ValidationException", message, 400, **extra)+++def _qp(query_params, *names):+ """Return the first value of the first matching query parameter."""+ for name in names:+ values = query_params.get(name)+ if values:+ return values[0]+ return None+++def _cluster_response(cluster, include_tags=False):+ resp = {+ "identifier": cluster["identifier"],+ "arn": cluster["arn"],+ "status": cluster["status"],+ "creationTime": cluster["creationTime"],+ "deletionProtectionEnabled": cluster["deletionProtectionEnabled"],+ }+ if cluster.get("multiRegionProperties"):+ resp["multiRegionProperties"] = cluster["multiRegionProperties"]+ if cluster.get("encryptionDetails"):+ resp["encryptionDetails"] = cluster["encryptionDetails"]+ if cluster.get("endpoint"):+ resp["endpoint"] = cluster["endpoint"]+ if include_tags:+ resp["tags"] = _tags.get(cluster["arn"], {})+ return resp+++# ---------------------------------------------------------------------------+# Backend container + wire proxy lifecycle (data plane)+# ---------------------------------------------------------------------------+++def _docker_available():+ """Cheap pre-flight check so we don't import docker-py for nothing."""+ sock = os.environ.get("DOCKER_HOST") or "unix:///var/run/docker.sock"+ if sock.startswith("unix://"):+ return os.path.exists(sock[len("unix://"):])+ return True+++def _backends_enabled():+ """Whether real Postgres backend containers should run for clusters.++ Opt-in via ``DSQL_CONTAINERS=1``; default is control-plane-only stubs.+ """+ return DSQL_CONTAINERS and _docker_available()+++def _container_name(identifier):+ return f"ministack-dsql-{identifier}"+++def _published_port(container):+ ports = container.attrs.get("NetworkSettings", {}).get("Ports", {})+ return int(ports["5432/tcp"][0]["HostPort"])+++def _run_backend_container(identifier):+ """Blocking: (re)start the Postgres container and wait for readiness.++ Returns ``(backend_host, backend_port)`` for the proxy to dial. Raises on+ failure; the caller degrades to metadata-only. ``rds`` is imported lazily+ so the heavy module (and its import-time restore) only loads when a+ Docker socket actually exists.+ """+ from ministack.services import rds++ docker_client = rds._get_docker()+ if not docker_client:+ raise RuntimeError("Docker daemon not available")++ name = _container_name(identifier)+ try:+ existing = docker_client.containers.get(name)+ try:+ existing.stop(timeout=2)+ except Exception:+ pass+ try:+ existing.remove(v=False)+ except Exception as e:+ logger.warning("DSQL: failed to remove stale container %s: %s", name, e)+ except Exception:+ pass # No existing container with that name — fine++ ms_network = rds._get_ministack_network(docker_client)+ container_kwargs = dict(+ image=PG_IMAGE,+ detach=True,+ environment={+ "POSTGRES_USER": "postgres",+ "POSTGRES_DB": "postgres",+ "POSTGRES_HOST_AUTH_METHOD": "trust",+ },+ ports={"5432/tcp": None}, # docker-assigned host port+ name=name,+ labels={"ministack": "dsql", "cluster_id": identifier},+ )+ if ms_network:+ container_kwargs["network"] = ms_network+ if DSQL_PERSIST:+ container_kwargs["volumes"] = {+ f"ministack-dsql-{identifier}-data": {+ "bind": "/var/lib/postgresql/data",+ "mode": "rw",+ },+ }+ else:+ container_kwargs["tmpfs"] = {+ "/var/lib/postgresql/data": "rw,noexec,nosuid,size=256m",+ }++ container = docker_client.containers.run(**container_kwargs)+ container.reload()++ host = port = None+ if ms_network:+ networks = container.attrs.get("NetworkSettings", {}).get("Networks", {})+ container_ip = networks.get(ms_network, {}).get("IPAddress", "")+ if container_ip:+ host, port = container_ip, 5432+ if host is None:+ host, port = "localhost", _published_port(container)++ def _container_alive():+ try:+ container.reload()+ return container.status == "running"+ except Exception:+ return False++ # TCP readiness is not enough: docker-proxy accepts connections while the+ # postgres entrypoint is still running initdb, then drops them. Poll a+ # real authenticated connection (trust auth, so no password) like RDS does.+ if not rds._wait_for_database_ready(+ host, port, "postgres", "postgres", "", "postgres", _container_alive+ ):+ raise RuntimeError(f"backend at {host}:{port} did not become ready")+ return host, port+++def _remove_container(identifier):+ """Best-effort blocking removal of a cluster's backend container."""+ if not _docker_available():+ return+ try:+ from ministack.services import rds++ docker_client = rds._get_docker()+ if not docker_client:+ return+ container = docker_client.containers.get(_container_name(identifier))+ container.remove(v=True, force=True)+ except Exception as e:+ logger.debug("DSQL: container removal for %s: %s", identifier, e)+++async def _start_backend(identifier, cluster):+ """Background task: container + proxy, then flip the cluster ACTIVE.++ Any failure degrades to metadata-only (mirroring rds' resilience): the+ cluster still goes ACTIVE, just with nothing listening on its port.+ """+ global _main_loop+ try:+ _main_loop = asyncio.get_running_loop()+ except RuntimeError:+ pass+ try:+ host, port = await asyncio.to_thread(_run_backend_container, identifier)+ await pgproxy.start_proxy(identifier, cluster["port"], host, port)+ cluster["_backend_host"] = host+ cluster["_backend_port"] = port+ cluster["status"] = "ACTIVE"+ logger.info("DSQL: cluster %s backend up at %s:%d", identifier, host, port)+ except Exception as e:+ logger.warning(+ "DSQL: backend startup failed for cluster %s: %s — metadata-only",+ identifier,+ e,+ )+ cluster["status"] = "ACTIVE"+ cluster["_has_backend"] = False+++async def _teardown_backend(identifier):+ try:+ await pgproxy.stop_proxy(identifier)+ except Exception as e:+ logger.debug("DSQL: proxy stop for %s: %s", identifier, e)+ try:+ await asyncio.to_thread(_remove_container, identifier)+ except Exception as e:+ logger.warning("DSQL: backend teardown failed for %s: %s", identifier, e)+++def _respawn_backend(identifier, cluster):+ """Blocking restore path (runs in a daemon thread at import time)."""+ try:+ host, port = _run_backend_container(identifier)+ except Exception as e:+ logger.warning(+ "DSQL: backend respawn failed for cluster %s: %s — metadata-only",+ identifier,+ e,+ )+ cluster["status"] = "ACTIVE"+ cluster["_has_backend"] = False+ return+ cluster["_backend_host"] = host+ cluster["_backend_port"] = port+ cluster["status"] = "ACTIVE"+ # Proxies need the app event loop; it may not exist yet at import time.+ # `start_restored_proxies` (lifespan hook) picks up anything missed here.+ loop = _main_loop+ if loop is not None and loop.is_running():+ asyncio.run_coroutine_threadsafe(+ pgproxy.start_proxy(identifier, cluster["port"], host, port), loop+ )+++async def start_restored_proxies():+ """Lifespan hook: capture the running loop and start wire proxies for+ restored clusters whose backends are already up."""+ global _main_loop+ _main_loop = asyncio.get_running_loop()+ for cluster in list(_clusters._data.values()):+ host = cluster.get("_backend_host")+ port = cluster.get("_backend_port")+ if host and port and cluster.get("status") == "ACTIVE":+ try:+ await pgproxy.start_proxy(+ cluster["identifier"], cluster["port"], host, int(port)+ )+ except Exception as e:+ logger.warning(+ "DSQL: proxy restore failed for %s: %s",+ cluster["identifier"],+ e,+ )+++# ---------------------------------------------------------------------------+# Cluster lifecycle+# ---------------------------------------------------------------------------+++def _create_cluster(data):+ account_id = get_account_id()+ region = get_region()++ client_token = data.get("clientToken")+ if client_token:+ existing_id = _client_tokens.get(client_token)+ if isinstance(existing_id, str) and existing_id in _clusters:+ return json_response(_cluster_response(_clusters[existing_id]))++ key_err = _kms_key_error(data.get("kmsEncryptionKey"))+ if key_err:+ return key_err++ identifier = _new_identifier()+ arn = _cluster_arn(identifier, account_id, region)+ port = _next_port()+ has_backend = _backends_enabled()+ if has_backend:+ backend_count = sum(+ 1 for c in _clusters._data.values() if c.get("_has_backend")+ )+ if backend_count >= MAX_CONTAINERS:+ logger.warning(+ "DSQL: backend container cap reached (%d) — cluster %s is "+ "metadata-only. Raise DSQL_MAX_CONTAINERS to allow more.",+ MAX_CONTAINERS,+ identifier,+ )+ has_backend = False++ cluster = {+ "identifier": identifier,+ "arn": arn,+ # With a backend the Postgres container + wire proxy spin up in the+ # background; without one the cluster is metadata-only ACTIVE.+ "status": "CREATING" if has_backend else "ACTIVE",+ "creationTime": time.time(),+ # AWS creates clusters with deletion protection ON by default.+ "deletionProtectionEnabled": bool(data.get("deletionProtectionEnabled", True)),+ "encryptionDetails": _encryption_details(data.get("kmsEncryptionKey"), account_id, region),+ "port": port,+ "endpoint": f"localhost:{port}",+ "_has_backend": has_backend,+ }+ if data.get("multiRegionProperties"):+ cluster["multiRegionProperties"] = data["multiRegionProperties"]+ if data.get("policy"):+ cluster["policy"] = data["policy"]+ cluster["_policy_version"] = 1++ _clusters[identifier] = cluster+ if client_token:+ _client_tokens[client_token] = identifier+ if data.get("tags"):+ _tags[arn] = dict(data["tags"])++ if has_backend:+ try:+ asyncio.get_running_loop().create_task(_start_backend(identifier, cluster))+ except RuntimeError:+ # No event loop (direct/module-level call) — metadata-only.+ cluster["status"] = "ACTIVE"+ cluster["_has_backend"] = False++ return json_response(_cluster_response(cluster))+++def _get_cluster(identifier):+ cluster = _clusters.get(identifier)+ if not cluster:+ return _not_found(identifier)+ return json_response(_cluster_response(cluster, include_tags=True))+++def _lifecycle_response(cluster):+ """UpdateCluster/DeleteCluster return only these four members per the model."""+ return json_response({+ "identifier": cluster["identifier"],+ "arn": cluster["arn"],+ "status": cluster["status"],+ "creationTime": cluster["creationTime"],+ })+++def _update_cluster(identifier, data):+ cluster = _clusters.get(identifier)+ if not cluster:+ return _not_found(identifier)++ if (+ "deletionProtectionEnabled" not in data+ and not data.get("kmsEncryptionKey")+ and not data.get("multiRegionProperties")+ ):+ return _validation(+ "UpdateCluster requires at least one of deletionProtectionEnabled, "+ "kmsEncryptionKey, multiRegionProperties",+ reason="fieldValidationFailed",+ )++ if "deletionProtectionEnabled" in data:+ cluster["deletionProtectionEnabled"] = bool(data["deletionProtectionEnabled"])+ if data.get("kmsEncryptionKey"):+ key_err = _kms_key_error(data["kmsEncryptionKey"])+ if key_err:+ return key_err+ cluster["encryptionDetails"] = _encryption_details(+ data["kmsEncryptionKey"], get_account_id(), get_region()+ )+ if data.get("multiRegionProperties"):+ cluster["multiRegionProperties"] = data["multiRegionProperties"]+ # Don't clobber CREATING while the backend container is still spinning up.+ if cluster["status"] != "CREATING":+ cluster["status"] = "ACTIVE"++ return _lifecycle_response(cluster)+++def _delete_cluster(identifier, query_params):+ client_token = _qp(query_params, "client-token", "clientToken")+ if client_token:+ cached = _client_tokens.get(client_token)+ # Idempotent retry after a successful delete returns the cached+ # response (the cluster dict is already gone at that point).+ if isinstance(cached, dict) and cached.get("_deleted"):+ return json_response({+ k: v for k, v in cached.items() if k != "_deleted"+ })++ cluster = _clusters.get(identifier)+ if not cluster:+ return _not_found(identifier)++ if client_token:+ seen_arn = _client_tokens.get(client_token)+ # An existing (string) entry is a create token bound to a different+ # cluster. If the seen token is for a different cluster, reject.+ if isinstance(seen_arn, str) and seen_arn != cluster["arn"]:+ return _not_found(identifier)++ if cluster.get("deletionProtectionEnabled"):+ return _validation(+ "Cannot delete cluster: deletion protection is enabled. "+ "Disable deletion protection before deleting the cluster.",+ reason="deletionProtectionEnabled",+ )++ # Stop the wire proxy and remove the backend container (best effort).+ try:+ asyncio.get_running_loop().create_task(_teardown_backend(identifier))+ except RuntimeError:+ _remove_container(identifier)+ cluster["status"] = "DELETING"+ resp = _lifecycle_response(cluster)+ _tags.pop(cluster["arn"], None)+ del _clusters[identifier]+ if client_token:+ # Cache the entire response so an idempotent retry succeeds.+ body = json.loads(resp[2].decode("utf-8"))+ body["_deleted"] = True+ _client_tokens[client_token] = body+ return resp+++def _list_clusters(query_params):+ raw_max = _qp(query_params, "max-results", "maxResults")+ next_token = _qp(query_params, "next-token", "nextToken")++ try:+ max_results = int(raw_max) if raw_max else 100+ except (TypeError, ValueError):+ return _validation(+ f"Invalid maxResults: {raw_max}",+ reason="fieldValidationFailed", field="maxResults",+ )+ if max_results < 1 or max_results > 100:+ return _validation(+ "maxResults must be between 1 and 100",+ reason="fieldValidationFailed", field="maxResults",+ )++ start = 0+ if next_token:+ try:+ start = int(next_token)+ if start < 0:+ raise ValueError+ except (TypeError, ValueError):+ return _validation(+ f"Invalid nextToken: {next_token}",+ reason="fieldValidationFailed", field="nextToken",+ )++ summaries = [+ {"identifier": c["identifier"], "arn": c["arn"]}+ for c in _clusters.values()+ ]+ page = summaries[start : start + max_results]++ resp = {"clusters": page}+ if start + max_results < len(summaries):+ resp["nextToken"] = str(start + max_results)+ return json_response(resp)+++# ---------------------------------------------------------------------------+# Cluster policy+# ---------------------------------------------------------------------------+++def _policy_not_found(identifier):+ return _error(+ "ResourceNotFoundException",+ f"No policy found for cluster '{identifier}'",+ 404,+ resourceId=identifier,+ resourceType="policy",+ )+++def _check_expected_version(cluster, data=None, query_params=None):+ """ConflictException if expectedPolicyVersion is given and stale.++ ``expectedPolicyVersion`` arrives in the JSON body for+ PutClusterPolicy, or as a URI query param for DeleteClusterPolicy.+ """+ expected = None+ if data:+ expected = data.get("expectedPolicyVersion")+ if expected is None and query_params is not None:+ expected = _qp(query_params, "expected-policy-version", "expectedPolicyVersion")+ if expected is not None and expected != str(cluster.get("_policy_version", 0)):+ return _error(+ "ConflictException",+ f"Policy version mismatch: expected {expected}, current is "+ f"{cluster.get('_policy_version', 0)}",+ 409,+ resourceId=cluster["identifier"],+ resourceType="cluster",+ )+ return None+++def _put_cluster_policy(identifier, data):+ cluster = _clusters.get(identifier)+ if not cluster:+ return _not_found(identifier)+ policy = data.get("policy")+ if not isinstance(policy, str) or not policy:+ return _validation(+ "policy is required and must be a non-empty string",+ reason="fieldValidationFailed", field="policy",+ )+ conflict = _check_expected_version(cluster, data)+ if conflict:+ return conflict+ version = cluster.get("_policy_version", 0) + 1+ cluster["policy"] = policy+ cluster["_policy_version"] = version+ return json_response({"policyVersion": str(version)})+++def _get_cluster_policy(identifier):+ cluster = _clusters.get(identifier)+ if not cluster:+ return _not_found(identifier)+ if "policy" not in cluster:+ return _policy_not_found(identifier)+ return json_response({+ "policy": cluster["policy"],+ "policyVersion": str(cluster["_policy_version"]),+ })+++def _delete_cluster_policy(identifier, query_params):+ cluster = _clusters.get(identifier)+ if not cluster:+ return _not_found(identifier)+ if "policy" not in cluster:+ return _policy_not_found(identifier)+ conflict = _check_expected_version(cluster, query_params=query_params)+ if conflict:+ return conflict+ deleted_version = str(cluster["_policy_version"])+ del cluster["policy"]+ cluster["_policy_version"] += 1+ return json_response({"policyVersion": deleted_version})+++# ---------------------------------------------------------------------------+# Tags+# ---------------------------------------------------------------------------+++def _tags_for_arn(arn):+ """Return tags for a known cluster ARN, or a 404 response."""+ identifier = _identifier_from_arn(arn)+ if not identifier or identifier not in _clusters:+ return None, _not_found(arn, resource_type="cluster")+ return _tags.get(arn, {}), None+++def _tag_resource(arn, data):+ _, err = _tags_for_arn(arn)+ if err:+ return err+ new_tags = data.get("tags")+ if not isinstance(new_tags, dict):+ return _validation(+ "tags must be a map of key/value pairs",+ reason="fieldValidationFailed", field="tags",+ )+ existing = _tags.get(arn, {})+ existing.update(new_tags)+ _tags[arn] = existing+ return json_response({})+++def _untag_resource(arn, query_params):+ _, err = _tags_for_arn(arn)+ if err:+ return err+ tag_keys = query_params.get("tagKeys", [])+ existing = _tags.get(arn, {})+ for key in tag_keys:+ existing.pop(key, None)+ _tags[arn] = existing+ return json_response({})+++def _list_tags_for_resource(arn):+ tags, err = _tags_for_arn(arn)+ if err:+ return err+ return json_response({"tags": tags})+++# ---------------------------------------------------------------------------+# Request routing+# ---------------------------------------------------------------------------+++async def handle_request(method, path, headers, body, query_params):+ try:+ data = json.loads(body) if body else {}+ if not isinstance(data, dict):+ return _validation("Request body must be a JSON object", reason="cannotParse")+ except json.JSONDecodeError:+ return _validation("Invalid JSON", reason="cannotParse")++ try:+ # Tag routes embed the ARN in the path: /tags/{resourceArn}+ if path.startswith("/tags/"):+ arn = path[len("/tags/") :]+ if method == "POST":+ return _tag_resource(arn, data)+ if method == "DELETE":+ return _untag_resource(arn, query_params)+ if method == "GET":+ return _list_tags_for_resource(arn)+ return _validation(f"Unknown method for /tags: {method}", reason="unknownOperation")++ if path == "/cluster":+ if method == "POST":+ return _create_cluster(data)+ if method == "GET":+ return _list_clusters(query_params)+ return _validation(f"Unknown method for /cluster: {method}", reason="unknownOperation")++ if path.startswith("/cluster/"):+ identifier = path[len("/cluster/") :]+ if identifier.endswith("/policy"):+ identifier = identifier[: -len("/policy")]+ if method == "POST":+ return _put_cluster_policy(identifier, data)+ if method == "GET":+ return _get_cluster_policy(identifier)+ if method == "DELETE":+ return _delete_cluster_policy(identifier, query_params)+ return _validation(+ f"Unknown method for /cluster/{{id}}/policy: {method}",+ reason="unknownOperation",+ )+ if "/" in identifier:+ return _validation(f"Unknown path: {method} {path}", reason="unknownOperation")+ if method == "GET":+ return _get_cluster(identifier)+ if method == "POST":+ return _update_cluster(identifier, data)+ if method == "DELETE":+ return _delete_cluster(identifier, query_params)+ return _validation(f"Unknown method for /cluster/{{id}}: {method}", reason="unknownOperation")++ return _validation(f"Unknown path: {method} {path}", reason="unknownOperation")+ except Exception as e:+ logger.exception("Error handling %s %s: %s", method, path, e)+ return _error("InternalServerException", str(e), 500)+++# ---------------------------------------------------------------------------+# Persistence+# ---------------------------------------------------------------------------+++def get_state():+ clusters = copy.deepcopy(dict(_clusters._data))+ # Backend addresses are runtime-only (recomputed on restore).+ for cluster in clusters.values():+ if isinstance(cluster, dict):+ cluster.pop("_backend_host", None)+ cluster.pop("_backend_port", None)+ return {+ "clusters": clusters,+ "client_tokens": copy.deepcopy(dict(_client_tokens._data)),+ "tags": copy.deepcopy(dict(_tags._data)),+ "port_counter": _port_counter[0],+ }+++def restore_state(data):+ if not data:+ return+ clusters = data.get("clusters", {})+ if clusters:+ _clusters._data.update(clusters)+ tokens = data.get("client_tokens", {})+ if tokens:+ _client_tokens._data.update(tokens)+ tags = data.get("tags", {})+ if tags:+ _tags._data.update(tags)+ saved_counter = data.get("port_counter")+ if isinstance(saved_counter, int) and saved_counter > _port_counter[0]:+ _port_counter[0] = saved_counter+ # Never reissue a port still owned by a restored cluster.+ for cluster in _clusters._data.values():+ port = cluster.get("port")+ if isinstance(port, int) and port >= _port_counter[0]:+ _port_counter[0] = port + 1+ # Capture the loop when the import fires during request handling, so+ # respawn threads can schedule proxy startup on it.+ global _main_loop+ try:+ _main_loop = asyncio.get_running_loop()+ except RuntimeError:+ pass+ # Respawn backend containers for restored clusters (blocking Docker work+ # in daemon threads, mirroring rds restore). Proxies start via+ # `_respawn_backend` once the loop is known, or via the lifespan hook+ # `start_restored_proxies`. The cap is deliberately not re-enforced here:+ # clusters that had a backend get it back (best effort), so a restart can+ # transiently exceed MAX_CONTAINERS.+ for cluster in _clusters._data.values():+ if not isinstance(cluster, dict) or "identifier" not in cluster:+ continue+ # Clusters persisted as stubs stay stubs. State saved before+ # `_has_backend` existed defaults to respawning, matching the old+ # always-container behavior.+ if not _backends_enabled() or not cluster.get("_has_backend", True):+ cluster["status"] = "ACTIVE"+ cluster["_has_backend"] = False+ continue+ cluster["status"] = "CREATING"+ threading.Thread(+ target=_respawn_backend,+ args=(cluster["identifier"], cluster),+ daemon=True,+ ).start()+++def reset():+ # `/_ministack/reset` runs resets via asyncio.to_thread, so there is no+ # running loop here — schedule proxy shutdown on the captured app loop.+ loop = _main_loop+ if loop is not None and loop.is_running():+ asyncio.run_coroutine_threadsafe(pgproxy.stop_all_proxies(), loop)+ pgproxy.clear_jobs()+ if _docker_available():+ try:+ from ministack.services import rds++ docker_client = rds._get_docker()+ if docker_client:+ for c in docker_client.containers.list(+ all=True, filters={"label": "ministack=dsql"}+ ):+ try:+ c.stop(timeout=2)+ c.remove(v=True)+ except Exception as e:+ logger.warning("reset: failed to remove container %s: %s", c.name, e)+ except Exception as e:+ logger.warning("reset: DSQL container sweep failed: %s", e)+ _clusters.clear()+ _client_tokens.clear()+ _tags.clear()+ _port_counter[0] = BASE_PORT+++try:+ _restored = load_state("dsql")+ if _restored:+ restore_state(_restored)+except Exception:+ logging.getLogger(__name__).exception("Failed to restore persisted state; continuing with fresh store")diff --git a/tests/conftest.py b/tests/conftest.pyindex a563c6f8..023cdaa9 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -562,6 +562,11 @@ def mq():return make_client("mq")[email protected](scope="session")+def dsql():+ return make_client("dsql")++@pytest.fixture(scope="session")def s3tables():return make_client("s3tables")diff --git a/tests/test_dsql.py b/tests/test_dsql.pynew file mode 100644index 00000000..2b589b92--- /dev/null+++ b/tests/test_dsql.py@@ -0,0 +1,2061 @@+"""+Integration tests for the Aurora DSQL control-plane emulator, plus+validator unit tests and live data-plane (wire proxy) tests.+"""++import asyncio+import re+import socket+import struct+import threading+import time++import pytest+from botocore.exceptions import ClientError++from ministack.core import pgproxy++ID_RE = re.compile(r"^[a-z0-9]{26}$")+++def _create(dsql, **kwargs):+ resp = dsql.create_cluster(**kwargs)+ return resp["identifier"]+++def _cleanup(dsql, identifier):+ try:+ dsql.update_cluster(identifier=identifier, deletionProtectionEnabled=False)+ except ClientError:+ pass+ try:+ dsql.delete_cluster(identifier=identifier)+ except ClientError:+ pass+++def _wait_active(dsql, identifier, timeout=60):+ """Poll GetCluster until the cluster is ACTIVE (backend may spin up async)."""+ deadline = time.time() + timeout+ while time.time() < deadline:+ cluster = dsql.get_cluster(identifier=identifier)+ if cluster["status"] == "ACTIVE":+ return cluster+ time.sleep(1)+ raise TimeoutError(f"cluster {identifier} not ACTIVE after {timeout}s")+++class TestClusterLifecycle:+ def test_create_get_list_update_delete(self, dsql):+ resp = dsql.create_cluster()+ identifier = resp["identifier"]+ assert ID_RE.match(identifier)+ assert resp["arn"].endswith(f":cluster/{identifier}")+ assert ":dsql:us-east-1:" in resp["arn"]+ # CREATING while the backend spins up (Docker), ACTIVE without it.+ assert resp["status"] in ("CREATING", "ACTIVE")+ assert resp["deletionProtectionEnabled"] is True # AWS default: on+ assert resp["creationTime"] is not None+ assert resp["endpoint"].startswith("localhost:")++ got = _wait_active(dsql, identifier)+ assert got["identifier"] == identifier+ assert got["tags"] == {}++ listed = dsql.list_clusters()+ assert any(c["identifier"] == identifier for c in listed["clusters"])+ summary = next(c for c in listed["clusters"] if c["identifier"] == identifier)+ assert summary["arn"] == resp["arn"]++ updated = dsql.update_cluster(identifier=identifier, deletionProtectionEnabled=True)+ assert updated["identifier"] == identifier+ assert updated["status"] == "ACTIVE"+ assert dsql.get_cluster(identifier=identifier)["deletionProtectionEnabled"] is True+ dsql.update_cluster(identifier=identifier, deletionProtectionEnabled=False)++ deleted = dsql.delete_cluster(identifier=identifier)+ assert deleted["identifier"] == identifier+ assert deleted["status"] == "DELETING"++ with pytest.raises(ClientError) as exc:+ dsql.get_cluster(identifier=identifier)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"++ def test_create_with_multi_region_properties_echoed(self, dsql):+ identifier = _create(+ dsql,+ multiRegionProperties={"witnessRegion": "us-west-2", "clusters": []},+ )+ try:+ got = dsql.get_cluster(identifier=identifier)+ assert got["multiRegionProperties"] == {+ "witnessRegion": "us-west-2",+ "clusters": [],+ }+ finally:+ _cleanup(dsql, identifier)++ def test_encryption_details_default_aws_owned(self, dsql):+ resp = dsql.create_cluster()+ try:+ enc = resp["encryptionDetails"]+ assert enc["encryptionType"] == "AWS_OWNED_KMS_KEY"+ assert enc["encryptionStatus"] == "ENABLED"+ assert ":key/aws/dsql" in enc["kmsKeyArn"]+ finally:+ _cleanup(dsql, resp["identifier"])++ def test_encryption_details_customer_managed(self, dsql, kms_client):+ key_arn = kms_client.create_key()["KeyMetadata"]["Arn"]+ resp = dsql.create_cluster(kmsEncryptionKey=key_arn)+ identifier = resp["identifier"]+ try:+ enc = resp["encryptionDetails"]+ assert enc["encryptionType"] == "CUSTOMER_MANAGED_KMS_KEY"+ assert enc["kmsKeyArn"] == key_arn+ assert enc["encryptionStatus"] == "ENABLED"++ # UpdateCluster with a new key updates encryptionDetails+ new_key = kms_client.create_key()["KeyMetadata"]["Arn"]+ dsql.update_cluster(identifier=identifier, kmsEncryptionKey=new_key)+ got = dsql.get_cluster(identifier=identifier)+ assert got["encryptionDetails"]["kmsKeyArn"] == new_key+ assert got["encryptionDetails"]["encryptionType"] == "CUSTOMER_MANAGED_KMS_KEY"+ finally:+ _cleanup(dsql, identifier)++ def test_kms_key_validated_against_kms_service(self, dsql, kms_client):+ bogus = "arn:aws:kms:us-east-1:000000000000:key/12345678-1234-1234-1234-123456789012"+ with pytest.raises(ClientError) as exc:+ dsql.create_cluster(kmsEncryptionKey=bogus)+ assert exc.value.response["Error"]["Code"] == "ValidationException"++ # A real key (by id, ARN, or alias) is accepted.+ key = kms_client.create_key()["KeyMetadata"]+ kms_client.create_alias(AliasName="alias/dsql-test", TargetKeyId=key["KeyId"])+ for ref in (key["KeyId"], key["Arn"], "alias/dsql-test"):+ resp = dsql.create_cluster(kmsEncryptionKey=ref)+ identifier = resp["identifier"]+ try:+ enc = resp["encryptionDetails"]+ assert enc["encryptionType"] == "CUSTOMER_MANAGED_KMS_KEY"+ # bare ids/aliases expand to a full key ARN+ assert enc["kmsKeyArn"].startswith("arn:aws:kms:")+ finally:+ _cleanup(dsql, identifier)++ # UpdateCluster validates too.+ identifier = _create(dsql)+ try:+ with pytest.raises(ClientError) as exc:+ dsql.update_cluster(identifier=identifier, kmsEncryptionKey=bogus)+ assert exc.value.response["Error"]["Code"] == "ValidationException"+ finally:+ _cleanup(dsql, identifier)++ def test_kms_aws_managed_alias_accepted(self, dsql):+ # AWS-managed aliases exist by definition — no KMS store entry needed.+ resp = dsql.create_cluster(kmsEncryptionKey="alias/aws/dsql")+ try:+ assert resp["encryptionDetails"]["encryptionType"] == "CUSTOMER_MANAGED_KMS_KEY"+ finally:+ _cleanup(dsql, resp["identifier"])++ def test_endpoints_get_unique_ports(self, dsql):+ a = dsql.create_cluster()+ b = dsql.create_cluster()+ try:+ assert a["endpoint"] != b["endpoint"]+ assert a["endpoint"].split(":")[1].isdigit()+ finally:+ _cleanup(dsql, a["identifier"])+ _cleanup(dsql, b["identifier"])++ def test_deletion_protection_blocks_delete(self, dsql):+ identifier = _create(dsql, deletionProtectionEnabled=True)+ try:+ with pytest.raises(ClientError) as exc:+ dsql.delete_cluster(identifier=identifier)+ err = exc.value.response["Error"]+ assert err["Code"] == "ValidationException"+ assert "deletion protection" in err["Message"].lower()+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400++ # Disable protection, then delete succeeds.+ dsql.update_cluster(identifier=identifier, deletionProtectionEnabled=False)+ deleted = dsql.delete_cluster(identifier=identifier)+ assert deleted["status"] == "DELETING"+ finally:+ _cleanup(dsql, identifier)++ def test_unknown_cluster_id(self, dsql):+ unknown = "a" * 26+ for call in (+ lambda: dsql.get_cluster(identifier=unknown),+ lambda: dsql.update_cluster(identifier=unknown, deletionProtectionEnabled=True),+ lambda: dsql.delete_cluster(identifier=unknown),+ ):+ with pytest.raises(ClientError) as exc:+ call()+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404++ def test_client_token_idempotency(self, dsql):+ token = "test-idempotency-token-12345"+ first = dsql.create_cluster(clientToken=token)+ second = dsql.create_cluster(clientToken=token)+ try:+ assert second["identifier"] == first["identifier"]+ assert second["arn"] == first["arn"]+ finally:+ _cleanup(dsql, first["identifier"])++ def test_list_clusters_pagination(self, dsql):+ ids = {_create(dsql), _create(dsql)}+ try:+ seen = []+ kwargs = {"maxResults": 1}+ pages = 0+ while True:+ resp = dsql.list_clusters(**kwargs)+ assert len(resp["clusters"]) <= 1+ seen.extend(c["identifier"] for c in resp["clusters"])+ pages += 1+ token = resp.get("nextToken")+ if not token:+ break+ kwargs["nextToken"] = token+ assert ids.issubset(set(seen))+ assert pages >= 2+ finally:+ for identifier in ids:+ _cleanup(dsql, identifier)++ def test_list_clusters_invalid_next_token(self, dsql):+ with pytest.raises(ClientError) as exc:+ dsql.list_clusters(nextToken="not-a-valid-token")+ assert exc.value.response["Error"]["Code"] == "ValidationException"++ def test_list_clusters_max_results_out_of_range(self):+ # botocore validates maxResults (model min 1 / max 100) client-side,+ # so exercise the server-side validation with a raw HTTP call.+ import urllib.error+ import urllib.request++ for bad in (0, 101):+ req = urllib.request.Request(+ f"http://127.0.0.1:4566/cluster?max-results={bad}",+ headers={"Host": "dsql.us-east-1.localhost"},+ )+ with pytest.raises(urllib.error.HTTPError) as exc:+ urllib.request.urlopen(req, timeout=10)+ assert exc.value.code == 400, bad+ assert exc.value.headers["x-amzn-errortype"] == "ValidationException"++ def test_update_cluster_empty_body_rejected(self, dsql):+ identifier = _create(dsql)+ try:+ with pytest.raises(ClientError) as exc:+ dsql.update_cluster(identifier=identifier)+ err = exc.value.response["Error"]+ assert err["Code"] == "ValidationException"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400+ finally:+ _cleanup(dsql, identifier)++ def test_delete_cluster_client_token_idempotency(self, dsql):+ identifier = _create(dsql)+ try:+ dsql.update_cluster(identifier=identifier, deletionProtectionEnabled=False)+ token = "delete-idempotency-token-abc"+ first = dsql.delete_cluster(identifier=identifier, clientToken=token)+ assert first["identifier"] == identifier+ assert first["status"] == "DELETING"++ # Second delete with same token returns idempotent success+ second = dsql.delete_cluster(identifier=identifier, clientToken=token)+ assert second["identifier"] == identifier+ assert second["status"] == "DELETING"+ assert second["arn"] == first["arn"]+ assert second["creationTime"] == first["creationTime"]+ finally:+ _cleanup(dsql, identifier)+++class TestTags:+ def test_tags_via_create_and_tag_ops(self, dsql):+ resp = dsql.create_cluster(tags={"env": "test", "team": "data"})+ identifier = resp["identifier"]+ arn = resp["arn"]+ try:+ got = dsql.get_cluster(identifier=identifier)+ assert got["tags"] == {"env": "test", "team": "data"}++ listed = dsql.list_tags_for_resource(resourceArn=arn)+ assert listed["tags"] == {"env": "test", "team": "data"}++ dsql.tag_resource(resourceArn=arn, tags={"owner": "alice", "env": "prod"})+ listed = dsql.list_tags_for_resource(resourceArn=arn)+ assert listed["tags"] == {"env": "prod", "team": "data", "owner": "alice"}++ dsql.untag_resource(resourceArn=arn, tagKeys=["team", "owner"])+ listed = dsql.list_tags_for_resource(resourceArn=arn)+ assert listed["tags"] == {"env": "prod"}+ finally:+ _cleanup(dsql, identifier)++ def test_tag_ops_unknown_arn(self, dsql):+ unknown_arn = "arn:aws:dsql:us-east-1:000000000000:cluster/" + "z" * 26+ with pytest.raises(ClientError) as exc:+ dsql.list_tags_for_resource(resourceArn=unknown_arn)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"++ with pytest.raises(ClientError) as exc:+ dsql.tag_resource(resourceArn=unknown_arn, tags={"k": "v"})+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"++ with pytest.raises(ClientError) as exc:+ dsql.untag_resource(resourceArn=unknown_arn, tagKeys=["k"])+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"+++class TestClusterPolicy:+ POLICY = '{"Version":"2012-10-17","Statement":[]}'++ def test_policy_put_get_delete(self, dsql):+ identifier = _create(dsql)+ try:+ with pytest.raises(ClientError) as exc:+ dsql.get_cluster_policy(identifier=identifier)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"++ put = dsql.put_cluster_policy(identifier=identifier, policy=self.POLICY)+ assert put["policyVersion"] == "1"++ got = dsql.get_cluster_policy(identifier=identifier)+ assert got["policy"] == self.POLICY+ assert got["policyVersion"] == "1"++ put2 = dsql.put_cluster_policy(+ identifier=identifier, policy=self.POLICY, expectedPolicyVersion="1"+ )+ assert put2["policyVersion"] == "2"++ deleted = dsql.delete_cluster_policy(identifier=identifier)+ assert deleted["policyVersion"] == "2"++ with pytest.raises(ClientError) as exc:+ dsql.get_cluster_policy(identifier=identifier)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"+ finally:+ _cleanup(dsql, identifier)++ def test_policy_version_conflict(self, dsql):+ identifier = _create(dsql)+ try:+ dsql.put_cluster_policy(identifier=identifier, policy=self.POLICY)+ with pytest.raises(ClientError) as exc:+ dsql.put_cluster_policy(+ identifier=identifier, policy=self.POLICY, expectedPolicyVersion="99"+ )+ assert exc.value.response["Error"]["Code"] == "ConflictException"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 409+ finally:+ _cleanup(dsql, identifier)++ def test_policy_on_unknown_cluster(self, dsql):+ unknown = "z" * 26+ with pytest.raises(ClientError) as exc:+ dsql.put_cluster_policy(identifier=unknown, policy=self.POLICY)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"++ def test_create_cluster_with_policy(self, dsql):+ resp = dsql.create_cluster(policy=self.POLICY)+ identifier = resp["identifier"]+ try:+ got = dsql.get_cluster_policy(identifier=identifier)+ assert got["policy"] == self.POLICY+ assert got["policyVersion"] == "1"+ finally:+ _cleanup(dsql, identifier)++ def test_delete_policy_with_expected_version_success(self, dsql):+ identifier = _create(dsql)+ try:+ dsql.put_cluster_policy(identifier=identifier, policy=self.POLICY)+ deleted = dsql.delete_cluster_policy(+ identifier=identifier, expectedPolicyVersion="1"+ )+ assert deleted["policyVersion"] == "1"+ finally:+ _cleanup(dsql, identifier)++ def test_delete_policy_expected_version_conflict(self, dsql):+ identifier = _create(dsql)+ try:+ dsql.put_cluster_policy(identifier=identifier, policy=self.POLICY)+ with pytest.raises(ClientError) as exc:+ dsql.delete_cluster_policy(+ identifier=identifier, expectedPolicyVersion="99"+ )+ assert exc.value.response["Error"]["Code"] == "ConflictException"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 409+ finally:+ _cleanup(dsql, identifier)+++class TestBotoParity:+ """Waiters, paginators, and structured error members (botocore model)."""++ def test_cluster_active_and_not_exists_waiters(self, dsql):+ identifier = _create(dsql)+ dsql.get_waiter("cluster_active").wait(+ identifier=identifier, WaiterConfig={"Delay": 1, "MaxAttempts": 60}+ )+ assert dsql.get_cluster(identifier=identifier)["status"] == "ACTIVE"++ dsql.update_cluster(identifier=identifier, deletionProtectionEnabled=False)+ dsql.delete_cluster(identifier=identifier)+ dsql.get_waiter("cluster_not_exists").wait(+ identifier=identifier, WaiterConfig={"Delay": 1, "MaxAttempts": 10}+ )++ def test_list_clusters_paginator(self, dsql):+ ids = {_create(dsql), _create(dsql)}+ try:+ paginator = dsql.get_paginator("list_clusters")+ seen = []+ for page in paginator.paginate(PaginationConfig={"PageSize": 1}):+ seen.extend(c["identifier"] for c in page["clusters"])+ assert ids.issubset(set(seen))+ assert len(seen) >= 2+ finally:+ for identifier in ids:+ _cleanup(dsql, identifier)++ def test_error_members_not_found(self, dsql):+ unknown = "z" * 26+ with pytest.raises(ClientError) as exc:+ dsql.get_cluster(identifier=unknown)+ resp = exc.value.response+ assert resp["Error"]["Code"] == "ResourceNotFoundException"+ # botocore parses modeled error members at the response top level+ assert resp["resourceId"] == unknown+ assert resp["resourceType"] == "cluster"++ def test_error_members_deletion_protection(self, dsql):+ identifier = _create(dsql, deletionProtectionEnabled=True)+ try:+ with pytest.raises(ClientError) as exc:+ dsql.delete_cluster(identifier=identifier)+ resp = exc.value.response+ assert resp["Error"]["Code"] == "ValidationException"+ assert resp["reason"] == "deletionProtectionEnabled"+ finally:+ _cleanup(dsql, identifier)++ def test_update_delete_response_shape(self, dsql):+ identifier = _create(dsql)+ try:+ updated = dsql.update_cluster(+ identifier=identifier, deletionProtectionEnabled=False+ )+ assert set(updated) == {+ "identifier", "arn", "status", "creationTime", "ResponseMetadata",+ }+ finally:+ deleted = dsql.delete_cluster(identifier=identifier)+ assert set(deleted) == {+ "identifier", "arn", "status", "creationTime", "ResponseMetadata",+ }+++class TestAuthTokens:+ def test_generate_db_connect_auth_token(self, dsql):+ token = dsql.generate_db_connect_auth_token("localhost", "us-east-1")+ assert isinstance(token, str)+ assert "Action=DbConnect" in token++ def test_generate_db_connect_admin_auth_token(self, dsql):+ token = dsql.generate_db_connect_admin_auth_token("localhost", "us-east-1")+ assert isinstance(token, str)+ assert "Action=DbConnectAdmin" in token+++# ---------------------------------------------------------------------------+# Validator unit tests (no server, no Postgres needed)+# ---------------------------------------------------------------------------+++class TestClassifyStatement:+ @pytest.mark.parametrize(+ "sql,expected",+ [+ ("CREATE TABLE t (a int)", "ddl"),+ ("alter table t add column b int", "ddl"),+ ("DROP INDEX idx", "ddl"),+ ("INSERT INTO t VALUES (1)", "dml"),+ ("update t set a = 1", "dml"),+ ("DELETE FROM t", "dml"),+ ("BEGIN", "tcl_begin"),+ ("START TRANSACTION", "tcl_begin"),+ ("COMMIT", "tcl_end"),+ ("END", "tcl_end"),+ ("ROLLBACK", "tcl_end"),+ ("ABORT", "tcl_end"),+ ("SELECT 1", "other"),+ ("SET search_path TO public", "other"),+ ("SHOW server_version", "other"),+ ("GRANT SELECT ON t TO r", "other"),+ ("REVOKE SELECT ON t FROM r", "other"),+ # Leading comments must not hide the statement keyword.+ ("-- migration 004\nCREATE TABLE t (a int)", "ddl"),+ ("/* flyway */ INSERT INTO t VALUES (1)", "dml"),+ ("/* a /* nested */ b */ COMMIT", "tcl_end"),+ ("\n\n-- one\n-- two\nBEGIN", "tcl_begin"),+ # CTE-leading writes are still writes.+ ("WITH x AS (SELECT 1) INSERT INTO t SELECT 1", "dml"),+ ("WITH x AS (SELECT 1) UPDATE t SET a = 1", "dml"),+ ("WITH x AS (SELECT 1) DELETE FROM t", "dml"),+ ("WITH RECURSIVE x AS (SELECT 1) INSERT INTO t SELECT 1", "dml"),+ ("WITH x (a, b) AS (SELECT 1, 2) INSERT INTO t SELECT 1", "dml"),+ ("WITH x AS MATERIALIZED (SELECT 1) INSERT INTO t SELECT 1", "dml"),+ ("WITH a AS (SELECT 1), b AS (SELECT 2) INSERT INTO t SELECT 1", "dml"),+ ("WITH x AS (SELECT 1) SELECT * FROM x", "other"),+ # EXPLAIN ANALYZE executes; plain EXPLAIN only plans.+ ("EXPLAIN ANALYZE INSERT INTO t VALUES (1)", "dml"),+ ("EXPLAIN (ANALYZE, BUFFERS) DELETE FROM t", "dml"),+ ("EXPLAIN INSERT INTO t VALUES (1)", "other"),+ ("EXPLAIN SELECT 1", "other"),+ ],+ )+ def test_classify(self, sql, expected):+ assert pgproxy.classify_statement(sql) == expected+++class TestSqlLexing:+ """Statement splitting and comment stripping must not be fooled by+ literals — a ';' or ')' inside a string used to truncate the text a+ validation rule saw, so the rule passed on the fragment."""++ @pytest.mark.parametrize(+ "sql,expected",+ [+ ("SELECT 1; SELECT 2", ["SELECT 1", "SELECT 2"]),+ ("SELECT 'a;b'", ["SELECT 'a;b'"]),+ ("SELECT 'it''s; fine'", ["SELECT 'it''s; fine'"]),+ ("SELECT $$a;b$$", ["SELECT $$a;b$$"]),+ ("SELECT $tag$a;b$tag$", ["SELECT $tag$a;b$tag$"]),+ ('SELECT "we;ird"', ['SELECT "we;ird"']),+ ("SELECT 1 -- c;omment\n; SELECT 2", ["SELECT 1 -- c;omment", "SELECT 2"]),+ ("SELECT 1 /* c;omment */; SELECT 2", ["SELECT 1 /* c;omment */", "SELECT 2"]),+ ("SELECT E'a\\';b'", ["SELECT E'a\\';b'"]),+ ("SELECT 1;", ["SELECT 1"]),+ (" ; ", []),+ ],+ )+ def test_split_statements(self, sql, expected):+ assert pgproxy.split_statements(sql) == expected++ @pytest.mark.parametrize(+ "sql,expected",+ [+ ("-- x\nSELECT 1", "SELECT 1"),+ ("/* x */SELECT 1", "SELECT 1"),+ ("/* a /* b */ c */ SELECT 1", "SELECT 1"),+ (" \n\t-- a\n /* b */\nSELECT 1", "SELECT 1"),+ ("SELECT 1 -- trailing", "SELECT 1 -- trailing"),+ ("'-- not a comment'", "'-- not a comment'"),+ ],+ )+ def test_strip_leading_comments(self, sql, expected):+ assert pgproxy.strip_leading_comments(sql) == expected++ def test_semicolon_in_literal_does_not_hide_a_bad_column(self):+ # The fragment left by a naive ';' split had unbalanced parens, so the+ # column-type rule silently passed and `serial` got through.+ err = pgproxy.validate(+ "CREATE TABLE t (a text DEFAULT 'x;y', b serial)"+ )+ assert isinstance(err, pgproxy.DsqlError)+ assert "serial" in err.message++ def test_paren_in_literal_does_not_hide_a_bad_column(self):+ err = pgproxy.validate("CREATE TABLE t (a text DEFAULT ')(', b serial)")+ assert isinstance(err, pgproxy.DsqlError)+ assert "serial" in err.message++ @pytest.mark.parametrize(+ "sql",+ [+ "-- migration\nCREATE EXTENSION pgcrypto",+ "/* tooling */ TRUNCATE t",+ "\n-- a\n/* b */ CREATE TEMP TABLE t (a int)",+ ],+ )+ def test_leading_comment_does_not_bypass_the_denylist(self, sql):+ assert isinstance(pgproxy.validate(sql), pgproxy.DsqlError)+++class TestDenylist:+ @pytest.mark.parametrize(+ "sql",+ [+ "CREATE DATABASE foo",+ "CREATE TYPE mood AS ENUM ('sad')",+ "CREATE EXTENSION pgcrypto",+ "CREATE TRIGGER trg AFTER INSERT ON t EXECUTE FUNCTION f()",+ "CREATE PROCEDURE p() LANGUAGE sql AS $$ SELECT 1 $$",+ "CREATE TABLESPACE ts LOCATION '/tmp/x'",+ "CREATE TEMP TABLE t (a int)",+ "CREATE TEMPORARY TABLE t (a int)",+ "CREATE UNLOGGED TABLE t (a int)",+ "COPY t FROM STDIN",+ "LISTEN chan",+ "NOTIFY chan",+ "UNLISTEN chan",+ "TRUNCATE t",+ "DO $$ BEGIN NULL; END $$",+ "SAVEPOINT sp",+ "LOCK TABLE t IN ACCESS EXCLUSIVE MODE",+ "CREATE FUNCTION f() RETURNS int LANGUAGE plpgsql AS $$ BEGIN RETURN 1; END $$",+ "CREATE FUNCTION f() RETURNS int LANGUAGE C AS 'x'",+ "CREATE MATERIALIZED VIEW mv AS SELECT 1",+ "CREATE TABLE t (a int) PARTITION BY RANGE (a)",+ "CREATE TABLE p (a int, EXCLUDE (a WITH =))",+ "CREATE RULE r AS ON DELETE TO t DO INSTEAD NOTHING",+ "CREATE AGGREGATE a (int) (sfunc = f, stype = int)",+ "CREATE DOMAIN d AS int",+ "CREATE CAST (int AS text) WITH FUNCTION f(int)",+ "CREATE COLLATION c (locale = 'en_US')",+ "CREATE OPERATOR + (leftarg = int, rightarg = int, function = f)",+ "CREATE PUBLICATION p FOR ALL TABLES",+ "CREATE SUBSCRIPTION s CONNECTION 'x' PUBLICATION p",+ "CREATE FOREIGN TABLE ft (a int) SERVER s",+ "CREATE SERVER s FOREIGN DATA WRAPPER w",+ "VACUUM t",+ "VACUUM FULL ANALYZE t",+ "CLUSTER t USING idx",+ "REINDEX TABLE t",+ "ALTER SYSTEM SET work_mem = '4MB'",+ "PREPARE TRANSACTION 'gid'",+ "COMMIT PREPARED 'gid'",+ "ROLLBACK PREPARED 'gid'",+ ],+ )+ def test_denied(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000"+ assert "is not supported" in err.message++ @pytest.mark.parametrize(+ "sql",+ [+ "CREATE TABLE t (a int, b int REFERENCES other (id))",+ "CREATE TABLE t (a int, FOREIGN KEY (a) REFERENCES other (id))",+ "ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (a) REFERENCES o (id)",+ ],+ )+ def test_foreign_keys_denied(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000"+ assert "foreign key" in err.message++ @pytest.mark.parametrize(+ "sql",+ [+ "ALTER TABLE t ADD COLUMN c int DEFAULT 0",+ "ALTER TABLE t ADD COLUMN c text NOT NULL",+ ],+ )+ def test_add_column_default_not_null_denied(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000"++ def test_create_function_language_sql_allowed(self):+ sql = "CREATE FUNCTION f() RETURNS int LANGUAGE sql AS $$ SELECT 1 $$"+ assert pgproxy.validate(sql, pgproxy.TxnState()) is None+++class TestColumnTypes:+ @pytest.mark.parametrize(+ "typ",+ [+ "serial", "bigserial", "smallserial",+ "money", "xml", "inet", "cidr", "macaddr", "macaddr8",+ "bit", "bit varying", "varbit",+ "point", "line", "lseg", "box", "path", "polygon", "circle",+ "hstore",+ "int[]", "text[]",+ ],+ )+ def test_unsupported_types_denied(self, typ):+ err = pgproxy.validate(+ f"CREATE TABLE t (id int, c {typ})", pgproxy.TxnState()+ )+ assert isinstance(err, pgproxy.DsqlError), typ+ assert err.sqlstate == "0A000"+ assert 'type "' in err.message and "is not supported" in err.message++ def test_unsupported_type_in_alter_add_column(self):+ err = pgproxy.validate(+ "ALTER TABLE t ADD COLUMN c serial", pgproxy.TxnState()+ )+ assert isinstance(err, pgproxy.DsqlError)+ assert err.sqlstate == "0A000"+ assert '"serial"' in err.message++ @pytest.mark.parametrize(+ "typ",+ [+ "smallint", "int2", "integer", "int", "int4", "bigint", "int8",+ "real", "float4", "double precision", "float8",+ "numeric", "decimal", "dec", "numeric(18,6)", "dec(10,2)",+ "char(1)", "character(1)", "bpchar", "varchar(255)",+ "character varying(10)", "text",+ "date", "time", "time with time zone", "timetz",+ "timestamp", "timestamp with time zone", "timestamptz",+ "interval", "interval year", "interval day to second",+ "interval year to month", "interval second", "boolean", "bool", "bytea", "uuid", "json", "jsonb",+ ],+ )+ def test_supported_types_allowed(self, typ):+ assert pgproxy.validate(+ f"CREATE TABLE t (id int, c {typ})", pgproxy.TxnState()+ ) is None++ def test_table_constraints_skipped(self):+ sql = "CREATE TABLE t (id int PRIMARY KEY, c text UNIQUE, CHECK (id > 0))"+ assert pgproxy.validate(sql, pgproxy.TxnState()) is None++ def test_storage_modifier_allowed(self):+ assert pgproxy.validate(+ "CREATE TABLE t (id int, j jsonb STORAGE PLAIN)", pgproxy.TxnState()+ ) is None+ assert pgproxy.validate(+ "ALTER TABLE t ADD COLUMN j jsonb STORAGE EXTERNAL", pgproxy.TxnState()+ ) is None+ assert pgproxy.validate(+ "ALTER TABLE t ADD COLUMN j jsonb STORAGE DEFAULT", pgproxy.TxnState()+ ) is None++ def test_bogus_interval_fields_rejected(self):+ err = pgproxy.validate(+ "CREATE TABLE t (id int, c interval foo)", pgproxy.TxnState()+ )+ assert isinstance(err, pgproxy.DsqlError)+ assert err.sqlstate == "0A000"+++class TestAlterTableSubset:+ """ALTER TABLE action subset per the AWS alter-table-syntax-support doc."""++ @pytest.mark.parametrize(+ "sql",+ [+ "ALTER TABLE t ALTER COLUMN c TYPE text",+ "ALTER TABLE t ALTER COLUMN c SET DATA TYPE text",+ "ALTER TABLE t ALTER COLUMN c SET NOT NULL",+ "ALTER TABLE t ADD CONSTRAINT pk PRIMARY KEY (id)",+ "ALTER TABLE t ADD PRIMARY KEY (id)",+ "ALTER TABLE t ADD CONSTRAINT u UNIQUE (a)",+ "ALTER TABLE t ADD CONSTRAINT c CHECK (a > 0)", # missing NOT VALID+ "ALTER TABLE t ALTER COLUMN c ADD GENERATED BY DEFAULT AS IDENTITY", # no CACHE+ ],+ )+ def test_unsupported_actions_denied(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000"++ @pytest.mark.parametrize(+ "sql",+ [+ "ALTER TABLE t DROP COLUMN c",+ "ALTER TABLE t DROP COLUMN IF EXISTS c CASCADE",+ "ALTER TABLE t ALTER COLUMN c SET DEFAULT 0",+ "ALTER TABLE t ALTER COLUMN c DROP DEFAULT",+ "ALTER TABLE t ALTER COLUMN c DROP NOT NULL",+ "ALTER TABLE t ALTER COLUMN c SET STORAGE PLAIN",+ "ALTER TABLE t RENAME COLUMN a TO b",+ "ALTER TABLE t RENAME TO t2",+ "ALTER TABLE t SET SCHEMA app",+ "ALTER TABLE t ADD CONSTRAINT c CHECK (a > 0) NOT VALID",+ "ALTER TABLE t ADD CONSTRAINT u UNIQUE USING INDEX idx",+ "ALTER TABLE t DROP CONSTRAINT c",+ "ALTER TABLE t ALTER COLUMN c ADD GENERATED BY DEFAULT AS IDENTITY (CACHE 1)",+ "ALTER TABLE t ADD COLUMN c integer",+ "ALTER TABLE t ADD COLUMN j jsonb STORAGE PLAIN",+ "ALTER TABLE t ADD COLUMN c bigint GENERATED BY DEFAULT AS IDENTITY (CACHE 1)",+ "ALTER TABLE t ADD COLUMN cl text GENERATED ALWAYS AS (lower(email)) STORED",+ ],+ )+ def test_supported_actions_allowed(self, sql):+ assert pgproxy.validate(sql, pgproxy.TxnState()) is None, sql++ @pytest.mark.parametrize(+ "sql,expected",+ [+ ("ALTER TABLE t DROP COLUMN c", ["c"]),+ ("ALTER TABLE t DROP c", ["c"]),+ ("ALTER TABLE t DROP COLUMN IF EXISTS c CASCADE", ["c"]),+ ("ALTER TABLE t DROP COLUMN a, DROP COLUMN b", ["a", "b"]),+ ("ALTER TABLE t DROP a, DROP COLUMN b, DROP IF EXISTS c", ["a", "b", "c"]),+ ('ALTER TABLE t DROP COLUMN "MixedCase"', ["MixedCase"]),+ # Actions that merely start with DROP are not column drops.+ ("ALTER TABLE t DROP CONSTRAINT c", []),+ ("ALTER TABLE t ALTER COLUMN c DROP DEFAULT", []),+ ("ALTER TABLE t ALTER COLUMN c DROP NOT NULL", []),+ ("ALTER TABLE t ALTER COLUMN c DROP EXPRESSION", []),+ ("ALTER TABLE t ALTER COLUMN c DROP IDENTITY", []),+ ("ALTER TABLE t ADD COLUMN c int", []),+ ("DROP TABLE t", []),+ ],+ )+ def test_dropped_columns(self, sql, expected):+ assert pgproxy.dropped_columns(sql) == expected++ def test_drop_column_still_passes_text_only_validation(self):+ # The primary-key restriction needs a catalog probe, so validate()+ # stays clean and the rule lives in the proxy's statement planner.+ assert pgproxy.validate("ALTER TABLE t DROP COLUMN c") is None++ def test_validate_constraint_async_rewrite(self):+ result = pgproxy.validate(+ "ALTER TABLE ASYNC t VALIDATE CONSTRAINT c", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+ assert result.sql == "ALTER TABLE t VALIDATE CONSTRAINT c"+ assert result.job_type == "VALIDATE_CONSTRAINT"+ assert result.object_name == "public.t"+++class TestIdentityAndSequences:+ @pytest.mark.parametrize(+ "sql",+ [+ "CREATE TABLE t (id bigint GENERATED BY DEFAULT AS IDENTITY (CACHE 1) PRIMARY KEY)",+ "CREATE TABLE t (id bigint GENERATED ALWAYS AS IDENTITY (CACHE 65536))",+ "CREATE TABLE t (id bigint GENERATED BY DEFAULT AS IDENTITY)",+ "CREATE SEQUENCE s CACHE 1",+ "CREATE SEQUENCE s CACHE 65536",+ "CREATE SEQUENCE s CACHE 1000000",+ "CREATE SEQUENCE s",+ ],+ )+ def test_allowed(self, sql):+ assert pgproxy.validate(sql, pgproxy.TxnState()) is None, sql++ @pytest.mark.parametrize(+ "sql",+ [+ # identity is bigint-only+ "CREATE TABLE t (id int GENERATED BY DEFAULT AS IDENTITY (CACHE 1))",+ "CREATE TABLE t (id integer GENERATED ALWAYS AS IDENTITY (CACHE 65536))",+ # CACHE must be 1 or >= 65536+ "CREATE TABLE t (id bigint GENERATED BY DEFAULT AS IDENTITY (CACHE 100))",+ "CREATE SEQUENCE s CACHE 100",+ "CREATE SEQUENCE s CACHE 65535",+ ],+ )+ def test_denied(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000"+++class TestIndexLimits:+ @pytest.mark.parametrize(+ "sql",+ [+ "CREATE INDEX i ON t USING gin (data)",+ "CREATE INDEX i ON t USING gist (coords)",+ "CREATE INDEX i ON t USING brin (created_at)",+ "CREATE INDEX i ON t USING hash (a)",+ "CREATE INDEX ASYNC i ON t USING gin (data)",+ "CREATE INDEX CONCURRENTLY i ON t (a)",+ "CREATE INDEX i ON t (a) WHERE a > 0", # partial index+ "CREATE INDEX ASYNC i ON t (a) WHERE a > 0",+ "CREATE INDEX i ON t (lower(email))", # expression index+ "CREATE INDEX ASYNC i ON t ((data->>'city'))",+ ],+ )+ def test_unsupported_index_forms_denied(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000"++ @pytest.mark.parametrize(+ "sql",+ [+ "CREATE INDEX i ON t (a)",+ "CREATE INDEX i ON t USING btree (a)",+ "CREATE UNIQUE INDEX i ON t (a NULLS LAST) INCLUDE (b)",+ "CREATE INDEX ASYNC i ON t (a, b DESC) INCLUDE (c)",+ ],+ )+ def test_supported_index_forms_allowed(self, sql):+ result = pgproxy.validate(sql, pgproxy.TxnState())+ assert not isinstance(result, pgproxy.DsqlError), sql++ def test_more_than_8_columns_denied(self):+ for kw in ("", "ASYNC "):+ err = pgproxy.validate(+ f"CREATE INDEX {kw}i ON t (a,b,c,d,e,f,g,h,i)", pgproxy.TxnState()+ )+ assert isinstance(err, pgproxy.DsqlError), kw+ assert err.sqlstate == "0A000"+ assert "8 columns" in err.message++ def test_exactly_8_columns_allowed(self):+ assert pgproxy.validate(+ "CREATE INDEX i ON t (a,b,c,d,e,f,g,h)", pgproxy.TxnState()+ ) is None+ result = pgproxy.validate(+ "CREATE INDEX ASYNC i ON t (a,b,c,d,e,f,g,h)", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+++class TestWaitForJob:+ def test_documented_form(self):+ sql = "SELECT sys.wait_for_job(job_id) 'abc123'"+ assert pgproxy.match_wait_for_job(sql) == "abc123"++ def test_conventional_form(self):+ sql = "SELECT sys.wait_for_job('abc123')"+ assert pgproxy.match_wait_for_job(sql) == "abc123"++ def test_call_form(self):+ assert pgproxy.match_wait_for_job("CALL sys.wait_for_job('abc123')") == "abc123"++ def test_no_match(self):+ assert pgproxy.match_wait_for_job("SELECT 1") is None+ assert pgproxy.match_wait_for_job("SELECT sys.wait_for_job(job_id)") is None+++class TestReset:+ def test_reset_stops_proxies_and_clears_jobs(self):+ """reset() runs in a worker thread (no running loop) — it must still+ schedule proxy shutdown on the captured app loop and drop job state."""+ from ministack.services import dsql as dsql_mod++ loop = asyncio.new_event_loop()+ thread = threading.Thread(target=loop.run_forever, daemon=True)+ thread.start()+ old_loop = dsql_mod._main_loop+ dsql_mod._main_loop = loop+ try:+ port = _free_port()+ asyncio.run_coroutine_threadsafe(+ pgproxy.start_proxy("resettest", port, "127.0.0.1", 1), loop+ ).result(timeout=5)+ pgproxy._jobs["resettest"] = [{"job_id": "x"}]+ assert "resettest" in pgproxy._proxies++ dsql_mod.reset() # called from this thread: no running loop here++ deadline = time.time() + 5+ while time.time() < deadline and "resettest" in pgproxy._proxies:+ time.sleep(0.05)+ assert "resettest" not in pgproxy._proxies+ assert pgproxy.get_jobs("resettest") == []+ finally:+ dsql_mod._main_loop = old_loop+ loop.call_soon_threadsafe(loop.stop)+ thread.join(timeout=5)+++class TestContainerCap:+ def test_cap_exceeded_logs_and_degrades_to_stub(self, monkeypatch, caplog):+ """When the backend container cap is reached, creating another cluster+ logs a warning naming DSQL_MAX_CONTAINERS and goes metadata-only."""+ import json+ import logging++ from ministack.services import dsql as dsql_mod++ monkeypatch.setattr(dsql_mod, "_backends_enabled", lambda: True)+ monkeypatch.setattr(dsql_mod, "MAX_CONTAINERS", 0)++ with caplog.at_level(logging.WARNING, logger="dsql"):+ status, _, body = dsql_mod._create_cluster({})++ assert status == 200+ cluster = json.loads(body)+ identifier = cluster["identifier"]+ try:+ assert cluster["status"] == "ACTIVE" # stub, not CREATING+ assert dsql_mod._clusters[identifier]["_has_backend"] is False+ assert any(+ "DSQL_MAX_CONTAINERS" in r.message and "cap reached" in r.message+ for r in caplog.records+ if r.name == "dsql" and r.levelno == logging.WARNING+ )+ finally:+ dsql_mod._clusters.clear()++ def test_containers_disabled_means_stub_even_with_docker(self, monkeypatch):+ """Default posture: Docker available but DSQL_CONTAINERS unset/off —+ clusters are metadata-only stubs and no backend is started."""+ import json++ from ministack.services import dsql as dsql_mod++ monkeypatch.setattr(dsql_mod, "DSQL_CONTAINERS", False)+ monkeypatch.setattr(dsql_mod, "_docker_available", lambda: True)+ assert dsql_mod._backends_enabled() is False++ status, _, body = dsql_mod._create_cluster({})+ assert status == 200+ cluster = json.loads(body)+ identifier = cluster["identifier"]+ try:+ assert cluster["status"] == "ACTIVE" # stub, not CREATING+ assert dsql_mod._clusters[identifier]["_has_backend"] is False+ finally:+ dsql_mod._clusters.clear()++ def test_backends_enabled_requires_flag_and_docker(self, monkeypatch):+ from ministack.services import dsql as dsql_mod++ monkeypatch.setattr(dsql_mod, "_docker_available", lambda: True)+ monkeypatch.setattr(dsql_mod, "DSQL_CONTAINERS", False)+ assert dsql_mod._backends_enabled() is False+ monkeypatch.setattr(dsql_mod, "DSQL_CONTAINERS", True)+ assert dsql_mod._backends_enabled() is True+ monkeypatch.setattr(dsql_mod, "_docker_available", lambda: False)+ assert dsql_mod._backends_enabled() is False++++class TestIndexAsync:+ def test_rewrite_strips_async(self):+ assert pgproxy.rewrite_index_async(+ "CREATE UNIQUE INDEX ASYNC IF NOT EXISTS idx ON t (a)"+ ) == "CREATE UNIQUE INDEX IF NOT EXISTS idx ON t (a)"++ def test_validate_returns_rewrite(self):+ result = pgproxy.validate(+ "CREATE INDEX ASYNC idx ON t (a)", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+ assert result.kind == "index_async"+ assert result.sql == "CREATE INDEX idx ON t (a)"+ assert result.object_name == "public.idx"+ assert result.table == "t"++ def test_named_index_takes_table_schema(self):+ result = pgproxy.validate(+ "CREATE INDEX ASYNC idx ON app.t (a)", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+ assert result.object_name == "app.idx"++ def test_schema_qualified_index_name_kept(self):+ result = pgproxy.validate(+ "CREATE INDEX ASYNC app.idx ON app.t (a)", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+ assert result.object_name == "app.idx"++ def test_unnamed_index_auto_named(self):+ result = pgproxy.validate(+ "CREATE INDEX ASYNC ON public.t (a, b)", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+ assert result.object_name == "public.t_a_b_idx"++ def test_unnamed_index_default_schema(self):+ result = pgproxy.validate(+ "CREATE INDEX ASYNC ON t (email)", pgproxy.TxnState()+ )+ assert isinstance(result, pgproxy.Rewrite)+ assert result.object_name == "public.t_email_idx"++ def test_plain_create_index_not_rewritten(self):+ result = pgproxy.validate(+ "CREATE INDEX idx ON t (a)", pgproxy.TxnState()+ )+ assert result is None+ assert pgproxy.is_plain_create_index("CREATE INDEX idx ON t (a)")+ assert not pgproxy.is_plain_create_index("CREATE INDEX ASYNC idx ON t (a)")+++class TestTxnDiscipline:+ def test_second_ddl_in_txn_rejected(self):+ txn = pgproxy.TxnState()+ txn.apply("BEGIN")+ txn.apply("CREATE TABLE a (x int)")+ err = pgproxy.validate("CREATE TABLE b (x int)", txn)+ assert isinstance(err, pgproxy.DsqlError)+ assert err.sqlstate == "25006"+ assert "only one DDL" in err.message++ def test_ddl_after_dml_rejected(self):+ txn = pgproxy.TxnState()+ txn.apply("BEGIN")+ txn.apply("INSERT INTO t VALUES (1)")+ err = pgproxy.validate("CREATE TABLE b (x int)", txn)+ assert isinstance(err, pgproxy.DsqlError)+ assert err.sqlstate == "25006"+ assert "cannot be mixed" in err.message++ def test_dml_after_ddl_rejected(self):+ txn = pgproxy.TxnState()+ txn.apply("BEGIN")+ txn.apply("CREATE TABLE a (x int)")+ err = pgproxy.validate("INSERT INTO a VALUES (1)", txn)+ assert isinstance(err, pgproxy.DsqlError)+ assert err.sqlstate == "25006"+ assert "cannot be mixed" in err.message++ def test_commit_resets_state(self):+ txn = pgproxy.TxnState()+ txn.apply("BEGIN")+ txn.apply("CREATE TABLE a (x int)")+ txn.apply("COMMIT")+ assert pgproxy.validate("CREATE TABLE b (x int)", txn) is None++ def test_autocommit_sequential_ddl_fine(self):+ txn = pgproxy.TxnState()+ assert pgproxy.validate("CREATE TABLE a (x int)", txn) is None+ txn.apply("CREATE TABLE a (x int)")+ assert pgproxy.validate("CREATE TABLE b (x int)", txn) is None++ def test_select_insert_outside_txn_fine(self):+ txn = pgproxy.TxnState()+ assert pgproxy.validate("SELECT 1", txn) is None+ assert pgproxy.validate("INSERT INTO t VALUES (1)", txn) is None+++class TestTxnLimits:+ def test_single_insert_over_3000_rows_rejected(self):+ sql = "INSERT INTO t VALUES " + ",".join(f"({i})" for i in range(3001))+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError)+ assert err.sqlstate == "25006"+ assert "3,000" in err.message++ def test_exactly_3000_rows_allowed(self):+ sql = "INSERT INTO t VALUES " + ",".join(f"({i})" for i in range(3000))+ assert pgproxy.validate(sql, pgproxy.TxnState()) is None++ def test_cumulative_rows_across_txn(self):+ txn = pgproxy.TxnState()+ txn.apply("BEGIN")+ batch = "INSERT INTO t VALUES " + ",".join(f"({i})" for i in range(2000))+ assert pgproxy.validate(batch, txn) is None+ txn.apply(batch)+ err = pgproxy.validate(batch, txn) # 2000 + 2000 > 3000+ assert isinstance(err, pgproxy.DsqlError)+ assert "3,000" in err.message++ def test_oversized_statement_rejected(self):+ sql = "INSERT INTO t VALUES ('" + "x" * (10 * 1024 * 1024 + 1) + "')"+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError)+ assert "10 MiB" in err.message++ def test_duration_limit(self):+ txn = pgproxy.TxnState()+ txn.apply("BEGIN")+ txn.started_at -= 301 # simulate a 5+ minute old transaction+ err = pgproxy.validate("INSERT INTO t VALUES (1)", txn)+ assert isinstance(err, pgproxy.DsqlError)+ assert "5 minute" in err.message++ def test_values_tuple_count(self):+ assert pgproxy._values_tuple_count("INSERT INTO t VALUES (1, '(a'), (2)") == 2+ assert pgproxy._values_tuple_count("INSERT INTO t SELECT * FROM s") == 0+ assert pgproxy._values_tuple_count("UPDATE t SET a = 1") == 0+++# ---------------------------------------------------------------------------+# Live proxy tests (need Docker — the backend is a real postgres container)+# ---------------------------------------------------------------------------+++def _docker_daemon_available():+ try:+ import docker++ docker.from_env().ping()+ return True+ except Exception:+ return False+++requires_docker = pytest.mark.skipif(+ not _docker_daemon_available(), reason="Docker daemon not available"+)+++def _free_port():+ with socket.socket() as s:+ s.bind(("127.0.0.1", 0))+ return s.getsockname()[1]++[email protected](scope="module")+def pg_backend():+ """Run a trust-auth postgres container on an ephemeral published port."""+ docker = pytest.importorskip("docker")+ psycopg2 = pytest.importorskip("psycopg2")+ client = docker.from_env()+ container = client.containers.run(+ image="postgres:16-alpine",+ detach=True,+ environment={+ "POSTGRES_USER": "postgres",+ "POSTGRES_DB": "postgres",+ "POSTGRES_HOST_AUTH_METHOD": "trust",+ },+ ports={"5432/tcp": None}, # docker-assigned host port+ )+ try:+ container.reload()+ port = int(+ container.attrs["NetworkSettings"]["Ports"]["5432/tcp"][0]["HostPort"]+ )+ deadline = time.time() + 60+ while time.time() < deadline:+ try:+ conn = psycopg2.connect(+ host="127.0.0.1",+ port=port,+ user="postgres",+ dbname="postgres",+ connect_timeout=2,+ )+ conn.close()+ break+ except psycopg2.OperationalError:+ time.sleep(0.5)+ else:+ pytest.fail(f"postgres container not ready on port {port}")+ yield "127.0.0.1", port+ finally:+ container.remove(force=True, v=True)++[email protected](scope="module")+def dsql_proxy(pg_backend):+ """Run a pgproxy in a background event loop in front of pg_backend."""+ loop = asyncio.new_event_loop()+ thread = threading.Thread(target=loop.run_forever, daemon=True)+ thread.start()+ proxy_port = _free_port()+ asyncio.run_coroutine_threadsafe(+ pgproxy.start_proxy("livetest", proxy_port, pg_backend[0], pg_backend[1]),+ loop,+ ).result(timeout=10)+ yield proxy_port+ asyncio.run_coroutine_threadsafe(pgproxy.stop_all_proxies(), loop).result(timeout=10)+ loop.call_soon_threadsafe(loop.stop)+ thread.join(timeout=5)+++def _pg_connect(port, autocommit=True):+ psycopg2 = pytest.importorskip("psycopg2")+ # Any user/password must be accepted (IAM-token posture).+ conn = psycopg2.connect(+ host="127.0.0.1",+ port=port,+ user="admin",+ password="anything",+ dbname="postgres",+ connect_timeout=10,+ )+ conn.autocommit = autocommit+ return conn+++@requires_docker+class TestContainersE2E:+ """End-to-end for DSQL_CONTAINERS=1: _create_cluster spins up a real+ Postgres container behind the wire proxy, reachable over SQL. Runs+ in-process (flag monkeypatched on) wherever a Docker daemon exists."""++ def test_env_flag_spins_up_real_backend(self, monkeypatch):+ import json++ psycopg2 = pytest.importorskip("psycopg2")+ pytest.importorskip("docker")++ from ministack.services import dsql as dsql_mod++ monkeypatch.setattr(dsql_mod, "DSQL_CONTAINERS", True)+ # _start_backend stashes the test's ephemeral loop in this global;+ # put the original back afterwards (all consumers guard is_running(),+ # but don't leak a closed loop into other tests).+ old_main_loop = dsql_mod._main_loop++ async def _run():+ status, _, body = dsql_mod._create_cluster({})+ assert status == 200+ cluster = json.loads(body)+ identifier = cluster["identifier"]+ try:+ # Backend container + proxy spin up in the background.+ assert cluster["status"] == "CREATING"+ deadline = time.time() + 120+ while dsql_mod._clusters[identifier]["status"] != "ACTIVE":+ assert time.time() < deadline, "backend did not start in time"+ await asyncio.sleep(0.5)+ assert dsql_mod._clusters[identifier]["_has_backend"] is True++ def _select_one():+ conn = psycopg2.connect(+ host="127.0.0.1",+ port=cluster["port"],+ user="admin",+ password="anything",+ dbname="postgres",+ connect_timeout=10,+ )+ try:+ conn.autocommit = True+ with conn.cursor() as cur:+ cur.execute("SELECT 1")+ return cur.fetchone()+ finally:+ conn.close()++ assert await asyncio.to_thread(_select_one) == (1,)+ finally:+ await dsql_mod._teardown_backend(identifier)+ dsql_mod._clusters.pop(identifier, None)++ try:+ asyncio.run(_run())+ finally:+ dsql_mod._main_loop = old_main_loop+++class _WireClient:+ """Raw Postgres wire-protocol client.++ psycopg2 interpolates parameters client-side and always sends the simple+ 'Q' protocol, so it cannot reach the extended (Parse/Bind/Execute) path+ that pgjdbc, pgx, asyncpg and psycopg3 use. These tests speak the protocol+ directly so both paths are covered.+ """++ def __init__(self, port, user="admin", database="postgres"):+ self.sock = socket.create_connection(("127.0.0.1", port), timeout=15)+ params = f"user\0{user}\0database\0{database}\0\0".encode()+ payload = struct.pack("!I", 196608) + params+ self.sock.sendall(struct.pack("!I", len(payload) + 4) + payload)+ self._until_ready()++ def __enter__(self):+ return self++ def __exit__(self, *exc):+ self.close()++ def _recv(self, n):+ buf = b""+ while len(buf) < n:+ chunk = self.sock.recv(n - len(buf))+ if not chunk:+ raise ConnectionError("proxy closed the connection")+ buf += chunk+ return buf++ def _until_ready(self):+ frames = []+ while True:+ type_byte = self._recv(1)+ (length,) = struct.unpack("!I", self._recv(4))+ payload = self._recv(length - 4)+ frames.append((type_byte, payload))+ if type_byte == b"Z":+ return frames++ @staticmethod+ def _frame(type_byte, payload):+ return type_byte + struct.pack("!I", len(payload) + 4) + payload++ def simple(self, sql):+ """Simple query protocol — what psql and psycopg2 send."""+ self.sock.sendall(self._frame(b"Q", sql.encode() + b"\0"))+ return _WireResult(self._until_ready())++ def extended(self, sql):+ """Parse/Bind/Describe/Execute/Sync — what prepared statements send."""+ msg = self._frame(b"P", b"\0" + sql.encode() + b"\0" + struct.pack("!H", 0))+ msg += self._frame(b"B", b"\0\0" + struct.pack("!HHH", 0, 0, 0))+ msg += self._frame(b"D", b"P\0")+ msg += self._frame(b"E", b"\0" + struct.pack("!I", 0))+ msg += self._frame(b"S", b"")+ self.sock.sendall(msg)+ return _WireResult(self._until_ready())++ def close(self):+ try:+ self.sock.sendall(self._frame(b"X", b""))+ except OSError:+ pass+ self.sock.close()+++class _WireResult:+ """Decoded response: error state, rows, command tag, txn status byte."""++ def __init__(self, frames):+ self.sqlstate = None+ self.message = None+ self.rows = []+ self.tag = None+ self.txn_status = None+ for type_byte, payload in frames:+ if type_byte == b"E":+ for part in payload.split(b"\0"):+ if part[:1] == b"C":+ self.sqlstate = part[1:].decode()+ elif part[:1] == b"M":+ self.message = part[1:].decode("utf-8", "replace")+ elif type_byte == b"C":+ self.tag = payload[:-1].decode("utf-8", "replace")+ elif type_byte == b"D":+ (n,) = struct.unpack("!H", payload[:2])+ values, off = [], 2+ for _ in range(n):+ (vlen,) = struct.unpack("!i", payload[off:off + 4])+ off += 4+ if vlen < 0:+ values.append(None)+ else:+ values.append(payload[off:off + vlen].decode("utf-8", "replace"))+ off += vlen+ self.rows.append(tuple(values))+ elif type_byte == b"Z":+ self.txn_status = payload.decode()++ @property+ def ok(self):+ return self.sqlstate is None+++@requires_docker+class TestExtendedProtocol:+ """The extended protocol must enforce the same DSQL subset as 'Q'.++ Relaying Parse unvalidated let every prepared-statement client — pgjdbc,+ pgx, asyncpg, psycopg3, and the ORMs on top of them — past the validator.+ """++ @pytest.mark.parametrize(+ "sql,expected",+ [+ ("CREATE TABLE xp_serial (id serial)", "serial"),+ ("CREATE TABLE xp_fk (id int, p int REFERENCES xp_serial (id))",+ "foreign key"),+ ("CREATE EXTENSION pgcrypto", "CREATE EXTENSION"),+ ("CREATE TEMP TABLE xp_tmp (id int)", "temporary"),+ ("TRUNCATE xp_base", "TRUNCATE"),+ ("CREATE MATERIALIZED VIEW xp_mv AS SELECT 1", "materialized view"),+ ("CREATE TABLE xp_part (id int) PARTITION BY RANGE (id)",+ "partitioning"),+ ("CREATE TABLE xp_bad (id int, ts money)", "money"),+ ],+ )+ def test_unsupported_sql_is_rejected_over_parse(self, dsql_proxy, sql, expected):+ with _WireClient(dsql_proxy) as c:+ result = c.extended(sql)+ assert not result.ok, f"{sql} was accepted over the extended protocol"+ assert result.sqlstate == "0A000"+ assert expected in result.message++ def test_rejected_parse_leaves_the_connection_usable(self, dsql_proxy):+ """After an error the client must be able to keep working (the proxy+ has to honour skip-until-Sync, then answer the Sync)."""+ with _WireClient(dsql_proxy) as c:+ assert not c.extended("CREATE TABLE xp_reuse (id serial)").ok+ result = c.extended("SELECT 1")+ assert result.ok and result.rows == [("1",)]++ def test_valid_ddl_still_passes_over_parse(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ assert c.extended("CREATE TABLE xp_good (id bigint, name text)").ok+ assert c.extended("INSERT INTO xp_good VALUES (1, 'a')").ok+ result = c.extended("SELECT name FROM xp_good WHERE id = 1")+ assert result.rows == [("a",)]++ def test_transaction_limits_apply_over_parse(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE xp_rows (id int)")+ big = "INSERT INTO xp_rows VALUES " + ", ".join(+ f"({i})" for i in range(3100)+ )+ result = c.extended(big)+ assert result.sqlstate == "25006"+ assert "3,000 row" in result.message++ def test_for_update_rules_apply_over_parse(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE xp_lock (id int PRIMARY KEY, v int)")+ result = c.extended("SELECT * FROM xp_lock WHERE id > 1 FOR UPDATE")+ assert result.sqlstate == "0A000"+ assert "equality predicates" in result.message++ def test_create_index_async_returns_a_job_over_parse(self, dsql_proxy):+ """DSQL-only syntax has to be rewritten on this path too, or the+ backend answers with a bare syntax error."""+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE xp_idx (id int, name text)")+ result = c.extended("CREATE INDEX ASYNC xp_i ON xp_idx (name)")+ assert result.ok, result.message+ assert len(result.rows) == 1+ job_id = result.rows[0][0]+ assert ID_RE.match(job_id)+ jobs = c.extended(+ f"SELECT job_id, status FROM sys.jobs WHERE job_id = '{job_id}'"+ )+ assert jobs.rows == [(job_id, "completed")]++ def test_sys_jobs_is_emulated_over_parse(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ result = c.extended("SELECT job_id FROM sys.jobs")+ assert result.ok, result.message++ def test_txn_state_survives_begin_and_commit_over_parse(self, dsql_proxy):+ """Transaction state comes from the backend's ReadyForQuery byte, so a+ BEGIN or COMMIT sent over either protocol keeps the validator honest."""+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE xp_txn (id int)")+ assert c.extended("BEGIN").ok+ assert c.simple("CREATE TABLE xp_txn_a (id int)").ok+ # Second DDL in the same transaction: DSQL allows only one.+ result = c.simple("CREATE TABLE xp_txn_b (id int)")+ assert result.sqlstate == "25006"+ c.simple("ROLLBACK")++ def test_commit_over_parse_does_not_strand_txn_state(self, dsql_proxy):+ """A stranded in_txn made the validator reject valid autocommit DDL —+ a false rejection, worse than a missed one."""+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE xp_strand (id int)")+ assert c.simple("BEGIN").ok+ assert c.extended("INSERT INTO xp_strand VALUES (1)").ok+ assert c.extended("COMMIT").ok+ assert c.simple("CREATE TABLE xp_strand_a (id int)").ok+ assert c.simple("CREATE TABLE xp_strand_b (id int)").ok+ assert c.simple("INSERT INTO xp_strand VALUES (2)").ok++ def test_ready_for_query_status_tracks_the_backend(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ assert c.extended("BEGIN").txn_status == "T"+ # Rejected by the proxy, but the block really is in a transaction.+ assert c.simple("TRUNCATE xp_base").txn_status == "E"+ assert c.simple("ROLLBACK").txn_status == "I"+++@requires_docker+class TestDropColumn:+ """Aurora DSQL gained ALTER TABLE ... DROP COLUMN on 2026-08-03, including+ several columns in one statement, but dropping a primary key column is not+ supported. The rule needs a catalog probe, so it runs in the proxy."""++ def test_drop_non_pk_column(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_a (id int PRIMARY KEY, x int, y int)")+ assert c.simple("ALTER TABLE dc_a DROP COLUMN x").ok+ cols = c.simple(+ "SELECT column_name FROM information_schema.columns "+ "WHERE table_name = 'dc_a' ORDER BY column_name"+ )+ assert cols.rows == [("id",), ("y",)]++ def test_drop_several_non_pk_columns_in_one_statement(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_b (id int PRIMARY KEY, x int, y int, z int)")+ assert c.simple("ALTER TABLE dc_b DROP COLUMN x, DROP COLUMN y").ok+ cols = c.simple(+ "SELECT column_name FROM information_schema.columns "+ "WHERE table_name = 'dc_b' ORDER BY column_name"+ )+ assert cols.rows == [("id",), ("z",)]++ def test_drop_pk_column_rejected(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_c (id int PRIMARY KEY, x int)")+ result = c.simple("ALTER TABLE dc_c DROP COLUMN id")+ assert result.sqlstate == "0A000"+ assert "primary key" in result.message+ # The column must still be there.+ assert c.simple("SELECT id FROM dc_c").ok++ def test_drop_pk_column_rejected_over_extended_protocol(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_d (id int PRIMARY KEY, x int)")+ result = c.extended("ALTER TABLE dc_d DROP COLUMN id")+ assert result.sqlstate == "0A000"+ assert "primary key" in result.message++ def test_pk_column_in_a_multi_drop_rejects_the_whole_statement(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_e (id int PRIMARY KEY, x int)")+ result = c.simple("ALTER TABLE dc_e DROP COLUMN x, DROP COLUMN id")+ assert result.sqlstate == "0A000"+ assert "primary key" in result.message+ # Nothing was dropped — the statement never reached the backend.+ assert c.simple("SELECT id, x FROM dc_e").ok++ def test_composite_pk_columns_are_all_protected(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_f (a int, b int, x int, PRIMARY KEY (a, b))")+ assert c.simple("ALTER TABLE dc_f DROP COLUMN b").sqlstate == "0A000"+ assert c.simple("ALTER TABLE dc_f DROP COLUMN x").ok++ def test_other_drop_actions_are_unaffected(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple(+ "CREATE TABLE dc_g (id int PRIMARY KEY, x int DEFAULT 1 NOT NULL)"+ )+ assert c.simple("ALTER TABLE dc_g ALTER COLUMN x DROP DEFAULT").ok+ assert c.simple("ALTER TABLE dc_g ALTER COLUMN x DROP NOT NULL").ok++ def test_table_without_a_primary_key(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE dc_h (x int, y int)")+ assert c.simple("ALTER TABLE dc_h DROP COLUMN x").ok+++@requires_docker+class TestTransactionAbortSemantics:+ """A statement the proxy rejects must poison the transaction block the way+ a real error does — otherwise the following statements still commit."""++ def test_rejection_aborts_the_block(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE ab_t (id int)")+ assert c.simple("BEGIN").ok+ assert c.simple("INSERT INTO ab_t VALUES (1)").ok+ assert c.simple("TRUNCATE ab_t").sqlstate == "0A000"++ blocked = c.simple("INSERT INTO ab_t VALUES (2)")+ assert blocked.sqlstate == "25P02"+ assert blocked.txn_status == "E"++ # Postgres reports ROLLBACK when a failed block is committed.+ assert c.simple("COMMIT").tag == "ROLLBACK"+ assert c.simple("SELECT count(*) FROM ab_t").rows == [("0",)]++ def test_rollback_clears_the_abort(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE ab_r (id int)")+ c.simple("BEGIN")+ assert c.simple("CREATE TEMP TABLE nope (id int)").sqlstate == "0A000"+ assert c.simple("ROLLBACK").ok+ assert c.simple("INSERT INTO ab_r VALUES (1)").ok+ assert c.simple("SELECT count(*) FROM ab_r").rows == [("1",)]++ def test_abort_applies_to_the_extended_protocol_too(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ c.simple("CREATE TABLE ab_x (id int)")+ c.simple("BEGIN")+ assert c.extended("TRUNCATE ab_x").sqlstate == "0A000"+ assert c.extended("INSERT INTO ab_x VALUES (1)").sqlstate == "25P02"+ assert c.simple("COMMIT").tag == "ROLLBACK"+ assert c.simple("SELECT count(*) FROM ab_x").rows == [("0",)]+++@requires_docker+class TestLiveProxy:+ def test_create_insert_select_round_trip(self, dsql_proxy):+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute(+ "CREATE TABLE rt_live (id int, name text, amt numeric(18,6), "+ "flag boolean, ts timestamp with time zone, u uuid, j jsonb)"+ )+ cur.execute(+ "INSERT INTO rt_live VALUES (1, 'alice', 12.5, true, now(), "+ "'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11', '{\"k\": 1}')"+ )+ cur.execute("SELECT id, name, flag FROM rt_live")+ rows = cur.fetchall()+ assert rows == [(1, "alice", True)]+ cur.execute("DROP TABLE rt_live")+ finally:+ conn.close()++ def test_create_index_async_returns_job(self, dsql_proxy):+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE idx_live (a int)")+ cur.execute("INSERT INTO idx_live VALUES (1)")+ cur.execute("CREATE INDEX ASYNC idx_live_a ON idx_live (a)")+ job_id = cur.fetchone()[0]+ assert re.match(r"^[a-z0-9]{26}$", job_id)++ cur.execute("SELECT job_id, status, job_type FROM sys.jobs")+ jobs = cur.fetchall()+ assert (job_id, "completed", "INDEX_BUILD") in jobs+ finally:+ conn.close()++ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute(+ f"SELECT job_id, status FROM sys.jobs WHERE job_id = '{job_id}'"+ )+ assert cur.fetchall() == [(job_id, "completed")]++ cur.execute(f"SELECT sys.wait_for_job(job_id) '{job_id}'")+ assert cur.fetchone()[0] is True+ cur.execute("SELECT sys.wait_for_job(job_id) 'nonexistent'")+ assert cur.fetchone()[0] is False++ # The rewritten (sync) index really exists on the backend.+ cur.execute(+ "SELECT indexname FROM pg_indexes WHERE tablename = 'idx_live'"+ )+ assert "idx_live_a" in {r[0] for r in cur.fetchall()}+ cur.execute("DROP TABLE idx_live")+ finally:+ conn.close()++ def test_plain_create_index_empty_table_ok(self, dsql_proxy):+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE ci_live (a int)")+ cur.execute("CREATE INDEX ci_live_a ON ci_live (a)")+ cur.execute("DROP TABLE ci_live")+ finally:+ conn.close()++ def test_plain_create_index_nonempty_table_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE cn_live (a int)")+ cur.execute("INSERT INTO cn_live VALUES (1)")+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("CREATE INDEX cn_live_a ON cn_live (a)")+ assert exc.value.pgcode == "0A000"+ # exact message published in the AWS DSQL troubleshooting docs+ assert "use CREATE INDEX ASYNC instead" in str(exc.value)+ cur.execute("DROP TABLE cn_live")+ finally:+ conn.close()++ def test_serial_column_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("CREATE TABLE ser_live (id serial)")+ assert exc.value.pgcode == "0A000"+ assert '"serial"' in str(exc.value)+ finally:+ conn.close()++ def test_create_extension_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("CREATE EXTENSION pgcrypto")+ assert exc.value.pgcode == "0A000"+ assert "is not supported" in str(exc.value)+ finally:+ conn.close()++ def test_foreign_key_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute(+ "CREATE TABLE fk_live (a int REFERENCES other_table (id))"+ )+ assert exc.value.pgcode == "0A000"+ assert "foreign key" in str(exc.value)+ finally:+ conn.close()++ def test_index_form_rejections(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE form_live (a int, data jsonb)")+ for sql, fragment in [+ ("CREATE INDEX fi1 ON form_live USING gin (data)", "btree"),+ ("CREATE INDEX fi2 ON form_live (a) WHERE a > 0", "partial"),+ ("CREATE INDEX fi3 ON form_live (lower(data::text))", "expression"),+ ("CREATE MATERIALIZED VIEW mv_live AS SELECT 1", "materialized"),+ ("CREATE TABLE part_live (a int) PARTITION BY RANGE (a)", "partition"),+ ]:+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute(sql)+ assert exc.value.pgcode == "0A000", sql+ assert fragment in str(exc.value), sql+ cur.execute("DROP TABLE form_live")+ finally:+ conn.close()++ def test_alter_column_type_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE alt_live (a int)")+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("ALTER TABLE alt_live ALTER COLUMN a TYPE text")+ assert exc.value.pgcode == "0A000"+ assert "ALTER COLUMN TYPE" in str(exc.value)+ cur.execute("DROP TABLE alt_live")+ finally:+ conn.close()++ def test_check_constraint_not_valid_and_async_validate(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE chk_live (a int)")+ cur.execute("INSERT INTO chk_live VALUES (1)")+ # CHECK via ALTER TABLE requires NOT VALID+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute(+ "ALTER TABLE chk_live ADD CONSTRAINT chk_a CHECK (a > 0)"+ )+ assert exc.value.pgcode == "0A000"+ assert "NOT VALID" in str(exc.value)+ cur.execute(+ "ALTER TABLE chk_live ADD CONSTRAINT chk_a CHECK (a > 0) NOT VALID"+ )+ # Async validation returns a job_id and registers a job+ cur.execute("ALTER TABLE ASYNC chk_live VALIDATE CONSTRAINT chk_a")+ job_id = cur.fetchone()[0]+ cur.execute(+ "SELECT job_type, status FROM sys.jobs WHERE job_id = %s" % repr(job_id)+ )+ assert cur.fetchone() == ("VALIDATE_CONSTRAINT", "completed")+ # Backend actually validated: violating rows are now rejected+ with pytest.raises(psycopg2.Error):+ cur.execute("INSERT INTO chk_live VALUES (-1)")+ cur.execute("DROP TABLE chk_live")+ finally:+ conn.close()++ def test_max_24_indexes_per_table(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE idx_live (a int)")+ for i in range(24):+ cur.execute(f"CREATE INDEX ASYNC idx_live_{i} ON idx_live (a)")+ cur.fetchone() # consume the job_id row+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("CREATE INDEX ASYNC idx_live_24 ON idx_live (a)")+ assert exc.value.pgcode == "0A000"+ assert "24 indexes" in str(exc.value)+ cur.execute("DROP TABLE idx_live")+ finally:+ conn.close()++ def test_oc001_stale_catalog(self, dsql_proxy):+ """A txn that outlives another session's DDL gets 40001 OC001 once."""+ psycopg2 = pytest.importorskip("psycopg2")+ stale = _pg_connect(dsql_proxy, autocommit=False)+ other = _pg_connect(dsql_proxy)+ try:+ stale.cursor().execute("SELECT 1") # opens txn, caches catalog+ other.cursor().execute("CREATE TABLE oc_live (a int)") # bumps it+ with pytest.raises(psycopg2.Error) as exc:+ stale.cursor().execute("SELECT 1")+ assert exc.value.pgcode == "40001"+ assert "OC001" in str(exc.value)+ # Retry on the same session sees the fresh catalog.+ stale.cursor().execute("SELECT 1")+ other.cursor().execute("DROP TABLE oc_live")+ finally:+ stale.close()+ other.close()++ def test_for_update_rules(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE fu_live (id int primary key, v text)")+ cur.execute("INSERT INTO fu_live VALUES (1, 'x')")+ # equality on the PK: fine+ cur.execute("SELECT * FROM fu_live WHERE id = 1 FOR UPDATE")+ cur.fetchall()+ for sql, fragment in [+ ("SELECT * FROM fu_live FOR UPDATE", "equality predicates"),+ ("SELECT * FROM fu_live WHERE id > 1 FOR UPDATE", "equality predicates"),+ ("SELECT * FROM fu_live WHERE v = 'x' FOR UPDATE", "equality predicates"),+ (+ "SELECT * FROM fu_live a JOIN fu_live b ON a.id = b.id "+ "WHERE a.id = 1 FOR UPDATE",+ "single table",+ ),+ ]:+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute(sql)+ assert exc.value.pgcode == "0A000", sql+ assert fragment in str(exc.value), sql+ cur.execute("DROP TABLE fu_live")+ finally:+ conn.close()++ def test_row_limit_enforced(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE rl_live (a int)")+ big = "INSERT INTO rl_live VALUES " + ",".join(+ f"({i})" for i in range(3001)+ )+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute(big)+ assert exc.value.pgcode == "25006"+ assert "3,000" in str(exc.value)+ cur.execute("SELECT count(*) FROM rl_live")+ assert cur.fetchone()[0] == 0+ cur.execute("DROP TABLE rl_live")+ finally:+ conn.close()++ def test_mixed_ddl_dml_in_txn_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy, autocommit=False)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE tx_live (a int)") # psycopg2 sends BEGIN+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("INSERT INTO tx_live VALUES (1)")+ assert exc.value.pgcode == "25006"+ assert "cannot be mixed" in str(exc.value)+ conn.rollback()+ finally:+ conn.close()+ # Rollback discarded the CREATE TABLE (PG transactional DDL) — the+ # DROP is just a safety net if the backend behaved differently.+ conn = _pg_connect(dsql_proxy)+ try:+ conn.cursor().execute("DROP TABLE IF EXISTS tx_live")+ finally:+ conn.close()++ def test_two_ddl_in_txn_rejected(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy, autocommit=False)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE dd1_live (a int)")+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute("CREATE TABLE dd2_live (a int)")+ assert exc.value.pgcode == "25006"+ assert "only one DDL" in str(exc.value)+ conn.rollback()+ finally:+ conn.close()+ conn = _pg_connect(dsql_proxy)+ try:+ conn.cursor().execute("DROP TABLE IF EXISTS dd1_live")+ finally:+ conn.close()++ def test_autocommit_sequential_ddl_fine(self, dsql_proxy):+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ cur.execute("CREATE TABLE ac1_live (a int)")+ cur.execute("CREATE TABLE ac2_live (a int)")+ cur.execute("DROP TABLE ac1_live")+ cur.execute("DROP TABLE ac2_live")+ finally:+ conn.close()++ def test_multi_statement_batch_txn_rules(self, dsql_proxy):+ psycopg2 = pytest.importorskip("psycopg2")+ conn = _pg_connect(dsql_proxy)+ try:+ cur = conn.cursor()+ with pytest.raises(psycopg2.Error) as exc:+ cur.execute(+ "CREATE TABLE ba1_live (a int); CREATE TABLE ba2_live (a int)"+ )+ assert exc.value.pgcode == "25006"+ # Nothing in the batch was forwarded.+ cur.execute(+ "SELECT tablename FROM pg_tables WHERE tablename LIKE 'ba%_live'"+ )+ assert cur.fetchall() == []+ finally:+ conn.close()+++# ---------------------------------------------------------------------------+# Container lifecycle smoke test (needs a Docker daemon)+# ---------------------------------------------------------------------------+++@requires_docker+def test_cluster_data_plane_end_to_end(dsql):+ """CreateCluster -> poll ACTIVE -> psycopg2 through the endpoint."""+ psycopg2 = pytest.importorskip("psycopg2")+ resp = dsql.create_cluster()+ identifier = resp["identifier"]+ try:+ deadline = time.time() + 180+ cluster = None+ while time.time() < deadline:+ cluster = dsql.get_cluster(identifier=identifier)+ if cluster["status"] == "ACTIVE":+ break+ time.sleep(2)+ else:+ pytest.fail(f"cluster {identifier} not ACTIVE after 180s")++ port = int(cluster["endpoint"].rsplit(":", 1)[1])+ deadline = time.time() + 60+ while True:+ try:+ conn = psycopg2.connect(+ host="127.0.0.1",+ port=port,+ user="admin",+ password="anything",+ dbname="postgres",+ connect_timeout=5,+ )+ break+ except psycopg2.OperationalError:+ if time.time() > deadline:+ raise+ time.sleep(1)+ try:+ conn.autocommit = True+ cur = conn.cursor()+ cur.execute("CREATE TABLE e2e (id int, name text)")+ cur.execute("INSERT INTO e2e VALUES (1, 'x')")+ cur.execute("SELECT name FROM e2e WHERE id = 1")+ assert cur.fetchone()[0] == "x"+ cur.execute("DROP TABLE e2e")+ finally:+ conn.close()+ finally:+ _cleanup(dsql, identifier)