Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(dsql): enforce the locking clause DSQL enforces, not the predicate (#1442)
Edoardo Vivianidd4cd6fb59eb
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4360f240..febff1f5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,6 +14,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).- **CloudFormation — API Gateway (v1) API keys and usage plans** — `AWS::ApiGateway::ApiKey`, `AWS::ApiGateway::UsagePlan`, and `AWS::ApiGateway::UsagePlanKey` failed stack creation with `Unsupported resource type`, even though the runtime `CreateApiKey` / `CreateUsagePlan` APIs were already served. The three resource types now provision through those same stores: an `ApiKey` supports a pinned `Value`, `Enabled`, and `StageKeys`, updates `Name` / `Description` / `Enabled` in place, and exposes both `Ref` and `Fn::GetAtt APIKeyId` as the key id; a `UsagePlan` maps `Throttle` / `Quota` / `ApiStages` and returns its id via `Ref` and `Fn::GetAtt Id`; a `UsagePlanKey` associates a key with a plan and is torn down from the plan on delete. This unblocks CDK `RestApi` / `ApiKey` constructs and Terraform `aws_api_gateway_api_key` deployed through CloudFormation. Contributed by @ryan-bennett.- **AWS IoT Jobs — control plane and device data plane** — `CreateJob` fell through to `Unsupported IoT path` and the `iot-jobs-data` endpoint did not exist, so device OTA loops could not be rehearsed locally. The `iot` service now serves the nine job operations and a new `iot-jobs-data` service the four device ones (incl. the `$next` sentinel), sharing one store and the AWS execution state machine. A `documentSource` job gets a placeholder, not the S3 object. Contributed by @iot-rocket.### Fixed+- **Aurora DSQL — `SELECT ... FOR UPDATE` is no longer refused for its predicate** — strict mode enforced a rule Aurora DSQL does not (any more): a locking read had to be a single table with an equality on every primary key column, or it was rejected with `0A000` "locking clause such as FOR UPDATE can be applied only on tables with equality predicates on the key". The check compared the WHERE clause's column against the key name as raw SQL text, so `WHERE "id" = $1` did not match key column `id` — and every mainstream ORM (Drizzle, Prisma, TypeORM, SQLAlchemy) quotes its identifiers, so in practice every locking read from an ORM failed, with a message that blamed the user's schema. Measured against a live cluster (eu-central-1, 2026-08-18, `server_version` 16.15), DSQL restricts the lock *strength*, not the predicate: `FOR UPDATE` locks whatever the query selects — no `WHERE` clause, a non-key column, an inequality, `IN`/`OR`, a join, or a table with no primary key all succeed, as do `NOWAIT`, `SKIP LOCKED` and `OF` — while `FOR NO KEY UPDATE`, `FOR SHARE` and `FOR KEY SHARE` are refused with `0A000` "locking clauses other than FOR UPDATE/FOR KEY SHARE are not supported", inside a CTE or subquery included. The predicate rule is gone and the lock-strength rule takes its place, so the emulator stops failing correct code and starts failing the code DSQL actually fails. The new rule is token-based (`FOR SHARE` inside a string literal is text, not a clause) and needs no catalog probe, so a locking read no longer costs a round-trip before it is forwarded.+- **Aurora DSQL — quoted identifiers are normalized the way the server stores them** — `ALTER TABLE ... DROP COLUMN` compared the column against the primary key case-insensitively after stripping quotes, so `DROP COLUMN "ID"` was read as key column `id` and refused, while the primary-key probe dropped the quotes from its relation name — `"myschema".t` became `myschema".t` and resolved to nothing, and a table created as `"MyTable"` case-folded to `mytable` and was not found, silently disabling the rule. Identifiers are now folded as PostgreSQL folds them (bare to lower case, quoted verbatim, `""` unescaped) and the relation is requoted part by part before the lookup, so schema-qualified and mixed-case names resolve.- **CloudFormation — `AWS::IoT::Policy` updates apply instead of rolling the stack back** — the type had a create and a delete but no update handler, so a stack update fell through to create, hit `ResourceAlreadyExistsException` on the policy it had provisioned itself, and rolled back with the document unchanged: an IoT policy could be deployed once and never edited again. A changed `PolicyDocument` is now a no-interruption update stored as a new policy version and made the default, so `Ref` keeps naming the same policy and `GetPolicy` returns the new document; a changed `PolicyName` is a replacement, creating the policy under the new name and removing the old one. Because IoT caps a policy at five versions, the update prunes the oldest non-default versions before storing a new document, as the CloudFormation handler does on AWS.- **EC2 — instance public IP and DNS reach the SDKs, and generated addresses are addresses** — `DescribeInstances` and `RunInstances` emitted the public address under `publicIpAddress` / `publicDnsName`, which are not the tags the EC2 wire schema defines (`ipAddress` / `dnsName`, per botocore's `ec2-2016-11-15` model), so every SDK dropped both fields silently and `PublicIpAddress` came back absent on instances that had one. They now ride the real tags. Fixing that exposed a second one: `_random_ip` appended two octets whatever it was given, so a one-octet prefix produced `52.55.218` — `AllocateAddress` has been handing back that shape as `PublicIp` all along, and no address parser accepts it. The generator now completes any prefix to four octets. Contributed by @iot-rocket.- **S3 — versioning edge cases: the null version, delete-marker signals, and versioned copies** — five behaviors around the edges of object versioning now match AWS. A PUT on a Suspended bucket minted a fresh UUID version id like an Enabled one; it is now stored as the `null` version — replacing any previous null version, answered without `x-amz-version-id` — and a suspended DELETE mints its marker under the literal `null` id the same way. The object written before versioning was enabled used to vanish from the index at the first versioned overwrite; it now stays addressable as `VersionId=null` and becomes current again when the newer version is deleted. Delete markers announce themselves: `DeleteObjects` without a per-key VersionId on a versioned bucket now creates a marker (for keys that never existed too) and reports `DeleteMarker`/`DeleteMarkerVersionId`, and a GET/HEAD 404 for a key hidden by a marker carries `x-amz-delete-marker: true` with the marker's id. `UploadPartCopy` ignored the copy source's `?versionId=` and copied the current bytes; it now copies that exact version and echoes `x-amz-copy-source-version-id`. A versioned `CopyObject` dropped the source version's user metadata (the index kept it; the synthesized source object just didn't carry it). And both copy operations decoded the copy-source header before splitting at `?`, so a key legitimately named `?versionId` — sent percent-encoded — read as an empty key with a version qualifier; the raw header is split first now. The five version-minting sites (PutObject, POST, CopyObject, CompleteMultipartUpload, and the delete-marker path) shared five hand-copied bookkeeping blocks; they now go through one helper, which is what makes the null-version rules uniform. Contributed by @gaul.diff --git a/README.md b/README.mdindex bd541781..0d16cd8c 100644--- a/README.md+++ b/README.md@@ -646,7 +646,7 @@ only `DeleteDBCluster` removes the container and its storage.Each `CreateCluster` returns an `ACTIVE` cluster with an endpoint. Set `DSQL_STRICT=1` to additionally spin up a real `postgres:16-alpine` container per cluster (capped by a fixed port window off `DSQL_BASE_PORT`, default 30 ports) behind an in-process Postgres wire-protocol proxy.-**The proxy fails like real DSQL**: SQL that DSQL rejects fails here too (`serial` columns, foreign keys, `CREATE EXTENSION`, `TRUNCATE`, temp tables, `ALTER COLUMN TYPE`, `SET NOT NULL`, materialized views, partitioned tables, ...), index creation is always asynchronous — plain `CREATE INDEX` is refused (`unsupported mode. please use CREATE INDEX ASYNC.`), as are `CONCURRENTLY`, `USING` and partial (`WHERE`) indexes — while expression index keys are supported as long as the functions used are immutable (violations fail with `42P17`, same as the backend; `INCLUDE` columns can't be expressions), `CREATE INDEX ASYNC` and `ALTER TABLE ASYNC ... VALIDATE CONSTRAINT` return a `job_id`, `sys.jobs` / `sys.wait_for_job` are emulated, CHECK constraints via ALTER TABLE require `NOT VALID`, `ALTER TABLE ... DROP COLUMN` works (several columns in one statement included) except on a primary key column, identity columns are bigint-only (CACHE 1 or ≥ 65536), index limits are enforced (8 key columns/index, 24/table), transactions allow only one DDL statement with no DDL+DML mixing, `SELECT ... FOR UPDATE` requires a single table with equality predicates on the key, a transaction that outlives another session's DDL fails once with `40001` (OC001) and succeeds on retry, and the transaction limits (3,000 rows / 10 MiB / 5 min) are enforced.+**The proxy fails like real DSQL**: SQL that DSQL rejects fails here too (`serial` columns, foreign keys, `CREATE EXTENSION`, `TRUNCATE`, temp tables, `ALTER COLUMN TYPE`, `SET NOT NULL`, materialized views, partitioned tables, ...), index creation is always asynchronous — plain `CREATE INDEX` is refused (`unsupported mode. please use CREATE INDEX ASYNC.`), as are `CONCURRENTLY`, `USING` and partial (`WHERE`) indexes — while expression index keys are supported as long as the functions used are immutable (violations fail with `42P17`, same as the backend; `INCLUDE` columns can't be expressions), `CREATE INDEX ASYNC` and `ALTER TABLE ASYNC ... VALIDATE CONSTRAINT` return a `job_id`, `sys.jobs` / `sys.wait_for_job` are emulated, CHECK constraints via ALTER TABLE require `NOT VALID`, `ALTER TABLE ... DROP COLUMN` works (several columns in one statement included) except on a primary key column, identity columns are bigint-only (CACHE 1 or ≥ 65536), index limits are enforced (8 key columns/index, 24/table), transactions allow only one DDL statement with no DDL+DML mixing, `SELECT ... FOR UPDATE` is the only locking clause supported (`FOR SHARE`, `FOR KEY SHARE` and `FOR NO KEY UPDATE` are refused, at any nesting depth; the predicate itself is unrestricted), a transaction that outlives another session's DDL fails once with `40001` (OC001) and succeeds on retry, and the transaction limits (3,000 rows / 10 MiB / 5 min) are enforced.`GetCluster` returns an `endpoint` of the form `localhost:<port>` (ports allocated from `DSQL_BASE_PORT`, default `25432`). Any user/password is accepted — IAM auth tokens from `generate_db_connect_auth_token` work but aren't validated. Connect with e.g. `psql "postgresql://admin@localhost:25432/postgres"` and enter any test password when prompted (no TLS — use `sslmode=disable`/`prefer`; with psycopg2 use `autocommit=True`).diff --git a/ministack/core/pgproxy.py b/ministack/core/pgproxy.pyindex 380ed764..338bba93 100644--- a/ministack/core/pgproxy.py+++ b/ministack/core/pgproxy.py@@ -19,9 +19,16 @@ Documented limitations:backend connection; a client that ``Parse``s such a statement and never``Execute``s it registers no job.- Type/name parsing is heuristic regex work, not a real SQL parser. Statement- splitting, comment stripping and parenthesis matching are lexer-aware- (string literals, dollar quotes, quoted identifiers and nested comments),- but clause-level analysis is still pattern matching.+ splitting, comment stripping, parenthesis matching and the locking-clause+ rule are lexer-aware (string literals, dollar quotes, quoted identifiers and+ nested comments), but the remaining clause-level analysis is still pattern+ matching.+- The locking-clause rule is syntactic: DSQL refuses a share-mode clause when+ the plan would actually take a share lock, so it accepts one that cannot+ reach a relation (``SELECT 1 FOR SHARE``) or that a stronger clause on the+ same relation supersedes (``SELECT * FROM (SELECT id FROM t FOR UPDATE) x+ FOR SHARE``). Both are no-ops; the proxy refuses them anyway rather than+ model the planner's strongest-lock-wins merge.- Transaction row counting is static (``VALUES`` tuples only); ``INSERT ...SELECT`` / ``UPDATE`` / ``DELETE`` affected-row counts are not tracked.- OC001 emulation is optimistic: the catalog version bumps on forwarded DDL,@@ -177,7 +184,7 @@ def _register_job(cluster_id, object_name, job_type="INDEX_BUILD"):# SQL lexing primitives## Everything that scans SQL text (statement splitting, comment stripping,-# parenthesis matching) goes through _skip_noise so that string literals,+# parenthesis matching, tokenizing) goes through _skip_noise so that literals,# dollar-quoted bodies, quoted identifiers and comments can never be mistaken# for structure. Without this a ';' or ')' inside a literal silently truncates# the text a validation rule sees, and the rule passes on the fragment.@@ -297,6 +304,77 @@ def _find_char(sql, i, chars):return -1+# One SQL identifier: double-quoted (case kept, "" escapes a quote) or bare+# (folded to lower case by the server). ``_IDENT_PATH`` covers a dotted name+# in any mix of the two, so schema."Tbl".col is one match.+_IDENT = r'(?:"(?:[^"]|"")*"|[^\W\d][\w$]*)'+_IDENT_PATH = rf"{_IDENT}(?:\s*\.\s*{_IDENT})*"+_IDENT_RE = re.compile(_IDENT)+_NAME_RE = re.compile(r"[^\W\d][\w$]*")+++def fold_identifier(raw):+ """Normalize one identifier to the name the server actually stores.++ A quoted identifier keeps its case verbatim (with ``""`` collapsed to a+ single quote); a bare one folds to lower case. So ``id``, ``ID`` and+ ``"id"`` all name the same column, while ``"ID"`` names a different one.+ Returns ``(name, was_quoted)``.+ """+ raw = raw.strip()+ if len(raw) >= 2 and raw.startswith('"') and raw.endswith('"'):+ return raw[1:-1].replace('""', '"'), True+ return raw.lower(), False+++def identifier_path(raw):+ """Normalized parts of a dotted identifier: ``s."T".c`` -> ``[s, T, c]``."""+ return [fold_identifier(m.group(0))[0] for m in _IDENT_RE.finditer(raw)]+++_OP_CHARS = "+-*/<>=~!@#%^&|?"+_PUNCT_CHARS = "().,;[]:"+++def _sql_tokens(sql):+ """Lex SQL into ``(kind, text)`` pairs, dropping whitespace and comments.++ Kinds: ``name`` (bare identifier or keyword, folded to lower case),+ ``qname`` (quoted identifier, verbatim), ``str`` (string or dollar-quoted+ literal), ``op`` (a maximal run of operator characters, the way the server+ lexes one), ``punct`` (a single ``().,;[]:`` character) and ``other``+ (numbers, ``$1`` placeholders, anything else).+ """+ i, n = 0, len(sql)+ while i < n:+ ch = sql[i]+ if ch.isspace():+ i += 1+ continue+ j = _skip_noise(sql, i)+ if j != i: # literal, quoted identifier or comment+ if ch == '"':+ yield "qname", sql[i + 1 : j - 1].replace('""', '"')+ elif ch in "'$":+ yield "str", sql[i:j]+ i = j+ continue+ m = _NAME_RE.match(sql, i)+ if m:+ yield "name", m.group(0).lower()+ i = m.end()+ continue+ if ch in _OP_CHARS:+ k = i + 1+ while k < n and sql[k] in _OP_CHARS:+ k += 1+ yield "op", sql[i:k]+ i = k+ continue+ yield ("punct" if ch in _PUNCT_CHARS else "other"), ch+ i += 1++# ---------------------------------------------------------------------------# DSQL validator (pure, unit-testable)# ---------------------------------------------------------------------------@@ -1009,6 +1087,45 @@ def match_wait_for_job(sql):return m.group(1) or m.group(2)+# --- Locking clauses (measured against Aurora DSQL, 2026-08-18) -------------++# DSQL supports one locking clause, FOR UPDATE, with its optional+# OF / NOWAIT / SKIP LOCKED tail. It places no restriction on the query the+# clause locks — no WHERE clause, a non-key column, an inequality, IN/OR, a+# join and a table without a primary key all lock fine.+_UNSUPPORTED_LOCKS = (("no", "key", "update"), ("key", "share"), ("share",))+# Cheap gate: only tokenize a statement that could carry one of those.+_LOCK_CLAUSE_RE = re.compile(+ r"\bFOR\s+(?:NO\s+KEY\s+UPDATE|(?:KEY\s+)?SHARE)\b", re.I+)+++def _check_locking_clause(sql):+ """Refuse every locking clause but FOR UPDATE.++ The check reads the statement's tokens, so ``FOR SHARE`` inside a string+ literal is text rather than a clause. A clause nested in a CTE or subquery+ still counts, and an earlier FOR UPDATE does not excuse a later share+ clause — both measured against a live cluster.+ """+ if not _LOCK_CLAUSE_RE.search(sql):+ return None+ toks = list(_sql_tokens(sql))+ for i, (kind, text) in enumerate(toks):+ if kind != "name" or text != "for":+ continue+ words = tuple(t for k, t in toks[i + 1 : i + 4] if k == "name")+ if any(words[: len(lock)] == lock for lock in _UNSUPPORTED_LOCKS):+ # Verbatim from the service, which names FOR KEY SHARE as+ # supported and then refuses it too.+ return DsqlError(+ "0A000",+ "locking clauses other than FOR UPDATE/FOR KEY SHARE "+ "are not supported",+ )+ return None++# --- Top-level validator -----------------------------------------------------@@ -1075,6 +1192,9 @@ def validate(sql, txn_state=None):if err:return errerr = _check_index_rules(s)+ if err:+ return err+ err = _check_locking_clause(s)if err:return err@@ -1285,15 +1405,33 @@ def _abort_gate(conn, sql):return None, sql-async def _primary_key_columns(conn, b_writer, table):- """Primary key column names for ``table``; None if the probe failed."""+def _relation_literal(parts):+ """Quote a normalized name path back into SQL: ``[s, T] -> "s"."T"``.++ Quoting every part keeps the exact case the catalog holds, so a relation+ created as ``"MyTable"`` resolves as itself instead of case-folding away.+ """+ return ".".join('"' + part.replace('"', '""') + '"' for part in parts)+++async def _primary_key_columns(conn, b_writer, relation):+ """Primary key columns of ``relation`` (a normalized name path).++ None when the probe failed or the relation could not be resolved, so the+ caller forwards and lets the backend answer. An unqualified name resolves+ through the backend's search_path, exactly as the client's own statement+ would; a schema-qualified one is looked up in that schema.+ """+ if not relation:+ return None+ literal = _relation_literal(relation).replace("'", "''")try:frames = await _run_backend_capture(conn,b_writer,"SELECT a.attname FROM pg_index i JOIN pg_attribute a ""ON a.attrelid = i.indrelid AND a.attnum = ANY(i.indkey) "- f"WHERE i.indrelid = '{table}'::regclass AND i.indisprimary",+ f"WHERE i.indrelid = '{literal}'::regclass AND i.indisprimary",)except Exception:return None@@ -1315,26 +1453,30 @@ async def _primary_key_columns(conn, b_writer, table):# DROP CONSTRAINT / DEFAULT / NOT NULL / EXPRESSION / IDENTITY — have to be# told apart from a bare column name._DROP_COLUMN_RE = re.compile(- r"\bDROP\s+(?:COLUMN\s+)?(?:IF\s+EXISTS\s+)?([\w\".]+)", re.I+ rf"\bDROP\s+(?:COLUMN\s+)?(?:IF\s+EXISTS\s+)?({_IDENT})", re.I)_DROP_NON_COLUMN_ACTIONS = frozenset(("constraint", "default", "not", "expression", "identity"))_ALTER_TABLE_TARGET_RE = re.compile(- r"\s*ALTER\s+TABLE\s+(?:ASYNC\s+)?(?:IF\s+EXISTS\s+)?(?:ONLY\s+)?([\w\".]+)", re.I+ rf"\s*ALTER\s+TABLE\s+(?:ASYNC\s+)?(?:IF\s+EXISTS\s+)?(?:ONLY\s+)?"+ rf"({_IDENT_PATH})",+ re.I,)def dropped_columns(sql):- """Column names an ALTER TABLE statement drops (may be several)."""+ """Column names an ALTER TABLE statement drops (may be several).++ Names come back normalized the way the server stores them, so they can be+ compared against catalog names directly.+ """if not re.match(r"\s*ALTER\s+TABLE\b", sql, re.I):return []cols = []for m in _DROP_COLUMN_RE.finditer(sql):- raw = m.group(1)- quoted = raw.startswith('"')- name = raw.strip('"')- if not quoted and name.lower() in _DROP_NON_COLUMN_ACTIONS:+ name, quoted = fold_identifier(m.group(1))+ if not quoted and name in _DROP_NON_COLUMN_ACTIONS:continuecols.append(name)return cols@@ -1348,64 +1490,19 @@ async def _check_drop_column(conn, b_writer, sql):m = _ALTER_TABLE_TARGET_RE.match(sql)if not m:return None- pk_cols = await _primary_key_columns(conn, b_writer, m.group(1).strip('"'))+ pk_cols = await _primary_key_columns(+ conn, b_writer, identifier_path(m.group(1))+ )if not pk_cols:return None # probe failed or no primary key — let the backend answer- lowered = {c.lower() for c in pk_cols}+ keys = set(pk_cols)for col in cols:- if col.lower() in lowered:+ if col in keys:# Exact message from real DSQL (verified in eu-west-2, Aug 2026).return DsqlError("0A000", f"cannot drop primary key column {col}")return None-_FOR_UPDATE_MULTI = (- "locking clause such as FOR UPDATE can be applied on a single table"-)-_FOR_UPDATE_EQ = (- "locking clause such as FOR UPDATE can be applied only on tables "- "with equality predicates on the key"-)-_FOR_UPDATE_RE = re.compile(r"\bFOR\s+(?:NO\s+KEY\s+)?UPDATE\b", re.I)---async def _check_for_update(conn, b_writer, sql):- """DSQL locking-clause rules (exact messages from the AWS features doc).-- FOR UPDATE needs a single table and equality predicates on all primary- key columns. Returns a DsqlError, or None to forward.- """- body = _FOR_UPDATE_RE.split(sql)[0]- m = re.search(- r"\bFROM\b(.+?)(?:\bWHERE\b|\bGROUP\b|\bORDER\b|\bLIMIT\b|$)",- body, re.I | re.S,- )- if not m:- return DsqlError("0A000", _FOR_UPDATE_EQ)- from_part = m.group(1)- if "," in from_part or re.search(r"\bJOIN\b", from_part, re.I):- return DsqlError("0A000", _FOR_UPDATE_MULTI)- table = from_part.strip().split()[0].strip('"')- pk_cols = await _primary_key_columns(conn, b_writer, table)- if pk_cols is None:- return None # probe failed — let the backend answer- wm = re.search(r"\bWHERE\b(.+)$", body, re.I | re.S)- if not pk_cols or not wm:- return DsqlError("0A000", _FOR_UPDATE_EQ)- where = wm.group(1)- if re.search(- r"<>|!=|<=|>=|<|>|\bOR\b|\bIN\s*\(|\bBETWEEN\b|\bLIKE\b", where, re.I- ):- return DsqlError("0A000", _FOR_UPDATE_EQ)- for col in pk_cols:- if not (- re.search(rf"\b{re.escape(col)}\b\s*=[^=]", where, re.I)- or re.search(rf"=[^=][^;]*\b{re.escape(col)}\b", where, re.I)- ):- return DsqlError("0A000", _FOR_UPDATE_EQ)- return None--_ASYNC_BATCH_ERR = DsqlError("0A000", "asynchronous DDL is not supported in multi-statement queries")@@ -1473,10 +1570,6 @@ async def _plan_statement(conn, s, b_writer, allow_probe=True):if err:return "error", err- if _FOR_UPDATE_RE.search(s) and allow_probe:- err = await _check_for_update(conn, b_writer, s)- if err:- return "error", errreturn "forward", Nonediff --git a/tests/test_dsql.py b/tests/test_dsql.pyindex c5cfb90e..e8b324e0 100644--- a/tests/test_dsql.py+++ b/tests/test_dsql.py@@ -1198,6 +1198,104 @@ class TestTxnLimits:assert pgproxy._values_tuple_count("UPDATE t SET a = 1") == 0+class TestIdentifierNormalization:+ """``"id"``, ``id`` and ``ID`` all name the same column; ``"ID"`` does not.+ Rules that compare a statement's identifiers against catalog names have to+ normalize the same way the server does, or quoted SQL stops matching."""++ @pytest.mark.parametrize(+ "raw,expected",+ [+ ("id", ("id", False)),+ ("ID", ("id", False)),+ (" id ", ("id", False)),+ ('"id"', ("id", True)),+ ('"ID"', ("ID", True)),+ ('"we ird"', ("we ird", True)),+ ('"quo""ted"', ('quo"ted', True)),+ ],+ )+ def test_fold_identifier(self, raw, expected):+ assert pgproxy.fold_identifier(raw) == expected++ @pytest.mark.parametrize(+ "raw,expected",+ [+ ("id", ["id"]),+ ("t.id", ["t", "id"]),+ ('"T".id', ["T", "id"]),+ ('t."ID"', ["t", "ID"]),+ ('"my schema"."My Table"', ["my schema", "My Table"]),+ ("Schema.Tbl.Col", ["schema", "tbl", "col"]),+ ],+ )+ def test_identifier_path(self, raw, expected):+ assert pgproxy.identifier_path(raw) == expected+++class TestLockingClauses:+ """What Aurora DSQL restricts about a locking read is the lock *strength*,+ not the predicate. Measured against a live cluster (eu-central-1,+ 2026-08-18, server_version 16.15): FOR UPDATE locks whatever the query+ selects — no WHERE clause, a non-key column, an inequality, IN/OR, a join,+ a table with no primary key — while FOR NO KEY UPDATE, FOR SHARE and+ FOR KEY SHARE are all refused with 0A000."""++ MESSAGE = "locking clauses other than FOR UPDATE/FOR KEY SHARE are not supported"++ @pytest.mark.parametrize(+ "sql",+ [+ "SELECT s FROM t WHERE id = '1' FOR UPDATE",+ 'SELECT "id", "s" FROM "t" WHERE "t"."id" = $1 FOR UPDATE',+ "SELECT s FROM t FOR UPDATE",+ "SELECT s FROM t WHERE s = 'a' FOR UPDATE",+ "SELECT s FROM t WHERE id > '1' FOR UPDATE",+ "SELECT s FROM t WHERE id IN (1, 2) FOR UPDATE",+ "SELECT s FROM t WHERE id = 1 OR s = 'a' FOR UPDATE",+ "SELECT a.s FROM t a JOIN t b ON a.id = b.id FOR UPDATE",+ "SELECT s FROM t, u WHERE t.id = u.id FOR UPDATE",+ "SELECT 1 FOR UPDATE",+ "SELECT s FROM t WHERE id = 1 FOR UPDATE NOWAIT",+ "SELECT s FROM t WHERE id = 1 FOR UPDATE SKIP LOCKED",+ "SELECT s FROM t WHERE id = 1 FOR UPDATE OF t NOWAIT",+ "SELECT * FROM (SELECT 1 FOR UPDATE) x FOR UPDATE",+ # The words only as text — not a locking clause at all.+ "SELECT s FROM t WHERE s = 'FOR SHARE'",+ "SELECT 'FOR NO KEY UPDATE' AS lit",+ # FOR introducing something else entirely.+ "DECLARE c CURSOR FOR SELECT 1",+ "CREATE POLICY p ON t FOR UPDATE TO someone",+ ],+ )+ def test_supported_locking_reads_pass(self, sql):+ assert pgproxy.validate(sql, pgproxy.TxnState()) is None, sql++ @pytest.mark.parametrize(+ "sql",+ [+ "SELECT s FROM t WHERE id = 1 FOR SHARE",+ "SELECT s FROM t WHERE id = 1 FOR KEY SHARE",+ "SELECT s FROM t WHERE id = 1 FOR NO KEY UPDATE",+ "SELECT s FROM t for share",+ "SELECT s FROM t FOR SHARE NOWAIT",+ # Any nesting depth: DSQL refuses it inside a CTE or a subquery.+ "WITH x AS (SELECT s FROM t WHERE id = 1 FOR SHARE) SELECT * FROM x",+ "SELECT * FROM (SELECT s FROM t FOR KEY SHARE) y",+ # A supported clause earlier in the statement must not hide one+ # that still reaches a relation (both verified rejected by DSQL).+ "SELECT x.id FROM (SELECT id FROM t FOR UPDATE) x, "+ "(SELECT a FROM two) y FOR SHARE",+ "WITH c AS (SELECT id FROM t FOR UPDATE) SELECT * FROM two FOR SHARE",+ ],+ )+ def test_unsupported_lock_strength_rejected(self, sql):+ err = pgproxy.validate(sql, pgproxy.TxnState())+ assert isinstance(err, pgproxy.DsqlError), sql+ assert err.sqlstate == "0A000", sql+ assert err.message == self.MESSAGE, sql++# ---------------------------------------------------------------------------# Live proxy tests (need Docker — the backend is a real postgres container)# ---------------------------------------------------------------------------@@ -1410,10 +1508,21 @@ class _WireClient:self.sock.sendall(self._frame(b"Q", sql.encode() + b"\0"))return _WireResult(self._until_ready())- def extended(self, sql):- """Parse/Bind/Describe/Execute/Sync — what prepared statements send."""+ def extended(self, sql, params=()):+ """Parse/Bind/Describe/Execute/Sync — what prepared statements send.++ ``params`` are bound as text, so a ``$1`` placeholder reaches the proxy+ as one, the way an ORM sends it."""+ bind = b"\0\0" + struct.pack("!HH", 0, len(params))+ for value in params:+ if value is None:+ bind += struct.pack("!i", -1)+ else:+ raw = str(value).encode()+ bind += struct.pack("!i", len(raw)) + raw+ bind += struct.pack("!H", 0) # result format codesmsg = self._frame(b"P", b"\0" + sql.encode() + b"\0" + struct.pack("!H", 0))- msg += self._frame(b"B", b"\0\0" + struct.pack("!HHH", 0, 0, 0))+ msg += self._frame(b"B", bind)msg += self._frame(b"D", b"P\0")msg += self._frame(b"E", b"\0" + struct.pack("!I", 0))msg += self._frame(b"S", b"")@@ -1521,12 +1630,15 @@ class TestExtendedProtocol:assert result.sqlstate == "25006"assert "3,000 row" in result.message- def test_for_update_rules_apply_over_parse(self, dsql_proxy):+ def test_locking_clause_rules_apply_over_parse(self, dsql_proxy):with _WireClient(dsql_proxy) as c:c.simple("CREATE TABLE xp_lock (id int PRIMARY KEY, v int)")- result = c.extended("SELECT * FROM xp_lock WHERE id > 1 FOR UPDATE")+ # An unsupported lock strength is refused ...+ result = c.extended("SELECT * FROM xp_lock WHERE id = 1 FOR SHARE")assert result.sqlstate == "0A000"- assert "equality predicates" in result.message+ assert "locking clauses other than FOR UPDATE" in result.message+ # ... while FOR UPDATE reaches the backend whatever the predicate.+ assert c.extended("SELECT * FROM xp_lock WHERE v > 1 FOR UPDATE").okdef test_create_index_async_returns_a_job_over_parse(self, dsql_proxy):"""DSQL-only syntax has to be rewritten on this path too, or the@@ -1652,6 +1764,153 @@ class TestDropColumn:assert c.simple("ALTER TABLE dc_h DROP COLUMN x").ok+@requires_docker+class TestLockingReads:+ """A locking read has to reach the backend whatever its predicate looks+ like — an ORM quotes every identifier, and DSQL itself places no+ restriction on the predicate (measured, eu-central-1, 2026-08-18). What it+ does refuse is a lock strength other than FOR UPDATE."""++ UUID = "11111111-1111-1111-1111-111111111111"+ MESSAGE = "locking clauses other than FOR UPDATE/FOR KEY SHARE are not supported"++ @staticmethod+ def _predicates(table, value):+ """The same key equality in every quoting and qualification form."""+ return [+ f"id = {value}",+ f'"id" = {value}',+ f"{table}.id = {value}",+ f'"{table}".id = {value}',+ f'{table}."id" = {value}',+ f'"{table}"."id" = {value}',+ ]++ def _setup(self, client, name, schema=None):+ relation = f"{schema}.{name}" if schema else name+ if schema:+ client.simple(f"CREATE SCHEMA IF NOT EXISTS {schema}")+ client.simple(f"DROP TABLE IF EXISTS {relation}")+ assert client.simple(+ f"CREATE TABLE {relation} (id uuid PRIMARY KEY, s text)"+ ).ok+ assert client.simple(+ f"INSERT INTO {relation} VALUES ('{self.UUID}', 'a')"+ ).ok++ def test_quoted_and_qualified_keys_accepted(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_q")+ for relation in ("fu_q", '"fu_q"', "public.fu_q", '"public"."fu_q"'):+ for predicate in self._predicates("fu_q", f"'{self.UUID}'"):+ sql = f"SELECT s FROM {relation} WHERE {predicate} FOR UPDATE"+ result = c.simple(sql)+ assert result.ok, f"{sql} -> {result.sqlstate} {result.message}"+ assert result.rows == [("a",)], sql++ def test_schema_qualified_relation_accepted(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_s", schema="myschema")+ for relation in ("myschema.fu_s", '"myschema".fu_s',+ 'myschema."fu_s"', '"myschema"."fu_s"'):+ for predicate in self._predicates("fu_s", f"'{self.UUID}'"):+ sql = f"SELECT s FROM {relation} WHERE {predicate} FOR UPDATE"+ result = c.simple(sql)+ assert result.ok, f"{sql} -> {result.sqlstate} {result.message}"+ assert result.rows == [("a",)], sql++ def test_bound_parameter_key_accepted_over_extended_protocol(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_b")+ for predicate in self._predicates("fu_b", "$1"):+ sql = f"SELECT s FROM fu_b WHERE {predicate} FOR UPDATE"+ result = c.extended(sql, params=(self.UUID,))+ assert result.ok, f"{sql} -> {result.sqlstate} {result.message}"+ assert result.rows == [("a",)], sql++ def test_orm_generated_locking_read(self, dsql_proxy):+ """Verbatim shape of a Drizzle ``eq(orders.id, id)`` locking read."""+ with _WireClient(dsql_proxy) as c:+ c.simple("DROP TABLE IF EXISTS orders")+ c.simple(+ 'CREATE TABLE "orders" ('+ '"id" uuid PRIMARY KEY, "order_status" text, "total" int)'+ )+ c.simple(f"INSERT INTO orders VALUES ('{self.UUID}', 'new', 7)")+ sql = (+ 'SELECT "id", "order_status", "total" FROM "orders" '+ 'WHERE "orders"."id" = $1 FOR UPDATE'+ )+ result = c.extended(sql, params=(self.UUID,))+ assert result.ok, f"{result.sqlstate} {result.message}"+ assert result.rows == [(self.UUID, "new", "7")]++ def test_locking_read_inside_a_transaction(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_t")+ assert c.simple("BEGIN").ok+ result = c.simple(+ f'SELECT s FROM "fu_t" WHERE "fu_t"."id" = \'{self.UUID}\' FOR UPDATE'+ )+ assert result.ok, f"{result.sqlstate} {result.message}"+ assert c.simple("COMMIT").ok++ @pytest.mark.parametrize(+ "tail",+ [+ "FOR UPDATE", # no predicate at all+ "WHERE s = 'a' FOR UPDATE", # a non-key column+ "WHERE id > '{uuid}' FOR UPDATE", # not an equality+ "WHERE id IN ('{uuid}') FOR UPDATE",+ "WHERE id = '{uuid}' OR s = 'a' FOR UPDATE",+ "WHERE id = '{uuid}' FOR UPDATE NOWAIT",+ "WHERE id = '{uuid}' FOR UPDATE SKIP LOCKED",+ "WHERE id = '{uuid}' FOR UPDATE OF fu_p",+ ],+ )+ def test_predicate_is_not_restricted(self, dsql_proxy, tail):+ """DSQL locks whatever the query selects — the emulator must not be+ stricter than the service it stands in for."""+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_p")+ sql = f"SELECT s FROM fu_p {tail.format(uuid=self.UUID)}"+ for send in (c.simple, c.extended):+ result = send(sql)+ assert result.ok, f"{sql} -> {result.sqlstate} {result.message}"++ def test_join_and_missing_primary_key_accepted(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_j")+ c.simple("DROP TABLE IF EXISTS fu_nokey")+ c.simple("CREATE TABLE fu_nokey (a int, s text)")+ c.simple("INSERT INTO fu_nokey VALUES (1, 'a')")+ for sql in (+ "SELECT a.s FROM fu_j a JOIN fu_j b ON a.id = b.id FOR UPDATE",+ "SELECT s FROM fu_nokey WHERE a = 1 FOR UPDATE",+ "SELECT s FROM fu_nokey FOR UPDATE",+ ):+ result = c.simple(sql)+ assert result.ok, f"{sql} -> {result.sqlstate} {result.message}"++ @pytest.mark.parametrize(+ "clause", ["FOR SHARE", "FOR KEY SHARE", "FOR NO KEY UPDATE"]+ )+ def test_other_lock_strengths_rejected(self, dsql_proxy, clause):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_l")+ sql = f"SELECT s FROM fu_l WHERE id = '{self.UUID}' {clause}"+ for send in (c.simple, c.extended):+ result = send(sql)+ assert result.sqlstate == "0A000", sql+ assert result.message == self.MESSAGE, sql++ def test_lock_strength_in_a_literal_is_not_a_clause(self, dsql_proxy):+ with _WireClient(dsql_proxy) as c:+ self._setup(c, "fu_lit")+ result = c.simple("SELECT s FROM fu_lit WHERE s = 'FOR SHARE'")+ assert result.ok, f"{result.sqlstate} {result.message}"++@requires_dockerclass TestTransactionAbortSemantics:"""A statement the proxy rejects must poison the transaction block the way@@ -1927,30 +2186,33 @@ class TestLiveProxy:stale.close()other.close()- def test_for_update_rules(self, dsql_proxy):+ def test_locking_clause_rules(self, dsql_proxy):psycopg2 = pytest.importorskip("psycopg2")conn = _pg_connect(dsql_proxy)try:cur = conn.cursor()cur.execute("CREATE TABLE fu_live (id int primary key, v text)")cur.execute("INSERT INTO fu_live VALUES (1, 'x')")- # equality on the PK: fine- cur.execute("SELECT * FROM fu_live WHERE id = 1 FOR UPDATE")- cur.fetchall()- for sql, fragment in [- ("SELECT * FROM fu_live FOR UPDATE", "equality predicates"),- ("SELECT * FROM fu_live WHERE id > 1 FOR UPDATE", "equality predicates"),- ("SELECT * FROM fu_live WHERE v = 'x' FOR UPDATE", "equality predicates"),- (- "SELECT * FROM fu_live a JOIN fu_live b ON a.id = b.id "- "WHERE a.id = 1 FOR UPDATE",- "single table",- ),- ]:+ # FOR UPDATE locks whatever the query selects, as on DSQL.+ for sql in (+ "SELECT * FROM fu_live WHERE id = 1 FOR UPDATE",+ 'SELECT * FROM "fu_live" WHERE "fu_live"."id" = 1 FOR UPDATE',+ "SELECT * FROM fu_live FOR UPDATE",+ "SELECT * FROM fu_live WHERE v = 'x' FOR UPDATE",+ "SELECT * FROM fu_live a JOIN fu_live b ON a.id = b.id FOR UPDATE",+ ):+ cur.execute(sql)+ cur.fetchall()+ # Any other lock strength is not supported.+ for sql in (+ "SELECT * FROM fu_live WHERE id = 1 FOR SHARE",+ "SELECT * FROM fu_live WHERE id = 1 FOR NO KEY UPDATE",+ "SELECT * FROM fu_live WHERE id = 1 FOR KEY SHARE",+ ):with pytest.raises(psycopg2.Error) as exc:cur.execute(sql)assert exc.value.pgcode == "0A000", sql- assert fragment in str(exc.value), sql+ assert "locking clauses other than FOR UPDATE" in str(exc.value), sqlcur.execute("DROP TABLE fu_live")finally:conn.close()