Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Writes keep working after a migration adds a column under a running server (GST-2314)
garrytane5ca8f4cb778
diff --git a/docs/architecture/key-files/persistence-stalls.md b/docs/architecture/key-files/persistence-stalls.mdindex cbf8febb34..07259aaf57 100644--- a/docs/architecture/key-files/persistence-stalls.md+++ b/docs/architecture/key-files/persistence-stalls.md@@ -44,5 +44,5 @@ owner (`owner_wedged_here`), movement surfaces (`data_moving`, doctor- `src/core/persistence/consumer-lane.ts` — #6317 (reporter ask 2): `consumerStatementEngine`, the Proxy that routes the consumer's own tick statements (idle probe, switch read, recovery and expired-claim scans, capacity, releases, floor) to `executeRawDirect` while the dual pool is active (`GBRAIN_DIRECT_DATABASE_URL` or the Supabase-derived direct URL), so the owner's round-trips avoid a transaction-mode pooler; `status().connection = { lane, pooler_mode, direct_host? }` says which lane is in use, and one `[persistence] phase=start reason=transaction_pooler` line prints when only a `prepare=false` URL exists.- `src/core/persistence/consumer-diagnostics.ts` — #6317 (B3, I1): `hostConsumersReport` (the `host.consumers` block of `writer status --json`), `consumerOverlap` (two resident-kind full rows alive over 30 s), `ownersWithoutHeartbeat` (claim owners on running claims with no row) and `hostIdentityMismatches` (this process's `host.json` id differs from a binding owner's while the owner's checkout resolves on this filesystem or the machine id matches), read-only and shared by doctor and writer status.- `src/core/persistence/publication-deadline.ts` (#6288 part 1, #6352; UC2) — what bounds a publish that neither resolves nor rejects. On Postgres the publish transaction is the real bound: `declareDurablePersistence` already sets a transaction-local `statement_timeout` (5 s) and `lock_timeout`, and `declarePublication` (#6712 L6: the opening statement of `publishMutation`'s and `publishGroup`'s transactions, the same `SELECT` as `declareDurablePersistence`, so no extra round trip even where a transaction-mode pooler turns pipelining off) adds a transaction-local `idle_in_transaction_session_timeout` of `persistence.publication_ceiling_ms` (default 300000, 60000 to 3600000, read per engine; never above the pool's own `GBRAIN_IDLE_TX_TIMEOUT`), so a publisher stuck between statements is ended by the server and commits nothing; both survive a transaction pooler that drops startup parameters. Never set on PGLite: it would enforce the bound, and a transaction it ends wedges the engine's only connection. `boundPublication` (the consumer's single path and `executeClaimedGroup`'s group publish and per-member fallback) settles only with the publish's own outcome, so nothing releases or reports a request or group while its transaction can still commit; at the ceiling the claim stamp reads `waiting_on: publication_deadline`, after `PUBLICATION_GRACE_MS` (30 s) it aborts the publication's signal (passed to `engine.transaction(fn, { signal })` through `publicationTransaction`: on Postgres the connection is discarded and nothing commits; the rejection waits for the transaction body to settle, then surfaces with the driver's `CONNECTION_CLOSED` code, so the claim is requeued for a retry; PGLite checks the signal only before BEGIN), and after `PUBLICATION_SETTLE_MS` (5 s) more the consumer lists it in `stuck_publications` and reports `restart_required` (its root stays held: restarting the owner is the recovery, never a promised reclaim). The lease keeps renewing, so a reclaim never races a publisher that may still write files. The wave 9 yield between groups is outside any one bound. `PublicationBound.later` is the scheduler seam (default a real unref'd timer); the unit test drives the three phases through a manual clock so a stalled runner cannot fire a phase early (the consumer's `clock` seam is separate). Tests: `test/persistence-publication-deadline.test.ts` (barriers after the token lock, after the file rename and before commit; the server-side idle end on Postgres; the wiring probe).-- `src/core/persistence/transient-retry.ts` — the retry budget for a managed write's transient interruptions: a statement the database ended with 57014 (`statement_cancel`) or a publication `boundPublication` cancelled at the ceiling (`publication_ceiling`, read from the publication's signal by `publicationDeadlinePassed`, whatever error it surfaces as). `chargeTransientInterruption` counts it in `persistence_requests.transient_interruptions` (v261, never `preparation_attempts`) and sets `next_attempt_at` to 1 s, 2 s, 4 s, 8 s, then 15 s (under the drain's 30 s no-progress window); `BACKOFF_ELAPSED` keeps every claim (`claimableWriteSql`, lane heads, group followers) off the row until then, so no claim is held while it waits, and `readHeadBlockers` names it `transient_backoff`. Past `persistence.max_transient_retries` (default 5, 0 to 20, a preparation-budget key) coordinator.ts ends the request `write_interrupted`; the charge commits with the claim's release. Off with `preparation_deadlines`. Tests: `test/pathological/lp-b-transient-retry-budget.test.ts`, `test/pathological/lp-c-publication-ceiling-retry.test.ts` (`_setPublicationTimingForTest`), `test/persistence-write-interrupted.test.ts`.+- `src/core/persistence/transient-retry.ts` — the retry budget for a managed write's transient interruptions: a statement the database ended with 57014 (`statement_cancel`), a statement in the write's transaction that ran on a plan cached before a migration changed its table (0A000 "cached plan must not change result type", `stale_plan`, matched by `retry-matcher.ts` `isStalePlanError`; GST-2314) or a publication `boundPublication` cancelled at the ceiling (`publication_ceiling`, read from the publication's signal by `publicationDeadlinePassed`, whatever error it surfaces as). `chargeTransientInterruption` counts it in `persistence_requests.transient_interruptions` (v261, never `preparation_attempts`) and sets `next_attempt_at` to 1 s, 2 s, 4 s, 8 s, then 15 s (under the drain's 30 s no-progress window); `BACKOFF_ELAPSED` keeps every claim (`claimableWriteSql`, lane heads, group followers) off the row until then, so no claim is held while it waits, and `readHeadBlockers` names it `transient_backoff`. Past `persistence.max_transient_retries` (default 5, 0 to 20, a preparation-budget key) coordinator.ts ends the request `write_interrupted`; the charge commits with the claim's release. Off with `preparation_deadlines`. Tests: `test/pathological/lp-b-transient-retry-budget.test.ts`, `test/pathological/lp-c-publication-ceiling-retry.test.ts` (`_setPublicationTimingForTest`), `test/persistence-write-interrupted.test.ts`.- `src/core/persistence/sync-discovery.ts` (#6349 staged catch-up) — when an incremental delta (not `--full`, not company) exceeds the cursor bound (100,000 entries, 16 MiB; `_setCursorBoundForTest` lowers it), `stagedSyncTarget` gallops forward over `git rev-list --first-parent last..HEAD`, verifying every candidate's delta from the last synced commit against the bound (a net delta is not monotone, so no binary search), and discovery reruns pinned at the furthest fitting commit with no working-tree entries, recording `stagedHead`; the cursor carries it and `SyncResult.staged { target, head, stages }` reports it. One commit alone above the bound refuses `request_too_large` naming the commit. `commands/sync/perform.ts` loops a drain stage by stage to HEAD (the drain's one extra HEAD-moved pass is skipped for a staged result). Tests: `test/sync-staged-catchup.test.ts`.diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex d1da8f6849..87f6a0ef0a 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -5634,7 +5634,7 @@ Reasons: `override`, `standing_instruction`, `exfiltration`, `credential`, `dete| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| The database kept ending this managed write's statements (SQLSTATE 57014, a statement timeout or cancel), or its publication kept running past persistence.publication_ceiling_ms and was cancelled, after persistence.max_transient_retries retries with backoff (default 5), so the owner finished it failed instead of preparing it again; nothing was published. | Each such interruption throws the write's preparation away. It is load or latency, not the write's content, so it has its own retry budget: the request is claimed again after a backoff (1 s, 2 s, 4 s, 8 s, then 15 s; no claim is held while it waits) up to persistence.max_transient_retries times, and the next interruption ends it instead of preparing it again without limit. It never charges persistence.max_preparation_attempts, which preparations cut off at their deadline keep. Contention with other transactions (lock timeout, serialization failure, deadlock), a dropped connection and a restarting database are never counted. The receipt names the interruptions (attempts), the retry limit, the cause and the SQLSTATE. | Inspect the owner with gbrain sources writer status --source <id> --json (read-only), then submit the write again with a new request_id once the database is less loaded; a held sync file needs gbrain sources retry-held <id> and the same sync. If the same write ends this way again, report its request id with gbrain doctor --json. Run: gbrain sources writer status --source '{source_id}' --json | agent | `repeat the read that failed` | 1 | no |+| The database kept ending this managed write's statements (SQLSTATE 57014, a statement timeout or cancel, or 0A000, a plan cached before a migration changed the table), or its publication kept running past persistence.publication_ceiling_ms and was cancelled, after persistence.max_transient_retries retries with backoff (default 5), so the owner finished it failed instead of preparing it again; nothing was published. | Each such interruption throws the write's preparation away. It is load or latency, not the write's content, so it has its own retry budget: the request is claimed again after a backoff (1 s, 2 s, 4 s, 8 s, then 15 s; no claim is held while it waits) up to persistence.max_transient_retries times, and the next interruption ends it instead of preparing it again without limit. It never charges persistence.max_preparation_attempts, which preparations cut off at their deadline keep. Contention with other transactions (lock timeout, serialization failure, deadlock), a dropped connection and a restarting database are never counted. The receipt names the interruptions (attempts), the retry limit, the cause and the SQLSTATE. | Inspect the owner with gbrain sources writer status --source <id> --json (read-only), then submit the write again with a new request_id once the database is less loaded; a held sync file needs gbrain sources retry-held <id> and the same sync. If the same write ends this way again, report its request id with gbrain doctor --json. Run: gbrain sources writer status --source '{source_id}' --json | agent | `repeat the read that failed` | 1 | no |More: [docs/guides/write-refusals.md#write_interrupted](../../docs/guides/write-refusals.md#write_interrupted)diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 41083194f8..4cc177beeb 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -110,7 +110,7 @@ of making a duplicate.| <a id="content_null_bytes"></a>`content_null_bytes` (sync hold) | `content_null_bytes` | A managed sync held one Markdown file because its text holds NUL bytes (U+0000). Page text is stored as UTF-8 without NUL bytes. The usual cause is a UTF-16 file (a Windows "Unicode" save; the hold says so when the file starts with a UTF-16 byte order mark), a binary file named `.md` or a crash-padded file. Until GBRA-133 the whole managed sync stopped at that file on every run (`storage_error`, SQLSTATE 22P05) and the rest of the source never imported; now only the file waits. With `sync.holds=fail` the run stops with `content_null_bytes` instead of holding. | Save a UTF-16 file as UTF-8, remove the NUL bytes, or remove a binary file (or leave it out with `sync.exclude`), then commit and `gbrain sync --source <source> --no-pull`. The hold re-screens when the file changes. || <a id="git_object_missing"></a>`git_object_missing` (sync hold) | `git_object_missing` | A managed sync held one committed file because the commit names a Git object the checkout no longer has (an interrupted `git gc`, a partial clone without its promisor remote, disk corruption) and the working-tree copy differs from it or is gone. When the working-tree bytes hash to the committed blob id, sync imports them and holds nothing. Until GBRA-133 the whole managed sync stopped at that file on every run. | Run `git fsck --connectivity-only` in the checkout to see what is missing, restore it with `git fetch` (a partial or shallow clone) or from a backup, then `gbrain sync --source <source> --no-pull`. Every sync re-tests the hold. || <a id="path_not_utf8"></a>`path_not_utf8` (sync hold) | `path_not_utf8` | A managed sync held one committed file because its name is not valid UTF-8 (a Latin-1 name from an older system, an archive or a network share). Git hands the name back with U+FFFD for the bytes that are not UTF-8, and no Git read or file matches that decoded name, so nothing is imported from it. Until GBRA-133 the managed sync either stopped at that file on every run or skipped it with no hold; now only the file waits and the hold names it. With `sync.holds=fail` the run stops with `path_not_utf8` instead of holding. | Rename the file to a UTF-8 name (`git -c core.quotePath=true ls-files` shows its exact bytes; `convmv -f latin1 -t utf8` converts a Latin-1 name), commit, then `gbrain sync --source <source> --no-pull`. The rename clears the hold. |-| <a id="write_interrupted"></a>`write_interrupted` | `write_interrupted` | A transient interruption kept ending this managed write: the database ended its own statements (SQLSTATE `57014`, a statement timeout or cancel) at preparation or inside its publication transaction, or its publication ran past `persistence.publication_ceiling_ms` and its grace and the owner cancelled it. Each interruption throws the preparation away, but it is load or latency, not the write's content, so it has its own budget: the request is claimed again after a backoff (1 s, 2 s, 4 s, 8 s, then 15 s, scheduled on the request as `next_attempt_at`; no claim, root lock or connection is held while it waits, and `gbrain sources writer status` names the waiting head `transient_backoff`) up to `persistence.max_transient_retries` times (default 5, 0 to 20), and the next interruption finishes it `failed` instead of preparing it again without limit. Interruptions never spend `persistence.max_preparation_attempts`, which preparations cut off at their deadline keep ([`preparation_stalled`](#preparation_stalled)). Nothing was published: a worktree-backed write's canonical file is restored before the failure is recorded. Contention with other transactions (`55P03` lock timeout, `40001` serialization failure, `40P01` deadlock) is released as `database_contention` uncounted, and a dropped connection or a database that is restarting or out of connections is never counted. The receipt's message and owner-only `error_detail` name the interruptions (`attempts`), the retry limit (`limit`, `limit_key`), the `cause` (`statement_cancel` or `publication_ceiling`) and the SQLSTATE (`sqlstate`, public). A managed sync member is held like [`preparation_stalled`](#write_interrupted-hold): the run moves past it and the rest of the source lands, `gbrain sync status` lists the hold (class `page`, safe action `retry`, [runbook row](sync-unblock-runbook.md)), and `gbrain sources retry-held` re-admits it. A group's shared read the database cancels is charged once, to the first member that settles with it, never to every member; and the charge commits with the claim's release, so a crash between them never counts it twice. With `persistence.preparation_deadlines` off nothing is counted and no backoff applies. | Run `gbrain sources writer status --source <source> --json` (read-only: the owner, its pool, what its writes wait on), then submit the write again with a new `request_id` once the database is less loaded; for a held sync file, run `gbrain sources retry-held <source>` and the same sync with its options. A `publication_ceiling` cause on a page that legitimately takes longer to publish is the user's call: ask before raising `persistence.publication_ceiling_ms` (or the database `statement_timeout` for a `statement_cancel`). If the same write ends this way again, report its request id with `gbrain doctor --json`. Do not raise `persistence.max_transient_retries` to get past it. |+| <a id="write_interrupted"></a>`write_interrupted` | `write_interrupted` | A transient interruption kept ending this managed write: the database ended its own statements (SQLSTATE `57014`, a statement timeout or cancel) at preparation or inside its publication transaction, a statement inside that transaction ran on a plan cached before a migration changed its table (SQLSTATE `0A000`, "cached plan must not change result type"; the next claim prepares it again), or its publication ran past `persistence.publication_ceiling_ms` and its grace and the owner cancelled it. Each interruption throws the preparation away, but it is load or latency, not the write's content, so it has its own budget: the request is claimed again after a backoff (1 s, 2 s, 4 s, 8 s, then 15 s, scheduled on the request as `next_attempt_at`; no claim, root lock or connection is held while it waits, and `gbrain sources writer status` names the waiting head `transient_backoff`) up to `persistence.max_transient_retries` times (default 5, 0 to 20), and the next interruption finishes it `failed` instead of preparing it again without limit. Interruptions never spend `persistence.max_preparation_attempts`, which preparations cut off at their deadline keep ([`preparation_stalled`](#preparation_stalled)). Nothing was published: a worktree-backed write's canonical file is restored before the failure is recorded. Contention with other transactions (`55P03` lock timeout, `40001` serialization failure, `40P01` deadlock) is released as `database_contention` uncounted, and a dropped connection or a database that is restarting or out of connections is never counted. The receipt's message and owner-only `error_detail` name the interruptions (`attempts`), the retry limit (`limit`, `limit_key`), the `cause` (`statement_cancel`, `stale_plan` or `publication_ceiling`) and the SQLSTATE (`sqlstate`, public). A managed sync member is held like [`preparation_stalled`](#write_interrupted-hold): the run moves past it and the rest of the source lands, `gbrain sync status` lists the hold (class `page`, safe action `retry`, [runbook row](sync-unblock-runbook.md)), and `gbrain sources retry-held` re-admits it. A group's shared read the database cancels is charged once, to the first member that settles with it, never to every member; and the charge commits with the claim's release, so a crash between them never counts it twice. With `persistence.preparation_deadlines` off nothing is counted and no backoff applies. | Run `gbrain sources writer status --source <source> --json` (read-only: the owner, its pool, what its writes wait on), then submit the write again with a new `request_id` once the database is less loaded; for a held sync file, run `gbrain sources retry-held <source>` and the same sync with its options. A `publication_ceiling` cause on a page that legitimately takes longer to publish is the user's call: ask before raising `persistence.publication_ceiling_ms` (or the database `statement_timeout` for a `statement_cancel`). If the same write ends this way again, report its request id with `gbrain doctor --json`. Do not raise `persistence.max_transient_retries` to get past it. || <a id="write_outcome_unknown"></a>`write_outcome_unknown` | `write_outcome_unknown` | The database session dropped while the write was being admitted (a pooler reap, a failover, `pg_terminate_backend`), and it kept dropping through the re-reads, so whether the write was accepted is unknown (#6355). A single drop never reaches you: the admission attempt is re-run after 100, 300 and 900 ms, and every attempt begins by reading the retained `request_id`, so a re-run replays an admitted row instead of admitting again. Before this code existed the raw socket error (`write CONNECTION_CLOSED …`, `57P01`) reached the caller while the write went on to commit, and a receipt oracle took it for a refusal. | Read the request first: `gbrain write-request -- <request_id>` or `get_write_request { request_id }` (the error's `fix` names it). A committed or pending row means the write was accepted: replay the same `request_id` to wait for it. No row means it was not: submit it again with the same `request_id`. Never resubmit under a new `request_id` without that read. || <a id="connection_inherited_read_only"></a>`connection_inherited_read_only` | `connection_inherited_read_only` | The database refused the write as read-only (SQLSTATE `25006`): the server connection defaults to read-only transactions (a session-level `SET` another client left on a pooled connection, a role or database default, or a read-only server). Nothing was written. A managed sync stops the run with it (systemic). | Ask the database operator to reset the pooler's server connections, stop the client issuing session-level `SET` through the pooler and check `ALTER ROLE` / `ALTER DATABASE` defaults; then submit the write again. || <a id="page_body_segmented"></a>`page_body_segmented` | `page_body_segmented` (a memory verb: `invalid_params` with `write_error: page_body_segmented`) | The target page stores its body in segments (a page over 1 MiB written while `pages.segmented_storage` was on), and this write needs the whole body as one string: `remember` against an entity page over 24 MiB (named, or inferred from the fact), and the other whole-body writers listed under [page_body_segmented](windowed-reads.md#page_body_segmented). It refused before changing anything and is not retryable. | For `remember`, save the claim without an entity (`infer_entity: false`); remembering it with `entity` once that page accepts fact rows replaces that copy. Otherwise edit the page's file and sync it, or read it with `get_page` `body: "auto"`. |diff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex fabcf5feb8..be45539c63 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -902,7 +902,7 @@ export const CODES = {write_gate_rejected: { class: 'caller', summary: "The write gate refused external instruction-like content because the operator set `write_gate.external_mode` to `reject`.", why: "Text from an untrusted source that reads like an instruction to an agent (an override, a standing rule, an exfiltration or credential request) is refused instead of held under that setting, so nothing was written. The same content refuses on every retry.", reasons: ['override', 'standing_instruction', 'exfiltration', 'credential', 'detector_error'], suggestion: 'Tell the user the content was refused; whether to accept it or switch `write_gate.external_mode` back to `quarantine` is their decision.' },write_held: { class: 'consent', summary: "The write gate held this content for the owner's review instead of saving it as memory.", why: "Content from an untrusted source read like an instruction to an agent, so it was quarantined in the write-gate holding table. It is not searchable or recalled until the owner releases it; retrying the same write only re-opens the same hold.", reasons: ['override', 'standing_instruction', 'exfiltration', 'credential', 'detector_error'], actor: 'user', exit: 3, suggestion: 'Do not retry. Tell the user what was held and give them the release command from the fix; releasing is their decision.' },write_interrupted: { class: 'retryable', retryable: false, actor: 'agent',- summary: "The database kept ending this managed write's statements (SQLSTATE 57014, a statement timeout or cancel), or its publication kept running past persistence.publication_ceiling_ms and was cancelled, after persistence.max_transient_retries retries with backoff (default 5), so the owner finished it failed instead of preparing it again; nothing was published.",+ summary: "The database kept ending this managed write's statements (SQLSTATE 57014, a statement timeout or cancel, or 0A000, a plan cached before a migration changed the table), or its publication kept running past persistence.publication_ceiling_ms and was cancelled, after persistence.max_transient_retries retries with backoff (default 5), so the owner finished it failed instead of preparing it again; nothing was published.",why: "Each such interruption throws the write's preparation away. It is load or latency, not the write's content, so it has its own retry budget: the request is claimed again after a backoff (1 s, 2 s, 4 s, 8 s, then 15 s; no claim is held while it waits) up to persistence.max_transient_retries times, and the next interruption ends it instead of preparing it again without limit. It never charges persistence.max_preparation_attempts, which preparations cut off at their deadline keep. Contention with other transactions (lock timeout, serialization failure, deadlock), a dropped connection and a restarting database are never counted. The receipt names the interruptions (attempts), the retry limit, the cause and the SQLSTATE.",docs: 'docs/guides/write-refusals.md#write_interrupted',suggestion: 'Inspect the owner with gbrain sources writer status --source <id> --json (read-only), then submit the write again with a new request_id once the database is less loaded; a held sync file needs gbrain sources retry-held <id> and the same sync. If the same write ends this way again, report its request id with gbrain doctor --json.',diff --git a/src/core/grants/model.ts b/src/core/grants/model.tsindex 512af3ed02..bdacf4c220 100644--- a/src/core/grants/model.ts+++ b/src/core/grants/model.ts@@ -244,7 +244,7 @@ function columnList(value: unknown): string[] | null {}/** The grant the unified columns carry; an inconsistent source state reads as `none`. */-export function tokenGrantFromColumns(row: Record<string, unknown>): TokenGrantAxes {+export function tokenGrantFromColumns(row: TokenGrantRow): TokenGrantAxes {const write = typeof row.source_id === 'string' && row.source_id.length > 0 ? row.source_id : null;const reads = columnList(row.federated_read);let sources: GrantSources = { kind: 'none' };@@ -268,14 +268,15 @@ function sameAxis(axis: LegacyGrantAxis, a: TokenGrantAxes, b: TokenGrantAxes):/*** The effective grant of an `access_tokens` row (SELECT * keeps this working- * on brains that predate the columns). Unified rows read the columns, and any+ * on brains that predate the columns; a reader inside a transaction selects+ * `TOKEN_GRANT_ROW_COLUMNS` instead). Unified rows read the columns, and any* axis whose JSONB mirror disagrees evaluates deny-all. A `source_grant IS* NULL` row has not been converted yet (the bulk migration converts every* active one; an older binary's `auth create` can still add one): it reads as* the conversion `migrateLegacyTokens` would write, and a malformed* `permissions` value denies every axis.*/-export function grantFromTokenRow(row: Record<string, unknown>): PrincipalGrant {+export function grantFromTokenRow(row: TokenGrantRow): PrincipalGrant {const base = {principal: { kind: 'legacy_token' as const, id: String(row.id) },scopes: normalizeTokenScopes(row.scopes) ?? ['read', 'write', 'admin'],@@ -390,3 +391,11 @@ export function tokenGrantColumnValues(g: Pick<PrincipalGrant, 'sources' | 'allo];}export const TOKEN_GRANT_COLUMNS = ['source_grant', 'source_id', 'federated_read', 'allowed_operations', 'takes_holders'] as const;+/**+ * GST-2314: every access_tokens column `grantFromTokenRow` reads (its row type allows no other). A reader inside a+ * transaction selects these by name: a prepared SELECT * whose plan was cached before a migration added a column+ * fails there with 0A000 ("cached plan must not change result type") until the connection re-prepares it.+ */+export const TOKEN_GRANT_ROW_COLUMNS = ['id', 'scopes', 'permissions', 'private_pages', 'grant_revision', ...TOKEN_GRANT_COLUMNS] as const;+export const TOKEN_GRANT_ROW_SELECT = TOKEN_GRANT_ROW_COLUMNS.join(',');+export type TokenGrantRow = Partial<Record<(typeof TOKEN_GRANT_ROW_COLUMNS)[number], unknown>>;diff --git a/src/core/persistence/admission-retry.ts b/src/core/persistence/admission-retry.tsindex a302c15c37..4bf57e47cf 100644--- a/src/core/persistence/admission-retry.ts+++ b/src/core/persistence/admission-retry.ts@@ -1,7 +1,7 @@import { AsyncLocalStorage } from 'node:async_hooks';import { setTimeout as delay } from 'node:timers/promises';import { markNotAdmitted, OperationError, opError } from '../ops/contract.ts';-import { getCode, isConnectionLoss } from '../retry-matcher.ts';+import { getCode, isConnectionLoss, isStalePlanError } from '../retry-matcher.ts';/** #6355: how many times an admission attempt is re-run after the session dropped under it, and the pauses before each re-run. */export const CONNECTION_LOSS_RETRY_MS = [100, 300, 900] as const;@@ -50,7 +50,7 @@ export async function retryWriteAdmission<T>(requestId: string, attempt: (remainawait recover?.(error).catch(() => undefined);continue;}- if (!['40001', '40P01', '55P03', '57014'].includes(code ?? '')) throw error;+ if (!['40001', '40P01', '55P03', '57014'].includes(code ?? '') && !isStalePlanError(error)) throw error;const remaining = deadline - performance.now();if (remaining <= 25) {const unavailable = new OperationError('storage_error', 'Write admission is temporarily blocked by database contention.',diff --git a/src/core/persistence/authority.ts b/src/core/persistence/authority.tsindex 19be131e30..47b5da3758 100644--- a/src/core/persistence/authority.ts+++ b/src/core/persistence/authority.ts@@ -6,7 +6,7 @@ import { slugUnderBoundPrefixes, matchesSlugAllowList, noSourceGrantError } fromimport { NO_SOURCES } from '../source-id.ts';import { hasScope } from '../scope.ts';import { normalizeTokenScopes } from '../legacy-token-scope.ts';-import { authSourcesFromGrant, grantFromTokenRow } from '../grants/model.ts';+import { authSourcesFromGrant, grantFromTokenRow, TOKEN_GRANT_ROW_SELECT, type TokenGrantRow } from '../grants/model.ts';import { readLocalWriter, currentVerifiedLocalWriter, takeRegisteredWriterRow, verifyLocalWriter, type LocalGrant } from './identity.ts';import type { Principal, SqlEngine, WriteAuthority, WriteRequest } from './model.ts';import { authorizePageVisibility, excludesPrivateWritesAtAdmission } from './page-visibility.ts';@@ -180,7 +180,8 @@ export async function authorizeWrite(engine: SqlEngine, a: WriteAuthority, operareturn;}if (a.principal.kind === 'legacy_token') {- const [row] = await engine.executeRaw<Record<string, unknown>>(`SELECT * FROM access_tokens WHERE id=$1${suffix}`, [a.principal.id]);+ // GST-2314: named columns, not SELECT *: inside this transaction a plan cached before a migration added a column fails 0A000.+ const [row] = await engine.executeRaw<TokenGrantRow & { revoked_at: unknown }>(`SELECT ${TOKEN_GRANT_ROW_SELECT},revoked_at FROM access_tokens WHERE id=$1${suffix}`, [a.principal.id]);if (!row || row.revoked_at != null) deny('The owning token is revoked.');const grant = grantFromTokenRow(row);if (grant.permissionsMalformed) deny('The current legacy grant is malformed.');diff --git a/src/core/persistence/coordinator.ts b/src/core/persistence/coordinator.tsindex ba2272a257..10e3c3b906 100644--- a/src/core/persistence/coordinator.ts+++ b/src/core/persistence/coordinator.ts@@ -282,7 +282,9 @@ function writeInterruptedFailure(row: WriteRequest, interruption: TransientInter: 'Submit the write again with a new request_id once the database is less loaded.';const what = interruption.cause === 'publication_ceiling'? `Publication of this write ran past persistence.publication_ceiling_ms and was cancelled ${attempts} time(s)${interruption.sqlstate ? ` (SQLSTATE ${interruption.sqlstate})` : ''}`- : `The database ended this write's statements ${attempts} time(s) (SQLSTATE ${interruption.sqlstate}, a statement timeout or cancel)`;+ : interruption.cause === 'stale_plan'+ ? `This write's statements ran on a plan cached before a schema change ${attempts} time(s) (SQLSTATE 0A000)`+ : `The database ended this write's statements ${attempts} time(s) (SQLSTATE ${interruption.sqlstate}, a statement timeout or cancel)`;return { code: 'write_interrupted',message: `${what}; it was retried with backoff ${limit} time(s), the limit of ${TRANSIENT_RETRIES_KEY} (${limit}), and is not prepared again. `+ `Run gbrain sources writer status --source ${row.source_id} --json to see the load on the owner. ${fix}`,diff --git a/src/core/persistence/transient-retry.ts b/src/core/persistence/transient-retry.tsindex 63cc60e4df..93e1263d8e 100644--- a/src/core/persistence/transient-retry.ts+++ b/src/core/persistence/transient-retry.ts@@ -6,7 +6,10 @@* - `statement_cancel`: the database ended one of the write's own statements with SQLSTATE 57014 (a statement timeout* or cancel) while it stayed up, at preparation or inside the publication transaction;* - `publication_ceiling`: the publication ran past `persistence.publication_ceiling_ms` and its grace, and the owner- * cancelled it (publication-deadline.ts `boundPublication`), whatever error the cancelled publish then rejects with.+ * cancelled it (publication-deadline.ts `boundPublication`), whatever error the cancelled publish then rejects with;+ * - `stale_plan` (GST-2314): a statement inside the write's transaction ran on a plan cached before a migration changed+ * its table (0A000 "cached plan must not change result type"); the transaction rolled back and the next claim+ * prepares it again.** Each one is counted in `transient_interruptions` (v261), never in `preparation_attempts` (the content-class budget* that preparations cut off at their deadline keep), and schedules the request's next claim at `next_attempt_at` with@@ -21,6 +24,7 @@* counted and no backoff applies: the switch restores the uncounted release, as it does for the deadlines.*/import { OperationError } from '../ops/contract.ts';+import { isStalePlanError } from '../retry-matcher.ts';import type { SqlEngine, WriteRequest } from './model.ts';import { PERSISTENCE_PROTOCOL_PREDICATE } from './protocol.ts';import { publicationDeadlinePassed } from './publication-deadline.ts';@@ -37,7 +41,7 @@ export function transientBackoffMs(interruptions: number): number {/** Whether a queued row's backoff has elapsed; every claim statement requires it. */export const BACKOFF_ELAPSED = (alias: string) => `(${alias}.next_attempt_at IS NULL OR ${alias}.next_attempt_at<=now())`;-export type TransientCause = 'statement_cancel' | 'publication_ceiling';+export type TransientCause = 'statement_cancel' | 'publication_ceiling' | 'stale_plan';export interface TransientInterruption { cause: TransientCause; sqlstate: string | null }/**@@ -49,6 +53,7 @@ export function transientInterruption(error: unknown): TransientInterruption | nconst code = (error as { code?: unknown } | null)?.code;const sqlstate = typeof code === 'string' && /^[0-9A-Z]{5}$/.test(code) ? code : null;if (publicationDeadlinePassed() && !(error instanceof OperationError)) return { cause: 'publication_ceiling', sqlstate };+ if (isStalePlanError(error)) return { cause: 'stale_plan', sqlstate: '0A000' };return code === '57014' ? { cause: 'statement_cancel', sqlstate: '57014' } : null;}diff --git a/src/core/retry-matcher.ts b/src/core/retry-matcher.tsindex 05d198d509..fe5f4c0238 100644--- a/src/core/retry-matcher.ts+++ b/src/core/retry-matcher.ts@@ -102,6 +102,16 @@ export function getMessage(err: unknown): string {return String(err ?? '');}+/**+ * GST-2314: a prepared statement whose cached plan went stale because a migration changed a table it reads (SQLSTATE+ * 0A000 "cached plan must not change result type"). Outside a transaction the driver re-prepares it; inside one the+ * transaction is already aborted, so the caller sees the error and a re-run on a fresh transaction succeeds. Every other+ * 0A000 (feature_not_supported) stays a refusal.+ */+export function isStalePlanError(err: unknown): boolean {+ return getCode(err) === '0A000' && /cached plan must not change result type/i.test(getMessage(err));+}+/*** SQLSTATE 57014: query_canceled / statement_timeout.* Postgres signals this when a statement exceeds `statement_timeout`.diff --git a/src/core/shared-skills/membership-authority.ts b/src/core/shared-skills/membership-authority.tsindex 5bb23990a0..88219d0d87 100644--- a/src/core/shared-skills/membership-authority.ts+++ b/src/core/shared-skills/membership-authority.ts@@ -4,7 +4,7 @@ import { sourceScopeOpts } from '../ops/context.ts';import { currentVerifiedLocalWriter, readLocalWriter, verifyLocalWriter, type LocalGrant } from '../persistence/identity.ts';import type { Principal } from '../persistence/model.ts';import { normalizeTokenScopes } from '../legacy-token-scope.ts';-import { authSourcesFromGrant, grantFromTokenRow } from '../grants/model.ts';+import { authSourcesFromGrant, grantFromTokenRow, TOKEN_GRANT_ROW_SELECT, type TokenGrantRow } from '../grants/model.ts';import { hasScope } from '../scope.ts';const deny = (): never => {@@ -30,7 +30,8 @@ export async function memberAuthority(ctx: OperationContext, operation: string):sources = [...new Set([row.source_id, ...(row.federated_read ?? [])])].filter(Boolean);grant = row;} else {- const [row] = await ctx.engine.executeRaw<Record<string, unknown>>('SELECT * FROM access_tokens WHERE id=$1 AND revoked_at IS NULL FOR SHARE', [principal.id]);+ // GST-2314: named columns, not SELECT *: inside this transaction a plan cached before a migration added a column fails 0A000.+ const [row] = await ctx.engine.executeRaw<TokenGrantRow>(`SELECT ${TOKEN_GRANT_ROW_SELECT} FROM access_tokens WHERE id=$1 AND revoked_at IS NULL FOR SHARE`, [principal.id]);if (!row) deny();const scopes = normalizeTokenScopes(row.scopes) ?? [];const tokenGrant = grantFromTokenRow(row);diff --git a/test/access-token-grant-columns.test.ts b/test/access-token-grant-columns.test.tsnew file mode 100644index 0000000000..45d86fa4f8--- /dev/null+++ b/test/access-token-grant-columns.test.ts@@ -0,0 +1,92 @@+/**+ * GST-2314: a migration that adds an `access_tokens` column under a running server must not fail the next+ * legacy-token write. Inside a transaction, a prepared `SELECT *` whose plan was cached before the column+ * existed fails with 0A000 ("cached plan must not change result type"): the driver's stale-plan retry lands+ * in the already-aborted transaction. The publication authority read and the shared-skills membership read+ * name their columns, so a new column leaves their result type unchanged.+ * 1. Protects: the publication authority read (FOR SHARE) and the membership authority read, run on every+ * pooled connection before and after `ALTER TABLE access_tokens ADD COLUMN`, succeed; so does a+ * legacy-token put_page after the column is added.+ * 2. Fails when: either read selects every column of access_tokens inside its transaction.+ * 3. Real seams: the token service, the serve --http token verifier, MCP dispatch and the durable write path.+ * Each read runs in POOL concurrent transactions held open together, so every pooled connection prepares it+ * first and every one reuses its cached plan after the column is added.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';+import { sqlQueryForEngine } from '../src/core/sql-query.ts';+import { mintLegacyToken } from '../src/core/token-mint.ts';+import { dispatchToolCall } from '../src/mcp/dispatch.ts';+import { authorizeWrite } from '../src/core/persistence/authority.ts';+import type { WriteAuthority } from '../src/core/persistence/model.ts';+import { memberAuthority } from '../src/core/shared-skills/membership-authority.ts';+import type { AuthInfo, OperationContext } from '../src/core/ops/contract.ts';+import { testBackends } from './helpers/test-backends.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';++const note = (title: string) => ['---', 'type: note', 'title: ' + title, '---', '# ' + title, '', 'Body of ' + title + '.'].join('\n');++for (const kind of testBackends()) {+ describe('legacy-token reads after a migration adds an access_tokens column (' + kind + ')', () => {+ const POOL = kind === 'postgres' ? 3 : 1;+ let engine: BrainEngine;+ let close: () => Promise<void>;+ let column = 0;+ const addColumn = () => engine.executeRaw('ALTER TABLE access_tokens ADD COLUMN probe_added_' + (column++) + ' TEXT');+ /** Runs fn in POOL transactions that all stay open until each has started, so each gets its own pooled connection. */+ async function onEveryConnection<T>(fn: (tx: BrainEngine) => Promise<T>): Promise<T[]> {+ let started = 0;+ let release!: () => void;+ const all = new Promise<void>(resolve => { release = resolve; });+ return Promise.all(Array.from({ length: POOL }, () => engine.transaction(async tx => {+ if (++started === POOL) release();+ await all;+ return fn(tx);+ })));+ }++ beforeAll(async () => {+ if (kind === 'postgres') ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!, 'instance', POOL));+ else { engine = new PGLiteEngine(); await engine.connect({ engine: 'pglite' } as never); await engine.initSchema(); close = () => engine.disconnect(); }+ }, 240_000);+ afterAll(async () => { await close?.(); }, 60_000);++ test('the publication authority read succeeds on every connection after the column is added', async () => {+ const minted = await mintLegacyToken(engine, { name: 'publisher', scopes: ['read', 'write'], takesHolders: ['world'] });+ const [source] = await engine.executeRaw<{ incarnation: string }>("SELECT incarnation FROM sources WHERE id='default'");+ const authority: WriteAuthority = { version: 1, principal: { kind: 'legacy_token', id: minted.id }, remote: true, sourceId: 'default',+ sourceIncarnation: source.incarnation, scopes: ['read', 'write'], operations: null, slugPrefixes: null, takesHolders: ['world'] };+ const authorize = () => onEveryConnection(tx => authorizeWrite(tx, authority, 'put_page', 'notes/publication-read', true));+ await authorize();+ await addColumn();+ await authorize();+ }, 240_000);++ test('the membership authority read succeeds on every connection after the column is added', async () => {+ const minted = await mintLegacyToken(engine, { name: 'member', scopes: ['read', 'skills_member_self'], takesHolders: ['world'], allowedOperations: ['join_brain'] });+ const read = () => onEveryConnection(async tx => (await memberAuthority({ engine: tx, remote: true, sourceId: 'default',+ auth: { principal: { kind: 'legacy_token', id: minted.id }, scopes: ['read', 'skills_member_self'], allowedOperations: ['join_brain'] } } as unknown as OperationContext, 'join_brain')).principal);+ const principal = { kind: 'legacy_token', id: minted.id };+ expect(await read()).toEqual(Array(POOL).fill(principal));+ await addColumn();+ expect(await read()).toEqual(Array(POOL).fill(principal));+ }, 240_000);++ test('a legacy-token put_page after the column is added publishes', async () => {+ const minted = await mintLegacyToken(engine, { name: 'writer', scopes: ['read', 'write'], takesHolders: ['world'] });+ const provider = new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });+ const put = async (slug: string) => {+ const auth = await provider.verifyAccessToken(minted.token) as unknown as AuthInfo;+ const out = await dispatchToolCall(engine, 'put_page', { slug, content: note(slug), request_id: randomUUID() },+ { remote: true, transport: 'http', auth, sourceId: 'default', takesHoldersAllowList: ['world'] });+ return { isError: out.isError === true, text: out.content.map(c => c.text).join('\n') };+ };+ expect(await put('notes/before-column')).toMatchObject({ isError: false });+ await addColumn();+ expect(await put('notes/after-column')).toMatchObject({ isError: false });+ }, 240_000);+ });+}diff --git a/test/access-ungranted-member-writes.test.ts b/test/access-ungranted-member-writes.test.tsindex f6b8f2da20..53db50678d 100644--- a/test/access-ungranted-member-writes.test.ts+++ b/test/access-ungranted-member-writes.test.ts@@ -50,6 +50,7 @@ for (const kind of testBackends()) {describe('writes to default from a member-bound credential (' + kind + ', repo-less)', () => {let engine: BrainEngine;let close: () => Promise<void>;+ let databaseUrl: string;const tokens: Record<string, string> = {};const provider = () => new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });@@ -84,7 +85,7 @@ for (const kind of testBackends()) {const none = { get_page: false, search: false, query: false, recall: false, list_pages: false };beforeAll(async () => {- if (kind === 'postgres') ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!));+ if (kind === 'postgres') ({ engine, close, databaseUrl } = await isolatedPersistencePostgres(process.env.DATABASE_URL!));else { engine = new PGLiteEngine(); await engine.connect({ engine: 'pglite' } as never); await engine.initSchema(); close = () => engine.disconnect(); }__resetPrivateVisibilityCacheForTests();await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING', [PRIVATE_SRC]);@@ -145,7 +146,12 @@ for (const kind of testBackends()) {try {await dispatchToolCall(engine, 'put_page', { slug, content: note('Alice lookup', 'Alice lookup code ' + canary + '.'), request_id: randomUUID() },{ remote: true, transport: 'http', auth, sourceId: 'default', takesHoldersAllowList: ['world'] });- } finally { await engine.executeRaw('ALTER TABLE access_tokens RENAME COLUMN member_id_unreadable TO member_id'); }+ } finally {+ await engine.executeRaw('ALTER TABLE access_tokens RENAME COLUMN member_id_unreadable TO member_id');+ // GST-2314: a pooled connection that prepared an access_tokens SELECT * while the column was renamed fails 0A000 on+ // its next use inside a transaction, so the later cases start on fresh connections.+ if (kind === 'postgres') { await disposePersistenceConsumer(engine); await engine.disconnect(); await engine.connect({ database_url: databaseUrl, poolSize: 4 } as never); }+ }expect(await stored(slug)).toBeUndefined();expect(await reads('public', slug, canary)).toEqual(none);await engine.executeRaw('INSERT INTO access_members (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', ['carol-example']);diff --git a/test/persistence-stale-plan-retry.test.ts b/test/persistence-stale-plan-retry.test.tsnew file mode 100644index 0000000000..4248c4ad76--- /dev/null+++ b/test/persistence-stale-plan-retry.test.ts@@ -0,0 +1,74 @@+/**+ * GST-2314: a migration that adds a `persistence_requests` column under a running writer must not fail accepted writes.+ * Inside a publication transaction, a prepared `SELECT *` whose plan was cached before the column existed fails with+ * 0A000 ("cached plan must not change result type"); the transaction rolls back and nothing is committed.+ * 1. Protects: after `ALTER TABLE persistence_requests ADD COLUMN`, every put_page on a pool of 3 publishes, with no+ * request charged more stale-plan interruptions than `persistence.max_transient_retries` allows, and no caller+ * refused at admission.+ * 2. Fails when: the stale-plan error is not a transient interruption (the write ends failed/storage_error, never+ * claimed again) or admission does not re-run it (the caller gets internal_error).+ * 3. Real seams: the serve --http token verifier, MCP dispatch, admission, the resident consumer and publication.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';+import { sqlQueryForEngine } from '../src/core/sql-query.ts';+import { mintLegacyToken } from '../src/core/token-mint.ts';+import { dispatchToolCall } from '../src/mcp/dispatch.ts';+import type { AuthInfo } from '../src/core/ops/contract.ts';+import { testBackends } from './helpers/test-backends.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';++const note = (title: string) => ['---', 'type: note', 'title: ' + title, '---', '# ' + title, '', 'Body of ' + title + '.'].join('\n');+const MAX_TRANSIENT_RETRIES = 5;++for (const kind of testBackends()) {+ describe('accepted writes after a migration adds a persistence_requests column (' + kind + ')', () => {+ const POOL = kind === 'postgres' ? 3 : 1;+ let engine: BrainEngine;+ let close: () => Promise<void>;++ beforeAll(async () => {+ if (kind === 'postgres') ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!, 'instance', POOL));+ else { engine = new PGLiteEngine(); await engine.connect({ engine: 'pglite' } as never); await engine.initSchema(); close = () => engine.disconnect(); }+ }, 240_000);+ afterAll(async () => { await close?.(); }, 60_000);++ test('every put_page after the column is added publishes within the transient retry limit', async () => {+ const minted = await mintLegacyToken(engine, { name: 'writer', scopes: ['read', 'write'], takesHolders: ['world'] });+ const provider = new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });+ let n = 0;+ const put = async () => {+ const slug = 'notes/stale-plan-' + (n++);+ const requestId = randomUUID();+ const auth = await provider.verifyAccessToken(minted.token) as unknown as AuthInfo;+ const out = await dispatchToolCall(engine, 'put_page', { slug, content: note(slug), request_id: requestId },+ { remote: true, transport: 'http', auth, sourceId: 'default', takesHoldersAllowList: ['world'] });+ const text = out.content.map(c => c.text).join('\n');+ return { requestId, code: out.isError === true ? String(JSON.parse(text).code) : 'ok' };+ };+ const warm = [];+ for (let i = 0; i < 4; i++) warm.push(await put());+ warm.push(...await Promise.all(Array.from({ length: POOL }, () => put())));+ expect(warm.map(w => w.code)).toEqual(warm.map(() => 'ok'));++ await engine.executeRaw('ALTER TABLE persistence_requests ADD COLUMN probe_added_column TEXT');+ const after = [];+ for (let i = 0; i < 6; i++) after.push(await put());+ after.push(...await Promise.all(Array.from({ length: POOL }, () => put())));+ expect(after.map(a => a.code).filter(code => code !== 'ok' && code !== 'write_pending')).toEqual([]);++ const ids = after.map(a => a.requestId);+ const settled = async () => engine.executeRaw<{ state: string; error_code: string | null; transient_interruptions: number }>(+ 'SELECT state, error_code, transient_interruptions FROM persistence_requests WHERE request_id = ANY($1::uuid[])', [ids]);+ let rows = await settled();+ for (const until = Date.now() + 120_000; Date.now() < until && rows.some(r => r.state !== 'committed' && r.state !== 'failed'); rows = await settled()) {+ await new Promise(resolve => setTimeout(resolve, 500));+ }+ expect(rows.map(r => r.state + (r.error_code ? '/' + r.error_code : ''))).toEqual(ids.map(() => 'committed'));+ expect(Math.max(...rows.map(r => r.transient_interruptions))).toBeLessThanOrEqual(MAX_TRANSIENT_RETRIES);+ }, 300_000);+ });+}