Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Lane the two new Postgres probes and keep persistence-stalls.md under its cap (GST-2314)

garrytan3cf7ccc5b02d
diff --git a/docs/architecture/key-files/persistence-stalls.md b/docs/architecture/key-files/persistence-stalls.md
index 07259aaf57..6d2bf4edd8 100644
--- a/docs/architecture/key-files/persistence-stalls.md
+++ b/docs/architecture/key-files/persistence-stalls.md
@@ -44,5 +44,5 @@ owner (`owner_wedged_here`), movement surfaces (`data_moving`, doctor
- `src/core/persistence/consumer-lane.ts` — #6317 (reporter ask 2): `consumerStatementEngine`, the Proxy that routes the consumer's own tick statements (idle probe, switch read, recovery and expired-claim scans, capacity, releases, floor) to `executeRawDirect` while the dual pool is active (`GBRAIN_DIRECT_DATABASE_URL` or the Supabase-derived direct URL), so the owner's round-trips avoid a transaction-mode pooler; `status().connection = { lane, pooler_mode, direct_host? }` says which lane is in use, and one `[persistence] phase=start reason=transaction_pooler` line prints when only a `prepare=false` URL exists.
- `src/core/persistence/consumer-diagnostics.ts` — #6317 (B3, I1): `hostConsumersReport` (the `host.consumers` block of `writer status --json`), `consumerOverlap` (two resident-kind full rows alive over 30 s), `ownersWithoutHeartbeat` (claim owners on running claims with no row) and `hostIdentityMismatches` (this process's `host.json` id differs from a binding owner's while the owner's checkout resolves on this filesystem or the machine id matches), read-only and shared by doctor and writer status.
- `src/core/persistence/publication-deadline.ts` (#6288 part 1, #6352; UC2) — what bounds a publish that neither resolves nor rejects. On Postgres the publish transaction is the real bound: `declareDurablePersistence` already sets a transaction-local `statement_timeout` (5 s) and `lock_timeout`, and `declarePublication` (#6712 L6: the opening statement of `publishMutation`'s and `publishGroup`'s transactions, the same `SELECT` as `declareDurablePersistence`, so no extra round trip even where a transaction-mode pooler turns pipelining off) adds a transaction-local `idle_in_transaction_session_timeout` of `persistence.publication_ceiling_ms` (default 300000, 60000 to 3600000, read per engine; never above the pool's own `GBRAIN_IDLE_TX_TIMEOUT`), so a publisher stuck between statements is ended by the server and commits nothing; both survive a transaction pooler that drops startup parameters. Never set on PGLite: it would enforce the bound, and a transaction it ends wedges the engine's only connection. `boundPublication` (the consumer's single path and `executeClaimedGroup`'s group publish and per-member fallback) settles only with the publish's own outcome, so nothing releases or reports a request or group while its transaction can still commit; at the ceiling the claim stamp reads `waiting_on: publication_deadline`, after `PUBLICATION_GRACE_MS` (30 s) it aborts the publication's signal (passed to `engine.transaction(fn, { signal })` through `publicationTransaction`: on Postgres the connection is discarded and nothing commits; the rejection waits for the transaction body to settle, then surfaces with the driver's `CONNECTION_CLOSED` code, so the claim is requeued for a retry; PGLite checks the signal only before BEGIN), and after `PUBLICATION_SETTLE_MS` (5 s) more the consumer lists it in `stuck_publications` and reports `restart_required` (its root stays held: restarting the owner is the recovery, never a promised reclaim). The lease keeps renewing, so a reclaim never races a publisher that may still write files. The wave 9 yield between groups is outside any one bound. `PublicationBound.later` is the scheduler seam (default a real unref'd timer); the unit test drives the three phases through a manual clock so a stalled runner cannot fire a phase early (the consumer's `clock` seam is separate). Tests: `test/persistence-publication-deadline.test.ts` (barriers after the token lock, after the file rename and before commit; the server-side idle end on Postgres; the wiring probe).
-- `src/core/persistence/transient-retry.ts` — the retry budget for a managed write's transient interruptions: a statement the database ended with 57014 (`statement_cancel`), a statement in the write's transaction that ran on a plan cached before a migration changed its table (0A000 "cached plan must not change result type", `stale_plan`, matched by `retry-matcher.ts` `isStalePlanError`; GST-2314) or a publication `boundPublication` cancelled at the ceiling (`publication_ceiling`, read from the publication's signal by `publicationDeadlinePassed`, whatever error it surfaces as). `chargeTransientInterruption` counts it in `persistence_requests.transient_interruptions` (v261, never `preparation_attempts`) and sets `next_attempt_at` to 1 s, 2 s, 4 s, 8 s, then 15 s (under the drain's 30 s no-progress window); `BACKOFF_ELAPSED` keeps every claim (`claimableWriteSql`, lane heads, group followers) off the row until then, so no claim is held while it waits, and `readHeadBlockers` names it `transient_backoff`. Past `persistence.max_transient_retries` (default 5, 0 to 20, a preparation-budget key) coordinator.ts ends the request `write_interrupted`; the charge commits with the claim's release. Off with `preparation_deadlines`. Tests: `test/pathological/lp-b-transient-retry-budget.test.ts`, `test/pathological/lp-c-publication-ceiling-retry.test.ts` (`_setPublicationTimingForTest`), `test/persistence-write-interrupted.test.ts`.
+- `src/core/persistence/transient-retry.ts` — the retry budget for a managed write's transient interruptions: a statement the database ended with 57014 (`statement_cancel`), a stale cached plan (0A000, `stale_plan`) or a publication `boundPublication` cancelled at the ceiling (`publication_ceiling`, read from its signal by `publicationDeadlinePassed`). `chargeTransientInterruption` counts it in `persistence_requests.transient_interruptions` (v261, never `preparation_attempts`) and sets `next_attempt_at` to 1 s, 2 s, 4 s, 8 s, then 15 s (under the drain's 30 s no-progress window); `BACKOFF_ELAPSED` keeps every claim (`claimableWriteSql`, lane heads, group followers) off the row until then, so no claim is held while it waits, and `readHeadBlockers` names it `transient_backoff`. Past `persistence.max_transient_retries` (default 5, 0 to 20, a preparation-budget key) coordinator.ts ends the request `write_interrupted`; the charge commits with the claim's release. Off with `preparation_deadlines`. Tests: `test/pathological/lp-b-transient-retry-budget.test.ts`, `test/pathological/lp-c-publication-ceiling-retry.test.ts` (`_setPublicationTimingForTest`), `test/persistence-write-interrupted.test.ts`.
- `src/core/persistence/sync-discovery.ts` (#6349 staged catch-up) — when an incremental delta (not `--full`, not company) exceeds the cursor bound (100,000 entries, 16 MiB; `_setCursorBoundForTest` lowers it), `stagedSyncTarget` gallops forward over `git rev-list --first-parent last..HEAD`, verifying every candidate's delta from the last synced commit against the bound (a net delta is not monotone, so no binary search), and discovery reruns pinned at the furthest fitting commit with no working-tree entries, recording `stagedHead`; the cursor carries it and `SyncResult.staged { target, head, stages }` reports it. One commit alone above the bound refuses `request_too_large` naming the commit. `commands/sync/perform.ts` loops a drain stage by stage to HEAD (the drain's one extra HEAD-moved pass is skipped for a staged result). Tests: `test/sync-staged-catchup.test.ts`.
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index dc6d271680..490bedc708 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -27,6 +27,7 @@ test/access-owner-default-writes.test.ts
test/access-owner-reads.test.ts
test/access-page-indexes-online.test.ts
test/access-qm-actor.test.ts
+test/access-token-grant-columns.test.ts
test/access-ungranted-member-writes.test.ts
test/access-writes.test.ts
test/activation-unbound-rooted-6449.test.ts
@@ -450,6 +451,7 @@ test/persistence-reconcile-git-dirty-6138.test.ts
test/persistence-skill-bundles.serial.test.ts
test/persistence-skill-crash.slow.test.ts
test/persistence-skip-cost.test.ts
+test/persistence-stale-plan-retry.test.ts
test/persistence-sync-cursor-options.test.ts
test/persistence-sync-cursor-strategy-6696.test.ts
test/persistence-sync-fence-holds.test.ts