Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(sync): bound engine-internal preparation reads, harden noop/waiver provenance, cap rows-behind batch memory (#6318, #8089, #8093, #8094)

garrytan612a1ae1b577
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bebd4b391d..cc8e1b640e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,20 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A table lock on `pages` no longer pins a managed sync's preparations past their budget, sync and waiver provenance records name a file only when it holds the bytes they record, and the rows-behind doctor check holds at most 64 MiB of committed files per batch (#6318, #8089, #8093, #8094).**
+
+A managed sync bounds each preparation by `persistence.sync_preparation_ms`. Its raw reads already ended on the server at that budget, but the engine's own reads (the page snapshot, the import pipeline's duplicate and content-hash lookups) ran as plain autocommit statements, so under `LOCK TABLE pages` held elsewhere they waited as long as the lock did, through a pooler that drops the session `statement_timeout`. Those reads now run in a transaction whose `statement_timeout` is the time the preparation has left, and a read the server ends there is the member's `preparation_deadline`, held as `preparation_stalled` like the head member. Nothing changes without a deadline, inside a transaction, or on PGLite.
+
+### Itemized changes
+
+- `src/core/read-deadline.ts` (new), `src/core/postgres-engine.ts` `withScopedReadTransaction`, `src/core/persistence/bounded-reads.ts`: a call through a preparation's bounded engine view carries the clock's deadline, and the Postgres engine's scoped reads issued under it set `statement_timeout` to the time left (#6318). Probe: `test/e2e/persistence-preparation-engine-reads-lock-postgres.test.ts`.
+- `src/core/persistence/sync-prepare.ts` (`publishedSyncNoop`, new `heldDiskBytes`), `src/core/persistence/sync-waivers.ts` (`waivedProvenance`): a `published` no-op re-reads its file at apply and records provenance only when the file still has the raw hash; a waiver records the file it read only when it holds the imported bytes (up to line endings), the rule `syncDiskBytes` already follows (#8089). Probe: `test/sync-provenance-writers-8089.test.ts`.
+- `src/core/persistence/rows-behind.ts`: doctor `sync_rows_behind_checkpoint` reads and compares a batch's committed files in chunks of at most 64 MiB (`ROWS_BEHIND_BATCH_BYTES`), drops each chunk's buffers after the comparison, checks the deadline before the per-source tree reads and each chunk, and reads its screen config only once a batch starts (#8093 item 1). The docstring names `not_eligible` (#8094).
+- `test/cycle/dream-reverse-write-commit.test.ts`: a refused commit in one source's checkout holds only that source's page (#8094).
+- `scripts/bench/managed-sync-stall-repro.ts`: `--sync-preparation-ms` and a `lock_release` result listing sync statements still waiting on a lock 3 s past the budget.
+
## [0.60.529.0] - 2026-10-31
**A pull request that adds a file to the PGLite snapshot inputs no longer gets stopped at gstack's pregate secrets check by its own workflow cache keys (#8107).**
diff --git a/docs/architecture/key-files/canonical-reconciliation-continued.md b/docs/architecture/key-files/canonical-reconciliation-continued.md
index d667f227db..04af8a7a73 100644
--- a/docs/architecture/key-files/canonical-reconciliation-continued.md
+++ b/docs/architecture/key-files/canonical-reconciliation-continued.md
@@ -3,6 +3,6 @@
[Subsystem index](../KEY_FILES.md) · [Previous entries](canonical-reconciliation.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-- `src/core/persistence/rows-behind.ts` — #8071: `scanRowsBehind(engine, sourceId, { deadlineAt })` finds live Markdown pages of a managed source whose row holds an older version of its file than `sources.last_commit`, with no hold. It reads the checkpoint tree from Git (`readHeadTree(root, commit)`), compares each page in id batches of 256 with the committed blob (`compareInlineExportFile`, or `compareSegmentedExportBlob` for a page stored in segments) and confirms a difference with `prepareRepairPublication`; direction comes from the page's sync import record (`behind` when its revision is the recorded one, `unproven` without a record, `database_moved` skipped). Paths sync would not import (`loadSyncEligibility` from `sync-discovery.ts`) are skipped `not_eligible`. A blob over `SYNC_READ_BOUND` is never read and reported with `file_too_large`. Owner host only; a repo-less brain is `not_applicable`.
+- `src/core/persistence/rows-behind.ts` — #8071: `scanRowsBehind(engine, sourceId, { deadlineAt })` finds live Markdown pages of a managed source whose row holds an older version of its file than `sources.last_commit`, with no hold. It reads the checkpoint tree from Git (`readHeadTree(root, commit)`), compares each page in id batches of 256 with the committed blob (`compareInlineExportFile`, or `compareSegmentedExportBlob` for a page stored in segments) and confirms a difference with `prepareRepairPublication`; direction comes from the page's sync import record (`behind` when its revision is the recorded one, `unproven` without a record, `database_moved` skipped). Paths sync would not import (`loadSyncEligibility` from `sync-discovery.ts`) are skipped `not_eligible`. A blob over `SYNC_READ_BOUND` is never read and reported with `file_too_large`. Blobs are read and compared in chunks of at most `ROWS_BEHIND_BATCH_BYTES` (64 MiB) and dropped after the comparison, and the deadline is checked before each tree read and each chunk, so one batch of large files neither holds every blob in memory nor runs past the budget. Owner host only; a repo-less brain is `not_applicable`.
- `src/commands/doctor/checks/rows-behind.ts` — doctor `sync_rows_behind_checkpoint` (BRAIN, after `sync_overwritten_db_edits`): per source counts and first slugs from `scanRowsBehind`; `warn` on `behind` rows with the read-only `gbrain repair rows-behind --source <id> --json` preview as the fix; `unknown` (info) when the `GBRAIN_DOCTOR_ROWS_BEHIND_TIMEOUT_MS` budget (default 10 s) stopped a scan or a checkout is unreadable; not applicable (ok) off the owner host and on repo-less brains.
- `src/core/repair/rows-behind.ts` — `gbrain repair rows-behind --source <id>`: explicit-only, preview-bound (`--apply --expect <hash>`, `--include-ambiguous` adds rows with no import record). `reimport` submits the committed bytes through `submitManagedFileRepair`; `hold:<code>` writes the hold sync would have written (`writeGitHold`). Refused off the owner host (`not_owner_host`). Never runs on upgrade. Tests: `test/rows-behind-checkpoint-8071.test.ts` (PGLite and Postgres via `test/e2e/rows-behind-checkpoint-8071-postgres.test.ts`).
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index 8f25bdc923..0cd38ec1b9 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -28,7 +28,7 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.
- `src/core/persistence/put-page-screen.ts` — #6741 B: `unchangedPutPage` screens a trusted local put_page (the transcript import) through `preparePageAdmission` (unadmitted), the page publication preparer and the no-op kernel; skips admission only on a genuine no-op with no pending request, no segmented body and no projection work. Tests: `test/transcripts-managed-noop.test.ts`.
- `src/core/persistence/import-reads.ts` — GBRA-75 wave 10: `readImportSnapshots` reads a run of pages as `readPageSnapshot(slug, { sourceId, includeDeleted: true })` returns each (soft-deleted rows, withdrawals and purge markers included) through `readPageSnapshotsBatch({ includeDeleted, absentIsNull })`, looping over its byte-budget prefixes; the managed import screen uses it for two or more new members. `importGroupReads` serves an import group's preparation (`group-publish.ts` `preparationReads`): each member's first own snapshot read comes from one batch, and `findDuplicatePage` calls that arrive together (all members, or a 2 ms window) are answered by one `findDuplicatePages` (`engine-sql/pages.ts`, same predicate and ranking per input); a failed batch falls back to each member's single read; with `reread` (the read-only screen) a member's later reads of its page come from the batch too. GBRA-108: `sliceReads` answers a screened slice's per-page reads (the origin claimants of `sync-origin.ts`, the stored timeline and take collisions of `canonical-projections.ts`, the safe-chunk fence of `page-state/projections.ts`, matched by their exported statements) from one statement per kind over the slice, row for row; a page with stored takes, a page or path outside the slice, a signalled call or a failed slice read runs its own statement (test `test/sync-legacy-sweep-gbra108.test.ts`). Tests: `test/import-reads.test.ts`, Postgres arm `test/e2e/import-reads.test.ts`.
- `src/core/persistence/screening-paths.ts` — GBRA-75 wave 7: `withScreeningPaths(run)` opens one screening batch's memo of root-level filesystem facts and `screeningPath(key, compute)` resolves each once inside it: other sources' root realpaths (`assertImportPaths`), shared skillpack root canonical paths and the source root's Git scope (`knowledge-guard.ts`). Per-file paths are never memoized; a nested batch joins the open one; the memo closes with `run`, so work that outlives it computes fresh; a throwing compute is never memoized. Opened by the managed import plan and screen loops (`import-mutations.ts`), each sync waiver screen (`screenWaiver`) and `sync-run.ts` `waiveRun`'s screens. Tests: `test/persistence-skip-cost.test.ts`.
-- `src/core/persistence/sync-waivers.ts` — managed-sync no-op waivers (#5470, #5984). `waiveNoopEntry` screens a frozen, unadmitted entry: an unchanged import (`unchangedSyncImport`, moved here from `sync-run.ts`, with the #5751 working-tree kernel waivers) or a delete whose page is already soft-deleted at the entry's frozen `expected_revision` with its file absent (never an unowned deletion or a rename). A candidate runs `validateSyncAuthority` first (a revoked grant throws before any waiver), then commits in one transaction with a fixed lock order: page guard (`lockPageKeys`), then the managed-sync cursor row `FOR UPDATE`; it re-reads the snapshot (and, for deletes, the origin and file absence) and refuses while `UNFINISHED_PAGE_REQUEST_SQL` finds an unfinished request for the page by slug, page id or a `renameFrom` naming it. That lookup is three `LIMIT 1` branches served by `persistence_requests_pending`, `persistence_requests_recovery` and the database-only pending index (EXPLAIN pinned in `test/persistence-managed-sync.test.ts`). A cursor that moved meanwhile is adopted instead of waived again, so `waived` counts only increment in the winning transaction. `GBRAIN_SYNC_WAIVE_NOOP=0` admits every entry. #6278: the lookup also matches the entry's file path, origin and a rename from that origin (`unfinishedPageRequestParams`), so the hold gate in `sync-run.ts` shares it. `unchangedSyncImport` (and so `sameContentAtOrigin`'s preparation) runs the sync preparer under `raceSyncBudget` against `persistence.sync_preparation_ms` (`syncPreparationBudgetMs`, read at most every 5 s, default 120 s) and the run's signal: past the budget nothing is waived and the origin refusal stands (fail open, nothing counted or held, `SyncPreparationBudgetExceeded`); a cancelled run propagates. GBRA-75 wave 8: `unchangedSyncImport` prepares with `{ unsaved: true }`, so its validation also accepts this run's cursor with nothing pending; before that every import waiver-run screen failed the cursor check and each unchanged file took a pending save, a second screen and its own waiver transaction. `waiveNoopRun` (up to 64 entries, one cursor write) re-reads the owner epoch under its lock (a change passes nothing; the per-entry path then refuses it) and each import's file bytes (an edit ends the prefix). Both waiver transactions read a waived import's file (a raw hash that no longer matches admits the entry) and write its `sync-import-provenance` record from the preparation's `importProvenance` (`waivedProvenance`: disk hash and blob id of those bytes, the page's current revision), so a waived page carries the same record a publication writes. Tests: `test/sync-waiver-budget.test.ts`, `test/sync-waiver-run-imports.test.ts`, `test/persistence-managed-sync.test.ts` (#5984 waiver cases), `test/persistence-sync-legacy-files-5751.test.ts`.
+- `src/core/persistence/sync-waivers.ts` — managed-sync no-op waivers (#5470, #5984). `waiveNoopEntry` screens a frozen, unadmitted entry: an unchanged import (`unchangedSyncImport`, moved here from `sync-run.ts`, with the #5751 working-tree kernel waivers) or a delete whose page is already soft-deleted at the entry's frozen `expected_revision` with its file absent (never an unowned deletion or a rename). A candidate runs `validateSyncAuthority` first (a revoked grant throws before any waiver), then commits in one transaction with a fixed lock order: page guard (`lockPageKeys`), then the managed-sync cursor row `FOR UPDATE`; it re-reads the snapshot (and, for deletes, the origin and file absence) and refuses while `UNFINISHED_PAGE_REQUEST_SQL` finds an unfinished request for the page by slug, page id or a `renameFrom` naming it. That lookup is three `LIMIT 1` branches served by `persistence_requests_pending`, `persistence_requests_recovery` and the database-only pending index (EXPLAIN pinned in `test/persistence-managed-sync.test.ts`). A cursor that moved meanwhile is adopted instead of waived again, so `waived` counts only increment in the winning transaction. `GBRAIN_SYNC_WAIVE_NOOP=0` admits every entry. #6278: the lookup also matches the entry's file path, origin and a rename from that origin (`unfinishedPageRequestParams`), so the hold gate in `sync-run.ts` shares it. `unchangedSyncImport` (and so `sameContentAtOrigin`'s preparation) runs the sync preparer under `raceSyncBudget` against `persistence.sync_preparation_ms` (`syncPreparationBudgetMs`, read at most every 5 s, default 120 s) and the run's signal: past the budget nothing is waived and the origin refusal stands (fail open, nothing counted or held, `SyncPreparationBudgetExceeded`); a cancelled run propagates. GBRA-75 wave 8: `unchangedSyncImport` prepares with `{ unsaved: true }`, so its validation also accepts this run's cursor with nothing pending. `waiveNoopRun` (up to 64 entries, one cursor write) re-reads the owner epoch under its lock (a change passes nothing; the per-entry path then refuses it) and each import's file bytes (an edit ends the prefix). Both waiver transactions read a waived import's file (a raw hash that no longer matches admits the entry) and write its `sync-import-provenance` record from the preparation's `importProvenance` (`waivedProvenance`: disk hash and blob id of those bytes when they hold the imported ones, `heldDiskBytes`, #8089; the page's current revision), so a waived page carries the same record a publication writes. Tests: `test/sync-waiver-budget.test.ts`, `test/sync-waiver-run-imports.test.ts`, `test/persistence-managed-sync.test.ts` (#5984 waiver cases), `test/persistence-sync-legacy-files-5751.test.ts`.
- `src/core/persistence/atom-maintenance.ts` (write failures) — #7514: a batch whose write ended without committing (`conflict`, `failed`, `cancelled`) is a write failure, `atomWriteFailure`: its record carries the write's request id and refusal code (the code survives receipt compaction, which erases the message to "The write did not commit."), and on the drain each time the batch is met counts one strike in an `op_checkpoints` row (`managed-atoms-uncommitted`, keyed by the failed run key) with no model call, since a failed batch is never re-extracted on its own. At `MAX_DETERMINISTIC_FAILURES` strikes a journaled failure completion (intent `attempts`) advances the drain-key checkpoint to `held`, so the origin leaves the backlog and doctor lists it with that completion's receipt and the reviewed retry; an explicit retry records no strike. Test: `test/extract-atoms-commit-failure-hold.test.ts`.
- `src/core/persistence/accepted-pending.ts` — `acceptedPendingReceipt(error)`: the shared test for an accepted, not yet committed write, used by managed atom publication, extract_atoms, managed import and the connector budget. `publicationHold(error)` returns `pending` for such a receipt or `contention` for an OperationError `storage_error` with detail `database_contention` and no receipt (admission rolled back, nothing recorded), else null — never from a message, a terminal receipt, a contention error carrying a receipt or a permission refusal; `publishOrHold(fn)` runs one publication and returns null on commit, the hold when there is one, and rethrows anything else. Used by the patterns, drift and synthesize dream phases.
- `src/core/persistence/connector-checkpoint-migration.ts` — migration 176 and orphan cleanup. The newest committed checkpoint receipt per source incarnation (found by slug, so a compacted one is still seen and re-walks) is copied to the stable key with `INSERT … ON CONFLICT DO NOTHING`; state rows are seeded; the cutoff is stored in `op_checkpoints` (`op=managed-connector-migration`, fingerprint `v2-cutoff`, excluded from the checkpoint purge). `orphanConnectorCheckpoints` keeps rows touched after the start, rows any registered source (archived included) can load under its stable or alias key (`loadableConnectorKeys`, #6561), and rows referenced by a non-terminal request or a pending set; timestamps bind as text. `gbrain repair connector-checkpoints` (`src/core/repair/connector-checkpoints.ts`, whose preview counts alias-kept rows as `alias_protected`) and the `connector_checkpoints` doctor check reuse it with a 7-day floor. No upgrade step rewrites alias rows (#6561): `connector-sync.ts` `resumeCheckpoint` resumes from an alias row written after the stable one when the stable key has none or the alias holds the newer completed Gmail backfill, and the run's first save moves that cursor to the stable key; every checkpoint and page write compares against and writes only the stable row. `connector-reset.ts` guards `gbrain sync --reset-checkpoint` to connector sources.
diff --git a/docs/architecture/key-files/persistence-stalls.md b/docs/architecture/key-files/persistence-stalls.md
index 6ca7944c74..1c0a852299 100644
--- a/docs/architecture/key-files/persistence-stalls.md
+++ b/docs/architecture/key-files/persistence-stalls.md
@@ -21,7 +21,7 @@ owner (`owner_wedged_here`), movement surfaces (`data_moving`, doctor
- `src/core/persistence/claim-phase.ts` — #6176 / #6278: the phase and step a running write request's claim is in. `startClaimPhase`/`enterClaimPhase` keep a per-claim `ClaimPhaseClock` in the consumer (single and group routes); `claimPhaseStamp` is the jsonb every claim, renewal and group dispatch mark stores in `persistence_requests.claim_phase` (v217): phase `preparing` or `publishing`, claim start, phase start, the execution token (so a later claim never reads an earlier claim's ages), the `step` in flight and when it began, `waiting_on` and the owning process (`owner`: command kind, pid, gbrain build; #6650 boot id, start). Preparers (`sync-prepare.ts`, `page-prepare.ts`, `prepared-maintenance.ts`, link resolution) call `enterClaimStep(clock, step, signal, waitingOn)` at each await boundary: it names the step, records what the await waits on (`git`, `fs`, `db`, `pool`, else `unknown`: set only where observed, never inferred) and throws the preparation's abort reason, so cancellation reaches a preparer between its queries even though the queries take no signal. `claimStateOf` reads a row back from the current claim's stamp only: `preparing`/`publishing` (live renewal), `publication_transaction` (claim lapsed while running, read on the database clock: a transaction holds the row or the owner is gone; requeued once it ends), `file_publication` (`publication_started`; recovery finishes it) or `unrecorded`, with claim and phase ages, `step`, `step_age_ms`, `waiting_on`, `owner`, `resumes_on_its_own` and a `why`. `claimStall` is the `preparation_overdue` verdict (`claim.stall`) for a preparing claim past its budget. `EXPIRED_PREPARING_CHARGE_SQL` is the one rule an expired-claim reclaim charges a preparation attempt by: only a claim stamped `preparing` under the token being reclaimed. `persistence.max_claim_ms` (default 600000, 60000 to 86400000; `validateMaxClaimConfigValue` is `config set`'s `invalid_params` refusal) is doctor `persistence_write_stall`'s threshold. Consumer and group-publish store the stamp on every renewal, so writer status (`diagnostics.ts`, `claim` on each running blocker, never the token) and doctor name a stuck phase and step without an extra statement per write. Tests: `test/persistence-write-stall.test.ts`, `test/persistence-claim-step.test.ts`.
- `src/core/persistence/preparation-budget.ts` + `src/commands/config/preparation-keys.ts` — #6278: how long a claimed write may prepare and how many cut-offs it survives. `preparationKind` sorts a request into `foreground` (`remember`, and `put_page`/`edit_page` with no internal intent: the consumer's 30 s `preparationMs`), `sync` (`managed_sync_*` kinds: `persistence.sync_preparation_ms`, default 120000) or `maintenance` (every other kind: `persistence.maintenance_preparation_ms`, default 120000). `persistence.preparation_ceiling_ms` (default 600000, 60000 to 3600000, floored at the larger budget plus 30 s) is the hard ceiling for a preparation that ignores cancellation and `persistence.max_preparation_attempts` (default 2, 1 to 10) the cut-offs before a request finishes `preparation_stalled`; a missing or out-of-range value keeps its default (`resolvePreparationPolicy`). These keys and `persistence.max_transient_retries` (transient-retry.ts) ride the write-switch snapshot (`switches.ts`, one statement, 5 s TTL) together with the `preparation_deadlines` kill switch, so a claim never reads config on its own. `startPreparation` runs a preparer under its budget and the budget wins the race even when the preparer ignores its signal (the signal aborts with `{ code: 'preparation_deadline' }`; a result that lands after the budget on the clock still reads as a deadline; the late result is the caller's to drop); `racePreparation` is its outcome-only form for the sync's out-of-consumer preparations (`sync-waivers.ts`). `validatePreparationConfigValue` is `config set`'s structured refusal (each key's range, and the ceiling rule across the sibling keys; usage error exit 2, nothing written), wired in by `refuseInvalidPreparationValue`. `persistence.max_claim_ms` keeps its doctor meaning. Tests: `test/persistence-preparation-budget.test.ts`.
-- `src/core/persistence/bounded-reads.ts` — #6278 (plan item 1.4): a preparation read that can wait on a relation lock ends at the budget on the server. Phase 0 (`docs/eval/managed-sync-stall-repro.md`) showed that through a transaction-mode pooler gbrain's `statement_timeout` startup parameter is dropped, so a preparation read outside the publication transaction had no server-side bound; an `ACCESS EXCLUSIVE` lock on `pages` held elsewhere pinned the member's origin check and its connection until the lock lifted, and with the budget alone the released member's statement kept running as a zombie that blocked its root until the ceiling. `boundedReads(engine, clock)` is the engine view every preparer reads through (`prepareManagedSyncMutation` and its helpers, `preparePageMutation`, `prepareFactFenceAdoption`, the sync's out-of-consumer preparations): each unmemoized raw statement runs with `executeRaw`'s `timeoutMs` set to the clock's remaining budget (`ClaimPhaseClock.deadlineAt`, set by `startClaimPhase(now, signal, budgetMs)`) and the clock's signal, which ends only the wait for a free connection. On Postgres `timeoutMs` runs the statement on a reserved connection as `BEGIN; SET LOCAL statement_timeout; <statement>; COMMIT` in one pipelined round trip, so the bound holds through the pooler and `SET LOCAL` ends with it; inside a transaction and on PGLite the option is ignored. A statement the server ends (57014, or a 55P03 raised by a shorter session `lock_timeout`) surfaces as the member's own `preparation_deadline` (`PreparationDeadlineError`, or the signal's reason once it aborted), which `preparationAbortReason` classifies as the deadline: the consumer and the group route handle it as the deadline even inside the budget (charged release, `preparation_stalled` at the limit, the group cut like the timer), never as a `storage_error` receipt or an uncharged release. The memo rule holds: `preparationReads` answers a bounded read once with the member's bound and signal dropped. Engine helpers that bypass `executeRaw` (`readPageSnapshot`, the import pipeline) stay under the consumer's race and the ceiling. Tests: `test/persistence-bounded-reads.test.ts`, `test/e2e/persistence-preparation-lock-wait-postgres.test.ts` (a real lock held in another session), the server-ended cases in `test/persistence-consumer-scheduling.test.ts` and `test/e2e/persistence-group-deadline-postgres.test.ts`. GBRA-75 wave 9: `withBoundedReadSession` gives a waiver screen (`unchangedSyncImport`) one reserved ordinary-pool connection (a long-hold permit; none left, or PGLite, runs the screen as before) on which its bounded reads share one transaction: `BEGIN; SET LOCAL statement_timeout` once, `COMMIT` when the screen settles. The bound stays per statement on the server: it is set again in the read's pipeline once the bound in force differs from the read's own by more than `BOUNDED_SESSION_SLACK_MS` (25 ms). A waiver run (`waiveRun` in `sync-run.ts`) opens one session for all its screens, and a screen inside it joins it. Session reads are named prepared statements (`ReservedConnection.executeRaw` `prepare`, ANDed with the connection option as `runUnsafe` does). A failed read rolls back and the reads it aborted (25P02) run again in a new transaction; a read after the screen settled takes the per-read path. Test: `test/persistence-bounded-read-session.test.ts` (real lock wait and no remaining statement on Postgres).
+- `src/core/persistence/bounded-reads.ts` — #6278 (plan item 1.4): a preparation read that can wait on a relation lock ends at the budget on the server. Phase 0 (`docs/eval/managed-sync-stall-repro.md`) showed that through a transaction-mode pooler gbrain's `statement_timeout` startup parameter is dropped, so a preparation read outside the publication transaction had no server-side bound; an `ACCESS EXCLUSIVE` lock on `pages` held elsewhere pinned the member's origin check and its connection until the lock lifted, and with the budget alone the released member's statement kept running as a zombie that blocked its root until the ceiling. `boundedReads(engine, clock)` is the engine view every preparer reads through (`prepareManagedSyncMutation` and its helpers, `preparePageMutation`, `prepareFactFenceAdoption`, the sync's out-of-consumer preparations): each unmemoized raw statement runs with `executeRaw`'s `timeoutMs` set to the clock's remaining budget (`ClaimPhaseClock.deadlineAt`, set by `startClaimPhase(now, signal, budgetMs)`) and the clock's signal, which ends only the wait for a free connection. On Postgres `timeoutMs` runs the statement on a reserved connection as `BEGIN; SET LOCAL statement_timeout; <statement>; COMMIT` in one pipelined round trip, so the bound holds through the pooler and `SET LOCAL` ends with it; inside a transaction and on PGLite the option is ignored. A statement the server ends (57014, or a 55P03 raised by a shorter session `lock_timeout`) surfaces as the member's own `preparation_deadline` (`PreparationDeadlineError`, or the signal's reason once it aborted), which `preparationAbortReason` classifies as the deadline: the consumer and the group route handle it as the deadline even inside the budget (charged release, `preparation_stalled` at the limit, the group cut like the timer), never as a `storage_error` receipt or an uncharged release. The memo rule holds: `preparationReads` answers a bounded read once with the member's bound and signal dropped. #6318: other calls through the view run under `withReadDeadline` (`read-deadline.ts`), so the engine's own reads (`readPageSnapshot`, the import pipeline) get `statement_timeout` at the time left. Tests: `test/persistence-bounded-reads.test.ts`, `test/e2e/persistence-preparation-lock-wait-postgres.test.ts` and `-engine-reads-lock-postgres` (a real lock held in another session), the server-ended cases in `test/persistence-consumer-scheduling.test.ts` and `test/e2e/persistence-group-deadline-postgres.test.ts`. GBRA-75 wave 9: `withBoundedReadSession` gives a waiver screen (`unchangedSyncImport`) one reserved ordinary-pool connection (a long-hold permit; none left, or PGLite, runs the screen as before) on which its bounded reads share one transaction: `BEGIN; SET LOCAL statement_timeout` once, `COMMIT` when the screen settles. The bound stays per statement on the server: it is set again in the read's pipeline once the bound in force differs from the read's own by more than `BOUNDED_SESSION_SLACK_MS` (25 ms). A waiver run (`waiveRun` in `sync-run.ts`) opens one session for all its screens, and a screen inside it joins it. Session reads are named prepared statements (`ReservedConnection.executeRaw` `prepare`, ANDed with the connection option as `runUnsafe` does). A failed read rolls back and the reads it aborted (25P02) run again in a new transaction; a read after the screen settled takes the per-read path. Test: `test/persistence-bounded-read-session.test.ts` (real lock wait and no remaining statement on Postgres).
- `src/core/persistence/owner-refusal.ts` — #6278: the `owner_unavailable` reasons matrix. `checkOwner(binding, incarnation, hostId)` names the first failing condition in a fixed order (`binding_missing`, `incarnation_changed`, `host_mismatch`, `transfer_in_progress` for a `draining` worktree, `clone_in_progress` for `recovering`, `local_path_missing`, `coordination_path_missing`) or returns the usable `ActiveOwnerBinding`; `ownerRefusal` gives each reason its message, `why`, actor, fix (always the read-only `gbrain sources writer status --source <id> --json`, with the same verify; `actor: provider` so the two in-progress reasons render `next: wait` with a 30 s delay, `host_admin` for the rest) and `retryable` (`host_mismatch` false at the site, carried by `OperationError.retryable`); `ownerUnavailableError` is the thrown form. `host_mismatch` prints both host ids' first 8 characters and, when `remote` is false, the `host.json` path this process read; nothing here claims or transfers ownership. Used by `maintenancePreflight`, `loadFenceSource` and `managedRepairRoot`; the registry's `owner_unavailable.reasons` and the `FENCE_REASONS` owner entries carry the same vocabulary. Tests: `test/owner-refusal.test.ts` (matrix, every preflight condition on a claimed worktree, the HTTP envelope without a local path), `test/fence-repair-reasons.test.ts`. `transfer_in_progress` is `wait` until the drain's `prepared_at` is `TRANSFER_STALE_MINUTES` (30) old, then `ask_user` naming the accept and the exact `transfer cancel` argv (a drain with no `prepared_at` keeps the wait).
- `src/core/persistence/repair-busy.ts` — #6278: the busy set a repair must not write while a managed sync runs. `loadRepairBusySet(engine, sourceId, incarnation)` returns the paths and slugs named by queued, running and recovering requests and terminal rows with an open recovery (both rename endpoints included) plus, for every unfinished `managed-sync` cursor of the incarnation, the entries of its frozen `managed-sync-manifest` from the cursor's index on (a path with a `sync-hold` record is left out: the sync retries a held file only when its bytes change); it never throws and fails closed (`unknown` set, so `repairBusy` answers true for everything) when a query fails or a cursor's manifest is missing. `repairBusyMessage` is the hold text, `repairBusyError` the `sync_in_progress` refusal (`reason` `candidate_in_flight` or `busy_set_unreadable`) `submitManagedFileRepair` raises at admission. Tests: `test/repair-busy.test.ts`, `test/repair-fences.test.ts`, `test/repair-fences-during-catchup.test.ts` (Postgres: `test/e2e/repair-fences-during-catchup-postgres.test.ts`).
- `src/core/persistence/consumer.ts` — resident canonical-write consumer with retained wake-ups, root-scoped retry backoff and locked-row-skipping expired-claim sweeps. A full tick fans out to every worker; when idle, each tick instead runs one work probe (a single statement, + the 5 s #6650 dead-owner scan, mirroring the claim, recovery, expired-claim, effect, projection (inline rows windowed by `projection-window.ts`) and topology predicates, excluding rows those workers would skip, including topology recoveries that failed to finish within the last 30 s) and the poll interval doubles from 250 ms to a 5 s cap (`idleMaxMs`), resetting on work. `wake()` runs a full tick immediately; `waitForWrite` calls it after a same-process admission commits, so cross-process writes wait at most 5 s plus one probe and no LISTEN is used. #5984: `awaitWrite` (`waitForWrite` is its row-only form) receives a request this consumer publishes through the `onSettled` handoff with no request read, and skips polls while `holds(id)` says the consumer has it claimed; otherwise it polls the narrow `WRITE_PROGRESS_SQL` (first poll after min(200 ms, half the wait), then 50 ms backing off to 250 ms) and reads the full row once when terminal. Its own cancelled reads are not failures; a database read failure ends the wait as a classified `read_failed` (auth/permission failures at once), and a pending head with an intervention `blocked_reason` ends as `blocked`. A probe failure is reported once and the next tick runs the full fan-out. On Postgres pools of three or more with no connection checked out, idle probes share one reserved connection from the ordinary pool, never the direct/session route (acquired and queried under the phase deadline; a timed-out lane returns to the pool once its query settles) so the rest of the pool drains through the 20 s `idle_timeout` (postgres.js reuses idle connections first-in-first-out); root refresh and receipt compaction run every 5 minutes while idle. Root refresh, recovery scan, sweep and capacity-marking SQL use cooperative five-second cancellation; failure never admits work under stale confinement. Cold direct initialization and queued transactions remain observed and awaited through shutdown. Recognized cancellation owned by the consumer's stop reason is not reported as a storage failure; deadline-first cancellation and unrelated storage errors remain visible. The rate-limited stderr line names the phase, the SQLSTATE or write-error code, a redacted one-line message of at most 200 characters, connection-wait evidence (`first_conn_ms`, or `checkout=not_observed conn_wait_ms`, attributed to the phase through AsyncLocalStorage over the engine's `onCheckout` seam) plus `loop_lag_ms`, a fix command and its docs anchor (`docs/ENGINES.md#persistence-consumer-log`); the trusted status `phase` carries the same fields. Every claimed write has a preparation budget (`preparation-budget.ts`): `execute` races the preparation against it (`startPreparation`), so the claim is released with `blocked_reason` `preparation_deadline` and one counted attempt at the budget even when the preparer ignores its signal, and a late result never publishes (the execution-token fence). A request already at `persistence.max_preparation_attempts` is finished `failed` with `preparation_stalled` at claim without preparing again, and the release that reaches the limit finishes it there (`coordinator.ts` `finishPreparationStalled`; a waiting `awaitWrite` caller gets the terminal receipt, foreground writes included); a preparer's rejection on the consumer's own abort (its reason, an `AbortError` or a cancelled statement, `preparationAbortReason`) is a release, never a receipt. The expired-claim sweep (Postgres, and PGLite's `releaseAbandonedClaims`) charges one attempt for a claim stamped `preparing` (a process killed mid-preparation); `claim_lost`, `consumer_stopping` and `group_member_waiting` never charge, and a commit resets the counter. A statement the database ended with 57014 (statement timeout or cancel: `coordinator.ts` `writeInterruption`) is charged too, in `finishUnpublishedFailure` and in `publishMutation`'s recovery route, and at the limit ends the request `failed` with `write_interrupted` instead of a `database_contention` release; contention with other transactions (55P03, 40001, 40P01), a dropped connection or a restarting database is never charged (`test/persistence-write-interrupted.test.ts`). The #5373 root barrier for an abandoned preparation ends at `persistence.preparation_ceiling_ms` measured from the claim: the root is freed, the counter is pinned at the limit (an overrun proves cancellation is ignored, so the next claim fails the request `preparation_stalled`) and the zombie stays in `status().outlived_ceiling`; at the per-process cap (`zombieCap`, default 1) the consumer claims nothing more and `status().restart_required` is true. `stop()` gives abandoned preparations `ABANDONED_STOP_GRACE_MS` (5 s) after aborting them instead of waiting forever. The fault point `consumer:preparing` sits inside the budget, `status()` reports `preparation_policy` and `abandoned_preparations`, and the stderr line carries `phase=preparation reason=deadline_exceeded | preparation_stalled | ceiling_exceeded`. With the `preparation_deadlines` switch off only `remember` and intent-less `put_page`/`edit_page` keep the 30 s budget, the old unbounded wait applies and no attempt is counted. Publication and shutdown retain the token/native-lock/recovery barriers. Trusted status exposes sampled process-local phases and attempts, not durable progress. A tick woken by a completed publication or a local admission claims the next write at once and runs root refresh plus recovery, expiry and topology scans at most once per poll interval; explicit, timer and idle-probe ticks always scan. Effects drain in batches of 20 without waiting for the next tick while full batches keep arriving; within a batch, git effects wait for their asynchronous durability probe (one per worktree root) after the other effects, and a single-file git effect in a repository without the durability hook acquires the worktree lock only to prove its root and records its outcome after releasing it. Each projection-worker invocation (`runResidentProjectionInvocation`, #5401) rebuilds up to 100 pages and starts no new page once 250 ms have passed since it began; when `hasClaimableWrite` (the `claimableWriteSql` predicate `claimNextWrite` claims with, in `journal.ts`) finds a claimable write at its start it takes 2 pages, and its selector skips failed rows for 30 s through the same `PROJECTION_RETRY_READY_SQL` fragment as the idle probe (PGLite, 1,000 pages: ~9.5 ms mean, ~18 ms p99 per page). On PGLite the first full tick releases claims last written before this process started (`releaseAbandonedClaims` in `effect-journal.ts`: running effects and unpublished requests without a recovery record), since the datastore admits one process and those owners have exited; Postgres keeps lease expiry (requests 30 s, effects 2 minutes); a provably dead owner's claim goes in 5 s (#6650). On PGLite, which has no autovacuum, the maintenance worker also runs `vacuumPersistenceQueues` (VACUUM ANALYZE of the queue tables) every minute or after 50 + 20% of the request table's rows in publications. Tests: `persistence-consumer-scheduling.test.ts`, `persistence-consumer-idle.test.ts`, `persistence-consumer-log.test.ts`, `projection-drain.test.ts` (injected clock and per-page cost), `persistence-root-refresh.test.ts`, `test/e2e/persistence-phase-liveness.test.ts`, `test/e2e/persistence-idle-pool.test.ts`. With `requestsOnly` (engine graduation's drain) the effect, projection, topology and maintenance workers never start; claim, recovery, expired-claim and publication run as usual and `stop()` remains the barrier. Both the single route and the grouped/lane route (`executeClaimedGroup` in `group-publish.ts`) prepare under a claim lease (`claim-lease.ts`: a renewal every `renewalIntervalMs`, default 10 s, each bounded by `phaseMs`); the claim is lost the first time a renewal says it is gone, throws or overruns, and `whileHeld` stops waiting on the preparation at that moment (the single route also aborts it). The claims still ours are released unpublished with `claim_lost` (token-fenced, so a consumer that took one over keeps it); the abandoned preparation is never published and keeps its root or lane slot until it settles or the ceiling passes, while the consumer slot frees at once. Renewals are never awaited on the way out: `stop()` drains in-flight renewals and gives abandoned preparations their grace before engine close. Tests: `test/persistence-claim-lease.test.ts`, `test/e2e/persistence-consumer-claim-loss.test.ts`, `test/persistence-preparation-kill-accounting.test.ts`, `test/e2e/persistence-preparation-deadline.e2e.test.ts`. #6355: `onLane` lends its warm lane's transaction through a wrapper that notes a connection loss (`isConnectionLoss`); in `finally` a lost lane is released and `idleLaneRetryAt` set a minute out, whatever `run` returned or threw (admission turns the loss into `write_outcome_unknown` or succeeds on the pool, so the catch alone never saw it), because the dead reservation answers every later statement with "Connection is no longer owned" (`test/persistence-consumer-lane-loss.test.ts`). #7106: `publishSingle` passes `projectionPending` (coordinator.ts `PublicationHooks`): a committed write that created its page (no page at its slug before the publication, and no additional page key such as a rename's) and whose outcome says `projection: 'pending'` (a new segmented page over SEGMENTED_CREATE_SEAL_MAX_BYTES, quarantined or not: `segmentedOutcome` in searchable.ts, recorded by the managed import, sync and put_page preparers) settles its waiter at commit, and its `afterCommit` projection goes to `detachBuild`, which runs at most `DETACHED_PROJECTION_BUILDS` (1) at a time, queued builds in order, each tracked in `outlived` so stop() waits for queued and running builds; it holds neither the reply nor the root. An edit (a deferred retirement included) and a publication without the hook await afterCommit as before (`test/large-pages/managed-import-projection-pending.test.ts`).
diff --git a/scripts/bench/managed-sync-stall-repro.ts b/scripts/bench/managed-sync-stall-repro.ts
index ee18d6ccdd..4bd17a0c28 100644
--- a/scripts/bench/managed-sync-stall-repro.ts
+++ b/scripts/bench/managed-sync-stall-repro.ts
@@ -119,6 +119,12 @@ const CHAOS_FOR_MS = Number(flag('chaos-for', '0')) * 1000;
const CHAOS_STREAM = flag('chaos-stream', 'downstream');
/** `--chaos-kind lock`: instead of a toxic, one connection through the pooler opens a transaction, runs `--chaos-lock` and holds it (a leaked or pending table lock behind a pooler that dropped statement_timeout). */
const CHAOS_LOCK = flag('chaos-lock', 'LOCK TABLE pages IN ACCESS EXCLUSIVE MODE');
+/**
+ * #6318: `--sync-preparation-ms <ms>` (0 = the product default) sets `persistence.sync_preparation_ms` before the passes, so a
+ * `--chaos-kind lock` run reaches the budget sooner. The report's `lock_release` lists, per lock episode, the sync process's
+ * statements that waited on a lock longer than the budget plus 3 s while it was held (the assertion is that there are none).
+ */
+const SYNC_PREPARATION_MS = Number(flag('sync-preparation-ms', '0'));
/** Phase 4.1: `--chaos-repeat N` holds the lock N times (default once), `--chaos-gap <seconds>` apart, so the second hold lands on a run that already recovered once. */
const CHAOS_REPEAT = Number(flag('chaos-repeat', '1'));
const CHAOS_GAP_MS = Number(flag('chaos-gap', '300')) * 1000;
@@ -389,19 +395,21 @@ function append(file: string, record: Record<string, unknown>): void { appendFil
async function sampleOnce(row: Row, deep: boolean): Promise<Record<string, unknown>> {
const t = Date.now();
- const via = row.poolerUrl ?? row.directUrl;
+ // #6318: under a held lock the pooler's server connections are pinned, so a lock run samples directly (the sampler never waits on the pooler).
+ const via = CHAOS_KIND === 'lock' ? row.directUrl : row.poolerUrl ?? row.directUrl;
const pooled = await admin(via, async sql => ({
running: await sql.unsafe(SAMPLE_SQL.running), activity: await sql.unsafe(SAMPLE_SQL.activity), states: await sql.unsafe(SAMPLE_SQL.states),
committed: Number((await sql.unsafe(SAMPLE_SQL.committedSync))[0]!.n),
...(deep ? { failed: await sql.unsafe(SAMPLE_SQL.failed), holds: await sql.unsafe(SAMPLE_SQL.holds) } : {}),
}), false).catch(error => ({ error: String(error instanceof Error ? error.message : error) }));
const locks = await admin(harness.adminUrl, sql => sql.unsafe(SAMPLE_SQL.locks, [row.db])).catch(error => [{ error: String(error) }]);
- const sample: Record<string, unknown> = { t, at: new Date(t).toISOString(), pass: state.pass?.pass ?? null, sync_pid: state.pass?.pid ?? null, via: row.poolerUrl ? 'pooler' : 'direct', ...pooled, locks };
+ const sample: Record<string, unknown> = { t, at: new Date(t).toISOString(), pass: state.pass?.pass ?? null, sync_pid: state.pass?.pid ?? null, via: via === row.poolerUrl ? 'pooler' : 'direct', ...pooled, locks };
if ('committed' in pooled && typeof pooled.committed === 'number') {
if (pooled.committed !== state.committed) { state.committed = pooled.committed; state.committedChangedAt = t; }
sample.committed_stale_ms = t - state.committedChangedAt;
}
- if (deep) {
+ // A writer status probe reads pages, so in a lock run it would wait out the held lock and stall the sampler.
+ if (deep && CHAOS_KIND !== 'lock') {
const status = await runCli(row.home, ['sources', 'writer', 'status', '--probe', '--json'], { GBRAIN_SQL_TRACE: row.trace, GBRAIN_SQL_TRACE_LABEL: 'writer-status' }, 120_000);
sample.writer_status = status.json ?? { exit: status.code, stdout: status.stdout.slice(-2000), stderr: status.stderr.slice(-2000) };
}
@@ -623,6 +631,25 @@ async function fenceRepairOnce(row: Row, stop: () => boolean): Promise<void> {
log(`fence_repair: exit=${r.code} status=${phase?.status ?? '?'} ${round1(r.wallMs / 1000)}s: ${String(phase?.summary ?? '').slice(0, 300)}`);
}
+/**
+ * #6318: per `--chaos-kind lock` episode, the sync process's statements (its `application_name` carries its pid) seen
+ * waiting on a lock for longer than the preparation budget plus 3 s in the samples taken while the lock was held.
+ */
+function lockRelease(events: Array<Record<string, unknown>>): Array<Record<string, unknown>> {
+ if (CHAOS_KIND !== 'lock' || !existsSync(join(OUT, 'samples.jsonl'))) return [];
+ const budgetMs = SYNC_PREPARATION_MS > 0 ? SYNC_PREPARATION_MS : 120_000;
+ const samples = readFileSync(join(OUT, 'samples.jsonl'), 'utf8').split('\n').filter(Boolean).map(line => JSON.parse(line) as Record<string, unknown>);
+ return events.filter(e => e.action === 'lock_held').map(held => {
+ const released = events.find(e => e.action === 'lock_released' && e.episode === held.episode);
+ const window = samples.filter(s => Number(s.t) >= Number(held.t) && Number(s.t) < (released ? Number(released.t) : Infinity));
+ const waiting = window.map(s => ({ at: s.at, statements: ((s.activity as Array<Record<string, unknown>> | undefined) ?? [])
+ .filter(a => a.wait_event_type === 'Lock' && Number(a.query_age_ms) > budgetMs + 3000 && s.sync_pid != null && String(a.application_name ?? '').includes(`:${s.sync_pid}:`))
+ .map(a => ({ wait: `${a.wait_event_type}/${a.wait_event}`, query_age_ms: a.query_age_ms, xact_age_ms: a.xact_age_ms, query: String(a.query ?? '').slice(0, 120) })) }));
+ return { episode: held.episode, budget_ms: budgetMs, samples: window.length,
+ max_waiting: Math.max(0, ...waiting.map(w => w.statements.length)), waiting: waiting.filter(w => w.statements.length) };
+ });
+}
+
/** Holds by code and reason, from the hold records themselves (the status listing is bounded). */
async function holdSummary(row: Row): Promise<Record<string, unknown>> {
return admin(row.directUrl, async sql => {
@@ -923,6 +950,7 @@ try {
via_pooler: await admin(row.poolerUrl ?? row.proxyUrl, showTimeouts, false, resolveSessionTimeouts()).catch(e => String(e)),
direct: await admin(row.directUrl, showTimeouts, true, resolveSessionTimeouts()).catch(e => String(e)) };
log(`session timeouts: ${JSON.stringify(report.session_timeouts)}`);
+ if (SYNC_PREPARATION_MS > 0) await mustCli(row.home, ['config', 'set', 'persistence.sync_preparation_ms', String(SYNC_PREPARATION_MS)]);
save();
const stop = () => stopped;
chaosRow = row;
@@ -962,7 +990,7 @@ try {
const doctorChecks = (doctorFences.json?.checks as Array<Record<string, unknown>> | undefined) ?? [];
const fenceCheck = doctorChecks.find(c => c.name === 'fence_integrity' || c.check === 'fence_integrity' || c.id === 'fence_integrity') ?? doctorChecks[0] ?? null;
const unrenderable = ((fenceCheck?.details ?? fenceCheck?.data ?? fenceCheck ?? {}) as Record<string, unknown>).unrenderable_legacy_facts as { total?: number; complete?: boolean; pages?: Array<{ slug: string; rows: Array<{ class: string; reason: string }> }> } | undefined;
- report.result = { committed_sync_requests: state.committed, chaos: chaosEvents, wedges: state.wedges, stall_debug_last: stallDebug.at(-1) ?? null, backlog_entries: backlogPages.length, states: final.states, stalls: state.stalls,
+ report.result = { committed_sync_requests: state.committed, chaos: chaosEvents, lock_release: lockRelease(chaosEvents), wedges: state.wedges, stall_debug_last: stallDebug.at(-1) ?? null, backlog_entries: backlogPages.length, states: final.states, stalls: state.stalls,
preparing_claims_observed: preparingClaimsObserved(),
doctor_fence_integrity: { exit: doctorFences.code, status: fenceCheck?.status ?? null, unrenderable_total: unrenderable?.total ?? null, unrenderable_complete: unrenderable?.complete ?? null,
unrenderable_by_class: Object.fromEntries([...(unrenderable?.pages ?? []).flatMap(p => p.rows).reduce((m, r) => m.set(`${r.class}/${r.reason}`, (m.get(`${r.class}/${r.reason}`) ?? 0) + 1), new Map<string, number>())]),
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index b2e11fd8fb..2649e35d7c 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -21,7 +21,7 @@ src/core/import-file.ts 2079 ratchet +2 #7880: importFromContent's holdNoncanoni
src/core/migrate.ts 602 ratchet schema-migration runner only; migrations live in src/core/schema-migrations/ (W3); +3 #5216: resumable page revision backfill runs after every migration pass while pages.knowledge_revision is nullable
src/core/operations.ts 245 ratchet +1 #7168: graphRankOperations import (ops/graph-rank.ts; spread shares the orphans line); peel target: containment sprint C4-C7; grown v0.46.26.0 megawave op wiring; +chat-connectors wiring (master merge); +5 v0.47 gmail-loops: loopsOperations spread + OP_AREAS; #5105: receipt operation registry wiring; shared-skills integration on company-ingestion base; +2 security wave: withOutputRedaction applied at op registration; +1 #5616: pageEditOperations import (spread shares the pages line); +1 Foundations 1 F1b: attributionOperations import (spread shares the admin line); +1 agent operator wave A6: ops/notices.ts import (mute_notice); +2 agent operator closeout A1: registerOpRoutes import + call (routing logic in fix-routing.ts); +2 retrieval feedback: feedbackOperations import and spread (ops/feedback.ts); +1 #6007: pageBatchOperations import (spread shares the pages line); +1 P5: wanted_pages area row; lowered GBRA-75 W11 cold start: OP_AREAS + finalize step moved to operation-load.ts (shared with the CLI per-op loader); +1 C4 pinned questions: questionsOperations import and spread (ops/questions.ts; its areas live in operation-load.ts OP_AREAS); +1 Wave B append: pageAppendOperations import (spread shares the pages line); +1 Wave B upload sessions: pageUploadOperations import (spread shares the pages line); +1 #7191: search-exact import; +1 GBRA-156: whois import (ops/whois.ts; spread shares the chunks line)
src/core/pglite-engine.ts 3097 ratchet +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2983 measured); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2983 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +5 GBRA-67: readPageSnapshotsBatch delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages delegate (SQL in engine-sql/pages.ts); +2 GBRA-76: transaction/transactionDirect take TransactionOptions and refuse an already-aborted signal before BEGIN; +2 #4772 (W14 PR3): entityTypes HealthDeps delegate + import (policy in schema-pack/entity-types.ts, SQL in engine-sql/health.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3023 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates (SQL in engine-sql/pages.ts); +6 large pages Wave A sub-a: getChunkPage delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +7 LHF wave 2 item 5: request config snapshot delegate lines in getConfig/getAllConfig/setConfig/unsetConfig (logic in config-snapshot.ts); +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3072 + 3; +5 PR2 fix lane X: keywordBoundedRead runs the read on the engine itself (engine methods cannot move): 3075 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3080 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3085); +2 #7159: the frontmatter where clause in searchKeyword and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); +2 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity with an orFallback rank parameter like Postgres (logic in search/keyword-or-rarity.ts; engine methods cannot move); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3097 measured)
-src/core/postgres-engine.ts 3155 ratchet +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); ceiling rationale: docs/architecture/key-files/engines-2.md entry for src/core/postgres-engine.ts; evidence delivery: getChunkWindows scoped-read delegate + type import (+5; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +10 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3242 #5824: searchVector SQL moved to search/vector-statement.ts, +explainVectorSearch/runVectorAttempt (-156); +6 #5730: onpoisoned wiring on the instance/module/connection-manager pools + poisonedDiscards in getPoolDiagnostics (counter in pool-gauge.ts); +5 #5801: onCheckout observer seam (fires after reserve resolution / transaction-callback entry; listeners in pool-gauge.ts); lowered to 3119 #5889: searchTitles SQL moved to engine-sql/titles.ts (-133); lowered to 2914 F4a: getHealth SQL moved to engine-sql/health.ts (-216); +2 #5984: transactions track held page guards like PGLite (withHeldPageKeys), so repeated guards in one transaction cost no round trips; +1 #5984: traceSqlOptions import for the env-gated GBRAIN_SQL_TRACE bench hook (src/core/sql-trace.ts); v0.60.48.0 merge: #5984 growth re-summed over v0.60.47.0 (2917 measured); +2 engine graduation (GBRA-50 G2a): initSchema sizing parameter (the source embedding layout sizes a fresh target); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2920 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); 6 #6085: initSchema pins advisory lock 42, bootstrap, schema replay and unlock to one reserved backend when the DDL pool has more than one slot (gauge only, no long-hold permit), released in finally; wave 10 collector: lane E (#6085), lane G (#5375) and master compose at 2939; +2 #6110: search transactions pass jitOff to withScopedReadTransaction (SET LOCAL jit = off in postgres-engine/search-settings.ts); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +1 #5984 single-write fast path composes with the lane apply diet at 2944; +2 #5984 shared parameter types: the instance pool passes shared_types and initSchema clears them after migrations (helpers in db.ts); +2 #6276: the remote alias-resolving readPageSnapshot runs with JIT off (withScopedReadTransaction jitOff); +5 #6278: executeRaw timeoutMs delegate (statement in postgres-engine/bounded-statement.ts); +23 #6278 PR 1.5: cancelSettleMs() and the settle timer in runUnsafe that discards a reserved connection whose cancel a transaction-mode pooler never completed; +3 #6317 lane 2: getPoolDiagnostics().pool from the vendored driver (postgres-engine/pool-stats.ts) and the prepare mark; +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +1 GBRA-69 extract batching: replaceDerivedLinksBatch delegate (batch logic in derived-links.ts); +9 GBRA-75: getBacklinkCounts and getHealth statements run JIT-off through the private jitOffRead transaction helper (engine methods cannot move); +3 GBRA-75 wave 7: searchKeyword and searchVector start the once-per-engine background search statistics guard (logic in search/projection-statistics.ts); +7 GBRA-67: readPageSnapshotsBatch scoped-read delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages scoped-read delegate (SQL in engine-sql/pages.ts); +7 #6383: poolIncidents hooks on the instance/module/connection-manager pools (counter and lines in pool-gauge.ts, budget in db.ts) and stuckDiscards/buildFailures in getPoolDiagnostics; +3 GBRA-76: transaction/transactionDirect take TransactionOptions; transactionOn attaches the begin handle to withTransactionAbort (logic in postgres-engine/transaction-abort.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +1 #5227: schema.sql replay under withSchemaLockTimeout (wave 14 PR1); +1 driver desync tripwire: desyncDiscards in getPoolDiagnostics (counter and line in pool-gauge.ts); +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages scoped-read delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3027 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates, reads scoped (SQL in engine-sql/pages.ts); +12 wave 16 G5.1 (#6495): keywordStatementTimeout memo field, its connect/disconnect resets and the keywordStatementTimeoutSetting accessor, plus one awaited setting per keyword/chunk/CJK transaction (resolver in postgres-engine/keyword-statement-timeout.ts); wave 16 G5.1 gate fix: keywordStatementTimeoutSetting resolves the knob once per connection on the first keyword statement from the saved connect config (connect issues no statement for it; 3042 measured); +7 large pages Wave A sub-a: getChunkPage scoped-read delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +3 wave 18b lane 2 (#6495): keywordTimeouts negative-cache field and the searchKeyword runner wrap (logic in postgres-engine/keyword-timeout-cache.ts); integrated at wave 18b: master 3082 + 3; +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3085 + 3; +3 PR2 fix lane E (E1): getChunks/getChunkPage/getChunkWindows run the manifest privilege probe in their scoped transaction (chunkManifestReads in search/safe-chunks.ts); integrated at the lane E merge onto b75a69a6d: 3088 + 3; +6 PR2 fix lane X: searchVector, explainVectorSearch, the hasMore witness and runVectorAttempt pick the chunk-manifest statement form inside each bounded attempt transaction, so its statement_timeout bounds the probe (form memo vectorStatementForms in search/vector-statement.ts; engine methods cannot move): 3091 + 6; +5 PR2 fix lane X: keywordBoundedRead runs the read in withScopedReadTransaction under the keyword statement timeout setting (engine methods cannot move): 3097 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3102 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3107); +3 #7159: the frontmatter where clause in the two keyword typesClause sites and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); GBRA-156 retrieval wave (#7201) composed with #7194 at the merge: re-summed (3114 measured); +2 warm CLI socket (#7218): connectedDatabaseUrl getter (the URL the engine connected with, bound into the socket identity), composed over the GBRA-156 retrieval wave (3116 measured); +1 GBRA-133 PR 2b: transactionOn hands the outermost transaction its commit checks (page-state/transactions.ts withCommitChecks), composed with the warm CLI socket at the merge (3117 measured); +18 GBRA-156: the private readTransaction helper behind withScopedReadTransaction and jitOffRead answers a top-level read before its COMMIT settles (engine methods cannot move; 3135 measured); +1 GBRA-156: transactionOn and runUnsafe hold a per-search connection slot (import of search/connection-budget.ts; 3136 measured); +2 GBRA-156: transactionOn passes TransactionOptions.pipelineFirst to the pool begin as { first } (driver gate in vendor/postgres index.js begin); an engine method cannot move; composed with the warm CLI socket and GBRA-133 PR 2b at the master v0.60.344.0 merge (3119 measured); GBRA-151 #7183/#7187/#7188 re-summed at the v0.60.402.0 merge (3135 measured); +2 #7183 (GBRA-151) at the v0.60.412.0 merge: read-write-transactions import and transactionOn wraps its connection in readWriteTransactions (logic in postgres-engine/read-write-transactions.ts; engine methods cannot move): 3135 + 2; +7 GBRA-156: transactionOn opens readOnly transactions as BEGIN READ ONLY and a top-level read inherits its request signal, runUnsafe refuses an aborted request (logic in request-signal.ts; engine methods cannot move): 3137 + 7; +3 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity (logic in search/keyword-or-rarity.ts; engine methods cannot move); +2 wave-b PR 4 merged with #7218 (both sides added imports); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3155 measured)
+src/core/postgres-engine.ts 3158 ratchet +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); ceiling rationale: docs/architecture/key-files/engines-2.md entry for src/core/postgres-engine.ts; evidence delivery: getChunkWindows scoped-read delegate + type import (+5; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +10 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3242 #5824: searchVector SQL moved to search/vector-statement.ts, +explainVectorSearch/runVectorAttempt (-156); +6 #5730: onpoisoned wiring on the instance/module/connection-manager pools + poisonedDiscards in getPoolDiagnostics (counter in pool-gauge.ts); +5 #5801: onCheckout observer seam (fires after reserve resolution / transaction-callback entry; listeners in pool-gauge.ts); lowered to 3119 #5889: searchTitles SQL moved to engine-sql/titles.ts (-133); lowered to 2914 F4a: getHealth SQL moved to engine-sql/health.ts (-216); +2 #5984: transactions track held page guards like PGLite (withHeldPageKeys), so repeated guards in one transaction cost no round trips; +1 #5984: traceSqlOptions import for the env-gated GBRAIN_SQL_TRACE bench hook (src/core/sql-trace.ts); v0.60.48.0 merge: #5984 growth re-summed over v0.60.47.0 (2917 measured); +2 engine graduation (GBRA-50 G2a): initSchema sizing parameter (the source embedding layout sizes a fresh target); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2920 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); 6 #6085: initSchema pins advisory lock 42, bootstrap, schema replay and unlock to one reserved backend when the DDL pool has more than one slot (gauge only, no long-hold permit), released in finally; wave 10 collector: lane E (#6085), lane G (#5375) and master compose at 2939; +2 #6110: search transactions pass jitOff to withScopedReadTransaction (SET LOCAL jit = off in postgres-engine/search-settings.ts); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +1 #5984 single-write fast path composes with the lane apply diet at 2944; +2 #5984 shared parameter types: the instance pool passes shared_types and initSchema clears them after migrations (helpers in db.ts); +2 #6276: the remote alias-resolving readPageSnapshot runs with JIT off (withScopedReadTransaction jitOff); +5 #6278: executeRaw timeoutMs delegate (statement in postgres-engine/bounded-statement.ts); +23 #6278 PR 1.5: cancelSettleMs() and the settle timer in runUnsafe that discards a reserved connection whose cancel a transaction-mode pooler never completed; +3 #6317 lane 2: getPoolDiagnostics().pool from the vendored driver (postgres-engine/pool-stats.ts) and the prepare mark; +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +1 GBRA-69 extract batching: replaceDerivedLinksBatch delegate (batch logic in derived-links.ts); +9 GBRA-75: getBacklinkCounts and getHealth statements run JIT-off through the private jitOffRead transaction helper (engine methods cannot move); +3 GBRA-75 wave 7: searchKeyword and searchVector start the once-per-engine background search statistics guard (logic in search/projection-statistics.ts); +7 GBRA-67: readPageSnapshotsBatch scoped-read delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages scoped-read delegate (SQL in engine-sql/pages.ts); +7 #6383: poolIncidents hooks on the instance/module/connection-manager pools (counter and lines in pool-gauge.ts, budget in db.ts) and stuckDiscards/buildFailures in getPoolDiagnostics; +3 GBRA-76: transaction/transactionDirect take TransactionOptions; transactionOn attaches the begin handle to withTransactionAbort (logic in postgres-engine/transaction-abort.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +1 #5227: schema.sql replay under withSchemaLockTimeout (wave 14 PR1); +1 driver desync tripwire: desyncDiscards in getPoolDiagnostics (counter and line in pool-gauge.ts); +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages scoped-read delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3027 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates, reads scoped (SQL in engine-sql/pages.ts); +12 wave 16 G5.1 (#6495): keywordStatementTimeout memo field, its connect/disconnect resets and the keywordStatementTimeoutSetting accessor, plus one awaited setting per keyword/chunk/CJK transaction (resolver in postgres-engine/keyword-statement-timeout.ts); wave 16 G5.1 gate fix: keywordStatementTimeoutSetting resolves the knob once per connection on the first keyword statement from the saved connect config (connect issues no statement for it; 3042 measured); +7 large pages Wave A sub-a: getChunkPage scoped-read delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +3 wave 18b lane 2 (#6495): keywordTimeouts negative-cache field and the searchKeyword runner wrap (logic in postgres-engine/keyword-timeout-cache.ts); integrated at wave 18b: master 3082 + 3; +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3085 + 3; +3 PR2 fix lane E (E1): getChunks/getChunkPage/getChunkWindows run the manifest privilege probe in their scoped transaction (chunkManifestReads in search/safe-chunks.ts); integrated at the lane E merge onto b75a69a6d: 3088 + 3; +6 PR2 fix lane X: searchVector, explainVectorSearch, the hasMore witness and runVectorAttempt pick the chunk-manifest statement form inside each bounded attempt transaction, so its statement_timeout bounds the probe (form memo vectorStatementForms in search/vector-statement.ts; engine methods cannot move): 3091 + 6; +5 PR2 fix lane X: keywordBoundedRead runs the read in withScopedReadTransaction under the keyword statement timeout setting (engine methods cannot move): 3097 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3102 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3107); +3 #7159: the frontmatter where clause in the two keyword typesClause sites and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); GBRA-156 retrieval wave (#7201) composed with #7194 at the merge: re-summed (3114 measured); +2 warm CLI socket (#7218): connectedDatabaseUrl getter (the URL the engine connected with, bound into the socket identity), composed over the GBRA-156 retrieval wave (3116 measured); +1 GBRA-133 PR 2b: transactionOn hands the outermost transaction its commit checks (page-state/transactions.ts withCommitChecks), composed with the warm CLI socket at the merge (3117 measured); +18 GBRA-156: the private readTransaction helper behind withScopedReadTransaction and jitOffRead answers a top-level read before its COMMIT settles (engine methods cannot move; 3135 measured); +1 GBRA-156: transactionOn and runUnsafe hold a per-search connection slot (import of search/connection-budget.ts; 3136 measured); +2 GBRA-156: transactionOn passes TransactionOptions.pipelineFirst to the pool begin as { first } (driver gate in vendor/postgres index.js begin); an engine method cannot move; composed with the warm CLI socket and GBRA-133 PR 2b at the master v0.60.344.0 merge (3119 measured); GBRA-151 #7183/#7187/#7188 re-summed at the v0.60.402.0 merge (3135 measured); +2 #7183 (GBRA-151) at the v0.60.412.0 merge: read-write-transactions import and transactionOn wraps its connection in readWriteTransactions (logic in postgres-engine/read-write-transactions.ts; engine methods cannot move): 3135 + 2; +7 GBRA-156: transactionOn opens readOnly transactions as BEGIN READ ONLY and a top-level read inherits its request signal, runUnsafe refuses an aborted request (logic in request-signal.ts; engine methods cannot move): 3137 + 7; +3 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity (logic in search/keyword-or-rarity.ts; engine methods cannot move); +2 wave-b PR 4 merged with #7218 (both sides added imports); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3155 measured); +3 #6318: withScopedReadTransaction takes the read deadline of a bounded preparation (read-deadline.ts) and sets statement_timeout to the time left (engine methods cannot move): 3155 + 3
src/core/search/hybrid.ts 1562 ratchet +5 GBRA-81 item 5 per-stage search timings: HybridSearchOpts._timings (interface field) and the timer bindings and import in hybridSearch/hybridSearchCached (collector logic in search/stage-timings.ts); +2 W5 page-unit ranking cut: HybridSearchOpts.pageRankCut (logic in search/page-rank-cut.ts); +9 W3 confidence-gated reranking: HybridSearchOpts.rerankGate and the gate/lookup threading through rerankAndPin and sizeReturnPool (logic in search/hybrid/rerank-gate.ts, crag.ts gradePreRerank); re-summed at the W3 merge onto master v0.60.168.0 (1641 measured); +2 System One v1: HybridSearchOpts.decide (search/decide-stage.ts); refactor wave 1 split target (W4 hybrid, cut line); ceiling rationale: docs/architecture/key-files/core-search-1.md entry for src/core/search/hybrid.ts; +10 #5691: per-request embedding_query_prefix read once and threaded to the embeds and cache key; +5 green master wave (embedding opt-out): HybridSearchOpts._embeddingOptedOut (interface field) read once per request and threaded to the cache-lookup embed and the keyword-only gate; +24 P2 explain RRF attribution stamp in rrfFusionWeighted, backlink_count stamp, explain_target stage observations, explain/explainTarget opts; +14 fix wave 13 P4.8 #6400: lazyQueryEmbedDeadline (the query-embed deadline starts when the embed starts) and the _queryEmbedDeadline getter; +1 query facts arm: HybridSearchOpts.onQueryEmbedding (interface field; its one-line call is paid inside the ceiling) after the vector arms, so the facts arm reuses the query embedding without a second model call; re-summed at the master a08578ca5 merge (1591 master lines + 3 integration lines: the onQueryEmbedding hook); lowered to 1562: LHF wave 2 item 3 peeled cosineReScore into search/cosine-rescore.ts
src/core/types.ts 2351 ratchet +4 Wave B PR 1 (1c): Page.body_storage/body_bytes (interface fields; guard logic in page-body/page-shape.ts); +8 W3 confidence-gated reranking: HybridSearchMeta.rerank_gate (RerankGateMeta in search/hybrid/rerank-gate.ts); re-summed at the W3 merge onto master v0.60.168.0 (2279 measured); +7 fix wave 11 W6.2/W6.3 (#5989, #6132): SearchOpts.hnswIterativeScan and cjkKeyword, the keyword_candidates_incomplete stage and HybridSearchMeta.keyword_candidates (logic in engine-sql/cjk-search.ts); +5 System One v1 retrieval lane: SearchResult injection_p/injection_suspected, HybridSearchMeta answerability, egress_denied degraded reason; +4 System One v1: SearchResult rerank_score_kind/decide_evidence, HybridSearchMeta decide/rerank; +3 wave review: PAGE_SORT_SQL.updated_asc source_id + id tiebreakers (total order under federated scopes); +5 #4881: OntologyReadOpts extends PageReadScope doc; +3 #4857: PAGE_SORT_SQL.slug total-order tiebreaker comment; grown v0.46.26.0 megawave: EvidenceOpts, pacing/usage result fields, PageFilters growth (#4352 #4218); wave-k: +9 SalienceOpts/AnomaliesOpts source scope (#3867); +20 2026-09 adversarial batch 2: keyword_relaxed_carried stage + relaxed_dropped meta field docs; +16 2026-09 fix wave: SearchResult.keyword_relaxed field + full fusion-demotion doc; (#3867); +9: #4648 rerank_passthrough degraded stage + empty_result_set/malformed_shape reasons | merged v0.48.1.0 true-up; +master v0.48 merge true-up (reranker succession: no_key preflight, reranker_skipped stage, readiness); corrective release: shared internal PageReadScope and PageReadPolicy; ranker wave v0.48.4.0: HybridSearchMeta relational_rerank_pin / keyword_arm_confidence / metadata_boost_gate + SearchResult.relational_pinned (+48 over master's peel); +12: #4681 Page.updated_at_iso + keyset cursor precision docs; #5105: canonical page state shared module and transaction/snapshot delegates; #5105: public page revision fields; upstream integration: preserve independently reviewed master additions with branch facade extraction; #5252 wave: typed vector pool and readiness metadata; merged company ingestion/schema routing changes; retired provider integration removed; +1 write-path wave: 'created' effective-date source; +3 read-path wave: rerank_failed degraded stage + budget reason (closed vocabularies) and its recall classification note; +8 fw3-search: SearchOpts.source_boosts + keyword_arm_failed/title_arm_failed stages; +2 wave-2 W4: ChunkInput.embedding_input_hash (#5553); fix wave 3 x fix wave 2 (v0.60.5.0) composition: +2 at the merge; evidence delivery: optional SearchResult.delivered (+2; shape in search/evidence-delivery.ts); +2 eval wave N1-3: OntologyReadOpts.visibility (remote world-only ontology reads); +4 #5827: GraphNode.source_id and GraphPath from_source_id/to_source_id; +2 #5824: SearchOpts.vectorLegacyGuard rollback switch; +7 #5154: PageFilters.listColumnsOnly (interface field); +11 Foundations 1 F1b: WriteAttributionView + AttributedPageVersion (get_versions / get_write_attribution read shapes); +1 #5988: SearchResult.stale carries the held-file detail (interface field); +6 retrieval feedback: SearchResult feedback_boost/content_hash/relational_path_edges stamps and RelationalFanoutRow.path_edges (interface fields); +2 RelationalFanoutOpts.wide (interface field); 1 GBRA-49: temporal annotation field on link rows; 1 GBRA-49: temporal annotation field on link rows; 1 GBRA-49: temporal annotation field on link rows; +44 P7 ChainHopOpts/ChainHopEdge chain-hop contract types; P7 SearchResult.relational evidence + RelationalEvidence type + relational_plan meta + planner per-call overrides; +2 P7 SearchOpts.relationalChainSlots per-call override; +1 P7 ChainHopOpts.temporal (relationship-validity policy per hop); P3 x P7 (v0.60.60.0) composition: deltas re-summed at the merge; 1 GBRA-49: temporal annotation field on link rows; +44 P7 ChainHopOpts/ChainHopEdge chain-hop contract types; P7 SearchResult.relational evidence + RelationalEvidence type + relational_plan meta + planner per-call overrides; +2 P7 SearchOpts.relationalChainSlots per-call override; +1 P7 ChainHopOpts.temporal (relationship-validity policy per hop); +1 green master wave: embedding_disabled degraded reason (closed vocabulary); +2 #5951: Chunk.source_id (get_chunks names each row's source); 1 P2 SearchResult explain fields (rrf attribution, blend_norm_rrf, score_details, backlink_count) + RrfAttribution (interface fields); +11 #5575 (L1b): PageReadScope minTrust/suppressFlagged, SearchResult trust_tier/origin/unconfirmed, TimelineOpts eligibility (interface fields cannot move); +8 #4772 (W14 PR3): BrainHealth.entity_types_status field (interface fields cannot move); +1 C4: question page type row; +6 entity anchoring: optional SearchResult.entity_anchored row stamp (logic in search/entity-anchor.ts); +4 query facts arm: optional SearchResult.fact_row and superseded_claim row fields (logic in search/facts-arm.ts); +2 query facts arm: SearchResult result_type/fact_id/page_slug/follow_up (fact rows are never page-shaped); +9 Cat 40 Hard fix wave (#6271): SearchResult matched_alias/evidence_omitted/provenance, SearchOpts.rankQuery, keyword_count_unavailable stage and rerank_failed reasons; +12 fix wave 15 P2.1 (#6500): PageFilters.slugAfter (single-source slug keyset, interface field + doc) and the updated_desc tiebreaker comment (the keyset walk lives in facts/conversation-enumeration.ts); +2 wave 16 G5.1 (#6495): EngineConfig.keyword_statement_timeout_ms (interface field + doc); +2 GBRA-92: EngineConfig.pglite_shared_buffers (interface field + doc); +11 large pages (Wave A): Chunk/ChunkInput projected_byte_start/end interface fields (span logic in chunkers/recursive.ts, page-window/document.ts); +4 wave 18b lane 2 (#6495): SearchOpts.keywordTimeoutCache, the skipped_after_timeout degraded reason and the useCache doc line; integrated at wave 18b: master 2319 + 4; +2 LHF wave 2 item 3: retrieval meta cosine_rescore field (js_fallback disclosure; logic in search/cosine-rescore.ts); +14 GBRA-154 ranking wave (#4612 #7153 #7156 #7162 #7164): SearchResult collapsed/redirect_to/image_page/rank_prior/rank_score/fused_score stamps and HybridSearchMeta.collapse (interface fields; logic in search/collapse.ts, rank-priors.ts, score-contract.ts); GBRA-151 retrieval P0 x operability wave merge: both sides grew the file, re-summed (2343 measured); GBRA-156 retrieval wave (#7201) composed with #7194 at the merge: re-summed (2345 measured); GBRA-151 #7147 follow-up re-summed at the v0.60.338.0 merge (2347 measured); +2 GBRA-95 G7a follow-up: Page.withheld_source_path field; +2 #7600: SearchResult.quoted (interface field; logic in search/transcript-quote.ts)
src/core/minions/queue.ts 2547 ratchet grown #4078 + megawave probe-liveness unlatch (#4310) + master private-queue lifecycle: recovery classifier + monotonic lease; security: durable queued authority and confined filesystem dispatch; master integration: grown #4078 + megawave probe-liveness unlatch (#4310) + master private-queue lifecycle: recovery classifier + monotonic lease; harness support: boundary wiring to shared grants, isolated setup, withdrawal and delegated guards; merged v0.49: durable authority and delegated admission both enforced; typed readiness, configuration-blocked lifecycle and bounded release; +9: prune archives completed dream synthesis keys; +18 Lane D: deferJob (a deferral returns a job to delayed with no attempt counted and no stacktrace growth); the ceiling had 17 lines of slack; +5 agent operator follow-up Item 5 merged over master: claim-time spend fence token and legacy_default stamp (logic in minions/spend-authorization.ts); +5 GBRA-130 A-30: claim excludeWatchdogSuspects option (signature, doc and the mark parameter), the releaseWatchdogClaim delegate, and the co-runner stall call site in handleStalled (logic in minions/watchdog-attribution.ts); +5 GBRA-158 #7239/#7240: failJob result parameter (the #7216 hunk), jsonbSafeJson import and its call in add (sanitizer in text-safe.ts); +2 #7242 lane reservation: claim excludeNames option (doc line and the bound array; policy in minions/lanes.ts); +1 #7241 submitter label import (job-insight.ts) + 1 queue-families type import; #7252 x #7249 merge: both sides grew the file, re-summed (2544 measured); +3 GBRA-133: the stall sweep deletes a drain-lease holder record whose lease lock is gone or expired
diff --git a/src/core/persistence/bounded-reads.ts b/src/core/persistence/bounded-reads.ts
index 1a16055fae..1557e8f1a1 100644
--- a/src/core/persistence/bounded-reads.ts
+++ b/src/core/persistence/bounded-reads.ts
@@ -20,12 +20,17 @@
* The memo rule stands: a read `preparationReads` answers once for a whole
* group never takes one member's bound or signal (group-publish.ts drops
* them), and a statement that already carries a caller's signal keeps the
- * foreground path's cancellation instead of the bound. Engine helpers that bypass `executeRaw`
- * (`readPageSnapshot` and the import pipeline) stay under the consumer's race
- * and the ceiling. Without a deadline (switch off, a clock from an older
+ * foreground path's cancellation instead of the bound. Without a deadline (switch off, a clock from an older
* caller) or on PGLite (one in-process connection, no other session to wait
* on) the engine is returned as it is.
*
+ * #6318: the engine's own reads (`readPageSnapshot`, `readPageSnapshotsBatch`, `findDuplicatePage`, the import
+ * pipeline's lookups) bypass `executeRaw`. Every other method called through the view runs under the clock's
+ * deadline (read-deadline.ts `withReadDeadline`), so the Postgres engine's scoped reads it issues end on the server at
+ * the deadline too; one the server ended there (or a 55P03 a shorter session `lock_timeout` raised) is the member's
+ * `preparation_deadline`. A read a group memo shares (group-publish.ts `preparationReads`, import-reads.ts) takes the
+ * deadline of the member that sent it, and its failure stays a shared one.
+ *
* #6317 (C1): the same wrapper records each statement's label on the clock
* (`recordClaimSql`) before issuing it, so the claim stamp's `last_sql` names
* the read a parked preparation is waiting on.
@@ -33,6 +38,7 @@
import type { BrainEngine, ReservedConnection } from '../engine.ts';
import { PoolCapacityError } from '../pool-budget.ts';
import { READ_WRITE_BEGIN } from '../postgres-engine/read-write-transactions.ts';
+import { withReadDeadline } from '../read-deadline.ts';
import { recordClaimSql, type ClaimPhaseClock } from './claim-phase.ts';
import { registerEngineView, viewedEngine } from './switches.ts';
@@ -105,7 +111,15 @@ export function boundedReads(engine: BrainEngine, clock: ClaimPhaseClock | undef
}
};
const value = Reflect.get(target, key, target);
- return typeof value === 'function' ? value.bind(target) : value;
+ if (typeof value !== 'function') return value;
+ return (...args: unknown[]) => {
+ const result = withReadDeadline(deadlineAt, () => value.apply(target, args));
+ return result instanceof Promise ? result.catch(error => {
+ if (!isStatementTimeout(error) || isSharedReadFailure(error) || Date.now() < deadlineAt) throw error;
+ if (clock!.signal?.aborted) throw clock!.signal.reason;
+ throw new PreparationDeadlineError(clock!.step, error.code, error);
+ }) : result;
+ };
} }), viewedEngine(engine));
}
diff --git a/src/core/persistence/rows-behind.ts b/src/core/persistence/rows-behind.ts
index f524a0ee33..f53aa4f181 100644
--- a/src/core/persistence/rows-behind.ts
+++ b/src/core/persistence/rows-behind.ts
@@ -12,15 +12,19 @@
* is the one its last sync import produced (the database has not moved since) is `behind`; one the database moved since
* is `database_moved` and not reported (file-drift's territory); one with no record (or a record without a revision) is
* `unproven`, listed apart. Not reported either: held paths (the hold already says it), paths with no blob at the
- * checkpoint, non-Markdown pages, declared database-only pages, and paths that changed between the checkpoint and HEAD
- * (the next sync imports them).
+ * checkpoint, non-Markdown pages, declared database-only pages, paths sync does not read (`not_eligible`: outside the
+ * source's scope, matched by `sync.exclude`, hidden, outside the sync strategy or a reserved skillpack path), and paths
+ * that changed between the checkpoint and HEAD (the next sync imports them).
*
* Bounded: the checkpoint tree is listed once (and HEAD's only when it differs), pages are read in id batches of
- * ROWS_BEHIND_BATCH in read-only transactions, committed blobs are read in bounded `cat-file --batch` calls, and a blob
- * over the sync read bound (SYNC_READ_BOUND, the page ceiling) is never read: its row is reported with the
- * `file_too_large` refusal sync would hold it with. The deadline is checked before each batch and before each row's
- * content comparison and confirmation (`complete: false`; `pages` and `counts` cover the rows decided before
- * `stopped_at`, the page id the scan stopped at). `confirmUnproven: false` (doctor) counts rows with no import record
+ * ROWS_BEHIND_BATCH in read-only transactions, and a blob over the sync read bound (SYNC_READ_BOUND, the page ceiling) is
+ * never read: its row is reported with the `file_too_large` refusal sync would hold it with. #8093: a batch's committed
+ * blobs and export pages are read and compared in chunks of at most ROWS_BEHIND_BATCH_BYTES (one blob when a single one
+ * is larger), each chunk's buffers dropped once it is compared, and the differing rows keep at most that many bytes for
+ * their confirmation (the rest re-read their blob there). The deadline is checked before the per-source tree reads,
+ * before each batch and before each row's content comparison and confirmation (`complete: false`; `pages` and
+ * `counts` cover the rows decided before `stopped_at`, the page id the scan stopped at); no chunk is read after it
+ * passes. The confirmation's screen config and the export withdrawals are read once the first batch starts. `confirmUnproven: false` (doctor) counts rows with no import record
* from the content comparison alone, without running the importer's confirmation.
* Applies only on the owner host of a managed source with a checkout and a checkpoint; everything else, a repo-less
* brain included, is `not_applicable` with its reason. Read-only. Readers: doctor `sync_rows_behind_checkpoint` and
@@ -43,6 +47,8 @@ import { METAFILE_HEAD_BYTES } from '../sync.ts';
import { isSourceDbOnlySlug } from './source-storage.ts';
export const ROWS_BEHIND_BATCH = 256;
+/** #8093: the committed-blob bytes one chunk of a batch reads, and the bytes the batch's differing rows keep for confirmation. */
+export const ROWS_BEHIND_BATCH_BYTES = 64 * 1024 * 1024;
/** Rows one scan lists per class; counts go on past it (`truncated`). */
export const ROWS_BEHIND_LIST_LIMIT = 1000;
export const ROWS_BEHIND_DOCS = 'docs/guides/repair.md#rows-behind';
@@ -103,7 +109,7 @@ const empty = (sourceId: string, status: RowsBehindScan['status'], reason?: stri
const isMarkdown = (path: string) => /\.mdx?$/i.test(path);
export async function scanRowsBehind(engine: BrainEngine, sourceId: string,
- opts: { deadlineAt?: number; confirmUnproven?: boolean; now?: () => number } = {}): Promise<RowsBehindScan> {
+ opts: { deadlineAt?: number; confirmUnproven?: boolean; now?: () => number; batchBytes?: number } = {}): Promise<RowsBehindScan> {
if (!(await managedPersistenceEnabled(engine))) return empty(sourceId, 'not_applicable', 'not_managed');
const context = await sourceContext(engine, sourceId);
if (!context.ok) return empty(sourceId, context.status, context.reason);
@@ -111,6 +117,9 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string,
if (!source?.last_commit) return empty(sourceId, 'not_applicable', 'no_checkpoint');
const root = context.root;
try { isSourceDbOnlySlug(root, '', 'refuse'); } catch { return empty(sourceId, 'unknown', 'invalid_source_config'); }
+ const now = opts.now ?? Date.now;
+ const late = () => opts.deadlineAt !== undefined && now() >= opts.deadlineAt;
+ if (late()) return { ...empty(sourceId, 'known'), checkpoint: source.last_commit, complete: false };
const checkpoint = await readHeadTree(root, source.last_commit);
if (!checkpoint?.commit) return { ...empty(sourceId, 'unknown', 'checkpoint_unreadable'), checkpoint: source.last_commit };
const headRead = await readHeadTree(root, 'HEAD');
@@ -123,13 +132,13 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string,
const policy = { mirror: false, databaseOnlySource: false, dbOnly: (slug: string) => isSourceDbOnlySlug(root, slug, 'refuse'), ignored: new Set<string>() };
const scan: RowsBehindScan = { ...empty(sourceId, 'known'), checkpoint: checkpoint.commit, head: head.commit };
const count = (why: RowsBehindSkip) => { scan.counts[why] = (scan.counts[why] ?? 0) + 1; };
- const now = opts.now ?? Date.now;
- const late = () => opts.deadlineAt !== undefined && now() >= opts.deadlineAt;
- const screen = await repairScreenConfig(engine, sourceId);
- const withdrawals: ExportWithdrawal[] = await readExportWithdrawals(engine, sourceId);
+ const batchBytes = opts.batchBytes ?? ROWS_BEHIND_BATCH_BYTES;
+ let screen: Awaited<ReturnType<typeof repairScreenConfig>> | undefined;
+ let withdrawals: ExportWithdrawal[] | undefined;
const incarnation = context.owner.source_incarnation;
for (let cursor = '0'; ;) {
if (late()) { scan.complete = false; break; }
+ const exportWithdrawals = withdrawals ??= await readExportWithdrawals(engine, sourceId);
let stop: number | null = null;
const skipped: Array<[number, RowsBehindSkip]> = [];
const skip = (id: string, why: RowsBehindSkip) => { skipped.push([Number(id), why]); };
@@ -174,28 +183,40 @@ export async function scanRowsBehind(engine: BrainEngine, sourceId: string,
if (head !== checkpoint && head.blobs.get(gitPath(row.rel))?.oid !== row.blob.oid) { skip(row.key.id, 'changed_after_checkpoint'); continue; }
compare.push(row);
}
- const readable = compare.filter(row => row.blob.size <= SYNC_READ_BOUND);
- const bytes = await readBlobs(root, readable.filter(row => !row.key.segmented).map(row => row.blob));
- if (!bytes) throw new Error(`Git could not read committed files of source ${sourceId} at ${checkpoint.commit}`);
- const snapshots = await readExportPages(tx, readable.map(row => row.key), withdrawals);
- const byId = new Map(snapshots.map(snapshot => [String(snapshot.page.id), snapshot]));
- for (const row of compare) {
- if (late()) { stop = Number(row.key.id); break; }
- const recordState: RowBehind['record'] = records.get(Number(row.key.id))?.revision != null ? 'recorded' : 'none';
- if (row.blob.size > SYNC_READ_BOUND) { candidates.push({ ...row, record: recordState, content: null, compared: null }); continue; }
- const snapshot = byId.get(row.key.id);
- if (!snapshot) continue;
- const compared = row.key.segmented ? await compareSegmentedExportBlob(tx, snapshot, withdrawals, row.blob, checkpoint.format)
- : compareInlineExportFile(bytes.get(row.blob.oid)!, scannerSourcePath(root, `${root}/${row.rel}`, context.mode), snapshot);
- if (compared.state === 'equal') { skip(row.key.id, 'current'); continue; }
- candidates.push({ ...row, record: recordState, content: bytes.get(row.blob.oid) ?? null, compared });
+ const blobBytes = (row: (typeof compare)[number]) => row.blob.size > SYNC_READ_BOUND || row.key.segmented ? 0 : row.blob.size;
+ let kept = 0;
+ for (let start = 0; start < compare.length && stop === null;) {
+ let end = start, total = 0;
+ while (end < compare.length && (end === start || total + blobBytes(compare[end]!) <= batchBytes)) total += blobBytes(compare[end++]!);
+ const chunk = compare.slice(start, end);
+ start = end;
+ const readable = chunk.filter(row => row.blob.size <= SYNC_READ_BOUND);
+ const bytes = await readBlobs(root, readable.filter(row => !row.key.segmented).map(row => row.blob));
+ if (!bytes) throw new Error(`Git could not read committed files of source ${sourceId} at ${checkpoint.commit}`);
+ const snapshots = await readExportPages(tx, readable.map(row => row.key), exportWithdrawals);
+ const byId = new Map(snapshots.map(snapshot => [String(snapshot.page.id), snapshot]));
+ for (const row of chunk) {
+ if (late()) { stop = Number(row.key.id); break; }
+ const recordState: RowBehind['record'] = records.get(Number(row.key.id))?.revision != null ? 'recorded' : 'none';
+ if (row.blob.size > SYNC_READ_BOUND) { candidates.push({ ...row, record: recordState, content: null, compared: null }); continue; }
+ const snapshot = byId.get(row.key.id);
+ if (!snapshot) continue;
+ const compared = row.key.segmented ? await compareSegmentedExportBlob(tx, snapshot, exportWithdrawals, row.blob, checkpoint.format)
+ : compareInlineExportFile(bytes.get(row.blob.oid)!, scannerSourcePath(root, `${root}/${row.rel}`, context.mode), snapshot);
+ if (compared.state === 'equal') { skip(row.key.id, 'current'); continue; }
+ const content = bytes.get(row.blob.oid) ?? null;
+ const keep = content !== null && kept + content.length <= batchBytes;
+ if (keep) kept += content.length;
+ candidates.push({ ...row, record: recordState, content: keep ? content : null, compared });
+ }
}
return batch.length < ROWS_BEHIND_BATCH;
});
for (const candidate of candidates) {
if (stop !== null && Number(candidate.key.id) >= stop) break;
if (late()) { stop = Number(candidate.key.id); break; }
- const row = candidate.record === 'none' && opts.confirmUnproven === false ? unconfirmedRow(candidate) : await confirm(engine, sourceId, root, screen, candidate);
+ const row = candidate.record === 'none' && opts.confirmUnproven === false ? unconfirmedRow(candidate)
+ : await confirm(engine, sourceId, root, screen ??= await repairScreenConfig(engine, sourceId), candidate);
if (typeof row === 'string') { count(row); continue; }
const list = row.record === 'recorded' ? scan.behind : scan.unproven;
if (row.record === 'recorded') scan.behind_count++; else scan.unproven_count++;
diff --git a/src/core/persistence/sync-prepare.ts b/src/core/persistence/sync-prepare.ts
index 2b5f5b9e40..f051678d74 100644
--- a/src/core/persistence/sync-prepare.ts
+++ b/src/core/persistence/sync-prepare.ts
@@ -482,9 +482,11 @@ function syncFileProvenance(row: WriteRequest, p: SyncIntent, root: string, impo
/**
* The no-op of an entry whose newer working-tree bytes are the current page (`published`): it records, as its import
* provenance, those working-tree bytes at the page's current revision (#6712 PR 1), so the commit that later carries
- * them is prescreened; the pinned bytes it did not import are not recorded.
+ * them is prescreened; the pinned bytes it did not import are not recorded. #8089: the file is re-read at apply, so the
+ * record names it only when it still holds those bytes (its hash is the raw hash); otherwise nothing is recorded and the
+ * later commit is imported, never matched by bytes the page does not hold.
*/
-function publishedSyncNoop(row: WriteRequest, p: SyncIntent, root: string, snapshot: Snapshot | null, schemaPack: string | null,
+export function publishedSyncNoop(row: WriteRequest, p: SyncIntent, root: string, snapshot: Snapshot | null, schemaPack: string | null,
validate: (tx: BrainEngine) => Promise<void>, releaseHold: (tx: BrainEngine) => Promise<void>): SyncPreparedMutation {
const importProvenance: SyncImportProvenanceBase = { source_id: row.source_id, incarnation: row.source_incarnation, origin: p.sourcePath!, raw_sha256: p.rawHash!,
working: true, gbrain_version: VERSION, parser: syncImportParser(schemaPack), ...(p.fullEpisode ? { full_episode: p.fullEpisode } : {}) };
@@ -492,8 +494,9 @@ function publishedSyncNoop(row: WriteRequest, p: SyncIntent, root: string, snaps
apply: async tx => {
await releaseHold(tx);
const current = await tx.readPageSnapshot(row.slug, { sourceId: row.source_id, includeDeleted: true });
- if (current && current.page.deleted_at == null && current.revision === snapshot?.revision) {
- await recordSyncImportProvenance(tx, { ...importProvenance, page_id: Number(current.page.id), revision: current.revision, ...syncDiskProvenance(readSyncFile(root, p.path!), p.target) });
+ const disk = readSyncFile(root, p.path!);
+ if (current && current.page.deleted_at == null && current.revision === snapshot?.revision && disk && sha256(disk) === p.rawHash) {
+ await recordSyncImportProvenance(tx, { ...importProvenance, page_id: Number(current.page.id), revision: current.revision, ...syncDiskProvenance(disk, p.target) });
}
return { status: 'skipped', slug: row.slug, source_id: row.source_id, chunks: 0, noop: true, imported_file: true };
} };
@@ -507,7 +510,11 @@ function publishedSyncNoop(row: WriteRequest, p: SyncIntent, root: string, snaps
*/
export function syncDiskBytes(root: string, path: string, imported: Buffer, rawHash: string | null): Uint8Array | null {
if (rawHash !== null && rawHash === sha256(imported)) return imported;
- const disk = readSyncFile(root, path);
+ return heldDiskBytes(readSyncFile(root, path), imported);
+}
+
+/** #8071 and #8089: `disk` (null: absent) when it holds the imported bytes up to line endings, else the imported bytes. */
+export function heldDiskBytes(disk: Buffer | null, imported: Buffer): Uint8Array | null {
if (!disk || disk.equals(imported)) return disk;
const text = disk.toString('utf8'), lf = (value: string) => value.replace(/\r\n/g, '\n');
return disk.equals(Buffer.from(text)) && lf(text) === lf(imported.toString('utf8')) ? disk : imported;
diff --git a/src/core/persistence/sync-waivers.ts b/src/core/persistence/sync-waivers.ts
index 436039e77f..5478f257c7 100644
--- a/src/core/persistence/sync-waivers.ts
+++ b/src/core/persistence/sync-waivers.ts
@@ -8,7 +8,7 @@
import { join } from 'node:path';
import type { BrainEngine, PageSnapshot } from '../engine.ts';
import type { GBrainConfig } from '../config.ts';
-import { prepareManagedSyncMutation, type SyncIntent } from './sync-prepare.ts';
+import { heldDiskBytes, prepareManagedSyncMutation, type SyncIntent } from './sync-prepare.ts';
import { heldGitPaths, readSyncImportProvenances, syncImportParserFor, recordSyncImportProvenances, syncDiskProvenance, type SyncImportProvenance, type SyncImportProvenanceBase } from './sync-holds.ts';
import { sameParser } from './sync-prescreen.ts';
import { isSyncReadBound, isUnreadableSyncFile } from './sync-screen.ts';
@@ -347,13 +347,20 @@ function waiverRead(root: string, path: string): Buffer | null | undefined {
* #6712: the import provenance a waived import records in its waiver transaction, from the file read there: false when
* the file no longer has the frozen raw hash (the entry is not waived), null for a waiver that records none (a delete,
* or an import whose preparation carried no provenance, such as a code file).
+ * #8089: the disk half follows `syncDiskBytes`' rule. The imported bytes are the ones whose hash the provenance records
+ * (the frozen content, or for a `published` no-op the working file); the file read here is recorded only when it holds
+ * them (up to line endings), else the imported bytes are. When neither is in hand, nothing is recorded.
*/
-function waivedProvenance(cursor: WaiverCursor, pending: WaiverEntry, waived: NoopWaiver, revision: string): Omit<SyncImportProvenance, 'recorded_at' | 'validated'> | false | null {
+export function waivedProvenance(cursor: WaiverCursor, pending: WaiverEntry, waived: NoopWaiver, revision: string): Omit<SyncImportProvenance, 'recorded_at' | 'validated'> | false | null {
if (waived.kind !== 'import') return null;
const bytes = waiverRead(cursor.root, pending.intent.path!);
if (bytes === undefined || (bytes === null ? null : sha256(bytes)) !== pending.intent.rawHash) return false;
if (!waived.provenance) return null;
- return { ...waived.provenance, page_id: pending.pageId!, revision, ...syncDiskProvenance(bytes, pending.intent.target) };
+ const content = typeof pending.intent.content === 'string' ? Buffer.from(pending.intent.content) : null;
+ const imported = content && sha256(content) === waived.provenance.raw_sha256 ? content
+ : bytes && sha256(bytes) === waived.provenance.raw_sha256 ? bytes : null;
+ if (!imported) return null;
+ return { ...waived.provenance, page_id: pending.pageId!, revision, ...syncDiskProvenance(heldDiskBytes(bytes, imported), pending.intent.target) };
}
function softDeletedAt(snapshot: Awaited<ReturnType<BrainEngine['readPageSnapshot']>>, pending: WaiverEntry): boolean {
diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts
index 947cfc21f5..426c87b877 100644
--- a/src/core/postgres-engine.ts
+++ b/src/core/postgres-engine.ts
@@ -8,6 +8,7 @@ import { assertPageRevision } from './page-state/types.ts';
import { assertOriginUnclaimed, assertRevivalUnclaimed, classifyOriginUniqueViolation } from './persistence/origin-guard.ts';
import { lockUnheldPageKeys, withHeldPageKeys, type HeldPageKeys } from './page-state/guards.ts';
import { readPageSnapshot as readCanonicalPageSnapshot } from './page-state/snapshot.ts';
+import { readDeadline } from './read-deadline.ts';
import { readPageSnapshotsBatch } from './page-snapshot-batch.ts';
import { createPageVersion } from './page-state/versions.ts';
import { moveSlugBindings, recordRenameAlias } from './page-state/rename-alias.ts';
@@ -359,10 +360,11 @@ export class PostgresEngine implements BrainEngine {
callback: (tx: ReturnType<typeof postgres>) => Promise<T>,
opts?: { alwaysTransaction?: boolean; jitOff?: boolean; settings?: SearchSettings | (() => SearchSettings) },
): Promise<T> {
+ const deadlineAt = this._pageTransaction ? undefined : readDeadline();
// Flag off + no pre-existing transaction need: call through on the
// shared pool exactly as master does. No tx round-trip, no pool slot
// held for the duration of the read.
- if (!this.rlsScopeBindingEnabled && !opts?.alwaysTransaction) {
+ if (!this.rlsScopeBindingEnabled && !opts?.alwaysTransaction && deadlineAt === undefined) {
return await callback(this.sql);
}
// Precedence matches sourceScopeOpts: federated array > scalar > '*'
@@ -384,6 +386,7 @@ export class PostgresEngine implements BrainEngine {
const settings = {
...(this.rlsScopeBindingEnabled ? { 'app.scopes': scopesValue } : {}),
...(opts?.jitOff ? SEARCH_JIT_OFF : {}),
+ ...(deadlineAt === undefined ? {} : { statement_timeout: String(Math.max(1, Math.ceil(deadlineAt - Date.now()))) }),
...(typeof opts?.settings === 'function' ? opts.settings() : opts?.settings),
};
return withSearchSettings(tx, this._pageTransaction, settings, () => callback(tx), VECTOR_SETTING_DEFAULTS);
diff --git a/src/core/read-deadline.ts b/src/core/read-deadline.ts
new file mode 100644
index 0000000000..67b2631b6d
--- /dev/null
+++ b/src/core/read-deadline.ts
@@ -0,0 +1,22 @@
+/**
+ * #6318: the deadline of the preparation a read runs for. A managed-sync preparation's raw reads take its remaining budget
+ * as `executeRaw`'s `timeoutMs` (persistence/bounded-reads.ts `boundedReads`), but the engine's own reads
+ * (`readPageSnapshot`, `readPageSnapshotsBatch`, `findDuplicatePage`, the import pipeline's lookups) issue their
+ * statements on the engine's connection directly. Under a relation lock held elsewhere those statements waited for as long
+ * as the lock lasted, through a pooler that drops the session's `statement_timeout`. Work run inside
+ * `withReadDeadline(deadlineAt, fn)` carries the deadline, and the Postgres engine's scoped reads issued from it run in a
+ * read-only transaction whose `statement_timeout` is the time left (one settings statement, pipelined behind BEGIN), so
+ * the server ends them at the deadline. A transaction's own statements and PGLite are unaffected.
+ */
+import { AsyncLocalStorage } from 'node:async_hooks';
+
+const store = new AsyncLocalStorage<number | undefined>();
+
+export function withReadDeadline<T>(deadlineAt: number, fn: () => T): T {
+ return store.run(deadlineAt, fn);
+}
+
+/** The epoch-millisecond deadline of the work this read runs for, or undefined. */
+export function readDeadline(): number | undefined {
+ return store.getStore();
+}
diff --git a/test/cycle/dream-reverse-write-commit.test.ts b/test/cycle/dream-reverse-write-commit.test.ts
index 7a45bb50df..dcefcc42d8 100644
--- a/test/cycle/dream-reverse-write-commit.test.ts
+++ b/test/cycle/dream-reverse-write-commit.test.ts
@@ -138,6 +138,35 @@ for (const { backend, databaseUrl } of pathologicalArms()) {
});
}, 180_000);
+ // Value: protects=#8087's root match in reverseWriteDreamRefs (a refused commit holds only the pages of the checkout that refused it);
+ // fails_when=the failure is matched to changes by relative path alone, so the same slug in another source's checkout is held and taken off
+ // the written count too; why_new=every other case here has one checkout (#8094); seam=none (two real checkouts, a pre-commit hook in one);
+ // no_claim=the hold token's text, which names the slug only.
+ test('a refused commit in one source\'s checkout holds only that source\'s page; the same slug in another checkout commits', async () => {
+ await withBrain(databaseUrl, 'git', async b => {
+ const other = `${b.root}-b`;
+ mkdirSync(other, { recursive: true });
+ const gitB = (...args: string[]) => execFileSync('git', ['-C', other, ...args], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim();
+ gitB('init', '-q');
+ gitB('config', 'user.name', 'Example Writer');
+ gitB('config', 'user.email', '[email protected]');
+ writeFileSync(join(other, 'README.md'), '# source-b\n');
+ gitB('add', 'README.md');
+ gitB('commit', '-qm', 'init');
+ await b.engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', ['source-b', other]);
+ await importFromContent(b.engine, IDEA, page('Alice Example idea', 'The source-b copy of the idea.'), { noEmbed: true, sourceId: 'source-b' });
+ refuseCommits(b);
+ const headA = b.git('rev-parse', 'HEAD'), commitsB = Number(gitB('rev-list', '--count', 'HEAD'));
+ const held: string[] = [];
+ const written = await reverseWriteDreamRefs(b.engine, b.brainDir, [{ slug: IDEA, source_id: SOURCE }, { slug: IDEA, source_id: 'source-b' }],
+ { phase: 'synthesize', nativeSourceId: SOURCE, render: (p, tags) => serializePageToMarkdown(p, tags), held });
+ expect({ written, held: failedHolds(held, IDEA).length }).toEqual({ written: 1, held: 1 });
+ expect({ headA: b.git('rev-parse', 'HEAD'), dirtyA: dirty(b), fileA: existsSync(b.path(IDEA)) }).toEqual({ headA, dirtyA: '', fileA: false });
+ expect({ commitsB: Number(gitB('rev-list', '--count', 'HEAD')) - commitsB, pathsB: gitB('show', '--name-only', '--format=', 'HEAD'),
+ dirtyB: gitB('status', '--porcelain', '--untracked-files=all') }).toEqual({ commitsB: 1, pathsB: `${IDEA}.md`, dirtyB: '' });
+ });
+ }, 180_000);
+
test('a checkout carrying a canonical owner marker commits through the named-paths-only path', async () => {
await withBrain(databaseUrl, 'git', async b => {
appendFileSync(join(b.root, '.git', 'info', 'exclude'), '.gbrain-owner.json\n');
diff --git a/test/e2e/persistence-preparation-engine-reads-lock-postgres.test.ts b/test/e2e/persistence-preparation-engine-reads-lock-postgres.test.ts
new file mode 100644
index 0000000000..aed0e7e7f5
--- /dev/null
+++ b/test/e2e/persistence-preparation-engine-reads-lock-postgres.test.ts
@@ -0,0 +1,144 @@
+/**
+ * #6318 on Postgres: a managed-sync preparation's engine-internal reads end at the budget on the server too. #6278 bounded
+ * the preparation's raw reads (`executeRaw` with `timeoutMs`), but the engine's own reads (`readPageSnapshot` and the
+ * import pipeline's lookups) went to the pool as autocommit statements: under `LOCK TABLE pages IN ACCESS EXCLUSIVE MODE`
+ * held in another session they waited for as long as the lock lasted, so the member was released at the budget while its
+ * statement stayed on the server, pinning a connection (docs/eval/managed-sync-stall-repro.md, Phase 4.1b).
+ *
+ * Value: protects=the read deadline on the engine's scoped reads issued through a preparation's bounded view
+ * (persistence/bounded-reads.ts, read-deadline.ts); fails_when=a statement of the sync waits on the lock past the budget plus
+ * slack while the lock is still held (the pre-fix path: the member's page snapshot read stayed waiting until the lock
+ * dropped); why_new=the #6278 lock-wait test takes the lock at the origin check, a raw read; seam=the claimed member's
+ * `readPageSnapshot` is intercepted only to take the lock right before it runs (the read itself runs unchanged);
+ * no_claim=the consumer's projection-queue probes (`page_projection_jobs`, not preparation reads), which this does not bound.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { execFileSync } from 'node:child_process';
+import postgres from '#postgres';
+import { isolatedPersistencePostgres } from '../helpers/persistence-postgres.ts';
+import type { BrainEngine } from '../../src/core/engine.ts';
+import { claimWorktree } from '../../src/core/persistence/ownership.ts';
+import { performSync } from '../../src/commands/sync/perform.ts';
+import { disposePersistenceConsumer } from '../../src/core/persistence/service.ts';
+import { resetWriteSwitches } from '../../src/core/persistence/switches.ts';
+import { hasDatabase } from './helpers.ts';
+import { withEnv } from '../helpers/with-env.ts';
+import { waitFor } from '../helpers/wait-for.ts';
+
+const home = mkdtempSync(join(tmpdir(), 'gbrain-prep-engine-reads-e2e-'));
+let engine: BrainEngine | undefined;
+let databaseUrl = '';
+let closePostgres: (() => Promise<void>) | undefined;
+const sources: string[] = [];
+const BUDGET_MS = 2_000;
+/** How long past the budget a statement may still wait on the lock (the server ends it at the budget). */
+const SLACK_MS = 3_000;
+function git(root: string, ...args: string[]): string { return execFileSync('git', ['-C', root, ...args], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim(); }
+const pad = (i: number) => String(i).padStart(3, '0');
+
+async function fixture(e: BrainEngine, count: number) {
+ const id = `enginereads-${randomUUID().replace(/-/g, '').slice(0, 16)}`; sources.push(id);
+ const root = join(home, id); mkdirSync(root); git(root, 'init', '-q');
+ for (let i = 0; i < count; i++) {
+ const full = join(root, 'notes', `n${pad(i)}.md`); mkdirSync(join(full, '..'), { recursive: true });
+ writeFileSync(full, `---\ntitle: Note ${i}\n---\nA durable observation number ${i}.\n`);
+ }
+ git(root, 'add', '.'); git(root, '-c', 'user.name=Example', '-c', '[email protected]', 'commit', '-qm', 'notes');
+ await e.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');
+ await e.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,\'{}\')', [id, root]);
+ await claimWorktree(e, id, root);
+ await e.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');
+ return { id, root };
+}
+
+/** Another session holding `pages` ACCESS EXCLUSIVE until `drop()`, and a reader of the preparation statements waiting on it. */
+async function holdPagesLock(url: string) {
+ const holder = postgres(url, { max: 2, prepare: false, onnotice: () => {} });
+ const dropped = Promise.withResolvers<void>();
+ const acquired = Promise.withResolvers<void>();
+ const transaction = holder.begin(async sql => {
+ for (;;) {
+ await sql.unsafe('SAVEPOINT lock_attempt');
+ try { await sql.unsafe('LOCK TABLE pages IN ACCESS EXCLUSIVE MODE NOWAIT'); break; }
+ catch (error) {
+ if ((error as { code?: string }).code !== '55P03') throw error;
+ await sql.unsafe('ROLLBACK TO SAVEPOINT lock_attempt');
+ await Bun.sleep(20);
+ }
+ }
+ acquired.resolve();
+ await dropped.promise;
+ }).catch(error => { acquired.reject(error); throw error; });
+ await acquired.promise;
+ const waiting = async () => Array.from(await holder.unsafe(
+ `SELECT pid, (extract(epoch FROM now()-query_start)*1000)::int AS age_ms, left(query,100) AS query FROM pg_stat_activity
+ WHERE datname=current_database() AND pid<>pg_backend_pid() AND wait_event_type='Lock' AND query NOT LIKE '%page_projection_jobs%'`)) as Array<{ pid: number; age_ms: number; query: string }>;
+ const request = async (sourceId: string, slug: string) => (Array.from(await holder.unsafe(
+ `SELECT state,error_code,error_message,preparation_attempts FROM persistence_requests WHERE source_id=$1 AND slug=$2 AND intent->>'kind'='managed_sync_import'`,
+ [sourceId, slug])) as Array<{ state: string; error_code: string | null; error_message: string | null; preparation_attempts: number }>)[0];
+ return { waiting, request, drop: async () => { dropped.resolve(); await transaction.catch(() => undefined); await holder.end(); } };
+}
+
+beforeAll(async () => {
+ if (!hasDatabase()) return;
+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!, 'instance', 12); engine = pg.engine; closePostgres = pg.close; databaseUrl = pg.databaseUrl;
+ await engine.setConfig('persistence.sync_preparation_ms', String(BUDGET_MS));
+ await engine.setConfig('persistence.preparation_ceiling_ms', '60000');
+ resetWriteSwitches();
+}, 120_000);
+afterAll(async () => {
+ if (engine) await withEnv({ GBRAIN_HOME: home }, async () => {
+ await disposePersistenceConsumer(engine!); await engine!.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');
+ for (const id of sources) await engine!.executeRaw('DELETE FROM sources WHERE id=$1', [id]);
+ });
+ resetWriteSwitches();
+ await closePostgres?.();
+ rmSync(home, { recursive: true, force: true });
+});
+
+describe.skipIf(!hasDatabase())('a lock-blocked engine read of a preparation ends at the budget (Postgres, #6318)', () => {
+ test('the page snapshot read ends on the server at the budget: no statement of the sync waits on the lock past it while the lock is held', async () => withEnv({ GBRAIN_HOME: home, GBRAIN_SYNC_FAILURES_DIR: home, GBRAIN_SYNC_BULK_SIZE: '1' }, async () => {
+ if (!engine) return;
+ const f = await fixture(engine, 30);
+ const stuckSlug = 'notes/n012';
+ let lock: Awaited<ReturnType<typeof holdPagesLock>> | undefined;
+ const oldest: { age_ms: number; query: string } = { age_ms: 0, query: '' };
+ let watched: Promise<void> = Promise.resolve();
+ const original = engine.readPageSnapshot;
+ let holds = 0;
+ (engine as { readPageSnapshot: unknown }).readPageSnapshot = async function (this: BrainEngine, slug: string, opts?: Parameters<BrainEngine['readPageSnapshot']>[1]) {
+ if (slug === stuckSlug && opts?.sourceId === f.id && !lock) {
+ const [claimed] = await engine!.executeRaw("SELECT 1 FROM persistence_requests WHERE source_id=$1 AND slug=$2 AND state='running' AND intent->>'kind'='managed_sync_import'", [f.id, stuckSlug]);
+ if (claimed) {
+ holds++;
+ lock = await holdPagesLock(databaseUrl);
+ // Held until the member's terminal receipt and one more budget plus slack, sampling the statements that wait on it.
+ watched = (async () => {
+ let terminalAt: number | null = null;
+ await waitFor(async () => {
+ for (const row of await lock!.waiting()) if (row.age_ms > oldest.age_ms) Object.assign(oldest, row);
+ if (terminalAt === null && (await lock!.request(f.id, stuckSlug))?.state === 'failed') terminalAt = performance.now();
+ return terminalAt !== null && performance.now() - terminalAt > BUDGET_MS + SLACK_MS;
+ }, { timeoutMs: BUDGET_MS * 4 + SLACK_MS * 4 + 10_000, intervalMs: 100, label: 'the lock window' }).catch(() => undefined);
+ await lock!.drop();
+ })();
+ }
+ }
+ return original.call(this, slug, opts);
+ };
+ await engine.setConfig('sync.holds', 'fail');
+ try {
+ await performSync(engine, { sourceId: f.id, noPull: true, noEmbed: true, noExtract: true, drain: true, lanes: 1 });
+ } finally { (engine as { readPageSnapshot: unknown }).readPageSnapshot = original; await watched; await lock?.drop().catch(() => undefined); await engine.unsetConfig('sync.holds'); }
+ expect(holds).toBe(1);
+ const stuck = await engine.executeRaw<{ state: string; error_code: string | null; error_message: string | null }>(
+ `SELECT state,error_code,error_message FROM persistence_requests WHERE source_id=$1 AND slug=$2 AND intent->>'kind'='managed_sync_import'`, [f.id, stuckSlug]);
+ expect(stuck[0]).toMatchObject({ state: 'failed', error_code: 'preparation_stalled' });
+ // Before the bound, the first attempt's snapshot read was still waiting when the lock dropped (its age, the whole window).
+ expect({ age_over_bound: oldest.age_ms >= BUDGET_MS + SLACK_MS, query: oldest.query }).toMatchObject({ age_over_bound: false });
+ }), 300_000);
+});
diff --git a/test/rows-behind-checkpoint-8071.test.ts b/test/rows-behind-checkpoint-8071.test.ts
index 9a965da5c7..9cc8cec9b5 100644
--- a/test/rows-behind-checkpoint-8071.test.ts
+++ b/test/rows-behind-checkpoint-8071.test.ts
@@ -31,6 +31,7 @@ import { randomUUID } from 'node:crypto';
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
+import { deflateSync } from 'node:zlib';
import type { BrainEngine } from '../src/core/engine.ts';
import type { OperationContext } from '../src/core/ops/contract.ts';
import { performManagedSync } from '../src/core/persistence/sync-run.ts';
@@ -245,10 +246,10 @@ for (const backend of testBackends()) describe(`[${backend}] #8071: rows behind
expect(full).toMatchObject({ complete: true, pages: 5, behind_count: 4, unproven_count: 1 });
expect(full.unproven[0]!.digest).toBeDefined();
- // A clock that passes the deadline on its eighth read (one before the batch, five before the comparisons, one before
- // the first confirmation): the one batch of five rows must stop part way through, after one confirmed row.
+ // A clock that passes the deadline on its ninth read (one before the tree reads, one before the batch, five before the
+ // comparisons, one before the first confirmation): the one batch of five rows must stop part way through, after one confirmed row.
let reads = 0;
- const stopped = await scanRowsBehind(engine, 'default', { deadlineAt: 1, now: () => (reads++ < 7 ? 0 : 1) });
+ const stopped = await scanRowsBehind(engine, 'default', { deadlineAt: 1, now: () => (reads++ < 8 ? 0 : 1) });
expect(stopped.complete).toBe(false);
expect(stopped.pages).toBeGreaterThan(0);
expect(stopped.pages).toBeLessThan(5);
@@ -269,6 +270,31 @@ for (const backend of testBackends()) describe(`[${backend}] #8071: rows behind
expect(partial.message).toMatch(/stopped after \d+ page\(s\)/);
}, { databaseUrl, setup }), 240_000);
+ // Value: protects=#8093 item 1 (doctor's budget bounds a batch's reads: blobs are read and compared in byte-capped chunks, and none is read
+ // after the deadline; the per-source tree reads wait on the deadline too); fails_when=a batch reads every comparable blob in one call before
+ // its first deadline check, or a scan whose deadline passed still lists the trees; why_new=the deadline test above only stops between rows;
+ // seam=the last row's committed object is rewritten as a non-blob of the same size (the tree listing still sizes it; `cat-file --batch`
+ // cannot serve it as a blob), standing in for a read that must not happen, and batchBytes=1 makes each blob its own chunk; no_claim=peak memory,
+ // which this does not measure.
+ test('the deadline bounds a batch\'s blob reads: no chunk is read after it passes, and no tree is listed once it has passed', () => managedBrain(async ({ engine, root }) => {
+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');
+ await seedBehind(engine, root, Object.fromEntries(SLUGS.map(slug => [slug, { content: note(slug, `The redacted body of ${slug}.`), record: 'checkpoint' as const }])));
+ const last = git(root, 'rev-parse', `HEAD:${SLUGS.at(-1)}.md`);
+ const object = join(root, '.git', 'objects', last.slice(0, 2), last.slice(2));
+ const payload = Buffer.from(git(root, 'cat-file', 'blob', last) + '\n');
+ rmSync(object);
+ writeFileSync(object, deflateSync(Buffer.concat([Buffer.from(`tree ${payload.length}\0`), payload])));
+ await expect(scanRowsBehind(engine, 'default')).rejects.toThrow('Git could not read committed files');
+ // Reads: before the tree reads, before the batch, before the first row's comparison; the second row's check passes the
+ // deadline (the first row, a differing one, then stops before its confirmation). The last row's chunk is never read.
+ let reads = 0;
+ const stopped = await scanRowsBehind(engine, 'default', { deadlineAt: 1, batchBytes: 1, now: () => (reads++ < 3 ? 0 : 1) });
+ expect(stopped).toMatchObject({ status: 'known', complete: false, pages: 0, stopped_at: (await pageOf(engine, SLUGS[0]!)).id });
+
+ await unguarded(engine, () => engine.executeRaw("UPDATE sources SET last_commit=$1 WHERE id='default'", ['f'.repeat(40)]).then(() => {}));
+ expect(await scanRowsBehind(engine, 'default', { deadlineAt: 1, now: () => 1 })).toMatchObject({ status: 'known', complete: false, pages: 0, head: null, stopped_at: null });
+ }, { databaseUrl, setup }), 240_000);
+
test('a hold row applies the refusal the apply computes, not the one saved with the preview', () => managedBrain(async ({ engine, root }) => {
expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');
await seedBehind(engine, root, { 'notes/a': { content: '---\ntype: concept\ntitle: [unclosed\n---\n\nThe redacted body of notes/a.\n', record: 'checkpoint' } });
diff --git a/test/sync-provenance-writers-8089.test.ts b/test/sync-provenance-writers-8089.test.ts
new file mode 100644
index 0000000000..7a05424f7a
--- /dev/null
+++ b/test/sync-provenance-writers-8089.test.ts
@@ -0,0 +1,86 @@
+/**
+ * #8089: the two other writers of a sync import record's disk half follow `syncDiskBytes`' rule (#8071): they record the
+ * working file only when it holds the bytes the record says were imported, so a later commit of a newer, uncommitted edit
+ * is never matched by bytes the page does not hold.
+ *
+ * Value: protects=the noop waiver (sync-waivers.ts `waivedProvenance`) and the published no-op (sync-prepare.ts
+ * `publishedSyncNoop`) provenance writers; fails_when=either records file bytes whose hash is not the record's raw hash;
+ * why_new=#8087 hardened only `syncDiskBytes`, and neither shape was reached by a sync probe (#8089); seam=forced probes: each
+ * writer is called directly, the waiver with a frozen entry whose raw hash names an uncommitted edit of the pinned content, the
+ * published no-op with its file replaced between preparation and apply over a stub transaction that captures statements;
+ * no_claim=that a sync reaches either shape today.
+ */
+import { afterEach, describe, expect, test } from 'bun:test';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { sha256 } from '../src/core/persistence/digest.ts';
+import { publishedSyncNoop, type SyncIntent } from '../src/core/persistence/sync-prepare.ts';
+import { waivedProvenance, type NoopWaiver, type WaiverCursor, type WaiverEntry } from '../src/core/persistence/sync-waivers.ts';
+
+const roots: string[] = [];
+afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
+const checkout = (bytes: string) => {
+ const root = mkdtempSync(join(tmpdir(), 'gbrain-8089-'));
+ roots.push(root);
+ mkdirSync(join(root, 'notes'), { recursive: true });
+ writeFileSync(join(root, 'notes/a.md'), bytes);
+ return root;
+};
+const note = (body: string) => `---\ntype: concept\ntitle: notes/a\n---\n\n${body}\n`;
+const PINNED = note('The committed body of notes/a.');
+const EDITED = note('An uncommitted edit of notes/a.');
+const TARGET = 'a'.repeat(40);
+const PARSER = { parser_version: 'v-example', markdown_chunker: 1, schema_pack: null };
+
+describe('#8089 waivedProvenance', () => {
+ const waiver = (root: string, content: string, rawHash: string, recordedHash: string) => waivedProvenance({ root } as WaiverCursor,
+ { requestId: 'request-a', slug: 'notes/a', pageId: 7, intent: { kind: 'managed_sync_import', path: 'notes/a.md', sourcePath: 'notes/a.md', content, rawHash, target: TARGET } as unknown as SyncIntent } satisfies WaiverEntry,
+ { kind: 'import', kernel: [], provenance: { source_id: 'default', incarnation: 'incarnation-a', origin: 'notes/a.md', raw_sha256: recordedHash, working: false, gbrain_version: 'v-example', parser: PARSER } } as NoopWaiver,
+ 'revision-a');
+ const disk = (result: ReturnType<typeof waivedProvenance>) => result ? result.disk_sha256 : result;
+
+ test('a pinned import whose file holds an uncommitted edit records the pinned bytes, not the edit', () => {
+ expect(disk(waiver(checkout(EDITED), PINNED, sha256(Buffer.from(EDITED)), sha256(Buffer.from(PINNED))))).toBe(sha256(Buffer.from(PINNED)));
+ });
+
+ test('controls: the imported file, a CRLF copy of it and a published no-op\'s working file are recorded as read', () => {
+ expect(disk(waiver(checkout(PINNED), PINNED, sha256(Buffer.from(PINNED)), sha256(Buffer.from(PINNED))))).toBe(sha256(Buffer.from(PINNED)));
+ const crlf = PINNED.replace(/\n/g, '\r\n');
+ expect(disk(waiver(checkout(crlf), PINNED, sha256(Buffer.from(crlf)), sha256(Buffer.from(PINNED))))).toBe(sha256(Buffer.from(crlf)));
+ expect(disk(waiver(checkout(EDITED), PINNED, sha256(Buffer.from(EDITED)), sha256(Buffer.from(EDITED))))).toBe(sha256(Buffer.from(EDITED)));
+ });
+
+ test('a file without the frozen raw hash is not waived; imported bytes in hand under neither hash record nothing', () => {
+ expect(waiver(checkout(EDITED), PINNED, sha256(Buffer.from(PINNED)), sha256(Buffer.from(PINNED)))).toBe(false);
+ expect(waiver(checkout(EDITED), PINNED, sha256(Buffer.from(EDITED)), sha256(Buffer.from('other bytes')))).toBeNull();
+ });
+});
+
+describe('#8089 publishedSyncNoop', () => {
+ const run = async (root: string, change?: string) => {
+ const statements: Array<{ sql: string; params: unknown[] }> = [];
+ const page = { id: 7, slug: 'notes/a', deleted_at: null };
+ const tx = {
+ readPageSnapshot: async () => ({ revision: 'revision-a', page }),
+ executeRaw: async (sql: string, params: unknown[] = []) => { statements.push({ sql, params }); return []; },
+ } as unknown as BrainEngine;
+ const prepared = publishedSyncNoop({ source_id: 'default', source_incarnation: 'incarnation-a', slug: 'notes/a', request_id: 'request-a' } as never,
+ { kind: 'managed_sync_import', path: 'notes/a.md', sourcePath: 'notes/a.md', rawHash: sha256(Buffer.from(EDITED)), target: TARGET } as unknown as SyncIntent,
+ root, { revision: 'revision-a', page } as never, null, async () => {}, async () => {});
+ if (change !== undefined) writeFileSync(join(root, 'notes/a.md'), change);
+ await prepared.apply!(tx);
+ return statements.filter(statement => /INSERT INTO op_checkpoints/.test(statement.sql))
+ .map(statement => (JSON.parse(String(statement.params[2])) as Array<{ disk_sha256: string | null; raw_sha256: string }>)[0]!);
+ };
+
+ test('a file that changed between preparation and apply is not recorded', async () => {
+ expect(await run(checkout(EDITED), note('A later edit of notes/a.'))).toEqual([]);
+ });
+
+ test('control: the working file the page holds is recorded at apply', async () => {
+ expect((await run(checkout(EDITED))).map(record => [record.raw_sha256, record.disk_sha256]))
+ .toEqual([[sha256(Buffer.from(EDITED)), sha256(Buffer.from(EDITED))]]);
+ });
+});