Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs: continue full key-files pages on chained pages linked from their last line; split engines-2 and core-services-2 (GST-2417)
garrytan64c22cd74929
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e51418d1f5..4b13781afd 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,16 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++### The doc size budgets have room again++**llms-full.txt is 751,119 bytes, 48,881 under its 800,000-byte budget, and the two key-files pages that most merges touch are under 54,000 bytes again. No budget was raised and no text was removed.** Before, llms-full.txt sat 532 bytes under its budget and `engines-2.md` and `core-services-2.md` were within 8 bytes of the 60,000-byte page cap. So any change that added a sentence to those pages failed `check:doc-history` or the build-llms size test, and splitting a page cost bytes in `KEY_FILES.md`, which is 9,994 of its 10,000-byte cap.++- `scripts/llms-config.ts`: the Grok Bot, Muse and in-agent setup guides are listed in llms.txt and no longer inlined in llms-full.txt. They only apply inside those hosted agents, and the inlined AGENTS.md routes those agents to them.+- A key-files page at its cap continues on a new page linked from its last line, and the new page links back with `[Previous entries](<page>.md)`. The index links only the first page of each chain. `engines-2.md` continues on `engines-2-continued.md` and `core-services-2.md` on `core-services-2-more.md`; the moved entries are unchanged.+- `test/docs-navigation.test.ts` checks that every key-files page is reachable from `KEY_FILES.md` by following key-files links: a page the index does not link must name one predecessor whose last line links it, its chain must reach an indexed page, and no page has two continuations. The `check:doc-history` failure message and `KEY_FILES.md` state the rule.+## [0.60.659.0] - 2026-11-11### A page purge removes the page's name from the volunteer log and the mention gazetteerdiff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.mdindex f7714f548d..7d9526be4d 100644--- a/docs/architecture/KEY_FILES.md+++ b/docs/architecture/KEY_FILES.md@@ -17,8 +17,8 @@ phase goes: [CONTRIBUTING.md](../../CONTRIBUTING.md#where-does-my-change-go).A branch from before refactor wave 1 follows the generated[porting guide](wave-1-porting.md) ([JSON map](wave-1-moves.json)).`scripts/check-key-files-current-state.sh` guards each subsystem against-history, duplicate entries and size growth. Split a growing subsystem and-link it here; never raise the cap.+history, duplicate entries and size growth. A full page links its next page+from its last line; never raise the cap.| Subsystem | Entry range / scope ||---|---|diff --git a/docs/architecture/key-files/core-services-2-more.md b/docs/architecture/key-files/core-services-2-more.mdnew file mode 100644index 0000000000..71ebda32c7--- /dev/null+++ b/docs/architecture/key-files/core-services-2-more.md@@ -0,0 +1,12 @@+# Key files: Core Services (2/3, more)++[Subsystem index](../KEY_FILES.md) · [Previous entries](core-services-2.md). Read only the entries relevant to your change.+Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.++- `src/core/ops/search.ts` — the search-domain op module (`search`, `query`, `search_stats`, `search_modes`, `search_tune`, `cache_stats`). Both resolve their source scope through `ops/search-source-scope.ts` and return no rows when that scope is empty. `query` is the hybrid entry (`hybridSearchCached`) plus two non-hybrid legs that share ONE effective-row-contract helper, `resolveEffectiveLimit(ctx, p)`: an explicit `limit` wins, otherwise the mode-derived `searchLimit` resolved through the same trust-gated chain `hybridSearch` uses (`resolvePerCallMode` ignores a remote caller's `mode`, so a remote client cannot select the tokenmax row count); the config reads run lazily on those paths only. The legs: the image-similarity branch (`embedding_image` vector search, `test/query-image-mode-limit.serial.test.ts`) and the CRAG escalation slice — when `search.crag_escalation` is on and `shouldEscalateRetrieval` (`src/core/search/crag.ts`) says the first pass graded `weak`, was not already escalated, AND did not already run with the caller's expansion on (`callerExpanded`), the op re-runs once at `limit: max(effectiveLimit, 50)` with expansion + relational on and autocut off, keeping the better-graded run. `buildRetrievalResponseMeta(ctx, scope, …)` is the ONE producer of the `retrieval` response-meta channel (keyword-only path included): when a remote caller's scope holds pages of any kind below the safe-chunk index version (`hasUnsealedPagesInScope`: sargable `chunker_version < SAFE_FENCE_CHUNKER_VERSION` LIMIT 1 probe served by the unsealed-only partial `pages_safe_chunk_pending_idx` — NOT the COALESCE `safeChunksFilter` form; pinned by `test/search-safe-index-probe.test.ts`; fail-open; retains source, private, quarantine and live-page policy), it appends the `safe_index_pending` degraded stage to empty and partial results alike, so neither reads as a clean or complete answer; the drain is `gbrain repair safe-chunks`. The withholding itself is unchanged. Empty and nonempty replies also carry projection_readiness, with projection_pending/projection_status_unknown degradation when appropriate (projection_by_design when the only unsealed pages in scope are protected-OCR images, #6803). Vector candidate incompleteness and scoped pool details survive the public CLI/MCP serialization boundary; public type filters constrain the current-projection probe. Retrieval output copies and sanitizes result/metadata after grading and capture using the canonical scanner, before snippet truncation; internal pipeline/capture remains unredacted. `searchOutput` then projects the returned rows to the caller's row shape (`src/core/search/lean-rows.ts`: lean for remote callers unless `fields: "full"`, the transport chose full rows, or the caller is trusted-local; a lean row keeps `id` only when `ctx.isCallable('fetch')`, the MCP dispatch predicate `buildOperationContext` threads in, says the caller can call `fetch` (unset keeps it), and prints `score` at 4 significant digits) and reports it as `retrieval.rows` for remote callers and whenever `fields` is passed. The CLI formatter repeats this on JSON, explain and human output. System One: the `query` op asks S4 answerability (`decide.answerability`, diagnostic only) and surfaces `meta.answerability`; the crag escalation re-run keeps S2-S5 off (`rerankOnly`). Each handler times its request (`SearchTimings`, search/stage-timings.ts) and `searchOutput` emits `retrieval.timings` last. The typed-search type check lives in `ops/search-type-filter.ts` (`reconcileTypeFilter`): one statement of per-type `EXISTS` probes over each type's schema-expanded set, with the `DISTINCT` type list for the `type_filter_notice` evaluated only when a probe misses, run `jitOff` and unprepared (a generic plan scans every page per missing type); pinned by `test/search-type-filter-reconcile.test.ts` and `test/e2e/typed-search-jit-postgres.test.ts`. `buildRetrievalResponseMeta` also stamps `stale` on hits whose page's newer file is held, adds `held_files` per source in scope and emits the `held_files` degraded notice (`persistence/held-reads.ts`; Test: `test/held-read-signals.test.ts`); `coverage` counts pending, held and embedding-held items with `could_affect` (`search/retrieval-coverage.ts`). Its safe-index probe, projection readiness, entity saved facts, saved-fact match and held-file coverage run concurrently, each with its own catch; the answer-feedback write runs after them (`test/search-response-meta-concurrency.test.ts`). The saved-fact match is one statement: an `EXISTS` probe skips its scan (`idx_facts_fts`; `LIKE ANY` for CJK) when no fact qualifies. The handlers start the alias fan-out's `resolveQueryEntity` alongside the hybrid search, and `evidenceOutput` reads page trust (`loadPageTrust` over the rows every delivered row is copied from) alongside evidence delivery (`test/search-post-rerank-concurrency.test.ts`). The starter list leaves out (`fullSurfaceOnly`, still accepted) `mode`, `snippet_chars`, `return_window`, `token_budget`, `salience`, `recency`, `fields`, and on `query` `image`/`image_mime`/`embedding_column`, the code params, `detail`, `adaptive_return`, `autocut` and `relational`.+- `src/core/ops/search-source-scope.ts` — per-request scope of `search`/`query`: `source_id` through `federatedSearchScope`, narrowed by `source_ids` (`parseSourceIdsParam`: trimmed, deduped, validated; `__all__` refused; `[]` or a blank string is absent, #5390). Each listed id is authorized like an explicit `source_id` (out-of-grant: permission_denied; non-live: unknown_source); with both params the scope is their intersection, and an empty one (`sourceIds: []`) emits `empty_retrieval` and nothing is searched. Tests: `test/search/search-source-ids.test.ts` (+ its `test/e2e/` Postgres twin), `test/e2e/serve-http-member-tokens.test.ts`.++- `src/core/ops/sources.ts` — `sources_remove` is confined to the caller's WRITE authority via `assertSourceInCallerWriteScope(ctx, id)` (context.ts): an untrusted caller may hard-remove only its own write source; a federated read grant naming a source does not make it removable; any other id answers `OperationError('not_found', 'Unknown source: <id>')`, byte-identical to a nonexistent source so the error is not an existence oracle. `sources_status` keeps the read-scope helper (`assertSourceInCallerScope`), so a federated grant can still diagnose a source it cannot remove. Trusted local CLI keeps the full operator view. Over HTTP every client carries a write source (registration default `default`) and its own `oauth_clients.source_id` row FK-RESTRICTs removing that source, so hard removal is effectively an operator action (`gbrain sources remove`); a source a remote `sources_admin` caller just created via `sources_add` stays outside its authority until the operator rescopes the client. There is no `sources_archive` op — archive lives in the CLI (`gbrain sources archive`). Pinned by `test/sources-mcp.test.ts` (federated-read-only id → not_found while status still answers; write-source id → removed; unbound → any; local → any; `remote: undefined` fail-closed) and `test/e2e/sources-remote-mcp.test.ts` (real server: not_found before AND after a read-grant widen; operator CLI completes; rescoping `--source` moves the authority).++- `src/core/page-summary.ts` — `generatePerChunkSynopsis`: the per-chunk synopsis call (utility-tier default, `thinking: 'off'`, `SYNOPSIS_DOC_MAX_CHARS` surrogate-safe head truncation, D27 failure envelope). One user message of TWO text blocks (PR #6341): the stable title + document block flagged `cache: true` (the only prefix every chunk of a page shares) and the per-chunk block; joined with `\n` they equal the former single string, so `SYNOPSIS_PROMPT_VERSION` stays 1. Pinned by `test/page-summary-prompt-cache.test.ts`.+- `src/core/notifier/` + `src/core/commit-notifier.ts` — the commit notifier, wanted markers, residents and search freshness: [Commit notifier](commit-notifier.md).diff --git a/docs/architecture/key-files/core-services-2.md b/docs/architecture/key-files/core-services-2.mdindex 5222ef8947..7fd4b47d58 100644--- a/docs/architecture/key-files/core-services-2.md+++ b/docs/architecture/key-files/core-services-2.md@@ -50,10 +50,4 @@ Fact and page purge modules (`src/core/facts/purge*.ts`, `deletion-inventory.ts`- `src/core/ops/raw-data.ts` — `put_raw_data` / `get_raw_data`. `get_raw_data` follows the page soft-delete (a tombstoned page reads as `[]`, `restore_page` brings the rows back; there is no remote `include_deleted` param — `get_page include_deleted: true` verifies the tombstone) and, per #4352, a `visibility: private` page's raw data reads exactly like a missing page's for untrusted callers. Scope comes from `readPolicyOpts(ctx)`, never from MCP parameters.-- `src/core/ops/search.ts` — the search-domain op module (`search`, `query`, `search_stats`, `search_modes`, `search_tune`, `cache_stats`). Both resolve their source scope through `ops/search-source-scope.ts` and return no rows when that scope is empty. `query` is the hybrid entry (`hybridSearchCached`) plus two non-hybrid legs that share ONE effective-row-contract helper, `resolveEffectiveLimit(ctx, p)`: an explicit `limit` wins, otherwise the mode-derived `searchLimit` resolved through the same trust-gated chain `hybridSearch` uses (`resolvePerCallMode` ignores a remote caller's `mode`, so a remote client cannot select the tokenmax row count); the config reads run lazily on those paths only. The legs: the image-similarity branch (`embedding_image` vector search, `test/query-image-mode-limit.serial.test.ts`) and the CRAG escalation slice — when `search.crag_escalation` is on and `shouldEscalateRetrieval` (`src/core/search/crag.ts`) says the first pass graded `weak`, was not already escalated, AND did not already run with the caller's expansion on (`callerExpanded`), the op re-runs once at `limit: max(effectiveLimit, 50)` with expansion + relational on and autocut off, keeping the better-graded run. `buildRetrievalResponseMeta(ctx, scope, …)` is the ONE producer of the `retrieval` response-meta channel (keyword-only path included): when a remote caller's scope holds pages of any kind below the safe-chunk index version (`hasUnsealedPagesInScope`: sargable `chunker_version < SAFE_FENCE_CHUNKER_VERSION` LIMIT 1 probe served by the unsealed-only partial `pages_safe_chunk_pending_idx` — NOT the COALESCE `safeChunksFilter` form; pinned by `test/search-safe-index-probe.test.ts`; fail-open; retains source, private, quarantine and live-page policy), it appends the `safe_index_pending` degraded stage to empty and partial results alike, so neither reads as a clean or complete answer; the drain is `gbrain repair safe-chunks`. The withholding itself is unchanged. Empty and nonempty replies also carry projection_readiness, with projection_pending/projection_status_unknown degradation when appropriate (projection_by_design when the only unsealed pages in scope are protected-OCR images, #6803). Vector candidate incompleteness and scoped pool details survive the public CLI/MCP serialization boundary; public type filters constrain the current-projection probe. Retrieval output copies and sanitizes result/metadata after grading and capture using the canonical scanner, before snippet truncation; internal pipeline/capture remains unredacted. `searchOutput` then projects the returned rows to the caller's row shape (`src/core/search/lean-rows.ts`: lean for remote callers unless `fields: "full"`, the transport chose full rows, or the caller is trusted-local; a lean row keeps `id` only when `ctx.isCallable('fetch')`, the MCP dispatch predicate `buildOperationContext` threads in, says the caller can call `fetch` (unset keeps it), and prints `score` at 4 significant digits) and reports it as `retrieval.rows` for remote callers and whenever `fields` is passed. The CLI formatter repeats this on JSON, explain and human output. System One: the `query` op asks S4 answerability (`decide.answerability`, diagnostic only) and surfaces `meta.answerability`; the crag escalation re-run keeps S2-S5 off (`rerankOnly`). Each handler times its request (`SearchTimings`, search/stage-timings.ts) and `searchOutput` emits `retrieval.timings` last. The typed-search type check lives in `ops/search-type-filter.ts` (`reconcileTypeFilter`): one statement of per-type `EXISTS` probes over each type's schema-expanded set, with the `DISTINCT` type list for the `type_filter_notice` evaluated only when a probe misses, run `jitOff` and unprepared (a generic plan scans every page per missing type); pinned by `test/search-type-filter-reconcile.test.ts` and `test/e2e/typed-search-jit-postgres.test.ts`. `buildRetrievalResponseMeta` also stamps `stale` on hits whose page's newer file is held, adds `held_files` per source in scope and emits the `held_files` degraded notice (`persistence/held-reads.ts`; Test: `test/held-read-signals.test.ts`); `coverage` counts pending, held and embedding-held items with `could_affect` (`search/retrieval-coverage.ts`). Its safe-index probe, projection readiness, entity saved facts, saved-fact match and held-file coverage run concurrently, each with its own catch; the answer-feedback write runs after them (`test/search-response-meta-concurrency.test.ts`). The saved-fact match is one statement: an `EXISTS` probe skips its scan (`idx_facts_fts`; `LIKE ANY` for CJK) when no fact qualifies. The handlers start the alias fan-out's `resolveQueryEntity` alongside the hybrid search, and `evidenceOutput` reads page trust (`loadPageTrust` over the rows every delivered row is copied from) alongside evidence delivery (`test/search-post-rerank-concurrency.test.ts`). The starter list leaves out (`fullSurfaceOnly`, still accepted) `mode`, `snippet_chars`, `return_window`, `token_budget`, `salience`, `recency`, `fields`, and on `query` `image`/`image_mime`/`embedding_column`, the code params, `detail`, `adaptive_return`, `autocut` and `relational`.-- `src/core/ops/search-source-scope.ts` — per-request scope of `search`/`query`: `source_id` through `federatedSearchScope`, narrowed by `source_ids` (`parseSourceIdsParam`: trimmed, deduped, validated; `__all__` refused; `[]` or a blank string is absent, #5390). Each listed id is authorized like an explicit `source_id` (out-of-grant: permission_denied; non-live: unknown_source); with both params the scope is their intersection, and an empty one (`sourceIds: []`) emits `empty_retrieval` and nothing is searched. Tests: `test/search/search-source-ids.test.ts` (+ its `test/e2e/` Postgres twin), `test/e2e/serve-http-member-tokens.test.ts`.--- `src/core/ops/sources.ts` — `sources_remove` is confined to the caller's WRITE authority via `assertSourceInCallerWriteScope(ctx, id)` (context.ts): an untrusted caller may hard-remove only its own write source; a federated read grant naming a source does not make it removable; any other id answers `OperationError('not_found', 'Unknown source: <id>')`, byte-identical to a nonexistent source so the error is not an existence oracle. `sources_status` keeps the read-scope helper (`assertSourceInCallerScope`), so a federated grant can still diagnose a source it cannot remove. Trusted local CLI keeps the full operator view. Over HTTP every client carries a write source (registration default `default`) and its own `oauth_clients.source_id` row FK-RESTRICTs removing that source, so hard removal is effectively an operator action (`gbrain sources remove`); a source a remote `sources_admin` caller just created via `sources_add` stays outside its authority until the operator rescopes the client. There is no `sources_archive` op — archive lives in the CLI (`gbrain sources archive`). Pinned by `test/sources-mcp.test.ts` (federated-read-only id → not_found while status still answers; write-source id → removed; unbound → any; local → any; `remote: undefined` fail-closed) and `test/e2e/sources-remote-mcp.test.ts` (real server: not_found before AND after a read-grant widen; operator CLI completes; rescoping `--source` moves the authority).--- `src/core/page-summary.ts` — `generatePerChunkSynopsis`: the per-chunk synopsis call (utility-tier default, `thinking: 'off'`, `SYNOPSIS_DOC_MAX_CHARS` surrogate-safe head truncation, D27 failure envelope). One user message of TWO text blocks (PR #6341): the stable title + document block flagged `cache: true` (the only prefix every chunk of a page shares) and the per-chunk block; joined with `\n` they equal the former single string, so `SYNOPSIS_PROMPT_VERSION` stays 1. Pinned by `test/page-summary-prompt-cache.test.ts`.-- `src/core/notifier/` + `src/core/commit-notifier.ts` — the commit notifier, wanted markers, residents and search freshness: [Commit notifier](commit-notifier.md).+Continued in [Core Services (2/3, more)](core-services-2-more.md).diff --git a/docs/architecture/key-files/engines-2-continued.md b/docs/architecture/key-files/engines-2-continued.mdnew file mode 100644index 0000000000..95ee00150f--- /dev/null+++ b/docs/architecture/key-files/engines-2-continued.md@@ -0,0 +1,11 @@+# Key files: Engines (2/2, continued)++[Subsystem index](../KEY_FILES.md) · [Previous entries](engines-2.md). Read only the entries relevant to your change.+Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.++- `src/core/postgres-engine/` + `src/core/pglite-engine/` — per-engine module directories for code that is genuinely dialect-specific. Migrated storage domains (the `migrated` rows of `scripts/engine-sql-baseline.tsv`) do not live here: their SQL exists once in `src/core/engine-sql/` (see that entry in engines-2.md). What remains: `pglite-engine/checkpoint-guard.ts`, `pglite-engine/code-edges.ts` (PGLite's `getEdgesByChunk`, whose master SQL behaves differently from Postgres's: one UNION ALL under a shared LIMIT; listed in `docs/designs/refactor-wave-1/w1-inventory.md`), `postgres-engine/cancellation.ts`, `postgres-engine/init-schema-lock.ts`, `postgres-engine/keyword-statement-timeout.ts` (#6495: `KEYWORD_STATEMENT_TIMEOUT_KEY` = `search.keyword_statement_timeout_ms`, default 8000, range 1000-60000; `keywordStatementTimeoutMs` accepts an integer in range, `resolveKeywordStatementTimeoutMs(file, db)` applies file > DB > default, `keywordStatementTimeoutValueProblem` is the `config set` refusal text, and `loadKeywordStatementTimeout(reader, fileValue)` yields the `<ms>ms` setting one connection binds; pinned by `test/keyword-statement-timeout-knob-6495.test.ts` on a recording handle and, with DATABASE_URL, on a real Postgres whose locked tables make each statement wait until the bound cancels it), `postgres-engine/keyword-timeout-cache.ts` (#6495 negative cache: `KeywordTimeoutNegativeCache.run(enabled, db, keyParts, exec)` wraps `searchKeyword`'s strict and relaxed statements when hybrid's keyword arm opts in with `SearchOpts.keywordTimeoutCache`; the key hashes the statement text, its parameters, the RLS source scope and the statement timeout, so only the exact statement is skipped; a genuine statement-timeout 57014, never a user cancel, records an entry for `search.keyword_timeout_negative_cache_seconds` (default 600, 0 = off, read once on the first timeout) and later identical runs throw `KeywordSkippedAfterTimeoutError` (code `keyword_skipped_after_timeout`, arm reason `skipped_after_timeout`) without a statement; at most 256 entries per engine; each timeout writes one bounded `search.keyword_timeout_stats` row that doctor `keyword_plan` reads; pinned by `test/keyword-timeout-negative-cache-6495.test.ts`), `postgres-engine/schema-lock-timeout.ts` (#5227: `resolveSchemaLockTimeoutMs` reads `GBRAIN_SCHEMA_LOCK_TIMEOUT_SECONDS`, default 60 s; `withSchemaLockTimeout` sets a session-level `lock_timeout` on the reserved DDL backend and restores the prior value on release, used by the `schema.sql` replay in `postgres-engine.ts` and the `transaction: false` path of `migrate.ts`, while a transaction-wrapped migration binds the same wait with `SET LOCAL`; `schemaDiagnosticConnection` opens a one-backend connection that samples `pg_blocking_pids()` of the waiting backend while the wait is active so `schemaLockBlockedError` names blockers by pid, application, state and transaction age, never query text; `init-schema-retry.ts` retries `schema_lock_blocked` / SQLSTATE 55P03 with a short backoff; handler migrations and the post-migration `verifySchema` run outside the bound) and `postgres-engine/search-settings.ts` (search transactions run with `jit = off`; `withSearchSettings` applies a read's settings in one statement and restores a caller's values when nested), `postgres-engine/reserved-transactions.ts` (`withReservedConnection` lends its connection a `transaction()`; BEGIN READ WRITE and SAVEPOINT go out with the statements after them, and a COMMIT answered with ROLLBACK throws), `postgres-engine/read-write-transactions.ts` (#7183: a server connection whose `default_transaction_read_only` is on, from a session SET left on a pooled connection or a role or database default, refuses writes with 25006; `beginReadWrite` / `readWriteTransactions` open every pool transaction `BEGIN READ WRITE`, falling back to a plain BEGIN on a hot standby that refuses it, and `retryReadOnlyAutocommit` reruns an autocommit statement refused with 25006 once inside one; explicit `begin('read only')` and `SET TRANSACTION ... READ ONLY` stay read-only; pinned by `test/inherited-read-only-7183.test.ts` and, on direct Postgres and PgBouncer, `test/e2e/pooler-inherited-read-only-7183.test.ts`), `postgres-engine/transaction-abort.ts` (`transaction(fn, { signal })`: an abort calls the begin handle's `cancelAndDiscard()`, which sends the running fenced statement's CancelRequest, keeps successors out until it settles (bounded by `GBRAIN_CANCEL_SETTLE_MS`), then discards the connection, so a statement blocked on a lock stops instead of waiting out its statement timeout (#6598) and the server rolls the transaction back; the caller gets an `AbortError` with the driver's `CONNECTION_CLOSED` as `cause` and, when the cancel could not be confirmed (unfenced statement, transport failure, settle timeout), the `CANCEL_UNCONFIRMED` rejection as `cancelFailure`; an abort after COMMIT keeps the committed result; an already-aborted signal rejects before BEGIN; pinned offline by `test/postgres-transaction-cancel-order.test.ts` and on direct Postgres and PgBouncer by `test/e2e/transaction-cancel-held-lock.test.ts`); the forward-reference bootstrap lives once in `engine-sql/bootstrap.ts` for both engines. Façades import these modules statically (engine-live no-runtime-dynamic-import rule) and keep their full public surface.++- `src/core/analyze-lock.ts` — #7083: `lockTableForAnalyze` takes a table's SHARE UPDATE EXCLUSIVE lock with NOWAIT before every ANALYZE gbrain runs on Postgres on its own (the projection and chunk refreshes, `repair planner-stats`, `analyzeAfterIndexBuild` through `analyzeYieldingToVacuum`), and `analyzeDeferred` recognizes its 55P03 refusal. An ANALYZE that waited `deadlock_timeout` on a running autovacuum made Postgres cancel it, so a refresh after every embed drain kept a 10M-chunk content_chunks from ever being vacuumed; a request that never waits never signals a vacuum, and the caller defers its ANALYZE. Doctor `planner_stats_stale` reports a hot table whose autovacuum stays starved (`readVacuumStarvation`). Test: `test/e2e/analyze-yields-to-autovacuum-postgres.test.ts`.+- `src/core/migration-preludes.ts` — #7108 runner-side preludes for published migrations whose SQL rewrites a large table in one transaction (published migration files never change; restamp refuses an edit). `runMigrationPrelude(engine, version)` runs right before that migration's SQL inside the runner's retry wrapper: `batchedUpdate` repeats the migration's own SET and predicate over id-ordered batches (5,000 rows, inlined so a generic prepared plan cannot scan the table per batch; `__setBackfillBatchRowsForTests`), each its own transaction with the migration statement budget and the schema lock timeout, so progress is the data itself and a rerun resumes; a column the migration adds first is added first, in the migration's column order (fresh and upgraded ordinals stay equal). Preludes: v011 legacy `links.link_source` (skipped below PG15, where the migration refuses), v028 chunk keyword vectors, v033 request-log agent names (NULL-token rows stay the migration's own), v116 code-edge `source_id`, v161 (Postgres) the partial pages index built concurrently under the budget on a reserved connection (the migration caps its own build at 30 s; another session's build or an invalid result stops the run), v221 `embedding_pending_since` (v221's active-column choice; the first stamping batch writes `embedding_pending_since_backfilled_at` in its transaction). Pinned by `test/e2e/migration-v221-batched-backfill-7108.test.ts`. Migrations that cannot be moved here are listed in docs/guides/troubleshooting.md#migration-statement-timeout.+- `src/core/migration-timeout.ts` — #7108 the migration statement budget (`migrationStatementTimeoutMs`, default 10 min; `setMigrationStatementTimeoutMs` from operator-only `gbrain apply-migrations --statement-timeout <minutes>`, restored when that run returns; 0 removes it) and the timed-out-migration record (config `schema_migration_timeout`: version, name, whether the run had a prelude, idle-in-transaction pids sampled at the failure): `runMigrations` writes it on a 57014 or a statement-timeout retry exhaustion and deletes it after a pass reaches head. A schema-graph leaf (no release version, so the PGLite snapshot hash stays stable). `migration-timeout-advice.ts` turns a record into the one fix both apply-migrations (`migrations_pending` still_behind) and doctor `schema_version` show: timed out unbatched and this release has a prelude → rerun; idle sessions open → ask_user, check `doctor --locks`, end the ones still open, rerun; none → ask_user, one run with `--statement-timeout 120`.+- `src/core/derived-state-maintainers.ts` + `src/core/schema-migrations/v281-derived-state-maintainers-fail-closed.ts` — #7637: the eight derived-state maintainer triggers (page retrievals, atom scan state, tag revision, version trust and attribution, page search vector, projection queue, trust generation) stay SECURITY INVOKER and fail closed. `requireMaintainerPoliciesSql(table, commands)` is the PL/pgSQL each body runs before touching another table: when `row_security_active(table)` holds and no permissive `pg_policy` row for the caller (PUBLIC or a role it has USAGE of) covers a command, it raises `writer_coordinator_required` (`managed_writer_guard:unreadable_state`, DETAIL `state` = the table), so an empty read never counts as nothing to do. Each body lives with its schema (`TAG_REVISION_FUNCTION_SQL`, `SNAPSHOT_VERSION_FUNCTION`, `STAMP_VERSION_FUNCTION`, `BUMP_FUNCTION`, `PAGE_PROJECTION_QUEUE_FUNCTION_SQL`, `pageSearchVectorTriggerFnSql`); the page retrievals body is a schema blob fragment here. v281 replaces only the functions (the search vector under the configured FTS language), never the triggers. The policies a scoped writer needs are the ENGINES.md maintainer recipe (`test/derived-state-maintainers-7637.test.ts`, applied verbatim).diff --git a/docs/architecture/key-files/engines-2.md b/docs/architecture/key-files/engines-2.mdindex e9b4960cb9..e1d8df3743 100644--- a/docs/architecture/key-files/engines-2.md+++ b/docs/architecture/key-files/engines-2.md@@ -62,9 +62,4 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/schema-migrations/v232-fts-cjk-boundary.ts` — #6370, handler-only and idempotent: creates `gbrain_fts_input`, recreates both search_vector trigger functions under `getFtsLanguage()` with the shared builders, then rebuilds only pages (`title`/`timeline`) and chunks (`chunk_text`/`doc_comment`/`symbol_name_qualified`) matching `FTS_CJK_BOUNDARY_REGEX` through `search-vector-backfill.ts` (checkpoints `fts_cjk_pages`/`fts_cjk_content_chunks`, cleared at the end); every other row's vector is byte-identical under the helper and is not touched. Pinned by `test/fts-cjk-boundary.test.ts` and `test/e2e/fts-cjk-boundary-postgres.test.ts`.- `src/core/schema-migrations/v245-page-search-vector-bounded-timeline.ts` — Large pages, handler-only and idempotent: recreates the pages search_vector trigger function under `getFtsLanguage()` with the bounded timeline (`PAGE_TIMELINE_FTS_MAX_CHARS`) and entries (`PAGE_TIMELINE_ENTRIES_FTS_MAX_CHARS`) cuts, then rebuilds only pages whose timeline is longer than half the timeline cap through `search-vector-backfill.ts` (checkpoint `page_vector_bounded_timeline`, cleared at the end); shorter timelines index byte-identically and are not touched. Pinned by `test/large-pages/page-timeline-tsvector-bound.test.ts` and `test/e2e/page-timeline-tsvector-bound-postgres.test.ts`.-- `src/core/postgres-engine/` + `src/core/pglite-engine/` — per-engine module directories for code that is genuinely dialect-specific. Migrated storage domains (the `migrated` rows of `scripts/engine-sql-baseline.tsv`) do not live here: their SQL exists once in `src/core/engine-sql/` (see that entry in engines-2.md). What remains: `pglite-engine/checkpoint-guard.ts`, `pglite-engine/code-edges.ts` (PGLite's `getEdgesByChunk`, whose master SQL behaves differently from Postgres's: one UNION ALL under a shared LIMIT; listed in `docs/designs/refactor-wave-1/w1-inventory.md`), `postgres-engine/cancellation.ts`, `postgres-engine/init-schema-lock.ts`, `postgres-engine/keyword-statement-timeout.ts` (#6495: `KEYWORD_STATEMENT_TIMEOUT_KEY` = `search.keyword_statement_timeout_ms`, default 8000, range 1000-60000; `keywordStatementTimeoutMs` accepts an integer in range, `resolveKeywordStatementTimeoutMs(file, db)` applies file > DB > default, `keywordStatementTimeoutValueProblem` is the `config set` refusal text, and `loadKeywordStatementTimeout(reader, fileValue)` yields the `<ms>ms` setting one connection binds; pinned by `test/keyword-statement-timeout-knob-6495.test.ts` on a recording handle and, with DATABASE_URL, on a real Postgres whose locked tables make each statement wait until the bound cancels it), `postgres-engine/keyword-timeout-cache.ts` (#6495 negative cache: `KeywordTimeoutNegativeCache.run(enabled, db, keyParts, exec)` wraps `searchKeyword`'s strict and relaxed statements when hybrid's keyword arm opts in with `SearchOpts.keywordTimeoutCache`; the key hashes the statement text, its parameters, the RLS source scope and the statement timeout, so only the exact statement is skipped; a genuine statement-timeout 57014, never a user cancel, records an entry for `search.keyword_timeout_negative_cache_seconds` (default 600, 0 = off, read once on the first timeout) and later identical runs throw `KeywordSkippedAfterTimeoutError` (code `keyword_skipped_after_timeout`, arm reason `skipped_after_timeout`) without a statement; at most 256 entries per engine; each timeout writes one bounded `search.keyword_timeout_stats` row that doctor `keyword_plan` reads; pinned by `test/keyword-timeout-negative-cache-6495.test.ts`), `postgres-engine/schema-lock-timeout.ts` (#5227: `resolveSchemaLockTimeoutMs` reads `GBRAIN_SCHEMA_LOCK_TIMEOUT_SECONDS`, default 60 s; `withSchemaLockTimeout` sets a session-level `lock_timeout` on the reserved DDL backend and restores the prior value on release, used by the `schema.sql` replay in `postgres-engine.ts` and the `transaction: false` path of `migrate.ts`, while a transaction-wrapped migration binds the same wait with `SET LOCAL`; `schemaDiagnosticConnection` opens a one-backend connection that samples `pg_blocking_pids()` of the waiting backend while the wait is active so `schemaLockBlockedError` names blockers by pid, application, state and transaction age, never query text; `init-schema-retry.ts` retries `schema_lock_blocked` / SQLSTATE 55P03 with a short backoff; handler migrations and the post-migration `verifySchema` run outside the bound) and `postgres-engine/search-settings.ts` (search transactions run with `jit = off`; `withSearchSettings` applies a read's settings in one statement and restores a caller's values when nested), `postgres-engine/reserved-transactions.ts` (`withReservedConnection` lends its connection a `transaction()`; BEGIN READ WRITE and SAVEPOINT go out with the statements after them, and a COMMIT answered with ROLLBACK throws), `postgres-engine/read-write-transactions.ts` (#7183: a server connection whose `default_transaction_read_only` is on, from a session SET left on a pooled connection or a role or database default, refuses writes with 25006; `beginReadWrite` / `readWriteTransactions` open every pool transaction `BEGIN READ WRITE`, falling back to a plain BEGIN on a hot standby that refuses it, and `retryReadOnlyAutocommit` reruns an autocommit statement refused with 25006 once inside one; explicit `begin('read only')` and `SET TRANSACTION ... READ ONLY` stay read-only; pinned by `test/inherited-read-only-7183.test.ts` and, on direct Postgres and PgBouncer, `test/e2e/pooler-inherited-read-only-7183.test.ts`), `postgres-engine/transaction-abort.ts` (`transaction(fn, { signal })`: an abort calls the begin handle's `cancelAndDiscard()`, which sends the running fenced statement's CancelRequest, keeps successors out until it settles (bounded by `GBRAIN_CANCEL_SETTLE_MS`), then discards the connection, so a statement blocked on a lock stops instead of waiting out its statement timeout (#6598) and the server rolls the transaction back; the caller gets an `AbortError` with the driver's `CONNECTION_CLOSED` as `cause` and, when the cancel could not be confirmed (unfenced statement, transport failure, settle timeout), the `CANCEL_UNCONFIRMED` rejection as `cancelFailure`; an abort after COMMIT keeps the committed result; an already-aborted signal rejects before BEGIN; pinned offline by `test/postgres-transaction-cancel-order.test.ts` and on direct Postgres and PgBouncer by `test/e2e/transaction-cancel-held-lock.test.ts`); the forward-reference bootstrap lives once in `engine-sql/bootstrap.ts` for both engines. Façades import these modules statically (engine-live no-runtime-dynamic-import rule) and keep their full public surface.--- `src/core/analyze-lock.ts` — #7083: `lockTableForAnalyze` takes a table's SHARE UPDATE EXCLUSIVE lock with NOWAIT before every ANALYZE gbrain runs on Postgres on its own (the projection and chunk refreshes, `repair planner-stats`, `analyzeAfterIndexBuild` through `analyzeYieldingToVacuum`), and `analyzeDeferred` recognizes its 55P03 refusal. An ANALYZE that waited `deadlock_timeout` on a running autovacuum made Postgres cancel it, so a refresh after every embed drain kept a 10M-chunk content_chunks from ever being vacuumed; a request that never waits never signals a vacuum, and the caller defers its ANALYZE. Doctor `planner_stats_stale` reports a hot table whose autovacuum stays starved (`readVacuumStarvation`). Test: `test/e2e/analyze-yields-to-autovacuum-postgres.test.ts`.-- `src/core/migration-preludes.ts` — #7108 runner-side preludes for published migrations whose SQL rewrites a large table in one transaction (published migration files never change; restamp refuses an edit). `runMigrationPrelude(engine, version)` runs right before that migration's SQL inside the runner's retry wrapper: `batchedUpdate` repeats the migration's own SET and predicate over id-ordered batches (5,000 rows, inlined so a generic prepared plan cannot scan the table per batch; `__setBackfillBatchRowsForTests`), each its own transaction with the migration statement budget and the schema lock timeout, so progress is the data itself and a rerun resumes; a column the migration adds first is added first, in the migration's column order (fresh and upgraded ordinals stay equal). Preludes: v011 legacy `links.link_source` (skipped below PG15, where the migration refuses), v028 chunk keyword vectors, v033 request-log agent names (NULL-token rows stay the migration's own), v116 code-edge `source_id`, v161 (Postgres) the partial pages index built concurrently under the budget on a reserved connection (the migration caps its own build at 30 s; another session's build or an invalid result stops the run), v221 `embedding_pending_since` (v221's active-column choice; the first stamping batch writes `embedding_pending_since_backfilled_at` in its transaction). Pinned by `test/e2e/migration-v221-batched-backfill-7108.test.ts`. Migrations that cannot be moved here are listed in docs/guides/troubleshooting.md#migration-statement-timeout.-- `src/core/migration-timeout.ts` — #7108 the migration statement budget (`migrationStatementTimeoutMs`, default 10 min; `setMigrationStatementTimeoutMs` from operator-only `gbrain apply-migrations --statement-timeout <minutes>`, restored when that run returns; 0 removes it) and the timed-out-migration record (config `schema_migration_timeout`: version, name, whether the run had a prelude, idle-in-transaction pids sampled at the failure): `runMigrations` writes it on a 57014 or a statement-timeout retry exhaustion and deletes it after a pass reaches head. A schema-graph leaf (no release version, so the PGLite snapshot hash stays stable). `migration-timeout-advice.ts` turns a record into the one fix both apply-migrations (`migrations_pending` still_behind) and doctor `schema_version` show: timed out unbatched and this release has a prelude → rerun; idle sessions open → ask_user, check `doctor --locks`, end the ones still open, rerun; none → ask_user, one run with `--statement-timeout 120`.-- `src/core/derived-state-maintainers.ts` + `src/core/schema-migrations/v281-derived-state-maintainers-fail-closed.ts` — #7637: the eight derived-state maintainer triggers (page retrievals, atom scan state, tag revision, version trust and attribution, page search vector, projection queue, trust generation) stay SECURITY INVOKER and fail closed. `requireMaintainerPoliciesSql(table, commands)` is the PL/pgSQL each body runs before touching another table: when `row_security_active(table)` holds and no permissive `pg_policy` row for the caller (PUBLIC or a role it has USAGE of) covers a command, it raises `writer_coordinator_required` (`managed_writer_guard:unreadable_state`, DETAIL `state` = the table), so an empty read never counts as nothing to do. Each body lives with its schema (`TAG_REVISION_FUNCTION_SQL`, `SNAPSHOT_VERSION_FUNCTION`, `STAMP_VERSION_FUNCTION`, `BUMP_FUNCTION`, `PAGE_PROJECTION_QUEUE_FUNCTION_SQL`, `pageSearchVectorTriggerFnSql`); the page retrievals body is a schema blob fragment here. v281 replaces only the functions (the search vector under the configured FTS language), never the triggers. The policies a scoped writer needs are the ENGINES.md maintainer recipe (`test/derived-state-maintainers-7637.test.ts`, applied verbatim).+Continued in [Engines (2/2, continued)](engines-2-continued.md).diff --git a/scripts/check-key-files-current-state.sh b/scripts/check-key-files-current-state.shindex b1f1729b9d..7b14decd4e 100755--- a/scripts/check-key-files-current-state.sh+++ b/scripts/check-key-files-current-state.sh@@ -15,7 +15,9 @@# marker is banned, so this never false-fires on legitimate version mentions.# 2. Size caps for CLAUDE.md, the index, each subsystem and# docs/TESTING.md. The TESTING.md cap is a ratchet: lower it when the file-# shrinks, never raise it; detail moves next to its code instead.+# shrinks, never raise it; detail moves next to its code instead. A subsystem+# page at its cap continues on a new page linked from its last line (the+# index links only a chain's first page), not on an unrelated page.# 3. One reference bullet per src file across the index and subsystems — a cherry-pick that keeps both sides# of a hunk leaves two "- `src/x.ts`" bullets describing two different# "current states" for one file. Merge into the survivor, never append a twin.@@ -112,7 +114,7 @@ for doc in "$ROOT"/docs/architecture/key-files/*.md; dobytes=$(wc -c < "$doc" | tr -d ' ')if [ "$bytes" -gt 60000 ]; thenfail=1- echo "FAIL: ${doc#"$ROOT/"} is $bytes bytes, over the 60000 cap. Split the subsystem and update KEY_FILES.md." >&2+ echo "FAIL: ${doc#"$ROOT/"} is $bytes bytes, over the 60000 cap. Move its last entries to a continuation page that links back with [Previous entries](<page>.md), and link it from this page's last line; never raise the cap." >&2fidonediff --git a/scripts/structural-suites.tsv b/scripts/structural-suites.tsvindex ec712333e1..353686b188 100644--- a/scripts/structural-suites.tsv+++ b/scripts/structural-suites.tsv@@ -119,7 +119,7 @@ test/destructive-guard.test.ts purgeExpiredSources — clone cleanup containmenttest/detached-stderr.test.ts jobs.ts detach branch (source pin, #4418) 2 source-surface-helpertest/divergence-safe-pull-5429.serial.test.ts --no-autostash (#5429, bug fix) 4 readFileSynctest/docs-mcp-deploy.test.ts DEPLOY.md documents the owner magic-link flow (#5007) 1 readFileSync,source-surface-helper-test/docs-navigation.test.ts primary documentation navigation 1 readFileSync+test/docs-navigation.test.ts primary documentation navigation 2 readFileSynctest/doctor-atom-provenance-drift.test.ts computeAtomProvenanceDriftCheck 19 doctor-source-helpertest/doctor-embedding-env-override.test.ts cross-surface parity (source-grep regression guard) 1 doctor-source-helpertest/doctor-fix.test.ts gbrain doctor --fix CLI integration 3 readFileSyncdiff --git a/test/docs-navigation.test.ts b/test/docs-navigation.test.tsindex 8ba8dbf544..f96485209d 100644--- a/test/docs-navigation.test.ts+++ b/test/docs-navigation.test.ts@@ -48,10 +48,67 @@ function anchors(markdown: string): Set<string> {}describe('primary documentation navigation', () => {- test('every subsystem is directly discoverable from the bounded index', () => {- const index = readFileSync(join(root, 'docs/architecture/KEY_FILES.md'), 'utf8');- for (const name of subsystems) expect(index).toContain(`](key-files/${name})`);+ const linkedPages = (markdown: string, prefix: string) => new Set(+ [...markdown.matchAll(/\]\(([^)#\s]+\.md)(?:#[^)\s]*)?\)/g)]+ .map(match => match[1])+ .filter(href => href.startsWith(prefix))+ .map(href => href.slice(prefix.length))+ .filter(name => subsystems.includes(name)),+ );+ const pageText = (name: string) => readFileSync(join(root, subsystemDir, name), 'utf8');+ const lastLine = (name: string) => pageText(name).trimEnd().split('\n').at(-1) ?? '';+ const indexed = linkedPages(readFileSync(join(root, 'docs/architecture/KEY_FILES.md'), 'utf8'), 'key-files/');++ test('every subsystem page is reachable from the bounded index', () => {expect(subsystems.length).toBeGreaterThan(1);+ const reached = new Set(indexed);+ const queue = [...indexed];+ while (queue.length > 0) {+ for (const next of linkedPages(pageText(queue.shift()!), '')) {+ if (!reached.has(next)) {+ reached.add(next);+ queue.push(next);+ }+ }+ }+ expect(subsystems.filter(name => !reached.has(name))).toEqual([]);+ });++ test('a page the index does not link continues exactly one chain from its predecessor', () => {+ const failures: string[] = [];+ const predecessorOf = new Map<string, string>();+ for (const name of subsystems.filter(page => !indexed.has(page))) {+ const previous = [...pageText(name).matchAll(/\[Previous entries\]\(([^)#\s]+\.md)\)/g)].map(match => match[1]);+ if (previous.length !== 1 || !subsystems.includes(previous[0])) {+ failures.push(`${name}: not linked from the index and has no single [Previous entries](<page>.md) link`);+ continue;+ }+ predecessorOf.set(name, previous[0]);+ if (!linkedPages(lastLine(previous[0]), '').has(name)) failures.push(`${name}: the last line of ${previous[0]} does not link it`);+ }+ for (const [name] of predecessorOf) {+ const seen = new Set([name]);+ let page = name;+ while (!indexed.has(page)) {+ const previous = predecessorOf.get(page);+ if (!previous || seen.has(previous)) {+ failures.push(`${name}: its chain never reaches a page the index links`);+ break;+ }+ seen.add(previous);+ page = previous;+ }+ }+ for (const name of subsystems) {+ for (const next of linkedPages(lastLine(name), '')) {+ if (!indexed.has(next) && predecessorOf.get(next) !== name) failures.push(`${next}: continued from ${name}, which is not its [Previous entries] page`);+ }+ }+ const continuations = [...predecessorOf.values()];+ for (const previous of new Set(continuations)) {+ if (continuations.filter(page => page === previous).length > 1) failures.push(`${previous}: continued by more than one page`);+ }+ expect(failures).toEqual([]);});const anchorCache = new Map<string, Set<string>>();