Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(extract): refuse a missing page on the inline timeline batch (#6805)
garrytan5931b0d6cf2c
diff --git a/docs/architecture/key-files/commands-3.md b/docs/architecture/key-files/commands-3.mdindex 2cb1df41f3..3af2701acb 100644--- a/docs/architecture/key-files/commands-3.md+++ b/docs/architecture/key-files/commands-3.md@@ -31,4 +31,4 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into bounded windows via `src/core/facts/conversation-windows.ts` (#6267: eligibility per 30-min gap group of ≥2 messages decided before windowing; windows close at 30 messages or when the rendered text with its header would exceed 6,500 chars; oversized turns split at sentence boundaries with the speaker prefix repeated; a trailing single reply of an eligible group is its own window, never glued across a gap; nothing is clipped and an unrepresentable header fails the page before prior-fact cleanup; a page's frontmatter `conversation_segment_gap_minutes`, an integer from 1 to 10080, overrides the gap for that page, and any other value is ignored with a stderr warning naming the accepted range), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES` = `CONVERSATION_BODY_MAX_BYTES` in `conversation-parser/body.ts`, `PAGE_MAX_BYTES` + 1 MiB = 25 MiB, measured on the parser input; `readConversationBodyForParsing` stats a `raw_transcript` sidecar and throws `ConversationBodyTooLargeError` over the cap before any read, the snapshot carries `oversizedSidecarBytes`, and a size skip writes a `skipped_too_large` receipt to `page_extraction_state` (doctor `extraction_input_truncated`) that a later run under the cap clears; test `test/conversation-facts-oversized-sidecar.test.ts`; a segmented page is read in ranges; one over `CONVERSATION_WINDOW_BYTES` with no conversation facts is extracted from its newest window (day-header date, `tail_window` receipt, `window_rule: no_prior_facts`); one with facts is skipped with nothing cleared (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears prior facts the window may not see, until PR 4's window anchors (test `test/large-pages/segmented-conversation-facts.test.ts`)); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd` (a DEFAULT cap is dropped with a warning when the extraction or parser chat model is unpriced, via `src/core/facts/conversation-budget.ts` `conversationFactsCostCap` (#5823); explicit CLI/caller/cycle caps stay fail-closed and a `no_pricing` stop reports `budget_reason`/`budget_model`, exits 1 and names `pricing.overrides`); **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); **`--background` via `maybeBackground`**; and **speaker-shaped-fold decline** — when the parse reports `unrecognized_headings` containing a speaker-shaped label (1-2 title-cased words, not in the doc-heading stoplist — the stoplist gates the DECLINE only, so a miss is warn-noise, never data loss) AND the parse produced fewer than two distinct speakers, the page is declined instead of extracted under wrong attribution: counted in `pages_skipped_unrecognized_speaker` (Result/CLI/cycle surfaces), warned to stderr with the folding pattern id, and deliberately NON-terminal (no durable audit row and no orphan cleanup, so a future parser/pattern fix retries the page); multi-speaker pages with folds proceed warn-only. `publishConversationFacts` skips a non-outcome row an active conversation fact of the source already states (`findCaptureDuplicate`, no time bound; `duplicates_skipped`, #7043). The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source except `cycle.conversation_facts_backfill.exclude_sources` (reported as `sources_excluded`), and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts:<source>:<slug>`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). A page's rows are buffered and published once after its last segment (`publishPageFacts` on an unmanaged brain, one `writeDerivedFacts` transaction; the managed batch otherwise), through `publishConversationFacts` (facts/conversation-publication.ts), so a crashed, failed, empty or partial run never removes the prior batch and a terminal audit row commits only with its page's facts (#7013); a run's `scope` is `partial` when fewer segments than in scope ran, `window` when `--since` dropped messages, else `full`, and a page that changed mid-run keeps an unmanaged run's facts as a `partial` batch. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Additive `pages_skipped_unparsed`, `pages_skipped_type_mismatch`, `pages_skipped_insufficient_turns`, and `pages_skipped_since` explain the unchanged `pages_skipped` aggregate across core, CLI, workers, and cycle sources; durable completion remains separate. `--dry-run` stops after segmentation, with no parser-fallback, extractor, or embedding calls and no durable knowledge/checkpoint/receipt/rollup writes. The narrow `python-dict-utterance` registry format and its quoted/nested-example rejection are documented in [conversation parser patterns](../conversation-parser-patterns.md). Checkpoint state is a shared `cpMap: Map<slug, endIso>` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-diagnostics.serial.test.ts` (pool workers, per-page lock skip + exit 3, replay replacing a prior partial run's facts, D15 preflight, `--workers` round-trip). `src/commands/doctor.ts` reads the same outcomes, which survive checkpoint GC. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Explicit `slugs` batches contain page failures like the pool (cancellation and budget stops halt); logs carry only the closed reason and a retry command. Completion requires every in-scope segment (exactly `--segment-limit` counts). Extracted rows take `facts.default_visibility` (resolved once per run); audit rows stay private. Each page's own entities go to the extractor as `entityHints` (slugs, `src/core/facts/conversation-entity-hints.ts`, computed once per page, cap `CONVERSATION_ENTITY_HINTS_CAP` = 40 through `ExtractInput.entityHintsCap`): authored edges only (the page's frontmatter edges with the page as origin, then outgoing markdown, wikilink-resolved and manual links), same source, live fact-entity pages, and a private entity only when the run's facts are private; the `extract_facts` op keeps its 5-hint cap and prompt. Pinned by `test/conversation-entity-hints.test.ts`. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed. `no_match`, errors, cancellation, and dry runs never become durable negatives, bar `facts/conversation-skip.ts` (prose meetings, single emails, undated turns). Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. On a managed brain the core runs `managedDerivedFactsPreflight` (local writer, active source) before any provider work, and a page's facts and terminal (or non-extractable) audit row are buffered, then published in its receipted batch after source activity, page id, revision and parser-input snapshot are rechecked under the page lock, so a failed or raced re-extraction keeps the prior batch; the CLI no longer refuses up front and receipt pages are skipped. Tests: `test/managed-facts-writers.test.ts`, `test/managed-unsupported-preflight.serial.test.ts` (refusal before spend for an unaccepted writer).-- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. A lost link or timeline write on the fs walks or the slug-scoped path leaves its pages unstamped and throws `extract_write_lost` after stamping the rest (`core/extract-watermark.ts`); the sync hooks report a page whose link or timeline write failed in `errors`, never `processed`, so the caller leaves it unstamped. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). The full fs walk replaces each page's own markdown-derived links through `replacePageFileLinks` (the per-slug contract below; a file with no page row is skipped and not stamped); `extract --stale` replaces through `replaceDerivedLinks`, the db link walk through `replaceDerivedLinksBatch`. Timeline rows are inserted in batches of 100 via `addTimelineEntriesBatch` (`ON CONFLICT DO NOTHING`; `created` counts real inserts) after every path — fs walk, db walk, per-slug, sync hooks, `--stale` — runs `retractRemovedTimelineEntries` for the page. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. Every per-slug extractor — `extractForSlugs` (the cycle's incremental pass) and the two sync hooks `extractLinksForSlugs` / `extractTimelineForSlugs` — resolves each requested slug through `resolveSlugRelPath`: the `buildSlugPathIndex(walkMarkdownFiles(dir))` index first, then — only on an index miss — the legacy `join(dir, slug + '.md')` existence check. The index is the primary because `pathToSlug` lowercases and slugifies, so a reconstructed path only finds files whose names are already slugs; the legacy fallback stays because sync ADMITS files the walker never emits (`_`-prefixed names, dot-dirs waived via `sync.include_hidden`) and those slugs still round-trip to their real path — without it such a page imports with no edges. On a slug collision the index prefers the file whose name IS the slug (else the first walked entry), so resolution never depends on read order. The per-slug extractors reconcile like `put_page`: each page's own markdown-derived links are replaced through `replacePageFileLinks` (`src/core/link-reconciliation.ts`, `replaceDerivedLinks` with `preserveExisting`; other producers survive, a page with no row counts as processed with no links, and a failed replacement is reported on stderr and leaves the page stale), and `retractRemovedTimelineEntries` (`src/core/timeline-extract.ts`) deletes the non-event rows whose (date, source, summary) any stored `page_versions` text of the page produced (under either timeline parser) and its current text no longer does; rows no version produced (enrichment, meeting fan-out, `--infer-dates` anchors) are kept, and the version scan only runs when the page holds a row the current text does not produce. `gbrain extract timeline --prune-orphans [--source-id] [--dry-run] [--json]` (`pruneTimelineOrphans`) runs that reconciliation over every page with timeline rows, without inserting — the one-time prune for rows orphaned while extraction was insert-only (managed brains retract each page's rows in a page-locked `withCoordinatedWrite` transaction; these rows are removals, so `gbrain repair timeline` / `timeline_history` exclude them; pinned in `test/persistence-managed-lifecycle.test.ts`); the read-only `timeline_orphans` doctor check (`src/commands/doctor/checks/timeline-orphans.ts`) samples 500 pages and points at it. Pinned by `test/timeline-reconcile-all-paths.test.ts`. `loadSlugAliasTargets` adds renamed pages' old slugs (`slug_aliases`) to `allSlugs` so links written before a rename resolve to the current slug. `extractLinksForSlugs` reads the requested files first, then loads page metadata only for those pages and their link endpoints (`fileLinkOwnership(engine, sourceId, { slugs })`); the unscoped whole-brain load happens only when a DB meeting or attendance origin needs `replaceFileLinks`. Both return `ExtractForSlugsResult { created, processed, errors? }`; `errors` names pages whose read or write failed, and sync reports them as `extract_error` (stderr line plus the `SyncResult` field) instead of swallowing them. `processed` is the subset actually read, and the shared `slugsSafeToStamp(links, timeline)` intersection is what every caller may stamp. Stamping the full requested set marks a silently-skipped page fresh and hides it from `extract --stale` permanently. All three call sites (`src/commands/sync.ts`, `src/core/github-source.ts`, `src/core/google/google-source.ts`) stamp the intersection. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id <id>` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id <id>] [--catch-up] [--retry-held] [--dry-run] [--json]` branch (`extractStaleFromDB`; on a managed brain it hands the sweep to `extractManagedStaleLinks` in `src/core/persistence/links-maintenance.ts` for every caller, CLI, cycle, jobs and maintain, and a live PGLite owner runs it through `src/commands/extract-stale-delegate.ts`; failing pages are reported and skipped there, and `--retry-held` clears held ones first) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale, then the mention pass. Embedded callers (the cycle) pass `quiet: true` so the helper writes nothing to stdout, including its JSON result; standalone CLI output is unchanged. In `jsonMode` every fs path, `extractForSlugs` included, reports a dropped flush as a `{event:'batch_error'}` stderr line rather than going silent. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush (the unmanaged sweep): a page's throw aborts the sweep after the batch's finished pages are flushed and stamped (S4), so the failing page and later ones stay stale and re-extract idempotently. `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. On a managed-persistence brain each page publishes its derived links (`prepareAutomaticLinks`), the coordinator's unrecorded canonical timeline tuples (`unrecordedCanonicalTimeline`, insert-only) and its watermark in one revision-bound `withCoordinatedWrite` transaction; a page edited since the read is skipped and stays stale. Tests: `test/extract-stale-managed.test.ts`, `test/persistence-managed-lifecycle.test.ts`. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Fresh-install schema in `src/schema.sql` (both generated blobs follow via `bun run build:schema`) + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor/checks/extraction-sync.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; pages whose attendance marker (`links_attendance_blocked_revision`, migration v180) equals their current `knowledge_revision` are left out of the lag count and reported as `details.attendance_blocked` (both `extract --stale` sweeps set the marker on an unresolved-attendance skip; pre-v180 brains fall back to the unfiltered count); strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` carries `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` at the inline-extract call site for the slugs the extractors actually read, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` lists `'extract.stale'` in `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` lists `links_extraction_lag` in `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp + crash-contract), `test/sync-inline-extract-stamps.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite case in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. W4.14: one basename index per slug set (rebuilt as it grows). GBRA-133 2b.1: the sync hooks (`extractLinksForSlugs` / `extractTimelineForSlugs`) share one `indexMarkdownTree` walk per run (`src/commands/extract-tree.ts`, with `markdownWalk` behind `walkMarkdownFiles`) (it yields after each directory) and type link endpoints through `withEndpointPageTypes`, which parses only the walked files the pass asks a type for (repeating until nothing new is asked) instead of `loadFsPageTypes` over the whole tree; full `gbrain extract` keeps the whole-tree parse, and the two give the same links (`test/pathological/s-sync-offpath-2b.test.ts`). #6805: `extractTimelineForSlugs` writes each page's rows as one `addProjectedTimelineBatch` (same tier, write gate and dedup as the per-row path); a batch the database refuses is retried row by row through `addProjectedTimelineEntry`, so the page keeps its other rows and is still reported in `errors`. `created` counts rows inserted (`test/extract-timeline-for-slugs-batch.test.ts`).+- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. A lost link or timeline write on the fs walks or the slug-scoped path leaves its pages unstamped and throws `extract_write_lost` after stamping the rest (`core/extract-watermark.ts`); the sync hooks report a page whose link or timeline write failed in `errors`, never `processed`, so the caller leaves it unstamped. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). The full fs walk replaces each page's own markdown-derived links through `replacePageFileLinks` (the per-slug contract below; a file with no page row is skipped and not stamped); `extract --stale` replaces through `replaceDerivedLinks`, the db link walk through `replaceDerivedLinksBatch`. Timeline rows are inserted in batches of 100 via `addTimelineEntriesBatch` (`ON CONFLICT DO NOTHING`; `created` counts real inserts) after every path — fs walk, db walk, per-slug, sync hooks, `--stale` — runs `retractRemovedTimelineEntries` for the page. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. Every per-slug extractor — `extractForSlugs` (the cycle's incremental pass) and the two sync hooks `extractLinksForSlugs` / `extractTimelineForSlugs` — resolves each requested slug through `resolveSlugRelPath`: the `buildSlugPathIndex(walkMarkdownFiles(dir))` index first, then — only on an index miss — the legacy `join(dir, slug + '.md')` existence check. The index is the primary because `pathToSlug` lowercases and slugifies, so a reconstructed path only finds files whose names are already slugs; the legacy fallback stays because sync ADMITS files the walker never emits (`_`-prefixed names, dot-dirs waived via `sync.include_hidden`) and those slugs still round-trip to their real path — without it such a page imports with no edges. On a slug collision the index prefers the file whose name IS the slug (else the first walked entry), so resolution never depends on read order. The per-slug extractors reconcile like `put_page`: each page's own markdown-derived links are replaced through `replacePageFileLinks` (`src/core/link-reconciliation.ts`, `replaceDerivedLinks` with `preserveExisting`; other producers survive, a page with no row counts as processed with no links, and a failed replacement is reported on stderr and leaves the page stale), and `retractRemovedTimelineEntries` (`src/core/timeline-extract.ts`) deletes the non-event rows whose (date, source, summary) any stored `page_versions` text of the page produced (under either timeline parser) and its current text no longer does; rows no version produced (enrichment, meeting fan-out, `--infer-dates` anchors) are kept, and the version scan only runs when the page holds a row the current text does not produce. `gbrain extract timeline --prune-orphans [--source-id] [--dry-run] [--json]` (`pruneTimelineOrphans`) runs that reconciliation over every page with timeline rows, without inserting — the one-time prune for rows orphaned while extraction was insert-only (managed brains retract each page's rows in a page-locked `withCoordinatedWrite` transaction; these rows are removals, so `gbrain repair timeline` / `timeline_history` exclude them; pinned in `test/persistence-managed-lifecycle.test.ts`); the read-only `timeline_orphans` doctor check (`src/commands/doctor/checks/timeline-orphans.ts`) samples 500 pages and points at it. Pinned by `test/timeline-reconcile-all-paths.test.ts`. `loadSlugAliasTargets` adds renamed pages' old slugs (`slug_aliases`) to `allSlugs` so links written before a rename resolve to the current slug. `extractLinksForSlugs` reads the requested files first, then loads page metadata only for those pages and their link endpoints (`fileLinkOwnership(engine, sourceId, { slugs })`); the unscoped whole-brain load happens only when a DB meeting or attendance origin needs `replaceFileLinks`. Both return `ExtractForSlugsResult { created, processed, errors? }`; `errors` names pages whose read or write failed, and sync reports them as `extract_error` (stderr line plus the `SyncResult` field) instead of swallowing them. `processed` is the subset actually read, and the shared `slugsSafeToStamp(links, timeline)` intersection is what every caller may stamp. Stamping the full requested set marks a silently-skipped page fresh and hides it from `extract --stale` permanently. All three call sites (`src/commands/sync.ts`, `src/core/github-source.ts`, `src/core/google/google-source.ts`) stamp the intersection. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id <id>` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id <id>] [--catch-up] [--retry-held] [--dry-run] [--json]` branch (`extractStaleFromDB`; on a managed brain it hands the sweep to `extractManagedStaleLinks` in `src/core/persistence/links-maintenance.ts` for every caller, CLI, cycle, jobs and maintain, and a live PGLite owner runs it through `src/commands/extract-stale-delegate.ts`; failing pages are reported and skipped there, and `--retry-held` clears held ones first) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale, then the mention pass. Embedded callers (the cycle) pass `quiet: true` so the helper writes nothing to stdout, including its JSON result; standalone CLI output is unchanged. In `jsonMode` every fs path, `extractForSlugs` included, reports a dropped flush as a `{event:'batch_error'}` stderr line rather than going silent. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush (the unmanaged sweep): a page's throw aborts the sweep after the batch's finished pages are flushed and stamped (S4), so the failing page and later ones stay stale and re-extract idempotently. `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. On a managed-persistence brain each page publishes its derived links (`prepareAutomaticLinks`), the coordinator's unrecorded canonical timeline tuples (`unrecordedCanonicalTimeline`, insert-only) and its watermark in one revision-bound `withCoordinatedWrite` transaction; a page edited since the read is skipped and stays stale. Tests: `test/extract-stale-managed.test.ts`, `test/persistence-managed-lifecycle.test.ts`. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Fresh-install schema in `src/schema.sql` (both generated blobs follow via `bun run build:schema`) + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor/checks/extraction-sync.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; pages whose attendance marker (`links_attendance_blocked_revision`, migration v180) equals their current `knowledge_revision` are left out of the lag count and reported as `details.attendance_blocked` (both `extract --stale` sweeps set the marker on an unresolved-attendance skip; pre-v180 brains fall back to the unfiltered count); strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` carries `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` at the inline-extract call site for the slugs the extractors actually read, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` lists `'extract.stale'` in `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` lists `links_extraction_lag` in `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp + crash-contract), `test/sync-inline-extract-stamps.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite case in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. W4.14: one basename index per slug set (rebuilt as it grows). GBRA-133 2b.1: the sync hooks (`extractLinksForSlugs` / `extractTimelineForSlugs`) share one `indexMarkdownTree` walk per run (`src/commands/extract-tree.ts`, with `markdownWalk` behind `walkMarkdownFiles`) (it yields after each directory) and type link endpoints through `withEndpointPageTypes`, which parses only the walked files the pass asks a type for (repeating until nothing new is asked) instead of `loadFsPageTypes` over the whole tree; full `gbrain extract` keeps the whole-tree parse, and the two give the same links (`test/pathological/s-sync-offpath-2b.test.ts`). #6805: `extractTimelineForSlugs` writes each page's rows as one batch through `addProjectedPageTimeline` (same tier, write gate and dedup as the per-row path; an absent or soft-deleted page fails the slug with addTimelineEntry's `PageMissingError`); a batch the database refuses is retried row by row through `addProjectedTimelineEntry`, so the page keeps its other rows and is still reported in `errors`. `created` counts rows inserted (`test/extract-timeline-for-slugs-batch.test.ts`).diff --git a/docs/architecture/key-files/security.md b/docs/architecture/key-files/security.mdindex eb1e400d51..6578597237 100644--- a/docs/architecture/key-files/security.md+++ b/docs/architecture/key-files/security.md@@ -30,7 +30,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/trust/` — memory trust tiers (#5575). `tier.ts`: the six tiers in filter order (`user_confirmed` > `operator_curated` > `tool_observed` > `agent_written` > `unknown` > `external_untrusted`), `TRUST_TIER_LABELS` (the fixed user-facing labels; enums only in JSON), `USER_SAID_TRUST_LABEL` (the label of an `agent_written` row whose `write_origin.content_origin` is `user_said`; the tier never rises, the eval harness reads the constant), `minTrust`/`maxTrust`/`admitsTrust`/`tiersAtOrAbove`, `contentOriginTier` (`tool_output` lowers to external), `storedMinTrust` (a damaged token floor fails closed to the strictest), and `effectiveWriteTrust`, which computes a write's tier once (derived writes capped at `agent_written` and their least trusted input; `write_origin.taint_inputs` keeps the 32 least trusted with `taint_inputs_truncated`/`taint_input_count`). `schema.ts` (migration only, like attribution): `trust_tier` + `write_origin` on facts, takes, timeline_entries and pages, the page_versions snapshot, `min_trust` on oauth_clients/access_tokens, `trust_proposals`, and the BEFORE ROW `trust_tier_stamp` trigger (fires last: INSERT gets the writer's tier from `gbrain.write_trust_tier` or `unknown`; a `TRUST_CONTENT_COLUMNS` change restamps for an owner-tier writer and takes min(prior, writer) otherwise; an unchanged row keeps its tier; a raise needs `gbrain.write_trust_promotion` covering it, or `gbrain.write_trust_backfill` from `unknown` to below `user_confirmed`, else `trust_raise_refused`). Writers declare through `WriteAttribution.trust`, `withWriteTrust` and `maintenanceTransaction(…, trust)` (`persistence/context.ts`; nested declarations combine as min, a group member resets). `confirm.ts`: `requireOwnerConfirmation` (a remote caller needs `memory_confirm`; the local CLI needs a TTY where the user types the target token; `--yes` never confirms), `tierRaiseFix`, `trustRaiseRefusal`, test seam `__setConfirmationIoForTests`. `min-trust.ts`: the local-only floor writes behind `gbrain auth create --min-trust` and `gbrain auth set-min-trust`. `backfill.ts`: `gbrain trust backfill` (deterministic signals, read-only dry run that works before the migration, resumable batches inside `withCoordinatedWrite` on managed brains) and the doctor `trust_tiers` counts. Tests: `test/trust-tier.test.ts`, `test/trust-tier-schema.test.ts` and `test/trust-backfill.test.ts` (PGLite and Postgres), `test/trust-confirm-scope.test.ts`. Write side (lane L1a): `channel.ts` computes the tier once per write from the channel (`requestChannelTrust`: agent verbs agent_written, `content_origin: tool_output` external_untrusted, connector jobs external_untrusted; owner sync/import `ownerImportTrust`: operator_curated or the `sources set-trust` default, lowered by frontmatter markers `frontmatterTrustCaps`) and `stampTrustMarker` (the lower-only `trust_tier` marker, CEO-21); `page-write.ts` wires it into page-prepare (marker stamp, fence edits keep the page tier via `withTrustKeep`, `lower_page` proposals on agent rewrites, the page gate tier; on an unclassified page `stampPageTrustMarker` also caps the marker by the stored frontmatter's own lowering markers, so a write that removes one cannot raise the page, #7873; a derived `submit_job` page with a derivation declaration stamps the marker too, #7905); `proposals.ts` is the `trust_proposals` store and handler dispatch (`decide.ts` loads every handler); `supersede-handlers.ts` is guarded supersession (`supersessionGuarded`, contested fact/take/ontology proposals, remote `forget_requires_owner`, the trust store for the checked decide_proposals supersede); `fence-guard.ts` guards the facts-fence re-projection and gates new fence rows, and holds take and timeline rows a lower-tier write would delete, replace or deactivate (`guardFenceTakes`, `heldRows`, `contestHeldRows`; projection, `takes_remove`, takes reconcile, timeline orphan prune; W1 #6783 §12 (a)); `held-removals.ts` is the owner's accept of those `forget` proposals (the page's text rules the row; a refused `takes_remove` reruns as an owner `takes_remove` bound by a nonce); `propagation.ts` lowers rows derived from a lowered input through `derivation_inputs` and never promotes (`propagateTierLowering`, §12 (b)); tests `test/trust-held-removals.test.ts` (PGLite and Postgres); `gate-outcomes.ts` is the DX-1 `gate` field and the held-write verb error. Owner actions: owner actions (#5575 lane L1a-owner: A4, CEO-2, CEO-9, CEO-12, DX-2, DX-7, DX-10, DX-13, DX-14, DX-15, ENG-15). `owner-actions.ts`: `previewOwnerAction` / `applyOwnerAction` for confirm, drop, release, revert, allow, allow_remove, disable and enable: the preview names the summary, the typed token (the ref, or a hash8 for page refs), the exact command, whether the action raises trust and a `binding` of the target's exact state (row content and tier, page revision, proposal status); apply re-previews, refuses a moved binding with `preview_changed` and a raising action without a confirmation with `confirmation_required` (`ownerActionFix` is the DX-3 `tell_user_to_run` fix). Fact and take confirms are tier-only updates under `withTrustPromotion` (inside `withCoordinatedWrite` on a managed brain); the hold store is an isolated adapter over `write_gate_holds` (`__setHoldStoreForTests`, `TODO(L2a-merge)`); on a managed brain its `publish` (`release-publish.ts`) releases a held fact whose page exists as a journaled `trust_release` request: `prepareTrustReleaseMutation` appends the fact to its entity page's facts fence with `prepareFenceFactAppend` (or, for a hold naming `payload.row_num`, indexes the row its file already holds), flips the hold and writes the row at user_confirmed (origin `trust_release`, ledger cause `{hold}`) in one publication, locks and rechecks the hold in validate (`preview_changed`), previews and binds an index-existing release's cell fields and refuses `preview_changed` when the cell no longer matches (`releaseCell`, `releaseCellFields`, origin `fields_previewed`; #7656; doctor `trust_release_unpreviewed_fields` lists earlier releases without it), while the database-only fallback re-checks the binding on the locked hold (#7657), keys the request id on the hold and its approved state, and refuses a deleted or private page with the fact still held; a hold with no entity slug or no page keeps the database-only release; `disable --all` writes `write_gate.external_mode=off`, `write_gate.agent_mode=off`, `trust.agent_activation=allow` with the `trust.kill_switch` receipt that `--undo` restores from. `page-handlers.ts`: `confirmPage` (removes the CEO-21 `trust_tier` marker through a journaled `trust_owner_page` mutation on a managed brain, `importFromContent` otherwise; tier only when no marker), `revertPageToVersion` (the version's content and its snapshot tier, `unknown` for legacy versions), the coordinator preparer `prepareTrustOwnerPageMutation` (dispatched from `persistence/service.ts`; re-checks the approved revision and refuses a remote request without `memory_confirm`), and the `lower_page` handler (accept = confirm, reject = dismiss, `revertLoweredPage` = restore the prior owner version). `refs.ts`: typed refs `f<id>`, `t<id>`, `h<id>`, `tp<id>`, `a<id>`, `p:<source>/<slug>`, bare slugs resolved by `--source` or a single source (ambiguity refuses). `review.ts`: `buildTrustReview` (pending proposals, lowered pages, unconfirmed standing preferences, holds, allow rules, kill-switch state, grouped by day and page, complete argv) and `explainTrust`. `allow-rules.ts`: the `trust_allow_rules` table (migration v229, receipted, removal keeps the row) and the pure `matchTrustAllowRule` over server-stamped fields. `owner-ipc.ts`: the `trust_read` / `trust_preview` / `trust_apply` administration handlers a resident owner serves over its 0600 socket to the verified local CLI only. Tests: `test/trust-owner-actions.test.ts` (PGLite and Postgres), `test/trust-release-fence.test.ts` (PGLite and Postgres), `test/trust-owner-ipc.test.ts`, `test/trust-cli.test.ts`. Derived-row taint (`taint.ts`, `derived-gate.ts`, `timeline-projection.ts`): see [security](security.md).- `src/core/eligibility/{policy,sql,labels,registry,stamp,activation,scan,explain}.ts` — the one reversible read eligibility policy for memory trust (#5575 A6, A7, CEO-18, CEO-20, DX-6, DX-10, ENG-8, ENG-14). `policy.ts`: `resolveReadEligibility` returns the effective floor (the strictest of `AuthInfo.minTrust`, the caller's `min_trust` and `trust.read_policy=filter`'s own `unknown` floor; label mode adds nothing, so ordering is byte-identical) and, for proactive reads, `suppressFlagged` unless `trust.agent_activation=allow`. `sql.ts`: vetted predicates over a caller alias (closed vocabularies only): `trustFloorSql`, `pageEligibleSql` (spliced by `buildVisibilityClause` and `pageReadFilter`, so every search arm filters before LIMIT; a chunk's tier is its page's), `projectionEligibleSql` (facts, takes and timeline reads: a quarantined page's projections and `needs_rederive` rows are hidden, never expired) and `activationSuppressedSql` (tier at or below `agent_written` plus an instruction-family `write_gate_receipts` flag). `labels.ts`: `trust_tier` + a short server-stamped `origin` (`trustFields` appends `:user_said` for a relayed user statement, which `trustLabelWords` reads as `USER_SAID_TRUST_LABEL` unless the row is flagged unconfirmed); compact per-item text labels with the fixed words, `unknown` as a short origin line, and the `<external-data>` envelope only below `unknown`. `registry.ts`: `OP_READ_TRUST` classifies every read-scope op `filtered | labeled | text_free` (the dispatcher refuses a `labeled` op for a floored token) and `PROACTIVE_SURFACES` lists the injecting surfaces, each calling `proactiveEligibility(ctx, id)`; drift-tested by `test/eligibility-registry.test.ts`. `stamp.ts` labels rows after ranking and backstops the floor for rows a stage adds by slug; explicit reads mark flagged rows `unconfirmed`. `activation.ts` partitions proactive candidates (`partitionForActivation`, `pageActivationVerdicts`) and builds the `memory_suppressed` notice (count + `gbrain trust review`, never content). `scan.ts` (`gbrain trust scan`, doctor `trust_scan`) records flag receipts for agent-written-or-lower rows written before the gate, in resumable keyset batches bounded by the id ceiling of its first run (later rows are gated by their writers; a detector bump rescans). `explain.ts` is the `gbrain trust explain` data API: tier, origin, receipts and the per-surface decision for a typed ref. Tests: `test/eligibility-{policy,registry,reads}.test.ts`.-- `src/core/trust/taint.ts`, `derived-gate.ts`, `timeline-projection.ts` (#5575 derived rows; the rest of `src/core/trust/` is above): `taint.ts` (I2, ENG-3/ENG-7): `readTaintInputs`/`deriveTrust` read the stored tiers of every row a deriver placed in its prompt (a vanished input reads `unknown`); `derivedWriteTrust` caps model output at `agent_written` and a deterministic projection (fence or timeline parse) at `operator_curated`; `recordTaintEdges` writes the complete `derivation_inputs` edges; `lowerToDerivedTier`/`lowerDerivedPage` (and `lowerDerivedPageIn`, the same inside a caller's coordinated transaction) lower a derived row whose republish changed no content column; `derivedMaintenanceTransaction` is the unmanaged deriver seam; `declareDerivation`/`declaredWriteTrust` carry a derivation in a maintenance or facts intent (the preparer stamps it, never above `agent_written`); `readDerivedTaintShare` is the doctor CEO-24 diagnostic. `derived-gate.ts`: the derivers' row-level write gate at the declared tier (`derivedGateConfig`, `applyGateDecision`: insert + flag receipt, hold, or skip a reject; `recordTimelineFlag`). `timeline-projection.ts`: `gbrain extract` timeline walks stamped per page tier and gated. Tests: `test/trust-taint.test.ts`, `test/trust-taint-origin.test.ts` (PGLite and Postgres), `test/trust-taint-dream.test.ts`.+- `src/core/trust/taint.ts`, `derived-gate.ts`, `timeline-projection.ts` (#5575 derived rows; the rest of `src/core/trust/` is above): `taint.ts` (I2, ENG-3/ENG-7): `readTaintInputs`/`deriveTrust` read the stored tiers of every row a deriver placed in its prompt (a vanished input reads `unknown`); `derivedWriteTrust` caps model output at `agent_written` and a deterministic projection (fence or timeline parse) at `operator_curated`; `recordTaintEdges` writes the complete `derivation_inputs` edges; `lowerToDerivedTier`/`lowerDerivedPage` (and `lowerDerivedPageIn`, the same inside a caller's coordinated transaction) lower a derived row whose republish changed no content column; `derivedMaintenanceTransaction` is the unmanaged deriver seam; `declareDerivation`/`declaredWriteTrust` carry a derivation in a maintenance or facts intent (the preparer stamps it, never above `agent_written`); `readDerivedTaintShare` is the doctor CEO-24 diagnostic. `derived-gate.ts`: the derivers' row-level write gate at the declared tier (`derivedGateConfig`, `applyGateDecision`: insert + flag receipt, hold, or skip a reject; `recordTimelineFlag`). `timeline-projection.ts`: `gbrain extract` timeline walks stamped per page tier and gated; `addProjectedPageTimeline` writes one page's own timeline as one batch that holds the live page FOR SHARE and refuses a missing one (`requirePages`), falling back row by row when the database refuses the batch (#6805). Tests: `test/trust-taint.test.ts`, `test/trust-taint-origin.test.ts` (PGLite and Postgres), `test/trust-taint-dream.test.ts`.- `src/core/trust/claim.ts` + `claim-state.ts` + `claim-notice.ts`, `src/commands/trust-claim.ts`: #5575 legacy content. `gbrain trust claim-sources` (local CLI; typed source id on a TTY, `--yes` never counts, non-TTY refuses `confirmation_required` with a tell_user_to_run fix) records a claim in `sources.config` (`trust_tier: operator_curated`, `trust_claimed_at`, `trust_claim_lifted_at`) and lifts the source's legacy `unknown` rows (ids at or below `write_gate.scan_baseline`) through `trust/backfill.ts` scoped to the claimed sources (own cursor, `write_origin.channel` `trust_claim`), lowered by `frontmatterTrustCapsSql` (channel.ts, pinned to `frontmatterTrustCaps`) and every backfill signal, never above operator_curated; connector sources refuse. Doctor `trust_sources_unclaimed`, `gbrain post-upgrade` and the behavior-change notice name the claim (`actor: user`, tell_user_to_run); doctor `trust_scan`'s fix is the user's to run; the scan refuses while a lift is pending; explain and review show pending rows as owner tier. `claim-sources --relower [--dry-run]` (`relowerClaimedRows`, backfill.ts) is the lower-only #6533 repair: `trust_claim` rows whose page rule now ranks below their stored tier (legacy `extracted_by: extract_atoms-*` atoms) drop to it, pages first with child rows following through the relower-mode page taint, receipt `write_origin.channel` `trust_relower`; the dry run reports the same rows; doctor `trust_sources_unclaimed` names it. Guide: `docs/guides/memory-trust.md`. Tests: `test/trust-claim.test.ts`, `test/trust-claim-relower.test.ts`.- `src/core/trust/marker-repair.ts`: #6550 `gbrain trust backfill --repair-unknown-markers` (local CLI, `src/cli/commands/trust.ts`). `previewMarkerRepair` lists live pages whose frontmatter `trust_tier` is `unknown` while the row is higher, with evidence (first `page_versions` snapshot showing the marker vs the fixed release `UNKNOWN_MARKER_FIX_RELEASE`: its commit time bounds the first run, `upgrade-state.json` can name it; whether the oldest kept version had the marker), an `assessment` (`consistent_with_6550` only when a pre-fix snapshot shows it and the first import did not, else `ambiguous`), pages already lowered through the marker (with their revert command), a `preview_hash` and the apply argv naming only the consistent pages. `applyMarkerRepair` refuses `preview_changed` unless the fresh preview hashes to `--expect`, then calls `page-handlers.ts` `repairPageMarker` per named slug: the `trust_owner_page` mode `repair_marker` (refused for remote requests) rewrites the marker to `repairedMarker(stored tier)` (none at owner tiers) at the previewed revision and keeps the tier. `review.ts` adds a `repair` command to `lower_page` items an `unknown` marker caused. Pinned by `test/trust-channel-writes.test.ts` (#6550 repair) and `test/trust-cli.test.ts`.- `src/core/trust/provenance-drift.ts`, `src/core/repair/provenance.ts`, `src/commands/doctor/checks/provenance-stamp-drift.ts`: #7891 / #7918. `analyzePageDrift` (pure) walks a page's `page_versions` plus its live row; each transition's writer is the older snapshot's `archived_write_request_id` (joined to `persistence_requests.id` for `authority.remote`, else the principal kind; none: `unattributed`). Verdicts: `remote`, `ambiguous`, `local`, `needs_review` (a later local write changed the same stamp) and `restamp` (a classified page whose live caps rank above its stored tier). A restore puts each stamp back to the server-written column or the value before the first non-local change, and stamps min(lost cap, live caps, stored tier) with `stampTrustMarker`, so it only lowers. `scanProvenanceDrift` is bounded (`DRIFT_SCAN_LIMIT` per source) and degrades to `pre_trust` / `pre_attribution`. The doctor check is read-only; the repair kind is explicit-only and preview-bound (`provenance-v1` hash, approved set) and writes through `trust_owner_page` mode `restore_provenance` (`restorePageProvenance` in `page-handlers.ts`, local CLI only, re-checked against `plannedDigest`, files `lower_page` with proposer `provenance_repair`) or `repair_marker` for `restamp`. A page sync holds is listed `held` by the preview and skipped `held` by the apply, both through `pageHolds` (#8009), so the apply writes what the preview counted. Both repair modes render the page to its file on a classic checkout (`writePageThrough` with the preimage bytes). #7917: `recordAgentPageLowering` also files for derived publications and for callers that name a proposer; `prepareMaintenanceMutation` and `derivedMaintenanceTransaction` (rows carrying `prior`) call it. #7918: `derivationWriteTrust` rates a present but unreadable declaration `unknown`.diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsvindex 8ab581d8de..91c4fa72d0 100644--- a/scripts/module-size-limits.tsv+++ b/scripts/module-size-limits.tsv@@ -8,7 +8,7 @@ src/commands/autopilot.ts 1662 ratchet refactor wave 1 split target (W4 autopilosrc/commands/doctor.ts 700 ratchet +2 Wave B decision #10: Check.skipped_segmented field (logic in doctor/check-fix.ts withSkippedSegmented); +19 #6696 doctor tiers: Check.elapsed_ms and DoctorReport tier/complete/unknown_sections/elapsed_ms/budget_ms fields, buildChecks/runDoctor deadline params (logic in doctor/budget.ts); +1 agent operator gate fixes: --probe consent gate (logic in doctor/probe-consent.ts); refactor wave 1 split target (W4 doctor); ceiling rationale: docs/architecture/key-files/commands-2.md entry for src/commands/doctor.ts; +4 #7142: DoctorReport.last_known_sections field and the buildChecks last-known/refresh scope params (logic in doctor/last-known.ts); +8 #7276: DoctorReport.unknown_count field and the unknown category/count lines of the text headline (scoring logic in doctor/budget.ts scoredChecks)src/commands/embed.ts 2306 ratchet +2 agent operator wave D2: --json failure envelope via writeCliError; lowered v0.46.26.0: embedBatchWithBackoff + retry cluster peeled to core/embed-retry.ts (core/commands layering fix); grown wave-g (#4530): per-model input-limit wiring; wave-k #3622 reimpl: --stale zero-progress failure quarantine; +29 #4668 (#4530): oversize-heal call sites in both --stale drains — kept low by peeling the healed-row remap to core/embed-oversize-heal.ts (shared with embed-stale.ts); +35 #4647 (refs #4599): per-tick timeout+abort on the single-flight lock heartbeat refresh (interval deliberately NOT unref'd); +122 refs #4599: stall-watchdog race around the drain + shared idempotent flushAndRelease + CLI stall exit mapping (watchdog machinery itself lives in core/embed-stall.ts); +12: adversarial-review hardening (healed_splits progress key + heal-tick comment); #5105: guarded derived projection preparation and read seals; #5105: conditional embedding installer wiring; archived eligibility: active-only dispatch with honest stranded-work reporting; explicit archived admission: live-source snapshots and honest refused-installation failures; +2 fw3: embed --stale --images dispatch (sweep lives in core/embed-stale-images.ts); +5 #5289: dry run reports restamp-only chunks apart from chunks to embed (count lives in core/embedding-invalidation.ts); +6 #4616: keep a partially refused batch in embedPageTexts and name the recovery command in failure samples (guard lives in core/ai/embedding-guard.ts); +5 #5226: embed summary counts distinct pages; +3 #5804: null projection snapshot counted as a failure with one batch summary in embed --stale; +18 wave 8 lane D1: #5527 mixed-space warning subtracts current-model chunks (+4), #5885 takes option/result fields and the stale-drain takes call (pass lives in core/embed-takes.ts); +9 Foundations 1 F4d: time-budget stop result fields, stdout verdict and stop recording in embed --stale (logic in core/embed-budget-stop.ts); +8 green-master lane A: chunkless install re-reads through retryProjectionConflict instead of recording an unavailable page (retry lives in core/page-state/projections.ts, failure line in core/agent-output.ts); +8 fix wave 13 P1.18: runEmbed consent parameter, cost-cap stop wiring and result fields, --slugs parsing to the next flag (cap, verdict and resume logic live in core/embed-cost-cap.ts); +28 large pages A9/T3/T16: --held listing and per-page cap metering on every embed path (logic in core/page-embedding-budget.ts meterPageBatch/spendGrant/heldEmbedArgv), approvedPageCapUsd from --max-usd on --slugs; +1 #6722: an explicit slug embed skips an embed_skip page (it keeps keyword chunks, never vectors); merge v0.60.240.0: large pages and master growth compose at 2265; +39 large pages Wave B lane 1e: segmented pages route to the keyset sweep on the slug, --all and --stale paths, with a dry-run count and provenance stamp (logic in core/page-body/segmented-embedding.ts); +2 agent operator wave D2: --json failure envelope via writeCliErrorsrc/commands/extract-conversation-facts.ts 1949 ratchet grown v0.46.x per-provider gateway-unavailable copy; grown wave-g (#4482): expected budget/deadline caps classified apart from error halts; wave-k: save-time resolution caller (#4052/#3729 — supersedes wave-g's per-row resolveEntitySlug mapper); +1: key-aware llm-fallback default (#3813); +1: pricingOverrides wiring into the owned BudgetTracker (#4633 adoption); +3: #4869 snapshot-mismatch counts as pages_failed; merged reliability/shared-skills source integration; +19: additive skip reasons and model-free dry-run (#5364); +21 #5757: managed extraction buffers a page's rows and replaces the prior batch in one coordinator transaction (replacement helper lives in core/persistence/derived-facts.ts); +18 #5823: result diagnostics, strict explicit-cap parsing and CLI recovery/help; default-cap decision lives in facts/conversation-budget.ts; +18 fix wave 7 (#5704, #5448): --json envelope on stdout (parser flag, help line, summary block gated behind the human-output branch); +2 fix wave 10 (#6037, issue #6033): explicit-slug page failures are contained per page through one bounded failure logger (recordPageFailure) shared with the pooled path; +4 fix wave 10 (#5120): facts.default_visibility resolved once per run (resolveDefaultVisibility) and applied to extracted rows only; 2 P2 segmentObservationDate + segmentValidFrom helpers (date-grounding observation time per segment; extractor event date wins over segment start); +4: ExtractConversationFactsResult.failed_pages (interface field) and the help text exit-status paragraph; partial-success logic lives in core/facts/conversation-run-status.ts; re-summed at the master 6ccf30c3e merge (2179 master lines + 20 integration lines over base); re-summed at the #6020 merge on actual lines (as rehearsed on capy/mpw-integration-p2); re-summed at the wave 10 merge on actual lines (+1 recordPageFailure fills failed_pages); lowered at the wave-9 follow-ups merge: the three zero-result literals became emptyExtractConversationFactsResult(); lowered to 2075 at wave 15 P2.5 (#6267): segment tunables, splitIntoSegments and renderSegmentForExtraction moved to core/facts/conversation-windows.ts, the message/segment types to core/facts/conversation-types.ts (re-exported here); +12 large pages T2: the oversized-sidecar snapshot and the durable size-skip receipt (skippedForSize; the cap and the bounded sidecar read live in core/conversation-parser/body.ts, the receipt in core/extraction-watermark.ts); merge v0.60.240.0: large pages and master growth compose at 2087; -6 wave 18 lane 3 (#6731 #6764): the two extractor call sites share one input object, inference lives in core/facts/conversation-subject-infer.ts; Wave B PR 2 (kept at the master 4e29d0904 merge): +7 large pages PR 2 lane 2e2: a segmented conversation over the window cap (snapshot.window, the tail_window receipt, the window fallback date; the window read lives in core/page-body/segment-conversation.ts); +9 lane 2e2 follow-up: the lossless window rule (a page over the window with conversation facts keeps them and is skipped, the receipt names the rule; the count lives in core/facts/conversation-publication.ts); integrated at the master 4e29d0904 merge: 2097 measured; lowered to 2044 by #7013: a page's rows publish once through publishConversationFacts (core/facts/conversation-publication.ts), so the delete-first cleanup, the per-segment inserts and the row-number peek left; +1 GBRA-159: facts from a private page are private (tightenToPage per page; helper in core/facts/visibility.ts); +3 #7438: ExtractConversationFactsResult.facts_skipped_purged (interface field) and one notePurgedSkips call (the count and log live in core/facts/conversation-publication.ts); grown v0.46.x per-provider gateway-unavailable copy; re-summed at the Wave B PR 4 merge with v0.60.452.0 (1949 measured)-src/commands/extract.ts 2706 ratchet +8 GBRA-130 A-1: extract --stale --retry-held flag, help lines and the managed result failure fields (logic in core/persistence/links-maintenance.ts and links-extraction-holds.ts); grown v0.46.26.0 megawave: FS-walk stamp snapshot + legacy timeline repair wiring (#3957 D4) + PR#4486 rebase onto current master; grown wave-g (#4542): --from-meetings zero-match warning + default-pass note; +29: #3478 federated-only gate on the cross-source 'default' link fallback; +44: #3908 cross_source opt-in — counted drops + deterministic cross-source pick; +36: #4611 configured-sources.default fallback (resolveLinkFallbackDefault); +14: #4890 extractStaleFromDB quiet opt + extractForSlugs jsonMode batch_error stderr fallback; +5: #4855 stroke-letter fold in link extraction; +5: #4793 by-mention scan honours link_resolution.cross_source; +18: wave review — ExtractOpts.quiet threaded through the fs-walk extractors (stdout silence decoupled from jsonMode); +77: shared buildSlugPathIndex — every per-slug extractor resolves the real on-disk path, and the sync hooks report which slugs they actually read; +10 #4999: includeFrontmatter default-resolved inside extractLinksForSlugs + extractStaleFromDB (resolveIncludeFrontmatter), --stale flag absent -> configured knob; +20: pre-landing review of #4967 — resolveSlugRelPath: index miss falls back to the legacy slug+'.md' path so admitted _-prefixed / include_hidden pages keep their edges; T2: source-qualified inference and atomic origin-scoped derived links; standard DB extraction fences captured endpoint revisions; shared authoritative attendance/source reconciliation moved to the core module; +7 A2 preview-bound attendance repair import, help and authority dispatch; bounded incomplete-attendance diagnostics and consistent basename resolution; fail closed before graph mutation when active ontology is unavailable; +16 source-selected ontology cache, type-scoped preflight and non-starving failed stale drain; +7 write-path wave: per-slug extractors replace page links + retract removed timeline rows + rename-alias resolution; +5 fw3 derived-sync: timeline --prune-orphans CLI (#4649) and full-walk page link replacement; +13 fw3 A16/A15: sync link hook reads scoped page metadata and reports per-page extraction errors; +1 fw3 managed-lifecycle: extract --stale dispatches managed brains to the coordinator-safe path; +4 fw3 integration: managed brains refuse the direct timeline orphan prune; +26 #5609: managed extract --stale publishes links, canonical timeline and watermark per page in one coordinated transaction; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +4 at the merge; +6 fix wave 3 lane B: a managed brain routes the extract phase file walk through extractManagedStaleLinks instead of the refused raw timeline batch, keeping the extract.links_fs progress phase; +6 #5761: the unmanaged extract --stale sweep marks unresolved-attendance skips attendance-blocked; +2 eval-wave N9-3/N12-7: a page-type-bound pack attended rule defers to canonical attendance on the filesystem path (ownsAttendanceInference); lowered to 2528 #5904: the DB timeline walk moved to commands/extract-timeline-db.ts (coordinated publication on managed brains, honest refusal + non-zero exit); +13 entity mention index: extract --stale runs the mention pass after link work (half the budget reserved while mention-due pages exist), dry-run and JSON report mention counts, and the help documents the pass and `extract mentions --explain` (logic in core/mentions/); cat40 entity wave merge with v0.60.61.0 (2541 measured); +6 fix wave 10 lane L1 #5556: quarantined pages skipped by the file walks, slug lists and DB walks (slug lookup in quarantine.ts quarantinedSlugs); +3 #5556: the abort check stays its own statement in each quarantine-aware scan loop (cycle-abort coverage pin); 2 P5 wanted pages: both DB extract paths record unresolved authored refs with each page's links; +5 P5 line grammar: stated relation types on the filesystem link scan; +3 PGLite stale link drains refresh planner statistics before and during the drain (logic in planner-stats.ts plannerStatsForLinkDrain); +10 fix wave 12 W2.1 S4 (#6228): extract --stale flushes and stamps the pages a batch finished before an aborting page; +12 P5 wanted pages: both DB extract paths record unresolved authored refs with each page's links; +9 Q2 parser gaps: both DB extract paths read the line-grammar settings once, publish through replaceDerivedLinksUnlessSettingsChanged and report pages a mid-run settings change left stale; +25 GBRA-69: extract --source db reads snapshots and publishes derived links BATCH_SIZE pages at a time (readPageSnapshotsBatch in core/page-snapshot-batch.ts, replay in derived-links.ts replaceDerivedLinksBatchOrReplay); +2 GBRA-75 w7: the --source db links walk refreshes planner statistics (logic in planner-stats.ts plannerStatsForLinkDrain); +10 fix wave 13 P2.16 (#6272): both unmanaged DB link drains defer and count a page a concurrent writer moved (skipped_concurrent_write) instead of aborting the run (classifier and skip line in core/derived-links.ts); +3 fix wave 13 P2.18 (S4): extract all --source db stamps the pages both phases completed at one revision, in a finally (stampCompletedIntersection), instead of every links-phase page; +1 fix wave 13 P2.15 (#6273): --from-meetings exits non-zero while managed entity pages are still pending; +7 fix wave 13 PR2: extractFromDB split out of runExtract so runExtract stays under its function-size baseline; Wave B PR 2 (kept at the master 4e29d0904 merge): +3 PR 2 fix lane J1: extract --stale reads a segmented page block by block (extractSegmentedPageLinks, segmentedTimelineText) now that the stale list includes it; integrated at the master 4e29d0904 merge: 2635 measured; +23 fix wave 20 #6954: the fs walks and the slug-scoped extract collect lost link/timeline writes, stamp only the pages whose writes landed and throw extract_write_lost (helpers in core/extract-watermark.ts); fix wave 20 integration: lane 1 (#6808, +11) and lane 2 (#6954, +23) compose at 2677 measured; +16 #7394: --type/--since/--infer-dates on the file walk refuse with the --source db command beside the other --source db refusals, and the help says so; +5 GBRA-161 PR 5 (T-2, L-6): timeline retraction and the mention rebuild sweep are bound to the revision they read (PageRevisionConflictError import, the scanned-revision read and its FOR SHARE check); +2 GBRA-133: ExtractOpts.timeBudgetMs, forwarded to the managed stale-page pass so the cycle bounds a pass with no slug list; +6 #6805: extractTimelineForSlugs writes each page's timeline rows as one projected batch, retried row by row when the database refuses it+src/commands/extract.ts 2700 ratchet +8 GBRA-130 A-1: extract --stale --retry-held flag, help lines and the managed result failure fields (logic in core/persistence/links-maintenance.ts and links-extraction-holds.ts); grown v0.46.26.0 megawave: FS-walk stamp snapshot + legacy timeline repair wiring (#3957 D4) + PR#4486 rebase onto current master; grown wave-g (#4542): --from-meetings zero-match warning + default-pass note; +29: #3478 federated-only gate on the cross-source 'default' link fallback; +44: #3908 cross_source opt-in — counted drops + deterministic cross-source pick; +36: #4611 configured-sources.default fallback (resolveLinkFallbackDefault); +14: #4890 extractStaleFromDB quiet opt + extractForSlugs jsonMode batch_error stderr fallback; +5: #4855 stroke-letter fold in link extraction; +5: #4793 by-mention scan honours link_resolution.cross_source; +18: wave review — ExtractOpts.quiet threaded through the fs-walk extractors (stdout silence decoupled from jsonMode); +77: shared buildSlugPathIndex — every per-slug extractor resolves the real on-disk path, and the sync hooks report which slugs they actually read; +10 #4999: includeFrontmatter default-resolved inside extractLinksForSlugs + extractStaleFromDB (resolveIncludeFrontmatter), --stale flag absent -> configured knob; +20: pre-landing review of #4967 — resolveSlugRelPath: index miss falls back to the legacy slug+'.md' path so admitted _-prefixed / include_hidden pages keep their edges; T2: source-qualified inference and atomic origin-scoped derived links; standard DB extraction fences captured endpoint revisions; shared authoritative attendance/source reconciliation moved to the core module; +7 A2 preview-bound attendance repair import, help and authority dispatch; bounded incomplete-attendance diagnostics and consistent basename resolution; fail closed before graph mutation when active ontology is unavailable; +16 source-selected ontology cache, type-scoped preflight and non-starving failed stale drain; +7 write-path wave: per-slug extractors replace page links + retract removed timeline rows + rename-alias resolution; +5 fw3 derived-sync: timeline --prune-orphans CLI (#4649) and full-walk page link replacement; +13 fw3 A16/A15: sync link hook reads scoped page metadata and reports per-page extraction errors; +1 fw3 managed-lifecycle: extract --stale dispatches managed brains to the coordinator-safe path; +4 fw3 integration: managed brains refuse the direct timeline orphan prune; +26 #5609: managed extract --stale publishes links, canonical timeline and watermark per page in one coordinated transaction; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +4 at the merge; +6 fix wave 3 lane B: a managed brain routes the extract phase file walk through extractManagedStaleLinks instead of the refused raw timeline batch, keeping the extract.links_fs progress phase; +6 #5761: the unmanaged extract --stale sweep marks unresolved-attendance skips attendance-blocked; +2 eval-wave N9-3/N12-7: a page-type-bound pack attended rule defers to canonical attendance on the filesystem path (ownsAttendanceInference); lowered to 2528 #5904: the DB timeline walk moved to commands/extract-timeline-db.ts (coordinated publication on managed brains, honest refusal + non-zero exit); +13 entity mention index: extract --stale runs the mention pass after link work (half the budget reserved while mention-due pages exist), dry-run and JSON report mention counts, and the help documents the pass and `extract mentions --explain` (logic in core/mentions/); cat40 entity wave merge with v0.60.61.0 (2541 measured); +6 fix wave 10 lane L1 #5556: quarantined pages skipped by the file walks, slug lists and DB walks (slug lookup in quarantine.ts quarantinedSlugs); +3 #5556: the abort check stays its own statement in each quarantine-aware scan loop (cycle-abort coverage pin); 2 P5 wanted pages: both DB extract paths record unresolved authored refs with each page's links; +5 P5 line grammar: stated relation types on the filesystem link scan; +3 PGLite stale link drains refresh planner statistics before and during the drain (logic in planner-stats.ts plannerStatsForLinkDrain); +10 fix wave 12 W2.1 S4 (#6228): extract --stale flushes and stamps the pages a batch finished before an aborting page; +12 P5 wanted pages: both DB extract paths record unresolved authored refs with each page's links; +9 Q2 parser gaps: both DB extract paths read the line-grammar settings once, publish through replaceDerivedLinksUnlessSettingsChanged and report pages a mid-run settings change left stale; +25 GBRA-69: extract --source db reads snapshots and publishes derived links BATCH_SIZE pages at a time (readPageSnapshotsBatch in core/page-snapshot-batch.ts, replay in derived-links.ts replaceDerivedLinksBatchOrReplay); +2 GBRA-75 w7: the --source db links walk refreshes planner statistics (logic in planner-stats.ts plannerStatsForLinkDrain); +10 fix wave 13 P2.16 (#6272): both unmanaged DB link drains defer and count a page a concurrent writer moved (skipped_concurrent_write) instead of aborting the run (classifier and skip line in core/derived-links.ts); +3 fix wave 13 P2.18 (S4): extract all --source db stamps the pages both phases completed at one revision, in a finally (stampCompletedIntersection), instead of every links-phase page; +1 fix wave 13 P2.15 (#6273): --from-meetings exits non-zero while managed entity pages are still pending; +7 fix wave 13 PR2: extractFromDB split out of runExtract so runExtract stays under its function-size baseline; Wave B PR 2 (kept at the master 4e29d0904 merge): +3 PR 2 fix lane J1: extract --stale reads a segmented page block by block (extractSegmentedPageLinks, segmentedTimelineText) now that the stale list includes it; integrated at the master 4e29d0904 merge: 2635 measured; +23 fix wave 20 #6954: the fs walks and the slug-scoped extract collect lost link/timeline writes, stamp only the pages whose writes landed and throw extract_write_lost (helpers in core/extract-watermark.ts); fix wave 20 integration: lane 1 (#6808, +11) and lane 2 (#6954, +23) compose at 2677 measured; +16 #7394: --type/--since/--infer-dates on the file walk refuse with the --source db command beside the other --source db refusals, and the help says so; +5 GBRA-161 PR 5 (T-2, L-6): timeline retraction and the mention rebuild sweep are bound to the revision they read (PageRevisionConflictError import, the scanned-revision read and its FOR SHARE check); +2 GBRA-133: ExtractOpts.timeBudgetMs, forwarded to the managed stale-page pass so the cycle bounds a pass with no slug listsrc/commands/jobs.ts 693 ratchet refactor wave 1 split target (W4 jobs, cut line); ceiling rationale: docs/architecture/key-files/commands-4.md entry for src/commands/jobs.ts; help lines: authorize-legacy --select (#5157 T3) and jobs cancel --select (DX-T2); submit --queue-only + PGLite work drain note (agent-first operator wave E5); +2 spend authorization: jobs submit --yes/--max-usd and jobs list --group help lines; +4 fix wave 9: supervisor stop help documents the 45 s wait, --queue and the verified-drain exit codes; +1 #7142: doctor-refresh handler registration (logic in doctor/last-known.ts); +20 #7242 jobs wait: subcommand row, import, usage lines and its --help text with the exit codes (logic in jobs/wait.ts); +6 GBRA-283 B3a: runJobs enters the local scope (withLocalScope; scope store in core/engine-scope-store.ts), the entry wrapper and its import cannot movesrc/commands/serve-http.ts 947 ratchet refactor wave 1 W4 split into serve-http-<area>.ts modules; ceiling rationale: docs/architecture/key-files/commands-5.md entry for src/commands/serve-http.ts; +2 agent operator wave A6: ServeHttpContext.noticeLedger (import + field; logic in core/notice-ledger.ts); +2 agent contract F1/F2: ServeHttpContext.readinessCache (probed readiness for initialize tails + gbrain://capabilities); +5 Lane D facts drain: serve --http starts the drain scheduler and stops it before the engine disconnect (logic in core/facts/drain-scheduler.ts); +2 agent-operator follow-up Item 3: ServeHttpOptions.adoptServer (status-only recovery hands its bound listener to listenOrRefuse; logic in serve-http-status.ts); +5 #6317: managed-sync movement watch start/stop (watch in persistence/sync-movement.ts); +1 #6589: imports loadServeConfigs (operations see the DB-plane config read once at startup; logic in mcp/serve-config.ts); +5 #6858 ask 6: on-commit HEAD watcher start/stop (watcher in core/serve-head-watcher.ts); +4 GBRA-199: serve --http --advertise parse, help line and pass-through (logic in mcp/surface.ts parseAdvertiseFlag); +6 GBRA-283 B3a: runServeHttp enters the local scope (withLocalScope; scope store in core/engine-scope-store.ts), the entry wrapper and its import cannot movesrc/core/ai/gateway.ts 4571 ratchet +5 GBRA-274 B1.3: withoutBudgetTracker export (runs a rollout hook outside the run's budget scope; the budget AsyncLocalStorage is gateway-private, so it cannot move); +2 #6997: ChatResult.fallbackReason (interface field + doc; recorded in ai/chat-fallback.ts); +17 #6607: lazy OpenAI model discovery (discoverBeforeOpenAICall, once per reconfigure, at the chat/expand/embed entry points; it re-resolves through the gateway-internal resolveGatewayModels, so it cannot move); +18 W11 item 3: embedQuery query-embedding cache lookup/store + clear at every model-cache clear site and the test transport seam (cache lives in ai/query-embed-cache.ts); +7 fix wave 5 (#5137): provider-key redactor on every upstream error path and the first embedding 401/403 report (logic in ai/key-redact.ts, ai/key-warnings.ts); +20 System One v1: decide() delegating export + transport seam, RerankInput.onMeta/RerankMeta, TypeSafe reranker delegation to ai/decide/rerank-adapter.ts; +3 #5585: reconfigureGatewayWithEngine records expansion/chat model provenance (ai/gateway-model-sources.ts) + clears it with gateway state; lowered #5585: tracker record sites collapsed onto ai/budget-record.ts (recordOnTracker + failedCallUsage) while adding ledger fields (requested model, purpose, failed, estimated, countsAsCall); +20: schema-rejection fallback in chat() (retry once schemaless on a json_schema 4xx, remember the recipe for the process — mirrors expand()); re-summed after merging master v0.50.0.0 with the wave (structured-output + pricing hunks); +48: #4863 ChatOpts.responseSchema → tolerant json_schema Output on the chat() generateText call for openai-compatible recipes that declare structured outputs (facts extraction on Ollama); +74: #3657 post-sunset rerank short-circuit (effective-model check, once-per-process audit row + stderr, injected-clock seam); grown v0.46.26.0 megawave: config-plane probes + halt telemetry (#3387 #4312); +21 lines (OpenAI Responses API reasoning-item echo: ChatBlock 'reasoning' variant, chat()/toModelMessages() round-trip); wave-k absorbs: OCR routing seam (#4107 class), config-snapshot reader block (#3980); +21: #4107 OCR-model routing (getImageOcrModel); +10: keyless OPENAI_BASE_URL embedding override (#4385); wave-k #3622 reimpl: GBRAIN_EMBED_MAX_BATCH_TOKENS env cap for no_batch_cap recipes; +9: DeepSeek judge thinking pin plumbed through chat opts (#4069); +master v0.48 merge true-up (reranker succession: no_key preflight, reranker_skipped stage, readiness); ranker wave v0.48.4.0: ChatOpts.temperature threaded to the transport (+15); +24: #4172 recipe-declared thinking headroom on the chat()/toolLoop() default — shared isThinkingModel predicate (Claude-5 regex || recipe thinking_by_default, fail-closed), also consumed by the subagent handler; +20: #4861 claude-cli expansion routed to the schemaless viaText path (the CLI adapter ignores responseFormat, so the native generateObject branch had no fallback); retired provider integration removed; safety wave: migration reranker input ceilings; lowered by fix wave 3 #5680 (embed request planning peeled to ai/embed-batch-plan.ts); +5 #4616: empty inputs skip the provider and every text/multimodal result is screened for zero-norm/NaN/Inf vectors (logic lives in ai/embedding-guard.ts); fix wave 3 integration: #4616 empty-input filtering feeds the shared #5680 batch planner; +3 #5691: embedQuery takes a per-request queryPrefix (no gateway state); +26 fix wave 7 (#5642): embed spend records bank provider-reported usage.tokens (all sub-batches reported → measured; chars-per-token estimate stays the fallback and the failure charge); +2 #5892: import + load-time call of installAiSdkWarningWriter (writer lives in ai/sdk-warnings.ts); +12 #5490: chat() delegation to ai/chat-fallback.ts when chat_fallback_chain is set (attempt loop lives there), ChatOpts.allowFallback + ToolLoopOpts.allowFallback fields and the toolLoop forward; +2 #5490 review: ChatResult.fallbackFrom (the call model a chat_fallback_chain answer replaced); +1 fix wave lane B3: configureGateway keeps chat_fallback_on_refusal (the refusal switch lives in ai/chat-fallback.ts); +18 fix wave 9 lane B: ChatOpts.thinking (#5331, per-route mapping lives in ai/thinking-off.ts), ChatOpts.timeoutMs + ToolLoopOpts.turnTimeoutMs and the turn forward (#4921); interface fields and their docs cannot move; +2 P8: GBRAIN_AI_CALL_LOG observer install in configureGateway (logic in ai/call-log.ts); +2 W9F item 5: thinkingOffOutputCap export shared by chat() and the judge preflight estimates (capability table lives in ai/thinking-off.ts); +3 A1 (wave 0): ChatResult.usage.reasoning_tokens (interface field + doc) and the chat() reasoning-subset read beside the cache counts; the ai-sdk importer allowlist is shrink-only, so the seam stays here); +9 GOOGLE_GENERATIVE_AI_BASE_URL: resolveNativeBaseUrl('google') + the three native-google construction sites (eval metering proxy); re-summed at the master 6ccf30c3e merge (4417 integration + 4423 master - 4408 base); lowered -5: the expand() prompt template peeled to ai/expansion-prompt.ts (buildExpansionPrompt) so the longmemeval call planner measures the same text; +5 #7176: expand() writes its chat_usage_log row (phase expansion) beside the budget record, and the guardedExpansion instance that names its invocation gateway.expand (ledger in ai/chat-usage.ts); +2 PR 4 (Wave B incremental derivation, transcripts) 4b round 4: chat() forwards a metered page step's providerRetries (BudgetTracker) as the SDK's maxRetries, so every attempt is one it reserved for; +24 GBRA-283 B2 (#8456): per-engine gateway state, the gw() accessor every config and model-cache read goes through (an embedded engine state or the process state), configure split into gatewayConfigOf, and configFor at the provider-resolution sites so a provider the engine lacks refuses before any call (state and refusal live in ai/engine-gateway.ts); +1 GBRA-283 B2: embedding diagnosis refuses a provider the embedded engine was not given; +1 GBRA-283 B2 (#8499 item 1): requireConfig refuses an unscoped call in a process holding embedded engines (engine_scope_missing; predicate in engine-scope.ts); +7 GBRA-283 B3a: every embed/chat/expand/rerank/toolLoop entry and requireConfig/configFor call refuseUnscoped (predicate and refusal in ai/engine-gateway.ts), the unscoped state, and the shrink/no_key/structured-output memory read through gw() (state in ai/engine-gateway.ts GatewayState)diff --git a/src/commands/extract.ts b/src/commands/extract.tsindex e4bead56d4..9af44eaa8f 100644--- a/src/commands/extract.ts+++ b/src/commands/extract.ts@@ -74,7 +74,7 @@ import { PageRegexBudget } from '../core/schema-pack/redos-guard.ts';export { extractTimelineFromContent, type ExtractedTimelineEntry } from '../core/timeline-extract.ts';import { extractTimelineFromContent, pruneTimelineOrphans, retractRemovedTimelineEntries, segmentedTimelineText, type ExtractedTimelineEntry } from '../core/timeline-extract.ts';import { managedPersistenceEnabled } from '../core/persistence/ownership.ts';-import { addProjectedTimelineBatch, addProjectedTimelineEntry } from '../core/trust/timeline-projection.ts';+import { addProjectedPageTimeline, addProjectedTimelineBatch } from '../core/trust/timeline-projection.ts';import { createProgress } from '../core/progress.ts';import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';import { pathToSlug, slugifyPath, slugifySegment, readMetafilePagesMode, DEFAULT_METAFILE_PAGES_MODE, type MetafilePagesMode } from '../core/sync.ts';@@ -1826,17 +1826,10 @@ export async function extractTimelineForSlugs(const content = readFileSync(filePath, 'utf-8');const entries = extractTimelineFromContent(content, slug);await retractRemovedTimelineEntries(engine, slug, sourceId, content);- // #6805: one projected batch per page, whatever its timeline length. A batch the database refuses (one bad row- // fails the statement) is retried row by row, so the page keeps its other rows and still reports the refusal.- try { created += await addProjectedTimelineBatch(engine, entries.map(entry => ({ ...entry, source_id: sourceId })), { auditSite: 'extract.timeline_inc' }); }- catch {- let refusal: unknown;- for (const entry of entries) {- try { if (await addProjectedTimelineEntry(engine, entry.slug, { date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail }, { sourceId })) created++; }- catch (e) { refusal ??= e; }- }- if (refusal) throw refusal;- }+ // #6805: one projected batch per page, whatever its timeline length; a missing page fails it like addTimelineEntry.+ const page = await addProjectedPageTimeline(engine, sourceId, entries, { auditSite: 'extract.timeline_inc' });+ created += page.created;+ if (page.refusal !== undefined) throw page.refusal;processed.push(slug);} catch (e) { errors.push({ slug, error: e instanceof Error ? e.message : String(e) }); }}diff --git a/src/core/trust/timeline-projection.ts b/src/core/trust/timeline-projection.tsindex 304de3679b..0ecf4826d8 100644--- a/src/core/trust/timeline-projection.ts+++ b/src/core/trust/timeline-projection.ts@@ -14,6 +14,7 @@ import { derivedWriteTrust } from './taint.ts';import { storedTrustTier, type TaintInput, type TrustTier } from './tier.ts';import { derivedGateConfig, derivedGateInput, recordTimelineFlag, timelineRowAllowed } from './derived-gate.ts';import { assessTimelineForGate } from '../write-gate.ts';+import { PageMissingError } from '../engine-errors.ts';const CHANNEL = 'derive:timeline';@@ -25,9 +26,15 @@ async function pageInputs(engine: BrainEngine, refs: Array<{ source_id: string;return new Map(rows.map(row => [`${row.source_id}:${row.slug}`, { table: 'pages', id: Number(row.id), tier: storedTrustTier(row.trust_tier) }]));}-/** addTimelineEntriesBatch, stamped per page tier. Returns the rows inserted. */-export async function addProjectedTimelineBatch(engine: BrainEngine, rows: TimelineBatchInput[], opts?: BatchOpts): Promise<number> {+/**+ * addTimelineEntriesBatch, stamped per page tier. Returns the rows inserted. The batch drops a row whose page is absent+ * or soft-deleted; `requirePages` instead refuses it with addTimelineEntry's PageMissingError, holding each live page+ * FOR SHARE through the insert so a concurrent delete cannot slip between the check and the batch.+ */+export async function addProjectedTimelineBatch(engine: BrainEngine, rows: TimelineBatchInput[],+ opts?: BatchOpts & { requirePages?: boolean }): Promise<number> {if (!rows.length) return 0;+ const { requirePages, ...batchOpts } = opts ?? {};const key = (row: TimelineBatchInput) => `${row.source_id ?? 'default'}:${row.slug}`;const refs = [...new Map(rows.map(row => [key(row), { source_id: row.source_id ?? 'default', slug: row.slug }])).values()];const inputs = await pageInputs(engine, refs);@@ -47,7 +54,15 @@ export async function addProjectedTimelineBatch(engine: BrainEngine, rows: Timelconst kept = group.rows.map(row => ({ row, assessment: assessTimelineForGate(row, derivedGateInput(trust), cfg) })).filter(r => timelineRowAllowed(r.assessment));if (!kept.length) continue;inserted += await maintenanceTransaction(engine, async tx => {- const count = await tx.addTimelineEntriesBatch(kept.map(r => r.row), opts);+ if (requirePages) {+ const pages = [...new Map(kept.map(({ row }) => [key(row), { source_id: row.source_id ?? 'default', slug: row.slug }])).values()];+ const live = new Set((await tx.executeRaw<{ source_id: string; slug: string }>(`SELECT p.source_id,p.slug FROM pages p+ JOIN jsonb_to_recordset($1::text::jsonb) AS r(source_id text,slug text) ON p.source_id=r.source_id AND p.slug=r.slug+ WHERE p.deleted_at IS NULL FOR SHARE OF p`, [JSON.stringify(pages)])).map(page => `${page.source_id}:${page.slug}`));+ const missing = pages.find(page => !live.has(`${page.source_id}:${page.slug}`));+ if (missing) throw new PageMissingError('addTimelineEntry', 'page', missing.slug, missing.source_id);+ }+ const count = await tx.addTimelineEntriesBatch(kept.map(r => r.row), batchOpts);for (const r of kept) await recordTimelineFlag(tx, r.assessment, r.row);return count;}, trust);@@ -68,3 +83,22 @@ export async function addProjectedTimelineEntry(engine: BrainEngine, slug: strinreturn added;}, trust);}++/**+ * #6805: one page's own timeline as one projected batch that refuses a missing page as addTimelineEntry does. A batch the+ * database refuses otherwise (one bad row fails the statement) is retried row by row, so the page keeps its other rows;+ * `refusal` is the first row's error, and `created` counts the rows inserted either way.+ */+export async function addProjectedPageTimeline(engine: BrainEngine, sourceId: string, entries: Array<TimelineInput & { slug: string }>,+ opts?: BatchOpts): Promise<{ created: number; refusal?: unknown }> {+ try { return { created: await addProjectedTimelineBatch(engine, entries.map(entry => ({ ...entry, source_id: sourceId })), { ...opts, requirePages: true }) }; }+ catch (error) {+ if (error instanceof PageMissingError) throw error;+ let created = 0, refusal: unknown;+ for (const { slug, date, source, summary, detail } of entries) {+ try { if (await addProjectedTimelineEntry(engine, slug, { date, source, summary, detail }, { sourceId })) created++; }+ catch (e) { refusal ??= e; }+ }+ return refusal === undefined ? { created } : { created, refusal };+ }+}diff --git a/test/extract-timeline-for-slugs-batch.test.ts b/test/extract-timeline-for-slugs-batch.test.tsindex 216a67563d..4f937a23ea 100644--- a/test/extract-timeline-for-slugs-batch.test.ts+++ b/test/extract-timeline-for-slugs-batch.test.ts@@ -105,3 +105,19 @@ test('a row the database refuses fails the page without costing it its other rowexpect(await summaries('notes/refused')).toEqual(['Before the refused row', 'After the refused row']);expect(result.created).toBe(2);});++test('a page missing from the database fails the slug as the per-row path did, whether never imported or soft-deleted', async () => {+ mkdirSync(join(repo, 'notes'), { recursive: true });+ writeFileSync(join(repo, 'notes/absent.md'), `---\ntype: note\ntitle: notes/absent\n---\nA synthetic page.\n\n## Timeline\n\n${bullets(2).join('\n')}\n`);+ const absent = await extract('notes/absent');+ expect(absent.processed).toEqual([]);+ expect(absent.errors).toEqual([{ slug: 'notes/absent', error: 'addTimelineEntry failed: page "notes/absent" (source=default) not found' }]);+ expect(absent.created).toBe(0);++ await page('notes/deleted', bullets(2));+ expect(await engine.softDeletePage('notes/deleted', { sourceId: 'default' })).not.toBeNull();+ const deleted = await extract('notes/deleted');+ expect(deleted.processed).toEqual([]);+ expect(deleted.errors).toEqual([{ slug: 'notes/deleted', error: 'addTimelineEntry failed: page "notes/deleted" (source=default) not found' }]);+ expect(await summaries('notes/deleted')).toEqual([]);+});