Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(conversation_facts_backfill): report managed pages still publishing instead of a bare 0 facts inserted (#7500)

garrytan4fe93d626d03
diff --git a/docs/architecture/key-files/commands-3.md b/docs/architecture/key-files/commands-3.md
index 918f4f6407..fe499f20c6 100644
--- a/docs/architecture/key-files/commands-3.md
+++ b/docs/architecture/key-files/commands-3.md
@@ -29,6 +29,6 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/commands/export.ts` + `src/core/export-snapshot.ts` + `src/core/export-stage.ts` + `src/core/export-publish.ts` — local Markdown export collects source/page/canonical-withdrawal/tag/raw state in one export-scoped repeatable-read read-only transaction, using 256-key ID batches and private disk-backed SQLite staging. Primary-key page reads reuse the canonical withdrawal overlay without source-wide slug scans under stale planner statistics. Each 256-key batch groups by source and retains only one capacity-checked withdrawal ledger at a time; bodies with no literal `gbrain:facts:` marker skip normalization, while malformed/orphan markers still use canonical SQL fingerprinting. No engine defaults or shared migration iterators change. Explicit source selection is validated; restore-only selects one source through `resolveRestoreTarget` (`src/core/restore-target.ts`, shared with `gbrain storage status`) and checks its recorded file path through the existing native path resolver. The manifest rejects unsafe paths, same/cross-source aliases, case/Unicode and file-prefix collisions before destination output, preserving the `<slug>.md` and `.raw/<name>.json` layout. Existing planned paths refuse rather than overwrite. The portable first-party native addon anchors no-follow directories and publishes leaves with atomic no-replace operations, retaining an exclusive persistent `.gbrain-export-status` marker; success requires flushed data and a final COMPLETE line. Unsupported native capabilities fail closed without a pathname fallback. A fresh destination is the retry path; unrelated files are never pruned. Fixed capacity/time limits fail visibly, never truncate. This is a historical snapshot, not a database backup or a sandbox against another same-user process renaming directories. See [safe export](../../storage-tiering.md#safe-export). Engine-free `export --help` documents scope, no-overwrite publication and incomplete-output recovery. Wave B PR 2: a segmented page streams (`segmentedExportMarkdown`: serializeMarkdown's head, the trusted stored body through `readBodyRange` under a body ledger, here `exportBodyLedger`, the same withdrawal overlay; `restore_page` of a page over the single-string ceiling passes its snapshot's ledger; the timeline separator before the first timeline byte) into its own stage file (`ExportStage.addStream`, a full collection per 1 MiB read), published with the native path API; its payload estimate is `body_bytes`, the per-page limit is `exportPayloadLimit(pages.max_bytes)` (32 MiB while segmented storage is off), a streamed page adds 1 ms per KiB to the 10-minute snapshot deadline and stage files must leave 512 MiB free on the staging filesystem (`test/e2e/large-pages-export.test.ts`: inline vs segmented same bytes at 2 and 20 MiB, export(import(F)) == F, a gated 128 MiB memory arm). `exportPagePath` writes a page whose slug is not a portable file name under an escaped name with its `slug:` line (`:` and other Windows-unsafe characters become `~`, a reserved device name or trailing dot/space gets `~` appended, an over-budget segment is cut with a `~<8 hex>` hash and marked not restorable); `runExport` returns an `ExportReport`, prints the escaped pages, and collects every `ExportPathCollision` into one refusal whose lines name each fix (`test/pathological/b5-export-unsafe-slugs.test.ts`). Pinned by both-engine export safety/independent-roundtrip/snapshot/failure-release suites, the 100001-page scale fixture with staging bytes and full export SQL statement accounting, the native publication/compiled-wrapper suite, actual CLI help tests, and the PostgreSQL export E2E wrapper. A page's file is rendered by `export-snapshot.ts` (`inlineExportMarkdown`, `segmentedExportHead` for `segmentedExportMarkdown`; frontmatter with the #5966 slug stamp), which the git-gap equality contract (`persistence/git-gap.ts`) shares.
-- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into bounded windows via `src/core/facts/conversation-windows.ts` (#6267: eligibility per 30-min gap group of ≥2 messages decided before windowing; windows close at 30 messages or when the rendered text with its header would exceed 6,500 chars; oversized turns split at sentence boundaries with the speaker prefix repeated; a trailing single reply of an eligible group is its own window, never glued across a gap; nothing is clipped and an unrepresentable header fails the page before prior-fact cleanup; a page's frontmatter `conversation_segment_gap_minutes`, an integer from 1 to 10080, overrides the gap for that page, and any other value is ignored with a stderr warning naming the accepted range), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES` = `CONVERSATION_BODY_MAX_BYTES` in `conversation-parser/body.ts`, `PAGE_MAX_BYTES` + 1 MiB = 25 MiB, measured on the parser input; `readConversationBodyForParsing` stats a `raw_transcript` sidecar and throws `ConversationBodyTooLargeError` over the cap before any read, the snapshot carries `oversizedSidecarBytes`, and a size skip writes a `skipped_too_large` receipt to `page_extraction_state` (doctor `extraction_input_truncated`) that a later run under the cap clears; test `test/conversation-facts-oversized-sidecar.test.ts`; a segmented page is read in ranges; one over `CONVERSATION_WINDOW_BYTES` with no conversation facts is extracted from its newest window (day-header date, `tail_window` receipt, `window_rule: no_prior_facts`); one with facts is skipped with nothing cleared (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears prior facts the window may not see, until PR 4's window anchors (test `test/large-pages/segmented-conversation-facts.test.ts`)); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd` (a DEFAULT cap is dropped with a warning when the extraction or parser chat model is unpriced, via `src/core/facts/conversation-budget.ts` `conversationFactsCostCap` (#5823); explicit CLI/caller/cycle caps stay fail-closed and a `no_pricing` stop reports `budget_reason`/`budget_model`, exits 1 and names `pricing.overrides`); **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); **`--background` via `maybeBackground`**; and **speaker-shaped-fold decline** — when the parse reports `unrecognized_headings` containing a speaker-shaped label (1-2 title-cased words, not in the doc-heading stoplist — the stoplist gates the DECLINE only, so a miss is warn-noise, never data loss) AND the parse produced fewer than two distinct speakers, the page is declined instead of extracted under wrong attribution: counted in `pages_skipped_unrecognized_speaker` (Result/CLI/cycle surfaces), warned to stderr with the folding pattern id, and deliberately NON-terminal (no durable audit row and no orphan cleanup, so a future parser/pattern fix retries the page); multi-speaker pages with folds proceed warn-only. `publishConversationFacts` skips a non-outcome row an active conversation fact of the source already states (`findCaptureDuplicate`, no time bound; `duplicates_skipped`, #7043). The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source except `cycle.conversation_facts_backfill.exclude_sources` (reported as `sources_excluded`), and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts:<source>:<slug>`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). A page's rows are buffered and published once after its last segment (`publishPageFacts` on an unmanaged brain, one `writeDerivedFacts` transaction; the managed batch otherwise), through `publishConversationFacts` (facts/conversation-publication.ts), so a crashed, failed, empty or partial run never removes the prior batch and a terminal audit row commits only with its page's facts (#7013); a run's `scope` is `partial` when fewer segments than in scope ran, `window` when `--since` dropped messages, else `full`, and a page that changed mid-run keeps an unmanaged run's facts as a `partial` batch. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Additive `pages_skipped_unparsed`, `pages_skipped_type_mismatch`, `pages_skipped_insufficient_turns`, and `pages_skipped_since` explain the unchanged `pages_skipped` aggregate across core, CLI, workers, and cycle sources; durable completion remains separate. `--dry-run` stops after segmentation, with no parser-fallback, extractor, or embedding calls and no durable knowledge/checkpoint/receipt/rollup writes. The narrow `python-dict-utterance` registry format and its quoted/nested-example rejection are documented in [conversation parser patterns](../conversation-parser-patterns.md). Checkpoint state is a shared `cpMap: Map<slug, endIso>` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-diagnostics.serial.test.ts` (pool workers, per-page lock skip + exit 3, replay replacing a prior partial run's facts, D15 preflight, `--workers` round-trip). `src/commands/doctor.ts` reads the same outcomes, which survive checkpoint GC. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Explicit `slugs` batches contain page failures like the pool (cancellation and budget stops halt); logs carry only the closed reason and a retry command. Completion requires every in-scope segment (exactly `--segment-limit` counts). Extracted rows take `facts.default_visibility` (resolved once per run); audit rows stay private. Each page's own entities go to the extractor as `entityHints` (slugs, `src/core/facts/conversation-entity-hints.ts`, computed once per page, cap `CONVERSATION_ENTITY_HINTS_CAP` = 40 through `ExtractInput.entityHintsCap`): authored edges only (the page's frontmatter edges with the page as origin, then outgoing markdown, wikilink-resolved and manual links), same source, live fact-entity pages, and a private entity only when the run's facts are private; the `extract_facts` op keeps its 5-hint cap and prompt. Pinned by `test/conversation-entity-hints.test.ts`. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed. `no_match`, errors, cancellation, and dry runs never become durable negatives, bar `facts/conversation-skip.ts` (prose meetings, single emails, undated turns). Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. On a managed brain the core runs `managedDerivedFactsPreflight` (local writer, active source) before any provider work, and a page's facts and terminal (or non-extractable) audit row are buffered, then published in its receipted batch after source activity, page id, revision and parser-input snapshot are rechecked under the page lock, so a failed or raced re-extraction keeps the prior batch; the CLI no longer refuses up front and receipt pages are skipped. Tests: `test/managed-facts-writers.test.ts`, `test/managed-unsupported-preflight.serial.test.ts` (refusal before spend for an unaccepted writer).
+- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into bounded windows via `src/core/facts/conversation-windows.ts` (#6267: eligibility per 30-min gap group of ≥2 messages decided before windowing; windows close at 30 messages or when the rendered text with its header would exceed 6,500 chars; oversized turns split at sentence boundaries with the speaker prefix repeated; a trailing single reply of an eligible group is its own window, never glued across a gap; nothing is clipped and an unrepresentable header fails the page before prior-fact cleanup; a page's frontmatter `conversation_segment_gap_minutes`, an integer from 1 to 10080, overrides the gap for that page, and any other value is ignored with a stderr warning naming the accepted range), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES` = `CONVERSATION_BODY_MAX_BYTES` in `conversation-parser/body.ts`, `PAGE_MAX_BYTES` + 1 MiB = 25 MiB, measured on the parser input; `readConversationBodyForParsing` stats a `raw_transcript` sidecar and throws `ConversationBodyTooLargeError` over the cap before any read, the snapshot carries `oversizedSidecarBytes`, and a size skip writes a `skipped_too_large` receipt to `page_extraction_state` (doctor `extraction_input_truncated`) that a later run under the cap clears; test `test/conversation-facts-oversized-sidecar.test.ts`; a segmented page is read in ranges; one over `CONVERSATION_WINDOW_BYTES` with no conversation facts is extracted from its newest window (day-header date, `tail_window` receipt, `window_rule: no_prior_facts`); one with facts is skipped with nothing cleared (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears prior facts the window may not see, until PR 4's window anchors (test `test/large-pages/segmented-conversation-facts.test.ts`)); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd` (a DEFAULT cap is dropped with a warning when the extraction or parser chat model is unpriced, via `src/core/facts/conversation-budget.ts` `conversationFactsCostCap` (#5823); explicit CLI/caller/cycle caps stay fail-closed and a `no_pricing` stop reports `budget_reason`/`budget_model`, exits 1 and names `pricing.overrides`); **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); **`--background` via `maybeBackground`**; and **speaker-shaped-fold decline** — when the parse reports `unrecognized_headings` containing a speaker-shaped label (1-2 title-cased words, not in the doc-heading stoplist — the stoplist gates the DECLINE only, so a miss is warn-noise, never data loss) AND the parse produced fewer than two distinct speakers, the page is declined instead of extracted under wrong attribution: counted in `pages_skipped_unrecognized_speaker` (Result/CLI/cycle surfaces), warned to stderr with the folding pattern id, and deliberately NON-terminal (no durable audit row and no orphan cleanup, so a future parser/pattern fix retries the page); multi-speaker pages with folds proceed warn-only. `publishConversationFacts` skips a non-outcome row an active conversation fact of the source already states (`findCaptureDuplicate`, no time bound; `duplicates_skipped`, #7043). The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source except `cycle.conversation_facts_backfill.exclude_sources` (reported as `sources_excluded`), reports managed pages left accepted and still publishing as `pages_pending` in its details and summary (`test/cycle-conversation-facts-backfill-pending-7500.test.ts`), and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts:<source>:<slug>`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). A page's rows are buffered and published once after its last segment (`publishPageFacts` on an unmanaged brain, one `writeDerivedFacts` transaction; the managed batch otherwise), through `publishConversationFacts` (facts/conversation-publication.ts), so a crashed, failed, empty or partial run never removes the prior batch and a terminal audit row commits only with its page's facts (#7013); a run's `scope` is `partial` when fewer segments than in scope ran, `window` when `--since` dropped messages, else `full`, and a page that changed mid-run keeps an unmanaged run's facts as a `partial` batch. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Additive `pages_skipped_unparsed`, `pages_skipped_type_mismatch`, `pages_skipped_insufficient_turns`, and `pages_skipped_since` explain the unchanged `pages_skipped` aggregate across core, CLI, workers, and cycle sources; durable completion remains separate. `--dry-run` stops after segmentation, with no parser-fallback, extractor, or embedding calls and no durable knowledge/checkpoint/receipt/rollup writes. The narrow `python-dict-utterance` registry format and its quoted/nested-example rejection are documented in [conversation parser patterns](../conversation-parser-patterns.md). Checkpoint state is a shared `cpMap: Map<slug, endIso>` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-diagnostics.serial.test.ts` (pool workers, per-page lock skip + exit 3, replay replacing a prior partial run's facts, D15 preflight, `--workers` round-trip). `src/commands/doctor.ts` reads the same outcomes, which survive checkpoint GC. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Explicit `slugs` batches contain page failures like the pool (cancellation and budget stops halt); logs carry only the closed reason and a retry command. Completion requires every in-scope segment (exactly `--segment-limit` counts). Extracted rows take `facts.default_visibility` (resolved once per run); audit rows stay private. Each page's own entities go to the extractor as `entityHints` (slugs, `src/core/facts/conversation-entity-hints.ts`, computed once per page, cap `CONVERSATION_ENTITY_HINTS_CAP` = 40 through `ExtractInput.entityHintsCap`): authored edges only (the page's frontmatter edges with the page as origin, then outgoing markdown, wikilink-resolved and manual links), same source, live fact-entity pages, and a private entity only when the run's facts are private; the `extract_facts` op keeps its 5-hint cap and prompt. Pinned by `test/conversation-entity-hints.test.ts`. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed. `no_match`, errors, cancellation, and dry runs never become durable negatives, bar `facts/conversation-skip.ts` (prose meetings, single emails, undated turns). Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. On a managed brain the core runs `managedDerivedFactsPreflight` (local writer, active source) before any provider work, and a page's facts and terminal (or non-extractable) audit row are buffered, then published in its receipted batch after source activity, page id, revision and parser-input snapshot are rechecked under the page lock, so a failed or raced re-extraction keeps the prior batch; the CLI no longer refuses up front and receipt pages are skipped. Tests: `test/managed-facts-writers.test.ts`, `test/managed-unsupported-preflight.serial.test.ts` (refusal before spend for an unaccepted writer).
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. A lost link or timeline write on the fs walks or the slug-scoped path leaves its pages unstamped and throws `extract_write_lost` after stamping the rest (`core/extract-watermark.ts`); the sync hooks report a page whose link or timeline write failed in `errors`, never `processed`, so the caller leaves it unstamped. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). The full fs walk replaces each page's own markdown-derived links through `replacePageFileLinks` (the per-slug contract below; a file with no page row is skipped and not stamped); `extract --stale` replaces through `replaceDerivedLinks`, the db link walk through `replaceDerivedLinksBatch`. Timeline rows are inserted in batches of 100 via `addTimelineEntriesBatch` (`ON CONFLICT DO NOTHING`; `created` counts real inserts) after every path — fs walk, db walk, per-slug, sync hooks, `--stale` — runs `retractRemovedTimelineEntries` for the page. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. Every per-slug extractor — `extractForSlugs` (the cycle's incremental pass) and the two sync hooks `extractLinksForSlugs` / `extractTimelineForSlugs` — resolves each requested slug through `resolveSlugRelPath`: the `buildSlugPathIndex(walkMarkdownFiles(dir))` index first, then — only on an index miss — the legacy `join(dir, slug + '.md')` existence check. The index is the primary because `pathToSlug` lowercases and slugifies, so a reconstructed path only finds files whose names are already slugs; the legacy fallback stays because sync ADMITS files the walker never emits (`_`-prefixed names, dot-dirs waived via `sync.include_hidden`) and those slugs still round-trip to their real path — without it such a page imports with no edges. On a slug collision the index prefers the file whose name IS the slug (else the first walked entry), so resolution never depends on read order. The per-slug extractors reconcile like `put_page`: each page's own markdown-derived links are replaced through `replacePageFileLinks` (`src/core/link-reconciliation.ts`, `replaceDerivedLinks` with `preserveExisting`; other producers survive, a page with no row counts as processed with no links, and a failed replacement is reported on stderr and leaves the page stale), and `retractRemovedTimelineEntries` (`src/core/timeline-extract.ts`) deletes the non-event rows whose (date, source, summary) any stored `page_versions` text of the page produced (under either timeline parser) and its current text no longer does; rows no version produced (enrichment, meeting fan-out, `--infer-dates` anchors) are kept, and the version scan only runs when the page holds a row the current text does not produce. `gbrain extract timeline --prune-orphans [--source-id] [--dry-run] [--json]` (`pruneTimelineOrphans`) runs that reconciliation over every page with timeline rows, without inserting — the one-time prune for rows orphaned while extraction was insert-only (managed brains retract each page's rows in a page-locked `withCoordinatedWrite` transaction; these rows are removals, so `gbrain repair timeline` / `timeline_history` exclude them; pinned in `test/persistence-managed-lifecycle.test.ts`); the read-only `timeline_orphans` doctor check (`src/commands/doctor/checks/timeline-orphans.ts`) samples 500 pages and points at it. Pinned by `test/timeline-reconcile-all-paths.test.ts`. `loadSlugAliasTargets` adds renamed pages' old slugs (`slug_aliases`) to `allSlugs` so links written before a rename resolve to the current slug. `extractLinksForSlugs` reads the requested files first, then loads page metadata only for those pages and their link endpoints (`fileLinkOwnership(engine, sourceId, { slugs })`); the unscoped whole-brain load happens only when a DB meeting or attendance origin needs `replaceFileLinks`. Both return `ExtractForSlugsResult { created, processed, errors? }`; `errors` names pages whose read or write failed, and sync reports them as `extract_error` (stderr line plus the `SyncResult` field) instead of swallowing them. `processed` is the subset actually read, and the shared `slugsSafeToStamp(links, timeline)` intersection is what every caller may stamp. Stamping the full requested set marks a silently-skipped page fresh and hides it from `extract --stale` permanently. All three call sites (`src/commands/sync.ts`, `src/core/github-source.ts`, `src/core/google/google-source.ts`) stamp the intersection. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id <id>` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id <id>] [--catch-up] [--retry-held] [--dry-run] [--json]` branch (`extractStaleFromDB`; on a managed brain it hands the sweep to `extractManagedStaleLinks` in `src/core/persistence/links-maintenance.ts` for every caller, CLI, cycle, jobs and maintain, and a live PGLite owner runs it through `src/commands/extract-stale-delegate.ts`; failing pages are reported and skipped there, and `--retry-held` clears held ones first) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale, then the mention pass. Embedded callers (the cycle) pass `quiet: true` so the helper writes nothing to stdout, including its JSON result; standalone CLI output is unchanged. In `jsonMode` every fs path, `extractForSlugs` included, reports a dropped flush as a `{event:'batch_error'}` stderr line rather than going silent. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush (the unmanaged sweep): a page's throw aborts the sweep after the batch's finished pages are flushed and stamped (S4), so the failing page and later ones stay stale and re-extract idempotently. `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. On a managed-persistence brain each page publishes its derived links (`prepareAutomaticLinks`), the coordinator's unrecorded canonical timeline tuples (`unrecordedCanonicalTimeline`, insert-only) and its watermark in one revision-bound `withCoordinatedWrite` transaction; a page edited since the read is skipped and stays stale. Tests: `test/extract-stale-managed.test.ts`, `test/persistence-managed-lifecycle.test.ts`. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Fresh-install schema in `src/schema.sql` (both generated blobs follow via `bun run build:schema`) + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor/checks/extraction-sync.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; pages whose attendance marker (`links_attendance_blocked_revision`, migration v180) equals their current `knowledge_revision` are left out of the lag count and reported as `details.attendance_blocked` (both `extract --stale` sweeps set the marker on an unresolved-attendance skip; pre-v180 brains fall back to the unfiltered count); strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` carries `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` at the inline-extract call site for the slugs the extractors actually read, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` lists `'extract.stale'` in `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` lists `links_extraction_lag` in `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp + crash-contract), `test/sync-inline-extract-stamps.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite case in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. W4.14: one basename index per slug set (rebuilt as it grows). GBRA-133 2b.1: the sync hooks (`extractLinksForSlugs` / `extractTimelineForSlugs`) share one `indexMarkdownTree` walk per run (`src/commands/extract-tree.ts`, with `markdownWalk` behind `walkMarkdownFiles`) (it yields after each directory) and type link endpoints through `withEndpointPageTypes`, which parses only the walked files the pass asks a type for (repeating until nothing new is asked) instead of `loadFsPageTypes` over the whole tree; full `gbrain extract` keeps the whole-tree parse, and the two give the same links (`test/pathological/s-sync-offpath-2b.test.ts`).
diff --git a/src/core/cycle/conversation-facts-backfill.ts b/src/core/cycle/conversation-facts-backfill.ts
index 30b8e64759..ef30898539 100644
--- a/src/core/cycle/conversation-facts-backfill.ts
+++ b/src/core/cycle/conversation-facts-backfill.ts
@@ -380,6 +380,7 @@ export async function runPhaseConversationFactsBackfill(
pages_skipped_unrecognized_speaker: 0,
pages_failed: 0,
facts_inserted: 0,
+ pages_pending: 0,
fallback_slugify_count: 0,
resolution_errors: 0,
sources_processed: 0,
@@ -398,6 +399,7 @@ export async function runPhaseConversationFactsBackfill(
totals.pages_skipped_unrecognized_speaker += r.pages_skipped_unrecognized_speaker;
totals.pages_failed += r.pages_failed;
totals.facts_inserted += r.facts_inserted;
+ totals.pages_pending += r.pages_pending ?? 0;
totals.fallback_slugify_count += r.fallback_slugify_count;
totals.resolution_errors += r.resolution_errors;
}
@@ -409,6 +411,7 @@ export async function runPhaseConversationFactsBackfill(
const noPricing = [...new Map(Object.values(perSourceResults)
.flatMap((r) => r.budget_pricing ? [[r.budget_pricing.model, r.budget_pricing] as const] : [])).values()];
const summary = `${totals.facts_inserted} facts inserted across ${totals.sources_processed}/${sources.length} sources, ~$${totalSpent.toFixed(4)} spent` +
+ (totals.pages_pending ? `. ${totals.pages_pending} page(s) accepted and still publishing; their facts commit after this run and are not counted here` : '') +
noPricing.map((g) => `. ${noPricingMessage(g)}`).join('') + notOwnedNote(sourcesNotOwned)
+ (excluded.length ? `. Skipped ${excluded.length} source(s) listed in ${CFG_PREFIX}.exclude_sources (${excluded.join(', ')})` : '');
@@ -432,6 +435,7 @@ export async function runPhaseConversationFactsBackfill(
pages_skipped_unrecognized_speaker: totals.pages_skipped_unrecognized_speaker,
pages_failed: totals.pages_failed,
facts_inserted: totals.facts_inserted,
+ pages_pending: totals.pages_pending,
fallback_slugify_count: totals.fallback_slugify_count,
resolution_errors: totals.resolution_errors,
spent_usd: totalSpent,
diff --git a/test/cycle-conversation-facts-backfill-pending-7500.test.ts b/test/cycle-conversation-facts-backfill-pending-7500.test.ts
new file mode 100644
index 0000000000..9be7765978
--- /dev/null
+++ b/test/cycle-conversation-facts-backfill-pending-7500.test.ts
@@ -0,0 +1,66 @@
+/**
+ * #7500: conversation_facts_backfill reports the pages a managed run left
+ * accepted and still publishing.
+ *
+ * Protects: when a managed batch is still pending after the job's write wait,
+ * the phase summary and `details.pages_pending` say so, instead of a bare
+ * "0 facts inserted" while the accepted batch commits its facts afterwards.
+ * Fails when: the phase aggregate drops the per-source `pages_pending`.
+ * Why new: managed-facts-writers covers a batch that commits within the wait only.
+ */
+import { afterAll, afterEach, beforeAll, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { configureGateway, resetGateway, __setChatTransportForTests, __setEmbedTransportForTests, type ChatResult } from '../src/core/ai/gateway.ts';
+import { runPhaseConversationFactsBackfill } from '../src/core/cycle/conversation-facts-backfill.ts';
+import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
+import { installFaultHook } from '../src/core/persistence/fault-points.ts';
+import { __setMaintenanceWriteWaitForTests } from '../src/core/persistence/maintenance-wait.ts';
+import { disposePersistenceConsumer, waitForWrite } from '../src/core/persistence/service.ts';
+import type { WriteRequest } from '../src/core/persistence/model.ts';
+import { managedBrain } from './helpers/managed-brain.ts';
+import { testWaitMs } from './helpers/wait-for.ts';
+
+const SLUG = 'conversations/synthetic-chat';
+const CONVERSATION = `---\ntitle: Synthetic chat\ntype: conversation\n---\n${[
+ '**Alice Example** (2024-03-15 9:00 AM): I just signed the offer letter for Acme Corp.',
+ '**Bob Demo** (2024-03-15 9:01 AM): Congrats! What is the title?',
+ '**Alice Example** (2024-03-15 9:02 AM): Staff engineer on the platform team.',
+].join('\n')}\n`;
+const usage = { input_tokens: 10, output_tokens: 10, cache_read_tokens: 0, cache_creation_tokens: 0 };
+
+beforeAll(() => {
+ configureGateway({ chat_model: 'anthropic:claude-sonnet-4-6', embedding_model: 'openai:text-embedding-3-large', embedding_dimensions: 1536,
+ env: { ANTHROPIC_API_KEY: 'sk-ant-test', OPENAI_API_KEY: 'sk-test' } });
+ __setChatTransportForTests(async (): Promise<ChatResult> => ({ text: JSON.stringify({ facts: [{ fact: 'Alice Example joined Acme Corp.', kind: 'event', entity: null, confidence: 1, notability: 'high' }] }),
+ blocks: [], stopReason: 'end', usage, model: 'anthropic:claude-sonnet-4-6', providerId: 'anthropic' }));
+ __setEmbedTransportForTests((async ({ values }: { values: string[] }) => ({ embeddings: values.map(() => Array(1536).fill(0.01)) })) as never);
+});
+afterEach(() => { installFaultHook(undefined); });
+afterAll(() => { __setChatTransportForTests(null); __setEmbedTransportForTests(null); resetGateway(); });
+
+const extractorFacts = async (engine: BrainEngine) => (await engine.executeRaw<{ fact: string }>(
+ "SELECT fact FROM facts WHERE source_id='default' AND source_markdown_slug=$1 AND source LIKE 'cli:extract-conversation-facts%' ORDER BY row_num", [SLUG])).map(r => r.fact);
+
+test('#7500: a managed batch still pending after the wait is reported as pending, then commits its facts', async () => {
+ await managedBrain(async ({ engine, ctx }) => {
+ await engine.setConfig('cycle.conversation_facts_backfill.enabled', 'true');
+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug: SLUG, request_id: randomUUID(), content: CONVERSATION } });
+ await disposePersistenceConsumer(engine);
+
+ let release!: () => void;
+ const stalled = new Promise<void>(resolve => { release = resolve; });
+ installFaultHook(async point => { if (point === 'consumer:prepared') await stalled; });
+ const restore = __setMaintenanceWriteWaitForTests(testWaitMs(300));
+ let phase: Awaited<ReturnType<typeof runPhaseConversationFactsBackfill>>;
+ try {
+ phase = await runPhaseConversationFactsBackfill(engine);
+ } finally { release(); restore(); }
+ expect(phase.details).toMatchObject({ facts_inserted: 0, pages_pending: 1 });
+ expect(phase.summary).toContain('1 page(s) accepted and still publishing; their facts commit after this run and are not counted here');
+
+ const open = await engine.executeRaw<WriteRequest>("SELECT * FROM persistence_requests WHERE source_id='default' AND state NOT IN ('committed','failed','cancelled')");
+ for (const row of open) expect((await waitForWrite(engine, row, { engine: engine.kind }, testWaitMs(20_000))).state).toBe('committed');
+ expect(await extractorFacts(engine)).toEqual(['Alice Example joined Acme Corp.', 'EXTRACTION_COMPLETE']);
+ });
+}, 120_000);