Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(timeline): extract and sync retire rows the old hyphen-split parser left

garrytana6ccb4a32779
diff --git a/docs/architecture/key-files/commands-3.md b/docs/architecture/key-files/commands-3.md
index fd0b6588e5..faa17f97ec 100644
--- a/docs/architecture/key-files/commands-3.md
+++ b/docs/architecture/key-files/commands-3.md
@@ -31,4 +31,4 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into bounded windows via `src/core/facts/conversation-windows.ts` (#6267: eligibility per 30-min gap group of ≥2 messages decided before windowing; windows close at 30 messages or when the rendered text with its header would exceed 6,500 chars; oversized turns split at sentence boundaries with the speaker prefix repeated; a trailing single reply of an eligible group is its own window, never glued across a gap; nothing is clipped and an unrepresentable header fails the page before prior-fact cleanup; a page's frontmatter `conversation_segment_gap_minutes`, an integer from 1 to 10080, overrides the gap for that page, and any other value is ignored with a stderr warning naming the accepted range), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES` = `CONVERSATION_BODY_MAX_BYTES` in `conversation-parser/body.ts`, `PAGE_MAX_BYTES` + 1 MiB = 25 MiB, measured on the parser input; `readConversationBodyForParsing` stats a `raw_transcript` sidecar and throws `ConversationBodyTooLargeError` over the cap before any read, the snapshot carries `oversizedSidecarBytes`, and a size skip writes a `skipped_too_large` receipt to `page_extraction_state` (doctor `extraction_input_truncated`) that a later run under the cap clears; test `test/conversation-facts-oversized-sidecar.test.ts`; a segmented page is read in ranges; one over `CONVERSATION_WINDOW_BYTES` with no conversation facts is extracted from its newest window (day-header date, `tail_window` receipt, `window_rule: no_prior_facts`); one with facts is skipped with nothing cleared (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears prior facts the window may not see, until PR 4's window anchors (test `test/large-pages/segmented-conversation-facts.test.ts`)); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd` (a DEFAULT cap is dropped with a warning when the extraction or parser chat model is unpriced, via `src/core/facts/conversation-budget.ts` `conversationFactsCostCap` (#5823); explicit CLI/caller/cycle caps stay fail-closed and a `no_pricing` stop reports `budget_reason`/`budget_model`, exits 1 and names `pricing.overrides`); **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); **`--background` via `maybeBackground`**; and **speaker-shaped-fold decline** — when the parse reports `unrecognized_headings` containing a speaker-shaped label (1-2 title-cased words, not in the doc-heading stoplist — the stoplist gates the DECLINE only, so a miss is warn-noise, never data loss) AND the parse produced fewer than two distinct speakers, the page is declined instead of extracted under wrong attribution: counted in `pages_skipped_unrecognized_speaker` (Result/CLI/cycle surfaces), warned to stderr with the folding pattern id, and deliberately NON-terminal (no durable audit row and no orphan cleanup, so a future parser/pattern fix retries the page); multi-speaker pages with folds proceed warn-only. `publishConversationFacts` skips a non-outcome row an active conversation fact of the source already states (`findCaptureDuplicate`, no time bound; `duplicates_skipped`, #7043). The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source except `cycle.conversation_facts_backfill.exclude_sources` (reported as `sources_excluded`), and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts:<source>:<slug>`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). A page's rows are buffered and published once after its last segment (`publishPageFacts` on an unmanaged brain, one `writeDerivedFacts` transaction; the managed batch otherwise), through `publishConversationFacts` (facts/conversation-publication.ts), so a crashed, failed, empty or partial run never removes the prior batch and a terminal audit row commits only with its page's facts (#7013); a run's `scope` is `partial` when fewer segments than in scope ran, `window` when `--since` dropped messages, else `full`, and a page that changed mid-run keeps an unmanaged run's facts as a `partial` batch. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Additive `pages_skipped_unparsed`, `pages_skipped_type_mismatch`, `pages_skipped_insufficient_turns`, and `pages_skipped_since` explain the unchanged `pages_skipped` aggregate across core, CLI, workers, and cycle sources; durable completion remains separate. `--dry-run` stops after segmentation, with no parser-fallback, extractor, or embedding calls and no durable knowledge/checkpoint/receipt/rollup writes. The narrow `python-dict-utterance` registry format and its quoted/nested-example rejection are documented in [conversation parser patterns](../conversation-parser-patterns.md). Checkpoint state is a shared `cpMap: Map<slug, endIso>` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-diagnostics.serial.test.ts` (pool workers, per-page lock skip + exit 3, replay replacing a prior partial run's facts, D15 preflight, `--workers` round-trip). `src/commands/doctor.ts` reads the same outcomes, which survive checkpoint GC. Each outcome is bound to the exact parsed snapshot: regular pages use `content_hash` plus the UTC effective date; raw-conversation sidecars and legacy null-hash pages use a canonical SHA-256 over every parser-relevant input. Selection checks the token before locking, refetches under the lock, and verifies it again before writing the outcome, so an edit cannot be certified by stale work. The strict extraction path treats provider, refusal, truncation, malformed/schema-invalid output, segment-write, cleanup, and terminal-write failures as unfinished work; bulk failures increment `pages_failed`, affect CLI/cycle receipts and exit status, and never advance the legacy checkpoint. Explicit `slugs` batches contain page failures like the pool (cancellation and budget stops halt); logs carry only the closed reason and a retry command. Completion requires every in-scope segment (exactly `--segment-limit` counts). Extracted rows take `facts.default_visibility` (resolved once per run); audit rows stay private. Each page's own entities go to the extractor as `entityHints` (slugs, `src/core/facts/conversation-entity-hints.ts`, computed once per page, cap `CONVERSATION_ENTITY_HINTS_CAP` = 40 through `ExtractInput.entityHintsCap`): authored edges only (the page's frontmatter edges with the page as origin, then outgoing markdown, wikilink-resolved and manual links), same source, live fact-entity pages, and a private entity only when the run's facts are private; the `extract_facts` op keeps its 5-hint cap and prompt. Pinned by `test/conversation-entity-hints.test.ts`. Checkpoints are only a scheduling hint: a slug without a matching v2 outcome is replayed. `no_match`, errors, cancellation, and dry runs never become durable negatives, bar `facts/conversation-skip.ts` (prose meetings, single emails, undated turns). Result, CLI, cycle, and doctor surfaces keep completed, scanned-not-extractable, unfinished, failed, and lock-skipped counts separate. See [Conversation backfill durable outcomes](../../operations/conversation-backfill-outcomes.md) for the operator and maintainer contract. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. On a managed brain the core runs `managedDerivedFactsPreflight` (local writer, active source) before any provider work, and a page's facts and terminal (or non-extractable) audit row are buffered, then published in its receipted batch after source activity, page id, revision and parser-input snapshot are rechecked under the page lock, so a failed or raced re-extraction keeps the prior batch; the CLI no longer refuses up front and receipt pages are skipped. Tests: `test/managed-facts-writers.test.ts`, `test/managed-unsupported-preflight.serial.test.ts` (refusal before spend for an unaccepted writer).
-- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. A lost link or timeline write on the fs walks or the slug-scoped path leaves its pages unstamped and throws `extract_write_lost` after stamping the rest (`core/extract-watermark.ts`); the sync hooks report a page whose link or timeline write failed in `errors`, never `processed`, so the caller leaves it unstamped. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). The full fs walk replaces each page's own markdown-derived links through `replacePageFileLinks` (the per-slug contract below; a file with no page row is skipped and not stamped); `extract --stale` replaces through `replaceDerivedLinks`, the db link walk through `replaceDerivedLinksBatch`. Timeline rows are inserted in batches of 100 via `addTimelineEntriesBatch` (`ON CONFLICT DO NOTHING`; `created` counts real inserts) after every path — fs walk, db walk, per-slug, sync hooks, `--stale` — runs `retractRemovedTimelineEntries` for the page. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. Every per-slug extractor — `extractForSlugs` (the cycle's incremental pass) and the two sync hooks `extractLinksForSlugs` / `extractTimelineForSlugs` — resolves each requested slug through `resolveSlugRelPath`: the `buildSlugPathIndex(walkMarkdownFiles(dir))` index first, then — only on an index miss — the legacy `join(dir, slug + '.md')` existence check. The index is the primary because `pathToSlug` lowercases and slugifies, so a reconstructed path only finds files whose names are already slugs; the legacy fallback stays because sync ADMITS files the walker never emits (`_`-prefixed names, dot-dirs waived via `sync.include_hidden`) and those slugs still round-trip to their real path — without it such a page imports with no edges. On a slug collision the index prefers the file whose name IS the slug (else the first walked entry), so resolution never depends on read order. The per-slug extractors reconcile like `put_page`: each page's own markdown-derived links are replaced through `replacePageFileLinks` (`src/core/link-reconciliation.ts`, `replaceDerivedLinks` with `preserveExisting`; other producers survive, a page with no row counts as processed with no links, and a failed replacement is reported on stderr and leaves the page stale), and `retractRemovedTimelineEntries` (`src/core/timeline-extract.ts`) deletes the non-event rows whose (date, source, summary) any stored `page_versions` text of the page produced (under either timeline parser) and its current text no longer does; rows no version produced (enrichment, meeting fan-out, `--infer-dates` anchors) are kept, and the version scan only runs when the page holds a row the current text does not produce. `gbrain extract timeline --prune-orphans [--source-id] [--dry-run] [--json]` (`pruneTimelineOrphans`) runs that reconciliation over every page with timeline rows, without inserting — the one-time prune for rows orphaned while extraction was insert-only (managed brains retract each page's rows in a page-locked `withCoordinatedWrite` transaction; these rows are removals, so `gbrain repair timeline` / `timeline_history` exclude them; pinned in `test/persistence-managed-lifecycle.test.ts`); the read-only `timeline_orphans` doctor check (`src/commands/doctor/checks/timeline-orphans.ts`) samples 500 pages and points at it. Pinned by `test/timeline-reconcile-all-paths.test.ts`. `loadSlugAliasTargets` adds renamed pages' old slugs (`slug_aliases`) to `allSlugs` so links written before a rename resolve to the current slug. `extractLinksForSlugs` reads the requested files first, then loads page metadata only for those pages and their link endpoints (`fileLinkOwnership(engine, sourceId, { slugs })`); the unscoped whole-brain load happens only when a DB meeting or attendance origin needs `replaceFileLinks`. Both return `ExtractForSlugsResult { created, processed, errors? }`; `errors` names pages whose read or write failed, and sync reports them as `extract_error` (stderr line plus the `SyncResult` field) instead of swallowing them. `processed` is the subset actually read, and the shared `slugsSafeToStamp(links, timeline)` intersection is what every caller may stamp. Stamping the full requested set marks a silently-skipped page fresh and hides it from `extract --stale` permanently. All three call sites (`src/commands/sync.ts`, `src/core/github-source.ts`, `src/core/google/google-source.ts`) stamp the intersection. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id <id>` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id <id>] [--catch-up] [--retry-held] [--dry-run] [--json]` branch (`extractStaleFromDB`; on a managed brain it hands the sweep to `extractManagedStaleLinks` in `src/core/persistence/links-maintenance.ts` for every caller, CLI, cycle, jobs and maintain, and a live PGLite owner runs it through `src/commands/extract-stale-delegate.ts`; failing pages are reported and skipped there, and `--retry-held` clears held ones first) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale, then the mention pass. Embedded callers (the cycle) pass `quiet: true` so the helper writes nothing to stdout, including its JSON result; standalone CLI output is unchanged. In `jsonMode` every fs path, `extractForSlugs` included, reports a dropped flush as a `{event:'batch_error'}` stderr line rather than going silent. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush (the unmanaged sweep): a page's throw aborts the sweep after the batch's finished pages are flushed and stamped (S4), so the failing page and later ones stay stale and re-extract idempotently. `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. On a managed-persistence brain each page publishes its derived links (`prepareAutomaticLinks`), the coordinator's unrecorded canonical timeline tuples (`unrecordedCanonicalTimeline`, insert-only) and its watermark in one revision-bound `withCoordinatedWrite` transaction; a page edited since the read is skipped and stays stale. Tests: `test/extract-stale-managed.test.ts`, `test/persistence-managed-lifecycle.test.ts`. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Fresh-install schema in `src/schema.sql` (both generated blobs follow via `bun run build:schema`) + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor/checks/extraction-sync.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; pages whose attendance marker (`links_attendance_blocked_revision`, migration v180) equals their current `knowledge_revision` are left out of the lag count and reported as `details.attendance_blocked` (both `extract --stale` sweeps set the marker on an unresolved-attendance skip; pre-v180 brains fall back to the unfiltered count); strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` carries `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` at the inline-extract call site for the slugs the extractors actually read, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` lists `'extract.stale'` in `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` lists `links_extraction_lag` in `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp + crash-contract), `test/sync-inline-extract-stamps.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite case in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. W4.14: one basename index per slug set (rebuilt as it grows). GBRA-133 2b.1: the sync hooks (`extractLinksForSlugs` / `extractTimelineForSlugs`) share one `indexMarkdownTree` walk per run (`src/commands/extract-tree.ts`, with `markdownWalk` behind `walkMarkdownFiles`) (it yields after each directory) and type link endpoints through `withEndpointPageTypes`, which parses only the walked files the pass asks a type for (repeating until nothing new is asked) instead of `loadFsPageTypes` over the whole tree; full `gbrain extract` keeps the whole-tree parse, and the two give the same links (`test/pathological/s-sync-offpath-2b.test.ts`).
+- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. A lost link or timeline write on the fs walks or the slug-scoped path leaves its pages unstamped and throws `extract_write_lost` after stamping the rest (`core/extract-watermark.ts`); the sync hooks report a page whose link or timeline write failed in `errors`, never `processed`, so the caller leaves it unstamped. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use for live brains with no local checkout). The full fs walk replaces each page's own markdown-derived links through `replacePageFileLinks` (the per-slug contract below; a file with no page row is skipped and not stamped); `extract --stale` replaces through `replaceDerivedLinks`, the db link walk through `replaceDerivedLinksBatch`. Timeline rows are inserted in batches of 100 via `addTimelineEntriesBatch` (`ON CONFLICT DO NOTHING`; `created` counts real inserts) after every path — fs walk, db walk, per-slug, sync hooks, `--stale` — runs `retractRemovedTimelineEntries` for the page. `ExtractOpts.slugs?: string[]` enables incremental extract via `extractForSlugs()` (single combined links+timeline pass); the cycle path threads sync's `pagesAffected` through. Every per-slug extractor — `extractForSlugs` (the cycle's incremental pass) and the two sync hooks `extractLinksForSlugs` / `extractTimelineForSlugs` — resolves each requested slug through `resolveSlugRelPath`: the `buildSlugPathIndex(walkMarkdownFiles(dir))` index first, then — only on an index miss — the legacy `join(dir, slug + '.md')` existence check. The index is the primary because `pathToSlug` lowercases and slugifies, so a reconstructed path only finds files whose names are already slugs; the legacy fallback stays because sync ADMITS files the walker never emits (`_`-prefixed names, dot-dirs waived via `sync.include_hidden`) and those slugs still round-trip to their real path — without it such a page imports with no edges. On a slug collision the index prefers the file whose name IS the slug (else the first walked entry), so resolution never depends on read order. The per-slug extractors reconcile like `put_page`: each page's own markdown-derived links are replaced through `replacePageFileLinks` (`src/core/link-reconciliation.ts`, `replaceDerivedLinks` with `preserveExisting`; other producers survive, a page with no row counts as processed with no links, and a failed replacement is reported on stderr and leaves the page stale), and `retractRemovedTimelineEntries` (`src/core/timeline-extract.ts`) deletes the non-event rows whose (date, source, summary) any stored `page_versions` text of the page produced (under either timeline parser) and its current text no longer does; rows no version produced (enrichment, meeting fan-out, `--infer-dates` anchors) are kept, except an extractor row an older reading of that text filed (`supersededTimelineReadings`, e.g. the pre-#3280 bullet split, #7949), and the version scan only runs when the page holds a row the current text does not produce. `gbrain extract timeline --prune-orphans [--source-id] [--dry-run] [--json]` (`pruneTimelineOrphans`) runs that reconciliation over every page with timeline rows, without inserting — the one-time prune for rows orphaned while extraction was insert-only (managed brains retract each page's rows in a page-locked `withCoordinatedWrite` transaction; these rows are removals, so `gbrain repair timeline` / `timeline_history` exclude them; pinned in `test/persistence-managed-lifecycle.test.ts`); the read-only `timeline_orphans` doctor check (`src/commands/doctor/checks/timeline-orphans.ts`) samples 500 pages and points at it. Pinned by `test/timeline-reconcile-all-paths.test.ts`. `loadSlugAliasTargets` adds renamed pages' old slugs (`slug_aliases`) to `allSlugs` so links written before a rename resolve to the current slug. `extractLinksForSlugs` reads the requested files first, then loads page metadata only for those pages and their link endpoints (`fileLinkOwnership(engine, sourceId, { slugs })`); the unscoped whole-brain load happens only when a DB meeting or attendance origin needs `replaceFileLinks`. Both return `ExtractForSlugsResult { created, processed, errors? }`; `errors` names pages whose read or write failed, and sync reports them as `extract_error` (stderr line plus the `SyncResult` field) instead of swallowing them. `processed` is the subset actually read, and the shared `slugsSafeToStamp(links, timeline)` intersection is what every caller may stamp. Stamping the full requested set marks a silently-skipped page fresh and hides it from `extract --stale` permanently. All three call sites (`src/commands/sync.ts`, `src/core/github-source.ts`, `src/core/google/google-source.ts`) stamp the intersection. `walkMarkdownFiles(brainDir)` still runs to build `allSlugs` for link resolution. `--source-id <id>` scopes extraction to one source on federated brains (resolved via `resolveSourceWithTier()` before any SQL; failures hint `gbrain sources list`). `gbrain extract --stale [--source-id <id>] [--catch-up] [--retry-held] [--dry-run] [--json]` branch (`extractStaleFromDB`; on a managed brain it hands the sweep to `extractManagedStaleLinks` in `src/core/persistence/links-maintenance.ts` for every caller, CLI, cycle, jobs and maintain, and a live PGLite owner runs it through `src/commands/extract-stale-delegate.ts`; failing pages are reported and skipped there, and `--retry-held` clears held ones first) — incremental DB-source link+timeline sweep over pages whose `pages.links_extracted_at` watermark is stale, then the mention pass. Embedded callers (the cycle) pass `quiet: true` so the helper writes nothing to stdout, including its JSON result; standalone CLI output is unchanged. In `jsonMode` every fs path, `extractForSlugs` included, reports a dropped flush as a `{event:'batch_error'}` stderr line rather than going silent. Stale predicate (shared by both engines + the doctor check): `links_extracted_at IS NULL OR links_extracted_at < LINK_EXTRACTOR_VERSION_TS::timestamptz OR updated_at > links_extracted_at` (the `updated_at` arm catches MCP `put_page` / `sync --no-extract` edited-since-extract). Three `BrainEngine` methods (parity in postgres-engine.ts + pglite-engine.ts + bootstrap probes): `countStalePagesForExtraction(opts?)`, `listStalePagesForExtraction({batchSize, afterPageId?, sourceId?, versionTs?})` (returns page CONTENT to avoid N+1 `getPage`; `rowToStalePage` in utils.ts maps the row, `StalePageRow` in types.ts), `markPagesExtractedBatch(refs, defaultExtractedAt)` (3-array unnest `slug[],source_id[],ts[]`; each ref may carry its own `extractedAt`). `STALE_BATCH_SIZE` default 25 (`GBRAIN_EXTRACT_STALE_BATCH`; small because page bodies are unbounded — the LIMIT is the only fetch-time memory bound); `STALE_TIME_BUDGET_MS` 30min wall-clock (`--catch-up` removes the cap). Non-swallowing flush (the unmanaged sweep): a page's throw aborts the sweep after the batch's finished pages are flushed and stamped (S4), so the failing page and later ones stay stale and re-extract idempotently. `extractStaleFromDB` stamps with each row's READ `updated_at` (not `now()`), so a concurrent edit during the sweep keeps the page stale and it re-extracts next run rather than marked fresh-with-old-content. On a managed-persistence brain each page publishes its derived links (`prepareAutomaticLinks`), the coordinator's unrecorded canonical timeline tuples (`unrecordedCanonicalTimeline`, insert-only) and its watermark in one revision-bound `withCoordinatedWrite` transaction; a page edited since the read is skipped and stays stale. Tests: `test/extract-stale-managed.test.ts`, `test/persistence-managed-lifecycle.test.ts`. Source-correct stamping at DB-extract sites via `stampExtracted` (best-effort, never throws); `extractLinksFromDB` only stamps the combined watermark when `subcommand === 'all'` (a links-only run must not hide timeline staleness). `LINK_EXTRACTOR_VERSION_TS` lives in `src/core/link-extraction.ts` (bump like `CHUNKER_VERSION` to invalidate all prior stamps). Migration v112 (`pages_links_extracted_at`) adds nullable `TIMESTAMPTZ` + composite `(source_id, links_extracted_at)` index (CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain on PGLite), NO backfill so the real backlog surfaces on first `gbrain doctor`. Fresh-install schema in `src/schema.sql` (both generated blobs follow via `bun run build:schema`) + `REQUIRED_BOOTSTRAP_COVERAGE`. `src/commands/doctor/checks/extraction-sync.ts:checkLinksExtractionLag` (the `links_extraction_lag` check, also in `doctorReportRemote`) warn-only by default (>`GBRAIN_EXTRACTION_LAG_WARN_PCT`, default 20%; shared `EXTRACTION_LAG_WARN_PCT_DEFAULT` + `EXTRACTION_LAG_MIN_PAGES=100` + exported `_resolveEnvNumber`), hard-fails only when `GBRAIN_EXTRACTION_LAG_FAIL_PCT` is set; vacuous-skips <100 pages (no `--source`); pre-v112 brains graceful-skip via `isUndefinedColumnError`; pages whose attendance marker (`links_attendance_blocked_revision`, migration v180) equals their current `knowledge_revision` are left out of the lag count and reported as `details.attendance_blocked` (both `extract --stale` sweeps set the marker on an unresolved-attendance skip; pre-v180 brains fall back to the unfiltered count); strictly a SQL COUNT (safe on remote/thin-client). `src/commands/sync.ts` carries `--no-extract` (threaded through single-source + `--all` + `syncOneSource`), stamps `links_extracted_at` at the inline-extract call site for the slugs the extractors actually read, and `maybeExtractionNudge` prints a one-line stderr nudge after a `synced | first_sync | up_to_date` sync that leaves a backlog (`shouldNudgeAfterSync` pure predicate; `GBRAIN_SYNC_NO_EXTRACT_NUDGE` suppresses). `src/core/retry.ts` lists `'extract.stale'` in `BATCH_AUDIT_SITES`; `src/core/doctor-categories.ts` lists `links_extraction_lag` in `BRAIN_CHECK_NAMES`. Pinned by `test/extract-stale.test.ts` (incl. edited-after-stamp + crash-contract), `test/sync-inline-extract-stamps.test.ts`, `test/sync-nudge-status-gate.test.ts`, `test/doctor-links-extraction-lag.test.ts`, engine-parity (Postgres↔PGLite) for the 3 methods + v112 round-trip. The stale SELECT in both engines projects a deterministic full-µs UTC string `to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso` (carried on `StalePageRow.updated_at_iso`, populated by `rowToStalePage` in utils.ts with an ISO-only fallback — never `String(Date)`, which `::timestamptz` misparses); `extractStaleFromDB` stamps that exact-precision value, not a JS `Date` (which truncates to milliseconds), so on Postgres `links_extracted_at` equals the row's `updated_at` to the microsecond and `links_extraction_lag` clears — a ms-truncated stamp stays strictly below the µs `updated_at` and leaves every page perpetually stale, which `extract --stale` could never satisfy. `to_char` (not raw `::text`, which is `DateStyle`-fragile) keeps the projection deterministic. The `markPagesExtractedBatch` SQL is unchanged, so callers passing an explicit (e.g. backdated) `extractedAt` still control the stamp and the edited-since arm is exact. A deterministic PGLite case in `test/extract-stale.test.ts` injects a µs `updated_at`, runs `--stale`, and asserts the lag is 0 and stays 0. W4.14: one basename index per slug set (rebuilt as it grows). GBRA-133 2b.1: the sync hooks (`extractLinksForSlugs` / `extractTimelineForSlugs`) share one `indexMarkdownTree` walk per run (`src/commands/extract-tree.ts`, with `markdownWalk` behind `walkMarkdownFiles`) (it yields after each directory) and type link endpoints through `withEndpointPageTypes`, which parses only the walked files the pass asks a type for (repeating until nothing new is asked) instead of `loadFsPageTypes` over the whole tree; full `gbrain extract` keeps the whole-tree parse, and the two give the same links (`test/pathological/s-sync-offpath-2b.test.ts`).
diff --git a/docs/architecture/key-files/core-utilities-2.md b/docs/architecture/key-files/core-utilities-2.md
index eca37ab876..dd5a56be42 100644
--- a/docs/architecture/key-files/core-utilities-2.md
+++ b/docs/architecture/key-files/core-utilities-2.md
@@ -46,7 +46,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/surface-audit.ts` — `writeSurfaceChangeAudit(engine, audit)`: the surface-mutation audit trail. Every surface mutation (rescope CLI, `POST /admin/api/rescope-client`, `request_tools` persist) writes one typed `mcp_request_log` row — `operation='surface_change'`, params a RAW object `{actor, client_id, old, new, via}` via `executeRawJsonb` (never `JSON.stringify` into `::jsonb`). Zero new DDL; rides `idx_mcp_log_time_agent` + the retention TODO. Best-effort: a failed audit write warns to stderr, never fails the committed mutation (callers needing fail-closed semantics inspect the returned boolean). CLI-actor rows are written even though stdio ops don't otherwise log (documented exception — audit outranks the transport-logging convention). The usage reader excludes these rows from op-call stats.
- `src/core/timeline-marker.ts` — the `<!-- gbrain:materialized v1 <12-hex> -->` marker that precedes a materialized timeline bullet: `timelineKey` (the one normalization for prior-set membership, dedup, stored-row matching and marker hashes), `timelineTupleHash`, `materializedMarker`, marker-line detection and `stripMaterializedMarkers`. Both timeline extractors skip marker lines (a marker never becomes bullet or header detail) and `sanitizeRemoteBody` drops them from chunk text unless `keepMaterializedMarkers` is set (remote `get_page`/`fetch` keep them so an edit round-trips). A marker whose hash no longer matches the bullet below it marks nothing.
-- `src/core/timeline-citations.ts` — `parseInlineCitationTimelineEntries`: inline `[Source: <source>, YYYY-MM-DD]` citations become timeline candidates whose summary is the citing paragraph without the citation; a page with no `[Source:` returns none before any paragraph is built. #6184: a comment-only line is a paragraph boundary and `stripHtmlComments` removes `<!-- … -->` spans (and dangling `<!--`/`-->`) before the summary is built, so section and materialized-row markers are never filed as entries. #6226: a citation files one entry per dated `source, date` segment (a `;` splits sources only after a date), summaries unwrap paired emphasis and keep inline code (#8893), and `supersededInlineCitationEntries` returns older readings' entries the current one drops; `timeline-extract.ts:supersededCitationTimeline` runs it with both extractors' bullet skips and `retractRemovedTimelineEntries` retracts those rows when their detail is extractor-written (`hasExtractorDetail`: empty or `Source: <source>`). The dated-bullet regexes (`TIMELINE_LINE_RE`, `_CN`, `isDatedTimelineLine`) live here and `link-extraction.ts` imports them.
+- `src/core/timeline-citations.ts` — `parseInlineCitationTimelineEntries`: inline `[Source: <source>, YYYY-MM-DD]` citations become timeline candidates whose summary is the citing paragraph without the citation; a page with no `[Source:` returns none before any paragraph is built. #6184: a comment-only line is a paragraph boundary and `stripHtmlComments` removes `<!-- … -->` spans (and dangling `<!--`/`-->`) before the summary is built, so section and materialized-row markers are never filed as entries. #6226: a citation files one entry per dated `source, date` segment (a `;` splits sources only after a date), summaries unwrap paired emphasis and keep inline code (#8893), and `supersededInlineCitationEntries` returns older readings' entries the current one drops; `timeline-extract.ts:supersededTimelineReadings` runs it with both extractors' bullet skips and `retractRemovedTimelineEntries` retracts those rows when their detail is extractor-written (`hasExtractorDetail`: empty or `Source: <source>`). The dated-bullet regexes (`TIMELINE_LINE_RE`, `_CN`, `isDatedTimelineLine`) live here and `link-extraction.ts` imports them.
- `src/core/transcripts.ts` — `listRecentTranscripts(engine, opts)` library reused by both the `gbrain transcripts recent` CLI and the `get_recent_transcripts` MCP op. Reads `dream.synthesize.session_corpus_dir` + `dream.synthesize.meeting_transcripts_dir` config (same as `discoverTranscripts`); walks `.txt` files within `days`; applies the `isDreamOutput` guard from `transcript-discovery.ts` (skips dream-generated files); returns `{path, date, mtime, length, summary}[]` sorted newest-first. Summary mode (default true) = first non-empty line + ~250 trailing chars; full mode caps at 100KB/file. Missing/non-existent corpus dirs return `[]`, not error. Trust gate lives in the op handler, not here: the op throws `permission_denied` for `ctx.remote === true`; this is a trusted library function used by both the gated op and the local CLI.
- `src/core/eligibility/fence-overlay.ts` + `src/core/trust/fence-append.ts` — the facts-fence eligibility overlay for chunks (#5575 ENG-1). `loadFenceChunkOverlay(engine, { sourceId, slug, compiled_truth, timeline })` reads, in one query and only when the body has world fence rows, which rows to omit (a `write_gate_holds` hold on the row's fingerprint, through the `registerFactHoldFingerprint` seam for the write gate's `holdFingerprint`; a `fact_purges` tombstone, the backstop to L3's import overlay) and which to demote (stored `facts.trust_tier` below the page tier, or a row a fence append is publishing, declared with `withPendingFenceRows`). `prepareMarkdownChunks(page, maxTokens, overlay)` (`markdown-chunks.ts`) drops both from the page's chunks and emits demoted rows as trailing chunks whose first line is `fenceTrustMarker(tier)` (a `compactTrustLabel` line with origin `facts-fence`); without overlay rows the output is byte-identical. The importer (`prepareFenceAwareMarkdownChunks`), projection rebuilds (`ProjectionSnapshot.fenceOverlay`), embed's chunkless rebuilds (`prepareProseChunks`) and evidence delivery's page document (`pageEvidenceText`) all cut through it; `healOversizedChunks` keeps the marker on every split piece. `fenceLowTierRows` / `renderFenceLowTier` give the demoted rows of one text for the segment reader's low-tier block. `eligibility/stamp.ts` labels a row carrying a marker min(page tier, marker tier) with origin `facts-fence` and applies floors at that tier; a marker can only lower. `withPageTierKept(tx, page, fn)` restores the page's prior `trust_tier`/`write_origin` after a fence append's page write (promotion capped at the prior tier when raising back), used by `fence-write.ts` (legacy mirror), `memory-prepare.ts` (remember) and `facts-prepare.ts` (managed entity facts). Tests: `test/fence-chunk-overlay.test.ts` (PGLite and Postgres via `test/e2e/fence-chunk-overlay-postgres.test.ts`).
diff --git a/src/core/persistence/canonical-projections.ts b/src/core/persistence/canonical-projections.ts
index e496787204..ac535f9a9c 100644
--- a/src/core/persistence/canonical-projections.ts
+++ b/src/core/persistence/canonical-projections.ts
@@ -8,7 +8,7 @@ import { TAKES_FENCE_BEGIN, TAKES_FENCE_END, parseTakesFence } from '../takes-fe
import { extractFactsFromFenceText, pageFactDate } from '../facts/extract-from-fence.ts';
import { takesPreparation } from '../takes-write.ts';
import { LINK_EXTRACTOR_VERSION_TS, parseTimelineEntries } from '../link-extraction.ts';
-import { eachStoredTimelineRow, extractTimelineFromContent, hasExtractorDetail, supersededCitationTimeline, TimelineKeySet,
+import { eachStoredTimelineRow, extractTimelineFromContent, hasExtractorDetail, supersededTimelineReadings, TimelineKeySet,
type ExtractedTimelineEntry } from '../timeline-extract.ts';
import { sanitizeRemoteBody } from '../remote-body.ts';
import { sanitizeForJsonb } from '../batch-rows.ts';
@@ -69,9 +69,9 @@ export type ProjectionWriter = 'editing' | 'preserving' | 'file' | 'immutable';
* `in_body` exactly matches a new bullet, `drifted` matches one only after
* normalization, `removed` / `removed_marked` had an unmarked / materialized
* bullet in the prior body that the new body dropped, `superseded` is what
- * the citation reading before #6226 filed for a citation either body still
- * or once carried (with no detail beyond what the extractor writes) and the
- * current reading replaces, `superseded_annotated` is such a row someone gave
+ * an older reading (supersededTimelineReadings) filed for text either body
+ * still or once carried (with no detail beyond what the extractor writes) and
+ * the current reading replaces, `superseded_annotated` is such a row someone gave
* its own detail, and `database_only` has no bullet in either body.
*/
export type TimelineRowState = 'in_body' | 'drifted' | 'removed' | 'removed_marked' | 'superseded' | 'superseded_annotated' | 'database_only';
@@ -153,17 +153,17 @@ function markedTimeline(safe: string, slug: string, marked = new Set<string>()):
return marked;
}
-/** What a body says about timeline rows: its canonical timeline, the tuples a marker still matches, the superseded citation readings. */
+/** What a body says about timeline rows: its canonical timeline, the tuples a marker still matches, the superseded older readings. */
interface TimelineView { entries: Map<string, ExtractedTimelineEntry>; marked: Set<string>; superseded: Set<string> }
function timelineView(body: CanonicalBody, slug: string): TimelineView {
const safe = safeBody(body);
- return { entries: extractTimeline(safe, slug), marked: markedTimeline(safe, slug), superseded: new Set(supersededCitationTimeline(safe).map(t => timelineKey(t))) };
+ return { entries: extractTimeline(safe, slug), marked: markedTimeline(safe, slug), superseded: new Set(supersededTimelineReadings(safe).map(t => timelineKey(t))) };
}
/**
* What a prior body says about the stored rows a write classifies, by the row's index in the rows read and its key:
- * a bullet (`entry`), a bullet a marker still matches (`marked`), a superseded citation reading (`superseded`).
+ * a bullet (`entry`), a bullet a marker still matches (`marked`), a superseded older reading (`superseded`).
*/
export interface PriorRows { entry(i: number, key: string): boolean; marked(i: number, key: string): boolean; superseded(i: number, key: string): boolean }
@@ -207,7 +207,7 @@ export async function segmentedPriorView(engine: BrainEngine, page: { id: number
for (const t of extractTimelineFromContent(safe, slug)) mark(timelineKey(t), PRIOR_ENTRY);
for (const t of parseTimelineEntries(safe)) mark(timelineKey({ ...t, source: t.source ?? 'markdown' }), PRIOR_ENTRY);
for (const key of markedTimeline(safe, slug)) mark(key, PRIOR_MARKED);
- for (const t of supersededCitationTimeline(safe)) mark(timelineKey(t), PRIOR_SUPERSEDED);
+ for (const t of supersededTimelineReadings(safe)) mark(timelineKey(t), PRIOR_SUPERSEDED);
}
const fields = [...takes.values()].map(scan => scan.result());
const view: PriorView = {
@@ -225,14 +225,14 @@ function blockTimeline(text: string, slug: string) {
extracted: extractTimelineFromContent(safe, slug).map(t => [timelineKey(t), t] as const),
parsed: parseTimelineEntries(safe).map(t => [timelineKey({ ...t, source: t.source ?? 'markdown' }), { ...t, source: t.source ?? 'markdown', slug }] as const),
marked: markedTimeline(safe, slug),
- superseded: supersededCitationTimeline(safe).map(t => timelineKey(t)),
+ superseded: supersededTimelineReadings(safe).map(t => timelineKey(t)),
};
}
/**
* What one run of a segmented body holds for the timeline (spec 8.2): the timelineDigest of the timelineKey of each
* entry its blocks yield (extracted, parsed; each in block order, repeats kept), of each materialized marker and of each
- * superseded citation. Numbers, not entries, so a row and the read that builds it stay a small fraction of the run's
+ * superseded older reading. Numbers, not entries, so a row and the read that builds it stay a small fraction of the run's
* text even when forced cuts join a whole timeline into one run.
*/
interface TimelineRunOutput { x: number[]; p: number[]; m: number[]; s: number[] }
@@ -257,7 +257,7 @@ async function eachTimelineRun(engine: BrainEngine, page: { id: number; slug: st
for (const t of extractTimelineFromContent(safe, slug)) out.x.push(timelineDigest(timelineKey(t)));
for (const t of parseTimelineEntries(safe)) out.p.push(timelineDigest(timelineKey({ ...t, source: t.source ?? 'markdown' })));
for (const key of markedTimeline(safe, slug)) out.m.push(timelineDigest(key));
- for (const t of supersededCitationTimeline(safe)) out.s.push(timelineDigest(timelineKey(t)));
+ for (const t of supersededTimelineReadings(safe)) out.s.push(timelineDigest(timelineKey(t)));
},
}, visit, opts.onBlock);
return scope;
diff --git a/src/core/timeline-extract.ts b/src/core/timeline-extract.ts
index b0fa6a163f..cdbeb6ae74 100644
--- a/src/core/timeline-extract.ts
+++ b/src/core/timeline-extract.ts
@@ -107,15 +107,39 @@ export function extractTimelineFromContent(content: string, slug: string): Extra
type TimelineTuple = Pick<ExtractedTimelineEntry, 'date' | 'source' | 'summary'>;
const tupleKey = (e: TimelineTuple) => JSON.stringify([e.date, e.source, e.summary]);
+const LEGACY_DASHES = '—–-';
+
+/**
+ * #7949: how the file walk read a `- **YYYY-MM-DD** | text` bullet before #3280: cut at its first dash of any kind,
+ * spaced or not (`foo-bar.de` -> source `foo`, summary `bar.de`), as its lazy `(.+?)\s*[—–-]\s*(.+)$` did. Scanned
+ * by hand so a long line costs linear time.
+ */
+function legacySplitBullets(content: string): TimelineTuple[] {
+ const out: TimelineTuple[] = [];
+ for (const line of content.split(/\r?\n/)) {
+ const m = /^-\s+\*\*(\d{4}-\d{2}-\d{2})\*\*\s*\|(.*)$/.exec(line);
+ if (!m) continue;
+ const rest = m[2]!;
+ const start = rest.search(/\S/);
+ if (start < 0) continue;
+ let at = -1;
+ for (let i = start + 1; i < rest.length - 1; i++) if (LEGACY_DASHES.includes(rest[i]!)) { at = i; break; }
+ if (at < 0 && start > 0 && start < rest.length - 1 && LEGACY_DASHES.includes(rest[start]!)) at = start;
+ if (at >= 0) out.push({ date: m[1]!, source: rest.slice(0, at).trim(), summary: rest.slice(at + 1).trim() });
+ }
+ return out;
+}
+
/**
- * #6226: the tuples the inline-citation pass of either parser filed for this
- * text before one citation could name several dated sources, and that the
- * current reading no longer produces (supersededInlineCitationEntries).
+ * The tuples older readings of this text filed and the current one may no longer produce: the inline-citation pass of
+ * either parser before one citation could name several dated sources (#6226, supersededInlineCitationEntries) and the
+ * file walk's bullet split before #3280 (#7949, legacySplitBullets). Callers compare only rows the text does not keep.
*/
-export function supersededCitationTimeline(content: string): TimelineTuple[] {
+export function supersededTimelineReadings(content: string): TimelineTuple[] {
return [
...supersededInlineCitationEntries(content, { skipLine: (line) => bulletLinePattern.test(line) }),
...supersededInlineCitationEntries(content, { skipLine: isDatedTimelineLine }),
+ ...legacySplitBullets(content),
];
}
@@ -221,7 +245,7 @@ function keyDigest(key: string): [number, number] {
}
/**
- * What a page text says about its timeline rows: the tuples either parser yields, and the older citation readings it
+ * What a page text says about its timeline rows: the tuples either parser yields, and the older readings it
* supersedes. Read from a segmented body (segmentedTimelineText) with `within`, both hold only those keys, and a
* stored row whose key is not in `within` (one written after the read) is never retracted against them. `runs` holds
* the keys of the body's runs whose blocks were their stored text, which a retained generation need not read again.
@@ -261,7 +285,7 @@ export async function segmentedTimelineText(engine: Pick<BrainEngine, 'executeRa
let parsed = 0;
for await (const block of bodyBlocks(engine as BrainEngine, page, { runKeys: runs, ...(opts.stop ? { stop: opts.stop } : {}) })) {
for (const key of markdownTimelineKeys(block.text, page.slug)) if (!within) kept.add(key); else within.mark(key);
- for (const entry of supersededCitationTimeline(block.text)) {
+ for (const entry of supersededTimelineReadings(block.text)) {
const key = tupleKey(entry);
if (!within || within.has(key)) superseded.add(key);
}
@@ -347,12 +371,12 @@ export async function retractRemovedTimelineEntries(
WHERE p.source_id = $1 AND p.slug = $2`, [sourceId, slug]);
const wanted = new Set(extra.map(tupleKey));
const produced = new Set<string>();
- // #6226: rows the older citation reading filed for this text or a stored version.
- const superseded = new Set(typeof current === 'string' ? supersededCitationTimeline(current).map(tupleKey)
+ // #6226, #7949: rows an older reading filed for this text or a stored version.
+ const superseded = new Set(typeof current === 'string' ? supersededTimelineReadings(current).map(tupleKey)
: [...wanted].filter(key => current.superseded.has(key)));
const addText = (text: string) => {
for (const key of markdownTimelineKeys(text, slug)) if (wanted.has(key)) produced.add(key);
- for (const entry of supersededCitationTimeline(text)) if (wanted.has(tupleKey(entry))) superseded.add(tupleKey(entry));
+ for (const entry of supersededTimelineReadings(text)) if (wanted.has(tupleKey(entry))) superseded.add(tupleKey(entry));
};
const seen = new Set(typeof current === 'string' ? [] : current.runs ?? []);
for (const version of versions) {
diff --git a/test/timeline-legacy-split-7949.test.ts b/test/timeline-legacy-split-7949.test.ts
new file mode 100644
index 0000000000..2ea42751cf
--- /dev/null
+++ b/test/timeline-legacy-split-7949.test.ts
@@ -0,0 +1,60 @@
+/**
+ * #7949: rows the file walk filed before #3280, when it cut a dated bullet at
+ * its first dash of any kind (`foo-bar.de` -> `foo` / `bar.de`), are retired.
+ *
+ * Protects: `retractRemovedTimelineEntries` (the core of `extract timeline
+ * --prune-orphans` and the sync reconcile) retracts a stored row equal to that
+ * older reading of the page text when its detail is extractor-written, keeps
+ * the current row, and keeps a legacy-shaped row someone gave their own detail.
+ * Fails when: the older split is not one of the superseded readings, so no
+ * version of the page "produced" the row and it stays forever.
+ * Seams: none; real PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { retractRemovedTimelineEntries, supersededTimelineReadings } from '../src/core/timeline-extract.ts';
+import { cpuMs } from './helpers/cpu-time.ts';
+
+const slug = 'people/alice-example';
+const text = [
+ '- **2020-05-03** | Email activity with acme-example.com (1 messages); link day [[days/2020/2020-05-03|2020-05-03]].',
+ '- **2021-04-04** | Call with alice-example about the plan',
+].join('\n');
+const legacy = [
+ { date: '2020-05-03', source: 'Email activity with acme', summary: 'example.com (1 messages); link day [[days/2020/2020-05-03|2020-05-03]].' },
+ { date: '2021-04-04', source: 'Call with alice', summary: 'example about the plan' },
+];
+
+let engine: PGLiteEngine;
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 60_000);
+afterAll(async () => { await engine.disconnect(); }, 60_000);
+beforeEach(async () => { await resetPgliteState(engine); });
+
+test('the pre-#3280 split of a dated bullet is a superseded reading', () => {
+ expect(supersededTimelineReadings(text)).toEqual(legacy);
+ expect(supersededTimelineReadings('- **2020-05-03** | No dash at all here')).toEqual([]);
+ expect(supersededTimelineReadings(text.replace(/\n/g, '\r\n') + '\r\n')).toEqual(legacy);
+});
+
+test('the older split stays linear on a long line with no dash', () => {
+ const line = `- **2020-05-03** | ${' '.repeat(200_000)}x${' '.repeat(200_000)}`;
+ expect(cpuMs(() => supersededTimelineReadings(line))).toBeLessThan(1_000);
+});
+
+test('legacy split rows are retracted and the current rows stay', async () => {
+ await engine.putPage(slug, { type: 'person', title: 'Alice Example', compiled_truth: text, timeline: '' });
+ for (const row of legacy) await engine.addTimelineEntry(slug, row);
+ await engine.addTimelineEntry(slug, { date: '2020-05-03', source: 'markdown', summary: 'Email activity with acme-example.com (1 messages); link day [[days/2020/2020-05-03|2020-05-03]].' });
+ await engine.addTimelineEntry(slug, { date: '2021-04-04', source: 'markdown', summary: 'Call with alice-example about the plan' });
+ const retired = await retractRemovedTimelineEntries(engine, slug, 'default', text);
+ expect(retired.map(r => r.source).sort()).toEqual(['Call with alice', 'Email activity with acme']);
+ const left = await engine.executeRaw<{ source: string }>('SELECT source FROM timeline_entries ORDER BY source');
+ expect(left.map(r => r.source)).toEqual(['markdown', 'markdown']);
+});
+
+test('a legacy-shaped row someone gave their own detail stays', async () => {
+ await engine.putPage(slug, { type: 'person', title: 'Alice Example', compiled_truth: text, timeline: '' });
+ await engine.addTimelineEntry(slug, { ...legacy[1]!, detail: 'confirmed by phone' });
+ expect(await retractRemovedTimelineEntries(engine, slug, 'default', text)).toEqual([]);
+});