Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Read a segmented page fence for fact evidence, mark body-less versions incomplete, and keep inventories current

garrytand0bcb0f38abc
diff --git a/docs/architecture/key-files/commands-2.md b/docs/architecture/key-files/commands-2.md
index e558aec7fd..2f33645466 100644
--- a/docs/architecture/key-files/commands-2.md
+++ b/docs/architecture/key-files/commands-2.md
@@ -14,7 +14,6 @@ Doctor's embedding-dimension warning treats existing chunks as existing data eve
- `src/commands/doctor/checks/unbound-source.ts` — `unbound_source` (#5254) counts live pages with `pages.database_only_reason='unbound_source'` and no recorded origin (blank `source_path` and `source_uri`) per source, via `databaseOnlyPageSql` (`persistence/unbound-pages.ts`), the SQL form of `isDatabaseOnlyPage` (#6536). A managed sync that records a page's origin, and a reconcile apply, clear the marker. While the source is unbound the count is ok and the message carries the filled `sources writer status`/`writer claim` bind command (database-only is the chosen mode); once the source has a binding it warns, because binding never materializes those pages into canonical files. Unreadable state warns. Registered on the `persistence_capacity` line in `doctor.ts`. Tests: `e2e/unbound-source-postgres`, `unbound-source-6536`.
- `src/commands/doctor/checks/revision-backfill.ts` — `revision_backfill` (#5216, META, registered before `search_mode`): reads `readRevisionBackfillStatus` (column state, pages still without `knowledge_revision`, failed rows still pending with their page). ok when the column is NOT NULL; warn with the pending count and `gbrain apply-migrations --force-schema` as `fix` while any row is resumable (or only the NOT NULL step remains); failed rows are listed by page id, `source:slug`, attempt and error, and when only rows with spent attempts remain the `fix` is the read-only `gbrain repair orphan-children` preview. Tests: `test/doctor-revision-backfill.test.ts`.
- `src/commands/doctor/checks/extractor-facts.ts` — `extractor_facts_expired` (#5731): counts conversation-extractor facts the pre-v0.60.11.0 projection expired, by class (`evidenced`, `ambiguous`; `excluded` reported but never a finding) through `classifyExtractorFacts`, and names `gbrain repair extractor-facts` (explicit-only, so the remediation plan and banner list it with its preview). Read-only; health unknown when the query fails.
-- `src/commands/doctor/checks/unwithdrawn-facts.ts` — `unwithdrawn_facts` (#8819): counts facts a facts-fence generation retired although no fence of their page held them, by class (`evidenced`, `ambiguous`; `excluded` reported but never a finding) through `classifyUnwithdrawnFacts`, and names the explicit-only `gbrain repair unwithdrawn-facts` preview. Read-only.
- `src/commands/doctor/checks/atom-retirements.ts` — `atom_retirements_restorable` (#7013, BRAIN, a `WAVE_CHECKS` entry with `resolution: 'repair'`, host-only, cleared by the explicit-only `atoms-restore`, so remediation classifies it `explicit_kind_required`): counts the `restorable` rows of `atomRetirementsSql` per source; `warn` names the count, origins, oldest and newest `retired_at`, the end of the earliest purge hold (`hold_until`) and the read-only preview `gbrain repair atoms-restore --source <id> --json` as an agent `fix`; the apply is the user's call.
- `src/commands/doctor/checks/conversation-outcomes.ts` — `conversation_label_facts` (wave check, host-only, resolution `repair`): bounded count (capped at 1000, reported as at least N) of the default set `gbrain repair conversation-labels` would retire; names the preview, never the apply. `conversation_outcomes_stale` (wave check, host-only, resolution `operator`): counts active conversation extraction outcomes stamped with an older `extractor_version` than `CONVERSATION_EXTRACTOR_VERSION` (outcomes are never reopened automatically) and names the `--force --dry-run` re-extract preview; outcomes older than the stamp are counted as `unstamped` and never warn. Bounded scan of the extractor rows, capped at 5000 (reported as at least N). Read-only.
- `src/commands/doctor/checks/captured-facts.ts` — `captured_facts_active` (wave check, host-only): counts active capture-lane facts (`hook:writeback`, `hook:compact`, `sweep:corpus`) that `classifyCapturedFacts` classes as self-capture (`evidenced`) or paste candidates (`ambiguous`); `excluded` and `unclassifiable` are reported, never a finding. Names `gbrain repair captured-facts` (explicit-only) and says it counts facts while `self_capture` counts corpus files. Read-only; health unknown when the scan fails.
diff --git a/docs/architecture/key-files/core-persistence.md b/docs/architecture/key-files/core-persistence.md
index 28ec21caba..89100dcca2 100644
--- a/docs/architecture/key-files/core-persistence.md
+++ b/docs/architecture/key-files/core-persistence.md
@@ -21,7 +21,7 @@ Managed facts extraction (withdrawn-candidate skip and recovery, conversation fa
Derived atom pages may regenerate or retire when their canonical file is already absent, while ordinary page writes retain missing-file refusal.
Accepted output replays provider-free after restart; page/take preparers recheck original/current grants, source incarnation and revisions. Unresolved facts remain DB-only without phantom pages. Retired takes skip; semantic fact snapshots exclude derived vector/cache fields and journal quotas reject overlarge retained intent. Managed consolidation retains the explicit chronological `valid_until` writeback from the legacy consolidation phase, scoped to the accepted source/fact IDs and revalidated semantic snapshots; contradiction probes never gain supersession authority. `submitMaintenanceIntent` admits a maintenance request with its own intent kind keyed by the intent; `prepareMaintenanceMutation` also handles `managed_maintenance_phantom_merge` and `managed_maintenance_phantom_delete` (both delegated to `cycle/phantom-redirect-managed.ts`). `publishMaintenancePage` accepts an optional `eventProjection` (Life Chronicle): the event page and its depth-page timeline row commit in the same publication with the depth page key locked. Remote maintenance remains refused.
-- `src/core/persistence/canonical-projections.ts` — installs the facts, takes and timeline projections of a canonical page body inside its publication transaction. Rows written by the conversation fact extractor (`cli:extract-conversation-facts*`) share the page coordinate without a fence and are never expired as rows that left it (#1928 parity with the extract_facts reconcile), so republishing a conversation keeps its extracted facts until their own replay replaces them. Preparation takes the caller's prior snapshot and writer class and pins the stored timeline rows and take row numbers. Writer classes: `editing` (revision-bound `put_page`, reconcile with an approved body decision), `preserving` (`put_page` without a revision, reconcile, connectors), `file` (managed sync of file edits and managed import) and `immutable` (company-brain imports). One pure decision table (`timelineRowAction`) maps each row state (`in_body`, `drifted`, `removed`, `removed_marked`, `database_only`) to `refresh_detail`, `delete`, `keep` or `materialize`: database-only rows are materialized by `editing` and `preserving` writers and kept by the others; #6226 rows the older citation reading filed for either body are `superseded` (deleted by every writer) or, carrying their own detail, `superseded_annotated` (kept, never materialized); a materialized bullet the new body drops is deleted by `editing`/`file`, rendered again by `preserving` and kept DB-side by `immutable`. `materializeTimeline` runs during preparation, before the body is imported, digested and rendered, and appends each round-trippable row after existing bullets in date-then-id order as a marked bullet (`timeline-marker.ts`); `renderMaterializedBullet` requires render-then-extract to return exactly the tuple and normalized detail, so backlink receipts, empty or delimiter-bearing sources are kept and never rendered. `pendingTimelineRows` counts materializable and kept rows for the repair and doctor. Deletes and detail refreshes match the pinned id, tuple and detail, so rows changed after preparation are untouched. Fence refusals are typed `invalid_fence` (`fence-repair/refusal.ts`): `compileCanonicalProjections`, the validation-only entry, refuses the first defect `scanCanonicalFences` finds (repeated marker, any strict-parser warning, a row number used twice on the page) with wire `invalid_params`; a new fence row whose number belongs to a database-only take refuses as `stored_row_collision` with wire `take_row_collision`. When the prior takes fence does not parse, the page's stored `takes.row_num` count as prior canonical rows (TE1), so a row Tier 1 normalized updates its take instead of colliding. `unrecordedCanonicalTimeline` returns the body's canonical tuples with no stored row under the normalized key (managed `extract --stale` inserts only these), and `materializedHistoryRanges` locates marker lines whose hash matches the bullet after it, plus that bullet's indented detail (the dream verifier's history exemption). Tests: `test/canonical-projection-history.test.ts`, `test/timeline-materialize.test.ts` (PGLite and Postgres). A write that would drop rows held by a fence quoted in markdown code (readers project nothing from it) is refused as `invalid_fence` / `quoted_fence_rows` instead of deleting them. Pinned by `test/fence-marker-projection.test.ts`. #8819: the fact generation (`projectFactFenceGeneration`) retires only rows a fence of the page held (`generations/fence-evidence.ts`, with the prior snapshot's body as evidence); the expire statement is limited to its `retireIds`, the trust guard skips its `kept` rows, and a kept row whose number an incoming fence row claims moves above the page's rows first (`moveKeptRows`). #5969 (D3): an ordinary `put_page` / `put_pages` intent carries a `TimelineWritePolicy` (`timeline-omission.ts`) that changes only the rows the writer saw (`removed`, `removed_marked`): an explicitly empty section deletes them for every writer class; an omitted section keeps them for a local preserving writer (`materialize`) and deletes them with `drop_timeline`; maintenance, reconcile and file writers pass no policy. The pinned timeline `DELETE` returns what it removed, and the projection closure returns `{ timelineRowsRemoved: { count, earliest, latest } }` counting only `removed`/`removed_marked` rows it actually deleted (a row changed after preparation is neither deleted nor counted); `timelineRowsToRemove` judges the same set ahead of publication. #6259: a quarantined page (gate `quarantine` marker) projects no facts or takes (timeline rows still project); rows projected before it was quarantined stay as they are.
+- `src/core/persistence/canonical-projections.ts` — installs the facts, takes and timeline projections of a canonical page body inside its publication transaction. Rows written by the conversation fact extractor (`cli:extract-conversation-facts*`) share the page coordinate without a fence and are never expired as rows that left it (#1928 parity with the extract_facts reconcile), so republishing a conversation keeps its extracted facts until their own replay replaces them. Preparation takes the caller's prior snapshot and writer class and pins the stored timeline rows and take row numbers. Writer classes: `editing` (revision-bound `put_page`, reconcile with an approved body decision), `preserving` (`put_page` without a revision, reconcile, connectors), `file` (managed sync of file edits and managed import) and `immutable` (company-brain imports). One pure decision table (`timelineRowAction`) maps each row state (`in_body`, `drifted`, `removed`, `removed_marked`, `database_only`) to `refresh_detail`, `delete`, `keep` or `materialize`: database-only rows are materialized by `editing` and `preserving` writers and kept by the others; #6226 rows the older citation reading filed for either body are `superseded` (deleted by every writer) or, carrying their own detail, `superseded_annotated` (kept, never materialized); a materialized bullet the new body drops is deleted by `editing`/`file`, rendered again by `preserving` and kept DB-side by `immutable`. `materializeTimeline` runs during preparation, before the body is imported, digested and rendered, and appends each round-trippable row after existing bullets in date-then-id order as a marked bullet (`timeline-marker.ts`); `renderMaterializedBullet` requires render-then-extract to return exactly the tuple and normalized detail, so backlink receipts, empty or delimiter-bearing sources are kept and never rendered. `pendingTimelineRows` counts materializable and kept rows for the repair and doctor. Deletes and detail refreshes match the pinned id, tuple and detail, so rows changed after preparation are untouched. Fence refusals are typed `invalid_fence` (`fence-repair/refusal.ts`): `compileCanonicalProjections`, the validation-only entry, refuses the first defect `scanCanonicalFences` finds (repeated marker, any strict-parser warning, a row number used twice on the page) with wire `invalid_params`; a new fence row whose number belongs to a database-only take refuses as `stored_row_collision` with wire `take_row_collision`. When the prior takes fence does not parse, the page's stored `takes.row_num` count as prior canonical rows (TE1), so a row Tier 1 normalized updates its take instead of colliding. `unrecordedCanonicalTimeline` returns the body's canonical tuples with no stored row under the normalized key (managed `extract --stale` inserts only these), and `materializedHistoryRanges` locates marker lines whose hash matches the bullet after it, plus that bullet's indented detail (the dream verifier's history exemption). Tests: `test/canonical-projection-history.test.ts`, `test/timeline-materialize.test.ts` (PGLite and Postgres). A write that would drop rows held by a fence quoted in markdown code (readers project nothing from it) is refused as `invalid_fence` / `quoted_fence_rows` instead of deleting them. Pinned by `test/fence-marker-projection.test.ts`. #8819: see `fence-evidence.ts`. #5969 (D3): an ordinary `put_page` / `put_pages` intent carries a `TimelineWritePolicy` (`timeline-omission.ts`) that changes only the rows the writer saw (`removed`, `removed_marked`): an explicitly empty section deletes them for every writer class; an omitted section keeps them for a local preserving writer (`materialize`) and deletes them with `drop_timeline`; maintenance, reconcile and file writers pass no policy. The pinned timeline `DELETE` returns what it removed, and the projection closure returns `{ timelineRowsRemoved: { count, earliest, latest } }` counting only `removed`/`removed_marked` rows it actually deleted (a row changed after preparation is neither deleted nor counted); `timelineRowsToRemove` judges the same set ahead of publication. #6259: a quarantined page (gate `quarantine` marker) projects no facts or takes (timeline rows still project); rows projected before it was quarantined stay as they are.
- `src/core/persistence/timeline-omission.ts` — #5969 (D3): `timelineSectionOf` reads a put_page body's Timeline section from the caller's raw content before normalization (`omitted`: no timeline sentinel and no `## Timeline`/`## History` heading outside code; `emptied`: a section with no entries; `present`). Admission (`page-mutations.ts`) records it on the intent as `timeline_section` (a caller-sent value is stripped; failed-writes replay passes the stored one through `submitPageMutation`'s `timelineSection`). `assertTimelineNotOmitted` refuses a remote omission that would delete rows with `timeline_rows_would_be_removed` (fix: `get_page include_content:true`, then the `drop_timeline: true` retry) before admission and again during preparation; `timelineWritePolicy` builds the projection policy. Successful deletes surface as the receipt's `timeline_rows_removed` advisory (`page-advisories.ts`, dates only), copied onto `put_pages` entries (`page-batch.ts`, which accepts `drop_timeline` per entry).
diff --git a/docs/architecture/key-files/core-utilities-2.md b/docs/architecture/key-files/core-utilities-2.md
index 9c3c9241e1..2e88a2a908 100644
--- a/docs/architecture/key-files/core-utilities-2.md
+++ b/docs/architecture/key-files/core-utilities-2.md
@@ -9,7 +9,6 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/repair/` — `gbrain repair` core. `core.ts`: `resolveRepairScope` (default every active source; prints brain and source ids), `runRepair` (dry run unless `apply`; each item is one revision-bound `put_page` with a deterministic request id (replayed while pending, a new attempt id after a terminal failure), re-decided against current state just before it is applied; a cursor in `op_checkpoints` keyed by kind + scope resumes an interrupted or `--limit` run and is cleared when a scan completes; the same row carries the run id (written before the first applied item and passed to `apply` as `runId`), so a resumed run keeps the id that any authorization it granted is bound to; a kind may return a named per-item outcome (`RepairItemOutcome`), tallied in `outcomes` and `outcome_items`; a `spends: 'llm'` kind's plan reports `llm` (its paid-model estimate, null for an unpriced model under a user-set cap, and what is left under its own daily cap), and its result's `cost` adds `llm_usd` (the estimate on a dry run, the actual spend summed from each item's `llm_usd` on an apply) and `llm_cap_remaining_usd` (the smaller of the daily cap and the run allowance, less the spend), keys other kinds never carry; `maxLlmUsd` is the run's paid-model allowance: each item's `apply` gets what is left as `llmAllowanceUsd`, and once it is used up, or an item's own `llm_usd` estimate exceeds what is left, the run stops `budget_exhausted` before that item, naming spent and allowance, and rerunning the apply command resumes it (a dry run sums the per-item estimates of the items it would apply, so `--limit` lowers it); the core opens no budget tracker; `deadline` (epoch ms) stops the run with `time_budget` before the next item; an item outcome's `stop` ends the run after that item with its reason, message and fix; a kind's `report` hook adds `repaired`, `remaining`, `verification` (or refines `stopped`) once the run ends, and a plan's `scan` is copied to the result; stops before crossing 90% of the brain or principal lifetime-ID / receipt-byte caps with the `gbrain config set` value that finishes; a held writer stops with `gbrain sources writer status <source>`; other item failures go through one `ItemRun` (`repair.apply_items`): an accepted pending write is checked first, a run-scoped failure is thrown, an item-scoped one is a `failed[]` entry (`reason` its code, `retry: 'next_full_scan'`, `fix` the kind's preview) and the run goes on; the breaker stops it `item_failures_systemic`; a limited run with only item failures clears its cursor; a kind's `onItemFailures` gets the failed items unless the run stopped; `--expect` applies throw all but `REPAIR_ITEM_FAILURE_CODES`). `rotation.ts`: the bare-apply rotation fences and slug-conflicts share (keys joined with U+0001 for jsonb). `timeline.ts`: `scanTimelineHistory` (bounded id-order scan shared with the doctor; one stored-row read per page batch for the retraction check) and the materialize kind. `visibility.ts`: `planVisibilityRepair`, the tighten-only atom/concept backfill (atoms from `effectiveVisibility` of their origin, concepts from the strictest input over provenance edges and atoms' `concepts:`; no-lineage concepts are counted and left alone). `safe-chunks.ts`: a `publication: 'projection'` kind (no admission, zero lifetime-ID and receipt cost, no capacity stop) that re-seals markdown and code pages below the safe-chunk fence through `resealSafeChunks`, then embeds their NULL chunks unless `--no-embed`; code pages without a recorded path and other page kinds are residuals; a page whose re-seal throws item-scoped is `held` `projection_unbuildable` (code, message, retry `fix`), stays withheld and is retried next run; the plan also selects pages the v0.60.31 re-chunk held (`CREDENTIAL_RESEAL_HELD`, segmented pages excluded). The pending count is reported only by the doctor check `safe_index_pending`, which the post-upgrade recovery banner uses. `contextual-mode.ts`: a `publication: 'projection'` kind (#5621) that stamps `contextual_retrieval_mode` + `corpus_generation` on sealed live markdown pages with a NULL mode through `resolveImportContextualMode` (`src/core/import-contextual-mode.ts`, the resolver `importFromContent` also uses, strict source-row read), then under the page guard clears only vectors whose recorded embedding input the stamped convention no longer accepts (the #5553 rule) and re-embeds that page once unless `--no-embed`; unsealed pages are the `unsealed_projection` residual. `connector-checkpoints.ts`: a `publication: 'projection'` bookkeeping kind (#5686, `embeds: false`) that deletes connector checkpoint rows and retry pointers no registered source can load and that are older than 7 days, keeping rows a non-terminal request or recorded pending set references; brain-wide. `request-indexes.ts`: a `publication: 'projection'` brain-wide kind (#5762, `embeds: false`) that builds a missing online-built request index (`PERSISTENCE_REQUEST_ONLINE_INDEXES`: the managed sync checkpoint pair and the page purge's slug index) or drops an INVALID one and rebuilds it through `buildIndexOnline` (`src/core/schema-migrations/helpers.ts`: CONCURRENTLY on Postgres under the session's startup timeouts with no session SET, inline on PGLite, the result re-checked for validity, an index another session is still building is left alone); it clears `persistence_request_indexes`. `orphan-bindings.ts`: a projection bookkeeping kind (#5732) over `persistence/orphan-bindings.ts` that deletes persistence source bindings of a removed source or an earlier incarnation, brain-wide, keeping one a pending request of that incarnation references. `embedding-effects.ts`: a projection kind (#5629/#5734, `embeds: 'effect'`) over `persistence/embedding-settlement.ts`; each stale queued or failed embedding effect of a committed write is `reconciled` (the effect projection verifier passes), `superseded` (page deleted, or a newer revision of the same page owns its own effect), `retry_queued` (stale queued re-queued for the owner; failed granted the retry-effects allowance; an exhausted allowance gets one new bounded cycle under `embeddingGrantId(runId, effect)`, which a resumed run replays) or `blocked` with a reason; every change is a compare-and-set on state, attempts, execution token and recovery, and a changed candidate is `changed_since_preview`. Retry items count the whole retry budget in the embedding estimate. `registry.ts`: `REPAIR_REGISTRY` (one entry per `REPAIR_KINDS` name in `--all` order: handler, help summary, `embeds` — `effect` when the publication's embedding effect re-embeds each page outside the run, `inline` when the kind embeds in-process and honors `--no-embed`, `none` for bookkeeping kinds — `spends: 'llm'` for a kind that may call a paid chat model, and the doctor `checks` it clears), `repairMayEmbed` (embedding spend under the flags) and `repairMaySpend` (that or `spends: 'llm'`), `repairForCheck`, `repairApplyCommand` and `repairRunner`, the one trusted local runner that both `gbrain repair` and doctor remediation use (an applying runner resolves the CLI write wait once, before any kind runs, and puts it on the ctx as `writeWaitMs`, so every publication waits like other CLI writes and a malformed `GBRAIN_WRITE_WAIT_MS` refuses before a checkpoint; a preview never reads it) (its `run` takes `maxLlmUsd`; `repairRunner` and `repairSpec` take an optional `registry` of stub specs for tests, and production callers use the registered kinds); a new kind plugs in by adding its name to `REPAIR_KINDS` and one registry entry. A kind with `explicit_only` (`stale-atoms`, `extractor-facts`; the `extractor-facts.ts` stub refuses with `unavailable` until its fix wave 5 lane lands) runs only when named: `AUTO_REPAIR_REGISTRY` is the `--all`, no-kind and remediation set, `explicitRepairNotices` lists the rest as `explicit_kind_required` with `repairPreviewCommand`, and `runRepair` refuses an unnamed explicit-only kind (`explicitKindRequired`), so a supplied remediation step cannot run one. Explicit-only kinds and kinds with `preview_bound` (`fences`, which is not explicit-only and so also runs from `--all`, the remediation run and the maintenance cycle with a bare `--apply`) accept `--expect <hash>`; only explicit-only kinds accept `--include-ambiguous`; `--only`/`--skip`/`--slug` selections, `--no-llm` and `--max-usd` (only `spends: 'llm'` kinds; it only lowers their cap) are passed to `plan` as `RepairPlanOptions` and printed in the apply command; a plan's `preview_hash` is appended to its apply command, and its `listing` (every previewed item with its class) is printed by the dry run. `stale-atoms.ts` (#5770): live page-bound atoms (never `imported_from`, never `source_path`-only) whose source page is gone (`origin_gone`) or whose stripped 16-character `source_hash` differs from a source page whose extraction at its current hash produced atoms (`origin_changed`; evidence is a live atom at that hash, managed also the completed `managed-atoms` checkpoint via `managedAtomCompletedSql`, so zero yield replaces nothing). The preview saves the set with `saveApprovedSet`; `--apply --expect <hash>` loads it (`preview_changed` otherwise; `--apply` alone refuses with the filled commands), rechecks each atom (revision, class, origin, evidence) and reports `retired`, `changed_since_preview` or `file_conflict` (an uncoordinated atom-file edit; its attempt-aware `repairRequestId` lets a later apply retry once the file is fixed). An approved set applies only under the source selection it was previewed with, and the apply's resume cursor is keyed by its approval hash. Managed brains admit one `managed_maintenance_retire_stale_atoms` request per atom bound to its revision (`prepareStaleAtomRetirement`, rechecked under the atom and origin page locks, no live origin needed); unmanaged brains soft-delete in one locked transaction. Both stamp `retired_by: stale-atoms`/`retired_at` and bump the origin page's `managed-atoms-generation` (clearing its page state and completed checkpoints, keeping the current hash for `origin_changed`), so a restored page is extracted again and `publishManagedAtoms` restores the retired atoms. Tests: `test/repair-explicit-only.test.ts`, `test/repair-stale-atoms.test.ts` and `test/journey-atoms-sync-race.test.ts` (PGLite and Postgres). Tests: `test/repair-command.test.ts`, `test/derived-visibility-repair.test.ts`, `test/safe-chunk-reseal.test.ts` (PGLite and Postgres), `test/repair-contextual-mode-5621.serial.test.ts` (PGLite and Postgres), `test/repair-flags.test.ts`, `test/persistence-request-indexes.test.ts`. `attribution-backfill.ts` (Foundations 1, F1b): an automatic `publication: 'projection'` bookkeeping kind (`embeds: 'none'`, no doctor check) that fills NULL write attribution only where exactly one committed request proves it: `pages.revision_*` and `page_versions.write_*` from the page mutation (or fenced `remember`) whose `outcome.revision` is the row's `knowledge_revision` (skipped, duplicate and observe-only requests prove nothing), and `facts.write_*` from the `remember` whose `outcome.id` names the fact with status `inserted` or `superseded`. The preview proofs and each batch run inside a short transaction with `enable_nestloop` off (the journal join on the outcome key has no index, and fresh attribution columns have no statistics), restoring the caller's setting before it ends. Items are batches of up to 1,000 ids per table in (table, id) cursor order; each batch re-runs the proof under `lockPageKeys` for its pages and commits on its own, so an interrupted run resumes after the last committed batch. Unproven NULL rows are the `unrecorded_<table>` residuals. Test: `test/write-attribution-read.test.ts`. `google-file-modes.ts`: an explicit-only (`explicit_only: true`), `publication: 'projection'` filesystem-only kind (`embeds: false`, no admission, no request ids) that plans one item per loose entry from `scanGoogleFileModes` (cursor phase = source index; the item carries root, relative path, kind and old mode) and applies `tightenGoogleEntry` (outcomes `tightened`, `already_private`, `skipped` with a reason); symlink, foreign-owner and wrong-type entries are `skipped_*` residuals. It is last in `REPAIR_KINDS`, so `--all` runs it after `embedding-effects`; registry check `google_file_modes`. Every run re-scans, so a rerun after `--limit` picks up exactly what is still loose.
- `src/core/rerank-audit.ts` — failure-only JSONL audit at `~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation, mirrors `src/core/audit-slug-fallback.ts`). Exports `logRerankFailure({reason, model, query_hash, doc_count, error_summary})` + `readRecentRerankFailures(days)`. The `no_key` reason is written ONCE per process per model by the gateway itself (not per query by `applyReranker`): the provider key is absent, so calls are skipped without HTTP and results pass through unreranked; `auth` means key present but rejected; `unreachable` means no HTTP response at all (connection refused, DNS, reset; a stale `provider_base_urls` entry fails every call this way, Cat 40 R0), while `network` stays a provider 5xx. Deliberately no `logRerankSuccess`: writing once per tokenmax search is hot-path I/O churn AND success events leak query volume + timing into a local audit file. `gbrain doctor`'s `reranker_health` check (`doctor/checks/calibration.ts`) resolves enablement + model through `resolveSearchMode(loadSearchModeConfig)` — the plane search reranks with — and `rerankerReadiness` before reading the audit: readiness is evaluated against the DB-merged config plane (`loadConfigWithEngine`, so DB-plane provider keys and `provider_base_urls` self-host overrides count, exactly like the gateway the CLI configures): a brain with NO embedding provider (`isAvailable('embedding')` false on a configured gateway) is `ok` even when not ready — search runs keyword-only there and never reaches `applyReranker`, so there is nothing to fix; otherwise enabled-but-not-ready (key absent / unknown model) warns with the paste-ready fix; only audit rows for the RESOLVED model count (rows for any other model never make the live default warn or send the operator to verify the wrong key); `no_key` skip rows are informational once ready (those processes ran unreranked; a long-lived worker started before the key was added keeps skipping until restarted) and never outlive the fix as a warn; then the auth / payload / budget / unreachable (3+ rows: names the `provider_base_urls.<provider>` key) / transient / unknown ladders on the non-skip rows; disabled → ok (with an enable hint when the default is ready). Query text SHA-256-prefix-hashed (8 hex chars) for privacy. `GBRAIN_AUDIT_DIR` env override honored via the shared `resolveAuditDir()`.
- `src/core/repair/extractor-facts.ts` — `gbrain repair extractor-facts` (#5731), explicit-only and preview-bound. `classifyExtractorFacts` (read-only; also the doctor check's count) takes conversation-extractor facts with `expired_at` set and `row_num` NULL and classes each: `excluded` (page missing or deleted, `superseded_by` set, withdrawn, an active same-text fact on the page, or an earlier same-text candidate), `ambiguous` (unmanaged brain; no committed request of the page with `completed_at = expired_at`, compared in SQL as microsecond UTC text, which only a same-transaction completion satisfies on Postgres; a consumer at or above v0.60.11.0; or the page has ever held fence-projected facts) or `evidenced`. The preview lists every candidate, hashes them with the selection (`--include-ambiguous`) and saves one approved item per page (with its request id) through `preview-approval.ts`; the apply loads only that set, replays a still-pending request with the ctx's CLI write wait, skips pages whose request already committed or whose facts are active again, and restores each still-identical fact with a fresh `row_num` above the page's facts and fence rows. Managed brains publish one database-only `managed_maintenance_restore_extractor_facts` request per page (`submitDatabaseMaintenanceIntent`, dispatched by `prepareMaintenanceMutation` to `prepareExtractorFactsRestore`, which reclassifies under the page lock); unmanaged brains run the same restore in one locked transaction. Previews warn, naming the host, when a consumer older than this release published after `writer_version_cutoff`. On PGLite `now()` has millisecond resolution, so the same-transaction rule is millisecond-exact there.
-- `src/core/repair/unwithdrawn-facts.ts` — `gbrain repair unwithdrawn-facts` (#8819), explicit-only and preview-bound. `classifyUnwithdrawnFacts` (read-only; also the doctor check's count) takes each fact a `fact_fence` generation retired (a `retired` ledger item with a fact preimage that was active and positioned) whose row is still as the retirement left it (expired at the generation's instant, `row_num` NULL, same claim and visibility; conversation-extractor rows excluded), and classes it `evidenced` (no fence ever held it: `generations/fence-evidence.ts` over the ledger, the current page and its stored versions), `ambiguous` (`version_history_gap_at` set, or versions left unread) or `excluded:<reason>` (page missing, superseded, withdrawn, changed since retirement, `fence_held`, active duplicate, duplicate candidate). The preview saves one item per page with its revision through `preview-approval.ts`; the apply restores the preimage's lifecycle (`expired_at`, `valid_until`, `superseded_by`, consolidation marks) on the same id at its recorded row number when free, else above the page's rows, after reclassifying under the page key: managed, one database-only `managed_maintenance_restore_unwithdrawn_facts` request per page bound to the previewed revision (`prepareUnwithdrawnFactsRestore`, refusing `revision_conflict`); unmanaged, a page-locked `maintenanceTransaction` that refuses a changed revision. Doctor `unwithdrawn_facts` (`commands/doctor/checks/unwithdrawn-facts.ts`). Tests: `test/repair-unwithdrawn-facts-8819.test.ts` (PGLite and Postgres).
- `src/core/repair/conversation-labels.ts` — `gbrain repair conversation-labels`, explicit-only, preview-bound and destructive (consent payload from the registry's `consentText`). `classifyLabelFacts` (read-only) takes active per-segment extractor facts (`source` exactly `cli:extract-conversation-facts`) on live pages with a bold `METADATA_LABELS` line and classes each `excluded` (page outcome stamped with `extractor_version`, withdrawn, superseded, named by another fact's `superseded_by`, referenced by `open_loops.fact_id`), `evidenced` (context carries `segment 1970-01-01`) or `ambiguous`. The preview decides each page's outcome with the current parser at no cost (`non_extractable`, `awaiting_reextraction`, `cleaned`), hashes it all, saves one approved item per page and prints the capped hand-off (`extract-conversation-facts --slugs … --dry-run`, then `--max-cost-usd`). The apply expires the approved ids still as previewed (appending `retired: conversation-labels <hash>` to the context), expires the completion marker and writes the stamped not-extractable outcome for a page still at the previewed parser input; approved pages apply in batches of `LABEL_BATCH_PAGES` (25): one database-only `managed_maintenance_conversation_label_retire` request per batch on `LABELS_ANCHOR_SLUG` on a managed brain (`prepareConversationLabelRetirement`, every page key locked, each page rechecked), one transaction per batch otherwise. No model calls. Segmented pages are not classified (the label filter reads the body columns): their unstamped per-segment facts are counted as `segmented_pages` / `segmented_facts` and named in a warning. `countLabelFacts` is doctor `conversation_label_facts`' bounded count. Pinned by `test/repair-conversation-labels.test.ts` (+ Postgres arm) and `test/conversation-facts-retained-history.test.ts`.
- `src/core/repair/captured-facts.ts` — `gbrain repair captured-facts`, explicit-only and preview-bound. `classifyCapturedFacts` (read-only; also the `captured_facts_active` count) takes active capture-lane facts and classes each by its session (`captureSessionId`: `sweep:corpus:<file>` resolves through `corpusFileSessionId`): `evidenced` when a gbrain claude-cli scratch project holds the session's harness transcript or its corpus file sits in `<corpus>.quarantine/`; `ambiguous` (paste candidate) when a retained corpus file of the session holds a paste block and at least 60% of the claim's content words occur there only inside paste blocks; `excluded:legitimate_duplicate` when the claim and entity also have an active row that is not proven suspect. Sessions with no transcript and no corpus file are counted `unclassifiable`. The preview hashes every candidate with the selection (`--include-ambiguous`) and saves one approved item per page through `preview-approval.ts`; the apply reclassifies each fact, expires database-only rows through one database-only `managed_maintenance_expire_captured_facts` request per page on a managed brain (owner-checked by `maintenancePreflight`; `prepareCapturedFactsExpiry`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain, and strikes fenced rows in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection expires them and a later page write cannot reactivate them. No withdrawal is recorded, so `remember` can save the same claim again. Pinned by `test/repair-captured-facts.test.ts` (+ Postgres arm).
- `src/core/repair/redacted-facts.ts` — `gbrain repair redacted-facts` (#7125), explicit-only and preview-bound. `classifyRedactedFacts` (read-only) takes active facts whose text holds a redaction marker and runs `restoreNeutralized` (`think/sanitize.ts`, with the pre-fix `LEGACY_EVAL_SHELL` shape) against the page the fact's context names and the page holding it (`pageTexts`: an inline page's body + timeline, a segmented page's body through `readBodyRange` up to 16 MiB, then up to 50 `page_versions` bodies): `evidenced` when every proving text proves the same restoration, `ambiguous` (`conflicting_sources`) when a text holds two spans around the words or texts disagree, `unproven` (`no_source_page` / `not_in_source`) otherwise. A proved fenced row on a segmented page is `segmented` (`segmented_page`): listed with its restoration, never rewritten (`restoreFencedRows` reads and publishes inline bodies only). The preview hashes every candidate with its restoration and saves one approved item per page; the apply reclassifies each fact, rewrites database-only rows through one database-only `managed_maintenance_restore_redacted_facts` request per page on a managed brain (`prepareRedactedFactsRestore`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain (exact-text compare-and-set; context gets `restored: redacted-facts repair`), and rewrites fenced rows' claims in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection replaces the row. Pinned by `test/repair-redacted-facts.test.ts` (+ Postgres arm).
diff --git a/docs/architecture/key-files/generations.md b/docs/architecture/key-files/generations.md
index 745d5cea40..e7752367f4 100644
--- a/docs/architecture/key-files/generations.md
+++ b/docs/architecture/key-files/generations.md
@@ -4,6 +4,8 @@
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
- `src/core/generations/{contract,plan,publish,proofs,config,repair,ledger-scrub}.ts` + `adapters/{page,atoms,index}.ts` (GBRA-161) — the generation primitive: one ledger row per derived key generation (`derived_generations`, items and bounded preimages in `derived_generation_items`); `publishGeneration` holds a shrunk, oversize, unproven-empty, valve or segmented result and otherwise writes, completes, rechecks, retires (versioned, stamped) and activates in one transaction. `gbrain repair generations` restores or releases from previews; doctor `generation_retirements` / `generation_held`. Reference: [generations](../generations.md). Tests: `test/pathological/g1-generations.test.ts`, `test/pathological/g1-atom-generations.test.ts`.
-- `src/core/generations/fence-evidence.ts` (#8819) — `factsFenceHeld`: which positioned fact rows of a page some fence held at the same row number, claim and visibility (a non-retired item of any `fact_fence` generation of the page, the body the writer read, or a stored `page_versions` body, newest first, at most 1000; `complete` false past that). The page projection (`projectFactFenceGeneration`, which now returns `retireIds` and `kept`) retires only held rows and conversation-extractor rows a fence row displaces; the managed `extract_facts` reconcile keeps unheld stale rows out of its retirements. `moveKeptRows` moves a kept row whose number an incoming fence row claims above every row of the page. Tests: `test/fence-first-generation-8819.test.ts` (PGLite and Postgres).
+- `src/core/generations/fence-evidence.ts` (#8819) — `factsFenceHeld`: which positioned fact rows of a page some fence held at the same row number, claim and visibility (a non-retired item of any `fact_fence` generation of the page, the replaced body's rows from `pageFenceRowKeys`, which reads a segmented page from its marker segments, or a stored `page_versions` body, newest first, at most 1000; `complete` is false past that or when a version has no body, as a segmented snapshot has none). The page projection (`projectFactFenceGeneration`, which returns `retireIds` and `kept`) retires only held rows and conversation-extractor rows a fence row displaces; the managed `extract_facts` reconcile keeps unheld stale rows out of its retirements. `moveKeptRows` moves a kept row whose number an incoming fence row claims above every row of the page. The projection's evidence is also why `canonical-projections.ts` reads the prior fence before publication. Tests: `test/fence-first-generation-8819.test.ts` (PGLite and Postgres).
+- `src/core/repair/unwithdrawn-facts.ts` — `gbrain repair unwithdrawn-facts` (#8819), explicit-only and preview-bound. `classifyUnwithdrawnFacts` (read-only; also the doctor check's count) takes each fact a `fact_fence` generation retired (a `retired` ledger item with a fact preimage that was active and positioned) whose row is still as the retirement left it (expired at the generation's instant, `row_num` NULL, same claim and visibility; conversation-extractor rows excluded), and classes it `evidenced` (no fence ever held it: `generations/fence-evidence.ts` over the ledger, the current page and its stored versions), `ambiguous` (`version_history_gap_at` set, or versions left unread) or `excluded:<reason>` (page missing, superseded, withdrawn, changed since retirement, `fence_held`, active duplicate, duplicate candidate). The preview saves one item per page with its revision through `preview-approval.ts`; the apply restores the preimage's lifecycle (`expired_at`, `valid_until`, `superseded_by`, consolidation marks) on the same id at its recorded row number when free, else above the page's rows, after reclassifying under the page key: managed, one database-only `managed_maintenance_restore_unwithdrawn_facts` request per page bound to the previewed revision (`prepareUnwithdrawnFactsRestore`, refusing `revision_conflict`); unmanaged, a page-locked `maintenanceTransaction` that refuses a changed revision. Doctor `unwithdrawn_facts` (`commands/doctor/checks/unwithdrawn-facts.ts`). Tests: `test/repair-unwithdrawn-facts-8819.test.ts` (PGLite and Postgres).
+- `src/commands/doctor/checks/unwithdrawn-facts.ts` — doctor `unwithdrawn_facts` (#8819): counts those retirements by class through `classifyUnwithdrawnFacts` (`excluded` reported, never a finding) and names the `gbrain repair unwithdrawn-facts` preview. Read-only.
- `src/core/generations/adapters/synthesis.ts`, `src/core/generations/synthesis-coverage.ts`, `src/core/generations/managed-retire.ts`, `src/core/cycle/synthesize-generations.ts` (GBRA-161 PR 4, #7065) — synthesis on the primitive: each re-synthesized transcript's outputs become its `synthesis` generation after postprocess; an earlier output retires (versioned, `synthesis-resynthesis`) only when it is an exact duplicate or the same subject with every claim, quote and citation covered (pure match in `synthesis-coverage.ts`); others stay live and `gbrain repair generations --kind synthesis --mode cleanup` lists and, on apply, retires them (`synthesis-cleanup`). A chunk failure, deferral or `generations.synthesis.retire_previous=false` makes the generation partial. Managed brains admit the activation plus one `managed_generation_retire` delete per retirement (`managed-retire.ts`). Reference: [generations](../generations.md#synthesis). Tests: `test/pathological/g4-synthesis-resynthesis-7065.test.ts`.
- `src/core/generations/adapters/single-page.ts` (GBRA-161 PR 5) — dream summary, `think` page and drift report as single-item generations: revision-bound, versioned first, restorable. Reference: [generations](../generations.md#single-page-kinds-and-fence-only-writers-pr-5). Tests: `test/pathological/g5-*` to `g7-*`.
diff --git a/scripts/check-core-guard-coverage.mjs b/scripts/check-core-guard-coverage.mjs
index 18bdb75d43..1e865ea68b 100644
--- a/scripts/check-core-guard-coverage.mjs
+++ b/scripts/check-core-guard-coverage.mjs
@@ -49,6 +49,7 @@ const EXEMPT = {
'src/core/repair/captured-facts.ts': 'owner-run repair that expires fact rows (shrinks only)',
'src/core/repair/agent-session-facts.ts': 'owner-run repair that re-attributes capture-lane facts and retires their takes (no content added)',
'src/core/repair/extractor-facts.ts': 'owner-run repair restoring expired fact rows',
+ 'src/core/repair/unwithdrawn-facts.ts': 'owner-run, preview-bound repair putting expired fact rows back to their recorded before-state (row number and lifecycle); page text is unchanged',
'src/core/repair/redacted-facts.ts': 'owner-run, preview-bound repair rewriting the text of database-only fact rows to words a source page proves; page text is unchanged (fenced rows publish through put_page, preparePageMutation)',
'src/core/repair/visibility.ts': 'owner-run, preview-first repair that only tightens database-only fact rows from world to private; page text is unchanged (fenced rows and page labels publish through put_page, preparePageMutation)',
'src/core/repair/conversation-labels.ts': 'owner-run repair expiring fact rows and writing an outcome row (shrinks only; page text unchanged)',
diff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsv
index 06b68e76a8..d53dac8bf9 100644
--- a/scripts/page-body-readers-baseline.tsv
+++ b/scripts/page-body-readers-baseline.tsv
@@ -304,6 +304,9 @@ src/core/generations/adapters/single-page.ts publishSinglePage field NONE hashes
src/core/generations/adapters/single-page.ts singlePageAdapter field NONE restoreItem reads the body field of a ledger preimage JSON copy; segmented and oversize pages have no preimage
src/core/generations/adapters/single-page.ts writePage call:putPage SEGW writes a single-item derived page (dream summary, think synthesis, drift report) from the writer's proposed content or a ledger preimage; a segmented live page is held segmented_unsupported by the generation before any write
src/core/generations/adapters/single-page.ts writePage field INLINE passes the proposed content's compiled_truth to putPage; no stored page body is read (the prior page is versioned by createVersion)
+src/core/generations/fence-evidence.ts factsFenceHeld sql INLINE page_versions rows that hold a body copy (compiled_truth IS NOT NULL); a segmented version has none, is not read, and marks the evidence incomplete
+src/core/generations/fence-evidence.ts fenceRowKeys field INLINE inline arm only: a segmented body yields no rows here (pageFenceRowKeys reads it from its marker segments)
+src/core/generations/fence-evidence.ts pageFenceRowKeys field INLINE inline arm; a segmented page reads its fence from its marker segments (segmentedFactsFenceView)
src/core/generations/fence-restore.ts planFenceRestore call:serializePageToMarkdown INLINE a segmented page is refused segmented_unsupported before the page is serialized for the guarded restore write
src/core/generations/fence-restore.ts planFenceRestore field INLINE a segmented page is refused segmented_unsupported before its fence blocks are read
src/core/generations/fence.ts fenceInput field NONE reads the fence block strings of a FenceRead (already parsed), never a stored page body
@@ -597,6 +600,7 @@ src/core/repair/timeline.ts removedRowIds field STREAM
src/core/repair/timeline.ts scanTimelineHistory sql STREAM
src/core/repair/timeline.ts timelineRepair call:serializePageToMarkdown SEGW
src/core/repair/timeline.ts timelineRepair field STREAM
+src/core/repair/unwithdrawn-facts.ts classifyUnwithdrawnFacts sql INLINE reads compiled_truth and timeline of inline pages only (NULL for a segmented page, whose fence pageFenceRowKeys reads from its segments)
src/core/repair/visibility.ts applyFactItem call:serializePageToMarkdown INLINE inline pages only: a segmented page returns before the read and its fenced fact is left for the owner
src/core/repair/visibility.ts applyFactItem field INLINE inline pages only: a segmented page returns before the read and its fenced fact is left for the owner
src/core/repair/visibility.ts planAliases sql META
diff --git a/src/core/generations/adapters/fact-rows.ts b/src/core/generations/adapters/fact-rows.ts
index 4425b7946d..b06a14882b 100644
--- a/src/core/generations/adapters/fact-rows.ts
+++ b/src/core/generations/adapters/fact-rows.ts
@@ -19,7 +19,7 @@ import { readGenerationsConfig } from '../config.ts';
import type { GenerationAdapter, GenerationKey, ItemRow } from '../contract.ts';
import { factRowIdentity, fenceRowKey, FACT_FENCE_KIND, publishFenceGeneration, type FenceGenerationOutcome, type FenceItem, type FenceRead } from '../fence.ts';
import { activateGeneration, generationKeyLock, itemSetHash, readLiveGeneration, recordHeld } from '../publish.ts';
-import { factsFenceHeld, type FenceBody } from '../fence-evidence.ts';
+import { factsFenceHeld } from '../fence-evidence.ts';
export const CONSOLIDATION_KIND = 'consolidation';
@@ -64,7 +64,7 @@ export async function factPreimages(db: BrainEngine, sourceId: string, ids: read
* Held: the caller expires nothing.
*/
export async function projectFactFenceGeneration(tx: BrainEngine, input: { sourceId: string; slug: string; read: FenceRead; incoming: string;
- rows: ReadonlyArray<{ row_num: number; fact: string; visibility?: string | null }>; writer: string; prior?: FenceBody | null }):
+ rows: ReadonlyArray<{ row_num: number; fact: string; visibility?: string | null }>; writer: string; prior?: ReadonlySet<string> | null }):
Promise<FenceGenerationOutcome & { retiring: number; retireIds: number[]; kept: number[] }> {
const candidates = (await tx.executeRaw<{ preimage: FactPreimage }>(`SELECT ${FACT_PREIMAGE_SQL} AS preimage FROM facts f
WHERE f.source_id=$1 AND f.source_markdown_slug=$2 AND f.row_num IS NOT NULL
diff --git a/src/core/generations/fence-evidence.ts b/src/core/generations/fence-evidence.ts
index ae403b1215..1c0f4fd8c2 100644
--- a/src/core/generations/fence-evidence.ts
+++ b/src/core/generations/fence-evidence.ts
@@ -9,9 +9,11 @@ import type { BrainEngine } from '../engine.ts';
import { scanCanonicalFences } from '../fence-repair/refusal.ts';
import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts';
import { factRowIdentity, fenceOwnerKey, fenceRowKey, FACT_FENCE_KIND } from './fence.ts';
+import { isSegmentedPage } from '../page-body/page-shape.ts';
+import { segmentedFactsFenceView } from '../page-body/segment-facts.ts';
export interface FenceRowRef { id: number; row_num: number | null; fact: string; visibility: string }
-export interface FenceBody { compiled_truth: string | null; timeline?: string | null; body_storage?: string | null }
+export interface FenceBody { id?: number | string | null; compiled_truth?: string | null; timeline?: string | null; body_storage?: string | null }
/** Stored versions read per batch, and at most in all, newest first. */
const VERSION_BATCH = 50;
@@ -21,16 +23,24 @@ const rowKey = (row: { row_num: number | null; fact: string; visibility?: string
/** The (row number, claim, visibility) of every row a body's facts fence holds; nothing for a body that was not read whole. */
export function fenceRowKeys(body: FenceBody, slug: string, sourceId: string): Set<string> {
- if (body.body_storage === 'segmented' || body.compiled_truth == null) return new Set();
+ if (isSegmentedPage(body as { body_storage?: 'segmented' | null }) || body.compiled_truth == null) return new Set();
const scan = scanCanonicalFences({ compiled_truth: body.compiled_truth, timeline: body.timeline ?? '' });
return new Set(extractFactsFromFenceText(scan.facts, slug, sourceId).map(rowKey));
}
+/** fenceRowKeys of a stored page, a segmented one read from the runs that hold its fence markers. */
+export async function pageFenceRowKeys(db: BrainEngine, page: FenceBody, slug: string, sourceId: string): Promise<Set<string>> {
+ if (!isSegmentedPage(page as { body_storage?: 'segmented' | null })) return fenceRowKeys(page, slug, sourceId);
+ const view = await segmentedFactsFenceView(db, { id: Number(page.id), slug, source_id: sourceId });
+ return new Set(extractFactsFromFenceText([...view.body.facts, ...view.timeline.facts], slug, sourceId).map(rowKey));
+}
+
/**
* The ids of `rows` some fence of the page held (see the module comment). `complete` is false when stored versions were
- * left unread (more than VERSIONS_MAX), so a row outside `held` was only not found, not proven absent.
+ * left unread (more than VERSIONS_MAX, or a version stored without a body, as a segmented page's are), so a row outside
+ * `held` was only not found, not proven absent. `prior` is pageFenceRowKeys of the body the writer replaces.
*/
-export async function factsFenceHeld(db: BrainEngine, input: { sourceId: string; slug: string; rows: readonly FenceRowRef[]; prior?: FenceBody | null }):
+export async function factsFenceHeld(db: BrainEngine, input: { sourceId: string; slug: string; rows: readonly FenceRowRef[]; prior?: ReadonlySet<string> | null }):
Promise<{ held: Set<number>; complete: boolean }> {
const { sourceId, slug } = input;
const held = new Set<number>();
@@ -43,9 +53,10 @@ export async function factsFenceHeld(db: BrainEngine, input: { sourceId: string;
[sourceId, FACT_FENCE_KIND, fenceOwnerKey(slug), open().map(keyOf)])).map(row => row.item_key));
for (const row of open()) if (recorded.has(keyOf(row))) held.add(Number(row.id));
const markHeld = (keys: Set<string>) => { for (const row of open()) if (keys.has(rowKey(row))) held.add(Number(row.id)); };
- if (input.prior) markHeld(fenceRowKeys(input.prior, slug, sourceId));
+ if (input.prior) markHeld(new Set(input.prior));
let before: number | null = null;
- let complete = true;
+ let complete = !open().length || !(await db.executeRaw(`SELECT 1 FROM page_versions v JOIN pages p ON p.id = v.page_id
+ WHERE p.source_id = $1 AND p.slug = $2 AND v.compiled_truth IS NULL LIMIT 1`, [sourceId, slug])).length;
for (let read = 0; open().length; read += VERSION_BATCH) {
if (read >= VERSIONS_MAX) { complete = false; break; }
const versions: Array<{ id: number; compiled_truth: string; timeline: string | null }> = await db.executeRaw(`SELECT v.id, v.compiled_truth, v.timeline FROM page_versions v
diff --git a/src/core/persistence/canonical-projections.ts b/src/core/persistence/canonical-projections.ts
index 247c2fe76a..8efcd2df76 100644
--- a/src/core/persistence/canonical-projections.ts
+++ b/src/core/persistence/canonical-projections.ts
@@ -29,7 +29,7 @@ import { propagateFactLowering } from '../trust/propagation.ts';
import { loadWriteGateConfig } from '../trust/gate-outcomes.ts';
import { fenceOwnerKey, readFenceBlocks, TAKE_FENCE_KIND } from '../generations/fence.ts';
import { projectFactFenceGeneration } from '../generations/adapters/fact-rows.ts';
-import { moveKeptRows } from '../generations/fence-evidence.ts';
+import { moveKeptRows, pageFenceRowKeys } from '../generations/fence-evidence.ts';
import { projectTakeFenceGeneration } from '../generations/adapters/take-rows.ts';
import { linksFenceProjectionRows, projectLinksFence } from '../links-fence-projection.ts';
@@ -676,6 +676,10 @@ export async function prepareCanonicalProjections(engine: BrainEngine, page: Par
const fenceLinks = projectFences && !bodyUnread(page) ? await linksFenceProjectionRows(engine, sourceId, page) : null;
const storedRows = prior ? await storedTimeline(engine, prior.page.id) : [];
const prior_ = prior ? await priorView(engine, prior.page, slug, storedRows) : null;
+ // #8819: the fence rows of the body this write replaces, read at the observed revision, for the fact generation's evidence.
+ const priorFenceKeys = prior && projectFences && (await engine.executeRaw(`SELECT 1 FROM facts WHERE source_id=$1 AND source_markdown_slug=$2
+ AND row_num IS NOT NULL AND COALESCE(source,'') NOT LIKE 'cli:extract-conversation-facts%' LIMIT 1`, [sourceId, slug])).length
+ ? await pageFenceRowKeys(engine, prior.page, slug, sourceId) : null;
const { timeline, exactIncoming, pinned } = classifyTimeline(storedRows, page, prior_?.rows ?? null, slug, writer, policy);
const deletionRows = pinned.filter(row => row.action === 'delete')
.map(({ id, date, source, summary, detail, state }) => ({ id, date, source, summary, detail, removed: state === 'removed' || state === 'removed_marked' }));
@@ -806,7 +810,7 @@ export async function prepareCanonicalProjections(engine: BrainEngine, page: Par
// GBRA-161 PR 2: the fact and take generations decide first. A held one (its fence absent from this body) retires and
// overwrites nothing, so its rows stay as they are; every row a published one retires is recorded with its before-state.
const factGeneration = await step('fact_generation', () => projectFactFenceGeneration(tx, { sourceId, slug, read: fenceReads.facts, incoming, rows: factRows,
- writer: 'projection', prior: prior?.page ?? null }));
+ writer: 'projection', prior: priorFenceKeys }));
const factsHeld = factGeneration.state === 'held';
// #8819: a row the fence never held stays active; a fence row claiming its number takes it, and the row moves past the page's rows.
const kept = factsHeld ? [] : factGeneration.kept;
diff --git a/src/core/repair/unwithdrawn-facts.ts b/src/core/repair/unwithdrawn-facts.ts
index 0799b616f1..1924133608 100644
--- a/src/core/repair/unwithdrawn-facts.ts
+++ b/src/core/repair/unwithdrawn-facts.ts
@@ -40,7 +40,7 @@ import { maintenancePreflight, submitDatabaseMaintenanceIntent } from '../persis
import { clearApprovedSet, loadApprovedSet, previewChangedError, previewHash, saveApprovedSet } from '../persistence/preview-approval.ts';
import { maintenanceTransaction } from '../persistence/attribution.ts';
import { FACT_FENCE_KIND } from '../generations/fence.ts';
-import { factsFenceHeld } from '../generations/fence-evidence.ts';
+import { factsFenceHeld, pageFenceRowKeys } from '../generations/fence-evidence.ts';
import type { FactPreimage } from '../generations/adapters/fact-rows.ts';
import { afterCursor, type RepairHandler, type RepairItem, type RepairItemOutcome, type RepairListing, type RepairPlan, type RepairScope } from './core.ts';
@@ -122,9 +122,11 @@ export async function classifyUnwithdrawnFacts(db: BrainEngine, sourceIds: strin
const key = `${row.source_id}\u0000${row.slug}`;
if (evidence.has(key) || row.page_id === null) continue;
const page = rows.filter(other => other.source_id === row.source_id && other.slug === row.slug);
- const current = await db.executeRaw<{ compiled_truth: string | null; timeline: string | null; body_storage: string | null }>(
- 'SELECT compiled_truth, timeline, body_storage FROM pages WHERE id = $1', [row.page_id]);
- evidence.set(key, await factsFenceHeld(db, { sourceId: row.source_id, slug: row.slug, prior: current[0] ?? null,
+ const [current] = await db.executeRaw<{ id: number; compiled_truth: string | null; timeline: string | null; body_storage: string | null }>(
+ `SELECT id, CASE WHEN body_storage = 'segmented' THEN NULL ELSE compiled_truth END AS compiled_truth,
+ CASE WHEN body_storage = 'segmented' THEN NULL ELSE timeline END AS timeline, body_storage FROM pages WHERE id = $1`, [row.page_id]);
+ evidence.set(key, await factsFenceHeld(db, { sourceId: row.source_id, slug: row.slug,
+ prior: current ? await pageFenceRowKeys(db, current, row.slug, row.source_id) : null,
rows: page.map(other => ({ id: Number(other.id), row_num: Number(other.preimage.row_num), fact: other.preimage.fact, visibility: other.preimage.visibility })) }));
}
const base = rows.map(row => classify(row, evidence.get(`${row.source_id}\u0000${row.slug}`)));
diff --git a/test/fence-first-generation-8819.test.ts b/test/fence-first-generation-8819.test.ts
index f6f2c0e08a..b56f66571b 100644
--- a/test/fence-first-generation-8819.test.ts
+++ b/test/fence-first-generation-8819.test.ts
@@ -6,11 +6,15 @@
* Seeds write the page without a projection, so the managed write under test publishes generation 1 with no predecessor.
* PGLite; Postgres when DATABASE_URL is set.
*/
-import { describe, expect, test } from 'bun:test';
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import type { BrainEngine } from '../src/core/engine.ts';
import { pathologicalArms } from './pathological/harness.ts';
import { FACTS_FENCE, managedSource, PERSON, type ManagedSource } from './pathological/managed-fixture.ts';
import { runExtractFacts } from '../src/core/cycle/extract-facts.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { fenceRowKeys, pageFenceRowKeys } from '../src/core/generations/fence-evidence.ts';
+import { conversationBody } from './helpers/conversation-page.ts';
const SLUG = 'people/alice-example';
const ROW = (n: number, claim: string) => `| ${n} | ${claim} | fact | 1.0 | private | medium | 2020-01-01 | | chat | |`;
@@ -120,3 +124,28 @@ describe('#8819 first managed facts-fence generation', () => {
}, 120_000);
}
});
+
+describe('#8819 fence evidence of a segmented page', () => {
+ let engine: PGLiteEngine;
+
+ beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ });
+
+ afterAll(async () => {
+ await engine.disconnect();
+ });
+
+ test('a segmented prior body yields its fence rows from its segments, so a removed row still counts as fence-held', async () => {
+ await engine.setConfig('pages.segmented_storage', 'on');
+ const body = `${conversationBody(800_000, { seed: 1 })}\n\n${FACTS_FENCE([FOUNDED, CHESS].join('\n'))}\n\n${conversationBody(800_000, { seed: 2 })}\n`;
+ expect(await importFromContent(engine, SLUG, PERSON('Alice Example', body), { sourceId: 'default', noEmbed: true })).toMatchObject({ status: 'imported' });
+ const page = (await engine.readPageSnapshot(SLUG, { sourceId: 'default' }))!.page;
+ expect(page.body_storage).toBe('segmented');
+ expect(fenceRowKeys(page, SLUG, 'default').size).toBe(0);
+ expect([...await pageFenceRowKeys(engine, page, SLUG, 'default')].map(key => key.split('\u0000').slice(0, 2)))
+ .toEqual([['1', 'Alice founded acme-example'], ['2', 'Alice plays chess']]);
+ }, 120_000);
+});
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index b7a3e81a18..43d7fd0bbb 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -95,6 +95,7 @@ test/facts-insert-guards-7105.test.ts
test/facts-publication-large-fence-6898.test.ts
test/facts-subject-infer-7405.test.ts
test/fence-fact-append.test.ts
+test/fence-first-generation-8819.test.ts
test/fence-repair-receipts.test.ts
test/fenced-embed-reuse-6961.test.ts
test/file-drift-soft-delete-hold-release-gbra133.test.ts
@@ -483,6 +484,7 @@ test/repair-quoted-markers-8059.test.ts
test/repair-receipt-compaction.test.ts
test/repair-rename-identity-6729.test.ts
test/repair-secret-spans.test.ts
+test/repair-unwithdrawn-facts-8819.test.ts
test/retrieval-coverage-notices.test.ts
test/schema-missing-tables.test.ts
test/schema-replay-deadlock-6143.test.ts
diff --git a/test/repair-unwithdrawn-facts-8819.test.ts b/test/repair-unwithdrawn-facts-8819.test.ts
index 405446a07c..defcd4efbe 100644
--- a/test/repair-unwithdrawn-facts-8819.test.ts
+++ b/test/repair-unwithdrawn-facts-8819.test.ts
@@ -117,4 +117,24 @@ describe('gbrain repair unwithdrawn-facts (#8819)', () => {
} finally { await close(); }
}, 120_000);
}
+
+ for (const { backend, databaseUrl } of pathologicalArms()) {
+ test(`[${backend}] a stored version kept without a body (a segmented snapshot) leaves the row ambiguous, never restored by default`, async () => {
+ const { engine, close } = await isolatedSharedSkillsEngine(databaseUrl);
+ try {
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');
+ const sourceId = 'default';
+ await engine.putPage(SLUG, { type: 'person', title: 'Alice Example', compiled_truth: FACTS_FENCE(FOUNDED) }, { sourceId });
+ await engine.executeRaw(`INSERT INTO page_versions (page_id, compiled_truth, timeline, body_storage, body_generation)
+ SELECT id, NULL, NULL, 'segmented', 1 FROM pages WHERE source_id=$1 AND slug=$2`, [sourceId, SLUG]);
+ await factRow(engine, sourceId, 1, 'Alice founded acme-example', 'chat');
+ const chess = await factRow(engine, sourceId, 2, 'Alice plays chess');
+ await prefixRetire(engine, sourceId, [chess]);
+ const preview = await repair(engine, sourceId);
+ expect(preview.listing!.map(row => row.class)).toEqual(['ambiguous']);
+ expect((await unwithdrawnFactsCheck(engine, [sourceId])).details).toMatchObject({ evidenced: 0, ambiguous: 1 });
+ expect(await state(engine, [chess])).toEqual([[chess, null, false]]);
+ } finally { await close(); }
+ }, 120_000);
+ }
});
diff --git a/test/trust-write-gate-drift.test.ts b/test/trust-write-gate-drift.test.ts
index 6c825a2ffe..57cffcc91b 100644
--- a/test/trust-write-gate-drift.test.ts
+++ b/test/trust-write-gate-drift.test.ts
@@ -59,6 +59,8 @@ const EXEMPT: Record<string, string> = {
'src/core/generations/adapters/fact-rows.ts': 'generation restore puts back a fact row\'s own recorded before-state (lifecycle); no new text',
'src/core/repair/fact-claim-dates.ts': 'owner repair moving the valid_from of existing database-only facts to the date their claim states (lifecycle)',
'src/core/repair/extractor-facts.ts': 'owner repair of existing rows',
+ 'src/core/repair/unwithdrawn-facts.ts': 'owner repair behind a preview hash: puts back an existing fact row\'s own ledger before-state (lifecycle and row number); no new text',
+ 'src/core/generations/fence-evidence.ts': 'renumbers a kept existing fact row inside the gated fence projection or reconcile; no new text',
'src/core/repair/secret-spans.ts': 'owner repair behind a preview hash: replaces only the credential spans the secret scanner finds in a page\'s stored copies with <REDACTED:rule> markers (no new text)',
'src/core/repair/redacted-facts.ts': 'owner repair behind a preview hash: puts back into existing fact rows the words a stored source page proves; fenced rows publish through put_page, which runs the gate',
'src/core/repair/facts-rehome.ts': 'owner repair moving existing facts onto their entity page (placement)',