Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

repair redacted-facts restores conversation-extractor facts that have no Facts fence row (#8820) (#9384)

Garry Tanaee3bee1dc7e
diff --git a/changelog.d/capy-redacted-facts-legacy-rows.md b/changelog.d/capy-redacted-facts-legacy-rows.md
new file mode 100644
index 0000000000..5676c83762
--- /dev/null
+++ b/changelog.d/capy-redacted-facts-legacy-rows.md
@@ -0,0 +1,6 @@
+**`gbrain repair redacted-facts` now restores facts the conversation extractor wrote, which keep a row number on a page with no Facts fence (#8820).**
+
+### Fixed
+- `src/core/repair/redacted-facts.ts`: a fact with a row number is treated as a Facts-fence row only while its page's fence holds an active row with its text at that number. A conversation-extractor fact with no such fence row is now restored in place, keeping its row number, by the same row-exact rewrite as a database-only fact; its page is not changed. Before, the preview listed these facts as `evidenced` and the apply left them damaged with `changed_since_preview`. Any other fact whose fence row is gone is now listed as `unproven` (`fence_row_missing`) instead of `evidenced`, so the preview no longer promises a restore the apply cannot make. The repair is still preview-first: `--apply --expect <hash>` restores only the previewed set. Reported by @tarush1989.
+- Docs: `docs/guides/repair.md` (redacted facts class table and apply paragraph).
+- Tests: `test/repair-redacted-facts.test.ts` (PGLite, and Postgres through `test/e2e/repair-redacted-facts-postgres.test.ts`): a managed brain restores an extractor fact and leaves its page revision unchanged; an unmanaged brain restores an extractor fact and keeps a non-extractor fact whose fence row is gone. Both failed before the fix on both engines.
diff --git a/docs/architecture/canonical-writers.tsv b/docs/architecture/canonical-writers.tsv
index 9a6cad1da1..46a213977c 100644
--- a/docs/architecture/canonical-writers.tsv
+++ b/docs/architecture/canonical-writers.tsv
@@ -242,7 +242,7 @@ src/core/source-delete.ts 1 engine_guard Unmanaged source delete with its incarn
src/core/repair/extractor-facts.ts 1 coordinator `gbrain repair extractor-facts` (#5731) clears expired_at on preview-approved conversation-extractor facts and gives each a fresh row number. On a managed brain the UPDATE runs only inside the managed_maintenance_restore_extractor_facts publication (withCoordinatedWrite, page key locked, each fact reclassified first); an unmanaged brain runs the same function in one transaction under the page lock.
src/core/repair/captured-facts.ts 1 coordinator `gbrain repair captured-facts` expires preview-approved capture-lane facts that have no fence row. On a managed brain the UPDATE runs only inside the managed_maintenance_expire_captured_facts publication (owner-checked by maintenancePreflight, page key locked); an unmanaged brain runs it in one transaction. Fenced rows are never updated here: they are struck in the page and expired by the canonical projection of a revision-bound put_page.
src/core/repair/agent-session-facts.ts 2 coordinator gbrain repair agent-session-facts (#8064) re-attributes preview-approved database-only capture facts to the agent (attributed_to other plus a speaker note) and retires fact/self takes whose every contributing fact is then non-user. On a managed brain both UPDATEs run only inside the managed_maintenance_agent_session_facts publication (owner-checked by maintenancePreflight, page key locked); an unmanaged brain runs them in one maintenanceTransaction. The UPDATEs recheck attribution, activity and fence position under the row lock. Fenced facts and takes are never updated here: they are rewritten in the page through a revision-bound put_page and the canonical projection rewrites the rows in place.
-src/core/repair/redacted-facts.ts 1 coordinator `gbrain repair redacted-facts` (#7125) rewrites the text of preview-approved database-only facts to the words a source page proves. On a managed brain the UPDATE runs only inside the managed_maintenance_restore_redacted_facts publication (owner-checked by maintenancePreflight, page key locked); an unmanaged brain runs it in one maintenanceTransaction. The UPDATE compares text, visibility, entity and holding page under the row lock. Fenced rows are never updated here: their claim is rewritten in the page through a revision-bound put_page and the canonical projection replaces the row.
+src/core/repair/redacted-facts.ts 1 coordinator `gbrain repair redacted-facts` (#7125) rewrites the text of preview-approved database-only and conversation-extractor facts to the words a source page proves. On a managed brain the UPDATE runs only inside the managed_maintenance_restore_redacted_facts publication (owner-checked by maintenancePreflight, page key locked); an unmanaged brain runs it in one maintenanceTransaction. The UPDATE compares text, visibility, entity and holding page under the row lock. Fenced rows are never updated here: their claim is rewritten in the page through a revision-bound put_page and the canonical projection replaces the row.
src/core/repair/secret-spans.ts 2 coordinator `gbrain repair secret-spans` (GST-327) replaces only the credential spans the secret scanner finds in the preview-approved pages' stored copies with <REDACTED:rule> markers, one page per transaction under the page key after a digest recheck: inside withCoordinatedWrite with maintenance attribution on a managed brain, withWriteAttribution on an unmanaged one. Database only: the file side of a redaction goes through Git, history is kept, and a sync hold stays.
src/core/repair/ontology-facts.ts 1 coordinator `gbrain repair ontology-facts` (#6264) gives preview-approved ontology observations their own provenance back, clears their fence row number and expiry. On a managed brain the UPDATE runs only inside coordinatedDatabaseWrite (the ontology_propose path: write authority checked, source incarnation revalidated, entity and page keys locked); an unmanaged brain runs it in one maintenance transaction. It writes database-only ontology rows and never a page or file.
src/core/repair/facts-rehome.ts 1 coordinator `gbrain repair facts-rehome` (B-7, #6925) moves preview-approved database-only facts off a page-less entity_slug onto the page the strict resolver reaches. On a managed brain the UPDATE runs only inside coordinatedDatabaseWrite (the relink_facts operation: write authority checked, source incarnation revalidated, both slugs locked); an unmanaged brain runs one maintenance transaction per slug. It rechecks the slug, target and fact ids against the preview and never writes a page or file.
diff --git a/docs/architecture/key-files/core-utilities-2.md b/docs/architecture/key-files/core-utilities-2.md
index 9da55b1a13..78dd2faa06 100644
--- a/docs/architecture/key-files/core-utilities-2.md
+++ b/docs/architecture/key-files/core-utilities-2.md
@@ -11,7 +11,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/repair/extractor-facts.ts` — `gbrain repair extractor-facts` (#5731), explicit-only and preview-bound. `classifyExtractorFacts` (read-only; also the doctor check's count) takes conversation-extractor facts with `expired_at` set and `row_num` NULL and classes each: `excluded` (page missing or deleted, `superseded_by` set, withdrawn, an active same-text fact on the page, or an earlier same-text candidate), `ambiguous` (unmanaged brain; no committed request of the page with `completed_at = expired_at`, compared in SQL as microsecond UTC text, which only a same-transaction completion satisfies on Postgres; a consumer at or above v0.60.11.0; or the page has ever held fence-projected facts) or `evidenced`. The preview lists every candidate, hashes them with the selection (`--include-ambiguous`) and saves one approved item per page (with its request id) through `preview-approval.ts`; the apply loads only that set, replays a still-pending request with the ctx's CLI write wait, skips pages whose request already committed or whose facts are active again, and restores each still-identical fact with a fresh `row_num` above the page's facts and fence rows. Managed brains publish one database-only `managed_maintenance_restore_extractor_facts` request per page (`submitDatabaseMaintenanceIntent`, dispatched by `prepareMaintenanceMutation` to `prepareExtractorFactsRestore`, which reclassifies under the page lock); unmanaged brains run the same restore in one locked transaction. Previews warn, naming the host, when a consumer older than this release published after `writer_version_cutoff`. On PGLite `now()` has millisecond resolution, so the same-transaction rule is millisecond-exact there.
- `src/core/repair/conversation-labels.ts` — `gbrain repair conversation-labels`, explicit-only, preview-bound and destructive (consent payload from the registry's `consentText`). `classifyLabelFacts` (read-only) takes active per-segment extractor facts (`source` exactly `cli:extract-conversation-facts`) on live pages with a bold `METADATA_LABELS` line and classes each `excluded` (page outcome stamped with `extractor_version`, withdrawn, superseded, named by another fact's `superseded_by`, referenced by `open_loops.fact_id`), `evidenced` (context carries `segment 1970-01-01`) or `ambiguous`. The preview decides each page's outcome with the current parser at no cost (`non_extractable`, `awaiting_reextraction`, `cleaned`), hashes it all, saves one approved item per page and prints the capped hand-off (`extract-conversation-facts --slugs … --dry-run`, then `--max-cost-usd`). The apply expires the approved ids still as previewed (appending `retired: conversation-labels <hash>` to the context), expires the completion marker and writes the stamped not-extractable outcome for a page still at the previewed parser input; approved pages apply in batches of `LABEL_BATCH_PAGES` (25): one database-only `managed_maintenance_conversation_label_retire` request per batch on `LABELS_ANCHOR_SLUG` on a managed brain (`prepareConversationLabelRetirement`, every page key locked, each page rechecked), one transaction per batch otherwise. No model calls. Segmented pages are not classified (the label filter reads the body columns): their unstamped per-segment facts are counted as `segmented_pages` / `segmented_facts` and named in a warning. `countLabelFacts` is doctor `conversation_label_facts`' bounded count. Pinned by `test/repair-conversation-labels.test.ts` (+ Postgres arm) and `test/conversation-facts-retained-history.test.ts`.
- `src/core/repair/captured-facts.ts` — `gbrain repair captured-facts`, explicit-only and preview-bound. `classifyCapturedFacts` (read-only; also the `captured_facts_active` count) takes active capture-lane facts and classes each by its session (`captureSessionId`: `sweep:corpus:<file>` resolves through `corpusFileSessionId`): `evidenced` when a gbrain claude-cli scratch project holds the session's harness transcript or its corpus file sits in `<corpus>.quarantine/`; `ambiguous` (paste candidate) when a retained corpus file of the session holds a paste block and at least 60% of the claim's content words occur there only inside paste blocks; `excluded:legitimate_duplicate` when the claim and entity also have an active row that is not proven suspect. Sessions with no transcript and no corpus file are counted `unclassifiable`. The preview hashes every candidate with the selection (`--include-ambiguous`) and saves one approved item per page through `preview-approval.ts`; the apply reclassifies each fact, expires database-only rows through one database-only `managed_maintenance_expire_captured_facts` request per page on a managed brain (owner-checked by `maintenancePreflight`; `prepareCapturedFactsExpiry`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain, and strikes fenced rows in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection expires them and a later page write cannot reactivate them. No withdrawal is recorded, so `remember` can save the same claim again. Pinned by `test/repair-captured-facts.test.ts` (+ Postgres arm).
-- `src/core/repair/redacted-facts.ts` — `gbrain repair redacted-facts` (#7125), explicit-only and preview-bound. `classifyRedactedFacts` (read-only) takes active facts whose text holds a redaction marker and runs `restoreNeutralized` (`think/sanitize.ts`, with the pre-fix `LEGACY_EVAL_SHELL` shape) against the page the fact's context names and the page holding it (`pageTexts`: an inline page's body + timeline, a segmented page's body through `readBodyRange` up to 16 MiB, then up to 50 `page_versions` bodies): `evidenced` when every proving text proves the same restoration, `ambiguous` (`conflicting_sources`) when a text holds two spans around the words or texts disagree, `unproven` (`no_source_page` / `not_in_source`) otherwise. A proved fenced row on a segmented page is `segmented` (`segmented_page`): listed with its restoration, never rewritten (`restoreFencedRows` reads and publishes inline bodies only). The preview hashes every candidate with its restoration and saves one approved item per page; the apply reclassifies each fact, rewrites database-only rows through one database-only `managed_maintenance_restore_redacted_facts` request per page on a managed brain (`prepareRedactedFactsRestore`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain (exact-text compare-and-set; context gets `restored: redacted-facts repair`), and rewrites fenced rows' claims in the page's `## Facts` fence through one revision-bound `put_page`, so the canonical projection replaces the row. Pinned by `test/repair-redacted-facts.test.ts` (+ Postgres arm).
+- `src/core/repair/redacted-facts.ts` — `gbrain repair redacted-facts` (#7125), explicit-only, preview-bound. `classifyRedactedFacts` (read-only) takes active facts holding a redaction marker and runs `restoreNeutralized` (with the pre-fix `LEGACY_EVAL_SHELL` shape) against the page its context names and the page holding it (`pageTexts`: body + timeline, a segmented body via `readBodyRange` up to 16 MiB, then up to 50 `page_versions` bodies): `evidenced` when every proving text agrees, `ambiguous` (`conflicting_sources`) when a text holds two spans or texts disagree, `unproven` (`no_source_page` / `not_in_source`) otherwise. A numbered row is fenced only while its page's fence holds an active row with its text at that number; else an extractor row takes the database route (#8820) and any other is `unproven` (`fence_row_missing`). A proved fenced row on a segmented page is `segmented` (`segmented_page`): listed with its restoration, never rewritten. The preview hashes every candidate and saves one approved item per page; the apply reclassifies each fact, rewrites database-route rows through one database-only `managed_maintenance_restore_redacted_facts` request per page on a managed brain (`prepareRedactedFactsRestore`, dispatched from `prepareMaintenanceMutation`) or one transaction on an unmanaged brain (exact compare-and-set; context gets `restored: redacted-facts repair`), and rewrites fenced rows' claims in the page's `## Facts` fence through one revision-bound `put_page`, so the projection replaces the row. Pinned by `test/repair-redacted-facts.test.ts` (+ Postgres arm).
- `src/core/repair/loop-facts.ts` — `gbrain repair loop-facts` (#5869), explicit-only and preview-bound (`--apply --expect <hash>`): candidates are `loopFactDrift` rows (closed `done`/`dropped` loops whose commitment fact is active, in the loop's source and shared with no open loop); each item is rechecked (`changed_since_preview`) and retired through `retireLoopFact`, the same coordinated strike-plus-expire publication `loops_close` uses.
- `src/core/repair/failed-writes.ts` — `gbrain repair failed-writes` (#5983), explicit-only and preview-bound: candidates are failed, uncompacted `persistence_requests` refused by the managed writer guard (`writer_coordinator_required`, or `storage_error` "Publication failed (P0001)"). Caller writes (`put_page` without an internal `kind`, `add_timeline_entry`, `remember`) are classified `replay` / `already_written` / `duplicate` / `superseded` / `unpinned_target` by later requests on the same page and by a committed replay request id, and `file_database_drift` (W4.11) when its last replay attempt was refused `source_changed` and the page has not been written since (`replayWalks`, set reads; the detail and an apply's `file_database_drift` outcome name `gbrain sources reconcile <source> <slug> --preview`; `test/repair-failed-writes-drift-w411.test.ts`); owner-produced writes are `producer_owned` residuals with the producer command (a failed `managed_file_repair` names both `gbrain repair frontmatter` and `gbrain repair fences`). #7602 fact rows: `factsRecovery` ([Facts](core-persistence-facts.md)). The approved set records each target page's revision. Apply resumes an admitted replay attempt, otherwise reclassifies and rechecks the revision, re-runs `authorizeStoredRequest` on the failed row, and resubmits the retained intent on the original trust lane (`laneContext`: remote, take holders, delegated prefixes) through `submitPageMutation` / `submitRememberMutation`, with `put_page` bound to the previewed revision and `remember` pinned to its resolved subject, under a request id derived from the failed request id and attempt (`repair-failed-writes-5983.test.ts`). #5994: `laneContext` sets `ctx.replayAuthority` to the stored authority, so admission (`submissionAuthority` → `replayedAuthority` in `persistence/authority.ts`, and `requestPrincipalForContext`) reuses its principal, delegation, scopes and autoLinkTrusted for the same source incarnation and re-authorizes it live; a restricted write gets back its recorded job (`delegatedJobId`, or the id in a single legacy `wiki/agents/<id>/*` prefix, replayed on the legacy sandboxed path when its original was), and one with neither is accepted by `enforceSubagentSlugFence` (`ops/context.ts`) only through that marker with the exact non-empty stored allow-list. The fence still fails closed for every live dispatch. A put_page replay passes the stored `timeline_section` (#5969). The superseded / already-written / duplicate rule lives in `persistence/terminal-classify.ts`, shared with the #6650 terminal-failure readers.
- `src/core/repair/ontology-facts.ts` — `gbrain repair ontology-facts` (#6264), explicit-only and preview-bound (`--apply --expect <hash>`). `classifyOntologyFacts` (read-only; also doctor `ontology_facts_fenced`) takes ontology rows (`dimension IS NOT NULL`) carrying the pre-fix fence step's signature, a `row_num` or a `source_markdown_slug` that differs from `source` (`mergeOntologyFact` writes them equal and never sets `row_num`), and classes each `fenced` or `retired`, or excluded as `withdrawn` (`fact_withdrawals`), `consolidated` or `duplicate` (restoring would collide with the ontology dedup key). The apply sets `row_num` NULL, `source_markdown_slug = source` and `expired_at` NULL in one statement that rechecks row number, page, expiry and the dedup key (`changed_since_preview` otherwise), through `coordinatedDatabaseWrite('ontology_propose')` on the entity and page keys on a managed brain or one maintenance transaction otherwise; the withdrawal trigger re-expires a claim forgotten after the preview (`withdrawn`). Page text is never rewritten. Tests: `test/ontology-fence-sweep.test.ts`.
diff --git a/docs/architecture/system-of-record.md b/docs/architecture/system-of-record.md
index 15cd31e0b8..c8592810dd 100644
--- a/docs/architecture/system-of-record.md
+++ b/docs/architecture/system-of-record.md
@@ -321,7 +321,7 @@ or a physical projection that leaves attribution untouched):
- `src/core/repair/ghost-pages.ts` (2): `gbrain repair ghost-pages` retirement: a receipted `managed_maintenance_retire_ghost_page` request (a prepared `delete_page` with no file bytes) on managed brains; the unmanaged version and soft delete run in `maintenanceTransaction` after the same recheck.
- `src/core/repair/moved-pages.ts` (1): `gbrain repair moved-pages` fold: a receipted `managed_maintenance_fold_moved_page` request on the old page that also holds the moved page's key, on managed brains; the unmanaged fold runs in `maintenanceTransaction` under both page locks after the same recheck.
- `src/core/repair/ontology-facts.ts` (1): a coordinated database-only write (`coordinatedDatabaseWrite`, the `ontology_propose` path) on managed brains; the unmanaged ontology-facts restore runs in `maintenanceTransaction`.
-- `src/core/repair/redacted-facts.ts` (1): a `managed_maintenance_restore_redacted_facts` request on managed brains; the unmanaged restore of database-only fact text runs in `maintenanceTransaction`.
+- `src/core/repair/redacted-facts.ts` (1): a `managed_maintenance_restore_redacted_facts` request on managed brains; the unmanaged restore of database-only and conversation-extractor fact text runs in `maintenanceTransaction`.
- `src/core/repair/rename-identity.ts` (5): `gbrain repair rename-identity` carry, unmanaged brains only (a managed brain refuses `managed_remedy_unavailable` before writing); the fence-row merge, the provisional page's retirement, the slug move and the import run in one `maintenanceTransaction` under both page locks after the plan-hash recheck.
- `src/core/repair/secret-spans.ts` (1): `gbrain repair secret-spans` (GST-327) rewrites the credential spans of each preview-approved page under the page key: inside `withCoordinatedWrite` with maintenance attribution on managed brains, `withWriteAttribution` with maintenance attribution on unmanaged ones.
- `src/core/repair/segmented-tiling.ts` (1): `gbrain repair segmented-tiling` (GST-1209) versions each preview-approved damaged page (`createVersion`) before republishing it from the file Git holds at HEAD, under the page key: inside `withCoordinatedWrite` with maintenance attribution on managed brains, `withWriteAttribution` with maintenance attribution on unmanaged ones.
diff --git a/docs/guides/repair.md b/docs/guides/repair.md
index fe316a5757..ec146e8461 100644
--- a/docs/guides/repair.md
+++ b/docs/guides/repair.md
@@ -2792,14 +2792,16 @@ matches. A page stored in segments is read as evidence up to 16 MiB.
| --- | --- | --- |
| `evidenced` | Every text that proves a restoration proves the same one. | By `--apply --expect <hash>` |
| `ambiguous` | A text holds two different spans around the same words, or two texts prove different restorations. | Never; listed |
-| `unproven` | No page it came from exists any more (`no_source_page`), or none still holds the words around the marker (`not_in_source`). | Never; listed |
+| `unproven` | No page it came from exists any more (`no_source_page`), or none still holds the words around the marker (`not_in_source`), or the fact's row number names no active row with its text in its page's `## Facts` fence and it did not come from the conversation extractor (`fence_row_missing`; the listing shows the restored text). | Never; listed |
| `segmented` | Proved, but the fact's row sits in the `## Facts` fence of a page stored in segments, which this repair does not rewrite (`segmented_page`). The listing shows the restored text. | Never; listed |
The apply restores exactly the previewed set. A fact that changed since the
preview reports `changed_since_preview` and keeps its text. A fact with a row
in its page's `## Facts` fence has that row's claim rewritten in the page
(one revision-bound `put_page` per page, which re-chunks and re-embeds the
-page); a fact with no fence row is rewritten database-only (one maintenance
+page); a fact with no row number, or a conversation-extractor fact whose
+row number names no fence row (its page has no `## Facts` fence), is
+rewritten database-only, row number kept (one maintenance
request per page on a managed brain) and gets `restored: redacted-facts
repair` appended to its context; its vector no longer matches its text, so
the next `gbrain embed` re-embeds it. Page history keeps the damaged
diff --git a/src/core/repair/redacted-facts.ts b/src/core/repair/redacted-facts.ts
index 2fe84cc13b..a7cc422733 100644
--- a/src/core/repair/redacted-facts.ts
+++ b/src/core/repair/redacted-facts.ts
@@ -27,6 +27,12 @@
* - segmented: proved, but the row sits in the facts fence of a page
* stored in segments, which this repair does not rewrite; listed with
* its restoration, never changed.
+ * A row with a row number is a fence row only while its page's `## Facts`
+ * fence holds an active row with its text at that number. A conversation-extractor row keeps the
+ * row number the extractor gave it on a page with no such row (#8820) and is
+ * restored like a database-only row; any other row whose fence row is gone
+ * is unproven (`fence_row_missing`), listed with its restoration, never
+ * changed.
* A segmented source page is read as evidence through the segment reader
* (up to 16 MiB).
*
@@ -34,7 +40,8 @@
* previewed. A fenced row (`row_num` on its page) gets its claim rewritten in
* the page's `## Facts` fence by one revision-bound `put_page` per page, so
* the canonical projection replaces the row and the page's chunks are
- * re-embedded by its publication. A database-only row gets its text rewritten
+ * re-embedded by its publication. A database-only or conversation-extractor
+ * row gets its text rewritten
* by one database-only maintenance request on a managed brain (one
* transaction on an unmanaged brain); its vector is then stale by its text
* hash and `gbrain embed` re-embeds it. Page history keeps the damaged
@@ -43,7 +50,7 @@
import type { BrainEngine } from '../engine.ts';
import { opError, OperationError } from '../ops/contract.ts';
import { readFix } from '../ops/op-fix.ts';
-import { parseFactsFence } from '../facts-fence.ts';
+import { parseFactsFence, parseFactsFenceFor } from '../facts-fence.ts';
import { strikeFenceRow } from '../facts/forget.ts';
import { serializePageToMarkdown } from '../markdown.ts';
import { digest } from '../persistence/digest.ts';
@@ -76,8 +83,10 @@ export interface RedactedFactCandidate {
/** The page the fact lives on (fenced rows) or its entity key (database-only rows). */
slug: string;
row_num: number | null;
+ /** `fence`: rewritten in its page's Facts fence; `database`: rewritten in place (no row number, or a conversation-extractor row with no fence row). */
+ route: 'fence' | 'database';
class: RedactedFactClass;
- /** `restorable`, `conflicting_sources`, `no_source_page`, `not_in_source` or `segmented_page`. */
+ /** `restorable`, `conflicting_sources`, `no_source_page`, `not_in_source`, `segmented_page` or `fence_row_missing`. */
reason: string;
/** The stored text, and the restored text when evidenced (or proved on a segmented page). */
fact: string;
@@ -98,7 +107,7 @@ interface PageItem extends RepairItem { page: RedactedFactsPage; hash: string; l
interface FactRow {
id: number | string; source_id: string; fact: string; visibility: string; entity_slug: string | null;
- row_num: number | null; source_markdown_slug: string | null; context: string | null;
+ row_num: number | null; source_markdown_slug: string | null; context: string | null; source: string | null;
}
const factHash = (row: FactRow) => digest(['redacted-fact-v1', row.fact, row.visibility, row.entity_slug, row.row_num, row.source_markdown_slug]);
@@ -109,7 +118,7 @@ const contextSlug = (context: string | null) => {
return /^[^\s|]+$/.test(first) ? first : null;
};
-interface PageTexts { segmented: boolean; texts: Array<{ label: string; text: string }> }
+interface PageTexts { segmented: boolean; fenceRows: Map<number, string>; fenceUncertain: boolean; writableRows: Map<number, string>; texts: Array<{ label: string; text: string }> }
/** A segmented body's stored text, read segment by segment; null when it is over TEXT_MAX_BYTES (it proves nothing). */
async function segmentedText(db: BrainEngine, pageId: number): Promise<string | null> {
@@ -132,25 +141,31 @@ async function segmentedText(db: BrainEngine, pageId: number): Promise<string |
*/
async function pageTexts(db: BrainEngine, sourceId: string, slug: string): Promise<PageTexts> {
const snapshot = await db.readPageSnapshot(slug, { sourceId }).catch(() => null);
- if (!snapshot) return { segmented: false, texts: [] };
+ if (!snapshot) return { segmented: false, fenceRows: new Map(), fenceUncertain: false, writableRows: new Map(), texts: [] };
const texts: PageTexts['texts'] = [];
const segmented = isSegmentedPage(snapshot.page);
+ let fenceBody: string | null;
if (segmented) {
- const text = await segmentedText(db, snapshot.page.id);
- if (text !== null) texts.push({ label: slug, text });
+ fenceBody = await segmentedText(db, snapshot.page.id);
+ if (fenceBody !== null) texts.push({ label: slug, text: fenceBody });
} else {
+ fenceBody = snapshot.page.compiled_truth ?? '';
texts.push({ label: slug, text: [snapshot.page.compiled_truth ?? '', snapshot.page.timeline ?? ''].join('\n') });
}
const versions = await db.executeRaw<{ id: number | string; compiled_truth: string }>(`SELECT id, compiled_truth FROM page_versions
WHERE page_id=$1 AND compiled_truth IS NOT NULL ORDER BY id DESC LIMIT ${VERSIONS_PER_PAGE}`, [snapshot.page.id]);
for (const v of versions) texts.push({ label: `${slug}@version:${v.id}`, text: v.compiled_truth });
- return { segmented, texts };
+ const activeRows = (parsed: ReturnType<typeof parseFactsFence> | null) => new Map((parsed?.facts ?? []).filter(row => row.active).map(row => [row.rowNum, row.claim.trim()]));
+ const fence = fenceBody === null ? null : parseFactsFenceFor(fenceBody, { slug });
+ const writable = fenceBody === null ? null : parseFactsFence(fenceBody);
+ return { segmented, fenceRows: activeRows(fence), fenceUncertain: !fence || fence.warnings.length > 0,
+ writableRows: activeRows(writable?.warnings.length ? null : writable), texts };
}
/** Read-only: every active fact of these sources whose text holds a redaction marker, classified, in (source, page, id) order. */
export async function classifyRedactedFacts(db: BrainEngine, sourceIds: string[], opts: { ids?: number[] } = {}): Promise<RedactedFactCandidate[]> {
const rows = await db.executeRaw<FactRow>(`
- SELECT f.id, f.source_id, f.fact, f.visibility, f.entity_slug, f.row_num, f.source_markdown_slug, f.context
+ SELECT f.id, f.source_id, f.fact, f.visibility, f.entity_slug, f.row_num, f.source_markdown_slug, f.context, f.source
FROM facts f
WHERE f.source_id=ANY($1::text[]) AND f.expired_at IS NULL AND f.superseded_by IS NULL
AND f.fact LIKE '%[redacted%' AND ($2::bigint[] IS NULL OR f.id=ANY($2::bigint[]))
@@ -176,12 +191,16 @@ export async function classifyRedactedFacts(db: BrainEngine, sourceIds: string[]
}
}
const proved = !conflicting && proofs.size === 1;
- const held = proved && row.row_num !== null && row.source_markdown_slug !== null
- && (await sourceTexts(row.source_id, row.source_markdown_slug)).segmented;
+ const holder = row.row_num !== null && row.source_markdown_slug !== null ? await sourceTexts(row.source_id, row.source_markdown_slug) : null;
+ const extractor = (row.source ?? '').startsWith('cli:extract-conversation-facts');
+ const inFence = !!holder && (holder.fenceRows.get(Number(row.row_num)) === row.fact.trim() || holder.segmented && !extractor || extractor && holder.fenceUncertain);
+ const route = row.row_num === null || !inFence && extractor ? 'database' : 'fence';
+ const fenceGone = route === 'fence' && holder?.writableRows.get(Number(row.row_num)) !== row.fact.trim();
const [klass, reason]: [RedactedFactClass, string] = conflicting || proofs.size > 1 ? ['ambiguous', 'conflicting_sources']
- : held ? ['segmented', 'segmented_page'] : proved ? ['evidenced', 'restorable'] : ['unproven', read ? 'not_in_source' : 'no_source_page'];
+ : proved && holder?.segmented && route === 'fence' ? ['segmented', 'segmented_page'] : proved && fenceGone ? ['unproven', 'fence_row_missing']
+ : proved ? ['evidenced', 'restorable'] : ['unproven', read ? 'not_in_source' : 'no_source_page'];
const [restored, evidence] = proved ? [...proofs][0]! : [null, [...proofs.values()].flat()];
- candidates.push({ id: Number(row.id), source_id: row.source_id, slug: targetSlug(row), row_num: row.row_num === null ? null : Number(row.row_num),
+ candidates.push({ id: Number(row.id), source_id: row.source_id, slug: targetSlug(row), row_num: row.row_num === null ? null : Number(row.row_num), route,
class: klass, reason, fact: row.fact, restored, evidence, visibility: row.visibility, entity_slug: row.entity_slug,
source_markdown_slug: row.source_markdown_slug, fact_hash: factHash(row) });
}
@@ -194,6 +213,7 @@ const DETAILS: Record<string, string> = {
no_source_page: 'no page it came from exists any more, so nothing can prove the original words; left unchanged',
not_in_source: 'the words around the marker are not in any page it came from, so nothing proves the original words; left unchanged',
segmented_page: 'its row sits in the facts fence of a page stored in segments, and this repair rewrites a fence by publishing the whole body; left unchanged',
+ fence_row_missing: 'its row number names no active row with its text in its page\'s Facts fence, so neither the fence nor the row can be rewritten safely; left unchanged',
};
function previewCommand(scope: RepairScope, all: string[]): string {
@@ -206,7 +226,7 @@ async function activeSources(engine: BrainEngine): Promise<string[]> {
async function hashParts(engine: BrainEngine, scope: RepairScope, facts: RedactedFactCandidate[]) {
const [brain] = await engine.executeRaw<{ brain_id: string }>('SELECT brain_id FROM persistence_brain WHERE singleton=1').catch(() => []);
- return ['redacted-facts-v1', brain?.brain_id ?? 'host', scope.source_ids, facts.map(f => [f.id, f.class, f.reason, f.fact_hash, f.restored])];
+ return ['redacted-facts-v1', brain?.brain_id ?? 'host', scope.source_ids, facts.map(f => [f.id, f.class, f.reason, f.fact_hash, f.restored, f.route])];
}
function pageItem(page: RedactedFactsPage, hash: string, index: number, last: boolean): PageItem {
@@ -258,12 +278,12 @@ export const redactedFactsRepair: RepairHandler = {
const live = new Map((await classifyRedactedFacts(engine, [page.source_id], { ids: page.facts.map(f => f.id) })).map(c => [c.id, c]));
const unchanged = page.facts.filter(f => {
const current = live.get(f.id);
- return !!current && (['class', 'fact_hash', 'slug', 'row_num', 'restored'] as const).every(field => current[field] === f[field]);
+ return !!current && (['class', 'fact_hash', 'slug', 'row_num', 'route', 'restored'] as const).every(field => current[field] === f[field]);
});
const restored: number[] = [];
- const dbOnly = unchanged.filter(f => f.row_num === null);
+ const dbOnly = unchanged.filter(f => f.route === 'database');
if (dbOnly.length) restored.push(...await restoreDatabaseRows(engine, page, hash, dbOnly));
- const fenced = unchanged.filter(f => f.row_num !== null);
+ const fenced = unchanged.filter(f => f.route === 'fence');
if (fenced.length) restored.push(...await restoreFencedRows(ctx, entry, page, fenced));
if (last) await clearApprovedSet(engine, { command: 'redacted-facts', hash });
const changed = page.facts.filter(f => !restored.includes(f.id)).map(f => f.id);
@@ -272,8 +292,8 @@ export const redactedFactsRepair: RepairHandler = {
},
};
-/** One approved database-only rewrite: the row as previewed (text, visibility, entity, holding page) and its restored text. */
-export interface Restore { id: number; from: string; to: string; visibility: string; entity_slug: string | null; source_markdown_slug: string | null }
+/** One approved in-place rewrite: the row as previewed (text, visibility, entity, holding page, row number) and its restored text. */
+export interface Restore { id: number; from: string; to: string; visibility: string; entity_slug: string | null; source_markdown_slug: string | null; row_num?: number | null }
/**
* Rewrites each row still exactly as previewed. The compare runs in the
@@ -283,21 +303,22 @@ export interface Restore { id: number; from: string; to: string; visibility: str
*/
export const restoreRedactedRows = (tx: BrainEngine, sourceId: string, restores: Restore[]) => tx.executeRaw<{ id: number | string }>(`UPDATE facts f
SET fact=r.to_text, context=concat_ws(' | ', NULLIF(f.context,''), $3::text)
- FROM jsonb_to_recordset($2::text::jsonb) AS r(id bigint, from_text text, to_text text, visibility text, entity_slug text, source_markdown_slug text)
+ FROM jsonb_to_recordset($2::text::jsonb) AS r(id bigint, from_text text, to_text text, visibility text, entity_slug text, source_markdown_slug text, row_num int)
WHERE f.source_id=$1 AND f.id=r.id AND f.fact=r.from_text AND f.visibility=r.visibility AND f.entity_slug IS NOT DISTINCT FROM r.entity_slug
- AND f.source_markdown_slug IS NOT DISTINCT FROM r.source_markdown_slug AND f.expired_at IS NULL AND f.superseded_by IS NULL AND f.row_num IS NULL
+ AND f.source_markdown_slug IS NOT DISTINCT FROM r.source_markdown_slug AND f.expired_at IS NULL AND f.superseded_by IS NULL
+ AND f.row_num IS NOT DISTINCT FROM r.row_num AND (r.row_num IS NULL OR f.source LIKE 'cli:extract-conversation-facts%')
RETURNING f.id`,
[sourceId, JSON.stringify(restores.map(r => ({ id: r.id, from_text: r.from, to_text: r.to, visibility: r.visibility, entity_slug: r.entity_slug,
- source_markdown_slug: r.source_markdown_slug }))), RESTORED_CONTEXT]);
+ source_markdown_slug: r.source_markdown_slug, row_num: r.row_num ?? null }))), RESTORED_CONTEXT]);
/**
- * Database-only rows: on a managed brain one database-only maintenance
+ * Database-only and conversation-extractor rows: on a managed brain one database-only maintenance
* request on the page key (owner-checked, with a publication receipt); on an
* unmanaged brain one transaction.
*/
async function restoreDatabaseRows(engine: BrainEngine, page: RedactedFactsPage, hash: string, facts: RedactedFactCandidate[]): Promise<number[]> {
const restores: Restore[] = facts.map(f => ({ id: f.id, from: f.fact, to: f.restored!, visibility: f.visibility, entity_slug: f.entity_slug,
- source_markdown_slug: f.source_markdown_slug }));
+ source_markdown_slug: f.source_markdown_slug, row_num: f.row_num }));
if (!await managedPersistenceEnabled(engine)) {
return (await maintenanceTransaction(engine, tx => restoreRedactedRows(tx, page.source_id, restores))).map(row => Number(row.id));
}
@@ -310,13 +331,14 @@ async function restoreDatabaseRows(engine: BrainEngine, page: RedactedFactsPage,
return (receipt.restored as number[] | undefined) ?? [];
}
-/** Preparer for `managed_maintenance_restore_redacted_facts`: rewrites the named database-only rows still holding their previewed text. */
+/** Preparer for `managed_maintenance_restore_redacted_facts`: rewrites the named database-only and conversation-extractor rows still holding their previewed text. */
export async function prepareRedactedFactsRestore(engine: BrainEngine, row: WriteRequest): Promise<PreparedMutation> {
const restores = (row.intent as { restores?: unknown } | null)?.restores;
const text = (v: unknown) => typeof v === 'string';
const valid = Array.isArray(restores) && restores.every(r => !!r && typeof r === 'object' && Number.isSafeInteger((r as Restore).id)
&& text((r as Restore).from) && text((r as Restore).to) && text((r as Restore).visibility)
- && ((r as Restore).entity_slug === null || text((r as Restore).entity_slug)) && ((r as Restore).source_markdown_slug === null || text((r as Restore).source_markdown_slug)));
+ && ((r as Restore).entity_slug === null || text((r as Restore).entity_slug)) && ((r as Restore).source_markdown_slug === null || text((r as Restore).source_markdown_slug))
+ && ((r as Restore).row_num == null || Number.isSafeInteger((r as Restore).row_num)));
if (!valid) {
throw opError('invalid_params', 'The redacted facts restore intent does not name its facts.',
`Request ${row.request_id} in source ${row.source_id} does not name the facts to restore, so nothing changed. Preview the repair again and apply the new preview after the user approves.`,
diff --git a/test/repair-redacted-facts.test.ts b/test/repair-redacted-facts.test.ts
index a8525754eb..93c1a30dec 100644
--- a/test/repair-redacted-facts.test.ts
+++ b/test/repair-redacted-facts.test.ts
@@ -57,6 +57,12 @@ async function refusal(run: () => Promise<unknown>): Promise<OperationError> {
const factText = async (engine: BrainEngine, id: number) =>
(await engine.executeRaw<{ fact: string; active: boolean }>('SELECT fact, expired_at IS NULL AS active FROM facts WHERE id=$1', [id]))[0];
+const CONVERSATION = 'conversations/planning-call';
+const insertExtractorFact = async (db: BrainEngine, fact: string, rowNum: number) => Number((await db.executeRaw<{ id: number }>(`INSERT INTO facts
+ (source_id, entity_slug, fact, kind, visibility, source, context, row_num, source_markdown_slug)
+ VALUES ('default', $1, $2, 'fact', 'private', 'cli:extract-conversation-facts', $3, $4, $3) RETURNING id`, [ACME, fact, CONVERSATION, rowNum]))[0].id);
+const extractorRow = async (engine: BrainEngine, id: number) => (await engine.executeRaw<{ fact: string; row_num: number; source_markdown_slug: string; active: boolean }>(
+ 'SELECT fact, row_num::int AS row_num, source_markdown_slug, expired_at IS NULL AS active FROM facts WHERE id=$1', [id]))[0];
const putPage = (ctx: OperationContext, slug: string, content: string) =>
submitPageMutation(ctx, { operation: 'put_page', params: { request_id: randomUUID(), slug, source_id: 'default', content } });
@@ -155,6 +161,22 @@ ${SOURCE_TEXT}
expect((await factText(engine, id)).fact).toBe(EXOS_DAMAGED);
}, { databaseUrl });
}, 180_000);
+
+ test('#8820: a conversation-extractor row (legacy row number, no Facts fence) is restored in place; its page is unchanged', async () => {
+ await managedBrain(async ({ engine, ctx }) => {
+ await putPage(ctx, CONVERSATION, `---\ntype: note\ntitle: Planning call\n---\n${SOURCE_TEXT}\n`);
+ const before = (await engine.readPageSnapshot(CONVERSATION, { sourceId: 'default' }))!;
+ const id = await engine.transaction(tx => withCoordinatedWrite(tx, ['default'], () => insertExtractorFact(tx, ABS_DAMAGED, 3), TEST_WRITE_ATTRIBUTION));
+ const preview = await repair(engine, []);
+ expect(preview.results[0].residuals).toEqual({ evidenced: 1, ambiguous: 0, unproven: 0, segmented: 0 });
+ expect((await repair(engine, applyArgs(preview))).results[0].outcomes).toEqual({ restored: 1 });
+ expect(await extractorRow(engine, id)).toEqual({ fact: ABS_RESTORED, row_num: 3, source_markdown_slug: CONVERSATION, active: true });
+ const after = (await engine.readPageSnapshot(CONVERSATION, { sourceId: 'default' }))!;
+ expect(after.revision).toBe(before.revision);
+ expect(after.page.compiled_truth).toBe(before.page.compiled_truth);
+ expect((await factsRedactionMarkerCheck(engine)).status).toBe('ok');
+ }, { databaseUrl });
+ }, 180_000);
});
describe(`gbrain repair redacted-facts, unmanaged (${backend})`, () => {
@@ -204,6 +226,75 @@ ${SOURCE_TEXT}
await engine.executeRaw('DELETE FROM facts WHERE id=$1', [fenced]);
}, 120_000);
+ test('#8820: a conversation-extractor row on a segmented page with no Facts fence is restored in place', async () => {
+ const conversation = 'conversations/segmented-call';
+ await createSegmentedPage(engine, { slug: conversation, truth: `${'Filler line.\n'.repeat(90_000)}${SOURCE_TEXT}\n` });
+ const id = Number((await engine.executeRaw<{ id: number }>(`INSERT INTO facts
+ (source_id, entity_slug, fact, kind, visibility, source, context, row_num, source_markdown_slug)
+ VALUES ('default', $1, $2, 'fact', 'private', 'cli:extract-conversation-facts', $3, 3, $3) RETURNING id`, [ACME, ABS_DAMAGED, conversation]))[0].id);
+ const preview = await repair(engine, []);
+ const listed = Object.fromEntries(preview.results[0].listing!.map(l => [Number(l.item.split('#')[1]), l.class]));
+ expect(listed[id]).toBe('evidenced');
+ expect((await repair(engine, applyArgs(preview))).results[0].outcomes).toEqual({ restored: 1 });
+ expect(await extractorRow(engine, id)).toEqual({ fact: ABS_RESTORED, row_num: 3, source_markdown_slug: conversation, active: true });
+ await engine.executeRaw('DELETE FROM facts WHERE id=$1', [id]);
+ }, 120_000);
+
+ test('#8820: an extractor row whose text sits in a segmented holding page\'s entity-column fence stays on the fence route and is left unchanged', async () => {
+ const hold = 'memory/unattributed';
+ const conversation = 'conversations/held-call';
+ await engine.putPage(conversation, { type: 'note', title: 'Held call', compiled_truth: SOURCE_TEXT }, { sourceId: 'default' });
+ await createSegmentedPage(engine, { slug: hold, truth: [`${'Filler line.\n'.repeat(90_000)}`, '## Facts', '', '<!--- gbrain:facts:begin -->',
+ '| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context | entity |',
+ '|---|---|---|---|---|---|---|---|---|---|---|',
+ `| 2 | ${ABS_DAMAGED} | fact | 1.0 | private | medium | 2026-11-11 | | chat 2026-11-11 | | ${ACME} |`,
+ '<!--- gbrain:facts:end -->', ''].join('\n') });
+ const id = Number((await engine.executeRaw<{ id: number }>(`INSERT INTO facts
+ (source_id, entity_slug, fact, kind, visibility, source, context, row_num, source_markdown_slug)
+ VALUES ('default', $1, $2, 'fact', 'private', 'cli:extract-conversation-facts', $3, 2, $4) RETURNING id`, [ACME, ABS_DAMAGED, conversation, hold]))[0].id);
+ const preview = await repair(engine, []);
+ const listed = Object.fromEntries(preview.results[0].listing!.map(l => [Number(l.item.split('#')[1]), l.class]));
+ expect(listed[id]).toBe('segmented');
+ expect(preview.results[0].residuals.evidenced).toBe(0);
+ expect((await factText(engine, id)).fact).toBe(ABS_DAMAGED);
+ await engine.executeRaw('DELETE FROM facts WHERE id=$1', [id]);
+ }, 120_000);
+
+ test('a non-extractor row in an inline holding page\'s entity-column fence is not listed as restorable, because the fence writer cannot rewrite that fence', async () => {
+ const hold = 'memory/unattributed';
+ const conversation = 'conversations/held-inline-call';
+ await engine.executeRaw("DELETE FROM pages WHERE slug=$1 AND source_id='default'", [hold]);
+ await engine.putPage(conversation, { type: 'note', title: 'Held inline call', compiled_truth: SOURCE_TEXT }, { sourceId: 'default' });
+ await engine.putPage(hold, { type: 'note', title: 'Unattributed', compiled_truth: ['## Facts', '', '<!--- gbrain:facts:begin -->',
+ '| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context | entity |',
+ '|---|---|---|---|---|---|---|---|---|---|---|',
+ `| 2 | ${ABS_DAMAGED} | fact | 1.0 | private | medium | 2026-11-11 | | chat 2026-11-11 | | ${ACME} |`,
+ '<!--- gbrain:facts:end -->', ''].join('\n') }, { sourceId: 'default' });
+ const id = Number((await engine.executeRaw<{ id: number }>(`INSERT INTO facts
+ (source_id, entity_slug, fact, kind, visibility, source, context, row_num, source_markdown_slug)
+ VALUES ('default', $1, $2, 'fact', 'private', 'sync:import', $3, 2, $4) RETURNING id`, [ACME, ABS_DAMAGED, conversation, hold]))[0].id);
+ const preview = await repair(engine, []);
+ const listed = Object.fromEntries(preview.results[0].listing!.map(l => [Number(l.item.split('#')[1]), l.class]));
+ expect(listed[id]).toBe('unproven');
+ expect(preview.results[0].residuals.evidenced).toBe(0);
+ await engine.executeRaw('DELETE FROM facts WHERE id=$1', [id]);
+ }, 120_000);
+
+ test('#8820: a conversation-extractor row is restored in place; a fenced row number whose fence row is gone is not', async () => {
+ await engine.putPage(CONVERSATION, { type: 'note', title: 'Planning call', compiled_truth: SOURCE_TEXT }, { sourceId: 'default' });
+ const id = await insertExtractorFact(engine, ABS_DAMAGED, 3);
+ const other = Number((await engine.executeRaw<{ id: number }>(`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, source, row_num, source_markdown_slug)
+ VALUES ('default', NULL, $1, 'fact', 'private', 'mcp:put_page', 4, $2) RETURNING id`, [EXOS_DAMAGED, CONVERSATION]))[0].id);
+ const preview = await repair(engine, []);
+ const listed = Object.fromEntries(preview.results[0].listing!.map(l => [Number(l.item.split('#')[1]), l.class]));
+ expect(listed[id]).toBe('evidenced');
+ expect(listed[other]).not.toBe('evidenced');
+ expect((await repair(engine, applyArgs(preview))).results[0].outcomes).toEqual({ restored: 1 });
+ expect(await extractorRow(engine, id)).toEqual({ fact: ABS_RESTORED, row_num: 3, source_markdown_slug: CONVERSATION, active: true });
+ expect((await factText(engine, other)).fact).toBe(EXOS_DAMAGED);
+ await engine.executeRaw('DELETE FROM facts WHERE id = ANY($1::bigint[])', [[id, other]]);
+ }, 120_000);
+
test('the publication rewrites a row only while it is exactly as previewed (a change after the recheck wins)', async () => {
const id = Number((await engine.executeRaw<{ id: number }>(`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, source, context)
VALUES ('default', $1, $2, 'fact', 'private', 'sync:import', $3) RETURNING id`, [ACME, EXOS_DAMAGED, NOTES]))[0].id);