Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
repair secret-spans: resume in page-id order, refuse colliding timeline rows and ontology values, match rows in memory, fingerprint field hashes
garrytan4748a0b6d54c
diff --git a/docs/guides/repair.md b/docs/guides/repair.mdindex 3b02a5264d..02cf84b469 100644--- a/docs/guides/repair.md+++ b/docs/guides/repair.md@@ -172,7 +172,7 @@ work behind exits 0, so scripts should also check `results[].complete`.| `missing-tables` | `schema_missing_tables` | Explicit-only, preview-bound. Recreates, empty, a rebuildable cache table (`query_cache`, `code_traversal_cache`) that the applied migrations created but the brain no longer has, on a brain whose schema version already covers its creating migration, so `apply-migrations` never recreates it (#6911). DDL only (`IF NOT EXISTS`), no journal admission, brain-wide. See [missing tables](#missing-tables). | `user_data_tables`: a missing table that holds user data; this repair never recreates it, so restore its rows from a backup first. `other_tables`: a missing table that holds no user data; `gbrain init --migrate-only` recreates it if the base schema defines it. || `url-clone-fences` | none (`gbrain sync --source <id>` stops at the pull with `pull_blocked_local_changes`) | Explicit-only, preview-bound, destructive consent. Clears the `## Facts` fence rows an older gbrain appended to files of a clone it owns (`gbrain sources add --url`), which block every `git pull --ff-only` that touches those files (#6946). The preview proves each changed file differs from HEAD only by appended fence rows; `--apply --expect <hash> --yes` backs each file up, keeps its rows database-only and restores the file from HEAD. Unmanaged brains only. See [URL clone fences](#url-clone-fences). | `non_fence_edit`, `unparsed_fence`, `staged_change`, `deleted`, `not_in_head`, `unreadable_checkout`: a file with any other difference from HEAD, named with why; a file whose HEAD would not import as its page (`visibility_noncanonical` and the other import holds): apply keeps its rows database-only and restores the file so the clone can pull, but does not import HEAD and reports `held`; files that changed after the preview (`changed_since_preview`). || `segmented-mirror` | none (doctor does not read segmented files; run the preview once if the brain ran a version before v0.60.299.0) | Explicit-only, preview-bound, destructive consent. Rewrites the canonical file of a page stored in segments that an older binary's withdrawal mirror passed by (`page_body_segmented`), so the file still holds a row `forget`, `forget_fact` or `purge_fact` removed from the database. The file becomes the page's canonical render (`gbrain export`'s bytes), written by the withdrawal mirror and committed by its Git effect. See [segmented mirror](#segmented-mirror). | `file_database_drift`: the file also differs from the database for another reason; it is listed held and never rewritten. `not_owner_host`: the preview ran off the source's owner host and read none of its files. |-| `secret-spans` | none (the operator names the pages) | Explicit-only, preview-bound, trusted local CLI. Replaces the credentials the secret scanner finds in the database copies of the named pages with `<REDACTED:rule>` markers and changes nothing else: title, frontmatter values (never keys), the body (inline or segmented), timeline, fact, take and link rows, raw data; chunks holding a value are rebuilt. Database only: no file write, no Git commit, and a sync hold stays. `--source <id> --slug <slug>... [--rules <list>]`, then `--apply --expect <hash>`. See [secret spans](#secret-spans). | `clean` (no match), `refused` (`page_not_found`, `page_kind_unsupported`, `page_too_large`, `proof_failed`); pages that moved after the preview (`preview_changed`, `changed_since_preview`). |+| `secret-spans` | none (the operator names the pages) | Explicit-only, preview-bound, trusted local CLI. Replaces the credentials the secret scanner finds in the database copies of the named pages with `<REDACTED:rule>` markers and changes nothing else: title, frontmatter values (never keys), the body (inline or segmented), timeline, fact, take and link rows, raw data; chunks holding a value are rebuilt. Database only: no file write, no Git commit, and a sync hold stays. `--source <id> --slug <slug>... [--rules <list>]`, then `--apply --expect <hash>`. See [secret spans](#secret-spans). | `clean` (no match), `refused` (`page_not_found`, `page_kind_unsupported`, `page_too_large`, `proof_failed`, `fact_value_unsupported`, `timeline_rows_collide`); pages that moved after the preview (`preview_changed`, `changed_since_preview`). |Timeline rows that an earlier version of a page produced and its current textno longer has are removals, not history, so `timeline` neither counts nor@@ -1234,12 +1234,15 @@ replaced too, and text that is already only markers is left alone, so a secondpreview of an applied page is empty.The preview lists each page with its spans per rule and, for each changed-field, the old and new SHA-256 and a hash of the span offsets. It never prints a-value. It also counts the retained page versions, which are kept as history and-not rewritten. A page is `refused` when it does not exist, is a code or image-page, has a segmented body over 24 MiB, or when the planned spans and the-scanner's own redaction of the text disagree (`proof_failed`); nothing on a-refused page changes.+field, the first 64 bits of its old and new SHA-256 and a hash of the span+offsets. It never prints a value. It also counts the retained page versions,+which are kept as history and not rewritten. A page is `refused`, and nothing+on it changes, when it does not exist, is a code or image page, has a segmented+body over 24 MiB, when the planned spans and the scanner's own redaction of the+text disagree (`proof_failed`), when an ontology fact's resolved value holds a+match (`fact_value_unsupported`: the value keys the fact's dedup; retire the+fact with `gbrain forget`), or when two timeline rows of one day would read the+same once redacted (`timeline_rows_collide`: delete the duplicate row first).The apply checks every previewed page again before writing anything andrefuses `preview_changed` if one moved; preview again. Each page is then one@@ -1249,12 +1252,16 @@ brains included). Trust tiers are kept. Chunks that held a value are deletedand rebuilt from the redacted body, keeping the vectors of unchanged chunks;changed chunks are embedded unless `--no-embed` or the page carries`embed_skip`. Fact and take vectors of a rewritten row are cleared. A-value-free receipt per page (field hashes and span offsets) is stored with the-write and appended to the `secret-spans` audit log, and the apply re-scans each+value-free receipt per page (field fingerprints and span offsets) is stored+with the write, where a resumed apply finds it for the next 7 days, and+appended to the `secret-spans` audit log, and the apply re-scans eachpage and reports any match left (`residual`, expected 0).There is no undo: the old value is gone from the live copies. Page versions,-write receipts and Git history are kept and still hold it. It runs only from+write receipts and Git history are kept and still hold it. On a brain with a+repository, redact the file too: a later import of the unredacted file (a full+sync, `gbrain repair file-drift --prefer file`, an edit) writes the value back+into the database. It runs only fromthe trusted local CLI on the brain host, only when named, and never from `--all`, doctor's remediation plan, the maintenancecycle or an upgrade.diff --git a/src/core/repair/secret-spans.ts b/src/core/repair/secret-spans.tsindex 9e3280bab7..978ccb0340 100644--- a/src/core/repair/secret-spans.ts+++ b/src/core/repair/secret-spans.ts@@ -10,7 +10,8 @@* are never touched), its body (inline columns, or the live generation of a* segmented body, restaged and flipped), its timeline rows (page or event* page), the fact rows it holds or that name it with no holding page, its take- * rows, its outgoing link contexts and the string leaves of its raw data. Chunk+ * rows, its outgoing link contexts and the string leaves of its raw data (an+ * ontology fact's resolved value is scanned, and a match refuses the page). Chunk* rows that hold a value are deleted and the projection is rebuilt from the* redacted body (vectors of unchanged chunk texts are kept); derivation rows and* view digests that may hold one are deleted. Fact and take vectors of a@@ -73,23 +74,27 @@ interface PageRow {}type Refusal = 'page_not_found' | 'page_kind_unsupported' | 'page_too_large';+type PlanRefusal = 'proof_failed' | 'fact_value_unsupported' | 'timeline_rows_collide';/** `text`: the body's truth and timeline regions as stored (inline columns, or a segmented body read in full). */interface Read { page: PageRow; fields: Field[]; text: { truth: string; tail: string } }interface Planned { read: Read; results: FieldRedaction[]; digest: string }interface Selection { source_ids: string[]; slugs: string[]; rules: string[] }-interface FieldSummary { key: string; spans: number; old_sha256: string; new_sha256: string; spans_sha256: string }+/** `old_fp` / `new_fp`: the first 64 bits of the field's SHA-256, so a field that is only a short value is not printed as its full hash. */+interface FieldSummary { key: string; spans: number; old_fp: string; new_fp: string; spans_sha256: string }interface PlannedPage {source_id: string; slug: string; id: number; revision: string; generation: string; storage: string; digest: string;rules: Record<string, number>; echo: number; fields: FieldSummary[]; versions: number;}interface SpanItem extends RepairItem { page: PlannedPage; hash: string; rules: string[]; last: boolean }-const REFUSAL_TEXT: Record<Refusal | 'proof_failed', string> = {+const REFUSAL_TEXT: Record<Refusal | PlanRefusal, string> = {page_not_found: 'no live page with this slug in the source',page_kind_unsupported: 'only markdown pages are redacted here (code and image pages index the file itself)',page_too_large: `the segmented body is over the ${PAGE_MAX_BYTES / 1024 ** 2} MiB single-string ceiling this repair reads`,proof_failed: 'the planned spans and the scanner\'s own redaction disagree, so nothing on the page is rewritten',+ fact_value_unsupported: 'an ontology fact\'s resolved value holds a match; its value_hash keys the fact\'s dedup, so this repair does not rewrite it (retire the fact with gbrain forget, then preview again)',+ timeline_rows_collide: 'two timeline rows of one day would read the same once redacted, which their dedup key refuses; delete the duplicate row, then preview again',};function previewCommand(selection: Selection): string {@@ -149,7 +154,7 @@ async function readFields(engine: BrainEngine, sourceId: string, slug: string):};rows('timeline_entries', ['summary', 'detail', 'source'], await engine.executeRaw('SELECT id,summary,detail,source FROM timeline_entries WHERE page_id=$1 OR event_page_id=$1 ORDER BY id', [id]));- rows('facts', ['fact', 'context'], await engine.executeRaw(`SELECT id,fact,context FROM facts+ rows('facts', ['fact', 'context', 'value'], await engine.executeRaw(`SELECT id,fact,context,value FROM factsWHERE source_id=$1 AND (source_markdown_slug=$2 OR (source_markdown_slug IS NULL AND entity_slug=$2)) ORDER BY id`, [sourceId, slug]));rows('takes', ['claim', 'source', 'resolved_source'], await engine.executeRaw('SELECT id,claim,source,resolved_source FROM takes WHERE page_id=$1 ORDER BY id', [id]));rows('links', ['context'], await engine.executeRaw('SELECT id,context FROM links WHERE from_page_id=$1 ORDER BY id', [id]));@@ -167,6 +172,27 @@ function planPage(read: Read, rules: SecretRuleSet): Planned {results.map(result => [result.key, result.old_sha256, result.spans_sha256])]) };}+/**+ * Why a planned page cannot be rewritten as planned, or null. A timeline row's dedup key (page, date, summary, source)+ * is checked against every row of the pages its rows belong to, because the event-page rows sit on other pages.+ */+async function planRefusal(engine: BrainEngine, planned: Planned): Promise<PlanRefusal | null> {+ if (planned.results.some(result => !result.proven)) return 'proof_failed';+ const changed = changedOf(planned);+ if (changed.some(result => /^facts\/\d+\/value$/.test(result.key))) return 'fact_value_unsupported';+ if (!changed.some(result => result.key.startsWith('timeline_entries/'))) return null;+ const next = new Map(planned.results.map(result => [result.key, result.next]));+ const rows = await engine.executeRaw<{ id: number | string; page_id: number | string; date: string; summary: string; source: string }>(`SELECT id,page_id,date::text AS date,summary,source+ FROM timeline_entries WHERE page_id IN (SELECT page_id FROM timeline_entries WHERE page_id=$1 OR event_page_id=$1)`, [planned.read.page.id]);+ const keys = new Set<string>();+ for (const row of rows) {+ const key = JSON.stringify([String(row.page_id), row.date, next.get(`timeline_entries/${row.id}/summary`) ?? row.summary, next.get(`timeline_entries/${row.id}/source`) ?? row.source]);+ if (keys.has(key)) return 'timeline_rows_collide';+ keys.add(key);+ }+ return null;+}+const changedOf = (planned: Planned) => planned.results.filter(result => result.spans.length);function summaryOf(planned: Planned, versions: number): PlannedPage {@@ -175,7 +201,7 @@ function summaryOf(planned: Planned, versions: number): PlannedPage {const counts = spanCounts(changed.flatMap(result => result.spans));return { source_id: page.source_id, slug: page.slug, id: page.id, revision: page.revision, generation: page.generation, storage: page.body_storage,digest: planned.digest, rules: counts.rules, echo: counts.echo, versions,- fields: changed.map(result => ({ key: result.key, spans: result.spans.length, old_sha256: result.old_sha256, new_sha256: result.new_sha256, spans_sha256: result.spans_sha256 })) };+ fields: changed.map(result => ({ key: result.key, spans: result.spans.length, old_fp: result.old_sha256.slice(0, 16), new_fp: result.new_sha256.slice(0, 16), spans_sha256: result.spans_sha256 })) };}const ruleText = (rules: Record<string, number>) => Object.entries(rules).sort().map(([rule, n]) => `${rule}=${n}`).join(', ');@@ -250,15 +276,22 @@ async function writePage(tx: BrainEngine, planned: Planned): Promise<{ unsealed:await tx.executeRaw('UPDATE pages SET knowledge_revision=gen_random_uuid() WHERE id=$1', [page.id]);if (page.body_storage === 'segmented') await sealSegmentedPage(tx, page.id);}+ // Rows holding a value are matched here, never in SQL, so no secret is ever sent as a statement parameter (which a+ // server's slow-statement log would keep).const values = valuesOf(planned);+ const holds = (text: string | null | undefined) => !!text && values.some(value => text.includes(value));if (page.body_storage !== 'segmented') {- await tx.executeRaw(`DELETE FROM content_chunks c WHERE c.page_id=$1 AND EXISTS (SELECT 1 FROM unnest($2::text[]) v- WHERE strpos(c.chunk_text,v)>0 OR strpos(COALESCE(c.doc_comment,''),v)>0)`, [page.id, values]);+ const chunks = await tx.executeRaw<{ id: number | string; chunk_text: string; doc_comment: string | null }>(+ 'SELECT id,chunk_text,doc_comment FROM content_chunks WHERE page_id=$1 -- chunk-visibility-ok: every stored row of the page is checked for a value', [page.id]);+ const doomed = chunks.filter(chunk => holds(chunk.chunk_text) || holds(chunk.doc_comment)).map(chunk => Number(chunk.id));+ if (doomed.length) await tx.executeRaw('DELETE FROM content_chunks WHERE id=ANY($1::int[])', [doomed]);await queuePageProjection(tx, page.source_id, page.slug, 'secret_spans');}- await tx.executeRaw(`DELETE FROM page_segment_derivations d WHERE d.page_id=$1 AND d.output IS NOT NULL- AND EXISTS (SELECT 1 FROM unnest($2::text[]) v WHERE strpos(d.output::text,v)>0)`,- [page.id, [...new Set([...values, ...values.map(value => JSON.stringify(value).slice(1, -1))])]]);+ const derivations = await tx.executeRaw<{ pipeline: string; occurrence_id: number | string; output: unknown }>(+ 'SELECT pipeline,occurrence_id,output FROM page_segment_derivations WHERE page_id=$1 AND output IS NOT NULL', [page.id]);+ for (const row of derivations.filter(row => leaves(row.output).some(leaf => holds(leaf.text)))) {+ await tx.executeRaw('DELETE FROM page_segment_derivations WHERE page_id=$1 AND pipeline=$2 AND occurrence_id=$3', [page.id, row.pipeline, row.occurrence_id]);+ }await tx.executeRaw('DELETE FROM page_view_digests WHERE page_id=$1', [page.id]);return { unsealed };}@@ -337,9 +370,10 @@ export const secretSpansRepair: RepairHandler = {}const plan = planPage(read, rules);const label = `${sourceId}:${slug} (revision ${read.page.revision})`;- if (plan.results.some(result => !result.proven)) {+ const refusal = await planRefusal(engine, plan);+ if (refusal) {residuals.refused!++;- listing.push({ item: label, class: 'refused', detail: `proof_failed: ${REFUSAL_TEXT.proof_failed}` });+ listing.push({ item: label, class: 'refused', detail: `${refusal}: ${REFUSAL_TEXT[refusal]}` });continue;}if (!changedOf(plan).length) { residuals.clean!++; continue; }@@ -351,6 +385,8 @@ export const secretSpansRepair: RepairHandler = {residuals.retained_versions! += page.versions;listing.push({ item: label, class: 'redact', detail: changeDetail(page) });}+ // Items run in cursor (page id) order, so a resumed apply after --limit or a stop continues with the rest.+ planned.sort((a, b) => a.id - b.id);const hash = await hashOf(engine, scope, selection, planned);if (planned.length) await saveApprovedSet<PlannedPage & { selection: Selection }>(engine, { command: 'secret-spans', hash }, planned.map(page => ({ ...page, selection })));return { items: planned.map((page, index) => item(page, hash, selection.rules, index === planned.length - 1)), preview_hash: hash, residuals, listing,@@ -391,11 +427,13 @@ export const secretSpansRepair: RepairHandler = {if (typeof read === 'string') return { refused: read };const planned = planPage(read, rules);if (read.page.id !== page.id || planned.digest !== page.digest) return { refused: 'changed_since_preview' };+ const refusal = await planRefusal(tx, planned);+ if (refusal) return { refused: refusal };const { unsealed } = await writePage(tx, planned);const [after] = await tx.executeRaw<{ revision: string }>('SELECT knowledge_revision::text AS revision FROM pages WHERE id=$1', [page.id]);const receipt = { kind: 'secret-spans', hash, source_id: page.source_id, slug: page.slug, page_id: page.id, revision_before: page.revision,- revision_after: after?.revision ?? null, rules: rules.rules, fields: changedOf(planned).map(result => ({ key: result.key, old_sha256: result.old_sha256,- new_sha256: result.new_sha256, spans_sha256: result.spans_sha256, spans: result.spans.map(span => [span.start, span.end, span.rule, span.echo ? 1 : 0]) })) };+ revision_after: after?.revision ?? null, rules: rules.rules, fields: changedOf(planned).map(result => ({ key: result.key, old_fp: result.old_sha256.slice(0, 16),+ new_fp: result.new_sha256.slice(0, 16), spans_sha256: result.spans_sha256, spans: result.spans.map(span => [span.start, span.end, span.rule, span.echo ? 1 : 0]) })) };await tx.executeRaw(`INSERT INTO op_checkpoints(op,fingerprint,completed_keys,updated_at) VALUES ($1,$2,$3::text::jsonb,now())ON CONFLICT (op,fingerprint) DO UPDATE SET completed_keys=EXCLUDED.completed_keys,updated_at=now()`,[SECRET_SPANS_RECEIPT_OP, receiptKey(hash, page.id), JSON.stringify([receipt])]);@@ -423,7 +461,7 @@ export const secretSpansRepair: RepairHandler = {},render(details) {return ((details.pages ?? []) as PlannedPage[]).flatMap(page => [` ${page.source_id}:${page.slug} (${page.storage}): ${changeDetail(page)}`,- ...page.fields.map(field => ` ${field.key}: ${field.spans} span(s), sha256 ${field.old_sha256.slice(0, 12)} -> ${field.new_sha256.slice(0, 12)}, spans ${field.spans_sha256.slice(0, 12)}`)]);+ ...page.fields.map(field => ` ${field.key}: ${field.spans} span(s), sha256 ${field.old_fp} -> ${field.new_fp}, spans ${field.spans_sha256.slice(0, 12)}`)]);},async report(ctx, scope, result, opts) {if (result.mode !== 'apply' || !opts.slugs?.length || scope.source_ids.length !== 1) return {};diff --git a/test/repair-secret-spans.test.ts b/test/repair-secret-spans.test.tsindex 36fbe95167..5774056fd3 100644--- a/test/repair-secret-spans.test.ts+++ b/test/repair-secret-spans.test.ts@@ -209,6 +209,41 @@ describe('GST-327 gbrain repair secret-spans on a repo-less brain', () => {}), 300_000);});+describe('GST-327 gbrain repair secret-spans: resume and refusals', () => {+ test('an apply stopped by --limit resumes in page-id order and finishes; colliding timeline rows and ontology values refuse their page', () => withEnv({ GBRAIN_HOME: home }, async () => {+ const { engine } = repoLess[0]!;+ const AWS2 = 'AKIA' + 'EXAMPLENOTREAL01';+ // Imported first, so its id is lower although its slug sorts last.+ await importFromContent(engine, 'notes/z-old-example', `---\ntype: note\ntitle: Old\n---\n\nOld key ${AWS}.\n`, { noEmbed: true, sourceId: 'default' });+ await importFromContent(engine, 'notes/a-new-example', `---\ntype: note\ntitle: New\n---\n\nNew key ${AWS2}.\n`, { noEmbed: true, sourceId: 'default' });+ await importFromContent(engine, 'notes/dup-example', '---\ntype: note\ntitle: Dup\n---\n\nNo key in the body.\n', { noEmbed: true, sourceId: 'default' });+ await importFromContent(engine, 'notes/value-example', '---\ntype: note\ntitle: Value\n---\n\nNo key in the body.\n', { noEmbed: true, sourceId: 'default' });+ const dup = await pageRow(engine, 'notes/dup-example');+ await engine.executeRaw("INSERT INTO timeline_entries(page_id,date,source,summary,detail) VALUES ($1,'2024-05-01','import',$2,''),($1,'2024-05-01','import',$3,'')",+ [dup.id, `Rotated ${AWS}`, `Rotated ${AWS2}`]);+ await engine.executeRaw(`INSERT INTO facts(source_id,entity_slug,fact,value,source,kind) VALUES ('default','notes/value-example','Credential on file',$1,'test','fact')`, [AWS]);+ const scope = await resolveRepairScope(engine, 'default');+ const slugs = ['notes/a-new-example', 'notes/z-old-example', 'notes/dup-example', 'notes/value-example'];+ const timelineBefore = await engine.executeRaw('SELECT id,summary FROM timeline_entries WHERE page_id=$1 ORDER BY id', [dup.id]);++ const preview = await (await repairRunner(engine, { apply: false, logger, noEmbed: true })).run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs });+ expect(preview.listing!.map(entry => [entry.class, entry.item.split(' ')[0], entry.detail?.split(':')[0]])).toEqual([+ ['redact', 'default:notes/a-new-example', '1 span(s) in 1 field(s)'], ['refused', 'default:notes/dup-example', 'timeline_rows_collide'],+ ['refused', 'default:notes/value-example', 'fact_value_unsupported'], ['redact', 'default:notes/z-old-example', '1 span(s) in 1 field(s)']]);+ const apply = (limit?: number) => repairRunner(engine, { apply: true, logger, noEmbed: true })+ .then(runner => runner.run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs, expect: preview.preview_hash, ...(limit ? { limit } : {}) }));+ const first = await apply(1);+ expect(first.applied).toBe(1);+ expect((await pageRow(engine, 'notes/z-old-example')).compiled_truth).not.toContain(AWS);+ const rest = await apply();+ expect(rest.applied).toBe(1);+ expect(rest.complete).toBe(true);+ expect((await pageRow(engine, 'notes/a-new-example')).compiled_truth).not.toContain(AWS2);+ expect(await engine.executeRaw('SELECT id,summary FROM timeline_entries WHERE page_id=$1 ORDER BY id', [dup.id])).toEqual(timelineBefore);+ expect(await engine.executeRaw("SELECT value FROM facts WHERE entity_slug='notes/value-example'")).toEqual([{ value: AWS }]);+ }), 120_000);+});+for (const backend of testBackends()) describe(`GST-327 gbrain repair secret-spans on a managed brain (${backend})`, () => {test('a held page is redacted in the database; its file, its hold and Git stay as they are', () => managedBrain(async ({ engine, ctx, root }) => {const slug = 'notes/held-example';