Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

repair secret-spans: resume in page-id order, refuse colliding timeline rows and ontology values, match rows in memory, fingerprint field hashes

garrytan4748a0b6d54c
diff --git a/docs/guides/repair.md b/docs/guides/repair.md
index 3b02a5264d..02cf84b469 100644
--- a/docs/guides/repair.md
+++ b/docs/guides/repair.md
@@ -172,7 +172,7 @@ work behind exits 0, so scripts should also check `results[].complete`.
| `missing-tables` | `schema_missing_tables` | Explicit-only, preview-bound. Recreates, empty, a rebuildable cache table (`query_cache`, `code_traversal_cache`) that the applied migrations created but the brain no longer has, on a brain whose schema version already covers its creating migration, so `apply-migrations` never recreates it (#6911). DDL only (`IF NOT EXISTS`), no journal admission, brain-wide. See [missing tables](#missing-tables). | `user_data_tables`: a missing table that holds user data; this repair never recreates it, so restore its rows from a backup first. `other_tables`: a missing table that holds no user data; `gbrain init --migrate-only` recreates it if the base schema defines it. |
| `url-clone-fences` | none (`gbrain sync --source <id>` stops at the pull with `pull_blocked_local_changes`) | Explicit-only, preview-bound, destructive consent. Clears the `## Facts` fence rows an older gbrain appended to files of a clone it owns (`gbrain sources add --url`), which block every `git pull --ff-only` that touches those files (#6946). The preview proves each changed file differs from HEAD only by appended fence rows; `--apply --expect <hash> --yes` backs each file up, keeps its rows database-only and restores the file from HEAD. Unmanaged brains only. See [URL clone fences](#url-clone-fences). | `non_fence_edit`, `unparsed_fence`, `staged_change`, `deleted`, `not_in_head`, `unreadable_checkout`: a file with any other difference from HEAD, named with why; a file whose HEAD would not import as its page (`visibility_noncanonical` and the other import holds): apply keeps its rows database-only and restores the file so the clone can pull, but does not import HEAD and reports `held`; files that changed after the preview (`changed_since_preview`). |
| `segmented-mirror` | none (doctor does not read segmented files; run the preview once if the brain ran a version before v0.60.299.0) | Explicit-only, preview-bound, destructive consent. Rewrites the canonical file of a page stored in segments that an older binary's withdrawal mirror passed by (`page_body_segmented`), so the file still holds a row `forget`, `forget_fact` or `purge_fact` removed from the database. The file becomes the page's canonical render (`gbrain export`'s bytes), written by the withdrawal mirror and committed by its Git effect. See [segmented mirror](#segmented-mirror). | `file_database_drift`: the file also differs from the database for another reason; it is listed held and never rewritten. `not_owner_host`: the preview ran off the source's owner host and read none of its files. |
-| `secret-spans` | none (the operator names the pages) | Explicit-only, preview-bound, trusted local CLI. Replaces the credentials the secret scanner finds in the database copies of the named pages with `<REDACTED:rule>` markers and changes nothing else: title, frontmatter values (never keys), the body (inline or segmented), timeline, fact, take and link rows, raw data; chunks holding a value are rebuilt. Database only: no file write, no Git commit, and a sync hold stays. `--source <id> --slug <slug>... [--rules <list>]`, then `--apply --expect <hash>`. See [secret spans](#secret-spans). | `clean` (no match), `refused` (`page_not_found`, `page_kind_unsupported`, `page_too_large`, `proof_failed`); pages that moved after the preview (`preview_changed`, `changed_since_preview`). |
+| `secret-spans` | none (the operator names the pages) | Explicit-only, preview-bound, trusted local CLI. Replaces the credentials the secret scanner finds in the database copies of the named pages with `<REDACTED:rule>` markers and changes nothing else: title, frontmatter values (never keys), the body (inline or segmented), timeline, fact, take and link rows, raw data; chunks holding a value are rebuilt. Database only: no file write, no Git commit, and a sync hold stays. `--source <id> --slug <slug>... [--rules <list>]`, then `--apply --expect <hash>`. See [secret spans](#secret-spans). | `clean` (no match), `refused` (`page_not_found`, `page_kind_unsupported`, `page_too_large`, `proof_failed`, `fact_value_unsupported`, `timeline_rows_collide`); pages that moved after the preview (`preview_changed`, `changed_since_preview`). |
Timeline rows that an earlier version of a page produced and its current text
no longer has are removals, not history, so `timeline` neither counts nor
@@ -1234,12 +1234,15 @@ replaced too, and text that is already only markers is left alone, so a second
preview of an applied page is empty.
The preview lists each page with its spans per rule and, for each changed
-field, the old and new SHA-256 and a hash of the span offsets. It never prints a
-value. It also counts the retained page versions, which are kept as history and
-not rewritten. A page is `refused` when it does not exist, is a code or image
-page, has a segmented body over 24 MiB, or when the planned spans and the
-scanner's own redaction of the text disagree (`proof_failed`); nothing on a
-refused page changes.
+field, the first 64 bits of its old and new SHA-256 and a hash of the span
+offsets. It never prints a value. It also counts the retained page versions,
+which are kept as history and not rewritten. A page is `refused`, and nothing
+on it changes, when it does not exist, is a code or image page, has a segmented
+body over 24 MiB, when the planned spans and the scanner's own redaction of the
+text disagree (`proof_failed`), when an ontology fact's resolved value holds a
+match (`fact_value_unsupported`: the value keys the fact's dedup; retire the
+fact with `gbrain forget`), or when two timeline rows of one day would read the
+same once redacted (`timeline_rows_collide`: delete the duplicate row first).
The apply checks every previewed page again before writing anything and
refuses `preview_changed` if one moved; preview again. Each page is then one
@@ -1249,12 +1252,16 @@ brains included). Trust tiers are kept. Chunks that held a value are deleted
and rebuilt from the redacted body, keeping the vectors of unchanged chunks;
changed chunks are embedded unless `--no-embed` or the page carries
`embed_skip`. Fact and take vectors of a rewritten row are cleared. A
-value-free receipt per page (field hashes and span offsets) is stored with the
-write and appended to the `secret-spans` audit log, and the apply re-scans each
+value-free receipt per page (field fingerprints and span offsets) is stored
+with the write, where a resumed apply finds it for the next 7 days, and
+appended to the `secret-spans` audit log, and the apply re-scans each
page and reports any match left (`residual`, expected 0).
There is no undo: the old value is gone from the live copies. Page versions,
-write receipts and Git history are kept and still hold it. It runs only from
+write receipts and Git history are kept and still hold it. On a brain with a
+repository, redact the file too: a later import of the unredacted file (a full
+sync, `gbrain repair file-drift --prefer file`, an edit) writes the value back
+into the database. It runs only from
the trusted local CLI on the brain host, only when named, and never from `--all`, doctor's remediation plan, the maintenance
cycle or an upgrade.
diff --git a/src/core/repair/secret-spans.ts b/src/core/repair/secret-spans.ts
index 9e3280bab7..978ccb0340 100644
--- a/src/core/repair/secret-spans.ts
+++ b/src/core/repair/secret-spans.ts
@@ -10,7 +10,8 @@
* are never touched), its body (inline columns, or the live generation of a
* segmented body, restaged and flipped), its timeline rows (page or event
* page), the fact rows it holds or that name it with no holding page, its take
- * rows, its outgoing link contexts and the string leaves of its raw data. Chunk
+ * rows, its outgoing link contexts and the string leaves of its raw data (an
+ * ontology fact's resolved value is scanned, and a match refuses the page). Chunk
* rows that hold a value are deleted and the projection is rebuilt from the
* redacted body (vectors of unchanged chunk texts are kept); derivation rows and
* view digests that may hold one are deleted. Fact and take vectors of a
@@ -73,23 +74,27 @@ interface PageRow {
}
type Refusal = 'page_not_found' | 'page_kind_unsupported' | 'page_too_large';
+type PlanRefusal = 'proof_failed' | 'fact_value_unsupported' | 'timeline_rows_collide';
/** `text`: the body's truth and timeline regions as stored (inline columns, or a segmented body read in full). */
interface Read { page: PageRow; fields: Field[]; text: { truth: string; tail: string } }
interface Planned { read: Read; results: FieldRedaction[]; digest: string }
interface Selection { source_ids: string[]; slugs: string[]; rules: string[] }
-interface FieldSummary { key: string; spans: number; old_sha256: string; new_sha256: string; spans_sha256: string }
+/** `old_fp` / `new_fp`: the first 64 bits of the field's SHA-256, so a field that is only a short value is not printed as its full hash. */
+interface FieldSummary { key: string; spans: number; old_fp: string; new_fp: string; spans_sha256: string }
interface PlannedPage {
source_id: string; slug: string; id: number; revision: string; generation: string; storage: string; digest: string;
rules: Record<string, number>; echo: number; fields: FieldSummary[]; versions: number;
}
interface SpanItem extends RepairItem { page: PlannedPage; hash: string; rules: string[]; last: boolean }
-const REFUSAL_TEXT: Record<Refusal | 'proof_failed', string> = {
+const REFUSAL_TEXT: Record<Refusal | PlanRefusal, string> = {
page_not_found: 'no live page with this slug in the source',
page_kind_unsupported: 'only markdown pages are redacted here (code and image pages index the file itself)',
page_too_large: `the segmented body is over the ${PAGE_MAX_BYTES / 1024 ** 2} MiB single-string ceiling this repair reads`,
proof_failed: 'the planned spans and the scanner\'s own redaction disagree, so nothing on the page is rewritten',
+ fact_value_unsupported: 'an ontology fact\'s resolved value holds a match; its value_hash keys the fact\'s dedup, so this repair does not rewrite it (retire the fact with gbrain forget, then preview again)',
+ timeline_rows_collide: 'two timeline rows of one day would read the same once redacted, which their dedup key refuses; delete the duplicate row, then preview again',
};
function previewCommand(selection: Selection): string {
@@ -149,7 +154,7 @@ async function readFields(engine: BrainEngine, sourceId: string, slug: string):
};
rows('timeline_entries', ['summary', 'detail', 'source'], await engine.executeRaw(
'SELECT id,summary,detail,source FROM timeline_entries WHERE page_id=$1 OR event_page_id=$1 ORDER BY id', [id]));
- rows('facts', ['fact', 'context'], await engine.executeRaw(`SELECT id,fact,context FROM facts
+ rows('facts', ['fact', 'context', 'value'], await engine.executeRaw(`SELECT id,fact,context,value FROM facts
WHERE source_id=$1 AND (source_markdown_slug=$2 OR (source_markdown_slug IS NULL AND entity_slug=$2)) ORDER BY id`, [sourceId, slug]));
rows('takes', ['claim', 'source', 'resolved_source'], await engine.executeRaw('SELECT id,claim,source,resolved_source FROM takes WHERE page_id=$1 ORDER BY id', [id]));
rows('links', ['context'], await engine.executeRaw('SELECT id,context FROM links WHERE from_page_id=$1 ORDER BY id', [id]));
@@ -167,6 +172,27 @@ function planPage(read: Read, rules: SecretRuleSet): Planned {
results.map(result => [result.key, result.old_sha256, result.spans_sha256])]) };
}
+/**
+ * Why a planned page cannot be rewritten as planned, or null. A timeline row's dedup key (page, date, summary, source)
+ * is checked against every row of the pages its rows belong to, because the event-page rows sit on other pages.
+ */
+async function planRefusal(engine: BrainEngine, planned: Planned): Promise<PlanRefusal | null> {
+ if (planned.results.some(result => !result.proven)) return 'proof_failed';
+ const changed = changedOf(planned);
+ if (changed.some(result => /^facts\/\d+\/value$/.test(result.key))) return 'fact_value_unsupported';
+ if (!changed.some(result => result.key.startsWith('timeline_entries/'))) return null;
+ const next = new Map(planned.results.map(result => [result.key, result.next]));
+ const rows = await engine.executeRaw<{ id: number | string; page_id: number | string; date: string; summary: string; source: string }>(`SELECT id,page_id,date::text AS date,summary,source
+ FROM timeline_entries WHERE page_id IN (SELECT page_id FROM timeline_entries WHERE page_id=$1 OR event_page_id=$1)`, [planned.read.page.id]);
+ const keys = new Set<string>();
+ for (const row of rows) {
+ const key = JSON.stringify([String(row.page_id), row.date, next.get(`timeline_entries/${row.id}/summary`) ?? row.summary, next.get(`timeline_entries/${row.id}/source`) ?? row.source]);
+ if (keys.has(key)) return 'timeline_rows_collide';
+ keys.add(key);
+ }
+ return null;
+}
+
const changedOf = (planned: Planned) => planned.results.filter(result => result.spans.length);
function summaryOf(planned: Planned, versions: number): PlannedPage {
@@ -175,7 +201,7 @@ function summaryOf(planned: Planned, versions: number): PlannedPage {
const counts = spanCounts(changed.flatMap(result => result.spans));
return { source_id: page.source_id, slug: page.slug, id: page.id, revision: page.revision, generation: page.generation, storage: page.body_storage,
digest: planned.digest, rules: counts.rules, echo: counts.echo, versions,
- fields: changed.map(result => ({ key: result.key, spans: result.spans.length, old_sha256: result.old_sha256, new_sha256: result.new_sha256, spans_sha256: result.spans_sha256 })) };
+ fields: changed.map(result => ({ key: result.key, spans: result.spans.length, old_fp: result.old_sha256.slice(0, 16), new_fp: result.new_sha256.slice(0, 16), spans_sha256: result.spans_sha256 })) };
}
const ruleText = (rules: Record<string, number>) => Object.entries(rules).sort().map(([rule, n]) => `${rule}=${n}`).join(', ');
@@ -250,15 +276,22 @@ async function writePage(tx: BrainEngine, planned: Planned): Promise<{ unsealed:
await tx.executeRaw('UPDATE pages SET knowledge_revision=gen_random_uuid() WHERE id=$1', [page.id]);
if (page.body_storage === 'segmented') await sealSegmentedPage(tx, page.id);
}
+ // Rows holding a value are matched here, never in SQL, so no secret is ever sent as a statement parameter (which a
+ // server's slow-statement log would keep).
const values = valuesOf(planned);
+ const holds = (text: string | null | undefined) => !!text && values.some(value => text.includes(value));
if (page.body_storage !== 'segmented') {
- await tx.executeRaw(`DELETE FROM content_chunks c WHERE c.page_id=$1 AND EXISTS (SELECT 1 FROM unnest($2::text[]) v
- WHERE strpos(c.chunk_text,v)>0 OR strpos(COALESCE(c.doc_comment,''),v)>0)`, [page.id, values]);
+ const chunks = await tx.executeRaw<{ id: number | string; chunk_text: string; doc_comment: string | null }>(
+ 'SELECT id,chunk_text,doc_comment FROM content_chunks WHERE page_id=$1 -- chunk-visibility-ok: every stored row of the page is checked for a value', [page.id]);
+ const doomed = chunks.filter(chunk => holds(chunk.chunk_text) || holds(chunk.doc_comment)).map(chunk => Number(chunk.id));
+ if (doomed.length) await tx.executeRaw('DELETE FROM content_chunks WHERE id=ANY($1::int[])', [doomed]);
await queuePageProjection(tx, page.source_id, page.slug, 'secret_spans');
}
- await tx.executeRaw(`DELETE FROM page_segment_derivations d WHERE d.page_id=$1 AND d.output IS NOT NULL
- AND EXISTS (SELECT 1 FROM unnest($2::text[]) v WHERE strpos(d.output::text,v)>0)`,
- [page.id, [...new Set([...values, ...values.map(value => JSON.stringify(value).slice(1, -1))])]]);
+ const derivations = await tx.executeRaw<{ pipeline: string; occurrence_id: number | string; output: unknown }>(
+ 'SELECT pipeline,occurrence_id,output FROM page_segment_derivations WHERE page_id=$1 AND output IS NOT NULL', [page.id]);
+ for (const row of derivations.filter(row => leaves(row.output).some(leaf => holds(leaf.text)))) {
+ await tx.executeRaw('DELETE FROM page_segment_derivations WHERE page_id=$1 AND pipeline=$2 AND occurrence_id=$3', [page.id, row.pipeline, row.occurrence_id]);
+ }
await tx.executeRaw('DELETE FROM page_view_digests WHERE page_id=$1', [page.id]);
return { unsealed };
}
@@ -337,9 +370,10 @@ export const secretSpansRepair: RepairHandler = {
}
const plan = planPage(read, rules);
const label = `${sourceId}:${slug} (revision ${read.page.revision})`;
- if (plan.results.some(result => !result.proven)) {
+ const refusal = await planRefusal(engine, plan);
+ if (refusal) {
residuals.refused!++;
- listing.push({ item: label, class: 'refused', detail: `proof_failed: ${REFUSAL_TEXT.proof_failed}` });
+ listing.push({ item: label, class: 'refused', detail: `${refusal}: ${REFUSAL_TEXT[refusal]}` });
continue;
}
if (!changedOf(plan).length) { residuals.clean!++; continue; }
@@ -351,6 +385,8 @@ export const secretSpansRepair: RepairHandler = {
residuals.retained_versions! += page.versions;
listing.push({ item: label, class: 'redact', detail: changeDetail(page) });
}
+ // Items run in cursor (page id) order, so a resumed apply after --limit or a stop continues with the rest.
+ planned.sort((a, b) => a.id - b.id);
const hash = await hashOf(engine, scope, selection, planned);
if (planned.length) await saveApprovedSet<PlannedPage & { selection: Selection }>(engine, { command: 'secret-spans', hash }, planned.map(page => ({ ...page, selection })));
return { items: planned.map((page, index) => item(page, hash, selection.rules, index === planned.length - 1)), preview_hash: hash, residuals, listing,
@@ -391,11 +427,13 @@ export const secretSpansRepair: RepairHandler = {
if (typeof read === 'string') return { refused: read };
const planned = planPage(read, rules);
if (read.page.id !== page.id || planned.digest !== page.digest) return { refused: 'changed_since_preview' };
+ const refusal = await planRefusal(tx, planned);
+ if (refusal) return { refused: refusal };
const { unsealed } = await writePage(tx, planned);
const [after] = await tx.executeRaw<{ revision: string }>('SELECT knowledge_revision::text AS revision FROM pages WHERE id=$1', [page.id]);
const receipt = { kind: 'secret-spans', hash, source_id: page.source_id, slug: page.slug, page_id: page.id, revision_before: page.revision,
- revision_after: after?.revision ?? null, rules: rules.rules, fields: changedOf(planned).map(result => ({ key: result.key, old_sha256: result.old_sha256,
- new_sha256: result.new_sha256, spans_sha256: result.spans_sha256, spans: result.spans.map(span => [span.start, span.end, span.rule, span.echo ? 1 : 0]) })) };
+ revision_after: after?.revision ?? null, rules: rules.rules, fields: changedOf(planned).map(result => ({ key: result.key, old_fp: result.old_sha256.slice(0, 16),
+ new_fp: result.new_sha256.slice(0, 16), spans_sha256: result.spans_sha256, spans: result.spans.map(span => [span.start, span.end, span.rule, span.echo ? 1 : 0]) })) };
await tx.executeRaw(`INSERT INTO op_checkpoints(op,fingerprint,completed_keys,updated_at) VALUES ($1,$2,$3::text::jsonb,now())
ON CONFLICT (op,fingerprint) DO UPDATE SET completed_keys=EXCLUDED.completed_keys,updated_at=now()`,
[SECRET_SPANS_RECEIPT_OP, receiptKey(hash, page.id), JSON.stringify([receipt])]);
@@ -423,7 +461,7 @@ export const secretSpansRepair: RepairHandler = {
},
render(details) {
return ((details.pages ?? []) as PlannedPage[]).flatMap(page => [` ${page.source_id}:${page.slug} (${page.storage}): ${changeDetail(page)}`,
- ...page.fields.map(field => ` ${field.key}: ${field.spans} span(s), sha256 ${field.old_sha256.slice(0, 12)} -> ${field.new_sha256.slice(0, 12)}, spans ${field.spans_sha256.slice(0, 12)}`)]);
+ ...page.fields.map(field => ` ${field.key}: ${field.spans} span(s), sha256 ${field.old_fp} -> ${field.new_fp}, spans ${field.spans_sha256.slice(0, 12)}`)]);
},
async report(ctx, scope, result, opts) {
if (result.mode !== 'apply' || !opts.slugs?.length || scope.source_ids.length !== 1) return {};
diff --git a/test/repair-secret-spans.test.ts b/test/repair-secret-spans.test.ts
index 36fbe95167..5774056fd3 100644
--- a/test/repair-secret-spans.test.ts
+++ b/test/repair-secret-spans.test.ts
@@ -209,6 +209,41 @@ describe('GST-327 gbrain repair secret-spans on a repo-less brain', () => {
}), 300_000);
});
+describe('GST-327 gbrain repair secret-spans: resume and refusals', () => {
+ test('an apply stopped by --limit resumes in page-id order and finishes; colliding timeline rows and ontology values refuse their page', () => withEnv({ GBRAIN_HOME: home }, async () => {
+ const { engine } = repoLess[0]!;
+ const AWS2 = 'AKIA' + 'EXAMPLENOTREAL01';
+ // Imported first, so its id is lower although its slug sorts last.
+ await importFromContent(engine, 'notes/z-old-example', `---\ntype: note\ntitle: Old\n---\n\nOld key ${AWS}.\n`, { noEmbed: true, sourceId: 'default' });
+ await importFromContent(engine, 'notes/a-new-example', `---\ntype: note\ntitle: New\n---\n\nNew key ${AWS2}.\n`, { noEmbed: true, sourceId: 'default' });
+ await importFromContent(engine, 'notes/dup-example', '---\ntype: note\ntitle: Dup\n---\n\nNo key in the body.\n', { noEmbed: true, sourceId: 'default' });
+ await importFromContent(engine, 'notes/value-example', '---\ntype: note\ntitle: Value\n---\n\nNo key in the body.\n', { noEmbed: true, sourceId: 'default' });
+ const dup = await pageRow(engine, 'notes/dup-example');
+ await engine.executeRaw("INSERT INTO timeline_entries(page_id,date,source,summary,detail) VALUES ($1,'2024-05-01','import',$2,''),($1,'2024-05-01','import',$3,'')",
+ [dup.id, `Rotated ${AWS}`, `Rotated ${AWS2}`]);
+ await engine.executeRaw(`INSERT INTO facts(source_id,entity_slug,fact,value,source,kind) VALUES ('default','notes/value-example','Credential on file',$1,'test','fact')`, [AWS]);
+ const scope = await resolveRepairScope(engine, 'default');
+ const slugs = ['notes/a-new-example', 'notes/z-old-example', 'notes/dup-example', 'notes/value-example'];
+ const timelineBefore = await engine.executeRaw('SELECT id,summary FROM timeline_entries WHERE page_id=$1 ORDER BY id', [dup.id]);
+
+ const preview = await (await repairRunner(engine, { apply: false, logger, noEmbed: true })).run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs });
+ expect(preview.listing!.map(entry => [entry.class, entry.item.split(' ')[0], entry.detail?.split(':')[0]])).toEqual([
+ ['redact', 'default:notes/a-new-example', '1 span(s) in 1 field(s)'], ['refused', 'default:notes/dup-example', 'timeline_rows_collide'],
+ ['refused', 'default:notes/value-example', 'fact_value_unsupported'], ['redact', 'default:notes/z-old-example', '1 span(s) in 1 field(s)']]);
+ const apply = (limit?: number) => repairRunner(engine, { apply: true, logger, noEmbed: true })
+ .then(runner => runner.run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs, expect: preview.preview_hash, ...(limit ? { limit } : {}) }));
+ const first = await apply(1);
+ expect(first.applied).toBe(1);
+ expect((await pageRow(engine, 'notes/z-old-example')).compiled_truth).not.toContain(AWS);
+ const rest = await apply();
+ expect(rest.applied).toBe(1);
+ expect(rest.complete).toBe(true);
+ expect((await pageRow(engine, 'notes/a-new-example')).compiled_truth).not.toContain(AWS2);
+ expect(await engine.executeRaw('SELECT id,summary FROM timeline_entries WHERE page_id=$1 ORDER BY id', [dup.id])).toEqual(timelineBefore);
+ expect(await engine.executeRaw("SELECT value FROM facts WHERE entity_slug='notes/value-example'")).toEqual([{ value: AWS }]);
+ }), 120_000);
+});
+
for (const backend of testBackends()) describe(`GST-327 gbrain repair secret-spans on a managed brain (${backend})`, () => {
test('a held page is redacted in the database; its file, its hold and Git stay as they are', () => managedBrain(async ({ engine, ctx, root }) => {
const slug = 'notes/held-example';