Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
repair: add secret-spans, span-only redaction of a page's database copies (GST-327)
garrytan39e6fbe3595a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 796bbb7591..6dd74c065b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,19 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**`gbrain repair secret-spans --source <id> --slug <slug>...` replaces the credentials the secret scanner finds in the database copies of the pages you name with `<REDACTED:rule>` markers and changes nothing else, held pages included (GST-327).**++A credential pasted into a page lives in more places than its file: the page row, timeline, fact and take rows, link contexts, raw connector data and the chunks built from them. A page that sync holds keeps its stale database copy, so redacting the file never reaches it. The new kind reads every one of those copies, plans the spans with the scanner's own rules (`--rules`, default every named credential rule; the two broad heuristics only when named) and proves the result equals the scanner's redaction byte for byte before it writes. The preview shows counts per rule and each changed field's old and new SHA-256, never a value; `--apply --expect <hash>` rewrites exactly that set, one transaction per page, keeps trust tiers, rebuilds the chunks from the redacted body and re-scans what it wrote. It writes no file and no commit, leaves sync holds as they are, keeps page versions as history, and works the same on a repo-less brain. Explicit-only and trusted-local-CLI-only.++### Itemized changes++- `src/core/secret-spans.ts` (new): the pure span planner and proof (`redactFields`, `resolveSecretRules`, `residualMatches`); `src/core/secret-scan.ts` exports `NAMED_SECRET_RULES`.+- `src/core/repair/secret-spans.ts` (new): the repair kind (preview, preview-bound apply, per-page receipt, re-scan), registered explicit-only and preview-bound; `gbrain repair` accepts `--rules` and `--slug` for it.+- `docs/guides/repair.md`: the `secret-spans` section and table row.+- Tests: `test/secret-spans.test.ts`, `test/repair-secret-spans.test.ts` (repo-less and managed brains, segmented embed_skip page, held page, impossible timeline date, `preview_changed`, remote refusal); the explicit-kind lists in `test/repair-explicit-only.test.ts` and `test/fix-wave-5-integration.test.ts` name the new kind.+## [0.60.513.0] - 2026-10-30**A search on a brain with many pages still waiting for a projection no longer spends seconds in the readiness check: the probe stops after it finds a pending page instead of also walking every unsealed page for the `by_design` answer it then discards (#8000).**diff --git a/docs/guides/repair.md b/docs/guides/repair.mdindex 8e8f24ab93..3b02a5264d 100644--- a/docs/guides/repair.md+++ b/docs/guides/repair.md@@ -172,6 +172,7 @@ work behind exits 0, so scripts should also check `results[].complete`.| `missing-tables` | `schema_missing_tables` | Explicit-only, preview-bound. Recreates, empty, a rebuildable cache table (`query_cache`, `code_traversal_cache`) that the applied migrations created but the brain no longer has, on a brain whose schema version already covers its creating migration, so `apply-migrations` never recreates it (#6911). DDL only (`IF NOT EXISTS`), no journal admission, brain-wide. See [missing tables](#missing-tables). | `user_data_tables`: a missing table that holds user data; this repair never recreates it, so restore its rows from a backup first. `other_tables`: a missing table that holds no user data; `gbrain init --migrate-only` recreates it if the base schema defines it. || `url-clone-fences` | none (`gbrain sync --source <id>` stops at the pull with `pull_blocked_local_changes`) | Explicit-only, preview-bound, destructive consent. Clears the `## Facts` fence rows an older gbrain appended to files of a clone it owns (`gbrain sources add --url`), which block every `git pull --ff-only` that touches those files (#6946). The preview proves each changed file differs from HEAD only by appended fence rows; `--apply --expect <hash> --yes` backs each file up, keeps its rows database-only and restores the file from HEAD. Unmanaged brains only. See [URL clone fences](#url-clone-fences). | `non_fence_edit`, `unparsed_fence`, `staged_change`, `deleted`, `not_in_head`, `unreadable_checkout`: a file with any other difference from HEAD, named with why; a file whose HEAD would not import as its page (`visibility_noncanonical` and the other import holds): apply keeps its rows database-only and restores the file so the clone can pull, but does not import HEAD and reports `held`; files that changed after the preview (`changed_since_preview`). || `segmented-mirror` | none (doctor does not read segmented files; run the preview once if the brain ran a version before v0.60.299.0) | Explicit-only, preview-bound, destructive consent. Rewrites the canonical file of a page stored in segments that an older binary's withdrawal mirror passed by (`page_body_segmented`), so the file still holds a row `forget`, `forget_fact` or `purge_fact` removed from the database. The file becomes the page's canonical render (`gbrain export`'s bytes), written by the withdrawal mirror and committed by its Git effect. See [segmented mirror](#segmented-mirror). | `file_database_drift`: the file also differs from the database for another reason; it is listed held and never rewritten. `not_owner_host`: the preview ran off the source's owner host and read none of its files. |+| `secret-spans` | none (the operator names the pages) | Explicit-only, preview-bound, trusted local CLI. Replaces the credentials the secret scanner finds in the database copies of the named pages with `<REDACTED:rule>` markers and changes nothing else: title, frontmatter values (never keys), the body (inline or segmented), timeline, fact, take and link rows, raw data; chunks holding a value are rebuilt. Database only: no file write, no Git commit, and a sync hold stays. `--source <id> --slug <slug>... [--rules <list>]`, then `--apply --expect <hash>`. See [secret spans](#secret-spans). | `clean` (no match), `refused` (`page_not_found`, `page_kind_unsupported`, `page_too_large`, `proof_failed`); pages that moved after the preview (`preview_changed`, `changed_since_preview`). |Timeline rows that an earlier version of a page produced and its current textno longer has are removals, not history, so `timeline` neither counts nor@@ -1198,6 +1199,65 @@ stays). An apply refuses `preview_changed` when a previewed page moved since thepreview; preview again. Undo a page on a managed source with `git revert` of itscommit, or `gbrain revert`.+### Secret spans++A credential pasted into a page is often stored in more places than the page+file: the page row, a held page the sync left alone, timeline, fact and take+rows, link contexts, raw connector data and the chunks built from them. This+explicit-only, preview-bound kind replaces each value the secret scanner finds+in those database copies with a `<REDACTED:rule>` marker and leaves every other+byte as it was. Redacting the files themselves is a separate step through the+normal write path; this kind never writes a file or a commit.++**Say to your agent:** *"Preview the secret spans on these pages, show me the+counts by rule, then apply after I agree."*++```bash+gbrain repair secret-spans --source <id> --slug <slug>... [--rules <rule,...>] # preview: counts and hashes, never values+gbrain repair secret-spans --source <id> --slug <slug>... --apply --expect <hash> # rewrites exactly the previewed pages+```++`--rules` picks the scanner rules (default: every named credential rule, such as+`openai`, `aws_access_key`, `github_token`, `jwt`, `bearer` and+`private_key_pem`). The two broad heuristics, `high_entropy_assignment` and+`labeled_credential`, run only when named, because they also match things that+are not keys. An unknown rule name refuses before anything is read.++What it reads and rewrites for each page: the title, every string value in the+frontmatter (keys are never touched), the body (inline, or the live generation+of a segmented page, which is restaged in place), timeline rows of the page or+its event page, fact rows held by the page or naming it with no holding page,+take rows, the context of its outgoing links and every string value of its raw+data. A value found by a rule in one place is also replaced where it appears+bare elsewhere on the same page (an echo). A key glued to an existing marker is+replaced too, and text that is already only markers is left alone, so a second+preview of an applied page is empty.++The preview lists each page with its spans per rule and, for each changed+field, the old and new SHA-256 and a hash of the span offsets. It never prints a+value. It also counts the retained page versions, which are kept as history and+not rewritten. A page is `refused` when it does not exist, is a code or image+page, has a segmented body over 24 MiB, or when the planned spans and the+scanner's own redaction of the text disagree (`proof_failed`); nothing on a+refused page changes.++The apply checks every previewed page again before writing anything and+refuses `preview_changed` if one moved; preview again. Each page is then one+transaction under its page lock: a coordinated maintenance write on a managed+brain, an attributed maintenance transaction on any other brain (repo-less+brains included). Trust tiers are kept. Chunks that held a value are deleted+and rebuilt from the redacted body, keeping the vectors of unchanged chunks;+changed chunks are embedded unless `--no-embed` or the page carries+`embed_skip`. Fact and take vectors of a rewritten row are cleared. A+value-free receipt per page (field hashes and span offsets) is stored with the+write and appended to the `secret-spans` audit log, and the apply re-scans each+page and reports any match left (`residual`, expected 0).++There is no undo: the old value is gone from the live copies. Page versions,+write receipts and Git history are kept and still hold it. It runs only from+the trusted local CLI on the brain host, only when named, and never from `--all`, doctor's remediation plan, the maintenance+cycle or an upgrade.+### Timeline history scan coverageDoctor's `timeline_history` check classifies at most 2,000 pages or 10@@ -3756,7 +3816,7 @@ Each heading below is the `docs` anchor a refusal carries.### Explicit-only repair kinds`google-file-modes`, `stale-atoms`, `extractor-facts`, `captured-facts`,-`loop-facts`, `ontology-facts`, `file-drift`, `missing-tables` and `segmented-mirror` run only when named: `gbrain repair <kind>` previews, and+`loop-facts`, `ontology-facts`, `file-drift`, `missing-tables`, `segmented-mirror` and `secret-spans` run only when named: `gbrain repair <kind>` previews, and`gbrain repair <kind> --apply` applies (every kind but `google-file-modes`also needs `--expect <hash>`, so it applies exactly the previewed set; `google-file-modes` re-checks each file's owner,type and mode at apply time). They are excluded everywhere else:diff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsvindex e5ffe202b8..7040ea2964 100644--- a/scripts/page-body-readers-baseline.tsv+++ b/scripts/page-body-readers-baseline.tsv@@ -545,6 +545,10 @@ src/core/repair/redacted-facts.ts pageTexts sql INLINE page_versions rows that hsrc/core/repair/redacted-facts.ts restoreFencedRows call:serializePageToMarkdown INLINE inline pages only: a segmented page returns before the read, and its fenced rows preview as class segmentedsrc/core/repair/redacted-facts.ts restoreFencedRows field INLINE inline pages only: a segmented page returns before the read, and its fenced rows preview as class segmentedsrc/core/repair/safe-chunks.ts safeChunksRepair sql META+src/core/repair/secret-spans.ts PAGE_SQL sql INLINE inline body columns; a segmented body is read whole through materializeSegmentedBody, refused page_too_large over the single-string ceiling+src/core/repair/secret-spans.ts readFields field INLINE inline body columns; a segmented body is read whole through materializeSegmentedBody, refused page_too_large over the single-string ceiling+src/core/repair/secret-spans.ts writePage call:contentHash INLINE runs only for an inline page; a segmented body is restaged through prepareSegmentedBody and applySegmentedBody+src/core/repair/secret-spans.ts writePage sql INLINE runs only for an inline page; a segmented body is restaged through prepareSegmentedBody and applySegmentedBodysrc/core/repair/segmented-mirror.ts fileHoldsWithdrawnRows field NONE the overlay result of one file fragment (a facts fence block) for the region digests (body-digests.ts); no page body is readsrc/core/repair/slug-conflicts.ts pageContent call:serializePageToMarkdown SEGWsrc/core/repair/split-facts.ts mergeOne call:serializePageToMarkdown SEGWdiff --git a/src/commands/repair.ts b/src/commands/repair.tsindex 9b10ba504b..58ba5a32b5 100644--- a/src/commands/repair.ts+++ b/src/commands/repair.ts@@ -35,6 +35,7 @@ import { ItemRun } from '../core/item-failure.ts';import { createProgress } from '../core/progress.ts';import { registerSignalOwner } from '../core/process-cleanup.ts';import { cliOptsToProgressOptions, getCliOptions } from '../core/cli-options.ts';+import { resolveSecretRules } from '../core/secret-spans.ts';/** The content-repair lane's command word (#6377): parsed like a kind, dispatched to repair-content.ts, never a RepairKind. */export const CONTENT_LANE = 'content';@@ -90,6 +91,7 @@ export const REPAIR_HELP = `Usage: gbrain repair [<kind>] [--apply] [--source <igbrain repair duplicate-facts [--source <id>] [--include-cross-entity] [--apply --expect <preview-hash> --yes] [--json]gbrain repair fact-claim-dates --source <id> [--day YYYY-MM-DD] [--apply --expect <preview-hash> --yes] [--json]gbrain repair provenance [--source <id>] [--slug <slug>]... [--include-ambiguous] [--apply --expect <preview-hash> --yes] [--json]+ gbrain repair secret-spans --source <id> --slug <slug>... [--rules <rule,...>] [--no-embed] [--apply --expect <preview-hash>] [--json]${REPAIR_ORDER_HELP}gbrain repair rename-identity --source <id> [--slug <new-slug> [--from <old-slug>] [--close][--apply --expect <preview-hash> --yes]] [--json]@@ -166,6 +168,9 @@ Options:--day <YYYY-MM-DD>fact-claim-dates: the UTC day a bulk load stamped on the facts, instead of the days the preview detects;the hash and the apply command carry it.+ --rules <rule,...>+ secret-spans: the secret scanner rules to redact (default every named credential rule;+ high_entropy_assignment and labeled_credential only when named); the hash and the apply command carry them.--kind <kind> [--mode restore|release-held] (--run <id> | --since <ISO-8601|7d> | --key <owner> | --generation <id> | --held)generations: the ledger kind and the operation (one of --run, --since, --key, --generationor --held is required; --mode defaults to restore); the hash and the apply command carry them.@@ -186,22 +191,22 @@ A sources add --url clone whose pull stops at pull_blocked_local_changes: docs/gRenames whose identity sync did not keep (rename-identity): docs/guides/live-sync.md#renaming-a-page.`;const BOOLEAN_FLAGS = new Set(['--apply', '--all', '--json', '--no-embed', '--include-ambiguous', '--include-cross-entity', '--diff', '--yes', '--no-llm', '--all-drifted', '--include-failures', '--force', '--close', '--held']);-const VALUE_FLAGS = new Set(['--source', '--limit', '--expect', '--only', '--skip', '--slug', '--max-usd', '--prefer', '--request', '--class', '--since', '--keep', '--request-id', '--older-than', '--from', '--side', '--state', '--after', '--cursor', '--provenance', '--kind', '--mode', '--run', '--key', '--generation', '--day']);+const VALUE_FLAGS = new Set(['--rules', '--source', '--limit', '--expect', '--only', '--skip', '--slug', '--max-usd', '--prefer', '--request', '--class', '--since', '--keep', '--request-id', '--older-than', '--from', '--side', '--state', '--after', '--cursor', '--provenance', '--kind', '--mode', '--run', '--key', '--generation', '--day']);/** Flags only some kinds accept, with the kinds that do. */const KIND_FLAGS: Record<string, readonly string[]> = {'--only': ['frontmatter', 'fences', 'slug-conflicts', 'file-drift', CONTENT_LANE], '--skip': ['frontmatter', 'fences', 'slug-conflicts', 'file-drift', CONTENT_LANE], '--diff': ['frontmatter', 'fences', 'slug-conflicts', CONTENT_LANE], '--yes': ['frontmatter', 'conversation-labels', 'stuck-recovery', 'ghost-pages', 'legacy-twins', 'rename-identity', 'url-clone-fences'],'--include-cross-entity': ['duplicate-facts'],- '--slug': ['fences', 'file-drift', 'rename-identity', 'overwritten-edits', 'origin-twins', 'timeline-order', 'provenance'], '--from': ['rename-identity'], '--close': ['rename-identity'], '--prefer': ['file-drift'], '--request': ['file-drift'], '--all-drifted': ['file-drift'], '--class': ['file-drift', 'visibility'], '--since': ['file-drift', 'generations'],+ '--slug': ['fences', 'file-drift', 'rename-identity', 'overwritten-edits', 'origin-twins', 'timeline-order', 'provenance', 'secret-spans'], '--from': ['rename-identity'], '--close': ['rename-identity'], '--prefer': ['file-drift'], '--request': ['file-drift'], '--all-drifted': ['file-drift'], '--class': ['file-drift', 'visibility'], '--since': ['file-drift', 'generations'],'--held': ['file-drift', 'generations'], '--side': ['file-drift'], '--state': ['file-drift'], '--cursor': ['file-drift'], '--provenance': ['file-drift'],'--keep': ['stuck-recovery'], '--request-id': ['stuck-recovery'],'--older-than': ['receipt-compaction'], '--include-failures': ['receipt-compaction'], '--force': ['receipt-compaction'],'--after': ['uncommitted-deletions'],- '--kind': ['generations'], '--mode': ['generations'], '--run': ['generations'], '--key': ['generations'], '--generation': ['generations'], '--day': ['fact-claim-dates'],+ '--kind': ['generations'], '--mode': ['generations'], '--run': ['generations'], '--key': ['generations'], '--generation': ['generations'], '--day': ['fact-claim-dates'], '--rules': ['secret-spans'],};const VALUE_EXAMPLES: Record<string, string> = { '--source': 'default', '--limit': '50', '--expect': 'PLAN_HASH', '--only': 'notes/a.md', '--skip': 'notes/a.md','--slug': 'notes/a', '--max-usd': '0.50', '--prefer': 'file', '--request': 'REQUEST_UUID', '--class': 'review_required', '--since': '7d', '--side': 'deleted', '--state': 'tombstone_edited', '--keep': 'prior','--request-id': '123e4567-e89b-42d3-a456-426614174000', '--older-than': '30d', '--from': 'notes/old', '--after': '1200', '--cursor': 'NEXT_CURSOR', '--provenance': 'never_committed',- '--kind': 'atom', '--mode': 'restore', '--run': 'RUN_ID', '--key': 'page:notes/a', '--generation': '42', '--day': '2026-01-15' };+ '--kind': 'atom', '--mode': 'restore', '--run': 'RUN_ID', '--key': 'page:notes/a', '--generation': '42', '--day': '2026-01-15', '--rules': 'openai,aws_access_key' };/** A usage refusal whose fix is the read-only preview (gbrain repair without --apply writes nothing). */const invalid = (message: string, suggestion: string, preview: string[] = ['gbrain', 'repair', '--json']) => opError('invalid_params', message, suggestion,{ fix: readFix('Previews the repair; without --apply it writes nothing.', { argv: preview }) });@@ -211,7 +216,7 @@ interface RepairArgs { kind?: string; apply: boolean; all: boolean; json: booleaallDrifted: boolean; prefer?: 'file' | 'database'; requests: string[]; driftClass?: string; since?: string;held: boolean; side?: string; state?: string; provenance?: string; cursor?: string;keep?: 'prior' | 'disk' | 'withdrawn'; requestId?: string; olderThan?: string; includeFailures: boolean; force: boolean; from?: string; close: boolean; afterPage?: number;- generationKind?: string; generationMode?: string; run?: string; key?: string; generation?: number; day?: string }+ generationKind?: string; generationMode?: string; run?: string; key?: string; generation?: number; day?: string; rules?: string[] }/** `--max-usd` on a kind that never calls a paid model: the cap belongs to the doctor remediation run. */function maxUsdRefusal(flag: string, cap: string | undefined, kind: string | undefined): OperationError {@@ -322,6 +327,7 @@ export function parseRepairArgs(args: string[]): RepairArgs {['gbrain', 'repair', 'fact-claim-dates', ...(parsed.source ? ['--source', parsed.source] : []), '--json']);parsed.day = value;}+ else if (flag === '--rules') parsed.rules = resolveSecretRules(value.split(',').map(rule => rule.trim()).filter(Boolean)).rules;else if (flag === '--kind') parsed.generationKind = value;else if (flag === '--mode') parsed.generationMode = value;else if (flag === '--run') parsed.run = value;@@ -422,7 +428,7 @@ export async function runRepairCommand(engine: BrainEngine, args: string[]): Proif (args.includes('--help') || args.includes('-h')) { console.log(REPAIR_HELP); return; }const { kind, apply, all, json, noEmbed, includeAmbiguous, includeCrossEntity, source, expect, only, skip, slugs, diff, noLlm, maxUsd, limit: limitText,prefer, requests, allDrifted, driftClass, since, held, side, state, provenance, cursor, keep, requestId, olderThan, includeFailures, force, from, close, afterPage,- generationKind, generationMode, run, key, generation, day } = parseRepairArgs(args);+ generationKind, generationMode, run, key, generation, day, rules } = parseRepairArgs(args);const generations: GenerationsSelection | undefined = kind === 'generations' ? { kind: generationKind ?? '', mode: (generationMode ?? 'restore') as GenerationMode,...(run ? { run } : {}), ...(since ? { since } : {}), ...(key ? { key } : {}), ...(generation !== undefined ? { generation } : {}), ...(held ? { held } : {}) } : undefined;if (generations) {@@ -484,7 +490,7 @@ export async function runRepairCommand(engine: BrainEngine, args: string[]): Pro...(k === kind && keep ? { keep } : {}), ...(k === kind && requestId ? { requestId } : {}),...(k === kind && k === 'receipt-compaction' ? { olderThan, includeFailures, force } : {}),...(k === kind && k === 'rename-identity' ? { from, close } : {}), ...(k === kind && afterPage !== undefined ? { afterPage } : {}),- ...(k === kind && generations ? { generations } : {}), ...(k === kind && day ? { day } : {}),+ ...(k === kind && generations ? { generations } : {}), ...(k === kind && day ? { day } : {}), ...(k === kind && rules ? { rules } : {}),...(apply ? { stop: stop.signal, recordDatabaseStops: true } : {}) })).finally(() => releaseSignals?.());if (apply) clearHealthMemo(engine);const paidKinds = results.filter(r => r.paid).map(r => r.kind);diff --git a/src/core/cli-flag-registry.generated.ts b/src/core/cli-flag-registry.generated.tsindex 08048c2e3c..97592b2824 100644--- a/src/core/cli-flag-registry.generated.ts+++ b/src/core/cli-flag-registry.generated.ts@@ -50,7 +50,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'db-repair': ['--accept-pending', '--apply', '--apply-rewrites', '--background', '--brain', '--break-lock', '--db-url', '--dry-run', '--explain', '--filter', '--follow', '--force', '--force-retry', '--format', '--help', '--http', '--json', '--legacy-lock', '--locks', '--name', '--no-accept-pending', '--path', '--prefer-postgres', '--probe', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--restart', '--skip-verify', '--source', '--statement-timeout', '--supabase', '--timeout', '--timings', '--undo-last-rewrite', '--url', '--version', '--wait', '--yes'],'decide': ['--accept-all', '--all', '--all-from', '--apply', '--brain', '--call-site', '--dataset', '--dry-run', '--egress', '--embedding-dimensions', '--embedding-model', '--expect', '--from', '--help', '--include-withdrawals', '--input', '--items-out', '--json', '--kind', '--limit', '--max-cost', '--max-usd', '--min', '--non-interactive', '--out', '--provenance', '--provider', '--query', '--rate', '--reason', '--reason-family', '--recommended', '--remove', '--shadow', '--since', '--slot', '--source', '--status', '--suite', '--target', '--threshold', '--trust', '--undo', '--uri-prefix', '--version', '--what-if-threshold', '--yes'],'diag': ['--access', '--add', '--all', '--allowed-operations', '--apply', '--asof', '--body', '--bound-slug-prefixes', '--brain', '--budget-ms', '--client', '--count', '--cursor', '--date', '--dated-since', '--days', '--detach', '--entities', '--expect', '--expected-revision', '--federated-read', '--format', '--from-index', '--full', '--fuzzy', '--group-by', '--held', '--help', '--http', '--id', '--include-ambiguous', '--include-deleted', '--json', '--kind', '--limit', '--max-usd', '--model', '--multimodal', '--no-autopilot-install', '--no-pull', '--no-walk', '--only', '--page', '--params', '--path', '--phase', '--prefer', '--preview', '--purge', '--query', '--recent', '--refresh-operations', '--replace', '--restart-only', '--reverse', '--row', '--scopes', '--session', '--side', '--since', '--slug', '--slugs', '--source', '--source-id', '--sources', '--stale', '--state', '--surface', '--to', '--updated-after', '--url', '--yes'],- 'doctor': ['--abandon', '--abbrev-ref', '--accept-capabilities', '--accept-pending', '--access', '--account', '--acknowledge-managed', '--add', '--admin-intent', '--adopt', '--adopt-columns', '--adopt-permissions', '--advertise', '--after', '--aggressive', '--aliases', '--all', '--all-drifted', '--all-from', '--allow-old-serve', '--allow-outside-source-root', '--allow-shell-jobs', '--allow-unverified-remote', '--allowed-operations', '--applied', '--apply', '--apply-rewrites', '--assign-corpus', '--audit', '--auto', '--auto-additive', '--auto-fix', '--autostash', '--background', '--batch', '--batch-check', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--branch', '--break-lock', '--budget-ms', '--budget-policy', '--budget-tokens', '--budget-usd', '--by-mention', '--cached', '--cacheinfo', '--call-site', '--catch-up', '--channel', '--check', '--cherry-mark', '--class', '--clear', '--clear-never-installed', '--cli-socket', '--client', '--close', '--concurrency', '--confidence', '--confirm', '--confirm-quiesced', '--confirm-relocated-root', '--conflicts', '--contacts-dir', '--content-audit', '--content-root', '--continue', '--conversation', '--count', '--cursor', '--dataset', '--day', '--db-url', '--deadline', '--delegated-namespace', '--delegated-slug-prefixes', '--delete-brain', '--dest', '--detach', '--detail', '--diff-filter', '--dim', '--dimensions', '--dir', '--disable', '--drain', '--drain-timeout', '--dry-run', '--egress', '--embedding-dimensions', '--embedding-model', '--enable', '--enable-dcr', '--end-of-options', '--entity', '--env', '--error-unmatch', '--exclude', '--exclude-standard', '--exit-code', '--expand', '--expect', '--expect-local', '--expect-remote', '--expect-upstream', '--expected-epoch', '--expected-revision', '--expected-state', '--explain', '--facts', '--fast', '--federated', '--federated-read', '--fetch-timeout-ms', '--ff-only', '--file', '--first-parent', '--fix', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-remove', '--force-retry', '--force-schema', '--format', '--fresh', '--fresh-token', '--from', '--from-meetings', '--from-pages', '--full', '--full-tree', '--fuzzy', '--generation', '--get', '--get-all', '--get-url', '--git-common-dir', '--git-dir', '--git-path', '--grammar', '--group', '--group-by', '--hard', '--hard-deadline', '--harness', '--held', '--help', '--hold-state', '--holds', '--home', '--http', '--id', '--if-version', '--ignore-env-override', '--images', '--include-ambiguous', '--include-cross-entity', '--include-deleted', '--include-failures', '--include-flagged', '--include-frontmatter', '--include-gitignored', '--include-hidden', '--include-pseudo', '--include-repairs', '--include-withdrawals', '--index-audit', '--index-info', '--infer-dates', '--init', '--input', '--install', '--instructions', '--instructions-only', '--is-ancestor', '--is-inside-work-tree', '--items-out', '--job-isolation', '--jq', '--json', '--keep', '--keep-checkout', '--key', '--kind', '--lanes', '--lang', '--left-right', '--legacy-lock', '--limit', '--link', '--link-source', '--link-type', '--list', '--literal-pathspecs', '--local', '--local-env-vars', '--locks', '--log-noop', '--manifest', '--markdown', '--max-cost', '--max-cost-usd', '--max-count', '--max-facts', '--max-jobs', '--max-requests', '--max-rss', '--max-runtime', '--max-usd', '--max-waiting', '--mcp-only', '--merge-base', '--migrate-legacy', '--migrate-only', '--migration', '--min', '--min-n', '--mode', '--model', '--multimodal', '--name', '--name-only', '--name-status', '--nice', '--no-accept-pending', '--no-autopilot-install', '--no-autostash', '--no-capture', '--no-commit-id', '--no-confirm', '--no-cron', '--no-delegate', '--no-edit', '--no-embed', '--no-embed-cache', '--no-embedding', '--no-expand', '--no-extract', '--no-federated', '--no-filters', '--no-gpg-sign', '--no-hooks', '--no-instructions', '--no-lanes', '--no-llm', '--no-merges', '--no-migrate', '--no-probe', '--no-pull', '--no-renames', '--no-repair', '--no-restart', '--no-schema-pack', '--no-seat', '--no-tags', '--no-verify', '--no-walk', '--no-write-fetch-head', '--non-interactive', '--not', '--now', '--older-than', '--once', '--only', '--operation', '--operations', '--others', '--out', '--output', '--oversized', '--overwrite', '--page', '--parallel', '--params', '--parents', '--parked-code', '--parked-pushes', '--pat-file', '--path', '--path-format', '--pending', '--pglite', '--phase', '--pid-file', '--plan', '--porcelain', '--port', '--prefer', '--prefer-postgres', '--preset', '--preview', '--probe', '--probe-pglite', '--profile', '--progress-interval', '--progress-json', '--project', '--provenance', '--provider', '--providers', '--prune-orphans', '--public-url', '--pull', '--pure', '--purge', '--push', '--push-only', '--query', '--question', '--queue', '--quiet', '--rate', '--reader-mode', '--reauth', '--rebase', '--rebuild', '--recommended', '--redact-paths', '--reference-date', '--refresh', '--refresh-operations', '--refresh-skills', '--regenerate', '--reissue', '--relative', '--relower', '--remediate', '--remediation-plan', '--remove', '--remove-core', '--repair', '--repair-unknown-markers', '--replace', '--repo', '--request', '--request-id', '--require-clean', '--reranker', '--reset', '--reset-checkpoint', '--reset-default', '--resolve', '--restart-only', '--restore-only', '--resume', '--retry-failed', '--retry-global', '--retry-held', '--retry-parked', '--reverse', '--review-lower', '--revision', '--right-only', '--rollback-to-source', '--root', '--run', '--runtime', '--sample', '--save', '--scope', '--scopes', '--search-mode', '--seat', '--section', '--segmented', '--select', '--self-transfer', '--semantic', '--session', '--session-id', '--set', '--set-upstream-to', '--shadow', '--shared-skills', '--short', '--show-current', '--show-object-format', '--show-prefix', '--show-token', '--show-toplevel', '--side', '--since', '--since-last-run', '--skills-dir', '--skip', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--slot', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slugs', '--source', '--source-id', '--source-ids', '--sources', '--src-subpath', '--staged', '--stale', '--stat', '--state', '--statement-timeout', '--status', '--stdin', '--stop-mcp', '--strategy', '--strict', '--suite', '--supabase', '--surface', '--swap-only', '--symbol-kind', '--symbolic-full-name', '--takes-holders', '--target', '--target-score', '--threshold', '--tier', '--timeout', '--timings', '--to', '--token', '--token-ttl', '--top-k', '--topo-order', '--trust', '--type', '--unindexed', '--uninstall', '--unsafe-bypass-dream-guard', '--unset-all', '--untracked-files', '--url', '--url-env', '--user', '--vacuum', '--vectors', '--verbose', '--verify', '--version', '--wait', '--wait-drain', '--warn-only', '--watch', '--what-if-threshold', '--window', '--workers', '--working-tree', '--workspace', '--write-tree', '--yes'],+ 'doctor': ['--abandon', '--abbrev-ref', '--accept-capabilities', '--accept-pending', '--access', '--account', '--acknowledge-managed', '--add', '--admin-intent', '--adopt', '--adopt-columns', '--adopt-permissions', '--advertise', '--after', '--aggressive', '--aliases', '--all', '--all-drifted', '--all-from', '--allow-old-serve', '--allow-outside-source-root', '--allow-shell-jobs', '--allow-unverified-remote', '--allowed-operations', '--applied', '--apply', '--apply-rewrites', '--assign-corpus', '--audit', '--auto', '--auto-additive', '--auto-fix', '--autostash', '--background', '--batch', '--batch-check', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--branch', '--break-lock', '--budget-ms', '--budget-policy', '--budget-tokens', '--budget-usd', '--by-mention', '--cached', '--cacheinfo', '--call-site', '--catch-up', '--channel', '--check', '--cherry-mark', '--class', '--clear', '--clear-never-installed', '--cli-socket', '--client', '--close', '--concurrency', '--confidence', '--confirm', '--confirm-quiesced', '--confirm-relocated-root', '--conflicts', '--contacts-dir', '--content-audit', '--content-root', '--continue', '--conversation', '--count', '--cursor', '--dataset', '--day', '--db-url', '--deadline', '--delegated-namespace', '--delegated-slug-prefixes', '--delete-brain', '--dest', '--detach', '--detail', '--diff-filter', '--dim', '--dimensions', '--dir', '--disable', '--drain', '--drain-timeout', '--dry-run', '--egress', '--embedding-dimensions', '--embedding-model', '--enable', '--enable-dcr', '--end-of-options', '--entity', '--env', '--error-unmatch', '--exclude', '--exclude-standard', '--exit-code', '--expand', '--expect', '--expect-local', '--expect-remote', '--expect-upstream', '--expected-epoch', '--expected-revision', '--expected-state', '--explain', '--facts', '--fast', '--federated', '--federated-read', '--fetch-timeout-ms', '--ff-only', '--file', '--first-parent', '--fix', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-remove', '--force-retry', '--force-schema', '--format', '--fresh', '--fresh-token', '--from', '--from-meetings', '--from-pages', '--full', '--full-tree', '--fuzzy', '--generation', '--get', '--get-all', '--get-url', '--git-common-dir', '--git-dir', '--git-path', '--grammar', '--group', '--group-by', '--hard', '--hard-deadline', '--harness', '--held', '--help', '--hold-state', '--holds', '--home', '--http', '--id', '--if-version', '--ignore-env-override', '--images', '--include-ambiguous', '--include-cross-entity', '--include-deleted', '--include-failures', '--include-flagged', '--include-frontmatter', '--include-gitignored', '--include-hidden', '--include-pseudo', '--include-repairs', '--include-withdrawals', '--index-audit', '--index-info', '--infer-dates', '--init', '--input', '--install', '--instructions', '--instructions-only', '--is-ancestor', '--is-inside-work-tree', '--items-out', '--job-isolation', '--jq', '--json', '--keep', '--keep-checkout', '--key', '--kind', '--lanes', '--lang', '--left-right', '--legacy-lock', '--limit', '--link', '--link-source', '--link-type', '--list', '--literal-pathspecs', '--local', '--local-env-vars', '--locks', '--log-noop', '--manifest', '--markdown', '--max-cost', '--max-cost-usd', '--max-count', '--max-facts', '--max-jobs', '--max-requests', '--max-rss', '--max-runtime', '--max-usd', '--max-waiting', '--mcp-only', '--merge-base', '--migrate-legacy', '--migrate-only', '--migration', '--min', '--min-n', '--mode', '--model', '--multimodal', '--name', '--name-only', '--name-status', '--nice', '--no-accept-pending', '--no-autopilot-install', '--no-autostash', '--no-capture', '--no-commit-id', '--no-confirm', '--no-cron', '--no-delegate', '--no-edit', '--no-embed', '--no-embed-cache', '--no-embedding', '--no-expand', '--no-extract', '--no-federated', '--no-filters', '--no-gpg-sign', '--no-hooks', '--no-instructions', '--no-lanes', '--no-llm', '--no-merges', '--no-migrate', '--no-probe', '--no-pull', '--no-renames', '--no-repair', '--no-restart', '--no-schema-pack', '--no-seat', '--no-tags', '--no-verify', '--no-walk', '--no-write-fetch-head', '--non-interactive', '--not', '--now', '--older-than', '--once', '--only', '--operation', '--operations', '--others', '--out', '--output', '--oversized', '--overwrite', '--page', '--parallel', '--params', '--parents', '--parked-code', '--parked-pushes', '--pat-file', '--path', '--path-format', '--pending', '--pglite', '--phase', '--pid-file', '--plan', '--porcelain', '--port', '--prefer', '--prefer-postgres', '--preset', '--preview', '--probe', '--probe-pglite', '--profile', '--progress-interval', '--progress-json', '--project', '--provenance', '--provider', '--providers', '--prune-orphans', '--public-url', '--pull', '--pure', '--purge', '--push', '--push-only', '--query', '--question', '--queue', '--quiet', '--rate', '--reader-mode', '--reauth', '--rebase', '--rebuild', '--recommended', '--redact-paths', '--reference-date', '--refresh', '--refresh-operations', '--refresh-skills', '--regenerate', '--reissue', '--relative', '--relower', '--remediate', '--remediation-plan', '--remove', '--remove-core', '--repair', '--repair-unknown-markers', '--replace', '--repo', '--request', '--request-id', '--require-clean', '--reranker', '--reset', '--reset-checkpoint', '--reset-default', '--resolve', '--restart-only', '--restore-only', '--resume', '--retry-failed', '--retry-global', '--retry-held', '--retry-parked', '--reverse', '--review-lower', '--revision', '--right-only', '--rollback-to-source', '--root', '--rules', '--run', '--runtime', '--sample', '--save', '--scope', '--scopes', '--search-mode', '--seat', '--section', '--segmented', '--select', '--self-transfer', '--semantic', '--session', '--session-id', '--set', '--set-upstream-to', '--shadow', '--shared-skills', '--short', '--show-current', '--show-object-format', '--show-prefix', '--show-token', '--show-toplevel', '--side', '--since', '--since-last-run', '--skills-dir', '--skip', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--slot', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slugs', '--source', '--source-id', '--source-ids', '--sources', '--src-subpath', '--staged', '--stale', '--stat', '--state', '--statement-timeout', '--status', '--stdin', '--stop-mcp', '--strategy', '--strict', '--suite', '--supabase', '--surface', '--swap-only', '--symbol-kind', '--symbolic-full-name', '--takes-holders', '--target', '--target-score', '--threshold', '--tier', '--timeout', '--timings', '--to', '--token', '--token-ttl', '--top-k', '--topo-order', '--trust', '--type', '--unindexed', '--uninstall', '--unsafe-bypass-dream-guard', '--unset-all', '--untracked-files', '--url', '--url-env', '--user', '--vacuum', '--vectors', '--verbose', '--verify', '--version', '--wait', '--wait-drain', '--warn-only', '--watch', '--what-if-threshold', '--window', '--workers', '--working-tree', '--workspace', '--write-tree', '--yes'],'dream': ['--audit-rejects', '--brain', '--cancel-unmatched', '--date', '--dir', '--drain', '--dry-run', '--eval-run', '--force', '--from', '--help', '--input', '--install', '--json', '--limit', '--list', '--max-usd', '--once', '--phase', '--probe', '--pull', '--reconcile-queue', '--since', '--source', '--source-id', '--threshold', '--to', '--unsafe-bypass-dream-guard', '--window', '--yes'],'edge-proposals': ['--access', '--add', '--all', '--all-applied', '--all-held', '--allowed-operations', '--asof', '--body', '--bound-slug-prefixes', '--brain', '--client', '--cursor', '--date', '--dated-since', '--days', '--entities', '--federated-read', '--from-index', '--full', '--fuzzy', '--group-by', '--help', '--http', '--id', '--include-deleted', '--json', '--kind', '--limit', '--max-usd', '--model', '--multimodal', '--no-autopilot-install', '--page', '--params', '--path', '--phase', '--prefer', '--purge', '--recent', '--refresh-operations', '--row', '--scopes', '--session', '--since', '--slugs', '--source', '--sources', '--stale', '--status', '--surface', '--to', '--updated-after', '--url', '--yes'],'edges-backfill': ['--all-sources', '--brain', '--concurrency', '--help', '--json', '--max-chunks', '--source', '--workers'],@@ -110,7 +110,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'reindex-search-vector': ['--accept-pending', '--background', '--brain', '--dry-run', '--explain', '--follow', '--help', '--json', '--no-accept-pending', '--progress-interval', '--progress-json', '--quiet', '--resume', '--source', '--timeout', '--timings', '--wait', '--yes'],'reinit-pglite': ['--abbrev-ref', '--acknowledge-managed', '--all', '--allow-create-db', '--allow-docker', '--allow-empty', '--apply', '--body', '--brain', '--break-lock', '--cached', '--chat-model', '--check', '--concurrency', '--confirm-destructive', '--content-root', '--cursor', '--db-only', '--detail', '--diff-filter', '--dim', '--embedding-dimensions', '--embedding-model', '--empty', '--exclude', '--exclude-standard', '--expansion-model', '--expect', '--filters', '--fix', '--force', '--force-break-lock', '--format', '--full', '--full-history', '--git', '--grant-types', '--hard-deadline', '--help', '--hold-state', '--http', '--include-gitignored', '--include-hidden', '--interval', '--is-ancestor', '--is-shallow-repository', '--issuer-url', '--json', '--key', '--lanes', '--limit', '--local-postgres', '--max-age', '--max-cost', '--max-sources', '--max-usd', '--mcp-only', '--mcp-url', '--migrate-only', '--missing-path', '--model', '--name-only', '--no-auto-embed', '--no-bulk', '--no-delegate', '--no-embed', '--no-embedding', '--no-extract', '--no-git', '--no-gpg-sign', '--no-hard-deadline', '--no-lanes', '--no-llm', '--no-migrate', '--no-pull', '--no-renames', '--no-repair', '--no-schema-pack', '--no-sync', '--no-verify', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--others', '--parallel', '--path', '--pglite', '--plan', '--prefer', '--prefer-postgres', '--preview', '--priority', '--provenance', '--query', '--quiet', '--redact-paths', '--repo', '--reset-checkpoint', '--retry-failed', '--retry-held', '--schema-pack', '--scopes', '--search-mode', '--serial', '--show-toplevel', '--skip-embed-check', '--skip-failed', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--status', '--strategy', '--supabase', '--surface', '--timeout', '--to', '--trust', '--url', '--verify', '--version', '--watch', '--workers', '--working-tree', '--yes'],'remote': ['--brain', '--help', '--json', '--mcp-only', '--only', '--remediation-plan', '--request-id', '--scopes', '--source', '--timeout'],- 'repair': ['--accept-pending', '--acknowledge-managed', '--after', '--all', '--all-drifted', '--apply', '--background', '--brain', '--break-lock', '--class', '--client', '--close', '--cursor', '--day', '--diff', '--expect', '--explain', '--ff-only', '--follow', '--force', '--force-break-lock', '--from', '--full', '--full-history', '--generation', '--held', '--help', '--id', '--include-ambiguous', '--include-cross-entity', '--include-failures', '--include-repairs', '--json', '--keep', '--key', '--kind', '--limit', '--max-cost', '--max-cost-usd', '--max-usd', '--mode', '--no-accept-pending', '--no-delegate', '--no-embed', '--no-llm', '--older-than', '--only', '--prefer', '--probe', '--progress-interval', '--progress-json', '--provenance', '--quiet', '--remediate', '--remediation-plan', '--request', '--request-id', '--reset-checkpoint', '--retry-failed', '--run', '--scopes', '--side', '--since', '--skip', '--skip-failed', '--slug', '--slugs', '--source', '--stale', '--state', '--timeout', '--timings', '--url', '--wait', '--yes'],+ 'repair': ['--accept-pending', '--acknowledge-managed', '--after', '--all', '--all-drifted', '--apply', '--background', '--brain', '--break-lock', '--class', '--client', '--close', '--cursor', '--day', '--diff', '--expect', '--explain', '--ff-only', '--follow', '--force', '--force-break-lock', '--from', '--full', '--full-history', '--generation', '--held', '--help', '--id', '--include-ambiguous', '--include-cross-entity', '--include-failures', '--include-repairs', '--json', '--keep', '--key', '--kind', '--limit', '--max-cost', '--max-cost-usd', '--max-usd', '--mode', '--no-accept-pending', '--no-delegate', '--no-embed', '--no-llm', '--older-than', '--only', '--prefer', '--probe', '--progress-interval', '--progress-json', '--provenance', '--quiet', '--remediate', '--remediation-plan', '--request', '--request-id', '--reset-checkpoint', '--retry-failed', '--rules', '--run', '--scopes', '--side', '--since', '--skip', '--skip-failed', '--slug', '--slugs', '--source', '--stale', '--state', '--timeout', '--timings', '--url', '--wait', '--yes'],'repair-jsonb': ['--accept-pending', '--background', '--brain', '--dry-run', '--explain', '--follow', '--help', '--json', '--no-accept-pending', '--probe', '--progress-interval', '--progress-json', '--quiet', '--source', '--supabase', '--timeout', '--timings', '--url', '--wait'],'report': ['--brain', '--content', '--dir', '--help', '--json', '--source', '--title', '--type'],'repos': ['--abandon', '--abbrev-ref', '--abort', '--accept-suggested', '--access', '--account', '--add', '--admin-intent', '--adopt-columns', '--adopt-permissions', '--adopt-slug-path', '--after', '--all', '--all-new', '--allow-unverified-remote', '--allowed-operations', '--app-id', '--app-install', '--app-pem', '--apply', '--audit', '--auto-additive', '--autostash', '--backups', '--batch-check', '--before', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--branch', '--break-lock', '--budget-usd-per-day', '--cached', '--calendar-id', '--category', '--classify', '--cleanup-dead-local-locks', '--clear', '--client', '--clone-dir', '--code', '--confirm-destructive', '--confirm-quiesced', '--confirm-relocated-root', '--conflicts', '--contacts-dir', '--count', '--cursor', '--days', '--decisions', '--delegated-namespace', '--delegated-slug-prefixes', '--depth', '--detect', '--diff-filter', '--dir', '--disable', '--dry-run', '--enable', '--error-unmatch', '--exclude', '--expect', '--expect-local', '--expect-remote', '--expect-upstream', '--expected-epoch', '--expected-incarnation', '--expected-state', '--federated', '--federated-read', '--fetch-timeout-ms', '--ff-only', '--file', '--fix', '--force', '--format', '--from', '--from-git', '--full', '--future-days', '--get', '--get-all', '--git-dir', '--git-path', '--github-repo', '--grant-types', '--group-by', '--hard', '--help', '--history-days', '--holds', '--http', '--id', '--if-version', '--include', '--include-warns', '--is-ancestor', '--is-inside-work-tree', '--json', '--keep', '--keep-checkout', '--keep-storage', '--kind', '--lanes', '--limit', '--local', '--local-env-vars', '--loops-exclude-labels', '--manifest', '--max-cost-usd', '--message', '--migrate-legacy', '--migration', '--min-trust', '--name', '--name-only', '--no-autostash', '--no-cron', '--no-embed', '--no-extract', '--no-federate', '--no-federated', '--no-harden', '--no-lanes', '--no-pull', '--no-tags', '--no-verify', '--not', '--only', '--operation', '--operations', '--origin', '--others', '--out', '--params', '--parked-code', '--parked-pushes', '--pat-file', '--path', '--path-format', '--plan', '--porcelain', '--preview', '--probe', '--profile', '--pull', '--push-only', '--push-remote', '--quiet', '--read-sources', '--rebase', '--redirect-uri', '--refresh-operations', '--remove-backup', '--repair', '--replace', '--repos', '--request-id', '--require-clean', '--reset-checkpoint', '--reset-default', '--resolve', '--resume', '--retry-failed', '--sample', '--scope', '--scopes', '--secret', '--self-transfer', '--services', '--set', '--set-upstream-to', '--shared-skills', '--short', '--show-toplevel', '--since', '--slug-prefixes', '--source', '--source-id', '--source-ids', '--sources', '--stale', '--state', '--status', '--stdin', '--surface', '--symbolic-full-name', '--takes-holders', '--to', '--token', '--token-command', '--token-endpoint-auth-method', '--token-env', '--token-ttl', '--unindexed', '--unset-all', '--unsettle', '--untracked-files', '--url', '--usage', '--verify', '--wait', '--wait-drain', '--warn-only', '--window', '--yes'],diff --git a/src/core/persistence/preview-approval.ts b/src/core/persistence/preview-approval.tsindex 75ec39513b..a62a4236e2 100644--- a/src/core/persistence/preview-approval.ts+++ b/src/core/persistence/preview-approval.ts@@ -40,7 +40,7 @@ export const PREVIEW_APPROVAL_OP = 'repair-approval';/** The default `purgeStaleCheckpoints` TTL; an older approved set may already be gone. */export const APPROVAL_MAX_AGE_DAYS = 7;-export type PreviewApprovalCommand = 'authorize-legacy' | 'jobs-cancel' | 'stale-atoms' | 'atoms-restore' | 'extractor-facts' | 'conversation-labels' | 'captured-facts' | 'redacted-facts' | 'loop-facts' | 'ontology-facts' | 'facts-rehome' | 'consolidate-ties' | 'duplicate-facts' | 'failed-writes' | 'split-facts' | 'frontmatter' | 'fences' | 'slug-conflicts' | 'managed-sync-orphans' | 'frontmatter-keys' | 'file-drift' | 'stuck-recovery' | 'ghost-pages' | 'origin-twins' | 'receipt-compaction' | 'accepted-takes' | 'rename-identity' | 'overwritten-edits' | 'legacy-twins' | 'missing-tables' | 'segmented-mirror' | 'uncommitted-deletions' | 'visibility' | 'generations' | 'fact-claim-dates' | 'timeline-order' | 'provenance';+export type PreviewApprovalCommand = 'authorize-legacy' | 'jobs-cancel' | 'stale-atoms' | 'atoms-restore' | 'extractor-facts' | 'conversation-labels' | 'captured-facts' | 'redacted-facts' | 'loop-facts' | 'ontology-facts' | 'facts-rehome' | 'consolidate-ties' | 'duplicate-facts' | 'failed-writes' | 'split-facts' | 'frontmatter' | 'fences' | 'slug-conflicts' | 'managed-sync-orphans' | 'frontmatter-keys' | 'file-drift' | 'stuck-recovery' | 'ghost-pages' | 'origin-twins' | 'receipt-compaction' | 'accepted-takes' | 'rename-identity' | 'overwritten-edits' | 'legacy-twins' | 'missing-tables' | 'segmented-mirror' | 'uncommitted-deletions' | 'visibility' | 'generations' | 'fact-claim-dates' | 'timeline-order' | 'provenance' | 'secret-spans';export interface ApprovedSetKey {command: PreviewApprovalCommand;diff --git a/src/core/repair/core.ts b/src/core/repair/core.tsindex 22335d049a..1f53ca7e86 100644--- a/src/core/repair/core.ts+++ b/src/core/repair/core.ts@@ -51,7 +51,7 @@ import { classifyItemFailure, ItemRun, type ItemFailure } from '../item-failure.import type { ProgressReporter } from '../progress.ts';import type { GenerationsSelection } from '../generations/repair.ts';-export const REPAIR_KINDS = ['timeline', 'visibility', 'safe-chunks', 'contextual-mode', 'connector-checkpoints', 'request-indexes', 'connector-fences', 'take-supersession', 'accepted-takes', 'orphan-bindings', 'embedding-effects', 'attribution-backfill', 'planner-stats', 'google-file-modes', 'stale-atoms', 'atoms-restore', 'extractor-facts', 'conversation-labels', 'captured-facts', 'redacted-facts', 'loop-facts', 'ontology-facts', 'facts-rehome', 'consolidate-ties', 'duplicate-facts', 'orphan-children', 'failed-writes', 'split-facts', 'frontmatter', 'fences', 'slug-conflicts', 'timeline-comments', 'timeline-order', 'managed-sync-orphans', 'content-receipt-root', 'frontmatter-keys', 'file-drift', 'stuck-recovery', 'ghost-pages', 'origin-twins', 'receipt-compaction', 'aliases', 'rename-identity', 'overwritten-edits', 'legacy-twins', 'missing-tables', 'url-clone-fences', 'segmented-mirror', 'uncommitted-deletions', 'generations', 'fact-claim-dates', 'provenance'] as const;+export const REPAIR_KINDS = ['timeline', 'visibility', 'safe-chunks', 'contextual-mode', 'connector-checkpoints', 'request-indexes', 'connector-fences', 'take-supersession', 'accepted-takes', 'orphan-bindings', 'embedding-effects', 'attribution-backfill', 'planner-stats', 'google-file-modes', 'stale-atoms', 'atoms-restore', 'extractor-facts', 'conversation-labels', 'captured-facts', 'redacted-facts', 'loop-facts', 'ontology-facts', 'facts-rehome', 'consolidate-ties', 'duplicate-facts', 'orphan-children', 'failed-writes', 'split-facts', 'frontmatter', 'fences', 'slug-conflicts', 'timeline-comments', 'timeline-order', 'managed-sync-orphans', 'content-receipt-root', 'frontmatter-keys', 'file-drift', 'stuck-recovery', 'ghost-pages', 'origin-twins', 'receipt-compaction', 'aliases', 'rename-identity', 'overwritten-edits', 'legacy-twins', 'missing-tables', 'url-clone-fences', 'segmented-mirror', 'uncommitted-deletions', 'generations', 'fact-claim-dates', 'provenance', 'secret-spans'] as const;export type RepairKind = typeof REPAIR_KINDS[number];export interface RepairScope { brain_id: string; source_ids: string[] }@@ -128,7 +128,9 @@ export interface RepairPlanOptions { apply: boolean; expect?: string; includeAmb/** `generations` (GBRA-161): the ledger kind, mode and selectors (`--kind`, `--mode`, `--run`/`--since`/`--key`/`--generation`/`--held`). */generations?: GenerationsSelection;/** `fact-claim-dates` (#7247): the bulk-loaded UTC day to repair (`--day`) instead of the detected ones. */- day?: string }+ day?: string;+ /** `secret-spans` (GST-327): the secret scanner rules to redact (`--rules`); default every named rule. */+ rules?: string[] }export interface RepairHandler {kind: RepairKind;@@ -330,7 +332,7 @@ export async function runRepair(ctx: OperationContext, handler: RepairHandler, sexplicit?: boolean; expect?: string; includeAmbiguous?: boolean; includeCrossEntity?: boolean; only?: string[]; skip?: string[]; slugs?: string[]; noLlm?: boolean;prefer?: 'file' | 'database'; requests?: string[]; allDrifted?: boolean; driftClass?: string; since?: string; held?: boolean; side?: string; state?: string;provenance?: string; cursor?: string; progress?: ProgressReporter;- keep?: 'prior' | 'disk' | 'withdrawn'; requestId?: string; olderThan?: string; includeFailures?: boolean; force?: boolean; from?: string; close?: boolean; afterPage?: number; generations?: GenerationsSelection; day?: string;+ keep?: 'prior' | 'disk' | 'withdrawn'; requestId?: string; olderThan?: string; includeFailures?: boolean; force?: boolean; from?: string; close?: boolean; afterPage?: number; generations?: GenerationsSelection; day?: string; rules?: string[];spec?: Pick<RepairKindSpec, 'explicit_only' | 'consent' | 'spends'>; maxLlmUsd?: number; deadline?: number; now?: () => number;/** GBRA-133: the CLI's termination signal; the apply stops before the next item with a result instead of dying mid-run. */stop?: AbortSignal;@@ -362,7 +364,7 @@ async function runRepairScoped(ctx: OperationContext, handler: RepairHandler, sc...(opts.olderThan ? { olderThan: opts.olderThan } : {}), ...(opts.includeFailures ? { includeFailures: true } : {}), ...(opts.force ? { force: true } : {}),...(opts.from ? { from: opts.from } : {}), ...(opts.close ? { close: true } : {}), ...(opts.afterPage !== undefined ? { afterPage: opts.afterPage } : {}),...(opts.limit !== undefined ? { limit: opts.limit } : {}), ...(opts.generations ? { generations: opts.generations } : {}),- ...(opts.day ? { day: opts.day } : {}) };+ ...(opts.day ? { day: opts.day } : {}), ...(opts.rules?.length ? { rules: opts.rules } : {}) };const plan = await handler.plan(ctx.engine, scope, resumed, planOpts);const pending = opts.limit !== undefined ? plan.items.slice(0, opts.limit) : plan.items;const counters = await capacity(ctx);diff --git a/src/core/repair/registry.ts b/src/core/repair/registry.tsindex bf15782277..ebe06d611e 100644--- a/src/core/repair/registry.ts+++ b/src/core/repair/registry.ts@@ -85,6 +85,7 @@ import { segmentedMirrorRepair } from './segmented-mirror.ts';import { uncommittedDeletionsRepair, UNCOMMITTED_DELETIONS_CONSENT_TEXT } from './uncommitted-deletions.ts';import { factClaimDatesRepair } from './fact-claim-dates.ts';import { provenanceRepair } from './provenance.ts';+import { secretSpansRepair } from './secret-spans.ts';import { generationsRepair, generationsSelectionArgv, type GenerationsSelection } from '../generations/repair.ts';import { restoreDaysHint } from '../generations/config.ts';import { ERROR_CATALOGUE, catalogueError } from '../error-catalogue.ts';@@ -570,6 +571,16 @@ const SPECS: Record<RepairKind, Omit<RepairKindSpec, 'kind'>> = {+ 'Each change only lowers, and a lowered page files a lower_page proposal for gbrain trust review. --slug <slug> narrows. Preview-bound and '+ 'destructive: --apply --expect <hash> --yes writes exactly the previewed pages through the owner page write; local CLI only; never on upgrade.',},+ 'secret-spans': {+ handler: secretSpansRepair, embeds: 'inline', checks: [], explicit_only: true, preview_bound: true,+ summary: 'Replace the credentials the secret scanner finds in the database copies of the named pages with <REDACTED:rule> markers, and nothing '+ + 'else (GST-327): title, frontmatter string values (never keys), the body (inline or segmented), timeline, fact, take and link rows and raw data '+ + 'of each page; chunks holding a value are rebuilt from the redacted body. Needs --source and --slug (repeatable); --rules <list> picks the '+ + 'scanner rules (default every named credential rule; high_entropy_assignment and labeled_credential only when named). The preview lists spans '+ + 'by rule and each changed field\'s old and new SHA-256, never a value; --apply --expect <hash> rewrites exactly that set, one transaction per '+ + 'page, then re-scans it. Database only: no file write, no Git commit, and a sync hold stays, so held pages are redacted where they are stored. '+ + 'History (page versions, receipts, Git) is kept. Trusted local CLI only; runs only when named; never on upgrade.',+ },'generations': {handler: generationsRepair, embeds: 'effect', checks: ['generation_retirements', 'generation_held', 'generation_pending'], explicit_only: true, preview_bound: true, consent: 'destructive',consentText: {@@ -671,16 +682,16 @@ export async function repairRunner(engine: BrainEngine, opts: { apply: boolean;async run(kind: RepairKind, scope: RepairScope, run: { limit?: number; sourceFlag?: string; explicit?: boolean; expect?: string; includeAmbiguous?: boolean; includeCrossEntity?: boolean; only?: string[]; skip?: string[];slugs?: string[]; noLlm?: boolean; maxLlmUsd?: number; deadline?: number; stop?: AbortSignal; recordDatabaseStops?: boolean;prefer?: 'file' | 'database'; requests?: string[]; allDrifted?: boolean; driftClass?: string; since?: string; held?: boolean; side?: string; state?: string; provenance?: string; cursor?: string; progress?: ProgressReporter;- keep?: 'prior' | 'disk' | 'withdrawn'; requestId?: string; olderThan?: string; includeFailures?: boolean; force?: boolean; from?: string; close?: boolean; afterPage?: number; generations?: GenerationsSelection; day?: string } = {}): Promise<RepairResult> {+ keep?: 'prior' | 'disk' | 'withdrawn'; requestId?: string; olderThan?: string; includeFailures?: boolean; force?: boolean; from?: string; close?: boolean; afterPage?: number; generations?: GenerationsSelection; day?: string; rules?: string[] } = {}): Promise<RepairResult> {const ctx = { engine, config, logger, dryRun: !opts.apply, remote: false, sourceId: scope.source_ids[0], writeWaitMs } as OperationContext;const spec = repairSpec(kind, opts.registry);return runRepair(ctx, spec.handler, scope, { apply: opts.apply, limit: run.limit, embeddingModel, sourceFlag: run.sourceFlag, spec,embed: !opts.noEmbed && embeddingModel !== undefined, applyArgs: [...(opts.noEmbed && spec.embeds === 'inline' ? ['--no-embed'] : []),...(run.olderThan ? ['--older-than', run.olderThan] : []), ...(run.includeFailures ? ['--include-failures'] : []), ...(run.force ? ['--force'] : []),- ...(run.afterPage !== undefined ? ['--after', String(run.afterPage)] : []), ...(run.generations ? generationsSelectionArgv(run.generations) : []), ...(run.day ? ['--day', run.day] : [])],+ ...(run.afterPage !== undefined ? ['--after', String(run.afterPage)] : []), ...(run.generations ? generationsSelectionArgv(run.generations) : []), ...(run.day ? ['--day', run.day] : []), ...(run.rules?.length ? ['--rules', run.rules.join(',')] : [])],explicit: run.explicit, expect: run.expect, includeAmbiguous: run.includeAmbiguous, includeCrossEntity: run.includeCrossEntity, only: run.only, skip: run.skip, slugs: run.slugs, noLlm: run.noLlm,maxLlmUsd: run.maxLlmUsd, deadline: run.deadline, ...(run.stop ? { stop: run.stop } : {}), ...(run.recordDatabaseStops ? { recordDatabaseStops: true } : {}), prefer: run.prefer, requests: run.requests, allDrifted: run.allDrifted, driftClass: run.driftClass, since: run.since,- held: run.held, side: run.side, state: run.state, provenance: run.provenance, cursor: run.cursor, progress: run.progress, keep: run.keep, requestId: run.requestId, olderThan: run.olderThan, includeFailures: run.includeFailures, force: run.force, from: run.from, close: run.close, afterPage: run.afterPage, generations: run.generations, day: run.day });+ held: run.held, side: run.side, state: run.state, provenance: run.provenance, cursor: run.cursor, progress: run.progress, keep: run.keep, requestId: run.requestId, olderThan: run.olderThan, includeFailures: run.includeFailures, force: run.force, from: run.from, close: run.close, afterPage: run.afterPage, generations: run.generations, day: run.day, rules: run.rules });},};}diff --git a/src/core/repair/secret-spans.ts b/src/core/repair/secret-spans.tsnew file mode 100644index 0000000000..9e3280bab7--- /dev/null+++ b/src/core/repair/secret-spans.ts@@ -0,0 +1,441 @@+/**+ * `gbrain repair secret-spans` (GST-327): replace the credentials the secret+ * scanner finds in the database copies of named pages with `<REDACTED:rule>`+ * markers, and change nothing else. The text change and its proof are+ * `redactFields` (secret-spans.ts, pure): every field of one page is planned+ * against one shared echo dictionary, and a page with a field whose independent+ * splice and `applyRedaction` disagree is refused (`proof_failed`).+ *+ * The copies of a page: its title, the string leaves of its frontmatter (keys+ * are never touched), its body (inline columns, or the live generation of a+ * segmented body, restaged and flipped), its timeline rows (page or event+ * page), the fact rows it holds or that name it with no holding page, its take+ * rows, its outgoing link contexts and the string leaves of its raw data. Chunk+ * rows that hold a value are deleted and the projection is rebuilt from the+ * redacted body (vectors of unchanged chunk texts are kept); derivation rows and+ * view digests that may hold one are deleted. Fact and take vectors of a+ * rewritten row are cleared. History is kept: page versions, retained segmented+ * generations, write receipts and Git are not rewritten, and the preview counts+ * the retained versions.+ *+ * Nothing is published: no journal admission, no Git effect, no file write, and+ * a sync hold on the page stays as it is, so a held page is redacted where it is+ * stored. One transaction per page under its key lock, a coordinated+ * maintenance write on a managed brain and an attributed maintenance+ * transaction otherwise (repo-less brains included); trust tiers are kept (a+ * redaction is not authorship). Each page's value-free receipt (field hashes and+ * span offsets) is written in the same transaction and appended to the+ * `secret-spans` audit log.+ *+ * Explicit-only, preview-bound and trusted-local-CLI-only: the preview needs+ * `--source` and at least one `--slug`, lists each page's spans by rule and+ * every changed field's old and new SHA-256 under a hash, and writes nothing but+ * the approval checkpoint. `--apply --expect <hash>` rechecks every page+ * before writing (`preview_changed`) and again under its lock+ * (`changed_since_preview`), then re-scans what it wrote.+ */+import type { BrainEngine } from '../engine.ts';+import { OperationError } from '../ops/contract.ts';+import { trustedCliRequired } from '../ops/op-fix.ts';+import { digest } from '../persistence/digest.ts';+import { maintenanceAttribution } from '../persistence/attribution.ts';+import { withCoordinatedWrite, withTrustKeep, withWriteAttribution } from '../persistence/context.ts';+import { readManagedMode } from '../persistence/managed-mode.ts';+import { clearApprovedSet, loadApprovedSet, previewChangedError, previewHash, saveApprovedSet } from '../persistence/preview-approval.ts';+import { applySegmentedBody, sealSegmentedPage } from '../persistence/segmented-apply.ts';+import { withPageTierKept } from '../trust/fence-append.ts';+import { materializeSegmentedBody, prepareSegmentedBody } from '../page-body/ingest.ts';+import { readPageBodyPolicy } from '../page-body/limits.ts';+import { buildSegmentedProjection } from '../page-body/projection.ts';+import { queuePageProjection, resealSafeChunks } from '../page-state/projections.ts';+import { embedStalePages } from '../embed-stale.ts';+import { isQuarantined } from '../quarantine.ts';+import { PAGE_MAX_BYTES } from '../page-size-policy.ts';+import { contentHash } from '../utils.ts';+import { createAuditWriter } from '../audit/audit-writer.ts';+import { redactFields, residualMatches, resolveSecretRules, spanCounts, type FieldRedaction, type SecretRuleSet } from '../secret-spans.ts';+import { afterCursor, type RepairHandler, type RepairItem, type RepairItemOutcome, type RepairListing, type RepairPlan, type RepairScope } from './core.ts';++const DOCS = 'docs/guides/repair.md#secret-spans';+export const SECRET_SPANS_RECEIPT_OP = 'secret-spans-receipt';++type Json = Record<string, unknown>;+type LeafPath = Array<string | number>;+type Store = 'page' | 'frontmatter' | 'timeline_entries' | 'facts' | 'takes' | 'links' | 'raw_data';++/** One stored text of a page: where it lives and its current value. */+interface Field { key: string; text: string; store: Store; id?: number; column?: string; path?: LeafPath }++interface PageRow {+ id: number; source_id: string; slug: string; title: string; type: string; page_kind: string; frontmatter: Json | null;+ compiled_truth: string | null; timeline: string | null; body_storage: string; body_bytes: number | string | null; generation: string;+ body_flags: Json | null; revision: string; contextual_retrieval_mode: string | null; corpus_generation: string | null; tags: string[];+}++type Refusal = 'page_not_found' | 'page_kind_unsupported' | 'page_too_large';+/** `text`: the body's truth and timeline regions as stored (inline columns, or a segmented body read in full). */+interface Read { page: PageRow; fields: Field[]; text: { truth: string; tail: string } }+interface Planned { read: Read; results: FieldRedaction[]; digest: string }++interface Selection { source_ids: string[]; slugs: string[]; rules: string[] }+interface FieldSummary { key: string; spans: number; old_sha256: string; new_sha256: string; spans_sha256: string }+interface PlannedPage {+ source_id: string; slug: string; id: number; revision: string; generation: string; storage: string; digest: string;+ rules: Record<string, number>; echo: number; fields: FieldSummary[]; versions: number;+}+interface SpanItem extends RepairItem { page: PlannedPage; hash: string; rules: string[]; last: boolean }++const REFUSAL_TEXT: Record<Refusal | 'proof_failed', string> = {+ page_not_found: 'no live page with this slug in the source',+ page_kind_unsupported: 'only markdown pages are redacted here (code and image pages index the file itself)',+ page_too_large: `the segmented body is over the ${PAGE_MAX_BYTES / 1024 ** 2} MiB single-string ceiling this repair reads`,+ proof_failed: 'the planned spans and the scanner\'s own redaction disagree, so nothing on the page is rewritten',+};++function previewCommand(selection: Selection): string {+ return `gbrain repair secret-spans --source ${selection.source_ids[0]}${selection.slugs.map(slug => ` --slug ${slug}`).join('')}`+ + (selection.rules.join(',') === resolveSecretRules().rules.join(',') ? '' : ` --rules ${selection.rules.join(',')}`);+}++const pointer = (path: LeafPath) => path.map(part => `/${String(part).replace(/~/g, '~0').replace(/\//g, '~1')}`).join('');++/** Every string leaf of a JSON value with its path, keys in sorted order. */+function leaves(value: unknown, path: LeafPath = []): Array<{ path: LeafPath; text: string }> {+ if (typeof value === 'string') return [{ path, text: value }];+ if (Array.isArray(value)) return value.flatMap((entry, index) => leaves(entry, [...path, index]));+ if (value && typeof value === 'object') return Object.keys(value).sort().flatMap(key => leaves((value as Json)[key], [...path, key]));+ return [];+}++/** A copy of `root` with the string leaf at `path` replaced. */+function withLeaf<T>(root: T, path: LeafPath, text: string): T {+ if (!path.length) return text as T;+ const copy = (Array.isArray(root) ? [...root] : { ...(root as Json) }) as Json;+ const [head, ...rest] = path;+ copy[head as string] = withLeaf(copy[head as string], rest, text);+ return copy as T;+}++const PAGE_SQL = `SELECT p.id,p.source_id,p.slug,p.title,p.type,p.page_kind,p.frontmatter,p.compiled_truth,p.timeline,p.body_storage,p.body_bytes,+ COALESCE(p.body_generation,0)::text AS generation,p.body_flags,p.knowledge_revision::text AS revision,p.contextual_retrieval_mode,p.corpus_generation,+ COALESCE((SELECT jsonb_agg(t.tag ORDER BY t.tag) FROM tags t WHERE t.page_id=p.id),'[]'::jsonb) AS tags+ FROM pages p WHERE p.source_id=$1 AND p.slug=$2 AND p.deleted_at IS NULL`;++/** Every stored text of one live page, in a stable order, or why the page is not redacted here. */+async function readFields(engine: BrainEngine, sourceId: string, slug: string): Promise<Read | Refusal> {+ const [page] = await engine.executeRaw<PageRow>(PAGE_SQL, [sourceId, slug]);+ if (!page) return 'page_not_found';+ if (page.page_kind !== 'markdown') return 'page_kind_unsupported';+ const id = Number(page.id);+ let text = { truth: page.compiled_truth ?? '', tail: page.timeline ?? '' };+ if (page.body_storage === 'segmented') {+ if (Number(page.body_bytes ?? 0) > PAGE_MAX_BYTES) return 'page_too_large';+ const stored = await materializeSegmentedBody(engine, id, sourceId);+ text = { truth: stored.compiled_truth, tail: stored.timeline };+ }+ const fields: Field[] = [+ { key: 'title', text: page.title ?? '', store: 'page', column: 'title' },+ ...leaves(page.frontmatter ?? {}).map(leaf => ({ key: `frontmatter${pointer(leaf.path)}`, text: leaf.text, store: 'frontmatter' as const, path: leaf.path })),+ { key: 'compiled_truth', text: text.truth, store: 'page', column: 'compiled_truth' },+ { key: 'timeline', text: text.tail, store: 'page', column: 'timeline' },+ ];+ const rows = (store: Store, columns: string[], list: Array<Record<string, unknown>>) => {+ for (const row of list) {+ for (const column of columns) {+ const value = row[column];+ if (typeof value === 'string' && value) fields.push({ key: `${store}/${row.id}/${column}`, text: value, store, id: Number(row.id), column });+ }+ }+ };+ rows('timeline_entries', ['summary', 'detail', 'source'], await engine.executeRaw(+ 'SELECT id,summary,detail,source FROM timeline_entries WHERE page_id=$1 OR event_page_id=$1 ORDER BY id', [id]));+ rows('facts', ['fact', 'context'], await engine.executeRaw(`SELECT id,fact,context FROM facts+ WHERE source_id=$1 AND (source_markdown_slug=$2 OR (source_markdown_slug IS NULL AND entity_slug=$2)) ORDER BY id`, [sourceId, slug]));+ rows('takes', ['claim', 'source', 'resolved_source'], await engine.executeRaw('SELECT id,claim,source,resolved_source FROM takes WHERE page_id=$1 ORDER BY id', [id]));+ rows('links', ['context'], await engine.executeRaw('SELECT id,context FROM links WHERE from_page_id=$1 ORDER BY id', [id]));+ for (const row of await engine.executeRaw<{ id: number; data: unknown }>('SELECT id,data FROM raw_data WHERE page_id=$1 ORDER BY id', [id])) {+ for (const leaf of leaves(row.data)) fields.push({ key: `raw_data/${row.id}/data${pointer(leaf.path)}`, text: leaf.text, store: 'raw_data', id: Number(row.id), path: leaf.path });+ }+ return { page: { ...page, id }, fields, text };+}++/** The page's redaction and the digest the plan hash binds: identity, revision, generation and every field's old text and spans. */+function planPage(read: Read, rules: SecretRuleSet): Planned {+ const results = redactFields(read.fields, rules);+ const { page } = read;+ return { read, results, digest: digest(['secret-spans-page-v1', page.id, page.revision, page.generation,+ results.map(result => [result.key, result.old_sha256, result.spans_sha256])]) };+}++const changedOf = (planned: Planned) => planned.results.filter(result => result.spans.length);++function summaryOf(planned: Planned, versions: number): PlannedPage {+ const { page } = planned.read;+ const changed = changedOf(planned);+ const counts = spanCounts(changed.flatMap(result => result.spans));+ return { source_id: page.source_id, slug: page.slug, id: page.id, revision: page.revision, generation: page.generation, storage: page.body_storage,+ digest: planned.digest, rules: counts.rules, echo: counts.echo, versions,+ fields: changed.map(result => ({ key: result.key, spans: result.spans.length, old_sha256: result.old_sha256, new_sha256: result.new_sha256, spans_sha256: result.spans_sha256 })) };+}++const ruleText = (rules: Record<string, number>) => Object.entries(rules).sort().map(([rule, n]) => `${rule}=${n}`).join(', ');+const spansOf = (page: PlannedPage) => page.fields.reduce((n, field) => n + field.spans, 0);++function changeDetail(page: PlannedPage): string {+ return `${spansOf(page)} span(s) in ${page.fields.length} field(s): ${ruleText(page.rules)}${page.echo ? ` (${page.echo} echo)` : ''}`+ + `; ${page.versions} retained version(s) kept`;+}++/** The secret values of one page's spans, kept in memory only: chunk and derivation rows holding one are deleted. */+function valuesOf(planned: Planned): string[] {+ const values = new Set<string>();+ planned.results.forEach((result, index) => {+ for (const span of result.spans) values.add(planned.read.fields[index]!.text.slice(span.start, span.end));+ });+ return [...values];+}++async function* once(text: string): AsyncIterable<Uint8Array> { if (text) yield Buffer.from(text, 'utf8'); }++const receiptKey = (hash: string, pageId: number) => `${hash}:${pageId}`;+const audit = createAuditWriter<{ ts: string } & Json>({ featureName: 'secret-spans', errorLabel: 'secret-spans-audit', errorTrailer: '; repair continues' });++/** Rewrites one page's changed fields in the caller's transaction; `unsealed`: a segmented body whose projection is built after commit. */+async function writePage(tx: BrainEngine, planned: Planned): Promise<{ unsealed: boolean }> {+ const { page, text } = planned.read;+ const fieldOf = (index: number) => planned.read.fields[index]!;+ const next = new Map(planned.results.map(result => [result.key, result.next]));+ let frontmatter: Json = page.frontmatter ?? {};+ planned.results.forEach((result, index) => {+ const field = fieldOf(index);+ if (field.store === 'frontmatter' && result.spans.length) frontmatter = withLeaf(frontmatter, field.path!, result.next);+ });+ const title = next.get('title') ?? page.title, compiled = next.get('compiled_truth') ?? text.truth, timeline = next.get('timeline') ?? text.tail;+ const pageChanged = planned.results.some((result, index) => result.spans.length > 0 && (fieldOf(index).store === 'page' || fieldOf(index).store === 'frontmatter'));+ let unsealed = false;+ if (pageChanged && page.body_storage === 'segmented') {+ const { principal } = await maintenanceAttribution(tx);+ const prepared = await prepareSegmentedBody(tx, { sourceId: page.source_id, slug: page.slug, title, type: page.type, frontmatter, tags: page.tags,+ live: { id: page.id, generation: Number(page.generation), storage: 'segmented', title: page.title, type: page.type, flags: page.body_flags },+ truth: once(compiled), timeline: once(timeline), hasTimeline: timeline.length > 0, flags: page.body_flags ?? {},+ declaredBytes: Buffer.byteLength(compiled, 'utf8') + Buffer.byteLength(timeline, 'utf8'),+ owner: { kind: 'edit', principalKind: principal.kind, principalId: principal.id, baseRevision: null },+ chunking: { quarantined: isQuarantined(frontmatter), ...(page.contextual_retrieval_mode === 'title'+ ? { tier: 'title' as const, corpusGeneration: page.corpus_generation } : { tier: 'none' as const, corpusGeneration: null }) } });+ const { flipRetireMaxRows } = await readPageBodyPolicy(tx);+ const applied = await withPageTierKept(tx, { sourceId: page.source_id, slug: page.slug }, () => applySegmentedBody(tx, { slug: page.slug, sourceId: page.source_id,+ pageId: page.id, prepared, promotion: false, retireMaxRows: flipRetireMaxRows,+ meta: { title, type: page.type as never, frontmatter, page_kind: page.page_kind as never } }));+ if (applied.complete) await sealSegmentedPage(tx, page.id);+ else unsealed = true;+ } else if (pageChanged) {+ await tx.executeRaw('UPDATE pages SET title=$2,frontmatter=$3::text::jsonb,compiled_truth=$4,timeline=$5,content_hash=$6,updated_at=now() WHERE id=$1',+ [page.id, title, JSON.stringify(frontmatter), compiled, timeline, contentHash({ title, type: page.type, compiled_truth: compiled, timeline, frontmatter, tags: page.tags } as never)]);+ }+ const rawData = new Map<number, unknown>();+ for (const [index, result] of planned.results.entries()) {+ const field = fieldOf(index);+ if (!result.spans.length || field.store === 'page' || field.store === 'frontmatter') continue;+ if (field.store === 'raw_data') {+ if (!rawData.has(field.id!)) rawData.set(field.id!, (await tx.executeRaw<{ data: unknown }>('SELECT data FROM raw_data WHERE id=$1', [field.id]))[0]?.data);+ rawData.set(field.id!, withLeaf(rawData.get(field.id!), field.path!, result.next));+ continue;+ }+ const vectors = field.store === 'facts' || field.store === 'takes' ? ',embedding=NULL,embedded_at=NULL,embedding_model=NULL,embedded_text_hash=NULL' : '';+ await tx.executeRaw(`UPDATE ${field.store} SET ${field.column}=$2${vectors}${field.store === 'takes' ? ',updated_at=now()' : ''} WHERE id=$1`, [field.id, result.next]);+ }+ for (const [id, data] of rawData) await tx.executeRaw('UPDATE raw_data SET data=$2::text::jsonb WHERE id=$1', [id, JSON.stringify(data)]);+ if (!pageChanged) {+ // Rows alone changed: a new revision tells caches and projections the page's knowledge moved.+ await tx.executeRaw('UPDATE pages SET knowledge_revision=gen_random_uuid() WHERE id=$1', [page.id]);+ if (page.body_storage === 'segmented') await sealSegmentedPage(tx, page.id);+ }+ const values = valuesOf(planned);+ if (page.body_storage !== 'segmented') {+ await tx.executeRaw(`DELETE FROM content_chunks c WHERE c.page_id=$1 AND EXISTS (SELECT 1 FROM unnest($2::text[]) v+ WHERE strpos(c.chunk_text,v)>0 OR strpos(COALESCE(c.doc_comment,''),v)>0)`, [page.id, values]);+ await queuePageProjection(tx, page.source_id, page.slug, 'secret_spans');+ }+ await tx.executeRaw(`DELETE FROM page_segment_derivations d WHERE d.page_id=$1 AND d.output IS NOT NULL+ AND EXISTS (SELECT 1 FROM unnest($2::text[]) v WHERE strpos(d.output::text,v)>0)`,+ [page.id, [...new Set([...values, ...values.map(value => JSON.stringify(value).slice(1, -1))])]]);+ await tx.executeRaw('DELETE FROM page_view_digests WHERE page_id=$1', [page.id]);+ return { unsealed };+}++interface Residual { total: number; rules: Record<string, number>; stores: Record<string, number> }++/** Matches of the selected rules still stored for the page, by rule and by store (chunks and derivations included). */+async function residualScan(engine: BrainEngine, sourceId: string, slug: string, rules: SecretRuleSet): Promise<Residual> {+ const out: Residual = { total: 0, rules: {}, stores: {} };+ const add = (store: string, text: string) => {+ for (const [rule, n] of Object.entries(residualMatches(text, rules))) {+ out.total += n; out.rules[rule] = (out.rules[rule] ?? 0) + n; out.stores[store] = (out.stores[store] ?? 0) + n;+ }+ };+ const read = await readFields(engine, sourceId, slug);+ if (typeof read === 'string') return out;+ for (const field of read.fields) add(field.store, field.text);+ for (const row of await engine.executeRaw<{ chunk_text: string; doc_comment: string | null }>(+ 'SELECT chunk_text,doc_comment FROM content_chunks WHERE page_id=$1 -- chunk-visibility-ok: the residual scan reads every stored chunk row, staged and retired included', [read.page.id])) add('content_chunks', `${row.chunk_text}\n${row.doc_comment ?? ''}`);+ for (const row of await engine.executeRaw<{ output: unknown }>('SELECT output FROM page_segment_derivations WHERE page_id=$1 AND output IS NOT NULL', [read.page.id])) {+ for (const leaf of leaves(row.output)) add('page_segment_derivations', leaf.text);+ }+ return out;+}++const sourcesOf = (engine: BrainEngine, selection: Selection) => engine.executeRaw<{ id: string; incarnation: string }>(+ 'SELECT id, incarnation::text AS incarnation FROM sources WHERE id=ANY($1::text[]) ORDER BY id', [selection.source_ids]);++const hashOf = async (engine: BrainEngine, scope: RepairScope, selection: Selection, pages: PlannedPage[]) =>+ previewHash({ kind: 'secret-spans-v1', brain_id: scope.brain_id, sources: await sourcesOf(engine, selection), selection,+ pages: pages.map(page => ({ id: page.id, slug: page.slug, revision: page.revision, generation: page.generation, digest: page.digest })) });++/** Pages already redacted under `hash` (a resumed apply), by page id. */+async function receiptsOf(engine: BrainEngine, hash: string, pages: readonly PlannedPage[]): Promise<Set<number>> {+ const rows = await engine.executeRaw<{ fingerprint: string }>('SELECT fingerprint FROM op_checkpoints WHERE op=$1 AND fingerprint=ANY($2::text[])',+ [SECRET_SPANS_RECEIPT_OP, pages.map(page => receiptKey(hash, page.id))]);+ return new Set(rows.map(row => Number(row.fingerprint.slice(hash.length + 1))));+}++function selectionOf(scope: RepairScope, slugs: readonly string[] | undefined, rules: readonly string[] | undefined): Selection {+ if (scope.source_ids.length !== 1) {+ throw new OperationError('invalid_params', 'gbrain repair secret-spans redacts pages of one source; name it with --source.',+ 'Pass --source with the source id and one --slug per page to preview; gbrain sources list shows the source ids.', DOCS);+ }+ const selected = [...new Set((slugs ?? []).map(slug => slug.trim()).filter(Boolean))].sort();+ if (!selected.length) {+ throw new OperationError('invalid_params', 'gbrain repair secret-spans redacts only the pages you name; pass --slug <slug> (repeatable).',+ `Add one --slug per page to gbrain repair secret-spans --source ${scope.source_ids[0]}, then preview.`, DOCS);+ }+ return { source_ids: scope.source_ids, slugs: selected, rules: resolveSecretRules(rules).rules };+}++function item(page: PlannedPage, hash: string, rules: string[], last: boolean): SpanItem {+ return { cursor: { phase: 0, id: page.id }, source_id: page.source_id, slug: page.slug, chars: 0, action: changeDetail(page), page, hash, rules, last };+}++export const secretSpansRepair: RepairHandler = {+ kind: 'secret-spans',+ publication: 'projection',+ async plan(engine, scope, after, opts): Promise<RepairPlan> {+ if (opts?.remote !== false) throw trustedCliRequired('gbrain repair secret-spans reads and rewrites stored credentials, so it runs only from the trusted local CLI on the brain host.');+ const selection = selectionOf(scope, opts.slugs, opts.rules);+ const rules = resolveSecretRules(selection.rules);+ const command = previewCommand(selection);+ const sourceId = selection.source_ids[0]!;+ if (!opts.apply) {+ const planned: PlannedPage[] = [];+ const listing: RepairListing[] = [];+ const residuals: Record<string, number> = { pages_to_redact: 0, spans: 0, clean: 0, refused: 0, retained_versions: 0 };+ for (const slug of selection.slugs) {+ const read = await readFields(engine, sourceId, slug);+ if (typeof read === 'string') {+ residuals.refused!++;+ listing.push({ item: `${sourceId}:${slug}`, class: 'refused', detail: `${read}: ${REFUSAL_TEXT[read]}` });+ continue;+ }+ const plan = planPage(read, rules);+ const label = `${sourceId}:${slug} (revision ${read.page.revision})`;+ if (plan.results.some(result => !result.proven)) {+ residuals.refused!++;+ listing.push({ item: label, class: 'refused', detail: `proof_failed: ${REFUSAL_TEXT.proof_failed}` });+ continue;+ }+ if (!changedOf(plan).length) { residuals.clean!++; continue; }+ const [versions] = await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM page_versions WHERE page_id=$1', [read.page.id]);+ const page = summaryOf(plan, Number(versions?.n ?? 0));+ planned.push(page);+ residuals.pages_to_redact!++;+ residuals.spans! += spansOf(page);+ residuals.retained_versions! += page.versions;+ listing.push({ item: label, class: 'redact', detail: changeDetail(page) });+ }+ const hash = await hashOf(engine, scope, selection, planned);+ if (planned.length) await saveApprovedSet<PlannedPage & { selection: Selection }>(engine, { command: 'secret-spans', hash }, planned.map(page => ({ ...page, selection })));+ return { items: planned.map((page, index) => item(page, hash, selection.rules, index === planned.length - 1)), preview_hash: hash, residuals, listing,+ details: { rules: selection.rules, pages: planned } };+ }+ if (!opts.expect) {+ throw new OperationError('invalid_params', 'gbrain repair secret-spans --apply redacts only the set a preview printed.',+ `Preview first: ${command} — show the user the listing, then run the apply command it prints: ${command} --apply --expect <preview-hash>`, DOCS);+ }+ const expect = opts.expect;+ const approved = (await loadApprovedSet<PlannedPage & { selection: Selection }>(engine, { command: 'secret-spans', hash: expect, previewCommand: command,+ emptyHash: () => hashOf(engine, scope, selection, []) })).items;+ if (approved.some(page => digest(page.selection) !== digest(selection))) throw previewChangedError(expect, command);+ const pages = approved.map(({ selection: _selection, ...page }) => page);+ const done = await receiptsOf(engine, expect, pages);+ // Every page not yet redacted under this hash still holds exactly what the user saw, or nothing is written.+ for (const page of pages) {+ if (done.has(page.id)) continue;+ const read = await readFields(engine, page.source_id, page.slug);+ if (typeof read === 'string' || read.page.id !== page.id || planPage(read, rules).digest !== page.digest) throw previewChangedError(expect, command);+ }+ return { items: pages.map((page, index) => item(page, expect, selection.rules, index === pages.length - 1)).filter(entry => afterCursor(entry.cursor, after)),+ preview_hash: expect, residuals: {} };+ },+ async apply(ctx, entry, opts): Promise<RepairItemOutcome> {+ if (ctx.remote !== false) throw trustedCliRequired('gbrain repair secret-spans runs only from the trusted local CLI on the brain host.');+ const { page, hash, last } = entry as SpanItem;+ const rules = resolveSecretRules((entry as SpanItem).rules);+ const detail = { spans: spansOf(page), fields: page.fields.length, rules: ruleText(page.rules) };+ try {+ if ((await receiptsOf(ctx.engine, hash, [page])).has(page.id)) return { applied: true, outcome: 'redacted', detail };+ const written = await ctx.engine.transaction(async tx => {+ const managed = await readManagedMode(tx, { refusal: { subject: 'repair secret-spans' } });+ const attribution = await maintenanceAttribution(tx);+ const write = () => withTrustKeep(tx, ['pages', 'facts', 'takes', 'timeline_entries'], async () => {+ await tx.lockPageKeys([{ sourceId: page.source_id, slug: page.slug }]);+ const read = await readFields(tx, page.source_id, page.slug);+ if (typeof read === 'string') return { refused: read };+ const planned = planPage(read, rules);+ if (read.page.id !== page.id || planned.digest !== page.digest) return { refused: 'changed_since_preview' };+ const { unsealed } = await writePage(tx, planned);+ const [after] = await tx.executeRaw<{ revision: string }>('SELECT knowledge_revision::text AS revision FROM pages WHERE id=$1', [page.id]);+ const receipt = { kind: 'secret-spans', hash, source_id: page.source_id, slug: page.slug, page_id: page.id, revision_before: page.revision,+ revision_after: after?.revision ?? null, rules: rules.rules, fields: changedOf(planned).map(result => ({ key: result.key, old_sha256: result.old_sha256,+ new_sha256: result.new_sha256, spans_sha256: result.spans_sha256, spans: result.spans.map(span => [span.start, span.end, span.rule, span.echo ? 1 : 0]) })) };+ await tx.executeRaw(`INSERT INTO op_checkpoints(op,fingerprint,completed_keys,updated_at) VALUES ($1,$2,$3::text::jsonb,now())+ ON CONFLICT (op,fingerprint) DO UPDATE SET completed_keys=EXCLUDED.completed_keys,updated_at=now()`,+ [SECRET_SPANS_RECEIPT_OP, receiptKey(hash, page.id), JSON.stringify([receipt])]);+ return { receipt, unsealed, embedSkip: !!(read.page.frontmatter ?? {}).embed_skip };+ });+ return managed ? withCoordinatedWrite(tx, [page.source_id], write, attribution) : withWriteAttribution(tx, attribution, write);+ });+ if ('refused' in written) {+ return { applied: false, outcome: 'changed_since_preview', reason: written.refused === 'changed_since_preview' ? 'the page changed since the preview' : written.refused };+ }+ audit.log(written.receipt);+ let projection = 'sealed';+ try {+ if (page.storage !== 'segmented') await resealSafeChunks(ctx.engine, page.slug, page.source_id);+ else if (written.unsealed) await buildSegmentedProjection(ctx.engine, { sourceId: page.source_id, slug: page.slug });+ } catch { projection = 'queued'; }+ const embedded = opts?.embed && !written.embedSkip+ ? (await embedStalePages(ctx.engine, [page.slug], page.source_id).catch(() => ({ embedded: 0 }))).embedded : 0;+ const residual = await residualScan(ctx.engine, page.source_id, page.slug, rules);+ return { applied: true, outcome: 'redacted', detail: { ...detail, projection, embedded, residual: residual.total,+ ...(residual.total ? { residual_rules: ruleText(residual.rules), residual_stores: ruleText(residual.stores) } : {}) } };+ } finally {+ if (last) await clearApprovedSet(ctx.engine, { command: 'secret-spans', hash });+ }+ },+ render(details) {+ return ((details.pages ?? []) as PlannedPage[]).flatMap(page => [` ${page.source_id}:${page.slug} (${page.storage}): ${changeDetail(page)}`,+ ...page.fields.map(field => ` ${field.key}: ${field.spans} span(s), sha256 ${field.old_sha256.slice(0, 12)} -> ${field.new_sha256.slice(0, 12)}, spans ${field.spans_sha256.slice(0, 12)}`)]);+ },+ async report(ctx, scope, result, opts) {+ if (result.mode !== 'apply' || !opts.slugs?.length || scope.source_ids.length !== 1) return {};+ const rules = resolveSecretRules(opts.rules);+ const byRule: Record<string, number> = {};+ let total = 0;+ const slugs = [...new Set(opts.slugs)].sort();+ for (const slug of slugs) {+ const scan = await residualScan(ctx.engine, scope.source_ids[0]!, slug, rules);+ total += scan.total;+ for (const [rule, n] of Object.entries(scan.rules)) byRule[rule] = (byRule[rule] ?? 0) + n;+ }+ return { verification: { rescanned_pages: slugs.length, residual_matches: total, residual_by_rule: byRule } };+ },+};diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.tsindex 8e29e51fbd..2cbbeeac47 100644--- a/src/core/secret-scan.ts+++ b/src/core/secret-scan.ts@@ -1091,6 +1091,9 @@ function sha256Hex(value: string): string {return createHash('sha256').update(value).digest('hex');}+/** Every named rule the default scan applies (the opt-in `high_entropy_assignment` and `labeled_credential` heuristics excluded), sorted. */+export const NAMED_SECRET_RULES: readonly string[] = [...new Set([...CORE_PATTERNS.map((p) => p.name), 'private_key_pem'])].sort();+const PATTERN_SINCE: ReadonlyMap<string, string> = new Map([...CORE_PATTERNS.flatMap((p): Array<[string, string]> => (p.since ? [[p.name, p.since]] : [])),['private_key_pem', SECURITY_WAVE_SINCE],diff --git a/src/core/secret-spans.ts b/src/core/secret-spans.tsnew file mode 100644index 0000000000..75a42c0481--- /dev/null+++ b/src/core/secret-spans.ts@@ -0,0 +1,142 @@+/**+ * Span-level secret redaction of stored text, the pure half of `gbrain repair+ * secret-spans` (repair/secret-spans.ts). Every field of one page is planned+ * with the secret scanner (`planRedaction`, the rules the operator selects)+ * against one shared echo dictionary, so a value the scanner claimed in one+ * field is also replaced where it reappears bare in another, as the corpus+ * writer does. A claim whose text is only redaction markers is dropped, so an+ * already-redacted field is a fixed point.+ *+ * The proof: the span list is computed here, independently of+ * `applyRedaction`; the new text is the old text with `<REDACTED:rule>`+ * spliced over exactly those spans, and `applyRedaction` over the same+ * filtered plan must return byte-identical text. A field whose two results+ * differ, or whose spans are not sorted, disjoint and inside the text, is not+ * proven, and the caller refuses its page.+ */+import { createHash } from 'node:crypto';+import { opError } from './ops/contract.ts';+import { applyRedaction, NAMED_SECRET_RULES, planRedaction, type EchoDictionary, type RedactionPlan, type ScanOpts } from './secret-scan.ts';++/** The opt-in heuristics: selectable by name, never part of the default rule set. */+export const BROAD_SECRET_RULES: readonly string[] = ['high_entropy_assignment', 'labeled_credential'];++const MARKER_ONLY = /^(?:<REDACTED:[^<>\s]+>)+$/;++export interface SecretRuleSet { rules: string[]; scan: ScanOpts }++/** The selected rules, sorted (default: every named rule), and the scan options they need; unknown names refuse. */+export function resolveSecretRules(names?: readonly string[]): SecretRuleSet {+ const known = [...NAMED_SECRET_RULES, ...BROAD_SECRET_RULES];+ const picked = names?.length ? [...new Set(names.map(name => name.trim()))].sort() : [...NAMED_SECRET_RULES];+ const unknown = picked.filter(name => !known.includes(name));+ if (unknown.length) {+ throw opError('invalid_params', `Unknown secret rule${unknown.length > 1 ? 's' : ''}: ${unknown.map(name => JSON.stringify(name)).join(', ')}; nothing was scanned.`,+ `--rules takes a comma-separated list of: ${known.join(', ')}. Leave it out to apply every named rule (the opt-in ${BROAD_SECRET_RULES.join(' and ')} heuristics only run when named).`);+ }+ return { rules: picked, scan: { highEntropy: picked.includes('high_entropy_assignment'), labeledCredentials: picked.includes('labeled_credential') } };+}++export interface SecretSpan { start: number; end: number; rule: string; echo: boolean }++export interface FieldRedaction {+ key: string;+ /** The text with every span replaced by its marker (the old text when there are no spans). */+ next: string;+ spans: SecretSpan[];+ old_sha256: string;+ new_sha256: string;+ /** SHA-256 over the span list (offsets, rule, echo flag), never over a value. */+ spans_sha256: string;+ /** False when the independent splice and `applyRedaction` disagree, or a span is malformed. */+ proven: boolean;+}++export const sha256Text = (text: string): string => createHash('sha256').update(text).digest('hex');+const markerOf = (rule: string) => `<REDACTED:${rule}>`;++/**+ * The spans `applyRedaction` replaces for these claims and this dictionary: each claim (an overlap's uncovered tail as+ * its own span) and, inside the gaps between claims, every dictionary occurrence leftmost then longest.+ */+function spansOf(text: string, claimed: RedactionPlan['claimed'], echo: EchoDictionary): SecretSpan[] {+ const values = [...echo].sort((a, b) => b[0].length - a[0].length);+ const next = values.map(([value]) => text.indexOf(value));+ const out: SecretSpan[] = [];+ const gap = (from: number, to: number) => {+ let at = from;+ while (at < to) {+ let best = -1, bestPos = to;+ for (let k = 0; k < values.length; k++) {+ let p = next[k]!;+ if (p === -1) continue;+ if (p < at) { p = text.indexOf(values[k]![0], at); next[k] = p; if (p === -1) continue; }+ if (p >= bestPos || p + values[k]![0].length > to) continue;+ best = k; bestPos = p;+ }+ if (best === -1) return;+ out.push({ start: bestPos, end: bestPos + values[best]![0].length, rule: values[best]![1], echo: true });+ at = bestPos + values[best]![0].length;+ }+ };+ let cur = 0;+ for (const c of claimed) {+ if (c.end <= cur) continue;+ if (c.abs > cur) gap(cur, c.abs);+ out.push({ start: Math.max(c.abs, cur), end: c.end, rule: c.pattern, echo: false });+ cur = c.end;+ }+ gap(cur, text.length);+ return out;+}++function splice(text: string, spans: readonly SecretSpan[]): string {+ const parts: string[] = [];+ let at = 0;+ for (const span of spans) { parts.push(text.slice(at, span.start), markerOf(span.rule)); at = span.end; }+ parts.push(text.slice(at));+ return parts.join('');+}++const wellFormed = (text: string, spans: readonly SecretSpan[]) =>+ spans.every((span, i) => span.start < span.end && span.end <= text.length && (i === 0 || spans[i - 1]!.end <= span.start)+ && !MARKER_ONLY.test(text.slice(span.start, span.end)));++/**+ * Plans every field of one page, in the given order, against one shared echo dictionary restricted to the selected+ * rules. Returns one result per field, changed or not.+ */+export function redactFields(fields: ReadonlyArray<{ key: string; text: string }>, rules: SecretRuleSet): FieldRedaction[] {+ const shared: EchoDictionary = new Map();+ const plans = fields.map(field => planRedaction(field.text, { ...rules.scan, echoValues: shared }));+ const selected = new Set(rules.rules);+ const echo: EchoDictionary = new Map([...shared].filter(([value, rule]) => selected.has(rule) && !MARKER_ONLY.test(value)));+ return fields.map((field, i) => {+ const plan = plans[i]!;+ const claimed = plan.claimed.filter(c => selected.has(c.pattern) && !MARKER_ONLY.test(field.text.slice(c.abs, c.end)));+ const spans = spansOf(field.text, claimed, echo);+ const next = spans.length ? splice(field.text, spans) : field.text;+ const proven = wellFormed(field.text, spans) && applyRedaction({ ...plan, claimed, echoValues: echo }) === next;+ return { key: field.key, next, spans, old_sha256: sha256Text(field.text), new_sha256: sha256Text(next), proven,+ spans_sha256: sha256Text(JSON.stringify(spans.map(span => [span.start, span.end, span.rule, span.echo ? 1 : 0]))) };+ });+}++/** Matches of the selected rules a fresh scan still finds in `text`, by rule (marker-only claims excluded). */+export function residualMatches(text: string, rules: SecretRuleSet): Record<string, number> {+ const selected = new Set(rules.rules);+ const out: Record<string, number> = {};+ for (const c of planRedaction(text, rules.scan).claimed) {+ if (!selected.has(c.pattern) || MARKER_ONLY.test(text.slice(c.abs, c.end))) continue;+ out[c.pattern] = (out[c.pattern] ?? 0) + 1;+ }+ return out;+}++/** Spans by rule, and how many of them are echoes. */+export function spanCounts(spans: Iterable<SecretSpan>): { rules: Record<string, number>; echo: number } {+ const rules: Record<string, number> = {};+ let echo = 0;+ for (const span of spans) { rules[span.rule] = (rules[span.rule] ?? 0) + 1; if (span.echo) echo++; }+ return { rules, echo };+}diff --git a/test/fix-wave-5-integration.test.ts b/test/fix-wave-5-integration.test.tsindex 86afe325d1..064a316e54 100644--- a/test/fix-wave-5-integration.test.ts+++ b/test/fix-wave-5-integration.test.ts@@ -201,7 +201,7 @@ for (const backend of testBackends()) {'uncommitted-deletions': 'gbrain repair uncommitted-deletions',// GBRA-161: the generations preview carries its ledger kind and the restore window, so the printed command runs as is.generations: 'gbrain repair generations --kind atom --since 30d', 'fact-claim-dates': 'gbrain repair fact-claim-dates',- provenance: 'gbrain repair provenance' };+ provenance: 'gbrain repair provenance', 'secret-spans': 'gbrain repair secret-spans' };// The post-upgrade banner names both findings with the kind's read-only preview.const banner = await postUpgradeRecoveryBanner(engine, 'host');diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txtindex 2341121142..51f073d68c 100644--- a/test/postgres-unit-arms.txt+++ b/test/postgres-unit-arms.txt@@ -443,6 +443,7 @@ test/repair-frontmatter.test.tstest/repair-ghost-database-ahead-gbra133.test.tstest/repair-receipt-compaction.test.tstest/repair-rename-identity-6729.test.ts+test/repair-secret-spans.test.tstest/retrieval-coverage-notices.test.tstest/schema-missing-tables.test.tstest/schema-replay-deadlock-6143.test.tsdiff --git a/test/repair-explicit-only.test.ts b/test/repair-explicit-only.test.tsindex f548167c08..eb3f353068 100644--- a/test/repair-explicit-only.test.ts+++ b/test/repair-explicit-only.test.ts@@ -24,7 +24,7 @@ import { REPAIR_HELP, parseRepairArgs, runRepairCommand } from '../src/commands/import { _resetCliExitVerdictForTests, currentExitCode } from '../src/core/cli-force-exit.ts';import { withEnv } from './helpers/with-env.ts';-const EXPLICIT: RepairKind[] = ['accepted-takes', 'google-file-modes', 'stale-atoms', 'atoms-restore', 'extractor-facts', 'conversation-labels', 'captured-facts', 'redacted-facts', 'loop-facts', 'ontology-facts', 'facts-rehome', 'consolidate-ties', 'duplicate-facts', 'orphan-children', 'failed-writes', 'split-facts', 'frontmatter', 'timeline-comments', 'timeline-order', 'managed-sync-orphans', 'content-receipt-root', 'frontmatter-keys', 'file-drift', 'stuck-recovery', 'ghost-pages', 'origin-twins', 'receipt-compaction', 'aliases', 'rename-identity', 'overwritten-edits', 'legacy-twins', 'missing-tables', 'url-clone-fences', 'segmented-mirror', 'uncommitted-deletions', 'generations', 'fact-claim-dates', 'provenance'];+const EXPLICIT: RepairKind[] = ['accepted-takes', 'google-file-modes', 'stale-atoms', 'atoms-restore', 'extractor-facts', 'conversation-labels', 'captured-facts', 'redacted-facts', 'loop-facts', 'ontology-facts', 'facts-rehome', 'consolidate-ties', 'duplicate-facts', 'orphan-children', 'failed-writes', 'split-facts', 'frontmatter', 'timeline-comments', 'timeline-order', 'managed-sync-orphans', 'content-receipt-root', 'frontmatter-keys', 'file-drift', 'stuck-recovery', 'ghost-pages', 'origin-twins', 'receipt-compaction', 'aliases', 'rename-identity', 'overwritten-edits', 'legacy-twins', 'missing-tables', 'url-clone-fences', 'segmented-mirror', 'uncommitted-deletions', 'generations', 'fact-claim-dates', 'provenance', 'secret-spans'];// GBRA-161: the generations preview carries its ledger kind and the restore window, so the printed command runs as is.const PREVIEW = (kind: RepairKind): string => kind === 'generations' ? 'gbrain repair generations --kind atom --since 30d' : `gbrain repair ${kind}`;let engine: PGLiteEngine;diff --git a/test/repair-secret-spans.test.ts b/test/repair-secret-spans.test.tsnew file mode 100644index 0000000000..36fbe95167--- /dev/null+++ b/test/repair-secret-spans.test.ts@@ -0,0 +1,256 @@+/**+ * GST-327: `gbrain repair secret-spans` redacts only the secret spans in every database copy of the named pages.+ *+ * Repo-less brain (no worktree, the database is the record): an inline page whose title, frontmatter value, body,+ * timeline rows, fact, take, link context and raw data hold credentials (one glued to an existing marker, one only an+ * echo), a segmented embed_skip page over 1 MB, a page whose timeline holds an impossible date, a clean page and a+ * missing slug in one batch. Managed brain: a held page is redacted in the database while its file, its hold and Git+ * stay as they are. Runs on PGLite, and on Postgres when DATABASE_URL is set.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { execFileSync } from 'node:child_process';+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { importFromContent } from '../src/core/import-file.ts';+import { materializeSegmentedBody } from '../src/core/page-body/ingest.ts';+import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';+import { writeGitHold, GIT_HOLD_OP } from '../src/core/persistence/sync-holds.ts';+import { resolveRepairScope, runRepair, type RepairResult } from '../src/core/repair/core.ts';+import { AUTO_REPAIR_REGISTRY, repairRunner } from '../src/core/repair/registry.ts';+import { secretSpansRepair } from '../src/core/repair/secret-spans.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';+import { managedBrain, settleManagedGitEffects } from './helpers/managed-brain.ts';+import { durableGitRepo } from './helpers/git-publication.ts';+import { withEnv } from './helpers/with-env.ts';+import { requirePostgresTestDatabase, testBackends } from './helpers/test-backends.ts';+import { parseRepairArgs } from '../src/commands/repair.ts';++const logger = { info() {}, warn() {}, error() {} };+// Obviously fake key shapes, assembled so no literal key sits in the source.+const OPENAI = ['sk', 'proj', 'EXAMPLExNOTxREALxEXAMPLExNOTxREAL'].join('-');+const AWS = 'AKIA' + 'EXAMPLENOTREAL00';+const GITHUB = 'ghp' + '_' + 'EXAMPLEnotREAL'.repeat(3);+const BEARER = 'EXAMPLEtokenNOTreal' + '1234567890abcdef';+const SECRETS = [OPENAI, AWS, GITHUB, BEARER];+const M = (rule: string) => `<REDACTED:${rule}>`;++const ALPHA_BODY = [`# Alpha Example`, '', `The deploy notes keep the old key ${OPENAI} here.`, `A pasted marker then a key: ${M('openai')}${OPENAI} glued.`,+ `Authorization: Bearer ${BEARER}`, `The token ${BEARER} appears again bare.`, 'Nothing else changes, not even this line.'].join('\n');+const ALPHA_TIMELINE = `## Timeline\n\n- **2024-01-02** | import — Rotated ${AWS} out of the build\n- **2024-03-04** | import — Unrelated entry`;+const ALPHA = `---\ntype: note\ntitle: Alpha Example\nnote_key: "copy of ${OPENAI}"\nowner: alice-example\n---\n\n${ALPHA_BODY}\n\n<!-- timeline -->\n\n${ALPHA_TIMELINE}\n`;+const filler = (tag: string, bytes: number) => Array.from({ length: Math.ceil(bytes / 1900) }, (_, i) =>+ `## ${tag} ${i}\n\n${`Runbook notes for the synthetic staging cluster, step ${i}. `.repeat(30)}\n\n`).join('');+const LARGE_BODY = `# Large Example\n\n${filler('Before', 600_000)}## Old key\n\nThe retired key was ${AWS} and nothing else.\n\n${filler('After', 600_000)}`;+const LARGE = `---\ntype: note\ntitle: Large Example\n---\n\n${LARGE_BODY}`;+const BAD_DATE = `---\ntype: note\ntitle: Bad Date Example\n---\n\nThe github token ${GITHUB} was pasted here.\n`;+const CLEAN = '---\ntype: note\ntitle: Clean Example\n---\n\nNo credentials on this page.\n';++const home = mkdtempSync(join(tmpdir(), 'gbrain-secret-spans-'));+afterAll(() => rmSync(home, { recursive: true, force: true }));++const leaked = (value: unknown) => SECRETS.filter(secret => JSON.stringify(value, (_key, v) => typeof v === 'bigint' ? String(v) : v).includes(secret));+const pageRow = async (engine: BrainEngine, slug: string, sourceId = 'default') => (await engine.executeRaw<Record<string, any>>(+ `SELECT id,title,frontmatter,compiled_truth,timeline,body_storage,knowledge_revision::text AS revision,content_hash,trust_tier FROM pages WHERE source_id=$1 AND slug=$2`, [sourceId, slug]))[0]!;+const snapshotDb = async (engine: BrainEngine) => ({+ pages: await engine.executeRaw('SELECT id,slug,title,frontmatter,compiled_truth,timeline,knowledge_revision::text AS revision,body_generation FROM pages ORDER BY id'),+ timeline: await engine.executeRaw('SELECT id,page_id,date::text AS date,summary,detail,source FROM timeline_entries ORDER BY id'),+ facts: await engine.executeRaw('SELECT id,fact,context,embedded_at FROM facts ORDER BY id'),+ takes: await engine.executeRaw('SELECT id,claim,source FROM takes ORDER BY id'),+ links: await engine.executeRaw('SELECT id,context FROM links ORDER BY id'),+ raw: await engine.executeRaw('SELECT id,data FROM raw_data ORDER BY id'),+ chunks: await engine.executeRaw('SELECT page_id,count(*)::int AS n FROM content_chunks GROUP BY page_id ORDER BY page_id'),+ versions: await engine.executeRaw('SELECT count(*)::int AS n FROM page_versions'),+});++const repoLess: Array<{ name: string; engine: BrainEngine }> = [];+let closePostgres: (() => Promise<void>) | undefined;+beforeAll(() => withEnv({ GBRAIN_HOME: home }, async () => {+ const lite = new PGLiteEngine(); await lite.connect({}); await lite.initSchema();+ repoLess.push({ name: 'pglite', engine: lite });+ if (process.env.DATABASE_URL) {+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL);+ repoLess.push({ name: 'postgres', engine: pg.engine });+ closePostgres = pg.close;+ }+}), 120_000);+afterAll(async () => { for (const { engine } of repoLess) await engine.disconnect(); await closePostgres?.(); });++async function seed(engine: BrainEngine) {+ await engine.setConfig('pages.segmented_storage', 'on');+ for (const [slug, content] of [['notes/alpha-example', ALPHA], ['notes/large-example', LARGE], ['notes/bad-date-example', BAD_DATE], ['notes/clean-example', CLEAN]] as const) {+ await importFromContent(engine, slug, content, { noEmbed: true, sourceId: 'default' });+ }+ // An older brain's size marker (the content gate no longer writes one at this size): the page gets no vectors.+ await engine.executeRaw(`UPDATE pages SET frontmatter=frontmatter||'{"embed_skip":"oversized"}'::jsonb WHERE source_id='default' AND slug='notes/large-example'`);+ // #8049 shape: a stored timeline whose bullet names a day that does not exist (an older writer stored it as text).+ await engine.executeRaw(`UPDATE pages SET timeline=$2 WHERE source_id='default' AND slug=$1`,+ ['notes/bad-date-example', `## Timeline\n\n- **2024-02-30** | import — Moved ${GITHUB} to the vault`]);+ const alpha = await pageRow(engine, 'notes/alpha-example');+ const clean = await pageRow(engine, 'notes/clean-example');+ await engine.executeRaw("INSERT INTO timeline_entries(page_id,date,source,summary,detail) VALUES ($1,'2024-01-02','import',$2,''),($1,'2024-03-04','import','Unrelated entry','')",+ [alpha.id, `Rotated ${AWS} out of the build`]);+ await engine.executeRaw(`INSERT INTO facts(source_id,entity_slug,fact,context,source,kind) VALUES ('default','notes/alpha-example',$1,$2,'test','fact')`,+ [`The CI token is ${GITHUB}.`, `Seen next to ${AWS} in the notes`]);+ await engine.executeRaw(`INSERT INTO takes(page_id,row_num,claim,kind,holder,source) VALUES ($1,1,$2,'take','alice-example',$3)`,+ [alpha.id, `Rotating ${OPENAI} was overdue`, 'notes']);+ await engine.executeRaw(`INSERT INTO links(from_page_id,to_page_id,link_type,context) VALUES ($1,$2,'mentions',$3)`, [alpha.id, clean.id, `see ${AWS} near the clean page`]);+ await engine.executeRaw(`INSERT INTO raw_data(page_id,source,data) VALUES ($1,'test',$2::text::jsonb)`,+ [alpha.id, JSON.stringify({ request: { headers: { authorization: `Bearer ${BEARER}` }, ids: [1, 2] }, note: 'kept as is' })]);+}++describe('GST-327 gbrain repair secret-spans on a repo-less brain', () => {+ test('preview writes nothing; refusals; preview_changed; the apply redacts only the spans; a second preview is empty', () => withEnv({ GBRAIN_HOME: home }, async () => {+ for (const { name, engine } of repoLess) {+ {+ await seed(engine);+ expect(AUTO_REPAIR_REGISTRY.map(spec => spec.kind)).not.toContain('secret-spans');+ const scope = await resolveRepairScope(engine, 'default');+ const slugs = ['notes/alpha-example', 'notes/large-example', 'notes/bad-date-example', 'notes/clean-example', 'notes/missing-example'];+ const preview = () => repairRunner(engine, { apply: false, logger }).then(runner => runner.run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs }));+ const apply = (hash: string) => repairRunner(engine, { apply: true, logger }).then(runner => runner.run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs, expect: hash }));++ // Explicit-only, trusted-local-CLI-only, and an apply needs a preview.+ await expect(repairRunner(engine, { apply: false, logger }).then(runner => runner.run('secret-spans', scope, { slugs }))).rejects.toThrow(/explicit/);+ const remote = { engine, config: { engine: engine.kind }, logger, dryRun: true, remote: true, sourceId: 'default' } as never;+ await expect(runRepair(remote, secretSpansRepair, scope, { apply: false, explicit: true, slugs })).rejects.toMatchObject({ code: 'permission_denied' });+ await expect(repairRunner(engine, { apply: true, logger }).then(runner => runner.run('secret-spans', scope, { explicit: true, slugs }))).rejects.toThrow(/preview/i);+ await expect(repairRunner(engine, { apply: false, logger }).then(runner => runner.run('secret-spans', scope, { explicit: true }))).rejects.toThrow(/--slug/);++ const before = await snapshotDb(engine);+ const first = await preview();+ expect(first.affected, name).toBe(3);+ expect(first.listing!.map(entry => [entry.class, entry.item.split(' ')[0]])).toEqual([+ ['redact', 'default:notes/alpha-example'], ['redact', 'default:notes/bad-date-example'], ['redact', 'default:notes/large-example'],+ ['refused', 'default:notes/missing-example']]);+ expect(first.residuals).toMatchObject({ pages_to_redact: 3, clean: 1, refused: 1 });+ expect(first.apply_command).toContain(`--apply --expect ${first.preview_hash}`);+ expect(leaked(first)).toEqual([]);+ expect(await snapshotDb(engine)).toEqual(before);++ // A fact edit after the preview moves the hash: nothing is written.+ await engine.executeRaw(`UPDATE facts SET context=context||' (edited)' WHERE entity_slug='notes/alpha-example'`);+ await expect(apply(first.preview_hash!)).rejects.toMatchObject({ code: 'preview_changed' });+ const edited = await snapshotDb(engine);+ expect(edited.pages).toEqual(before.pages);++ const second = await preview();+ expect(second.preview_hash).not.toBe(first.preview_hash);+ const alphaBefore = await pageRow(engine, 'notes/alpha-example');+ const cleanBefore = await pageRow(engine, 'notes/clean-example');+ const largeBefore = await pageRow(engine, 'notes/large-example');+ const largeBody = await materializeSegmentedBody(engine, Number(largeBefore.id), 'default');+ expect(largeBody.compiled_truth).toContain(AWS);+ // The apply embeds what it changed (no key here, so each attempt logs), except the embed_skip page.+ const logged: string[] = [];+ const write = process.stderr.write;+ process.stderr.write = ((chunk: string | Uint8Array) => { logged.push(String(chunk)); return true; }) as typeof process.stderr.write;+ const result: RepairResult = await apply(second.preview_hash!).finally(() => { process.stderr.write = write; });+ expect(logged.some(line => line.includes('notes/alpha-example'))).toBe(true);+ expect(logged.some(line => line.includes('notes/large-example'))).toBe(false);+ expect(result.applied, name).toBe(3);+ expect(result.outcomes).toEqual({ redacted: 3 });+ for (const outcome of result.outcome_items ?? []) expect(outcome.detail?.residual, outcome.item).toBe(0);+ expect(result.verification).toMatchObject({ rescanned_pages: 5, residual_matches: 0 });+ expect(leaked(result)).toEqual([]);++ // Only the spans changed, everywhere the page is stored.+ const alpha = await pageRow(engine, 'notes/alpha-example');+ expect(alpha.compiled_truth).toBe(ALPHA_BODY.replace(`${M('openai')}${OPENAI}`, `${M('openai')}${M('openai')}`)+ .replaceAll(OPENAI, M('openai')).replaceAll(BEARER, M('bearer')));+ expect(alpha.timeline).toBe(ALPHA_TIMELINE.replace(AWS, M('aws_access_key')));+ expect(alpha.frontmatter).toEqual({ ...alphaBefore.frontmatter, note_key: `copy of ${M('openai')}` });+ expect(alpha.title).toBe('Alpha Example');+ expect(alpha.revision).not.toBe(alphaBefore.revision);+ expect(alpha.trust_tier).toBe(alphaBefore.trust_tier);+ const rows = await snapshotDb(engine);+ expect(rows.timeline.map((row: any) => row.summary).join('\n')).toContain(`Rotated ${M('aws_access_key')} out of the build`);+ expect(rows.timeline.map((row: any) => row.date)).toEqual(edited.timeline.map((row: any) => row.date));+ expect(rows.facts).toEqual([expect.objectContaining({ fact: `The CI token is ${M('github_token')}.`, context: `Seen next to ${M('aws_access_key')} in the notes (edited)`, embedded_at: null })]);+ expect(rows.takes).toEqual([expect.objectContaining({ claim: `Rotating ${M('openai')} was overdue`, source: 'notes' })]);+ expect(rows.links).toEqual([expect.objectContaining({ context: `see ${M('aws_access_key')} near the clean page` })]);+ expect(rows.raw).toEqual([expect.objectContaining({ data: { request: { headers: { authorization: `Bearer ${M('bearer')}` }, ids: [1, 2] }, note: 'kept as is' } })]);+ expect(leaked(rows)).toEqual([]);+ expect(rows.versions).toEqual(before.versions);+ expect(leaked(await engine.executeRaw('SELECT chunk_text FROM content_chunks'))).toEqual([]);++ // The segmented embed_skip page: still segmented, flag kept, body identical but for the span.+ const large = await pageRow(engine, 'notes/large-example');+ expect(large.body_storage).toBe('segmented');+ expect(large.frontmatter.embed_skip).toBe('oversized');+ expect(large.revision).not.toBe(largeBefore.revision);+ expect(await materializeSegmentedBody(engine, Number(large.id), 'default'))+ .toEqual({ compiled_truth: largeBody.compiled_truth.replace(AWS, M('aws_access_key')), timeline: largeBody.timeline });++ // Every rewritten page is sealed again at its new revision, with chunks rebuilt from the redacted body.+ const sealed = await engine.executeRaw<{ slug: string; sealed: boolean; chunks: number }>(`SELECT p.slug,p.text_projection_revision=p.knowledge_revision AS sealed,+ (SELECT count(*)::int FROM content_chunks c WHERE c.page_id=p.id) AS chunks FROM pages p WHERE p.slug=ANY($1::text[]) ORDER BY p.slug`,+ [['notes/alpha-example', 'notes/large-example', 'notes/bad-date-example']]);+ for (const page of sealed) { expect(page.sealed, page.slug).toBe(true); expect(page.chunks, page.slug).toBeGreaterThan(0); }+ // The receipts are value-free.+ const receipts = await engine.executeRaw("SELECT completed_keys FROM op_checkpoints WHERE op='secret-spans-receipt'");+ expect(receipts).toHaveLength(3);+ expect(leaked(receipts)).toEqual([]);++ // The impossible-date page was redacted in the same batch, its date left as stored.+ const bad = await pageRow(engine, 'notes/bad-date-example');+ expect(bad.timeline).toBe(`## Timeline\n\n- **2024-02-30** | import — Moved ${M('github_token')} to the vault`);++ // The clean page is untouched, and a second preview finds nothing.+ expect(await pageRow(engine, 'notes/clean-example')).toEqual(cleanBefore);+ const again = await preview();+ expect(again.affected).toBe(0);+ expect(again.residuals).toMatchObject({ pages_to_redact: 0, clean: 4, refused: 1 });+ }+ }+ }), 300_000);+});++for (const backend of testBackends()) describe(`GST-327 gbrain repair secret-spans on a managed brain (${backend})`, () => {+ test('a held page is redacted in the database; its file, its hold and Git stay as they are', () => managedBrain(async ({ engine, ctx, root }) => {+ const slug = 'notes/held-example';+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug, source_id: 'default', request_id: randomUUID(),+ content: `---\ntype: note\ntitle: Held Example\n---\n\nThe old key ${OPENAI} sits in this note.\n` } });+ await settleManagedGitEffects(engine);+ const page = await pageRow(engine, slug);+ const [source] = await engine.executeRaw<{ incarnation: string }>("SELECT incarnation::text AS incarnation FROM sources WHERE id='default'");+ await writeGitHold(engine, { source_id: 'default', incarnation: source!.incarnation, path: `${slug}.md`, source_path: `${slug}.md`, slug,+ page_id: Number(page.id), code: 'preparation_stalled', message: 'Preparation exhausted its attempts.', upstream_version: null,+ observed_at: new Date().toISOString(), run_id: null, mode: 'managed', meta: { recovery_version: 1 } });+ const hold = () => engine.executeRaw('SELECT completed_keys FROM op_checkpoints WHERE op=$1', [GIT_HOLD_OP]);+ const holdBefore = await hold();+ expect(holdBefore).toHaveLength(1);+ const file = readFileSync(join(root, `${slug}.md`), 'utf8');+ const commits = () => execFileSync('git', ['-C', root, 'rev-list', '--count', 'HEAD'], { encoding: 'utf8' }).trim();+ const commitsBefore = commits();+ const requests = () => engine.executeRaw('SELECT count(*)::int AS n FROM persistence_requests');+ const requestsBefore = await requests();++ const scope = await resolveRepairScope(engine, 'default');+ const preview = await (await repairRunner(engine, { apply: false, logger })).run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs: [slug] });+ expect(preview.affected).toBe(1);+ const result = await (await repairRunner(engine, { apply: true, logger })).run('secret-spans', scope, { explicit: true, sourceFlag: 'default', slugs: [slug], expect: preview.preview_hash });+ expect(result.outcomes).toEqual({ redacted: 1 });+ expect(result.verification).toMatchObject({ residual_matches: 0 });++ const after = await pageRow(engine, slug);+ expect(after.compiled_truth).toBe(page.compiled_truth.replace(OPENAI, M('openai')));+ expect(after.revision).not.toBe(page.revision);+ expect(await hold()).toEqual(holdBefore);+ expect(readFileSync(join(root, `${slug}.md`), 'utf8')).toBe(file);+ expect(commits()).toBe(commitsBefore);+ expect(await requests()).toEqual(requestsBefore);+ }, { setup: ({ root }) => { writeFileSync(join(root, '.gitignore'), '\n'); durableGitRepo(root, ['.gitignore']); },+ databaseUrl: backend === 'postgres' ? requirePostgresTestDatabase() : undefined }), 120_000);+});++describe('GST-327 gbrain repair secret-spans command line', () => {+ test('--rules is parsed, sorted and checked before anything runs, and belongs to secret-spans only', () => {+ expect(parseRepairArgs(['secret-spans', '--source', 'default', '--slug', 'notes/a', '--rules', 'openai, aws_access_key']).rules).toEqual(['aws_access_key', 'openai']);+ expect(() => parseRepairArgs(['secret-spans', '--source', 'default', '--slug', 'notes/a', '--rules', 'openai,not_a_rule'])).toThrow(/Unknown secret rule/);+ expect(() => parseRepairArgs(['timeline-order', '--rules', 'openai'])).toThrow(/--rules/);+ });+});diff --git a/test/secret-spans.test.ts b/test/secret-spans.test.tsnew file mode 100644index 0000000000..4d01fe4d57--- /dev/null+++ b/test/secret-spans.test.ts@@ -0,0 +1,47 @@+/**+ * GST-327: the pure span planner behind `gbrain repair secret-spans`. Spans come from the scanner's claims plus echoes+ * of claimed values across a page's fields; the splice must equal `applyRedaction` byte for byte; a marker-only claim is+ * dropped, so redacted text is a fixed point; unknown rule names refuse before anything is scanned.+ */+import { describe, expect, test } from 'bun:test';+import { NAMED_SECRET_RULES } from '../src/core/secret-scan.ts';+import { BROAD_SECRET_RULES, redactFields, residualMatches, resolveSecretRules } from '../src/core/secret-spans.ts';++const OPENAI = ['sk', 'proj', 'EXAMPLExNOTxREALxEXAMPLExNOTxREAL'].join('-');+const AWS = 'AKIA' + 'EXAMPLENOTREAL00';+const BEARER = 'EXAMPLEtokenNOTreal' + '1234567890abcdef';++describe('secret-spans planner', () => {+ test('default rules are every named rule; broad heuristics only when named; unknown names refuse', () => {+ expect(resolveSecretRules().rules).toEqual([...NAMED_SECRET_RULES]);+ expect(resolveSecretRules().rules.some(rule => BROAD_SECRET_RULES.includes(rule))).toBe(false);+ expect(resolveSecretRules(['labeled_credential', 'openai']).scan).toEqual({ highEntropy: false, labeledCredentials: true });+ expect(() => resolveSecretRules(['openai', 'nope'])).toThrow(/Unknown secret rule: "nope"; nothing was scanned/);+ });++ test('claims, echoes across fields and a key glued to a marker are replaced; everything else is byte-identical; the result is a fixed point', () => {+ const rules = resolveSecretRules();+ const fields = [+ { key: 'title', text: 'Notes' },+ { key: 'compiled_truth', text: `Key ${OPENAI} and ${AWS}.\n<REDACTED:openai>${OPENAI} glued.\nAuthorization: Bearer ${BEARER}` },+ { key: 'facts/1/fact', text: `The bare token ${BEARER} again.` },+ ];+ const results = redactFields(fields, rules);+ expect(results.every(result => result.proven)).toBe(true);+ expect(results.map(result => result.next)).toEqual(['Notes',+ 'Key <REDACTED:openai> and <REDACTED:aws_access_key>.\n<REDACTED:openai><REDACTED:openai> glued.\nAuthorization: Bearer <REDACTED:bearer>',+ 'The bare token <REDACTED:bearer> again.']);+ expect(results[0]!.spans).toEqual([]);+ expect(results[2]!.spans).toEqual([expect.objectContaining({ rule: 'bearer', echo: true })]);+ expect(results[0]!.new_sha256).toBe(results[0]!.old_sha256);+ expect(JSON.stringify(results.map(({ next: _next, ...rest }) => rest))).not.toContain(OPENAI);+ const again = redactFields(results.map(result => ({ key: result.key, text: result.next })), rules);+ expect(again.every(result => result.spans.length === 0 && result.proven)).toBe(true);+ for (const result of results) expect(residualMatches(result.next, rules)).toEqual({});+ });++ test('a rule left out of the selection is neither claimed nor echoed', () => {+ const [result] = redactFields([{ key: 'compiled_truth', text: `${OPENAI} ${AWS} ${OPENAI}` }], resolveSecretRules(['aws_access_key']));+ expect(result!.next).toBe(`${OPENAI} <REDACTED:aws_access_key> ${OPENAI}`);+ });+});