Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(repair): provenance repair never overwrites an unsynced page file
garrytan89f2a684d968
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e4ca4abd2d..1aff2685b5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -23,7 +23,7 @@ Maintenance writes that lower a page now tell you (#7917). A dream or patterns p- `src/core/trust/provenance-drift.ts` (new): the detector (`analyzePageDrift`, `scanProvenanceDrift`, `readPageDrift`), bounded at 50,000 pages per source in doctor.- `src/commands/doctor/checks/provenance-stamp-drift.ts` (new): doctor `provenance_stamp_drift`, read-only; its fix previews the repair (`ask_user`).- `src/core/repair/provenance.ts` (new): repair kind `provenance`, explicit-only, preview-bound and destructive; `--slug` narrows, `--include-ambiguous` widens.-- `src/core/trust/page-handlers.ts`: owner page mode `restore_provenance` (local CLI only, bound to the previewed revision and change). On a classic checkout, both repair modes (`restore_provenance` and `repair_marker`, which `gbrain trust backfill --repair-unknown-markers` also uses) write the repaired page to its file in the same transaction.+- `src/core/trust/page-handlers.ts`: owner page mode `restore_provenance` (local CLI only, bound to the previewed revision and change). On a classic checkout, both repair modes (`restore_provenance` and `repair_marker`, which `gbrain trust backfill --repair-unknown-markers` also uses) write the repaired page to its file in the same transaction; a page whose file holds an edit sync has not imported, or whose checkout cannot be written, is listed `needs_review` and left alone.- `src/core/trust/page-write.ts`, `src/core/persistence/prepared-maintenance.ts`, `src/core/trust/taint.ts`, `src/core/cycle/patterns.ts`: maintenance lowerings file `lower_page` on managed and repo-less brains; an unreadable derivation declaration rates the write `unknown`.- Two `BEHAVIOR_CHANGES` rows; docs: `docs/guides/repair.md#provenance`, `docs/architecture/key-files/security.md`.- Tests: `test/provenance-drift-7891.test.ts` and `test/maintenance-lowering-7917-7918.test.ts` (PGLite and Postgres; classic repository, managed repository and repo-less brains).diff --git a/docs/guides/repair.md b/docs/guides/repair.mdindex beb73bc035..9db9f7142c 100644--- a/docs/guides/repair.md+++ b/docs/guides/repair.md@@ -782,8 +782,10 @@ follow it down. The apply writes one owner page write per page, bound to thepreviewed revision and change, so a page that moved since is `preview_changed`and nothing is written. On a managed brain the write publishes the file throughthe coordinator (Git stays the parent); on a classic checkout it renders the-repaired page to its file in the same transaction; on a repo-less brain the-database is the record. It runs only from the local CLI on the brain host, and+repaired page to its file in the same transaction, and a page whose file holds+an edit the database lacks (run `gbrain sync` first) or whose checkout cannot+be written is listed `needs_review` instead; on a repo-less brain the database+is the record. It runs only from the local CLI on the brain host, andnever from `--all`, the remediation plan or an upgrade. A second preview listsnothing.diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.tsindex 298ae5b1f7..b6c0963c28 100644--- a/src/core/behavior-change-notice.ts+++ b/src/core/behavior-change-notice.ts@@ -1346,7 +1346,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;// #7891 provenance-stamp drift check and repair. `since` is stamped at the merge slot.{ since: '0.60.492.0', id: 'provenance_stamp_drift_7891', text: 'New doctor check `provenance_stamp_drift` warns when a page\'s provenance stamps (`source_kind`, `ingested_via`, `ingested_at`) or its lowering `trust_tier` marker were removed or rewritten by a write that is not the owner\'s own (before v0.60.486.0, #7892, made the stamps server-owned), or when a classified page\'s file would re-read above its stored tier. It reads page history and write receipts only and writes nothing. New explicit-only repair `gbrain repair provenance --source <id>` previews each page with its stamps, marker and tier before and after; `--apply --expect <hash> --yes` puts the stamps back, stamps the lower marker, lowers the page to it and files a `lower_page` proposal that `gbrain trust review` confirms or reverts. It only lowers trust, never writes the owner\'s own edits, writes unattributed ones only with `--include-ambiguous`, and never runs from `--all`, the remediation plan or an upgrade.' },// #7917 / #7918 maintenance lowerings and unreadable derivation declarations. `since` is stamped at the merge slot.- { since: '0.60.492.0', id: 'maintenance_lowering_7917_7918', text: 'A maintenance publication that lowers a page\'s trust tier (a dream or patterns write over a page the owner wrote, such as a grounding repair outside the output prefix) now files a `lower_page` proposal, so the page shows up in `gbrain trust review`; before, only agent content writes did. A maintenance write whose derivation declaration is present but unreadable now publishes at `unknown` and stamps an `unknown` trust marker into the file, instead of taking the write channel\'s tier and leaving the file unmarked. Both only lower trust. `gbrain trust backfill --repair-unknown-markers` and `gbrain repair provenance` on a classic (unmanaged) checkout now also write the repaired page to its file in the same step.' },+ { since: '0.60.492.0', id: 'maintenance_lowering_7917_7918', text: 'A maintenance publication that lowers a page\'s trust tier (a dream or patterns write over a page the owner wrote, such as a grounding repair outside the output prefix) now files a `lower_page` proposal, so the page shows up in `gbrain trust review`; before, only agent content writes did. A maintenance write whose derivation declaration is present but unreadable now publishes at `unknown` and stamps an `unknown` trust marker into the file, instead of taking the write channel\'s tier and leaving the file unmarked. Both only lower trust. `gbrain trust backfill --repair-unknown-markers` and `gbrain repair provenance` on a classic (unmanaged) checkout now also write the repaired page to its file in the same step, and leave a page whose file holds an edit sync has not imported untouched (run `gbrain sync` first).' },];/** The newest disclosed change's release: the notice id moves only when a release adds rows. */diff --git a/src/core/chronicle/publish.ts b/src/core/chronicle/publish.tsindex 0083a80d07..2d3aa608e4 100644--- a/src/core/chronicle/publish.ts+++ b/src/core/chronicle/publish.ts@@ -42,7 +42,8 @@ import { preparePageMutation } from '../persistence/page-prepare.ts';import type { Visibility } from '../search/private-visibility.ts';import type { ChronicleEventProposal } from './extract-events.ts';import { resolveChronicleEventSlugs } from './event-identity.ts';-import { declareDerivation, declaredWriteTrust, deriveTrust, readDerivationDeclaration } from '../trust/taint.ts';+import { declareDerivation, deriveTrust } from '../trust/taint.ts';+import { derivationWriteTrust } from '../trust/page-write.ts';import { derivedGateConfig, derivedGateInput, timelineRowAllowed } from '../trust/derived-gate.ts';import { assessTimelineForGate, type WriteGateAssessment } from '../write-gate.ts';import { recordWriteGateReceipt } from '../write-gate-store.ts';@@ -281,10 +282,10 @@ export async function prepareChronicleMutation(engine: BrainEngine, row: WriteReif (!projection) return outcome;const { projected } = await tx.upsertEventProjection({ depthSlug: projection.depth_slug, eventSlug: row.slug,date: projection.date, summary: projection.summary, sourceId: row.source_id });- const declaration = readDerivationDeclaration(p.derivation);- if (projected && declaration) {+ const trust = derivationWriteTrust({ derivation: p.derivation } as never);+ if (projected && trust) {const assessment = assessTimelineForGate({ summary: projection.summary, source: `life-chronicle:event:${row.slug}` },- derivedGateInput(declaredWriteTrust(declaration), row.id), await derivedGateConfig(tx));+ derivedGateInput(trust, row.id), await derivedGateConfig(tx));const [event] = await tx.executeRaw<{ id: number }>('SELECT id FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [row.source_id, row.slug]);if (event) await recordProjectionFlag(tx, assessment, Number(event.id), projection.date, row.source_id, row.id);}diff --git a/src/core/persistence/facts-prepare.ts b/src/core/persistence/facts-prepare.tsindex ea0854c5f9..6637a25288 100644--- a/src/core/persistence/facts-prepare.ts+++ b/src/core/persistence/facts-prepare.ts@@ -30,6 +30,7 @@ import { fenceAppendPendingTier, withPendingFenceRows } from '../eligibility/fenimport { withPageTierKept } from '../trust/fence-append.ts';import { pageFencesNormalized } from '../fence-repair/report.ts';import { declaredWriteTrust, derivedWriteTrust, readDerivationDeclaration, readTaintInputs, recordTaintEdges } from '../trust/taint.ts';+import { derivationWriteTrust } from '../trust/page-write.ts';import { minTrust, type TaintInput, type WriteTrust } from '../trust/tier.ts';import { applyGateDecision, derivedGateConfig, derivedGateInput, emptyGateTally } from '../trust/derived-gate.ts';import { decideFactWrite, recordFlaggedRow, type GatedRowDecision } from '../write-gate-store.ts';@@ -67,6 +68,8 @@ function thawFact(fact: FrozenExtractedFact): NewFact & { entity_slug: string |*/async function managedFactsTrust(engine: BrainEngine, p: ManagedFactIntent): Promise<{ trust: WriteTrust; inputs: Array<Pick<TaintInput, 'table' | 'id'>> } | undefined> {const declaration = readDerivationDeclaration(p.derivation);+ // #7918: a declaration that is present but unreadable fails closed at unknown, never the channel tier.+ if (!declaration && p.derivation != null) return { trust: derivationWriteTrust({ derivation: p.derivation } as never)!, inputs: [] };if (!declaration && !p.origin) return undefined;const page = p.origin ? await readTaintInputs(engine, [{ table: 'pages', id: p.origin.pageId }]) : [];const live = derivedWriteTrust({ channel: declaration?.origin.channel ?? 'derive:facts_backstop', inputs: page, requestId: p.originalRequestId });diff --git a/src/core/repair/provenance.ts b/src/core/repair/provenance.tsindex 5ec7a7a9d3..309ea849cc 100644--- a/src/core/repair/provenance.ts+++ b/src/core/repair/provenance.ts@@ -20,7 +20,8 @@import type { BrainEngine } from '../engine.ts';import { OperationError, opError } from '../ops/contract.ts';import { clearApprovedSet, loadApprovedSet, previewChangedError, previewHash, saveApprovedSet } from '../persistence/preview-approval.ts';-import { repairPageMarker, restorePageProvenance } from '../trust/page-handlers.ts';+import { managedPersistenceEnabled } from '../persistence/ownership.ts';+import { ownerRepairFile, repairPageMarker, restorePageProvenance } from '../trust/page-handlers.ts';import { plannedDigest, readPageDrift, scanProvenanceDrift, type DriftItem } from '../trust/provenance-drift.ts';import { afterCursor, type RepairHandler, type RepairItem, type RepairItemOutcome, type RepairListing, type RepairPlan, type RepairScope } from './core.ts';@@ -67,6 +68,7 @@ async function scan(engine: BrainEngine, scope: RepairScope, selection: Selectioconst planned: PlannedPage[] = [];const listing: RepairListing[] = [];const residuals: Record<string, number> = { restore: 0, restamp: 0, ambiguous: 0, needs_review: 0, local: 0 };+ const managedBrain = await managedPersistenceEnabled(engine);for (const sourceId of scope.source_ids) {const drift = await scanProvenanceDrift(engine, sourceId, { limit: Number.MAX_SAFE_INTEGER, includeAmbiguous: selection.include_ambiguous, slugs: selection.slugs });for (const item of drift.items) {@@ -84,6 +86,14 @@ async function scan(engine: BrainEngine, scope: RepairScope, selection: Selectiolisting.push({ item: label, class: 'needs_review', detail: 'the page stores its body in segments, which this repair does not rewrite; edit its frontmatter by hand' });continue;}+ const file = managedBrain ? null : await ownerRepairFile(engine, item.source_id, item.slug);+ if (file && !file.ok) {+ residuals.needs_review!++;+ listing.push({ item: label, class: 'needs_review', detail: file.reason === 'file_database_drift'+ ? 'the page file holds an edit the database lacks; run gbrain sync first, then preview again'+ : `the page file cannot be written (${file.reason}); fix the checkout, then preview again` });+ continue;+ }const action = verdict.class === 'restamp' ? 'restamp' : 'restore';residuals[action]!++;const page: PlannedPage = { source_id: item.source_id, slug: item.slug, id: item.page_id, revision: item.revision, action, planned: plannedDigest(item), summary: summaryOf(item) };@@ -102,7 +112,8 @@ function item(page: PlannedPage, hash: string, includeAmbiguous: boolean, last:/** The page still needs exactly the previewed change. */async function stillPlanned(engine: BrainEngine, page: PlannedPage, includeAmbiguous: boolean): Promise<'planned' | 'done' | 'moved'> {const current = await readPageDrift(engine, page.source_id, page.slug, { includeAmbiguous });- if (!current || current.page_id !== page.id || !current.verdict.restore) return 'done';+ if (!current) return 'done';+ if (current.page_id !== page.id || !current.verdict.restore) return 'moved';return current.revision === page.revision && plannedDigest(current) === page.planned ? 'planned' : 'moved';}diff --git a/src/core/trust/page-handlers.ts b/src/core/trust/page-handlers.tsindex cdfc6e5432..894a8e645a 100644--- a/src/core/trust/page-handlers.ts+++ b/src/core/trust/page-handlers.ts@@ -311,12 +311,15 @@ async function publishOwnerPage(engine: BrainEngine, input: PageActionInput, intconst { importFromContent } = await import('../import-file.ts');const attribution = await attributionFor(engine, input.by);const trust = ownerTrust(tier, intent.mode, intent.via);- // A repair mode on an unmanaged checkout renders the repaired row to the page's file in the same transaction (put_page's- // write-through), so a re-read of the checkout does not undo it; the file must still be the bytes read here.- const writeThrough = LOCAL_ONLY_MODES.includes(intent.mode) ? await import('../write-through.ts') : null;- const target = writeThrough ? await writeThrough.resolvePageWriteTarget(engine, input.slug, input.sourceId) : null;- const { existsSync, readFileSync } = await import('node:fs');- const fileBytes = target?.ok ? (existsSync(target.filePath) ? readFileSync(target.filePath, 'utf8') : null) : undefined;+ // A repair mode on an unmanaged checkout renders the repaired row to the page's file in the same transaction, so a+ // re-read of the checkout does not undo it; a file holding an edit the database lacks refuses the page instead.+ const file = LOCAL_ONLY_MODES.includes(intent.mode) ? await ownerRepairFile(engine, input.sourceId, input.slug) : null;+ if (file && !file.ok) {+ if (file.reason === 'file_database_drift') throw changedSincePreview(pageRef(input.sourceId, input.slug), ACTION_NAMES[intent.mode]);+ throw opError('unavailable', `The page file of ${pageRef(input.sourceId, input.slug)} cannot be written (${file.reason}); nothing was changed.`,+ 'Fix the checkout (gbrain sources status), then preview the repair again.');+ }+ const writeThrough = file?.ok && file.target ? await import('../write-through.ts') : null;await engine.transaction(tx => withWriteAttribution(tx, { ...attribution, trust }, () => withTrustPromotion(tx, tier, () =>withTrustKeep(tx, FENCE_TABLES, async () => {if ((await tx.readPageSnapshot(input.slug, { sourceId: input.sourceId }))?.revision !== input.expectedRevision) {@@ -326,10 +329,10 @@ async function publishOwnerPage(engine: BrainEngine, input: PageActionInput, intconst prior = intent.mode === 'restore_provenance' ? await readPageTrustState(tx, input.sourceId, input.slug) : null;await importFromContent(tx, input.slug, markdown, { sourceId: input.sourceId, noEmbed: true, preserveGateMarkers: intent.mode !== 'revert', forceRechunk: true, allowEmptyOverwrite: intent.mode === 'revert' });if (prior) await recordAgentPageLowering(tx, { operation: TRUST_OWNER_PAGE_OPERATION, source_id: input.sourceId, slug: input.slug }, { tier: prior.tier, revision: prior.revision }, PROVENANCE_PROPOSER);- if (writeThrough && target?.ok) {- const written = await writeThrough.writePageThrough(tx, input.slug, { sourceId: input.sourceId, expectedFileBytes: fileBytes ?? null });+ if (writeThrough && file?.ok && file.target) {+ const written = await writeThrough.writePageThrough(tx, input.slug, { sourceId: input.sourceId, expectedFileBytes: file.target.expectedFileBytes });if (written.skipped === 'file_changed') throw changedSincePreview(pageRef(input.sourceId, input.slug), ACTION_NAMES[intent.mode]);- if (!written.written && written.skipped !== 'disabled_by_config' && written.skipped !== 'url_clone') {+ if (!written.written) {throw opError('unavailable', `The page file of ${pageRef(input.sourceId, input.slug)} could not be written (${written.skipped ?? written.error}); nothing was changed.`,'Fix the checkout (gbrain sources status), then preview the repair again.');}@@ -338,6 +341,28 @@ async function publishOwnerPage(engine: BrainEngine, input: PageActionInput, int}))));}+/** Write-through outcomes that leave a repair database-only on purpose: no checkout, write-through off, an ignored or db_only target. */+const DATABASE_ONLY_TARGETS = ['no_repo_configured', 'disabled_by_config', 'not_canonical'];++/**+ * The checkout file an owner repair mode on an unmanaged brain renders the page to: `target: null` when the page has+ * none on purpose (DATABASE_ONLY_TARGETS), the bytes the write must still find, or why the page cannot be repaired:+ * `file_database_drift` when the file holds an edit the database lacks (a hand edit sync has not imported), else the+ * unresolvable target (a missing checkout, a path outside the source root) or `url_clone` (a clone gbrain pulls, whose+ * next pull would undo a database-only repair). Read-only.+ */+export async function ownerRepairFile(engine: BrainEngine, sourceId: string, slug: string):+ Promise<{ ok: true; target: { expectedFileBytes: string | null } | null } | { ok: false; reason: string }> {+ const { derivedFileTarget } = await import('../cycle/derived-write-through.ts');+ const resolved = await derivedFileTarget(engine, slug, sourceId);+ if (!resolved.ok) return DATABASE_ONLY_TARGETS.includes(resolved.skipped) ? { ok: true, target: null } : { ok: false, reason: resolved.skipped };+ const { urlCloneReadOnly } = await import('../url-clone.ts');+ if (await urlCloneReadOnly(engine, sourceId)) return { ok: false, reason: 'url_clone' };+ const { replaceableFile } = await import('../facts/file-body-guard.ts');+ const file = await replaceableFile(engine, sourceId, slug, resolved.target.filePath);+ return file.ok ? { ok: true, target: { expectedFileBytes: file.expectedFileBytes } } : { ok: false, reason: 'file_database_drift' };+}+/** Admits the journaled owner page mutation and waits for its publication (the decide_proposal precedent). */async function submitOwnerPage(engine: BrainEngine, input: PageActionInput, intent: OwnerPageIntent): Promise<void> {const { initializeLocalPersistence, requestPrincipalForContext } = await import('../persistence/page-mutations.ts');diff --git a/test/provenance-drift-7891.test.ts b/test/provenance-drift-7891.test.tsindex 6815e46580..85a1a22824 100644--- a/test/provenance-drift-7891.test.ts+++ b/test/provenance-drift-7891.test.ts@@ -195,6 +195,49 @@ for (const kind of testBackends()) {expect(await tierOf(slug)).toBe('external_untrusted');}, 120_000);+ test.skipIf(!repo)('a page file holding an edit the database lacks is listed needs_review and never overwritten', async () => {+ const slug = 'inbox/unsynced-edit';+ await seed(slug, STORED);+ await rewrite(slug, {}, 'oauth_client');+ const ready = await run(false);+ expect(ready.affected).toBe(1);+ const edited = fs.readFileSync(fileOf(slug), 'utf8').replace('Captured body, edited', 'Captured body, edited by hand in the checkout');+ fs.writeFileSync(fileOf(slug), edited);+ expect((await refusal(() => run(true, { expect: ready.preview_hash }))).code).toBe('preview_changed');+ expect(fs.readFileSync(fileOf(slug), 'utf8')).toBe(edited);+ const dry = await run(false);+ expect(dry.affected).toBe(0);+ expect(dry.listing.map((entry: Record<string, string>) => [entry.class, entry.detail])).toEqual([['needs_review', expect.stringContaining('gbrain sync first')]]);+ expect(fs.readFileSync(fileOf(slug), 'utf8')).toBe(edited);+ expect(await proposals(slug)).toEqual([]);+ }, 120_000);++ test.skipIf(!repo)('a page whose checkout cannot be written is listed needs_review, not repaired in the database alone', async () => {+ const slug = 'inbox/missing-checkout';+ await seed(slug, STORED);+ await rewrite(slug, {}, 'oauth_client');+ await engine.setConfig('sync.repo_path', path.join(tmpRoot, 'gone'));+ const dry = await run(false);+ expect(dry.affected).toBe(0);+ expect(dry.listing.map((entry: Record<string, string>) => entry.class)).toEqual(['needs_review']);+ }, 120_000);++ test.skipIf(!repo)('a page of a clone gbrain pulls is listed needs_review, not repaired in the database alone', async () => {+ const slug = 'inbox/owned-clone';+ await seed(slug, STORED);+ await rewrite(slug, {}, 'oauth_client');+ const before = fs.readFileSync(fileOf(slug), 'utf8');+ await engine.executeRaw(`UPDATE sources SET local_path = $1, config = COALESCE(config, '{}'::jsonb) || '{"managed_clone":true}'::jsonb WHERE id = 'default'`, [brainDir]);+ try {+ const dry = await run(false);+ expect(dry.affected).toBe(0);+ expect(dry.listing.map((entry: Record<string, string>) => [entry.class, entry.detail])).toEqual([['needs_review', expect.stringContaining('url_clone')]]);+ expect(fs.readFileSync(fileOf(slug), 'utf8')).toBe(before);+ } finally {+ await engine.executeRaw(`UPDATE sources SET local_path = NULL, config = config - 'managed_clone' WHERE id = 'default'`);+ }+ }, 120_000);+test('#7918: a classified page whose file would re-read higher gets its stored tier as the marker; the row keeps its tier', async () => {const slug = 'inbox/derived-unmarked';await seed(slug, {}, 'agent_written');