Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(ingest): frontmatter-only report type, grant preflight, undo keeps relied-on claims, re-plan on create conflict (GBRA-215 L3)
garrytan525fc82bd553
diff --git a/docs/guides/ingest.md b/docs/guides/ingest.mdindex 087077916e..3ca09e6c66 100644--- a/docs/guides/ingest.md+++ b/docs/guides/ingest.md@@ -148,24 +148,28 @@ receipts' `persistence.receipt_retention_days` window, default 30) and cross-cheno other attempt uses it.It keeps anything edited, promoted or linked from another page since, support it counted for a claim that already-existed, and claims another attempt counts as support. Apply runs under the source's publication guard and refuses+existed, and claims another attempt counts as support, with their page and the source page they cite. Apply runs under the source's publication guard and refuses`preview_changed` when the plan moved since the preview. A repeat is a no-op. After a rebuild that renumbered the-facts, undo refuses `ingest_undo_unverifiable` rather than guess. A page whose body is segmented keeps its rows+facts, or without the attempt record, undo refuses `ingest_undo_unverifiable` rather than guess. A page whose body is segmented keeps its rows(reported `segmented`).## Reserved namespaceReport pages (type `ingest-report`, slugs `ingest/<attempt_id>`) are written only by the attempt that owns them.Every generic page write (put, edit, append, delete, restore, revert, batch, upload) that names one, or declares the-type, is refused `ingest_namespace_reserved`. Other slugs under `ingest/` stay ordinary pages; an owner who wants+type in a page's leading frontmatter, is refused `ingest_namespace_reserved`; the same line in a page body is ordinary text. Other slugs under `ingest/` stay ordinary pages; an owner who wantsthe whole prefix reserved sets `gbrain config set ingest.reserve_namespace true`.## Remote callers`ingest` and `ingest_undo` are on the full and starter MCP surfaces. A remote ingest always proposes new pages,cannot set `visibility: private` or `trusted_extraction`, and sees only its own attempts; pages it cannot read are-withheld from reports by slug. A bound-slug client writes entities only inside its prefixes (others are-`skipped: outside_grant`), while its report page is still written.+withheld from reports by slug. Before any provider call, ingest checks that the caller's grant covers what it+commits: `put_page` on the source page slug and on `ingest/` (the report page), and `remember` when it extracts+facts. Otherwise it refuses `permission_denied` with nothing extracted or written. A bound-slug client writes+entities only inside its prefixes (others are `skipped: outside_grant`). When another writer creates a planned slug+first, ingest plans that entity once more and reports what it found (`existing`, a hash-tagged slug, or+`skipped: create_conflict`).## Errorsdiff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsvindex 7e89396c38..f81a3f0b0f 100644--- a/scripts/page-body-readers-baseline.tsv+++ b/scripts/page-body-readers-baseline.tsv@@ -318,6 +318,7 @@ src/core/import-identity.ts decideImportIdentity field METAsrc/core/import-screen.ts assessImportSanity field STREAMsrc/core/import-screen.ts screenImportContent call:contentHash METAsrc/core/import-screen.ts screenImportContent field STREAM+src/core/ingest/namespace.ts editedHead call:serializePageToMarkdown NONE serializes the frontmatter head with an empty body to apply edit_page edits that reach it; the body is never readsrc/core/line-grammar-report.ts grammarDiagnosticsForPage field INLINE grammar_diagnostics parses inline bodies only; a segmented page gets the skip (segmentedGrammarSkip)src/core/link-reconciliation.ts reconcileSourceLinks field INLINE inline arm; a segmented page streams through extractSegmentedPageLinks (page-body/segment-links.ts)src/core/link-temporal-apply.ts applyTemporalEvidence field STREAMdiff --git a/src/core/ingest/namespace.ts b/src/core/ingest/namespace.tsindex 679c7398bb..cedf786d1e 100644--- a/src/core/ingest/namespace.ts+++ b/src/core/ingest/namespace.ts@@ -11,11 +11,43 @@import { opError, type OperationError } from '../ops/contract.ts';import { readFix } from '../ops/op-fix.ts';import type { BrainEngine } from '../engine.ts';+import { coerceFrontmatterString, parseMarkdownHead, serializePageToMarkdown } from '../markdown.ts';+import { withoutBody } from '../page-body/page-shape.ts';+import type { PageSnapshot } from '../page-state/types.ts';+import type { Page } from '../types.ts';import { INGEST_REPORT_PREFIX, INGEST_REPORT_TYPE } from './report.ts';export const RESERVE_NAMESPACE_KEY = 'ingest.reserve_namespace';const REPORT_SLUG = /^ingest\/a[0-9a-f]{16}$/;-const DECLARES_REPORT_TYPE = /^type:\s*["']?ingest-report["']?\s*$/m;+const LEADING_FRONTMATTER = /^---[^\n]*\n(?:[\s\S]*?\n)?---[ \t]*(?:\n|$)/;++/** Whether a document's leading frontmatter declares the report type; the body never does. */+function declaresReportType(text: string): boolean {+ return coerceFrontmatterString(parseMarkdownHead(text).frontmatter.type) === INGEST_REPORT_TYPE;+}++/**+ * The existing page's frontmatter head after the edits that reach it. An edit matched only after the head's closing+ * fence changes the body, which cannot change the frontmatter; one that straddles the closing fence is applied to the+ * head as far as it reaches. No page body is read.+ */+function editedHead(existing: Pick<PageSnapshot, 'page' | 'tags'>, edits: unknown[]): string {+ let head = serializePageToMarkdown({ ...withoutBody(existing.page), compiled_truth: '', timeline: '' } as Page, existing.tags).slice(0, -1);+ for (const edit of edits) {+ const { old_text: oldText, new_text: newText } = (edit ?? {}) as { old_text?: unknown; new_text?: unknown };+ if (typeof oldText !== 'string' || !oldText || typeof newText !== 'string') continue;+ const close = LEADING_FRONTMATTER.exec(head)?.[0].length ?? head.length;+ const at = head.indexOf(oldText);+ if (at !== -1 && at < close) {+ head = head.slice(0, at) + newText + head.slice(at + oldText.length);+ continue;+ }+ for (let from = Math.max(0, head.length - oldText.length + 1); from < close - 1; from++) {+ if (oldText.startsWith(head.slice(from))) { head = head.slice(0, from) + newText; break; }+ }+ }+ return head;+}function refused(slug: string, why: string): OperationError {const attempt = slug.slice(INGEST_REPORT_PREFIX.length);@@ -25,13 +57,14 @@ function refused(slug: string, why: string): OperationError {}/** The reserved-type and reserved-namespace check every generic page mutation passes at admission. */-export async function assertIngestNamespaceWritable(engine: BrainEngine, input: { operation: string; slug: string; existingType: string | null | undefined; params: Record<string, unknown> }): Promise<void> {- if (input.existingType === INGEST_REPORT_TYPE) throw refused(input.slug, 'Ingest reports are written only by the ingest attempt that owns them.');- const texts: string[] = [];- if (typeof input.params.content === 'string') texts.push(input.params.content.slice(0, 8192));- if (input.params.type === INGEST_REPORT_TYPE) throw refused(input.slug, 'The ingest-report page type is reserved for ingest.');- if (Array.isArray(input.params.edits)) for (const e of input.params.edits) if (typeof (e as { new_text?: unknown })?.new_text === 'string') texts.push((e as { new_text: string }).new_text);- if (texts.some(t => DECLARES_REPORT_TYPE.test(t))) throw refused(input.slug, 'The ingest-report page type is reserved for ingest.');+export async function assertIngestNamespaceWritable(engine: BrainEngine, input: { operation: string; slug: string; existing: Pick<PageSnapshot, 'page' | 'tags'> | null | undefined; params: Record<string, unknown> }): Promise<void> {+ const existing = input.existing && !input.existing.page.deleted_at ? input.existing : null;+ if (input.existing?.page.type === INGEST_REPORT_TYPE) throw refused(input.slug, 'Ingest reports are written only by the ingest attempt that owns them.');+ const { content, edits } = input.params;+ if (input.params.type === INGEST_REPORT_TYPE || (typeof content === 'string' && declaresReportType(content))+ || (Array.isArray(edits) && existing && declaresReportType(editedHead(existing, edits)))) {+ throw refused(input.slug, 'The ingest-report page type is reserved for ingest.');+ }if (!input.slug.startsWith(INGEST_REPORT_PREFIX)) return;if (REPORT_SLUG.test(input.slug)) throw refused(input.slug, 'Slugs of the form ingest/<attempt_id> are written only by the ingest attempt that owns them.');const reserve = await engine.getConfig(RESERVE_NAMESPACE_KEY);diff --git a/src/core/ops/ingest.ts b/src/core/ops/ingest.tsindex 777c46e696..34d63f926f 100644--- a/src/core/ops/ingest.ts+++ b/src/core/ops/ingest.ts@@ -31,9 +31,9 @@ import {INGEST_ENTITY_CAP, INGEST_MAX_CHARS, IngestProviderFailure, extractWindow, extractionKillSwitchOn, ingestModel, keylessCandidates,mergeWindows, windowDocument, type ExtractedWindow,} from '../ingest/extract.ts';-import { planIngest, type IngestPlan } from '../ingest/plan.ts';+import { planIngest, type IngestPlan, type PlannedPage } from '../ingest/plan.ts';import {- INGEST_REPORT_TYPE, filterReportForCaller, reportActions, reportPageContent, reportSlugFor, resolveStatus, summarize,+ INGEST_REPORT_PREFIX, INGEST_REPORT_TYPE, filterReportForCaller, reportActions, reportPageContent, reportSlugFor, resolveStatus, summarize,type IngestReport, type ReportEvidence, type ReportFact, type ReportPage,} from '../ingest/report.ts';@@ -195,6 +195,26 @@ async function principalOf(ctx: OperationContext) {return requestPrincipalForContext(ctx);}+/**+ * Plan step 1: before any provider call or write, the caller's grant must cover what ingest commits: the source page,+ * the report page under ingest/ and, when facts are extracted, remember. The door's own authorization answers it.+ */+async function assertCallerCanCommit(ctx: OperationContext, sourceId: string, incarnation: string, sourceSlug: string, facts: boolean): Promise<void> {+ const { submissionAuthority } = await import('../persistence/authority.ts');+ const needs: Array<[string, string, string]> = [['put_page', sourceSlug, 'the source page ' + sourceSlug], ['put_page', reportSlugFor('a' + '0'.repeat(16)), 'the report page under ' + INGEST_REPORT_PREFIX],+ ...(facts ? [['remember', sourceSlug, 'facts (remember)'] as [string, string, string]] : [])];+ for (const [operation, slug, what] of needs) {+ try {+ await submissionAuthority(ctx, operation, sourceId, incarnation, slug);+ } catch (error) {+ if (!(error instanceof OperationError)) throw error;+ throw opError('permission_denied', 'This caller cannot write ' + what + ', which ingest commits: ' + error.message,+ 'Nothing was extracted or written. The grant must cover the source page slug, ' + INGEST_REPORT_PREFIX + ', put_page and remember; ask the brain owner to widen it, or pass a slug inside it.',+ { docs: DOCS + '#remote-callers' });+ }+ }+}+/** Status, resume and undo by attempt id: the local owner reads every attempt, any other caller only its own. */async function callerMayRead(ctx: OperationContext, state: IngestAttemptState): Promise<boolean> {return ctx.remote === false || principalKey(await principalOf(ctx)) === principalKey(state.principal);@@ -302,17 +322,30 @@ async function commitPlan(ctx: OperationContext, state: IngestAttemptState, planconst base: ReportPage = { name: page.name, slug: page.slug, type: page.type, status: 'skipped', ...(page.near_names ? { near_names: page.near_names } : {}) };if (page.status === 'skipped') { report.pages.push({ ...base, reason: page.reason }); continue; }if (page.status === 'existing') { const row = { ...base, status: 'existing' as const }; report.pages.push(row); written.set(page.slug, row); continue; }- const proposed = page.status === 'propose';- const row = await withEntityCreateLock(ctx, sourceId, async (): Promise<ReportPage> => {- const snap = await ctx.engine.readPageSnapshot(page.slug, { sourceId, includeDeleted: true });- if (snap && !snap.page.deleted_at) return { ...base, status: 'skipped', reason: 'create_conflict' };- const r = await publish(ctx, sourceId, { slug: page.slug, content: entityPageContent(page, proposed, state), ...(snap ? { expected_revision: snap.revision } : {}) });- if (!r.ok) return { ...base, status: 'failed', reason: r.code };- report.write_requests.push({ slug: page.slug, request_id: r.request_id, state: r.pending ? 'pending' : 'committed' });- if (r.pending) return { ...base, status: 'pending', request_id: r.request_id };- evidence.created_pages.push({ slug: page.slug, request_id: r.request_id, revision: null, proposed });- return { ...base, status: proposed ? 'proposed' : 'created', request_id: r.request_id };+ const create = (target: PlannedPage) => withEntityCreateLock(ctx, sourceId, async (): Promise<ReportPage | null> => {+ const proposed = target.status === 'propose';+ const at = { ...base, slug: target.slug };+ const snap = await ctx.engine.readPageSnapshot(target.slug, { sourceId, includeDeleted: true });+ if (snap && !snap.page.deleted_at) return null;+ const r = await publish(ctx, sourceId, { slug: target.slug, content: entityPageContent(target, proposed, state), ...(snap ? { expected_revision: snap.revision } : {}) });+ if (!r.ok) return { ...at, status: 'failed', reason: r.code };+ report.write_requests.push({ slug: target.slug, request_id: r.request_id, state: r.pending ? 'pending' : 'committed' });+ if (r.pending) return { ...at, status: 'pending', request_id: r.request_id };+ evidence.created_pages.push({ slug: target.slug, request_id: r.request_id, revision: null, proposed });+ return { ...at, status: proposed ? 'proposed' : 'created', request_id: r.request_id };});+ const conflict: ReportPage = { ...base, status: 'skipped', reason: 'create_conflict' };+ let row = await create(page);+ if (!row) {+ // Another writer took the slug after planning: plan this one entity again against the brain as it is now, once.+ const [again] = (await planIngest({ ctx, sourceId, text, entities: [{ name: page.name, type: page.type, context: page.context, source: page.source }], facts: [],+ visibility: state.options.visibility, validFrom: null, trusted: state.options.trusted_extraction, factsSkipped: null })).pages;+ row = !again ? conflict+ : again.status === 'existing' ? { ...base, slug: again.slug, status: 'existing' }+ : again.status === 'skipped' ? { ...base, slug: again.slug || page.slug, status: 'skipped', reason: again.reason }+ : (await create(again)) ?? conflict;+ if (row.status === 'existing') { report.pages.push(row); written.set(page.slug, row); continue; }+ }report.pages.push(row);if (row.status === 'proposed' || row.status === 'created') written.set(page.slug, row);}@@ -330,27 +363,27 @@ async function commitPlan(ctx: OperationContext, state: IngestAttemptState, planrow.copies.push({ entity_slug: c.entity_slug, status: pageRow?.status === 'pending' ? 'pending' : 'skipped', reason: pageRow?.status === 'pending' ? 'page_pending' : 'page_not_written' });continue;}- const check = await checkFactAgainstSaved(ctx.engine, sourceId, { fact: f.fact, kind: f.kind, visibility: state.options.visibility, entity_slug: c.entity_slug }, { excludePrivate });- if (check.verdict === 'withdrawn') { row.copies.push({ entity_slug: c.entity_slug, status: 'skipped', reason: 'withdrawn' }); continue; }+ const check = await checkFactAgainstSaved(ctx.engine, sourceId, { fact: f.fact, kind: f.kind, visibility: state.options.visibility, entity_slug: target.slug }, { excludePrivate });+ if (check.verdict === 'withdrawn') { row.copies.push({ entity_slug: target.slug, status: 'skipped', reason: 'withdrawn' }); continue; }if (check.verdict === 'duplicate') {- row.copies.push({ entity_slug: c.entity_slug, status: 'support', fact_id: check.id });+ row.copies.push({ entity_slug: target.slug, status: 'support', fact_id: check.id });if (!evidence.support_fact_ids.includes(check.id)) evidence.support_fact_ids.push(check.id);continue;}const request_id = randomUUID();try {const res = asRecord(await remember.handler(ctx, {- fact: f.fact, entity: c.entity_slug, provenance: 'ingest:' + state.source_slug, kind: f.kind, request_id,+ fact: f.fact, entity: target.slug, provenance: 'ingest:' + state.source_slug, kind: f.kind, request_id,visibility: state.options.visibility, ...(f.valid_from ? { valid_from: f.valid_from } : {}), infer_entity: false,}));const id = Number(res.fact_id ?? res.id ?? asRecord((res.facts as unknown[] | undefined)?.[0]).id);const pending = pendingOf(res);- row.copies.push({ entity_slug: c.entity_slug, status: pending ? 'pending' : 'created', ...(Number.isFinite(id) ? { fact_id: id } : {}), request_id });- report.write_requests.push({ slug: c.entity_slug, request_id, state: pending ? 'pending' : 'committed' });- if (!pending && Number.isFinite(id)) evidence.created_facts.push({ id, entity_slug: c.entity_slug, request_id });+ row.copies.push({ entity_slug: target.slug, status: pending ? 'pending' : 'created', ...(Number.isFinite(id) ? { fact_id: id } : {}), request_id });+ report.write_requests.push({ slug: target.slug, request_id, state: pending ? 'pending' : 'committed' });+ if (!pending && Number.isFinite(id)) evidence.created_facts.push({ id, entity_slug: target.slug, request_id });} catch (error) {if (!(error instanceof OperationError)) throw error;- row.copies.push({ entity_slug: c.entity_slug, status: 'failed', reason: error.code, request_id });+ row.copies.push({ entity_slug: target.slug, status: 'failed', reason: error.code, request_id });}}// D3 and round-4 item 11: the links effect derives no mentions edge from fence rows on either storage (checked in@@ -409,6 +442,7 @@ async function runIngest(ctx: OperationContext, p: Params): Promise<Record<strin}if (state && state.phase === 'complete') return storedReport(ctx, state, { repeated: true });if (state && state.phase !== 'undone' && p.resume !== true) return storedReport(ctx, state, { repeated: true, resumable: true });+ await assertCallerCanCommit(ctx, sourceId, topology.source_incarnation, state?.source_slug ?? v.slug ?? 'inbox/ingest-' + documentHash.slice(0, 12), v.options.keyed && !v.options.no_facts);if (!state || state.phase === 'undone') {const now = new Date().toISOString();const attempt_id = newAttemptId();@@ -515,6 +549,7 @@ async function planUndo(ctx: OperationContext, state: IngestAttemptState, ev: Reconst sourceId = state.source_id;const out: UndoPlan = { retract: [], delete: [], skipped: [] };const ids = ev.created_facts.map(f => f.id);+ const supportedOn = new Set<string>();const rows = ids.length ? await ctx.engine.executeRaw<{ id: number; entity_slug: string | null; row_num: number | null; expired_at: unknown; valid_until: unknown; source: string | null }>('SELECT id, entity_slug, row_num, expired_at, valid_until, source FROM facts WHERE source_id=$1 AND id = ANY($2::int[])', [sourceId, ids]) : [];const byId = new Map(rows.map(r => [Number(r.id), r]));@@ -523,7 +558,7 @@ async function planUndo(ctx: OperationContext, state: IngestAttemptState, ev: Reconst row = byId.get(f.id);if (!row || row.entity_slug !== f.entity_slug || row.source !== 'ingest:' + ev.source_slug) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'not_found' }); continue; }if (row.expired_at || row.valid_until) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'already_retracted' }); continue; }- if ((await supportingAttempts(ctx.engine, f.id, state.attempt_id)).length) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'supported' }); continue; }+ if ((await supportingAttempts(ctx.engine, f.id, state.attempt_id)).length) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'supported' }); supportedOn.add(f.entity_slug); continue; }if (row.row_num == null) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'not_fenced' }); continue; }out.retract.push({ fact_id: f.id, entity_slug: f.entity_slug, row_num: Number(row.row_num) });}@@ -537,13 +572,14 @@ async function planUndo(ctx: OperationContext, state: IngestAttemptState, ev: Reif (!snap || snap.page.deleted_at) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'not_found' }); continue; }if (pg.proposed && !isUnverifiedExtraction(asRecord(snap.page.frontmatter))) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'promoted' }); continue; }if (pg.revision && snap.revision !== pg.revision) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'edited' }); continue; }- if ((await referencedFrom(pg.slug)).length) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'referenced' }); continue; }+ // A claim another attempt relies on keeps its page, and the source page it cites.+ if (supportedOn.has(pg.slug) || (await referencedFrom(pg.slug)).length) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'referenced' }); continue; }out.delete.push({ slug: pg.slug });}if (ev.source_created) {const others = await attemptsOfDocument(ctx.engine, sourceId, state.document_hash, state.attempt_id);if (others.length) out.skipped.push({ kind: 'page', ref: ev.source_slug, reason: 'used_by_other_attempt' });- else if ((await referencedFrom(ev.source_slug)).length) out.skipped.push({ kind: 'page', ref: ev.source_slug, reason: 'referenced' });+ else if (supportedOn.size || (await referencedFrom(ev.source_slug)).length) out.skipped.push({ kind: 'page', ref: ev.source_slug, reason: 'referenced' });else out.delete.push({ slug: ev.source_slug });}return out;diff --git a/src/core/persistence/page-mutations.ts b/src/core/persistence/page-mutations.tsindex 95316e7b59..69dbdad085 100644--- a/src/core/persistence/page-mutations.ts+++ b/src/core/persistence/page-mutations.ts@@ -363,12 +363,10 @@ export async function preparePageAdmission(ctx: OperationContext,if (typeof intent.capture_path === 'string') intent.capture_file_hash = sha256(p.content);}await validateMutationSlug(ctx, input.operation, slug, sourceId);- // GBRA-215: an attempt's own report keeps the original principal but not its slug fence (a bound client's ingest writes it).+ enforceClientSlugFence(ctx, slug, input.operation);+ enforceSubagentSlugFence(ctx, slug, input.operation);+ // GBRA-215: an attempt's own report page (ingest/<attempt_id>) passes the reserved-namespace check; every grant still applies.const ingestReport = input.internalIngestReport === true && input.operation === 'put_page' && slug.startsWith('ingest/');- if (!ingestReport) {- enforceClientSlugFence(ctx, slug, input.operation);- enforceSubagentSlugFence(ctx, slug, input.operation);- }// Preserve same-source diagnostics for new timeline writes without making// terminal replay depend on a page that may have since been purged.if (input.operation === 'add_timeline_entry') await requireWritablePage({ ...ctx, sourceId }, slug, input.operation, 'page');@@ -383,7 +381,7 @@ export async function preparePageAdmission(ctx: OperationContext,async () => !source.local_path && sourceId === 'default' ? ctx.engine.getConfig('sync.repo_path') : null,]) as [Awaited<ReturnType<typeof submissionAuthority>>, unknown, Awaited<ReturnType<BrainEngine['readPageSnapshot']>>, Awaited<ReturnType<typeof getWorktreeBinding>>, string | null, string | null];if (!ingestReport && input.managedFileImport !== true) {- await (await import('../ingest/namespace.ts')).assertIngestNamespaceWritable(ctx.engine, { operation: input.operation, slug, existingType: snapshot?.page.type, params: p });+ await (await import('../ingest/namespace.ts')).assertIngestNamespaceWritable(ctx.engine, { operation: input.operation, slug, existing: snapshot, params: p });}if (typeof intent.content === 'string' && (typeof p.expected_revision !== 'string' || p.expected_revision === snapshot?.revision)) {await assertTimelineNotOmitted(ctx.engine, { intent, remote: ctx.remote !== false, writer: typeof p.expected_revision === 'string' ? 'editing' : 'preserving',diff --git a/test/e2e/ingest-postgres.test.ts b/test/e2e/ingest-postgres.test.tsindex 6d82d60d55..71d1e650e6 100644--- a/test/e2e/ingest-postgres.test.ts+++ b/test/e2e/ingest-postgres.test.ts@@ -71,6 +71,13 @@ d('ingest and ingest_undo (live Postgres)', () => {test('validation (' + label + ')', async () => { fx = await fixture(storage); await S.validationScenario(engine); }, 60_000);test('two-entity claim (' + label + ')', async () => { fx = await fixture(storage); await S.twoEntityScenario(engine); }, 60_000);test('reserved report namespace (' + label + ')', async () => { fx = await fixture(storage); await S.namespaceScenario(engine); }, 60_000);+ test('private ingest (' + label + ')', async () => { fx = await fixture(storage); await S.privacyScenario(engine); }, 60_000);+ test('held natural slug (' + label + ')', async () => { fx = await fixture(storage); await S.heldSlugScenario(engine); }, 60_000);+ test('grants (' + label + ')', async () => { fx = await fixture(storage); await S.grantScenario(engine); }, 60_000);+ test('create conflict re-plan (' + label + ')', async () => { fx = await fixture(storage); await S.createConflictScenario(engine); }, 60_000);+ test('undo keeps changed pages (' + label + ')', async () => { fx = await fixture(storage); await S.undoKeepsChangedPagesScenario(engine); }, 90_000);+ test('undo keeps relied-on claims (' + label + ')', async () => { fx = await fixture(storage); await S.undoKeepsReliedOnScenario(engine); }, 90_000);+ test('undo unverifiable (' + label + ')', async () => { fx = await fixture(storage); await S.undoUnverifiableScenario(engine); }, 60_000);}test('Git target unavailable refuses before writes', async () => { fx = await fixture('git'); await S.storageUnavailableScenario(engine); }, 60_000);});diff --git a/test/helpers/ingest-scenarios.ts b/test/helpers/ingest-scenarios.tsindex f028421be8..50b964fe2f 100644--- a/test/helpers/ingest-scenarios.ts+++ b/test/helpers/ingest-scenarios.ts@@ -11,6 +11,8 @@ import { localHostId } from '../../src/core/persistence/identity.ts';import { parseMarkdown } from '../../src/core/markdown.ts';import { __setChatTransportForTests, configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';import { reviewCtx, committedFile, type ReviewDoorFixture } from './review-door-scenarios.ts';+import { randomUUID } from 'node:crypto';+import { tryAcquireDbLock } from '../../src/core/db-lock.ts';export type Report = Record<string, any>;@@ -331,8 +333,30 @@ export async function namespaceScenario(engine: BrainEngine): Promise<void> {}const typed = await put.handler(reviewCtx(engine), { slug: 'notes/fake-report', content: '---\ntype: ingest-report\ntitle: x\n---\n\nmine\n' }).catch(e => e) as OperationError;expect(typed.code).toBe('ingest_namespace_reserved');+ const reportRev = (await engine.readPageSnapshot(report.report_page))!.revision;+ const editReport = await operationsByName.edit_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: report.report_page, expected_revision: reportRev,+ edits: [{ old_text: 'ingest', new_text: 'mine' }] }).catch(e => e) as OperationError;+ expect(editReport.code).toBe('ingest_namespace_reserved');const del = await operationsByName.delete_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: report.report_page }).catch(e => e) as OperationError;expect(del.code).toBe('ingest_namespace_reserved');+ // Only the leading frontmatter declares a type: the same line in the body is ordinary text, written and edited as before.+ await put.handler(reviewCtx(engine, { remote: true } as never), { slug: 'notes/config-example', content: '---\ntype: note\ntitle: Config example\n---\n\nA report page starts with:\n\ntype: ingest-report\n' });+ await drain(engine);+ const bodyOnly = (await engine.readPageSnapshot('notes/config-example'))!;+ expect(bodyOnly.page.type).toBe('note');+ await operationsByName.edit_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: 'notes/config-example', expected_revision: bodyOnly.revision,+ edits: [{ old_text: 'A report page starts with:', new_text: 'A report page starts with:\n\n---\ntype: ingest-report\n---' }] });+ await drain(engine);+ const edited = (await engine.readPageSnapshot('notes/config-example'))!;+ expect(edited.revision).not.toBe(bodyOnly.revision);+ expect(edited.page.type).toBe('note');+ // An edit that puts the type into the frontmatter head is refused, whole line or value alone.+ for (const change of [{ old_text: 'type: note', new_text: 'type: ingest-report' }, { old_text: 'type: note', new_text: 'type: "ingest-report"' }]) {+ const headEdit = await operationsByName.edit_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: 'notes/config-example', expected_revision: edited.revision,+ edits: [change] }).catch(e => e) as OperationError;+ expect(headEdit.code).toBe('ingest_namespace_reserved');+ }+ expect((await engine.readPageSnapshot('notes/config-example'))!.revision).toBe(edited.revision);// Other slugs under ingest/ stay ordinary pages unless the owner reserves the namespace.await put.handler(reviewCtx(engine), { slug: 'ingest/notes', content: '---\ntype: note\ntitle: x\n---\n\nmine\n' });await engine.setConfig('ingest.reserve_namespace', 'true');@@ -372,3 +396,174 @@ export async function twoEntityScenario(engine: BrainEngine): Promise<Array<{ frreturn edges;} finally { stub.restore(); }}++/** A remote OAuth client on the default source with the given operation and slug grants. */+async function oauthCaller(engine: BrainEngine, grant: { allowedOperations?: string[]; boundSlugPrefixes?: string[] }): Promise<Partial<OperationContext>> {+ const clientId = 'ingest-client-' + randomUUID().slice(0, 8);+ await engine.executeRaw("INSERT INTO oauth_clients(client_id,client_secret_hash,client_name,scope,source_id,allowed_operations,bound_slug_prefixes) VALUES($1,'fixture-hash','example-client','read write','default',$2,$3)",+ [clientId, grant.allowedOperations ?? null, grant.boundSlugPrefixes ?? null]);+ return { remote: true, transport: 'http', auth: { token: 'fixture', clientId, principal: { kind: 'oauth_client', id: clientId }, scopes: ['read', 'write'], sourceId: 'default', ...grant } } as Partial<OperationContext>;+}++const ingestCheckpoints = async (engine: BrainEngine) => Number((await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM op_checkpoints WHERE op LIKE 'ingest-%'"))[0]!.n);++/** A private ingest: every page it derives and every claim it saves is private, and a remote caller reads none of it. */+export async function privacyScenario(engine: BrainEngine): Promise<void> {+ await seedEntities(engine);+ const stub = stubChat();+ try {+ const report = await ingest(engine, { content: goldenDocument(), title: 'Call notes', visibility: 'private' });+ expect(report.status).toBe('complete');+ const derived = [report.source_page.slug, report.report_page, ...report.pages.filter((p: Report) => p.status === 'proposed').map((p: Report) => p.slug)];+ expect(derived).toEqual(expect.arrayContaining(['companies/zentrix', 'companies/acmx']));+ const remote = reviewCtx(engine, { remote: true } as never);+ for (const slug of derived) {+ expect((await engine.getPage(slug))!.frontmatter.visibility).toBe('private');+ expect((await operationsByName.get_page!.handler(remote, { slug }).catch(e => e) as OperationError).code).toBe('page_not_found');+ }+ const facts = await engine.executeRaw<{ visibility: string }>('SELECT visibility FROM facts WHERE source = $1', ['ingest:' + report.source_page.slug]);+ expect(facts.length).toBe(3);+ expect(facts.every(f => f.visibility === 'private')).toBe(true);+ // The claim saved on an existing world page stays out of a remote read of that page.+ const alice = JSON.stringify(await operationsByName.get_page!.handler(remote, { slug: 'people/alice-example' }));+ expect(alice).toContain('Alice Example is a founder.');+ expect(alice).not.toContain('Lisbon');+ } finally { stub.restore(); }+}++/** A natural slug held by a page the caller cannot read takes the hash-tagged slug and stays proposed, never existing. */+export async function heldSlugScenario(engine: BrainEngine): Promise<void> {+ await seedEntities(engine);+ await operationsByName.put_page!.handler(reviewCtx(engine), { slug: 'companies/zentrix', content: '---\ntype: company\ntitle: Zentrix\nvisibility: private\n---\n\nPrivate notes.\n' });+ await drain(engine);+ const stub = stubChat();+ try {+ const report = await ingest(engine, { content: goldenDocument(), title: 'Call notes' }, { remote: true } as never);+ const zentrix = report.pages.find((p: Report) => p.name === 'Zentrix');+ expect(zentrix.status).toBe('proposed');+ expect(zentrix.slug).toMatch(/^companies\/zentrix-[0-9a-f]{6}$/);+ expect(report.pages.some((p: Report) => p.slug === 'companies/zentrix')).toBe(false);+ expect((await engine.getPage('companies/zentrix'))!.compiled_truth.trim()).toBe('Private notes.');+ expect((await factsOn(engine, 'companies/zentrix')).length).toBe(0);+ } finally { stub.restore(); }+}++/** Grants: a caller that cannot write what ingest commits is refused before any provider call; a bound one skips entities outside its prefixes. */+export async function grantScenario(engine: BrainEngine): Promise<void> {+ await seedEntities(engine);+ const stub = stubChat();+ try {+ for (const grant of [{ allowedOperations: ['ingest'] }, { boundSlugPrefixes: ['companies/'] }]) {+ const before = await pageCount(engine);+ const err = await ingest(engine, { content: goldenDocument(), title: 'Call notes' }, await oauthCaller(engine, grant)).catch(e => e) as OperationError;+ expect(err.code).toBe('permission_denied');+ expect(stub.calls).toBe(0);+ expect(await pageCount(engine)).toBe(before);+ expect(await ingestCheckpoints(engine)).toBe(0);+ }+ const report = await ingest(engine, { content: goldenDocument(), title: 'Call notes' }, await oauthCaller(engine, { boundSlugPrefixes: ['companies/', 'inbox/', 'ingest/'] }));+ expect(report.status).toBe('complete');+ expect(report.pages.find((p: Report) => p.name === 'Alice Example')).toMatchObject({ slug: 'people/alice-example', status: 'skipped', reason: 'outside_grant' });+ expect(report.pages.find((p: Report) => p.name === 'Zentrix')).toMatchObject({ slug: 'companies/zentrix', status: 'proposed' });+ expect((await engine.getPage(report.report_page))!.type).toBe('ingest-report');+ expect((await factsOn(engine, 'people/alice-example')).length).toBe(0);+ } finally { stub.restore(); }+}++/** Another writer creates a planned slug before the attempt does: the entity is planned again once, and its claim lands on that page. */+export async function createConflictScenario(engine: BrainEngine): Promise<void> {+ await seedEntities(engine);+ const stub = stubChat();+ const lock = await tryAcquireDbLock(engine, 'ingest-entity-create:default', 2);+ expect(lock).not.toBeNull();+ let held = true;+ try {+ const running = ingest(engine, { content: goldenDocument(), title: 'Call notes' });+ // The report page is written before any entity page, so its row means planning is over and the run waits on the lock.+ for (let i = 0; i < 150; i++) {+ const [row] = await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM pages WHERE slug LIKE 'ingest/%' AND deleted_at IS NULL");+ if (Number(row!.n)) break;+ await new Promise(resolve => setTimeout(resolve, 100));+ }+ await operationsByName.put_page!.handler(reviewCtx(engine), { slug: 'companies/zentrix', content: '---\ntype: company\ntitle: Zentrix\n---\n\nZentrix builds sensors.\n' });+ await drain(engine);+ await lock!.release();+ held = false;+ const report = await running;+ expect(report.status).toBe('complete');+ expect(report.pages.find((p: Report) => p.name === 'Zentrix')).toMatchObject({ slug: 'companies/zentrix', status: 'existing' });+ const fact = report.facts.find((f: Report) => f.fact === 'Zentrix signed its first enterprise customer.');+ expect(fact.copies.map((c: Report) => [c.entity_slug, c.status])).toEqual([['companies/zentrix', 'created']]);+ const page = (await engine.getPage('companies/zentrix'))!;+ expect(page.compiled_truth).toContain('Zentrix builds sensors.');+ expect(page.frontmatter.status).toBeUndefined();+ expect((await factsOn(engine, 'companies/zentrix')).map(f => f.fact)).toEqual(['Zentrix signed its first enterprise customer.']);+ // The other writer's page is not the attempt's to delete.+ const preview = await undo(engine, { attempt_id: report.attempt_id });+ expect(preview.delete.map((d: Report) => d.slug)).not.toContain('companies/zentrix');+ } finally {+ if (held) await lock!.release();+ stub.restore();+ }+}++async function goldenRun(engine: BrainEngine): Promise<Report> {+ await seedEntities(engine);+ const stub = stubChat();+ try { return await ingest(engine, { content: goldenDocument(), title: 'Call notes' }); } finally { stub.restore(); }+}++/** Undo keeps a proposed page someone promoted and a page someone edited after the attempt wrote it. */+export async function undoKeepsChangedPagesScenario(engine: BrainEngine): Promise<void> {+ const report = await goldenRun(engine);+ await operationsByName.extraction_review!.handler(reviewCtx(engine), { action: 'promote', slugs: ['companies/acmx'] });+ await drain(engine);+ const snap = (await engine.readPageSnapshot('companies/zentrix'))!;+ await operationsByName.edit_page!.handler(reviewCtx(engine), { slug: 'companies/zentrix', expected_revision: snap.revision,+ edits: [{ old_text: 'Zentrix signed its first enterprise customer last week.', new_text: 'Zentrix signed its first enterprise customer last week, per the call.' }] });+ await drain(engine);+ const preview = await undo(engine, { attempt_id: report.attempt_id });+ expect(preview.skipped).toEqual(expect.arrayContaining([{ kind: 'page', ref: 'companies/acmx', reason: 'promoted' }, { kind: 'page', ref: 'companies/zentrix', reason: 'edited' }]));+ expect(preview.delete.map((d: Report) => d.slug)).toEqual([report.source_page.slug]);+ const applied = await undo(engine, { attempt_id: report.attempt_id, yes: true, expect: preview.plan_hash });+ expect(applied.status).toBe('undone');+ expect((await engine.getPage('companies/acmx'))!.frontmatter.status).not.toBe('unverified');+ expect((await engine.getPage('companies/zentrix'))!.compiled_truth).toContain('per the call.');+}++/** Undo keeps a claim another attempt saved as support, with its page and the source page it cites. */+export async function undoKeepsReliedOnScenario(engine: BrainEngine): Promise<void> {+ const first = await goldenRun(engine);+ const zentrixFact = first.facts.find((f: Report) => f.fact === 'Zentrix signed its first enterprise customer.').copies[0].fact_id;+ const stub = stubChat();+ let second: Report;+ try { second = await ingest(engine, { content: 'Follow-up call. Zentrix signed its first enterprise customer last week, the team said again.', title: 'Follow-up' }); } finally { stub.restore(); }+ expect(second.facts.find((f: Report) => f.fact === 'Zentrix signed its first enterprise customer.').copies).toEqual([expect.objectContaining({ entity_slug: 'companies/zentrix', status: 'support', fact_id: zentrixFact })]);+ const preview = await undo(engine, { attempt_id: first.attempt_id });+ expect(preview.skipped).toEqual(expect.arrayContaining([{ kind: 'fact', ref: String(zentrixFact), reason: 'supported' },+ { kind: 'page', ref: 'companies/zentrix', reason: 'referenced' }, { kind: 'page', ref: first.source_page.slug, reason: 'referenced' }]));+ expect(preview.delete.map((d: Report) => d.slug)).toEqual(['companies/acmx']);+ const applied = await undo(engine, { attempt_id: first.attempt_id, yes: true, expect: preview.plan_hash });+ expect(applied.status).toBe('undone');+ expect(await engine.getPage('companies/zentrix')).not.toBeNull();+ expect(await engine.getPage(first.source_page.slug)).not.toBeNull();+ expect(await engine.getPage('companies/acmx')).toBeNull();+ const kept = (await factsOn(engine, 'companies/zentrix')).find(f => Number(f.id) === zentrixFact)!;+ expect(kept.valid_until).toBeNull();+ expect(kept.expired_at).toBeNull();+}++/** Without the attempt's journal rows undo cannot prove ownership, so it refuses and changes nothing. */+export async function undoUnverifiableScenario(engine: BrainEngine): Promise<void> {+ const report = await goldenRun(engine);+ const preview = await undo(engine, { attempt_id: report.attempt_id });+ await engine.executeRaw("DELETE FROM op_checkpoints WHERE op LIKE 'ingest-%'");+ const before = await pageCount(engine);+ for (const params of [{ attempt_id: report.attempt_id }, { attempt_id: report.attempt_id, yes: true, expect: preview.plan_hash }]) {+ expect((await undo(engine, params).catch(e => e) as OperationError).code).toBe('ingest_undo_unverifiable');+ }+ expect(await pageCount(engine)).toBe(before);+ expect(await engine.getPage('companies/zentrix')).not.toBeNull();+ expect((await factsOn(engine, 'companies/zentrix')).every(f => f.valid_until === null && f.expired_at === null)).toBe(true);+}+diff --git a/test/ingest.serial.test.ts b/test/ingest.serial.test.tsindex 6372a188bb..154e70618f 100644--- a/test/ingest.serial.test.ts+++ b/test/ingest.serial.test.ts@@ -80,6 +80,34 @@ for (const storage of ['git', 'database'] as ReviewStorage[]) {fx = await reviewDoorFixture(engine, storage);await S.namespaceScenario(engine);}, 60_000);+ test('a private ingest derives only private pages and claims, unreadable remotely', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.privacyScenario(engine);+ }, 60_000);+ test('a natural slug held by an unreadable page takes the hash-tagged slug, proposed', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.heldSlugScenario(engine);+ }, 60_000);+ test('grants: refused before any provider call without the child capabilities; bound prefixes skip outside_grant', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.grantScenario(engine);+ }, 60_000);+ test('a slug created by another writer after planning is planned again once', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.createConflictScenario(engine);+ }, 60_000);+ test('undo keeps promoted and edited pages', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.undoKeepsChangedPagesScenario(engine);+ }, 90_000);+ test('undo keeps a claim another attempt relies on, with its pages', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.undoKeepsReliedOnScenario(engine);+ }, 90_000);+ test('undo refuses ingest_undo_unverifiable without the attempt journal', async () => {+ fx = await reviewDoorFixture(engine, storage);+ await S.undoUnverifiableScenario(engine);+ }, 60_000);});}