Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(ingest): frontmatter-only report type, grant preflight, undo keeps relied-on claims, re-plan on create conflict (GBRA-215 L3)

garrytan525fc82bd553
diff --git a/docs/guides/ingest.md b/docs/guides/ingest.md
index 087077916e..3ca09e6c66 100644
--- a/docs/guides/ingest.md
+++ b/docs/guides/ingest.md
@@ -148,24 +148,28 @@ receipts' `persistence.receipt_retention_days` window, default 30) and cross-che
no other attempt uses it.
It keeps anything edited, promoted or linked from another page since, support it counted for a claim that already
-existed, and claims another attempt counts as support. Apply runs under the source's publication guard and refuses
+existed, and claims another attempt counts as support, with their page and the source page they cite. Apply runs under the source's publication guard and refuses
`preview_changed` when the plan moved since the preview. A repeat is a no-op. After a rebuild that renumbered the
-facts, undo refuses `ingest_undo_unverifiable` rather than guess. A page whose body is segmented keeps its rows
+facts, or without the attempt record, undo refuses `ingest_undo_unverifiable` rather than guess. A page whose body is segmented keeps its rows
(reported `segmented`).
## Reserved namespace
Report pages (type `ingest-report`, slugs `ingest/<attempt_id>`) are written only by the attempt that owns them.
Every generic page write (put, edit, append, delete, restore, revert, batch, upload) that names one, or declares the
-type, is refused `ingest_namespace_reserved`. Other slugs under `ingest/` stay ordinary pages; an owner who wants
+type in a page's leading frontmatter, is refused `ingest_namespace_reserved`; the same line in a page body is ordinary text. Other slugs under `ingest/` stay ordinary pages; an owner who wants
the whole prefix reserved sets `gbrain config set ingest.reserve_namespace true`.
## Remote callers
`ingest` and `ingest_undo` are on the full and starter MCP surfaces. A remote ingest always proposes new pages,
cannot set `visibility: private` or `trusted_extraction`, and sees only its own attempts; pages it cannot read are
-withheld from reports by slug. A bound-slug client writes entities only inside its prefixes (others are
-`skipped: outside_grant`), while its report page is still written.
+withheld from reports by slug. Before any provider call, ingest checks that the caller's grant covers what it
+commits: `put_page` on the source page slug and on `ingest/` (the report page), and `remember` when it extracts
+facts. Otherwise it refuses `permission_denied` with nothing extracted or written. A bound-slug client writes
+entities only inside its prefixes (others are `skipped: outside_grant`). When another writer creates a planned slug
+first, ingest plans that entity once more and reports what it found (`existing`, a hash-tagged slug, or
+`skipped: create_conflict`).
## Errors
diff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsv
index 7e89396c38..f81a3f0b0f 100644
--- a/scripts/page-body-readers-baseline.tsv
+++ b/scripts/page-body-readers-baseline.tsv
@@ -318,6 +318,7 @@ src/core/import-identity.ts decideImportIdentity field META
src/core/import-screen.ts assessImportSanity field STREAM
src/core/import-screen.ts screenImportContent call:contentHash META
src/core/import-screen.ts screenImportContent field STREAM
+src/core/ingest/namespace.ts editedHead call:serializePageToMarkdown NONE serializes the frontmatter head with an empty body to apply edit_page edits that reach it; the body is never read
src/core/line-grammar-report.ts grammarDiagnosticsForPage field INLINE grammar_diagnostics parses inline bodies only; a segmented page gets the skip (segmentedGrammarSkip)
src/core/link-reconciliation.ts reconcileSourceLinks field INLINE inline arm; a segmented page streams through extractSegmentedPageLinks (page-body/segment-links.ts)
src/core/link-temporal-apply.ts applyTemporalEvidence field STREAM
diff --git a/src/core/ingest/namespace.ts b/src/core/ingest/namespace.ts
index 679c7398bb..cedf786d1e 100644
--- a/src/core/ingest/namespace.ts
+++ b/src/core/ingest/namespace.ts
@@ -11,11 +11,43 @@
import { opError, type OperationError } from '../ops/contract.ts';
import { readFix } from '../ops/op-fix.ts';
import type { BrainEngine } from '../engine.ts';
+import { coerceFrontmatterString, parseMarkdownHead, serializePageToMarkdown } from '../markdown.ts';
+import { withoutBody } from '../page-body/page-shape.ts';
+import type { PageSnapshot } from '../page-state/types.ts';
+import type { Page } from '../types.ts';
import { INGEST_REPORT_PREFIX, INGEST_REPORT_TYPE } from './report.ts';
export const RESERVE_NAMESPACE_KEY = 'ingest.reserve_namespace';
const REPORT_SLUG = /^ingest\/a[0-9a-f]{16}$/;
-const DECLARES_REPORT_TYPE = /^type:\s*["']?ingest-report["']?\s*$/m;
+const LEADING_FRONTMATTER = /^---[^\n]*\n(?:[\s\S]*?\n)?---[ \t]*(?:\n|$)/;
+
+/** Whether a document's leading frontmatter declares the report type; the body never does. */
+function declaresReportType(text: string): boolean {
+ return coerceFrontmatterString(parseMarkdownHead(text).frontmatter.type) === INGEST_REPORT_TYPE;
+}
+
+/**
+ * The existing page's frontmatter head after the edits that reach it. An edit matched only after the head's closing
+ * fence changes the body, which cannot change the frontmatter; one that straddles the closing fence is applied to the
+ * head as far as it reaches. No page body is read.
+ */
+function editedHead(existing: Pick<PageSnapshot, 'page' | 'tags'>, edits: unknown[]): string {
+ let head = serializePageToMarkdown({ ...withoutBody(existing.page), compiled_truth: '', timeline: '' } as Page, existing.tags).slice(0, -1);
+ for (const edit of edits) {
+ const { old_text: oldText, new_text: newText } = (edit ?? {}) as { old_text?: unknown; new_text?: unknown };
+ if (typeof oldText !== 'string' || !oldText || typeof newText !== 'string') continue;
+ const close = LEADING_FRONTMATTER.exec(head)?.[0].length ?? head.length;
+ const at = head.indexOf(oldText);
+ if (at !== -1 && at < close) {
+ head = head.slice(0, at) + newText + head.slice(at + oldText.length);
+ continue;
+ }
+ for (let from = Math.max(0, head.length - oldText.length + 1); from < close - 1; from++) {
+ if (oldText.startsWith(head.slice(from))) { head = head.slice(0, from) + newText; break; }
+ }
+ }
+ return head;
+}
function refused(slug: string, why: string): OperationError {
const attempt = slug.slice(INGEST_REPORT_PREFIX.length);
@@ -25,13 +57,14 @@ function refused(slug: string, why: string): OperationError {
}
/** The reserved-type and reserved-namespace check every generic page mutation passes at admission. */
-export async function assertIngestNamespaceWritable(engine: BrainEngine, input: { operation: string; slug: string; existingType: string | null | undefined; params: Record<string, unknown> }): Promise<void> {
- if (input.existingType === INGEST_REPORT_TYPE) throw refused(input.slug, 'Ingest reports are written only by the ingest attempt that owns them.');
- const texts: string[] = [];
- if (typeof input.params.content === 'string') texts.push(input.params.content.slice(0, 8192));
- if (input.params.type === INGEST_REPORT_TYPE) throw refused(input.slug, 'The ingest-report page type is reserved for ingest.');
- if (Array.isArray(input.params.edits)) for (const e of input.params.edits) if (typeof (e as { new_text?: unknown })?.new_text === 'string') texts.push((e as { new_text: string }).new_text);
- if (texts.some(t => DECLARES_REPORT_TYPE.test(t))) throw refused(input.slug, 'The ingest-report page type is reserved for ingest.');
+export async function assertIngestNamespaceWritable(engine: BrainEngine, input: { operation: string; slug: string; existing: Pick<PageSnapshot, 'page' | 'tags'> | null | undefined; params: Record<string, unknown> }): Promise<void> {
+ const existing = input.existing && !input.existing.page.deleted_at ? input.existing : null;
+ if (input.existing?.page.type === INGEST_REPORT_TYPE) throw refused(input.slug, 'Ingest reports are written only by the ingest attempt that owns them.');
+ const { content, edits } = input.params;
+ if (input.params.type === INGEST_REPORT_TYPE || (typeof content === 'string' && declaresReportType(content))
+ || (Array.isArray(edits) && existing && declaresReportType(editedHead(existing, edits)))) {
+ throw refused(input.slug, 'The ingest-report page type is reserved for ingest.');
+ }
if (!input.slug.startsWith(INGEST_REPORT_PREFIX)) return;
if (REPORT_SLUG.test(input.slug)) throw refused(input.slug, 'Slugs of the form ingest/<attempt_id> are written only by the ingest attempt that owns them.');
const reserve = await engine.getConfig(RESERVE_NAMESPACE_KEY);
diff --git a/src/core/ops/ingest.ts b/src/core/ops/ingest.ts
index 777c46e696..34d63f926f 100644
--- a/src/core/ops/ingest.ts
+++ b/src/core/ops/ingest.ts
@@ -31,9 +31,9 @@ import {
INGEST_ENTITY_CAP, INGEST_MAX_CHARS, IngestProviderFailure, extractWindow, extractionKillSwitchOn, ingestModel, keylessCandidates,
mergeWindows, windowDocument, type ExtractedWindow,
} from '../ingest/extract.ts';
-import { planIngest, type IngestPlan } from '../ingest/plan.ts';
+import { planIngest, type IngestPlan, type PlannedPage } from '../ingest/plan.ts';
import {
- INGEST_REPORT_TYPE, filterReportForCaller, reportActions, reportPageContent, reportSlugFor, resolveStatus, summarize,
+ INGEST_REPORT_PREFIX, INGEST_REPORT_TYPE, filterReportForCaller, reportActions, reportPageContent, reportSlugFor, resolveStatus, summarize,
type IngestReport, type ReportEvidence, type ReportFact, type ReportPage,
} from '../ingest/report.ts';
@@ -195,6 +195,26 @@ async function principalOf(ctx: OperationContext) {
return requestPrincipalForContext(ctx);
}
+/**
+ * Plan step 1: before any provider call or write, the caller's grant must cover what ingest commits: the source page,
+ * the report page under ingest/ and, when facts are extracted, remember. The door's own authorization answers it.
+ */
+async function assertCallerCanCommit(ctx: OperationContext, sourceId: string, incarnation: string, sourceSlug: string, facts: boolean): Promise<void> {
+ const { submissionAuthority } = await import('../persistence/authority.ts');
+ const needs: Array<[string, string, string]> = [['put_page', sourceSlug, 'the source page ' + sourceSlug], ['put_page', reportSlugFor('a' + '0'.repeat(16)), 'the report page under ' + INGEST_REPORT_PREFIX],
+ ...(facts ? [['remember', sourceSlug, 'facts (remember)'] as [string, string, string]] : [])];
+ for (const [operation, slug, what] of needs) {
+ try {
+ await submissionAuthority(ctx, operation, sourceId, incarnation, slug);
+ } catch (error) {
+ if (!(error instanceof OperationError)) throw error;
+ throw opError('permission_denied', 'This caller cannot write ' + what + ', which ingest commits: ' + error.message,
+ 'Nothing was extracted or written. The grant must cover the source page slug, ' + INGEST_REPORT_PREFIX + ', put_page and remember; ask the brain owner to widen it, or pass a slug inside it.',
+ { docs: DOCS + '#remote-callers' });
+ }
+ }
+}
+
/** Status, resume and undo by attempt id: the local owner reads every attempt, any other caller only its own. */
async function callerMayRead(ctx: OperationContext, state: IngestAttemptState): Promise<boolean> {
return ctx.remote === false || principalKey(await principalOf(ctx)) === principalKey(state.principal);
@@ -302,17 +322,30 @@ async function commitPlan(ctx: OperationContext, state: IngestAttemptState, plan
const base: ReportPage = { name: page.name, slug: page.slug, type: page.type, status: 'skipped', ...(page.near_names ? { near_names: page.near_names } : {}) };
if (page.status === 'skipped') { report.pages.push({ ...base, reason: page.reason }); continue; }
if (page.status === 'existing') { const row = { ...base, status: 'existing' as const }; report.pages.push(row); written.set(page.slug, row); continue; }
- const proposed = page.status === 'propose';
- const row = await withEntityCreateLock(ctx, sourceId, async (): Promise<ReportPage> => {
- const snap = await ctx.engine.readPageSnapshot(page.slug, { sourceId, includeDeleted: true });
- if (snap && !snap.page.deleted_at) return { ...base, status: 'skipped', reason: 'create_conflict' };
- const r = await publish(ctx, sourceId, { slug: page.slug, content: entityPageContent(page, proposed, state), ...(snap ? { expected_revision: snap.revision } : {}) });
- if (!r.ok) return { ...base, status: 'failed', reason: r.code };
- report.write_requests.push({ slug: page.slug, request_id: r.request_id, state: r.pending ? 'pending' : 'committed' });
- if (r.pending) return { ...base, status: 'pending', request_id: r.request_id };
- evidence.created_pages.push({ slug: page.slug, request_id: r.request_id, revision: null, proposed });
- return { ...base, status: proposed ? 'proposed' : 'created', request_id: r.request_id };
+ const create = (target: PlannedPage) => withEntityCreateLock(ctx, sourceId, async (): Promise<ReportPage | null> => {
+ const proposed = target.status === 'propose';
+ const at = { ...base, slug: target.slug };
+ const snap = await ctx.engine.readPageSnapshot(target.slug, { sourceId, includeDeleted: true });
+ if (snap && !snap.page.deleted_at) return null;
+ const r = await publish(ctx, sourceId, { slug: target.slug, content: entityPageContent(target, proposed, state), ...(snap ? { expected_revision: snap.revision } : {}) });
+ if (!r.ok) return { ...at, status: 'failed', reason: r.code };
+ report.write_requests.push({ slug: target.slug, request_id: r.request_id, state: r.pending ? 'pending' : 'committed' });
+ if (r.pending) return { ...at, status: 'pending', request_id: r.request_id };
+ evidence.created_pages.push({ slug: target.slug, request_id: r.request_id, revision: null, proposed });
+ return { ...at, status: proposed ? 'proposed' : 'created', request_id: r.request_id };
});
+ const conflict: ReportPage = { ...base, status: 'skipped', reason: 'create_conflict' };
+ let row = await create(page);
+ if (!row) {
+ // Another writer took the slug after planning: plan this one entity again against the brain as it is now, once.
+ const [again] = (await planIngest({ ctx, sourceId, text, entities: [{ name: page.name, type: page.type, context: page.context, source: page.source }], facts: [],
+ visibility: state.options.visibility, validFrom: null, trusted: state.options.trusted_extraction, factsSkipped: null })).pages;
+ row = !again ? conflict
+ : again.status === 'existing' ? { ...base, slug: again.slug, status: 'existing' }
+ : again.status === 'skipped' ? { ...base, slug: again.slug || page.slug, status: 'skipped', reason: again.reason }
+ : (await create(again)) ?? conflict;
+ if (row.status === 'existing') { report.pages.push(row); written.set(page.slug, row); continue; }
+ }
report.pages.push(row);
if (row.status === 'proposed' || row.status === 'created') written.set(page.slug, row);
}
@@ -330,27 +363,27 @@ async function commitPlan(ctx: OperationContext, state: IngestAttemptState, plan
row.copies.push({ entity_slug: c.entity_slug, status: pageRow?.status === 'pending' ? 'pending' : 'skipped', reason: pageRow?.status === 'pending' ? 'page_pending' : 'page_not_written' });
continue;
}
- const check = await checkFactAgainstSaved(ctx.engine, sourceId, { fact: f.fact, kind: f.kind, visibility: state.options.visibility, entity_slug: c.entity_slug }, { excludePrivate });
- if (check.verdict === 'withdrawn') { row.copies.push({ entity_slug: c.entity_slug, status: 'skipped', reason: 'withdrawn' }); continue; }
+ const check = await checkFactAgainstSaved(ctx.engine, sourceId, { fact: f.fact, kind: f.kind, visibility: state.options.visibility, entity_slug: target.slug }, { excludePrivate });
+ if (check.verdict === 'withdrawn') { row.copies.push({ entity_slug: target.slug, status: 'skipped', reason: 'withdrawn' }); continue; }
if (check.verdict === 'duplicate') {
- row.copies.push({ entity_slug: c.entity_slug, status: 'support', fact_id: check.id });
+ row.copies.push({ entity_slug: target.slug, status: 'support', fact_id: check.id });
if (!evidence.support_fact_ids.includes(check.id)) evidence.support_fact_ids.push(check.id);
continue;
}
const request_id = randomUUID();
try {
const res = asRecord(await remember.handler(ctx, {
- fact: f.fact, entity: c.entity_slug, provenance: 'ingest:' + state.source_slug, kind: f.kind, request_id,
+ fact: f.fact, entity: target.slug, provenance: 'ingest:' + state.source_slug, kind: f.kind, request_id,
visibility: state.options.visibility, ...(f.valid_from ? { valid_from: f.valid_from } : {}), infer_entity: false,
}));
const id = Number(res.fact_id ?? res.id ?? asRecord((res.facts as unknown[] | undefined)?.[0]).id);
const pending = pendingOf(res);
- row.copies.push({ entity_slug: c.entity_slug, status: pending ? 'pending' : 'created', ...(Number.isFinite(id) ? { fact_id: id } : {}), request_id });
- report.write_requests.push({ slug: c.entity_slug, request_id, state: pending ? 'pending' : 'committed' });
- if (!pending && Number.isFinite(id)) evidence.created_facts.push({ id, entity_slug: c.entity_slug, request_id });
+ row.copies.push({ entity_slug: target.slug, status: pending ? 'pending' : 'created', ...(Number.isFinite(id) ? { fact_id: id } : {}), request_id });
+ report.write_requests.push({ slug: target.slug, request_id, state: pending ? 'pending' : 'committed' });
+ if (!pending && Number.isFinite(id)) evidence.created_facts.push({ id, entity_slug: target.slug, request_id });
} catch (error) {
if (!(error instanceof OperationError)) throw error;
- row.copies.push({ entity_slug: c.entity_slug, status: 'failed', reason: error.code, request_id });
+ row.copies.push({ entity_slug: target.slug, status: 'failed', reason: error.code, request_id });
}
}
// D3 and round-4 item 11: the links effect derives no mentions edge from fence rows on either storage (checked in
@@ -409,6 +442,7 @@ async function runIngest(ctx: OperationContext, p: Params): Promise<Record<strin
}
if (state && state.phase === 'complete') return storedReport(ctx, state, { repeated: true });
if (state && state.phase !== 'undone' && p.resume !== true) return storedReport(ctx, state, { repeated: true, resumable: true });
+ await assertCallerCanCommit(ctx, sourceId, topology.source_incarnation, state?.source_slug ?? v.slug ?? 'inbox/ingest-' + documentHash.slice(0, 12), v.options.keyed && !v.options.no_facts);
if (!state || state.phase === 'undone') {
const now = new Date().toISOString();
const attempt_id = newAttemptId();
@@ -515,6 +549,7 @@ async function planUndo(ctx: OperationContext, state: IngestAttemptState, ev: Re
const sourceId = state.source_id;
const out: UndoPlan = { retract: [], delete: [], skipped: [] };
const ids = ev.created_facts.map(f => f.id);
+ const supportedOn = new Set<string>();
const rows = ids.length ? await ctx.engine.executeRaw<{ id: number; entity_slug: string | null; row_num: number | null; expired_at: unknown; valid_until: unknown; source: string | null }>(
'SELECT id, entity_slug, row_num, expired_at, valid_until, source FROM facts WHERE source_id=$1 AND id = ANY($2::int[])', [sourceId, ids]) : [];
const byId = new Map(rows.map(r => [Number(r.id), r]));
@@ -523,7 +558,7 @@ async function planUndo(ctx: OperationContext, state: IngestAttemptState, ev: Re
const row = byId.get(f.id);
if (!row || row.entity_slug !== f.entity_slug || row.source !== 'ingest:' + ev.source_slug) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'not_found' }); continue; }
if (row.expired_at || row.valid_until) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'already_retracted' }); continue; }
- if ((await supportingAttempts(ctx.engine, f.id, state.attempt_id)).length) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'supported' }); continue; }
+ if ((await supportingAttempts(ctx.engine, f.id, state.attempt_id)).length) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'supported' }); supportedOn.add(f.entity_slug); continue; }
if (row.row_num == null) { out.skipped.push({ kind: 'fact', ref: String(f.id), reason: 'not_fenced' }); continue; }
out.retract.push({ fact_id: f.id, entity_slug: f.entity_slug, row_num: Number(row.row_num) });
}
@@ -537,13 +572,14 @@ async function planUndo(ctx: OperationContext, state: IngestAttemptState, ev: Re
if (!snap || snap.page.deleted_at) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'not_found' }); continue; }
if (pg.proposed && !isUnverifiedExtraction(asRecord(snap.page.frontmatter))) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'promoted' }); continue; }
if (pg.revision && snap.revision !== pg.revision) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'edited' }); continue; }
- if ((await referencedFrom(pg.slug)).length) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'referenced' }); continue; }
+ // A claim another attempt relies on keeps its page, and the source page it cites.
+ if (supportedOn.has(pg.slug) || (await referencedFrom(pg.slug)).length) { out.skipped.push({ kind: 'page', ref: pg.slug, reason: 'referenced' }); continue; }
out.delete.push({ slug: pg.slug });
}
if (ev.source_created) {
const others = await attemptsOfDocument(ctx.engine, sourceId, state.document_hash, state.attempt_id);
if (others.length) out.skipped.push({ kind: 'page', ref: ev.source_slug, reason: 'used_by_other_attempt' });
- else if ((await referencedFrom(ev.source_slug)).length) out.skipped.push({ kind: 'page', ref: ev.source_slug, reason: 'referenced' });
+ else if (supportedOn.size || (await referencedFrom(ev.source_slug)).length) out.skipped.push({ kind: 'page', ref: ev.source_slug, reason: 'referenced' });
else out.delete.push({ slug: ev.source_slug });
}
return out;
diff --git a/src/core/persistence/page-mutations.ts b/src/core/persistence/page-mutations.ts
index 95316e7b59..69dbdad085 100644
--- a/src/core/persistence/page-mutations.ts
+++ b/src/core/persistence/page-mutations.ts
@@ -363,12 +363,10 @@ export async function preparePageAdmission(ctx: OperationContext,
if (typeof intent.capture_path === 'string') intent.capture_file_hash = sha256(p.content);
}
await validateMutationSlug(ctx, input.operation, slug, sourceId);
- // GBRA-215: an attempt's own report keeps the original principal but not its slug fence (a bound client's ingest writes it).
+ enforceClientSlugFence(ctx, slug, input.operation);
+ enforceSubagentSlugFence(ctx, slug, input.operation);
+ // GBRA-215: an attempt's own report page (ingest/<attempt_id>) passes the reserved-namespace check; every grant still applies.
const ingestReport = input.internalIngestReport === true && input.operation === 'put_page' && slug.startsWith('ingest/');
- if (!ingestReport) {
- enforceClientSlugFence(ctx, slug, input.operation);
- enforceSubagentSlugFence(ctx, slug, input.operation);
- }
// Preserve same-source diagnostics for new timeline writes without making
// terminal replay depend on a page that may have since been purged.
if (input.operation === 'add_timeline_entry') await requireWritablePage({ ...ctx, sourceId }, slug, input.operation, 'page');
@@ -383,7 +381,7 @@ export async function preparePageAdmission(ctx: OperationContext,
async () => !source.local_path && sourceId === 'default' ? ctx.engine.getConfig('sync.repo_path') : null,
]) as [Awaited<ReturnType<typeof submissionAuthority>>, unknown, Awaited<ReturnType<BrainEngine['readPageSnapshot']>>, Awaited<ReturnType<typeof getWorktreeBinding>>, string | null, string | null];
if (!ingestReport && input.managedFileImport !== true) {
- await (await import('../ingest/namespace.ts')).assertIngestNamespaceWritable(ctx.engine, { operation: input.operation, slug, existingType: snapshot?.page.type, params: p });
+ await (await import('../ingest/namespace.ts')).assertIngestNamespaceWritable(ctx.engine, { operation: input.operation, slug, existing: snapshot, params: p });
}
if (typeof intent.content === 'string' && (typeof p.expected_revision !== 'string' || p.expected_revision === snapshot?.revision)) {
await assertTimelineNotOmitted(ctx.engine, { intent, remote: ctx.remote !== false, writer: typeof p.expected_revision === 'string' ? 'editing' : 'preserving',
diff --git a/test/e2e/ingest-postgres.test.ts b/test/e2e/ingest-postgres.test.ts
index 6d82d60d55..71d1e650e6 100644
--- a/test/e2e/ingest-postgres.test.ts
+++ b/test/e2e/ingest-postgres.test.ts
@@ -71,6 +71,13 @@ d('ingest and ingest_undo (live Postgres)', () => {
test('validation (' + label + ')', async () => { fx = await fixture(storage); await S.validationScenario(engine); }, 60_000);
test('two-entity claim (' + label + ')', async () => { fx = await fixture(storage); await S.twoEntityScenario(engine); }, 60_000);
test('reserved report namespace (' + label + ')', async () => { fx = await fixture(storage); await S.namespaceScenario(engine); }, 60_000);
+ test('private ingest (' + label + ')', async () => { fx = await fixture(storage); await S.privacyScenario(engine); }, 60_000);
+ test('held natural slug (' + label + ')', async () => { fx = await fixture(storage); await S.heldSlugScenario(engine); }, 60_000);
+ test('grants (' + label + ')', async () => { fx = await fixture(storage); await S.grantScenario(engine); }, 60_000);
+ test('create conflict re-plan (' + label + ')', async () => { fx = await fixture(storage); await S.createConflictScenario(engine); }, 60_000);
+ test('undo keeps changed pages (' + label + ')', async () => { fx = await fixture(storage); await S.undoKeepsChangedPagesScenario(engine); }, 90_000);
+ test('undo keeps relied-on claims (' + label + ')', async () => { fx = await fixture(storage); await S.undoKeepsReliedOnScenario(engine); }, 90_000);
+ test('undo unverifiable (' + label + ')', async () => { fx = await fixture(storage); await S.undoUnverifiableScenario(engine); }, 60_000);
}
test('Git target unavailable refuses before writes', async () => { fx = await fixture('git'); await S.storageUnavailableScenario(engine); }, 60_000);
});
diff --git a/test/helpers/ingest-scenarios.ts b/test/helpers/ingest-scenarios.ts
index f028421be8..50b964fe2f 100644
--- a/test/helpers/ingest-scenarios.ts
+++ b/test/helpers/ingest-scenarios.ts
@@ -11,6 +11,8 @@ import { localHostId } from '../../src/core/persistence/identity.ts';
import { parseMarkdown } from '../../src/core/markdown.ts';
import { __setChatTransportForTests, configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
import { reviewCtx, committedFile, type ReviewDoorFixture } from './review-door-scenarios.ts';
+import { randomUUID } from 'node:crypto';
+import { tryAcquireDbLock } from '../../src/core/db-lock.ts';
export type Report = Record<string, any>;
@@ -331,8 +333,30 @@ export async function namespaceScenario(engine: BrainEngine): Promise<void> {
}
const typed = await put.handler(reviewCtx(engine), { slug: 'notes/fake-report', content: '---\ntype: ingest-report\ntitle: x\n---\n\nmine\n' }).catch(e => e) as OperationError;
expect(typed.code).toBe('ingest_namespace_reserved');
+ const reportRev = (await engine.readPageSnapshot(report.report_page))!.revision;
+ const editReport = await operationsByName.edit_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: report.report_page, expected_revision: reportRev,
+ edits: [{ old_text: 'ingest', new_text: 'mine' }] }).catch(e => e) as OperationError;
+ expect(editReport.code).toBe('ingest_namespace_reserved');
const del = await operationsByName.delete_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: report.report_page }).catch(e => e) as OperationError;
expect(del.code).toBe('ingest_namespace_reserved');
+ // Only the leading frontmatter declares a type: the same line in the body is ordinary text, written and edited as before.
+ await put.handler(reviewCtx(engine, { remote: true } as never), { slug: 'notes/config-example', content: '---\ntype: note\ntitle: Config example\n---\n\nA report page starts with:\n\ntype: ingest-report\n' });
+ await drain(engine);
+ const bodyOnly = (await engine.readPageSnapshot('notes/config-example'))!;
+ expect(bodyOnly.page.type).toBe('note');
+ await operationsByName.edit_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: 'notes/config-example', expected_revision: bodyOnly.revision,
+ edits: [{ old_text: 'A report page starts with:', new_text: 'A report page starts with:\n\n---\ntype: ingest-report\n---' }] });
+ await drain(engine);
+ const edited = (await engine.readPageSnapshot('notes/config-example'))!;
+ expect(edited.revision).not.toBe(bodyOnly.revision);
+ expect(edited.page.type).toBe('note');
+ // An edit that puts the type into the frontmatter head is refused, whole line or value alone.
+ for (const change of [{ old_text: 'type: note', new_text: 'type: ingest-report' }, { old_text: 'type: note', new_text: 'type: "ingest-report"' }]) {
+ const headEdit = await operationsByName.edit_page!.handler(reviewCtx(engine, { remote: true } as never), { slug: 'notes/config-example', expected_revision: edited.revision,
+ edits: [change] }).catch(e => e) as OperationError;
+ expect(headEdit.code).toBe('ingest_namespace_reserved');
+ }
+ expect((await engine.readPageSnapshot('notes/config-example'))!.revision).toBe(edited.revision);
// Other slugs under ingest/ stay ordinary pages unless the owner reserves the namespace.
await put.handler(reviewCtx(engine), { slug: 'ingest/notes', content: '---\ntype: note\ntitle: x\n---\n\nmine\n' });
await engine.setConfig('ingest.reserve_namespace', 'true');
@@ -372,3 +396,174 @@ export async function twoEntityScenario(engine: BrainEngine): Promise<Array<{ fr
return edges;
} finally { stub.restore(); }
}
+
+/** A remote OAuth client on the default source with the given operation and slug grants. */
+async function oauthCaller(engine: BrainEngine, grant: { allowedOperations?: string[]; boundSlugPrefixes?: string[] }): Promise<Partial<OperationContext>> {
+ const clientId = 'ingest-client-' + randomUUID().slice(0, 8);
+ await engine.executeRaw("INSERT INTO oauth_clients(client_id,client_secret_hash,client_name,scope,source_id,allowed_operations,bound_slug_prefixes) VALUES($1,'fixture-hash','example-client','read write','default',$2,$3)",
+ [clientId, grant.allowedOperations ?? null, grant.boundSlugPrefixes ?? null]);
+ return { remote: true, transport: 'http', auth: { token: 'fixture', clientId, principal: { kind: 'oauth_client', id: clientId }, scopes: ['read', 'write'], sourceId: 'default', ...grant } } as Partial<OperationContext>;
+}
+
+const ingestCheckpoints = async (engine: BrainEngine) => Number((await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM op_checkpoints WHERE op LIKE 'ingest-%'"))[0]!.n);
+
+/** A private ingest: every page it derives and every claim it saves is private, and a remote caller reads none of it. */
+export async function privacyScenario(engine: BrainEngine): Promise<void> {
+ await seedEntities(engine);
+ const stub = stubChat();
+ try {
+ const report = await ingest(engine, { content: goldenDocument(), title: 'Call notes', visibility: 'private' });
+ expect(report.status).toBe('complete');
+ const derived = [report.source_page.slug, report.report_page, ...report.pages.filter((p: Report) => p.status === 'proposed').map((p: Report) => p.slug)];
+ expect(derived).toEqual(expect.arrayContaining(['companies/zentrix', 'companies/acmx']));
+ const remote = reviewCtx(engine, { remote: true } as never);
+ for (const slug of derived) {
+ expect((await engine.getPage(slug))!.frontmatter.visibility).toBe('private');
+ expect((await operationsByName.get_page!.handler(remote, { slug }).catch(e => e) as OperationError).code).toBe('page_not_found');
+ }
+ const facts = await engine.executeRaw<{ visibility: string }>('SELECT visibility FROM facts WHERE source = $1', ['ingest:' + report.source_page.slug]);
+ expect(facts.length).toBe(3);
+ expect(facts.every(f => f.visibility === 'private')).toBe(true);
+ // The claim saved on an existing world page stays out of a remote read of that page.
+ const alice = JSON.stringify(await operationsByName.get_page!.handler(remote, { slug: 'people/alice-example' }));
+ expect(alice).toContain('Alice Example is a founder.');
+ expect(alice).not.toContain('Lisbon');
+ } finally { stub.restore(); }
+}
+
+/** A natural slug held by a page the caller cannot read takes the hash-tagged slug and stays proposed, never existing. */
+export async function heldSlugScenario(engine: BrainEngine): Promise<void> {
+ await seedEntities(engine);
+ await operationsByName.put_page!.handler(reviewCtx(engine), { slug: 'companies/zentrix', content: '---\ntype: company\ntitle: Zentrix\nvisibility: private\n---\n\nPrivate notes.\n' });
+ await drain(engine);
+ const stub = stubChat();
+ try {
+ const report = await ingest(engine, { content: goldenDocument(), title: 'Call notes' }, { remote: true } as never);
+ const zentrix = report.pages.find((p: Report) => p.name === 'Zentrix');
+ expect(zentrix.status).toBe('proposed');
+ expect(zentrix.slug).toMatch(/^companies\/zentrix-[0-9a-f]{6}$/);
+ expect(report.pages.some((p: Report) => p.slug === 'companies/zentrix')).toBe(false);
+ expect((await engine.getPage('companies/zentrix'))!.compiled_truth.trim()).toBe('Private notes.');
+ expect((await factsOn(engine, 'companies/zentrix')).length).toBe(0);
+ } finally { stub.restore(); }
+}
+
+/** Grants: a caller that cannot write what ingest commits is refused before any provider call; a bound one skips entities outside its prefixes. */
+export async function grantScenario(engine: BrainEngine): Promise<void> {
+ await seedEntities(engine);
+ const stub = stubChat();
+ try {
+ for (const grant of [{ allowedOperations: ['ingest'] }, { boundSlugPrefixes: ['companies/'] }]) {
+ const before = await pageCount(engine);
+ const err = await ingest(engine, { content: goldenDocument(), title: 'Call notes' }, await oauthCaller(engine, grant)).catch(e => e) as OperationError;
+ expect(err.code).toBe('permission_denied');
+ expect(stub.calls).toBe(0);
+ expect(await pageCount(engine)).toBe(before);
+ expect(await ingestCheckpoints(engine)).toBe(0);
+ }
+ const report = await ingest(engine, { content: goldenDocument(), title: 'Call notes' }, await oauthCaller(engine, { boundSlugPrefixes: ['companies/', 'inbox/', 'ingest/'] }));
+ expect(report.status).toBe('complete');
+ expect(report.pages.find((p: Report) => p.name === 'Alice Example')).toMatchObject({ slug: 'people/alice-example', status: 'skipped', reason: 'outside_grant' });
+ expect(report.pages.find((p: Report) => p.name === 'Zentrix')).toMatchObject({ slug: 'companies/zentrix', status: 'proposed' });
+ expect((await engine.getPage(report.report_page))!.type).toBe('ingest-report');
+ expect((await factsOn(engine, 'people/alice-example')).length).toBe(0);
+ } finally { stub.restore(); }
+}
+
+/** Another writer creates a planned slug before the attempt does: the entity is planned again once, and its claim lands on that page. */
+export async function createConflictScenario(engine: BrainEngine): Promise<void> {
+ await seedEntities(engine);
+ const stub = stubChat();
+ const lock = await tryAcquireDbLock(engine, 'ingest-entity-create:default', 2);
+ expect(lock).not.toBeNull();
+ let held = true;
+ try {
+ const running = ingest(engine, { content: goldenDocument(), title: 'Call notes' });
+ // The report page is written before any entity page, so its row means planning is over and the run waits on the lock.
+ for (let i = 0; i < 150; i++) {
+ const [row] = await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM pages WHERE slug LIKE 'ingest/%' AND deleted_at IS NULL");
+ if (Number(row!.n)) break;
+ await new Promise(resolve => setTimeout(resolve, 100));
+ }
+ await operationsByName.put_page!.handler(reviewCtx(engine), { slug: 'companies/zentrix', content: '---\ntype: company\ntitle: Zentrix\n---\n\nZentrix builds sensors.\n' });
+ await drain(engine);
+ await lock!.release();
+ held = false;
+ const report = await running;
+ expect(report.status).toBe('complete');
+ expect(report.pages.find((p: Report) => p.name === 'Zentrix')).toMatchObject({ slug: 'companies/zentrix', status: 'existing' });
+ const fact = report.facts.find((f: Report) => f.fact === 'Zentrix signed its first enterprise customer.');
+ expect(fact.copies.map((c: Report) => [c.entity_slug, c.status])).toEqual([['companies/zentrix', 'created']]);
+ const page = (await engine.getPage('companies/zentrix'))!;
+ expect(page.compiled_truth).toContain('Zentrix builds sensors.');
+ expect(page.frontmatter.status).toBeUndefined();
+ expect((await factsOn(engine, 'companies/zentrix')).map(f => f.fact)).toEqual(['Zentrix signed its first enterprise customer.']);
+ // The other writer's page is not the attempt's to delete.
+ const preview = await undo(engine, { attempt_id: report.attempt_id });
+ expect(preview.delete.map((d: Report) => d.slug)).not.toContain('companies/zentrix');
+ } finally {
+ if (held) await lock!.release();
+ stub.restore();
+ }
+}
+
+async function goldenRun(engine: BrainEngine): Promise<Report> {
+ await seedEntities(engine);
+ const stub = stubChat();
+ try { return await ingest(engine, { content: goldenDocument(), title: 'Call notes' }); } finally { stub.restore(); }
+}
+
+/** Undo keeps a proposed page someone promoted and a page someone edited after the attempt wrote it. */
+export async function undoKeepsChangedPagesScenario(engine: BrainEngine): Promise<void> {
+ const report = await goldenRun(engine);
+ await operationsByName.extraction_review!.handler(reviewCtx(engine), { action: 'promote', slugs: ['companies/acmx'] });
+ await drain(engine);
+ const snap = (await engine.readPageSnapshot('companies/zentrix'))!;
+ await operationsByName.edit_page!.handler(reviewCtx(engine), { slug: 'companies/zentrix', expected_revision: snap.revision,
+ edits: [{ old_text: 'Zentrix signed its first enterprise customer last week.', new_text: 'Zentrix signed its first enterprise customer last week, per the call.' }] });
+ await drain(engine);
+ const preview = await undo(engine, { attempt_id: report.attempt_id });
+ expect(preview.skipped).toEqual(expect.arrayContaining([{ kind: 'page', ref: 'companies/acmx', reason: 'promoted' }, { kind: 'page', ref: 'companies/zentrix', reason: 'edited' }]));
+ expect(preview.delete.map((d: Report) => d.slug)).toEqual([report.source_page.slug]);
+ const applied = await undo(engine, { attempt_id: report.attempt_id, yes: true, expect: preview.plan_hash });
+ expect(applied.status).toBe('undone');
+ expect((await engine.getPage('companies/acmx'))!.frontmatter.status).not.toBe('unverified');
+ expect((await engine.getPage('companies/zentrix'))!.compiled_truth).toContain('per the call.');
+}
+
+/** Undo keeps a claim another attempt saved as support, with its page and the source page it cites. */
+export async function undoKeepsReliedOnScenario(engine: BrainEngine): Promise<void> {
+ const first = await goldenRun(engine);
+ const zentrixFact = first.facts.find((f: Report) => f.fact === 'Zentrix signed its first enterprise customer.').copies[0].fact_id;
+ const stub = stubChat();
+ let second: Report;
+ try { second = await ingest(engine, { content: 'Follow-up call. Zentrix signed its first enterprise customer last week, the team said again.', title: 'Follow-up' }); } finally { stub.restore(); }
+ expect(second.facts.find((f: Report) => f.fact === 'Zentrix signed its first enterprise customer.').copies).toEqual([expect.objectContaining({ entity_slug: 'companies/zentrix', status: 'support', fact_id: zentrixFact })]);
+ const preview = await undo(engine, { attempt_id: first.attempt_id });
+ expect(preview.skipped).toEqual(expect.arrayContaining([{ kind: 'fact', ref: String(zentrixFact), reason: 'supported' },
+ { kind: 'page', ref: 'companies/zentrix', reason: 'referenced' }, { kind: 'page', ref: first.source_page.slug, reason: 'referenced' }]));
+ expect(preview.delete.map((d: Report) => d.slug)).toEqual(['companies/acmx']);
+ const applied = await undo(engine, { attempt_id: first.attempt_id, yes: true, expect: preview.plan_hash });
+ expect(applied.status).toBe('undone');
+ expect(await engine.getPage('companies/zentrix')).not.toBeNull();
+ expect(await engine.getPage(first.source_page.slug)).not.toBeNull();
+ expect(await engine.getPage('companies/acmx')).toBeNull();
+ const kept = (await factsOn(engine, 'companies/zentrix')).find(f => Number(f.id) === zentrixFact)!;
+ expect(kept.valid_until).toBeNull();
+ expect(kept.expired_at).toBeNull();
+}
+
+/** Without the attempt's journal rows undo cannot prove ownership, so it refuses and changes nothing. */
+export async function undoUnverifiableScenario(engine: BrainEngine): Promise<void> {
+ const report = await goldenRun(engine);
+ const preview = await undo(engine, { attempt_id: report.attempt_id });
+ await engine.executeRaw("DELETE FROM op_checkpoints WHERE op LIKE 'ingest-%'");
+ const before = await pageCount(engine);
+ for (const params of [{ attempt_id: report.attempt_id }, { attempt_id: report.attempt_id, yes: true, expect: preview.plan_hash }]) {
+ expect((await undo(engine, params).catch(e => e) as OperationError).code).toBe('ingest_undo_unverifiable');
+ }
+ expect(await pageCount(engine)).toBe(before);
+ expect(await engine.getPage('companies/zentrix')).not.toBeNull();
+ expect((await factsOn(engine, 'companies/zentrix')).every(f => f.valid_until === null && f.expired_at === null)).toBe(true);
+}
+
diff --git a/test/ingest.serial.test.ts b/test/ingest.serial.test.ts
index 6372a188bb..154e70618f 100644
--- a/test/ingest.serial.test.ts
+++ b/test/ingest.serial.test.ts
@@ -80,6 +80,34 @@ for (const storage of ['git', 'database'] as ReviewStorage[]) {
fx = await reviewDoorFixture(engine, storage);
await S.namespaceScenario(engine);
}, 60_000);
+ test('a private ingest derives only private pages and claims, unreadable remotely', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.privacyScenario(engine);
+ }, 60_000);
+ test('a natural slug held by an unreadable page takes the hash-tagged slug, proposed', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.heldSlugScenario(engine);
+ }, 60_000);
+ test('grants: refused before any provider call without the child capabilities; bound prefixes skip outside_grant', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.grantScenario(engine);
+ }, 60_000);
+ test('a slug created by another writer after planning is planned again once', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.createConflictScenario(engine);
+ }, 60_000);
+ test('undo keeps promoted and edited pages', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.undoKeepsChangedPagesScenario(engine);
+ }, 90_000);
+ test('undo keeps a claim another attempt relies on, with its pages', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.undoKeepsReliedOnScenario(engine);
+ }, 90_000);
+ test('undo refuses ingest_undo_unverifiable without the attempt journal', async () => {
+ fx = await reviewDoorFixture(engine, storage);
+ await S.undoUnverifiableScenario(engine);
+ }, 60_000);
});
}