Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
perf(persistence): batch export reads in the canonical conversion plan (#7617)
garrytan9bb96fe7917d
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.mdindex b6dfaeb406..17dc71b898 100644--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@ -5,7 +5,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGELThe #6188 fence check and fence repair live in [fence repair](core-fence-repair.md). Managed connector ingestion, connector item holds and the no-op kernel (checkpoint validation and the fault seams: [request indexes and fault seams](core-persistence-request-indexes.md)). Also the persistence verb types and errors, database-write and manual links, write attribution and loop-fact retirement; the managed purge of deleted pages is under [memory purge](memory-purge.md).- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies`, so on owner start) touches only interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command), which the refusal hints point at. Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts`; while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.-- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).+- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).- `src/core/persistence/{types,params,preconditions,verb-errors}.ts` — public mutation receipt states, error reasons and revision/idempotency parameter contracts. Receipt serialization explicitly selects public fields; queued content and execution credentials stay internal. `frozenVerbWriteError` uses MEMORY_VERBS v1's existing error enum with additive `write_request`/`write_error`; `page_body_segmented` is a write error code, so a verb refused by a segmented target keeps it as `write_error` and canonical code under the frozen `invalid_params` (caller class, not retryable) on both the receipt and the pre-admission path (`runMemoryWrite`). `remember` reads a segmented entity page up to PAGE_MAX_BYTES back whole (`materializedPage`) for its inferred-target checks and fence write; over it, a named or inferred entity refuses before admission with `segmentedRememberRefusal` (fix: remember with `infer_entity: false`), never a silent `NO_ENTITY` (`test/large-pages/segmented-remember-refusal.test.ts`); `committedVerbOutcome` refuses pending or failed receipts. `OperationError` and the thin MCP client preserve this metadata, including when a timeout races an already-received receipt. The registry-safe parameter/schema leaf never imports operation values. Pinned by `test/persistence-write-contracts.test.ts`, `test/mcp-client-hardening.test.ts` and a real HTTP envelope in `test/mcp-client.test.ts`.diff --git a/src/commands/sources-git-gap.ts b/src/commands/sources-git-gap.tsindex e6fdbb5980..33b3ac37ec 100644--- a/src/commands/sources-git-gap.ts+++ b/src/commands/sources-git-gap.ts@@ -17,7 +17,7 @@ import type { BrainEngine } from '../core/engine.ts';import { setCliExitVerdict } from '../core/cli-force-exit.ts';import { OperationError } from '../core/ops/contract.ts';import { cliRenderContext, toAgentError } from '../core/agent-output.ts';-import { readExportPage, readExportWithdrawals, type ExportWithdrawal } from '../core/export-snapshot.ts';+import { readExportPages, readExportWithdrawals, type ExportSnapshot, type ExportWithdrawal } from '../core/export-snapshot.ts';import { RestoreTargetError, restoreFilePath } from '../core/restore-target.ts';import { privatePagesFilterFragment, readHonorAliases } from '../core/search/private-visibility.ts';import { resolveSourceId } from '../core/source-resolver.ts';@@ -121,7 +121,8 @@ interface Checkout { root: string; head: HeadTree; mode: Awaited<ReturnType<typeconst refuse = (message: string, suggestion: string) => new OperationError('storage_error', message, suggestion, GIT_GAP_DOCS);-export async function censusGitGap(engine: BrainEngine, sourceId: string, opts: { sample: number; deadlineAt?: number; collect?: (page: GitGapPage) => void }): Promise<GitGapCensus> {+/** `collect` receives each classified page, with the export snapshot the census compared for a database_ahead or git_ahead page. */+export async function censusGitGap(engine: BrainEngine, sourceId: string, opts: { sample: number; deadlineAt?: number; collect?: GitGapCollect }): Promise<GitGapCensus> {await resolveSourceId(engine, sourceId, undefined, { skipLocalSignals: true });const [source] = await engine.executeRaw<{ local_path: string | null; incarnation: string }>('SELECT local_path, incarnation::text AS incarnation FROM sources WHERE id=$1', [sourceId]);const root = source?.local_path && existsSync(source.local_path) ? source.local_path : null;@@ -158,8 +159,10 @@ export async function censusGitGap(engine: BrainEngine, sourceId: string, opts:return census;}+export type GitGapCollect = (page: GitGapPage, snapshot?: ExportSnapshot) => void;+interface BatchContext { checkout: Checkout | null; policy: { mirror: boolean; databaseOnlySource: boolean; dbOnly: (slug: string) => boolean; ignored: Set<string> }; incarnation: string; sample: number;- collect?: (page: GitGapPage) => void }+ collect?: GitGapCollect }const collected = (key: Key, path: string | null, state: PageState, extra: Partial<GitGapPage> = {}): GitGapPage =>({ id: key.id, slug: key.slug, revision: key.revision, path, state, segmented: key.segmented === true, ...extra });@@ -240,9 +243,10 @@ async function compareBatch(tx: BrainEngine, compare: Array<{ key: Key; path: stif (!blobs) throw refuse(`Git could not read committed files of the checkout at ${checkout.root}.`,`Nothing changed. Check the checkout with git -C ${checkout.root} fsck, then rerun gbrain sources git-gap ${sourceId}.`);const withdrawals: ExportWithdrawal[] = await readExportWithdrawals(tx, sourceId);- const differs: Array<{ key: Key; rel: string; blob: TreeBlob; worktree: WorktreeState; fields: string[] }> = [];- for (const item of compare) {- const snapshot = await readExportPage(tx, item.key, withdrawals);+ const differs: Array<{ key: Key; rel: string; blob: TreeBlob; worktree: WorktreeState; fields: string[]; snapshot: ExportSnapshot }> = [];+ const snapshots = await readExportPages(tx, compare.map(item => item.key), withdrawals);+ for (const [i, item] of compare.entries()) {+ const snapshot = snapshots[i]!;const compared: ExportFileComparison = item.key.segmented ? await compareSegmentedExportBlob(tx, snapshot, withdrawals, item.blob, checkout.head.format): compareInlineExportFile(blobs.get(item.blob.oid)!, scannerSourcePath(checkout.root, item.path, checkout.mode), snapshot);if (compared.state === 'equal') { census.states.equal++; ctx.collect?.(collected(item.key, item.rel, 'equal', { worktree: item.worktree, headBlob: item.blob.oid })); continue; }@@ -252,7 +256,7 @@ async function compareBatch(tx: BrainEngine, compare: Array<{ key: Key; path: stnote(census, ctx.sample, { slug: item.key.slug, path: item.rel, state: 'unparseable', worktree: item.worktree });continue;}- differs.push({ ...item, fields: compared.paths });+ differs.push({ ...item, fields: compared.paths, snapshot });}if (!differs.length) return;const records = await readSyncImportProvenances(tx, sourceId, ctx.incarnation, differs.map(item => Number(item.key.id)));@@ -265,7 +269,7 @@ async function compareBatch(tx: BrainEngine, compare: Array<{ key: Key; path: stif (!imported.length) census.database_ahead_without_import_record++;const databaseAlsoMoved = gitMoved && record?.revision != null && String(record.revision) !== String(item.key.revision);if (databaseAlsoMoved) census.git_ahead_database_also_moved++;- ctx.collect?.(collected(item.key, item.rel, state, { worktree: item.worktree, headBlob: item.blob.oid, databaseAlsoMoved }));+ ctx.collect?.(collected(item.key, item.rel, state, { worktree: item.worktree, headBlob: item.blob.oid, databaseAlsoMoved }), item.snapshot);note(census, ctx.sample, { slug: item.key.slug, path: item.rel, state, fields: item.fields, basis: imported.length ? 'import_record' : 'no_import_record', worktree: item.worktree });}}diff --git a/src/core/export-snapshot.ts b/src/core/export-snapshot.tsindex 7e1d905626..4628b50050 100644--- a/src/core/export-snapshot.ts+++ b/src/core/export-snapshot.ts@@ -1,5 +1,5 @@import type { BrainEngine } from './engine.ts';-import { overlayCanonicalBodies } from './page-state/snapshot.ts';+import { overlayCanonicalBodies, overlayCanonicalBodiesBatch } from './page-state/snapshot.ts';import type { PageWithdrawal } from './page-state/types.ts';import { rowToPage } from './utils.ts';import { EXPORT_PAYLOAD_LIMIT } from './export-stage.ts';@@ -43,6 +43,31 @@ export async function readExportPage(tx: BrainEngine, key: { id: string; source_export type ExportSnapshot = Awaited<ReturnType<typeof readExportPage>>;+/**+ * readExportPage of every key, in key order, in two statements however many keys there are: the rows with their tags,+ * and one fence normalization for the pages whose overlay needs it (#7617: a per-page read made a large source's+ * conversion plan one round trip per page). Each snapshot is exactly the one readExportPage returns for its key.+ */+export async function readExportPages(tx: BrainEngine, keys: ReadonlyArray<{ id: string; source_id: string; slug: string }>, withdrawals: ExportWithdrawal[]): Promise<ExportSnapshot[]> {+ if (!keys.length) return [];+ const rows = await tx.executeRaw<Record<string, unknown>>(`SELECT p.*,+ COALESCE((SELECT jsonb_agg(t.tag ORDER BY t.tag) FROM tags t WHERE t.page_id=p.id), '[]'::jsonb) AS export_tags+ FROM unnest($1::bigint[], $2::text[], $3::text[]) AS k(id, source_id, slug)+ JOIN pages p ON p.id=k.id AND p.source_id=k.source_id AND p.slug=k.slug AND p.deleted_at IS NULL`,+ [keys.map(key => key.id), keys.map(key => key.source_id), keys.map(key => key.slug)]);+ const byId = new Map(rows.map(row => [String(row.id), row]));+ const snapshots = keys.map(key => {+ const row = byId.get(String(key.id));+ if (!row) throw new Error('Export snapshot unexpectedly lost a selected page.');+ return { page: rowToPage(row), tags: row.export_tags as string[] };+ });+ const fenced = snapshots.filter(({ page }) => !isSegmentedPage(page) && (page.compiled_truth.includes('gbrain:facts:') || page.timeline.includes('gbrain:facts:')));+ const overlaid = await overlayCanonicalBodiesBatch(tx.executeRaw.bind(tx),+ fenced.map(({ page }) => ({ body: page.compiled_truth, timeline: page.timeline, withdrawals: applicableWithdrawals(page.slug, withdrawals) })));+ fenced.forEach(({ page }, i) => Object.assign(page, overlaid[i]));+ return snapshots;+}+/*** The frontmatter and header fields of a page's exported file. #5966: the slug stamp rule lives in* slugStampedFrontmatter (markdown.ts), shared with every render; an escaped file always carries its slug.diff --git a/src/core/page-state/snapshot.ts b/src/core/page-state/snapshot.tsindex 2f6ce8aa1c..71eaa32b42 100644--- a/src/core/page-state/snapshot.ts+++ b/src/core/page-state/snapshot.ts@@ -30,6 +30,10 @@ export async function resolveGlobalPurges(query: ReadQuery, sourceId: string, mareturn rows.map(r => typeof r.w === 'string' ? JSON.parse(r.w) as PageWithdrawal : r.w);}+/** The ledger fingerprint text of `text` (a SQL expression): each line lowercased, whitespace runs collapsed to one space. */+const normalizedLines = (text: string) => `(SELECT string_agg(regexp_replace(lower(line),'[[:space:]]+',' ','g'),chr(10) ORDER BY ord)+ FROM unnest(string_to_array(${text},chr(10))) WITH ORDINALITY AS lines(line,ord))`;+/*** Normalize filesystem bytes with the exact ledger fingerprint rules before comparing them. `global` names the* snapshot's source and '*' purge marker, resolved against this text's fence rows (not the stored page's).@@ -39,17 +43,34 @@ export async function overlayCanonicalBodies(query: ReadQuery, body: string, timconst pending = global?.marker && global.marker.count > 0 ? global : undefined;if (!withdrawals.length && !pending) return { compiled_truth: body, timeline };if (!body.includes('gbrain:facts:begin') && !timeline.includes('gbrain:facts:begin')) return { compiled_truth: body, timeline };- const [normalized] = await query<{ body: string; timeline: string }>(`SELECT- (SELECT string_agg(regexp_replace(lower(line),'[[:space:]]+',' ','g'),chr(10) ORDER BY ord)- FROM unnest(string_to_array($1::text,chr(10))) WITH ORDINALITY AS lines(line,ord)) AS body,- (SELECT string_agg(regexp_replace(lower(line),'[[:space:]]+',' ','g'),chr(10) ORDER BY ord)- FROM unnest(string_to_array($2::text,chr(10))) WITH ORDINALITY AS lines(line,ord)) AS timeline`, [body, timeline]);+ const [normalized] = await query<{ body: string; timeline: string }>(`SELECT ${normalizedLines('$1::text')} AS body, ${normalizedLines('$2::text')} AS timeline`, [body, timeline]);const resolved = pending ? await resolveGlobalPurges(query, pending.sourceId, pending.marker, [normalized.body ?? '', normalized.timeline ?? '']) : [];const all = resolved.length ? [...withdrawals, ...resolved] : withdrawals;return { compiled_truth: overlayWithdrawalBody(body, normalized.body ?? '', all),timeline: overlayWithdrawalBody(timeline, normalized.timeline ?? '', all) };}+/**+ * overlayCanonicalBodies (without a '*' purge marker) of many texts: the texts that need it are normalized in one+ * statement, so a batch of pages costs one round trip instead of one per page. Results are in `items` order.+ */+export async function overlayCanonicalBodiesBatch(query: ReadQuery, items: ReadonlyArray<{ body: string; timeline: string; withdrawals: PageWithdrawal[] }>):+ Promise<Array<{ compiled_truth: string; timeline: string }>> {+ const out = items.map(item => ({ compiled_truth: item.body, timeline: item.timeline }));+ const pending = items.flatMap((item, i) => item.withdrawals.length+ && (item.body.includes('gbrain:facts:begin') || item.timeline.includes('gbrain:facts:begin')) ? [i] : []);+ if (!pending.length) return out;+ const rows = await query<{ i: number; body: string | null; timeline: string | null }>(`SELECT t.i::int AS i, ${normalizedLines('t.body')} AS body,+ ${normalizedLines('t.timeline')} AS timeline FROM unnest($1::text[], $2::text[]) WITH ORDINALITY AS t(body, timeline, i)`,+ [pending.map(i => items[i]!.body), pending.map(i => items[i]!.timeline)]);+ for (const row of rows) {+ const at = pending[Number(row.i) - 1]!, item = items[at]!;+ out[at] = { compiled_truth: overlayWithdrawalBody(item.body, row.body ?? '', item.withdrawals),+ timeline: overlayWithdrawalBody(item.timeline, row.timeline ?? '', item.withdrawals) };+ }+ return out;+}+/*** One MVCC statement binds content, tags, identity and withdrawals to one revision.* The normalized body and timeline fingerprints feed only the withdrawal overlay,diff --git a/src/core/persistence/git-conversion.ts b/src/core/persistence/git-conversion.tsindex aa0d5b12cf..e62316dbac 100644--- a/src/core/persistence/git-conversion.ts+++ b/src/core/persistence/git-conversion.ts@@ -39,7 +39,7 @@ import { resolveSourceId } from '../source-resolver.ts';import { opError, type OperationError } from '../ops/contract.ts';import { trustedCliRequired } from '../ops/op-fix.ts';import { currentSubmissionAuthority } from '../minions/submission-authority.ts';-import { inlineExportMarkdown, readExportPage, readExportWithdrawals } from '../export-snapshot.ts';+import { inlineExportMarkdown, readExportPage, readExportPages, readExportWithdrawals, type ExportSnapshot } from '../export-snapshot.ts';import { scannerSlugRootMode, scannerSourcePath } from '../write-through.ts';import { censusGitGap, type GitGapCensus, type GitGapPage } from '../../commands/sources-git-gap.ts';import { compareInlineExportFile, ignoredPaths } from './git-gap.ts';@@ -58,6 +58,8 @@ import { previewChangedError, previewHash } from './preview-approval.ts';import { sha256 } from './digest.ts';export const CONVERSION_GROUP_MAX = 200;+/** Adoption candidates whose export view the plan reads per read-only transaction (the census's batch size). */+export const ADOPTION_BATCH = 256;export const CONVERSION_OP = 'git-canonical-conversion';const DOCS = 'docs/designs/GIT_CANONICAL_MEMORY.md#conversion';const SAMPLE = 20;@@ -132,7 +134,12 @@ export async function planGitConversion(engine: BrainEngine, sourceId: string):const context = await resolveManagedSyncContext(engine, { sourceId, noPull: true, dryRun: true });const pages: GitGapPage[] = [];const equal = new Set<string>();- const census = await censusGitGap(engine, sourceId, { sample: 0, collect: page => { if (page.state !== 'equal') pages.push(page); else if (page.path) equal.add(page.path); } });+ const exported = new ExportChecks(context.root, await scannerSlugRootMode(engine, sourceId, context.root));+ const census = await censusGitGap(engine, sourceId, { sample: 0, collect: (page, snapshot) => {+ if (page.state === 'equal') { if (page.path) equal.add(page.path); return; }+ pages.push(page);+ if (snapshot && page.state === 'database_ahead' && page.path && !page.segmented) exported.add(page.id, page.path, snapshot);+ } });if (!census.checkout || realpathSync(census.checkout) !== context.root) {throw opError('source_changed', `Source ${sourceId}'s registered checkout is not its owner's root, so nothing was planned.`,`Check the owner with gbrain sources writer status --source ${sourceId} --json, then preview again.`);@@ -186,7 +193,7 @@ export async function planGitConversion(engine: BrainEngine, sourceId: string):default: continue;}}- await planAdoption(engine, sourceId, root, candidates, preview, adopt, discard);+ await planAdoption(engine, sourceId, root, candidates, exported, preview, adopt, discard);const parked = await readParked(engine, sourceId);for (const item of parked) for (const code of item.codes) preview.parked_effects.by_code[`${item.kind}:${code}`] = (preview.parked_effects.by_code[`${item.kind}:${code}`] ?? 0) + 1;preview.parked_effects.count = parked.length;@@ -217,27 +224,50 @@ function holdItem(hold: GitHoldRecord): HoldItem {return { path: hold.path, code: hold.code, side, observed_at: hold.observed_at, action };}+/**+ * What the adoption plan needs of a page's export view, kept instead of the view: the sha256 of its exported bytes, and+ * whether those bytes read back equal under the census contract.+ */+class ExportChecks {+ private readonly checks = new Map<string, { digest: string; readsBack: boolean }>();+ constructor(private readonly root: string, private readonly mode: Awaited<ReturnType<typeof scannerSlugRootMode>>) {}+ add(id: string, path: string, snapshot: ExportSnapshot): void {+ const bytes = Buffer.from(inlineExportMarkdown(snapshot));+ this.checks.set(id, { digest: sha256(bytes), readsBack: compareInlineExportFile(bytes, scannerSourcePath(this.root, join(this.root, path), this.mode), snapshot).state === 'equal' });+ }+ get(id: string) { return this.checks.get(id); }+}+/*** Each adoption candidate's export bytes must read back equal under the census contract, its target must not be ignored,- * and the working tree must hold HEAD's bytes, the export's, or nothing: an uncommitted edit is never overwritten.+ * and the working tree must hold HEAD's bytes, the export's, or nothing: an uncommitted edit is never overwritten. A+ * database_ahead page's export view is the one the census compared; the other candidates are read ADOPTION_BATCH at a+ * time in read-only transactions (#7617).*/-async function planAdoption(engine: BrainEngine, sourceId: string, root: string, candidates: Array<{ page: GitGapPage; state: AdoptState }>, preview: GitConversionPreview,- adopt: AdoptItem[], discard: (category: DiscardCategory, path: string) => void): Promise<void> {+async function planAdoption(engine: BrainEngine, sourceId: string, root: string, candidates: Array<{ page: GitGapPage; state: AdoptState }>, exported: ExportChecks,+ preview: GitConversionPreview, adopt: AdoptItem[], discard: (category: DiscardCategory, path: string) => void): Promise<void> {if (!candidates.length) return;const ignored = await ignoredPaths(root, candidates.map(({ page }) => page.path!));- const mode = await scannerSlugRootMode(engine, sourceId, root);- const withdrawals = await readExportWithdrawals(engine, sourceId);+ const unread = candidates.map(({ page }) => page).filter(page => ignored && !ignored.has(page.path!) && !page.segmented && !exported.get(page.id));+ const withdrawals = unread.length ? await readExportWithdrawals(engine, sourceId) : [];+ for (let start = 0; start < unread.length; start += ADOPTION_BATCH) {+ const batch = unread.slice(start, start + ADOPTION_BATCH);+ const snapshots = await engine.transaction(async tx => {+ await tx.executeRaw('SET TRANSACTION READ ONLY');+ return readExportPages(tx, batch.map(page => ({ ...page, source_id: sourceId })), withdrawals);+ });+ batch.forEach((page, i) => exported.add(page.id, page.path!, snapshots[i]!));+ }for (const { page, state } of candidates) {const path = page.path!;// Fail-closed fallback: an ignored path with no file is already declared git_ignored by the census, and Git commits a// tracked file whatever its ignore rules say, so this fires only when reading the ignore rules fails (ignored is null).if (!ignored || ignored.has(path)) { discard('ignored_target', path); continue; }if (page.segmented) { discard('segmented_page', path); continue; }- const snapshot = await readExportPage(engine, { ...page, source_id: sourceId }, withdrawals);- const bytes = Buffer.from(inlineExportMarkdown(snapshot));- if (compareInlineExportFile(bytes, scannerSourcePath(root, join(root, path), mode), snapshot).state !== 'equal') { discard('export_mismatch', path); continue; }+ const check = exported.get(page.id)!;+ if (!check.readsBack) { discard('export_mismatch', path); continue; }const disk = diskBytes(join(root, path));- if (page.worktree !== 'clean' && page.worktree !== 'absent' && !disk?.equals(bytes)) { discard('uncommitted_edit', path); continue; }+ if (page.worktree !== 'clean' && page.worktree !== 'absent' && !(disk && sha256(disk) === check.digest)) { discard('uncommitted_edit', path); continue; }adopt.push({ id: page.id, source_id: sourceId, slug: page.slug, revision: page.revision, path, state, disk: disk ? sha256(disk) : null });tally(preview.adopt, state, path);}diff --git a/test/export-snapshot.test.ts b/test/export-snapshot.test.tsindex 1257293865..2b79735fd7 100644--- a/test/export-snapshot.test.ts+++ b/test/export-snapshot.test.ts@@ -8,7 +8,7 @@ import type { BrainEngine } from '../src/core/engine.ts';import { runExport } from '../src/commands/export.ts';import { parseFactsFence, renderFactsTable } from '../src/core/facts-fence.ts';import { importFromContent } from '../src/core/import-file.ts';-import { readExportPage, readExportWithdrawals } from '../src/core/export-snapshot.ts';+import { inlineExportMarkdown, readExportPage, readExportPages, readExportWithdrawals } from '../src/core/export-snapshot.ts';import { overlayCanonicalBodies } from '../src/core/page-state/snapshot.ts';import { requirePostgresTestDatabase, testBackends } from './helpers/test-backends.ts';@@ -82,6 +82,80 @@ for (const backend of testBackends()) describe(`coherent export: ${backend}`, ()}});}, 60000);+ test('#7617: readExportPages gives every page exactly the snapshot readExportPage gives, in two statements', async () => {+ await engine.executeRaw('DELETE FROM pages');+ await engine.executeRaw("DELETE FROM fact_withdrawals WHERE source_id='default'");+ await engine.executeRaw("DELETE FROM fact_purges WHERE source_id='default'");+ const fence = (...claims: string[]) => renderFactsTable(claims.map((claim, i) => ({ rowNum: i + 1, claim, kind: 'fact', confidence: 1,+ visibility: 'world', notability: 'medium', active: true, context: '' })));+ const withdraw = (claim: string, subject = '*') => engine.executeRaw(+ "INSERT INTO fact_withdrawals(source_id,visibility,fact_hash,subject) VALUES ('default','world',gbrain_fact_fingerprint($1),$2)", [claim, subject]);+ await withdraw('Withdrawn everywhere claim');+ await withdraw('Withdrawn here claim', 'shapes/fence-subject');+ await withdraw('MiXeD\t synthetic claim');+ await engine.executeRaw("INSERT INTO fact_purges(source_id,visibility,fact_hash,subject) VALUES ('default','world',gbrain_fact_fingerprint($1),'shapes/fence-purged')", ['Purged claim']);+ const shapes: Array<[string, string, string, Record<string, unknown>]> = [+ ['shapes/plain', 'A plain body.', '', {}],+ ['shapes/empty', '', '', {}],+ ['shapes/timeline', 'Body.', '- 2026-01-01: An event.\n- 2026-01-02: Another event.', {}],+ ['shapes/crlf', 'Line one\r\nLine two\r\n', 'Timeline\r\n', {}],+ ['shapes/fence-star', 'Lead.\n\n' + fence('Withdrawn everywhere claim', 'A kept claim'), '', {}],+ ['shapes/fence-subject', fence('Withdrawn here claim'), fence('Withdrawn here claim', 'Withdrawn everywhere claim'), {}],+ ['shapes/fence-other-subject', fence('Withdrawn here claim'), '', {}],+ ['shapes/fence-purged', fence('Purged claim', 'A kept claim'), '', {}],+ ['shapes/fence-kept', fence('A kept claim'), '', {}],+ ['shapes/fence-mixed-case', fence('MIXED synthetic claim'), '', {}],+ ['shapes/fence-timeline-only', 'No fence in the body.', fence('Withdrawn everywhere claim'), {}],+ ['shapes/marker-mention', 'Mentions gbrain:facts: without a fence.', '', {}],+ ['shapes/fence-end-only', '<!--- gbrain:facts:end -->', '', {}],+ ['shapes/fence-unclosed', fence('Withdrawn everywhere claim') + '\n<!--- gbrain:facts:begin -->', '', {}],+ ['shapes/fence-broken', '<!--- gbrain:facts:begin -->\nbroken\n<!--- gbrain:facts:end -->', '', {}],+ ['shapes/frontmatter', 'Body.', '', { aliases: ['alpha-example', 'beta-example'], nested: { key: 'value', list: [1, 2, { deep: true }] },+ empty: '', nothing: null, date: '2026-01-01', unicode: 'ünïcödé — ✓', colon: 'a: b', quote: "it's \"quoted\"", multiline: 'one\ntwo' }],+ ['shapes/slug-in-frontmatter', 'Body.', '', { slug: 'shapes/another-slug', title: 'A frontmatter title' }],+ ['shapes/tagged', 'Tagged body.', '', { tags: ['zeta', 'alpha'] }],+ ];+ for (const [slug, compiled, timeline, frontmatter] of shapes) {+ await engine.executeRaw("INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter) VALUES ('default',$1,'note',$1,$2,$3,$4::text::jsonb)",+ [slug, compiled, timeline, JSON.stringify(frontmatter)]);+ }+ await engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter)+ SELECT 'default','bulk/p'||n,'note','Bulk '||n,CASE WHEN n%5=0 THEN $1 ELSE 'Bulk body '||n END,'','{}'::jsonb FROM generate_series(1,300) n`,+ [fence('Withdrawn everywhere claim', 'A kept claim')]);+ await engine.executeRaw(`INSERT INTO tags(page_id,tag) SELECT id,t FROM pages, unnest(ARRAY['zeta','alpha','mid']) t WHERE slug IN ('shapes/tagged','bulk/p7')`);+ await engine.executeRaw("INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter,deleted_at) VALUES ('default','shapes/deleted','note','Deleted','Gone.','','{}'::jsonb,now())");+ // Reverse id order: the result follows the keys, not the table.+ const keys = await engine.executeRaw<{ id: string; source_id: string; slug: string }>("SELECT id::text,source_id,slug FROM pages WHERE source_id='default' AND deleted_at IS NULL ORDER BY id DESC");+ expect(keys.length).toBe(shapes.length + 300);+ await engine.transaction(async tx => {+ await tx.executeRaw('SET TRANSACTION READ ONLY');+ const withdrawals = await readExportWithdrawals(tx, 'default');+ const real = tx.executeRaw;+ let statements = 0;+ tx.executeRaw = async function (query, params, options) { statements++; return real.call(this, query, params, options) as never; };+ let batch;+ try { batch = await readExportPages(tx, keys, withdrawals); } finally { tx.executeRaw = real; }+ expect(statements).toBe(2);+ expect(batch.length).toBe(keys.length);+ let overlaid = 0;+ for (const [i, key] of keys.entries()) {+ const single = await readExportPage(tx, key, withdrawals);+ expect(batch[i]!.page.slug).toBe(key.slug);+ expect(JSON.stringify(batch[i])).toBe(JSON.stringify(single));+ expect(inlineExportMarkdown(batch[i]!)).toBe(inlineExportMarkdown(single));+ if ([single.page.compiled_truth, single.page.timeline].some(text => parseFactsFence(text).facts.some(fact => fact.active === false))) overlaid++;+ }+ // The overlay really ran: withdrawn rows read inactive in the fenced shapes and every fifth bulk page.+ expect(overlaid).toBeGreaterThan(60);+ expect(parseFactsFence(batch[keys.findIndex(key => key.slug === 'shapes/fence-subject')]!.page.compiled_truth).facts[0].active).toBe(false);+ expect(parseFactsFence(batch[keys.findIndex(key => key.slug === 'shapes/fence-other-subject')]!.page.compiled_truth).facts[0].active).toBe(true);+ expect(parseFactsFence(batch[keys.findIndex(key => key.slug === 'shapes/fence-purged')]!.page.compiled_truth).facts.map(fact => fact.claim)).toEqual(['A kept claim']);+ expect(batch[keys.findIndex(key => key.slug === 'shapes/tagged')]!.tags).toEqual(['alpha', 'mid', 'zeta']);+ expect(await readExportPages(tx, [], withdrawals)).toEqual([]);+ const [deleted] = await tx.executeRaw<{ id: string; source_id: string; slug: string }>("SELECT id::text,source_id,slug FROM pages WHERE slug='shapes/deleted'");+ await expect(readExportPages(tx, [keys[0]!, deleted!], withdrawals)).rejects.toThrow('Export snapshot unexpectedly lost a selected page.');+ });+ }, 60000);test('source/page/tag/raw changes after snapshot do not tear the export', async () => {await engine.executeRaw('DELETE FROM pages');await engine.executeRaw("INSERT INTO sources (id,name) VALUES ('snapshot-source','Snapshot') ON CONFLICT (id) DO UPDATE SET archived=false");diff --git a/test/git-canonical-conversion-g8a.test.ts b/test/git-canonical-conversion-g8a.test.tsindex 8dba644763..2a93e4f4cd 100644--- a/test/git-canonical-conversion-g8a.test.ts+++ b/test/git-canonical-conversion-g8a.test.ts@@ -23,7 +23,8 @@ import { localHostId } from '../src/core/persistence/identity.ts';import { acquireWorktree } from '../src/core/persistence/ownership.ts';import { performManagedSync } from '../src/core/persistence/sync-run.ts';import { resolveManagedSyncContext } from '../src/core/persistence/sync-discovery.ts';-import { applyGitConversion, applyMixedConversion, CONVERSION_GROUP_MAX, planGitConversion, planMixedConversion } from '../src/core/persistence/git-conversion.ts';+import { ADOPTION_BATCH, applyGitConversion, applyMixedConversion, CONVERSION_GROUP_MAX, planGitConversion, planMixedConversion } from '../src/core/persistence/git-conversion.ts';+import { renderFactsTable } from '../src/core/facts-fence.ts';import { withSubmissionAuthority } from '../src/core/minions/submission-authority.ts';import { censusGitGap } from '../src/commands/sources-git-gap.ts';import { runSourcesCanonical } from '../src/commands/sources-canonical.ts';@@ -97,6 +98,23 @@ const bulkSetup = ({ root }: { root: string }) => {git(root, 'add', '-A'); git(root, 'commit', '-qm', 'bulk');};+const WIDE = 300, FENCED_EVERY = 5;+const factsFence = (claim: string) => renderFactsTable([{ rowNum: 1, claim, kind: 'fact', confidence: 1, visibility: 'world', notability: 'medium', active: true, context: '' }]);+const wideSetup = ({ root }: { root: string }) => {+ mkdirSync(join(root, 'wide'), { recursive: true }); git(root, 'init', '-q');+ git(root, 'config', 'user.name', 'Example Writer'); git(root, 'config', 'user.email', '[email protected]');+ for (let i = 0; i < WIDE; i++) {+ writeFileSync(join(root, `wide/p${String(i).padStart(3, '0')}.md`), note(`wide/p${String(i).padStart(3, '0')}`, i % FENCED_EVERY ? `Committed body ${i}.` : `Committed body ${i}.\n\n${factsFence(`Wide claim ${i}`)}`));+ }+ git(root, 'add', '-A'); git(root, 'commit', '-qm', 'wide');+};+/**+ * Statements that read a page's export view: the per-page read (readExportPage), the batch read (readExportPages) and the+ * fence normalization of either (overlayCanonicalBodies, overlayCanonicalBodiesBatch). Page snapshot reads by background+ * projection work normalize stored columns, never a parameter, so they do not match.+ */+const EXPORT_READ = /export_page AS MATERIALIZED|AS k\(id, source_id, slug\)|string_to_array\(\$1::text|AS t\(body, timeline, i\)/;+/** The page loses its sync import record, as a page imported before records were written has none. */async function dropImportRecord(engine: BrainEngine, slug: string): Promise<void> {const [row] = await engine.executeRaw<{ id: string }>("SELECT id::text FROM pages WHERE source_id='default' AND slug=$1", [slug]);@@ -274,6 +292,30 @@ for (const backend of testBackends()) describe(`[${backend}] G8a: sources canoniexpect(git(root, 'status', '--porcelain')).toBe('');}, { databaseUrl, setup: bulkSetup }), 300_000);+ test('#7617: the plan reads export views a batch at a time, never one page per round trip', () => managedBrain(async brain => {+ const { engine } = brain;+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');+ await engine.transaction(tx => withCoordinatedWrite(tx, ['default'], async () => {+ await tx.executeRaw(`UPDATE pages SET compiled_truth=compiled_truth || ' Edited in the database.', knowledge_revision=gen_random_uuid(), revision_write_request_id=NULL+ WHERE source_id='default' AND slug LIKE 'wide/%' AND right(slug,1)::int % 2 = 1`);+ await tx.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter)+ SELECT 'default','dbonly/p'||lpad(n::text,3,'0'),'concept','dbonly/p'||lpad(n::text,3,'0'),CASE WHEN n%$2::int=0 THEN 'Only in the database '||n||'.'||chr(10)||chr(10)||$3 ELSE 'Only in the database '||n||'.' END,'',+ jsonb_build_object('type','concept','title','dbonly/p'||lpad(n::text,3,'0'))+ FROM generate_series(0,$1::int - 1) n`, [WIDE, FENCED_EVERY, factsFence('A database-only claim')]);+ // An unrelated withdrawal: every fenced page's export view runs the overlay, and none changes.+ await tx.executeRaw("INSERT INTO fact_withdrawals(source_id,visibility,fact_hash) VALUES ('default','world',gbrain_fact_fingerprint('An unrelated withdrawn claim'))");+ }, TEST_WRITE_ATTRIBUTION));+ const real = engine.executeRaw;+ let reads = 0;+ engine.executeRaw = async function (query, params, options) { if (EXPORT_READ.test(query)) reads++; return real.call(this, query, params, options) as never; };+ let preview;+ try { ({ preview } = await planGitConversion(engine, 'default')); } finally { engine.executeRaw = real; }+ expect(preview).toMatchObject({ pages: 2 * WIDE, equal: WIDE / 2, adopt: { count: WIDE / 2 + WIDE, by: { database_ahead: WIDE / 2, undeclared_database_only: WIDE } }, discard: { count: 0 } });+ // The census reads each batch of compared pages in one statement plus one fence normalization; adoption reuses the+ // census's view of a database_ahead page and reads the database-only pages ADOPTION_BATCH at a time the same way.+ expect(reads).toBeLessThanOrEqual(2 * (Math.ceil(2 * WIDE / 256) + Math.ceil(WIDE / ADOPTION_BATCH)));+ }, { databaseUrl, setup: wideSetup }), 300_000);+test('work that lands after the strict-equivalent pass leaves the source strict and reports incomplete', () => managedBrain(async brain => {const { engine, root } = brain;expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');