Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

perf(persistence): batch export reads in the canonical conversion plan (#7617)

garrytan9bb96fe7917d
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index b6dfaeb406..17dc71b898 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -5,7 +5,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
The #6188 fence check and fence repair live in [fence repair](core-fence-repair.md). Managed connector ingestion, connector item holds and the no-op kernel (checkpoint validation and the fault seams: [request indexes and fault seams](core-persistence-request-indexes.md)). Also the persistence verb types and errors, database-write and manual links, write attribution and loop-fact retirement; the managed purge of deleted pages is under [memory purge](memory-purge.md).
- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies`, so on owner start) touches only interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command), which the refusal hints point at. Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts`; while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.
-- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).
+- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).
- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).
- `src/core/persistence/{types,params,preconditions,verb-errors}.ts` — public mutation receipt states, error reasons and revision/idempotency parameter contracts. Receipt serialization explicitly selects public fields; queued content and execution credentials stay internal. `frozenVerbWriteError` uses MEMORY_VERBS v1's existing error enum with additive `write_request`/`write_error`; `page_body_segmented` is a write error code, so a verb refused by a segmented target keeps it as `write_error` and canonical code under the frozen `invalid_params` (caller class, not retryable) on both the receipt and the pre-admission path (`runMemoryWrite`). `remember` reads a segmented entity page up to PAGE_MAX_BYTES back whole (`materializedPage`) for its inferred-target checks and fence write; over it, a named or inferred entity refuses before admission with `segmentedRememberRefusal` (fix: remember with `infer_entity: false`), never a silent `NO_ENTITY` (`test/large-pages/segmented-remember-refusal.test.ts`); `committedVerbOutcome` refuses pending or failed receipts. `OperationError` and the thin MCP client preserve this metadata, including when a timeout races an already-received receipt. The registry-safe parameter/schema leaf never imports operation values. Pinned by `test/persistence-write-contracts.test.ts`, `test/mcp-client-hardening.test.ts` and a real HTTP envelope in `test/mcp-client.test.ts`.
diff --git a/src/commands/sources-git-gap.ts b/src/commands/sources-git-gap.ts
index e6fdbb5980..33b3ac37ec 100644
--- a/src/commands/sources-git-gap.ts
+++ b/src/commands/sources-git-gap.ts
@@ -17,7 +17,7 @@ import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { OperationError } from '../core/ops/contract.ts';
import { cliRenderContext, toAgentError } from '../core/agent-output.ts';
-import { readExportPage, readExportWithdrawals, type ExportWithdrawal } from '../core/export-snapshot.ts';
+import { readExportPages, readExportWithdrawals, type ExportSnapshot, type ExportWithdrawal } from '../core/export-snapshot.ts';
import { RestoreTargetError, restoreFilePath } from '../core/restore-target.ts';
import { privatePagesFilterFragment, readHonorAliases } from '../core/search/private-visibility.ts';
import { resolveSourceId } from '../core/source-resolver.ts';
@@ -121,7 +121,8 @@ interface Checkout { root: string; head: HeadTree; mode: Awaited<ReturnType<type
const refuse = (message: string, suggestion: string) => new OperationError('storage_error', message, suggestion, GIT_GAP_DOCS);
-export async function censusGitGap(engine: BrainEngine, sourceId: string, opts: { sample: number; deadlineAt?: number; collect?: (page: GitGapPage) => void }): Promise<GitGapCensus> {
+/** `collect` receives each classified page, with the export snapshot the census compared for a database_ahead or git_ahead page. */
+export async function censusGitGap(engine: BrainEngine, sourceId: string, opts: { sample: number; deadlineAt?: number; collect?: GitGapCollect }): Promise<GitGapCensus> {
await resolveSourceId(engine, sourceId, undefined, { skipLocalSignals: true });
const [source] = await engine.executeRaw<{ local_path: string | null; incarnation: string }>('SELECT local_path, incarnation::text AS incarnation FROM sources WHERE id=$1', [sourceId]);
const root = source?.local_path && existsSync(source.local_path) ? source.local_path : null;
@@ -158,8 +159,10 @@ export async function censusGitGap(engine: BrainEngine, sourceId: string, opts:
return census;
}
+export type GitGapCollect = (page: GitGapPage, snapshot?: ExportSnapshot) => void;
+
interface BatchContext { checkout: Checkout | null; policy: { mirror: boolean; databaseOnlySource: boolean; dbOnly: (slug: string) => boolean; ignored: Set<string> }; incarnation: string; sample: number;
- collect?: (page: GitGapPage) => void }
+ collect?: GitGapCollect }
const collected = (key: Key, path: string | null, state: PageState, extra: Partial<GitGapPage> = {}): GitGapPage =>
({ id: key.id, slug: key.slug, revision: key.revision, path, state, segmented: key.segmented === true, ...extra });
@@ -240,9 +243,10 @@ async function compareBatch(tx: BrainEngine, compare: Array<{ key: Key; path: st
if (!blobs) throw refuse(`Git could not read committed files of the checkout at ${checkout.root}.`,
`Nothing changed. Check the checkout with git -C ${checkout.root} fsck, then rerun gbrain sources git-gap ${sourceId}.`);
const withdrawals: ExportWithdrawal[] = await readExportWithdrawals(tx, sourceId);
- const differs: Array<{ key: Key; rel: string; blob: TreeBlob; worktree: WorktreeState; fields: string[] }> = [];
- for (const item of compare) {
- const snapshot = await readExportPage(tx, item.key, withdrawals);
+ const differs: Array<{ key: Key; rel: string; blob: TreeBlob; worktree: WorktreeState; fields: string[]; snapshot: ExportSnapshot }> = [];
+ const snapshots = await readExportPages(tx, compare.map(item => item.key), withdrawals);
+ for (const [i, item] of compare.entries()) {
+ const snapshot = snapshots[i]!;
const compared: ExportFileComparison = item.key.segmented ? await compareSegmentedExportBlob(tx, snapshot, withdrawals, item.blob, checkout.head.format)
: compareInlineExportFile(blobs.get(item.blob.oid)!, scannerSourcePath(checkout.root, item.path, checkout.mode), snapshot);
if (compared.state === 'equal') { census.states.equal++; ctx.collect?.(collected(item.key, item.rel, 'equal', { worktree: item.worktree, headBlob: item.blob.oid })); continue; }
@@ -252,7 +256,7 @@ async function compareBatch(tx: BrainEngine, compare: Array<{ key: Key; path: st
note(census, ctx.sample, { slug: item.key.slug, path: item.rel, state: 'unparseable', worktree: item.worktree });
continue;
}
- differs.push({ ...item, fields: compared.paths });
+ differs.push({ ...item, fields: compared.paths, snapshot });
}
if (!differs.length) return;
const records = await readSyncImportProvenances(tx, sourceId, ctx.incarnation, differs.map(item => Number(item.key.id)));
@@ -265,7 +269,7 @@ async function compareBatch(tx: BrainEngine, compare: Array<{ key: Key; path: st
if (!imported.length) census.database_ahead_without_import_record++;
const databaseAlsoMoved = gitMoved && record?.revision != null && String(record.revision) !== String(item.key.revision);
if (databaseAlsoMoved) census.git_ahead_database_also_moved++;
- ctx.collect?.(collected(item.key, item.rel, state, { worktree: item.worktree, headBlob: item.blob.oid, databaseAlsoMoved }));
+ ctx.collect?.(collected(item.key, item.rel, state, { worktree: item.worktree, headBlob: item.blob.oid, databaseAlsoMoved }), item.snapshot);
note(census, ctx.sample, { slug: item.key.slug, path: item.rel, state, fields: item.fields, basis: imported.length ? 'import_record' : 'no_import_record', worktree: item.worktree });
}
}
diff --git a/src/core/export-snapshot.ts b/src/core/export-snapshot.ts
index 7e1d905626..4628b50050 100644
--- a/src/core/export-snapshot.ts
+++ b/src/core/export-snapshot.ts
@@ -1,5 +1,5 @@
import type { BrainEngine } from './engine.ts';
-import { overlayCanonicalBodies } from './page-state/snapshot.ts';
+import { overlayCanonicalBodies, overlayCanonicalBodiesBatch } from './page-state/snapshot.ts';
import type { PageWithdrawal } from './page-state/types.ts';
import { rowToPage } from './utils.ts';
import { EXPORT_PAYLOAD_LIMIT } from './export-stage.ts';
@@ -43,6 +43,31 @@ export async function readExportPage(tx: BrainEngine, key: { id: string; source_
export type ExportSnapshot = Awaited<ReturnType<typeof readExportPage>>;
+/**
+ * readExportPage of every key, in key order, in two statements however many keys there are: the rows with their tags,
+ * and one fence normalization for the pages whose overlay needs it (#7617: a per-page read made a large source's
+ * conversion plan one round trip per page). Each snapshot is exactly the one readExportPage returns for its key.
+ */
+export async function readExportPages(tx: BrainEngine, keys: ReadonlyArray<{ id: string; source_id: string; slug: string }>, withdrawals: ExportWithdrawal[]): Promise<ExportSnapshot[]> {
+ if (!keys.length) return [];
+ const rows = await tx.executeRaw<Record<string, unknown>>(`SELECT p.*,
+ COALESCE((SELECT jsonb_agg(t.tag ORDER BY t.tag) FROM tags t WHERE t.page_id=p.id), '[]'::jsonb) AS export_tags
+ FROM unnest($1::bigint[], $2::text[], $3::text[]) AS k(id, source_id, slug)
+ JOIN pages p ON p.id=k.id AND p.source_id=k.source_id AND p.slug=k.slug AND p.deleted_at IS NULL`,
+ [keys.map(key => key.id), keys.map(key => key.source_id), keys.map(key => key.slug)]);
+ const byId = new Map(rows.map(row => [String(row.id), row]));
+ const snapshots = keys.map(key => {
+ const row = byId.get(String(key.id));
+ if (!row) throw new Error('Export snapshot unexpectedly lost a selected page.');
+ return { page: rowToPage(row), tags: row.export_tags as string[] };
+ });
+ const fenced = snapshots.filter(({ page }) => !isSegmentedPage(page) && (page.compiled_truth.includes('gbrain:facts:') || page.timeline.includes('gbrain:facts:')));
+ const overlaid = await overlayCanonicalBodiesBatch(tx.executeRaw.bind(tx),
+ fenced.map(({ page }) => ({ body: page.compiled_truth, timeline: page.timeline, withdrawals: applicableWithdrawals(page.slug, withdrawals) })));
+ fenced.forEach(({ page }, i) => Object.assign(page, overlaid[i]));
+ return snapshots;
+}
+
/**
* The frontmatter and header fields of a page's exported file. #5966: the slug stamp rule lives in
* slugStampedFrontmatter (markdown.ts), shared with every render; an escaped file always carries its slug.
diff --git a/src/core/page-state/snapshot.ts b/src/core/page-state/snapshot.ts
index 2f6ce8aa1c..71eaa32b42 100644
--- a/src/core/page-state/snapshot.ts
+++ b/src/core/page-state/snapshot.ts
@@ -30,6 +30,10 @@ export async function resolveGlobalPurges(query: ReadQuery, sourceId: string, ma
return rows.map(r => typeof r.w === 'string' ? JSON.parse(r.w) as PageWithdrawal : r.w);
}
+/** The ledger fingerprint text of `text` (a SQL expression): each line lowercased, whitespace runs collapsed to one space. */
+const normalizedLines = (text: string) => `(SELECT string_agg(regexp_replace(lower(line),'[[:space:]]+',' ','g'),chr(10) ORDER BY ord)
+ FROM unnest(string_to_array(${text},chr(10))) WITH ORDINALITY AS lines(line,ord))`;
+
/**
* Normalize filesystem bytes with the exact ledger fingerprint rules before comparing them. `global` names the
* snapshot's source and '*' purge marker, resolved against this text's fence rows (not the stored page's).
@@ -39,17 +43,34 @@ export async function overlayCanonicalBodies(query: ReadQuery, body: string, tim
const pending = global?.marker && global.marker.count > 0 ? global : undefined;
if (!withdrawals.length && !pending) return { compiled_truth: body, timeline };
if (!body.includes('gbrain:facts:begin') && !timeline.includes('gbrain:facts:begin')) return { compiled_truth: body, timeline };
- const [normalized] = await query<{ body: string; timeline: string }>(`SELECT
- (SELECT string_agg(regexp_replace(lower(line),'[[:space:]]+',' ','g'),chr(10) ORDER BY ord)
- FROM unnest(string_to_array($1::text,chr(10))) WITH ORDINALITY AS lines(line,ord)) AS body,
- (SELECT string_agg(regexp_replace(lower(line),'[[:space:]]+',' ','g'),chr(10) ORDER BY ord)
- FROM unnest(string_to_array($2::text,chr(10))) WITH ORDINALITY AS lines(line,ord)) AS timeline`, [body, timeline]);
+ const [normalized] = await query<{ body: string; timeline: string }>(`SELECT ${normalizedLines('$1::text')} AS body, ${normalizedLines('$2::text')} AS timeline`, [body, timeline]);
const resolved = pending ? await resolveGlobalPurges(query, pending.sourceId, pending.marker, [normalized.body ?? '', normalized.timeline ?? '']) : [];
const all = resolved.length ? [...withdrawals, ...resolved] : withdrawals;
return { compiled_truth: overlayWithdrawalBody(body, normalized.body ?? '', all),
timeline: overlayWithdrawalBody(timeline, normalized.timeline ?? '', all) };
}
+/**
+ * overlayCanonicalBodies (without a '*' purge marker) of many texts: the texts that need it are normalized in one
+ * statement, so a batch of pages costs one round trip instead of one per page. Results are in `items` order.
+ */
+export async function overlayCanonicalBodiesBatch(query: ReadQuery, items: ReadonlyArray<{ body: string; timeline: string; withdrawals: PageWithdrawal[] }>):
+ Promise<Array<{ compiled_truth: string; timeline: string }>> {
+ const out = items.map(item => ({ compiled_truth: item.body, timeline: item.timeline }));
+ const pending = items.flatMap((item, i) => item.withdrawals.length
+ && (item.body.includes('gbrain:facts:begin') || item.timeline.includes('gbrain:facts:begin')) ? [i] : []);
+ if (!pending.length) return out;
+ const rows = await query<{ i: number; body: string | null; timeline: string | null }>(`SELECT t.i::int AS i, ${normalizedLines('t.body')} AS body,
+ ${normalizedLines('t.timeline')} AS timeline FROM unnest($1::text[], $2::text[]) WITH ORDINALITY AS t(body, timeline, i)`,
+ [pending.map(i => items[i]!.body), pending.map(i => items[i]!.timeline)]);
+ for (const row of rows) {
+ const at = pending[Number(row.i) - 1]!, item = items[at]!;
+ out[at] = { compiled_truth: overlayWithdrawalBody(item.body, row.body ?? '', item.withdrawals),
+ timeline: overlayWithdrawalBody(item.timeline, row.timeline ?? '', item.withdrawals) };
+ }
+ return out;
+}
+
/**
* One MVCC statement binds content, tags, identity and withdrawals to one revision.
* The normalized body and timeline fingerprints feed only the withdrawal overlay,
diff --git a/src/core/persistence/git-conversion.ts b/src/core/persistence/git-conversion.ts
index aa0d5b12cf..e62316dbac 100644
--- a/src/core/persistence/git-conversion.ts
+++ b/src/core/persistence/git-conversion.ts
@@ -39,7 +39,7 @@ import { resolveSourceId } from '../source-resolver.ts';
import { opError, type OperationError } from '../ops/contract.ts';
import { trustedCliRequired } from '../ops/op-fix.ts';
import { currentSubmissionAuthority } from '../minions/submission-authority.ts';
-import { inlineExportMarkdown, readExportPage, readExportWithdrawals } from '../export-snapshot.ts';
+import { inlineExportMarkdown, readExportPage, readExportPages, readExportWithdrawals, type ExportSnapshot } from '../export-snapshot.ts';
import { scannerSlugRootMode, scannerSourcePath } from '../write-through.ts';
import { censusGitGap, type GitGapCensus, type GitGapPage } from '../../commands/sources-git-gap.ts';
import { compareInlineExportFile, ignoredPaths } from './git-gap.ts';
@@ -58,6 +58,8 @@ import { previewChangedError, previewHash } from './preview-approval.ts';
import { sha256 } from './digest.ts';
export const CONVERSION_GROUP_MAX = 200;
+/** Adoption candidates whose export view the plan reads per read-only transaction (the census's batch size). */
+export const ADOPTION_BATCH = 256;
export const CONVERSION_OP = 'git-canonical-conversion';
const DOCS = 'docs/designs/GIT_CANONICAL_MEMORY.md#conversion';
const SAMPLE = 20;
@@ -132,7 +134,12 @@ export async function planGitConversion(engine: BrainEngine, sourceId: string):
const context = await resolveManagedSyncContext(engine, { sourceId, noPull: true, dryRun: true });
const pages: GitGapPage[] = [];
const equal = new Set<string>();
- const census = await censusGitGap(engine, sourceId, { sample: 0, collect: page => { if (page.state !== 'equal') pages.push(page); else if (page.path) equal.add(page.path); } });
+ const exported = new ExportChecks(context.root, await scannerSlugRootMode(engine, sourceId, context.root));
+ const census = await censusGitGap(engine, sourceId, { sample: 0, collect: (page, snapshot) => {
+ if (page.state === 'equal') { if (page.path) equal.add(page.path); return; }
+ pages.push(page);
+ if (snapshot && page.state === 'database_ahead' && page.path && !page.segmented) exported.add(page.id, page.path, snapshot);
+ } });
if (!census.checkout || realpathSync(census.checkout) !== context.root) {
throw opError('source_changed', `Source ${sourceId}'s registered checkout is not its owner's root, so nothing was planned.`,
`Check the owner with gbrain sources writer status --source ${sourceId} --json, then preview again.`);
@@ -186,7 +193,7 @@ export async function planGitConversion(engine: BrainEngine, sourceId: string):
default: continue;
}
}
- await planAdoption(engine, sourceId, root, candidates, preview, adopt, discard);
+ await planAdoption(engine, sourceId, root, candidates, exported, preview, adopt, discard);
const parked = await readParked(engine, sourceId);
for (const item of parked) for (const code of item.codes) preview.parked_effects.by_code[`${item.kind}:${code}`] = (preview.parked_effects.by_code[`${item.kind}:${code}`] ?? 0) + 1;
preview.parked_effects.count = parked.length;
@@ -217,27 +224,50 @@ function holdItem(hold: GitHoldRecord): HoldItem {
return { path: hold.path, code: hold.code, side, observed_at: hold.observed_at, action };
}
+/**
+ * What the adoption plan needs of a page's export view, kept instead of the view: the sha256 of its exported bytes, and
+ * whether those bytes read back equal under the census contract.
+ */
+class ExportChecks {
+ private readonly checks = new Map<string, { digest: string; readsBack: boolean }>();
+ constructor(private readonly root: string, private readonly mode: Awaited<ReturnType<typeof scannerSlugRootMode>>) {}
+ add(id: string, path: string, snapshot: ExportSnapshot): void {
+ const bytes = Buffer.from(inlineExportMarkdown(snapshot));
+ this.checks.set(id, { digest: sha256(bytes), readsBack: compareInlineExportFile(bytes, scannerSourcePath(this.root, join(this.root, path), this.mode), snapshot).state === 'equal' });
+ }
+ get(id: string) { return this.checks.get(id); }
+}
+
/**
* Each adoption candidate's export bytes must read back equal under the census contract, its target must not be ignored,
- * and the working tree must hold HEAD's bytes, the export's, or nothing: an uncommitted edit is never overwritten.
+ * and the working tree must hold HEAD's bytes, the export's, or nothing: an uncommitted edit is never overwritten. A
+ * database_ahead page's export view is the one the census compared; the other candidates are read ADOPTION_BATCH at a
+ * time in read-only transactions (#7617).
*/
-async function planAdoption(engine: BrainEngine, sourceId: string, root: string, candidates: Array<{ page: GitGapPage; state: AdoptState }>, preview: GitConversionPreview,
- adopt: AdoptItem[], discard: (category: DiscardCategory, path: string) => void): Promise<void> {
+async function planAdoption(engine: BrainEngine, sourceId: string, root: string, candidates: Array<{ page: GitGapPage; state: AdoptState }>, exported: ExportChecks,
+ preview: GitConversionPreview, adopt: AdoptItem[], discard: (category: DiscardCategory, path: string) => void): Promise<void> {
if (!candidates.length) return;
const ignored = await ignoredPaths(root, candidates.map(({ page }) => page.path!));
- const mode = await scannerSlugRootMode(engine, sourceId, root);
- const withdrawals = await readExportWithdrawals(engine, sourceId);
+ const unread = candidates.map(({ page }) => page).filter(page => ignored && !ignored.has(page.path!) && !page.segmented && !exported.get(page.id));
+ const withdrawals = unread.length ? await readExportWithdrawals(engine, sourceId) : [];
+ for (let start = 0; start < unread.length; start += ADOPTION_BATCH) {
+ const batch = unread.slice(start, start + ADOPTION_BATCH);
+ const snapshots = await engine.transaction(async tx => {
+ await tx.executeRaw('SET TRANSACTION READ ONLY');
+ return readExportPages(tx, batch.map(page => ({ ...page, source_id: sourceId })), withdrawals);
+ });
+ batch.forEach((page, i) => exported.add(page.id, page.path!, snapshots[i]!));
+ }
for (const { page, state } of candidates) {
const path = page.path!;
// Fail-closed fallback: an ignored path with no file is already declared git_ignored by the census, and Git commits a
// tracked file whatever its ignore rules say, so this fires only when reading the ignore rules fails (ignored is null).
if (!ignored || ignored.has(path)) { discard('ignored_target', path); continue; }
if (page.segmented) { discard('segmented_page', path); continue; }
- const snapshot = await readExportPage(engine, { ...page, source_id: sourceId }, withdrawals);
- const bytes = Buffer.from(inlineExportMarkdown(snapshot));
- if (compareInlineExportFile(bytes, scannerSourcePath(root, join(root, path), mode), snapshot).state !== 'equal') { discard('export_mismatch', path); continue; }
+ const check = exported.get(page.id)!;
+ if (!check.readsBack) { discard('export_mismatch', path); continue; }
const disk = diskBytes(join(root, path));
- if (page.worktree !== 'clean' && page.worktree !== 'absent' && !disk?.equals(bytes)) { discard('uncommitted_edit', path); continue; }
+ if (page.worktree !== 'clean' && page.worktree !== 'absent' && !(disk && sha256(disk) === check.digest)) { discard('uncommitted_edit', path); continue; }
adopt.push({ id: page.id, source_id: sourceId, slug: page.slug, revision: page.revision, path, state, disk: disk ? sha256(disk) : null });
tally(preview.adopt, state, path);
}
diff --git a/test/export-snapshot.test.ts b/test/export-snapshot.test.ts
index 1257293865..2b79735fd7 100644
--- a/test/export-snapshot.test.ts
+++ b/test/export-snapshot.test.ts
@@ -8,7 +8,7 @@ import type { BrainEngine } from '../src/core/engine.ts';
import { runExport } from '../src/commands/export.ts';
import { parseFactsFence, renderFactsTable } from '../src/core/facts-fence.ts';
import { importFromContent } from '../src/core/import-file.ts';
-import { readExportPage, readExportWithdrawals } from '../src/core/export-snapshot.ts';
+import { inlineExportMarkdown, readExportPage, readExportPages, readExportWithdrawals } from '../src/core/export-snapshot.ts';
import { overlayCanonicalBodies } from '../src/core/page-state/snapshot.ts';
import { requirePostgresTestDatabase, testBackends } from './helpers/test-backends.ts';
@@ -82,6 +82,80 @@ for (const backend of testBackends()) describe(`coherent export: ${backend}`, ()
}
});
}, 60000);
+ test('#7617: readExportPages gives every page exactly the snapshot readExportPage gives, in two statements', async () => {
+ await engine.executeRaw('DELETE FROM pages');
+ await engine.executeRaw("DELETE FROM fact_withdrawals WHERE source_id='default'");
+ await engine.executeRaw("DELETE FROM fact_purges WHERE source_id='default'");
+ const fence = (...claims: string[]) => renderFactsTable(claims.map((claim, i) => ({ rowNum: i + 1, claim, kind: 'fact', confidence: 1,
+ visibility: 'world', notability: 'medium', active: true, context: '' })));
+ const withdraw = (claim: string, subject = '*') => engine.executeRaw(
+ "INSERT INTO fact_withdrawals(source_id,visibility,fact_hash,subject) VALUES ('default','world',gbrain_fact_fingerprint($1),$2)", [claim, subject]);
+ await withdraw('Withdrawn everywhere claim');
+ await withdraw('Withdrawn here claim', 'shapes/fence-subject');
+ await withdraw('MiXeD\t synthetic claim');
+ await engine.executeRaw("INSERT INTO fact_purges(source_id,visibility,fact_hash,subject) VALUES ('default','world',gbrain_fact_fingerprint($1),'shapes/fence-purged')", ['Purged claim']);
+ const shapes: Array<[string, string, string, Record<string, unknown>]> = [
+ ['shapes/plain', 'A plain body.', '', {}],
+ ['shapes/empty', '', '', {}],
+ ['shapes/timeline', 'Body.', '- 2026-01-01: An event.\n- 2026-01-02: Another event.', {}],
+ ['shapes/crlf', 'Line one\r\nLine two\r\n', 'Timeline\r\n', {}],
+ ['shapes/fence-star', 'Lead.\n\n' + fence('Withdrawn everywhere claim', 'A kept claim'), '', {}],
+ ['shapes/fence-subject', fence('Withdrawn here claim'), fence('Withdrawn here claim', 'Withdrawn everywhere claim'), {}],
+ ['shapes/fence-other-subject', fence('Withdrawn here claim'), '', {}],
+ ['shapes/fence-purged', fence('Purged claim', 'A kept claim'), '', {}],
+ ['shapes/fence-kept', fence('A kept claim'), '', {}],
+ ['shapes/fence-mixed-case', fence('MIXED synthetic claim'), '', {}],
+ ['shapes/fence-timeline-only', 'No fence in the body.', fence('Withdrawn everywhere claim'), {}],
+ ['shapes/marker-mention', 'Mentions gbrain:facts: without a fence.', '', {}],
+ ['shapes/fence-end-only', '<!--- gbrain:facts:end -->', '', {}],
+ ['shapes/fence-unclosed', fence('Withdrawn everywhere claim') + '\n<!--- gbrain:facts:begin -->', '', {}],
+ ['shapes/fence-broken', '<!--- gbrain:facts:begin -->\nbroken\n<!--- gbrain:facts:end -->', '', {}],
+ ['shapes/frontmatter', 'Body.', '', { aliases: ['alpha-example', 'beta-example'], nested: { key: 'value', list: [1, 2, { deep: true }] },
+ empty: '', nothing: null, date: '2026-01-01', unicode: 'ünïcödé — ✓', colon: 'a: b', quote: "it's \"quoted\"", multiline: 'one\ntwo' }],
+ ['shapes/slug-in-frontmatter', 'Body.', '', { slug: 'shapes/another-slug', title: 'A frontmatter title' }],
+ ['shapes/tagged', 'Tagged body.', '', { tags: ['zeta', 'alpha'] }],
+ ];
+ for (const [slug, compiled, timeline, frontmatter] of shapes) {
+ await engine.executeRaw("INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter) VALUES ('default',$1,'note',$1,$2,$3,$4::text::jsonb)",
+ [slug, compiled, timeline, JSON.stringify(frontmatter)]);
+ }
+ await engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter)
+ SELECT 'default','bulk/p'||n,'note','Bulk '||n,CASE WHEN n%5=0 THEN $1 ELSE 'Bulk body '||n END,'','{}'::jsonb FROM generate_series(1,300) n`,
+ [fence('Withdrawn everywhere claim', 'A kept claim')]);
+ await engine.executeRaw(`INSERT INTO tags(page_id,tag) SELECT id,t FROM pages, unnest(ARRAY['zeta','alpha','mid']) t WHERE slug IN ('shapes/tagged','bulk/p7')`);
+ await engine.executeRaw("INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter,deleted_at) VALUES ('default','shapes/deleted','note','Deleted','Gone.','','{}'::jsonb,now())");
+ // Reverse id order: the result follows the keys, not the table.
+ const keys = await engine.executeRaw<{ id: string; source_id: string; slug: string }>("SELECT id::text,source_id,slug FROM pages WHERE source_id='default' AND deleted_at IS NULL ORDER BY id DESC");
+ expect(keys.length).toBe(shapes.length + 300);
+ await engine.transaction(async tx => {
+ await tx.executeRaw('SET TRANSACTION READ ONLY');
+ const withdrawals = await readExportWithdrawals(tx, 'default');
+ const real = tx.executeRaw;
+ let statements = 0;
+ tx.executeRaw = async function (query, params, options) { statements++; return real.call(this, query, params, options) as never; };
+ let batch;
+ try { batch = await readExportPages(tx, keys, withdrawals); } finally { tx.executeRaw = real; }
+ expect(statements).toBe(2);
+ expect(batch.length).toBe(keys.length);
+ let overlaid = 0;
+ for (const [i, key] of keys.entries()) {
+ const single = await readExportPage(tx, key, withdrawals);
+ expect(batch[i]!.page.slug).toBe(key.slug);
+ expect(JSON.stringify(batch[i])).toBe(JSON.stringify(single));
+ expect(inlineExportMarkdown(batch[i]!)).toBe(inlineExportMarkdown(single));
+ if ([single.page.compiled_truth, single.page.timeline].some(text => parseFactsFence(text).facts.some(fact => fact.active === false))) overlaid++;
+ }
+ // The overlay really ran: withdrawn rows read inactive in the fenced shapes and every fifth bulk page.
+ expect(overlaid).toBeGreaterThan(60);
+ expect(parseFactsFence(batch[keys.findIndex(key => key.slug === 'shapes/fence-subject')]!.page.compiled_truth).facts[0].active).toBe(false);
+ expect(parseFactsFence(batch[keys.findIndex(key => key.slug === 'shapes/fence-other-subject')]!.page.compiled_truth).facts[0].active).toBe(true);
+ expect(parseFactsFence(batch[keys.findIndex(key => key.slug === 'shapes/fence-purged')]!.page.compiled_truth).facts.map(fact => fact.claim)).toEqual(['A kept claim']);
+ expect(batch[keys.findIndex(key => key.slug === 'shapes/tagged')]!.tags).toEqual(['alpha', 'mid', 'zeta']);
+ expect(await readExportPages(tx, [], withdrawals)).toEqual([]);
+ const [deleted] = await tx.executeRaw<{ id: string; source_id: string; slug: string }>("SELECT id::text,source_id,slug FROM pages WHERE slug='shapes/deleted'");
+ await expect(readExportPages(tx, [keys[0]!, deleted!], withdrawals)).rejects.toThrow('Export snapshot unexpectedly lost a selected page.');
+ });
+ }, 60000);
test('source/page/tag/raw changes after snapshot do not tear the export', async () => {
await engine.executeRaw('DELETE FROM pages');
await engine.executeRaw("INSERT INTO sources (id,name) VALUES ('snapshot-source','Snapshot') ON CONFLICT (id) DO UPDATE SET archived=false");
diff --git a/test/git-canonical-conversion-g8a.test.ts b/test/git-canonical-conversion-g8a.test.ts
index 8dba644763..2a93e4f4cd 100644
--- a/test/git-canonical-conversion-g8a.test.ts
+++ b/test/git-canonical-conversion-g8a.test.ts
@@ -23,7 +23,8 @@ import { localHostId } from '../src/core/persistence/identity.ts';
import { acquireWorktree } from '../src/core/persistence/ownership.ts';
import { performManagedSync } from '../src/core/persistence/sync-run.ts';
import { resolveManagedSyncContext } from '../src/core/persistence/sync-discovery.ts';
-import { applyGitConversion, applyMixedConversion, CONVERSION_GROUP_MAX, planGitConversion, planMixedConversion } from '../src/core/persistence/git-conversion.ts';
+import { ADOPTION_BATCH, applyGitConversion, applyMixedConversion, CONVERSION_GROUP_MAX, planGitConversion, planMixedConversion } from '../src/core/persistence/git-conversion.ts';
+import { renderFactsTable } from '../src/core/facts-fence.ts';
import { withSubmissionAuthority } from '../src/core/minions/submission-authority.ts';
import { censusGitGap } from '../src/commands/sources-git-gap.ts';
import { runSourcesCanonical } from '../src/commands/sources-canonical.ts';
@@ -97,6 +98,23 @@ const bulkSetup = ({ root }: { root: string }) => {
git(root, 'add', '-A'); git(root, 'commit', '-qm', 'bulk');
};
+const WIDE = 300, FENCED_EVERY = 5;
+const factsFence = (claim: string) => renderFactsTable([{ rowNum: 1, claim, kind: 'fact', confidence: 1, visibility: 'world', notability: 'medium', active: true, context: '' }]);
+const wideSetup = ({ root }: { root: string }) => {
+ mkdirSync(join(root, 'wide'), { recursive: true }); git(root, 'init', '-q');
+ git(root, 'config', 'user.name', 'Example Writer'); git(root, 'config', 'user.email', '[email protected]');
+ for (let i = 0; i < WIDE; i++) {
+ writeFileSync(join(root, `wide/p${String(i).padStart(3, '0')}.md`), note(`wide/p${String(i).padStart(3, '0')}`, i % FENCED_EVERY ? `Committed body ${i}.` : `Committed body ${i}.\n\n${factsFence(`Wide claim ${i}`)}`));
+ }
+ git(root, 'add', '-A'); git(root, 'commit', '-qm', 'wide');
+};
+/**
+ * Statements that read a page's export view: the per-page read (readExportPage), the batch read (readExportPages) and the
+ * fence normalization of either (overlayCanonicalBodies, overlayCanonicalBodiesBatch). Page snapshot reads by background
+ * projection work normalize stored columns, never a parameter, so they do not match.
+ */
+const EXPORT_READ = /export_page AS MATERIALIZED|AS k\(id, source_id, slug\)|string_to_array\(\$1::text|AS t\(body, timeline, i\)/;
+
/** The page loses its sync import record, as a page imported before records were written has none. */
async function dropImportRecord(engine: BrainEngine, slug: string): Promise<void> {
const [row] = await engine.executeRaw<{ id: string }>("SELECT id::text FROM pages WHERE source_id='default' AND slug=$1", [slug]);
@@ -274,6 +292,30 @@ for (const backend of testBackends()) describe(`[${backend}] G8a: sources canoni
expect(git(root, 'status', '--porcelain')).toBe('');
}, { databaseUrl, setup: bulkSetup }), 300_000);
+ test('#7617: the plan reads export views a batch at a time, never one page per round trip', () => managedBrain(async brain => {
+ const { engine } = brain;
+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');
+ await engine.transaction(tx => withCoordinatedWrite(tx, ['default'], async () => {
+ await tx.executeRaw(`UPDATE pages SET compiled_truth=compiled_truth || ' Edited in the database.', knowledge_revision=gen_random_uuid(), revision_write_request_id=NULL
+ WHERE source_id='default' AND slug LIKE 'wide/%' AND right(slug,1)::int % 2 = 1`);
+ await tx.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth,timeline,frontmatter)
+ SELECT 'default','dbonly/p'||lpad(n::text,3,'0'),'concept','dbonly/p'||lpad(n::text,3,'0'),CASE WHEN n%$2::int=0 THEN 'Only in the database '||n||'.'||chr(10)||chr(10)||$3 ELSE 'Only in the database '||n||'.' END,'',
+ jsonb_build_object('type','concept','title','dbonly/p'||lpad(n::text,3,'0'))
+ FROM generate_series(0,$1::int - 1) n`, [WIDE, FENCED_EVERY, factsFence('A database-only claim')]);
+ // An unrelated withdrawal: every fenced page's export view runs the overlay, and none changes.
+ await tx.executeRaw("INSERT INTO fact_withdrawals(source_id,visibility,fact_hash) VALUES ('default','world',gbrain_fact_fingerprint('An unrelated withdrawn claim'))");
+ }, TEST_WRITE_ATTRIBUTION));
+ const real = engine.executeRaw;
+ let reads = 0;
+ engine.executeRaw = async function (query, params, options) { if (EXPORT_READ.test(query)) reads++; return real.call(this, query, params, options) as never; };
+ let preview;
+ try { ({ preview } = await planGitConversion(engine, 'default')); } finally { engine.executeRaw = real; }
+ expect(preview).toMatchObject({ pages: 2 * WIDE, equal: WIDE / 2, adopt: { count: WIDE / 2 + WIDE, by: { database_ahead: WIDE / 2, undeclared_database_only: WIDE } }, discard: { count: 0 } });
+ // The census reads each batch of compared pages in one statement plus one fence normalization; adoption reuses the
+ // census's view of a database_ahead page and reads the database-only pages ADOPTION_BATCH at a time the same way.
+ expect(reads).toBeLessThanOrEqual(2 * (Math.ceil(2 * WIDE / 256) + Math.ceil(WIDE / ADOPTION_BATCH)));
+ }, { databaseUrl, setup: wideSetup }), 300_000);
+
test('work that lands after the strict-equivalent pass leaves the source strict and reports incomplete', () => managedBrain(async brain => {
const { engine, root } = brain;
expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');