Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(persistence): the census reads a withheld page's file at its withheld path, so a crash between the withhold stamp and its commit recovers (#7618)

garrytan8f51689ee62c
diff --git a/docs/designs/GIT_CANONICAL_MEMORY.md b/docs/designs/GIT_CANONICAL_MEMORY.md
index 3ef3312b8d..737c40911c 100644
--- a/docs/designs/GIT_CANONICAL_MEMORY.md
+++ b/docs/designs/GIT_CANONICAL_MEMORY.md
@@ -90,7 +90,7 @@ G8a ships it as `gbrain sources canonical <source-id> --to git|mixed [--dry-run
| parked effects | Retried once, or, for targets Git ignores, the page is stamped `git_ignored` (declared) in the same transaction that marks each parked target `declared: git_ignored`; the target stays as a record and the next preview no longer lists it. |
| discard | Refused: `git_conversion_blocked` names each category (both sides moved, a both-sides drift hold, an unknown-side drift hold whose row differs from the committed file, a database edit Git deleted, an uncommitted working-tree edit that differs from HEAD and the export, an ignored target, a target whose ignore status Git cannot read (`ignore_status_unknown`: beyond a symlink, outside the checkout, a failed check; only that path is refused), unparseable, unsafe path, segmented page, export mismatch, or a row a strict pass would overwrite unplanned). There is no override. |
-`--apply --expect <hash>` re-plans and refuses `preview_changed` on any difference, adopts group by group (a crash leaves the source mixed and the rerun preview lists only what is left), withholds group by group, rechecks what a strict pass would overwrite, clears the holds only after that recheck passes, settles the parked effects, runs the full strict-equivalent pass, then sets `canonical: git` in the transaction that records the conversion. Writes can land between that pass and the flag, so it runs one genuine strict pass and previews again: the conversion is complete only when nothing is left; otherwise it reports `git_conversion_incomplete` with what remains and the source stays strict, because Git wins. Where durability applies the apply pushes its door commits itself, after the groups and once at the end; a push that does not land leaves them local, the receipt says so, and the next page write's push-only effect sends them. The apply prints a progress line per phase and every 10 seconds on stderr, and SIGTERM, SIGINT or SIGHUP prints a partial report (phase, groups committed, items scanned, canonical state); `GBRAIN_MEMORY_TRACE` adds a memory trace. `--to mixed` removes the flag and changes nothing else; Git keeps every door commit.
+`--apply --expect <hash>` re-plans and refuses `preview_changed` on any difference, adopts group by group (a crash leaves the source mixed and the rerun preview lists only what is left), withholds group by group, settles the parked effects, rechecks what a strict pass would overwrite, clears the holds only after that recheck passes, runs the full strict-equivalent pass, then sets `canonical: git` in the transaction that records the conversion. Writes can land between that pass and the flag, so it runs one genuine strict pass and previews again: the conversion is complete only when nothing is left; otherwise it reports `git_conversion_incomplete` with what remains and the source stays strict, because Git wins. Where durability applies the apply pushes its door commits itself, after the groups and once at the end; a push that does not land leaves them local, the receipt says so, and the next page write's push-only effect sends them. The apply prints a progress line per phase and every 10 seconds on stderr, and SIGTERM, SIGINT or SIGHUP prints a partial report (phase, groups committed, items scanned, canonical state); `GBRAIN_MEMORY_TRACE` adds a memory trace. `--to mixed` removes the flag and changes nothing else; Git keeps every door commit.
## The git-gap census
diff --git a/src/commands/sources-git-gap.ts b/src/commands/sources-git-gap.ts
index f710d95e7c..c141846898 100644
--- a/src/commands/sources-git-gap.ts
+++ b/src/commands/sources-git-gap.ts
@@ -177,7 +177,9 @@ function note(census: GitGapCensus, limit: number, item: GitGapSample): void {
async function censusBatch(tx: BrainEngine, batch: Key[], census: GitGapCensus, ctx: BatchContext): Promise<void> {
const { checkout } = ctx;
const target = (key: Key): string | null => {
- try { return restoreFilePath(checkout!.root, key.slug, key.source_path, Number(key.source_path_bytes), checkout!.mode); }
+ // A withheld page's file is looked for where its private transition removed it, as page-prepare writes it there again.
+ const recorded = key.source_path || key.withheld_source_path;
+ try { return restoreFilePath(checkout!.root, key.slug, recorded, key.source_path ? Number(key.source_path_bytes) : Buffer.byteLength(recorded ?? ''), checkout!.mode); }
catch (error) { if (error instanceof RestoreTargetError) return null; throw error; }
};
const rows = batch.map(key => {
diff --git a/src/core/persistence/fault-points.ts b/src/core/persistence/fault-points.ts
index 00ac492334..5d8c1bcdc6 100644
--- a/src/core/persistence/fault-points.ts
+++ b/src/core/persistence/fault-points.ts
@@ -105,7 +105,7 @@ export type FaultPoint = `publication:${PublicationBoundary}` | `effect:${Effect
// GBRA-161 PR 1b: as a managed generation member (an overwrite or retirement after its activation committed) is prepared.
| 'generation:member'
// G8a (git-conversion.ts): after a conversion group's door commit, and after the strict-equivalent pass, before the flag is set.
- | 'conversion:group_committed' | 'conversion:withhold_committed' | 'conversion:before_flip';
+ | 'conversion:group_committed' | 'conversion:withhold_stamped' | 'conversion:withhold_committed' | 'conversion:before_flip';
/**
* `signal` (#6278, `consumer:preparing` only): the preparation's abort signal, so
* a stalling hook can model both the reporter's cases: a preparer that honours
diff --git a/src/core/persistence/git-conversion.ts b/src/core/persistence/git-conversion.ts
index 1d9ff3c090..71948c3a0e 100644
--- a/src/core/persistence/git-conversion.ts
+++ b/src/core/persistence/git-conversion.ts
@@ -14,8 +14,9 @@
* reads back equal to it (the census contract) keeps that file in HEAD and is settled: a strict import changes nothing.
* - withhold: a private, alias-private or lineage-private page whose row still records a file that is absent from HEAD
* or differs from it. The apply takes G7a's
- * private transition: a committed file's removal goes through the door, then the row withholds its origin
- * (withheld_source_path), so the strict pass leaves the page live; with no file at HEAD, only the row changes. Any
+ * private transition: the row withholds its origin (withheld_source_path), then a committed file's removal goes
+ * through the door, so no sync reads the removal as a deletion and the page stays live; with no file at HEAD, only
+ * the row changes. Any
* other declared page with a committed file is a discard (git_overwrites_database), as Git's import would replace it.
* - git_wins: Git's verdict replaces the row and loses nothing (Git moved and the database did not; Git deleted a page
* the database had not edited; a held deletion; a held file_drift whose side is unknown and whose row already equals
@@ -142,7 +143,7 @@ export interface GitConversionPreview {
/** `committed_equal`: declared pages whose committed file reads back equal to the page, so the file stays in HEAD. */
declared: Counted<string> & { action: 'none'; committed_equal: number };
git_wins: Counted<GitWinsReason> & { action: 'index_committed_blob' };
- /** G7a's private transition for a privacy-class page that still records a file: its file's removal is committed, then the row withholds its origin. */
+ /** G7a's private transition for a privacy-class page that still records a file: the row withholds its origin, then its file's removal is committed. */
withhold: Counted<WithholdKind> & { action: 'commit_removal_and_withhold'; groups: number };
/** `unplaced`: holds at a path no category above placed (a declared page, no live page), which the index check judges as the strict pass will see them. */
holds: Counted<string> & { action: 'clear'; unplaced: { count: number; by: Record<string, number> } };
@@ -614,19 +615,17 @@ async function adoptGroups(engine: BrainEngine, plan: GitConversionPlan, preview
* branch), which runs only inside a publication: group by group under the writer lock, a transaction stamps the class and
* moves `source_path` to `withheld_source_path`, then the files' removal goes through the door. Stamped first, no sync
* (the strict pass or a mixed one run before a rerun) reads the removal as the deletion of a live page; a crash between
- * the stamp and the commit leaves the file in HEAD, which a mixed sync holds and the rerun preview withholds again. A page
- * whose file is already absent from HEAD only gets the stamp. A page whose revision or working tree moved since the
- * preview stops the apply (preview_changed) before anything is stamped.
+ * the stamp and the commit leaves the file in HEAD, which a mixed sync holds and the rerun preview (which reads a withheld
+ * page's file at withheld_source_path) withholds again. A page whose file is already absent from HEAD only gets the stamp,
+ * CONVERSION_GROUP_MAX at a time. The stamp updates only the planned revision: a page whose revision or working tree moved
+ * since the preview stops the apply (preview_changed) before its group is stamped; the groups before it stay withheld and
+ * the rerun lists only what is left.
*/
async function withholdGroups(engine: BrainEngine, plan: GitConversionPlan, previewCommand: string,
onGroup: (done: { pages: number; commits: number }) => void): Promise<GitConversionResult['withheld']> {
const { context } = plan;
const root = context.root;
const commits: string[] = [];
- const assertUnchanged = async (item: WithholdItem) => {
- const [live] = await engine.executeRaw<{ revision: string | null }>('SELECT knowledge_revision::text AS revision FROM pages WHERE id=$1::bigint AND deleted_at IS NULL', [item.id]);
- if (!live || live.revision !== item.revision) throw previewChangedError(plan.preview.preview_hash, previewCommand);
- };
const stamp = (items: WithholdItem[]) => engine.transaction(async tx => {
await declarePersistenceProtocol(tx);
for (const item of items) {
@@ -640,14 +639,14 @@ async function withholdGroups(engine: BrainEngine, plan: GitConversionPlan, prev
const group = committed.slice(start, start + CONVERSION_GROUP_MAX);
const lock = await acquireWorktree(context.binding, LOCK_WAIT_MS, undefined, engine, { yieldLanes: true });
if (!lock) throw opError('writer_busy', `The writer lock of source ${context.sourceId}'s checkout stayed busy for ${LOCK_WAIT_MS / 1000}s, so the removal of private pages' files was not committed.`,
- `Run ${previewCommand} again; the removals already committed read as file_absent, so the new plan holds only what is left.`, { retryable: true });
+ `Run ${previewCommand} again; the pages already withheld drop out of the plan, so the new plan holds only what is left.`, { retryable: true });
try {
for (const item of group) {
- await assertUnchanged(item);
const disk = diskBytes(join(root, item.path));
if ((disk ? sha256(disk) : null) !== item.disk) throw previewChangedError(plan.preview.preview_hash, previewCommand);
}
await stamp(group);
+ await faultPoint('conversion:withhold_stamped', { sourceId: context.sourceId, round: start / CONVERSION_GROUP_MAX + 1 });
for (const item of group) removeConfined(root, join(root, item.path));
const commit = await commitThroughDoor(root, group.map(item => ({ path: join(root, item.path), planned: null })), lock);
if (commit) commits.push(commit);
diff --git a/test/git-canonical-conversion-g8a.test.ts b/test/git-canonical-conversion-g8a.test.ts
index 471b61a31f..a0da231abc 100644
--- a/test/git-canonical-conversion-g8a.test.ts
+++ b/test/git-canonical-conversion-g8a.test.ts
@@ -560,7 +560,7 @@ for (const backend of testBackends()) describe(`[${backend}] G8a: sources canoni
}, { databaseUrl, setup }), 180_000);
/**
- * Value: protects=the withhold never removes a file or stamps a row that moved after the preview; fails_when=assertUnchanged
+ * Value: protects=the withhold never removes a file or stamps a row that moved after the preview; fails_when=the stamp's revision compare-and-set
* or the disk-hash check in withholdGroups is removed; why_new=preview_changed is otherwise tested only for a stale hash
* before any write; seam=none; no_claim=the writer_busy and git_target_unsafe branches of withholdGroups
*/
@@ -701,6 +701,41 @@ for (const backend of testBackends()) describe(`[${backend}] G8a: sources canoni
expect(await pageRow(engine, 'notes/private')).toMatchObject({ body: before.body, deleted: false, withheld: 'notes/private.md' });
}, { databaseUrl, setup }), 180_000);
+ /**
+ * Value: protects=a crash after the withhold stamp and before its removal commit loses nothing: a mixed sync keeps the
+ * page live and unchanged, and the rerun finds the file at the path the stamp withheld and removes it; fails_when=the
+ * census resolves a withheld page's file from its slug alone (a file at another path is then left in HEAD unplanned);
+ * why_new=the crash test crashes after the commit, at a slug path; seam=the conversion:withhold_stamped fault point;
+ * no_claim=a crash inside commitThroughDoor itself
+ */
+ test('#7618: a crash between the withhold stamp and its commit keeps the page through a mixed sync, and the rerun withholds the file at its recorded path', () => managedBrain(async brain => {
+ const { engine, root } = brain;
+ writeFileSync(join(root, 'notes/Private Note.md'), PRIVATE('notes/private-note', 'A private note.'));
+ git(root, 'add', 'notes/Private Note.md'); git(root, 'commit', '-qm', 'a private note');
+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');
+ const [{ slug }] = await engine.executeRaw<{ slug: string }>("SELECT slug FROM pages WHERE source_id='default' AND source_path='notes/Private Note.md'");
+ await databaseOnlyEdit(engine, slug, 'A private note, edited in the database.');
+ const before = await pageRow(engine, slug), base = git(root, 'rev-parse', 'HEAD');
+ const { preview } = await planGitConversion(engine, 'default');
+ expect(preview).toMatchObject({ withhold: { count: 1, by: { file_committed: 1 }, sample: ['notes/Private Note.md'] }, discard: { count: 0 } });
+ installFaultHook(point => { if (point === 'conversion:withhold_stamped') throw new Error('crash before the withhold commit'); });
+ try { await expect(applyGitConversion(engine, 'default', preview.preview_hash, PROCESSING)).rejects.toThrow('crash before the withhold commit'); }
+ finally { installFaultHook(undefined); }
+ expect(git(root, 'rev-parse', 'HEAD')).toBe(base);
+ expect(atHead(root, 'notes/Private Note.md')).toBe('notes/Private Note.md');
+ const stamped = { body: before.body, deleted: false, source_path: null, withheld: 'notes/Private Note.md', reason: 'private' };
+ expect(await pageRow(engine, slug)).toEqual(stamped);
+ await performManagedSync(engine, OPTS);
+ expect(await pageRow(engine, slug)).toEqual(stamped);
+ const rerun = (await planGitConversion(engine, 'default')).preview;
+ expect(rerun).toMatchObject({ withhold: { count: 1, by: { file_committed: 1 }, sample: ['notes/Private Note.md'] }, discard: { count: 0 } });
+ expect(await applyGitConversion(engine, 'default', rerun.preview_hash, PROCESSING)).toMatchObject({ status: 'converted', withheld: { pages: 1 }, after: { remaining: 0 } });
+ expect(conversionCommits(root, base)).toEqual([{ subject: expect.any(String), paths: ['notes/Private Note.md'] }]);
+ expect(atHead(root, 'notes/Private Note.md')).toBe('');
+ expect(await pageRow(engine, slug)).toEqual(stamped);
+ expect(await strict(engine)).toBe(true);
+ }, { databaseUrl, setup }), 180_000);
+
/**
* Value: protects=the text preview and receipt name the withheld page and a push that did not land, with the command that
* pushes it; fails_when=describeGitPreview or describeResult drops the Withhold line, the withheld count or the push-pending