Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: owed sync after an aborted refresh runs once per member and names members it leaves behind

garrytan20b875a1ccaf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f23edc33e4..851fc4de89 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,9 +16,9 @@ identifiers and attribution are available in the pre-removal Git revision
### Itemized changes
-- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` ends every run (returned or refused, not `--dry-run`) with `syncAfterUnfinishedRefresh`. When the worktree's newest refresh row is `aborted`, it runs the bounded member sync for each member whose `last_commit` is not HEAD, after the run's locks are released. The run's `synced` lists those syncs, and a refusal's message names them.
+- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` ends every run (returned or refused, not `--dry-run`) with `syncAfterUnfinishedRefresh`. When the worktree's newest refresh row is `aborted`, it runs the bounded member sync for each member whose `last_commit` is not HEAD and that the run did not already sync, after the run's locks are released. The run's `synced` lists those syncs once each; a member left behind HEAD is named with its `gbrain sync --source <id> --no-pull` command in `next` (and `sync_blocked` when blocked) or in a refusal's message. A refusal of the run's own arguments, `refresh_in_progress` and an unexpected failure skip it.
- `spec/tla/refresh-barrier`: new `RefreshBarrierAbortLiveness.cfg` checks liveness with no scheduled sync and no forced expiry (`OutsideHelp = FALSE`). `OwedSyncBegin` models the new sync, and the guard mutant `abort-skips-owed-sync` removes it (`DeferredLeaves` fails; the code edit fails the new tests). `scripts/spec/mutants.ts` names the temporal property that fails by checking each one on its own.
-- Tests: `test/refresh-barrier-held-after-abort.test.ts` (PGLite, and Postgres when `DATABASE_URL` is set). It covers a refresh aborted by another writer's commit followed by a diverged retry, a refresh killed at fenced and then abandoned by `--resume`, and an owner-scan abort followed by a diverged tick.
+- Tests: `test/refresh-barrier-held-after-abort.test.ts` (PGLite, and Postgres when `DATABASE_URL` is set). It covers a refresh aborted by another writer's commit followed by a diverged retry, a refresh killed at fenced and then abandoned by `--resume`, and an owner-scan abort followed by a diverged tick, plus a run that already synced the members (no second sync, each member listed once), a member left behind HEAD named with its resume command, and an argument refusal that runs no owed sync.
## [0.60.613.0] - 2026-11-08
diff --git a/docs/designs/REFRESH_WRITE_BARRIER.md b/docs/designs/REFRESH_WRITE_BARRIER.md
index 182fe66820..94d6440be1 100644
--- a/docs/designs/REFRESH_WRITE_BARRIER.md
+++ b/docs/designs/REFRESH_WRITE_BARRIER.md
@@ -229,12 +229,15 @@ it, so a write released from it and then held for that import would wait for som
while the checkout stays diverged. GST-1853: while the worktree's newest refresh is `aborted`, every refresh run on the
owner host (a tick, `--resume`, `--abandon`, even one its precheck refuses `refresh_diverged`) ends, after its own
locks are released, by running the managed `--no-pull` sync of each member whose `last_commit` is not the checkout
-HEAD, within `sources.refresh_sync_budget_ms` (`syncAfterUnfinishedRefresh`; the run lists them in `synced`, and a
-refusal's message names them). The held writes are then prepared again against the checkout as it is, as above: an
+HEAD and that the run did not already sync, within `sources.refresh_sync_budget_ms` (`syncAfterUnfinishedRefresh`;
+the run lists them in `synced` once each, and a refusal's message names them). A refusal of the run's own arguments,
+`refresh_in_progress` and an unexpected failure skip it. The held writes are then prepared again against the checkout as it is, as above: an
`expected_revision` put ends `revision_conflict`, a blind put or capture lands and names what it replaced, an append
applies on top. Releasing them this way, instead of refusing them, gives the caller an answer within the run, and a
refusal would only be held again on retry while the import stays pending. A member this sync cannot bring to HEAD (its
-drain blocked) waits for the next run or sync, and its held writes stay bound by the wait limit below.
+drain blocked or out of budget) is named with `gbrain sync --source <id> --no-pull` in `next` (a blocked one also in
+`sync_blocked`) or the refusal's message, and waits for the next run or sync; its held writes stay bound by the wait
+limit below.
**What a blind write replaced.** A held `put_page` or `capture` that replaces content an import brought in names
it in three places, so the replacement is auditable and recoverable without digging: the receipt's
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index e25dae0f58..e6ec31b742 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -674,13 +674,17 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt
catch (error) {
if (!(error instanceof OperationError) && classifyPgAccessError(error).reason !== 'unknown') throw error;
const failure = error instanceof OperationError ? error : await unexpectedRefreshFailure(engine, sourceId, opts, error);
- const owed = opts.dryRun ? [] : await syncAfterUnfinishedRefresh(engine, sourceId, opts).catch(() => []);
- if (owed.length) failure.message += ` ${owedSyncNote(owed)}`;
+ const owed = opts.dryRun || NO_OWED_SYNC_AFTER.has(failure.code) ? NO_OWED_SYNC
+ : await syncAfterUnfinishedRefresh(engine, sourceId, opts, new Set()).catch(owedSyncFailed(sourceId));
+ if (owed.synced.length) failure.message += ` ${owedSyncNote(owed)}`;
throw failure;
}
if (opts.dryRun) return result;
- const owed = await syncAfterUnfinishedRefresh(engine, sourceId, opts).catch(() => []);
- return owed.length ? { ...result, synced: [...result.synced, ...owed] } : result;
+ const owed = await syncAfterUnfinishedRefresh(engine, sourceId, opts, new Set(result.synced.map(member => member.source_id))).catch(owedSyncFailed(sourceId));
+ if (!owed.synced.length) return result;
+ const blocked = [...(result.sync_blocked ?? []), ...owed.blocked];
+ const next = [result.next, ...owed.resume].filter(Boolean).join(' ; ');
+ return { ...result, synced: [...result.synced, ...owed.synced], ...(blocked.length ? { sync_blocked: blocked } : {}), ...(next ? { next } : {}) };
}
/**
@@ -688,36 +692,50 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt
* the member syncs its fence refused, so a write held for an import on a member (import-deferral.ts) would wait for some
* later sync, indefinitely while the checkout stays diverged. While the worktree's newest refresh is `aborted`, every
* refresh run on the owner host therefore ends, after its own locks are released, by running the managed `--no-pull`
- * sync of each member whose last_commit is not the checkout HEAD, within the sync budget, so those writes are prepared
- * again against the checkout as it is. Best effort: a member it cannot sync waits for the next run or sync, and its held
- * writes stay bound by the wait limit (`pending_import_expired`).
+ * sync of each member whose last_commit is not the checkout HEAD and that the run did not sync itself (`done`), within the
+ * sync budget, so those writes are prepared again against the checkout as it is. A refusal of the run's own arguments, a
+ * live refresh or an unexpected failure skips it. Best effort: a member it leaves behind HEAD is named with its resume
+ * command and waits for the next run or sync, and its held writes stay bound by the wait limit (`pending_import_expired`).
*/
-async function syncAfterUnfinishedRefresh(engine: BrainEngine, sourceId: string, opts: RefreshOptions): Promise<RefreshMemberSync[]> {
+async function syncAfterUnfinishedRefresh(engine: BrainEngine, sourceId: string, opts: RefreshOptions, done: ReadonlySet<string>): Promise<OwedSync> {
const hostId = existingHostIdFor(engine);
- if (!hostId || !await managedPersistenceEnabled(engine)) return [];
+ if (!hostId || !await managedPersistenceEnabled(engine)) return NO_OWED_SYNC;
const binding = await getWorktreeBinding(engine, sourceId, hostId);
- if (!binding || binding.owner_host_id !== hostId || !binding.local_path) return [];
+ if (!binding || binding.owner_host_id !== hostId || !binding.local_path) return NO_OWED_SYNC;
const root = binding.local_path;
const [newest] = await engine.executeRaw<Pick<WorktreeRefreshRow, 'state' | 'source_ids'>>(
'SELECT state,source_ids FROM persistence_worktree_refreshes WHERE worktree_id=$1::uuid ORDER BY created_at DESC LIMIT 1', [binding.worktree_id]);
- if (newest?.state !== 'aborted') return [];
+ if (newest?.state !== 'aborted') return NO_OWED_SYNC;
const head = await withGit(git => headOf(git, root));
- if (!head) return [];
+ if (!head) return NO_OWED_SYNC;
const lagging: string[] = [];
- for (const member of newest.source_ids) if (await lastCommit(engine, member) !== head) lagging.push(member);
- if (!lagging.length) return [];
+ for (const member of newest.source_ids) if (!done.has(member) && await lastCommit(engine, member) !== head) lagging.push(member);
+ if (!lagging.length) return NO_OWED_SYNC;
startPersistenceConsumer(engine, consumerConfig(engine));
const sync = await boundedMemberSync(await configuredMs(engine, opts.syncBudgetMs, 'GBRAIN_REFRESH_SYNC_BUDGET_MS', 'sources.refresh_sync_budget_ms', SYNC_BUDGET_MS));
- const synced: RefreshMemberSync[] = [];
+ const owed: OwedSync = { synced: [], blocked: [], resume: [] };
for (const [index, member] of lagging.entries()) {
- const status = await sync.member(engine, member, lagging.length - index).then(result => result.status, () => 'blocked');
- synced.push({ source_id: member, status, last_commit: await lastCommit(engine, member) });
+ const resume = `gbrain sync --source ${member} --no-pull`;
+ const result = await sync.member(engine, member, lagging.length - index).catch((error): MemberSync => ({ status: 'blocked', blocked: { source_id: member,
+ code: error instanceof OperationError ? error.code : 'storage_error', message: error instanceof Error ? error.message : String(error), resume } }));
+ const commit = await lastCommit(engine, member);
+ owed.synced.push({ source_id: member, status: result.status, last_commit: commit });
+ if (result.blocked) owed.blocked.push(result.blocked);
+ if (commit !== head) owed.resume.push(resume);
}
- return synced;
+ return owed;
}
+type OwedSync = { synced: RefreshMemberSync[]; blocked: RefreshSyncBlocked[]; resume: string[] };
+const NO_OWED_SYNC: OwedSync = { synced: [], blocked: [], resume: [] };
+const NO_OWED_SYNC_AFTER = new Set(['invalid_params', 'refresh_in_progress', 'internal_error']);
+const owedSyncFailed = (sourceId: string) => (error: unknown): OwedSync => {
+ console.error(`[gbrain] the sync owed after an unfinished refresh of ${sourceId} failed (${error instanceof Error ? error.message : String(error)}); its members' next sync or refresh run retries it.`);
+ return NO_OWED_SYNC;
+};
const consumerConfig = (engine: BrainEngine) => loadConfig() ?? { engine: engine.kind };
-const owedSyncNote = (synced: RefreshMemberSync[]) => `The worktree's last refresh ended without completing, so this run then ran the managed sync of ${synced
- .map(member => `${member.source_id} (${member.status})`).join(', ')} at the checkout HEAD, for writes held for that import.`;
+const owedSyncNote = (owed: OwedSync) => `The worktree's last refresh ended without completing, so this run then ran the managed sync of ${owed.synced
+ .map(member => `${member.source_id} (${member.status})`).join(', ')} at the checkout HEAD, for writes held for that import.${owed.resume.length
+ ? ` To finish it, run ${owed.resume.join(' ; ')}.` : ''}`;
const redact = (text: string) => {
const home = homedir();
diff --git a/test/refresh-barrier-held-after-abort.test.ts b/test/refresh-barrier-held-after-abort.test.ts
index a056295cb0..07c33a48c5 100644
--- a/test/refresh-barrier-held-after-abort.test.ts
+++ b/test/refresh-barrier-held-after-abort.test.ts
@@ -26,7 +26,7 @@ import type { WriteRequest } from '../src/core/persistence/model.ts';
import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
import { testBackends } from './helpers/test-backends.ts';
import { withEnv } from './helpers/with-env.ts';
-import { commitFile, page, withRefreshFixture, type RefreshFixture } from './helpers/worktree-refresh-fixture.ts';
+import { commitFile, git, page, withRefreshFixture, type RefreshFixture } from './helpers/worktree-refresh-fixture.ts';
const backends = testBackends();
const engines: BrainEngine[] = [];
@@ -157,3 +157,38 @@ test('GST-1853: when the owner scan aborts the killed refresh, the next refresh
await expect(refreshWorktree(f.engine, f.alpha)).rejects.toMatchObject({ code: 'refresh_diverged' });
await expectDecided(f, ids);
}), 180_000);
+
+/** The killed refresh aborted by the owner scan, its held writes still pending and the checkout diverged. */
+async function abortedByScan(f: RefreshFixture): Promise<void> {
+ await killedAtFenced(f);
+ await resumeWorktreeRefreshes(f.engine);
+ expect((await f.refreshRows()).map(r => r.state)).toEqual(['aborted']);
+}
+
+test('GST-1853: a refresh run that already synced the members runs no second owed sync and lists each member once', () => each(async f => {
+ await abortedByScan(f);
+ git(f.root, 'push', '-q', '-f', 'origin', 'HEAD:main');
+ const result = await refreshWorktree(f.engine, f.alpha, { syncBudgetMs: 0 });
+ expect(result.status).toBe('sync_deferred');
+ expect(result.synced.map(member => member.source_id).sort()).toEqual([f.alpha, f.beta].sort());
+}), 180_000);
+
+test('GST-1853: a member the owed sync leaves behind HEAD is named with its resume command, on success and on refusal', () => each(async f => {
+ await abortedByScan(f);
+ const resume = `gbrain sync --source ${f.alpha} --no-pull`;
+ const refused = await refreshWorktree(f.engine, f.alpha, { syncBudgetMs: 0 }).then(() => null, thrown => thrown as { code?: string; message: string });
+ expect(refused?.code).toBe('refresh_diverged');
+ expect(refused?.message).toContain(resume);
+ const resumed = await refreshWorktree(f.engine, f.alpha, { resume: true, syncBudgetMs: 0 });
+ expect(resumed.synced.map(member => member.source_id)).toContain(f.alpha);
+ expect(resumed.next ?? '').toContain(resume);
+}), 180_000);
+
+test('GST-1853: a refusal of the run\'s own arguments runs no owed sync', () => each(async f => {
+ await abortedByScan(f);
+ const before = await f.lastCommit(f.alpha);
+ const refused = await refreshWorktree(f.engine, f.alpha, { waitDrainMs: -1 }).then(() => null, thrown => thrown as { code?: string; message: string });
+ expect(refused?.code).toBe('invalid_params');
+ expect(refused?.message).not.toContain('ended without completing');
+ expect(await f.lastCommit(f.alpha)).toBe(before);
+}), 180_000);