Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pending worktree recovery refuses before an active refresh; state-matrix tests for unreadable HEAD and pending recovery (#7679)

garrytana3c7d79d9caa
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 56b2eac454..74ea98466d 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -519,7 +519,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.
| <a id="refresh_diverged"></a>`refresh_diverged` | `refresh_diverged` | The checkout has commits its upstream lacks, so a fast-forward is impossible. Nothing changed. | Ask the user, then publish the local commits merged with the upstream: `gbrain git commit-push --push-only --keep-checkout --path <checkout>` merges in memory, pushes the merge, refuses on a conflicting file and never moves the checkout (the refusal's fix fills in the real path and counts the local commits). Then retry `gbrain sources refresh <source>`, which fast-forwards to the merge. `gbrain sources reclone <source>` replaces the checkout instead and drops the local commits. |
| <a id="refresh_dirty"></a>`refresh_dirty` | `refresh_dirty` | Uncommitted or untracked files overlap the incoming upstream changes (the first 20 are named), or `git merge --ff-only` refused for the same reason. HEAD did not move. Uncommitted files outside the incoming changes do not refuse; they are kept and listed as `preserved_uncommitted`. | Commit or discard the named paths, then retry. Check `gbrain sources writer status <source> --json` first: a pending Git effect may be about to commit a gbrain-published file. |
| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |
-| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands). An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
+| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
| <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |
| <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. |
| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index ba9777a753..04b2f64b8f 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -234,11 +234,11 @@ async function precheck(engine: BrainEngine, git: Git, sourceId: string, fetchTi
throw refusal('refresh_recovery_required', stored.cause ?? `Refresh ${active.id} could not verify the checkout HEAD, so no refresh recovers it on its own.`,
stored.fix ?? `gbrain sources writer status ${sourceId}, then gbrain sources refresh ${sourceId} --resume`, active.id);
}
- if (active) throw inProgress(`Refresh ${active.id} of this worktree is ${active.state}, held by ${describeHolder(holderOf(active))}; only one refresh per worktree runs at a time.`,
- sourceId, active.id);
if (binding.state !== 'active' || await worktreeRecoveryPending(engine, binding)) throw refusal('refresh_recovery_required',
'Publication or topology recovery is pending on this worktree; the checkout must not move until it is recovered.',
`gbrain sources writer status ${sourceId}, let the owner finish recovery, then gbrain sources refresh ${sourceId}`);
+ if (active) throw inProgress(`Refresh ${active.id} of this worktree is ${active.state}, held by ${describeHolder(holderOf(active))}; only one refresh per worktree runs at a time.`,
+ sourceId, active.id);
const members = (await engine.executeRaw<{ source_id: string }>('SELECT source_id FROM persistence_source_bindings WHERE worktree_id=$1::uuid ORDER BY source_id',
[binding.worktree_id])).map(row => row.source_id);
await assertNoUnfinishedSync(engine, members);
@@ -575,7 +575,7 @@ async function hostRoot(engine: BrainEngine, worktreeId: string, hostId = localH
*/
async function recoverOne(engine: BrainEngine, git: Git, row: WorktreeRefreshRow, root: string, unattended = false): Promise<WorktreeRefreshRow> {
const head = await headOf(git, root);
- if (head === null && unattended) throw inProgress(`Refresh ${row.id} is orphaned, but the checkout HEAD could not be read, so it stays ${row.state}; the next refresh retries it.`, row.source_ids[0], row.id);
+ if (head === null && unattended) throw inProgress(`Refresh ${row.id} is orphaned, but the checkout HEAD could not be read, so it stays ${row.state}; once git can read HEAD again, the next refresh retries it.`, row.source_ids[0], row.id);
if (row.state === 'draining') {
await engine.transaction(tx => transition(tx, row, ['draining'], 'aborted', { refusal: { code: 'refresh_interrupted', cause: 'The refreshing process stopped while draining; no merge was started.' } }));
} else if (row.state === 'fenced' && head === row.old_head) {
@@ -621,7 +621,7 @@ export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostI
if (after.state !== row.state) moved++;
});
} catch (error) {
- // GBRA-130 A-34: one refresh's unexpected failure leaves that row for the next owner start and never stops the others or the topology scan.
+ // GBRA-130 A-34: one refresh's unexpected failure leaves that row for the next topology scan and never stops the others or the scan.
if (!(error instanceof OperationError && ['write_pending', 'recovery_required', 'refresh_recovery_required', 'owner_unavailable'].includes(error.code))) {
console.error(`[gbrain] worktree refresh ${row.id} (${row.source_ids.join(', ')}) did not converge at owner start (${String((error as { code?: unknown } | null)?.code ?? (error as Error)?.name ?? 'error')}); it stays ${row.state} and the owner's next topology scan retries it. Check it with gbrain sources writer status ${row.source_ids[0]}.`);
}
@@ -713,17 +713,18 @@ async function resumeHeld(engine: BrainEngine, git: Git, sourceId: string, refre
* #7679: a plain `gbrain sources refresh` that finds an active refresh whose lock no process holds (its refresher died,
* for example a container replaced mid-refresh) finishes it through the `--resume` path instead of refusing
* `refresh_in_progress` on every scheduled run. Returns null when there is nothing to adopt: no active row, a
- * `recovery_required` row (the user decides how to reset), a live holder (the lock stays busy for a second), or a
- * caller that is not the owner (the precheck refuses each case by name).
+ * `recovery_required` row (the user decides how to reset), publication or topology recovery pending on the worktree,
+ * a live holder (the lock stays busy for a second), a caller that is not the owner, or a checkout HEAD git cannot read
+ * (an unattended run never escalates on it); the precheck then refuses each case by name.
*/
async function adoptOrphanedRefresh(engine: BrainEngine, sourceId: string): Promise<WorktreeRefreshResult | null> {
if (!await managedPersistenceEnabled(engine)) return null;
const hostId = existingLocalHostId();
const binding = await getWorktreeBinding(engine, sourceId, hostId);
if (!binding || !hostId || binding.owner_host_id !== hostId || !binding.local_path) return null;
- if (binding.state !== 'active' || await worktreeRecoveryPending(engine, binding)) return null;
const active = await readActiveRefresh(engine, binding.worktree_id);
if (!active || active.state === 'recovery_required') return null;
+ if (binding.state !== 'active' || await worktreeRecoveryPending(engine, binding)) return null;
// A holder recorded on another host is never adopted: this host's lock cannot say whether it is alive.
if ((holderOf(active)?.host_id ?? hostId) !== hostId) return null;
const lock = await acquireNativeLock(refreshLockPath(binding.worktree_id), { timeoutMs: 1000 });
diff --git a/test/persistence-worktree-refresh.test.ts b/test/persistence-worktree-refresh.test.ts
index 89ccf3ceec..3823cdcbcd 100644
--- a/test/persistence-worktree-refresh.test.ts
+++ b/test/persistence-worktree-refresh.test.ts
@@ -421,6 +421,61 @@ test('7c. #7679 state matrix: an interrupted refresh of each kind, settled by th
});
}, 600_000);
+test('7d. #7679 state matrix: an unreadable HEAD and pending worktree recovery never move an interrupted refresh the unattended paths cannot verify', async () => {
+ // Value: protects=a plain refresh leaves an orphan untouched (state, holder, previous_holder, HEAD) when git cannot read HEAD (refusing
+ // refresh_in_progress) or when publication/topology recovery is pending (refusing refresh_recovery_required, the blocker that must clear first);
+ // the owner scan aborts a dead drain whatever HEAD reads, never adopts an unverified HEAD for a fenced row, and leaves a lagging syncing row;
+ // fails_when=adoption runs recovery on a null HEAD (escalating to recovery_required) or ahead of the recovery gate, or precheck names the
+ // orphan instead of the pending recovery; why_new=7c covers readable HEADs with no pending recovery only; seam=.git/HEAD overwritten with
+ // garbage, a recovering topology change row naming the worktree; no_claim=the race where HEAD becomes unreadable between adoption's two reads.
+ type Kind = 'draining' | 'fenced_old' | 'fenced_target' | 'syncing_lagging' | 'recovery_required';
+ type Action = 'plain_head_unreadable' | 'scan_head_unreadable' | 'plain_recovery_pending';
+ const crashAt = { draining: 'drained', fenced_old: 'fenced', fenced_target: 'fenced', syncing_lagging: 'syncing', recovery_required: 'fenced' } as const;
+ const kinds: Kind[] = ['draining', 'fenced_old', 'fenced_target', 'syncing_lagging', 'recovery_required'];
+ const cells: Array<[Kind, Action]> = [
+ ...(['draining', 'fenced_old', 'syncing_lagging'] as Kind[]).map(kind => [kind, 'plain_head_unreadable'] as [Kind, Action]),
+ ...(['draining', 'fenced_old', 'syncing_lagging'] as Kind[]).map(kind => [kind, 'scan_head_unreadable'] as [Kind, Action]),
+ ...kinds.map(kind => [kind, 'plain_recovery_pending'] as [Kind, Action])];
+ for (const engine of engines) for (const [kind, action] of cells) await withRefreshFixture(engine, async f => {
+ const label = `${kind} x ${action}`;
+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ await expect(refreshWorktree(f.engine, f.alpha, { hooks: { boundary: point => { if (point === crashAt[kind]) throw new Error(`simulated crash at ${point}`); } } }))
+ .rejects.toThrow('simulated crash');
+ await disposePersistenceConsumer(f.engine);
+ const [orphan] = await f.refreshRows();
+ const state = kind === 'draining' ? 'draining' : kind === 'syncing_lagging' ? 'syncing' : kind === 'recovery_required' ? 'recovery_required' : 'fenced';
+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET state=$2,completed_at=NULL WHERE id=$1::uuid`, [orphan.id, state]);
+ if (kind === 'fenced_target') git(f.root, 'merge', '-q', '--ff-only', target);
+ const head = f.head();
+ const settled = async () => (await f.refreshRows()).map(r => [r.state, r.outcome.previous_holder]);
+ if (action === 'plain_recovery_pending') {
+ await f.engine.executeRaw(`INSERT INTO persistence_topology_changes(principal_id,request_id,digest,operation,source_id,worktree_ids,state,recovery)
+ VALUES($1::uuid,$2::uuid,'example','reclone',$3,ARRAY[$4::uuid],'recovering',$5::text::jsonb)`,
+ [randomUUID(), randomUUID(), f.alpha, f.worktreeId, JSON.stringify({ worktreeId: f.worktreeId })]);
+ await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_recovery_required');
+ expect(await settled(), label).toEqual([[state, undefined]]);
+ expect(f.head(), label).toBe(head);
+ return;
+ }
+ const headFile = join(f.root, '.git', 'HEAD');
+ const saved = readFileSync(headFile, 'utf8');
+ writeFileSync(headFile, 'not a ref\n');
+ try {
+ if (action === 'plain_head_unreadable') {
+ const refused = await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_in_progress');
+ expect(refused.suggestion, label).toContain(`gbrain sources writer status --source ${f.alpha} --json`);
+ expect(await settled(), label).toEqual([[state, undefined]]);
+ } else {
+ const moved = await resumeWorktreeRefreshes(f.engine);
+ const expected = { draining: 'aborted', fenced_old: 'recovery_required', syncing_lagging: 'syncing' }[kind as 'draining' | 'fenced_old' | 'syncing_lagging'];
+ expect(moved, label).toBe(expected === state ? 0 : 1);
+ expect((await f.refreshRows()).map(r => r.state), label).toEqual([expected]);
+ }
+ } finally { writeFileSync(headFile, saved); }
+ expect(f.head(), label).toBe(head);
+ });
+}, 600_000);
+
test('8. a queued git effect drains before fenced, and nothing on the worktree is claimed while fenced', () => each(async f => {
f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
await f.put(f.beta, 'notes/effect', page('Effect', 'Has a git effect.'));