Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(refresh): wave #8123 part A: abandon state-matrix cells, owner-scan lock and cache fixes, HEAD-aware writer status, delegated fix

garrytanfb2dcde77e65
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bebd4b391d..1f54b882e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,18 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**`gbrain sources refresh` says what is actually holding an interrupted refresh, and its owner recovery stops holding locks it doesn't need (#8123).**
+
+Writer status for a `recovery_required` refresh now reads the checkout HEAD: at the upstream commit it names `--resume`, at the start commit `--abandon`, and only another HEAD asks the user to reset. An orphan whose HEAD git cannot read names `git -C <checkout> status` instead of promising that the next refresh finishes it, and the owner scan logs it once every 15 minutes instead of swallowing it. A refresh lock file this host cannot open reads `liveness: unknown` with the lock path, and a plain refresh refuses `refresh_in_progress` instead of failing with a raw lock error. A refresh held on another host names that host in `next`. A plain refresh that rolled back an orphan and then refused keeps that recovery in its refusal message. A delegated refresh (a resident PGLite owner) prints the same structured `fix` as a direct one.
+
+### Itemized changes
+
+- `src/core/persistence/worktree-refresh.ts`: the owner scan checks member progress before it takes the refresh lock, caches a syncing orphan it could not complete because of its HEAD, fails one row (not the scan) on a lock error, and prunes its caches each scan (#7712, #7728, #7730, #7731); `refreshHolderReport` reads HEAD (#7712, #7731); `next` is host-scoped and shell-quoted (#7731).
+- `src/core/persistence/ipc.ts`: a refusal's `fix` survives the owner socket (#7707).
+- Tests: the #7679 state matrix gains the `--abandon` action for every kind (#7718-#7727, #7729), plus red tests for the owner scan's syncing filter and test 7's adoption guard (#7708) and the unreadable-lock branch (#7728).
+
## [0.60.529.0] - 2026-10-31
**A pull request that adds a file to the PGLite snapshot inputs no longer gets stopped at gstack's pregate secrets check by its own workflow cache keys (#8107).**
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index e43d770ea2..96ad4aa2b8 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -551,11 +551,11 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.
| <a id="refresh_diverged"></a>`refresh_diverged` | `refresh_diverged` | The checkout has commits its upstream lacks, so a fast-forward is impossible. Nothing changed. | Ask the user, then publish the local commits merged with the upstream: `gbrain git commit-push --push-only --keep-checkout --path <checkout>` merges in memory, pushes the merge, refuses on a conflicting file and never moves the checkout (the refusal's fix fills in the real path and counts the local commits). Then retry `gbrain sources refresh <source>`, which fast-forwards to the merge. `gbrain sources reclone <source>` replaces the checkout instead and drops the local commits. |
| <a id="refresh_dirty"></a>`refresh_dirty` | `refresh_dirty` | Uncommitted or untracked files overlap the incoming upstream changes (the first 20 are named), or `git merge --ff-only` refused for the same reason. HEAD did not move. Uncommitted files outside the incoming changes do not refuse; they are kept and listed as `preserved_uncommitted`. | Commit or discard the named paths, then retry. Check `gbrain sources writer status <source> --json` first: a pending Git effect may be about to commit a gbrain-published file. |
| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |
-| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). A refresh whose holder is recorded on another host is never taken over either: writer status shows its `liveness` as `unknown`, and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
+| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again; writer status then names `git -C <checkout> status` instead of the next refresh). A refresh whose holder is recorded on another host, or whose lock file this host cannot open, is never taken over either: writer status shows its `liveness` as `unknown` (its `next` names the host to run on), and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
| <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |
| <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within its share of the refresh sync budget (each member gets an equal share of what is left; `sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. |
| <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. |
-| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |
+| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. Writer status reads HEAD and names the one step left when HEAD already sits at one of those commits: `--resume` at the upstream commit, `--abandon` at the start commit. |
| <a id="worktree_refreshing"></a>`worktree_refreshing` | `worktree_refreshing` | A write, or a managed sync started by a cycle or another command, reached a checkout while a refresh is draining or merging it. It was not journaled. `detail` is `retry_after_ms=1000`. Writes to database-only memory are not affected. | Retry the same request ID after about a second. `gbrain sources writer status <source>` shows the refresh. A cycle's sync is retried by the next cycle. |
A refresh is recorded in `persistence_worktree_refreshes` and survives a crash.
diff --git a/src/core/persistence/ipc.ts b/src/core/persistence/ipc.ts
index 41f147a0d6..3d5ecb3d7e 100644
--- a/src/core/persistence/ipc.ts
+++ b/src/core/persistence/ipc.ts
@@ -353,9 +353,36 @@ function remoteOperationError(value: unknown): OperationError {
if (isWriteReceipt(value.write_request)) error.writeRequest = publicWriteReceipt(value.write_request);
if (isWriteRequestId(value.request_id)) error.requestId = value.request_id;
if (isWriteErrorCode(value.write_error)) error.writeError = value.write_error;
+ const fix = wireAction(value.fix);
+ if (fix) error.fix = fix;
return error;
}
+/**
+ * #7707: the owner sends a refusal's `fix` already rendered (`toJSON`), or as the stored Action when it never loaded the renderer.
+ * Back to an Action, so a delegated refusal prints the same structured fix as the direct path; a value of any other shape is dropped.
+ */
+function wireAction(value: unknown): Action | undefined {
+ if (!record(value) || !Array.isArray(value.consent) || typeof value.actor !== 'string' || typeof value.why !== 'string'
+ || typeof value.requires_exclusive !== 'boolean') return undefined;
+ const argv = Array.isArray(value.argv) && value.argv.every(arg => typeof arg === 'string') ? value.argv as string[] : undefined;
+ const mcp = record(value.mcp) && typeof value.mcp.tool === 'string' && record(value.mcp.arguments) ? value.mcp as unknown as Action['mcp'] : undefined;
+ if (!argv && !mcp) return undefined;
+ const then = value.then === undefined ? undefined : wireAction(value.then);
+ return {
+ ...(value.next === 'ask_user' || value.ask_user === true ? { ask_user: true as const } : {}),
+ ...(argv ? { argv } : {}), ...(mcp ? { mcp } : {}),
+ consent: value.consent as Action['consent'], actor: value.actor as Action['actor'], why: value.why, requires_exclusive: value.requires_exclusive,
+ ...(typeof value.user_message === 'string' ? { user_message: value.user_message } : {}),
+ ...(record(value.verify) ? { verify: value.verify as Action['verify'] } : {}),
+ ...(typeof value.docs === 'string' ? { docs: value.docs } : {}),
+ ...(Array.isArray(value.inputs) ? { inputs: value.inputs as Action['inputs'] } : {}),
+ ...(typeof value.plan_hash === 'string' ? { plan_hash: value.plan_hash } : {}),
+ ...(Array.isArray(value.preview_argv) && value.preview_argv.every(arg => typeof arg === 'string') ? { preview_argv: value.preview_argv as string[] } : {}),
+ ...(then ? { then } : {}),
+ };
+}
+
/** One bounded exchange. No automatic reconnect or replay after any request bytes are sent. */
async function exchange(socketPath: string, request: unknown, timeoutMs: number, requestId?: string): Promise<unknown> {
const frame = responseFrame(request);
diff --git a/src/core/persistence/sync-status.ts b/src/core/persistence/sync-status.ts
index 32390f5955..1dbbb5c8b7 100644
--- a/src/core/persistence/sync-status.ts
+++ b/src/core/persistence/sync-status.ts
@@ -230,7 +230,7 @@ export interface SyncStatus {
landing_read: SyncStatusRead;
}
export type SyncStatusState = 'synced' | 'resumable' | 'blocked';
-type RefreshProbeRow = { id: string; worktree_id: string; state: import('./worktree-refresh-schema.ts').WorktreeRefreshState; source_ids: string[]; outcome: Record<string, unknown>; updated_at: string; owner_host_id: string | null };
+type RefreshProbeRow = { id: string; worktree_id: string; state: import('./worktree-refresh-schema.ts').WorktreeRefreshState; source_ids: string[]; outcome: Record<string, unknown>; old_head: string; target_head: string; updated_at: string; owner_host_id: string | null };
/** Why the source is in its state; one closed list. */
export const SYNC_STATUS_REASONS = ['nothing_pending', 'held_by_design', 'held_needs_human', 'cursor_unfinished', 'live_job', 'last_error', 'holds_actionable', 'recovery_pending', 'request_pending',
'status_unknown', 'needs_human', 'head_blocked', 'recovery_blocked',
@@ -341,7 +341,7 @@ export async function readSyncStatus(engine: BrainEngine, sourceId: string, opts
(SELECT r.request_id::text FROM persistence_effects e JOIN persistence_requests r ON r.id=e.request_id WHERE e.kind='git' AND e.state='failed' AND NOT (e.data ? 'settled')
AND e.request_id IN (SELECT id::uuid FROM jsonb_array_elements_text(CASE WHEN jsonb_typeof(p.completed_keys->0->'obligations'->'converged'->'writeback')='array'
THEN p.completed_keys->0->'obligations'->'converged'->'writeback' ELSE '[]'::jsonb END) AS w(id)) ORDER BY e.id LIMIT 1) AS writeback_failed,
- (SELECT row_to_json(f) FROM (SELECT r.id,r.worktree_id,r.state,r.source_ids,r.outcome,r.updated_at,w.owner_host_id FROM persistence_worktree_refreshes r
+ (SELECT row_to_json(f) FROM (SELECT r.id,r.worktree_id,r.state,r.source_ids,r.outcome,r.old_head,r.target_head,r.updated_at,w.owner_host_id FROM persistence_worktree_refreshes r
LEFT JOIN persistence_worktrees w ON w.id=r.worktree_id WHERE s.id=ANY(r.source_ids) AND r.state IN ${ACTIVE_REFRESH_STATES_SQL} LIMIT 1) f) AS refresh FROM sources s
LEFT JOIN op_checkpoints p ON p.op=$2 AND p.fingerprint=s.id||':'||s.incarnation::text
LEFT JOIN op_checkpoints w ON w.op=$3 AND w.fingerprint=s.id||':'||s.incarnation::text
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index 93aab6ef82..37b485127d 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -488,8 +488,14 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt
const waitDrainMs = await configuredMs(engine, opts.waitDrainMs, 'GBRAIN_REFRESH_DRAIN_WAIT_MS', 'sources.refresh_drain_wait_ms', 60_000);
const recovered = opts.dryRun ? null : await adoptOrphanedRefresh(engine, sourceId, syncBudgetMs);
if (recovered && recovered.status !== 'completed' && recovered.status !== 'abandoned') return recovered;
- const result = await refreshOnce(engine, sourceId, opts, fetchTimeoutMs, waitDrainMs, syncBudgetMs);
- return recovered?.recovered ? { ...result, recovered: recovered.recovered } : result;
+ if (!recovered?.recovered) return refreshOnce(engine, sourceId, opts, fetchTimeoutMs, waitDrainMs, syncBudgetMs);
+ const { refresh_id, status, previous_holder } = recovered.recovered;
+ const result = await refreshOnce(engine, sourceId, opts, fetchTimeoutMs, waitDrainMs, syncBudgetMs).catch(error => {
+ // #7712: the recovery this run already made stays on record when its own refresh then refuses.
+ if (error instanceof OperationError) error.message += ` Before that, this run finished orphaned refresh ${refresh_id} (${status}; previous holder: ${describeHolder(previous_holder)}).`;
+ throw error;
+ });
+ return { ...result, recovered: recovered.recovered };
}
async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshOptions, fetchTimeoutMs: number, waitDrainMs: number, syncBudgetMs: number): Promise<WorktreeRefreshResult> {
@@ -649,6 +655,20 @@ async function recoverOne(engine: BrainEngine, git: Git, row: WorktreeRefreshRow
return readRefresh(engine, row.id);
}
+/**
+ * Orphaned `syncing` rows the scan could not complete, keyed by refresh id, with the member `last_commit`s it saw and when: one whose
+ * members lag behind the target, or whose HEAD is not the target or a descendant of it (#7731). The scan re-checks it once a member moves,
+ * or after LAGGING_RECHECK_MS (a git failure reads as lagging, so a cached verdict never outlives a transient one). Each scan drops the
+ * ids it no longer lists, so a long-lived owner keeps no entry for a refresh that settled.
+ */
+const laggingSyncing = new Map<string, { commits: string; at: number }>();
+const LAGGING_RECHECK_MS = 60_000;
+/** #7731: when the scan last logged that an orphan's checkout HEAD is unreadable, so it says so once in a while instead of every tick. */
+const unreadableHeadLogged = new Map<string, number>();
+const UNREADABLE_HEAD_LOG_MS = 15 * 60_000;
+/** @internal tests only: how many refresh ids the owner scan's caches hold. */
+export const _refreshScanCacheSizeForTests = () => laggingSyncing.size + unreadableHeadLogged.size;
+
/**
* Owner recovery (the consumer's topology scan): every interrupted refresh on
* this host's worktrees whose refreshing process has exited (its refresh lock
@@ -656,21 +676,17 @@ async function recoverOne(engine: BrainEngine, git: Git, row: WorktreeRefreshRow
* Member syncs are not run here. #7679: an orphaned `syncing` row whose members
* all synced to the target or a descendant of it (a later cycle sync caught
* them up) and whose HEAD is still at the target is completed here (a lagging one is re-checked after a member moves or a minute passes); one whose members lag is finished by the next
- * plain `gbrain sources refresh`, which runs the `--resume` path, or by `--resume`.
+ * plain `gbrain sources refresh`, which runs the `--resume` path, or by `--resume`. #7712: member progress is checked before the refresh
+ * lock is taken, so a lagging row never holds it.
*/
-/**
- * A lagging orphaned `syncing` row, keyed by refresh id, with the member `last_commit`s it lagged at and when: the scan re-checks it once a
- * member moves, or after LAGGING_RECHECK_MS (a git failure reads as lagging, so a cached verdict never outlives a transient one).
- */
-const laggingSyncing = new Map<string, { commits: string; at: number }>();
-const LAGGING_RECHECK_MS = 60_000;
-
export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostId?: string } = {}): Promise<number> {
const host = opts.hostId ?? localHostId();
const rows = await engine.executeRaw<WorktreeRefreshRow>(`SELECT f.* FROM persistence_worktree_refreshes f
JOIN persistence_worktrees w ON w.id=f.worktree_id WHERE w.owner_host_id=$1::uuid AND f.state IN ('draining','fenced','merged','syncing')
AND (f.state<>'syncing' OR NOT EXISTS (SELECT 1 FROM sources s WHERE s.id=ANY(f.source_ids) AND (s.last_commit IS NULL OR s.last_commit=f.old_head)))
ORDER BY f.state='syncing', f.created_at LIMIT 16`, [host]);
+ const listed = new Set(rows.map(row => row.id));
+ for (const cache of [laggingSyncing, unreadableHeadLogged]) for (const id of cache.keys()) if (!listed.has(id)) cache.delete(id);
let moved = 0;
for (const row of rows) {
const root = await hostRoot(engine, row.worktree_id, host);
@@ -679,23 +695,35 @@ export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostI
'SELECT last_commit FROM sources WHERE id=ANY($1::text[]) ORDER BY id', [row.source_ids])).map(source => source.last_commit ?? '').join(',') : null;
const lagging = commits === null ? undefined : laggingSyncing.get(row.id);
if (lagging?.commits === commits && Date.now() - lagging.at < LAGGING_RECHECK_MS) continue;
- const lock = await tryAcquireNativeLock(refreshLockPath(row.worktree_id));
- if (!lock) continue;
try {
await withGit(async git => {
if (commits !== null && !await membersAtTarget(engine, git, row, root)) { laggingSyncing.set(row.id, { commits, at: Date.now() }); return; }
- const after = await withTopologyLocks(engine, row.source_ids[0], async () => {
- const current = await readRefresh(engine, row.id);
- return current.state === 'syncing' ? completeSyncedOrphan(engine, git, current, root) : recoverOne(engine, git, current, root, true);
- }, undefined, 0);
- if (after.state !== row.state) moved++;
+ const lock = await tryAcquireNativeLock(refreshLockPath(row.worktree_id));
+ if (!lock) return;
+ try {
+ const after = await withTopologyLocks(engine, row.source_ids[0], async () => {
+ const current = await readRefresh(engine, row.id);
+ return current.state === 'syncing' ? completeSyncedOrphan(engine, git, current, root) : recoverOne(engine, git, current, root, true);
+ }, undefined, 0);
+ if (after.state !== row.state) moved++;
+ else if (commits !== null) laggingSyncing.set(row.id, { commits, at: Date.now() });
+ } finally { await lock.release(); }
});
} catch (error) {
+ if (error instanceof OperationError && error.code === 'refresh_in_progress' && row.state !== 'draining') {
+ // #7731: recoverOne refuses an unattended step on an unreadable HEAD; the fence stays, so say so now and then, not on every tick.
+ const logged = unreadableHeadLogged.get(row.id);
+ if (logged === undefined || Date.now() - logged >= UNREADABLE_HEAD_LOG_MS) {
+ unreadableHeadLogged.set(row.id, Date.now());
+ console.error(`[gbrain] worktree refresh ${row.id} (${row.source_ids.join(', ')}) is orphaned in ${row.state}, but git cannot read the checkout HEAD in ${root}, so the owner's topology scan leaves it and writes to its sources stay refused. Check git -C ${q(root)} status, then gbrain sources refresh ${row.source_ids[0]} --resume.`);
+ }
+ continue;
+ }
// GBRA-130 A-34: one refresh's unexpected failure leaves that row for the next topology scan and never stops the others or the scan.
- if (!(error instanceof OperationError && ['write_pending', 'recovery_required', 'refresh_recovery_required', 'refresh_in_progress', 'owner_unavailable'].includes(error.code))) {
+ if (!(error instanceof OperationError && ['write_pending', 'recovery_required', 'refresh_recovery_required', 'owner_unavailable'].includes(error.code))) {
console.error(`[gbrain] worktree refresh ${row.id} (${row.source_ids.join(', ')}) did not converge in the owner's topology scan (${String((error as { code?: unknown } | null)?.code ?? (error as Error)?.name ?? 'error')}); it stays ${row.state} and the owner's next topology scan retries it. Check it with gbrain sources writer status ${row.source_ids[0]}.`);
}
- } finally { await lock.release(); }
+ }
}
return moved;
}
@@ -803,7 +831,8 @@ async function adoptOrphanedRefresh(engine: BrainEngine, sourceId: string, syncB
if (binding.state !== 'active' || await worktreeRecoveryPending(engine, binding)) return null;
// A holder recorded on another host is never adopted: this host's lock cannot say whether it is alive.
if ((holderOf(active)?.host_id ?? hostId) !== hostId) return null;
- const lock = await acquireNativeLock(refreshLockPath(binding.worktree_id), { timeoutMs: 1000 });
+ // #7728: a lock this process cannot read says nothing about the holder, so it is not adopted; the precheck refuses refresh_in_progress.
+ const lock = await acquireNativeLock(refreshLockPath(binding.worktree_id), { timeoutMs: 1000 }).catch(() => null as NativeLockHandle | null);
if (!lock) return null;
try {
const orphan = await readActiveRefresh(engine, binding.worktree_id);
@@ -852,7 +881,7 @@ export async function refreshLiveness(worktreeId: string, holderHostId: string |
}
/** #7679: the holder of an active refresh, whether it is orphaned, and the step that moves it on (`fix` names `--resume` whenever no process will). */
-export async function refreshHolderReport(engine: BrainEngine, row: Pick<WorktreeRefreshRow, 'id' | 'worktree_id' | 'state' | 'source_ids' | 'outcome'> & { owner_host_id: string | null }):
+export async function refreshHolderReport(engine: BrainEngine, row: Pick<WorktreeRefreshRow, 'id' | 'worktree_id' | 'state' | 'source_ids' | 'outcome' | 'old_head' | 'target_head'> & { owner_host_id: string | null }):
Promise<{ holder: RefreshHolder | null; liveness: RefreshLiveness; orphaned: boolean | null; auto_recovery: 'next_refresh' | 'none' | 'unknown'; fix: Action | null }> {
const holder = holderOf(row);
const holderHost = holder?.host_id ?? row.owner_host_id;
@@ -861,15 +890,28 @@ export async function refreshHolderReport(engine: BrainEngine, row: Pick<Worktre
const resume = ['gbrain', 'sources', 'refresh', source, '--resume'];
const verify = { argv: ['gbrain', 'sources', 'writer', 'status', '--source', source, '--json'] };
const base = { holder, liveness, orphaned: liveness === 'unknown' ? null : liveness === 'orphaned' };
- if (row.state === 'recovery_required') return { ...base, auto_recovery: 'none', fix: { argv: resume, consent: [], actor: 'agent', requires_exclusive: false, verify,
- why: `Refresh ${row.id} could not verify the checkout HEAD, so nothing recovers it on its own. --resume changes nothing here: it prints the two ways out (reset to the start commit and --abandon, or to the upstream commit and --resume); resetting is destructive, so ask the user first.` } };
+ const binding = liveness === 'unknown' ? null : await getWorktreeBinding(engine, source, holderHost);
+ // #7712, #7731: what the next step does depends on the checkout HEAD, read on the host that holds the checkout.
+ const head = binding?.local_path ? await withGit(git => headOf(git, binding.local_path!)).catch(() => null) : undefined;
+ if (row.state === 'recovery_required') {
+ if (head === row.target_head) return { ...base, auto_recovery: 'none', fix: { argv: resume, consent: [], actor: 'agent', requires_exclusive: false, verify,
+ why: `Refresh ${row.id} stopped without verifying the checkout HEAD, which is now the refresh target ${row.target_head}, so --resume records the merge and finishes the refresh; nothing needs resetting.` } };
+ if (head === row.old_head) return { ...base, auto_recovery: 'none', fix: { argv: ['gbrain', 'sources', 'refresh', source, '--abandon'], consent: [], actor: 'agent', requires_exclusive: false, verify,
+ why: `Refresh ${row.id} stopped without verifying the checkout HEAD, which is back at the pre-refresh commit ${row.old_head}, so --abandon lifts the fence and nothing moved; then gbrain sources refresh ${source} starts again.` } };
+ return { ...base, auto_recovery: 'none', fix: { argv: resume, consent: [], actor: 'agent', requires_exclusive: false, verify,
+ why: `Refresh ${row.id} could not verify the checkout HEAD, so nothing recovers it on its own. --resume changes nothing here: it prints the two ways out (reset to the start commit and --abandon, or to the upstream commit and --resume); resetting is destructive, so ask the user first.` } };
+ }
if (liveness === 'live') return { ...base, auto_recovery: 'none', fix: null };
- const binding = liveness === 'orphaned' ? await getWorktreeBinding(engine, source, holderHost) : null;
if (binding && (binding.state !== 'active' || await worktreeRecoveryPending(engine, binding))) return { ...base, auto_recovery: 'none', fix: { argv: verify.argv, consent: [], actor: 'agent', requires_exclusive: false, verify,
why: `No process holds refresh ${row.id}'s lock, but publication or topology recovery is pending on its worktree, so no refresh adopts it until the owner finishes that recovery (this read shows it, read-only). Then the next gbrain sources refresh ${source} finishes the refresh on its own.` } };
+ if (liveness === 'orphaned' && head === null && row.state !== 'draining') return { ...base, auto_recovery: 'none', fix: readFix(
+ `No process holds refresh ${row.id}'s lock, but git cannot read the checkout HEAD in ${binding!.local_path}, so neither the owner nor the next gbrain sources refresh ${source} moves it and writes to its sources stay refused. Git prints why (read-only); once it reads HEAD again, gbrain sources refresh ${source} --resume finishes the refresh.`,
+ { argv: ['git', '-C', binding!.local_path!, 'status'] }) };
if (liveness === 'orphaned') return { ...base, auto_recovery: 'next_refresh', fix: { argv: resume, consent: [], actor: 'agent', requires_exclusive: false, verify,
why: `No process holds refresh ${row.id}'s lock (recorded holder: ${describeHolder(holder)}), so it is orphaned. The next gbrain sources refresh ${source} finishes it on its own through the --resume path`
+ `${row.state === 'syncing' ? ', and the owner completes it once every member has synced to its target' : ''}; --resume finishes it now.` } };
+ if (holderHost !== null && holderHost === existingLocalHostId()) return { ...base, auto_recovery: 'unknown', fix: { argv: verify.argv, consent: [], actor: 'host_admin', requires_exclusive: false,
+ why: `This host cannot read refresh ${row.id}'s lock file ${refreshLockPath(row.worktree_id)}, so whether ${describeHolder(holder)} is alive is unknown and no refresh adopts it. Make the file readable and writable by the user gbrain runs as, then run this again: an orphaned refresh then names gbrain sources refresh ${source} --resume.` } };
return { ...base, auto_recovery: 'unknown', fix: { argv: verify.argv, consent: [], actor: 'host_admin', requires_exclusive: false,
why: `Refresh ${row.id}'s lock lives on host ${holderHost ?? 'unknown'}, not this one, so only that host can tell whether ${describeHolder(holder)} is alive. Run this there; if it reports the refresh orphaned, or that host is gone for good, gbrain sources refresh ${source} --resume on the owner host finishes it (a plain refresh does not adopt another host's refresh).` } };
}
@@ -886,7 +928,8 @@ export async function activeWorktreeRefreshes(engine: BrainEngine, sourceId?: st
...(row.outcome.refusal ? { refusal: row.outcome.refusal } : {}), ...report,
next: report.liveness === 'live' && row.state === 'draining' ? 'Writes to this worktree are refused while queued work drains; retry them after about a second.'
: report.liveness === 'live' ? `Wait for it: ${describeHolder(report.holder)} holds the refresh lock and finishes it.`
- : report.fix?.argv?.join(' ') ?? `gbrain sources refresh ${row.source_ids[0]} --resume` };
+ : report.liveness === 'unknown' && report.fix?.argv ? `On host ${report.holder?.hostname ?? 'unknown'} (${report.holder?.host_id ?? row.owner_host_id ?? 'unknown host id'}), which holds the refresh lock: ${report.fix.argv.map(q).join(' ')}`
+ : report.fix?.argv?.map(q).join(' ') ?? `gbrain sources refresh ${q(row.source_ids[0])} --resume` };
}));
}
diff --git a/test/persistence-ipc.test.ts b/test/persistence-ipc.test.ts
index 3177ab322f..c9c72dd7a1 100644
--- a/test/persistence-ipc.test.ts
+++ b/test/persistence-ipc.test.ts
@@ -5,9 +5,11 @@ import { mkdtempSync, rmSync, statSync, writeFileSync, readFileSync } from 'node
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { OperationError } from '../src/core/ops/contract.ts';
+import { printRefreshOutcome } from '../src/commands/sources-refresh.ts';
+import { _resetCliExitVerdictForTests } from '../src/core/cli-force-exit.ts';
import {
PERSISTENCE_IPC_MAX_BYTES, PersistenceIpcTransportError,
- persistenceSocketPathForConfig, requestPersistenceCapabilities, requestPersistenceOperation,
+ persistenceSocketPathForConfig, requestPersistenceAdministration, requestPersistenceCapabilities, requestPersistenceOperation,
startPersistenceIpcServer, type PersistenceIpcBinding, type PersistenceIpcRequest,
} from '../src/core/persistence/ipc.ts';
@@ -220,4 +222,32 @@ describe('dedicated persistence IPC', () => {
await expect(startPersistenceIpcServer(path, { brainId: BRAIN, dispatch: async () => ({}) })).rejects.toThrow('not a socket');
expect(readFileSync(path, 'utf8')).toBe('keep');
});
+ test('#7707: a refusal\'s structured fix survives the owner socket, so a delegated sources refresh prints the direct path\'s v1 fix', async () => {
+ // Value: protects=a refusal delegated to a resident owner keeps its structured fix (argv, actor, ask_user) instead of falling back to the
+ // prose suggestion; fails_when=remoteOperationError drops `fix`; why_new=test 4e reads the direct path only; seam=an owner whose
+ // administration throws the refusal; no_claim=a real resident refresh.
+ const path = socketPath();
+ const refusal = () => {
+ const error = new OperationError('refresh_diverged', 'The checkout HEAD is not an ancestor of origin/main.', 'Publish the local commits first.');
+ error.fix = { argv: ['gbrain', 'git', 'commit-push', '--push-only', '--keep-checkout', '--path', '/tmp/brain-example', '--json'], consent: [], actor: 'user',
+ ask_user: true, requires_exclusive: false, why: 'HEAD has a local commit the upstream lacks.', user_message: 'Push the local commit of source-a?',
+ docs: 'docs/guides/write-refusals.md#refresh_diverged' };
+ return error;
+ };
+ const result = await startPersistenceIpcServer(path, { brainId: BRAIN, dispatch: async () => ({}), administer: async () => { throw refusal(); } });
+ bindings.push(result!);
+ const delegated = await requestPersistenceAdministration(path, { version: 1, kind: 'administration', brain_id: BRAIN, operation: 'writer_refresh',
+ params: { source_id: 'source-a' }, registration: REGISTRATION }).then(() => null, error => error);
+ expect(delegated).toBeInstanceOf(OperationError);
+ const printed = (error: OperationError) => {
+ const out: string[] = [];
+ const log = console.log;
+ console.log = (line: string) => { out.push(line); };
+ try { printRefreshOutcome({ error }, true); } finally { console.log = log; _resetCliExitVerdictForTests(); process.exitCode = 0; }
+ return JSON.parse(out.join('\n'));
+ };
+ const direct = printed(refusal()).fix;
+ expect(direct).toMatchObject({ argv: refusal().fix!.argv, next: 'ask_user', actor: 'user' });
+ expect(printed(delegated as OperationError).fix).toEqual(direct);
+ });
});
diff --git a/test/persistence-worktree-refresh.test.ts b/test/persistence-worktree-refresh.test.ts
index 21d510a86a..207d8c63e4 100644
--- a/test/persistence-worktree-refresh.test.ts
+++ b/test/persistence-worktree-refresh.test.ts
@@ -35,7 +35,7 @@ import { runManagedSourceLifecycle } from '../src/core/persistence/source-lifecy
import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';
import { performManagedSync } from '../src/core/persistence/sync-run.ts';
import { readSyncStatus } from '../src/core/persistence/sync-status.ts';
-import { refreshWorktree, resumeWorktreeRefreshes } from '../src/core/persistence/worktree-refresh.ts';
+import { _refreshScanCacheSizeForTests, refreshLockPath, refreshWorktree, resumeWorktreeRefreshes } from '../src/core/persistence/worktree-refresh.ts';
import { commitAndPush } from '../src/core/git/commit-push.ts';
import { gitConvergenceCheck } from '../src/commands/doctor/checks/git-convergence.ts';
import { MANAGED_SYNC_PASS_OP, writePassRow, type ManagedSyncPassValue } from '../src/core/persistence/sync-pass-row.ts';
@@ -331,9 +331,12 @@ test('7. a crash leaving HEAD on an unrelated commit is recovery_required; write
// own (auto_recovery none, ask the user before resetting); fails_when=adoption drops its recovery_required guard, or the holder report treats
// it as an orphan the next refresh finishes; why_new=the #7679 tests cover syncing orphans only; seam=none; no_claim=the reset choice itself.
// The plain refresh refuses with the row's own code and stored fix (both resets), not refresh_in_progress: no process will finish it.
+ // #7708: a distinct holder pid shows the plain refresh never took the row over (adoption would record this process as its holder).
+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=jsonb_set(outcome,'{holder,pid}','535353'::jsonb) WHERE state='recovery_required'`);
const plain = await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_recovery_required');
expect(plain.suggestion).toContain(`reset --hard ${old}`);
expect((await f.refreshRows()).map(r => r.state)).toEqual(['recovery_required']);
+ expect((await f.refreshRows())[0].outcome).toMatchObject({ holder: { pid: 535353 } });
const [held] = (await runPersistenceAdministration(f.engine, 'writer_status', { source_id: f.alpha }) as { worktree_refreshes: Array<Record<string, any>> }).worktree_refreshes;
expect(held).toMatchObject({ state: 'recovery_required', auto_recovery: 'none', fix: { argv: ['gbrain', 'sources', 'refresh', f.alpha, '--resume'], actor: 'agent' } });
expect(held.fix.why).toContain('ask the user');
@@ -486,6 +489,192 @@ test('7g. #7679: a backlog of 16 orphaned syncing refreshes waiting on their mem
}
}), 180_000);
+/** Crash a refresh at `point` and restore the state the crash left (the resident consumer's owner scan may already have settled it). */
+async function orphanAt(f: RefreshFixture, point: 'drained' | 'fenced' | 'merged' | 'syncing', state: string) {
+ await expect(refreshWorktree(f.engine, f.alpha, { hooks: { boundary: at => { if (at === point) throw new Error(`simulated crash at ${at}`); } } }))
+ .rejects.toThrow('simulated crash');
+ await disposePersistenceConsumer(f.engine);
+ const [orphan] = await f.refreshRows();
+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET state=$2,completed_at=NULL WHERE id=$1::uuid`, [orphan.id, state]);
+ return orphan;
+}
+const setLastCommit = (f: RefreshFixture, member: string, sha: string) => f.engine.transaction(tx => withCoordinatedWrite(tx, [member],
+ () => tx.executeRaw('UPDATE sources SET last_commit=$2 WHERE id=$1', [member, sha]), TEST_WRITE_ATTRIBUTION));
+const writerStatusRow = async (f: RefreshFixture) => (await runPersistenceAdministration(f.engine, 'writer_status', { source_id: f.alpha }) as
+ { worktree_refreshes: Array<Record<string, any>> }).worktree_refreshes[0];
+
+test('7i. #7708: the owner scan skips syncing orphans whose members never left the start commit, so 16 of them do not hide one it can complete', () => each(async f => {
+ // Value: protects=the scan's SQL filter keeps syncing rows whose members still sit at old_head (or have no last_commit) out of its LIMIT 16;
+ // fails_when=the filter is dropped, so 16 older lagging rows fill the LIMIT and a completable orphan is never scanned; why_new=7g backlogs
+ // rows the ORDER BY puts after a drain, which never competes with syncing rows; seam=backlog rows inserted with SQL (no host binding);
+ // no_claim=16 real worktrees.
+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ const orphan = await orphanAt(f, 'syncing', 'syncing');
+ for (const member of [f.alpha, f.beta]) await setLastCommit(f, member, target);
+ const [{ owner_host_id: host, principal_id: principal }] = await f.engine.executeRaw<{ owner_host_id: string; principal_id: string }>(
+ `SELECT w.owner_host_id::text AS owner_host_id, r.principal_id::text AS principal_id FROM persistence_worktrees w
+ JOIN persistence_worktree_refreshes r ON r.worktree_id=w.id WHERE r.id=$1::uuid`, [orphan.id]);
+ const backlog: string[] = [];
+ try {
+ for (let i = 0; i < 16; i++) {
+ const [{ id }] = await f.engine.executeRaw<{ id: string }>('INSERT INTO persistence_worktrees(owner_host_id) VALUES($1::uuid) RETURNING id::text AS id', [host]);
+ backlog.push(id);
+ // The member's last_commit is this row's old_head: it never synced past the start commit.
+ await f.engine.executeRaw(`INSERT INTO persistence_worktree_refreshes(worktree_id,source_ids,principal_id,owner_epoch,topology_generation,state,old_head,target_head,upstream_ref,created_at)
+ VALUES($1::uuid,ARRAY[$2]::text[],$3::uuid,0,1,'syncing',$4,$5,'origin/main',now()-interval '1 hour')`, [id, f.alpha, principal, target, 'f'.repeat(40)]);
+ }
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(1);
+ expect(await f.refreshRows()).toMatchObject([{ id: orphan.id, state: 'completed', outcome: { completed_by: 'owner_recovery' } }]);
+ } finally {
+ if (backlog.length) {
+ await f.engine.executeRaw('DELETE FROM persistence_worktree_refreshes WHERE worktree_id=ANY($1::uuid[])', [backlog]);
+ await f.engine.executeRaw('DELETE FROM persistence_worktrees WHERE id=ANY($1::uuid[])', [backlog]);
+ }
+ }
+}), 180_000);
+
+test('7j. #7712: a lagging syncing orphan is checked without its refresh lock, and an unreadable lock never stops the scan', () => each(async f => {
+ // Value: protects=the owner scan reads member progress before it takes the refresh lock, so a lagging row never holds it (a refresh or
+ // liveness probe landing then would see it busy), and a lock it cannot open fails that row only; fails_when=the scan takes the lock before
+ // checking members, or lets the lock error escape the scan; why_new=7e checks the verdict, not the lock; seam=the lock file made unreadable
+ // (chmod 000); no_claim=the race window itself.
+ const first = f.push('alpha/two.md', page('Alpha two', 'Upstream first.'));
+ const target = f.push('alpha/three.md', page('Alpha three', 'Upstream second.'));
+ const orphan = await orphanAt(f, 'syncing', 'syncing');
+ await setLastCommit(f, f.alpha, target);
+ await setLastCommit(f, f.beta, first);
+ const lockFile = refreshLockPath(f.worktreeId);
+ mkdirSync(join(lockFile, '..'), { recursive: true });
+ if (!existsSync(lockFile)) writeFileSync(lockFile, '');
+ chmodSync(lockFile, 0o000);
+ const errors: string[] = [];
+ const consoleError = console.error;
+ console.error = (...args: unknown[]) => { errors.push(args.join(' ')); };
+ try {
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ await setLastCommit(f, f.beta, target);
+ // Every member reached the target, so this scan needs the lock: it cannot open it, says so, and returns normally.
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ } finally { console.error = consoleError; chmodSync(lockFile, 0o644); }
+ expect(errors.filter(line => line.includes(orphan.id))).toHaveLength(1);
+ expect((await f.refreshRows()).map(r => r.state)).toEqual(['syncing']);
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(1);
+ expect((await f.refreshRows()).map(r => r.state)).toEqual(['completed']);
+}), 180_000);
+
+test('7k. #7728: a refresh lock this host cannot read reports liveness unknown with a host_admin fix, and a plain refresh does not adopt the orphan', () => each(async f => {
+ // Value: protects=refreshLiveness's unreadable-lock branch: writer status says unknown (not orphaned) and names the lock file, and the plain
+ // refresh refuses refresh_in_progress instead of adopting or failing with a raw lock error; fails_when=the probe's catch is dropped, or
+ // adoption lets the lock error escape; why_new=7d reaches 'unknown' only through a holder on another host; seam=the lock file made
+ // unreadable (chmod 000); no_claim=other ways a lock read fails.
+ f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ const orphan = await orphanAt(f, 'fenced', 'fenced');
+ const before = JSON.stringify(await f.refreshRows());
+ const lockFile = refreshLockPath(f.worktreeId);
+ mkdirSync(join(lockFile, '..'), { recursive: true });
+ if (!existsSync(lockFile)) writeFileSync(lockFile, '');
+ chmodSync(lockFile, 0o000);
+ try {
+ const status = await writerStatusRow(f);
+ expect(status).toMatchObject({ refresh_id: orphan.id, liveness: 'unknown', orphaned: null, auto_recovery: 'unknown',
+ fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', f.alpha, '--json'], actor: 'host_admin' } });
+ expect(status.fix.why).toContain(lockFile);
+ await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_in_progress');
+ } finally { chmodSync(lockFile, 0o644); }
+ expect(JSON.stringify(await f.refreshRows())).toBe(before);
+}), 180_000);
+
+test('7l. #7730: the owner scan forgets a refresh it no longer lists', () => each(async f => {
+ // Value: protects=a long-lived owner keeps no cache entry for a refresh that settled; fails_when=the scan stops pruning ids absent from its
+ // rows; why_new=no test read the cache; seam=a test-only size read; no_claim=memory growth over a real process lifetime.
+ const first = f.push('alpha/two.md', page('Alpha two', 'Upstream first.'));
+ const target = f.push('alpha/three.md', page('Alpha three', 'Upstream second.'));
+ await orphanAt(f, 'syncing', 'syncing');
+ await setLastCommit(f, f.alpha, target);
+ await setLastCommit(f, f.beta, first);
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ expect(_refreshScanCacheSizeForTests()).toBe(1);
+ await setLastCommit(f, f.beta, target);
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(1);
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ expect(_refreshScanCacheSizeForTests()).toBe(0);
+}), 180_000);
+
+test('7m. #7731: a syncing orphan whose HEAD is unrelated is re-checked after a minute, not on every owner scan tick', () => each(async f => {
+ // Value: protects=the owner scan caches a syncing orphan it could not complete because of its HEAD, as it does a lagging one, so it does not
+ // retake the refresh and topology locks every tick; fails_when=only lagging members are cached; why_new=7h caches a member verdict only;
+ // seam=HEAD reset with git, the clock moved with setSystemTime; no_claim=the lock contention itself.
+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ const orphan = await orphanAt(f, 'syncing', 'syncing');
+ for (const member of [f.alpha, f.beta]) await setLastCommit(f, member, target);
+ const old = git(f.root, 'rev-parse', 'HEAD~1');
+ git(f.root, 'reset', '-q', '--hard', old);
+ commitFile(f.root, 'beta/unrelated.md', page('Unrelated', 'Not the upstream commit.'));
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ git(f.root, 'reset', '-q', '--hard', target);
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ expect((await f.refreshRows()).map(r => r.state)).toEqual(['syncing']);
+ setSystemTime(new Date(Date.now() + 61_000));
+ try { expect(await resumeWorktreeRefreshes(f.engine)).toBe(1); } finally { setSystemTime(); }
+ expect(await f.refreshRows()).toMatchObject([{ id: orphan.id, state: 'completed' }]);
+}), 180_000);
+
+test('7n. #7731: an orphan whose checkout HEAD git cannot read says so in writer status and in one owner-scan log line, not on every tick', () => each(async f => {
+ // Value: protects=writer status does not promise that the next refresh finishes an orphan whose HEAD is unreadable (it names git status),
+ // and the owner scan logs it once instead of swallowing it on every tick; fails_when=the holder report ignores HEAD for orphans, or the scan
+ // drops the log or its rate limit; why_new=7d checks that nothing moves, not what status or the log says; seam=.git/HEAD overwritten with
+ // garbage; no_claim=the 15-minute repeat.
+ f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ const orphan = await orphanAt(f, 'fenced', 'fenced');
+ const headFile = join(f.root, '.git', 'HEAD');
+ const saved = readFileSync(headFile, 'utf8');
+ writeFileSync(headFile, 'not a ref\n');
+ const errors: string[] = [];
+ const consoleError = console.error;
+ console.error = (...args: unknown[]) => { errors.push(args.join(' ')); };
+ try {
+ expect(await writerStatusRow(f)).toMatchObject({ refresh_id: orphan.id, liveness: 'orphaned', auto_recovery: 'none',
+ fix: { argv: ['git', '-C', f.root, 'status'] }, next: `git -C ${f.root} status` });
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);
+ } finally { console.error = consoleError; writeFileSync(headFile, saved); }
+ expect(errors.filter(line => line.includes(orphan.id))).toEqual([expect.stringContaining('cannot read the checkout HEAD')]);
+ expect(await writerStatusRow(f)).toMatchObject({ refresh_id: orphan.id, auto_recovery: 'next_refresh' });
+}), 180_000);
+
+test('7o. #7712: a plain refresh that rolls back an orphan and then refuses keeps that recovery in its refusal', () => each(async f => {
+ // Value: protects=the rollback a plain refresh made before its own refresh refused is still reported (a cron log otherwise shows only the
+ // refusal); fails_when=refreshWorktree attaches the recovery to successful results only; why_new=7b and 7f succeed after adopting;
+ // seam=an uncommitted edit of the incoming path; no_claim=other refusals (they take the same path).
+ f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ const orphan = await orphanAt(f, 'fenced', 'fenced');
+ writeFileSync(join(f.root, 'alpha/two.md'), page('Alpha two', 'Edited locally, not committed.'));
+ const refused = await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_dirty');
+ expect(refused.message).toContain(`finished orphaned refresh ${orphan.id} (abandoned; previous holder: pid ${process.pid}`);
+ expect((await f.refreshRows()).map(r => r.state)).toEqual(['aborted']);
+}), 180_000);
+
+test('7p. #7712: writer status for a recovery_required refresh names the step HEAD allows: --resume at the target, --abandon at the start commit', () => each(async f => {
+ // Value: protects=the fix of a recovery_required row follows the checkout HEAD: at the target --resume converges it with nothing to reset, at
+ // the start commit --abandon lifts the fence, and only another HEAD asks the user to reset; fails_when=the fix text stays unconditional;
+ // why_new=test 7 reads it at an unrelated HEAD only; seam=the row's state set with SQL after a crash at merged; no_claim=none.
+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
+ const old = f.head();
+ const orphan = await orphanAt(f, 'merged', 'recovery_required');
+ expect(f.head()).toBe(target);
+ const atTarget = await writerStatusRow(f);
+ expect(atTarget).toMatchObject({ refresh_id: orphan.id, state: 'recovery_required', auto_recovery: 'none', fix: { argv: ['gbrain', 'sources', 'refresh', f.alpha, '--resume'] } });
+ expect(atTarget.fix.why).toContain('nothing needs resetting');
+ expect(atTarget.fix.why).not.toContain('ask the user');
+ git(f.root, 'reset', '-q', '--hard', old);
+ const atOld = await writerStatusRow(f);
+ expect(atOld).toMatchObject({ fix: { argv: ['gbrain', 'sources', 'refresh', f.alpha, '--abandon'] }, next: `gbrain sources refresh ${f.alpha} --abandon` });
+ expect((await refreshWorktree(f.engine, f.alpha, { abandon: true })).status).toBe('abandoned');
+ git(f.root, 'merge', '-q', '--ff-only', target);
+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET state='recovery_required',completed_at=NULL WHERE id=$1::uuid`, [orphan.id]);
+ expect(await refreshWorktree(f.engine, f.alpha, { resume: true })).toMatchObject({ refresh_id: orphan.id, status: 'completed' });
+}), 180_000);
+
test('8. a queued git effect drains before fenced, and nothing on the worktree is claimed while fenced', () => each(async f => {
f.push('alpha/two.md', page('Alpha two', 'Upstream.'));
await f.put(f.beta, 'notes/effect', page('Effect', 'Has a git effect.'));
diff --git a/test/worktree-refresh-settle-matrix-7679.test.ts b/test/worktree-refresh-settle-matrix-7679.test.ts
index 6a680ed8e1..24e498a3f0 100644
--- a/test/worktree-refresh-settle-matrix-7679.test.ts
+++ b/test/worktree-refresh-settle-matrix-7679.test.ts
@@ -42,25 +42,30 @@ const refusedWith = async (promise: Promise<unknown>, code: string) => {
return error!;
};
-test('7c. #7679 state matrix: an interrupted refresh of each kind, settled by the owner scan, a plain refresh and --resume', async () => {
+test('7c. #7679 state matrix: an interrupted refresh of each kind, settled by the owner scan, a plain refresh, --resume and --abandon', async () => {
// Value: protects=each kind of interrupted refresh row leaves the row section 1.5 names when the owner scan, a plain refresh (adoption) or
// --resume (resumeHeld) settles it; fails_when=a kind falls through to a sibling's path (an unverified HEAD adopted as the target, a syncing
// row with an unrelated HEAD completed or refreshed over, a dead drain left standing); why_new=the #7679 tests cover syncing orphans with HEAD at
// the target only; seam=none; no_claim=a real process death (the crash is a hook throw; the refresh lock is released as on exit).
+ // #7718-#7727, #7729: --abandon rolls back only a row whose checkout never moved (a dead drain, a fence at the start commit) and admits writes
+ // again; a row whose HEAD reached the target refuses refresh_in_progress and stays as it is, and an unrelated HEAD escalates to
+ // recovery_required; fails_when=abandon drops its syncing or at-target guard (files left ahead of the database) or aborts over an unrelated HEAD.
type Kind = 'draining' | 'fenced_old' | 'fenced_target' | 'fenced_unrelated' | 'merged_target' | 'merged_unrelated' | 'syncing_at_target' | 'syncing_descendant'
- | 'syncing_unrelated' | 'syncing_unrelated_synced';
- type Action = 'owner_scan' | 'plain' | 'resume';
+ | 'syncing_unrelated' | 'syncing_unrelated_synced' | 'syncing_lagging';
+ type Action = 'owner_scan' | 'plain' | 'resume' | 'abandon';
const kinds: Kind[] = ['draining', 'fenced_old', 'fenced_target', 'fenced_unrelated', 'merged_target', 'merged_unrelated', 'syncing_at_target', 'syncing_descendant',
'syncing_unrelated', 'syncing_unrelated_synced'];
const stateOf = (kind: Kind) => kind.split('_')[0] as 'draining' | 'fenced' | 'merged' | 'syncing';
const crashAt = { draining: 'drained', fenced: 'fenced', merged: 'fenced', syncing: 'syncing' } as const;
// The owner scan's row and moved count; a merged row's bookkeeping leaves it syncing, and a syncing row with an unrelated HEAD waits for --resume.
const scanned: Record<Kind, [string, number]> = { draining: ['aborted', 1], fenced_old: ['aborted', 1], fenced_target: ['syncing', 1], fenced_unrelated: ['recovery_required', 1],
- merged_target: ['syncing', 1], merged_unrelated: ['recovery_required', 1], syncing_at_target: ['completed', 1], syncing_descendant: ['syncing', 0], syncing_unrelated: ['syncing', 0], syncing_unrelated_synced: ['syncing', 0] };
- const cells = kinds.flatMap(kind => (['owner_scan', 'plain', 'resume'] as Action[]).map(action => [kind, action] as [Kind, Action]));
+ merged_target: ['syncing', 1], merged_unrelated: ['recovery_required', 1], syncing_at_target: ['completed', 1], syncing_descendant: ['syncing', 0], syncing_unrelated: ['syncing', 0], syncing_unrelated_synced: ['syncing', 0], syncing_lagging: ['syncing', 0] };
+ // syncing_lagging (HEAD at the target, members still at the start commit) is what the crash at syncing leaves; the other paths cover it in 7e/7d.
+ const cells = [...kinds.flatMap(kind => (['owner_scan', 'plain', 'resume', 'abandon'] as Action[]).map(action => [kind, action] as [Kind, Action])),
+ ['syncing_lagging', 'abandon'] as [Kind, Action]];
// Test cost (#7782): a cell that leaves the checkout settled (no unfinished refresh, a HEAD the upstream contains, every member synced to it) hands
// its fixture to the next cell instead of building a brain per cell, so each chain is at most one settled cell and then one unsettled cell.
- const endsSettled = ([kind, action]: [Kind, Action]) => !kind.includes('_unrelated') && kind !== 'syncing_descendant'
+ const endsSettled = ([kind, action]: [Kind, Action]) => action === 'abandon' ? kind === 'fenced_old' : !kind.includes('_unrelated') && kind !== 'syncing_descendant'
&& !(action === 'owner_scan' && (kind === 'fenced_target' || kind === 'merged_target'));
const settledCells = cells.filter(endsSettled), unsettledCells = cells.filter(cell => !endsSettled(cell));
const chains = [...unsettledCells.map((cell, i) => settledCells[i] ? [settledCells[i], cell] : [cell]), ...settledCells.slice(unsettledCells.length).map(cell => [cell])];
@@ -104,6 +109,27 @@ test('7c. #7679 state matrix: an interrupted refresh of each kind, settled by th
expect(f.head(), label).toBe(head);
continue;
}
+ if (action === 'abandon') {
+ const members = [await f.lastCommit(f.alpha), await f.lastCommit(f.beta)];
+ const run = refreshWorktree(f.engine, f.alpha, { abandon: true });
+ if (kind === 'draining' || kind === 'fenced_old') {
+ expect(await run, label).toMatchObject({ status: 'abandoned', refresh_id: orphan.id, old_head: old, target_head: target });
+ expect(await rows(), label).toEqual(['aborted']);
+ expect(((await cellRows())[0].outcome.refusal as { code: string }).code, label).toBe('refresh_abandoned');
+ } else if (kind === 'fenced_unrelated' || kind === 'merged_unrelated') {
+ const refused = await refusedWith(run, 'refresh_recovery_required');
+ expect(refused.message, label).toContain(head);
+ expect(await rows(), label).toEqual(['recovery_required']);
+ } else {
+ const refused = await refusedWith(run, 'refresh_in_progress');
+ expect(refused.suggestion, label).toContain(`gbrain sources refresh ${f.alpha} --resume`);
+ expect(await rows(), label).toEqual([stateOf(kind)]);
+ }
+ expect(f.head(), label).toBe(head);
+ expect([await f.lastCommit(f.alpha), await f.lastCommit(f.beta)], label).toEqual(members);
+ if (kind === 'draining') expect((await f.put(f.alpha, 'notes/after-abandon', page('After', 'admitted again'))).state ?? 'committed', label).toBe('committed');
+ continue;
+ }
const run = refreshWorktree(f.engine, f.alpha, action === 'resume' ? { resume: true } : {});
if (kind.includes('_unrelated')) {
await refusedWith(run, 'refresh_recovery_required');
diff --git a/test/worktree-refresh-unmoved-matrix-7679.test.ts b/test/worktree-refresh-unmoved-matrix-7679.test.ts
index f0138433fb..a095c0923a 100644
--- a/test/worktree-refresh-unmoved-matrix-7679.test.ts
+++ b/test/worktree-refresh-unmoved-matrix-7679.test.ts
@@ -7,6 +7,7 @@
import { afterAll, beforeAll, expect, test } from 'bun:test';
import { randomUUID } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
+import { hostname } from 'node:os';
import { join } from 'node:path';
import type { BrainEngine } from '../src/core/engine.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
@@ -108,10 +109,12 @@ test('7d. #7679 state matrix: an interrupted refresh that the unattended paths c
} finally { await lock!.release(); }
} else if (action === 'other_host_holder') {
const [{ outcome }] = await f.refreshRows();
- await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=jsonb_set(outcome,'{holder,host_id}',to_jsonb($2::text)) WHERE id=$1::uuid`, [orphan.id, randomUUID()]);
+ const foreignHost = randomUUID();
+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=jsonb_set(outcome,'{holder,host_id}',to_jsonb($2::text)) WHERE id=$1::uuid`, [orphan.id, foreignHost]);
await refusedWith(refreshWorktree(f.engine, f.alpha), plainCode);
+ // #7731: next names the host the command must run on.
if (kind !== 'recovery_required') expect(await writerStatus(), label).toMatchObject({ liveness: 'unknown', auto_recovery: 'unknown', fix: { actor: 'host_admin' },
- next: `gbrain sources writer status --source ${f.alpha} --json` });
+ next: `On host ${hostname()} (${foreignHost}), which holds the refresh lock: gbrain sources writer status --source ${f.alpha} --json` });
await f.engine.executeRaw('UPDATE persistence_worktree_refreshes SET outcome=$2::text::jsonb WHERE id=$1::uuid', [orphan.id, JSON.stringify(outcome)]);
} else {
const headFile = join(f.root, '.git', 'HEAD');