Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pass 2 review fixes (--abandon/--resume kept in the internal_error fix; the post-drain abort is bounded by its own lock timeout)

garrytana13d888ce56c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 517d04fe02..7d9fcb5bcb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,10 +18,10 @@ Before, a stuck transaction holding a source, worktree or brain row (a long-runn
### Itemized changes
-- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists up to five open transactions in the current database holding row-level locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open); the blocking transaction is among them. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a failed lookup says so. A refusal after the drain aborts the draining row as other fence refusals do.
-- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, SQLSTATE and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
+- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists up to five open transactions in the current database holding row-level locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open); the blocking transaction is among them. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a failed lookup says so. A refusal after the drain aborts the draining row as other fence refusals do; that abort has its own 5 s lock timeout, and when the same holder blocks it the row stays `draining` for the next run instead of the refresh hanging.
+- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, SQLSTATE and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read. Under `--abandon` or `--resume` the fix repeats that flag. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
- `src/commands/sources-refresh.ts`: refusals print `why` (a `why` key under `--json`, a `Why:` line otherwise).
-- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, and credentials and the home directory are redacted).
+- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, credentials and the home directory are redacted, `--abandon` survives into the fix, and an abort blocked by the same holder still returns).
## [0.60.529.0] - 2026-10-31
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index d68a566d09..7aeba1987a 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -488,7 +488,7 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt
try { return await runRefresh(engine, sourceId, opts); }
catch (error) {
if (error instanceof OperationError || classifyPgAccessError(error).reason !== 'unknown') throw error;
- throw await unexpectedRefreshFailure(engine, sourceId, error);
+ throw await unexpectedRefreshFailure(engine, sourceId, opts, error);
}
}
@@ -503,17 +503,18 @@ const redact = (text: string) => {
* A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so the CLI
* keeps its own repair fix for it.
*/
-async function unexpectedRefreshFailure(engine: BrainEngine, sourceId: string, error: unknown): Promise<OperationError> {
+async function unexpectedRefreshFailure(engine: BrainEngine, sourceId: string, opts: RefreshOptions, error: unknown): Promise<OperationError> {
const thrown = error instanceof Error ? error : new Error(String(error));
const sqlstate = typeof (error as { code?: unknown } | null)?.code === 'string' ? ` (${(error as { code: string }).code})` : '';
const lookup = await Promise.resolve().then(async () => {
const binding = await getWorktreeBinding(engine, sourceId, existingLocalHostId());
return { active: binding ? await readActiveRefresh(engine, binding.worktree_id) : null };
}).catch(() => null);
+ const command = `gbrain sources refresh ${sourceId}${opts.abandon ? ' --abandon' : opts.resume ? ' --resume' : ''}`;
const left = !lookup ? 'The refresh row could not be read, so a fenced or syncing refresh may still be active.'
- : lookup.active ? `Refresh ${lookup.active.id} stays ${lookup.active.state}, and the next gbrain sources refresh ${sourceId} resumes it.` : 'No refresh row was left active.';
+ : lookup.active ? `Refresh ${lookup.active.id} stays ${lookup.active.state}${opts.abandon ? '.' : `, and the next gbrain sources refresh ${sourceId} resumes it.`}` : 'No refresh row was left active.';
return opError('internal_error', redact(`The refresh of source ${sourceId} failed unexpectedly: ${thrown.name}${sqlstate}: ${thrown.message}`),
- `${left} Run gbrain sources refresh ${sourceId} again; if it fails the same way, report this output (why holds the stack) with gbrain doctor --json.`,
+ `${left} Run ${command} again; if it fails the same way, report this output (why holds the stack) with gbrain doctor --json.`,
{ why: redact(thrown.stack ?? `${thrown.name}: ${thrown.message}`) });
}
@@ -674,7 +675,10 @@ async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, pl
} catch (error) {
const stored = await readRefresh(engine, row.id);
if (stored.state === 'draining' && error instanceof OperationError) {
- await engine.transaction(tx => transition(tx, stored, ['draining'], 'aborted', refusalOutcome(error)));
+ await engine.transaction(async tx => {
+ await tx.executeRaw("SELECT set_config('lock_timeout','5s',true)");
+ await transition(tx, stored, ['draining'], 'aborted', refusalOutcome(error));
+ }).catch(abortError => { if ((abortError as { code?: string }).code !== LOCK_NOT_AVAILABLE) throw abortError; });
error.detail ??= `refresh_id=${row.id}`;
}
throw error;
diff --git a/test/worktree-refresh-unexpected-throw.test.ts b/test/worktree-refresh-unexpected-throw.test.ts
index d7b6b37ff0..6d41f78779 100644
--- a/test/worktree-refresh-unexpected-throw.test.ts
+++ b/test/worktree-refresh-unexpected-throw.test.ts
@@ -98,6 +98,14 @@ test('an unexpected throw whose refresh row cannot be read says so, with credent
expect(failure.why).toContain('~/brain-checkout/src/x.ts');
});
+test('an unexpected throw under --abandon repeats --abandon in its fix, never a plain refresh that would finish the refresh', async () => {
+ const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });
+ const fake = { kind: 'postgres', getConfig: async () => { throw statement; }, executeRaw: async () => { throw statement; } } as unknown as BrainEngine;
+ const failure = await withEnv({ GBRAIN_REFRESH_SYNC_BUDGET_MS: undefined }, () => refreshWorktree(fake, 'source-a', { abandon: true }).then(() => null, error => error));
+ expect(failure.code).toBe('internal_error');
+ expect(failure.suggestion).toContain('Run gbrain sources refresh source-a --abandon again');
+});
+
test('a host identity file that cannot be read still yields a typed internal_error refusal', () => withRefreshFixture(engine, async f => {
const hostFile = join(persistenceHome(), 'host.json');
const original = readFileSync(hostFile, 'utf8');
@@ -113,10 +121,11 @@ test('a host identity file that cannot be read still yields a typed internal_err
/**
* Another connection holds one source row until `release` (a stuck transaction on the brain host): `FOR UPDATE` by
- * default, or a plain read (AccessShareLock only, which cannot block the fence) with `plainRead`. Its last statement
+ * default, or a plain read (AccessShareLock only, which cannot block the fence) with `plainRead`; with `refreshId` it also
+ * holds that refresh row. Its last statement
* carries a marker literal, which the refusal must never echo.
*/
-async function holdSourceRow(sourceId: string, plainRead = false): Promise<{ pid: number; release: () => Promise<void> }> {
+async function holdSourceRow(sourceId: string, plainRead = false, refreshId?: string): Promise<{ pid: number; release: () => Promise<void> }> {
const sql = postgres(databaseUrl, { max: 1, prepare: false });
let release!: () => void;
const released = new Promise<void>(resolve => { release = resolve; });
@@ -126,6 +135,7 @@ async function holdSourceRow(sourceId: string, plainRead = false): Promise<{ pid
const [row] = plainRead
? await t`SELECT pg_backend_pid() AS pid FROM sources WHERE id=${sourceId}`
: await t`SELECT pg_backend_pid() AS pid FROM sources WHERE id=${sourceId} FOR UPDATE`;
+ if (refreshId) await t`SELECT id FROM persistence_worktree_refreshes WHERE id=${refreshId}::uuid FOR UPDATE`;
await t.unsafe("SELECT 'holder-sql-marker' AS note");
locked(Number(row!.pid));
await released;
@@ -189,3 +199,18 @@ test.skipIf(!process.env.DATABASE_URL)('the lock-timeout why names only row-lock
expect(failure!.why).not.toContain('holder-sql-marker');
expect(failure!.why).not.toContain('SELECT');
}), 120_000);
+
+test.skipIf(!process.env.DATABASE_URL)('a lock timeout after the drain whose holder also holds the refresh row still refuses, leaving the row draining for the next run', () => withRefreshFixture(engine, async f => {
+ const before = f.head();
+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));
+ let held: Awaited<ReturnType<typeof holdSourceRow>> | undefined;
+ const failure = await refreshWorktree(f.engine, f.alpha, { hooks: { boundary: async (point, row) => { if (point === 'drained') held = await holdSourceRow(f.alpha, false, row.id); } } })
+ .then(() => null, error => error).finally(() => held?.release());
+ expect(failure).toBeInstanceOf(OperationError);
+ expect(failure.code).toBe('refresh_lock_timeout');
+ const [row] = await f.refreshRows();
+ expect(row!.state).toBe('draining');
+ expect(f.head()).toBe(before);
+ expect((await refreshWorktree(f.engine, f.alpha)).status).toBe('completed');
+ expect(await f.lastCommit(f.alpha)).toBe(target);
+}), 60_000);