Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(sources refresh): look up lock holders after the topology lock is released, keep --wait-drain/--fetch-timeout-ms in the retry, mark 55P03 retryable (GBRA-250 voice findings)
garrytane7732d2165b9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3af839b896..11980cc04b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,8 +18,8 @@ Before, a stuck transaction holding a source, worktree or brain row (a long-runn### Itemized changes-- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a lookup that fails or takes longer than 3 s says so (the statement is cancelled at that bound, so the refusal never holds the topology lock waiting on it). A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.-- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s; its statements are cancelled at that bound). A deadlock (`40P01`), serialization failure (`40001`) or cancelled statement (`57014`) is marked retryable. The fix repeats `--dry-run`, `--abandon` or `--resume` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.+- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a lookup that fails or takes longer than 3 s says so (the statement is cancelled at that bound, and after the drain the lookup runs only once the topology and worktree locks are released). A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.+- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s; its statements are cancelled at that bound). A deadlock (`40P01`), serialization failure (`40001`), cancelled statement (`57014`) or lock timeout outside the fence (`55P03`) is marked retryable. The fix repeats `--dry-run`, `--abandon`, `--resume`, `--wait-drain` and `--fetch-timeout-ms` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.- `src/commands/sources-refresh.ts`: refusals print `why` (a `why` key under `--json`, a `Why:` line otherwise).- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, credentials and the home directory are redacted without touching a sibling path, `--abandon` and `--dry-run` survive into the fix, a lookup that never answers still gives the refusal and is cancelled (row lookup and lock-holder lookup), transient SQLSTATEs are retryable, the holder list is worded as candidates, and an abort blocked by the same holder still returns and says the row stays draining).diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.tsindex bf8d0561b8..8bbb8d6c9c 100644--- a/src/core/persistence/worktree-refresh.ts+++ b/src/core/persistence/worktree-refresh.ts@@ -517,7 +517,7 @@ async function boundedLookup<T>(engine: BrainEngine, read: (engine: BrainEngine)const timedOut = new Promise<null>(resolve => { timer = setTimeout(() => { controller.abort(); resolve(null); }, LOOKUP_MS); });return Promise.race([Promise.resolve().then(() => read(bounded)), timedOut]).catch(() => null).finally(() => clearTimeout(timer));}-const TRANSIENT_SQLSTATES = new Set(['40P01', '40001', '57014']);+const TRANSIENT_SQLSTATES = new Set(['40P01', '40001', '57014', '55P03']);/*** GBRA-250: an untyped throw (a statement error, a TypeError) leaves the refresh where it stood, so a fenced or syncing row@@ -532,7 +532,8 @@ async function unexpectedRefreshFailure(engine: BrainEngine, sourceId: string, oconst binding = await getWorktreeBinding(reads, sourceId, existingLocalHostId());return { active: binding ? await readActiveRefresh(reads, binding.worktree_id) : null };});- const command = `gbrain sources refresh ${sourceId}${opts.dryRun ? ' --dry-run' : ''}${opts.abandon ? ' --abandon' : opts.resume ? ' --resume' : ''}`;+ const command = `gbrain sources refresh ${sourceId}${opts.dryRun ? ' --dry-run' : ''}${opts.abandon ? ' --abandon' : opts.resume ? ' --resume' : ''}`+ + `${opts.waitDrainMs !== undefined ? ` --wait-drain ${opts.waitDrainMs / 1000}` : ''}${opts.fetchTimeoutMs !== undefined ? ` --fetch-timeout-ms ${opts.fetchTimeoutMs}` : ''}`;const left = !lookup ? 'The refresh row could not be read, so a fenced or syncing refresh may still be active.': lookup.active ? `Refresh ${lookup.active.id} stays ${lookup.active.state}${opts.abandon ? '.' : `, and the next gbrain sources refresh ${sourceId} resumes it.`}` : 'No refresh row was left active.';const failure = opError('internal_error', redact(`The refresh of source ${sourceId} failed unexpectedly: ${thrown.name}${code ? ` (${code})` : ''}: ${thrown.message}`),@@ -656,6 +657,9 @@ async function lockTimeoutRefusal(engine: BrainEngine, sourceId: string, refreshreturn error;}+/** A fence transaction past its lock timeout; fenceAndMerge builds the refusal after the native locks are released. */+class FenceLockTimeout extends Error {}+/** Steps 3-5 under the worktree native lock: recheck drained, fence, verify HEAD, ff-only merge, bookkeeping. */async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, plan: Plan, row: WorktreeRefreshRow, hooks?: RefreshHooks): Promise<WorktreeRefreshRow> {let fenced: WorktreeRefreshRow;@@ -674,7 +678,7 @@ async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, pl`Work reached the worktree after the drain check (${counts.requests} request(s), ${counts.effects} checkout effect(s), ${counts.recovery} recovery record(s)); nothing moved.`,`Retry gbrain sources refresh ${sourceId}`, row.id);if (!await transition(tx, row, ['draining'], 'fenced')) throw refusal('refresh_source_changed', 'The refresh record was moved by recovery while draining.', `gbrain sources refresh ${sourceId} --resume`, row.id);- }).catch(async error => { throw (error as { code?: string }).code === LOCK_NOT_AVAILABLE ? await lockTimeoutRefusal(engine, sourceId, row.id) : error; });+ }).catch(error => { throw (error as { code?: string }).code === LOCK_NOT_AVAILABLE ? new FenceLockTimeout() : error; });const current = await readRefresh(engine, row.id);await hooks?.boundary?.('fenced', current);const head = await headOf(git, plan.root);@@ -697,7 +701,8 @@ async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, plawait bookkeeping(engine, current);return readRefresh(engine, row.id);}, undefined, 30_000);- } catch (error) {+ } catch (thrown) {+ const error = thrown instanceof FenceLockTimeout ? await lockTimeoutRefusal(engine, sourceId, row.id) : thrown;const stored = await readRefresh(engine, row.id);if (stored.state === 'draining' && error instanceof OperationError) {await engine.transaction(async tx => {diff --git a/test/worktree-refresh-unexpected-throw.test.ts b/test/worktree-refresh-unexpected-throw.test.tsindex 820b347935..cf6f7fea59 100644--- a/test/worktree-refresh-unexpected-throw.test.ts+++ b/test/worktree-refresh-unexpected-throw.test.ts@@ -21,6 +21,7 @@ import { runSourcesRefresh } from '../src/commands/sources-refresh.ts';import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';import { refreshWorktree } from '../src/core/persistence/worktree-refresh.ts';import { persistenceHome } from '../src/core/persistence/identity.ts';+import { tryAcquireNativeLock } from '../src/core/persistence/native-lock.ts';import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';import { page, withRefreshFixture } from './helpers/worktree-refresh-fixture.ts';import { withEnv } from './helpers/with-env.ts';@@ -114,6 +115,13 @@ test('an unexpected throw under --dry-run repeats --dry-run in its fix, never aexpect(failure.suggestion).toContain('Run gbrain sources refresh source-a --dry-run again');});+test('an unexpected throw repeats --wait-drain and --fetch-timeout-ms in its fix', async () => {+ const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });+ const fake = { kind: 'postgres', getConfig: async () => { throw statement; }, executeRaw: async () => { throw statement; } } as unknown as BrainEngine;+ const failure = await withEnv({ GBRAIN_REFRESH_SYNC_BUDGET_MS: undefined }, () => refreshWorktree(fake, 'source-a', { waitDrainMs: 120_000, fetchTimeoutMs: 5_000 }).then(() => null, error => error));+ expect(failure.suggestion).toContain('Run gbrain sources refresh source-a --wait-drain 120 --fetch-timeout-ms 5000 again');+});+test('a refresh-row lookup that never answers still yields the refusal, and the stalled read is bounded and cancelled', async () => {const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });const reads: Array<{ signal?: AbortSignal; timeoutMs?: number }> = [];@@ -135,6 +143,7 @@ test('a deadlock or serialization failure is a retryable internal_error; an ordi};expect((await refuse('40P01')).retryable).toBe(true);expect((await refuse('40001')).retryable).toBe(true);+ expect((await refuse('55P03')).retryable).toBe(true);expect((await refuse('42883')).retryable).toBeUndefined();});@@ -263,6 +272,27 @@ test.skipIf(!process.env.DATABASE_URL)('a lock-holder lookup that never answersexpect(stalled?.aborted).toBe(true);}), 60_000);+test.skipIf(!process.env.DATABASE_URL)('a lock timeout after the drain looks up the holders only after the topology lock is released', () => withRefreshFixture(engine, async f => {+ f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));+ const target = f.engine as unknown as { executeRaw: (this: unknown, sql: string, params?: unknown[], opts?: object) => Promise<Array<Record<string, string>>> };+ const original = target.executeRaw;+ const [brain] = await original.call(f.engine, 'SELECT brain_id FROM persistence_brain WHERE singleton=1');+ let topologyFree: boolean | undefined;+ target.executeRaw = async function (sql, params, opts) {+ if (sql.includes('a.xact_start IS NOT NULL') && topologyFree === undefined) {+ const handle = await tryAcquireNativeLock(join(persistenceHome(), 'locks', `topology-${brain!.brain_id}.lock`));+ topologyFree = handle !== null;+ await handle?.release();+ }+ return original.call(this, sql, params, opts);+ };+ let held: Awaited<ReturnType<typeof holdSourceRow>> | undefined;+ const failure = await refreshWorktree(f.engine, f.alpha, { hooks: { boundary: async point => { if (point === 'drained') held = await holdSourceRow(f.alpha); } } })+ .then(() => null, error => error).finally(async () => { target.executeRaw = original; await held?.release(); });+ expect(failure.code).toBe('refresh_lock_timeout');+ expect(topologyFree).toBe(true);+}), 120_000);+test.skipIf(!process.env.DATABASE_URL)('a lock timeout after the drain whose holder also holds the refresh row still refuses, leaving the row draining for the next run', () => withRefreshFixture(engine, async f => {const before = f.head();const target = f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));