Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(sources refresh): look up lock holders after the topology lock is released, keep --wait-drain/--fetch-timeout-ms in the retry, mark 55P03 retryable (GBRA-250 voice findings)

garrytane7732d2165b9
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3af839b896..11980cc04b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,8 +18,8 @@ Before, a stuck transaction holding a source, worktree or brain row (a long-runn
### Itemized changes
-- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a lookup that fails or takes longer than 3 s says so (the statement is cancelled at that bound, so the refusal never holds the topology lock waiting on it). A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.
-- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s; its statements are cancelled at that bound). A deadlock (`40P01`), serialization failure (`40001`) or cancelled statement (`57014`) is marked retryable. The fix repeats `--dry-run`, `--abandon` or `--resume` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
+- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a lookup that fails or takes longer than 3 s says so (the statement is cancelled at that bound, and after the drain the lookup runs only once the topology and worktree locks are released). A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.
+- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s; its statements are cancelled at that bound). A deadlock (`40P01`), serialization failure (`40001`), cancelled statement (`57014`) or lock timeout outside the fence (`55P03`) is marked retryable. The fix repeats `--dry-run`, `--abandon`, `--resume`, `--wait-drain` and `--fetch-timeout-ms` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
- `src/commands/sources-refresh.ts`: refusals print `why` (a `why` key under `--json`, a `Why:` line otherwise).
- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, credentials and the home directory are redacted without touching a sibling path, `--abandon` and `--dry-run` survive into the fix, a lookup that never answers still gives the refusal and is cancelled (row lookup and lock-holder lookup), transient SQLSTATEs are retryable, the holder list is worded as candidates, and an abort blocked by the same holder still returns and says the row stays draining).
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index bf8d0561b8..8bbb8d6c9c 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -517,7 +517,7 @@ async function boundedLookup<T>(engine: BrainEngine, read: (engine: BrainEngine)
const timedOut = new Promise<null>(resolve => { timer = setTimeout(() => { controller.abort(); resolve(null); }, LOOKUP_MS); });
return Promise.race([Promise.resolve().then(() => read(bounded)), timedOut]).catch(() => null).finally(() => clearTimeout(timer));
}
-const TRANSIENT_SQLSTATES = new Set(['40P01', '40001', '57014']);
+const TRANSIENT_SQLSTATES = new Set(['40P01', '40001', '57014', '55P03']);
/**
* GBRA-250: an untyped throw (a statement error, a TypeError) leaves the refresh where it stood, so a fenced or syncing row
@@ -532,7 +532,8 @@ async function unexpectedRefreshFailure(engine: BrainEngine, sourceId: string, o
const binding = await getWorktreeBinding(reads, sourceId, existingLocalHostId());
return { active: binding ? await readActiveRefresh(reads, binding.worktree_id) : null };
});
- const command = `gbrain sources refresh ${sourceId}${opts.dryRun ? ' --dry-run' : ''}${opts.abandon ? ' --abandon' : opts.resume ? ' --resume' : ''}`;
+ const command = `gbrain sources refresh ${sourceId}${opts.dryRun ? ' --dry-run' : ''}${opts.abandon ? ' --abandon' : opts.resume ? ' --resume' : ''}`
+ + `${opts.waitDrainMs !== undefined ? ` --wait-drain ${opts.waitDrainMs / 1000}` : ''}${opts.fetchTimeoutMs !== undefined ? ` --fetch-timeout-ms ${opts.fetchTimeoutMs}` : ''}`;
const left = !lookup ? 'The refresh row could not be read, so a fenced or syncing refresh may still be active.'
: lookup.active ? `Refresh ${lookup.active.id} stays ${lookup.active.state}${opts.abandon ? '.' : `, and the next gbrain sources refresh ${sourceId} resumes it.`}` : 'No refresh row was left active.';
const failure = opError('internal_error', redact(`The refresh of source ${sourceId} failed unexpectedly: ${thrown.name}${code ? ` (${code})` : ''}: ${thrown.message}`),
@@ -656,6 +657,9 @@ async function lockTimeoutRefusal(engine: BrainEngine, sourceId: string, refresh
return error;
}
+/** A fence transaction past its lock timeout; fenceAndMerge builds the refusal after the native locks are released. */
+class FenceLockTimeout extends Error {}
+
/** Steps 3-5 under the worktree native lock: recheck drained, fence, verify HEAD, ff-only merge, bookkeeping. */
async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, plan: Plan, row: WorktreeRefreshRow, hooks?: RefreshHooks): Promise<WorktreeRefreshRow> {
let fenced: WorktreeRefreshRow;
@@ -674,7 +678,7 @@ async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, pl
`Work reached the worktree after the drain check (${counts.requests} request(s), ${counts.effects} checkout effect(s), ${counts.recovery} recovery record(s)); nothing moved.`,
`Retry gbrain sources refresh ${sourceId}`, row.id);
if (!await transition(tx, row, ['draining'], 'fenced')) throw refusal('refresh_source_changed', 'The refresh record was moved by recovery while draining.', `gbrain sources refresh ${sourceId} --resume`, row.id);
- }).catch(async error => { throw (error as { code?: string }).code === LOCK_NOT_AVAILABLE ? await lockTimeoutRefusal(engine, sourceId, row.id) : error; });
+ }).catch(error => { throw (error as { code?: string }).code === LOCK_NOT_AVAILABLE ? new FenceLockTimeout() : error; });
const current = await readRefresh(engine, row.id);
await hooks?.boundary?.('fenced', current);
const head = await headOf(git, plan.root);
@@ -697,7 +701,8 @@ async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, pl
await bookkeeping(engine, current);
return readRefresh(engine, row.id);
}, undefined, 30_000);
- } catch (error) {
+ } catch (thrown) {
+ const error = thrown instanceof FenceLockTimeout ? await lockTimeoutRefusal(engine, sourceId, row.id) : thrown;
const stored = await readRefresh(engine, row.id);
if (stored.state === 'draining' && error instanceof OperationError) {
await engine.transaction(async tx => {
diff --git a/test/worktree-refresh-unexpected-throw.test.ts b/test/worktree-refresh-unexpected-throw.test.ts
index 820b347935..cf6f7fea59 100644
--- a/test/worktree-refresh-unexpected-throw.test.ts
+++ b/test/worktree-refresh-unexpected-throw.test.ts
@@ -21,6 +21,7 @@ import { runSourcesRefresh } from '../src/commands/sources-refresh.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
import { refreshWorktree } from '../src/core/persistence/worktree-refresh.ts';
import { persistenceHome } from '../src/core/persistence/identity.ts';
+import { tryAcquireNativeLock } from '../src/core/persistence/native-lock.ts';
import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
import { page, withRefreshFixture } from './helpers/worktree-refresh-fixture.ts';
import { withEnv } from './helpers/with-env.ts';
@@ -114,6 +115,13 @@ test('an unexpected throw under --dry-run repeats --dry-run in its fix, never a
expect(failure.suggestion).toContain('Run gbrain sources refresh source-a --dry-run again');
});
+test('an unexpected throw repeats --wait-drain and --fetch-timeout-ms in its fix', async () => {
+ const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });
+ const fake = { kind: 'postgres', getConfig: async () => { throw statement; }, executeRaw: async () => { throw statement; } } as unknown as BrainEngine;
+ const failure = await withEnv({ GBRAIN_REFRESH_SYNC_BUDGET_MS: undefined }, () => refreshWorktree(fake, 'source-a', { waitDrainMs: 120_000, fetchTimeoutMs: 5_000 }).then(() => null, error => error));
+ expect(failure.suggestion).toContain('Run gbrain sources refresh source-a --wait-drain 120 --fetch-timeout-ms 5000 again');
+});
+
test('a refresh-row lookup that never answers still yields the refusal, and the stalled read is bounded and cancelled', async () => {
const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });
const reads: Array<{ signal?: AbortSignal; timeoutMs?: number }> = [];
@@ -135,6 +143,7 @@ test('a deadlock or serialization failure is a retryable internal_error; an ordi
};
expect((await refuse('40P01')).retryable).toBe(true);
expect((await refuse('40001')).retryable).toBe(true);
+ expect((await refuse('55P03')).retryable).toBe(true);
expect((await refuse('42883')).retryable).toBeUndefined();
});
@@ -263,6 +272,27 @@ test.skipIf(!process.env.DATABASE_URL)('a lock-holder lookup that never answers
expect(stalled?.aborted).toBe(true);
}), 60_000);
+test.skipIf(!process.env.DATABASE_URL)('a lock timeout after the drain looks up the holders only after the topology lock is released', () => withRefreshFixture(engine, async f => {
+ f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));
+ const target = f.engine as unknown as { executeRaw: (this: unknown, sql: string, params?: unknown[], opts?: object) => Promise<Array<Record<string, string>>> };
+ const original = target.executeRaw;
+ const [brain] = await original.call(f.engine, 'SELECT brain_id FROM persistence_brain WHERE singleton=1');
+ let topologyFree: boolean | undefined;
+ target.executeRaw = async function (sql, params, opts) {
+ if (sql.includes('a.xact_start IS NOT NULL') && topologyFree === undefined) {
+ const handle = await tryAcquireNativeLock(join(persistenceHome(), 'locks', `topology-${brain!.brain_id}.lock`));
+ topologyFree = handle !== null;
+ await handle?.release();
+ }
+ return original.call(this, sql, params, opts);
+ };
+ let held: Awaited<ReturnType<typeof holdSourceRow>> | undefined;
+ const failure = await refreshWorktree(f.engine, f.alpha, { hooks: { boundary: async point => { if (point === 'drained') held = await holdSourceRow(f.alpha); } } })
+ .then(() => null, error => error).finally(async () => { target.executeRaw = original; await held?.release(); });
+ expect(failure.code).toBe('refresh_lock_timeout');
+ expect(topologyFree).toBe(true);
+}), 120_000);
+
test.skipIf(!process.env.DATABASE_URL)('a lock timeout after the drain whose holder also holds the refresh row still refuses, leaving the row draining for the next run', () => withRefreshFixture(engine, async f => {
const before = f.head();
const target = f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));