Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(sources refresh): bound and cancel both refusal-path lookups, mark deadlock/serialization/cancel retryable (GBRA-250 voice findings)
garrytan2edcfb59d278
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 53c595ae35..3af839b896 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,10 +18,10 @@ Before, a stuck transaction holding a source, worktree or brain row (a long-runn### Itemized changes-- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a failed lookup says so. A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.-- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s). The fix repeats `--dry-run`, `--abandon` or `--resume` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.+- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a lookup that fails or takes longer than 3 s says so (the statement is cancelled at that bound, so the refusal never holds the topology lock waiting on it). A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.+- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s; its statements are cancelled at that bound). A deadlock (`40P01`), serialization failure (`40001`) or cancelled statement (`57014`) is marked retryable. The fix repeats `--dry-run`, `--abandon` or `--resume` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.- `src/commands/sources-refresh.ts`: refusals print `why` (a `why` key under `--json`, a `Why:` line otherwise).-- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, credentials and the home directory are redacted without touching a sibling path, `--abandon` and `--dry-run` survive into the fix, a lookup that never answers still gives the refusal, the holder list is worded as candidates, and an abort blocked by the same holder still returns and says the row stays draining).+- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, credentials and the home directory are redacted without touching a sibling path, `--abandon` and `--dry-run` survive into the fix, a lookup that never answers still gives the refusal and is cancelled (row lookup and lock-holder lookup), transient SQLSTATEs are retryable, the holder list is worded as candidates, and an abort blocked by the same holder still returns and says the row stays draining).## [0.60.529.0] - 2026-10-31diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex d0ef2ba239..de58eee842 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -553,7 +553,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). || <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). A refresh whose holder is recorded on another host is never taken over either: writer status shows its `liveness` as `unknown`, and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. || <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |-| <a id="refresh_lock_timeout"></a>`refresh_lock_timeout` | `refresh_lock_timeout` | The refresh fence locks the brain, worktree, source and refresh rows with a 5 s lock timeout, and another database transaction held one of them longer. `why` lists the five oldest open transactions in this database holding locks on those tables (pid, application, state, how long the transaction has been open). They are candidates: the blocker may not be among them, because Postgres does not show which row each one holds. It never repeats their SQL, and says so when that lookup failed. HEAD did not move: no fence was set, or the draining refresh was aborted. When the same holder also blocked that abort, the refusal says the row stays `draining`, and the next `gbrain sources refresh <source>` resumes it. | Retry `gbrain sources refresh <source>`. If it repeats, a transaction holding one of those rows is stuck: `why` lists the oldest candidates, so confirm the blocker on the brain host before ending any transaction. |+| <a id="refresh_lock_timeout"></a>`refresh_lock_timeout` | `refresh_lock_timeout` | The refresh fence locks the brain, worktree, source and refresh rows with a 5 s lock timeout, and another database transaction held one of them longer. `why` lists the five oldest open transactions in this database holding locks on those tables (pid, application, state, how long the transaction has been open). They are candidates: the blocker may not be among them, because Postgres does not show which row each one holds. It never repeats their SQL, and says so when that lookup failed or took longer than 3 s. HEAD did not move: no fence was set, or the draining refresh was aborted. When the same holder also blocked that abort, the refusal says the row stays `draining`, and the next `gbrain sources refresh <source>` resumes it. | Retry `gbrain sources refresh <source>`. If it repeats, a transaction holding one of those rows is stuck: `why` lists the oldest candidates, so confirm the blocker on the brain host before ending any transaction. || <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within its share of the refresh sync budget (each member gets an equal share of what is left; `sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. || <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. || <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.tsindex 65e03e5c88..bf8d0561b8 100644--- a/src/core/persistence/worktree-refresh.ts+++ b/src/core/persistence/worktree-refresh.ts@@ -498,7 +498,26 @@ const redact = (text: string) => {const ownHome = new RegExp(`${escaped}(?![\\w.-])`, 'g');return redactConnectionInfo(redactUrlsInText(home.length > 1 ? text.replace(ownHome, '~') : text));};-const FAILURE_LOOKUP_MS = 3_000;+const LOOKUP_MS = 3_000;+/**+ * A best-effort diagnostic read on a refusal path, bounded at LOOKUP_MS: each raw statement runs under a transaction-local+ * statement_timeout and a signal that cancels it at the bound, so neither the refusal nor a pooled connection waits on it.+ * A read still pending at the bound resolves null.+ */+async function boundedLookup<T>(engine: BrainEngine, read: (engine: BrainEngine) => Promise<T>): Promise<T | null> {+ const controller = new AbortController();+ const deadlineAt = Date.now() + LOOKUP_MS;+ const bounded = new Proxy(engine, { get(target, key) {+ if (key === 'executeRaw') return (sql: string, params?: unknown[], opts?: object) =>+ target.executeRaw(sql, params, { ...opts, signal: controller.signal, timeoutMs: LOOKUP_MS, deadlineAt });+ const value = Reflect.get(target, key, target);+ return typeof value === 'function' ? value.bind(target) : value;+ } });+ let timer: ReturnType<typeof setTimeout> | undefined;+ const timedOut = new Promise<null>(resolve => { timer = setTimeout(() => { controller.abort(); resolve(null); }, LOOKUP_MS); });+ return Promise.race([Promise.resolve().then(() => read(bounded)), timedOut]).catch(() => null).finally(() => clearTimeout(timer));+}+const TRANSIENT_SQLSTATES = new Set(['40P01', '40001', '57014']);/*** GBRA-250: an untyped throw (a statement error, a TypeError) leaves the refresh where it stood, so a fenced or syncing row@@ -508,21 +527,19 @@ const FAILURE_LOOKUP_MS = 3_000;*/async function unexpectedRefreshFailure(engine: BrainEngine, sourceId: string, opts: RefreshOptions, error: unknown): Promise<OperationError> {const thrown = error instanceof Error ? error : new Error(String(error));- const errorCode = typeof (error as { code?: unknown } | null)?.code === 'string' ? ` (${(error as { code: string }).code})` : '';- let timer: ReturnType<typeof setTimeout> | undefined;- const lookup = await Promise.race([- Promise.resolve().then(async () => {- const binding = await getWorktreeBinding(engine, sourceId, existingLocalHostId());- return { active: binding ? await readActiveRefresh(engine, binding.worktree_id) : null };- }),- new Promise<null>(resolve => { timer = setTimeout(resolve, FAILURE_LOOKUP_MS, null); }),- ]).catch(() => null).finally(() => clearTimeout(timer));+ const code = typeof (error as { code?: unknown } | null)?.code === 'string' ? (error as { code: string }).code : null;+ const lookup = await boundedLookup(engine, async reads => {+ const binding = await getWorktreeBinding(reads, sourceId, existingLocalHostId());+ return { active: binding ? await readActiveRefresh(reads, binding.worktree_id) : null };+ });const command = `gbrain sources refresh ${sourceId}${opts.dryRun ? ' --dry-run' : ''}${opts.abandon ? ' --abandon' : opts.resume ? ' --resume' : ''}`;const left = !lookup ? 'The refresh row could not be read, so a fenced or syncing refresh may still be active.': lookup.active ? `Refresh ${lookup.active.id} stays ${lookup.active.state}${opts.abandon ? '.' : `, and the next gbrain sources refresh ${sourceId} resumes it.`}` : 'No refresh row was left active.';- return opError('internal_error', redact(`The refresh of source ${sourceId} failed unexpectedly: ${thrown.name}${errorCode}: ${thrown.message}`),+ const failure = opError('internal_error', redact(`The refresh of source ${sourceId} failed unexpectedly: ${thrown.name}${code ? ` (${code})` : ''}: ${thrown.message}`),`${left} Run ${command} again; if it fails the same way, report this output (why holds the stack) with gbrain doctor --json.`,{ why: redact(thrown.stack ?? `${thrown.name}: ${thrown.message}`) });+ if (code && TRANSIENT_SQLSTATES.has(code)) failure.retryable = true;+ return failure;}async function runRefresh(engine: BrainEngine, sourceId: string, opts: RefreshOptions): Promise<WorktreeRefreshResult> {@@ -623,17 +640,17 @@ const LOCK_NOT_AVAILABLE = '55P03';* which row each one holds), instead of an untyped internal_error.*/async function lockTimeoutRefusal(engine: BrainEngine, sourceId: string, refreshId?: string): Promise<OperationError> {- const holders = engine.kind !== 'postgres' ? [] : await engine.executeRaw<{ pid: number; application_name: string; state: string | null; open_seconds: number }>(+ const holders = engine.kind !== 'postgres' ? [] : await boundedLookup(engine, reads => reads.executeRaw<{ pid: number; application_name: string; state: string | null; open_seconds: number }>(`SELECT a.pid, a.application_name, a.state, extract(epoch FROM now() - a.xact_start)::int AS open_secondsFROM pg_stat_activity aWHERE a.pid <> pg_backend_pid() AND a.xact_start IS NOT NULL AND a.datname = current_database()AND EXISTS (SELECT 1 FROM pg_locks l WHERE l.pid = a.pid AND l.granted AND l.mode <> 'AccessShareLock'AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database())AND l.relation IN ('persistence_brain'::regclass, 'persistence_worktrees'::regclass, 'sources'::regclass, 'persistence_worktree_refreshes'::regclass))- ORDER BY a.xact_start LIMIT 5`).catch(() => null);+ ORDER BY a.xact_start LIMIT 5`));const error = refusal('refresh_lock_timeout', `The refresh fence of source ${sourceId} waited past its 5 s lock timeout for a brain, worktree, source or refresh row another database transaction holds; nothing moved.`,`Retry gbrain sources refresh ${sourceId}. If it repeats, a transaction holding one of those rows is stuck: why lists the oldest candidates, which may not include it, so confirm the blocker on the brain host before ending any transaction.`, refreshId);- error.why = holders === null ? 'The lookup of open transactions holding those rows failed.'+ error.why = holders === null ? 'The lookup of open transactions holding those rows failed or took longer than 3 s.': holders.length ? redact(`The oldest open transactions holding locks on those tables, at most five; the one blocking the fence may not be listed: ${holders.map(h => `pid ${h.pid} (${h.application_name || 'no application name'}, ${h.state ?? 'unknown state'}, open ${h.open_seconds} s)`).join('; ')}`): 'No open transaction holding a lock on those tables was visible to this database role.';return error;diff --git a/test/worktree-refresh-unexpected-throw.test.ts b/test/worktree-refresh-unexpected-throw.test.tsindex f1689ad48e..820b347935 100644--- a/test/worktree-refresh-unexpected-throw.test.ts+++ b/test/worktree-refresh-unexpected-throw.test.ts@@ -114,16 +114,30 @@ test('an unexpected throw under --dry-run repeats --dry-run in its fix, never aexpect(failure.suggestion).toContain('Run gbrain sources refresh source-a --dry-run again');});-test('a refresh-row lookup that never answers still yields the refusal, saying the row could not be read', async () => {+test('a refresh-row lookup that never answers still yields the refusal, and the stalled read is bounded and cancelled', async () => {const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });- let lookups = 0;- const fake = { kind: 'postgres', getConfig: async () => { throw statement; }, executeRaw: () => { lookups++; return new Promise(() => {}); } } as unknown as BrainEngine;+ const reads: Array<{ signal?: AbortSignal; timeoutMs?: number }> = [];+ const fake = { kind: 'postgres', getConfig: async () => { throw statement; },+ executeRaw: (_sql: string, _params?: unknown[], opts?: { signal?: AbortSignal; timeoutMs?: number }) => { reads.push(opts ?? {}); return new Promise(() => {}); } } as unknown as BrainEngine;const failure = await withEnv({ GBRAIN_REFRESH_SYNC_BUDGET_MS: undefined }, () => refreshWorktree(fake, 'source-a').then(() => null, error => error));- expect(lookups).toBeGreaterThan(0);+ expect(reads.length).toBeGreaterThan(0);expect(failure.code).toBe('internal_error');expect(failure.suggestion).toContain('The refresh row could not be read');+ expect(reads[0]!.timeoutMs).toBe(3_000);+ expect(reads[0]!.signal?.aborted).toBe(true);}, 15_000);+test('a deadlock or serialization failure is a retryable internal_error; an ordinary statement error is not marked retryable', async () => {+ const refuse = async (code: string) => {+ const thrown = Object.assign(new Error('deadlock detected'), { code });+ const fake = { kind: 'postgres', getConfig: async () => { throw thrown; }, executeRaw: async () => { throw thrown; } } as unknown as BrainEngine;+ return withEnv({ GBRAIN_REFRESH_SYNC_BUDGET_MS: undefined }, () => refreshWorktree(fake, 'source-a').then(() => null, error => error));+ };+ expect((await refuse('40P01')).retryable).toBe(true);+ expect((await refuse('40001')).retryable).toBe(true);+ expect((await refuse('42883')).retryable).toBeUndefined();+});+test('home-directory redaction leaves a sibling path that only shares the home prefix intact', async () => {const statement = Object.assign(new Error('canceling statement due to user request'), { code: '57014' });statement.stack = `Error: canceling statement\n at fence (${homedir()}/brain-checkout/src/x.ts:1:1)\n at load (${homedir()}-example/src/y.ts:1:1)`;@@ -230,6 +244,25 @@ test.skipIf(!process.env.DATABASE_URL)('the lock-timeout why names only row-lockexpect(failure!.why).not.toContain('is among them');}), 120_000);+test.skipIf(!process.env.DATABASE_URL)('a lock-holder lookup that never answers still refuses refresh_lock_timeout, and the stalled read is cancelled', () => withRefreshFixture(engine, async f => {+ f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));+ const held = await holdSourceRow(f.alpha);+ const target = f.engine as unknown as { executeRaw: (this: unknown, sql: string, params?: unknown[], opts?: { signal?: AbortSignal }) => Promise<unknown> };+ const original = target.executeRaw;+ let stalled: AbortSignal | undefined;+ target.executeRaw = function (sql, params, opts) {+ if (!sql.includes('a.xact_start IS NOT NULL')) return original.call(this, sql, params, opts);+ stalled = opts?.signal;+ return new Promise((_, reject) => opts?.signal?.addEventListener('abort', () => reject(opts.signal!.reason), { once: true }));+ };+ let failure: OperationError | null;+ try { failure = await refreshWorktree(f.engine, f.alpha).then(() => null, error => error); }+ finally { target.executeRaw = original; await held.release(); }+ expect(failure!.code).toBe('refresh_lock_timeout');+ expect(failure!.why).toBe('The lookup of open transactions holding those rows failed or took longer than 3 s.');+ expect(stalled?.aborted).toBe(true);+}), 60_000);+test.skipIf(!process.env.DATABASE_URL)('a lock timeout after the drain whose holder also holds the refresh row still refuses, leaving the row draining for the next run', () => withRefreshFixture(engine, async f => {const before = f.head();const target = f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));