Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pre-landing review fixes (refresh_lock_timeout names only row-lock holders in this database and never echoes their SQL; an unreadable host identity still refuses typed)

garrytanfc6edfb9fe9e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c60ad91127..e511348bb1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,10 +18,10 @@ Before, a stuck transaction holding a source, worktree or brain row (a long-runn
### Itemized changes
-- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists up to five open transactions holding locks on `persistence_brain`, `persistence_worktrees` or `sources` (pid, application, state, seconds open, the start of the query). A refusal after the drain aborts the draining row as other fence refusals do.
-- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, SQLSTATE and message (connection details and URLs redacted), the stack in `why`, and a fix naming the active refresh row and its state. The row is not changed. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
+- `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists up to five open transactions in the current database holding row-level locks on `persistence_brain`, `persistence_worktrees` or `sources` (pid, application, state, seconds open). Plain readers, which cannot block the fence, are left out, and the holders' SQL text is never copied into `why`. A refusal after the drain aborts the draining row as other fence refusals do.
+- `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, SQLSTATE and message (connection details and URLs redacted), the stack in `why`, and a fix naming the active refresh row and its state. The row is not changed, and a host identity file that cannot be read still yields that refusal. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
- `src/commands/sources-refresh.ts`: refusals print `why` (a `why` key under `--json`, a `Why:` line otherwise).
-- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain).
+- Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal).
## [0.60.529.0] - 2026-10-31
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 7cb81cc2b6..9c71fd71cd 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -553,7 +553,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.
| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |
| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). A refresh whose holder is recorded on another host is never taken over either: writer status shows its `liveness` as `unknown`, and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
| <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |
-| <a id="refresh_lock_timeout"></a>`refresh_lock_timeout` | `refresh_lock_timeout` | The refresh fence locks the brain, worktree and source rows with a 5 s lock timeout, and another database transaction held one of them longer. `why` lists the open transactions holding locks on those tables (pid, application, state, how long the transaction has been open, the start of its query). Nothing moved: no fence was set, or the draining refresh was aborted, and HEAD did not move. | Retry `gbrain sources refresh <source>`. If it repeats, the transaction in `why` is stuck: find out what it is on the brain host before ending it. |
+| <a id="refresh_lock_timeout"></a>`refresh_lock_timeout` | `refresh_lock_timeout` | The refresh fence locks the brain, worktree and source rows with a 5 s lock timeout, and another database transaction held one of them longer. `why` lists the open transactions in this database holding row locks on those tables (pid, application, state, how long the transaction has been open); it never repeats their SQL. Nothing moved: no fence was set, or the draining refresh was aborted, and HEAD did not move. | Retry `gbrain sources refresh <source>`. If it repeats, the transaction in `why` is stuck: find out what it is on the brain host before ending it. |
| <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within its share of the refresh sync budget (each member gets an equal share of what is left; `sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. |
| <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. |
| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index 7d55f2894a..15e1b4c41b 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -488,6 +488,8 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt
catch (error) { throw error instanceof OperationError ? error : await unexpectedRefreshFailure(engine, sourceId, error); }
}
+const redact = (text: string) => redactUrlsInText(redactConnectionInfo(text));
+
/**
* GBRA-250: an untyped throw (a statement error, a TypeError) leaves the refresh where it stood, so a fenced or syncing row
* resumes on the next run. The refusal names that row and carries the stack in `why`, so a failure seen once leaves a trace.
@@ -495,8 +497,7 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt
async function unexpectedRefreshFailure(engine: BrainEngine, sourceId: string, error: unknown): Promise<OperationError> {
const thrown = error instanceof Error ? error : new Error(String(error));
const sqlstate = typeof (error as { code?: unknown } | null)?.code === 'string' ? ` (${(error as { code: string }).code})` : '';
- const redact = (text: string) => redactUrlsInText(redactConnectionInfo(text));
- const binding = await getWorktreeBinding(engine, sourceId, existingLocalHostId()).catch(() => null);
+ const binding = await Promise.resolve().then(() => getWorktreeBinding(engine, sourceId, existingLocalHostId())).catch(() => null);
const active = binding ? await readActiveRefresh(engine, binding.worktree_id).catch(() => null) : null;
const left = active ? `Refresh ${active.id} stays ${active.state}, and the next gbrain sources refresh ${sourceId} resumes it.` : 'No refresh row was left active.';
return opError('internal_error', redact(`The refresh of source ${sourceId} failed unexpectedly: ${thrown.name}${sqlstate}: ${thrown.message}`),
@@ -601,17 +602,18 @@ const LOCK_NOT_AVAILABLE = '55P03';
* typed refusal naming the open transactions that hold locks on the fenced tables, instead of an untyped internal_error.
*/
async function lockTimeoutRefusal(engine: BrainEngine, sourceId: string, refreshId?: string): Promise<OperationError> {
- const holders = engine.kind !== 'postgres' ? [] : await engine.executeRaw<{ pid: number; application_name: string; state: string | null; open_seconds: number; query: string | null }>(
- `SELECT a.pid, a.application_name, a.state, extract(epoch FROM now() - a.xact_start)::int AS open_seconds, left(a.query, 160) AS query
+ const holders = engine.kind !== 'postgres' ? [] : await engine.executeRaw<{ pid: number; application_name: string; state: string | null; open_seconds: number }>(
+ `SELECT a.pid, a.application_name, a.state, extract(epoch FROM now() - a.xact_start)::int AS open_seconds
FROM pg_stat_activity a
- WHERE a.pid <> pg_backend_pid() AND a.xact_start IS NOT NULL
- AND EXISTS (SELECT 1 FROM pg_locks l WHERE l.pid = a.pid AND l.granted
+ WHERE a.pid <> pg_backend_pid() AND a.xact_start IS NOT NULL AND a.datname = current_database()
+ AND EXISTS (SELECT 1 FROM pg_locks l WHERE l.pid = a.pid AND l.granted AND l.mode <> 'AccessShareLock'
+ AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database())
AND l.relation IN ('persistence_brain'::regclass, 'persistence_worktrees'::regclass, 'sources'::regclass))
ORDER BY a.xact_start LIMIT 5`).catch(() => []);
const error = refusal('refresh_lock_timeout', `The refresh fence of source ${sourceId} waited past its 5 s lock timeout for a brain, worktree or source row another database transaction holds; nothing moved.`,
`Retry gbrain sources refresh ${sourceId}. If it repeats, the transaction named in why is stuck: find out what it is on the brain host before ending it.`, refreshId);
error.why = holders.length
- ? `Open transactions holding locks on those tables: ${holders.map(h => `pid ${h.pid} (${h.application_name || 'no application name'}, ${h.state ?? 'unknown state'}, open ${h.open_seconds} s): ${(h.query ?? '').replace(/\s+/g, ' ').trim()}`).join('; ')}`
+ ? redact(`Open transactions holding row locks on those tables: ${holders.map(h => `pid ${h.pid} (${h.application_name || 'no application name'}, ${h.state ?? 'unknown state'}, open ${h.open_seconds} s)`).join('; ')}`)
: 'No open transaction holding a lock on those tables was visible to this database role.';
return error;
}
diff --git a/test/worktree-refresh-unexpected-throw.test.ts b/test/worktree-refresh-unexpected-throw.test.ts
index 6d694d5f4f..2a384ee116 100644
--- a/test/worktree-refresh-unexpected-throw.test.ts
+++ b/test/worktree-refresh-unexpected-throw.test.ts
@@ -9,6 +9,8 @@
* refresh_lock_timeout naming the open transactions instead.
*/
import { afterAll, beforeAll, expect, test } from 'bun:test';
+import { readFileSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
import postgres from '#postgres';
import type { BrainEngine } from '../src/core/engine.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
@@ -17,6 +19,7 @@ import { _resetCliExitVerdictForTests, currentExitCode } from '../src/core/cli-f
import { runSourcesRefresh } from '../src/commands/sources-refresh.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
import { refreshWorktree } from '../src/core/persistence/worktree-refresh.ts';
+import { persistenceHome } from '../src/core/persistence/identity.ts';
import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
import { page, withRefreshFixture } from './helpers/worktree-refresh-fixture.ts';
import { withEnv } from './helpers/with-env.ts';
@@ -72,15 +75,35 @@ test('sources refresh prints an untyped throw as a refused document carrying the
expect(doc.fix).toContain('gbrain sources refresh source-a');
});
-/** Another connection holds `SELECT … FOR UPDATE` on one source row until `release` (a stuck transaction on the brain host). */
-async function holdSourceRow(sourceId: string): Promise<{ pid: number; release: () => Promise<void> }> {
+test('a host identity file that cannot be read still yields a typed internal_error refusal', () => withRefreshFixture(engine, async f => {
+ const hostFile = join(persistenceHome(), 'host.json');
+ const original = readFileSync(hostFile, 'utf8');
+ writeFileSync(hostFile, '{');
+ let failure: unknown;
+ try { failure = await refreshWorktree(f.engine, f.alpha).then(() => null, error => error); }
+ finally { writeFileSync(hostFile, original); }
+ expect(failure).toBeInstanceOf(OperationError);
+ expect((failure as OperationError).code).toBe('internal_error');
+ expect((failure as OperationError).message).toContain('SyntaxError');
+ expect((failure as OperationError).suggestion).toContain('No refresh row was left active.');
+}), 120_000);
+
+/**
+ * Another connection holds one source row until `release` (a stuck transaction on the brain host): `FOR UPDATE` by
+ * default, or a plain read (AccessShareLock only, which cannot block the fence) with `plainRead`. Its last statement
+ * carries a connection string, which the refusal must never echo.
+ */
+async function holdSourceRow(sourceId: string, plainRead = false): Promise<{ pid: number; release: () => Promise<void> }> {
const sql = postgres(databaseUrl, { max: 1, prepare: false });
let release!: () => void;
const released = new Promise<void>(resolve => { release = resolve; });
let locked!: (pid: number) => void;
const pid = new Promise<number>(resolve => { locked = resolve; });
const tx = sql.begin(async t => {
- const [row] = await t`SELECT pg_backend_pid() AS pid FROM sources WHERE id=${sourceId} FOR UPDATE`;
+ const [row] = plainRead
+ ? await t`SELECT pg_backend_pid() AS pid FROM sources WHERE id=${sourceId}`
+ : await t`SELECT pg_backend_pid() AS pid FROM sources WHERE id=${sourceId} FOR UPDATE`;
+ await t.unsafe("SELECT 'postgres://alice-example:[email protected]/brain' AS note");
locked(Number(row!.pid));
await released;
});
@@ -129,3 +152,17 @@ test.skipIf(!process.env.DATABASE_URL)('a source row taken after the drain abort
expect((await refreshWorktree(f.engine, f.alpha)).status).toBe('completed');
expect(await f.lastCommit(f.alpha)).toBe(target);
}), 120_000);
+
+test.skipIf(!process.env.DATABASE_URL)('the lock-timeout why names only row-lock holders in this database and never echoes their SQL', () => withRefreshFixture(engine, async f => {
+ f.push('alpha/two.md', page('Alpha two', 'Upstream side.'));
+ const reader = await holdSourceRow(f.alpha, true);
+ const held = await holdSourceRow(f.alpha);
+ let failure: OperationError | null;
+ try { failure = await refreshWorktree(f.engine, f.alpha).then(() => null, error => error); }
+ finally { await held.release(); await reader.release(); }
+ expect(failure!.code).toBe('refresh_lock_timeout');
+ expect(failure!.why).toContain(`pid ${held.pid} `);
+ expect(failure!.why).not.toContain(`pid ${reader.pid} `);
+ expect(failure!.why).not.toContain('hunter2');
+ expect(failure!.why).not.toContain('SELECT');
+}), 120_000);