Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: a links or facts-backstop effect behind a publication in flight waits as writer_busy, not recovery_required

garrytan5a9851f581dd
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1b24974bc7..3922eca0ba 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,15 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A `links` or `facts-backstop` effect that meets another write's publication in flight now waits as `writer_busy` and retries within a second, instead of reporting `recovery_required` and waiting 30 s.** These effects run without the worktree lock, so on PostgreSQL, where several processes publish at once, a concurrent writer could record its publication's recovery between the effect's claim and its guard. The guard read that record as an unfinished recovery: the effect was requeued with `recovery_required`, a code that tells an agent to inspect the brain host, and a fixed 30 s delay, which often outlasted the process that queued it. Nothing was lost (the next effect pass after the delay derived the links), but the receipt said a recovery was needed when none was.
+
+### Itemized changes
+
+- `src/core/persistence/effects.ts`: the effect guard tells a publication in flight (its request `running` with a live claim, or already terminal with its record not yet cleared) from one that needs recovery. An effect that does not hold the worktree lock and meets only publications in flight is requeued `writer_busy` with the existing busy backoff (250 ms doubling, at most 60 s, never parked). A record whose publisher's claim lapsed, or a request in `recovering`, still requeues it `recovery_required`, and an effect that holds the lock (Git, withdrawal mirror) is unchanged. An `embedding` effect in the same wait records `writer_busy` too, on its unchanged retry schedule.
+- `test/persistence-derived-effect-publication-race.test.ts` (PGLite, and PostgreSQL through `test/e2e/persistence-derived-effect-publication-race-postgres.test.ts`): drives a real put_page publication to its `prepared` and `after_commit` boundaries between a links effect's claim and its guard, and checks the effect waits as `writer_busy` with a delay under 5 s, then commits its link on the next pass; with the publisher's claim lapsed it stays `recovery_required` and still links once the record settles.
+
## [0.60.597.0] - 2026-11-06
**`gbrain upgrade` now restarts a jobs supervisor started from a source checkout as `bun run src/cli.ts jobs supervisor` instead of refusing the upgrade with `restart_refused`.** The restart planner recognised a gbrain process only when its command line held a path containing "gbrain", so a supervisor launched with a relative `src/cli.ts` was invisible, its worker looked unsupervised, and the upgrade stopped. Four smaller fixes ride along with it, covering a flaky spend-guard SIGTERM test, the PGLite snapshot cache keys, the stress gate's pooled-owner match, and the ownership stamp on a checkout that became a Git repository after it was stamped.
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index a49669ba9d..20661308c8 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1470,6 +1470,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.594.0', id: 'file_drift_held_needs_prefer_8615', text: '`gbrain repair file-drift` with `--held`, `--side`, `--state`, `--cursor` or `--provenance` and no `--prefer` now refuses with the usage line (`invalid_params`, and `file_drift_state_required` for a deleted side without `--state`) instead of silently running the all-drifted survey, which counted a different set of pages (#8615).' },
{ since: '0.60.594.0', id: 'refresh_source_changed_retry_8617', text: 'A `gbrain sources refresh` run refused `refresh_source_changed` because the checkout HEAD moved (a writer committing) or work reached the checkout after the drain check now retries once from a fresh precheck in the same run, so the tick is not lost (#8617). When the retry is refused too, the run returns the first `refresh_source_changed` refusal with the retry\'s cause appended; ownership or source-membership changes are not retried.' },
{ since: '0.60.594.0', id: 'commit_push_checkout_not_a_source_8662', text: '`gbrain git commit-push` now refuses `checkout_not_a_source` before staging anything when this brain home records its source checkouts (a managed brain) and the checkout it resolved from the working directory (no `--path`) is none of them and has no `gbrain.sourceRoot`, for example a wrapper that changed into another repository (#8662). The refusal names the resolved path and the brain\'s checkouts; pass `--path` with the absolute brain checkout. Hosts whose brain records no source checkout behave as before.' },
+ { since: 'TBD', id: 'derived_effect_publication_busy_gst1644', text: 'A `links` or `facts-backstop` effect that meets another write\'s publication in flight in the same worktree (its claim live, or committed with its recovery record not yet cleared) now waits as `writer_busy` with the busy backoff (250 ms doubling, at most 60 s) instead of `recovery_required` with a fixed 30 s retry, so a write receipt no longer reports `recovery_required` for that routine wait. A recovery record whose publisher\'s claim lapsed, or a request in `recovering`, still holds the effect as `recovery_required`. Neither wait parks a links or facts-backstop effect, and no derived link is lost: the next effect pass after it is due derives it. An `embedding` effect in the same wait now records `writer_busy` too; its retry schedule is unchanged.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/persistence/effects.ts b/src/core/persistence/effects.ts
index df9f25e24c..148beaba91 100644
--- a/src/core/persistence/effects.ts
+++ b/src/core/persistence/effects.ts
@@ -885,8 +885,12 @@ async function runEffectPass(engine: BrainEngine, config: GBrainConfig, opts: Ef
await engine.transaction(async tx => {
await guardEffectSource(tx, effect, opts.hostId);
if (effect.worktree_id) {
- const blocked = await tx.executeRaw('SELECT id FROM persistence_requests WHERE worktree_id=$1::uuid AND recovery IS NOT NULL LIMIT 1', [effect.worktree_id]);
- if (blocked.length) throw recoveryFirst(effect);
+ // GST-1644: an effect that runs without the worktree lock can meet another writer's publication in flight (its
+ // claim live, or committed with its record not yet cleared); that is contention, not a recovery.
+ const [blocked] = await tx.executeRaw<{ publishing: boolean | null }>(`SELECT bool_and(state='running' AND claim_expires_at>now()
+ OR state IN ('committed','conflict','failed','cancelled')) AS publishing FROM persistence_requests WHERE worktree_id=$1::uuid AND recovery IS NOT NULL`, [effect.worktree_id]);
+ if (blocked?.publishing === true && !lock) throw writerBusy(effect);
+ if (blocked && blocked.publishing !== null) throw recoveryFirst(effect);
}
}, { pipelineFirst: isPersistenceDeclaration });
effect = await upgradeWithdrawalEffect(engine, effect, opts.hostId);
diff --git a/test/e2e/persistence-derived-effect-publication-race-postgres.test.ts b/test/e2e/persistence-derived-effect-publication-race-postgres.test.ts
new file mode 100644
index 0000000000..64a25b7cb7
--- /dev/null
+++ b/test/e2e/persistence-derived-effect-publication-race-postgres.test.ts
@@ -0,0 +1,3 @@
+import { registerPostgresTests } from '../helpers/test-backends.ts';
+
+await registerPostgresTests(() => import('../persistence-derived-effect-publication-race.test.ts'));
diff --git a/test/persistence-derived-effect-publication-race.test.ts b/test/persistence-derived-effect-publication-race.test.ts
new file mode 100644
index 0000000000..371387b372
--- /dev/null
+++ b/test/persistence-derived-effect-publication-race.test.ts
@@ -0,0 +1,146 @@
+/**
+ * GST-1644: a links or facts-backstop effect runs without the worktree lock, so on PostgreSQL another writer's
+ * publication can record its recovery between the effect's claim and its guard. That record is the publication in
+ * flight, not a crash: the effect waits for it as `writer_busy` with the short busy backoff. Only a recovery whose
+ * publisher's claim lapsed (or a request in `recovering`) stays `recovery_required`. Either way the effect stays
+ * queued and the next pass after it is due derives the links.
+ *
+ * The interleaving is forced by wrapping the engine's transaction (no production seam): the first transaction after the
+ * effect is claimed is its guard, and before it runs a real put_page publication is driven to a boundary.
+ */
+import { describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import type { BrainEngine } from '../src/core/engine.ts';
+import type { OperationContext } from '../src/core/ops/contract.ts';
+import type { PublicationHooks } from '../src/core/persistence/coordinator.ts';
+import { publishMutation } from '../src/core/persistence/coordinator.ts';
+import { submissionAuthority } from '../src/core/persistence/authority.ts';
+import { localHostId } from '../src/core/persistence/identity.ts';
+import { admitWrite, claimNextWrite } from '../src/core/persistence/journal.ts';
+import { getWorktreeBinding } from '../src/core/persistence/ownership.ts';
+import { preparePageMutation } from '../src/core/persistence/page-prepare.ts';
+import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
+import { declarePersistenceProtocol } from '../src/core/persistence/protocol.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import type { WriteRequest } from '../src/core/persistence/model.ts';
+import { runPersistenceEffects } from '../src/core/persistence/effects.ts';
+import { managedBrain } from './helpers/managed-brain.ts';
+import { due } from './pathological/effects-fixture.ts';
+import { pathologicalArms } from './pathological/harness.ts';
+
+const TARGET = 'notes/target-example';
+const PAGE = 'notes/page-example';
+const OTHER = 'notes/other-example';
+const doc = (slug: string, body: string) => `---\ntype: note\ntitle: ${slug}\n---\n\n${body}\n`;
+const put = (ctx: OperationContext, slug: string, body: string) => submitPageMutation(ctx, { operation: 'put_page', params: { slug, request_id: randomUUID(), content: doc(slug, body) } });
+
+type Row = { kind: string; state: string; error_code: string | null; delay_ms: number; links: string | null };
+const derived = (engine: BrainEngine) => engine.executeRaw<Row>(`SELECT e.kind,e.state,e.error_code,e.outcome->>'links' AS links,
+ GREATEST(0,EXTRACT(EPOCH FROM (e.next_attempt_at-now()))*1000)::int AS delay_ms
+ FROM persistence_effects e WHERE e.kind IN ('links','facts-backstop') ORDER BY e.kind`);
+const mentions = (engine: BrainEngine) => engine.executeRaw<{ slug: string }>(`SELECT t.slug FROM links l JOIN pages f ON f.id=l.from_page_id
+ JOIN pages t ON t.id=l.to_page_id WHERE f.source_id='default' AND f.slug=$1 ORDER BY t.slug`, [PAGE]);
+
+/** PAGE mentions TARGET; its request gets a links and then a facts-backstop effect (the shapes a remote put_page queues), and every seed effect settles. */
+async function seed(engine: BrainEngine, ctx: OperationContext): Promise<void> {
+ await put(ctx, TARGET, 'The target.');
+ await put(ctx, PAGE, `See [[${TARGET}]].`);
+ await disposePersistenceConsumer(engine);
+ await due(engine);
+ await runPersistenceEffects(engine, ctx.config, { hostId: localHostId(), limit: 20 });
+ await engine.transaction(async tx => {
+ await declarePersistenceProtocol(tx);
+ await tx.executeRaw(`DELETE FROM links l USING pages f WHERE f.id=l.from_page_id AND f.slug=$1`, [PAGE]);
+ await tx.executeRaw(`INSERT INTO persistence_effects(request_id,kind,revision,data,source_id,source_incarnation,worktree_id)
+ SELECT e.request_id,k.kind,e.revision,e.data,e.source_id,e.source_incarnation,e.worktree_id FROM persistence_effects e
+ JOIN persistence_requests r ON r.id=e.request_id CROSS JOIN (VALUES ('links'),('facts-backstop')) AS k(kind)
+ WHERE e.kind='embedding' AND r.slug=$1`, [PAGE]);
+ });
+}
+
+/** Another writer's put_page, claimed and prepared, publishing until `release` resolves; its publication pauses at `at`. */
+async function otherWriter(engine: BrainEngine, ctx: OperationContext, at: Parameters<NonNullable<PublicationHooks['boundary']>>[0]) {
+ const binding = (await getWorktreeBinding(engine, 'default', localHostId()))!;
+ const authority = await submissionAuthority(ctx, 'put_page', 'default', binding.source_incarnation, OTHER);
+ const intent = { content: doc(OTHER, 'Another writer.') };
+ await admitWrite(engine, { principal: authority.principal, authority, operation: 'put_page', sourceId: 'default', sourceIncarnation: binding.source_incarnation,
+ slug: OTHER, pageId: null, requestId: randomUUID(), callerIntent: intent, intent, worktreeId: binding.worktree_id, topologyGeneration: binding.topology_generation });
+ const row = (await claimNextWrite(engine, localHostId()))!;
+ expect(row.slug).toBe(OTHER);
+ let reached!: () => void; let release!: () => void;
+ const paused = new Promise<void>(resolve => { reached = resolve; });
+ const gate = new Promise<void>(resolve => { release = resolve; });
+ return { row, paused, release, start: async (): Promise<WriteRequest> => publishMutation(engine, row, await preparePageMutation(engine, row, ctx.config), localHostId(),
+ { boundary: async name => { if (name === at) { reached(); await gate; } } }) };
+}
+
+/** Runs one effect pass; before the guard of the first claimed derived effect, `interleave` runs. */
+async function racedPass(engine: BrainEngine, ctx: OperationContext, interleave: () => Promise<void>): Promise<void> {
+ const original = engine.transaction;
+ let armed = true;
+ engine.transaction = (async function (this: BrainEngine, fn: (tx: BrainEngine) => Promise<unknown>, opts?: unknown) {
+ if (armed && this === engine) {
+ const [claimed] = await engine.executeRaw(`SELECT 1 FROM persistence_effects WHERE kind IN ('links','facts-backstop') AND state='running'`);
+ if (claimed) { armed = false; await interleave(); }
+ }
+ return original.call(this, fn, opts as never);
+ }) as BrainEngine['transaction'];
+ try {
+ await due(engine);
+ await runPersistenceEffects(engine, ctx.config, { hostId: localHostId(), limit: 20 });
+ } finally { engine.transaction = original; }
+ expect(armed).toBe(false);
+}
+
+for (const { backend, databaseUrl } of pathologicalArms()) {
+ describe(`GST-1644 derived effects behind a concurrent publication (${backend})`, () => {
+ for (const at of ['prepared', 'after_commit'] as const) {
+ test(`a publication in flight (${at}) holds them as writer_busy with the short backoff; the next pass links`, async () => {
+ await managedBrain(async ({ engine, ctx }) => {
+ await seed(engine, ctx);
+ const other = await otherWriter(engine, ctx, at);
+ let publication: Promise<WriteRequest> | undefined;
+ await racedPass(engine, ctx, async () => { publication = other.start(); await other.paused; });
+ const held = await derived(engine);
+ // The links effect was claimed first; once the record exists the claim skips the worktree, so facts-backstop waits unclaimed.
+ expect(held.map(row => ({ kind: row.kind, state: row.state, error_code: row.error_code }))).toEqual([
+ { kind: 'facts-backstop', state: 'queued', error_code: null }, { kind: 'links', state: 'queued', error_code: 'writer_busy' }]);
+ expect(held.find(row => row.kind === 'links')!.delay_ms).toBeLessThan(5_000);
+ other.release();
+ expect((await publication!).state).toBe('committed');
+ await due(engine);
+ await runPersistenceEffects(engine, ctx.config, { hostId: localHostId(), limit: 20 });
+ expect((await derived(engine)).map(row => ({ kind: row.kind, state: row.state }))).toEqual([
+ { kind: 'facts-backstop', state: 'committed' }, { kind: 'links', state: 'committed' }]);
+ expect(await mentions(engine)).toEqual([{ slug: TARGET }]);
+ }, { databaseUrl });
+ }, 120_000);
+ }
+
+ test('a recovery whose publisher\'s claim lapsed still holds them as recovery_required; once it settles the next pass links', async () => {
+ await managedBrain(async ({ engine, ctx }) => {
+ await seed(engine, ctx);
+ const other = await otherWriter(engine, ctx, 'prepared');
+ let publication: Promise<WriteRequest> | undefined;
+ await racedPass(engine, ctx, async () => {
+ publication = other.start();
+ await other.paused;
+ await engine.transaction(async tx => {
+ await declarePersistenceProtocol(tx);
+ await tx.executeRaw("UPDATE persistence_requests SET claim_expires_at=now()-interval '1 second' WHERE id=$1::uuid", [other.row.id]);
+ });
+ });
+ const held = await derived(engine);
+ expect(held.map(row => ({ kind: row.kind, state: row.state, error_code: row.error_code }))).toEqual([
+ { kind: 'facts-backstop', state: 'queued', error_code: null }, { kind: 'links', state: 'queued', error_code: 'recovery_required' }]);
+ other.release();
+ await publication!.catch(() => undefined);
+ await due(engine);
+ await runPersistenceEffects(engine, ctx.config, { hostId: localHostId(), limit: 20 });
+ expect((await derived(engine)).map(row => ({ kind: row.kind, state: row.state }))).toEqual([
+ { kind: 'facts-backstop', state: 'committed' }, { kind: 'links', state: 'committed' }]);
+ expect(await mentions(engine)).toEqual([{ slug: TARGET }]);
+ }, { databaseUrl });
+ }, 120_000);
+ });
+}