Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: pre-landing review fixes, pass 2 (#8368)
garrytan82426792eaf2
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 62a83a5643..37770d21c2 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -59,7 +59,7 @@ of making a duplicate.| <a id="shared_origin"></a>`shared_origin` | `source_changed` (purge, or a delete whose survivor changed); a soft delete commits with `write_through.skipped: shared_origin` | Two live pages of one source recorded the same canonical file (a duplicate-origin twin: the shared file renders one of them, so a delete of either used to refuse `file_database_drift`, reconcile refused "Several pages claim the recorded canonical file" and every managed sync refused `page_identity_changed`). A soft delete (`gbrain delete <slug> --source-id <id> --expected-revision <rev>`) of a page whose file another live page records now keeps the file and publishes database-only: the selected database page is hidden; the file and the surviving page remain. This does not erase that file's contents. The receipt's `write_through.warning` says so and names the surviving slug to the trusted local owner, or to a remote writer whose grant and visibility cover that slug; other remote writers read a generic "another page of this source records the same file". Sync's census and both origin guards apply one rule: a live page wins over tombstones, several live pages refuse, a single tombstone may be reused, several tombstones stay ambiguous. An owner purge (`--purge`) of a shared-origin page refuses with this reason: a purge tombstones the content source-wide and a same-content survivor would refuse `purged_content` on its next import. The 72 h retention purge (the autopilot purge phase, `gbrain pages purge-deleted --older-than 72h --yes`) of such a page once it is soft-deleted writes no tombstone, so it removes the page row, its chunks, links and facts database-only and leaves the file and the surviving page untouched (#7311); a census that changed under it defers to the next run, which re-reads the claimants and, when the tombstone now owns the file alone, purges it normally, file included. A delete whose surviving claimant changed between preparation and publication (the file moved or removed, a survivor deleted or re-pointed) refuses and never removes the file as a fallback. | Delete the page that should not own the file; the kept file still holds whatever text it had, so edit it first if it holds the deleted page's text, then import its current content into the surviving page with `gbrain repair file-drift --source <id> --slug <surviving-slug> --prefer file` (preview, then the apply command it prints) and `gbrain sync --source <id> --no-pull`. Sync does not overwrite a surviving page that differs from the file: it holds it [`file_drift`](#file_drift), whose fix leads with the same command. For a purge of a live page, soft-delete it instead; a soft-deleted one is removed by the retention purge (a tombstone an older release parked after its refusals is retried once by the first purge run after the upgrade). After a `source_changed` on a changed survivor, submit the delete again with a new `request_id`; it re-reads the file's claimants. || <a id="writer-coordinator-required"></a>`database_guard` or `database_trigger` | `writer_coordinator_required` (or `storage_error` for another trigger) | A database trigger refused a row while a write was being prepared or published, so nothing was committed and any canonical file already replaced was restored. With `database_guard`, a gbrain process wrote a row its publication does not own (another source, an unresolvable source, or a source checkpoint/topology change outside owner administration). This is a gbrain defect or a version mismatch between gbrain processes sharing the database, not a content conflict. The exception is relationship `unreadable_state` (#7636): the database role gbrain connects as cannot see the writer state the guard reads (the `persistence_brain` or `git_canonical_brain` row, or the source or page row the write names) because row-level security hides it, or a `persistence_brain` / `git_canonical_brain` row is missing, so the guard refuses instead of letting the write through as if the brain were unmanaged or the row absent. A derived-state maintainer trigger raises the same refusal (#7637, raiser `gbrain_forget_page_retrievals`, `gbrain_queue_page_projection` and the others [listed in ENGINES.md](../ENGINES.md#derived-state-maintainers)) when row-level security applies to the role on a table it keeps up to date and the role holds no policy for it there, instead of leaving that derived state stale. The receipt's `write_error_detail` names the table, operation, guard branch, source relationship and stage; it never contains page text or source names. For another trigger it also carries a closed `reason`: `attribution_unclassified_table` or `trust_unclassified_table` (a gbrain stamp trigger met a table it is not built for: a version mismatch), `graduation_in_progress` (the engine-graduation fence, SQLSTATE 55000, refuses writes until the move finishes: retry later), or `unknown` (an operator trigger gbrain does not know; its message text stays on the brain host). The same `reason`, raiser and table appear on the managed sync failure (`gbrain doctor`, the failure ledger) and in `gbrain sync status --json` `last_error.write_error_detail`. | On the brain host run `gbrain sources writer status --probe --json`. Its `recent_failures[].error_detail` adds the source ids and the gbrain build and host that ran the attempt (`attempt.consumer_version`). If that build differs from `gbrain --version`, upgrade or restart every gbrain process that shares the database (serve, autopilot, sync jobs), then retry with a **new** request ID. If the builds match, report `error_detail` on the gbrain issue tracker. For relationship `unreadable_state` the build is not the cause: run gbrain as a role that can see that state (the table owner, a role that bypasses RLS, or a scoped role whose policies cover those rows, though a change to which sources are Git-canonical passes only the first two; see [writer-protocol fences](../ENGINES.md#opt-in-rls-source-scope-binding-gbrain_rls_scope_binding), and for a maintainer's table the [policies a scoped writer needs](../ENGINES.md#derived-state-maintainers)), then retry with a **new** request ID; a missing row, which `gbrain init` never leaves, is reported like a defect. Never force-write, edit SQL or disable the guard. || `ambiguous_source_path` | `page_identity_changed` | A source registered at a Git subfolder `<sub>` has a page whose stored path `<sub>/<file>` could mean either `<file>` in the source directory (the older Git-root spelling) or `<sub>/<file>` inside a folder of the source that repeats its name, and both files exist. Sync refuses instead of guessing. See [sources in a Git subfolder](multi-source-brains.md#sources-in-a-git-subfolder). | Rename or move one of the two files, commit, then `gbrain sync --source <source> --no-pull`. |-| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner (a cleanup or a restore that dropped dotfiles). No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released); a sync stops `blocked_by_failures` on the refused file, without a hold. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then run the same `gbrain sync` again. Never create or copy the marker by hand. |+| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner and identifies this exact directory: device, inode and birth time all equal (a cleanup or a restore that dropped dotfiles). A directory the reservation does not identify exactly, a birth time within the 1 µs tolerance included, keeps the generic `recovery_required` and its [recreated-root](#recreated-root) exit. No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released); a sync stops `blocked_by_failures` on the refused file, without a hold. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then run the same `gbrain sync` again. Never create or copy the marker by hand. || `physical_root_device_changed` | `recovery_required` | The checkout's filesystem device number changed while everything else matches, which macOS can do after a reboot. When the owner token, brain, worktree, root, inode and a non-zero birth time all match and the caller can verify database ownership, the write path re-stamps ownership by itself and the write proceeds. This refusal means the automatic re-stamp could not be verified; the suggestion says why. | Do a deliberate self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Retry the original write with the **same** request ID. Never delete ownership marker files. || `physical_root_directory_changed` | `recovery_required` | The directory at the checkout's recorded path is not the one the owner stamped: its inode changed, or its birth time differs from the stamp by more than 1 µs, while the owner token, brain, worktree, root and device all still match. A checkout deleted and recreated at the same path (restore from backup, rebuilt container, Migration Assistant copy) looks like this. A birth time within 1 µs of the stamp is accepted (#6501: APFS reports a new directory's birth time tens of nanoseconds apart across processes); a stamp of `0` against a filesystem that now reports a birth time, or a damaged stamp, still refuses. | A refused page write carries `detail: physical_root_directory_changed` and this self-transfer as its `fix` (`next: tell_user_to_run`, both phases): an agent relays it and never runs it. If the checkout was recreated at its recorded path, do a deliberate relocated-root self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. `writer status --json` prints the commands filled in. Retry the original write with the **same** request ID. Never delete ownership marker files. || <a id="ownership_record_not_private"></a>`ownership_record_not_private` | `recovery_required` | A private ownership record of the checkout (`.gbrain-owner.json` inside it, or `.gbrain-owner-<hash>.json` beside it) is readable by other users, typically after a recursive `chmod` of a copied brain, so it no longer proves who owns the checkout. Nothing was written; the message names no path. | The refusal's `fix` is `gbrain doctor --only foreign_ownership_marker` on the brain host, which names each such record and the exact `chmod 600 '<path>'`. Run it, then submit the write again with a new request_id. Never remove the record. |diff --git a/src/commands/doctor/checks/persistence-capacity.ts b/src/commands/doctor/checks/persistence-capacity.tsindex 7f9abee91e..2a88ce782d 100644--- a/src/commands/doctor/checks/persistence-capacity.ts+++ b/src/commands/doctor/checks/persistence-capacity.ts@@ -23,7 +23,8 @@ const OUTSTANDING_STALL_S = 600;* The fix verifies through doctor and the read-scope `get_brain_health` band (#6696 B2).* #8368: an outstanding-request cap at 90% or more whose oldest outstanding request is over ten minutes old is a warning* too, naming the sources holding its slots (requests that never settle hold them, so raising the cap is not the fix); its- * fix is the holder's read-only writer status. A healthy sync drain runs near the line with young requests and stays ok.+ * fix is the read-only writer status of the holder with the most stalled requests. A healthy sync drain runs near the line+ * with young requests and stays ok.*/export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Check> {try {@@ -38,11 +39,12 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Chefor (const row of capacityDiagnostics(counters, limits)) {if (row.resource !== 'outstanding_count' || row.limit <= 0 || row.used < Math.ceil(row.limit * OUTSTANDING_WARN)) continue;const principal = /^principal:([^:]+):(.+)$/.exec(row.scope);- const holders = await engine.executeRaw<{ source_id: string; n: number | string; oldest_s: number | string }>(`SELECT source_id,count(*) AS n,- EXTRACT(EPOCH FROM now()-min(created_at)) AS oldest_s FROM persistence_requests- WHERE state IN ('queued','running','recovering')${principal ? ' AND principal_kind=$1 AND principal_id=$2' : ''} GROUP BY source_id ORDER BY count(*) DESC,source_id LIMIT 3`,+ const stalled = `count(*) FILTER (WHERE created_at <= now()-interval '${OUTSTANDING_STALL_S} seconds')`;+ const holders = await engine.executeRaw<{ source_id: string; n: number | string; stalled: number | string }>(`SELECT source_id,count(*) AS n,${stalled} AS stalled+ FROM persistence_requests WHERE state IN ('queued','running','recovering')${principal ? ' AND principal_kind=$1 AND principal_id=$2' : ''}+ GROUP BY source_id ORDER BY ${stalled} DESC,count(*) DESC,source_id LIMIT 3`,principal ? [principal[1], principal[2]] : []);- if (!holders.some(holder => Number(holder.oldest_s) >= OUTSTANDING_STALL_S)) continue;+ if (!holders.length || Number(holders[0].stalled) === 0) continue;outstanding.push({ scope: row.scope, resource: row.resource, used: row.used, limit: row.limit, config_key: row.config_key,holders: holders.map(holder => ({ source_id: holder.source_id, outstanding: Number(holder.n) })) });}diff --git a/src/core/persistence/physical-root-record.ts b/src/core/persistence/physical-root-record.tsindex 8dbebc7151..f2af349edf 100644--- a/src/core/persistence/physical-root-record.ts+++ b/src/core/persistence/physical-root-record.ts@@ -253,10 +253,11 @@ export function physicalRootDirectoryChangedError(): OperationError {export function assertPhysicalRootStamp(directory: string, reservation: PhysicalRootReservation): void {const value = readPrivate(join(directory, PHYSICAL_ROOT_MARKER)) as PhysicalRootStamp | null;const info = statSync(directory, { bigint: true });- // #8368: no stamp in a directory the reservation still identifies (device, inode, birth) lost only its marker; a recreated- // or staged directory, or one the reservation cannot identify, keeps the generic refusal and the relocated-root exit.+ // #8368: no stamp in a directory the reservation still identifies exactly (device, inode, birth) lost only its marker, and+ // the plain self-transfer re-creates it. A recreated or staged directory, one whose birth only drifted within tolerance+ // (the self-transfer compares it exactly), or one the reservation cannot identify keeps the generic refusal.if (value === null && reservation.initialDevice === info.dev.toString() && reservation.initialInode === info.ino.toString()- && samePhysicalBirth(reservation.initialBirth, info.birthtimeNs)) throw ownerMarkerMissingError();+ && reservation.initialBirth === info.birthtimeNs.toString()) throw ownerMarkerMissingError();if (!value || value.version !== 1 || value.token !== reservation.token || value.brainId !== reservation.brainId || value.worktreeId !== reservation.worktreeId|| value.root !== reservation.root) throw physicalRootError();if (value.inode !== info.ino.toString() || !samePhysicalBirth(value.birth, info.birthtimeNs)) {diff --git a/test/write-cap-wedge-8368.test.ts b/test/write-cap-wedge-8368.test.tsindex 3330a6cf5f..82eba4b6ca 100644--- a/test/write-cap-wedge-8368.test.ts+++ b/test/write-cap-wedge-8368.test.ts@@ -102,13 +102,30 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (const lock = await acquireWorktree((await getWorktreeBinding(engine, s.id))!, 0, undefined, engine); expect(lock).not.toBeNull(); await lock?.release();}), 120_000);- test('(a) requests refused for a missing marker end terminally, release every principal slot and name the restore', () => withEnv(env(), async () => {+ test('(c) a birth time that only drifted within tolerance keeps the generic refusal the plain self-transfer would also refuse', () => withEnv(env(), async () => {+ const s = await managedSource({ 'notes/a.md': note('A') });+ const binding = (await getWorktreeBinding(engine, s.id))!;+ const reservation = JSON.parse(readFileSync(physicalRootReservationPath(s.root), 'utf8'));+ expect(BigInt(reservation.initialBirth)).toBeGreaterThan(0n);+ writeFileSync(physicalRootReservationPath(s.root), JSON.stringify({ ...reservation, initialBirth: (BigInt(reservation.initialBirth) + 500n).toString() }), { mode: 0o600 });+ rmSync(join(s.root, PHYSICAL_ROOT_MARKER));+ const refusal = await acquireWorktree(binding, 0, undefined, engine).then(() => null, (error: unknown) => error as Record<string, any>);+ expect(refusal).toMatchObject({ code: 'recovery_required' });+ await expect(runPersistenceAdministration(engine, 'writer_transfer_prepare', { source_id: s.id, self_transfer: true,+ ...await reviewedWriterIntent(engine, 'writer_transfer_prepare') })).rejects.toMatchObject({ code: 'recovery_required' });+ }), 120_000);++ for (const [code, fault] of [+ ['owner_marker_missing', (root: string) => rmSync(join(root, PHYSICAL_ROOT_MARKER), { force: true })],+ ['recovery_required', (root: string) => writeFileSync(join(root, PHYSICAL_ROOT_MARKER), JSON.stringify({ version: 1, token: randomUUID(), brainId: randomUUID(),+ worktreeId: randomUUID(), root, device: '1', inode: '1', birth: '1' }), { mode: 0o600 })],+ ] as const) test(`(a) requests refused ${code} by the ownership check end terminally, release every principal slot and name the fix`, () => withEnv(env(), async () => {const s = await managedSource({ 'notes/first.md': note('First') });expect((await performSync(engine, { sourceId: s.id, noPull: true, noEmbed: true, noExtract: true, drain: true })).status).toBe('first_sync');for (let i = 0; i < 40; i++) writeFileSync(join(s.root, 'notes', `n${i}.md`), note(`N${i}`));commit(s.root, 'more');// The marker goes missing while the drain runs: on Postgres the next publication group is claimed with its followers.- installFaultHook(async at => { if (at === 'sync:before_group_admission' || at === 'consumer:preparing') rmSync(join(s.root, PHYSICAL_ROOT_MARKER), { force: true }); });+ installFaultHook(async at => { if (at === 'sync:before_group_admission' || at === 'consumer:preparing') fault(s.root); });try {const run = performSync(engine, { sourceId: s.id, noPull: true, noEmbed: true, noExtract: true, drain: true }).then(r => r, (e: unknown) => e);const outcome = await Promise.race([run, new Promise(resolve => setTimeout(() => resolve('timed_out'), 60_000))]);@@ -118,9 +135,10 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (`SELECT state,error_code,claim_expires_at IS NOT NULL AS claimed,request_id::text AS request_id,slug,operation,error_message,error_detail,intent->>'kind' AS intent_kindFROM persistence_requests WHERE source_id=$1 AND state<>'committed' ORDER BY sequence`, [s.id]);expect(rows.length).toBeGreaterThan(0);- for (const row of rows) expect(row).toMatchObject({ state: 'failed', error_code: 'owner_marker_missing', claimed: false });+ for (const row of rows) expect(row).toMatchObject({ state: 'failed', error_code: code, claimed: false });const [principal] = await engine.executeRaw<{ outstanding: number }>(`SELECT max(outstanding_count)::int AS outstanding FROM persistence_counters WHERE key LIKE 'principal:%'`);expect(Number(principal!.outstanding)).toBe(0);+ if (code !== 'owner_marker_missing') return;// The receipt's diagnosis is typed and names the restore step, then a read-only verify.const verdict = classifyTerminal({ ...rows[0]!, source_id: s.id, error_detail: rows[0]!.error_detail as object | null, recovery: null, blocked_reason: null, compacted: false, publication_started: false });expect(verdict).toMatchObject({ code: 'owner_marker_missing', class: 'systemic', needs_human: true });@@ -179,6 +197,11 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (expect(past!.message).toContain(`source ${stuck} holds 4`);// The stuck source cannot borrow back: it already holds more than its share.expect(await refusedWith(stuck)).toMatchObject({ code: 'queue_capacity' });+ // Doctor names the source whose requests stopped settling first, even when a busier source with young requests outranks it.+ await engine.executeRaw(`UPDATE persistence_requests SET created_at=now() WHERE source_id=$1 AND state IN ('queued','running')`, [stuck]);+ await engine.executeRaw(`UPDATE persistence_requests SET created_at=now()-interval '1 hour' WHERE source_id=$1 AND state IN ('queued','running')`, [other]);+ expect(await checkPersistenceCapacity(engine)).toMatchObject({ status: 'warn',+ fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', other, '--json'] } });// The brain cap is never borrowed past.await engine.executeRaw(`UPDATE persistence_requests SET state='cancelled',execution_token=NULL WHERE source_id=$1 AND state='queued' AND id=(SELECT id FROM persistence_requests WHERE source_id=$1 AND state='queued' ORDER BY sequence DESC LIMIT 1)`, [other]);await engine.executeRaw(`UPDATE persistence_counters SET outstanding_count=outstanding_count-1 WHERE key='brain' OR key LIKE 'principal:%'`);