Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pre-landing review fixes, pass 2 (#8368)

garrytan82426792eaf2
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 62a83a5643..37770d21c2 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -59,7 +59,7 @@ of making a duplicate.
| <a id="shared_origin"></a>`shared_origin` | `source_changed` (purge, or a delete whose survivor changed); a soft delete commits with `write_through.skipped: shared_origin` | Two live pages of one source recorded the same canonical file (a duplicate-origin twin: the shared file renders one of them, so a delete of either used to refuse `file_database_drift`, reconcile refused "Several pages claim the recorded canonical file" and every managed sync refused `page_identity_changed`). A soft delete (`gbrain delete <slug> --source-id <id> --expected-revision <rev>`) of a page whose file another live page records now keeps the file and publishes database-only: the selected database page is hidden; the file and the surviving page remain. This does not erase that file's contents. The receipt's `write_through.warning` says so and names the surviving slug to the trusted local owner, or to a remote writer whose grant and visibility cover that slug; other remote writers read a generic "another page of this source records the same file". Sync's census and both origin guards apply one rule: a live page wins over tombstones, several live pages refuse, a single tombstone may be reused, several tombstones stay ambiguous. An owner purge (`--purge`) of a shared-origin page refuses with this reason: a purge tombstones the content source-wide and a same-content survivor would refuse `purged_content` on its next import. The 72 h retention purge (the autopilot purge phase, `gbrain pages purge-deleted --older-than 72h --yes`) of such a page once it is soft-deleted writes no tombstone, so it removes the page row, its chunks, links and facts database-only and leaves the file and the surviving page untouched (#7311); a census that changed under it defers to the next run, which re-reads the claimants and, when the tombstone now owns the file alone, purges it normally, file included. A delete whose surviving claimant changed between preparation and publication (the file moved or removed, a survivor deleted or re-pointed) refuses and never removes the file as a fallback. | Delete the page that should not own the file; the kept file still holds whatever text it had, so edit it first if it holds the deleted page's text, then import its current content into the surviving page with `gbrain repair file-drift --source <id> --slug <surviving-slug> --prefer file` (preview, then the apply command it prints) and `gbrain sync --source <id> --no-pull`. Sync does not overwrite a surviving page that differs from the file: it holds it [`file_drift`](#file_drift), whose fix leads with the same command. For a purge of a live page, soft-delete it instead; a soft-deleted one is removed by the retention purge (a tombstone an older release parked after its refusals is retried once by the first purge run after the upgrade). After a `source_changed` on a changed survivor, submit the delete again with a new `request_id`; it re-reads the file's claimants. |
| <a id="writer-coordinator-required"></a>`database_guard` or `database_trigger` | `writer_coordinator_required` (or `storage_error` for another trigger) | A database trigger refused a row while a write was being prepared or published, so nothing was committed and any canonical file already replaced was restored. With `database_guard`, a gbrain process wrote a row its publication does not own (another source, an unresolvable source, or a source checkpoint/topology change outside owner administration). This is a gbrain defect or a version mismatch between gbrain processes sharing the database, not a content conflict. The exception is relationship `unreadable_state` (#7636): the database role gbrain connects as cannot see the writer state the guard reads (the `persistence_brain` or `git_canonical_brain` row, or the source or page row the write names) because row-level security hides it, or a `persistence_brain` / `git_canonical_brain` row is missing, so the guard refuses instead of letting the write through as if the brain were unmanaged or the row absent. A derived-state maintainer trigger raises the same refusal (#7637, raiser `gbrain_forget_page_retrievals`, `gbrain_queue_page_projection` and the others [listed in ENGINES.md](../ENGINES.md#derived-state-maintainers)) when row-level security applies to the role on a table it keeps up to date and the role holds no policy for it there, instead of leaving that derived state stale. The receipt's `write_error_detail` names the table, operation, guard branch, source relationship and stage; it never contains page text or source names. For another trigger it also carries a closed `reason`: `attribution_unclassified_table` or `trust_unclassified_table` (a gbrain stamp trigger met a table it is not built for: a version mismatch), `graduation_in_progress` (the engine-graduation fence, SQLSTATE 55000, refuses writes until the move finishes: retry later), or `unknown` (an operator trigger gbrain does not know; its message text stays on the brain host). The same `reason`, raiser and table appear on the managed sync failure (`gbrain doctor`, the failure ledger) and in `gbrain sync status --json` `last_error.write_error_detail`. | On the brain host run `gbrain sources writer status --probe --json`. Its `recent_failures[].error_detail` adds the source ids and the gbrain build and host that ran the attempt (`attempt.consumer_version`). If that build differs from `gbrain --version`, upgrade or restart every gbrain process that shares the database (serve, autopilot, sync jobs), then retry with a **new** request ID. If the builds match, report `error_detail` on the gbrain issue tracker. For relationship `unreadable_state` the build is not the cause: run gbrain as a role that can see that state (the table owner, a role that bypasses RLS, or a scoped role whose policies cover those rows, though a change to which sources are Git-canonical passes only the first two; see [writer-protocol fences](../ENGINES.md#opt-in-rls-source-scope-binding-gbrain_rls_scope_binding), and for a maintainer's table the [policies a scoped writer needs](../ENGINES.md#derived-state-maintainers)), then retry with a **new** request ID; a missing row, which `gbrain init` never leaves, is reported like a defect. Never force-write, edit SQL or disable the guard. |
| `ambiguous_source_path` | `page_identity_changed` | A source registered at a Git subfolder `<sub>` has a page whose stored path `<sub>/<file>` could mean either `<file>` in the source directory (the older Git-root spelling) or `<sub>/<file>` inside a folder of the source that repeats its name, and both files exist. Sync refuses instead of guessing. See [sources in a Git subfolder](multi-source-brains.md#sources-in-a-git-subfolder). | Rename or move one of the two files, commit, then `gbrain sync --source <source> --no-pull`. |
-| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner (a cleanup or a restore that dropped dotfiles). No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released); a sync stops `blocked_by_failures` on the refused file, without a hold. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then run the same `gbrain sync` again. Never create or copy the marker by hand. |
+| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner and identifies this exact directory: device, inode and birth time all equal (a cleanup or a restore that dropped dotfiles). A directory the reservation does not identify exactly, a birth time within the 1 µs tolerance included, keeps the generic `recovery_required` and its [recreated-root](#recreated-root) exit. No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released); a sync stops `blocked_by_failures` on the refused file, without a hold. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then run the same `gbrain sync` again. Never create or copy the marker by hand. |
| `physical_root_device_changed` | `recovery_required` | The checkout's filesystem device number changed while everything else matches, which macOS can do after a reboot. When the owner token, brain, worktree, root, inode and a non-zero birth time all match and the caller can verify database ownership, the write path re-stamps ownership by itself and the write proceeds. This refusal means the automatic re-stamp could not be verified; the suggestion says why. | Do a deliberate self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Retry the original write with the **same** request ID. Never delete ownership marker files. |
| `physical_root_directory_changed` | `recovery_required` | The directory at the checkout's recorded path is not the one the owner stamped: its inode changed, or its birth time differs from the stamp by more than 1 µs, while the owner token, brain, worktree, root and device all still match. A checkout deleted and recreated at the same path (restore from backup, rebuilt container, Migration Assistant copy) looks like this. A birth time within 1 µs of the stamp is accepted (#6501: APFS reports a new directory's birth time tens of nanoseconds apart across processes); a stamp of `0` against a filesystem that now reports a birth time, or a damaged stamp, still refuses. | A refused page write carries `detail: physical_root_directory_changed` and this self-transfer as its `fix` (`next: tell_user_to_run`, both phases): an agent relays it and never runs it. If the checkout was recreated at its recorded path, do a deliberate relocated-root self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. `writer status --json` prints the commands filled in. Retry the original write with the **same** request ID. Never delete ownership marker files. |
| <a id="ownership_record_not_private"></a>`ownership_record_not_private` | `recovery_required` | A private ownership record of the checkout (`.gbrain-owner.json` inside it, or `.gbrain-owner-<hash>.json` beside it) is readable by other users, typically after a recursive `chmod` of a copied brain, so it no longer proves who owns the checkout. Nothing was written; the message names no path. | The refusal's `fix` is `gbrain doctor --only foreign_ownership_marker` on the brain host, which names each such record and the exact `chmod 600 '<path>'`. Run it, then submit the write again with a new request_id. Never remove the record. |
diff --git a/src/commands/doctor/checks/persistence-capacity.ts b/src/commands/doctor/checks/persistence-capacity.ts
index 7f9abee91e..2a88ce782d 100644
--- a/src/commands/doctor/checks/persistence-capacity.ts
+++ b/src/commands/doctor/checks/persistence-capacity.ts
@@ -23,7 +23,8 @@ const OUTSTANDING_STALL_S = 600;
* The fix verifies through doctor and the read-scope `get_brain_health` band (#6696 B2).
* #8368: an outstanding-request cap at 90% or more whose oldest outstanding request is over ten minutes old is a warning
* too, naming the sources holding its slots (requests that never settle hold them, so raising the cap is not the fix); its
- * fix is the holder's read-only writer status. A healthy sync drain runs near the line with young requests and stays ok.
+ * fix is the read-only writer status of the holder with the most stalled requests. A healthy sync drain runs near the line
+ * with young requests and stays ok.
*/
export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Check> {
try {
@@ -38,11 +39,12 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Che
for (const row of capacityDiagnostics(counters, limits)) {
if (row.resource !== 'outstanding_count' || row.limit <= 0 || row.used < Math.ceil(row.limit * OUTSTANDING_WARN)) continue;
const principal = /^principal:([^:]+):(.+)$/.exec(row.scope);
- const holders = await engine.executeRaw<{ source_id: string; n: number | string; oldest_s: number | string }>(`SELECT source_id,count(*) AS n,
- EXTRACT(EPOCH FROM now()-min(created_at)) AS oldest_s FROM persistence_requests
- WHERE state IN ('queued','running','recovering')${principal ? ' AND principal_kind=$1 AND principal_id=$2' : ''} GROUP BY source_id ORDER BY count(*) DESC,source_id LIMIT 3`,
+ const stalled = `count(*) FILTER (WHERE created_at <= now()-interval '${OUTSTANDING_STALL_S} seconds')`;
+ const holders = await engine.executeRaw<{ source_id: string; n: number | string; stalled: number | string }>(`SELECT source_id,count(*) AS n,${stalled} AS stalled
+ FROM persistence_requests WHERE state IN ('queued','running','recovering')${principal ? ' AND principal_kind=$1 AND principal_id=$2' : ''}
+ GROUP BY source_id ORDER BY ${stalled} DESC,count(*) DESC,source_id LIMIT 3`,
principal ? [principal[1], principal[2]] : []);
- if (!holders.some(holder => Number(holder.oldest_s) >= OUTSTANDING_STALL_S)) continue;
+ if (!holders.length || Number(holders[0].stalled) === 0) continue;
outstanding.push({ scope: row.scope, resource: row.resource, used: row.used, limit: row.limit, config_key: row.config_key,
holders: holders.map(holder => ({ source_id: holder.source_id, outstanding: Number(holder.n) })) });
}
diff --git a/src/core/persistence/physical-root-record.ts b/src/core/persistence/physical-root-record.ts
index 8dbebc7151..f2af349edf 100644
--- a/src/core/persistence/physical-root-record.ts
+++ b/src/core/persistence/physical-root-record.ts
@@ -253,10 +253,11 @@ export function physicalRootDirectoryChangedError(): OperationError {
export function assertPhysicalRootStamp(directory: string, reservation: PhysicalRootReservation): void {
const value = readPrivate(join(directory, PHYSICAL_ROOT_MARKER)) as PhysicalRootStamp | null;
const info = statSync(directory, { bigint: true });
- // #8368: no stamp in a directory the reservation still identifies (device, inode, birth) lost only its marker; a recreated
- // or staged directory, or one the reservation cannot identify, keeps the generic refusal and the relocated-root exit.
+ // #8368: no stamp in a directory the reservation still identifies exactly (device, inode, birth) lost only its marker, and
+ // the plain self-transfer re-creates it. A recreated or staged directory, one whose birth only drifted within tolerance
+ // (the self-transfer compares it exactly), or one the reservation cannot identify keeps the generic refusal.
if (value === null && reservation.initialDevice === info.dev.toString() && reservation.initialInode === info.ino.toString()
- && samePhysicalBirth(reservation.initialBirth, info.birthtimeNs)) throw ownerMarkerMissingError();
+ && reservation.initialBirth === info.birthtimeNs.toString()) throw ownerMarkerMissingError();
if (!value || value.version !== 1 || value.token !== reservation.token || value.brainId !== reservation.brainId || value.worktreeId !== reservation.worktreeId
|| value.root !== reservation.root) throw physicalRootError();
if (value.inode !== info.ino.toString() || !samePhysicalBirth(value.birth, info.birthtimeNs)) {
diff --git a/test/write-cap-wedge-8368.test.ts b/test/write-cap-wedge-8368.test.ts
index 3330a6cf5f..82eba4b6ca 100644
--- a/test/write-cap-wedge-8368.test.ts
+++ b/test/write-cap-wedge-8368.test.ts
@@ -102,13 +102,30 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
const lock = await acquireWorktree((await getWorktreeBinding(engine, s.id))!, 0, undefined, engine); expect(lock).not.toBeNull(); await lock?.release();
}), 120_000);
- test('(a) requests refused for a missing marker end terminally, release every principal slot and name the restore', () => withEnv(env(), async () => {
+ test('(c) a birth time that only drifted within tolerance keeps the generic refusal the plain self-transfer would also refuse', () => withEnv(env(), async () => {
+ const s = await managedSource({ 'notes/a.md': note('A') });
+ const binding = (await getWorktreeBinding(engine, s.id))!;
+ const reservation = JSON.parse(readFileSync(physicalRootReservationPath(s.root), 'utf8'));
+ expect(BigInt(reservation.initialBirth)).toBeGreaterThan(0n);
+ writeFileSync(physicalRootReservationPath(s.root), JSON.stringify({ ...reservation, initialBirth: (BigInt(reservation.initialBirth) + 500n).toString() }), { mode: 0o600 });
+ rmSync(join(s.root, PHYSICAL_ROOT_MARKER));
+ const refusal = await acquireWorktree(binding, 0, undefined, engine).then(() => null, (error: unknown) => error as Record<string, any>);
+ expect(refusal).toMatchObject({ code: 'recovery_required' });
+ await expect(runPersistenceAdministration(engine, 'writer_transfer_prepare', { source_id: s.id, self_transfer: true,
+ ...await reviewedWriterIntent(engine, 'writer_transfer_prepare') })).rejects.toMatchObject({ code: 'recovery_required' });
+ }), 120_000);
+
+ for (const [code, fault] of [
+ ['owner_marker_missing', (root: string) => rmSync(join(root, PHYSICAL_ROOT_MARKER), { force: true })],
+ ['recovery_required', (root: string) => writeFileSync(join(root, PHYSICAL_ROOT_MARKER), JSON.stringify({ version: 1, token: randomUUID(), brainId: randomUUID(),
+ worktreeId: randomUUID(), root, device: '1', inode: '1', birth: '1' }), { mode: 0o600 })],
+ ] as const) test(`(a) requests refused ${code} by the ownership check end terminally, release every principal slot and name the fix`, () => withEnv(env(), async () => {
const s = await managedSource({ 'notes/first.md': note('First') });
expect((await performSync(engine, { sourceId: s.id, noPull: true, noEmbed: true, noExtract: true, drain: true })).status).toBe('first_sync');
for (let i = 0; i < 40; i++) writeFileSync(join(s.root, 'notes', `n${i}.md`), note(`N${i}`));
commit(s.root, 'more');
// The marker goes missing while the drain runs: on Postgres the next publication group is claimed with its followers.
- installFaultHook(async at => { if (at === 'sync:before_group_admission' || at === 'consumer:preparing') rmSync(join(s.root, PHYSICAL_ROOT_MARKER), { force: true }); });
+ installFaultHook(async at => { if (at === 'sync:before_group_admission' || at === 'consumer:preparing') fault(s.root); });
try {
const run = performSync(engine, { sourceId: s.id, noPull: true, noEmbed: true, noExtract: true, drain: true }).then(r => r, (e: unknown) => e);
const outcome = await Promise.race([run, new Promise(resolve => setTimeout(() => resolve('timed_out'), 60_000))]);
@@ -118,9 +135,10 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
`SELECT state,error_code,claim_expires_at IS NOT NULL AS claimed,request_id::text AS request_id,slug,operation,error_message,error_detail,intent->>'kind' AS intent_kind
FROM persistence_requests WHERE source_id=$1 AND state<>'committed' ORDER BY sequence`, [s.id]);
expect(rows.length).toBeGreaterThan(0);
- for (const row of rows) expect(row).toMatchObject({ state: 'failed', error_code: 'owner_marker_missing', claimed: false });
+ for (const row of rows) expect(row).toMatchObject({ state: 'failed', error_code: code, claimed: false });
const [principal] = await engine.executeRaw<{ outstanding: number }>(`SELECT max(outstanding_count)::int AS outstanding FROM persistence_counters WHERE key LIKE 'principal:%'`);
expect(Number(principal!.outstanding)).toBe(0);
+ if (code !== 'owner_marker_missing') return;
// The receipt's diagnosis is typed and names the restore step, then a read-only verify.
const verdict = classifyTerminal({ ...rows[0]!, source_id: s.id, error_detail: rows[0]!.error_detail as object | null, recovery: null, blocked_reason: null, compacted: false, publication_started: false });
expect(verdict).toMatchObject({ code: 'owner_marker_missing', class: 'systemic', needs_human: true });
@@ -179,6 +197,11 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
expect(past!.message).toContain(`source ${stuck} holds 4`);
// The stuck source cannot borrow back: it already holds more than its share.
expect(await refusedWith(stuck)).toMatchObject({ code: 'queue_capacity' });
+ // Doctor names the source whose requests stopped settling first, even when a busier source with young requests outranks it.
+ await engine.executeRaw(`UPDATE persistence_requests SET created_at=now() WHERE source_id=$1 AND state IN ('queued','running')`, [stuck]);
+ await engine.executeRaw(`UPDATE persistence_requests SET created_at=now()-interval '1 hour' WHERE source_id=$1 AND state IN ('queued','running')`, [other]);
+ expect(await checkPersistenceCapacity(engine)).toMatchObject({ status: 'warn',
+ fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', other, '--json'] } });
// The brain cap is never borrowed past.
await engine.executeRaw(`UPDATE persistence_requests SET state='cancelled',execution_token=NULL WHERE source_id=$1 AND state='queued' AND id=(SELECT id FROM persistence_requests WHERE source_id=$1 AND state='queued' ORDER BY sequence DESC LIMIT 1)`, [other]);
await engine.executeRaw(`UPDATE persistence_counters SET outstanding_count=outstanding_count-1 WHERE key='brain' OR key LIKE 'principal:%'`);