Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: adversarial review fixes (#8368)

garrytanc56e6479dab9
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bae20bbb81..30190c7ec2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,13 +14,13 @@ identifiers and attribution are available in the pre-removal Git revision
**One source's stuck writes can no longer starve another source's sync: a write refused for a missing ownership marker ends terminally with its own code and restore step, the shared local principal keeps a fair share for every source, and doctor names the source holding the slots (#8368)**
-On Postgres, a bulk sync group whose checkout failed the ownership check when its worktree lock was taken threw out of the consumer and stayed `running`; after each lease lapse the owner claimed it into the same refusal, forever, while the source kept admitting more. A hundred of them filled the local CLI principal's outstanding cap, which every source's sync shares, so another source's sync stopped `write_capacity` and doctor read `ok`. Now a deterministic ownership refusal finishes every claimed, unpublished member `failed` (the single path already did), which releases its slot; transient failures keep retrying as before. A missing `.gbrain-owner.json` is refused `owner_marker_missing` instead of the generic identity refusal, and its fix is the verified self-transfer that re-creates it. At the principal cap a source holding less than its fair share (the cap over the sources holding outstanding requests on the local CLI principal) is still admitted, the brain cap stays hard, and a refusal names the source holding the most slots. `persistence_capacity` warns at 90% of an outstanding-request cap and names the sources holding it.
+On Postgres, a bulk sync group whose checkout failed the ownership check when its worktree lock was taken threw out of the consumer and stayed `running`; after each lease lapse the owner claimed it into the same refusal, forever, while the source kept admitting more. A hundred of them filled the local CLI principal's outstanding cap, which every source's sync shares, so another source's sync stopped `write_capacity` and doctor read `ok`. Now a deterministic ownership refusal finishes every claimed, unpublished member `failed` (the single path already did), which releases its slot; transient failures keep retrying as before. A missing `.gbrain-owner.json` is refused `owner_marker_missing` instead of the generic identity refusal, and its fix is the verified self-transfer that re-creates it. At the principal cap a source holding less than its fair share (the cap over the sources holding outstanding requests on the local CLI principal) is still admitted, a sync publication group whole, the brain cap stays hard, and a refusal names the source holding the most slots. `persistence_capacity` warns at 90% of an outstanding-request cap and names the sources holding it.
### Itemized changes
- `src/core/persistence/consumer.ts`: `failRefusedClaims` settles a claimed group refused `recovery_required` or `owner_marker_missing` before publication, fenced on the claim token, no recovery record and publication not started.
- `src/core/persistence/owner-marker.ts`, `physical-root-record.ts`, `ownership.ts`: `owner_marker_missing` for an absent stamp under a valid reservation, with the self-transfer restore as its fix; a stamp naming another owner keeps `recovery_required`. Registered in `error-registry.ts`, `WRITE_ERROR_CODES`, the sync fault table and the terminal classifier (the receipt's fix is the restore step).
-- `src/core/persistence/journal.ts`: `outstandingRefusal` applies the local CLI principal fair share on single and group admission and names the top holder.
+- `src/core/persistence/journal.ts`: `outstandingRefusal` applies the local CLI principal fair share on single and group admission (checked before the admission, so a group larger than the share is admitted whole) and names the top holder.
- `src/commands/doctor/checks/persistence-capacity.ts`: `details.outstanding[]` with holders; a holder's writer status as the fix when it is the only finding.
- `test/write-cap-wedge-8368.test.ts` (PGLite and Postgres, with `test/e2e/write-cap-wedge-8368-postgres.test.ts`).
diff --git a/docs/guides/concurrent-writes.md b/docs/guides/concurrent-writes.md
index 123dc5415c..12ca548877 100644
--- a/docs/guides/concurrent-writes.md
+++ b/docs/guides/concurrent-writes.md
@@ -996,7 +996,7 @@ When `persistence_capacity` warns or fails, or a write is refused with
`fix` is the reversible unblock. `details.outstanding[]` lists an
outstanding-request cap at 90% or more whose oldest request is over ten
minutes old (a healthy drain near the line stays ok), with the sources holding its slots;
- requests that never settle hold them, so when it is the only finding, `fix`
+ requests that stopped settling (or wait behind a slow publication) hold them, so when it is the only finding, `fix`
is that source's read-only `gbrain sources writer status`, not a raise.
2. Run the `gbrain config set <key> <value>` it prints on the brain host
(undo with `gbrain config unset <key>`). Writes resume at once; retry the
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 37770d21c2..6093bc5528 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -69,7 +69,7 @@ of making a duplicate.
| `cursor_processing_options_conflict` | `invalid_params` | Retired in 0.60.255.0 (#6712): no longer returned. An unfinished sync's processing opt-outs (`--no-embed`, `--no-extract`, `--no-schema-pack`) are inherited by every run that resumes it, and the result names them in `inherited_options`. An older gbrain still returns it when a run's processing flags conflict with the cursor's. | On an older gbrain, run the printed resume command (`gbrain sync --source <source> --no-pull` plus the saved flags) or drop the conflicting flag. After upgrading, rerun the sync: it finishes the cursor with its stored opt-outs. |
| `take_row_collision` | `take_row_collision` | A save adds a takes-table row whose row number already belongs to a different take that exists only in the database. The save stops instead of overwriting that take. | Renumber the new takes row, or add the existing take to the page's takes table, then save with the current `expected_revision` and a **new** request ID (the content changed). |
| `invalid_source_uri` | `invalid_source_uri` | The brain has shared skillpacks, and the page's stored `source_uri` is a `file:` URI that cannot be turned into a local path, so gbrain cannot prove the write stays outside a skillpack. Shared-skill protection stays on. | The source owner inspects the page's stored `source_uri` on the brain host and replaces it with an absolute file URI or clears it; there is no dedicated command yet. Retry with a **new** request ID. |
-| `queue_capacity` | `queue_capacity` | Admission would exceed a write-journal limit. Existing requests keep their place; nothing is evicted. For the cumulative limits (lifetime request IDs and receipt bytes), `detail` names the limit, for example `principal_lifetime_ids`; for the outstanding-request caps, `detail` carries the counts (`outstanding=<used> limit=<limit>`), the message names the source holding the most of the local CLI principal's slots, and a managed `gbrain sync` drain waits them out ([`write_capacity`](#drain-write-capacity)). At the shared local CLI principal's outstanding cap, a page or sync write from a source holding less than its fair share (the cap divided by the sources holding outstanding requests on it) is still admitted, and the brain cap stays hard. Gates that read the principal's headroom before admission (maintenance backpressure, purge deferral, topology lifecycle checks) still count every source's requests. | For a cumulative limit, run the printed `gbrain config set persistence.limits.<limit> <value>` (sized for about one more year at the current rate), then retry with the **same** request ID. For outstanding-request, queued-byte or recovery-byte limits, let outstanding requests finish and check `gbrain sources writer status`. See [bounded admission and retention](concurrent-writes.md#bounded-admission-and-retention). |
+| `queue_capacity` | `queue_capacity` | Admission would exceed a write-journal limit. Existing requests keep their place; nothing is evicted. For the cumulative limits (lifetime request IDs and receipt bytes), `detail` names the limit, for example `principal_lifetime_ids`; for the outstanding-request caps, `detail` carries the counts (`outstanding=<used> limit=<limit>`), the message names the source holding the most of the local CLI principal's slots, and a managed `gbrain sync` drain waits them out ([`write_capacity`](#drain-write-capacity)). At the shared local CLI principal's outstanding cap, a page or sync write from a source holding less than its fair share (the cap divided by the sources holding outstanding requests on it) is still admitted, a sync publication group whole, so a source passes its share by at most one group; the brain cap stays hard. Gates that read the principal's headroom before admission (maintenance backpressure, purge deferral, topology lifecycle checks) still count every source's requests. | For a cumulative limit, run the printed `gbrain config set persistence.limits.<limit> <value>` (sized for about one more year at the current rate), then retry with the **same** request ID. For outstanding-request, queued-byte or recovery-byte limits, let outstanding requests finish and check `gbrain sources writer status`. See [bounded admission and retention](concurrent-writes.md#bounded-admission-and-retention). |
| <a id="checkpoint-validation-timeout"></a>`index_building`, `index_missing` or `indexes_valid` | `checkpoint_validation_timeout` | A managed sync's checkpoint validation (the check that every page receipt of the run committed) hit the coordinator's 5-second statement timeout on a large `persistence_requests` table. The checkpoint request failed terminally instead of being retried ahead of every other write to the source, so other writes proceed; `last_commit` is unchanged and every page the run already committed stays committed. `detail` says which of three states the request indexes were in when the hint was built. | `index_building`: migration 179 is still building the indexes; wait (doctor `persistence_request_indexes` shows progress), then run the printed retry. `index_missing`: run `gbrain repair request-indexes --apply` (it drops an INVALID index and rebuilds it concurrently on Postgres), then the printed retry. `indexes_valid`: run the printed retry once; if it times out again, report the request id with the `persistence_request_indexes` and `persistence_request_growth` entries of `gbrain doctor --json`. The retry is `gbrain sync --source <source> --no-pull` plus the run's `--repo` base and cursor options (`--full`, `--working-tree`, `--src-subpath`, `--exclude`, `--include-hidden`, `--strategy`) and saved processing flags, so it resumes the same cursor. Do not cancel the request: a cancelled checkpoint still blocks the next one. |
| <a id="connector-account-changed"></a>`account_changed`, `account_unresolved` | `connector_account_changed` | A managed Google or GitHub connector's credential resolves to a different account than the one pinned for the source (Google email, GitHub App installation or login), or to no account at all. Google checks every enabled service before any service runs. Nothing was imported. No reset flag authorizes an account change. | The suggestion gives two branches with real values. (A) Restore the credential for the pinned account (the configured `g_token_env`, `g_token_command`, vault credential, `gh_token_env` or GitHub App key), then `gbrain sync --source <source>`. (B) For a deliberate change, add a new source for the new account (`gbrain google setup --account <email>`, or `gbrain sources add <new-id> --kind github … --app-install <id>`), then `gbrain sources archive <source>`; existing pages stay under the old source. The refused run is recorded, and autopilot then skips the source (an `unavailable-path` skip in the fan-out, `freshness_source_path_skipped` with `code: connector_account_changed` in the freshness loop) instead of retrying it every interval; doctor `connector_account_mismatch` lists such sources. A `gbrain sync --source <source>` that passes the account check clears the skip, and so does a change to the source's connector config (one more automatic try). |
| <a id="connector-intent-outdated"></a>`pre_upgrade` | `connector_intent_outdated` | A connector write was admitted in the intent format retired in v0.60.11.0. With detail `pre_upgrade` it was admitted before this brain's upgrade; otherwise a connector host older than v0.60.11.0 admitted it. The consumer recovers any file publication in progress first, then fails the request. | `pre_upgrade`: nothing to do; the item is fetched again on the next `gbrain sync --source <source>` under a new request ID. Otherwise: `gbrain upgrade` on the host that runs connector jobs, then `gbrain sync --source <source>`. |
diff --git a/src/commands/doctor/checks/persistence-capacity.ts b/src/commands/doctor/checks/persistence-capacity.ts
index 2a88ce782d..6eb3ea3f4c 100644
--- a/src/commands/doctor/checks/persistence-capacity.ts
+++ b/src/commands/doctor/checks/persistence-capacity.ts
@@ -22,7 +22,8 @@ const OUTSTANDING_STALL_S = 600;
* A scope past the background pause line (A5) names the producers it pauses.
* The fix verifies through doctor and the read-scope `get_brain_health` band (#6696 B2).
* #8368: an outstanding-request cap at 90% or more whose oldest outstanding request is over ten minutes old is a warning
- * too, naming the sources holding its slots (requests that never settle hold them, so raising the cap is not the fix); its
+ * too, naming the sources holding its slots (requests that stopped settling, or wait behind a slow publication, hold them,
+ * so raising the cap is not the fix); its
* fix is the read-only writer status of the holder with the most stalled requests. A healthy sync drain runs near the line
* with young requests and stays ok.
*/
@@ -78,7 +79,7 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Che
const holder = outstanding.flatMap(row => row.holders)[0];
const held = outstanding.length ? `${outstanding.length} outstanding-request cap(s) at or near the limit: ${outstanding.map(row => `${row.scope} ${row.used}/${row.limit}`
+ (row.holders.length ? ` held by ${row.holders.map(h => `source ${h.source_id} (${h.outstanding})`).join(', ')}` : '')).join('; ')}. `
- + 'Admission refuses past the cap; requests that never settle hold it, so inspect the holder before raising anything. ' : '';
+ + 'Admission refuses past the cap; requests unsettled for over ten minutes hold it (stuck, or behind a slow publication), so inspect the holder before raising anything. ' : '';
if (!resources.length && outstanding.length) return { name: 'persistence_capacity', status: 'warn', details,
fix: { argv: holder ? ['gbrain', 'sources', 'writer', 'status', '--source', holder.source_id, '--json'] : ['gbrain', 'sources', 'writer', 'status', '--json'],
consent: [], actor: 'host_admin', requires_exclusive: false, verify: capacityVerify(), docs: DOCS,
diff --git a/src/core/persistence/consumer.ts b/src/core/persistence/consumer.ts
index 4c8c0e948d..ecf4ac02ce 100644
--- a/src/core/persistence/consumer.ts
+++ b/src/core/persistence/consumer.ts
@@ -1225,16 +1225,24 @@ export class PersistenceConsumer implements PersistenceConsumerLike {
* #8368: a claimed group refused before it published by a deterministic ownership refusal (the checkout identity check
* of the group's worktree lock) is finished the way the single path finishes one request: each member still running
* under this claim, unpublished and without a recovery record, ends failed with the refusal's code, which releases its
- * outstanding-request slot. Left running, every lease lapse re-claimed it into the same refusal, forever.
+ * outstanding-request slot. Left running, every lease lapse re-claimed it into the same refusal, forever. A member whose
+ * claim moved meanwhile is left to its new claimant; a connection loss gives the unfinished members back at once.
*/
private async failRefusedClaims(rows: WriteRequest[], error: OperationError): Promise<boolean> {
let progressed = false;
- for (const row of rows) {
- const current = await getWriteRequestById(this.engine, row.id);
- if (!current || current.state !== 'running' || current.execution_token !== row.execution_token || current.recovery || current.publication_started) continue;
- const done = await finishUnpublishedFailure(this.engine, current, error, 'publication');
- if (done.state === 'failed') this.log('publication', done.error_code ?? error.code, failureLogText(done), '', row);
- if (this.settled(done)) progressed = true;
+ for (const [index, row] of rows.entries()) {
+ try {
+ const current = await getWriteRequestById(this.engine, row.id);
+ if (!current || current.state !== 'running' || current.execution_token !== row.execution_token || current.recovery || current.publication_started) continue;
+ const done = await finishUnpublishedFailure(this.engine, current, error, 'publication');
+ if (done.state === 'failed') this.log('publication', done.error_code ?? error.code, failureLogText(done), '', row);
+ if (this.settled(done)) progressed = true;
+ } catch (failure) {
+ if (failure instanceof OperationError && failure.code === 'write_claim_lost') continue;
+ if (!transientDatabaseFailure(failure)) throw failure;
+ await this.releaseAfterConnectionLoss(rows.slice(index));
+ return progressed;
+ }
}
return progressed;
}
diff --git a/src/core/persistence/journal.ts b/src/core/persistence/journal.ts
index 535722ac9e..d66c61d05a 100644
--- a/src/core/persistence/journal.ts
+++ b/src/core/persistence/journal.ts
@@ -113,7 +113,9 @@ export function capacityError(resource: string, outstanding?: { used: number; li
* shared by every source's sync and maintenance on the host, so one source's requests that never settle could hold the
* whole principal cap and starve every other source. At that cap, a source holding less than its fair share (the cap
* divided by the sources holding outstanding requests on the principal, this one included) is still admitted; the brain
- * cap and every other principal's cap stay hard. A refusal names the source holding the most slots.
+ * cap and every other principal's cap stay hard. The share is checked before the admission, so a sync publication group
+ * larger than the share is admitted whole and a source passes its share by at most one group. A refusal names the source
+ * holding the most slots.
*/
async function outstandingRefusal(tx: SqlEngine, row: Counter, principal: Principal, sourceId: string, need: number, limit: number): Promise<OperationError | null> {
const used = Number(row.outstanding_count);
@@ -123,7 +125,7 @@ async function outstandingRefusal(tx: SqlEngine, row: Counter, principal: Princi
WHERE principal_kind=$1 AND principal_id=$2 AND state IN ('queued','running','recovering') GROUP BY source_id ORDER BY count(*) DESC,source_id`, [principal.kind, principal.id]);
const own = Number(holders.find(holder => holder.source_id === sourceId)?.n ?? 0);
const sources = holders.length + (own > 0 ? 0 : 1);
- if (sources > 1 && own + need <= Math.floor(limit / sources)) return null;
+ if (sources > 1 && own < Math.floor(limit / sources)) return null;
const top = holders[0];
return capacityError(`principal outstanding requests${top ? ` (source ${top.source_id} holds ${Number(top.n)})` : ''}`, { used, limit });
}
diff --git a/test/write-cap-wedge-8368.test.ts b/test/write-cap-wedge-8368.test.ts
index 82eba4b6ca..10359b05bb 100644
--- a/test/write-cap-wedge-8368.test.ts
+++ b/test/write-cap-wedge-8368.test.ts
@@ -22,9 +22,12 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import type { OperationContext } from '../src/core/ops/contract.ts';
import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';
import { acquireWorktree, claimWorktree, getWorktreeBinding } from '../src/core/persistence/ownership.ts';
-import { registerLocalWriter } from '../src/core/persistence/identity.ts';
+import { localHostId, registerLocalWriter } from '../src/core/persistence/identity.ts';
import { activatePersistence } from '../src/core/persistence/activation.ts';
-import { admitWrite, type WriteAdmission } from '../src/core/persistence/journal.ts';
+import { admitWrite, admitWriteGroupInTransaction, getWriteRequestById, type WriteAdmission } from '../src/core/persistence/journal.ts';
+import { PersistenceConsumer } from '../src/core/persistence/consumer.ts';
+import type { WriteRequest } from '../src/core/persistence/model.ts';
+import { editRequest, requestFixture } from './helpers/persistence-request-fixture.ts';
import { submissionAuthority } from '../src/core/persistence/authority.ts';
import { PHYSICAL_ROOT_MARKER, physicalRootReservationPath } from '../src/core/persistence/physical-root-record.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
@@ -207,6 +210,14 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
await engine.executeRaw(`UPDATE persistence_counters SET outstanding_count=outstanding_count-1 WHERE key='brain' OR key LIKE 'principal:%'`);
await engine.setConfig('persistence.limits.brain_outstanding', '5');
expect(await refusedWith(other)).toMatchObject({ code: 'queue_capacity', message: expect.stringContaining('brain outstanding') });
+ // A publication group larger than a starved source's share (4 / 3 sources = 1) is admitted whole, once.
+ await engine.unsetConfig('persistence.limits.brain_outstanding');
+ const third = await dbSource();
+ const group = async () => Promise.all(Array.from({ length: 3 }, () => admission(third)));
+ const members = await group();
+ expect(await engine.transaction(tx => admitWriteGroupInTransaction(tx, members))).toHaveLength(3);
+ const again = await group().then(members => engine.transaction(tx => admitWriteGroupInTransaction(tx, members))).then(() => null, (error: Record<string, any>) => error);
+ expect(again).toMatchObject({ code: 'queue_capacity' });
} finally {
await engine.unsetConfig('persistence.limits.principal_outstanding');
await engine.unsetConfig('persistence.limits.brain_outstanding');
@@ -215,6 +226,34 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
});
});
+test('#8368: a refused group member whose claim moved is left to its claimant and the rest still end failed', async () => {
+ const engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema();
+ try {
+ const fixture = await requestFixture(engine);
+ // A claimed group: three members running under one claim.
+ const claimed: WriteRequest[] = [];
+ for (const slug of ['notes/one', 'notes/two', 'notes/three']) {
+ const admitted = await fixture.admit(slug);
+ await editRequest(engine, admitted.id, "state='running',execution_token=$2::uuid,claim_expires_at=now()+interval '1 hour'", [randomUUID()]);
+ claimed.push((await getWriteRequestById(engine, admitted.id))!);
+ }
+ const consumer = new PersistenceConsumer(engine, { engine: 'pglite' }, async () => { throw new Error('unused'); }, { hostId: localHostId() });
+ // The first member's lease lapses and another claimant takes it between the consumer's read and its completion.
+ const transaction = engine.transaction.bind(engine);
+ let moved = false;
+ engine.transaction = (async (fn: Parameters<BrainEngine['transaction']>[0]) => {
+ if (!moved) { moved = true; await editRequest(engine, claimed[0]!.id, 'execution_token=$2::uuid', [randomUUID()]); }
+ return transaction(fn);
+ }) as BrainEngine['transaction'];
+ await (consumer as unknown as { failRefusedClaims(rows: unknown[], error: OperationError): Promise<boolean> })
+ .failRefusedClaims(claimed, new OperationError('owner_marker_missing', 'The ownership marker is missing.'));
+ engine.transaction = transaction;
+ const states = await engine.executeRaw<{ id: string; state: string; error_code: string | null }>(
+ 'SELECT id::text AS id,state,error_code FROM persistence_requests WHERE source_id=$1 ORDER BY sequence', [fixture.sourceId]);
+ expect(states.map(row => [row.state, row.error_code])).toEqual([['running', null], ['failed', 'owner_marker_missing'], ['failed', 'owner_marker_missing']]);
+ } finally { await engine.disconnect(); }
+}, 60_000);
+
test('#8368: owner_marker_missing stops a sync or connector run for the source instead of counting against one item', () => {
const refusal = new OperationError('owner_marker_missing', 'The ownership marker is missing.');
expect(classifyItemFailure(refusal).scope).not.toBe('item');