Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(persistence): release slots of deterministically refused writes, fair share on the local principal, owner_marker_missing, doctor cap holder (#8368)

garrytan8a61daca34e5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 620c007315..24c4c0fc92 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,20 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**One source's stuck writes can no longer starve another source's sync: a write refused for a missing ownership marker ends terminally with its own code and restore step, the shared local principal keeps a fair share for every source, and doctor names the source holding the slots (#8368)**
+
+On Postgres, a bulk sync group whose checkout failed the ownership check when its worktree lock was taken threw out of the consumer and stayed `running`; after each lease lapse the owner claimed it into the same refusal, forever, while the source kept admitting more. A hundred of them filled the local CLI principal's outstanding cap, which every source's sync shares, so another source's sync stopped `write_capacity` and doctor read `ok`. Now a deterministic ownership refusal finishes every claimed, unpublished member `failed` (the single path already did), which releases its slot; transient failures keep retrying as before. A missing `.gbrain-owner.json` is refused `owner_marker_missing` instead of the generic identity refusal, and its fix is the verified self-transfer that re-creates it. At the principal cap a source holding less than its fair share (the cap over the sources holding outstanding requests on the local CLI principal) is still admitted, the brain cap stays hard, and a refusal names the source holding the most slots. `persistence_capacity` warns at 90% of an outstanding-request cap and names the sources holding it.
+
+### Itemized changes
+
+- `src/core/persistence/consumer.ts`: `failRefusedClaims` settles a claimed group refused `recovery_required` or `owner_marker_missing` before publication, fenced on the claim token, no recovery record and publication not started.
+- `src/core/persistence/owner-marker.ts`, `physical-root-record.ts`, `ownership.ts`: `owner_marker_missing` for an absent stamp under a valid reservation, with the self-transfer restore as its fix; a stamp naming another owner keeps `recovery_required`. Registered in `error-registry.ts`, `WRITE_ERROR_CODES`, the sync fault table and the terminal classifier (the receipt's fix is the restore step).
+- `src/core/persistence/journal.ts`: `outstandingRefusal` applies the local CLI principal fair share on single and group admission and names the top holder.
+- `src/commands/doctor/checks/persistence-capacity.ts`: `details.outstanding[]` with holders; a holder's writer status as the fix when it is the only finding.
+- `test/write-cap-wedge-8368.test.ts` (PGLite and Postgres, with `test/e2e/write-cap-wedge-8368-postgres.test.ts`).
+
## [0.60.532.0] - 2026-11-01
**CI wave 3: a passed full-ci pair can now clear the remote gate without a second full suite on VMs, the import-rate scale gate no longer fails on one stall, an effect pass that loses its database connection gives its claims back at once, and the Ubicloud runner's teardown retries instead of leaking VMs (#7939, #8103, #8096, #7607, #8119).**
diff --git a/docs/guides/concurrent-writes.md b/docs/guides/concurrent-writes.md
index af330554ce..97825a4fb2 100644
--- a/docs/guides/concurrent-writes.md
+++ b/docs/guides/concurrent-writes.md
@@ -993,7 +993,10 @@ When `persistence_capacity` warns or fails, or a write is refused with
1. Read the finding: `gbrain doctor --only persistence_capacity --json`.
`details.resources[]` names the scope, resource, days left and `basis`;
- `fix` is the reversible unblock.
+ `fix` is the reversible unblock. `details.outstanding[]` lists an
+ outstanding-request cap at 90% or more with the sources holding its slots;
+ requests that never settle hold them, so when it is the only finding, `fix`
+ is that source's read-only `gbrain sources writer status`, not a raise.
2. Run the `gbrain config set <key> <value>` it prints on the brain host
(undo with `gbrain config unset <key>`). Writes resume at once; retry the
refused write with the same `request_id`.
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index 685531a3c9..74086a3cf6 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -2865,6 +2865,16 @@ More: [docs/guides/repair.md#origin-twins](../../docs/guides/repair.md#origin-tw
|---|---|---|---|---|---|---|
| Sources cannot claim overlapping canonical directories. | The request itself was wrong or no longer matches the brain; nothing was changed. | Correct the request using the message above, then retry. | agent | `repeat the read that failed` | 1 | no |
+### owner_marker_missing
+
+<a id="owner_marker_missing"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| The ownership marker (`.gbrain-owner.json`) is missing from a managed source's checkout root while the reservation beside the checkout still names this owner, so no write publishes there. | The marker binds the checkout directory to its owner. A cleanup or a restore that drops dotfiles removes it; a retry never brings it back, so a write refused with this code ends terminally and releases its outstanding-request slot. The verified self-transfer re-creates it for the same root on the owner host; creating or copying it by hand is refused. | On the brain host: gbrain sources writer status --source <id> --json (note admin_state), then gbrain sources writer transfer prepare <id> --self-transfer and transfer accept with the manifest it prints; then gbrain sources retry-held <id> and the same gbrain sync. Run: gbrain sources writer status --source '{source_id}' --json | host_admin | `gbrain doctor --json` | 1 | no |
+
+More: [docs/guides/write-refusals.md#owner_marker_missing](../../docs/guides/write-refusals.md#owner_marker_missing)
+
### owner_unavailable
<a id="owner_unavailable"></a>
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 25f3fb5beb..f016a2ce17 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -59,6 +59,7 @@ of making a duplicate.
| <a id="shared_origin"></a>`shared_origin` | `source_changed` (purge, or a delete whose survivor changed); a soft delete commits with `write_through.skipped: shared_origin` | Two live pages of one source recorded the same canonical file (a duplicate-origin twin: the shared file renders one of them, so a delete of either used to refuse `file_database_drift`, reconcile refused "Several pages claim the recorded canonical file" and every managed sync refused `page_identity_changed`). A soft delete (`gbrain delete <slug> --source-id <id> --expected-revision <rev>`) of a page whose file another live page records now keeps the file and publishes database-only: the selected database page is hidden; the file and the surviving page remain. This does not erase that file's contents. The receipt's `write_through.warning` says so and names the surviving slug to the trusted local owner, or to a remote writer whose grant and visibility cover that slug; other remote writers read a generic "another page of this source records the same file". Sync's census and both origin guards apply one rule: a live page wins over tombstones, several live pages refuse, a single tombstone may be reused, several tombstones stay ambiguous. An owner purge (`--purge`) of a shared-origin page refuses with this reason: a purge tombstones the content source-wide and a same-content survivor would refuse `purged_content` on its next import. The 72 h retention purge (the autopilot purge phase, `gbrain pages purge-deleted --older-than 72h --yes`) of such a page once it is soft-deleted writes no tombstone, so it removes the page row, its chunks, links and facts database-only and leaves the file and the surviving page untouched (#7311); a census that changed under it defers to the next run, which re-reads the claimants and, when the tombstone now owns the file alone, purges it normally, file included. A delete whose surviving claimant changed between preparation and publication (the file moved or removed, a survivor deleted or re-pointed) refuses and never removes the file as a fallback. | Delete the page that should not own the file; the kept file still holds whatever text it had, so edit it first if it holds the deleted page's text, then import its current content into the surviving page with `gbrain repair file-drift --source <id> --slug <surviving-slug> --prefer file` (preview, then the apply command it prints) and `gbrain sync --source <id> --no-pull`. Sync does not overwrite a surviving page that differs from the file: it holds it [`file_drift`](#file_drift), whose fix leads with the same command. For a purge of a live page, soft-delete it instead; a soft-deleted one is removed by the retention purge (a tombstone an older release parked after its refusals is retried once by the first purge run after the upgrade). After a `source_changed` on a changed survivor, submit the delete again with a new `request_id`; it re-reads the file's claimants. |
| <a id="writer-coordinator-required"></a>`database_guard` or `database_trigger` | `writer_coordinator_required` (or `storage_error` for another trigger) | A database trigger refused a row while a write was being prepared or published, so nothing was committed and any canonical file already replaced was restored. With `database_guard`, a gbrain process wrote a row its publication does not own (another source, an unresolvable source, or a source checkpoint/topology change outside owner administration). This is a gbrain defect or a version mismatch between gbrain processes sharing the database, not a content conflict. The exception is relationship `unreadable_state` (#7636): the database role gbrain connects as cannot see the writer state the guard reads (the `persistence_brain` or `git_canonical_brain` row, or the source or page row the write names) because row-level security hides it, or a `persistence_brain` / `git_canonical_brain` row is missing, so the guard refuses instead of letting the write through as if the brain were unmanaged or the row absent. A derived-state maintainer trigger raises the same refusal (#7637, raiser `gbrain_forget_page_retrievals`, `gbrain_queue_page_projection` and the others [listed in ENGINES.md](../ENGINES.md#derived-state-maintainers)) when row-level security applies to the role on a table it keeps up to date and the role holds no policy for it there, instead of leaving that derived state stale. The receipt's `write_error_detail` names the table, operation, guard branch, source relationship and stage; it never contains page text or source names. For another trigger it also carries a closed `reason`: `attribution_unclassified_table` or `trust_unclassified_table` (a gbrain stamp trigger met a table it is not built for: a version mismatch), `graduation_in_progress` (the engine-graduation fence, SQLSTATE 55000, refuses writes until the move finishes: retry later), or `unknown` (an operator trigger gbrain does not know; its message text stays on the brain host). The same `reason`, raiser and table appear on the managed sync failure (`gbrain doctor`, the failure ledger) and in `gbrain sync status --json` `last_error.write_error_detail`. | On the brain host run `gbrain sources writer status --probe --json`. Its `recent_failures[].error_detail` adds the source ids and the gbrain build and host that ran the attempt (`attempt.consumer_version`). If that build differs from `gbrain --version`, upgrade or restart every gbrain process that shares the database (serve, autopilot, sync jobs), then retry with a **new** request ID. If the builds match, report `error_detail` on the gbrain issue tracker. For relationship `unreadable_state` the build is not the cause: run gbrain as a role that can see that state (the table owner, a role that bypasses RLS, or a scoped role whose policies cover those rows, though a change to which sources are Git-canonical passes only the first two; see [writer-protocol fences](../ENGINES.md#opt-in-rls-source-scope-binding-gbrain_rls_scope_binding), and for a maintainer's table the [policies a scoped writer needs](../ENGINES.md#derived-state-maintainers)), then retry with a **new** request ID; a missing row, which `gbrain init` never leaves, is reported like a defect. Never force-write, edit SQL or disable the guard. |
| `ambiguous_source_path` | `page_identity_changed` | A source registered at a Git subfolder `<sub>` has a page whose stored path `<sub>/<file>` could mean either `<file>` in the source directory (the older Git-root spelling) or `<sub>/<file>` inside a folder of the source that repeats its name, and both files exist. Sync refuses instead of guessing. See [sources in a Git subfolder](multi-source-brains.md#sources-in-a-git-subfolder). | Rename or move one of the two files, commit, then `gbrain sync --source <source> --no-pull`. |
+| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner (a cleanup or a restore that dropped dotfiles). No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released) and a sync file is held with this code. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then `gbrain sources retry-held <source>` and the same `gbrain sync`. Never create or copy the marker by hand. |
| `physical_root_device_changed` | `recovery_required` | The checkout's filesystem device number changed while everything else matches, which macOS can do after a reboot. When the owner token, brain, worktree, root, inode and a non-zero birth time all match and the caller can verify database ownership, the write path re-stamps ownership by itself and the write proceeds. This refusal means the automatic re-stamp could not be verified; the suggestion says why. | Do a deliberate self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Retry the original write with the **same** request ID. Never delete ownership marker files. |
| `physical_root_directory_changed` | `recovery_required` | The directory at the checkout's recorded path is not the one the owner stamped: its inode changed, or its birth time differs from the stamp by more than 1 µs, while the owner token, brain, worktree, root and device all still match. A checkout deleted and recreated at the same path (restore from backup, rebuilt container, Migration Assistant copy) looks like this. A birth time within 1 µs of the stamp is accepted (#6501: APFS reports a new directory's birth time tens of nanoseconds apart across processes); a stamp of `0` against a filesystem that now reports a birth time, or a damaged stamp, still refuses. | If the checkout was recreated at its recorded path, do a deliberate relocated-root self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. `writer status --json` prints the commands filled in. Retry the original write with the **same** request ID. Never delete ownership marker files. |
| <a id="recreated-root"></a>recreated or moved canonical root | `recovery_required` | The checkout at the recorded path is a different directory (new inode and birth time after a restore, a rebuilt container or a Migration Assistant copy) or the recorded path now resolves through a symlink; without this release every verb, self-transfer included, refused with no exit (#5914). | Same path, directory recreated: the self-transfer above with `--confirm-relocated-root` on **both** phases; it waives only the directory identity comparison and needs the reservation beside the root (`.gbrain-owner-<hash>.json`) and the stamp to agree on token, brain, worktree, root and coordination path. Recorded path now a symlink to the moved checkout: `gbrain sources set-path <source> <path>` on the owner host keeps the worktree and moves its records. `gbrain sources writer deactivate --dry-run` lists `physical_root` when its real run would refuse, and the dry runs of `gbrain sources remove` and `gbrain sources set-path` refuse with the same `recovery_required` their real runs would (#6849). Runbook: [recreated root](../architecture/topologies.md#recreated-root). Never delete ownership marker files. |
@@ -67,7 +68,7 @@ of making a duplicate.
| `cursor_processing_options_conflict` | `invalid_params` | Retired in 0.60.255.0 (#6712): no longer returned. An unfinished sync's processing opt-outs (`--no-embed`, `--no-extract`, `--no-schema-pack`) are inherited by every run that resumes it, and the result names them in `inherited_options`. An older gbrain still returns it when a run's processing flags conflict with the cursor's. | On an older gbrain, run the printed resume command (`gbrain sync --source <source> --no-pull` plus the saved flags) or drop the conflicting flag. After upgrading, rerun the sync: it finishes the cursor with its stored opt-outs. |
| `take_row_collision` | `take_row_collision` | A save adds a takes-table row whose row number already belongs to a different take that exists only in the database. The save stops instead of overwriting that take. | Renumber the new takes row, or add the existing take to the page's takes table, then save with the current `expected_revision` and a **new** request ID (the content changed). |
| `invalid_source_uri` | `invalid_source_uri` | The brain has shared skillpacks, and the page's stored `source_uri` is a `file:` URI that cannot be turned into a local path, so gbrain cannot prove the write stays outside a skillpack. Shared-skill protection stays on. | The source owner inspects the page's stored `source_uri` on the brain host and replaces it with an absolute file URI or clears it; there is no dedicated command yet. Retry with a **new** request ID. |
-| `queue_capacity` | `queue_capacity` | Admission would exceed a write-journal limit. Existing requests keep their place; nothing is evicted. For the cumulative limits (lifetime request IDs and receipt bytes), `detail` names the limit, for example `principal_lifetime_ids`; for the outstanding-request caps, `detail` carries the counts (`outstanding=<used> limit=<limit>`), and a managed `gbrain sync` drain waits them out ([`write_capacity`](#drain-write-capacity)). | For a cumulative limit, run the printed `gbrain config set persistence.limits.<limit> <value>` (sized for about one more year at the current rate), then retry with the **same** request ID. For outstanding-request, queued-byte or recovery-byte limits, let outstanding requests finish and check `gbrain sources writer status`. See [bounded admission and retention](concurrent-writes.md#bounded-admission-and-retention). |
+| `queue_capacity` | `queue_capacity` | Admission would exceed a write-journal limit. Existing requests keep their place; nothing is evicted. For the cumulative limits (lifetime request IDs and receipt bytes), `detail` names the limit, for example `principal_lifetime_ids`; for the outstanding-request caps, `detail` carries the counts (`outstanding=<used> limit=<limit>`), the message names the source holding the most of the local CLI principal's slots, and a managed `gbrain sync` drain waits them out ([`write_capacity`](#drain-write-capacity)). One source cannot take every slot of the shared local CLI principal from the others: at that cap a source holding less than its fair share (the cap divided by the sources holding outstanding requests on it) is still admitted, and the brain cap stays hard. | For a cumulative limit, run the printed `gbrain config set persistence.limits.<limit> <value>` (sized for about one more year at the current rate), then retry with the **same** request ID. For outstanding-request, queued-byte or recovery-byte limits, let outstanding requests finish and check `gbrain sources writer status`. See [bounded admission and retention](concurrent-writes.md#bounded-admission-and-retention). |
| <a id="checkpoint-validation-timeout"></a>`index_building`, `index_missing` or `indexes_valid` | `checkpoint_validation_timeout` | A managed sync's checkpoint validation (the check that every page receipt of the run committed) hit the coordinator's 5-second statement timeout on a large `persistence_requests` table. The checkpoint request failed terminally instead of being retried ahead of every other write to the source, so other writes proceed; `last_commit` is unchanged and every page the run already committed stays committed. `detail` says which of three states the request indexes were in when the hint was built. | `index_building`: migration 179 is still building the indexes; wait (doctor `persistence_request_indexes` shows progress), then run the printed retry. `index_missing`: run `gbrain repair request-indexes --apply` (it drops an INVALID index and rebuilds it concurrently on Postgres), then the printed retry. `indexes_valid`: run the printed retry once; if it times out again, report the request id with the `persistence_request_indexes` and `persistence_request_growth` entries of `gbrain doctor --json`. The retry is `gbrain sync --source <source> --no-pull` plus the run's `--repo` base and cursor options (`--full`, `--working-tree`, `--src-subpath`, `--exclude`, `--include-hidden`, `--strategy`) and saved processing flags, so it resumes the same cursor. Do not cancel the request: a cancelled checkpoint still blocks the next one. |
| <a id="connector-account-changed"></a>`account_changed`, `account_unresolved` | `connector_account_changed` | A managed Google or GitHub connector's credential resolves to a different account than the one pinned for the source (Google email, GitHub App installation or login), or to no account at all. Google checks every enabled service before any service runs. Nothing was imported. No reset flag authorizes an account change. | The suggestion gives two branches with real values. (A) Restore the credential for the pinned account (the configured `g_token_env`, `g_token_command`, vault credential, `gh_token_env` or GitHub App key), then `gbrain sync --source <source>`. (B) For a deliberate change, add a new source for the new account (`gbrain google setup --account <email>`, or `gbrain sources add <new-id> --kind github … --app-install <id>`), then `gbrain sources archive <source>`; existing pages stay under the old source. The refused run is recorded, and autopilot then skips the source (an `unavailable-path` skip in the fan-out, `freshness_source_path_skipped` with `code: connector_account_changed` in the freshness loop) instead of retrying it every interval; doctor `connector_account_mismatch` lists such sources. A `gbrain sync --source <source>` that passes the account check clears the skip, and so does a change to the source's connector config (one more automatic try). |
| <a id="connector-intent-outdated"></a>`pre_upgrade` | `connector_intent_outdated` | A connector write was admitted in the intent format retired in v0.60.11.0. With detail `pre_upgrade` it was admitted before this brain's upgrade; otherwise a connector host older than v0.60.11.0 admitted it. The consumer recovers any file publication in progress first, then fails the request. | `pre_upgrade`: nothing to do; the item is fetched again on the next `gbrain sync --source <source>` under a new request ID. Otherwise: `gbrain upgrade` on the host that runs connector jobs, then `gbrain sync --source <source>`. |
diff --git a/scripts/structural-suites.tsv b/scripts/structural-suites.tsv
index bac40de485..7db13b2d1d 100644
--- a/scripts/structural-suites.tsv
+++ b/scripts/structural-suites.tsv
@@ -438,6 +438,7 @@ test/v0_37_gap_fill.serial.test.ts reinit-pglite — backup + reinit 8 source-su
test/w13-small-hardening.test.ts P1.14 paid-consent lint wording 1 exec-scan
test/worker-supervised-db-probe.test.ts issue #1801 fix #2 — supervised DB self-defense 5 readFileSync
test/write-attribution-legacy.test.ts write attribution inventory 1 readFileSync
+test/write-cap-wedge-8368.test.ts #8368 write-cap wedge (${kind}) 2 readFileSync
test/yaml-load-bounded-only.test.ts js-yaml load is imported only by data-frontmatter.ts 2 readFileSync
# unknown (detectors present, no attributable suite):
# unknown test/e2e/postgres-driver-install.test.ts
diff --git a/src/commands/doctor/checks/persistence-capacity.ts b/src/commands/doctor/checks/persistence-capacity.ts
index d9f8574567..196c64f888 100644
--- a/src/commands/doctor/checks/persistence-capacity.ts
+++ b/src/commands/doctor/checks/persistence-capacity.ts
@@ -9,6 +9,7 @@ import { BACKGROUND_PRODUCERS, readBackgroundPauseShare } from '../../../core/pe
const VERIFY = 'gbrain doctor --only persistence_capacity --json';
const DOCS = 'docs/guides/concurrent-writes.md#quota-near-exhaustion';
+const OUTSTANDING_WARN = 0.9;
/**
* #5470/#6696 (A3): a cumulative managed-write cap (permanent request IDs,
@@ -19,6 +20,8 @@ const DOCS = 'docs/guides/concurrent-writes.md#quota-near-exhaustion';
* their compacted (net) size. Each key gets the one value that covers a year.
* A scope past the background pause line (A5) names the producers it pauses.
* The fix verifies through doctor and the read-scope `get_brain_health` band (#6696 B2).
+ * #8368: an outstanding-request cap at 90% or more is a warning too, naming the sources holding its slots (requests
+ * that never settle hold them, so raising the cap is not the fix); its fix is the holder's read-only writer status.
*/
export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Check> {
try {
@@ -29,6 +32,16 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Che
const forecast: CapacityForecast | null = await doctorCapacityForecast(engine, { limits }).catch(forecastOrNull);
const pause = await readBackgroundPauseShare(engine);
const cumulative = capacityDiagnostics(counters, limits).filter(row => row.resource === 'lifetime_ids' || row.resource === 'terminal_bytes');
+ const outstanding = [];
+ for (const row of capacityDiagnostics(counters, limits)) {
+ if (row.resource !== 'outstanding_count' || row.limit <= 0 || row.used < Math.ceil(row.limit * OUTSTANDING_WARN)) continue;
+ const principal = /^principal:([^:]+):(.+)$/.exec(row.scope);
+ const holders = await engine.executeRaw<{ source_id: string; n: number | string }>(`SELECT source_id,count(*) AS n FROM persistence_requests
+ WHERE state IN ('queued','running','recovering')${principal ? ' AND principal_kind=$1 AND principal_id=$2' : ''} GROUP BY source_id ORDER BY count(*) DESC,source_id LIMIT 3`,
+ principal ? [principal[1], principal[2]] : []);
+ outstanding.push({ scope: row.scope, resource: row.resource, used: row.used, limit: row.limit, config_key: row.config_key,
+ holders: holders.map(holder => ({ source_id: holder.source_id, outstanding: Number(holder.n) })) });
+ }
const resources = [];
const needed = new Map<string, number>();
let worst: 'ok' | 'warn' | 'fail' = 'ok';
@@ -51,11 +64,20 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Che
}
const anyPaused = resources.some(r => r.background_paused);
const commands = [...needed].map(([key, value]) => `gbrain config set ${key} ${value}`);
- const details = { exact: true, inspected: counters.length, truncated: false, resources, commands, verify: VERIFY, docs: DOCS,
+ const details = { exact: true, inspected: counters.length, truncated: false, resources, outstanding, commands, verify: VERIFY, docs: DOCS,
background_pause_pct: Math.round(pause * 100), paused_producers: anyPaused ? [...BACKGROUND_PRODUCERS] : [],
forecast: forecast ? { thresholds: forecast.thresholds, window_days: forecast.window_days, retention_days: forecast.retention_days,
table_bytes: forecast.table_bytes, rows: forecast.rows } : { health: 'unknown' } };
const thresholds = forecast?.thresholds;
+ const holder = outstanding.flatMap(row => row.holders)[0];
+ const held = outstanding.length ? `${outstanding.length} outstanding-request cap(s) at or near the limit: ${outstanding.map(row => `${row.scope} ${row.used}/${row.limit}`
+ + (row.holders.length ? ` held by ${row.holders.map(h => `source ${h.source_id} (${h.outstanding})`).join(', ')}` : '')).join('; ')}. `
+ + 'Admission refuses past the cap; requests that never settle hold it, so inspect the holder before raising anything. ' : '';
+ if (!resources.length && outstanding.length) return { name: 'persistence_capacity', status: 'warn', details,
+ fix: { argv: holder ? ['gbrain', 'sources', 'writer', 'status', '--source', holder.source_id, '--json'] : ['gbrain', 'sources', 'writer', 'status', '--json'],
+ consent: [], actor: 'host_admin', requires_exclusive: false, verify: capacityVerify(), docs: DOCS,
+ why: `Read-only: names the requests${holder ? ` source ${holder.source_id}` : ''} holds and why they have not settled (a refused write ends terminally and frees its slot).` },
+ message: `${held}Run on the brain host: ${holder ? `gbrain sources writer status --source ${holder.source_id} --json` : 'gbrain sources writer status --json'}, then verify with ${VERIFY}.` };
if (!resources.length) return { name: 'persistence_capacity', status: 'ok', details,
message: thresholds ? `Managed write capacity is below ${thresholds.warn_pct}% of every cumulative limit and none runs out within ${thresholds.warn_days} days at its 7-day rate.`
: 'Managed write capacity is below 80% of every cumulative limit.' };
@@ -64,7 +86,7 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Che
fix: { argv: ['gbrain', 'config', 'set', first![0], String(first![1])], consent: [], actor: 'host_admin', requires_exclusive: false,
verify: capacityVerify(), docs: DOCS,
why: `Raises ${first![0]} to a value that covers about one more year at the 7-day rate; reversible with gbrain config unset.${commands.length > 1 ? ` Also run: ${commands.slice(1).join('; ')}.` : ''}` },
- message: `${resources.length} managed write limit(s) near exhaustion: ${resources.map(r => `${r.scope} ${r.resource} ${r.used}/${r.limit}`
+ message: `${held}${resources.length} managed write limit(s) near exhaustion: ${resources.map(r => `${r.scope} ${r.resource} ${r.used}/${r.limit}`
+ ('days_to_exhaustion' in r && r.days_to_exhaustion !== null ? ` (about ${r.days_to_exhaustion} day(s) left at the 7-day rate, ${r.basis})` : '')).join('; ')}. `
+ `Admission refuses at the limit. Run on the brain host: ${commands.join('; ')} (each covers about one more year at the current admission rate), `
+ `then verify with ${VERIFY}. Permanent request IDs are never evicted.`
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 8a6575475f..e958a45ace 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -489,6 +489,12 @@ export const CODES = {
no_brain: { class: 'caller', summary: "No brain is configured on this machine.", suggestion: 'Run `gbrain init --pglite --no-embedding` to create a local keyless brain (or `gbrain init --help` for hosted and Postgres options).' },
not_found: { class: 'caller', summary: "The requested resource does not exist or is not visible to this caller.", legacy_error: 'invalid_params' },
overlapping_path: { class: 'caller', summary: "Sources cannot claim overlapping canonical directories." },
+ owner_marker_missing: { class: 'host_only', retryable: false, actor: 'host_admin',
+ summary: "The ownership marker (`.gbrain-owner.json`) is missing from a managed source's checkout root while the reservation beside the checkout still names this owner, so no write publishes there.",
+ why: "The marker binds the checkout directory to its owner. A cleanup or a restore that drops dotfiles removes it; a retry never brings it back, so a write refused with this code ends terminally and releases its outstanding-request slot. The verified self-transfer re-creates it for the same root on the owner host; creating or copying it by hand is refused.",
+ docs: 'docs/guides/write-refusals.md#owner_marker_missing',
+ suggestion: 'On the brain host: gbrain sources writer status --source <id> --json (note admin_state), then gbrain sources writer transfer prepare <id> --self-transfer and transfer accept with the manifest it prints; then gbrain sources retry-held <id> and the same gbrain sync.',
+ fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'host_admin', why: 'Read-only: shows admin_state and the refused writes before the self-transfer that re-creates the marker.', requires_exclusive: false } },
owner_unavailable: { class: 'retryable', summary: "The source's canonical owner cannot run this write or maintenance step right now; `reason` says which condition.",
why: "A managed source publishes only through the host that owns its checkout. `host_mismatch`: another host id owns it (two GBRAIN_HOME identity files on one machine look the same; both ids are printed, and a retry on this host cannot help). `transfer_in_progress`: the worktree is draining for a writer transfer, which ends when the successor accepts it or the owner host runs `gbrain sources writer transfer cancel`; a drain prepared over 30 minutes ago asks the user which (`ask_user`) instead of waiting. `clone_in_progress`: a topology clone is recovering it and clears by itself. `binding_missing`, `incarnation_changed`, `local_path_missing` and `coordination_path_missing` name an incomplete or outdated registration. `not_sent` / `outcome_unknown`: the local owner's IPC lane did not take the write, or lost its acknowledgment. `detail: unbound_source`: on Postgres the source has a checkout but no canonical owner, a configuration state rather than a transient one (not retryable); in classic mode the write path is the file, a commit and `gbrain sync`.",
reasons: ['host_mismatch', 'transfer_in_progress', 'clone_in_progress', 'binding_missing', 'incarnation_changed', 'coordination_path_missing', 'local_path_missing', 'not_sent', 'outcome_unknown'],
diff --git a/src/core/persistence/consumer.ts b/src/core/persistence/consumer.ts
index 73e05465cc..4c8c0e948d 100644
--- a/src/core/persistence/consumer.ts
+++ b/src/core/persistence/consumer.ts
@@ -1217,9 +1217,27 @@ export class PersistenceConsumer implements PersistenceConsumerLike {
} finally { for (const member of rows) this.executing.delete(member.id); }
} catch (error) {
if (transientDatabaseFailure(error)) await this.releaseAfterConnectionLoss(claimed);
+ else if (error instanceof OperationError && (error.code === 'recovery_required' || error.code === 'owner_marker_missing')) return this.failRefusedClaims(claimed, error);
throw error;
}
}
+ /**
+ * #8368: a claimed group refused before it published by a deterministic ownership refusal (the checkout identity check
+ * of the group's worktree lock) is finished the way the single path finishes one request: each member still running
+ * under this claim, unpublished and without a recovery record, ends failed with the refusal's code, which releases its
+ * outstanding-request slot. Left running, every lease lapse re-claimed it into the same refusal, forever.
+ */
+ private async failRefusedClaims(rows: WriteRequest[], error: OperationError): Promise<boolean> {
+ let progressed = false;
+ for (const row of rows) {
+ const current = await getWriteRequestById(this.engine, row.id);
+ if (!current || current.state !== 'running' || current.execution_token !== row.execution_token || current.recovery || current.publication_started) continue;
+ const done = await finishUnpublishedFailure(this.engine, current, error, 'publication');
+ if (done.state === 'failed') this.log('publication', done.error_code ?? error.code, failureLogText(done), '', row);
+ if (this.settled(done)) progressed = true;
+ }
+ return progressed;
+ }
/**
* #6712 (convergence mode f): a pass whose database connection died (a pooler restart, a dropped connection) gives
* back every claim it took that did not commit, as soon as the database answers again, so the drain claims them anew
diff --git a/src/core/persistence/embedding-noop-settle.ts b/src/core/persistence/embedding-noop-settle.ts
index 6aac0044fd..fda40f12bc 100644
--- a/src/core/persistence/embedding-noop-settle.ts
+++ b/src/core/persistence/embedding-noop-settle.ts
@@ -40,7 +40,7 @@ import { refreshFenceClear } from './worktree-refresh-schema.ts';
export const NOOP_SETTLE_BATCH = 50;
/** Guard failures that leave an effect for the runner, which records them as it does today. */
-const LEFT_FOR_RUNNER = new Set(['owner_unavailable', 'source_changed', 'recovery_required']);
+const LEFT_FOR_RUNNER = new Set(['owner_unavailable', 'source_changed', 'recovery_required', 'owner_marker_missing']);
type Candidate = Pick<PersistenceEffect, 'id' | 'kind' | 'source_id' | 'source_incarnation' | 'worktree_id'>;
diff --git a/src/core/persistence/journal.ts b/src/core/persistence/journal.ts
index fc748a994e..535722ac9e 100644
--- a/src/core/persistence/journal.ts
+++ b/src/core/persistence/journal.ts
@@ -108,6 +108,25 @@ export function capacityError(resource: string, outstanding?: { used: number; li
if (outstanding) error.detail = outstandingCapacityDetail(outstanding.used, outstanding.limit);
return error;
}
+/**
+ * #8368: the outstanding-request refusal for `row`, or null when the write may be admitted. The local CLI principal is
+ * shared by every source's sync and maintenance on the host, so one source's requests that never settle could hold the
+ * whole principal cap and starve every other source. At that cap, a source holding less than its fair share (the cap
+ * divided by the sources holding outstanding requests on the principal, this one included) is still admitted; the brain
+ * cap and every other principal's cap stay hard. A refusal names the source holding the most slots.
+ */
+async function outstandingRefusal(tx: SqlEngine, row: Counter, principal: Principal, sourceId: string, need: number, limit: number): Promise<OperationError | null> {
+ const used = Number(row.outstanding_count);
+ if (used + need <= limit) return null;
+ if (row.key === 'brain' || principal.kind !== 'local_cli') return capacityError(`${row.key === 'brain' ? 'brain' : 'principal'} outstanding requests`, { used, limit });
+ const holders = await tx.executeRaw<{ source_id: string; n: number | string }>(`SELECT source_id,count(*) AS n FROM persistence_requests
+ WHERE principal_kind=$1 AND principal_id=$2 AND state IN ('queued','running','recovering') GROUP BY source_id ORDER BY count(*) DESC,source_id`, [principal.kind, principal.id]);
+ const own = Number(holders.find(holder => holder.source_id === sourceId)?.n ?? 0);
+ const sources = holders.length + (own > 0 ? 0 : 1);
+ if (sources > 1 && own + need <= Math.floor(limit / sources)) return null;
+ const top = holders[0];
+ return capacityError(`principal outstanding requests${top ? ` (source ${top.source_id} holds ${Number(top.n)})` : ''}`, { used, limit });
+}
/**
* #6696 (A6): a cumulative cap refusal is `write_quota_exhausted` (host only;
* waiting does not free a permanent id, and receipt bytes free only as
@@ -306,8 +325,8 @@ async function prepareAdmission(engine: BrainEngine, input: WriteAdmission, over
if (input.targetKind === 'skill_bundle') await assertSharedSkillPersistence(tx, input.sourceId);
for (const row of counters) {
const brain = row.key === 'brain';
- const outstandingLimit = brain ? limits.brainOutstanding : limits.principalOutstanding;
- if (Number(row.outstanding_count) + 1 > outstandingLimit) throw capacityError(`${brain ? 'brain' : 'principal'} outstanding requests`, { used: Number(row.outstanding_count), limit: outstandingLimit });
+ const outstanding = await outstandingRefusal(tx, row, input.principal, input.sourceId, 1, brain ? limits.brainOutstanding : limits.principalOutstanding);
+ if (outstanding) throw outstanding;
if (Number(row.intent_bytes) + bytes > (brain ? limits.brainIntentBytes : limits.principalIntentBytes)) throw capacityError(`${brain ? 'brain' : 'principal'} intent bytes`);
const scope = brain ? 'brain' : 'principal';
if (Number(row.lifetime_ids) + 1 > limits[`${scope}LifetimeIds`]) throw await cumulativeCapacityError(tx, `${scope} permanent request IDs`,
@@ -390,8 +409,8 @@ export async function admitWriteGroupInTransaction(tx: BrainEngine, inputs: Writ
const bytes = fresh.reduce((sum, item) => sum + item.bytes, 0), terminalBytes = fresh.reduce((sum, item) => sum + item.terminalBytes, 0);
for (const row of counters) {
const brain = row.key === 'brain', scope = brain ? 'brain' : 'principal';
- const outstandingLimit = brain ? limits.brainOutstanding : limits.principalOutstanding;
- if (Number(row.outstanding_count) + fresh.length > outstandingLimit) throw capacityError(`${scope} outstanding requests`, { used: Number(row.outstanding_count), limit: outstandingLimit });
+ const outstanding = await outstandingRefusal(tx, row, first.principal, first.sourceId, fresh.length, brain ? limits.brainOutstanding : limits.principalOutstanding);
+ if (outstanding) throw outstanding;
if (Number(row.intent_bytes) + bytes > (brain ? limits.brainIntentBytes : limits.principalIntentBytes)) throw capacityError(`${scope} intent bytes`);
if (Number(row.lifetime_ids) + fresh.length > limits[`${scope}LifetimeIds`]) throw await cumulativeCapacityError(tx, `${scope} permanent request IDs`,
row.key, `${scope}LifetimeIds`, Number(row.lifetime_ids), limits[`${scope}LifetimeIds`]);
diff --git a/src/core/persistence/owner-marker.ts b/src/core/persistence/owner-marker.ts
new file mode 100644
index 0000000000..5e7669be92
--- /dev/null
+++ b/src/core/persistence/owner-marker.ts
@@ -0,0 +1,38 @@
+import { OperationError } from '../ops/contract.ts';
+import type { Action } from '../agent-output.ts';
+import { PHYSICAL_ROOT_MARKER } from './root-metadata.ts';
+
+const DOCS = 'docs/guides/write-refusals.md#owner_marker_missing';
+
+/**
+ * #8368: the verified self-transfer that re-creates the ownership marker for the same root on this host: prepare, then
+ * accept. `root` and `epoch` are filled when the caller holds the binding; otherwise they are inputs from writer status.
+ */
+export function ownerMarkerRestoreFix(sourceId: string, binding?: { root: string; epoch: string }): Action {
+ const intent = (phase: string) => ['--admin-intent', `writer_transfer_${phase}`, '--expected-state', '<admin_state>'];
+ const adminState = { name: 'admin_state', how: `admin_state from gbrain sources writer status --source ${sourceId} --json on the owner host (read it again before the accept)` };
+ const prepare = ['gbrain', 'sources', 'writer', 'transfer', 'prepare', sourceId, '--self-transfer'];
+ return { argv: [...prepare, ...intent('prepare')], preview_argv: [...prepare, '--dry-run'], inputs: [adminState], consent: [], actor: 'host_admin', requires_exclusive: false, docs: DOCS,
+ why: `Source ${sourceId}'s checkout lost its ownership marker (${PHYSICAL_ROOT_MARKER}) while the reservation beside it still names this owner. The self-transfer verifies ownership and re-creates the marker for this same root on this host; never create or copy the marker by hand.`,
+ verify: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', sourceId, '--json'] },
+ then: { argv: ['gbrain', 'sources', 'writer', 'transfer', 'accept', sourceId, '--path', binding?.root ?? '<root>', '--expected-epoch', binding?.epoch ?? '<owner_epoch>',
+ '--manifest', '<manifest digest printed by prepare>', '--self-transfer', ...intent('accept')],
+ inputs: [adminState, ...(binding ? [] : [{ name: 'root', how: 'the binding root writer status prints' }, { name: 'owner_epoch', how: 'the owner_epoch prepare printed' }])],
+ consent: [], actor: 'host_admin', requires_exclusive: false,
+ why: 'Accepts the self-transfer: ownership stays on this host, the marker is written again and writes to the source resume. A sync file refused meanwhile is retried by gbrain sources retry-held and the same sync.',
+ verify: { argv: ['gbrain', 'sources', 'writer', 'failures', sourceId, '--json'] } } };
+}
+
+/**
+ * #8368: the ownership marker inside the checkout is gone while the outside-root reservation still names this owner (a
+ * cleanup or restore that dropped dotfiles). A retry never brings it back, so a write refused with it ends terminally;
+ * only the self-transfer in `fix` restores it. lockWorktree fills the source and root.
+ */
+export function ownerMarkerMissingError(sourceId?: string, binding?: { root: string; epoch: string }): OperationError {
+ const error = new OperationError('owner_marker_missing',
+ `The ownership marker ${PHYSICAL_ROOT_MARKER} is missing from the checkout root, while the reservation beside the checkout still names this owner.`,
+ `On the brain host, read gbrain sources writer status${sourceId ? ` --source ${sourceId}` : ''} --json and note admin_state, then run the verified self-transfer `
+ + `(gbrain sources writer transfer prepare ${sourceId ?? '<source>'} --self-transfer, then transfer accept with the manifest it prints) to re-create the marker. Never create or copy the marker by hand.`, DOCS);
+ error.fix = ownerMarkerRestoreFix(sourceId ?? '<source>', binding);
+ return error;
+}
diff --git a/src/core/persistence/ownership.ts b/src/core/persistence/ownership.ts
index bf2461f832..09913ad4fd 100644
--- a/src/core/persistence/ownership.ts
+++ b/src/core/persistence/ownership.ts
@@ -22,6 +22,7 @@ import { assertPhysicalRootReservation, readPhysicalRootStamp } from './physical
import { nativeFilesystemPath } from './root-registry.ts';
import { assertWriterAdminState } from './admin-intent.ts';
import { readGraduationRow } from './graduation-schema.ts';
+import { ownerMarkerMissingError } from './owner-marker.ts';
import { assertWriterAdminUnlocked } from './admin-lock.ts';
import { inspectPhysicalRootRecovery, repairPhysicalRoot, type PhysicalRootRecovery } from './physical-root-recovery.ts';
import type { ContentReceiptRebind } from '../shared-skills/receipt-root.ts';
@@ -225,6 +226,7 @@ async function lockWorktree(binding: WorktreeBinding, waitMs: number, signal: Ab
try { assertPhysicalRoot(binding.local_path, identity); return lock; }
catch (error) {
try {
+ if (error instanceof OperationError && error.code === 'owner_marker_missing') throw ownerMarkerMissingError(binding.source_id, { root: binding.local_path, epoch: String(binding.owner_epoch) });
if (error instanceof OperationError && error.detail === 'physical_root_directory_changed') {
// #6501: a directory recreated (or drifted beyond tolerance) at its recorded path; fill the #5914 exit for this source.
error.suggestion = `The directory at ${binding.local_path} is not the one this owner stamped (inode or birth time changed; owner token, brain, worktree, root and device match). `
diff --git a/src/core/persistence/physical-root-record.ts b/src/core/persistence/physical-root-record.ts
index 2e84dd90dc..fef4f2df8e 100644
--- a/src/core/persistence/physical-root-record.ts
+++ b/src/core/persistence/physical-root-record.ts
@@ -8,6 +8,7 @@ import { flushDirectory } from '../fs-durable.ts';
import { digest, sha256 } from './digest.ts';
import { canonicalFilesystemPath } from './root-registry.ts';
import { OWNERSHIP_MARKER_GLOBS, PHYSICAL_ROOT_MARKER } from './root-metadata.ts';
+import { ownerMarkerMissingError } from './owner-marker.ts';
export { OWNERSHIP_MARKER_GLOBS, PHYSICAL_ROOT_MARKER, isPhysicalRootMetadata } from './root-metadata.ts';
const RESERVATION_PREFIX = '.gbrain-owner-';
@@ -229,6 +230,10 @@ export function physicalRootDirectoryChangedError(): OperationError {
export function assertPhysicalRootStamp(directory: string, reservation: PhysicalRootReservation): void {
const value = readPrivate(join(directory, PHYSICAL_ROOT_MARKER)) as PhysicalRootStamp | null;
const info = statSync(directory, { bigint: true });
+ // #8368: no stamp in a directory the reservation still identifies (device, inode, birth) lost only its marker; a recreated
+ // or staged directory, or one the reservation cannot identify, keeps the generic refusal and the relocated-root exit.
+ if (value === null && reservation.initialDevice === info.dev.toString() && reservation.initialInode === info.ino.toString()
+ && samePhysicalBirth(reservation.initialBirth, info.birthtimeNs)) throw ownerMarkerMissingError();
if (!value || value.version !== 1 || value.token !== reservation.token || value.brainId !== reservation.brainId || value.worktreeId !== reservation.worktreeId
|| value.root !== reservation.root) throw physicalRootError();
if (value.inode !== info.ino.toString() || !samePhysicalBirth(value.birth, info.birthtimeNs)) {
diff --git a/src/core/persistence/sync-fault-class.ts b/src/core/persistence/sync-fault-class.ts
index c58d0313d6..75a6ff7ea7 100644
--- a/src/core/persistence/sync-fault-class.ts
+++ b/src/core/persistence/sync-fault-class.ts
@@ -171,6 +171,9 @@ export const SYNC_FAULT_TABLE: ReadonlyArray<SyncFaultRule> = [
{ code: 'preparation_systemic', class: 'systemic', safe_actions: ['upgrade', 'retry'], needs_human: true, escalate: 'always', human_reason: 'Several writes could not finish preparing: the write owner is the likely cause. Writer status names it; fix, restart or upgrade it.' },
{ code: 'drain_stalled', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'The write at the head of the source is wedged in its owner; restart the owner process writer status names.' },
{ code: 'recovery_required', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'A write needs recovery on the brain host (gbrain sources writer status --json names it).' },
+ // #8368: the checkout lost its ownership marker; only the verified self-transfer re-creates it.
+ { code: 'owner_marker_missing', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always',
+ human_reason: 'The checkout lost its ownership marker (.gbrain-owner.json). On the brain host, run the verified self-transfer (gbrain sources writer transfer prepare <source> --self-transfer, then transfer accept) to re-create it; never create it by hand. Then gbrain sources retry-held <source> and the same sync.' },
{ code: 'owner_unavailable', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'The write owner is gone; start or restart it on the brain host.' },
{ code: 'writer_coordinator_required', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'Managed persistence is not active for this source; activate it or sync unmanaged.' },
{ code: 'permission_denied', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'This principal may not sync the source; the owner grants it.' },
diff --git a/src/core/persistence/terminal-classify.ts b/src/core/persistence/terminal-classify.ts
index 6aed4ccfab..ee8f6105e7 100644
--- a/src/core/persistence/terminal-classify.ts
+++ b/src/core/persistence/terminal-classify.ts
@@ -25,6 +25,7 @@ import { classifySyncFault, SYNC_FAULT_TABLE, type SyncFaultClass, type SyncSafe
import { isSyncKnockOn, knockOnRoot, SYNC_CURSOR_FENCE_ORIGIN, SYNC_KNOCK_ON_ORIGIN, cursorFenceCause, syncKnockOnSql } from './sync-knock-on.ts';
import { writeFailureDiagnostic } from './verb-errors.ts';
import { publicCategory, type PublicCategory } from './public-category.ts';
+import { ownerMarkerRestoreFix } from './owner-marker.ts';
export type TerminalCategory =
| 'recovery_pending' | 'recovery_blocked'
@@ -296,6 +297,7 @@ function route(input: TerminalInput): Partial_ {
why: 'The page changed after the caller read it. Read the current revision and resubmit against it with a new request id.',
verify: { argv: failuresArgv(source) } } };
}
+ if (code === 'owner_marker_missing') return fromTable('sync_fault', code, origin, ownerMarkerRestoreFix(source));
if (FAULT_CODES.has(code)) return fromTable('sync_fault', code, origin, faultAction(code, input), null, input.error_message);
const entry = (CODES as Record<string, { class: string } | undefined>)[code];
if (!entry) return diagnosis(input, code, origin, `The write ended with code ${code}, which this build does not know.`);
diff --git a/src/core/persistence/topology-recovery.ts b/src/core/persistence/topology-recovery.ts
index 8b75d91e9f..60b576a248 100644
--- a/src/core/persistence/topology-recovery.ts
+++ b/src/core/persistence/topology-recovery.ts
@@ -192,7 +192,7 @@ export async function finishTopologyClone(engine:BrainEngine,id:string,hooks:Clo
row=await readChange(engine,id);
if(row.state!=='committed'){
const code=(error as {code?:string}).code;
- if(['permission_denied','source_changed','recovery_required'].includes(code??''))return abortClone(engine,row,record,code!);
+ if(['permission_denied','source_changed','recovery_required','owner_marker_missing'].includes(code??''))return abortClone(engine,row,record,code!);
throw error;
}
}
@@ -232,7 +232,7 @@ export async function recoverSourceTopologies(engine:BrainEngine,opts:{hostId?:s
for(const row of rows){
let finished=false;
try{await withTopologyLocks(engine,row.source_id,async()=>{const done=await finishTopologyClone(engine,row.id);if(!done.recovery){recovered++;finished=true;}},undefined,0);}
- catch(error){if(!(error instanceof OperationError&&['write_pending','recovery_required'].includes(error.code)))throw error;}
+ catch(error){if(!(error instanceof OperationError&&['write_pending','recovery_required','owner_marker_missing'].includes(error.code)))throw error;}
finally{opts.onAttempt?.(row.id,finished);}
}
return recovered;
diff --git a/src/core/persistence/types.ts b/src/core/persistence/types.ts
index 286087eb57..146f28b8d1 100644
--- a/src/core/persistence/types.ts
+++ b/src/core/persistence/types.ts
@@ -20,7 +20,7 @@ export const WRITE_ERROR_CODES = [
'core_budget_exceeded', 'core_mark_owner_only', 'core_remote_edit_refused', 'core_delete_owner_only',
'timeline_rows_would_be_removed', 'purged_content', 'preparation_stalled',
'fence_unrenderable', 'page_body_segmented', 'revision_changed', 'tail_changed',
- 'write_outcome_unknown', 'write_interrupted', 'connection_inherited_read_only',
+ 'write_outcome_unknown', 'write_interrupted', 'connection_inherited_read_only', 'owner_marker_missing',
] as const;
export type WriteErrorCode = typeof WRITE_ERROR_CODES[number];
diff --git a/test/e2e/write-cap-wedge-8368-postgres.test.ts b/test/e2e/write-cap-wedge-8368-postgres.test.ts
new file mode 100644
index 0000000000..08ffb8adc5
--- /dev/null
+++ b/test/e2e/write-cap-wedge-8368-postgres.test.ts
@@ -0,0 +1,3 @@
+import { registerPostgresTests } from '../helpers/test-backends.ts';
+
+await registerPostgresTests(() => import('../write-cap-wedge-8368.test.ts'));
diff --git a/test/persistence-physical-root.test.ts b/test/persistence-physical-root.test.ts
index 9854da872e..f196f387b2 100644
--- a/test/persistence-physical-root.test.ts
+++ b/test/persistence-physical-root.test.ts
@@ -114,9 +114,10 @@ test('missing or torn ownership metadata fails closed even for an existing owner
const f = await fixture();
const binding = await withEnv({ GBRAIN_HOME: f.homes[0] }, () => claimWorktree(engine, f.sources[0], f.root, f.hosts[0]));
rmSync(join(f.root, PHYSICAL_ROOT_MARKER));
- await expect(acquireWorktree(binding)).rejects.toMatchObject({ code: 'recovery_required' });
+ // #8368: a marker missing from the directory its reservation still identifies has its own code, and still fails closed.
+ await expect(acquireWorktree(binding)).rejects.toMatchObject({ code: 'owner_marker_missing' });
await withEnv({ GBRAIN_HOME: f.homes[0] }, async () => {
- await expect(claimWorktree(engine, f.sources[0], f.root, f.hosts[0])).rejects.toMatchObject({ code: 'recovery_required' });
+ await expect(claimWorktree(engine, f.sources[0], f.root, f.hosts[0])).rejects.toMatchObject({ code: 'owner_marker_missing' });
});
writeFileSync(physicalRootReservationPath(f.root), '{torn');
await expect(acquireWorktree(binding)).rejects.toMatchObject({ code: 'recovery_required' });
diff --git a/test/write-cap-wedge-8368.test.ts b/test/write-cap-wedge-8368.test.ts
new file mode 100644
index 0000000000..925f1dfdce
--- /dev/null
+++ b/test/write-cap-wedge-8368.test.ts
@@ -0,0 +1,187 @@
+/**
+ * #8368: one source's write requests that can never succeed held every slot of the shared local CLI principal's
+ * outstanding-request cap, so another source's sync refused `write_capacity` indefinitely.
+ *
+ * (a) A deterministic ownership refusal on a claimed publication group finishes every claimed member terminally (its
+ * slot is released) instead of leaving it running for the next claim; the single path already did.
+ * (b) At the principal cap, a source holding less than its fair share of the local CLI principal (the cap over the
+ * sources holding outstanding requests on it) is still admitted; the brain cap stays hard.
+ * (c) A missing ownership marker is refused `owner_marker_missing`, and its fix (the verified self-transfer) restores it.
+ * (d) doctor persistence_capacity warns at 90% of an outstanding-request cap and names the source holding the slots.
+ *
+ * Synthetic content only. PGLite and, with DATABASE_URL, Postgres (test/e2e/write-cap-wedge-8368-postgres.test.ts).
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { execFileSync } from 'node:child_process';
+import { randomUUID } from 'node:crypto';
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import type { OperationContext } from '../src/core/ops/contract.ts';
+import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';
+import { acquireWorktree, claimWorktree, getWorktreeBinding } from '../src/core/persistence/ownership.ts';
+import { registerLocalWriter } from '../src/core/persistence/identity.ts';
+import { activatePersistence } from '../src/core/persistence/activation.ts';
+import { admitWrite, type WriteAdmission } from '../src/core/persistence/journal.ts';
+import { submissionAuthority } from '../src/core/persistence/authority.ts';
+import { PHYSICAL_ROOT_MARKER, physicalRootReservationPath } from '../src/core/persistence/physical-root-record.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { installFaultHook } from '../src/core/persistence/fault-points.ts';
+import { classifyTerminal } from '../src/core/persistence/terminal-classify.ts';
+import { checkPersistenceCapacity } from '../src/commands/doctor/checks/persistence-capacity.ts';
+import { performSync } from '../src/commands/sync/perform.ts';
+import { CODES } from '../src/core/error-registry.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { reviewedWriterIntent } from './helpers/writer-admin-intent.ts';
+import { testBackends } from './helpers/test-backends.ts';
+import { withEnv } from './helpers/with-env.ts';
+
+const git = (root: string, ...args: string[]) => execFileSync('git', ['-C', root, ...args], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim();
+const commit = (root: string, message: string) => { git(root, 'add', '-A'); git(root, '-c', 'user.name=Example', '-c', '[email protected]', 'commit', '-qm', message); };
+const note = (title: string) => `---\ntitle: ${title}\ntype: note\n---\nA synthetic observation about ${title}.\n`;
+
+for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, () => {
+ const directory = realpathSync.native(mkdtempSync(join(tmpdir(), 'gbrain-wedge-8368-')));
+ let engine: BrainEngine, close: (() => Promise<void>) | undefined;
+ beforeAll(async () => {
+ if (kind === 'postgres') ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!));
+ else { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }
+ }, 120_000);
+ afterAll(async () => {
+ installFaultHook(undefined);
+ await withEnv({ GBRAIN_HOME: directory }, () => disposePersistenceConsumer(engine));
+ if (close) await close(); else await engine.disconnect();
+ rmSync(directory, { recursive: true, force: true });
+ });
+
+ /** A managed git source with `files`, claimed on this host. */
+ async function managedSource(files: Record<string, string>) {
+ const id = `wedge-${randomUUID().replace(/-/g, '').slice(0, 16)}`, root = join(directory, id);
+ mkdirSync(root, { recursive: true }); git(root, 'init', '-q');
+ for (const [path, content] of Object.entries(files)) { mkdirSync(join(root, path, '..'), { recursive: true }); writeFileSync(join(root, path), content); }
+ commit(root, 'fixture');
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');
+ await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::text::jsonb)', [id, root, '{}']);
+ await claimWorktree(engine, id, root);
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');
+ return { id, root };
+ }
+ const env = () => ({ GBRAIN_HOME: directory, GBRAIN_SYNC_FAILURES_DIR: directory, GBRAIN_BRAIN_ID: 'host', GBRAIN_SOURCE: undefined });
+
+ test('(c) a missing ownership marker refuses owner_marker_missing, and its fix (the verified self-transfer) restores writes', () => withEnv(env(), async () => {
+ const s = await managedSource({ 'notes/a.md': note('A') });
+ const binding = (await getWorktreeBinding(engine, s.id))!;
+ const reservation = readFileSync(physicalRootReservationPath(s.root), 'utf8');
+ rmSync(join(s.root, PHYSICAL_ROOT_MARKER));
+ const refusal = await acquireWorktree(binding, 0, undefined, engine).then(() => null, (error: unknown) => error as Record<string, any>);
+ expect(refusal).toMatchObject({ code: 'owner_marker_missing', docs: 'docs/guides/write-refusals.md#owner_marker_missing' });
+ expect(refusal!.message).toContain(PHYSICAL_ROOT_MARKER);
+ expect(refusal!.fix).toMatchObject({ actor: 'host_admin', argv: ['gbrain', 'sources', 'writer', 'transfer', 'prepare', s.id, '--self-transfer',
+ '--admin-intent', 'writer_transfer_prepare', '--expected-state', '<admin_state>'] });
+ expect(refusal!.fix.then.argv).toEqual(expect.arrayContaining(['gbrain', 'sources', 'writer', 'transfer', 'accept', s.id, '--path', s.root, '--self-transfer']));
+ expect(CODES.owner_marker_missing).toMatchObject({ class: 'host_only' });
+ // Nothing was rewritten by the refusal, and no marker was invented.
+ expect(existsSync(join(s.root, PHYSICAL_ROOT_MARKER))).toBe(false);
+ expect(readFileSync(physicalRootReservationPath(s.root), 'utf8')).toBe(reservation);
+ // A marker that is present but names another owner keeps the generic identity refusal.
+ writeFileSync(join(s.root, PHYSICAL_ROOT_MARKER), JSON.stringify({ version: 1, token: randomUUID(), brainId: randomUUID(), worktreeId: binding.worktree_id,
+ root: s.root, device: '1', inode: '1', birth: '1' }), { mode: 0o600 });
+ await expect(acquireWorktree(binding, 0, undefined, engine)).rejects.toMatchObject({ code: 'recovery_required' });
+ rmSync(join(s.root, PHYSICAL_ROOT_MARKER));
+ // The fix: prepare and accept the verified self-transfer for the same root on this host.
+ const administer = async (operation: 'writer_transfer_prepare' | 'writer_transfer_accept', params: Record<string, unknown>) =>
+ runPersistenceAdministration(engine, operation, { ...params, ...await reviewedWriterIntent(engine, operation) });
+ const prepared = await administer('writer_transfer_prepare', { source_id: s.id, self_transfer: true }) as { owner_epoch: string; manifest: { digest: string } };
+ await administer('writer_transfer_accept', { source_id: s.id, path: s.root, expected_epoch: prepared.owner_epoch, manifest: prepared.manifest.digest, self_transfer: true });
+ expect(existsSync(join(s.root, PHYSICAL_ROOT_MARKER))).toBe(true);
+ const lock = await acquireWorktree((await getWorktreeBinding(engine, s.id))!, 0, undefined, engine); expect(lock).not.toBeNull(); await lock?.release();
+ }), 120_000);
+
+ test('(a) requests refused for a missing marker end terminally, release every principal slot and name the restore', () => withEnv(env(), async () => {
+ const s = await managedSource({ 'notes/first.md': note('First') });
+ expect((await performSync(engine, { sourceId: s.id, noPull: true, noEmbed: true, noExtract: true, drain: true })).status).toBe('first_sync');
+ for (let i = 0; i < 40; i++) writeFileSync(join(s.root, 'notes', `n${i}.md`), note(`N${i}`));
+ commit(s.root, 'more');
+ // The marker goes missing while the drain runs: on Postgres the next publication group is claimed with its followers.
+ installFaultHook(async at => { if (at === 'sync:before_group_admission' || at === 'consumer:preparing') rmSync(join(s.root, PHYSICAL_ROOT_MARKER), { force: true }); });
+ try {
+ const run = performSync(engine, { sourceId: s.id, noPull: true, noEmbed: true, noExtract: true, drain: true }).then(r => r, (e: unknown) => e);
+ const outcome = await Promise.race([run, new Promise(resolve => setTimeout(() => resolve('timed_out'), 60_000))]);
+ expect(outcome).not.toBe('timed_out');
+ } finally { installFaultHook(undefined); }
+ const rows = await engine.executeRaw<{ state: string; error_code: string | null; claimed: boolean; request_id: string; slug: string; operation: string; error_message: string | null; error_detail: unknown; intent_kind: string | null }>(
+ `SELECT state,error_code,claim_expires_at IS NOT NULL AS claimed,request_id::text AS request_id,slug,operation,error_message,error_detail,intent->>'kind' AS intent_kind
+ FROM persistence_requests WHERE source_id=$1 AND state<>'committed' ORDER BY sequence`, [s.id]);
+ expect(rows.length).toBeGreaterThan(0);
+ for (const row of rows) expect(row).toMatchObject({ state: 'failed', error_code: 'owner_marker_missing', claimed: false });
+ const [principal] = await engine.executeRaw<{ outstanding: number }>(`SELECT max(outstanding_count)::int AS outstanding FROM persistence_counters WHERE key LIKE 'principal:%'`);
+ expect(Number(principal!.outstanding)).toBe(0);
+ // The receipt's diagnosis is typed and names the restore step, then a read-only verify.
+ const verdict = classifyTerminal({ ...rows[0]!, source_id: s.id, error_detail: rows[0]!.error_detail as object | null, recovery: null, blocked_reason: null, compacted: false, publication_started: false });
+ expect(verdict).toMatchObject({ code: 'owner_marker_missing', class: 'systemic', needs_human: true });
+ expect(verdict!.action.argv).toEqual(['gbrain', 'sources', 'writer', 'transfer', 'prepare', s.id, '--self-transfer', '--admin-intent', 'writer_transfer_prepare', '--expected-state', '<admin_state>']);
+ expect(verdict!.action.verify?.argv).toBeDefined();
+ }), 150_000);
+
+ describe('(b, d) one source cannot starve another on the shared local principal', () => {
+ async function dbSource() {
+ const id = `fair-${randomUUID().replace(/-/g, '').slice(0, 16)}`;
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');
+ await engine.executeRaw('INSERT INTO sources(id,name) VALUES($1,$1)', [id]);
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');
+ return id;
+ }
+ async function admission(sourceId: string): Promise<WriteAdmission> {
+ const [source] = await engine.executeRaw<{ incarnation: string }>('SELECT incarnation FROM sources WHERE id=$1', [sourceId]);
+ const slug = `notes/fair-${randomUUID().slice(0, 8)}`;
+ const context: OperationContext = { engine, sourceId, remote: false, dryRun: false, config: { engine: engine.kind }, logger: { info() {}, warn() {}, error() {} } };
+ const authority = await submissionAuthority(context, 'put_page', sourceId, source!.incarnation, slug);
+ const content = note('Fair share');
+ return { principal: authority.principal, operation: 'put_page', sourceId, sourceIncarnation: source!.incarnation, slug, pageId: null, requestId: randomUUID(),
+ callerIntent: { slug, content }, intent: { slug, content }, authority };
+ }
+ const refusedWith = (sourceId: string) => admission(sourceId).then(input => admitWrite(engine, input)).then(() => null, (error: Record<string, any>) => error);
+
+ test('a stuck source fills the cap; another source still gets its fair share, the brain cap stays hard, and doctor names the holder', () => withEnv(env(), async () => {
+ await disposePersistenceConsumer(engine);
+ await registerLocalWriter(engine, 'cli');
+ expect((await activatePersistence(engine, { confirmQuiesced: true })).enabled).toBe(true);
+ await engine.executeRaw(`UPDATE persistence_requests SET state='cancelled',execution_token=NULL,claim_expires_at=NULL WHERE state IN ('queued','running')`);
+ await engine.executeRaw(`UPDATE persistence_counters SET outstanding_count=0`);
+ const stuck = await dbSource(), other = await dbSource();
+ await engine.setConfig('persistence.limits.principal_outstanding', '4');
+ try {
+ for (let i = 0; i < 4; i++) await admitWrite(engine, await admission(stuck));
+ // A single source at the cap is refused as before, and the refusal names the source holding the slots.
+ const own = await refusedWith(stuck);
+ expect(own).toMatchObject({ code: 'queue_capacity', detail: 'outstanding=4 limit=4' });
+ expect(own!.message).toContain(`source ${stuck} holds 4`);
+ // (d) doctor warns at the cap and names the holder; its fix is the read-only writer status of that source.
+ const check = await checkPersistenceCapacity(engine);
+ expect(check.status).toBe('warn');
+ expect(check.message).toContain(stuck);
+ expect(check.details).toMatchObject({ outstanding: [expect.objectContaining({ resource: 'outstanding_count', used: 4, limit: 4,
+ holders: [expect.objectContaining({ source_id: stuck, outstanding: 4 })] })] });
+ expect(check.fix).toMatchObject({ argv: ['gbrain', 'sources', 'writer', 'status', '--source', stuck, '--json'] });
+ // (b) the other source is admitted up to its share (4 / 2 sources = 2), then refused naming the holder.
+ await admitWrite(engine, await admission(other));
+ await admitWrite(engine, await admission(other));
+ const past = await refusedWith(other);
+ expect(past).toMatchObject({ code: 'queue_capacity' });
+ expect(past!.message).toContain(`source ${stuck} holds 4`);
+ // The stuck source cannot borrow back: it already holds more than its share.
+ expect(await refusedWith(stuck)).toMatchObject({ code: 'queue_capacity' });
+ // The brain cap is never borrowed past.
+ await engine.executeRaw(`UPDATE persistence_requests SET state='cancelled',execution_token=NULL WHERE source_id=$1 AND state='queued' AND id=(SELECT id FROM persistence_requests WHERE source_id=$1 AND state='queued' ORDER BY sequence DESC LIMIT 1)`, [other]);
+ await engine.executeRaw(`UPDATE persistence_counters SET outstanding_count=outstanding_count-1 WHERE key='brain' OR key LIKE 'principal:%'`);
+ await engine.setConfig('persistence.limits.brain_outstanding', '5');
+ expect(await refusedWith(other)).toMatchObject({ code: 'queue_capacity', message: expect.stringContaining('brain outstanding') });
+ } finally {
+ await engine.unsetConfig('persistence.limits.principal_outstanding');
+ await engine.unsetConfig('persistence.limits.brain_outstanding');
+ }
+ }), 120_000);
+ });
+});