Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pre-landing review fixes (#8368)

garrytan4278dceb3354
diff --git a/docs/guides/concurrent-writes.md b/docs/guides/concurrent-writes.md
index 97825a4fb2..123dc5415c 100644
--- a/docs/guides/concurrent-writes.md
+++ b/docs/guides/concurrent-writes.md
@@ -994,7 +994,8 @@ When `persistence_capacity` warns or fails, or a write is refused with
1. Read the finding: `gbrain doctor --only persistence_capacity --json`.
`details.resources[]` names the scope, resource, days left and `basis`;
`fix` is the reversible unblock. `details.outstanding[]` lists an
- outstanding-request cap at 90% or more with the sources holding its slots;
+ outstanding-request cap at 90% or more whose oldest request is over ten
+ minutes old (a healthy drain near the line stays ok), with the sources holding its slots;
requests that never settle hold them, so when it is the only finding, `fix`
is that source's read-only `gbrain sources writer status`, not a raise.
2. Run the `gbrain config set <key> <value>` it prints on the brain host
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index 60c1da3e60..ee69192b8f 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -2871,7 +2871,7 @@ More: [docs/guides/repair.md#origin-twins](../../docs/guides/repair.md#origin-tw
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| The ownership marker (`.gbrain-owner.json`) is missing from a managed source's checkout root while the reservation beside the checkout still names this owner, so no write publishes there. | The marker binds the checkout directory to its owner. A cleanup or a restore that drops dotfiles removes it; a retry never brings it back, so a write refused with this code ends terminally and releases its outstanding-request slot. The verified self-transfer re-creates it for the same root on the owner host; creating or copying it by hand is refused. | On the brain host: gbrain sources writer status --source <id> --json (note admin_state), then gbrain sources writer transfer prepare <id> --self-transfer and transfer accept with the manifest it prints; then gbrain sources retry-held <id> and the same gbrain sync. Run: gbrain sources writer status --source '{source_id}' --json | host_admin | `gbrain doctor --json` | 1 | no |
+| The ownership marker (`.gbrain-owner.json`) is missing from a managed source's checkout root while the reservation beside the checkout still names this owner, so no write publishes there. | The marker binds the checkout directory to its owner. A cleanup or a restore that drops dotfiles removes it; a retry never brings it back, so a write refused with this code ends terminally and releases its outstanding-request slot. The verified self-transfer re-creates it for the same root on the owner host; creating or copying it by hand is refused. | On the brain host: gbrain sources writer status --source <id> --json (note admin_state), then gbrain sources writer transfer prepare <id> --self-transfer and transfer accept with the manifest it prints; then run the same gbrain sync again (a sync stops blocked_by_failures on the refused file, with no hold). Run: gbrain sources writer status --source '{source_id}' --json | host_admin | `gbrain doctor --json` | 1 | no |
More: [docs/guides/write-refusals.md#owner_marker_missing](../../docs/guides/write-refusals.md#owner_marker_missing)
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 3e20793608..62a83a5643 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -59,7 +59,7 @@ of making a duplicate.
| <a id="shared_origin"></a>`shared_origin` | `source_changed` (purge, or a delete whose survivor changed); a soft delete commits with `write_through.skipped: shared_origin` | Two live pages of one source recorded the same canonical file (a duplicate-origin twin: the shared file renders one of them, so a delete of either used to refuse `file_database_drift`, reconcile refused "Several pages claim the recorded canonical file" and every managed sync refused `page_identity_changed`). A soft delete (`gbrain delete <slug> --source-id <id> --expected-revision <rev>`) of a page whose file another live page records now keeps the file and publishes database-only: the selected database page is hidden; the file and the surviving page remain. This does not erase that file's contents. The receipt's `write_through.warning` says so and names the surviving slug to the trusted local owner, or to a remote writer whose grant and visibility cover that slug; other remote writers read a generic "another page of this source records the same file". Sync's census and both origin guards apply one rule: a live page wins over tombstones, several live pages refuse, a single tombstone may be reused, several tombstones stay ambiguous. An owner purge (`--purge`) of a shared-origin page refuses with this reason: a purge tombstones the content source-wide and a same-content survivor would refuse `purged_content` on its next import. The 72 h retention purge (the autopilot purge phase, `gbrain pages purge-deleted --older-than 72h --yes`) of such a page once it is soft-deleted writes no tombstone, so it removes the page row, its chunks, links and facts database-only and leaves the file and the surviving page untouched (#7311); a census that changed under it defers to the next run, which re-reads the claimants and, when the tombstone now owns the file alone, purges it normally, file included. A delete whose surviving claimant changed between preparation and publication (the file moved or removed, a survivor deleted or re-pointed) refuses and never removes the file as a fallback. | Delete the page that should not own the file; the kept file still holds whatever text it had, so edit it first if it holds the deleted page's text, then import its current content into the surviving page with `gbrain repair file-drift --source <id> --slug <surviving-slug> --prefer file` (preview, then the apply command it prints) and `gbrain sync --source <id> --no-pull`. Sync does not overwrite a surviving page that differs from the file: it holds it [`file_drift`](#file_drift), whose fix leads with the same command. For a purge of a live page, soft-delete it instead; a soft-deleted one is removed by the retention purge (a tombstone an older release parked after its refusals is retried once by the first purge run after the upgrade). After a `source_changed` on a changed survivor, submit the delete again with a new `request_id`; it re-reads the file's claimants. |
| <a id="writer-coordinator-required"></a>`database_guard` or `database_trigger` | `writer_coordinator_required` (or `storage_error` for another trigger) | A database trigger refused a row while a write was being prepared or published, so nothing was committed and any canonical file already replaced was restored. With `database_guard`, a gbrain process wrote a row its publication does not own (another source, an unresolvable source, or a source checkpoint/topology change outside owner administration). This is a gbrain defect or a version mismatch between gbrain processes sharing the database, not a content conflict. The exception is relationship `unreadable_state` (#7636): the database role gbrain connects as cannot see the writer state the guard reads (the `persistence_brain` or `git_canonical_brain` row, or the source or page row the write names) because row-level security hides it, or a `persistence_brain` / `git_canonical_brain` row is missing, so the guard refuses instead of letting the write through as if the brain were unmanaged or the row absent. A derived-state maintainer trigger raises the same refusal (#7637, raiser `gbrain_forget_page_retrievals`, `gbrain_queue_page_projection` and the others [listed in ENGINES.md](../ENGINES.md#derived-state-maintainers)) when row-level security applies to the role on a table it keeps up to date and the role holds no policy for it there, instead of leaving that derived state stale. The receipt's `write_error_detail` names the table, operation, guard branch, source relationship and stage; it never contains page text or source names. For another trigger it also carries a closed `reason`: `attribution_unclassified_table` or `trust_unclassified_table` (a gbrain stamp trigger met a table it is not built for: a version mismatch), `graduation_in_progress` (the engine-graduation fence, SQLSTATE 55000, refuses writes until the move finishes: retry later), or `unknown` (an operator trigger gbrain does not know; its message text stays on the brain host). The same `reason`, raiser and table appear on the managed sync failure (`gbrain doctor`, the failure ledger) and in `gbrain sync status --json` `last_error.write_error_detail`. | On the brain host run `gbrain sources writer status --probe --json`. Its `recent_failures[].error_detail` adds the source ids and the gbrain build and host that ran the attempt (`attempt.consumer_version`). If that build differs from `gbrain --version`, upgrade or restart every gbrain process that shares the database (serve, autopilot, sync jobs), then retry with a **new** request ID. If the builds match, report `error_detail` on the gbrain issue tracker. For relationship `unreadable_state` the build is not the cause: run gbrain as a role that can see that state (the table owner, a role that bypasses RLS, or a scoped role whose policies cover those rows, though a change to which sources are Git-canonical passes only the first two; see [writer-protocol fences](../ENGINES.md#opt-in-rls-source-scope-binding-gbrain_rls_scope_binding), and for a maintainer's table the [policies a scoped writer needs](../ENGINES.md#derived-state-maintainers)), then retry with a **new** request ID; a missing row, which `gbrain init` never leaves, is reported like a defect. Never force-write, edit SQL or disable the guard. |
| `ambiguous_source_path` | `page_identity_changed` | A source registered at a Git subfolder `<sub>` has a page whose stored path `<sub>/<file>` could mean either `<file>` in the source directory (the older Git-root spelling) or `<sub>/<file>` inside a folder of the source that repeats its name, and both files exist. Sync refuses instead of guessing. See [sources in a Git subfolder](multi-source-brains.md#sources-in-a-git-subfolder). | Rename or move one of the two files, commit, then `gbrain sync --source <source> --no-pull`. |
-| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner (a cleanup or a restore that dropped dotfiles). No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released) and a sync file is held with this code. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then `gbrain sources retry-held <source>` and the same `gbrain sync`. Never create or copy the marker by hand. |
+| <a id="owner_marker_missing"></a>`owner_marker_missing` | `owner_marker_missing` | The ownership marker `.gbrain-owner.json` is gone from the checkout root while the reservation beside the checkout still names this owner (a cleanup or a restore that dropped dotfiles). No write publishes there, and a retry never brings the marker back, so the owner finishes each refused write `failed` (its outstanding-request slot is released); a sync stops `blocked_by_failures` on the refused file, without a hold. | Re-create the marker with the verified self-transfer on the brain host: `gbrain sources writer status --source <source> --json` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Then run the same `gbrain sync` again. Never create or copy the marker by hand. |
| `physical_root_device_changed` | `recovery_required` | The checkout's filesystem device number changed while everything else matches, which macOS can do after a reboot. When the owner token, brain, worktree, root, inode and a non-zero birth time all match and the caller can verify database ownership, the write path re-stamps ownership by itself and the write proceeds. This refusal means the automatic re-stamp could not be verified; the suggestion says why. | Do a deliberate self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. Retry the original write with the **same** request ID. Never delete ownership marker files. |
| `physical_root_directory_changed` | `recovery_required` | The directory at the checkout's recorded path is not the one the owner stamped: its inode changed, or its birth time differs from the stamp by more than 1 µs, while the owner token, brain, worktree, root and device all still match. A checkout deleted and recreated at the same path (restore from backup, rebuilt container, Migration Assistant copy) looks like this. A birth time within 1 µs of the stamp is accepted (#6501: APFS reports a new directory's birth time tens of nanoseconds apart across processes); a stamp of `0` against a filesystem that now reports a birth time, or a damaged stamp, still refuses. | A refused page write carries `detail: physical_root_directory_changed` and this self-transfer as its `fix` (`next: tell_user_to_run`, both phases): an agent relays it and never runs it. If the checkout was recreated at its recorded path, do a deliberate relocated-root self-transfer: `gbrain sources writer status <source>` (note `admin_state`), then `gbrain sources writer transfer prepare <source> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_prepare --expected-state <admin_state>`, then `gbrain sources writer transfer accept <source> --path <root> --expected-epoch <epoch> --manifest <digest from prepare> --self-transfer --confirm-relocated-root --admin-intent writer_transfer_accept --expected-state <fresh admin_state>`. `writer status --json` prints the commands filled in. Retry the original write with the **same** request ID. Never delete ownership marker files. |
| <a id="ownership_record_not_private"></a>`ownership_record_not_private` | `recovery_required` | A private ownership record of the checkout (`.gbrain-owner.json` inside it, or `.gbrain-owner-<hash>.json` beside it) is readable by other users, typically after a recursive `chmod` of a copied brain, so it no longer proves who owns the checkout. Nothing was written; the message names no path. | The refusal's `fix` is `gbrain doctor --only foreign_ownership_marker` on the brain host, which names each such record and the exact `chmod 600 '<path>'`. Run it, then submit the write again with a new request_id. Never remove the record. |
@@ -69,7 +69,7 @@ of making a duplicate.
| `cursor_processing_options_conflict` | `invalid_params` | Retired in 0.60.255.0 (#6712): no longer returned. An unfinished sync's processing opt-outs (`--no-embed`, `--no-extract`, `--no-schema-pack`) are inherited by every run that resumes it, and the result names them in `inherited_options`. An older gbrain still returns it when a run's processing flags conflict with the cursor's. | On an older gbrain, run the printed resume command (`gbrain sync --source <source> --no-pull` plus the saved flags) or drop the conflicting flag. After upgrading, rerun the sync: it finishes the cursor with its stored opt-outs. |
| `take_row_collision` | `take_row_collision` | A save adds a takes-table row whose row number already belongs to a different take that exists only in the database. The save stops instead of overwriting that take. | Renumber the new takes row, or add the existing take to the page's takes table, then save with the current `expected_revision` and a **new** request ID (the content changed). |
| `invalid_source_uri` | `invalid_source_uri` | The brain has shared skillpacks, and the page's stored `source_uri` is a `file:` URI that cannot be turned into a local path, so gbrain cannot prove the write stays outside a skillpack. Shared-skill protection stays on. | The source owner inspects the page's stored `source_uri` on the brain host and replaces it with an absolute file URI or clears it; there is no dedicated command yet. Retry with a **new** request ID. |
-| `queue_capacity` | `queue_capacity` | Admission would exceed a write-journal limit. Existing requests keep their place; nothing is evicted. For the cumulative limits (lifetime request IDs and receipt bytes), `detail` names the limit, for example `principal_lifetime_ids`; for the outstanding-request caps, `detail` carries the counts (`outstanding=<used> limit=<limit>`), the message names the source holding the most of the local CLI principal's slots, and a managed `gbrain sync` drain waits them out ([`write_capacity`](#drain-write-capacity)). One source cannot take every slot of the shared local CLI principal from the others: at that cap a source holding less than its fair share (the cap divided by the sources holding outstanding requests on it) is still admitted, and the brain cap stays hard. | For a cumulative limit, run the printed `gbrain config set persistence.limits.<limit> <value>` (sized for about one more year at the current rate), then retry with the **same** request ID. For outstanding-request, queued-byte or recovery-byte limits, let outstanding requests finish and check `gbrain sources writer status`. See [bounded admission and retention](concurrent-writes.md#bounded-admission-and-retention). |
+| `queue_capacity` | `queue_capacity` | Admission would exceed a write-journal limit. Existing requests keep their place; nothing is evicted. For the cumulative limits (lifetime request IDs and receipt bytes), `detail` names the limit, for example `principal_lifetime_ids`; for the outstanding-request caps, `detail` carries the counts (`outstanding=<used> limit=<limit>`), the message names the source holding the most of the local CLI principal's slots, and a managed `gbrain sync` drain waits them out ([`write_capacity`](#drain-write-capacity)). At the shared local CLI principal's outstanding cap, a page or sync write from a source holding less than its fair share (the cap divided by the sources holding outstanding requests on it) is still admitted, and the brain cap stays hard. Gates that read the principal's headroom before admission (maintenance backpressure, purge deferral, topology lifecycle checks) still count every source's requests. | For a cumulative limit, run the printed `gbrain config set persistence.limits.<limit> <value>` (sized for about one more year at the current rate), then retry with the **same** request ID. For outstanding-request, queued-byte or recovery-byte limits, let outstanding requests finish and check `gbrain sources writer status`. See [bounded admission and retention](concurrent-writes.md#bounded-admission-and-retention). |
| <a id="checkpoint-validation-timeout"></a>`index_building`, `index_missing` or `indexes_valid` | `checkpoint_validation_timeout` | A managed sync's checkpoint validation (the check that every page receipt of the run committed) hit the coordinator's 5-second statement timeout on a large `persistence_requests` table. The checkpoint request failed terminally instead of being retried ahead of every other write to the source, so other writes proceed; `last_commit` is unchanged and every page the run already committed stays committed. `detail` says which of three states the request indexes were in when the hint was built. | `index_building`: migration 179 is still building the indexes; wait (doctor `persistence_request_indexes` shows progress), then run the printed retry. `index_missing`: run `gbrain repair request-indexes --apply` (it drops an INVALID index and rebuilds it concurrently on Postgres), then the printed retry. `indexes_valid`: run the printed retry once; if it times out again, report the request id with the `persistence_request_indexes` and `persistence_request_growth` entries of `gbrain doctor --json`. The retry is `gbrain sync --source <source> --no-pull` plus the run's `--repo` base and cursor options (`--full`, `--working-tree`, `--src-subpath`, `--exclude`, `--include-hidden`, `--strategy`) and saved processing flags, so it resumes the same cursor. Do not cancel the request: a cancelled checkpoint still blocks the next one. |
| <a id="connector-account-changed"></a>`account_changed`, `account_unresolved` | `connector_account_changed` | A managed Google or GitHub connector's credential resolves to a different account than the one pinned for the source (Google email, GitHub App installation or login), or to no account at all. Google checks every enabled service before any service runs. Nothing was imported. No reset flag authorizes an account change. | The suggestion gives two branches with real values. (A) Restore the credential for the pinned account (the configured `g_token_env`, `g_token_command`, vault credential, `gh_token_env` or GitHub App key), then `gbrain sync --source <source>`. (B) For a deliberate change, add a new source for the new account (`gbrain google setup --account <email>`, or `gbrain sources add <new-id> --kind github … --app-install <id>`), then `gbrain sources archive <source>`; existing pages stay under the old source. The refused run is recorded, and autopilot then skips the source (an `unavailable-path` skip in the fan-out, `freshness_source_path_skipped` with `code: connector_account_changed` in the freshness loop) instead of retrying it every interval; doctor `connector_account_mismatch` lists such sources. A `gbrain sync --source <source>` that passes the account check clears the skip, and so does a change to the source's connector config (one more automatic try). |
| <a id="connector-intent-outdated"></a>`pre_upgrade` | `connector_intent_outdated` | A connector write was admitted in the intent format retired in v0.60.11.0. With detail `pre_upgrade` it was admitted before this brain's upgrade; otherwise a connector host older than v0.60.11.0 admitted it. The consumer recovers any file publication in progress first, then fails the request. | `pre_upgrade`: nothing to do; the item is fetched again on the next `gbrain sync --source <source>` under a new request ID. Otherwise: `gbrain upgrade` on the host that runs connector jobs, then `gbrain sync --source <source>`. |
diff --git a/src/commands/doctor/checks/persistence-capacity.ts b/src/commands/doctor/checks/persistence-capacity.ts
index 196c64f888..7f9abee91e 100644
--- a/src/commands/doctor/checks/persistence-capacity.ts
+++ b/src/commands/doctor/checks/persistence-capacity.ts
@@ -10,6 +10,7 @@ import { BACKGROUND_PRODUCERS, readBackgroundPauseShare } from '../../../core/pe
const VERIFY = 'gbrain doctor --only persistence_capacity --json';
const DOCS = 'docs/guides/concurrent-writes.md#quota-near-exhaustion';
const OUTSTANDING_WARN = 0.9;
+const OUTSTANDING_STALL_S = 600;
/**
* #5470/#6696 (A3): a cumulative managed-write cap (permanent request IDs,
@@ -20,8 +21,9 @@ const OUTSTANDING_WARN = 0.9;
* their compacted (net) size. Each key gets the one value that covers a year.
* A scope past the background pause line (A5) names the producers it pauses.
* The fix verifies through doctor and the read-scope `get_brain_health` band (#6696 B2).
- * #8368: an outstanding-request cap at 90% or more is a warning too, naming the sources holding its slots (requests
- * that never settle hold them, so raising the cap is not the fix); its fix is the holder's read-only writer status.
+ * #8368: an outstanding-request cap at 90% or more whose oldest outstanding request is over ten minutes old is a warning
+ * too, naming the sources holding its slots (requests that never settle hold them, so raising the cap is not the fix); its
+ * fix is the holder's read-only writer status. A healthy sync drain runs near the line with young requests and stays ok.
*/
export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Check> {
try {
@@ -36,9 +38,11 @@ export async function checkPersistenceCapacity(engine: BrainEngine): Promise<Che
for (const row of capacityDiagnostics(counters, limits)) {
if (row.resource !== 'outstanding_count' || row.limit <= 0 || row.used < Math.ceil(row.limit * OUTSTANDING_WARN)) continue;
const principal = /^principal:([^:]+):(.+)$/.exec(row.scope);
- const holders = await engine.executeRaw<{ source_id: string; n: number | string }>(`SELECT source_id,count(*) AS n FROM persistence_requests
+ const holders = await engine.executeRaw<{ source_id: string; n: number | string; oldest_s: number | string }>(`SELECT source_id,count(*) AS n,
+ EXTRACT(EPOCH FROM now()-min(created_at)) AS oldest_s FROM persistence_requests
WHERE state IN ('queued','running','recovering')${principal ? ' AND principal_kind=$1 AND principal_id=$2' : ''} GROUP BY source_id ORDER BY count(*) DESC,source_id LIMIT 3`,
principal ? [principal[1], principal[2]] : []);
+ if (!holders.some(holder => Number(holder.oldest_s) >= OUTSTANDING_STALL_S)) continue;
outstanding.push({ scope: row.scope, resource: row.resource, used: row.used, limit: row.limit, config_key: row.config_key,
holders: holders.map(holder => ({ source_id: holder.source_id, outstanding: Number(holder.n) })) });
}
diff --git a/src/core/connectors/item-holds.ts b/src/core/connectors/item-holds.ts
index 5a2a08c98e..1c8012b822 100644
--- a/src/core/connectors/item-holds.ts
+++ b/src/core/connectors/item-holds.ts
@@ -95,7 +95,7 @@ export interface ItemHoldsState {
const SOURCE_CODES = new Set([
'writer_coordinator_required', 'owner_unavailable', 'database_contention', 'writer_lock_unavailable', 'writer_pool_capacity',
- 'queue_capacity', 'write_quota_exhausted', 'write_capacity_backpressure', 'recovery_required', 'write_pending', 'source_changed', 'permission_denied', 'scope_denied',
+ 'queue_capacity', 'write_quota_exhausted', 'write_capacity_backpressure', 'recovery_required', 'owner_marker_missing', 'write_pending', 'source_changed', 'permission_denied', 'scope_denied',
'writer_registration_required', 'writer_identity_invalid', 'writer_not_initialized', 'writer_upgrade_required',
'writer_not_quiesced', 'writer_admin_locked', 'connector_account_changed', 'connector_intent_outdated',
'unsupported_mutation_protocol', 'cancelled', 'write_claim_lost', 'lock_stolen', 'statement_timeout', 'lock_timeout',
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 3fb3c735ce..55e6cab5fc 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -493,7 +493,7 @@ export const CODES = {
summary: "The ownership marker (`.gbrain-owner.json`) is missing from a managed source's checkout root while the reservation beside the checkout still names this owner, so no write publishes there.",
why: "The marker binds the checkout directory to its owner. A cleanup or a restore that drops dotfiles removes it; a retry never brings it back, so a write refused with this code ends terminally and releases its outstanding-request slot. The verified self-transfer re-creates it for the same root on the owner host; creating or copying it by hand is refused.",
docs: 'docs/guides/write-refusals.md#owner_marker_missing',
- suggestion: 'On the brain host: gbrain sources writer status --source <id> --json (note admin_state), then gbrain sources writer transfer prepare <id> --self-transfer and transfer accept with the manifest it prints; then gbrain sources retry-held <id> and the same gbrain sync.',
+ suggestion: 'On the brain host: gbrain sources writer status --source <id> --json (note admin_state), then gbrain sources writer transfer prepare <id> --self-transfer and transfer accept with the manifest it prints; then run the same gbrain sync again (a sync stops blocked_by_failures on the refused file, with no hold).',
fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'host_admin', why: 'Read-only: shows admin_state and the refused writes before the self-transfer that re-creates the marker.', requires_exclusive: false } },
owner_unavailable: { class: 'retryable', summary: "The source's canonical owner cannot run this write or maintenance step right now; `reason` says which condition.",
why: "A managed source publishes only through the host that owns its checkout. `host_mismatch`: another host id owns it (two GBRAIN_HOME identity files on one machine look the same; both ids are printed, and a retry on this host cannot help). `transfer_in_progress`: the worktree is draining for a writer transfer, which ends when the successor accepts it or the owner host runs `gbrain sources writer transfer cancel`; a drain prepared over 30 minutes ago asks the user which (`ask_user`) instead of waiting. `clone_in_progress`: a topology clone is recovering it and clears by itself. `binding_missing`, `incarnation_changed`, `local_path_missing` and `coordination_path_missing` name an incomplete or outdated registration. `not_sent` / `outcome_unknown`: the local owner's IPC lane did not take the write, or lost its acknowledgment. `detail: unbound_source`: on Postgres the source has a checkout but no canonical owner, a configuration state rather than a transient one (not retryable); in classic mode the write path is the file, a commit and `gbrain sync`.",
diff --git a/src/core/item-failure.ts b/src/core/item-failure.ts
index 3631d9e0c5..4dd93b9f5f 100644
--- a/src/core/item-failure.ts
+++ b/src/core/item-failure.ts
@@ -78,7 +78,7 @@ export const RUN_SCOPED_CODES: ReadonlySet<string> = new Set([
'writer_not_quiesced', 'writer_admin_locked', 'unsupported_mutation_protocol', 'write_claim_lost', 'connection_lost',
'permission_denied', 'schema_owner_required', 'scope_denied', 'config', 'auth', 'connector_holds_exhausted', 'transfer_in_progress', 'disk_full',
'invalid_token', 'insufficient_scope', 'trusted_local_only', 'requires_local_engine', 'not_owner_host', 'host_identity_mismatch',
- 'recovery_required', 'cancelled', 'persistence_restart_required', 'writer_regrant_required', 'writer_manifest_rescope_required',
+ 'recovery_required', 'owner_marker_missing', 'cancelled', 'persistence_restart_required', 'writer_regrant_required', 'writer_manifest_rescope_required',
'connection_inherited_read_only',
]);
diff --git a/src/core/persistence/effects.ts b/src/core/persistence/effects.ts
index 413d3224a6..ee21370268 100644
--- a/src/core/persistence/effects.ts
+++ b/src/core/persistence/effects.ts
@@ -747,7 +747,7 @@ async function settleDerivedFailures(engine: BrainEngine, derived: DerivedFailur
/** Waits that say nothing about the target never count toward parking. */
const CONTENTION_CODES = ['projection_pending', 'revision_conflict', 'writer_busy', 'writer_pool_capacity', 'git_index_locked'];
// #6305: writer_lock_unavailable is this process failing to open the lock file, not the effect failing; it never parks the effect.
-const DEPENDENCY_CODES = ['recovery_required', 'owner_unavailable', 'write_claim_lost', 'queue_capacity', 'write_quota_exhausted', 'write_capacity_backpressure', 'writer_lock_unavailable'];
+const DEPENDENCY_CODES = ['recovery_required', 'owner_marker_missing', 'owner_unavailable', 'write_claim_lost', 'queue_capacity', 'write_quota_exhausted', 'write_capacity_backpressure', 'writer_lock_unavailable'];
/**
* A scan sets its failing target aside and moves on; a single-target effect,
diff --git a/src/core/persistence/owner-marker.ts b/src/core/persistence/owner-marker.ts
index 5e7669be92..c85ec93f82 100644
--- a/src/core/persistence/owner-marker.ts
+++ b/src/core/persistence/owner-marker.ts
@@ -19,7 +19,7 @@ export function ownerMarkerRestoreFix(sourceId: string, binding?: { root: string
'--manifest', '<manifest digest printed by prepare>', '--self-transfer', ...intent('accept')],
inputs: [adminState, ...(binding ? [] : [{ name: 'root', how: 'the binding root writer status prints' }, { name: 'owner_epoch', how: 'the owner_epoch prepare printed' }])],
consent: [], actor: 'host_admin', requires_exclusive: false,
- why: 'Accepts the self-transfer: ownership stays on this host, the marker is written again and writes to the source resume. A sync file refused meanwhile is retried by gbrain sources retry-held and the same sync.',
+ why: 'Accepts the self-transfer: ownership stays on this host, the marker is written again and writes to the source resume. A sync that stopped blocked_by_failures on a refused file is finished by running the same sync again.',
verify: { argv: ['gbrain', 'sources', 'writer', 'failures', sourceId, '--json'] } } };
}
diff --git a/src/core/persistence/sync-fault-class.ts b/src/core/persistence/sync-fault-class.ts
index 8543aa345c..fdecfda11e 100644
--- a/src/core/persistence/sync-fault-class.ts
+++ b/src/core/persistence/sync-fault-class.ts
@@ -176,7 +176,7 @@ export const SYNC_FAULT_TABLE: ReadonlyArray<SyncFaultRule> = [
{ code: 'recovery_required', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'A write needs recovery on the brain host (gbrain sources writer status --json names it).' },
// #8368: the checkout lost its ownership marker; only the verified self-transfer re-creates it.
{ code: 'owner_marker_missing', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always',
- human_reason: 'The checkout lost its ownership marker (.gbrain-owner.json). On the brain host, run the verified self-transfer (gbrain sources writer transfer prepare <source> --self-transfer, then transfer accept) to re-create it; never create it by hand. Then gbrain sources retry-held <source> and the same sync.' },
+ human_reason: 'The checkout lost its ownership marker (.gbrain-owner.json). On the brain host, run the verified self-transfer (gbrain sources writer transfer prepare <source> --self-transfer, then transfer accept) to re-create it; never create it by hand. The sync stopped blocked_by_failures on the refused file: run the same sync again.' },
{ code: 'owner_unavailable', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'The write owner is gone; start or restart it on the brain host.' },
{ code: 'writer_coordinator_required', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'Managed persistence is not active for this source; activate it or sync unmanaged.' },
{ code: 'permission_denied', class: 'systemic', safe_actions: ['none'], needs_human: true, escalate: 'always', human_reason: 'This principal may not sync the source; the owner grants it.' },
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index 08ec0c5603..5671b55bbe 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -897,7 +897,7 @@ export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostI
});
} catch (error) {
// GBRA-130 A-34: one refresh's unexpected failure leaves that row for the next topology scan and never stops the others or the scan.
- if (!(error instanceof OperationError && ['write_pending', 'recovery_required', 'refresh_recovery_required', 'refresh_in_progress', 'owner_unavailable'].includes(error.code))) {
+ if (!(error instanceof OperationError && ['write_pending', 'recovery_required', 'owner_marker_missing', 'refresh_recovery_required', 'refresh_in_progress', 'owner_unavailable'].includes(error.code))) {
console.error(`[gbrain] worktree refresh ${row.id} (${row.source_ids.join(', ')}) did not converge in the owner's topology scan (${String((error as { code?: unknown } | null)?.code ?? (error as Error)?.name ?? 'error')}); it stays ${row.state} and the owner's next topology scan retries it. Check it with gbrain sources writer status ${row.source_ids[0]}.`);
}
} finally { await lock.release(); }
diff --git a/test/persistence-effects.test.ts b/test/persistence-effects.test.ts
index f5b12fd32b..2bb29592a3 100644
--- a/test/persistence-effects.test.ts
+++ b/test/persistence-effects.test.ts
@@ -20,6 +20,7 @@ import { serializePageToMarkdown } from '../src/core/markdown.ts';
import { installPageProjection, readProjectionSnapshot } from '../src/core/page-state/projections.ts';
import { durableGitRepo } from './helpers/git-publication.ts';
import { declarePersistenceProtocol } from '../src/core/persistence/protocol.ts';
+import { PHYSICAL_ROOT_MARKER } from '../src/core/persistence/physical-root-record.ts';
let engine: PGLiteEngine;
const roots: string[] = [];
@@ -322,3 +323,29 @@ test('#6305: a Git effect this process cannot lock never parks and commits once
await runPersistenceEffects(engine, config, { hostId, limit: 1 });
expect((await effect())!.state).toBe('committed');
});
+
+test('#8368: a Git effect whose checkout lost its ownership marker never parks and commits once the marker is back', async () => {
+ const f = await fixture();
+ durableGitRepo(f.root, ['page.md']);
+ const admitted = await admit(f);
+ const row = (await claimNextWrite(engine, hostId))!;
+ expect(row.id).toBe(admitted.id);
+ const published = await publishMutation(engine, row, { observedRevision: f.snapshot.revision, file: { path: f.file, root: f.root, content: 'After' },
+ apply: async tx => { await tx.putPage('page', page('After'), { sourceId: f.sourceId }); return {}; } }, hostId);
+ expect(published.state).toBe('committed');
+ await engine.executeRaw("UPDATE persistence_effects SET next_attempt_at=now()+interval '1 hour' WHERE request_id=$1::uuid AND kind<>'git'", [row.id]);
+ const marker = join(f.root, PHYSICAL_ROOT_MARKER);
+ const bytes = readFileSync(marker);
+ rmSync(marker);
+ const effect = async () => (await engine.executeRaw<{ state: string; error_code: string | null }>(
+ "SELECT state,error_code FROM persistence_effects WHERE request_id=$1::uuid AND kind='git'", [row.id]))[0];
+ for (let attempt = 0; attempt < 7; attempt++) {
+ await engine.executeRaw("UPDATE persistence_effects SET next_attempt_at=now() WHERE request_id=$1::uuid AND kind='git'", [row.id]);
+ await runPersistenceEffects(engine, config, { hostId, limit: 1 });
+ }
+ expect(await effect()).toMatchObject({ state: 'queued', error_code: 'owner_marker_missing' });
+ writeFileSync(marker, bytes, { mode: 0o600 });
+ await engine.executeRaw("UPDATE persistence_effects SET next_attempt_at=now() WHERE request_id=$1::uuid AND kind='git'", [row.id]);
+ await runPersistenceEffects(engine, config, { hostId, limit: 1 });
+ expect((await effect())!.state).toBe('committed');
+});
diff --git a/test/write-cap-wedge-8368.test.ts b/test/write-cap-wedge-8368.test.ts
index 925f1dfdce..3330a6cf5f 100644
--- a/test/write-cap-wedge-8368.test.ts
+++ b/test/write-cap-wedge-8368.test.ts
@@ -33,6 +33,9 @@ import { classifyTerminal } from '../src/core/persistence/terminal-classify.ts';
import { checkPersistenceCapacity } from '../src/commands/doctor/checks/persistence-capacity.ts';
import { performSync } from '../src/commands/sync/perform.ts';
import { CODES } from '../src/core/error-registry.ts';
+import { OperationError } from '../src/core/ops/contract.ts';
+import { classifyItemFailure } from '../src/core/item-failure.ts';
+import { classifyConnectorError } from '../src/core/connectors/item-holds.ts';
import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
import { reviewedWriterIntent } from './helpers/writer-admin-intent.ts';
import { testBackends } from './helpers/test-backends.ts';
@@ -158,7 +161,10 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
const own = await refusedWith(stuck);
expect(own).toMatchObject({ code: 'queue_capacity', detail: 'outstanding=4 limit=4' });
expect(own!.message).toContain(`source ${stuck} holds 4`);
- // (d) doctor warns at the cap and names the holder; its fix is the read-only writer status of that source.
+ // (d) young requests at the cap are a busy drain, not a stall: doctor stays ok.
+ expect(await checkPersistenceCapacity(engine)).toMatchObject({ status: 'ok', details: { outstanding: [] } });
+ // Once they stop settling, doctor warns at the cap and names the holder; its fix is the read-only writer status of that source.
+ await engine.executeRaw(`UPDATE persistence_requests SET created_at=now()-interval '1 hour' WHERE source_id=$1 AND state IN ('queued','running')`, [stuck]);
const check = await checkPersistenceCapacity(engine);
expect(check.status).toBe('warn');
expect(check.message).toContain(stuck);
@@ -185,3 +191,9 @@ for (const kind of testBackends()) describe(`#8368 write-cap wedge (${kind})`, (
}), 120_000);
});
});
+
+test('#8368: owner_marker_missing stops a sync or connector run for the source instead of counting against one item', () => {
+ const refusal = new OperationError('owner_marker_missing', 'The ownership marker is missing.');
+ expect(classifyItemFailure(refusal).scope).not.toBe('item');
+ expect(classifyConnectorError(refusal)).toMatchObject({ code: 'owner_marker_missing', scope: 'source' });
+});