Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs: key-files entry for the refresh-fence wait (ship docs audit)
garrytan08997a59f8f5
diff --git a/docs/architecture/key-files/core-persistence-request-indexes.md b/docs/architecture/key-files/core-persistence-request-indexes.mdindex 64a233b047..afda8884ba 100644--- a/docs/architecture/key-files/core-persistence-request-indexes.md+++ b/docs/architecture/key-files/core-persistence-request-indexes.md@@ -8,7 +8,7 @@ The online-built request-table indexes (the managed sync checkpoint pair, the pa- `src/core/persistence/fault-points.ts` — named crash seams for the persistence gate. `faultPoint(point, detail)` is a no-op unless a gate worker installed a hook with `installFaultHook`; production never does. Seams: `publication:<boundary>` (the five `PublicationHooks.boundary` names, routed through `withFaultPoints` in `publishMutation`), `effect:<kind>:mid` for every `EffectKind` in `completeEffect` (after the side effect, before its completion is recorded; `EFFECT_FAULT_POINTS` is a `Record<EffectKind, FaultPoint>`), `effect_recovery:before_clear` (`effect-recovery.ts`), `large_file:stage_streaming` (after each piece `stageStreamedContent` writes to a stage's in-progress name), `effect_recovery:before_stage` / `effect_recovery:staged` (a streamed, version 2 withdrawal mirror: once its record is durable, before its file is streamed into the publication stage, and after that step, before the recovery transaction renames the stage), `publication_recovery:before_clear` (`recoverPublication`), `sync:mid_checkpoint` (after each committed cursor step in `sync-run.ts` `saveCursor`), `sync:after_freeze` (#6712 PR 0: at the end of `freezeEntry` for an import entry, after its bytes are read and before the request is admitted; detail carries `slug` and `path`), `consumer:prepared` (between preparing a claimed request and publishing it), `stuck_recovery:*` (between the steps of `recovery-resolve.ts`) and the purge lookups `purge:*` (page-purge.ts, intent-redaction.ts, facts/purge.ts; a hook throwing a 57014 models a statement timeout). Used by `scripts/persistence/crash-robot.ts`, the #6511 tests and `test/sync-convergence.serial.test.ts`, which installs it in real `serve`/`sync` processes through the `test/fixtures/convergence-fault-kill.ts` preload (one action registry: kill, uncommitted edit, stall until released, or throw a given SQLSTATE, at the nth hit).- `src/core/persistence/recovery-resolve.ts` + `src/core/repair/stuck-recovery.ts` — `gbrain repair stuck-recovery [--source <id>] [--request-id <uuid>]` (#6650 §1.10; GBRA-130 C-1, C-4), explicit-only, preview-bound and destructive (registry `consentText`; `--apply --expect <hash> --keep prior|disk|withdrawn --yes`, a non-interactive apply without `--yes` exits 3; a `--keep` some approved record does not offer refuses `recovery_choice_refused` in the kind's plan before anything runs). `scanStuckRecoveries` (owner host only, else `owner_unavailable` for a named source) reads the source's queued head blocker (`readHeadBlockers`; its `request_recovery` ids select the records) and every non-terminal recovery record on the worktree blocked on `unexpected_file_bytes` or whose file matches neither recorded version: a page record (`target: 'page'`) or a v2 shared-skill bundle record (`target: 'skill_bundle'`, every file of the bundle; a file matches a recorded version only when `bundleFileHash` also accepts its mode and shape). A bundle record `recoveryFiles` cannot read is listed in `refused` as `recovery_bundle_unsupported` (residual `bundle_refused`; named with `--request-id` it throws with the same ask_user fix). Every withdrawal-mirror effect recovery on the worktree whose file is not a regular file holding its pre-withdrawal or withdrawn bytes is an item too (`target: 'withdrawal_mirror'`, id `effect:<id>`, `effect_id`, the forget's `request_id`; its file view adds `withdrawn`, the recorded post-withdrawal fingerprint, and its only choice is `withdrawn`); a record without usable withdrawn bytes, or bound to another owner epoch, root or incarnation, is listed in `refused` as `recovery_effect_unsupported` (residual `effect_recovery`) with `fix` `{ ask_user, argv: gbrain repair stuck-recovery --source <id> --request-id <request> }` and a `user_message`, and named with `--request-id` it throws with that fix; a blocked terminal request is listed in `refused` (residual `terminal_refused`; named with `--request-id` it refuses `recovery_choice_refused`, as does a choice the record cannot support), and unblocked terminal cleanups, staging-byte blocks and self-recovering records are residuals. Each item lists every file's prior (or absent) and disk (or deleted) fingerprints and what recovery sees; a local preview writes the prior (an effect's `withdrawn`), disk and unified-diff files 0600 under `~/.gbrain/backups/stuck-recovery/<request>/<digest>/` (a private directory outside every checkout) and prints the `diff -u` command. The item digest binds the recovery identity (attempt, owner epoch, path and published hash per file), the recorded prior hashes, each file's disk view and the page's `knowledge_revision`. `resolveStuckRecovery` runs under `acquireWorktree` with a bounded wait (10 s, then `writer_lock_unavailable`; so the resident recovery pass returns unchanged), re-verifies the digest (`preview_changed`) and journals in `op_checkpoints` op `stuck-recovery-resolution` keyed `<request row id>:<preview hash>` (exempt from the checkpoint purge; the finished row is the audit entry): `backed_up` (disk and prior bytes durable, always), then `written` (`--keep prior`: `upgradeRecoveryStaging` reserves an older record's restoration stage within the recovery budget, `queue_capacity` otherwise, then the prior bytes are published through `publishPersistenceFile` at that stage after `cleanupRecoveryStaging`, an absent prior unlinked) or `abandoned` (`--keep disk`, #6640's drop: `completeWrite` ends the live request `conflict` `source_changed`, or `finishUnpublishedFailure` with its recorded failure; the record is never rewritten, so its reserved bytes are released by the terminal path's `clearResolvedRecovery`), then `recoverPublication(…, alreadyLocked)` finishes by its unchanged rule and the journal records `done` with the result. A step refuses `unexpected_file_bytes` when the file changed again; a different `--keep` for the same hash refuses `idempotency_conflict`; an identical apply after `done` replays the result (also after the approved set aged out). A record the resident consumer (any version) or an earlier killed run already finished reports `finished_by: another_pass`. `--keep prior` on a request with no recorded failure requeues it (the accepted write publishes again). `--keep disk` hands off to `gbrain sources reconcile <source> <slug> --brain host --preview` and never reports the page repaired. A record whose prior file is over the single-string ceiling (`staging.before`) resolves the same way in bounded memory: the disk file and the staged copy are hashed in pieces (the copy's state, `verified`, `missing` or `mismatched`, is in the prior view and the digest), backups are kernel copies checked by streamed hash, the diff file is a summary (both fingerprints, the first differing byte, a window of at most 1 KiB; `cmp` reproduces it), `--keep prior` runs recovery's `restoreBeforeStage` after `cleanupRecoveryStaging(keepBefore)`, and `--keep disk` backs up a mismatched copy (`<index>.prior-stage`) and removes it before abandoning. `--keep prior` without a verified copy refuses `recovery_prior_stage_unavailable` (registry entry; its fix is the `--keep disk` apply under the same hash). An effect item resolves through `resolveEffectRecovery` under the same lock and journal (key `effect:<id>:<preview hash>`): `backed_up` (the edited bytes beside the withdrawn bytes and their diff), `written` (first `rebaseStaleEffectRecovery` (`effect-recovery.ts`): when a later withdrawal moved the page past the record, the record is rebased under its page lock on the latest snapshot's `mirrorContent` bytes (the rendering `mirrorPage` itself writes), its publication stage and reserved bytes following them, and the journal stores the rebased `identity` in the same transaction, so the recorded bytes never put a later-withdrawn fact back on disk and a rerun resumes; a page deleted, replaced or segmented meanwhile refuses `recovery_effect_unsupported` with nothing written; then `upgradeRecoveryStaging` `forward` and the record's `after` bytes published at its publication stage, mode restored), then `claimAndRecoverEffect` (`effects.ts`, the resident pass's own claim + `recoverEffectPublication` + failure recording) materializes the page, clears the record and advances the effect's cursor, and `trueUpRequests` settles the forget's receipt; `--keep disk`/`prior` on it refuse `recovery_choice_refused`, since keeping the edited or pre-withdrawal bytes would let the next sync re-import the forgotten fact. A bundle's `--keep prior` writes the prior bytes of its edited files and `recoverPublication` restores the rest; `--keep disk` keeps every file and abandons the publication; both back up every file first. Fault seams `stuck_recovery:{backed_up,written,abandoned,recovered}`. Tests: `test/repair-stuck-recovery.test.ts`, `test/e2e/repair-stuck-recovery-postgres.test.ts`, `test/pathological/c1-effect-stuck-recovery.test.ts` and `test/pathological/c4-bundle-stuck-recovery.test.ts` (both engines; the effect probe asserts the forgotten fact stays forgotten after the resolution, a commit and a sync).- `src/core/persistence/native-file-target.ts` — shared preparation/Git execution spelling resolution. Native directory-entry inode/device identity proves aliases; no global case folding or separator rewrite. Missing suffixes retain exact spelling, descendant symlinks and ambiguous aliases refuse, and physical root identity is rechecked. Exact POSIX case-distinct names and literal backslashes remain distinct. #5032: `isWindowsColonTarget(rel)` and `colonSlugWindowsRefusal(slug, sourceId)` name the one Windows-only refusal (`colon_slug_windows_write_through`: `:` names an NTFS alternate data stream); `submitPageMutation` raises it before admission when this host publishes the file, `prepareFileTarget` at publication, and managed (`fileRefusals`) and unmanaged (`failedFiles`) sync skip each such path by name (`test/colon-slug-windows-5032.test.ts`). A path segment over 255 bytes, or `ENAMETOOLONG` from the lookup, refuses `invalid_params` with detail `file_name_too_long` and a fix naming a slug whose segments fit 240 bytes (`test/pathological/b12-slug-segment-bytes.test.ts`).-- `src/core/persistence/admission-retry.ts` — bounded retries for database-confirmed admission transaction aborts. Each retry reuses the UUID and normalized intent, rechecks guarded authority and reserves quota only on commitment. Standalone journal admission uses 100ms lock waits within the five-second retry budget to retain queue position behind short counter transactions; backoff occurs after rollback releases the connection so counter contention leaves pool capacity for reads. Ambiguous connection/commit errors are never classified as confirmed rollback. #6355: a session that drops under the attempt (`isConnectionLoss` in `retry-matcher.ts`: connection-class SQLSTATEs, postgres.js's connection codes, `57P01`/`57P02`/`57P03`, the socket errnos) re-runs the attempt after `CONNECTION_LOSS_RETRY_MS` (100/300/900 ms), which is safe because every attempt reads the retained request id first and replays an admitted row; when the session keeps dropping the caller gets `writeOutcomeUnknown` (`write_outcome_unknown`, `reason: connection_lost`, fix: read the request by id), never the raw socket error the crash robot's receipt oracle took for a refusal while the write committed (`test/persistence-admission-connection-loss.test.ts`, forced probes: a lost COMMIT acknowledgment; a dead lent transaction used once with the pool finishing the write). The optional `recover(error)` hook runs before each re-run; `admitWrite` uses it to rebuild the pool (`engine.reconnect`) and to move the re-run from a lent transaction (the consumer's lane, `onPersistenceLane`) to `engine.transaction`, since a dropped reservation fails every later use the same way ("Connection is no longer owned"). Real counter-lock, sustained-queue progress, read-capacity and after-insert rollback tests live in `test/persistence-journal.test.ts`. Withdrawal retries wrap its whole source-exclusive transaction with its existing lock policy; `admitWriteInTransaction` never backs off inside an existing unit of work. `test/persistence-withdrawal-retry.test.ts` verifies guard release, complete rollback and bounded contention. #6278 (B7): also the outstanding-request cap's `detail` format (`outstandingCapacityDetail`, `outstanding=N limit=M`, stamped by `journal.ts` `capacityError`) and its readers `isWriteCapacityWait` / `outstandingCapacityOf`, which the managed drain and `sync-run.ts`'s failure ledger use to tell a refusal that clears by itself from a cumulative cap.+- `src/core/persistence/admission-retry.ts` — bounded retries for database-confirmed admission transaction aborts. Each retry reuses the UUID and normalized intent, rechecks guarded authority and reserves quota only on commitment. Standalone journal admission uses 100ms lock waits within the five-second retry budget to retain queue position behind short counter transactions; backoff occurs after rollback releases the connection so counter contention leaves pool capacity for reads. Ambiguous connection/commit errors are never classified as confirmed rollback. #6355: a session that drops under the attempt (`isConnectionLoss` in `retry-matcher.ts`: connection-class SQLSTATEs, postgres.js's connection codes, `57P01`/`57P02`/`57P03`, the socket errnos) re-runs the attempt after `CONNECTION_LOSS_RETRY_MS` (100/300/900 ms), which is safe because every attempt reads the retained request id first and replays an admitted row; when the session keeps dropping the caller gets `writeOutcomeUnknown` (`write_outcome_unknown`, `reason: connection_lost`, fix: read the request by id), never the raw socket error the crash robot's receipt oracle took for a refusal while the write committed (`test/persistence-admission-connection-loss.test.ts`, forced probes: a lost COMMIT acknowledgment; a dead lent transaction used once with the pool finishing the write). The optional `recover(error)` hook runs before each re-run; `admitWrite` uses it to rebuild the pool (`engine.reconnect`) and to move the re-run from a lent transaction (the consumer's lane, `onPersistenceLane`) to `engine.transaction`, since a dropped reservation fails every later use the same way ("Connection is no longer owned"). Real counter-lock, sustained-queue progress, read-capacity and after-insert rollback tests live in `test/persistence-journal.test.ts`. Withdrawal retries wrap its whole source-exclusive transaction with its existing lock policy; `admitWriteInTransaction` never backs off inside an existing unit of work. `test/persistence-withdrawal-retry.test.ts` verifies guard release, complete rollback and bounded contention. #6278 (B7): also the outstanding-request cap's `detail` format (`outstandingCapacityDetail`, `outstanding=N limit=M`, stamped by `journal.ts` `capacityError`) and its readers `isWriteCapacityWait` / `outstandingCapacityOf`, which the managed drain and `sync-run.ts`'s failure ledger use to tell a refusal that clears by itself from a cumulative cap. #8621: `waitOutRefreshFence` re-runs an interactive write's whole admission while a refresh fence refuses it (`worktree_refreshing`; 250 ms to 2 s backoff, `REFRESH_FENCE_WAIT_MS` = 10 s cap shared by every item of a `remember` batch via `withRefreshFenceBudget`, lowered by an explicit `wait_ms`); background writes and the managed sync are not waited for (`test/write-refusal-refresh-fence-8621.test.ts`).- `src/core/persistence/unbound-source.ts` — #5254 policy for a Postgres page write to a filesystem source (`local_path`, or `sync.repo_path` for `default`) with default write-through and no worktree binding. `submitPageMutation` refuses `owner_unavailable` with `detail: unbound_source`, a `docs` pointer to `write-refusals.md#unbound-sources-on-postgres`, and a hint naming the filled `sources writer status`/`writer claim ... --admin-intent writer_claim --expected-state <admin_state>` bind command (host path only for trusted local callers) and, for `put_page` to a page without `source_path`, `gbrain config set persistence.unbound_write database_only`. That key (`refuse` default, validated by `config set`; any other stored value refuses) lets such a `put_page` admit with `databaseOnlyReason: 'unbound_source'`. `decideUnboundWrite` is the one rule page writers and fact writers (`remember`, loop-fact retirement and relink through `fact-write-target.ts`, #6608) apply to a rooted source with no binding: connector → `connector_database`; PGLite not activation-listed database-only (#6449) → automatic claim; otherwise a page with a recorded canonical file, or the key unset, refuses and the rest is database-only. `remember` keeps its frozen `unavailable` code and carries the refusal's detail, docs and fix (`verb-errors.ts`). On a classic brain (managed persistence not activated) the hint leads with the classic write path: write `<checkout>/<slug>.md`, commit it, `gbrain sync --source <id>` (a remote caller is told to ask the user to). The refusal is `retryable: false` with a `fix` of `next: ask_user` and a `user_message` (classic or managed wording), because every way out is the user's choice (`test/unbound-source-hint.test.ts`). Publication rechecks inside its transaction, after `authorizeStoredRequest` share-locks the source row, that the source is still unbound and the page still has no `source_path` (else `owner_unavailable`/`unbound_source`, hint rebuilt from the persisted message by `writeResponse`), then stamps `pages.database_only_reason='unbound_source'` (migration 177). After binding, `prepareFileTarget` keeps a stamped page with no file database-only (receipt `write_through.skipped: unbound_source` with a warning) and managed sync refuses to import a file at its path (`source_changed`, diagnostic reason `unbound_source`). PGLite keeps auto-claiming. Tests: `test/e2e/unbound-source-postgres.test.ts`, `test/persistence-unbound-source.test.ts`. See [persistence stalls](persistence-stalls.md).- Segmented withdrawal mirror (`src/core/persistence/{effects,effect-recovery,effect-model,staging}.ts`, flip lane A) — a withdrawal's mirror and Git scans treat a page stored in segments as an inline page (older binaries skipped it with `page_body_segmented`). `mirrorPage` renders its file from the segments (`page-prepare.ts` `unreadPageFile`, a `StreamedFileContent`) and reserves a version 2 `EffectRecovery` (`after: null`; `staging.publication` names the streamed file by path and SHA-256; the reservation counts the file once). `recoverEffectPublication` writes that stage before its transaction (`stageStreamedMirror`: from the measured file, or after a crash from the segments at the recorded revision, first discarding a stream a crash cut short under its in-progress name; seams `effect_recovery:before_stage` / `effect_recovery:staged`), then renames the verified stage (`staging.ts` `publishStreamedStage`). A segmented page's withdrawal is not materialized again: its rows were struck in its segments when the withdrawal committed. v0.60.290.0 reads a version 2 record as version 1 (`Buffer.from(null)`, untyped `effect_unavailable` on every attempt, never parks) but never deletes it or its stage, so `mirrorPage` writes none while that binary could claim it (next entry). Test: `test/large-pages/segmented-withdrawal-mirror.test.ts` (purge and forget twins, Git commits, two interruption seams, both engines).- `src/core/persistence/{recovery-version,segmented-mirror-gate,segmented-gate-hold}.ts` — forward compatibility of recovery records. `recovery-version.ts` lists the record versions this binary reads (`EFFECT_RECOVERY_VERSIONS`, `REQUEST_RECOVERY_VERSIONS`); `assertRecoveryVersion` refuses any other `recovery_version_unsupported` (fix `gbrain upgrade`, `host_admin`) before a reader touches the record: `recoverEffectPublication`, `reserveEffectRecovery`, `rebaseStaleEffectRecovery` (the `--keep withdrawn` rebase), `upgradeRecoveryStaging` (both tables) and `model.ts` `recoveryFiles` (request recovery, clearing, stuck-recovery); the stuck-recovery preview (`recovery-resolve.ts` `scanStuckRecoveries`) lists such a record refused with that code and fix; `head-blocker.ts` reports such a record as an intervention with the same fix. `segmented-mirror-gate.ts`: `mirrorPage` calls `assertSegmentedMirrorGate` before streaming a segmented page's file; while `mirrorGateBlockers` sees a live writer below SEGMENTED_WRITER_FLOOR (the switch's TTL-cached `segmentedWriterBlockers`, filtered by `segmented-gate-hold.ts` `liveSegmentedBlockers`, the classification doctor `segmented_storage` uses: this host's own pre-upgrade writes superseded by the same kind of write from this build are not live; cached per gate read and host) it refuses `segmented_mirror_writer_gate` without reserving a record, and the target retries and parks as usual. `resumeGatedMirrors` (each effect pass, at most once per gate TTL per engine) requeues, via `retryParkedEffects`, requests whose parked targets all carry that code, or `withdrawal_mirror_failed` for the Git/embedding siblings parked behind the mirror (re-armed once it commits), once the gate is clear. No version 1 fallback: it would hold the file whole, and v0.60.290.0's recovery of it refuses to materialize a segmented page forever. Tests: `test/pathological/lp-d-recovery-version.test.ts`, `test/pathological/lp-d-segmented-mirror-gate.test.ts` (both engines).