Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
58.0 KB
Raw Blame

Key files: Core Persistence

Subsystem index. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG. The page operations that submit through this layer are in Core Services (2/3). Connector ingestion, item holds, checkpoint validation and the single-write fast path (switches.ts) continue in Core Persistence (continued). Managed facts extraction (withdrawn-candidate skip and recovery, conversation facts batches, the unmanaged derived index) is in Core Persistence (managed facts).

  • src/core/persistence/authority.ts — durable write authority. submissionAuthority records the caller's grant ceiling on every write request (scopes, operations, slug prefixes, delegated namespace and, for a delegated job, its native subagentId when a positive safe integer); authorizeWrite re-checks it against the CURRENT principal at publication and replay. For a delegated OAuth client the current namespace is the stored delegated_slug_prefixes in prefixes mode, or only wiki/agents/<retained subagentId>/* for the native job form (prefixes NULL); anything else allows nothing, and it always intersects the original accepted prefixes, bound tools, scope and source, so a revoked, narrowed or switched grant or a foreign or invalid job id denies (test/persistence-page-authority.test.ts, Postgres: test/e2e/minions-authority-parity.test.ts). #5994: a context carrying replayAuthority (set only by gbrain repair failed-writes) gets that stored authority back from submissionAuthority (replayedAuthority): same source incarnation or refused, databaseOnlyReason/takeHoldersUsed dropped for admission to recompute, then authorizeWrite and page visibility against the live grant.

  • src/core/persistence/activation.ts — private CLI sources writer activate --confirm-quiesced boundary. Verifies native capability and filesystem bindings, takes root locks before DB waits and rechecks topology under transaction guards. Live/young/foreign legacy leases refuse regardless of expiry; only explicitly requested --cleanup-dead-local-locks removes exact dead same-host holders under the lease lock. Dry-run deletes nothing. Never silently claims a source. Tests: test/persistence-activation.test.ts, test/persistence-onboarding.test.ts. Unbound Google/GitHub API sources may activate as reviewed connector_database without fabricating a Git root, while bound sources still need physical-root checks. Activation increments persistence_brain.mode_epoch (migration 183), which the writer admin-state hash includes. A non-connector source whose recorded local_path no longer exists or is relative (unclaimed, or claimed by this host) refuses source_changed with detail source_path_missing, naming the source, the path and the sources set-path / sources remove exits (the claimed checkout filled in), in --dry-run too (#5206, test/activation-missing-source-path.test.ts). The report's warnings[] (dry run and real run) carries one git_durability_not_enabled entry per filesystem source the effect runner's policy (gitDurabilityPolicy) would leave non-durable, with enable_command, a read-only git-durability --enable --dry-run fix and the note that enabling re-queues skipped Git effects; persistence-admin.ts also prints them to stderr without --json, and activation proceeds (#5182, test/writer-activate-durability-warnings-5182.test.ts).

  • src/core/persistence/git-durability-policy.ts + git-durability-administration.ts — whether the persistence owner commits and pushes page writes through its Git effect is a per-host-binding setting (#5182): persistence_host_bindings.git_durability (enabled | disabled | NULL, migration v234, column SQL kept out of the replayed schema blob as PERSISTENCE_GIT_DURABILITY_COLUMN_SQL). gitDurabilityPolicy(binding, root) is tri-state and fail-closed: a recorded value wins, NULL falls back to the legacy post-commit-hook probe, a probe error fails the effect instead of skipping it; effects.ts takes its per-root verdict from it. gitDurabilityState renders on/off/unknown for writer status and writer activate's git_durability listing. planGitDurabilityCatchUp/requeueGitDurabilityCatchUp re-queue the Git effects completed durability_not_enabled while the setting was off by resetting the existing rows, never inserting under UNIQUE(request_id, kind). The verb writer_git_durability (gbrain sources writer git-durability <source> --enable|--disable [--pat-file <path>] [--dry-run], administration-gated like the other writer verbs; a non-preview checks its approval before any side effect, #7634) runs a read-only git push --dry-run probe before enabling (codes git_branch_unborn, git_checkout_required, git_detached_head), wires the repo-scoped credential under gbrain.durability.managedcredential from a 0600 --pat-file (pat_file_unreadable otherwise) and retries the worktree's parked pushes (codes in GIT_PUSH_CODES; retry-effects <source> --parked-pushes does the same without a credential; a per-request refusal such as page_not_found is listed in refused and the run ends partial, #7110). sources push (managedPushRefusal in workspace-push.ts), bootstrap verify's checkPushProbe and the skill-pack writer guard name this route on a managed worktree (#5606). Pinned by test/persistence-admin-git-durability.test.ts, test/managed-push-remedy.serial.test.ts.

  • src/core/persistence/effect-git.ts (pushGitRoot) + git-push-holds.ts — a Git effect reads the remote tip (ls-remote), scans <tip>..HEAD and pushes HEAD as a fast-forward leased on it (a missing branch: whole history first). A non-ancestor tip or a moved remote runs gbrain git commit-push --push-only --keep-checkout --scan-secrets as a child (off the event loop, never PATH gbrain): it merges in memory and pushes without moving the checkout or running pre-push hooks, so sources refresh stays the only importer. Holds: publish_secret_found, publish_scan_unreadable, or integrate_conflict (the user picks --resolve). readGitPushHolds counts each source's effects retrying or parked on a push code, with conflict state and one fix, for doctor parked_effects and sync status git_push_holds (next falls back to a held fix). A hold in an unbound worktree is former_worktree (fix: settle preview, #7378). Pinned by test/pathological/h2-git-effect-moved-remote.test.ts.

  • src/core/persistence/deactivation.ts — private CLI sources writer deactivate (#5455/#5628), brain-wide and trusted-local. deactivationBlockers lists the admin lock, queued/running/recovering requests, recovering topology changes, queued, running or recovering effects (the predicate activation uses; each exit says what clears it, and a failing links or facts-backstop effect settles after five attempts), held connector items (managed holds that classic mode would not read) and live gbrain_cycle_locks leases, each with its CLI exit; failedEffectHistory lists terminally failed effects as history (failed_effects on the dry run and the applied report, never blockers), each with its code and the exit that retries it: retry-effects for parked targets and failed Git, withdrawal, links and facts-backstop effects, repair embedding-effects for a failed embedding effect, none for embedding_zero_norm or an effect whose source was archived, replaced or removed (test/pathological/a15-deactivate-exits.test.ts runs every printed exit and checks it clears its entry); physicalRootBlockers(engine, mode) (#5914 [R9a]) adds physical_root for this host's owned checkouts, probed as the mode's real run will: strict assertPhysicalRoot for managed, assertPhysicalRootReservation for the classic pre-activation release, so each preview lists exactly what its real operation refuses on; --dry-run returns them plus the worktrees that would retire, the kept shared-skill settings and, when clean, apply_command bound to the admin state read before the preview. deactivatePersistence takes the local native worktree locks, then in one transaction locks persistence_brain FOR UPDATE, checks the expected admin state and the admin lock, locks worktrees then sources FOR UPDATE (admission's order), rechecks the blockers, sets enabled=false, increments mode_epoch, retires every worktree, deletes source and host bindings and records a committed writer_deactivate topology change (brain id, retired epoch, retired worktree ids and roots, source roots). cleanupRetiredManagedMarkers runs on CLI connect, after deactivate and from writer status: it removes this brain's registry records, .gbrain-managed / Git gbrain-managed.json, .gbrain-owner.json and reservations only when their mode_epoch is a retired one below the current epoch, or, without an epoch, their worktree id (or, for markers without one, their root outside any active binding) is recorded as retired; unknown or newer epochs are kept and reported pending, each file is re-read before unlinking and other brains' files are never touched. root-registry.ts writes mode_epoch into new registry records and managed markers. Tests: test/persistence-deactivate.test.ts (PGLite and Postgres), test/e2e/persistence-deactivate-race.test.ts. See persistence stalls.

  • src/core/persistence/{admin-lock,blocking-effects,writer-versions}.ts — admin-lock.ts owns the opt-in brain-level writer admin lock: reserved config key persistence.writer_admin_lock ({locked, set_at, host_id, generation}), written only by local gbrain sources writer lock|unlock under the persistence_brain row FOR UPDATE; lock refuses while a prepared transfer is unaccepted. assertWriterAdminUnlocked runs inside the claim, managed claim, activation, shared-skill activation and transfer prepare/accept transactions for every caller (share lock on the brain row) and refuses with writer_admin_locked; only the ordinary first-write auto-claim (claimWorktree(..., { automatic: true }) from page and memory mutations) is exempt. Generic config set/unset/unset --pattern refuse the key; engine migration copies it. It is not a security boundary and older binaries ignore it. blocking-effects.ts lists queued, running or recovering effects (listBlockingEffects) and builds writer_not_quiesced suggestions naming the first blocking request or effect with gbrain sources writer status <source> --json. writer-versions.ts stamps admitter version/host at admission and consumer version/host/published_at only on committed publication (migration 178, cutoff persistence_brain.writer_version_cutoff on the database clock); replay and compaction keep stamps. sources writer status reports admin_lock, local_host_id, blocking_effects and writer_versions; src/commands/doctor/checks/writer-version.ts is the seven-day writer_version advisory. SEGMENTED_WRITER_FLOOR (page-body/writer-floor.ts): a seven-day writer below it or unmarked keeps pages.segmented_storage off (page-body/limits.ts, doctor segmented_storage); marked rows of any role meet the floor (ingest log: two partial indexes, v254/v259). Tests: test/persistence-writer-admin-lock.test.ts, test/persistence-writer-versions.test.ts, test/persistence-blocking-effects.test.ts, test/e2e/persistence-writer-{admin-lock,stamps}.test.ts.

  • src/core/persistence/{atom-maintenance,atom-retry,page-prepare,facts-maintenance,facts-prepare,prepared-maintenance}.ts — resident dispatch for durable atom publication/completion, explicit atom retry (retryManagedAtomBatch takes the job's signal and deadline, stops between stages on caller cancel, deadline or cycle-lock loss, shares one deadline-bound MaintenanceWriteWait with the model rerun, and reports a stop after admission as write_pending or an AbortError carrying the receipts, never cancelled; #6622; a transcript item that does not resolve to the retry's source is refused transcript_source_mismatch before anything starts, #7869), authorized multi-entity fact-fence/backstop append, local synthesize/patterns/consolidation, and the cycle's lint repairs on a managed brain (runLintCore publishes each fix through maintenancePreflight + publishMaintenancePage; #5180, test/managed-lint.test.ts). page-prepare.ts keeps the stored provenance stamps on every remote page write, repo or not (keepStoredProvenance, provenanceWriteTrust: tier by the lower of caller and published frontmatter; #7873); it exports fileMatchesSnapshot, the canonical file-versus-page comparison prepareFileTarget applies, so lint can bind a repair to the revision it read (it takes the importer's path for the file: a frontmatter-less file under a directory-inference rule also matches when readCanonicalAsImported (BOM strip + applyInference at scannerSourcePath) reproduces the stored row, with an inferred type treated as undeclared, #6605; reconcile reads the file side through reconcileFileReading in reconcile-state.ts); the uncoordinated-local-edit refusal carries detail file_database_drift and a sources reconcile <src> <slug> --brain host --preview fix; a shared-origin delete: canonical-reconciliation.md. In prepareFileTarget, a live page whose canonical file is missing refuses source_changed unless it is declared or derive-phase database-only, or it is a delete_page (deleting, no content) of a page with no recorded source_path and no captured file URI: that delete plans no file, so it publishes database-only and its receipt reports no file or Git effect (test/persistence-captured-file-target.test.ts, test/persistence-never-published-delete.test.ts). atomOwnerTarget is the one owner decision for an atom session and for atomDrainOwnerRefusal, the auto-drain dispatch pre-check; an unbound Google/GitHub source publishes atom maintenance through the connector_database target (its local_path is connector state, not a checkout) and the publication refuses with source_changed when the source gained an owner after preflight. submitMaintenance scans every content-carrying intent with scanCanonicalFences (the check the canonical projection runs) over the whole submitted body and timeline before a request exists (#6278): a defect no adoption clears (a second facts fence in the timeline; replaceOrInsertFactsFence replaces only the first) is refused with its fence reason, so fenceManaged reports the page in failed_pages and the others still adopt, while a repair whose postimage is clean passes. prepareFactFenceAdoption names its steps for enterClaimStep, compares the fence cell with the legacy fact modulo whitespace, writes the cell's parsed text back into facts.fact in the same UPDATE that sets row_num (so the projection keeps the fact's id), and throws the registered fence_unrenderable (class server) as its last-line defence. maintenancePreflight creates the job's MaintenanceWriteWait (optional deadlineAtMs) and submitMaintenance uses it for new and replayed requests; its owner check goes through checkOwner / ownerUnavailableError (owner-refusal.ts), so a missing binding, a changed incarnation, another owner host, a draining or recovering worktree and a missing local or coordination path each refuse owner_unavailable with their own reason, and nothing claims or transfers ownership to get past them. #6325: resumeManagedAtoms returns run | done | held. The drain does not replay a failed batch: it resumes the next automatic attempt, whose run key is digest([drainKey, failedRequestId, 'automatic']) (one stable id per failure, so a crashed attempt resumes and two runs admit the same requests), and whose completion advances the failed drain-key checkpoint by the same compare-and-set an explicit retry uses. The completion counts one strike per completed failure on that checkpoint (attempts); at MAX_DETERMINISTIC_FAILURES it sets held, which MANAGED_ATOM_DISCOVERY_SQL excludes from discovery and the backlog and listHeldManagedAtoms reports (doctor extract_atoms_held_failed, code held_failed). Reading a held or failed checkpoint never strikes; storage and authority errors fail the attempt without completing it, so they never strike either. #7514 write-failure strikes and hold: continued. A content change is a new drain key, so the count starts over. An explicit retry (session.retry) ignores held, and its failure refusal (extraction_failed) names the recorded failure reason (malformed output or a response stopped before the end) and carries a paid fix (gbrain jobs submit extract-atoms-drain --params '{sourceId, retryRequestId}') with a read-only gbrain write-request verify; the drain raises it only for a failed batch whose checkpoint is gone. Tests: test/managed-atoms-retry.test.ts. Managed facts extraction (the withdrawn-candidate skip and its recovery route, the conversation facts batches, the unmanaged derived index) continues in Core Persistence (managed facts). Atom completion admits replacement and revision-pinned stale-atom deletion children together. A failed, partial-input or zero-yield batch admits no retirement, and each retirement child refuses revision_conflict until every replacement page of its run is committed (replacementPublished), so a partly published batch leaves the prior generation live. Each retirement stamps retired_by/retired_at into the tombstone in the same publication; publishManagedAtoms restores a retired atom of the same origin when an extraction produces it again, and a user-deleted atom stays deleted. Each committed retirement bumps the origin page's regeneration generation (op_checkpoints op managed-atoms-generation, part of the drain's run key only above 0, never of the explicit retry identity; the content whose extraction retired keeps the generation its accepted batch used, so that batch stays resumable) and clears that page's older extract_atoms_page_state rows and completed managed-atoms checkpoints in the same publication, so an A→B→A edit, even after a partly failed batch, re-extracts instead of replaying a retained receipt. Retirements are admitted in slug order. Derived atom pages may regenerate or retire when their canonical file is already absent, while ordinary page writes retain missing-file refusal. Accepted output replays provider-free after restart; page/take preparers recheck original/current grants, source incarnation and revisions. Unresolved facts remain DB-only without phantom pages. Retired takes skip; semantic fact snapshots exclude derived vector/cache fields and journal quotas reject overlarge retained intent. Managed consolidation retains the explicit chronological valid_until writeback from the legacy consolidation phase, scoped to the accepted source/fact IDs and revalidated semantic snapshots; contradiction probes never gain supersession authority. submitMaintenanceIntent admits a maintenance request with its own intent kind keyed by the intent; prepareMaintenanceMutation also handles managed_maintenance_phantom_merge and managed_maintenance_phantom_delete (both delegated to cycle/phantom-redirect-managed.ts). publishMaintenancePage accepts an optional eventProjection (Life Chronicle): the event page and its depth-page timeline row commit in the same publication with the depth page key locked. Remote maintenance remains refused.

  • src/core/persistence/canonical-projections.ts — installs the facts, takes and timeline projections of a canonical page body inside its publication transaction. Rows written by the conversation fact extractor (cli:extract-conversation-facts*) share the page coordinate without a fence and are never expired as rows that left it (#1928 parity with the extract_facts reconcile), so republishing a conversation keeps its extracted facts until their own replay replaces them. Preparation takes the caller's prior snapshot and writer class and pins the stored timeline rows and take row numbers. Writer classes: editing (revision-bound put_page, reconcile with an approved body decision), preserving (put_page without a revision, reconcile, connectors), file (managed sync of file edits and managed import) and immutable (company-brain imports). One pure decision table (timelineRowAction) maps each row state (in_body, drifted, removed, removed_marked, database_only) to refresh_detail, delete, keep or materialize: database-only rows are materialized by editing and preserving writers and kept by the others; #6226 rows the older citation reading filed for either body are superseded (deleted by every writer) or, carrying their own detail, superseded_annotated (kept, never materialized); a materialized bullet the new body drops is deleted by editing/file, rendered again by preserving and kept DB-side by immutable. materializeTimeline runs during preparation, before the body is imported, digested and rendered, and appends each round-trippable row after existing bullets in date-then-id order as a marked bullet (timeline-marker.ts); renderMaterializedBullet requires render-then-extract to return exactly the tuple and normalized detail, so backlink receipts, empty or delimiter-bearing sources are kept and never rendered. pendingTimelineRows counts materializable and kept rows for the repair and doctor. Deletes and detail refreshes match the pinned id, tuple and detail, so rows changed after preparation are untouched. Fence refusals are typed invalid_fence (fence-repair/refusal.ts): compileCanonicalProjections, the validation-only entry, refuses the first defect scanCanonicalFences finds (repeated marker, any strict-parser warning, a row number used twice on the page) with wire invalid_params; a new fence row whose number belongs to a database-only take refuses as stored_row_collision with wire take_row_collision. When the prior takes fence does not parse, the page's stored takes.row_num count as prior canonical rows (TE1), so a row Tier 1 normalized updates its take instead of colliding. unrecordedCanonicalTimeline returns the body's canonical tuples with no stored row under the normalized key (managed extract --stale inserts only these), and materializedHistoryRanges locates marker lines whose hash matches the bullet after it, plus that bullet's indented detail (the dream verifier's history exemption). Tests: test/canonical-projection-history.test.ts, test/timeline-materialize.test.ts (PGLite and Postgres). A write that would drop rows held by a fence quoted in markdown code (readers project nothing from it) is refused as invalid_fence / quoted_fence_rows instead of deleting them. Pinned by test/fence-marker-projection.test.ts. #8819: see fence-evidence.ts. #5969 (D3): an ordinary put_page / put_pages intent carries a TimelineWritePolicy (timeline-omission.ts) that changes only the rows the writer saw (removed, removed_marked): an explicitly empty section deletes them for every writer class; an omitted section keeps them for a local preserving writer (materialize) and deletes them with drop_timeline; maintenance, reconcile and file writers pass no policy. The pinned timeline DELETE returns what it removed, and the projection closure returns { timelineRowsRemoved: { count, earliest, latest } } counting only removed/removed_marked rows it actually deleted (a row changed after preparation is neither deleted nor counted); timelineRowsToRemove judges the same set ahead of publication. #6259: a quarantined page (gate quarantine marker) projects no facts or takes (timeline rows still project); rows projected before it was quarantined stay as they are.

  • src/core/persistence/timeline-omission.ts — #5969 (D3): timelineSectionOf reads a put_page body's Timeline section from the caller's raw content before normalization (omitted: no timeline sentinel and no ## Timeline/## History heading outside code; emptied: a section with no entries; present). Admission (page-mutations.ts) records it on the intent as timeline_section (a caller-sent value is stripped; failed-writes replay passes the stored one through submitPageMutation's timelineSection). assertTimelineNotOmitted refuses a remote omission that would delete rows with timeline_rows_would_be_removed (fix: get_page include_content:true, then the drop_timeline: true retry) before admission and again during preparation; timelineWritePolicy builds the projection policy. Successful deletes surface as the receipt's timeline_rows_removed advisory (page-advisories.ts, dates only), copied onto put_pages entries (page-batch.ts, which accepts drop_timeline per entry).

  • src/core/persistence/health.ts — pure advisory age/reason/action policy shared by receipts and trusted diagnostics. The nested public contract is optional, validated and explicitly allowlisted; thresholds are 30/120 seconds and polls 1/5/30 seconds. The journal batches at most 100 authorized receipts into one projected root query with a 500ms caller budget, retaining one in-flight query per engine until settlement. Database-only pending lookups use a partial source-incarnation/sequence index shared by both engines. Synchronous service reads coalesce only for the same request ID, retain at most four unresolved reads per engine until settlement, and permit cancellation only by the read's initiating waiter or shutdown. No health observation changes authority or durable state. Real authenticated legacy/OAuth recovery and the synthetic three-client import live in test/e2e/persistence-http-liveness.test.ts.

  • src/core/persistence/effect-facts.ts + page-advisories.ts + derived-extraction-gate.ts — bounded writer-lint/facts-backstop outbox. dispatchFactsBackstopEffect stamps the job's notabilityFilter from facts.page_write_notability_filter (default all; facts/notability-filter.ts), read inside the dispatch transaction so retries keep one filter (#6231). The job also carries the stable session put_page:<slug> (mirrors sync's sync:<slug>), so a re-extraction of the page feeds the #6333 would_dedup_same_page shadow counter and the facts it writes record that source_session (visible to recall --session). derivedExtractionSkip is the writer-confinement rule facts and Life Chronicle share (slug-bound/delegated/restricted → slug_bound_client; an operation-bound grant without extract_facts → operation_bound_client). Eligibility, anti-loop, selected configuration, original/current grants and source/page revision guards precede scheduling; a write that keeps the compiled_truth (the only text extraction reads) of a live, already-eligible page skips with body_unchanged only when a facts-absorb job bound to that page id has completed and no facts-backstop effect or startable facts-absorb job for it is still pending (the new revision would supersede it), so a timeline or frontmatter-only republish such as repair timeline queues no extraction for an extracted page, while a page no extraction ever completed for still queues one. Wave B PR 2: a segmented page queues the effect like an inline one (bodyAlreadyOffered is false for a segmented prior or an unread body); the facts-absorb job hands the backstop the page's bounded truthLead and its id, and runPipeline plans the extraction text with planSegmentedExtraction (extraction-watermark.ts: one streamed pass over the stored truth, head plus the newest tail window), so the job never reads the throwing body (test/large-pages/segmented-derived-work.test.ts). Page publication records the effect; job insertion and completion use its stable idempotency key. Managed backstop retains the committed original page request as the facts-absorb job authority, including OAuth/stdio origin; no fabricated CLI authority or blanket managed skip. Legacy unsupported raw queue/fence paths remain refused. Tests: test/persistence-facts-backstop.test.ts, test/managed-facts-backstop.test.ts.

  • src/core/persistence/effect-recovery-scan.ts — bounded owner-scoped recovery selection rotates through durable effect IDs even when an earlier root has no available native lock, and wraps to retry prior roots. It never changes recovery authority or releases reservations. test/persistence-effect-recovery-fairness.test.ts holds two real native roots, recovers a later mirror, then verifies wraparound cleanup and exact counters on both engines.

  • src/core/persistence/page-input.ts — freezes scoped subagent type normalization at durable admission, after existing-request replay. An explicit undeclared model-authored type becomes note with legacy_type retained using the then-active schema pack; a replay keeps its original normalized payload when the pack changes. Caller intent remains unchanged for idempotency checks. Raw input size is checked before normalization and malformed Markdown is left to the importer’s sanitized error contract. test/brain-allowlist.test.ts pins changed-pack replay, raw-size refusal, and private parse-error redaction. undeclaredPageTypeWarning covers every other put_page (not managed file imports): a type the write source's pack neither declares nor aliases is stored as written and the committed result carries type_warning (code: page_type_undeclared, cause, fix, docs), also logged; pinned by test/put-page-type-warning.test.ts.

  • src/core/persistence/params.ts — runtime-leaf mutation parameter and receipt schemas shared with the frozen verb registry. Its operation-contract import is type-only; runtime validators live in separate modules to avoid a registry initialization cycle. preconditions.ts validates UUID/revision/force syntax. purge-params.ts validates local-only purge shape/trust before admission and publication, keeping purge diagnostics out of generic precondition imports and unrelated command flag scans. test/persistence-cold-import.test.ts loads each validator/contract/registry entry in a fresh process and preserves schema and typed-error identity. CAPTURE_EVENT_PARAMS are full-surface-only.

  • src/core/persistence/write-wait.ts — the CLI write-wait and exit contract (#5232). resolveCliWriteWaitMs (--wait seconds > GBRAIN_WRITE_WAIT_MS > file-plane persistence.write_wait_ms > 30 s; agents keep AGENT_WRITE_WAIT_MS); writeExchangeBudget (transport deadline = wait + 15 s admission headroom; an explicit --timeout shortens the requested owner wait instead); replayWhilePending (replays the identical request ID while an older owner, a remote server or a subagent tool's own bounded wait reports pending; the caller-supplied sleep decides how each pause ends, which is how subagent tool writes stop on abort); writeErrorExitCode keys on an admitted non-terminal receipt (admittedPendingReceipt in types.ts): pending → PENDING_WRITE_EXIT_CODE (0 with --accept-pending/GBRAIN_ACCEPT_PENDING=1 when acceptPendingCovers holds: no diagnostic.owner stale-owner block, GBRA-248), anything else → 1. Pinned by test/pending-write-exit.serial.test.ts and test/pending-write-exit-owner.test.ts. pollCommand (#6255) renders the envelope's receipt fix.argv (gbrain write-request --brain <id> -- <request_id>), else pins that argv to the CLI routing; each argument is shell-quoted (src/core/shell-quote.ts, fix wave 13 P1.15).

  • src/core/persistence/{effect-schema,effect-model,effect-journal,effect-recovery,effect-git,effects}.ts — bounded postcommit outbox (migration 155). Canonical publication atomically queues Git and embedding work; withdrawals queue three versioned, bounded exact-page manifests with durable slug cursors. effect-targets.ts converts legacy source scans with complete bounded discovery keyed on the request's own forgotten fact (falling back to the whole source ledger only when that fact row is gone) while retaining each cursor; ambiguous or over-capacity legacy intent stays pending. Older workers must be quiesced before rollout. Independent mirror recovery reserves capacity before file effects, finishes forward without changing revision or receipt, and blocks only its worktree until reconciled. Git holds the native root lock, disables hooks, uses literal pathspecs and target-only commits, and pushes without pull/rebase. A claimed single-file Git effect brings up to 99 more ready single-file Git effects of its worktree (claimCoalescedGitEffects, same readiness, recovery and withdrawal-mirror ordering predicates, SKIP LOCKED); the runner validates each path under the worktree lock, makes one commit-tree per group (commitGitTargets: one ls-files, status, add and diff for the group; untracked absent or ignored targets take the per-target refusal path) and pushes each root once at the end of the pass (pushGitRoot). A page file target's optional commit note (GitCommitNote, set only by trusted local preparers: a fence repair's fenceRepairCommit) is queued as commit_subject/commit_line in the Git effect's data and passed by the runner to commitGitTargets/publishGitEffect: a noted path committing alone takes its subject; a group keeps gbrain: persist N canonical memory updates and lists the noted paths' lines in the body, in path order; control characters are stripped, and targets without a note commit exactly as before (test/fence-repair-receipts.test.ts). A group yields to publications still queued or running for its worktree (one update requeues it with publication_pending, 250 ms apart) for at most 20 claims, so a serial producer publishes ahead and its Git work lands as one group, and a write admitted behind a backlog of full groups takes the worktree lock after the group already committing (wave 9; on master it waited out the backlog and its 5 s write wait). claimCoalescedGitEffects checks the worktree-wide conditions (owner, refresh fence, recovery blocks) once as an InitPlan, so the claim walks persistence_effects_pending in order and stops at its limit (351 → 25 ms per claim with 22k ready effects). A path failure fails only its effect; a push failure leaves the group queued with git_push_unavailable, and the next pass finds nothing to commit and pushes once. Every path listing compares relative to the source root, which may sit below the Git toplevel (#7091: status prefix stripped, diff --relative). That holds for a removal too (#7089): its retry finds the path untracked and absent (target_absent), and it joins the push while the root is ahead of its upstream (gitRootAhead, a local rev-list @{upstream}..HEAD, no upstream is not ahead), as does publishGitEffect, recording target_absent with the push result; on a root that is not ahead it completes without a push, so a down remote never parks it. A Git page walk that ends while its durable root is ahead pushes before completing (test/persistence-git-removal-push-7089.test.ts + Postgres wrapper). In a root without the durability hook a group runs no git command: it takes each root's lock once, guards each source incarnation once, validates every path as before and records the durability_not_enabled outcomes in one update (a 50k import's ~30k queued effects drain inside a minute of serve) (#5530; tests: persistence-git-coalescing-5530.slow.test.ts + Postgres wrapper). A Git effect for a declared db_only page completes as skipped (reason: db_only), keeping its gitignored cache file out of Git. A Git or withdrawal target parks after five consecutive execution failures (data.target_failures for data.failing_target, reset when the target advances or a different target fails; shutdown cancellation, contention, dependency waits (including recovery-capacity waits), transient database failures and raw errors classifyItemFailure calls run-scoped (a lost connection, a failed transaction, an abort, a full disk) do not count). A links or facts-backstop effect counts its failures the same way (data.target_failures, no target; run-scoped failures, including run-scoped refusal codes, never count, and a 57014 counts as statement_timeout): at five a links effect completes as skipped with the code (outcome: { links: 'skipped', reason }; the edges are derived and gbrain extract --stale recovers them) and a facts-backstop effect fails with it; a pass's counted failures settle when the pass ends under an ItemRun (effects.derived_retry_budget), and a pass where most of them failed counts none (each retries as item_failures_systemic) (GBRA-130 A-11, test/pathological/a11-effect-retry-budget.test.ts). An embedding scan (a withdrawal's exact targets or a source scan) parks the page whose provider call keeps failing once its per-page budget (MAX_RATE_LIMIT_RETRIES) is spent, with the failure's code, and moves on with a fresh budget; provider throttling (isEmbedRetriableError), configuration, budget and disabled errors and run-scoped failures stay effect-wide (A-13, a13-embedding-target-park.test.ts): a scan records it in data.parked and moves on, a single-target effect or a failure outside any target parks the whole effect, and an effect with parked targets finishes as failed/targets_parked, never committed. A mirror or Git scan never parks on a sync-skip metafile page (RESOLVER.md carries the managed durability block by design) or on a file whose bytes hold an uncoordinated local edit (file_database_drift): the mirror materializes the withdrawal in the database without touching the file, both scans record the page in data.skipped (at most 100) and advance, so the request's Git and embedding effects are no longer gated behind a parked mirror (#5396). A page stored in segments is mirrored and committed too, streamed (see the segmented withdrawal mirror entry in core-persistence-request-indexes.md). Both claim queries hold a request's Git and embedding effects only while its withdrawal mirror is queued or running; each Git or embedding attempt rereads its mirror (mirrorGates, every mirror not committed plus a committed one that recorded its publications): a mirror queued or running again (retry-effects after the claim) makes the attempt wait (withdrawal_mirror_pending); otherwise the mirror's recovery records each page whose file it rewrote (data.published) and its cursor is never proof: a Git page walk commits a page the mirror passed only when the file's raw bytes hash-equal the mirror rendering of the current snapshot (the serialization mirrorPage writes; this also covers a mirror from before publications were recorded and a published file later reverted by hand) and otherwise skips it (withdrawal_not_mirrored), a single-file Git effect needs the recorded publication, embedding behind a failed mirror needs the recorded publication, and behind a failed mirror every other page, including a single-file Git effect's page in a coalesced group, is parked with withdrawal_mirror_failed, so Git never commits a file that may still hold withdrawn text. When the mirror commits, the same transaction re-arms its request's Git and embedding targets parked withdrawal_mirror_failed (rearmWithheldTargets: a scan revisits them through data.retry_slugs; other parks stay), and each effect pass repeats it for the requests it touched, so no second retry-effects is needed. Lifecycle settlement (topology-locks.ts) waits on a queued mirror, not a failed one (A-12, a12-withdrawal-mirror-gate.test.ts). Migration 170 indexes parked effects for the parked_effects doctor check. retryParkedEffects (the retry-effects admin path) previews parked targets or authorizes one more attempt per target (data.retry_slugs, visited before the cursor); it also covers parked embedding targets, failed Git, withdrawal-mirror, links and facts-backstop effects without parked targets (one more attempt; a mirror that failed source_changed reruns from its cursor once the source is back) and an embedding effect that failed source_changed, so one call reruns a failed mirror and re-arms the pages it withheld; parkedOnly keeps the credential-change retry (retryParkedPushes) to parked Git and withdrawal effects; tests: persistence-effect-parking.test.ts. repair embedding-effects skips failed embedding effects with parked targets and zero-norm refusals. Old targets resolve under the same hash/confinement checks without changing frozen data; ambiguous absent spellings refuse after scoped staged/unstaged deletion checks. Embedding runs outside transactions/OS locks and installs with projection CAS. Receipt helpers return only public effect state/reason/push. Consumer shutdown aborts providers and awaits workers; receipt compaction is bounded. Tests: persistence-effects.test.ts, persistence-git-publication.test.ts, persistence-publication-native.serial.test.ts. Embedding completion checks active column/vector, embedded stamp, text hash, model and page signature; partial work sends only stale chunks and preserves valid siblings. A partial synopsis rebuild completes before title demotion. Provider timeouts use the gateway budget, five durable page claims/backoff, execution-token CAS after provider work and atomic vector/stamp/effect completion. Keyless work skips cleanly. retry-effects is an explicit one-shot failed-effect allowance; current vectors reconcile provider-free without changing canonical receipts. A git refusal on an index lock (git's own message names the lock, so linked worktrees resolve) is git_index_locked contention (retried, not counted toward parking) while the lock is younger than 10 minutes, and git_index_stale (counted, naming the lock) when older or future-dated. Only the writer, holding its writer lock, removes a lock index-lock-reclaim.ts proves abandoned (its header lists the proof), with a receipt doctor parked_effects reports, then retries the step and the effects that lock parked; otherwise git_index_stale carries an ask_user fix. Steps run with GIT_OPTIONAL_LOCKS=0, an index-scaled deadline (gitStepTimeoutMs) and a 10 s SIGTERM grace (#6932). #9227: gitReceiptState maps a classic write's Git effect to the receipt publicReceipt (ops/persistence.ts) returns in place of the stored git_state: queued: committed with git_commit, or superseded, skipped, failed or retrying with git_reason (test/persistence-receipt-git-state.test.ts).

  • src/core/persistence/{ipc,local-client}.ts — dedicated local persistence socket and engine-free CLI delegation. The narrow protocol carries a durable brain UUID, existing local writer registration, client source/cwd, and one request UUID; the production provider verifies current authority and dispatches through operations. Max frame 32 MiB (five-million-byte content even with JSON escaping), eight connections/in-flight dispatches, incomplete-frame timeout 15s. Lost responses preserve the original request ID with unknown submission state and never automatically replay or open another engine. Brain/mount routing precedes lock inspection. A lock holder that is not a serve and answers no owner socket (a concurrent CLI call) is not a resident owner: maybeDelegateLocalOperation returns unhandled and the engine path waits for the PGLite lock instead of answering owner_unavailable; a serve holder or an answering socket keeps every failure final. Owner-side resolveSourceId(..., { skipLocalSignals: true }) prevents serve env/dotfiles from overriding client routing. For an op that owns a source param (opOwnsSource, src/core/ops/contract.ts), params.source stays on the wire as the op's own value (provenance) and routing comes from the ambient tiers, as on the direct and thin-client CLI routes (test/cli-op-owned-source.test.ts). The production provider dispatches the trusted local_cli principal's allow-listed skill-administration ops with the 'stdio' locality marker, so the four localOnly ones (get_skill_retention, prune_skill_revisions, retain_skill_revision, import_skill_proposal) work while a serve owns the brain, as do takes_remove and takes_rebuild (LOCAL_CLI_LOCAL_ONLY_OPS) for the local_cli principal (write scope still checked; gbrain takes remove and gbrain takes rebuild defer their engine like the other takes verbs); other local calls keep no marker (test/persistence-local-only-ops.test.ts). bindResolveIpcForServe accepts the provider as its third argument and closes the dedicated listener with hook IPC. residentPersistenceConfig resolves a resident serve's brain the same way (--brain, GBRAIN_BRAIN_ID, .gbrain-mount, mount path), so the owner socket and the provider's consumer config follow a mounted brain instead of the host config (#5237); hook IPC stays keyed off the host config. Tests: test/persistence-ipc.test.ts, test/persistence-local-client.test.ts, test/resolve-ipc-binding.test.ts, test/serve-mounted-owner.test.ts. A credential-free, read-only projection_status request (like capabilities) returns the resident's projection backlog counts (PersistenceProjectionStatus, no identifiers) so doctor's text_projection_readiness can report it while the resident holds a PGLite datastore (#5401, test/projection-resident-doctor.test.ts).

  • src/core/persistence/{schema,model,journal,authority,identity,ownership,coordinator,consumer,service,page-mutations,page-prepare,source-storage,control,context,digest,prepared-import}.ts — dedicated durable mutation admission and permanent principal-scoped replay IDs, atomic quota reservations, registered canonical worktree ownership, native lock publication, conditional file recovery, and coherent receipt completion. Recovery presence blocks its whole worktree even when the receipt is terminal; admission and publication both enforce the fence. A file write is not claimed while a queued or running withdrawal mirror targets its page (any page, for an untargeted mirror), so the mirror cannot rewrite the file under an accepted request; when file bytes differ from preparation and the page revision also moved, publication reprepares (revision_conflict) instead of refusing as raw_file_changed, which stays the outcome for changed bytes at an unchanged revision (test/withdrawal-followup-writes.test.ts). Migration 159 adds a partial worktree/sequence recovery index so permanent receipt retention does not turn the fence into a history scan. Canonical file preparation honors a recorded source path or contained absolute capture file URI within the registered owner root (relative or host-only file:// URIs are absent); symlink confinement and coherent snapshot checks still apply. A slug under a storage.db_only dir declared in gbrain.yml (matched case-insensitively, derive-phase defaults excluded) publishes database-only when a live page's file is absent (the case that otherwise refuses as an uncoordinated removal), with receipt write_through.skipped: db_only; new pages, tombstones, missing-file-tolerant callers and present cache files keep the full file treatment and never read gbrain.yml. gbrain.yml is read from the source root, like doctor, export and backup, not the git root manageGitignore uses, which differs for a monorepo-subdir source. source-storage.ts caches the parsed file per version and logs an invalid one (unreadable, overlapping tiers, db_only declared in syntax that resolves nothing) once; that missing-file refusal then reports storage_error, and the Git effect treats the page as not db_only. The drift guard applies the import rule for a file without an explicit type: (the stored type stands) and compares titles trimmed, as the parser reads them; a real drift refuses with detail: file_database_drift (over 24 MiB: body-digests.ts) and a filled sources reconcile command; when sync holds the page's file (sync-holds.ts), the drift and occupied-path refusals name the hold instead (location-free message from verb-errors.ts heldFileMessage, mapped back by heldFileDiagnostic in receipts and service.ts) and its repair, and the page stays read-only for put_page until the file is repaired. The first write that binds an origin in a Git-subdirectory source with no slug-root pin pins the inferred mode in its publication transaction (a concurrent different pin refuses with revision_conflict), so later pages cannot flip the inference. A new page captured from a file inside the source (capture_path, resolved from the trusted CLI local_file in page-mutations, only when sync would give that file the capture slug under the source's slug-root mode) publishes to that file when its bytes are exactly the captured input; otherwise capture records source_uri: cli-file, never a host path. acquireWorktree(binding, waitMs, signal, engine) re-stamps a device-only physical-root change under the native lock (see physical-root.ts); without an engine, or when birth time is zero, it refuses with detail: physical_root_device_changed and the filled self-transfer commands. Version snapshots retain nullable is_deleted: new snapshots restore recorded deletion state atomically with content, tags, files and receipts; legacy NULL preserves current deletion state. Canonical no-op detection compares deletion state as well as content. Page replacement preconditions are checked before no-op detection; providers prepare outside publication transactions; recovery never restores unexpected bytes. Local registrations retain immutable CLI/stdio trust lanes and hashed credentials; accepted authority is intersected with current grants. limits.ts reads every persistence.limits.* cap and persistence.receipt_retention_days (default 30) from the database config table (JOURNAL_CONFIG_KEYS, registered for gbrain config set; malformed values are rejected at set time). Cumulative defaults (250,000 lifetime IDs and 1.5 GiB receipt bytes per principal, 8 GiB brain receipt bytes) cover 600 admissions a day for a year; each admission reserves at least 16 KiB and a compacted receipt keeps about 4 KiB. A cumulative queue_capacity refusal names its key in detail and suggests a gbrain config set value covering one more year at the recent admission rate (oneYearCapacity). Compaction filters out receipts with unfinished effects before its batch limit. test/persistence-journal.test.ts exercises real PGLite and optional guarded Postgres admission, cancellation, compaction, conflict and publication recovery; test/persistence-capacity.test.ts covers retention, the compaction filter, capacity hints and default sizing. service.ts also dispatches operation decide_proposal (S9 proposal accept/undo) to facts/proposal-supersede.ts. FENCE REPAIR RECEIPT: fence_repair is a reserved put_page param; preparePageAdmission refuses it (invalid_params) on every operation unless the call is a trusted local (ctx.remote === false) put_page whose value parseFenceRepairReceipt accepts; page-prepare.ts re-parses it from the stored intent, returns it in the outcome as fence_repair and gives the page's file target the fenceRepairCommit note for its source-relative path. GOOGLE PUBLISH MODE: a page target is a PageMutationFile (MutationFile plus optional publishMode and commit). The Google callers set publishMode: 0o600 at prepare time from the stored connector (prepareConnectorMutation in connector-sync.ts when p.connector === 'google', and prepareGoogleReceiptPatch); it is never part of the submitted intent, so request ids are unchanged. publishFile then creates missing directories with mkdirPrivate(dirname(path), root) (0700) instead of mkdirSync and opens the stage with mode ?? publishMode through AtomicWriteOpts.mode, so a rewrite of a legacy 0644 page lands 0600. Restoration keeps restoring the journaled before-mode and opens its stage with it (the post-flush chmod stays), so a restored private page is never staged wider than its mode.

  • src/core/persistence/protocol.ts — the persistence protocol declaration and the global row-lock order: the persistence_brain row, then worktree rows (id order), then source rows (id order), then persistence_counters, request rows and page keys. The request and effect protocol triggers (shared-skills/persistence-schema.ts) read the brain row FOR SHARE, and worktree claims, topology changes (lockTopologyRows, then settleTopologyRequests locks the brain counter) and activation hold it FOR UPDATE before their worktrees, sources and counters. So declarePersistenceProtocol and declareDurablePersistence take the brain row FOR SHARE in the same statement as their settings (after lock_timeout, a missing row still declares), and every journal transaction declares before any other row lock: admission, publication, recovery, completion, cancellation (control.ts), forget (memory-mutations.ts), connector retry approval, embedding effect retry and settlement, and withdrawal-target upgrades (effect-targets.ts). A transaction that needs the brain row FOR UPDATE takes it first and declares after it (skill-activation.ts). Before this order, an ordinary write that locked its source and counters before the trigger's brain read inverted against a concurrent claim or topology change and ended in a 1 s lock_timeout (or a deadlock), a retry, and writes left pending. Tests: test/e2e/persistence-publish-lock-order-postgres.test.ts (paused topology change and a claim/topology race, direct and through PgBouncer), the crash robot's brain_before_rows rule (scripts/persistence/lock-order.ts, test/persistence-crash-robot.test.ts). isPersistenceDeclaration and the BEGIN pipelining built on it: Core Persistence (continued).

  • src/core/persistence/{semantic,semantic-pages}.ts — revision-bound semantic tag/timeline merges through the journal. Supported merges reprepare against a newer snapshot (mayReprepare: revision_conflict or, #5385 [R13], revision_required, both only when the caller named no expected_revision); exact timeline replay leaves canonical bytes, revision, versions and structured rows unchanged. The add_timeline_entry receipt carries bullet, the rendered Markdown line (#5970): there is no update/remove verb because the page is the system of record, so the caller corrects an entry by editing that line with edit_page and the row follows the page (both timeline op descriptions say so; test/timeline-correction-path-5970.test.ts). #5251: add_tag / remove_tag prepare with deferEmbedding: true (a tag change leaves every chunk's text as it was, so no embedding effect is queued and embedding_state reads deferred; the stale sweep still covers any chunk the republication left without a vector); the full page republication they ride stays, so the issue stays open. The public timeline handler resolves durable replay before page-existence preflight, retaining original outcomes after page purge or recreation; fresh writes preserve same-source diagnostics. page-state/projections.ts completes sanitized text and page search vectors together, and installs delayed embeddings with UPDATE-only revision/chunk/text/index-context checks. New vectors receive the full provider:model captured before provider execution, and a runtime model switch invalidates the prepared indexing context. Metadata updates preserve the sanitized vector. Tests: persistence-semantic-pages.test.ts, page-projection-concurrency.test.ts.

  • src/core/persistence/{source-lifecycle,managed-sources,topology-schema,topology-locks,topology-receipts,topology-clone,topology-clone-model,topology-filesystem,topology-recovery}.ts — migration 157 coordinates source add/claim/archive/restore/remove/purge/rebind/reclone. Native root locks precede topology/source/grant/counter/request guards; withTopologyLocks(..., { relocation: true }) (rebind only, #5914 P1.3c) verifies a recorded root that now resolves through a symlink to the rebind target with inspectRelocatedPhysicalRoot instead of the realpath check, and the rebind transaction then calls relocatePhysicalRoot and moves local_path, the host binding and relative_path in place of a new worktree; a shared root advances every membership and permanently conflicts accepted old-topology requests. Withdrawal mirrors and running publication must settle first. Administrative UUIDs share the page journal's principal-scoped conflict domain and permanent capacity counters. Directory replacement reserves the configured recovery budget before provider/staging work, records a verified manifest and physical identity, and recovers only known bytes; clone failure retains the original checkout and a replayable failure receipt. A managed add by a remote caller records added_by_remote_caller in the source config (managedSourceAddInput), and an add or reclone of such a source passes skipLfs to cloneTopologyCheckout, which writes LFS_OFF_CLONE_CONFIG (git-remote.ts) into the clone as cloneRepo does. Rebinding accepts a fresh candidate with exactly matching canonical content, including deletions. Stored worktree manifests and clone recovery records keep digest, file_count, canonical_stamp and self_transfer but no per-file map (#5790), so their size is independent of the worktree; compactStoredManifest drops a legacy map on rewrite, and transfer prepare/accept hash under the native lock before their transaction. #5200 (W14 P1.4): only a lifecycle operation that compares or records checkout bytes (add at an existing root, claim, rebind, restore; reclone in topology-clone.ts) walks and hashes the checkout; archive, remove and purge skip the walk when every binding of the source already carries a verified digest and refresh the stored manifest's canonical_stamp in place (manifest = COALESCE(manifest,'{}') || {canonical_stamp}), so a symlink that appeared after the claim no longer refuses a retire (a binding claimed before activation has no digest yet, and the retire records that first manifest because clone recovery needs it), and every dry run reports manifest_required so preview and real run agree (test/source-retire-symlink-manifest.test.ts). The writer_manifest_unsafe text names the manifest, not a transfer, since claim and rebind raise it too. Large-repo tests: test/persistence-large-manifest.test.ts (12k files, both engines via test/e2e/persistence-large-manifest-postgres.test.ts) and test/e2e/persistence-large-manifest-50k.test.ts. Old root refusal records remain. Metadata deletion retains local storage. Restart recovery rotates a bounded root scan so one conflict cannot starve other roots. Tests: test/persistence-source-lifecycle.test.ts on PGLite and guarded isolated Postgres. #6099: manifests are Git-scoped for Git checkouts; see worktree-manifest.ts in managed sync and persistence stalls.