Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs: keep core-persistence key-files under the 60000-byte cap (#7873)
garrytan3a09522f6645
diff --git a/docs/architecture/key-files/core-persistence.md b/docs/architecture/key-files/core-persistence.mdindex 8e66f445c1..f58abe7055 100644--- a/docs/architecture/key-files/core-persistence.md+++ b/docs/architecture/key-files/core-persistence.md@@ -6,8 +6,6 @@ The page operations that submit through this layer are in [Core Services (2/3)](Connector ingestion, item holds, checkpoint validation and the single-write fast path (`switches.ts`) continue in [Core Persistence (continued)](core-persistence-continued.md).Managed facts extraction (withdrawn-candidate skip and recovery, conversation facts batches, the unmanaged derived index) is in [Core Persistence (managed facts)](core-persistence-facts.md).-- `src/core/persistence/connector-google-receipts.ts` — the managed Google attachment-repair preparer checks account/thread/message ownership from the current canonical snapshot, changes only the owned receipt frontmatter field, and installs provider-free text projections under the existing source-before-page publication locks. Page identity/revision, current authorization, source incarnation and file recovery guards remain mandatory; body, unrelated frontmatter, tags and the withdrawal ledger are preserved. It schedules no embeddings, loop detection or fact extraction. `connector-sync.ts` admits this narrow mutation and banks its committed receipts through the existing managed connector checkpoints.-- `src/core/persistence/authority.ts` — durable write authority. `submissionAuthority` records the caller's grant ceiling on every write request (scopes, operations, slug prefixes, delegated namespace and, for a delegated job, its native `subagentId` when a positive safe integer); `authorizeWrite` re-checks it against the CURRENT principal at publication and replay. For a delegated OAuth client the current namespace is the stored `delegated_slug_prefixes` in `prefixes` mode, or only `wiki/agents/<retained subagentId>/*` for the native `job` form (prefixes NULL); anything else allows nothing, and it always intersects the original accepted prefixes, bound tools, scope and source, so a revoked, narrowed or switched grant or a foreign or invalid job id denies (`test/persistence-page-authority.test.ts`, Postgres: `test/e2e/minions-authority-parity.test.ts`). #5994: a context carrying `replayAuthority` (set only by `gbrain repair failed-writes`) gets that stored authority back from `submissionAuthority` (`replayedAuthority`): same source incarnation or refused, `databaseOnlyReason`/`takeHoldersUsed` dropped for admission to recompute, then `authorizeWrite` and page visibility against the live grant.- `src/core/persistence/activation.ts` — private CLI `sources writer activate --confirm-quiesced` boundary. Verifies native capability and filesystem bindings, takes root locks before DB waits and rechecks topology under transaction guards. Live/young/foreign legacy leases refuse regardless of expiry; only explicitly requested `--cleanup-dead-local-locks` removes exact dead same-host holders under the lease lock. Dry-run deletes nothing. Never silently claims a source. Tests: `test/persistence-activation.test.ts`, `test/persistence-onboarding.test.ts`. Unbound Google/GitHub API sources may activate as reviewed `connector_database` without fabricating a Git root, while bound sources still need physical-root checks. Activation increments `persistence_brain.mode_epoch` (migration 183), which the writer admin-state hash includes. A non-connector source whose recorded `local_path` no longer exists or is relative (unclaimed, or claimed by this host) refuses `source_changed` with detail `source_path_missing`, naming the source, the path and the `sources set-path` / `sources remove` exits (the claimed checkout filled in), in `--dry-run` too (#5206, `test/activation-missing-source-path.test.ts`). The report's `warnings[]` (dry run and real run) carries one `git_durability_not_enabled` entry per filesystem source the effect runner's policy (`gitDurabilityPolicy`) would leave non-durable, with `enable_command`, a read-only `git-durability --enable --dry-run` fix and the note that enabling re-queues skipped Git effects; `persistence-admin.ts` also prints them to stderr without `--json`, and activation proceeds (#5182, `test/writer-activate-durability-warnings-5182.test.ts`).@@ -17,7 +15,7 @@ Managed facts extraction (withdrawn-candidate skip and recovery, conversation fa- `src/core/persistence/{admin-lock,blocking-effects,writer-versions}.ts` — `admin-lock.ts` owns the opt-in brain-level writer admin lock: reserved config key `persistence.writer_admin_lock` (`{locked, set_at, host_id, generation}`), written only by local `gbrain sources writer lock|unlock` under the `persistence_brain` row `FOR UPDATE`; lock refuses while a prepared transfer is unaccepted. `assertWriterAdminUnlocked` runs inside the claim, managed claim, activation, shared-skill activation and transfer prepare/accept transactions for every caller (share lock on the brain row) and refuses with `writer_admin_locked`; only the ordinary first-write auto-claim (`claimWorktree(..., { automatic: true })` from page and memory mutations) is exempt. Generic `config set`/`unset`/`unset --pattern` refuse the key; engine migration copies it. It is not a security boundary and older binaries ignore it. `blocking-effects.ts` lists queued, running or recovering effects (`listBlockingEffects`) and builds `writer_not_quiesced` suggestions naming the first blocking request or effect with `gbrain sources writer status <source> --json`. `writer-versions.ts` stamps admitter version/host at admission and consumer version/host/`published_at` only on committed publication (migration 178, cutoff `persistence_brain.writer_version_cutoff` on the database clock); replay and compaction keep stamps. `sources writer status` reports `admin_lock`, `local_host_id`, `blocking_effects` and `writer_versions`; `src/commands/doctor/checks/writer-version.ts` is the seven-day `writer_version` advisory. `SEGMENTED_WRITER_FLOOR` (`page-body/writer-floor.ts`): a seven-day writer below it or unmarked keeps `pages.segmented_storage` off (`page-body/limits.ts`, doctor `segmented_storage`); marked rows of any role meet the floor (ingest log: two partial indexes, v254/v259). Tests: `test/persistence-writer-admin-lock.test.ts`, `test/persistence-writer-versions.test.ts`, `test/persistence-blocking-effects.test.ts`, `test/e2e/persistence-writer-{admin-lock,stamps}.test.ts`.-- `src/core/persistence/{atom-maintenance,atom-retry,page-prepare,facts-maintenance,facts-prepare,prepared-maintenance}.ts` — resident dispatch for durable atom publication/completion, explicit atom retry (`retryManagedAtomBatch` takes the job's signal and deadline, stops between stages on caller cancel, deadline or cycle-lock loss, shares one deadline-bound `MaintenanceWriteWait` with the model rerun, and reports a stop after admission as `write_pending` or an `AbortError` carrying the receipts, never `cancelled`; #6622; a transcript item that does not resolve to the retry's source is refused `transcript_source_mismatch` before anything starts, #7869), authorized multi-entity fact-fence/backstop append, local synthesize/patterns/consolidation, and the cycle's lint repairs on a managed brain (`runLintCore` publishes each fix through `maintenancePreflight` + `publishMaintenancePage`; #5180, `test/managed-lint.test.ts`). `page-prepare.ts` makes the provenance stamps (`source_kind`, `ingested_via`, `ingested_at`) server-owned on every remote `put_page`, `edit_page`, `append_page` and `capture`, with or without a repository (`keepStoredProvenance` drops the caller's values and keeps the stored ones; `provenanceWriteTrust` rates the write at the lower of the caller's frontmatter and the frontmatter it publishes; #7873; local callers unchanged); it exports `fileMatchesSnapshot`, the canonical file-versus-page comparison `prepareFileTarget` applies, so lint can bind a repair to the revision it read (it takes the importer's path for the file: a frontmatter-less file under a directory-inference rule also matches when `readCanonicalAsImported` (BOM strip + `applyInference` at `scannerSourcePath`) reproduces the stored row, with an inferred type treated as undeclared, #6605; reconcile reads the file side through `reconcileFileReading` in `reconcile-state.ts`); the uncoordinated-local-edit refusal carries detail `file_database_drift` and a `sources reconcile <src> <slug> --brain host --preview` fix; a shared-origin delete: canonical-reconciliation.md. In `prepareFileTarget`, a live page whose canonical file is missing refuses `source_changed` unless it is declared or derive-phase database-only, or it is a `delete_page` (`deleting`, no content) of a page with no recorded `source_path` and no captured file URI: that delete plans no file, so it publishes database-only and its receipt reports no file or Git effect (`test/persistence-captured-file-target.test.ts`, `test/persistence-never-published-delete.test.ts`). `atomOwnerTarget` is the one owner decision for an atom session and for `atomDrainOwnerRefusal`, the auto-drain dispatch pre-check; an unbound Google/GitHub source publishes atom maintenance through the `connector_database` target (its `local_path` is connector state, not a checkout) and the publication refuses with `source_changed` when the source gained an owner after preflight. `submitMaintenance` scans every content-carrying intent with `scanCanonicalFences` (the check the canonical projection runs) over the whole submitted body and timeline before a request exists (#6278): a defect no adoption clears (a second facts fence in the timeline; `replaceOrInsertFactsFence` replaces only the first) is refused with its fence reason, so `fenceManaged` reports the page in `failed_pages` and the others still adopt, while a repair whose postimage is clean passes. `prepareFactFenceAdoption` names its steps for `enterClaimStep`, compares the fence cell with the legacy fact modulo whitespace, writes the cell's parsed text back into `facts.fact` in the same `UPDATE` that sets `row_num` (so the projection keeps the fact's id), and throws the registered `fence_unrenderable` (class `server`) as its last-line defence. `maintenancePreflight` creates the job's `MaintenanceWriteWait` (optional `deadlineAtMs`) and `submitMaintenance` uses it for new and replayed requests; its owner check goes through `checkOwner` / `ownerUnavailableError` (`owner-refusal.ts`), so a missing binding, a changed incarnation, another owner host, a draining or recovering worktree and a missing local or coordination path each refuse `owner_unavailable` with their own `reason`, and nothing claims or transfers ownership to get past them. #6325: `resumeManagedAtoms` returns `run | done | held`. The drain does not replay a failed batch: it resumes the next automatic attempt, whose run key is `digest([drainKey, failedRequestId, 'automatic'])` (one stable id per failure, so a crashed attempt resumes and two runs admit the same requests), and whose completion advances the failed drain-key checkpoint by the same compare-and-set an explicit retry uses. The completion counts one strike per completed failure on that checkpoint (`attempts`); at `MAX_DETERMINISTIC_FAILURES` it sets `held`, which `MANAGED_ATOM_DISCOVERY_SQL` excludes from discovery and the backlog and `listHeldManagedAtoms` reports (doctor `extract_atoms_held_failed`, code `held_failed`). Reading a held or failed checkpoint never strikes; storage and authority errors fail the attempt without completing it, so they never strike either. #7514 write-failure strikes and hold: [continued](core-persistence-continued.md). A content change is a new drain key, so the count starts over. An explicit retry (`session.retry`) ignores `held`, and its failure refusal (`extraction_failed`) names the recorded failure reason (malformed output or a response stopped before the end) and carries a paid `fix` (`gbrain jobs submit extract-atoms-drain --params '{sourceId, retryRequestId}'`) with a read-only `gbrain write-request` verify; the drain raises it only for a failed batch whose checkpoint is gone. Tests: `test/managed-atoms-retry.test.ts`. Managed facts extraction (the withdrawn-candidate skip and its recovery route, the conversation facts batches, the unmanaged derived index) continues in [Core Persistence (managed facts)](core-persistence-facts.md).+- `src/core/persistence/{atom-maintenance,atom-retry,page-prepare,facts-maintenance,facts-prepare,prepared-maintenance}.ts` — resident dispatch for durable atom publication/completion, explicit atom retry (`retryManagedAtomBatch` takes the job's signal and deadline, stops between stages on caller cancel, deadline or cycle-lock loss, shares one deadline-bound `MaintenanceWriteWait` with the model rerun, and reports a stop after admission as `write_pending` or an `AbortError` carrying the receipts, never `cancelled`; #6622; a transcript item that does not resolve to the retry's source is refused `transcript_source_mismatch` before anything starts, #7869), authorized multi-entity fact-fence/backstop append, local synthesize/patterns/consolidation, and the cycle's lint repairs on a managed brain (`runLintCore` publishes each fix through `maintenancePreflight` + `publishMaintenancePage`; #5180, `test/managed-lint.test.ts`). `page-prepare.ts` keeps the stored provenance stamps on every remote page write, repo or not (`keepStoredProvenance`, `provenanceWriteTrust`: tier by the lower of caller and published frontmatter; #7873); it exports `fileMatchesSnapshot`, the canonical file-versus-page comparison `prepareFileTarget` applies, so lint can bind a repair to the revision it read (it takes the importer's path for the file: a frontmatter-less file under a directory-inference rule also matches when `readCanonicalAsImported` (BOM strip + `applyInference` at `scannerSourcePath`) reproduces the stored row, with an inferred type treated as undeclared, #6605; reconcile reads the file side through `reconcileFileReading` in `reconcile-state.ts`); the uncoordinated-local-edit refusal carries detail `file_database_drift` and a `sources reconcile <src> <slug> --brain host --preview` fix; a shared-origin delete: canonical-reconciliation.md. In `prepareFileTarget`, a live page whose canonical file is missing refuses `source_changed` unless it is declared or derive-phase database-only, or it is a `delete_page` (`deleting`, no content) of a page with no recorded `source_path` and no captured file URI: that delete plans no file, so it publishes database-only and its receipt reports no file or Git effect (`test/persistence-captured-file-target.test.ts`, `test/persistence-never-published-delete.test.ts`). `atomOwnerTarget` is the one owner decision for an atom session and for `atomDrainOwnerRefusal`, the auto-drain dispatch pre-check; an unbound Google/GitHub source publishes atom maintenance through the `connector_database` target (its `local_path` is connector state, not a checkout) and the publication refuses with `source_changed` when the source gained an owner after preflight. `submitMaintenance` scans every content-carrying intent with `scanCanonicalFences` (the check the canonical projection runs) over the whole submitted body and timeline before a request exists (#6278): a defect no adoption clears (a second facts fence in the timeline; `replaceOrInsertFactsFence` replaces only the first) is refused with its fence reason, so `fenceManaged` reports the page in `failed_pages` and the others still adopt, while a repair whose postimage is clean passes. `prepareFactFenceAdoption` names its steps for `enterClaimStep`, compares the fence cell with the legacy fact modulo whitespace, writes the cell's parsed text back into `facts.fact` in the same `UPDATE` that sets `row_num` (so the projection keeps the fact's id), and throws the registered `fence_unrenderable` (class `server`) as its last-line defence. `maintenancePreflight` creates the job's `MaintenanceWriteWait` (optional `deadlineAtMs`) and `submitMaintenance` uses it for new and replayed requests; its owner check goes through `checkOwner` / `ownerUnavailableError` (`owner-refusal.ts`), so a missing binding, a changed incarnation, another owner host, a draining or recovering worktree and a missing local or coordination path each refuse `owner_unavailable` with their own `reason`, and nothing claims or transfers ownership to get past them. #6325: `resumeManagedAtoms` returns `run | done | held`. The drain does not replay a failed batch: it resumes the next automatic attempt, whose run key is `digest([drainKey, failedRequestId, 'automatic'])` (one stable id per failure, so a crashed attempt resumes and two runs admit the same requests), and whose completion advances the failed drain-key checkpoint by the same compare-and-set an explicit retry uses. The completion counts one strike per completed failure on that checkpoint (`attempts`); at `MAX_DETERMINISTIC_FAILURES` it sets `held`, which `MANAGED_ATOM_DISCOVERY_SQL` excludes from discovery and the backlog and `listHeldManagedAtoms` reports (doctor `extract_atoms_held_failed`, code `held_failed`). Reading a held or failed checkpoint never strikes; storage and authority errors fail the attempt without completing it, so they never strike either. #7514 write-failure strikes and hold: [continued](core-persistence-continued.md). A content change is a new drain key, so the count starts over. An explicit retry (`session.retry`) ignores `held`, and its failure refusal (`extraction_failed`) names the recorded failure reason (malformed output or a response stopped before the end) and carries a paid `fix` (`gbrain jobs submit extract-atoms-drain --params '{sourceId, retryRequestId}'`) with a read-only `gbrain write-request` verify; the drain raises it only for a failed batch whose checkpoint is gone. Tests: `test/managed-atoms-retry.test.ts`. Managed facts extraction (the withdrawn-candidate skip and its recovery route, the conversation facts batches, the unmanaged derived index) continues in [Core Persistence (managed facts)](core-persistence-facts.md).Atom completion admits replacement and revision-pinned stale-atom deletion children together. A failed, partial-input or zero-yield batch admits no retirement, and each retirement child refuses `revision_conflict` until every replacement page of its run is committed (`replacementPublished`), so a partly published batch leaves the prior generation live.Each retirement stamps `retired_by`/`retired_at` into the tombstone in the same publication; `publishManagedAtoms` restores a retired atom of the same origin when an extraction produces it again, and a user-deleted atom stays deleted. Each committed retirement bumps the origin page's regeneration generation (`op_checkpoints` op `managed-atoms-generation`, part of the drain's run key only above 0, never of the explicit retry identity; the content whose extraction retired keeps the generation its accepted batch used, so that batch stays resumable) and clears that page's older `extract_atoms_page_state` rows and completed `managed-atoms` checkpoints in the same publication, so an A→B→A edit, even after a partly failed batch, re-extracts instead of replaying a retained receipt. Retirements are admitted in slug order.Derived atom pages may regenerate or retire when their canonical file is already absent, while ordinary page writes retain missing-file refusal.diff --git a/docs/architecture/key-files/google-api-access.md b/docs/architecture/key-files/google-api-access.mdindex a387e7dc8f..8ce949aede 100644--- a/docs/architecture/key-files/google-api-access.md+++ b/docs/architecture/key-files/google-api-access.md@@ -9,3 +9,4 @@- When the Gmail label list itself fails, the loop exclusion log (`loops-enqueue.ts`) says the names could not be checked, never that they are not labels of the account.- Tests: `test/google-api-access-7073.test.ts`, `test/google-api-access-cmd-7073.serial.test.ts`, `test/google-api-access-managed-7073.test.ts` (real PGLite, and Postgres under `DATABASE_URL`).- `src/commands/doctor/checks/google-api-access.ts` — doctor `google_api_access` (wave check, resolution operator, zero network): warns per Google source whose stored `api_access` names a service it still syncs, listing each disabled API with project and enable link, with `googleApiAccessFix` for the first source.+- `src/core/persistence/connector-google-receipts.ts` — the managed Google attachment-repair preparer checks account/thread/message ownership from the current canonical snapshot, changes only the owned receipt frontmatter field, and installs provider-free text projections under the existing source-before-page publication locks. Page identity/revision, current authorization, source incarnation and file recovery guards remain mandatory; body, unrelated frontmatter, tags and the withdrawal ledger are preserved. It schedules no embeddings, loop detection or fact extraction. `connector-sync.ts` admits this narrow mutation and banks its committed receipts through the existing managed connector checkpoints.