Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs(key-files): keep canonical-reconciliation.md under its cap after the lane merges
garrytan35c5f7327a2d
diff --git a/docs/architecture/key-files/canonical-reconciliation.md b/docs/architecture/key-files/canonical-reconciliation.mdindex 4279cb09b3..543304c959 100644--- a/docs/architecture/key-files/canonical-reconciliation.md+++ b/docs/architecture/key-files/canonical-reconciliation.md@@ -17,7 +17,7 @@ See [generations](generations.md).- `src/core/persistence/managed-mode.ts` — the one reader of the brain's persistence mode (#6482). `readManagedMode` / `readPersistenceBrain` accept exactly one `persistence_brain` row with a boolean `enabled` (optionally `FOR SHARE` / `FOR UPDATE`) and refuse anything else (`storage_error`, `detail: persistence_mode_unknown`, fix `gbrain sources writer status --probe --json`); only a well-formed `enabled: false` row selects the unmanaged path, so a missing, duplicated or malformed row can never route a write around the owner. Every non-test reader of the mode goes through it (`test/persistence-mode-reader-guard.test.ts` lists the few schema and admin-intent files allowed their own statement); `ownership.ts` re-exports it for callers that already import ownership.- `src/core/persistence/sync-slices.ts` — #6349 path-sliced staging for a managed target that exceeds the cursor bound with no commit axis to stage on (a first sync, a `--full` tree walk, one commit above the bound). `sliceEntries` cuts the whole resolved manifest in `(phase, code-unit path)` order: every import slice, then bounded deletion slices, the longest prefix the bound admits. The slice cursor `{ phase, after, head, target }` lives in `op_checkpoints` (`managed-sync-slice`, keyed by source, incarnation and the run's cursor options so a resumed invocation with other exclusions starts over), written or cleared by the slice's checkpoint in the same transaction that leaves `last_commit` untouched until the last slice settles; `sync-discovery.ts` resumes an unfinished target pinned at its recorded commit.-- **`src/core/persistence/sync-discovery.ts`** + **`sync-origin.ts`** — managed owner-only Git discovery with source-relative exclusions and one-statement page identity/revision/liveness snapshots. A root with no Git checkout at or above it refuses `sync_not_applicable` (`reason: plain_directory`, fix `gbrain sync --source <id>`). #6517: `selectOriginClaimant` (sync-origin.ts) is the one origin rule, applied by the discovery census and by `assertSyncPageOrigin` at preparation and publication: a live page wins over tombstones, several live pages refuse `page_identity_changed` naming the soft-delete exit (`sharedOriginExit`), a single tombstone may be reused, several tombstones stay ambiguous. Reserved skillpack paths (`skills/` at any depth and `skillpack.json`, `src/core/skill-reserved-paths.ts`) never enter discovery: the shared skill publisher owns them and `managedImportContent` refuses them, so a seeded pack cannot block a sync. Nested full discovery filters source scope before slicing paths. Windows separator aliases retain their page identity; ambiguous aliases refuse, while POSIX backslashes/case remain exact. Under `source-root` mode a stored `<scope>/<path>` origin (the Git-root form older write-through minted from the slug, recognized because `<path>` names the page slug) is accepted as `<path>` at discovery, freeze, preparation and the page-origin query, and an import rewrites it in place; when both readings name existing files the stored origin refuses with `detail: ambiguous_source_path`. A genuine origin mismatch refuses with `detail: sync_origin_mismatch` and the filled paths. Deletions must be absent from the pinned Git target or their explicitly selected working tree. Deleting a file that no page records as its origin (for example a skipped slug twin) while its slug's page keeps another origin commits an explicit no-op receipt (`unownedDeletion`, outcome reason `unowned_deleted_path`) that preserves that live page or tombstone; freeze and publication re-prove that no page owns the path and that the file is still absent. Existing cursors reuse their accepted manifests, with unknown old deletion provenance refused rather than rewritten. Active Git holds of the source are re-screened when their blob or working bytes changed, the file left, a `sources retry-held` request names them, the hold predates the current `RECOVERY_VERSION`, or the code is `frontmatter_slug_conflict`, `file_too_large`, `rename_held` or `parser_regression`; holds of now-excluded paths are released at the checkpoint. #6278 (B7): `retryTaken` lists every scheduled path the discovery read (screened as an entry, as a candidate, or no longer eligible); `sync-run.ts` clears them in the transaction that saves the fresh cursor. A held rename destination (or a held path renamed again) carries the hold's `renameFrom` when the old page is unchanged, else `renameHeld` (held as `rename_held` / `rename_source_changed`), and a full walk never deletes an origin a hold's `rename_from` names. Imports whose origin has no page claim their slug together: a live page whose own file is still in the tree keeps the slug, otherwise the file named like the slug (else the first path) wins, and the losers leave the manifest and are reported as `slugCollisions` (the library path's `slug_collision` skip) instead of failing the run. A winner whose slug is held by a page with a vanished or deleted origin takes that page over (`renameFrom`, same slug). A Git-detected rename to an unoccupied slug becomes one import carrying `renameFrom` in place of a delete plus a create. Origins for one slug that differ only by case or separator spelling refuse, except a case-only respelling whose old file the same sync removes. Code files keep exact per-origin identity. On Windows a path containing `:` is reported in `SyncDiscovery.fileRefusals` (code `colon_slug_windows_write_through`, with its filled `gbrain sync --source <id> --no-pull` hint) and kept out of the run and its origin checks, so one such file no longer fails the whole sync (#5032). `resolveManagedSyncContext` refuses `skipFailed` / `includeGitignored` on a managed source with `managed_sync_flag_refused` (`managed-sync-flags.ts`, `detail` = the flag, fix = the same `gbrain sync` argv without it), the one refusal jobs, MCP and the cycle reach; `gbrain sync` raises it at parse, before the legacy ledger acknowledgement and the cost gate, `--all` included (#6696). #6865: the per-path eligibility (scope, `sync.exclude`, `sync.include_hidden`, strategy, reserved skillpack paths) and the origin a path maps to under the slug mode come from one exported `loadSyncEligibility(engine, context, opts?, company?)`, which the unindexed-file detector shares.+- **`src/core/persistence/sync-discovery.ts`** + **`sync-origin.ts`** — managed owner-only Git discovery with source-relative exclusions and one-statement page identity/revision/liveness snapshots. A non-Git root refuses `sync_not_applicable`. #6517: `selectOriginClaimant` (sync-origin.ts) is the one origin rule, applied by the discovery census and by `assertSyncPageOrigin` at preparation and publication: a live page wins over tombstones, several live pages refuse `page_identity_changed` naming the soft-delete exit (`sharedOriginExit`), a single tombstone may be reused, several tombstones stay ambiguous. Reserved skillpack paths (`skills/` at any depth and `skillpack.json`, `src/core/skill-reserved-paths.ts`) never enter discovery: the shared skill publisher owns them and `managedImportContent` refuses them, so a seeded pack cannot block a sync. Nested full discovery filters source scope before slicing paths. Windows separator aliases retain their page identity; ambiguous aliases refuse, while POSIX backslashes/case remain exact. Under `source-root` mode a stored `<scope>/<path>` origin (the Git-root form older write-through minted from the slug, recognized because `<path>` names the page slug) is accepted as `<path>` at discovery, freeze, preparation and the page-origin query, and an import rewrites it in place; when both readings name existing files the stored origin refuses with `detail: ambiguous_source_path`. A genuine origin mismatch refuses with `detail: sync_origin_mismatch` and the filled paths. Deletions must be absent from the pinned Git target or their explicitly selected working tree. Deleting a file that no page records as its origin (for example a skipped slug twin) while its slug's page keeps another origin commits an explicit no-op receipt (`unownedDeletion`, outcome reason `unowned_deleted_path`) that preserves that live page or tombstone; freeze and publication re-prove that no page owns the path and that the file is still absent. Existing cursors reuse their accepted manifests, with unknown old deletion provenance refused rather than rewritten. Active Git holds of the source are re-screened when their blob or working bytes changed, the file left, a `sources retry-held` request names them, the hold predates the current `RECOVERY_VERSION`, or the code is `frontmatter_slug_conflict`, `file_too_large`, `rename_held` or `parser_regression`; holds of now-excluded paths are released at the checkpoint. #6278 (B7): `retryTaken` lists every scheduled path the discovery read (screened as an entry, as a candidate, or no longer eligible); `sync-run.ts` clears them in the transaction that saves the fresh cursor. A held rename destination (or a held path renamed again) carries the hold's `renameFrom` when the old page is unchanged, else `renameHeld` (held as `rename_held` / `rename_source_changed`), and a full walk never deletes an origin a hold's `rename_from` names. Imports whose origin has no page claim their slug together: a live page whose own file is still in the tree keeps the slug, otherwise the file named like the slug (else the first path) wins, and the losers leave the manifest and are reported as `slugCollisions` (the library path's `slug_collision` skip) instead of failing the run. A winner whose slug is held by a page with a vanished or deleted origin takes that page over (`renameFrom`, same slug). A Git-detected rename to an unoccupied slug becomes one import carrying `renameFrom` in place of a delete plus a create. Origins for one slug that differ only by case or separator spelling refuse, except a case-only respelling whose old file the same sync removes. Code files keep exact per-origin identity. On Windows a path containing `:` is reported in `SyncDiscovery.fileRefusals` (code `colon_slug_windows_write_through`, with its filled `gbrain sync --source <id> --no-pull` hint) and kept out of the run and its origin checks, so one such file no longer fails the whole sync (#5032). `resolveManagedSyncContext` refuses `skipFailed` / `includeGitignored` on a managed source with `managed_sync_flag_refused` (`managed-sync-flags.ts`, `detail` = the flag, fix = the same `gbrain sync` argv without it), the one refusal jobs, MCP and the cycle reach; `gbrain sync` raises it at parse, before the legacy ledger acknowledgement and the cost gate, `--all` included (#6696). #6865: the per-path eligibility (scope, `sync.exclude`, `sync.include_hidden`, strategy, reserved skillpack paths) and the origin a path maps to under the slug mode come from one exported `loadSyncEligibility(engine, context, opts?, company?)`, which the unindexed-file detector shares.- `src/core/hardened-git.ts` (#6712 additions; base entry in [persistence-stalls](persistence-stalls.md)) — `assertHardenedGitArgs` checks exact options in every wrapper: `cat-file` with at most `--batch` (never `--filters`, `--textconv` or `--batch-command`), `hash-object --no-filters` optionally with `--stdin-paths`, `check-attr`/`check-ignore` with `-z`/`--stdin`, and `diff-tree` only as `-r -z --no-renames`. `hardenedGitStream` is a third wrapper: it spawns async, streams stdout to a callback, feeds stdin with backpressure, SIGKILLs past its timeout or byte cap and rejects when the consumer or stdin iterable throws; `NulRecordSplitter` and `hardenedStreamComplete` give the completeness rule (exit 0, no signal, output ending at a NUL or empty). Tests: `test/hardened-git-stream-6712.test.ts`, `test/hardened-git-filters.test.ts`.- `src/core/persistence/sync-git-stream.ts` — the managed-sync reconcile's Git level reads (#6712), all through `hardenedGitStream` (allow-listed, filter-free plumbing; the checkout's clean/process filters and textconv never run) and parsed as they stream, so memory is the result maps. `readTreeLevel` (`ls-tree -r -z <commit> -- <scope>`: blobs plus symlinks and gitlinks with their mode), `readIndexLevel` (`ls-files -s -z`, the index only; a conflicted path keeps its highest stage) and `readRenameHints` (`diff-tree -r -z --no-renames`: a deleted and an added regular file paired only when their blob OID is unique to them; hints, never candidates) throw `SyncTreeReadIncomplete` (code `reconcile_read_incomplete`) unless Git exited 0, was not signalled and its output ended at a NUL. `readBlobs` reads contents through one `cat-file --batch`, leaving out missing, oversized and over-total objects. `probeDirtiness` is the batched `hardenedPathDirty` (`Dp`): per batch of 2,000 paths one `hash-object --no-filters --stdin-paths` and one `check-attr -z --stdin`, presence and bytes from `readSyncFile`; verdicts in order `untracked`, `staged` (index and tree differ in OID or mode, a non-zero stage, or one side missing), `deleted`, `unknown` (symlink or gitlink, unreadable file, a path with a newline or a leading quote, any Git failure, an OID that disagrees with the bytes read), then `clean` (raw OID equals the tree with the executable bit matching), `clean_eol` (only `text`/`eol` set and the CRLF-folded bytes hash to the tree), `clean_lfs` (a valid pointer whose sha256 and size match the bytes) or `modified`; another filter, `working-tree-encoding`, an executable-bit-only change or CRLF bytes that fold to the tree with no attribute are `unknown`, never clean. Tests: `test/hardened-git-stream-6712.test.ts` (every verdict, completeness, and a forced probe that 1,000 passes of every read never run a configured filter or textconv while `git status`/`git diff` on the same checkout do). Working mode: `probeDirtiness(..., { working: true })` never reads the index (a path with no file is `deleted`, a present file outside the tree `untracked` with its raw OID, every other present file judged from its bytes against the tree through the same `clean_eol`/`clean_lfs` domains), so a staged edit, a staged new file, `git rm --cached`, a staged deletion and a staged file removed from disk probe as the same disk does with a pristine index: the working level is the physical bytes (design v3.2 §2.1). `hashWorkingLevel` hashes every path of a working round that way with hash-then-restat (a path whose bigint `lstat` moved across the hash is rehashed, up to three attempts) and returns the stat cache it leaves: `WorkingStatCache` is plain JSON (`{v: 1, root, entries: {path: 'dev:ino:mode:uid:size:mtime_ns:ctime_ns' | 'absent'}}`), an entry is written only for a stable stat whose `mtime` and `ctime` are older than 2 s before the attempt began (Git's racy rule) and a probe with a level, and at most `WORKING_STAT_CACHE_MAX_ENTRIES` (100,000) entries are kept in path order. `workingChanged(cache, paths)` is the idle tick's check: a path the cache does not hold, a moved stat, a failed stat, a zero `ctime`, a missing cache or a moved root all read as changed, never as unchanged. `parseWorkingStatCache` validates a serialized copy (junk is null, so everything reads changed); `rememberWorkingStatCache`/`recallWorkingStatCache` keep up to 64 caches per process. Working-mode tests: `test/sync-working-level-6712.test.ts` (probe 41, on the Git reads and on a managed PGLite pass), `test/sync-working-stat-cache-6712.test.ts` (probe 42); the probe 28 marker test also runs `hashWorkingLevel`.- `src/core/persistence/sync-classify.ts` — the managed-sync level classifier (#6712 PR 4, design v3.2 §2.3): pure (no I/O), total, ordered, first match over one path's facts (`SyncPathState`: mode, `T[p]`, the level `L`, presence, `Dp`, the page and its import record, the attestation of its revision, hold, deletion marker, baseline blob, open work). Rows in order: `held`, `spelling`, `untracked`, `kept` (and an absent db_only page), `delete`, `delete_drift`, `gone`, `deleted_same`, `deleted_edited`, `legacy_same`, `dirty`, `matched`, `adopt`, `reparse`, `db_ahead` (sub-labels `pending`/`db_only`/`unproven`; keep_db writes only `db_only` on a clean committed path; an open full obligation skips `db_only`/`unproven`), `import`, `legacy_unverified`, `keep_db`, `drift` (side `both` or `unknown`); then the overlay (unsettled with open work is `in_flight`). `syncClassAction` is the glossary tag (`none`, `owner`, `repair`); `syncPassKind` decides `idle`, `hold_only` or `full` from the converged state and every obligation. Tests: `test/sync-classify-6712.test.ts` (each row, and a cross product against a reference model).@@ -25,7 +25,7 @@ See [generations](generations.md).- `src/core/persistence/sync-convergence.ts` — the convergence receipt (#6712 PR 4a, schema `convergence/1`, always `dry_run: true`): per-class `counts` and `classes` with action tags (`in_flight`, `keep_db_pending` beyond 200 per pass, and `deletes_withheld` when #2828's valve trips are classes too), predicted `decisions` by side after `readOnlySameContentScreen` (the dry run's `sameContentAtOrigin`: a screening request through `prepareManagedSyncMutation(..., { unsaved: true })`, at most 200 paths; one call's preparations share group-publish.ts's `preparationReads` memo plus the binding, trust-config and purge-tombstone reads, read their pages through `importGroupReads`' batch (`reread`), and answer each page's origin, timeline, take-collision and safe-chunk-fence reads from one read per kind per call (`sliceReads`), so a call costs a constant, not a statement per page (GBRA-108); equal content is `same_content_noops`, unscreened is `screen_unknown`), `would_hold`, `db_ahead` by prefix, `legacy_unverified`, `legacy_pageless`, `deletes` with the valve verdict and set hash, up to 20 samples per non-matched class, timings and statement count. `--redact-paths` replaces paths with sha256 prefixes and drops slugs.- `src/core/persistence/sync-dry-run-writer.ts` — #7167: `readDryRunWriterState(engine, sourceId)`, the writer state a managed dry run reports (`SyncResult.writer`: open queued/running/recovering requests of the source, the oldest one's age, and `gbrain sources writer status` as the next step) instead of waiting on the writer. A CLI dry run runs once outside the drain (`commands/sync/perform.ts`), so the drain's no-progress governor never abandons a long read as a writer stall, and a dry run that is not delegated to a serve says it previews in this process (`commands/sync-delegate.ts`). Test: `test/sync-dry-run-drain-7167.test.ts`.- `src/core/persistence/sync-pass-run.ts` — the live level-triggered pass of a managed, non-company source (#6712 PR 4b, design v3.2 §2.2, §3.1–§3.5, §8, §9), run under the source's drain lease. `runSyncPass` retires an unfinished legacy cursor first (its options seed the episode), pins the pass row (`sync.admission`, `HEAD`, the mode; the row is written with the lease token), decides the kind (`idle` stamps only `last_sync_at` as `up_to_date`; `hold_only`; `full`), then runs at most `PASS_MAX_ROUNDS` (4) rounds: read and classify the level (`sync-pass.ts`), admit `C_k = unsettled − S` in batches of at most `PASS_BATCH_MAX` (10,000; deletes, then imports) through sync-run.ts's cursor machinery on an in-memory batch, and add each settled `(path, level)` to `S` (never admitted twice in one pass). With nothing unsettled and nothing in flight it stamps through `applyPassCheckpoint`; four rounds without converging stop `resumable` / `not_converged`. Every stop records its outcome, stop reason, `settled` and `remaining` on the pass row (a stop mid-batch counts the entries its batch passed); a blocked page refusal is replayed with no new request while nothing that decided it changed. #7149: a requested re-screen (the `sync-hold-retry` row) is consumed as soon as the pass settles its path (the batch carrying it drained, or the round classified it settled), not only by the stamp, so a pass that stops before its stamp leaves no imported file reported as waiting. Tests: `test/sync-pass-live-6712.test.ts`, `test/unindexed-files-convergence-7149.test.ts`, `test/sync-pass-completion-6712.test.ts`, `test/sync-pass-outcomes-6712.test.ts`.-- `src/core/persistence/sync-pass-row.ts` — the pass row and the rows beside it (#6712 PR 4b, design §3.4), free of heavy imports so doctor, status, the lease, compaction and GC read it without loading the pass. `managed-sync-pass` (fingerprint `<source>:<incarnation>`) holds `obligations` (baseline, episode, cost, the converged state, mismatch and not-converged streaks, page faults, a blocked request to replay) and `telemetry` (the newest pass's kind, round, outcome, stop reason, candidates, settled, remaining, counts, timings). `writePassRow` writes it at pin under `assertDrainLease` and afterwards only `WHERE lease_token = $token`; an `onlyOpen` stop never inserts a gone row. Readers age `updated_at` on the read's database clock (`passReadClock`). Beside it: `sync-origin-deleted` (deletion markers), `sync-deletes-withheld` (the mass-delete trip) and `sync-pass-cost` (one reservation per group); `SYNC_PASS_ROW_OPS` lists the ops the checkpoint GC keeps for a live incarnation. `passHealth` is the one health rule (`blocked`, `not_converged`, `settle_mismatch`, `open_stalled`, `nothing_pending`); `SETTLE_MISMATCH_STREAK` and `NOT_CONVERGED_REPORT_STREAK` are 3.+- `src/core/persistence/sync-pass-row.ts` — the pass row and the rows beside it (#6712 PR 4b, design §3.4), free of heavy imports so doctor, status, the lease, compaction and GC read it without loading the pass. `managed-sync-pass` (fingerprint `<source>:<incarnation>`) holds `obligations` (baseline, episode, cost, the converged state, mismatch and not-converged streaks, page faults, a blocked request to replay) and `telemetry` (the newest pass's kind, round, outcome, stop reason, candidates, settled, remaining, counts, timings). `writePassRow` writes it at pin under `assertDrainLease` and afterwards only `WHERE lease_token = $token`; an `onlyOpen` stop never inserts. Readers use the database clock (`passReadClock`). Beside it: `sync-origin-deleted` (deletion markers), `sync-deletes-withheld` (the mass-delete trip) and `sync-pass-cost` (one reservation per group); `SYNC_PASS_ROW_OPS` lists the ops the checkpoint GC keeps for a live incarnation. `passHealth` is the one health rule (`blocked`, `not_converged`, `settle_mismatch`, `open_stalled`, `nothing_pending`); `SETTLE_MISMATCH_STREAK` and `NOT_CONVERGED_REPORT_STREAK` are 3.- `src/core/persistence/sync-pass-readers.ts` — the pass row as the former cursor readers see it (#6712 PR 4b, design §3.7): doctor's movement, git-holds and terminal-failure checks, the drain backlog (`readManagedSyncBacklog`), the blocked-run reader and `sources status` read a non-company source's pass row through `readPassReadings` (`passHealth` plus the checkout HEAD from a hardened `rev-parse` on this host; no checkout means `head: null` and the recorded converged target is trusted). `passResumeArgs` is the plain sync that continues the pass, with the open episode's inherited options; `passStopCode` and `passStuckSince` name a stuck pass. Company sources and unretired legacy cursors keep their cursor arm in each reader. Tests: `test/sync-pass-readers-6712.test.ts`, `test/sync-pass-reader-gaps-6712.test.ts`.- `src/core/persistence/sync-pass-apply.ts` — what a live pass round does besides admitting one-sided imports and deletes (#6712 PR 4b, design v3.2 §2.5–§2.7, §3.1, §5): holds with side and state for the two-sided and deleted rows (`file_drift` with `side`, `tombstone_edited`, `purged_differs`, `file_gone`), the `settle_mismatch` hold after three passes at one level, arm-1 adoption of a record without a revision on `fileSame`, the bounded legacy sweep (500 per round) that records equal legacy pages on the first full round (GBRA-108: in slices of `LEGACY_SWEEP_SLICE` (100) whose records commit before the next slice, each followed by a pass-row save and a `managed_sync.legacy_swept` progress event the drain's in-pass governor counts, so a pass stopped mid-sweep keeps its slices and the next pass sweeps only the rest; seam `sync:legacy_slice`; tests `test/sync-legacy-sweep-gbra108.test.ts` and its Postgres twin), the mass-delete valve (one `sync-deletes-withheld` record; imports and the stamp proceed), `keep_db` admission (200 per pass), deletion-marker retirement in full rounds, and the remote ending (`reconcile_host_required`: one-sided work commits, drift stays unsettled with no hold). Entry points: `applyPassRound`, `finishPassApply`, `summarizeRound`. Tests: `test/sync-pass-completion-6712.test.ts`.- `src/core/persistence/sync-pass-episode.ts` — pass episodes and cost (#6712 PR 4b, design §6, §7): `episodeClosure` closes an episode by inclusion or at the configured selection (a narrower episode never stamps `chunker_version`; a widened one releases its extra paths), `passCostDecision` defers paid embedding when the envelope is unknown or spent, and `reservePassCost` writes one `sync-pass-cost` row per admitted group (fingerprint `<source>:<incarnation>:<episode_or_pass_id>:<group_id>`, insert-once) inside the group's admission transaction, so a kill keeps the reservation and a replay adds nothing. Tests: `test/sync-pass-completion-6712.test.ts`.