Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

docs: git-gap live_pages, PARTIAL and direction_evidence in the reference rows (ship docs audit)

garrytan09cf77a8b196
diff --git a/docs/architecture/key-files/core-persistence-git.md b/docs/architecture/key-files/core-persistence-git.md
index 83bcbc3c13..e3b7ddecdd 100644
--- a/docs/architecture/key-files/core-persistence-git.md
+++ b/docs/architecture/key-files/core-persistence-git.md
@@ -5,7 +5,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
The managed Git writer (`gbrain sources refresh`), git-canonical memory (the git gap census, declared database-only classes, the door) and the checkout reads beside them.
- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member in one budget (`sources.refresh_sync_budget_ms`; past it a member is `deferred`) (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` / `sync_deferred` when a member blocks or is deferred. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit, and never on a `recovery_required` row once a member's `last_commit` is past it (not it or an ancestor; GST-1457: the next sync would soft-delete what it synced), and `--resume` finishes a `recovery_required` row only at a HEAD that contains every member commit on that refresh's line. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command). Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured `fix` when it has one; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), `test/oversize-hold-rescreen-8072.test.ts` (budget), Postgres via `test/e2e/worktree-refresh-*postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.
-- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).
+- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, split by content evidence in `direction_evidence`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only with each source's `live_pages` (counted in SQL before the scan), a source the budget stopped or never reached as partial (`PARTIAL, <read> of <live> live page(s) read`). Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).
- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60, connector imports #7971). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The guard reads its state as the writing role and fails closed: a hidden or missing `persistence_brain` or `git_canonical_brain` row, or a source or page row hidden by row-level security, raises `writer_coordinator_required` (`managed_writer_guard:unreadable_state`), and the counter refuses a recount under RLS on `sources` (`test/writer-fence-fail-closed-7636.test.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).
- `src/core/persistence/checkout-state.ts` (`readCheckoutState`) — Q1: a canonical checkout against its last fetched upstream, read-only and time-bounded (no fetch): ahead/behind, `diverged`, each commit the upstream lacks (sha, author, date, subject, paths; at most 20) and porcelain counts. `sources writer status` reports it as `checkouts`; doctor `git_convergence` adds the commits and counts for a managed checkout and warns at once on a diverged one.
- `src/core/persistence/git-door.ts`: G2, the door of git-canonical memory (Git is the source of truth, the database derives from it). A source with `sources.config.canonical = "git"` (strict, default off; set by `gbrain sources canonical <id> --to git --apply --expect <hash>`, `git-conversion.ts`) commits each published file before its database row. `publishMutation` (`coordinator.ts`) and `publishGroup` (`group-publish.ts`, one commit per group, after its lane turn) call `commitThroughDoor` under the writer lock, between the file rename and the transaction commit. It commits by plumbing without the checkout's index: `hash-object -w` per file, the trees along the changed paths rebuilt from HEAD (`cat-file tree`, `hash-object -w -t tree`), `commit-tree` with `commitGitTargets`' message, `update-ref` compare-and-swap (one rebuild if the branch moved, then a refusal), then `update-index --cacheinfo` for exactly the published paths (an entry someone staged during the write is left as it is, and the write says its commit landed). That refresh runs under Git's own index lock: `syncIndex` creates `<index>.lock` exclusively (an existing lock the #6932 proof finds abandoned is reclaimed under the caller's writer lock through `reclaimAbandonedIndexLock` in `effect-git.ts` and taken again; any other existing lock refuses with the landed failure, which says why it was not reclaimed), compares the entries, updates a hard-linked copy of the index through `GIT_INDEX_FILE`, and renames the copy onto the lock and the lock onto the index, so no writer that takes the lock can stage between the compare and the write; detached HEAD and unborn branches refuse typed. Each `DoorFile` carries its planned state (`planned`: the sha256 of the bytes the publication wrote, or null for a removal), and the door commits exactly that state or nothing: the file must hash to the plan before and after `hash-object -w` (else `git_target_changed`: nothing is committed and the edit stays on disk; the write is held in recovery as `unexpected_file_bytes` when the file holds neither its bytes nor the version before it, and fails with nothing held when the file holds the version before it), and each target's index entry must equal HEAD's (else `git_index_conflict`, index untouched). An entry that still holds the version its path's last commit replaced (an interrupted door refresh) counts as clean: the write restores it to HEAD and reports the paths as `outcome.persistence.git_index_restored` and the `git_index_restored` notice (`indexRestoredNotice`, emitted by `page-mutations.ts` `emitFenceNotice`). The step deadline is sized from one stat of the index (`gitStepTimeoutMs`), and `refuseIgnored` asks `check-ignore` only about new targets `ignore-rules.ts` cannot clear. Door commits are not coalesced across single writes (each holds its worktree lock exclusively and the consumer runs one per root); a batch commits once through publishGroup. The publication stamps the commit as `outcome.persistence.git_commit` (`git_state: committed`); a target Git refuses rolls the publication back. The Git effect carries `push_only` and joins the push without staging (`effects.ts`). The recovery record carries `door` (checkout and starting HEAD); `recoverPublication` keeps a file whose commit landed (`doorCommitLanded`) instead of restoring it, and the replay completes against it: `replayAfterDoorCommit` in `publishMutation` and the strict exceptions in `page-prepare.ts` (own bytes committed, or a removal HEAD already holds). Old binaries ignore `door`; strict stays off until conversion fences them. Tests: `test/git-door-strict-gbra133.test.ts` (both engines), `test/git-door-plumbing-gbra133.test.ts`, `test/git-door-planned-state.test.ts`, `test/git-door-write-cost.test.ts`, `test/git-door-index-lock-7742.test.ts` (both engines); per-write overhead bench `scripts/bench/door-write-overhead.ts` (opt-in, smoke `test/bench-door-write-overhead.test.ts`).
diff --git a/docs/guides/git-canonical.md b/docs/guides/git-canonical.md
index 271955e3a6..9cc63c7b77 100644
--- a/docs/guides/git-canonical.md
+++ b/docs/guides/git-canonical.md
@@ -236,8 +236,8 @@ next write and sync pass.
| Where | What it tells you |
|---|---|
| `gbrain doctor` → `git_index_lag` | Per strict source, how many commits and seconds the index trails HEAD. It warns once the oldest unindexed commit is older than 900 s (`GBRAIN_DOCTOR_GIT_INDEX_LAG_WARN_SECONDS`), and its fix is `gbrain sync --source <id> --no-pull`. Until then, search and reads show the older version. |
-| `gbrain doctor` → `db_ahead_of_git` | The git-gap totals for every source with a checkout, strict or not. Informational: always ok. Bounded by `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (10 s by default). |
-| `gbrain sources git-gap <id> [--sample n] [--json]` | Each page's state against HEAD: `equal`, `database_ahead`, `git_ahead`, declared or undeclared database-only, `worktree_only`, `unparseable`, `unsafe_path`, with a sample of paths. Read-only. |
+| `gbrain doctor` → `db_ahead_of_git` | The git-gap totals for every source with a checkout, strict or not, with each source's live pages (a scan the budget stopped reads `PARTIAL, <read> of <live> live page(s) read`). Informational: always ok. Bounded by `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (10 s by default). |
+| `gbrain sources git-gap <id> [--sample n] [--json]` | Each page's state against HEAD: `equal`, `database_ahead`, `git_ahead`, declared or undeclared database-only, `worktree_only`, `unparseable`, `unsafe_path`, with a sample of paths, the source's `live_pages`, and `direction_evidence` for differing pages with no import record. Read-only. |
| `gbrain sources reindex <id> --from-git` | A preview of every database version Git would replace, bound to HEAD by a hash; `--apply --expect <hash>` runs that pass. Use it to re-derive a strict source's index on purpose. |
| `gbrain sync status --source <id> --json` | The sync pass that indexes commits: state, reason and the last error. |
| A write receipt | `persistence.git_commit` and `git_state: committed`. A door commit that landed before its row, on a write that then ended without committing, keeps `landed_before_row: true`: Git holds those bytes and the indexer imports them. |