Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
21.3 KB
Raw Blame

Key files: Core Persistence (Git)

Subsystem index · Core Persistence · continued. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG. The managed Git writer (gbrain sources refresh), git-canonical memory (the git gap census, declared database-only classes, the door) and the checkout reads beside them.

  • src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts — gbrain sources refresh (F0), the one sanctioned managed Git writer. persistence_worktree_refreshes holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to refresh_in_progress). refreshWorktree prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened git fetch bounded by sources.refresh_fetch_timeout_ms, ancestry, dirty paths intersected with the incoming diff), then: draining (inserted under withTopologyLocks + lockTopologyRows; admission in journal.ts refuses worktree_refreshing for every request with that worktree_id, and assertManagedSyncAllowed refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running git/withdrawal-mirror effect and no recovery record remain (else refresh_drain_timeout, aborted), fenced (both native locks held, rechecked in one transaction; refreshFenceClear stops request and effect claims), HEAD verified (#8617: a refresh_source_changed for a moved HEAD or work that arrived after the drain check is retried once in the same run from a fresh precheck; a second refusal returns the first with the retry's cause appended), dirt rechecked, git merge --ff-only, HEAD verified again (a HEAD that descends from the target after a successful merge, with no member synced off it, counts as merged: a plain-git commit landed in between; a refused fast-forward whose HEAD only gained commits on the start commit aborts refresh_source_changed, GST-1674), then one transaction merged + upstream observation + syncing; native locks are released and the managed --no-pull sync runs per member in one budget (sources.refresh_sync_budget_ms; past it a member is deferred) (only managed_sync_* submit_job admissions pass during syncing); completed when every member's last_commit is the target or descends from it, or with outcome.sync_blocked / sync_deferred when a member blocks or is deferred. A per-worktree refresh-<worktree>.lock held for the whole run proves liveness: resumeWorktreeRefreshes (called from recoverSourceTopologies on every consumer topology scan) touches interrupted draining/fenced/merged rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; so does a fenced row whose merge started (outcome.merge_started_at, epoch ms) and whose HEAD descends from the old HEAD without the target, when the reflogs of HEAD and its branch show no move to the target since then (git refused the fast-forward, GST-1674; no readable reflog stays recovery_required); HEAD at the verified target moves to syncing; anything else is recovery_required, which admission refuses with refresh_recovery_required). Admission also completes a syncing row whose members all reached the target. #7679: each row records its holder (outcome.holder: host id, hostname, pid, process start; previous_holder after a recovery adopts it). A row whose lock is free is orphaned: resumeWorktreeRefreshes also completes an orphaned syncing row once every member's last_commit reached the target or a descendant, or (#9649) once the row has not moved for one refresh sync budget (sources.refresh_sync_budget_ms) with the checkout still at the target or a descendant, recording completed_by: owner_reclaim and the lagging members in sync_deferred as a run whose budget ran out does (#8072), and a plain gbrain sources refresh on the owner host finishes any orphaned non-recovery_required row through the --resume path (adoptOrphanedRefresh + resumeHeld) before its own refresh, reporting it as recovered. A holder recorded on another host is never adopted. refreshHolderReport (writer status worktree_refreshes, sync status worktree_refresh) names the holder, liveness (live / orphaned / unknown off the lock's host) and a fix naming --resume. While the worktree's newest refresh is aborted, every run (tick, --resume, --abandon, even one its precheck refuses) ends by running the managed --no-pull sync of each member behind the checkout HEAD (syncAfterUnfinishedRefresh, listed in synced), so a write held for an import that refresh fenced off is decided in that run, not at a later refresh. --resume converges and runs the member syncs; --abandon lifts a fence only with HEAD back at the start commit or on a row recovery would abort (a HEAD moved before the merge, or a refused merge, GST-1674), and never on a recovery_required row once a member is past it (membersPastStart: a last_commit that is not it or an ancestor, GST-1457: the next sync would soft-delete what it synced; or a managed-sync import or delete admitted after the row's barrier_sequence, GST-1465: a pass stopped before its stamp), checking and aborting under every member's drain lease (underMemberLeases, so a pass in flight refuses sync_drain_lease_held), and --resume finishes a recovery_required row only at a HEAD that contains every member commit on that refresh's line (adoptsHead, which --abandon also reads to name --resume instead of a reset). keepPagesWayOut names the way out (GST-1472): a fast-forward to the newest member commit on the target's line, a merge of member commits on diverged lines (ask the user), or a fetch when git cannot place a member commit (ancestry tells exit 1 from a failure); the refusals also name the revert route for a deliberate rollback (GST-1477), and off the owner host writer status names no step. gbrain sources writer status lists active refreshes as worktree_refreshes (state, heads, the next command). Doctor worktree_refresh_stuck lists rows active over 15 minutes. CLI body: src/commands/sources-refresh.ts (#7685: --json prints a refusal's structured fix when it has one; refresh_in_progress carries the writer-status read); while a resident PGLite owner (gbrain serve, stdio MCP) holds the database, src/commands/sources-refresh-delegate.ts runs the refresh inside it through the writer_refresh administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies refreshFenceClear to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: test/persistence-worktree-refresh.test.ts, test/persistence-worktree-refresh-restart.test.ts (child-process crash after merge, and after a refused merge), test/worktree-refresh-orphan-7679.test.ts (SIGKILL mid-syncing, live-holder no-steal), test/oversize-hold-rescreen-8072.test.ts (budget), test/refresh-barrier-held-after-abort.test.ts (the sync after an aborted refresh), Postgres via test/e2e/worktree-refresh-*postgres.test.ts, owner delegation in test/persistence-sync-stdio-owner.test.ts.
  • src/core/persistence/refresh-barrier.ts (+ deferred admission and claim in journal.ts, migration v292 src/core/schema-migrations/v292-refresh-write-barrier.ts) — the refresh write barrier (docs/designs/REFRESH_WRITE_BARRIER.md, docs/guides/write-refusals.md#deferred-writes). Off by default (persistence.refresh_barrier). With it on, a single page or memory write that reaches a worktree during a refresh is journaled in state deferred (tied to deferred_refresh_id) and answered pending instead of refused worktree_refreshing. Only a claim that declares the barrier moves a deferred row, and only after its refresh is released. deferredReplayRefusal re-checks the write's authority and reads the refresh's old_head..target_head diff with git, so a file the refresh changed ends revision_conflict with nothing applied. A write still waiting past persistence.refresh_barrier_max_wait_ms (15 minutes) ends refresh_barrier_expired. A write during a refresh in recovery_required defers too, and since GST-1582 there is no per-refresh cap (persistence.refresh_barrier_max_deferred is no longer read). A write whose page file holds a committed change sync has not imported yet is held with deferred_import_from (migration v294) instead of refused pending_import, and ends pending_import_expired past the wait limit (or four wait limits, at least four hours, after admission when it is held again and again). GST-1623: either kind of deferred write ends revision_conflict when a later write to its page committed first, and a refresh-deferred write is checked against its refresh before an import hold, which then clears deferred_refresh_id. refresh-retry-hint.ts derives worktree_refreshing's retry_after_ms from the worktree's refresh history. gbrain upgrade --check-downgrade exits 3 downgrade_blocked_deferred_writes while any deferred write is unfinished. Tests: test/refresh-write-barrier.test.ts, test/refresh-barrier-state-audit.test.ts.
  • src/core/persistence/git-gap.ts + src/commands/sources-git-gap.ts + src/commands/doctor/checks/db-ahead-of-git.ts — git-canonical memory, wave G1 (docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census), read-only. declaredDatabaseOnly is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (isPrivatePage), an alias-private page (isDerivationPrivate), lineage-private (computed in SQL from privatePagesFilterFragment), database_only_reason, a storage.db_only prefix (isSourceDbOnlySlug, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts isNeverFiledDerivedPage: no source_path/source_uri under DERIVE_PHASE_DB_ONLY_DEFAULTS), a path the checkout's Git ignores (ignoredPaths: filter-free check-ignore -z --stdin, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (readExportPage, withdrawals overlaid; the census reads a batch's views with readExportPages, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by contentCanonical digest (reconcileCanonical without the write gate's database-only markers, so a flagged page is not reported behind its own file, #9511); a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (readHeadTree: one ls-tree -r -l -z --full-tree under the source's directory; readBlobs: cat-file --batch in calls of at most 64 MiB) and the working tree by hashing raw bytes (worktreeBlobOid), never git status/git diff, so no repository filter, textconv or fsmonitor runs. gbrain sources git-gap <id> [--sample n] [--json] (trusted local CLI, not an MCP op) puts every live page in one state (equal, database_ahead, git_ahead when the sync import record's blob_oid/disk_blob_oid differs from HEAD, else folded into database_ahead and counted in database_ahead_without_import_record, split by content evidence in direction_evidence, declared_database_only per class, undeclared_database_only, worktree_only, unparseable, unsafe_path), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose link_source is not in DERIVED_LINK_SOURCES, timeline rows on pages without a file), open sync-hold records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor db_ahead_of_git runs the census per non-archived source whose directory is a Git checkout (others are only counted) under GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS (default 10 s) and the entry deadline; the census reads the DOCTOR_GIT_GAP_NEWEST (800) most recently changed live pages per source (censusGitGap option newest, keyed by updated_at; the census records newest and newest_since, the oldest updated_at it read) and the message states that window (checked the <n> most recently changed of <live> live page(s), back to <time>). The gap is informational (never a warning); a source the budget stopped or never reached (PARTIAL, <read> of <live> live page(s) read) or one that could not be compared makes the check the unknown convention (warn, health: unknown), never ok (GBRA-541). It lists totals only with each source's live_pages (counted in SQL before the scan). Tests: test/git-gap-census.test.ts (PGLite and Postgres).
  • src/core/persistence/git-declared.ts + src/core/persistence/git-canonical-schema.ts (migration git_canonical_declared) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (sources.config.canonical = "git", default off). strictPageClass judges a page by its slug and prior file (declaredPathClass: db_only_prefix for a storage.db_only dir, derive_folder for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (privacyClass: private, alias_private, or lineage_private from the same SQL remote reads use, privatePagesFilterFragment, over the page's own values). page-prepare.ts prepareFileTarget asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is git_ignored, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose filter attribute is lfs passes assertLfsTarget only with storage.git_lfs and a checkout that configures the LFS filter (git_lfs_not_enabled). preparePrivateTransition (page-prepare.ts) is that removal for other writers (GBRA-60, connector imports #7971). In publishMutation and publishGroup, assertNoPrivateCommit refuses a file target carrying a private page's bytes before anything is written (git_private_commit_refused; a group rolls back whole), and declareStrictPublication (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (git_home_required), stamps pages.database_only_reason and, for a privacy class, clears source_path/source_uri in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. assertStrictGitHome is the refusal for writers with no file form yet (G4, G5). The managed writer guard (writer-guard-schema.ts) refuses rows of a strict source unless the writer declares gbrain.git_protocol (withCoordinatedWrite does), behind the git_canonical_brain.strict_sources counter kept by sources triggers (git_protocol_required, mapped in publication-failure.ts). The guard reads its state as the writing role and fails closed: a hidden or missing persistence_brain or git_canonical_brain row, or a source or page row hidden by row-level security, raises writer_coordinator_required (managed_writer_guard:unreadable_state), and the counter refuses a recount under RLS on sources (test/writer-fence-fail-closed-7636.test.ts). The census (git-gap.ts) adds database_only_source (a source with no checkout) and maps the stamped reasons to their own classes. Tests: test/git-canonical-declared-g7a.test.ts (both engines).
  • src/core/persistence/checkout-state.ts (readCheckoutState) — Q1: a canonical checkout against its last fetched upstream, read-only and time-bounded (no fetch): ahead/behind, diverged, each commit the upstream lacks (sha, author, date, subject, paths; at most 20) and porcelain counts. sources writer status reports it as checkouts; doctor git_convergence adds the commits and counts for a managed checkout and warns at once on a diverged one.
  • src/core/persistence/git-door.ts: G2, the door of git-canonical memory. A source with sources.config.canonical = "git" (strict, default off; set by gbrain sources canonical <id> --to git --apply --expect <hash>, git-conversion.ts) commits each published file before its database row. publishMutation (coordinator.ts) and publishGroup (group-publish.ts, one commit per group, after its lane turn) call commitThroughDoor under the writer lock, between the file rename and the transaction commit. It commits by plumbing without the checkout's index: hash-object -w per file, the trees along the changed paths rebuilt from HEAD (cat-file tree, hash-object -w -t tree), commit-tree with commitGitTargets' message, update-ref compare-and-swap (one rebuild if the branch moved, then a refusal), then update-index --cacheinfo for exactly the published paths (an entry someone staged during the write is left as it is, and the write says its commit landed). That refresh runs under Git's own index lock: syncIndex creates <index>.lock exclusively (an existing lock the #6932 proof finds abandoned is reclaimed under the caller's writer lock through reclaimAbandonedIndexLock in effect-git.ts and taken again; any other existing lock refuses with the landed failure, which says why it was not reclaimed), compares the entries, updates a hard-linked copy of the index through GIT_INDEX_FILE, and renames the copy onto the lock and the lock onto the index, so no writer that takes the lock can stage between the compare and the write; it also sweeps dead processes' index copies (#7755), retries win32 renames (#7756); detached HEAD and unborn branches refuse typed. Each DoorFile carries its planned state (planned: the sha256 of the bytes the publication wrote, or null for a removal), and the door commits exactly that state or nothing: the file must hash to the plan before and after hash-object -w (else git_target_changed: nothing is committed and the edit stays on disk; the write is held in recovery as unexpected_file_bytes when the file holds neither its bytes nor the version before it, and fails with nothing held when the file holds the version before it), and each target's index entry must equal HEAD's (else git_index_conflict, index untouched). An entry that still holds the version its path's last commit replaced (an interrupted door refresh) counts as clean: the write restores it to HEAD and reports the paths as outcome.persistence.git_index_restored and the git_index_restored notice (indexRestoredNotice, emitted by page-mutations.ts emitFenceNotice). The step deadline is sized from one stat of the index (gitStepTimeoutMs), and refuseIgnored asks check-ignore only about new targets ignore-rules.ts cannot clear. Door commits are not coalesced across single writes (each holds its worktree lock exclusively and the consumer runs one per root); a batch commits once through publishGroup. The publication stamps the commit as outcome.persistence.git_commit (git_state: committed); a target Git refuses rolls the publication back. The Git effect carries push_only and joins the push without staging (effects.ts). The recovery record carries door (checkout and starting HEAD); recoverPublication keeps a file whose commit landed (doorCommitLanded) instead of restoring it, and the replay completes against it: replayAfterDoorCommit in publishMutation and the strict exceptions in page-prepare.ts (own bytes committed, or a removal HEAD already holds). Old binaries ignore door; strict stays off until conversion fences them. Tests: test/git-door-strict-gbra133.test.ts (both engines), test/git-door-plumbing-gbra133.test.ts, test/git-door-planned-state.test.ts, test/git-door-write-cost.test.ts, test/git-door-index-lock-7742.test.ts (both engines); per-write overhead bench scripts/bench/door-write-overhead.ts (opt-in, smoke test/bench-door-write-overhead.test.ts).
  • src/core/persistence/ignore-rules.ts — the ignore filter in front of git check-ignore; Git stays the only judge of an ignored path. ignoreCandidates(root, paths) returns every path some positive ignore rule could match, read from every .gitignore from the top level down to the path, info/exclude and core.excludesFile; a path it drops is one no rule could match. It over-approximates: negated rules are dropped, every wildcard matches any run of characters including /, directory-only rules match files, letters compare case-insensitively, each rule is tested against the path and every directory above it in its own and NFC spelling, and an unreadable or non-UTF-8 ignore file or a rule with a backslash or a character outside printable ASCII matches everything under its directory. Every path goes to Git on a sparse checkout or skip-worktree entry, a split or sparse index, an unresolvable core.excludesFile or include, a gitlink or symlink along the path, or a failed probe. The probe (rev-parse layout and the configuration with its origin files) is cached per checkout and taken again when a configuration file it read or could read, HEAD or the configuration environment changes; ignore files are re-read on every call and parsed once per content. Callers: git-door.ts refuseIgnored and sync-origin-evidence.ts gitIgnoresUntracked. Tests: test/ignore-rules.test.ts.