Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Describe the review fixes in the changelog and key-files docs
garrytan407f48a1505b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f14e5766b..166137a4e6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -33,12 +33,12 @@ identifiers and attribution are available in the pre-removal Git revision|---|---|---|| A query or search with eval capture on, whose capture runs after the response, racing a page purge | the capture could write the purged slug back into `eval_candidates.retrieved_slugs` | the insert keeps only slugs whose page still exists, checked under the same row lock the feedback writers take || A connector, GitHub or Google sync deletes a page (`deletePages`), or a transcript re-ingest deletes a stale part | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction |-| `gbrain sources remove`, `sources purge`, the expired-archive sweep, or a managed `remove`/`purge` | the source's retrieval logs, ratings, volunteer rows and captured eval slugs stayed | deleted in the statement that deletes the source, and swept again right after for a write that raced it |+| `gbrain sources remove`, `sources purge`, the expired-archive sweep, or a managed `remove`/`purge` | the source's retrieval logs, ratings, volunteer rows and captured eval slugs stayed | deleted in the statement that deletes the source, and swept again right after for a write that raced it, unless a source with the same id was created in between || `gbrain sources federate` or `sources restore` on a source with a remote-written `visibility: private` page and no readers | the warning named that page, though no-grant connections still cannot read it | the warning applies the read paths' privacy rules, unless `search.remote_private_pages` or `GBRAIN_REMOTE_PRIVATE_PAGES` relaxes them |-- `src/core/feedback/store.ts`: `insertEvalCandidate` is the one eval capture insert both engines run.+- `src/core/feedback/store.ts`: `insertEvalCandidate` is the one eval capture insert both engines run. A facts-arm result (`facts/<id>`) names a fact, not a page, so it stays while that fact exists.- `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. The transcript importer's stale-part delete now calls `deletePages`. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.-- `src/core/source-delete.ts`: `removedSourceScrubCtes` runs inside `deleteSourceRow` and the managed lifecycle's source delete, then once more ungated. `retrieval_weights` and `mention_gazetteer_entries` already cascaded from `sources`.+- `src/core/source-delete.ts`: `removedSourceScrubCtes` runs inside `deleteSourceRow` and the managed lifecycle's source delete, then once more after it. `deleteSourceRow` skips that second pass when a source with the same id exists again, so a re-created source keeps its rows. `retrieval_weights` and `mention_gazetteer_entries` already cascaded from `sources`.- `src/core/access/federation-exposure.ts`: the exposure read adds `privacyPartSql` unless `remotePrivacyRelaxed`.- Tests: `test/purge-scrub-followups-9068.test.ts` and `test/federation-exposure-private-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`). `test/eval-candidates.test.ts` now seeds the pages its round-trip captures, since a capture keeps only live slugs.diff --git a/docs/architecture/key-files/core-utilities-1.md b/docs/architecture/key-files/core-utilities-1.mdindex 0e3ad04ff9..a7aa5aee9b 100644--- a/docs/architecture/key-files/core-utilities-1.md+++ b/docs/architecture/key-files/core-utilities-1.md@@ -53,7 +53,7 @@ The `src/core/cycle.ts` entry is on its own page: [Core Cycle primitive](core-cy- `src/core/eval-capture-scrub.ts` — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.-- `src/core/eval-capture.ts` — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers (catches MCP + CLI + subagent tool-bridge from one site). Fire-and-forget; the insert (`insertEvalCandidate` in `feedback/store.ts`, both engines) keeps only slugs a page of the captured sources still holds, probed under the feedback writers' key-share lock, so a capture deferred past its response cannot write a purged slug back (#9068). Failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. Capture is off by default — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Contributors set `export GBRAIN_CONTRIBUTOR_MODE=1`. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.+- `src/core/eval-capture.ts` — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers (catches MCP + CLI + subagent tool-bridge from one site). Fire-and-forget; the insert (`insertEvalCandidate` in `feedback/store.ts`, both engines) keeps only slugs a page of the captured sources still holds, probed under the feedback writers' key-share lock, so a capture deferred past its response cannot write a purged slug back (#9068); a facts-arm result (`facts/<id>`) stays while that fact exists. Failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. Capture is off by default — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Contributors set `export GBRAIN_CONTRIBUTOR_MODE=1`. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.- `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` — Check 6 of `check-resolvable`. Parses `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON (error severity). Internal `parseFrontmatter` is a thin wrapper over the shared `src/core/skill-frontmatter.ts` parser so both filing-audit and skill-brain-first read the same shape (`tools?`, `triggers?`, `brain_first?: 'exempt'`, typed `brain_first_typo`) from one source of truth.diff --git a/docs/architecture/key-files/files-and-sync-1.md b/docs/architecture/key-files/files-and-sync-1.mdindex d08280740f..e862a0e11a 100644--- a/docs/architecture/key-files/files-and-sync-1.md+++ b/docs/architecture/key-files/files-and-sync-1.md@@ -51,7 +51,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/source-resolver.ts` — 6-tier source resolution. `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside `resolveSourceId()` (unchanged). `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'sole_non_default', 'brain_default', 'seed_default']` (7 entries; order matches priority). Tier `sole_non_default` slots between `local_path` and `brain_default`: when NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'` AND the `'default'` source holds no active pages (the emptiness guard — the tier's charter is rescuing brains whose `'default'` is empty; when `'default'` carries an established corpus, auto-routing would hijack every bare write into the side-source, so the resolver falls through to `seed_default` and prints a one-line stderr notice naming both sides so the suppressed flip is diagnosable, same `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1` suppression knob as the routing nudge; a failed pages probe on an exotic/legacy schema keeps the auto-route), auto-route to it; archived sources excluded (try/catch for pre-v34 brains); private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points so they cannot drift. Exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `resolveCliSyncSource` in `src/commands/sync/run.ts` calls `resolveSourceWithTier` unconditionally so the tier fires; `src/commands/import.ts:96-128` mirrors with the tier-gated nudge. Consumed by `gbrain sources current`, `import --source-id`, `extract --source-id`, and the `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (`withEnv()` per test-isolation lint), `test/source-resolver-sole-non-default.test.ts` (20 cases incl. both-ways emptiness-guard + probe-failure fallback), `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync`), `test/pages-source-scoping-4329.test.ts` (real-engine both-ways guard coverage). Also exports `noGrantFederatedScope(engine, hasSourceGrant, sourceId)`, the widening decision transports share: returns `localFederatedSourceIds(..., 'seed_default')` ONLY when `hasSourceGrant === false` (the legacy-bearer no-grant floor), `undefined` for a granted token, for an OAuth client (flag `undefined` — a falsy gate here would widen every OAuth client), and when the resolver throws (best-effort; the scalar scope stands rather than failing the request). Called by `src/commands/serve-http.ts`; pinned by `test/no-grant-federated-scope.test.ts` (6 cases). `explicitReadBinding(engine, sourceId, tier)` (#5081) is the explicit-tier twin used by the stdio transport for `GBRAIN_SOURCE` (`env`) and `.gbrain-source` (`dotfile`) bindings: `{ sourceId, via, sourceIds, optedOut }`, where `sourceIds` is the same set `localFederatedSourceIds` computes (shared `federatedSetFor`, so an isolated anchor admits only itself) and `optedOut` lists `federated === false` sources for the denial hint; `undefined` for every other tier. Pinned by `test/explicit-read-admission-5081.test.ts`. Unscoped default-write guard family (the `seed_default` tier is the only one that actually lands a write in `'default'`): `assessDefaultWriteGuard(engine)` → `DefaultWriteAssessment { shouldGuard, defaultPages, nonDefaultPages, nonDefaultSources, failed? }` — an unindexed full-`pages` aggregate; `shouldGuard` when ≥1 non-default source holds pages AND non-default pages outnumber default's; a query failure returns the fail-open no-guard verdict stamped `failed: true`, which cachers/latchers must not treat as settled. `assessDefaultWriteGuardOnce(engine)` is the per-engine WeakMap memo (arms on the first SUCCESSFUL assessment whatever its verdict, forgets a failed attempt so the next caller retries, coalesces concurrent callers; `__resetDefaultWriteGuardMemo` is the test seam). `assessUnscopedDefaultWrite(engine, tier, mutating)` → `{ warning, assessed }` (`assessed: false` only when the aggregate failed); `formatDefaultWriteWarning(a, sourceFlag?)` / `formatDefaultWriteRefusal(cmd, a)` render the two surfaces; `defaultWriteAllowedByEnv()` is the `GBRAIN_ALLOW_DEFAULT_WRITE=1` escape hatch. Consumers: `sync` refuses (dry-run warns), `import` warns via the memo, MCP stdio's `createDefaultWriteAdvisory` prints once. Also exports `isResolverUserError(e)` — the ONE predicate for this module's user-facing throws (`Source "…" not found.` / `not found or is archived.`, `Invalid --source value`, `Invalid GBRAIN_SOURCE value`), consumed by dream.ts, agent.ts and code-scope.ts so a CLI command turns an unknown/archived source into a clean stderr line + exit 1 while anything else keeps propagating. A request snapshot serves tiers 4, 5.5 (its rows carry the 'default' emptiness probe, `default_has_pages`), `assertSourceExists` and the federated set from one sources read, sent with the config table (`resolveSourceWithTier` reads tier 5's key beside the tier-4 listing) (GBRA-156).-- `src/core/source-delete.ts` — `deleteSourceRow`: every unmanaged source delete (`sources remove`, `sources purge`, the `sources_remove` op, expired-archive purge) deletes the `sources` row and that incarnation's `persistence_source_bindings` row in one statement; bindings have no FK to `sources`. #9068: that statement also runs `removedSourceScrubCtes` (the managed `remove`/`purge` in `source-lifecycle.ts` runs them in its final transaction too), which delete the source's `retrieval_event_pages`, `retrieval_event_links`, `retrieval_feedback` and `context_volunteer_events` rows and strip from `eval_candidates.retrieved_slugs` each slug no other source the candidate covers has a page for (`retrieval_weights` and `mention_gazetteer_entries` cascade from `sources`); a second statement runs them again ungated, for a feedback write or capture that held a page row while the delete waited. Pinned by `test/purge-scrub-followups-9068.test.ts`. Bindings of other incarnations are left to `gbrain repair orphan-bindings` (`src/core/persistence/orphan-bindings.ts`, doctor `orphan_persistence_bindings`). The unmanaged claim check (`sync-authority.ts`) and doctor `unbound_source` count only a binding of the source's current incarnation. `EXPIRED_ARCHIVE_SQL` is the one purge-eligibility predicate (unmanaged sweep, its per-row DELETE and the managed `expiredOnly` purge): both archive timestamps must be later than the epoch and the expiry past; `UNVERIFIED_ARCHIVE_SQL` lists archived rows with a missing or epoch timestamp, which the sweep reports blocked with restore-then-archive guidance instead of purging.+- `src/core/source-delete.ts` — `deleteSourceRow`: every unmanaged source delete (`sources remove`, `sources purge`, the `sources_remove` op, expired-archive purge) deletes the `sources` row and that incarnation's `persistence_source_bindings` row in one statement; bindings have no FK to `sources`. #9068: that statement also runs `removedSourceScrubCtes` (the managed `remove`/`purge` in `source-lifecycle.ts` runs them in its final transaction too), which delete the source's `retrieval_event_pages`, `retrieval_event_links`, `retrieval_feedback` and `context_volunteer_events` rows and strip from `eval_candidates.retrieved_slugs` each slug no other source the candidate covers has a page for (`retrieval_weights` and `mention_gazetteer_entries` cascade from `sources`); a second statement runs them again, for a feedback write or capture that held a page row while the delete waited, unless a source with the same id exists again (so a re-created source keeps its rows). Pinned by `test/purge-scrub-followups-9068.test.ts`. Bindings of other incarnations are left to `gbrain repair orphan-bindings` (`src/core/persistence/orphan-bindings.ts`, doctor `orphan_persistence_bindings`). The unmanaged claim check (`sync-authority.ts`) and doctor `unbound_source` count only a binding of the source's current incarnation. `EXPIRED_ARCHIVE_SQL` is the one purge-eligibility predicate (unmanaged sweep, its per-row DELETE and the managed `expiredOnly` purge): both archive timestamps must be later than the epoch and the expiry past; `UNVERIFIED_ARCHIVE_SQL` lists archived rows with a missing or epoch timestamp, which the sweep reports blocked with restore-then-archive guidance instead of purging.- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. `assertNoOverlappingPath(engine, id, path)` is the ONE overlapping-path guard shared by every surface that binds a `local_path` (`addSource`, `gbrain sources set-path`): a path equal to, nested inside, or enclosing another source's `local_path` throws `SourceOpError('overlapping_path')` — overlapping trees make sync/write-through attribute files to the wrong source, so no add or repair path may skip it. The guard compares by spelling AND by realpath (both the candidate and each sibling `local_path` are `realpathSync`'d when they exist, fail-open to the spelling otherwise), so a symlink resolving into or over another source's tree cannot slip past it; the stored pointer and the error message keep the operator's spelling. #6946: `isOwnedClone`/`defaultCloneDir` live in `src/core/url-clone.ts`, re-exported here.- `src/core/sync-concurrency.ts` — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the sites can't drift. `DEFAULT_PARALLEL_SOURCES = 4` is a SEPARATE constant for the per-source fan-out under `gbrain sync --all` — kept distinct from `DEFAULT_PARALLEL_WORKERS` because total live Postgres connections per wave ≈ `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-file pool)` = 32 at both defaults (each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`); `sync.ts` warns when `parallel × workers × 2 > 16`. `resolveWorkersWithClamp(engine, override, commandName, fileCount)` wraps `autoConcurrency` with a per-command stderr clamp warning on PGLite (per-(command, requested) dedup via module-scoped warned-once set with `_resetWorkersClampWarningsForTest()` seam) and is the canonical surface for every bulk-command `--workers N` flag (extract-conversation-facts, extract, edges-backfill, reindex-multimodal, reindex, reindex-code); embed.ts deliberately bypasses it and keeps `GBRAIN_EMBED_CONCURRENCY || 20`. `resolveMaxConnections()` (reads `GBRAIN_MAX_CONNECTIONS`, undefined when unset) + `clampWorkersForConnectionBudget(workers, perWorkerPool, maxConnections, parentPool)` back the opt-in single-sync connection-footprint clamp so a big sync stays under a low pooler cap (`parent_pool + workers×perWorkerPool ≤ budget`); `gbrain doctor`'s `pool_budget` check (`computePoolBudgetCheck` / `checkPoolBudget` in `src/commands/doctor/checks/consolidation-cycle.ts`, re-exported by `src/commands/doctor.ts`) warns when the budget leaves no room for a worker, pointing at `GBRAIN_POOL_SIZE=2`. Pinned by `test/pglite-workers-clamp.test.ts`.