Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs: keep key-files pages under their size cap
garrytaneb9d87805460
diff --git a/docs/architecture/key-files/core-search-1.md b/docs/architecture/key-files/core-search-1.mdindex cd7ff0220e..198dd347ac 100644--- a/docs/architecture/key-files/core-search-1.md+++ b/docs/architecture/key-files/core-search-1.md@@ -52,7 +52,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGELthe cache-lookup and vector-arm embeds still share one budget(`test/search/query-embed-deadline-start.test.ts`).-- `src/core/search/image-loader.ts` — `loadImageInput(input, opts)` accepts a `data:` URI or `http(s)://` URL, and a local path or `file://` URI only when the caller sets `allowLocalPath` (the trusted local CLI; `search_by_image` refuses `image_path` and any non-http(s) `image_url` for every other caller). A local path must be a regular file within the size cap, checked before it is read. Magic-byte sniff for PNG/JPEG/WebP. Hard size cap (default 10 MB, configurable via `search.image_query.max_bytes`). URLs route through `fetchWithSSRFGuard` so DNS rebinding + redirect chains are defeated; shared guarded transport pins validated DNS answers and streams decoded bodies with a 2 MiB remote cap (10 MiB local cap). `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).+- `src/core/search/image-loader.ts` — `loadImageInput(input, opts)` accepts a `data:` URI or `http(s)://` URL; a local path or `file://` URI only with `allowLocalPath` (trusted local CLI only), after a regular-file and size check. Magic-byte sniff for PNG/JPEG/WebP. Size cap (default 10 MB, `search.image_query.max_bytes`). URLs go through `fetchWithSSRFGuard` (pinned DNS answers, no redirect chains), with a 2 MiB cap on decoded remote bodies. `ImageLoadError` with discriminated `code` (INVALID_FORMAT / OVERSIZED / INVALID_URL / FETCH_FAILED / TIMEOUT / SSRF_BLOCKED / NOT_FOUND).- `src/core/search/keyword-paging.ts` — keyword counts and keyword-mode paging for `search`. `parseSearchPaging` validates `match` (`hybrid` default, `keyword`), `cursor`, the 10,000 offset cap (`search_offset_over_cap`; hybrid `offset` keeps its ranked-list meaning) and refuses an agent-facing `mode` (`search_mode_local_only`) and a cursor outside keyword mode (`search_cursor_requires_keyword`), each with a fix naming `match: "keyword"`. `startKeywordCount` / `countKeywordMatches` run `engine.countKeywordPages` beside the search until max(2 s, search end + `KEYWORD_COUNT_GRACE_MS`); the result is `keyword_total` / `keyword_truncated` (`keyword_total_capped` past 10,000) or the `keyword_count_unavailable` degraded stage, never a guess. `readKeywordPage` reads `limit + 1` rows from `engine.searchKeywordPages` (so `truncated` and `next` never depend on the count) and builds `next`, a ready-to-send argument object whose `cursor` is a base64url keyset `(score, page_id, rows shown)` bound to the query, `types` and `source_id`. `keepEnumeratedRows` keeps every enumerated page after evidence delivery (`evidence_omitted: true`, empty text, when its text did not fit). `searchCountLine` renders the one model-visible line `src/mcp/dispatch.ts` appends after the rows on every search call. Tests: `test/search-keyword-paging.test.ts`, `test/search-keyword-cap.slow.test.ts`, `test/e2e/search-keyword-paging-parity.test.ts`.diff --git a/docs/architecture/key-files/files-and-sync-1.md b/docs/architecture/key-files/files-and-sync-1.mdindex d98c805486..3571b167b7 100644--- a/docs/architecture/key-files/files-and-sync-1.md+++ b/docs/architecture/key-files/files-and-sync-1.md@@ -25,7 +25,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/git-head.ts` — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true also requires a clean working tree (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). `requireCleanWorkingTree` is `boolean | 'ignore-untracked'` — in `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no` so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" (sync's incremental path keys off the commit diff and does not import untracked files by default — it counts them as `uncommitted` drift with a stderr warning; `--working-tree` opts into importing them); `GitCleanProbe` gains an `ignoreUntracked?` second arg. Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) keep unit tests R2-compliant (no `mock.module`). Uses `execFileSync` with array args so shell metachars in `local_path` cannot escape to a shell (the test runs real `execFileSync` against `'/nonexistent/$(touch <sentinel>)/repo'` and asserts the sentinel is never created). Fail-open on every error (missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → false) so the caller falls back to its time-based check. The chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Pinned by `test/core/git-head.test.ts` (incl. the shell-injection guard).-- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last used by the sync cost-estimator's fetch-first path, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `http.followRedirects=false`, `protocol.file.allow=never`, `protocol.ext.allow=never`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git <GIT_SSRF_FLAGS> clone <GIT_SSRF_SUBCOMMAND_FLAGS> --depth=1 [--branch X] [skipLfs flags] <url> <dir>`; `skipLfs` (set for URL sources a remote caller adds, recorded as `added_by_remote_caller: true` and honored by `recloneIfMissing`) writes `lfs.fetchexclude=*`, `core.hooksPath=/dev/null` and `lfs.locksverify=false` into the clone's config, so the repository's own `.lfsconfig` cannot point later git commands at an LFS endpoint. `pullRepo` argv: `git -C <dir> <durableSsrfFlags()> pull <GIT_SSRF_SUBCOMMAND_FLAGS> --ff-only` — `durableSsrfFlags()`, not the hardcoded `GIT_SSRF_FLAGS`: identical `-c` flags except `protocol.file.allow` honors the `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1` escape hatch. `pullRepo(repoPath, { timeoutMs?, signal? })` runs git through `execFileBounded`, so the caller's event loop keeps running; an aborted `signal` or the `timeoutMs` deadline (300 s by default) stops it, and `isStoppedGitPull(err)` separates that stop from a real git failure by reading the `GitOperationError` cause (`ETIMEDOUT`, `ABORT_ERR` or `SIGTERM`). Only `git pull` itself is signalled; the `git fetch` and transport it spawned are not. `fetchRemote` uses the same helper. Pinned by `test/git-remote.test.ts` position-anchored guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). Also exports the durability-side helpers that power `gbrain sources harden/pull`: `GIT_ENV_AUTH` (the no-prompt env minus the askpass `/bin/false` overrides, so an auth'd push/fetch can consult the repo's configured credential helper while `GIT_TERMINAL_PROMPT=0` still fails fast on a missing credential), `divergenceSafePull(repoPath, branch)` (fetch + `pull --rebase`; returns `skipped_dirty` on a dirty tree, `conflict_aborted` on ANY `pull --rebase` failure (the catch doesn't classify the cause — it's treated as a conflict) after up to two best-effort `rebase --abort` attempts (each swallows its own failure; `conflict_aborted` is still returned even if rebase state remains), else `up_to_date`/`advanced`), `detectDefaultBranch` (origin/HEAD → current branch → `main`), `pushProbe(repoPath, branch)` (authenticated `push --dry-run` that proves push access and classifies `auth`/`protected`/`unreachable`), and `isWorkingTreeDirty`. These auth'd paths route their `protocol.file.allow` through `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` (default `never`; set `=1` for self-hosted filesystem remotes), unlike `cloneRepo`, which still uses the hardcoded `GIT_SSRF_FLAGS` and stays strict.+- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the sync cost-estimator's fetch-first path, with the same hardened flags). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `http.followRedirects=false`, `protocol.file.allow=never`, `protocol.ext.allow=never`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (before the verb, git exits 129). `cloneRepo` argv: `git <GIT_SSRF_FLAGS> clone <GIT_SSRF_SUBCOMMAND_FLAGS> --depth=1 [--branch X] <url> <dir>`; `skipLfs` (URL sources a remote caller adds; `recloneIfMissing` follows their `added_by_remote_caller`) writes `lfs.fetchexclude=*`, `core.hooksPath=/dev/null` and `lfs.locksverify=false` into the clone's config. `pullRepo` argv: `git -C <dir> <durableSsrfFlags()> pull <GIT_SSRF_SUBCOMMAND_FLAGS> --ff-only` — `durableSsrfFlags()`, not the hardcoded `GIT_SSRF_FLAGS`: identical `-c` flags except `protocol.file.allow` honors the `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1` escape hatch. `pullRepo(repoPath, { timeoutMs?, signal? })` runs git through `execFileBounded`, so the caller's event loop keeps running; an aborted `signal` or the `timeoutMs` deadline (300 s by default) stops it, and `isStoppedGitPull(err)` separates that stop from a real git failure by reading the `GitOperationError` cause (`ETIMEDOUT`, `ABORT_ERR` or `SIGTERM`). Only `git pull` itself is signalled; the `git fetch` and transport it spawned are not. `fetchRemote` uses the same helper. Pinned by `test/git-remote.test.ts` position-anchored guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). Also exports the durability-side helpers that power `gbrain sources harden/pull`: `GIT_ENV_AUTH` (the no-prompt env minus the askpass `/bin/false` overrides, so an auth'd push/fetch can consult the repo's configured credential helper while `GIT_TERMINAL_PROMPT=0` still fails fast on a missing credential), `divergenceSafePull(repoPath, branch)` (fetch + `pull --rebase`; returns `skipped_dirty` on a dirty tree, `conflict_aborted` on ANY `pull --rebase` failure after up to two best-effort `rebase --abort` attempts, else `up_to_date`/`advanced`), `detectDefaultBranch` (origin/HEAD → current branch → `main`), `pushProbe(repoPath, branch)` (authenticated `push --dry-run` that proves push access and classifies `auth`/`protected`/`unreachable`), and `isWorkingTreeDirty`. These auth'd paths route their `protocol.file.allow` through `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` (default `never`; set `=1` for self-hosted filesystem remotes), unlike `cloneRepo`, which still uses the hardcoded `GIT_SSRF_FLAGS` and stays strict.- `src/core/import-checkpoint.ts` — `loadCheckpoint(path, currentDir)`, `saveCheckpoint(path, cp)`, `resumeFilter(allFiles, dir, completed)`, `clearCheckpoint(path)`, `resolveImportTargetDir(dir)`, and the `ImportCheckpoint` type. Payload: `{schema_version: 1, owner: 'gbrain', kind: 'import', dir, completedPaths, timestamp}`; `dir` is the absolute realpath of the import target, never the caller's CWD. Paths record successful/unchanged progress, not content identity; ordinary `runImport` revalidates all current files via `importFile`'s existing content-hash comparison rather than using `resumeFilter`. Company-brain ingestion uses separate protected database checkpoints for an immutable approved manifest. The exported pure filter returns a path array and skips completed paths for company-brain manifest resume; by itself it is not proof that mutable file content is unchanged. `loadCheckpoint` returns `null` for missing/malformed payloads, target mismatch, invalid path/timestamp fields, or incompatible producer metadata; absent producer fields in legacy path-set checkpoints remain accepted. Old positional checkpoints log a stderr diagnostic before discard. `saveCheckpoint` sorts paths and atomically writes via `.tmp` + `rename()`; save/clear failures are non-fatal and silent. The caller supplies the checkpoint path via `gbrainPath()`, preserving `GBRAIN_HOME` isolation. Pinned by `test/import-checkpoint.test.ts`, `test/import-resume.test.ts`, and `test/import-json-stdout.test.ts`.