Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs: key-files corpus-source entry names the dream-synthesize reader and extracted-session listing (#7805)
garrytan87283a37040c
diff --git a/docs/architecture/key-files/agent-bootstrap-continued.md b/docs/architecture/key-files/agent-bootstrap-continued.mdindex 8c94f86be4..cbbace6db3 100644--- a/docs/architecture/key-files/agent-bootstrap-continued.md+++ b/docs/architecture/key-files/agent-bootstrap-continued.md@@ -23,7 +23,7 @@- Tests: `test/writeback-config.test.ts`, `test/mcp-instructions-writeback.test.ts`, `test/facts/writeback-gate.test.ts`, `test/hook-writeback-stop.serial.test.ts`, writeback-lane cases in `test/checkpoint-harvest.serial.test.ts`, `test/bootstrap-instructions-block.test.ts` + harness serial additions, `test/writeback-nudge.serial.test.ts`, `test/doctor-memory-writeback.test.ts`, the hermetic 5-step `test/ambient-writeback-lifecycle.serial.test.ts`, and the NON-GATING real-codex door (`test/e2e/bootstrap-real-codex.serial.test.ts`, ambient-writeback describe).- `src/core/sweep.ts` + `src/commands/sweep.ts` — the serve-resident maintenance sweep (the lock owner closes the persistence loop): facts-fence reconciliation (zero-LLM, reuses the cycle extractor with a slug subset), deterministic link/timeline extraction over recent workspace pages (the same cores as `gbrain extract` — remote put_page deliberately skips these, the sweep is where the graph compounds), and spend-gated corpus ingest (skipped keyless; turn-boundary windows through `src/core/context/corpus-windows.ts`, at most `CORPUS_WINDOWS_PER_SWEEP` (8) per file and `CORPUS_WINDOWS_PER_SWEEP_TOTAL` (32, env `GBRAIN_CORPUS_WINDOWS_PER_SWEEP`) per sweep, each windowed file reported in `corpus_files[]` as `{file, windows_done, windows_remaining}`; `.ingested` is written when the last window of the current snapshot finishes, and a finished file re-enters when its stat no longer matches its `.progress`; writeback `.wb-` turn files stay one call; dream outputs and claude-cli self-captures classified by `isClaudeCliSelfSessionId` get a terminal `.ingested` sidecar with `skipped: dream_output` / `self_capture` (the shared `selfCaptureSidecarJson` writer) and are never extracted; the extractor input is `corpusTextForExtraction`, so pasted blocks never reach it). Bounded, fail-soft, never throws; the wall-clock budget stops the corpus pass between files and windows, never mid-extraction (an in-flight extraction finishes and writes its sidecar, E-N2, pinned by `test/sweep-budget-between-items.test.ts`); armed at serve startup (3s, best-effort) and on 10-min idle ticks through the injectable timer seam, everything unref'd; `serve --http` arms `armCorpusDrain` instead (X8: every `CORPUS_DRAIN_INTERVAL_MS`, 10 min, one sweep on a `CORPUS_DRAIN_BUDGET_MS` 60 s budget while a corpus `.txt` has no final `.ingested` sidecar; never overlaps itself; its sweep diagnostics and thrown ticks reach stderr, each distinct line once per run, plus the skip reasons whenever their set changes (#6785); pinned by `test/serve-http-corpus-drain.test.ts`); `GBRAIN_SWEEP=0` kills both. `bootstrap verify` runs the sweep in-process with `corpus: false` (#6786: its engine has no configured gateway, so every window answered `chat_unavailable`). Before any model call the corpus pass runs the facts-absorb owner-readiness preflight per source (`assertSourceLegacyWriter` in `persistence/filesystem-guard.ts`): a claimed, not activated canonical worktree skips the file as `writer_coordinator_required` and logs `gbrain sources writer status <id>` (#6785; the compact and writeback harvest lanes do the same). A `chat_unavailable` result is not terminal: the windowed file or `.wb-` turn keeps no sidecar, nothing is recorded, and the pass stops (`corpus_chat_unavailable`). Failures thrown after a chat call are counted per window (`<file>.failures.json`, see `corpus-windows.ts`) and skipped as `corpus_window_backoff` / `corpus_window_dead_lettered`; `gbrain sweep --retry-failed [--apply]` previews or clears them (doctor `memory_writeback` names the paid retry as its fix). Pinned by `test/sweep-corpus-chat-unavailable-6786.test.ts` and `test/sweep-corpus-failure-cap-6785.test.ts`. `gbrain sweep --once` is the trusted CLI seam for operator catch-up runs (CLI-only, never over MCP). The link pass threads the same `link_resolution.cross_source` opt-in and configured `sources.default` the CLI extract lanes pass into `resolveCandidateSources`, so an edge into another source is created (flag on) or counted as `cross_source_link` in `skipped` (flag off) — never silently dropped, and the reconcile never deletes a cross-source edge `extract links --source db` created. A corpus window's facts batch is keyed by its text (the sweep passes `reAdmitFileRefusals`), so when the canonical file check refused some of its entity requests, `resumeManagedFacts` admits their retained facts as a follow-up batch once every refused page passes that check again: no new extraction, the page's current revision, never wider than the current default visibility, committed requests reported rather than re-admitted, the stored and current writer grants re-checked, at most three follow-ups. Pinned by `test/managed-sweep-refused-window-retry.test.ts`.- `src/core/context/corpus-windows.ts` — windowed fact extraction of session-corpus files (#5887). `retireCorpusFile` / `retireCorpusTurns` (#6091) record a file or its off-period turns as finished without extracting (`skipped_reason`; a retire overrides another extractor's lease, and a later rewrite extracts only unrecorded turns). `parseCorpusTurns` (defined in the provider-free `corpus-turns.ts`, re-exported here) splits only at the exact `\n\n[user]\n` / `\n\n[assistant]\n` markers `toCorpusText` emits (text before the first marker is a header-less preamble); each user turn is paste-stripped whole before windowing; `planCorpusWindows` packs whole turns into windows of at most `MAX_TURN_TEXT_CHARS` (exported from `facts/extract.ts`) and splits a turn that cannot fit a fresh window at a newline or space, never inside a surrogate pair, repeating its `[role]` header on every piece. `<file>.progress` (suffix constants in `corpus-segments.ts`, GC'd by the session-end hook with the `.txt`, never deleted by its resume rewrite) holds `{version, generation, turns[{start,end,sha256}], continuation{turn_sha256, stripped_offset}|null, windows_done, finished(stat)|null, totals, entity_slugs, skipped_reason?, lease}`: `resumePoint` resumes at the first turn whose hash is unrecorded (appends, compaction remainders and same-size or shorter rewrites cost only their new turns). Every write is compare-and-set under the O_EXCL `<file>.progress.lock` (stale after 30 s): a window commits only when generation, last turn hash and lease owner still match and progress grows; the lease (stale after an hour, like the claim) stops a sweep whose `.in-progress` claim the hook deleted from extracting a window another extractor holds. `runCorpusWindows` returns `complete`/`partial`/`aborted`/`contended`/`changed`/`source_changed`/`unavailable`/`backoff`/`dead_lettered` with cumulative counts; an aborted or `unavailable` window records nothing, and a progress a pre-#6786 build finished with `skipped_reason: chat_unavailable` is rewound under the lease CAS to its retired turns (source binding, totals and links kept). Transport and write errors release the lease and throw; one thrown after a chat call (`extractUnderFailureCount`, observed through the AI invocation preflight) counts against the window's start key (`<turn sha256>@<offset>`) and source incarnation in `<file>.failures.json`: retried on the next sweep, then after 10, 20 and 40 minutes (`CORPUS_WINDOW_RETRY_BASE_MS`), stopped at `CORPUS_WINDOW_FAILURE_CAP` (5); tail growth and error labels never reset it, a commit of that window removes it, and one change of `extractionConfigFingerprint` (model, provider, key presence) re-arms a stopped window once (#6785). `corpusFileCompletion` in `corpus-segments.ts` is the one completion classifier (selection, the under-claim re-check, retention GC; `corpusSidecarState` for the drain and the doctor backlog): a `chat_unavailable` sidecar is pending, and only `done`/`legacy` files get the plain retention (#6786); `adoptFinishedCorpusFile` records a pre-window `.ingested` file at its current turns once (only when the sidecar is newer than the file). `checkpoint-harvest.ts` (`hook:compact`) extracts window 1 only: when more remains it publishes window 1's links, releases the receipt and leaves `.progress` for the sweep; an existing `.progress` means window 1 is done. Pinned by `test/corpus-windows.test.ts`, `test/sweep-corpus-windows.test.ts` and `test/managed-sweep-corpus-windows.test.ts` (managed coordinator, PGLite and Postgres).-- `src/core/context/corpus-source.ts` — #6268, the session's source for every corpus file. Writers (`hook.ts` compact/stop/session-end, the OpenClaw `compact()`) call `openSessionCorpus`, which freezes one source per session at first use (GBRAIN_SOURCE, then the `.gbrain-source` dotfile, else unresolved) into `<sid>.source.json` in the `sourced/` spool and stamps it into the file name (`.src-<id>`, `default` included; `_unresolved` cannot collide with a real id). Pre-stamp sweepers never list the spool. `resolveCorpusFileSource` is the sweep/harvest/serve read: the stamp or frozen record, else the brain's cwd chain once (frozen, tier `engine`); legacy top-level files are held (`corpus_legacy_unstamped`) unless a legacy wb stamp, a frozen record or a single-source brain proves the source, never assigned the pass source. The first engine resolution binds the source incarnation; a recreated source holds the session (`corpus_source_incarnation_changed`). `gbrain sweep --assign-corpus <source>` is the operator mapping (preview, then `--apply`); doctor `memory_writeback` reports `details.corpus_source_holds`. Tests: `test/corpus-source-stamp.test.ts`.+- `src/core/context/corpus-source.ts` — #6268, the session's source for every corpus file. Writers (`hook.ts` compact/stop/session-end, the OpenClaw `compact()`) call `openSessionCorpus`, which freezes one source per session at first use (GBRAIN_SOURCE, then the `.gbrain-source` dotfile, else unresolved) into `<sid>.source.json` in the `sourced/` spool and stamps it into the file name (`.src-<id>`, `default` included; `_unresolved` cannot collide with a real id). Pre-stamp sweepers never list the spool. `resolveCorpusFileSource` is the sweep/harvest/serve and dream-synthesize read: the stamp or frozen record, else the brain's cwd chain once (frozen, tier `engine`); legacy top-level files are held (`corpus_legacy_unstamped`) unless a legacy wb stamp, a frozen record or a single-source brain proves the source, never assigned the pass source. The first engine resolution binds the source incarnation; a recreated source holds the session (`corpus_source_incarnation_changed`). `gbrain sweep --assign-corpus <source>` is the operator mapping (preview, then `--apply`; it also lists sessions the sweep already extracted, because dream synthesis holds those until their source is mapped, #7805); doctor `memory_writeback` reports `details.corpus_source_holds`. Tests: `test/corpus-source-stamp.test.ts`.- `src/core/context/sync-ipc.ts` + `src/core/serve-sync-runner.ts` + `src/commands/sync-delegate.ts` — serve-delegated sync. `sync-ipc.ts` is the LEAF wire module: `DELEGATED_SYNC_OPTION_FIELDS` (the fields `SYNC_FLAGS` in `src/core/sync-flags.ts` forwards through the serve, plus `sourceId`, `embedDeferred`, `timeoutSeconds` and `origin`) drives the validator, the CLI wire-builder, and the serve-side SyncOpts builder; `validateDelegatedSyncOptions` rejects unknown keys fail-closed (repoPath/skipLock/lockId/concurrency are unreachable from the socket), requires `timeoutSeconds` (0 = the explicit `--no-hard-deadline` unbounded encoding; everything else clamps to 24h), and `toWireSyncResult` truncates `pagesAffected` to 50 + a true total under the 256KB message cap. `serve-sync-runner.ts` is the serve-side module singleton: one job at a time (correctness still rests on the `gbrain-sync:<source>` row lock inside performSync), clientToken attach (a lost-ack retry finds its own running OR retained-terminal job instead of duplicate-running), per-job deadline timer, `[serve-sync]` lifecycle stderr lines, an idempotent shared `shutdownDelegatedSync()` both serve shutdown paths await BEFORE `engine.disconnect()` (the disconnect-mode drain is allowAbort:false after `_db` is nulled, so settle writes need the live engine; a registered drainer remains as backstop), and `maybeDrainDeferredEmbeds` (delegated jobs always run noEmbed — the cost gate lives in runSync — and the serve drains stale embeds afterwards via `runEmbedCore`, keyless-safe, cleared only when a drain finds nothing left; wire `noEmbed` records that the USER declined embeds and suppresses the drain). `sync-delegate.ts` is the CLI half: read-only holder probe (`probeLivePgliteHolder`), DEFAULT-DENY argv gate (any unclassified token refuses by name — a silently dropped `--exclude` would perform the wrong sync), engine-free source tiers (`resolveSourceIdEngineFree`), 1s poll loop tolerant of event-loop-blocked serves with a PID-probe backstop, `unknown_job` → serve-restarted resume hint, Ctrl-C → `sync_abort` (second Ctrl-C exits 130), and runSync's `pull_failed` verdict mirroring. Mounts never delegate; opt-outs `--no-delegate` / `GBRAIN_SYNC_NO_DELEGATE=1` (client) and `GBRAIN_SERVE_SYNC_IPC=0` (serve). #6317 (D1 a′), managed Postgres brains: `maybeDelegateManagedSyncToServe(engine, args, opts)` runs post-connect from `runSingleSourceSync` right before `performSync`; it probes the host's live full `serve` consumer through `persistence_consumers` (`consumer-heartbeat.ts`, the database, never a socket), admits the catch-up's flags (`--no-bulk`, `--lanes`/`--no-lanes`, `--json`, the processing flags), sends the CLI's durable `cli` registration in `sync_start`, prints the drain's own numbered lines from `sync_status` (`managed`, `lines`, `drain`, `next`) and returns the SyncResult to the in-process printers, so exit codes and the JSON envelope do not change. The serve verifies the registration (`withVerifiedLocalRegistration`) and runs `performSync({ drain: true })` as that writer under `withHumanLineSink`; the shared-secret lane stays under `withLegacySyncDelegation` and is still refused on a managed brain; a denied, revoked or stdio registration answers `sync_start` with `permission_denied` and the envelope (`refusal`), which the CLI rethrows. Every other rung (no owner, a non-serve owner, an unsupported flag, no socket, the kill switch, `stale_serve` / `unsupported_kind` from an older serve, `source_mismatch`, a serve lost mid-drain) runs without the serve with one actionable line; only the opt-out, a serve not answering its socket and a `stale_serve` serve call `forceOwnConsumer` (#6543), the rest admit writes the serve consumes under `persistence.single_consumer`. #6712 PR 1: neither builder sends `explicitProcessing` any more; the serve validates one from an older CLI by shape and drops it (`WIRE_SKEW_WINDOWS` in `src/core/wire-skew.ts`), and a resume inherits the cursor's processing opt-outs. #6712 PR 2: `WIRE_BOOL_FLAGS` / `VALUE_FLAGS` / `IGNORED_FLAGS` and the managed pick list are views of `SYNC_FLAGS`; a retired field the validator drops comes back as `ignored`, and the serve logs `retiredWireNotice` and streams it to the older client as a progress line (`test/sync-flag-table-6712.test.ts`). The serve's delegated run follows the source's drain lease as holder `serve` (`followDrainLease`, `drainHolder`). Wire `embedDeferred` (sent only when true) carries the cost gate's runtime deferral, so adopting a cursor that stores `noEmbed=false` never embeds in that run. `maybeDelegateManagedSyncToServe` refuses `--skip-failed` / `--include-gitignored` on a managed source with `managed_sync_flag_refused` before any probe or socket call (`test/sync-delegated-options-6696.serial.test.ts`, `test/sync-managed-forbidden-flags-6696.test.ts`). Pinned by `test/context/sync-ipc-validation.test.ts`, `test/sync-delegate-ladder.test.ts`, `test/serve-sync-runner.serial.test.ts`, `test/serve-sync-handoff.serial.test.ts`, `test/context/resolve-ipc-sync-kinds.test.ts`, `test/e2e/sync-delegation-under-serve.serial.test.ts` (gating tier1 step) and `test/e2e/serve-delegated-managed-sync.serial.test.ts` (Postgres, spawned serve, MCP latency printed). GBRA-133 2b.3: the deferred-embed drain re-arms itself until the backlog is gone: a pass that left work retries after `GBRAIN_DEFERRED_EMBED_RETRY_MS` (default 5000), a failing provider backs off 1x/3x/12x/60x, a running delegated sync defers it at most `DEFERRED_EMBED_BUSY_MAX` (3) times before it drains beside the sync, and a keyless brain keeps the backlog with no provider call or timer; `deferredEmbedDecisions` records each decision. #7243: `startDelegatedSync`'s in-process `logLines` option (never on the wire) also writes a verified job's human lines to the serve log, for the resident notifier's runs that nobody polls.- `src/core/capability.ts` — config-plane keyless/keyed detection + the honest capability report (per-provider keyless banner: OpenAI = semantic search + auto-extraction, Voyage = semantic search, Anthropic = auto-extraction) rendered by verify and the runbook. The extraction probe resolves through the SAME shared `resolveEffectiveChatModel` the gateway's reconfigure fallback uses (`GBRAIN_MODEL` > servable file pin > key-aware tier default), so a stale unservable pin degrades identically in the report and at runtime. Key/env fold comes from `mergedProviderEnv` (`src/core/ai/provider-env.ts`). Accepted limitation (documented in the module): DB-plane overrides (`models.default`, `facts.extraction_model`) are invisible to this engine-less probe; the runtime gate in `facts/extract.ts` and the engine-aware pre-enqueue gate in `facts/backstop.ts` are the backstops.- `src/core/readiness.ts` — capability readiness (agent operator contract A7): `ReadinessEntry {capability, state, reason, why, fix, tier, asset?, http_visible}` for embeddings, chat_llm, writeback, tool_surface, sync, harness_wiring (config tier) and worker, backup per asset, migrations, sync (probed tier); each capability's reasons are a closed exported const (`EMBEDDINGS_REASONS`, …). `configReadiness(cfg, {transport})` is synchronous and memoized per (config object, transport); its only file read is one memoized `peekLock()` (none when this process holds the lock, via `heldLockFor`), plus one process-wide gbrain-binary PATH lookup on the CLI transport only. The HTTP view drops `http_visible:false` entries and passes the rest through `redactForTransport`. `probedReadiness(engine, {cache})` never calls `getStats`/`getHealth`; 60 s cache (`createReadinessCache()` per ServeHttpContext, process-wide otherwise), single-flight, 2 s per-probe bound, stale-while-revalidate; `readinessTail` is MCP initialize's best-effort 250 ms tail. `embeddingEnablement(cfg)` is the one "turn on embeddings" fix: resolves the datastore (PGLite path, Postgres, a mount from `--brain`/`GBRAIN_BRAIN_ID`) and the column width, picks a keyed provider that fits (or asks for a key with `inputs`), and emits in-place `init --force --embedding-model … --embedding-dimensions … [--path …]` or, for a mount, `embeddings enable --brain …`; it never rebuilds a column. `exclusiveFix` turns an exclusive fix into a two-step plan (actor `user` stops the owning serve named by `lock_owner`, `then` the command) unless the command delegates to the owner. A live status-only `serve --http` marker (`liveStatusMarkers`, config plane) makes CLI `harness_wiring` `serve_status_only` with `transport: 'http'` and the status reason's fix (`httpStatusServerEntry`, shared with doctor), never "start `gbrain serve --http`" on the port it holds. `harnessWiringEntry` builds harness-native stdio registration (absolute binary, `stdioServeArgv` with `REGISTRATION_SURFACE` = `starter`, or `input.surface` from `gbrain init --surface`; `configReadiness(cfg, {transport, surface})` skips the memo when a surface is passed), shared-HTTP `bootstrap harness`, or the MEMORY_VERBS install anchor, with effects and destinations disclosed. Consumers never coach on `disabled_by_choice`. Tests: `test/readiness.test.ts`, `test/readiness-http-cost.test.ts`, `test/readiness-advice-lint.test.ts` (no hand-written enablement advice outside this module; shrink-only baseline), `test/readiness-embedding-enablement.serial.test.ts` (the E2E matrix).