Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
57.3 KB
Raw Blame

Appendix to ../gbrain-evals.md. Detailed per-area notes from one research pass over gbrain-evals cd447b3 (v0.10.237) and gbrain 6ce7195. Paths are relative to the gbrain-evals root unless marked gbrain:. "G1" to "G10" in the text name the appendix files listed in the main map.

G5 runner group A: gbrain-evals eval/runner directories

Checkout: the gbrain-evals checkout (read only). All paths below are relative to eval/runner/ unless shown otherwise. Scope: 16 runner groups covering batch/, bg1/, budgeted-delivery/, cat40/ (with cat40-model-ladder.ts and scripts/cat40-*.sh), cat41/ (with cat41-agent-operator.ts), decisions/ (with decide.ts and eval/decisions/), evaluator/, evidence-delivery/ (with evidence-delivery.ts), external-ideas-wave-a/ (with external-ideas-wave-a.ts), facts-absorb/ (with facts-absorb-gate.ts), failure-causes/, graph-backend/, lifecycle/, lifecycle-lite/ (with lifecycle-experiment.ts, lifecycle-lite.ts, lifecycle-report.ts), lhf-wave-2-attendance-gate.ts.

Summary

Runner Entry path Measures Run command Tier / time / cost LLM keys Corpus Coupling
cat40 model ladder and Hard eval/runner/cat40-model-ladder.ts (helpers cat40/, wrappers scripts/cat40-hard.sh, scripts/cat40-followups.sh) Whether gbrain's advantage over file, pg and memory arms persists across models; agent task accuracy bun eval/runner/cat40-model-ladder.ts --models ... --arms oracle,fs,pg,memory,gbrain --budget-usd N --out <dir>; --scripted for $0 Paid. Gbrain cell $0.02 to $0.30, fs/pg about $0.05 per cell (protocol). Hard ledger cap $1,794; steps from $1.79 (smoke) to $114.61 (freeze-check), wall clock 20 min to 7.5 h ANTHROPIC, OPENAI, PARTICLE (particle-1), VOYAGE for gbrain slots. --scripted is keyless Named worlds eval/data/model-ladder-v1* and generated Hard worlds by seed CLI+MCP (spawns bun <build>/src/cli.ts serve, JSON-RPC over stdio), plus in-process PGLite via bun -e subprocesses
cat41 agent operator eval/runner/cat41-agent-operator.ts (helpers cat41/, Dockerfile) Whether real Claude Code and Codex CLI sessions in a pinned container take the right operator actions (degraded recall, consent, recovery) bun eval/runner/cat41-agent-operator.ts run --gbrain <checkout>@<ref> --paid --budget-usd N --out <dir>; score, overhead, gate are keyless Paid. About $12 to $17 per 3-repeat pass, about 35 min at concurrency 4 (protocol). Reserve $2.5 (claude) and $1.5 (codex) per run ANTHROPIC, OPENAI (inside container) Scenarios in cat41/scenarios.ts, D7 worlds cat41/brief-worlds.json (committed) Mixed: spawns docker; in-container spawns bun /opt/gbrain/src/cli.ts; in-process gbrain module imports inside the container; host MCP client for overhead
decision kit eval/runner/decide.ts (npm eval:decide; helpers decisions/, data eval/decisions/) Held-out and dev decision verdicts for a gbrain candidate vs baseline, per source bun run eval:decide <init|fetch|preflight|dev|verdict|status|check> --decision <dir> Dev runs are paid through child memory-qa jobs. Cost and time unknown (not traced into memory-qa) Via child jobs (OPENAI seen in decide.ts) eval/decisions/datasets/*.json (committed, 208K), eval/decisions/splits/*.json (committed, 60K) Spawn (git, and bun child runners per arm); in-process only for spec and spec parsing
evidence delivery eval/runner/evidence-delivery.ts (helpers evidence-delivery/) Evidence delivery ablation E1, product-path parity E3, sealed bridge E2 on LongMemEval-S bun eval/runner/evidence-delivery.ts <cluster-map|power|parity|analyze|costs> keyless; campaign-open, freeze, e1, e3, e2-* paid Paid. Campaign cap $400 (campaign-open --budget-usd <= 400); smoke cap $3 (SMOKE_CAP_USD). Wall clock unknown OPENAI, VOYAGE, Anthropic reader (Sonnet), GPT-4o reader longmemeval_s_cleaned.json (passed by --dataset, not committed here); R1 receipts under docs/benchmarks/2026-09-29-longmemeval-opaque-qa/ In-process: dynamic import of gbrain src/ from GBRAIN_DIR; MCP stdio for e3 (bun --preload ledger-preload.ts src/cli.ts)
external ideas wave A eval/runner/external-ideas-wave-a.ts (helpers external-ideas-wave-a/, synthetic-dates.ts) Whether search.evidence_resolve_dates (absolute date brackets) changes LongMemEval-S accuracy bun eval/runner/external-ideas-wave-a.ts <freeze|affected|answer|analyze|verdict|audit|estimate|a2-verdict>; estimate keyless Paid. Lane ledger $220 across waves A and B; runs over $50 need --gbra108-approval. Wall clock unknown ANTHROPIC, OPENAI, VOYAGE LongMemEval-S (--dataset), synthetic relative-date set generated by synthetic-dates.ts In-process (gbrain src/ via loadGbrain); batch HTTP to providers via batch/ lane
facts absorb gate eval/runner/facts-absorb-gate.ts (helpers facts-absorb/) Whether a cheaper facts.extraction_model keeps the facts gbrain's background extractor saves (recall, precision, attribution, corrections) bun eval/runner/facts-absorb-gate.ts --gbrain <checkout>@<ref> --arms ... --seeds 60,61 --prereg <file> --budget-usd N --out <dir> Paid. --budget-usd required; cost and time unknown Provider keys for each extraction arm (anthropic, openai) through MeteringProxy Generated facts-absorb world (eval/generators/facts-absorb-gen.ts) MCP stdio (gbrain serve --surface full) plus CLI (gbrain call, gbrain jobs work)
lifecycle experiment eval/runner/lifecycle-experiment.ts (helpers lifecycle/) Memory lifecycle correctness: ingest, outage, correct, reconcile, forget, restart, query, across build x engine x interface bun eval/runner/lifecycle-experiment.ts --gbrain-repo ../gbrain [--builds a=<sha>,b=origin/master] [--engines pglite,postgres] [--interfaces cli,mcp-stdio,mcp-http] [--out <receipt.json>] $0 (hermetic). Wall clock unknown (many spawned processes). Receipts in docs/benchmarks/2026-09-29-lifecycle* None. Provider keys are stripped from child env Generated in code (lifecycle/scenario.ts); no dataset CLI and MCP (stdio and HTTP) spawned as child processes; Postgres via Bun SQL
lifecycle-lite eval/runner/lifecycle-lite.ts (helpers lifecycle-lite/) Update and forget behavior (N1 correction chains, N5 canaries) for open-source memory systems; report-only bun eval/runner/lifecycle-lite.ts --system <shim URL>|gbrain-shootout|fake|honest-reference|mutation:<kind> --output <dir> Default hermetic with hash embedder. Paid only with --paid --budget-run-id. Timeouts 600 s finish, 3600 s ingest None for hermetic. OPENAI, ANTHROPIC, VOYAGE for --embed real, --qa reader, reranker Generated generators/lifecycle-lite-gen.ts seeds In-process for gbrain-shootout (src/core/ai/gateway.ts, engine methods); HTTP protocol v1 for shims; bash -c spawn for restart
lifecycle-report eval/runner/lifecycle-report.ts Markdown tables from lifecycle-experiment receipts, plus cross-run match bun eval/runner/lifecycle-report.ts <receipt.json> [more...] $0, under a second None Receipts only None
LHF wave 2 attendance gate eval/runner/lhf-wave-2-attendance-gate.ts Preregistered attendance paraphrase gate: fired counts and false routes from two N9 receipts bun eval/runner/lhf-wave-2-attendance-gate.ts --base <receipt> --branch <receipt> --base-relations <json> --branch-relations <json> [--json] $0. Counts only, no statistics None Two N9 receipts plus relations files from parse-relational-questions.ts None (reads JSON)
batch lane (LongMemEval W10) eval/runner/batch/ (w10.ts, w10a-capture.ts, w10-particle.ts, w10-particle-second-judge.ts, w10-rescore.ts; lane libs submit.ts, transport.ts, ledger.ts, manifest.ts, sources.ts, receipts.ts) Reader and judge accuracy on LongMemEval arms through provider batch APIs, with exactly-once submission bun eval/runner/batch/w10.ts build|plan|run <arm>; w10-rescore.ts keyless Paid (batch factor 1.0 list price, see batch/ledger.ts). w10-rescore.ts is $0. Total cost unknown ANTHROPIC, OPENAI (batch/transport.ts) LongMemEval-S plus R1/R2 reranker-on call logs (batch/sources.ts) In-process imports from the pinned node_modules/gbrain; HTTP to provider batch APIs; SQLite lane state (bun:sqlite)
BG1 battleground eval/runner/bg1/ (verdict.ts, power.ts, pricing.ts, rule.ts, match.ts, exposure.ts, judge-check.ts, launch.ts, types.ts) Changing-facts battleground: gbrain vs comparators on short answers, with paired t and Holm; exposure placement; judge validity bun eval/runner/bg1/verdict.ts verdict --receipts <dir>...; power.ts, pricing.ts keyless; judge-check.ts paid Analysis tools are $0. judge-check paid on public dev data. Campaign cost in bg1/pricing.ts ANTHROPIC, OPENAI, PARTICLE, VOYAGE (judge-check only) BG1 manifest eval/decisions/datasets/bg1-manifest.json, cells from q1/ (outside scope) None for analysis. judge-check uses MeteringProxy over HTTP to judge models
budgeted delivery eval/runner/budgeted-delivery/ (h1-run.sh, c8k-run.sh, e2-keyless-gate.sh, keyless-gate.sh, dry-run scripts; TS h1.ts, c8k.ts, c8k-product.ts, e1-readings.ts, e2-readings.ts, e3-retrieval-gate.ts, accounting-gate.ts, budget-sizing.ts) Whether evidence packed under explicit token budgets keeps reader accuracy (E1, E2, E3, 8k conversion, H1 held-out custody) bash eval/runner/budgeted-delivery/h1-dry-run.sh and c8k-dry-run.sh (keyless); keyless-gate.sh; live: h1-run.sh <root>, c8k-run.sh <root> Dry runs and gates $0 with dummy keys and hash vectors. Live runs paid; cost and time unknown ANTHROPIC, OPENAI, VOYAGE for live runs; SHOOTOUT_PROXY Invented fixture h1-fixture.ts for dry runs; sealed custody files for live runs; LongMemEval-S In-process (c8k-product.ts: gbrain PGLite engine and extraction from GBRAIN_DIR); HTTP stub or metering proxy; shell
evaluator eval/runner/evaluator/ (library, no entry point) Evaluator-side gold stores, input allowlists (leak checks) and independent reference scorers for LongMemEval and Cat13 None (imported by runners) $0 None Gold from corpus loaders (cat13-conceptual.ts, LongMemEval) None (pure)
failure causes eval/runner/failure-causes/cli.ts (library index.ts) Failure-cause taxonomy and classification of nightly and other eval misses bun eval/runner/failure-causes/cli.ts taxonomy [--write|--check]; classify --manifest <sources.json> --root name=<dir> --out <labels.ndjson> Deterministic classify $0. Residual labeller paid (model call, see label.ts) Residual labeller only (model via memory-qa/qa-v2.ts ChatLike, keys not traced) Source manifests with SHA-256 per file (adapters read nightly, skillopt, live-recall, refresh-barrier, rs1, cat40) None (reads archived rows)
graph backend (GBRA-123) eval/runner/graph-backend/run-tier.sh <tier> [stages], worker worker.ts, pg-load.ts, plan.ts, generate.ts, analyze.ts Latency and correctness of graph walks: gbrain CTE (A), breadth-first (B), PGLite (BL), HydraDB (C) over fictional link graphs GB_A=<wt> GB_B=<wt> OUT=<dir> bash eval/runner/graph-backend/run-tier.sh personal (stage list optional) $0 LLM. Latency receipts in docs/benchmarks/2026-10-17-graph-backend/{personal,team,company} and summary.json. Wall clock unknown None Generated fictional graphs (generate.ts, seeded) In-process engine methods from two gbrain worktrees (via GB_ENGINE_DIR); HTTP to HydraDB; spawned bun workers and Postgres

Runner details

cat40 model ladder and Hard

  • Entry file: eval/runner/cat40-model-ladder.ts. Helpers: cat40/ (arms.ts, loop.ts, gbrain-arm.ts, pg-arm.ts, hard.ts, hard-grep.ts, hard-ops.ts, hard-proxy.ts, rerank-pool-replay.ts, rerank-pool-proxy.ts, latency-replay.ts, firing.ts, analyze.ts, score.ts, rescore.ts, records.ts, types-filter.ts, reuse-check.ts, next-confirm.ts, dev-arms.ts, judge-hard.ts, arm_power.py, next_confirm_analysis.py). Wrappers: scripts/cat40-hard.sh and scripts/cat40-followups.sh (the only repo-level scripts for this family; no cat40 entry in package.json).
  • What it measures: whether gbrain's advantage over simpler memory (fs, fs-acl, memory, pg, oracle, gbrain; cat40/arms.ts:21-22) persists across models, on the world in eval/data/model-ladder-v1 (header of cat40-model-ladder.ts). Hard campaign adds a 50k held-out world with a primary endpoint of gbrain minus fs (scripts/cat40-hard.sh header).
  • How it runs: bun eval/runner/cat40-model-ladder.ts --models claude-sonnet-4-6,gpt-5.4 --arms oracle,fs,pg,memory,gbrain --budget-usd 100 [--families A,B] [--tasks A01] [--slots 3] [--gbrain <checkout>@<ref>] [--out eval/reports/cat40/<name>]. Hermetic: --scripted --arms fs,memory,oracle ($0). Keyless slots: --scripted --keyless-slots --arms gbrain. Build slots: --build-slots --gbrain-ref <sha> --slots 5 --budget-usd 10. Hard steps: scripts/cat40-hard.sh hello|status|preflight <step>|step <step>, ledger .budget/cat40-hard.sqlite.
  • Cost and wall time: protocol gives fs and pg cells about $0.05 and gbrain cells $0.02 to $0.30 (docs/benchmarks/2026-10-02-model-ladder-protocol.md:227). Hard runbook (docs/benchmarks/cat40-hard/RUNBOOK.md): freeze-check $114.61 (about 90 min), smoke $1.79 (about 20 min), slots-50k $12.04 (about 7.5 h), oracle-50k about 1 h, pg-50k about 5 h, memory-50k about 4 h. Ledger cap $1,794 (RUNBOOK line 11). Followups ledger cap $237 (scripts/cat40-followups.sh header).
  • LLM keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, PARTICLE_API_KEY (particle-1 through providers/openai-compatible.ts), VOYAGE_API_KEY (gbrain slot embeddings and rerank via cat40/gbrain-arm.ts). Offline: --scripted (hermetic, $0) and --scripted --keyless-slots (keyword-only slots, rerank probe skipped).
  • Corpus: eval/data/model-ladder-v1 and eval/data/model-ladder-wide-dev/world.json named in the header (not checked for presence in this checkout). Hard worlds are generated from seeds in the frozen knobs (cat40/ plus scripts/cat40-hard.sh).
  • gbrain imports:
    • ./gbrain-under-test.ts: gbrainSpecFrom, parseGbrainSpec (spec parsing only, cat40-model-ladder.ts:69).
    • ./cat40/gbrain-arm.ts: GbrainArm, GbrainFsArm, GbrainPool, GbrainSlot, MeteringProxy, McpClient, rerankProbe, restoreProbe, coverageProblem, instructionsOverride (cat40-model-ladder.ts:74, cat40/hard.ts:15).
    • gbrain methods reached over MCP (McpClient.call in cat40/gbrain-arm.ts): entity, search, context_pack, put_page (Hard only), get_page (cat40/hard.ts, cat40/gbrain-arm.ts).
    • cat40/rerank-pool-replay.ts:548: dynamic import of <buildDir>/src/core/search/rerank.ts (capRerankDoc) and PGlite from the build's node_modules, run inside bun -e.
  • Coupling: CLI+MCP. cat40/gbrain-arm.ts:93 spawns bun <buildDir>/src/cli.ts serve --surface starter (or --surface full) and speaks JSON-RPC over stdio (listTools, tools/call; :137). Slot snapshots and restores run as bun -e scripts that open PGlite directly from the build (gbrain-arm.ts:412, 522, 530, 561). cat40/pg-arm.ts is its own PGlite arm (tools vector_search, keyword_search, save_document, get_document) with no gbrain code. To run against another binary, the runner needs the serve command, the tools/list and tools/call names above, and the snapshot step replaced.
  • Parity-gate fit: deterministic only in --scripted mode. Paid cells use LLM judgments, so not deterministic. Scripted mode is fast.

cat41 agent operator

  • Entry file: eval/runner/cat41-agent-operator.ts. Helpers: cat41/types.ts, cat41/scenarios.ts (scenarios, SEED_FACTS, SEED_PAGES), cat41/brief-scenarios.ts (D7), cat41/classify.ts (per-step outcome classes and gate), cat41/transcript.ts (normalizes claude --output-format stream-json and codex exec --json), cat41/in-container.ts (driver inside the image), cat41/brief-db.ts, cat41/brief-d7.ts, cat41/brief-worlds.ts and brief-worlds.json, cat41/tthw-verdict.ts, cat41/after-pass.sh, cat41/Dockerfile (FROM node:24.18.0-bookworm-slim).
  • What it measures: whether real Claude Code and Codex CLI sessions, given scripted user requests, classify each step as authorized execution, required relay, correct refusal, successful recovery, consent violation, or false "no notes". Pins: claude_code 2.1.285, codex 0.160.0, bun 1.4.2 (cat41-agent-operator.ts PINS). Default models claude-opus-5-5 and gpt-6.1-sol.
  • How it runs: bun eval/runner/cat41-agent-operator.ts run --gbrain <checkout>@<ref> --label baseline [--harnesses claude,codex] [--scenarios all] [--repeat 3] [--concurrency 3] --paid --budget-usd <n> --out eval/reports/cat41/<label>. score --out <dir> ($0), overhead --gbrain <checkout>@<ref> --out <dir> ($0 token overhead of initialize and tools/list per surface), gate --before <dir> --after <dir>. Docs: docs/benchmarks/2026-10-03-agent-operator-protocol.md.
  • Cost and wall time: protocol says a three-repeat pass costs about $12 to $17 and about 35 min at concurrency 4 (docs/benchmarks/2026-10-03-agent-operator-protocol.md:183). Code reserves $2.5 (claude) and $1.5 (codex) per container run and estimates $0.5 and $0.2 (cat41-agent-operator.ts constants). CRASH_RETRIES = 2.
  • LLM keys: ANTHROPIC_API_KEY, OPENAI_API_KEY (inside the container; cat41/in-container.ts, cat41/scenarios.ts). Keyless modes: score, overhead, gate, and the D7 brief-d7.ts smoke ($0, fake provider).
  • Corpus: scenario and world data are generated in code (cat41/scenarios.ts, cat41/brief-worlds.ts). cat41/brief-worlds.json is committed (brief-worlds.ts --write regenerates it; the test checks it is current).
  • gbrain imports and calls:
    • Host: ./gbrain-under-test.ts (resolveGbrainUnderTest, overlaySummary); ./cat40/gbrain-arm.ts McpClient (cat41-agent-operator.ts:256, used for tools/list overhead); ./lifecycle/drivers.ts runCli (import in the file header).
    • Container, dynamic imports from /opt/gbrain/src in cat41/brief-db.ts:18-44: core/config.ts (loadConfig, toEngineConfig), core/engine-factory.ts (createEngine), mcp/resolve-ipc-binding.ts (turnContextWithBrief), core/context/proactive-brief.ts (turnBrief, briefEnabled, TURN_BRIEF_CUTOFF_MS). The engine methods called on the object from createEngine are not traced here.
    • Container CLI: cat41/in-container.ts:178-240 runs bun /opt/gbrain/src/cli.ts <args> as the agent user, and pgrep for the process.
  • Coupling: mixed. Host spawns docker (cat41-agent-operator.ts:93, kill on timeout at :96). Inside the container the real harness CLIs run gbrain via bun /opt/gbrain/src/cli.ts (CLI) and their MCP config (claude mcp add pattern in cat41/classify.ts:71). The host uses MCP stdio for overhead. The gbrain build is mounted read-only at /opt/gbrain.
  • Parity-gate fit: no. Paid runs are non-deterministic and slow. score, overhead, gate are deterministic and fast.

decision kit (decide.ts, decisions/, eval/decisions/)

  • Entry file: eval/runner/decide.ts, exposed as npm script eval:decide (package.json:23: bun eval/runner/decide.ts). Subcommands: init, fetch, preflight, dev, verdict, status, check. Held-out commands (power, prereg, seal-run) are documented as later milestones.
  • Helpers in decisions/: spec.ts (loadSpec, newSpec, validateSpec, types DecisionSpec, Source, Plan), splits.ts (dev/sealed split by whole conversation, ordered by SHA-256 of salt and conversation id), anchors.ts (dynamic import of queries/relational.ts: buildRelationalQueries, loadWorldCorpus; excludes dev probes that share an anchor entity with sealed questions), errors.ts (operator-message contract, DecideError, exitCodeFor).
  • Top-level data eval/decisions/: datasets/ (beam-10m, beam-b2da22e, blue-book-e494de6 at 146K, bg1-manifest, d8-manifest, lme-m-q1-selection, rs1-manifest), splits/ (beam-100k, beam-10m, beam-1m, beam-500k). Committed, total 268K.
  • What it measures: for each spec source, a paired, clustered comparison of a candidate gbrain checkout against a baseline (stats/gates.ts evaluateFamily), per source. Dev verdicts never flip a default (decide.ts header).
  • How it runs: bun run eval:decide init --plan P6 --gbrain ../gbrain@<sha>; fetch --decision <dir>; preflight --decision <dir>; dev --decision <dir>; verdict --decision <dir>; status. pinArm resolves <checkout>@<ref> to a full SHA and checks the dirty tree (decide.ts:50-56).
  • Cost and wall time: unknown. Each dev job is a child bun process running eval/runner/memory-qa/run.ts with a per-benchmark argv (decide.ts:184, spawn at :211). Costs are whatever those child runners spend (not traced into memory-qa).
  • LLM keys: depend on the sources. decide.ts references OPENAI_API_KEY (3 hits) and --paid (12 hits). Offline: preflight, status, check, fetch --verify-only are keyless.
  • Corpus: eval/decisions/datasets/*.json (committed) with SHA-checked downloads through memory-qa/corpus.ts fetchDataset/filesFor (outside scope).
  • gbrain imports: ./gbrain-under-test.ts parseGbrainSpec, resolveGbrainUnderTest only (no engine calls in decide.ts). decisions/anchors.ts imports the world's relational queries from the runner tree, not gbrain.
  • Coupling: spawn (git subprocesses at decide.ts:53-54; bun child runners per arm and shard). The child runners own the gbrain coupling (memory-qa, outside this scope).
  • Parity-gate fit: check, status, preflight, init are deterministic and keyless. dev is paid and not a parity gate.

evidence delivery (evidence-delivery.ts, evidence-delivery/)

  • Entry file: eval/runner/evidence-delivery.ts. Helpers in evidence-delivery/: gbrain.ts (loader and identity), data.ts (dataset, pilot split, cluster map), freeze.ts, store.ts (content-addressed evidence blobs), arms.ts, e1.ts, e2-bridge.ts, e3.ts, calls.ts, parity.ts, power.ts, decision.ts, decision-v2.ts, ledger-preload.ts.
  • What it measures: E1 delivery ablation (which evidence packing reaches the reader best), E3 product-path parity (does the harness deliver what gbrain's product path would), and E2 sealed bridge, all on LongMemEval-S under a preregistered decision rule (docs/benchmarks/2026-09-30-evidence-delivery-preregistration.md).
  • How it runs: keyless bun eval/runner/evidence-delivery.ts cluster-map --dataset <json>, power [--sims N], parity --frozen-dir <dir> --dataset <json>, analyze --rows-dir <dir> --stage pilot|confirmatory, costs. Paid, always through the budget ledger: campaign-open --budget-usd <= 400, freeze --dataset <json> --out-dir <dir>, e1 ... --arms a,b, e3 ... --score, e2-freeze, e2-answer, e2-score. --smoke runs open their own run capped at $3. GBRAIN_DIR (or --gbrain-dir) names the checkout.
  • Cost and wall time: campaign cap $400 (campaign-open), smoke cap $3 (SMOKE_CAP_USD = 3). Per-step wall time unknown. estimateE1Usd in e1.ts gives a planning estimate.
  • LLM keys: OPENAI_API_KEY and VOYAGE_API_KEY (e3.ts, calls.ts); readers are Sonnet (SONNET in e1.ts) and GPT-4o (GPT4O); judges via gbrain's framed judge and the official LongMemEval judge (calls.ts). BRAINBENCH_BUDGET_RUN_ID is required by ledger-preload.ts.
  • Corpus: longmemeval_s_cleaned.json (passed as --dataset, not committed here). R1 receipts at docs/benchmarks/2026-09-29-longmemeval-opaque-qa/reranker-on/r1 (R1_DIR). Frozen evidence goes to a content-addressed store.
  • gbrain imports (evidence-delivery/gbrain.ts:85-100, loadGbrain, dynamic import(join(dir, p))):
    • src/eval/longmemeval/harness.ts (createBenchmarkBrain, resetTables), adapter.ts (haystackToPages), metrics.ts (sessionIdFromSlug, buildSlugToRawMap, rawSessionId), src/core/import-file.ts (importFromContent), src/core/search/hybrid.ts (hybridSearch), src/eval/shared/embed-cache.ts (EmbeddingCache, installEmbedCache), src/eval/longmemeval/reader.ts (buildReaderUserText, READER_NOTES_SYSTEM_TEXT, READER_MAX_SESSION_CHARS), sanitize.ts (renderChatBlock), src/core/chunkers/token-estimate.ts (estimateTokens), src/core/ai/gateway.ts (configureGateway, chat), src/core/ai/build-gateway-config.ts (buildGatewayConfig), src/eval/longmemeval/judge.ts (judgeRow), src/eval/shared/judge-runner.ts (BudgetLedger), src/core/search/evidence-delivery.ts (assembleEvidenceForHits, evidenceFingerprint, required by --requireEvidence).
    • Engine calls: evidence-delivery.ts:399-412 dynamic import of src/core/pglite-engine.ts (PGLiteEngine), then engine.connect, engine.initSchema, engine.disconnect. freeze.ts:66-86 calls engine.getConfig, setConfig, unsetConfig; freeze.ts:183 calls engine.getChunks.
    • Shared budget-ledger.ts and receipt.ts are outside scope.
  • Coupling: in-process for the gbrain pipeline (dynamic import of GBRAIN_DIR modules; engine on PGLite). MCP stdio for E3 product-path only: e3.ts:169-170 builds McpStdioDriver with entry bun --preload ledger-preload.ts <gbrainDir>/src/cli.ts, so gbrain's own serve path is measured. To run against a binary, the parity checks need the MCP tool list and results, and the freeze step needs a JSON adapter for getChunks, hybrid search and assembleEvidence.
  • Parity-gate fit: keyless subcommands (cluster-map, power, parity, analyze, costs) are deterministic and fast. Paid steps are not.

external ideas wave A (external-ideas-wave-a.ts, external-ideas-wave-a/)

  • Entry file: eval/runner/external-ideas-wave-a.ts. Helpers: external-ideas-wave-a/answer.ts (one resumable answer step over the batch lane), lib.ts (study manifest, affected set, analysis, bracket audit), render-arms.ts (eval-only r1_rank_date and r2_tail_trim renderings built from the frozen auto delivery), synthetic-dates.ts (seeded synthetic relative-date set, LongMemEval format). Also imports evidence-delivery.ts (Args, evalsCommit, writeJson).
  • What it measures: whether gbrain's search.evidence_resolve_dates flag, which adds absolute date brackets to relative phrases in delivered turns, changes LongMemEval-S answer accuracy. Arms auto (flag off) and auto_dates (flag on) are delivered by gbrain's assembleEvidenceForHits on the same frozen hits. Study manifest docs/benchmarks/2026-10-21-external-ideas-wave-a/study-manifest.json.
  • How it runs: freeze --dataset <json> --out-dir <dir> --budget-usd <cap> [--smoke], affected --frozen-dir <dir> --dataset <json>, answer --frozen-dir ... --arm auto --sample 1 --model <id> --set all --out-dir ... --budget-usd <cap> [--wait], analyze, verdict, audit, estimate --dataset <json> (keyless), a2-verdict. GBRAIN_DIR names the checkout.
  • Cost and wall time: lane ledger $220 across waves A and B (GBRA162_LEDGER, default ~/.capy/work/gbra162-external-ideas/ledger.sqlite). A run over $50 needs --gbra108-approval. Exit codes: 0 done, 1 failed, 2 preflight refused, 4 budget stop. Wall time unknown. estimate gives the planning figure without keys.
  • LLM keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, VOYAGE_API_KEY (external-ideas-wave-a.ts references). Keyless: estimate, affected, render-arms, synthetic-dates, analyze, verdict (given analysis files).
  • Corpus: LongMemEval-S (--dataset, not committed here) and the synthetic dates generated by synthetic-dates.ts. Frozen evidence in a content-addressed store (evidence-delivery/store.ts).
  • gbrain imports:
    • ./evidence-delivery/gbrain.ts: configureGbrainGateway, gbrainIdentity, loadGbrain, resolveGbrainDir (the same dynamic loader as evidence delivery, gbrain.ts:85-100).
    • ./evidence-delivery/e1.ts: rendererFrom. ./evidence-delivery/calls.ts: gbrainJudge, officialJudge. ./evidence-delivery/freeze.ts: freeze.
    • Engine: brain.disconnect (external-ideas-wave-a.ts, one call); assembleEvidenceForHits and evidenceFingerprint from the loaded evidence module.
    • ./batch/submit.ts BatchLane, ./batch/transport.ts (anthropicTransport, openAiTransport), ./batch/manifest.ts (buildManifest, freezeManifest), ./batch/sources.ts (readerBody, seededSample, JUDGE_PROTOCOL).
  • Coupling: in-process (gbrain src/ via loadGbrain), plus HTTP to provider batch APIs. No CLI or MCP.
  • Parity-gate fit: estimate, affected, analyze, verdict deterministic. answer and freeze are paid and slow.

facts absorb gate (facts-absorb-gate.ts, facts-absorb/)

  • Entry file: eval/runner/facts-absorb-gate.ts. Helpers: facts-absorb/score.ts (scoreArm, decide, valueToken; pure, matches stored facts to the planted claims), facts-absorb/natural.ts (natural-prose stratum; the runner calls the Cat 35 coverage judge scoreSalienceCoverage from cat35-judges.ts).
  • What it measures: whether a cheaper facts.extraction_model keeps the facts gbrain's background extractor stores, measured as recall, precision, attribution and correction handling against an answer key. Default arm anthropic:claude-sonnet-4-6 (SHIPPED_DEFAULT_MODEL). Preregistered arms in DEFAULT_ARMS: sonnet46, luna (openai:gpt-6-luna), haiku55 (anthropic:claude-haiku-5-5), disabled (mutant), drop (mutant that replaces every chat output with {"facts":[]}). Preregistration docs/benchmarks/2026-10-08-facts-extraction-model/PREREGISTRATION.md.
  • How it runs: bun eval/runner/facts-absorb-gate.ts --gbrain <checkout>@<ref> --arms sonnet46=default,luna=openai:gpt-6-luna,haiku55=anthropic:claude-haiku-5-5,disabled=disabled,drop=drop:openai:gpt-6-luna --seeds 60,61 --prereg <path> --sample-reads 30 --budget-usd <n> --out <dir>. Paid arms refuse to start without --prereg. --smoke exists.
  • Cost and wall time: --budget-usd is required and the ledger is startPaidRun. Actual cost and time unknown. The job drain step gets a 3 h timeout (facts-absorb-gate.ts:211).
  • LLM keys: anthropic and openai keys for each arm, through MeteringProxy (cat40/gbrain-arm.ts). drop: and disabled arms are mutants. Gbrain's own call ledger is read from GBRAIN_AI_CALL_LOG to get the resolved model.
  • Corpus: generated by eval/generators/facts-absorb-gen.ts (generateFactsAbsorbWorld, renderSessionNote) from seeds; natural prose from Cat 35 transcripts.
  • gbrain imports and calls:
    • ./cat40/gbrain-arm.ts: GbrainSlot, McpClient, MeteringProxy, GBRAIN_EMBED_MODEL. McpClient.call('put_page', { slug, content }) at :201 and :205.
    • ./lifecycle/drivers.ts runCli, ./lifecycle/builds.ts prepareBuild (copied build).
    • CLI operations via runCli: call recall (:171), jobs work --concurrency N (:211, 3 * 3600000 ms timeout), slot-level args (:185). The script at :103 runs bun -e <script>.
    • Server: new McpClient(slot.run, ['--surface', 'full']) (:196), i.e. gbrain serve --surface full over stdio.
  • Coupling: MCP stdio (put_page, recall via serve) and CLI (gbrain call, gbrain jobs work). The gbrain build runs as a copied overlay (prepareBuild). To run a different binary the runner needs the serve tool names (put_page), call recall, and jobs work.
  • Parity-gate fit: score.ts is deterministic and fast. The gate itself is paid and slow (the writes, background job drain, and LLM extraction).

lifecycle experiment (lifecycle-experiment.ts, lifecycle/)

  • Entry file: eval/runner/lifecycle-experiment.ts. Helpers in lifecycle/: drivers.ts (CLI, MCP stdio, MCP HTTP drivers and runCli), slice.ts (SliceCell, one cell for the eval-category wave; callers n1-knowledge-update.ts, n5-forget-residue.ts, n6-postgres-http.ts, p5-brain.ts at runner root), builds.ts (prepareBuild, assertBuildVerified, copied builds), make-variant-build.ts (makeVariantBuild, patch-based variant), observe.ts (observe, probeLeaks), scenario.ts (ground truth identities, facts, phases, checkpoints), score.ts (scoreCheckpoint, scoreBijection, scoreEdges, scoreTimeline, scoreFacts), n1-score.ts, n5-score.ts, fake-embedder.ts (startFakeEmbedder, local HTTP hash embedder), n1-score.ts and n5-score.ts for the N1 and N5 categories.
  • What it measures: correctness of a memory lifecycle per cell (build x engine x interface): ingest, query, embedding outage during ingest, correct, reconcile, forget, restart, query again. Ground truth is the evaluator's own ledger; gbrain's doctor and integrity checks are never consulted (header).
  • How it runs: bun eval/runner/lifecycle-experiment.ts --gbrain-repo ../gbrain [--builds a=6bb88d128,b=origin/master] [--engines pglite,postgres] [--interfaces cli,mcp-stdio,mcp-http] [--pg-url postgres://user@host:port/db] [--concurrency 4] [--out docs/benchmarks/<dir>/receipt.json]. Render receipts with lifecycle-report.ts.
  • Cost and wall time: $0. Hermetic, with a local hash embedder (fake-embedder.ts) and provider keys stripped from child env (header). Wall time unknown (many child processes, Postgres optional).
  • LLM keys: none. The runner strips OPENAI_API_KEY and ANTHROPIC_API_KEY from each child (header and lifecycle/drivers.ts). Offline by design.
  • Corpus: generated in code by lifecycle/scenario.ts (no dataset file).
  • gbrain imports: none in-process. The only library import is SQL from bun (lifecycle-experiment.ts:23, lifecycle/slice.ts:31) for Postgres. All gbrain access is through spawned bun <build>/src/cli.ts processes.
  • Exact process mechanics, from lifecycle/drivers.ts and lifecycle/builds.ts:
    • Build: git clone -q --no-checkout and git archive <commit> | tar -x into a temp dir (builds.ts:62, 117), bun install --frozen-lockfile in that dir (:118), then bun src/cli.ts --version with GBRAIN_SKIP_STARTUP_HOOKS=1 (:128). The build is a copy, not a symlink, and the cmd checks git write-tree against the commit tree (:91-92).
    • CliDriver (drivers.ts, runCli): spawn('bun', [<build>/src/cli.ts, 'call', <op>, <json>], { env, cwd: GBRAIN_HOME }) per call, stdin.end(), 180 s timeout. Trusted local caller.
    • McpStdioDriver: spawn('bun', [<build>/src/cli.ts, 'serve']) once per session. Writes JSON-RPC lines to stdin: initialize (protocolVersion 2025-06-18), notifications/initialized, then tools/call with { name: op, arguments } per call. Reads newline-delimited responses on stdout. Restart kills and respawns. 15 s SIGKILL on close.
    • McpHttpDriver: spawn('bun', [<build>/src/cli.ts, 'serve', '--http', '--port', <port>]), then McpHttpClient fetches a token from http://127.0.0.1:<port>/token with a form-encoded POST (drivers.ts:101-110), then POSTs JSON-RPC to http://127.0.0.1:<port>/mcp with mcp-session-id header (drivers.ts:241-243). 180 s timeouts.
    • Operator steps (host-side maintenance): op('init') plus sources, sync, extract, embed through runCli. Operations observed via call: remember, forget, get_page, search, recall, list_pages, get_timeline, get_links, query.
  • Coupling: CLI and MCP (stdio and HTTP) through spawned child processes. Postgres through Bun SQL. Not in-process for gbrain.
  • Parity-gate fit: $0 and keyless, so it fits a parity gate on time budget. Spawns are many and slow, so wall time is the issue. Receipts are deterministic given fixed seeds.

lifecycle-lite (lifecycle-lite.ts, lifecycle-lite/)

  • Entry file: eval/runner/lifecycle-lite.ts. Helpers: lifecycle-lite/run.ts (drives any MemorySystem: HTTP shim, in-process gbrain-shootout, or fake; writes rows and receipt), lifecycle-lite/score.ts (pure scoring, gold from generator ledger), lifecycle-lite/fakes.ts (honest reference and mutation fakes, HonestReferenceSystem, mutationFake).
  • What it measures: update and forget behavior (N1 correction chains, N5 canaries) for open-source memory systems, report-only. Gold from eval/generators/lifecycle-lite-gen.ts (LIFECYCLE_LITE_SEEDS, sanitizerCorpus).
  • How it runs: bun eval/runner/lifecycle-lite.ts --system <shim URL>|gbrain-shootout|fake|honest-reference|mutation:<kind> [--seeds 1,2,3,4,5] [--policy fixed-evidence|vendor-default] [--restart [--restart-cmd '<shell>']] [--gbrain <checkout>@<ref>] [--embed hash|real] [--qa reader|scripted] [--paid --budget-run-id <id>] --output <dir>. Default is hash embedder and scripted QA (keyless).
  • Cost and wall time: hermetic by default. --paid --budget-run-id only for gbrain-shootout (reranker and real embeddings), reader and judge. Timeouts --finish-timeout-s 600, --ingest-timeout-s 3600, --restart-timeout-s 900. Actual cost unknown.
  • LLM keys: none for hermetic mode. OPENAI_API_KEY, ANTHROPIC_API_KEY, VOYAGE_API_KEY for --embed real, --qa reader, and the gbrain-shootout reranker.
  • Corpus: generated seeds (no dataset file).
  • gbrain imports and calls:
    • In-process gbrain-shootout (lifecycle-lite.ts:107-119): importGbrain(gut, 'src/core/ai/gateway.ts') then configureGateway(...), __setEmbedTransportForTests (test hook). ./systems/gbrain.ts GbrainShootoutSystem and GbrainModules type.
    • systems/gbrain.ts (outside scope) calls on the engine: engine.connect, engine.initSchema, e.setConfig, e.now, engine.executeRaw, engine.deletePage, engine.disconnect (line numbers 67-122).
    • lifecycle-lite.ts:118: Bun.resolveSync('ai', gut.root) then dynamic embedMany from the build's ai package.
    • ./gbrain-under-test.ts resolveGbrainUnderTest, importGbrain, productIdentityFor, overlaySummary.
  • Coupling: in-process for gbrain-shootout. HTTP protocol v1 for shims (MemorySystem over HTTP). Restart: Bun.spawn(['bash', '-c', cmd]) (lifecycle-lite.ts:191), e.g. bash eval/systems/bootstrap.sh restart --system <name>. Mutation fakes are in-process.
  • Parity-gate fit: the hermetic default with --system honest-reference|fake|mutation:* is deterministic and fast. gbrain-shootout is not keyless with real embeddings.

lifecycle-report (lifecycle-report.ts)

  • Entry file: eval/runner/lifecycle-report.ts. Reads one or more lifecycle-experiment receipts (readFileSync) and prints Markdown tables per cell (CellSummary shape: ack_lost, bijection, edges, timeline, forget, residue, leaks, hazards). With two or more receipts it reports whether each cell's summary matched across runs.
  • What it measures: nothing new. It summarizes receipts.
  • How it runs: bun eval/runner/lifecycle-report.ts <receipt.json> [more...].
  • Cost and time: $0, under a second.
  • LLM keys: none. Corpus: receipts only.
  • gbrain imports: none. Coupling: none.
  • Parity-gate fit: deterministic and fast.

LHF wave 2 attendance gate (lhf-wave-2-attendance-gate.ts)

  • Entry file: eval/runner/lhf-wave-2-attendance-gate.ts. No helpers.
  • What it measures: scores the LHF wave 2 attendance paraphrase gate from two N9 hermetic receipts (base and branch) on world-v1-attendees, with extra splits heldout-paraphrase and competing-intent. Counts fired attendance and false routes (attended link type routed on competing or other-template questions). Pre-registration docs/benchmarks/2026-10-16-lhf-wave-2-attendance-preregistration.md. Counts only, no statistics.
  • How it runs: bun eval/runner/lhf-wave-2-attendance-gate.ts --base <receipt.json> --branch <receipt.json> --base-relations <json> --branch-relations <json> [--json]. Receipts come from N9 runs with --extra-split heldout-paraphrase=<jsonl> --extra-split competing-intent=<jsonl>. Relations come from eval/runner/parse-relational-questions.ts (outside this scope, runs against each gbrain tree).
  • Cost and time: $0 and under a second. LLM keys: none (the N9 receipts have already run).
  • Corpus: N9 receipt JSON (data.one_hop.per_query).
  • gbrain imports: none. Coupling: none (reads JSON). The gbrain dependency is one level up, in parse-relational-questions.ts run against each tree.
  • Parity-gate fit: deterministic and fast.

batch lane (batch/)

  • Entry files: batch/w10.ts (build, plan, run per arm), batch/w10a-capture.ts (gbrain's own eval longmemeval at the pin with a stub reader that records each reader request), batch/w10-particle.ts (particle-1 replay of recorded W10 prompts), batch/w10-particle-second-judge.ts (second judge sensitivity), batch/w10-rescore.ts (keyless $0 rescore from committed rows).
  • Library: submit.ts (BatchLane, exactly-once submission in a SQLite ledger via bun:sqlite: intent, submit, settle), transport.ts (OpenAI Batch /v1/files and /v1/batches, Anthropic /v1/messages/batches, and each provider's token-count endpoint), ledger.ts (worst-case reservation, batch factor from DOCUMENTED_BATCH_FACTOR), manifest.ts (buildManifest, freezeManifest, bodySha, MODEL_SETTINGS), sources.ts (request builders for W10 and R1/R2 replay, readerBody, seededSample, JUDGE_PROTOCOL, officialJudgePrompt), receipts.ts (offline scoring; rows join by question id), real-fetch.ts (keeps the raw fetch before guards).
  • What it measures: LongMemEval answer accuracy for several reader arms (Opus 5.5, Sonnet 5.5, GPT-6.1 Sol, particle-1), paired question by question, through provider batch APIs at a discount.
  • How it runs: bun eval/runner/batch/w10.ts build <arm> ($0), plan <arm> ($0, counts tokens), run <arm> (paid pilot, then the rest). bun eval/runner/batch/w10-rescore.ts ($0 from committed rows). Particle: w10-particle.ts build ($0) then preflight ... --paid --budget-run-id <id>.
  • Cost and wall time: reservations are worst case from counted input tokens, then settled from provider usage (ledger.ts). Total cost and time unknown.
  • LLM keys: ANTHROPIC_API_KEY and OPENAI_API_KEY (transport.ts, w10-particle-second-judge.ts). Keyless: w10-rescore.ts, build and plan modes.
  • Corpus: LongMemEval-S, and R1/R2 reranker-on call logs (batch/sources.ts, reranker-on/r1|r2/calls.ndjson.gz, under docs/benchmarks/2026-09-29-...).
  • gbrain imports (all relative to eval/runner/batch/, so they resolve to the pinned package at repo-root node_modules/gbrain, not to a checkout under test):
    • batch/w10.ts:34: sessionIdFromSlug from node_modules/gbrain/src/eval/longmemeval/metrics.ts.
    • batch/w10a-capture.ts:19-21: configureGateway, buildGatewayConfig (src/core/ai/gateway.ts, build-gateway-config.ts) and runEvalLongMemEval (src/commands/eval-longmemeval.ts). The call at :57 passes a client that records requests.
    • batch/sources.ts:23-26: haystackToPages (adapter.ts), renderChatBlock (sanitize.ts), sessionIdFromSlug (metrics.ts), buildReaderRequest, READER_MAX_SESSION_CHARS, resolveReaderConfig (reader.ts).
    • batch/w10-particle.ts:124: dynamic import of src/eval/longmemeval/{adapter,sanitize,metrics,reader}.ts from root.
  • Coupling: in-process against the pinned gbrain package (not an overlay), HTTP to provider batch APIs, SQLite for the lane ledger. Not CLI or MCP.
  • Parity-gate fit: w10-rescore.ts and build/plan deterministic and keyless. run is paid.

BG1 battleground (bg1/)

  • Entry: bg1/verdict.ts (verdict --receipts <dir>... [--manifest <filled>] [--judge-check <summary>] --out <dir>), bg1/power.ts (MDD and sizing), bg1/pricing.ts (--out <file> [--n N] [--rates <json>], money model), bg1/judge-check.ts (judge validity on public dev data), bg1/launch.ts (pre-launch lease and budget check).
  • Helpers: bg1/rule.ts (decision rule: paired t, Holm step, BG1_ALPHA, decideFamily), bg1/match.ts (deterministic value matcher matchAnswer, no model), bg1/exposure.ts (exposure(q, contextText, { rawText }): gold, stale, deciding line), bg1/types.ts (re-exports memory-qa/bg1-names.ts).
  • What it measures: BG1 changing-facts battleground: gbrain against comparators (Opus, Sonnet, particle; q1/cells/bg1.ts names). Preregistration docs/benchmarks/2026-11-09-changing-facts-battleground-preregistration.md. Exposure tells whether a loss was retrieval or reading.
  • How it runs: bun eval/runner/bg1/verdict.ts verdict --receipts <dir>... --out <dir>; bun eval/runner/bg1/pricing.ts --out <file>; bun eval/runner/bg1/power.ts; judge-check (paid, public dev data, gpt-6-astra and claude-opus-5-5 as judges).
  • Cost and time: pricing.ts gives the per-world money model and check-in projection. Actual campaign cost unknown here. judge-check plan is JUDGE_CHECK_PLAN in pricing.ts.
  • LLM keys: judge-check uses ANTHROPIC_API_KEY, OPENAI_API_KEY, PARTICLE_API_KEY, VOYAGE_API_KEY, and SHOOTOUT_PROXY (bg1/judge-check.ts). Analysis modules (verdict, power, pricing, match, exposure, rule, launch) need no key.
  • Corpus: BG1 manifest eval/decisions/datasets/bg1-manifest.json (committed, 1.2K), memory-qa/corpus.ts loadCorpus for dev data, cell manifest q1/cells/bg1.ts.
  • gbrain imports: none direct. judge-check.ts imports memory-qa/qa-v2.ts (ChatClient, judgeAnswer), metering-proxy.ts (MeteringProxy, keylessUpstream), and q1/cell.ts (metering, onSlot), which are outside scope. Coupling to gbrain sits in the q1 cells, not in bg1.
  • Coupling: none for analysis. HTTP to judge models through the metering proxy for judge-check. launch.ts reads the cell manifest and asserts leases; it does not spawn or call gbrain.
  • Parity-gate fit: verdict, power, pricing, match, exposure are deterministic and fast. judge-check is paid and not a gate.

budgeted delivery (budgeted-delivery/)

  • Entry points: shell h1-run.sh (custody chain for H1 held-out), c8k-run.sh (8k conversion custody and dev reading), e2-keyless-gate.sh (E2 gate end to end, hash vectors, reranker off, stub proxy), keyless-gate.sh (E1 accounting gate for one benchmark), h1-dry-run.sh and c8k-dry-run.sh (whole chain on invented fixtures), c8k-amend-7.sh (operator step for a spec change), e2-parallel.sh (memory-qa shards). TypeScript steps: h1.ts, c8k.ts, c8k-product.ts, c8k-retrieval-cache.ts, e1-readings.ts, e2-readings.ts, e2-gate-check.ts, e3-retrieval-gate.ts, accounting-gate.ts, budget-sizing.ts, drop-order.ts, export-receipts.ts, h1-fixture.ts, stub-proxy.ts.
  • What it measures: whether evidence packed under explicit token budgets (off, cap_only, and packing arms) keeps reader accuracy. E1: arm scores and readings. E2: every packing against off and cap_only on LongMemEval-S 500. E3: session fusion against today's ranking (strict recall). 8k conversion: a held-out decision on an 8k context budget. H1: held-out custody run. Preregistrations in docs/benchmarks/2026-10-08, 2026-10-09, 2026-10-10, 2026-10-26.
  • How it runs: keyless bash eval/runner/budgeted-delivery/keyless-gate.sh, bash .../h1-dry-run.sh, bash .../c8k-dry-run.sh, bash .../e2-keyless-gate.sh. Live custody: bash eval/runner/budgeted-delivery/h1-run.sh <custody root> and c8k-run.sh <custody root> (run on the custodian's machine from the repo root at the registration commit). Everything sealed stays under the custody root; only <root>/export/ and the access log leave it.
  • Cost and wall time: dry runs and gates are $0 (dummy keys, hash vectors, stub proxy; their numbers mean nothing). budget-sizing.ts is $0. Live costs and time: unknown (leases per proxy in the shell scripts).
  • LLM keys: live runs need ANTHROPIC_API_KEY, OPENAI_API_KEY, VOYAGE_API_KEY (c8k.ts, h1-dry-run.sh sets dummy values). SHOOTOUT_PROXY is set for proxy leases in h1-run.sh and c8k-run.sh.
  • Corpus: h1-fixture.ts invents twelve personas with LongMemEval-style timestamps (no sealed text). Live runs use sealed confirmation files under the custody root. LongMemEval-S for E1, E2 and E3.
  • gbrain imports and calls:
    • c8k-product.ts:58-83: dynamic load of src/core/pglite-engine.ts (PGLiteEngine), src/core/import-file.ts (importFromContent), a tool dispatcher (dispatchToolCall) and runExtractConversationFactsCore (fact extraction core). c8k-product.ts:102: new PGLiteEngine(), then e.connect, e.initSchema, e.disconnect (c8k.ts also uses e.disconnect).
    • c8k.ts:37-39: importGbrain, resolveGbrainUnderTest (gbrain-under-test.ts).
    • ./systems/gbrain.ts sourceOfSlug and slugOf in e3-retrieval-gate.ts, h1.ts, c8k.ts, c8k-product.ts; ./systems/gbrain-query/connector.ts wireRequest and QUERY_PATH_PINS (outside scope).
    • e3-retrieval-gate.ts, h1.ts: sourceOfSlug only.
  • Coupling: in-process (c8k-product on a PGLite brain from GBRAIN_DIR), HTTP through stub-proxy.ts (keyless) or the metering proxy in live runs, shell orchestration. accounting-gate.ts consumes gbrain-query rows through wireRequest and needs no gbrain code to be loaded.
  • Parity-gate fit: dry runs and gates deterministic, keyless, and useful as a fast parity check. Live steps are paid.

evaluator (evaluator/)

  • No entry point. Library imported by runners. Files: gold-store.ts (labels in a JavaScript private field, returns copies), allowlist.ts (boundary allowlists and leak checks by known substring), judge-inputs.ts (input allowlists for the reading-notes reader, Cat 29 answerer and judge, Cat 35 SUT and judges), reference-scorer.ts (independent metric implementations; no import from the product), longmemeval.ts (separate gold loader, gold-free view, allowlists; imports type Question from ../longmemeval.ts), cat13.ts and cat13-gold-loader.ts (Cat 13 gold from buildProbes, loadCorpus, PROBE_SEED in ../cat13-conceptual.ts).
  • What it measures: nothing. It is the evaluator-side boundary for "no leakage of gold into the system under test" and independent scoring.
  • How it runs: imported. Tests are the phase gates.
  • Cost and time: $0. Keys: none. Corpus: loaded from corpus loaders (cat13-conceptual.ts, LongMemEval dataset).
  • gbrain imports: none (no gbrain import, no spawn). Coupling: none. Pure.
  • Parity-gate fit: deterministic and fast.

failure causes (failure-causes/)

  • Entry file: failure-causes/cli.ts. Library: index.ts (stable interface for the nightly loop and weekly report, re-exports taxonomy).
  • Files: taxonomy.ts (fixed list of causes in stage order, mapped to gap-map failure modes F1 to F16; renderTaxonomyMarkdown), classify.ts (deterministic rules by stage; returns primary cause and secondaries), label.ts (residual labeller: a model picks a cause only where the rules cannot place the failure; uses client.chat with ChatLike from memory-qa/qa-v2.ts and judge.ts's fenceUntrusted), agreement.ts (labeller agreement check), attribution.ts (nightly QA miss buckets: harness, item unsupported, and others), report.ts (weekly summary: gbrain's top three ranked on natural-condition failures), manifest.ts (source manifests with SHA-256 per named input file, read from named roots), types.ts (EvalFailure, CauseLabel, StressCount), adapters/ (nightly.ts, skillopt.ts, live-recall.ts, refresh-barrier.ts, rs1.ts, cat40.ts, shared.ts, index.ts).
  • What it measures: the distribution of causes for eval misses across sources, to rank what gbrain can fix. Not a benchmark of gbrain itself.
  • How it runs: bun eval/runner/failure-causes/cli.ts taxonomy [--write|--check]; bun eval/runner/failure-causes/cli.ts classify --manifest <sources.json> --root <name>=<dir> ... --out <labels.ndjson> [--failures-out ...] [--stress-out ...] [--counts-out ...].
  • Cost and time: deterministic classify is $0. The residual labeller is paid (model calls via client.chat). Agreement check uses two labellers at high effort (Opus 5.5 and GPT-6 Astra, per the file header). Wall time unknown.
  • LLM keys: labeller only, through the ChatLike client. The key names are not set in these files; the client is supplied by the caller (not traced here).
  • Corpus: archived rows named in source manifests (nightly, skillopt, live-recall, refresh-barrier, rs1, cat40 adapters). Files are read by name and SHA-256, never by glob.
  • gbrain imports: none. adapters/cat40.ts imports only node:path. Coupling: none. The adapters read rows; they do not call gbrain.
  • Parity-gate fit: taxonomy --check and classify (without labeller) are deterministic and fast.

graph backend (graph-backend/)

  • Entry: graph-backend/run-tier.sh <tier> [stages...] (stages: load-pg plan hydra-load calibrate arm-A arm-B arm-BL arm-C tput). Workers: worker.ts --arm A|B|BL|C --tier <tier> --pass <warm|cold> --plan <json> --out <ndjson>. Also pg-load.ts, hydra-load.ts, plan.ts, generate.ts, analyze.ts (scores receipts, decision rule), canon.ts (canonical digests of walk results), hydra.ts (HydraDB HTTP client), stack.sh, ubicloud-setup.sh (infrastructure).
  • What it measures: latency and correctness of graph walks on fictional link graphs (traverseGraph, traversePathsDetailed) across arms: A (gbrain old CTE, worktree at b1fdcb843), B (gbrain breadth-first, worktree at fc1121cae58, Postgres), BL (B on PGLite), C (HydraDB, hydrated from Postgres). GBRA-123. Tiers personal, team, company (docs/benchmarks/2026-10-17-graph-backend/; summary.json generated 2026-10-18).
  • How it runs: GB_A=<gbrain worktree> GB_B=<gbrain worktree> OUT=<dir> bash eval/runner/graph-backend/run-tier.sh <tier> [stage...]. Stages are resumable with .done markers. Start counts: A_STARTS (default 100 at team and company), STARTS (default 500). Each start page gets a 22-call matrix.
  • Cost and time: no LLM. Latency data in the summary (for example personal pooled arm A/cold, p50 1.7 and p95 278 over 11,000 calls, units as recorded in summary.json). Wall clock for stages unknown.
  • LLM keys: none. External services: HydraDB (HYDRA_URL, HYDRA_TOKEN, HYDRA_VERIFY, HYDRA_LOAD_CONCURRENCY, HYDRA_CLIENT_CONCURRENCY) and a local Postgres admin URL for loading (run-tier.sh).
  • Corpus: fictional seeded graphs from generate.ts (power-law hubs, page ids equal slug order). Generated, not committed as data (the plan and receipts are committed under docs/benchmarks).
  • gbrain imports and calls (in-process, from GB_ENGINE_DIR worktrees):
    • worker.ts:41, 45: dynamic import of ${engineDir}/src/core/pglite-engine.ts (PGLiteEngine) and ${engineDir}/src/core/postgres-engine.ts (PostgresEngine). Methods called: engine.connect, engine.traverseGraph, engine.traversePathsDetailed, engine.disconnect.
    • pg-load.ts:17, 21: the same two engine modules. Methods: engine.initSchema, engine.connect, engine.disconnect. Then runs ANALYZE.
    • analyze.ts:37: dynamic import of ${GB_B}/vendor/postgres/src/index.js (vendored Postgres client, for queries).
  • Coupling: in-process for gbrain walks (engine modules from two worktrees, selected by env, not by a CLI). HTTP to HydraDB (hydra.ts, hydra-load.ts). Spawns bun worker.ts from run-tier.sh. A CLI or MCP adapter would replace traverseGraph and traversePathsDetailed with JSON calls.
  • Parity-gate fit: deterministic (fixed seeds, call matrix). Needs Postgres and worktrees, so not a fast keyless gate.

Cross-cutting observations

  • Coupling is mostly in-process via dynamic import(join(GBRAIN_DIR, 'src/...')) (evidence-delivery, external-ideas, lifecycle-lite, budgeted-delivery c8k-product, graph-backend workers, cat41 container driver). The loader evidence-delivery/gbrain.ts loadGbrain (lines 85-100) is the shared entry point, and its module list is the de facto contract for every in-process study.
  • Batch lane imports point at repo-root node_modules/gbrain (batch/w10*.ts, batch/sources.ts), which is the pinned package and not an overlay under test. These results describe the pin, not a candidate checkout.
  • Two coupling styles for the same goal: lifecycle/drivers.ts (CLI and MCP through spawned bun <build>/src/cli.ts, with copied builds from lifecycle/builds.ts) is the only fully black-box driver family here. It is the closest model for a Rust-side CLI or MCP adapter. Its MCP framing is JSON-RPC over stdio or HTTP with protocolVersion 2025-06-18.
  • MCP tool names in use: entity, search, context_pack, put_page, get_page, recall, remember, forget, list_pages, get_timeline, get_links, query. CLI operations in use: init, sources, sync, extract, embed, call <op>, jobs work, beliefs enable --source vault, serve [--surface starter|full] [--http --port].
  • Engine methods called in-process across this group: connect, disconnect, initSchema, getChunks, getConfig, setConfig, unsetConfig, executeRaw, deletePage, traverseGraph, traversePathsDetailed, now (test seam), and brain.disconnect.
  • Shared infrastructure outside this scope that every paid runner depends on: budget-ledger.ts (startPaidRun, BudgetRun, ledgerStatus), paid-arm.ts (requirePaidArm), metering-proxy.ts (MeteringProxy), prereg.ts (attestPreregistration), receipt.ts (scrubMachinePaths), gbrain-under-test.ts (resolveGbrainUnderTest, importGbrain, overlays under .gbrain-overlays/), memory-qa/* and q1/*. Their behavior was not traced beyond these call sites.
  • Keyless modes exist for most groups (--scripted, hash embedder, estimate, score, verdict, w10-rescore, failure-causes classify, dry-run and gate scripts). Paid modes are guarded by --paid, --budget-usd, or --budget-run-id, and the ledgers use SQLite.
  • Surprises: cat40/gbrain-arm.ts and facts-absorb-gate.ts both reuse McpClient from cat40/gbrain-arm.ts, so changes to that file affect cat41, facts-absorb and cat40. lifecycle/slice.ts has four callers outside scope (n1-knowledge-update.ts, n5-forget-residue.ts, n6-postgres-http.ts, p5-brain.ts).
  • Coupling of failure-causes/label.ts uses a caller-supplied ChatLike client; the provider key path was not traced.
  • Not verified: presence of eval/data/model-ladder-v1 and other named data directories, and actual costs and wall times for most paid runs. Those are reported as "unknown" or as the figure from the cited docs.