Garry's Opinionated OpenClaw/Hermes Agent Brain
Appendix to ../gbrain-evals.md. Detailed per-area notes from one research pass over gbrain-evals
cd447b3(v0.10.237) and gbrain6ce7195. Paths are relative to the gbrain-evals root unless markedgbrain:. "G1" to "G10" in the text name the appendix files listed in the main map.
G5 runner group A: gbrain-evals eval/runner directories
Checkout: the gbrain-evals checkout (read only). All paths below are relative to eval/runner/ unless shown otherwise.
Scope: 16 runner groups covering batch/, bg1/, budgeted-delivery/, cat40/ (with cat40-model-ladder.ts and scripts/cat40-*.sh), cat41/ (with cat41-agent-operator.ts), decisions/ (with decide.ts and eval/decisions/), evaluator/, evidence-delivery/ (with evidence-delivery.ts), external-ideas-wave-a/ (with external-ideas-wave-a.ts), facts-absorb/ (with facts-absorb-gate.ts), failure-causes/, graph-backend/, lifecycle/, lifecycle-lite/ (with lifecycle-experiment.ts, lifecycle-lite.ts, lifecycle-report.ts), lhf-wave-2-attendance-gate.ts.
Summary
| Runner | Entry path | Measures | Run command | Tier / time / cost | LLM keys | Corpus | Coupling |
|---|---|---|---|---|---|---|---|
| cat40 model ladder and Hard | eval/runner/cat40-model-ladder.ts (helpers cat40/, wrappers scripts/cat40-hard.sh, scripts/cat40-followups.sh) |
Whether gbrain's advantage over file, pg and memory arms persists across models; agent task accuracy | bun eval/runner/cat40-model-ladder.ts --models ... --arms oracle,fs,pg,memory,gbrain --budget-usd N --out <dir>; --scripted for $0 |
Paid. Gbrain cell $0.02 to $0.30, fs/pg about $0.05 per cell (protocol). Hard ledger cap $1,794; steps from $1.79 (smoke) to $114.61 (freeze-check), wall clock 20 min to 7.5 h | ANTHROPIC, OPENAI, PARTICLE (particle-1), VOYAGE for gbrain slots. --scripted is keyless |
Named worlds eval/data/model-ladder-v1* and generated Hard worlds by seed |
CLI+MCP (spawns bun <build>/src/cli.ts serve, JSON-RPC over stdio), plus in-process PGLite via bun -e subprocesses |
| cat41 agent operator | eval/runner/cat41-agent-operator.ts (helpers cat41/, Dockerfile) |
Whether real Claude Code and Codex CLI sessions in a pinned container take the right operator actions (degraded recall, consent, recovery) | bun eval/runner/cat41-agent-operator.ts run --gbrain <checkout>@<ref> --paid --budget-usd N --out <dir>; score, overhead, gate are keyless |
Paid. About $12 to $17 per 3-repeat pass, about 35 min at concurrency 4 (protocol). Reserve $2.5 (claude) and $1.5 (codex) per run | ANTHROPIC, OPENAI (inside container) | Scenarios in cat41/scenarios.ts, D7 worlds cat41/brief-worlds.json (committed) |
Mixed: spawns docker; in-container spawns bun /opt/gbrain/src/cli.ts; in-process gbrain module imports inside the container; host MCP client for overhead |
| decision kit | eval/runner/decide.ts (npm eval:decide; helpers decisions/, data eval/decisions/) |
Held-out and dev decision verdicts for a gbrain candidate vs baseline, per source | bun run eval:decide <init|fetch|preflight|dev|verdict|status|check> --decision <dir> |
Dev runs are paid through child memory-qa jobs. Cost and time unknown (not traced into memory-qa) | Via child jobs (OPENAI seen in decide.ts) | eval/decisions/datasets/*.json (committed, 208K), eval/decisions/splits/*.json (committed, 60K) |
Spawn (git, and bun child runners per arm); in-process only for spec and spec parsing |
| evidence delivery | eval/runner/evidence-delivery.ts (helpers evidence-delivery/) |
Evidence delivery ablation E1, product-path parity E3, sealed bridge E2 on LongMemEval-S | bun eval/runner/evidence-delivery.ts <cluster-map|power|parity|analyze|costs> keyless; campaign-open, freeze, e1, e3, e2-* paid |
Paid. Campaign cap $400 (campaign-open --budget-usd <= 400); smoke cap $3 (SMOKE_CAP_USD). Wall clock unknown |
OPENAI, VOYAGE, Anthropic reader (Sonnet), GPT-4o reader | longmemeval_s_cleaned.json (passed by --dataset, not committed here); R1 receipts under docs/benchmarks/2026-09-29-longmemeval-opaque-qa/ |
In-process: dynamic import of gbrain src/ from GBRAIN_DIR; MCP stdio for e3 (bun --preload ledger-preload.ts src/cli.ts) |
| external ideas wave A | eval/runner/external-ideas-wave-a.ts (helpers external-ideas-wave-a/, synthetic-dates.ts) |
Whether search.evidence_resolve_dates (absolute date brackets) changes LongMemEval-S accuracy |
bun eval/runner/external-ideas-wave-a.ts <freeze|affected|answer|analyze|verdict|audit|estimate|a2-verdict>; estimate keyless |
Paid. Lane ledger $220 across waves A and B; runs over $50 need --gbra108-approval. Wall clock unknown |
ANTHROPIC, OPENAI, VOYAGE | LongMemEval-S (--dataset), synthetic relative-date set generated by synthetic-dates.ts |
In-process (gbrain src/ via loadGbrain); batch HTTP to providers via batch/ lane |
| facts absorb gate | eval/runner/facts-absorb-gate.ts (helpers facts-absorb/) |
Whether a cheaper facts.extraction_model keeps the facts gbrain's background extractor saves (recall, precision, attribution, corrections) |
bun eval/runner/facts-absorb-gate.ts --gbrain <checkout>@<ref> --arms ... --seeds 60,61 --prereg <file> --budget-usd N --out <dir> |
Paid. --budget-usd required; cost and time unknown |
Provider keys for each extraction arm (anthropic, openai) through MeteringProxy | Generated facts-absorb world (eval/generators/facts-absorb-gen.ts) |
MCP stdio (gbrain serve --surface full) plus CLI (gbrain call, gbrain jobs work) |
| lifecycle experiment | eval/runner/lifecycle-experiment.ts (helpers lifecycle/) |
Memory lifecycle correctness: ingest, outage, correct, reconcile, forget, restart, query, across build x engine x interface | bun eval/runner/lifecycle-experiment.ts --gbrain-repo ../gbrain [--builds a=<sha>,b=origin/master] [--engines pglite,postgres] [--interfaces cli,mcp-stdio,mcp-http] [--out <receipt.json>] |
$0 (hermetic). Wall clock unknown (many spawned processes). Receipts in docs/benchmarks/2026-09-29-lifecycle* |
None. Provider keys are stripped from child env | Generated in code (lifecycle/scenario.ts); no dataset |
CLI and MCP (stdio and HTTP) spawned as child processes; Postgres via Bun SQL |
| lifecycle-lite | eval/runner/lifecycle-lite.ts (helpers lifecycle-lite/) |
Update and forget behavior (N1 correction chains, N5 canaries) for open-source memory systems; report-only | bun eval/runner/lifecycle-lite.ts --system <shim URL>|gbrain-shootout|fake|honest-reference|mutation:<kind> --output <dir> |
Default hermetic with hash embedder. Paid only with --paid --budget-run-id. Timeouts 600 s finish, 3600 s ingest |
None for hermetic. OPENAI, ANTHROPIC, VOYAGE for --embed real, --qa reader, reranker |
Generated generators/lifecycle-lite-gen.ts seeds |
In-process for gbrain-shootout (src/core/ai/gateway.ts, engine methods); HTTP protocol v1 for shims; bash -c spawn for restart |
| lifecycle-report | eval/runner/lifecycle-report.ts |
Markdown tables from lifecycle-experiment receipts, plus cross-run match | bun eval/runner/lifecycle-report.ts <receipt.json> [more...] |
$0, under a second | None | Receipts only | None |
| LHF wave 2 attendance gate | eval/runner/lhf-wave-2-attendance-gate.ts |
Preregistered attendance paraphrase gate: fired counts and false routes from two N9 receipts | bun eval/runner/lhf-wave-2-attendance-gate.ts --base <receipt> --branch <receipt> --base-relations <json> --branch-relations <json> [--json] |
$0. Counts only, no statistics | None | Two N9 receipts plus relations files from parse-relational-questions.ts |
None (reads JSON) |
| batch lane (LongMemEval W10) | eval/runner/batch/ (w10.ts, w10a-capture.ts, w10-particle.ts, w10-particle-second-judge.ts, w10-rescore.ts; lane libs submit.ts, transport.ts, ledger.ts, manifest.ts, sources.ts, receipts.ts) |
Reader and judge accuracy on LongMemEval arms through provider batch APIs, with exactly-once submission | bun eval/runner/batch/w10.ts build|plan|run <arm>; w10-rescore.ts keyless |
Paid (batch factor 1.0 list price, see batch/ledger.ts). w10-rescore.ts is $0. Total cost unknown |
ANTHROPIC, OPENAI (batch/transport.ts) |
LongMemEval-S plus R1/R2 reranker-on call logs (batch/sources.ts) |
In-process imports from the pinned node_modules/gbrain; HTTP to provider batch APIs; SQLite lane state (bun:sqlite) |
| BG1 battleground | eval/runner/bg1/ (verdict.ts, power.ts, pricing.ts, rule.ts, match.ts, exposure.ts, judge-check.ts, launch.ts, types.ts) |
Changing-facts battleground: gbrain vs comparators on short answers, with paired t and Holm; exposure placement; judge validity | bun eval/runner/bg1/verdict.ts verdict --receipts <dir>...; power.ts, pricing.ts keyless; judge-check.ts paid |
Analysis tools are $0. judge-check paid on public dev data. Campaign cost in bg1/pricing.ts |
ANTHROPIC, OPENAI, PARTICLE, VOYAGE (judge-check only) | BG1 manifest eval/decisions/datasets/bg1-manifest.json, cells from q1/ (outside scope) |
None for analysis. judge-check uses MeteringProxy over HTTP to judge models |
| budgeted delivery | eval/runner/budgeted-delivery/ (h1-run.sh, c8k-run.sh, e2-keyless-gate.sh, keyless-gate.sh, dry-run scripts; TS h1.ts, c8k.ts, c8k-product.ts, e1-readings.ts, e2-readings.ts, e3-retrieval-gate.ts, accounting-gate.ts, budget-sizing.ts) |
Whether evidence packed under explicit token budgets keeps reader accuracy (E1, E2, E3, 8k conversion, H1 held-out custody) | bash eval/runner/budgeted-delivery/h1-dry-run.sh and c8k-dry-run.sh (keyless); keyless-gate.sh; live: h1-run.sh <root>, c8k-run.sh <root> |
Dry runs and gates $0 with dummy keys and hash vectors. Live runs paid; cost and time unknown | ANTHROPIC, OPENAI, VOYAGE for live runs; SHOOTOUT_PROXY | Invented fixture h1-fixture.ts for dry runs; sealed custody files for live runs; LongMemEval-S |
In-process (c8k-product.ts: gbrain PGLite engine and extraction from GBRAIN_DIR); HTTP stub or metering proxy; shell |
| evaluator | eval/runner/evaluator/ (library, no entry point) |
Evaluator-side gold stores, input allowlists (leak checks) and independent reference scorers for LongMemEval and Cat13 | None (imported by runners) | $0 | None | Gold from corpus loaders (cat13-conceptual.ts, LongMemEval) |
None (pure) |
| failure causes | eval/runner/failure-causes/cli.ts (library index.ts) |
Failure-cause taxonomy and classification of nightly and other eval misses | bun eval/runner/failure-causes/cli.ts taxonomy [--write|--check]; classify --manifest <sources.json> --root name=<dir> --out <labels.ndjson> |
Deterministic classify $0. Residual labeller paid (model call, see label.ts) |
Residual labeller only (model via memory-qa/qa-v2.ts ChatLike, keys not traced) |
Source manifests with SHA-256 per file (adapters read nightly, skillopt, live-recall, refresh-barrier, rs1, cat40) |
None (reads archived rows) |
| graph backend (GBRA-123) | eval/runner/graph-backend/run-tier.sh <tier> [stages], worker worker.ts, pg-load.ts, plan.ts, generate.ts, analyze.ts |
Latency and correctness of graph walks: gbrain CTE (A), breadth-first (B), PGLite (BL), HydraDB (C) over fictional link graphs | GB_A=<wt> GB_B=<wt> OUT=<dir> bash eval/runner/graph-backend/run-tier.sh personal (stage list optional) |
$0 LLM. Latency receipts in docs/benchmarks/2026-10-17-graph-backend/{personal,team,company} and summary.json. Wall clock unknown |
None | Generated fictional graphs (generate.ts, seeded) |
In-process engine methods from two gbrain worktrees (via GB_ENGINE_DIR); HTTP to HydraDB; spawned bun workers and Postgres |
Runner details
cat40 model ladder and Hard
- Entry file:
eval/runner/cat40-model-ladder.ts. Helpers:cat40/(arms.ts,loop.ts,gbrain-arm.ts,pg-arm.ts,hard.ts,hard-grep.ts,hard-ops.ts,hard-proxy.ts,rerank-pool-replay.ts,rerank-pool-proxy.ts,latency-replay.ts,firing.ts,analyze.ts,score.ts,rescore.ts,records.ts,types-filter.ts,reuse-check.ts,next-confirm.ts,dev-arms.ts,judge-hard.ts,arm_power.py,next_confirm_analysis.py). Wrappers:scripts/cat40-hard.shandscripts/cat40-followups.sh(the only repo-level scripts for this family; nocat40entry in package.json). - What it measures: whether gbrain's advantage over simpler memory (fs, fs-acl, memory, pg, oracle, gbrain;
cat40/arms.ts:21-22) persists across models, on the world ineval/data/model-ladder-v1(header ofcat40-model-ladder.ts). Hard campaign adds a 50k held-out world with a primary endpoint of gbrain minus fs (scripts/cat40-hard.shheader). - How it runs:
bun eval/runner/cat40-model-ladder.ts --models claude-sonnet-4-6,gpt-5.4 --arms oracle,fs,pg,memory,gbrain --budget-usd 100 [--families A,B] [--tasks A01] [--slots 3] [--gbrain <checkout>@<ref>] [--out eval/reports/cat40/<name>]. Hermetic:--scripted --arms fs,memory,oracle($0). Keyless slots:--scripted --keyless-slots --arms gbrain. Build slots:--build-slots --gbrain-ref <sha> --slots 5 --budget-usd 10. Hard steps:scripts/cat40-hard.sh hello|status|preflight <step>|step <step>, ledger.budget/cat40-hard.sqlite. - Cost and wall time: protocol gives
fsandpgcells about $0.05 andgbraincells $0.02 to $0.30 (docs/benchmarks/2026-10-02-model-ladder-protocol.md:227). Hard runbook (docs/benchmarks/cat40-hard/RUNBOOK.md): freeze-check $114.61 (about 90 min), smoke $1.79 (about 20 min), slots-50k $12.04 (about 7.5 h), oracle-50k about 1 h, pg-50k about 5 h, memory-50k about 4 h. Ledger cap $1,794 (RUNBOOK line 11). Followups ledger cap $237 (scripts/cat40-followups.shheader). - LLM keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, PARTICLE_API_KEY (particle-1 through
providers/openai-compatible.ts), VOYAGE_API_KEY (gbrain slot embeddings and rerank viacat40/gbrain-arm.ts). Offline:--scripted(hermetic, $0) and--scripted --keyless-slots(keyword-only slots, rerank probe skipped). - Corpus:
eval/data/model-ladder-v1andeval/data/model-ladder-wide-dev/world.jsonnamed in the header (not checked for presence in this checkout). Hard worlds are generated from seeds in the frozen knobs (cat40/plusscripts/cat40-hard.sh). - gbrain imports:
./gbrain-under-test.ts:gbrainSpecFrom,parseGbrainSpec(spec parsing only,cat40-model-ladder.ts:69)../cat40/gbrain-arm.ts:GbrainArm,GbrainFsArm,GbrainPool,GbrainSlot,MeteringProxy,McpClient,rerankProbe,restoreProbe,coverageProblem,instructionsOverride(cat40-model-ladder.ts:74,cat40/hard.ts:15).- gbrain methods reached over MCP (
McpClient.callincat40/gbrain-arm.ts):entity,search,context_pack,put_page(Hard only),get_page(cat40/hard.ts,cat40/gbrain-arm.ts). cat40/rerank-pool-replay.ts:548: dynamic import of<buildDir>/src/core/search/rerank.ts(capRerankDoc) and PGlite from the build's node_modules, run insidebun -e.
- Coupling: CLI+MCP.
cat40/gbrain-arm.ts:93spawnsbun <buildDir>/src/cli.ts serve --surface starter(or--surface full) and speaks JSON-RPC over stdio (listTools,tools/call;:137). Slot snapshots and restores run asbun -escripts that open PGlite directly from the build (gbrain-arm.ts:412, 522, 530, 561).cat40/pg-arm.tsis its own PGlite arm (toolsvector_search,keyword_search,save_document,get_document) with no gbrain code. To run against another binary, the runner needs theservecommand, thetools/listandtools/callnames above, and the snapshot step replaced. - Parity-gate fit: deterministic only in
--scriptedmode. Paid cells use LLM judgments, so not deterministic. Scripted mode is fast.
cat41 agent operator
- Entry file:
eval/runner/cat41-agent-operator.ts. Helpers:cat41/types.ts,cat41/scenarios.ts(scenarios, SEED_FACTS, SEED_PAGES),cat41/brief-scenarios.ts(D7),cat41/classify.ts(per-step outcome classes and gate),cat41/transcript.ts(normalizesclaude --output-format stream-jsonandcodex exec --json),cat41/in-container.ts(driver inside the image),cat41/brief-db.ts,cat41/brief-d7.ts,cat41/brief-worlds.tsandbrief-worlds.json,cat41/tthw-verdict.ts,cat41/after-pass.sh,cat41/Dockerfile(FROM node:24.18.0-bookworm-slim). - What it measures: whether real Claude Code and Codex CLI sessions, given scripted user requests, classify each step as authorized execution, required relay, correct refusal, successful recovery, consent violation, or false "no notes". Pins: claude_code 2.1.285, codex 0.160.0, bun 1.4.2 (
cat41-agent-operator.tsPINS). Default models claude-opus-5-5 and gpt-6.1-sol. - How it runs:
bun eval/runner/cat41-agent-operator.ts run --gbrain <checkout>@<ref> --label baseline [--harnesses claude,codex] [--scenarios all] [--repeat 3] [--concurrency 3] --paid --budget-usd <n> --out eval/reports/cat41/<label>.score --out <dir>($0),overhead --gbrain <checkout>@<ref> --out <dir>($0 token overhead of initialize and tools/list per surface),gate --before <dir> --after <dir>. Docs:docs/benchmarks/2026-10-03-agent-operator-protocol.md. - Cost and wall time: protocol says a three-repeat pass costs about $12 to $17 and about 35 min at concurrency 4 (
docs/benchmarks/2026-10-03-agent-operator-protocol.md:183). Code reserves $2.5 (claude) and $1.5 (codex) per container run and estimates $0.5 and $0.2 (cat41-agent-operator.tsconstants).CRASH_RETRIES = 2. - LLM keys: ANTHROPIC_API_KEY, OPENAI_API_KEY (inside the container;
cat41/in-container.ts,cat41/scenarios.ts). Keyless modes:score,overhead,gate, and the D7brief-d7.ts smoke($0, fake provider). - Corpus: scenario and world data are generated in code (
cat41/scenarios.ts,cat41/brief-worlds.ts).cat41/brief-worlds.jsonis committed (brief-worlds.ts --writeregenerates it; the test checks it is current). - gbrain imports and calls:
- Host:
./gbrain-under-test.ts(resolveGbrainUnderTest,overlaySummary);./cat40/gbrain-arm.tsMcpClient(cat41-agent-operator.ts:256, used for tools/list overhead);./lifecycle/drivers.tsrunCli(import in the file header). - Container, dynamic imports from
/opt/gbrain/srcincat41/brief-db.ts:18-44:core/config.ts(loadConfig,toEngineConfig),core/engine-factory.ts(createEngine),mcp/resolve-ipc-binding.ts(turnContextWithBrief),core/context/proactive-brief.ts(turnBrief,briefEnabled,TURN_BRIEF_CUTOFF_MS). The engine methods called on the object fromcreateEngineare not traced here. - Container CLI:
cat41/in-container.ts:178-240runsbun /opt/gbrain/src/cli.ts <args>as the agent user, andpgrepfor the process.
- Host:
- Coupling: mixed. Host spawns
docker(cat41-agent-operator.ts:93, kill on timeout at:96). Inside the container the real harness CLIs run gbrain viabun /opt/gbrain/src/cli.ts(CLI) and their MCP config (claude mcp addpattern incat41/classify.ts:71). The host uses MCP stdio for overhead. The gbrain build is mounted read-only at/opt/gbrain. - Parity-gate fit: no. Paid runs are non-deterministic and slow.
score,overhead,gateare deterministic and fast.
decision kit (decide.ts, decisions/, eval/decisions/)
- Entry file:
eval/runner/decide.ts, exposed as npm scripteval:decide(package.json:23:bun eval/runner/decide.ts). Subcommands:init,fetch,preflight,dev,verdict,status,check. Held-out commands (power, prereg, seal-run) are documented as later milestones. - Helpers in
decisions/:spec.ts(loadSpec,newSpec,validateSpec, typesDecisionSpec,Source,Plan),splits.ts(dev/sealed split by whole conversation, ordered by SHA-256 of salt and conversation id),anchors.ts(dynamic import ofqueries/relational.ts:buildRelationalQueries,loadWorldCorpus; excludes dev probes that share an anchor entity with sealed questions),errors.ts(operator-message contract,DecideError,exitCodeFor). - Top-level data
eval/decisions/:datasets/(beam-10m, beam-b2da22e, blue-book-e494de6 at 146K, bg1-manifest, d8-manifest, lme-m-q1-selection, rs1-manifest),splits/(beam-100k, beam-10m, beam-1m, beam-500k). Committed, total 268K. - What it measures: for each spec source, a paired, clustered comparison of a candidate gbrain checkout against a baseline (
stats/gates.tsevaluateFamily), per source. Dev verdicts never flip a default (decide.ts header). - How it runs:
bun run eval:decide init --plan P6 --gbrain ../gbrain@<sha>;fetch --decision <dir>;preflight --decision <dir>;dev --decision <dir>;verdict --decision <dir>;status.pinArmresolves<checkout>@<ref>to a full SHA and checks the dirty tree (decide.ts:50-56). - Cost and wall time: unknown. Each
devjob is a childbunprocess runningeval/runner/memory-qa/run.tswith a per-benchmark argv (decide.ts:184, spawn at:211). Costs are whatever those child runners spend (not traced into memory-qa). - LLM keys: depend on the sources.
decide.tsreferences OPENAI_API_KEY (3 hits) and--paid(12 hits). Offline:preflight,status,check,fetch --verify-onlyare keyless. - Corpus:
eval/decisions/datasets/*.json(committed) with SHA-checked downloads throughmemory-qa/corpus.tsfetchDataset/filesFor(outside scope). - gbrain imports:
./gbrain-under-test.tsparseGbrainSpec,resolveGbrainUnderTestonly (no engine calls in decide.ts).decisions/anchors.tsimports the world's relational queries from the runner tree, not gbrain. - Coupling: spawn (git subprocesses at
decide.ts:53-54;bunchild runners per arm and shard). The child runners own the gbrain coupling (memory-qa, outside this scope). - Parity-gate fit:
check,status,preflight,initare deterministic and keyless.devis paid and not a parity gate.
evidence delivery (evidence-delivery.ts, evidence-delivery/)
- Entry file:
eval/runner/evidence-delivery.ts. Helpers inevidence-delivery/:gbrain.ts(loader and identity),data.ts(dataset, pilot split, cluster map),freeze.ts,store.ts(content-addressed evidence blobs),arms.ts,e1.ts,e2-bridge.ts,e3.ts,calls.ts,parity.ts,power.ts,decision.ts,decision-v2.ts,ledger-preload.ts. - What it measures: E1 delivery ablation (which evidence packing reaches the reader best), E3 product-path parity (does the harness deliver what gbrain's product path would), and E2 sealed bridge, all on LongMemEval-S under a preregistered decision rule (
docs/benchmarks/2026-09-30-evidence-delivery-preregistration.md). - How it runs: keyless
bun eval/runner/evidence-delivery.ts cluster-map --dataset <json>,power [--sims N],parity --frozen-dir <dir> --dataset <json>,analyze --rows-dir <dir> --stage pilot|confirmatory,costs. Paid, always through the budget ledger:campaign-open --budget-usd <= 400,freeze --dataset <json> --out-dir <dir>,e1 ... --arms a,b,e3 ... --score,e2-freeze,e2-answer,e2-score.--smokeruns open their own run capped at $3.GBRAIN_DIR(or--gbrain-dir) names the checkout. - Cost and wall time: campaign cap $400 (
campaign-open), smoke cap $3 (SMOKE_CAP_USD = 3). Per-step wall time unknown.estimateE1Usdine1.tsgives a planning estimate. - LLM keys: OPENAI_API_KEY and VOYAGE_API_KEY (
e3.ts,calls.ts); readers are Sonnet (SONNETine1.ts) and GPT-4o (GPT4O); judges via gbrain's framed judge and the official LongMemEval judge (calls.ts).BRAINBENCH_BUDGET_RUN_IDis required byledger-preload.ts. - Corpus:
longmemeval_s_cleaned.json(passed as--dataset, not committed here). R1 receipts atdocs/benchmarks/2026-09-29-longmemeval-opaque-qa/reranker-on/r1(R1_DIR). Frozen evidence goes to a content-addressed store. - gbrain imports (
evidence-delivery/gbrain.ts:85-100,loadGbrain, dynamicimport(join(dir, p))):src/eval/longmemeval/harness.ts(createBenchmarkBrain,resetTables),adapter.ts(haystackToPages),metrics.ts(sessionIdFromSlug,buildSlugToRawMap,rawSessionId),src/core/import-file.ts(importFromContent),src/core/search/hybrid.ts(hybridSearch),src/eval/shared/embed-cache.ts(EmbeddingCache,installEmbedCache),src/eval/longmemeval/reader.ts(buildReaderUserText,READER_NOTES_SYSTEM_TEXT,READER_MAX_SESSION_CHARS),sanitize.ts(renderChatBlock),src/core/chunkers/token-estimate.ts(estimateTokens),src/core/ai/gateway.ts(configureGateway,chat),src/core/ai/build-gateway-config.ts(buildGatewayConfig),src/eval/longmemeval/judge.ts(judgeRow),src/eval/shared/judge-runner.ts(BudgetLedger),src/core/search/evidence-delivery.ts(assembleEvidenceForHits,evidenceFingerprint, required by--requireEvidence).- Engine calls:
evidence-delivery.ts:399-412dynamic import ofsrc/core/pglite-engine.ts(PGLiteEngine), thenengine.connect,engine.initSchema,engine.disconnect.freeze.ts:66-86callsengine.getConfig,setConfig,unsetConfig;freeze.ts:183callsengine.getChunks. - Shared
budget-ledger.tsandreceipt.tsare outside scope.
- Coupling: in-process for the gbrain pipeline (dynamic import of
GBRAIN_DIRmodules; engine on PGLite). MCP stdio for E3 product-path only:e3.ts:169-170buildsMcpStdioDriverwith entrybun --preload ledger-preload.ts <gbrainDir>/src/cli.ts, so gbrain's ownservepath is measured. To run against a binary, the parity checks need the MCP tool list and results, and the freeze step needs a JSON adapter forgetChunks, hybrid search and assembleEvidence. - Parity-gate fit: keyless subcommands (
cluster-map,power,parity,analyze,costs) are deterministic and fast. Paid steps are not.
external ideas wave A (external-ideas-wave-a.ts, external-ideas-wave-a/)
- Entry file:
eval/runner/external-ideas-wave-a.ts. Helpers:external-ideas-wave-a/answer.ts(one resumable answer step over the batch lane),lib.ts(study manifest, affected set, analysis, bracket audit),render-arms.ts(eval-onlyr1_rank_dateandr2_tail_trimrenderings built from the frozenautodelivery),synthetic-dates.ts(seeded synthetic relative-date set, LongMemEval format). Also importsevidence-delivery.ts(Args,evalsCommit,writeJson). - What it measures: whether gbrain's
search.evidence_resolve_datesflag, which adds absolute date brackets to relative phrases in delivered turns, changes LongMemEval-S answer accuracy. Armsauto(flag off) andauto_dates(flag on) are delivered by gbrain'sassembleEvidenceForHitson the same frozen hits. Study manifestdocs/benchmarks/2026-10-21-external-ideas-wave-a/study-manifest.json. - How it runs:
freeze --dataset <json> --out-dir <dir> --budget-usd <cap> [--smoke],affected --frozen-dir <dir> --dataset <json>,answer --frozen-dir ... --arm auto --sample 1 --model <id> --set all --out-dir ... --budget-usd <cap> [--wait],analyze,verdict,audit,estimate --dataset <json>(keyless),a2-verdict.GBRAIN_DIRnames the checkout. - Cost and wall time: lane ledger $220 across waves A and B (
GBRA162_LEDGER, default~/.capy/work/gbra162-external-ideas/ledger.sqlite). A run over $50 needs--gbra108-approval. Exit codes: 0 done, 1 failed, 2 preflight refused, 4 budget stop. Wall time unknown.estimategives the planning figure without keys. - LLM keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, VOYAGE_API_KEY (
external-ideas-wave-a.tsreferences). Keyless:estimate,affected,render-arms,synthetic-dates,analyze,verdict(given analysis files). - Corpus: LongMemEval-S (
--dataset, not committed here) and the synthetic dates generated bysynthetic-dates.ts. Frozen evidence in a content-addressed store (evidence-delivery/store.ts). - gbrain imports:
./evidence-delivery/gbrain.ts:configureGbrainGateway,gbrainIdentity,loadGbrain,resolveGbrainDir(the same dynamic loader as evidence delivery,gbrain.ts:85-100)../evidence-delivery/e1.ts:rendererFrom../evidence-delivery/calls.ts:gbrainJudge,officialJudge../evidence-delivery/freeze.ts:freeze.- Engine:
brain.disconnect(external-ideas-wave-a.ts, one call);assembleEvidenceForHitsandevidenceFingerprintfrom the loadedevidencemodule. ./batch/submit.tsBatchLane,./batch/transport.ts(anthropicTransport,openAiTransport),./batch/manifest.ts(buildManifest,freezeManifest),./batch/sources.ts(readerBody,seededSample,JUDGE_PROTOCOL).
- Coupling: in-process (gbrain
src/vialoadGbrain), plus HTTP to provider batch APIs. No CLI or MCP. - Parity-gate fit:
estimate,affected,analyze,verdictdeterministic.answerandfreezeare paid and slow.
facts absorb gate (facts-absorb-gate.ts, facts-absorb/)
- Entry file:
eval/runner/facts-absorb-gate.ts. Helpers:facts-absorb/score.ts(scoreArm,decide,valueToken; pure, matches stored facts to the planted claims),facts-absorb/natural.ts(natural-prose stratum; the runner calls the Cat 35 coverage judgescoreSalienceCoveragefromcat35-judges.ts). - What it measures: whether a cheaper
facts.extraction_modelkeeps the facts gbrain's background extractor stores, measured as recall, precision, attribution and correction handling against an answer key. Default armanthropic:claude-sonnet-4-6(SHIPPED_DEFAULT_MODEL). Preregistered arms inDEFAULT_ARMS: sonnet46, luna (openai:gpt-6-luna), haiku55 (anthropic:claude-haiku-5-5), disabled (mutant), drop (mutant that replaces every chat output with{"facts":[]}). Preregistrationdocs/benchmarks/2026-10-08-facts-extraction-model/PREREGISTRATION.md. - How it runs:
bun eval/runner/facts-absorb-gate.ts --gbrain <checkout>@<ref> --arms sonnet46=default,luna=openai:gpt-6-luna,haiku55=anthropic:claude-haiku-5-5,disabled=disabled,drop=drop:openai:gpt-6-luna --seeds 60,61 --prereg <path> --sample-reads 30 --budget-usd <n> --out <dir>. Paid arms refuse to start without--prereg.--smokeexists. - Cost and wall time:
--budget-usdis required and the ledger isstartPaidRun. Actual cost and time unknown. The job drain step gets a 3 h timeout (facts-absorb-gate.ts:211). - LLM keys: anthropic and openai keys for each arm, through
MeteringProxy(cat40/gbrain-arm.ts).drop:anddisabledarms are mutants. Gbrain's own call ledger is read fromGBRAIN_AI_CALL_LOGto get the resolved model. - Corpus: generated by
eval/generators/facts-absorb-gen.ts(generateFactsAbsorbWorld,renderSessionNote) from seeds; natural prose from Cat 35 transcripts. - gbrain imports and calls:
./cat40/gbrain-arm.ts:GbrainSlot,McpClient,MeteringProxy,GBRAIN_EMBED_MODEL.McpClient.call('put_page', { slug, content })at:201and:205../lifecycle/drivers.tsrunCli,./lifecycle/builds.tsprepareBuild(copied build).- CLI operations via
runCli:call recall(:171),jobs work --concurrency N(:211,3 * 3600000ms timeout), slot-levelargs(:185). The script at:103runsbun -e <script>. - Server:
new McpClient(slot.run, ['--surface', 'full'])(:196), i.e.gbrain serve --surface fullover stdio.
- Coupling: MCP stdio (put_page, recall via serve) and CLI (
gbrain call,gbrain jobs work). The gbrain build runs as a copied overlay (prepareBuild). To run a different binary the runner needs the serve tool names (put_page),call recall, andjobs work. - Parity-gate fit:
score.tsis deterministic and fast. The gate itself is paid and slow (the writes, background job drain, and LLM extraction).
lifecycle experiment (lifecycle-experiment.ts, lifecycle/)
- Entry file:
eval/runner/lifecycle-experiment.ts. Helpers inlifecycle/:drivers.ts(CLI, MCP stdio, MCP HTTP drivers andrunCli),slice.ts(SliceCell, one cell for the eval-category wave; callersn1-knowledge-update.ts,n5-forget-residue.ts,n6-postgres-http.ts,p5-brain.tsat runner root),builds.ts(prepareBuild,assertBuildVerified, copied builds),make-variant-build.ts(makeVariantBuild, patch-based variant),observe.ts(observe,probeLeaks),scenario.ts(ground truth identities, facts, phases, checkpoints),score.ts(scoreCheckpoint,scoreBijection,scoreEdges,scoreTimeline,scoreFacts),n1-score.ts,n5-score.ts,fake-embedder.ts(startFakeEmbedder, local HTTP hash embedder),n1-score.tsandn5-score.tsfor the N1 and N5 categories. - What it measures: correctness of a memory lifecycle per cell (build x engine x interface): ingest, query, embedding outage during ingest, correct, reconcile, forget, restart, query again. Ground truth is the evaluator's own ledger; gbrain's doctor and integrity checks are never consulted (header).
- How it runs:
bun eval/runner/lifecycle-experiment.ts --gbrain-repo ../gbrain [--builds a=6bb88d128,b=origin/master] [--engines pglite,postgres] [--interfaces cli,mcp-stdio,mcp-http] [--pg-url postgres://user@host:port/db] [--concurrency 4] [--out docs/benchmarks/<dir>/receipt.json]. Render receipts withlifecycle-report.ts. - Cost and wall time: $0. Hermetic, with a local hash embedder (
fake-embedder.ts) and provider keys stripped from child env (header). Wall time unknown (many child processes, Postgres optional). - LLM keys: none. The runner strips OPENAI_API_KEY and ANTHROPIC_API_KEY from each child (header and
lifecycle/drivers.ts). Offline by design. - Corpus: generated in code by
lifecycle/scenario.ts(no dataset file). - gbrain imports: none in-process. The only library import is
SQLfrombun(lifecycle-experiment.ts:23,lifecycle/slice.ts:31) for Postgres. All gbrain access is through spawnedbun <build>/src/cli.tsprocesses. - Exact process mechanics, from
lifecycle/drivers.tsandlifecycle/builds.ts:- Build:
git clone -q --no-checkoutandgit archive <commit> | tar -xinto a temp dir (builds.ts:62, 117),bun install --frozen-lockfilein that dir (:118), thenbun src/cli.ts --versionwithGBRAIN_SKIP_STARTUP_HOOKS=1(:128). The build is a copy, not a symlink, and the cmd checksgit write-treeagainst the commit tree (:91-92). - CliDriver (
drivers.ts,runCli):spawn('bun', [<build>/src/cli.ts, 'call', <op>, <json>], { env, cwd: GBRAIN_HOME })per call,stdin.end(), 180 s timeout. Trusted local caller. - McpStdioDriver:
spawn('bun', [<build>/src/cli.ts, 'serve'])once per session. Writes JSON-RPC lines to stdin:initialize(protocolVersion2025-06-18),notifications/initialized, thentools/callwith{ name: op, arguments }per call. Reads newline-delimited responses on stdout. Restart kills and respawns. 15 s SIGKILL on close. - McpHttpDriver:
spawn('bun', [<build>/src/cli.ts, 'serve', '--http', '--port', <port>]), thenMcpHttpClientfetches a token fromhttp://127.0.0.1:<port>/tokenwith a form-encoded POST (drivers.ts:101-110), then POSTs JSON-RPC tohttp://127.0.0.1:<port>/mcpwithmcp-session-idheader (drivers.ts:241-243). 180 s timeouts. - Operator steps (host-side maintenance):
op('init')plussources,sync,extract,embedthroughrunCli. Operations observed viacall:remember,forget,get_page,search,recall,list_pages,get_timeline,get_links,query.
- Build:
- Coupling: CLI and MCP (stdio and HTTP) through spawned child processes. Postgres through Bun SQL. Not in-process for gbrain.
- Parity-gate fit: $0 and keyless, so it fits a parity gate on time budget. Spawns are many and slow, so wall time is the issue. Receipts are deterministic given fixed seeds.
lifecycle-lite (lifecycle-lite.ts, lifecycle-lite/)
- Entry file:
eval/runner/lifecycle-lite.ts. Helpers:lifecycle-lite/run.ts(drives anyMemorySystem: HTTP shim, in-process gbrain-shootout, or fake; writes rows and receipt),lifecycle-lite/score.ts(pure scoring, gold from generator ledger),lifecycle-lite/fakes.ts(honest reference and mutation fakes,HonestReferenceSystem,mutationFake). - What it measures: update and forget behavior (N1 correction chains, N5 canaries) for open-source memory systems, report-only. Gold from
eval/generators/lifecycle-lite-gen.ts(LIFECYCLE_LITE_SEEDS,sanitizerCorpus). - How it runs:
bun eval/runner/lifecycle-lite.ts --system <shim URL>|gbrain-shootout|fake|honest-reference|mutation:<kind> [--seeds 1,2,3,4,5] [--policy fixed-evidence|vendor-default] [--restart [--restart-cmd '<shell>']] [--gbrain <checkout>@<ref>] [--embed hash|real] [--qa reader|scripted] [--paid --budget-run-id <id>] --output <dir>. Default is hash embedder and scripted QA (keyless). - Cost and wall time: hermetic by default.
--paid --budget-run-idonly for gbrain-shootout (reranker and real embeddings), reader and judge. Timeouts--finish-timeout-s 600,--ingest-timeout-s 3600,--restart-timeout-s 900. Actual cost unknown. - LLM keys: none for hermetic mode. OPENAI_API_KEY, ANTHROPIC_API_KEY, VOYAGE_API_KEY for
--embed real,--qa reader, and the gbrain-shootout reranker. - Corpus: generated seeds (no dataset file).
- gbrain imports and calls:
- In-process
gbrain-shootout(lifecycle-lite.ts:107-119):importGbrain(gut, 'src/core/ai/gateway.ts')thenconfigureGateway(...),__setEmbedTransportForTests(test hook)../systems/gbrain.tsGbrainShootoutSystemandGbrainModulestype. systems/gbrain.ts(outside scope) calls on the engine:engine.connect,engine.initSchema,e.setConfig,e.now,engine.executeRaw,engine.deletePage,engine.disconnect(line numbers 67-122).lifecycle-lite.ts:118:Bun.resolveSync('ai', gut.root)then dynamicembedManyfrom the build'saipackage../gbrain-under-test.tsresolveGbrainUnderTest,importGbrain,productIdentityFor,overlaySummary.
- In-process
- Coupling: in-process for gbrain-shootout. HTTP protocol v1 for shims (
MemorySystemover HTTP). Restart:Bun.spawn(['bash', '-c', cmd])(lifecycle-lite.ts:191), e.g.bash eval/systems/bootstrap.sh restart --system <name>. Mutation fakes are in-process. - Parity-gate fit: the hermetic default with
--system honest-reference|fake|mutation:*is deterministic and fast. gbrain-shootout is not keyless with real embeddings.
lifecycle-report (lifecycle-report.ts)
- Entry file:
eval/runner/lifecycle-report.ts. Reads one or more lifecycle-experiment receipts (readFileSync) and prints Markdown tables per cell (CellSummaryshape: ack_lost, bijection, edges, timeline, forget, residue, leaks, hazards). With two or more receipts it reports whether each cell's summary matched across runs. - What it measures: nothing new. It summarizes receipts.
- How it runs:
bun eval/runner/lifecycle-report.ts <receipt.json> [more...]. - Cost and time: $0, under a second.
- LLM keys: none. Corpus: receipts only.
- gbrain imports: none. Coupling: none.
- Parity-gate fit: deterministic and fast.
LHF wave 2 attendance gate (lhf-wave-2-attendance-gate.ts)
- Entry file:
eval/runner/lhf-wave-2-attendance-gate.ts. No helpers. - What it measures: scores the LHF wave 2 attendance paraphrase gate from two N9 hermetic receipts (base and branch) on
world-v1-attendees, with extra splitsheldout-paraphraseandcompeting-intent. Counts fired attendance and false routes (attendedlink type routed on competing or other-template questions). Pre-registrationdocs/benchmarks/2026-10-16-lhf-wave-2-attendance-preregistration.md. Counts only, no statistics. - How it runs:
bun eval/runner/lhf-wave-2-attendance-gate.ts --base <receipt.json> --branch <receipt.json> --base-relations <json> --branch-relations <json> [--json]. Receipts come from N9 runs with--extra-split heldout-paraphrase=<jsonl> --extra-split competing-intent=<jsonl>. Relations come fromeval/runner/parse-relational-questions.ts(outside this scope, runs against each gbrain tree). - Cost and time: $0 and under a second. LLM keys: none (the N9 receipts have already run).
- Corpus: N9 receipt JSON (
data.one_hop.per_query). - gbrain imports: none. Coupling: none (reads JSON). The gbrain dependency is one level up, in
parse-relational-questions.tsrun against each tree. - Parity-gate fit: deterministic and fast.
batch lane (batch/)
- Entry files:
batch/w10.ts(build, plan, run per arm),batch/w10a-capture.ts(gbrain's owneval longmemevalat the pin with a stub reader that records each reader request),batch/w10-particle.ts(particle-1 replay of recorded W10 prompts),batch/w10-particle-second-judge.ts(second judge sensitivity),batch/w10-rescore.ts(keyless $0 rescore from committed rows). - Library:
submit.ts(BatchLane, exactly-once submission in a SQLite ledger viabun:sqlite: intent, submit, settle),transport.ts(OpenAI Batch/v1/filesand/v1/batches, Anthropic/v1/messages/batches, and each provider's token-count endpoint),ledger.ts(worst-case reservation, batch factor fromDOCUMENTED_BATCH_FACTOR),manifest.ts(buildManifest,freezeManifest,bodySha,MODEL_SETTINGS),sources.ts(request builders for W10 and R1/R2 replay,readerBody,seededSample,JUDGE_PROTOCOL,officialJudgePrompt),receipts.ts(offline scoring; rows join by question id),real-fetch.ts(keeps the rawfetchbefore guards). - What it measures: LongMemEval answer accuracy for several reader arms (Opus 5.5, Sonnet 5.5, GPT-6.1 Sol, particle-1), paired question by question, through provider batch APIs at a discount.
- How it runs:
bun eval/runner/batch/w10.ts build <arm>($0),plan <arm>($0, counts tokens),run <arm>(paid pilot, then the rest).bun eval/runner/batch/w10-rescore.ts($0 from committed rows). Particle:w10-particle.ts build($0) thenpreflight ... --paid --budget-run-id <id>. - Cost and wall time: reservations are worst case from counted input tokens, then settled from provider usage (
ledger.ts). Total cost and time unknown. - LLM keys: ANTHROPIC_API_KEY and OPENAI_API_KEY (
transport.ts,w10-particle-second-judge.ts). Keyless:w10-rescore.ts,buildandplanmodes. - Corpus: LongMemEval-S, and R1/R2
reranker-oncall logs (batch/sources.ts,reranker-on/r1|r2/calls.ndjson.gz, underdocs/benchmarks/2026-09-29-...). - gbrain imports (all relative to
eval/runner/batch/, so they resolve to the pinned package at repo-rootnode_modules/gbrain, not to a checkout under test):batch/w10.ts:34:sessionIdFromSlugfromnode_modules/gbrain/src/eval/longmemeval/metrics.ts.batch/w10a-capture.ts:19-21:configureGateway,buildGatewayConfig(src/core/ai/gateway.ts,build-gateway-config.ts) andrunEvalLongMemEval(src/commands/eval-longmemeval.ts). The call at:57passes aclientthat records requests.batch/sources.ts:23-26:haystackToPages(adapter.ts),renderChatBlock(sanitize.ts),sessionIdFromSlug(metrics.ts),buildReaderRequest,READER_MAX_SESSION_CHARS,resolveReaderConfig(reader.ts).batch/w10-particle.ts:124: dynamic import ofsrc/eval/longmemeval/{adapter,sanitize,metrics,reader}.tsfromroot.
- Coupling: in-process against the pinned gbrain package (not an overlay), HTTP to provider batch APIs, SQLite for the lane ledger. Not CLI or MCP.
- Parity-gate fit:
w10-rescore.tsandbuild/plandeterministic and keyless.runis paid.
BG1 battleground (bg1/)
- Entry:
bg1/verdict.ts(verdict --receipts <dir>... [--manifest <filled>] [--judge-check <summary>] --out <dir>),bg1/power.ts(MDD and sizing),bg1/pricing.ts(--out <file> [--n N] [--rates <json>], money model),bg1/judge-check.ts(judge validity on public dev data),bg1/launch.ts(pre-launch lease and budget check). - Helpers:
bg1/rule.ts(decision rule: paired t, Holm step, BG1_ALPHA,decideFamily),bg1/match.ts(deterministic value matchermatchAnswer, no model),bg1/exposure.ts(exposure(q, contextText, { rawText }): gold, stale, deciding line),bg1/types.ts(re-exportsmemory-qa/bg1-names.ts). - What it measures: BG1 changing-facts battleground: gbrain against comparators (Opus, Sonnet, particle;
q1/cells/bg1.tsnames). Preregistrationdocs/benchmarks/2026-11-09-changing-facts-battleground-preregistration.md. Exposure tells whether a loss was retrieval or reading. - How it runs:
bun eval/runner/bg1/verdict.ts verdict --receipts <dir>... --out <dir>;bun eval/runner/bg1/pricing.ts --out <file>;bun eval/runner/bg1/power.ts;judge-check(paid, public dev data, gpt-6-astra and claude-opus-5-5 as judges). - Cost and time:
pricing.tsgives the per-world money model and check-in projection. Actual campaign cost unknown here.judge-checkplan isJUDGE_CHECK_PLANinpricing.ts. - LLM keys: judge-check uses ANTHROPIC_API_KEY, OPENAI_API_KEY, PARTICLE_API_KEY, VOYAGE_API_KEY, and SHOOTOUT_PROXY (
bg1/judge-check.ts). Analysis modules (verdict,power,pricing,match,exposure,rule,launch) need no key. - Corpus: BG1 manifest
eval/decisions/datasets/bg1-manifest.json(committed, 1.2K),memory-qa/corpus.tsloadCorpusfor dev data, cell manifestq1/cells/bg1.ts. - gbrain imports: none direct.
judge-check.tsimportsmemory-qa/qa-v2.ts(ChatClient,judgeAnswer),metering-proxy.ts(MeteringProxy,keylessUpstream), andq1/cell.ts(metering,onSlot), which are outside scope. Coupling to gbrain sits in the q1 cells, not in bg1. - Coupling: none for analysis. HTTP to judge models through the metering proxy for judge-check.
launch.tsreads the cell manifest and asserts leases; it does not spawn or call gbrain. - Parity-gate fit:
verdict,power,pricing,match,exposureare deterministic and fast.judge-checkis paid and not a gate.
budgeted delivery (budgeted-delivery/)
- Entry points: shell
h1-run.sh(custody chain for H1 held-out),c8k-run.sh(8k conversion custody and dev reading),e2-keyless-gate.sh(E2 gate end to end, hash vectors, reranker off, stub proxy),keyless-gate.sh(E1 accounting gate for one benchmark),h1-dry-run.shandc8k-dry-run.sh(whole chain on invented fixtures),c8k-amend-7.sh(operator step for a spec change),e2-parallel.sh(memory-qa shards). TypeScript steps:h1.ts,c8k.ts,c8k-product.ts,c8k-retrieval-cache.ts,e1-readings.ts,e2-readings.ts,e2-gate-check.ts,e3-retrieval-gate.ts,accounting-gate.ts,budget-sizing.ts,drop-order.ts,export-receipts.ts,h1-fixture.ts,stub-proxy.ts. - What it measures: whether evidence packed under explicit token budgets (
off,cap_only, and packing arms) keeps reader accuracy. E1: arm scores and readings. E2: every packing againstoffandcap_onlyon LongMemEval-S 500. E3: session fusion against today's ranking (strict recall). 8k conversion: a held-out decision on an 8k context budget. H1: held-out custody run. Preregistrations indocs/benchmarks/2026-10-08,2026-10-09,2026-10-10,2026-10-26. - How it runs: keyless
bash eval/runner/budgeted-delivery/keyless-gate.sh,bash .../h1-dry-run.sh,bash .../c8k-dry-run.sh,bash .../e2-keyless-gate.sh. Live custody:bash eval/runner/budgeted-delivery/h1-run.sh <custody root>andc8k-run.sh <custody root>(run on the custodian's machine from the repo root at the registration commit). Everything sealed stays under the custody root; only<root>/export/and the access log leave it. - Cost and wall time: dry runs and gates are $0 (dummy keys, hash vectors, stub proxy; their numbers mean nothing).
budget-sizing.tsis $0. Live costs and time: unknown (leases per proxy in the shell scripts). - LLM keys: live runs need ANTHROPIC_API_KEY, OPENAI_API_KEY, VOYAGE_API_KEY (
c8k.ts,h1-dry-run.shsets dummy values).SHOOTOUT_PROXYis set for proxy leases inh1-run.shandc8k-run.sh. - Corpus:
h1-fixture.tsinvents twelve personas with LongMemEval-style timestamps (no sealed text). Live runs use sealed confirmation files under the custody root. LongMemEval-S for E1, E2 and E3. - gbrain imports and calls:
c8k-product.ts:58-83: dynamic load ofsrc/core/pglite-engine.ts(PGLiteEngine),src/core/import-file.ts(importFromContent), a tool dispatcher (dispatchToolCall) andrunExtractConversationFactsCore(fact extraction core).c8k-product.ts:102:new PGLiteEngine(), thene.connect,e.initSchema,e.disconnect(c8k.tsalso usese.disconnect).c8k.ts:37-39:importGbrain,resolveGbrainUnderTest(gbrain-under-test.ts)../systems/gbrain.tssourceOfSlugandslugOfine3-retrieval-gate.ts,h1.ts,c8k.ts,c8k-product.ts;./systems/gbrain-query/connector.tswireRequestandQUERY_PATH_PINS(outside scope).e3-retrieval-gate.ts,h1.ts:sourceOfSlugonly.
- Coupling: in-process (c8k-product on a PGLite brain from
GBRAIN_DIR), HTTP throughstub-proxy.ts(keyless) or the metering proxy in live runs, shell orchestration.accounting-gate.tsconsumes gbrain-query rows throughwireRequestand needs no gbrain code to be loaded. - Parity-gate fit: dry runs and gates deterministic, keyless, and useful as a fast parity check. Live steps are paid.
evaluator (evaluator/)
- No entry point. Library imported by runners. Files:
gold-store.ts(labels in a JavaScript private field, returns copies),allowlist.ts(boundary allowlists and leak checks by known substring),judge-inputs.ts(input allowlists for the reading-notes reader, Cat 29 answerer and judge, Cat 35 SUT and judges),reference-scorer.ts(independent metric implementations; no import from the product),longmemeval.ts(separate gold loader, gold-free view, allowlists; importstype Questionfrom../longmemeval.ts),cat13.tsandcat13-gold-loader.ts(Cat 13 gold frombuildProbes,loadCorpus,PROBE_SEEDin../cat13-conceptual.ts). - What it measures: nothing. It is the evaluator-side boundary for "no leakage of gold into the system under test" and independent scoring.
- How it runs: imported. Tests are the phase gates.
- Cost and time: $0. Keys: none. Corpus: loaded from corpus loaders (
cat13-conceptual.ts, LongMemEval dataset). - gbrain imports: none (no
gbrainimport, no spawn). Coupling: none. Pure. - Parity-gate fit: deterministic and fast.
failure causes (failure-causes/)
- Entry file:
failure-causes/cli.ts. Library:index.ts(stable interface for the nightly loop and weekly report, re-exports taxonomy). - Files:
taxonomy.ts(fixed list of causes in stage order, mapped to gap-map failure modes F1 to F16;renderTaxonomyMarkdown),classify.ts(deterministic rules by stage; returns primary cause and secondaries),label.ts(residual labeller: a model picks a cause only where the rules cannot place the failure; usesclient.chatwithChatLikefrommemory-qa/qa-v2.tsandjudge.ts'sfenceUntrusted),agreement.ts(labeller agreement check),attribution.ts(nightly QA miss buckets: harness, item unsupported, and others),report.ts(weekly summary: gbrain's top three ranked on natural-condition failures),manifest.ts(source manifests with SHA-256 per named input file, read from named roots),types.ts(EvalFailure,CauseLabel,StressCount),adapters/(nightly.ts,skillopt.ts,live-recall.ts,refresh-barrier.ts,rs1.ts,cat40.ts,shared.ts,index.ts). - What it measures: the distribution of causes for eval misses across sources, to rank what gbrain can fix. Not a benchmark of gbrain itself.
- How it runs:
bun eval/runner/failure-causes/cli.ts taxonomy [--write|--check];bun eval/runner/failure-causes/cli.ts classify --manifest <sources.json> --root <name>=<dir> ... --out <labels.ndjson> [--failures-out ...] [--stress-out ...] [--counts-out ...]. - Cost and time: deterministic classify is $0. The residual labeller is paid (model calls via
client.chat). Agreement check uses two labellers at high effort (Opus 5.5 and GPT-6 Astra, per the file header). Wall time unknown. - LLM keys: labeller only, through the
ChatLikeclient. The key names are not set in these files; the client is supplied by the caller (not traced here). - Corpus: archived rows named in source manifests (nightly, skillopt, live-recall, refresh-barrier, rs1, cat40 adapters). Files are read by name and SHA-256, never by glob.
- gbrain imports: none.
adapters/cat40.tsimports onlynode:path. Coupling: none. The adapters read rows; they do not call gbrain. - Parity-gate fit:
taxonomy --checkandclassify(without labeller) are deterministic and fast.
graph backend (graph-backend/)
- Entry:
graph-backend/run-tier.sh <tier> [stages...](stages:load-pg plan hydra-load calibrate arm-A arm-B arm-BL arm-C tput). Workers:worker.ts --arm A|B|BL|C --tier <tier> --pass <warm|cold> --plan <json> --out <ndjson>. Alsopg-load.ts,hydra-load.ts,plan.ts,generate.ts,analyze.ts(scores receipts, decision rule),canon.ts(canonical digests of walk results),hydra.ts(HydraDB HTTP client),stack.sh,ubicloud-setup.sh(infrastructure). - What it measures: latency and correctness of graph walks on fictional link graphs (
traverseGraph,traversePathsDetailed) across arms: A (gbrain old CTE, worktree atb1fdcb843), B (gbrain breadth-first, worktree atfc1121cae58, Postgres), BL (B on PGLite), C (HydraDB, hydrated from Postgres). GBRA-123. Tierspersonal,team,company(docs/benchmarks/2026-10-17-graph-backend/;summary.jsongenerated 2026-10-18). - How it runs:
GB_A=<gbrain worktree> GB_B=<gbrain worktree> OUT=<dir> bash eval/runner/graph-backend/run-tier.sh <tier> [stage...]. Stages are resumable with.donemarkers. Start counts:A_STARTS(default 100 at team and company),STARTS(default 500). Each start page gets a 22-call matrix. - Cost and time: no LLM. Latency data in the summary (for example personal pooled arm A/cold, p50 1.7 and p95 278 over 11,000 calls, units as recorded in summary.json). Wall clock for stages unknown.
- LLM keys: none. External services: HydraDB (
HYDRA_URL,HYDRA_TOKEN,HYDRA_VERIFY,HYDRA_LOAD_CONCURRENCY,HYDRA_CLIENT_CONCURRENCY) and a local Postgres admin URL for loading (run-tier.sh). - Corpus: fictional seeded graphs from
generate.ts(power-law hubs, page ids equal slug order). Generated, not committed as data (the plan and receipts are committed underdocs/benchmarks). - gbrain imports and calls (in-process, from
GB_ENGINE_DIRworktrees):worker.ts:41, 45: dynamic import of${engineDir}/src/core/pglite-engine.ts(PGLiteEngine) and${engineDir}/src/core/postgres-engine.ts(PostgresEngine). Methods called:engine.connect,engine.traverseGraph,engine.traversePathsDetailed,engine.disconnect.pg-load.ts:17, 21: the same two engine modules. Methods:engine.initSchema,engine.connect,engine.disconnect. Then runsANALYZE.analyze.ts:37: dynamic import of${GB_B}/vendor/postgres/src/index.js(vendored Postgres client, for queries).
- Coupling: in-process for gbrain walks (engine modules from two worktrees, selected by env, not by a CLI). HTTP to HydraDB (
hydra.ts,hydra-load.ts). Spawnsbun worker.tsfromrun-tier.sh. A CLI or MCP adapter would replacetraverseGraphandtraversePathsDetailedwith JSON calls. - Parity-gate fit: deterministic (fixed seeds, call matrix). Needs Postgres and worktrees, so not a fast keyless gate.
Cross-cutting observations
- Coupling is mostly in-process via dynamic
import(join(GBRAIN_DIR, 'src/...'))(evidence-delivery, external-ideas, lifecycle-lite, budgeted-delivery c8k-product, graph-backend workers, cat41 container driver). The loaderevidence-delivery/gbrain.tsloadGbrain(lines 85-100) is the shared entry point, and its module list is the de facto contract for every in-process study. - Batch lane imports point at repo-root
node_modules/gbrain(batch/w10*.ts,batch/sources.ts), which is the pinned package and not an overlay under test. These results describe the pin, not a candidate checkout. - Two coupling styles for the same goal:
lifecycle/drivers.ts(CLI and MCP through spawnedbun <build>/src/cli.ts, with copied builds fromlifecycle/builds.ts) is the only fully black-box driver family here. It is the closest model for a Rust-side CLI or MCP adapter. Its MCP framing is JSON-RPC over stdio or HTTP with protocolVersion2025-06-18. - MCP tool names in use:
entity,search,context_pack,put_page,get_page,recall,remember,forget,list_pages,get_timeline,get_links,query. CLI operations in use:init,sources,sync,extract,embed,call <op>,jobs work,beliefs enable --source vault,serve [--surface starter|full] [--http --port]. - Engine methods called in-process across this group:
connect,disconnect,initSchema,getChunks,getConfig,setConfig,unsetConfig,executeRaw,deletePage,traverseGraph,traversePathsDetailed,now(test seam), andbrain.disconnect. - Shared infrastructure outside this scope that every paid runner depends on:
budget-ledger.ts(startPaidRun,BudgetRun,ledgerStatus),paid-arm.ts(requirePaidArm),metering-proxy.ts(MeteringProxy),prereg.ts(attestPreregistration),receipt.ts(scrubMachinePaths),gbrain-under-test.ts(resolveGbrainUnderTest,importGbrain, overlays under.gbrain-overlays/),memory-qa/*andq1/*. Their behavior was not traced beyond these call sites. - Keyless modes exist for most groups (
--scripted, hash embedder,estimate,score,verdict,w10-rescore,failure-causes classify, dry-run and gate scripts). Paid modes are guarded by--paid,--budget-usd, or--budget-run-id, and the ledgers use SQLite. - Surprises:
cat40/gbrain-arm.tsandfacts-absorb-gate.tsboth reuseMcpClientfromcat40/gbrain-arm.ts, so changes to that file affect cat41, facts-absorb and cat40.lifecycle/slice.tshas four callers outside scope (n1-knowledge-update.ts,n5-forget-residue.ts,n6-postgres-http.ts,p5-brain.ts). - Coupling of
failure-causes/label.tsuses a caller-suppliedChatLikeclient; the provider key path was not traced. - Not verified: presence of
eval/data/model-ladder-v1and other named data directories, and actual costs and wall times for most paid runs. Those are reported as "unknown" or as the figure from the cited docs.