Garry's Opinionated OpenClaw/Hermes Agent Brain
Appendix to ../gbrain-evals.md. Detailed per-area notes from one research pass over gbrain-evals
cd447b3(v0.10.237) and gbrain6ce7195. Paths are relative to the gbrain-evals root unless markedgbrain:. "G1" to "G10" in the text name the appendix files listed in the main map.
G6 runner dirs B: gbrain-evals runners (live-recall through takes-bootstrap)
Source root: the gbrain-evals checkout (HEAD cd447b3, Release v0.10.237). Paths below are relative to that root. Scope: eval/runner/{live-recall, memory-qa, memory-trust, messy-company, miss-mining, nightly, outcomes, predictive, providers, refresh-barrier, rs1, skillopt-method, sota, strict-pageless, system-one, systems, takes-bootstrap}, eval/runner/mcp-contract.ts, eval/runner/tool-bridge.ts, docs/nightly.md, scripts/nightly-.sh, scripts/refresh-barrier-.ts. Method: header and usage reads, rg for gbrain imports, spawns, JSON-RPC and tool names. Not every file was read in full. Files under cat40/ (McpClient, execGbrain-like helpers, loop.ts) and lifecycle/drivers.ts are outside this scope and were read only where a scoped file calls them.
Summary table
| Runner | Entry path | Measures | Run command | Tier / time / cost | LLM keys | Corpus | Coupling |
|---|---|---|---|---|---|---|---|
| BrainBench Cat 12 MCP contract | eval/runner/mcp-contract.ts | Remote vs local trust boundary and limits on gbrain operation handlers (depth cap, list clamp, search mode, injection) | bun eval/runner/mcp-contract.ts [--json] |
offline; seconds; $0 | none | generated in-run (~150 pages) | in-process (PGLiteEngine, OPERATIONS handlers) |
| Tool bridge (library, no runner) | eval/runner/tool-bridge.ts | Adapts read ops to Anthropic tool defs; executes them with remote:true | imported by Cat 8/9 agent loops (outside scope) | n/a | none | n/a | in-process (gbrain/operations) |
| Nightly gbrain check | eval/runner/nightly/run.ts (+ tool-contract.ts, smokes/, worker.ts, adapters.ts, bisect.ts) | Dev-set pass rates per suite, cost per correct answer, regressions vs previous night | bun eval/runner/nightly/run.ts --ref master --tier matrix --fake --out <dir> --gbrain-clone <path> (docs/nightly.md:251) |
--fake keyless; --paid needs ledger run; wall time unknown |
only for model suites (--paid) |
eval/data dev-set, docs/benchmarks inputs (exported by scripts/nightly-export.sh) | mixed: CLI (smokes), in-process (tool-contract, worker), HTTP (serve --http health only) |
| Nightly tool-contract replay | eval/runner/nightly/tool-contract.ts | Refusals of recorded frontier tool calls on rollout, write-capture and MCP dispatch paths | bun eval/runner/nightly/tool-contract.ts --gbrain-root <dir> --out <file.json> [--families sol,opus] [--skills query,briefing,fact-check] |
keyless; wall time unknown | none | docs/benchmarks/2026-11-01-skillopt-method/probe/frontier | in-process (dynamic import of build src/mcp/dispatch.ts) |
| Nightly system smokes | eval/runner/nightly/smokes/*.ts | Seven keyless CLI and serve behaviours (upgrade, sync, ingest retry, Postgres 15) | via nightly/run.ts system-smokes suite |
keyless; unknown | none | seeded synthetic brain | CLI (spawn bun src/cli.ts), HTTP (serve --http, GET /health) |
| Refresh-barrier validation | eval/runner/refresh-barrier/run.ts (+ trial.ts, child.ts, agent.ts, runtime.ts, schedule.ts, analyze.ts) | Refresh write barrier: write landing, refusals, concurrency, quiet latency; agent-level writes | bun eval/runner/refresh-barrier/run.ts --block trials|named|quiet --engine postgres|pglite --gbrain-root <G> --out <dir> |
paid agent caps: opus $65, sol $35, smoke $5 (refresh-barrier/agent.ts:72-76); trials keyless; wall time unknown | Anthropic, OpenAI for agent block | seeded synthetic git fixture (fixture.ts) | in-process (dispatchToolCall, PGLite/Postgres engines), child processes (bun child.ts) |
| Refresh-barrier helpers | scripts/refresh-barrier-b1-guard.ts, scripts/refresh-barrier-partc-adjudicate.ts | Idle-machine guard; Part C write adjudication | bun scripts/refresh-barrier-b1-guard.ts --out <dir> -- <cmd> |
n/a | none | n/a | none (scripts) |
| Live recall | eval/runner/live-recall/run.ts (+ task-arms.ts, tasks.ts, brain.ts, synthetic.ts, scorecard.ts) | recall@k and retrieval arms A/B/C/D on private questions | bun eval/runner/live-recall/run.ts run --fixture or run --synthetic <dir>; freeze; --paid for Voyage |
--paid cost from synthetic.ts estimateVoyageUsd; time unknown |
Voyage (embedding, rerank), Anthropic/OpenAI for readers | private brain on the box (not committed) or synthetic dir | in-process for embeddings; MCP-shaped dispatch (dispatchToolCall) |
| Memory QA (arms) | eval/runner/memory-qa/run.ts + run-systems.ts (+ corpus.ts, qa.ts, arms.ts, beam10m-*.ts) | LoCoMo, LongMemEval-S, BEAM accuracy per gbrain build | bun eval/runner/memory-qa/run.ts --benchmark locomo|lme-s|beam-100k|beam-1m|fixture [--embed hash|real] [--paid --budget-run-id <id>] |
--embed hash and --qa none keyless; paid estimate in code (run.ts:572); wall time unknown |
QA reader and judge when --qa reader |
benchmark datasets (eval/data), generated synthetic corpora | in-process (PGLite, hybridSearch, importFromContent, runThink) |
| Memory trust flag-rate (amendment 5) | eval/runner/memory-trust/flag-rate.ts | How often the write gate flags ordinary use; review queue size on LongMemEval-S users | bun eval/runner/memory-trust/flag-rate.ts --out <dir> [--users 16 --control-users 4 --inject 5] [--smoke --keyless] |
paid budget via startPaidRun; time unknown | Anthropic/OpenAI for agent | LongMemEval-S haystacks (--lme) |
MCP over stdio (serve --surface full) + CLI |
| Memory trust sut / repro / utility guard | eval/runner/memory-trust/sut.ts, repro.ts, utility-guard.ts, agent.ts, flag-rate-score.ts, join-cat37.ts | Cat 37/38/39 trust checks on a hermetic PGLite brain; finding repros; LongMemEval utility | bun eval/runner/memory-trust/repro.ts --gbrain <checkout>@<ref> <finding> |
repro keyless; utility guard paid | utility guard: reader and judge keys | synthetic brain; LongMemEval-S | in-process (operationsByName handlers), CLI capture for utility-guard |
| Messy company loader and scorers | eval/runner/messy-company/load.ts, score-*.ts, common.ts | Ownership windows, timelines, aliases, document status, entity mentions on messy-company-v1 | bun eval/runner/messy-company/load.ts --split dev|heldout|all [--db <dir>] [--json]; score-*.ts <system output.json> |
keyless; seconds | none | eval/data/messy-company-v1/corpus/ (committed) | in-process (load only: importFromContent, PGLiteEngine); scorers none |
| Miss mining | eval/runner/miss-mining/weekly-counts.ts, to-live-recall.ts, schema.ts, custody.ts | Private miss log to counts and live-recall questions | bun eval/runner/miss-mining/weekly-counts.ts --log <log.jsonl> --week <date> --out <json> |
keyless | none | private miss log (not in repo) | none |
| Outcomes v3 | eval/runner/outcomes/v3.ts, rescore-a4.ts, validate-labeler.ts | Versioned answer-outcome labels (abstain, hedge, commit) beside frozen scorer | bun eval/runner/outcomes/rescore-a4.ts <receipt.json> ($0) ; validate-labeler.ts --paid ... |
rescore $0; labeler paid | labeler: model ladder keys | A4 receipts; labeler samples (custodian folder) | none (pure scoring) |
| Predictive measures | eval/runner/predictive/run.ts (+ cli-arm.ts, write-contract.ts, episodes.ts, out-of-view/, scale-latency/, ubicloud/) | Agent behaviour on CLI and MCP surfaces: write landing, call contract, reach, latency | bun eval/runner/predictive/run.ts ... (see predictive/README.md); scale-latency via ubicloud/run.sh |
keyless grammar; paid episodes; wall time unknown | Anthropic/OpenAI for agent episodes | synthetic brain built per run | CLI (execFile bun src/cli.ts) + MCP (gbrain serve stdio) + in-process brain build |
| Provider transport | eval/runner/providers/openai-compatible.ts | Request/response/error shape for Particle and Gemini judge endpoints | imported by reader and judge runners | n/a | Particle, Gemini keys at call time | n/a | none |
| RS1 verdict | eval/runner/rs1/verdict.ts, iut.ts, power.ts, pricing.ts, paid-window.ts | RS1 pre-registered decision rule (gbrain vs memory-bank vs extract-first) | bun eval/runner/rs1/verdict.ts verdict --receipts <dir>... |
$0 analysis; pricing and paid-window are guards | none directly | receipts from cell.ts (outside scope) | none (pure analysis) |
| SkillOpt method | eval/runner/skillopt-method/*.ts | SkillOpt optimizer headroom, one-shot, scoring, gate on gbrain skills | headroom.ts, one-shot.ts, score.ts (see file headers) |
frontier probe paid; test seam for chat transport | Anthropic (claude-opus-5-5), OpenAI (gpt-6.1-sol) | skillopt-method-v2/v3 dataset (docs/benchmarks) | in-process (gbrain/core/skillopt, ai/gateway, pglite-engine, operations) |
| SOTA loop | eval/runner/sota/*.ts (+ ubicloud/) | Monthly gbrain minus leader (whole-history reader, grep agent) on dev and sealed questions | bun eval/runner/sota/leader.ts plan --cache <dir>; oracle.ts run ...; scorecard.ts --rows ... |
paid via q1 cells (outside scope) | Anthropic, OpenAI, Particle | BEAM-1M and LongMemEval-S outcome cache | indirect (memory-qa reader via q1 cells); scoring none |
| Strict-pageless cell | eval/runner/strict-pageless/agent.ts (+ items.ts, analyze.ts, report.ts) | Does a remember for an entity with no page survive a rebuild from Git; is the reply truthful | bun eval/runner/strict-pageless/agent.ts --model <p:id>|fake --gbrain-root <checkout> --items 1-24 --k 1 --out <dir> |
paid (budgetOptionsFrom); --model fake keyless |
Anthropic, OpenAI when not fake | refresh-barrier fixture, synthetic | MCP-shaped dispatch (dispatchToolCall, remote stdio) + in-process sync/extract |
| System One (Jev) per-slot evals | eval/runner/system-one-jev.ts (+ slots.ts, datasets.ts, recount.ts) | Nine gbrain System One decision slots, matched pair per slot | bun eval/runner/system-one-jev.ts verify / plan / run |
paid (typesafe:jev-1.13.0); plan prints commands | TypeSafe Jev key | LongMemEval / dataset runners | CLI (spawnSync bun src/cli.ts, env allowlist) |
| Memory-system shootout (systems) | eval/runner/systems/*.ts | Adapter layer: gbrain in process, gbrain-agent, gbrain-query connector, neutral MCP tools, HTTP systems | consumed by memory-qa and shootout drivers | keyless for gbrain legacy; baselines paid | per baseline | shootout datasets | in-process (gbrain.ts, gbrain-agent.ts, gbrain-query), MCP-shaped neutral adapter (mcp-tools.ts), HTTP client (http.ts, non-gbrain) |
| Takes bootstrap overlay | eval/runner/takes-bootstrap/harness-overlay.mjs (+ .diff) | gbrain takes-bootstrap harness with price overrides, summary and request log | bun eval/runner/takes-bootstrap/harness-overlay.mjs --gbrain-dir <tree> ... |
paid (provider requests logged) | provider key for the model under test | gbrain's evals/takes-bootstrap corpus | in-process (loads gbrain harness and src/ from --gbrain-dir) |
MCP and CLI surface (priority section)
This section seeds the adapter-surface recommendation. Two kinds of call exist:
- Real MCP wire calls: a child process runs
bun <build>/src/cli.ts serve ...and the client speaks JSON-RPC 2.0 over newline-delimited stdio. - In-process "MCP-shaped" calls: gbrain's own
dispatchToolCall(engine, name, args, opts)fromsrc/mcp/dispatch.tsis imported and called, returning{content:[{type:'text',text}], isError?}. No protocol framing or process boundary is crossed. Several runners label this "as gbrain serve receives them".
Server launch (real wire)
| Caller | Command | Transport | Notes |
|---|---|---|---|
| McpClient in eval/runner/cat40/gbrain-arm.ts:93 (used by predictive/cli-arm.ts:232, memory-trust/flag-rate.ts:156, predictive/out-of-view/agent.ts) | spawn('bun', [<buildDir>/src/cli.ts, 'serve', ...args], {env, cwd: GBRAIN_HOME}) |
stdio, newline JSON-RPC | args e.g. ['--surface','full'], ['--surface','starter'], ['--access','read-only'] |
| McpClient warm start (cat40/gbrain-arm.ts:482) | same | stdio | waits for reply to one read-only entity call before timing |
| nightly/smokes/harness.ts:174 | spawn('bun', ['--no-env-file', <root>/src/cli.ts, 'serve', '--http', '--port', N, ...]) |
HTTP | ready when GET http://127.0.0.1:<port>/health returns 200 |
| nightly/smokes/harness.ts:149 | writes a shim bin/gbrain = exec bun --no-env-file <root>/src/cli.ts "$@" |
CLI | used for upgrade-from-previous installs |
JSON-RPC methods and shapes (McpClient, cat40/gbrain-arm.ts:60-145)
| Method | Params sent | Result read |
|---|---|---|
initialize |
{protocolVersion:'2025-06-18', capabilities:{}, clientInfo:{name:'gbrain-evals-cat40', version:'1'}} (line 120) |
init.error checked; instructions taken from the result |
notifications/initialized |
none (notification, no id) | none |
tools/list |
{} |
list of {name, description, inputSchema} (listTools, line 137) |
tools/call |
{name, arguments} (line 143) |
{content:[{type:'text',text}], isError?}; text blocks joined by newline |
server notification notifications/tools/list_changed |
n/a | triggers relist (line 105) |
Error and invalid-JSON lines are skipped (not JSON-RPC, line 104). Per-call timeout is callTimeoutMs; init timeout defaults to 180 s (line 91).
MCP tool names called (by file)
| Tool | Called from | Request arguments observed | Response assertions |
|---|---|---|---|
search |
cat40 McpClient call (outside scope), predictive/run.ts (call op) |
{query} |
text result |
query |
live-recall/task-arms.ts:186 (in-process), live-recall/tasks.ts (.call('query'), memory-trust/repro.ts:41, nightly/smokes via CLI |
{query, limit, expand:false forced} |
_meta.retrieval via emitResponseMeta; meta.degraded[].stage |
get_page |
live-recall/task-arms.ts:210, predictive/out-of-view/agent.ts (.call('get_page'), refresh-barrier/agent.ts (call('get_page'), refresh-barrier/runtime.ts (call(s, sourceId, 'get_page'), nightly/tool-contract.ts (REPLAY_SLUG) |
{slug} |
text contains "slug" when readable; isError otherwise |
get_backlinks |
nightly/tool-contract.ts (replayed), skillopt rollout registry | {slug} |
refusal classification only |
traverse_graph |
nightly/tool-contract.ts (replayed), mcp-contract.ts | {slug, depth, direction?} |
GraphPath rows {from_slug,to_slug,depth}; depth cap 10 |
list_pages |
mcp-contract.ts | {limit} |
remote clamp at REMOTE_LIST_PAGES_CAP=100 |
remember |
refresh-barrier/runtime.ts (remember), strict-pageless/agent.ts, memory-trust/flag-rate.ts (.call('remember'), memory-trust/repro.ts |
{fact, entity, provenance, request_id} or {items:[...]} |
response items[].page or page; notice text entity_page_missing |
put_page |
refresh-barrier/runtime.ts (MCP path; put_blind goes via CLI submitPageMutation), memory-trust/flag-rate.ts, memory-trust/repro.ts |
{slug, content, force?, request_id} |
retry_after_ms, write_request.state, code, retryable/class |
get_write_request |
refresh-barrier/runtime.ts (poll), refresh-barrier/agent.ts | {request_id} |
state in committed, conflict, failed, cancelled (pending otherwise) |
submit_answer |
refresh-barrier/agent.ts, strict-pageless/agent.ts, write-contract (predictive) | Anthropic tool input, not gbrain | not a gbrain tool; it is the eval's own tool |
get_chunks |
live-recall/task-arms.ts:202 (count chunk_index) | {slug} |
count of "chunk_index" matches |
context_pack, recall |
memory-trust/repro.ts | per finding | per finding |
entity |
predictive/scale-latency/probe.ts (OPS), cat40 coverage probe | {name} |
parsed by parseCoverage |
sync |
predictive/scale-latency/probe.ts (call('sync') |
n/a | n/a |
Tool names in tool-bridge.ts (EXPOSED_READ_TOOLS, line 41): search, query, get_page, list_pages, get_backlinks, get_links, get_timeline, get_tags, traverse_graph, resolve_slugs, get_chunks, get_stats. Dry-run tools (line 59): dry_run_put_page, dry_run_add_link, dry_run_add_timeline_entry. Mutating ops are rejected with ForbiddenOpError (line 73).
Live-recall TOOL_NAMES (task-arms.ts:33): query, search, get_page. GBRAIN_AGENT_TOOLS (systems/gbrain-agent.ts:49): same three. Server-side tool defs are generated by buildToolDefs(chosen) from src/mcp/tool-defs.ts and filtered: expand, source_id, source_ids are dropped (DROPPED_PARAMS in task-arms.ts, expand forced false for query).
Request and response shapes asserted
- Dispatch context used everywhere in MCP-shaped calls:
{remote: true, transport: 'stdio', sourceId, admissionOrigin: 'interactive', takesHoldersAllowList: ['world'], surface: 'full'}(systems/gbrain-agent.ts:226; refresh-barrier/runtime.ts:88; strict-pageless/agent.ts:95). Live-recall addsmetaHook. - CLI-path write context (refresh-barrier/runtime.ts:71):
{engine, sourceId, remote:false, dryRun:false, admissionOrigin:'interactive', config:{engine, embedding_disabled:true}}, callings.g.pageMutations.submitPageMutation(ctx, {operation:'put_page', params})in process. - memory-trust/sut.ts uses
remote: true, transport: 'http'and an auth object{token:'', clientId, principal:{kind:'legacy_token'}, scopes:['read','write']}but invokesoperationsByName[name].handler(ctx, {request_id, ...params})in process, with no HTTP. - Error envelope parsed (refresh-barrier/agent.ts:269-275): text starts with
Error:then a JSON object. classifyResult (runtime.ts:39-47) readscode,retryable,class==='retryable',retry_after_ms(orretry_after_ms=Nindetail),write_request.state. - Refusal shape (nightly/tool-contract.ts:51):
{code?, message, suggestion?, fix?: {mcp?: {tool, arguments}}}. A refusal whosefix.mcp.arguments.request_idnames the same tool is replayed with the same request id (refresh-barrier/agent.ts:253-259). - Live-recall access check (task-arms.ts:~120): when
caller.appliedAccessSummaryexists, every response must carry_meta.accesswithmember: null,groups: [],audience: null,relaxation: 'none'; otherwise HarnessError. - mcp-contract.ts assertions: traverse_graph depth=1000 frontier exactly 10 hops (checkDepthCap, line 194); depth=5 honored; remote default direction=both; list_pages limit=1M remote clamped to 100 (line 351-359); search with unknown
moderejected locally and ignored remotely (line 384-387); 10 MB query returns within 5 s (line 460-466); injection payloads not silently accepted (line 443-450).
CLI commands and JSON fields parsed
Launch: every CLI call is bun --no-env-file <build>/src/cli.ts <argv...> via execFile or spawn, shell:false, with a hermetic env (GBRAIN_HOME set, no provider keys). Source: predictive/cli-arm.ts:102 (execGbrain), nightly/smokes/harness.ts:108, refresh-barrier/trial.ts:156, memory-trust/flag-rate.ts (runCli from lifecycle/drivers.ts, outside scope).
| CLI command | Caller | Output parsed |
|---|---|---|
init --pglite --path <dir> --embedding-model <m> --non-interactive |
memory-trust/flag-rate.ts:149 | exit code only |
init --pglite --non-interactive --content-root <dir> --git |
predictive/cli-arm.ts:144 | exit code |
put <slug> [--force] [--wait-ms N] with markdown on stdin |
predictive/cli-arm.ts:147; nightly/smokes/postgres-15.ts:112 | exit code; error text |
put <slug> --source-id <id> --expected-revision <rev> |
nightly/smokes/ingest-retry-across-upgrade.ts:41 | exit code |
sync [--source <id>] [--no-pull] |
predictive/cli-arm.ts:151; nightly/smokes/* | exit code; sync status --source X --json gives last_error.code (source-pinned-sync.ts:40) |
sync status --source <id> --json |
nightly/smokes/source-pinned-sync.ts:40 | last_error?.code |
get <slug> [--source <id>|--source-id <id>] --json |
nightly/smokes/source-pinned-sync.ts:25, ingest-retry-across-upgrade.ts:95, upgrade-from-previous.ts:59 | JSON page |
search <q> --json |
nightly/smokes/source-pinned-sync.ts:44; upgrade-from-previous.ts:65 | JSON array with slug |
query <q> --limit N --json |
nightly/smokes/query-contract.ts:18 | JSON hits |
recall <slug> --json, recall --since <d> --limit N --json |
nightly/smokes/query-contract.ts:13; predictive/cli-arm.ts:193 | JSON |
list --limit N --offset M --json |
nightly/smokes/upgrade-from-previous.ts:87 | JSON array |
export --dir <dir> |
predictive/cli-arm.ts:183 | files on disk, then read as snapshot |
call <op> <json-args> |
predictive/run.ts:252 (execGbrain(brain.gut, brain.env, ['call', op, JSON.stringify(args)])); predictive/cli-arm.ts:272 |
stdout+stderr text; error code from Error [code], "code":"...", or "error":"..." (predictive/write-contract.ts:71); exit code |
sources writer status --json |
nightly/smokes/upgrade-from-old.ts:56, 114 | JSON |
sources writer claim <src> --path <repo> --admin-intent writer_claim --expected-state <s> |
upgrade-from-old.ts:119 | exit code |
sources writer activate --confirm-quiesced [--dry-run] --json |
upgrade-from-old.ts:122-124 | JSON |
sources retry-held <src> --dry-run --json |
nightly/smokes/managed-sync.ts:44 | items[].key, items[].code |
sources set-path <src> <repo> --json |
nightly/smokes/source-pinned-sync.ts:32 | JSON |
sources add vault --path <p> |
cat40 (outside scope) | exit code |
config set <key> <value> |
nightly/smokes/managed-sync.ts:39 | exit code |
jobs work [--concurrency N] |
memory-trust/flag-rate.ts:182; nightly/smokes/ingest-retry-across-upgrade.ts:87 | exit code |
jobs get <id> --json, jobs list --limit N --json |
ingest-retry-across-upgrade.ts:89, 102 | JSON status fields |
trust review --json |
memory-trust/flag-rate.ts:185-187 | JSON after first { |
serve (stdio) / serve --http --port N |
see Server launch | GET /health 200 |
system-one decide flags |
system-one-jev.ts:301 via spawnSync | JSON execution block with package, declared_pin, loaded_git_head, package_sha256 (system-one-jev.ts:213) |
In-process imports used as an MCP/CLI substitute
These runners bypass the wire and import gbrain source directly. They are the parts an adapter must replace.
| Runner | Import (dynamic from build root) | Symbols and methods called |
|---|---|---|
| mcp-contract.ts | gbrain/operations, gbrain/pglite-engine, gbrain/config |
operations (find by name, op.handler(ctx, params)); PGLiteEngine (connect, initSchema, setConfig, putPage, addLink, disconnect) |
| tool-bridge.ts | gbrain/operations, gbrain/engine, gbrain/config |
operations (buildToolDefs maps ParamDef), op.handler |
| nightly/tool-contract.ts | <root>/src/mcp/dispatch.ts, src/core/pglite-engine.ts, src/core/import-file.ts, src/core/ai/gateway.ts, src/core/skillopt/index.ts |
dispatchToolCall(engine, tool, args, {remote, transport, sourceId}); PGLiteEngine (connect, initSchema, setConfig); importFromContent(engine, slug, content, {noEmbed}); configureGateway; runRollout |
| refresh-barrier/runtime.ts, agent.ts | <root>/src/core/config.ts, src/core/operation-manifest.generated.ts, src/mcp/surface.ts, src/mcp/server.ts, src/mcp/tool-defs.ts, src/mcp/validate-params.ts, src/mcp/instructions.ts, src/mcp/initialize-context.ts, src/core/facts/writeback-config.ts, src/core/facts/writeback-settings.ts, src/version.ts, src/mcp/dispatch.ts, src/core/pglite-engine.ts, src/core/postgres-engine.ts, src/commands/sync.ts, src/core/cycle/extract-facts.ts, src/core/persistence/service.ts |
dispatch.dispatchToolCall; pageMutations.submitPageMutation; buildToolDefs(listed, {strictParams}); resolveMcpInstructions; instructionReadiness(engine, cfg, 'stdio'); persistence service disposePersistenceConsumer; PGLiteEngine and PostgresEngine (connect, initSchema, disconnect) |
| live-recall/task-arms.ts | src/mcp/dispatch.ts, src/core/operations.ts, src/mcp/tool-defs.ts, src/mcp/server.ts, src/core/ai/invocation-guard.ts, src/core/search/hybrid.ts, src/core/search/private-visibility.ts, src/core/access/caller.ts (optional) |
dispatchToolCall; buildToolDefs(chosen); resolveMcpStdioSourceScope(engine); withAIInvocationGuard(guard, run) (allows only embedding and rerank kinds); appliedAccessSummary |
| live-recall/synthetic.ts, run.ts, brain.ts | src/core/ai/gateway.ts, src/core/ai/build-gateway-config.ts, src/core/config.ts, src/core/engine-factory.ts, src/core/import-file.ts, src/core/pglite-engine.ts, src/core/search/eval-pool-depth.ts, src/core/search/sql-ranking.ts, src/core/fts-language.ts |
configureGateway; importFromContent; PGLiteEngine; LATEST_VERSION |
| memory-trust/sut.ts | src/core/pglite-engine.ts, src/core/operations.ts (operationsByName), src/core/token-mint.ts (mintLegacyToken), src/core/persistence/identity.ts (registerLocalWriter, readLocalWriter), src/core/persistence/service.ts (disposePersistenceConsumer), src/core/ai/gateway.ts (configureGateway, resetGateway), src/core/import-file.ts (importFromContent), src/core/context/turn-context.ts (assembleTurnContext), src/core/trust/channel.ts (optional) |
engine.executeRaw (SQL on persistence_brain, sources, information_schema); engine.setConfig; operationsByName[name].handler(ctx, {request_id, ...params}) for remote (transport:'http') and local (remote:false) callers |
| memory-trust/flag-rate-score.ts | <buildDir> via bun -e |
src/core/write-gate.ts (dynamic) |
| memory-trust/utility-guard.ts | src/commands/eval-longmemeval.ts, src/core/ai/build-gateway-config.ts, src/core/ai/gateway.ts |
gbrain's own eval longmemeval command, with the recording stub reader |
| strict-pageless/agent.ts | src/core/pglite-engine.ts, src/commands/sync.ts (performSync), src/core/cycle/extract-facts.ts (runExtractFacts) |
dispatchToolCall; engine.executeRaw on pages; performSync; runExtractFacts |
| predictive/out-of-view/reach.ts | src/core/import-file.ts, src/core/pglite-engine.ts, src/mcp/dispatch.ts |
importFromContent; dispatchToolCall |
| predictive/scale-latency/build.ts | src/core/ai/defaults.ts, src/core/ai/gateway.ts, src/core/engine-factory.ts, src/core/pglite-engine.ts |
configuration and engine construction |
| memory-qa/run-systems.ts, run.ts | PGLiteEngine, hybridSearch, importFromContent, runThink, DEFAULT_RERANKER_MODEL |
hybridSearch with expansion off (systems/gbrain.ts header) |
| messy-company/load.ts | gbrain/import-file (importFromContent), gbrain/pglite-engine (PGLiteEngine) |
importFromContent(engine, slug, content, {noEmbed}) |
| skillopt-method/*.ts | gbrain/core/skillopt (runRollout, scoreSkillPerTask, runSkillOpt, rolloutToolsForSkill, describeJudges, SKILLOPT_HOOK_BUDGET_EXHAUSTED, BenchmarkTask type, ScoreOnTasksOpts type), gbrain/ai/gateway (chat, toolLoop, configureGateway, __setChatTransportForTests, ChatOpts, ChatResult), gbrain/ai/invocation-guard (withAIInvocationGuard), gbrain/pglite-engine (PGLiteEngine), gbrain/operations (operations), gbrain/import-file (importFromContent) |
rollout, scoring and chat; __setChatTransportForTests is a stub seam |
| systems/gbrain-agent.ts | ../gbrain-under-test.ts (gbrainSpecFrom, importGbrain, resolveGbrainUnderTest, productIdentityFor), src/mcp/dispatch.ts (dispatchToolCall), src/mcp/server.ts (MCP_STDIO_MAX_MESSAGE_BYTES), src/core/operation-manifest (OPERATION_MANIFEST) |
mcpToolArm('gbrain', {listTools, callTool}, GBRAIN_AGENT_TOOLS) |
| systems/gbrain.ts | gbrain/pglite-engine, gbrain/import-file, gbrain/search/hybrid (per header) |
hybridSearch, importFromContent, PGLite schema setup |
Coupling classification (the runner's gbrain contact)
| Runner or file | Classification | Evidence |
|---|---|---|
| mcp-contract.ts | in-process | import ... from 'gbrain/operations'; runOp calls op.handler(c, params) (mcp-contract.ts:156-167) |
| tool-bridge.ts | in-process | import ... from 'gbrain/operations'; op.handler(ctx, safeInput) (tool-bridge.ts ~line 440) |
| nightly/tool-contract.ts | in-process (MCP-shaped) | dynamic import of <root>/src/mcp/dispatch.ts; dispatchToolCall with {remote:true, transport:'stdio'} |
| nightly/smokes/*.ts | CLI and HTTP (real process) | spawn bun src/cli.ts; serve --http plus GET /health |
| nightly/worker.ts, adapters.ts, bisect.ts | in-process plus spawn | worker imports src/core/postgres-engine.ts; adapters spawn bun (spawn(o.cmd[0], ...), line 84) and cat40 runner commands |
| refresh-barrier | in-process plus child processes (no gbrain serve) |
dispatchToolCall; submitPageMutation; spawn('bun', [child.ts, ...]) (trial.ts:156). Comment at child.ts:3 says each agent's gbrain serve session is emulated as a resident process |
| live-recall | in-process MCP-shaped | dispatchToolCall on phase-1 engine (task-arms.ts:186) |
| memory-qa | in-process | PGLite, hybridSearch, importFromContent |
| memory-trust/flag-rate.ts | MCP (real wire) plus CLI | McpClient serve --surface full (flag-rate.ts:156); runCli for init, jobs work, trust review --json |
| memory-trust/sut.ts, repro.ts | in-process | operationsByName[...].handler |
| memory-trust/utility-guard.ts | in-process | eval-longmemeval import |
| messy-company | in-process (load only) | importFromContent; scorers none |
| miss-mining, outcomes, rs1, providers, sota scorers | none | pure data, scoring or transport; no gbrain import |
| predictive | CLI (real), MCP (real) plus in-process build | execGbrain (bun src/cli.ts, execFile); McpClient serve; brain.gut import |
| skillopt-method | in-process | gbrain/core/skillopt, gbrain/ai/gateway |
| strict-pageless | in-process MCP-shaped | dispatchToolCall (agent.ts:95) |
| system-one-jev.ts | CLI (real) | spawnSync bun argv (system-one-jev.ts:301) |
| systems/gbrain*.ts | in-process | gbrain.ts legacy adapter; gbrain-agent.ts (header line 13: "In process rather than a gbrain serve subprocess") |
| systems/http.ts | HTTP to a non-gbrain system | HttpMemorySystem; wire protocol in eval/systems/PROTOCOL.md (not read) |
| takes-bootstrap overlay | in-process (gbrain harness loaded from tree) | harness-overlay.mjs:54 --gbrain-dir, default node_modules/gbrain |
Per-runner sections
Adapter-relevant runners (MCP/CLI or in-process gbrain)
eval/runner/mcp-contract.ts (BrainBench Category 12)
- Entry: eval/runner/mcp-contract.ts (main at line 492). Helpers: eval/runner/gbrain-version.ts, probe-accounting.ts.
- Measures: whether gbrain operation handlers enforce the trust boundary (local vs remote) and limits. Question: does a remote caller get tighter behaviour, and do caps and injection checks bite.
- Run:
bun eval/runner/mcp-contract.ts [--json]. Output receipt viareceiptPath('mcp-contract'). - Cost and time: unknown; no network. Hermetic PGLite with about 150 pages (RING 10, CHAIN 16, NOTE 110).
- LLM keys: none. Keyless by construction (
search.mcp_keyword_only=true, setup at line 100-110). - Corpus: generated in-run (setupEngine).
- Metrics: pass/fail assertions (checkDepthCap, checkRemoteDirectionDefault, checkRemoteListClamp, checkListPagesTrustMatrix, checkModeDifferential) plus a handler sanity walk (
handlerSanityWalk, line 485). - gbrain imports:
gbrain/operations(operations;op.handler),gbrain/pglite-engine(PGLiteEngine:connect,initSchema,setConfig,putPage,addLink,disconnect),gbrain/config(type). OperationContext fields:engine, config, logger, dryRun, remote, sourceId. - Ops invoked:
traverse_graph,list_pages,search(mode),get_page,add_timeline_entry. - Coupling: in-process. For a different binary the runner would need a CLI JSON or MCP adapter:
calleach op overgbrain servestdio withremoteposture, then assert the same response shapes (GraphEdgeRow, list clamp 100, mode rejection on local only). - Parity-gate fit: deterministic, keyless, fast. Strong candidate for a first MCP-based parity check.
eval/runner/tool-bridge.ts (library)
- Entry: no CLI. Exports
buildToolDefs()(line 198),createToolBridge(config)(line 378),sanitizeToolInput,checkCitationFormat,ForbiddenOpError,UnknownToolError. - Measures: serves as the Cat 8/9 agent tool surface: read tools plus dry-run write intent recording.
- Behaviour:
executeToolforcesremote:trueandsourceId:'default'; stripsexpandfromquery(sanitizeToolInput); truncates results to DEFAULT_MAX_CHARS (32768 * 4); scans for poison fixtures. - Cost, keys, corpus: n/a (library).
- gbrain imports:
gbrain/operations(operations, typesOperation,OperationContext,ParamDef),gbrain/engine(typeBrainEngine),gbrain/config(typeGBrainConfig). - Coupling: in-process. Reusable reference for which read tools a remote agent exposes (EXPOSED_READ_TOOLS, line 41) and the exact result envelope (
{content, truncated, matched_poison_fixture_ids}). - Parity-gate fit: deterministic; a library, not a runner.
eval/runner/nightly/ (nightly check, run.ts plus helpers)
- Entry: eval/runner/nightly/run.ts. Usage at header lines 3-12. Helpers: tool-contract.ts, smokes/index.ts and smokes/*.ts (7 smokes), worker.ts, adapters.ts, bisect.ts, devset.ts, select-devset.ts, ledger.ts, outcome.ts, report.ts, write-score.ts, keyless.ts, judge-cache.ts, item-support.ts.
- Measures: pass rate per dev-set suite per gbrain ref; cost per correct answer; regressions vs previous night, with paired bootstrap; a bisect mode for regressions (bisect.ts).
- Run:
bun eval/runner/nightly/run.ts --ref master|stable|<sha> --tier nightly|matrix|smoke --out <dir> [--fake | --paid --budget-ledger <path> (--budget-run-id <id> | --night-cap-usd N --month-cap-usd N)] [--suites a,b] [--items x,y] [--gbrain-clone <path>]. Resume:--resume <night dir>. - Cost and time:
--fakemakes no paid call and refuses--paid(docs/nightly.md:246-251). Night caps via--night-cap-usd(run.ts:104). Wall time: unknown. Ubicloud VM bootstrap is in scripts/nightly-vm.sh (docs/nightly.md, header lines 1-6). - LLM keys: model suites need
--paid. The nightly tier usesgpt-6.1-solalone (docs/nightly.md:59). Keyless suites use keylessEnv (nightly/keyless.ts): provider keys removed and base URLs pointed at a dead loopback port. - Corpus: dev-set frozen (
devset.ts), docs/benchmarks inputs exported by scripts/nightly-export.sh. - Metrics:
report.tswith thresholds (pass-rate drop of at least 5 points with paired bootstrap interval excluding zero, or cost per correct answer rise of 25 percent or more, docs/nightly.md:119-120). - gbrain imports:
src/core/postgres-engine.ts(worker.ts, dynamic);gbrain/core/skilloptBenchmarkTasktype (tool-contract.ts:20); git on clone (execFileSync('git', ['-C', clone, ...])). - Coupling: mixed. Smokes: CLI and
serve --http(see tables). Adapters: spawnbunfor cat40 runners (adapters.ts:84, 162). Worker: in-process gbrain agents with keyedFetch (keyless.ts header). - Parity-gate fit: suites are mostly deterministic when
--fake; model suites need keys and are not a parity gate. The keyless smokes and tool-contract are the most useful parity signals.
eval/runner/nightly/tool-contract.ts (tool-contract replay)
- Entry: eval/runner/nightly/tool-contract.ts (usage at header lines 1-14):
bun eval/runner/nightly/tool-contract.ts --gbrain-root <dir> --out <file.json> [--families sol,opus] [--skills query,briefing,fact-check]. - Measures: whether gbrain refuses tool calls that real frontier models make, counted by family, path, tool and error code. Each replayed call keeps its recorded arguments; a refusal caused by the replay brain's content is an isContentMiss, counted as answered.
- Inputs: docs/benchmarks/2026-11-01-skillopt-method/probe/frontier (
PROBE_DIR, line 36), 2 families (sol, opus) times 3 skills. - Replay tools: get_page, get_backlinks, traverse_graph (REPLAY_SLUG line 46:
people/vera-chen-14,companies/ranger-22; these are synthetic example slugs). - Paths: rollout (SkillOpt registry, read-only), write-capture (SkillOpt registry), mcp (MCP dispatcher as remote stdio client). Source for the mcp path:
withMcp(line ~100). - Cost and time: keyless; wall time unknown.
- LLM keys: none (withHermeticEnv and keyless env).
- Metrics: refusal counts by family, path, tool, error code.
isContentMissfor coverage_from, coverage_gap and "has no heading" refusals (line 26-27 region). - gbrain imports: dynamic
src/core/ai/gateway.ts(configureGateway),src/core/pglite-engine.ts(PGLiteEngine:connect,initSchema,setConfig),src/core/import-file.ts(importFromContent(engine, slug, content, {noEmbed:true})),src/mcp/dispatch.ts(dispatchToolCall),src/core/skillopt/index.ts(runRolloutwithwriteCaptureorallowedTools). Alsorefresh-barrier/agent.ts(readResult,resultText) andskillopt-method/*constants. - Coupling: in-process (MCP-shaped). For a different binary, replace
withMcpwith agbrain servestdio client using tools/call with{name, arguments}, and parseisErrorplus theError:text envelope. - Parity-gate fit: deterministic, keyless. Strong parity candidate on argument handling and refusal codes.
eval/runner/nightly/smokes/ (system smokes)
- Entry: smokes/index.ts; 7 smokes: query-contract, source-pinned-sync, managed-sync, upgrade-from-previous, upgrade-from-old, ingest-retry-across-upgrade, postgres-15 (plus harness.ts, brain.ts).
- Measures: CLI and serve behaviours across upgrades and sync: JSON output contracts, writer claim and activate, held-sync retry, Postgres 15 concurrent writers.
- Run: via
nightly/run.tssystem-smokessuite, keyless. - Cost: $0. Time: unknown (waits up to 600 s per sync in code).
- Keys: none (
assertKeylessin harness.ts:106, 171). - Corpus: seeded synthetic brain (brain.ts).
- gbrain imports: none directly; the harness spawns
bun --no-env-file <root>/src/cli.ts. - CLI JSON fields read: see the CLI table (sync status
last_error.code,items[].key/code, searchslug,sources writer status --json,jobs get --json). - HTTP:
serve --http --port Nready on GET /health 200 (harness.ts:170-193); the ingest-retry smoke also drivesgbrain serve --httpwith an OAuth client_credentials client (ingest-retry-across-upgrade.ts header line 4). - Coupling: CLI (real) and HTTP (real). Most directly reusable as a parity adapter because it already drives CLI JSON.
- Parity-gate fit: deterministic, keyless, moderate speed.
eval/runner/refresh-barrier/ (refresh write barrier)
- Entry: eval/runner/refresh-barrier/run.ts (usage at header lines 1-12). Helpers: trial.ts (in-process PGLite trial and child spawn), child.ts (Postgres child), agent.ts (agent arm, catalog capture, paid runs), runtime.ts (write/poll client), schedule.ts (B_KINDS), named-case.ts, verify.ts, retrieval.ts, analyze.ts, rescore.ts, ledger.ts, ledger-preload.ts, adjudicate.ts, fixture.ts, plus repo-root scripts (refresh-barrier-b1-guard.ts, refresh-barrier-partc-adjudicate.ts).
- Measures: whether concurrent writers and refresh ticks land writes correctly under a refresh barrier, with latency. Agent block measures whether a model's writes land and its report is truthful.
- Run:
bun eval/runner/refresh-barrier/run.ts --block trials|named|quiet --engine postgres|pglite --gbrain-root <G> --out <dir> [--seeds 1001-1500] [--arms off,on] [--concurrency 6](quiet uses concurrency 2). - Cost: agent arms capped (agent.ts:72-76):
anthropic:claude-opus-5-5$65,openai:gpt-6.1-sol$35, smoke $5. Trials and named cases are keyless (keylessEnv). - Time: unknown.
- LLM keys: Anthropic and OpenAI for the agent block (fixture.ts PROVIDER_KEYS; keyedFetch in agent.ts).
- Corpus: synthetic git fixture (fixture.ts: bare remote plus managed clone, two sources alpha and beta, eight people-example pages and eight notes each), generated per trial.
- Metrics: refused, retry and landing counts;
classifyWrite(agent.ts:240); analysis in analyze.ts; preregistration docs/benchmarks/2026-11-06-refresh-barrier-preregistration.md. - gbrain imports:
src/core/pglite-engine.ts(PGLiteEngine:connect,initSchema,disconnect),src/core/postgres-engine.ts(PostgresEngine:connect({database_url, poolSize})),src/core/persistence/service.ts(disposePersistenceConsumer),src/mcp/dispatch.ts(dispatchToolCall),pageMutations.submitPageMutation(runtime.ts:72),src/core/persistence/switches.ts(readWriteSwitches,viewedEngine), MCP surface modules (agent.ts:175-178). - MCP calls in the runtime:
remember,put_page(MCP surface),get_page,get_write_request, andsubmit_answer(agent only).put_blinduses the CLI-pathsubmitPageMutation. - Coupling: in-process MCP-shaped (dispatchToolCall, not wire), CLI-shaped write (submitPageMutation in process), and child processes (
spawn('bun', ['--no-env-file', child.ts ...]), trial.ts:156). - Parity-gate fit: trials and named cases are deterministic and keyless but heavy (multi-process Postgres). Agent block needs keys.
eval/runner/live-recall/ (live recall phase 1 and 2)
- Entry: eval/runner/live-recall/run.ts (usage at header lines 1-12, subcommands
run,freeze). Helpers: task-arms.ts (four arms A/B/C/D), tasks.ts, brain.ts (phase 1 guarded engine), synthetic.ts (Voyage pricing and estimate), scorecard.ts, task-scorecard.ts, task-judge.ts, redact-scorecard.ts, visibility-epoch.ts, paraphrase-opus.ts, snapshot.sh. - Measures: recall@k and retrieval-ablation over private questions; arms: A no retrieval, B gbrain query/search/get_page, C summariser loop, D hybridSearch plus get_page.
- Run:
bun eval/runner/live-recall/run.ts run --fixture;run --synthetic <dir> [--protocol batch3] [--paid --budget-run-id <id>];freeze --questions <private.jsonl> --out <freeze.json> [--protocol batch1..batch4] [--gbrain <checkout>@<commit>]. - Cost: paid synthetic run uses Voyage (embedding, rerank); cost estimate
estimateVoyageUsd(synthetic.ts, called at run.ts:755), recorded in paid-receipt.json. Amount not in headers; unknown here. - Time: unknown.
- LLM keys: Voyage (synthetic paid run); reader and judge models through Cat 40 loop (medium effort).
- Corpus: private live-recall questions on the brain's box, not committed; synthetic dirs from eval/generators/conditions-recall-gen.ts or batch3-synthetic-gen.ts.
- Metrics: recall@k, per-query-type ablations; judge verdict via task-judge.ts; scorecard redacted (redact-scorecard.ts).
- gbrain imports:
src/mcp/dispatch.ts,src/core/operations.ts,src/mcp/tool-defs.ts,src/mcp/server.ts(resolveMcpStdioSourceScope),src/core/ai/invocation-guard.ts(withAIInvocationGuard),src/core/search/hybrid.ts,src/core/search/private-visibility.ts,src/core/access/caller.ts(optional),src/core/ai/gateway.ts,src/core/import-file.ts,src/core/pglite-engine.ts,src/core/config.ts,src/core/engine-factory.ts,src/core/search/sql-ranking.ts,src/core/search/eval-pool-depth.ts,src/core/fts-language.ts. - Coupling: in-process MCP-shaped (dispatchToolCall with
remote:true, transport:'stdio',metaHook). The_meta.accessand_meta.retrievalfields are a contract an adapter must reproduce. - Parity-gate fit: needs Voyage for the paid run; the arm logic is deterministic with a stubbed embed (live-recall
--fixture). Partial fit.
eval/runner/memory-trust/ (trust categories and flag-rate)
- Entry points: flag-rate.ts (amendment 5, the LongMemEval user simulation, usage in header lines 1-30 and main at line 218), repro.ts (finding repros,
--gbrain <checkout>@<ref> <finding>), utility-guard.ts (LongMemEval utility;capture,submit,judge,poll,summary), sut.ts (shared hermetic brain for Cat 37/38/39), agent.ts (remote agent loop), flag-rate-score.ts, join-cat37.ts, models.ts, user-preferences.ts. - Measures: flag-rate (how often the gate flags ordinary use, review items per user); repro (does a known finding still reproduce); utility (LongMemEval recall and answers vs published W10 rows).
- Run:
bun eval/runner/memory-trust/flag-rate.ts --out <dir> [--users 16] [--control-users 4] [--inject 5] [--smoke] [--keyless] [--no-agent] [--max-sessions N] [--concurrency 4] [--user-allowance-usd 2]. Repro keyless. - Cost: flag-rate uses
startPaidRun('memory-trust-flag-rate', ...)with estimate;--keylesson smoke skips the ledger (flag-rate.ts:252). Per-user allowance default $2 (line 255). Time: unknown. - LLM keys: agent model (Anthropic/OpenAI via McpClient-driven agent); judge for flags; utility-guard uses reader and judge keys (official gpt-4o-2024-08-06 judge batch per header).
- Corpus: LongMemEval-S haystacks (
--lme, default DEFAULT_LME), injected marker lines (marker-inj-N). - Metrics: flag rate; review queue counts from
trust review --json; repro exit code 1 means finding still reproduces; utility recall_all@5 and McNemar vs published W10. - gbrain imports: sut.ts (see table); flag-rate-score.ts
src/core/write-gate.tsviabun -e; utility-guard.tssrc/commands/eval-longmemeval.tsplussrc/core/ai/gateway.ts,src/core/ai/build-gateway-config.ts. - Coupling: flag-rate uses the real MCP wire (McpClient
serve --surface full, toolsrememberandput_page) plus CLI (init,jobs work,trust review --json). sut.ts and repro.ts are in-processoperationsByNamehandlers. utility-guard runs gbrain's owneval longmemevalcommand in process through a copied overlay. - Parity-gate fit: sut and repro are deterministic and keyless (strong fit). flag-rate needs keys unless
--keyless --smoke.
eval/runner/predictive/ (agent stack, CLI and MCP surfaces)
- Entry: run.ts (
runGrammarat line 240,mainat 266), README.md (Modes, Paid runs). Helpers: cli-arm.ts (SyntheticBrain, execGbrain, GbrainCliArm, GbrainMcpArm, validateArgv), write-contract.ts (classifyCall, schemaShapes, runSchemaGrammar), episodes.ts (CLI and MCP episode shapes), causes.ts, passk.ts, verdict.ts, validity.ts, freshness.ts, mapping.ts, paid.ts, entity-agent.ts, export-references.ts, out-of-view/ (agent.ts, run.ts, tasks.ts, gen.ts, reach.ts, README.md), scale-latency/ (build.ts, probe.ts, ubicloud.ts, vm-setup.sh, README.md), ubicloud/ (setup.sh, run.sh, vm-run.sh). - Measures: agent behaviour on the CLI and MCP surfaces: write landing (G3), refusal causes, reach (out-of-view), latency (scale-latency), pass^k and cost per correct answer.
- Run:
bun eval/runner/predictive/run.ts(flags in code; see predictive/README.md Modes). Scale-latency throughubicloud/run.sh. Out-of-viewrun.tswithgbrain serve --access read-only. - Cost and time: paid episodes gated by paid.ts; time unknown. The README gives modes but no figures here.
- LLM keys: Anthropic and OpenAI for the model side (cat40 toolLoop);
--model fakestyle paths keyless (grammar is keyless). - Corpus: synthetic brain built per run (SyntheticBrain: content root under GBRAIN_HOME, git-backed); GRAMMAR_PAGES; out-of-view generated (gen.ts).
- Metrics: passk.ts (pass^k), causes.ts, validity.ts; write-contract classifyCall reads
Error [code],"code","error"and detailpending_import,file_database_drift(write-contract.ts:33-71). - gbrain surfaces: CLI
init --pglite --non-interactive --content-root <dir> --git,put <slug>(markdown on stdin),sync,export --dir,recall --since 2000-01-01 --limit 1000 --json,get,search,call <op> <json>. MCP:gbrain servestdio via McpClient; toolsget_page,remember,query(per episode shapes). - gbrain imports:
gbrain/config,src/core/ai/defaults.ts,src/core/ai/gateway.ts,src/core/engine-factory.ts,src/core/pglite-engine.ts,src/core/import-file.ts,src/mcp/dispatch.ts(out-of-view reach). - Coupling: CLI (real, execFile
bun src/cli.ts) and MCP (real,servestdio) and in-process brain build. This is the closest existing model for a multi-surface adapter: onegbraintool taking an argv array (cli-arm.ts:29, GBRAIN_CLI_TOOL) and the same schema throughtools/list. - Parity-gate fit: keyless grammar (
runSchemaGrammar,schemaShapes) is deterministic and cheap; episodes need keys.
eval/runner/strict-pageless/ (GBRA-449 cell)
- Entry: eval/runner/strict-pageless/agent.ts (usage at header lines 13-20; main at line 300). Helpers: items.ts (item ids and keys), analyze.ts, report.ts (scoreReport, Truth).
- Measures: on a Git-canonical source, does a fact remembered for an entity with no page survive rebuild from Git, and is the reply truthful.
- Run:
bun eval/runner/strict-pageless/agent.ts --model <provider:id>|fake --gbrain-root <checkout> --items 1-24 --k 1 --out <dir> [--budget-ledger <path> --program-cap-usd N --budget-usd N] [--pilot]. - Cost and time: paid run via
startPaidRun(agent.ts:317, estimateUsd null). Time unknown. - LLM keys: model provider key when not
fake. - Corpus: fresh PGLite brain built by refresh-barrier fixture (
openBrain, trial.ts), eight people pages and eight notes per source. - Metrics: kept_in_git (active fact row with token whose entity slug names the item), truthful (report.ts scoreReport), exposure (remember call carrying token with entity having no page;
entity_page_beforefrom SQL onpages). - MCP calls:
remember(itemsarray or single entity),put_page,get_page,submit_answer(eval tool). Tool list fromcaptureCatalog(refresh-barrier/agent.ts), with server instructions shown in system hint (strict-pageless/agent.ts:74). - gbrain imports:
src/core/pglite-engine.ts,src/commands/sync.ts(performSync),src/core/cycle/extract-facts.ts(runExtractFacts), plusdispatchToolCallvia the openedg.dispatch. - Coupling: MCP-shaped in process (
remote:true, transport:'stdio', sourceId: alpha). Response envelope read withreadResultandresultText. - Parity-gate fit:
--model fakedeterministic; uses the same fixture as refresh-barrier.
eval/runner/system-one/ and system-one-jev.ts
- Entry: system-one-jev.ts (usage header;
verifyandplanin package.json at eval:system-one, eval:system-one:plan). Helpers: system-one/slots.ts (nine slot definitions,--decidemodes), datasets.ts, recount.ts. - Measures: each of gbrain's nine System One slots (S1 rerank, S2 route, S3 prune, S4 answerable, S5 prompt-injection flag, S6 needs-memory, S7 dream triage, S8 claim support, S9 contradiction) as a matched pair on the same commit.
- Run:
bun eval/runner/system-one-jev.ts verify(package.json eval:system-one) orplan(eval:system-one:plan, prints all commands). - Cost and time: paid through TypeSafe Jev (
typesafe:jev-1.13.0); unknown amount. - LLM keys: TypeSafe Jev key (not in the grep list; check system-one-jev.ts env allowlist).
- Coupling: CLI.
spawnSync('bun', argv, {cwd, env})(system-one-jev.ts:301). The execution block readsproduct.package,declared_pin,loaded_git_head,package_sha256,version(system-one-jev.ts:213). - Parity-gate fit:
planis deterministic and keyless;runis paid.
eval/runner/systems/ (adapter layer)
- Entry: no runner. Consumed by memory-qa and the shootout driver. Files: types.ts (
MemorySystemcontract, sanitized inputs), gbrain.ts (GbrainLegacySystem, GbrainShootoutSystem, in process), gbrain-agent.ts (GbrainAgentSystem, in process), gbrain-query/ (connector.ts, system.ts, README.md), mcp-tools.ts (neutral MCP tool adapter), http.ts (HttpMemorySystem, HTTP to a non-gbrain stack,POST /answeroptional), file-agent.ts (rg, not gbrain), agent-runtime.ts (docker, not gbrain), baselines.ts (extract-first, memory-bank, pgvector), fake.ts, render.ts and render-v2.ts, sanitize.ts. - mcp-tools.ts: the neutral MCP adapter. Types
McpTool {name, description?, inputSchema},McpToolResult {content: [{type, text?}], isError?},McpToolServer {listTools(), callTool(name, args)}.mcpToolArm(name, server, allowed)refuses to build when a named tool is absent and refuses calls outsideallowed. Result text is the text blocks joined by newline;isErroris thrown. This is the shape an adapter for a Rust gbrain MCP server should match. - gbrain-agent.ts:
GBRAIN_AGENT_TOOLS = ['query','search','get_page'];tools/list=buildToolDefsover the build's OPERATION_MANIFEST;tools/call=dispatchToolCall(engine, name, args, dispatch)withtransport:'stdio',inboundFrameBytes: MCP_STDIO_MAX_MESSAGE_BYTES,takesHoldersAllowList:['world'],sourceId:'default',surface:'full'(line 226). Runs in process so gbrain's provider calls pass the paid-request guard (header lines 13-14). - gbrain-query connector:
wireRequest(name, query, {limit, budget})(connector.ts:50),assembleRequest(hits, unit, budget, packing)(line 69),QUERY_PATH_PINS(line 28),AUTO_PACKINGS(off, cap_only, breadth_capped, depth_first; line 74),search.auto_packingconfig key (line 76),deliveryRecord,parity(live, assembled)(line 245). ItsloadConnectorModules(load)dynamically imports gbrain modules through a loader. This is the closest existing code to a request-shape spec for query delivery. - http.ts: a generic HTTP client for an external memory system (
HttpMemorySystem), not gbrain. Its wire protocol is in eval/systems/PROTOCOL.md (not read). - Coupling: gbrain.ts and gbrain-agent.ts in-process; mcp-tools.ts neutral; http.ts HTTP to a non-gbrain system.
- Parity-gate fit: in-process paths are deterministic; the neutral MCP adapter is the natural seam for a binary-based parity run.
eval/runner/takes-bootstrap/
- Entry: harness-overlay.mjs (overlay of gbrain's evals/takes-bootstrap/harness.mjs at gbrain c5fb0201, v0.60.95.0) and harness-overlay.diff.
- Measures: the takes-bootstrap harness, with pricing overrides, a summary JSON (model, estimate, ceiling, hard cap, spend, prices used, per-case classifier errors, wall time) and a request log.
- Run:
bun eval/runner/takes-bootstrap/harness-overlay.mjs --gbrain-dir <tree> [--price-input N --price-output N] [--summary <path>] [--request-log <path>] --model <m>(flags from header lines 1-15). - Cost: spend estimate and hard cap via BudgetTracker; provider key needed for the model under test.
- Coupling: in-process by loading the gbrain harness, corpus, scorer and
src/tree from--gbrain-dir(line 54; defaultnode_modules/gbrain). Detail beyond the header was not read. - Parity-gate fit: paid; not a parity gate as is.
Non-adapter runners (in scope, no gbrain contact or only indirect)
eval/runner/memory-qa/ (benchmark arm runner)
- Entry: run.ts (usage header lines 1-14), run-systems.ts (dispatched for memory-system flags). Helpers: corpus.ts, qa.ts, qa-v2.ts, instruments.ts, arms.ts, records.ts, outcomes.ts, sealed-profile.ts, bg1-names.ts, beam10m-corpus.ts, beam10m-questions.ts.
- Measures: LoCoMo, LongMemEval-S, BEAM accuracy per gbrain build (and per memory system via run-systems).
- Run:
bun eval/runner/memory-qa/run.ts --benchmark locomo|lme-s|beam-100k|beam-1m|fixture [--split dev] [--embed hash|real] [--qa reader|think] [--paid --budget-run-id <id>]. Sealed split needs a custodian decision id. - Cost: paid estimate computed in code (run.ts:572,
Math.max(0.05, ...)); per-question and per-QA constants. Not published in headers; unknown. - LLM keys:
--embed realneeds embedding provider;--qa reader|thinkneeds QA model keys (ChatClient, run.ts:575).--qa nonekeyless.--embed hashkeyless. - Corpus: benchmark datasets under eval/data (LoCoMo, LME-S, BEAM); generated in part (beam10m-corpus.ts).
- Metrics: accuracy per category, judge via instruments.ts; receipts with cost and fidelity fields.
- gbrain imports:
PGLiteEngine,hybridSearch(expansion off),importFromContent,runThink,DEFAULT_RERANKER_MODEL(run.ts, run-systems.ts). Coupling: in-process. - Parity-gate fit:
--embed hash --qa nonedeterministic and keyless.
eval/runner/messy-company/
- Entry: load.ts (usage header lines 1-12), score-ownership.ts, score-timeline.ts, score-agent-tasks.ts, score-document-status.ts, score-entity-mentions.ts, score-changes.ts, score-aliases.ts, score-owner-phrases.ts, score-governing-records.ts, common.ts (Wilson intervals, normalization).
- Measures: messy-company-v1 lanes: ownership windows (owner-on-date accuracy, acting-cover, exact-window recall/precision), timeline, aliases, document status, entity mentions, governing records, agent tasks.
- Run:
bun eval/runner/messy-company/load.ts --split dev|heldout|all [--db <dir>] [--public-only] [--json]then the score-*.ts scorers on a system's output JSON (shape in each header and eval/data/messy-company-v1/SCORING.md). - Cost: $0; keyless (importFromContent with noEmbed). Time: seconds.
- Corpus: eval/data/messy-company-v1/corpus/ (committed Markdown pages), labels/ (gold).
- Metrics: counts with 95 percent Wilson intervals (common.ts).
- gbrain imports:
gbrain/import-file(importFromContent),gbrain/pglite-engine(PGLiteEngine) in load.ts only. - Coupling: in-process (load) and none (scorers). Parity-gate fit: deterministic and keyless.
eval/runner/miss-mining/
- Entry: weekly-counts.ts (usage header), to-live-recall.ts, schema.ts, custody.ts.
- Measures: weekly counts of private misses by cause x ask kind x detection mode; conversion of logged misses into live-recall questions.
- Run:
bun eval/runner/miss-mining/weekly-counts.ts --log <miss-log.jsonl> --week <YYYY-MM-DD> --out <weekly.json>;to-live-recall.ts --log <log> --out <questions.jsonl>. - Cost $0; keyless. Corpus: private log on the box; output passes custody.ts.
- gbrain imports: none. Coupling: none.
eval/runner/outcomes/
- Entry: v3.ts (outcome-v3 instrument, header: versioned answer outcomes, replaces no historical scorer), rescore-a4.ts (
$0rescoring of A4 receipts), validate-labeler.ts (labeler validation, paid). - Measures: outcome labels (abstain, hedge, commit) and a judged commitment/hedge labeler validated against GBRA-49 hand labels. Preregistered rule: first model with precision >= 0.90 on both
hedgedandabstainis selected; ladder gpt-6-luna, claude-haiku-5-5, claude-sonnet-5-5. - Run:
bun eval/runner/outcomes/rescore-a4.ts <receipt.json>...($0);validate-labeler.ts --labels-dir <dir> --stage select|confirm --paid --budget-run-id <id> --budget-ledger <path>. - Cost: rescore $0; labeler paid (amount unknown).
- LLM keys: labeler ladder keys (Anthropic, OpenAI).
- Corpus: A4 receipts (committed) and hand-labelled answers (custodian folder).
- gbrain imports: none. Coupling: none.
eval/runner/providers/openai-compatible.ts
- Entry: module only (no CLI). Pieces:
compatRequest,compatSend(sends once, never retries),parseCompatResponse,classifyCompatError,compatError. Providers: Particle AI (https://api.particle.ai/v1, US endpointhttps://us.api.particle.ai/v1) in COMPAT_PROVIDERS; Google Gemini OpenAI-compatible endpoint as judge-only. - Measures: transport facts only. Checked against provider docs on 2026-10-31 (header).
- Keys: Particle and Gemini keys at call time. Coupling: none (no gbrain). Parity-gate fit: n/a.
eval/runner/rs1/
- Entry: verdict.ts (
verdictandpilotsubcommands, header lines 1-10), iut.ts (estimand and decision rule, the single source the power simulation and verdict both call), power.ts, pricing.ts, paid-window.ts (guard CLI:check --paid-window <unit> --window-cap-usd N --budget-ledger <file> --budget-run-id <id>, exit 2 on refusal). - Measures: RS1 pre-registered comparison of gbrain against memory-bank and extract-first servers, paired by conversation, with a one-sided 95 percent t lower bound on n minus 1 df (iut.ts header).
- Run:
bun eval/runner/rs1/verdict.ts verdict --receipts <dir>... [--manifest <file>] [--set RS1|RS1-dev2] [--out <dir>]. - Cost: analysis $0. pricing.ts estimates spend from
--n,--spent-usd,--gbrain-api-per-mtok. - Keys: none direct (receipts come from cell.ts, outside scope).
- gbrain imports: none. Coupling: none.
- Parity-gate fit: deterministic analysis; not a gbrain parity gate.
eval/runner/skillopt-method/
- Entry points: headroom.ts (A1 headroom probe; frontier targets
opus= claude-opus-5-5,sol= gpt-6.1-sol), one-shot.ts (harness one-shot arm, runs seed on 24 train tasks), score.ts / score-v3.ts (scoring), gate.ts, rollouts.ts (toolLoop and runRollout), tool-calls.ts (rollout tool-call accounting), test-stage.ts (test split reader), ceiling-stages.ts, dataset.ts and dataset-context.ts (skillopt-method-v2/v3 dataset), kept-rewrites.ts, audit.ts, contract.ts (output contracts per skill). - Measures: whether gbrain's shipped skills (query, briefing, fact-check) leave headroom for an optimizer; one-shot rewrite; gate decision on kept rewrites.
- Run: headroom and one-shot via their file usage text (not read in full). Frontier probe is paid.
- Cost: paid frontier probe; amounts not read here.
- LLM keys: Anthropic (claude-opus-5-5) and OpenAI (gpt-6.1-sol) for the frontier probe. A test seam exists:
__setChatTransportForTests(headroom.ts import) for offline stubs. - Corpus: skillopt-method dataset (docs/benchmarks/2026-11-01-skillopt-method/...), skills from gbrain's skills tree.
- gbrain imports: see table (gbrain/core/skillopt, gbrain/ai/gateway, gbrain/ai/invocation-guard, gbrain/pglite-engine, gbrain/operations, gbrain/import-file, gbrain/core/skillopt describeJudges).
- Coupling: in-process. Parity-gate fit: the rollout and scoring path is in process; no CLI or MCP seam.
eval/runner/sota/
- Entry: leader.ts (plan and import subcommands), oracle.ts (plan, import-a4, run), scorecard.ts (monthly scorecard), answerability.ts, fixlist.ts, ledger.ts, cache.ts, custody.ts, sealed-guard.ts, lme-support.ts, scorecard.ts; ubicloud/ (setup.sh, run.sh, vm-run.sh).
- Measures: monthly gbrain minus leader (whole-history reader on LongMemEval, grep file agent on BEAM-1M) on dev and sealed questions, with conversation-clustered 95 percent interval, in points; status ahead, behind, parity (inside ±3.5 points).
- Run:
bun eval/runner/sota/leader.ts plan --cache <dir>;leader.ts import --from <q1 out dir> --cell <id> --cache <dir>;oracle.ts run --benchmark beam-1m|lme-s --reader <provider:model> --items <ids.txt> --cache <dir>;scorecard.ts --rows <rows.json> [--previous <file>] --takeaways <file> --out <file outside repo>. - Cost and time: runs go through q1 cells (outside scope); unknown.
- LLM keys: readers claude-opus-5-5, gpt-6.1-sol, particle-1; judges gpt-6-astra (Claude rows) and claude-opus-5-5 (GPT and particle rows) per oracle.ts header.
- Corpus: BEAM-1M and LongMemEval-S outcome cache.
- gbrain imports: none directly in scoped files; memory-qa reader indirectly via q1 cells (outside scope).
- Coupling: none for scorecard, ledger, cache, custody; indirect in-process via q1 for leader and oracle.
- Parity-gate fit: scorecard deterministic; oracle and leader paid.
eval/runner/nightly and eval/runner/mcp-contract.ts also feed docs
- docs/nightly.md describes the nightly tiers (
concurrent-writers,system-smokes,tool-contract-replay, dev-set suites),--fake,--paid, and the Ubicloud VM (scripts/nightly-vm.sh, scripts/nightly-export.sh).
Cross-cutting observations
- Two protocols exist in this scope, and only one is a real wire. The real MCP wire is McpClient (cat40/gbrain-arm.ts, outside scope) spawning
bun <build>/src/cli.ts serve. Everything else called "MCP" callsdispatchToolCallin process. A Rust adapter can match thetools/callresult envelope{content:[{type:'text',text}], isError}but must still speak JSON-RPC over stdio to be tested as a real server. - The CLI is driven through
bun --no-env-file <build>/src/cli.ts <argv>in every case. No runner calls an installedgbrainbinary directly, except via the shim written by nightly/smokes/harness.ts:149. That makes the entry point swappable: replace<build>/src/cli.tswith a binary path. - JSON output is parsed in several places with inconsistent tolerance. flag-rate.ts:187 slices from the first
{; predictive/write-contract.ts:71 regex-matches error codes from text (Error [code],"code":,"error":). A CLI JSON contract would need to replace these regexes. - Error classification depends on text. refresh-barrier/agent.ts:269-275 parses
Error:plus a JSON body; refresh-barrier/runtime.ts:41 regex-matchesretry_after_ms=Nfromdetail. A Rust adapter must keepcode,retryableorclass,retry_after_ms,write_request.stateas structured fields. - Shared helpers across runners: refresh-barrier/agent.ts is imported by nightly/tool-contract.ts (readResult, resultText), strict-pageless/agent.ts (captureCatalog, keyedFetch, readResult, resultText, writeToolsOf), and skillopt via refresh-barrier/fixture.ts (gitHead). refresh-barrier/trial.ts (openBrain, quiesce) is shared by strict-pageless. Treat refresh-barrier as a library layer for the agent/MCP surface.
- Keyless enforcement is structural: keylessEnv (nightly/keyless.ts), assertKeyless (nightly/smokes/harness.ts:106), enterHermeticEnv (hermetic-env.ts), and
--fakemodes. Any adapter must run under a stripped env with provider base URLs on a dead port. - Two different gbrain loaders exist: the npm pin
node_modules/gbrain(package exports such asgbrain/operations) and a build root path (dynamicimport(join(root, 'src/...'))) via gbrain-under-test.ts (importGbrain, overlay withgit archiveinto.gbrain-overlays/). Most MCP-shaped runners need the source tree, not the package exports. - The
mcp-contract.tsheader says "Category 12" and points out assertion design: many checks were rewritten after audit (the header notes that pre-fix checks could not fail). Those are the assertions an adapter must reproduce to claim parity. - Surprises: refresh-barrier and strict-pageless call
put_blindthrough the CLI write path (submitPageMutation) and not through MCP, so a write-path adapter must support both. memory-trust/sut.ts buildstransport:'http'contexts but never opens HTTP. - Not verified: the full contents of many files (nightly/report.ts, live-recall/tasks.ts, memory-qa/corpus.ts, sota/*), exact costs in several headers, and the cat40/loop.ts and lifecycle/drivers.ts helpers that the scope calls. The "unknown" entries in the table mark these.
Summary (5 lines)
- Real MCP wire exists only via McpClient (cat40/gbrain-arm.ts) spawning
bun <build>/src/cli.ts serve --surface <s>over newline JSON-RPC stdio (2025-06-18); used by flag-rate, predictive, out-of-view. - Other MCP-shaped calls are in-process
dispatchToolCall(src/mcp/dispatch.ts) with{remote:true, transport:'stdio'}: live-recall, refresh-barrier, strict-pageless, nightly tool-contract, systems/gbrain-agent. - CLI runners spawn
bun --no-env-file <build>/src/cli.ts <argv>and parse JSON from--jsonfor get/search/query/recall/list/jobs/sources/sync status/trust review; predictive also usescall <op> <json>. - Keyless and deterministic candidates for a parity gate: mcp-contract.ts, tool-bridge.ts, memory-trust/repro.ts and sut.ts, nightly/tool-contract.ts, nightly/smokes, messy-company load, predictive schema grammar.
- Costs are capped in code only for refresh-barrier agent arms ($65 opus, $35 sol, $5 smoke); other paid runs estimate in code or are unknown. No runner uses a published gbrain binary.