Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
5.7 KB
Raw Blame

Efficiency bench harness

Opt-in benchmark scripts for the Efficiency crew. Nothing here runs in CI or ships in the CLI. Brain data (pulled pages, synthetic brains, PGLite files, results) lives under $BENCH_WORK (default ~/.capy/work/brain) and must never be committed. Result rows hold counts, sizes and timings only.

Requires Bun >= 1.4.4 (the repo's engines.bun) and Docker for the Postgres engine.

File What it does
lib.ts Row schema (machine, engine, brain, pages, chunks, command, mode, N, p50, p95), percentiles, CLI runner, provider price table.
preload-instrument.ts bun --preload hook. BENCH_USAGE_LOG records every outbound HTTP call (host, path, ms, model, token usage; no bodies). BENCH_ENGINE_TIMING times every PGLite/Postgres engine method (outermost calls only) and process CPU. No product code changes.
materialize-pull.ts Pull directory -> importable markdown dir (content copied, field-only pages re-serialized, oversize pages replaced by size-matched synthetic stand-ins).
synth-brain.ts shape (manifest -> shape JSON, counts/sizes only), full-shape (~5,000 pages over 8 sources), gen (deterministic synthetic brain, --scale, --seed).
bench-import.ts Fresh isolated brain (GBRAIN_HOME) per engine; times init, per-source import, extract all, embed --stale; engine/HTTP/CPU split per phase; in-process parse and chunk probe; tokens and USD from provider usage.
bench-hot.ts Hot paths, cold (fresh CLI process per sample) and warm (one long-lived gbrain serve stdio MCP process): search, query with/without expansion, get_page, list_pages, recall, doctor, MCP stdio startup + call, HTTP serve startup + call, one-page sync.
probe-pg.ts Warm stdio MCP tool against a Postgres bench brain with pg_stat_statements reset around N calls: p50/p95, DB time per call, top statements (placeholders only). --jit off A/Bs Postgres JIT for the run.
probe-cold.ts Cold CLI start split: bun start, module graph, --version, real reads, each with stdout piped and with GBRAIN_FLUSH_GRACE_MS=0 (the non-TTY exit-flush grace).
probe-rerank.ts Direct Voyage /v1/rerank grid over top_n_in x max_doc_tokens and models on candidate sets drawn from a Postgres bench brain: billed tokens/call, $/1k queries, p50/p95, top-5 agreement with the 25x1400 default (a drift signal, not quality). Spends ~$0.1 per 20-query grid.
probe-mcp-start.ts MCP stdio cold start: fresh gbrain serve per sample, spawn -> initialize, spawn -> tools/list, first tools/call latency and spawn -> first call, after each --idle-ms gap; interleaves several --cli trees and prints the tools/list size and hash per tree (equal hashes = byte-identical list).
probe-pglite-open.ts PGLite open cost on a copy of a data directory: PGlite.create with PGLite's stock (eager) vs gbrain's lazy extensions, at the 128MB default vs the 32MB one-shot-read shared_buffers, one fresh process per sample, plus the first query that loads vector.so.
bench-chunk.ts In-process chunk pass: parse every page of a synthetic brain directory (or --cjk <pages> for a generated mixed CJK / Latin / emoji corpus), then time the importer's prepareMarkdownChunks over all pages for --n passes after a warm-up. Prints p50/p95 per pass and a sha256 over every chunk, so a before/after pair doubles as an equivalence check.
vector-scope-share.ts Vector search latency and recall per source scope against an existing Postgres or PGLite brain (GBRA-71): searchVector per scope, recall of result pages against the exact statement, page share, the chunk count the engine routes on, underfilled exits; --remote adds the private-page rule.
remote-rows.ts Rows for hosted-brain read latency recorded during a pull.
report.ts JSONL -> markdown tables (results/report.md).
run-all.sh Orchestrates data prep, both engines, all datasets, report.

Run

export BENCH_WORK=~/.capy/work/brain          # default
bun run bench:efficiency default-pull synth-1x synth-full   # = scripts/bench/efficiency/run-all.sh
# one brain, one engine
bun scripts/bench/efficiency/bench-import.ts --engine pglite --data $BENCH_WORK/import/synth-1x --label synth-1x --embed --max-usd 1
bun scripts/bench/efficiency/bench-hot.ts --engine pglite --label synth-1x --n 20 --only search,query_expand
bun scripts/bench/efficiency/report.ts
# Postgres statement probe and cold-start split
bun scripts/bench/efficiency/probe-pg.ts --label synth-full --tool list_pages --args '{"limit":50}' --jit off
bun scripts/bench/efficiency/probe-cold.ts --label synth-full --engine postgres
# chunk pass and scoped vector search, before/after from two pinned worktrees
bun scripts/bench/efficiency/bench-chunk.ts --data $BENCH_WORK/import/synth-full --n 20
bun scripts/bench/efficiency/vector-scope-share.ts postgresql://postgres:[email protected]:5440/bench_synth-full sessions notes-b all --queries 25 [--remote]
# CPU profile of one phase
bun scripts/bench/efficiency/bench-import.ts ... --cpu-prof   # writes .cpuprofile under engines/<engine>-<label>/bench-run/

Postgres uses a pgvector/pgvector:pg16 container named gbrain-bench-pg (BENCH_PG_CONTAINER) on 127.0.0.1:5440 (run-all.sh starts it); each brain gets its own database bench_<label>.

Caveats

  • Cold CLI rows include Bun start, TypeScript load, config, engine connect and exit, the cost a shell-out agent pays per call. Warm rows are what an MCP-connected agent pays.
  • Query, search and recall rows include provider round trips (query embedding, rerank, and for query with expansion an LLM call), so they move with provider latency.
  • Embedding cost is the provider-reported token usage times lib.ts prices.