Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
7.1 KB
Raw Blame

Scale tier harness

The nightly and label-gated scale tier (.github/workflows/scale-tier.yml) runs this harness; the gate shape and cadence are in docs/TESTING.md.

bun run test:scale -- --pages 10000 [--engine pglite|postgres] [--seed 1] [--corpus-dir <dir>] [--import-mode cli|content] [--enforce] [--out <file.json>] (scripts/scale/run.ts) generates a deterministic two-source brain from the seed (scripts/scale/fixture.ts: links, timeline bullets, ## Facts and ## Takes fences, partly overlapping bodies, island pages, a seeded dense vector in 16 dimensions per page), and imports it into a fresh brain under a temporary GBRAIN_HOME: PGLite in a fresh data dir, or Postgres in a fresh database created from DATABASE_URL and dropped afterwards. The default --import-mode cli writes the Markdown corpus once into --corpus-dir (reused while its manifest matches) and runs the real gbrain import per source, timing each file from its progress events; --import-mode content keeps the per-page importFromContent loop. It then extracts links, timeline, facts and takes, writes the vectors onto every chunk so the vector arm runs keylessly through queryEmbedFn, and measures p50 over five runs after a warmup for each op, each with a known-answer check: get_health, list_pages, local and MCP-path search, a source-scoped grant search, hybrid query with an injected vector, traverse_graph, get_backlinks and find_orphans, plus a cold-process first query and two concurrent receipt-bearing put_pages. find_orphans is checked with one call at the op's maximum page size: its rows must hold every fixture island and match total_orphans. It captures every statement each op sends and replays the reads under EXPLAIN ANALYZE for the planner check. With the brain closed, it then runs the large-brain operational ceilings through the real CLI (scripts/scale/f4d.ts), each an enforced data check with its measurement in the report's f4d section: a gbrain sync of a fresh source (1,000 files at 10k and up) past a 1 s progress-aware deadline completes (f4d_sync_deadline); gbrain embed --stale against a local stub embedding endpoint stops at its time budget with exit 11, the remaining count and the resume command (f4d_embed_budget_stop; nothing leaves the machine); gbrain serve answers initialize and a search without hitting its boot deadline (f4d_serve_boot); and at 20,000 pages and up, gbrain sources add registers a 20,000-file checkout on a fresh managed brain (f4d_sources_add_20k). Each phase logs its start and end, so a run stopped by a job timeout shows where it was. The report leads with the headline metric, MCP search p50 at the run's size as shipped (no manual ANALYZE).

Exit codes: 0 when every enforced gate passes, or always without --enforce; 1 when an enforced gate fails (each failure names the gate and op and prints its EXPLAIN; the JSON report and a .explain.txt land next to --out); 2 on a usage error; 3 when the harness itself crashed (not a verdict). The import-rate gate compares the last 10% of pages' per-page cost with the first 10% (<= 1.5x) and the total import with the time at the halfway mark (<= 2.5x). Both ratios set aside at most one stall per import (#7939): a paused database container or a WAL checkpoint lands in one page's time and once failed unrelated pull requests on its own. Each tenth is cut into ten slices; the slice with the largest excess over its tenth's median slice is lowered to the next-largest excess anywhere in the import, and its tenth never drops below the Theil-Sen line through the ten tenth means. When the next-largest excess is over a quarter of the largest, the cost showed up twice and nothing is set aside, so work that recurs (a rebuild every few hundred pages, once per tenth, or twice in the import) is kept at the plain measure's sensitivity. A cost that lands only once, or whose second landing is over 4x its first (on a 10k import, a rebuild every 5,000 pages or more, or every 4,500 starting early), looks like a stall; the 20k nightly tier sees it again. The gate message names the stall and the report's import.<basis>.excused records it. The report and its JSON import section carry both bases for every engine: wall time (elapsed_ms) and the import process's main-thread CPU time (cpu_ms, user + system, from each progress tick; in-process timing under --import-mode content). PGLite is judged on CPU time: the database runs synchronously on the import's main thread, so that CPU time holds all of its work (and not the runtime's GC and JIT helper threads), while wall time also counts time a shared CI host gives the CPU to other tenants, which once failed a run with no code change. Postgres is judged on wall time, because the server's work never shows up in the import process's CPU time.

Enforced under --enforce: import rate, known answers, no-op re-import, no duplicates across sources, the import phase timer, and the stats-dependent gates (PLANNER_HEALTH_ENFORCED in scripts/scale/gates.ts): the Nested Loop inner-loop gate on the key plans, the budgets phase timer, and planner stats. Planner stats are probed after the first timed op, because F4b analyzes on the first planner-sensitive read, and only hot tables above 500 rows must have pg_stats rows (PGLite only: on Postgres autovacuum owns statistics, so a missing row is report-only). Interactive ceilings and calibrated budgets (scripts/scale/budgets.json, written by --calibrate) stay report-only until bun scripts/scale/trend.ts prints "ceilings stable" over the last five nightly runs; a reviewer then sets the repo variable GBRAIN_SCALE_ENFORCE_CEILINGS=1. The same script picks the nightly sizes. Reproduce any report with the command it prints. The fixture's determinism is pinned by test/scripts/scale-fixture.test.ts, the gate policy by test/scripts/scale-gates.test.ts and scale-trend.test.ts, the find_orphans known answer by test/scripts/scale-orphans-verifier.test.ts, and a 40-page enforced run by test/scripts/scale-harness.slow.test.ts.

The harness runs as a child of scripts/scale/watchdog.ts. A PGLite statement runs synchronously inside WASM, so an in-process timer cannot fire during a stall; the supervisor times each watched phase (import, extract, vectors, budgets) from the child's [scale] phase <name> start lines and kills the child's process group at 1.5x the phase's gate ceiling (extract and vectors get 1.5x of half and a quarter of the import ceiling). The diagnostic names the phase, elapsed time against the limit and the last progress line; the supervisor then removes the brain home and scale database. Override one limit for a local investigation with GBRAIN_SCALE_PHASE_LIMIT_MS_<PHASE> (for example GBRAIN_SCALE_PHASE_LIMIT_MS_VECTORS=3600000). The vectors phase prints one line per batch with its time, dimension and RSS. 20k cells run under a 130-minute job cap, and scripts/scale/trend.ts promotes the nightly size to 50k only when both engines wrote a 20k report. test/scripts/scale-watchdog.test.ts blocks a child's main thread and checks the named failure arrives within the limit plus grace.