Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
48.7 KB
Raw Blame

Key files: Evaluation

  • src/eval/longmemeval/evidence-packet.ts + scripts/eval-answer-packet.ts — experimental presentation-only paired comparison over pinned historical session IDs. The pure formatter uses original structured conversational rounds with recoverable pointers and explicit omissions. The runner keeps the existing full-session baseline, uses a leakage-safe development/holdout split, and journals guarded reader/judge spending under one resumable cap. It does not retrieve or change production defaults. Usage and caveats: answer packets. Tests: test/longmemeval-evidence-packet.test.ts, test/eval-answer-packet.test.ts.

Subsystem index. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.

  • scripts/eval-c2-candidate-fusion.ts + scripts/eval-c2-embed.py — the C2 offline eval: a seeded synthetic fixture (40 companies; single-valued claims with changed numbers/dates, coexisting similar claims, private rows, a dense crowded slice, back-to-back corrections) replayed on PGLite through listSupersessionCandidates for facts.candidate_fusion rrf_free vs interleave, embedded once with a free local model. Reports twin recall@5, false supersession, preserved distinct claims and correct decisions per threshold, paired by probe and clustered by company (pairedClusterStatistics); at 0.95 the real decideSingleFact runs beside the replayed rule and must agree. Output: docs/eval/decisions/c2-interleave-dev/verdict.json.

  • src/eval/longmemeval/evidence-brief.ts — experimental, eval-only evidence brief (10x memory advantage plan, wave 1 item A3). Pure: buildBriefPrompt (question-aware) and buildDigestPrompt (one session, no question) build a cheap builder's prompt over the intact delivered sessions; parseBriefDraft reads its JSON; validateBrief grounds every quote with the dream-cycle verifier (groundSource/groundQuote/unsupportedNumericClaims, LongMemEval **user:** turns via lmeTurns), drops ungrounded, misattributed, number-unsupported and instruction-context claims (counted in validation.dropped), withholds a paraphrase whose negation disagrees with its quote, appends uncited later corrections verbatim, never presents a count as complete, replaces absence wording with "no supporting evidence was returned from the searched scope" plus the searched sessions, renders claims inside sanitized <chat_session> blocks within a cl100k budget, and falls back to the full text when parsing or grounding fails. Pointers are UTF-16 spans into the sanitized session text with span and body SHA-256 (verifyPointer). Grounding is provenance, not truth. No model call, no production default. Tests: test/evidence-brief.test.ts (pointer round trip, adversarial fixtures, fallback).

  • src/eval/longmemeval/reader.ts + gateway-client.ts — the LongMemEval answer request seam, exported as gbrain/eval/longmemeval/reader. resolveReaderConfig selects notes-first/1024 by default or the frozen direct-v3/512 baseline; buildReaderRequest changes only the system instruction and budget, retaining the same full-session sanitizer, evidence order and dates. readerConfigHash pins mode, system-prompt version/sha, requested model and token budget. generateAnswer reports the provider finish reason, joins every text block and keeps the call's provider-normalized usage (A1: ReaderUsage with total, uncached, cache-read and cache-write input, output and reasoning tokens; the cache counts are subsets of the total), which the command writes as reader_usage on every answered row (null when no reader call ran). readerUsageFromMessage reads the gateway adapter's gbrain_usage record or the Anthropic Messages convention. The command records reader_partial_output as a diagnostic, never a completed hypothesis, when the reader stops at the limit or returns empty/unknown completion; those failures count in the judged headline denominator and fail the run without automatic paid retries. Tests: test/longmemeval-reader.test.ts, test/eval-longmemeval-reader-mode.slow.test.ts, test/eval-answer-packet.test.ts (historical explicit direct baseline).

  • evals/brainbench/ — the committed BrainBench corpus: 141 fixtures (135 generated + 6 hand-authored spike) / 241 gold-annotated turns across 7 categories (kta-pos/kta-neg/push/write-back/continuity/multi-source/adversarial), ~15% holdout (excluded from the CI gate, scored in published --include-holdout runs). generator/gen.ts rebuilds the corpus byte-identically (Mulberry32, seed 42; whole-cloth fictional universe from curated synthetic name pools so scenario privacy is structural; prose is template-synthesized with PRNG-selected variants — deliberately no LLM pass, difficulty stays controlled; several know-to-ask variants intentionally exercise documented v1 reflex limits so the baseline measures the roadmap). gold/ is sealed; schema/ is the foreign-runner contract (gbrain-evals drives the suite as a subprocess via --fixtures DIR --gold DIR --json --out FILE); baselines/main.json is the committed gate baseline; _ledger.json records counts/seed/rebuild command. CI: the .github/workflows/test.yml brainbench job + scripts/ci-brainbench-gate.sh (fetches MAIN's baseline via git show origin/master:… — a PR cannot rewrite what it's compared against; first-landing path runs ungated) + scripts/render-brainbench-delta.ts (compact step-summary/PR-body delta block from the --out artifact). Privacy: scripts/check-synthetic-corpus-privacy.sh scans evals/brainbench/{fixtures,gold} in bun run verify.

  • evals/longmemeval/splits-seed42.json + evals/longmemeval/dev-slice-seed42.txt — the committed, seeded question-id splits that carry the LongMemEval decision discipline (ids only, never question text or session content). schema_version: 1; dataset_sha256 + scored: 470 / abstention_excluded: 30 pin the exact cleaned _s corpus the splits were drawn from; prng = mulberry32 over question_ids sorted ascending with Fisher-Yates; seeds dev: 42, halves470: 4242, halves430: 4243. Lists: dev40 (the 40-question dev slice, per-type counts in dev40_type_counts), decision430 (the held-out remainder every pre-registered success rule is decided on), halfA470/halfB470 (235 each) and halfA430/halfB430 (215 each) for select-on-A / confirm-on-B decisions; type_counts records the per-type totals of the 470. dev-slice-seed42.txt is dev40 one id per line for gbrain eval longmemeval --question-ids. Invariant: mechanisms are chosen on the dev slice or half A and decided on decision430 or half B; a full-470 row published alongside is labelled as including the dev slice.

  • src/core/bench/baseline-file.ts + src/core/bench/qrels-file.ts + src/core/bench/correctness-gate.ts + src/commands/bench-publish.ts + src/commands/eval-gate.ts — the eval loop. gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> writes a baseline (stamps stable query_hash per row; metadata header carries _kind: 'baseline_metadata' + thresholds + source_hash + baseline_mean_latency_ms; deterministic sort by (tool_name, query_hash); strict: empty=fail, dupes=fail with paste-ready hint, --to exists=refuse without --force). gbrain eval gate [--baseline X] [--qrels Y] is the two-gate dispatcher (regression gate via in-process replayCore, correctness gate via bare hybridSearch for determinism, both must pass when both flags set, exit 0 PASS / 1 FAIL / 2 USAGE). Source-id-aware: bench publish dedup key is (tool_name, source_ids, query_hash); qrels compare keys are ${source_id}::${slug} everywhere (so multi-source brains are keyed correctly at the file-shape layer). Latency math: (baseline + delta) / baseline <= multiplier. Fail-closed: ANY in-process throw flips verdict to fail with named breach in breaches[] — never silently exit 0. .qrels.json preserves the 12-row test/fixtures/eval-baselines/qrels-search.json fixture (slug-only relevant_slugs + first_relevant_slug auto-promote to source_id='default') AND supports the federated shape (explicit relevant: [{source_id, slug}] + expected_top1). correctness-gate.ts runs each qrels query via bare hybridSearch; per-query throw recorded as errored: true and flagged as gate failure. Audit JSONL at ~/.gbrain/audit/bench-publish-YYYY-Www.jsonl. Hermetic mode: --embedder deterministic (correctness gate ONLY — rejected with --baseline, requires --qrels) embeds each query as the qrels fixture's basis vector via src/eval/deterministic-embed.ts (basisEmbedding unit vectors; FNV-1a-derived fallback dim for off-fixture query texts) and threads it into bare hybridSearch through the queryEmbedFn seam — no API keys, no network; bare hybridSearch neither reads nor writes the semantic query cache (both live in hybridSearchCached), so deterministic runs cannot poison cached production results. scripts/run-eval-canary.ts (check:eval-canary package script, on-demand; in CI the same runner executes via test/eval-canary.test.ts in the unit matrix, not in the verify battery) is the hermetic CLI retrieval canary built on it: boots a throwaway PGLite brain under a temp GBRAIN_HOME, seeds the qrels fixture corpus (the expected-top1 page carries its query text in timeline too — page-grain FTS indexes title(A) + timeline(C) only, compiled_truth is deliberately unindexed), spawns the REAL CLI with engine-reroute/provider env stripped, and asserts exit 0 + metric floors; --record additionally appends an EvalRunRecord-shaped line to .gbrain-evals/eval-results.jsonl. Honest scope: the canary gates the hybrid ranking pipeline (keyword/title/alias arms + RRF against gold qrels) with synthetic vectors — semantic-embedding regressions remain the keyed eval suites' job. Pinned by test/bench/baseline-file.test.ts, test/bench/qrels-file.test.ts, test/bench/correctness-gate.test.ts, test/bench-publish.test.ts, test/eval-gate.test.ts, test/eval-canary.test.ts, test/eval-replay-metadata-skip.test.ts, test/cycle/nightly-probe-adapters.test.ts, test/autopilot-nightly-probe-wiring.test.ts, test/e2e/eval-loop.test.ts. D2: bench declares json: 'ndjson': bench publish --json writes its result as one line and every failure ends with a {status:'error'} envelope line (refusals keep their stderr lines).

  • src/core/cross-modal-eval/aggregate.ts — pure verdict logic. Judge dimension keys are normalized with trim + lowercase before cross-model roll-up, so CORRECTNESS and correctness cannot split into separate one-model dimensions. Pass criterion: (successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5). Inconclusive when <2/3 models returned parseable scores (guards the Object.values({}).every(...) === true empty-array PASS trap).

  • src/core/cross-modal-eval/json-repair.ts — parseModelJSON(raw) named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.

  • src/core/cross-modal-eval/receipt-name.ts — receipt filename binds (slug, SKILL.md sha-8). findReceiptForSkill(skillPath, receiptDir) returns 'found' | 'stale' | 'missing'. Skillify-check surfaces the status as informational; the audit does NOT fail on missing/stale receipts.

  • src/core/cross-modal-eval/receipt-write.ts — wraps fs.writeFileSync with mkdirSync({recursive:true}) ahead of every write (gbrainPath() does NOT auto-mkdir).

  • src/core/cross-modal-eval/runner.ts — orchestrator. The judge prompt pins the exact scores JSON keys per dimension (dimensionScoreKey = the label before the em-dash) so judges cannot invent spellings; aggregate.ts's normalization is the backstop. buildPrompt wraps the task and the candidate in <task_to_grade>/<candidate_output> data blocks, and neutralizeClosingTag rewrites every closing form of those tags inside the untrusted text (case-insensitive, whitespace-tolerant) to the visibly-escaped <\/tag, so a candidate carrying its own closing delimiter cannot terminate the block and land text outside the data boundary — the post-candidate grading-only instruction stays the last thing the judge reads. callSlot sends EVALUATOR_SYSTEM_PROMPT as system and the bounded prompt as the single user turn. Each cycle runs Promise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)]) (bare allSettled, no rate-leases for the CLI path). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots: openai:gpt-5.2 / anthropic:claude-opus-4-7 / deepseek:deepseek-v4-pro. estimateCost() prices via the canonical model-pricing table, each slot at the output cap its thinking-off call sends (thinkingOffOutputCap; a raised cap is named in the notes, which both the single-run and batch banners print); test/cross-modal-default-slots.test.ts pins recipe support, pricing coverage, and three distinct providers.

  • src/core/eval/cycle-default.ts — single source of truth for the eval cycle-count default. Exports DEFAULT_CYCLES_TTY = 3, DEFAULT_CYCLES_NONTTY = 1, resolveCycleDefault(explicit, isTty): {cycles, usedNonTtyDefault}, and cycleDefaultSuffix(r) (returns (non-interactive default; --cycles N for more) only when the non-TTY default was applied, else ''). Consumed by eval-cross-modal.ts, eval-takes-quality.ts (run + regress), and takes-quality-eval/runner.ts (core uses only the constant — library stays TTY-agnostic; the CLI owns the TTY=3 upgrade + banner annotation). eval-suspected-contradictions.ts applies the same transparency to its $5/$1 budget default via a budgetUsdExplicit flag (the budget is overwritten in-place so explicitness can't be inferred post-hoc). Not shared with resolveWorkersWithClamp (different domain, no engine, no dedup). Pinned by test/eval/cycle-default.test.ts, test/eval-suspected-contradictions-budget-default.test.ts.

  • src/core/eval/paired-bootstrap.ts — paired, clustered comparison statistics for per-query eval rows, ported from the gbrain-evals situation-recall comparator (eval/runner/situation-recall-regression.ts). pairedClusterStatistics(pairs, { seed, draws }) returns n, clusters, means, delta, wins/losses/ties, a percentile 95% CI from whole-cluster bootstrap resamples, and a two-sided sign-flip p-value (exact enumeration at ≤16 clusters, else Monte Carlo with the (k+1)/(n+1) correction); no discordant pairs → CI [0, 0], p = 1; fewer than two clusters → unbounded CI, p = 1. holmAdjusted is the Holm step-down. Deterministic via the same seeded mulberry32 generator. Requires ≥1000 draws and finite values. Consumed by src/commands/eval-compare.ts. Pinned by test/eval-paired-bootstrap.test.ts.

  • src/core/eval/ranked-docs.ts — dependency-free page-level ranking primitive shared by the retrieval eval harness, NamedThingBench, and the qrels correctness gate. dedupeRankedKeys() preserves the first/best occurrence of each ${source_id}::${slug} or slug key before any cutoff, so repeated chunks neither inflate relevance gain nor consume document ranks. Precision retains its fixed-k denominator; Recall and nDCG are structurally bounded to [0,1]. Pinned across test/eval.test.ts, test/retrieval-quality-harness.test.ts, test/bench/qrels-file.test.ts, and test/bench/correctness-gate.test.ts.

  • src/core/takes-quality-eval/pricing.ts — budget pricing for eval takes-quality run. getPricing(model, overrides?) returns the operator rate registered with gbrain pricing set (pricing.overrides), else the canonical rate (canonicalLookup), else null; schema {input_per_1m, output_per_1m}. Every canonically priced model is gateable. Miss policy (new models must run): with no --budget-usd an unpriced model runs with a warning naming the registration command and its spend is not counted; under a user cap the runner refuses before any call with no_pricing (unpricedUnderCapError), whose fix is gbrain pricing set <model> --input … --output … --source … (no-pricing.ts). Rates are never copied here; test/model-pricing.test.ts pins equality with canonical for every canonical key.

  • src/core/takes-quality-eval/runner.ts — eval takes-quality run orchestrator: samples takes, sends the rendered rubric prompt to every panel model per cycle (allowFallback: false, thinking: 'off'), and re-asks each malformed slot once with the same model and sample plus the validator's error (slotFormatFailure in aggregate.ts: parse_failed including an empty reply, or incomplete_scores; provider errors and valid low scores never re-run). The per-cycle projection and each correction price judgeCallCostUsd(model): 5k input tokens plus the output cap the call sends (thinkingOffOutputCap(model, JUDGE_MAX_TOKENS); a raised cap is announced on stderr). A correction is priced against --budget-usd before it is sent and recorded corrected: null with skipped_reason when it does not fit or the run was aborted; it replaces the first attempt only when it validates. The receipt carries protocol_version: TAKES_QUALITY_PROTOCOL_VERSION (2), correction_selection_rule and corrections (additive at schema_version: 1); regress.ts lists a protocol change among the dissimilar inputs. Pinned by test/eval-takes-quality-runner.serial.test.ts and test/eval-takes-quality-regress.test.ts.

  • src/eval/brainbench/ + src/commands/eval-brainbench.ts — BrainBench, the cross-harness memory conformance suite (gbrain eval brainbench; methodology in docs/eval/BRAINBENCH.md). types.ts carries the PUBLISHED interchange shapes (fixture/gold/result/baseline — mirrored as JSON Schemas in evals/brainbench/schema/; breaking changes bump the schema versions). fixtures.ts: strict loader/validator + corpus fixtures_hash (covers fixture AND gold files); a gold key inside a fixture turn is a validation error — gold is SEALED in the gold dir and adapters only ever see sanitized PublicTurns. seed.ts: fail-fast hermetic seeding (importFromContent noEmbed + NULL-embedding insertFact; any non-imported status ⇒ SeedError ⇒ fixture seed_failed ⇒ run exit 2). adapters/shared.ts: ONE runReflexPipeline all three adapters drive with declarative config (pointer budget, suppression mode) — cross-harness comparability is structural; openclaw.ts (seam production, the shipped pipeline), claude-code.ts (seam production; drives the shipped gbrain hook user-prompt path end-to-end — fixture turns become UserPromptSubmit stdin JSON, synthesized Claude Code JSONL transcripts feed the real window parse + cross-turn dedupe, and resolution rides a run-scoped resolve-IPC server with the real shared secret; bench-pinned deviations disclosed in docs/eval/BRAINBENCH.md: generous userPromptDeadlineMs, push-failure banner suppressed), codex.ts (seam contract; static entity-index preamble whose slugs deliberately don't count as injections + ≤1 per-turn fragment; fixture conversations round-trip through the real rollout format + the shipped parser src/core/transcripts/codex.ts for turn selection — fragment DELIVERY remains harness-shaped until a shipped codex injection path lands). metrics/: know-to-ask (+false-fire anti-gaming companion), push (micro-averaged P/R), write-back (drives the PRODUCTION conversation→facts pipeline via the injectable-extractor seam; gold extractor in CI, real extractor under --llm), continuity (writer→reader pairs on a shared brain through DIFFERENT adapters; pointer-injection OR stored-fact keyword probe). harness.ts: ONE in-memory PGLite per run + resetTables between fixtures (longmemeval engine-sharing pattern); read-only suites share one seeding across all adapters; emits per-(harness×suite) cells + re-scoreable turn rows; source_isolation_violations counted per turn and gated at zero. scoreboard.ts: markdown render, canonical diff-stable committed baseline (4-decimal rounding, sorted keys, receipts excluded), compareBaselines with main-baseline governance — same-hash count-aware gate vs corpus-bless mode (the committed baseline must byte-match the run; regressions vs main require a justification). The CLI brings its own PGLite (cli.ts routes before connectEngine), writes --out as the canonical CI artifact, and terminates via an explicit grace-tick process.exit(verdict) (0 pass / 1 regression / 2 error) because PGLite stomps process.exitCode and Bun discards queued stdout on exit. runBrainBenchCore() is the in-process entry eval run-all uses (one record per sweep, EvalRunRecord schema_version 3, mode: 'n/a'). Pinned by test/brainbench-*.test.ts + test/eval-brainbench-e2e.slow.test.ts. System One arm: --decide* flags (src/eval/decide-eval-flags.ts) configure the benchmark brain; the claude-code adapter captures the turn-context decide.recall_needed meta onto turn_rows[].decide, the result gains a decide block, and --update-baseline is refused for an arm (test/eval-brainbench-decide.serial.test.ts).

  • src/eval/longmemeval/diagnostics.ts + scripts/lme-miss-diagnostics.ts — LongMemEval miss diagnostics: locate WHERE a strict miss is lost before any fix is chosen. Input is a harness receipt (ndjson) plus the dataset; for every question with recall_all_hit=false (every scored question under --all) the question's brain is re-created EXACTLY as the harness built it (one in-memory PGLite per run, resetTables per question, haystackToPages → importFromContent, the embed cache installed through the same seam so every page / question vector is a cache hit) and each gold session missing from the receipt's top-k is located per arm: vector_rank (first chunk row of the session in engine.searchVector, paged by offset to depth, default 200, because one call caps at MAX_SEARCH_LIMIT), keyword_rank (engine.searchKeyword, OR-fallback as hybrid), title_rank (engine.searchTitles), fused_rank_* (the pre-rerank RRF order of ONE hybridSearch call at fusedLimit, default 50, under the same pins, captured via onRerankPool(pool, preRerank)), post_rerank_* (that call's post-rerank pool) and final_rank_rows (what the call returned after autocut / limit). Frozen classes (classifyMiss): (i) absent from every arm's top-depth, (ii) in an arm pool but outside the fused top-k pre-rerank, (iii) in the fused top-k pre-rerank but reranked out, (iv) ceiling (more gold sessions than k); two observational classes keep the receipt honest — rerun_hit (the re-created run places the gold inside the top-k: the receipt's miss did not reproduce under these pins) and autocut_dropped / post_fusion_dropped (the gold survived fusion + rerank and a later trim removed it). Frozen hypothesis probes: h1Signature (≥ 2 gold, one at fused session rank 1–3, the missing one at 6–15), the counterfactual clause sub-queries (splitClauses / splitClausesDetailed over the frozen pattern list how_many_between | between | first_or | before_after, with the content-token and quoted-span guardrails; H1 is supported when a clause's own vector top-5 contains the missing gold) and the h3Candidates split — H3a candidate generation (gold in the vector top-depth but outside the pre-fusion pool, computeInnerLimit) vs H3b reranker depth (in the fused pool but beyond reranker_top_n_in). splitMembership tags each question with the committed evals/longmemeval/splits-seed42.json splits (dev40 | decision430 | halfA430 | halfB430 | halfA470 | halfB470). INVARIANT: the clause sub-query embeds BYPASS the embed cache, so a diagnostics run never adds rows to the shared like-for-like cache (its canonical hash is a receipt field). INVARIANT: every helper except runDiagnostics (the one engine-touching entry point; applyPins writes the pins) is pure over plain data — classifyMiss, splitClauses, h1Signature, h3Candidates, the rank builders, parseReceipt / pinsFromReceipt / receiptTopKSessions, summarizeDiagnostics, renderDiagnosticsMarkdown — so the tests pin them without PGLite, and every printed metric routes through the shared glossary (glossFor, with LOCAL_GLOSSARY for the diagnostics-only names). The script owns argv, gateway bootstrap, file I/O and printing (<receipt.ndjson> --dataset FILE [--splits FILE] [--mode M] [--reranker on|off] [--autocut on|off] [--expansion-variant-budget legacy|B] [--embed-cache PATH | --no-embed-cache] [--k N] [--depth 200] [--fused-limit 50] [--all] [--question-ids FILE] [--limit N] [--out-ndjson FILE] [--out-md FILE] [--json]); pins default to the receipt's flat run_config on the by_type_summary line (mode, reranker.{enabled,model}, autocut, expansion, expansion_variant_budget, topK, embedder; a legacy nested pins block is still accepted) and explicit flags win; it is NOT a gbrain subcommand, so its flags sit outside the CLI flag registry by design. Spend: with the receipt's cache every page / question embed is a hit; the ≤ 2 clause sub-query embeds per miss and, under --reranker on, one rerank call per miss are the only paid calls. Exit 0 ok · 1 bad input / run error · 2 gateway or reranker not ready. Pinned by test/longmemeval-diagnostics.test.ts.

  • src/eval/longmemeval/judge.ts + judge-lane.ts + qa-accuracy.ts — the LongMemEval LLM-as-judge lane behind gbrain eval longmemeval --judge. judge.ts is a faithful port of the official evaluate_qa.py::get_anscheck_prompt: the per-type instruction (standard for single-session-user / single-session-assistant / multi-session; temporal-reasoning adds the off-by-one-days clause; knowledge-update; the single-session-preference rubric) and the abstention instruction for _abs ids (the id suffix beats the type; an unknown type falls back to standard), ONE user message per question, DEFAULT_JUDGE_MODEL openai:gpt-4o, JUDGE_TEMPERATURE 0, JUDGE_MAX_TOKENS 16 (the OpenAI API's minimum; the official 10 is rejected, and a one-token yes/no verdict is unaffected), and the official verdict rule (parseJudgeVerdict: 'yes' substring of the lowercased completion). Disclosed deviations, carried verbatim in JUDGE_METHODOLOGY_NOTE on every summary: (1) the question / reference / response sit inside <judge_input> data-boundary framing under JUDGE_DATA_BOUNDARY_INSTRUCTION (#4338) — escapeJudgeData neutralises tag closures inside the data and the response text is otherwise unaltered, so the judge grades what the reader actually said; (2) the judge_error class — classifyJudgeResponse returns null for a completion that is neither a yes nor a standalone no (malformed, re-judged) instead of a silent no; (3) abstention detected by the _abs suffix; (4) the unknown-type fallback. JUDGE_PROMPT_VERSION bumps whenever any instruction text, framing or field label changes. judgeConfigHash = sha256 over the stable JSON of {judge_model, prompt_version, max_tokens, temperature, reader_model, reader_prompt_sha, context: {k, max_tokens}}. judgeRow judges one row from its stored hypothesis: budget-skips on the PROJECTED cost (estimateJudgeCallUsd) BEFORE spending, records actual cost after, never throws on a transport failure, and stripJudgeFields removes every prior judge_* key first so a re-judge leaves no stale field. judge-lane.ts is the harness-side orchestration, pure except runJudgeBackfill (whose only effects are the injected client and in-place row updates; no engine, no file I/O): parseMaxUsd (N, or off / none / unlimited → no cap), makeJudgeConfigHasher (per-row hasher — a prior row's recorded reader pins win over this run's; k is the run's --top-k, already gated by retrieval_config_hash), hasJudgeAttempt / hasSettledVerdict, selectBackfillRows (candidates = rows with a hypothesis and no settled verdict; counts settled, mismatched + the foreign hashes, retrievalOnly, missingFromDataset), judgePreflight (availability → pricing → estimate → cap: exit 1 no provider, exit 2 unpriced-with-budget or over-cap without --yes) and runJudgeBackfill (runWithLimit at --judge-concurrency; the dataset's answer is the reference, the row's own answer the fallback). qa-accuracy.ts builds the qa_accuracy block from ALL rows (pure; last row per question_id wins): accuracy_headline (alias accuracy) = correct / total_questions over EVERY question including _abs — a judge_error, a budget-skipped row, a reader-error row and a never-judged row all score INCORRECT, so it is never more lenient than the official scorer; accuracy_excluding_errors = correct / judged (secondary); accuracy_470 = the headline rule over the non-_abs questions (the retrieval-metric denominator); by_type and an abstention sub-block; judge_error_classes; ci95_bootstrap (percentile bootstrap over the headline 0/1 vector, label: 'question-sampling only'); complete (no error, no skip, no unjudged row — the publishability bit the run-end gate reads); mixed_judge_config; est_cost_usd, actual_cost_usd (summed over the rows' own judge_cost_usd across resumes), run_cost_usd (this run's ledger); methodology_note. Pinned by test/longmemeval-judge.test.ts and test/eval-longmemeval-judge.slow.test.ts.

  • src/eval/longmemeval/metrics.ts — LongMemEval strict-recall metrics: the raw-id join, recall_all@k / recall_any@k, per-type buckets, the schema-v2 by_type_summary, and the per-question row assembler. PURE: no engine, no I/O, no LLM, so the harness and the tests score the same bytes. The join is on RAW session ids: haystackToPages imports every session under an OPAQUE per-question id (sessionSlug → chat/s- + first 10 hex of sha256(question_id:normalized session id), the same id in frontmatter and in the reader's <chat_session id>), because every public gold id starts with answer_ and no distractor does — a raw id in a slug, title, chunk or reader prompt is a gold label (pinned by test/longmemeval-gold-leak.slow.test.ts); buildSlugToRawMap is the private per-question inverse used only for scoring, distinctRetrievedSessions joins through it, detectSlugCollisions / collisionsTouchingGold name the ambiguous slugs (a gold-touching collision makes the harness emit an error row), goldMissingFromHaystack counts dataset defects, and normalizeSessionId exists only for the opaque-id hash input and legacy-row rawification — never for the gold compare. k semantics: recall_*@k is scored over the DISTINCT sessions among the top-k CHUNK rows returned at limit: k (the caller slices first; scoreRecall treats k as a guard); empty gold scores both hits false, not vacuously true, and such rows stay out of the denominator. addRowToBucket folds a v2 row into total/all_hit/any_hit; a row carrying only the deprecated any-only recall_hit counts toward total + any_hit and bumps legacy_rows (its all_rate is therefore a lower bound); buildByTypeSummaryV2 emits sorted type keys, null rates on empty buckets, mean_distinct_sessions, and the caller's run_config. buildRow assembles the JSONL row — scored fields over results.slice(0, k), retrieved[] + retrieved_session_ids over EVERY returned row so replay can re-score at any smaller k — and harness passthrough extra keys never override the scored fields. Pinned by test/longmemeval-metrics.test.ts.

  • src/eval/longmemeval/resume.ts — reading a prior run's JSONL back: row scanning, the mixed-run check, expansion-variant replay maps, and per-type bucket re-seeding. INVARIANT: recall is RECOMPUTED, never trusted — seedBucketsFromRows re-derives both metrics for every prior row from retrievedIdsAtK (prefers retrieved[] sliced to k; falls back to the older retrieved_session_ids shape) joined against the dataset's gold, so a file written before the raw-id join simply scores false instead of poisoning the summary, and there is no legacy any-only counter. readJsonlRows skips corrupt lines (a SIGKILL tail); isScoredQuestionRow excludes summaries and hypothesis-less error rows (those are retried); checkResumeConfigHash reports rows stamped with a foreign retrieval_config_hash (refused by the harness unless --allow-mixed-run-config) and tolerates unstamped rows; loadExpansionReplay maps question_id → recorded expansion_variants. Pure given its inputs (file reads are the only I/O).

  • src/eval/longmemeval/run-config.ts — the resolved retrieval pins, retrieval_config_hash, the run_config receipt block, secret redaction, and the --question-ids loader. Pure given its inputs (the harness resolves the mode bundle and the embedder string and passes them in). RetrievalPins = {mode, keyword_only, reranker: {enabled, model}, autocut, expansion, expansion_variant_budget, embedder (model@dims or 'unconfigured'), top_k, trajectory}; retrievalConfigHash(pins, {knobs_hash, knobs_hash_version}) is sha256 over stableStringify (sorted keys at every level) of the pins PLUS the resolved knobsHash, so two runs hash equal regardless of flag order, dataset, output path or clock, and a resume file written under a config snapshot that differs in ANY result-shaping knob — not just a pin — is a different run. redactSecrets (URL userinfo, bearer tokens, secret-looking key=value pairs, provider key prefixes, ?password=) is applied to every error string BEFORE it lands in a receipt row or the eval ledger. loadQuestionIds reads one id per line with # comments, dedupes, and throws on a missing or empty file. buildRunConfig stamps the summary's run_config (pins, expansion_replay, dataset sha256 + count, question_ids_file, both hashes, the CacheReceipt {path, hits, misses, bypassed, infra_faults, canonical_sha256, sha256} or cache: null + cache_skipped reason, and the counters reranker_skipped_rows, vector_degraded_rows, expansion_failed_rows, expansion_replay_miss, gold_missing_from_haystack, slug_collisions, excluded_abstention, errors).

  • src/eval/retrieval-quality/harness.ts + src/commands/eval-retrieval-quality.ts + test/fixtures/retrieval-quality/namedthing.jsonl + test/fixtures/retrieval-quality/namedthing/corpus.ts — NamedThingBench, the retrieval-quality eval that pins the named-thing-miss failure class. Seven query families, each a distinct failure class: title-substring (the named-thing miss itself), generic-to-named (tourist label → named thing), alias-synonym (declared alias / romanization → canonical), multi-chunk-dilution (one strong chunk among many weak — stresses max-pool), short-vs-rich, graph-relationship (guardrail), hard-negative (precision guard, must NOT return a page). gbrain eval retrieval-quality <fixture.jsonl> runs it with hard gates (e.g. title-substring Hit@1 ≥ 0.95, alias Hit@1 ≥ 0.98, multi-chunk-dilution Hit@3 = 1.0). Pure: caller injects a SearchFn (CLI uses hybridSearch, tests stub) so it's engine-agnostic. A search call that throws is stamped errored: true and scored as a miss in every family (never a clean hard-negative); report.errored / per-family errored count them and any errored question fails the gate. Metric glossary entries (hit@1/hit@3) added to src/core/eval/metric-glossary.ts. Pinned by test/eval-retrieval-quality.test.ts + test/retrieval-quality-harness.test.ts. The seed corpus (NAMEDTHING_CORPUS, seedNamedThingCorpus(engine, { embed? }), loadNamedThingQuestions()) lives in test/fixtures/retrieval-quality/namedthing/corpus.ts — the ONE brain the fixture is written against (sibling of relational/corpus.ts), seeded without vectors by the hermetic gate and with real vectors by the R1 A/B script, so a paid receipt and the CI gate describe the same pages. The committed fixture is 12 queries — 11 non-relational core queries plus one graph-relationship query, which R1 scores together with relational/corpus.ts's 38 RELATIONAL_QUESTIONS as the 39 relational questions. seedNamedThingCorpus seeds exactly what the gate always seeded (putPage with compiled_truth = the chunks joined, one upsertChunks row per chunk with token_count: 10, lower-cased setPageAliases when declared) and, when embed is supplied, embeds every chunk text in ONE batch and stores the vectors (a count mismatch throws); the result reports pages, chunks, embedded, embedded_chars for spend accounting. Change pages, chunk boundaries or aliases only together with the fixture and its gate expectations. --decide* runs read-only on the connected brain: missing provider/key/consent refuses with the catalogued line, write-only flags are refused, rerank=on pins the Jev reranker per call, and --json adds decide with per-query receipts (test/eval-retrieval-quality-decide.serial.test.ts).

  • src/eval/shared/autocut-replay.ts + scripts/replay-autocut-floor.ts — pure replay of the autocut weak-top-floor sweep over captured rerank pools, and its CLI. INVARIANT: the replay uses the SAME applyAutocut the live path uses, over the SAME pre-autocut reranked pool (gbrain eval longmemeval --capture-pool → rerank_pool, unscored alias/exact-lookup rows included — dropping them would change decision.total), with the SAME preserve predicate (alias_hit === true || exact_lookup === true), and slices to k AFTER autocut exactly like hybrid.ts, so validateLive(rows, floor) can demand byte-for-byte agreement with the recorded live decision (search_meta.autocut, plus autocut_kept_keys when present) before any other floor cell is trusted. Every cell — including floor off — comes from ONE capture; no second reranker call. Metric semantics mirror the harness (distinct sessions among the first k kept chunk rows; recall_all = gold ⊆ distinct; recall_any = non-empty intersection), with the benefit metrics mean_returned_results / mean_returned_est_tokens (why autocut exists) and recall as the guardrail; pairedDelta gives wins/losses/net per type vs the first floor, splitHalf(rows, seed) gives a seeded half-A/half-B selection/confirmation split, topScoreHistogram publishes the reranker's top-score distribution. normalizePoolRow throws on a row without slug/session_id (a silently dropped row shifts every cliff). The script owns argv (<capture.ndjson> --floors off,0.10,… [--k 5] [--validate-live F] [--split-half SEED] [--jump 0.2] [--min-keep 1] [--json]; --validate-live takes exactly one floor), file I/O and printing; every metric it prints routes through src/core/eval/metric-glossary.ts (REPLAY_GLOSSARY_KEYS); exit 0 ok · 1 validate-live mismatch or bad input · 2 usage. Pinned by test/replay-autocut-floor.test.ts.

  • src/eval/shared/embed-cache.ts — content-addressed embedding cache for fixed-corpus evals (dataset-agnostic; the LongMemEval harness is its first consumer). INVARIANT: a cached vector is served ONLY for the exact (model@dims, text, side) it was computed for, and every row is integrity-checked on read (declared dims == stored bytes / 4 == vector length); a mismatch is a HARD EmbedCacheIntegrityError naming the file and the key — never a silent re-embed, because a silently corrupted cache makes every arm's vectors non-comparable. Key = ${model}@${dims} (+ #query for query-side asymmetric embeds, so asymmetric query vectors can never be served a document-side row) + : + sha256(text). Storage: bun:sqlite, WAL journal + synchronous=NORMAL, busy retry; local filesystem only. A question's writes are buffered in memory (reads see them) and written in one short BEGIN IMMEDIATE transaction when its withTransaction unit ends, so no SQLite lock is held across embed round-trips and concurrent runs sharing one cache file neither lose writes to SQLITE_BUSY_SNAPSHOT nor stall behind each other's questions. installEmbedCache installs it through the gateway's __setEmbedTransportForTests seam (gateway.ts sits at its module-size ceiling; a named setEmbedTransport() hook is a filed follow-up) and restores the caller-supplied realTransport ?? null on uninstall. Stats: hits, misses (genuine misses PLUS every value of a batch that fell open on an infrastructure fault), bypassed (SDK model id ≠ the installed model — never cached), infra_faults (file deleted mid-run / table dropped / disk error → uncached re-embed or lost write-back, so a run that lost its cache can never show a clean misses: 0). The canonical hash (PRAGMA wal_checkpoint(TRUNCATE) then sha256 over the sorted (key, dims, sha256(vector)) rows) goes into run_config.cache.canonical_sha256 so two runs can prove they saw the same vectors. Pinned by test/longmemeval-embed-cache.test.ts.

  • src/eval/shared/gateway-bootstrap.ts — configureEvalGateway({ chatModel? }), the one AI-gateway bootstrap for eval runners that bring their own brain instead of calling connectEngine: configureGateway(buildGatewayConfig(loadConfig() ?? env)), where env supplies GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS and provider keys fold from both planes. chatModel fills in only when the brain configures no chat_model. Consumers: the gbrain eval longmemeval branch of cli.ts, scripts/r1-namedthing-rerank-ab.ts, scripts/lme-miss-diagnostics.ts, scripts/eval-answer-packet.ts, evals/functional-area-resolver/harness-runner.ts and evals/takes-bootstrap/harness.mjs.

  • src/eval/shared/judge-runner.ts + src/eval/shared/bootstrap.ts — dataset-agnostic judge mechanics shared by the LongMemEval lane and any later judged eval. judge-runner.ts owns the chat-client seam (JudgeChatFn — the gateway chat in production, a canned fn in tests), bounded retries with exponential backoff (default 2 retries, base 500 ms, sleep seam), the closed judge_error vocabulary (timeout | rate_limit | empty | refusal | malformed | provider_error, exported as JUDGE_ERROR_CLASSES), usage summed over every attempt, per-call cost through the ONE canonical pricing table (canonicalLookup; an unpriced model yields cost_usd: null, never 0, and isJudgeModelPriced lets a budget refuse it), and the BudgetLedger (maxUsd === null = no cap; canAfford(next) is PROJECTED so a run never overshoots by more than the calls already in flight; unpriced calls are counted, not summed; snapshot() for receipts). INVARIANT: a judge malfunction is an error outcome, never a verdict: 'incorrect' — runJudge never throws on a transport failure (only an already-aborted AbortSignal or a bug in the caller's parse can), classifies 429 / rate-limit prose → rate_limit, abort / timeout → timeout, everything else → provider_error, and maps a refusal / content_filter stop, an empty completion and a null from parse to refusal / empty / malformed; the caller decides how the headline scores them. It passes temperature and maxTokens through ChatOpts and records ChatResult.responseModel as response_model. bootstrap.ts: bootstrapMeanCi(xs, {resamples = 10000, seed = 42, alpha = 0.05}) is a seeded (mulberry32) percentile bootstrap of the mean, byte-reproducible; n = 0 → nulls, n = 1 → a degenerate [x, x]. INVARIANT: every block carries label: 'question-sampling only' — the interval quantifies how much the number would move under a different draw of questions and says nothing about reader / judge nondeterminism, dataset revision or prompt drift. Pinned by test/longmemeval-judge.test.ts.

  • src/eval/longmemeval/retrieval-arms.ts — eval-only retrieval arms for gbrain eval longmemeval, flags spread into the harness flag table. --fact-keys chunk|page extracts each imported session's facts (--fact-extractor paper = the benchmark's published user-turn fact prompt, production = extractFactsFromTurnWithOutcome as shipped; --fact-keys-model, --fact-keys-max-usd cap on uncached spend, default 20) through a content-hash JSONL cache under ~/.cache/gbrain-eval/, then re-embeds that session's chunks with factKeyedEmbeddingInput inside the embed-cache transaction (row field fact_keys). --time-scope reserved|partition widens the search limit to --time-scope-pool (default 50), parses the question against question_date, re-orders by each session's observation + mentioned dates, and records time_scope plus the unscoped top-k's unscoped_recall_all_hit / unscoped_recall_any_hit from the same pool (a paired control inside one run). Active arms fold into retrieval_config_hash as retrieval_arms; with no arm on the hash is unchanged. Test: test/fact-keys.test.ts.

  • src/eval/longmemeval/synopsis-tier.ts — makes --mode tokenmax measure the production per-chunk synopsis tier: when the resolved tier is per_chunk_synopsis, every imported session is re-embedded through reembedPageWithContextualRetrieval with the backfill's synopsis model (resolveContextualSynopsisModel), behind a JSONL cache keyed by buildSynopsisCacheKey (only answers the model returned are cached) and a --synopsis-max-usd cap metered from provider usage (default 20; the run stops after the question that reached it and exits 1). Transient service errors retry with backoff. Row field contextual_synopsis; pins in run_config.contextual_synopsis and retrieval_config_hash. Test: test/eval-longmemeval-synopsis-tier.serial.test.ts.

  • src/eval/longmemeval/via-operation.ts — gbrain eval longmemeval --via-operation query|search (--op-expand on|off): runs each question through the real operation handler with an agent's stdio MCP context (buildOperationContext, remote, stdio, source default) at the operation's default limit, records via_operation per row (context, expand sent, model calls per touchpoint from a per-question BudgetTracker, latency, CRAG grade, expansion outcome) and run_config.via_operation_summary (tree, p50/p95 latency, model calls per query); folds the op and context into retrieval_config_hash; a run that expects expansion pushes expansion_not_applied on any row that did not expand, which resume.ts counts as an expansion failure (exit 1). Pinned by test/eval-longmemeval-via-operation.serial.test.ts.

  • src/eval/longmemeval/call-plan.ts — gbrain eval longmemeval --call-plan FILE (--call-plan-gate CMD): writes gstack's eval call plan (GSTACK_EVAL_CALLS v1, gstack docs/eval-calls.md; formatEvalCalls copied from its reference emitter) for the run the other flags describe and exits 0. The command hands over right after flags, pins, mode knobs, models and the question filters resolve — before the resume skip, any client, engine or embed call — so the plan describes the same run. One row per active lane (expansion, embed_doc, embed_query, rerank, trajectory_extract, synopsis, fact_keys, reader, judge; lanes the flags turn off are absent, so calls is never 0). Haystacks are chunked offline with the importer's pure functions (screenImportContent, prepareMarkdownChunks at resolveMaxChunkTokens()), so trajectory_extract (distinct page contents) and synopsis (non-code chunks) counts are exact; reader (the K largest rendered sessions plus a worst-case trajectory block), rerank (the top_n_in largest chunks, each with the query) and judge (a reader-max-tokens hypothesis) inputs are upper bounds. ONE token rule, ceil(chars / CALL_PLAN_CHARS_PER_TOKEN) (3), over the real prompt text each lane builds (buildExpansionPrompt, SYNOPSIS_SYSTEM_PROMPT + buildPagePrompt/buildChunkPrompt, EXTRACTOR_SYSTEM_PROMPT, paperFactUserPrompt, buildReaderUserText, buildJudgePrompt); out_tokens = calls × the per-call cap sent (expansion sends none, so the AI SDK default for claude-haiku-4-5, 64000, is used and any other expansion model refuses). Caches, the resume skip and retries are not subtracted; --expansion-replay drops the expansion lane and its file sets the query-embed count. GATE = the command minus --call-plan, POSIX-quoted only where needed; HEAD = gbrain's own checkout; a tracked, unmodified dataset / --question-ids / replay file is an INPUT line, any other file a PRECONDITION with its sha256 and basename. Preconditions name the token rule, the uncached worst case, each lane's model source and env, and the spend caps. Refuses (exit 1, nothing written; FILE.tmp then rename) on --decide* / an inherited GBRAIN_DECIDE_SLOTS, --via-operation, search.cross_modal.llm_intent=true and an unconfigured embedding model. Tests: test/eval-longmemeval-call-plan.test.ts (throwing RunOpts seams, local v1 grammar check, lane matrix, GATE round trip, refusals), test/eval-longmemeval-call-plan.serial.test.ts (exact counts and token bounds against an instrumented run).

  • src/eval/longmemeval/locomo.ts + scripts/locomo-to-longmemeval.ts — LoCoMo locomo10.json → LongMemEval-format questions for an explicit conversation list: one question per QA except category 5, the conversation's sessions as the haystack (stable ids conv-N_sK, LongMemEval-format dates, the last session's date as the question date), gold sessions from evidence dialog ids, speaker names as turn roles. Sealed conversations (LOCOMO_SEALED_CONVERSATIONS) convert only with custodian. Test: test/longmemeval-locomo.test.ts.