Garry's Opinionated OpenClaw/Hermes Agent Brain
Key files: Evaluation
src/eval/longmemeval/evidence-packet.ts+scripts/eval-answer-packet.ts— experimental presentation-only paired comparison over pinned historical session IDs. The pure formatter uses original structured conversational rounds with recoverable pointers and explicit omissions. The runner keeps the existing full-session baseline, uses a leakage-safe development/holdout split, and journals guarded reader/judge spending under one resumable cap. It does not retrieve or change production defaults. Usage and caveats: answer packets. Tests:test/longmemeval-evidence-packet.test.ts,test/eval-answer-packet.test.ts.
Subsystem index. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
scripts/eval-c2-candidate-fusion.ts+scripts/eval-c2-embed.py— the C2 offline eval: a seeded synthetic fixture (40 companies; single-valued claims with changed numbers/dates, coexisting similar claims, private rows, a dense crowded slice, back-to-back corrections) replayed on PGLite throughlistSupersessionCandidatesforfacts.candidate_fusionrrf_freevsinterleave, embedded once with a free local model. Reports twin recall@5, false supersession, preserved distinct claims and correct decisions per threshold, paired by probe and clustered by company (pairedClusterStatistics); at 0.95 the realdecideSingleFactruns beside the replayed rule and must agree. Output:docs/eval/decisions/c2-interleave-dev/verdict.json.src/eval/longmemeval/evidence-brief.ts— experimental, eval-only evidence brief (10x memory advantage plan, wave 1 item A3). Pure:buildBriefPrompt(question-aware) andbuildDigestPrompt(one session, no question) build a cheap builder's prompt over the intact delivered sessions;parseBriefDraftreads its JSON;validateBriefgrounds every quote with the dream-cycle verifier (groundSource/groundQuote/unsupportedNumericClaims, LongMemEval**user:**turns vialmeTurns), drops ungrounded, misattributed, number-unsupported and instruction-context claims (counted invalidation.dropped), withholds a paraphrase whose negation disagrees with its quote, appends uncited later corrections verbatim, never presents a count as complete, replaces absence wording with "no supporting evidence was returned from the searched scope" plus the searched sessions, renders claims inside sanitized<chat_session>blocks within a cl100k budget, and falls back to the full text when parsing or grounding fails. Pointers are UTF-16 spans into the sanitized session text with span and body SHA-256 (verifyPointer). Grounding is provenance, not truth. No model call, no production default. Tests:test/evidence-brief.test.ts(pointer round trip, adversarial fixtures, fallback).src/eval/longmemeval/reader.ts+gateway-client.ts— the LongMemEval answer request seam, exported asgbrain/eval/longmemeval/reader.resolveReaderConfigselects notes-first/1024 by default or the frozen direct-v3/512 baseline;buildReaderRequestchanges only the system instruction and budget, retaining the same full-session sanitizer, evidence order and dates.readerConfigHashpins mode, system-prompt version/sha, requested model and token budget.generateAnswerreports the provider finish reason, joins every text block and keeps the call's provider-normalizedusage(A1:ReaderUsagewith total, uncached, cache-read and cache-write input, output and reasoning tokens; the cache counts are subsets of the total), which the command writes asreader_usageon every answered row (null when no reader call ran).readerUsageFromMessagereads the gateway adapter'sgbrain_usagerecord or the Anthropic Messages convention. The command recordsreader_partial_outputas a diagnostic, never a completed hypothesis, when the reader stops at the limit or returns empty/unknown completion; those failures count in the judged headline denominator and fail the run without automatic paid retries. Tests:test/longmemeval-reader.test.ts,test/eval-longmemeval-reader-mode.slow.test.ts,test/eval-answer-packet.test.ts(historical explicit direct baseline).evals/brainbench/— the committed BrainBench corpus: 141 fixtures (135 generated + 6 hand-authored spike) / 241 gold-annotated turns across 7 categories (kta-pos/kta-neg/push/write-back/continuity/multi-source/adversarial), ~15% holdout (excluded from the CI gate, scored in published--include-holdoutruns).generator/gen.tsrebuilds the corpus byte-identically (Mulberry32, seed 42; whole-cloth fictional universe from curated synthetic name pools so scenario privacy is structural; prose is template-synthesized with PRNG-selected variants — deliberately no LLM pass, difficulty stays controlled; several know-to-ask variants intentionally exercise documented v1 reflex limits so the baseline measures the roadmap).gold/is sealed;schema/is the foreign-runner contract (gbrain-evals drives the suite as a subprocess via--fixtures DIR --gold DIR --json --out FILE);baselines/main.jsonis the committed gate baseline;_ledger.jsonrecords counts/seed/rebuild command. CI: the.github/workflows/test.ymlbrainbenchjob +scripts/ci-brainbench-gate.sh(fetches MAIN's baseline viagit show origin/master:…— a PR cannot rewrite what it's compared against; first-landing path runs ungated) +scripts/render-brainbench-delta.ts(compact step-summary/PR-body delta block from the--outartifact). Privacy:scripts/check-synthetic-corpus-privacy.shscansevals/brainbench/{fixtures,gold}inbun run verify.evals/longmemeval/splits-seed42.json+evals/longmemeval/dev-slice-seed42.txt— the committed, seeded question-id splits that carry the LongMemEval decision discipline (ids only, never question text or session content).schema_version: 1;dataset_sha256+scored: 470/abstention_excluded: 30pin the exact cleaned_scorpus the splits were drawn from;prng= mulberry32 over question_ids sorted ascending with Fisher-Yates; seedsdev: 42,halves470: 4242,halves430: 4243. Lists:dev40(the 40-question dev slice, per-type counts indev40_type_counts),decision430(the held-out remainder every pre-registered success rule is decided on),halfA470/halfB470(235 each) andhalfA430/halfB430(215 each) for select-on-A / confirm-on-B decisions;type_countsrecords the per-type totals of the 470.dev-slice-seed42.txtisdev40one id per line forgbrain eval longmemeval --question-ids. Invariant: mechanisms are chosen on the dev slice or half A and decided ondecision430or half B; a full-470 row published alongside is labelled as including the dev slice.src/core/bench/baseline-file.ts+src/core/bench/qrels-file.ts+src/core/bench/correctness-gate.ts+src/commands/bench-publish.ts+src/commands/eval-gate.ts— the eval loop.gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson>writes a baseline (stamps stablequery_hashper row; metadata header carries_kind: 'baseline_metadata'+ thresholds +source_hash+baseline_mean_latency_ms; deterministic sort by(tool_name, query_hash); strict: empty=fail, dupes=fail with paste-ready hint,--toexists=refuse without--force).gbrain eval gate [--baseline X] [--qrels Y]is the two-gate dispatcher (regression gate via in-processreplayCore, correctness gate via barehybridSearchfor determinism, both must pass when both flags set, exit 0 PASS / 1 FAIL / 2 USAGE). Source-id-aware:bench publishdedup key is(tool_name, source_ids, query_hash); qrels compare keys are${source_id}::${slug}everywhere (so multi-source brains are keyed correctly at the file-shape layer). Latency math:(baseline + delta) / baseline <= multiplier. Fail-closed: ANY in-process throw flips verdict to fail with named breach inbreaches[]— never silently exit 0..qrels.jsonpreserves the 12-rowtest/fixtures/eval-baselines/qrels-search.jsonfixture (slug-onlyrelevant_slugs+first_relevant_slugauto-promote tosource_id='default') AND supports the federated shape (explicitrelevant: [{source_id, slug}]+expected_top1).correctness-gate.tsruns each qrels query via barehybridSearch; per-query throw recorded aserrored: trueand flagged as gate failure. Audit JSONL at~/.gbrain/audit/bench-publish-YYYY-Www.jsonl. Hermetic mode:--embedder deterministic(correctness gate ONLY — rejected with--baseline, requires--qrels) embeds each query as the qrels fixture's basis vector viasrc/eval/deterministic-embed.ts(basisEmbeddingunit vectors; FNV-1a-derived fallback dim for off-fixture query texts) and threads it into barehybridSearchthrough thequeryEmbedFnseam — no API keys, no network; barehybridSearchneither reads nor writes the semantic query cache (both live inhybridSearchCached), so deterministic runs cannot poison cached production results.scripts/run-eval-canary.ts(check:eval-canarypackage script, on-demand; in CI the same runner executes viatest/eval-canary.test.tsin the unit matrix, not in the verify battery) is the hermetic CLI retrieval canary built on it: boots a throwaway PGLite brain under a temp GBRAIN_HOME, seeds the qrels fixture corpus (the expected-top1 page carries its query text intimelinetoo — page-grain FTS indexes title(A) + timeline(C) only,compiled_truthis deliberately unindexed), spawns the REAL CLI with engine-reroute/provider env stripped, and asserts exit 0 + metric floors;--recordadditionally appends an EvalRunRecord-shaped line to.gbrain-evals/eval-results.jsonl. Honest scope: the canary gates the hybrid ranking pipeline (keyword/title/alias arms + RRF against gold qrels) with synthetic vectors — semantic-embedding regressions remain the keyed eval suites' job. Pinned bytest/bench/baseline-file.test.ts,test/bench/qrels-file.test.ts,test/bench/correctness-gate.test.ts,test/bench-publish.test.ts,test/eval-gate.test.ts,test/eval-canary.test.ts,test/eval-replay-metadata-skip.test.ts,test/cycle/nightly-probe-adapters.test.ts,test/autopilot-nightly-probe-wiring.test.ts,test/e2e/eval-loop.test.ts. D2:benchdeclaresjson: 'ndjson':bench publish --jsonwrites its result as one line and every failure ends with a{status:'error'}envelope line (refusals keep their stderr lines).src/core/cross-modal-eval/aggregate.ts— pure verdict logic. Judge dimension keys are normalized with trim + lowercase before cross-model roll-up, soCORRECTNESSandcorrectnesscannot split into separate one-model dimensions. Pass criterion:(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5). Inconclusive when <2/3 models returned parseable scores (guards theObject.values({}).every(...) === trueempty-array PASS trap).src/core/cross-modal-eval/json-repair.ts—parseModelJSON(raw)named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.src/core/cross-modal-eval/receipt-name.ts— receipt filename binds (slug, SKILL.md sha-8).findReceiptForSkill(skillPath, receiptDir)returns'found' | 'stale' | 'missing'. Skillify-check surfaces the status as informational; the audit does NOT fail on missing/stale receipts.src/core/cross-modal-eval/receipt-write.ts— wrapsfs.writeFileSyncwithmkdirSync({recursive:true})ahead of every write (gbrainPath()does NOT auto-mkdir).src/core/cross-modal-eval/runner.ts— orchestrator. The judge prompt pins the exactscoresJSON keys per dimension (dimensionScoreKey= the label before the em-dash) so judges cannot invent spellings; aggregate.ts's normalization is the backstop.buildPromptwraps the task and the candidate in<task_to_grade>/<candidate_output>data blocks, andneutralizeClosingTagrewrites every closing form of those tags inside the untrusted text (case-insensitive, whitespace-tolerant) to the visibly-escaped<\/tag, so a candidate carrying its own closing delimiter cannot terminate the block and land text outside the data boundary — the post-candidate grading-only instruction stays the last thing the judge reads.callSlotsendsEVALUATOR_SYSTEM_PROMPTassystemand the bounded prompt as the single user turn. Each cycle runsPromise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)])(bare allSettled, no rate-leases for the CLI path). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots:openai:gpt-5.2/anthropic:claude-opus-4-7/deepseek:deepseek-v4-pro.estimateCost()prices via the canonical model-pricing table, each slot at the output cap its thinking-off call sends (thinkingOffOutputCap; a raised cap is named in the notes, which both the single-run and batch banners print);test/cross-modal-default-slots.test.tspins recipe support, pricing coverage, and three distinct providers.src/core/eval/cycle-default.ts— single source of truth for the eval cycle-count default. ExportsDEFAULT_CYCLES_TTY = 3,DEFAULT_CYCLES_NONTTY = 1,resolveCycleDefault(explicit, isTty): {cycles, usedNonTtyDefault}, andcycleDefaultSuffix(r)(returns(non-interactive default; --cycles N for more)only when the non-TTY default was applied, else''). Consumed byeval-cross-modal.ts,eval-takes-quality.ts(run + regress), andtakes-quality-eval/runner.ts(core uses only the constant — library stays TTY-agnostic; the CLI owns the TTY=3 upgrade + banner annotation).eval-suspected-contradictions.tsapplies the same transparency to its$5/$1budget default via abudgetUsdExplicitflag (the budget is overwritten in-place so explicitness can't be inferred post-hoc). Not shared withresolveWorkersWithClamp(different domain, no engine, no dedup). Pinned bytest/eval/cycle-default.test.ts,test/eval-suspected-contradictions-budget-default.test.ts.src/core/eval/paired-bootstrap.ts— paired, clustered comparison statistics for per-query eval rows, ported from the gbrain-evals situation-recall comparator (eval/runner/situation-recall-regression.ts).pairedClusterStatistics(pairs, { seed, draws })returns n, clusters, means, delta, wins/losses/ties, a percentile 95% CI from whole-cluster bootstrap resamples, and a two-sided sign-flip p-value (exact enumeration at ≤16 clusters, else Monte Carlo with the (k+1)/(n+1) correction); no discordant pairs → CI [0, 0], p = 1; fewer than two clusters → unbounded CI, p = 1.holmAdjustedis the Holm step-down. Deterministic via the same seeded mulberry32 generator. Requires ≥1000 draws and finite values. Consumed bysrc/commands/eval-compare.ts. Pinned bytest/eval-paired-bootstrap.test.ts.src/core/eval/ranked-docs.ts— dependency-free page-level ranking primitive shared by the retrieval eval harness, NamedThingBench, and the qrels correctness gate.dedupeRankedKeys()preserves the first/best occurrence of each${source_id}::${slug}or slug key before any cutoff, so repeated chunks neither inflate relevance gain nor consume document ranks. Precision retains its fixed-kdenominator; Recall and nDCG are structurally bounded to[0,1]. Pinned acrosstest/eval.test.ts,test/retrieval-quality-harness.test.ts,test/bench/qrels-file.test.ts, andtest/bench/correctness-gate.test.ts.src/core/takes-quality-eval/pricing.ts— budget pricing foreval takes-quality run.getPricing(model, overrides?)returns the operator rate registered withgbrain pricing set(pricing.overrides), else the canonical rate (canonicalLookup), else null; schema{input_per_1m, output_per_1m}. Every canonically priced model is gateable. Miss policy (new models must run): with no--budget-usdan unpriced model runs with a warning naming the registration command and its spend is not counted; under a user cap the runner refuses before any call withno_pricing(unpricedUnderCapError), whose fix isgbrain pricing set <model> --input … --output … --source …(no-pricing.ts). Rates are never copied here;test/model-pricing.test.tspins equality with canonical for every canonical key.src/core/takes-quality-eval/runner.ts—eval takes-quality runorchestrator: samples takes, sends the rendered rubric prompt to every panel model per cycle (allowFallback: false,thinking: 'off'), and re-asks each malformed slot once with the same model and sample plus the validator's error (slotFormatFailureinaggregate.ts:parse_failedincluding an empty reply, orincomplete_scores; provider errors and valid low scores never re-run). The per-cycle projection and each correction pricejudgeCallCostUsd(model): 5k input tokens plus the output cap the call sends (thinkingOffOutputCap(model, JUDGE_MAX_TOKENS); a raised cap is announced on stderr). A correction is priced against--budget-usdbefore it is sent and recordedcorrected: nullwithskipped_reasonwhen it does not fit or the run was aborted; it replaces the first attempt only when it validates. The receipt carriesprotocol_version: TAKES_QUALITY_PROTOCOL_VERSION(2),correction_selection_ruleandcorrections(additive atschema_version: 1);regress.tslists a protocol change among the dissimilar inputs. Pinned bytest/eval-takes-quality-runner.serial.test.tsandtest/eval-takes-quality-regress.test.ts.src/eval/brainbench/+src/commands/eval-brainbench.ts— BrainBench, the cross-harness memory conformance suite (gbrain eval brainbench; methodology indocs/eval/BRAINBENCH.md).types.tscarries the PUBLISHED interchange shapes (fixture/gold/result/baseline — mirrored as JSON Schemas inevals/brainbench/schema/; breaking changes bump the schema versions).fixtures.ts: strict loader/validator + corpusfixtures_hash(covers fixture AND gold files); agoldkey inside a fixture turn is a validation error — gold is SEALED in the gold dir and adapters only ever see sanitizedPublicTurns.seed.ts: fail-fast hermetic seeding (importFromContentnoEmbed + NULL-embeddinginsertFact; any non-importedstatus ⇒SeedError⇒ fixtureseed_failed⇒ run exit 2).adapters/shared.ts: ONErunReflexPipelineall three adapters drive with declarative config (pointer budget, suppression mode) — cross-harness comparability is structural;openclaw.ts(seamproduction, the shipped pipeline),claude-code.ts(seamproduction; drives the shippedgbrain hook user-promptpath end-to-end — fixture turns becomeUserPromptSubmitstdin JSON, synthesized Claude Code JSONL transcripts feed the real window parse + cross-turn dedupe, and resolution rides a run-scoped resolve-IPC server with the real shared secret; bench-pinned deviations disclosed indocs/eval/BRAINBENCH.md: generoususerPromptDeadlineMs, push-failure banner suppressed),codex.ts(seamcontract; static entity-index preamble whose slugs deliberately don't count as injections + ≤1 per-turn fragment; fixture conversations round-trip through the real rollout format + the shipped parsersrc/core/transcripts/codex.tsfor turn selection — fragment DELIVERY remains harness-shaped until a shipped codex injection path lands).metrics/: know-to-ask (+false-fire anti-gaming companion), push (micro-averaged P/R), write-back (drives the PRODUCTION conversation→facts pipeline via the injectable-extractor seam; gold extractor in CI, real extractor under--llm), continuity (writer→reader pairs on a shared brain through DIFFERENT adapters; pointer-injection OR stored-fact keyword probe).harness.ts: ONE in-memory PGLite per run +resetTablesbetween fixtures (longmemeval engine-sharing pattern); read-only suites share one seeding across all adapters; emits per-(harness×suite) cells + re-scoreable turn rows;source_isolation_violationscounted per turn and gated at zero.scoreboard.ts: markdown render, canonical diff-stable committed baseline (4-decimal rounding, sorted keys, receipts excluded),compareBaselineswith main-baseline governance — same-hash count-aware gate vs corpus-bless mode (the committed baseline must byte-match the run; regressions vs main require ajustification). The CLI brings its own PGLite (cli.ts routes before connectEngine), writes--outas the canonical CI artifact, and terminates via an explicit grace-tickprocess.exit(verdict)(0 pass / 1 regression / 2 error) because PGLite stompsprocess.exitCodeand Bun discards queued stdout on exit.runBrainBenchCore()is the in-process entryeval run-alluses (one record per sweep,EvalRunRecordschema_version 3,mode: 'n/a'). Pinned bytest/brainbench-*.test.ts+test/eval-brainbench-e2e.slow.test.ts. System One arm:--decide*flags (src/eval/decide-eval-flags.ts) configure the benchmark brain; the claude-code adapter captures the turn-contextdecide.recall_neededmeta ontoturn_rows[].decide, the result gains adecideblock, and--update-baselineis refused for an arm (test/eval-brainbench-decide.serial.test.ts).src/eval/longmemeval/diagnostics.ts+scripts/lme-miss-diagnostics.ts— LongMemEval miss diagnostics: locate WHERE a strict miss is lost before any fix is chosen. Input is a harness receipt (ndjson) plus the dataset; for every question withrecall_all_hit=false(every scored question under--all) the question's brain is re-created EXACTLY as the harness built it (one in-memory PGLite per run,resetTablesper question,haystackToPages→importFromContent, the embed cache installed through the same seam so every page / question vector is a cache hit) and each gold session missing from the receipt's top-k is located per arm:vector_rank(first chunk row of the session inengine.searchVector, paged by offset todepth, default 200, because one call caps atMAX_SEARCH_LIMIT),keyword_rank(engine.searchKeyword, OR-fallback as hybrid),title_rank(engine.searchTitles),fused_rank_*(the pre-rerank RRF order of ONEhybridSearchcall atfusedLimit, default 50, under the same pins, captured viaonRerankPool(pool, preRerank)),post_rerank_*(that call's post-rerank pool) andfinal_rank_rows(what the call returned after autocut / limit). Frozen classes (classifyMiss): (i) absent from every arm's top-depth, (ii) in an arm pool but outside the fused top-k pre-rerank, (iii) in the fused top-k pre-rerank but reranked out, (iv) ceiling (more gold sessions than k); two observational classes keep the receipt honest —rerun_hit(the re-created run places the gold inside the top-k: the receipt's miss did not reproduce under these pins) andautocut_dropped/post_fusion_dropped(the gold survived fusion + rerank and a later trim removed it). Frozen hypothesis probes:h1Signature(≥ 2 gold, one at fused session rank 1–3, the missing one at 6–15), the counterfactual clause sub-queries (splitClauses/splitClausesDetailedover the frozen pattern listhow_many_between | between | first_or | before_after, with the content-token and quoted-span guardrails; H1 is supported when a clause's own vector top-5 contains the missing gold) and theh3Candidatessplit — H3a candidate generation (gold in the vector top-depth but outside the pre-fusion pool,computeInnerLimit) vs H3b reranker depth (in the fused pool but beyondreranker_top_n_in).splitMembershiptags each question with the committedevals/longmemeval/splits-seed42.jsonsplits (dev40 | decision430 | halfA430 | halfB430 | halfA470 | halfB470). INVARIANT: the clause sub-query embeds BYPASS the embed cache, so a diagnostics run never adds rows to the shared like-for-like cache (its canonical hash is a receipt field). INVARIANT: every helper exceptrunDiagnostics(the one engine-touching entry point;applyPinswrites the pins) is pure over plain data —classifyMiss,splitClauses,h1Signature,h3Candidates, the rank builders,parseReceipt/pinsFromReceipt/receiptTopKSessions,summarizeDiagnostics,renderDiagnosticsMarkdown— so the tests pin them without PGLite, and every printed metric routes through the shared glossary (glossFor, withLOCAL_GLOSSARYfor the diagnostics-only names). The script owns argv, gateway bootstrap, file I/O and printing (<receipt.ndjson> --dataset FILE [--splits FILE] [--mode M] [--reranker on|off] [--autocut on|off] [--expansion-variant-budget legacy|B] [--embed-cache PATH | --no-embed-cache] [--k N] [--depth 200] [--fused-limit 50] [--all] [--question-ids FILE] [--limit N] [--out-ndjson FILE] [--out-md FILE] [--json]); pins default to the receipt's flatrun_configon theby_type_summaryline (mode,reranker.{enabled,model},autocut,expansion,expansion_variant_budget,topK,embedder; a legacy nestedpinsblock is still accepted) and explicit flags win; it is NOT agbrainsubcommand, so its flags sit outside the CLI flag registry by design. Spend: with the receipt's cache every page / question embed is a hit; the ≤ 2 clause sub-query embeds per miss and, under--reranker on, one rerank call per miss are the only paid calls. Exit 0 ok · 1 bad input / run error · 2 gateway or reranker not ready. Pinned bytest/longmemeval-diagnostics.test.ts.src/eval/longmemeval/judge.ts+judge-lane.ts+qa-accuracy.ts— the LongMemEval LLM-as-judge lane behindgbrain eval longmemeval --judge. judge.ts is a faithful port of the officialevaluate_qa.py::get_anscheck_prompt: the per-type instruction (standard forsingle-session-user/single-session-assistant/multi-session; temporal-reasoning adds the off-by-one-days clause; knowledge-update; the single-session-preference rubric) and the abstention instruction for_absids (the id suffix beats the type; an unknown type falls back to standard), ONE user message per question,DEFAULT_JUDGE_MODELopenai:gpt-4o,JUDGE_TEMPERATURE0,JUDGE_MAX_TOKENS16 (the OpenAI API's minimum; the official 10 is rejected, and a one-token yes/no verdict is unaffected), and the official verdict rule (parseJudgeVerdict:'yes'substring of the lowercased completion). Disclosed deviations, carried verbatim inJUDGE_METHODOLOGY_NOTEon every summary: (1) the question / reference / response sit inside<judge_input>data-boundary framing underJUDGE_DATA_BOUNDARY_INSTRUCTION(#4338) —escapeJudgeDataneutralises tag closures inside the data and the response text is otherwise unaltered, so the judge grades what the reader actually said; (2) thejudge_errorclass —classifyJudgeResponsereturns null for a completion that is neither a yes nor a standalone no (malformed, re-judged) instead of a silentno; (3) abstention detected by the_abssuffix; (4) the unknown-type fallback.JUDGE_PROMPT_VERSIONbumps whenever any instruction text, framing or field label changes.judgeConfigHash= sha256 over the stable JSON of{judge_model, prompt_version, max_tokens, temperature, reader_model, reader_prompt_sha, context: {k, max_tokens}}.judgeRowjudges one row from its stored hypothesis: budget-skips on the PROJECTED cost (estimateJudgeCallUsd) BEFORE spending, records actual cost after, never throws on a transport failure, andstripJudgeFieldsremoves every priorjudge_*key first so a re-judge leaves no stale field. judge-lane.ts is the harness-side orchestration, pure exceptrunJudgeBackfill(whose only effects are the injected client and in-place row updates; no engine, no file I/O):parseMaxUsd(N, oroff/none/unlimited→ no cap),makeJudgeConfigHasher(per-row hasher — a prior row's recorded reader pins win over this run's; k is the run's--top-k, already gated byretrieval_config_hash),hasJudgeAttempt/hasSettledVerdict,selectBackfillRows(candidates = rows with a hypothesis and no settled verdict; countssettled,mismatched+ the foreign hashes,retrievalOnly,missingFromDataset),judgePreflight(availability → pricing → estimate → cap: exit 1 no provider, exit 2 unpriced-with-budget or over-cap without--yes) andrunJudgeBackfill(runWithLimitat--judge-concurrency; the dataset's answer is the reference, the row's ownanswerthe fallback). qa-accuracy.ts builds theqa_accuracyblock from ALL rows (pure; last row perquestion_idwins):accuracy_headline(aliasaccuracy) = correct /total_questionsover EVERY question including_abs— ajudge_error, a budget-skipped row, a reader-error row and a never-judged row all score INCORRECT, so it is never more lenient than the official scorer;accuracy_excluding_errors= correct / judged (secondary);accuracy_470= the headline rule over the non-_absquestions (the retrieval-metric denominator);by_typeand anabstentionsub-block;judge_error_classes;ci95_bootstrap(percentile bootstrap over the headline 0/1 vector,label: 'question-sampling only');complete(no error, no skip, no unjudged row — the publishability bit the run-end gate reads);mixed_judge_config;est_cost_usd,actual_cost_usd(summed over the rows' ownjudge_cost_usdacross resumes),run_cost_usd(this run's ledger);methodology_note. Pinned bytest/longmemeval-judge.test.tsandtest/eval-longmemeval-judge.slow.test.ts.src/eval/longmemeval/metrics.ts— LongMemEval strict-recall metrics: the raw-id join,recall_all@k/recall_any@k, per-type buckets, the schema-v2by_type_summary, and the per-question row assembler. PURE: no engine, no I/O, no LLM, so the harness and the tests score the same bytes. The join is on RAW session ids:haystackToPagesimports every session under an OPAQUE per-question id (sessionSlug→chat/s-+ first 10 hex of sha256(question_id:normalized session id), the same id in frontmatter and in the reader's<chat_session id>), because every public gold id starts withanswer_and no distractor does — a raw id in a slug, title, chunk or reader prompt is a gold label (pinned bytest/longmemeval-gold-leak.slow.test.ts);buildSlugToRawMapis the private per-question inverse used only for scoring,distinctRetrievedSessionsjoins through it,detectSlugCollisions/collisionsTouchingGoldname the ambiguous slugs (a gold-touching collision makes the harness emit an error row),goldMissingFromHaystackcounts dataset defects, andnormalizeSessionIdexists only for the opaque-id hash input and legacy-row rawification — never for the gold compare. k semantics:recall_*@kis scored over the DISTINCT sessions among the top-k CHUNK rows returned atlimit: k(the caller slices first;scoreRecalltreats k as a guard); empty gold scores both hits false, not vacuously true, and such rows stay out of the denominator.addRowToBucketfolds a v2 row intototal/all_hit/any_hit; a row carrying only the deprecated any-onlyrecall_hitcounts towardtotal+any_hitand bumpslegacy_rows(itsall_rateis therefore a lower bound);buildByTypeSummaryV2emits sorted type keys,nullrates on empty buckets,mean_distinct_sessions, and the caller'srun_config.buildRowassembles the JSONL row — scored fields overresults.slice(0, k),retrieved[]+retrieved_session_idsover EVERY returned row so replay can re-score at any smaller k — and harness passthroughextrakeys never override the scored fields. Pinned bytest/longmemeval-metrics.test.ts.src/eval/longmemeval/resume.ts— reading a prior run's JSONL back: row scanning, the mixed-run check, expansion-variant replay maps, and per-type bucket re-seeding. INVARIANT: recall is RECOMPUTED, never trusted —seedBucketsFromRowsre-derives both metrics for every prior row fromretrievedIdsAtK(prefersretrieved[]sliced to k; falls back to the olderretrieved_session_idsshape) joined against the dataset's gold, so a file written before the raw-id join simply scores false instead of poisoning the summary, and there is no legacy any-only counter.readJsonlRowsskips corrupt lines (a SIGKILL tail);isScoredQuestionRowexcludes summaries and hypothesis-less error rows (those are retried);checkResumeConfigHashreports rows stamped with a foreignretrieval_config_hash(refused by the harness unless--allow-mixed-run-config) and tolerates unstamped rows;loadExpansionReplaymapsquestion_id→ recordedexpansion_variants. Pure given its inputs (file reads are the only I/O).src/eval/longmemeval/run-config.ts— the resolved retrieval pins,retrieval_config_hash, therun_configreceipt block, secret redaction, and the--question-idsloader. Pure given its inputs (the harness resolves the mode bundle and the embedder string and passes them in).RetrievalPins={mode, keyword_only, reranker: {enabled, model}, autocut, expansion, expansion_variant_budget, embedder (model@dims or 'unconfigured'), top_k, trajectory};retrievalConfigHash(pins, {knobs_hash, knobs_hash_version})is sha256 overstableStringify(sorted keys at every level) of the pins PLUS the resolvedknobsHash, so two runs hash equal regardless of flag order, dataset, output path or clock, and a resume file written under a config snapshot that differs in ANY result-shaping knob — not just a pin — is a different run.redactSecrets(URL userinfo, bearer tokens, secret-lookingkey=valuepairs, provider key prefixes,?password=) is applied to every error string BEFORE it lands in a receipt row or the eval ledger.loadQuestionIdsreads one id per line with#comments, dedupes, and throws on a missing or empty file.buildRunConfigstamps the summary'srun_config(pins,expansion_replay, dataset sha256 + count,question_ids_file, both hashes, theCacheReceipt{path, hits, misses, bypassed, infra_faults, canonical_sha256, sha256}orcache: null+cache_skippedreason, and the countersreranker_skipped_rows,vector_degraded_rows,expansion_failed_rows,expansion_replay_miss,gold_missing_from_haystack,slug_collisions,excluded_abstention,errors).src/eval/retrieval-quality/harness.ts+src/commands/eval-retrieval-quality.ts+test/fixtures/retrieval-quality/namedthing.jsonl+test/fixtures/retrieval-quality/namedthing/corpus.ts— NamedThingBench, the retrieval-quality eval that pins the named-thing-miss failure class. Seven query families, each a distinct failure class:title-substring(the named-thing miss itself),generic-to-named(tourist label → named thing),alias-synonym(declared alias / romanization → canonical),multi-chunk-dilution(one strong chunk among many weak — stresses max-pool),short-vs-rich,graph-relationship(guardrail),hard-negative(precision guard, must NOT return a page).gbrain eval retrieval-quality <fixture.jsonl>runs it with hard gates (e.g. title-substring Hit@1 ≥ 0.95, alias Hit@1 ≥ 0.98, multi-chunk-dilution Hit@3 = 1.0). Pure: caller injects aSearchFn(CLI useshybridSearch, tests stub) so it's engine-agnostic. A search call that throws is stampederrored: trueand scored as a miss in every family (never a clean hard-negative);report.errored/ per-familyerroredcount them and any errored question fails the gate. Metric glossary entries (hit@1/hit@3) added tosrc/core/eval/metric-glossary.ts. Pinned bytest/eval-retrieval-quality.test.ts+test/retrieval-quality-harness.test.ts. The seed corpus (NAMEDTHING_CORPUS,seedNamedThingCorpus(engine, { embed? }),loadNamedThingQuestions()) lives intest/fixtures/retrieval-quality/namedthing/corpus.ts— the ONE brain the fixture is written against (sibling ofrelational/corpus.ts), seeded without vectors by the hermetic gate and with real vectors by the R1 A/B script, so a paid receipt and the CI gate describe the same pages. The committed fixture is 12 queries — 11 non-relational core queries plus onegraph-relationshipquery, which R1 scores together withrelational/corpus.ts's 38RELATIONAL_QUESTIONSas the 39 relational questions.seedNamedThingCorpusseeds exactly what the gate always seeded (putPagewithcompiled_truth= the chunks joined, oneupsertChunksrow per chunk withtoken_count: 10, lower-casedsetPageAliaseswhen declared) and, whenembedis supplied, embeds every chunk text in ONE batch and stores the vectors (a count mismatch throws); the result reportspages,chunks,embedded,embedded_charsfor spend accounting. Change pages, chunk boundaries or aliases only together with the fixture and its gate expectations.--decide*runs read-only on the connected brain: missing provider/key/consent refuses with the catalogued line, write-only flags are refused,rerank=onpins the Jev reranker per call, and--jsonaddsdecidewith per-query receipts (test/eval-retrieval-quality-decide.serial.test.ts).src/eval/shared/autocut-replay.ts+scripts/replay-autocut-floor.ts— pure replay of the autocut weak-top-floor sweep over captured rerank pools, and its CLI. INVARIANT: the replay uses the SAMEapplyAutocutthe live path uses, over the SAME pre-autocut reranked pool (gbrain eval longmemeval --capture-pool→rerank_pool, unscored alias/exact-lookup rows included — dropping them would changedecision.total), with the SAME preserve predicate (alias_hit === true || exact_lookup === true), and slices to k AFTER autocut exactly like hybrid.ts, sovalidateLive(rows, floor)can demand byte-for-byte agreement with the recorded live decision (search_meta.autocut, plusautocut_kept_keyswhen present) before any other floor cell is trusted. Every cell — including flooroff— comes from ONE capture; no second reranker call. Metric semantics mirror the harness (distinctsessions among the first k kept chunk rows;recall_all= gold ⊆ distinct;recall_any= non-empty intersection), with the benefit metricsmean_returned_results/mean_returned_est_tokens(why autocut exists) and recall as the guardrail;pairedDeltagives wins/losses/net per type vs the first floor,splitHalf(rows, seed)gives a seeded half-A/half-B selection/confirmation split,topScoreHistogrampublishes the reranker's top-score distribution.normalizePoolRowthrows on a row withoutslug/session_id(a silently dropped row shifts every cliff). The script owns argv (<capture.ndjson> --floors off,0.10,… [--k 5] [--validate-live F] [--split-half SEED] [--jump 0.2] [--min-keep 1] [--json];--validate-livetakes exactly one floor), file I/O and printing; every metric it prints routes throughsrc/core/eval/metric-glossary.ts(REPLAY_GLOSSARY_KEYS); exit 0 ok · 1 validate-live mismatch or bad input · 2 usage. Pinned bytest/replay-autocut-floor.test.ts.src/eval/shared/embed-cache.ts— content-addressed embedding cache for fixed-corpus evals (dataset-agnostic; the LongMemEval harness is its first consumer). INVARIANT: a cached vector is served ONLY for the exact(model@dims, text, side)it was computed for, and every row is integrity-checked on read (declared dims == stored bytes / 4 == vector length); a mismatch is a HARDEmbedCacheIntegrityErrornaming the file and the key — never a silent re-embed, because a silently corrupted cache makes every arm's vectors non-comparable. Key =${model}@${dims}(+#queryfor query-side asymmetric embeds, so asymmetric query vectors can never be served a document-side row) +:+ sha256(text). Storage:bun:sqlite, WAL journal +synchronous=NORMAL, busy retry; local filesystem only. A question's writes are buffered in memory (reads see them) and written in one shortBEGIN IMMEDIATEtransaction when itswithTransactionunit ends, so no SQLite lock is held across embed round-trips and concurrent runs sharing one cache file neither lose writes toSQLITE_BUSY_SNAPSHOTnor stall behind each other's questions.installEmbedCacheinstalls it through the gateway's__setEmbedTransportForTestsseam (gateway.ts sits at its module-size ceiling; a namedsetEmbedTransport()hook is a filed follow-up) and restores the caller-suppliedrealTransport ?? nullon uninstall. Stats:hits,misses(genuine misses PLUS every value of a batch that fell open on an infrastructure fault),bypassed(SDK model id ≠ the installed model — never cached),infra_faults(file deleted mid-run / table dropped / disk error → uncached re-embed or lost write-back, so a run that lost its cache can never show a cleanmisses: 0). The canonical hash (PRAGMA wal_checkpoint(TRUNCATE)then sha256 over the sorted(key, dims, sha256(vector))rows) goes intorun_config.cache.canonical_sha256so two runs can prove they saw the same vectors. Pinned bytest/longmemeval-embed-cache.test.ts.src/eval/shared/gateway-bootstrap.ts—configureEvalGateway({ chatModel? }), the one AI-gateway bootstrap for eval runners that bring their own brain instead of callingconnectEngine:configureGateway(buildGatewayConfig(loadConfig() ?? env)), where env suppliesGBRAIN_EMBEDDING_MODEL/GBRAIN_EMBEDDING_DIMENSIONSand provider keys fold from both planes.chatModelfills in only when the brain configures nochat_model. Consumers: thegbrain eval longmemevalbranch of cli.ts,scripts/r1-namedthing-rerank-ab.ts,scripts/lme-miss-diagnostics.ts,scripts/eval-answer-packet.ts,evals/functional-area-resolver/harness-runner.tsandevals/takes-bootstrap/harness.mjs.src/eval/shared/judge-runner.ts+src/eval/shared/bootstrap.ts— dataset-agnostic judge mechanics shared by the LongMemEval lane and any later judged eval. judge-runner.ts owns the chat-client seam (JudgeChatFn— the gatewaychatin production, a canned fn in tests), bounded retries with exponential backoff (default 2 retries, base 500 ms,sleepseam), the closedjudge_errorvocabulary (timeout | rate_limit | empty | refusal | malformed | provider_error, exported asJUDGE_ERROR_CLASSES), usage summed over every attempt, per-call cost through the ONE canonical pricing table (canonicalLookup; an unpriced model yieldscost_usd: null, never 0, andisJudgeModelPricedlets a budget refuse it), and theBudgetLedger(maxUsd === null= no cap;canAfford(next)is PROJECTED so a run never overshoots by more than the calls already in flight; unpriced calls are counted, not summed;snapshot()for receipts). INVARIANT: a judge malfunction is an error outcome, never averdict: 'incorrect'—runJudgenever throws on a transport failure (only an already-abortedAbortSignalor a bug in the caller'sparsecan), classifies 429 / rate-limit prose →rate_limit, abort / timeout →timeout, everything else →provider_error, and maps arefusal/content_filterstop, an empty completion and a null fromparsetorefusal/empty/malformed; the caller decides how the headline scores them. It passestemperatureandmaxTokensthroughChatOptsand recordsChatResult.responseModelasresponse_model. bootstrap.ts:bootstrapMeanCi(xs, {resamples = 10000, seed = 42, alpha = 0.05})is a seeded (mulberry32) percentile bootstrap of the mean, byte-reproducible; n = 0 → nulls, n = 1 → a degenerate[x, x]. INVARIANT: every block carrieslabel: 'question-sampling only'— the interval quantifies how much the number would move under a different draw of questions and says nothing about reader / judge nondeterminism, dataset revision or prompt drift. Pinned bytest/longmemeval-judge.test.ts.src/eval/longmemeval/retrieval-arms.ts— eval-only retrieval arms forgbrain eval longmemeval, flags spread into the harness flag table.--fact-keys chunk|pageextracts each imported session's facts (--fact-extractor paper= the benchmark's published user-turn fact prompt,production=extractFactsFromTurnWithOutcomeas shipped;--fact-keys-model,--fact-keys-max-usdcap on uncached spend, default 20) through a content-hash JSONL cache under~/.cache/gbrain-eval/, then re-embeds that session's chunks withfactKeyedEmbeddingInputinside the embed-cache transaction (row fieldfact_keys).--time-scope reserved|partitionwidens the search limit to--time-scope-pool(default 50), parses the question againstquestion_date, re-orders by each session's observation + mentioned dates, and recordstime_scopeplus the unscoped top-k'sunscoped_recall_all_hit/unscoped_recall_any_hitfrom the same pool (a paired control inside one run). Active arms fold intoretrieval_config_hashasretrieval_arms; with no arm on the hash is unchanged. Test:test/fact-keys.test.ts.src/eval/longmemeval/synopsis-tier.ts— makes--mode tokenmaxmeasure the production per-chunk synopsis tier: when the resolved tier isper_chunk_synopsis, every imported session is re-embedded throughreembedPageWithContextualRetrievalwith the backfill's synopsis model (resolveContextualSynopsisModel), behind a JSONL cache keyed bybuildSynopsisCacheKey(only answers the model returned are cached) and a--synopsis-max-usdcap metered from provider usage (default 20; the run stops after the question that reached it and exits 1). Transient service errors retry with backoff. Row fieldcontextual_synopsis; pins inrun_config.contextual_synopsisandretrieval_config_hash. Test:test/eval-longmemeval-synopsis-tier.serial.test.ts.src/eval/longmemeval/via-operation.ts—gbrain eval longmemeval --via-operation query|search(--op-expand on|off): runs each question through the real operation handler with an agent's stdio MCP context (buildOperationContext, remote, stdio, sourcedefault) at the operation's default limit, recordsvia_operationper row (context,expandsent, model calls per touchpoint from a per-question BudgetTracker, latency, CRAG grade, expansion outcome) andrun_config.via_operation_summary(tree, p50/p95 latency, model calls per query); folds the op and context intoretrieval_config_hash; a run that expects expansion pushesexpansion_not_appliedon any row that did not expand, whichresume.tscounts as an expansion failure (exit 1). Pinned bytest/eval-longmemeval-via-operation.serial.test.ts.src/eval/longmemeval/call-plan.ts—gbrain eval longmemeval --call-plan FILE(--call-plan-gate CMD): writes gstack's eval call plan (GSTACK_EVAL_CALLS v1, gstackdocs/eval-calls.md;formatEvalCallscopied from its reference emitter) for the run the other flags describe and exits 0. The command hands over right after flags, pins, mode knobs, models and the question filters resolve — before the resume skip, any client, engine or embed call — so the plan describes the same run. One row per active lane (expansion,embed_doc,embed_query,rerank,trajectory_extract,synopsis,fact_keys,reader,judge; lanes the flags turn off are absent, socallsis never 0). Haystacks are chunked offline with the importer's pure functions (screenImportContent,prepareMarkdownChunksatresolveMaxChunkTokens()), sotrajectory_extract(distinct page contents) andsynopsis(non-code chunks) counts are exact; reader (the K largest rendered sessions plus a worst-case trajectory block), rerank (the top_n_in largest chunks, each with the query) and judge (a reader-max-tokens hypothesis) inputs are upper bounds. ONE token rule,ceil(chars / CALL_PLAN_CHARS_PER_TOKEN)(3), over the real prompt text each lane builds (buildExpansionPrompt,SYNOPSIS_SYSTEM_PROMPT+buildPagePrompt/buildChunkPrompt,EXTRACTOR_SYSTEM_PROMPT,paperFactUserPrompt,buildReaderUserText,buildJudgePrompt); out_tokens = calls × the per-call cap sent (expansion sends none, so the AI SDK default for claude-haiku-4-5, 64000, is used and any other expansion model refuses). Caches, the resume skip and retries are not subtracted;--expansion-replaydrops the expansion lane and its file sets the query-embed count. GATE = the command minus--call-plan, POSIX-quoted only where needed; HEAD = gbrain's own checkout; a tracked, unmodified dataset /--question-ids/ replay file is an INPUT line, any other file a PRECONDITION with its sha256 and basename. Preconditions name the token rule, the uncached worst case, each lane's model source and env, and the spend caps. Refuses (exit 1, nothing written; FILE.tmp then rename) on--decide*/ an inheritedGBRAIN_DECIDE_SLOTS,--via-operation,search.cross_modal.llm_intent=trueand an unconfigured embedding model. Tests:test/eval-longmemeval-call-plan.test.ts(throwing RunOpts seams, local v1 grammar check, lane matrix, GATE round trip, refusals),test/eval-longmemeval-call-plan.serial.test.ts(exact counts and token bounds against an instrumented run).src/eval/longmemeval/locomo.ts+scripts/locomo-to-longmemeval.ts— LoCoMolocomo10.json→ LongMemEval-format questions for an explicit conversation list: one question per QA except category 5, the conversation's sessions as the haystack (stable idsconv-N_sK, LongMemEval-format dates, the last session's date as the question date), gold sessions from evidence dialog ids, speaker names as turn roles. Sealed conversations (LOCOMO_SEALED_CONVERSATIONS) convert only withcustodian. Test:test/longmemeval-locomo.test.ts.