Garry's Opinionated OpenClaw/Hermes Agent Brain
System One labelled datasets
Labelled inputs for gbrain decide calibrate / qualify and the matched-pair evals of three slots:
S6 recall_needed, S9 conflict and S8 grounding. Every dataset line is in the
src/core/ai/decide/dataset.ts schema, written by gbrain decide dataset (which assigns the frozen
calibrate/eval split by a stable hash of the family), then stamped with its label provenance by
generators/label-source.ts. That step only adds fields (label_source, and per dataset design, case,
origin, baseline_*), so the split and split_hash are exactly the builder's and parseDatasetJsonl
reads the files unchanged. Hashes, item and family counts: HASHES.md.
Label provenance
Every line carries label_source:
| Value | Meaning |
|---|---|
synthetic-construction |
The label is fixed by the deterministic template that built the item (the generator's intent). No person or model labelled it. |
upstream-gold |
The label comes from an existing corpus or test in this repository (BrainBench sealed gold; labelled or verdict-bearing fact pairs in test/). |
llm:<model> |
A model labelled it (S8: llm:claude-sonnet-5). These are not hand labels. |
human |
Labelled by a person. No dataset here contains human labels yet. |
The design plan asks for a hand-labelled S8 sample for quarantine precision; the S8 set below is an LLM-judged set and does not satisfy that requirement. A human pass over a sample of it is still owed before any S8 quarantine precision is reported as hand-validated.
S6 recall_needed (know-to-ask)
Files. s6-recall-needed.brainbench.jsonl (the committed BrainBench corpus),
s6-recall-needed.extra.jsonl (the extension corpus below) and s6-recall-needed.combined.jsonl
(the two concatenated; ids never collide).
Source. evals/brainbench (sealed, unchanged) and the new BrainBench-schema corpus
know-to-ask-extra/{fixtures,gold} (132 fixtures, 528 gold user turns, suite know-to-ask), generated by
bun docs/eval/system-one/generators/know-to-ask-extra.ts (Mulberry32, seed 60601, byte-identical reruns,
synthetic name pools only: common first names, invented surnames and company names). _design.json records
the designed intent and subtype of every gold turn and is copied into each extra line as design.
Generator commands.
bun docs/eval/system-one/generators/know-to-ask-extra.ts
gbrain decide dataset --slot recall_needed --from know-to-ask evals/brainbench --out bb.raw.jsonl
gbrain decide dataset --slot recall_needed --from know-to-ask docs/eval/system-one/datasets/know-to-ask-extra --out extra.raw.jsonl
bun docs/eval/system-one/generators/label-source.ts bb.raw.jsonl docs/eval/system-one/datasets/s6-recall-needed.brainbench.jsonl upstream-gold
bun docs/eval/system-one/generators/label-source.ts extra.raw.jsonl docs/eval/system-one/datasets/s6-recall-needed.extra.jsonl synthetic-construction docs/eval/system-one/datasets/know-to-ask-extra/_design.json
cat docs/eval/system-one/datasets/s6-recall-needed.{brainbench,extra}.jsonl > docs/eval/system-one/datasets/s6-recall-needed.combined.jsonl
The builder seeds every fixture into an in-memory brain and replays the shipped reflex
(assembleTurnContext, all decide slots off, the hook's 4-turn window, cross-turn dedupe), so slice
(reflex:fired / reflex:silent) and protected (alias or exact-title identity hit) record what the
reflex actually did.
Family. One item per user turn (<fixture>#<turn>, one request each, the production shape); the split
is by fixture, so turns of one conversation never straddle the halves.
Labels. BrainBench lines are upstream-gold (the corpus's sealed gold, generator-constructed and
validated by the blind double-label pass its README describes). Extra lines are synthetic-construction.
Each extra turn targets its own seeded entity, never re-mentioned in the same fixture, so BrainBench's
"re-mentions demote" convention never applies. The categories and why each label is defensible:
| Design | Built so the reflex… | Gold | Why |
|---|---|---|---|
fire/*: no-alias first name, lowercase surname, lowercase full name (no alias), lowercase company, partial company name, indirect person ("my cofounder"), indirect vendor ("the vendor we picked for payroll"), indirect project ("that pricing thing") |
stays silent | should_retrieve: true, gold_slugs = the seeded page |
The seeded page holds the fact the question asks for; the prompt does not state it. |
suppress/surname-*: spelling, arithmetic from the prompt, copyedit, general-knowledge question, reformat, translate, letter count, unit conversion |
fires on the capitalized surname (title-surname arm, unprotected) | false |
The request is self-contained: everything needed is in the prompt or is general knowledge. |
suppress/alias-name-question, full-name-self-contained, company-title-self-contained |
fires with an identity hit (protected) | false |
Same self-contained requests; S6 may never suppress these (identity protection), so they test that the reducer leaves them alone. |
keep/surname-needed |
fires on the capitalized surname (unprotected) | true |
The answer needs the seeded page. These are the turns a wrong suppression would hurt, so suppression precision is measured against real risk. |
negative/*: small talk, generic question, incidental lowercase name (pronunciation, character naming), other-referent name ("Captain X" in a novel, "X Street"), unseeded name |
stays silent | false |
Nothing stored is needed to answer. |
Reflex replay (actual counts). Every one of the 528 extra turns came out on the designed reflex slice
and protection flag (0 mismatches against _design.json).
| Dataset | fired, needed, protected | fired, needed, unprotected | fired, not needed, protected | fired, not needed, unprotected (suppress opportunity) | silent, needed (fire opportunity) | silent, not needed |
|---|---|---|---|---|---|---|
| BrainBench | 57 | 5 | 0 | 0 | 0 | 87 |
| Extra | 0 | 66 | 44 | 121 | 165 | 132 |
| Combined | 57 | 71 | 44 | 121 | 165 | 219 |
The shipped reflex resolves every BrainBench know-to-ask turn (its baseline failure rate is 0), so BrainBench
alone offers S6 no fire and no suppress opportunity; the extra corpus supplies all of them. Eval half of the
extra corpus: 62 unprotected suppress opportunities (at least 35 of 35 correct are needed for
action_precision_lb ≥ 0.90), 21 protected, 33 fired-and-needed unprotected turns, 82 fire opportunities,
66 silent negatives.
S9 conflict (hot-memory contradiction)
Files. s9-conflict/pairs.jsonl (the facts-fixtures input, with baselines),
s9-conflict/pairs.sweep-eligible.jsonl (the pairs at cosine ≥ 0.80), and the builder outputs
s9-conflict.jsonl (all pairs) and s9-conflict.sweep-eligible.jsonl.
Source. The repository holds few labelled fact pairs: the hot-memory classifier tests use placeholder
texts ('x', 'new'), and the suspected-contradictions probe set (test/fixtures/contradictions-mini.jsonl)
is five queries, not pairs. Collected as upstream-gold (6 pairs, source_ref names the test): the three
labelled facts-fixtures cases and the proposal accept/undo pair in test/decide/conflict.test.ts, the MRR
contradiction in test/eval-contradictions-judge.test.ts, and the temporal take supersession in
test/eval-contradictions-auto-supersession.test.ts (verdicts mapped to supersede as the tests resolve
them). The five probe queries seed 13 placeholder pairs (case: probe:*). The rest is
synthetic-construction from bun docs/eval/system-one/generators/conflict-pairs.ts (Mulberry32, seed
90909): 12 attributes over invented people and companies (role, city, employer, drink, contact channel,
one-on-one cadence, headcount, customers, stage, pricing, database, headquarters).
duplicate: the candidate restates the new fact's value (restatement,paraphrase-low-overlap, andrestates-updated-value: an older memory that already holds the value an update states).supersede: the new fact updates the candidate's value with explicit newer framing (value-update-near: same wording plus "as of / since / starting 2026";value-update: narrative "promoted / moved / switched from … to …";negation-update: "no longer"). Ordered attributes only move forward.independent: same entity, a different or compatible attribute (compatible-same-entity,other-attribute, andnear-independent: same sentence shape and shared tokens, a different attribute or referent, e.g. "has N employees" vs "has M paying customers").
Family. Pairs sharing a new fact share a family (the production request: the new fact is the state, one
choice question per candidate); 408 families, 1 to 3 candidates each.
Counts. 786 pairs: duplicate 159, supersede 179, independent 448. slice is the choice label on
purpose: the facts-fixtures builder keeps only a duplicate boolean as label, and slice is where the
three-way label survives (hard-case tags are in case).
Baseline (today's rule). generators/conflict-baseline.ts embeds every distinct text with OpenAI
text-embedding-3-large at 1536 dimensions and stores per pair baseline_cosine, baseline_decision (the
decideSingleFact rule applied within the family: exact normalized match → duplicate; otherwise only the
family's highest-cosine candidate can match, and at cosine ≥ 0.95 it supersedes when the kinds agree, else
duplicate; everything else is inserted, independent), baseline_model and sweep_eligible (cosine ≥ 0.80,
the S9 sweep's candidate floor). Vectors are cached locally, not committed. Results:
- The cosine rule marks no true supersede (0 of 179; the highest supersede cosine is 0.926), supersedes 12 duplicates of the same kind (cosine ≥ 0.95) and inserts everything else.
- Median cosine: duplicate 0.818, supersede 0.763, independent 0.581. Only 172 pairs clear the sweep floor
(supersede 59, duplicate 96, independent 17); a supersede whose new value is an unseen name or number often
sits below 0.80 with this embedding model, so the floor, not the classifier, bounds what S9 can find.
s9-conflict.sweep-eligible.jsonlis the set production would actually ask about (eval half: supersede 30, duplicate 49, independent 9).
S8 grounding (claim support)
Files. s8-grounding/labels.jsonl (the grounding-labels input with provenance),
s8-grounding/transcripts/ (the 16 Cat 35 transcripts it references) and the builder output
s8-grounding.jsonl.
Source. Cat 35 transcript-distill corpus in github.com/garrytan/gbrain-evals at
88d0b1997e4b7fb3bd61e04d6eb637ef166ea6d4 (MIT; a fictional, model-generated corpus with invented people and
companies): dream pages docs/benchmarks/2026-08-16-brainbench-cat35-transcript-distill/artifacts/*.dream.md
against eval/data/transcript-distill-v1/transcripts-txt/*.txt. The transcripts are copied unchanged so
transcript_path resolves from the repository root (transcripts/SOURCE.md). Eight of the 24 dream
artifacts are empty (routine transcripts produce no page), so 16 pages contribute.
Units. verifyBody (src/core/cycle/synthesize-verify.ts) against the page's transcript;
groundingUnits are exactly the units S8 judges in production (they pass the mechanical checks only because
they carry no quote, number or attribution): 567 units.
Labels. llm:claude-sonnet-5, one call per page with the transcript and all its units, strict rubric
matching the production question (supported only when the transcript states or directly implies the claim;
added reasons, motives, generalizations or evaluations are unsupported; when unsure, unsupported). The
judge's one-line reason is kept as judge_reason. The model's temperature is not settable, so a rerun without
the local response cache can differ.
Perturbation negatives. Per page, up to 12 evenly spaced units the judge found supported were rewritten
by the same model into fluent claims the transcript does not support (one key element changed; no numbers,
quotes or speaker attribution). A rewrite is kept only when it still passes the mechanical checks as a
grounding unit and a separate blind judge call labels it unsupported: 183 kept. They carry
origin: perturbation, source_unit (the unit rewritten) and label_source: llm:claude-sonnet-5
(model-written and model-judged, not hand-labelled).
Counts. 750 items: dream-page supported 320, dream-page unsupported 247, perturbation unsupported 183.
Family. The page (cat35/<id>): 16 families, so the frozen split is coarse: 5 pages / 254 items
calibrate, 11 pages / 496 items eval. Family-level qualification over 16 independent families cannot reach
insufficient_n's minimum, so S8 is expected to stay shadow-only with this data, as the plan predicts.
Builder. gbrain decide dataset --slot grounding --from grounding-labels docs/eval/system-one/datasets/s8-grounding/labels.jsonl run from the repository root on a brain initialised
with --no-embedding: source windows were selected by normalized substring and keyword neighbours only (the
output is byte-identical with the embedding keys unset). slice is the coverage the builder found
(adequate 571, weak 179; weak items are protected and can only become insufficient_context).
Generator command.
bun docs/eval/system-one/generators/grounding-labels.ts ~/.capy/work/gbrain-evals
gbrain decide dataset --slot grounding --from grounding-labels docs/eval/system-one/datasets/s8-grounding/labels.jsonl --out s8.raw.jsonl
bun docs/eval/system-one/generators/label-source.ts s8.raw.jsonl docs/eval/system-one/datasets/s8-grounding.jsonl llm:claude-sonnet-5 docs/eval/system-one/datasets/s8-grounding/labels.jsonl
S7 triage (dream triage)
s7-triage.jsonl is built with gbrain decide dataset --slot triage --from cat35 docs/eval/system-one/datasets/s7-triage.
The directory holds 254 transcripts in the Cat 35 layout (gold/<id>.json + transcripts-txt/<date>-<id>.txt):
- 24 Cat 35 transcripts copied unchanged from garrytan/gbrain-evals
eval/data/transcript-distill-v1at88d0b1997e4b7fb3bd61e04d6eb637ef166ea6d4(MIT; itself a synthetic corpus). Label: the corpus's ownexpected_triage(label_source: upstream-gold). - 230 synthetic transcripts from
docs/eval/system-one/generators/s7-synthetic-transcripts.ts(openai:gpt-5.6-luna, reasoning none, temperature 1, seeded spec list): 180 entirely routine (label false) and 50 mostly routine with one buried synthesis-worthy passage (label true). Labels are the generator's instructions (label_source: synthetic-construction); nobody read and labelled them. A name check found no planted person or project name in any routine transcript.
Family = transcript. Eval half: 109 transcripts (79 negative, 30 positive).
S2 intent (query routing)
s2-intent.jsonl = gbrain decide dataset --slot intent --from longmemeval longmemeval_s_cleaned.json (470
non-abstention questions, search and think call sites, labels from question_type) plus --from brainbench test/fixtures/retrieval-quality/relational/relational.jsonl (38 relational queries). Labels are the corpora's own
annotations mapped by the builder (upstream-gold). The search label set this produces is only temporal and
general, so it measures a slice of the search question.
S5 injection
s5-injection.jsonl = gbrain decide dataset --slot injection --from injection-fixtures test/fixtures/decide/injection-cases.jsonl: 12 cases, 4 with an attack candidate (upstream-gold). Too small to
calibrate or qualify on; used for a detection check only.
S3 evidence and S4 answerable (not committed: size)
Both are regenerated from LongMemEval-S cleaned (MIT, sha256 d6f21ea9d60a0d56…):
- S3:
gbrain decide dataset --slot evidence --from longmemeval longmemeval_s_cleaned.json→ 10,000 items in 500 families (question), 20 sessions per family, label = the session is an answer session; split_hash036999bf920b7039(57 MB). Candidate text is the session capped at 6,000 characters, the same capcapRerankDocapplies to production candidates. - S4:
gbrain decide dataset --slot answerable --from longmemeval longmemeval_s_cleaned.json→ 500 items, label false for the 30_absquestions; split_hashc2d7aea440b81e88.
The LongMemEval question lists the retrieval arms ran on are in longmemeval/: s-eval-half.txt (the 248
eval-half families of the S3 split), s-eval-judged-100.txt (85 answerable by hash order + all 15 abstention
questions of the eval half) and m-pilot-28.txt (the frozen gbrain-evals LongMemEval-M pilot selection).
Spend
Paid calls are logged one line per request batch in ../ledger-datasets.jsonl (tokens from the provider's
reported usage, priced at list: Claude Sonnet 5 $3 / $15 per million input / output tokens, OpenAI
text-embedding-3-large $0.13 per million). Total for these datasets: $2.52 (S8 labelling and perturbation
$2.51, S9 embeddings $0.004 including superseded iterations, two model-id probes $0.0002), under the $3 cap.
The S8 generator refuses to start a call that would take the ledger past $3.
Regenerating hashes
bun docs/eval/system-one/generators/hashes.ts rewrites HASHES.md.