Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
17.0 KB
Raw Blame

System One labelled datasets

Labelled inputs for gbrain decide calibrate / qualify and the matched-pair evals of three slots: S6 recall_needed, S9 conflict and S8 grounding. Every dataset line is in the src/core/ai/decide/dataset.ts schema, written by gbrain decide dataset (which assigns the frozen calibrate/eval split by a stable hash of the family), then stamped with its label provenance by generators/label-source.ts. That step only adds fields (label_source, and per dataset design, case, origin, baseline_*), so the split and split_hash are exactly the builder's and parseDatasetJsonl reads the files unchanged. Hashes, item and family counts: HASHES.md.

Label provenance

Every line carries label_source:

Value Meaning
synthetic-construction The label is fixed by the deterministic template that built the item (the generator's intent). No person or model labelled it.
upstream-gold The label comes from an existing corpus or test in this repository (BrainBench sealed gold; labelled or verdict-bearing fact pairs in test/).
llm:<model> A model labelled it (S8: llm:claude-sonnet-5). These are not hand labels.
human Labelled by a person. No dataset here contains human labels yet.

The design plan asks for a hand-labelled S8 sample for quarantine precision; the S8 set below is an LLM-judged set and does not satisfy that requirement. A human pass over a sample of it is still owed before any S8 quarantine precision is reported as hand-validated.

S6 recall_needed (know-to-ask)

Files. s6-recall-needed.brainbench.jsonl (the committed BrainBench corpus), s6-recall-needed.extra.jsonl (the extension corpus below) and s6-recall-needed.combined.jsonl (the two concatenated; ids never collide).

Source. evals/brainbench (sealed, unchanged) and the new BrainBench-schema corpus know-to-ask-extra/{fixtures,gold} (132 fixtures, 528 gold user turns, suite know-to-ask), generated by bun docs/eval/system-one/generators/know-to-ask-extra.ts (Mulberry32, seed 60601, byte-identical reruns, synthetic name pools only: common first names, invented surnames and company names). _design.json records the designed intent and subtype of every gold turn and is copied into each extra line as design.

Generator commands.

bun docs/eval/system-one/generators/know-to-ask-extra.ts
gbrain decide dataset --slot recall_needed --from know-to-ask evals/brainbench --out bb.raw.jsonl
gbrain decide dataset --slot recall_needed --from know-to-ask docs/eval/system-one/datasets/know-to-ask-extra --out extra.raw.jsonl
bun docs/eval/system-one/generators/label-source.ts bb.raw.jsonl docs/eval/system-one/datasets/s6-recall-needed.brainbench.jsonl upstream-gold
bun docs/eval/system-one/generators/label-source.ts extra.raw.jsonl docs/eval/system-one/datasets/s6-recall-needed.extra.jsonl synthetic-construction docs/eval/system-one/datasets/know-to-ask-extra/_design.json
cat docs/eval/system-one/datasets/s6-recall-needed.{brainbench,extra}.jsonl > docs/eval/system-one/datasets/s6-recall-needed.combined.jsonl

The builder seeds every fixture into an in-memory brain and replays the shipped reflex (assembleTurnContext, all decide slots off, the hook's 4-turn window, cross-turn dedupe), so slice (reflex:fired / reflex:silent) and protected (alias or exact-title identity hit) record what the reflex actually did.

Family. One item per user turn (<fixture>#<turn>, one request each, the production shape); the split is by fixture, so turns of one conversation never straddle the halves.

Labels. BrainBench lines are upstream-gold (the corpus's sealed gold, generator-constructed and validated by the blind double-label pass its README describes). Extra lines are synthetic-construction. Each extra turn targets its own seeded entity, never re-mentioned in the same fixture, so BrainBench's "re-mentions demote" convention never applies. The categories and why each label is defensible:

Design Built so the reflex… Gold Why
fire/*: no-alias first name, lowercase surname, lowercase full name (no alias), lowercase company, partial company name, indirect person ("my cofounder"), indirect vendor ("the vendor we picked for payroll"), indirect project ("that pricing thing") stays silent should_retrieve: true, gold_slugs = the seeded page The seeded page holds the fact the question asks for; the prompt does not state it.
suppress/surname-*: spelling, arithmetic from the prompt, copyedit, general-knowledge question, reformat, translate, letter count, unit conversion fires on the capitalized surname (title-surname arm, unprotected) false The request is self-contained: everything needed is in the prompt or is general knowledge.
suppress/alias-name-question, full-name-self-contained, company-title-self-contained fires with an identity hit (protected) false Same self-contained requests; S6 may never suppress these (identity protection), so they test that the reducer leaves them alone.
keep/surname-needed fires on the capitalized surname (unprotected) true The answer needs the seeded page. These are the turns a wrong suppression would hurt, so suppression precision is measured against real risk.
negative/*: small talk, generic question, incidental lowercase name (pronunciation, character naming), other-referent name ("Captain X" in a novel, "X Street"), unseeded name stays silent false Nothing stored is needed to answer.

Reflex replay (actual counts). Every one of the 528 extra turns came out on the designed reflex slice and protection flag (0 mismatches against _design.json).

Dataset fired, needed, protected fired, needed, unprotected fired, not needed, protected fired, not needed, unprotected (suppress opportunity) silent, needed (fire opportunity) silent, not needed
BrainBench 57 5 0 0 0 87
Extra 0 66 44 121 165 132
Combined 57 71 44 121 165 219

The shipped reflex resolves every BrainBench know-to-ask turn (its baseline failure rate is 0), so BrainBench alone offers S6 no fire and no suppress opportunity; the extra corpus supplies all of them. Eval half of the extra corpus: 62 unprotected suppress opportunities (at least 35 of 35 correct are needed for action_precision_lb ≥ 0.90), 21 protected, 33 fired-and-needed unprotected turns, 82 fire opportunities, 66 silent negatives.

S9 conflict (hot-memory contradiction)

Files. s9-conflict/pairs.jsonl (the facts-fixtures input, with baselines), s9-conflict/pairs.sweep-eligible.jsonl (the pairs at cosine ≥ 0.80), and the builder outputs s9-conflict.jsonl (all pairs) and s9-conflict.sweep-eligible.jsonl.

Source. The repository holds few labelled fact pairs: the hot-memory classifier tests use placeholder texts ('x', 'new'), and the suspected-contradictions probe set (test/fixtures/contradictions-mini.jsonl) is five queries, not pairs. Collected as upstream-gold (6 pairs, source_ref names the test): the three labelled facts-fixtures cases and the proposal accept/undo pair in test/decide/conflict.test.ts, the MRR contradiction in test/eval-contradictions-judge.test.ts, and the temporal take supersession in test/eval-contradictions-auto-supersession.test.ts (verdicts mapped to supersede as the tests resolve them). The five probe queries seed 13 placeholder pairs (case: probe:*). The rest is synthetic-construction from bun docs/eval/system-one/generators/conflict-pairs.ts (Mulberry32, seed 90909): 12 attributes over invented people and companies (role, city, employer, drink, contact channel, one-on-one cadence, headcount, customers, stage, pricing, database, headquarters).

  • duplicate: the candidate restates the new fact's value (restatement, paraphrase-low-overlap, and restates-updated-value: an older memory that already holds the value an update states).
  • supersede: the new fact updates the candidate's value with explicit newer framing (value-update-near: same wording plus "as of / since / starting 2026"; value-update: narrative "promoted / moved / switched from … to …"; negation-update: "no longer"). Ordered attributes only move forward.
  • independent: same entity, a different or compatible attribute (compatible-same-entity, other-attribute, and near-independent: same sentence shape and shared tokens, a different attribute or referent, e.g. "has N employees" vs "has M paying customers").

Family. Pairs sharing a new fact share a family (the production request: the new fact is the state, one choice question per candidate); 408 families, 1 to 3 candidates each.

Counts. 786 pairs: duplicate 159, supersede 179, independent 448. slice is the choice label on purpose: the facts-fixtures builder keeps only a duplicate boolean as label, and slice is where the three-way label survives (hard-case tags are in case).

Baseline (today's rule). generators/conflict-baseline.ts embeds every distinct text with OpenAI text-embedding-3-large at 1536 dimensions and stores per pair baseline_cosine, baseline_decision (the decideSingleFact rule applied within the family: exact normalized match → duplicate; otherwise only the family's highest-cosine candidate can match, and at cosine ≥ 0.95 it supersedes when the kinds agree, else duplicate; everything else is inserted, independent), baseline_model and sweep_eligible (cosine ≥ 0.80, the S9 sweep's candidate floor). Vectors are cached locally, not committed. Results:

  • The cosine rule marks no true supersede (0 of 179; the highest supersede cosine is 0.926), supersedes 12 duplicates of the same kind (cosine ≥ 0.95) and inserts everything else.
  • Median cosine: duplicate 0.818, supersede 0.763, independent 0.581. Only 172 pairs clear the sweep floor (supersede 59, duplicate 96, independent 17); a supersede whose new value is an unseen name or number often sits below 0.80 with this embedding model, so the floor, not the classifier, bounds what S9 can find. s9-conflict.sweep-eligible.jsonl is the set production would actually ask about (eval half: supersede 30, duplicate 49, independent 9).

S8 grounding (claim support)

Files. s8-grounding/labels.jsonl (the grounding-labels input with provenance), s8-grounding/transcripts/ (the 16 Cat 35 transcripts it references) and the builder output s8-grounding.jsonl.

Source. Cat 35 transcript-distill corpus in github.com/garrytan/gbrain-evals at 88d0b1997e4b7fb3bd61e04d6eb637ef166ea6d4 (MIT; a fictional, model-generated corpus with invented people and companies): dream pages docs/benchmarks/2026-08-16-brainbench-cat35-transcript-distill/artifacts/*.dream.md against eval/data/transcript-distill-v1/transcripts-txt/*.txt. The transcripts are copied unchanged so transcript_path resolves from the repository root (transcripts/SOURCE.md). Eight of the 24 dream artifacts are empty (routine transcripts produce no page), so 16 pages contribute.

Units. verifyBody (src/core/cycle/synthesize-verify.ts) against the page's transcript; groundingUnits are exactly the units S8 judges in production (they pass the mechanical checks only because they carry no quote, number or attribution): 567 units.

Labels. llm:claude-sonnet-5, one call per page with the transcript and all its units, strict rubric matching the production question (supported only when the transcript states or directly implies the claim; added reasons, motives, generalizations or evaluations are unsupported; when unsure, unsupported). The judge's one-line reason is kept as judge_reason. The model's temperature is not settable, so a rerun without the local response cache can differ.

Perturbation negatives. Per page, up to 12 evenly spaced units the judge found supported were rewritten by the same model into fluent claims the transcript does not support (one key element changed; no numbers, quotes or speaker attribution). A rewrite is kept only when it still passes the mechanical checks as a grounding unit and a separate blind judge call labels it unsupported: 183 kept. They carry origin: perturbation, source_unit (the unit rewritten) and label_source: llm:claude-sonnet-5 (model-written and model-judged, not hand-labelled).

Counts. 750 items: dream-page supported 320, dream-page unsupported 247, perturbation unsupported 183.

Family. The page (cat35/<id>): 16 families, so the frozen split is coarse: 5 pages / 254 items calibrate, 11 pages / 496 items eval. Family-level qualification over 16 independent families cannot reach insufficient_n's minimum, so S8 is expected to stay shadow-only with this data, as the plan predicts.

Builder. gbrain decide dataset --slot grounding --from grounding-labels docs/eval/system-one/datasets/s8-grounding/labels.jsonl run from the repository root on a brain initialised with --no-embedding: source windows were selected by normalized substring and keyword neighbours only (the output is byte-identical with the embedding keys unset). slice is the coverage the builder found (adequate 571, weak 179; weak items are protected and can only become insufficient_context).

Generator command.

bun docs/eval/system-one/generators/grounding-labels.ts ~/.capy/work/gbrain-evals
gbrain decide dataset --slot grounding --from grounding-labels docs/eval/system-one/datasets/s8-grounding/labels.jsonl --out s8.raw.jsonl
bun docs/eval/system-one/generators/label-source.ts s8.raw.jsonl docs/eval/system-one/datasets/s8-grounding.jsonl llm:claude-sonnet-5 docs/eval/system-one/datasets/s8-grounding/labels.jsonl

S7 triage (dream triage)

s7-triage.jsonl is built with gbrain decide dataset --slot triage --from cat35 docs/eval/system-one/datasets/s7-triage. The directory holds 254 transcripts in the Cat 35 layout (gold/<id>.json + transcripts-txt/<date>-<id>.txt):

  • 24 Cat 35 transcripts copied unchanged from garrytan/gbrain-evals eval/data/transcript-distill-v1 at 88d0b1997e4b7fb3bd61e04d6eb637ef166ea6d4 (MIT; itself a synthetic corpus). Label: the corpus's own expected_triage (label_source: upstream-gold).
  • 230 synthetic transcripts from docs/eval/system-one/generators/s7-synthetic-transcripts.ts (openai:gpt-5.6-luna, reasoning none, temperature 1, seeded spec list): 180 entirely routine (label false) and 50 mostly routine with one buried synthesis-worthy passage (label true). Labels are the generator's instructions (label_source: synthetic-construction); nobody read and labelled them. A name check found no planted person or project name in any routine transcript.

Family = transcript. Eval half: 109 transcripts (79 negative, 30 positive).

S2 intent (query routing)

s2-intent.jsonl = gbrain decide dataset --slot intent --from longmemeval longmemeval_s_cleaned.json (470 non-abstention questions, search and think call sites, labels from question_type) plus --from brainbench test/fixtures/retrieval-quality/relational/relational.jsonl (38 relational queries). Labels are the corpora's own annotations mapped by the builder (upstream-gold). The search label set this produces is only temporal and general, so it measures a slice of the search question.

S5 injection

s5-injection.jsonl = gbrain decide dataset --slot injection --from injection-fixtures test/fixtures/decide/injection-cases.jsonl: 12 cases, 4 with an attack candidate (upstream-gold). Too small to calibrate or qualify on; used for a detection check only.

S3 evidence and S4 answerable (not committed: size)

Both are regenerated from LongMemEval-S cleaned (MIT, sha256 d6f21ea9d60a0d56…):

  • S3: gbrain decide dataset --slot evidence --from longmemeval longmemeval_s_cleaned.json → 10,000 items in 500 families (question), 20 sessions per family, label = the session is an answer session; split_hash 036999bf920b7039 (57 MB). Candidate text is the session capped at 6,000 characters, the same cap capRerankDoc applies to production candidates.
  • S4: gbrain decide dataset --slot answerable --from longmemeval longmemeval_s_cleaned.json → 500 items, label false for the 30 _abs questions; split_hash c2d7aea440b81e88.

The LongMemEval question lists the retrieval arms ran on are in longmemeval/: s-eval-half.txt (the 248 eval-half families of the S3 split), s-eval-judged-100.txt (85 answerable by hash order + all 15 abstention questions of the eval half) and m-pilot-28.txt (the frozen gbrain-evals LongMemEval-M pilot selection).

Spend

Paid calls are logged one line per request batch in ../ledger-datasets.jsonl (tokens from the provider's reported usage, priced at list: Claude Sonnet 5 $3 / $15 per million input / output tokens, OpenAI text-embedding-3-large $0.13 per million). Total for these datasets: $2.52 (S8 labelling and perturbation $2.51, S9 embeddings $0.004 including superseded iterations, two model-id probes $0.0002), under the $3 cap. The S8 generator refuses to start a call that would take the ledger past $3.

Regenerating hashes

bun docs/eval/system-one/generators/hashes.ts rewrites HASHES.md.