Skip to content

BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.

chore(benchmarks): update REPORTS ledger and multi-suite docs

Christian0b6f31cdbb73
diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md
index 89c62fa..f5bcf0e 100644
--- a/benchmarks/AGENTS.md
+++ b/benchmarks/AGENTS.md
@@ -1,21 +1,47 @@
# Benchmarks — agent guidance
-## LoCoMo harness
+## Suites
-The LoCoMo harness lives in `benchmarks/locomo/`. It talks to a running BrainAPI over HTTP only (no `src/` imports). Typical flow: `download` → `ingest` → `evaluate` → `report` via `./locomo.sh` or `.venv/bin/python -m locomo`.
+| Suite | Package | Wrapper | Ledger key |
+| --- | --- | --- | --- |
+| **LoCoMo** | `benchmarks/locomo/` | `./locomo.sh` | `benchmarks.locomo` |
+| **LongMemEval** | `benchmarks/longmemeval/` | `./longmemeval.sh` | `benchmarks.longmemeval` |
+| **BEAM** | `benchmarks/beam/` | `./beam.sh` | `benchmarks.beam` |
+
+All harnesses talk to a running BrainAPI over HTTP only (no `src/` imports).
+
+### LoCoMo
+
+Typical flow: `download` → `ingest` → `evaluate` → `report` via `./locomo.sh` or `.venv/bin/python -m locomo`.
+
+### LongMemEval
+
+Typical flow: `download` → `ingest` → `evaluate` → `report` via `./longmemeval.sh` or `.venv/bin/python -m longmemeval`. Protocol: `docs/research/09-longmemeval-protocol.md`.
+
+### BEAM
+
+Typical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh` or `.venv/bin/python -m beam`.
+
+- Dataset: HuggingFace `Mohammadta/BEAM` splits `100K` / `500K` / `1M` (BEAM-10M deferred).
+- Ingest turns only — never probing questions, rubrics, or ideal answers.
+- Scoring: rubric LLM-judge (`judge_prompt_variant: beam-rubric-v1-question-aware`); headline = mean of 10 ability means; `event_ordering` uses `tau_norm`.
+- Brain IDs: `beam{size}{convid}` → e.g. `beam100k1`.
+- Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md).
+- Ingest parallelism: prefer `--concurrency 2` (≤4); resume skips permanent embed-8192 fails. See protocol “Parallel ingest” + `scripts/run_beam_1m_1_ingest_eval.sh` (`BEAM_INGEST_CONCURRENCY`). Do not wipe live eval brains.
## Results ledger (`REPORTS.json`)
`benchmarks/REPORTS.json` is the public BrainAPI results ledger: top published scores for **all** benchmark suites under `benchmarks.<suite>`.
- LoCoMo harness upserts `benchmarks.locomo` via `write_report` → `update_reports_json` when report `status` is `ok`.
+- BEAM harness upserts `benchmarks.beam` the same way (field: `headline_score`, continuous `[0,1]`).
- After any completed evaluate/report that produces a successful scored run, ensure the suite entry in `REPORTS.json` reflects it.
- If you write or patch `runs/<id>/report.json` manually, update `REPORTS.json` yourself (same schema; upsert by `run_id` under the suite).
- When adding a new suite, add a `benchmarks.<suite_id>` key (`name` + `leaderboard`) and upsert the same way.
- Skip failed/junk runs (`status: failed`, 0% judge, nonsense scores) — they do not belong on the leaderboard.
- Research caveats and checkpoint narrative live in `runs/CHECKPOINT_NOTES.md` (local), not in `REPORTS.json`.
-## Tracks
+## Tracks (LoCoMo)
| Track | Profile | Intent |
| --- | --- | --- |
@@ -30,6 +56,6 @@ TUI / `brainapi start` may run from `~/.brainapi/source`, not this workspace. Sy
## Conventions
-- Brain IDs are alphanumeric (`conv-26` → `locomoconv26`). Prefer clean brains (`locomoconv26clean`) for product/SOTA claims after Phase 0.
-- Compare judge accuracy with `./locomo.sh compare` (paired McNemar). Do not A/B graph EvR until identical-config session-set agreement clears the ≥95% gate.
-- Do not ingest QA annotations. Keep answer prompts prompt-audit clean (`./locomo.sh prompt-audit`).
+- Brain IDs are alphanumeric (`conv-26` → `locomoconv26`; BEAM `100K/1` → `beam100k1`). Prefer clean brains for product/SOTA claims after Phase 0.
+- Compare LoCoMo judge accuracy with `./locomo.sh compare` (paired McNemar). Do not A/B graph EvR until identical-config session-set agreement clears the ≥95% gate.
+- Do not ingest QA annotations / probing questions. Keep answer prompts prompt-audit clean where the suite provides it (`./locomo.sh prompt-audit`).
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 4d9a950..43a2fad 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -1,3 +1,16 @@
+# Benchmarks for BrainAPI
+
+Standalone HTTP-only harnesses (no `src/` imports):
+
+| Suite | Package | Wrapper |
+| --- | --- | --- |
+| [LoCoMo](https://github.com/snap-research/LoCoMo) | `locomo/` | `./locomo.sh` |
+| [LongMemEval](https://github.com/xiaowu0162/LongMemEval) | `longmemeval/` | `./longmemeval.sh` |
+
+Shared setup: `requirements.txt`, `.env` (`BRAINPAT_TOKEN`, LLM keys). Results ledger: [`REPORTS.json`](REPORTS.json). Agent notes: [`AGENTS.md`](AGENTS.md).
+
+---
+
# LoCoMo benchmark for BrainAPI
Standalone harness that:
@@ -204,15 +217,104 @@ Create a sibling package under `benchmarks/`:
```text
benchmarks/
- locomo/ # this package
+ locomo/ # LoCoMo package
+ longmemeval/ # LongMemEval package
yourbench/ # new package with its own cli / client / metrics
requirements.txt # shared deps, or a per-bench requirements file
```
-Reuse patterns from `locomo/`:
+Reuse patterns from `locomo/` / `longmemeval/`:
- HTTP-only `BrainAPIClient`
- JSONL resume files under `runs/`
- Separate ingest → evaluate → report stages
+- Upsert into `REPORTS.json` under `benchmarks.<suite_id>`
Keep each benchmark self-contained and free of imports from BrainAPI's `src/`.
+
+---
+
+# LongMemEval benchmark for BrainAPI
+
+Sibling harness for [LongMemEval](https://github.com/xiaowu0162/LongMemEval) (cleaned HF release).
+
+1. Downloads cleaned LongMemEval_S (default), oracle, or M
+2. Ingests each question's haystack into its own brain (`lme…`)
+3. Answers via `POST /retrieve/context` + LLM (includes `question_date`)
+4. Scores with official LongMemEval yes/no judge prompts by `question_type`
+
+Protocol: [`docs/research/09-longmemeval-protocol.md`](../docs/research/09-longmemeval-protocol.md).
+
+## Quickstart
+
+```bash
+./longmemeval.sh download --variant s
+./longmemeval.sh dataset-stats --variant s
+./longmemeval.sh smoke --limit 1
+./longmemeval.sh ingest --run lme-s-smoke --limit 3
+./longmemeval.sh evaluate --run lme-s-smoke --limit 3
+./longmemeval.sh report --run lme-s-smoke
+./longmemeval.sh selftest-metrics
+./longmemeval.sh prompt-audit
+```
+
+### Useful flags
+
+- `--variant s|oracle|m`
+- `--question-id <id>` (repeatable)
+- `--limit N` / `--limit-sessions N`
+- `--concurrency N`
+- `--dry-run` / `--no-resume`
+
+### Scoring
+
+- **Headline**: LLM yes/no accuracy over all questions (including abstention).
+- Also: per-`question_type` accuracy, abstention accuracy, session recall vs `answer_session_ids` (abstention skipped), retrieval latency, LLM tokens.
+- Successful runs with ≥50 scored questions upsert `benchmarks.longmemeval` in `REPORTS.json`.
+
+Do **not** ingest gold answers or `has_answer` turn labels.
+
+---
+
+# BEAM benchmark for BrainAPI
+
+Sibling harness for [BEAM](https://github.com/mohammadtavakoli78/BEAM) (ICLR 2026). Evaluates BrainAPI as long-term memory on multi-scale chats (`100K` / `500K` / `1M`). BEAM-10M is deferred.
+
+1. Downloads HuggingFace `Mohammadta/BEAM` and normalizes under `data/beam/`
+2. Ingests each batch turn into a brain (`beam100k1`, …)
+3. Answers probing questions via `POST /retrieve/context` + LLM
+4. Scores with BEAM rubric LLM-judge (10 abilities); headline = mean of ability means
+
+Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md).
+
+## Quickstart
+
+```bash
+./beam.sh download --size 100K
+./beam.sh dataset-stats --size 100K
+./beam.sh smoke --size 100K --sample 1
+./beam.sh ingest --size 100K --sample 1 --run beam-100k-1 --limit-turns 10
+./beam.sh evaluate --size 100K --sample 1 --run beam-100k-1 --limit 5
+./beam.sh report --run beam-100k-1
+./beam.sh selftest
+```
+
+### Useful flags
+
+- `--size 100K|500K|1M`
+- `--sample <id>` (repeatable; id or `size/id`)
+- `--limit-turns N` / `--limit N` / `--abilities a,b,…`
+- `--concurrency N` (ingest default 2; BEAM 1M prefer 2–3, ≤4; keep ≤ `CELERY_WORKER_CONCURRENCY`)
+- `--dry-run` / `--no-resume` (resume skips completed + permanent embed-8192 fails)
+
+### Scoring
+
+- **Headline**: mean of 10 ability means (`headline_score`).
+- `event_ordering` uses Kendall `tau_norm`; others use mean rubric `llm_judge_score`.
+- Judge variant: `beam-rubric-v1-question-aware` (question-aware, float scores).
+- Successful runs with ≥20 scored questions upsert `benchmarks.beam` in `REPORTS.json`.
+
+Do **not** ingest probing questions, rubrics, or ideal answers.
+
+**Cost warning:** even one 100K chat is ~100+ turns; start with `--limit-turns` / `--limit`.
+**1M wall time:** ~40h at concurrency 1; ~20–25h at 2 with healthy API (see `docs/research/10-beam-protocol.md`).
diff --git a/benchmarks/REPORTS.json b/benchmarks/REPORTS.json
index d5d79f0..49ede06 100644
--- a/benchmarks/REPORTS.json
+++ b/benchmarks/REPORTS.json
@@ -1,11 +1,182 @@
{
"schema_version": 2,
- "updated_at": "2026-07-29T18:51:58.401379+00:00",
+ "updated_at": "2026-08-03T09:15:33.804119+00:00",
"description": "BrainAPI benchmark results. Top published scores across suites. Updated when a suite evaluate/report completes successfully.",
"benchmarks": {
"locomo": {
"name": "LoCoMo",
"leaderboard": [
+ {
+ "run_id": "locomo-compose-sota-conv26-v4d",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.9539473684210527,
+ "headline_judge_accuracy_pct": 95.39,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26-v4d/report.json",
+ "recorded_at": "2026-07-31T13:14:01.363020+00:00"
+ },
+ {
+ "run_id": "locomo-compose-sota-conv26-v4c",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.9276315789473685,
+ "headline_judge_accuracy_pct": 92.76,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26-v4c/report.json",
+ "recorded_at": "2026-07-31T13:07:26.775227+00:00"
+ },
+ {
+ "run_id": "locomo-compose-sota-conv26-v2",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.9210526315789473,
+ "headline_judge_accuracy_pct": 92.11,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26-v2/report.json",
+ "recorded_at": "2026-07-30T20:01:08.915975+00:00"
+ },
+ {
+ "run_id": "locomo-compose-sota-conv26-v4",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.9210526315789473,
+ "headline_judge_accuracy_pct": 92.11,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26-v4/report.json",
+ "recorded_at": "2026-07-31T11:28:19.452764+00:00"
+ },
+ {
+ "run_id": "locomo-compose-sota-conv26-v3",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.9013157894736842,
+ "headline_judge_accuracy_pct": 90.13,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26-v3/report.json",
+ "recorded_at": "2026-07-31T10:29:50.328215+00:00"
+ },
+ {
+ "run_id": "locomo-conv26-beamreg-product",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.9013157894736842,
+ "headline_judge_accuracy_pct": 90.13,
+ "answerable_rate": 0.9473684210526315,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-conv26-beamreg-product/report.json",
+ "recorded_at": "2026-08-01T01:58:34.693097+00:00"
+ },
+ {
+ "run_id": "locomo-compose-sota-conv26",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8947368421052632,
+ "headline_judge_accuracy_pct": 89.47,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26/report.json",
+ "recorded_at": "2026-07-30T19:18:56.740730+00:00"
+ },
+ {
+ "run_id": "locomo-compose-sota-conv26-v4b",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.875,
+ "headline_judge_accuracy_pct": 87.5,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-compose-sota-conv26-v4b/report.json",
+ "recorded_at": "2026-07-31T12:26:52.469668+00:00"
+ },
+ {
+ "run_id": "locomo-cat3fix-sota-conv26",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.868421052631579,
+ "headline_judge_accuracy_pct": 86.84,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-cat3fix-sota-conv26/report.json",
+ "recorded_at": "2026-07-30T18:26:15.214852+00:00"
+ },
{
"run_id": "oldbrain-recheck-a",
"track": "product",
@@ -25,6 +196,25 @@
"report_path": "runs/oldbrain-recheck-a/report.json",
"recorded_at": "2026-07-28T22:05:39.768798+00:00"
},
+ {
+ "run_id": "locomo-batch-nostorm-cfgb",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8552631578947368,
+ "headline_judge_accuracy_pct": 85.53,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-batch-nostorm-cfgb/report.json",
+ "recorded_at": "2026-07-30T17:03:35.449258+00:00"
+ },
{
"run_id": "phase-sota-d1-conv26",
"track": "sota",
@@ -44,6 +234,44 @@
"report_path": "runs/phase-sota-d1-conv26/report.json",
"recorded_at": "2026-07-29T18:51:58.400341+00:00"
},
+ {
+ "run_id": "locomo-nostorm-sota-conv26",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8486842105263158,
+ "headline_judge_accuracy_pct": 84.87,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26nostorm",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-nostorm-sota-conv26/report.json",
+ "recorded_at": "2026-07-30T17:41:32.563092+00:00"
+ },
+ {
+ "run_id": "locomo-batch-cfgb",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8421052631578947,
+ "headline_judge_accuracy_pct": 84.21,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26batch",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-batch-cfgb/report.json",
+ "recorded_at": "2026-07-30T15:23:27.730891+00:00"
+ },
{
"run_id": "clean-brain-eval-a",
"track": "product",
@@ -139,6 +367,34 @@
"report_path": "runs/baseline-clean-a/report.json",
"recorded_at": "2026-07-28T18:18:19.581660+00:00"
},
+ {
+ "run_id": "sota-locomo10-batch-compose",
+ "track": "sota",
+ "scope": "full-locomo10",
+ "samples": [
+ "conv-26",
+ "conv-30",
+ "conv-41",
+ "conv-42",
+ "conv-43",
+ "conv-44",
+ "conv-47",
+ "conv-48",
+ "conv-49",
+ "conv-50"
+ ],
+ "n_non_adversarial": 1540,
+ "headline_judge_accuracy": 0.8097402597402598,
+ "headline_judge_accuracy_pct": 80.97,
+ "answerable_rate": 0.9363636363636364,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomof10c*",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/sota-locomo10-batch-compose/report.json",
+ "recorded_at": "2026-07-31T13:35:58.561407+00:00"
+ },
{
"run_id": "phase-d1-paths-a",
"track": "product",
@@ -157,6 +413,375 @@
"git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
"report_path": "runs/phase-d1-paths-a/report.json",
"recorded_at": "2026-07-29T16:26:21.919353+00:00"
+ },
+ {
+ "run_id": "locomo-batch-score",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8026315789473685,
+ "headline_judge_accuracy_pct": 80.26,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26batch",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/locomo-batch-score/report.json",
+ "recorded_at": "2026-07-30T14:59:39.743019+00:00"
+ }
+ ]
+ },
+ "longmemeval": {
+ "name": "LongMemEval",
+ "leaderboard": []
+ },
+ "beam": {
+ "name": "BEAM",
+ "leaderboard": [
+ {
+ "run_id": "beam-1m-1-product-v3c-compose",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "1M/1",
+ "size": "1M",
+ "samples": [
+ "1M/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.7897396010333467,
+ "headline_score_pct": 78.97,
+ "per_ability": {
+ "abstention": 0.5,
+ "contradiction_resolution": 0.6875,
+ "event_ordering": 0.42170156588902286,
+ "information_extraction": 1.0,
+ "instruction_following": 1.0,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.9375,
+ "preference_following": 0.875,
+ "summarization": 0.4756944444444444,
+ "temporal_reasoning": 1.0
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam1m1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-1m-1-product-v3c-compose/report.json",
+ "recorded_at": "2026-08-03T09:15:33.803818+00:00"
+ },
+ {
+ "run_id": "beam-100k-1-clean-ordv1",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.7887500000000001,
+ "headline_score_pct": 78.88,
+ "per_ability": {
+ "abstention": 1.0,
+ "contradiction_resolution": 0.625,
+ "event_ordering": 0.7875,
+ "information_extraction": 1.0,
+ "instruction_following": 0.75,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.625,
+ "preference_following": 1.0,
+ "summarization": 0.6000000000000001,
+ "temporal_reasoning": 0.5
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1-clean-ordv1/report.json",
+ "recorded_at": "2026-08-01T02:00:50.310070+00:00"
+ },
+ {
+ "run_id": "beam-100k-1-clean-product",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.7398223304703363,
+ "headline_score_pct": 73.98,
+ "per_ability": {
+ "abstention": 0.5,
+ "contradiction_resolution": 0.6875,
+ "event_ordering": 0.1357233047033631,
+ "information_extraction": 0.75,
+ "instruction_following": 1.0,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.875,
+ "preference_following": 1.0,
+ "summarization": 0.7,
+ "temporal_reasoning": 0.75
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1-clean-product/report.json",
+ "recorded_at": "2026-08-01T02:00:51.059632+00:00"
+ },
+ {
+ "run_id": "beam-1m-1-product-v2",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "1M/1",
+ "size": "1M",
+ "samples": [
+ "1M/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.7014047619826247,
+ "headline_score_pct": 70.14,
+ "per_ability": {
+ "abstention": 0.5,
+ "contradiction_resolution": 0.4375,
+ "event_ordering": 0.2883531753818027,
+ "information_extraction": 1.0,
+ "instruction_following": 1.0,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.625,
+ "preference_following": 0.75,
+ "summarization": 0.4131944444444444,
+ "temporal_reasoning": 1.0
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam1m1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-1m-1-product-v2/report.json",
+ "recorded_at": "2026-08-03T07:58:27.006989+00:00"
+ },
+ {
+ "run_id": "beam-1m-1-product-v3b",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "1M/1",
+ "size": "1M",
+ "samples": [
+ "1M/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.6497747205349994,
+ "headline_score_pct": 64.98,
+ "per_ability": {
+ "abstention": 0.5,
+ "contradiction_resolution": 0.5625,
+ "event_ordering": 0.4213583164611052,
+ "information_extraction": 1.0,
+ "instruction_following": 1.0,
+ "knowledge_update": 0.5,
+ "multi_session_reasoning": 0.75,
+ "preference_following": 0.875,
+ "summarization": 0.3888888888888889,
+ "temporal_reasoning": 0.5
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam1m1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-1m-1-product-v3b/report.json",
+ "recorded_at": "2026-08-03T09:14:47.079177+00:00"
+ },
+ {
+ "run_id": "beam-100k-1-sota",
+ "track": "sota",
+ "bench_profile": "sota",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.6483928571428572,
+ "headline_score_pct": 64.84,
+ "per_ability": {
+ "abstention": 0.0,
+ "contradiction_resolution": 0.625,
+ "event_ordering": 0.1339285714285714,
+ "information_extraction": 0.75,
+ "instruction_following": 0.75,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.875,
+ "preference_following": 1.0,
+ "summarization": 0.6,
+ "temporal_reasoning": 0.75
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1-sota/report.json",
+ "recorded_at": "2026-08-01T02:01:11.568901+00:00"
+ },
+ {
+ "run_id": "beam-100k-1-clean-sota",
+ "track": "sota",
+ "bench_profile": "sota",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.6245833333333334,
+ "headline_score_pct": 62.46,
+ "per_ability": {
+ "abstention": 0.0,
+ "contradiction_resolution": 0.875,
+ "event_ordering": 0.1458333333333333,
+ "information_extraction": 0.5,
+ "instruction_following": 0.75,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.625,
+ "preference_following": 1.0,
+ "summarization": 0.6000000000000001,
+ "temporal_reasoning": 0.75
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1-clean-sota/report.json",
+ "recorded_at": "2026-08-01T02:01:12.316230+00:00"
+ },
+ {
+ "run_id": "beam-100k-1-product-prompts",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.6042261904761904,
+ "headline_score_pct": 60.42,
+ "per_ability": {
+ "abstention": 0.0,
+ "contradiction_resolution": 0.5,
+ "event_ordering": 0.1339285714285714,
+ "information_extraction": 0.9583333333333333,
+ "instruction_following": 0.75,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.75,
+ "preference_following": 0.75,
+ "summarization": 0.7,
+ "temporal_reasoning": 0.5
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1-product-prompts/report.json",
+ "recorded_at": "2026-08-01T02:01:13.074254+00:00"
+ },
+ {
+ "run_id": "beam-100k-1-configb",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.559333376188726,
+ "headline_score_pct": 55.93,
+ "per_ability": {
+ "abstention": 0.25,
+ "contradiction_resolution": 0.5,
+ "event_ordering": 0.14333376188726035,
+ "information_extraction": 0.5,
+ "instruction_following": 0.75,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.625,
+ "preference_following": 0.75,
+ "summarization": 0.7,
+ "temporal_reasoning": 0.375
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1-configb/report.json",
+ "recorded_at": "2026-08-01T02:01:13.832770+00:00"
+ },
+ {
+ "run_id": "beam-100k-1",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "100K/1",
+ "size": "100K",
+ "samples": [
+ "100K/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.5558928571428572,
+ "headline_score_pct": 55.59,
+ "per_ability": {
+ "abstention": 0.5,
+ "contradiction_resolution": 0.5,
+ "event_ordering": 0.1339285714285714,
+ "information_extraction": 0.5,
+ "instruction_following": 0.5,
+ "knowledge_update": 1.0,
+ "multi_session_reasoning": 0.625,
+ "preference_following": 0.75,
+ "summarization": 0.55,
+ "temporal_reasoning": 0.5
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam100k1",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-100k-1/report.json",
+ "recorded_at": "2026-08-01T02:01:14.597463+00:00"
+ },
+ {
+ "run_id": "beam-1m-1-clean-product",
+ "track": "product",
+ "bench_profile": "product",
+ "scope": "1M/1",
+ "size": "1M",
+ "samples": [
+ "1M/1"
+ ],
+ "n_questions": 20,
+ "headline_score": 0.5520385817869101,
+ "headline_score_pct": 55.2,
+ "per_ability": {
+ "abstention": 0.5,
+ "contradiction_resolution": 0.375,
+ "event_ordering": 0.35024692898021254,
+ "information_extraction": 1.0,
+ "instruction_following": 0.5,
+ "knowledge_update": 0.5,
+ "multi_session_reasoning": 0.625,
+ "preference_following": 0.75,
+ "summarization": 0.4201388888888889,
+ "temporal_reasoning": 0.5
+ },
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "brain": "beam1m1clean",
+ "git_sha": "78cbbae50ac37777ce0315f85156f7812b28248c",
+ "report_path": "runs/beam-1m-1-clean-product/report.json",
+ "recorded_at": "2026-08-03T06:59:14.016975+00:00"
}
]
}
diff --git a/benchmarks/requirements.txt b/benchmarks/requirements.txt
index fccaa95..40855cb 100644
--- a/benchmarks/requirements.txt
+++ b/benchmarks/requirements.txt
@@ -2,3 +2,6 @@ httpx>=0.27.0,<1.0.0
openai>=1.40.0,<2.0.0
python-dotenv>=1.0.0,<2.0.0
rich>=13.0.0,<15.0.0
+datasets>=2.19.0,<4.0.0
+scipy>=1.11.0,<2.0.0
+json-repair>=0.30.0,<1.0.0