Skip to content

BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.

feat(benchmarks): SOTA LoCoMo harness and multi-suite REPORTS ledger

Christian785a507524b7
diff --git a/AGENTS.md b/AGENTS.md
index e69de29..69adc92 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -0,0 +1,5 @@
+# Agent guidance
+
+Project agent guidance for benchmarks: see [`benchmarks/AGENTS.md`](benchmarks/AGENTS.md).
+
+Benchmark results ledger (all suites): [`benchmarks/REPORTS.json`](benchmarks/REPORTS.json).
diff --git a/benchmarks/.env.example b/benchmarks/.env.example
index 24405da..8d0c17e 100644
--- a/benchmarks/.env.example
+++ b/benchmarks/.env.example
@@ -1,14 +1,34 @@
BRAINAPI_URL=http://localhost:8000
BRAINPAT_TOKEN=
+# Dual profile: product (greedy) | sota (deepseek-v4-flash + SC + gap-fill)
+# BENCH_PROFILE=product
+# BENCH_PROFILE=sota
+# BENCH_SC_SAMPLES=5
+# BENCH_SC_TEMPERATURE=0.7
+# BENCH_GAP_FILL=1
+# Optional overrides (default SOTA answerer is deepseek-v4-flash):
+# BENCH_SOTA_ANSWER_MODEL=deepseek-v4-flash
+# BENCH_SOTA_JUDGE_MODEL=deepseek-v4-flash
+
# DeepSeek (OpenAI-compatible). Get a key at https://platform.deepseek.com/
DEEPSEEK_API_KEY=
BENCH_LLM_BASE_URL=https://api.deepseek.com
BENCH_ANSWER_MODEL=deepseek-v4-flash
-BENCH_JUDGE_MODEL=deepseek-v4-flash
+
+# The judge must be a different model family from the answerer, or judge accuracy
+# carries self-preference bias. Leave BENCH_JUDGE_MODEL unset to auto-select the
+# Azure deployment below; set it explicitly to pin a judge.
+# BENCH_JUDGE_MODEL=
+# BENCH_JUDGE_BASE_URL=
+# BENCH_JUDGE_API_KEY=
+
+# Azure OpenAI judge (falls back to the repo-root AZURE_LARGE_LLM_* variables):
+# BENCH_JUDGE_AZURE_ENDPOINT=
+# BENCH_JUDGE_AZURE_KEY=
+# BENCH_JUDGE_AZURE_API_VERSION=2024-12-01-preview
# Optional aliases if you prefer another OpenAI-compatible provider:
# OPENAI_API_KEY=
# BENCH_LLM_BASE_URL=https://api.openai.com/v1
# BENCH_ANSWER_MODEL=gpt-4o-mini
-# BENCH_JUDGE_MODEL=gpt-4o
diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md
index e69de29..89c62fa 100644
--- a/benchmarks/AGENTS.md
+++ b/benchmarks/AGENTS.md
@@ -0,0 +1,35 @@
+# Benchmarks — agent guidance
+
+## LoCoMo harness
+
+The LoCoMo harness lives in `benchmarks/locomo/`. It talks to a running BrainAPI over HTTP only (no `src/` imports). Typical flow: `download` → `ingest` → `evaluate` → `report` via `./locomo.sh` or `.venv/bin/python -m locomo`.
+
+## Results ledger (`REPORTS.json`)
+
+`benchmarks/REPORTS.json` is the public BrainAPI results ledger: top published scores for **all** benchmark suites under `benchmarks.<suite>`.
+
+- LoCoMo harness upserts `benchmarks.locomo` via `write_report` → `update_reports_json` when report `status` is `ok`.
+- After any completed evaluate/report that produces a successful scored run, ensure the suite entry in `REPORTS.json` reflects it.
+- If you write or patch `runs/<id>/report.json` manually, update `REPORTS.json` yourself (same schema; upsert by `run_id` under the suite).
+- When adding a new suite, add a `benchmarks.<suite_id>` key (`name` + `leaderboard`) and upsert the same way.
+- Skip failed/junk runs (`status: failed`, 0% judge, nonsense scores) — they do not belong on the leaderboard.
+- Research caveats and checkpoint narrative live in `runs/CHECKPOINT_NOTES.md` (local), not in `REPORTS.json`.
+
+## Tracks
+
+| Track | Profile | Intent |
+| --- | --- | --- |
+| **SOTA** | `BENCH_PROFILE=sota` | Beat HyperMem-class LoCoMo (≥93% on full-10). Prefer **deepseek-v4-flash** answerer/judge. SC + gap-fill live in the harness, not product latency path. |
+| **Product** | `BENCH_PROFILE=product` (default) | ADR-006 sub-second `/retrieve/context` path — greedy, no online LLM loops. |
+
+Protocol and competitor bar: `docs/research/08-sota-locomo-protocol.md`. Historical checkpoint narrative: `runs/CHECKPOINT_NOTES.md`.
+
+## Live stack
+
+TUI / `brainapi start` may run from `~/.brainapi/source`, not this workspace. Sync or restart from the intended checkout before measuring, or scores will reflect stale code.
+
+## Conventions
+
+- Brain IDs are alphanumeric (`conv-26` → `locomoconv26`). Prefer clean brains (`locomoconv26clean`) for product/SOTA claims after Phase 0.
+- Compare judge accuracy with `./locomo.sh compare` (paired McNemar). Do not A/B graph EvR until identical-config session-set agreement clears the ≥95% gate.
+- Do not ingest QA annotations. Keep answer prompts prompt-audit clean (`./locomo.sh prompt-audit`).
diff --git a/benchmarks/README.md b/benchmarks/README.md
index a4c06e2..4d9a950 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -65,6 +65,10 @@ Reports land in `runs/<run_id>/`:
- `manifest.json` — run config
- `report.md` / `report.json` — aggregated metrics
+## Results ledger
+
+[`REPORTS.json`](REPORTS.json) is the shared BrainAPI benchmark ledger (multi-suite). LoCoMo writes the `locomo` entry; successful `evaluate` / `report` upserts automatically. See [`AGENTS.md`](AGENTS.md).
+
## Commands
| Command | Purpose |
@@ -74,9 +78,11 @@ Reports land in `runs/<run_id>/`:
| `smoke` | One-shot ingest + retrieve health check |
| `ingest` | Push sessions (or turns) into BrainAPI |
| `answer-once` | Retrieve + answer a single question |
-| `selftest-metrics` | Local F1 / BLEU-1 sanity checks |
+| `selftest-metrics` | Local metric, channel-split and record-construction checks |
+| `prompt-audit` | Fail if the answer prompt shares an n-gram with any gold answer |
| `evaluate` | Retrieve → answer → judge for QA items |
| `report` | Aggregate `answers.jsonl` into markdown/JSON |
+| `compare` | Paired exact-McNemar comparison between two run arms |
### Useful flags
@@ -99,19 +105,84 @@ Reports land in `runs/<run_id>/`:
- `--concurrency 2`
- `--run <id>`
+**compare**
+
+- `--baseline <run>` / `--candidate <run>`, both repeatable — repeat to pool several runs of the same configuration into one arm
+- `--include-adversarial`
+- `--json`
+
## Scoring
- **Headline number**: LLM-judge accuracy excluding category 5 (adversarial), matching common LoCoMo memory-system reporting.
- Also reported: overall accuracy, per-category breakdown, mean F1, mean BLEU-1, retrieval latency p50/p95, total LLM tokens.
- Categories: `1` multi-hop, `2` temporal, `3` open-domain, `4` single-hop, `5` adversarial.
+- Gold for adversarial questions is read from `adversarial_answer` when `answer` is absent. F1 and BLEU-1 are `null` (unscorable) when gold is empty rather than a perfect 1.0.
+- Evidence-session recall is reported for the **graph** and **passage** channels separately as well as combined, so the event graph's contribution is visible on its own.
+
+### Comparing two runs
+
+Judge accuracy is noisy: 11–12% of questions flip between identical-config runs at
+`temperature=0`. **Passage** retrieval metrics are deterministic for a fixed brain
+and config. **Graph** session sets are a measurement only after identical-config
+agreement clears the **≥95%** gate (see `compare` graph-session stability output).
+Do not A/B graph EvR below that gate.
+
+```bash
+./locomo.sh compare --baseline locomo-conv26-push75-b --candidate locomo-conv26-push75-c
+./locomo.sh compare \
+ --baseline locomo-conv26-push75-a --baseline locomo-conv26-push75-b \
+ --candidate locomo-conv26-push75-c --candidate locomo-conv26-push75-d
+
+# Checkpoint A — identical-config graph stability (gate ≥95% session-set agreement)
+./locomo.sh evaluate --sample conv-26 --brain locomoconv26clean \
+ --run graph-stable-a --historical-limit 16 --max-passages 16 \
+ --max-facts 50 --use-ppr --no-sufficiency-retry
+./locomo.sh evaluate --sample conv-26 --brain locomoconv26clean \
+ --run graph-stable-b --historical-limit 16 --max-passages 16 \
+ --max-facts 50 --use-ppr --no-sufficiency-retry
+./locomo.sh compare --baseline graph-stable-a --candidate graph-stable-b
+```
+
+This reports the exact McNemar p-value with the flip table (how many questions moved
+to correct and how many to wrong), overall and per category. Two independent Wilson
+intervals are not a substitute: they discard the pairing and most of the power.
+When both arms are single runs, `compare` also prints graph-session set agreement
+vs the 95% gate.
-Defaults use **DeepSeek** (override in `.env`):
+### Models
+
+Defaults use **DeepSeek** for the answerer (override in `.env`):
- Base URL: `https://api.deepseek.com` (`BENCH_LLM_BASE_URL`)
-- Answerer + judge: `deepseek-v4-flash` (`BENCH_ANSWER_MODEL` / `BENCH_JUDGE_MODEL`)
+- Answerer: `deepseek-v4-flash` (`BENCH_ANSWER_MODEL`)
- API key: `DEEPSEEK_API_KEY` (or `OPENAI_API_KEY` for other providers)
-Other OpenAI-compatible providers work the same way — set `BENCH_LLM_BASE_URL`, model ids, and the key.
+The judge is configured independently and should be a **different model family** from
+the answerer, otherwise judge accuracy carries self-preference bias. `evaluate` prints a
+warning and records `judge_shares_answer_family` in the manifest when the families match.
+
+- `BENCH_JUDGE_MODEL`, `BENCH_JUDGE_BASE_URL`, `BENCH_JUDGE_API_KEY` for any OpenAI-compatible provider
+- `BENCH_JUDGE_AZURE_ENDPOINT` / `BENCH_JUDGE_AZURE_KEY` / `BENCH_JUDGE_AZURE_API_VERSION` for Azure OpenAI; the repo-root `AZURE_LARGE_LLM_*` variables are picked up automatically when they are exported and no judge model is set explicitly
+- With no judge configuration at all, the judge falls back to the answerer's provider and model, and the run is marked as sharing a family
+
+Other OpenAI-compatible providers work the same way — set the base URL, model ids, and the key.
+
+## Reproducibility
+
+Every `ingest` and `evaluate` manifest records the git SHA and dirty flag, SHA-256 of the
+answer and judge prompts, SHA-256 of the dataset file, and the resolved answerer/judge
+model ids, families and providers. A prompt edit is a measurement-instrument change: it
+changes `answer_prompt_sha256`, and any comparison across that boundary is invalid.
+
+`prompt-audit` guards against tuning the answer prompt on gold answers:
+
+```bash
+./locomo.sh prompt-audit # fails if any 3-gram of ANSWER_SYSTEM appears in a gold answer
+```
+
+`report` de-duplicates answers by `(sample_id, qa_index)`, counts errored rows, and marks
+a run that failed wholesale as `status: failed` with a banner, so a broken run can no
+longer render as a clean 0.0%.
## Cost and time expectations
diff --git a/benchmarks/REPORTS.json b/benchmarks/REPORTS.json
new file mode 100644
index 0000000..d5d79f0
--- /dev/null
+++ b/benchmarks/REPORTS.json
@@ -0,0 +1,164 @@
+{
+ "schema_version": 2,
+ "updated_at": "2026-07-29T18:51:58.401379+00:00",
+ "description": "BrainAPI benchmark results. Top published scores across suites. Updated when a suite evaluate/report completes successfully.",
+ "benchmarks": {
+ "locomo": {
+ "name": "LoCoMo",
+ "leaderboard": [
+ {
+ "run_id": "oldbrain-recheck-a",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8552631578947368,
+ "headline_judge_accuracy_pct": 85.53,
+ "answerable_rate": 0.9473684210526315,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/oldbrain-recheck-a/report.json",
+ "recorded_at": "2026-07-28T22:05:39.768798+00:00"
+ },
+ {
+ "run_id": "phase-sota-d1-conv26",
+ "track": "sota",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8486842105263158,
+ "headline_judge_accuracy_pct": 84.87,
+ "answerable_rate": 0.9407894736842105,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "sota",
+ "brain": "locomoconv26clean",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/phase-sota-d1-conv26/report.json",
+ "recorded_at": "2026-07-29T18:51:58.400341+00:00"
+ },
+ {
+ "run_id": "clean-brain-eval-a",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8355263157894737,
+ "headline_judge_accuracy_pct": 83.55,
+ "answerable_rate": 0.9539473684210527,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26clean",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/clean-brain-eval-a/report.json",
+ "recorded_at": "2026-07-28T21:29:30.713806+00:00"
+ },
+ {
+ "run_id": "graph-stable-b",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8355263157894737,
+ "headline_judge_accuracy_pct": 83.55,
+ "answerable_rate": 0.9671052631578947,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26clean",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/graph-stable-b/report.json",
+ "recorded_at": "2026-07-29T10:59:22.348091+00:00"
+ },
+ {
+ "run_id": "clean-brain-a",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8289473684210527,
+ "headline_judge_accuracy_pct": 82.89,
+ "answerable_rate": 0.9473684210526315,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26clean",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/clean-brain-a/report.json",
+ "recorded_at": "2026-07-28T21:57:06.650866+00:00"
+ },
+ {
+ "run_id": "clean-brain-b",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8289473684210527,
+ "headline_judge_accuracy_pct": 82.89,
+ "answerable_rate": 0.9473684210526315,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26clean",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/clean-brain-b/report.json",
+ "recorded_at": "2026-07-28T22:13:25.651713+00:00"
+ },
+ {
+ "run_id": "baseline-clean-a",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8223684210526315,
+ "headline_judge_accuracy_pct": 82.24,
+ "answerable_rate": 0.9473684210526315,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/baseline-clean-a/report.json",
+ "recorded_at": "2026-07-28T18:18:19.581660+00:00"
+ },
+ {
+ "run_id": "phase-d1-paths-a",
+ "track": "product",
+ "scope": "conv-26",
+ "samples": [
+ "conv-26"
+ ],
+ "n_non_adversarial": 152,
+ "headline_judge_accuracy": 0.8092105263157895,
+ "headline_judge_accuracy_pct": 80.92,
+ "answerable_rate": 0.9671052631578947,
+ "answer_model": "deepseek-v4-flash",
+ "judge_model": "deepseek-v4-flash",
+ "bench_profile": "product",
+ "brain": "locomoconv26clean",
+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",
+ "report_path": "runs/phase-d1-paths-a/report.json",
+ "recorded_at": "2026-07-29T16:26:21.919353+00:00"
+ }
+ ]
+ }
+ }
+}
diff --git a/benchmarks/locomo/answer.py b/benchmarks/locomo/answer.py
index 36d6764..a741aa5 100644
--- a/benchmarks/locomo/answer.py
+++ b/benchmarks/locomo/answer.py
@@ -4,10 +4,11 @@ import time
from dataclasses import dataclass
from typing import Any
-from openai import OpenAI
+from openai import AzureOpenAI, OpenAI
from locomo.config import Settings
from locomo.prompts import build_answer_messages
+from locomo.sota import majority_vote
@dataclass
@@ -18,40 +19,110 @@ class AnswerResult:
prompt_tokens: int | None
completion_tokens: int | None
total_tokens: int | None
+ sc_samples_used: int = 1
-def _make_openai_client(settings: Settings) -> OpenAI:
+def _make_openai_client(settings: Settings) -> OpenAI | AzureOpenAI:
settings.require_llm()
+ if settings.answer_azure_endpoint:
+ return AzureOpenAI(
+ api_key=settings.openai_api_key,
+ azure_endpoint=settings.answer_azure_endpoint,
+ api_version=settings.answer_azure_api_version,
+ )
kwargs: dict[str, Any] = {"api_key": settings.openai_api_key}
if settings.llm_base_url:
kwargs["base_url"] = settings.llm_base_url
return OpenAI(**kwargs)
+def _one_completion(
+ client: OpenAI | AzureOpenAI,
+ *,
+ model_name: str,
+ messages: list[dict[str, str]],
+ temperature: float,
+) -> tuple[str, Any]:
+ response = client.chat.completions.create(
+ model=model_name,
+ messages=messages,
+ temperature=temperature,
+ )
+ content = (response.choices[0].message.content or "").strip()
+ return content, response
+
+
def answer_question(
settings: Settings,
question: str,
context: dict[str, Any],
*,
model: str | None = None,
+ sc_samples: int | None = None,
+ sc_temperature: float | None = None,
) -> AnswerResult:
client = _make_openai_client(settings)
model_name = model or settings.answer_model
messages = build_answer_messages(question, context)
- started = time.perf_counter()
- response = client.chat.completions.create(
- model=model_name,
- messages=messages,
- temperature=0,
+ n = max(1, int(sc_samples if sc_samples is not None else settings.sc_samples))
+ temperature = (
+ float(sc_temperature)
+ if sc_temperature is not None
+ else (settings.sc_temperature if n > 1 else 0.0)
)
+ started = time.perf_counter()
+ prompt_tokens = 0
+ completion_tokens = 0
+ total_tokens = 0
+ samples: list[str] = []
+ last_response: Any = None
+
+ for _ in range(n):
+ content, response = _one_completion(
+ client,
+ model_name=model_name,
+ messages=messages,
+ temperature=0.0 if n == 1 else temperature,
+ )
+ last_response = response
+ if response.usage:
+ prompt_tokens += getattr(response.usage, "prompt_tokens", 0) or 0
+ completion_tokens += getattr(response.usage, "completion_tokens", 0) or 0
+ total_tokens += getattr(response.usage, "total_tokens", 0) or 0
+ if content:
+ samples.append(content)
+
+ content = majority_vote(samples) if samples else ""
+
+ if not content:
+ retry_messages = messages + [
+ {
+ "role": "user",
+ "content": (
+ "Your previous reply was empty. Answer the question now using the "
+ "retrieved context. Prefer a short factual list or phrase."
+ ),
+ }
+ ]
+ content, response = _one_completion(
+ client,
+ model_name=model_name,
+ messages=retry_messages,
+ temperature=0,
+ )
+ last_response = response
+ if response.usage:
+ prompt_tokens += getattr(response.usage, "prompt_tokens", 0) or 0
+ completion_tokens += getattr(response.usage, "completion_tokens", 0) or 0
+ total_tokens += getattr(response.usage, "total_tokens", 0) or 0
+
latency_ms = (time.perf_counter() - started) * 1000
- content = (response.choices[0].message.content or "").strip()
- usage = response.usage
return AnswerResult(
answer=content,
- model=model_name,
+ model=getattr(last_response, "model", None) or model_name,
latency_ms=latency_ms,
- prompt_tokens=getattr(usage, "prompt_tokens", None) if usage else None,
- completion_tokens=getattr(usage, "completion_tokens", None) if usage else None,
- total_tokens=getattr(usage, "total_tokens", None) if usage else None,
+ prompt_tokens=prompt_tokens or None,
+ completion_tokens=completion_tokens or None,
+ total_tokens=total_tokens or None,
+ sc_samples_used=n,
)
diff --git a/benchmarks/locomo/cli.py b/benchmarks/locomo/cli.py
index 9f3fcb8..c3590d1 100644
--- a/benchmarks/locomo/cli.py
+++ b/benchmarks/locomo/cli.py
@@ -3,6 +3,7 @@ from __future__ import annotations
import argparse
import json
from pathlib import Path
+from typing import Any
from rich.console import Console
from rich.table import Table
@@ -18,10 +19,17 @@ from locomo.dataset import (
load_dataset,
resolve_samples,
)
-from locomo.evaluate import evaluate_samples
-from locomo.ingest import ensure_run_dir, ingest_samples, write_manifest
-from locomo.metrics import selftest_metrics
-from locomo.report import print_report_table, write_report
+from locomo.evaluate import evaluate_samples, selftest_records
+from locomo.ingest import ensure_run_dir, ingest_samples, load_jsonl, write_manifest
+from locomo.metrics import (
+ compare_arms,
+ retrieval_arm_summary,
+ selftest_metrics,
+ tokenize,
+)
+from locomo.prompts import ANSWER_SYSTEM
+from locomo.provenance import build_provenance
+from locomo.report import print_comparison, print_report_table, write_report
console = Console()
@@ -99,8 +107,8 @@ def cmd_ingest(args: argparse.Namespace, settings: Settings) -> int:
"limit_sessions": args.limit_sessions,
"dry_run": args.dry_run,
"brainapi_url": settings.brainapi_url,
- "answer_model": settings.answer_model,
- "judge_model": settings.judge_model,
+ "brain_override": args.brain,
+ **build_provenance(settings),
},
)
ingest_samples(
@@ -113,6 +121,7 @@ def cmd_ingest(args: argparse.Namespace, settings: Settings) -> int:
dry_run=args.dry_run,
resume=not args.no_resume,
task_timeout_s=args.timeout,
+ brain_override=args.brain,
)
console.print(f"[green]Run directory:[/green] {run_dir}")
return 0
@@ -141,7 +150,7 @@ def cmd_answer_once(args: argparse.Namespace, settings: Settings) -> int:
def cmd_selftest_metrics(args: argparse.Namespace, settings: Settings) -> int:
- errors = selftest_metrics()
+ errors = selftest_metrics() + selftest_records()
if errors:
for err in errors:
console.print(f"[red]FAIL[/red] {err}")
@@ -155,6 +164,14 @@ def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:
samples = resolve_samples(settings, sample_ids)
run_id, run_dir = ensure_run_dir(settings, args.run)
categories = _parse_categories(args.categories)
+ if settings.judge_shares_answer_family:
+ console.print(
+ f"[yellow]Warning: judge model ({settings.judge_model}) is the same "
+ f"family as the answer model ({settings.answer_model}); judge accuracy "
+ "carries self-preference bias. Set BENCH_JUDGE_MODEL (with "
+ "BENCH_JUDGE_BASE_URL / BENCH_JUDGE_API_KEY) to a different "
+ "family.[/yellow]"
+ )
write_manifest(
run_dir,
{
@@ -166,9 +183,14 @@ def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:
"categories": sorted(categories) if categories else None,
"limit": args.limit,
"brainapi_url": settings.brainapi_url,
- "answer_model": settings.answer_model,
- "judge_model": settings.judge_model,
+ "brain_override": args.brain,
+ **build_provenance(settings),
"historical_limit": args.historical_limit,
+ "max_passages": args.max_passages,
+ "max_facts": args.max_facts,
+ "apply_fact_filter": not args.no_fact_filter,
+ "use_ppr": args.use_ppr,
+ "sufficiency_retry": args.sufficiency_retry,
},
)
evaluate_samples(
@@ -181,6 +203,12 @@ def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:
limit=args.limit,
resume=not args.no_resume,
historical_limit=args.historical_limit,
+ max_passages=args.max_passages,
+ max_facts=args.max_facts,
+ apply_fact_filter=not args.no_fact_filter,
+ use_ppr=args.use_ppr,
+ sufficiency_retry=args.sufficiency_retry,
+ brain_override=args.brain,
)
report = write_report(run_dir)
print_report_table(report)
@@ -200,6 +228,79 @@ def cmd_report(args: argparse.Namespace, settings: Settings) -> int:
return 0
+def _load_run_answers(settings: Settings, run_id: str) -> list[dict[str, Any]]:
+ run_dir = settings.runs_dir / run_id
+ answers_path = run_dir / "answers.jsonl"
+ if not answers_path.exists():
+ raise SystemExit(f"No answers.jsonl in run: {run_dir}")
+ rows = load_jsonl(answers_path)
+ latest: dict[tuple[str, int], dict[str, Any]] = {}
+ for row in rows:
+ if row.get("error"):
+ continue
+ latest[(str(row.get("sample_id")), int(row.get("qa_index") or 0))] = row
+ return list(latest.values())
+
+
+def cmd_compare(args: argparse.Namespace, settings: Settings) -> int:
+ baseline = [_load_run_answers(settings, run) for run in args.baseline]
+ candidate = [_load_run_answers(settings, run) for run in args.candidate]
+ comparison = compare_arms(
+ baseline, candidate, skip_adversarial=not args.include_adversarial
+ )
+ retrieval = {
+ "baseline": {"labels": args.baseline, **retrieval_arm_summary(baseline)},
+ "candidate": {"labels": args.candidate, **retrieval_arm_summary(candidate)},
+ }
+ print_comparison(
+ comparison,
+ " + ".join(args.baseline),
+ " + ".join(args.candidate),
+ retrieval,
+ )
+ if args.json:
+ console.print_json(
+ json.dumps({"comparison": comparison, "retrieval": retrieval})
+ )
+ return 0
+
+
+def cmd_prompt_audit(args: argparse.Namespace, settings: Settings) -> int:
+ dataset = load_dataset(settings.dataset_path)
+ prompt_tokens = tokenize(ANSWER_SYSTEM)
+ n = args.ngram
+ prompt_ngrams = {
+ tuple(prompt_tokens[i : i + n])
+ for i in range(0, max(0, len(prompt_tokens) - n + 1))
+ }
+ hits: list[tuple[str, int, str]] = []
+ for sample in dataset:
+ sample_id = str(sample.get("sample_id"))
+ for idx, qa in enumerate(sample.get("qa") or []):
+ gold = qa.get("answer")
+ if gold is None:
+ gold = qa.get("adversarial_answer")
+ gold_tokens = tokenize(str(gold or ""))
+ for i in range(0, max(0, len(gold_tokens) - n + 1)):
+ gram = tuple(gold_tokens[i : i + n])
+ if gram in prompt_ngrams:
+ hits.append((sample_id, idx, " ".join(gram)))
+ break
+ if hits:
+ console.print(
+ f"[red]FAIL[/red] answer prompt shares a {n}-gram with "
+ f"{len(hits)} gold answers"
+ )
+ for sample_id, idx, gram in hits[:20]:
+ console.print(f" {sample_id}::{idx} {gram!r}")
+ return 1
+ console.print(
+ f"[green]prompt-audit passed[/green] no {n}-gram of the answer prompt "
+ f"appears in any gold answer ({len(dataset)} conversations)"
+ )
+ return 0
+
+
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="python -m locomo",
@@ -231,6 +332,11 @@ def build_parser() -> argparse.ArgumentParser:
help="Sample id (repeatable). Default: all samples.",
)
p_ingest.add_argument("--run", default=None, help="Existing or new run id")
+ p_ingest.add_argument(
+ "--brain",
+ default=None,
+ help="Override the derived brain id (single --sample only)",
+ )
p_ingest.add_argument(
"--granularity", choices=("session", "turn"), default="session"
)
@@ -252,6 +358,11 @@ def build_parser() -> argparse.ArgumentParser:
p_eval = sub.add_parser("evaluate", help="Answer and score LoCoMo QA")
p_eval.add_argument("--sample", action="append")
p_eval.add_argument("--run", default=None)
+ p_eval.add_argument(
+ "--brain",
+ default=None,
+ help="Override the derived brain id (single --sample only)",
+ )
p_eval.add_argument("--concurrency", type=int, default=2)
p_eval.add_argument("--categories", default=None, help="e.g. 1,2,4")
p_eval.add_argument("--limit", type=int, default=None)
@@ -262,6 +373,11 @@ def build_parser() -> argparse.ArgumentParser:
)
p_eval.add_argument("--no-resume", action="store_true")
p_eval.add_argument("--historical-limit", type=int, default=10)
+ p_eval.add_argument("--max-passages", type=int, default=8)
+ p_eval.add_argument("--max-facts", type=int, default=40)
+ p_eval.add_argument("--no-fact-filter", action="store_true")
+ p_eval.add_argument("--use-ppr", action="store_true")
+ p_eval.add_argument("--sufficiency-retry", action="store_true")
p_eval.set_defaults(func=cmd_evaluate)
p_report = sub.add_parser("report", help="Rebuild report from a run directory")
@@ -269,6 +385,22 @@ def build_parser() -> argparse.ArgumentParser:
p_report.add_argument("--json", action="store_true")
p_report.set_defaults(func=cmd_report)
+ p_compare = sub.add_parser(
+ "compare", help="Paired McNemar comparison between two run arms"
+ )
+ p_compare.add_argument("--baseline", action="append", required=True)
+ p_compare.add_argument("--candidate", action="append", required=True)
+ p_compare.add_argument("--include-adversarial", action="store_true")
+ p_compare.add_argument("--json", action="store_true")
+ p_compare.set_defaults(func=cmd_compare)
+
+ p_audit = sub.add_parser(
+ "prompt-audit",
+ help="Fail if the answer prompt shares an n-gram with any gold answer",
+ )
+ p_audit.add_argument("--ngram", type=int, default=3)
+ p_audit.set_defaults(func=cmd_prompt_audit)
+
return parser
diff --git a/benchmarks/locomo/client.py b/benchmarks/locomo/client.py
index 8951fea..51837f0 100644
--- a/benchmarks/locomo/client.py
+++ b/benchmarks/locomo/client.py
@@ -137,11 +137,23 @@ class BrainAPIClient:
brain_id: str,
*,
historical_limit: int = 10,
+ max_passages: int = 8,
+ max_facts: int = 40,
+ apply_fact_filter: bool = True,
+ use_ppr: bool = False,
+ sufficiency_retry: bool = False,
+ cross_event_bridges: int = 3,
) -> TimedResult:
- body = {
+ body: dict[str, Any] = {
"text": text,
"brain_id": brain_id,
"historical_limit": historical_limit,
+ "max_passages": max_passages,
+ "max_facts": max_facts,
+ "apply_fact_filter": apply_fact_filter,
+ "use_ppr": use_ppr,
+ "sufficiency_retry": sufficiency_retry,
+ "cross_event_bridges": cross_event_bridges,
}
return self._request("POST", "/retrieve/context", json=body)
diff --git a/benchmarks/locomo/config.py b/benchmarks/locomo/config.py
index d2f39cf..a859090 100644
--- a/benchmarks/locomo/config.py
+++ b/benchmarks/locomo/config.py
@@ -15,7 +15,33 @@ DEFAULT_DATASET_URL = (
DEFAULT_DATASET_PATH = DATA_DIR / "locomo10.json"
DEFAULT_LLM_BASE_URL = "https://api.deepseek.com"
DEFAULT_ANSWER_MODEL = "deepseek-v4-flash"
-DEFAULT_JUDGE_MODEL = "deepseek-v4-flash"
+DEFAULT_SOTA_ANSWER_MODEL = "deepseek-v4-flash"
+DEFAULT_AZURE_JUDGE_MODEL = "gpt-4o"
+DEFAULT_AZURE_API_VERSION = "2024-12-01-preview"
+DEFAULT_BENCH_PROFILE = "product"
+
+_MODEL_FAMILY_PREFIXES = {
+ "deepseek": "deepseek",
+ "gpt": "openai",
+ "o1": "openai",
+ "o3": "openai",
+ "o4": "openai",
+ "chatgpt": "openai",
+ "claude": "anthropic",
+ "gemini": "google",
+ "grok": "xai",
+ "qwen": "qwen",
+ "llama": "meta",
+ "mistral": "mistral",
+}
+
+
+def model_family(model: str) -> str:
+ name = (model or "").strip().lower().rsplit("/", 1)[-1]
+ for prefix, family in _MODEL_FAMILY_PREFIXES.items():
+ if name.startswith(prefix):
+ return family
+ return name.split("-", 1)[0] or "unknown"
TERMINAL_TASK_STATUSES = frozenset({"completed", "failed", "partial_failed"})
CATEGORY_NAMES = {
@@ -35,9 +61,19 @@ class Settings:
llm_base_url: str | None
answer_model: str
judge_model: str
+ judge_api_key: str
+ judge_base_url: str | None
+ judge_azure_endpoint: str | None
+ judge_azure_api_version: str
dataset_path: Path
dataset_url: str
runs_dir: Path
+ bench_profile: str
+ sc_samples: int
+ sc_temperature: float
+ gap_fill: bool
+ answer_azure_endpoint: str | None = None
+ answer_azure_api_version: str = DEFAULT_AZURE_API_VERSION
@classmethod
def load(cls, env_file: Path | None = None) -> "Settings":
@@ -48,26 +84,114 @@ class Settings:
else:
load_dotenv(override=False)
+ from locomo.sota import profile_defaults
+
+ profile = (
+ os.getenv("BENCH_PROFILE", DEFAULT_BENCH_PROFILE) or DEFAULT_BENCH_PROFILE
+ ).strip().lower()
+ if profile not in {"product", "sota"}:
+ profile = DEFAULT_BENCH_PROFILE
+ defaults = profile_defaults(profile)
+
api_key = (
os.getenv("DEEPSEEK_API_KEY")
or os.getenv("OPENAI_API_KEY")
or ""
)
+ llm_base_url = os.getenv("BENCH_LLM_BASE_URL", DEFAULT_LLM_BASE_URL) or None
+ answer_model = os.getenv("BENCH_ANSWER_MODEL", DEFAULT_ANSWER_MODEL)
+
+ azure_endpoint = (
+ os.getenv("BENCH_JUDGE_AZURE_ENDPOINT")
+ or os.getenv("AZURE_LARGE_LLM_ENDPOINT")
+ or ""
+ ).rstrip("/")
+ azure_key = (
+ os.getenv("BENCH_JUDGE_AZURE_KEY")
+ or os.getenv("AZURE_LARGE_LLM_SUBSCRIPTION_KEY")
+ or ""
+ )
+ azure_available = bool(azure_endpoint and azure_key)
+
+ answer_azure_endpoint: str | None = None
+ if profile == "sota":
+ # Maintainer: deepseek-v4-flash is the preferred SOTA answerer (stronger
+ # than gpt-4o on this stack). Only override via BENCH_SOTA_* envs.
+ sota_answer = os.getenv("BENCH_SOTA_ANSWER_MODEL", "").strip()
+ if sota_answer:
+ answer_model = sota_answer
+ sota_base = os.getenv("BENCH_SOTA_LLM_BASE_URL", "").strip()
+ if sota_base:
+ llm_base_url = sota_base
+ else:
+ answer_model = os.getenv(
+ "BENCH_ANSWER_MODEL", DEFAULT_SOTA_ANSWER_MODEL
+ )
+
+ judge_model = os.getenv("BENCH_JUDGE_MODEL", "")
+ judge_base_url = os.getenv("BENCH_JUDGE_BASE_URL", "")
+ judge_api_key = os.getenv("BENCH_JUDGE_API_KEY", "")
+ use_azure_judge = (
+ azure_available and not judge_model and not judge_base_url
+ )
+ if use_azure_judge:
+ judge_model = os.getenv(
+ "AZURE_LARGE_LLM_MODEL", DEFAULT_AZURE_JUDGE_MODEL
+ )
+ if profile == "sota":
+ sota_judge = os.getenv("BENCH_SOTA_JUDGE_MODEL", "").strip()
+ if sota_judge:
+ judge_model = sota_judge
+ use_azure_judge = False
+ elif not os.getenv("BENCH_JUDGE_MODEL"):
+ judge_model = answer_model
+ use_azure_judge = False
+ judge_base_url = llm_base_url or ""
+ judge_api_key = api_key
return cls(
brainapi_url=os.getenv("BRAINAPI_URL", "http://localhost:8000").rstrip("/"),
brainpat_token=os.getenv("BRAINPAT_TOKEN", ""),
openai_api_key=api_key,
- llm_base_url=os.getenv("BENCH_LLM_BASE_URL", DEFAULT_LLM_BASE_URL) or None,
- answer_model=os.getenv("BENCH_ANSWER_MODEL", DEFAULT_ANSWER_MODEL),
- judge_model=os.getenv("BENCH_JUDGE_MODEL", DEFAULT_JUDGE_MODEL),
+ llm_base_url=llm_base_url,
+ answer_model=answer_model,
+ judge_model=judge_model or answer_model,
+ judge_api_key=(
+ azure_key if use_azure_judge else (judge_api_key or api_key)
+ ),
+ judge_base_url=(
+ None if use_azure_judge else (judge_base_url or llm_base_url)
+ ),
+ judge_azure_endpoint=azure_endpoint if use_azure_judge else None,
+ judge_azure_api_version=os.getenv(
+ "BENCH_JUDGE_AZURE_API_VERSION",
+ os.getenv("AZURE_LARGE_LLM_API_VERSION", DEFAULT_AZURE_API_VERSION),
+ ),
dataset_path=Path(
os.getenv("BENCH_DATASET_PATH", str(DEFAULT_DATASET_PATH))
),
dataset_url=os.getenv("BENCH_DATASET_URL", DEFAULT_DATASET_URL),
runs_dir=Path(os.getenv("BENCH_RUNS_DIR", str(RUNS_DIR))),
+ bench_profile=str(defaults["bench_profile"]),
+ sc_samples=max(1, int(defaults["sc_samples"])),
+ sc_temperature=float(defaults["sc_temperature"]),
+ gap_fill=bool(defaults["gap_fill"]),
+ answer_azure_endpoint=answer_azure_endpoint,
+ answer_azure_api_version=os.getenv(
+ "BENCH_JUDGE_AZURE_API_VERSION",
+ os.getenv("AZURE_LARGE_LLM_API_VERSION", DEFAULT_AZURE_API_VERSION),
+ ),
)
+ @property
+ def judge_shares_answer_family(self) -> bool:
+ return model_family(self.judge_model) == model_family(self.answer_model)
+
+ def judge_provider(self) -> str:
+ if self.judge_azure_endpoint:
+ return "azure"
+ return self.judge_base_url or "openai-default"
+
def require_brainapi(self) -> None:
if not self.brainpat_token:
raise SystemExit(
@@ -80,3 +204,11 @@ class Settings:
"DEEPSEEK_API_KEY (or OPENAI_API_KEY) is required for answer/judge "
"LLM calls. Copy .env.example to .env and set it."
)
+
+ def require_judge_llm(self) -> None:
+ if not self.judge_api_key:
+ raise SystemExit(
+ "No judge API key. Set BENCH_JUDGE_API_KEY (with BENCH_JUDGE_BASE_URL "
+ "and BENCH_JUDGE_MODEL), or DEEPSEEK_API_KEY / OPENAI_API_KEY to reuse "
+ "the answerer's provider."
+ )
diff --git a/benchmarks/locomo/evaluate.py b/benchmarks/locomo/evaluate.py
index 6bbfb1e..52f1615 100644
--- a/benchmarks/locomo/evaluate.py
+++ b/benchmarks/locomo/evaluate.py
@@ -1,8 +1,9 @@
from __future__ import annotations
+import re
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
-from dataclasses import asdict, dataclass
+from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
@@ -16,11 +17,19 @@ from locomo.config import Settings
from locomo.dataset import brain_id_for
from locomo.ingest import append_jsonl, load_jsonl
from locomo.judge import judge_answer
-from locomo.metrics import bleu1_score, f1_score
+from locomo.metrics import evidence_coverage, overlap_scores
+from locomo.prompts import enrich_paths_from_triples, flatten_path, flatten_triple
+from locomo.sota import merge_contexts, plan_gap_fill
console = Console()
_write_lock = threading.Lock()
+_GRAPH_CONTEXT_CAP = 60000
+_TRIPLES_CAP = 500
+_PASSAGE_PREFIX = "[passage] "
+_SESSION_RE = re.compile(r"session_(\d+)", re.IGNORECASE)
+_DIALOG_RE = re.compile(r"\bD(\d+):", re.IGNORECASE)
+
@dataclass
class AnswerRecord:
@@ -35,8 +44,8 @@ class AnswerRecord:
judge_correct: bool
judge_reason: str
judge_raw: str
- f1: float
- bleu1: float
+ f1: float | None
+ bleu1: float | None
retrieve_latency_ms: float | None
answer_latency_ms: float | None
judge_latency_ms: float | None
@@ -44,15 +53,101 @@ class AnswerRecord:
judge_model: str | None
answer_total_tokens: int | None
judge_total_tokens: int | None
- text_context: str
- error: str | None
- timestamp: str
+ graph_context: str
+ triples: list[str] = field(default_factory=list)
+ paths: list[str] = field(default_factory=list)
+ source_passages: list[str] = field(default_factory=list)
+ historical_context: list[str] = field(default_factory=list)
+ retrieved_session_ids: list[str] = field(default_factory=list)
+ retrieved_session_ids_graph: list[str] = field(default_factory=list)
+ retrieved_session_ids_passages: list[str] = field(default_factory=list)
+ context_truncated: dict[str, int] = field(default_factory=dict)
+ is_adversarial: bool = False
+ error: str | None = None
+ timestamp: str = ""
+
+
+def _extract_session_ids(*blobs: Any) -> list[str]:
+ found: list[str] = []
+ seen: set[str] = set()
+ for blob in blobs:
+ if isinstance(blob, (list, tuple)):
+ text = "\n".join(str(item) for item in blob)
+ else:
+ text = str(blob or "")
+ for match in _SESSION_RE.finditer(text):
+ sid = f"session_{match.group(1)}"
+ if sid not in seen:
+ seen.add(sid)
+ found.append(sid)
+ for match in _DIALOG_RE.finditer(text):
+ sid = f"session_{match.group(1)}"
+ if sid not in seen:
+ seen.add(sid)
+ found.append(sid)
+ return found
+
+
+def _graph_channel(text_context: str, source_passages: list[str]) -> str:
+ passage_block = "\n".join(f"{_PASSAGE_PREFIX}{p}" for p in source_passages)
+ if passage_block and text_context.startswith(passage_block):
+ return text_context[len(passage_block) :].lstrip("\n")
+ return "\n".join(
+ line
+ for line in text_context.splitlines()
+ if not line.startswith(_PASSAGE_PREFIX)
+ )
+
+
+def extract_context_channels(context: dict[str, Any]) -> dict[str, Any]:
+ context = enrich_paths_from_triples(context)
+ source_passages = [str(p) for p in (context.get("source_passages") or []) if p]
+ historical_context = [
+ str(h) for h in (context.get("historical_context") or []) if h
+ ]
+ triples = [flatten_triple(t) for t in (context.get("triples") or []) if t]
+ paths = [flatten_path(p) for p in (context.get("paths") or []) if p]
+ graph_context = _graph_channel(
+ str(context.get("text_context") or ""), source_passages
+ )
+ session_ids_graph = _extract_session_ids(graph_context, triples, paths)
+ session_ids_passages = _extract_session_ids(source_passages, historical_context)
+ truncated: dict[str, int] = {}
+ if len(graph_context) > _GRAPH_CONTEXT_CAP:
+ truncated["graph_context"] = len(graph_context)
+ graph_context = graph_context[:_GRAPH_CONTEXT_CAP]
+ if len(triples) > _TRIPLES_CAP:
+ truncated["triples"] = len(triples)
+ triples = triples[:_TRIPLES_CAP]
+ return {
+ "graph_context": graph_context,
+ "triples": triples,
+ "paths": paths,
+ "source_passages": source_passages,
+ "historical_context": historical_context,
+ "retrieved_session_ids": _extract_session_ids(
+ session_ids_passages, session_ids_graph
+ ),
+ "retrieved_session_ids_graph": session_ids_graph,
+ "retrieved_session_ids_passages": session_ids_passages,
+ "context_truncated": truncated,
+ }
def _utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
+def qa_gold(qa: dict[str, Any]) -> tuple[str, bool]:
+ answer = qa.get("answer")
+ if answer is not None and str(answer).strip():
+ return str(answer), int(qa.get("category") or 0) == 5
+ adversarial = qa.get("adversarial_answer")
+ if adversarial is not None:
+ return str(adversarial), True
+ return "", int(qa.get("category") or 0) == 5
+
+
def qa_key(sample_id: str, qa_index: int) -> str:
return f"{sample_id}::{qa_index}"
@@ -74,11 +169,14 @@ def iter_qa_jobs(
categories: set[int] | None = None,
skip_adversarial: bool = True,
limit: int | None = None,
+ brain_override: str | None = None,
) -> list[dict[str, Any]]:
+ if brain_override and len(samples) > 1:
+ raise SystemExit("--brain can only be used with a single --sample")
jobs: list[dict[str, Any]] = []
for sample in samples:
sample_id = str(sample["sample_id"])
- brain_id = brain_id_for(sample_id)
+ brain_id = brain_override or brain_id_for(sample_id)
for idx, qa in enumerate(sample.get("qa") or []):
cat = int(qa.get("category") or 0)
if skip_adversarial and cat == 5:
@@ -98,6 +196,120 @@ def iter_qa_jobs(
return jobs
+def selftest_records() -> list[str]:
+ errors: list[str] = []
+ passages = [
+ "Conversation between Caroline and Melanie.\nSession id: session_9.\n"
+ + ("filler " * 4000),
+ "Session id: session_11. Caroline talked about the marathon.",
+ ]
+ context = {
+ "text_context": "\n".join(
+ [f"{_PASSAGE_PREFIX}{p}" for p in passages]
+ + [
+ "Melanie | PUT | keys | IN | slipper (session_3)",
+ "[dossier] Melanie: runs charity races (session_3)",
+ ]
+ ),
+ "source_passages": passages,
+ "historical_context": ["Session id: session_11. Earlier chunk."],
+ "triples": [
+ {
+ "identified_entity": "Melanie",
+ "triple": ["Melanie", "PUT", "keys", "IN", "slipper"],
+ }
+ ],
+ "paths": [
+ {
+ "hubs": ["event-a", "event-b"],
+ "shared_entity_name": "Melanie",
+ "legs": [
+ "Actor: Melanie | MADE | Event: put keys | TARGETED | Target: slipper",
+ "Actor: Melanie | MADE | Event: run | TARGETED | Target: race",
+ ],
+ }
+ ],
+ }
+ channels = extract_context_channels(context)
+ record = AnswerRecord(
+ sample_id="conv-26",
+ brain_id="locomoconv26",
+ qa_index=0,
+ question="Where did Melanie put the keys?",
+ gold="In Melanie's slipper",
+ prediction="In Melanie's slipper",
+ category=4,
+ evidence=["D3:2"],
+ judge_correct=True,
+ judge_reason="",
+ judge_raw="",
+ f1=1.0,
+ bleu1=1.0,
+ retrieve_latency_ms=1.0,
+ answer_latency_ms=1.0,
+ judge_latency_ms=1.0,
+ answer_model="answer-model",
+ judge_model="judge-model",
+ answer_total_tokens=1,
+ judge_total_tokens=1,
+ error=None,
+ timestamp=_utc_now(),
+ **channels,
+ )
+ row = asdict(record)
+
+ if _PASSAGE_PREFIX in row["graph_context"]:
+ errors.append("graph_context must not contain passage lines")
+ if "PUT" not in row["graph_context"]:
+ errors.append("graph fact line missing from graph_context")
+ if not row["triples"]:
+ errors.append("triples must be logged as their own field")
+ if "slipper" not in row["triples"][0]:
+ errors.append("triples must be flattened to readable text")
+ if not row["paths"]:
+ errors.append("paths must be logged as their own field")
+ if "via Melanie" not in row["paths"][0] or "-->" not in row["paths"][0]:
+ errors.append("paths must flatten to via-entity leg chains")
+ if row["retrieved_session_ids_graph"] != ["session_3"]:
+ errors.append(
+ f"graph channel session ids wrong: {row['retrieved_session_ids_graph']}"
+ )
+ if sorted(row["retrieved_session_ids_passages"]) != ["session_11", "session_9"]:
+ errors.append(
+ "passage channel session ids wrong: "
+ f"{row['retrieved_session_ids_passages']}"
+ )
+ if sorted(row["retrieved_session_ids"]) != [
+ "session_11",
+ "session_3",
+ "session_9",
+ ]:
+ errors.append("combined session ids should be the union of both channels")
+ if evidence_coverage(row, "graph") != "full":
+ errors.append("graph-only evidence should score full recall on the graph channel")
+ if evidence_coverage(row, "passages") != "none":
+ errors.append("graph-only evidence should score no recall on the passage channel")
+
+ long_context = dict(context)
+ long_context["text_context"] = "x" * (_GRAPH_CONTEXT_CAP + 10)
+ long_context["source_passages"] = []
+ capped = extract_context_channels(long_context)
+ if len(capped["graph_context"]) != _GRAPH_CONTEXT_CAP:
+ errors.append("graph_context should be capped")
+ if capped["context_truncated"].get("graph_context") != _GRAPH_CONTEXT_CAP + 10:
+ errors.append("truncation must be recorded with the original size")
+
+ gold, adversarial = qa_gold(
+ {"category": 5, "adversarial_answer": "No information available"}
+ )
+ if gold != "No information available" or not adversarial:
+ errors.append("adversarial gold must be read from adversarial_answer")
+ gold, adversarial = qa_gold({"category": 4, "answer": "7 May 2023"})
+ if gold != "7 May 2023" or adversarial:
+ errors.append("non-adversarial gold must be read from answer")
+ return errors
+
+
def evaluate_samples(
settings: Settings,
samples: list[dict[str, Any]],
@@ -109,6 +321,12 @@ def evaluate_samples(
limit: int | None = None,
resume: bool = True,
historical_limit: int = 10,
+ max_passages: int = 8,
+ max_facts: int = 40,
+ apply_fact_filter: bool = True,
+ use_ppr: bool = False,
+ sufficiency_retry: bool = False,
+ brain_override: str | None = None,
) -> list[AnswerRecord]:
run_dir.mkdir(parents=True, exist_ok=True)
answers_path = run_dir / "answers.jsonl"
@@ -119,6 +337,7 @@ def evaluate_samples(
categories=categories,
skip_adversarial=skip_adversarial,
limit=limit,
+ brain_override=brain_override,
)
jobs = [
j
@@ -140,7 +359,7 @@ def evaluate_samples(
def _one(job: dict[str, Any]) -> AnswerRecord:
qa = job["qa"]
question = str(qa.get("question") or "")
- gold = str(qa.get("answer") if qa.get("answer") is not None else "")
+ gold, is_adversarial = qa_gold(qa)
category = int(qa.get("category") or 0)
evidence = list(qa.get("evidence") or [])
@@ -150,11 +369,46 @@ def evaluate_samples(
question,
job["brain_id"],
historical_limit=historical_limit,
+ max_passages=max_passages,
+ max_facts=max_facts,
+ apply_fact_filter=apply_fact_filter,
+ use_ppr=use_ppr,
+ sufficiency_retry=sufficiency_retry,
)
context = retrieved.data or {}
- answered = answer_question(settings, question, context)
+ retrieve_ms = retrieved.latency_ms
+ draft = answer_question(settings, question, context)
+ if settings.gap_fill:
+ plan = plan_gap_fill(question, draft.answer, context)
+ if plan.needs_retry:
+ with BrainAPIClient(settings) as client:
+ second = client.retrieve_context(
+ plan.reformulated_query,
+ job["brain_id"],
+ historical_limit=historical_limit,
+ max_passages=max_passages,
+ max_facts=max_facts,
+ apply_fact_filter=apply_fact_filter,
+ use_ppr=use_ppr,
+ sufficiency_retry=sufficiency_retry,
+ )
+ retrieve_ms = (retrieve_ms or 0) + (second.latency_ms or 0)
+ context = merge_contexts(context, second.data or {})
+ answered = answer_question(settings, question, context)
+ else:
+ answered = draft
+ else:
+ answered = draft
+ channels = extract_context_channels(context)
judged = judge_answer(
- settings, question, gold, answered.answer
+ settings,
+ question,
+ gold,
+ answered.answer,
+ adversarial=is_adversarial,
+ )
+ f1, bleu1 = overlap_scores(
+ answered.answer, gold, adversarial=is_adversarial
)
record = AnswerRecord(
sample_id=job["sample_id"],
@@ -168,18 +422,19 @@ def evaluate_samples(
judge_correct=judged.correct,
judge_reason=judged.reason,
judge_raw=judged.raw,
- f1=f1_score(answered.answer, gold),
- bleu1=bleu1_score(answered.answer, gold),
- retrieve_latency_ms=retrieved.latency_ms,
+ f1=f1,
+ bleu1=bleu1,
+ retrieve_latency_ms=retrieve_ms,
answer_latency_ms=answered.latency_ms,
judge_latency_ms=judged.latency_ms,
answer_model=answered.model,
judge_model=judged.model,
answer_total_tokens=answered.total_tokens,
judge_total_tokens=judged.total_tokens,
- text_context=str(context.get("text_context") or "")[:4000],
+ is_adversarial=is_adversarial,
error=None,
timestamp=_utc_now(),
+ **channels,
)
except Exception as exc:
record = AnswerRecord(
@@ -194,8 +449,8 @@ def evaluate_samples(
judge_correct=False,
judge_reason="",
judge_raw="",
- f1=0.0,
- bleu1=0.0,
+ f1=None,
+ bleu1=None,
retrieve_latency_ms=None,
answer_latency_ms=None,
judge_latency_ms=None,
@@ -203,9 +458,10 @@ def evaluate_samples(
judge_model=None,
answer_total_tokens=None,
judge_total_tokens=None,
- text_context="",
+ is_adversarial=is_adversarial,
error=str(exc),
timestamp=_utc_now(),
+ **extract_context_channels({}),
)
with _write_lock:
diff --git a/benchmarks/locomo/ingest.py b/benchmarks/locomo/ingest.py
index 4339bfd..9134e13 100644
--- a/benchmarks/locomo/ingest.py
+++ b/benchmarks/locomo/ingest.py
@@ -79,15 +79,19 @@ def ingest_samples(
dry_run: bool = False,
resume: bool = True,
task_timeout_s: float = 900.0,
+ brain_override: str | None = None,
) -> list[IngestRecord]:
run_dir.mkdir(parents=True, exist_ok=True)
ingest_path = run_dir / "ingest.jsonl"
already_done = completed_unit_ids(ingest_path) if resume else set()
+ if brain_override and len(samples) > 1:
+ raise SystemExit("--brain can only be used with a single --sample")
+
jobs: list[dict[str, Any]] = []
for sample in samples:
sample_id = str(sample["sample_id"])
- brain_id = brain_id_for(sample_id)
+ brain_id = brain_override or brain_id_for(sample_id)
for unit in iter_ingest_units(
sample, granularity=granularity, limit_sessions=limit_sessions
):
diff --git a/benchmarks/locomo/judge.py b/benchmarks/locomo/judge.py
index 2fead6d..d97d7d3 100644
--- a/benchmarks/locomo/judge.py
+++ b/benchmarks/locomo/judge.py
@@ -6,7 +6,7 @@ import time
from dataclasses import dataclass
from typing import Any
-from openai import OpenAI
+from openai import AzureOpenAI, OpenAI
from locomo.config import Settings
from locomo.prompts import build_judge_messages
@@ -24,11 +24,17 @@ class JudgeResult:
total_tokens: int | None
-def _make_openai_client(settings: Settings) -> OpenAI:
- settings.require_llm()
- kwargs: dict[str, Any] = {"api_key": settings.openai_api_key}
- if settings.llm_base_url:
- kwargs["base_url"] = settings.llm_base_url
+def _make_openai_client(settings: Settings) -> OpenAI | AzureOpenAI:
+ settings.require_judge_llm()
+ if settings.judge_azure_endpoint:
+ return AzureOpenAI(
+ api_key=settings.judge_api_key,
+ azure_endpoint=settings.judge_azure_endpoint,
+ api_version=settings.judge_azure_api_version,
+ )
+ kwargs: dict[str, Any] = {"api_key": settings.judge_api_key}
+ if settings.judge_base_url:
+ kwargs["base_url"] = settings.judge_base_url
return OpenAI(**kwargs)
@@ -60,10 +66,13 @@ def judge_answer(
prediction: str,
*,
model: str | None = None,
+ adversarial: bool = False,
) -> JudgeResult:
client = _make_openai_client(settings)
model_name = model or settings.judge_model
- messages = build_judge_messages(question, gold, prediction)
+ messages = build_judge_messages(
+ question, gold, prediction, adversarial=adversarial
+ )
started = time.perf_counter()
response = client.chat.completions.create(
model=model_name,
@@ -79,7 +88,7 @@ def judge_answer(
correct=correct,
reason=reason,
raw=raw,
- model=model_name,
+ model=getattr(response, "model", None) or model_name,
latency_ms=latency_ms,
prompt_tokens=getattr(usage, "prompt_tokens", None) if usage else None,
completion_tokens=getattr(usage, "completion_tokens", None) if usage else None,
diff --git a/benchmarks/locomo/metrics.py b/benchmarks/locomo/metrics.py
index dfecc15..a0884af 100644
--- a/benchmarks/locomo/metrics.py
+++ b/benchmarks/locomo/metrics.py
@@ -1,12 +1,15 @@
from __future__ import annotations
import math
+import re
import string
from collections import Counter
from typing import Any, Iterable
_ARTICLES = {"a", "an", "the"}
+_DIALOG_EVIDENCE_RE = re.compile(r"^D(\d+)", re.IGNORECASE)
+_SESSION_RE = re.compile(r"session_(\d+)", re.IGNORECASE)
def normalize_answer(text: str) -> str:
@@ -20,12 +23,12 @@ def tokenize(text: str) -> list[str]:
return normalize_answer(text).split()
-def f1_score(prediction: str, gold: str) -> float:
+def f1_score(prediction: str, gold: str) -> float | None:
pred_tokens = tokenize(prediction)
gold_tokens = tokenize(gold)
- if not pred_tokens and not gold_tokens:
- return 1.0
- if not pred_tokens or not gold_tokens:
+ if not gold_tokens:
+ return None
+ if not pred_tokens:
return 0.0
common = Counter(pred_tokens) & Counter(gold_tokens)
num_same = sum(common.values())
@@ -36,11 +39,13 @@ def f1_score(prediction: str, gold: str) -> float:
return 2 * precision * recall / (precision + recall)
-def bleu1_score(prediction: str, gold: str) -> float:
+def bleu1_score(prediction: str, gold: str) -> float | None:
pred_tokens = tokenize(prediction)
gold_tokens = tokenize(gold)
- if not pred_tokens or not gold_tokens:
- return 1.0 if not pred_tokens and not gold_tokens else 0.0
+ if not gold_tokens:
+ return None
+ if not pred_tokens:
+ return 0.0
gold_counts = Counter(gold_tokens)
match = 0
for token in pred_tokens:
@@ -52,6 +57,14 @@ def bleu1_score(prediction: str, gold: str) -> float:
return precision * brevity
+def overlap_scores(
+ prediction: str, gold: str, *, adversarial: bool = False
+) -> tuple[float | None, float | None]:
+ if adversarial:
+ return None, None
+ return f1_score(prediction, gold), bleu1_score(prediction, gold)
+
+
def percentile(values: list[float], p: float) -> float | None:
if not values:
return None
@@ -87,6 +100,117 @@ def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float | None, fl
return max(0.0, center - margin), min(1.0, center + margin)
+def evidence_session_ids(evidence: Any) -> set[str]:
+ sessions: set[str] = set()
+ for item in evidence or []:
+ text = str(item or "")
+ match = _DIALOG_EVIDENCE_RE.match(text)
+ if match:
+ sessions.add(f"session_{match.group(1)}")
+ continue
+ for session_match in _SESSION_RE.finditer(text):
+ sessions.add(f"session_{session_match.group(1)}")
+ return sessions
+
+
+CHANNELS = ("graph", "passages")
+
+_CHANNEL_ID_FIELD = {
+ "combined": "retrieved_session_ids",
+ "graph": "retrieved_session_ids_graph",
+ "passages": "retrieved_session_ids_passages",
+}
+_CHANNEL_TEXT_FIELDS = {
+ "combined": (
+ "text_context",
+ "graph_context",
+ "triples",
+ "paths",
+ "source_passages",
+ "historical_context",
+ ),
+ "graph": ("graph_context", "triples", "paths"),
+ "passages": ("source_passages", "historical_context"),
+}
+
+
+def _channel_blobs(row: dict[str, Any], channel: str) -> list[str]:
+ blobs: list[str] = []
+ for field in _CHANNEL_TEXT_FIELDS[channel]:
+ value = row.get(field)
+ if isinstance(value, (list, tuple)):
+ blobs.append("\n".join(str(item) for item in value))
+ elif value:
+ blobs.append(str(value))
+ return blobs
+
+
+def _scrape_session_ids(blobs: list[str]) -> set[str]:
+ sessions: set[str] = set()
+ for blob in blobs:
+ for match in _SESSION_RE.finditer(blob):
+ sessions.add(f"session_{match.group(1)}")
+ for match in re.finditer(r"\bD(\d+):", blob, re.IGNORECASE):
+ sessions.add(f"session_{match.group(1)}")
+ return sessions
+
+
+def retrieved_session_ids(
+ row: dict[str, Any], channel: str = "combined"
+) -> set[str] | None:
+ field = _CHANNEL_ID_FIELD[channel]
+ explicit = row.get(field)
+ if explicit is not None:
+ sessions = {str(s) for s in explicit if s}
+ if sessions or channel != "combined":
+ return sessions
+ if channel != "combined" and not any(
+ f in row for f in _CHANNEL_TEXT_FIELDS[channel]
+ ):
+ return None
+ return _scrape_session_ids(_channel_blobs(row, channel))
+
+
+def full_context_blob(row: dict[str, Any]) -> str:
+ return "\n".join(_channel_blobs(row, "combined")).lower()
+
+
+def gold_in_context(row: dict[str, Any], threshold: float = 0.5) -> bool:
+ gold_tokens = [t for t in tokenize(str(row.get("gold") or "")) if len(t) > 2]
+ if not gold_tokens:
+ return False
+ blob = full_context_blob(row)
+ hits = sum(1 for token in gold_tokens if token in blob)
+ return hits / len(gold_tokens) >= threshold
+
+
+def evidence_coverage(row: dict[str, Any], channel: str = "combined") -> str | None:
+ needed = evidence_session_ids(row.get("evidence"))
+ if not needed:
+ return None
+ have = retrieved_session_ids(row, channel)
+ if have is None:
+ return None
+ if needed <= have:
+ return "full"
+ if needed & have:
+ return "partial"
+ return "none"
+
+
+def _recall_bucket(items: list[dict[str, Any]], channel: str) -> dict[str, Any]:
+ coverages = [evidence_coverage(r, channel) for r in items]
+ with_evidence = [c for c in coverages if c is not None]
+ n_ev = len(with_evidence)
+ full = sum(1 for c in with_evidence if c == "full")
+ partial = sum(1 for c in with_evidence if c in {"full", "partial"})
+ return {
+ "evidence_session_recall_full": (full / n_ev) if n_ev else None,
+ "evidence_session_recall_partial": (partial / n_ev) if n_ev else None,
+ "n_with_evidence": n_ev,
+ }
+
+
def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:
by_category: dict[int, list[dict[str, Any]]] = {}
for row in rows:
@@ -95,27 +219,49 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:
def _bucket(items: list[dict[str, Any]]) -> dict[str, Any]:
if not items:
- return {
+ empty = {
"n": 0,
"judge_accuracy": None,
"judge_accuracy_ci95": None,
"mean_f1": None,
"mean_bleu1": None,
+ "n_scored_f1": 0,
+ "answerable_rate": None,
+ "evidence_session_recall_full": None,
+ "evidence_session_recall_partial": None,
+ "n_with_evidence": 0,
}
+ for channel in CHANNELS:
+ empty[f"evidence_session_recall_full_{channel}"] = None
+ empty[f"evidence_session_recall_partial_{channel}"] = None
+ empty[f"n_with_evidence_{channel}"] = 0
+ return empty
correct = [1.0 if r.get("judge_correct") else 0.0 for r in items]
successes = int(sum(correct))
lo, hi = wilson_ci(successes, len(items))
- return {
+ answerable = [1.0 if gold_in_context(r) else 0.0 for r in items]
+ f1s = [float(r["f1"]) for r in items if r.get("f1") is not None]
+ bleus = [float(r["bleu1"]) for r in items if r.get("bleu1") is not None]
+ bucket = {
"n": len(items),
"judge_accuracy": mean(correct),
"judge_accuracy_ci95": (
{"low": lo, "high": hi} if lo is not None and hi is not None else None
),
- "mean_f1": mean(float(r.get("f1") or 0.0) for r in items),
- "mean_bleu1": mean(float(r.get("bleu1") or 0.0) for r in items),
+ "mean_f1": mean(f1s),
+ "mean_bleu1": mean(bleus),
+ "n_scored_f1": len(f1s),
+ "answerable_rate": mean(answerable),
+ **_recall_bucket(items, "combined"),
}
+ for channel in CHANNELS:
+ channel_bucket = _recall_bucket(items, channel)
+ for key, value in channel_bucket.items():
+ bucket[f"{key}_{channel}"] = value
+ return bucket
non_adv = [r for r in rows if int(r.get("category") or 0) != 5]
+ adversarial = [r for r in rows if int(r.get("category") or 0) == 5]
all_rows = rows
non_adv_bucket = _bucket(non_adv)
all_bucket = _bucket(all_rows)
@@ -130,6 +276,16 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:
if r.get(key) is not None:
total_tokens += int(r[key])
+ answerer_gap = None
+ if (
+ non_adv_bucket.get("answerable_rate") is not None
+ and non_adv_bucket.get("judge_accuracy") is not None
+ ):
+ answerer_gap = (
+ float(non_adv_bucket["answerable_rate"])
+ - float(non_adv_bucket["judge_accuracy"])
+ )
+
return {
"n_total": len(all_rows),
"n_non_adversarial": len(non_adv),
@@ -139,6 +295,26 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:
"overall_judge_accuracy_ci95": all_bucket["judge_accuracy_ci95"],
"mean_f1_non_adversarial": non_adv_bucket["mean_f1"],
"mean_bleu1_non_adversarial": non_adv_bucket["mean_bleu1"],
+ "answerable_rate": non_adv_bucket["answerable_rate"],
+ "evidence_session_recall_full": non_adv_bucket["evidence_session_recall_full"],
+ "evidence_session_recall_partial": non_adv_bucket[
+ "evidence_session_recall_partial"
+ ],
+ "evidence_session_recall_by_channel": {
+ channel: {
+ "full": non_adv_bucket[f"evidence_session_recall_full_{channel}"],
+ "partial": non_adv_bucket[
+ f"evidence_session_recall_partial_{channel}"
+ ],
+ "n_with_evidence": non_adv_bucket[f"n_with_evidence_{channel}"],
+ }
+ for channel in CHANNELS
+ },
+ "answerer_gap": answerer_gap,
+ "abstention": {
+ "n": len(adversarial),
+ "accuracy": _bucket(adversarial)["judge_accuracy"],
+ },
"by_category": {
str(cat): _bucket(items) for cat, items in sorted(by_category.items())
},
@@ -151,6 +327,213 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:
}
+def mcnemar_exact(flipped_wrong: int, flipped_right: int) -> float:
+ n = flipped_wrong + flipped_right
+ if n <= 0:
+ return 1.0
+ k = min(flipped_wrong, flipped_right)
+ tail = sum(math.comb(n, i) for i in range(0, k + 1)) / (2**n)
+ return min(1.0, 2.0 * tail)
+
+
+def _correct_by_question(rows: list[dict[str, Any]]) -> dict[tuple[str, int], bool]:
+ scores: dict[tuple[str, int], bool] = {}
+ for row in rows:
+ if row.get("error"):
+ continue
+ key = (str(row.get("sample_id")), int(row.get("qa_index") or 0))
+ scores[key] = bool(row.get("judge_correct"))
+ return scores
+
+
+def _arm_scores(arm: list[list[dict[str, Any]]]) -> dict[tuple[str, int], float]:
+ per_run = [_correct_by_question(rows) for rows in arm if rows]
+ if not per_run:
+ return {}
+ keys = set(per_run[0])
+ for run in per_run[1:]:
+ keys &= set(run)
+ return {
+ key: sum(1.0 for run in per_run if run[key]) / len(per_run) for key in keys
+ }
+
+
+def _category_by_question(
+ arm: list[list[dict[str, Any]]],
+) -> dict[tuple[str, int], int]:
+ categories: dict[tuple[str, int], int] = {}
+ for rows in arm:
+ for row in rows:
+ key = (str(row.get("sample_id")), int(row.get("qa_index") or 0))
+ categories.setdefault(key, int(row.get("category") or 0))
+ return categories
+
+
+def _flip_table(
+ baseline: dict[tuple[str, int], float],
+ candidate: dict[tuple[str, int], float],
+ keys: Iterable[tuple[str, int]],
+) -> dict[str, Any]:
+ flipped_right: list[tuple[str, int]] = []
+ flipped_wrong: list[tuple[str, int]] = []
+ agreed = 0
+ for key in keys:
+ base = baseline[key]
+ cand = candidate[key]
+ if cand > base:
+ flipped_right.append(key)
+ elif cand < base:
+ flipped_wrong.append(key)
+ else:
+ agreed += 1
+ p_value = mcnemar_exact(len(flipped_wrong), len(flipped_right))
+ return {
+ "n_paired": agreed + len(flipped_right) + len(flipped_wrong),
+ "agreed": agreed,
+ "flipped_right": len(flipped_right),
+ "flipped_wrong": len(flipped_wrong),
+ "mcnemar_exact_p": p_value,
+ "significant_at_05": p_value < 0.05,
+ "flipped_right_questions": [f"{s}::{i}" for s, i in sorted(flipped_right)],
+ "flipped_wrong_questions": [f"{s}::{i}" for s, i in sorted(flipped_wrong)],
+ }
+
+
+def compare_arms(
+ baseline_runs: list[list[dict[str, Any]]],
+ candidate_runs: list[list[dict[str, Any]]],
+ *,
+ skip_adversarial: bool = True,
+) -> dict[str, Any]:
+ baseline = _arm_scores(baseline_runs)
+ candidate = _arm_scores(candidate_runs)
+ categories = _category_by_question(baseline_runs + candidate_runs)
+ keys = sorted(set(baseline) & set(candidate))
+ if skip_adversarial:
+ keys = [k for k in keys if categories.get(k, 0) != 5]
+
+ overall = _flip_table(baseline, candidate, keys)
+ by_category: dict[str, Any] = {}
+ for cat in sorted({categories.get(k, 0) for k in keys}):
+ cat_keys = [k for k in keys if categories.get(k, 0) == cat]
+ by_category[str(cat)] = _flip_table(baseline, candidate, cat_keys)
+
+ graph_stability = None
+ if len(baseline_runs) == 1 and len(candidate_runs) == 1:
+ graph_stability = graph_session_stability(
+ baseline_runs[0],
+ candidate_runs[0],
+ skip_adversarial=skip_adversarial,
+ )
+
+ return {
+ "baseline_runs": len(baseline_runs),
+ "candidate_runs": len(candidate_runs),
+ "baseline_accuracy": mean(baseline[k] for k in keys),
+ "candidate_accuracy": mean(candidate[k] for k in keys),
+ "overall": overall,
+ "by_category": by_category,
+ "graph_session_stability": graph_stability,
+ }
+
+
+_GRAPH_STABILITY_GATE = 0.95
+
+
+def _graph_session_set(row: dict[str, Any]) -> frozenset[str]:
+ raw = row.get("retrieved_session_ids_graph")
+ if raw is None:
+ raw = row.get("graph_session_ids")
+ if not raw:
+ return frozenset()
+ return frozenset(str(s) for s in raw if s)
+
+
+def _row_key(row: dict[str, Any]) -> tuple[str, int]:
+ return (str(row.get("sample_id") or ""), int(row.get("qa_index") or 0))
+
+
+def graph_session_stability(
+ baseline_rows: list[dict[str, Any]],
+ candidate_rows: list[dict[str, Any]],
+ *,
+ skip_adversarial: bool = True,
+ gate: float = _GRAPH_STABILITY_GATE,
+) -> dict[str, Any]:
+ baseline_map = {_row_key(r): r for r in baseline_rows if not r.get("error")}
+ candidate_map = {_row_key(r): r for r in candidate_rows if not r.get("error")}
+ keys = sorted(set(baseline_map) & set(candidate_map))
+ if skip_adversarial:
+ keys = [
+ k
+ for k in keys
+ if int(baseline_map[k].get("category") or 0) != 5
+ and int(candidate_map[k].get("category") or 0) != 5
+ ]
+
+ identical = 0
+ identical_coverage = 0
+ coverage_comparable = 0
+ disagreeing: list[str] = []
+ for key in keys:
+ base = baseline_map[key]
+ cand = candidate_map[key]
+ base_set = _graph_session_set(base)
+ cand_set = _graph_session_set(cand)
+ if base_set == cand_set:
+ identical += 1
+ else:
+ disagreeing.append(f"{key[0]}::{key[1]}")
+ base_cov = evidence_coverage(base, "graph")
+ cand_cov = evidence_coverage(cand, "graph")
+ if base_cov is not None and cand_cov is not None:
+ coverage_comparable += 1
+ if base_cov == cand_cov:
+ identical_coverage += 1
+
+ n = len(keys)
+ agreement_rate = (identical / n) if n else None
+ coverage_agreement_rate = (
+ (identical_coverage / coverage_comparable) if coverage_comparable else None
+ )
+ return {
+ "n_paired": n,
+ "identical_session_sets": identical,
+ "agreement_rate": agreement_rate,
+ "identical_graph_coverage": identical_coverage,
+ "n_coverage_comparable": coverage_comparable,
+ "coverage_agreement_rate": coverage_agreement_rate,
+ "gate": gate,
+ "passes_gate": (
+ agreement_rate is not None and agreement_rate >= gate
+ ),
+ "disagreeing_questions": disagreeing,
+ "note": (
+ "Do not A/B graph EvR below the gate: graph EvR is a measurement only "
+ f"when ≥{gate:.0%} of questions have identical graph-session sets "
+ "across two identical-config runs."
+ ),
+ }
+
+
+def retrieval_arm_summary(arm: list[list[dict[str, Any]]]) -> dict[str, Any]:
+ per_run = []
+ for rows in arm:
+ scored = [r for r in rows if not r.get("error")]
+ metrics = aggregate_answers(scored)
+ per_run.append(
+ {
+ "evidence_session_recall_full": metrics["evidence_session_recall_full"],
+ "evidence_session_recall_by_channel": metrics[
+ "evidence_session_recall_by_channel"
+ ],
+ "answerable_rate": metrics["answerable_rate"],
+ }
+ )
+ identical = all(run == per_run[0] for run in per_run) if per_run else True
+ return {"per_run": per_run, "identical_across_runs": identical}
+
+
def selftest_metrics() -> list[str]:
errors: list[str] = []
if abs(f1_score("7 May 2023", "7 May 2023") - 1.0) > 1e-9:
@@ -164,4 +547,124 @@ def selftest_metrics() -> list[str]:
lo, hi = wilson_ci(5, 10)
if lo is None or hi is None or not (0.0 <= lo <= 0.5 <= hi <= 1.0):
errors.append("wilson_ci(5,10) should contain 0.5")
+ if evidence_session_ids(["D7:3", "D12:1"]) != {"session_7", "session_12"}:
+ errors.append("evidence_session_ids failed")
+ sample = {
+ "gold": "Charlotte's Web",
+ "text_context": "Melanie loves Charlotte's Web",
+ "source_passages": [],
+ "historical_context": [],
+ "evidence": ["D7:1"],
+ "retrieved_session_ids": ["session_7"],
+ }
+ if not gold_in_context(sample):
+ errors.append("gold_in_context should detect gold tokens")
+ if evidence_coverage(sample) != "full":
+ errors.append("evidence_coverage should be full")
+
+ if f1_score("", "") is not None:
+ errors.append("empty prediction vs empty gold must not score F1")
+ if bleu1_score("", "") is not None:
+ errors.append("empty prediction vs empty gold must not score BLEU-1")
+ if f1_score("Not mentioned", "") is not None:
+ errors.append("empty gold must not be scorable by F1")
+ if f1_score("", "Paris") != 0.0:
+ errors.append("empty prediction vs non-empty gold should have F1=0")
+ if overlap_scores("", "", adversarial=True) != (None, None):
+ errors.append("adversarial items must not be scored by n-gram overlap")
+ if overlap_scores("LGBTQ+ folks", "LGBTQ+ individuals", adversarial=True) != (
+ None,
+ None,
+ ):
+ errors.append("matching an adversarial trap answer must not score as overlap")
+
+ channels = {
+ "gold": "slipper",
+ "graph_context": "Melanie | OWNS | slipper (session_3)",
+ "triples": ["Melanie: Melanie | PUT | keys | IN | slipper"],
+ "source_passages": ["[passage] Session id: session_9. Caroline said hi."],
+ "historical_context": [],
+ "evidence": ["D3:2"],
+ }
+ if retrieved_session_ids(channels, "graph") != {"session_3"}:
+ errors.append("graph channel session ids failed")
+ if retrieved_session_ids(channels, "passages") != {"session_9"}:
+ errors.append("passage channel session ids failed")
+ if evidence_coverage(channels, "graph") != "full":
+ errors.append("graph channel evidence coverage should be full")
+ if evidence_coverage(channels, "passages") != "none":
+ errors.append("passage channel evidence coverage should be none")
+ legacy = {"evidence": ["D3:2"], "retrieved_session_ids": ["session_3"]}
+ if evidence_coverage(legacy, "graph") is not None:
+ errors.append("graph coverage must be unknown for rows without a graph channel")
+
+ if abs(mcnemar_exact(9, 20) - 0.06142835) > 1e-6:
+ errors.append("mcnemar_exact(9, 20) should be ~0.0614")
+ if abs(mcnemar_exact(10, 32) - 0.00094067) > 1e-7:
+ errors.append("mcnemar_exact(10, 32) should be ~0.00094")
+ if mcnemar_exact(0, 0) != 1.0:
+ errors.append("mcnemar_exact with no discordant pairs should be 1.0")
+
+ base_rows = [
+ {"sample_id": "s", "qa_index": 0, "category": 1, "judge_correct": False},
+ {"sample_id": "s", "qa_index": 1, "category": 1, "judge_correct": True},
+ ]
+ cand_rows = [
+ {"sample_id": "s", "qa_index": 0, "category": 1, "judge_correct": True},
+ {"sample_id": "s", "qa_index": 1, "category": 1, "judge_correct": True},
+ ]
+ comparison = compare_arms([base_rows], [cand_rows])
+ if comparison["overall"]["flipped_right"] != 1:
+ errors.append("compare_arms should count one flip to correct")
+ if comparison["overall"]["flipped_wrong"] != 0:
+ errors.append("compare_arms should count no flip to wrong")
+
+ stable_a = [
+ {
+ "sample_id": "s",
+ "qa_index": 0,
+ "category": 1,
+ "retrieved_session_ids_graph": ["session_1", "session_2"],
+ "evidence": ["D1:1", "D2:1"],
+ "judge_correct": True,
+ },
+ {
+ "sample_id": "s",
+ "qa_index": 1,
+ "category": 4,
+ "retrieved_session_ids_graph": ["session_3"],
+ "evidence": ["D3:1"],
+ "judge_correct": True,
+ },
+ ]
+ stable_b = [
+ {
+ "sample_id": "s",
+ "qa_index": 0,
+ "category": 1,
+ "graph_session_ids": ["session_2", "session_1"],
+ "evidence": ["D1:1", "D2:1"],
+ "judge_correct": True,
+ },
+ {
+ "sample_id": "s",
+ "qa_index": 1,
+ "category": 4,
+ "retrieved_session_ids_graph": ["session_3"],
+ "evidence": ["D3:1"],
+ "judge_correct": True,
+ },
+ ]
+ stability = graph_session_stability(stable_a, stable_b)
+ if stability["agreement_rate"] != 1.0 or not stability["passes_gate"]:
+ errors.append("graph_session_stability should pass on identical sets")
+ unstable_b = [
+ dict(stable_b[0], retrieved_session_ids_graph=["session_9"]),
+ stable_b[1],
+ ]
+ unstable = graph_session_stability(stable_a, unstable_b)
+ if unstable["agreement_rate"] != 0.5:
+ errors.append("graph_session_stability should report 50% on one disagreement")
+ if "s::0" not in unstable["disagreeing_questions"]:
+ errors.append("graph_session_stability should list disagreeing qa keys")
return errors
diff --git a/benchmarks/locomo/prompts.py b/benchmarks/locomo/prompts.py
index abb9063..b4f6baa 100644
--- a/benchmarks/locomo/prompts.py
+++ b/benchmarks/locomo/prompts.py
@@ -1,14 +1,21 @@
from __future__ import annotations
+import hashlib
from typing import Any
ANSWER_SYSTEM = """You are answering questions about a long-term conversation using only the retrieved memory context provided.
Rules:
-- Prefer short factual answers: dates, names, places, or short phrases when that is what the question asks for.
+- Prefer short factual answers: dates, names, places, or short phrases when that is what the question asks for. Never return an empty answer.
+- Refer to people by the names used in the dialogue rather than by the first-person pronouns the speakers use.
- You MAY draw reasonable conclusions that are clearly supported by combining facts in the context (e.g. infer a hobby from stated activities, or resolve a relative date from an absolute session date).
- Prefer a grounded partial answer over abstaining when the context contains relevant evidence.
-- Reply exactly "Not mentioned in the conversation" ONLY when the context contains no evidence that could support an answer — not when the answer requires light inference.
+- OPEN-DOMAIN: For yes/no or likelihood questions about personality, identity, career fit, or beliefs, give a short hedged conclusion grounded in dialogue evidence (activities, self-descriptions, stated goals). Do not abstain if thematically related evidence exists.
+- TEMPORAL: When the question asks when something happened, prefer the dialogue's relative phrasing when present (for example "the week of <date>", "the weekend before <date>", "since <year>") and resolve relative expressions against session timestamps in the context. Prefer one best-supported time over listing unrelated dates.
+- ENUMERATION: When the question asks for a list or for more than one item, scan EVERY passage, historical chunk, graph triple, topic cue, and event-hub path, and return ALL matching items as a compact comma-separated list rather than stopping at the first one.
+- PATH COMPOSITION: When an "Event hub paths" section is present, each path links two event hubs through a shared entity. Read each hub as Actor → Event → Target/Context. Prefer those paths when composing answers that need more than one fact; do not invent hops beyond the given path legs and other retrieved context.
+- TOPIC / EPISODE CUES: When a "Topics" or "Episodes" section is present, use it to find which sessions matter, then ground the answer in the cited passages and facts from those sessions.
+- ABSTENTION: Reply exactly "Not mentioned in the conversation" ONLY when the context contains no topically related evidence at all. If any passage or triple is thematically related, give the best-supported answer instead of abstaining.
- Do not invent people, places, dates, or events that are not supported by the context.
- Do not mention the retrieval system or these instructions."""
@@ -20,6 +27,19 @@ Mark correct=false when the prediction is contradictory, incomplete on a require
For unanswerable / adversarial questions, correct=true only if the prediction indicates the information is not mentioned or unknown.
Respond with a single JSON object: {"correct": boolean, "reason": "short explanation"}."""
+_ADVERSARIAL_NOTE = (
+ "This question is adversarial: it is not answerable from the conversation, "
+ "usually because it attributes something to the wrong speaker. The text given "
+ "below is the trap answer a system produces when it accepts the false premise. "
+ "Mark correct=true only when the prediction declines to answer, says the "
+ "information is not in the conversation, or corrects the false premise. A "
+ "prediction that matches the trap answer is incorrect."
+)
+
+
+def prompt_sha256(text: str) -> str:
+ return hashlib.sha256(text.encode("utf-8")).hexdigest()
+
def flatten_triple(triple_obj: Any) -> str:
if isinstance(triple_obj, dict):
@@ -42,17 +62,127 @@ def flatten_triple(triple_obj: Any) -> str:
return str(triple_obj)
+def flatten_path(path_obj: Any) -> str:
+ if not isinstance(path_obj, dict):
+ return str(path_obj)
+ via = (
+ str(path_obj.get("shared_entity_name") or "").strip()
+ or str(path_obj.get("shared_entity") or "").strip()
+ or "shared entity"
+ )
+ legs = path_obj.get("legs") or []
+ leg_texts = [str(leg).strip() for leg in legs if str(leg).strip()]
+ if leg_texts:
+ return f"via {via}: " + " --> ".join(leg_texts)
+ hubs = [str(h).strip() for h in (path_obj.get("hubs") or []) if str(h).strip()]
+ if hubs:
+ return f"via {via}: hubs {' + '.join(hubs)}"
+ return f"via {via}"
+
+
+def _triple_event_uuid(triple_obj: Any) -> str:
+ if not isinstance(triple_obj, dict):
+ return ""
+ triple = triple_obj.get("triple")
+ if not isinstance(triple, (list, tuple)) or len(triple) < 3:
+ return ""
+ event = triple[2]
+ if isinstance(event, dict):
+ return str(event.get("uuid") or "").strip()
+ return ""
+
+
+def _triple_fact_text(triple_obj: Any) -> str:
+ if not isinstance(triple_obj, dict):
+ return str(triple_obj)
+ triple = triple_obj.get("triple")
+ if isinstance(triple, (list, tuple)) and len(triple) >= 5:
+ parts = []
+ for item in triple:
+ if isinstance(item, dict):
+ parts.append(
+ item.get("name")
+ or item.get("type")
+ or item.get("description")
+ or str(item)
+ )
+ else:
+ parts.append(str(item))
+ return " | ".join(parts)
+ return flatten_triple(triple_obj)
+
+
+def enrich_paths_from_triples(context: dict[str, Any]) -> dict[str, Any]:
+ """Attach readable path legs from triples when the API omitted them."""
+ paths = context.get("paths") or []
+ triples = context.get("triples") or []
+ if not paths:
+ return context
+ if any(isinstance(p, dict) and p.get("legs") for p in paths):
+ return context
+
+ hub_texts: dict[str, str] = {}
+ for triple in triples:
+ hub = _triple_event_uuid(triple)
+ text = _triple_fact_text(triple).strip()
+ if hub and text and hub not in hub_texts:
+ hub_texts[hub] = text
+ if not hub_texts:
+ return context
+
+ enriched: list[dict[str, Any]] = []
+ for path in paths:
+ if not isinstance(path, dict):
+ continue
+ hubs = [str(h) for h in (path.get("hubs") or []) if h]
+ if len(hubs) < 2 or not all(h in hub_texts for h in hubs):
+ continue
+ enriched.append(
+ {
+ **path,
+ "legs": [hub_texts[h] for h in hubs],
+ }
+ )
+ if not enriched:
+ return context
+ out = dict(context)
+ out["paths"] = enriched
+ return out
+
+
def build_context_block(context: dict[str, Any]) -> str:
+ context = enrich_paths_from_triples(context)
text_context = (context.get("text_context") or "").strip()
triples = context.get("triples") or []
+ paths = context.get("paths") or []
historical = context.get("historical_context") or []
source_passages = context.get("source_passages") or []
+ topics = context.get("topics") or []
+ episodes = context.get("episodes") or []
sections = ["## Retrieved text context", text_context or "(empty)"]
+ if topics:
+ sections.append("## Topics")
+ for topic in topics[:20]:
+ if isinstance(topic, dict):
+ label = topic.get("label") or topic.get("topic_id") or topic
+ sessions = topic.get("sessions") or []
+ sess = ", ".join(str(s) for s in sessions[:12])
+ sections.append(f"- {label}" + (f" ({sess})" if sess else ""))
+ else:
+ sections.append(f"- {topic}")
+ if episodes:
+ sections.append("## Episodes")
+ for episode in episodes[:20]:
+ sections.append(f"- {episode}")
if source_passages:
sections.append("## Source passages")
for passage in source_passages[:20]:
sections.append(f"- {passage}")
+ if paths:
+ sections.append("## Event hub paths")
+ for path in paths[:40]:
+ sections.append(f"- {flatten_path(path)}")
if triples:
sections.append("## Graph triples")
for t in triples[:80]:
@@ -77,13 +207,16 @@ def build_answer_messages(question: str, context: dict[str, Any]) -> list[dict[s
def build_judge_messages(
- question: str, gold: str, prediction: str
+ question: str, gold: str, prediction: str, *, adversarial: bool = False
) -> list[dict[str, str]]:
+ gold_label = "Trap answer" if adversarial else "Gold answer"
user = (
f"Question: {question}\n"
- f"Gold answer: {gold}\n"
+ f"{gold_label}: {gold}\n"
f"Predicted answer: {prediction}\n"
)
+ if adversarial:
+ user = f"{_ADVERSARIAL_NOTE}\n\n{user}"
return [
{"role": "system", "content": JUDGE_SYSTEM},
{"role": "user", "content": user},
diff --git a/benchmarks/locomo/provenance.py b/benchmarks/locomo/provenance.py
new file mode 100644
index 0000000..3c5a936
--- /dev/null
+++ b/benchmarks/locomo/provenance.py
@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+import hashlib
+import subprocess
+from pathlib import Path
+from typing import Any
+
+from locomo.config import BENCHMARKS_ROOT, Settings, model_family
+from locomo.prompts import ANSWER_SYSTEM, JUDGE_SYSTEM, prompt_sha256
+
+REPO_ROOT = BENCHMARKS_ROOT.parent
+
+
+def _git(*args: str) -> str | None:
+ try:
+ result = subprocess.run(
+ ["git", *args],
+ cwd=REPO_ROOT,
+ capture_output=True,
+ text=True,
+ timeout=10,
+ )
+ except (OSError, subprocess.SubprocessError):
+ return None
+ if result.returncode != 0:
+ return None
+ return result.stdout.strip()
+
+
+def git_sha() -> str | None:
+ return _git("rev-parse", "HEAD")
+
+
+def git_dirty() -> bool | None:
+ status = _git("status", "--porcelain")
+ if status is None:
+ return None
+ return bool(status.strip())
+
+
+def file_sha256(path: Path) -> str | None:
+ if not path.exists():
+ return None
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def build_provenance(settings: Settings) -> dict[str, Any]:
+ return {
+ "git_sha": git_sha(),
+ "git_dirty": git_dirty(),
+ "answer_prompt_sha256": prompt_sha256(ANSWER_SYSTEM),
+ "judge_prompt_sha256": prompt_sha256(JUDGE_SYSTEM),
+ "dataset_path": str(settings.dataset_path),
+ "dataset_sha256": file_sha256(settings.dataset_path),
+ "answer_model": settings.answer_model,
+ "answer_model_family": model_family(settings.answer_model),
+ "answer_llm_base_url": settings.llm_base_url,
+ "judge_model": settings.judge_model,
+ "judge_model_family": model_family(settings.judge_model),
+ "judge_provider": settings.judge_provider(),
+ "judge_shares_answer_family": settings.judge_shares_answer_family,
+ "bench_profile": settings.bench_profile,
+ "sc_samples": settings.sc_samples,
+ "sc_temperature": settings.sc_temperature,
+ "gap_fill": settings.gap_fill,
+ }
diff --git a/benchmarks/locomo/report.py b/benchmarks/locomo/report.py
index 9946c8d..530cdf5 100644
--- a/benchmarks/locomo/report.py
+++ b/benchmarks/locomo/report.py
@@ -1,18 +1,46 @@
from __future__ import annotations
import json
+import tempfile
+from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from rich.console import Console
from rich.table import Table
-from locomo.config import CATEGORY_NAMES
+from locomo.config import BENCHMARKS_ROOT, CATEGORY_NAMES
from locomo.ingest import load_jsonl
from locomo.metrics import aggregate_answers
console = Console()
+REPORTS_PATH = BENCHMARKS_ROOT / "REPORTS.json"
+_BENCHMARK_ID = "locomo"
+_BENCHMARK_NAME = "LoCoMo"
+_REPORTS_DESCRIPTION = (
+ "BrainAPI benchmark results. Top published scores across suites. "
+ "Updated when a suite evaluate/report completes successfully."
+)
+_LEADERBOARD_ENTRY_KEYS = (
+ "run_id",
+ "track",
+ "scope",
+ "samples",
+ "n_non_adversarial",
+ "headline_judge_accuracy",
+ "headline_judge_accuracy_pct",
+ "answerable_rate",
+ "answer_model",
+ "judge_model",
+ "bench_profile",
+ "brain",
+ "git_sha",
+ "report_path",
+ "recorded_at",
+)
+_MIN_LEADERBOARD_N = 50
+
def _latest_ingest_rows(ingest_rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Keep the last status per sample/unit (resume appends retries)."""
@@ -26,12 +54,28 @@ def _latest_ingest_rows(ingest_rows: list[dict[str, Any]]) -> list[dict[str, Any
return list(latest.values())
+def _latest_answer_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ latest: dict[tuple[str, int], dict[str, Any]] = {}
+ for row in rows:
+ key = (str(row.get("sample_id") or ""), int(row.get("qa_index") or 0))
+ latest[key] = row
+ return list(latest.values())
+
+
def build_report(run_dir: Path) -> dict[str, Any]:
answers_path = run_dir / "answers.jsonl"
ingest_path = run_dir / "ingest.jsonl"
manifest_path = run_dir / "manifest.json"
- answers = [r for r in load_jsonl(answers_path) if not r.get("error")]
+ all_answer_rows = load_jsonl(answers_path)
+ scored_rows = [r for r in all_answer_rows if not r.get("error")]
+ answers = _latest_answer_rows(scored_rows)
+ errored = [r for r in all_answer_rows if r.get("error")]
+ duplicates_dropped = len(scored_rows) - len(answers)
+ empty_predictions = sum(
+ 1 for r in answers if not str(r.get("prediction") or "").strip()
+ )
+ truncated_rows = sum(1 for r in answers if r.get("context_truncated"))
ingest_rows = load_jsonl(ingest_path)
ingest_latest = _latest_ingest_rows(ingest_rows)
manifest = {}
@@ -50,35 +94,128 @@ def build_report(run_dir: Path) -> dict[str, Any]:
"attempts": len(ingest_rows),
}
+ integrity = {
+ "rows_in_file": len(all_answer_rows),
+ "errored_rows": len(errored),
+ "scored_rows": len(answers),
+ "duplicates_dropped": duplicates_dropped,
+ "empty_predictions": empty_predictions,
+ "rows_with_truncated_context": truncated_rows,
+ }
+
report = {
"run_dir": str(run_dir),
"manifest": manifest,
"ingest": ingest_summary,
+ "integrity": integrity,
"metrics": metrics,
+ "status": "ok",
"warnings": [],
}
if ingest_summary["partial_failed"] or ingest_summary["failed"]:
report["warnings"].append(
"Some ingest units failed or partially failed; scores may be skewed."
)
+ if duplicates_dropped:
+ report["warnings"].append(
+ f"{duplicates_dropped} duplicate answer rows dropped "
+ f"({len(scored_rows)} scored rows over {len(answers)} unique questions)."
+ )
+ if errored:
+ report["warnings"].append(
+ f"{len(errored)} rows errored and are excluded from every metric."
+ )
+ if truncated_rows:
+ report["warnings"].append(
+ f"{truncated_rows} rows had a retrieved-context channel truncated "
+ "before logging; see context_truncated in answers.jsonl."
+ )
+ graph_rows = [r for r in answers if r.get("graph_context") or r.get("triples")]
+ if graph_rows and not any(r.get("retrieved_session_ids_graph") for r in graph_rows):
+ report["warnings"].append(
+ f"The graph channel was logged for {len(graph_rows)} rows but carries no "
+ "session ids, so its evidence recall reads 0% by construction: "
+ "/retrieve/context returns graph facts without session provenance."
+ )
+
+ failure_reasons: list[str] = []
+ if not answers:
+ failure_reasons.append("no answer rows were scored")
+ else:
+ if empty_predictions == len(answers):
+ failure_reasons.append("every scored answer is empty")
+ if metrics.get("headline_judge_accuracy") == 0.0:
+ failure_reasons.append("every scored answer was judged incorrect")
+ if failure_reasons:
+ report["status"] = "failed"
+ report["warnings"].insert(
+ 0,
+ "RUN FAILED WHOLESALE: "
+ + "; ".join(failure_reasons)
+ + ". The percentages below are not a measurement of quality.",
+ )
+ elif errored or duplicates_dropped:
+ report["status"] = "degraded"
return report
def render_markdown(report: dict[str, Any]) -> str:
metrics = report["metrics"]
manifest = report.get("manifest") or {}
+ integrity = report.get("integrity") or {}
+ channels = metrics.get("evidence_session_recall_by_channel") or {}
lines = [
"# LoCoMo BrainAPI Benchmark Report",
"",
- f"- Run dir: `{report['run_dir']}`",
- f"- Created: {manifest.get('created_at', 'n/a')}",
- f"- Granularity: {manifest.get('granularity', 'n/a')}",
- f"- Answer model: {manifest.get('answer_model', 'n/a')}",
- f"- Judge model: {manifest.get('judge_model', 'n/a')}",
- f"- Samples: {', '.join(manifest.get('samples', []) or ['n/a'])}",
- "",
- "## Headline",
- "",
+ ]
+ if report.get("status") == "failed":
+ lines.extend(
+ [
+ "> **RUN FAILED WHOLESALE — the numbers below are not a measurement.**",
+ f"> {report['warnings'][0]}",
+ "",
+ ]
+ )
+ elif report.get("status") == "degraded":
+ lines.extend(
+ [
+ "> **Run degraded** — errored or duplicate rows were dropped; "
+ "see Warnings.",
+ "",
+ ]
+ )
+ lines.extend(
+ [
+ f"- Run dir: `{report['run_dir']}`",
+ f"- Created: {manifest.get('created_at', 'n/a')}",
+ f"- Granularity: {manifest.get('granularity', 'n/a')}",
+ f"- Answer model: {manifest.get('answer_model', 'n/a')}",
+ f"- Judge model: {manifest.get('judge_model', 'n/a')} "
+ f"(provider: {manifest.get('judge_provider', 'n/a')}, "
+ f"same family as answerer: "
+ f"{manifest.get('judge_shares_answer_family', 'n/a')})",
+ f"- Git SHA: {manifest.get('git_sha', 'n/a')} "
+ f"(dirty: {manifest.get('git_dirty', 'n/a')})",
+ f"- Answer prompt sha256: {manifest.get('answer_prompt_sha256', 'n/a')}",
+ f"- Judge prompt sha256: {manifest.get('judge_prompt_sha256', 'n/a')}",
+ f"- Dataset sha256: {manifest.get('dataset_sha256', 'n/a')}",
+ f"- Samples: {', '.join(manifest.get('samples', []) or ['n/a'])}",
+ "",
+ "## Run integrity",
+ "",
+ f"- Rows in answers.jsonl: {integrity.get('rows_in_file', 'n/a')}",
+ f"- Scored unique questions: {integrity.get('scored_rows', 'n/a')}",
+ f"- Errored rows (excluded): {integrity.get('errored_rows', 'n/a')}",
+ f"- Duplicate rows dropped: {integrity.get('duplicates_dropped', 'n/a')}",
+ f"- Empty predictions: {integrity.get('empty_predictions', 'n/a')}",
+ f"- Rows with truncated context: "
+ f"{integrity.get('rows_with_truncated_context', 'n/a')}",
+ "",
+ "## Headline",
+ "",
+ ]
+ )
+ lines.extend([
f"- Judge accuracy (excl. adversarial / cat 5): "
f"**{_pct(metrics.get('headline_judge_accuracy'))}** "
f"{_ci(metrics.get('headline_judge_accuracy_ci95'))} "
@@ -91,18 +228,43 @@ def render_markdown(report: dict[str, Any]) -> str:
f"**{_num(metrics.get('mean_f1_non_adversarial'))}**",
f"- Mean BLEU-1 (excl. adversarial): "
f"**{_num(metrics.get('mean_bleu1_non_adversarial'))}**",
+ f"- Answerable rate (gold tokens in context): "
+ f"**{_pct(metrics.get('answerable_rate'))}**",
+ f"- Evidence-session recall (full): "
+ f"**{_pct(metrics.get('evidence_session_recall_full'))}**",
+ f"- Evidence-session recall (partial+full): "
+ f"**{_pct(metrics.get('evidence_session_recall_partial'))}**",
+ f"- Evidence-session recall, graph channel: "
+ f"**{_pct((channels.get('graph') or {}).get('full'))}** "
+ f"(n={(channels.get('graph') or {}).get('n_with_evidence', 0)})",
+ f"- Evidence-session recall, passage channel: "
+ f"**{_pct((channels.get('passages') or {}).get('full'))}** "
+ f"(n={(channels.get('passages') or {}).get('n_with_evidence', 0)})",
+ f"- Answerer gap (answerable − judge): "
+ f"**{_pct(metrics.get('answerer_gap'))}**",
+ f"- Abstention accuracy (cat 5): "
+ f"**{_pct((metrics.get('abstention') or {}).get('accuracy'))}** "
+ f"(n={(metrics.get('abstention') or {}).get('n', 0)})",
+ "",
+ "Retrieval-side metrics are deterministic for a fixed brain and config; "
+ "they are reported exactly and carry no significance test. "
+ "Judge accuracy is not — compare runs with `python -m locomo compare`.",
"",
"## Per-category",
"",
- "| Category | Name | N | Judge Acc | 95% CI | Mean F1 | Mean BLEU-1 |",
- "|---|---|---:|---:|---|---:|---:|",
- ]
+ "| Category | Name | N | Judge Acc | 95% CI | Answerable | EvRecall | EvRecall graph | EvRecall passages | Mean F1 | Mean BLEU-1 |",
+ "|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|",
+ ])
for cat, bucket in (metrics.get("by_category") or {}).items():
name = CATEGORY_NAMES.get(int(cat), "unknown")
lines.append(
f"| {cat} | {name} | {bucket.get('n', 0)} | "
f"{_pct(bucket.get('judge_accuracy'))} | "
f"{_ci(bucket.get('judge_accuracy_ci95'))} | "
+ f"{_pct(bucket.get('answerable_rate'))} | "
+ f"{_pct(bucket.get('evidence_session_recall_full'))} | "
+ f"{_pct(bucket.get('evidence_session_recall_full_graph'))} | "
+ f"{_pct(bucket.get('evidence_session_recall_full_passages'))} | "
f"{_num(bucket.get('mean_f1'))} | "
f"{_num(bucket.get('mean_bleu1'))} |"
)
@@ -142,6 +304,186 @@ def render_markdown(report: dict[str, Any]) -> str:
return "\n".join(lines)
+def entry_from_report(report: dict[str, Any]) -> dict[str, Any] | None:
+ if report.get("status") == "failed":
+ return None
+ metrics = report.get("metrics") or {}
+ headline = metrics.get("headline_judge_accuracy")
+ n_non_adv = int(metrics.get("n_non_adversarial") or 0)
+ if headline is None or n_non_adv < _MIN_LEADERBOARD_N:
+ return None
+
+ manifest = report.get("manifest") or {}
+ run_id = str(manifest.get("run_id") or "").strip()
+ if not run_id:
+ run_dir = str(report.get("run_dir") or "").rstrip("/")
+ run_id = Path(run_dir).name if run_dir else ""
+ if not run_id:
+ return None
+
+ samples = list(manifest.get("samples") or [])
+ if len(samples) == 1:
+ scope = str(samples[0])
+ elif len(samples) >= 10:
+ scope = "locomo10"
+ elif samples:
+ scope = ",".join(str(s) for s in samples)
+ else:
+ scope = "unknown"
+
+ bench_profile = manifest.get("bench_profile") or "product"
+ track = "sota" if bench_profile == "sota" else "product"
+ brain = manifest.get("brain_override")
+ acc = float(headline)
+ now = datetime.now(timezone.utc).isoformat()
+
+ return {
+ "run_id": run_id,
+ "track": track,
+ "scope": scope,
+ "samples": samples,
+ "n_non_adversarial": n_non_adv,
+ "headline_judge_accuracy": acc,
+ "headline_judge_accuracy_pct": round(acc * 100, 2),
+ "answerable_rate": metrics.get("answerable_rate"),
+ "answer_model": manifest.get("answer_model"),
+ "judge_model": manifest.get("judge_model"),
+ "bench_profile": bench_profile,
+ "brain": brain,
+ "git_sha": manifest.get("git_sha"),
+ "report_path": f"runs/{run_id}/report.json",
+ "recorded_at": now,
+ }
+
+
+def _empty_reports_ledger() -> dict[str, Any]:
+ return {
+ "schema_version": 2,
+ "updated_at": datetime.now(timezone.utc).isoformat(),
+ "description": _REPORTS_DESCRIPTION,
+ "benchmarks": {
+ _BENCHMARK_ID: {
+ "name": _BENCHMARK_NAME,
+ "leaderboard": [],
+ }
+ },
+ }
+
+
+def _public_leaderboard_entry(row: dict[str, Any]) -> dict[str, Any]:
+ return {key: row[key] for key in _LEADERBOARD_ENTRY_KEYS if key in row}
+
+
+def _sorted_leaderboard(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ leaderboard = [
+ _public_leaderboard_entry(row)
+ for row in rows
+ if isinstance(row, dict) and row.get("run_id")
+ ]
+ leaderboard.sort(
+ key=lambda row: float(row.get("headline_judge_accuracy") or 0.0),
+ reverse=True,
+ )
+ return leaderboard
+
+
+def _normalize_reports_ledger(data: dict[str, Any]) -> dict[str, Any]:
+ benchmarks: dict[str, Any] = {}
+ raw_benchmarks = data.get("benchmarks")
+ if isinstance(raw_benchmarks, dict):
+ for suite_id, suite in raw_benchmarks.items():
+ if not isinstance(suite, dict):
+ continue
+ benchmarks[str(suite_id)] = {
+ "name": suite.get("name") or str(suite_id),
+ "leaderboard": _sorted_leaderboard(list(suite.get("leaderboard") or [])),
+ }
+
+ # One-shot migrate schema v1 flat leaderboard into benchmarks.locomo.
+ legacy = data.get("leaderboard")
+ if isinstance(legacy, list) and legacy:
+ suite = dict(benchmarks.get(_BENCHMARK_ID) or {})
+ by_id: dict[str, dict[str, Any]] = {
+ str(row["run_id"]): row
+ for row in (suite.get("leaderboard") or [])
+ if isinstance(row, dict) and row.get("run_id")
+ }
+ for row in legacy:
+ if not isinstance(row, dict) or not row.get("run_id"):
+ continue
+ rid = str(row["run_id"])
+ if rid not in by_id:
+ by_id[rid] = _public_leaderboard_entry(row)
+ suite["name"] = suite.get("name") or _BENCHMARK_NAME
+ suite["leaderboard"] = _sorted_leaderboard(list(by_id.values()))
+ benchmarks[_BENCHMARK_ID] = suite
+
+ if _BENCHMARK_ID not in benchmarks:
+ benchmarks[_BENCHMARK_ID] = {
+ "name": _BENCHMARK_NAME,
+ "leaderboard": [],
+ }
+
+ return {
+ "schema_version": 2,
+ "updated_at": data.get("updated_at")
+ or datetime.now(timezone.utc).isoformat(),
+ "description": _REPORTS_DESCRIPTION,
+ "benchmarks": benchmarks,
+ }
+
+
+def update_reports_json(report: dict[str, Any]) -> None:
+ entry = entry_from_report(report)
+ if entry is None:
+ return
+
+ if REPORTS_PATH.exists():
+ data = json.loads(REPORTS_PATH.read_text(encoding="utf-8"))
+ else:
+ data = _empty_reports_ledger()
+
+ data = _normalize_reports_ledger(data)
+
+ suite = data["benchmarks"].setdefault(
+ _BENCHMARK_ID,
+ {"name": _BENCHMARK_NAME, "leaderboard": []},
+ )
+ run_id = entry["run_id"]
+ leaderboard = list(suite.get("leaderboard") or [])
+ lb_idx = next(
+ (i for i, row in enumerate(leaderboard) if row.get("run_id") == run_id),
+ None,
+ )
+ if lb_idx is not None:
+ leaderboard[lb_idx] = entry
+ else:
+ leaderboard.append(entry)
+
+ suite["name"] = suite.get("name") or _BENCHMARK_NAME
+ suite["leaderboard"] = leaderboard
+ data["benchmarks"][_BENCHMARK_ID] = suite
+ data["updated_at"] = datetime.now(timezone.utc).isoformat()
+ data["description"] = _REPORTS_DESCRIPTION
+ data = _normalize_reports_ledger(data)
+
+ REPORTS_PATH.parent.mkdir(parents=True, exist_ok=True)
+ fd, tmp_name = tempfile.mkstemp(
+ prefix="REPORTS.",
+ suffix=".json.tmp",
+ dir=str(REPORTS_PATH.parent),
+ )
+ tmp_path = Path(tmp_name)
+ try:
+ with open(fd, "w", encoding="utf-8") as fh:
+ json.dump(data, fh, indent=2)
+ fh.write("\n")
+ tmp_path.replace(REPORTS_PATH)
+ except Exception:
+ tmp_path.unlink(missing_ok=True)
+ raise
+
+
def write_report(run_dir: Path) -> dict[str, Any]:
report = build_report(run_dir)
md = render_markdown(report)
@@ -149,6 +491,8 @@ def write_report(run_dir: Path) -> dict[str, Any]:
json.dumps(report, indent=2), encoding="utf-8"
)
(run_dir / "report.md").write_text(md, encoding="utf-8")
+ if report.get("status") == "ok":
+ update_reports_json(report)
return report
@@ -160,6 +504,10 @@ def print_report_table(report: dict[str, Any]) -> None:
table.add_column("N", justify="right")
table.add_column("Judge Acc", justify="right")
table.add_column("95% CI")
+ table.add_column("Answerable", justify="right")
+ table.add_column("EvRecall", justify="right")
+ table.add_column("EvR graph", justify="right")
+ table.add_column("EvR passages", justify="right")
table.add_column("F1", justify="right")
table.add_column("BLEU-1", justify="right")
for cat, bucket in (metrics.get("by_category") or {}).items():
@@ -169,19 +517,143 @@ def print_report_table(report: dict[str, Any]) -> None:
str(bucket.get("n", 0)),
_pct(bucket.get("judge_accuracy")),
_ci(bucket.get("judge_accuracy_ci95")),
+ _pct(bucket.get("answerable_rate")),
+ _pct(bucket.get("evidence_session_recall_full")),
+ _pct(bucket.get("evidence_session_recall_full_graph")),
+ _pct(bucket.get("evidence_session_recall_full_passages")),
_num(bucket.get("mean_f1")),
_num(bucket.get("mean_bleu1")),
)
+ if report.get("status") == "failed":
+ console.print(
+ f"[bold red]{(report.get('warnings') or ['RUN FAILED'])[0]}[/bold red]"
+ )
console.print(table)
console.print(
f"Headline judge accuracy (excl. cat 5): "
f"[bold]{_pct(metrics.get('headline_judge_accuracy'))}[/bold] "
f"{_ci(metrics.get('headline_judge_accuracy_ci95'))}"
)
+ channels = metrics.get("evidence_session_recall_by_channel") or {}
+ console.print(
+ f"Answerable rate: {_pct(metrics.get('answerable_rate'))} | "
+ f"EvRecall full: {_pct(metrics.get('evidence_session_recall_full'))} "
+ f"(graph {_pct((channels.get('graph') or {}).get('full'))}, "
+ f"passages {_pct((channels.get('passages') or {}).get('full'))}) | "
+ f"Answerer gap: {_pct(metrics.get('answerer_gap'))}"
+ )
for warning in report.get("warnings") or []:
console.print(f"[yellow]Warning: {warning}[/yellow]")
+def print_comparison(
+ comparison: dict[str, Any],
+ baseline_label: str,
+ candidate_label: str,
+ retrieval: dict[str, Any] | None = None,
+) -> None:
+ overall = comparison["overall"]
+ console.print(
+ f"[bold]{baseline_label}[/bold] -> [bold]{candidate_label}[/bold] "
+ f"({comparison['baseline_runs']} vs {comparison['candidate_runs']} run(s), "
+ f"n={overall['n_paired']} paired questions)"
+ )
+ table = Table(title="Paired judge-accuracy comparison (exact McNemar)")
+ table.add_column("Scope")
+ table.add_column("N", justify="right")
+ table.add_column("Flipped right", justify="right")
+ table.add_column("Flipped wrong", justify="right")
+ table.add_column("Exact p", justify="right")
+ table.add_column("Significant")
+ rows = [("overall", overall)]
+ for cat, bucket in (comparison.get("by_category") or {}).items():
+ rows.append((f"{cat} {CATEGORY_NAMES.get(int(cat), 'unknown')}", bucket))
+ for label, bucket in rows:
+ table.add_row(
+ label,
+ str(bucket["n_paired"]),
+ str(bucket["flipped_right"]),
+ str(bucket["flipped_wrong"]),
+ f"{bucket['mcnemar_exact_p']:.4g}",
+ "yes" if bucket["significant_at_05"] else "no",
+ )
+ console.print(table)
+ console.print(
+ f"Accuracy: {_pct(comparison['baseline_accuracy'])} -> "
+ f"{_pct(comparison['candidate_accuracy'])}"
+ )
+ if retrieval:
+ console.print(
+ "[dim]Passage retrieval metrics are exact for a fixed brain/config. "
+ "Graph EvR is a measurement only after graph-session set agreement "
+ "clears the ≥95% gate below.[/dim]"
+ )
+ ret_table = Table(title="Retrieval metrics (exact, no significance test)")
+ ret_table.add_column("Arm")
+ ret_table.add_column("Run")
+ ret_table.add_column("EvRecall full", justify="right")
+ ret_table.add_column("EvR graph", justify="right")
+ ret_table.add_column("EvR passages", justify="right")
+ ret_table.add_column("Answerable", justify="right")
+ for arm_label, arm in (
+ ("baseline", retrieval.get("baseline") or {}),
+ ("candidate", retrieval.get("candidate") or {}),
+ ):
+ for name, run in zip(
+ arm.get("labels") or [], arm.get("per_run") or []
+ ):
+ channels = run.get("evidence_session_recall_by_channel") or {}
+ ret_table.add_row(
+ arm_label,
+ name,
+ _pct(run.get("evidence_session_recall_full")),
+ _pct((channels.get("graph") or {}).get("full")),
+ _pct((channels.get("passages") or {}).get("full")),
+ _pct(run.get("answerable_rate")),
+ )
+ console.print(ret_table)
+ for arm_label, arm in (
+ ("baseline", retrieval.get("baseline") or {}),
+ ("candidate", retrieval.get("candidate") or {}),
+ ):
+ if len(arm.get("per_run") or []) > 1:
+ verdict = (
+ "identical across runs"
+ if arm.get("identical_across_runs")
+ else "NOT identical across runs"
+ )
+ console.print(f"[dim]{arm_label} retrieval metrics: {verdict}[/dim]")
+
+ stability = comparison.get("graph_session_stability")
+ if stability:
+ rate = stability.get("agreement_rate")
+ gate = stability.get("gate")
+ n = stability.get("n_paired")
+ identical = stability.get("identical_session_sets")
+ passes = stability.get("passes_gate")
+ console.print(
+ f"Graph-session set agreement: "
+ f"{_pct(rate)} ({identical}/{n} questions identical); "
+ f"gate {_pct(gate)} — "
+ f"{'PASS' if passes else 'FAIL (do not A/B graph EvR)'}"
+ )
+ cov_rate = stability.get("coverage_agreement_rate")
+ if cov_rate is not None:
+ console.print(
+ f"Graph coverage-label agreement: {_pct(cov_rate)} "
+ f"({stability.get('identical_graph_coverage')}/"
+ f"{stability.get('n_coverage_comparable')})"
+ )
+ console.print(f"[dim]{stability.get('note')}[/dim]")
+ disagreeing = stability.get("disagreeing_questions") or []
+ if disagreeing:
+ preview = ", ".join(disagreeing[:12])
+ more = (
+ f" (+{len(disagreeing) - 12} more)" if len(disagreeing) > 12 else ""
+ )
+ console.print(f"Disagreeing questions: {preview}{more}")
+
+
def _pct(value: float | None) -> str:
if value is None:
return "n/a"
diff --git a/benchmarks/locomo/sota.py b/benchmarks/locomo/sota.py
new file mode 100644
index 0000000..135e859
--- /dev/null
+++ b/benchmarks/locomo/sota.py
@@ -0,0 +1,119 @@
+from __future__ import annotations
+
+import os
+import re
+from collections import Counter
+from dataclasses import dataclass
+from typing import Any
+
+
+_NORMALIZE_RE = re.compile(r"[^a-z0-9\s]+")
+_WS_RE = re.compile(r"\s+")
+
+
+def normalize_answer(text: str) -> str:
+ lowered = (text or "").strip().lower()
+ lowered = _NORMALIZE_RE.sub(" ", lowered)
+ return _WS_RE.sub(" ", lowered).strip()
+
+
+def majority_vote(answers: list[str]) -> str:
+ """Pick the most common normalized answer; ties keep first occurrence order."""
+ nonempty = [a.strip() for a in answers if (a or "").strip()]
+ if not nonempty:
+ return ""
+ counts = Counter(normalize_answer(a) for a in nonempty)
+ winner_norm, _ = max(
+ counts.items(),
+ key=lambda item: (item[1], -next(
+ i for i, a in enumerate(nonempty) if normalize_answer(a) == item[0]
+ )),
+ )
+ for answer in nonempty:
+ if normalize_answer(answer) == winner_norm:
+ return answer
+ return nonempty[0]
+
+
+@dataclass(frozen=True)
+class GapFillPlan:
+ needs_retry: bool
+ reformulated_query: str
+ missing_slots: list[str]
+
+
+def plan_gap_fill(
+ question: str,
+ draft_answer: str,
+ context: dict[str, Any],
+) -> GapFillPlan:
+ """Heuristic FAIR-RAG-lite: abstention or thin draft → re-retrieve with question."""
+ draft = (draft_answer or "").strip()
+ text_blob = " ".join(
+ [
+ str(context.get("text_context") or ""),
+ " ".join(str(p) for p in (context.get("source_passages") or [])[:20]),
+ " ".join(str(t) for t in (context.get("triples") or [])[:40]),
+ ]
+ ).lower()
+ abstained = bool(
+ re.search(
+ r"not mentioned|no information|unknown|cannot (tell|determine)",
+ draft,
+ re.I,
+ )
+ )
+ q_tokens = [t for t in re.findall(r"[a-z0-9]{3,}", question.lower()) if t]
+ covered = sum(1 for t in q_tokens if t in text_blob) if q_tokens else 0
+ coverage = covered / len(q_tokens) if q_tokens else 1.0
+ missing = [t for t in q_tokens if t not in text_blob][:8]
+ needs = abstained or (coverage < 0.45 and len(draft) < 40)
+ reform = question.strip()
+ if missing:
+ reform = f"{question.strip()} (focus: {', '.join(missing[:5])})"
+ return GapFillPlan(
+ needs_retry=needs,
+ reformulated_query=reform,
+ missing_slots=missing,
+ )
+
+
+def merge_contexts(
+ primary: dict[str, Any], secondary: dict[str, Any]
+) -> dict[str, Any]:
+ """Union passage/fact channels from a second retrieve into the first."""
+ out = dict(primary)
+ for key in ("source_passages", "historical_context", "triples", "paths"):
+ a = list(primary.get(key) or [])
+ b = list(secondary.get(key) or [])
+ seen: set[str] = set()
+ merged: list[Any] = []
+ for item in a + b:
+ sig = str(item)
+ if sig in seen:
+ continue
+ seen.add(sig)
+ merged.append(item)
+ out[key] = merged
+ text_a = str(primary.get("text_context") or "")
+ text_b = str(secondary.get("text_context") or "")
+ if text_b and text_b not in text_a:
+ out["text_context"] = (text_a + "\n" + text_b).strip()
+ return out
+
+
+def profile_defaults(profile: str) -> dict[str, Any]:
+ name = (profile or "product").strip().lower()
+ if name == "sota":
+ return {
+ "bench_profile": "sota",
+ "sc_samples": int(os.getenv("BENCH_SC_SAMPLES", "5")),
+ "sc_temperature": float(os.getenv("BENCH_SC_TEMPERATURE", "0.7")),
+ "gap_fill": os.getenv("BENCH_GAP_FILL", "1") not in {"0", "false", "False"},
+ }
+ return {
+ "bench_profile": "product",
+ "sc_samples": int(os.getenv("BENCH_SC_SAMPLES", "1")),
+ "sc_temperature": 0.0,
+ "gap_fill": os.getenv("BENCH_GAP_FILL", "0") in {"1", "true", "True"},
+ }
diff --git a/benchmarks/locomo/taxonomy.py b/benchmarks/locomo/taxonomy.py
new file mode 100644
index 0000000..52c5126
--- /dev/null
+++ b/benchmarks/locomo/taxonomy.py
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+"""Classify wrong LoCoMo answers into generation vs retrieval failure modes."""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from collections import Counter, defaultdict
+from pathlib import Path
+from typing import Any
+
+from locomo.config import CATEGORY_NAMES, RUNS_DIR
+from locomo.metrics import evidence_coverage, gold_in_context, tokenize
+
+_RELATIVE_DATE = re.compile(
+ r"\b(week of|since \d{4}|sunday before|saturday before|the day before)\b",
+ re.I,
+)
+_ABSTAIN = re.compile(
+ r"not mentioned|no information|unknown|cannot (tell|determine)",
+ re.I,
+)
+
+
+def _load_latest(path: Path) -> list[dict[str, Any]]:
+ latest: dict[tuple[str, int], dict[str, Any]] = {}
+ for line in path.read_text().splitlines():
+ if not line.strip():
+ continue
+ row = json.loads(line)
+ if row.get("error"):
+ continue
+ key = (str(row.get("sample_id")), int(row.get("qa_index") or 0))
+ latest[key] = row
+ return list(latest.values())
+
+
+def classify(row: dict[str, Any]) -> str:
+ cat = int(row.get("category") or 0)
+ gold = str(row.get("gold") or "")
+ pred = str(row.get("prediction") or "")
+ reason = str(row.get("judge_reason") or "").lower()
+ answerable = gold_in_context(row)
+ coverage = evidence_coverage(row, "combined")
+
+ if not answerable and coverage in {None, "none", "partial"}:
+ return "missing_evidence"
+ if _ABSTAIN.search(pred) and gold.strip():
+ return "present_but_unused"
+ if cat == 3:
+ return "open_domain_inference"
+ if cat == 2 and (
+ _RELATIVE_DATE.search(gold)
+ or "date" in reason
+ or "temporal" in reason
+ or "phrasing" in reason
+ ):
+ return "temporal_format"
+ if cat == 1:
+ return "multi_hop_composition"
+ if answerable:
+ gold_toks = set(tokenize(gold))
+ pred_toks = set(tokenize(pred))
+ if gold_toks and len(gold_toks & pred_toks) / len(gold_toks) >= 0.4:
+ return "phrasing"
+ return "present_but_unused"
+ return "missing_evidence"
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--run",
+ default="phase-d1-paths-a",
+ help="Run id under benchmarks/runs/",
+ )
+ parser.add_argument(
+ "--out",
+ type=Path,
+ default=None,
+ help="Markdown output path",
+ )
+ args = parser.parse_args()
+ run_dir = RUNS_DIR / args.run
+ answers_path = run_dir / "answers.jsonl"
+ if not answers_path.exists():
+ raise SystemExit(f"missing {answers_path}")
+
+ rows = _load_latest(answers_path)
+ wrong = [
+ r
+ for r in rows
+ if not r.get("judge_correct") and int(r.get("category") or 0) != 5
+ ]
+ by_class: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for row in wrong:
+ by_class[classify(row)].append(row)
+
+ counts = Counter({k: len(v) for k, v in by_class.items()})
+ n_wrong = len(wrong)
+ gen_classes = {
+ "present_but_unused",
+ "phrasing",
+ "open_domain_inference",
+ "temporal_format",
+ "multi_hop_composition",
+ }
+ gen_n = sum(counts[c] for c in gen_classes)
+ gen_share = (gen_n / n_wrong) if n_wrong else 0.0
+ n_total = len([r for r in rows if int(r.get("category") or 0) != 5])
+
+ lines: list[str] = []
+ lines.append(f"# SOTA failure taxonomy — `{args.run}`")
+ lines.append("")
+ lines.append(f"Non-adversarial scored: **{n_total}**; wrong: **{n_wrong}**.")
+ lines.append(
+ f"Generation-side share (expected ≥70%): **{gen_share:.0%}** "
+ f"({gen_n}/{n_wrong})."
+ )
+ lines.append("")
+ lines.append("| Class | Count | Est. headline pp if all fixed |")
+ lines.append("| --- | ---: | ---: |")
+ for cls, n in counts.most_common():
+ pp = 100.0 * n / n_total if n_total else 0.0
+ lines.append(f"| `{cls}` | {n} | ~{pp:.1f} |")
+ lines.append("")
+ lines.append("## Top examples (≤20)")
+ lines.append("")
+ shown = 0
+ for cls, items in sorted(by_class.items(), key=lambda kv: -len(kv[1])):
+ for row in items[: max(1, 20 // max(1, len(by_class)))]:
+ if shown >= 20:
+ break
+ cat = CATEGORY_NAMES.get(int(row.get("category") or 0), "?")
+ lines.append(
+ f"- **{cls}** / {cat}: Q={row.get('question')!r} "
+ f"gold={row.get('gold')!r} pred={row.get('prediction')!r} "
+ f"reason={row.get('judge_reason')!r}"
+ )
+ shown += 1
+ if shown >= 20:
+ break
+
+ out = args.out or (RUNS_DIR / "sota-failure-taxonomy.md")
+ out.write_text("\n".join(lines) + "\n", encoding="utf-8")
+ print(out)
+ print(f"generation_share={gen_share:.3f} wrong={n_wrong}")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/locomo/test_sota_helpers.py b/benchmarks/locomo/test_sota_helpers.py
new file mode 100644
index 0000000..7d96525
--- /dev/null
+++ b/benchmarks/locomo/test_sota_helpers.py
@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+import unittest
+
+from locomo.sota import majority_vote, merge_contexts, plan_gap_fill, profile_defaults
+
+
+class SotaHelpersTests(unittest.TestCase):
+ def test_majority_vote(self) -> None:
+ self.assertEqual(majority_vote(["Yes", "yes", "No"]), "Yes")
+ self.assertEqual(majority_vote(["", " "]), "")
+
+ def test_profile_defaults(self) -> None:
+ self.assertEqual(profile_defaults("product")["sc_samples"], 1)
+ self.assertEqual(profile_defaults("sota")["sc_samples"], 5)
+ self.assertTrue(profile_defaults("sota")["gap_fill"])
+
+ def test_gap_fill_on_abstain(self) -> None:
+ plan = plan_gap_fill(
+ "Would Caroline pursue writing?",
+ "Not mentioned in the conversation",
+ {"text_context": "Caroline likes reading and wants to be a counselor"},
+ )
+ self.assertTrue(plan.needs_retry)
+
+ def test_merge_contexts(self) -> None:
+ merged = merge_contexts(
+ {"source_passages": ["a"], "text_context": "a"},
+ {"source_passages": ["b", "a"], "text_context": "b"},
+ )
+ self.assertEqual(merged["source_passages"], ["a", "b"])
+ self.assertIn("b", merged["text_context"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/benchmarks/scripts/run_sota_locomo10.sh b/benchmarks/scripts/run_sota_locomo10.sh
new file mode 100755
index 0000000..61e3628
--- /dev/null
+++ b/benchmarks/scripts/run_sota_locomo10.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+# Full LoCoMo10 SOTA leaderboard run (requires live BrainAPI + DeepSeek keys).
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+
+export BENCH_PROFILE="${BENCH_PROFILE:-sota}"
+export BENCH_SC_SAMPLES="${BENCH_SC_SAMPLES:-5}"
+export BENCH_GAP_FILL="${BENCH_GAP_FILL:-1}"
+# Preferred answerer/judge: deepseek-v4-flash (maintainer: stronger than gpt-4o here).
+export BENCH_SOTA_ANSWER_MODEL="${BENCH_SOTA_ANSWER_MODEL:-deepseek-v4-flash}"
+export BENCH_SOTA_JUDGE_MODEL="${BENCH_SOTA_JUDGE_MODEL:-deepseek-v4-flash}"
+
+RUN_INGEST="${RUN_INGEST:-sota-locomo10-ingest}"
+RUN_EVAL="${RUN_EVAL:-sota-locomo10-eval}"
+
+echo "== prompt-audit =="
+../.venv/bin/python -m locomo prompt-audit
+
+echo "== ingest all samples =="
+../.venv/bin/python -m locomo ingest --run "$RUN_INGEST" --concurrency 2 --granularity session
+
+echo "== evaluate (non-adversarial) =="
+../.venv/bin/python -m locomo evaluate \
+ --run "$RUN_EVAL" \
+ --max-facts 50 \
+ --max-passages 16 \
+ --historical-limit 16 \
+ --use-ppr \
+ --no-fact-filter \
+ --concurrency 2
+
+echo "Done. See benchmarks/runs/$RUN_EVAL/report.md"