BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
feat(benchmarks): SOTA LoCoMo harness and multi-suite REPORTS ledger
Christian785a507524b7
diff --git a/AGENTS.md b/AGENTS.mdindex e69de29..69adc92 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -0,0 +1,5 @@+# Agent guidance++Project agent guidance for benchmarks: see [`benchmarks/AGENTS.md`](benchmarks/AGENTS.md).++Benchmark results ledger (all suites): [`benchmarks/REPORTS.json`](benchmarks/REPORTS.json).diff --git a/benchmarks/.env.example b/benchmarks/.env.exampleindex 24405da..8d0c17e 100644--- a/benchmarks/.env.example+++ b/benchmarks/.env.example@@ -1,14 +1,34 @@BRAINAPI_URL=http://localhost:8000BRAINPAT_TOKEN=+# Dual profile: product (greedy) | sota (deepseek-v4-flash + SC + gap-fill)+# BENCH_PROFILE=product+# BENCH_PROFILE=sota+# BENCH_SC_SAMPLES=5+# BENCH_SC_TEMPERATURE=0.7+# BENCH_GAP_FILL=1+# Optional overrides (default SOTA answerer is deepseek-v4-flash):+# BENCH_SOTA_ANSWER_MODEL=deepseek-v4-flash+# BENCH_SOTA_JUDGE_MODEL=deepseek-v4-flash+# DeepSeek (OpenAI-compatible). Get a key at https://platform.deepseek.com/DEEPSEEK_API_KEY=BENCH_LLM_BASE_URL=https://api.deepseek.comBENCH_ANSWER_MODEL=deepseek-v4-flash-BENCH_JUDGE_MODEL=deepseek-v4-flash++# The judge must be a different model family from the answerer, or judge accuracy+# carries self-preference bias. Leave BENCH_JUDGE_MODEL unset to auto-select the+# Azure deployment below; set it explicitly to pin a judge.+# BENCH_JUDGE_MODEL=+# BENCH_JUDGE_BASE_URL=+# BENCH_JUDGE_API_KEY=++# Azure OpenAI judge (falls back to the repo-root AZURE_LARGE_LLM_* variables):+# BENCH_JUDGE_AZURE_ENDPOINT=+# BENCH_JUDGE_AZURE_KEY=+# BENCH_JUDGE_AZURE_API_VERSION=2024-12-01-preview# Optional aliases if you prefer another OpenAI-compatible provider:# OPENAI_API_KEY=# BENCH_LLM_BASE_URL=https://api.openai.com/v1# BENCH_ANSWER_MODEL=gpt-4o-mini-# BENCH_JUDGE_MODEL=gpt-4odiff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.mdindex e69de29..89c62fa 100644--- a/benchmarks/AGENTS.md+++ b/benchmarks/AGENTS.md@@ -0,0 +1,35 @@+# Benchmarks — agent guidance++## LoCoMo harness++The LoCoMo harness lives in `benchmarks/locomo/`. It talks to a running BrainAPI over HTTP only (no `src/` imports). Typical flow: `download` → `ingest` → `evaluate` → `report` via `./locomo.sh` or `.venv/bin/python -m locomo`.++## Results ledger (`REPORTS.json`)++`benchmarks/REPORTS.json` is the public BrainAPI results ledger: top published scores for **all** benchmark suites under `benchmarks.<suite>`.++- LoCoMo harness upserts `benchmarks.locomo` via `write_report` → `update_reports_json` when report `status` is `ok`.+- After any completed evaluate/report that produces a successful scored run, ensure the suite entry in `REPORTS.json` reflects it.+- If you write or patch `runs/<id>/report.json` manually, update `REPORTS.json` yourself (same schema; upsert by `run_id` under the suite).+- When adding a new suite, add a `benchmarks.<suite_id>` key (`name` + `leaderboard`) and upsert the same way.+- Skip failed/junk runs (`status: failed`, 0% judge, nonsense scores) — they do not belong on the leaderboard.+- Research caveats and checkpoint narrative live in `runs/CHECKPOINT_NOTES.md` (local), not in `REPORTS.json`.++## Tracks++| Track | Profile | Intent |+| --- | --- | --- |+| **SOTA** | `BENCH_PROFILE=sota` | Beat HyperMem-class LoCoMo (≥93% on full-10). Prefer **deepseek-v4-flash** answerer/judge. SC + gap-fill live in the harness, not product latency path. |+| **Product** | `BENCH_PROFILE=product` (default) | ADR-006 sub-second `/retrieve/context` path — greedy, no online LLM loops. |++Protocol and competitor bar: `docs/research/08-sota-locomo-protocol.md`. Historical checkpoint narrative: `runs/CHECKPOINT_NOTES.md`.++## Live stack++TUI / `brainapi start` may run from `~/.brainapi/source`, not this workspace. Sync or restart from the intended checkout before measuring, or scores will reflect stale code.++## Conventions++- Brain IDs are alphanumeric (`conv-26` → `locomoconv26`). Prefer clean brains (`locomoconv26clean`) for product/SOTA claims after Phase 0.+- Compare judge accuracy with `./locomo.sh compare` (paired McNemar). Do not A/B graph EvR until identical-config session-set agreement clears the ≥95% gate.+- Do not ingest QA annotations. Keep answer prompts prompt-audit clean (`./locomo.sh prompt-audit`).diff --git a/benchmarks/README.md b/benchmarks/README.mdindex a4c06e2..4d9a950 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -65,6 +65,10 @@ Reports land in `runs/<run_id>/`:- `manifest.json` — run config- `report.md` / `report.json` — aggregated metrics+## Results ledger++[`REPORTS.json`](REPORTS.json) is the shared BrainAPI benchmark ledger (multi-suite). LoCoMo writes the `locomo` entry; successful `evaluate` / `report` upserts automatically. See [`AGENTS.md`](AGENTS.md).+## Commands| Command | Purpose |@@ -74,9 +78,11 @@ Reports land in `runs/<run_id>/`:| `smoke` | One-shot ingest + retrieve health check || `ingest` | Push sessions (or turns) into BrainAPI || `answer-once` | Retrieve + answer a single question |-| `selftest-metrics` | Local F1 / BLEU-1 sanity checks |+| `selftest-metrics` | Local metric, channel-split and record-construction checks |+| `prompt-audit` | Fail if the answer prompt shares an n-gram with any gold answer || `evaluate` | Retrieve → answer → judge for QA items || `report` | Aggregate `answers.jsonl` into markdown/JSON |+| `compare` | Paired exact-McNemar comparison between two run arms |### Useful flags@@ -99,19 +105,84 @@ Reports land in `runs/<run_id>/`:- `--concurrency 2`- `--run <id>`+**compare**++- `--baseline <run>` / `--candidate <run>`, both repeatable — repeat to pool several runs of the same configuration into one arm+- `--include-adversarial`+- `--json`+## Scoring- **Headline number**: LLM-judge accuracy excluding category 5 (adversarial), matching common LoCoMo memory-system reporting.- Also reported: overall accuracy, per-category breakdown, mean F1, mean BLEU-1, retrieval latency p50/p95, total LLM tokens.- Categories: `1` multi-hop, `2` temporal, `3` open-domain, `4` single-hop, `5` adversarial.+- Gold for adversarial questions is read from `adversarial_answer` when `answer` is absent. F1 and BLEU-1 are `null` (unscorable) when gold is empty rather than a perfect 1.0.+- Evidence-session recall is reported for the **graph** and **passage** channels separately as well as combined, so the event graph's contribution is visible on its own.++### Comparing two runs++Judge accuracy is noisy: 11–12% of questions flip between identical-config runs at+`temperature=0`. **Passage** retrieval metrics are deterministic for a fixed brain+and config. **Graph** session sets are a measurement only after identical-config+agreement clears the **≥95%** gate (see `compare` graph-session stability output).+Do not A/B graph EvR below that gate.++```bash+./locomo.sh compare --baseline locomo-conv26-push75-b --candidate locomo-conv26-push75-c+./locomo.sh compare \+ --baseline locomo-conv26-push75-a --baseline locomo-conv26-push75-b \+ --candidate locomo-conv26-push75-c --candidate locomo-conv26-push75-d++# Checkpoint A — identical-config graph stability (gate ≥95% session-set agreement)+./locomo.sh evaluate --sample conv-26 --brain locomoconv26clean \+ --run graph-stable-a --historical-limit 16 --max-passages 16 \+ --max-facts 50 --use-ppr --no-sufficiency-retry+./locomo.sh evaluate --sample conv-26 --brain locomoconv26clean \+ --run graph-stable-b --historical-limit 16 --max-passages 16 \+ --max-facts 50 --use-ppr --no-sufficiency-retry+./locomo.sh compare --baseline graph-stable-a --candidate graph-stable-b+```++This reports the exact McNemar p-value with the flip table (how many questions moved+to correct and how many to wrong), overall and per category. Two independent Wilson+intervals are not a substitute: they discard the pairing and most of the power.+When both arms are single runs, `compare` also prints graph-session set agreement+vs the 95% gate.-Defaults use **DeepSeek** (override in `.env`):+### Models++Defaults use **DeepSeek** for the answerer (override in `.env`):- Base URL: `https://api.deepseek.com` (`BENCH_LLM_BASE_URL`)-- Answerer + judge: `deepseek-v4-flash` (`BENCH_ANSWER_MODEL` / `BENCH_JUDGE_MODEL`)+- Answerer: `deepseek-v4-flash` (`BENCH_ANSWER_MODEL`)- API key: `DEEPSEEK_API_KEY` (or `OPENAI_API_KEY` for other providers)-Other OpenAI-compatible providers work the same way — set `BENCH_LLM_BASE_URL`, model ids, and the key.+The judge is configured independently and should be a **different model family** from+the answerer, otherwise judge accuracy carries self-preference bias. `evaluate` prints a+warning and records `judge_shares_answer_family` in the manifest when the families match.++- `BENCH_JUDGE_MODEL`, `BENCH_JUDGE_BASE_URL`, `BENCH_JUDGE_API_KEY` for any OpenAI-compatible provider+- `BENCH_JUDGE_AZURE_ENDPOINT` / `BENCH_JUDGE_AZURE_KEY` / `BENCH_JUDGE_AZURE_API_VERSION` for Azure OpenAI; the repo-root `AZURE_LARGE_LLM_*` variables are picked up automatically when they are exported and no judge model is set explicitly+- With no judge configuration at all, the judge falls back to the answerer's provider and model, and the run is marked as sharing a family++Other OpenAI-compatible providers work the same way — set the base URL, model ids, and the key.++## Reproducibility++Every `ingest` and `evaluate` manifest records the git SHA and dirty flag, SHA-256 of the+answer and judge prompts, SHA-256 of the dataset file, and the resolved answerer/judge+model ids, families and providers. A prompt edit is a measurement-instrument change: it+changes `answer_prompt_sha256`, and any comparison across that boundary is invalid.++`prompt-audit` guards against tuning the answer prompt on gold answers:++```bash+./locomo.sh prompt-audit # fails if any 3-gram of ANSWER_SYSTEM appears in a gold answer+```++`report` de-duplicates answers by `(sample_id, qa_index)`, counts errored rows, and marks+a run that failed wholesale as `status: failed` with a banner, so a broken run can no+longer render as a clean 0.0%.## Cost and time expectationsdiff --git a/benchmarks/REPORTS.json b/benchmarks/REPORTS.jsonnew file mode 100644index 0000000..d5d79f0--- /dev/null+++ b/benchmarks/REPORTS.json@@ -0,0 +1,164 @@+{+ "schema_version": 2,+ "updated_at": "2026-07-29T18:51:58.401379+00:00",+ "description": "BrainAPI benchmark results. Top published scores across suites. Updated when a suite evaluate/report completes successfully.",+ "benchmarks": {+ "locomo": {+ "name": "LoCoMo",+ "leaderboard": [+ {+ "run_id": "oldbrain-recheck-a",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8552631578947368,+ "headline_judge_accuracy_pct": 85.53,+ "answerable_rate": 0.9473684210526315,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/oldbrain-recheck-a/report.json",+ "recorded_at": "2026-07-28T22:05:39.768798+00:00"+ },+ {+ "run_id": "phase-sota-d1-conv26",+ "track": "sota",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8486842105263158,+ "headline_judge_accuracy_pct": 84.87,+ "answerable_rate": 0.9407894736842105,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "sota",+ "brain": "locomoconv26clean",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/phase-sota-d1-conv26/report.json",+ "recorded_at": "2026-07-29T18:51:58.400341+00:00"+ },+ {+ "run_id": "clean-brain-eval-a",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8355263157894737,+ "headline_judge_accuracy_pct": 83.55,+ "answerable_rate": 0.9539473684210527,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26clean",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/clean-brain-eval-a/report.json",+ "recorded_at": "2026-07-28T21:29:30.713806+00:00"+ },+ {+ "run_id": "graph-stable-b",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8355263157894737,+ "headline_judge_accuracy_pct": 83.55,+ "answerable_rate": 0.9671052631578947,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26clean",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/graph-stable-b/report.json",+ "recorded_at": "2026-07-29T10:59:22.348091+00:00"+ },+ {+ "run_id": "clean-brain-a",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8289473684210527,+ "headline_judge_accuracy_pct": 82.89,+ "answerable_rate": 0.9473684210526315,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26clean",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/clean-brain-a/report.json",+ "recorded_at": "2026-07-28T21:57:06.650866+00:00"+ },+ {+ "run_id": "clean-brain-b",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8289473684210527,+ "headline_judge_accuracy_pct": 82.89,+ "answerable_rate": 0.9473684210526315,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26clean",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/clean-brain-b/report.json",+ "recorded_at": "2026-07-28T22:13:25.651713+00:00"+ },+ {+ "run_id": "baseline-clean-a",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8223684210526315,+ "headline_judge_accuracy_pct": 82.24,+ "answerable_rate": 0.9473684210526315,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/baseline-clean-a/report.json",+ "recorded_at": "2026-07-28T18:18:19.581660+00:00"+ },+ {+ "run_id": "phase-d1-paths-a",+ "track": "product",+ "scope": "conv-26",+ "samples": [+ "conv-26"+ ],+ "n_non_adversarial": 152,+ "headline_judge_accuracy": 0.8092105263157895,+ "headline_judge_accuracy_pct": 80.92,+ "answerable_rate": 0.9671052631578947,+ "answer_model": "deepseek-v4-flash",+ "judge_model": "deepseek-v4-flash",+ "bench_profile": "product",+ "brain": "locomoconv26clean",+ "git_sha": "649f5e569de8894c05eec5f1f423024be5e9b6a4",+ "report_path": "runs/phase-d1-paths-a/report.json",+ "recorded_at": "2026-07-29T16:26:21.919353+00:00"+ }+ ]+ }+ }+}diff --git a/benchmarks/locomo/answer.py b/benchmarks/locomo/answer.pyindex 36d6764..a741aa5 100644--- a/benchmarks/locomo/answer.py+++ b/benchmarks/locomo/answer.py@@ -4,10 +4,11 @@ import timefrom dataclasses import dataclassfrom typing import Any-from openai import OpenAI+from openai import AzureOpenAI, OpenAIfrom locomo.config import Settingsfrom locomo.prompts import build_answer_messages+from locomo.sota import majority_vote@dataclass@@ -18,40 +19,110 @@ class AnswerResult:prompt_tokens: int | Nonecompletion_tokens: int | Nonetotal_tokens: int | None+ sc_samples_used: int = 1-def _make_openai_client(settings: Settings) -> OpenAI:+def _make_openai_client(settings: Settings) -> OpenAI | AzureOpenAI:settings.require_llm()+ if settings.answer_azure_endpoint:+ return AzureOpenAI(+ api_key=settings.openai_api_key,+ azure_endpoint=settings.answer_azure_endpoint,+ api_version=settings.answer_azure_api_version,+ )kwargs: dict[str, Any] = {"api_key": settings.openai_api_key}if settings.llm_base_url:kwargs["base_url"] = settings.llm_base_urlreturn OpenAI(**kwargs)+def _one_completion(+ client: OpenAI | AzureOpenAI,+ *,+ model_name: str,+ messages: list[dict[str, str]],+ temperature: float,+) -> tuple[str, Any]:+ response = client.chat.completions.create(+ model=model_name,+ messages=messages,+ temperature=temperature,+ )+ content = (response.choices[0].message.content or "").strip()+ return content, response++def answer_question(settings: Settings,question: str,context: dict[str, Any],*,model: str | None = None,+ sc_samples: int | None = None,+ sc_temperature: float | None = None,) -> AnswerResult:client = _make_openai_client(settings)model_name = model or settings.answer_modelmessages = build_answer_messages(question, context)- started = time.perf_counter()- response = client.chat.completions.create(- model=model_name,- messages=messages,- temperature=0,+ n = max(1, int(sc_samples if sc_samples is not None else settings.sc_samples))+ temperature = (+ float(sc_temperature)+ if sc_temperature is not None+ else (settings.sc_temperature if n > 1 else 0.0))+ started = time.perf_counter()+ prompt_tokens = 0+ completion_tokens = 0+ total_tokens = 0+ samples: list[str] = []+ last_response: Any = None++ for _ in range(n):+ content, response = _one_completion(+ client,+ model_name=model_name,+ messages=messages,+ temperature=0.0 if n == 1 else temperature,+ )+ last_response = response+ if response.usage:+ prompt_tokens += getattr(response.usage, "prompt_tokens", 0) or 0+ completion_tokens += getattr(response.usage, "completion_tokens", 0) or 0+ total_tokens += getattr(response.usage, "total_tokens", 0) or 0+ if content:+ samples.append(content)++ content = majority_vote(samples) if samples else ""++ if not content:+ retry_messages = messages + [+ {+ "role": "user",+ "content": (+ "Your previous reply was empty. Answer the question now using the "+ "retrieved context. Prefer a short factual list or phrase."+ ),+ }+ ]+ content, response = _one_completion(+ client,+ model_name=model_name,+ messages=retry_messages,+ temperature=0,+ )+ last_response = response+ if response.usage:+ prompt_tokens += getattr(response.usage, "prompt_tokens", 0) or 0+ completion_tokens += getattr(response.usage, "completion_tokens", 0) or 0+ total_tokens += getattr(response.usage, "total_tokens", 0) or 0+latency_ms = (time.perf_counter() - started) * 1000- content = (response.choices[0].message.content or "").strip()- usage = response.usagereturn AnswerResult(answer=content,- model=model_name,+ model=getattr(last_response, "model", None) or model_name,latency_ms=latency_ms,- prompt_tokens=getattr(usage, "prompt_tokens", None) if usage else None,- completion_tokens=getattr(usage, "completion_tokens", None) if usage else None,- total_tokens=getattr(usage, "total_tokens", None) if usage else None,+ prompt_tokens=prompt_tokens or None,+ completion_tokens=completion_tokens or None,+ total_tokens=total_tokens or None,+ sc_samples_used=n,)diff --git a/benchmarks/locomo/cli.py b/benchmarks/locomo/cli.pyindex 9f3fcb8..c3590d1 100644--- a/benchmarks/locomo/cli.py+++ b/benchmarks/locomo/cli.py@@ -3,6 +3,7 @@ from __future__ import annotationsimport argparseimport jsonfrom pathlib import Path+from typing import Anyfrom rich.console import Consolefrom rich.table import Table@@ -18,10 +19,17 @@ from locomo.dataset import (load_dataset,resolve_samples,)-from locomo.evaluate import evaluate_samples-from locomo.ingest import ensure_run_dir, ingest_samples, write_manifest-from locomo.metrics import selftest_metrics-from locomo.report import print_report_table, write_report+from locomo.evaluate import evaluate_samples, selftest_records+from locomo.ingest import ensure_run_dir, ingest_samples, load_jsonl, write_manifest+from locomo.metrics import (+ compare_arms,+ retrieval_arm_summary,+ selftest_metrics,+ tokenize,+)+from locomo.prompts import ANSWER_SYSTEM+from locomo.provenance import build_provenance+from locomo.report import print_comparison, print_report_table, write_reportconsole = Console()@@ -99,8 +107,8 @@ def cmd_ingest(args: argparse.Namespace, settings: Settings) -> int:"limit_sessions": args.limit_sessions,"dry_run": args.dry_run,"brainapi_url": settings.brainapi_url,- "answer_model": settings.answer_model,- "judge_model": settings.judge_model,+ "brain_override": args.brain,+ **build_provenance(settings),},)ingest_samples(@@ -113,6 +121,7 @@ def cmd_ingest(args: argparse.Namespace, settings: Settings) -> int:dry_run=args.dry_run,resume=not args.no_resume,task_timeout_s=args.timeout,+ brain_override=args.brain,)console.print(f"[green]Run directory:[/green] {run_dir}")return 0@@ -141,7 +150,7 @@ def cmd_answer_once(args: argparse.Namespace, settings: Settings) -> int:def cmd_selftest_metrics(args: argparse.Namespace, settings: Settings) -> int:- errors = selftest_metrics()+ errors = selftest_metrics() + selftest_records()if errors:for err in errors:console.print(f"[red]FAIL[/red] {err}")@@ -155,6 +164,14 @@ def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:samples = resolve_samples(settings, sample_ids)run_id, run_dir = ensure_run_dir(settings, args.run)categories = _parse_categories(args.categories)+ if settings.judge_shares_answer_family:+ console.print(+ f"[yellow]Warning: judge model ({settings.judge_model}) is the same "+ f"family as the answer model ({settings.answer_model}); judge accuracy "+ "carries self-preference bias. Set BENCH_JUDGE_MODEL (with "+ "BENCH_JUDGE_BASE_URL / BENCH_JUDGE_API_KEY) to a different "+ "family.[/yellow]"+ )write_manifest(run_dir,{@@ -166,9 +183,14 @@ def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:"categories": sorted(categories) if categories else None,"limit": args.limit,"brainapi_url": settings.brainapi_url,- "answer_model": settings.answer_model,- "judge_model": settings.judge_model,+ "brain_override": args.brain,+ **build_provenance(settings),"historical_limit": args.historical_limit,+ "max_passages": args.max_passages,+ "max_facts": args.max_facts,+ "apply_fact_filter": not args.no_fact_filter,+ "use_ppr": args.use_ppr,+ "sufficiency_retry": args.sufficiency_retry,},)evaluate_samples(@@ -181,6 +203,12 @@ def cmd_evaluate(args: argparse.Namespace, settings: Settings) -> int:limit=args.limit,resume=not args.no_resume,historical_limit=args.historical_limit,+ max_passages=args.max_passages,+ max_facts=args.max_facts,+ apply_fact_filter=not args.no_fact_filter,+ use_ppr=args.use_ppr,+ sufficiency_retry=args.sufficiency_retry,+ brain_override=args.brain,)report = write_report(run_dir)print_report_table(report)@@ -200,6 +228,79 @@ def cmd_report(args: argparse.Namespace, settings: Settings) -> int:return 0+def _load_run_answers(settings: Settings, run_id: str) -> list[dict[str, Any]]:+ run_dir = settings.runs_dir / run_id+ answers_path = run_dir / "answers.jsonl"+ if not answers_path.exists():+ raise SystemExit(f"No answers.jsonl in run: {run_dir}")+ rows = load_jsonl(answers_path)+ latest: dict[tuple[str, int], dict[str, Any]] = {}+ for row in rows:+ if row.get("error"):+ continue+ latest[(str(row.get("sample_id")), int(row.get("qa_index") or 0))] = row+ return list(latest.values())+++def cmd_compare(args: argparse.Namespace, settings: Settings) -> int:+ baseline = [_load_run_answers(settings, run) for run in args.baseline]+ candidate = [_load_run_answers(settings, run) for run in args.candidate]+ comparison = compare_arms(+ baseline, candidate, skip_adversarial=not args.include_adversarial+ )+ retrieval = {+ "baseline": {"labels": args.baseline, **retrieval_arm_summary(baseline)},+ "candidate": {"labels": args.candidate, **retrieval_arm_summary(candidate)},+ }+ print_comparison(+ comparison,+ " + ".join(args.baseline),+ " + ".join(args.candidate),+ retrieval,+ )+ if args.json:+ console.print_json(+ json.dumps({"comparison": comparison, "retrieval": retrieval})+ )+ return 0+++def cmd_prompt_audit(args: argparse.Namespace, settings: Settings) -> int:+ dataset = load_dataset(settings.dataset_path)+ prompt_tokens = tokenize(ANSWER_SYSTEM)+ n = args.ngram+ prompt_ngrams = {+ tuple(prompt_tokens[i : i + n])+ for i in range(0, max(0, len(prompt_tokens) - n + 1))+ }+ hits: list[tuple[str, int, str]] = []+ for sample in dataset:+ sample_id = str(sample.get("sample_id"))+ for idx, qa in enumerate(sample.get("qa") or []):+ gold = qa.get("answer")+ if gold is None:+ gold = qa.get("adversarial_answer")+ gold_tokens = tokenize(str(gold or ""))+ for i in range(0, max(0, len(gold_tokens) - n + 1)):+ gram = tuple(gold_tokens[i : i + n])+ if gram in prompt_ngrams:+ hits.append((sample_id, idx, " ".join(gram)))+ break+ if hits:+ console.print(+ f"[red]FAIL[/red] answer prompt shares a {n}-gram with "+ f"{len(hits)} gold answers"+ )+ for sample_id, idx, gram in hits[:20]:+ console.print(f" {sample_id}::{idx} {gram!r}")+ return 1+ console.print(+ f"[green]prompt-audit passed[/green] no {n}-gram of the answer prompt "+ f"appears in any gold answer ({len(dataset)} conversations)"+ )+ return 0++def build_parser() -> argparse.ArgumentParser:parser = argparse.ArgumentParser(prog="python -m locomo",@@ -231,6 +332,11 @@ def build_parser() -> argparse.ArgumentParser:help="Sample id (repeatable). Default: all samples.",)p_ingest.add_argument("--run", default=None, help="Existing or new run id")+ p_ingest.add_argument(+ "--brain",+ default=None,+ help="Override the derived brain id (single --sample only)",+ )p_ingest.add_argument("--granularity", choices=("session", "turn"), default="session")@@ -252,6 +358,11 @@ def build_parser() -> argparse.ArgumentParser:p_eval = sub.add_parser("evaluate", help="Answer and score LoCoMo QA")p_eval.add_argument("--sample", action="append")p_eval.add_argument("--run", default=None)+ p_eval.add_argument(+ "--brain",+ default=None,+ help="Override the derived brain id (single --sample only)",+ )p_eval.add_argument("--concurrency", type=int, default=2)p_eval.add_argument("--categories", default=None, help="e.g. 1,2,4")p_eval.add_argument("--limit", type=int, default=None)@@ -262,6 +373,11 @@ def build_parser() -> argparse.ArgumentParser:)p_eval.add_argument("--no-resume", action="store_true")p_eval.add_argument("--historical-limit", type=int, default=10)+ p_eval.add_argument("--max-passages", type=int, default=8)+ p_eval.add_argument("--max-facts", type=int, default=40)+ p_eval.add_argument("--no-fact-filter", action="store_true")+ p_eval.add_argument("--use-ppr", action="store_true")+ p_eval.add_argument("--sufficiency-retry", action="store_true")p_eval.set_defaults(func=cmd_evaluate)p_report = sub.add_parser("report", help="Rebuild report from a run directory")@@ -269,6 +385,22 @@ def build_parser() -> argparse.ArgumentParser:p_report.add_argument("--json", action="store_true")p_report.set_defaults(func=cmd_report)+ p_compare = sub.add_parser(+ "compare", help="Paired McNemar comparison between two run arms"+ )+ p_compare.add_argument("--baseline", action="append", required=True)+ p_compare.add_argument("--candidate", action="append", required=True)+ p_compare.add_argument("--include-adversarial", action="store_true")+ p_compare.add_argument("--json", action="store_true")+ p_compare.set_defaults(func=cmd_compare)++ p_audit = sub.add_parser(+ "prompt-audit",+ help="Fail if the answer prompt shares an n-gram with any gold answer",+ )+ p_audit.add_argument("--ngram", type=int, default=3)+ p_audit.set_defaults(func=cmd_prompt_audit)+return parserdiff --git a/benchmarks/locomo/client.py b/benchmarks/locomo/client.pyindex 8951fea..51837f0 100644--- a/benchmarks/locomo/client.py+++ b/benchmarks/locomo/client.py@@ -137,11 +137,23 @@ class BrainAPIClient:brain_id: str,*,historical_limit: int = 10,+ max_passages: int = 8,+ max_facts: int = 40,+ apply_fact_filter: bool = True,+ use_ppr: bool = False,+ sufficiency_retry: bool = False,+ cross_event_bridges: int = 3,) -> TimedResult:- body = {+ body: dict[str, Any] = {"text": text,"brain_id": brain_id,"historical_limit": historical_limit,+ "max_passages": max_passages,+ "max_facts": max_facts,+ "apply_fact_filter": apply_fact_filter,+ "use_ppr": use_ppr,+ "sufficiency_retry": sufficiency_retry,+ "cross_event_bridges": cross_event_bridges,}return self._request("POST", "/retrieve/context", json=body)diff --git a/benchmarks/locomo/config.py b/benchmarks/locomo/config.pyindex d2f39cf..a859090 100644--- a/benchmarks/locomo/config.py+++ b/benchmarks/locomo/config.py@@ -15,7 +15,33 @@ DEFAULT_DATASET_URL = (DEFAULT_DATASET_PATH = DATA_DIR / "locomo10.json"DEFAULT_LLM_BASE_URL = "https://api.deepseek.com"DEFAULT_ANSWER_MODEL = "deepseek-v4-flash"-DEFAULT_JUDGE_MODEL = "deepseek-v4-flash"+DEFAULT_SOTA_ANSWER_MODEL = "deepseek-v4-flash"+DEFAULT_AZURE_JUDGE_MODEL = "gpt-4o"+DEFAULT_AZURE_API_VERSION = "2024-12-01-preview"+DEFAULT_BENCH_PROFILE = "product"++_MODEL_FAMILY_PREFIXES = {+ "deepseek": "deepseek",+ "gpt": "openai",+ "o1": "openai",+ "o3": "openai",+ "o4": "openai",+ "chatgpt": "openai",+ "claude": "anthropic",+ "gemini": "google",+ "grok": "xai",+ "qwen": "qwen",+ "llama": "meta",+ "mistral": "mistral",+}+++def model_family(model: str) -> str:+ name = (model or "").strip().lower().rsplit("/", 1)[-1]+ for prefix, family in _MODEL_FAMILY_PREFIXES.items():+ if name.startswith(prefix):+ return family+ return name.split("-", 1)[0] or "unknown"TERMINAL_TASK_STATUSES = frozenset({"completed", "failed", "partial_failed"})CATEGORY_NAMES = {@@ -35,9 +61,19 @@ class Settings:llm_base_url: str | Noneanswer_model: strjudge_model: str+ judge_api_key: str+ judge_base_url: str | None+ judge_azure_endpoint: str | None+ judge_azure_api_version: strdataset_path: Pathdataset_url: strruns_dir: Path+ bench_profile: str+ sc_samples: int+ sc_temperature: float+ gap_fill: bool+ answer_azure_endpoint: str | None = None+ answer_azure_api_version: str = DEFAULT_AZURE_API_VERSION@classmethoddef load(cls, env_file: Path | None = None) -> "Settings":@@ -48,26 +84,114 @@ class Settings:else:load_dotenv(override=False)+ from locomo.sota import profile_defaults++ profile = (+ os.getenv("BENCH_PROFILE", DEFAULT_BENCH_PROFILE) or DEFAULT_BENCH_PROFILE+ ).strip().lower()+ if profile not in {"product", "sota"}:+ profile = DEFAULT_BENCH_PROFILE+ defaults = profile_defaults(profile)+api_key = (os.getenv("DEEPSEEK_API_KEY")or os.getenv("OPENAI_API_KEY")or "")+ llm_base_url = os.getenv("BENCH_LLM_BASE_URL", DEFAULT_LLM_BASE_URL) or None+ answer_model = os.getenv("BENCH_ANSWER_MODEL", DEFAULT_ANSWER_MODEL)++ azure_endpoint = (+ os.getenv("BENCH_JUDGE_AZURE_ENDPOINT")+ or os.getenv("AZURE_LARGE_LLM_ENDPOINT")+ or ""+ ).rstrip("/")+ azure_key = (+ os.getenv("BENCH_JUDGE_AZURE_KEY")+ or os.getenv("AZURE_LARGE_LLM_SUBSCRIPTION_KEY")+ or ""+ )+ azure_available = bool(azure_endpoint and azure_key)++ answer_azure_endpoint: str | None = None+ if profile == "sota":+ # Maintainer: deepseek-v4-flash is the preferred SOTA answerer (stronger+ # than gpt-4o on this stack). Only override via BENCH_SOTA_* envs.+ sota_answer = os.getenv("BENCH_SOTA_ANSWER_MODEL", "").strip()+ if sota_answer:+ answer_model = sota_answer+ sota_base = os.getenv("BENCH_SOTA_LLM_BASE_URL", "").strip()+ if sota_base:+ llm_base_url = sota_base+ else:+ answer_model = os.getenv(+ "BENCH_ANSWER_MODEL", DEFAULT_SOTA_ANSWER_MODEL+ )++ judge_model = os.getenv("BENCH_JUDGE_MODEL", "")+ judge_base_url = os.getenv("BENCH_JUDGE_BASE_URL", "")+ judge_api_key = os.getenv("BENCH_JUDGE_API_KEY", "")+ use_azure_judge = (+ azure_available and not judge_model and not judge_base_url+ )+ if use_azure_judge:+ judge_model = os.getenv(+ "AZURE_LARGE_LLM_MODEL", DEFAULT_AZURE_JUDGE_MODEL+ )+ if profile == "sota":+ sota_judge = os.getenv("BENCH_SOTA_JUDGE_MODEL", "").strip()+ if sota_judge:+ judge_model = sota_judge+ use_azure_judge = False+ elif not os.getenv("BENCH_JUDGE_MODEL"):+ judge_model = answer_model+ use_azure_judge = False+ judge_base_url = llm_base_url or ""+ judge_api_key = api_keyreturn cls(brainapi_url=os.getenv("BRAINAPI_URL", "http://localhost:8000").rstrip("/"),brainpat_token=os.getenv("BRAINPAT_TOKEN", ""),openai_api_key=api_key,- llm_base_url=os.getenv("BENCH_LLM_BASE_URL", DEFAULT_LLM_BASE_URL) or None,- answer_model=os.getenv("BENCH_ANSWER_MODEL", DEFAULT_ANSWER_MODEL),- judge_model=os.getenv("BENCH_JUDGE_MODEL", DEFAULT_JUDGE_MODEL),+ llm_base_url=llm_base_url,+ answer_model=answer_model,+ judge_model=judge_model or answer_model,+ judge_api_key=(+ azure_key if use_azure_judge else (judge_api_key or api_key)+ ),+ judge_base_url=(+ None if use_azure_judge else (judge_base_url or llm_base_url)+ ),+ judge_azure_endpoint=azure_endpoint if use_azure_judge else None,+ judge_azure_api_version=os.getenv(+ "BENCH_JUDGE_AZURE_API_VERSION",+ os.getenv("AZURE_LARGE_LLM_API_VERSION", DEFAULT_AZURE_API_VERSION),+ ),dataset_path=Path(os.getenv("BENCH_DATASET_PATH", str(DEFAULT_DATASET_PATH))),dataset_url=os.getenv("BENCH_DATASET_URL", DEFAULT_DATASET_URL),runs_dir=Path(os.getenv("BENCH_RUNS_DIR", str(RUNS_DIR))),+ bench_profile=str(defaults["bench_profile"]),+ sc_samples=max(1, int(defaults["sc_samples"])),+ sc_temperature=float(defaults["sc_temperature"]),+ gap_fill=bool(defaults["gap_fill"]),+ answer_azure_endpoint=answer_azure_endpoint,+ answer_azure_api_version=os.getenv(+ "BENCH_JUDGE_AZURE_API_VERSION",+ os.getenv("AZURE_LARGE_LLM_API_VERSION", DEFAULT_AZURE_API_VERSION),+ ),)+ @property+ def judge_shares_answer_family(self) -> bool:+ return model_family(self.judge_model) == model_family(self.answer_model)++ def judge_provider(self) -> str:+ if self.judge_azure_endpoint:+ return "azure"+ return self.judge_base_url or "openai-default"+def require_brainapi(self) -> None:if not self.brainpat_token:raise SystemExit(@@ -80,3 +204,11 @@ class Settings:"DEEPSEEK_API_KEY (or OPENAI_API_KEY) is required for answer/judge ""LLM calls. Copy .env.example to .env and set it.")++ def require_judge_llm(self) -> None:+ if not self.judge_api_key:+ raise SystemExit(+ "No judge API key. Set BENCH_JUDGE_API_KEY (with BENCH_JUDGE_BASE_URL "+ "and BENCH_JUDGE_MODEL), or DEEPSEEK_API_KEY / OPENAI_API_KEY to reuse "+ "the answerer's provider."+ )diff --git a/benchmarks/locomo/evaluate.py b/benchmarks/locomo/evaluate.pyindex 6bbfb1e..52f1615 100644--- a/benchmarks/locomo/evaluate.py+++ b/benchmarks/locomo/evaluate.py@@ -1,8 +1,9 @@from __future__ import annotations+import reimport threadingfrom concurrent.futures import ThreadPoolExecutor, as_completed-from dataclasses import asdict, dataclass+from dataclasses import asdict, dataclass, fieldfrom datetime import datetime, timezonefrom pathlib import Pathfrom typing import Any@@ -16,11 +17,19 @@ from locomo.config import Settingsfrom locomo.dataset import brain_id_forfrom locomo.ingest import append_jsonl, load_jsonlfrom locomo.judge import judge_answer-from locomo.metrics import bleu1_score, f1_score+from locomo.metrics import evidence_coverage, overlap_scores+from locomo.prompts import enrich_paths_from_triples, flatten_path, flatten_triple+from locomo.sota import merge_contexts, plan_gap_fillconsole = Console()_write_lock = threading.Lock()+_GRAPH_CONTEXT_CAP = 60000+_TRIPLES_CAP = 500+_PASSAGE_PREFIX = "[passage] "+_SESSION_RE = re.compile(r"session_(\d+)", re.IGNORECASE)+_DIALOG_RE = re.compile(r"\bD(\d+):", re.IGNORECASE)+@dataclassclass AnswerRecord:@@ -35,8 +44,8 @@ class AnswerRecord:judge_correct: booljudge_reason: strjudge_raw: str- f1: float- bleu1: float+ f1: float | None+ bleu1: float | Noneretrieve_latency_ms: float | Noneanswer_latency_ms: float | Nonejudge_latency_ms: float | None@@ -44,15 +53,101 @@ class AnswerRecord:judge_model: str | Noneanswer_total_tokens: int | Nonejudge_total_tokens: int | None- text_context: str- error: str | None- timestamp: str+ graph_context: str+ triples: list[str] = field(default_factory=list)+ paths: list[str] = field(default_factory=list)+ source_passages: list[str] = field(default_factory=list)+ historical_context: list[str] = field(default_factory=list)+ retrieved_session_ids: list[str] = field(default_factory=list)+ retrieved_session_ids_graph: list[str] = field(default_factory=list)+ retrieved_session_ids_passages: list[str] = field(default_factory=list)+ context_truncated: dict[str, int] = field(default_factory=dict)+ is_adversarial: bool = False+ error: str | None = None+ timestamp: str = ""+++def _extract_session_ids(*blobs: Any) -> list[str]:+ found: list[str] = []+ seen: set[str] = set()+ for blob in blobs:+ if isinstance(blob, (list, tuple)):+ text = "\n".join(str(item) for item in blob)+ else:+ text = str(blob or "")+ for match in _SESSION_RE.finditer(text):+ sid = f"session_{match.group(1)}"+ if sid not in seen:+ seen.add(sid)+ found.append(sid)+ for match in _DIALOG_RE.finditer(text):+ sid = f"session_{match.group(1)}"+ if sid not in seen:+ seen.add(sid)+ found.append(sid)+ return found+++def _graph_channel(text_context: str, source_passages: list[str]) -> str:+ passage_block = "\n".join(f"{_PASSAGE_PREFIX}{p}" for p in source_passages)+ if passage_block and text_context.startswith(passage_block):+ return text_context[len(passage_block) :].lstrip("\n")+ return "\n".join(+ line+ for line in text_context.splitlines()+ if not line.startswith(_PASSAGE_PREFIX)+ )+++def extract_context_channels(context: dict[str, Any]) -> dict[str, Any]:+ context = enrich_paths_from_triples(context)+ source_passages = [str(p) for p in (context.get("source_passages") or []) if p]+ historical_context = [+ str(h) for h in (context.get("historical_context") or []) if h+ ]+ triples = [flatten_triple(t) for t in (context.get("triples") or []) if t]+ paths = [flatten_path(p) for p in (context.get("paths") or []) if p]+ graph_context = _graph_channel(+ str(context.get("text_context") or ""), source_passages+ )+ session_ids_graph = _extract_session_ids(graph_context, triples, paths)+ session_ids_passages = _extract_session_ids(source_passages, historical_context)+ truncated: dict[str, int] = {}+ if len(graph_context) > _GRAPH_CONTEXT_CAP:+ truncated["graph_context"] = len(graph_context)+ graph_context = graph_context[:_GRAPH_CONTEXT_CAP]+ if len(triples) > _TRIPLES_CAP:+ truncated["triples"] = len(triples)+ triples = triples[:_TRIPLES_CAP]+ return {+ "graph_context": graph_context,+ "triples": triples,+ "paths": paths,+ "source_passages": source_passages,+ "historical_context": historical_context,+ "retrieved_session_ids": _extract_session_ids(+ session_ids_passages, session_ids_graph+ ),+ "retrieved_session_ids_graph": session_ids_graph,+ "retrieved_session_ids_passages": session_ids_passages,+ "context_truncated": truncated,+ }def _utc_now() -> str:return datetime.now(timezone.utc).isoformat()+def qa_gold(qa: dict[str, Any]) -> tuple[str, bool]:+ answer = qa.get("answer")+ if answer is not None and str(answer).strip():+ return str(answer), int(qa.get("category") or 0) == 5+ adversarial = qa.get("adversarial_answer")+ if adversarial is not None:+ return str(adversarial), True+ return "", int(qa.get("category") or 0) == 5++def qa_key(sample_id: str, qa_index: int) -> str:return f"{sample_id}::{qa_index}"@@ -74,11 +169,14 @@ def iter_qa_jobs(categories: set[int] | None = None,skip_adversarial: bool = True,limit: int | None = None,+ brain_override: str | None = None,) -> list[dict[str, Any]]:+ if brain_override and len(samples) > 1:+ raise SystemExit("--brain can only be used with a single --sample")jobs: list[dict[str, Any]] = []for sample in samples:sample_id = str(sample["sample_id"])- brain_id = brain_id_for(sample_id)+ brain_id = brain_override or brain_id_for(sample_id)for idx, qa in enumerate(sample.get("qa") or []):cat = int(qa.get("category") or 0)if skip_adversarial and cat == 5:@@ -98,6 +196,120 @@ def iter_qa_jobs(return jobs+def selftest_records() -> list[str]:+ errors: list[str] = []+ passages = [+ "Conversation between Caroline and Melanie.\nSession id: session_9.\n"+ + ("filler " * 4000),+ "Session id: session_11. Caroline talked about the marathon.",+ ]+ context = {+ "text_context": "\n".join(+ [f"{_PASSAGE_PREFIX}{p}" for p in passages]+ + [+ "Melanie | PUT | keys | IN | slipper (session_3)",+ "[dossier] Melanie: runs charity races (session_3)",+ ]+ ),+ "source_passages": passages,+ "historical_context": ["Session id: session_11. Earlier chunk."],+ "triples": [+ {+ "identified_entity": "Melanie",+ "triple": ["Melanie", "PUT", "keys", "IN", "slipper"],+ }+ ],+ "paths": [+ {+ "hubs": ["event-a", "event-b"],+ "shared_entity_name": "Melanie",+ "legs": [+ "Actor: Melanie | MADE | Event: put keys | TARGETED | Target: slipper",+ "Actor: Melanie | MADE | Event: run | TARGETED | Target: race",+ ],+ }+ ],+ }+ channels = extract_context_channels(context)+ record = AnswerRecord(+ sample_id="conv-26",+ brain_id="locomoconv26",+ qa_index=0,+ question="Where did Melanie put the keys?",+ gold="In Melanie's slipper",+ prediction="In Melanie's slipper",+ category=4,+ evidence=["D3:2"],+ judge_correct=True,+ judge_reason="",+ judge_raw="",+ f1=1.0,+ bleu1=1.0,+ retrieve_latency_ms=1.0,+ answer_latency_ms=1.0,+ judge_latency_ms=1.0,+ answer_model="answer-model",+ judge_model="judge-model",+ answer_total_tokens=1,+ judge_total_tokens=1,+ error=None,+ timestamp=_utc_now(),+ **channels,+ )+ row = asdict(record)++ if _PASSAGE_PREFIX in row["graph_context"]:+ errors.append("graph_context must not contain passage lines")+ if "PUT" not in row["graph_context"]:+ errors.append("graph fact line missing from graph_context")+ if not row["triples"]:+ errors.append("triples must be logged as their own field")+ if "slipper" not in row["triples"][0]:+ errors.append("triples must be flattened to readable text")+ if not row["paths"]:+ errors.append("paths must be logged as their own field")+ if "via Melanie" not in row["paths"][0] or "-->" not in row["paths"][0]:+ errors.append("paths must flatten to via-entity leg chains")+ if row["retrieved_session_ids_graph"] != ["session_3"]:+ errors.append(+ f"graph channel session ids wrong: {row['retrieved_session_ids_graph']}"+ )+ if sorted(row["retrieved_session_ids_passages"]) != ["session_11", "session_9"]:+ errors.append(+ "passage channel session ids wrong: "+ f"{row['retrieved_session_ids_passages']}"+ )+ if sorted(row["retrieved_session_ids"]) != [+ "session_11",+ "session_3",+ "session_9",+ ]:+ errors.append("combined session ids should be the union of both channels")+ if evidence_coverage(row, "graph") != "full":+ errors.append("graph-only evidence should score full recall on the graph channel")+ if evidence_coverage(row, "passages") != "none":+ errors.append("graph-only evidence should score no recall on the passage channel")++ long_context = dict(context)+ long_context["text_context"] = "x" * (_GRAPH_CONTEXT_CAP + 10)+ long_context["source_passages"] = []+ capped = extract_context_channels(long_context)+ if len(capped["graph_context"]) != _GRAPH_CONTEXT_CAP:+ errors.append("graph_context should be capped")+ if capped["context_truncated"].get("graph_context") != _GRAPH_CONTEXT_CAP + 10:+ errors.append("truncation must be recorded with the original size")++ gold, adversarial = qa_gold(+ {"category": 5, "adversarial_answer": "No information available"}+ )+ if gold != "No information available" or not adversarial:+ errors.append("adversarial gold must be read from adversarial_answer")+ gold, adversarial = qa_gold({"category": 4, "answer": "7 May 2023"})+ if gold != "7 May 2023" or adversarial:+ errors.append("non-adversarial gold must be read from answer")+ return errors++def evaluate_samples(settings: Settings,samples: list[dict[str, Any]],@@ -109,6 +321,12 @@ def evaluate_samples(limit: int | None = None,resume: bool = True,historical_limit: int = 10,+ max_passages: int = 8,+ max_facts: int = 40,+ apply_fact_filter: bool = True,+ use_ppr: bool = False,+ sufficiency_retry: bool = False,+ brain_override: str | None = None,) -> list[AnswerRecord]:run_dir.mkdir(parents=True, exist_ok=True)answers_path = run_dir / "answers.jsonl"@@ -119,6 +337,7 @@ def evaluate_samples(categories=categories,skip_adversarial=skip_adversarial,limit=limit,+ brain_override=brain_override,)jobs = [j@@ -140,7 +359,7 @@ def evaluate_samples(def _one(job: dict[str, Any]) -> AnswerRecord:qa = job["qa"]question = str(qa.get("question") or "")- gold = str(qa.get("answer") if qa.get("answer") is not None else "")+ gold, is_adversarial = qa_gold(qa)category = int(qa.get("category") or 0)evidence = list(qa.get("evidence") or [])@@ -150,11 +369,46 @@ def evaluate_samples(question,job["brain_id"],historical_limit=historical_limit,+ max_passages=max_passages,+ max_facts=max_facts,+ apply_fact_filter=apply_fact_filter,+ use_ppr=use_ppr,+ sufficiency_retry=sufficiency_retry,)context = retrieved.data or {}- answered = answer_question(settings, question, context)+ retrieve_ms = retrieved.latency_ms+ draft = answer_question(settings, question, context)+ if settings.gap_fill:+ plan = plan_gap_fill(question, draft.answer, context)+ if plan.needs_retry:+ with BrainAPIClient(settings) as client:+ second = client.retrieve_context(+ plan.reformulated_query,+ job["brain_id"],+ historical_limit=historical_limit,+ max_passages=max_passages,+ max_facts=max_facts,+ apply_fact_filter=apply_fact_filter,+ use_ppr=use_ppr,+ sufficiency_retry=sufficiency_retry,+ )+ retrieve_ms = (retrieve_ms or 0) + (second.latency_ms or 0)+ context = merge_contexts(context, second.data or {})+ answered = answer_question(settings, question, context)+ else:+ answered = draft+ else:+ answered = draft+ channels = extract_context_channels(context)judged = judge_answer(- settings, question, gold, answered.answer+ settings,+ question,+ gold,+ answered.answer,+ adversarial=is_adversarial,+ )+ f1, bleu1 = overlap_scores(+ answered.answer, gold, adversarial=is_adversarial)record = AnswerRecord(sample_id=job["sample_id"],@@ -168,18 +422,19 @@ def evaluate_samples(judge_correct=judged.correct,judge_reason=judged.reason,judge_raw=judged.raw,- f1=f1_score(answered.answer, gold),- bleu1=bleu1_score(answered.answer, gold),- retrieve_latency_ms=retrieved.latency_ms,+ f1=f1,+ bleu1=bleu1,+ retrieve_latency_ms=retrieve_ms,answer_latency_ms=answered.latency_ms,judge_latency_ms=judged.latency_ms,answer_model=answered.model,judge_model=judged.model,answer_total_tokens=answered.total_tokens,judge_total_tokens=judged.total_tokens,- text_context=str(context.get("text_context") or "")[:4000],+ is_adversarial=is_adversarial,error=None,timestamp=_utc_now(),+ **channels,)except Exception as exc:record = AnswerRecord(@@ -194,8 +449,8 @@ def evaluate_samples(judge_correct=False,judge_reason="",judge_raw="",- f1=0.0,- bleu1=0.0,+ f1=None,+ bleu1=None,retrieve_latency_ms=None,answer_latency_ms=None,judge_latency_ms=None,@@ -203,9 +458,10 @@ def evaluate_samples(judge_model=None,answer_total_tokens=None,judge_total_tokens=None,- text_context="",+ is_adversarial=is_adversarial,error=str(exc),timestamp=_utc_now(),+ **extract_context_channels({}),)with _write_lock:diff --git a/benchmarks/locomo/ingest.py b/benchmarks/locomo/ingest.pyindex 4339bfd..9134e13 100644--- a/benchmarks/locomo/ingest.py+++ b/benchmarks/locomo/ingest.py@@ -79,15 +79,19 @@ def ingest_samples(dry_run: bool = False,resume: bool = True,task_timeout_s: float = 900.0,+ brain_override: str | None = None,) -> list[IngestRecord]:run_dir.mkdir(parents=True, exist_ok=True)ingest_path = run_dir / "ingest.jsonl"already_done = completed_unit_ids(ingest_path) if resume else set()+ if brain_override and len(samples) > 1:+ raise SystemExit("--brain can only be used with a single --sample")+jobs: list[dict[str, Any]] = []for sample in samples:sample_id = str(sample["sample_id"])- brain_id = brain_id_for(sample_id)+ brain_id = brain_override or brain_id_for(sample_id)for unit in iter_ingest_units(sample, granularity=granularity, limit_sessions=limit_sessions):diff --git a/benchmarks/locomo/judge.py b/benchmarks/locomo/judge.pyindex 2fead6d..d97d7d3 100644--- a/benchmarks/locomo/judge.py+++ b/benchmarks/locomo/judge.py@@ -6,7 +6,7 @@ import timefrom dataclasses import dataclassfrom typing import Any-from openai import OpenAI+from openai import AzureOpenAI, OpenAIfrom locomo.config import Settingsfrom locomo.prompts import build_judge_messages@@ -24,11 +24,17 @@ class JudgeResult:total_tokens: int | None-def _make_openai_client(settings: Settings) -> OpenAI:- settings.require_llm()- kwargs: dict[str, Any] = {"api_key": settings.openai_api_key}- if settings.llm_base_url:- kwargs["base_url"] = settings.llm_base_url+def _make_openai_client(settings: Settings) -> OpenAI | AzureOpenAI:+ settings.require_judge_llm()+ if settings.judge_azure_endpoint:+ return AzureOpenAI(+ api_key=settings.judge_api_key,+ azure_endpoint=settings.judge_azure_endpoint,+ api_version=settings.judge_azure_api_version,+ )+ kwargs: dict[str, Any] = {"api_key": settings.judge_api_key}+ if settings.judge_base_url:+ kwargs["base_url"] = settings.judge_base_urlreturn OpenAI(**kwargs)@@ -60,10 +66,13 @@ def judge_answer(prediction: str,*,model: str | None = None,+ adversarial: bool = False,) -> JudgeResult:client = _make_openai_client(settings)model_name = model or settings.judge_model- messages = build_judge_messages(question, gold, prediction)+ messages = build_judge_messages(+ question, gold, prediction, adversarial=adversarial+ )started = time.perf_counter()response = client.chat.completions.create(model=model_name,@@ -79,7 +88,7 @@ def judge_answer(correct=correct,reason=reason,raw=raw,- model=model_name,+ model=getattr(response, "model", None) or model_name,latency_ms=latency_ms,prompt_tokens=getattr(usage, "prompt_tokens", None) if usage else None,completion_tokens=getattr(usage, "completion_tokens", None) if usage else None,diff --git a/benchmarks/locomo/metrics.py b/benchmarks/locomo/metrics.pyindex dfecc15..a0884af 100644--- a/benchmarks/locomo/metrics.py+++ b/benchmarks/locomo/metrics.py@@ -1,12 +1,15 @@from __future__ import annotationsimport math+import reimport stringfrom collections import Counterfrom typing import Any, Iterable_ARTICLES = {"a", "an", "the"}+_DIALOG_EVIDENCE_RE = re.compile(r"^D(\d+)", re.IGNORECASE)+_SESSION_RE = re.compile(r"session_(\d+)", re.IGNORECASE)def normalize_answer(text: str) -> str:@@ -20,12 +23,12 @@ def tokenize(text: str) -> list[str]:return normalize_answer(text).split()-def f1_score(prediction: str, gold: str) -> float:+def f1_score(prediction: str, gold: str) -> float | None:pred_tokens = tokenize(prediction)gold_tokens = tokenize(gold)- if not pred_tokens and not gold_tokens:- return 1.0- if not pred_tokens or not gold_tokens:+ if not gold_tokens:+ return None+ if not pred_tokens:return 0.0common = Counter(pred_tokens) & Counter(gold_tokens)num_same = sum(common.values())@@ -36,11 +39,13 @@ def f1_score(prediction: str, gold: str) -> float:return 2 * precision * recall / (precision + recall)-def bleu1_score(prediction: str, gold: str) -> float:+def bleu1_score(prediction: str, gold: str) -> float | None:pred_tokens = tokenize(prediction)gold_tokens = tokenize(gold)- if not pred_tokens or not gold_tokens:- return 1.0 if not pred_tokens and not gold_tokens else 0.0+ if not gold_tokens:+ return None+ if not pred_tokens:+ return 0.0gold_counts = Counter(gold_tokens)match = 0for token in pred_tokens:@@ -52,6 +57,14 @@ def bleu1_score(prediction: str, gold: str) -> float:return precision * brevity+def overlap_scores(+ prediction: str, gold: str, *, adversarial: bool = False+) -> tuple[float | None, float | None]:+ if adversarial:+ return None, None+ return f1_score(prediction, gold), bleu1_score(prediction, gold)++def percentile(values: list[float], p: float) -> float | None:if not values:return None@@ -87,6 +100,117 @@ def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float | None, flreturn max(0.0, center - margin), min(1.0, center + margin)+def evidence_session_ids(evidence: Any) -> set[str]:+ sessions: set[str] = set()+ for item in evidence or []:+ text = str(item or "")+ match = _DIALOG_EVIDENCE_RE.match(text)+ if match:+ sessions.add(f"session_{match.group(1)}")+ continue+ for session_match in _SESSION_RE.finditer(text):+ sessions.add(f"session_{session_match.group(1)}")+ return sessions+++CHANNELS = ("graph", "passages")++_CHANNEL_ID_FIELD = {+ "combined": "retrieved_session_ids",+ "graph": "retrieved_session_ids_graph",+ "passages": "retrieved_session_ids_passages",+}+_CHANNEL_TEXT_FIELDS = {+ "combined": (+ "text_context",+ "graph_context",+ "triples",+ "paths",+ "source_passages",+ "historical_context",+ ),+ "graph": ("graph_context", "triples", "paths"),+ "passages": ("source_passages", "historical_context"),+}+++def _channel_blobs(row: dict[str, Any], channel: str) -> list[str]:+ blobs: list[str] = []+ for field in _CHANNEL_TEXT_FIELDS[channel]:+ value = row.get(field)+ if isinstance(value, (list, tuple)):+ blobs.append("\n".join(str(item) for item in value))+ elif value:+ blobs.append(str(value))+ return blobs+++def _scrape_session_ids(blobs: list[str]) -> set[str]:+ sessions: set[str] = set()+ for blob in blobs:+ for match in _SESSION_RE.finditer(blob):+ sessions.add(f"session_{match.group(1)}")+ for match in re.finditer(r"\bD(\d+):", blob, re.IGNORECASE):+ sessions.add(f"session_{match.group(1)}")+ return sessions+++def retrieved_session_ids(+ row: dict[str, Any], channel: str = "combined"+) -> set[str] | None:+ field = _CHANNEL_ID_FIELD[channel]+ explicit = row.get(field)+ if explicit is not None:+ sessions = {str(s) for s in explicit if s}+ if sessions or channel != "combined":+ return sessions+ if channel != "combined" and not any(+ f in row for f in _CHANNEL_TEXT_FIELDS[channel]+ ):+ return None+ return _scrape_session_ids(_channel_blobs(row, channel))+++def full_context_blob(row: dict[str, Any]) -> str:+ return "\n".join(_channel_blobs(row, "combined")).lower()+++def gold_in_context(row: dict[str, Any], threshold: float = 0.5) -> bool:+ gold_tokens = [t for t in tokenize(str(row.get("gold") or "")) if len(t) > 2]+ if not gold_tokens:+ return False+ blob = full_context_blob(row)+ hits = sum(1 for token in gold_tokens if token in blob)+ return hits / len(gold_tokens) >= threshold+++def evidence_coverage(row: dict[str, Any], channel: str = "combined") -> str | None:+ needed = evidence_session_ids(row.get("evidence"))+ if not needed:+ return None+ have = retrieved_session_ids(row, channel)+ if have is None:+ return None+ if needed <= have:+ return "full"+ if needed & have:+ return "partial"+ return "none"+++def _recall_bucket(items: list[dict[str, Any]], channel: str) -> dict[str, Any]:+ coverages = [evidence_coverage(r, channel) for r in items]+ with_evidence = [c for c in coverages if c is not None]+ n_ev = len(with_evidence)+ full = sum(1 for c in with_evidence if c == "full")+ partial = sum(1 for c in with_evidence if c in {"full", "partial"})+ return {+ "evidence_session_recall_full": (full / n_ev) if n_ev else None,+ "evidence_session_recall_partial": (partial / n_ev) if n_ev else None,+ "n_with_evidence": n_ev,+ }++def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:by_category: dict[int, list[dict[str, Any]]] = {}for row in rows:@@ -95,27 +219,49 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:def _bucket(items: list[dict[str, Any]]) -> dict[str, Any]:if not items:- return {+ empty = {"n": 0,"judge_accuracy": None,"judge_accuracy_ci95": None,"mean_f1": None,"mean_bleu1": None,+ "n_scored_f1": 0,+ "answerable_rate": None,+ "evidence_session_recall_full": None,+ "evidence_session_recall_partial": None,+ "n_with_evidence": 0,}+ for channel in CHANNELS:+ empty[f"evidence_session_recall_full_{channel}"] = None+ empty[f"evidence_session_recall_partial_{channel}"] = None+ empty[f"n_with_evidence_{channel}"] = 0+ return emptycorrect = [1.0 if r.get("judge_correct") else 0.0 for r in items]successes = int(sum(correct))lo, hi = wilson_ci(successes, len(items))- return {+ answerable = [1.0 if gold_in_context(r) else 0.0 for r in items]+ f1s = [float(r["f1"]) for r in items if r.get("f1") is not None]+ bleus = [float(r["bleu1"]) for r in items if r.get("bleu1") is not None]+ bucket = {"n": len(items),"judge_accuracy": mean(correct),"judge_accuracy_ci95": ({"low": lo, "high": hi} if lo is not None and hi is not None else None),- "mean_f1": mean(float(r.get("f1") or 0.0) for r in items),- "mean_bleu1": mean(float(r.get("bleu1") or 0.0) for r in items),+ "mean_f1": mean(f1s),+ "mean_bleu1": mean(bleus),+ "n_scored_f1": len(f1s),+ "answerable_rate": mean(answerable),+ **_recall_bucket(items, "combined"),}+ for channel in CHANNELS:+ channel_bucket = _recall_bucket(items, channel)+ for key, value in channel_bucket.items():+ bucket[f"{key}_{channel}"] = value+ return bucketnon_adv = [r for r in rows if int(r.get("category") or 0) != 5]+ adversarial = [r for r in rows if int(r.get("category") or 0) == 5]all_rows = rowsnon_adv_bucket = _bucket(non_adv)all_bucket = _bucket(all_rows)@@ -130,6 +276,16 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:if r.get(key) is not None:total_tokens += int(r[key])+ answerer_gap = None+ if (+ non_adv_bucket.get("answerable_rate") is not None+ and non_adv_bucket.get("judge_accuracy") is not None+ ):+ answerer_gap = (+ float(non_adv_bucket["answerable_rate"])+ - float(non_adv_bucket["judge_accuracy"])+ )+return {"n_total": len(all_rows),"n_non_adversarial": len(non_adv),@@ -139,6 +295,26 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:"overall_judge_accuracy_ci95": all_bucket["judge_accuracy_ci95"],"mean_f1_non_adversarial": non_adv_bucket["mean_f1"],"mean_bleu1_non_adversarial": non_adv_bucket["mean_bleu1"],+ "answerable_rate": non_adv_bucket["answerable_rate"],+ "evidence_session_recall_full": non_adv_bucket["evidence_session_recall_full"],+ "evidence_session_recall_partial": non_adv_bucket[+ "evidence_session_recall_partial"+ ],+ "evidence_session_recall_by_channel": {+ channel: {+ "full": non_adv_bucket[f"evidence_session_recall_full_{channel}"],+ "partial": non_adv_bucket[+ f"evidence_session_recall_partial_{channel}"+ ],+ "n_with_evidence": non_adv_bucket[f"n_with_evidence_{channel}"],+ }+ for channel in CHANNELS+ },+ "answerer_gap": answerer_gap,+ "abstention": {+ "n": len(adversarial),+ "accuracy": _bucket(adversarial)["judge_accuracy"],+ },"by_category": {str(cat): _bucket(items) for cat, items in sorted(by_category.items())},@@ -151,6 +327,213 @@ def aggregate_answers(rows: list[dict[str, Any]]) -> dict[str, Any]:}+def mcnemar_exact(flipped_wrong: int, flipped_right: int) -> float:+ n = flipped_wrong + flipped_right+ if n <= 0:+ return 1.0+ k = min(flipped_wrong, flipped_right)+ tail = sum(math.comb(n, i) for i in range(0, k + 1)) / (2**n)+ return min(1.0, 2.0 * tail)+++def _correct_by_question(rows: list[dict[str, Any]]) -> dict[tuple[str, int], bool]:+ scores: dict[tuple[str, int], bool] = {}+ for row in rows:+ if row.get("error"):+ continue+ key = (str(row.get("sample_id")), int(row.get("qa_index") or 0))+ scores[key] = bool(row.get("judge_correct"))+ return scores+++def _arm_scores(arm: list[list[dict[str, Any]]]) -> dict[tuple[str, int], float]:+ per_run = [_correct_by_question(rows) for rows in arm if rows]+ if not per_run:+ return {}+ keys = set(per_run[0])+ for run in per_run[1:]:+ keys &= set(run)+ return {+ key: sum(1.0 for run in per_run if run[key]) / len(per_run) for key in keys+ }+++def _category_by_question(+ arm: list[list[dict[str, Any]]],+) -> dict[tuple[str, int], int]:+ categories: dict[tuple[str, int], int] = {}+ for rows in arm:+ for row in rows:+ key = (str(row.get("sample_id")), int(row.get("qa_index") or 0))+ categories.setdefault(key, int(row.get("category") or 0))+ return categories+++def _flip_table(+ baseline: dict[tuple[str, int], float],+ candidate: dict[tuple[str, int], float],+ keys: Iterable[tuple[str, int]],+) -> dict[str, Any]:+ flipped_right: list[tuple[str, int]] = []+ flipped_wrong: list[tuple[str, int]] = []+ agreed = 0+ for key in keys:+ base = baseline[key]+ cand = candidate[key]+ if cand > base:+ flipped_right.append(key)+ elif cand < base:+ flipped_wrong.append(key)+ else:+ agreed += 1+ p_value = mcnemar_exact(len(flipped_wrong), len(flipped_right))+ return {+ "n_paired": agreed + len(flipped_right) + len(flipped_wrong),+ "agreed": agreed,+ "flipped_right": len(flipped_right),+ "flipped_wrong": len(flipped_wrong),+ "mcnemar_exact_p": p_value,+ "significant_at_05": p_value < 0.05,+ "flipped_right_questions": [f"{s}::{i}" for s, i in sorted(flipped_right)],+ "flipped_wrong_questions": [f"{s}::{i}" for s, i in sorted(flipped_wrong)],+ }+++def compare_arms(+ baseline_runs: list[list[dict[str, Any]]],+ candidate_runs: list[list[dict[str, Any]]],+ *,+ skip_adversarial: bool = True,+) -> dict[str, Any]:+ baseline = _arm_scores(baseline_runs)+ candidate = _arm_scores(candidate_runs)+ categories = _category_by_question(baseline_runs + candidate_runs)+ keys = sorted(set(baseline) & set(candidate))+ if skip_adversarial:+ keys = [k for k in keys if categories.get(k, 0) != 5]++ overall = _flip_table(baseline, candidate, keys)+ by_category: dict[str, Any] = {}+ for cat in sorted({categories.get(k, 0) for k in keys}):+ cat_keys = [k for k in keys if categories.get(k, 0) == cat]+ by_category[str(cat)] = _flip_table(baseline, candidate, cat_keys)++ graph_stability = None+ if len(baseline_runs) == 1 and len(candidate_runs) == 1:+ graph_stability = graph_session_stability(+ baseline_runs[0],+ candidate_runs[0],+ skip_adversarial=skip_adversarial,+ )++ return {+ "baseline_runs": len(baseline_runs),+ "candidate_runs": len(candidate_runs),+ "baseline_accuracy": mean(baseline[k] for k in keys),+ "candidate_accuracy": mean(candidate[k] for k in keys),+ "overall": overall,+ "by_category": by_category,+ "graph_session_stability": graph_stability,+ }+++_GRAPH_STABILITY_GATE = 0.95+++def _graph_session_set(row: dict[str, Any]) -> frozenset[str]:+ raw = row.get("retrieved_session_ids_graph")+ if raw is None:+ raw = row.get("graph_session_ids")+ if not raw:+ return frozenset()+ return frozenset(str(s) for s in raw if s)+++def _row_key(row: dict[str, Any]) -> tuple[str, int]:+ return (str(row.get("sample_id") or ""), int(row.get("qa_index") or 0))+++def graph_session_stability(+ baseline_rows: list[dict[str, Any]],+ candidate_rows: list[dict[str, Any]],+ *,+ skip_adversarial: bool = True,+ gate: float = _GRAPH_STABILITY_GATE,+) -> dict[str, Any]:+ baseline_map = {_row_key(r): r for r in baseline_rows if not r.get("error")}+ candidate_map = {_row_key(r): r for r in candidate_rows if not r.get("error")}+ keys = sorted(set(baseline_map) & set(candidate_map))+ if skip_adversarial:+ keys = [+ k+ for k in keys+ if int(baseline_map[k].get("category") or 0) != 5+ and int(candidate_map[k].get("category") or 0) != 5+ ]++ identical = 0+ identical_coverage = 0+ coverage_comparable = 0+ disagreeing: list[str] = []+ for key in keys:+ base = baseline_map[key]+ cand = candidate_map[key]+ base_set = _graph_session_set(base)+ cand_set = _graph_session_set(cand)+ if base_set == cand_set:+ identical += 1+ else:+ disagreeing.append(f"{key[0]}::{key[1]}")+ base_cov = evidence_coverage(base, "graph")+ cand_cov = evidence_coverage(cand, "graph")+ if base_cov is not None and cand_cov is not None:+ coverage_comparable += 1+ if base_cov == cand_cov:+ identical_coverage += 1++ n = len(keys)+ agreement_rate = (identical / n) if n else None+ coverage_agreement_rate = (+ (identical_coverage / coverage_comparable) if coverage_comparable else None+ )+ return {+ "n_paired": n,+ "identical_session_sets": identical,+ "agreement_rate": agreement_rate,+ "identical_graph_coverage": identical_coverage,+ "n_coverage_comparable": coverage_comparable,+ "coverage_agreement_rate": coverage_agreement_rate,+ "gate": gate,+ "passes_gate": (+ agreement_rate is not None and agreement_rate >= gate+ ),+ "disagreeing_questions": disagreeing,+ "note": (+ "Do not A/B graph EvR below the gate: graph EvR is a measurement only "+ f"when ≥{gate:.0%} of questions have identical graph-session sets "+ "across two identical-config runs."+ ),+ }+++def retrieval_arm_summary(arm: list[list[dict[str, Any]]]) -> dict[str, Any]:+ per_run = []+ for rows in arm:+ scored = [r for r in rows if not r.get("error")]+ metrics = aggregate_answers(scored)+ per_run.append(+ {+ "evidence_session_recall_full": metrics["evidence_session_recall_full"],+ "evidence_session_recall_by_channel": metrics[+ "evidence_session_recall_by_channel"+ ],+ "answerable_rate": metrics["answerable_rate"],+ }+ )+ identical = all(run == per_run[0] for run in per_run) if per_run else True+ return {"per_run": per_run, "identical_across_runs": identical}++def selftest_metrics() -> list[str]:errors: list[str] = []if abs(f1_score("7 May 2023", "7 May 2023") - 1.0) > 1e-9:@@ -164,4 +547,124 @@ def selftest_metrics() -> list[str]:lo, hi = wilson_ci(5, 10)if lo is None or hi is None or not (0.0 <= lo <= 0.5 <= hi <= 1.0):errors.append("wilson_ci(5,10) should contain 0.5")+ if evidence_session_ids(["D7:3", "D12:1"]) != {"session_7", "session_12"}:+ errors.append("evidence_session_ids failed")+ sample = {+ "gold": "Charlotte's Web",+ "text_context": "Melanie loves Charlotte's Web",+ "source_passages": [],+ "historical_context": [],+ "evidence": ["D7:1"],+ "retrieved_session_ids": ["session_7"],+ }+ if not gold_in_context(sample):+ errors.append("gold_in_context should detect gold tokens")+ if evidence_coverage(sample) != "full":+ errors.append("evidence_coverage should be full")++ if f1_score("", "") is not None:+ errors.append("empty prediction vs empty gold must not score F1")+ if bleu1_score("", "") is not None:+ errors.append("empty prediction vs empty gold must not score BLEU-1")+ if f1_score("Not mentioned", "") is not None:+ errors.append("empty gold must not be scorable by F1")+ if f1_score("", "Paris") != 0.0:+ errors.append("empty prediction vs non-empty gold should have F1=0")+ if overlap_scores("", "", adversarial=True) != (None, None):+ errors.append("adversarial items must not be scored by n-gram overlap")+ if overlap_scores("LGBTQ+ folks", "LGBTQ+ individuals", adversarial=True) != (+ None,+ None,+ ):+ errors.append("matching an adversarial trap answer must not score as overlap")++ channels = {+ "gold": "slipper",+ "graph_context": "Melanie | OWNS | slipper (session_3)",+ "triples": ["Melanie: Melanie | PUT | keys | IN | slipper"],+ "source_passages": ["[passage] Session id: session_9. Caroline said hi."],+ "historical_context": [],+ "evidence": ["D3:2"],+ }+ if retrieved_session_ids(channels, "graph") != {"session_3"}:+ errors.append("graph channel session ids failed")+ if retrieved_session_ids(channels, "passages") != {"session_9"}:+ errors.append("passage channel session ids failed")+ if evidence_coverage(channels, "graph") != "full":+ errors.append("graph channel evidence coverage should be full")+ if evidence_coverage(channels, "passages") != "none":+ errors.append("passage channel evidence coverage should be none")+ legacy = {"evidence": ["D3:2"], "retrieved_session_ids": ["session_3"]}+ if evidence_coverage(legacy, "graph") is not None:+ errors.append("graph coverage must be unknown for rows without a graph channel")++ if abs(mcnemar_exact(9, 20) - 0.06142835) > 1e-6:+ errors.append("mcnemar_exact(9, 20) should be ~0.0614")+ if abs(mcnemar_exact(10, 32) - 0.00094067) > 1e-7:+ errors.append("mcnemar_exact(10, 32) should be ~0.00094")+ if mcnemar_exact(0, 0) != 1.0:+ errors.append("mcnemar_exact with no discordant pairs should be 1.0")++ base_rows = [+ {"sample_id": "s", "qa_index": 0, "category": 1, "judge_correct": False},+ {"sample_id": "s", "qa_index": 1, "category": 1, "judge_correct": True},+ ]+ cand_rows = [+ {"sample_id": "s", "qa_index": 0, "category": 1, "judge_correct": True},+ {"sample_id": "s", "qa_index": 1, "category": 1, "judge_correct": True},+ ]+ comparison = compare_arms([base_rows], [cand_rows])+ if comparison["overall"]["flipped_right"] != 1:+ errors.append("compare_arms should count one flip to correct")+ if comparison["overall"]["flipped_wrong"] != 0:+ errors.append("compare_arms should count no flip to wrong")++ stable_a = [+ {+ "sample_id": "s",+ "qa_index": 0,+ "category": 1,+ "retrieved_session_ids_graph": ["session_1", "session_2"],+ "evidence": ["D1:1", "D2:1"],+ "judge_correct": True,+ },+ {+ "sample_id": "s",+ "qa_index": 1,+ "category": 4,+ "retrieved_session_ids_graph": ["session_3"],+ "evidence": ["D3:1"],+ "judge_correct": True,+ },+ ]+ stable_b = [+ {+ "sample_id": "s",+ "qa_index": 0,+ "category": 1,+ "graph_session_ids": ["session_2", "session_1"],+ "evidence": ["D1:1", "D2:1"],+ "judge_correct": True,+ },+ {+ "sample_id": "s",+ "qa_index": 1,+ "category": 4,+ "retrieved_session_ids_graph": ["session_3"],+ "evidence": ["D3:1"],+ "judge_correct": True,+ },+ ]+ stability = graph_session_stability(stable_a, stable_b)+ if stability["agreement_rate"] != 1.0 or not stability["passes_gate"]:+ errors.append("graph_session_stability should pass on identical sets")+ unstable_b = [+ dict(stable_b[0], retrieved_session_ids_graph=["session_9"]),+ stable_b[1],+ ]+ unstable = graph_session_stability(stable_a, unstable_b)+ if unstable["agreement_rate"] != 0.5:+ errors.append("graph_session_stability should report 50% on one disagreement")+ if "s::0" not in unstable["disagreeing_questions"]:+ errors.append("graph_session_stability should list disagreeing qa keys")return errorsdiff --git a/benchmarks/locomo/prompts.py b/benchmarks/locomo/prompts.pyindex abb9063..b4f6baa 100644--- a/benchmarks/locomo/prompts.py+++ b/benchmarks/locomo/prompts.py@@ -1,14 +1,21 @@from __future__ import annotations+import hashlibfrom typing import AnyANSWER_SYSTEM = """You are answering questions about a long-term conversation using only the retrieved memory context provided.Rules:-- Prefer short factual answers: dates, names, places, or short phrases when that is what the question asks for.+- Prefer short factual answers: dates, names, places, or short phrases when that is what the question asks for. Never return an empty answer.+- Refer to people by the names used in the dialogue rather than by the first-person pronouns the speakers use.- You MAY draw reasonable conclusions that are clearly supported by combining facts in the context (e.g. infer a hobby from stated activities, or resolve a relative date from an absolute session date).- Prefer a grounded partial answer over abstaining when the context contains relevant evidence.-- Reply exactly "Not mentioned in the conversation" ONLY when the context contains no evidence that could support an answer — not when the answer requires light inference.+- OPEN-DOMAIN: For yes/no or likelihood questions about personality, identity, career fit, or beliefs, give a short hedged conclusion grounded in dialogue evidence (activities, self-descriptions, stated goals). Do not abstain if thematically related evidence exists.+- TEMPORAL: When the question asks when something happened, prefer the dialogue's relative phrasing when present (for example "the week of <date>", "the weekend before <date>", "since <year>") and resolve relative expressions against session timestamps in the context. Prefer one best-supported time over listing unrelated dates.+- ENUMERATION: When the question asks for a list or for more than one item, scan EVERY passage, historical chunk, graph triple, topic cue, and event-hub path, and return ALL matching items as a compact comma-separated list rather than stopping at the first one.+- PATH COMPOSITION: When an "Event hub paths" section is present, each path links two event hubs through a shared entity. Read each hub as Actor → Event → Target/Context. Prefer those paths when composing answers that need more than one fact; do not invent hops beyond the given path legs and other retrieved context.+- TOPIC / EPISODE CUES: When a "Topics" or "Episodes" section is present, use it to find which sessions matter, then ground the answer in the cited passages and facts from those sessions.+- ABSTENTION: Reply exactly "Not mentioned in the conversation" ONLY when the context contains no topically related evidence at all. If any passage or triple is thematically related, give the best-supported answer instead of abstaining.- Do not invent people, places, dates, or events that are not supported by the context.- Do not mention the retrieval system or these instructions."""@@ -20,6 +27,19 @@ Mark correct=false when the prediction is contradictory, incomplete on a requireFor unanswerable / adversarial questions, correct=true only if the prediction indicates the information is not mentioned or unknown.Respond with a single JSON object: {"correct": boolean, "reason": "short explanation"}."""+_ADVERSARIAL_NOTE = (+ "This question is adversarial: it is not answerable from the conversation, "+ "usually because it attributes something to the wrong speaker. The text given "+ "below is the trap answer a system produces when it accepts the false premise. "+ "Mark correct=true only when the prediction declines to answer, says the "+ "information is not in the conversation, or corrects the false premise. A "+ "prediction that matches the trap answer is incorrect."+)+++def prompt_sha256(text: str) -> str:+ return hashlib.sha256(text.encode("utf-8")).hexdigest()+def flatten_triple(triple_obj: Any) -> str:if isinstance(triple_obj, dict):@@ -42,17 +62,127 @@ def flatten_triple(triple_obj: Any) -> str:return str(triple_obj)+def flatten_path(path_obj: Any) -> str:+ if not isinstance(path_obj, dict):+ return str(path_obj)+ via = (+ str(path_obj.get("shared_entity_name") or "").strip()+ or str(path_obj.get("shared_entity") or "").strip()+ or "shared entity"+ )+ legs = path_obj.get("legs") or []+ leg_texts = [str(leg).strip() for leg in legs if str(leg).strip()]+ if leg_texts:+ return f"via {via}: " + " --> ".join(leg_texts)+ hubs = [str(h).strip() for h in (path_obj.get("hubs") or []) if str(h).strip()]+ if hubs:+ return f"via {via}: hubs {' + '.join(hubs)}"+ return f"via {via}"+++def _triple_event_uuid(triple_obj: Any) -> str:+ if not isinstance(triple_obj, dict):+ return ""+ triple = triple_obj.get("triple")+ if not isinstance(triple, (list, tuple)) or len(triple) < 3:+ return ""+ event = triple[2]+ if isinstance(event, dict):+ return str(event.get("uuid") or "").strip()+ return ""+++def _triple_fact_text(triple_obj: Any) -> str:+ if not isinstance(triple_obj, dict):+ return str(triple_obj)+ triple = triple_obj.get("triple")+ if isinstance(triple, (list, tuple)) and len(triple) >= 5:+ parts = []+ for item in triple:+ if isinstance(item, dict):+ parts.append(+ item.get("name")+ or item.get("type")+ or item.get("description")+ or str(item)+ )+ else:+ parts.append(str(item))+ return " | ".join(parts)+ return flatten_triple(triple_obj)+++def enrich_paths_from_triples(context: dict[str, Any]) -> dict[str, Any]:+ """Attach readable path legs from triples when the API omitted them."""+ paths = context.get("paths") or []+ triples = context.get("triples") or []+ if not paths:+ return context+ if any(isinstance(p, dict) and p.get("legs") for p in paths):+ return context++ hub_texts: dict[str, str] = {}+ for triple in triples:+ hub = _triple_event_uuid(triple)+ text = _triple_fact_text(triple).strip()+ if hub and text and hub not in hub_texts:+ hub_texts[hub] = text+ if not hub_texts:+ return context++ enriched: list[dict[str, Any]] = []+ for path in paths:+ if not isinstance(path, dict):+ continue+ hubs = [str(h) for h in (path.get("hubs") or []) if h]+ if len(hubs) < 2 or not all(h in hub_texts for h in hubs):+ continue+ enriched.append(+ {+ **path,+ "legs": [hub_texts[h] for h in hubs],+ }+ )+ if not enriched:+ return context+ out = dict(context)+ out["paths"] = enriched+ return out++def build_context_block(context: dict[str, Any]) -> str:+ context = enrich_paths_from_triples(context)text_context = (context.get("text_context") or "").strip()triples = context.get("triples") or []+ paths = context.get("paths") or []historical = context.get("historical_context") or []source_passages = context.get("source_passages") or []+ topics = context.get("topics") or []+ episodes = context.get("episodes") or []sections = ["## Retrieved text context", text_context or "(empty)"]+ if topics:+ sections.append("## Topics")+ for topic in topics[:20]:+ if isinstance(topic, dict):+ label = topic.get("label") or topic.get("topic_id") or topic+ sessions = topic.get("sessions") or []+ sess = ", ".join(str(s) for s in sessions[:12])+ sections.append(f"- {label}" + (f" ({sess})" if sess else ""))+ else:+ sections.append(f"- {topic}")+ if episodes:+ sections.append("## Episodes")+ for episode in episodes[:20]:+ sections.append(f"- {episode}")if source_passages:sections.append("## Source passages")for passage in source_passages[:20]:sections.append(f"- {passage}")+ if paths:+ sections.append("## Event hub paths")+ for path in paths[:40]:+ sections.append(f"- {flatten_path(path)}")if triples:sections.append("## Graph triples")for t in triples[:80]:@@ -77,13 +207,16 @@ def build_answer_messages(question: str, context: dict[str, Any]) -> list[dict[sdef build_judge_messages(- question: str, gold: str, prediction: str+ question: str, gold: str, prediction: str, *, adversarial: bool = False) -> list[dict[str, str]]:+ gold_label = "Trap answer" if adversarial else "Gold answer"user = (f"Question: {question}\n"- f"Gold answer: {gold}\n"+ f"{gold_label}: {gold}\n"f"Predicted answer: {prediction}\n")+ if adversarial:+ user = f"{_ADVERSARIAL_NOTE}\n\n{user}"return [{"role": "system", "content": JUDGE_SYSTEM},{"role": "user", "content": user},diff --git a/benchmarks/locomo/provenance.py b/benchmarks/locomo/provenance.pynew file mode 100644index 0000000..3c5a936--- /dev/null+++ b/benchmarks/locomo/provenance.py@@ -0,0 +1,70 @@+from __future__ import annotations++import hashlib+import subprocess+from pathlib import Path+from typing import Any++from locomo.config import BENCHMARKS_ROOT, Settings, model_family+from locomo.prompts import ANSWER_SYSTEM, JUDGE_SYSTEM, prompt_sha256++REPO_ROOT = BENCHMARKS_ROOT.parent+++def _git(*args: str) -> str | None:+ try:+ result = subprocess.run(+ ["git", *args],+ cwd=REPO_ROOT,+ capture_output=True,+ text=True,+ timeout=10,+ )+ except (OSError, subprocess.SubprocessError):+ return None+ if result.returncode != 0:+ return None+ return result.stdout.strip()+++def git_sha() -> str | None:+ return _git("rev-parse", "HEAD")+++def git_dirty() -> bool | None:+ status = _git("status", "--porcelain")+ if status is None:+ return None+ return bool(status.strip())+++def file_sha256(path: Path) -> str | None:+ if not path.exists():+ return None+ digest = hashlib.sha256()+ with path.open("rb") as handle:+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):+ digest.update(chunk)+ return digest.hexdigest()+++def build_provenance(settings: Settings) -> dict[str, Any]:+ return {+ "git_sha": git_sha(),+ "git_dirty": git_dirty(),+ "answer_prompt_sha256": prompt_sha256(ANSWER_SYSTEM),+ "judge_prompt_sha256": prompt_sha256(JUDGE_SYSTEM),+ "dataset_path": str(settings.dataset_path),+ "dataset_sha256": file_sha256(settings.dataset_path),+ "answer_model": settings.answer_model,+ "answer_model_family": model_family(settings.answer_model),+ "answer_llm_base_url": settings.llm_base_url,+ "judge_model": settings.judge_model,+ "judge_model_family": model_family(settings.judge_model),+ "judge_provider": settings.judge_provider(),+ "judge_shares_answer_family": settings.judge_shares_answer_family,+ "bench_profile": settings.bench_profile,+ "sc_samples": settings.sc_samples,+ "sc_temperature": settings.sc_temperature,+ "gap_fill": settings.gap_fill,+ }diff --git a/benchmarks/locomo/report.py b/benchmarks/locomo/report.pyindex 9946c8d..530cdf5 100644--- a/benchmarks/locomo/report.py+++ b/benchmarks/locomo/report.py@@ -1,18 +1,46 @@from __future__ import annotationsimport json+import tempfile+from datetime import datetime, timezonefrom pathlib import Pathfrom typing import Anyfrom rich.console import Consolefrom rich.table import Table-from locomo.config import CATEGORY_NAMES+from locomo.config import BENCHMARKS_ROOT, CATEGORY_NAMESfrom locomo.ingest import load_jsonlfrom locomo.metrics import aggregate_answersconsole = Console()+REPORTS_PATH = BENCHMARKS_ROOT / "REPORTS.json"+_BENCHMARK_ID = "locomo"+_BENCHMARK_NAME = "LoCoMo"+_REPORTS_DESCRIPTION = (+ "BrainAPI benchmark results. Top published scores across suites. "+ "Updated when a suite evaluate/report completes successfully."+)+_LEADERBOARD_ENTRY_KEYS = (+ "run_id",+ "track",+ "scope",+ "samples",+ "n_non_adversarial",+ "headline_judge_accuracy",+ "headline_judge_accuracy_pct",+ "answerable_rate",+ "answer_model",+ "judge_model",+ "bench_profile",+ "brain",+ "git_sha",+ "report_path",+ "recorded_at",+)+_MIN_LEADERBOARD_N = 50+def _latest_ingest_rows(ingest_rows: list[dict[str, Any]]) -> list[dict[str, Any]]:"""Keep the last status per sample/unit (resume appends retries)."""@@ -26,12 +54,28 @@ def _latest_ingest_rows(ingest_rows: list[dict[str, Any]]) -> list[dict[str, Anyreturn list(latest.values())+def _latest_answer_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:+ latest: dict[tuple[str, int], dict[str, Any]] = {}+ for row in rows:+ key = (str(row.get("sample_id") or ""), int(row.get("qa_index") or 0))+ latest[key] = row+ return list(latest.values())++def build_report(run_dir: Path) -> dict[str, Any]:answers_path = run_dir / "answers.jsonl"ingest_path = run_dir / "ingest.jsonl"manifest_path = run_dir / "manifest.json"- answers = [r for r in load_jsonl(answers_path) if not r.get("error")]+ all_answer_rows = load_jsonl(answers_path)+ scored_rows = [r for r in all_answer_rows if not r.get("error")]+ answers = _latest_answer_rows(scored_rows)+ errored = [r for r in all_answer_rows if r.get("error")]+ duplicates_dropped = len(scored_rows) - len(answers)+ empty_predictions = sum(+ 1 for r in answers if not str(r.get("prediction") or "").strip()+ )+ truncated_rows = sum(1 for r in answers if r.get("context_truncated"))ingest_rows = load_jsonl(ingest_path)ingest_latest = _latest_ingest_rows(ingest_rows)manifest = {}@@ -50,35 +94,128 @@ def build_report(run_dir: Path) -> dict[str, Any]:"attempts": len(ingest_rows),}+ integrity = {+ "rows_in_file": len(all_answer_rows),+ "errored_rows": len(errored),+ "scored_rows": len(answers),+ "duplicates_dropped": duplicates_dropped,+ "empty_predictions": empty_predictions,+ "rows_with_truncated_context": truncated_rows,+ }+report = {"run_dir": str(run_dir),"manifest": manifest,"ingest": ingest_summary,+ "integrity": integrity,"metrics": metrics,+ "status": "ok","warnings": [],}if ingest_summary["partial_failed"] or ingest_summary["failed"]:report["warnings"].append("Some ingest units failed or partially failed; scores may be skewed.")+ if duplicates_dropped:+ report["warnings"].append(+ f"{duplicates_dropped} duplicate answer rows dropped "+ f"({len(scored_rows)} scored rows over {len(answers)} unique questions)."+ )+ if errored:+ report["warnings"].append(+ f"{len(errored)} rows errored and are excluded from every metric."+ )+ if truncated_rows:+ report["warnings"].append(+ f"{truncated_rows} rows had a retrieved-context channel truncated "+ "before logging; see context_truncated in answers.jsonl."+ )+ graph_rows = [r for r in answers if r.get("graph_context") or r.get("triples")]+ if graph_rows and not any(r.get("retrieved_session_ids_graph") for r in graph_rows):+ report["warnings"].append(+ f"The graph channel was logged for {len(graph_rows)} rows but carries no "+ "session ids, so its evidence recall reads 0% by construction: "+ "/retrieve/context returns graph facts without session provenance."+ )++ failure_reasons: list[str] = []+ if not answers:+ failure_reasons.append("no answer rows were scored")+ else:+ if empty_predictions == len(answers):+ failure_reasons.append("every scored answer is empty")+ if metrics.get("headline_judge_accuracy") == 0.0:+ failure_reasons.append("every scored answer was judged incorrect")+ if failure_reasons:+ report["status"] = "failed"+ report["warnings"].insert(+ 0,+ "RUN FAILED WHOLESALE: "+ + "; ".join(failure_reasons)+ + ". The percentages below are not a measurement of quality.",+ )+ elif errored or duplicates_dropped:+ report["status"] = "degraded"return reportdef render_markdown(report: dict[str, Any]) -> str:metrics = report["metrics"]manifest = report.get("manifest") or {}+ integrity = report.get("integrity") or {}+ channels = metrics.get("evidence_session_recall_by_channel") or {}lines = ["# LoCoMo BrainAPI Benchmark Report","",- f"- Run dir: `{report['run_dir']}`",- f"- Created: {manifest.get('created_at', 'n/a')}",- f"- Granularity: {manifest.get('granularity', 'n/a')}",- f"- Answer model: {manifest.get('answer_model', 'n/a')}",- f"- Judge model: {manifest.get('judge_model', 'n/a')}",- f"- Samples: {', '.join(manifest.get('samples', []) or ['n/a'])}",- "",- "## Headline",- "",+ ]+ if report.get("status") == "failed":+ lines.extend(+ [+ "> **RUN FAILED WHOLESALE — the numbers below are not a measurement.**",+ f"> {report['warnings'][0]}",+ "",+ ]+ )+ elif report.get("status") == "degraded":+ lines.extend(+ [+ "> **Run degraded** — errored or duplicate rows were dropped; "+ "see Warnings.",+ "",+ ]+ )+ lines.extend(+ [+ f"- Run dir: `{report['run_dir']}`",+ f"- Created: {manifest.get('created_at', 'n/a')}",+ f"- Granularity: {manifest.get('granularity', 'n/a')}",+ f"- Answer model: {manifest.get('answer_model', 'n/a')}",+ f"- Judge model: {manifest.get('judge_model', 'n/a')} "+ f"(provider: {manifest.get('judge_provider', 'n/a')}, "+ f"same family as answerer: "+ f"{manifest.get('judge_shares_answer_family', 'n/a')})",+ f"- Git SHA: {manifest.get('git_sha', 'n/a')} "+ f"(dirty: {manifest.get('git_dirty', 'n/a')})",+ f"- Answer prompt sha256: {manifest.get('answer_prompt_sha256', 'n/a')}",+ f"- Judge prompt sha256: {manifest.get('judge_prompt_sha256', 'n/a')}",+ f"- Dataset sha256: {manifest.get('dataset_sha256', 'n/a')}",+ f"- Samples: {', '.join(manifest.get('samples', []) or ['n/a'])}",+ "",+ "## Run integrity",+ "",+ f"- Rows in answers.jsonl: {integrity.get('rows_in_file', 'n/a')}",+ f"- Scored unique questions: {integrity.get('scored_rows', 'n/a')}",+ f"- Errored rows (excluded): {integrity.get('errored_rows', 'n/a')}",+ f"- Duplicate rows dropped: {integrity.get('duplicates_dropped', 'n/a')}",+ f"- Empty predictions: {integrity.get('empty_predictions', 'n/a')}",+ f"- Rows with truncated context: "+ f"{integrity.get('rows_with_truncated_context', 'n/a')}",+ "",+ "## Headline",+ "",+ ]+ )+ lines.extend([f"- Judge accuracy (excl. adversarial / cat 5): "f"**{_pct(metrics.get('headline_judge_accuracy'))}** "f"{_ci(metrics.get('headline_judge_accuracy_ci95'))} "@@ -91,18 +228,43 @@ def render_markdown(report: dict[str, Any]) -> str:f"**{_num(metrics.get('mean_f1_non_adversarial'))}**",f"- Mean BLEU-1 (excl. adversarial): "f"**{_num(metrics.get('mean_bleu1_non_adversarial'))}**",+ f"- Answerable rate (gold tokens in context): "+ f"**{_pct(metrics.get('answerable_rate'))}**",+ f"- Evidence-session recall (full): "+ f"**{_pct(metrics.get('evidence_session_recall_full'))}**",+ f"- Evidence-session recall (partial+full): "+ f"**{_pct(metrics.get('evidence_session_recall_partial'))}**",+ f"- Evidence-session recall, graph channel: "+ f"**{_pct((channels.get('graph') or {}).get('full'))}** "+ f"(n={(channels.get('graph') or {}).get('n_with_evidence', 0)})",+ f"- Evidence-session recall, passage channel: "+ f"**{_pct((channels.get('passages') or {}).get('full'))}** "+ f"(n={(channels.get('passages') or {}).get('n_with_evidence', 0)})",+ f"- Answerer gap (answerable − judge): "+ f"**{_pct(metrics.get('answerer_gap'))}**",+ f"- Abstention accuracy (cat 5): "+ f"**{_pct((metrics.get('abstention') or {}).get('accuracy'))}** "+ f"(n={(metrics.get('abstention') or {}).get('n', 0)})",+ "",+ "Retrieval-side metrics are deterministic for a fixed brain and config; "+ "they are reported exactly and carry no significance test. "+ "Judge accuracy is not — compare runs with `python -m locomo compare`.","","## Per-category","",- "| Category | Name | N | Judge Acc | 95% CI | Mean F1 | Mean BLEU-1 |",- "|---|---|---:|---:|---|---:|---:|",- ]+ "| Category | Name | N | Judge Acc | 95% CI | Answerable | EvRecall | EvRecall graph | EvRecall passages | Mean F1 | Mean BLEU-1 |",+ "|---|---|---:|---:|---|---:|---:|---:|---:|---:|---:|",+ ])for cat, bucket in (metrics.get("by_category") or {}).items():name = CATEGORY_NAMES.get(int(cat), "unknown")lines.append(f"| {cat} | {name} | {bucket.get('n', 0)} | "f"{_pct(bucket.get('judge_accuracy'))} | "f"{_ci(bucket.get('judge_accuracy_ci95'))} | "+ f"{_pct(bucket.get('answerable_rate'))} | "+ f"{_pct(bucket.get('evidence_session_recall_full'))} | "+ f"{_pct(bucket.get('evidence_session_recall_full_graph'))} | "+ f"{_pct(bucket.get('evidence_session_recall_full_passages'))} | "f"{_num(bucket.get('mean_f1'))} | "f"{_num(bucket.get('mean_bleu1'))} |")@@ -142,6 +304,186 @@ def render_markdown(report: dict[str, Any]) -> str:return "\n".join(lines)+def entry_from_report(report: dict[str, Any]) -> dict[str, Any] | None:+ if report.get("status") == "failed":+ return None+ metrics = report.get("metrics") or {}+ headline = metrics.get("headline_judge_accuracy")+ n_non_adv = int(metrics.get("n_non_adversarial") or 0)+ if headline is None or n_non_adv < _MIN_LEADERBOARD_N:+ return None++ manifest = report.get("manifest") or {}+ run_id = str(manifest.get("run_id") or "").strip()+ if not run_id:+ run_dir = str(report.get("run_dir") or "").rstrip("/")+ run_id = Path(run_dir).name if run_dir else ""+ if not run_id:+ return None++ samples = list(manifest.get("samples") or [])+ if len(samples) == 1:+ scope = str(samples[0])+ elif len(samples) >= 10:+ scope = "locomo10"+ elif samples:+ scope = ",".join(str(s) for s in samples)+ else:+ scope = "unknown"++ bench_profile = manifest.get("bench_profile") or "product"+ track = "sota" if bench_profile == "sota" else "product"+ brain = manifest.get("brain_override")+ acc = float(headline)+ now = datetime.now(timezone.utc).isoformat()++ return {+ "run_id": run_id,+ "track": track,+ "scope": scope,+ "samples": samples,+ "n_non_adversarial": n_non_adv,+ "headline_judge_accuracy": acc,+ "headline_judge_accuracy_pct": round(acc * 100, 2),+ "answerable_rate": metrics.get("answerable_rate"),+ "answer_model": manifest.get("answer_model"),+ "judge_model": manifest.get("judge_model"),+ "bench_profile": bench_profile,+ "brain": brain,+ "git_sha": manifest.get("git_sha"),+ "report_path": f"runs/{run_id}/report.json",+ "recorded_at": now,+ }+++def _empty_reports_ledger() -> dict[str, Any]:+ return {+ "schema_version": 2,+ "updated_at": datetime.now(timezone.utc).isoformat(),+ "description": _REPORTS_DESCRIPTION,+ "benchmarks": {+ _BENCHMARK_ID: {+ "name": _BENCHMARK_NAME,+ "leaderboard": [],+ }+ },+ }+++def _public_leaderboard_entry(row: dict[str, Any]) -> dict[str, Any]:+ return {key: row[key] for key in _LEADERBOARD_ENTRY_KEYS if key in row}+++def _sorted_leaderboard(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:+ leaderboard = [+ _public_leaderboard_entry(row)+ for row in rows+ if isinstance(row, dict) and row.get("run_id")+ ]+ leaderboard.sort(+ key=lambda row: float(row.get("headline_judge_accuracy") or 0.0),+ reverse=True,+ )+ return leaderboard+++def _normalize_reports_ledger(data: dict[str, Any]) -> dict[str, Any]:+ benchmarks: dict[str, Any] = {}+ raw_benchmarks = data.get("benchmarks")+ if isinstance(raw_benchmarks, dict):+ for suite_id, suite in raw_benchmarks.items():+ if not isinstance(suite, dict):+ continue+ benchmarks[str(suite_id)] = {+ "name": suite.get("name") or str(suite_id),+ "leaderboard": _sorted_leaderboard(list(suite.get("leaderboard") or [])),+ }++ # One-shot migrate schema v1 flat leaderboard into benchmarks.locomo.+ legacy = data.get("leaderboard")+ if isinstance(legacy, list) and legacy:+ suite = dict(benchmarks.get(_BENCHMARK_ID) or {})+ by_id: dict[str, dict[str, Any]] = {+ str(row["run_id"]): row+ for row in (suite.get("leaderboard") or [])+ if isinstance(row, dict) and row.get("run_id")+ }+ for row in legacy:+ if not isinstance(row, dict) or not row.get("run_id"):+ continue+ rid = str(row["run_id"])+ if rid not in by_id:+ by_id[rid] = _public_leaderboard_entry(row)+ suite["name"] = suite.get("name") or _BENCHMARK_NAME+ suite["leaderboard"] = _sorted_leaderboard(list(by_id.values()))+ benchmarks[_BENCHMARK_ID] = suite++ if _BENCHMARK_ID not in benchmarks:+ benchmarks[_BENCHMARK_ID] = {+ "name": _BENCHMARK_NAME,+ "leaderboard": [],+ }++ return {+ "schema_version": 2,+ "updated_at": data.get("updated_at")+ or datetime.now(timezone.utc).isoformat(),+ "description": _REPORTS_DESCRIPTION,+ "benchmarks": benchmarks,+ }+++def update_reports_json(report: dict[str, Any]) -> None:+ entry = entry_from_report(report)+ if entry is None:+ return++ if REPORTS_PATH.exists():+ data = json.loads(REPORTS_PATH.read_text(encoding="utf-8"))+ else:+ data = _empty_reports_ledger()++ data = _normalize_reports_ledger(data)++ suite = data["benchmarks"].setdefault(+ _BENCHMARK_ID,+ {"name": _BENCHMARK_NAME, "leaderboard": []},+ )+ run_id = entry["run_id"]+ leaderboard = list(suite.get("leaderboard") or [])+ lb_idx = next(+ (i for i, row in enumerate(leaderboard) if row.get("run_id") == run_id),+ None,+ )+ if lb_idx is not None:+ leaderboard[lb_idx] = entry+ else:+ leaderboard.append(entry)++ suite["name"] = suite.get("name") or _BENCHMARK_NAME+ suite["leaderboard"] = leaderboard+ data["benchmarks"][_BENCHMARK_ID] = suite+ data["updated_at"] = datetime.now(timezone.utc).isoformat()+ data["description"] = _REPORTS_DESCRIPTION+ data = _normalize_reports_ledger(data)++ REPORTS_PATH.parent.mkdir(parents=True, exist_ok=True)+ fd, tmp_name = tempfile.mkstemp(+ prefix="REPORTS.",+ suffix=".json.tmp",+ dir=str(REPORTS_PATH.parent),+ )+ tmp_path = Path(tmp_name)+ try:+ with open(fd, "w", encoding="utf-8") as fh:+ json.dump(data, fh, indent=2)+ fh.write("\n")+ tmp_path.replace(REPORTS_PATH)+ except Exception:+ tmp_path.unlink(missing_ok=True)+ raise++def write_report(run_dir: Path) -> dict[str, Any]:report = build_report(run_dir)md = render_markdown(report)@@ -149,6 +491,8 @@ def write_report(run_dir: Path) -> dict[str, Any]:json.dumps(report, indent=2), encoding="utf-8")(run_dir / "report.md").write_text(md, encoding="utf-8")+ if report.get("status") == "ok":+ update_reports_json(report)return report@@ -160,6 +504,10 @@ def print_report_table(report: dict[str, Any]) -> None:table.add_column("N", justify="right")table.add_column("Judge Acc", justify="right")table.add_column("95% CI")+ table.add_column("Answerable", justify="right")+ table.add_column("EvRecall", justify="right")+ table.add_column("EvR graph", justify="right")+ table.add_column("EvR passages", justify="right")table.add_column("F1", justify="right")table.add_column("BLEU-1", justify="right")for cat, bucket in (metrics.get("by_category") or {}).items():@@ -169,19 +517,143 @@ def print_report_table(report: dict[str, Any]) -> None:str(bucket.get("n", 0)),_pct(bucket.get("judge_accuracy")),_ci(bucket.get("judge_accuracy_ci95")),+ _pct(bucket.get("answerable_rate")),+ _pct(bucket.get("evidence_session_recall_full")),+ _pct(bucket.get("evidence_session_recall_full_graph")),+ _pct(bucket.get("evidence_session_recall_full_passages")),_num(bucket.get("mean_f1")),_num(bucket.get("mean_bleu1")),)+ if report.get("status") == "failed":+ console.print(+ f"[bold red]{(report.get('warnings') or ['RUN FAILED'])[0]}[/bold red]"+ )console.print(table)console.print(f"Headline judge accuracy (excl. cat 5): "f"[bold]{_pct(metrics.get('headline_judge_accuracy'))}[/bold] "f"{_ci(metrics.get('headline_judge_accuracy_ci95'))}")+ channels = metrics.get("evidence_session_recall_by_channel") or {}+ console.print(+ f"Answerable rate: {_pct(metrics.get('answerable_rate'))} | "+ f"EvRecall full: {_pct(metrics.get('evidence_session_recall_full'))} "+ f"(graph {_pct((channels.get('graph') or {}).get('full'))}, "+ f"passages {_pct((channels.get('passages') or {}).get('full'))}) | "+ f"Answerer gap: {_pct(metrics.get('answerer_gap'))}"+ )for warning in report.get("warnings") or []:console.print(f"[yellow]Warning: {warning}[/yellow]")+def print_comparison(+ comparison: dict[str, Any],+ baseline_label: str,+ candidate_label: str,+ retrieval: dict[str, Any] | None = None,+) -> None:+ overall = comparison["overall"]+ console.print(+ f"[bold]{baseline_label}[/bold] -> [bold]{candidate_label}[/bold] "+ f"({comparison['baseline_runs']} vs {comparison['candidate_runs']} run(s), "+ f"n={overall['n_paired']} paired questions)"+ )+ table = Table(title="Paired judge-accuracy comparison (exact McNemar)")+ table.add_column("Scope")+ table.add_column("N", justify="right")+ table.add_column("Flipped right", justify="right")+ table.add_column("Flipped wrong", justify="right")+ table.add_column("Exact p", justify="right")+ table.add_column("Significant")+ rows = [("overall", overall)]+ for cat, bucket in (comparison.get("by_category") or {}).items():+ rows.append((f"{cat} {CATEGORY_NAMES.get(int(cat), 'unknown')}", bucket))+ for label, bucket in rows:+ table.add_row(+ label,+ str(bucket["n_paired"]),+ str(bucket["flipped_right"]),+ str(bucket["flipped_wrong"]),+ f"{bucket['mcnemar_exact_p']:.4g}",+ "yes" if bucket["significant_at_05"] else "no",+ )+ console.print(table)+ console.print(+ f"Accuracy: {_pct(comparison['baseline_accuracy'])} -> "+ f"{_pct(comparison['candidate_accuracy'])}"+ )+ if retrieval:+ console.print(+ "[dim]Passage retrieval metrics are exact for a fixed brain/config. "+ "Graph EvR is a measurement only after graph-session set agreement "+ "clears the ≥95% gate below.[/dim]"+ )+ ret_table = Table(title="Retrieval metrics (exact, no significance test)")+ ret_table.add_column("Arm")+ ret_table.add_column("Run")+ ret_table.add_column("EvRecall full", justify="right")+ ret_table.add_column("EvR graph", justify="right")+ ret_table.add_column("EvR passages", justify="right")+ ret_table.add_column("Answerable", justify="right")+ for arm_label, arm in (+ ("baseline", retrieval.get("baseline") or {}),+ ("candidate", retrieval.get("candidate") or {}),+ ):+ for name, run in zip(+ arm.get("labels") or [], arm.get("per_run") or []+ ):+ channels = run.get("evidence_session_recall_by_channel") or {}+ ret_table.add_row(+ arm_label,+ name,+ _pct(run.get("evidence_session_recall_full")),+ _pct((channels.get("graph") or {}).get("full")),+ _pct((channels.get("passages") or {}).get("full")),+ _pct(run.get("answerable_rate")),+ )+ console.print(ret_table)+ for arm_label, arm in (+ ("baseline", retrieval.get("baseline") or {}),+ ("candidate", retrieval.get("candidate") or {}),+ ):+ if len(arm.get("per_run") or []) > 1:+ verdict = (+ "identical across runs"+ if arm.get("identical_across_runs")+ else "NOT identical across runs"+ )+ console.print(f"[dim]{arm_label} retrieval metrics: {verdict}[/dim]")++ stability = comparison.get("graph_session_stability")+ if stability:+ rate = stability.get("agreement_rate")+ gate = stability.get("gate")+ n = stability.get("n_paired")+ identical = stability.get("identical_session_sets")+ passes = stability.get("passes_gate")+ console.print(+ f"Graph-session set agreement: "+ f"{_pct(rate)} ({identical}/{n} questions identical); "+ f"gate {_pct(gate)} — "+ f"{'PASS' if passes else 'FAIL (do not A/B graph EvR)'}"+ )+ cov_rate = stability.get("coverage_agreement_rate")+ if cov_rate is not None:+ console.print(+ f"Graph coverage-label agreement: {_pct(cov_rate)} "+ f"({stability.get('identical_graph_coverage')}/"+ f"{stability.get('n_coverage_comparable')})"+ )+ console.print(f"[dim]{stability.get('note')}[/dim]")+ disagreeing = stability.get("disagreeing_questions") or []+ if disagreeing:+ preview = ", ".join(disagreeing[:12])+ more = (+ f" (+{len(disagreeing) - 12} more)" if len(disagreeing) > 12 else ""+ )+ console.print(f"Disagreeing questions: {preview}{more}")++def _pct(value: float | None) -> str:if value is None:return "n/a"diff --git a/benchmarks/locomo/sota.py b/benchmarks/locomo/sota.pynew file mode 100644index 0000000..135e859--- /dev/null+++ b/benchmarks/locomo/sota.py@@ -0,0 +1,119 @@+from __future__ import annotations++import os+import re+from collections import Counter+from dataclasses import dataclass+from typing import Any+++_NORMALIZE_RE = re.compile(r"[^a-z0-9\s]+")+_WS_RE = re.compile(r"\s+")+++def normalize_answer(text: str) -> str:+ lowered = (text or "").strip().lower()+ lowered = _NORMALIZE_RE.sub(" ", lowered)+ return _WS_RE.sub(" ", lowered).strip()+++def majority_vote(answers: list[str]) -> str:+ """Pick the most common normalized answer; ties keep first occurrence order."""+ nonempty = [a.strip() for a in answers if (a or "").strip()]+ if not nonempty:+ return ""+ counts = Counter(normalize_answer(a) for a in nonempty)+ winner_norm, _ = max(+ counts.items(),+ key=lambda item: (item[1], -next(+ i for i, a in enumerate(nonempty) if normalize_answer(a) == item[0]+ )),+ )+ for answer in nonempty:+ if normalize_answer(answer) == winner_norm:+ return answer+ return nonempty[0]+++@dataclass(frozen=True)+class GapFillPlan:+ needs_retry: bool+ reformulated_query: str+ missing_slots: list[str]+++def plan_gap_fill(+ question: str,+ draft_answer: str,+ context: dict[str, Any],+) -> GapFillPlan:+ """Heuristic FAIR-RAG-lite: abstention or thin draft → re-retrieve with question."""+ draft = (draft_answer or "").strip()+ text_blob = " ".join(+ [+ str(context.get("text_context") or ""),+ " ".join(str(p) for p in (context.get("source_passages") or [])[:20]),+ " ".join(str(t) for t in (context.get("triples") or [])[:40]),+ ]+ ).lower()+ abstained = bool(+ re.search(+ r"not mentioned|no information|unknown|cannot (tell|determine)",+ draft,+ re.I,+ )+ )+ q_tokens = [t for t in re.findall(r"[a-z0-9]{3,}", question.lower()) if t]+ covered = sum(1 for t in q_tokens if t in text_blob) if q_tokens else 0+ coverage = covered / len(q_tokens) if q_tokens else 1.0+ missing = [t for t in q_tokens if t not in text_blob][:8]+ needs = abstained or (coverage < 0.45 and len(draft) < 40)+ reform = question.strip()+ if missing:+ reform = f"{question.strip()} (focus: {', '.join(missing[:5])})"+ return GapFillPlan(+ needs_retry=needs,+ reformulated_query=reform,+ missing_slots=missing,+ )+++def merge_contexts(+ primary: dict[str, Any], secondary: dict[str, Any]+) -> dict[str, Any]:+ """Union passage/fact channels from a second retrieve into the first."""+ out = dict(primary)+ for key in ("source_passages", "historical_context", "triples", "paths"):+ a = list(primary.get(key) or [])+ b = list(secondary.get(key) or [])+ seen: set[str] = set()+ merged: list[Any] = []+ for item in a + b:+ sig = str(item)+ if sig in seen:+ continue+ seen.add(sig)+ merged.append(item)+ out[key] = merged+ text_a = str(primary.get("text_context") or "")+ text_b = str(secondary.get("text_context") or "")+ if text_b and text_b not in text_a:+ out["text_context"] = (text_a + "\n" + text_b).strip()+ return out+++def profile_defaults(profile: str) -> dict[str, Any]:+ name = (profile or "product").strip().lower()+ if name == "sota":+ return {+ "bench_profile": "sota",+ "sc_samples": int(os.getenv("BENCH_SC_SAMPLES", "5")),+ "sc_temperature": float(os.getenv("BENCH_SC_TEMPERATURE", "0.7")),+ "gap_fill": os.getenv("BENCH_GAP_FILL", "1") not in {"0", "false", "False"},+ }+ return {+ "bench_profile": "product",+ "sc_samples": int(os.getenv("BENCH_SC_SAMPLES", "1")),+ "sc_temperature": 0.0,+ "gap_fill": os.getenv("BENCH_GAP_FILL", "0") in {"1", "true", "True"},+ }diff --git a/benchmarks/locomo/taxonomy.py b/benchmarks/locomo/taxonomy.pynew file mode 100644index 0000000..52c5126--- /dev/null+++ b/benchmarks/locomo/taxonomy.py@@ -0,0 +1,152 @@+#!/usr/bin/env python3+"""Classify wrong LoCoMo answers into generation vs retrieval failure modes."""+from __future__ import annotations++import argparse+import json+import re+from collections import Counter, defaultdict+from pathlib import Path+from typing import Any++from locomo.config import CATEGORY_NAMES, RUNS_DIR+from locomo.metrics import evidence_coverage, gold_in_context, tokenize++_RELATIVE_DATE = re.compile(+ r"\b(week of|since \d{4}|sunday before|saturday before|the day before)\b",+ re.I,+)+_ABSTAIN = re.compile(+ r"not mentioned|no information|unknown|cannot (tell|determine)",+ re.I,+)+++def _load_latest(path: Path) -> list[dict[str, Any]]:+ latest: dict[tuple[str, int], dict[str, Any]] = {}+ for line in path.read_text().splitlines():+ if not line.strip():+ continue+ row = json.loads(line)+ if row.get("error"):+ continue+ key = (str(row.get("sample_id")), int(row.get("qa_index") or 0))+ latest[key] = row+ return list(latest.values())+++def classify(row: dict[str, Any]) -> str:+ cat = int(row.get("category") or 0)+ gold = str(row.get("gold") or "")+ pred = str(row.get("prediction") or "")+ reason = str(row.get("judge_reason") or "").lower()+ answerable = gold_in_context(row)+ coverage = evidence_coverage(row, "combined")++ if not answerable and coverage in {None, "none", "partial"}:+ return "missing_evidence"+ if _ABSTAIN.search(pred) and gold.strip():+ return "present_but_unused"+ if cat == 3:+ return "open_domain_inference"+ if cat == 2 and (+ _RELATIVE_DATE.search(gold)+ or "date" in reason+ or "temporal" in reason+ or "phrasing" in reason+ ):+ return "temporal_format"+ if cat == 1:+ return "multi_hop_composition"+ if answerable:+ gold_toks = set(tokenize(gold))+ pred_toks = set(tokenize(pred))+ if gold_toks and len(gold_toks & pred_toks) / len(gold_toks) >= 0.4:+ return "phrasing"+ return "present_but_unused"+ return "missing_evidence"+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument(+ "--run",+ default="phase-d1-paths-a",+ help="Run id under benchmarks/runs/",+ )+ parser.add_argument(+ "--out",+ type=Path,+ default=None,+ help="Markdown output path",+ )+ args = parser.parse_args()+ run_dir = RUNS_DIR / args.run+ answers_path = run_dir / "answers.jsonl"+ if not answers_path.exists():+ raise SystemExit(f"missing {answers_path}")++ rows = _load_latest(answers_path)+ wrong = [+ r+ for r in rows+ if not r.get("judge_correct") and int(r.get("category") or 0) != 5+ ]+ by_class: dict[str, list[dict[str, Any]]] = defaultdict(list)+ for row in wrong:+ by_class[classify(row)].append(row)++ counts = Counter({k: len(v) for k, v in by_class.items()})+ n_wrong = len(wrong)+ gen_classes = {+ "present_but_unused",+ "phrasing",+ "open_domain_inference",+ "temporal_format",+ "multi_hop_composition",+ }+ gen_n = sum(counts[c] for c in gen_classes)+ gen_share = (gen_n / n_wrong) if n_wrong else 0.0+ n_total = len([r for r in rows if int(r.get("category") or 0) != 5])++ lines: list[str] = []+ lines.append(f"# SOTA failure taxonomy — `{args.run}`")+ lines.append("")+ lines.append(f"Non-adversarial scored: **{n_total}**; wrong: **{n_wrong}**.")+ lines.append(+ f"Generation-side share (expected ≥70%): **{gen_share:.0%}** "+ f"({gen_n}/{n_wrong})."+ )+ lines.append("")+ lines.append("| Class | Count | Est. headline pp if all fixed |")+ lines.append("| --- | ---: | ---: |")+ for cls, n in counts.most_common():+ pp = 100.0 * n / n_total if n_total else 0.0+ lines.append(f"| `{cls}` | {n} | ~{pp:.1f} |")+ lines.append("")+ lines.append("## Top examples (≤20)")+ lines.append("")+ shown = 0+ for cls, items in sorted(by_class.items(), key=lambda kv: -len(kv[1])):+ for row in items[: max(1, 20 // max(1, len(by_class)))]:+ if shown >= 20:+ break+ cat = CATEGORY_NAMES.get(int(row.get("category") or 0), "?")+ lines.append(+ f"- **{cls}** / {cat}: Q={row.get('question')!r} "+ f"gold={row.get('gold')!r} pred={row.get('prediction')!r} "+ f"reason={row.get('judge_reason')!r}"+ )+ shown += 1+ if shown >= 20:+ break++ out = args.out or (RUNS_DIR / "sota-failure-taxonomy.md")+ out.write_text("\n".join(lines) + "\n", encoding="utf-8")+ print(out)+ print(f"generation_share={gen_share:.3f} wrong={n_wrong}")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/locomo/test_sota_helpers.py b/benchmarks/locomo/test_sota_helpers.pynew file mode 100644index 0000000..7d96525--- /dev/null+++ b/benchmarks/locomo/test_sota_helpers.py@@ -0,0 +1,36 @@+from __future__ import annotations++import unittest++from locomo.sota import majority_vote, merge_contexts, plan_gap_fill, profile_defaults+++class SotaHelpersTests(unittest.TestCase):+ def test_majority_vote(self) -> None:+ self.assertEqual(majority_vote(["Yes", "yes", "No"]), "Yes")+ self.assertEqual(majority_vote(["", " "]), "")++ def test_profile_defaults(self) -> None:+ self.assertEqual(profile_defaults("product")["sc_samples"], 1)+ self.assertEqual(profile_defaults("sota")["sc_samples"], 5)+ self.assertTrue(profile_defaults("sota")["gap_fill"])++ def test_gap_fill_on_abstain(self) -> None:+ plan = plan_gap_fill(+ "Would Caroline pursue writing?",+ "Not mentioned in the conversation",+ {"text_context": "Caroline likes reading and wants to be a counselor"},+ )+ self.assertTrue(plan.needs_retry)++ def test_merge_contexts(self) -> None:+ merged = merge_contexts(+ {"source_passages": ["a"], "text_context": "a"},+ {"source_passages": ["b", "a"], "text_context": "b"},+ )+ self.assertEqual(merged["source_passages"], ["a", "b"])+ self.assertIn("b", merged["text_context"])+++if __name__ == "__main__":+ unittest.main()diff --git a/benchmarks/scripts/run_sota_locomo10.sh b/benchmarks/scripts/run_sota_locomo10.shnew file mode 100755index 0000000..61e3628--- /dev/null+++ b/benchmarks/scripts/run_sota_locomo10.sh@@ -0,0 +1,33 @@+#!/usr/bin/env bash+# Full LoCoMo10 SOTA leaderboard run (requires live BrainAPI + DeepSeek keys).+set -euo pipefail+ROOT="$(cd "$(dirname "$0")/.." && pwd)"+cd "$ROOT"++export BENCH_PROFILE="${BENCH_PROFILE:-sota}"+export BENCH_SC_SAMPLES="${BENCH_SC_SAMPLES:-5}"+export BENCH_GAP_FILL="${BENCH_GAP_FILL:-1}"+# Preferred answerer/judge: deepseek-v4-flash (maintainer: stronger than gpt-4o here).+export BENCH_SOTA_ANSWER_MODEL="${BENCH_SOTA_ANSWER_MODEL:-deepseek-v4-flash}"+export BENCH_SOTA_JUDGE_MODEL="${BENCH_SOTA_JUDGE_MODEL:-deepseek-v4-flash}"++RUN_INGEST="${RUN_INGEST:-sota-locomo10-ingest}"+RUN_EVAL="${RUN_EVAL:-sota-locomo10-eval}"++echo "== prompt-audit =="+../.venv/bin/python -m locomo prompt-audit++echo "== ingest all samples =="+../.venv/bin/python -m locomo ingest --run "$RUN_INGEST" --concurrency 2 --granularity session++echo "== evaluate (non-adversarial) =="+../.venv/bin/python -m locomo evaluate \+ --run "$RUN_EVAL" \+ --max-facts 50 \+ --max-passages 16 \+ --historical-limit 16 \+ --use-ppr \+ --no-fact-filter \+ --concurrency 2++echo "Done. See benchmarks/runs/$RUN_EVAL/report.md"