Skip to content

BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.

Add BEAM 10M harness: plan concat normalize and campaign scripts.

Christiancf61c99b8097
diff --git a/benchmarks/.env.example b/benchmarks/.env.example
index 8d0c17e..5cbb772 100644
--- a/benchmarks/.env.example
+++ b/benchmarks/.env.example
@@ -32,3 +32,7 @@ BENCH_ANSWER_MODEL=deepseek-v4-flash
# OPENAI_API_KEY=
# BENCH_LLM_BASE_URL=https://api.openai.com/v1
# BENCH_ANSWER_MODEL=gpt-4o-mini
+
+# BEAM HF datasets (defaults):
+# BENCH_BEAM_HF_DATASET=Mohammadta/BEAM
+# BENCH_BEAM_HF_DATASET_10M=Mohammadta/BEAM-10M
diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md
index f5bcf0e..04f8507 100644
--- a/benchmarks/AGENTS.md
+++ b/benchmarks/AGENTS.md
@@ -22,12 +22,14 @@ Typical flow: `download` → `ingest` → `evaluate` → `report` via `./longmem
Typical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh` or `.venv/bin/python -m beam`.
-- Dataset: HuggingFace `Mohammadta/BEAM` splits `100K` / `500K` / `1M` (BEAM-10M deferred).
+- Dataset: HuggingFace `Mohammadta/BEAM` (`100K` / `500K` / `1M`) and `Mohammadta/BEAM-10M` (`10M`). Harness supports all four sizes in `CHAT_SIZES`.
+- 10M normalize: concatenate interlocking plans chronologically into one chat (`bN_tM` / `session_N`); see [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md) § BEAM-10M. Brain `beam10m1` / `--brain beam10m1clean`; never wipe `beam1m1clean`.
+- Download sample 1 once `.env` exists: `./beam.sh download --size 10M` then `./beam.sh dataset-stats --size 10M` (read `n_turns` before any ETA). Ingest/eval still need live API + explicit go (`scripts/run_beam_10m_1_ingest_eval.sh`).
- Ingest turns only — never probing questions, rubrics, or ideal answers.
- Scoring: rubric LLM-judge (`judge_prompt_variant: beam-rubric-v1-question-aware`); headline = mean of 10 ability means; `event_ordering` uses `tau_norm`.
-- Brain IDs: `beam{size}{convid}` → e.g. `beam100k1`.
+- Brain IDs: `beam{size}{convid}` → e.g. `beam100k1`, `beam10m1`.
- Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md).
-- Ingest parallelism: prefer `--concurrency 2` (≤4); resume skips permanent embed-8192 fails. See protocol “Parallel ingest” + `scripts/run_beam_1m_1_ingest_eval.sh` (`BEAM_INGEST_CONCURRENCY`). Do not wipe live eval brains.
+- Ingest parallelism: prefer `--concurrency 2` (≤4); resume skips permanent embed-8192 fails. See protocol “Parallel ingest” + `scripts/run_beam_1m_1_ingest_eval.sh` / `scripts/run_beam_10m_1_ingest_eval.sh` (`BEAM_INGEST_CONCURRENCY`). Do not wipe live eval brains.
## Results ledger (`REPORTS.json`)
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 43a2fad..69ea41e 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -278,14 +278,14 @@ Do **not** ingest gold answers or `has_answer` turn labels.
# BEAM benchmark for BrainAPI
-Sibling harness for [BEAM](https://github.com/mohammadtavakoli78/BEAM) (ICLR 2026). Evaluates BrainAPI as long-term memory on multi-scale chats (`100K` / `500K` / `1M`). BEAM-10M is deferred.
+Sibling harness for [BEAM](https://github.com/mohammadtavakoli78/BEAM) (ICLR 2026). Evaluates BrainAPI as long-term memory on multi-scale chats (`100K` / `500K` / `1M` / `10M`).
-1. Downloads HuggingFace `Mohammadta/BEAM` and normalizes under `data/beam/`
-2. Ingests each batch turn into a brain (`beam100k1`, …)
+1. Downloads HuggingFace `Mohammadta/BEAM` (and `Mohammadta/BEAM-10M` for `--size 10M`) and normalizes under `data/beam/`
+2. Ingests each batch turn into a brain (`beam100k1`, `beam10m1`, …)
3. Answers probing questions via `POST /retrieve/context` + LLM
4. Scores with BEAM rubric LLM-judge (10 abilities); headline = mean of ability means
-Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md).
+Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md). 10M rows concatenate interlocking plans chronologically into one chat (same `bN_tM` / `session_N` path as 1M).
## Quickstart
@@ -299,12 +299,21 @@ Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protoco
./beam.sh selftest
```
+10M (after `benchmarks/.env` + live API; do not start long ingest casually):
+
+```bash
+./beam.sh download --size 10M
+./beam.sh dataset-stats --size 10M # read n_turns / ingest_target first
+./beam.sh smoke --size 10M --sample 1 --limit-turns 2
+# durable campaign: scripts/boot_beam_10m_screen.sh + beam_10m_keepalive.sh
+```
+
### Useful flags
-- `--size 100K|500K|1M`
+- `--size 100K|500K|1M|10M`
- `--sample <id>` (repeatable; id or `size/id`)
- `--limit-turns N` / `--limit N` / `--abilities a,b,…`
-- `--concurrency N` (ingest default 2; BEAM 1M prefer 2–3, ≤4; keep ≤ `CELERY_WORKER_CONCURRENCY`)
+- `--concurrency N` (ingest default 2; BEAM 1M/10M prefer 2–3, ≤4; keep ≤ `CELERY_WORKER_CONCURRENCY`)
- `--dry-run` / `--no-resume` (resume skips completed + permanent embed-8192 fails)
### Scoring
diff --git a/benchmarks/beam/cli.py b/benchmarks/beam/cli.py
index 2548d2e..8176c78 100644
--- a/benchmarks/beam/cli.py
+++ b/benchmarks/beam/cli.py
@@ -73,7 +73,9 @@ def cmd_dataset_stats(args: argparse.Namespace, settings: Settings) -> int:
)
console.print(table)
assert brain_id_for("100K", "1") == "beam100k1"
+ assert brain_id_for("10M", "1") == "beam10m1"
console.print("[dim]brain_id_for('100K', '1') == 'beam100k1' OK[/dim]")
+ console.print("[dim]brain_id_for('10M', '1') == 'beam10m1' OK[/dim]")
return 0
@@ -239,6 +241,8 @@ def cmd_selftest(args: argparse.Namespace, settings: Settings) -> int:
errors.append("brain_id_for('100K','1') != 'beam100k1'")
if brain_id_for("1M", "12") != "beam1m12":
errors.append("brain_id_for('1M','12') != 'beam1m12'")
+ if brain_id_for("10M", "1") != "beam10m1":
+ errors.append("brain_id_for('10M','1') != 'beam10m1'")
prompt = build_rubric_judge_prompt(
"When does my sprint end?",
@@ -311,7 +315,7 @@ def build_parser() -> argparse.ArgumentParser:
"--size",
choices=CHAT_SIZES,
default=None,
- help="Download one size only (default: all 100K/500K/1M)",
+ help="Download one size only (default: all 100K/500K/1M/10M)",
)
p_download.add_argument(
"--force",
diff --git a/benchmarks/beam/config.py b/benchmarks/beam/config.py
index a509ccf..538b177 100644
--- a/benchmarks/beam/config.py
+++ b/benchmarks/beam/config.py
@@ -11,12 +11,19 @@ DATA_DIR = BENCHMARKS_ROOT / "data"
RUNS_DIR = BENCHMARKS_ROOT / "runs"
DEFAULT_BEAM_DATA_DIR = DATA_DIR / "beam"
DEFAULT_HF_DATASET = "Mohammadta/BEAM"
+DEFAULT_HF_DATASET_10M = "Mohammadta/BEAM-10M"
DEFAULT_LLM_BASE_URL = "https://api.deepseek.com"
DEFAULT_ANSWER_MODEL = "deepseek-v4-flash"
DEFAULT_AZURE_JUDGE_MODEL = "gpt-4o"
DEFAULT_AZURE_API_VERSION = "2024-12-01-preview"
-CHAT_SIZES = ("100K", "500K", "1M")
+CHAT_SIZES = ("100K", "500K", "1M", "10M")
+HF_DATASET_BY_SIZE = {
+ "100K": DEFAULT_HF_DATASET,
+ "500K": DEFAULT_HF_DATASET,
+ "1M": DEFAULT_HF_DATASET,
+ "10M": DEFAULT_HF_DATASET_10M,
+}
ABILITY_NAMES = (
"abstention",
"contradiction_resolution",
@@ -70,6 +77,7 @@ class Settings:
judge_azure_api_version: str
beam_data_dir: Path
hf_dataset: str
+ hf_dataset_10m: str
runs_dir: Path
bench_profile: str = "product"
sc_samples: int = 1
@@ -78,6 +86,11 @@ class Settings:
answer_azure_endpoint: str | None = None
answer_azure_api_version: str = DEFAULT_AZURE_API_VERSION
+ def hf_dataset_for(self, size: str) -> str:
+ if size == "10M":
+ return self.hf_dataset_10m
+ return self.hf_dataset
+
@classmethod
def load(cls, env_file: Path | None = None) -> "Settings":
if env_file is None:
@@ -143,6 +156,9 @@ class Settings:
os.getenv("BENCH_BEAM_DATA_DIR", str(DEFAULT_BEAM_DATA_DIR))
),
hf_dataset=os.getenv("BENCH_BEAM_HF_DATASET", DEFAULT_HF_DATASET),
+ hf_dataset_10m=os.getenv(
+ "BENCH_BEAM_HF_DATASET_10M", DEFAULT_HF_DATASET_10M
+ ),
runs_dir=Path(os.getenv("BENCH_RUNS_DIR", str(RUNS_DIR))),
bench_profile=str(defaults["bench_profile"]),
sc_samples=max(1, int(defaults["sc_samples"])),
diff --git a/benchmarks/beam/dataset.py b/benchmarks/beam/dataset.py
index 525678a..eb6e6eb 100644
--- a/benchmarks/beam/dataset.py
+++ b/benchmarks/beam/dataset.py
@@ -8,6 +8,10 @@ from typing import Any
from beam.config import ABILITY_NAMES, CHAT_SIZES, Settings
+_PLAN_KEY_RE = re.compile(r"^plan[-_]?(\d+)$", re.IGNORECASE)
+NORMALIZE_STRATEGY_SINGLE = "single_chat"
+NORMALIZE_STRATEGY_PLANS = "concatenate_plans_chronological"
+
def brain_id_for(size: str, conversation_id: str) -> str:
slug = re.sub(r"[^a-zA-Z0-9]", "", f"beam{size}{conversation_id}")
@@ -79,6 +83,171 @@ def convert_chat_batches(chat: list[Any]) -> list[dict[str, Any]]:
return json_object
+def plan_sort_key(label: Any) -> tuple[int, str]:
+ text = str(label or "").strip()
+ match = _PLAN_KEY_RE.match(text)
+ if match:
+ return (int(match.group(1)), text)
+ if text.isdigit():
+ return (int(text), text)
+ return (10**9, text)
+
+
+def _as_mapping(value: Any) -> dict[str, Any] | None:
+ if isinstance(value, dict):
+ return value
+ if value is None:
+ return None
+ keys = getattr(value, "keys", None)
+ if callable(keys):
+ try:
+ return {str(k): value[k] for k in keys()}
+ except Exception:
+ return None
+ return None
+
+
+def _iter_plan_keyed_entries(chat: Any) -> list[tuple[str, Any]]:
+ """Yield (plan_id, payload) from BEAM-10M main chat (plan-1..plan-10)."""
+ mapping = _as_mapping(chat)
+ if mapping is not None:
+ entries = [
+ (key, mapping[key])
+ for key in mapping
+ if _PLAN_KEY_RE.match(str(key).strip())
+ ]
+ entries.sort(key=lambda item: plan_sort_key(item[0]))
+ return entries
+ if isinstance(chat, list):
+ collected: list[tuple[str, Any]] = []
+ for item in chat:
+ nested = _as_mapping(item)
+ if nested is None:
+ continue
+ for key, payload in nested.items():
+ if _PLAN_KEY_RE.match(str(key).strip()):
+ collected.append((str(key), payload))
+ collected.sort(key=lambda item: plan_sort_key(item[0]))
+ return collected
+ return []
+
+
+def _batches_from_chat_payload(value: Any) -> list[dict[str, Any]]:
+ """Accept already-normalized batches or raw HF message-list chat."""
+ if not isinstance(value, list) or not value:
+ return []
+ first = value[0]
+ if isinstance(first, dict) and "turns" in first:
+ return [row for row in value if isinstance(row, dict) and row.get("turns")]
+ if isinstance(first, list):
+ return [row for row in convert_chat_batches(value) if row.get("turns")]
+ return []
+
+
+def concatenate_plan_batches(
+ plan_batches: list[tuple[str, list[dict[str, Any]]]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """
+ Merge plan chats in chronological order into one chat.json-compatible list.
+
+ Reassigns batch_number globally so unit ids remain unique under the existing
+ bN_tM / session_N conventions used by the 1M ingest path.
+ """
+ out: list[dict[str, Any]] = []
+ plan_meta: list[dict[str, Any]] = []
+ next_batch = 1
+ for plan_id, batches in plan_batches:
+ start = next_batch
+ n_turns = 0
+ for batch in batches:
+ turns = batch.get("turns") or []
+ if not isinstance(turns, list) or not turns:
+ continue
+ n_turns += len(turns)
+ out.append(
+ {
+ "batch_number": next_batch,
+ "turns": turns,
+ "time_anchor": batch.get("time_anchor"),
+ "plan_id": plan_id,
+ }
+ )
+ next_batch += 1
+ if next_batch > start:
+ plan_meta.append(
+ {
+ "plan_id": plan_id,
+ "batch_number_start": start,
+ "batch_number_end": next_batch - 1,
+ "n_batches": next_batch - start,
+ "n_turns": n_turns,
+ }
+ )
+ return out, plan_meta
+
+
+def extract_normalized_chat(
+ row: dict[str, Any],
+ *,
+ size: str,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """
+ Build normalized chat batches for a HF row.
+
+ BEAM-10M: concatenate plan-1..plan-10 (or plans[]) chronologically into one
+ long chat. Smaller sizes: single-chat convert_chat_batches path.
+ """
+ chat = row.get("chat")
+ plans = row.get("plans")
+ plan_entries = _iter_plan_keyed_entries(chat)
+ use_multi_plan = size == "10M" or bool(plan_entries) or (
+ isinstance(plans, list) and bool(plans)
+ )
+
+ if use_multi_plan:
+ ordered: list[tuple[str, list[dict[str, Any]]]] = []
+ if plan_entries:
+ for plan_id, payload in plan_entries:
+ batches = _batches_from_chat_payload(payload)
+ if batches:
+ ordered.append((str(plan_id), batches))
+ source = "chat_plan_keys"
+ else:
+ source = "plans_array"
+ plan_rows = [p for p in (plans or []) if isinstance(p, dict)]
+ plan_rows.sort(
+ key=lambda p: plan_sort_key(p.get("plan_id") or p.get("id") or "")
+ )
+ for index, plan in enumerate(plan_rows):
+ plan_id = str(
+ plan.get("plan_id") or plan.get("id") or f"plan-{index + 1}"
+ )
+ batches = _batches_from_chat_payload(plan.get("chat"))
+ if batches:
+ ordered.append((plan_id, batches))
+
+ if ordered:
+ chat_json, plan_meta = concatenate_plan_batches(ordered)
+ return chat_json, {
+ "normalize_strategy": NORMALIZE_STRATEGY_PLANS,
+ "n_plans": len(plan_meta),
+ "plans": plan_meta,
+ "plan_source": source,
+ }
+
+ chat_list = chat if isinstance(chat, list) else []
+ return convert_chat_batches(chat_list), {
+ "normalize_strategy": NORMALIZE_STRATEGY_SINGLE,
+ }
+
+
+def ingest_target_turns(n_turns: int, *, slack: int = 5) -> int:
+ """Ops TARGET from normalized turn count (never invent without n_turns)."""
+ if n_turns <= 0:
+ return 0
+ return max(1, n_turns - max(0, slack))
+
+
def _parse_probing_questions(raw: Any) -> dict[str, list[dict[str, Any]]]:
if isinstance(raw, dict):
data = raw
@@ -106,11 +275,23 @@ def write_conversation(
size: str,
conversation_id: str,
topic: Any,
- chat: list[Any],
+ chat: list[Any] | None = None,
probing_questions: Any,
+ plans: Any = None,
+ row: dict[str, Any] | None = None,
) -> Path:
dest.mkdir(parents=True, exist_ok=True)
- chat_json = convert_chat_batches(chat)
+ if row is not None:
+ chat_json, extra_meta = extract_normalized_chat(row, size=size)
+ if topic is None:
+ topic = row.get("conversation_seed")
+ if probing_questions is None:
+ probing_questions = row.get("probing_questions")
+ else:
+ chat_json, extra_meta = extract_normalized_chat(
+ {"chat": chat or [], "plans": plans},
+ size=size,
+ )
probing = _parse_probing_questions(probing_questions)
(dest / "chat.json").write_text(
json.dumps(chat_json, ensure_ascii=False, indent=2),
@@ -125,14 +306,17 @@ def write_conversation(
json.dumps(topic, ensure_ascii=False, indent=2),
encoding="utf-8",
)
+ n_turns = sum(len(b.get("turns") or []) for b in chat_json)
meta = {
"conversation_id": str(conversation_id),
"size": size,
"sample_id": sample_id_for(size, str(conversation_id)),
"brain_id": brain_id_for(size, str(conversation_id)),
"n_batches": len(chat_json),
- "n_turns": sum(len(b.get("turns") or []) for b in chat_json),
+ "n_turns": n_turns,
"n_questions": sum(len(v) for v in probing.values()),
+ "ingest_target": ingest_target_turns(n_turns),
+ **extra_meta,
}
(dest / "meta.json").write_text(
json.dumps(meta, ensure_ascii=False, indent=2),
@@ -163,7 +347,8 @@ def download_and_normalize(
settings.beam_data_dir.mkdir(parents=True, exist_ok=True)
for size in selected:
- conversations = load_dataset(settings.hf_dataset, split=size)
+ dataset_name = settings.hf_dataset_for(size)
+ conversations = load_dataset(dataset_name, split=size)
for conversation in conversations:
conversation_id = str(conversation["conversation_id"])
dest = conversation_dir(
@@ -175,13 +360,14 @@ def download_and_normalize(
and (dest / "probing_questions.json").exists()
):
continue
+ row = dict(conversation)
write_conversation(
dest,
size=size,
conversation_id=conversation_id,
- topic=conversation.get("conversation_seed"),
- chat=list(conversation.get("chat") or []),
- probing_questions=conversation.get("probing_questions"),
+ topic=row.get("conversation_seed"),
+ probing_questions=row.get("probing_questions"),
+ row=row,
)
return settings.beam_data_dir
diff --git a/benchmarks/beam/provenance.py b/benchmarks/beam/provenance.py
index 42d35b2..5212511 100644
--- a/benchmarks/beam/provenance.py
+++ b/benchmarks/beam/provenance.py
@@ -89,7 +89,9 @@ def build_provenance(settings: Settings, *, size: str | None = None) -> dict[str
"judge_model_family": model_family(settings.judge_model),
"judge_provider": settings.judge_provider(),
"judge_shares_answer_family": settings.judge_shares_answer_family,
- "hf_dataset": settings.hf_dataset,
+ "hf_dataset": (
+ settings.hf_dataset_for(size) if size else settings.hf_dataset
+ ),
"bench_profile": settings.bench_profile,
"sc_samples": settings.sc_samples,
"sc_temperature": settings.sc_temperature,
diff --git a/benchmarks/beam/test_beam_helpers.py b/benchmarks/beam/test_beam_helpers.py
index 4b5401e..3d70a06 100644
--- a/benchmarks/beam/test_beam_helpers.py
+++ b/benchmarks/beam/test_beam_helpers.py
@@ -2,7 +2,16 @@ from __future__ import annotations
from pathlib import Path
-from beam.dataset import brain_id_for, convert_chat_batches, format_turn_text
+from beam.dataset import (
+ brain_id_for,
+ concatenate_plan_batches,
+ convert_chat_batches,
+ extract_normalized_chat,
+ format_turn_text,
+ ingest_target_turns,
+ iter_ingest_units,
+ write_conversation,
+)
from beam.ingest import (
completed_unit_ids,
is_permanent_ingest_error,
@@ -18,6 +27,8 @@ def test_brain_id_for() -> None:
assert brain_id_for("100K", "1") == "beam100k1"
assert brain_id_for("500K", "3") == "beam500k3"
assert brain_id_for("1M", "12") == "beam1m12"
+ assert brain_id_for("10M", "1") == "beam10m1"
+ assert brain_id_for("10M", "1clean") == "beam10m1clean"
def test_convert_chat_batches_splits_on_main_question() -> None:
@@ -45,6 +56,179 @@ def test_convert_chat_batches_splits_on_main_question() -> None:
assert batches[0]["turns"][1][0]["content"] == "Q2"
+def _raw_turn(content: str, *, anchor: str = "March-15-2024") -> list[dict]:
+ return [
+ {
+ "role": "user",
+ "content": content,
+ "question_type": "main_question",
+ "time_anchor": anchor,
+ },
+ {"role": "assistant", "content": f"re:{content}"},
+ ]
+
+
+def test_extract_normalized_chat_10m_plan_keys() -> None:
+ row = {
+ "chat": {
+ "plan-2": [
+ {
+ "batch_number": 1,
+ "time_anchor": None,
+ "turns": [_raw_turn("p2")],
+ }
+ ],
+ "plan-1": [
+ {
+ "batch_number": 1,
+ "time_anchor": None,
+ "turns": [_raw_turn("p1a"), _raw_turn("p1b")],
+ }
+ ],
+ },
+ "plans": [],
+ "probing_questions": {"summarization": [{"question": "Sum?", "rubric": ["x"]}]},
+ }
+ chat_json, meta = extract_normalized_chat(row, size="10M")
+ assert meta["normalize_strategy"] == "concatenate_plans_chronological"
+ assert meta["n_plans"] == 2
+ assert meta["plan_source"] == "chat_plan_keys"
+ assert [b["plan_id"] for b in chat_json] == ["plan-1", "plan-2"]
+ assert [b["batch_number"] for b in chat_json] == [1, 2]
+ assert len(chat_json[0]["turns"]) == 2
+ assert chat_json[0]["turns"][0][0]["content"] == "p1a"
+
+
+def test_extract_normalized_chat_10m_plans_array_raw() -> None:
+ row = {
+ "chat": None,
+ "plans": [
+ {
+ "plan_id": "plan-10",
+ "chat": [
+ [
+ {
+ "role": "user",
+ "content": "late",
+ "question_type": "main_question",
+ "time_anchor": "April-01-2024",
+ },
+ {"role": "assistant", "content": "re:late"},
+ ]
+ ],
+ },
+ {
+ "plan_id": "plan-1",
+ "chat": [
+ [
+ {
+ "role": "user",
+ "content": "early-a",
+ "question_type": "main_question",
+ "time_anchor": "March-15-2024",
+ },
+ {"role": "assistant", "content": "a"},
+ {
+ "role": "user",
+ "content": "early-b",
+ "question_type": "main_question",
+ "time_anchor": "March-16-2024",
+ },
+ {"role": "assistant", "content": "b"},
+ ]
+ ],
+ },
+ ],
+ }
+ chat_json, meta = extract_normalized_chat(row, size="10M")
+ assert meta["normalize_strategy"] == "concatenate_plans_chronological"
+ assert meta["plan_source"] == "plans_array"
+ assert [b["plan_id"] for b in chat_json] == ["plan-1", "plan-10"]
+ assert [b["batch_number"] for b in chat_json] == [1, 2]
+ sample = {
+ "sample_id": "10M/1",
+ "size": "10M",
+ "conversation_id": "1",
+ "chat": chat_json,
+ "probing_questions": {},
+ }
+ units = iter_ingest_units(sample)
+ assert len(units) == 3
+ assert [u["unit_id"] for u in units] == ["b1_t1", "b1_t2", "b2_t1"]
+ assert [u["session_key"] for u in units] == [
+ "session_1",
+ "session_2",
+ "session_3",
+ ]
+ assert "early-a" in units[0]["text"]
+ assert "late" in units[2]["text"]
+
+
+def test_concatenate_plan_batches_renumbers_globally() -> None:
+ chat_json, plan_meta = concatenate_plan_batches(
+ [
+ (
+ "plan-1",
+ [
+ {"turns": [_raw_turn("a")], "time_anchor": None},
+ {"turns": [_raw_turn("b")], "time_anchor": None},
+ ],
+ ),
+ ("plan-2", [{"turns": [_raw_turn("c")], "time_anchor": None}]),
+ ]
+ )
+ assert [b["batch_number"] for b in chat_json] == [1, 2, 3]
+ assert plan_meta[0]["batch_number_start"] == 1
+ assert plan_meta[0]["batch_number_end"] == 2
+ assert plan_meta[1]["batch_number_start"] == 3
+ assert ingest_target_turns(100) == 95
+ assert ingest_target_turns(0) == 0
+
+
+def test_write_conversation_10m_fixture(tmp_path: Path) -> None:
+ row = {
+ "conversation_id": "1",
+ "conversation_seed": {"title": "Fixture 10M"},
+ "chat": {
+ "plan-1": [
+ {
+ "batch_number": 1,
+ "turns": [_raw_turn("hello")],
+ "time_anchor": None,
+ }
+ ]
+ },
+ "plans": [],
+ "probing_questions": {
+ "information_extraction": [{"question": "What?", "rubric": ["hello"]}]
+ },
+ }
+ dest = tmp_path / "10M" / "1"
+ write_conversation(
+ dest,
+ size="10M",
+ conversation_id="1",
+ topic=row["conversation_seed"],
+ probing_questions=row["probing_questions"],
+ row=row,
+ )
+ meta = __import__("json").loads((dest / "meta.json").read_text(encoding="utf-8"))
+ assert meta["brain_id"] == "beam10m1"
+ assert meta["n_turns"] == 1
+ assert meta["ingest_target"] == 1
+ assert meta["normalize_strategy"] == "concatenate_plans_chronological"
+ chat = __import__("json").loads((dest / "chat.json").read_text(encoding="utf-8"))
+ units = iter_ingest_units(
+ {
+ "sample_id": "10M/1",
+ "chat": chat,
+ "probing_questions": {},
+ }
+ )
+ assert len(units) == 1
+ assert units[0]["unit_id"] == "b1_t1"
+
+
def test_format_turn_text_normalizes_time_anchor() -> None:
text, ts = format_turn_text(
[
diff --git a/benchmarks/scripts/beam_10m_keepalive.sh b/benchmarks/scripts/beam_10m_keepalive.sh
new file mode 100755
index 0000000..626fd2f
--- /dev/null
+++ b/benchmarks/scripts/beam_10m_keepalive.sh
@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Tiny keep-alive for BEAM 10M: only ensures API LISTEN + celery worker. Never pkill.
+# screen -dmS beam10m-keep bash benchmarks/scripts/beam_10m_keepalive.sh
+#
+# Safe under concurrent ingest: flock around starts; longer /docs probe.
+# Does not restart a healthy LISTEN uvicorn (avoids killing live eval).
+set +e
+SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"
+UV_LOG=/tmp/uvicorn-beam10m.log
+CEL_LOG=/tmp/brainapi-beam-worker.log
+LOCK_DIR=/tmp/beam10m-locks
+mkdir -p "$LOCK_DIR"
+CELERY_WORKER_CONCURRENCY="${CELERY_WORKER_CONCURRENCY:-4}"
+
+echo "KEEPALIVE_START $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> /tmp/beam-10m-keepalive.log
+while [[ ! -f /tmp/beam-10m-1.STOP ]]; do
+ if ! lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ echo "API_RESTART $(date -u +%H:%M:%S)" >> /tmp/beam-10m-keepalive.log
+ (
+ flock -n 9 || exit 0
+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ exit 0
+ fi
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \
+ >> "$UV_LOG" 2>&1 &
+ echo $! > /tmp/beam10m-uvicorn.pid
+ sleep 5
+ ) 9>"$LOCK_DIR/uvicorn.lock"
+ else
+ code=$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' http://127.0.0.1:8000/docs || echo 000)
+ if [[ "$code" != "200" && "$code" != "401" ]]; then
+ echo "API_BUSY_OR_SLOW code=$code $(date -u +%H:%M:%S)" >> /tmp/beam-10m-keepalive.log
+ fi
+ fi
+ if ! pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then
+ echo "CELERY_RESTART $(date -u +%H:%M:%S)" >> /tmp/beam-10m-keepalive.log
+ (
+ flock -n 9 || exit 0
+ if pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then
+ exit 0
+ fi
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads \
+ --concurrency=${CELERY_WORKER_CONCURRENCY} \
+ -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \
+ >> "$CEL_LOG" 2>&1 &
+ echo $! > /tmp/beam10m-celery.pid
+ sleep 5
+ ) 9>"$LOCK_DIR/celery.lock"
+ fi
+ sleep 20
+done
+echo "KEEPALIVE_STOP $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> /tmp/beam-10m-keepalive.log
diff --git a/benchmarks/scripts/boot_beam_10m_screen.sh b/benchmarks/scripts/boot_beam_10m_screen.sh
new file mode 100755
index 0000000..f5c3169
--- /dev/null
+++ b/benchmarks/scripts/boot_beam_10m_screen.sh
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Boot BEAM 10M ingest inside screen. Keep this file free of long uvicorn/celery
+# command strings in the screen argv — use this script path only:
+# screen -dmS beam10m /path/to/boot_beam_10m_screen.sh
+set +e
+SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"
+UV_LOG=/tmp/uvicorn-beam10m.log
+CEL_LOG=/tmp/brainapi-beam-worker.log
+UV_PID=/tmp/beam10m-uvicorn.pid
+CEL_PID=/tmp/beam10m-celery.pid
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+
+start_api() {
+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ return 0
+ fi
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \
+ >> "$UV_LOG" 2>&1 &
+ echo $! > "$UV_PID"
+ sleep 5
+}
+
+start_worker() {
+ if pgrep -f 'celery -A src.workers.app worker' >/dev/null 2>&1; then
+ return 0
+ fi
+ CELERY_C="${CELERY_WORKER_CONCURRENCY:-4}"
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads --concurrency=${CELERY_C} -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \
+ >> "$CEL_LOG" 2>&1 &
+ echo $! > "$CEL_PID"
+ sleep 5
+}
+
+start_api
+start_worker
+exec bash "$ROOT/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh" >> /tmp/beam-10m-1-run.log 2>&1
diff --git a/benchmarks/scripts/eval_beam_10m_1_product.sh b/benchmarks/scripts/eval_beam_10m_1_product.sh
new file mode 100755
index 0000000..1d66fa6
--- /dev/null
+++ b/benchmarks/scripts/eval_beam_10m_1_product.sh
@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+# Evaluate-only BEAM 10M/1 product on existing brain (no ingest / no wipe).
+# Never touches beam1m1clean.
+set +e
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+unset BRAINPAT_TOKEN BRAINAPI_TOKEN
+set -a; source ./.env; set +a
+export TERM=dumb PYTHONUNBUFFERED=1
+
+SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"
+UV_LOG=/tmp/uvicorn-beam10m.log
+CEL_LOG=/tmp/brainapi-beam-worker.log
+RUN_ID="${BEAM_10M_RUN:-beam-10m-1-clean}"
+BRAIN_ID="${BEAM_10M_BRAIN:-beam10m1clean}"
+EVAL_RUN_ID="${BEAM_10M_EVAL_RUN:-beam-10m-1-clean-product}"
+SAMPLE_ID="${BEAM_10M_SAMPLE:-1}"
+
+ensure_api() {
+ for _ in 1 2 3 4 5 6 7 8; do
+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ code=$(curl -s -o /dev/null --max-time 2 -w '%{http_code}' http://127.0.0.1:8000/docs || echo 000)
+ if [[ "$code" == "200" || "$code" == "401" ]]; then
+ if ! pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \
+ >> "$CEL_LOG" 2>&1 &
+ sleep 4
+ fi
+ return 0
+ fi
+ fi
+ if ! lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \
+ >> "$UV_LOG" 2>&1 &
+ sleep 5
+ fi
+ done
+ return 1
+}
+
+echo "START_10M_EVAL $(date -u +%Y-%m-%dT%H:%M:%SZ) brain=$BRAIN_ID eval_run=$EVAL_RUN_ID"
+ensure_api || { echo FATAL_API_DOWN; exit 1; }
+
+for ab in \
+ abstention \
+ contradiction_resolution \
+ event_ordering \
+ information_extraction \
+ instruction_following \
+ knowledge_update \
+ multi_session_reasoning \
+ preference_following \
+ summarization \
+ temporal_reasoning
+do
+ echo "==== EVAL $ab $(date -u +%H:%M:%S) ===="
+ ensure_api
+ ./beam.sh evaluate \
+ --size 10M --sample "$SAMPLE_ID" \
+ --run "$EVAL_RUN_ID" \
+ --brain "$BRAIN_ID" \
+ --profile product \
+ --abilities "$ab" \
+ --concurrency 1 \
+ --historical-limit 12 --max-passages 12 --max-facts 40 \
+ --no-ppr
+done
+
+./beam.sh report --run "$EVAL_RUN_ID"
+.venv/bin/python - <<PY
+import json, statistics
+from pathlib import Path
+ingest = Path('runs/${RUN_ID}/ingest.jsonl')
+rows=[json.loads(l) for l in ingest.read_text().splitlines() if l.strip()] if ingest.exists() else []
+last={}
+for r in rows: last[r.get('unit_id')]=r
+comp=[r for r in last.values() if r.get('status')=='completed' and (r.get('cost') or {}).get('llm_source_multiplier') is not None]
+mult=[r['cost']['llm_source_multiplier'] for r in comp]
+print('INGEST completed', sum(1 for r in last.values() if r.get('status')=='completed'), '/', len(last))
+print('PERMANENT_FAILS', sorted(
+ uid for uid,r in last.items()
+ if r.get('status')=='permanent_failed'
+ or (r.get('status')=='failed' and '8192' in (r.get('error') or ''))
+))
+print('MULTIPLIER n', len(mult),
+ 'median', round(statistics.median(mult),1) if mult else None,
+ 'mean', round(statistics.mean(mult),1) if mult else None)
+rep_path=Path('runs/${EVAL_RUN_ID}/report.json')
+if rep_path.exists():
+ rep=json.loads(rep_path.read_text())
+ m=rep.get('metrics') or {}
+ hs=m.get('headline_score')
+ print('HEADLINE', hs, f"({(hs or 0)*100:.1f}%)" if isinstance(hs,(int,float)) else '')
+ print('PER', m.get('per_ability'))
+PY
+date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/beam-10m-1.DONE
+echo EVAL_DONE
diff --git a/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh b/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh
new file mode 100755
index 0000000..a34c9d7
--- /dev/null
+++ b/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh
@@ -0,0 +1,253 @@
+#!/usr/bin/env bash
+# Resilient BEAM 10M/1 ingest + product evaluate.
+# Resume-safe. LISTEN-based ensure_api (never pkill healthy uvicorn).
+# Uses flock so watchdog/ensure never double-bind :8000.
+# NEVER touches beam1m1clean — uses beam10m1clean / beam-10m-1-clean only.
+#
+# TARGET comes from data/beam/10M/1/meta.json (ingest_target or n_turns-5).
+# Do not invent TARGET without a normalized sample.
+set +e
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+unset BRAINPAT_TOKEN BRAINAPI_TOKEN
+set -a; source ./.env; set +a
+export TERM=dumb PYTHONUNBUFFERED=1
+
+SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"
+UV_LOG=/tmp/uvicorn-beam10m.log
+CEL_LOG=/tmp/brainapi-beam-worker.log
+LOCK_DIR=/tmp/beam10m-locks
+mkdir -p "$LOCK_DIR"
+
+BEAM_INGEST_CONCURRENCY="${BEAM_INGEST_CONCURRENCY:-2}"
+BEAM_INGEST_TIMEOUT="${BEAM_INGEST_TIMEOUT:-2400}"
+CELERY_WORKER_CONCURRENCY="${CELERY_WORKER_CONCURRENCY:-4}"
+RUN_ID="${BEAM_10M_RUN:-beam-10m-1-clean}"
+BRAIN_ID="${BEAM_10M_BRAIN:-beam10m1clean}"
+EVAL_RUN_ID="${BEAM_10M_EVAL_RUN:-beam-10m-1-clean-product}"
+SAMPLE_ID="${BEAM_10M_SAMPLE:-1}"
+
+api_ok() {
+ lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1 || return 1
+ code=$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' http://127.0.0.1:8000/docs || echo 000)
+ [[ "$code" == "200" || "$code" == "401" ]]
+}
+
+start_uvicorn() {
+ (
+ flock -n 9 || return 0
+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ return 0
+ fi
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \
+ >> "$UV_LOG" 2>&1 &
+ echo $! > /tmp/beam10m-uvicorn.pid
+ sleep 5
+ ) 9>"$LOCK_DIR/uvicorn.lock"
+}
+
+start_celery() {
+ (
+ flock -n 9 || return 0
+ if pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then
+ return 0
+ fi
+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads \
+ --concurrency=${CELERY_WORKER_CONCURRENCY} \
+ -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \
+ >> "$CEL_LOG" 2>&1 &
+ echo $! > /tmp/beam10m-celery.pid
+ sleep 5
+ ) 9>"$LOCK_DIR/celery.lock"
+}
+
+ensure_api() {
+ for _ in 1 2 3 4 5 6 7 8 9 10 11 12; do
+ if api_ok; then
+ start_celery
+ return 0
+ fi
+ start_uvicorn
+ start_celery
+ sleep 3
+ done
+ return 1
+}
+
+resolve_target() {
+ .venv/bin/python - <<PY
+import json
+from pathlib import Path
+meta = Path("data/beam/10M/${SAMPLE_ID}/meta.json")
+if not meta.exists():
+ print(0)
+ raise SystemExit(0)
+row = json.loads(meta.read_text(encoding="utf-8"))
+target = row.get("ingest_target")
+if target is None:
+ n = int(row.get("n_turns") or 0)
+ target = max(1, n - 5) if n else 0
+print(int(target))
+PY
+}
+
+terminal_count() {
+ .venv/bin/python - <<PY
+import json
+import re
+from pathlib import Path
+p = Path("runs/${RUN_ID}/ingest.jsonl")
+rows = [json.loads(l) for l in p.read_text().splitlines() if l.strip()] if p.exists() else []
+last = {}
+for r in rows:
+ last[r.get("unit_id")] = r
+pat = re.compile(r"maximum context length is 8192", re.I)
+n = 0
+for r in last.values():
+ st = r.get("status")
+ err = r.get("error") or ""
+ if st in {"completed", "partial_failed", "permanent_failed"}:
+ n += 1
+ elif st == "failed" and pat.search(err):
+ n += 1
+print(n)
+PY
+}
+
+completed_count() {
+ terminal_count
+}
+
+watchdog() {
+ while [[ ! -f /tmp/beam-10m-1.STOP ]]; do
+ if ! lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then
+ echo "WATCHDOG_API_RESTART $(date -u +%H:%M:%S)"
+ start_uvicorn
+ fi
+ if ! pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then
+ echo "WATCHDOG_CELERY_RESTART $(date -u +%H:%M:%S)"
+ start_celery
+ fi
+ sleep 30
+ done
+}
+
+TARGET=$(resolve_target)
+if [[ "$TARGET" -le 0 ]]; then
+ echo "FATAL_NO_TARGET: normalize 10M sample first: ./beam.sh download --size 10M"
+ echo "Then re-run; TARGET is read from data/beam/10M/${SAMPLE_ID}/meta.json"
+ exit 1
+fi
+
+N_TURNS=$(.venv/bin/python - <<PY
+import json
+from pathlib import Path
+meta = Path("data/beam/10M/${SAMPLE_ID}/meta.json")
+print(json.loads(meta.read_text(encoding="utf-8")).get("n_turns", 0))
+PY
+)
+
+STALL_LIMIT=3
+prev_n=-1
+stalls=0
+MIN_EVAL_TURNS=$(( TARGET * 70 / 100 ))
+if [[ "$MIN_EVAL_TURNS" -lt 1 ]]; then
+ MIN_EVAL_TURNS=1
+fi
+
+rm -f /tmp/beam-10m-1.STOP /tmp/beam-10m-1.DONE
+echo "START_10M_INGEST $(date -u +%Y-%m-%dT%H:%M:%SZ) concurrency=$BEAM_INGEST_CONCURRENCY celery_c=$CELERY_WORKER_CONCURRENCY target=$TARGET n_turns=$N_TURNS brain=$BRAIN_ID run=$RUN_ID"
+ensure_api || { echo FATAL_API_DOWN; exit 1; }
+watchdog &
+WD_PID=$!
+echo "watchdog_pid=$WD_PID"
+
+for try in $(seq 1 300); do
+ ensure_api || { echo API_DOWN; sleep 20; continue; }
+ done_n=$(completed_count)
+ echo "try=$try terminal=$done_n target=$TARGET $(date -u +%H:%M:%S)"
+ if [[ "$done_n" -ge "$TARGET" ]]; then
+ break
+ fi
+ if [[ "$done_n" -eq "$prev_n" ]]; then
+ stalls=$((stalls + 1))
+ if [[ "$stalls" -ge "$STALL_LIMIT" ]]; then
+ echo "INGEST_STALLED terminal=$done_n (no progress ${stalls} tries) — proceed to eval"
+ break
+ fi
+ else
+ stalls=0
+ fi
+ prev_n=$done_n
+ ./beam.sh ingest \
+ --size 10M --sample "$SAMPLE_ID" \
+ --run "$RUN_ID" \
+ --brain "$BRAIN_ID" \
+ --concurrency "$BEAM_INGEST_CONCURRENCY" \
+ --timeout "$BEAM_INGEST_TIMEOUT"
+ sleep 2
+done
+
+done_n=$(completed_count)
+echo "INGEST_DONE=$done_n"
+kill "$WD_PID" 2>/dev/null
+wait "$WD_PID" 2>/dev/null
+
+if [[ "$done_n" -lt "$MIN_EVAL_TURNS" ]]; then
+ echo "INGEST_INCOMPLETE abort eval ($done_n/$N_TURNS target=$TARGET)"
+ date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/beam-10m-1.DONE
+ exit 2
+fi
+
+for ab in \
+ abstention \
+ contradiction_resolution \
+ event_ordering \
+ information_extraction \
+ instruction_following \
+ knowledge_update \
+ multi_session_reasoning \
+ preference_following \
+ summarization \
+ temporal_reasoning
+do
+ echo "==== EVAL $ab $(date -u +%H:%M:%S) ===="
+ ensure_api
+ ./beam.sh evaluate \
+ --size 10M --sample "$SAMPLE_ID" \
+ --run "$EVAL_RUN_ID" \
+ --brain "$BRAIN_ID" \
+ --profile product \
+ --abilities "$ab" \
+ --concurrency 1 \
+ --historical-limit 12 --max-passages 12 --max-facts 40 \
+ --no-ppr
+done
+./beam.sh report --run "$EVAL_RUN_ID"
+.venv/bin/python - <<PY
+import json, statistics, re
+from pathlib import Path
+rows=[json.loads(l) for l in Path('runs/${RUN_ID}/ingest.jsonl').read_text().splitlines() if l.strip()]
+last={}
+for r in rows: last[r.get('unit_id')]=r
+pat=re.compile(r'maximum context length is 8192', re.I)
+comp=[r for r in last.values() if r.get('status')=='completed' and (r.get('cost') or {}).get('llm_source_multiplier') is not None]
+mult=[r['cost']['llm_source_multiplier'] for r in comp]
+perm=sorted(
+ uid for uid,r in last.items()
+ if r.get('status')=='permanent_failed'
+ or (r.get('status')=='failed' and pat.search(r.get('error') or ''))
+)
+print('MULTIPLIER n', len(mult),
+ 'median', round(statistics.median(mult),1) if mult else None,
+ 'mean', round(statistics.mean(mult),1) if mult else None)
+print('PERMANENT_FAILS', perm)
+rep_path=Path('runs/${EVAL_RUN_ID}/report.json')
+if rep_path.exists():
+ rep=json.loads(rep_path.read_text())
+ m=rep.get('metrics') or {}
+ print('HEADLINE', m.get('headline_score'))
+ print('PER', m.get('per_ability'))
+PY
+date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/beam-10m-1.DONE
+echo MULTI_DONE
diff --git a/docs/research/10-beam-protocol.md b/docs/research/10-beam-protocol.md
index 06121e3..e9109c7 100644
--- a/docs/research/10-beam-protocol.md
+++ b/docs/research/10-beam-protocol.md
@@ -8,14 +8,38 @@ This is **not** a port of LIGHT. BrainAPI replaces RAG/LIGHT via `POST /ingest/`
| Item | Value |
| --- | --- |
-| Source | HuggingFace [`Mohammadta/BEAM`](https://huggingface.co/datasets/Mohammadta/BEAM) |
+| Source (≤1M) | HuggingFace [`Mohammadta/BEAM`](https://huggingface.co/datasets/Mohammadta/BEAM) |
+| Source (10M) | HuggingFace [`Mohammadta/BEAM-10M`](https://huggingface.co/datasets/Mohammadta/BEAM-10M) |
| License | CC BY-SA 4.0 |
-| Splits (v1) | `100K`, `500K`, `1M` (paper “128K” ≡ HF `100K`) |
-| Deferred | [`Mohammadta/BEAM-10M`](https://huggingface.co/datasets/Mohammadta/BEAM-10M) |
+| Splits | `100K`, `500K`, `1M`, `10M` (paper “128K” ≡ HF `100K`) |
| Local path | `benchmarks/data/beam/{size}/{conversation_id}/` |
+| Env overrides | `BENCH_BEAM_HF_DATASET`, `BENCH_BEAM_HF_DATASET_10M` |
Do **not** ingest probing questions, rubrics, or ideal answers.
+### BEAM-10M multi-plan normalize
+
+Paper + HF schema: each 10M row is ~10 interlocking plans (`plan-1`…`plan-10` / `plans[]`), generated sequentially so the narrative is chronological.
+
+**Harness decision:** concatenate plans in plan-number order into **one** long `chat.json`, then ingest with the same turn units as 1M (`bN_tM`, `session_N`, `time_anchor` stamps).
+
+| Choice | Why |
+| --- | --- |
+| Concatenate chronologically (not parallel brains) | Matches sequential plan expansion in the paper; probing spans the full dialogue |
+| Prefer main `chat` plan-* keys when present | HF already ships per-plan batches with `turns`; fall back to `plans[].chat` + `convert_chat_batches` |
+| Global `batch_number` renumber | Keeps unit ids unique under existing `bN_tM` resume keys |
+| `brain_id` | `beam10m1`; clean product claims use `--brain beam10m1clean` (never wipe `beam1m1clean`) |
+
+`meta.json` records `normalize_strategy: concatenate_plans_chronological`, `n_plans`, per-plan batch ranges, and `ingest_target` (`n_turns - 5`, floor 1). **Do not invent ETA/TARGET without reading `n_turns` after download.**
+
+```bash
+./beam.sh download --size 10M
+./beam.sh dataset-stats --size 10M
+# then smoke / ingest with --size 10M --sample 1 --brain beam10m1clean --run beam-10m-1-clean
+```
+
+Still requires `benchmarks/.env`, a live API + celery, and an **explicit** go before long ingest (see `scripts/run_beam_10m_1_ingest_eval.sh`).
+
## Pipeline
```text
@@ -26,7 +50,7 @@ Do **not** ingest probing questions, rubrics, or ideal answers.
./beam.sh report --run beam-100k-1
```
-Brain IDs: `beam{size}{convid}` → `beam100k1`.
+Brain IDs: `beam{size}{convid}` → `beam100k1`, `beam10m1`.
Ingest unit = one batch turn (user + assistant messages), with `time_anchor` as `source_timestamp` when present.
@@ -41,6 +65,8 @@ Ingest unit = one batch turn (user + assistant messages), with `time_anchor` as
**ETA (1M/1, ~625 turns):** prior run ≈ **40 h** at concurrency **1**. Expect roughly **20–25 h** at **2**, **14–18 h** at **3** (contention / uvicorn restarts can erase ideal linear speedup).
+**10M:** derive TARGET from `meta.json` `n_turns` / `ingest_target` after normalize — do not scale 1M hours blindly.
+
**Caveats:** concurrent units on one brain contend on Neo4j/Postgres and can knock over uvicorn under load. Keepalive must only restart when port **8000** is not LISTENing (never `pkill` a healthy API while evaluate-only is running). Prefer a **new brain id** for any full re-ingest; do not wipe brains mid-eval.
Example next 1M re-ingest (new brain; do not start casually):
@@ -69,7 +95,7 @@ abstention, contradiction_resolution, event_ordering, information_extraction, in
## Cost notes
-100K chats average ~100+ turns; accurate ingest is slow/expensive. Start with `--limit-turns` / `--limit` and a single sample. Rubric judging can issue several LLM calls per question.
+100K chats average ~100+ turns; accurate ingest is slow/expensive. Start with `--limit-turns` / `--limit` and a single sample. Rubric judging can issue several LLM calls per question. 10M is far larger — count turns first.
## Citation