BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
Add BEAM 10M harness: plan concat normalize and campaign scripts.
Christiancf61c99b8097
diff --git a/benchmarks/.env.example b/benchmarks/.env.exampleindex 8d0c17e..5cbb772 100644--- a/benchmarks/.env.example+++ b/benchmarks/.env.example@@ -32,3 +32,7 @@ BENCH_ANSWER_MODEL=deepseek-v4-flash# OPENAI_API_KEY=# BENCH_LLM_BASE_URL=https://api.openai.com/v1# BENCH_ANSWER_MODEL=gpt-4o-mini++# BEAM HF datasets (defaults):+# BENCH_BEAM_HF_DATASET=Mohammadta/BEAM+# BENCH_BEAM_HF_DATASET_10M=Mohammadta/BEAM-10Mdiff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.mdindex f5bcf0e..04f8507 100644--- a/benchmarks/AGENTS.md+++ b/benchmarks/AGENTS.md@@ -22,12 +22,14 @@ Typical flow: `download` → `ingest` → `evaluate` → `report` via `./longmemTypical flow: `download` → `ingest` → `evaluate` → `report` via `./beam.sh` or `.venv/bin/python -m beam`.-- Dataset: HuggingFace `Mohammadta/BEAM` splits `100K` / `500K` / `1M` (BEAM-10M deferred).+- Dataset: HuggingFace `Mohammadta/BEAM` (`100K` / `500K` / `1M`) and `Mohammadta/BEAM-10M` (`10M`). Harness supports all four sizes in `CHAT_SIZES`.+- 10M normalize: concatenate interlocking plans chronologically into one chat (`bN_tM` / `session_N`); see [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md) § BEAM-10M. Brain `beam10m1` / `--brain beam10m1clean`; never wipe `beam1m1clean`.+- Download sample 1 once `.env` exists: `./beam.sh download --size 10M` then `./beam.sh dataset-stats --size 10M` (read `n_turns` before any ETA). Ingest/eval still need live API + explicit go (`scripts/run_beam_10m_1_ingest_eval.sh`).- Ingest turns only — never probing questions, rubrics, or ideal answers.- Scoring: rubric LLM-judge (`judge_prompt_variant: beam-rubric-v1-question-aware`); headline = mean of 10 ability means; `event_ordering` uses `tau_norm`.-- Brain IDs: `beam{size}{convid}` → e.g. `beam100k1`.+- Brain IDs: `beam{size}{convid}` → e.g. `beam100k1`, `beam10m1`.- Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md).-- Ingest parallelism: prefer `--concurrency 2` (≤4); resume skips permanent embed-8192 fails. See protocol “Parallel ingest” + `scripts/run_beam_1m_1_ingest_eval.sh` (`BEAM_INGEST_CONCURRENCY`). Do not wipe live eval brains.+- Ingest parallelism: prefer `--concurrency 2` (≤4); resume skips permanent embed-8192 fails. See protocol “Parallel ingest” + `scripts/run_beam_1m_1_ingest_eval.sh` / `scripts/run_beam_10m_1_ingest_eval.sh` (`BEAM_INGEST_CONCURRENCY`). Do not wipe live eval brains.## Results ledger (`REPORTS.json`)diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 43a2fad..69ea41e 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -278,14 +278,14 @@ Do **not** ingest gold answers or `has_answer` turn labels.# BEAM benchmark for BrainAPI-Sibling harness for [BEAM](https://github.com/mohammadtavakoli78/BEAM) (ICLR 2026). Evaluates BrainAPI as long-term memory on multi-scale chats (`100K` / `500K` / `1M`). BEAM-10M is deferred.+Sibling harness for [BEAM](https://github.com/mohammadtavakoli78/BEAM) (ICLR 2026). Evaluates BrainAPI as long-term memory on multi-scale chats (`100K` / `500K` / `1M` / `10M`).-1. Downloads HuggingFace `Mohammadta/BEAM` and normalizes under `data/beam/`-2. Ingests each batch turn into a brain (`beam100k1`, …)+1. Downloads HuggingFace `Mohammadta/BEAM` (and `Mohammadta/BEAM-10M` for `--size 10M`) and normalizes under `data/beam/`+2. Ingests each batch turn into a brain (`beam100k1`, `beam10m1`, …)3. Answers probing questions via `POST /retrieve/context` + LLM4. Scores with BEAM rubric LLM-judge (10 abilities); headline = mean of ability means-Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md).+Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protocol.md). 10M rows concatenate interlocking plans chronologically into one chat (same `bN_tM` / `session_N` path as 1M).## Quickstart@@ -299,12 +299,21 @@ Protocol: [`docs/research/10-beam-protocol.md`](../docs/research/10-beam-protoco./beam.sh selftest```+10M (after `benchmarks/.env` + live API; do not start long ingest casually):++```bash+./beam.sh download --size 10M+./beam.sh dataset-stats --size 10M # read n_turns / ingest_target first+./beam.sh smoke --size 10M --sample 1 --limit-turns 2+# durable campaign: scripts/boot_beam_10m_screen.sh + beam_10m_keepalive.sh+```+### Useful flags-- `--size 100K|500K|1M`+- `--size 100K|500K|1M|10M`- `--sample <id>` (repeatable; id or `size/id`)- `--limit-turns N` / `--limit N` / `--abilities a,b,…`-- `--concurrency N` (ingest default 2; BEAM 1M prefer 2–3, ≤4; keep ≤ `CELERY_WORKER_CONCURRENCY`)+- `--concurrency N` (ingest default 2; BEAM 1M/10M prefer 2–3, ≤4; keep ≤ `CELERY_WORKER_CONCURRENCY`)- `--dry-run` / `--no-resume` (resume skips completed + permanent embed-8192 fails)### Scoringdiff --git a/benchmarks/beam/cli.py b/benchmarks/beam/cli.pyindex 2548d2e..8176c78 100644--- a/benchmarks/beam/cli.py+++ b/benchmarks/beam/cli.py@@ -73,7 +73,9 @@ def cmd_dataset_stats(args: argparse.Namespace, settings: Settings) -> int:)console.print(table)assert brain_id_for("100K", "1") == "beam100k1"+ assert brain_id_for("10M", "1") == "beam10m1"console.print("[dim]brain_id_for('100K', '1') == 'beam100k1' OK[/dim]")+ console.print("[dim]brain_id_for('10M', '1') == 'beam10m1' OK[/dim]")return 0@@ -239,6 +241,8 @@ def cmd_selftest(args: argparse.Namespace, settings: Settings) -> int:errors.append("brain_id_for('100K','1') != 'beam100k1'")if brain_id_for("1M", "12") != "beam1m12":errors.append("brain_id_for('1M','12') != 'beam1m12'")+ if brain_id_for("10M", "1") != "beam10m1":+ errors.append("brain_id_for('10M','1') != 'beam10m1'")prompt = build_rubric_judge_prompt("When does my sprint end?",@@ -311,7 +315,7 @@ def build_parser() -> argparse.ArgumentParser:"--size",choices=CHAT_SIZES,default=None,- help="Download one size only (default: all 100K/500K/1M)",+ help="Download one size only (default: all 100K/500K/1M/10M)",)p_download.add_argument("--force",diff --git a/benchmarks/beam/config.py b/benchmarks/beam/config.pyindex a509ccf..538b177 100644--- a/benchmarks/beam/config.py+++ b/benchmarks/beam/config.py@@ -11,12 +11,19 @@ DATA_DIR = BENCHMARKS_ROOT / "data"RUNS_DIR = BENCHMARKS_ROOT / "runs"DEFAULT_BEAM_DATA_DIR = DATA_DIR / "beam"DEFAULT_HF_DATASET = "Mohammadta/BEAM"+DEFAULT_HF_DATASET_10M = "Mohammadta/BEAM-10M"DEFAULT_LLM_BASE_URL = "https://api.deepseek.com"DEFAULT_ANSWER_MODEL = "deepseek-v4-flash"DEFAULT_AZURE_JUDGE_MODEL = "gpt-4o"DEFAULT_AZURE_API_VERSION = "2024-12-01-preview"-CHAT_SIZES = ("100K", "500K", "1M")+CHAT_SIZES = ("100K", "500K", "1M", "10M")+HF_DATASET_BY_SIZE = {+ "100K": DEFAULT_HF_DATASET,+ "500K": DEFAULT_HF_DATASET,+ "1M": DEFAULT_HF_DATASET,+ "10M": DEFAULT_HF_DATASET_10M,+}ABILITY_NAMES = ("abstention","contradiction_resolution",@@ -70,6 +77,7 @@ class Settings:judge_azure_api_version: strbeam_data_dir: Pathhf_dataset: str+ hf_dataset_10m: strruns_dir: Pathbench_profile: str = "product"sc_samples: int = 1@@ -78,6 +86,11 @@ class Settings:answer_azure_endpoint: str | None = Noneanswer_azure_api_version: str = DEFAULT_AZURE_API_VERSION+ def hf_dataset_for(self, size: str) -> str:+ if size == "10M":+ return self.hf_dataset_10m+ return self.hf_dataset+@classmethoddef load(cls, env_file: Path | None = None) -> "Settings":if env_file is None:@@ -143,6 +156,9 @@ class Settings:os.getenv("BENCH_BEAM_DATA_DIR", str(DEFAULT_BEAM_DATA_DIR))),hf_dataset=os.getenv("BENCH_BEAM_HF_DATASET", DEFAULT_HF_DATASET),+ hf_dataset_10m=os.getenv(+ "BENCH_BEAM_HF_DATASET_10M", DEFAULT_HF_DATASET_10M+ ),runs_dir=Path(os.getenv("BENCH_RUNS_DIR", str(RUNS_DIR))),bench_profile=str(defaults["bench_profile"]),sc_samples=max(1, int(defaults["sc_samples"])),diff --git a/benchmarks/beam/dataset.py b/benchmarks/beam/dataset.pyindex 525678a..eb6e6eb 100644--- a/benchmarks/beam/dataset.py+++ b/benchmarks/beam/dataset.py@@ -8,6 +8,10 @@ from typing import Anyfrom beam.config import ABILITY_NAMES, CHAT_SIZES, Settings+_PLAN_KEY_RE = re.compile(r"^plan[-_]?(\d+)$", re.IGNORECASE)+NORMALIZE_STRATEGY_SINGLE = "single_chat"+NORMALIZE_STRATEGY_PLANS = "concatenate_plans_chronological"+def brain_id_for(size: str, conversation_id: str) -> str:slug = re.sub(r"[^a-zA-Z0-9]", "", f"beam{size}{conversation_id}")@@ -79,6 +83,171 @@ def convert_chat_batches(chat: list[Any]) -> list[dict[str, Any]]:return json_object+def plan_sort_key(label: Any) -> tuple[int, str]:+ text = str(label or "").strip()+ match = _PLAN_KEY_RE.match(text)+ if match:+ return (int(match.group(1)), text)+ if text.isdigit():+ return (int(text), text)+ return (10**9, text)+++def _as_mapping(value: Any) -> dict[str, Any] | None:+ if isinstance(value, dict):+ return value+ if value is None:+ return None+ keys = getattr(value, "keys", None)+ if callable(keys):+ try:+ return {str(k): value[k] for k in keys()}+ except Exception:+ return None+ return None+++def _iter_plan_keyed_entries(chat: Any) -> list[tuple[str, Any]]:+ """Yield (plan_id, payload) from BEAM-10M main chat (plan-1..plan-10)."""+ mapping = _as_mapping(chat)+ if mapping is not None:+ entries = [+ (key, mapping[key])+ for key in mapping+ if _PLAN_KEY_RE.match(str(key).strip())+ ]+ entries.sort(key=lambda item: plan_sort_key(item[0]))+ return entries+ if isinstance(chat, list):+ collected: list[tuple[str, Any]] = []+ for item in chat:+ nested = _as_mapping(item)+ if nested is None:+ continue+ for key, payload in nested.items():+ if _PLAN_KEY_RE.match(str(key).strip()):+ collected.append((str(key), payload))+ collected.sort(key=lambda item: plan_sort_key(item[0]))+ return collected+ return []+++def _batches_from_chat_payload(value: Any) -> list[dict[str, Any]]:+ """Accept already-normalized batches or raw HF message-list chat."""+ if not isinstance(value, list) or not value:+ return []+ first = value[0]+ if isinstance(first, dict) and "turns" in first:+ return [row for row in value if isinstance(row, dict) and row.get("turns")]+ if isinstance(first, list):+ return [row for row in convert_chat_batches(value) if row.get("turns")]+ return []+++def concatenate_plan_batches(+ plan_batches: list[tuple[str, list[dict[str, Any]]]],+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:+ """+ Merge plan chats in chronological order into one chat.json-compatible list.++ Reassigns batch_number globally so unit ids remain unique under the existing+ bN_tM / session_N conventions used by the 1M ingest path.+ """+ out: list[dict[str, Any]] = []+ plan_meta: list[dict[str, Any]] = []+ next_batch = 1+ for plan_id, batches in plan_batches:+ start = next_batch+ n_turns = 0+ for batch in batches:+ turns = batch.get("turns") or []+ if not isinstance(turns, list) or not turns:+ continue+ n_turns += len(turns)+ out.append(+ {+ "batch_number": next_batch,+ "turns": turns,+ "time_anchor": batch.get("time_anchor"),+ "plan_id": plan_id,+ }+ )+ next_batch += 1+ if next_batch > start:+ plan_meta.append(+ {+ "plan_id": plan_id,+ "batch_number_start": start,+ "batch_number_end": next_batch - 1,+ "n_batches": next_batch - start,+ "n_turns": n_turns,+ }+ )+ return out, plan_meta+++def extract_normalized_chat(+ row: dict[str, Any],+ *,+ size: str,+) -> tuple[list[dict[str, Any]], dict[str, Any]]:+ """+ Build normalized chat batches for a HF row.++ BEAM-10M: concatenate plan-1..plan-10 (or plans[]) chronologically into one+ long chat. Smaller sizes: single-chat convert_chat_batches path.+ """+ chat = row.get("chat")+ plans = row.get("plans")+ plan_entries = _iter_plan_keyed_entries(chat)+ use_multi_plan = size == "10M" or bool(plan_entries) or (+ isinstance(plans, list) and bool(plans)+ )++ if use_multi_plan:+ ordered: list[tuple[str, list[dict[str, Any]]]] = []+ if plan_entries:+ for plan_id, payload in plan_entries:+ batches = _batches_from_chat_payload(payload)+ if batches:+ ordered.append((str(plan_id), batches))+ source = "chat_plan_keys"+ else:+ source = "plans_array"+ plan_rows = [p for p in (plans or []) if isinstance(p, dict)]+ plan_rows.sort(+ key=lambda p: plan_sort_key(p.get("plan_id") or p.get("id") or "")+ )+ for index, plan in enumerate(plan_rows):+ plan_id = str(+ plan.get("plan_id") or plan.get("id") or f"plan-{index + 1}"+ )+ batches = _batches_from_chat_payload(plan.get("chat"))+ if batches:+ ordered.append((plan_id, batches))++ if ordered:+ chat_json, plan_meta = concatenate_plan_batches(ordered)+ return chat_json, {+ "normalize_strategy": NORMALIZE_STRATEGY_PLANS,+ "n_plans": len(plan_meta),+ "plans": plan_meta,+ "plan_source": source,+ }++ chat_list = chat if isinstance(chat, list) else []+ return convert_chat_batches(chat_list), {+ "normalize_strategy": NORMALIZE_STRATEGY_SINGLE,+ }+++def ingest_target_turns(n_turns: int, *, slack: int = 5) -> int:+ """Ops TARGET from normalized turn count (never invent without n_turns)."""+ if n_turns <= 0:+ return 0+ return max(1, n_turns - max(0, slack))++def _parse_probing_questions(raw: Any) -> dict[str, list[dict[str, Any]]]:if isinstance(raw, dict):data = raw@@ -106,11 +275,23 @@ def write_conversation(size: str,conversation_id: str,topic: Any,- chat: list[Any],+ chat: list[Any] | None = None,probing_questions: Any,+ plans: Any = None,+ row: dict[str, Any] | None = None,) -> Path:dest.mkdir(parents=True, exist_ok=True)- chat_json = convert_chat_batches(chat)+ if row is not None:+ chat_json, extra_meta = extract_normalized_chat(row, size=size)+ if topic is None:+ topic = row.get("conversation_seed")+ if probing_questions is None:+ probing_questions = row.get("probing_questions")+ else:+ chat_json, extra_meta = extract_normalized_chat(+ {"chat": chat or [], "plans": plans},+ size=size,+ )probing = _parse_probing_questions(probing_questions)(dest / "chat.json").write_text(json.dumps(chat_json, ensure_ascii=False, indent=2),@@ -125,14 +306,17 @@ def write_conversation(json.dumps(topic, ensure_ascii=False, indent=2),encoding="utf-8",)+ n_turns = sum(len(b.get("turns") or []) for b in chat_json)meta = {"conversation_id": str(conversation_id),"size": size,"sample_id": sample_id_for(size, str(conversation_id)),"brain_id": brain_id_for(size, str(conversation_id)),"n_batches": len(chat_json),- "n_turns": sum(len(b.get("turns") or []) for b in chat_json),+ "n_turns": n_turns,"n_questions": sum(len(v) for v in probing.values()),+ "ingest_target": ingest_target_turns(n_turns),+ **extra_meta,}(dest / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2),@@ -163,7 +347,8 @@ def download_and_normalize(settings.beam_data_dir.mkdir(parents=True, exist_ok=True)for size in selected:- conversations = load_dataset(settings.hf_dataset, split=size)+ dataset_name = settings.hf_dataset_for(size)+ conversations = load_dataset(dataset_name, split=size)for conversation in conversations:conversation_id = str(conversation["conversation_id"])dest = conversation_dir(@@ -175,13 +360,14 @@ def download_and_normalize(and (dest / "probing_questions.json").exists()):continue+ row = dict(conversation)write_conversation(dest,size=size,conversation_id=conversation_id,- topic=conversation.get("conversation_seed"),- chat=list(conversation.get("chat") or []),- probing_questions=conversation.get("probing_questions"),+ topic=row.get("conversation_seed"),+ probing_questions=row.get("probing_questions"),+ row=row,)return settings.beam_data_dirdiff --git a/benchmarks/beam/provenance.py b/benchmarks/beam/provenance.pyindex 42d35b2..5212511 100644--- a/benchmarks/beam/provenance.py+++ b/benchmarks/beam/provenance.py@@ -89,7 +89,9 @@ def build_provenance(settings: Settings, *, size: str | None = None) -> dict[str"judge_model_family": model_family(settings.judge_model),"judge_provider": settings.judge_provider(),"judge_shares_answer_family": settings.judge_shares_answer_family,- "hf_dataset": settings.hf_dataset,+ "hf_dataset": (+ settings.hf_dataset_for(size) if size else settings.hf_dataset+ ),"bench_profile": settings.bench_profile,"sc_samples": settings.sc_samples,"sc_temperature": settings.sc_temperature,diff --git a/benchmarks/beam/test_beam_helpers.py b/benchmarks/beam/test_beam_helpers.pyindex 4b5401e..3d70a06 100644--- a/benchmarks/beam/test_beam_helpers.py+++ b/benchmarks/beam/test_beam_helpers.py@@ -2,7 +2,16 @@ from __future__ import annotationsfrom pathlib import Path-from beam.dataset import brain_id_for, convert_chat_batches, format_turn_text+from beam.dataset import (+ brain_id_for,+ concatenate_plan_batches,+ convert_chat_batches,+ extract_normalized_chat,+ format_turn_text,+ ingest_target_turns,+ iter_ingest_units,+ write_conversation,+)from beam.ingest import (completed_unit_ids,is_permanent_ingest_error,@@ -18,6 +27,8 @@ def test_brain_id_for() -> None:assert brain_id_for("100K", "1") == "beam100k1"assert brain_id_for("500K", "3") == "beam500k3"assert brain_id_for("1M", "12") == "beam1m12"+ assert brain_id_for("10M", "1") == "beam10m1"+ assert brain_id_for("10M", "1clean") == "beam10m1clean"def test_convert_chat_batches_splits_on_main_question() -> None:@@ -45,6 +56,179 @@ def test_convert_chat_batches_splits_on_main_question() -> None:assert batches[0]["turns"][1][0]["content"] == "Q2"+def _raw_turn(content: str, *, anchor: str = "March-15-2024") -> list[dict]:+ return [+ {+ "role": "user",+ "content": content,+ "question_type": "main_question",+ "time_anchor": anchor,+ },+ {"role": "assistant", "content": f"re:{content}"},+ ]+++def test_extract_normalized_chat_10m_plan_keys() -> None:+ row = {+ "chat": {+ "plan-2": [+ {+ "batch_number": 1,+ "time_anchor": None,+ "turns": [_raw_turn("p2")],+ }+ ],+ "plan-1": [+ {+ "batch_number": 1,+ "time_anchor": None,+ "turns": [_raw_turn("p1a"), _raw_turn("p1b")],+ }+ ],+ },+ "plans": [],+ "probing_questions": {"summarization": [{"question": "Sum?", "rubric": ["x"]}]},+ }+ chat_json, meta = extract_normalized_chat(row, size="10M")+ assert meta["normalize_strategy"] == "concatenate_plans_chronological"+ assert meta["n_plans"] == 2+ assert meta["plan_source"] == "chat_plan_keys"+ assert [b["plan_id"] for b in chat_json] == ["plan-1", "plan-2"]+ assert [b["batch_number"] for b in chat_json] == [1, 2]+ assert len(chat_json[0]["turns"]) == 2+ assert chat_json[0]["turns"][0][0]["content"] == "p1a"+++def test_extract_normalized_chat_10m_plans_array_raw() -> None:+ row = {+ "chat": None,+ "plans": [+ {+ "plan_id": "plan-10",+ "chat": [+ [+ {+ "role": "user",+ "content": "late",+ "question_type": "main_question",+ "time_anchor": "April-01-2024",+ },+ {"role": "assistant", "content": "re:late"},+ ]+ ],+ },+ {+ "plan_id": "plan-1",+ "chat": [+ [+ {+ "role": "user",+ "content": "early-a",+ "question_type": "main_question",+ "time_anchor": "March-15-2024",+ },+ {"role": "assistant", "content": "a"},+ {+ "role": "user",+ "content": "early-b",+ "question_type": "main_question",+ "time_anchor": "March-16-2024",+ },+ {"role": "assistant", "content": "b"},+ ]+ ],+ },+ ],+ }+ chat_json, meta = extract_normalized_chat(row, size="10M")+ assert meta["normalize_strategy"] == "concatenate_plans_chronological"+ assert meta["plan_source"] == "plans_array"+ assert [b["plan_id"] for b in chat_json] == ["plan-1", "plan-10"]+ assert [b["batch_number"] for b in chat_json] == [1, 2]+ sample = {+ "sample_id": "10M/1",+ "size": "10M",+ "conversation_id": "1",+ "chat": chat_json,+ "probing_questions": {},+ }+ units = iter_ingest_units(sample)+ assert len(units) == 3+ assert [u["unit_id"] for u in units] == ["b1_t1", "b1_t2", "b2_t1"]+ assert [u["session_key"] for u in units] == [+ "session_1",+ "session_2",+ "session_3",+ ]+ assert "early-a" in units[0]["text"]+ assert "late" in units[2]["text"]+++def test_concatenate_plan_batches_renumbers_globally() -> None:+ chat_json, plan_meta = concatenate_plan_batches(+ [+ (+ "plan-1",+ [+ {"turns": [_raw_turn("a")], "time_anchor": None},+ {"turns": [_raw_turn("b")], "time_anchor": None},+ ],+ ),+ ("plan-2", [{"turns": [_raw_turn("c")], "time_anchor": None}]),+ ]+ )+ assert [b["batch_number"] for b in chat_json] == [1, 2, 3]+ assert plan_meta[0]["batch_number_start"] == 1+ assert plan_meta[0]["batch_number_end"] == 2+ assert plan_meta[1]["batch_number_start"] == 3+ assert ingest_target_turns(100) == 95+ assert ingest_target_turns(0) == 0+++def test_write_conversation_10m_fixture(tmp_path: Path) -> None:+ row = {+ "conversation_id": "1",+ "conversation_seed": {"title": "Fixture 10M"},+ "chat": {+ "plan-1": [+ {+ "batch_number": 1,+ "turns": [_raw_turn("hello")],+ "time_anchor": None,+ }+ ]+ },+ "plans": [],+ "probing_questions": {+ "information_extraction": [{"question": "What?", "rubric": ["hello"]}]+ },+ }+ dest = tmp_path / "10M" / "1"+ write_conversation(+ dest,+ size="10M",+ conversation_id="1",+ topic=row["conversation_seed"],+ probing_questions=row["probing_questions"],+ row=row,+ )+ meta = __import__("json").loads((dest / "meta.json").read_text(encoding="utf-8"))+ assert meta["brain_id"] == "beam10m1"+ assert meta["n_turns"] == 1+ assert meta["ingest_target"] == 1+ assert meta["normalize_strategy"] == "concatenate_plans_chronological"+ chat = __import__("json").loads((dest / "chat.json").read_text(encoding="utf-8"))+ units = iter_ingest_units(+ {+ "sample_id": "10M/1",+ "chat": chat,+ "probing_questions": {},+ }+ )+ assert len(units) == 1+ assert units[0]["unit_id"] == "b1_t1"++def test_format_turn_text_normalizes_time_anchor() -> None:text, ts = format_turn_text([diff --git a/benchmarks/scripts/beam_10m_keepalive.sh b/benchmarks/scripts/beam_10m_keepalive.shnew file mode 100755index 0000000..626fd2f--- /dev/null+++ b/benchmarks/scripts/beam_10m_keepalive.sh@@ -0,0 +1,52 @@+#!/usr/bin/env bash+# Tiny keep-alive for BEAM 10M: only ensures API LISTEN + celery worker. Never pkill.+# screen -dmS beam10m-keep bash benchmarks/scripts/beam_10m_keepalive.sh+#+# Safe under concurrent ingest: flock around starts; longer /docs probe.+# Does not restart a healthy LISTEN uvicorn (avoids killing live eval).+set +e+SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"+UV_LOG=/tmp/uvicorn-beam10m.log+CEL_LOG=/tmp/brainapi-beam-worker.log+LOCK_DIR=/tmp/beam10m-locks+mkdir -p "$LOCK_DIR"+CELERY_WORKER_CONCURRENCY="${CELERY_WORKER_CONCURRENCY:-4}"++echo "KEEPALIVE_START $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> /tmp/beam-10m-keepalive.log+while [[ ! -f /tmp/beam-10m-1.STOP ]]; do+ if ! lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ echo "API_RESTART $(date -u +%H:%M:%S)" >> /tmp/beam-10m-keepalive.log+ (+ flock -n 9 || exit 0+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ exit 0+ fi+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \+ >> "$UV_LOG" 2>&1 &+ echo $! > /tmp/beam10m-uvicorn.pid+ sleep 5+ ) 9>"$LOCK_DIR/uvicorn.lock"+ else+ code=$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' http://127.0.0.1:8000/docs || echo 000)+ if [[ "$code" != "200" && "$code" != "401" ]]; then+ echo "API_BUSY_OR_SLOW code=$code $(date -u +%H:%M:%S)" >> /tmp/beam-10m-keepalive.log+ fi+ fi+ if ! pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then+ echo "CELERY_RESTART $(date -u +%H:%M:%S)" >> /tmp/beam-10m-keepalive.log+ (+ flock -n 9 || exit 0+ if pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then+ exit 0+ fi+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads \+ --concurrency=${CELERY_WORKER_CONCURRENCY} \+ -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \+ >> "$CEL_LOG" 2>&1 &+ echo $! > /tmp/beam10m-celery.pid+ sleep 5+ ) 9>"$LOCK_DIR/celery.lock"+ fi+ sleep 20+done+echo "KEEPALIVE_STOP $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> /tmp/beam-10m-keepalive.logdiff --git a/benchmarks/scripts/boot_beam_10m_screen.sh b/benchmarks/scripts/boot_beam_10m_screen.shnew file mode 100755index 0000000..f5c3169--- /dev/null+++ b/benchmarks/scripts/boot_beam_10m_screen.sh@@ -0,0 +1,36 @@+#!/usr/bin/env bash+# Boot BEAM 10M ingest inside screen. Keep this file free of long uvicorn/celery+# command strings in the screen argv — use this script path only:+# screen -dmS beam10m /path/to/boot_beam_10m_screen.sh+set +e+SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"+UV_LOG=/tmp/uvicorn-beam10m.log+CEL_LOG=/tmp/brainapi-beam-worker.log+UV_PID=/tmp/beam10m-uvicorn.pid+CEL_PID=/tmp/beam10m-celery.pid+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"++start_api() {+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ return 0+ fi+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \+ >> "$UV_LOG" 2>&1 &+ echo $! > "$UV_PID"+ sleep 5+}++start_worker() {+ if pgrep -f 'celery -A src.workers.app worker' >/dev/null 2>&1; then+ return 0+ fi+ CELERY_C="${CELERY_WORKER_CONCURRENCY:-4}"+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads --concurrency=${CELERY_C} -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \+ >> "$CEL_LOG" 2>&1 &+ echo $! > "$CEL_PID"+ sleep 5+}++start_api+start_worker+exec bash "$ROOT/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh" >> /tmp/beam-10m-1-run.log 2>&1diff --git a/benchmarks/scripts/eval_beam_10m_1_product.sh b/benchmarks/scripts/eval_beam_10m_1_product.shnew file mode 100755index 0000000..1d66fa6--- /dev/null+++ b/benchmarks/scripts/eval_beam_10m_1_product.sh@@ -0,0 +1,97 @@+#!/usr/bin/env bash+# Evaluate-only BEAM 10M/1 product on existing brain (no ingest / no wipe).+# Never touches beam1m1clean.+set +e+ROOT="$(cd "$(dirname "$0")/.." && pwd)"+cd "$ROOT"+unset BRAINPAT_TOKEN BRAINAPI_TOKEN+set -a; source ./.env; set +a+export TERM=dumb PYTHONUNBUFFERED=1++SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"+UV_LOG=/tmp/uvicorn-beam10m.log+CEL_LOG=/tmp/brainapi-beam-worker.log+RUN_ID="${BEAM_10M_RUN:-beam-10m-1-clean}"+BRAIN_ID="${BEAM_10M_BRAIN:-beam10m1clean}"+EVAL_RUN_ID="${BEAM_10M_EVAL_RUN:-beam-10m-1-clean-product}"+SAMPLE_ID="${BEAM_10M_SAMPLE:-1}"++ensure_api() {+ for _ in 1 2 3 4 5 6 7 8; do+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ code=$(curl -s -o /dev/null --max-time 2 -w '%{http_code}' http://127.0.0.1:8000/docs || echo 000)+ if [[ "$code" == "200" || "$code" == "401" ]]; then+ if ! pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \+ >> "$CEL_LOG" 2>&1 &+ sleep 4+ fi+ return 0+ fi+ fi+ if ! lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \+ >> "$UV_LOG" 2>&1 &+ sleep 5+ fi+ done+ return 1+}++echo "START_10M_EVAL $(date -u +%Y-%m-%dT%H:%M:%SZ) brain=$BRAIN_ID eval_run=$EVAL_RUN_ID"+ensure_api || { echo FATAL_API_DOWN; exit 1; }++for ab in \+ abstention \+ contradiction_resolution \+ event_ordering \+ information_extraction \+ instruction_following \+ knowledge_update \+ multi_session_reasoning \+ preference_following \+ summarization \+ temporal_reasoning+do+ echo "==== EVAL $ab $(date -u +%H:%M:%S) ===="+ ensure_api+ ./beam.sh evaluate \+ --size 10M --sample "$SAMPLE_ID" \+ --run "$EVAL_RUN_ID" \+ --brain "$BRAIN_ID" \+ --profile product \+ --abilities "$ab" \+ --concurrency 1 \+ --historical-limit 12 --max-passages 12 --max-facts 40 \+ --no-ppr+done++./beam.sh report --run "$EVAL_RUN_ID"+.venv/bin/python - <<PY+import json, statistics+from pathlib import Path+ingest = Path('runs/${RUN_ID}/ingest.jsonl')+rows=[json.loads(l) for l in ingest.read_text().splitlines() if l.strip()] if ingest.exists() else []+last={}+for r in rows: last[r.get('unit_id')]=r+comp=[r for r in last.values() if r.get('status')=='completed' and (r.get('cost') or {}).get('llm_source_multiplier') is not None]+mult=[r['cost']['llm_source_multiplier'] for r in comp]+print('INGEST completed', sum(1 for r in last.values() if r.get('status')=='completed'), '/', len(last))+print('PERMANENT_FAILS', sorted(+ uid for uid,r in last.items()+ if r.get('status')=='permanent_failed'+ or (r.get('status')=='failed' and '8192' in (r.get('error') or ''))+))+print('MULTIPLIER n', len(mult),+ 'median', round(statistics.median(mult),1) if mult else None,+ 'mean', round(statistics.mean(mult),1) if mult else None)+rep_path=Path('runs/${EVAL_RUN_ID}/report.json')+if rep_path.exists():+ rep=json.loads(rep_path.read_text())+ m=rep.get('metrics') or {}+ hs=m.get('headline_score')+ print('HEADLINE', hs, f"({(hs or 0)*100:.1f}%)" if isinstance(hs,(int,float)) else '')+ print('PER', m.get('per_ability'))+PY+date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/beam-10m-1.DONE+echo EVAL_DONEdiff --git a/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh b/benchmarks/scripts/run_beam_10m_1_ingest_eval.shnew file mode 100755index 0000000..a34c9d7--- /dev/null+++ b/benchmarks/scripts/run_beam_10m_1_ingest_eval.sh@@ -0,0 +1,253 @@+#!/usr/bin/env bash+# Resilient BEAM 10M/1 ingest + product evaluate.+# Resume-safe. LISTEN-based ensure_api (never pkill healthy uvicorn).+# Uses flock so watchdog/ensure never double-bind :8000.+# NEVER touches beam1m1clean — uses beam10m1clean / beam-10m-1-clean only.+#+# TARGET comes from data/beam/10M/1/meta.json (ingest_target or n_turns-5).+# Do not invent TARGET without a normalized sample.+set +e+ROOT="$(cd "$(dirname "$0")/.." && pwd)"+cd "$ROOT"+unset BRAINPAT_TOKEN BRAINAPI_TOKEN+set -a; source ./.env; set +a+export TERM=dumb PYTHONUNBUFFERED=1++SRC="${BRAINAPI_SRC:-$HOME/.brainapi/source}"+UV_LOG=/tmp/uvicorn-beam10m.log+CEL_LOG=/tmp/brainapi-beam-worker.log+LOCK_DIR=/tmp/beam10m-locks+mkdir -p "$LOCK_DIR"++BEAM_INGEST_CONCURRENCY="${BEAM_INGEST_CONCURRENCY:-2}"+BEAM_INGEST_TIMEOUT="${BEAM_INGEST_TIMEOUT:-2400}"+CELERY_WORKER_CONCURRENCY="${CELERY_WORKER_CONCURRENCY:-4}"+RUN_ID="${BEAM_10M_RUN:-beam-10m-1-clean}"+BRAIN_ID="${BEAM_10M_BRAIN:-beam10m1clean}"+EVAL_RUN_ID="${BEAM_10M_EVAL_RUN:-beam-10m-1-clean-product}"+SAMPLE_ID="${BEAM_10M_SAMPLE:-1}"++api_ok() {+ lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1 || return 1+ code=$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' http://127.0.0.1:8000/docs || echo 000)+ [[ "$code" == "200" || "$code" == "401" ]]+}++start_uvicorn() {+ (+ flock -n 9 || return 0+ if lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ return 0+ fi+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/uvicorn src.services.api.app:app --host 127.0.0.1 --port 8000" \+ >> "$UV_LOG" 2>&1 &+ echo $! > /tmp/beam10m-uvicorn.pid+ sleep 5+ ) 9>"$LOCK_DIR/uvicorn.lock"+}++start_celery() {+ (+ flock -n 9 || return 0+ if pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then+ return 0+ fi+ nohup bash -c "cd '$SRC' && exec ./.venv/bin/celery -A src.workers.app worker --loglevel=info --pool=threads \+ --concurrency=${CELERY_WORKER_CONCURRENCY} \+ -Q ingest_data,finalize_ingestion,process_architect_relationships,consolidate_graph,chatbot_memory,ingest_file,ingest_structured_data" \+ >> "$CEL_LOG" 2>&1 &+ echo $! > /tmp/beam10m-celery.pid+ sleep 5+ ) 9>"$LOCK_DIR/celery.lock"+}++ensure_api() {+ for _ in 1 2 3 4 5 6 7 8 9 10 11 12; do+ if api_ok; then+ start_celery+ return 0+ fi+ start_uvicorn+ start_celery+ sleep 3+ done+ return 1+}++resolve_target() {+ .venv/bin/python - <<PY+import json+from pathlib import Path+meta = Path("data/beam/10M/${SAMPLE_ID}/meta.json")+if not meta.exists():+ print(0)+ raise SystemExit(0)+row = json.loads(meta.read_text(encoding="utf-8"))+target = row.get("ingest_target")+if target is None:+ n = int(row.get("n_turns") or 0)+ target = max(1, n - 5) if n else 0+print(int(target))+PY+}++terminal_count() {+ .venv/bin/python - <<PY+import json+import re+from pathlib import Path+p = Path("runs/${RUN_ID}/ingest.jsonl")+rows = [json.loads(l) for l in p.read_text().splitlines() if l.strip()] if p.exists() else []+last = {}+for r in rows:+ last[r.get("unit_id")] = r+pat = re.compile(r"maximum context length is 8192", re.I)+n = 0+for r in last.values():+ st = r.get("status")+ err = r.get("error") or ""+ if st in {"completed", "partial_failed", "permanent_failed"}:+ n += 1+ elif st == "failed" and pat.search(err):+ n += 1+print(n)+PY+}++completed_count() {+ terminal_count+}++watchdog() {+ while [[ ! -f /tmp/beam-10m-1.STOP ]]; do+ if ! lsof -iTCP:8000 -sTCP:LISTEN >/dev/null 2>&1; then+ echo "WATCHDOG_API_RESTART $(date -u +%H:%M:%S)"+ start_uvicorn+ fi+ if ! pgrep -f '/.venv/bin/celery -A src.workers.app worker' >/dev/null 2>&1; then+ echo "WATCHDOG_CELERY_RESTART $(date -u +%H:%M:%S)"+ start_celery+ fi+ sleep 30+ done+}++TARGET=$(resolve_target)+if [[ "$TARGET" -le 0 ]]; then+ echo "FATAL_NO_TARGET: normalize 10M sample first: ./beam.sh download --size 10M"+ echo "Then re-run; TARGET is read from data/beam/10M/${SAMPLE_ID}/meta.json"+ exit 1+fi++N_TURNS=$(.venv/bin/python - <<PY+import json+from pathlib import Path+meta = Path("data/beam/10M/${SAMPLE_ID}/meta.json")+print(json.loads(meta.read_text(encoding="utf-8")).get("n_turns", 0))+PY+)++STALL_LIMIT=3+prev_n=-1+stalls=0+MIN_EVAL_TURNS=$(( TARGET * 70 / 100 ))+if [[ "$MIN_EVAL_TURNS" -lt 1 ]]; then+ MIN_EVAL_TURNS=1+fi++rm -f /tmp/beam-10m-1.STOP /tmp/beam-10m-1.DONE+echo "START_10M_INGEST $(date -u +%Y-%m-%dT%H:%M:%SZ) concurrency=$BEAM_INGEST_CONCURRENCY celery_c=$CELERY_WORKER_CONCURRENCY target=$TARGET n_turns=$N_TURNS brain=$BRAIN_ID run=$RUN_ID"+ensure_api || { echo FATAL_API_DOWN; exit 1; }+watchdog &+WD_PID=$!+echo "watchdog_pid=$WD_PID"++for try in $(seq 1 300); do+ ensure_api || { echo API_DOWN; sleep 20; continue; }+ done_n=$(completed_count)+ echo "try=$try terminal=$done_n target=$TARGET $(date -u +%H:%M:%S)"+ if [[ "$done_n" -ge "$TARGET" ]]; then+ break+ fi+ if [[ "$done_n" -eq "$prev_n" ]]; then+ stalls=$((stalls + 1))+ if [[ "$stalls" -ge "$STALL_LIMIT" ]]; then+ echo "INGEST_STALLED terminal=$done_n (no progress ${stalls} tries) — proceed to eval"+ break+ fi+ else+ stalls=0+ fi+ prev_n=$done_n+ ./beam.sh ingest \+ --size 10M --sample "$SAMPLE_ID" \+ --run "$RUN_ID" \+ --brain "$BRAIN_ID" \+ --concurrency "$BEAM_INGEST_CONCURRENCY" \+ --timeout "$BEAM_INGEST_TIMEOUT"+ sleep 2+done++done_n=$(completed_count)+echo "INGEST_DONE=$done_n"+kill "$WD_PID" 2>/dev/null+wait "$WD_PID" 2>/dev/null++if [[ "$done_n" -lt "$MIN_EVAL_TURNS" ]]; then+ echo "INGEST_INCOMPLETE abort eval ($done_n/$N_TURNS target=$TARGET)"+ date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/beam-10m-1.DONE+ exit 2+fi++for ab in \+ abstention \+ contradiction_resolution \+ event_ordering \+ information_extraction \+ instruction_following \+ knowledge_update \+ multi_session_reasoning \+ preference_following \+ summarization \+ temporal_reasoning+do+ echo "==== EVAL $ab $(date -u +%H:%M:%S) ===="+ ensure_api+ ./beam.sh evaluate \+ --size 10M --sample "$SAMPLE_ID" \+ --run "$EVAL_RUN_ID" \+ --brain "$BRAIN_ID" \+ --profile product \+ --abilities "$ab" \+ --concurrency 1 \+ --historical-limit 12 --max-passages 12 --max-facts 40 \+ --no-ppr+done+./beam.sh report --run "$EVAL_RUN_ID"+.venv/bin/python - <<PY+import json, statistics, re+from pathlib import Path+rows=[json.loads(l) for l in Path('runs/${RUN_ID}/ingest.jsonl').read_text().splitlines() if l.strip()]+last={}+for r in rows: last[r.get('unit_id')]=r+pat=re.compile(r'maximum context length is 8192', re.I)+comp=[r for r in last.values() if r.get('status')=='completed' and (r.get('cost') or {}).get('llm_source_multiplier') is not None]+mult=[r['cost']['llm_source_multiplier'] for r in comp]+perm=sorted(+ uid for uid,r in last.items()+ if r.get('status')=='permanent_failed'+ or (r.get('status')=='failed' and pat.search(r.get('error') or ''))+)+print('MULTIPLIER n', len(mult),+ 'median', round(statistics.median(mult),1) if mult else None,+ 'mean', round(statistics.mean(mult),1) if mult else None)+print('PERMANENT_FAILS', perm)+rep_path=Path('runs/${EVAL_RUN_ID}/report.json')+if rep_path.exists():+ rep=json.loads(rep_path.read_text())+ m=rep.get('metrics') or {}+ print('HEADLINE', m.get('headline_score'))+ print('PER', m.get('per_ability'))+PY+date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/beam-10m-1.DONE+echo MULTI_DONEdiff --git a/docs/research/10-beam-protocol.md b/docs/research/10-beam-protocol.mdindex 06121e3..e9109c7 100644--- a/docs/research/10-beam-protocol.md+++ b/docs/research/10-beam-protocol.md@@ -8,14 +8,38 @@ This is **not** a port of LIGHT. BrainAPI replaces RAG/LIGHT via `POST /ingest/`| Item | Value || --- | --- |-| Source | HuggingFace [`Mohammadta/BEAM`](https://huggingface.co/datasets/Mohammadta/BEAM) |+| Source (≤1M) | HuggingFace [`Mohammadta/BEAM`](https://huggingface.co/datasets/Mohammadta/BEAM) |+| Source (10M) | HuggingFace [`Mohammadta/BEAM-10M`](https://huggingface.co/datasets/Mohammadta/BEAM-10M) || License | CC BY-SA 4.0 |-| Splits (v1) | `100K`, `500K`, `1M` (paper “128K” ≡ HF `100K`) |-| Deferred | [`Mohammadta/BEAM-10M`](https://huggingface.co/datasets/Mohammadta/BEAM-10M) |+| Splits | `100K`, `500K`, `1M`, `10M` (paper “128K” ≡ HF `100K`) || Local path | `benchmarks/data/beam/{size}/{conversation_id}/` |+| Env overrides | `BENCH_BEAM_HF_DATASET`, `BENCH_BEAM_HF_DATASET_10M` |Do **not** ingest probing questions, rubrics, or ideal answers.+### BEAM-10M multi-plan normalize++Paper + HF schema: each 10M row is ~10 interlocking plans (`plan-1`…`plan-10` / `plans[]`), generated sequentially so the narrative is chronological.++**Harness decision:** concatenate plans in plan-number order into **one** long `chat.json`, then ingest with the same turn units as 1M (`bN_tM`, `session_N`, `time_anchor` stamps).++| Choice | Why |+| --- | --- |+| Concatenate chronologically (not parallel brains) | Matches sequential plan expansion in the paper; probing spans the full dialogue |+| Prefer main `chat` plan-* keys when present | HF already ships per-plan batches with `turns`; fall back to `plans[].chat` + `convert_chat_batches` |+| Global `batch_number` renumber | Keeps unit ids unique under existing `bN_tM` resume keys |+| `brain_id` | `beam10m1`; clean product claims use `--brain beam10m1clean` (never wipe `beam1m1clean`) |++`meta.json` records `normalize_strategy: concatenate_plans_chronological`, `n_plans`, per-plan batch ranges, and `ingest_target` (`n_turns - 5`, floor 1). **Do not invent ETA/TARGET without reading `n_turns` after download.**++```bash+./beam.sh download --size 10M+./beam.sh dataset-stats --size 10M+# then smoke / ingest with --size 10M --sample 1 --brain beam10m1clean --run beam-10m-1-clean+```++Still requires `benchmarks/.env`, a live API + celery, and an **explicit** go before long ingest (see `scripts/run_beam_10m_1_ingest_eval.sh`).+## Pipeline```text@@ -26,7 +50,7 @@ Do **not** ingest probing questions, rubrics, or ideal answers../beam.sh report --run beam-100k-1```-Brain IDs: `beam{size}{convid}` → `beam100k1`.+Brain IDs: `beam{size}{convid}` → `beam100k1`, `beam10m1`.Ingest unit = one batch turn (user + assistant messages), with `time_anchor` as `source_timestamp` when present.@@ -41,6 +65,8 @@ Ingest unit = one batch turn (user + assistant messages), with `time_anchor` as**ETA (1M/1, ~625 turns):** prior run ≈ **40 h** at concurrency **1**. Expect roughly **20–25 h** at **2**, **14–18 h** at **3** (contention / uvicorn restarts can erase ideal linear speedup).+**10M:** derive TARGET from `meta.json` `n_turns` / `ingest_target` after normalize — do not scale 1M hours blindly.+**Caveats:** concurrent units on one brain contend on Neo4j/Postgres and can knock over uvicorn under load. Keepalive must only restart when port **8000** is not LISTENing (never `pkill` a healthy API while evaluate-only is running). Prefer a **new brain id** for any full re-ingest; do not wipe brains mid-eval.Example next 1M re-ingest (new brain; do not start casually):@@ -69,7 +95,7 @@ abstention, contradiction_resolution, event_ordering, information_extraction, in## Cost notes-100K chats average ~100+ turns; accurate ingest is slow/expensive. Start with `--limit-turns` / `--limit` and a single sample. Rubric judging can issue several LLM calls per question.+100K chats average ~100+ turns; accurate ingest is slow/expensive. Start with `--limit-turns` / `--limit` and a single sample. Rubric judging can issue several LLM calls per question. 10M is far larger — count turns first.## Citation