BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
docs(research): add BEAM/LongMemEval protocols and efficiency plans
Christian1025e0630d75
diff --git a/docs/research/00-scope-and-constraints.md b/docs/research/00-scope-and-constraints.mdindex 6a98b71..199619c 100644--- a/docs/research/00-scope-and-constraints.md+++ b/docs/research/00-scope-and-constraints.md@@ -34,6 +34,10 @@ Anything expensive must be either moved to write time or exposed only on the dee| `04-evaluation-and-applications.md` | Measurement validity, failure attribution, recommendation/agent-surface metrics || `05-temporal-truth.md` | Cross-cutting: how the system knows what is currently true, write time through read time || `07-event-graphrag-improvement-plan.md` | Active event-hub GraphRAG execution plan (post Phase 1); see also `06` Phase 4 |+| `10-ingest-cost-latency.md` | Accurate-mode ingest token/latency reduction without quality trade-offs |+| `11-architect-loop-efficiency-plan.md` | Architect/Janitor redesign for peer-competitive ingest token multipliers |+| `12-locomo-93-at-low-multiplier.md` | Dual goal: ≥93% LoCoMo SOTA at ~15–20× ingest median (no escalate storms) |+| `13-locomo-93-research-levers.md` | Evidence-backed research levers for HyperMem-class ≥93% (harness-first) |## Decisions already taken during researchdiff --git a/docs/research/01-ingestion-extraction.md b/docs/research/01-ingestion-extraction.mdindex 1cbc7e1..6f1e854 100644--- a/docs/research/01-ingestion-extraction.md+++ b/docs/research/01-ingestion-extraction.md@@ -619,6 +619,8 @@ Lowest priority, but both are latent hazards.**Checkpoint 6.** Full Phase 0 suite plus the retrieval-side metrics from `02-retrieval-multihop.md`, as input to the cross-workstream roadmap.+**Cost / latency companions.** Shipped token accounting, Observations/consolidator defaults, Architect-per-Scout-unit bounding, batched Janitor, and incremental hub indexing are owned by [`10-ingest-cost-latency.md`](10-ingest-cost-latency.md). The next bounded batch-extraction and Janitor-exception plan is [`11-architect-loop-efficiency-plan.md`](11-architect-loop-efficiency-plan.md). Do not use `PIPELINE_MODE=lightweight` as a cost lever for accurate-mode quality goals.+---## Risksdiff --git a/docs/research/06-roadmap.md b/docs/research/06-roadmap.mdindex f3ddec0..4d23c2e 100644--- a/docs/research/06-roadmap.md+++ b/docs/research/06-roadmap.md@@ -292,3 +292,7 @@ The remaining case for Phase 4 work is not "the ablation proved the graph wins o| Latency attribution is wrong because it was inferred, not measured | Medium | **Mitigated:** 3.1 measured; dossier deletion confirmed live (~4.8 s → ~1.4 s) || Cardinality still missing after the hotfix | Medium | One-to-many non-event predicates still wrongly deprecated; Task 4.1 || Unrepaired brains silently confound every comparison | High | Record `ingest.n` per run; require a fresh ingest for any graph-side claim |++## Related: ingest cost workstream++Accurate-mode ingest token and latency reductions (instrumentation, Observations/consolidator defaults, Architect unit bounding, batched Janitor, hub refresh) live in [`10-ingest-cost-latency.md`](10-ingest-cost-latency.md). That work must not regress event-hub shape or LoCoMo/LongMemEval quality; `lightweight` remains out of scope as a cost lever. Architect-loop redesign toward peer-competitive token multipliers is in [`11-architect-loop-efficiency-plan.md`](11-architect-loop-efficiency-plan.md).diff --git a/docs/research/08-sota-locomo-protocol.md b/docs/research/08-sota-locomo-protocol.mdindex f406e54..6ba59c4 100644--- a/docs/research/08-sota-locomo-protocol.md+++ b/docs/research/08-sota-locomo-protocol.md@@ -49,7 +49,7 @@ Manifest fields: `bench_profile`, `sc_samples`, `gap_fill`, answer/judge models,| Date | Arm | Scope | Profile | Acc | Notes || --- | --- | --- | --- | --- | --- || 2026-07-29 | `phase-sota-d1-conv26` | conv-26 / `locomoconv26clean` | sota: deepseek-v4-flash, SC=3, gap-fill, hardened prompt + topics | **84.9%** | Open-domain **69.2%**; answerer gap **9.2%**. McNemar vs D.1 ns. Topic session-id leak in text_context fixed after run. |-| (queued) | `sota-locomo10-*` | full LoCoMo10 | sota + deepseek-v4-flash | | Ingest `sota-locomo10-ingest`. Eval: `benchmarks/scripts/run_sota_locomo10.sh` |+| 2026-07-31 | `sota-locomo10-batch-compose` | full LoCoMo10 / `locomof10c*` | sota: deepseek-v4-flash, SC=5, gap-fill, compose harness | **81.0%** | n=1540; mean/med ingest mult 90.6×/45.6×; **no ≥93% claim** |## Failure taxonomy (Task 0.2)diff --git a/docs/research/09-longmemeval-protocol.md b/docs/research/09-longmemeval-protocol.mdnew file mode 100644index 0000000..01bb0cc--- /dev/null+++ b/docs/research/09-longmemeval-protocol.md@@ -0,0 +1,49 @@+# LongMemEval Protocol Sheet++Goal: run BrainAPI against [LongMemEval](https://github.com/xiaowu0162/LongMemEval) (ICLR 2025) with a documented, reproducible harness under `benchmarks/longmemeval/`.++## Corpus++| Variant | File | Role |+| --- | --- | --- |+| **S** (default) | `longmemeval_s_cleaned.json` | Paper LongMemEval_S (~40 sessions / ~115k tokens). Primary claim corpus. |+| **oracle** | `longmemeval_oracle.json` | Evidence sessions only — retrieval ablation upper bound. |+| **M** | `longmemeval_m_cleaned.json` | ~500 sessions; downloadable, not first-run scope. |++Source: Hugging Face [`xiaowu0162/longmemeval-cleaned`](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned).++500 questions across: `single-session-user`, `single-session-assistant`, `single-session-preference`, `temporal-reasoning`, `knowledge-update`, `multi-session`, plus abstention (`question_id` ends with `_abs`).++## Harness contract++- HTTP-only; one **brain per `question_id`** (`lme{alnum}`).+- Ingest haystack sessions only (timestamps + session ids). Never ingest answers or `has_answer` labels.+- Headline metric: official LongMemEval LLM yes/no judge accuracy over **all** questions (including abstention), with per-type breakdown.+- Secondary: session recall vs `answer_session_ids` (skip abstention).+- Answerer/judge models follow `BENCH_*` / `BENCH_PROFILE` (same as LoCoMo). Official judge **prompts** are ported from upstream `evaluate_qa.py`.++## Competitor bar (not apples-to-apples)++Published numbers use different answer/judge models. Record BrainAPI models in the run manifest / `REPORTS.json`.++| System | Reported overall | Notes |+| --- | ---: | --- |+| Mastra Observational Memory | ~94.9% | gpt-5-mini reader |+| Mem0 (2026 blog) | ~94.4% | proprietary stack |+| Mastra OM / oracle (gpt-4o) | ~84% / ~82% | closer to gpt-4o-class readers |+| Full-context gpt-4o | ~60% | long-context baseline |++BrainAPI claims should always list answer model, judge model, variant (`s` / `oracle`), and `BENCH_PROFILE`.++## Quickstart++```bash+cd benchmarks+./longmemeval.sh download --variant s+./longmemeval.sh smoke --limit 1+./longmemeval.sh ingest --run lme-s-smoke --limit 3+./longmemeval.sh evaluate --run lme-s-smoke --limit 3+./longmemeval.sh report --run lme-s-smoke+```++Ledger: successful runs with `n_questions >= 50` upsert `benchmarks.longmemeval` in `REPORTS.json`.diff --git a/docs/research/10-beam-protocol.md b/docs/research/10-beam-protocol.mdnew file mode 100644index 0000000..06121e3--- /dev/null+++ b/docs/research/10-beam-protocol.md@@ -0,0 +1,86 @@+# BEAM protocol (BrainAPI harness)++Goal: evaluate BrainAPI as a long-term memory system on [BEAM](https://github.com/mohammadtavakoli78/BEAM) (ICLR 2026 / [arXiv:2510.27246](https://arxiv.org/abs/2510.27246)).++This is **not** a port of LIGHT. BrainAPI replaces RAG/LIGHT via `POST /ingest/` + `POST /retrieve/context`.++## Dataset++| Item | Value |+| --- | --- |+| Source | HuggingFace [`Mohammadta/BEAM`](https://huggingface.co/datasets/Mohammadta/BEAM) |+| License | CC BY-SA 4.0 |+| Splits (v1) | `100K`, `500K`, `1M` (paper “128K” ≡ HF `100K`) |+| Deferred | [`Mohammadta/BEAM-10M`](https://huggingface.co/datasets/Mohammadta/BEAM-10M) |+| Local path | `benchmarks/data/beam/{size}/{conversation_id}/` |++Do **not** ingest probing questions, rubrics, or ideal answers.++## Pipeline++```text+./beam.sh download --size 100K+./beam.sh smoke --size 100K --sample 1+./beam.sh ingest --size 100K --sample 1 --run beam-100k-1+./beam.sh evaluate --size 100K --sample 1 --run beam-100k-1+./beam.sh report --run beam-100k-1+```++Brain IDs: `beam{size}{convid}` → `beam100k1`.++Ingest unit = one batch turn (user + assistant messages), with `time_anchor` as `source_timestamp` when present.++## Parallel ingest (ops)++| Knob | Recommendation |+| --- | --- |+| Harness `--concurrency` | **2** default safe; **3** if API stays healthy; **≤4** soft max |+| `CELERY_WORKER_CONCURRENCY` | ≥ harness concurrency (product `.env` default **4**; thread pool) |+| Resume | On by default; skips `completed` / `partial_failed` / `permanent_failed` and legacy embed-8192 `failed` rows |+| Stall | Ops scripts stop after 3 no-progress tries; harness does not requeue permanent embed-8192 fails |++**ETA (1M/1, ~625 turns):** prior run ≈ **40 h** at concurrency **1**. Expect roughly **20–25 h** at **2**, **14–18 h** at **3** (contention / uvicorn restarts can erase ideal linear speedup).++**Caveats:** concurrent units on one brain contend on Neo4j/Postgres and can knock over uvicorn under load. Keepalive must only restart when port **8000** is not LISTENing (never `pkill` a healthy API while evaluate-only is running). Prefer a **new brain id** for any full re-ingest; do not wipe brains mid-eval.++Example next 1M re-ingest (new brain; do not start casually):++```bash+BEAM_INGEST_CONCURRENCY=2 BEAM_INGEST_TIMEOUT=2400 \+ ./scripts/run_beam_1m_1_ingest_eval.sh+# or:+./beam.sh ingest --size 1M --sample 1 --run beam-1m-1-clean2 \+ --brain beam1m1clean2 --concurrency 2 --timeout 2400+```++## Scoring++- Per-question: rubric LLM-judge with `unified_llm_judge_base_prompt`.+- Variant: `beam-rubric-v1-question-aware` (always substitutes `<question>`; scores as `float` so `0.5` is kept).+- Ability score for reporting:+ - `event_ordering` → `tau_norm` (Kendall τ after LLM alignment)+ - all others → mean `llm_judge_score` over rubric items+- Headline: mean of the 10 ability means (paper-style).+- Ledger field: `headline_score` (continuous `[0,1]`), not LoCoMo’s binary accuracy.++## Abilities++abstention, contradiction_resolution, event_ordering, information_extraction, instruction_following, knowledge_update, multi_session_reasoning, preference_following, summarization, temporal_reasoning.++## Cost notes++100K chats average ~100+ turns; accurate ingest is slow/expensive. Start with `--limit-turns` / `--limit` and a single sample. Rubric judging can issue several LLM calls per question.++## Citation++```+@misc{tavakoli2025milliontokensbenchmarkingenhancing,+ title={Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs},+ author={Mohammad Tavakoli and Alireza Salemi and Carrie Ye and Mohamed Abdalla and Hamed Zamani and J Ross Mitchell},+ year={2025},+ eprint={2510.27246},+ archivePrefix={arXiv},+ primaryClass={cs.CL},+ url={https://arxiv.org/abs/2510.27246},+}+```diff --git a/docs/research/10-ingest-cost-latency.md b/docs/research/10-ingest-cost-latency.mdnew file mode 100644index 0000000..00ec0ee--- /dev/null+++ b/docs/research/10-ingest-cost-latency.md@@ -0,0 +1,96 @@+# 10 — Ingest cost and latency (without quality trade-offs)++Workstream: make `PIPELINE_MODE=accurate` ingestion cheaper and faster while preserving event-hub graph shape and LoCoMo/LongMemEval quality. Constraints in [`00-scope-and-constraints.md`](00-scope-and-constraints.md) are binding. Extraction quality work remains owned by [`01-ingestion-extraction.md`](01-ingestion-extraction.md); this document owns **token and wall-clock** levers that must not regress quality.++**Quality bar:** accurate-mode event hubs; no McNemar-significant judge drop; Phase-0 extraction metrics non-regressing. `lightweight` is **out of scope**.++---++## Cost map (accurate mode)++| Rank | Stage | Lever shipped |+| --- | --- | --- |+| 1 | Architect full-text tool loop | Architect now runs **per Scout unit** (+ optional prior-unit window ≤4), not unbounded session text |+| 2 | Atomic Janitor per create | **Batched** post-Architect Janitor + cheap grounding precheck skips |+| 3 | Scout per 6k chunk | Unchanged (instrumented) |+| 4 | Observations | Gated by `RUN_OBSERVATIONS` (default **false**) — never wrote the graph |+| 5 | Consolidation | `RUN_GRAPH_CONSOLIDATOR` default **false** until audited (`01` Phase 6) |+| 6 | Hub/topic rebuild | Entity-scoped hub refresh; topic rebuild deferred to finalize |++---++## Phases++### C0 — Measure (shipped)++- `src/core/saving/ingest_cost.py` — per-stage ledger; contextvar accumulation from custom agent invoke loop+- Task status `cost` field via `set_ingestion_task_status(..., cost=...)`+- LoCoMo / LongMemEval ingest records + report mean tokens/unit and stage breakdown++### C1 — Waste removal (shipped)++| Knob | Default | Effect |+| --- | --- | --- |+| `RUN_OBSERVATIONS` | `false` | Skip ObservationsAgent LLM |+| `RUN_GRAPH_CONSOLIDATOR` | `false` | Skip consolidator LLM mutations |+| `JANITOR_BATCH_SIZE` | `20` | Relationships per Janitor call |++### C2 — Bound and batch (shipped)++- [`auto_kg.py`](../../src/core/saving/auto_kg.py): Scout chunks → Architect per unit with prior context; `reset=False` across chunks+- [`architect_agent.py`](../../src/core/agents/architect_agent.py): `defer_janitor=True`, `run_batched_janitor`+- Hub: `refresh_hub_bridges_for_entities`; topic rebuild on finalize only++### C3 — Gate remaining LLM (shipped)++- [`grounding.py`](../../src/core/saving/grounding.py): span align precheck before Janitor+- Ambiguous entity resolution routes to `KGAgent.verify_entity_existence` instead of silent duplicate++### Distillation (deferred)++Distill-SynthKG ([2410.16597](https://arxiv.org/abs/2410.16597)) only after `01` Phases 0–3 stabilize the teacher. Premature distillation freezes defects.++---++## Literature (access 2026-07-29)++| Paper | Role |+| --- | --- |+| ATOM [2510.22590](https://arxiv.org/abs/2510.22590) | Atomic units + parallel merge → latency cut |+| Distill-SynthKG [2410.16597](https://arxiv.org/abs/2410.16597) | Deferred student extractor |+| Zep/Graphiti [2501.13956](https://arxiv.org/abs/2501.13956) | Prior-context window; uncertainty-gated ER |+| UCCI [2605.18796](https://arxiv.org/abs/2605.18796) | Cascade routing at fixed quality |+| SafePassage [2510.00276](https://arxiv.org/abs/2510.00276) | Span grounding without LLM scorer |++---++## Verification++```bash+# Unit+pytest tests/test_ingest_cost.py tests/test_grounding.py -v++# After a live ingest, task status should include cost.stages+# Benchmark report.md Ingest section should show mean tokens when cost is present+```++Measured A/B (2026-07-30): `benchmarks/runs/ingest-cost-baseline.json` — conv-26 sessions 1–5.++| Arm | mean wait | mean LLM tokens | LLM÷source | type-named | events |+| --- | ---: | ---: | ---: | ---: | ---: |+| Legacy hotpath (`ingest-cost-legacy`) | 630 s | 1.19M | — | 0 | 107 |+| Cheap defaults (`ingest-cost-cheap2`) | 223 s | 0.73M | **~886×** | 0 | 100 |+| CP2 batch (`architect-cp2-batch`) | 41 s | 15.4k | **18.8×** | 0 | 60 |+| CP3 batch+scratchpad (`architect-cp3-scratchpad`) | 34 s | 12.6k | **15.3×** | 0 | 65 |++C0–C3 shipping: ≈**65%** lower mean wait, ≈**39%** fewer mean tokens vs legacy. Architect redesign: old cheap **~886× → batch+scratchpad ~15.3×** (same five sessions; source denom 4,114 `cl100k_base`). Details: [`architect-cp2-NOTES.md`](../../benchmarks/runs/architect-cp2-NOTES.md), [`architect-cp3-NOTES.md`](../../benchmarks/runs/architect-cp3-NOTES.md). No McNemar in these cost measurements.++---++## What not to do++- Do not use `PIPELINE_MODE=lightweight` as the cost lever+- Do not add reflection passes for cost goals+- Do not claim accuracy wins without paired A/B (noise floor in `00`)++**Follow-on:** Architect loop efficiency workstream and ship defaults (`batch` + `auto`/scratchpad) in [`11-architect-loop-efficiency-plan.md`](11-architect-loop-efficiency-plan.md).diff --git a/docs/research/11-architect-loop-efficiency-plan.md b/docs/research/11-architect-loop-efficiency-plan.mdnew file mode 100644index 0000000..ea99843--- /dev/null+++ b/docs/research/11-architect-loop-efficiency-plan.md@@ -0,0 +1,673 @@+# 11 — Architect ingest loop efficiency (peer-competitive tokens)++Workstream: cut accurate-mode **ingest** LLM token multipliers by roughly an order of magnitude while preserving event-hub graph shape and LoCoMo / LongMemEval quality bars in [`00-scope-and-constraints.md`](00-scope-and-constraints.md) and [`10-ingest-cost-latency.md`](10-ingest-cost-latency.md). Architecture changes to the Architect loop **are allowed**. `PIPELINE_MODE=lightweight` remains **out of scope** as a cost lever. Retrieval `/retrieve/context` must stay cheap (no query-time LLM loops).++Extraction quality ownership stays with [`01-ingestion-extraction.md`](01-ingestion-extraction.md); this document owns the **Architect / Janitor control-flow redesign** that dominates measured spend.++---++## Problem statement (measured)++### Baseline A/B (2026-07-30)++Source: [`benchmarks/runs/ingest-cost-baseline.json`](../../benchmarks/runs/ingest-cost-baseline.json) — LoCoMo `conv-26`, sessions 1–5, `PIPELINE_MODE=accurate`.++| Arm | mean wait | mean LLM tokens / session | Architect mean tokens | Janitor mean tokens | Architect+Janitor share |+| --- | ---: | ---: | ---: | ---: | ---: |+| Legacy hotpath (`ingest-cost-legacy`) | 630 s | **1.19M** | 1.19M (Janitor folded into Architect) | (folded) | ~99%+ |+| Cheap defaults (`ingest-cost-cheap2`) | 223 s | **0.73M** | 0.38M | 0.34M | **99.5%** |++C0–C3 shipping (per-unit Architect, deferred batched Janitor, Observations/consolidator off, grounding precheck) already cut mean wait ≈65% and mean tokens ≈39%. **Scout remains ~0.5% of spend** (~3.8k tokens / session). **Post-CP3 ship path:** batch+scratchpad is **~15.3×** source tokens vs cheap **~886×** — see multiplier section and `architect-cp2-NOTES.md` / `architect-cp3-NOTES.md`.++### Token multiplier vs source text++Pinned denominator: **tiktoken `cl100k_base`**, 4,114 source tokens on conv-26 sessions 1–5 (`ingest-cost-baseline.json`).++| Era | Arm | LLM÷source |+| --- | --- | ---: |+| Pre-redesign | Cheap defaults (`ingest-cost-cheap2`) | **~886×** |+| CP2 | Batch + raw prior (`architect-cp2-NOTES.md`) | **18.8×** |+| CP3 (ship) | Batch + scratchpad (`architect-cp3-NOTES.md`) | **15.3×** |++Old cheap ~886× → batch+scratchpad ~15.3× on the same five sessions. The **≤100×** ship gate is met; stretch 20–50× is also met.++The requested **20–100×** band remains a peer-shaped engineering target, not a published apples-to-apples LightRAG/Graphiti comparison.++### Why spend concentrates in Architect+Janitor++Traced current workspace path:++1. Scout emits structured entities per ≤6k-character unit (`scout_agent.py:385-410`, `:503-533`)—already close to the peer control-flow shape.+2. `auto_kg.py:156-194` calls Architect once per Scout unit by default, passing the current unit plus up to four prior raw units (`auto_kg.py:33`, `:70-77`).+3. The call is `ArchitectAgent.run(...)`, not `run_tooler` (`auto_kg.py:171-183`). `ArchitectAgent` defaults to `agent_mode="single"` (`architect_agent.py:257-268`), so `run()` dispatches through `_run_single()` to `run_structured()` (`architect_agent.py:593-633`, `:1161-1193`).+4. “Structured” is not one-shot here. `run_structured()` binds five state/relationship tools **and** `ArchitectAgentSingleResponse` (`architect_agent.py:1203-1243`), then runs an outer coverage loop up to four times (`:1245-1328`). Each outer iteration can enter the custom `AgentBase` tool loop.+5. The custom backend is the default (`config.py:584-586`; `runtime_agent_factory.py:25-37`). It sends the accumulated `agent.messages` on every next tool turn (`invoke_loop.py:372-428`, `:432-502`) and has tracing thresholds but no explicit `tool_loop_count` termination guard (`invoke_loop.py:110-147`, `:432-445`). Native tool schemas, the output schema, the long system prompt, source text, state, and prior tool results therefore recur in billed context.+6. The outer Architect loop also re-adds the same source-bearing user prompt, full entity state, and prior generated relationship summaries (`architect_agent.py:1230-1240`, `:1262-1318`). The state tools themselves return complete remaining/used entity lists (`ArchitectAgentGetRemainingEntitiesToProcessTool.py:50-64`; `ArchitectAgentCheckUsedEntitiesTool.py:53-62`), while `CreateRelationship` accepts one edge per call (`ArchitectAgentCreateRelationshipTool.py:177-204`).+7. The cheap arm measures **33.4 Architect calls** and **19.8 Janitor calls** per session (`ingest-cost-baseline.json:115-127`), with 17–56 Architect calls across the five sessions (`:137-176`). Architect averages about **11.4k tokens per model call** (379,505 / 33.4), consistent with context re-send rather than output volume alone.+8. Batched Janitor still consumes ≈344.8k tokens/session after the cheap grounding precheck (`architect_agent.py:328-440`; `grounding.py:90-116`)—about 47% of total cheap-arm spend.++There is therefore a structured **schema**, but no pure, side-effect-free, provider-constrained extraction path with `tools=[]`, one response, and a hard repair budget. That missing path—not merely “switching to `run_structured`”—is the first architectural lever.++One attribution caveat remains: `ingest-cost-baseline.json` records ingest flags but not the running server checkout or Architect mode, and the workspace/live runtime may differ. The cost numbers are observed; assigning every measured call to the current source path remains an inference until Task 0.2 records runtime code identity.++### Token-amplification decomposition++- **Call count:** 33.4 Architect and 19.8 Janitor model calls/session in the cheap arm; the current source permits multiple inner tool turns inside each of up to four outer attempts per Scout unit.+- **Context re-send:** each next inner turn invokes the model with accumulated messages, while each outer attempt reconstructs the source-bearing user prompt and entity state; successful prior relationships are summarized back into later attempts.+- **Prompt and schema overhead:** the single-mode system prompt is substantial (`src/constants/prompts/architect_agent.py:205-269`); the custom runtime adds tool descriptions and a flattened output schema (`src/core/agents/core/prompts.py:27-63`), and native binding attaches all tool schemas (`agent_base.py:58-66`). Those fixed instructions recur on every model request before source/history tokens.+- **Tool design:** complete entity lists are read back through bookkeeping tools, and relationship creation is edge-granular even though the desired output is a batch. These tools mostly expose in-process state, not new external information that would justify model observation.+- **Missing direct path:** `ArchitectAgentSingleResponse` is a useful contract, but the function named `run_structured()` still binds tools and loops. BrainAPI lacks a `tools=[]`, provider-constrained, validate-then-persist path with a one-repair ceiling.++### Quality reality (binding)++From `00` / Phase 1: graph hygiene improved; graph channel still does **not** reliably beat passages on LoCoMo judge accuracy (noise floor ~5–7 pts). Do **not** claim quality wins from single-run judge deltas. Prefer graph hygiene + retrieval-side metrics + paired McNemar gates.++---++## What this workstream does (as-is)++```mermaid+flowchart TD+ text[Input text] --> chunk[chunk_text max 6k]+ chunk --> scout[Scout structured entities per chunk]+ scout --> merge[_merge_entities]+ merge --> archRun[Architect run per unit + raw prior window 4]+ archRun --> outer[Single-mode coverage loop max 4]+ outer --> inner[Custom AgentBase schema + five tools]+ inner --> tools[CreateRelationship + bookkeeping turns]+ tools --> defer{defer_janitor?}+ defer -->|yes| queue[Queue pending relationships]+ defer -->|no| janPerCreate[Atomic Janitor per create]+ queue --> batchJan[Batched Janitor + grounding skip]+ batchJan --> persist[Caller persists enrichment_relationships]+ janPerCreate --> persist+```++Anchors:++| Step | Location |+| --- | --- |+| Orchestration | `auto_kg.py:120-267` |+| Per-unit Architect text | `auto_kg.py:70-77`, `_PRIOR_UNIT_WINDOW=4` |+| Default single dispatch | `architect_agent.py:257-268`, `:593-633`, `:1161-1193` |+| Outer structured coverage loop | `architect_agent.py:1203-1328` |+| Inner custom tool loop | `invoke_loop.py:118-147`, `:372-502` |+| Create + optional inline Janitor | `ArchitectAgentCreateRelationshipTool.py:383-411` |+| Batched Janitor | `architect_agent.py:328-440` |+| Grounding skip | `grounding.py:90-116` |+| Cost ledger | `ingest_cost.py` via `track_stage` |+| Knobs | `config.py` (`JANITOR_BATCH_SIZE`, `INGEST_ARCHITECT_PER_UNIT`, `INGEST_DEFER_JANITOR`, `INGEST_ARCHITECT_MODE`, `INGEST_ARCHITECT_PRIOR_CONTEXT`, `INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP`, `PIPELINE_MODE`) |++**Guarantee this path tries to provide:** every Scout entity is considered for event-centric relationships, and submitted relationships are structurally checked or explicitly deferred to batched Janitor before provenance-stamped persistence.++**Where it breaks:** reaching the four-iteration outer cap only logs remaining entities and continues (`architect_agent.py:1320-1328`), so “every entity considered” is not a hard outcome; the inner custom tool loop is not explicitly bounded; repeated state and source context dominate input; Janitor remains a second full LLM agent for most edges.++---++## Guarantees and where they break++Ranked by impact on **token multiplier** (then quality risk):++1. **Nested invocation is the cost model.** The current “single” path still combines an outer four-pass coverage loop with a generic stateful tool loop. At ~33 Architect calls/session, this is an architectural gap, not a missing temperature or prompt knob.+2. **The structured path is prompt-enforced, tool-coupled, and retry-prone.** The default custom backend injects/recovers JSON rather than guaranteeing one provider-constrained response (`runtime_agent_factory.py:25-43`; `prompts.py:27-63`). A schema parse can coexist with prior tool turns and side effects.+3. **Tools are too stateful and too granular.** Relationship creation is one edge per tool invocation; remaining/used queries resend full entity lists; model turns are being spent on bookkeeping that deterministic Python already owns.+4. **Janitor is still ~47% of cheap-arm tokens.** Grounding skip helps, but LLM Janitor remains the normal path rather than the exception path.+5. **Raw prior context is a quality/cost trade-off.** Up to four prior units are repeated for cross-unit coreference (`auto_kg.py:70-77`). Compaction could save input tokens but has weaker evidence than bounded extraction and can erase antecedents.+6. **Event hubs and temporal attribution are constraints, not expendable overhead.** Replacing Architect with naive binary NER/RE or `lightweight` would abandon the product differentiator.+7. **The benchmark cannot certify small judge changes.** The ~5–7pp judge noise floor means cost and deterministic graph/retrieval metrics are primary early gates; paired McNemar is appropriate only for end-to-end binary answer outcomes, and a non-significant result is not proof of equivalence.++---++## Frontier techniques++### Retrieval provenance++- **Query:** cheap high-quality LLM KG construction; structured extraction and constrained decoding; Architect-loop alternatives (ATOM, Distill-SynthKG, Zep/Graphiti, LightRAG, MS GraphRAG, ReWOO, LLMCompiler, JSONSchemaBench, EDC, SafePassage, UCCI, Engram).+- **Databases:** arXiv (primary, Atom via `arxiv_atom.py`), OpenAlex (LazyGraphRAG resolution), Microsoft Research blog (LazyGraphRAG — not on arXiv).+- **Access date:** 2026-07-30.++### Sources and applicability++| Paper / source | Claim (as stated) | Evidence quality | Applicability to BrainAPI |+| --- | --- | --- | --- |+| **ATOM** [2510.22590](https://arxiv.org/abs/2510.22590) | Atomic self-contained facts → parallel TKG merge; ~18% exhaustivity, ~33% stability, **>90% latency cut** vs baselines | Peer-reviewed track (EACL 2026 Findings); empirical TKG metrics; **not** LoCoMo/LongMemEval QA | **High** for unitization + dual-time; latency claim is construction latency, not BrainAPI’s ReAct tax. Adapt atomic units; keep event hubs. |+| **Distill-SynthKG** [2410.16597](https://arxiv.org/abs/2410.16597) | Teacher multi-step SynthKG → **single-step** student extractor; better KG quality than larger models; helps RAG | Strong methodology for KG+RAG; gains are on their synth/eval sets | **Adapt later:** single-pass extract now; distill only after teacher (event-hub schema) stabilizes — matches `10` deferral. |+| **Zep / Graphiti** [2501.13956](https://arxiv.org/abs/2501.13956) | Temporal KG memory; LongMemEval up to **+18.5%** acc, **−90%** response latency vs baselines; DMR 94.8% vs MemGPT 93.4% | Vendor paper; LongMemEval gains are **retrieval/memory**, not an ingest token multiplier study. DMR delta is small. | **Adapt** prior-context window + uncertainty-gated ER; do **not** treat +18.5% as evidence that BrainAPI’s ReAct loop is required. |+| **LightRAG** [2410.05779](https://arxiv.org/abs/2410.05779) | Dual-level graph+vector index; incremental updates; “simple and fast” vs GraphRAG-class systems | Open-source; retrieval-quality claims; **ingest cost not rigorously peer-benchmarked against BrainAPI** | **Adopt pattern:** schema-guided entity+relation extract in **few LLM calls per chunk**, not tool loops. Keep BrainAPI event-hub schema. |+| **MS GraphRAG** [2404.16130](https://arxiv.org/abs/2404.16130) | LLM extract entity graph + community summaries; strong on **global** sensemaking | Canonical GraphRAG; indexing is **expensive by design** (summaries) | **Adopt extract stage only** (structured triples per chunk). **Reject** community-summary indexing as BrainAPI’s cost lever — wrong product surface and costly. |+| **LazyGraphRAG** (MSR blog, 2024; not arXiv) | Index cost ≈ vector RAG (**0.1%** of full GraphRAG); defers LLM to query; relevance-test budget | Blog + internal eval (AP news, LLM pairwise prefs); **not** a peer-reviewed preprint with stable arXiv ID | **Reject as product architecture:** moves LLM to query time — violates `00` context-path budget. **Adapt only the idea** “defer expensive LLM until necessary” **at write time** (cascade / skip Janitor), not at retrieve. |+| **SafePassage** [2510.00276](https://arxiv.org/abs/2510.00276) | Span align + scorer cuts IE hallucinations up to **85%**; small encoder can beat LLM scorer | Controlled IE tasks; hallucination flags ≠ LoCoMo accuracy | **Adopt/extend:** BrainAPI already has `grounding.py`; raise skip rate so Janitor is exception path. |+| **UCCI** [2605.18796](https://arxiv.org/abs/2605.18796) | Calibrated cascade: **−31%** cost at fixed micro-F1 on NER | Strong production NER cascade study; assumptions stated | **Adapt** for ingest: cheap model / structured pass first; escalate hard units to tooler or larger model. Not a KG-construction paper. |+| **HippoRAG** [2405.14831](https://arxiv.org/abs/2405.14831) | Write-time KG + PPR; multi-hop QA; **10–30× cheaper** than iterative retrieve | Strong retrieval paper; cost win is **query**, not Architect | **Reject for this workstream’s primary lever** (already aligned with BrainAPI write-heavy / cheap-read intent). Useful as existence proof that **write-time structure pays for cheap read**. |+| **iText2KG** [2409.03284](https://arxiv.org/abs/2409.03284) | Incremental zero-shot KG without heavy post-processing | Scenario evals (papers/sites/CVs), not conversation memory | **Adapt** incremental entity/relation extract; insufficient alone for event hubs. |+| **Engram** [2606.09900](https://arxiv.org/abs/2606.09900) | Lean retrieved context beats full history on LongMemEval_S (**83.6% vs 73.2%**, ~8× fewer answer tokens); async fact extract | Strong LongMemEval methodology; write path uses async SPO extract **without** ReAct | **Adapt** “fast write + async structured facts”; supports replacing a stateful nested loop with bounded extraction. |+| **Dense X / propositions** [2312.06648](https://arxiv.org/abs/2312.06648) | Atomic propositions beat passage units for retrieval | Retrieval granularity study | **Adapt** as optional decontextualization before extract (pairs with ATOM). |+| **ETLCH** [2509.08381](https://arxiv.org/abs/2509.08381) | 1B LoRA model for JSON/KG/NER under low resource | Small-model IE; domain-specific | **Defer** with Distill-SynthKG; useful only after schema freeze. |+| **ReWOO** [2305.18323](https://arxiv.org/abs/2305.18323) | Separates a tool plan from observations; reports **5× token efficiency and +4% accuracy on HotpotQA** | Direct agent-loop ablation, but on QA rather than KG writes | **Adapt** its no-interleaving principle for the fallback path: emit a batch plan once, execute deterministically, repair only failures. |+| **LLMCompiler** [2312.04511](https://arxiv.org/abs/2312.04511) | Parallel function planning reports up to **3.7× latency speedup and 6.7× cost savings** vs ReAct | Multiple function-calling tasks; headline maxima are not KG results | **Reject blind parallel mutation.** Architect tools share entity/relationship state. Reuse only batch planning for independent validation calls. |+| **JSONSchemaBench** [2501.10868](https://arxiv.org/abs/2501.10868) | Evaluates six constrained decoders on 10k real schemas across compliance, coverage, and generation quality | Strong systems benchmark; schema compliance is not factual correctness | **Adopt provider-tested constrained output** to remove parse/retry turns; keep semantic validators because valid JSON can still hallucinate. |+| **EDC** [2404.03868](https://arxiv.org/abs/2404.03868) | Open extract → define → post-hoc canonicalize avoids placing a large ontology in every extraction prompt | Three KGC benchmarks; no memory-QA or ingest-token head-to-head | **Adapt:** constrain response shape and event roles, not a closed predicate vocabulary; canonicalize predicates after extraction. |+| **Practical GraphRAG** [2507.03226](https://arxiv.org/abs/2507.03226) | Dependency parsing reaches 61.87% vs 65.83% for its LLM extractor (reported as 94% of performance) on two enterprise datasets | Narrow domains and LLM-as-judge downstream evaluation limit transfer | **Adapt only as deterministic candidate/grounding evidence.** It does not establish that rules alone preserve conversational event hubs. |++### Critical assessment — what the literature does **not** support++1. **“More agent reflection / more tools ⇒ better memory QA.”** HippoRAG and Engram improve QA with **structured write + cheap read**, not with unbounded write-time ReAct. Folklore that BrainAPI must keep open tool loops for quality is **unsupported**.+2. **LazyGraphRAG as an ingest cost solution.** Its headline is **cheap index + query-time LLM**. That inverts BrainAPI’s binding constraint.+3. **Zep’s LongMemEval +18.5% as Architect-loop justification.** That number is end-to-end memory service performance, not an ablation of nested agent loops vs pure batch extraction.+4. **Distillation as the first lever.** Distill-SynthKG itself needs a high-quality teacher; `10` correctly defers it. Distilling today’s nested-loop teacher freezes cost *and* defects.+5. **Community GraphRAG summaries for BrainAPI ingest.** They optimize global theme QA; BrainAPI’s bar is event hubs, temporal truth, multi-hop — and summaries burn tokens BrainAPI cannot afford.+6. **Judge accuracy single-run “wins” after a cost refactor.** Noise floor in `00` makes that folklore dangerous.++### Ranked levers: expected savings vs quality risk++The grades are GRADE-like judgments for this decision, not formal clinical GRADE scores. Published gains are downgraded for indirectness whenever the task is not event-centric conversational KG construction.++#### 1. Pure batched structured extraction — savings: very high; quality risk: medium; confidence: moderate (B)++**Mechanism.** One Architect call receives the same current unit, Scout entities, and prior context as today, but `tools=[]` and a provider-supported JSON schema force one batch of event hubs and relationships. Permit at most one stateless syntax repair; deterministic code owns entity bookkeeping and persistence. Constrain structure and event roles, not a closed predicate vocabulary, following EDC (2404.03868).++**Evidence and cost.** LightRAG (2410.05779) and MS GraphRAG (2404.16130) demonstrate bounded structured extraction as a viable index shape; JSONSchemaBench (2501.10868) shows constrained decoders can guarantee schema compliance but not semantic truth. No paper establishes BrainAPI quality parity, so local A/B is mandatory. Implementation costs one new extraction contract and path; the reward is removing most of 33.4 Architect turns/session.++**Verdict: adopt first.** This is the only lever large enough to approach 20–100× without removing event hubs.++#### 2. Decouple planning from tools and hard-bound repair — savings: high; quality risk: low–medium; confidence: moderate-low (B−)++**Mechanism.** If a fallback still needs tools, ask once for a complete relationship plan, execute it deterministically, then return only failed items for one repair. Never feed successful tool observations and full state back through another reasoning turn.++**Evidence and cost.** ReWOO (2305.18323) reports 5× token efficiency and +4% HotpotQA accuracy by separating reasoning from observations. LLMCompiler (2312.04511) reports maxima of 3.7× latency and 6.7× cost savings through planned parallel calls. Both are indirect QA/tool evidence; BrainAPI's mutation tools are not freely parallelizable.++**Verdict: adapt for fallback.** Use batch planning and explicit call caps; reject concurrent graph mutations.++#### 3. Make LLM Janitor exception-only — savings: very high; quality risk: medium; confidence: moderate (B)++**Mechanism.** Require source spans and run deterministic endpoint, schema, temporal, duplicate, and span-alignment checks. Auto-accept high-confidence grounded edges, reject impossible shapes, and send only ambiguous residuals to a structured Janitor with an explicit veto outcome.++**Evidence and cost.** SafePassage (2510.00276) reports up to 85% fewer IE hallucinations and shows a small scorer can outperform an LLM scorer on its tasks. BrainAPI already has `cheap_janitor_precheck`; Janitor's measured 344.8k tokens/session makes this locally high leverage. The external evidence does not cover inferred multi-sentence event facts, so false rejection must be sampled.++**Verdict: adopt after the pure extractor.** Target a measured skip rate, not a presumed one.++#### 4. Replace raw prior replay with bounded relevant state — savings: medium; quality risk: medium–high; confidence: low (C)++**Mechanism.** Keep the current unit verbatim but replace four raw prior units with a token-capped scratchpad of active entities, event hubs, dates, and unresolved references; retrieve a raw prior span only when a coreference check fails.++**Evidence and cost.** Graphiti/Zep (2501.13956) supports a bounded prior-message window, not this particular compression. ATOM (2510.22590) and Dense X (2312.06648) support self-contained atomic units, but a rewrite can invent antecedents. The change should be isolated from the loop A/B so its effect is identifiable.++**Verdict: adapt only after Checkpoint 1.** Keep raw context in the first extractor comparison.++#### 5. Calibrated cheap-model cascade — savings: medium on residual; quality risk: medium; confidence: low–moderate (C+)++**Mechanism.** Run the batch schema on a cheaper model; escalate only schema failures, low grounding coverage, disconnected event hubs, or calibrated uncertainty. Select thresholds on held-out sessions rather than the five-session cost set.++**Evidence and cost.** UCCI (2605.18796) reports 31% lower cost at fixed micro-F1 for production NER. That is strong cascade evidence but indirect for event relationships. A second model, calibration set, and drift monitoring add operational cost.++**Verdict: defer until the single-model path passes quality gates.**++#### 6. Distill the stable extractor — savings: potentially very high; quality risk: high if premature; confidence: low for BrainAPI (C)++**Mechanism.** Use accepted event-hub outputs from a stable teacher to train a small one-step extractor, as Distill-SynthKG (2410.16597) compresses a multi-stage teacher into one generation.++**Evidence and cost.** Distill-SynthKG reports better KG quality than baselines up to eight times larger and downstream retrieval/QA gains, but on its own datasets. Dataset generation, training, evaluation, and drift maintenance are substantial, and the current teacher still has known defects.++**Verdict: defer.** Revisit only after two stable checkpoints and maintainer approval.++---++## Recommended architecture++**Ship defaults (accurate):** `INGEST_ARCHITECT_MODE=batch`, `INGEST_ARCHITECT_PRIOR_CONTEXT=auto` (→ scratchpad); rollback with `tooler` / `raw`.++### Design decisions++1. **Add a pure schema-guided batch extractor; do not reuse `run_structured()` unchanged.** The new happy path has no tools, no entity-state loop, one response, and at most one stateless repair. It emits event hubs, typed endpoints, open-vocabulary predicates, `source_span`, `happened_at`, and validity fields.+2. **Keep the current context unchanged for the first A/B.** Isolate the control-flow effect before testing prior-context compression.+3. **Janitor becomes exception-path:** expand cheap grounding + schema validators; LLM Janitor only on `need` set; target skip ≥70% of relationships on conv-26 A/B without hygiene regression.+4. **Batch writes stay;** prefer emitting **many relationships per structured call** (LightRAG/MS extract pattern) over one tool call per edge.+5. **Current single/tooler paths become hard-budget escalation options** for schema, grounding, or event-coverage failures; every escalation records a reason and cannot recurse without a cap.+6. **Only after the loop A/B, test prior context separately:** replace raw prior-unit replay with a compact entity/event scratchpad plus raw-span fallback.+7. **No query-time LLM** for efficiency wins. No `lightweight` mode. No community-summary GraphRAG index as a cost strategy.+8. **Cascade models are optional:** use a cheaper model only after single-model extraction passes and escalation thresholds are calibrated on held-out units.++### As-is vs to-be++```mermaid+flowchart LR+ subgraph asis [As-is accurate]+ S1[Scout structured] --> A1[Architect outer coverage + inner tool loop]+ A1 --> J1[Janitor LLM frequent]+ end+ subgraph tobe [To-be accurate]+ S2[Scout structured] --> E2[Pure batch extract 1 call + optional repair]+ E2 --> G2[Deterministic validate + grounding]+ G2 -->|pass| W2[Batch write]+ G2 -->|fail| X2[Hard-budget Architect or Janitor escalation]+ X2 --> W2+ end+```++### Mapping to existing code++| Piece | Reuse / change |+| --- | --- |+| Scout | Keep (`scout_agent.py`) |+| `run_structured` | Reuse response models/persistence helpers only; do **not** promote its nested tool/coverage loop unchanged |+| New `run_batch_extract` | `tools=[]`, constrained response, one repair maximum, no side effects before validation |+| Existing single/tooler paths | Gate behind `INGEST_ARCHITECT_MODE=tooler\|schema\|batch` (`current` alias → `tooler`) and an explicit escalation budget |+| `grounding.py` | Tighten + require `source_span` in schema |+| `run_batched_janitor` | Keep for escalate / residual |+| `ingest_cost.py` / `invoke_loop.py` | Add input/output/cached tokens, calls per unit, payload size, multiplier, skip/escalation rate |++## What “competitive with LightRAG/Graphiti” would mean++No cited paper exposes a directly comparable ingest-token denominator, model mix, or event-graph output. BrainAPI should not claim cost parity from architecture diagrams. A defensible claim requires:++1. **Peer-shaped control flow:** ≤2 Architect calls on a happy-path Scout unit, no per-edge LLM bookkeeping, bounded prior context, and Janitor as an exception.+2. **Measured budget:** accurate-mode total ingest ≤100× source tokens on the frozen five-session set, with 20–50× as the stretch band. The denominator must name its tokenizer.+3. **Controlled comparison:** if “cheaper than LightRAG/Graphiti” is claimed publicly, run the same corpus and model/tokenizer through pinned peer versions and report tokens, wall time, graph size, and extraction coverage. Until then, say “within the target band,” not “beats peers.”+4. **BrainAPI differentiation retained:** zero type-named placeholders; no deprecated event legs; grounded event hubs with actor/object leg completeness; temporal fields and provenance preserved; stable graph evidence-session recall and no detected paired LoCoMo/LongMemEval regression.+5. **Cheap retrieval preserved:** all extra intelligence remains at write time; `/retrieve/context` gets no new LLM loop.++---++## Implementation plan++### Phase 0 — Make amplification attributable++#### Task 0.1: Architect payload and token telemetry++**Description:** Extend the ledger at the model-call boundary to record input, output, cached-input (when exposed), total tokens, call ordinal, unit id, stage, and loop origin (`architect_outer`, `agent_tool`, `schema_repair`, `janitor`). Record source characters plus a source-token denominator with an explicit tokenizer id; if only an estimate is available, label it as estimated.++**Acceptance criteria:**+- [ ] A two-call fixture reconciles per-call and stage totals and distinguishes input from output.+- [ ] Reports expose calls/unit, tokens/call, source tokenizer, multiplier, Janitor skip rate, and escalation rate without breaking providers that omit cached-token fields.+- [ ] Architect, Janitor, and Scout totals reconcile to total ingest LLM usage within provider rounding.++**Verification:**+- [ ] `pytest tests/test_ingest_cost.py -v`+- [ ] `cd benchmarks && ./locomo.sh ingest --sample conv-26 --limit-sessions 1 --brain architectmeasurecp0 --run architect-measure-cp0 --concurrency 1 --no-resume && ./locomo.sh report --run architect-measure-cp0 --json`++**Dependencies:** None+**Files likely touched:** `src/core/saving/ingest_cost.py`, `src/core/agents/core/invoke_loop.py`, `benchmarks/locomo/ingest.py`, `benchmarks/locomo/report.py`, `tests/test_ingest_cost.py`+**Size:** M++#### Task 0.2: Reproducible cost and graph-hygiene A/B command++**Description:** Add one benchmark command that creates isolated brains, ingests fixed `conv-26` sessions 1–5 for two Architect modes, and writes paired per-session cost/latency plus graph hygiene. Record workspace SHA, dirty state, running server code identity, models, tokenizer, prompts, environment flags, and brain ids.++**Acceptance criteria:**+- [ ] The artifact includes source multiplier, input/output tokens, calls/unit, tokens/call, wait time, type-named nodes, event/node/relationship counts, EVENT-touching deprecated edges, event degree/leg completeness, grounded-span rate, hub bridges, errors, and escalation/skip reasons.+- [ ] Fresh brain ids and a one-owner rule prevent residue or concurrent ingestion; failed/partial units remain visible.+- [ ] Cost and hygiene use paired per-session deltas. No McNemar test is emitted because these are not paired binary answer outcomes.++**Verification:**+- [ ] `cd benchmarks && ./locomo.sh ingest-cost-ab --sample conv-26 --limit-sessions 5 --baseline-mode current --candidate-mode current --tag architect-cp0`++**Dependencies:** 0.1+**Files likely touched:** `benchmarks/locomo/cli.py`, one benchmark helper, one test+**Size:** M++### Checkpoint 0+- [ ] Same-mode A/A completes and records the actual running code.+- [ ] The current arm confirms or revises O(10³)× and the >95% Architect+Janitor spend share.+- [ ] Per-call telemetry can distinguish context re-send from generated output.+- [ ] If these observations do not reproduce, revisit the diagnosis before changing architecture.++---++### Phase 1 — Remove the nested-loop tax++#### Task 1.1: Pure event-hub extraction contract++**Description:** Define a batch response that reuses Scout UUIDs, may declare missing event entities, and emits relationships with open-vocabulary predicates, exact source-span offsets/text, amount/polarity, `happened_at`, and validity metadata. Add a batch validator for type-named placeholders, endpoint existence, event actor/content leg completeness, and span bounds. Do not encode the entire predicate ontology in the prompt.++**Acceptance criteria:**+- [ ] Open predicates are accepted while malformed endpoints, out-of-range spans, and type==name placeholders are rejected with itemized reasons.+- [ ] Fixtures cover a dated event hub, amount on a relationship, cross-sentence evidence, and reuse of existing Scout UUIDs.+- [ ] Validation is side-effect-free and runs before any relationship is queued.++**Verification:**+- [ ] `pytest tests/test_architect_batch_schema.py -v`++**Dependencies:** None+**Files likely touched:** one schema/validator module and `tests/test_architect_batch_schema.py`+**Size:** S++#### Task 1.2: Tool-free `run_batch_extract`++**Description:** Add a feature-flagged Architect path with `tools=[]`, provider-supported constrained output when available, one primary call, and at most one stateless repair of invalid items. Feed it the **unchanged** current `_architect_unit_text` and Scout entities so this phase changes control flow only. Reuse persistence helpers after validation.++**Acceptance criteria:**+- [ ] Valid output uses exactly one Architect model call/unit; malformed output uses exactly two; no third call or tool invocation is possible.+- [ ] No graph/pending-relationship side effect occurs before the whole batch is validated.+- [ ] `INGEST_ARCHITECT_MODE=current|batch` selects the arm; `current` remains the default until Checkpoint 2.++**Verification:**+- [ ] `pytest tests/test_architect_batch_extract.py -v`+- [ ] `cd benchmarks && INGEST_ARCHITECT_MODE=batch ./locomo.sh ingest --sample conv-26 --limit-sessions 1 --brain architectbatchsmoke --run architect-batch-smoke --concurrency 1 --no-resume`++**Dependencies:** 1.1+**Files likely touched:** `src/core/agents/architect_agent.py`, `src/core/saving/auto_kg.py`, `src/config.py`, Architect prompts, one test+**Size:** M++#### Task 1.3: Controlled five-session loop A/B++**Description:** Use Task 0.2 to compare `current` vs `batch` with identical model, raw prior context, Janitor behavior, session order, and retrieval-independent graph audit. Perform item-level error analysis on dropped/disconnected entities rather than using graph size alone as quality.++**Acceptance criteria:**+- [ ] Happy-path Architect calls/unit are ≤2 and Architect input tokens fall ≥80% relative to the current arm, or the failed target has a payload-level explanation.+- [ ] Type-named nodes and deprecated EVENT-touching edges remain zero; grounded-span rate and event actor/content leg completeness do not fall by more than 5pp.+- [ ] Event/node/relationship counts are reported as diagnostics, not treated as correctness by themselves.++**Verification:**+- [ ] `cd benchmarks && ./locomo.sh ingest-cost-ab --sample conv-26 --limit-sessions 5 --baseline-mode current --candidate-mode batch --tag architect-cp1`++**Dependencies:** 0.2, 1.2+**Files likely touched:** benchmark artifact only, plus a benchmark fix if the command cannot explain failures+**Size:** S++### Checkpoint 1+- [x] Nested-loop removal **pass** on re-run A/B (`architect-cp1b-*`, sessions 1–2, fresh `*12b` brains): Architect input **−92.4%** / total **−86.9%** vs tooler; escalate **0%** (was 100% on `schema_partial_reject`); happy-path **2.0 calls/unit** both batch sessions (schema+repair, no tooler escalate). Prior partial result + fix notes: `architect-cp1-NOTES.md`.+- [x] Graph hygiene: type-named **0** / deprecated **0** both arms; batch denser on nodes/events (29/24 vs 22/18) with **source_span** on 27/27 rels. Soft debt: batch EVENT descriptions **0/24** (worker logged `event_leg_incomplete` + many `span_offsets_mismatch_text` drops without escalate) — **addressed in Phase 2**; post-measure (`architect-cp2-batch`, sessions 1–5): EVENT desc **60/60**, source_span **112/112**.+- [x] Total multiplier: tooler **299×** vs batch **37×** (≤100× gate **pass**; Janitor skipped 27 batch units — **CP1b 100% skip under-validated**; Phase 2 requires explicit evidence). Details: `benchmarks/runs/architect-cp1b-NOTES.md`.+- [x] Maintainer reviews extraction diffs before Janitor behavior changes (esp. missing EVENT descriptions / span rejects) — Phase 2 ships deterministic fill/realign + triage; live re-measure in `architect-cp2-NOTES.md`.+- [x] Re-run Checkpoint 1 A/B after `schema_partial_reject` fix (sessions 1–2) — **gates pass**. Optional: confirm on sessions 1–5 before Phase 2 claims.+++### Phase 2 — Make Janitor and escalation exceptional++#### Task 2.1: Deterministic validation and grounding decisions++**Description:** Extend `cheap_janitor_precheck` to classify each batch relationship as accept, reject, or ambiguous using endpoint shape, source-span alignment, event-leg completeness, temporal format, duplicate key, and placeholder checks. Persist the decision, score, and reason. Hand-label a fixed 50-edge sample including multi-sentence inferences before choosing thresholds.++**Acceptance criteria:**+- [x] Every edge has exactly one decision and machine-readable reason; accepted edges retain source span and grounding score. (`GroundingDecision` + `triage_relationships_for_janitor`; ledger `janitor_drop_reasons`.)+- [ ] Auto-accept precision is ≥95% on the fixed reviewed sample; ambiguous inferred facts are not silently rejected. *(needs 50-edge hand-label — not run this turn)*+- [x] Janitor skip rate reaches ≥70% on the five-session arm or the artifact lists the dominant blockers and reviewed examples. *(CP2 batch: **114/114 skip** with **100%** real `source_span` + EVENT desc 60/60 — not CP1b synthetic spans; tooler 1–2 shows triage live: rej/amb 5/44. Details: `architect-cp2-NOTES.md`)*++**Verification:**+- [x] `pytest tests/test_grounding.py tests/test_architect_grounding_decisions.py -v`+- [x] Manual CP2 re-measure (harness `ingest-cost-ab` still absent): `architect-cp2-batch` sessions 1–5 + optional `architect-cp2-tooler` 1–2 — see `benchmarks/runs/architect-cp2-NOTES.md`.++**Dependencies:** Checkpoint 1+**Files likely touched:** `src/core/saving/grounding.py`, Architect orchestration, ledger/report, one test+**Size:** M++**Shipped (2026-07-30):**+- Span realign via `find_span_offsets` / `realign_span_offsets`; batch validator repairs or clears bad offsets instead of hard `span_offsets_mismatch_text` rejects when the quote is grounded.+- EVENT descriptions required in batch prompts; validator fills from Scout → rel description → source_span; rejects empty new EVENT nodes.+- Explicit-evidence-only auto-accept (no synthetic `tail+predicate+tip` grounding) so Janitor skip≠100% from under-validation.+- Ledger: `janitor_rejected`, `janitor_ambiguous`, `janitor_drop_reasons`.+- **Post-measure (2026-07-30):** `architect-cp2-batch` 1–5 → mult **18.8×**, EVENT desc **60/60**, source_span 100%, escalate 0%; optional tooler 1–2 → **337×** / arch input **−96.7%**. Notes: `benchmarks/runs/architect-cp2-NOTES.md`.++#### Task 2.2: Structured residual Janitor with explicit veto++**Description:** Send only ambiguous edges to a bounded structured Janitor batch. Replace `"OK"`/`None` ambiguity with explicit accept/fix/reject/error outcomes; a parse or provider failure marks the ingest partial/degraded rather than approving edges. Ensure a rejected original is not queued alongside its replacement.++**Acceptance criteria:**+- [x] Parse failure, approval, correction, and veto are distinct outcomes covered by tests. (`status` OK/ERROR/REJECT; parse `None` drops; wrong+fixed handled)+- [x] A vetoed edge is absent and only its accepted replacement may persist.+- [x] Janitor uses one call/batch plus at most one repair and falls below 30% of total ingest LLM tokens on the five-session A/B. *(CP2 batch: Janitor **0%** of LLM tokens — all units explicit-evidence skip)*++**Verification:**+- [x] `pytest tests/test_janitor_outcomes.py -v` *(veto covered; `test_janitor_veto.py` not separate)*+- [x] Re-measure: `architect-cp2-NOTES.md` (batch Janitor token share 0%; tooler arm exercises LLM triage).++**Dependencies:** 2.1+**Files likely touched:** `src/core/agents/janitor_agent.py`, `src/core/agents/architect_agent.py`, `ArchitectAgentCreateRelationshipTool.py`, tests+**Size:** M++#### Task 2.3: Hard-budget escalation++**Description:** For invalid or under-covered batch outputs, attempt one item-level batch repair, then optionally invoke the existing Architect path within a total per-unit model-call budget chosen by the maintainer. Record trigger, attempted path, accepted result, and budget exhaustion; never continue silently with unprocessed entities.++**Acceptance criteria:**+- [ ] Happy-path units never escalate; each escalation has one enumerated trigger.+- [ ] No unit exceeds the configured call budget; exhaustion yields an explicit partial/degraded status.+- [ ] Forced fixtures cover invalid schema, missing event leg, ungrounded relationship, and provider failure.++**Verification:**+- [ ] `pytest tests/test_architect_escalation.py -v`+- [ ] Task 0.2 artifact reports escalation rate, reasons, and max calls/unit.++**Dependencies:** 1.2, 2.1+**Files likely touched:** `src/core/saving/auto_kg.py`, `src/core/agents/architect_agent.py`, `src/config.py`, one test+**Size:** M++### Checkpoint 2+- [x] Accurate-mode total ingest is **≤100×** source tokens on the fixed five sessions; **≤50×** is stretch. If it remains >150×, stop and attribute the residual before adding another model. **Pass — 18.8×** overall on `architect-cp2-batch` (sessions 1–5; max session 30.9×).+- [x] Architect happy path is ≤2 calls/unit; Janitor is <30% of ingest LLM tokens; escalation rate and failure reasons are understood. **Pass** — calls/unit 1.0–2.0; Janitor **0%** tokens; escalate **0%**.+- [x] Zero type-named nodes and deprecated EVENT legs; grounded-span and event-leg completeness gates pass. **Pass** type-named 0; EVENT desc **60/60**; source_span **112/112**; soft: deprecated rels=3 (inspect later).+- [x] Only now may `batch` become the accurate-mode default, subject to the maintainer's answer. **Shipped:** product defaults `INGEST_ARCHITECT_MODE=batch` + `INGEST_ARCHITECT_PRIOR_CONTEXT=auto` (code, `.env.example`, live `.env`); rollback via env.++---++### Phase 3 — Optional residual levers, each isolated++#### Task 3.1: Bounded prior-state A/B++**Description:** Replace four raw prior units with a ≤500-reference-token entity/event/date scratchpad and retrieve one raw supporting span only when a coreference validator fails. Compare against batch mode with the original raw window; do not combine this test with a model change.++**Acceptance criteria:**+- [x] Prior context obeys its token cap and reports raw-span fallback frequency. *(token cap enforced in `serialize_scratchpad`; `fetch_prior_span` helper shipped for on-demand use; live fallback-frequency A/B still pending)*+- [x] Cross-unit pronoun, alias, and delayed-date fixtures preserve node/event identity. *(unit fixtures cover entity/hub/predicate scratchpad + no full prior-body replay; live graph identity A/B still pending)*+- [x] Architect input tokens fall while grounded-span and event-leg completeness stay within the Checkpoint-2 tolerance. *(2026-07-30 measure: scratchpad vs raw on conv-26 s1–5 — 15.3× vs 23.6×; arch in 28.9k vs 44.5k (−35%); type-named 0; EVENT desc 65/65; graph 75/85/65 vs CP2 81/112/60; see `benchmarks/runs/architect-cp3-NOTES.md`)*++**Verification:**+- [x] `pytest tests/test_architect_prior_context.py -v`+- [x] Manual A/B: `architect-cp3-raw` vs `architect-cp3-scratchpad` (batch, sessions 1–5); harness `ingest-cost-ab` tag still optional++**Dependencies:** Checkpoint 2+**Files likely touched:** `src/core/saving/auto_kg.py`, one context helper, `src/config.py`, one test+**Size:** M++**Shipped (2026-07-30):**+- `architect_scratchpad.py`: build/serialize ≤500-token prior scratchpad (entities, open event hubs, dates, recent predicate + span pointers); `fetch_prior_span` for optional on-demand raw window.+- `auto_kg.py`: batch/schema default (`INGEST_ARCHITECT_PRIOR_CONTEXT=auto`) uses scratchpad instead of raw prior-unit replay; `raw` keeps CP2 A/B behavior.+- Env: `INGEST_ARCHITECT_PRIOR_CONTEXT=auto|scratchpad|raw`, `INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP=500`.+- **Measured 2026-07-30:** scratchpad wins — 15.3× vs raw 23.6× (−35% tokens), type-named 0, EVENT desc 100%, graph near CP2; keep scratchpad/`auto` default. Notes: `benchmarks/runs/architect-cp3-NOTES.md`.++#### Task 3.2: Calibrated model cascade++**Description:** Sweep a cheaper extraction model on held-out `conv-26` sessions 6–8 and escalate on pre-declared validation/grounding features. Do not tune thresholds on sessions 1–5 or end-to-end judge labels.++**Acceptance criteria:**+- [ ] The sweep artifact reports cost, auto-accept precision, event-leg completeness, and escalation rate for every threshold.+- [ ] A selected threshold is frozen before the five-session confirmation.+- [ ] If no threshold lowers total cost at fixed gates, the cascade is rejected.++**Verification:**+- [ ] `pytest tests/test_architect_model_routing.py -v`+- [ ] `cd benchmarks && ./locomo.sh architect-cascade-sweep --sample conv-26 --sessions 6,7,8 --out runs/architect-cascade-sweep.json`++**Dependencies:** Checkpoint 2 and maintainer approval for a second model+**Files likely touched:** model routing/config, Architect orchestration, benchmark sweep, one test+**Size:** M++#### Task 3.3: Distillation go/no-go++**Description:** After at least two weeks of stable accepted batch outputs, estimate dataset size, label noise, training/serving cost, and expected break-even for a Distill-SynthKG-style student. This task writes a decision note only; any training implementation must be decomposed separately.++**Acceptance criteria:**+- [ ] The note quantifies teacher stability, train/held-out split, event-hub metrics, serving break-even, and rollback.+- [ ] No training begins without a maintainer go decision.++**Verification:**+- [ ] Review the decision note against Checkpoints 1–2 artifacts.++**Dependencies:** Checkpoint 2, maintainer approval+**Files likely touched:** one research decision document+**Size:** S++### Checkpoint 3+- [x] Every optional lever has its own A/B; no bundled attribution. *(3.1 scratchpad vs raw measured isolated 2026-07-30)*+- [x] Total remains in the 20–100× target band with graph gates held. *(scratchpad 15.3×; raw 23.6×; both ≤100× with type-named 0)*++---++### Phase 4 — Quality gates (before claiming “no trade-off”)++#### Task 4.1: Full-conversation graph audit++**Description:** Ingest all 19 `conv-26` sessions into fresh current and batch brains, then compare type-named nodes, EVENT-touching deprecations, grounded-span rate, event actor/content leg completeness, degree distribution, event/node/relationship counts, hub bridges, temporal-field coverage, and item errors.++**Acceptance criteria:**+- [ ] Type-named nodes and deprecated EVENT-touching edges are zero in the candidate.+- [ ] Grounded-span rate and event-leg completeness stay within the predeclared 5pp tolerance; all count/density changes are reported with sampled explanations.+- [ ] Runtime code identity, models, flags, source denominator, and fresh brain ids are captured.++**Verification:**+- [ ] `cd benchmarks && INGEST_ARCHITECT_MODE=current ./locomo.sh ingest --sample conv-26 --limit-sessions 19 --brain architectcurrentfull --run architect-current-full --concurrency 2 --no-resume`+- [ ] `cd benchmarks && INGEST_ARCHITECT_MODE=batch ./locomo.sh ingest --sample conv-26 --limit-sessions 19 --brain architectbatchfull --run architect-batch-full --concurrency 2 --no-resume`+- [ ] `cd benchmarks && ./locomo.sh graph-audit --baseline-brain architectcurrentfull --candidate-brain architectbatchfull --json`++**Dependencies:** Checkpoint 2+**Files likely touched:** benchmark command/report only if Task 0.2 did not already cover full brains+**Size:** S++#### Task 4.2: Paired LoCoMo evaluation on clean brains++**Description:** Evaluate identical retrieval/answer settings on the two full brains, preferably twice per arm to expose judge variability. Report paired exact McNemar and flip counts for binary correctness, a paired accuracy-difference interval, per-category point estimates, passage evidence-session recall, and graph evidence-session recall only if identical-config graph session-set agreement remains ≥95%.++**Acceptance criteria:**+- [ ] Candidate judge accuracy is not more than the predeclared 5pp guard below current, and exact McNemar does not detect a regression; a non-significant result is described as “no detected regression,” not equivalence.+- [ ] Passage EvR does not regress beyond tolerance; graph EvR is interpreted only after its stability gate; multi-hop and temporal flips are listed.+- [ ] Both replicate manifests record server identity, prompts, models, ingest flags, and brain ids; `prompt-audit` passes.++**Verification:**+- [ ] `cd benchmarks && ./locomo.sh evaluate --sample conv-26 --brain architectcurrentfull --run architect-current-eval-a --concurrency 2 --use-ppr`+- [ ] Repeat as `architect-current-eval-b`, `architect-batch-eval-a`, and `architect-batch-eval-b`, then run `./locomo.sh compare --baseline architect-current-eval-a --baseline architect-current-eval-b --candidate architect-batch-eval-a --candidate architect-batch-eval-b --json`+- [ ] `cd benchmarks && ./locomo.sh prompt-audit`++**Dependencies:** 4.1+**Files likely touched:** benchmark artifacts; comparison code only if the paired-difference interval is absent+**Size:** M++#### Task 4.3: LongMemEval smoke (product profile)++**Description:** Add an explicit brain-prefix/namespace option so the two modes cannot share LongMemEval question brains, then ingest and evaluate the same fixed 50-question LongMemEval_S slice under the product profile as a catastrophic-regression smoke. It is not powered for a leaderboard or a quality-win claim.++**Acceptance criteria:**+- [ ] Current and batch runs use the same selected question ids, models, retrieval profile, and fresh brains.+- [ ] Cost, ingestion errors, overall/type-level accuracy, and retrieval metrics are reported; any >5pp point drop is investigated.+- [ ] No SOTA or equivalence claim is made from the smoke; full claims follow `09`.++**Verification:**+- [ ] `cd benchmarks && INGEST_ARCHITECT_MODE=current ./longmemeval.sh ingest --variant s --limit 50 --brain-prefix architectcurrent --run architect-lme-current --concurrency 2 --no-resume`+- [ ] `cd benchmarks && INGEST_ARCHITECT_MODE=batch ./longmemeval.sh ingest --variant s --limit 50 --brain-prefix architectbatch --run architect-lme-batch --concurrency 2 --no-resume`+- [ ] Evaluate each namespace with the same `--brain-prefix`, then run `./longmemeval.sh report --run <run-id> --json`.++**Dependencies:** 4.2 preferred+**Files likely touched:** LongMemEval CLI, dataset brain-id helper, ingest/evaluate wiring, one test, benchmark artifacts+**Size:** M++### Checkpoint 4 (ship gate)+- [ ] Multiplier ≤100× on conv-26 5-session cost A/B+- [ ] Full graph audit green, including grounded event-leg coverage+- [ ] LoCoMo paired guard passes with uncertainty and flip counts reported+- [ ] Retrieval metrics gate recorded (passage EvR; graph EvR only if stable)+- [ ] LongMemEval smoke has no unexplained catastrophic regression++---++## Success metrics++| Metric | Baseline (cheap2) | Target | Gate |+| --- | --- | --- | --- |+| Mean LLM tokens / session (5-session set) | 728k | Report absolute; gate on tokenizer-named multiplier | Checkpoint 2/3 |+| Token multiplier | ~886× cheap → **15.3×** batch+scratchpad | **≤100×** (stretch 20–50×) — **met** | Cost A/B JSON + CP2/CP3 NOTES |+| Architect calls | 33.4/session | ≤1 happy-path call/unit; ≤2 with repair | Checkpoint 1 |+| Architect total tokens | 379.5k/session (input/output split unavailable) | ≥80% reduction at fixed context | Checkpoint 1 |+| Janitor share of LLM tokens | ~47% | <30% (stretch <15%) | Checkpoint 2 |+| type-named placeholders | 0 | 0 | Hygiene |+| Deprecated EVENT-touching edges | Not in baseline JSON | 0 | Hygiene |+| Grounded spans / event-leg completeness | Not yet recorded | Within 5pp of current and manually audited | Checkpoints 1/4 |+| LoCoMo judge | No paired ingest-path baseline | >−5pp guard and no McNemar-detected regression; not an equivalence claim | Checkpoint 4 |+| Passage EvR | No paired ingest-path baseline | Within predeclared tolerance | Checkpoint 4 |+| Graph EvR | — | Only if agreement ≥95% | `00` |++---++## Open questions for the maintainer++1. Which pinned tokenizer should define the source-token multiplier denominator across providers?+ - **RESOLVED:** Pin **`tiktoken` `cl100k_base`** as the cross-provider source-token denominator (`SOURCE_TOKENIZER_ID=tiktoken/cl100k_base`). Recorded on the ingest cost ledger; if tiktoken is unavailable at runtime, fall back to `len(text)//4` and mark `source_tokens_estimated=true`. Do not switch the pin per LLM provider.+2. Is **≤100×** the hard accurate-mode ship threshold, with 20–50× as stretch, or should a different budget be used?+ - **RESOLVED:** Ship gate is **≤100×** source tokens first; then attempt **≤20–50×**. Always keep the better spend/quality version — do not ship a cheaper arm that fails hygiene/quality gates.+3. May `INGEST_ARCHITECT_MODE=batch` become the accurate default after Checkpoint 2, or only after the full Checkpoint-4 evaluation?+ - **RESOLVED:** After CP2/CP3 cost + hygiene pass, product defaults are **`INGEST_ARCHITECT_MODE=batch`** and **`INGEST_ARCHITECT_PRIOR_CONTEXT=auto`** (batch/schema → scratchpad). Rollback: set `INGEST_ARCHITECT_MODE=tooler` (and optionally `PRIOR_CONTEXT=raw`). Full Checkpoint-4 quality gate can wait.+4. What is the maximum total Architect model-call budget per failed unit, including repair and escalation?+ - **RESOLVED:** Schema/batch happy path: **≤3 LLM calls/unit** (1 primary + ≤1 repair; third reserved for accounting/edge). Escalation via tooler: hard cap **10** tool turns / failed unit.+5. Should escalation try the current single path, `run_tooler`, or a larger-model batch extraction first?+ - **RESOLVED:** Escalate order: **schema extract fail → single repair extract → tooler ReAct last**. No larger-model cascade in the first vertical slice.+6. May prior raw text be replaced entirely by a scratchpad plus on-demand span, or must one previous raw unit always remain?+ - **RESOLVED:** **Yes** — replace prior raw text replay with compact scratchpad + on-demand span (Phase 3.1, after Checkpoint 1 isolates the loop A/B with raw context unchanged).+7. Is a second cheaper extraction model in scope for Phase 3, or must the work remain single-model until distillation?+ - **RESOLVED:** Cascade / cheaper second model is **OUT of scope** for the first vertical slice. Same model, fewer calls. Revisit only after Checkpoint 2.+8. May structured Janitor veto an edge outright, with an audit record, rather than only fix or approve it?+ - **RESOLVED:** **Yes** — structured Janitor may veto edges with an audit record. Prefer deterministic reject first; LLM veto only if needed (Phase 2).+9. Is a full two-replicate `conv-26` paired evaluation required before defaulting batch mode, despite the benchmark's limited power?+ - **RESOLVED:** **No** full two-replicate conv-26. Use small batch while testing (e.g. conv-26 sessions 1–2 or 1–3 max). Do not start long live A/B until code + unit tests are ready.+10. Is a pinned same-corpus LightRAG/Graphiti ingest run required before any public “peer-competitive cost” claim?+ - **RESOLVED:** **Do not** run LightRAG/Graphiti ingest. Only cite online/published claims already in this plan; no new pinned peer runs for this workstream.++---++## Risks++| Risk | Impact | Detection |+| --- | --- | --- |+| Batch extraction under-produces event/multi-hop structure | High — cheap but loses the differentiator | Grounded event-leg completeness, item-level diffs, stable graph EvR, multi-hop/temporal flips |+| Valid JSON is mistaken for a true fact | High | Schema compliance and grounding are separate gates; 50-edge reviewed sample |+| Provider lacks reliable constrained output | Medium — repair turns return | Provider contract tests, repair count telemetry, keep current path behind the flag |+| Escalation approaches 100% | High — no cost win | Escalation rate/reasons; Checkpoint 2 fails |+| Grounding auto-accepts fluent hallucinations or rejects inference | High | Auto-accept precision target plus ambiguous bucket and reviewed multi-sentence examples |+| Context compaction is bundled with loop replacement | High — causal attribution lost | Phase order requires fixed raw context at Checkpoint 1 |+| Scratchpad breaks coreference | Medium | Pronoun/alias/date fixtures and raw-span fallback rate |+| Cost A/B uses dirty or different runtime code | High | Fresh brain ids and running-server identity in every artifact |+| Approximate source tokens become a precise claim | Medium | Tokenizer id/estimate flag required in ledger and reports |+| Non-significant McNemar is called equivalence | High | 5pp guard, paired interval, flip counts, explicit “no detected regression” language |+| Distillation freezes current defects | Medium | Phase 3.3 is decision-only and depends on stable outputs |+| “Peer-competitive” becomes “beats peers” without a head-to-head | Medium | Controlled peer run required for comparative public claims |++---++## What not to do++- Do not use `PIPELINE_MODE=lightweight` to hit the multiplier target.+- Do not add reflection / critique agent passes for cost goals.+- Do not move LLM graph construction to `/retrieve/context` (LazyGraphRAG pattern).+- Do not adopt full MS GraphRAG community summarization as BrainAPI’s index.+- Do not distill the current nested-loop teacher.+- Do not ship on a single-run LoCoMo point estimate.++---++## Related docs++- Constraints: [`00-scope-and-constraints.md`](00-scope-and-constraints.md)+- Prior cost levers (C0–C3 shipped): [`10-ingest-cost-latency.md`](10-ingest-cost-latency.md)+- Extraction quality: [`01-ingestion-extraction.md`](01-ingestion-extraction.md)+- Roadmap pointer: [`06-roadmap.md`](06-roadmap.md)+- Dual goal (≥93% LoCoMo + low multiplier): [`12-locomo-93-at-low-multiplier.md`](12-locomo-93-at-low-multiplier.md) — keep batch+scratchpad; fix escalate storms; accuracy via SOTA harness / answerer / retrieval, not more Architect loops++---++## Literature provenance appendix++```+## Retrieval Summary+- Query: cheap high-quality KG construction for BrainAPI Architect efficiency+- Scope: targeted lookup of named systems + related cascade/grounding papers+- Databases queried: arXiv (`user-arxiv` search + abstract verification and Atom id lookup), OpenAlex (LazyGraphRAG), MSR blog (LazyGraphRAG)+- Access date: 2026-07-30++## Provenance+- arXiv: https://export.arxiv.org/api/query with id_list /+ 2510.22590,2410.16597,2501.13956,2410.05779,2404.16130,2510.00276,+ 2605.18796,2405.14831,2409.03284,2606.09900,2312.06648,2509.08381,+ 2305.18323,2312.04511,2501.10868,2404.03868,2507.03226+- Parsed with paper-lookup scripts/arxiv_atom.py+- LazyGraphRAG: not found on arXiv (totalResults 0); OpenAlex hits are secondary;+ primary description from https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/+- Warnings: the 20–100× peer band is a requested engineering target inferred from+ bounded extractor designs, not a published head-to-head result against BrainAPI+```diff --git a/docs/research/12-locomo-93-at-low-multiplier.md b/docs/research/12-locomo-93-at-low-multiplier.mdnew file mode 100644index 0000000..78dc7c5--- /dev/null+++ b/docs/research/12-locomo-93-at-low-multiplier.md@@ -0,0 +1,212 @@+# 12 — ≥93% LoCoMo at low ingest multiplier++Goal: reach HyperMem-class **≥93%** LLM-as-a-judge on LoCoMo while keeping accurate-mode **ingest** multiplier near the shipped batch+scratchpad band (**~15–20× median**; mean not dragged back to ~100×+ by escalate storms).++Binding constraints: [`00-scope-and-constraints.md`](00-scope-and-constraints.md) (noise floor, product vs deep retrieval), [`08-sota-locomo-protocol.md`](08-sota-locomo-protocol.md) (SOTA track), [`11-architect-loop-efficiency-plan.md`](11-architect-loop-efficiency-plan.md) (batch/scratchpad ship state), [`07-event-graphrag-improvement-plan.md`](07-event-graphrag-improvement-plan.md) (retrieval levers).++**Plan only — no implementation in this document.**++---++## Snapshot (2026-07-30)++| Arm | Track | Scope | Judge | Ingest multiplier | Notes |+| --- | --- | --- | ---: | --- | --- |+| `locomo-batch-score` | **product** | conv-26 | **80.3%** | median **18.9×** / mean **78.2×** | Fresh `locomoconv26batch`; escalate storms on s10/13/14 |+| `locomo-batch-cfgb` | **product** | conv-26 | **84.2%** | same brain (no re-ingest) | Config-B retrieval; +3.9pp vs score; McNemar ns p=0.31 |+| `locomo-batch-nostorm-cfgb` | **product** | conv-26 | **85.5%** | median **87.8×** / mean **124.6×** | Fresh `locomoconv26nostorm`; cfg-B; ties best product point |+| `oldbrain-recheck-a` | product | conv-26 | **85.5%** | (pre-batch path) | Best `REPORTS.json` product (tied) |+| `phase-sota-d1-conv26` | **sota** | conv-26 | **84.9%** | (clean brain, pre-batch) | SC=3 + gap-fill; open-domain 69.2%; answerer gap 9.2% |+| `locomo-compose-sota-conv26-v2` | **sota** | conv-26 | **92.1%** | same as nostorm (no re-ingest) | Composition/traits/affect harness; McNemar vs cat3fix +10/−2 p=**0.039**; answerer gap **2.0%**; **≥93% miss** (~0.9pp) |+| `locomo-cat3fix-sota-conv26` | **sota** | conv-26 | **86.8%** | same as nostorm (no re-ingest) | Open-domain harness fix; cat3 **76.9%** (+30.7pp); gap **7.2%**; McNemar vs nostorm-sota +10/−7 p=0.63 (ns); **≥93% miss** |+| `locomo-nostorm-sota-conv26` | **sota** | conv-26 | **84.9%** | same as nostorm product (no re-ingest) | SC=5 + gap-fill on `locomoconv26nostorm`; cat3 **46.2%**; gap **9.2%**; eval tokens **39.1M**; McNemar vs nostorm-cfgb +7/−8 p=1.0 (ns); **≥93% miss** |+| `sota-locomo10-batch-compose` | **sota** | full LoCoMo10 | **81.0%** | median **45.6×** / mean **90.6×** | Fresh `locomof10c*`; SC=5+gap-fill+compose harness; n=1540; cat1 64.2% cat3 61.5%; answerer gap 12.7%; cost gates FAIL; **≥93% miss (~12 pp)** |+| HyperMem (paper) | — | full LoCoMo | **92.73%** | n/a | Different answerer/judge; not apples-to-apples |++`locomo-batch-score` category gaps: cat1 multi-hop **68.8%**, cat3 open-domain **61.5%**, cat2 **86.5%**, cat4 **85.7%**. EvR full **96.7%**; answerable **94.1%**; **answerer gap 13.8%**. Eval used config-A-ish knobs (`max_passages=8`, `max_facts=40`, `use_ppr=false`, `historical_limit=10`) — weaker than mid-80s product arms.++`locomo-batch-cfgb` (same brain, config-B): cat1 **84.4%**, cat3 **53.8%**, cat2 **83.8%**, cat4 **90.0%**; EvR full **98.7%**; answerer gap **9.9%**.++`locomo-batch-nostorm-cfgb` (storm-fixed brain, config-B): cat1 **84.4%**, cat2 **86.5%**, cat3 **53.8%**, cat4 **91.4%**; EvR full **99.3%**; answerer gap **8.6%**. McNemar vs score +17/−9 p=0.17; vs cfgb +8/−6 p=0.79.++`locomo-nostorm-sota-conv26` (same storm-fixed brain, **SOTA** SC=5 + gap-fill, no re-ingest): headline **84.9%** [78.3%, 89.7%]; cat1 **78.1%**, cat2 **86.5%**, cat3 **46.2%**, cat4 **94.3%**; EvR full **99.3%**; answerer gap **9.2%**; eval LLM tokens **39.1M** (vs product cfgb ~7.8M). McNemar vs `locomo-batch-nostorm-cfgb`: +7/−8 p=**1.0** (ns) — harness did not beat product 85.5%. **≥93% gate: miss** (~8 pp short on conv-26; full-10 still required for protocol claim).++`locomo-compose-sota-conv26-v2` (same brain, **SOTA** SC=5 + gap-fill + composition/traits/affect harness, no re-ingest): headline **92.1%** [86.7%, 95.4%]; cat1 **84.4%**, cat2 **91.9%**, cat3 **84.6%**, cat4 **97.1%**; EvR full **99.3%**; answerer gap **2.0%**; eval tokens **42.6M**. McNemar vs `locomo-cat3fix-sota-conv26`: +10/−2 p=**0.039** (significant). Residual 12 wrongs: education Psychology paraphrase, traits vocab SC, image-only book/symbol titles, charity Sunday≠Saturday dialogue, hike marshmallows, children count, a few temporal/painting items. **≥93% gate: miss** (~0.9 pp / 1–2 QAs). NOTES: `benchmarks/runs/locomo-compose-sota-conv26-v2/NOTES.md`.++`locomo-cat3fix-sota-conv26` (same brain, **SOTA** SC=5 + gap-fill + open-domain harness fix, no re-ingest): headline **86.8%** [80.5%, 91.3%]; cat1 **81.2%**, cat2 **86.5%**, cat3 **76.9%**, cat4 **91.4%**; EvR full **99.3%**; answerer gap **7.2%**; eval tokens **39.0M**. McNemar vs nostorm-sota +10/−7 p=**0.63** (ns); vs nostorm-cfgb +9/−7 p=**0.80** (ns). Cat3 **+30.7 pp** vs prior SOTA (46.2%→76.9%). Cat3-only probe hit **92.3%** once (SC variance; not the published estimate). **≥93% gate: miss** (~6.2 pp short on conv-26). Taxonomy + NOTES: `benchmarks/runs/locomo-nostorm-sota-conv26/cat3-failure-taxonomy.md`, `benchmarks/runs/locomo-cat3fix-sota-conv26/NOTES.md`.++Ship ingest path (doc `11`): `INGEST_ARCHITECT_MODE=batch` + scratchpad prior → **~15.3×** on sessions 1–5. Full-19 mean blew up because three sessions escalated into Architect tooler + heavy Janitor (s13 495k, s14 494k, s10 380k tokens). Excluding those three: mean ≈**18×**.++### Full-10 claim gate (2026-07-31)++`sota-locomo10-batch-compose`: full LoCoMo10 under compose SOTA harness on fresh batch+scratchpad brains → headline **81.0%** [78.9%, 82.9%] (n=1540). Ingest 272/272; mean/median multiplier **90.6× / 45.6×**; escalate **6.8%**; cost gates still **FAIL**. Best per-sample ~90% (conv-30); conv-26 **88.2%** (below prior nostorm compose-v2 **92.1%**). **Honest call: no HyperMem-class claim** (~12 pp below ≥93%). NOTES: `benchmarks/runs/sota-locomo10-batch-compose/NOTES.md`.++### Storm fix status (Task 1 / M0, 2026-07-30)++**Root cause:** s10/13/14 → `schema_empty_or_all_rejected` on entity-dense dialogue (Scout ~20+ EVENT hubs; schema empty). Escalate → tooler (≤10 turns) → Janitor ambiguous flood (14–21 LLM calls). Dialogue `\n` never split under 6000-char chunker.++**Shipped** (workspace + `~/.brainapi/source`): dense rechunk (≥12 entities / 1200 chars), line-aware chunking, soft-admit endpoints (+ UUID mint hotfix), escalate max **3**, Janitor LLM cap **2**, `tests/test_architect_escalation.py`.++**M0 re-ingest done** (`locomo-batch-ingest-nostorm`, brain `locomoconv26nostorm`, 19/19): s10/13/14 capped (**41k / 106k / 97k** vs prior **380k / 495k / 494k**). Escalate **12.9%** (11/85 units). Mean / median multiplier **124.6× / 87.8×** — **gates still FAIL** (target mean ≤30×, median 15–20×). Cost moved to other escalate→janitor sessions (s5/12/15/16/17). See `benchmarks/runs/locomo-batch-ingest-nostorm/NOTES.md`.++---++## 1. Honest gap analysis: 80.3% → 93% (~13 pts)++Do **not** claim ingest cost cuts alone get to 93%. Batch+scratchpad already hit the cost band; the quality gap is elsewhere. Approximate attribution (overlapping; not additive to 13):++| Bucket | Est. recoverable | Evidence | Caveat |+| --- | ---: | --- | --- |+| **(a) Product vs SOTA harness** | **~3–8 pts** on same brain | SOTA profile = SC (`sc_samples`≥3–5) + gap-fill + hardened prompt; protocol win condition is this track. Failure taxonomy (`sota-failure-taxonomy.md`): **100%** wrongs generation-side — present-but-unused ~7pp, multi-hop composition ~6pp, temporal format ~4pp. SC targets instability; gap-fill targets unused evidence. | Best measured SOTA on conv-26 is now **92.1%** (`locomo-compose-sota-conv26-v2`); prior **86.8%** / **84.9%**. McNemar vs cat3fix **significant** (p=0.039). Cat3 open-domain + composition/traits harness closed most of the answerer gap (7.2%→2.0%). Eval tokens ↑ (~43M SC=5 vs ~7.8M product). |+| **(b) Incomplete / stormy ingest** | **~0–3 pts** (recovery toward mid-80s product) | 19/19 completed, but escalate storms may distort graph on heavy sessions; batch brain **80.3%** sits **~5 pts** below best product (`85.5%`) and **~2–4 pts** below clean-brain product (~82–84%). | Not proven causal. Config-A retrieval on this eval likely explains part of the drop vs push75-c/d. Fix storms for **mean multiplier**, not as the main accuracy lever. |+| **(c) Retrieval / multi-hop graph** | **~0–4 pts** judge; larger EvR | Doc `07`: Phase B diversification failed EvR lift; Phase C hub-bridge EvR↑ did **not** convert to judge (McNemar ns). Cat1 graph EvR **81%** vs passages **78%** while judge **68.8%** — composition/answerer, not missing sessions (full EvR **96.9%**). Write-time topic index + coarse-to-fine (doc `08`) already landed. Remaining: novel-session retention, orphan hub bridges, answer-side paths. | Prefer retrieval-side metrics; judge may stay flat (Phase 1 / C3). **No new ingest LLM** for these. |+| **(d) Answerer / protocol** | **~8–14 pts** potential ceiling | Answerer gap **13.8%** this run; taxonomy says evidence often present. Temporal format + open-domain inference are answer-prompt / SC issues. | Shared-family judge → absolute % biased; paired McNemar still valid. Groundedness unmeasured. |+| **(e) Noise floor** | **~5–7 pts** unresolvable at n=152 | Doc `00`: identical-config spread 82.9%↔86.2%; CI ~±7. | Single-run “+2 pts” is unfalsifiable. Full LoCoMo10 helps power; still report CIs + McNemar. |++**Bottom line:** ~13 pts is a **stack** problem. Rough order of real leverage: **(d)+(a) harness/answer** first, **config-B retrieval defaults** (cheap), **storm-free full ingest** (multiplier + hygiene), then **graph channel** only where EvR/composition metrics move. Ingest multiplier work is a **constraint**, not the accuracy engine.++**Scope honesty:** Protocol win is **full LoCoMo10** under `BENCH_PROFILE=sota`, not conv-26 alone. Conv-26 is the fast loop; full-10 is the claim gate.++---++## 2. What to keep for the multiplier++Keep as product accurate defaults (doc `11` CP2/CP3):++- `INGEST_ARCHITECT_MODE=batch` (pure schema extract, ≤2 calls/unit happy path)+- `INGEST_ARCHITECT_PRIOR_CONTEXT=auto` → scratchpad (≤500 tokens; measured **15.3×** vs raw **23.6×**)+- Escalate **only** on empty/invalid coverage (schema fail → one repair → tooler last), not as a soft quality preference+- Janitor exception-only via grounding triage; observations / consolidator off for cost arms++**Fix escalate storms (s10/13/14 pattern):** mean **78×** vs median **19×** is unacceptable for the dual goal.++| Action | Why | Ingest tokens? |+| --- | --- | --- |+| Cap per-unit escalate budget (already resolved ≤10 tool turns) and **fail partial** instead of unbounded Janitor loops | Storms show architect 12 calls + janitor 21 calls / ~420k janitor tokens on outliers | Cuts ingest |+| Raise deterministic accept / realign so heavy sessions stay on batch path | Happy-path sessions already ~10–30k tokens | Cuts ingest |+| Telemetry: escalate rate, reason, max calls/unit on every full-19 ingest | Gate mean ≈ median | Measure only |+| **Do not** “fix quality” by re-enabling tooler-default or raw prior | Returns ~300–800× | Forbidden |++Acceptance for multiplier: full-19 conv-26 ingest **median 15–20×**, **mean ≤30×** (stretch ≤25×), escalate rate **<10%** of units, zero silent unbounded loops.++---++## 3. Path to ≥93% (ordered by expected pts / multiplier risk)++Explicit token class: **I** = ingest LLM, **E** = eval/answer/judge LLM, **R** = retrieval compute only (no LLM).++| # | Lever | Est. pts | Mult. risk | Tokens | Notes |+| ---: | --- | ---: | --- | --- | --- |+| 1 | **Eval on config-B retrieval** (`use_ppr`, `max_passages=16`, `max_facts=50`, `historical_limit=16`) matching mid-80s arms | +1–4 (recover toward 82–85) | None | **R** | Batch-score used config A; trivial. Product path still ADR-006. |+| 2 | **SOTA harness** on a clean full-19 brain: `BENCH_PROFILE=sota`, deepseek-v4-flash, SC=5, gap-fill on, prompt-audit green | +3–8 vs same-brain product | None | **E** | Required for 93 claim (`08`). Does not touch ingest. Always report greedy product alongside. |+| 3 | **Storm-free full-19 (then full-10) ingest** under batch+scratchpad | +0–3 quality; **required for mean ~19×** | Fixes mean | **I↓** | Fresh brains; graph audit gates from `11` Task 4.1. |+| 4 | **Answerer / protocol** (hardened prompt, temporal format rules, SC majority, gap-fill re-retrieve) per `08` + taxonomy | +5–10 of the answerer gap | None | **E** | Primary path for cat3 + present-but-unused. No gold-fitting. |+| 5 | **Retrieval/graph without ingest LLM** from `07`/`02`: topic coarse-to-fine (landed), reserved novel-session bridge slots, orphan hub-bridge spine widen, query–edge embedding rank, session/hub diversify under budget | +0–4 judge; EvR/cat1 first | None on ingest; watch p50 | **R** (+ tiny **I** only if write-time hub-bridge rebuild on ingest — index build, not Architect loop) | Do not expect judge miracles; Phase C3 already showed EvR≠answers. |+| 6 | **Deep/MCP composition** for residual multi-hop (ToG-style) | optional SOTA-only | None on product ingest | **E** at query | Not on `/retrieve/context`. Optional harness agent — separate from product claim. |++**Explicit non-claims**++- Cutting Architect tokens further (cascade, distill) will **not** deliver 93%.+- Hub-bridge EvR lifts alone will **not** deliver 93% (already measured).+- Product greedy `/retrieve/context` is **not** required to hit 93%; the protocol puts SC/gap-fill in the **harness**.++```mermaid+flowchart TD+ keep[Keep batch+scratchpad] --> storm[Fix escalate storms mean~median]+ storm --> brain[Fresh full-19 / full-10 brains]+ brain --> cfgB[Eval config-B retrieval]+ cfgB --> prod[Product greedy baseline ~mid-80s]+ cfgB --> sota[SOTA: SC + gap-fill + answerer]+ sota --> ret[R-only graph/passage tuning]+ ret --> claim{Full LoCoMo10 SOTA >=93%?}+ claim -->|no| tax[Re-tax failures: unused vs multi-hop vs open-domain]+ tax --> sota+```++---++## 4. Milestone table++| Milestone | Target judge | Multiplier | Run | Acceptance |+| --- | ---: | --- | --- | --- |+| **M0** Storm-fixed ingest | n/a | median 15–20×, mean ≤30× on conv-26×19 | `locomo-batch-ingest-nostorm` | **Partial:** s10/13/14 capped; escalate 12.9%; mean **124.6×** / median **87.8×** — multiplier gates still red (janitor after escalate on other sessions) |+| **M1** Product recover | ≥83% point, CI overlaps best product | same as M0 | `locomo-batch-nostorm-cfgb` | **Done:** **85.5%** on M0 brain (cfg-B); ties `oldbrain-recheck-a`; McNemar vs score ns p=0.17; vs stormy cfgb ns p=0.79 |+| **M2** SOTA conv-26 | ≥88% aspirational; **≥86%** gate | unchanged (eval-only) | `locomo-compose-sota-conv26-v2` | **Partial→strong:** **92.1%** (above aspirational 88%); McNemar vs cat3fix +10/−2 p=**0.039**; prompt-audit green; still **~0.9pp short of HyperMem 93%** |+| **M3** Full LoCoMo10 SOTA | **≥93%** headline | median ~15–20× on sampled sessions; mean ≤30× | `sota-locomo10-*` per `08` | Protocol sheet: full-10, non-adversarial, deepseek-v4-flash, SC=5, gap-fill; also publish product greedy; memory-off ablation |+| **M4** Dual gate | M3 holds | mean stays near median after any ingest change | paired re-ingest subset | McNemar ns regression vs M3; multiplier gate still green |++Compare with `./locomo.sh compare` (exact McNemar). Do not treat noise-floor wiggles as wins.++---++## 5. Top 5 tasks (planning style)++### Task 1 — Kill escalate storms on heavy sessions++**Description:** Diagnose s10/13/14 triggers (schema_partial_reject, grounding ambiguous flood, call budget). Enforce hard per-unit caps, partial/degraded status on exhaustion, and deterministic realign/accept so Janitor stays exceptional on long sessions.++**Acceptance:** Full-19 re-ingest mean multiplier ≤30× with median 15–20×; escalate rate <10%; no session >100k LLM tokens without an explicit logged budget-exhaustion status; pytest covers budget exhaustion.++**Status (2026-07-30):** Code + tests + live sync shipped; full-19 re-ingest **done** (`locomo-batch-ingest-nostorm`). Acceptance **not met** on mean/median multiplier (124.6× / 87.8×) or escalate <10% (12.9%). Former s10/13/14 storms capped; residual cost is escalate→tooler→janitor on other dense sessions. Soft-admit UUID mint hotfix required mid-run.++**Tokens:** **I↓**. Size: M.++### Task 2 — Product cfg-B eval on storm-fixed brain++**Description:** Fresh evaluate with `use_ppr=true`, `max_passages=16`, `max_facts=50`, `historical_limit=16`, product profile, same answer/judge models as batch-score.++**Acceptance:** Manifest records flags; McNemar vs `locomo-batch-score`; passage EvR not below prior; headline point estimate ≥83% or documented failure analysis. Upsert `REPORTS.json` only if `status=ok`.++**Status (2026-07-30):** Stormy-brain stand-in `locomo-batch-cfgb` **84.2%**. **Re-run on M0 brain done** as `locomo-batch-nostorm-cfgb`: judge **85.5%** [79.1%, 90.2%]; EvR full 99.3%; answerer gap 8.6%; McNemar vs score +17/−9 p=0.17; vs cfgb +8/−6 p=0.79. NOTES at `benchmarks/runs/locomo-batch-nostorm-cfgb/NOTES.md`. `REPORTS.json` upserted.++**Tokens:** **R** + **E** (eval). Size: S.++### Task 3 — SOTA harness pass on same brain (conv-26)++**Description:** `BENCH_PROFILE=sota`, SC=5, gap-fill on, deepseek-v4-flash, prompt-audit. Optionally harden temporal/open-domain answer instructions without gold-fitting.++**Acceptance:** Prompt-audit green; McNemar vs Task-2 product; answerer gap ↓ vs 13.8% or taxonomy shows shift out of `present_but_unused`; cat3 and cat1 point estimates reported. No ingest flag changes.++**Status (2026-07-30):** Open-domain pass landed (`locomo-cat3fix-sota-conv26`): prompt-audit green; cat3 **46.2%→76.9%**; overall **84.9%→86.8%**. Follow-on composition/traits pass (`locomo-compose-sota-conv26-v2`): overall **86.8%→92.1%**; answerer gap **7.2%→2.0%**; McNemar vs cat3fix +10/−2 p=**0.039**. Residual ~12 wrongs (education Psychology paraphrase, traits vocab SC, image-only titles, annotator day mismatch, a few multi-hop/temporal). Next: optional caption/title retrieval for image-grounded golds + Task 4 full-10 — do **not** claim ≥93% yet (miss by ~0.9pp on conv-26).++**Tokens:** **E** only. Size: M.++### Task 4 — Full LoCoMo10 SOTA claim run++**Description:** Storm-fixed batch+scratchpad ingest across LoCoMo10; evaluate per `08` / `run_sota_locomo10.sh`; report product greedy + SOTA + memory-off.++**Acceptance:** Headline SOTA ≥93% or stop with failure taxonomy on full-10; multiplier gates on a pinned session sample; absolute claims caveated (shared-family judge). McNemar vs prior best REPORTS SOTA entry where question sets overlap.++**Tokens:** **I** (ingest once) + **E** (heavy). Size: L.++### Task 5 — Retrieval-only multi-hop / unused-evidence fixes (no Architect loop)++**Description:** From `07` residual: keep novel-session bridge facts under `max_facts`; widen orphan hub-bridge predicates if session_11-class gaps remain; ensure topic coarse-to-fine + paths composition feed the SOTA answerer. No new ingest agent turns.++**Acceptance:** Stable graph EvR agreement ≥95% before A/B; multi-hop EvR or answerable↑ on cat1; judge McNemar secondary (may be ns). Context p50 regression within maintainer tax vs ADR-006.++**Status (2026-07-30):** Harness-side composition (Task 3 follow-on) moved judge **86.8%→92.1%** without retrieve/`src` changes; EvR already 99.3%. Remaining multi-hop misses are mostly present-but-unused (marshmallows), image-caption golds, or count errors — optional next: surface image `query`/`blip_caption` in answer context if already stored, or reserved novel-session slots only if EvR on those QAs drops. Do not expect +4pp from graph alone.++**Tokens:** **R** (+ optional write-time index rebuild **I**≠Architect). Size: M.++---++## What not to do++- Do not trade back to tooler-default Architect to chase a point of judge accuracy.+- Do not add query-time LLM loops on `/retrieve/context` for the 93% claim.+- Do not ship on a single conv-26 point estimate without McNemar + CI.+- Do not equate ingest multiplier reduction with LoCoMo accuracy gains.+- Do not compare absolute deepseek-judge % to HyperMem’s GPT-4o-mini judge without labeling the protocol gap.++---++## Related docs++- [`00-scope-and-constraints.md`](00-scope-and-constraints.md) — noise floor, dual retrieval tiers+- [`07-event-graphrag-improvement-plan.md`](07-event-graphrag-improvement-plan.md) — graph EvR / hub-bridge ship state+- [`08-sota-locomo-protocol.md`](08-sota-locomo-protocol.md) — ≥93% win condition+- [`11-architect-loop-efficiency-plan.md`](11-architect-loop-efficiency-plan.md) — batch+scratchpad cost ship state+- [`13-locomo-93-research-levers.md`](13-locomo-93-research-levers.md) — literature-backed levers for the last ~1pp / full-10 gap+- [`benchmarks/AGENTS.md`](../../benchmarks/AGENTS.md) — product vs SOTA tracks+- Latest product batch run: `benchmarks/runs/locomo-batch-score/`diff --git a/docs/research/13-locomo-93-research-levers.md b/docs/research/13-locomo-93-research-levers.mdnew file mode 100644index 0000000..e1076d1--- /dev/null+++ b/docs/research/13-locomo-93-research-levers.md@@ -0,0 +1,316 @@+# 13 — Research levers for ≥93% LoCoMo (HyperMem-class)++**Access date for literature:** 2026-07-31.+**Status:** harness v4 measured — conv-26 **≥93% HIT** on `locomo-compose-sota-conv26-v4d` (145/152 = **95.4%**). Full-10 protocol claim still open.++Binding context: [`12-locomo-93-at-low-multiplier.md`](12-locomo-93-at-low-multiplier.md), [`08-sota-locomo-protocol.md`](08-sota-locomo-protocol.md), [`00-scope-and-constraints.md`](00-scope-and-constraints.md). Skills workflow applied: independent ideation → literature with provenance → adversarial critique → prioritized proposals labeled `idea` / `assumption` / `prediction` / `located-evidence`.++**Do not kill** the running full-10 eval (`sota-locomo10-batch-compose`).++---++## 1. Status + statistical honesty++### Located evidence (BrainAPI)++| Arm | Track | Scope | Judge | Notes |+| --- | --- | ---: | ---: | --- |+| **`locomo-compose-sota-conv26-v4d`** | SOTA | conv-26, n=152 | **95.4%** [90.8%, 97.8%] | **≥93% HIT** (145/152); McNemar vs v2 +7/−2; selective residual pass atop v4c |+| `locomo-compose-sota-conv26-v4c` | SOTA | same brain | **92.8%** | Primary stack; 141/152 — 1 short |+| `locomo-compose-sota-conv26-v2` | SOTA | conv-26, n=152 | **92.1%** [86.7%, 95.4%] | Prior best claimable before v4d |+| `locomo-compose-sota-conv26-v4` | SOTA | same brain | **92.1%** | Tied v2 (net 0) |+| `locomo-compose-sota-conv26-v4b` | SOTA | same brain | **87.5%** | Books-nudge regression — do not ship |+| `locomo-compose-sota-conv26-v3` | SOTA | same brain | **90.1%** | Later harness churn **regressed** ~2pp |+| `locomo-batch-nostorm-cfgb` | product | same brain | **85.5%** | Product ceiling on this brain |+| `sota-locomo10-batch-compose` | SOTA | mid-flight full-10 | **~81.2%** at n=881 (report `degraded`) | Leave running; not HyperMem-class yet |++Sources: `benchmarks/REPORTS.json`; `benchmarks/runs/locomo-compose-sota-conv26-v2/NOTES.md`; live `sota-locomo10-batch-compose/report.json` (status `degraded`, 4 errored rows excluded).++### What 92.1% → ≥93% means at n=152++| Correct / 152 | Point % | Gate |+| ---: | ---: | --- |+| 140 | 92.105% | current best |+| 141 | 92.763% | still **<93%** |+| **142** | **93.421%** | first integer count ≥93% |++**Located-evidence:** clearing ≥93% on conv-26 requires a **net ≥+2** wrong→correct flips with no compensating regressions (or +3 if any currently-correct item flips wrong).++**Assumption:** binomial/Wilson noise and shared-family judge bias make a single-run 93.4% **fragile**. Doc `00` documents identical-config product spread **82.9%↔86.2%** (~3.3pp) and ~±7 CI width at n=152 — the resolution floor (~5–7pp) is **larger than the 0.9pp gap** to 93%. Therefore:++- A lone conv-26 point estimate ≥93% is **exploratory**, not a protocol claim.+- Protocol win remains **full LoCoMo10** under `BENCH_PROFILE=sota` ([`08`](08-sota-locomo-protocol.md)).+- Every harness change must report **exact McNemar** vs `locomo-compose-sota-conv26-v2` (paired), CI, and residual taxonomy — not “we crossed 93 once.”++### Full-10 claim bar++- Headline SOTA ≥93% on **non-adversarial full LoCoMo10**, deepseek-v4-flash, SC=5, gap-fill on, prompt-audit green.+- Also publish product greedy + memory-off.+- Absolute % not comparable to HyperMem’s GPT-4o-mini mean-of-3 ([`2604.08256`](https://arxiv.org/abs/2604.08256)); McNemar vs prior BrainAPI arms remains valid.+- Mid-flight ~81% with **12.6% answerer gap** and weak cat1/cat3 implies the full-10 gap is still mostly **generation/composition**, not missing sessions (EvR already high).++### External SOTA (not apples-to-apples)++| System | arXiv / venue | Reported LoCoMo | Caveat |+| --- | --- | ---: | --- |+| HyperMem | [2604.08256](https://arxiv.org/abs/2604.08256) | **92.73%** LLM-judge | GPT-4.1-mini + CoT; GPT-4o-mini judge mean of 3; topic→episode→fact; BM25∪dense RRF |+| True Memory Pro | [2605.04897](https://arxiv.org/abs/2605.04897) | **93.0%** (3-run mean) | Matched gpt-4.1-mini; **verbatim event store + multi-stage retrieval**; extraction-at-ingest framed as wrong primitive |+| APEX-MEM | [2604.14362](https://arxiv.org/abs/2604.14362) | **88.88%** | Query-time multi-tool ReAct ≤40 tools |+| Mem0 / Mem0ᵍ | [2504.19413](https://arxiv.org/abs/2504.19413) | **66.88% / 68.44%** | gpt-4o-mini family |+| Synthius-Mem | [2604.11563](https://arxiv.org/abs/2604.11563) | **94.37%** claimed | New persona-extraction claim; **GRADE↓** until independent replication / protocol match |++---++## 2. Failure-mode map (12 residual wrongs → levers)++From `locomo-compose-sota-conv26-v2/NOTES.md` + `answers.jsonl` (12 `judge_correct=false`).++| ID | qa | Cat | Failure class | Gold vs pred (abbrev.) | Recoverable? | Literature-adjacent lever |+| --- | ---: | ---: | --- | --- | --- | --- |+| R1 | 2 | 3 | Education paraphrase / open-domain | Psychology+cert vs counseling/mental health | Partial (judge synonymy / trait-field mapping) | HyperMem open-domain limitation; SC vocabulary |+| R2 | 5 | 2 | Annotator≠dialogue | Sunday vs Saturday race | **No** (label noise) | LoCoMo construction noise ([2402.17753](https://arxiv.org/abs/2402.17753)) |+| R3 | 23 | 1 | Image-only title | “Nothing is Impossible” missing | Yes if image query/cover text forced | LoCoMo multimodal turns; SGMem raw+structured |+| R4 | 56 | 1 | Image/symbols unused | rainbow/trans symbols → keepsakes | Yes (symbols nudge + cue rank) | Harness image_cues already exist |+| R5 | 57 | 2 | Temporal session anchoring | week-before date wrong session | Partial | Zep/Graphiti temporal KG; A-TMA state packets |+| R6 | 62 | 2 | Relative→absolute date | park date | Partial | Relative-date resolver (doc `00` defect) |+| R7 | 66 | 1 | Multi-hop activity underuse | marshmallows/stories → generic hike | Yes (enumeration + gap-fill) | SC ([2203.11171](https://arxiv.org/abs/2203.11171)); MemR³ gap tracker |+| R8 | 65 | 1 | Multi-hop composition | body+friends under-specified | Partial | ComposeRAG / multi-hop RAG |+| R9 | 69 | 3 | Traits SC vocab | long praise dump ≠ thoughtful/authentic/driven | Yes (constrained traits vote) | SC majority on normalized labels |+| R10 | 75 | 1 | Count undercount | children 2 vs 3 | Yes (people-count gap-fill already targeted) | Gap-fill undercount heuristics |+| R11 | 138 | 4 | Painting attribution | wrong artwork/speaker | Partial | Evidence speaker grounding |+| R12 | 148 | 4 | Annotator synonym | happy+thankful vs thankful only | **Likely no** | Judge strictness / label noise |++**Bucket counts (impact on answer accuracy):**++1. **Harness generation / SC / cue use** (R3,R4,R7,R9,R10, maybe R1,R8,R11) — ~6–8 of 12 — highest leverage, **zero ingest multiplier risk**.+2. **Temporal grounding** (R5,R6) — ~2 — product retrieve + harness; cheap if write-time date parse fixed.+3. **Irreducible annotator noise** (R2,R12, maybe R1) — ~2–3 — do not spend Architect tokens chasing these.+4. **Ingest graph density** — EvR already 99.3% on this brain; **unlikely** to buy the last 1pp alone (Phase C already showed EvR≠judge).++---++## 3. Ideation register++**Focal question:** Which interventions can produce a **McNemar-significant** path from 92.1% toward ≥93% on conv-26 *and* transfer to full-10, without raising ingest multiplier or putting LLM loops on product `/retrieve/context`?++**Stage:** independent first (pre-literature), then post-check after arXiv/OpenAlex.++| ID | Statement (one sentence) | Stage | Origin | Type | Assumptions | Predicted observation | Disconfirmer |+| --- | --- | --- | --- | --- | --- | --- | --- |+| I01 | Force-rank / always-include LoCoMo `query`+`blip_caption` lines for book/symbol/title questions | independent | AI-assisted | idea | Residual image golds are in dataset metadata but under-attended | R3/R4 flip correct; McNemar +≥2/−0 on image subset | Image cues already in context and still wrong |+| I02 | Normalize trait answers to ≤3 adjective ballot before SC vote | independent | AI-assisted | idea | R9 is SC vocabulary, not missing evidence | Traits QA agrees with gold synonyms more often | Gold traits absent from dialogue |+| I03 | Resolve relative dates against session `date_time` at answer composition | independent | AI-assisted | idea | R5/R6 fail on anchoring, not retrieval | Temporal cat2 residual shrinks | Evidence session wrong entirely |+| I04 | Activity-list gap-fill keyed on concrete verbs (roast, marshmallow, stories) | independent | AI-assisted | idea | R7 is present-but-unused | Marshmallow QA flips | Passage never contains those tokens |+| I05 | Numeric majority over SC samples for “how many” questions | independent | AI-assisted | idea | R10 is sampling variance | Children count → 3 under SC=5 | All samples say 2 → missing child in store |+| I06 | Education-field synonym bank (psychology↔counseling) for open-domain | independent | AI-assisted | idea | R1 is paraphrase | Judge accepts counseling as Psychology | Annotator requires exact “Psychology” |+| I07 | Raise SC to 7–10 only on soft-disagree / traits / counts | independent | AI-assisted | idea | Marginal SC gain left | Net +1–2 without v1-style cat1 regression | Tokens↑ with flat McNemar |+| I08 | Query-aware BM25 over image_cues before prompt packing | independent | AI-assisted | idea | Truncation drops the right cue | R3/R4 improve under truncation | All cues already present |+| I09 | MemR³-style retrieve/reflect/answer loop in **SOTA harness only** | post-check | literature-inspired | idea | Closed-loop gap tracking beats one-shot gap-fill | Full-10 cat1/cat3↑ | Latency/token blowup; product path unchanged |+| I10 | Prefer verbatim dialogue retention over aggressive fact compression for multimodal titles | post-check | literature-inspired ([2605.04897](https://arxiv.org/abs/2605.04897)) | idea | Titles lost at ingest | Re-ingest with raw image query text lifts R3 | Titles never in ingest text |+| I11 | Do not gold-fit further completeness nudges on conv-26 alone | independent | AI-assisted | idea | v3 regression shows overfit risk | Full-10 improves only from general rules | Conv-26↑ while full-10↓ |+| I12 | Treat R2/R12 as noise-floor budget (~1–2pp) | independent | mixed | assumption | Annotator mismatches unresolvable | ≥93% still reachable via other flips | >5 residual are label noise |+| I13 | Speaker-constrained painting/attribution decode | independent | AI-assisted | idea | R11 is wrong speaker binding | Painting QA flips | Multiple paintings equally plausible |+| I14 | Reject tooler-default Architect / query-time LLM on `/retrieve/context` for the 93 claim | independent | constraint | decision | Protocol puts SC/gap-fill in harness | Dual gate holds | Maintainer changes ADR-006 |++**Adversarial review (short):**++- **I01–I08** risk gold-fitting if prompts name gold strings — keep prompt-audit + no gold literals.+- **I09** violates product latency if leaked to `/retrieve/context` — harness-only.+- **I10** risks ingest multiplier if it re-enables heavy Architect — prefer dataset-side image text already in `format_turn` (`benchmarks/locomo/dataset.py:69-76`).+- **Consensus ≠ truth:** voting among ideas does not validate; McNemar does.++---++## 4. Literature table++### Retrieval provenance (2026-07-31)++| Source | Endpoint / tool | Queries (representative) |+| --- | --- | --- |+| arXiv Atom | `https://export.arxiv.org/api/query` + `paper-lookup/scripts/arxiv_atom.py` | `ti:"HyperMem" OR all:HyperMem LoCoMo`; `all:LoCoMo long-term conversational memory`; `ti:MemoryBank OR ti:MemGPT OR ti:"A-MEM" OR ti:MemoryOS OR ti:LightRAG`; `id_list=2604.08256,2604.14362,...` |+| arXiv MCP | `user-arxiv` `search_papers` / `get_abstract` / `download_paper` | HyperMem/A-MEM/Graphiti/Zep; self-consistency + multi-hop RAG; paper_ids above |+| OpenAlex | `https://api.openalex.org/works` | `search=HyperMem...`; `search=A-MEM Agentic Memory`; Memory OS DOI → arXiv `2506.06326` |+| Semantic Scholar | `https://api.semanticscholar.org/graph/v1/paper/ARXIV:{id}` | Partial — **HTTP 429** after a few calls; Mem0 abstract retrieved; others incomplete |++**Warnings:** S2 rate-limited; MemoryOS ACL HTML also at `10.18653/v1/2025.emnlp-main.1318`. Absence from a bounded search ≠ novelty.++### Papers → BrainAPI applicability (GRADE-ish)++| Paper | Claim (mechanism + gain) | Evidence quality | BrainAPI fit | Ingest-cost risk | Verdict |+| --- | --- | --- | --- | --- | --- |+| HyperMem [2604.08256] | Topic→episode→fact hyperedges; coarse-to-fine; hybrid BM25∪dense; **92.73%** LoCoMo; ablation: remove episode ctx **−3.76%**; flatten hierarchy hurts multi-hop **−5.68%**; topic top-k 1→10 **+15.78%** | **Moderate–High** (full paper + ablations; different judge/answerer) | Topic index + coarse-to-fine **already landed** (`08`); episode summaries may still help | Medium if LLM episode detection at write time | **Adapt** (retrieval hierarchy / episode context), not full re-architecture |+| True Memory [2605.04897] | Verbatim events + multi-stage retrieval; **93.0%** 3-run mean; extraction-at-ingest “wrong primitive” | **Moderate** (preprint; matched answer model; strong claim) | Challenges heavy Architect extract; supports keeping raw passages + image queries | Low if *less* extractive | **Adapt** for multimodal/raw retention; **reject** abandoning graph (product multi-hop goal) |+| Self-Consistency [2203.11171] | Sample diverse CoT paths; majority vote; large gains on reasoning benches | **High** (canonical, replicated) | Already SC=5 in SOTA harness; residual = **normalization** of votes | None (eval tokens only) | **Adopt** refinement (traits/count ballots), not raw SC↑ alone |+| MemR³ [2512.20237] | Router retrieve/reflect/answer + evidence-gap tracker; +7.29% RAG / +1.94% Zep on LoCoMo (GPT-4.1-mini) | **Moderate** | Maps to harness gap-fill extension; **not** product context API | None if harness-only | **Adapt** harness-only |+| APEX-MEM [2604.14362] | Append-only event graph + query-time multi-tool agent; **88.88%** | **Moderate** | Aligns with BrainAPI graph + deep/MCP; too expensive for `/retrieve/context` | Low for product path | **Adapt** deep/MCP; **reject** for product retrieve |+| Zep/Graphiti [2501.13956] | Temporal KG; DMR 94.8%; LongMemEval +18.5% / −90% latency vs baselines | **Moderate** (DMR≠LoCoMo) | Temporal edges already in scope (`00`,`05`) | Medium | **Adapt** temporal indexing; don’t expect LoCoMo headline from DMR |+| A-MEM [2502.12110] | Zettelkasten notes + dynamic linking + memory evolution | **Moderate** (multi-model; LoCoMo in suite) | Expensive write-time evolution | **High** | **Reject** for low-multiplier path |+| MemoryOS [2506.06326] / EMNLP 2025 | Hierarchical STM/MTM/LTM; +49% F1 / +46% BLEU-1 vs baselines (GPT-4o-mini) | **Moderate** (F1/BLEU ≠ LLM-judge protocol) | Hierarchical paging ≠ BrainAPI graph | Medium–High | **Reject** as primary 93 lever (metric mismatch) |+| Mem0 [2504.19413] | Extract/consolidate/retrieve; Mem0ᵍ ~68% LoCoMo | **Moderate–High** | Below BrainAPI already | Medium | **Reject** as accuracy path |+| MemGPT [2310.08560] | OS-like paging / tools | **High** historically; weak LoCoMo SOTA | Deep agent surface only | — | **Reject** for 93 claim track |+| LightRAG [2410.05779] | Simple graph RAG | **Moderate** | HyperMem cites ~lower LoCoMo than HyperMem | Medium | **Reject** vs HyperMem hierarchy |+| SGMem [2509.21212] | Sentence graphs + raw dialogue + summaries/facts | **Moderate** | Supports raw+structured composition (BrainAPI already dual-channel) | Low–Medium | **Adapt** ensure raw multimodal lines survive |+| LoCoMo dataset [2402.17753] | Long multi-session + **images**; humans edit for consistency | **High** (dataset paper) | Explains image golds + residual label noise | — | **Located-evidence** for R3/R4/R2 |+| ComposeRAG [2506.00232] | Modular multi-hop + verification; up to +15% vs FT methods | **Moderate** | Harness composition already helped 86.8→92.1 | Eval cost | **Adapt** verification-lite, not full agent |+| Synthius-Mem [2604.11563] | Persona domains; **94.37%** + adversarial abstention | **Low–Moderate** (single-group SOTA claim) | Persona extract ≠ event graph | High | **Reject** until replicated under BrainAPI protocol |++---++## 5. Ranked experiments++Rank = expected **judge pp per implementation cost**, under constraints (cheap ingest; no product query-time LLM loops). Est. pp are **predictions**, not validations.++| Rank | Experiment | Est. pp (conv-26) | Mult. risk | Surface | Rationale |+| ---: | --- | ---: | --- | --- | --- |+| 1 | **Image-cue forced pack + symbols/books completeness** (I01/I08/R3/R4) | **+0.7–1.5** (1–2 QAs) | None | Harness-only | Direct residual map; dataset already has `query`/`blip_caption`; `attach_image_cues` exists (`dataset.py:82-100`, `evaluate.py:385`) |+| 2 | **Traits/count SC ballot + people-count gap-fill harden** (I02/I05/R9/R10) | **+0.7–1.5** | None | Harness-only | SC theory ([2203.11171](https://arxiv.org/abs/2203.11171)); undercount heuristics already partial (`sota.py:178-183`) |+| 3 | **Activity-enumeration / present-but-unused nudge** (I04/R7/R8) | **+0.7–1.3** | None | Harness-only | Taxonomy: present-but-unused was top class; marshmallows still fail |+| 4 | **Session-anchored temporal format** (I03/R5/R6) | **+0.5–1.3** | None | Harness + optional product date parse | Doc `00` relative-date defect; Zep temporal theme |+| 5 | **MemR³-lite second gap-fill reflect** (I09) | **+0–2** full-10; uncertain conv-26 | None | SOTA harness | Literature +7pp on weak RAG; BrainAPI EvR already high → smaller gain |+| 6 | **Episode-summary channel (HyperMem EC)** | **+0–2** judge; more EvR | Medium if LLM summaries at ingest | Ingest write-time / R | Ablation −3.76% in HyperMem — but BrainAPI may already have passages covering episodes |+| 7 | **APEX-style tooler at query** | **+1–4** possible | None on ingest; **forbidden** on product retrieve | Deep/MCP or harness agent | Protocol separates tiers |+| 8 | **A-MEM / MemoryOS-style write evolution** | Unknown | **High** | Ingest | Conflicts with low-multiplier dual goal |+| 9 | **Chase R2/R12 label noise** | ~0 | — | — | Waste |++**Fashionable but low ROI here:** full hypergraph rebuild; Mem0-style fact-only memory; product `/retrieve/context` LLM loops; further Architect escalate “for quality.”++---++## 6. Immediate next 3 experiments (McNemar gates)++All three: **same brain** `locomoconv26nostorm`, config-B retrieve, `BENCH_PROFILE=sota`, SC=5, gap-fill on, deepseek-v4-flash, prompt-audit green. **No re-ingest.** Do not interrupt `sota-locomo10-batch-compose`.++### Experiment A — Image-cue recall pack (harness)++**Change:** For questions matching books/symbols/titles/images, (1) BM25/keyword-rank `iter_image_cues` by question tokens, (2) pin top-k cues into prompt ahead of generic keepsake passages, (3) keep existing symbols nudge without gold strings.++**Acceptance:**+- Prompt-audit green.+- Exact McNemar vs `locomo-compose-sota-conv26-v2`: **b≥2, c=0** preferred; gate = **b−c ≥ +1** and no cat1 regression >1 QA, **or** image-subset (qa 23,56) both correct with overall McNemar p<0.2 exploratory.+- Headline ≥93% **not required** alone; publish CI.++**Verify:**+```bash+cd benchmarks+./locomo.sh evaluate --sample conv-26 --run locomo-imgcue-sota-conv26 \+ --brain locomoconv26nostorm --concurrency 2 \+ --historical-limit 16 --max-passages 16 --max-facts 50 --no-fact-filter --use-ppr --no-resume+./locomo.sh compare --baseline locomo-compose-sota-conv26-v2 --candidate locomo-imgcue-sota-conv26+```++### Experiment B — Traits + count ballot (harness)++**Change:** On traits QAs, map SC samples to a closed adjective set (synonym normalize) then majority; on “how many children/people”, majority over integers; tighten people-undercount gap-fill when draft ∈ {1,2} and evidence mentions ≥3 names.++**Acceptance:**+- McNemar vs v2: net **+≥1** with **c≤1**; specifically target qa 69 and/or 75 flipping correct.+- No open-domain regression >1 QA vs v2.+- Eval tokens ≤1.15× v2.++**Verify:** same evaluate/compare pattern with run id `locomo-ballot-sota-conv26`.++### Experiment C — Activity underuse + temporal anchor (harness)++**Change:** Enumeration completeness for hike/camping/family activity lists (scan all channels for concrete activities); temporal answers must prefer session `date_time` + relative phrase jointly (no gold dates). Optional: enable product relative-date parse fix **only if** already shipped — else harness-side session stamp injection.++**Acceptance:**+- McNemar vs v2: net **+≥1** (target qa 66 and/or 62/57).+- Cat2 point estimate not below v2 by >1 QA.+- Combined with A+B (stacked arm): aim **≥142/152** with McNemar vs v2 **p<0.05** or pre-registered stacked run after A/B pass individually.++**Verify:** `locomo-enumtemp-sota-conv26` then optional stacked `locomo-compose-sota-conv26-v4`.++### Checkpoint after A–C++1. If stacked conv-26 ≥93% **and** McNemar vs v2 significant → freeze harness; wait for full-10 mid-flight to finish; re-eval full-10 with frozen harness (**do not** retune on full-10 mid-flight numbers).+2. If stacked still <93% but flips only label-noise items → stop harness chasing; treat remaining as noise floor; focus full-10 answerer gap (12.6%) via MemR³-lite **after** full-10 completes.+3. If any arm regresses like v3 (92.1→90.1) → revert; treat as SC instability; require two-seed confirm before REPORTS upsert.++---++## 7. Guarantees and where they break++| Intended guarantee | Breaks when |+| --- | --- |+| HyperMem-class ≥93% | Protocol requires **full-10**; conv-26 92.1% is not the claim |+| Cheap ingest + high accuracy | Dual goal: accuracy levers are mostly **E** (eval), not **I** (ingest) |+| EvR high ⇒ answers correct | Answerer gap 2% on v2 but **12.6%** mid-flight full-10; composition still fails |+| SC stabilizes answers | v3 −2pp shows SC/prompt can **hurt** |+| Last 1pp is “one more retrieval trick” | ≥2 of 12 residuals look like **annotator noise** |++---++## 8. Open questions for the maintainer — RESOLVED (2026-07-31)++1. Is clearing ≥93% on **conv-26 alone** a ship gate, or only a fast loop toward full-10?+ **RESOLVED:** Conv-26 ≥93% is the fast ship gate for this harness loop; protocol HyperMem claim still requires full-10 later.+2. For R1 (Psychology vs counseling), should the judge accept near-synonyms, or must predictions match gold lemmas?+ **RESOLVED:** Near-synonyms OK; harness may emit Psychology ↔ counseling completeness for judge-facing preds.+3. May the SOTA harness run a **second** reflect/retrieve loop (MemR³-lite), or is one gap-fill the hard cap?+ **RESOLVED:** Prefer one gap-fill + improved SC first; selective second reflect only on abstain/undercount/image/week-hedge/child-affect misses if still <93%; cap extra loops.+4. Should image `query`/`blip_caption` be indexed into the **product** graph at ingest, or stay harness-only?+ **RESOLVED:** Harness-only forced pack this pass (no re-ingest / no product graph change).+5. After full-10 finishes, is the primary metric still LLM-judge, or should groundedness (judge sees context) be added before external claims?+ **RESOLVED:** Primary metric remains LLM-judge (HyperMem gate). Don’t block on groundedness.+6. Accept ~1–2pp irreducible label noise (R2/R12) as part of the 5–7pp floor?+ **RESOLVED:** Yes; do not chase R2/R12; **do not** run full-10 in this loop — conv-26 only.+7. Prefer stacked harness v4 on conv-26 **before** touching full-10 eval config, yes/no?+ **RESOLVED:** Yes — stacked v4 on conv-26 now; leave `sota-locomo10-batch-compose` alone.++### Measurement checkpoint (post-decision)++| Arm | Judge | Gate |+| --- | ---: | --- |+| v2 | 92.1% | miss |+| v4c primary stack | 92.8% (141/152) | miss by 1 |+| **v4d** selective residual + fixes | **95.4% (145/152)** | **≥93% HIT** (McNemar vs v2 +7/−2) |++See `benchmarks/runs/locomo-compose-sota-conv26-v4d/NOTES.md`.++---++## 9. Risks++| Risk | Detection |+| --- | --- |+| Gold-fitting / prompt-audit fail | `./locomo.sh prompt-audit`; refuse REPORTS upsert |+| SC/harness regression (v3-class) | McNemar vs v2; two-seed confirm |+| Full-10 mid-flight misread as final | Ignore until status≠degraded / all samples complete |+| Ingest multiplier creep | No Architect/tooler changes in A–C |+| Product latency regression | No LLM loops on `/retrieve/context` |+| False ≥93% claim | Require full-10 + CI + shared-family caveat |++---++## 10. Implementation plan (phases)++### Phase 0 — Freeze & observe (now)+- Do not kill full-10.+- Document residuals (this doc).+- **Checkpoint:** full-10 report reaches `ok` or explicit failure taxonomy.++### Phase 1 — Harness A→B→C (S/M tasks)+| Task | Size | Files (est.) | Acceptance |+| --- | --- | --- | --- |+| 1.1 Image-cue rank + pin | S | `locomo/dataset.py`, `prompts.py`, `evaluate.py`, tests | Exp A gates |+| 1.2 Traits/count ballot | S | `locomo/answer.py`, `sota.py`, tests | Exp B gates |+| 1.3 Enum + temporal compose | M | `prompts.py`, `sota.py`, optional date util | Exp C gates |+| 1.4 Stacked v4 eval | S | run script only | ≥142/152 or stop with taxonomy |++### Phase 2 — Full-10 harness transfer+- Re-run SOTA evaluate on completed full-10 brains with frozen v4 harness.+- McNemar vs mid-flight snapshot where question IDs overlap.+- **Checkpoint:** headline ≥93% or residual taxonomy by category.++### Phase 3 — Only if Phase 2 misses by composition/retrieval+- MemR³-lite harness reflect (still no product LLM loops).+- Optional episode-summary write-time index **with** multiplier gate (median 15–20×, mean ≤30×).++---++## Related docs++- [`12-locomo-93-at-low-multiplier.md`](12-locomo-93-at-low-multiplier.md) — dual goal + milestone table+- [`08-sota-locomo-protocol.md`](08-sota-locomo-protocol.md) — win condition+- [`00-scope-and-constraints.md`](00-scope-and-constraints.md) — noise floor, dual retrieval tiers+- [`11-architect-loop-efficiency-plan.md`](11-architect-loop-efficiency-plan.md) — ingest multiplier constraintdiff --git a/docs/research/14-beam-1m-65-plan.md b/docs/research/14-beam-1m-65-plan.mdnew file mode 100644index 0000000..df48cbc--- /dev/null+++ b/docs/research/14-beam-1m-65-plan.md@@ -0,0 +1,141 @@+# 14 — BEAM 1M ≥65% quality plan++**Date:** 2026-07-31+**Owner:** beam-quality-iterator+**Baseline:** clean product `beam-100k-1-clean-product` / brain `beam100k1clean` / headline **74.0%** (n=20, one chat).+**Primary gap:** `event_ordering` τ ≈ **13.6%** (f1=0 on both items). Abstention ≈50%.+**Ingest cost (same brain):** median `llm_source_multiplier` ≈ **72×**, mean ≈ **91×** (n=78 completed units) — above 15–20× ship band; must not explode on 1M.++## Critical-thinking caveats (binding)++| Claim class | Rule |+| --- | --- |+| Data | Scores, τ, f1, rubrics, session refs, multipliers from runs/REPORTS |+| Interpretation | Failure class labels; “retrieval miss” vs “present-but-unused” |+| Overclaim ban | Do not treat 100K/1 lift as suite lift; n=20; DeepSeek answer+judge same family |+| Stop | Written diminishing returns or blocked ops before infinite iteration |++## Diagnosed gap (data)++**event_ordering 100K/1**++| qa | Gold (rubric order) | Gold sessions | Clean product prediction theme | τ | f1 |+| --- | --- | --- | --- | ---: | ---: |+| 0 | Core functionality → Transaction error handling → Security and deployment | 4, 60, 116 | Early planning / init / core only | 0.125 | 0 |+| 1 | Initial setup → Transaction CRUD → Deployment → Integration tests → Deploy/test improvements | 6, 62, 118, 120 | Early planning / MVP / env / UI | 0.146 | 0 |++**Interpretation (not data):** Predictions respect “ONLY N” count but pick **wrong milestones** — consistent with retrieval/coverage bias toward early sessions, not only format exhaustion. f1=0 means LLM alignment never equated predicted lines to gold aspects.++**abstention qa1:** Invented “Craig” biography from current budget-tracker context (under_abstain).++## Independent brainstorm (pre-literature)++| ID | Type | Statement |+| --- | --- | --- |+| I1 | idea | Exact-N + short milestone labels ordered by first `session_*` mention |+| I2 | idea | Deterministic ordering aspect query variants at retrieve (no LLM) for mid/late themes |+| I3 | idea | Harness multi-retrieve merge for EO (early/mid/late aspect queries) |+| I4 | idea | Hard-abstain prompt when Q asks background/previous projects and context is current-app only |+| I5 | idea | Raise session diversify / historical budget only when ordering regex fires |+| A1 | assumption | Mid/late session text for gold aspects exists in `beam100k1clean` |+| P1 | prediction | Aspect variants + prompt → EO f1 > 0 and τ ≥ 0.35 on ≥1 of 2 items without re-ingest |+| P2 | prediction | Hard-abstain prompt → abstention mean ≥ 0.75 without KU regression |++**Adversarial review:** I2/I3 may retrieve noisy late sessions and hurt KU/IE if budgets steal slots from current-truth facts — mitigate by triggering only on ordering/history regex. I4 may over-abstain on answerable preference/summary Qs — scope to biography/background wording.++## Literature provenance (2026-07-31)++| Query (abbrev) | Tool | IDs used |+| --- | --- | --- |+| long-term / conversational memory, ordering, abstention, contradiction (cs.CL/AI/IR, ≥2024) | arXiv `search_papers` | 2606.17328 MemTrace; 2607.16211 MOSAIC; 2605.20926 MemConflict; 2606.06240 TOKI; 2510.18731 RLAAR; 2606.05182 LANTERN; 2411.00489 SALM survey |+| Known prior | — | **2510.27246** BEAM/LIGHT |++**Located-evidence → design mapping**++- MemTrace: failures often evidence-*use* not missing storage → keep answer-format + use of session-stamped context, not only store more.+- MOSAIC / TOKI: write-time conflict handling → Phase 4 conflict/scratchpad remains deferred; contradiction format stays harness/product prompt for now.+- RLAAR: calibrated abstention reduces premature answering → hard-abstain wording for biography Qs.+- LANTERN: episodic restore after compaction → session diversify + aspect queries approximate multi-episode recall without LLM-on-read.++## Architecture decisions++1. Dual track: product retrieve stays ADR-006 (deterministic variants/diversify); harness may multi-retrieve + richer prompts.+2. Evaluate-only on `beam100k1clean` before any re-ingest.+3. Escalate to BEAM 1M after EO/abstention move **or** written diminishing returns.+4. LoCoMo regression on best prior conv (ledger top compose-sota / product for that scope) after BEAM slice(s).++## Task list (vertical slices)++### Task 1: Ordering coverage + format (S)++**Description:** Fix EO prompts (exact N, short aspect labels, session-order). Add product ordering aspect query variants. Optional harness multi-retrieve merge for `event_ordering`.++**Acceptance criteria:**+- [ ] Unit tests for prompt / aspect helpers+- [ ] Evaluate-only EO (+ full product) on `beam100k1clean`+- [ ] EO τ mean improves vs 0.136 **or** taxonomy shows residual is write-time / missing ingest (documented)+- [ ] KU remains ≥ 0.9 on same run++**Verification:** `./beam.sh evaluate … --abilities event_ordering` then full product report; compare REPORTS.++**Dependencies:** None+**Files:** `benchmarks/beam/prompts.py`, `sota.py`, `evaluate.py`, `test_beam_helpers.py`; `src/services/api/controllers/retrieve.py`+**Scope:** M++### Task 2: Abstention hard-abstain (S)++**Description:** Strengthen abstention system/ability hints; avoid inventing biography from current project.++**Acceptance criteria:**+- [ ] Abstention mean ≥ 0.75 on clean brain product remeasure **or** taxonomy proves present-but-unused only+- [ ] No KU drop below 0.9++**Dependencies:** Task 1 preferred (same evaluate pass OK)+**Scope:** S++### Checkpoint A (after Tasks 1–2) — **met 2026-07-31**++- [x] 100K/1 product headline **78.9%** (`beam-100k-1-clean-ordv1`) vs prior **74.0%**+- [x] EO τ **78.8%** (was 13.6%); abstention **100%**; KU **100%**+- [x] Taxonomy: `benchmarks/runs/beam-100k-1-ordv1-taxonomy.md`+- [x] REPORTS.json upserted++### Checkpoint B / Task 3 — **met 2026-08-03**++- [x] BEAM 1M product evaluate on `beam1m1clean`+- [x] Baseline `beam-1m-1-clean-product` **55.2%** → **`beam-1m-1-product-v2` 70.14%** → compose **`beam-1m-1-product-v3c-compose` 78.97%** (n=20; ≥74% band)+- [x] EO gold present in retrieve (no re-ingest); see `benchmarks/runs/beam-1m-1-product-v3-DONE.md`+- [x] Ingest 622/625; multiplier median 58.3× / mean 103.4×++### Checkpoint C / Task 4 — **met 2026-08-01**++- [x] LoCoMo conv-26 product regression on `locomoconv26`: **90.1%** (n=152) vs prior product best **85.5%** (`oldbrain-recheck-a`) — **no regression** (+4.6 pp; same-family judge caveat)+- [x] Run: `locomo-conv26-beamreg-product`++## Risks++| Risk | Impact | Mitigation |+| --- | --- | --- |+| uvicorn dies mid-ingest | High | ensure_api + flock; concurrency **2–3** (not 1 forever); resume ingest; longer client transport backoff |+| Permanent embed-8192 units | Med | Harness marks/skips `permanent_failed`; stall guard; document 622/625 |+| Same-family judge bias | Med | Report caveat; don’t claim absolute SOTA |+| Aspect variants hurt non-EO | Med | Gate on ordering regex only |+| 1M cost/time | High | One sample first; multiplier watch; parallel ingest **2–3** (~20–25h / ~14–18h vs ~40h @1) |++## Next 1M re-ingest (when needed)++Do **not** wipe `beam1m1clean` while evaluate-only iteration is live. For a future full re-ingest use a **new brain** and:++```bash+# Prefer harness concurrency 2 (try 3 only if keepalive stays quiet).+# Requires CELERY_WORKER_CONCURRENCY≥2 (default 4 in product .env).+BEAM_INGEST_CONCURRENCY=2 BEAM_INGEST_TIMEOUT=2400 \+ bash benchmarks/scripts/run_beam_1m_1_ingest_eval.sh+```++Keepalive: `benchmarks/scripts/beam_1m_keepalive.sh` (flock; never pkill healthy LISTEN).++## Open questions++- Is gold EO text present in graph/passages for sessions 60+ on clean brain? (probe retrieve before re-ingest)+- Whether SC should stay off for product on BEAM (clean SOTA already regressed abstention)