BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
docs: add ADRs and GraphRAG/SOTA research notes
Christian5eb50cc2c276
diff --git a/.gitignore b/.gitignoreindex b3b0200..6c2a2df 100644--- a/.gitignore+++ b/.gitignore@@ -18,11 +18,13 @@ mcp-stdio-http-bridge/target/mcp-stdio-http-bridge/dist/plugins/**-docs/**+.gittui/node_modulestui/dist.cursor-tui/embedded\ No newline at end of file+tui/embedded++.agent_tmp\ No newline at end of filediff --git a/docs/decisions/001-ingestion-structure.md b/docs/decisions/001-ingestion-structure.mdnew file mode 100644index 0000000..9ed4dd2--- /dev/null+++ b/docs/decisions/001-ingestion-structure.md@@ -0,0 +1,25 @@+# ADR-001: Ingestion Structure++## Status++Accepted++## Date++2026-07-27++## Context++--++## Decision++--++## Alternatives Considered++--++## Consequences++--diff --git a/docs/decisions/002-structured-ingestion-specific-processing.md b/docs/decisions/002-structured-ingestion-specific-processing.mdnew file mode 100644index 0000000..076e1a7--- /dev/null+++ b/docs/decisions/002-structured-ingestion-specific-processing.md@@ -0,0 +1,11 @@+/\*++- File: /docs/decisions/002-structured-ingestion-specific-processing.md+- Project: brainapi2+- Created Date: Monday July 27th 2026+- Author: Christian Nonis <[email protected]>+- ***+- Last Modified: Monday July 27th 2026 4:23:55 pm+- Modified By: Christian Nonis <[email protected]>+- ***+ \*/diff --git a/docs/decisions/003-simple-one-shot-context-api-retrieval.md b/docs/decisions/003-simple-one-shot-context-api-retrieval.mdnew file mode 100644index 0000000..7b9e7f4--- /dev/null+++ b/docs/decisions/003-simple-one-shot-context-api-retrieval.md@@ -0,0 +1,11 @@+/\*++- File: /docs/decisions/003-simple-one-shot-context-api-retrieval.md+- Project: brainapi2+- Created Date: Monday July 27th 2026+- Author: Christian Nonis <[email protected]>+- ***+- Last Modified: Monday July 27th 2026 4:24:19 pm+- Modified By: Christian Nonis <[email protected]>+- ***+ \*/diff --git a/docs/decisions/004-agentic-navigation-retrieval.md b/docs/decisions/004-agentic-navigation-retrieval.mdnew file mode 100644index 0000000..4772207--- /dev/null+++ b/docs/decisions/004-agentic-navigation-retrieval.md@@ -0,0 +1,11 @@+/\*++- File: /docs/decisions/004-agentic-navigation-retrieval.md+- Project: brainapi2+- Created Date: Monday July 27th 2026+- Author: Christian Nonis <[email protected]>+- ***+- Last Modified: Monday July 27th 2026 4:24:37 pm+- Modified By: Christian Nonis <[email protected]>+- ***+ \*/diff --git a/docs/decisions/005-hybrid-one-shot-context-retrieval.md b/docs/decisions/005-hybrid-one-shot-context-retrieval.mdnew file mode 100644index 0000000..9877f10--- /dev/null+++ b/docs/decisions/005-hybrid-one-shot-context-retrieval.md@@ -0,0 +1,59 @@+# ADR-005: Hybrid One-Shot Context Retrieval over Agentic Navigation++## Status++Accepted — **partially superseded by ADR-006** on retrieval latency and request defaults. The hybrid one-shot design below stands; the accepted ~5s p50 and the "defaults keep `use_ppr` and `sufficiency_retry` off" statement in the Decision section no longer hold.++## Date++2026-07-28++## Context++ADR-003 keeps `/retrieve/context` as a single request/response. ADR-004 considers agentic multi-step graph navigation. On LoCoMo conv-26, early GraphRAG work reached ~75% judge accuracy, but truncated eval logs hid the real split of blame: with full instrumentation, answerable rate was already ~93% while judge accuracy lagged by ~20 points. Remaining misses were mostly incomplete lists, relative-date phrasing, open-domain abstentions, and under-budget multi-hop evidence—not a need for a general agent loop.++We needed a retrieval posture that raises evidence coverage and multi-hop recall without replacing the simple context API or re-ingesting brains for every experiment.++## Decision++Keep the one-shot `GetContext` API, and enrich a single call with hybrid retrieval:++1. **Passage budget** — raise default passage/historical limits so evidence-session recall is not budget-capped.+2. **Hybrid passage ranking** — vector + keyword retrieval fused with reciprocal rank fusion (RRF).+3. **Enumeration query decomposition** — for list/what/which questions, issue entity/relation sub-queries and union passage sets before fusion.+4. **Optional Personalized PageRank** over seed neighborhoods (`use_ppr`) when multi-hop still lags min-seed-distance ranking.+5. **Optional sufficiency retry** (`sufficiency_retry`) — one targeted follow-up retrieval when assembled context looks insufficient.+6. **Recall-preserving fact filter fallback** — without an LLM adapter, do not truncate candidates before score-based capping.+7. **Benchmark answerer prompts** — enumeration completeness, relative temporal phrasing, and hedged open-domain answers; leave `JUDGE_SYSTEM` unchanged so scores stay comparable.++Defaults keep `use_ppr` and `sufficiency_retry` off for API compatibility; the LoCoMo harness enables them when measuring the upgraded path.++## Alternatives Considered++### Full agentic navigation (ADR-004)+- Pros: Can chase missing evidence across many hops; flexible for open-ended exploration.+- Cons: Higher latency/cost variance; harder to attribute failures; changes the product contract away from one-shot context.+- Rejected for now: LoCoMo gaps closed enough with one sufficiency pass and better budgets; agent loop not required for the measured failures.++### Prompt-only answerer changes+- Pros: Cheap, no API/runtime coupling.+- Cons: Cannot fix gold that never reaches context; alone left multi-hop weak.+- Rejected as sole strategy: used as Phase B, then combined with retrieval upgrades.++### Changing the judge prompt+- Pros: Would inflate headline accuracy on near-synonyms/supersets.+- Cons: Breaks comparability with v5/v6 and hides real system quality.+- Rejected: judge stays fixed; soft metrics (answerable rate, evidence-session recall, answerer gap) diagnose loss.++### Community-level GraphRAG summarization+- Pros: Better global sense-making for open-domain themes.+- Cons: Failures were local list completeness and temporal phrasing more than global summaries.+- Rejected for this increment.++## Consequences++- `/retrieve/context` remains one-shot but accepts richer options (`max_passages`, `max_facts`, `use_ppr`, `sufficiency_retry`).+- Eval must log full `source_passages` / `historical_context` / `retrieved_session_ids`; truncated context logs misdiagnose retrieval.+- Latency and token cost rise with larger passage budgets and optional PPR/sufficiency (p50 retrieve ~5s on the upgraded LoCoMo run).+- On LoCoMo conv-26 with the upgraded path + answerer prompts: ~86% judge accuracy, ~97% full evidence-session recall, multi-hop ~81%, temporal ~92%.+- Agentic navigation (ADR-004) stays open for workloads where one sufficiency retry is not enough; it is not the default retrieval model.diff --git a/docs/decisions/006-two-tier-retrieval-latency-budget.md b/docs/decisions/006-two-tier-retrieval-latency-budget.mdnew file mode 100644index 0000000..d9e2811--- /dev/null+++ b/docs/decisions/006-two-tier-retrieval-latency-budget.md@@ -0,0 +1,68 @@+# ADR-006: Two-Tier Retrieval with a Sub-Second Context API++## Status++Accepted — supersedes ADR-005 on retrieval latency and on request defaults. ADR-005 remains the record of the hybrid one-shot design, which this ADR keeps.++## Date++2026-07-28++## Context++ADR-005 accepted rising latency as a consequence of larger passage budgets and optional PPR/sufficiency retry, recording "p50 retrieve ~5s on the upgraded LoCoMo run" as an acceptable cost of the accuracy gain. That trade is no longer acceptable as stated, for two reasons.++First, `/retrieve/context` is the surface that agents, recommenders, and search integrations call synchronously. A ~5 s p50 is not viable there regardless of accuracy. Measured from `benchmarks/runs/*/report.json`: p50 4053 ms at the pre-upgrade config and 4793 ms after, p95 up to 6333 ms, and a **minimum** observed latency of 2508 ms across 152 questions — so no question, however simple, is close to budget. The two best-scoring configurations are also the two slowest and the two most expensive (4.06M → 7.74M LLM tokens for the same question set).++Second, the accuracy justification for spending that latency is weaker than ADR-005 assumed, because the measurement it rested on could not see the graph channel at all. `benchmarks/locomo/evaluate.py` caps the logged `text_context` at 20,000 characters and the API writes passages into that field before graph facts; verified against `locomo-conv26-push75-d/answers.jsonl`, all 152 rows are exactly 20,000 characters, none contains a graph fact line, and no `triples` field is logged. Both `answerable_rate` and `evidence_session_recall_full` derive from that field, so every reported retrieval number describes the passage channel only. ADR-005 itself recorded the requirement to log full context; the cap defeated it.++Separately, ADR-005's statement that defaults keep `use_ppr` and `sufficiency_retry` off is no longer true of the code, and its stated reason — API compatibility — was undermined by a latent defect: the controller read these flags as `getattr(request, "use_ppr", True)` while the Pydantic model declared `False`, so the fallback was dead and the flags were unreachable for every HTTP caller. The code and the ADR disagreed with each other and with the intent.++## Decision++**1. Two tiers, one graph.**++| Surface | Budget | Admissible techniques |+| --- | --- | --- |+| `/retrieve/context` | **p50 < 1000 ms**, cheap | Index-backed lookups, precomputed structures, write-time work, feature-based ranking over already-computed signals |+| Other REST endpoints and MCP tools | Depth allowed | Iterative retrieve-then-reason, agentic traversal, LLM and cross-encoder reranking, verification loops |++No LLM call on the context path. Gains there must come from moving work to write time, from indexing, or from cheap ranking — not from spending more inference at query time.++**2. Relocate rather than optimise.** Measurement of serial depth indicates two stages cannot fit the budget at any per-operation latency: the sufficiency retry, which is by definition a second full retrieval pass, and the per-entity dossier walk, which issues one unbatched vector fetch per edge. Both move to the deep tier. Passage retrieval is flattened to a single query with batched embedding and concurrent dense/keyword search. What remains is roughly 6-8 serial I/O operations.++**3. The cheap insufficiency signal stays, as a signal.** The lexical sufficiency check remains on the context path, but it returns an explicit insufficiency indicator to the caller instead of triggering a retry. This closes the gap where the API silently returned weak context, and lets a caller escalate to the deep tier by choice.++**4. Defaults follow the budget.** `use_ppr` defaults to `True`: it is one Cypher call over the seed set plus in-process power iterations, no LLM, and it is the change credited with moving multi-hop full evidence-session recall from 62.5% to 96.9% on a deterministic retrieval metric. `sufficiency_retry` defaults to `False` because decision 2 removes it from this path; defaulting it on would institutionalise what this ADR exists to remove. The `getattr` fallbacks are deleted so the request schema is the single source of defaults and the two can no longer drift.++**5. Instrument before optimising.** The stage attribution above is derived from serial depth in code, not from measurement. Per-stage timing lands before any relocation work, and Phase 3 gates on measured p50 < 1000 ms with per-category evidence recall held.++## Alternatives Considered++### Keep one uniform retrieval path and accept ~5 s (status quo per ADR-005)+- Pros: no routing complexity; a single contract to reason about.+- Cons: unusable for synchronous agent, recommender, and search integrations, which are the stated product targets.+- Rejected: the accuracy that latency buys is real but is not worth making the primary surface unusable, particularly while the graph channel's contribution is unmeasured.++### Make agentic navigation the default and retire one-shot+- Pros: highest ceiling on hard multi-hop; handles composition across event hubs, which the current two-hop flow-key-constrained traversal cannot.+- Cons: roughly an order of magnitude more cost and latency variance; harder failure attribution; abandons the one-shot product contract.+- Rejected, consistent with ADR-005: with evidence coverage already high on the passage channel, an agentic loop would mostly re-retrieve what is already in the prompt. It remains the escalation tier.++### Optimise the existing stages in place+- Pros: no contract change; no relocation work.+- Cons: the dossier walk alone exceeds 1000 ms at an optimistic 5 ms per operation, and the sufficiency retry is a second full pass by construction. Neither is reachable by constant-factor improvement.+- Rejected: these must be deleted from the path, not tuned.++### Cross-encoder reranking on the context path+- Pros: strong precision gains reported in the literature; the current fact filter is a no-op, so ranking is pure vector distance.+- Cons: a hosted model is a network hop on a path with no headroom; in-process over ~50 candidates costs 50-200 ms, i.e. 5-20% of the entire budget, for a component that is not the bottleneck.+- Rejected for this tier; adopted as the deep-tier escalation. The context tier gets a linear or small-GBDT ranker over features already computed — vector distance, PPR score, recency, provenance overlap, variant strength.++## Consequences++- `/retrieve/context` gains an explicit insufficiency signal and loses the automatic retry. Callers relying on the retry must opt into the deep tier.+- The response may carry optional per-stage timings when instrumentation is enabled; this is additive and off by default.+- A router is required to decide which tier serves a query. Candidate signals: cross-event-hub composition, enumeration completeness, multi-entity comparison, and the insufficiency indicator. Entity resolution confidence must gate escalation — a beam-search agent on top of low-confidence resolution will pursue a wrong entity confidently and expensively.+- ADR-005's reported figures (~86% judge accuracy, ~97% evidence recall, multi-hop ~81%, temporal ~92%) are retained as history but must not be quoted as current quality. They were produced with an answer prompt containing conv-26 gold answer strings, on brains carrying wrongly-deprecated event edges, and with the graph channel absent from the logged context. A re-baseline is in progress.+- Benchmark arms must send `use_ppr` unconditionally. `benchmarks/locomo/client.py` omits the flag when false, so with the new server default an arm configured `use_ppr=false` would silently run with PPR and be unreproducible.diff --git a/docs/research/00-scope-and-constraints.md b/docs/research/00-scope-and-constraints.mdnew file mode 100644index 0000000..6a98b71--- /dev/null+++ b/docs/research/00-scope-and-constraints.md@@ -0,0 +1,122 @@+# Research Scope and Confirmed Constraints++Confirmed with the maintainer on 2026-07-28. Every proposal in `01`-`05` is judged against this.++## Product goal++A memory layer that (a) answers accurately every time, (b) surfaces non-obvious connections via multi-hop, and (c) is a substrate for agents, recommenders, and search engines.++## Where quality hurts most today++1. **Multi-hop** — single-hop questions are acceptable; chained questions fail.+2. **Temporal** — superseded facts come back as current truth.++These two set the priority order. A proposal that improves single-hop precision while leaving these untouched ranks below one that moves either.++## Retrieval architecture: two tiers, one graph++The maintainer's intent for the empty ADRs 003/004 is **a router, not a winner**:++| Surface | Budget | Implication for proposals |+| --- | --- | --- |+| `/retrieve/context` (one-shot context API) | **Sub-second, fast and cheap** | No iterative LLM loops, no per-query LLM reranking, no multi-call decomposition on this path. Gains must come from indexing, precomputation, graph structure, or cheap learned rerankers. |+| Other REST endpoints + MCP tools | Deeper navigation allowed | Agents may search iteratively and in depth here. Iterative retrieve-then-reason, agentic traversal, and verification loops are admissible. |++Anything expensive must be either moved to write time or exposed only on the deep-navigation surface. "Make retrieval smarter with another LLM call" is not an acceptable answer for the context API.++## Workstreams++| Doc | Workstream |+| --- | --- |+| `01-ingestion-extraction.md` | Raw input to written graph: chunking, agent swarm, entity resolution, consolidation |+| `02-retrieval-multihop.md` | Query-time entry points, traversal, filtering, ranking, agentic navigation |+| `03-memory-substrate.md` | Data model, indexing, knowledge lifecycle, scaling |+| `04-evaluation-and-applications.md` | Measurement validity, failure attribution, recommendation/agent-surface metrics |+| `05-temporal-truth.md` | Cross-cutting: how the system knows what is currently true, write time through read time |+| `07-event-graphrag-improvement-plan.md` | Active event-hub GraphRAG execution plan (post Phase 1); see also `06` Phase 4 |++## Decisions already taken during research++- **Event-leg supersession is a bug, not a trade-off.** `_invalidate_superseded_relationships` (`src/workers/tasks/ingestion.py:465-505`) deprecates an actor's older `MADE` edges because the Architect is instructed to use generic predicate names, and `_is_currently_valid` (`src/services/api/controllers/retrieve.py:420-426`) then hides them from retrieval. The maintainer does not recall the original intent and has ruled it a bug; a minimal fix plus regression test is being shipped ahead of this roadmap. Proposals must assume the incorrect deprecation is gone, and must not assume existing brains have been repaired — no backfill was authorised.+- **The 3072-dim embedding default silently disables the pgvector HNSW index** (`src/lib/postgresql/vectors.py:62-63` returns no DDL above 2000 dims; `.env.example:43-47` ships 3072). The fix is deferred to this roadmap and must be decided by a recall-versus-latency measurement, not asserted. Candidates to evaluate: dimension truncation, `halfvec` indexing up to 4000 dims, and a loud warning when an index is skipped.++## Measurement noise floor — binding on every claimed improvement++`locomo-conv26-push75-c` and `locomo-conv26-push75-d` have **byte-identical manifests** (same brain, same config, same `deepseek-v4-flash` answer and judge models, `historical_limit=16`, `max_passages=16`, `max_facts=50`, `use_ppr=true`, `sufficiency_retry=true`) and yet score **82.9% and 86.2%**. That is a **3.3-point run-to-run spread with nothing changed.**++At `n_total=152` the 95% CI is roughly ±7 points, and the `push75-b` (75.7%, CI 68.3–81.8) and `push75-c` (82.9%, CI 76.1–88.1) intervals **overlap**. Consequences:++- The "+7.2 points from the push75-c config" is **not statistically established**. Do not quote it as a settled gain.+- The benchmark currently **cannot resolve differences below roughly 5–7 points**, which is larger than most individual improvements worth shipping. Any proposal justified by a single-run delta of a few points is unfalsifiable as measured today.+- **Evidence recall is the trustworthy signal** in these runs *for the passage channel*, because it is computed on retrieval output rather than through the judge: full session recall moved **82.7% → 97.3%** from `push75-b` to `push75-c`. Prefer retrieval-side metrics over judge accuracy when arguing for a change. **Caveat (post Phase 1):** graph-channel evidence recall is logged but **not yet trustworthy** — identical-config runs on the same brain agree on graph session recall for only ~39–42% of questions until fact selection is order-stable. Do not A/B on graph EvR until that defect is fixed. Passage recall remains deterministic.+- Latency and cost are also from `report.json` and are trustworthy: p50 retrieval **4053 ms → 4793 ms** and total LLM tokens **4.06M → 7.74M** for the same 152 questions.++## Latency reality vs the stated budget++The sub-second target for `/retrieve/context` is **not met yet**, but the dominant cost is gone. Pre-deletion measured retrieval p50 was **~4.8 s**; post-dossier-deletion live p50 is **~1.4 s** (clean-brain eval arms, after sync+restart), with no detectable accuracy loss. Remaining gap to budget ≈ **400 ms**, likely embedding fanout (Phase 3.2). Any roadmap must still treat "close the last ~400 ms" as first-class — and must state explicitly whether the sub-second target or an accuracy gain wins where they conflict.++## Measurement outcomes (Phase 1, live after sync+restart)++Clean brain `locomoconv26clean` (19/19 conv-26 sessions), de-fitted prompt, same config as `baseline-clean-a`. Full table in `06-roadmap.md`.++1. **Extraction fixes earn their keep on graph hygiene**, not on detectable judge accuracy at n=152 (0 deprecated EVENT-touching edges vs 219; 0 type-named placeholders vs 3; denser graph 537/752/410 vs 411/584/281).+2. **The graph channel does not move judge accuracy above the resolution floor** vs passages-only (McNemar ns). Graph EvR is logged (~53% point estimate) but cross-run agreement is only ~39–42% — measurable, not yet a measurement. The harness can see the channel; it is not yet converting into answers.+3. **Dossier deletion earns its keep on latency**: ~4.8 s → ~1.4 s p50, no accuracy loss. Sub-second not yet met.++**Still open in the measurement stack:** graph EvR non-determinism (Phase 0 follow-up); judge remains shared-family with the answerer (`deepseek-v4-flash`); groundedness remains unmeasured because the judge never sees retrieved context. Absolute accuracy claims stay unsupportable; paired A/B comparisons remain valid.++## Temporal semantics — decided by the maintainer++- **`valid_at` means document/ingest time.** This is correct as implemented; event time lives in `happened_at` only. Do not propose "fixing" that as a conflation.+- **`/retrieve/context` will default to current-truth-only**, with an opt-in mode for history. This is an accepted breaking change to what callers receive. Plans must account for the migration, and for the fact that the filter must be index-backed rather than LLM-mediated to fit the sub-second budget.+- **No targeted supersession benchmark for now.** The maintainer chose to prioritise fixing over measuring, with the explicit understanding that LoCoMo category 2 measures event *localization* ("when did X happen?") rather than supersession — it scores 0.86, above multi-hop — and that roughly 12 of 1,986 questions probe current truth. Consequence for the roadmap: temporal fixes are verifiable by unit test but their end-to-end quality effect is **not currently measurable**, so do not claim benchmark gains for them.++### Verified temporal defects (fixes authorised and in progress)++- **Relative-date resolution never runs.** `_DATE_INPUT_FORMATS` (`src/utils/dates.py:5-16`) has no time-bearing format, so the real `source_timestamp` from `benchmarks/data/locomo10.json` (`'1:56 pm on 8 May, 2023'`, passed through at `benchmarks/locomo/dataset.py:131`) yields `None` and `resolve_relative_date` returns every expression verbatim. Confirmed by execution. The resolver logic itself is correct against a parseable reference, so this is a format gap.+- **The recency scorer is dead.** `src/core/search/entity_info.py:230` calls `fromisoformat` on the `dd/mm/YYYY` that `src/utils/dates.py:76` writes; it always raises, a bare `except` swallows it, and `recency` is exactly `1.0` for every node — making the `recency * 0.2` term a constant across all candidates.+- **The current-truth filter guards only one of two evidence channels.** `_is_currently_valid` is called at `src/services/api/controllers/retrieve.py:787` on the triple channel only; the passage channel (`:667-725`) and `historical_context` (`:851-880`) have no temporal predicate, so an invalidated edge and the original sentence asserting it can be returned in the same payload. This is the seam that produces the reported symptom.++### Citation caveat++`2606.01435` (*Don't Ask the LLM to Track Freshness*) is verified real and correctly quoted on its headline result (+10.8 on FactConsolidation single-hop; Zep/Graphiti at 7%). **But its strong result is for `max(serial)` with explicit monotone serials.** The paper's own LongMemEval port to `max(timestamp)` — which is what BrainAPI would actually do — ties or slightly loses to LLM judgment (57.8% vs 64.4%, n=45). Cite this paper for the latency argument for a no-LLM query path, not as evidence that deterministic aggregation is more accurate.++## Maintainer decisions (2026-07-29) — Event GraphRAG track++Mirror of answers recorded in `07-event-graphrag-improvement-plan.md`. Full rationale lives there.++1. Context-path cross-event = write-time hub adjacency + cheap hub PPR; full iterative composition = deep/MCP only (ADR-006).+2. Primary metric = **stable graph EvR** (multi-hop / single-hop); judge accuracy secondary (may stay flat).+3. Graph EvR is a measurement only at **≥95%** identical graph-session sets across two identical-config runs.+4. Diversify under `max_facts` by **session + event-hub blend**; maintainer latency tax ceiling **2000 ms**, but ADR-006 still targets **p50 < 1000 ms** — prefer both; escalate with numbers if tax exceeds headroom to 1 s.+5. Cross-hub path strings not in default `text_context`; structured `paths` for deep/MCP only.+6. Temporal conflict: prefer recency in ranking and return both as meta.+7. Leiden out; k-core / deterministic hierarchy deferred (open-domain later).+8. Query–edge embedding similarity allowed on context ranker (no LLM); A/B.+9. Diversification harness arms: both `max_facts=50` and `max_facts=40`.+10. Hub-bridge index owned by **memory substrate (`03`)** — Phase A is retrieval determinism only.++## Roadmap decisions (2026-07-28, after synthesis)++- **The conv-26-fitted answer prompt rules were deleted and the benchmark re-baselined** at 82.2% (`baseline-clean-a`). Old fitted numbers must stay labelled as such.+- **The passages-only ablation ran on the clean brain** (`locomoconv26clean`). Result: no LoCoMo judge gain above the resolution floor; see Measurement outcomes above. Phase 4 is gated on non-LoCoMo-accuracy goals (EvR lift, multi-hop composition, product surfaces).+- **Phase 0 (harness) and Phase 2 (cheap defects) completed** in parallel; Phase 1 measurement is complete.+- **The judge stays shared-family** (`deepseek-v4-flash`); a human labelling pass was declined for now, and the judge still never sees the retrieved context — so **groundedness remains unmeasured**, and no claim about faithfulness is currently supportable.++## Judge configuration — decided++**The judge stays `deepseek-v4-flash`, the same model as the answerer.** The harness now supports full separation (`BENCH_JUDGE_*`, with Azure `gpt-4o` auto-selected when credentials are visible) and records `judge_shares_answer_family: true` when it is not used, so the condition is at least declared in every manifest.++What this does and does not cost:++- **Paired comparisons remain valid.** When two arms are scored by the same judge on the same questions, self-preference bias applies to both and largely cancels. Since nearly all planned work is A/B — config arms, the passages-only ablation, before/after a fix — the headline is usable for the comparisons that matter.+- **Absolute claims are not supportable.** "BrainAPI achieves X%" carries unmeasured self-preference bias and no human calibration. Do not publish or compare the figure against externally reported systems.+- **Groundedness stays unmeasured** regardless of model choice, because the judge never receives the retrieved context. A prediction that is correct from the model's parametric knowledge scores identically to one derived from the graph — the exact distinction a memory product exists to make. This is the most significant remaining gap in the measurement stack.++## Harness default divergence — deferred++The harness CLI defaults `use_ppr` off while production now defaults it on, so a plain `evaluate` run does not measure production behaviour. Deliberately deferred until Phase 3 settles which features remain on the context path. Every arm must state its flags explicitly in the meantime; the manifest now records what was actually sent.++## Deliverable++A synthesized, ranked cross-workstream roadmap for review. Implementation does not start until that roadmap is approved — with the single exception of the supersession fix above, which the maintainer authorised immediately because it corrupts data on every ingest.diff --git a/docs/research/01-ingestion-extraction.md b/docs/research/01-ingestion-extraction.mdnew file mode 100644index 0000000..1cbc7e1--- /dev/null+++ b/docs/research/01-ingestion-extraction.md@@ -0,0 +1,644 @@+# 01 — Ingestion and Knowledge Extraction++Workstream: everything from raw input to written graph. Judged against `00-scope-and-constraints.md`, which fixes the priority order as **multi-hop first, temporal second**, and states that expensive work must move to write time. This workstream *is* write time, so most of the burden of both named pain points lands here.++Read-time consumption of temporal fields is deferred to `05-temporal-truth.md`; this document owns what gets written and how it is marked.++All `file:line` anchors below reflect the tree as read on 2026-07-28. One file, `src/workers/tasks/ingestion.py`, was modified by concurrent work while this analysis was in progress: roughly 24 lines were inserted around line 462, so anchors in that file beyond that point are shifted by about that much relative to the current working tree. Gap 4 is where it matters substantively.++---++## What this workstream does++### Entry points++Three, all landing in `src/workers/tasks/ingestion.py`:++| Entry | Task | Notes |+| --- | --- | --- |+| Free text / textual payload | `ingest_data` (`src/workers/tasks/ingestion.py:527`) | The main path. |+| Structured triples | `ingest_structured_data` (`src/workers/tasks/ingestion.py:1054`) | Caller supplies subject/event/object triples; Scout backfills missing entities. |+| Files | `ingest_file` (`src/workers/tasks/ingestion.py:1379`) | Docling → markdown per page → one `ingest_data` per page. Only when `OCR_MODE=docling`. |+| Files, remote OCR | `src/services/api/routes/ingest.py:138-179` | When `OCR_MODE=docparser` (`src/config.py:571-572`) the route instead posts the file to an external `DOCPARSER_ENDPOINT/ingest` with a bearer token, bypassing `ingest_file` entirely. |++`src/core/files/` contains only an `__init__.py`; `src/services/input/` and `src/services/data/` are thin adapter re-export modules (392 and 79 bytes). There is no separate file-handling or input-normalization layer — file handling is inline in `ingest_file`.++### The free-text path, traced++1. **Raw text is persisted losslessly first.** `ingest_data` builds a `TextChunk` from the whole payload and saves it (`src/workers/tasks/ingestion.py:574-581`), embeds it, and adds it to the `"data"` vector store (`:582-592`). This is a genuine strength and is the same "non-lossy episode store" that Zep (2501.13956, §2.1) and Engram (2606.09900) argue is essential. Graph elements point back to it via `source_chunk_ids`.++2. **Observations** run only in `accurate` mode (`:597-614`), and are called with `context=None` (`:601`).++3. **Enrichment** — `enrich_kg_from_input` → `_enrich_kg_impl` (`src/core/saving/auto_kg.py:81`):+ - Mode select: `mode = "coarse" if config.pipeline_mode == "lightweight" else "granular"` (`auto_kg.py:112`).+ - **Scout** `scout_agent.run(...)` (`auto_kg.py:115-123`) with `reference_time=source_timestamp`.+ - **Architect** `architect_agent.run_tooler(...)` (`auto_kg.py:125-133`) — receives the **entire** input text, not chunks.+ - Relationships are drained with `take_pending_relationships()` (`auto_kg.py:135`) and stamped with provenance (`:139-151`).+ - `should_consolidate = pipeline_mode == "accurate" and config.run_graph_consolidator and enrichment_relationships` (`auto_kg.py:153-157`).++4. **Scout** (`src/core/agents/scout_agent.py:333`):+ - Chunks with `_chunk_text(text, max_chars=6000)` (`:344`, `:346`), implemented in `src/utils/text_chunking.py:1-27`: greedy paragraph packing, hard character slice for oversized paragraphs (`text_chunking.py:20-23`), **zero overlap, no context carry-over**.+ - Each chunk goes through an independent `_run_chunk` call (`:360-371`). Chunks are merged on the key `(name.lower(), type.lower(), happened_at or "")` with **first-wins** semantics (`:372-379`).+ - If the LLM returns nothing parseable, returns an empty entity list silently (`:511-512`).+ - `src/core/chunking/text_chunker.py` is unreachable — the only reference to `text_chunker` anywhere in the repo is its own file header. It is worth noting what is being left unused: it wraps a compiled extension, `chunker_cpp.chunk_text_semantically`, with `max_chunk_size=2000`, `min_chunk_size=500`, and `min_coherence_threshold=0.3` (`text_chunker.py:21-51`), falling back to fixed slices on failure (`:65-74`). So the project already contains a coherence-aware semantic chunker, and the live path instead uses 6000-character paragraph packing with no overlap. Whatever its quality, the boundary problem in gap 6 is being solved by the worse of two implementations already present.++5. **Architect** in tooler mode drives `ArchitectAgentCreateRelationshipTool` (`src/core/agents/architect_agent.py:350-359`). Inside the tool (`src/core/agents/tools/architect_agent/ArchitectAgentCreateRelationshipTool.py`):+ - Entities are resolved by UUID or `TYPE:NAME`; unknown ones become new `ScoutEntity` objects.+ - **Janitor** is invoked only when `self.mode == "granular"` (`:383`), via `run_atomic_janitor` (`:400-409`).+ - Janitor-supplied `fixed_relationships` suppress the corresponding input relationship if the endpoints match or embeddings agree above `0.90` (`:478-513`, threshold carries a literal `TODO` at `:512`).+ - Everything surviving is queued (`:550-551`), and *then* `wrong_relationships` is checked and returned as `status: "ERROR"` to the LLM (`:553-573`).++6. **Persistence** — `process_architect_relationships` (`src/workers/tasks/ingestion.py:707`):+ - `_normalize_relationship_dates` (`:192-205`) resolves `DATE`-typed entity names and `happened_at`.+ - `_resolve_relationship_entities` (`:208-446`) does entity resolution (detailed below).+ - Relationship vectors are computed, then a **global** relationship vector search (k=10) looks for near-duplicates (`:818-831`); if the nearest is within `RELATIONSHIP_DEDUP_MAX_DISTANCE = 0.1` (`:86`, `:165-170`) **and** endpoints match in either direction (`:842-848`), the new relationship is dropped with a bare `continue` (`:854`).+ - Nodes and the edge are written (`:911-971`).+ - `_invalidate_superseded_relationships` runs unconditionally for every persisted edge (`:972-975`).+ - Item-level failures accumulate in `item_errors` and produce `partial_failed`/`failed` status plus a raised `RuntimeError` (`:990-1013`).++7. **Consolidation** — `consolidate_graph` (`src/core/layers/graph_consolidation/graph_consolidation.py:31`): batches of 20, Janitor produces free-form natural-language tasks (`:64-69`), `KGAgent.run_graph_consolidator_operator` executes each (`:84-88`), failures printed and skipped (`:89-94`).++### Identity and provenance++`src/core/saving/identity.py`:+- `stable_node_id(name, type, happened_at, supplied_uuid)` (`:13-25`) — a supplied UUID short-circuits (`:19-20`); otherwise `sha256(lower(name) | lower(type) [| normalized_date if type == event])`.+- `stable_relationship_id(tail, predicate, tip, flow_key)` (`:28-42`).+- `stamp_provenance` (`:85-109`) merges a **list** of `source_chunk_ids` and a single `source_timestamp`.++### Entity resolution, in detail++`_resolve_relationship_entities` (`src/workers/tasks/ingestion.py:208-446`), three cascading strategies per non-event entity (`:298-307`):++1. `_uuid_match` (`:241-248`).+2. `_exact_match` on name + type (`:250-258`).+3. `_vector_match` (`:260-295`): embed the name, search `k=5` in the `nodes` store, **require the entity type string to appear verbatim (lowercased) in the candidate's labels** (`:278`), require cosine ≥ `NODE_RESOLUTION_SIMILARITY = 0.9` (`:84`, `:286`), and **abstain entirely if the top two candidates are within 0.02** (`:290-291`).++Events are excluded from that path (`:303-304`) and get a second, neighbour-anchored pass (`:341-408`): candidate events must be shared by at least half the already-resolved anchors (`:384`, `:387`), must not have a conflicting normalized date (`:394`), and must clear `EVENT_RESOLUTION_NAME_SIMILARITY = 0.7` (`:85`, `:400`), abstaining on ambiguity (`:402-404`). This is a reasonable cross-document event-coreference heuristic and deserves credit.++Anything unresolved gets `stable_node_id(...)` (`:431-436`).++### The agent swarm, as it actually exists++| Agent | File | Status in the live path |+| --- | --- | --- |+| Scout | `scout_agent.py` (518 lines) | Active. |+| Architect | `architect_agent.py` (1518 lines) | Active via `run_tooler`. |+| Janitor | `janitor_agent.py` (735 lines) | Active in `granular` only. |+| Observations | `observations_agent.py` (47 lines) | Called at `ingestion.py:598`, always with `context=None`. Writes to the observation store, **not to the graph**. |+| Temporal | `temporal_agent.py` (9 lines) | Header comment only. Referenced nowhere. |+| Validator | `validator_agent.py` (9 lines) | Header comment only. Referenced nowhere. |++`ArchitectAgent.run_structured` (`architect_agent.py:932`) contains the Janitor retry loop but is used by the structured-triples path, not free text.++### ADR status++`docs/decisions/001-ingestion-structure.md` and `docs/decisions/002-structured-ingestion-specific-processing.md` are **empty stubs**: 001 has `Status: Accepted`, `Date: 2026-07-27`, and `--` under Context, Decision, Alternatives Considered, and Consequences. 002 is the same shape and shorter. I am not inferring their intent. `AGENTS.md`, which `CLAUDE.md` points at, is a zero-byte file.++---++## Guarantees and where they break++The stated guarantee: every fact in the source text becomes a correct, deduplicated, temporally-marked element of an append-only event-centric graph, and nothing enters the graph that the source does not support.++Ranked by impact on the two confirmed pain points (multi-hop first, temporal second).++### 1. Node identity is decided by exact surface string, which both fragments and catastrophically over-merges — BUG (multi-hop)++Identity is `sha256(lower(name) | lower(type))`. Everything about multi-hop depends on the middle node of a chain being *one* node. Reproduced against the real code:++```+Money/MONEY -> 8f6db1fc-a23b-400a-2d5b-423758246dfb+money/money -> 8f6db1fc-a23b-400a-2d5b-423758246dfb+Acme Inc./ORGANIZATION -> e224b2c3-80c0-08e5-f017-165758f7806d+Acme Inc/ORGANIZATION -> 47ac63a4-3a19-7210-20ac-96ff9d0d9b92+Acme Inc./ORG -> 37229514-298f-6027-53c5-bcd3bdabc774+EVENT WENT_TO 08/05/2023 -> eb91dafe-84d5-868d-d3e4-d9339241d609+EVENT Went 08/05/2023 -> ef241913-3b5d-e544-d468-cda44d9c9f0c+EVENT WENT_TO 8 May, 2023 -> 45ace7ed-4cb5-a54b-6be4-36f408bb5a35+```++Two distinct failures, both severe:++**Over-merge.** The Architect prompt's own worked examples instruct the model to emit placeholder nodes named after their type: `{"type": "UNIT", "name": "Friends"}` (`src/constants/prompts/architect_agent.py:31`) and `{"type": "MONEY", "name": "Money"}` (`:111`). Every monetary amount ever ingested therefore lands on **one shared node**. The value survives on the edge's `amount` field (`ingestion.py:950`), but the object node becomes a hub joining every funding round, salary, and price in the graph to every other. For a system whose second goal is surfacing non-obvious connections, this manufactures an unbounded supply of obvious-and-wrong ones. Any 2-hop traversal through `MONEY` is garbage.++**Fragmentation.** A trailing period, an abbreviation, or a type-label variation forks the node. `_vector_match` is the only defence, and it is gated on `entity_type not in labels` (`:278`) — an exact lowercase string comparison against a type vocabulary the LLM invents freely. `PERSON` vs `PEOPLE` vs `HUMAN` can never resolve to each other no matter how similar the names are. The type string is simultaneously load-bearing for identity and completely uncontrolled.++Compounding this, the alias mechanism intended to mitigate it is dead code. At `:315` the resolved entity's name is overwritten with the graph node's name; at `:330` the guard is `entity.name != node.name`, which is now always false. `aliases` is never populated.++### 1b. Ingestion never populates the `triplets` vector store that triple-level retrieval reads — BUG (multi-hop)++Ranked second overall, behind gap 1 only because gap 1 is upstream of it. Found late, while cross-checking a parallel trace.++Every `add_vectors` call in the repository, with its target store:++| Writer | Store |+| --- | --- |+| `src/workers/tasks/ingestion.py:611` (post-shift) | `"data"` |+| `src/core/saving/ingestion_manager.py:83-84` | `"nodes"` |+| `src/core/saving/ingestion_manager.py:129-130` | `"relationships"` |+| `src/core/agents/tools/kg_agent/KGAgentCreateNodeTool.py:105-106` | `"nodes"` |+| `src/core/agents/tools/kg_agent/KGAgentCreateRelationshipTool.py:163-164` | `"relationships"` |+| `src/core/agents/tools/kg_agent/KGAgentAddTripletsTool.py:137-138`, `:160-161` | `"nodes"` |+| **`src/core/agents/tools/kg_agent/KGAgentAddTripletsTool.py:180`** | **`"triplets"`** |++That last row is the only writer to `"triplets"` anywhere in `src/`, and `KGAgentAddTripletsTool` is an agent tool that is not on the ingestion path. Meanwhile the retrieval controller's triple-level entry point calls `vector_search.search_triplets(...)` (`src/services/api/controllers/retrieve.py:79`), which searches `store="triplets"` (`src/utils/vector_search.py:33-40`). The store is declared with its own dimension (`src/constants/embeddings.py:23`) and is searched by `KGAgentSearchGraphTool.py:115` as well.++**So no fact ingested through `/ingest/`, `/ingest/structured`, or `/ingest/file` is ever retrievable by triple-level vector search.** Single-hop lookups that enter through `"nodes"` still work, which is consistent with the maintainer's report in `00-scope-and-constraints.md` that single-hop is acceptable while chained questions fail. This is an ingestion defect with a retrieval-side symptom, and it may account for a meaningful share of the multi-hop gap on its own.++Two things I did not determine: whether `"triplets"` was intended to be populated by ingestion and the wiring was lost, or whether `retrieve.py:79` is reading the wrong store; and how much of the measured multi-hop deficit this accounts for. Both need the maintainer and `02-retrieval-multihop.md`. It is cheap to test either way — populate the store for one brain and re-run the multi-hop slice.++### 2. The Janitor cannot reject anything, and fails open silently — BUG (accuracy)++`README.md:163` draws the Janitor as the gate into the graph — `Janitor -->|"OK"| Graph` — and `README.md:170` promises it loops "until it's clean." The code does neither. Two independent defects compose badly.++**No veto.** In the live path, relationships are queued for persistence at `ArchitectAgentCreateRelationshipTool.py:550-551`, and only afterwards is `wrong_relationships` inspected and returned as an error to the LLM (`:553-573`). The LLM then produces a corrected relationship, which is *also* queued. Net result: **both the rejected edge and its replacement are written.** The only edges the Janitor can actually suppress are ones it rewrote itself into `fixed_relationships` with matching endpoints (`:478-513`). "This fact is wrong, do not write it" is not expressible — the output schema (`src/constants/prompts/janitor_agent.py:226-246`) has `required_new_nodes`, `fixed_relationships`, and `wrong_relationships` (which carries fix `instructions`), and no delete verdict.++The same inversion exists in `run_structured`: `relationships_to_persist.extend(pending_batch)` executes at `architect_agent.py:1254`, *before* `_persist_relationships` invokes the Janitor at `:1255`.++**Fail-open.** `run_atomic_janitor` returns the string `"OK"` on success (`janitor_agent.py:732-733`) and `structured_response` otherwise (`:735`) — which is `None` when the LLM produced nothing parseable. Callers use `getattr(janitor_response, "wrong_relationships", [])` (`architect_agent.py:927`, tool `:554`). `getattr` on `"OK"` and on `None` both yield `[]`. **A Janitor that failed to produce output is indistinguishable from a Janitor that approved everything**, and nothing records which happened.++**Convergence.** The loop terminates — `max_janitor_iterations = 3` (`architect_agent.py:1250`, `:1263`) with a `(tail, tip, name)` dedup guard at `:1305-1307`. But it does not converge in any useful sense: it accumulates edges, and on exhausting its budget it breaks and returns everything with no flag, no error, no marker (`:1263-1264`). "Looping until it's clean" (`README.md:170`) is three tries and then whatever we have.++Downstream, `retrieve.py:422-424` filters on `invalid_at` and `deprecated`, so these are the only two levers that can ever remove a written fact from an answer — and neither is under the Janitor's control.++### 3. Temporal marking is broken end to end, and demonstrably no-ops on the benchmark in use — BUG (temporal)++`src/utils/dates.py` accepts ten formats (`:5-16`), all requiring full day-month-year. Unparseable values pass through verbatim (`:77`). The canonical stored form is `%d/%m/%Y` (`:76`) — day-first, **not ISO 8601**, so no lexicographic ordering or range comparison is possible downstream. `resolve_relative_date` requires the reference timestamp itself to parse with those same ten formats (`:96-98`), returning the input unchanged otherwise.++LOCOMO supplies `source_timestamp` verbatim from the dataset (`benchmarks/locomo/dataset.py:131`, `:148`). The actual values look like `'1:56 pm on 8 May, 2023'`. Executed against the real module:++```+parse('1:56 pm on 8 May, 2023') -> None+'yesterday' -> 'yesterday'+'two weeks ago' -> 'two weeks ago'+'3 months ago' -> '3 months ago'+'2023-05-08T13:56:00Z' -> '2023-05-08T13:56:00Z'+```++**Every relative date in the LOCOMO run is left unresolved**, because the reference timestamp never parses. Even with a parseable reference, `3 months ago`, `March 2026`, `2019`, and ISO-8601 all fail. `valid_at` is then set to that same unparseable string (`ingestion.py:418-422`), and `invalid_at` likewise (`:499`), with a `%d/%m/%Y` fallback (`:500`) — so the graph carries a mixture of formats, none of them sortable.++Separately, there is **one timeline, not two.** `valid_at` is populated from `source_timestamp` — when the content was observed — with no extraction of when the fact became true. `happened_at` on events is the only fact-time signal and it is a free-form string that participates in the identity hash. The `temporal_agent.py` that would presumably own this is a 9-line stub.++### 4. Supersession blindly deprecates the event graph it is built on — BUG (temporal)++`_invalidate_superseded_relationships` (`ingestion.py:465-505`) matches existing edges from the same subject on **predicate name equality alone** (`:484`), skips same-tip (`:486`), and marks everything else `invalid_at` + `deprecated: True` (`:494-503`). It runs unconditionally for every persisted edge (`:972`).++There is no model of predicate cardinality. For a genuinely functional predicate this is the right rule. For BrainAPI's own event-hub vocabulary — where a subject has many `:MADE` edges to many event hubs by design — **every new event a subject participates in deprecates all of their prior events**. Since `retrieve.py` filters on `deprecated`/`invalid_at`, those events become invisible. This directly produces the maintainer's stated symptom in reverse: history vanishes, and it contradicts the append-only design in `README.md`.++Note the shape is right and the guard is missing: MemStrata (2606.26511) uses exactly a deterministic `(subject, relation, object)` supersession rule, but only for facts it has established are contradictory.++**Partially fixed while this document was being written.** An uncommitted change now in the working tree adds `_is_event_entity` and returns early when either endpoint is an event, and additionally requires the existing edge's direction to be outgoing. That closes the event-hub case described above — independent confirmation that this defect is real — and it shifts the line anchors in this section by roughly 19 lines. Two parts remain open: there is still no cardinality model, so a legitimately one-to-many predicate between two *non-event* nodes is still blindly deprecated; and `invalid_at` is still written in the unparseable form described in gap 3.++### 5. Duplicate detection cannot distinguish a duplicate from a contradiction — BUG (temporal + accuracy)++Relationship dedup is embedding-similarity plus an endpoint check: nearest neighbour within distance `0.1` and matching endpoints → `continue` (`ingestion.py:835-854`).++MemStrata (2606.26511) measured this directly: cosine similarity separates a *contradicted* fact from a *duplicated* one at **AUROC 0.59, near chance**, because contradictions are often more embedding-similar to the original than a rephrasing is. So this code path preferentially swallows the updates it most needs to act on. And the drop is a bare `continue` — no log, no `item_errors` entry, no record that a fact was discarded.++Two further problems in the same block: the endpoint check accepts **either direction** (`:842-848`), silently collapsing `A→B` and `B→A` for asymmetric predicates; and the candidate search is **global** (`:826-831`, `k=10`) rather than constrained to the subject-object pair. Graphiti constrains edge dedup to edges between the same entity pair specifically to avoid both problems and to shrink the search space (2501.13956, §2.2.2). Here, for a common predicate, all ten neighbours can belong to other entity pairs, so a genuine duplicate for *this* pair is never examined.++### 6. Extraction is schema-free, and the prompts disagree with each other and with the README — BUG (multi-hop)++`README.md` specifies the triangle of attribution with fixed predicates: `:MADE`, `:TARGETED`, `:OCCURRED_WITHIN`. The prompts do not implement that.++`src/constants/prompts/architect_agent.py` contains three incompatible policies:+- `:15-20` — the fixed three.+- `:226-231` — `:(MADE|COVERED_ROLE|EXPERIENCED|etc..)`, `:(TARGETED|RESULTED_IN|etc..)`, `:(OCCURRED_WITHIN|etc..)`. The `etc..` licenses open invention, and the worked example immediately below uses `MOVED`, `INTO_LOCATION`, `ACCOMPLISHED_ACTION`, `HAPPENED_WITHIN`, `EXPERIENCED` (`:261-310`) — `HAPPENED_WITHIN` for `OCCURRED_WITHIN`, `INTO_LOCATION` for `TARGETED`.+- `:365-370`, `:407-410` — fully open `--(predicate)-->`, constrained only by a style note at `:382`.++Event-hub naming is likewise inconsistent, and because event identity hashes the name (`identity.py:22-25`), naming convention *is* identity. The Architect prompt uses `WENT_TO`, `KNEW`, `PARTICIPATED_IN` (`:28-36`); the Scout prompt uses `Went`, `Was in`, `Partecipated in`, `Covered role` (`src/constants/prompts/scout_agent.py:49-74`, typo included). Two agents in the same pipeline are told to name the same event two different ways, and `EVENT_RESOLUTION_NAME_SIMILARITY = 0.7` has to bridge that on embeddings of two- and three-token strings.++Entity types are equally free (`MONEY`, `UNIT`, `ROLE`, `CITY`, `ORGANIZATION` appear in examples with no registry), while `_vector_match` requires exact type equality. Unconstrained vocabulary plus exact-match gating is the worst of both.++The consequence for multi-hop is direct: reliable Cypher patterns cannot be written against a predicate vocabulary that is invented per-ingestion.++### 7. The Janitor never checks whether a fact is in the source text — GAP (accuracy)++Every item in the Janitor's revision protocol (`src/constants/prompts/janitor_agent.py:211-218`) is structural: identity resolution, direction audit, stripping numbers from names, never merging events, label normalization, UUID formatting. `:224` instructs it to "Maintain original intent." Nothing asks whether the relationship is supported by `CONTEXT_TEXT`.++So a fluent, well-formed, correctly-directed hallucination passes validation cleanly. For a product whose first goal is "always-accurate answers," the validation layer has no faithfulness check at all.++### 8. `lightweight` mode produces a structurally different graph, not a cheaper one — TRADE-OFF, mispriced++`PIPELINE_MODE` defaults to `accurate` (`src/config.py:568-570`) and is process-global. In `lightweight`:++- Scout uses the coarse prompt, which puts facts into **properties of ordinary entities instead of event hubs**: `{"type": "CITY", "name": "New York City", "properties": {"friends_count": 12}}` (`scout_agent.py` prompt `:124`) and `{"type": "ORGANIZATION", "name": "Acme Inc.", "properties": {"funding_amount": 100000000, "funding_date": "19/01/2026"}}` (`:134`). No `EVENT` nodes.+- The Janitor never runs (`ArchitectAgentCreateRelationshipTool.py:383`).+- Observations never run (`ingestion.py:597`).+- Consolidation never runs (`auto_kg.py:153-154`).++This is not a speed/quality dial. Facts buried in property maps are invisible to relationship-level retrieval and to the entire triangle traversal. Because the setting is global and no node or edge records which mode produced it (only `TextChunk` carries `BRAIN_VERSION`, `ingestion.py:578`), flipping the env var permanently heterogenizes the graph with no way to tell the halves apart.++### 9. File ingestion discards all provenance and splits on page boundaries — BUG++`ingest_file` builds each page's payload as `{"data": {"data_type": "text", "text_data": markdown}, "brain_id": ...}` (`:1435-1440`). No filename, no page number, no `source_timestamp`, no `meta_keys`. Consequences:++- Nothing in the graph records which file or page a fact came from; provenance terminates at an opaque chunk id.+- `source_timestamp` is `None`, so `valid_at` is never set (`:418`) and Scout's `reference_time` is `None` (`auto_kg.py:121`) — **no relative date in any ingested document can ever be resolved.**+- Each page is an independent Celery task with its own Scout/Architect/Janitor run and no shared context (`:1420-1423`, `:1434-1446`). A sentence or fact spanning a page break is split across two isolated extractions.+- Pages run concurrently. Identical surface forms still converge because `stable_node_id` is deterministic, so this is not a duplicate-explosion risk — but *variant* surface forms fragment, and each page independently re-derives entity context from nothing.++### 10. Chunk isolation defeats cross-chunk coreference — BUG (multi-hop)++Scout's chunks are processed by independent stateless calls (`scout_agent.py:360-371`) against a chunker with zero overlap (`src/utils/text_chunking.py`). A pronoun or shortened reference in chunk 2 has no access to its antecedent in chunk 1. Merging is `(name, type, happened_at)` first-wins (`:372-379`), so a later chunk's richer description and properties are silently discarded rather than merged.++There is also an asymmetry worth noting: **Scout sees 6000-character chunks; the Architect and Janitor see the entire text** (`auto_kg.py:126`, tool `:403`). For long inputs the Architect prompt is unbounded, and the two agents are reasoning over different views of the same document.++### 11. Failure surfacing is inconsistent by layer — BUG++Persistence-level failures are handled well: collected into `item_errors`, reported as `partial_failed`/`failed` with counts, and raised (`ingestion.py:990-1013`). Everything above and around it is not:++| Silent path | Location |+| --- | --- |+| Scout produces nothing parseable → empty entity list | `scout_agent.py:511-512` |+| Janitor produces nothing parseable → treated as approval | `janitor_agent.py:735` + `getattr` callers |+| Janitor loop exhausts its budget with edges still wrong | `architect_agent.py:1263-1264` |+| Relationship dropped as a "duplicate" | `ingestion.py:854` |+| Self-relationship skipped | `ingestion.py:789-793` (print only) |+| Node embedding fails, node still written via the edge | `ingestion.py:900-909` |+| Consolidation task fails | `graph_consolidation.py:89-94` (print only) |++The node-embedding case deserves emphasis: on failure the node is dropped from `graph_nodes` (`:904`, `:909`) but `add_relationship` still runs at `:913` with that node as an endpoint. The node exists in the graph and **not** in the vector store, so it is permanently invisible to `_vector_match` — guaranteeing future duplicates of exactly the entity that already had trouble.++### 12. The graph is not append-only: consolidation can physically delete, driven by an LLM from prose — BUG++`consolidate_graph` has the Janitor emit free-form natural-language tasks (`graph_consolidation.py:64-69`) which `KGAgent.run_graph_consolidator_operator` executes (`:86-88`). The tool set that agent is handed (`src/core/agents/kg_agent.py:507-539`) is:++| Tool | `kg_agent.py` |+| --- | --- |+| `KGAgentExecuteGraphOperationTool` | `:507` |+| `KGAgentCreateNodeTool` | `:514` |+| `KGAgentCreateRelationshipTool` | `:521` |+| **`KGAgentRemoveNodeTool`** | `:527` |+| **`KGAgentRemoveRelationshipTool`** | `:533` |++So an LLM, acting on natural-language instructions produced by another LLM, holds arbitrary graph-operation execution plus **physical delete** on nodes and relationships. `README.md:163` labels the graph "append-only"; it is not. This runs by default — `accurate` is the default mode (`src/config.py:568-570`) and `RUN_GRAPH_CONSOLIDATOR` defaults to true (`:562-564`) — and failures are printed and skipped (`graph_consolidation.py:89-94`) with no record of which mutations were applied. There is no audit log, so a deletion is unattributable and unrecoverable.++This is the sharpest disagreement with the literature in this document. Zep explicitly rejected the weaker version of it: "We chose this approach [predefined Cypher queries] over LLM-generated database queries to ensure consistent schema formats and reduce the potential for hallucinations" (2501.13956, §2.2.1). Both Zep and Engram (2606.09900) invalidate and never delete, precisely so provenance and supersession chains survive.++Scope, to be fair to the design: the Janitor builds a 2-hop neighbourhood snapshot around the new relationships' endpoints, vector-filtered at `0.35` (`janitor_agent.py:268`, `:280`), and processes relationships in batches of 20 (`graph_consolidation.py:28`, `:45`). It is a local normalizer over recent writes, not an all-pairs pass — the blast radius is bounded, but within that radius it is unconstrained.++### 13. Documented behaviour that does not exist — GAP++`README.md:168` states the Observations agent writes notes "taking the previously known context into account." It is called with `context=None` (`ingestion.py:601`) and writes to the observation store, never to the graph. The Temporal and Validator agents are 9-line stubs referenced nowhere. Together with gap 2, three of the five documented swarm behaviours do not hold: the Janitor does not gate writes (`README.md:163`), it does not loop until clean (`:170`), and Observations has no prior context (`:168`).++### 14. Retry safety and session bookkeeping are backend-dependent and partly broken — BUG++Two smaller defects that share a cause — orchestration state that assumes a single clean pass:++**Source-chunk persistence is idempotent on one backend and not the other.** PostgreSQL writes the chunk with `ON CONFLICT (id) DO UPDATE` (`src/lib/postgresql/data.py:183`); Mongo uses a bare `insert_one` (`src/lib/mongo/client.py:48`). Celery is configured with `task_acks_late=True` and `task_reject_on_worker_lost=True`, so redelivery is a designed-for event, not an edge case. On Mongo, a worker lost mid-task therefore duplicates the source chunk, and since `source_chunk_ids` provenance is list-unioned at the graph layer (`src/lib/neo4j/client.py:271-276`), the duplicate propagates into provenance rather than being absorbed.++**The session pending-task counter only counts down.** `{brain_id}:session:{session_id}:pending_tasks` is read (`ingestion.py:1042`), decremented (`:1072`), and deleted (`:1388`) — and there is no `incr` or `incrby` anywhere in `src/`. The counter is never initialized or raised, so it descends into negative values from the first decrement. Any logic gating on it reaching zero is either dead or firing at the wrong time. I did not determine which, because it depends on Redis's behaviour for the read path when the key is absent, but no reachable code raises it.++### Assets already in the codebase and unused where they matter++Three capabilities the project has already built and does not apply on the ingestion path. This matters for sizing the recommendations below: several are less "build" than "wire up."++1. **LLM adjudication of entity identity.** The structured path resolves its anchor node with vector candidate generation followed by `verify_entity_existence(entity_name, entity_types, entity_meta_description, pool_nodes, brain_id)` (`src/core/agents/kg_agent.py:219`), called at `ingestion.py:1134-1142`. That is precisely the Graphiti entity-resolution pattern (2501.13956, §2.2.1). It is used for one node in one path, and never in the free-text path where nearly all data flows — including at the two abstention branches that most need it (gap 1, task 2.4).++2. **Linguistic analysis.** `src/utils/nlp/` provides NER, lemmas, POS tags, and noun chunks across 12 languages (`src/constants/spacy_models.py`, `src/utils/nlp/ner.py:39-146`). The only importer in the repository is the retrieval controller (`retrieve.py:44`, `:754`). So query strings get linguistic normalization and ingested entity names — the ones that must match them — get only `strip().lower()`. Any lemmatization or singularization in the canonicalization work of task 2.1 can reuse this rather than add a dependency.++3. **Semantic chunking.** The unreachable `chunker_cpp` wrapper described above, versus 6000-character packing in the live path (task 1.8).++### Deliberate trade-offs, correctly made++Not everything here is a defect, and these should not be "fixed":++- **Abstaining on ambiguity** (`ingestion.py:290-291`, `:402-404`) — failing toward duplication rather than a wrong merge is the right default for an accuracy-first system. It is the *absence of a follow-up* that is the gap, not the abstention.+- **Never merging EVENT nodes** (`janitor_agent.py:216`) — preserving distinct historical instances is correct for an event-centric model.+- **Lossless raw-text store** (`ingestion.py:574-592`) — matches Zep and Engram, and Engram measured that facts alone lose recall while facts plus retrieved chunks recover it (2606.09900).+- **Deterministic content-addressed IDs** (`identity.py`) — makes ingestion idempotent and concurrent page ingestion safe. The problem is the *inputs* to the hash, not the hashing.+- **Neighbour-anchored event resolution** (`ingestion.py:341-408`) — a genuinely thoughtful cross-document event coreference heuristic.+- **Bounded Janitor iterations** (`architect_agent.py:1250`) — guarantees termination.++---++## Open questions for the maintainer++1. Is the predicate vocabulary meant to be the fixed `:MADE` / `:TARGETED` / `:OCCURRED_WITHIN` triangle from `README.md`, or an open set that the Architect invents per ingestion?+2. Should an EVENT node's name be a normalized verb phrase (`WENT_TO`) or a sentence-case label (`Went`) — given that the choice determines the node's identity hash?+3. Is `_invalidate_superseded_relationships` intended to apply to event-hub predicates like `:MADE`, or only to functional attribute predicates?+4. What is the intended format of the `source_timestamp` API field, given that the current parser rejects both ISO-8601 and the LOCOMO dataset's own format?+5. Should `valid_at` mean "when we observed this" or "when this became true in the world" — and do you want both tracked separately?+6. When the Janitor concludes a relationship is simply wrong rather than fixable, should it be able to prevent the write entirely?+7. When the Janitor exhausts its three iterations with relationships still marked wrong, should the ingestion be reported as degraded rather than completed?+8. Is `lightweight` mode intended to produce a graph without event hubs, or is that an unintended consequence of the coarse Scout prompt?+9. Are `lightweight` and `accurate` ever mixed within one brain, and if so should each node and edge record which pipeline wrote it?+10. Should the placeholder nodes the prompts demonstrate (`MONEY`/"Money", `UNIT`/"Friends") be per-instance nodes, or are they deliberately intended as shared type-level hubs?+11. Should file ingestion preserve filename and page number as queryable provenance, or is the opaque chunk id considered sufficient?+12. Are `temporal_agent.py` and `validator_agent.py` planned work, or abandoned ideas that should be deleted?+13. Should the Observations agent's output ever reach the graph, or is the observation store a separate retrieval surface by design?+14. `src/core/chunking/text_chunker.py` wraps a compiled semantic chunker (`chunker_cpp`, coherence-thresholded) and is unreachable, while the live path uses fixed 6000-character paragraph packing. Was the C++ chunker abandoned for a reason — quality, build friction, the threading lock — or did the wiring simply get lost?+15. Was the `triplets` vector store meant to be populated by ingestion (gap 1b), or is `retrieve.py:79` reading the wrong store? Either way, was triple-level vector retrieval ever observed working on ingested data?+16. Consolidation currently holds physical delete on nodes and relationships (gap 12). Is destructive consolidation intended, or should it be restricted to invalidation like the rest of the write path?+17. What is the acceptable added write latency and token cost per ingested unit for accuracy improvements — is doubling write cost acceptable if it removes a class of error?++---++## Frontier techniques++### A. Ontology-grounded post-extraction correction++**Mechanism.** Keep extraction open-domain, then apply embedding-based canonicalization of types and predicates, then targeted LLM correction of only the ontology violations that remain. Deferring correction to a post-extraction stage avoids repeated LLM calls during extraction.++**arXiv.** 2605.29168 — *Better Later Than Sooner: Neuro-Symbolic Knowledge Graph Construction via Ontology-grounded Post-extraction Correction*.++**Reported gain.** Improved KG consistency and substantially reduced token usage versus constrain-at-extraction pipelines, while preserving downstream QA quality; validated by measuring SPARQL graph-pattern occurrence, i.e. by whether the resulting graph is actually queryable.++**Cost.** One canonicalization pass over new types/predicates plus a small number of correction calls. Cheaper than schema-constrained extraction, and it is write-time work, which `00-scope-and-constraints.md` explicitly prefers.++**Fit.** Very high, and it maps onto a component that already exists. The Janitor is already a post-extraction corrector; it is already told to "use `get_schema` (target: `relationship_types`) to normalize relationship labels" (`janitor_agent.py:217`). What is missing is a persisted canonical registry and an embedding-based merge of near-synonym labels, rather than ad-hoc per-batch alignment against whatever noise is already in the graph.++**Verdict. Adopt.** This is the single highest-leverage item, because it attacks the top-priority pain point (multi-hop) at its root: gap 1 and gap 6 are the same problem seen from two sides.++### B. Simultaneous schema induction with entity-and-event modelling++**Mechanism.** Extract triples and induce the schema in the same pass, modelling both entities and events, and use conceptualization to organize instances into semantic categories.++**arXiv.** 2505.23628 — *AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corpora*.++**Reported gain.** Schema induction reaches **92% semantic alignment with human-crafted schemas with zero manual intervention**; outperforms baselines on multi-hop QA. Demonstrated at 50M+ documents, 900M+ nodes.++**Cost.** Schema induction over a corpus; the resulting type system then needs storage and versioning.++**Fit.** High, and unusually good: AutoSchemaKG models events as first-class alongside entities, which most schema work does not, and BrainAPI is event-centric. It answers question 1 and 2 empirically rather than by decree — you do not have to choose between schema-free and hand-authored ontology.++**Verdict. Adapt.** Do not adopt the full web-scale pipeline. Use it to induce a type/predicate registry from the graph BrainAPI has already built, then feed that registry to technique A as the canonicalization target.++Supporting references for the same problem: 2607.21610 (*SCOPE and SCION*) is a benchmark and auditable reference pipeline for schema induction whose core target is **event types and within-event argument roles** — structurally the closest published work to the triangle of attribution, and useful for evaluating an induced registry. 2412.20942 (*Ontology-grounded Automatic KG Construction under Wikidata schema*) authors an ontology from generated competency questions and maps relations onto Wikidata, which is a cheaper route if interoperability matters more than fit.++### C. Entity resolution as a first-class stage, with triple reflection++**Mechanism.** Treat LLM-generated KGs as noisy by default and denoise explicitly: entity resolution to eliminate redundant entities, plus "triple reflection" to remove erroneous relations.++**arXiv.** 2510.14271 — *Less is More: Denoising Knowledge Graphs For Retrieval Augmented Generation* (DEG-RAG).++**Reported gain.** Drastically reduced graph size and consistently improved QA across diverse Graph-RAG variants. The paper also contributes what it describes as the first systematic evaluation of entity resolution for LLM-generated KGs, sweeping **blocking strategies, embedding choices, similarity metrics, and merging techniques**.++**Cost.** A resolution pass over the graph; the empirical sweep means the parameters need tuning rather than guessing.++**Fit.** Directly applicable, and the surface is larger than it first appeared. A full sweep of the tree turns up at least twelve independent hardcoded similarity constants governing identity and dedup decisions:++| Value | Location | Governs |+| --- | --- | --- |+| `0.9` | `ingestion.py:84`, `:286` | node resolution by vector |+| `0.7` | `ingestion.py:85`, `:400` | event resolution by name |+| `0.02` | `ingestion.py:290`, `:402` | ambiguity abstention band |+| `0.1` | `ingestion.py:86`, `:165` | relationship dedup max distance |+| `0.90` | `architect_agent.py:886`, `ArchitectAgentCreateRelationshipTool.py:511` | near-duplicate descriptions (carries the `TODO`) |+| `0.92` | `list_reduction.py:89` | `reduce_list` pairwise dedup default |+| `0.35` | `janitor_agent.py:280` | consolidation 2-hop snapshot filter |+| `0.75` | `names.py:66` | Levenshtein+label acceptance |+| `0.6` / `0.4` | `names.py:49` | Levenshtein vs label-Jaccard blend weights |+| `0.8` | `adapters/graph.py:71` | neighbour vector reduction |+| `0.5` | `retrieve.py:199`, `entity_sibilings.py:35` | retrieval-side neighbour filters |++Nothing ties these to each other and none has a recorded justification. Two additional details compound it: `src/utils/nlp/names.py` implements a *different* matching function (Levenshtein blended with label Jaccard, accepted at `0.75`) reached from `KGAgentAddTripletsTool.py:108-114`, so the codebase has two unrelated entity-matching algorithms with unrelated thresholds; and `list_reduction.py:57-65` falls back, when no embeddings client resolves, to an **8-dimensional character-sum hash** used as an embedding — under which nearly any two strings of similar character composition will exceed `0.92`. That fallback is currently only reachable from retrieval-side neighbour reduction, but it is a silent correctness cliff rather than a degradation.++This paper is the evidence base these numbers lack.++**Verdict. Adopt** the methodology for choosing thresholds and merge strategy. Do not adopt wholesale graph compaction — "less is more" pruning risks removing exactly the low-degree bridge nodes that multi-hop depends on, which is the opposite of this project's priority.++### D. LLM adjudication of ambiguous resolution candidates++**Mechanism.** Cheap candidate generation (embeddings + full-text), then an LLM decides duplicate/not-duplicate over the candidate set, and on a merge emits a canonical name and summary. Graphiti runs this per entity with the episode as context (2501.13956, §2.2.1). Cost-efficient variants only spend LLM calls where the decision is uncertain: 2401.03426 initializes candidate partitions, defines uncertainty, and selects the most valuable pairs to query, with error-tolerant handling of LLM mistakes. 2506.02509 goes further and has the LLM cluster records directly rather than comparing pairs.++**arXiv.** 2501.13956, 2401.03426, 2506.02509.++**Reported gain.** LLM-CER (2506.02509) reports up to **150% higher accuracy, +10% F-measure, and 5× fewer API calls** than pairwise baselines across nine real-world datasets. Zep attributes part of its DMR/LongMemEval results to this resolution design.++**Cost.** LLM calls at write time, bounded if gated on uncertainty.++**Fit.** Excellent, and nearly free to wire up: `KGAgent.verify_entity_existence` (`kg_agent.py:219`) already implements exactly this and is already called with a vector-search candidate pool at `ingestion.py:1116-1142`. The abstain-on-ambiguity branches (`:290-291`, `:402-404`) are precisely the uncertainty signal that 2401.03426 says should trigger the LLM — and today they instead silently create a duplicate.++**Verdict. Adopt.** Highest ratio of gain to implementation cost in this document: route the existing abstention branches into the existing adjudicator.++### E. Bi-temporal model with contradiction-triggered supersession++**Mechanism.** Track four timestamps: transaction-time created/expired, and valid-time valid/invalid. On a new edge, compare against semantically related existing edges; on a genuine temporally-overlapping contradiction, set the old edge's `t_invalid` to the new edge's `t_valid`. Invalidate, never delete, so provenance and the supersession chain survive (2501.13956, §2.2.3). Engram (2606.09900) runs the same model with **no LLM call per fact**. MemStrata (2606.26511) reduces it further to a deterministic `(subject, relation, object)` supersession rule in a bi-temporal ledger with no similarity threshold and no LLM call.++**arXiv.** 2606.26511, 2501.13956, 2606.09900.++**Reported gain.** MemStrata: ties RAG on static knowledge, reaches **0.95–1.00 accuracy on evolving knowledge where RAG reaches 0.20–0.47**, and drives the **stale-fact-error rate from 15–40% to ~0%**, at ~2.1s retrieval latency versus ~16–18s for LLM-reranking baselines. Engram: 83.6% vs 73.2% full-context on the full 500-question LongMemEval_S at ~8× fewer tokens. Zep: up to 18.5% accuracy improvement on LongMemEval with 90% lower latency.++**Crucially,** MemStrata quantifies why BrainAPI's current approach cannot work: cosine similarity distinguishes a contradicted fact from a duplicated one at **AUROC 0.59**, near chance, because contradictions are often *more* embedding-similar to the original than rephrasings are.++**Fit.** Exact. This is gaps 3, 4, and 5 with a published, cheap, benchmarked answer, and it targets the maintainer's confirmed #2 pain point ("superseded facts come back as current truth"). The stale-fact-error rate is also the metric this project currently lacks.++**Verdict. Adopt.** Note that the mechanisms are deliberately *not* LLM-heavy, which suits the write-time budget.++### F. Grounding verification via required source spans++**Mechanism.** Require the extractor to emit, alongside each structured item, the context it came from. Align that context against the document with a string-based global aligner, then score the alignment. Anything unaligned or low-scoring is flagged as unsafe.++**arXiv.** 2510.00276 — *SafePassage: High-Fidelity Information Extraction with Black Box LLMs*.++**Reported gain.** **Up to 85% reduction in hallucinations** with minimal risk of flagging non-hallucinations, and high agreement with human judgments of extraction quality — so the same pipeline doubles as an evaluator. Notably, a **transformer encoder fine-tuned on a small number of task-specific examples outperformed an LLM scoring model** at flagging unsafe passages, with annotations collectible in 1–2 hours.++**Cost.** Extra output tokens per extraction (the span), a cheap string alignment, and optionally a small encoder. No extra LLM call on the critical path if the encoder is used.++**Fit.** Very high. It fills gap 7 exactly: the Janitor validates structure and never grounding. BrainAPI already has the raw text stored (`ingestion.py:574-581`) and `source_chunk_ids` on every element, so the document side of the alignment is already available. And the "small encoder beats LLM scorer" finding matters given the write-time cost constraint.++**Verdict. Adopt.** This is the mechanism that makes gap 2's veto meaningful — a veto with no faithfulness signal has nothing to act on.++### G. Atomic decontextualized facts as the extraction unit++**Mechanism.** Split input into minimal, **self-contained** atomic facts before graph construction, build per-fact temporal graphs with dual-time modelling, then merge in parallel (2510.22590). SynthKG (2410.16597) reaches the same conclusion from the data-generation side: systematic chunking plus **decontextualization** plus structured extraction, then distilled into a single-step small model. Dense X Retrieval (2312.06648) independently establishes propositions — atomic, self-contained natural-language expressions of a single factoid — as a superior retrieval unit.++**arXiv.** 2510.22590 (ATOM), 2410.16597 (SynthKG / Distill-SynthKG), 2312.06648 (Dense X Retrieval).++**Reported gain.** ATOM: **~18% higher exhaustivity, ~33% better stability across runs, >90% latency reduction** versus baselines. Distill-SynthKG: a fine-tuned smaller model surpasses baselines up to eight times larger on KG quality, and improves downstream retrieval and QA. Dense X: proposition-level indexing significantly outperforms passage-level.++**Cost.** An extra decomposition pass per input, offset by parallelism; ATOM reports a large net latency *win*.++**Fit.** High. Decontextualization is the correct fix for gaps 9 and 10: instead of trying to preserve context across chunk and page boundaries, rewrite each unit to stand alone before extraction. ATOM's stability result is directly relevant — non-determinism across runs is invisible in the current design because nothing measures it.++**Verdict. Adapt.** Adopt decontextualization and dual-time extraction. Defer the distillation half (2410.16597) until the pipeline is stable enough to be worth distilling; it is a cost optimization, not an accuracy one, and fine-tuning now would freeze in the current defects.++### H. Multi-round extraction with reflection++**Mechanism.** After an initial extraction pass, run a reflection step to catch omissions and suppress hallucinations. Zep applies a Reflexion-inspired pass after entity extraction specifically "to minimize hallucinations and enhance extraction coverage," and supplies the previous `n = 4` messages as context for named entity recognition (2501.13956, §2.2.1).++**arXiv.** 2501.13956; GraphRAG's two-stage graph index (2404.16130) is the canonical reference for LLM-built graph indexes.++**Reported gain.** Not separately ablated in the Zep paper — I could not determine the isolated contribution of the reflection pass.++**Cost.** Roughly doubles extraction calls.++**Fit.** Moderate. Recall is a real problem (Scout is single-pass and silently returns empty on parse failure, `scout_agent.py:511-512`), and the `n = 4` prior-context window is a cheap partial answer to cross-chunk coreference that requires no decontextualization machinery. But without a measurement of omission rate, adding a pass is unfalsifiable.++**Verdict. Adapt, after measurement.** Establish an omission baseline first (technique J), then decide. Ranked below the others because the gain is unquantified in the source.++### I. Late chunking++**Mechanism.** Embed all tokens of the long text with a long-context embedding model, then apply chunk boundaries after the transformer and before mean pooling, so each chunk embedding carries whole-document context. No additional training required.++**arXiv.** 2409.04701 — *Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models*.++**Reported gain.** Superior results across retrieval tasks versus independently-encoded chunks.++**Cost.** Requires a long-context embedding model; changes the embedding pipeline.++**Fit.** Low **for this workstream**. Late chunking improves chunk *embeddings*; BrainAPI's chunking problem is that the *LLM extraction call* lacks cross-chunk context (`scout_agent.py:360-371`), which a better pooled vector does not address. It is potentially relevant to the `"data"` text-chunk store used at retrieval time.++**Verdict. Reject for ingestion; refer to `02-retrieval-multihop.md`.** Rejected because the assumption it addresses — that the retrieval unit is an embedding — does not hold for the extraction path, where the unit is an LLM prompt.++### J. Extraction evaluation that does not depend on exact match++**Mechanism.** Evaluate generative extraction on multiple dimensions — topic similarity, uniqueness, granularity, factualness, completeness — rather than precision/recall against human-annotated reference triples (2402.10744). Complementarily, evaluate KG construction with explicit hallucination and omission metrics plus BERTScore-based graph similarity (2502.05239).++**arXiv.** 2402.10744 (GenRES), 2502.05239.++**Reported gain.** GenRES establishes that precision/recall fails to justify generative extraction performance, that human-annotated reference relations are frequently incomplete, and — importantly for technique A — that **prompting LLMs with a fixed set of relations or entities can itself cause hallucinations**. Its scores align with human preference judgments.++**Cost.** Harness work only.++**Fit.** Essential and unavoidable. Every other proposal here needs a measurement to be a task rather than a hope, and BrainAPI's current schema-free extraction has no reference triples to match against by construction.++**Verdict. Adopt first.** This is a prerequisite, not an improvement.++The GenRES finding is also the most important tension in this literature and should temper technique A: naive schema constraining at extraction time *increases* hallucination. That is precisely why 2605.29168's deferred post-extraction correction is the right architecture — canonicalize after open extraction, not before.++### K. Rejected, with reasons++- **Hyper-relational / n-ary KG embedding** (2305.18256, 2308.06512, 2306.02199, 2404.09848, 2411.06191, 2508.03280, 2505.11803, 2307.10219, 2305.06588, 2411.07019, 2312.09219). I searched this area specifically because BrainAPI's event hub is a reification of an n-ary fact. Almost all of it is representation learning for link prediction over a *given* graph — it assumes the facts already exist and asks how to embed them. It says nothing about how to extract or canonicalize, so it is out of scope for ingestion. **Rejected: assumes the graph as input.**+- **Event-centric attribution / viewpoint modelling** (2503.03563). Parameterized predicates for facts valid only within a stated viewpoint, for controversial events with multiple contested accounts. Genuinely relevant to a problem BrainAPI has no answer for — two sources asserting incompatible facts — but it presumes a conflict-detection layer that does not exist yet, and BrainAPI's current conflict handling (technique E) must land first. **Defer, revisit after E.**+- **Fine-tuned / distilled extraction models** (2410.16597 distillation half, and the fine-tuning result in 2502.05239). 2502.05239 found fine-tuned models improved accuracy and reduced hallucination and omission **but generalized worse** on a held-out dataset. Distilling the current pipeline would bake in the defects catalogued above. **Rejected for now: premature.**+- **Community detection / summarization at write time** (2404.16130, 2410.05779, and Zep's label-propagation variant). Real write-time work with real retrieval benefit, but it is graph-structure enrichment rather than extraction quality, and it operates on whatever the extractor produced. Building communities over a graph with a `MONEY` super-node and a fragmented type system would summarize the noise. **Defer: fix identity first; owned by `03-memory-substrate.md`.**+- **Memory-architecture comparators** — MemGPT (2310.08560), HippoRAG (2405.14831), A-MEM (2502.12110), Mem0 (2504.19413). Read for comparison, not adoption. HippoRAG's contribution is Personalized-PageRank retrieval, not writing. A-MEM's "memory evolution," where a new memory triggers updates to the attributes of existing ones, is the closest conceptual match to what BrainAPI's append-only model forbids, and is worth revisiting for `05-temporal-truth.md`. Mem0 is the most directly comparable published number, since it evaluates on **LOCOMO** — the same benchmark in `benchmarks/locomo/` — across single-hop, temporal, multi-hop, and open-domain categories, and reports a graph-memory variant scoring ~2% above its base configuration. **Use as baselines, not as designs.**++---++## Implementation plan++Sizing: **S** = 1–2 files, **M** = 3–5 files. Every task has a verification command. Phases are ordered by dependency; each checkpoint is a go/no-go.++### Phase 0 — Make the pipeline measurable++Nothing below can be judged without this. No production code changes.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 0.1 | Extraction-quality harness: for a fixed corpus, report per-input hallucination rate (triples with no supporting span), omission rate, and predicate/type cardinality, following 2502.05239 and 2402.10744 | M | Runs on ≥50 LOCOMO units; emits JSON with all four metrics; two runs on identical input report a stability score | `python -m benchmarks.extraction_quality --units 50 --out runs/eq-baseline.json` |+| 0.2 | Graph-health report: count of nodes sharing a name with their own type, top-20 nodes by degree, type-vocabulary size, predicate-vocabulary size, count of predicate pairs with cosine > 0.9 | S | Identifies the `MONEY`/"Money" class of super-node without being told to look for it | `python -m scripts.graph_health --brain-id <id>` |+| 0.3 | Stale-fact-error probe, per 2606.26511: ingest a fact, ingest its update, query, and record how often the superseded value is returned | S | Produces a single stale-fact-error-rate number | `python -m benchmarks.stale_facts --brain-id <id>` |+| 0.4 | Date-coverage report over every distinct `happened_at`, `valid_at`, and `invalid_at` string in the graph, bucketed into parsed / passed-through-verbatim | S | Reports the percentage of temporal values that are unparseable | `python -m scripts.date_coverage --brain-id <id>` |++**Checkpoint 0.** Baselines recorded and reviewed. Expected from the code read: high type/predicate cardinality, at least one type-named super-node, ~100% unparseable temporal values on LOCOMO. If those do not appear, the analysis above is wrong and the plan must be revisited before any change ships.++### Phase 1 — Stop the bleeding (small, independent, high-confidence)++Each of these is a localized fix for a defect established above. None depends on the others.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 1.1 | Extend `src/utils/dates.py`: accept ISO-8601 with time, year-only, month-year, and the LOCOMO `'H:MM am/pm on D Month, YYYY'` form; store canonically as ISO-8601; extend relative patterns to word numerals and months | S | `resolve_relative_date('two weeks ago', '1:56 pm on 8 May, 2023')` returns an absolute date; existing `%d/%m/%Y` inputs still parse; 0.4 shows unparseable share below 5% | `pytest tests/test_dates.py -v && python -m scripts.date_coverage --brain-id <id>` |+| 1.2 | Gate `_invalidate_superseded_relationships` on an explicit allowlist of functional predicates. The event-hub exemption already landed in the working tree; what remains is the cardinality model for non-event predicates | S | A subject with three `:MADE` edges to three event hubs retains all three as valid (already satisfied); a legitimately one-to-many non-event predicate also retains all edges; a functional attribute predicate still supersedes | `pytest tests/test_event_hub_invalidation.py -v` |+| 1.3 | Distinguish Janitor failure from Janitor approval: return an explicit outcome type instead of `"OK"` / `None`, and record unvalidated batches in `item_errors` | S | A Janitor whose response fails to parse yields a `partial_failed` task status; a genuine approval does not | `pytest tests/test_janitor_outcomes.py -v` |+| 1.4 | Make silent drops observable: log and record `item_errors` entries for dedup drops (`ingestion.py:854`), self-relationship skips (`:789`), node-embedding failures (`:900-909`), and consolidation task failures (`graph_consolidation.py:89`) | S | Every discarded fact appears in the task record with a reason; counts reconcile against relationships submitted | `pytest tests/test_ingestion_accounting.py -v` |+| 1.5 | Do not write a node whose embedding failed as a bare relationship endpoint; either retry the embedding or fail the item | S | No node exists in the graph without a corresponding vector-store entry | `python -m scripts.graph_health --check orphan-vectors --brain-id <id>` |+| 1.6 | Thread `filename`, page number, and `source_timestamp` through `ingest_file` into each page's `ingest_data` payload | S | A fact extracted from page 7 of `report.pdf` reports both; relative dates in documents resolve | `pytest tests/test_ingest_file_provenance.py -v` |+| 1.7 | Fix the dead alias branch at `ingestion.py:329-336` by capturing the incoming surface form before `:315` overwrites it | S | Resolving "Acme Inc" onto "Acme Inc." leaves "Acme Inc" in `aliases` | `pytest tests/test_entity_resolution.py -k alias -v` |+| 1.8 | Resolve the two-chunker split: either delete `src/core/chunking/text_chunker.py` or bench its C++ semantic chunker against the live 6000-char packer and keep the winner — pending open question 14 | S | Exactly one chunking implementation is reachable from the ingestion path, and the choice is recorded with a number behind it | `rg -n "chunk_text\|TextChunker" src/` |+| 1.9 | Resolve the `triplets` store mismatch (gap 1b): either populate it during relationship persistence alongside `"relationships"`, or point `retrieve.py:79` at the store ingestion actually writes — pending open question 15. **Do 1.9 first and measure it alone**, since it may move multi-hop more than the rest of Phase 1 combined | S | A fact ingested via `/ingest/` is returned by triple-level vector search; the multi-hop slice from 0.1 is re-measured against this change in isolation | `pytest tests/test_triplet_store_population.py -v && python -m benchmarks.locomo.cli run --slice multihop` |+| 1.10 | Give Mongo's `save_text_chunk` (`src/lib/mongo/client.py:48`) upsert semantics matching PostgreSQL's `ON CONFLICT (id) DO UPDATE`, so late-acked task redelivery does not duplicate source chunks | S | Ingesting the same chunk id twice yields one stored chunk on both backends; a simulated worker loss and redelivery produces no duplicate | `pytest tests/test_chunk_idempotency.py -v` |+| 1.11 | Fix or remove the session pending-task counter (gap 14): it is decremented and deleted but never incremented anywhere in `src/` | S | The counter reflects outstanding tasks and reaches zero exactly once per session, or the dead gate is removed outright | `rg -n "pending_tasks" src/ && pytest tests/test_session_lifecycle.py -v` |++**Checkpoint 1.** Re-run all of Phase 0. Stale-fact-error rate and unparseable-temporal share should both drop sharply from 1.1 and 1.2 alone, and multi-hop should move on 1.9 — attribute it before proceeding, because a large jump there changes the priority of Phase 2. Hallucination and cardinality metrics should be unchanged; if they move, something in Phase 1 had unintended reach.++### Phase 2 — Canonicalization (the multi-hop fix)++Depends on Checkpoint 1 and on 0.2 for a target to move.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 2.1 | Induce a type and predicate registry from the existing graph per 2505.23628, with embedding-clustered canonical forms and recorded surface variants | M | Registry covers ≥95% of existing type and predicate occurrences; every cluster is human-reviewable | `python -m scripts.induce_registry --brain-id <id> --out registry.json` |+| 2.2 | Canonicalize types and predicates at write time against the registry, after open extraction, per 2605.29168 | M | Predicate cardinality drops by ≥50% with no loss in 0.1 completeness; `PERSON`/`PEOPLE` collapse to one type | `python -m benchmarks.extraction_quality --units 50 --out runs/eq-canon.json && python -m scripts.graph_health --brain-id <id>` |+| 2.3 | Forbid type-named placeholder nodes: reject or per-instance-qualify any node whose name matches its type, and correct the prompt examples at `architect_agent.py:31` and `:111` | S | 0.2 reports zero type-named nodes; monetary amounts no longer share a node | `python -m scripts.graph_health --check type-named --brain-id <id>` |+| 2.4 | Route the abstain-on-ambiguity branches (`ingestion.py:290-291`, `:402-404`) into the existing `KGAgent.verify_entity_existence` adjudicator, per 2401.03426 and 2501.13956 | M | Ambiguous pairs are decided rather than duplicated; duplicate-node count falls; LLM calls fire only on abstention | `pytest tests/test_entity_resolution.py -v && python -m scripts.graph_health --brain-id <id>` |+| 2.5 | Constrain relationship dedup search to the subject-object pair and stop treating direction-reversed edges as duplicates, per 2501.13956 §2.2.2 | S | A duplicate for a given pair is found regardless of how many unrelated similar edges exist; `A→B` and `B→A` remain distinct | `pytest tests/test_relationship_dedup.py -v` |+| 2.6 | Choose thresholds by sweep rather than assertion, following 2510.14271: blocking strategy, embedding, similarity metric, merge rule. Collect the twelve constants inventoried in technique C into one configuration surface first | M | Every identity/dedup threshold is swept with a recorded precision/recall curve and read from one place; the `TODO` at `ArchitectAgentCreateRelationshipTool.py:512` is resolved | `python -m benchmarks.er_sweep --out runs/er-sweep.json` |+| 2.8 | Collapse the two entity-matching algorithms into one: either route `KGAgentAddTripletsTool`'s Levenshtein+Jaccard path (`names.py:47-66`) through the vector resolver, or state why triplet-tool matching differs | S | One matching function governs entity identity; `most_similar_name_with_labels_or_none` is either the single path or removed | `rg -n "most_similar_name_with_labels_or_none" src/` |+| 2.9 | Make `list_reduction.py`'s missing-embeddings path fail loudly instead of substituting the 8-dimensional character-sum hash (`:57-65`) | S | Absent an embeddings client, reduction raises rather than returning near-random similarity | `pytest tests/test_list_reduction.py -k fallback -v` |+| 2.7 | Reconcile the three predicate policies and two event-naming conventions in `src/constants/prompts/` into one, per answers to open questions 1 and 2 | M | One vocabulary statement across Scout, Architect, and Janitor prompts; the triangle in `README.md` matches the prompts | `pytest tests/test_prompt_consistency.py -v` |++**Checkpoint 2.** Multi-hop is the top-priority pain point, so this is the decisive gate. Requires: measurable drop in type and predicate cardinality, zero type-named nodes, reduced duplicate-node count, and **no regression in 0.1 completeness** — canonicalization must not be silently discarding facts. Hand off to `02-retrieval-multihop.md` for the read-side multi-hop measurement.++### Phase 3 — Grounding and the Janitor veto++Depends on Checkpoint 2, because a veto over a fragmented graph would reject correct facts for the wrong reason.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 3.1 | Require every extracted relationship to carry the source span it came from | M | Every new edge has a non-empty span; 0.1 hallucination rate becomes directly computable rather than estimated | `pytest tests/test_source_spans.py -v` |+| 3.2 | Add a string-based global aligner scoring each span against the stored `TextChunk`, per 2510.00276 | M | Fabricated spans score below unaligned threshold; genuine paraphrases are not flagged; false-positive rate on a labelled sample under 5% | `python -m benchmarks.grounding --units 50 --out runs/grounding.json` |+| 3.3 | Give the Janitor a delete verdict and honour it: move the queue-for-persistence call in `ArchitectAgentCreateRelationshipTool.py` to **after** the `wrong_relationships` check, and fix the same inversion at `architect_agent.py:1254` | M | A relationship the Janitor rejects never appears in the graph; the corrected version does; both-versions-written no longer occurs | `pytest tests/test_janitor_veto.py -v` |+| 3.4 | Report non-convergence: when the Janitor loop exhausts `max_janitor_iterations` with edges still wrong, mark the ingestion degraded and name the edges | S | Task status distinguishes clean completion from budget exhaustion | `pytest tests/test_janitor_convergence.py -v` |+| 3.5 | Optional, per 2510.00276: replace the LLM span scorer with a small fine-tuned encoder on 1–2 hours of annotations | S | Matches or beats the LLM scorer at lower latency; skip if 3.2 already meets the write-latency budget | `python -m benchmarks.grounding --scorer encoder --out runs/grounding-enc.json` |++**Checkpoint 3.** Hallucination rate from 0.1 must fall substantially — SafePassage reports up to 85% reduction as the achievable ceiling. Completeness must not regress: a veto that improves precision by discarding facts has failed.++### Phase 4 — Bi-temporal writes++Depends on Checkpoint 1 (1.1 and 1.2 specifically). Can proceed in parallel with Phases 2–3. Coordinate the read side with `05-temporal-truth.md`.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 4.1 | Separate transaction time from valid time on edges: `created_at`/`expired_at` and `valid_at`/`invalid_at`, per 2501.13956 §2.2.3 and 2606.09900 | M | Every new edge carries all four where determinable; observation time is never conflated with fact time | `pytest tests/test_bitemporal.py -v` |+| 4.2 | Extract fact-time explicitly against a reference timestamp, in ISO-8601, only when the text establishes or changes the relationship — the Graphiti temporal-extraction contract | M | "started my job two weeks ago" yields an absolute `valid_at` distinct from `created_at`; dates unrelated to the relationship are not attached | `pytest tests/test_temporal_extraction.py -v` |+| 4.3 | Replace similarity-based supersession with a deterministic `(subject, predicate, object)` rule over the canonical predicate registry, per 2606.26511 | M | Stale-fact-error rate from 0.3 approaches zero; no similarity threshold on the supersession path | `python -m benchmarks.stale_facts --brain-id <id>` |+| 4.4 | Set the superseded edge's `invalid_at` to the superseding edge's `valid_at` and record a supersession chain, so history is queryable rather than merely hidden | S | An "as-of" query returns the value valid at that time; the full chain is traversable | `pytest tests/test_supersession_chain.py -v` |+| 4.5 | Decide whether the 9-line `temporal_agent.py` becomes the home for 4.2 or is deleted, per open question 12 | S | No stub files referenced nowhere remain under `src/core/agents/` | `rg -n "temporal_agent\|validator_agent" src/` |++**Checkpoint 4.** Stale-fact-error rate near zero, and Phase 0 completeness and hallucination metrics unmoved. This closes the maintainer's confirmed #2 pain point on the write side.++### Phase 5 — Chunk-boundary integrity++Depends on Checkpoint 3, since 0.1's completeness metric is the only way to detect whether decontextualization is adding facts or inventing them.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 5.1 | Decontextualize each unit before extraction so it stands alone, per 2510.22590 and 2410.16597 | M | Pronouns and shortened references are resolved before Scout sees them; 0.1 completeness rises on multi-chunk inputs; stability score improves | `python -m benchmarks.extraction_quality --units 50 --multichunk --out runs/eq-decon.json` |+| 5.2 | Supply prior-unit context to Scout (Graphiti uses `n = 4`) and merge rather than discard richer duplicates at `scout_agent.py:372-379` | S | An entity first named in chunk 1 and referenced in chunk 2 lands on one node with merged description | `pytest tests/test_scout_chunking.py -v` |+| 5.3 | Resolve the Scout-chunks-versus-Architect-whole-text asymmetry (`auto_kg.py:126`) so both agents see the same view, with a bounded Architect prompt | M | A 100k-character input completes without prompt overflow; extraction is stable across runs | `python -m benchmarks.extraction_quality --units 5 --long-input --out runs/eq-long.json` |+| 5.4 | Give file ingestion overlap or shared context across page boundaries so a fact spanning pages 3–4 is extracted once | M | A deliberately page-split fact is captured; no duplicate hub for it | `pytest tests/test_ingest_file_boundaries.py -v` |++**Checkpoint 5.** Completeness up on multi-chunk and multi-page inputs, hallucination rate flat, stability improved toward ATOM's reported ~33%.++### Phase 6 — Make consolidation and mode-splitting safe++Lowest priority, but both are latent hazards.++| # | Task | Size | Acceptance criteria | Verification |+| --- | --- | --- | --- | --- |+| 6.1 | Replace free-form LLM graph mutations in `graph_consolidation.py:84-88` with parameterized operations from a fixed set, per 2501.13956 §2.2.1, and remove `KGAgentRemoveNodeTool` / `KGAgentRemoveRelationshipTool` from the consolidator's tool list (`kg_agent.py:527`, `:533`) in favour of invalidation | M | Every consolidation action is one of a known, audited set; nothing on the consolidation path issues a physical delete; each application is logged and reversible; failures surface rather than print | `pytest tests/test_consolidation_ops.py -v` |+| 6.1a | Interim mitigation, do this in Phase 1 if 6.1 will not land soon: log every consolidation mutation with its originating task string before applying it, so the deletions described in gap 12 are at least attributable | S | Every applied consolidation mutation is recoverable from logs, including deletes | `pytest tests/test_consolidation_audit.py -v` |+| 6.2 | Record on every node and edge which pipeline mode wrote it, pending open question 9 | S | A mixed-mode brain can be partitioned by writer; `graph_health` reports the split | `python -m scripts.graph_health --check pipeline-mode --brain-id <id>` |+| 6.3 | Resolve the `lightweight` structural divergence, pending open question 8: either make coarse mode emit event hubs, or document it as a separate incompatible product | M | Either both modes produce traversable event hubs, or the incompatibility is explicit in an ADR and enforced per brain | `pytest tests/test_pipeline_modes.py -v` |+| 6.4 | Fill ADRs 001 and 002, and decide the fate of the empty `AGENTS.md` | S | Neither ADR contains `--` placeholders | `rg -n '^\-\-$' docs/decisions/` |++**Checkpoint 6.** Full Phase 0 suite plus the retrieval-side metrics from `02-retrieval-multihop.md`, as input to the cross-workstream roadmap.++---++## Risks++**Canonicalization over-merges and destroys multi-hop precision.** Collapsing `PERSON`/`PEOPLE` is right; collapsing two genuinely distinct predicates is not, and it silently fuses unrelated subgraphs. *Detect:* 0.2's top-degree node list should not grow a new super-node after Phase 2; 0.1's completeness must hold; require human review of every merge cluster above a size threshold before it is applied.++**The Janitor veto suppresses correct facts.** Once the Janitor can delete, its false-positive rate becomes a data-loss rate, and there is no record of what was rejected. *Detect:* completeness at Checkpoint 3 must not regress; log every veto with its reason and span score, and sample them manually before enabling enforcement. Note that unaudited LLM-driven deletion is not a risk this plan introduces — per gap 12 it exists today on the consolidation path, and task 6.1a is the cheap way to make it visible in the meantime.++**Grounding verification rejects valid inference.** Not every true fact is a verbatim span — "John moved to New York" may be legitimately inferred across two sentences. A strict aligner flags it. *Detect:* 3.2's acceptance criterion is explicitly a false-positive rate on a labelled sample, not just a hallucination reduction. SafePassage reports "minimal risk of flagging non-hallucinations," which is a claim to verify locally, not assume.++**Fixing supersession un-hides facts that were wrong to begin with.** Task 1.2 will resurrect edges currently marked `deprecated`. Some were deprecated by the blind rule, but some may be genuinely superseded. *Detect:* count and sample the resurrected set before and after; do not backfill blindly, and treat the existing `deprecated` flag as untrustworthy rather than as ground truth.++**Date-format migration corrupts existing values.** Moving from `%d/%m/%Y` to ISO-8601 means reinterpreting stored strings, and `01/02/2026` is ambiguous without knowing which writer produced it. *Detect:* 0.4 before and after; migrate to a new field rather than rewriting in place, and keep the original string.++**Decontextualization introduces hallucinations.** Rewriting a chunk to be self-contained means an LLM inventing the missing context — exactly the failure mode being fixed elsewhere. *Detect:* 5.1's acceptance pairs completeness gain with a flat hallucination rate; if hallucination rises, the rewriting is fabricating antecedents.++**Write latency and cost exceed what the product can bear.** Phases 2–5 add LLM calls to the write path. Open question 15 exists because I could not determine the budget. *Detect:* record tokens and wall-clock per ingested unit at every checkpoint; ATOM (2510.22590) and Engram (2606.09900) both report large *reductions*, so a large increase is a signal that the design has drifted from the papers.++**Phase 0 metrics are themselves wrong.** Every gate here depends on the harness. A completeness metric that misses a class of fact will approve a change that discards it. *Detect:* validate 0.1 against a small hand-annotated set before trusting it as a gate, and follow 2402.10744's finding that human-annotated reference relations are themselves frequently incomplete.++**Concurrency assumptions break under the new resolution path.** Today, deterministic `stable_node_id` makes concurrent page ingestion safe for identical surface forms. Adding LLM adjudication (2.4) introduces a non-deterministic resolution decision into a path where multiple workers race on the same entity. *Detect:* ingest a document whose pages repeatedly mention the same entity under varying surface forms, concurrently, and assert a single resulting node.++**These changes are invisible to the current benchmark.** LOCOMO's temporal and multi-hop categories will move, but nothing in `benchmarks/` currently measures write-side quality, so a regression in extraction can be masked by a retrieval gain. *Detect:* report Phase 0 metrics alongside every benchmark run, and treat `04-evaluation-and-applications.md` failure-attribution work as a hard dependency for interpreting any end-to-end number.diff --git a/docs/research/02-retrieval-multihop.md b/docs/research/02-retrieval-multihop.mdnew file mode 100644index 0000000..c0b7f4d--- /dev/null+++ b/docs/research/02-retrieval-multihop.md@@ -0,0 +1,614 @@+# 02 — Query-time retrieval, ranking and multi-hop reasoning++Workstream owner: research analyst (this document). Scope: `src/core/search/`, the `/retrieve`, `/entity`, `/kg` API surface, and the agentic navigation surface (`src/core/agents/kg_agent.py`, `src/core/agents/tools/kg_agent/`, `src/services/mcp/`). Out of scope: extraction/ingestion quality, storage internals, the benchmark harness itself, and the pgvector HNSW dimension question (assigned to the roadmap per `00-scope-and-constraints.md:40`).++Revision 2. Judged against `docs/research/00-scope-and-constraints.md`, which is binding. Three constraints from that document drive most of what changed since revision 1:++- **`/retrieve/context` must be sub-second, fast and cheap** (`00:22`). No iterative LLM loops, no per-query LLM reranking, no multi-call decomposition on that path. This answers what were previously my open questions 4 and 5, and it invalidates part of my earlier plan.+- **The benchmark cannot resolve judge-accuracy differences below roughly 5-7 points** (`00:44-51`). Retrieval-side metrics are the trustworthy signal.+- **Event-leg supersession was a bug, is fixed, and existing brains were not repaired** (`00:39`). Every benchmark run analysed here was measured on a graph with wrongly-deprecated actor edges. I treat that as a confound, not a proposal.++Status of design intent: **`docs/decisions/003-simple-one-shot-context-api-retrieval.md` and `docs/decisions/004-agentic-navigation-retrieval.md` are still empty stubs.** Per `00:18` the maintainer's intent is a router rather than a winner, which matches the recommendation below.++All measurements come from the maintainer's LoCoMo runs in `benchmarks/runs/` (conv-26, n=152 non-adversarial, `deepseek-v4-flash` answering and judging at `temperature=0`). I ran no new benchmark runs. I did compute new statistics over the existing `answers.jsonl` and `report.json` files; every such number is labelled and reproducible from those files.++---++## What this workstream does++### 1. There are two retrieval surfaces, and only one of them can actually multi-hop++**Surface A — one-shot context API.** `POST /retrieve/context` → `get_context` (`src/services/api/controllers/retrieve.py:749-1065`). Single round trip, no LLM in the retrieval loop, returns a flat context blob. This is what the benchmark exercises and the surface with the sub-second budget.++**Surface B — agentic MCP tools.** `src/services/mcp/main.py:168-280` exposes `search_semantically`, `traverse_graph` (`main.py:231-260`), and `search_memory` (raw read-only Cypher/SQL, `main.py:272-280`). `traverse_graph` reaches `GraphAdapter.traverse_graph` (`src/adapters/graph.py:733-778`), which clamps to `MAX_TRAVERSE_DEPTH = 5` and `MAX_TRAVERSE_HOPS = 100` (`src/core/search/traverse.py:7-8`, clamped at `graph.py:746-747`).++These surfaces share no code, ranking, or bounds. **`traverse_graph` is not reachable from the REST API** — I searched repo-wide and only `src/services/mcp/` and tests call it. The depth-5 capability exists but only for MCP clients, and the depth-2 path is the only one ever benchmarked.++`src/core/agents/chat_agent.py` and `src/core/search/context.py` are empty. `KGAgent.search_kg` is a stub. So "agentic navigation" today is a tool surface plus a prompt (`src/services/mcp/prompt.py:62-83`) that says "you may call another tool or reply to the user" — **no iteration cap, no budget, no stopping criterion** (`prompt.py:81-82`).++### 2. Entry point: vector search only, fanned out over every content word++`get_context` runs spaCy over the query (`retrieve.py:754`, `src/utils/nlp/ner.py:97-118`) and builds query *variants* (`retrieve.py:429-460`): the full question; **every token whose POS is not in the ignore set** (`ner.py:82-83`), unbounded; up to 5 noun chunks (`retrieve.py:48`, `451-457`); and up to 8 synthesized `"<Entity> <relation-word>"` probes for enumeration-looking questions (`retrieve.py:463-515`).++Each variant is processed independently by `_collect_facts_for_variant` (`retrieve.py:770-811`), and all variants are fanned out concurrently (`retrieve.py:883-888`). Per variant: one embedding call (`retrieve.py:610`), a node vector search at `k=25`, a relationship vector search at `k=25` (`retrieve.py:51`, `614-643`), and one Cypher call.++There is **no entity linking**. `src/core/search/entities.py` has `extract_str_entities_from_text` and `graph_adapter.search_entities`, but neither is called from `get_context`. Entry points are pure ANN lookups against node and relationship embeddings — no alias table, no exact-string fallback, no name-match tiebreak, no acceptance threshold.++*Correction to revision 1:* I previously wrote that this fanout makes latency scale with query length. **That is wrong and the data refutes it.** Over 152 questions the correlation between question content-word count and `retrieve_latency_ms` is **r = 0.089** (config A) and **r = 0.068** (config B), and per-bucket medians are flat from 2 to 10 content words (3795-4331 ms in config A). The fanout is concurrent, so extra variants add load and cost, not wall-clock. Bounding the fanout is still worth doing, but it is a **cost** optimisation, not a latency one.++### 3. Traversal: hard-capped at two relationship hops inside one event++`_collect_facts_for_variant` calls `graph_adapter.get_event_centric_neighbors(seed_uuids)` (`retrieve.py:782-784`). The Cypher (`src/lib/neo4j/client.py`) matches `(n)-[r]-(m)-[r2]-(b)` with `r2.flow_key = r.flow_key`. So traversal is **exactly two relationship hops**, and the `flow_key` equality constrains both hops to the *same event hub*: it returns one event's subject/event/object triangle and **cannot cross from one event to another**.++The one-shot API therefore does not do multi-hop reasoning in the graph sense. It does fact lookup — whole events, one at a time. Composition across events is left to the answering LLM over the union of retrieved events. That is bag-of-facts multi-hop, not path-based multi-hop.++| Path | Bound | Anchor |+|---|---|---|+| `get_context` facts | 2 hops, same `flow_key` | `retrieve.py:782`, `lib/neo4j/client.py` |+| `get_context` dossiers | depth 3, branch factor 3, 50 node-visit budget per entity, ≤5 entities | `retrieve.py:49-50`, `entity_info.py:28-29` |+| `/kg/hops` | always 2nd degree; the `degrees` argument is accepted and **silently ignored** | `controllers/kg.py:22-48` |+| `/retrieve/entity/context` | `context_depth` default 3 | `entity_context.py:35-37,72-74` |+| MCP `traverse_graph` | depth 1-5 (default 2), limit ≤100 | `graph.py:746-747` |+| MCP agent loop | **unbounded** | `prompt.py:81-82` |++### 4. Passages: hybrid RRF over dense + keyword, retrieved twice and prompted three times++`_retrieve_passages` (`retrieve.py:667-725`) is the only genuinely hybrid component: dense `search_data` at `k≥24` (`retrieve.py:52`) fused with `data_adapter.search` keyword hits via reciprocal rank fusion (`retrieve.py:705`, `fact_filter.py:67-78`). `_collect_passages` (`retrieve.py:813-849`) runs this for up to 3 query variants (6 for enumeration) and RRF-fuses the result lists again.++Then `_get_historical_context` (`retrieve.py:851-862`) calls `_retrieve_passages` **with the same `request.text`** and puts the top `historical_limit` results in a separate response field. Measured over `answers.jsonl`: **91.5% of `historical_context` entries are byte-identical to a `source_passage`** in config B (80.0% in config A).++The waste compounds downstream. `text_context` is built as `[passage] <chunk>` lines followed by fact lines (`retrieve.py:1057-1058`), and the answering prompt renders `text_context`, *then* `source_passages` again, *then* `historical_context` (`benchmarks/locomo/prompts.py:55-67`). Reconstructing the prompt from the logged fields:++| Copy | Origin | Anchor | Chars per question (config B) |+|---|---|---|---:|+| 1 | passages inlined into `text_context` | `retrieve.py:1057` | 61,955 |+| 2 | `## Source passages` section | `prompts.py:56-59` | 61,827 |+| 3 | `## Historical context` section (91.5% duplicate of copy 2) | `prompts.py:64-67` | 60,196 |+| | **total passage-derived** | | **183,978** |+| | if sent once | | **61,955 (−66.3%)** |++Mean answer prompt+completion is 51,397 tokens per question, and at ~3.6 chars/token the passage text alone accounts for essentially all of it. Two of the three copies originate in this workstream (copy 1 is `text_context` composition, copy 3 is the duplicate `historical_context` retrieval); copy 2 is the harness. This is the single largest cost defect in the system and it costs nothing in accuracy to fix.++### 5. Ranking: distance, then optionally PPR, then a filter that never runs++Candidates are deduplicated by `(r.uuid, r2.uuid)` keeping the best score (`retrieve.py:890-895`), scored as `min` vector distance over the event's three nodes (`retrieve.py:789-798`). Then:++1. **Semantic** — sort ascending by that min distance (`retrieve.py:897`).+2. **Structural** — if `use_ppr`, run personalized PageRank over an adjacency built from the seed set and re-sort by `-max(ppr over the event's three nodes)` (`retrieve.py:898-912`, `554-565`; `personalized_pagerank` in `fact_filter.py:79-115`). Personalization mass is `1/(1+distance)` per seed (`retrieve.py:518-525`). The PPR score **completely replaces** distance ordering rather than blending, edge transitions are uniform, and the adjacency comes from a single `get_event_centric_neighbors` call over the seeds (`retrieve.py:528-551`) so propagation cannot leave the seed neighbourhood. Note also that `_build_adjacency_from_seeds` does **not** apply `_is_currently_valid`, so PPR propagates over edges that the candidate filter at `retrieve.py:787` then hides — the structural signal and the returned fact set disagree about which edges exist.+3. **LLM-based** — `filter_relevant_facts` (`retrieve.py:916-921`). **This is a no-op.** The call site passes no `llm_adapter`, and `fact_filter.py:21-25` returns every index in that case, so `apply_fact_filter=True` is provably identical to the `else` branch `ranked[:max_facts]`. The prompt at `fact_filter.py:27-37` is dead code on this path.+4. **Temporal** — the only temporal signal is a hard filter dropping predicates with `invalid_at` or `deprecated=True` (`retrieve.py:420-426`, applied at `787`). No recency weighting, no time-scoped filtering, no `happened_at` comparison. `happened_at` is string-appended to the fact text (`retrieve.py:405-408`) and left to the LLM.++Ranking is therefore **semantic + optionally structural, with no working LLM stage and effectively no temporal stage.**++### 6. Sufficiency retry: a lexical overlap heuristic that costs a second full pass++`_context_looks_insufficient` (`retrieve.py:568-605`) fires when the context blob is under 80 characters or fewer than 35% of the question's content words appear verbatim. On firing, it re-runs up to 3 follow-up queries and merges results (`retrieve.py:967-1012`). The check is purely lexical, so it misses paraphrase hits and fires on lexically-adjacent-but-wrong context. And on the retry path **PPR and the fact filter are both skipped** — `curated = ranked[:max_facts]` directly (`retrieve.py:1000-1001`) — so triggering the retry silently downgrades ranking. Critically for the budget, the retry is a serial second retrieval round (`retrieve.py:979-988` is a `for` loop with `await` inside).++### 7. Output: triples and text are assembled independently, and grounding is not enforced++The response (`retrieve.py:1060-1065`) has four fields: `text_context` (passages, then facts, then `[dossier:...]` lines), `triples` (the same curated facts, structured), `historical_context`, and `source_passages`.++Provenance flows one way: facts contribute `source_chunk_ids` (`retrieve.py:806`, `648-664`) and those chunks are appended to `source_passages` (`retrieve.py:950-963`). Nothing checks the reverse. **No triple carries a citation, no passage is linked to the fact it supports, and the consumer gets three undifferentiated prose buckets.** The answer is grounded in the union of retrieved text, not in a retrieved path.++### 8. What the measurements actually establish++Four runs form two arms with byte-identical manifests within each arm (`manifest.json` per run dir; `ingest.n = 0` in all four, so all four queried the same pre-existing `locomoconv26` brain with no re-ingest):++- **Config A** = `push75-a`, `push75-b`: `use_ppr=false`, `sufficiency_retry=false`, `max_passages=8`, `max_facts=40`, `historical_limit=10`.+- **Config B** = `push75-c`, `push75-d`: `use_ppr=true`, `sufficiency_retry=true`, `max_passages=16`, `max_facts=50`, `historical_limit=16`.++| | a | b | c | d |+|---|---:|---:|---:|---:|+| Judge accuracy | 72.4% | 75.7% | 82.9% | **86.2%** |+| 95% CI | 64.8-78.9 | 68.3-81.8 | 76.1-88.1 | 79.8-90.8 |+| `answerable_rate` | 0.927632 | 0.927632 | 0.940789 | 0.940789 |+| `evidence_session_recall_full` | 0.826667 | 0.826667 | 0.973333 | 0.973333 |+| `evidence_session_recall_partial` | 0.913333 | 0.913333 | 0.986667 | 0.986667 |+| `answerer_gap` | 20.4% | 17.1% | 11.2% | 7.9% |+| retrieval p50 / p95 (ms) | 4081 / 5683 | 4053 / 5446 | 4793 / 6333 | 4835 / 6860 |+| min retrieval latency (ms) | — | 2508 | — | 3410 |+| total LLM tokens | 4.007M | 4.057M | 7.736M | 7.864M |++**The retrieval-side metrics are bit-identical within each arm** — `0.826667` twice, then `0.973333` twice, to every decimal, including every per-category value. Retrieval is deterministic for a fixed graph and config. **All observed run-to-run noise lives in the answer+judge layer.** Measured directly: with byte-identical retrieval, **19 of 152 questions (12.5%) flip correctness between `push75-a` and `push75-b`, and 17 of 152 (11.2%) between `push75-c` and `push75-d`** — despite `temperature=0` at `benchmarks/locomo/answer.py:45`.++That has three consequences, and they pull in different directions.++**(a) Single-run judge-accuracy comparisons are not evidence, exactly as `00:48` says.** Comparing `push75-b` to `push75-c` — the comparison I used in revision 1 to claim "+7.2 points" — gives an exact McNemar p of **0.0614**. Not significant. My revision-1 framing was wrong and I have removed it.++**(b) But the correct test for this design does establish the effect.** The same 152 questions are asked in both arms, so this is paired data and independent-proportion CIs are the wrong instrument. Pooling both runs per arm and scoring each question 0-2:++- Config A pooled **74.01%**, config B pooled **84.54%**, delta **+10.5 points**.+- Config B better on **32** questions, config A better on **10**, tied on 110. Exact sign test **p = 9.4 × 10⁻⁴**.+- Per category (pooled): multi-hop **+26.6** (15 better / 2 worse), open-domain **+11.5** (2 / 0), temporal **+9.5** (6 / 2), single-hop **+3.6** (9 / 6).++So the config change is established at p < 0.001 *as a paired effect over two runs per arm*, and the gain is concentrated in multi-hop and temporal — the two priorities in `00:11-12`. What is *not* established is any particular point estimate; +10.5 is the best estimate from four runs on one conversation with one answerer model, and external validity beyond `conv-26` is untested.++**(c) The retrieval-side case is much stronger than the accuracy case and needs no statistics at all.** Because retrieval is deterministic, these deltas are exact, not estimates:++| Retrieval metric | Config A | Config B | Δ |+|---|---:|---:|---:|+| Full evidence-session recall, overall | 82.7% | 97.3% | **+14.6** |+| … multi-hop | 62.5% | 96.9% | **+34.4** |+| … open-domain | 63.6% | 90.9% | +27.3 |+| … temporal | 86.5% | 94.6% | +8.1 |+| … single-hop | 92.9% | 100.0% | +7.1 |+| `answerable_rate`, overall | 92.8% | 94.1% | +1.3 |++**Multi-hop full evidence recall going 62.5% → 96.9% is the argument for the config change.** It is measured on retrieval output, it is reproducible to the digit, and it moves the metric the maintainer cares most about. The judge-accuracy delta is corroboration, not the case.++**On the two recall numbers, precisely.** `94.1%` is `metrics.answerable_rate`, labelled "Answerable rate (gold tokens in context)" in `report.md` — a lexical check that the gold answer's tokens appear somewhere in the retrieved context. `97.3%` is `metrics.evidence_session_recall_full`, the fraction of questions where all gold evidence *sessions* were among the retrieved sessions. Both figures are real, distinct, and were quoted correctly in revision 1; they are not two measurements of the same thing. The important caveat is about what they are computed *over*: `retrieved_session_ids` is derived from `text_context`, `source_passages` and `historical_context` (`benchmarks/locomo/evaluate.py:201-203`), and `text_context` is truncated to 20,000 characters before being stored (`evaluate.py:25`, `194`). Since `text_context` puts passages first and passages average 61,955 chars, **the stored `text_context` contains zero graph fact lines and zero dossier lines in 152 of 152 questions** (verified: zero occurrences of the `" | "` fact separator and of `[dossier:` across the file). So evidence-session recall and the answerable rate are effectively measuring the **passage channel**, not the graph channel. They are still trustworthy retrieval-side signals — they just do not credit or debit the graph. The answerer itself did see triples, because `answer_question` receives the raw API response (`evaluate.py:204`) and `prompts.py:60-63` renders up to 80 of them; but nothing in the logs lets me attribute any answer to them.++**The answerer gap is partly instability, not inability.** The gap is 11.2% (c) and 7.9% (d) with identical retrieved evidence, while the within-arm flip rate is 11-12%. So a substantial share of the "evidence was there and the answer was still wrong" gap is the answerer being non-deterministic rather than incapable. The qualitative conclusion from revision 1 survives — recall is no longer the binding constraint in aggregate, and roughly 8-11 points are lost after the right evidence is in the prompt — but the remedy splits: instability calls for self-consistency or a less noisy answerer, while inability calls for better pruning and ordering. These are different interventions and the current numbers cannot separate them.++**Cost.** Config B costs ~51.4k answer tokens per question versus ~26.4k for config A — 1.95x — for the +10.5 paired points. Per §4, roughly two-thirds of that is literal duplication.++### 9. Every run above was measured on a corrupted graph++Per `00:39`, `_invalidate_superseded_relationships` wrongly deprecated an actor's older event legs. The fix is present in the working tree (`src/workers/tasks/ingestion.py:474-506` now guards with `_is_event_entity` at `484-485` and `505-506`, and the docstring at `481-482` states event hub legs are never invalidated), but no backfill was authorised, and all four runs show `ingest.n = 0` — they reused the brain ingested before the fix.++The interaction with this workstream is direct and severe. `retrieve.py:787` drops a candidate if **either** leg fails `_is_currently_valid`, so a single wrongly-deprecated `MADE` leg removes the entire event triangle from the fact set. On an unrepaired brain, an actor's non-most-recent events are largely invisible to `get_context`'s fact channel.++Conclusions that this plausibly confounds, flagged explicitly:++- **The multi-hop numbers are depressed by an unknown amount.** Multi-hop questions need older events; those are exactly what was hidden. Pooled multi-hop accuracy of 53.1% (config A) and 79.7% (config B) are lower bounds on what the same code would score on a clean graph.+- **The measured value of PPR may be inflated or deflated.** Because `_build_adjacency_from_seeds` skips the validity filter (§5.2), PPR propagated over the deprecated topology while the fact filter hid it. On a repaired brain, PPR and the fact set will agree for the first time, and the +34.4-point evidence-recall gain may not reproduce at that magnitude.+- **Any claim about the graph channel's contribution is currently unmeasurable**, both because of this and because the logged context excludes facts entirely (§8).++This is why the passages-only ablation cannot be run first on the existing brain: on an unrepaired graph it would show the event graph contributing little, and that result would be an artifact of the bug rather than a property of the design.++---++## Guarantees and where they break++What this workstream is trying to guarantee: *given a natural-language question, return — in under a second — a compact evidence set that contains everything needed to answer it, ordered so the most load-bearing evidence comes first, with enough structure that the consumer can tell which stored event supports which claim.* Ranked by impact on the priorities in `00:11-12` (multi-hop, then temporal):++**G1. The path is ~5x over its own latency budget, and the better configuration is the slower one. (Gap; now the defining constraint.)** Measured p50 is 4053 ms (config A) and 4793-4835 ms (config B) against a sub-second target (`00:22`, `00:55`). The **fastest single question in either arm is 2508 ms / 3410 ms** — nothing in the benchmark comes within 2.5x of budget. Config B is +782 ms and 1.95x tokens for its accuracy gain, so the accuracy and budget goals are in direct conflict and the conflict cannot be split by tuning constants. §"Latency decomposition" below works out which stages can and cannot fit.++**G2. The best-measured configuration is not the shipped default. (Gap, trivially fixable.)** `GetContextRequestBody` sets `use_ppr = False`, `sufficiency_retry = False` (`requests.py:399-400`), `max_passages = 8`, `max_facts = 40` (`requests.py:396-397`). The controller reads them with `getattr(request, "use_ppr", True)` (`retrieve.py:898`, `967`, `914`), but Pydantic always materialises the field so the `True` fallback never fires for an HTTP request. A caller posting `{"text": ...}` gets config A. The `getattr` defaults also disagree with the schema (`max_facts` 50 vs 40, `_DEFAULT_MAX_PASSAGES` 16 vs 8), so the code reads as though the good config were the default. Note this is now entangled with G1: config B is the better-retrieving config *and* the more budget-violating one.++**G3. Multi-hop is structurally impossible in the one-shot path. (Deliberate trade-off, now the dominant accuracy ceiling.)** The `flow_key`-constrained two-hop query cannot leave an event hub (§3). Nothing finds a *path* between two entities, which is the product goal in `00:7`. The +34.4-point multi-hop evidence-recall gain from PPR shows graph propagation does real work here — and that more is available.++**G4. Passages are billed three times. (Bug, pure cost, measured.)** 183,978 characters of passage text per question where 61,955 would do, a 66.3% reduction available at zero information loss (§4). Two of the three copies are this workstream's.++**G5. The LLM fact filter does not exist. (Gap.)** `apply_fact_filter` is a no-op (§5.3). The pipeline has *no* mechanism that can judge a candidate fact irrelevant; the only pruning is `[:max_facts]` over a distance ordering. Under `00:22` the fix cannot be an LLM call on this path, which changes what I recommend here versus revision 1.++**G6. Grounding is asserted, not verified. (Gap; biggest risk to "answers accurately every time".)** No triple carries a citation (§7). Combined with an answering prompt that instructs hedging over abstention (`prompts.py:14-15`), the failure mode is confident, unattributable answers, and nothing in the response lets a downstream verifier check an answer against a path.++**G7. Query fanout is unbounded and noise-generating. (Gap — cost, not latency.)** Every content-word token becomes an independent seed query (§2). A variant like `read` still pulls 25 nodes + 25 relationships into `seed_hits`, which feeds both the PPR personalization vector (`retrieve.py:899`) and dossier entity selection (`retrieve.py:1014-1029`), so weak variants corrupt the structural ranking signal and the dossier choice. Correcting revision 1: this does **not** drive p50 (measured r ≈ 0.07-0.09, §2).++**G8. Time-scoped, aggregation, negation and comparison questions have no dedicated handling. (Gap; `00:12` names temporal a top priority.)**+- *Time-scoped*: only invalidity filtering; `happened_at` is a prompt string. Temporal scores 86.5-91.9% on LoCoMo because LoCoMo puts explicit dates in the dialogue and `prompts.py:13` contains a hand-tuned temporal rubric. That is benchmark prompt engineering, not a retrieval capability, and it will not transfer to a corpus without literal dates.+- *Aggregation / enumeration*: a regex-triggered probe generator (`retrieve.py:463-515`) plus "scan EVERY passage" in the prompt (`prompts.py:12`). No count, no completeness check.+- *Negation*: none. "What did X not do" retrieves the same events as "what did X do", and nothing distinguishes absence of evidence from evidence of absence.+- *Comparison*: none. Two entities' facts merge into one pool ranked by distance to the whole question.++**G9. Empty and contradictory results are handled asymmetrically. (Gap.)** Empty: `_collect_facts_for_variant` returns silently on no seeds (`retrieve.py:772-773`); `_retrieve_passages` swallows all exceptions (`702-703`, `715-716`); the chunk fetch swallows exceptions (`962-963`); the response can be empty with no error and no signal. `_get_historical_context` then falls back to **the most recent chunks in the brain regardless of query** (`864-880`) — topically random context presented as relevant. Contradictory: nothing detects contradiction; `invalid_at` is the only mechanism and it depends on the write path.++**G10. Two endpoints crash on a cache miss. (Bug.)** `EntityContext.get_context` returns a 2-tuple on "no vector hit" and "node not found" (`entity_context.py:61-62`, `69-70`) but the caller unpacks 4 (`controllers/entities.py:72-74`) → `ValueError` → 500 for any unknown entity. `controllers/entities.py:50` reads `paths.target_node` with no `None` guard while `retrieve_matches` can return `None`.++**G11. The agentic surface is unbounded and partly broken. (Gap.)** No step cap or budget (`prompt.py:81-82`); `search_memory` hands a raw query language to a model. `KGAgentSearchGraphTool._run` initialises `v_results: list[Vector] = []` (`KGAgentSearchGraphTool.py:109`), never populates it, then uses it to drive the only relationship-aware lookup (`:150-162`), so that branch is dead — and the final projection discards relationships, returning bare nodes (`:164-173`). An agent using this tool cannot see edges.++**G12. `traverse.py` is not a traversal.** `flatten_neighborhood` (`traverse.py:50-95`) formats an already-materialised neighborhood and issues no queries; `MAX_TRAVERSE_DEPTH`/`MAX_TRAVERSE_HOPS` bound serialization. Actual depth comes from `get_neighborhood(depth)` (`graph.py:724-731`, `765`).++**G13. No acceptance threshold on entity resolution. (Gap; blocks the agentic tier.)** `entity_context.py:56-59`, `entity_info.py:170-172` and `controllers/kg.py:31-34` call `search_nodes` without `k`, taking the facade default of 10 (`src/utils/vector_search.py:24-33`); the first two then take `[0]` with no similarity threshold (`entity_context.py:64`, `entity_info.py:182-183`). Resolution is "nearest of 10, always accept", with no confidence returned. `/kg/hops` compounds it with `similarity_threshold=0.0` (`kg.py:46`).++**G14. The dossier explorer is the most expensive component and its scoring is unsound. (Gap.)** `_score_neighbors` (`entity_info.py:57-86`) issues **one `get_neighbors` Cypher per visited node and one `vector_store_adapter.get_by_ids` per neighbour edge**, unbatched, in a Python loop. `_recursive_explorer` visits up to `_MAX_EXPLORATION_WORK = 50` nodes per entity (`entity_info.py:29`, `114`), and `_run_dossiers` runs up to 5 entities **serially** (`retrieve.py:1034-1055`). `retrieve_matches` also embeds twice per entity (`entity_info.py:167-168`). Scoring is `path_score = max(score for _, score in candidate)` (`entity_info.py:145`), so a long path containing one good hop beats a short uniformly-good path, with no length penalty. The file's own header comment says it is "raw, unprecise and w/ poor devx" and "not fully supported by the Event-Centric v2 kg" (`entity_info.py:32-36`). And per §8 its output never even reached the logged context.++---++## Latency decomposition and the sub-second verdict++I could not isolate per-stage milliseconds — there is no instrumentation in `get_context` and I am not permitted to add any. What I *can* do exactly is count the sequential I/O round trips each stage requires from the code, and mark what is concurrent. Serial depth, not operation count, sets wall-clock. Config B, typical non-enumeration question, V variants:++| # | Stage | Anchor | Round trips | Serial depth | Concurrency |+|---|---|---|---|---|---|+| 1 | spaCy NLP | `retrieve.py:754`; `ner.py:116-117` | 0 network, **2 spaCy passes** (`_process_doc` called twice) | 2 | **on the event loop**, blocking |+| 2a | facts per variant | `retrieve.py:770-811` | 4 per variant (embed, node search, rel search, Cypher) | 4 | V branches concurrent (`883-885`) |+| 2b | passages | `retrieve.py:813-849`, `667-725` | 4 per query × 3 queries (6 if enumeration) | **12-24** | **serial loop in one thread** (`825-830`) |+| 2c | historical | `retrieve.py:851-862` | 4 | 4 | concurrent with 2a/2b — and 91.5% duplicate output |+| 3 | PPR | `retrieve.py:898-912`, `528-551` | 1 Cypher | 1 | + 20 Python iterations **on the event loop** |+| 4 | fact filter | `retrieve.py:916-921` | 0 | 0 | no-op |+| 5 | provenance chunks | `retrieve.py:950-957` | 1 (≤80 ids) | 1 | — |+| 6 | sufficiency retry | `retrieve.py:967-1012` | 4×3 passages + 4×3 facts = 24 | **24** | **serial `for` with `await`** (`979-988`) |+| 7 | dossiers | `retrieve.py:1031-1055`; `entity_info.py:57-150` | ≤5 × (2 embeds + 1 search + ≤50 Cypher + one vector fetch **per edge**) | **hundreds** | serial per entity, serial per node, unbatched per edge |++Critical path ≈ stage 1 + max(2a, 2b, 2c) + 3 + 5 + (6 if it fires) + 7, where stage 7 alone can exceed everything else combined. Stages 6 and 7 also explain why config B's *minimum* latency is 3410 ms: even the easiest question pays for the dossier walk.++**Which stages could plausibly fit a sub-second budget**++- **Yes, as-is:** 1 (after removing the duplicate spaCy pass and moving it off the event loop), 2a (genuinely parallel, only 4 deep, and the per-variant embeds are batchable into one call), 3 (PPR arithmetic is cheap — but its adjacency Cypher should be replaced by a precomputed table, and the 20 iterations must move off the event loop), 5 (one batched fetch), 4 (free).+- **Yes, if restructured:** 2b, but only collapsed to a single query with the embed batched and the dense and keyword searches issued concurrently — serial depth 12-24 cannot fit. 2c must simply be deleted.+- **No, at any per-operation latency:** 6 (a second full retrieval round is by definition a second pass) and 7 (hundreds of serial unbatched round trips; even at an optimistic 5 ms per operation, stage 7 alone blows a 1000 ms budget).++**Verdict: sub-second is achievable, but only by deleting stages 6 and 7 from this path and flattening 2b — not by optimising them.** A path consisting of stage 1 + one parallel wave of serial depth 4 + a precomputed-adjacency PPR + one chunk fetch has a serial depth of roughly 6-8 I/O operations, which fits 1 s at any plausible per-operation latency. The measured 2508 ms floor today is consistent with the dossier walk dominating, but **I could not verify that attribution**, so the first task in the plan is instrumentation, not optimisation. If per-stage timing shows the floor is spread evenly rather than concentrated in stage 7, the sub-second target and the current architecture are irreconcilable and the honest answer is to relax one of them — the trade being roughly **+10.5 paired accuracy points and +14.6 points of evidence recall against a 4-5x budget overrun**, which on `00:11-12`'s priorities I would resolve in favour of a two-tier split rather than by abandoning either goal.++**Where the reranker runs.** Revision 1 recommended a cross-encoder on the context path. Under `00:22` and `00:25` that is inadmissible as stated: a hosted reranker is a network hop on a path with no headroom, and an in-process cross-encoder over 50 candidates costs 50-200 ms on CPU, which is 5-20% of the entire budget for a component that is not the bottleneck. **Revised: on the context path, use a cheap learned reranker over features already computed** — vector distance, PPR score, event `happened_at` recency, provenance-chunk overlap with the top passages, seed-variant strength. A linear model or small GBDT over five features is microseconds and no model server. The cross-encoder and any listwise LLM reranking move to the deep-navigation tier, where `00:23` permits them.++---++## Open questions for the maintainer++Answered since revision 1 and removed: whether the context path may make LLM calls, and what its latency budget is (both settled by `00:22` and `00:55`). Not re-proposed: the supersession fix and the pgvector dimension question (`00:39-40`).++1. Was `filter_relevant_facts` ever meant to receive an `llm_adapter`, or was the LLM path abandoned deliberately — and given `00:22`, should the dead prompt at `fact_filter.py:27-37` be deleted or moved to the deep tier?+2. Is the `flow_key` equality constraint in `get_event_centric_neighbors` a deliberate decision that cross-event chaining belongs to the agentic surface, or an artifact of the schema you would lift if a cross-event query existed?+3. Does the sub-second budget apply to p50, p95, or a hard timeout — and is a degraded-but-fast response (fewer passages, no PPR) preferable to a slower complete one when the budget is at risk?+4. Should the dossier channel (`EventSynergyRetriever`, whose own header calls it "raw, unprecise") be deleted from `/retrieve/context` outright, or preserved behind a flag for the deep tier?+5. Who is the primary consumer of `triples` versus `text_context` — does any caller reason over the structured triples, or do they all concatenate `text_context`?+6. Is `historical_context` intended to be query-relevant (it currently re-runs the same passage search) or a genuinely separate recency channel — and if the latter, should it be recency-ordered rather than relevance-ordered?+7. Should `/retrieve/context` return an explicit "insufficient evidence" signal, or is silently returning a weak context the intended contract?+8. When two events contradict and neither has `invalid_at`, should retrieval return both, prefer recency, or refuse?+9. Is the MCP `traverse_graph` surface meant to become the deep-navigation retrieval path, or is it an integration tool a future `chat_agent` would bypass?+10. Should `search_memory` (arbitrary read-only Cypher/SQL from an LLM) survive into production?+11. Are the LoCoMo-specific rubrics in `benchmarks/locomo/prompts.py:12-15` meant to migrate into the product, or are they benchmark-only tuning?+12. Is a re-ingested clean brain available for benchmarking after the supersession fix, and if not, who owns creating one — the ablation in Phase 0 is uninterpretable without it?+13. Given the 11-12% per-question answerer flip rate at `temperature=0`, is reducing answerer variance (self-consistency, a different model) in scope for anyone, or must this workstream treat it as fixed noise?+14. Is `entity_sibilings.py` / `EntitySinergyRetriever` in scope for retrieval quality work?++---++## Frontier techniques++Each verdict is now also judged against the two-tier budget in `00:22-23`. "Context-tier" means admissible on `/retrieve/context`; "deep-tier" means MCP and other REST endpoints only; "write-time" means it must be precomputed.++### A. Personalized PageRank over an entity–passage graph (HippoRAG / HippoRAG 2)++**Mechanism.** Build an open KG from the corpus, index passages alongside entity nodes, seed PPR with query-linked entities, and let one propagation step surface multi-hop-relevant passages without iterative LLM calls.++**arXiv.** 2405.14831 (HippoRAG), 2502.14802 (HippoRAG 2).++**Reported gain.** HippoRAG: up to +20% over SOTA on multi-hop QA; single-step retrieval matches or beats IRCoT at 10-30x lower cost and 6-13x lower latency. HippoRAG 2: +7% on associative memory over the best embedding model while fixing HippoRAG's regression on simple factual recall.++**Cost.** Offline graph construction (already paid). Online PPR arithmetic is cheap; the expensive part is materialising the adjacency.++**Fit.** Partially validated here already: `use_ppr` moved multi-hop full evidence recall 62.5% → 96.9% (exact, deterministic). Three divergences from the paper matter: the adjacency is built only from seeds via a live Cypher call (`retrieve.py:528-551`) so propagation cannot go further; passages are not graph nodes, so PPR cannot rank passages; and the PPR score replaces rather than blends with the semantic score (`retrieve.py:910-912`). The single-step property is exactly what the budget requires — this is the one technique in the literature whose whole selling point is getting iterative-retrieval quality without iteration.++**Verdict: adopt — context-tier, with the adjacency moved to write time.** A materialised event-adjacency table replaces stage 3's Cypher call and makes wider propagation free at query time.++### B. Query-aware edge weighting for graph propagation (CatRAG)++**Mechanism.** Attacks the "static graph fallacy": PPR with indexing-time transition probabilities lets walks drift into high-degree hubs and stop short of the final hop. Adds symbolic anchoring, query-aware dynamic edge weighting, and a key-fact passage weight bias on top of HippoRAG 2.++**arXiv.** 2602.01965.++**Reported gain.** Consistent wins over SOTA on four multi-hop benchmarks; notably, plain recall improves modestly while *reasoning completeness* — recovering the whole evidence path without gaps — improves substantially.++**Cost.** One scoring pass over candidate edges per query; no LLM calls if embedding-based.++**Fit.** High, and the diagnosis matches: event hubs and `OCCURRED_WITHIN` context nodes are natural hubs, transitions are uniform, and `flow_key` is a ready-made symbolic anchor. The measured profile — 94.1% answerable but an 8-11 point residual gap — is "partial recall, incomplete chain."++**Verdict: adopt — context-tier**, after A, with weights precomputed per edge and only the query-similarity term computed online.++### C. Path-based retrieval and flow pruning (PathRAG)++**Mechanism.** Argues graph-RAG's problem is redundancy, not insufficiency. Retrieves relational *paths* using flow-based pruning and linearises paths, not flat chunk lists, into the prompt.++**arXiv.** 2502.14902.++**Reported gain.** Beats SOTA graph-RAG baselines across six datasets and five evaluation dimensions.++**Cost.** Path enumeration between node pairs; prompts get smaller, not larger.++**Fit.** Addresses G3 and G6 at once, and the *output-format* half is nearly free: BrainAPI already has ordered candidates and flattens them with `" | "` (`retrieve.py:412-413`). Path-shaped output also gives each fact a position in a chain, which is the raw material for attribution. The event-hub schema suits it — a path is an alternating node/event/node sequence. The flow-pruning half needs path enumeration, which is not sub-second on demand.++**Verdict: adapt — split by tier.** Path-shaped *formatting* is context-tier and cheap. Flow-based path *enumeration* is deep-tier or write-time.++### D. Listwise LLM reranking (RankGPT and successors)++**Mechanism.** Prompt an LLM to reorder candidates by relevance (sliding window for long lists), optionally distilled into a small specialised reranker.++**arXiv.** 2304.09542.++**Reported gain.** Properly instructed LLMs match or beat supervised SOTA rerankers zero-shot on BEIR; a distilled 440M model beats a 3B supervised model.++**Cost.** One or more LLM calls per query, 300 ms to seconds. A distilled cross-encoder removes the LLM but adds a hosted model and 50-200 ms on CPU.++**Fit.** The socket exists and is empty (G5), and precision-after-retrieval is where accuracy is lost. But `00:22` and `00:25` explicitly rule out per-query LLM reranking on the context path, and the distilled variant still spends 5-20% of a 1 s budget on a component that is not the bottleneck.++**Verdict: reject on the context path; adopt deep-tier.** Context-tier substitute: a linear/GBDT reranker over features already computed (distance, PPR score, `happened_at` recency, provenance overlap, variant strength) — microseconds, no model server, and it can be trained on the existing `answers.jsonl` labels. This is the significant change from revision 1.++### E. Complexity-routed retrieval (Adaptive-RAG)++**Mechanism.** A small classifier predicts query complexity and routes to no-retrieval, single-step, or iterative multi-step retrieval, with labels harvested from which strategy actually succeeded.++**arXiv.** 2403.14403.++**Reported gain.** Better efficiency *and* accuracy than both fixed-strategy and prior adaptive baselines on mixed-complexity open-domain QA.++**Cost.** One small-LM forward pass (single-digit ms) — comfortably inside the budget.++**Fit.** This is the mechanism that implements `00:18`'s "a router, not a winner". The labels already exist in `benchmarks/runs/*/answers.jsonl`: per-question category, `answerable_rate` inputs, and `judge_correct`. With four runs there are now two observations per arm per question, which is exactly what is needed to label a question as "one-shot suffices" versus "one-shot unreliable" rather than "one-shot failed once."++**Verdict: adopt — context-tier (the router itself), routing to the deep tier.** Load-bearing for the recommendation.++### F. Iterative retrieve-then-reason (IRCoT, Self-Ask, StructGPT, ToG)++**Mechanism.** Interleave retrieval with reasoning so what to retrieve next depends on what has been derived. IRCoT alternates CoT sentences and retrievals; Self-Ask emits explicit follow-up questions; StructGPT formalises invoke→linearise→generate over structured data; ToG runs LLM-guided beam search over KG paths.++**arXiv.** 2212.10509 (IRCoT), 2210.03350 (Self-Ask), 2305.09645 (StructGPT), 2307.07697 (ToG).++**Reported gain.** IRCoT: up to +21 retrieval points and +15 QA points on HotpotQA / 2WikiMultihopQA / MuSiQue / IIRC with reduced hallucination. ToG: SOTA on 6 of 9 datasets, training-free, with knowledge traceability and correctability. Self-Ask quantifies the *compositionality gap* — models answer all sub-questions correctly yet fail the composition, and the gap does not shrink with scale.++**Cost.** Multiple LLM calls plus multiple retrievals. HippoRAG measured IRCoT at 10-30x cost and 6-13x latency versus single-step graph retrieval.++**Fit.** Self-Ask's finding is the strongest argument for keeping these available: an 8-11 point gap with the evidence present is a compositionality gap by definition. ToG is the best template for the deep tier because its beam search maps onto `traverse_graph` and it preserves traceability, which G6 needs. Categorically inadmissible on the context path.++**Verdict: adopt ToG-style bounded beam search — deep-tier only. Reject on the context path.**++### G. Explicit evidence-gap analysis as the stopping criterion (FAIR-RAG)++**Mechanism.** Decompose the query into a checklist of required findings, audit accumulated evidence against it, emit targeted sub-queries for the specific gaps, iterate until sufficient.++**arXiv.** 2510.22344.++**Reported gain.** +8.3 F1 over the strongest iterative baseline on HotpotQA (0.453 F1), with 2Wiki and MuSiQue also reported.++**Cost.** An LLM call per audit round.++**Fit.** `_context_looks_insufficient` (`retrieve.py:568-605`) is a lexical stand-in for this module. The *check* is cheap and worth keeping on the context path as a **signal to the caller** (G9, open question 7). The *retry* is stage 6 — a second full retrieval pass — and per the decomposition it cannot fit the budget.++**Verdict: split. Keep the cheap check context-tier as an emitted signal; move the retry and any LLM audit deep-tier.** Also fix the existing defect that the retry skips PPR and the reranker (`retrieve.py:1000-1001`).++### H. Episodic memory graph with an agentic retriever (REMem)++**Mechanism.** Offline, convert experiences into a hybrid memory graph linking *time-aware gists* to facts. Online, an agentic retriever with purpose-built tools iterates over that graph. The paper names prior failure modes as overlooking episodicity, lacking explicit event modelling, and overemphasising simple retrieval over reasoning.++**arXiv.** 2602.13530.++**Reported gain.** +3.4 on episodic recollection and +13.4 on episodic *reasoning* over Mem0 and HippoRAG 2 across four benchmarks; also more robust refusal on unanswerable questions.++**Cost.** Agentic inference at query time; offline gist construction.++**Fit.** The closest paper to what BrainAPI already is. BrainAPI has the rarer half — explicit event modelling — paired with the weaker retrieval strategy. The offline **time-aware gist layer is write-time work and therefore admissible under `00:25`**, and it is the direct answer to G8's aggregation and time-scoping problems as well as a cheap replacement for the online dossier walk (G14). Its refusal-robustness result speaks to G9.++**Verdict: adopt — architectural reference. Gist layer is write-time (crosses into the ingestion workstream); the agentic retriever is deep-tier.**++### I. Does agentic search remove the need for a graph? (RAGSearch benchmark)++**Mechanism.** A controlled benchmark holding LLM backbone, retrieval budget and inference protocol fixed, comparing dense RAG and several GraphRAG variants *as retrieval infrastructure underneath* agentic search, training-free and RL-based.++**arXiv.** 2604.09666.++**Finding.** Agentic search substantially improves dense RAG and narrows the gap to GraphRAG, but GraphRAG **remains advantageous for complex multi-hop reasoning** and yields more stable agentic behaviour once offline cost is amortised. Explicit structure and agentic search are complementary.++**Verdict: adopt the conclusion** as the framing for ADRs 003/004 — it is the empirical backing for `00:18`'s router.++### J. RL-trained search agents (Search-R1, Search-o1)++**Mechanism.** Train the model to interleave reasoning and search calls with outcome-based rewards and retrieved-token masking (Search-R1); or add an agentic search workflow plus a Reason-in-Documents module that compresses retrieved text before it enters the reasoning chain (Search-o1).++**arXiv.** 2503.09516 (Search-R1), 2501.05366 (Search-o1).++**Reported gain.** Search-R1: +41% (Qwen2.5-7B) and +20% (Qwen2.5-3B) over RAG baselines across seven QA datasets.++**Cost.** RL infrastructure and a trainable open-weights model; BrainAPI calls a hosted `deepseek-v4-flash`.++**Verdict: reject the training requirement.** **Adopt one component, write-time:** Search-o1's Reason-in-Documents idea — compress passages before they enter a prompt — attacks the 51k-token cost directly, and if the compression is computed at ingest rather than per query it is admissible under `00:25`. That is the same artifact as REMem's gists.++### K. Recursive summary trees (RAPTOR) and community summaries (GraphRAG, LightRAG)++**Mechanism.** RAPTOR recursively embeds, clusters and summarises chunks into a tree and retrieves across abstraction levels. GraphRAG pregenerates community summaries for entity clusters and map-reduces partial answers. LightRAG adds dual-level retrieval with incremental index updates.++**arXiv.** 2401.18059 (RAPTOR), 2404.16130 (GraphRAG), 2410.05779 (LightRAG).++**Reported gain.** RAPTOR: +20% absolute on QuALITY with GPT-4. GraphRAG: substantial gains in comprehensiveness and diversity for global sensemaking over ~1M-token corpora. LightRAG: retrieval accuracy and latency improvements plus incremental updates.++**Cost.** GraphRAG's community summarisation is a heavy offline LLM pass that must be redone as the graph grows — a poor match for an append-only store with continuous writes. LightRAG's incremental update is the mitigation.++**Fit.** Open-domain is the category where recall is still clearly binding (76.9% answerable, 90.9% evidence recall, pooled 61.5% accuracy) and it is exactly the global-sensemaking class these target. But it is 13 questions, and static communities fight append-only writes.++**Verdict: reject GraphRAG community summarisation. Adapt the narrow idea — per-entity gists maintained incrementally, write-time.** Note that `[dossier:...]` (`retrieve.py:1031-1055`) is already a crude *online* version of this, computed per query at depth 3 with a 50-node budget — the expensive way to do it (G14).++### L. Grounding, attribution, and citation faithfulness (ALCE, Self-RAG)++**Mechanism.** ALCE defines the benchmark and automatic metrics (fluency, correctness, citation quality) for generating text with citations. Self-RAG trains a model to emit reflection tokens deciding when to retrieve and critiquing whether its output is supported.++**arXiv.** 2305.14627 (ALCE), 2310.11511 (Self-RAG).++**Reported gain.** ALCE: even the best systems lack complete citation support 50% of the time on ELI5 — unattributed generation is the norm unless measured. Self-RAG (7B/13B) beats ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning and fact verification, with gains in factuality and citation accuracy.++**Cost.** ALCE-style metrics are nearly free if the response carries passage IDs. Self-RAG needs a trained model.++**Fit.** `get_context` already computes `provenance_ids` per candidate (`retrieve.py:806`, `942`) and throws the association away when flattening into `source_passages`. Emitting `{triple, supporting_chunk_ids}` is a dict-shape change with no extra I/O — fully admissible on the context path — and it makes citation quality measurable for the first time.++**Verdict: adopt per-triple attribution — context-tier, zero added latency. Reject Self-RAG's trained variant; its verification loop belongs deep-tier.**++### M. Generate-then-ground and tree-structured iterative retrieval++**Mechanism.** GenGround alternates emitting a simpler single-hop question answered from parametric knowledge, then grounding that Q-A pair against retrieved documents and correcting it. Tree of Reviews expands a tree of reasoning paths, handling each retrieved paragraph separately and dynamically choosing to search, reject or accept, so one bad paragraph does not poison a chain.++**arXiv.** 2406.14891 (GenGround), 2404.14464 (Tree of Reviews).++**Reported gain.** Both report SOTA-class results on the HotpotQA / 2Wiki / MuSiQue family.++**Fit.** ToR's cascade-error argument is a design constraint on any deep-tier agent built over BrainAPI, because a single-chain agent on top of "nearest of 10, always accept" resolution (G13) will cascade confidently.++**Verdict: adapt — deep-tier design constraint, not a standalone priority.**++### N. Temporal QA benchmarking (ChronoQA) and temporal KG memory (Zep)++**Mechanism.** ChronoQA is a 5,176-question benchmark over 300k news articles covering absolute, aggregate and relative temporal types with explicit and implicit time expressions. Zep/Graphiti maintains a temporally-aware KG synthesising conversational and business data while preserving historical relationships and edge validity intervals.++**arXiv.** 2508.12282 (ChronoQA), 2501.13956 (Zep).++**Reported gain.** Zep: 94.8% vs 93.4% on DMR, up to +18.5% accuracy on LongMemEval with 90% lower latency, strongest on cross-session synthesis.++**Fit.** Temporal is a stated priority (`00:12`) and 86.5-91.9% on LoCoMo is not evidence of temporal capability: LoCoMo puts dates in the dialogue and `prompts.py:13` carries a hand-tuned rubric. Zep's validity-interval model is close to what `_is_currently_valid` gestures at, but retrieval never uses time as a ranking or filtering dimension. An as-of-time *filter* is an indexed predicate and therefore context-tier admissible; it is cheaper than what is there now, not more expensive.++**Verdict: adopt as-of-time filtering and recency features — context-tier.** ChronoQA-style question types are an eval gap to hand to `04-evaluation-and-applications.md`, coordinated with `05-temporal-truth.md`.++---++## Recommendation: one-shot vs agentic navigation++**Route, do not choose** — which matches `00:18`. RAGSearch (2604.09666) holds backbone and budget fixed and finds that agentic search lifts flat dense RAG and narrows the gap to graph RAG, while explicit graph structure stays advantageous for complex multi-hop and makes agentic behaviour more stable. REMem (2602.13530) arrives at the same place from the other side: an explicit *event* graph plus an agentic retriever beats HippoRAG 2 and Mem0, with the largest margin (+13.4) on episodic *reasoning*. BrainAPI has the rarer half of that pair and is pairing it with the weaker strategy.++The sub-second budget makes the split sharper than it was in revision 1: the context path is not merely the cheaper option, it is now constitutionally incapable of hosting anything iterative. So the two tiers get different jobs.++1. **The context path's job is recall and grounding within a hard budget** — one bounded parallel wave, precomputed structure, cheap feature-based ranking, per-triple provenance, an explicit sufficiency signal. It must get *faster* while its evidence recall stays at config B levels. Everything that made config B good and slow must be re-sourced from write-time precomputation rather than query-time work.+2. **The deep tier's job is composition** — iterative retrieve-then-reason, ToG-style bounded beam search over `traverse_graph`, gap-analysis stopping, LLM or cross-encoder reranking, verification.+3. **A cheap router decides**, trained on the labels already in `benchmarks/runs/*/answers.jsonl` (Adaptive-RAG, 2403.14403).++**Route to one-shot when:** the question names entities that resolve above a confidence threshold (which must first exist — G13); the answer plausibly lives inside one or two event hubs (attribute reads, "when did X do Y"); the lexical sufficiency check passes on the first pass; or a latency SLA applies. On LoCoMo that is most of the single-hop category, already at 88.6% with nothing an agent would add.++**Route to deep navigation when:** (a) the question requires composing facts across *different* event hubs, which the `flow_key`-constrained query structurally cannot retrieve (G3) — the strongest signal, and detectable from the question rather than the results; (b) it is a set/enumeration question where completeness matters and the regex probe generator cannot know whether it found every member (G8); (c) it is a multi-entity comparison, which needs separate retrievals and a joint ranking that the shared candidate pool destroys; (d) the sufficiency check fails on the context path — note this replaces the in-path retry, which moves here wholesale; or (e) the caller asks for discovery rather than an answer, since "non-obvious connections" (`00:7`) is an agentic workload.++**Never route to deep navigation when** entity resolution is low-confidence, because a beam-search agent on "nearest of 10, always accept" will chase a wrong entity for several expensive steps. Fix G13 first.++**Where each fails.** One-shot fails when the answer requires a path rather than a set: it returns many events ranked by distance and delegates composition to the answerer — the compositionality gap Self-Ask measured (2210.03350), which does not close with a bigger model. Deep navigation fails on cost and variance: 6-13x latency and 10-30x cost versus single-step graph retrieval (2405.14831), an unbounded loop today (G11), cascading errors from one bad expansion unless paths are isolated (2404.14464), and a tail that a p50 hides.++**Honest statement of the conflict.** As measured, the higher-recall configuration is also the slower and 1.95x more expensive one, and *both* configurations are 4-5x over budget. There is no setting of the existing knobs that is simultaneously sub-second and config-B-quality. The plan below resolves this by moving config B's expensive ingredients to write time and deleting two stages from the query path; if per-stage instrumentation shows that is not enough, the target and the quality goal are irreconcilable on this path and the maintainer has to relax one.++---++## Implementation plan++Sizing: **S** = 1-2 files, under a day. **M** = 3-5 files, a few days. Baseline is config B as measured (pooled 84.5% judge accuracy, 97.3% full evidence recall, 94.1% answerable, 4793-4835 ms p50, ~51.4k answer tokens per question).++**Measurement protocol, binding on every task below** (derived from §8 and `00:44-51`):++- **Primary readout is retrieval-side** (`evidence_session_recall_full`, `answerable_rate`, per-category). These are deterministic for a fixed graph and config, so **one run per arm is exact.** Report them per category, since the overall figure hides the multi-hop movement that matters.+- **Judge accuracy is secondary and requires ≥2 runs per arm plus a paired test** (exact sign test or McNemar on per-question correctness), never a comparison of independent CIs. I verified the difference this makes: `push75-b` vs `push75-c` as single runs gives p = 0.0614, while pooling two runs per arm and testing paired gives p = 9.4 × 10⁻⁴ for the same underlying change.+- **What this benchmark can and cannot resolve.** Simulating from the observed per-question outcomes: a *concentrated* effect of the magnitude actually seen (+10.5 points, driven by ~32 questions flipping) is detected with 0.94 power at 1 run per arm and 0.99 at 2. A *diffuse* uniform +5-point shift is detected with only 0.14 power at 2 runs per arm and 0.52 even at 6; a diffuse +3-point shift never exceeds 0.32 power at 8 runs. **So: fixes that repair specific broken question classes are measurable here; broad precision polish is not, at any affordable run count, and must be justified on retrieval-side metrics instead.**+- Latency and token figures from `report.json` are trustworthy (`00:51`) but were collected at `concurrency=2` against a local server; treat cross-run latency deltas under ~200 ms as noise.++### Phase 0 — Establish what is actually true before optimising anything++Ordered first because three of my previous conclusions turned out to rest on unmeasured assumptions.++**T0.1 (S) — Instrument `get_context` per stage.** Add timing around each stage in the §"Latency decomposition" table and emit it as a debug field or log line. No behaviour change.+*Acceptance:* a single request yields per-stage wall-clock for stages 1-7 plus a count of round trips issued.+*Verify:* run 20 questions and publish the stage breakdown. **This gates every latency claim in the plan, including my own.** If the 2508 ms floor is not concentrated in stage 7, the redesign in Phase 2 is misdirected.++**T0.2 (S) — Graph-channel ablation, on a clean brain.** Re-ingest `conv-26` on the fixed supersession code (`00:39`), then run three arms: facts+passages (config B), `max_facts=0` (passages only), and — because §8 shows the logged context never contained facts — one arm with passages capped low enough that facts survive any downstream truncation.+*Acceptance:* per-category `evidence_session_recall_full` and pooled paired judge accuracy for each arm; the graph channel's contribution stated as a point estimate with the resolvable floor attached.+*Verify:* **2 runs per arm** (0.99 power for a concentrated 10-point effect, per the protocol above). One run per arm suffices for the retrieval-side metrics. If the passages-only arm is within 5 points of config B, report "no contribution detectable above the ~5-point resolution floor" — *not* "no contribution". **Do not run this on the existing unrepaired brain**: §9 explains why the result would be an artifact of the supersession bug rather than a property of the design.++**T0.3 (S) — Re-baseline config A vs config B on the clean brain.** The +34.4-point multi-hop evidence-recall gain from PPR was measured on a graph where most historical events were filtered out at `retrieve.py:787`.+*Acceptance:* config A and config B rerun on the repaired brain, retrieval metrics reported per category.+*Verify:* 1 run per arm for retrieval metrics, 2 for accuracy. If the PPR gain shrinks substantially, Phase 1's ordering changes.++**T0.4 (S) — Audit the answerer gap by hand.** Sample 20 questions that are `answerable` but judged wrong, and classify each as (i) evidence present, answerer failed; (ii) lexical token overlap only, evidence not really present; (iii) flipped between the two runs of the arm.+*Acceptance:* a 20-row classification with counts.+*Verify:* the split between (i), (ii) and (iii) decides whether Phase 2's precision work or a variance-reduction measure is the better investment. §8 shows the 11-12% flip rate is the same order as the gap itself, so this is not a formality.++> **Checkpoint C0.** Publish: per-stage latency, the graph channel's measured contribution, clean-brain baselines, and the answerer-gap split. **No optimisation work starts before this.** Three revision-1 claims were wrong or unverifiable without it.++### Phase 1 — Free wins: correctness, cost, and defaults++None of these trade accuracy for latency; all are admissible on the context path.++**T1.1 (S) — Stop billing passages three times.** Make `_get_historical_context` (`retrieve.py:851-882`) either reuse `passage_hits` or exclude chunk IDs already in `source_passages`, and stop inlining `[passage]` blocks into `text_context` (`retrieve.py:1057`) now that `source_passages` is a separate field.+*Acceptance:* `set(historical_context) ∩ set(source_passages) == ∅`; `text_context` contains no passage text.+*Verify:* answer tokens per question **below 20k** (from 51.4k — the measured single-copy figure is 61,955 of 183,978 chars, a 66.3% cut) with per-category evidence recall unchanged. Deletes one full round trip from stage 2c as a side effect. Coordinate with `04-evaluation-and-applications.md`, since `prompts.py:55-67` also renders `source_passages` separately.++**T1.2 (S) — Align defaults with the config that retrieves better.** Set `use_ppr=True`, `sufficiency_retry` per T2.3's outcome, `max_passages=16`, `max_facts=50` in `GetContextRequestBody` (`requests.py:392-400`), and delete the `getattr(..., True)` fallbacks at `retrieve.py:898`, `914`, `967` so the schema is the single source of truth.+*Acceptance:* `POST /retrieve/context {"text": "..."}` with no other fields reproduces the config-B retrieval metrics; no `getattr` default disagrees with a Pydantic default.+*Verify:* `evidence_session_recall_full` ≥ the clean-brain config-B figure from T0.3. **Justified on retrieval-side evidence** (+14.6 overall, +34.4 multi-hop, deterministic), not on the judge-accuracy delta. Note this raises p50; it is provisional until Phase 2 restores the budget, and T1.2 and T2.x should ship together.++**T1.3 (S) — Per-triple provenance.** `provenance_ids` are computed per candidate (`retrieve.py:806`, `942`) then merged away. Attach `supporting_chunk_ids` to each `GetContextTriple`.+*Acceptance:* every returned triple carries its own chunk IDs, matching the candidate it came from; zero-provenance triples are counted.+*Verify:* a script reports the fraction of triples with ≥1 supporting chunk — the ALCE-style citation-support number (2305.14627), currently unmeasurable. No added I/O. If it comes back very low, that is a finding for `01-ingestion-extraction.md` about `source_chunk_ids` coverage, not a bug here.++**T1.4 (S) — Fix the crashing endpoints and the dead agent tool.** Return a 4-tuple from both early exits in `EntityContext.get_context` (`entity_context.py:61-62`, `69-70`); guard `paths is None` at `controllers/entities.py:50`; remove the dead `v_results` branch and stop discarding relationships in `KGAgentSearchGraphTool` (`:109`, `150-173`).+*Acceptance:* `GET /retrieve/entity/context?target=zzzznotanentity` returns 200-with-nulls or 404, not 500; the agent tool returns edges.+*Verify:* `pytest tests/ -k "entity_context or entity_info or kg_agent"` plus negative-path tests.++**T1.5 (S) — Bound the query fanout.** Restrict seed variants to the full question, NER entities and noun chunks; drop bare single tokens or down-weight them in `_seed_personalization` (`retrieve.py:518-525`) and exclude them from dossier entity selection (`retrieve.py:1014-1029`).+*Acceptance:* ≤8 variants per question; no `seed_hits` entry sourced from a single non-entity token.+*Verify:* embedding calls and vector searches per request down ≥50%; per-category evidence recall unchanged. **Expect no p50 improvement** — measured r ≈ 0.07-0.09 between question length and latency (§2). This is a cost and precision task; if evidence recall *drops*, the weak variants were carrying recall and this must be reverted and re-scoped.++> **Checkpoint C1.** Target: answer tokens per question under 20k (from 51.4k), evidence recall at or above the clean-brain config-B baseline, citation support measured for the first time, no 500s on unknown entities. Latency unchanged or slightly worse — that is Phase 2's job.++### Phase 2 — Make the context path meet its budget++This phase exists because of `00:22` and `00:55`. Its success criterion is latency, not accuracy, and its constraint is that per-category evidence recall must not regress.++**T2.1 (M) — Precompute the PPR adjacency.** Replace the live `get_event_centric_neighbors` call in `_build_adjacency_from_seeds` (`retrieve.py:528-551`) with a materialised event-adjacency table maintained at write time, and apply `_is_currently_valid` when building it so the structural signal and the fact set stop disagreeing (§5.2). Move the 20 PPR iterations off the event loop. Refs: 2405.14831, 2502.14802.+*Acceptance:* stage 3 issues zero live graph queries; PPR output is identical to today's for the same seed set on a fixed graph (regression test); adjacency covers ≥2 rings, not 1.+*Verify:* stage-3 time from T0.1 under 50 ms; multi-hop `evidence_session_recall_full` at or above baseline. Crosses into `03-memory-substrate.md` for the table itself.++**T2.2 (M) — Collapse passage retrieval to one wave.** `_collect_passages` (`retrieve.py:813-849`) runs 3-6 `_retrieve_passages` calls serially in one thread, each itself 4 serial round trips. Reduce to a single query, batch the embedding, and issue the dense and keyword searches concurrently.+*Acceptance:* stage 2b serial depth ≤4 round trips, down from 12-24.+*Verify:* stage-2b time from T0.1 cut ≥60%; `answerable_rate` and per-category evidence recall unchanged. If recall drops, the multi-query RRF was load-bearing and the fallback is to keep 2 queries concurrently rather than 6 serially.++**T2.3 (M) — Move the sufficiency retry to the deep tier; keep the check as a signal.** Delete the in-path retry (`retrieve.py:967-1012`, stage 6, 24 serial round trips) from `/retrieve/context`. Keep `_context_looks_insufficient` and return its verdict in the response so the caller — or the router — can escalate. Ref: 2510.22344.+*Acceptance:* `sufficiency_retry=true` on the context path is either removed or documented as deep-tier only; the response carries a sufficiency field; the escalation path exists on the MCP surface.+*Verify:* p95 improvement on the subset where the retry currently fires (instrument via the new field); evidence recall on that subset must be recovered by the escalation, not lost. This is the task that resolves G9 and open question 7 together.++**T2.4 (M) — Replace the online dossier walk with precomputed entity gists.** Delete `_run_dossiers` (`retrieve.py:1031-1055`) from the context path. Substitute a per-entity gist maintained incrementally at write time. Refs: 2602.13530 (time-aware gists), 2410.05779 (incremental update), 2501.05366 (compress before prompting).+*Acceptance:* stage 7 issues zero live graph queries on the context path; gists are available for the top seed entities from a single indexed lookup.+*Verify:* the largest single latency win available if T0.1 confirms stage 7 dominates — track p50 and, critically, the 2508/3410 ms *minimum*, which should fall furthest. Per §8 the dossier output never reached the logged context anyway, so the accuracy risk is low; confirm with per-category evidence recall. Gist construction crosses into `01-ingestion-extraction.md`.++**T2.5 (S) — Remove the duplicate spaCy pass and get NLP off the event loop.** `extract_elements` calls `_process_doc` twice (`ner.py:116-117`) and `retrieve.py:754` calls it synchronously on the event loop.+*Acceptance:* one spaCy pass per request, executed in a thread.+*Verify:* stage-1 time halved; no change to variant output (regression test on `_collect_query_variants`).++**T2.6 (S) — As-of-time filtering and recency features.** Add an optional as-of timestamp that filters candidates by validity interval, and expose `happened_at` recency as a ranking feature rather than a prompt string. Ref: 2501.13956.+*Acceptance:* a query with an as-of time excludes facts not valid then; recency is a numeric feature on each candidate.+*Verify:* temporal-category evidence recall and accuracy. Coordinate with `05-temporal-truth.md`, which owns validity semantics; this task consumes them rather than defining them.++> **Checkpoint C2.** **The gate is p50 under 1000 ms with per-category evidence recall at or above the C1 figure.** If instrumented stage times show that is unreachable even after T2.1-T2.5, stop and escalate: report the achieved p50, the residual serial depth, and the measured accuracy cost of the remaining cuts, so the maintainer can relax either the budget or the quality target with numbers in hand. Do not proceed to Phase 3 with the budget unmet — a router that escalates from an already-over-budget path cannot help.++### Phase 3 — Ranking quality within the budget++**T3.1 (M) — Blend PPR with the semantic score.** Change `_score_candidate_with_ppr` (`retrieve.py:554-565`) to a normalised combination, α configurable, default 0.5. Refs: 2405.14831, 2502.14802.+*Acceptance:* α=1.0 reproduces today's ordering exactly and α=0.0 the pure-distance ordering, both unit-tested.+*Verify:* sweep α ∈ {0, 0.3, 0.5, 0.7, 1.0}, reporting per-category evidence recall (deterministic, 1 run each). Judge accuracy only for the best and worst α, 2 runs each. A blend that beats both endpoints on multi-hop evidence recall wins; expect the accuracy difference to be **below the benchmark's resolution**, so decide on the retrieval metric.++**T3.2 (M) — Feature-based learned reranker in the empty filter socket.** Pass a cheap ranker into `filter_relevant_facts` (`retrieve.py:916-921`) over features already computed: vector distance, PPR score, `happened_at` recency, provenance overlap with top passages, variant strength. A linear model or small GBDT, trained on `answers.jsonl` labels. Then lower `max_facts` to 15-20.+*Acceptance:* inference under 5 ms for 50 candidates, no network call, no model server; with `max_facts=15` per-category evidence recall matches `max_facts=50`.+*Verify:* tokens per question down further with recall held; ranker latency reported as its own stage. **This replaces revision 1's cross-encoder recommendation, which `00:22` and `00:25` rule out on this path** — see technique D. If the features prove insufficient, the escalation is a cross-encoder on the *deep* tier, not on this one.++**T3.3 (S) — Query-aware edge weighting, precomputed where possible.** Modulate the materialised adjacency by a query-similarity term plus a `flow_key` symbolic anchor. Ref: 2602.01965.+*Acceptance:* two different queries produce different weight vectors over the same subgraph; the per-query term is O(edges in the seed neighbourhood) with no extra round trip.+*Verify:* multi-hop `evidence_session_recall_full` above T2.1 — CatRAG's "reasoning completeness" proxy — with stage-3 time still under 50 ms.++**T3.4 (S) — Path-shaped output.** Replace `" | "` flattening (`retrieve.py:412-413`, `1057-1058`) with ordered path strings and add a `paths` field. Ref: 2502.14902.+*Acceptance:* every returned fact appears in exactly one path with an explicit position; `text_context` stays byte-compatible until a consumer opts in.+*Verify:* diff old and new `text_context` over 50 questions requiring zero change; then measure with a path-aware answering prompt. Formatting only — the flow-pruning half is deep-tier.++> **Checkpoint C3.** Target: per-category evidence recall at or above C2 with `max_facts` cut ~3x, p50 still under 1000 ms, and paths plus per-triple citations in the response.++### Phase 4 — The deep tier and the router++**T4.1 (M) — Confidence-thresholded entity resolution.** Add an explicit `k` and a similarity threshold to the `search_nodes` calls at `entity_context.py:56-59`, `entity_info.py:170-172`, `controllers/kg.py:31-34`, plus an exact-name tiebreak, and return a confidence score.+*Acceptance:* below-threshold resolution returns "unresolved" rather than the nearest of 10; confidence is in the response.+*Verify:* prerequisite for T4.3 — an agent must not chase a wrong entity (G13, R7).++**T4.2 (M) — Train the router.** Small classifier on `benchmarks/runs/*/answers.jsonl`, predicting whether the context path suffices. With two runs per arm now available, label questions as "one-shot reliable" (correct in both) versus "one-shot unreliable" (0 or 1 of 2) rather than from a single verdict. Ref: 2403.14403.+*Acceptance:* held-out AUC > 0.7; inference under 50 ms so it fits the budget.+*Verify:* offline first — report the accuracy the system would reach if the router's "hard" bucket had oracle-perfect retrieval. **That is the ceiling on everything in Phase 4; if it is under 3 points, stop and do not build the loop.**++**T4.3 (M) — Bounded agentic route on the MCP surface.** Hard step cap, token budget, and gap-analysis stopping. Reuse `traverse_graph` for expansion; keep retrieved paths isolated so one bad path cannot poison the chain. Refs: 2602.13530, 2307.07697, 2510.22344, 2404.14464.+*Acceptance:* terminates within N steps for 100% of a 50-question sample; every answer carries its paths; `search_memory` removed from the agentic path or allowlisted.+*Verify:* per-category accuracy on the router's hard bucket versus the context path on the same bucket, **2 runs per arm, paired test**; report p50/p95 and tokens for that bucket separately so a 10x tail is not hidden inside a headline.++**T4.4 (S) — Write the ADRs.** Fill `003-simple-one-shot-context-api-retrieval.md` and `004-agentic-navigation-retrieval.md` with the routing decision, the conditions in each direction, and each tier's latency and token budget. Cite 2604.09666 for complementarity and `00:22-23` for the budgets.+*Acceptance:* both files state the decision, the alternatives, the measured evidence, and the revisit trigger.++> **Checkpoint C4.** Decide whether the deep route ships on by default for the routed bucket, stays flagged, or is dropped — gated on T4.2's ceiling against T4.3's measured cost.++### Deferred and handed off++Per-entity gist construction (T2.4's write-time half) and Reason-in-Documents-style passage compression (2602.13530, 2410.05779, 2501.05366) belong to `01-ingestion-extraction.md`. The materialised adjacency table (T2.1) belongs to `03-memory-substrate.md`. Validity-interval semantics consumed by T2.6 belong to `05-temporal-truth.md`. ChronoQA-style temporal question types and the `_TEXT_CONTEXT_CAP` truncation that hides the graph channel from the logs (`benchmarks/locomo/evaluate.py:25`, `194`) belong to `04-evaluation-and-applications.md`, along with the finding that answerer non-determinism at `temperature=0` flips 11-12% of questions per run.++---++## Risks++**R1. Phase 2 buys latency by deleting recall sources, and the benchmark cannot see small losses.** T2.2, T2.3 and T2.4 each remove work that might be carrying evidence. Per the protocol, a diffuse 3-5 point accuracy loss is undetectable at any affordable run count. *Detect:* gate every Phase 2 task on per-category `evidence_session_recall_full`, which is deterministic and exact, and never on judge accuracy. *Mitigate:* keep each removal behind a config flag so it can be reverted per request, and re-measure recall after each one individually rather than as a batch.++**R2. The whole evidence base is one conversation, one brain, one answerer, and a corrupted graph.** n=152 on `conv-26`; ±7-point CIs; a 3.3-point run-to-run spread; 11-12% of questions flip with identical retrieval; and per §9 every run was measured on wrongly-deprecated actor edges. *Detect:* T0.3 re-baselines on a clean brain before any conclusion is carried forward; add a second conversation before treating any cross-config delta as general. *Mitigate:* prefer deterministic retrieval-side metrics, which are immune to the answerer noise entirely.++**R3. The trustworthy metrics measure the passage channel, not the graph.** `evidence_session_recall_full` and `answerable_rate` are computed from `retrieved_session_ids`, derived from the 20,000-char-truncated `text_context` plus the passage lists (`evaluate.py:25`, `194`, `201-203`), and the stored `text_context` contains zero fact lines in 152/152 questions. So the metrics I am telling the plan to trust cannot credit or debit the event graph. *Detect:* T0.2's ablation is the only instrument that can, and it needs the truncation raised or the field ordering changed — a harness change owned by `04-evaluation-and-applications.md`. *Mitigate:* until then, treat every claim about the graph channel's value as unverified, including the PPR gain.++**R4. `answerable_rate` is lexical, so the precision-bound diagnosis could still be wrong.** It counts gold tokens appearing in context, which coincidental overlap satisfies. If true recall is materially below 94.1%, Phase 3's precision focus is misdirected. *Detect:* T0.4's hand audit, before Phase 3. *Mitigate:* T0.4 also separates answerer instability from answerer inability, which decides whether the fix is ranking or variance reduction.++**R5. Turning on PPR by default (T1.2) exposes an untested path to all callers, and it currently raises latency.** `personalized_pagerank` is uncommitted, has a dense-fallback branch that redistributes mass to *every* node when a node has no neighbours (`fact_filter.py:104-108`), and is O(nodes × iterations) on the event loop. On a large brain with a big seed set this could be slow or memory-heavy — and T1.2 lands before T2.1 fixes the adjacency. *Detect:* T0.1's stage-3 timer, plus a test on the largest available brain rather than `conv-26`. *Mitigate:* cap adjacency size with a fallback to distance ordering, and consider shipping T1.2 and T2.1 together rather than sequentially.++**R6. Path-shaped output and provenance change the response contract.** Open question 5 is unanswered — I do not know who reads `triples` versus `text_context`. T1.1 additionally *removes* passage text from `text_context`, which is a behaviour change for any consumer relying on it. *Detect:* keep `text_context` byte-compatible in T3.4 and diff over 50 questions; for T1.1, announce the change explicitly since byte-compatibility is not preservable there. *Mitigate:* version the response or gate T1.1's `text_context` change behind a request flag for one release.++**R7. The deep route regresses easy questions if routing is imprecise.** A ToG-style agent over "nearest of 10, always accept" resolution (G13) chases wrong entities confidently, and a router at AUC 0.7 will send genuine single-hop questions there. *Detect:* report per-category accuracy for routed versus not; require the deep route to beat the context path on the routed bucket by more than the resolution floor before it ships on. *Mitigate:* T4.1 is a hard prerequisite for T4.3.++**R8. Phase 2 succeeds on p50 and fails on p95.** The measured p95/p50 ratio is 1.3-1.4, and the tail is where the sufficiency retry and long dossier walks live — exactly what Phase 2 removes, so the tail should improve most. But the router adds a second failure mode: escalations pay both tiers. *Detect:* report p95 and p99 alongside p50 for every Phase 2 task, and report end-to-end latency for escalated queries separately from context-path latency. *Mitigate:* set an explicit escalation budget in T4.4's ADR rather than letting it emerge.diff --git a/docs/research/03-memory-substrate.md b/docs/research/03-memory-substrate.mdnew file mode 100644index 0000000..57cb67a--- /dev/null+++ b/docs/research/03-memory-substrate.md@@ -0,0 +1,717 @@+# 03 — Memory Substrate: data model, indexing, and knowledge lifecycle++Scope per `docs/research/00-scope-and-constraints.md`: the physical model underneath the graph, the vector stores, the text/observation stores, and the lifecycle machinery that is supposed to keep them true and bounded over time. Extraction prompt quality (`01`), query-time ranking (`02`), the benchmark harness (`04`), and temporal *policy* (`05`) are owned elsewhere; this document owns whether the storage layer can physically **express and index** what those workstreams need.++Every implementation claim below carries a `file:line` anchor. Every research claim carries an arXiv ID that was verified against the arXiv metadata API during this review.++---++## What this workstream does++### 1. Backend selection is a runtime switch with three independent axes++`src/config.py:550-552` reads three env vars:++```550:552:src/config.py+ self.vector_db = os.getenv("VECTOR_DB", "milvus")+ self.data_db = os.getenv("DATA_DB", "mongo")+ self.graph_db = os.getenv("GRAPH_DB", "neo4j")+```++`src/core/instances.py:108-153` binds adapters from those values: graph → Neo4j (`instances.py:109-112`) or a NetworkX-over-Postgres client (`instances.py:113-116`); data → Mongo (`instances.py:118-122`) or Postgres (`instances.py:123-127`); vectors → Milvus (`instances.py:144-148`) or pgvector (`instances.py:149-153`).++Two facts follow immediately:++- **The code defaults and the shipped defaults disagree.** `.env.example:144-146` sets `GRAPH_DB="networkx"`, `DATA_DB="postgresql"`, `VECTOR_DB="postgresql"`. So the configuration a new user actually runs exercises an entirely different set of storage code paths from the in-code defaults. Anything measured on one says little about the other.+- **Qdrant is dead code.** `src/lib/qdrant/` contains only `__init__.py`, and no branch in `instances.py` references it. `src/lib/embeddings/local.py` is likewise unwired. Both should be deleted or they will keep implying capabilities that do not exist.++There is also no `else` on those branches. Setting `GRAPH_DB=foo` yields a `GraphAdapter` with `self.graph is None` and the first traversal fails with an `AttributeError`; setting `VECTOR_DB=foo` leaves `vector_store_adapter` undefined and the failure is an import-time `NameError`. Neither is a config error message.++There is also no schema migration framework anywhere — no Alembic, no `migrations/` directory. All DDL is lazy `CREATE TABLE IF NOT EXISTS` executed on first use (`src/lib/postgresql/data.py:148-172`). Any Phase 2 schema change has to bring its own migration mechanism with it.++### 2. The graph model: labels, relationship types, and properties++Nodes are written by `Neo4jClient.add_nodes` (`src/lib/neo4j/client.py:204-306`). Labels come straight from the extractor's `type` string (`src/workers/tasks/ingestion.py:883`, `labels=[node_data.type]`), cleaned by `_clean_labels`. The identity key is `uuid`:++```284:288:src/lib/neo4j/client.py+ cypher_query = f"""+ MERGE (n:{labels_expression} {{uuid: {self._format_value(node.uuid)}}})+ SET {properties_set}+ RETURN n+ """+```++The uuid is content-derived, not random — `stable_node_id` (`src/core/saving/identity.py:13-25`) hashes `(name.lower(), type.lower())`, plus a normalized `happened_at` when the type is `EVENT`. That is the deduplication key: same name + same type = same node. Relationship identity is `stable_relationship_id` over `(tail_uuid, PREDICATE, tip_uuid, flow_key)` (`identity.py:28-42`).++Node properties actually written (`client.py:243-250`): `description`, `happened_at`, `last_updated`, `metadata`, `observations`, `polarity`, plus `name`, `uuid`, and whatever free-form `properties` dict the extractor produced. `description` and the list-valued `source_chunk_ids`/`aliases` get merge semantics rather than overwrite (`client.py:262-277`), which is a genuinely good append-only touch.++Relationships are written by `add_relationship` (`client.py:308-366`), which MERGEs on `uuid` and sets `description`, `v_id`, `flow_key`, and any of `properties`/`happened_at`/`last_updated`/`amount` found on the subject, predicate, **or** object (`client.py:330-350` — note the loop copies attributes from all three objects onto the edge, so a subject's `happened_at` can end up as the edge's `happened_at`).++The event-hub triangle from the README is not encoded in the schema. It is an emergent convention: the extractor is prompted to emit generic relation names (`ArchitectAgentCreateRelationshipTool.py:102` — *"'TARGETED' = correct"*), and the two legs of one hub are tied together by a shared `flow_key` (`identity.py:45-61`, consumed by `get_nexts_by_flow_key` at `client.py:1752-1810` and the two-hop query at `client.py:1670-1672`, which joins on `r2['flow_key'] = r['flow_key']`). There is no `:EventHub` label, no constraint that a hub has exactly one `MADE` in-edge, and nothing prevents the LLM from emitting a relation type that breaks the convention.++Provenance is real but thin. `stamp_provenance` (`identity.py:85-109`) writes `source_chunk_ids` (a merged list) and `source_timestamp` onto both entity and relationship property bags. There is **no confidence, no extractor/model version, and no agent attribution**.++### 3. Tenancy: database-per-brain, on all three backends++This is the strongest part of the substrate. Neo4j uses `database_=brain_id` on every call plus `ensure_database` (`client.py:57-77`), Milvus uses `db_name=brain_id` (`src/lib/milvus/client.py:141-155`), and Postgres provisions `brain_<sanitized>` databases via `brain_db_name` (`src/lib/postgresql/_naming.py:30-49`), with a deterministic SHA-256 suffix when the name would exceed the 63-byte identifier limit. Isolation is physical, not a `WHERE brain_id = ...` predicate, so there is no class of query that can leak across tenants by forgetting a filter.++The cost is operational: every brain is a database. Neo4j Community Edition supports exactly one user database — `ensure_database` swallows `"not supported in community edition"` (`client.py:71`) and silently continues, meaning **on Community Edition every brain shares the `neo4j` database with no `brain_id` predicate anywhere**. That is a real multi-tenant leak on the most likely self-hosted configuration. Milvus similarly caps databases (64 by default), and Postgres pays a connection pool per brain (`_provisioning.get_brain_pool`).++The deeper cost is that isolation now depends entirely on one argument being threaded correctly through every call, with no second line of defence — and it is not. `GraphAdapter.add_nodes` and `execute_operation` both default `brain_id="default"` (`src/adapters/graph.py:145`, `graph.py:122`), and two call sites rely on that default by accident:++```67:67:src/core/agents/tools/kg_agent/KGAgentAddNodesTool.py+ self.kg.add_nodes(nodes, self.identification_params, self.metadata)+```++The signature is `add_nodes(nodes, brain_id, identification_params, metadata)`, so `identification_params` is being passed as the database name. And `src/core/agents/kg_agent.py:121` calls `self.kg.execute_operation(operation)` with no `brain_id` at all, so arbitrary agent-authored Cypher executes against a shared `default` database. Database-per-tenant is the right design, but a silent `"default"` fallback converts every threading mistake into a cross-tenant write instead of an error. See "Guarantees and where they break" §5.++### 4. Vector stores++**Collections.** Five logical stores with independently configured dimensions: `nodes`, `triplets`, `observations`, `data`, `relationships` (`src/constants/embeddings.py:21-27`, fed from `EMBEDDING_*_DIMENSION` env vars at `config.py:321-345`). `.env.example:43-47` sets all five to 3072 (`text-embedding-3-large`).++Four of the five are actually written. **Nothing anywhere calls `add_vectors(..., "observations", ...)`** — the store is declared and provisioned but stays empty, so the agent-written notes in `data_observations` are reachable only by `ILIKE` substring match and by `resource_id` join. Observations are the one part of the substrate designed to hold synthesized insight, and they are the one part with no semantic index.++**Milvus** (`src/lib/milvus/client.py:158-193`): quick-setup `create_collection(store, dimension=..., vector_field_name="embeddings")`, then `AUTOINDEX` with `COSINE`. No partition key, no scalar index on any metadata field, and `search_vectors` (`client.py:218-255`) exposes **no filter parameter at all** — filtered ANN is not possible through this interface.++**pgvector** (`src/lib/postgresql/vectors.py:124-160`): a table per store with `id BIGINT PRIMARY KEY, uuid TEXT, embeddings vector(d), metadata JSONB`, a btree index on `uuid`, and an HNSW index — conditionally:++```60:67:src/lib/postgresql/vectors.py+def _vector_index_ddl(table: str, dimension: int) -> str:+ dim = int(dimension)+ if dim > 2000:+ return ""+ return f"""+ CREATE INDEX IF NOT EXISTS idx_{table}_embeddings+ ON {table} USING hnsw (embeddings vector_cosine_ops);+ """+```++pgvector's HNSW caps at 2000 dimensions, so at the shipped 3072 this returns the empty string and **no vector index is created**. Every `search_vectors` (`vectors.py:205-232`) then runs `ORDER BY embeddings <=> %s LIMIT k` as a full sequential scan. There is no log line, no warning, no metric. See "Guarantees and where they break" §2.++**Embeddings.** `EmbeddingsAdapter` (`src/adapters/embeddings.py:74-140`) retries five times with exponential backoff and then falls back to `ReturnEmptyVectorStrategy` (default, `embeddings.py:77`), which returns `Vector(id=uuid4(), embeddings=[], metadata={})` (`embeddings.py:56-63`). Nothing calls `set_failure_strategy`, so `RaiseEmbeddingFailureStrategy` (`embeddings.py:66-71`) is unreachable in production. The vector id is a fresh `uuid4()` (`embeddings.py:123`), unrelated to the graph node uuid; the graph→vector link is the integer `v_id` stored in node properties (`src/core/saving/ingestion_manager.py:86-89`) and the vector→graph link is `metadata["uuid"]` (`ingestion_manager.py:77-81`). **No embedding model name or version is stored anywhere on the vector.**++### 5. Relational and text stores++`src/lib/postgresql/data.py:55-101` defines four per-brain tables: `data_text_chunks`, `data_observations`, `data_structured_data`, and `data_kg_changes`. Indexes exist on `inserted_at`, `resource_id`, a GIN on `metadata->'labels'`, a GIN on `types`, and `timestamp`. There is **no tsvector/GIN full-text index**; text search is `ILIKE '%...%'` (`data.py:104-106`). `data_kg_changes` is a real append-only change log and is the only durable audit trail in the system.++Redis is always wired (`instances.py:105-106`) and keys are brain-prefixed (`src/lib/redis/client.py:39-43`). Task status is deliberately TTL'd to seven days, but the brain-registry and PAT entries (`src/services/api/middlewares/brains.py:132-136`, `middlewares/auth.py:77-79`) are written with **no TTL and no invalidation on write** — nothing calls `cache_adapter.delete` when a brain or its PAT changes. A revoked token stays valid in cache indefinitely.++### 6. Knowledge lifecycle: what exists, and what is an empty file++Live:++- **Graph consolidation** (`src/core/layers/graph_consolidation/graph_consolidation.py:31-94`): batches of 20 relationships handed to the Janitor agent, which emits normalization/dedup tasks executed by the KG agent. Failures per task are caught and `continue`d (`graph_consolidation.py:89-94`).+- **Supersession** (`src/workers/tasks/ingestion.py:465-505`): marks older same-type edges from the same subject with `invalid_at` and `deprecated=True`.+- **Deprecation via agent tool** (`client.py:1147-1150`, `SET r['deprecated'] = true`).+- **Observations**: an `ObservationsAgent` singleton (`src/services/observations/main.py:11-15`) writing into `data_observations`.++Not implemented — these are files containing only a docstring header:++| File | State |+| --- | --- |+| `src/core/agents/temporal_agent.py` | 9 lines, docstring only. No temporal agent exists. |+| `src/core/backups/backup_creator.py` | `create_backup` is `pass` with a TODO comment list (`backup_creator.py:24-37`); its `-> Backup` annotation references an undefined name, so importing the module raises `NameError`. |+| `src/core/backups/scheduler.py` | 10 lines, docstring only. |+| `src/services/triggers/neighbourhood.py` | 9 lines, docstring only. |+| `src/services/triggers/synthetic_kg.py` | 9 lines, docstring only. |++There is no community detection, no hierarchical summarization, no forgetting or decay, no compaction, and no backup. The append-only graph currently has **no counterweight of any kind**.++---++## Guarantees and where they break++What this layer is trying to guarantee, in my words: *a fact written once is retrievable forever, attributable to its source, distinguishable from the facts that superseded it, isolated from other tenants, and findable in bounded time as the graph grows.* Ranked by damage to answer accuracy and multi-hop recall:++### 1. Supersession invalidates the wrong edges, and in an event-centric graph it is actively destructive — **bug, highest impact**++`_invalidate_superseded_relationships` (`ingestion.py:465-505`) is the entire mechanism by which BrainAPI knows a fact stopped being true. It works like this: fetch the subject's neighbours, and for every edge whose relation name matches the new edge's name but whose other endpoint differs, stamp `invalid_at` and `deprecated=True`.++```481:503:src/workers/tasks/ingestion.py+ for predicate, neighbor in pairs:+ if not predicate or not neighbor:+ continue+ if (predicate.name or "").strip().upper() != (relationship.name or "").strip().upper():+ continue+ if neighbor.uuid == relationship.tip.uuid:+ continue+ ...+ graph_adapter.update_properties(+ predicate.uuid,+ "relationship",+ brain_id=brain_id,+ new_properties={+ "invalid_at": valid_at+ or datetime.datetime.utcnow().strftime("%d/%m/%Y"),+ "deprecated": True,+ },+ )+```++The assumption is functional: *"for a given subject and relation, only one object can be current."* That holds for `(Emily)-[:LIVES_IN]->(city)`. It is exactly wrong for the event-hub model, where relation names are deliberately generic — `MADE`, `TARGETED`, `OCCURRED_WITHIN`. Emily has one `MADE` edge per event she participates in. When she does a second thing, `relationship.name == "MADE"` matches her first `MADE` edge, the neighbour differs (it is a different hub), and **the first event's `MADE` edge is marked deprecated**. Ingest N events by the same actor and N−1 of them lose their actor attribution.++Three amplifiers:++- `get_neighbors` is undirected (`client.py:697`, `MATCH (n)-[r]-(c)`), so incoming edges of the same name are caught too.+- `valid_at` is a `"%d/%m/%Y"` string (`ingestion.py:500`) — no time zone, no ordering, no range query, no Neo4j temporal type.+- There is no `valid_from` on the *new* edge, only `invalid_at` on the old one, so you cannot reconstruct an interval and cannot answer "what was true on date X."++This is the single biggest data-model limitation and it is the direct mechanical cause of the maintainer's stated symptom #2 in `00-scope-and-constraints.md` ("superseded facts come back as current truth") — with the twist that the failure runs in both directions: valid facts get invalidated, and readers mostly ignore the flag anyway (see §3).++### 2. At the shipped configuration there is no vector index — **gap, second highest impact**++`_vector_index_ddl` returns `""` for any dimension above 2000 (`vectors.py:60-63`) and `.env.example:43-47` ships 3072. Every one of the five stores is therefore an unindexed sequential scan under the default install, and the failure is silent. `search_similar_by_ids` (`vectors.py:265-336`) runs one such scan **per source vector in a Python loop** (`vectors.py:306-317`), so a dedup pass over a batch of 20 relationships is 20 full table scans.++The Milvus path does build an index (`milvus/client.py:175-186`) but `AUTOINDEX` with no `m`/`ef_construct`/`ef` tuning and no quantization choice means recall/latency is whatever the server defaults to, unmeasured. Neither backend can filter — `search_vectors` takes no predicate on either path — so any "search only current facts" or "search only nodes of type PERSON" has to be post-filtered after top-k, which silently truncates recall exactly when the filter is selective.++### 3. Readers cannot tell current truth from history++`deprecated`/`invalid_at` is consulted in exactly one place in the entire codebase:++```420:426:src/services/api/controllers/retrieve.py+def _is_currently_valid(predicate: Predicate) -> bool:+ props = getattr(predicate, "properties", None) or {}+ if props.get("invalid_at"):+ return False+ if getattr(predicate, "deprecated", False):+ return False+ return True+```++`get_neighbors`, `list_triples`, `get_event_centric_neighbors`, `get_nexts_by_flow_key`, the hops endpoints, and every MCP tool return deprecated edges undifferentiated. A validity flag that only one of a dozen read paths honours is not a validity model.++### 4. The dual write is vector-first, graph-second, with no compensation++`process_node_vectors` (`ingestion_manager.py:57-94`) commits the vector, then `graph_adapter.add_nodes` (`ingestion.py:911`) commits the graph. If the graph write throws, the vector is already durable and orphaned. There is no outbox, no saga, no reconciliation job, and — because `backup_creator.py` is a stub — no snapshot to diff against.++The reverse leak is worse because it is silent. When an embedding future fails or times out, the loop `continue`s without adding the node to `graph_nodes` (`ingestion.py:900-909`), and then `add_relationship` is called anyway at `ingestion.py:913` referencing that node's uuid. The Cypher is `MATCH (a) WHERE a['uuid'] = ...` (`client.py:354`); no match means zero rows, means the `MERGE` never executes, means **the relationship is dropped with no exception and no error record**. And when the embedding itself fails, `ReturnEmptyVectorStrategy` yields `embeddings=[]`, `process_node_vectors` takes the `else` branch (`ingestion_manager.py:92-93`), prints, and returns the uuid as if nothing happened — so the graph node exists with no vector, invisible to every similarity query, forever, with no repair path.++### 5. Tenant isolation has a silent `"default"` fallback, and two call sites hit it — **bug**++Database-per-brain has no second layer: no query anywhere carries a `brain_id` predicate, so if the wrong database name is threaded in, the write lands in the wrong tenant with no error. `GraphAdapter` defaults that argument to the string `"default"` (`src/adapters/graph.py:122`, `graph.py:145`), which turns a missing argument into a shared database rather than an exception. Two call sites depend on it:++- `src/core/agents/kg_agent.py:121` — `self.kg.execute_operation(operation)`. Agent-authored Cypher, unscoped, against `default`.+- `src/core/agents/tools/kg_agent/KGAgentAddNodesTool.py:67` — `self.kg.add_nodes(nodes, self.identification_params, self.metadata)` against the signature `add_nodes(nodes, brain_id, identification_params, metadata)`. The identification dict is being passed as the database name, so this write is simply broken.++Combined with the Community Edition case in "What this workstream does" §3, isolation is currently a convention enforced by argument order.++### 6. Milvus primary keys are not stable across processes — **bug**++```205:205:src/lib/milvus/client.py+ "id": hash(vector.id) % (2**63),+```++Python salts string hashing per process by default; `PYTHONHASHSEED` is not set anywhere in this repo. The same logical vector therefore gets a different Milvus primary key in each Celery worker and after every restart, so `get_by_ids` and `remove_vectors` miss, and re-inserting a logical vector appends a duplicate row instead of upserting. The deterministic helper is right there in the same file and unused:++```30:32:src/lib/milvus/client.py+def string_to_int64(s: str) -> int:+ sha256 = hashlib.sha256(s.encode()).digest()+ return int.from_bytes(sha256[:8], byteorder="big") % (2**63)+```++The Postgres store uses it correctly (`vectors.py:172`). This is a one-line fix with a backfill caveat.++### 7. No Neo4j index or constraint exists anywhere in the repository++A repo-wide search for `CREATE INDEX` / `CREATE CONSTRAINT` / `db.index.*` returns only the Postgres DDL in `data.py`, `vectors.py`, and `graph_store.py`. Neo4j gets nothing — no uniqueness constraint on `uuid`, no lookup index, no full-text index. Meanwhile the hot queries are label-free scans:++- `client.py:354-355` — `MATCH (a) WHERE a['uuid'] = ...`, twice per relationship write.+- `client.py:530`, `client.py:562`, `client.py:698`, `client.py:1877` — same pattern for reads.+- `client.py:391-395` — `MATCH (n) WHERE toLower(n['name']) CONTAINS toLower(...)`, a full scan with a per-node `toLower` on every text search.++Also note `n['uuid']` (bracket syntax) rather than `n.uuid`: Neo4j's planner will not use a property index for dynamic-key access even if one existed, so adding indexes requires also rewriting the accessors. Every write is currently O(|V|) and every text search is O(|V|) with string work per node.++### 8. Unbounded growth with no compaction and hub-node blowup++Append-only is stated policy (README "Append-only by design") but it is not what the code does — `DELETE n` (`client.py:1890`) and `DELETE r` (`client.py:1929`, `1948`) are both reachable from agent tools, and `add_nodes`/`add_relationship` overwrite properties in place via `SET`. So the graph carries the costs of append-only (unbounded growth) without the guarantee (history is recoverable). The only durable audit trail is `data_kg_changes`.++Meanwhile there is no counterweight to growth: no TTL, no decay, no summarization, no community layer, no archival tier. Two specific pressures:++- **Hub blowup.** A frequently-mentioned entity accumulates one edge per event. `get_neighbors` (`client.py:696-709`) has an optional `limit` but callers such as `_invalidate_superseded_relationships` (`ingestion.py:472-473`) pass none, so every write touching a popular actor materializes that actor's entire neighbourhood. This is quadratic in the number of events per entity. `get_connected_nodes` is worse: it accepts a `limit` parameter (`client.py:846`) that never appears in the Cypher it builds (`client.py:862-871`), so callers believe they are bounded and are not.+- **Vector store growth.** Five stores, one vector per node/relationship/triplet/chunk, no deletion path except the dedup `remove_vectors` at `ingestion.py:849-853`.++There is also no write atomicity to lean on. Every operation is a bare `driver.execute_query`, one auto-commit transaction each; `add_nodes` issues one query per node in a Python loop (`client.py:226-290`). A batch of nodes can therefore be half-written, and `_execute_query_with_retry` is wired into only two methods (`client.py:290`, `1102`) — `add_relationship`, the deletes, and most reads have no retry at all.++### 9. Re-embedding is impossible; changing the model is destructive++Nothing records which model produced a vector, so you cannot identify stale vectors. And if you change `EMBEDDING_NODES_DIMENSION`, `_ensure_store` notices the mismatch and drops the table:++```148:158:src/lib/postgresql/vectors.py+ existing_dim = _table_vector_dimension(cur, table)+ if existing_dim is not None and existing_dim != dimension:+ logger.warning(+ "Recreating %s for brain %s: vector dimension %s -> %s", ...+ )+ cur.execute(f"DROP TABLE IF EXISTS {table} CASCADE")+ cur.execute(ddl)+```++The graph keeps its now-dangling `v_id` properties. There is no re-embed job in `scripts/` or `src/workers/`. Today, a "full re-index" means re-ingesting every source document through the full LLM swarm — the entire extraction cost, again.++### 10. Expressiveness the model does not have++Judged against what `02`/`05` will need:++| Needed | Present? | Evidence |+| --- | --- | --- |+| Validity interval `[valid_from, valid_to)` | No — only `invalid_at` on the superseded edge | `ingestion.py:499` |+| Ingestion time vs. event time (bitemporal) | Partially — `source_timestamp` and `happened_at` exist but are unordered strings on a free-form property bag | `identity.py:104-108`, `client.py:243-250` |+| Provenance to source span | Chunk-level only (`source_chunk_ids`), no offsets | `identity.py:94-102` |+| Confidence / extraction certainty | No | no such field anywhere in `add_nodes`/`add_relationship` |+| Logical negation ("Emily did *not* attend") | No. `polarity` is sentiment good/bad/neutral, not truth value | `src/constants/prompts/scout_agent.py:25-41`, `scout_agent.py:59-61` |+| n-ary facts beyond the 3-slot triangle | Only via the `flow_key` convention; a 4th role has nowhere to go | `identity.py:45-61` |+| Statement-level metadata (who said it, when, how sure) | No reified statement node exists | — |++The `flow_key` convention is a *reification by naming*: the event hub node is the reified statement, and `flow_key` is the correlation id. That is architecturally the right instinct (it is what StarE-style qualifier models formalize) but it is enforced by prompt text, not by schema, and it is invisible to the storage layer.++### Deliberate trade-offs, not bugs++- **Database-per-brain over a `brain_id` column** — costs operational scale, buys leak-proof isolation. Right call.+- **Content-derived uuids** — makes dedup idempotent and re-ingestion safe.+- **Free-form labels and relation types from the LLM** — buys domain generality (the plugin story), costs the ability to index or constrain anything. Reasonable, but it means quality controls have to live in properties, not labels.+- **Merge-on-write for `description`/`aliases`/`source_chunk_ids`** (`client.py:262-277`) — good append-only hygiene, already correct.++---++## Open questions for the maintainer++1. Is the `.env.example` triple (`GRAPH_DB=networkx`, `DATA_DB=postgresql`, `VECTOR_DB=postgresql`) the configuration you consider production, or is the Neo4j/Milvus/Mongo path the one that matters?+2. Is `_invalidate_superseded_relationships` intended to fire on event-hub legs like `MADE`/`TARGETED`, or was it written for functional attribute edges only?+3. Should a superseded fact be excluded from retrieval by default, or returned with a "no longer current" marker so the caller can reason about change?+4. Is Neo4j Community Edition a supported deployment target, given that `CREATE DATABASE` is unavailable there and all brains would then share one database?+5. What is the largest single brain you need to support in the next twelve months, measured in entities and in events per entity?+6. Is `polarity` meant to carry sentiment only, or did you also intend it to express factual negation?+7. Are you willing to add a small fixed vocabulary of reserved relationship types (`MADE`/`TARGETED`/`OCCURRED_WITHIN`) that the extractor must use, so the substrate can index the event-hub shape?+8. Should re-embedding after a model change preserve old vectors under a version tag, or is a destructive rebuild acceptable if the source text is still in `data_text_chunks`?+9. Is `src/lib/qdrant/` intended to become a real backend, or can it be deleted?+10. Do you want confidence to be a stored number per fact, or should uncertainty stay implicit in the Janitor's accept/reject decision?+11. The `observations` vector store is configured but never written to, so agent-written notes are only reachable by substring match. Should observations be embedded and semantically searchable, or are they intentionally a write-only annotation layer?+12. Hard `DELETE` of nodes and relationships is reachable from the agent tools despite the README's "append-only by design". Is deletion a capability you want agents to have, or should it be soft-delete only?++---++## Frontier techniques++### A. Bitemporal edge invalidation with explicit validity intervals++**Mechanism.** Zep/Graphiti stores each edge with four timestamps: when the fact became true and stopped being true in the world (`valid_at`/`invalid_at`), and when the system learned and unlearned it (`created_at`/`expired_at`). New facts are checked against existing edges for contradiction; contradicted edges get `invalid_at` set rather than deleted, so point-in-time queries stay answerable. ATOM sharpens the same idea by splitting documents into minimal self-contained "atomic" facts before building the temporal graph, explicitly separating observation time from validity time.++**arXiv.** Zep: `2501.13956`. ATOM: `2510.22590`. Ontology-driven legal variant with versioned expressions and reified legislative events: `2505.00039`.++**Reported gain.** Zep: 94.8% vs 93.4% on DMR, and up to +18.5 points accuracy on LongMemEval with ~90% latency reduction versus the baseline. ATOM: ~18% higher extraction exhaustivity, ~33% better stability, >90% latency reduction versus baselines for dynamic TKG construction.++**Cost.** Contradiction detection is an LLM call per candidate-conflict at write time. Storage cost is four timestamp properties per edge plus the indexes to range-scan them.++**Fit.** BrainAPI already has the two halves in the wrong shape: `source_timestamp` is an ingestion time and `valid_at` is an event time, but both are unordered strings in a property bag, and the invalidation rule keys on relation-name collision rather than semantic contradiction. Moving to typed temporal properties on the *hub node* rather than on the legs sidesteps the `MADE`-collision bug entirely, because a hub represents one event and is never in competition with another hub.++**Verdict: adopt**, restricted to the physical shape (typed `valid_from`/`valid_to`/`ingested_at`/`retracted_at`, range-indexed). The contradiction *policy* — when to set `valid_to` — belongs to `05-temporal-truth.md`; this workstream's job is to make sure there is somewhere correct to write it.++### B. Continuous-phase temporal rotation instead of hard invalidation++**Mechanism.** RoMem learns a per-relation "volatility" score from the relation's text embedding (fast for `president of`, slow for `born in`) and applies continuous phase rotation in complex vector space, so obsolete facts rotate out of phase and rank below current ones without ever being deleted or flagged.++**arXiv.** `2604.11544`.++**Reported gain.** 72.6 MRR on ICEWS05-15 (state of the art for temporal KG completion); 2–3× MRR and answer accuracy on MultiTQ temporal reasoning; reported to dominate on LoCoMo with zero degradation on static memory (DMR-MSC).++**Cost.** Requires training or importing a Semantic Speed Gate, plus complex-valued embeddings — a second embedding space alongside the existing five stores.++**Fit.** Attractive precisely because it needs *no* LLM call at ingestion, which suits the sub-second context API budget in `00-scope-and-constraints.md`. But it presumes a learned KG embedding space that BrainAPI does not have: today the only vectors are text embeddings of names and descriptions, with no structural training signal.++**Verdict: reject for now, revisit after A.** The volatility idea is separable and cheap, though: a static per-relation-type decay prior, stored as a property, would give most of the ranking benefit for a fraction of the work. Note the paper is very recent (April 2026) with no independent replication.++### C. Hyper-relational / qualifier representation instead of ad-hoc property bags++**Mechanism.** StarE represents a fact as a main triple plus an arbitrary set of `(qualifier_relation, qualifier_entity)` pairs, and message-passes over that structure so qualifiers modulate the main triple's representation without collapsing into it. GRAN models each n-ary fact as a small heterogeneous graph with edge-biased fully-connected attention. VITA extends this to temporal validity as a first-class qualifier type covering all four cases (since / until / period / time-invariant).++**arXiv.** StarE: `2009.10847`. GRAN: `2105.08476`. VITA: `2505.11803`. Inductive extension THOR: `2602.05424`. (HINGE, mentioned in the brief, is a WWW'19 paper — I could not find it on arXiv and am not citing it.)++**Reported gain.** StarE: up to +25 MRR points over triple-only representations on WD50K. VITA: up to 75.3% improvement over the best baselines across link-prediction variants. Ali et al. `2107.04894`: +6 absolute Hits@10 for qualifier-aware inductive link prediction.++**Cost.** These are *embedding* models with training pipelines. Adopting the learning machinery is a large project.++**Fit.** The representational insight transfers without the models. BrainAPI's event hub *is* a reified n-ary fact; what is missing is that the qualifier slots are unnamed. Today a fourth role (instrument, beneficiary, quantity) has to be smuggled into a free-form `properties` dict on one of the legs, where nothing can index or reason over it.++**Verdict: adapt.** Take the data shape — a hub node with typed, named role edges and typed qualifier properties — and skip the embedding models. This is the schema change that makes n-ary facts expressible.++### D. Which reification style to use for statement metadata++**Mechanism.** Standard RDF reification, singleton properties, RDF-star quoted triples, and named graphs are four ways to attach metadata to a statement. Egami et al. compare them head-to-head under a fair evaluation protocol for link prediction.++**arXiv.** `2503.21804`. Foundations of RDF-star: `1406.3399`. Singleton-property reasoning: `1509.04513`. RDF-star → property-graph transformation: `2210.05781`.++**Reported finding.** Reification performs well on simple hyper-relational graphs while singleton property is less effective; on complex hyper-relational graphs the differences between representation models are minimal.++**Cost.** Free — this is a design-choice input, not an implementation.++**Fit.** BrainAPI is a labelled property graph, not RDF, and it *already* reifies (the event hub node). The literature's conclusion — that on complex graphs the choice barely matters, so pick the one your engine indexes best — is a direct argument for keeping the hub-node design and investing in indexing it, rather than migrating to edge properties or a nested representation.++**Verdict: adopt the conclusion, reject a migration.** Keep hub-node reification. Explicitly do not adopt RDF-star.++### E. Community detection and hierarchical summarization as a bounded-growth mechanism++**Mechanism.** GraphRAG runs Leiden community detection over the entity graph and pre-generates a summary per community at each hierarchy level, so global questions are answered from summaries rather than from the raw graph. This is also the only structural answer in the literature to unbounded growth: the raw layer keeps growing, but the queried layer is bounded by the number of communities.++**arXiv.** GraphRAG: `2404.16130`. Core-based deterministic alternative: `2603.05207`. Dual-perception community detection: `2508.19855`. Design-space survey: `2411.05844`. Skeptical counterpoint: `2603.29875`, and the when-does-GraphRAG-help analysis `2506.05690`.++**Reported gain.** GraphRAG: substantial improvements in comprehensiveness and diversity over conventional RAG for global sensemaking on ~1M-token corpora. `2603.05207` proves that on sparse graphs (constant average degree) modularity optimization admits exponentially many near-optimal partitions, making **Leiden communities non-reproducible run to run**, and proposes k-core decomposition for a deterministic, linear-time hierarchy instead.++**Cost.** One LLM summarization call per community per level at index time, and re-clustering on drift. Substantial token cost, but entirely at write time, which is exactly where `00-scope-and-constraints.md` says expensive work belongs.++**Fit.** BrainAPI's graph is sparse and event-centric — precisely the regime where `2603.05207` says Leiden is unstable. A hub node has degree 2–3 by construction.++**Verdict: adapt — use k-core, not Leiden.** Non-reproducible communities are unacceptable for a memory layer that must give the same answer twice. The `flow_key` grouping also gives a free deterministic pre-clustering (one hub = one event), so a hierarchy could be built bottom-up from hubs without any clustering randomness at the leaf level.++### F. Mega-hub mitigation via concept-mediated nodes++**Mechanism.** GAAMA argues that entity-centric graphs over conversational data develop "mega-hub" nodes that dilute relevance propagation, and inserts a concept node layer (episode / fact / reflection / concept, five edge types) so traversal crosses via concepts instead of piling onto hot entities. CatRAG attacks the same failure from the retrieval side, showing that PPR random walks get diverted into high-degree hubs before reaching downstream evidence.++**arXiv.** GAAMA: `2603.27910`. CatRAG: `2602.01965`. HippoRAG (the PPR baseline both build on): `2405.14831`; HippoRAG 2: `2502.14802`.++**Reported gain.** GAAMA: 79.1% mean reward on LoCoMo-10, +4.2 points over a tuned RAG baseline. HippoRAG: up to 20% over prior RAG on multi-hop QA at 10–30× lower cost than iterative retrieval. HippoRAG 2: +7% on associative memory over the state-of-the-art embedding model.++**Cost.** An extra node layer and extra LLM calls to mint concepts at write time.++**Fit.** Direct. BrainAPI *will* develop mega-hubs — the append-only policy plus one edge per event per participant guarantees it, and §8 above shows the write path already scans full neighbourhoods.++**Verdict: adopt the diagnosis; defer the concept layer.** The immediate substrate action is to bound neighbourhood expansion (degree caps, ordered adjacency, indexed `flow_key`) and to measure the degree distribution. Whether the graph needs a concept layer is a question for `02`, and should be answered with a measured degree histogram rather than assumed.++### G. Vector index and quantization choices++**Mechanism.** HNSW gives the best recall/latency balance in memory; DiskANN/Vamana trades to SSD for billion-scale; product and binary quantization compress vectors at a measurable recall cost; Matryoshka Representation Learning trains embeddings so that a truncated prefix is itself a valid embedding, letting you store 512 dimensions for search and 3072 for reranking from one model.++**arXiv.** Matryoshka: `2205.13147`. In-place streaming DiskANN updates: `2502.13826`. Filtered ANN across FAISS/Milvus/pgvector: `2602.11443`. Low-selectivity filtered search: `2607.00768`. Binary-quantization-native topology and its limits: `2605.02171`. Declarative recall targets: `2505.19001`. ColBERT late interaction: `2004.12832`, ColBERTv2: `2112.01488`, PLAID: `2205.09707`.++**Reported gain.** Matryoshka: up to 14× smaller embeddings at equal accuracy and up to 14× real-world retrieval speedup. `2602.11443` finds Milvus achieves superior recall stability through hybrid approximate/exact execution while pgvector's cost-based optimizer frequently picks suboptimal plans, and that IVFFlat beats HNSW at low selectivity. `2605.02171` finds binary-quantization-native topology works well on cosine-native contrastive embeddings (≥88% Recall@10) but collapses (<15%) on Euclidean-native data — a falsifiable applicability boundary. `2502.13826` (IP-DiskANN) keeps recall stable under long insert/delete streams where batch-consolidation approaches degrade.++**Cost.** Matryoshka requires an MRL-trained model; `text-embedding-3-large` supports dimension truncation natively, so this is a config change for the current default provider and a model change for Ollama/local users.++**Fit.** Immediate and cheap. Truncating to ≤2000 dimensions makes the pgvector HNSW index legal *today* and directly fixes §2. `2602.11443` is worth reading before choosing between Milvus and pgvector at scale.++**Verdict: adopt — this is the highest gain-per-hour item in the document.** Reject binary quantization for now: `2605.02171`'s boundary condition is exactly the kind of thing that fails silently on a mixed corpus.++### H. Governed evolving memory: making lifecycle a first-class workload++**Mechanism.** Orogat & Mansour argue that agent memory is a distinct data-management workload whose correctness is a property of the *state trajectory*, not of individual records, and formalize four state-level operators (ingestion, revision, forgetting, retrieval) under six correctness conditions. They name four recurring failure modes: unregulated growth, missing semantic revision, capacity-driven forgetting, and read-only retrieval. Uddin et al. contribute Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory.++**arXiv.** GEM/MemState: `2605.26252`. Memora/FAMA: `2604.20006`. Storage-time provenance and versioning as a security requirement: `2604.16548`. Placement of the control plane: `2606.15903`.++**Reported finding.** `2604.20006` evaluates four LLMs and six memory agents and finds *frequent reuse of invalid memories and failures to reconcile evolving memories*, with memory agents offering only marginal improvements. `2606.15903` finds production failures are predominantly forgetting failures rather than recall failures, yet existing benchmarks measure only recall.++**Cost.** Conceptual adoption is free. FAMA needs labelled obsolescence in the eval set, which is a `04` dependency.++**Fit.** BrainAPI exhibits three of the four named failure modes verbatim: unregulated growth (§8), missing semantic revision (§1), and read-only retrieval (§3). The one it avoids is capacity-driven forgetting — because it never forgets.++**Verdict: adopt the framing and the metric.** This is the argument for why substrate work ranks above ranking work: a validity flag nothing reads and a growth curve nothing bounds are not ranking problems.++### I. Confidence and provenance modelling++**Mechanism.** ProVe automatically verifies that a KG triple is supported by the text at its documented provenance URL. Adobe's KG-RAG pipeline assigns confidence scores to entity-relation pairs and filters to high-confidence pairs, and links every fact to its source document.++**arXiv.** ProVe: `2210.14846`. Adobe KG-RAG: `2502.15237`. Provenance semirings for query-result probabilities: `2108.07758`. Epistemic stance over attributed claims: `2606.15246`. LLM confidence miscalibration and grouping loss: `2402.04957`.++**Reported gain.** ProVe: 87.5% accuracy / 82.9% macro-F1 detecting provenance support on text-rich sources. Adobe: >50% reduction in irrelevant answers and +88% fully-relevant answers versus their production system, from a combination of incremental entity resolution, similarity dedup, confidence filtering, and source linking.++**Cost.** A confidence float per node and edge is nearly free to store. Populating it well is not — and `2402.04957` shows LLM self-reported confidence is systematically overconfident and unevenly so across subpopulations, so a raw LLM score would be actively misleading.++**Fit.** BrainAPI already has the provenance half (`source_chunk_ids`, `source_timestamp`) and can therefore afford the cheap, well-calibrated version: count distinct supporting chunks. A fact seen in five independent chunks is more reliable than one seen once, and that number is already computable from `source_chunk_ids`.++**Verdict: adapt.** Store `support_count` (derived, calibrated by construction) rather than an LLM-reported confidence. Reject asking the LLM for a probability.++### Explicitly rejected++| Technique | Why not here |+| --- | --- |+| **RDF-star / named-graph migration** (`1406.3399`, `2211.16195`) | BrainAPI is an LPG. `2503.21804` finds representation choice barely matters on complex hyper-relational graphs. All cost, no measured gain. |+| **Learned hyper-relational KG embeddings** (`2602.05424`, `2605.24064`, `2306.02199`) | Transductive or requiring a training pipeline over a vocabulary that changes on every ingest. BrainAPI's entity vocabulary is open and per-tenant. |+| **Binary-quantization-native ANN** (`2605.02171`) | The paper's own applicability boundary excludes Euclidean-native and "structureless" distributions, and BrainAPI cannot characterize its embedding distribution today. Revisit after recall is measured. |+| **Parametric / KV-cache knowledge injection** (`2510.17934`) | Requires model-side integration and precludes per-tenant isolation. Incompatible with database-per-brain. |+| **Optical / resolution-based forgetting** (`2605.03804`) | Multimodal, on-device framing. BrainAPI's pressure is graph edges, not image bytes. |+| **Leiden-based communities** (`2404.16130` as-implemented) | `2603.05207` proves non-reproducibility on sparse graphs. Use k-core (technique E). |++---++## Implementation plan++Ordered by dependency and by gain per unit of cost. Phases 1 and 2 are correctness and are worth more than everything after them. Nothing here changes retrieval ranking — that is `02`.++**Baseline needed before Task 1.** Every acceptance criterion below that mentions latency or recall assumes a captured baseline. If `benchmarks/` cannot produce one, Task 0 becomes a blocker.++### Phase 1 — Stop the bleeding++#### Task 1: Make Milvus primary keys deterministic++**Description:** Replace `hash()` with the existing `string_to_int64` in `MilvusClient.add_vectors`, and add a one-off script that rewrites existing rows to their deterministic ids.++**Acceptance criteria:**+- [ ] `add_vectors` uses `string_to_int64(vector.id)`; `hash()` no longer appears in `src/lib/milvus/`.+- [ ] Inserting the same `Vector` twice from two separate Python processes yields one row, not two.+- [ ] A migration script maps existing rows to new ids or documents that a rebuild is required.++**Verification:** `poetry run pytest tests/ -k milvus`; manual — insert a vector, restart the worker, `get_by_ids` with the same logical id returns it.++**Dependencies:** None. **Files:** `src/lib/milvus/client.py`, `scripts/`. **Scope:** S.++#### Task 2: Give the default configuration a working vector index++**Description:** The 3072-dimension default silently disables HNSW on pgvector. Make the store request a dimension the index supports, and make the unindexed case loud.++**Acceptance criteria:**+- [ ] `_vector_index_ddl` logs an explicit error (not silence) when it declines to build an index.+- [ ] `.env.example` ships a dimension ≤ 2000 for all five stores, using Matryoshka truncation (`2205.13147`) rather than a different model.+- [ ] `EXPLAIN ANALYZE` on `search_vectors` shows an index scan on a fresh default install.+- [ ] A recall@10 comparison of truncated vs. full-dimension embeddings on the existing benchmark set is recorded in the PR.++**Verification:** `EXPLAIN ANALYZE SELECT ... ORDER BY embeddings <=> ...` shows `Index Scan using idx_vectors_nodes_embeddings`; benchmark latency for `/retrieve/context` recorded before and after.++**Dependencies:** None. **Files:** `src/lib/postgresql/vectors.py`, `.env.example`, `src/lib/embeddings/client*.py`. **Scope:** M.++#### Task 3: Scope supersession so it cannot invalidate event-hub legs++**Description:** `_invalidate_superseded_relationships` treats `MADE` as a functional relation. Restrict it to an explicit allow-list of functional relation types, defaulting to empty, and never let it fire on an edge whose endpoint carries an `EVENT` label or a `flow_key`.++**Acceptance criteria:**+- [ ] Ingesting three events by the same actor leaves all three `MADE` edges with `deprecated` unset.+- [ ] A functional-attribute change (same subject, same relation, new object, relation in the allow-list) still marks the old edge `invalid_at`.+- [ ] `get_neighbors` is called with a `limit`, or the function is rewritten to a targeted single query rather than a full neighbourhood fetch.++**Verification:** New test in `tests/` asserting both branches; `poetry run pytest tests/ -k supersede`.++**Dependencies:** None. **Files:** `src/workers/tasks/ingestion.py`, `tests/`. **Scope:** S.++#### Task 4: Fail loudly when a node is written without its vector++**Description:** Three silent-loss paths converge: `ReturnEmptyVectorStrategy` swallowing embedding failures, `process_node_vectors` returning success with no vector, and `add_relationship` silently no-op'ing when its endpoint MATCH finds nothing.++**Acceptance criteria:**+- [ ] An embedding failure produces an entry in the task's `item_errors`, not just a `print`.+- [ ] `add_relationship` checks the driver result and raises when zero rows were returned.+- [ ] A node written without a vector is recorded in `data_kg_changes` so a repair job can find it.++**Verification:** Inject an embedding-provider failure in a test and assert the ingestion task reports `partial_failed`; `poetry run pytest tests/ -k ingestion`.++**Dependencies:** None. **Files:** `src/adapters/embeddings.py`, `src/core/saving/ingestion_manager.py`, `src/lib/neo4j/client.py`, `src/workers/tasks/ingestion.py`. **Scope:** M.++#### Task 5: Remove the `"default"` brain_id fallback++**Description:** With database-per-tenant and no `brain_id` predicate in any query, a defaulted `brain_id` is a cross-tenant write. Make it a required argument and fix the two call sites that currently rely on the default.++**Acceptance criteria:**+- [ ] `brain_id` has no default value on any `GraphAdapter`, `DataAdapter`, or `VectorStoreAdapter` method; omitting it is a `TypeError` at call time.+- [ ] `KGAgentAddNodesTool` passes `brain_id` in the correct position.+- [ ] `KGAgent.execute_operation` threads the caller's `brain_id`.+- [ ] A grep for `brain_id: str = "default"` in `src/` returns nothing.++**Verification:** `poetry run pytest tests/`; a test asserting that calling `add_nodes` without `brain_id` raises rather than writing.++**Dependencies:** None. **Files:** `src/adapters/graph.py`, `src/adapters/data.py`, `src/adapters/embeddings.py`, `src/core/agents/kg_agent.py`, `src/core/agents/tools/kg_agent/`. **Scope:** S.++### Checkpoint: Phase 1++- [ ] Full test suite passes.+- [ ] A default-config install shows an index scan for vector search.+- [ ] The three-events-one-actor scenario preserves all attribution.+- [ ] No code path can write to a brain database without an explicit `brain_id`.+- [ ] `/retrieve/context` p95 measured and recorded against the pre-Phase-1 baseline.+- [ ] **Review with the maintainer.** Open questions 1, 2, and 4 must be answered before Phase 2, because the target schema depends on which backend is canonical.++### Phase 2 — Target schema++The proposed diff. Additive; nothing is removed, so old data stays readable.++```+EventHub node (currently: an ordinary node whose legs share a flow_key)++ :EventHub label, reserved, set by the writer not the LLM++ flow_key string promoted from edge property to node property, indexed++ valid_from datetime Neo4j temporal type; when the fact became true in the world++ valid_to datetime? null = still current; set on supersession++ ingested_at datetime when BrainAPI learned it++ retracted_at datetime? when BrainAPI stopped believing it (audit, not world-time)++ support_count int count of distinct source_chunk_ids; derived, recomputed on merge++ negated boolean default false; true = "this event did NOT happen"++ embedding_model string model id that produced the linked vectors++Role edges (currently: free-form LLM-chosen names)++ reserved types MADE | TARGETED | OCCURRED_WITHIN | QUALIFIED_BY+ LLM-chosen names remain legal but are no longer the mechanism for the triangle++ role string on QUALIFIED_BY: instrument | beneficiary | quantity | ...+ this is the n-ary slot the model currently lacks (technique C)++Entity node++ degree_hint int maintained edge count, for hub detection (technique F)++Neo4j indexes (currently: none exist anywhere in the repo)++ CONSTRAINT uuid uniqueness per label++ INDEX :EventHub(flow_key)++ RANGE INDEX :EventHub(valid_from), :EventHub(valid_to)++ FULLTEXT INDEX name, description -- replaces the toLower(...) CONTAINS scan++pgvector++ GIN index metadata jsonb_path_ops -- makes vector→graph lookup by node uuid indexable++ column embedding_model text -- makes selective re-embedding possible+```++What this unlocks, concretely: point-in-time queries (`valid_from <= t < valid_to`) become an index range scan instead of impossible; the `MADE` collision bug becomes structurally impossible because validity lives on the hub, and two hubs are never in competition; a fourth participant in an event has a typed home; text search stops being a full scan; and re-embedding becomes a filtered update instead of a table drop.++#### Task 6: Reserved labels and relationship types++**Description:** The writer, not the LLM, stamps `:EventHub` and the three reserved role types when the extractor's output matches the triangle shape. Existing free-form names keep working.++**Acceptance criteria:**+- [ ] Newly ingested hub nodes carry `:EventHub` and a node-level `flow_key`.+- [ ] The two legs of one hub use reserved types; a mismatch is logged, not silently accepted.+- [ ] Existing data without `:EventHub` still returns from all read paths.++**Verification:** `poetry run pytest tests/ -k event_hub`; Cypher spot-check that `MATCH (h:EventHub) RETURN count(h)` is non-zero after an ingest.++**Dependencies:** Task 3, open question 7. **Files:** `src/workers/tasks/ingestion.py`, `src/core/saving/identity.py`, `src/lib/neo4j/client.py`. **Scope:** M.++#### Task 7: Neo4j indexes and constraints, with accessor rewrite++**Description:** Add a startup migration creating the constraints and indexes above. This requires rewriting `n['uuid']` to `n.uuid` in the hot queries, because the planner will not use a property index for dynamic-key access.++**Acceptance criteria:**+- [ ] A startup migration is idempotent and runs per brain database.+- [ ] `PROFILE` on the uuid lookup at `client.py:354` shows `NodeUniqueIndexSeek`, not `AllNodesScan`.+- [ ] `node_text_search` uses the fulltext index instead of `toLower(...) CONTAINS`.+- [ ] Community Edition, where `CREATE DATABASE` is unavailable, is either supported explicitly or refused at startup with a clear message.++**Verification:** `PROFILE MATCH (a) WHERE a.uuid = '...' RETURN a` in the Neo4j browser; ingest-throughput benchmark before and after on a graph of ≥50k nodes.++**Dependencies:** Task 6, open question 4. **Files:** `src/lib/neo4j/client.py`, a new migration module — note that no migration framework exists today, so this task establishes one. **Scope:** M.++#### Task 8: Typed temporal properties on the hub++**Description:** Write `valid_from`/`valid_to`/`ingested_at`/`retracted_at` as Neo4j temporal types on `:EventHub`, replacing the `"%d/%m/%Y"` string. Backfill by parsing existing `happened_at`/`source_timestamp` where parseable.++**Acceptance criteria:**+- [ ] A point-in-time query returns only hubs valid at that instant and uses the range index.+- [ ] Unparseable legacy dates are left null and counted, not guessed.+- [ ] Setting `valid_to` on supersession writes to the hub, not to a leg.++**Verification:** `poetry run pytest tests/ -k temporal`; a fixture asserting that a superseded and a current fact are distinguishable by query alone.++**Dependencies:** Tasks 6, 7. **Files:** `src/lib/neo4j/client.py`, `src/workers/tasks/ingestion.py`, `src/core/saving/identity.py`. **Scope:** M.++#### Task 9: Make validity visible to every reader++**Description:** `_is_currently_valid` exists in one controller. Push the filter into the graph client as an opt-out parameter so `get_neighbors`, `list_triples`, `get_event_centric_neighbors`, and the hops queries all honour it by default.++**Acceptance criteria:**+- [ ] Every read method accepts `include_superseded: bool = False`.+- [ ] With the default, a superseded fact does not appear in any read path.+- [ ] With `include_superseded=True`, the fact returns with its `valid_to` populated so a caller can render history.++**Verification:** `poetry run pytest tests/ -k validity`; a test that ingests a fact, supersedes it, and asserts every read method's default excludes it.++**Dependencies:** Task 8. **Files:** `src/lib/neo4j/client.py`, `src/lib/postgresql/networkx_client.py`, `src/adapters/graph.py`, `src/services/api/controllers/retrieve.py`. **Scope:** M.++### Checkpoint: Phase 2++- [ ] Point-in-time query works and is index-backed.+- [ ] Superseded facts are invisible by default and retrievable on request from every read path.+- [ ] Ingest throughput has not regressed more than 10% (indexes cost write time).+- [ ] `/retrieve/context` still meets its sub-second budget.+- [ ] **Review with the maintainer** before Phase 3.++### Phase 3 — Bounded growth and reversibility++#### Task 10: Record the embedding model with every vector++**Description:** Add `embedding_model` to vector metadata and to the pgvector schema, populated from config at write time.++**Acceptance criteria:**+- [ ] Every newly written vector carries the model id.+- [ ] A query can count vectors per model per store.+- [ ] `_ensure_store` no longer drops a table on dimension change; it refuses and points at the re-embed job.++**Verification:** `SELECT embedding_model, count(*) FROM vectors_nodes GROUP BY 1`; a test that a dimension change raises rather than deletes.++**Dependencies:** Task 2. **Files:** `src/lib/postgresql/vectors.py`, `src/lib/milvus/client.py`, `src/core/saving/ingestion_manager.py`. **Scope:** M.++#### Task 11: Re-embed and reconcile job++**Description:** A Celery task that (a) re-embeds vectors whose `embedding_model` differs from current config, sourcing text from the graph and `data_text_chunks`, and (b) reports graph nodes with no vector and vectors with no graph node.++**Acceptance criteria:**+- [ ] Re-embedding a brain of 10k nodes completes without dropping a table and without a full LLM re-extraction.+- [ ] The reconcile report lists orphan counts in both directions.+- [ ] The job is resumable and idempotent.++**Verification:** Change the model in a test brain, run the job, assert `SELECT count(*) WHERE embedding_model = <old>` is zero and node count is unchanged.++**Dependencies:** Task 10. **Files:** `src/workers/tasks/`, `scripts/`. **Scope:** M.++#### Task 12: Degree instrumentation and hub bounds++**Description:** Before designing a concept layer (technique F), measure. Emit the entity degree distribution and put a hard `limit` on every unbounded `get_neighbors` call site.++**Acceptance criteria:**+- [ ] A script prints p50/p95/p99/max entity degree for a brain.+- [ ] No `get_neighbors` call site in `src/` passes `limit=None`.+- [ ] The degree histogram for the LoCoMo benchmark brain is recorded in this document.++**Verification:** Run the script against the benchmark brain; grep confirms no unbounded call sites.++**Dependencies:** Task 7. **Files:** `scripts/`, `src/workers/tasks/ingestion.py`, `src/core/search/`. **Scope:** S.++#### Task 13: Deterministic k-core hierarchy over event hubs++**Description:** A write-time job building a k-core hierarchy (`2603.05207`) over the entity graph, with one summary per group. Deterministic by construction — do not use Leiden.++**Acceptance criteria:**+- [ ] Running the job twice on the same graph produces byte-identical group membership.+- [ ] Group summaries are stored as nodes with edges to their members.+- [ ] The job is incremental — a new ingest does not require a full rebuild.++**Verification:** Run twice, diff the membership table; assert equality.++**Dependencies:** Tasks 7, 12, and a decision from `02` on whether retrieval will use it. **Scope:** M.++#### Task 14: Backup and restore++**Description:** Implement `create_backup` — currently `pass` — across graph, vectors, text/structured data, and observations, as a resumable Celery task with progress in Redis.++**Acceptance criteria:**+- [ ] A backup of one brain restores into an empty instance with identical node, edge, and vector counts.+- [ ] The module imports without `NameError`.+- [ ] Progress is observable while running.++**Verification:** Backup then restore a benchmark brain; assert counts and spot-check ten facts.++**Dependencies:** Task 10. **Files:** `src/core/backups/`, `src/workers/tasks/`. **Scope:** M.++### Checkpoint: Phase 3++- [ ] A model change is a job, not a data-loss event.+- [ ] Degree distribution is measured and recorded.+- [ ] Backup/restore round-trips.+- [ ] **Review with the maintainer.**++### Parallelization++Tasks 1 through 5 are independent and can run concurrently. Tasks 6→7→8→9 are a strict chain. Tasks 10 and 12 can run alongside the Phase 2 chain. Task 13 must wait on `02`'s decision about whether retrieval consumes a hierarchy.++---++## Risks++| Risk | Impact | How it shows up | Mitigation |+| --- | --- | --- | --- |+| Truncating embeddings to ≤2000 dims loses recall | High | Benchmark recall@k drops after Task 2 | Measure recall before and after in the same PR; Matryoshka models are designed for this (`2205.13147`) but the local/Ollama models in `.env.example` are not MRL-trained and must be checked separately |+| Fixing the Milvus key breaks existing deployments | High | Post-deploy, `get_by_ids` misses every pre-existing vector | Ship the migration script with the fix; if a deployment cannot migrate, document a rebuild as the supported path |+| Neo4j indexes slow ingestion | Medium | Ingest throughput regresses after Task 7 | Benchmark ingest before/after; a uniqueness constraint on `uuid` is cheap, a fulltext index is not free |+| Restricting supersession leaves genuinely stale facts current | Medium | Temporal benchmark accuracy drops after Task 3 | Task 3 narrows an over-firing rule; the replacement policy is `05`'s deliverable and must land before Phase 2 closes, or the system trades one temporal failure for another |+| Reserved labels conflict with existing extractor output | Medium | Nodes get both `:EventHub` and an LLM-chosen `:EVENT` label | Additive only; never remove an LLM label; assert old read paths still work in Task 6's tests |+| Community Edition users silently lose tenant isolation | High | Two brains' data interleaves in one database | Tasks 5 and 7 together must either refuse startup or add a `brain_id` predicate everywhere; this is a correctness and possibly a compliance issue, not a performance one (see open question 4) |+| Making `brain_id` required breaks callers found only at runtime | Medium | An MCP tool or agent path raises `TypeError` in production after Task 5 | Grep every call site before landing; the alternative — leaving the default in place — is a silent cross-tenant write, so a loud break is the better failure mode |+| The whole plan optimizes the wrong backend | High | Work lands on Neo4j while users run NetworkX-over-Postgres | Open question 1 gates Phase 2; every Phase 2 task must be mirrored in `src/lib/postgresql/networkx_client.py` or explicitly scoped out |+| Degree instrumentation reveals the real problem is elsewhere | Low | Hub degree turns out to be small | Good outcome — Task 12 is cheap precisely so that Task 13 and technique F are not built on assumption |++---++## What I could not determine++- Whether the Mongo data client is exercised by anyone; it is wired (`instances.py:118-122`) and 501 lines long, but `.env.example` selects Postgres.+- Actual production graph sizes. Every scale claim here is analytical (query shape × index absence), not measured, because no profiling data exists in the repo. Task 12 exists to replace that gap with numbers.+- Whether the Milvus `AUTOINDEX` resolves to HNSW or IVF in the maintainer's deployment — it depends on the Milvus version and deployment mode, and nothing pins it.diff --git a/docs/research/04-evaluation-and-applications.md b/docs/research/04-evaluation-and-applications.mdnew file mode 100644index 0000000..7260eac--- /dev/null+++ b/docs/research/04-evaluation-and-applications.md@@ -0,0 +1,673 @@+# Evaluation, Measurement, and Downstream Application Surfaces++Workstream 04. Scope and constraints: `docs/research/00-scope-and-constraints.md`.++Every implementation claim below is anchored to `file:line` as read at commit `649f5e5` plus the uncommitted working-tree changes to `benchmarks/locomo/`. Every research claim carries an arXiv ID. Where I could not determine something, I say so.++A note on freshness: while I was reading, run `locomo-conv26-push75-d` was executing against a working tree whose answer prompt had been edited minutes earlier. That run turned out to be the most informative artifact in the repository, for reasons in section 2.1.++## What this workstream does++### The harness, end to end++`benchmarks/locomo/` is a self-contained CLI that drives the public BrainAPI HTTP surface. It never imports `src/`, so it measures the deployed server as a black box.++**Command surface** (`benchmarks/locomo/cli.py`): `download`, `stats`, `smoke`, `ingest`, `answer-once`, `selftest-metrics`, `evaluate`, `report`.++**Dataset** (`benchmarks/locomo/dataset.py`). `locomo10.json` is fetched from `snap-research/locomo` (`config.py:12-15`), 10 conversations, 1,986 QA pairs. Turns are flattened to `f"{speaker} ({dia_id}): {text}"` (`dataset.py:76`) and each session is prefixed with its wall-clock time and a synthetic `Session id: {session_key}.` line (`dataset.py:87`). That injected session id is what evidence-recall scoring later keys on.++**Ingestion** (`benchmarks/locomo/ingest.py`). One unit per session, submitted to the observation endpoint, then polled to a terminal status (`config.py:20`). Per-unit records capture `submit_latency_ms` and `wait_latency_ms` (`ingest.py:22-36`). They do not capture tokens, extraction counts, or cost. `client.py:92-100` hardcodes the ingestion directive (`observate_for`, `preferred_extraction_entities` = Person/Event/Location/Date), so ingestion is configured for LoCoMo in code, not in the manifest.++**Retrieval + answer** (`benchmarks/locomo/client.py`, `answer.py`). `retrieve_context` posts the question to `/retrieve/context` with `historical_limit`, `max_passages`, `max_facts`, `apply_fact_filter`, `use_ppr`, `sufficiency_retry`. `answer.py:42-46` calls the answer model at `temperature=0` with no seed; `answer.py:54-79` retries once if the model returns empty.++**Judging** (`benchmarks/locomo/judge.py`, `prompts.py`). `judge_answer` (`judge.py:56-87`) sends `JUDGE_SYSTEM` (`prompts.py:20-25`) with a user message containing exactly three fields: question, gold, prediction (`prompts.py:86-90`). `temperature=0`, `response_format=json_object` (`judge.py:68-73`). The judge returns `{correct: bool, reason: str}`.++**Scoring** (`benchmarks/locomo/evaluate.py`, `metrics.py`). Per question the harness writes an `AnswerRecord` (`evaluate.py:30-57`) that is unusually rich: prediction, gold, category, evidence, judge verdict and raw payload, F1, BLEU-1, three separate latencies, both model names, answer and judge token counts, and the full retrieved context (`text_context` capped at 20,000 chars, `evaluate.py:25,194`), source passages, historical context, and the session ids parsed out of them. Because the context blob is persisted per row, most of the offline analysis in section 2 is possible after the fact.++`metrics.aggregate_answers` (`metrics.py:154-245`) computes, per category and overall: judge accuracy with a Wilson 95% interval (`metrics.py:79-90`), mean F1 and BLEU-1, `answerable_rate`, evidence-session recall at full and partial, `answerer_gap`, retrieval latency p50/p95/mean, and `total_llm_tokens`.++**Reporting** (`benchmarks/locomo/report.py`). `build_report` filters out errored rows (`report.py:34`), aggregates, and renders `report.md` plus `report.json` with a headline, a per-category table, latency, tokens, and ingest status.++### What the instrumentation already gets right++This is a better-instrumented harness than most. Specifically:++- **Per-category breakdown exists** and uses the real LoCoMo taxonomy: 1 multi-hop, 2 temporal, 3 open-domain, 4 single-hop, 5 adversarial (`config.py:21-27`), surfaced in the report table (`report.py:105-118`).+- **Confidence intervals exist** on every accuracy figure (`metrics.py:175,185-187`). Point estimates are not reported bare.+- **A retrieval ceiling proxy exists.** `gold_in_context` (`metrics.py:133-139`) checks whether at least half the gold tokens longer than two characters appear anywhere in the retrieved blob, and is reported as `answerable_rate`.+- **A first attempt at failure attribution exists.** `answerer_gap = answerable_rate − judge_accuracy` (`metrics.py:211-219`) is intended to separate "memory did not supply it" from "the answerer had it and still failed."+- **Cost and latency sit next to accuracy** in the same report (`report.py:120-129`), which is exactly the right instinct.+- **`selftest-metrics`** (`metrics.py:248-275`, `cli.py:143-150`) unit-tests the metric functions themselves, including a `gold_in_context` and an `evidence_coverage` case.++So the gaps below are not gaps of effort or of instinct. They are gaps of *validity*: the harness measures many things carefully, and almost none of those measurements are currently defended against the ways they can be wrong.++### The unmeasured application surfaces++**Synergies / recommendations.** `EntitySinergyRetriever.retrieve_sibilings` (`src/core/search/entity_sibilings.py`) is reached from `GET /retrieve/entity/synergies` (`src/services/api/routes/retrieve.py:501-526`) via `get_entity_sibilings` (`src/services/api/controllers/entities.py:83-131`). Scoring weights are module-level constants (`entity_sibilings.py:26,29,32,35`: `DIRECT_MULTIPLIER=1.30`, `REMOTE_MULTIPLIER=0.70`, `FACTORS_INCREMENTAL_WEIGHT=0.3`, `NODE_SIM_DESC_INCREMENTAL_WEIGHT=0.5`). The `polarity` argument is declared and documented (`entity_sibilings.py:53,65`) but never read in the method body — the `same`/`opposite` distinction the route advertises (`routes/retrieve.py:515`) does not exist in the implementation. The return is unbounded (`entity_sibilings.py:437`). There is no benchmark, no test, and no metric for this endpoint.++**MCP tool surface.** Five tools (`src/services/mcp/main.py:168,231,272,303,328`). Authorization failures return the bare string `"Unauthorized"` (`main.py:212,266,288,321`) rather than a structured error, so a calling agent cannot distinguish "not allowed" from "no results" without string matching. There is no benchmark for whether an agent equipped with these tools completes tasks better.++**Observability.** Tracing is a bespoke in-process tracker (`src/lib/tracing/tracker.py`), enabled by default (`tracker.py:83`). `start_subscribers` (`tracker.py:353`) is not invoked on the production path, so events stay in an in-memory queue and are dropped when it fills. Spans emit on breach, not on success, so the happy path is invisible. Token usage is logged to stdout only (`src/core/agents/core/invoke_loop.py:51`), never attached to a trace or aggregated. Retrieval and synergies are not instrumented at all.++**Tests as a gate.** 25 files under `tests/`, all `unittest`, no `conftest.py`, no coverage configuration in `pyproject.toml`. No workflow runs them: `staging.yaml:15` is literally `- run: "[ NOT IMPLEMENTED YET ]"`, and `deploy.yaml` / `tags.yaml` build and ship without a test step. There is no quality gate in CI today.++## Guarantees and where they break++The guarantee this workstream is trying to make: **a number in `report.md` moving up means the memory layer got better, and a number moving down means it got worse.** Everything else — cost tracking, attribution, CI gating — exists to make that inference safe.++Here is the recorded history, recomputed from `answers.jsonl` rather than taken from `report.json`, so that duplicate and errored rows are visible:++| run | scored rows | unique QA | judge acc | multi-hop (n=32) | temporal (n=37) | open-domain (n=13) | single-hop (n=70) | p50 ms | p95 ms | harness tokens |+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |+| checkpoint-a | 156 | 152 | 0.0% | 0% | 0% | 0% | 0% | 327 | 540 | 0.08M |+| v3 | 151 | 151 | 38.4% | 22% | 30% | 31% | 52% | 484 | 992 | 1.76M |+| v4-context | 152 | 152 | 44.1% | 22% | 38% | 46% | 57% | 6184 | 8368 | 2.12M |+| v5 | 152 | 152 | 38.2% | 22% | 38% | 15% | 50% | 7111 | 10615 | 2.20M |+| v6 | 152 | 152 | 75.0% | 66% | 81% | 46% | 81% | 4414 | 5694 | 4.01M |+| push75-a | 152 | 152 | 72.4% | 50% | 76% | 46% | 86% | 4081 | 5683 | 4.01M |+| push75-b | 152 | 152 | 75.7% | 56% | 84% | 54% | 84% | 4053 | 5446 | 4.06M |+| push75-c | 152 | 152 | 82.9% | 78% | 86% | 54% | 89% | 4793 | 6333 | 7.74M |+| push75-d | 152 | 152 | 86.2% | 81% | 92% | 69% | 89% | 4848 | 6902 | 7.86M |++The gaps below are ranked by how much they distort that inference, worst first.++### 1. The answer prompt has been fitted to conv-26 gold strings, so the headline now partly measures prompt-to-annotator alignment++This is the most serious problem, it is currently active in the working tree, and it is demonstrable rather than suspected.++`push75-c` and `push75-d` have **byte-identical manifests** apart from `run_id` and timestamps: same brain, same `use_ppr=True`, `max_passages=16`, `max_facts=50`, `apply_fact_filter=True`, `sufficiency_retry=True`, same answer and judge model. The only thing that changed between them is `ANSWER_SYSTEM` in `benchmarks/locomo/prompts.py`, which is uncommitted. Accuracy moved 82.9% → 86.2%.++Now compare the added prompt rules against the gold answers they recover:++| Added rule (`prompts.py`, uncommitted) | conv-26 gold answer | push75-c prediction | push75-d prediction |+| --- | --- | --- | --- |+| L9: `Keep speaker names explicit (e.g. "Melanie's slipper", not "my slipper")` | `In Melanie's slipper` | `In my slipper.` — wrong | `...in Melanie's slipper...` — right |+| L14: `always answer with a short hedged conclusion ... (e.g. "Likely no; ...")` | `LIkely no; though she likes reading...` | `Not mentioned in the conversation.` — wrong | `Likely no; the conversation shows...` — right |+| L13: `Prefer the relative phrasing ... "the week of <date>"` | `The week of 23 August 2023` | `23 August 2023 (during the session...)` — wrong | `...the week of August 23, 2023...` — right |+| L13: `For "how long" questions, prefer "since <year>"` | `Since 2016` | `Seven years.` — wrong | `Seven years (as of September 2023).` — right |+| L12: `ENUMERATION: ... Return ALL matching items` | `Her mentors, family, and friends` | `Melanie, friends, and family.` — wrong | `Melanie, friends, family, and mentors.` — right |++The prompt does not merely describe a formatting preference. Line 9 contains the gold answer string `Melanie's slipper` *and* the exact wrong answer the system gave on the previous run (`my slipper`). Line 14's example hedge `"Likely no; ..."` is the opening of the gold answer for the open-domain items. Line 13's Sunday-versus-Saturday rule resolves one specific question (`When did Melanie run a charity race?`, gold `The sunday before 25 May 2023`). `README.md` warns not to ingest the QA annotations because that leaks the test set; the letter of that rule is respected while its purpose is defeated through the prompt instead.++The `Since 2016` / `Seven years` case deserves separate attention, because it shows the failure is not only leakage. In both runs the memory layer supplied the fact — the system knew the duration. `push75-c` was marked wrong and `push75-d` right for the same underlying retrieval, purely because `push75-d` appended a parenthetical date. Neither prediction contains "2016". That question was never a memory failure in either run, yet it counts as one datapoint of memory quality in both.++Consequence: the headline conflates *did the memory layer supply the fact* with *did the answer prompt phrase it the way LoCoMo's annotator did*. Because the prompt is tuned on the same 152 questions that produce the score, there is no held-out set on which to detect this. The gains from `v5` (38.2%) to `push75-d` (86.2%) are a mixture of genuine retrieval improvement, which section 2.5 shows is real and large, and prompt fitting, which is not, and the harness cannot currently tell you the ratio.++Classification: **bug**, in the sense that it invalidates the measurement. The prompt edits are individually defensible as product behaviour ("never return an empty answer" is a reasonable rule); the problem is that they were derived from and validated on the eval set.++### 2. Every recorded number comes from one conversation out of ten++All ten run directories are `locomo-conv26-*`, and every manifest has `samples: ["conv-26"]`. That is 152 of the dataset's 1,540 non-adversarial questions — 9.9%. Two speakers, Caroline and Melanie, one persona pair, one event graph.++Sample size drives everything downstream. Wilson intervals on the headline are 11-14pp wide, and per category they are far worse. For `push75-c`:++| category | n | accuracy | Wilson 95% CI | width |+| --- | ---: | ---: | --- | ---: |+| multi-hop | 32 | 78.1% | [61.2, 89.0] | 27.7pp |+| temporal | 37 | 86.5% | [72.0, 94.1] | 22.1pp |+| open-domain | 13 | 53.8% | [29.1, 76.8] | 47.7pp |+| single-hop | 70 | 88.6% | [79.0, 94.1] | 15.1pp |++The two categories the maintainer named as the priority — multi-hop and temporal — are measured at n=32 and n=37. A real 10pp improvement in multi-hop is invisible at this sample size. The harness computes and prints these intervals honestly; the problem is that decisions are being taken on differences much smaller than the intervals.++### 3. Configurations are compared with an unpaired test when a paired one is available and free++The runs are paired by construction: the same 152 questions, same ids. The harness compares them by eyeballing two independent Wilson intervals, which throws away the pairing and loses most of the power. Running an exact McNemar test on the recorded rows instead:++| comparison | flipped to correct | flipped to wrong | exact p | verdict |+| --- | ---: | ---: | ---: | --- |+| v6 → push75-a | 7 | 11 | 0.481 | not distinguishable |+| push75-a → push75-b | 12 | 7 | 0.359 | not distinguishable |+| push75-b → push75-c | 20 | 9 | 0.061 | not distinguishable at 0.05 |+| push75-c → push75-d | 11 | 6 | 0.332 | not distinguishable |+| push75-a → push75-c | 24 | 8 | 0.007 | real |+| push75-a → push75-d | 25 | 4 | 0.0001 | real |++**Not one of the individual steps the team took is statistically distinguishable from the step before it.** Only the accumulated a→c and a→d differences are. Meanwhile the churn is large: even between configurations that are statistically identical, 15-20 questions flip in each direction. The `+3.3pp` from `push75-a` to `push75-b` that reads as progress in `report.md` is 12 questions gained and 7 lost, p=0.36.++This also means `v6` → `push75-a` — which looks like a small 2.6pp regression — actually broke 11 questions while fixing 7, and dropped multi-hop from 66% to 50%. A headline that moves 2.6pp hid a 16pp per-category swing.++Classification: **gap**, and the cheapest one to close. McNemar on paired runs needs no new data, only the rows already on disk.++### 4. The judge is unvalidated, is its own answerer, and cannot see the evidence++Three distinct problems, all in `judge.py` and `prompts.py`.++**No human calibration.** There is no file of human labels anywhere in the repo, and no agreement statistic is computed. Judge accuracy *is* the metric; nothing measures the judge. The systematic study in `2606.19544` (21 judges, ~541,000 judgments) finds that exact-match agreement, the metric practitioners default to, overstates discriminative ability, with kappa deflation of 33-41pp against Cohen's kappa on MT-Bench, and that judge rankings shift by up to 14 positions across benchmarks. An uncalibrated judge is an unknown-magnitude bias, not a small one.++**Answerer and judge are the same model.** `config.py:17-18` sets `DEFAULT_ANSWER_MODEL = DEFAULT_JUDGE_MODEL = "deepseek-v4-flash"`, and every manifest confirms both fields are that value. Self-preference bias is measurable and directional (`2410.21819`, `2604.22891`); the mechanism proposed in `2410.21819` is that judges over-reward low-perplexity, familiar-looking text, which is precisely the text their own decoder produces. The `Seven years.` → `Seven years (as of September 2023).` flip in section 2.1 is consistent with the verbosity-flavoured version of this.++**The judge never sees the retrieved context.** `build_judge_messages` passes question, gold, prediction and nothing else (`prompts.py:86-90`); `evaluate.py:205-207` calls it with no context argument. So the judge cannot assess groundedness at all. A prediction that is correct by luck or by parametric knowledge of the base model scores identically to one derived from retrieved evidence. For a memory product this is the wrong thing to be blind to: the entire value proposition is that the answer came from the stored graph.++`temperature=0` (`judge.py:68-73`) is set, which helps, but it does not make the judge unbiased and it does not make it reproducible across provider-side model updates, since `deepseek-v4-flash` is a moving alias with no pinned snapshot.++### 5. Failure attribution stops one step short of being actionable — and the missing step reverses the current priority++`answerer_gap` (`metrics.py:211-219`) is a scalar difference of two aggregate rates. It cannot tell you *which* questions failed for which reason, and it can go negative when the `gold_in_context` proxy under-counts, which it does.++Because `AnswerRecord` persists the full context per row, a proper joint decomposition is computable offline today. Assigning each question to the first failing stage — evidence session never retrieved, or retrieved but gold string absent from the assembled context, or gold present and the answer still wrong:++| run | evidence session missing | gold not in context | generation or judge | correct | context ceiling |+| --- | ---: | ---: | ---: | ---: | ---: |+| v3 | 60.9% | 0.7% | 0.0% | 38.4% | 21.9% |+| v5 | 61.2% | 0.7% | 0.0% | 38.2% | 32.9% |+| v6 | 24.3% | 0.7% | 0.0% | 75.0% | 62.5% |+| push75-a | 11.8% | 1.3% | 14.5% | 72.4% | 90.8% |+| push75-b | 9.9% | 1.3% | 13.2% | 75.7% | 90.8% |+| push75-c | 2.0% | 3.9% | 11.2% | 82.9% | 88.8% |++This changes the picture materially. Retrieval coverage went from missing the evidence session on 61% of questions to missing it on 2%. That work was real, large, and is the true story of `v5` → `push75-c`. But **as of `push75-c`, 17 of the 26 remaining failures — 65% — are on the generation-or-judge side of the boundary, not the retrieval side.** The evidence was in the context and the pipeline still did not convert it.++Meanwhile the tuning between `push75-a` and `push75-c` was entirely retrieval-side (`use_ppr` off→on, `max_passages` 8→16, `max_facts` 40→50, `sufficiency_retry` off→on) and cost 93% more tokens (4.01M → 7.74M) to push a component that was already at a 91% ceiling. Without this table on the report, effort keeps flowing to the stage that is already nearly saturated.++Two caveats I want to be explicit about. First, the boundary between the last two columns is only as good as `gold_in_context`, and that proxy is weak: `metrics.py:138` uses `token in blob`, a raw substring test with no word boundary, so gold token `may` matches `maybe` and `2022` matches `20225`. Second, the proxy under-counts as well as over-counts, because the judge accepts paraphrase that shares no gold tokens — visible in `v3`, where accuracy (38.4%) exceeds the measured ceiling (21.9%), which is impossible if the proxy were sound. Treat the split as directionally right and numerically soft; the fix is claim-level entailment rather than substring matching, per section 4.4.++### 6. Adversarial questions — 22% of LoCoMo, and the only test of abstention — are silently excluded, and would be scored wrong if included++`iter_qa_jobs` skips category 5 when `skip_adversarial=True` (`evaluate.py:104,113`), which is the default and is set in every recorded run. So abstention is not measured at all. That matters because abstention is one of the five core abilities LongMemEval was built to isolate (`2410.10813`), and for a memory product "I don't have that" is a correctness requirement, not a nicety.++If the flag is ever flipped, the numbers will be wrong rather than merely absent. In `locomo10.json`, 444 of the 446 category-5 items carry their gold under the key `adversarial_answer`, not `answer`:++```+key signatures across 1,986 QA items:+ 1540 ('answer', 'category', 'evidence', 'question')+ 444 ('adversarial_answer', 'category', 'evidence', 'question')+ 2 ('adversarial_answer', 'answer', 'category', 'evidence', 'question')+```++`evaluate.py:177` reads only `qa.get("answer")`, so gold becomes `""` for those 444. Then `f1_score` returns 1.0 when both prediction and gold are empty (`metrics.py:29-30`) and `bleu1_score` does the same (`metrics.py:45-46`) — an empty prediction scores a perfect 1.0 — while the judge is handed a blank `Gold answer:` line and asked to apply the rule at `prompts.py:24`, which it cannot evaluate without knowing the question was adversarial by design.++This is not a local mistake. `2604.10981` documents the identical defect in the upstream LoCoMo reference implementation, describing "an empty-gold scoring bug in the LOCOMO reference implementation that renders 23% of its corpus unscorable by construction" — matching the 444/1,986 = 22.4% measured here. The bug was inherited, and `--skip-adversarial` currently masks it.++### 7. A run cannot be reproduced from what the run records++The manifest captures `run_id`, `command`, `samples`, `concurrency`, `skip_adversarial`, `categories`, `limit`, `brainapi_url`, `answer_model`, `judge_model`, `historical_limit`, `max_passages`, `max_facts`, `apply_fact_filter`, `use_ppr`, `sufficiency_retry`, and timestamps. Missing:++- **Git SHA.** Nothing ties a run to the code that produced it. The `push75-c` → `push75-d` prompt change is invisible in the manifest, which is why that pair is indistinguishable in the recorded metadata despite being the cleanest demonstration of prompt fitting in the repo.+- **Prompt hash.** Same problem, one level finer. `ANSWER_SYSTEM` and `JUDGE_SYSTEM` are load-bearing measurement instruments and are not versioned.+- **Dataset digest.** `dataset_path` is not hashed. A re-download that changes upstream is undetectable.+- **Model pinning.** `deepseek-v4-flash` is an alias. `temperature=0` is set (`answer.py:42-46`, `judge.py:68-73`) but no seed is requested and no provider snapshot is recorded, so run-to-run and month-to-month drift are both unbounded and invisible.+- **The graph under test.** This is the largest hole. `push75-a/b/c/d`, `v4-context`, and `checkpoint-a` have **no `ingest.jsonl` at all** — they evaluate against a brain that some earlier run populated. The ingestion configuration, extraction model, and graph contents in effect are recorded nowhere in those runs. `v5` (38.2%) and `v6` (75.0%) each did re-ingest, so the single largest accuracy jump in the project's history spans a re-ingestion whose parameters are only partly recoverable.+- **Manifest schema stability.** `checkpoint-a` predates the retrieval flags entirely, so cross-run comparison requires knowing what the code defaulted to at that moment.++There is also a live default-drift hazard between the harness and the server that I should flag as an observation rather than a finding, since retrieval internals are a sibling workstream's scope: the uncommitted diff to `src/services/api/controllers/retrieve.py` reads `getattr(request, "use_ppr", True)` while the Pydantic model in `src/services/api/constants/requests.py` declares `use_ppr: bool = False`. The Pydantic default wins for any client that omits the field. Which means "what the server does by default" and "what the benchmark measured" can diverge without either side erroring.++### 8. Duplicate rows are counted, and a fully failed run renders as a clean report++`completed_qa_keys` (`evaluate.py:89-97`) de-duplicates *jobs* on resume, so `--resume` is safe. But `build_report` reads every non-errored row with no de-duplication (`report.py:34`). The author knew to solve this for ingestion — `_latest_ingest_rows` (`report.py:17-26`) keeps the last status per unit, with the comment "resume appends retries" — and the same treatment was never applied to answers.++`checkpoint-a` is the proof: 176 rows, 20 errored, 156 scored, but only **152 unique** `(sample_id, qa_index)` pairs. Four questions are counted twice, all in the temporal category, which is why that run reports `n=156` with `cat2 n=41` where every other run has 37. And because `report.py:34` drops errored rows silently, that run — in which every single answer failed, 0.0% across all four categories, 82K tokens against the 4M+ of a healthy run — renders as a complete, plausible-looking report with a valid `n`. Nothing in `report.md` says "this run is broken." The only warning path (`report.py:60-63`) triggers on ingest failures, and that run has no ingest rows.++Running `report` mid-flight has the same shape: it will happily aggregate a partial `answers.jsonl` and print a headline.++### 9. Cost is measured for the harness, not for the product++`total_llm_tokens` sums `answer_total_tokens` and `judge_total_tokens` only (`metrics.py:205-209`, labelled "answer+judge" at `report.py:129`). So:++- **Judge tokens are counted as system cost.** They are evaluation overhead and inflate the figure.+- **Server-side tokens are invisible.** Ingestion agent swarm calls, and the `fact_filter` LLM calls on the retrieval path, appear nowhere. `invoke_loop.py:51` logs usage to stdout and drops it. The uncommitted diff to `src/core/search/fact_filter.py` changes work on the retrieval path whose token cost the benchmark structurally cannot see.+- **Ingestion cost is not measured at all.** `IngestRecord` (`ingest.py:22-36`) has two latencies and no token or cost field.+- **There is no cost-per-correct-answer.** From `push75-a` to `push75-c`, accuracy rose 14% relative while harness tokens rose 93%. Cost per correct answer went from roughly 36K to 61K tokens, a 68% increase, and no reported metric shows this.+- **Denominators disagree.** Latency and tokens aggregate over all rows (`metrics.py:200-209`) while accuracy uses the non-adversarial subset (`metrics.py:196`). Identical today because category 5 is always skipped; wrong the moment it isn't.++### 10. Latency is reported but not governed, and the stated budget is being violated++`00-scope-and-constraints.md` fixes the budget for `/retrieve/context` at **sub-second, fast and cheap**. Measured p50 has been between 4.0 and 4.8 seconds in every run since `v6`, with p95 between 5.4 and 6.9 seconds. `push75-c` and `push75-d`, the two best-scoring configurations, are also the two slowest since `v5`.++The harness reports these numbers accurately and nothing acts on them, so accuracy tuning has been silently spending a budget the project has already declared fixed. Two things are worth separating here. Enabling `use_ppr` and `sufficiency_retry` for a benchmark is a legitimate experiment. Reporting the resulting accuracy as *the* headline, without a gate that fails when p95 exceeds the documented budget, is what turns an experiment into a misleading claim.++A measurement detail makes this worse rather than better: `client.py:54-78` resets its timer inside the retry loop, so `retrieve_latency_ms` records the duration of the last attempt only, excluding earlier attempts and backoff sleeps. Reported latency therefore under-states user-visible latency whenever a retry occurred.++### 11. Evidence-session recall is coarser than the evidence annotation it consumes++LoCoMo annotates evidence at dialogue-turn granularity (`D1:12`). `evidence_session_ids` (`metrics.py:93-103`) reduces `D1:12` to `session_1`, and `retrieved_session_ids` falls back to regex-scraping session ids out of the context blob (`metrics.py:106-121`). So a run that retrieves the right session but the wrong turn inside it scores full evidence recall. This is why `evidence_session_recall_full` reads 97.3% for `push75-c` while the finer-grained context ceiling is 88.8% — the coarse metric is the more flattering one, and it is the one on the report's headline.++### 12. The synergies endpoint has no metric, and one advertised feature does not exist++Ranked below the harness issues because it is a smaller surface, but it is the clearest instance of "unmeasured means unknown." `polarity` is accepted, documented in both the route (`routes/retrieve.py:515`) and the retriever (`entity_sibilings.py:65`), and never read (`entity_sibilings.py:53`). A test suite for this endpoint would have caught that on day one. The scoring constants (`entity_sibilings.py:26,29,32,35`) were chosen without any offline metric to choose them against, and the unbounded return (`entity_sibilings.py:437`) means response size is a function of graph density.++### 13. Agent-facing usefulness is unmeasured, and the tool surface makes failure ambiguous++The product goal is to be a substrate for smarter agents. Nothing measures whether an agent holding these five MCP tools completes tasks better than one without them. The `"Unauthorized"` string returns (`main.py:212,266,288,321`) are a measurement problem as much as an ergonomics one: a bare string is indistinguishable from a legitimate empty result, so an eval harness cannot classify tool-call outcomes without string matching.++### 14. No CI gate exists++`staging.yaml:15` is a placeholder. 25 `unittest` files run only when a human runs them. No coverage measurement. Nothing prevents a change that drops multi-hop accuracy 20pp from shipping, and given section 2.3, nothing would reliably detect it even if the benchmark were run, because a 20pp drop on n=32 sits inside the noise band of an unpaired comparison.++## Open questions for the maintainer++1. Were the conv-26-specific rules in `ANSWER_SYSTEM` (`prompts.py:9,13,14`) written by reading conv-26's gold answers, and are you willing to delete them and re-baseline, accepting that the headline will drop?+2. Should the LoCoMo headline be defined on a held-out split of conversations that prompts are never tuned against, and if so which conversations do you want reserved?+3. Is 9.9% of the dataset (conv-26 only) a deliberate cost decision, and what per-run token budget would let us move to all ten conversations?+4. Is `deepseek-v4-flash` acceptable as both answerer and judge, or should the judge move to a different model family to remove self-preference bias?+5. Would you fund a one-time human labelling pass — I estimate 200-300 items, a few hours — to calibrate the judge, given that without it every accuracy number has unknown bias?+6. Is `skip_adversarial=True` a deliberate scoping choice or an artefact, and do you want abstention to count toward the headline once the `adversarial_answer` key is read correctly?+7. Is the sub-second budget for `/retrieve/context` in `00-scope-and-constraints.md` still binding, given measured p50 of 4.0-4.8s in every run since `v6`?+8. Should `use_ppr` and `sufficiency_retry` be treated as the intended production default (in which case the latency budget needs restating) or as benchmark-only settings (in which case the headline should be reported with them off)?+9. Which single number do you want CI to gate on, and what drop are you willing to block a merge for?+10. Should the CI gate run against a live BrainAPI plus a live LLM, or against recorded fixtures, given that the former makes CI cost money and flake on provider outages?+11. Is `polarity` on the synergies endpoint (`entity_sibilings.py:53`) intended to work, and if so what does "opposite" mean concretely enough to write a test for?+12. What is a synergy *for* — a user-facing "you might also care about X", or an internal retrieval expansion signal — since the two need different metrics?+13. Do you have any source of ground-truth judgement about synergy quality (your own labels on a few entities would be enough) or must the first metric be fully unsupervised?+14. Which downstream agent task should define "the memory made the agent better", since we need one concrete task to measure MCP usefulness at all?+15. Is it acceptable for the benchmark to require a fresh ingestion per run — roughly doubling wall-clock — in exchange for runs being reproducible from their own manifest?+16. Should ingestion token cost be plumbed into the API response so the harness can see it, or captured server-side and correlated by run id?++## Frontier techniques++### 4.1 Validate the judge before trusting it: kappa, not exact match++**Mechanism.** Sample a stratified subset of scored questions, have a human label correct/incorrect, then report chance-corrected agreement (Cohen's kappa) between judge and human rather than raw agreement. Separately audit the judge for position and consistency by re-running it with swapped or repeated inputs. `2606.19544` distils this into a "Minimum Viable Validation Protocol" over three protocols: agreement, consistency, bias audit.++**arXiv.** `2606.19544` (Reliability without Validity), 21 judges, ~541,000 judgments. Supporting: `2406.07791` (position bias; introduces repetition stability, position consistency, preference fairness), `2410.21819` and `2604.22891` (self-preference bias quantification; the latter reduces it 31.5% via structured multi-dimensional evaluation).++**Reported gain.** Not an accuracy gain — a validity gain. The headline finding is that exact-match agreement overstates discriminative ability with kappa deflation of 33-41pp on MT-Bench, and that judge rankings shift by up to 14 positions across benchmarks. Two production-deployed judges combined test-retest reliability above 0.95 with position bias above 0.10, so a stable judge is not necessarily an unbiased one.++**Cost.** One human labelling pass of 200-300 items. Then a re-scoring script; no new model. Ongoing cost is one extra judge call per audited item if you add swap-consistency checks.++**Fit.** Direct. The harness already persists `judge_raw` and `judge_reason` per row, so a labelling UI can be a CSV export. Because the judge here is pointwise (not pairwise) with a fixed field order, classical position bias does not apply; the relevant biases are self-preference and the verbosity effect visible in section 2.1.++**Verdict: adopt.** This is the precondition for every other number in this document meaning anything.++### 4.2 Component-wise memory evaluation instead of end-to-end-only++**Mechanism.** Decompose the memory system into representation/storage, extraction, retrieval/routing, and maintenance, then measure each module separately with ablations, rather than reporting a single end-to-end QA score. `2606.24775` does exactly this across 12 memory systems and 11 datasets and argues that end-to-end F1/BLEU treats the system as a monolithic black box, hiding operational cost, architectural trade-offs, and robustness under knowledge updates.++**arXiv.** `2606.24775` (Are We Ready For An Agent-Native Memory System?). Complementary industrial template: `2607.13157` (Oracle Agent Memory), which explicitly complements downstream task accuracy with "memory-centric measures such as evidence retrieval, recall, latency, and estimated token use" — the exact quartet missing from `report.md` on the server side.++**Reported gain.** `2606.24775`'s finding that no single architecture dominates, and that effectiveness depends on whether the memory structure matches the workload bottleneck, is the useful one here: it is the general form of section 2.5's result that BrainAPI's bottleneck has already moved from retrieval to generation. Also finds localized maintenance is more cost-efficient than global reorganization.++**Cost.** Reporting-layer work plus server-side token accounting. No new models.++**Fit.** Strong, and partially already built — `answerer_gap` is a one-dimensional version of this. The blocker is that server-side token usage is logged to stdout (`invoke_loop.py:51`) instead of being returned or traced.++**Verdict: adopt.** This is the framing for the whole measurement stack in section 5.++### 4.3 Add LongMemEval to isolate the five abilities LoCoMo blurs together++**Mechanism.** 500 curated questions over scalable chat histories, targeting five separable abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Also supplies a three-stage decomposition — indexing, retrieval, reading — for attributing failures.++**arXiv.** `2410.10813` (LongMemEval). Successor for agent environments: `2605.12493` (LongMemEval-V2, 451 questions over trajectories up to 115M tokens, reporting an accuracy-latency Pareto frontier rather than accuracy alone).++**Reported gain.** Commercial assistants and long-context LLMs drop ~30% accuracy on sustained interactions, so the benchmark has headroom where LoCoMo is closer to saturation for this system (single-hop already 89%).++**Why it matters here specifically.** Two of the five abilities are directly on the project's critical path and are currently unmeasured or barely measured: **knowledge updates** is the whole subject of workstream 05 (superseded facts returning as current truth) and LoCoMo has no category for it; **abstention** is skipped entirely (section 2.6). Adding LongMemEval buys measurement of temporal truth, which is otherwise being worked on blind.++**Cost.** A new ingestion path and a new adapter — the largest single item in the plan. Histories are large; expect ingestion cost well above conv-26. Mitigate by starting with the small history configuration.++**Fit.** Good. It is the same shape of task (conversational history in, QA out) so `evaluate.py` needs an adapter, not a rewrite.++**Verdict: adopt, phase 3.** Do it after the harness is trustworthy; adding a second benchmark to an uncalibrated harness doubles the unreliable surface.++### 4.4 Claim-level faithfulness and groundedness in place of substring matching++**Mechanism.** Decompose an answer into atomic claims and check each against the retrieved context with an entailment model, yielding reference-free faithfulness. RAGAS scores faithfulness, context relevance, and answer relevance without ground truth (`2309.15217`). ARES trains lightweight LM judges on synthetic data and — the important part — corrects their errors with prediction-powered inference against a few hundred human annotations (`2311.09476`).++**arXiv.** `2309.15217` (RAGAS), `2311.09476` (ARES), `2601.04196` (RAGVUE, which decomposes into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and *judge calibration* as a first-class metric, and reports surfacing failures RAGAS misses).++**Reported gain.** ARES evaluates accurately across eight KILT/SuperGLUE/AIS tasks with only a few hundred human annotations, and its judges survive domain shift. For this project the gain is that the section 2.5 attribution boundary stops depending on `token in blob`.++**Cost.** An entailment or judge call per claim. Too expensive to run on every question every time; run it on the failure set and on a fixed sampled subset.++**Fit.** Very good. `text_context`, `source_passages`, and `historical_context` are already persisted per row, so faithfulness can be computed offline from existing `answers.jsonl` files with no re-run.++**Verdict: adopt the faithfulness metric; adopt ARES's prediction-powered inference idea for judge calibration.** Reject the framing that reference-free metrics can replace the gold-answer judge here — LoCoMo has gold answers and they are the stronger signal; use faithfulness as the *second* axis that catches right-for-the-wrong-reason.++### 4.5 Treat multi-hop scores as suspect until shortcut-resistance is tested++**Mechanism.** Establish how much of a multi-hop score survives when the shortcut is removed: run a single-hop-only baseline, and evaluate on adversarial or debiased variants where word-matching the question to one sentence no longer works.++**arXiv.** `1906.02900` (Compositional Questions Do Not Necessitate Multi-hop Reasoning) is canonical: a single-hop BERT model reaches 67 F1 on HotpotQA, comparable to multi-hop models of the time, and humans answer over 80% of questions without being shown all the required paragraphs. `1906.07132` constructs adversarial documents that contradict the shortcut without invalidating the gold answer, and strong baselines drop significantly. `2302.05963` builds four debiased datasets over 2WikiMultiHopQA and HotpotQA-small and shows underlying-reasoning supervision reduces shortcut reliance but does not confer robustness to sub-questions or inverted questions.++**Reported gain.** Diagnostic. The relevant number is how much a score falls when the shortcut is closed.++**Fit and the honest caveat.** LoCoMo's category 1 is "multi-hop" by annotator intent, and the evidence annotations name multiple turns, but nothing verifies that answering actually required combining them. With 32 questions and a context window holding 16 passages plus 40-50 facts, a single passage containing the answer is entirely plausible for a meaningful fraction. A cheap check exists and needs no new data: for each multi-hop question, count how many distinct annotated evidence turns are strictly necessary by re-answering with each one ablated.++**Verdict: adapt.** Do not import HotpotQA — different domain, and the retrieval corpus assumption (Wikipedia paragraphs) does not match a conversational event graph. Do import the methodology as a leave-one-evidence-out ablation on LoCoMo's own multi-hop set.++### 4.6 Multi-hop datasets: which to add and which to refuse++**Adopt, later: MuSiQue-style compositional construction and MultiHop-RAG.** Rejected for now on cost grounds, not merit — a second QA benchmark before the judge is calibrated multiplies unreliable surface.++**Adopt the design principle from `2605.12361` (MedHopQA) immediately, at zero cost.** Its anti-gaming construction embeds 1,000 scored questions inside a public set of 10,000 with answers withheld, explicitly treating saturation resistance and contamination resistance as design constraints. The transferable idea: **a held-out set whose gold answers the prompt author has not read.** That single practice would have prevented section 2.1.++**Reject `2412.17032` (MINTQA)** for this workstream: it targets new and long-tail *world* knowledge in the model's parametric memory, whereas BrainAPI's task is recall from a graph built from supplied conversations. The failure mode it isolates is not the one here.++**Reject `1601.02789`-style n-gram metric surveys and the reliance on BLEU/F1 generally as headline signals.** They remain useful as cheap regression tripwires — a large F1 drop with flat judge accuracy means the answer *format* changed, which is exactly the signal that would have flagged section 2.1 — but they should not be optimized. Note `mean_f1` is 0.474 while judge accuracy is 82.9% for `push75-c`; the two are nearly decoupled.++### 4.7 Attribution and citation verification, for the "did it come from memory" question++**Mechanism.** Require the answer to cite the passage or triple that supports it, then verify the citation independently. `2509.21557` separates generation-time citation (one pass) from post-hoc citation (draft, then attach and verify) and finds retrieval is the dominant driver of attribution quality in both, with post-hoc achieving high coverage at competitive correctness and moderate latency. `2510.11394` (VeriCite) verifies claims with an NLI model, selects supporting evidence, then refines.++**arXiv.** `2509.21557`, `2510.11394`, `2408.12398` (metrics vs humans on fine-grained citation support), `2606.28358` (mechanistic account of when models cite).++**Reported gain.** VeriCite improves citation quality while maintaining answer correctness across five open-source LLMs and four datasets.++**Two caveats I will not paper over.** `2408.12398` finds no single faithfulness metric excels across all evaluations, and that the best metrics specifically struggle to distinguish *partial* support from full or none — which is the interesting case for multi-hop. `2606.28358` shows citation behaviour is a distributed "attributional ensemble" partly disconnected from the model's actual reasoning path, concluding inline citations "can create a false sense of security." So citations are a useful measurement instrument and a weak guarantee.++**Fit.** Good for the deep-navigation surface and for offline evaluation. Not for `/retrieve/context`, whose sub-second budget forbids an extra verification pass.++**Verdict: adopt for evaluation only.** Compute attribution offline on stored rows. Do not add a verification hop to the hot path.++### 4.8 Intrinsic knowledge-graph construction quality, to measure ingestion without gold graphs++**Mechanism.** Score an automatically extracted graph against an "ideal" graph derived from the source text: entity-level completeness, resolution quality, and connectivity; relation-level predicate preservation and multiplicity via lexical similarity, dependency-parse alignment, and negation handling. `2607.10212` (KGCQual) does this, and validates that its scores correlate significantly with downstream link-prediction performance on the same extracted graphs — which is what makes it more than a vanity metric.++**arXiv.** `2607.10212` (KGCQual), `2502.05239` (KG construction evaluation emphasising hallucination and omission with BERTScore graph similarity), `2605.05476` (a dual-purpose benchmark with an expert-curated reference graph as an upper bound, designed to separate "is it the model or the graph").++**Reported gain.** `2607.10212` reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook, on WebNLG, TinyButMighty, and BenchIE.++**Fit.** Conceptually strong: this is the missing ingestion-side metric, and it needs no gold graph. Practically it needs adaptation — these tools target sentence-level triple extraction from encyclopedic text, whereas BrainAPI produces event-centric structures from dialogue with speaker attribution and time. The transferable core is the **omission/hallucination split**: for a session, what fraction of the entities and relations a reader would extract are present, and what fraction of extracted triples are unsupported.++**Verdict: adapt, phase 4.** The cheap version is a fixed 3-5 session gold-annotation set, hand-built once, scored on entity recall and triple precision. Reject wholesale adoption of the published scorers; their linguistic assumptions do not transfer.++### 4.9 Recommendation quality for the synergies endpoint++**Mechanism.** Score a ranked list on beyond-accuracy axes — diversity, novelty, serendipity, coverage — instead of relevance alone. Serendipity is the one that matters for "non-obvious connections", and the validated decomposition is unexpectedness plus relevance plus timeliness plus user curiosity.++**arXiv.** `1906.11431` (User Validation of Recommendation Serendipity Metrics) is the load-bearing citation: over 10,000 users of real feedback, finding that user-profile-based and especially content-based unexpectedness metrics outperform popularity-based ones, and that the full four-component metric indicates serendipity more accurately than any subset. Also `2310.02294` (beyond-accuracy review for GNN recommenders), `2307.14951` (Widespread Flaws in Offline Evaluation), `2309.05892` (report distributions, not point estimates).++**LLM-as-judge for serendipity: adopt with a stated ceiling.** `2508.17571` proposes a universally applicable LLM-based serendipity evaluator, finding chain-of-thought prompting most accurate and, notably, that no serendipity-oriented recommender consistently beat a general one across three datasets. But `2507.17290`'s meta-evaluation against real user studies reports a best-case Pearson correlation of **21.5%** with human judgement. That is a weak instrument. It is still better than the current state, which is no instrument, but it must not be used to gate merges.++**Reject** simulation-based evaluation (`2209.08642`) and prequential streaming evaluation (`1504.08175`): both assume an interaction log and a user-feedback loop that BrainAPI does not have.++**Verdict: adopt the unsupervised structural metrics first** (intra-list diversity, catalogue coverage over entity types, graph distance from the seed entity as an unexpectedness proxy) because they are deterministic, free, and gate-able. Add the LLM serendipity judge as a **reported, non-gating** diagnostic. Fix `polarity` before either, since a parameter that does nothing cannot be measured.++### 4.10 Agent memory utility as downstream task success++**Mechanism.** Define a concrete agent task, run it with and without memory access, and measure task completion. `2605.12493` (LongMemEval-V2) formalizes this for environment experience across five abilities — static state recall, dynamic state tracking, workflow knowledge, environment gotchas, premise awareness — using a context-gathering formulation where the memory system consumes history and returns compact evidence for downstream QA.++**arXiv.** `2605.12493`, and `2607.13157` for the industrial pattern of pairing downstream accuracy with memory-centric measures.++**Reported gain.** Best method 72.5% average accuracy versus 48.5% for the strongest RAG baseline, at substantial latency cost — the paper is explicit that it advances an accuracy-latency Pareto frontier rather than a single number, which is the right reporting shape for BrainAPI's two-tier architecture.++**Fit.** The context-gathering formulation maps almost exactly onto BrainAPI's MCP surface: tools consume the graph and return compact evidence. The obstacle is that BrainAPI has no defined downstream agent task, which is open question 14.++**Verdict: adapt, phase 5, and keep it small.** A 20-30 task suite exercising the five MCP tools, scored on completion with and without memory, plus tool-call success rate and tokens per completed task. Reject building a full agent benchmark; the goal is a tripwire showing the tools are usable, not a leaderboard.++### 4.11 Paired significance testing++**Mechanism.** Use the test that matches the design. For paired binary outcomes on identical items, that is McNemar's exact test, not a comparison of two independent proportions.++**arXiv.** `1809.01448` (Recommended Statistical Significance Tests for NLP Tasks), the appendix to the Dror et al. hitchhiker's guide, which maps task and measure to the valid test. Supporting cautionary evidence on underpowered protocols: `2305.01633` finds only 13% of surveyed NLP human evaluations had low enough barriers and enough information to even attempt reproduction.++**Reported gain.** Free statistical power. Section 2.3 is the demonstration: the same recorded rows, read with the correct test, reverse the conclusion about which steps were real.++**Cost.** About 30 lines of Python. No new data, no new models, no new runs.++**Verdict: adopt first.** Highest ratio of decision quality to effort in this entire document.++## Proposed measurement stack++### Benchmarks and datasets++| Layer | Dataset | Measures | Status | Priority |+| --- | --- | --- | --- | --- |+| Smoke | conv-26, 20 fixed questions | pipeline liveness, latency | exists, needs pinning | P0 |+| Headline | LoCoMo, **4 tuning + 6 held-out conversations** | multi-hop, temporal, open-domain, single-hop, abstention | exists but single-conversation and prompt-contaminated | P0 |+| Abstention | LoCoMo category 5 with `adversarial_answer` read correctly | refusal correctness | broken, masked by a flag | P1 |+| Update / temporal truth | LongMemEval (`2410.10813`) | knowledge updates, multi-session reasoning | absent | P2 |+| Ingestion quality | 3-5 hand-annotated sessions | entity recall, triple precision, omission, hallucination | absent | P3 |+| Recommendations | 20-30 seed entities, structural metrics | diversity, coverage, unexpectedness | absent | P3 |+| Agent utility | 20-30 MCP tasks, with/without memory | task completion, tool-call success | absent | P4 |++The tuning/held-out split is the single most important structural change. Prompts, thresholds, and scoring constants may be tuned on the 4 tuning conversations. The headline is reported on the 6 held-out ones, whose gold answers nobody who edits a prompt has read. Concretely: keep conv-26 in the tuning set (it is already contaminated and cannot be decontaminated), and never tune against the rest.++### Per-component metrics++Ingestion, per session: units submitted / completed / failed; extraction latency; **LLM tokens and cost**; entities and triples written; entity-resolution merge count; omission and hallucination rate against the annotated sessions.++Retrieval, per query: evidence recall at **turn** granularity, not just session (`metrics.py:93-103` is too coarse); context precision, the fraction of returned passages containing any evidence turn; the gold-in-context ceiling, computed by entailment rather than substring (`metrics.py:138`); passages and facts returned; **latency measured across all retry attempts** (`client.py:54-78` currently measures only the last); server-side filter tokens.++Generation, per answer: judge correctness; claim-level faithfulness against the retrieved context; abstention correctness split into correct refusal, wrong refusal, and hallucinated answer; F1 and BLEU-1 retained as *format-drift tripwires only*.++Judge, per release: Cohen's kappa against human labels; self-consistency across repeated calls; swap consistency; the human-label set version.++Run-level: cost per correct answer; total server plus harness tokens; p50 and p95 end-to-end latency; and the full reproducibility tuple — git SHA, prompt hashes, dataset digest, model snapshots, ingestion manifest reference.++### The failure-attribution breakdown++Every question lands in exactly one bucket, evaluated in order, so the buckets sum to 100% and each names one owning workstream:++| # | Bucket | Test | Owner |+| --- | --- | --- | --- |+| A | Not ingested | no annotated evidence turn exists in the graph | ingestion (01) |+| B | Ingested, not retrieved | evidence turn in graph, absent from result set | retrieval (02) |+| C | Retrieved, dropped in assembly | evidence in result set, absent from final context blob | retrieval (02) |+| D | In context, generation failed | context entails gold, answer wrong | generation / prompt |+| E | Answer correct, judge disagreed | human says correct, judge says wrong | judge calibration |+| F | Correct but ungrounded | answer correct, faithfulness fails | measurement validity |+| G | Correct and grounded | — | — |++The current harness collapses A+B+C into one number and D+E+F into another (`answerer_gap`). Splitting A from B is what tells you whether extraction or ranking is at fault. Splitting E out is what stops judge noise being read as a quality regression. Splitting F out is what catches right-for-the-wrong-reason, which for a memory product is a failure even when the score says success.++Report it as a stacked breakdown per category, since section 2.5 shows the mix differs sharply by category, and the aggregate hides that.++### What the CI regression gate asserts++Three tiers, because a benchmark that costs money and needs a live LLM cannot run on every push.++**Tier 1 — every push, no network, target under 2 minutes.**+- All 25 `unittest` files pass. This is the entire content of `staging.yaml:15` today and it is unimplemented.+- `selftest-metrics` passes (`metrics.py:248-275`), extended with cases that currently fail by omission: empty gold must not score F1 = 1.0; `gold_in_context` must respect word boundaries; duplicate rows must not be double-counted.+- Prompt-hash check: if `ANSWER_SYSTEM` or `JUDGE_SYSTEM` changed, the job fails with an instruction to re-baseline. A prompt change is a measurement-instrument change and must not pass silently.+- Static check: the synergies scoring constants and any declared-but-unread parameter (`entity_sibilings.py:53`) are covered by a test.++**Tier 2 — every pull request, live server, roughly 15 minutes, one held-out conversation.**+- Headline accuracy on the held-out conversation has not dropped by a margin that is **significant under McNemar's exact test at p < 0.05** against the stored baseline. Not "has not dropped 2pp" — section 2.3 shows raw thresholds at this n are noise.+- No per-category regression significant at p < 0.05, evaluated on multi-hop and temporal specifically, since those are the declared priorities.+- p95 retrieval latency for `/retrieve/context` is under the documented budget. Given measured p95 of 5.4-6.9s against a stated sub-second target, this gate must start as a **warning with a recorded value** and become blocking once the budget question (open question 7) is settled. A gate that fails on day one gets disabled.+- Cost per correct answer has not risen more than 20% without an explicit override label on the PR. This would have flagged `push75-a` → `push75-c`, which spent 93% more tokens for a 14% relative accuracy gain.+- Zero errored rows, and scored row count equals unique question count. This catches both the `checkpoint-a` class of silent failure and the duplicate-counting bug.+- Abstention correctness has not regressed, once category 5 is scored correctly.++**Tier 3 — nightly or on release tags, full dataset.**+- All 10 conversations, all categories, full attribution breakdown A-G.+- Judge kappa against the frozen human label set is above an agreed floor. If kappa drops, the *judge* regressed, and no accuracy number from that run is admissible.+- Faithfulness on a fixed sampled subset.+- Synergies structural metrics and MCP task-completion suite.+- The full reproducibility tuple recorded, and the run reproducible from its own manifest.++The gate's most important property is what it refuses to do: it does not assert that a number went up. It asserts that a number did not go **significantly** down, that the cost of any gain is visible, and that the measurement instrument itself has not changed underneath the comparison.++## Implementation plan++Sized S (1-2 files) or M (3-5 files). Every task has an acceptance criterion and a command. No task changes retrieval, extraction, or storage behaviour — this workstream measures, it does not tune. Note that `benchmarks/` currently has uncommitted changes; phase 0 assumes those are committed or stashed first so that a baseline is attributable to a SHA.++### Phase 0 — Stop the measurement from lying (no new data, no new models)++**0.1 Record the reproducibility tuple in every manifest.** S — `benchmarks/locomo/cli.py`, `benchmarks/locomo/ingest.py`.+Add git SHA, dirty-tree flag, SHA-256 of `ANSWER_SYSTEM` and `JUDGE_SYSTEM`, SHA-256 of the dataset file, and a reference to the ingest run that populated the brain.+*Acceptance:* a fresh `evaluate` manifest contains all six fields; a run started with a dirty tree is marked.+*Verify:* `python -m locomo evaluate --samples conv-26 --limit 5 --run-id repro-check && python -c "import json;m=json.load(open('benchmarks/runs/repro-check/manifest.json'));assert all(k in m for k in ('git_sha','dirty','answer_prompt_sha','judge_prompt_sha','dataset_sha','ingest_run_id'))"`++**0.2 De-duplicate answers in the report and fail loudly on broken runs.** S — `benchmarks/locomo/report.py`.+Apply the `_latest_ingest_rows` treatment (`report.py:17-26`) to answers keyed on `(sample_id, qa_index)`. Add warnings when the error count is above zero, when duplicates were dropped, or when scored rows are below the expected job count.+*Acceptance:* re-reporting `locomo-conv26-checkpoint-a` yields n=152 not 156, and `report.md` carries a warning naming 20 errored rows.+*Verify:* `python -m locomo report --run-id locomo-conv26-checkpoint-a && grep -c "Warning" benchmarks/runs/locomo-conv26-checkpoint-a/report.md`++**0.3 Add paired significance testing between runs.** S — `benchmarks/locomo/metrics.py`, `benchmarks/locomo/cli.py`.+A `compare` command taking two run ids, emitting McNemar exact p overall and per category, plus the flipped-question lists in both directions.+*Acceptance:* `compare push75-a push75-b` reports p ≈ 0.36 and flags the difference as not significant; `compare push75-a push75-c` reports p ≈ 0.007 and significant.+*Verify:* `python -m locomo compare --baseline locomo-conv26-push75-a --candidate locomo-conv26-push75-b`++**0.4 Fix the adversarial gold key and the empty-gold scoring path.** S — `benchmarks/locomo/evaluate.py`, `benchmarks/locomo/metrics.py`.+Read `adversarial_answer` when `answer` is absent; mark the record as an abstention item; make F1 and BLEU-1 return `None` rather than 1.0 for empty gold (`metrics.py:29-30,45-46`); score abstention items on refusal correctness instead.+*Acceptance:* `--no-skip-adversarial` on 10 category-5 questions produces no F1 = 1.0 rows and an `abstention_accuracy` figure.+*Verify:* `python -m locomo evaluate --samples conv-26 --categories 5 --no-skip-adversarial --limit 10 --run-id adv-check && python -m locomo report --run-id adv-check`++**0.5 Quarantine the contaminated prompt and re-baseline.** S — `benchmarks/locomo/prompts.py`.+Remove the conv-26-derived specifics from `ANSWER_SYSTEM` (`prompts.py:9,13,14`): the `Melanie's slipper` example, the charity-race Sunday rule, the `"Likely no; ..."` hedge template, and the enumeration keyword list that mirrors conv-26 gold answers. Keep the generalizable rules ("never return an empty answer", "prefer relative phrasing when the dialogue uses it", "do not abstain when thematically related evidence exists").+*Acceptance:* no string in `prompts.py` appears in any conv-26 gold answer; the re-baselined score is recorded as the new reference with its drop documented.+*Verify:* a script asserting that no gold answer from `locomo10.json` shares a 4-gram with `ANSWER_SYSTEM`, then `python -m locomo evaluate --samples conv-26 --run-id baseline-clean`++**CHECKPOINT 0.** Re-report all nine historical runs with the fixed reporting path, and publish a corrected history table with McNemar p-values between consecutive runs. Expect the headline to fall from 86.2%. That drop is not a regression; it is the removal of a measurement error. Do not proceed until the maintainer has seen and accepted the corrected baseline — this is where open questions 1, 2, and 6 must be answered.++### Phase 1 — Make the judge an instrument instead of an assumption++Depends on phase 0.++**1.1 Export a human labelling set.** S — `benchmarks/locomo/cli.py`.+A `label-export` command emitting a CSV stratified by category and judge verdict, with question, gold, prediction, judge verdict, judge reason, and a blank human column. Target 250 items weighted toward disagreement-prone categories.+*Acceptance:* CSV has 250 rows with proportional category representation and no leaked judge verdict in the column a human fills.+*Verify:* `python -m locomo label-export --run-id baseline-clean --n 250 --out benchmarks/runs/labels/round1.csv`++**1.2 Compute judge agreement.** S — `benchmarks/locomo/metrics.py`, `benchmarks/locomo/cli.py`.+Ingest completed labels; report raw agreement, Cohen's kappa, per-category kappa, and a confusion matrix. Report both raw and kappa side by side, since `2606.19544` shows the gap between them is the interesting quantity.+*Acceptance:* `judge-validate` prints kappa with a CI and flags any category where kappa is below 0.6.+*Verify:* `python -m locomo judge-validate --labels benchmarks/runs/labels/round1.csv`++**1.3 Audit judge self-consistency and self-preference.** S — `benchmarks/locomo/judge.py`, `benchmarks/locomo/cli.py`.+Re-judge a 100-item subset three times to measure flip rate; re-judge with a different model family to quantify the self-preference delta, given answerer and judge are currently the same model (`config.py:17-18`).+*Acceptance:* a reported flip rate and a cross-family accuracy delta; if the delta exceeds 3pp, the judge model is changed and phase 0's baseline re-run.+*Verify:* `python -m locomo judge-audit --run-id baseline-clean --repeats 3 --alt-model <other-family>`++**CHECKPOINT 1.** If kappa is below 0.6 in any category, that category's historical numbers are not admissible evidence and must be labelled as such in `CHECKPOINT_NOTES.md`. Answers open questions 4 and 5.++### Phase 2 — Attribution and honest cost++Depends on phase 1 for the E bucket.++**2.1 Turn-level evidence recall.** S — `benchmarks/locomo/metrics.py`.+Track `D<session>:<turn>` rather than collapsing to `session_N` (`metrics.py:93-103`), and require the harness to record which turns appear in the returned passages.+*Acceptance:* turn-level recall is reported alongside session-level and is lower than it, confirming the coarse metric was optimistic.+*Verify:* `python -m locomo report --run-id baseline-clean && grep "evidence_turn_recall" benchmarks/runs/baseline-clean/report.json`++**2.2 Entailment-based context ceiling.** M — `benchmarks/locomo/metrics.py`, plus a new judge-side helper.+Replace the `token in blob` substring test (`metrics.py:138`) with a claim-level entailment check on gold against the retrieved context, following RAGAS/RAGVUE (`2309.15217`, `2601.04196`). Retain the substring version as a cheap fallback and report both, so the divergence is visible.+*Acceptance:* the ceiling no longer falls below measured accuracy on any historical run — the `v3` impossibility in section 2.5 disappears.+*Verify:* recompute on `v3` and assert ceiling ≥ accuracy.++**2.3 Full A-G attribution breakdown.** M — `benchmarks/locomo/metrics.py`, `benchmarks/locomo/report.py`.+Implement the seven-bucket assignment, per category, summing to 100%. Bucket A needs a graph-presence probe, which is a new client call; if that endpoint does not exist, merge A into B and say so explicitly in the report rather than guessing.+*Acceptance:* buckets sum to the row count for every historical run; the `push75-c` split reproduces roughly 2% / 4% / 11% for the retrieval, assembly, and generation stages.+*Verify:* `python -m locomo report --run-id locomo-conv26-push75-c && python -c "..."` asserting the buckets sum.++**2.4 Cost per correct answer, including server-side tokens.** M — `benchmarks/locomo/metrics.py`, `benchmarks/locomo/report.py`, `benchmarks/locomo/ingest.py`.+Separate judge tokens from product tokens; add ingestion tokens to `IngestRecord` (`ingest.py:22-36`); report cost per correct answer. Server-side tokens require plumbing from `invoke_loop.py:51`, which is a `src/` change and therefore out of this workstream's scope — until then, report the harness-side figure explicitly labelled as a **lower bound**, not as total cost.+*Acceptance:* `report.md` shows product tokens, judge tokens, and cost per correct answer separately, and the `push75-a` → `push75-c` comparison shows the 68% cost-per-correct increase.+*Verify:* `python -m locomo report --run-id locomo-conv26-push75-c && grep "cost_per_correct" benchmarks/runs/locomo-conv26-push75-c/report.json`++**2.5 Fix the latency measurement.** S — `benchmarks/locomo/client.py`.+Measure from first attempt to final response including backoff (`client.py:54-78`), and report attempt count.+*Acceptance:* a run with an induced retry shows latency exceeding single-attempt duration and a retry count above zero.+*Verify:* `python -m locomo smoke --samples conv-26` against a server returning one 429.++**CHECKPOINT 2.** Publish the attribution breakdown for the corrected baseline and re-rank the sibling workstreams' priorities against it. If the generation bucket still dominates as it does in `push75-c`, the retrieval workstream's next task list needs revisiting before it is executed.++### Phase 3 — Scale the evidence base++Depends on phase 2 so that new data lands in a trustworthy harness.++**3.1 Tuning / held-out conversation split.** S — `benchmarks/locomo/config.py`, `benchmarks/locomo/cli.py`.+Declare 4 tuning conversations (including conv-26) and 6 held-out. Refuse to report a headline from tuning conversations without an explicit flag.+*Acceptance:* `evaluate` on a tuning conversation prints a banner that the result is not a headline; the held-out path forbids prompt-hash mismatch against the recorded baseline.+*Verify:* `python -m locomo evaluate --split tuning --limit 5` and confirm the banner.++**3.2 Full-dataset run.** M — orchestration only.+All 10 conversations, all categories including abstention. This is roughly 13× current volume; at `push75-c`'s rate that is on the order of 75-100M harness tokens, so the cost question (open question 3) must be answered first.+*Acceptance:* a run with n ≈ 1,986, per-category n at least 96, and per-category CI widths under 12pp.+*Verify:* `python -m locomo evaluate --all-samples --no-skip-adversarial --run-id full-v1`++**3.3 Multi-hop shortcut ablation.** M — new module under `benchmarks/locomo/`.+For each multi-hop question, re-answer with each annotated evidence turn ablated, and record how many turns are strictly necessary, per `1906.02900` and `1906.07132`.+*Acceptance:* a distribution of necessary-evidence counts; the fraction of "multi-hop" questions answerable from a single turn is reported.+*Verify:* `python -m locomo shortcut-probe --run-id full-v1 --categories 1`++**3.4 LongMemEval adapter.** M — new dataset adapter plus an ingestion path.+Start with the smallest history configuration. Map its five abilities onto the existing category machinery.+*Acceptance:* a LongMemEval run produces a report with per-ability breakdown, including knowledge-updates, which LoCoMo cannot measure at all.+*Verify:* `python -m locomo evaluate --dataset longmemeval --limit 50 --run-id lme-smoke`++**CHECKPOINT 3.** With full-dataset CIs and the shortcut probe in hand, the multi-hop and temporal claims can be stated with real confidence for the first time. This is the earliest point at which "the memory layer is X% accurate" is a defensible sentence.++### Phase 4 — Gate it++Depends on phase 3 for stable baselines. Tier 1 can land after phase 0.++**4.1 Tier 1 gate.** S — `.github/workflows/`.+Replace `staging.yaml:15` with a real job: `unittest` discovery, `selftest-metrics`, the prompt-hash check, and a no-gold-4-gram-in-prompt assertion.+*Acceptance:* a PR that edits `ANSWER_SYSTEM` fails with a re-baseline instruction; a PR that breaks a metric function fails.+*Verify:* `python -m unittest discover -s tests -v && python -m locomo selftest-metrics`++**4.2 Tier 2 gate.** M — workflow plus a comparison harness.+One held-out conversation against a live server, asserting no significant McNemar regression overall or on multi-hop and temporal, plus zero errored rows, unique-equals-scored, and cost-per-correct within 20%. Latency starts as a recorded warning.+*Acceptance:* an artificially degraded retrieval config fails the gate; a no-op PR passes; the job stays under 20 minutes.+*Verify:* `python -m locomo gate --baseline <ref> --candidate <run> --tier 2`++**4.3 Tier 3 nightly.** S — workflow.+Full dataset, attribution breakdown, judge kappa floor, faithfulness sample.+*Acceptance:* nightly artefact contains all sections; a judge-kappa drop marks the run inadmissible rather than reporting an accuracy delta.+*Verify:* manual dispatch of the workflow.++**CHECKPOINT 4.** Every subsequent workstream's claims are now falsifiable in CI. This is the point at which the roadmap in `00-scope-and-constraints.md` becomes provable rather than plausible.++### Phase 5 — The unmeasured surfaces++Depends on phase 4 only for CI plumbing; can run in parallel with phase 3.++**5.1 Synergies test suite.** S — `tests/`.+First tests for `EntitySinergyRetriever`, covering the scoring constants (`entity_sibilings.py:26,29,32,35`) and asserting the `polarity` contract. The `polarity` test will fail against current behaviour (`entity_sibilings.py:53`); land it as an expected failure with a reference to open question 11 rather than deleting the assertion.+*Acceptance:* tests exist and run in tier 1; the polarity gap is recorded as a known failure, not hidden.+*Verify:* `python -m unittest tests.test_entity_synergies -v`++**5.2 Synergies structural metrics.** M — new benchmark module.+20-30 seed entities; report intra-list diversity, catalogue coverage over entity types, mean graph distance from seed as an unexpectedness proxy, and result-set size distribution (relevant given the unbounded return at `entity_sibilings.py:437`).+*Acceptance:* a report with all four, and a stable baseline for a future gate.+*Verify:* `python -m locomo synergy-report --brain locomoconv26 --seeds benchmarks/data/synergy_seeds.json`++**5.3 LLM serendipity diagnostic.** S — extends 5.2.+Chain-of-thought serendipity judging per `2508.17571`, reported and explicitly **non-gating**, annotated with the 21.5% human correlation ceiling from `2507.17290` so nobody later mistakes it for ground truth.+*Acceptance:* the metric appears in the report with its stated ceiling next to it.+*Verify:* `python -m locomo synergy-report --with-llm-judge`++**5.4 MCP agent-task suite.** M — new module.+20-30 tasks over the five tools, run with and without memory access. Report task completion, tool-call success rate, and tokens per completed task.+*Acceptance:* a with-versus-without delta; ambiguous `"Unauthorized"` string returns (`main.py:212,266,288,321`) are counted separately from empty results and reported as a measurability defect.+*Verify:* `python -m locomo agent-eval --tasks benchmarks/data/mcp_tasks.json`++**5.5 Ingestion quality set.** M — new annotation plus scorer.+Hand-annotate entities and relations for 3-5 sessions; score entity recall, triple precision, omission, and hallucination, adapting KGCQual's split (`2607.10212`).+*Acceptance:* a per-session scorecard, and a demonstration that the scores separate two different extraction configurations.+*Verify:* `python -m locomo kg-quality --sessions benchmarks/data/kg_gold/`++### Ranked by expected value per unit cost++| Rank | Task | Cost | Why |+| --- | --- | --- | --- |+| 1 | 0.3 paired significance | S | Reverses conclusions about past work using data already on disk |+| 2 | 0.5 decontaminate the prompt | S | Without it the headline is not a memory metric |+| 3 | 0.2 de-duplicate and fail loudly | S | A fully broken run currently renders as a clean report |+| 4 | 0.1 reproducibility tuple | S | Nothing is attributable to code today |+| 5 | 2.3 attribution breakdown | M | Redirects every other workstream's effort |+| 6 | 1.1-1.2 judge calibration | S+S | Bounds the error on every number |+| 7 | 3.1 tuning/held-out split | S | Structurally prevents recurrence of finding 2.1 |+| 8 | 4.1-4.2 CI gates | S+M | Makes all of the above permanent |+| 9 | 3.2 full dataset | M | Halves CI widths; largest token cost |+| 10 | 2.4 cost per correct answer | M | Reveals cost regressions currently invisible |+| 11 | 3.4 LongMemEval | M | Only way to measure knowledge updates |+| 12 | 5.1-5.2 synergies | S+M | Unblocks an unmeasured product surface |+| 13 | 5.4 MCP agent tasks | M | Highest ambiguity about what to measure |+| 14 | 5.5 ingestion quality | M | Most annotation effort per unit of insight |++Explicitly **not worth it here**, despite being fashionable: adding HotpotQA or MuSiQue as headline benchmarks (wrong domain, and a second unreliable surface before the judge is calibrated); simulation-based recommender evaluation (`2209.08642`, no interaction log exists); inline citation verification on the `/retrieve/context` hot path (`2606.28358` shows citations are a weak guarantee, and the sub-second budget forbids the extra pass); and replacing the gold-answer judge with reference-free RAGAS metrics (LoCoMo has gold answers, which are the stronger signal — faithfulness belongs alongside, not instead).++## Risks++**Re-baselining will look like a large regression.** Phase 0 removes prompt contamination, counts duplicates correctly, and scores abstention. The headline will fall from 86.2%, possibly substantially. *Detection and mitigation:* publish the corrected history table at checkpoint 0 with the drop attributed line by line to each fix, and record in `benchmarks/runs/CHECKPOINT_NOTES.md` that pre-phase-0 numbers are not comparable to post-phase-0 numbers. The risk is not the drop; it is someone later comparing across the boundary.++**The judge may fail calibration, invalidating the recorded history.** If kappa comes back below 0.6 for multi-hop or temporal, the categories the project cares most about have no usable history. *Detection:* phase 1.2. *Mitigation:* label 250 items before committing to any accuracy target, and treat the labelled set as a permanent asset that is re-used at every judge change.++**A stricter gate gets disabled rather than satisfied.** A latency gate that fails on day one — which it would, at p95 of 5.4-6.9s against a sub-second budget — invites a bypass that then covers real regressions. *Mitigation:* latency starts as a recorded warning; only accuracy and correctness gates block initially; every override requires an explicit PR label so bypasses are countable.++**Full-dataset runs may be unaffordable.** Extrapolating `push75-c`, 10 conversations is on the order of 75-100M tokens per full run. *Detection:* cost per correct answer from 2.4. *Mitigation:* tier 2 uses one held-out conversation; the full run is nightly or release-only; the judge is the cheapest model that passes calibration.++**Attribution may be miscalibrated and misdirect effort.** The A-G buckets depend on a graph-presence probe (bucket A) that may not have an endpoint, and on entailment quality (buckets C/D). A wrong split sends sibling workstreams after the wrong stage — precisely the failure this document accuses the current `answerer_gap` of. *Detection:* cross-check the split against manual inspection of 30 failures. *Mitigation:* if the probe does not exist, merge A into B and label it as merged in the report, rather than inventing a boundary.++**Measuring more surfaces raises maintenance cost enough that measurement stops.** Six benchmark families is a lot for one project. *Mitigation:* only tiers 1 and 2 are on the critical path of a PR; everything else is nightly, and a nightly suite that breaks does not block shipping. Keep the synergies and MCP suites at 20-30 items — deliberately small tripwires, not leaderboards.++**A live-server gate makes CI flaky.** Tier 2 depends on a running BrainAPI and an LLM provider. *Detection:* track gate failure causes separately from gate failures. *Mitigation:* distinguish infrastructure failure from quality failure in the exit code, retry infrastructure failures, and never let an infrastructure failure read as a quality pass.++**Optimizing the new metrics instead of the product.** Faithfulness, diversity, and serendipity are all gameable, and the pattern this document documents in section 2.1 — tuning a prompt until a metric moves — will recur. *Mitigation:* the tuning/held-out split from 3.1 is the structural defence, and it must apply to every new metric, not just to accuracy.++**The uncommitted working tree makes the current state unattributable.** Seven harness files and three `src/` files are modified but uncommitted, and `push75-d` was produced from that tree. *Detection:* the dirty-tree flag from 0.1. *Mitigation:* commit or stash before establishing any baseline; treat every existing run in `benchmarks/runs/` as provenance-unknown until re-run.diff --git a/docs/research/05-temporal-truth.md b/docs/research/05-temporal-truth.mdnew file mode 100644index 0000000..5fe989f--- /dev/null+++ b/docs/research/05-temporal-truth.md@@ -0,0 +1,628 @@+# 05 — Temporal Truth (cross-cutting)++**Concern:** how BrainAPI knows what is *currently* true, traced from write time to read time.+**Confirmed problem being investigated:** superseded facts are returned as if they were current truth (`docs/research/00-scope-and-constraints.md:12`).+**Binding constraint honoured throughout:** `/retrieve/context` must stay sub-second and cheap, so any temporal filtering on that path must be answerable by index or precomputed structure — never by an LLM call at query time (`00-scope-and-constraints.md:20-25`).++---++## Answer up front++Temporal correctness breaks at **five stages simultaneously**, but there is one seam where responsibility is visibly *dropped* and one bug that makes every other stage inert.++**The seam (produces the reported symptom):** `get_context` returns two independent evidence channels in the same response — graph triples and raw text passages. A current-truth filter exists and is applied to the first channel only:++```420:426:src/services/api/controllers/retrieve.py+def _is_currently_valid(predicate: Predicate) -> bool:+ props = getattr(predicate, "properties", None) or {}+ if props.get("invalid_at"):+ return False+ if getattr(predicate, "deprecated", False):+ return False+ return True+```++It is called in exactly one place in the entire repository, guarding the graph-fact path (`src/services/api/controllers/retrieve.py:787`). The passage path (`src/services/api/controllers/retrieve.py:667-725`) and the `historical_context` path (`src/services/api/controllers/retrieve.py:851-880`) are pure vector + keyword fusion with **no temporal predicate at all**. Those passages are the verbatim sentences that stated the now-superseded fact. So the system can correctly invalidate the edge and still hand the caller the original sentence asserting it, in the same payload, unlabelled. That is precisely "superseded facts come back as current truth."++**The bug that makes everything else inert:** `src/utils/dates.py:59-68` cannot parse the timestamp format the system actually receives, so no comparable time value ever enters the graph. Verified empirically against the real benchmark data:++| input | `parse_date_string` | `resolve_relative_date('yesterday', ref)` |+| --- | --- | --- |+| `'1:56 pm on 8 May, 2023'` (real LoCoMo `session_1_date_time`) | `None` | `'yesterday'` (unresolved) |++`_DATE_INPUT_FORMATS` (`src/utils/dates.py:5-16`) contains ten formats and **not one includes a time component**, so every LoCoMo session timestamp fails to parse. `resolve_relative_date` then bails at `src/utils/dates.py:91-98` (`ref is None → return cleaned`) and returns the relative expression verbatim. All of `yesterday`, `last week`, `last Tuesday`, `next quarter`, `last summer`, `two weeks ago`, `a few months ago`, `last month` pass through unresolved for the entire benchmark.++**One bug or a missing capability?** A **missing capability wrapped around three real bugs**. Fixing `dates.py` alone changes nothing observable, because there is still no validity-interval representation (Stage 3), no notion of which claims are single-valued (Stage 2), and the read filter covers one of roughly six read paths (Stage 4). The three concrete bugs are listed in *Guarantees and where they break* (G1, G3, G6).++---++## Stage-by-stage assessment++| # | Stage | Status | Evidence |+| --- | --- | --- | --- |+| 1 | **Write time** — event vs ingestion vs document time | **Partial, broken in practice** | `source_timestamp` is the only temporal input (`src/constants/tasks/ingestion.py:81-87`) and is *document time*. It is injected as a reference date into the Scout prompt (`src/core/agents/scout_agent.py:412-414`, `src/constants/prompts/scout_agent.py:144,154`) and used for post-hoc resolution (`src/workers/tasks/ingestion.py:192-205`). Both mechanisms are defeated by `src/utils/dates.py:59-68` for the format actually supplied. `src/core/agents/temporal_agent.py` is a 9-line stub — the named component does not exist. |+| 2 | **Conflict detection** — does anything notice supersession? | **Absent** (one unsound heuristic) | `_invalidate_superseded_relationships` (`src/workers/tasks/ingestion.py:465-505`) is the only mechanism; it is a syntactic predicate-name match with no recency comparison. `src/core/agents/validator_agent.py` and all three `src/services/triggers/*.py` are 9-line stubs. The Janitor has zero temporal awareness (no `happened_at`/`valid_at`/date logic in `src/core/agents/janitor_agent.py`), and its prompts cover direction, dedup and schema only — no prompt in `src/constants/prompts/` mentions contradiction, supersession or "no longer true". |+| 3 | **Representation** — what the graph physically stores | **Partial, ad hoc** | Edges may carry `valid_at`, `invalid_at`, `deprecated`, `source_timestamp` (`src/workers/tasks/ingestion.py:418-422,494-503`; `src/core/saving/identity.py:104-108`). Nodes carry only free-text `happened_at`. Format is `dd/mm/YYYY` (`src/utils/dates.py:76`) — not sortable, not a temporal type. No `valid_to`, no invalidation edges, no bitemporal pair. **No `CREATE INDEX` / `CREATE CONSTRAINT` anywhere in `src/lib/neo4j/client.py`**, so no indexed temporal filter is possible today. Postgres has `inserted_at`/`created_at TIMESTAMPTZ` (`src/lib/postgresql/data.py:60,71,84`) never used for retrieval. |+| 4 | **Read time** — filtering, ranking, current vs historical | **Partial — 1 of ~6 paths** | Filtered: graph facts in `get_context` (`src/services/api/controllers/retrieve.py:787`). Unfiltered: passages (`:667-725`), `historical_context` (`:851-880`), `/retrieve/entity/status`, `/retrieve/entity/synergies`, and all of `src/core/search/` (zero temporal keywords across all 8 modules; `src/core/search/context.py` is a 9-line stub). No `as_of` parameter exists anywhere in `src/` and the response exposes no validity fields, so a caller cannot distinguish current from historical. |+| 5 | **Measurement** — is supersession scored? | **Partial — measures the wrong thing** | Per-category reporting exists and category 2 is `temporal` (`benchmarks/locomo/config.py:21-27`, `benchmarks/locomo/metrics.py:155-158,236-238`). But cat-2 scores **0.86** in the best run — *above* multi-hop's 0.78 (`benchmarks/runs/locomo-conv26-push75-c/report.json`). Every cat-2 question is "When did X happen?" (event localization), and across all 1,986 LoCoMo questions only ~12 contain a current-truth marker (`current` 3, `used to` 6, `still` 1, `no longer` 1, `latest` 1, `now` 0, `anymore` 0). Supersession is ~0.6% of the question set: **the maintainer's #1 temporal complaint is invisible to CI.** |++---++## What this workstream does (traced flow)++### Write path++1. **Ingress.** `source_timestamp` arrives as a free-text string described as "Reference timestamp for the source content (e.g. session date). Used to resolve relative dates like 'yesterday' into absolute dates" (`src/constants/tasks/ingestion.py:81-87`). This is the *only* temporal input to the system. There is no separate event-time or valid-from input.++2. **Prompt-side resolution.** It is passed as `reference_time` into the Scout (`src/core/saving/auto_kg.py:121` → `src/core/agents/scout_agent.py:412-414`), which renders `Reference date for resolving relative dates: {reference_time}` into the prompt. The prompts mandate `DD/MM/YYYY` output stored as `happened_at` (`src/constants/prompts/scout_agent.py:153-154,174-175`; `src/constants/prompts/architect_agent.py:108`).++3. **Code-side resolution.** In the worker, `_normalize_relationship_dates` re-resolves `DATE`-typed entity names and `happened_at` values against `reference_time` (`src/workers/tasks/ingestion.py:192-205`) using `resolve_relative_date` (`src/utils/dates.py:80-123`).++4. **Stamping.** `stamp_provenance` attaches `source_chunk_ids` and `source_timestamp` to relationships and both endpoints (`src/core/saving/identity.py:85-109`). Relationships additionally get `valid_at = source_timestamp` when unset (`src/workers/tasks/ingestion.py:418-422`).++5. **Invalidation.** After each relationship is written, `_invalidate_superseded_relationships` fetches the subject's neighbours and stamps `invalid_at` + `deprecated: True` on any same-predicate-name edge pointing at a different object (`src/workers/tasks/ingestion.py:465-505`, called at `:972-975`).++6. **Consolidation.** `consolidate_graph` runs the Janitor then the KG agent for name/connection normalisation and dedup (`src/core/layers/graph_consolidation/graph_consolidation.py:31-94`). Nothing in this layer reads or writes time.++### Read path++`get_context` fans out over query variants, collecting two independent channels:++- **Graph facts** — `get_event_centric_neighbors` returns 5-tuples `(n, r, m, r2, b)` joined on `flow_key` (`src/lib/neo4j/client.py:1669-1681`); candidates are dropped unless *both* predicates pass `_is_currently_valid` (`src/services/api/controllers/retrieve.py:787-788`), then ranked by seed distance, optionally re-ranked by personalized PageRank, then optionally LLM-filtered.+- **Text passages** — `_retrieve_passages` fuses vector and keyword hits via RRF (`src/services/api/controllers/retrieve.py:667-725`), plus `historical_context` (`:851-880`). Neither consults time.++Both channels are concatenated into the response. Only one is temporally guarded.++---++## Guarantees and where they break++The guarantee this workstream should provide: *given a query and an optional as-of time, BrainAPI returns the facts that were true at that time, labels them as current or historical, and never presents a superseded assertion as current — while retaining full provenance of how the fact evolved.* None of those four clauses holds today.++Ranked by impact on answer accuracy.++### G1 — Reference time is unparseable, so no comparable time value ever enters the graph *(bug)*++`_DATE_INPUT_FORMATS` (`src/utils/dates.py:5-16`) has no time-bearing format. The real LoCoMo `session_N_date_time` value `'1:56 pm on 8 May, 2023'` therefore yields `parse_date_string → None`, `resolve_relative_date` returns the input unchanged (`src/utils/dates.py:91-98`), and `valid_at` is stamped with that raw string (`src/workers/tasks/ingestion.py:418-422`). Downstream, `invalid_at` is set from that same unorderable string (`src/workers/tasks/ingestion.py:499-500`).++Secondary gap: `_RELATIVE_PATTERNS` (`src/utils/dates.py:18-38`) is anchored `^…$` and covers nine literal patterns. Even with a valid reference date, `next quarter`, `last summer`, `two weeks ago` (word numerals), `a few months ago` and any expression with trailing words are unsupported. This is the "relative expressions resolved against a reference date" question in the brief: the intent is present in two places, the capability is not.++### G2 — Event time and document time are conflated *(design gap)*++`valid_at` on an edge is set to `source_timestamp`, i.e. *when the system read the document*, while the event's own date lives on the node as `happened_at`. Two different clocks, never reconciled. A LoCoMo session dated May 2023 discussing an event from 2019 produces an edge whose "validity" is 2023. Zep's model keeps these strictly separate as `t_valid`/`t_invalid ∈ T` (event timeline) versus `t'_created`/`t'_expired ∈ T'` (transaction timeline) — see *Frontier techniques* F1.++### G3 — The invalidation heuristic is unsound in both directions *(bug)*++`_invalidate_superseded_relationships` (`src/workers/tasks/ingestion.py:481-503`) invalidates an existing edge when: same tail, same predicate name, different tip. Three independent defects:++- **No recency comparison.** It never checks whether the existing edge is *older* than the incoming one. Out-of-order ingestion (Celery concurrency, chunk reordering, backfill) will invalidate the *newer* fact and leave the older one live. There is no `if existing.valid_at < new.valid_at` guard anywhere in the function.+- **Predicate names are deliberately generic, so the match condition is not a contradiction test.** The prompts explicitly instruct general labels — `"TARGET_PRODUCT_OBJECT_CROISSANTS"=wrong, "TARGETED"=correct` (`src/constants/prompts/architect_agent.py:382,424`) — and the schema is `MADE` / `TARGETED` / `OCCURRED_WITHIN` (`src/constants/prompts/architect_agent.py:16-20`). "Same subject, same predicate, different object" is therefore the *normal shape of accumulated history*, not a functional-dependency violation. Every second event by the same actor triggers invalidation of the first. Because `_is_currently_valid` requires **both** `r` and `r2` to be valid (`src/services/api/controllers/retrieve.py:787`), invalidating the actor-side `MADE` edge silently deletes the whole triple from `get_context`.+- **Asymmetric coverage.** Event hubs are fresh nodes per event, so the object-side (`TARGETED`) edge of a new hub has no prior siblings and is never invalidated; only actor-side edges are ever touched.++Net effect: the mechanism both over-fires (destroying true history) and under-fires (missing genuine supersession that changes predicate label or object type). It is a coin flip, not a rule.++### G4 — The read-time filter covers one of ~six paths *(the seam)*++Detailed in *Answer up front*. The ambiguity is structural: nothing in the codebase assigns ownership of temporal validity to a layer. `src/core/search/` — the module that ought to own it — contains no temporal logic whatsoever, and its `context.py` is an empty stub. The one filter lives in the API controller, so every other consumer (`/retrieve/entity/status`, `/retrieve/entity/synergies`, `/retrieve/hops`, neighbours, MCP tools) silently inherits no filtering.++### G5 — Callers cannot express or observe time *(design gap)*++`GetContextRequestBody` (`src/services/api/constants/requests.py:396-400`, including the uncommitted `use_ppr` / `sufficiency_retry` additions) has no `as_of` or `temporal_mode`. `grep` for `as_of|point_in_time|valid_from|valid_to` across `src/` returns nothing. `GetContextTriple` carries no validity fields. So a caller can neither ask "what was true in March 2024" nor tell whether what came back is current.++### G6 — The only recency scorer is a permanent no-op *(bug)*++```226:234:src/core/search/entity_info.py+ if node.happened_at:+ try:+ happened_at = node.happened_at+ if isinstance(happened_at, str):+ happened_at = datetime.fromisoformat(happened_at)+ days_ago = max(0, (datetime.now() - happened_at).days)+ except Exception:+ days_ago = 0+ recency = 1 / (1 + np.log1p(days_ago)) if days_ago > 0 else 1.0+```++`normalize_date_string` writes `dd/mm/YYYY` (`src/utils/dates.py:76`); `datetime.fromisoformat` cannot parse that and raises; the bare `except` sets `days_ago = 0`; `recency` becomes the constant `1.0` for every node. The `* 0.2` recency term at `src/core/search/entity_info.py:235` contributes an identical constant to every candidate and therefore has zero effect on ranking. Note the normalizer actively *destroys* the one format the consumer can read — an ISO input `2024-03-08` is matched by `"%Y-%m-%d"` in `_DATE_INPUT_FORMATS` and rewritten to `08/03/2024`.++### G7 — Batch-level time smearing *(bug, minor but real)*++```754:766:src/workers/tasks/ingestion.py+ reference_time = None+ source_chunk_id = None+ for rel in relationships:+ props = rel.properties or {}+ if not reference_time:+ reference_time = props.get("source_timestamp") or props.get("valid_at")+```++`reference_time` is taken from the *first* relationship in the batch that has one, then applied to every relationship in the batch. When a batch spans chunks from different sessions, all facts inherit one session's date.++### G8 — Measurement hides the failure, and the reported score is prompt-compensated *(methodology gap)*++Beyond the ~0.6% coverage noted in the stage table, the cat-2 score is not evidence that the *pipeline* handles time. The LoCoMo answer prompt carries hand-written, benchmark-specific temporal rules:++> `- TEMPORAL: Prefer the relative phrasing used in dialogue when available… For "last weekend" / charity race timing relative to a Thursday/Friday session, prefer the preceding Sunday over Saturday when choosing one day.` (`benchmarks/locomo/prompts.py:13`)++The answer LLM is re-deriving temporal facts at read time from the session date embedded in the raw passage text, guided by rules tuned to this dataset. That produces 0.86 on cat 2 while the pipeline never normalises a single relative expression — and none of it transfers to a real `/retrieve/context` consumer, who has neither those rules nor a reason to invent them. The healthy signal already present is `answerable_rate = gold_in_context` (`benchmarks/locomo/metrics.py:176`), which decouples retrieval from answering; that is the hook a supersession metric should attach to.++### Deliberate trade-offs (not bugs)++- Running invalidation at write time rather than query time is **correct** given the sub-second budget, and should be kept.+- Generic predicate labels are a deliberate, defensible schema choice for multi-hop traversal. The fix is not to specialise labels but to add an explicit claim identity (see below).+- Doing entity/event resolution before invalidation is the right order.++---++## The append-only tension, stated directly++`README.md:198-200` says nothing is deleted, events are never merged away, and the provenance trail always survives. **The current implementation already violates this.** `_invalidate_superseded_relationships` mutates existing edges in place via `update_properties` (`src/workers/tasks/ingestion.py:494-503`), and it does so in the least auditable way available: it records *that* an edge became invalid but not *why*, or *by what*. The philosophy and the code have already diverged; the question is only how to resolve it deliberately.++There is no fundamental tension between append-only provenance and current-truth queries, provided you separate two things the codebase currently conflates:++- **Assertions** — "at ingestion time T', source S claimed P held from t_valid". These are immutable and append-only. This is the provenance record.+- **Validity** — "P is currently true". This is a *derived projection* over assertions, not a fact in its own right.++Append-only should be a property of assertions. Validity is a materialised view. The only reason to store validity on the edge at all is the sub-second budget: a projection you must compute at query time is a projection you cannot afford. So the resolution is to store validity as a **write-once, monotone materialisation** of an appended invalidation fact — a cache with a source of truth behind it.++---++## Proposed minimal representation change++Four fields on every relationship, plus one new edge type, plus one classification. Nothing is deleted; nothing is rewritten twice.++### 1. Bitemporal quad on the edge — all epoch-seconds integers++| Field | Meaning | Timeline |+| --- | --- | --- |+| `t_valid_from` | event time the fact began holding | valid time |+| `t_valid_to` | event time it stopped holding; `NULL` = currently true | valid time |+| `t_ingested` | when BrainAPI learned it | transaction time |+| `t_expired` | when BrainAPI marked it superseded | transaction time |++Integers, not `dd/mm/YYYY` strings. This is the single change that makes range scans, ordering and indexes possible, and it retires G1/G6 at the storage layer. Keep the existing `valid_at`/`invalid_at`/`happened_at` properties untouched for backwards compatibility; write the new fields alongside.++### 2. `statement_key` — the identity of the *claim*, not the event++This is the part with no equivalent in the codebase today, and it is what makes conflict detection sound. Supersession is not "same subject, same predicate, different object" (G3); it is "same claim slot, different filler". Derive:++```+statement_key = hash(subject_uuid, claim_type, object_role)+```++where `claim_type` is a normalised semantic slot taken from the **event hub's type** (`EMPLOYMENT`, `RESIDENCE`, `RELATIONSHIP_STATUS`, `HOBBY`), not from the generic edge label. Two assertions conflict only if they share a `statement_key`.++### 3. `cardinality` — single-valued vs multi-valued++The honest core of the problem: **supersession requires knowing which claims are functional.** "Employer" is single-valued (a new one supersedes the old); "hobby" and "books read" are multi-valued (a new one accumulates). Nothing in BrainAPI knows this today, which is exactly why G3 cannot be repaired by tightening the match condition. This is one boolean per `claim_type`, obtainable from the LLM at write time and cacheable per claim type — never at query time.++Invalidation then becomes sound and cheap: *on write, close the interval of the most recent live assertion sharing this `statement_key` if and only if `cardinality = single` and its `t_valid_from < new.t_valid_from`.* The recency guard fixes the out-of-order defect; `statement_key` + `cardinality` fixes the over-firing.++### 4. `SUPERSEDES` edge — append-only provenance++```+(new_event_hub)-[:SUPERSEDES {at, reason, confidence}]->(old_event_hub)+```++An appended fact, never a mutation. This makes provenance **strictly stronger than today**: you can reconstruct who superseded what, when, and why, which the current in-place stamp cannot express. `t_valid_to` on the superseded edge becomes a write-once monotone cache of this edge's existence, present only so the read path stays index-answerable.++### 5. Indexes — currently zero exist++```cypher+CREATE INDEX rel_valid_to IF NOT EXISTS FOR ()-[r:MADE]-() ON (r.t_valid_to);+CREATE INDEX rel_valid_from IF NOT EXISTS FOR ()-[r:MADE]-() ON (r.t_valid_from);+CREATE INDEX stmt_key IF NOT EXISTS FOR ()-[r:MADE]-() ON (r.statement_key);+```++repeated per relationship type in use. Without these the read-time filter degrades to a scan and the sub-second budget is at risk — this is not optional.++### 6. Chunk-level supersession rollup (fixes the actual reported symptom)++Edges already link to their source chunks via `source_chunk_ids` (`src/core/saving/identity.py:64-82`). At write time, roll that up per chunk into `supports_live_claims: bool` and `latest_claim_valid_to: int|NULL`. `_retrieve_passages` can then deprioritise and — more importantly — **annotate** passages that only support superseded claims.++Annotate rather than filter: a chunk typically contains many statements and only some are superseded, so dropping it would cost recall. Zep does exactly this, surfacing validity ranges into the context string rather than hiding facts (F1).++### Read-time query pattern this enables++```cypher+MATCH (n)-[r]-(m)-[r2]-(b)+WHERE n.uuid IN $seeds+ AND r2.flow_key = r.flow_key+ AND r.t_valid_from <= $as_of+ AND (r.t_valid_to IS NULL OR r.t_valid_to > $as_of)+ AND r2.t_valid_from <= $as_of+ AND (r2.t_valid_to IS NULL OR r2.t_valid_to > $as_of)+RETURN ...+```++This is a two-predicate index-backed extension of the existing query at `src/lib/neo4j/client.py:1669-1681`. Cost is an index lookup, not an LLM call — it satisfies the binding constraint by construction.++The API surface it unlocks, all on the cheap path:++- `temporal_mode: "current"` (default, `$as_of = now`) — the fix for the reported bug.+- `temporal_mode: "as_of", as_of: <ts>` — point-in-time truth, a genuinely new capability.+- `temporal_mode: "all"` — today's behaviour, for callers who want the full history.+- `GetContextTriple` gains `valid_from` / `valid_to` / `is_current`, and passages gain a supersession annotation, so a caller can finally *tell* (closes G5).++Deeper temporal reasoning — interval algebra, "what changed between X and Y", contradiction explanation — belongs on the MCP/deep surface where iteration is permitted, per `00-scope-and-constraints.md:20-25`.++---++## Open questions for the maintainer++1. In what format(s) will `source_timestamp` arrive from real (non-LoCoMo) callers — ISO-8601, epoch, or free text — and may we require ISO-8601 and reject unparseable values loudly instead of silently passing them through?+2. Should `valid_at` mean *event time* (when the fact began holding in the world) or *document time* (when we read it), given the code currently sets it to the latter while `happened_at` holds the former?+3. Is `_invalidate_superseded_relationships` (`src/workers/tasks/ingestion.py:465-505`) currently believed to be working, or already suspected of over-firing on generic `MADE` predicates?+4. Is it acceptable for a write-time LLM call to classify each claim type as single-valued or multi-valued, given that the result is cached per claim type and never invoked at query time?+5. Should `/retrieve/context` default to current-truth-only (a breaking change in returned facts) or keep today's behaviour and require callers to opt in via `temporal_mode`?+6. Does "append-only" forbid write-once monotone fields like `t_valid_to`, or only forbid deletion and destructive overwrite of assertions?+7. Should superseded *text chunks* be filtered out of passages, or returned with a "no longer current" annotation and left for the consumer to weigh?+8. Is a full re-ingest of the LoCoMo brain acceptable to validate the change, given `benchmarks/runs/CHECKPOINT_NOTES.md:13-20` already establishes that pattern for provenance work?+9. Should we add a temporal/supersession sub-benchmark (conflict-heavy, LTP-style) rather than relying on LoCoMo category 2, which we show measures event localization rather than supersession?+10. Are the recency term at `src/core/search/entity_info.py:225-235` and the `deprecated` flag on `Predicate` (`src/constants/kg.py:77-78`) intended to be load-bearing, or vestigial and safe to re-specify?++---++## Frontier techniques++### F1 — Bitemporal edge model with LLM-driven edge invalidation (Zep / Graphiti)++**arXiv:** 2501.13956 — *Zep: A Temporal Knowledge Graph Architecture for Agent Memory* (Rasmussen et al., Jan 2025).++**Mechanism (verified from the paper text, §2.1 and §2.2.3, not from recollection).** Zep implements an explicitly bi-temporal model: timeline `T` is the chronological ordering of events, timeline `T'` the transactional order of ingestion. Four timestamps are stored **on edges**: `t'_created` and `t'_expired ∈ T'` track when a fact was created or invalidated *in the system*; `t_valid` and `t_invalid ∈ T` track the range during which the fact *held true*. Each episode carries a reference timestamp `t_ref` which is what lets the extractor resolve relative and partial dates — the paper names exactly the cases BrainAPI drops: *"next Thursday," "in two weeks," "last summer"*. Invalidation uses an LLM to compare a new edge against semantically related existing edges; on a temporally overlapping contradiction it sets the affected edge's `t_invalid` to the invalidating edge's `t_valid`, consistently prioritising newer information along `T'`. Two further details matter here: edge deduplication is **constrained to edges between the same entity pair**, which is the scoping discipline BrainAPI's global predicate-name match lacks; and the retrieval constructor returns `t_valid, t_invalid` alongside each fact, with the context template stating *"These are the most relevant facts and their valid date ranges."*++**Reported gain.** 94.8% vs MemGPT's 93.4% on DMR; up to +18.5% accuracy on LongMemEval with 90% lower response latency, strongest on cross-session synthesis and long-term context maintenance.++**Cost.** One LLM call per candidate contradiction at **write** time. Four extra edge properties. No query-time model.++**Fit.** Very high, and it maps almost field-for-field onto the representation proposed above. The write-time LLM placement is explicitly compatible with the binding constraint. The gap Zep does not close for BrainAPI is that "semantically related existing edges" is scoped by entity pair, which is weaker than a `statement_key` + `cardinality` test when predicates are generic — and Zep is independently reported to score only 7% on MemoryAgentBench FactConsolidation (see F2), so its invalidation is *not* sufficient on its own.++**Verdict: adapt.** Take the four-timestamp bitemporal quad, `t_ref`-anchored extraction, and surfacing validity ranges into the returned context. Do not take LLM-only contradiction detection as the sole gate; combine it with the deterministic `statement_key` test.++### F2 — Deterministic version-aware aggregation instead of LLM freshness judgement++**arXiv:** 2606.01435 — *Don't Ask the LLM to Track Freshness: A Deterministic Recipe for Memory Conflict Resolution*.++**Mechanism.** On MemoryAgentBench's FactConsolidation task, the authors argue the bottleneck is the **assembly step**, not storage: baselines delegate conflict resolution to LLM-mediated retrieval or generation instead of version-aware aggregation. Replacing the LLM-judgement answer pipeline with candidate extraction plus a deterministic `max(serial)` (matched backbone, retrieval, chunking, `TOP_K`) yields +10.8 points on single-hop, widening from +8 at 6K context to +21 at 262K. The mechanism ports from `max(serial)` to `max(timestamp)`.++**Reported gain.** 78.0% FC-SH with gpt-4o-mini (94.8% with gpt-4o) vs HippoRAG-v2 54%, BM25 48%, Mem0 18%, **Zep/Graphiti 7%**. Multi-hop 30.2% → 51.5%. At matched 262K, +28 points over HippoRAG-v2.++**Cost.** Essentially zero — it *removes* an LLM call. The authors are candid that this is a whole-pipeline effect and isolating the resolver is future work, and that on LongMemEval's knowledge-update slice deterministic aggregation only ties LLM judgement (57.8% vs 64.4%, n=45).++**Fit.** Directly on point for the binding constraint: this is the strongest published evidence that current-value conflicts should be resolved by deterministic aggregation over a version field, which is exactly what `t_valid_from` + `statement_key` + `cardinality` provides. The Zep/Graphiti 7% figure is also the clearest warning that adopting F1's shape without a deterministic resolver is not enough.++**Verdict: adopt.** This is the cheapest high-leverage change available and it is the intellectual justification for the whole representation proposal.++### F3 — State-aware overlay: keep superseded records, label current / historical / transition++**arXiv:** 2607.01935 — *A-TMA: Decoupling State-Aware Memory Failures in Long-Term Agent Memory*.++**Mechanism.** Names BrainAPI's exact failure: **"ghost memory"** — old, current, and transition facts coexist in the memory bank, remain mixed during retrieval, and mislead the answer model. ATMA is an overlay that *keeps* superseded and transition records (compatible with append-only), builds evidence packets for the query's requested state view, and exposes explicit `current` / `historical` / `transition` labels to QA. The authors argue for **decoupled evaluation** of bank-, retrieval-, and answer-level failures, "since final QA accuracy can hide where ghost memory occurs."++**Reported gain.** On LTP (LoCoMo Temporal Plus, their conflict-heavy benchmark) Graphiti+ATMA improves conflict accuracy by **+0.240 absolute** over Graphiti. On LoCoMo, temporal F1 rises from **0.0295 to 0.1705**. Gains are host-dependent.++**Cost.** An overlay layer plus state labels on records; retrieval must be state-view aware.++**Fit.** Two things transfer directly. First, the label triple maps onto the proposed `is_current` / `valid_from` / `valid_to` response fields, and "keep superseded records" is precisely how append-only and current-truth are reconciled. Second — and this is the more important point for BrainAPI — their finding that *final QA accuracy hides where ghost memory occurs* is an independent confirmation of G8: LoCoMo cat-2 at 0.86 is exactly the kind of aggregate that conceals the problem. Note their LoCoMo temporal F1 numbers are not comparable to this repo's judge-accuracy metric.++**Verdict: adopt** the state-labelling and decoupled-evaluation discipline; treat LTP as the model for the missing conflict-heavy benchmark.++### F4 — Bitemporal property graphs as a storage model++**arXiv:** 2111.13499 — *Bitemporal Property Graphs to Organize Evolving Systems* (Rost et al., Oracle / University of Leipzig).++**Mechanism.** A one-year industrial collaboration producing four artefacts: a bitemporal property graph model, a temporal graph query language, a conception of continuous event detection, and a prototype bitemporal graph database. Establishes that property graphs with valid-time and transaction-time extensions are a prime candidate for organising evolving, multi-dimensional time-series-like relationships.++**Cost.** Conceptual; adopting the full model would mean a query-language layer BrainAPI does not need.++**Fit.** Useful as the canonical grounding that valid-time/transaction-time separation on *property graph edges* is a solved, industrially validated design rather than an invention — which matters because BrainAPI is a property graph and the proposal above adds exactly these two dimensions. The event-detection and dedicated-query-language portions are out of scope.++**Verdict: adapt** the model, reject the query-language and event-detection machinery as over-scoped.++### F5 — Forgetting-aware accuracy as a metric++**arXiv:** 2604.20006 — *From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents* (Memora).++**Mechanism.** Argues existing benchmarks frame long-term memory as fact retrieval from past conversations, giving little insight into consolidation or frequent knowledge updates — the precise criticism that applies to LoCoMo cat 2. Introduces **FAMA (Forgetting-Aware Memory Accuracy)**, a metric that *penalises reliance on obsolete or invalidated memory*, over weeks-to-months conversations across remembering / reasoning / recommending.++**Reported finding.** Across four LLMs and six memory agents: frequent reuse of invalid memories and failure to reconcile evolving memories; memory agents offer only marginal improvements.++**Cost.** Requires labelled supersession pairs. Cheap to approximate in this repo: the graph already knows which assertions were invalidated, so a superseded-fact-leakage rate can be computed directly from returned context without new human labels.++**Fit.** This is the missing Stage-5 instrument. A FAMA-style metric attaches naturally to the existing `gold_in_context` / `answerable_rate` hook (`benchmarks/locomo/metrics.py:176`), which already separates retrieval quality from answer quality.++**Verdict: adopt** as the primary regression guard, because without it no temporal fix here is verifiable.++### F6 — Reference-time temporal expression normalization and timeline extraction++**arXiv:** 2406.05265 — *TLEX: An Efficient Method for Extracting Exact Timelines from TimeML Temporal Graphs*; supporting context 2503.18085 (*GRAPHTREX*, clinical temporal relation extraction, +5.5% tempeval F1 and up to +8.9% on long-range relations).++**Mechanism.** TLEX converts TimeML annotations into exact timelines in a trunk-and-branch structure, checks temporal-graph consistency, identifies the specific relations involved in an inconsistency, and — the distinctive contribution — explicitly identifies **sections whose order is indeterminate**. Evaluated on 385 TimeML texts across four corpora: 123 are inconsistent, 181 have more than one main timeline, 2,541 indeterminate sections; sampling accuracy 98–100% at 95% confidence.++**Cost.** Full TimeML annotation is far heavier than BrainAPI needs.++**Fit.** Reject the pipeline; adopt two ideas. (a) **Indeterminate order is a first-class outcome.** BrainAPI currently coerces unresolvable expressions into free-text `happened_at` that silently poisons every consumer (G1/G6); representing "unknown/indeterminate" explicitly is strictly better than a string that fails to parse. (b) TIMEX-style normalization against an explicit reference time is a well-established task with known failure modes — worth treating as a bounded, testable unit (a fixture table of expression × reference date → expected ISO output) rather than a prompt instruction, which is what `src/constants/prompts/scout_agent.py:154` currently relies on.++**Verdict: adapt** narrowly — normalization contract plus an explicit indeterminate state.++### F7 — Knowledge-conflict taxonomy as shared vocabulary++**arXiv:** 2403.08319 — *Knowledge Conflicts for LLMs: A Survey*.++**Mechanism.** Categorises conflicts as context-memory, **inter-context**, and intra-memory, reviewing causes, LLM behaviour under each, and available mitigations.++**Fit.** Names BrainAPI's specific pathology precisely: returning a filtered graph triple *and* an unfiltered contradicting passage in one payload is an **inter-context conflict that the memory layer manufactures itself** (G4). Useful because it reframes the seam as a first-class known failure mode rather than an implementation slip, and because the survey's finding that LLMs resolve such conflicts unpredictably is the argument for why the memory layer must not delegate this to the consumer.++**Verdict: adopt** as framing/vocabulary. No implementation.++### F8 — Cheap NLI-based contradiction detection at write time++**arXiv:** 2410.04068 — *ECon: On the Detection and Resolution of Evidence Conflicts*.++**Mechanism.** Generates diverse validated evidence conflicts and evaluates detectors. Findings: NLI and LLM models show **high precision** in detecting answer conflicts, though weaker models suffer low recall; factual-consistency models struggle with lexically similar answer conflicts where NLI and LLMs do better; for resolution, LLMs often favour one side without justification and fall back on internal knowledge when they hold prior beliefs.++**Cost.** A small NLI model at write time — materially cheaper than an LLM call, and off the query path entirely.++**Fit.** Good as the *second* gate behind the deterministic `statement_key` test: use the cheap structural test to find candidate pairs, then NLI to confirm genuine contradiction before closing an interval. High precision is the property that matters, since a false invalidation destroys true history (the G3 over-firing failure). The resolution findings reinforce F2: do not let a model arbitrate freshness.++**Verdict: adapt** as an optional precision gate, only after the deterministic path is in place.++### F9 — Point-in-time primitives on the deep-navigation surface++**arXiv:** 2510.06002 — *Deterministic Legal Agents: A Canonical Primitive API for Auditable Reasoning over Temporal Knowledge Graphs*.++**Mechanism.** Argues that standard RAG over text fragments cannot preserve hierarchy, temporality, or causal provenance. Specifies a typed, atomic, composable primitive API mediating between a probabilistic LLM and a deterministic symbolic substrate, under a principle of **"Probability Isolation"**: uncertainty is confined to intent translation, semantic anchoring, and final synthesis, while structural, temporal, and causal traversals execute as deterministic operations. Shifts interaction from single-shot retrieve-then-generate to Reason–Act–Observe with primitives for point-in-time retrieval, context reconstruction, provenance tracing, and impact analysis. This is a formal architectural specification, not an empirical benchmark.++**Fit.** This is essentially a formalisation of the maintainer's own two-tier router intent (`00-scope-and-constraints.md:16-25`): deterministic, index-answerable temporal primitives on the cheap path; iterative reasoning confined to the deep surface. "Point-in-time retrieval" is the `as_of` primitive proposed above, and "impact analysis" is the natural MCP-tier companion ("what changed about X between T1 and T2"). No reported accuracy numbers, so cite it as design justification only.++**Verdict: adapt** for the MCP/deep tier once the cheap path is correct.++### F10 — Rejected: temporal KG completion and forecasting++**arXiv:** 2405.18106 (*TPAR*, unified interpolation/extrapolation TKG reasoning), 2607.14886 (*RAPTOR*, reachability-aware pretraining for RL-based multi-hop TKG forecasting on ICEWS).++**Why rejected.** These predict *missing or future* timestamped links. BrainAPI's problem is the opposite: it has the facts and cannot tell which are current. They also assume a dense corpus of quadruples with clean, orderable timestamps (ICEWS14/05-15/18) — the precise asset BrainAPI lacks, per G1. RL-based multi-hop path exploration is additionally incompatible with a sub-second budget. Revisit only if BrainAPI ever needs to *infer* unstated validity intervals, and only on the deep surface.++**Verdict: reject** for this workstream.++### F11 — Rejected for now: decay and strategic forgetting++**arXiv:** 2607.22562 (*SF-AMS*, strategic forgetting with utility-driven survival; reports +9.65 F1 multi-hop on Qwen2.5-7B and +6.91 F1 temporal on GPT-4o-mini over LightMem/Mem0/A-Mem), 2603.15642 (*CraniMem*, gated bounded memory with consolidation and pruning).++**Why rejected now.** Both *prune* low-utility memory. That is in direct tension with `README.md:198-200`, and more importantly it solves a different problem: BrainAPI's failure is not that the store is too large, it is that stored facts are unordered in time and unlabelled as to currency. Introducing forgetting before validity intervals exist would delete history while leaving the supersession bug intact — the worst of both. The reported temporal gains are also plausibly attributable to reducing distractor volume rather than to correct state resolution.++**Verdict: reject** for this workstream; reconsider under `03-memory-substrate` once scaling, not correctness, is the binding constraint.++### Context on what time-sensitive QA systems still fail at++Supporting evidence that this is a hard, unsolved area rather than a local implementation slip: **2311.08002** (*TempTabQA*, 11,454 QA pairs over 1,208 Infobox tables) finds top LLMs trail humans by **>13.5 F1**. **2310.19292** shows that fusing externally extracted temporal graphs into the input substantially improves temporal reasoning and sets SOTA on SituatedQA and three TimeQA splits — i.e. explicit temporal structure beats asking the model to infer time from raw text, which is the current de facto strategy at `benchmarks/locomo/prompts.py:13`. **2203.00255** (TSQA over temporal KGs) reports a 32% absolute error reduction on complex multi-step temporal questions by injecting timestamp ordering into KG embeddings, having identified that off-the-shelf temporal KG embeddings ignore temporal order — the same ordering deficiency G1 creates here. **2601.07978** independently measures Graphiti at 55–56% on LoCoMo versus 77–81% for mem0/RAG/full-context, attributing the gap to retrieval incompleteness rather than reasoning failure — a caution that adding temporal filters can cost recall if they over-fire, which is the central risk in R1 below.++---++## Implementation plan++Ordering principle: **measurement first.** The reported bug is currently invisible to CI (G8), and three of the defects (G1, G3, G6) fail silently inside bare `except` blocks. Any fix shipped before instrumentation is unfalsifiable.++Sizes per the task-breakdown discipline: S = 1–2 files, M = 3–5 files.++### Phase 0 — Make the failure visible++#### Task 0.1: Temporal-format regression fixtures+**Description.** Pin the observed behaviour of `parse_date_string` / `normalize_date_string` / `resolve_relative_date` against the timestamp formats actually supplied, including the real LoCoMo `session_N_date_time` shape, as failing-then-passing tests. Pure test addition; no `src/` change.+**Acceptance criteria.**+- [ ] A fixture table covers `'1:56 pm on 8 May, 2023'`, ISO-8601 with and without timezone, epoch, and `dd/mm/YYYY`.+- [ ] Relative expressions `yesterday`, `last week`, `last Tuesday`, `next quarter`, `last summer`, `two weeks ago`, `a few months ago` are asserted against a fixed reference date.+- [ ] Currently-broken cases are marked `xfail` with the G-number, so Task 1.1 flips them without touching assertions.++**Verification.** `poetry run pytest tests/test_temporal_normalization.py -q`+**Dependencies.** None.+**Files.** `tests/test_temporal_normalization.py`+**Scope.** S++#### Task 0.2: Superseded-fact leakage metric+**Description.** Add a FAMA-style (F5) metric to the LoCoMo harness measuring how often returned context contains an assertion the graph itself has marked invalid. Attaches to the existing `gold_in_context` hook (`benchmarks/locomo/metrics.py:176`) so it scores *retrieval*, not the answer LLM — the decoupling F3 argues for.+**Acceptance criteria.**+- [ ] `metrics.py` emits `superseded_leakage_rate` overall and per category.+- [ ] The metric counts both channels separately: leakage via graph triples and leakage via text passages (this is what makes G4 visible).+- [ ] `report.py` prints it alongside judge accuracy.++**Verification.** `cd benchmarks && ./locomo.sh evaluate --sample conv-26 --run temporal-baseline --no-resume`; confirm a non-null rate, and that passage-channel leakage is materially higher than triple-channel leakage.+**Dependencies.** None.+**Files.** `benchmarks/locomo/metrics.py`, `benchmarks/locomo/report.py`+**Scope.** S++#### Task 0.3: Conflict-heavy temporal probe set+**Description.** Add a small LTP-style (F3) question set targeting supersession specifically — "what is X's current …", "does X still …", "what did X used to …" — since only ~12 of 1,986 LoCoMo questions probe current truth (G8). Derive from existing conversations so no new ingest is needed.+**Acceptance criteria.**+- [ ] ≥40 questions, each with a gold *current* answer and the superseded value it must not return.+- [ ] Scored separately from LoCoMo categories, reusing the existing judge.+- [ ] Baseline recorded before any `src/` change.++**Verification.** `cd benchmarks && ./locomo.sh evaluate --sample conv-26 --run temporal-probe-baseline --no-resume`+**Dependencies.** 0.2+**Files.** `benchmarks/locomo/` (probe set + wiring)+**Scope.** M++### Checkpoint A — the bug is measurable+- [ ] `superseded_leakage_rate` is non-zero and attributed per channel.+- [ ] Probe-set baseline recorded; expected to be far below the 0.86 cat-2 figure, demonstrating cat 2 was measuring event localization.+- [ ] **Review with maintainer before any `src/` change.** Open questions 1, 2, 5 and 9 should be answered here.++### Phase 1 — Make time comparable++#### Task 1.1: Robust reference-time parsing with explicit indeterminate state+**Description.** Fix G1. Extend `parse_date_string` to accept ISO-8601 (with/without time and timezone), epoch, and the observed `'H:MM am/pm on D Month, YYYY'` shape; return an explicit indeterminate marker rather than passing free text through (F6). Add epoch-int emission alongside the existing `dd/mm/YYYY` string so nothing downstream breaks yet.+**Acceptance criteria.**+- [ ] All Task 0.1 `xfail` cases pass with no assertion edits.+- [ ] A new `to_epoch_seconds` helper returns `int | None`, never a string.+- [ ] Unparseable input yields an explicit indeterminate result that callers can branch on; no bare pass-through.+- [ ] `normalize_date_string`'s existing string output is unchanged for already-working inputs.++**Verification.** `poetry run pytest tests/test_temporal_normalization.py tests/test_ingestion_identity.py -q`+**Dependencies.** Checkpoint A.+**Files.** `src/utils/dates.py`, `tests/test_temporal_normalization.py`+**Scope.** S++#### Task 1.2: Broaden relative-expression coverage+**Description.** Extend `_RELATIVE_PATTERNS` beyond the nine anchored literals: word numerals, quarters, seasons, and non-anchored matching within a longer phrase.+**Acceptance criteria.**+- [ ] `next quarter`, `last summer`, `two weeks ago`, `a few months ago` resolve or return indeterminate — never silent pass-through.+- [ ] Ambiguous expressions (`last summer` without a hemisphere) return indeterminate rather than guessing.+- [ ] No regression in Task 0.1 fixtures.++**Verification.** `poetry run pytest tests/test_temporal_normalization.py -q`+**Dependencies.** 1.1+**Files.** `src/utils/dates.py`, `tests/test_temporal_normalization.py`+**Scope.** S++#### Task 1.3: Fix batch time smearing+**Description.** Fix G7. Resolve `reference_time` per relationship from its own `source_timestamp` instead of hoisting the batch's first value (`src/workers/tasks/ingestion.py:754-766`).+**Acceptance criteria.**+- [ ] A batch spanning two source timestamps produces two distinct `valid_from` values.+- [ ] Relationships with no `source_timestamp` are marked indeterminate, not silently given a sibling's date.++**Verification.** `poetry run pytest tests/test_ingestion_standard.py tests/test_ingestion_orchestration_collection.py -q`+**Dependencies.** 1.1+**Files.** `src/workers/tasks/ingestion.py`, `tests/test_ingestion_standard.py`+**Scope.** S++### Checkpoint B — timestamps are orderable+- [ ] Every new edge carries an epoch-int event time or an explicit indeterminate marker.+- [ ] Ingest a two-session fixture and confirm in-graph timestamps sort correctly.+- [ ] `superseded_leakage_rate` re-measured — expected roughly flat, since nothing reads the new fields yet. A change here means an unintended coupling.++### Phase 2 — Represent validity++#### Task 2.1: Bitemporal quad on edges + indexes+**Description.** Write `t_valid_from`, `t_valid_to`, `t_ingested`, `t_expired` as epoch ints on every relationship, and create the temporal indexes — `src/lib/neo4j/client.py` currently has none.+**Acceptance criteria.**+- [ ] All four fields written on every new edge; existing `valid_at`/`invalid_at` retained unchanged.+- [ ] Index creation is idempotent (`IF NOT EXISTS`) and runs on `ensure_database`.+- [ ] `t_valid_to` is `NULL` on creation, never `0` or `""`.+- [ ] Parity between the Neo4j and NetworkX/Postgres backends.++**Verification.** `poetry run pytest tests/test_graph_upsert_contract.py tests/test_networkx_event_centric.py -q`+**Dependencies.** Checkpoint B.+**Files.** `src/lib/neo4j/client.py`, `src/lib/postgresql/networkx_client.py`, `src/workers/tasks/ingestion.py`, `tests/test_graph_upsert_contract.py`+**Scope.** M++#### Task 2.2: `statement_key` and `cardinality`+**Description.** Derive `statement_key = hash(subject_uuid, claim_type, object_role)` from the event hub's semantic type rather than the generic edge label, and attach a `cardinality` marker per claim type (write-time LLM, cached per type — subject to open question 4).+**Acceptance criteria.**+- [ ] Two assertions about the same claim slot share a `statement_key`; two unrelated events by the same actor do not.+- [ ] `cardinality` is resolved once per claim type and cached; no per-edge LLM call.+- [ ] `single` vs `multi` classification verified on a fixture set covering employer/residence (single) and hobbies/books (multi).++**Verification.** `poetry run pytest tests/test_ingestion_identity.py -q`+**Dependencies.** 2.1+**Files.** `src/core/saving/identity.py`, `src/workers/tasks/ingestion.py`, `tests/test_ingestion_identity.py`+**Scope.** M++### Checkpoint C — representation is sound+- [ ] Ingest a fixture where a single-valued fact changes twice and a multi-valued fact accumulates three times; confirm three distinct `statement_key`s for the multi-valued fact and one shared key for the single-valued one.+- [ ] Confirm the temporal indexes are used (`EXPLAIN` on the extended neighbour query).+- [ ] **Review with maintainer** — open questions 3, 4, 6.++### Phase 3 — Detect supersession soundly++#### Task 3.1: Replace the invalidation heuristic+**Description.** Fix G3. Rewrite `_invalidate_superseded_relationships` to close intervals only when `statement_key` matches, `cardinality = single`, and the existing assertion's `t_valid_from` is strictly earlier. Emit the `SUPERSEDES` edge.+**Acceptance criteria.**+- [ ] Two unrelated events by the same actor sharing predicate `MADE` no longer invalidate each other (the over-firing regression test).+- [ ] Out-of-order ingestion leaves the newer fact live (the recency regression test).+- [ ] A genuine single-valued change closes exactly one interval and appends exactly one `SUPERSEDES` edge.+- [ ] `t_valid_to` is written at most once per edge; a second attempt is a no-op.++**Verification.** `poetry run pytest tests/test_ingestion_standard.py tests/test_graphrag_upgrade_helpers.py -q`+**Dependencies.** Checkpoint C.+**Files.** `src/workers/tasks/ingestion.py`, `src/lib/neo4j/client.py`, `tests/test_ingestion_standard.py`+**Scope.** M++#### Task 3.2: Chunk-level supersession rollup+**Description.** Precompute per text chunk whether it supports any currently-live claim, using the existing edge→chunk link (`src/core/saving/identity.py:64-82`). This is what makes the passage channel fixable within budget.+**Acceptance criteria.**+- [ ] Each chunk carries `supports_live_claims` and `latest_claim_valid_to`.+- [ ] Recomputed when an interval closes, not only at first write.+- [ ] Chunks with no extracted claims default to `supports_live_claims = true` (never penalise unclassified text).++**Verification.** `poetry run pytest tests/test_ingestion_standard.py -q`+**Dependencies.** 3.1+**Files.** `src/workers/tasks/ingestion.py`, `src/lib/postgresql/data.py`, `tests/test_ingestion_standard.py`+**Scope.** M++### Checkpoint D — write side correct+- [ ] Full re-ingest of the LoCoMo brain (per `benchmarks/runs/CHECKPOINT_NOTES.md:13-20`, subject to open question 8).+- [ ] Count closed intervals; manually audit a sample of 20 for correctness in both directions (false closure and missed closure).+- [ ] Re-measure leakage — expected still high, because read paths are unchanged.++### Phase 4 — Read current truth everywhere++#### Task 4.1: Index-backed temporal predicate in the graph query+**Description.** Extend `get_event_centric_neighbors` with the `as_of` predicate from the query pattern above, defaulting to now.+**Acceptance criteria.**+- [ ] Superseded triples are absent for `as_of = now` and present for an earlier `as_of`.+- [ ] Latency on the existing context benchmark does not regress beyond the sub-second budget.+- [ ] `NULL` `t_valid_to` is treated as live; indeterminate `t_valid_from` never silently excludes a fact.++**Verification.** `poetry run pytest tests/test_context_retrieval.py tests/test_postgresql_graph_read_query.py -q`, plus a timing check on `/retrieve/context`.+**Dependencies.** Checkpoint D.+**Files.** `src/lib/neo4j/client.py`, `src/lib/postgresql/networkx_client.py`, `tests/test_context_retrieval.py`+**Scope.** M++#### Task 4.2: `temporal_mode` / `as_of` on the request, validity on the response+**Description.** Fix G5. Add `temporal_mode` (`current` default | `as_of` | `all`) and `as_of`; surface `valid_from` / `valid_to` / `is_current` per triple and a supersession annotation per passage — the labelling F1 and F3 both converge on.+**Acceptance criteria.**+- [ ] `temporal_mode: "all"` reproduces pre-change behaviour exactly.+- [ ] `current` is the default (subject to open question 5).+- [ ] Every returned triple carries validity fields; passages carry the annotation.++**Verification.** `poetry run pytest tests/test_context_retrieval.py -q`+**Dependencies.** 4.1+**Files.** `src/services/api/constants/requests.py`, `src/services/api/controllers/retrieve.py`, `tests/test_context_retrieval.py`+**Scope.** M++#### Task 4.3: Close the seam — passages and the remaining read paths+**Description.** Fix G4 and G6. Apply the supersession annotation in `_retrieve_passages` and `historical_context`; move `_is_currently_valid` out of the controller into `src/core/search/` so `/retrieve/entity/status`, `/retrieve/entity/synergies`, hops and the MCP tools inherit it; repair or explicitly retire the dead recency term at `src/core/search/entity_info.py:225-235`.+**Acceptance criteria.**+- [ ] Passage-channel leakage drops materially in the Task 0.2 metric.+- [ ] The validity check lives in exactly one place and is reachable from every read path.+- [ ] The recency term either uses epoch ints and demonstrably changes ranking, or is removed — no silent constant.++**Verification.** `poetry run pytest tests/test_context_retrieval.py tests/test_mcp_traverse_graph.py -q`; then `cd benchmarks && ./locomo.sh evaluate --sample conv-26 --run temporal-phase4 --no-resume`+**Dependencies.** 4.2+**Files.** `src/core/search/` (validity module), `src/services/api/controllers/retrieve.py`, `src/core/search/entity_info.py`, `tests/test_context_retrieval.py`+**Scope.** M++### Checkpoint E — the reported bug is fixed+- [ ] `superseded_leakage_rate` materially reduced on **both** channels.+- [ ] Probe-set accuracy (0.3) materially improved.+- [ ] LoCoMo cat 1/3/4 **not regressed** — the key guard against over-invalidation (see R1).+- [ ] `/retrieve/context` p95 still sub-second.++### Phase 5 — Deep temporal navigation (optional, after E)++#### Task 5.1: Temporal primitives on the MCP surface+**Description.** Expose point-in-time and change-analysis primitives on the deep tier per F9 — "what changed about X between T1 and T2", "why is this fact no longer current" (answerable by walking `SUPERSEDES`).+**Acceptance criteria.**+- [ ] Primitives are deterministic graph operations; no LLM in the traversal.+- [ ] Supersession chains are walkable end to end, demonstrating provenance is stronger than before.++**Verification.** `poetry run pytest tests/test_mcp_traverse_graph.py -q`+**Dependencies.** Checkpoint E.+**Files.** MCP tool registration + `src/core/search/`+**Scope.** M++---++## Risks++| Risk | Impact | Detection | Mitigation |+| --- | --- | --- | --- |+| **R1 — Over-invalidation destroys true history.** The most likely way this work makes things worse; it is the current failure mode (G3) and 2601.07978 measures Graphiti losing 20+ points to retrieval incompleteness rather than reasoning failure. | **High** | LoCoMo cat 1/3/4 accuracy and `answerable_rate` at every checkpoint; count of closed intervals per ingest. | `cardinality = single` gate; strict recency guard; treat closure as opt-in per claim type; F8's high-precision NLI as an optional second gate. |+| **R2 — Latency regression on the cheap path.** Two extra predicates per edge across a 5-tuple join, on a graph with **zero indexes today**. | High | Time `/retrieve/context` at Checkpoints B, D, E; `EXPLAIN` the extended query. | Ship indexes in the same task as the predicate (2.1); epoch ints not strings; fail the checkpoint on p95 regression. |+| **R3 — Default flip breaks existing callers.** `temporal_mode: current` changes returned facts for everyone. | Medium | Contract tests asserting `all` reproduces pre-change output byte-for-byte. | Answer open question 5 before Task 4.2; ship `all` first, flip the default behind a separate decision. |+| **R4 — Backfill gap.** Existing brains have no `t_valid_from`/`statement_key`; `NULL` handling could hide every legacy fact or expose every superseded one. | Medium | Query legacy brain for edges missing the new fields; assert count before/after. | Treat missing `t_valid_from` as live-and-indeterminate (fail open, preserving recall); require re-ingest for full benefit, as already precedented for provenance. |+| **R5 — Wrong `cardinality` classification.** A claim type misclassified `single` silently deletes history across the whole graph. | Medium | Audit the per-type cache; assert on a fixture of known single/multi types. | Cache is per-type and inspectable; start with a small allowlist of high-confidence single-valued types and expand. |+| **R6 — Metric captures the answer LLM, not retrieval.** `benchmarks/locomo/prompts.py:13` already compensates for pipeline gaps (G8), so improvements could be masked or faked. | Medium | Score leakage on returned context independently of the judge (Task 0.2 does this by construction). | Keep the retrieval-level metric primary, per F3's decoupled-evaluation argument; do not add further dataset-specific prompt rules while this work is in flight. |+| **R7 — Event time vs document time resolved wrongly.** If open question 2 is answered differently than assumed, `t_valid_from` semantics invert and every interval is wrong. | High | Fixture where a 2023 session discusses a 2019 event; assert `t_valid_from` is 2019. | Blocking: get open question 2 answered at Checkpoint A, before Phase 2. |+| **R8 — `t_valid_to` mutation is judged to violate append-only.** Would invalidate the materialisation strategy and force query-time projection, which the latency budget cannot absorb. | High | N/A — a design decision, not a detectable defect. | Open question 6 at Checkpoint C. Fallback: keep `SUPERSEDES` as the sole truth and materialise into a separate index structure rather than an edge property. |diff --git a/docs/research/06-roadmap.md b/docs/research/06-roadmap.mdnew file mode 100644index 0000000..f3ddec0--- /dev/null+++ b/docs/research/06-roadmap.md@@ -0,0 +1,294 @@+# Cross-Workstream Roadmap++Synthesis of `01`-`05`, ranked. Constraints in `00-scope-and-constraints.md` are binding. Every claim below was verified against the code or the run artifacts; where a claim could not be verified it is marked as such.++## Reconciliation with ADR-005++`docs/decisions/005-hybrid-one-shot-context-retrieval.md` (Accepted, 2026-07-28) was written by the maintainer during this research and post-dates the analysts' reading of ADRs 001-004, all four of which are empty stubs. **ADR-005 is the authoritative statement of the current retrieval design and it already settles several points this roadmap's source documents treated as open.** Read it before acting on Phases 1-3. Specifically:++- **The one-shot vs agentic question is decided**, not open: keep the one-shot `GetContext` API and enrich a single call with hybrid retrieval; agentic navigation "stays open for workloads where one sufficiency retry is not enough; it is not the default retrieval model." The retrieval analyst reached the same conclusion independently, which is corroboration rather than new information.+- **The answerer prompt changes were deliberate and documented** (decision item 7), not inadvertent leakage. The maintainer explicitly rejected changing the *judge* prompt on the grounds that it "breaks comparability with v5/v6 and hides real system quality," while treating answerer-side enumeration, temporal phrasing, and hedged open-domain rules as legitimate tuning. That distinction is defensible in principle; the problem is narrower than "leakage" and specific: several of the shipped rules encode conv-26 gold answer strings and one resolves a single question, which fits the test set regardless of intent. Task 0.2 stands, and the maintainer has authorised it.+- **The truncated-logs problem was already known.** Consequence: "Eval must log full `source_passages` / `historical_context` / `retrieved_session_ids`; truncated context logs misdiagnose retrieval." The 20,000-character cap nevertheless still truncates, and no `triples` field is logged at all — so Task 0.1 implements an intent the ADR already recorded rather than introducing a new one.+- **The ~5 s p50 was accepted knowingly** ("Latency and token cost rise with larger passage budgets and optional PPR/sufficiency"). Phase 3 changed that accepted decision: dossier deletion is now live at **~1.4 s p50**; ADR-005 should be superseded to reflect the new latency posture and the remaining ~400 ms gap to sub-second.++**ADR-005 line 29 is now stale.** It states that defaults keep `use_ppr` and `sufficiency_retry` off for API compatibility. Task 2.2 flipped `use_ppr` to `True` (with `sufficiency_retry` left off, because Phase 3 removes it from this path). The ADR needs an amendment or a superseding ADR; leaving it as-is documents a default the code no longer has.++## The finding that reordered everything++**The event graph — the product's entire differentiator — had never been measured** until Phase 0/1.++`benchmarks/locomo/evaluate.py` previously capped logged `text_context` at 20,000 characters, and `retrieve.py` wrote passages into that field before graph facts. Verified against `locomo-conv26-push75-d/answers.jsonl`: all 152 of 152 rows were exactly 20,000 characters, **0 of 152 contained a graph fact line**, and no `triples` field was logged at all.++Both quality metrics — `answerable_rate` and `evidence_session_recall_full` — were derived from that truncated field. They therefore measured the **passage channel only**. Every headline number in `benchmarks/runs/` through push75 was consistent with the event graph contributing nothing whatsoever.++**Phase 0 fixed the harness; Phase 1 answered the product question.** Graph EvR is logged (~53% point estimate on the clean brain) but **not yet a reliable metric** — identical-config runs agree on graph session recall for only ~39–42% of questions (passage recall stays deterministic). See Phase 0 follow-up below. The channel is visible and does **not** move judge accuracy above the resolution floor vs passages-only. Everything after Phase 1 is downstream of that result — see Phase 4 gate.++## What is already fixed++| Defect | Status |+| --- | --- |+| Event-leg supersession deprecating valid actor attributions (`ingestion.py:465-505`) | **Fixed**, 3 tests, 2 fail before / pass after. Existing brains **not** repaired — no backfill authorised. |+| Relative-date resolution inert; recency scorer permanently 1.0 | **Fixed**, 27 tests. LoCoMo timestamps and ISO variants now parse; recency varies instead of being pinned at 1.0. |+| Test suite aborted collection entirely (`retrieve.py:40` imports `kg_agent`, stale stub in `tests/test_architecture_refactors.py`) | **Fixed.** No tests ran at all before this; every earlier "141 passed" baseline was measured with collection interrupted. Suite now runs. |+| Unreachable config defaults + no-op fact filter (Task 2.2) | **Fixed.** `use_ppr` now defaults `True`; all `getattr` fallbacks removed so the schema is the only source of defaults; filter genuinely gated on a caller-supplied adapter. |+| Type-named placeholder super-nodes (Task 2.3) | **Fixed.** Five demonstrated patterns and the Scout instructions to emit Unit entities removed; quantities travel as the relationship `amount` property, which the Janitor already prescribed. Existing graphs still affected — repair pass described, not authorised. |++| Phase 0 harness (all six tasks) | **Done and verified against the live stack.** Graph channel logged, prompt de-fitted, paired McNemar, adversarial scoring, reproducible manifests, judge separable. |+| `benchmarks/locomo/client.py` dropped `use_ppr` when false, so with the new server default an arm configured `use_ppr=false` would silently run **with** PPR | **Fixed.** Both flags now sent unconditionally, so the manifest matches the wire. |+| Dossier channel on `/retrieve/context` (~79% of wall / ~3918 ms p50) | **Fixed and measured live** after sync+restart. Context path no longer imports/calls `EventSynergyRetriever` / `retrieve_matches` and does not emit `[dossier:` lines. Synergies retained on entity routes (`/entity/info`, `/entity/synergies`). Live p50 **~1.4 s** (was ~4.8 s). |++## The honest baseline++`baseline-clean-a` — same config as `push75-c/-d`, conv-26 gold rules removed from `ANSWER_SYSTEM`, measured on the **pre-fix** (`locomoconv26`) brain:++| | Prompt-fitted (`push75-d`) | De-fitted (`baseline-clean-a`) |+| --- | --- | --- |+| Headline | 86.2% | **82.2%** [75.4, 87.5] |+| Multi-hop | 88.6% | 71.9% |+| Temporal | 91.9% | 86.5% |+| Open-domain | — | 46.2% |+| Single-hop | — | 91.4% |+| Evidence recall (passages) | 97.3% | 97.3% |++Paired McNemar: 4 flipped right, 10 flipped wrong, **p = 0.18**. The drop is real as a point estimate and not statistically distinguishable at n=152 — which is the same resolution problem as before, now applied honestly. Evidence recall is identical, exactly as expected when only the prompt changed. Open-domain lost 3 and gained 0, consistent with removing the hedge template.++**Use 82.2% as the reference point** for the old brain. A `prompt-audit` command now fails if any 3-gram of `ANSWER_SYSTEM` appears in a gold answer, so the fitting cannot silently recur.++## Phase 1 — done (live, after sync+restart)++Clean brain `locomoconv26clean`: **19/19** conv-26 sessions ingested (not 35 — that was a miscount of `date_time` keys). Extraction fixes confirmed in-graph vs the old brain:++| Signal | Clean | Old |+| --- | --- | --- |+| Deprecated EVENT-touching edges | **0** | 219 |+| Type-named placeholders | **0** | 3 |+| Nodes / rels / events | **537 / 752 / 410** | 411 / 584 / 281 |++Eval arms (de-fitted prompt, same config as `baseline-clean-a`):++| Arm | Acc | Multi-hop | Graph EvR | Passage EvR | p50 ms |+| --- | --- | --- | --- | --- | --- |+| `baseline-clean-a` (old brain) | 82.2% | 71.9% | 0%\* | 97.3% | 4776 |+| `clean-brain-eval-a` | 83.6% | 78.1% | 54.0% | 96.7% | 1372 |+| `clean-brain-eval-b` | 81.6% | 71.9% | 52.0% | 96.7% | 1423 |+| `ablate-passages-a` (`max_facts=0`) | 82.2% | 75.0% | 0% | 96.7% | 1337 |+| `ablate-passages-b` | 81.6% | 81.2% | 0% | 97.3% | 1306 |++\*Old baseline graph EvR was unmeasurable due to logging, not necessarily empty.++McNemar all ns: full vs baseline p≈0.77–1.0; ablate vs full p≈0.77–1.0.++### Verdicts++1. **Extraction fixes earn their keep on graph hygiene**, not on detectable judge accuracy at n=152.+2. **The graph channel does not move judge accuracy above the resolution floor** vs passages-only. Graph evidence recall is logged (~53% point estimate) — the harness can see the channel; it is not yet converting into answers. Treat that EvR figure as provisional until fact selection is order-stable (below).+3. **Dossier deletion earns its keep on latency**: ~4.8 s → ~1.4 s p50, no accuracy loss. Sub-second is not yet met; residual is likely embedding fanout (Phase 3.2). Gap to budget ≈ **400 ms**.++### Follow-up findings (second clean-brain measurement agent)++1. **Graph EvR is non-deterministic across identical-config runs** — open defect / Phase 0 follow-up. Even with `graph_session_ids` emitted, identical-config runs on the same brain agree on graph session recall for only ~39–42% of questions. Passage recall stays deterministic. Until fact selection is made order-stable, **do not treat graph EvR as a reliable metric** — it is measurable but not yet a measurement.+2. **The clean graph is larger and more fragmented, which hurts under a fixed `max_facts` budget.** Clean: 537 nodes / 410 events vs old 411 / 281. Same `max_facts=50` therefore covers fewer distinct sessions. Extraction correctly stopped collapsing history; **fact selection / budget is now the binding constraint** if the graph channel is to pay off — a better next lever than more extraction fidelity *if* graph work continues.+3. **Process:** two agents ingested the same brain id / run directory concurrently; `ingest.jsonl` was interleaved and needed dedup. Detail: `benchmarks/runs/clean-brain-ingest/NOTES.md`. Recommend: one owner per brain id; the manifest should record the running server's code identity, not only the harness git SHA (a SHA is useless when the TUI install is a different checkout).+4. **Provenance correction:** both old and clean brains already have `source_chunk_ids` on 100% of nodes and relationships — re-ingest is not required for provenance stamping. See `benchmarks/runs/CHECKPOINT_NOTES.md`.++## Latency — measured, not inferred (2026-07-28)++### Pre-deletion attribution (old path)++Live measurement against `locomoconv26` at the push75-d config, n=20 questions, in-process, matching the harness p50 within noise (4813 ms vs 4835 ms). Instrumentation: `profile_stages=true` on `/retrieve/context` → `stage_timings`, or `TRACE_STAGE_PROFILER_ENABLED=true` for a `LATENCY` trace event. Off by default; measured overhead below noise.++| Stage | Claimed | Measured p50 | Share | Verdict |+| --- | --- | --- | --- | --- |+| dossiers | hundreds of per-edge fetches | **3918 ms** | **79%** | right, and understated |+| passages | 12–24 serial | 829 ms (682 embed) | 17% | cost is embeddings, not search |+| facts (concurrent) | depth 4 | 481 ms union / 2622 sum | 10% | correct |+| historical | concurrent | 355 ms, fully overlapped | 0% of wall | free; delete for cost only |+| PPR | loop-bound | 35.9 ms | 0.7% | irrelevant |+| spaCy | 2 passes on loop | 13.1 ms | 0.3% | irrelevant |+| sufficiency retry | depth 24 | **fired 0/20** | 0% | unpaid on this brain |+| fact filter | 0 | 0 | 0% | correct |++### Post-deletion (live, after sync+restart)++Dossier deletion is **deployed and measured** on the clean-brain eval arms: harness p50 **~1372–1423 ms** (full) / **~1306–1337 ms** (passages-only), versus **4776 ms** on the old-brain baseline that still paid for dossiers. That is a ~3.4 s win with no detectable accuracy loss. Remaining gap to the 1000 ms budget is **~300–400 ms**, consistent with the predicted embedding-fanout residual.++### Consequences for Phase 3++1. **Dossier channel deleted from `/retrieve/context` — done and live.** Predicted remaining ~895 ms; measured remaining ~1.4 s. The component's own docstring called its output "raw, unprecise," and `_MAX_DOSSIER_ENTITIES = 5` was unconditional. Dominant cost inside it was **10 serial embedding HTTP calls**, not the per-edge vector fetches.+2. **Next: batch embeddings across the fanout (Phase 3.2).** Passages alone make three serial ~200 ms embed calls. One shared embedding is the leading candidate for closing the remaining ~400 ms gap.+3. **Do not spend effort on spaCy dedup, PPR off-loop moves, or the sufficiency retry** for latency — combined they are under 50 ms, and the retry never fired in 20/20 questions at both configs, including the arm whose manifest claims `sufficiency_retry=true`. That means `push75-d` did not measure the knob its manifest says it varies.+4. **Event-loop blocking is ~50 ms p50 (~1%).** Nearly everything is I/O wait or off-loop Python. One exception worth fixing when touching the file: `_build_adjacency_from_seeds` runs a synchronous graph query on the loop (28.5 ms) — small alone, a queuing multiplier under concurrency.+5. **Historical context recovers cost, not latency** — 91.5% duplicate of source passages and fully overlapped.++Sub-second now depends on embedding batching, not further channel deletion. This brain is tiny (19 sessions); vector/Cypher times will grow with corpus size while embedding latency will not.++## Phase 1 blockers — resolved and live++1. **`max_facts=0` honouring.** `retrieve.py` uses `max(0, request.max_facts)`; schema `Field(40, ge=0)`. Verified by `tests/test_context_max_facts_and_provenance.py` (6/6) and by the live ablate arms (graph EvR = 0%).+2. **Graph session provenance.** Response exposes per-triple `source_chunk_ids` / `source_session_ids` and top-level `graph_session_ids`. Fact lines also get `(session_N)` annotations so the existing harness scraper can score graph recall. Live graph EvR ~52–54% on clean-brain full arms confirms the channel is logged and scored — but **cross-run agreement is only ~39–42%**, so the number is not yet trustworthy (Phase 0 follow-up: order-stable fact selection).+3. **Sync + restart.** Workspace → `~/.brainapi/source` sync and TUI/API restart completed before the clean-brain eval and ablation runs above.++## Dataset subtlety worth recording++LoCoMo's `adversarial_answer` is **not** a refusal marker. There are 439 distinct values across 446 questions, all substantive: it is the **trap** answer produced by accepting the question's false premise, usually attributing an event to the wrong speaker. So F1/BLEU overlap against it is an *inverted* signal — high overlap means the system fell for the trap. The harness now returns `None` for lexical metrics on adversarial rows and instructs the judge that correct means declining or correcting the premise. Any future work on abstention must not treat this field as a gold refusal string.++Both were found during research, not by the benchmark. That is itself the argument for Phase 0.++---++## Phase 0 — Make the benchmark able to detect a real change++Nothing after this phase is verifiable without it. Every task here is small and touches only `benchmarks/`.++### Task 0.1 — Log the graph channel++**Why:** without it the graph's contribution is unmeasurable, and the passages-only ablation is meaningless.+**Do:** log `triples` as its own field; either raise `_TEXT_CONTEXT_CAP` or write graph facts before passages; recompute `retrieved_session_ids` from the untruncated response.+**Acceptance:** a graph fact line appears in the logged artifact for >0 questions; `evidence_session_recall` is computable separately for the graph and passage channels.+**Verification:** re-run `evaluate` on the existing brain; assert both channels non-empty in `answers.jsonl`.+**Size:** S. **Depends on:** none.++### Task 0.2 — Delete the conv-26-fitted rules from `ANSWER_SYSTEM` and re-baseline++**Why:** the uncommitted `benchmarks/locomo/prompts.py` diff contains gold answer strings verbatim — `Keep speaker names explicit (e.g. "Melanie's slipper", not "my slipper")`, the phrasings `"the week of <date>"` and `"since <year>"`, the open-domain template `"Likely no; ..."`, and `For "last weekend" / charity race timing relative to a Thursday/Friday session, prefer the preceding Sunday over Saturday`. That last rule resolves one specific question. `README.md` forbids ingesting QA annotations because it leaks the test set; routing the same information through the answer prompt defeats that rule while observing its letter.++`push75-c` → `push75-d` is the proof: byte-identical manifests, same brain, same flags, same models, **only** the answer prompt differs, and accuracy moved 82.9% → 86.2%. The gap between those two runs is prompt fitting, not memory quality.+**Do:** remove question-specific rules; keep only format guidance that is defensible for an arbitrary conversation. Re-baseline and accept the drop.+**Acceptance:** no gold-answer string or single-question rule remains in the prompt; a new baseline is recorded.+**Verification:** diff review plus a fresh run recorded as the reference point.+**Size:** S. **Depends on:** none. **Expect the headline to fall.**++### Task 0.3 — Paired significance testing++**Why:** the harness compares independent Wilson intervals, discarding the pairing on identical question sets and most of the statistical power. **Passage** retrieval is deterministic — passage `evidence_session_recall` is bit-identical across independent runs within a config arm (`0.826667` in both a and b; `0.973333` in both c and d) — while 11-12% of questions flip correctness between identical-config runs at `temperature=0`. **Graph** session recall is *not* deterministic yet (~39–42% cross-run agreement on identical-config clean-brain arms); do not treat graph EvR deltas as signal until fact selection is order-stable. Judge deltas need a paired test.+**Do:** exact McNemar between runs; report flipped-right and flipped-wrong counts, not just the delta.+**Acceptance:** every run comparison emits a p-value and a flip table.+**Verification:** reproduce the known results — `push75-b`→`push75-c` p≈0.06 single-run; pooled two-run arms p≈9×10⁻⁴.+**Size:** S. **Depends on:** none.++### Task 0.4 — Fix adversarial scoring, then decide whether to enable it++**Why:** 444 of 446 adversarial questions carry gold under `adversarial_answer`, but `evaluate.py:177` reads only `answer`; `metrics.py:29-30` returns F1 = 1.0 for empty prediction against empty gold. Abstention — the only defence against confident wrong answers, and the thing a memory product is judged on in production — is both unscorable and skipped by default.+**Acceptance:** adversarial questions score correctly against `adversarial_answer`; empty-vs-empty no longer scores 1.0.+**Size:** S. **Depends on:** none.++### Task 0.5 — Make runs reproducible++**Why:** no manifest records a git SHA, prompt hash, dataset digest, or pinned model, and `push75-a/b/c/d` have no `ingest.jsonl`, so the graph under test is recorded nowhere. Separately `report.py:34` does not de-duplicate, so `checkpoint-a` scored 156 rows from 152 questions — and that run, 0.0% across every category with every answer failing, rendered as a clean plausible report. Concurrent agents sharing a brain id / run directory also interleaved `ingest.jsonl` (see `benchmarks/runs/clean-brain-ingest/NOTES.md`).+**Acceptance:** manifest captures harness SHA **and the running server's code identity** (TUI install may be a different checkout — harness SHA alone is useless), prompt hash, dataset digest, model IDs; one owner per brain id; a run that fails wholesale is visibly distinguishable from a run that scored badly.+**Size:** S.++### Checkpoint 0+- Graph and passage channels measured separately+- Prompt no longer fitted to conv-26; honest baseline recorded+- Paired tests in place; flip counts reported+- **Do not proceed until the honest baseline exists.** Everything after this is judged against it.++---++## Phase 1 — Answer the product question — **DONE**++### Task 1.1 — Passages-only ablation on a re-ingested clean brain — **DONE**++**Why:** this decides whether the graph earns its cost. It had to run on a **freshly ingested** brain: all four `push75` runs showed `ingest.n = 0`, so every one reused a pre-fix brain carrying wrongly-deprecated actor edges.+**Done:** re-ingested `locomoconv26clean` (19/19 sessions), then compared `max_facts=0` against the full config with everything else fixed (de-fitted prompt, same flags as `baseline-clean-a`). Two runs per accuracy arm.+**Result:** no contribution detectable above the resolution floor — McNemar ablate vs full p≈0.77–1.0; judge accuracy within noise of the old-brain baseline. Graph EvR logged at ~53% but cross-run agreement only ~39–42% — not yet a reliable metric. Full table and verdicts in the Phase 1 results section above.++**This result hardens the Phase 4 gate** (see below): graph design investment is no longer justified by LoCoMo judge accuracy alone. If graph work continues, prefer fact-selection / `max_facts` budget over further extraction fidelity.++---++## Phase 2 — Confirmed defects with cheap fixes, independent of measurement++Ranked by gain per unit of cost. None require the benchmark to be trustworthy, because each is a provable defect rather than a quality hypothesis.++### Task 2.1 — Stop billing passages three times+The same passage text reaches the answerer three times: inlined into `text_context` (`retrieve.py:1057`), again as `## Source passages` (`prompts.py:56-59`), and again as `## Historical context` (`prompts.py:64-67`), where 91.5% of entries are byte-identical to a source passage. 183,978 characters per question where 61,955 suffice — a **66.3% cut with no information loss**, accounting for essentially the entire 51.4k-token prompt. Improves cost, latency, and the budget simultaneously. **Size:** S. **Highest gain per hour in this document.**++### Task 2.2 — Make the intended config reachable, and fix the no-op filter+`use_ppr` and `sufficiency_retry` are declared `False` in `GetContextRequestBody` while `retrieve.py:898` and `:967` read them as `getattr(request, ..., True)` — Pydantic always materialises the field, so the `True` fallback is dead and the better config is off for every HTTP caller. Separately `filter_relevant_facts` is called at `retrieve.py:917-921` with no `llm_adapter`, which under the current diff returns every index, making `apply_fact_filter=True` provably identical to the `else` branch it guards; the LLM prompt inside it never executes. Decide intent, then make the code say it. **Size:** S.++### Task 2.3 — Fix the type-name super-node collapse+`src/constants/prompts/architect_agent.py:111` instructs the model to emit `{"type": "MONEY", "name": "Money"}`. Node identity is `sha256(lower(name)|lower(type))` (`src/core/saving/identity.py:13-25`), so **every monetary amount ever ingested collapses onto one shared node**, manufacturing false multi-hop paths between unrelated facts. Verified: `stable_node_id('Money','MONEY')` is constant. This actively fabricates connections, which is worse than missing them for a product whose promise is a traceable path. **Size:** S-M.++### Task 2.4 — Index the graph, and stop silently skipping the vector index+Nothing in the repository creates a single Neo4j index or constraint; every uuid lookup is a full scan, twice per relationship write, using bracket syntax that would defeat a property index if one existed. And `_vector_index_ddl` returns no DDL above 2000 dimensions (`src/lib/postgresql/vectors.py:62-63`) while `.env.example:43-47` ships 3072, so no HNSW index is ever created on a default install. Decide the embedding question by recall-vs-latency measurement per `00`; `halfvec` indexes up to 4000 dims and is the option that keeps 3072. **Size:** S each.++### Task 2.5 — Give the Janitor a veto+Relationships are queued for persistence at `ArchitectAgentCreateRelationshipTool.py:550-551` and only *then* is `wrong_relationships` checked, so a rejected edge **and** its replacement are both written. And `run_atomic_janitor` returns `"OK"` on success and `None` on parse failure while callers use `getattr`, making a crashed Janitor indistinguishable from an approving one. The validation loop the README advertises does not currently gate anything. **Size:** M.++### Checkpoint 2+- Token cost per question down ~66% with accuracy unchanged within the resolution floor+- No shared type-level super-nodes; false path count measurable+- A failed Janitor is distinguishable from an approving one++---++## Phase 3 — Meet the latency budget (context path)++`/retrieve/context` must be sub-second. **Pre-deletion** measured p50 was 4053–4793 ms; **post-deletion live** p50 is **~1.4 s** on clean-brain full arms (~1.3 s passages-only). Gap to budget ≈ **400 ms**. Latency does not scale with query length (r = 0.089), so bounding fanout is a cost win, not a latency win.++Dossier deletion (the predicted dominant win) is **done and measured**. The remaining path is embedding fanout plus a handful of serial I/O operations.++- **3.1** Instrument per-stage timing — **done**. Live attribution confirmed dossiers at 79% of wall; post-deletion residual matches the embedding-fanout prediction.+- **3.2** Batch embeddings across the fanout (leading residual); optionally move to write time: PPR adjacency table, per-entity gists, passage compression. **This is the remaining work to close ~400 ms.**+- **3.3** Move to the deep/MCP tier: the sufficiency **retry** (keep the cheap lexical check on the context path, but only to return an explicit insufficiency signal to the caller — which also closes the "silently returns weak context" gap), listwise LLM and cross-encoder reranking, beam search, and all iterative retrieve-then-reason.+- **3.4** Context-tier ranker: a linear or small-GBDT model over features already computed — vector distance, PPR score, recency, provenance overlap, variant strength — trained on labels in `answers.jsonl`. Microseconds, no model server. A cross-encoder costs 50-200 ms on CPU for a component that is not the bottleneck; it is the deep-tier escalation path, not the context-path answer.++**Checkpoint 3:** p50 under 1000 ms with per-category evidence recall held. Dossier deletion got from ~4.8 s to ~1.4 s; embedding batching is the next gate. If the budget cannot be met, escalate with numbers rather than quietly missing it.++---++## Phase 4 — Design work, gated on Phase 1 (tension now explicit)++Phase 1 reported: **the graph channel does not move LoCoMo judge accuracy above the resolution floor** vs passages-only (McNemar ns). Graph design investment is therefore **harder to justify on LoCoMo judge accuracy alone.**++The remaining case for Phase 4 work is not "the ablation proved the graph wins on LoCoMo," but:++1. **Stabilize then raise graph EvR** — first make fact selection order-stable (Phase 0 follow-up; current ~39–42% cross-run agreement makes the ~53% figure provisional), then lift coverage under the fixed `max_facts` budget (clean graph is larger/more fragmented, so 50 facts cover fewer sessions). Converting retrieval into answers remains open.+2. **Multi-hop / composition where passages fail** — product priority #1; LoCoMo multi-hop moved within noise, not past the floor.+3. **Non-LoCoMo product surfaces** — synergies, entity routes, agent substrates — where the graph is the product, not an optional evidence channel.++**Active GraphRAG track:** `07-event-graphrag-improvement-plan.md` — event-hub-first plan (determinism → diversified `max_facts` / typed PPR → tiered cross-event composition per ADR-006). Prefer that doc for retrieval GraphRAG task breakdown; do not start 4.x as a LoCoMo accuracy programme. Start only when one of (a)–(c) is the explicit goal, and measure that goal directly.++**SOTA LoCoMo track:** `08-sota-locomo-protocol.md` — dual product/SOTA profiles to compete with HyperMem/Mem0/APEX-MEM (≥93% LLM-as-judge on full LoCoMo10). Generation + hierarchical topic retrieval; not more GraphRAG EvR alone.++**Phase A (2026-07-29):** order-stable fact selection + harness agreement. Live: graph-session stability **99.3%** on `graph-stable-a/b`; graph EvR measurable at **47.3%** (`graph-stable-a` p50 ~1351 ms).++**Phase B (2026-07-29):** hub/session MMR, intra-hub completeness, typed PPR, path-shaped single-hub format, optional `temporal_conflicts`. Live: stability held; graph EvR **flat 47.3%** vs graph-stable-a — **Checkpoint B EvR lift NOT met**. Sequential p50 ~948 ms. Conclusion recorded in `07`: re-ranking the same intra-hub pool cannot raise session coverage; Phase C cross-event hub-bridge expansion is required.++**Phase C (2026-07-29):** write-time hub-bridge index + context ≤1-bridge expansion (backfill 13 660 bridges on `locomoconv26clean`). Live: stability **98.7%**; graph EvR **47.3–48.0%** (noise vs 47.3%; +1 question on arm a); cat1 still 25%. Sequential p50 ~1340–1391 ms. **Checkpoint C EvR lift NOT met.** Novel-session bridge ranking (C2) also null on EvR (+~500 ms). Diagnostic: 91% of incomplete QAs are 1-bridge-reachable; curated set never kept the missing session → retention, not substrate reachability.++**Phase C3 + ANN (2026-07-29):** reserved curated slots for novel-session bridge facts (`reserve_hub_cap=12`) **plus** ANN seed stability (over-fetch+uuid order, seed stabilize, 3dp distance quantization). **Dual gate PASS:** agreement **99.3%**, graph EvR **56.0%** (was 47.3%), cat1 **40.6%** (was 25%), p50 ~1320 ms. Detail and ship setting in `07` Checkpoint C3.++**Judge secondary (same day):** full evaluate `phase-c3-judge-a/b` → **77.6% / 78.3%** (multi-hop **65.6%** both). Graph EvR **56.0%** and agreement **99.3%** held on full evaluate. McNemar pooled vs `baseline-clean-a`: 9↑/15↓ p=0.31; vs `graph-stable-a`: 10↑/16↓ p=0.33 — **ns**. EvR lift did **not** convert to answers (G4); answerer gap ~18–19%.++**Checkpoint D.1 (2026-07-29):** paths into answerer on `locomoconv26clean` (mf=50, `use_ppr`) → **80.9% / 80.3%** acc (multi-hop 71.9% / 75.0%); Graph EvR **56.0%** held; A↔B **100%**; `prompt-audit` passed. Pooled **80.6%** vs C3 **78.0%**: McNemar 17↑/9↓ p=0.17 ns; vs `baseline-clean-a` 82.2%: 10↑/16↓ p=0.33 ns. **Soft convert** (directionally up, not significant); answerer gap ~16%. Measured with harness `enrich_paths_from_triples` after TUI restart was initially blocked; later sync+restart confirmed native `_paths_for_curated` legs. Detail in `07`. **Next (user fork — do not auto-start D.2):** D.2 feature reranker **vs** Checkpoint D product-surface write-up (graph value = EvR/provenance, not LoCoMo judge).+- **4.1 Bitemporal representation.** Store validity as typed, range-queryable properties on the event hub plus an appended `SUPERSEDES` edge, with a `statement_key` and a `cardinality` marker so an interval closes only when the claim is genuinely single-valued and the existing assertion is strictly older. This fixes the remaining over-firing the hotfix did not cover — there is still no cardinality model, so a legitimately one-to-many predicate between two non-event nodes is still blindly deprecated — and the out-of-order case, where ingesting older data invalidates the *newer* fact. Adopt with `2501.13956`; cite `2606.01435` for the latency argument only, per the caveat in `00`.+- **4.2 Apply the current-truth filter to every channel.** It is called in exactly one place (`retrieve.py:787`) and guards only the triple channel; the passage channel and `historical_context` have no temporal predicate, so an invalidated edge and the sentence asserting it ship in the same payload. Per the maintainer's decision, current-truth becomes the default with opt-in history. Passage annotation must come from a write-time rollup — no LLM call on the query path.+- **4.3 Canonicalize types and predicates *after* open extraction.** `2402.10744` found that constraining an LLM to a fixed relation set *causes* hallucination, so post-hoc canonicalization is the right architecture, not schema-constrained extraction. Backing: `2605.29168`, `2505.23628`.+- **4.4 Ground each fact in a verified source span.** The Janitor's revision protocol is entirely structural; nothing asks whether a relationship is supported by the source text, so a fluent hallucination passes. `2510.00276`.+- **4.5 Contradiction-aware dedup.** `2606.26511` measured cosine similarity separating a contradicted fact from a duplicate at AUROC 0.59 — near chance — so the current similarity-based dedup preferentially swallows the very updates it should act on.+- **4.6** Reuse `KGAgent.verify_entity_existence` (`kg_agent.py:219`), which already implements candidate-generation-plus-LLM-adjudication, on the free-text path's abstain-on-ambiguity branches (`ingestion.py:290-291`), which currently create a duplicate instead of asking it.++---++## Deliberately deferred++- **Judge validation against human labels.** No human labels exist and no agreement statistic is computed, so judge accuracy *is* the metric while nothing measures the judge. `config.py:17-18` uses the same model as answerer and judge, and the judge never sees the retrieved context (`prompts.py:86-90`), so it cannot assess groundedness — the one property that matters for a memory product. A 200-300 item labelling pass would calibrate it. **Recommended; awaiting the maintainer's decision.**+- **Supersession sub-benchmark.** Explicitly deferred by the maintainer. Consequence: temporal fixes are unit-testable but their end-to-end effect is unmeasurable. LoCoMo category 2 scores 0.86 while measuring event *localization*; roughly 12 of 1,986 questions probe current truth.+- **Scaling to all ten conversations.** Currently 152 of 1,540 non-adversarial questions (9.9%), one conversation. The two priority categories are measured at n=32 (multi-hop, CI width 27.7pp) and n=37 (temporal, 22.1pp); open-domain is n=13 (47.7pp). Power analysis: a *concentrated* effect that repairs a broken question class is detectable at 1-2 runs per arm, but a **diffuse uniform +5-point improvement reaches only 0.14 power at two runs and 0.52 at six**. This benchmark can validate targeted fixes and cannot validate broad polish at any affordable run count.+- **Backfill for wrongly-deprecated edges.** Not authorised. Mechanical when wanted: clear `deprecated`/`invalid_at` where either endpoint carries the `EVENT` label.++## Risks++| Risk | Impact | Detection / mitigation |+| --- | --- | --- |+| The ablation shows the graph adds little | High — **realized:** no LoCoMo judge gain above floor; graph EvR ~53% provisional and not converting | Do not justify Phase 4 on LoCoMo accuracy alone; stabilize EvR first, then fact-budget / selection, multi-hop composition, or non-LoCoMo surfaces |+| Graph EvR treated as a measurement before order-stability | High — identical-config runs agree on only ~39–42% of questions | Phase 0 follow-up: order-stable fact selection; do not A/B on graph EvR until then |+| Concurrent agents share a brain id / run dir | Medium — interleaved `ingest.jsonl`, duplicate extractions | One owner per brain id; manifest records server code identity (see clean-brain-ingest/NOTES.md) |+| Removing fitted prompt rules drops the headline and looks like regression | Medium | Re-baseline explicitly and label the old numbers as fitted |+| Sub-second and accuracy are irreconcilable on one path | Medium | Already mitigated by the two-tier split; escalate with numbers |+| Latency attribution is wrong because it was inferred, not measured | Medium | **Mitigated:** 3.1 measured; dossier deletion confirmed live (~4.8 s → ~1.4 s) |+| Cardinality still missing after the hotfix | Medium | One-to-many non-event predicates still wrongly deprecated; Task 4.1 |+| Unrepaired brains silently confound every comparison | High | Record `ingest.n` per run; require a fresh ingest for any graph-side claim |diff --git a/docs/research/07-event-graphrag-improvement-plan.md b/docs/research/07-event-graphrag-improvement-plan.mdnew file mode 100644index 0000000..0375334--- /dev/null+++ b/docs/research/07-event-graphrag-improvement-plan.md@@ -0,0 +1,739 @@+# 07 — Event-centric GraphRAG improvement plan++Workstream: retrieval quality via GraphRAG techniques that respect the **5-node event path**. Constraints in `00-scope-and-constraints.md` and ADR-006 are binding. Phase 1 results in `06-roadmap.md` are not re-litigated.++**Plan mode only** — no `src/` or `benchmarks/` changes in this document's delivery. Tasks below are ordered for a future implementer.++Skill: planning-and-task-breakdown (vertical slices, S/M, acceptance criteria, verification, checkpoints).++---++## What this workstream does++### Product intent++Return, under the two-tier budget, evidence that preserves the Triangle of Attribution so a consumer can answer single-hop questions from one event hub and compose multi-hop answers across hubs — without flattening events into binary triples.++### The 5-node fact (non-negotiable)++README models every action as an **Event Hub**:++```+(subject)-[:MADE-family]->(Event)-[:TARGETED-family]->(object)+ \-[:OCCURRED_WITHIN]->(context)+```++In code this is the event-centric 5-tuple `(n, r, m, r2, b)` tied by `flow_key` on both legs.++### Traced one-shot path (`/retrieve/context`)++| Stage | What happens | Anchors |+| --- | --- | --- |+| Entry | `get_context` → `_build_context`; spaCy elements → query variants | `retrieve.py:789-822`, `:427-458` |+| Seeds | Per variant: embed → ANN nodes (`k=_SEED_K`) + ANN relationships → seed UUIDs | `retrieve.py:606-646`, `:831-845` |+| Event expand | `get_event_centric_neighbors(seed_uuids)`: Cypher `MATCH (n)-[r]-(m)-[r2]-(b) WHERE … AND r2.flow_key = r.flow_key` | `retrieve.py:849-851`; `neo4j/client.py:1654-1681` |+| Validity | Drop candidate if either leg fails `_is_currently_valid` | `retrieve.py:855-857`, `:418-424` |+| Score | Min vector distance over `{n,m,b}` among seeds; dedupe by `(r.uuid, r2.uuid)` | `retrieve.py:858-875`, `:414-415`, `:969-977` |+| Optional PPR | Adjacency from another `get_event_centric_neighbors` over seeds; clique edges among `{n,m,b}`; reorder by `-max(PPR)` | `retrieve.py:526-549`, `:981-1002`; `fact_filter.py:81-115` |+| Cap | `curated = ranked[:max_facts]` (LLM filter only if deep tier supplies adapter) | `retrieve.py:1004-1021`; docstring `:797-799` |+| Render | Path-shaped string via `" \| ".join(...)`; session tags from provenance chunks; passages then facts in `text_context` | `retrieve.py:410-411`, `:692-720`, `:1137-1139` |+| Passages | Separate hybrid RRF channel; no graph edges | `retrieve.py:723-786`, `:882-922` |++NetworkX backend mirrors the same flow-key path (`networkx_client.py:1069-1090`).++### What is configurable vs hardcoded++| Knob | Where | Notes |+| --- | --- | --- |+| `max_facts` | `requests.py:396` default **40**; harness often 50 | Binding budget on a fragmented clean graph |+| `use_ppr` | `requests.py:399` default **True** | Replaces distance order; no blend |+| `apply_fact_filter` | schema True; no-op without adapter | Context path intentionally empty |+| `_PPR_DAMPING` / `_PPR_ITERS` | `retrieve.py:51-52` | Hardcoded 0.85 / 20 |+| Flow-key equality | Cypher only | Hardcoded; no cross-event hop |++### Silently dropped / structurally impossible today++- **Cross-event composition on the one-shot path.** `flow_key` equality keeps both hops inside one hub (`neo4j/client.py:1670-1672`). Multi-hop across events is bag-of-facts composition by the answerer, not path retrieval.+- **PPR cannot leave the seed neighbourhood.** Adjacency is rebuilt from the same event-centric call over seeds only (`retrieve.py:526-549`).+- **PPR ignores edge types.** Transitions are uniform over the three event nodes; MADE vs TARGETED vs OCCURRED_WITHIN are not distinguished.+- **Validity filter skipped when building PPR adjacency** (`retrieve.py:538-549` vs filter at `:855-857`) — structural signal and returned set can disagree.+- **No session / hub diversification under `max_facts`.** Truncation is pure score order (`:1021`).+- **Sort ties have no uuid tiebreak** (`:977`, `:1000-1002`) while candidates arrive from concurrent threads (`:961-967`) → **order-unstable fact sets** and non-deterministic graph EvR (~39–42% cross-run agreement per `06`).++Deep / MCP tier: `traverse_graph` (depth ≤5) and agentic tools exist separately; they do not share ranking with `get_context` (see `02-retrieval-multihop.md`).++---++## Guarantees and where they break++**Intended guarantee:** given a question, return a compact, **reproducible** set of 5-node event facts (plus passages) that covers the sessions needed to answer — including single-hub lookups and cross-hub composition — within ADR-006 budgets.++Ranked by impact on accuracy / multi-hop recall:++| # | Break | Kind | Why it hurts |+| --- | --- | --- | --- |+| **G1** | Graph EvR is not a measurement | Bug | Concurrent candidate merge + score-only sorts → ~40% cross-run agreement. Cannot A/B graph work until fixed. |+| **G2** | Multi-hop across events impossible on context path | Deliberate trade-off (now the accuracy ceiling) | Flow-key 2-hop is intra-hub only. Product priority #1 in `00`. |+| **G3** | Fixed `max_facts` under-covers fragmented clean graph | Gap | Clean brain denser (410 events vs 281); score truncation collapses to few sessions → ~53% provisional graph EvR. |+| **G4** | Graph facts do not move LoCoMo judge accuracy above floor | Measured trade-off | Phase 1: passages-only ≈ full graph (McNemar ns). Risk: more graph facts that still do not change answers. |+| **G5** | Single-hop via hubs is incomplete under budget | Gap | Seeds may hit object/context without ranking the full triangle highly; no hub-level scoring or path completeness check. |+| **G6** | PPR adjacency is seed-local, type-blind, validity-inconsistent | Gap | Weak multi-hop signal; CatRAG-style hub drift risk on EVENT / OCCURRED_WITHIN nodes. |+| **G7** | Context p50 ~1.4 s vs <1 s budget | Gap | Remaining ~400 ms likely embedding fanout (Phase 3.2). Graph work must not blow the budget; expensive composition → deep tier or write-time. |+| **G8** | Flat prompt of facts, no cross-event path objects | Gap | PathRAG diagnosis: redundancy + flat organisation; BrainAPI already linearises one hub but not chains of hubs. |++---++## Maintainer decisions (2026-07-29)++Answers to the open questions below. Binding for Phases A–D.++1. **Cross-event on context path** = write-time entity-mediated hub adjacency + cheap PPR over hubs; full iterative composition = deep/MCP only (ADR-006).+2. **Primary metric** = **stable graph EvR** (esp. multi-hop / single-hop); judge accuracy is a secondary paired check and **may stay flat**.+3. Graph EvR is a measurement only when **≥95%** of questions have identical graph-session sets across two identical-config runs.+4. Diversification under `max_facts`: implement a **blend of session + event-hub coverage**. Maintainer max latency tax on the context path: **2000 ms**. **Tension with ADR-006:** the accepted context budget remains **p50 < 1000 ms**. Do not silently ignore either number — prefer meeting both if possible; if the diversification tax would exceed remaining headroom to 1 s, escalate with measured numbers rather than quietly spending the 2000 ms allowance.+5. Path-shaped **cross-hub** strings: **not** in `text_context` by default; structured `paths` field for deep/MCP only.+6. Temporal conflict: **prefer recency** in ranking **and return both** as meta information.+7. **Leiden out** for now; **k-core / deterministic hierarchy still interesting later** for open-domain.+8. Context-path ranker **may use query–edge embedding similarity** (no LLM); evaluate A/B.+9. Harness: evaluate both `max_facts=50` and `max_facts=40` arms when measuring diversification.+10. Write-time event-adjacency / hub-bridge index is owned by **memory substrate (`03`)** — Phase A does not build it; only retrieval determinism.++---++## Open questions for the maintainer++*Answered 2026-07-29 — see Maintainer decisions above.*++1. Should cross-event composition on the **context** path be limited to a write-time **entity-mediated hub adjacency** (cheap PPR over hubs), with full iterative composition reserved for the deep tier per ADR-006?+2. Is the primary success metric for this track **stable graph EvR** (especially multi-hop / single-hop categories), with judge accuracy as a secondary paired check — accepting Phase 1's finding that judge may stay flat?+3. What minimum **cross-run graph-session agreement** (e.g. ≥95% of questions identical on two identical-config runs) gates treating graph EvR as a measurement?+4. For diversification under `max_facts`, prefer **session coverage**, **distinct event-hub coverage**, or a blend — and what is the max latency tax (ms) you will accept on the context path?+5. Should path-shaped **cross-hub** strings enter `text_context` by default, or only a structured `paths` field for deep/MCP consumers?+6. When two hubs share an entity but disagree temporally, should context-path ranking prefer recency, return both, or defer to the current-truth filter only?+7. Is Leiden / community-summary GraphRAG explicitly out of scope for this brain (sparse event graph), or is a deterministic hierarchy (e.g. k-core) still interesting for open-domain sensemaking later?+8. May the context-path ranker use **query–edge embedding similarity** (no LLM) for CatRAG-style weights, or must all query-dependent weights stay at write time / deep tier?+9. For harness A/B, is `max_facts=50` the locked comparison budget, or should we also report a diversification arm at `max_facts=40` (schema default)?+10. Who owns the write-time event-adjacency / hub-bridge index — retrieval (`02`) or memory substrate (`03`)?++---++## Frontier techniques++Each technique judged against: (1) 5-node event model, (2) sub-second context budget / ADR-006, (3) Phase 1 finding that naive graph inclusion did not lift LoCoMo judge accuracy.++### A. HippoRAG / HippoRAG 2 — PPR over a memory graph++**Mechanism.** Offline open KG + passages; online seed entities; Personalized PageRank surfaces multi-hop-relevant nodes/passages in one shot.++**arXiv.** `2405.14831` (HippoRAG), `2502.14802` (HippoRAG 2).++**Reported gain.** Up to +20% on multi-hop QA vs prior SOTA; single-step matches/beats IRCoT at 10–30× lower cost and 6–13× lower latency. HippoRAG 2: +7% associative memory vs best embedding model without factual regression.++**Cost.** Offline graph build; online PPR cheap if adjacency is local/precomputed.++**Fit.** BrainAPI already runs PPR (`fact_filter.py:81-115`) but over a **clique of three nodes inside each flow_key**, not an entity–passage graph spanning hubs. Passages are a separate RRF channel.++**Verdict: adapt.** Keep PPR on the context path; rebuild adjacency as **hub-aware** (event nodes + entity bridges across `flow_key`s) at write time. Do not flatten events to OpenIE triples.++### B. CatRAG — query-aware traversal / anti-hub-drift++**Mechanism.** Attacks static transition matrices: symbolic anchoring, query-aware edge weights, key-fact passage bias so walks complete evidence chains instead of sinking into hubs.++**arXiv.** `2602.01965`.++**Reported gain.** Consistent wins on four multi-hop benchmarks; larger gains on *reasoning completeness* (full chain) than on partial recall.++**Cost.** Paper uses LLM edge scoring (deep-tier). Embedding-only reweight of outgoing legs can be context-tier.++**Fit.** EVENT hubs and OCCURRED_WITHIN contexts are natural sinks; uniform clique PPR matches the failure mode. `flow_key` is a ready symbolic anchor for *intra*-hub integrity.++**Verdict: adapt.** Context-tier: type-aware / query–embedding edge weights + symbolic seed anchors, no LLM. LLM edge scoring → deep tier only.++### C. PathRAG — relational paths + flow pruning + path prompts++**Mechanism.** Retrieve paths between query nodes; flow-based prune; prompt with path text (not flat bags).++**arXiv.** `2502.14902`.++**Reported gain.** Beats GraphRAG/LightRAG-style baselines across six datasets / five dimensions; average win rates ~58–60% vs those systems.++**Cost.** Path enumeration can be heavy online; prompting gets smaller.++**Fit.** Intra-hub path formatting already exists (`_format_event_fact`). Missing piece is **paths that chain hubs** via shared entities. Online multi-pair enumeration risks the latency budget.++**Verdict: adapt — split by tier.** Context: path-shaped rendering + cheap 1-bridge hub chains from a precomputed index. Deep: flow pruning / beam over `traverse_graph`.++### D. REMem — episodic memory graph + agentic retriever++**Mechanism.** Offline time-aware gist/fact hybrid graph; online agentic tools for iterative episodic reasoning.++**arXiv.** `2602.13530`.++**Reported gain.** +3.4 recollection / +13.4 episodic *reasoning* over Mem0 and HippoRAG 2; stronger refusal on unanswerable questions.++**Cost.** Agentic inference at query time; offline gists.++**Fit.** Closest to BrainAPI's event hubs. Phase 1 showed one-shot graph bag does not buy LoCoMo judge points — REMem's gain is on the *reasoning* half.++**Verdict: adopt as architectural reference.** Gists / indexes = write-time (context-admissible). Agentic episodic tools = deep tier only.++### E. Zep / Graphiti — temporal KG memory++**Mechanism.** Temporally-aware KG with validity; synthesises conversational + structured data.++**arXiv.** `2501.13956`.++**Reported gain.** DMR 94.8% vs 93.4%; up to +18.5% on LongMemEval with ~90% lower latency vs baselines.++**Fit.** Validity already filtered on triples; ranking still barely temporal. Crosses into `05-temporal-truth.md`.++**Verdict: adapt (ranking/filter features only).** Not a substitute for cross-event path retrieval.++### F. KAG — logical-form hybrid reasoning++**Mechanism.** LLM-friendly KG representation, chunk↔graph mutual index, logical-form-guided hybrid reasoning.++**arXiv.** `2409.13731`.++**Reported gain.** Relative F1 +19.6% (2Wiki) / +33.5% (HotpotQA) vs prior RAG in their setup.++**Cost.** Logical-form engine and alignment stack; multi-step.++**Fit.** Mutual indexing matches provenance goals; logical forms are iterative → deep tier. Must not collapse n-ary events to binary SPO.++**Verdict: adapt selectively.** Mutual index / provenance: yes. Full logical-form engine: deep tier; reject as context-path default.++### G. Leiden / community GraphRAG — reject for this graph++**Mechanism.** Community detection + hierarchical summaries for global sensemaking (`2404.16130` GraphRAG family).++**Counter-arXiv.** `2603.05207` proves modularity on sparse, low-degree graphs admits exponentially many near-optimal partitions → Leiden communities are **inherently non-reproducible**; proposes deterministic k-core hierarchies instead.++**Fit.** Clean event graph is sparse and hub-centric. Leiden fights G1 (determinism) and the 5-node model.++**Verdict: reject Leiden communities.** Optionally revisit **k-core** later for open-domain sensemaking only — out of critical path for multi-hop event QA.++### H. Budget-aware diversification (MMR / coverage)++**Mechanism.** Maximal Marginal Relevance and coverage objectives select items that are relevant *and* non-redundant under a hard budget (classic IR; widely reused in RAG pruning).++**arXiv / canon.** Carbonell & Goldstein MMR (SIGIR 1998) is the reference mechanism; recent graph-RAG work (PathRAG `2502.14902`) restates the redundancy diagnosis empirically.++**Reported gain.** Classic: better coverage at fixed k; PathRAG: quality↑ when redundancy↓.++**Cost.** O(k · n) over candidates; microseconds–low ms at `max_facts≤50`.++**Fit.** Direct answer to G3: diversify by **source session** and/or **event hub uuid** under `max_facts`.++**Verdict: adopt — context-tier.** Highest gain/$ after determinism.++### I. Deterministic ranking / stable ordering++**Mechanism.** Total orders with explicit tie-breaks; avoid concurrent unordered aggregation; optional ORDER BY in graph queries.++**Fit.** Root cause of G1 in this codebase (not a fashion paper). Concurrent `candidates.extend` + `sorted(key=score)` without secondary keys.++**Verdict: adopt immediately.** Prerequisite to every graph metric claim.++### J. Hybrid passage+graph fusion with channel attribution++**Mechanism.** Keep channels separate in logs/metrics; fuse only at answer time with explicit credit (Phase 0 already logs channels).++**Fit.** Phase 1: passages carry ~97% EvR and all detectable judge accuracy. Graph must earn keep on **stable graph EvR** and hard multi-hop misses, not by drowning the prompt.++**Verdict: adopt measurement discipline.** Prefer RRF/feature fusion that can ablate channels; reject “more facts into `text_context`” without paired McNemar + stable EvR.++### K. ToG / IRCoT-style iterative KG agents++**Mechanism.** LLM-guided beam / retrieve-then-reason loops.++**arXiv.** `2307.07697` (ToG); IRCoT family cited via HippoRAG comparisons in `2405.14831`.++**Verdict: adopt deep-tier only; reject on context path** (ADR-006).++### L. Hyper-relational / n-ary KG embeddings (HyNT, HEHRGNN, …)++**Mechanism.** Qualifier-aware or hyperedge embeddings for link prediction.++**arXiv.** e.g. `2305.18256`, `2602.18897`.++**Verdict: reject for now.** Training/serving cost; does not fix query-time hub composition or determinism; wrong layer vs ADR-006.++---++## Implementation plan++### Architecture decisions (locked for this plan)++1. **Determinism before lift.** No graph EvR A/B until identical-config agreement clears the gate (**≥95%** questions with identical graph-session sets — Q3 decided 2026-07-29).+2. **Event hubs stay first-class.** Never project to binary SPO for retrieval primary keys; diversify and traverse in hub/`flow_key` units.+3. **Context vs deep (ADR-006).** Context: indexes, precompute, cheap rank/diversify, ≤1 bridge hub composition from materialised adjacency. Deep: iterative composition, LLM/CatRAG edge scoring, ToG-style beams.+4. **Success metrics (binding).** On `locomoconv26clean`, de-fitted prompt, same flags as clean-brain eval arms:+ - **Primary:** stable graph EvR overall + per-category (multi-hop, single-hop); passage EvR must not regress.+ - **Secondary:** paired McNemar judge accuracy (≥2 runs/arm).+ - **Guardrail:** retrieve p50 (and stage timings); ADR-006 targets p50 < 1000 ms. Diversification latency tax ceiling is 2000 ms per maintainer — prefer both; escalate with numbers if tax exceeds remaining headroom to 1 s.+5. **Leiden out.** Per `2603.05207`. Hub-bridge index owned by memory substrate (`03`).+6. **Phase A scope.** Retrieval determinism only (total order, concurrent merge, harness stability). No write-time hub index, no MMR/diversification, no typed PPR.++### Measurement protocol (every phase)++```bash+# Identical-config stability (after Phase A)+./locomo.sh evaluate --sample conv-26 --brain locomoconv26clean \+ --run graph-stable-a --historical-limit 16 --max-passages 16 \+ --max-facts 50 --use-ppr --no-sufficiency-retry+./locomo.sh evaluate --sample conv-26 --brain locomoconv26clean \+ --run graph-stable-b ... # same flags+./locomo.sh compare --baseline graph-stable-a --candidate graph-stable-b++# Lift A/B (after Phase B+) — always 2 runs/arm for judge; 1 enough for passage EvR+./locomo.sh compare --baseline <control> --candidate <treatment>+```++Report: per-category judge accuracy, **graph** EvR full/partial, **passage** EvR, p50 ms, McNemar flip table. Do not claim judge wins below ~5–7 pts without paired significance (`00`).++---++### Phase A — Make graph metrics trustworthy++#### Task A.1 — Total order for fact ranking++**Description:** Add deterministic secondary keys to every sort that feeds `curated`: e.g. `(score, ppr_score, flow_key, r.uuid, r2.uuid)` (direction depends on ascending/descending). Ensure PPR reorder uses the same total order.++**Acceptance criteria:**+- [ ] Two sequential in-process `get_context` calls with fixed mock seeds/neighbors return byte-identical `triples` UUIDs and order.+- [ ] Unit test covers equal primary scores.++**Verification:**+- [ ] `pytest tests/ -k "fact_order or context_max_facts or deterministic"` (new test file OK).++**Dependencies:** None+**Files likely touched:** `src/services/api/controllers/retrieve.py`, `tests/test_context_*determinism*.py`+**Estimated scope:** S++#### Task A.2 — Deterministic candidate aggregation under concurrency++**Description:** Stop relying on race order from `asyncio.gather` + `candidates.extend`. Collect per-variant lists, then merge with a deterministic reduce (sort variants, then facts by total order before dedupe). Optionally `ORDER BY` in Cypher for neighbor rows.++**Acceptance criteria:**+- [ ] Under thread pool stress (e.g. 20 parallel identical requests), set of `(r.uuid,r2.uuid)` in top `max_facts` is identical across requests.+- [ ] Neo4j/NetworkX backends agree on membership for the same seeds (order may still be normalised in Python).++**Verification:**+- [ ] Stress unit/integration test; two harness runs `graph-stable-a/b` show graph-session agreement ≥ gate (Q3).++**Dependencies:** A.1+**Files likely touched:** `retrieve.py`, optionally `neo4j/client.py`+**Estimated scope:** M++#### Task A.3 — Harness stability report for graph channel++**Description:** Compare tool (or extend `compare`) emits per-question graph-session agreement rate and lists disagreeing `qa_index`s. Document gate in `CHECKPOINT_NOTES.md` style comment in compare output.++**Acceptance criteria:**+- [ ] One command prints % questions with identical graph session-recall boolean and identical retrieved graph session sets.+- [ ] README/research note: “do not A/B graph EvR below gate.”++**Verification:**+- [ ] Run on pre-fix vs post-A.2 artifacts; agreement rises past gate after A.2.++**Dependencies:** A.2 for the fix; can implement reporter in parallel with A.1+**Files likely touched:** `benchmarks/locomo/report.py` or `compare` path only+**Estimated scope:** S++### Checkpoint A++- [ ] Identical-config graph agreement **≥95%** questions identical session sets (maintainer gate, 2026-07-29).+- [ ] Passage EvR still bit-identical across runs.+- [ ] **Stop:** no diversification or composition work until this passes.+- [ ] `compare` prints graph-session agreement rate and disagreeing `qa_index`s; do not A/B graph EvR below the gate.++---++### Phase B — Raise stable graph EvR (single-hop + multi-hop) without blowing latency++#### Task B.1 — Hub- and session-aware diversification under `max_facts`++**Description:** Replace pure `ranked[:max_facts]` with MMR/coverage: relevance = distance/PPR score; redundancy = shared `flow_key` / event hub uuid / source session. Keep 5-tuples intact.++**Acceptance criteria:**+- [ ] At fixed `max_facts=50`, distinct `graph_session_ids` and distinct hub ids in curated set both increase vs control on a frozen candidate list fixture.+- [ ] Stable under A.1 total order.+- [ ] Added latency < 5 ms at 200 candidates (unit benchmark).++**Verification:**+- [ ] Two-run arm vs control on `locomoconv26clean`: **stable** graph EvR ↑ on multi-hop and single-hop; passage EvR within noise; p50 delta < 50 ms.+- [ ] McNemar recorded (expect possibly ns — still ship if EvR↑).++**Dependencies:** Checkpoint A+**Files likely touched:** `retrieve.py`, `fact_filter.py` (or new small helper), tests+**Estimated scope:** M++#### Task B.2 — Intra-hub completeness scoring (single-hop via event hubs)++**Description:** When a seed hits any of `{n,m,b}`, score the **whole** 5-tuple using a hub completeness feature (both legs present, MADE/TARGETED polarity, OCCURRED_WITHIN as context bonus not hub replacement). Prefer complete triangles over partial/noisy paths.++**Acceptance criteria:**+- [ ] Fixture: seed on object alone still surfaces the actor–event–object fact when present in neighbors.+- [ ] No binary triple projection.++**Verification:**+- [ ] Unit tests on synthetic 5-tuples; category single-hop stable graph EvR non-decreasing on clean brain.++**Dependencies:** A.1+**Files likely touched:** `retrieve.py`, tests+**Estimated scope:** S++#### Task B.3 — Type-aware PPR transitions (CatRAG-lite, no LLM)++**Description:** Replace uniform clique edges with typed weights (e.g. higher mass along MADE/TARGETED spine than OCCURRED_WITHIN; optional query–predicate embedding cosine). Apply `_is_currently_valid` when building adjacency so PPR and facts agree.++**Acceptance criteria:**+- [ ] Adjacency builder skips invalid legs.+- [ ] Ablation: typed vs uniform on frozen graph improves multi-hop graph EvR without p50 > +30 ms.++**Verification:**+- [ ] Unit tests for validity + weights; clean-brain eval arm.++**Dependencies:** A.2, ideally B.1+**Files likely touched:** `retrieve.py`, `fact_filter.py`, tests+**Estimated scope:** M++#### Task B.4 — Path-shaped prompt hygiene (single hub)++**Description:** Make `_format_event_fact` explicitly label roles (Actor / Event / Target / Context) without adding LLM calls; ensure answerer sees structure PathRAG argues for. Keep backward-compatible plain line or gate behind flag.++**Acceptance criteria:**+- [ ] Structured path string round-trips hub roles in unit test.+- [ ] Token delta measured; no passage EvR regression.++**Verification:**+- [ ] Optional one-run prompt A/B; decide on retrieval+judge protocol.++**Dependencies:** None (can parallelise after A)+**Files likely touched:** `retrieve.py`, possibly harness prompt only if needed+**Estimated scope:** S++### Checkpoint B — **FAILED primary EvR lift (2026-07-29)**++**Shipped (code):**+- Hub/session MMR diversification under `max_facts` (`_diversify_facts`)+- Intra-hub completeness scoring+- Typed / validity-consistent PPR adjacency+- Path-shaped single-hub format (`Actor: … | Event: … | Target/Context: …`)+- Optional `temporal_conflicts` meta (prefer recency in rank, keep both as meta)++**Live on `locomoconv26clean` (retrieve-only arms `phase-b-ret-mf50-a/b`, `phase-b-ret-mf40-a`):**+- Graph-session stability **99.3%** — Checkpoint A gate **held**+- Graph EvR **flat 47.3%** vs `graph-stable-a` (same overall; cat1 multi-hop **25%**, cat4 single-hop **57%**) — Checkpoint B EvR lift **NOT met**+- Concurrent retrieve p50 ~2.2 s under 4 workers; sequential/harness-comparable ~**948 ms** (−404 ms vs ~1351 ms `graph-stable-a` baseline) — latency tax of diversification itself is negligible++**Conclusion:** re-ranking / diversifying within the same intra-hub candidate pool cannot raise session coverage. Need **cross-event candidate expansion** (Phase C hub-bridge index). Proceed to Phase C without requiring judge McNemar on Phase B.++Checkpoint B checklist status:+- [x] Code shipped (MMR, completeness, typed PPR, path format, temporal_conflicts)+- [ ] Stable graph EvR materially above control — **failed** (flat 47.3%)+- [x] Multi-hop / single-hop reported (25% / 57%); no concentrated lift+- [x] Passage EvR not targeted; stability gate held+- [ ] Judge McNemar — deferred (EvR flat → composition first)++---++### Phase C — Cross-event composition (tiered)++#### Task C.1 — Write-time entity-mediated hub bridge index++**Description:** At ingest (or rebuild job), materialise edges between **event hubs** that share a non-EVENT entity (actor/object/context), with weight = co-occurrence / recency. This is the context-tier substrate for cross-`flow_key` PPR without online multi-hop Cypher.++**Acceptance criteria:**+- [ ] Index maps `event_uuid → [(neighbor_event_uuid, shared_entity, weight)]`.+- [ ] Rebuild deterministic for a fixed brain.+- [ ] Document ownership (open Q10).++**Verification:**+- [ ] Unit test on a 2-hub diamond fixture; rebuild idempotent.++**Dependencies:** Checkpoint A; substrate coordination+**Files likely touched:** ingestion/write path (`03`/`01`), thin read API in graph adapter+**Estimated scope:** M++#### Task C.2 — Context-path 1-bridge expansion via hub index++**Description:** After seed hubs from `get_event_centric_neighbors`, pull top-K neighboring hubs from C.1 (K small, e.g. 2–5), expand their 5-tuples, merge into candidates, then B.1 diversify. Hard cap on extra Cypher/index lookups to protect p50.++**Acceptance criteria:**+- [ ] At least one fixture question requires two hubs; curated set contains both after expansion.+- [ ] Stage timing: expansion < 50 ms p50 on conv-26-scale brain.+- [ ] Config flag `cross_event_bridges: 0|K` default conservative.++**Verification:**+- [ ] Clean-brain arm: multi-hop **stable** graph EvR ↑; p50 within budget headroom; McNemar vs Checkpoint B best arm.++**Dependencies:** C.1, B.1+**Files likely touched:** `retrieve.py`, requests schema, tests+**Estimated scope:** M++#### Task C.3 — Deep-tier cross-event composition route++**Description:** MCP / deep REST: ToG-style bounded beam or REMem-like tools over hub bridges + `traverse_graph`, with step cap and path isolation. Context path emits insufficiency / `needs_composition` signal (ADR-006) instead of silent weak context.++**Acceptance criteria:**+- [ ] Terminates within N steps on 50-question sample.+- [ ] Returns explicit paths across ≥2 hubs.+- [ ] Not invoked from default `/retrieve/context`.++**Verification:**+- [ ] Deep-tier eval bucket (multi-hop failures of context path); paired metrics; separate latency report.++**Dependencies:** C.1; ADR-006 router signals+**Files likely touched:** MCP tools, optional new deep retrieve controller+**Estimated scope:** M++#### Task C.4 — Channel-attributed fusion report++**Description:** For each eval arm, report questions where gold sessions appear in graph-only, passages-only, both, neither — so graph “earns keep” is visible even when judge is flat.++**Acceptance criteria:**+- [ ] Report table in `report.md` / compare output.+- [ ] Used in Checkpoint C go/no-go.++**Verification:**+- [ ] Recompute on Phase 1 artifacts for sanity.++**Dependencies:** Phase 0 logging (done)+**Files likely touched:** `benchmarks/locomo/report.py`, `metrics.py`+**Estimated scope:** S++### Checkpoint C — **primary EvR lift NOT met (2026-07-29)**++**Shipped (code):**+- Write-time / rebuild entity-mediated hub-bridge index (`kg_hub_bridges` on NetworkX/Postgres; `:HUB_BRIDGE` on Neo4j)+- Context-path ≤1-bridge expansion (`cross_event_bridges`, default 3; seed-hub cap 12) then Phase B diversify+- Additive structured `paths` on `GetContextResponse` (not dumped into `text_context`)+- Backfill on `locomoconv26clean`: **13 660** bridges (no re-ingest)++**Live retrieve-only (`phase-c-ret-mf50-a/b`, sequential):**+- Stability **98.7%** — gate held+- Graph EvR **47.3–48.0%** vs phase-b / graph-stable-a **47.3%** — Δ within 1/150 questions (noise); cat1 multi-hop still **25%**, cat4 **57–58.5%**+- Sequential p50 **~1340–1391 ms** (≈ baseline 1351 ms; under 2000 ms tax ceiling; still above ADR-006 1 s target)++**Conclusion:** entity-mediated 1-bridge expansion runs and emits paths, but does **not** move session coverage. Bridges via shared speakers mostly stay inside already-covered session neighbourhoods under `max_facts`. Checkpoint C EvR lift **NOT met**.++Next (recommended): rank/filter bridges by **novel source session** (prefer hubs whose provenance sessions are absent from the seed set); optionally raise bridge priority in MMR; keep deep-tier composition for residual multi-hop. Do not enable larger K without a session-novelty prior.++Checkpoint C checklist status:+- [x] ≤1-bridge expansion on context path (flag `cross_event_bridges`)+- [ ] Multi-hop stable graph EvR improved — **failed** (flat)+- [ ] Deep-tier composition route — out of this delivery (stub later)+- [ ] Fusion report C.4 — not required for go/no-go given flat EvR+- [x] Maintainer: keep `cross_event_bridges` default-on for substrate readiness; EvR payoff needs session-aware bridge ranking++### Checkpoint C2 — novel-session ranking **null** (2026-07-29)++**Shipped:** prefer ≤1-bridge expansions whose target hubs contribute source sessions absent from the seed/selected set (`select_bridge_neighbors` + pool fetch of hub sessions). K unchanged (`cross_event_bridges=3`).++**Live (`phase-c2-novel-session-a/b`, retrieve-only):**+- Stability **98.0%** — gate held+- Graph EvR **47.3%** flat vs phase-b / graph-stable-a; cat1 **25%**, cat4 **57.1%**+- Sequential p50 **~1876–1924 ms** (~**+500 ms** vs phase-c ~1340 ms) — pool fetch + chunk→session resolve++**Verdict:** novelty ranking alone does **not** lift EvR. Cost is real; EvR benefit is zero.++### Diagnostic — 1-bridge gold-session reachability (2026-07-29)++**Question:** among QAs where the graph channel misses gold evidence sessions on `locomoconv26clean`, does *any* 1-bridge neighbor of the retrieved seed hubs (cap 12) intersect those missing sessions?++**Method:** incomplete rows from `phase-c2-novel-session-a`; fresh `/retrieve/context` with `cross_event_bridges=0` for seed hubs; full `kg_hub_bridges` neighbourhood in Postgres; neighbor hub → session via `source_chunk_ids` → chunk text. Selection check: `cross_event_bridges=3` paths (`hubs` field) + final `graph_session_ids`. Artifacts: `.agent_tmp/hub_bridge_reachability.json`, `hub_bridge_reachability_selection.json`.++| Slice | Rate |+| --- | --- |+| Incomplete QAs where ≥1 gold-missing session is in the 1-bridge neighbourhood | **72/79 = 91.1%** |+| Same, cat1 multi-hop only | **22/24 = 91.7%** |+| Among reachable: emitted `paths` include a gold-covering neighbor hub | **64%** overall / **82%** cat1 |+| Among reachable: final curated `graph_session_ids` covers a gold-missing session | **0%** |+| Mean neighbor hubs that actually carry a missing gold session | **~5** (not “the whole graph”) |++**Unreachable residual (~9%):** mostly **session_11** — its 5 EVENT hubs use non-spine predicates (`ATTENDED`, `INVOLVED`, …) and have **0** rows in `kg_hub_bridges` (202/410 events are bridge-orphans under current spine filters). That slice is a substrate-construction gap, not a ranker bug.++**Interpretation:** intersection ≫ 0 → **do not pivot off speaker-mediated 1-bridge**. The substrate *can* reach gold sessions; C/C2 fail because **bridge facts lose the `max_facts` budget** after expand (paths often right, curated set never keeps the missing session). C2 novelty is insufficient as the sole lever and adds ~500 ms — optional latency revert is safe for EvR (null), but prefer fixing post-expand retention first so path hits can become EvR hits.++**Recommended next move (single):** reserve / hard-prioritize a small number of curated slots for **bridge-expanded facts that add novel sessions** (post-expand diversify must not drop them). Secondary (parallel, not instead): widen spine predicates (or membership rules) so orphan events like session_11 enter `kg_hub_bridges`. Deep-tier only for residual after retention lands.++### Checkpoint C3 — reserved bridge slots + ANN seed stability (**dual gate PASS**, 2026-07-29)++**Shipped (code):**+- Post-expand **reserved curated slots** for bridge-expanded facts that add novel sessions (`_BRIDGE_RESERVE_HUB_CAP`, ship setting **`reserve_hub_cap=12`** matching the expand seed shortlist)+- ANN seed determinism fixes for HNSW top-k without uuid order + embed float noise:+ - over-fetch then **uuid-stable** top-k (`ann_overfetch_k` + `stable_top_k_vectors`)+ - seed-hit stabilize (`_stabilize_seed_hits`)+ - **3 decimal-place** distance quantization (`_quantize_distance`)++**Live retrieve-only (`phase-c3-ann-stable-a/b`, sequential, `locomoconv26clean`, mf=50):**++| Metric | C3+ANN | Prior |+| --- | --- | --- |+| Graph-session agreement | **99.3%** | C3 reserved-slots pre-ANN **77.6%**; Phase A/B **99.3%** / C **98.7%** |+| Graph EvR full | **56.0%** | Phase A–C2 **47.3%** |+| Cat1 multi-hop EvR | **40.6%** | **25%** |+| Sequential p50 | **~1320 ms** | ~1351 ms graph-stable / ~1381 ms C3 reserved |++**Dual gate:** stability ≥95% **and** EvR lift (full ≥56%, cat1 ≥37.5%) → **PASS** (`dual_gate_ok=true`). Artifact: `benchmarks/runs/phase-c3-ann-stable-summary.json`.++**Root cause of the C3 stability regression (77.6%):** HNSW approximate top-k without uuid tie-break, amplified by embedding float noise → unstable seed sets → unstable reserved hubs. Fixes above restore Checkpoint A agreement while keeping the EvR lift.++**Ship setting:** `reserve_hub_cap=12` (code: `_BRIDGE_RESERVE_HUB_CAP = 12`).++**Secondary metric — judge accuracy (full `evaluate`, 2026-07-29):**++Config: de-fitted prompt, `locomoconv26clean`, mf=50, `use_ppr`, historical 16, max_passages 16. Arms `phase-c3-judge-a/b`.++| Arm | Acc | Multi-hop | Single-hop | Graph EvR | Agree A↔B | p50 ms |+| --- | --- | --- | --- | --- | --- | --- |+| `phase-c3-judge-a` | 77.6% | 65.6% | 87.1% | **56.0%** | — | 1456 |+| `phase-c3-judge-b` | 78.3% | 65.6% | 90.0% | **56.0%** | **99.3%** | 1341 |+| `graph-stable-a` (clean, pre-C3) | 80.3% | 75.0% | 90.0% | 47.3% | — | ~1351 |+| `baseline-clean-a` (old brain) | 82.2% | 71.9% | 91.4% | 0%\* | — | 4776 |++\*Old-brain graph EvR unmeasurable in that run's logging era.++Identical-config full-evaluate stability: graph-session agreement **99.3%** (151/152); EvR full identical **56.0%** both arms — dual gate **held** under answerer load.++McNemar (exact):++| Comparison | Acc | Right / Wrong | Overall p | Multi-hop p | Single-hop p |+| --- | --- | --- | --- | --- | --- |+| A → B (noise) | 77.6→78.3 | 8 / 7 | 1.0 | 1.0 | 0.69 |+| `baseline-clean-a` → A | 82.2→77.6 | 3 / 10 | 0.092 | 0.63 | 0.25 |+| `baseline-clean-a` → B | 82.2→78.3 | 7 / 13 | 0.26 | 0.69 | 1.0 |+| `baseline-clean-a` → A+B pooled | 82.2→78.0 | 9 / 15 | **0.31** | 0.73 | 1.0 |+| `graph-stable-a` → A+B pooled | 80.3→78.0 | 10 / 16 | **0.33** | 0.34 | 0.73 |++**Verdict:** EvR lift **did not convert to answers**. Point estimate is slightly down vs clean-brain `graph-stable-a` and old-brain `baseline-clean-a`; all McNemar ns at n=152. Answerable stays **96.7%** while judge ~78% → answerer gap **~18–19%** (G4 / Phase D territory). Primary C3 claim remains the dual-gate EvR unlock.++**Recommended next:** Phase D.1 measured (see Checkpoint D.1). **Do not auto-start D.2** — maintainer fork: D.2 feature reranker **vs** Checkpoint D product-surface write-up (graph value = EvR/provenance, not LoCoMo judge).++Checkpoint C3 checklist status:+- [x] Reserved slots for novel-session bridge facts under `max_facts`+- [x] Stable graph EvR materially above 47.3% control (**56.0%**, cat1 **40.6%**)+- [x] Identical-config agreement ≥95% (**99.3%** retrieve-only and full evaluate)+- [x] Dual gate PASS; ship `reserve_hub_cap=12`+- [x] Judge McNemar secondary: ns vs baseline / graph-stable; EvR↑ answers flat → Phase D++---++### Phase D — Only if graph EvR↑ but answers stay flat++#### Task D.1 — Answerer consumes paths, not bags — **measured (soft convert, 2026-07-29)**++**Description:** When structured paths exist, harness/product prompt prefers path blocks for multi-hop questions (no gold fitting). Measure judge with paired McNemar; keep prompt-audit.++**Acceptance criteria:**+- [x] `prompt-audit` still passes.+- [x] Multi-hop McNemar vs bag prompt recorded (vs C3 pooled; ns).++**Dependencies:** B.4, C.2+**Estimated scope:** S++#### Task D.2 — Feature reranker in empty filter socket (no LLM) — **not started**++**Description:** Linear/GBDT over distance, PPR, hub completeness, session novelty, provenance overlap with top passages; lower `max_facts` if EvR holds (ADR-006 §Alternatives).++**Acceptance criteria:**+- [ ] <5 ms / 50 candidates; no network.+- [ ] `max_facts=20` matches `max_facts=50` stable graph EvR within tolerance.++**Dependencies:** Checkpoint B+**Estimated scope:** M++### Checkpoint D.1 — paths into answerer (**soft convert**, 2026-07-29)++Config: de-fitted prompt, `locomoconv26clean`, mf=50, `use_ppr`. Arms `phase-d1-paths-a/b`. Artifact: `benchmarks/runs/phase-d1-paths-summary.json`.++| Arm | Acc | Multi-hop | Graph EvR | Agree A↔B |+| --- | --- | --- | --- | --- |+| `phase-d1-paths-a` | **80.9%** | 71.9% | **56.0%** | — |+| `phase-d1-paths-b` | **80.3%** | 75.0% | **56.0%** | **100%** |+| pooled D.1 | **80.6%** | — | 56.0% | — |+| C3 judge pooled | 78.0% | 65.6% | 56.0% | 99.3% |+| `baseline-clean-a` (old brain) | 82.2% | 71.9% | 0%\* | — |++\*Old-brain graph EvR unmeasurable in that run's logging era.++McNemar (exact, pooled):++| Comparison | Acc | Right / Wrong | p |+| --- | --- | --- | --- |+| C3 pooled → D.1 pooled | 78.0% → 80.6% | **17↑ / 9↓** | **0.17** ns |+| `baseline-clean-a` → D.1 pooled | 82.2% → 80.6% | 10↑ / 16↓ | **0.33** ns |++**Held:** Graph EvR **56.0%** (C3 dual-gate level); A↔B agreement **100%**; `prompt-audit` **passed**. No gold-fitting claims.++**Verdict:** **soft convert** — directionally up vs C3 on judge (point estimate +2.6 pp; McNemar ns at n=152). Answerable stays high while judge ~80.6% → answerer gap **~16%** (G4 still open). Paths help the answerer somewhat but do not clear the significance floor.++**Measurement note (TUI):** D.1 evaluate used harness `enrich_paths_from_triples` because the first TUI restart was blocked (`EPERM` unlink on `~/.brainapi/source/src/.DS_Store`). Workspace already had `retrieve.py` `_paths_for_curated` attaching fact `legs` to curated bridge paths. **Follow-up (same day):** `.DS_Store` removed; `brainapi start --no-services` synced workspace → `~/.brainapi/source` and restarted API; live `/retrieve/context` returns paths with native `legs` (probe: `native_legs_present=true`). Harness enrich remains a safe fallback when legs are already present (no-op).++**Next (maintainer fork — do not auto-start D.2):**+1. **D.2** feature reranker in the empty filter socket, **or**+2. **Checkpoint D product-surface write-up** — graph value = stable EvR / provenance / product surfaces, not LoCoMo judge conversion (Phase 4 gate in `06`).++### Checkpoint D++- [ ] Either judge multi-hop moves with paired significance, or written decision that graph value is EvR/product-surface only (per Phase 4 gate in `06`).+ - D.1: soft convert only (ns); fork above still open.+---++### Explicitly out of scope / reject list++| Item | Why |+| --- | --- |+| Leiden community summaries | Non-reproducible on sparse KGs (`2603.05207`); ignores event hubs |+| Flattening to (s,p,o) for primary retrieval | Violates Triangle of Attribution |+| LLM fact filter / CatRAG LLM edge scorer on context path | ADR-006 |+| Online IRCoT/ToG on `/retrieve/context` | Latency |+| Justifying work solely by LoCoMo headline | Phase 1 gate |++---++## Risks++| Risk | Impact | Detection |+| --- | --- | --- |+| **More graph facts, same answers (G4)** | High — wasted complexity | Channel fusion report (C.4); McNemar ns + flat answerer gap → pivot to D.1 or stop |+| Treat graph EvR as signal before determinism | High | Checkpoint A gate; refuse A/B |+| Diversification helps EvR but hurts precision | Medium | Watch single-hop judge flips; tune λ in MMR |+| Cross-event bridges create false multi-hop | High | Manual audit of new paths; require shared-entity + type constraints; Janitor hygiene |+| Latency regression undoes dossier win | High | `profile_stages` p50; budget CI on arms |+| PPR typed weights unstable across embed models | Medium | Freeze embed model in manifest; unit golden vectors |+| Clean brain still one conversation | Medium | Do not generalise; optional second sample later (`00` deferred scaling) |+| Concurrent ingest / wrong server SHA | Medium | One owner per brain; server code identity in manifest (`06`) |++---++## Critical path (task titles only)++1. Total order for fact ranking+2. Deterministic candidate aggregation under concurrency+3. Harness stability report for graph channel+4. **Checkpoint A — graph metrics trustworthy**+5. Hub- and session-aware diversification under `max_facts`+6. Intra-hub completeness scoring+7. Type-aware PPR transitions (CatRAG-lite)+8. Path-shaped prompt hygiene (single hub)+9. **Checkpoint B — stable graph EvR lifted**+10. Write-time entity-mediated hub bridge index+11. Context-path 1-bridge expansion via hub index+12. Deep-tier cross-event composition route+13. Channel-attributed fusion report+14. **Checkpoint C — composition tiered**+15. Reserved novel-session slots + ANN seed stability+16. **Checkpoint C3 — dual gate (stable EvR lift) PASS**+17. **Checkpoint D.1 — paths into answerer (soft convert; ns)**+18. Maintainer fork: D.2 feature reranker **or** Checkpoint D product-surface write-up (stop justifying on LoCoMo judge)++---++## Relation to other docs++- Extends `02-retrieval-multihop.md` with an **event-hub-first** execution plan after Phase 1.+- Satisfies `06` Phase 4 items (1)–(2): stabilize EvR, then composition — without reopening passages-only judge flatness.+- Obeys ADR-006 two-tier split and `00` measurement noise floor.diff --git a/docs/research/08-sota-locomo-protocol.md b/docs/research/08-sota-locomo-protocol.mdnew file mode 100644index 0000000..f406e54--- /dev/null+++ b/docs/research/08-sota-locomo-protocol.md@@ -0,0 +1,74 @@+# SOTA LoCoMo Protocol Sheet++Goal: beat HyperMem-class LoCoMo (≥93% LLM-as-a-judge) under a documented protocol. Product `/retrieve/context` stays ADR-006 (sub-second, no online LLM loops). SOTA gains live in write-time index + harness generation stack.++**Answerer choice (maintainer):** `deepseek-v4-flash` is the preferred SOTA answer/judge model — treated as substantially stronger than gpt-4o on this stack. Do not require OpenAI/Azure for the leaderboard track.++## Competitor stacks (from papers)++| System | Paper | Overall J | Corpus | Answer model | Judge | Notes |+| --- | --- | ---: | --- | --- | --- | --- |+| **HyperMem** | [2604.08256](https://arxiv.org/abs/2604.08256) | **92.73%** | LoCoMo (reported full) | GPT-4.1-mini + CoT | GPT-4o-mini (mean of 3 runs) | Topic→episode→fact; BM25∪dense RRF; Qwen3-Embedding/Reranker-4B; top-10/10/30 |+| **APEX-MEM** | [2604.14362](https://arxiv.org/abs/2604.14362) | **88.88%** (w/ Adv overall; 89.49% w/o Adv with GPT5) | LoCoMo full cats | GPT5 / Claude / GPT4o agent | LLM-as-judge (mean of 3) | Append-only event graph + multi-tool ReAct (≤40 tools) |+| **Mem0 / Mem0ᵍ** | [2504.19413](https://arxiv.org/abs/2504.19413) | **66.88% / 68.44%** | LoCoMo full, excl. adversarial | gpt-4o-mini family | LLM-as-judge (10 runs ±1σ) | Dense NL memories (+ optional graph); T=0 |+| **MIRIX** (cited) | via HyperMem/APEX tables | ~85.38% | LoCoMo | unknown | LLM-as-judge | Strong memory-system baseline |+| BrainAPI product | this repo | ~82–84% honest | **conv-26 only** (n=152) | deepseek-v4-flash | same family | Greedy product profile |+| BrainAPI SOTA (conv-26) | this repo | **84.9%** | conv-26 | deepseek-v4-flash + SC/gap-fill | same family | `phase-sota-d1-conv26` |++Gaps marked **unknown** when a paper omits temperature, exact prompt, or sample filter. Cross-paper J scores are not strictly comparable (different judges/answerers).++## BrainAPI dual profiles++| Env / knob | `product` (default) | `sota` |+| --- | --- | --- |+| `BENCH_PROFILE` | `product` | `sota` |+| Answer model | `deepseek-v4-flash` | same (`BENCH_SOTA_ANSWER_MODEL` optional override) |+| Judge model | answerer family by default | same as answerer unless `BENCH_SOTA_JUDGE_MODEL` set |+| `sc_samples` | 1 (greedy T=0) | 5 (Self-Consistency, T=0.7) |+| Gap-fill | off | on (`BENCH_GAP_FILL=1`) |+| prompt-audit | must pass | must pass |++Manifest fields: `bench_profile`, `sc_samples`, `gap_fill`, answer/judge models, `judge_shares_answer_family`.++## Win condition++≥93% headline LLM-as-a-judge on **full LoCoMo10** (non-adversarial) under `BENCH_PROFILE=sota` with **deepseek-v4-flash**, prompt-audit green, no gold-fitting. Always also report the product (greedy) number and a memory-off ablation when claiming SOTA.++## Technique mapping++| Paper technique | BrainAPI landing |+| --- | --- |+| HyperMem hierarchy + RRF | Write-time topic session index + coarse-to-fine session expand on context path (no LLM) |+| APEX-MEM append-only + query-time resolve | Keep append-only graph; agentic resolve only on deep/MCP + SOTA harness |+| Self-Consistency | Harness SOTA answer decoding |+| FAIR-RAG evidence gaps | Harness one-shot gap-fill re-retrieve |+| SGMem raw+structured | Passages + facts + paths composition |++## Results log++| Date | Arm | Scope | Profile | Acc | Notes |+| --- | --- | --- | --- | --- | --- |+| 2026-07-29 | `phase-sota-d1-conv26` | conv-26 / `locomoconv26clean` | sota: deepseek-v4-flash, SC=3, gap-fill, hardened prompt + topics | **84.9%** | Open-domain **69.2%**; answerer gap **9.2%**. McNemar vs D.1 ns. Topic session-id leak in text_context fixed after run. |+| (queued) | `sota-locomo10-*` | full LoCoMo10 | sota + deepseek-v4-flash | | Ingest `sota-locomo10-ingest`. Eval: `benchmarks/scripts/run_sota_locomo10.sh` |++## Failure taxonomy (Task 0.2)++From `phase-d1-paths-a`: **100%** of 29 wrong non-adversarial rows classified generation-side (`benchmarks/runs/sota-failure-taxonomy.md`). Top classes: present-but-unused (~7pp), multi-hop composition (~6pp), temporal format (~4pp).++## Implementation landed++- Harness: `BENCH_PROFILE`, SC majority vote, gap-fill, hardened prompt, prompt-audit green+- Memory: write-time topic session index (`topic_hyperedges.py`), coarse-to-fine on `/retrieve/context`, topics in response + context composition+- Ops: `benchmarks/scripts/run_sota_locomo10.sh`++## Comparison vs published SOTA (not apples-to-apples)++| System | Overall J | Notes |+| --- | ---: | --- |+| HyperMem | 92.73% | GPT-4.1-mini / GPT-4o-mini judge; full LoCoMo |+| APEX-MEM | 88.88% | GPT5 agent + tools |+| Mem0ᵍ | 68.44% | gpt-4o-mini family; full LoCoMo |+| BrainAPI product (greedy) | ~82–84% | conv-26 |+| BrainAPI SOTA profile | **84.9%** | conv-26; deepseek-v4-flash + SC/gap-fill |++To claim ≥93%: finish full-10 ingest, then `BENCH_PROFILE=sota` evaluate with deepseek-v4-flash (SC=5). Always report greedy product number alongside.