BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
11 — Architect ingest loop efficiency (peer-competitive tokens)
Workstream: cut accurate-mode ingest LLM token multipliers by roughly an order of magnitude while preserving event-hub graph shape and LoCoMo / LongMemEval quality bars in 00-scope-and-constraints.md and 10-ingest-cost-latency.md. Architecture changes to the Architect loop are allowed. PIPELINE_MODE=lightweight remains out of scope as a cost lever. Retrieval /retrieve/context must stay cheap (no query-time LLM loops).
Extraction quality ownership stays with 01-ingestion-extraction.md; this document owns the Architect / Janitor control-flow redesign that dominates measured spend.
Problem statement (measured)
Baseline A/B (2026-07-30)
Source: benchmarks/runs/ingest-cost-baseline.json — LoCoMo conv-26, sessions 1–5, PIPELINE_MODE=accurate.
| Arm | mean wait | mean LLM tokens / session | Architect mean tokens | Janitor mean tokens | Architect+Janitor share |
|---|---|---|---|---|---|
Legacy hotpath (ingest-cost-legacy) |
630 s | 1.19M | 1.19M (Janitor folded into Architect) | (folded) | ~99%+ |
Cheap defaults (ingest-cost-cheap2) |
223 s | 0.73M | 0.38M | 0.34M | 99.5% |
C0–C3 shipping (per-unit Architect, deferred batched Janitor, Observations/consolidator off, grounding precheck) already cut mean wait ≈65% and mean tokens ≈39%. Scout remains ~0.5% of spend (3.8k tokens / session). Post-CP3 ship path: batch+scratchpad is **15.3×** source tokens vs cheap ~886× — see multiplier section and architect-cp2-NOTES.md / architect-cp3-NOTES.md.
Token multiplier vs source text
Pinned denominator: tiktoken cl100k_base, 4,114 source tokens on conv-26 sessions 1–5 (ingest-cost-baseline.json).
| Era | Arm | LLM÷source |
|---|---|---|
| Pre-redesign | Cheap defaults (ingest-cost-cheap2) |
~886× |
| CP2 | Batch + raw prior (architect-cp2-NOTES.md) |
18.8× |
| CP3 (ship) | Batch + scratchpad (architect-cp3-NOTES.md) |
15.3× |
Old cheap ~886× → batch+scratchpad ~15.3× on the same five sessions. The ≤100× ship gate is met; stretch 20–50× is also met.
The requested 20–100× band remains a peer-shaped engineering target, not a published apples-to-apples LightRAG/Graphiti comparison.
Why spend concentrates in Architect+Janitor
Traced current workspace path:
- Scout emits structured entities per ≤6k-character unit (
scout_agent.py:385-410,:503-533)—already close to the peer control-flow shape. auto_kg.py:156-194calls Architect once per Scout unit by default, passing the current unit plus up to four prior raw units (auto_kg.py:33,:70-77).- The call is
ArchitectAgent.run(...), notrun_tooler(auto_kg.py:171-183).ArchitectAgentdefaults toagent_mode="single"(architect_agent.py:257-268), sorun()dispatches through_run_single()torun_structured()(architect_agent.py:593-633,:1161-1193). - “Structured” is not one-shot here.
run_structured()binds five state/relationship tools andArchitectAgentSingleResponse(architect_agent.py:1203-1243), then runs an outer coverage loop up to four times (:1245-1328). Each outer iteration can enter the customAgentBasetool loop. - The custom backend is the default (
config.py:584-586;runtime_agent_factory.py:25-37). It sends the accumulatedagent.messageson every next tool turn (invoke_loop.py:372-428,:432-502) and has tracing thresholds but no explicittool_loop_counttermination guard (invoke_loop.py:110-147,:432-445). Native tool schemas, the output schema, the long system prompt, source text, state, and prior tool results therefore recur in billed context. - The outer Architect loop also re-adds the same source-bearing user prompt, full entity state, and prior generated relationship summaries (
architect_agent.py:1230-1240,:1262-1318). The state tools themselves return complete remaining/used entity lists (ArchitectAgentGetRemainingEntitiesToProcessTool.py:50-64;ArchitectAgentCheckUsedEntitiesTool.py:53-62), whileCreateRelationshipaccepts one edge per call (ArchitectAgentCreateRelationshipTool.py:177-204). - The cheap arm measures 33.4 Architect calls and 19.8 Janitor calls per session (
ingest-cost-baseline.json:115-127), with 17–56 Architect calls across the five sessions (:137-176). Architect averages about 11.4k tokens per model call (379,505 / 33.4), consistent with context re-send rather than output volume alone. - Batched Janitor still consumes ≈344.8k tokens/session after the cheap grounding precheck (
architect_agent.py:328-440;grounding.py:90-116)—about 47% of total cheap-arm spend.
There is therefore a structured schema, but no pure, side-effect-free, provider-constrained extraction path with tools=[], one response, and a hard repair budget. That missing path—not merely “switching to run_structured”—is the first architectural lever.
One attribution caveat remains: ingest-cost-baseline.json records ingest flags but not the running server checkout or Architect mode, and the workspace/live runtime may differ. The cost numbers are observed; assigning every measured call to the current source path remains an inference until Task 0.2 records runtime code identity.
Token-amplification decomposition
- Call count: 33.4 Architect and 19.8 Janitor model calls/session in the cheap arm; the current source permits multiple inner tool turns inside each of up to four outer attempts per Scout unit.
- Context re-send: each next inner turn invokes the model with accumulated messages, while each outer attempt reconstructs the source-bearing user prompt and entity state; successful prior relationships are summarized back into later attempts.
- Prompt and schema overhead: the single-mode system prompt is substantial (
src/constants/prompts/architect_agent.py:205-269); the custom runtime adds tool descriptions and a flattened output schema (src/core/agents/core/prompts.py:27-63), and native binding attaches all tool schemas (agent_base.py:58-66). Those fixed instructions recur on every model request before source/history tokens. - Tool design: complete entity lists are read back through bookkeeping tools, and relationship creation is edge-granular even though the desired output is a batch. These tools mostly expose in-process state, not new external information that would justify model observation.
- Missing direct path:
ArchitectAgentSingleResponseis a useful contract, but the function namedrun_structured()still binds tools and loops. BrainAPI lacks atools=[], provider-constrained, validate-then-persist path with a one-repair ceiling.
Quality reality (binding)
From 00 / Phase 1: graph hygiene improved; graph channel still does not reliably beat passages on LoCoMo judge accuracy (noise floor ~5–7 pts). Do not claim quality wins from single-run judge deltas. Prefer graph hygiene + retrieval-side metrics + paired McNemar gates.
What this workstream does (as-is)
flowchart TD
text[Input text] --> chunk[chunk_text max 6k]
chunk --> scout[Scout structured entities per chunk]
scout --> merge[_merge_entities]
merge --> archRun[Architect run per unit + raw prior window 4]
archRun --> outer[Single-mode coverage loop max 4]
outer --> inner[Custom AgentBase schema + five tools]
inner --> tools[CreateRelationship + bookkeeping turns]
tools --> defer{defer_janitor?}
defer -->|yes| queue[Queue pending relationships]
defer -->|no| janPerCreate[Atomic Janitor per create]
queue --> batchJan[Batched Janitor + grounding skip]
batchJan --> persist[Caller persists enrichment_relationships]
janPerCreate --> persist
Anchors:
| Step | Location |
|---|---|
| Orchestration | auto_kg.py:120-267 |
| Per-unit Architect text | auto_kg.py:70-77, _PRIOR_UNIT_WINDOW=4 |
| Default single dispatch | architect_agent.py:257-268, :593-633, :1161-1193 |
| Outer structured coverage loop | architect_agent.py:1203-1328 |
| Inner custom tool loop | invoke_loop.py:118-147, :372-502 |
| Create + optional inline Janitor | ArchitectAgentCreateRelationshipTool.py:383-411 |
| Batched Janitor | architect_agent.py:328-440 |
| Grounding skip | grounding.py:90-116 |
| Cost ledger | ingest_cost.py via track_stage |
| Knobs | config.py (JANITOR_BATCH_SIZE, INGEST_ARCHITECT_PER_UNIT, INGEST_DEFER_JANITOR, INGEST_ARCHITECT_MODE, INGEST_ARCHITECT_PRIOR_CONTEXT, INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP, PIPELINE_MODE) |
Guarantee this path tries to provide: every Scout entity is considered for event-centric relationships, and submitted relationships are structurally checked or explicitly deferred to batched Janitor before provenance-stamped persistence.
Where it breaks: reaching the four-iteration outer cap only logs remaining entities and continues (architect_agent.py:1320-1328), so “every entity considered” is not a hard outcome; the inner custom tool loop is not explicitly bounded; repeated state and source context dominate input; Janitor remains a second full LLM agent for most edges.
Guarantees and where they break
Ranked by impact on token multiplier (then quality risk):
- Nested invocation is the cost model. The current “single” path still combines an outer four-pass coverage loop with a generic stateful tool loop. At ~33 Architect calls/session, this is an architectural gap, not a missing temperature or prompt knob.
- The structured path is prompt-enforced, tool-coupled, and retry-prone. The default custom backend injects/recovers JSON rather than guaranteeing one provider-constrained response (
runtime_agent_factory.py:25-43;prompts.py:27-63). A schema parse can coexist with prior tool turns and side effects. - Tools are too stateful and too granular. Relationship creation is one edge per tool invocation; remaining/used queries resend full entity lists; model turns are being spent on bookkeeping that deterministic Python already owns.
- Janitor is still ~47% of cheap-arm tokens. Grounding skip helps, but LLM Janitor remains the normal path rather than the exception path.
- Raw prior context is a quality/cost trade-off. Up to four prior units are repeated for cross-unit coreference (
auto_kg.py:70-77). Compaction could save input tokens but has weaker evidence than bounded extraction and can erase antecedents. - Event hubs and temporal attribution are constraints, not expendable overhead. Replacing Architect with naive binary NER/RE or
lightweightwould abandon the product differentiator. - The benchmark cannot certify small judge changes. The ~5–7pp judge noise floor means cost and deterministic graph/retrieval metrics are primary early gates; paired McNemar is appropriate only for end-to-end binary answer outcomes, and a non-significant result is not proof of equivalence.
Frontier techniques
Retrieval provenance
- Query: cheap high-quality LLM KG construction; structured extraction and constrained decoding; Architect-loop alternatives (ATOM, Distill-SynthKG, Zep/Graphiti, LightRAG, MS GraphRAG, ReWOO, LLMCompiler, JSONSchemaBench, EDC, SafePassage, UCCI, Engram).
- Databases: arXiv (primary, Atom via
arxiv_atom.py), OpenAlex (LazyGraphRAG resolution), Microsoft Research blog (LazyGraphRAG — not on arXiv). - Access date: 2026-07-30.
Sources and applicability
| Paper / source | Claim (as stated) | Evidence quality | Applicability to BrainAPI |
|---|---|---|---|
| ATOM 2510.22590 | Atomic self-contained facts → parallel TKG merge; ~18% exhaustivity, ~33% stability, >90% latency cut vs baselines | Peer-reviewed track (EACL 2026 Findings); empirical TKG metrics; not LoCoMo/LongMemEval QA | High for unitization + dual-time; latency claim is construction latency, not BrainAPI’s ReAct tax. Adapt atomic units; keep event hubs. |
| Distill-SynthKG 2410.16597 | Teacher multi-step SynthKG → single-step student extractor; better KG quality than larger models; helps RAG | Strong methodology for KG+RAG; gains are on their synth/eval sets | Adapt later: single-pass extract now; distill only after teacher (event-hub schema) stabilizes — matches 10 deferral. |
| Zep / Graphiti 2501.13956 | Temporal KG memory; LongMemEval up to +18.5% acc, −90% response latency vs baselines; DMR 94.8% vs MemGPT 93.4% | Vendor paper; LongMemEval gains are retrieval/memory, not an ingest token multiplier study. DMR delta is small. | Adapt prior-context window + uncertainty-gated ER; do not treat +18.5% as evidence that BrainAPI’s ReAct loop is required. |
| LightRAG 2410.05779 | Dual-level graph+vector index; incremental updates; “simple and fast” vs GraphRAG-class systems | Open-source; retrieval-quality claims; ingest cost not rigorously peer-benchmarked against BrainAPI | Adopt pattern: schema-guided entity+relation extract in few LLM calls per chunk, not tool loops. Keep BrainAPI event-hub schema. |
| MS GraphRAG 2404.16130 | LLM extract entity graph + community summaries; strong on global sensemaking | Canonical GraphRAG; indexing is expensive by design (summaries) | Adopt extract stage only (structured triples per chunk). Reject community-summary indexing as BrainAPI’s cost lever — wrong product surface and costly. |
| LazyGraphRAG (MSR blog, 2024; not arXiv) | Index cost ≈ vector RAG (0.1% of full GraphRAG); defers LLM to query; relevance-test budget | Blog + internal eval (AP news, LLM pairwise prefs); not a peer-reviewed preprint with stable arXiv ID | Reject as product architecture: moves LLM to query time — violates 00 context-path budget. Adapt only the idea “defer expensive LLM until necessary” at write time (cascade / skip Janitor), not at retrieve. |
| SafePassage 2510.00276 | Span align + scorer cuts IE hallucinations up to 85%; small encoder can beat LLM scorer | Controlled IE tasks; hallucination flags ≠ LoCoMo accuracy | Adopt/extend: BrainAPI already has grounding.py; raise skip rate so Janitor is exception path. |
| UCCI 2605.18796 | Calibrated cascade: −31% cost at fixed micro-F1 on NER | Strong production NER cascade study; assumptions stated | Adapt for ingest: cheap model / structured pass first; escalate hard units to tooler or larger model. Not a KG-construction paper. |
| HippoRAG 2405.14831 | Write-time KG + PPR; multi-hop QA; 10–30× cheaper than iterative retrieve | Strong retrieval paper; cost win is query, not Architect | Reject for this workstream’s primary lever (already aligned with BrainAPI write-heavy / cheap-read intent). Useful as existence proof that write-time structure pays for cheap read. |
| iText2KG 2409.03284 | Incremental zero-shot KG without heavy post-processing | Scenario evals (papers/sites/CVs), not conversation memory | Adapt incremental entity/relation extract; insufficient alone for event hubs. |
| Engram 2606.09900 | Lean retrieved context beats full history on LongMemEval_S (83.6% vs 73.2%, ~8× fewer answer tokens); async fact extract | Strong LongMemEval methodology; write path uses async SPO extract without ReAct | Adapt “fast write + async structured facts”; supports replacing a stateful nested loop with bounded extraction. |
| Dense X / propositions 2312.06648 | Atomic propositions beat passage units for retrieval | Retrieval granularity study | Adapt as optional decontextualization before extract (pairs with ATOM). |
| ETLCH 2509.08381 | 1B LoRA model for JSON/KG/NER under low resource | Small-model IE; domain-specific | Defer with Distill-SynthKG; useful only after schema freeze. |
| ReWOO 2305.18323 | Separates a tool plan from observations; reports 5× token efficiency and +4% accuracy on HotpotQA | Direct agent-loop ablation, but on QA rather than KG writes | Adapt its no-interleaving principle for the fallback path: emit a batch plan once, execute deterministically, repair only failures. |
| LLMCompiler 2312.04511 | Parallel function planning reports up to 3.7× latency speedup and 6.7× cost savings vs ReAct | Multiple function-calling tasks; headline maxima are not KG results | Reject blind parallel mutation. Architect tools share entity/relationship state. Reuse only batch planning for independent validation calls. |
| JSONSchemaBench 2501.10868 | Evaluates six constrained decoders on 10k real schemas across compliance, coverage, and generation quality | Strong systems benchmark; schema compliance is not factual correctness | Adopt provider-tested constrained output to remove parse/retry turns; keep semantic validators because valid JSON can still hallucinate. |
| EDC 2404.03868 | Open extract → define → post-hoc canonicalize avoids placing a large ontology in every extraction prompt | Three KGC benchmarks; no memory-QA or ingest-token head-to-head | Adapt: constrain response shape and event roles, not a closed predicate vocabulary; canonicalize predicates after extraction. |
| Practical GraphRAG 2507.03226 | Dependency parsing reaches 61.87% vs 65.83% for its LLM extractor (reported as 94% of performance) on two enterprise datasets | Narrow domains and LLM-as-judge downstream evaluation limit transfer | Adapt only as deterministic candidate/grounding evidence. It does not establish that rules alone preserve conversational event hubs. |
Critical assessment — what the literature does not support
- “More agent reflection / more tools ⇒ better memory QA.” HippoRAG and Engram improve QA with structured write + cheap read, not with unbounded write-time ReAct. Folklore that BrainAPI must keep open tool loops for quality is unsupported.
- LazyGraphRAG as an ingest cost solution. Its headline is cheap index + query-time LLM. That inverts BrainAPI’s binding constraint.
- Zep’s LongMemEval +18.5% as Architect-loop justification. That number is end-to-end memory service performance, not an ablation of nested agent loops vs pure batch extraction.
- Distillation as the first lever. Distill-SynthKG itself needs a high-quality teacher;
10correctly defers it. Distilling today’s nested-loop teacher freezes cost and defects. - Community GraphRAG summaries for BrainAPI ingest. They optimize global theme QA; BrainAPI’s bar is event hubs, temporal truth, multi-hop — and summaries burn tokens BrainAPI cannot afford.
- Judge accuracy single-run “wins” after a cost refactor. Noise floor in
00makes that folklore dangerous.
Ranked levers: expected savings vs quality risk
The grades are GRADE-like judgments for this decision, not formal clinical GRADE scores. Published gains are downgraded for indirectness whenever the task is not event-centric conversational KG construction.
1. Pure batched structured extraction — savings: very high; quality risk: medium; confidence: moderate (B)
Mechanism. One Architect call receives the same current unit, Scout entities, and prior context as today, but tools=[] and a provider-supported JSON schema force one batch of event hubs and relationships. Permit at most one stateless syntax repair; deterministic code owns entity bookkeeping and persistence. Constrain structure and event roles, not a closed predicate vocabulary, following EDC (2404.03868).
Evidence and cost. LightRAG (2410.05779) and MS GraphRAG (2404.16130) demonstrate bounded structured extraction as a viable index shape; JSONSchemaBench (2501.10868) shows constrained decoders can guarantee schema compliance but not semantic truth. No paper establishes BrainAPI quality parity, so local A/B is mandatory. Implementation costs one new extraction contract and path; the reward is removing most of 33.4 Architect turns/session.
Verdict: adopt first. This is the only lever large enough to approach 20–100× without removing event hubs.
2. Decouple planning from tools and hard-bound repair — savings: high; quality risk: low–medium; confidence: moderate-low (B−)
Mechanism. If a fallback still needs tools, ask once for a complete relationship plan, execute it deterministically, then return only failed items for one repair. Never feed successful tool observations and full state back through another reasoning turn.
Evidence and cost. ReWOO (2305.18323) reports 5× token efficiency and +4% HotpotQA accuracy by separating reasoning from observations. LLMCompiler (2312.04511) reports maxima of 3.7× latency and 6.7× cost savings through planned parallel calls. Both are indirect QA/tool evidence; BrainAPI's mutation tools are not freely parallelizable.
Verdict: adapt for fallback. Use batch planning and explicit call caps; reject concurrent graph mutations.
3. Make LLM Janitor exception-only — savings: very high; quality risk: medium; confidence: moderate (B)
Mechanism. Require source spans and run deterministic endpoint, schema, temporal, duplicate, and span-alignment checks. Auto-accept high-confidence grounded edges, reject impossible shapes, and send only ambiguous residuals to a structured Janitor with an explicit veto outcome.
Evidence and cost. SafePassage (2510.00276) reports up to 85% fewer IE hallucinations and shows a small scorer can outperform an LLM scorer on its tasks. BrainAPI already has cheap_janitor_precheck; Janitor's measured 344.8k tokens/session makes this locally high leverage. The external evidence does not cover inferred multi-sentence event facts, so false rejection must be sampled.
Verdict: adopt after the pure extractor. Target a measured skip rate, not a presumed one.
4. Replace raw prior replay with bounded relevant state — savings: medium; quality risk: medium–high; confidence: low (C)
Mechanism. Keep the current unit verbatim but replace four raw prior units with a token-capped scratchpad of active entities, event hubs, dates, and unresolved references; retrieve a raw prior span only when a coreference check fails.
Evidence and cost. Graphiti/Zep (2501.13956) supports a bounded prior-message window, not this particular compression. ATOM (2510.22590) and Dense X (2312.06648) support self-contained atomic units, but a rewrite can invent antecedents. The change should be isolated from the loop A/B so its effect is identifiable.
Verdict: adapt only after Checkpoint 1. Keep raw context in the first extractor comparison.
5. Calibrated cheap-model cascade — savings: medium on residual; quality risk: medium; confidence: low–moderate (C+)
Mechanism. Run the batch schema on a cheaper model; escalate only schema failures, low grounding coverage, disconnected event hubs, or calibrated uncertainty. Select thresholds on held-out sessions rather than the five-session cost set.
Evidence and cost. UCCI (2605.18796) reports 31% lower cost at fixed micro-F1 for production NER. That is strong cascade evidence but indirect for event relationships. A second model, calibration set, and drift monitoring add operational cost.
Verdict: defer until the single-model path passes quality gates.
6. Distill the stable extractor — savings: potentially very high; quality risk: high if premature; confidence: low for BrainAPI (C)
Mechanism. Use accepted event-hub outputs from a stable teacher to train a small one-step extractor, as Distill-SynthKG (2410.16597) compresses a multi-stage teacher into one generation.
Evidence and cost. Distill-SynthKG reports better KG quality than baselines up to eight times larger and downstream retrieval/QA gains, but on its own datasets. Dataset generation, training, evaluation, and drift maintenance are substantial, and the current teacher still has known defects.
Verdict: defer. Revisit only after two stable checkpoints and maintainer approval.
Recommended architecture
Ship defaults (accurate): INGEST_ARCHITECT_MODE=batch, INGEST_ARCHITECT_PRIOR_CONTEXT=auto (→ scratchpad); rollback with tooler / raw.
Design decisions
- Add a pure schema-guided batch extractor; do not reuse
run_structured()unchanged. The new happy path has no tools, no entity-state loop, one response, and at most one stateless repair. It emits event hubs, typed endpoints, open-vocabulary predicates,source_span,happened_at, and validity fields. - Keep the current context unchanged for the first A/B. Isolate the control-flow effect before testing prior-context compression.
- Janitor becomes exception-path: expand cheap grounding + schema validators; LLM Janitor only on
needset; target skip ≥70% of relationships on conv-26 A/B without hygiene regression. - Batch writes stay; prefer emitting many relationships per structured call (LightRAG/MS extract pattern) over one tool call per edge.
- Current single/tooler paths become hard-budget escalation options for schema, grounding, or event-coverage failures; every escalation records a reason and cannot recurse without a cap.
- Only after the loop A/B, test prior context separately: replace raw prior-unit replay with a compact entity/event scratchpad plus raw-span fallback.
- No query-time LLM for efficiency wins. No
lightweightmode. No community-summary GraphRAG index as a cost strategy. - Cascade models are optional: use a cheaper model only after single-model extraction passes and escalation thresholds are calibrated on held-out units.
As-is vs to-be
flowchart LR
subgraph asis [As-is accurate]
S1[Scout structured] --> A1[Architect outer coverage + inner tool loop]
A1 --> J1[Janitor LLM frequent]
end
subgraph tobe [To-be accurate]
S2[Scout structured] --> E2[Pure batch extract 1 call + optional repair]
E2 --> G2[Deterministic validate + grounding]
G2 -->|pass| W2[Batch write]
G2 -->|fail| X2[Hard-budget Architect or Janitor escalation]
X2 --> W2
end
Mapping to existing code
| Piece | Reuse / change |
|---|---|
| Scout | Keep (scout_agent.py) |
run_structured |
Reuse response models/persistence helpers only; do not promote its nested tool/coverage loop unchanged |
New run_batch_extract |
tools=[], constrained response, one repair maximum, no side effects before validation |
| Existing single/tooler paths | Gate behind INGEST_ARCHITECT_MODE=tooler|schema|batch (current alias → tooler) and an explicit escalation budget |
grounding.py |
Tighten + require source_span in schema |
run_batched_janitor |
Keep for escalate / residual |
ingest_cost.py / invoke_loop.py |
Add input/output/cached tokens, calls per unit, payload size, multiplier, skip/escalation rate |
What “competitive with LightRAG/Graphiti” would mean
No cited paper exposes a directly comparable ingest-token denominator, model mix, or event-graph output. BrainAPI should not claim cost parity from architecture diagrams. A defensible claim requires:
- Peer-shaped control flow: ≤2 Architect calls on a happy-path Scout unit, no per-edge LLM bookkeeping, bounded prior context, and Janitor as an exception.
- Measured budget: accurate-mode total ingest ≤100× source tokens on the frozen five-session set, with 20–50× as the stretch band. The denominator must name its tokenizer.
- Controlled comparison: if “cheaper than LightRAG/Graphiti” is claimed publicly, run the same corpus and model/tokenizer through pinned peer versions and report tokens, wall time, graph size, and extraction coverage. Until then, say “within the target band,” not “beats peers.”
- BrainAPI differentiation retained: zero type-named placeholders; no deprecated event legs; grounded event hubs with actor/object leg completeness; temporal fields and provenance preserved; stable graph evidence-session recall and no detected paired LoCoMo/LongMemEval regression.
- Cheap retrieval preserved: all extra intelligence remains at write time;
/retrieve/contextgets no new LLM loop.
Implementation plan
Phase 0 — Make amplification attributable
Task 0.1: Architect payload and token telemetry
Description: Extend the ledger at the model-call boundary to record input, output, cached-input (when exposed), total tokens, call ordinal, unit id, stage, and loop origin (architect_outer, agent_tool, schema_repair, janitor). Record source characters plus a source-token denominator with an explicit tokenizer id; if only an estimate is available, label it as estimated.
Acceptance criteria:
- A two-call fixture reconciles per-call and stage totals and distinguishes input from output.
- Reports expose calls/unit, tokens/call, source tokenizer, multiplier, Janitor skip rate, and escalation rate without breaking providers that omit cached-token fields.
- Architect, Janitor, and Scout totals reconcile to total ingest LLM usage within provider rounding.
Verification:
-
pytest tests/test_ingest_cost.py -v -
cd benchmarks && ./locomo.sh ingest --sample conv-26 --limit-sessions 1 --brain architectmeasurecp0 --run architect-measure-cp0 --concurrency 1 --no-resume && ./locomo.sh report --run architect-measure-cp0 --json
Dependencies: None
Files likely touched: src/core/saving/ingest_cost.py, src/core/agents/core/invoke_loop.py, benchmarks/locomo/ingest.py, benchmarks/locomo/report.py, tests/test_ingest_cost.py
Size: M
Task 0.2: Reproducible cost and graph-hygiene A/B command
Description: Add one benchmark command that creates isolated brains, ingests fixed conv-26 sessions 1–5 for two Architect modes, and writes paired per-session cost/latency plus graph hygiene. Record workspace SHA, dirty state, running server code identity, models, tokenizer, prompts, environment flags, and brain ids.
Acceptance criteria:
- The artifact includes source multiplier, input/output tokens, calls/unit, tokens/call, wait time, type-named nodes, event/node/relationship counts, EVENT-touching deprecated edges, event degree/leg completeness, grounded-span rate, hub bridges, errors, and escalation/skip reasons.
- Fresh brain ids and a one-owner rule prevent residue or concurrent ingestion; failed/partial units remain visible.
- Cost and hygiene use paired per-session deltas. No McNemar test is emitted because these are not paired binary answer outcomes.
Verification:
-
cd benchmarks && ./locomo.sh ingest-cost-ab --sample conv-26 --limit-sessions 5 --baseline-mode current --candidate-mode current --tag architect-cp0
Dependencies: 0.1
Files likely touched: benchmarks/locomo/cli.py, one benchmark helper, one test
Size: M
Checkpoint 0
- Same-mode A/A completes and records the actual running code.
- The current arm confirms or revises O(10³)× and the >95% Architect+Janitor spend share.
- Per-call telemetry can distinguish context re-send from generated output.
- If these observations do not reproduce, revisit the diagnosis before changing architecture.
Phase 1 — Remove the nested-loop tax
Task 1.1: Pure event-hub extraction contract
Description: Define a batch response that reuses Scout UUIDs, may declare missing event entities, and emits relationships with open-vocabulary predicates, exact source-span offsets/text, amount/polarity, happened_at, and validity metadata. Add a batch validator for type-named placeholders, endpoint existence, event actor/content leg completeness, and span bounds. Do not encode the entire predicate ontology in the prompt.
Acceptance criteria:
- Open predicates are accepted while malformed endpoints, out-of-range spans, and type==name placeholders are rejected with itemized reasons.
- Fixtures cover a dated event hub, amount on a relationship, cross-sentence evidence, and reuse of existing Scout UUIDs.
- Validation is side-effect-free and runs before any relationship is queued.
Verification:
-
pytest tests/test_architect_batch_schema.py -v
Dependencies: None
Files likely touched: one schema/validator module and tests/test_architect_batch_schema.py
Size: S
Task 1.2: Tool-free run_batch_extract
Description: Add a feature-flagged Architect path with tools=[], provider-supported constrained output when available, one primary call, and at most one stateless repair of invalid items. Feed it the unchanged current _architect_unit_text and Scout entities so this phase changes control flow only. Reuse persistence helpers after validation.
Acceptance criteria:
- Valid output uses exactly one Architect model call/unit; malformed output uses exactly two; no third call or tool invocation is possible.
- No graph/pending-relationship side effect occurs before the whole batch is validated.
-
INGEST_ARCHITECT_MODE=current|batchselects the arm;currentremains the default until Checkpoint 2.
Verification:
-
pytest tests/test_architect_batch_extract.py -v -
cd benchmarks && INGEST_ARCHITECT_MODE=batch ./locomo.sh ingest --sample conv-26 --limit-sessions 1 --brain architectbatchsmoke --run architect-batch-smoke --concurrency 1 --no-resume
Dependencies: 1.1
Files likely touched: src/core/agents/architect_agent.py, src/core/saving/auto_kg.py, src/config.py, Architect prompts, one test
Size: M
Task 1.3: Controlled five-session loop A/B
Description: Use Task 0.2 to compare current vs batch with identical model, raw prior context, Janitor behavior, session order, and retrieval-independent graph audit. Perform item-level error analysis on dropped/disconnected entities rather than using graph size alone as quality.
Acceptance criteria:
- Happy-path Architect calls/unit are ≤2 and Architect input tokens fall ≥80% relative to the current arm, or the failed target has a payload-level explanation.
- Type-named nodes and deprecated EVENT-touching edges remain zero; grounded-span rate and event actor/content leg completeness do not fall by more than 5pp.
- Event/node/relationship counts are reported as diagnostics, not treated as correctness by themselves.
Verification:
-
cd benchmarks && ./locomo.sh ingest-cost-ab --sample conv-26 --limit-sessions 5 --baseline-mode current --candidate-mode batch --tag architect-cp1
Dependencies: 0.2, 1.2 Files likely touched: benchmark artifact only, plus a benchmark fix if the command cannot explain failures Size: S
Checkpoint 1
- Nested-loop removal pass on re-run A/B (
architect-cp1b-*, sessions 1–2, fresh*12bbrains): Architect input −92.4% / total −86.9% vs tooler; escalate 0% (was 100% onschema_partial_reject); happy-path 2.0 calls/unit both batch sessions (schema+repair, no tooler escalate). Prior partial result + fix notes:architect-cp1-NOTES.md. - Graph hygiene: type-named 0 / deprecated 0 both arms; batch denser on nodes/events (29/24 vs 22/18) with source_span on 27/27 rels. Soft debt: batch EVENT descriptions 0/24 (worker logged
event_leg_incomplete+ manyspan_offsets_mismatch_textdrops without escalate) — addressed in Phase 2; post-measure (architect-cp2-batch, sessions 1–5): EVENT desc 60/60, source_span 112/112. - Total multiplier: tooler 299× vs batch 37× (≤100× gate pass; Janitor skipped 27 batch units — CP1b 100% skip under-validated; Phase 2 requires explicit evidence). Details:
benchmarks/runs/architect-cp1b-NOTES.md. - Maintainer reviews extraction diffs before Janitor behavior changes (esp. missing EVENT descriptions / span rejects) — Phase 2 ships deterministic fill/realign + triage; live re-measure in
architect-cp2-NOTES.md. - Re-run Checkpoint 1 A/B after
schema_partial_rejectfix (sessions 1–2) — gates pass. Optional: confirm on sessions 1–5 before Phase 2 claims.
Phase 2 — Make Janitor and escalation exceptional
Task 2.1: Deterministic validation and grounding decisions
Description: Extend cheap_janitor_precheck to classify each batch relationship as accept, reject, or ambiguous using endpoint shape, source-span alignment, event-leg completeness, temporal format, duplicate key, and placeholder checks. Persist the decision, score, and reason. Hand-label a fixed 50-edge sample including multi-sentence inferences before choosing thresholds.
Acceptance criteria:
- Every edge has exactly one decision and machine-readable reason; accepted edges retain source span and grounding score. (
GroundingDecision+triage_relationships_for_janitor; ledgerjanitor_drop_reasons.) - Auto-accept precision is ≥95% on the fixed reviewed sample; ambiguous inferred facts are not silently rejected. (needs 50-edge hand-label — not run this turn)
- Janitor skip rate reaches ≥70% on the five-session arm or the artifact lists the dominant blockers and reviewed examples. (CP2 batch: 114/114 skip with 100% real
source_span+ EVENT desc 60/60 — not CP1b synthetic spans; tooler 1–2 shows triage live: rej/amb 5/44. Details:architect-cp2-NOTES.md)
Verification:
-
pytest tests/test_grounding.py tests/test_architect_grounding_decisions.py -v - Manual CP2 re-measure (harness
ingest-cost-abstill absent):architect-cp2-batchsessions 1–5 + optionalarchitect-cp2-tooler1–2 — seebenchmarks/runs/architect-cp2-NOTES.md.
Dependencies: Checkpoint 1
Files likely touched: src/core/saving/grounding.py, Architect orchestration, ledger/report, one test
Size: M
Shipped (2026-07-30):
- Span realign via
find_span_offsets/realign_span_offsets; batch validator repairs or clears bad offsets instead of hardspan_offsets_mismatch_textrejects when the quote is grounded. - EVENT descriptions required in batch prompts; validator fills from Scout → rel description → source_span; rejects empty new EVENT nodes.
- Explicit-evidence-only auto-accept (no synthetic
tail+predicate+tipgrounding) so Janitor skip≠100% from under-validation. - Ledger:
janitor_rejected,janitor_ambiguous,janitor_drop_reasons. - Post-measure (2026-07-30):
architect-cp2-batch1–5 → mult 18.8×, EVENT desc 60/60, source_span 100%, escalate 0%; optional tooler 1–2 → 337× / arch input −96.7%. Notes:benchmarks/runs/architect-cp2-NOTES.md.
Task 2.2: Structured residual Janitor with explicit veto
Description: Send only ambiguous edges to a bounded structured Janitor batch. Replace "OK"/None ambiguity with explicit accept/fix/reject/error outcomes; a parse or provider failure marks the ingest partial/degraded rather than approving edges. Ensure a rejected original is not queued alongside its replacement.
Acceptance criteria:
- Parse failure, approval, correction, and veto are distinct outcomes covered by tests. (
statusOK/ERROR/REJECT; parseNonedrops; wrong+fixed handled) - A vetoed edge is absent and only its accepted replacement may persist.
- Janitor uses one call/batch plus at most one repair and falls below 30% of total ingest LLM tokens on the five-session A/B. (CP2 batch: Janitor 0% of LLM tokens — all units explicit-evidence skip)
Verification:
-
pytest tests/test_janitor_outcomes.py -v(veto covered;test_janitor_veto.pynot separate) - Re-measure:
architect-cp2-NOTES.md(batch Janitor token share 0%; tooler arm exercises LLM triage).
Dependencies: 2.1
Files likely touched: src/core/agents/janitor_agent.py, src/core/agents/architect_agent.py, ArchitectAgentCreateRelationshipTool.py, tests
Size: M
Task 2.3: Hard-budget escalation
Description: For invalid or under-covered batch outputs, attempt one item-level batch repair, then optionally invoke the existing Architect path within a total per-unit model-call budget chosen by the maintainer. Record trigger, attempted path, accepted result, and budget exhaustion; never continue silently with unprocessed entities.
Acceptance criteria:
- Happy-path units never escalate; each escalation has one enumerated trigger.
- No unit exceeds the configured call budget; exhaustion yields an explicit partial/degraded status.
- Forced fixtures cover invalid schema, missing event leg, ungrounded relationship, and provider failure.
Verification:
-
pytest tests/test_architect_escalation.py -v - Task 0.2 artifact reports escalation rate, reasons, and max calls/unit.
Dependencies: 1.2, 2.1
Files likely touched: src/core/saving/auto_kg.py, src/core/agents/architect_agent.py, src/config.py, one test
Size: M
Checkpoint 2
- Accurate-mode total ingest is ≤100× source tokens on the fixed five sessions; ≤50× is stretch. If it remains >150×, stop and attribute the residual before adding another model. Pass — 18.8× overall on
architect-cp2-batch(sessions 1–5; max session 30.9×). - Architect happy path is ≤2 calls/unit; Janitor is <30% of ingest LLM tokens; escalation rate and failure reasons are understood. Pass — calls/unit 1.0–2.0; Janitor 0% tokens; escalate 0%.
- Zero type-named nodes and deprecated EVENT legs; grounded-span and event-leg completeness gates pass. Pass type-named 0; EVENT desc 60/60; source_span 112/112; soft: deprecated rels=3 (inspect later).
- Only now may
batchbecome the accurate-mode default, subject to the maintainer's answer. Shipped: product defaultsINGEST_ARCHITECT_MODE=batch+INGEST_ARCHITECT_PRIOR_CONTEXT=auto(code,.env.example, live.env); rollback via env.
Phase 3 — Optional residual levers, each isolated
Task 3.1: Bounded prior-state A/B
Description: Replace four raw prior units with a ≤500-reference-token entity/event/date scratchpad and retrieve one raw supporting span only when a coreference validator fails. Compare against batch mode with the original raw window; do not combine this test with a model change.
Acceptance criteria:
- Prior context obeys its token cap and reports raw-span fallback frequency. (token cap enforced in
serialize_scratchpad;fetch_prior_spanhelper shipped for on-demand use; live fallback-frequency A/B still pending) - Cross-unit pronoun, alias, and delayed-date fixtures preserve node/event identity. (unit fixtures cover entity/hub/predicate scratchpad + no full prior-body replay; live graph identity A/B still pending)
- Architect input tokens fall while grounded-span and event-leg completeness stay within the Checkpoint-2 tolerance. (2026-07-30 measure: scratchpad vs raw on conv-26 s1–5 — 15.3× vs 23.6×; arch in 28.9k vs 44.5k (−35%); type-named 0; EVENT desc 65/65; graph 75/85/65 vs CP2 81/112/60; see
benchmarks/runs/architect-cp3-NOTES.md)
Verification:
-
pytest tests/test_architect_prior_context.py -v - Manual A/B:
architect-cp3-rawvsarchitect-cp3-scratchpad(batch, sessions 1–5); harnessingest-cost-abtag still optional
Dependencies: Checkpoint 2
Files likely touched: src/core/saving/auto_kg.py, one context helper, src/config.py, one test
Size: M
Shipped (2026-07-30):
architect_scratchpad.py: build/serialize ≤500-token prior scratchpad (entities, open event hubs, dates, recent predicate + span pointers);fetch_prior_spanfor optional on-demand raw window.auto_kg.py: batch/schema default (INGEST_ARCHITECT_PRIOR_CONTEXT=auto) uses scratchpad instead of raw prior-unit replay;rawkeeps CP2 A/B behavior.- Env:
INGEST_ARCHITECT_PRIOR_CONTEXT=auto|scratchpad|raw,INGEST_ARCHITECT_SCRATCHPAD_TOKEN_CAP=500. - Measured 2026-07-30: scratchpad wins — 15.3× vs raw 23.6× (−35% tokens), type-named 0, EVENT desc 100%, graph near CP2; keep scratchpad/
autodefault. Notes:benchmarks/runs/architect-cp3-NOTES.md.
Task 3.2: Calibrated model cascade
Description: Sweep a cheaper extraction model on held-out conv-26 sessions 6–8 and escalate on pre-declared validation/grounding features. Do not tune thresholds on sessions 1–5 or end-to-end judge labels.
Acceptance criteria:
- The sweep artifact reports cost, auto-accept precision, event-leg completeness, and escalation rate for every threshold.
- A selected threshold is frozen before the five-session confirmation.
- If no threshold lowers total cost at fixed gates, the cascade is rejected.
Verification:
-
pytest tests/test_architect_model_routing.py -v -
cd benchmarks && ./locomo.sh architect-cascade-sweep --sample conv-26 --sessions 6,7,8 --out runs/architect-cascade-sweep.json
Dependencies: Checkpoint 2 and maintainer approval for a second model Files likely touched: model routing/config, Architect orchestration, benchmark sweep, one test Size: M
Task 3.3: Distillation go/no-go
Description: After at least two weeks of stable accepted batch outputs, estimate dataset size, label noise, training/serving cost, and expected break-even for a Distill-SynthKG-style student. This task writes a decision note only; any training implementation must be decomposed separately.
Acceptance criteria:
- The note quantifies teacher stability, train/held-out split, event-hub metrics, serving break-even, and rollback.
- No training begins without a maintainer go decision.
Verification:
- Review the decision note against Checkpoints 1–2 artifacts.
Dependencies: Checkpoint 2, maintainer approval Files likely touched: one research decision document Size: S
Checkpoint 3
- Every optional lever has its own A/B; no bundled attribution. (3.1 scratchpad vs raw measured isolated 2026-07-30)
- Total remains in the 20–100× target band with graph gates held. (scratchpad 15.3×; raw 23.6×; both ≤100× with type-named 0)
Phase 4 — Quality gates (before claiming “no trade-off”)
Task 4.1: Full-conversation graph audit
Description: Ingest all 19 conv-26 sessions into fresh current and batch brains, then compare type-named nodes, EVENT-touching deprecations, grounded-span rate, event actor/content leg completeness, degree distribution, event/node/relationship counts, hub bridges, temporal-field coverage, and item errors.
Acceptance criteria:
- Type-named nodes and deprecated EVENT-touching edges are zero in the candidate.
- Grounded-span rate and event-leg completeness stay within the predeclared 5pp tolerance; all count/density changes are reported with sampled explanations.
- Runtime code identity, models, flags, source denominator, and fresh brain ids are captured.
Verification:
-
cd benchmarks && INGEST_ARCHITECT_MODE=current ./locomo.sh ingest --sample conv-26 --limit-sessions 19 --brain architectcurrentfull --run architect-current-full --concurrency 2 --no-resume -
cd benchmarks && INGEST_ARCHITECT_MODE=batch ./locomo.sh ingest --sample conv-26 --limit-sessions 19 --brain architectbatchfull --run architect-batch-full --concurrency 2 --no-resume -
cd benchmarks && ./locomo.sh graph-audit --baseline-brain architectcurrentfull --candidate-brain architectbatchfull --json
Dependencies: Checkpoint 2 Files likely touched: benchmark command/report only if Task 0.2 did not already cover full brains Size: S
Task 4.2: Paired LoCoMo evaluation on clean brains
Description: Evaluate identical retrieval/answer settings on the two full brains, preferably twice per arm to expose judge variability. Report paired exact McNemar and flip counts for binary correctness, a paired accuracy-difference interval, per-category point estimates, passage evidence-session recall, and graph evidence-session recall only if identical-config graph session-set agreement remains ≥95%.
Acceptance criteria:
- Candidate judge accuracy is not more than the predeclared 5pp guard below current, and exact McNemar does not detect a regression; a non-significant result is described as “no detected regression,” not equivalence.
- Passage EvR does not regress beyond tolerance; graph EvR is interpreted only after its stability gate; multi-hop and temporal flips are listed.
- Both replicate manifests record server identity, prompts, models, ingest flags, and brain ids;
prompt-auditpasses.
Verification:
-
cd benchmarks && ./locomo.sh evaluate --sample conv-26 --brain architectcurrentfull --run architect-current-eval-a --concurrency 2 --use-ppr - Repeat as
architect-current-eval-b,architect-batch-eval-a, andarchitect-batch-eval-b, then run./locomo.sh compare --baseline architect-current-eval-a --baseline architect-current-eval-b --candidate architect-batch-eval-a --candidate architect-batch-eval-b --json -
cd benchmarks && ./locomo.sh prompt-audit
Dependencies: 4.1 Files likely touched: benchmark artifacts; comparison code only if the paired-difference interval is absent Size: M
Task 4.3: LongMemEval smoke (product profile)
Description: Add an explicit brain-prefix/namespace option so the two modes cannot share LongMemEval question brains, then ingest and evaluate the same fixed 50-question LongMemEval_S slice under the product profile as a catastrophic-regression smoke. It is not powered for a leaderboard or a quality-win claim.
Acceptance criteria:
- Current and batch runs use the same selected question ids, models, retrieval profile, and fresh brains.
- Cost, ingestion errors, overall/type-level accuracy, and retrieval metrics are reported; any >5pp point drop is investigated.
- No SOTA or equivalence claim is made from the smoke; full claims follow
09.
Verification:
-
cd benchmarks && INGEST_ARCHITECT_MODE=current ./longmemeval.sh ingest --variant s --limit 50 --brain-prefix architectcurrent --run architect-lme-current --concurrency 2 --no-resume -
cd benchmarks && INGEST_ARCHITECT_MODE=batch ./longmemeval.sh ingest --variant s --limit 50 --brain-prefix architectbatch --run architect-lme-batch --concurrency 2 --no-resume - Evaluate each namespace with the same
--brain-prefix, then run./longmemeval.sh report --run <run-id> --json.
Dependencies: 4.2 preferred Files likely touched: LongMemEval CLI, dataset brain-id helper, ingest/evaluate wiring, one test, benchmark artifacts Size: M
Checkpoint 4 (ship gate)
- Multiplier ≤100× on conv-26 5-session cost A/B
- Full graph audit green, including grounded event-leg coverage
- LoCoMo paired guard passes with uncertainty and flip counts reported
- Retrieval metrics gate recorded (passage EvR; graph EvR only if stable)
- LongMemEval smoke has no unexplained catastrophic regression
Success metrics
| Metric | Baseline (cheap2) | Target | Gate |
|---|---|---|---|
| Mean LLM tokens / session (5-session set) | 728k | Report absolute; gate on tokenizer-named multiplier | Checkpoint 2/3 |
| Token multiplier | ~886× cheap → 15.3× batch+scratchpad | ≤100× (stretch 20–50×) — met | Cost A/B JSON + CP2/CP3 NOTES |
| Architect calls | 33.4/session | ≤1 happy-path call/unit; ≤2 with repair | Checkpoint 1 |
| Architect total tokens | 379.5k/session (input/output split unavailable) | ≥80% reduction at fixed context | Checkpoint 1 |
| Janitor share of LLM tokens | ~47% | <30% (stretch <15%) | Checkpoint 2 |
| type-named placeholders | 0 | 0 | Hygiene |
| Deprecated EVENT-touching edges | Not in baseline JSON | 0 | Hygiene |
| Grounded spans / event-leg completeness | Not yet recorded | Within 5pp of current and manually audited | Checkpoints 1/4 |
| LoCoMo judge | No paired ingest-path baseline | >−5pp guard and no McNemar-detected regression; not an equivalence claim | Checkpoint 4 |
| Passage EvR | No paired ingest-path baseline | Within predeclared tolerance | Checkpoint 4 |
| Graph EvR | — | Only if agreement ≥95% | 00 |
Open questions for the maintainer
- Which pinned tokenizer should define the source-token multiplier denominator across providers?
- RESOLVED: Pin
tiktokencl100k_baseas the cross-provider source-token denominator (SOURCE_TOKENIZER_ID=tiktoken/cl100k_base). Recorded on the ingest cost ledger; if tiktoken is unavailable at runtime, fall back tolen(text)//4and marksource_tokens_estimated=true. Do not switch the pin per LLM provider.
- RESOLVED: Pin
- Is ≤100× the hard accurate-mode ship threshold, with 20–50× as stretch, or should a different budget be used?
- RESOLVED: Ship gate is ≤100× source tokens first; then attempt ≤20–50×. Always keep the better spend/quality version — do not ship a cheaper arm that fails hygiene/quality gates.
- May
INGEST_ARCHITECT_MODE=batchbecome the accurate default after Checkpoint 2, or only after the full Checkpoint-4 evaluation?- RESOLVED: After CP2/CP3 cost + hygiene pass, product defaults are
INGEST_ARCHITECT_MODE=batchandINGEST_ARCHITECT_PRIOR_CONTEXT=auto(batch/schema → scratchpad). Rollback: setINGEST_ARCHITECT_MODE=tooler(and optionallyPRIOR_CONTEXT=raw). Full Checkpoint-4 quality gate can wait.
- RESOLVED: After CP2/CP3 cost + hygiene pass, product defaults are
- What is the maximum total Architect model-call budget per failed unit, including repair and escalation?
- RESOLVED: Schema/batch happy path: ≤3 LLM calls/unit (1 primary + ≤1 repair; third reserved for accounting/edge). Escalation via tooler: hard cap 10 tool turns / failed unit.
- Should escalation try the current single path,
run_tooler, or a larger-model batch extraction first?- RESOLVED: Escalate order: schema extract fail → single repair extract → tooler ReAct last. No larger-model cascade in the first vertical slice.
- May prior raw text be replaced entirely by a scratchpad plus on-demand span, or must one previous raw unit always remain?
- RESOLVED: Yes — replace prior raw text replay with compact scratchpad + on-demand span (Phase 3.1, after Checkpoint 1 isolates the loop A/B with raw context unchanged).
- Is a second cheaper extraction model in scope for Phase 3, or must the work remain single-model until distillation?
- RESOLVED: Cascade / cheaper second model is OUT of scope for the first vertical slice. Same model, fewer calls. Revisit only after Checkpoint 2.
- May structured Janitor veto an edge outright, with an audit record, rather than only fix or approve it?
- RESOLVED: Yes — structured Janitor may veto edges with an audit record. Prefer deterministic reject first; LLM veto only if needed (Phase 2).
- Is a full two-replicate
conv-26paired evaluation required before defaulting batch mode, despite the benchmark's limited power?- RESOLVED: No full two-replicate conv-26. Use small batch while testing (e.g. conv-26 sessions 1–2 or 1–3 max). Do not start long live A/B until code + unit tests are ready.
- Is a pinned same-corpus LightRAG/Graphiti ingest run required before any public “peer-competitive cost” claim?
- RESOLVED: Do not run LightRAG/Graphiti ingest. Only cite online/published claims already in this plan; no new pinned peer runs for this workstream.
Risks
| Risk | Impact | Detection |
|---|---|---|
| Batch extraction under-produces event/multi-hop structure | High — cheap but loses the differentiator | Grounded event-leg completeness, item-level diffs, stable graph EvR, multi-hop/temporal flips |
| Valid JSON is mistaken for a true fact | High | Schema compliance and grounding are separate gates; 50-edge reviewed sample |
| Provider lacks reliable constrained output | Medium — repair turns return | Provider contract tests, repair count telemetry, keep current path behind the flag |
| Escalation approaches 100% | High — no cost win | Escalation rate/reasons; Checkpoint 2 fails |
| Grounding auto-accepts fluent hallucinations or rejects inference | High | Auto-accept precision target plus ambiguous bucket and reviewed multi-sentence examples |
| Context compaction is bundled with loop replacement | High — causal attribution lost | Phase order requires fixed raw context at Checkpoint 1 |
| Scratchpad breaks coreference | Medium | Pronoun/alias/date fixtures and raw-span fallback rate |
| Cost A/B uses dirty or different runtime code | High | Fresh brain ids and running-server identity in every artifact |
| Approximate source tokens become a precise claim | Medium | Tokenizer id/estimate flag required in ledger and reports |
| Non-significant McNemar is called equivalence | High | 5pp guard, paired interval, flip counts, explicit “no detected regression” language |
| Distillation freezes current defects | Medium | Phase 3.3 is decision-only and depends on stable outputs |
| “Peer-competitive” becomes “beats peers” without a head-to-head | Medium | Controlled peer run required for comparative public claims |
What not to do
- Do not use
PIPELINE_MODE=lightweightto hit the multiplier target. - Do not add reflection / critique agent passes for cost goals.
- Do not move LLM graph construction to
/retrieve/context(LazyGraphRAG pattern). - Do not adopt full MS GraphRAG community summarization as BrainAPI’s index.
- Do not distill the current nested-loop teacher.
- Do not ship on a single-run LoCoMo point estimate.
Related docs
- Constraints:
00-scope-and-constraints.md - Prior cost levers (C0–C3 shipped):
10-ingest-cost-latency.md - Extraction quality:
01-ingestion-extraction.md - Roadmap pointer:
06-roadmap.md - Dual goal (≥93% LoCoMo + low multiplier):
12-locomo-93-at-low-multiplier.md— keep batch+scratchpad; fix escalate storms; accuracy via SOTA harness / answerer / retrieval, not more Architect loops
Literature provenance appendix
## Retrieval Summary
- Query: cheap high-quality KG construction for BrainAPI Architect efficiency
- Scope: targeted lookup of named systems + related cascade/grounding papers
- Databases queried: arXiv (`user-arxiv` search + abstract verification and Atom id lookup), OpenAlex (LazyGraphRAG), MSR blog (LazyGraphRAG)
- Access date: 2026-07-30
## Provenance
- arXiv: https://export.arxiv.org/api/query with id_list /
2510.22590,2410.16597,2501.13956,2410.05779,2404.16130,2510.00276,
2605.18796,2405.14831,2409.03284,2606.09900,2312.06648,2509.08381,
2305.18323,2312.04511,2501.10868,2404.03868,2507.03226
- Parsed with paper-lookup scripts/arxiv_atom.py
- LazyGraphRAG: not found on arXiv (totalResults 0); OpenAlex hits are secondary;
primary description from https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/
- Warnings: the 20–100× peer band is a requested engineering target inferred from
bounded extractor designs, not a published head-to-head result against BrainAPI