Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

docs(adr): ADR-0006 — scope focused-iteration changes, reject final_max_results truncation (#4873)

LearningCircuitb69b14a20abb
diff --git a/docs/decisions/0006-focused-iteration-scope-and-final-max-results.md b/docs/decisions/0006-focused-iteration-scope-and-final-max-results.md
new file mode 100644
index 0000000000..a1defcfc7a
--- /dev/null
+++ b/docs/decisions/0006-focused-iteration-scope-and-final-max-results.md
@@ -0,0 +1,336 @@
+# ADR-0006: Keep `focused-iteration` minimal — accept the state-leak/citation fixes, defer the `final_max_results` truncation
+
+**Date:** 2026-06-28 (updated 2026-07-03)
+**Status:** Accepted
+**Relates to:** PR #4850 (merged 2026-06-28), issue #4851, #2001 (closed), #2067 (closed), #2716/#2724/#2799 (open), #4420 (merged)
+
+> **Outcome:** this decision was enacted the same day it was drafted. The
+> contributor dropped the truncation and added tests for both fixes; #4850
+> merged on 2026-06-28 as "fix(focused-iteration): reset per-call state and
+> offset citations across subsections". See the Decision section at the end.
+
+## Context
+
+PR #4850 (originally "fix(focused-iteration): resolve state leak, add
+truncation ceiling and correct citation offset") bundled three independent
+changes to `advanced_search_system/strategies/focused_iteration_strategy.py`.
+Two are genuine bug fixes; the third re-introduces a change we have already
+rejected twice. This ADR records the per-change verdict and the surrounding
+history so we do not re-litigate it a fourth time.
+
+`focused-iteration` is one of the five production strategies that remain after
+#4420 removed ~22 experimental ones (and #4548 later removed `mcp`). It is the documented 96.51%-SimpleQA strategy
+(see the module docstring), but it is **not** the primary direction anymore: the
+default strategy is now **`langgraph-agent`**. langgraph-agent is considered the
+more flexible and stronger-performing option — it can drive multiple tools and
+adapt its approach per query — and it is the recommended path when there is
+enough VRAM to run a capable model (or a hosted model is used). `focused-iteration`
+remains supported and strong for SimpleQA-style factual lookups, but it is a
+secondary path. So the bar for changing it is "fix real bugs, change behaviour as
+little as possible" — not "make it converge with `source-based`."
+
+## The three changes in PR #4850
+
+### 1. State-leak reset — **ACCEPT**
+
+```python
+# top of analyze_topic()
+self.all_search_results = []
+self.results_by_iteration = {}
+```
+
+In report ("detailed") mode the **same** strategy instance handles every
+subsection sequentially (`report_generator.py:476-493`, which also pins
+`max_iterations=1` per subsection). On `main`, `all_search_results` was never
+reset, so subsection *N* synthesised over subsections 1..*N* combined →
+unbounded growth → context-window overflow (issue #4851).
+
+Why we want it: it is the actual root cause. Three independent reviewers
+confirmed the reset fixes the leak and introduces no new cross-subsection
+corruption. Residual un-reset state (`explorer.progress`,
+`findings_repository.documents`) is pre-existing and inert under report mode's
+1-iteration pin — a possible follow-up, not a blocker. The sibling
+`source_based_strategy` avoids the class of bug structurally by using a *local*
+accumulator instead of a `self.` attribute; that is the cleaner long-term shape,
+but the reset is correct as-is.
+
+**Completeness check (does the reset drop sources?).** A natural worry is that
+this change — which also moves the bibliography collection from a per-iteration
+`all_links_of_system.extend(iteration_results)` *inside* the loop to a single
+post-loop `all_links_of_system.extend(self.all_search_results)` — could leave the
+source list incomplete. It does not, because `all_links_of_system` is **not**
+reset: it is the cumulative, shared list (the *same* object as
+`search_system.all_links_of_system`, search_system.py:229; the `id()` guard at
+search_system.py:472-475 prevents the back-compat re-extend from doubling it).
+Each `analyze_topic` call appends only *its own* accumulated results to that
+shared list, so across report subsections it accumulates correctly:
+
+| Step | `all_search_results` | `all_links_of_system` (shared) | offset |
+|------|----------------------|--------------------------------|--------|
+| Sub1 | reset→`[A,B,C]` | `[A,B,C]` | 0 → [1,2,3] |
+| Sub2 | reset→`[D,E]` | `[A,B,C,D,E]` | 3 → [4,5] |
+| Sub3 | reset→`[F]` | `[A,B,C,D,E,F]` | 5 → [6] |
+
+On `main` the same scenario also ends with `[A…F]` in `all_links` — the `main`
+bug was that Sub3's *synthesis* re-ingested A–E (overflow), not that sources went
+missing. So **on the success path**, with the truncation ceiling (change #2) held
+high enough never to fire, the PR's `all_links_of_system` contains the same
+sources as `main` (verified for single-research mode, report mode, and the
+follow-up delegate path, which propagates the delegate's sources via
+`result["all_links_of_system"]` in `enhanced_contextual_followup.py:208-244`).
+Nothing reads `all_links_of_system` mid-loop, so removing the per-iteration extend
+breaks no incremental reader.
+
+**One error-path caveat (adversarially verified).** Because the PR's *only*
+`all_links_of_system.extend(...)` now lives *after* the loop, inside the same
+`try` that wraps the loop (the catch-all `except Exception` in
+`analyze_topic`; line 382 as of the #4850 merge), an **uncaught exception mid-loop** — e.g.
+the unguarded `model.invoke` in question generation (`browsecomp_question.py:96`
+and `:282`), which fails routinely on LLM rate-limit/timeout — skips that extend.
+On `main`, the per-iteration extend had already recorded the completed iterations'
+sources, so a degraded/failed single-research run still surfaces them in its
+Sources list; on the PR, that failed run surfaces none. This is a real divergence,
+but only on the *error* path (the run already returns `_create_error_response`),
+and the PR's all-or-nothing behaviour is arguably *preferable* (no half-populated
+bibliography from a run that errored out). In report mode it does not arise
+(`max_iterations=1`: a mid-loop throw means iteration 1 never completed, so
+neither side accumulated anything). It is **not** the "source list suddenly
+incomplete" symptom on successful research — that one is change #2 alone (below).
+
+### 2. `final_max_results` truncation ceiling — **DROP FROM THIS PR (cap is open for separate, benchmark-gated debate)**
+
+The objection is to the *blind insertion-order slice as shipped in #4850*, not to
+the idea of capping results in principle. A relevance-aware cap may have merit;
+the maintainer is open to it as its own PR, decided with a benchmark (see the
+"standing position" note below).
+
+```python
+max_results = int(self.get_setting("search.final_max_results", 100))
+if len(self.all_search_results) > max_results:
+ self.all_search_results = self.all_search_results[:max_results] # blind, insertion-order
+```
+
+Why we do **not** want this:
+
+- **It keeps the *first* 100, not the *best* 100.** Results accumulate
+ iteration 1→8; the strategy's highest-value work (verification searches gated
+ on `iteration > 3`) lands at the *tail* and is the first thing dropped. The
+ proven sibling applies the *same* setting via
+ `cross_engine_filter.filter_results(..., reorder=True, reindex=True)` —
+ relevance-rank + dedup, then top-N (`source_based_strategy.py:421-431`). A raw
+ `[:100]` slice does neither; the setting's own description even promises
+ "deduplicating and filtering."
+- **It runs unconditionally on the protected SimpleQA path** (8 iterations),
+ which the docstring asks us to preserve. `main` applies **no** truncation
+ here, so the 96.51% number was measured without it. At default
+ `search.max_results=50` × 5 questions, the 100 ceiling is hit by ~iteration 2,
+ discarding iterations 3–8 from synthesis — an unbenchmarked accuracy risk.
+- **It is mostly unnecessary for the bug it targets.** Report mode already
+ forces `max_iterations=1`, and change #1 (the reset) already fixes the
+ overflow. The truncation's cost falls almost entirely on the *non-report* path
+ it was not meant to touch.
+- **It is the one change that actually shrinks the source list.** Per the
+ completeness check under #1, `all_links_of_system` is otherwise complete. But
+ in single-research mode (8 iterations producing, say, 250 results) the slice
+ caps `all_search_results` at 100 *before* the `all_links` extend, so the
+ returned `sources` / Sources section silently drops the other 150 (and keeps
+ the *first* 100 by insertion order, not the best). On `main` all 250 are
+ returned. This is exactly the "source list suddenly incomplete" failure mode —
+ and it is caused by #2, not #1. Dropping #2 restores a complete, correctly
+ numbered source list.
+- **Minor robustness smell:** `int(get_setting(..., 100))` raises on a
+ present-but-null/non-numeric value, and the surrounding `except Exception`
+ converts that into a generic "Search iteration failed" *after* all search work
+ (reachable via direct DB / programmatic `settings_snapshot`, not the validated
+ UI).
+
+This is **the third time** this exact change has been proposed:
+
+- **#2001** — "apply final_max_results filtering in focused_iteration_strategy"
+ (CrossEngineFilter, matching source-based). **Closed.** Maintainer:
+ *"I am not so sure about this change. It is not what the strategy is supposed
+ to be. Not every strategy should do the same thing."* and *"Outdated due to
+ langraph strategy."*
+- **#2067** — same idea, simpler. **Closed.**
+- **#4850 (#2 of 3)** — same idea, blunter (`[:N]` instead of the filter).
+
+**Standing position (open, not a flat no).** The decision for *this PR* is to drop
+#2; the broader question of whether `focused-iteration` should cap results at all
+is open for separate debate. Guardrails for that debate: (a) not every strategy
+needs the same final-cap behaviour; (b) any cap must preserve the *best* sources
+(relevance-rank/dedup, e.g. harvested from the V2/V3 work below), not drop them by
+position; and (c) it should be gated on a SimpleQA benchmark proving no
+regression. Note also that an earlier informal evaluation (maintainer's
+recollection, details unverified) did not show a clear performance gain from a
+results cap — a reason to benchmark before committing, not a settled conclusion.
+
+**Post-merge evidence (2026-06-29).** After #4850 merged (with the truncation
+dropped), the contributor reported still hitting the context-length error on
+report mode: the reset fixes the *cross-subsection* leak, but nothing in the
+strategy accounts for the model's context window within a single call, so a
+sufficiently large single run can still overflow. That confirms #4851 had two
+components — the state leak (fixed) and missing context-length awareness (not
+fixed) — and it strengthens the case for having the capping debate. It does
+*not* change the verdict on the blind head-slice: a context-aware or
+relevance-aware mechanism under guardrails (a)–(c) above is the follow-up to
+propose, not `[:100]`.
+
+### 3. Citation offset — **ACCEPT**
+
+```python
+nr_of_links = total_citation_count_before_this_search # = len(all_links_of_system), was hardcoded 0
+```
+
+In report mode each subsection previously re-numbered citations from `[1]`
+(`nr_of_links=0`), so `[1]` meant different sources in different subsections and
+desynced from the single shared bibliography that `format_links_to_markdown`
+builds from `all_links_of_system`. Offsetting by the running link count makes
+inline citations contiguous and collision-free across subsections, matching the
+proven `source_based_strategy` invariant (verified by walking subsections 1→3:
+no off-by-one, gap, or orphan). For single (non-report) research the offset is 0,
+so citations still start at `[1]`.
+
+Test coverage: an early revision of #4850 only tested change #1; the merged
+version covers everything we asked for —
+`test_does_not_accumulate_results_across_calls` (#1),
+`test_citation_offset_uses_existing_all_links_count` and
+`test_report_mode_indices_stay_sequential_across_sections` (#3).
+
+## Cross-strategy comparison — #1 and #3 were focused-only oversights
+
+The two fixes are not a cross-cutting change; they bring `focused-iteration` in
+line with what the other production strategies already do. The contested
+truncation (#2) matches *neither* sibling.
+
+| Concern | focused (pre-#4850) | focused (post-#4850) | `langgraph-agent` (default) | `source-based` |
+|---|---|---|---|---|
+| **#1** Reset per-subsection working set | ❌ never reset (the leak) | ✅ resets `all_search_results` | ✅ `self.collector.reset()` (`langgraph_agent_strategy.py:788`) | ✅ *local* accumulator — no shared state to leak |
+| **#3** Citation offset | ❌ `nr_of_links=0` | ✅ `len(all_links_of_system)` | ✅ `nr_of_links = len(self.all_links_of_system)` (`:789`) | ✅ `:178` / `:481` |
+| **#2** `final_max_results` cap | none | blind `[:100]` | **none** (agent manages its own context) | proper: cross-engine filter `reorder=True, reindex=True` |
+
+Takeaways:
+
+- **#1 and #3 fix bugs unique to `focused-iteration`.** `langgraph-agent` and
+ `source-based` already reset per subsection and already offset citations, so the
+ fixes match two independent, already-shipping implementations — strong evidence
+ they are correct.
+- **#2 is an outlier in both directions.** Only `source-based` caps, and it does so
+ *properly* (relevance reorder + reindex); `langgraph-agent` deliberately doesn't
+ cap. A blind head-slice matches neither — extra evidence to drop it.
+- Because `langgraph-agent` is the **default**, the #4851 overflow this PR fixes
+ mainly bites users who switched to `focused-iteration` for report mode — which
+ fits the "not the primary investment area" framing.
+
+## Cross-section context & citation resolution (report mode)
+
+Resetting the per-subsection working set raises a fair question: does a later
+subsection still "know" what earlier ones used, and do in-text citations still
+resolve? Both hold, because the relevant state is **not** what gets reset:
+
+- **Coherence channel is independent of the reset.** Report generation feeds each
+ subsection a `=== CONTENT ALREADY WRITTEN (DO NOT REPEAT) ===` block built from
+ the previous sections' *generated content* (`report_generator.py:261-297`,
+ `_build_previous_context`; default last 3 sections, char-capped), injected into
+ the subsection query that drives both question generation and synthesis. This
+ survives change #1 untouched. The `main` leak added *raw search results* on top
+ of that — which is the overflow bug, not the coherence mechanism.
+- **In-text citations still resolve.** Inline `[N]` markers are hyperlinked by
+ index-matching against the structured source list
+ (`text_optimization/citation_formatter.py` `apply_inline_hyperlinks`:
+ `{str(s["index"]): (title, url)}`), and that list is the cumulative, un-reset
+ `all_links_of_system`. With #3 giving each subsection a distinct index range,
+ every `[N]` maps to exactly one bibliography entry. The reset removes sources
+ from the per-subsection *synthesis input*, not from the *source list citations
+ resolve against* — only #2's truncation removes anything from the latter.
+
+**Known soft nuance (separate follow-up, not a blocker for #4850).**
+`_build_previous_context` injects prior prose *with its old `[N]` markers intact*
+(no stripping). After the reset, the current subsection's synthesis LLM sees those
+numbers but no longer holds their source content (only its own offset-numbered
+sources). They still resolve globally in the assembled report, and the "do not
+repeat" instruction discourages echoing them, so this is a quality wrinkle rather
+than broken links — and it is strictly *less* bad than `main`, where `nr_of_links=0`
+made the numbers collide across sections. The clean fix (strip inline markers from
+the context block, or wire a real `previous_knowledge` in place of the hardcoded
+`""` in `focused_iteration_strategy.py`'s `analyze_followup` call, line 325 as of
+the #4850 merge) belongs in its own PR.
+
+## History of focused-iteration (for future readers)
+
+- **#917** — introduced the strategy (standard + adaptive search).
+- **#1277** — configurable options (the A/B knobs in `__init__`).
+- **#1258 / #1282** — robustness (empty-questions ThreadPool crash; instruction
+ injection when adaptive questions disabled).
+- **#2293** — docs: suggest 10–20 iterations.
+- **#2001 / #2067** — `final_max_results` truncation. **Both closed** (see above).
+- **V2/V3/V4 A/B era (#2716 = V2, #2724 = V2+V3 combined, #2799 = V4 — all
+ open):** separate, selectable strategies for benchmarking, leaving V1
+ untouched. V2 adds **URL dedup**; V3 adds **semantic dedup + relevance
+ sorting** (places most-relevant results last for context-window use); V4 swaps
+ in a task-oriented question generator. Merged review-fix PRs from this era:
+ #2797, #2980, #2996.
+- **#4420 (merged):** removed ~22 experimental strategies + the
+ `show_all_strategies` toggle. Survivors: `source-based`, `focused-iteration`,
+ `focused-iteration-standard`, `topic-organization`, `mcp`, `langgraph-agent`
+ (default). `mcp` was subsequently removed by #4548 (the factory now redirects
+ it to `langgraph-agent`). The V2/V3/V4 strategy files are **not** on `main`.
+- **#4711 / #4717 / #4729** — DRY refactors (base error-response + citation
+ helpers, `run_parallel_searches`, post-merge cleanups).
+
+## Open V2/V3/V4 PRs (#2716, #2724, #2799) — keep open, revisit later
+
+**Status: undecided — left open on purpose.** This ADR records the context for a
+future revisit rather than recommending closure now.
+
+Factors to weigh when we come back to them:
+
+- **Against merging as-is:** exploratory A/B artifacts (created March 2026;
+ 785 / 2,406 / 3,687 lines; `mergeable` UNKNOWN — likely heavy conflicts after
+ #4711/#4717/#4729 and #4420); **never benchmarked** against V1 (#2716's
+ SimpleQA checkbox is unchecked; #2724/#2799 don't mention a benchmark at
+ all); and as standalone strategies they would re-add
+ experimental options that #4420 deliberately removed, against the
+ langgraph-first direction.
+- **Worth keeping for:** the *ideas*, not the parallel strategies. V2's URL dedup
+ and V3's relevance-sorting-before-cap are exactly the "keep the best, not the
+ first" mechanism a future `focused-iteration` cap would need (see change #2).
+ If a cap is ever justified, the cleanest path is to port that — small, targeted,
+ into the single strategy — and benchmark it, rather than reviving three
+ separate strategies.
+
+No action on these PRs for now; this ADR just makes the trade-offs explicit so
+the eventual decision is a quick one.
+
+## Decision
+
+We asked the contributor to **split**: keep changes #1 (reset) and #3 (citation
+offset) — both correct, valuable bug fixes — and **drop #2 (truncation)**, plus
+add the missing tests for #3. The contributor did exactly that, and #4850
+merged on 2026-06-28 with #1 + #3 and tests for both. That shipped the
+state-leak half of #4851 without reshaping the strategy's synthesis behaviour
+or re-opening a question we had already closed twice.
+
+## Consequences
+
+- `focused-iteration` now resets its per-call working set and numbers citations
+ contiguously across report subsections, matching `langgraph-agent` and
+ `source-based`.
+- No truncation was added; the 96.51%-SimpleQA synthesis path is
+ behaviourally unchanged.
+- The context-length half of #4851 remains open (see the post-merge evidence
+ under change #2): any future cap must follow guardrails (a)–(c) —
+ per-strategy justification, keep-the-best (not head-slice), SimpleQA
+ benchmark gate.
+- Follow-ups (separate PRs):
+ 1. **Cross-section context quality** — strip inline `[N]` markers from
+ `_build_previous_context`, and/or wire a real `previous_knowledge` in
+ place of the hardcoded `""` in `focused_iteration_strategy.py`'s
+ `analyze_followup` call; verify with an LLM-judged before/after on a
+ report-mode query (not unit-testable).
+ 2. **Optional refactor** — move `focused-iteration` to a *local* accumulator
+ like `source-based`, so the per-call working set cannot leak by
+ construction (removes the need to remember the reset).
+ 3. **If a result cap is ever wanted on `focused-iteration`** — port V2/V3's
+ relevance-rank + dedup (not a blind slice), consider context-length
+ awareness per the post-merge evidence, and gate it behind a SimpleQA
+ benchmark.