~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs(adr): ADR-0006 — scope focused-iteration changes, reject final_max_results truncation (#4873)
LearningCircuitb69b14a20abb
diff --git a/docs/decisions/0006-focused-iteration-scope-and-final-max-results.md b/docs/decisions/0006-focused-iteration-scope-and-final-max-results.mdnew file mode 100644index 0000000000..a1defcfc7a--- /dev/null+++ b/docs/decisions/0006-focused-iteration-scope-and-final-max-results.md@@ -0,0 +1,336 @@+# ADR-0006: Keep `focused-iteration` minimal — accept the state-leak/citation fixes, defer the `final_max_results` truncation++**Date:** 2026-06-28 (updated 2026-07-03)+**Status:** Accepted+**Relates to:** PR #4850 (merged 2026-06-28), issue #4851, #2001 (closed), #2067 (closed), #2716/#2724/#2799 (open), #4420 (merged)++> **Outcome:** this decision was enacted the same day it was drafted. The+> contributor dropped the truncation and added tests for both fixes; #4850+> merged on 2026-06-28 as "fix(focused-iteration): reset per-call state and+> offset citations across subsections". See the Decision section at the end.++## Context++PR #4850 (originally "fix(focused-iteration): resolve state leak, add+truncation ceiling and correct citation offset") bundled three independent+changes to `advanced_search_system/strategies/focused_iteration_strategy.py`.+Two are genuine bug fixes; the third re-introduces a change we have already+rejected twice. This ADR records the per-change verdict and the surrounding+history so we do not re-litigate it a fourth time.++`focused-iteration` is one of the five production strategies that remain after+#4420 removed ~22 experimental ones (and #4548 later removed `mcp`). It is the documented 96.51%-SimpleQA strategy+(see the module docstring), but it is **not** the primary direction anymore: the+default strategy is now **`langgraph-agent`**. langgraph-agent is considered the+more flexible and stronger-performing option — it can drive multiple tools and+adapt its approach per query — and it is the recommended path when there is+enough VRAM to run a capable model (or a hosted model is used). `focused-iteration`+remains supported and strong for SimpleQA-style factual lookups, but it is a+secondary path. So the bar for changing it is "fix real bugs, change behaviour as+little as possible" — not "make it converge with `source-based`."++## The three changes in PR #4850++### 1. State-leak reset — **ACCEPT**++```python+# top of analyze_topic()+self.all_search_results = []+self.results_by_iteration = {}+```++In report ("detailed") mode the **same** strategy instance handles every+subsection sequentially (`report_generator.py:476-493`, which also pins+`max_iterations=1` per subsection). On `main`, `all_search_results` was never+reset, so subsection *N* synthesised over subsections 1..*N* combined →+unbounded growth → context-window overflow (issue #4851).++Why we want it: it is the actual root cause. Three independent reviewers+confirmed the reset fixes the leak and introduces no new cross-subsection+corruption. Residual un-reset state (`explorer.progress`,+`findings_repository.documents`) is pre-existing and inert under report mode's+1-iteration pin — a possible follow-up, not a blocker. The sibling+`source_based_strategy` avoids the class of bug structurally by using a *local*+accumulator instead of a `self.` attribute; that is the cleaner long-term shape,+but the reset is correct as-is.++**Completeness check (does the reset drop sources?).** A natural worry is that+this change — which also moves the bibliography collection from a per-iteration+`all_links_of_system.extend(iteration_results)` *inside* the loop to a single+post-loop `all_links_of_system.extend(self.all_search_results)` — could leave the+source list incomplete. It does not, because `all_links_of_system` is **not**+reset: it is the cumulative, shared list (the *same* object as+`search_system.all_links_of_system`, search_system.py:229; the `id()` guard at+search_system.py:472-475 prevents the back-compat re-extend from doubling it).+Each `analyze_topic` call appends only *its own* accumulated results to that+shared list, so across report subsections it accumulates correctly:++| Step | `all_search_results` | `all_links_of_system` (shared) | offset |+|------|----------------------|--------------------------------|--------|+| Sub1 | reset→`[A,B,C]` | `[A,B,C]` | 0 → [1,2,3] |+| Sub2 | reset→`[D,E]` | `[A,B,C,D,E]` | 3 → [4,5] |+| Sub3 | reset→`[F]` | `[A,B,C,D,E,F]` | 5 → [6] |++On `main` the same scenario also ends with `[A…F]` in `all_links` — the `main`+bug was that Sub3's *synthesis* re-ingested A–E (overflow), not that sources went+missing. So **on the success path**, with the truncation ceiling (change #2) held+high enough never to fire, the PR's `all_links_of_system` contains the same+sources as `main` (verified for single-research mode, report mode, and the+follow-up delegate path, which propagates the delegate's sources via+`result["all_links_of_system"]` in `enhanced_contextual_followup.py:208-244`).+Nothing reads `all_links_of_system` mid-loop, so removing the per-iteration extend+breaks no incremental reader.++**One error-path caveat (adversarially verified).** Because the PR's *only*+`all_links_of_system.extend(...)` now lives *after* the loop, inside the same+`try` that wraps the loop (the catch-all `except Exception` in+`analyze_topic`; line 382 as of the #4850 merge), an **uncaught exception mid-loop** — e.g.+the unguarded `model.invoke` in question generation (`browsecomp_question.py:96`+and `:282`), which fails routinely on LLM rate-limit/timeout — skips that extend.+On `main`, the per-iteration extend had already recorded the completed iterations'+sources, so a degraded/failed single-research run still surfaces them in its+Sources list; on the PR, that failed run surfaces none. This is a real divergence,+but only on the *error* path (the run already returns `_create_error_response`),+and the PR's all-or-nothing behaviour is arguably *preferable* (no half-populated+bibliography from a run that errored out). In report mode it does not arise+(`max_iterations=1`: a mid-loop throw means iteration 1 never completed, so+neither side accumulated anything). It is **not** the "source list suddenly+incomplete" symptom on successful research — that one is change #2 alone (below).++### 2. `final_max_results` truncation ceiling — **DROP FROM THIS PR (cap is open for separate, benchmark-gated debate)**++The objection is to the *blind insertion-order slice as shipped in #4850*, not to+the idea of capping results in principle. A relevance-aware cap may have merit;+the maintainer is open to it as its own PR, decided with a benchmark (see the+"standing position" note below).++```python+max_results = int(self.get_setting("search.final_max_results", 100))+if len(self.all_search_results) > max_results:+ self.all_search_results = self.all_search_results[:max_results] # blind, insertion-order+```++Why we do **not** want this:++- **It keeps the *first* 100, not the *best* 100.** Results accumulate+ iteration 1→8; the strategy's highest-value work (verification searches gated+ on `iteration > 3`) lands at the *tail* and is the first thing dropped. The+ proven sibling applies the *same* setting via+ `cross_engine_filter.filter_results(..., reorder=True, reindex=True)` —+ relevance-rank + dedup, then top-N (`source_based_strategy.py:421-431`). A raw+ `[:100]` slice does neither; the setting's own description even promises+ "deduplicating and filtering."+- **It runs unconditionally on the protected SimpleQA path** (8 iterations),+ which the docstring asks us to preserve. `main` applies **no** truncation+ here, so the 96.51% number was measured without it. At default+ `search.max_results=50` × 5 questions, the 100 ceiling is hit by ~iteration 2,+ discarding iterations 3–8 from synthesis — an unbenchmarked accuracy risk.+- **It is mostly unnecessary for the bug it targets.** Report mode already+ forces `max_iterations=1`, and change #1 (the reset) already fixes the+ overflow. The truncation's cost falls almost entirely on the *non-report* path+ it was not meant to touch.+- **It is the one change that actually shrinks the source list.** Per the+ completeness check under #1, `all_links_of_system` is otherwise complete. But+ in single-research mode (8 iterations producing, say, 250 results) the slice+ caps `all_search_results` at 100 *before* the `all_links` extend, so the+ returned `sources` / Sources section silently drops the other 150 (and keeps+ the *first* 100 by insertion order, not the best). On `main` all 250 are+ returned. This is exactly the "source list suddenly incomplete" failure mode —+ and it is caused by #2, not #1. Dropping #2 restores a complete, correctly+ numbered source list.+- **Minor robustness smell:** `int(get_setting(..., 100))` raises on a+ present-but-null/non-numeric value, and the surrounding `except Exception`+ converts that into a generic "Search iteration failed" *after* all search work+ (reachable via direct DB / programmatic `settings_snapshot`, not the validated+ UI).++This is **the third time** this exact change has been proposed:++- **#2001** — "apply final_max_results filtering in focused_iteration_strategy"+ (CrossEngineFilter, matching source-based). **Closed.** Maintainer:+ *"I am not so sure about this change. It is not what the strategy is supposed+ to be. Not every strategy should do the same thing."* and *"Outdated due to+ langraph strategy."*+- **#2067** — same idea, simpler. **Closed.**+- **#4850 (#2 of 3)** — same idea, blunter (`[:N]` instead of the filter).++**Standing position (open, not a flat no).** The decision for *this PR* is to drop+#2; the broader question of whether `focused-iteration` should cap results at all+is open for separate debate. Guardrails for that debate: (a) not every strategy+needs the same final-cap behaviour; (b) any cap must preserve the *best* sources+(relevance-rank/dedup, e.g. harvested from the V2/V3 work below), not drop them by+position; and (c) it should be gated on a SimpleQA benchmark proving no+regression. Note also that an earlier informal evaluation (maintainer's+recollection, details unverified) did not show a clear performance gain from a+results cap — a reason to benchmark before committing, not a settled conclusion.++**Post-merge evidence (2026-06-29).** After #4850 merged (with the truncation+dropped), the contributor reported still hitting the context-length error on+report mode: the reset fixes the *cross-subsection* leak, but nothing in the+strategy accounts for the model's context window within a single call, so a+sufficiently large single run can still overflow. That confirms #4851 had two+components — the state leak (fixed) and missing context-length awareness (not+fixed) — and it strengthens the case for having the capping debate. It does+*not* change the verdict on the blind head-slice: a context-aware or+relevance-aware mechanism under guardrails (a)–(c) above is the follow-up to+propose, not `[:100]`.++### 3. Citation offset — **ACCEPT**++```python+nr_of_links = total_citation_count_before_this_search # = len(all_links_of_system), was hardcoded 0+```++In report mode each subsection previously re-numbered citations from `[1]`+(`nr_of_links=0`), so `[1]` meant different sources in different subsections and+desynced from the single shared bibliography that `format_links_to_markdown`+builds from `all_links_of_system`. Offsetting by the running link count makes+inline citations contiguous and collision-free across subsections, matching the+proven `source_based_strategy` invariant (verified by walking subsections 1→3:+no off-by-one, gap, or orphan). For single (non-report) research the offset is 0,+so citations still start at `[1]`.++Test coverage: an early revision of #4850 only tested change #1; the merged+version covers everything we asked for —+`test_does_not_accumulate_results_across_calls` (#1),+`test_citation_offset_uses_existing_all_links_count` and+`test_report_mode_indices_stay_sequential_across_sections` (#3).++## Cross-strategy comparison — #1 and #3 were focused-only oversights++The two fixes are not a cross-cutting change; they bring `focused-iteration` in+line with what the other production strategies already do. The contested+truncation (#2) matches *neither* sibling.++| Concern | focused (pre-#4850) | focused (post-#4850) | `langgraph-agent` (default) | `source-based` |+|---|---|---|---|---|+| **#1** Reset per-subsection working set | ❌ never reset (the leak) | ✅ resets `all_search_results` | ✅ `self.collector.reset()` (`langgraph_agent_strategy.py:788`) | ✅ *local* accumulator — no shared state to leak |+| **#3** Citation offset | ❌ `nr_of_links=0` | ✅ `len(all_links_of_system)` | ✅ `nr_of_links = len(self.all_links_of_system)` (`:789`) | ✅ `:178` / `:481` |+| **#2** `final_max_results` cap | none | blind `[:100]` | **none** (agent manages its own context) | proper: cross-engine filter `reorder=True, reindex=True` |++Takeaways:++- **#1 and #3 fix bugs unique to `focused-iteration`.** `langgraph-agent` and+ `source-based` already reset per subsection and already offset citations, so the+ fixes match two independent, already-shipping implementations — strong evidence+ they are correct.+- **#2 is an outlier in both directions.** Only `source-based` caps, and it does so+ *properly* (relevance reorder + reindex); `langgraph-agent` deliberately doesn't+ cap. A blind head-slice matches neither — extra evidence to drop it.+- Because `langgraph-agent` is the **default**, the #4851 overflow this PR fixes+ mainly bites users who switched to `focused-iteration` for report mode — which+ fits the "not the primary investment area" framing.++## Cross-section context & citation resolution (report mode)++Resetting the per-subsection working set raises a fair question: does a later+subsection still "know" what earlier ones used, and do in-text citations still+resolve? Both hold, because the relevant state is **not** what gets reset:++- **Coherence channel is independent of the reset.** Report generation feeds each+ subsection a `=== CONTENT ALREADY WRITTEN (DO NOT REPEAT) ===` block built from+ the previous sections' *generated content* (`report_generator.py:261-297`,+ `_build_previous_context`; default last 3 sections, char-capped), injected into+ the subsection query that drives both question generation and synthesis. This+ survives change #1 untouched. The `main` leak added *raw search results* on top+ of that — which is the overflow bug, not the coherence mechanism.+- **In-text citations still resolve.** Inline `[N]` markers are hyperlinked by+ index-matching against the structured source list+ (`text_optimization/citation_formatter.py` `apply_inline_hyperlinks`:+ `{str(s["index"]): (title, url)}`), and that list is the cumulative, un-reset+ `all_links_of_system`. With #3 giving each subsection a distinct index range,+ every `[N]` maps to exactly one bibliography entry. The reset removes sources+ from the per-subsection *synthesis input*, not from the *source list citations+ resolve against* — only #2's truncation removes anything from the latter.++**Known soft nuance (separate follow-up, not a blocker for #4850).**+`_build_previous_context` injects prior prose *with its old `[N]` markers intact*+(no stripping). After the reset, the current subsection's synthesis LLM sees those+numbers but no longer holds their source content (only its own offset-numbered+sources). They still resolve globally in the assembled report, and the "do not+repeat" instruction discourages echoing them, so this is a quality wrinkle rather+than broken links — and it is strictly *less* bad than `main`, where `nr_of_links=0`+made the numbers collide across sections. The clean fix (strip inline markers from+the context block, or wire a real `previous_knowledge` in place of the hardcoded+`""` in `focused_iteration_strategy.py`'s `analyze_followup` call, line 325 as of+the #4850 merge) belongs in its own PR.++## History of focused-iteration (for future readers)++- **#917** — introduced the strategy (standard + adaptive search).+- **#1277** — configurable options (the A/B knobs in `__init__`).+- **#1258 / #1282** — robustness (empty-questions ThreadPool crash; instruction+ injection when adaptive questions disabled).+- **#2293** — docs: suggest 10–20 iterations.+- **#2001 / #2067** — `final_max_results` truncation. **Both closed** (see above).+- **V2/V3/V4 A/B era (#2716 = V2, #2724 = V2+V3 combined, #2799 = V4 — all+ open):** separate, selectable strategies for benchmarking, leaving V1+ untouched. V2 adds **URL dedup**; V3 adds **semantic dedup + relevance+ sorting** (places most-relevant results last for context-window use); V4 swaps+ in a task-oriented question generator. Merged review-fix PRs from this era:+ #2797, #2980, #2996.+- **#4420 (merged):** removed ~22 experimental strategies + the+ `show_all_strategies` toggle. Survivors: `source-based`, `focused-iteration`,+ `focused-iteration-standard`, `topic-organization`, `mcp`, `langgraph-agent`+ (default). `mcp` was subsequently removed by #4548 (the factory now redirects+ it to `langgraph-agent`). The V2/V3/V4 strategy files are **not** on `main`.+- **#4711 / #4717 / #4729** — DRY refactors (base error-response + citation+ helpers, `run_parallel_searches`, post-merge cleanups).++## Open V2/V3/V4 PRs (#2716, #2724, #2799) — keep open, revisit later++**Status: undecided — left open on purpose.** This ADR records the context for a+future revisit rather than recommending closure now.++Factors to weigh when we come back to them:++- **Against merging as-is:** exploratory A/B artifacts (created March 2026;+ 785 / 2,406 / 3,687 lines; `mergeable` UNKNOWN — likely heavy conflicts after+ #4711/#4717/#4729 and #4420); **never benchmarked** against V1 (#2716's+ SimpleQA checkbox is unchecked; #2724/#2799 don't mention a benchmark at+ all); and as standalone strategies they would re-add+ experimental options that #4420 deliberately removed, against the+ langgraph-first direction.+- **Worth keeping for:** the *ideas*, not the parallel strategies. V2's URL dedup+ and V3's relevance-sorting-before-cap are exactly the "keep the best, not the+ first" mechanism a future `focused-iteration` cap would need (see change #2).+ If a cap is ever justified, the cleanest path is to port that — small, targeted,+ into the single strategy — and benchmark it, rather than reviving three+ separate strategies.++No action on these PRs for now; this ADR just makes the trade-offs explicit so+the eventual decision is a quick one.++## Decision++We asked the contributor to **split**: keep changes #1 (reset) and #3 (citation+offset) — both correct, valuable bug fixes — and **drop #2 (truncation)**, plus+add the missing tests for #3. The contributor did exactly that, and #4850+merged on 2026-06-28 with #1 + #3 and tests for both. That shipped the+state-leak half of #4851 without reshaping the strategy's synthesis behaviour+or re-opening a question we had already closed twice.++## Consequences++- `focused-iteration` now resets its per-call working set and numbers citations+ contiguously across report subsections, matching `langgraph-agent` and+ `source-based`.+- No truncation was added; the 96.51%-SimpleQA synthesis path is+ behaviourally unchanged.+- The context-length half of #4851 remains open (see the post-merge evidence+ under change #2): any future cap must follow guardrails (a)–(c) —+ per-strategy justification, keep-the-best (not head-slice), SimpleQA+ benchmark gate.+- Follow-ups (separate PRs):+ 1. **Cross-section context quality** — strip inline `[N]` markers from+ `_build_previous_context`, and/or wire a real `previous_knowledge` in+ place of the hardcoded `""` in `focused_iteration_strategy.py`'s+ `analyze_followup` call; verify with an LLM-judged before/after on a+ report-mode query (not unit-testable).+ 2. **Optional refactor** — move `focused-iteration` to a *local* accumulator+ like `source-based`, so the per-call working set cannot leak by+ construction (removes the need to remember the reset).+ 3. **If a result cap is ever wanted on `focused-iteration`** — port V2/V3's+ relevance-rank + dedup (not a blind slice), consider context-length+ awareness per the post-merge evidence, and gate it behind a SimpleQA+ benchmark.