Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

feat(search): make fetch content max length a user setting (#7194)

S2a1b207d592b
diff --git a/changelog.d/+fetch-content-max-length.feature.md b/changelog.d/+fetch-content-max-length.feature.md
new file mode 100644
index 0000000000..074ac45ce5
--- /dev/null
+++ b/changelog.d/+fetch-content-max-length.feature.md
@@ -0,0 +1 @@
+Added a `search.fetch.content_max_length` setting (Fetch Content Max Length, default 10000, range 1000–50000) below the fetch mode setting. It caps fetched page text per URL before it is returned directly (`full` mode) or sent to the LLM for focused extraction (`summary_focus` / `summary_focus_query`).
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index 3643a7c6c3..c4b2ed51c0 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -643,6 +643,7 @@ They are not available in the Web UI because they are needed to start the applic
| `search.engine.web.zenodo.use_in_auto_search` | `LDR_SEARCH_ENGINE_WEB_ZENODO_USE_IN_AUTO_SEARCH` | `false` | Include Zenodo in auto search mode | SEARCH |
| `search.engine.web.zenodo.weaknesses` | `LDR_SEARCH_ENGINE_WEB_ZENODO_WEAKNESSES` | ``["research focus", "variable quality", "metadata only"]`` | Limitations of Zenodo. | SEARCH |
| `search.favorites` | `LDR_SEARCH_FAVORITES` | ``["arxiv", "searxng", "library", "openalex"]`` | List of favorite search engine IDs that appear at the top of the dropdown. Edit as JSON array, e.g. ["arxiv", "brave"] | SEARCH |
+| `search.fetch.content_max_length` | `LDR_SEARCH_FETCH_CONTENT_MAX_LENGTH` | `10000` | Maximum page text (in characters) passed to the fetch_content tool per URL before summarization. The fetched page (HTTP or local library document) is truncated to this length before it is returned directly ('full' mode) or sent to the LLM for focused extraction ('summary_focus' / 'summary_focus_query'). Increase for more complete context, decrease to keep small-model context clean. Default is 10000. Note: This setting applies exclusively to the langgraph-agent strategy's fetch_content tool; pipeline strategies retrieve full page text upfront during search when search.snippets_only is disabled. | SEARCH |
| `search.fetch.max_per_topic` | `LDR_SEARCH_FETCH_MAX_PER_TOPIC` | `2` | Maximum number of page content fetches (fetch_content calls) allowed per subagent (and lead agent) per topic/section in autonomous agent strategies (like langgraph-agent). Prevents agents from exhausting their recursion/turn budget on page fetches instead of searching and reasoning. Set to 0 for unlimited fetches (only capped by agent recursion limit). Default is 2 (subagents typically need at most 1-2 targeted page fetches to supplement search snippets; a conservative per-agent cap prevents runaway fetches without starving parallel subtopics or consuming entire turn budgets). | SEARCH |
| `search.fetch.mode` | `LDR_SEARCH_FETCH_MODE` | `summary_focus_query` | Controls the LangGraph agent's fetch_content tool. 'summary_focus_query' (default) asks the LLM to extract only spans relevant to a focus question the agent supplies per call AND the original research query, so vague focus can be disambiguated — keeps small-model context clean. 'summary_focus' is the same but uses only the per-call focus question. 'full' returns the entire extracted page text (legacy behavior; can flood small-model context with boilerplate/metadata and degrade benchmark accuracy). 'disabled' removes the tool entirely so the agent works from search snippets only — useful for diagnosing whether fetching is helping or hurting accuracy. Note: This setting applies exclusively to the langgraph-agent strategy; pipeline strategies retrieve full page text upfront during search when search.snippets_only is disabled. | SEARCH |
| `search.final_max_results` | `LDR_SEARCH_FINAL_MAX_RESULTS` | `100` | Maximum unique sources to include in the final research output. Applied in pipeline/source-based strategies once after all search iterations complete, deduplicating and filtering across all accumulated results. Note: Autonomous agent strategies (like langgraph-agent) manage context autonomously and do not cap final results. | SEARCH |
diff --git a/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py b/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py
index 8271cb2260..969b08bea9 100644
--- a/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py
+++ b/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py
@@ -56,6 +56,12 @@ from .prompts import SUMMARY_FOCUS_PROMPT, SUMMARY_FOCUS_QUERY_PROMPT
CONTENT_FETCH_TIMEOUT = 30
CONTENT_MAX_LENGTH = 10_000
+# Bounds for the user-configurable ``search.fetch.content_max_length``
+# setting (see default_settings.json). The fetch tool truncates page text
+# to this length BEFORE summary_focus / summary_focus_query extraction.
+CONTENT_MAX_LENGTH_MIN = 1_000
+CONTENT_MAX_LENGTH_MAX = 50_000
+
FETCH_MODES = (
"disabled",
@@ -97,6 +103,55 @@ def _read_max_fetches_setting(
return default
+def _read_content_max_length_setting(
+ settings_snapshot: dict | None,
+ default: int = CONTENT_MAX_LENGTH,
+) -> int:
+ """Read the ``search.fetch.content_max_length`` setting from a settings snapshot.
+
+ The value caps page text (HTTP or local library document) BEFORE it is
+ returned directly (``full`` mode) or sent to the LLM for focused
+ extraction (``summary_focus`` / ``summary_focus_query``).
+
+ Returns the configured length clamped to
+ [CONTENT_MAX_LENGTH_MIN, CONTENT_MAX_LENGTH_MAX], or default if
+ missing/invalid/non-positive.
+ """
+ if not settings_snapshot:
+ return default
+ from local_deep_research.config.thread_settings import (
+ get_setting_from_snapshot,
+ )
+
+ try:
+ val = get_setting_from_snapshot(
+ "search.fetch.content_max_length",
+ default=None,
+ settings_snapshot=settings_snapshot,
+ )
+ except Exception:
+ return default
+ if val is None:
+ return default
+ unwrapped = unwrap_setting(val)
+ try:
+ if isinstance(unwrapped, str):
+ stripped = unwrapped.strip()
+ try:
+ limit = int(stripped)
+ except ValueError:
+ # Allow float strings like "10000.7" (truncate toward zero,
+ # matching int(float) semantics for numeric inputs).
+ limit = int(float(stripped))
+ else:
+ limit = int(unwrapped)
+ if limit <= 0:
+ return default
+ return min(CONTENT_MAX_LENGTH_MAX, max(CONTENT_MAX_LENGTH_MIN, limit))
+ except (TypeError, ValueError, OverflowError):
+ return default
+
+
_warned_collector_missing_fetch_cap = False
@@ -270,6 +325,7 @@ def _try_resolve_url(
library_resolver: Any,
collector: Any,
_visited: set[str] | None = None,
+ content_max_length: int | None = None,
) -> tuple[str, Any] | None:
"""Pre-resolve a fetch URL that isn't a network URL.
@@ -329,7 +385,11 @@ def _try_resolve_url(
# the fetch tool updates its ``url`` variable before the egress
# gate / HTTP fetch runs against the resolved URL — not the marker.
recursive = _try_resolve_url(
- resolved_url, library_resolver, collector, _visited=_visited
+ resolved_url,
+ library_resolver,
+ collector,
+ _visited=_visited,
+ content_max_length=content_max_length,
)
if recursive is not None:
return recursive
@@ -355,8 +415,13 @@ def _try_resolve_url(
content = library_resolver(url)
if content is not None:
text = content.get("content") or ""
- if len(text) > CONTENT_MAX_LENGTH:
- text = text[:CONTENT_MAX_LENGTH]
+ cap = (
+ content_max_length
+ if content_max_length is not None
+ else CONTENT_MAX_LENGTH
+ )
+ if len(text) > cap:
+ text = text[:cap]
return (
_KIND_RESULT,
{
@@ -379,6 +444,7 @@ def _fetch_raw_content(
mode_label: str,
max_fetches: int | None = None,
subagent_id: str | None = None,
+ content_max_length: int | None = None,
) -> tuple[dict | None, str, str | None, int | None]:
"""Pre-resolve non-network URL shapes or fetch via HTTP.
@@ -390,7 +456,17 @@ def _fetch_raw_content(
"""
from local_deep_research.content_fetcher import ContentFetcher
- pre = _try_resolve_url(url, library_resolver, collector)
+ effective_max_length = (
+ content_max_length
+ if content_max_length is not None
+ else _read_content_max_length_setting(settings_snapshot)
+ )
+ pre = _try_resolve_url(
+ url,
+ library_resolver,
+ collector,
+ content_max_length=effective_max_length,
+ )
result: dict | None = None
if pre is not None:
kind, payload = pre
@@ -434,7 +510,7 @@ def _fetch_raw_content(
enable_js_rendering=enable_js,
egress_context=egress_context,
) as fetcher:
- result = fetcher.fetch(url, max_length=CONTENT_MAX_LENGTH)
+ result = fetcher.fetch(url, max_length=effective_max_length)
return result, url, None, current_count
@@ -446,6 +522,7 @@ def _make_full_fetch_tool(
library_resolver: Any = None,
max_fetches: int | None = None,
subagent_id: str | None = None,
+ content_max_length: int | None = None,
):
mode_label = "full"
limit = (
@@ -477,6 +554,7 @@ def _make_full_fetch_tool(
mode_label,
max_fetches=limit,
subagent_id=subagent_id,
+ content_max_length=content_max_length,
)
if err_msg is not None:
return err_msg
@@ -538,6 +616,7 @@ def _make_summary_fetch_tool(
library_resolver: Any = None,
max_fetches: int | None = None,
subagent_id: str | None = None,
+ content_max_length: int | None = None,
):
"""Build the summary-mode fetch tool.
@@ -581,6 +660,7 @@ def _make_summary_fetch_tool(
mode_label,
max_fetches=limit,
subagent_id=subagent_id,
+ content_max_length=content_max_length,
)
if err_msg is not None:
return err_msg
@@ -709,6 +789,7 @@ def build_fetch_tool(
library_resolver: Any = None,
max_fetches: int | None = None,
subagent_id: str | None = None,
+ content_max_length: int | None = None,
):
"""Build the agent-facing ``fetch_content`` tool for *mode*.
@@ -743,6 +824,7 @@ def build_fetch_tool(
library_resolver=library_resolver,
max_fetches=max_fetches,
subagent_id=subagent_id,
+ content_max_length=content_max_length,
)
if mode == "summary_focus":
if model is None:
@@ -756,6 +838,7 @@ def build_fetch_tool(
library_resolver=library_resolver,
max_fetches=max_fetches,
subagent_id=subagent_id,
+ content_max_length=content_max_length,
)
if mode == "summary_focus_query":
if model is None:
@@ -771,6 +854,7 @@ def build_fetch_tool(
library_resolver=library_resolver,
max_fetches=max_fetches,
subagent_id=subagent_id,
+ content_max_length=content_max_length,
)
raise ValueError(
f"Unknown fetch mode {mode!r}; expected one of {FETCH_MODES}"
diff --git a/src/local_deep_research/defaults/default_settings.json b/src/local_deep_research/defaults/default_settings.json
index fcc6bd92ab..8749e4bb63 100644
--- a/src/local_deep_research/defaults/default_settings.json
+++ b/src/local_deep_research/defaults/default_settings.json
@@ -1022,6 +1022,20 @@
"value": "summary_focus_query",
"visible": true
},
+ "search.fetch.content_max_length": {
+ "category": "search_parameters",
+ "description": "Maximum page text (in characters) passed to the fetch_content tool per URL before summarization. The fetched page (HTTP or local library document) is truncated to this length before it is returned directly ('full' mode) or sent to the LLM for focused extraction ('summary_focus' / 'summary_focus_query'). Increase for more complete context, decrease to keep small-model context clean. Default is 10000. Note: This setting applies exclusively to the langgraph-agent strategy's fetch_content tool; pipeline strategies retrieve full page text upfront during search when search.snippets_only is disabled.",
+ "editable": true,
+ "max_value": 50000,
+ "min_value": 1000,
+ "name": "Fetch Content Max Length",
+ "options": null,
+ "step": 500,
+ "type": "SEARCH",
+ "ui_element": "number",
+ "value": 10000,
+ "visible": true
+ },
"search.fetch.max_per_topic": {
"category": "search_parameters",
"description": "Maximum number of page content fetches (fetch_content calls) allowed per subagent (and lead agent) per topic/section in autonomous agent strategies (like langgraph-agent). Prevents agents from exhausting their recursion/turn budget on page fetches instead of searching and reasoning. Set to 0 for unlimited fetches (only capped by agent recursion limit). Default is 2 (subagents typically need at most 1-2 targeted page fetches to supplement search snippets; a conservative per-agent cap prevents runaway fetches without starving parallel subtopics or consuming entire turn budgets).",
diff --git a/tests/advanced_search_system/tools/test_fetch_modes.py b/tests/advanced_search_system/tools/test_fetch_modes.py
index 28183eeade..559ceb5fdc 100644
--- a/tests/advanced_search_system/tools/test_fetch_modes.py
+++ b/tests/advanced_search_system/tools/test_fetch_modes.py
@@ -1473,6 +1473,209 @@ def test_read_max_fetches_setting_clamping():
)
+def test_read_content_max_length_setting_defaults_and_clamping():
+ from local_deep_research.advanced_search_system.tools.fetch import (
+ CONTENT_MAX_LENGTH,
+ CONTENT_MAX_LENGTH_MAX,
+ CONTENT_MAX_LENGTH_MIN,
+ _read_content_max_length_setting,
+ )
+
+ assert _read_content_max_length_setting(None) == CONTENT_MAX_LENGTH
+ assert _read_content_max_length_setting({}) == CONTENT_MAX_LENGTH
+ assert (
+ _read_content_max_length_setting(
+ {"search.fetch.content_max_length": {"value": 5000}}
+ )
+ == 5000
+ )
+ # Raw-int snapshot form (no {"value": ...} wrapper).
+ assert (
+ _read_content_max_length_setting(
+ {"search.fetch.content_max_length": 2000}
+ )
+ == 2000
+ )
+ assert (
+ _read_content_max_length_setting(
+ {"search.fetch.content_max_length": {"value": 100000}}
+ )
+ == CONTENT_MAX_LENGTH_MAX
+ )
+ assert (
+ _read_content_max_length_setting(
+ {"search.fetch.content_max_length": {"value": 500}}
+ )
+ == CONTENT_MAX_LENGTH_MIN
+ )
+ assert (
+ _read_content_max_length_setting(
+ {"search.fetch.content_max_length": {"value": -5}}
+ )
+ == CONTENT_MAX_LENGTH
+ )
+ assert (
+ _read_content_max_length_setting(
+ {"search.fetch.content_max_length": {"value": "invalid"}}
+ )
+ == CONTENT_MAX_LENGTH
+ )
+
+
+def test_library_doc_respects_content_max_length_setting():
+ """A snapshot-configured cap truncates library docs instead of the default."""
+ long_text = "A" * 20000
+
+ def _large_resolver(_url):
+ return {
+ "title": "Large Doc",
+ "content": long_text,
+ "url": _url,
+ "snippet": "A" * 200,
+ }
+
+ collector = SearchResultsCollector([])
+ tool = build_fetch_tool(
+ "full",
+ collector,
+ library_resolver=_large_resolver,
+ settings_snapshot={"search.fetch.content_max_length": {"value": 3000}},
+ )
+
+ out = tool.invoke({"url": "/library/document/large-doc"})
+ body = out.split("\n\n", 1)[1]
+ assert len(body) == 3000
+
+
+def test_http_fetch_receives_configured_max_length():
+ """The snapshot cap is forwarded to ContentFetcher as max_length."""
+ collector = SearchResultsCollector([])
+ tool = build_fetch_tool(
+ "full",
+ collector,
+ settings_snapshot={"search.fetch.content_max_length": {"value": 4200}},
+ )
+
+ cm = _fetcher_cm(title="Network", content="Network page")
+ with patch(
+ "local_deep_research.content_fetcher.ContentFetcher", return_value=cm
+ ):
+ out = tool.invoke({"url": "https://example.com/net"})
+
+ assert "Title: Network" in out
+ fetcher = cm.__enter__.return_value
+ fetcher.fetch.assert_called_once_with(
+ "https://example.com/net", max_length=4200
+ )
+
+
+def test_explicit_content_max_length_overrides_snapshot():
+ """An explicit arg wins over the snapshot for library-doc truncation."""
+ long_text = "B" * 20000
+
+ def _large_resolver(_url):
+ return {
+ "title": "Large Doc",
+ "content": long_text,
+ "url": _url,
+ "snippet": "B" * 200,
+ }
+
+ collector = SearchResultsCollector([])
+ tool = build_fetch_tool(
+ "full",
+ collector,
+ library_resolver=_large_resolver,
+ settings_snapshot={"search.fetch.content_max_length": {"value": 3000}},
+ content_max_length=1500,
+ )
+
+ out = tool.invoke({"url": "/library/document/large-doc"})
+ body = out.split("\n\n", 1)[1]
+ assert len(body) == 1500
+
+
+def test_summary_mode_respects_content_max_length_setting():
+ """Summary mode truncates library-doc text before LLM extraction."""
+ long_text = "C" * 20000
+
+ def _large_resolver(_url):
+ return {
+ "title": "Large Doc",
+ "content": long_text,
+ "url": _url,
+ "snippet": "C" * 200,
+ }
+
+ collector = SearchResultsCollector([])
+ model = _model_returning("relevant excerpt")
+ tool = build_fetch_tool(
+ "summary_focus",
+ collector,
+ model=model,
+ library_resolver=_large_resolver,
+ settings_snapshot={"search.fetch.content_max_length": {"value": 3000}},
+ )
+
+ out = tool.invoke(
+ {"url": "/library/document/large-doc", "focus": "key fact"}
+ )
+ assert "relevant excerpt" in out
+ assert model.invoke.call_count == 1
+ prompt = model.invoke.call_args[0][0]
+ assert "C" * 3000 in prompt
+ assert "C" * 3001 not in prompt
+
+
+def test_http_fetch_real_fetcher_truncates_to_configured_max_length():
+ """End-to-end: snapshot cap flows through the real ContentFetcher."""
+ from local_deep_research.content_fetcher.url_classifier import URLType
+
+ long_text = "D" * 20000
+ long_bytes = long_text.encode("utf-8")
+
+ download_result = MagicMock()
+ download_result.is_success = True
+ download_result.content = long_bytes
+ download_result.skip_reason = None
+
+ mock_dl = MagicMock()
+ mock_dl.download_with_result.return_value = download_result
+ mock_dl.get_metadata.return_value = {"title": "Large Page"}
+
+ collector = SearchResultsCollector([])
+ tool = build_fetch_tool(
+ "full",
+ collector,
+ settings_snapshot={"search.fetch.content_max_length": {"value": 2500}},
+ )
+
+ with (
+ patch(
+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",
+ return_value=True,
+ ),
+ patch(
+ "local_deep_research.content_fetcher.fetcher.URLClassifier"
+ ) as classifier_mock,
+ patch(
+ "local_deep_research.content_fetcher.fetcher.ContentFetcher._get_downloader",
+ return_value=mock_dl,
+ ),
+ ):
+ classifier_mock.classify.return_value = URLType.HTML
+ classifier_mock.get_source_name.return_value = "Web Page"
+ out = tool.invoke({"url": "https://example.com/large"})
+
+ assert "[... content truncated ...]" in out
+ body = out.split("\n\n", 1)[1]
+ # Strip the budget footer ("(N of M page fetches used)") if present.
+ content_only = body.split("\n\n(", 1)[0]
+ # Real ContentFetcher appends "\n\n[... content truncated ...]" after cap.
+ assert content_only.startswith("D" * 2500)
+ assert len(content_only) == 2500 + len("\n\n[... content truncated ...]")
+
+
def test_full_mode_fetch_puts_the_page_title_in_the_bibliography():
"""The whole production chain for #6024, entered at the tool and with
only the HTTP fetcher stubbed: a source the agent found by search and
diff --git a/tests/settings/golden_master_settings.json b/tests/settings/golden_master_settings.json
index 311a990e08..219d873983 100644
--- a/tests/settings/golden_master_settings.json
+++ b/tests/settings/golden_master_settings.json
@@ -9093,6 +9093,20 @@
],
"visible": true
},
+ "search.fetch.content_max_length": {
+ "category": "search_parameters",
+ "description": "Maximum page text (in characters) passed to the fetch_content tool per URL before summarization. The fetched page (HTTP or local library document) is truncated to this length before it is returned directly ('full' mode) or sent to the LLM for focused extraction ('summary_focus' / 'summary_focus_query'). Increase for more complete context, decrease to keep small-model context clean. Default is 10000. Note: This setting applies exclusively to the langgraph-agent strategy's fetch_content tool; pipeline strategies retrieve full page text upfront during search when search.snippets_only is disabled.",
+ "editable": true,
+ "max_value": 50000,
+ "min_value": 1000,
+ "name": "Fetch Content Max Length",
+ "options": null,
+ "step": 500,
+ "type": "SEARCH",
+ "ui_element": "number",
+ "value": 10000,
+ "visible": true
+ },
"search.fetch.max_per_topic": {
"category": "search_parameters",
"description": "Maximum number of page content fetches (fetch_content calls) allowed per subagent (and lead agent) per topic/section in autonomous agent strategies (like langgraph-agent). Prevents agents from exhausting their recursion/turn budget on page fetches instead of searching and reasoning. Set to 0 for unlimited fetches (only capped by agent recursion limit). Default is 2 (subagents typically need at most 1-2 targeted page fetches to supplement search snippets; a conservative per-agent cap prevents runaway fetches without starving parallel subtopics or consuming entire turn budgets).",