~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(llm): enforce bounded provider request timeouts (#5656)
LearningCircuitceeed035427b
diff --git a/changelog.d/5339.bugfix.md b/changelog.d/5339.bugfix.mdnew file mode 100644index 0000000000..015d2ffbba--- /dev/null+++ b/changelog.d/5339.bugfix.md@@ -0,0 +1 @@+LLM clients and the Ollama and OpenAI embedding clients now use an explicit per-operation HTTP timeout. `llm.request_timeout` defaults to 1800 seconds, clamps values below 1 second, and has no upper limit; malformed or non-finite values use the default. It bounds connect, write, pool, and response-read operations individually. A peer that keeps sending data can exceed this value even for a non-streaming response, so this is not a total-response deadline. OpenAI-compatible and Ollama clients cap connection attempts at 5 seconds; Anthropic inference uses its configured scalar timeout for connecting too. `llm.max_retries` defaults to 2 and is clamped to 0-5 for OpenAI-compatible and Anthropic SDKs. Each retry has its own operation timeouts, and SDK backoff or server-directed delays add to elapsed time. OpenAI-compatible, Anthropic, and OpenAI embedding model discovery use a 30-second operation budget with SDK retries disabled; Google and Ollama retain their 10- and 5-second discovery limits. Timeout errors name the setting and environment variable, and say that connection attempts are bounded at most at 5 seconds so raising the setting above that will not help an unreachable endpoint. Configured Ollama authentication headers now reach both the synchronous and the asynchronous HTTP client, for chat and for embeddings alike. `ChatOpenAI` receives the bound as a hashable `(connect, read, write, pool)` tuple rather than a `Timeout` object, which is what keeps `langchain_openai`'s shared httpx client cache in effect; every other client, including OpenAI embeddings and the OpenAI-compatible discovery clients, is built fresh per instance or per call and takes the SDK's own `Timeout` object; that shared async client stays bound to the event loop that built it (see issue #6293), so synchronous entry points continue to use synchronous `invoke`. On asynchronous streams `langchain_openai` applies its own 120-second `stream_chunk_timeout` between chunks, which fires before this setting. A non-streaming request receives nothing until generation finishes, so there the bound is in effect a cap on a single generation: a local model slow enough to exceed it needs a value above the 1800-second default, which has no upper limit.diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex 36809f1d78..c33484796a 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -142,6 +142,7 @@ They are not available in the Web UI because they are needed to start the applic| `llm.lmstudio.api_key` | `LDR_LLM_LMSTUDIO_API_KEY` | `` | Optional API key for LM Studio if you've enabled authentication. Leave empty for the default no-auth setup. | LLM || `llm.lmstudio.url` | `LDR_LLM_LMSTUDIO_URL` | `http://localhost:1234/v1` | HTTP endpoint URL where LM Studio is running locally. Include the full path (e.g., http://localhost:1234/v1 for OpenAI-compatible API format). | LLM || `llm.local_context_window_size` | `LDR_LLM_LOCAL_CONTEXT_WINDOW_SIZE` | `20480` | Context window size in tokens for local LLMs (Ollama, LlamaCpp, LM Studio). The default of 20480 tokens fits on systems with limited VRAM while still leaving room for the synthesis prompt. If your synthesis prompts get truncated, increase this; if you run out of VRAM, reduce it and switch to the source-based strategy. | LLM |+| `llm.max_retries` | `LDR_LLM_MAX_RETRIES` | `2` | Maximum provider SDK retries per failed inference or embedding request. Applies to OpenAI-compatible and Anthropic SDKs; the Ollama client does not retry. Values outside 0-5 are clamped. Each retry starts another request with its own per-operation timeouts; SDK backoff or server-requested retry delays can further extend elapsed time. This setting does not impose a total request deadline. Model discovery SDK retries are disabled. | LLM || `llm.max_tokens` | `LDR_LLM_MAX_TOKENS` | `30000` | Maximum tokens in model responses. Automatically capped at 80% of the context window size to leave room for prompt tokens. | LLM || `llm.model` | `LDR_LLM_MODEL` | `` | The language model to use. Available models depend on the selected provider (e.g., 'gpt-4o' for OpenAI, 'gemma:7b' for Ollama). You can type any model name supported by your provider. | LLM || `llm.ollama.api_key` | `LDR_LLM_OLLAMA_API_KEY` | `` | Optional API key for Ollama if you've put it behind a proxy that requires authentication. Leave empty for the default no-auth setup. | LLM |@@ -154,6 +155,7 @@ They are not available in the Web UI because they are needed to start the applic| `llm.openrouter.api_key` | `LDR_LLM_OPENROUTER_API_KEY` | `null` | API key to use for the OpenRouter provider. | SEARCH || `llm.orcarouter.api_key` | `LDR_LLM_ORCAROUTER_API_KEY` | `null` | API key to use for the OrcaRouter provider. | SEARCH || `llm.provider` | `LDR_LLM_PROVIDER` | `ollama` | The LLM service provider. Choose local providers (Ollama, LM Studio, LlamaCpp) for free on-device inference, or cloud APIs (OpenAI, Anthropic, Google, OpenRouter) for hosted models. | LLM |+| `llm.request_timeout` | `LDR_LLM_REQUEST_TIMEOUT` | `1800` | Per-operation HTTP timeout in seconds for LLM inference and embedding requests, applied to every supported provider. It bounds each connect, write and wait for the next response chunk individually; it is not a cap on the total duration of a response. A peer that keeps sending data can exceed this value, including for a non-streaming response. Connecting is separately capped at 5 seconds for OpenAI-compatible and Ollama clients; Anthropic inference uses this value for connecting too. There is no upper limit; values below 1 second are clamped to 1, and non-numeric or non-finite values use the default. Model discovery is unaffected: OpenAI-compatible providers and Anthropic use 30 seconds, while Google and Ollama retain provider-specific limits of 10 and 5 seconds, respectively. | LLM || `llm.requesty.api_key` | `LDR_LLM_REQUESTY_API_KEY` | `null` | API key to use for the Requesty provider. | SEARCH || `llm.require_local_endpoint` | `LDR_LLM_REQUIRE_LOCAL_ENDPOINT` | `false` | When enabled, the LLM provider's resolved endpoint must be on a local network (loopback, RFC1918, link-local). Cloud-only providers (OpenAI, Anthropic, Google, OpenRouter) are blocked. Independent of egress_scope. | LLM || `llm.supports_max_tokens` | `LDR_LLM_SUPPORTS_MAX_TOKENS` | `true` | Feature flag for max_tokens parameter. Disable if your LLM API returns errors when max_tokens is set. When disabled, the parameter is not passed to the API. | LLM |diff --git a/src/local_deep_research/defaults/default_settings.json b/src/local_deep_research/defaults/default_settings.jsonindex 7193add033..b3202ba8eb 100644--- a/src/local_deep_research/defaults/default_settings.json+++ b/src/local_deep_research/defaults/default_settings.json@@ -537,6 +537,34 @@"value": true,"visible": true},+ "llm.request_timeout": {+ "category": "llm_parameters",+ "description": "Per-operation HTTP timeout in seconds for LLM inference and embedding requests, applied to every supported provider. It bounds each connect, write and wait for the next response chunk individually; it is not a cap on the total duration of a response. A peer that keeps sending data can exceed this value, including for a non-streaming response. Connecting is separately capped at 5 seconds for OpenAI-compatible and Ollama clients; Anthropic inference uses this value for connecting too. There is no upper limit; values below 1 second are clamped to 1, and non-numeric or non-finite values use the default. Model discovery is unaffected: OpenAI-compatible providers and Anthropic use 30 seconds, while Google and Ollama retain provider-specific limits of 10 and 5 seconds, respectively.",+ "editable": true,+ "max_value": null,+ "min_value": 1,+ "name": "LLM Request Inactivity Timeout (seconds)",+ "options": null,+ "step": 1,+ "type": "LLM",+ "ui_element": "number",+ "value": 1800,+ "visible": true+ },+ "llm.max_retries": {+ "category": "llm_parameters",+ "description": "Maximum provider SDK retries per failed inference or embedding request. Applies to OpenAI-compatible and Anthropic SDKs; the Ollama client does not retry. Values outside 0-5 are clamped. Each retry starts another request with its own per-operation timeouts; SDK backoff or server-requested retry delays can further extend elapsed time. This setting does not impose a total request deadline. Model discovery SDK retries are disabled.",+ "editable": true,+ "max_value": 5,+ "min_value": 0,+ "name": "LLM Max Retries",+ "options": null,+ "step": 1,+ "type": "LLM",+ "ui_element": "number",+ "value": 2,+ "visible": true+ },"llm.model": {"category": "llm_general","description": "The language model to use. Available models depend on the selected provider (e.g., 'gpt-4o' for OpenAI, 'gemma:7b' for Ollama). You can type any model name supported by your provider.",diff --git a/src/local_deep_research/embeddings/providers/implementations/ollama.py b/src/local_deep_research/embeddings/providers/implementations/ollama.pyindex 868dac9622..94031b5950 100644--- a/src/local_deep_research/embeddings/providers/implementations/ollama.py+++ b/src/local_deep_research/embeddings/providers/implementations/ollama.py@@ -3,6 +3,7 @@import weakreffrom typing import Any, Dict, List, Optional+from httpx import Timeoutfrom langchain_ollama import OllamaEmbeddingsfrom langchain_core.embeddings import Embeddings@@ -85,6 +86,37 @@ class OllamaEmbeddingsProvider(BaseEmbeddingProvider):if num_ctx:ollama_kwargs["num_ctx"] = int(num_ctx)+ # Same bound as the LLM side: langchain_ollama otherwise builds+ # its httpx clients with ``timeout=None``, so a stalled embedding+ # request would hold an indexing worker forever. client_kwargs+ # reaches both the sync and the async ``ollama.Client``. Imported+ # lazily — the ``llm.providers`` package runs provider+ # auto-discovery at import time.+ from ....llm.providers._helpers import (+ build_httpx_timeout,+ resolve_request_timeout,+ )+ from ....llm.providers.implementations.ollama import OllamaProvider++ client_kwargs: Dict[str, Any] = {+ "timeout": build_httpx_timeout(+ resolve_request_timeout(settings_snapshot), Timeout+ )+ }++ # An authenticated Ollama instance sits behind the same proxy for+ # embeddings as for chat, and there is a single ``llm.ollama.api_key``+ # for both. OllamaEmbeddings has no ``headers`` field either, so the+ # header has to travel via client_kwargs exactly as it does on the+ # chat path — without this, chat authenticates and indexing 401s.+ auth_headers = OllamaProvider.build_bearer_header(+ settings_snapshot=settings_snapshot+ )+ if auth_headers:+ client_kwargs["headers"] = auth_headers++ ollama_kwargs["client_kwargs"] = client_kwargs+instance = OllamaEmbeddings(**ollama_kwargs)# Safety net for callers that bypass LocalEmbeddingManager (e.g.,diff --git a/src/local_deep_research/embeddings/providers/implementations/openai.py b/src/local_deep_research/embeddings/providers/implementations/openai.pyindex 8405148dca..9d38b3f3a9 100644--- a/src/local_deep_research/embeddings/providers/implementations/openai.py+++ b/src/local_deep_research/embeddings/providers/implementations/openai.py@@ -171,6 +171,32 @@ class OpenAIEmbeddingsProvider(BaseEmbeddingProvider):effective_chunk_size = _DEFAULT_CHUNK_SIZEparams["chunk_size"] = effective_chunk_size+ # Same bound as the LLM side: langchain otherwise forwards an+ # explicit ``timeout=None`` that overrides the SDK default, leaving+ # embedding reads unbounded, and the SDK's own 2 retries would+ # multiply the wait. Imported lazily — the ``llm.providers``+ # package runs provider auto-discovery at import time.+ from ....llm.providers._helpers import (+ build_httpx_timeout,+ resolve_max_retries,+ resolve_request_timeout,+ )++ # An SDK ``Timeout`` object, not the chat path's hashable tuple:+ # ``OpenAIEmbeddings`` never consults langchain's ``@lru_cache``'d+ # ``_get_default_httpx_client``, so every instance owns its httpx+ # clients whatever the timeout's type. The object keeps the SDK's+ # informational ``x-stainless-read-timeout`` header well formed+ # while the effective bounds stay identical. ``openai.Timeout`` is+ # the httpx2 flavour the SDK ships; ``httpx.Timeout`` is a+ # different class and would not be recognised.+ from openai import Timeout++ params["request_timeout"] = build_httpx_timeout(+ resolve_request_timeout(settings_snapshot), Timeout+ )+ params["max_retries"] = resolve_max_retries(settings_snapshot)+embeddings = OpenAIEmbeddings(**params)logger.info("Creating OpenAIEmbeddings with model={}, chunk_size={}",@@ -236,7 +262,7 @@ class OpenAIEmbeddingsProvider(BaseEmbeddingProvider):# even if an early statement (e.g. the openai import) raises.api_key = Nonetry:- from openai import OpenAI+ from openai import OpenAI, Timeoutapi_key = get_setting_from_snapshot("embeddings.openai.api_key",@@ -261,6 +287,19 @@ class OpenAIEmbeddingsProvider(BaseEmbeddingProvider):client_kwargs: Dict[str, Any] = {"api_key": api_key}if base_url:client_kwargs["base_url"] = normalize_url(base_url)+ # Bounded discovery request — mirrors the LLM-side provider.+ from ....llm.providers._helpers import (+ MODEL_DISCOVERY_TIMEOUT_SECONDS,+ build_httpx_timeout,+ )++ # Fresh client per call, outside langchain's cached factory —+ # see ``create_embeddings`` above for why the object form,+ # not the tuple, is right here.+ client_kwargs["timeout"] = build_httpx_timeout(+ MODEL_DISCOVERY_TIMEOUT_SECONDS, Timeout+ )+ client_kwargs["max_retries"] = 0client = OpenAI(**client_kwargs)models_response = client.models.list()diff --git a/src/local_deep_research/error_handling/openai_compat_errors.py b/src/local_deep_research/error_handling/openai_compat_errors.pyindex d0f90528f9..97b68685c7 100644--- a/src/local_deep_research/error_handling/openai_compat_errors.py+++ b/src/local_deep_research/error_handling/openai_compat_errors.py@@ -95,6 +95,28 @@ def _walk_cause(exc: BaseException) -> BaseException:return deepest+# The setting name is the label the UI actually renders (see+# ``defaults/default_settings.json``), not a Settings -> ... path, so a user+# can search for it. The connect bound is stated separately because+# ``_helpers.build_timeout`` caps connect at ``CONNECT_TIMEOUT_SECONDS``+# (``min(5, seconds)``), so raising this setting above 5 cannot move it:+# the openai SDK wraps a connect timeout in the same ``APITimeoutError``+# as a read timeout, so without this sentence the hint would tell a user+# with an unreachable endpoint to raise a knob that cannot help them.+# "at most" rather than "not affected by" because below 5 seconds the+# setting *is* the connect bound. The literal is pinned against the+# constant by ``tests/error_handling/test_openai_compat_timeout_hint.py``.+_TIMEOUT_HINT = (+ ' If the server needs longer, raise the "LLM Request Inactivity Timeout '+ '(seconds)" setting (`llm.request_timeout`, no upper limit), or set '+ "`LDR_LLM_REQUEST_TIMEOUT`. It limits individual network operations, "+ "not total response duration. A response that keeps sending data, SDK "+ "retries, and retry delays can make the complete request last longer. "+ "Connection attempts are bounded at most at 5 seconds, so raising it "+ "above that will not help if the endpoint cannot be reached at all."+)++_DOCKER_HINT = (" (from inside Docker, localhost is the container itself -- use ""host.docker.internal, the host IP, or run with --network=host to share "@@ -122,7 +144,7 @@ def _dispatch(return ("openai_timeout",f"{provider} at {base_url} did not respond in time. The server "- "may be loading a model or overloaded.",+ f"may be loading a model or overloaded.{_TIMEOUT_HINT}",)# Connection-refused / network-unreachable familydiff --git a/src/local_deep_research/llm/providers/_helpers.py b/src/local_deep_research/llm/providers/_helpers.pyindex ab2532cb27..cc93f58383 100644--- a/src/local_deep_research/llm/providers/_helpers.py+++ b/src/local_deep_research/llm/providers/_helpers.py@@ -6,10 +6,46 @@ provider-class ``create_llm`` methods are the single source of truth forLLM construction logic — no per-provider drift."""+import math+from typing import Any, Final++from ...security.secure_logging import logger++DEFAULT_LOCAL_CONTEXT_WINDOW_SIZE = 8192DEFAULT_CLOUD_CONTEXT_WINDOW_SIZE = 128000LOCAL_PROVIDERS = ("ollama", "llamacpp", "lmstudio")+# ``llm.request_timeout`` is a *per-network-operation* bound, not a cap on+# the total wall-clock duration of a response. httpx (which every provider+# SDK here sits on) has no total-deadline knob: a scalar expands to+# ``Timeout(connect=X, read=X, write=X, pool=X)`` where ``read`` is the+# time allowed *between* received bytes. A provider that keeps dripping+# data therefore keeps the request alive past X. See+# ``docs/CONFIGURATION.md`` — the setting is documented as an inactivity+# bound for exactly this reason.+DEFAULT_REQUEST_TIMEOUT: Final = 1800+MIN_REQUEST_TIMEOUT: Final = 1+# Deliberately no ceiling: a long-running local model on modest hardware+# can legitimately think for hours, and a cap that is generous enough for+# those is not a meaningful guard against a typo anyway. Non-finite values+# are still rejected below, which is what actually prevents an unbounded+# wait; MAX_MAX_RETRIES stays, because retries MULTIPLY this bound.++# Connecting is not inference: a blackholed IP should fail fast instead of+# holding a worker for the full inference budget. Matches the OpenAI and+# Anthropic SDK defaults (``Timeout(connect=5.0, ...)``), which a scalar+# timeout would otherwise overwrite.+CONNECT_TIMEOUT_SECONDS: Final = 5.0++# Limit the number of SDK retries. Each attempt has its own per-operation+# timeouts, with SDK backoff or server-directed delays between attempts.+DEFAULT_MAX_RETRIES: Final = 2+MIN_MAX_RETRIES: Final = 0+MAX_MAX_RETRIES: Final = 5++MODEL_DISCOVERY_TIMEOUT_SECONDS: Final = 30+def get_context_window_for_provider(provider_type, settings_snapshot=None):"""Resolve effective context window size for a provider.@@ -94,3 +130,234 @@ def compute_max_tokens(if context_window_size is not None:max_tokens = min(max_tokens, int(context_window_size * 0.8))return max_tokens+++def _get_setting_value(settings_snapshot, key: str, default: Any) -> Any:+ """Read ``key`` from the snapshot, degrading to ``default``.++ Every other setting reader on the LLM-construction path degrades to a+ default rather than raising; this one must too. A bad value here would+ otherwise take down every LLM construction in the process.+ """+ from ...config.thread_settings import (+ get_setting_from_snapshot,+ NoSettingsContextError,+ )++ try:+ return get_setting_from_snapshot(+ key, default, settings_snapshot=settings_snapshot+ )+ except (NoSettingsContextError, OverflowError):+ # Full-format number entries are coerced by the settings reader+ # before _clamp_number runs. An oversized integer can overflow there.+ return default+++def _raw_snapshot_value(settings_snapshot, key: str) -> Any:+ """Return the pre-coercion snapshot value for ``key`` (or ``None``)."""+ if not settings_snapshot:+ return None+ raw = settings_snapshot.get(key)+ if isinstance(raw, dict):+ return raw.get("value")+ return raw+++def _clamp_number(+ key: str,+ value: Any,+ *,+ default: float,+ minimum: float,+ maximum: float | None,+) -> float:+ """Coerce ``value`` to a finite float inside ``[minimum, maximum]``.++ ``maximum=None`` means no upper bound; the value is still required to be+ finite, so ``inf`` cannot pass as 'no timeout'.++ Unusable shapes fall back to ``default`` and out-of-range numbers are+ clamped — never raised. An operator typo in a single numeric field+ must not be a self-inflicted outage of every LLM call.+ """+ if value is None:+ return float(default)+ if isinstance(value, bool):+ logger.warning(+ "Setting {} is a boolean, which is not a valid number; using {}.",+ key,+ default,+ )+ return float(default)+ try:+ number = float(value)+ except (TypeError, ValueError, OverflowError):+ logger.warning(+ "Setting {} is not a usable number; using {}.", key, default+ )+ return float(default)+ if not math.isfinite(number):+ logger.warning("Setting {} is not finite; using {}.", key, default)+ return float(default)+ if number < minimum:+ logger.warning(+ "Setting {}={} is below the supported minimum; clamping to {}.",+ key,+ number,+ minimum,+ )+ return float(minimum)+ if maximum is not None and number > maximum:+ logger.warning(+ "Setting {}={} is above the supported maximum; clamping to {}.",+ key,+ number,+ maximum,+ )+ return float(maximum)+ return number+++def _resolve_bounded_number(+ settings_snapshot,+ key: str,+ *,+ default: float,+ minimum: float,+ maximum: float | None,+) -> float:+ """Read a numeric setting and force it into ``[minimum, maximum]``.++ ``maximum=None`` means no upper bound; the value is still required to+ be a finite number, so ``inf`` cannot slip through as 'no timeout'.+ """+ if isinstance(_raw_snapshot_value(settings_snapshot, key), bool):+ # A full-format snapshot entry would be coerced from ``True`` to+ # ``1`` by the number parser before we could see it, so the raw+ # stored value has to be inspected separately.+ logger.warning(+ "Setting {} is a boolean, which is not a valid number; using {}.",+ key,+ default,+ )+ return float(default)+ return _clamp_number(+ key,+ _get_setting_value(settings_snapshot, key, default),+ default=default,+ minimum=minimum,+ maximum=maximum,+ )+++def resolve_request_timeout(settings_snapshot=None) -> float:+ """Resolve the per-operation request timeout shared by all clients.++ The returned value bounds each individual network operation (connect,+ write, and the wait for the *next* response byte) — not the total+ duration of a response. Always returns a finite number of at least+ ``MIN_REQUEST_TIMEOUT``; there is no upper bound.+ """+ return _resolve_bounded_number(+ settings_snapshot,+ "llm.request_timeout",+ default=DEFAULT_REQUEST_TIMEOUT,+ minimum=MIN_REQUEST_TIMEOUT,+ maximum=None,+ )+++def resolve_max_retries(settings_snapshot=None) -> int:+ """Resolve the bounded SDK retry count shared by all clients.++ Retries multiply the number of attempts, so this is capped at+ ``MAX_MAX_RETRIES``. Each attempt has its own per-operation timeouts;+ SDK backoff and server-directed delays also affect total elapsed time.+ """+ return int(+ _resolve_bounded_number(+ settings_snapshot,+ "llm.max_retries",+ default=DEFAULT_MAX_RETRIES,+ minimum=MIN_MAX_RETRIES,+ maximum=MAX_MAX_RETRIES,+ )+ )+++def build_timeout(seconds: float) -> tuple[float, float, float, float]:+ """Expand a scalar bound into a hashable per-operation timeout tuple.++ Returns ``(connect, read, write, pool)``, the 4-tuple form httpx+ accepts wherever it accepts a ``Timeout`` object+ (``httpx._config.Timeout.__init__`` unpacks a tuple in exactly this+ order). The effective client timeout is identical either way:+ ``Timeout(connect=min(5, X), read=X, write=X, pool=X)``. The connect+ slot is bounded *at most* at 5 seconds, not pinned to it — below a+ 5-second ``seconds`` the ``min`` yields ``seconds``, so raising the+ setting from 2 to 4 does raise the connect bound as well.++ **For client construction only.** ``openai._models+ .FinalRequestOptions.timeout`` is pydantic-validated and rejects a+ tuple, so a per-request ``timeout=`` keyword (e.g.+ ``client.models.list(timeout=...)``) must use ``build_httpx_timeout``+ instead. Only the client-level ``timeout=`` is untyped.++ The openai SDK also stringifies whatever it is given into the+ informational ``x-stainless-read-timeout`` request header, so the+ tuple's repr rides along on the clients that take it. The effective+ bounds are unaffected; the header is not read by this codebase.++ **The tuple shape is load-bearing, not cosmetic.** ``langchain_openai``+ builds ``ChatOpenAI``'s httpx clients through an ``@lru_cache``'d+ factory and silently falls back to a per-instance client when the+ timeout is unhashable — ``_client_utils._get_default_httpx_client``:+ *"Uses cached client unless timeout is ``httpx.Timeout``, which is not+ hashable"*. ``httpx.Timeout`` defines ``__eq__`` without ``__hash__``,+ so handing a ``Timeout`` object to ``ChatOpenAI(request_timeout=...)``+ would give every LLM its own connection pool. That would invert the+ invariants recorded in ``utilities/llm_utils.py`` (``_close_base_llm``+ deliberately does *not* close ChatOpenAI/ChatAnthropic clients because+ they are shared) and in ``docs/developing/resource-cleanup.md``. With+ the tuple those stay true, so neither needs changing.++ Sharing is also why sync entry points must keep calling the sync+ ``invoke``: the cached *async* client is bound to the event loop that+ created it, so a second ``asyncio.run()`` in the same process reuses a+ client whose loop is closed (issue #6293). That is a property of+ langchain's cache, not of this bound — an unhashable timeout only+ masked it by accident.++ Passing a bare scalar instead would set ``connect`` to the same large+ value, so a TCP connect to a blackholed address would hold a worker+ for the whole inference budget. A 2-tuple is not an option either: it+ leaves ``write`` and ``pool`` at ``None`` (unbounded), so all four+ slots are always filled.+ """+ bounded = float(seconds)+ return (min(CONNECT_TIMEOUT_SECONDS, bounded), bounded, bounded, bounded)+++def build_httpx_timeout(seconds: float, timeout_cls):+ """``build_timeout`` as the SDK's own ``Timeout`` object.++ For every client whose httpx client is per-instance regardless of+ hashability, which is all of them except ``ChatOpenAI``:+ ``ollama.Client`` is constructed once per ``ChatOllama``+ /``OllamaEmbeddings``; ``OpenAIEmbeddings`` never consults+ ``_get_default_httpx_client`` at all; and the model-discovery clients+ are fresh SDK instances built per call. Only+ ``ChatOpenAI`` routes through the ``@lru_cache``'d factory, so only+ there does the tuple of ``build_timeout`` buy anything.++ Use the target SDK's exported type — ``anthropic.Timeout``,+ ``openai.Timeout``, or ``httpx.Timeout`` for Ollama — to match its+ native transport. This is not interchangeable: Anthropic and Ollama+ ship ``httpx`` while openai ships ``httpx2``, and the two ``Timeout``+ classes do not recognise each other. Handing an ``httpx.Timeout`` to+ ``openai.OpenAI(timeout=...)`` is silently treated as a scalar, which+ nests it into all four slots of an ``httpx2.Timeout``.+ """+ connect, read, write, pool = build_timeout(seconds)+ return timeout_cls(connect=connect, read=read, write=write, pool=pool)diff --git a/src/local_deep_research/llm/providers/implementations/anthropic.py b/src/local_deep_research/llm/providers/implementations/anthropic.pyindex c20b1f6057..0f85c7151b 100644--- a/src/local_deep_research/llm/providers/implementations/anthropic.py+++ b/src/local_deep_research/llm/providers/implementations/anthropic.py@@ -10,6 +10,12 @@ from ....security.secure_logging import loggerfrom ..base import OPTIONAL_API_KEY_PLACEHOLDER, Exposurefrom ..openai_base import OpenAICompatibleProviderfrom ....security.ssrf_validator import assert_base_url_safe+from .._helpers import (+ MODEL_DISCOVERY_TIMEOUT_SECONDS,+ build_httpx_timeout,+ resolve_max_retries,+ resolve_request_timeout,+)class AnthropicProvider(OpenAICompatibleProvider):@@ -99,6 +105,14 @@ class AnthropicProvider(OpenAICompatibleProvider):except NoSettingsContextError:pass # Optional parameter+ # ``ChatAnthropic.default_request_timeout`` is typed ``float | None``+ # and its httpx client comes from a cached factory, so — unlike the+ # OpenAI-flavoured providers — a per-operation ``Timeout`` object+ # cannot be injected here. The scalar still bounds every operation+ # (connect included), just without the shorter connect cap.+ anthropic_params["timeout"] = resolve_request_timeout(settings_snapshot)+ anthropic_params["max_retries"] = resolve_max_retries(settings_snapshot)+# Operator-configurable base_url for self-hosted Anthropic-format# endpoints. No-op for the official cloud provider, whose url_setting# is None (so it always talks to api.anthropic.com via the SDK@@ -170,7 +184,7 @@ class AnthropicProvider(OpenAICompatibleProvider):)return []- from anthropic import Anthropic+ from anthropic import Anthropic, Timeout as AnthropicTimeout# Pass the key explicitly (placeholder when keyless) so the SDK does# not read ANTHROPIC_API_KEY from the environment and ship a real@@ -178,6 +192,10 @@ class AnthropicProvider(OpenAICompatibleProvider):client = Anthropic(api_key=api_key or OPTIONAL_API_KEY_PLACEHOLDER,base_url=base_url or None,+ timeout=build_httpx_timeout(+ MODEL_DISCOVERY_TIMEOUT_SECONDS, AnthropicTimeout+ ),+ max_retries=0,)logger.debug(f"Fetching models from {cls.provider_name}")diff --git a/src/local_deep_research/llm/providers/implementations/ollama.py b/src/local_deep_research/llm/providers/implementations/ollama.pyindex 0f774e7e45..cd07b19fd9 100644--- a/src/local_deep_research/llm/providers/implementations/ollama.py+++ b/src/local_deep_research/llm/providers/implementations/ollama.py@@ -1,6 +1,7 @@"""Ollama LLM provider for Local Deep Research."""import requests+from httpx import Timeoutfrom langchain_ollama import ChatOllamafrom ....config.thread_settings import get_setting_from_snapshot@@ -156,18 +157,16 @@ class OllamaProvider(BaseLLMProvider):"temperature": temperature,}- # Add authentication headers if configured- headers = cls._get_auth_headers(settings_snapshot=settings_snapshot)- if headers:- # ChatOllama supports auth via headers parameter- ollama_params["headers"] = headers-# Resolve the context window through the shared helper so num_ctx# always matches the context_limit reported for overflow detection# (wrap_llm_without_think_tags uses the same resolution). No# max_tokens kwarg: ChatOllama ignores it (its output-length control# is num_predict), so passing it would only feign a cap.- from .._helpers import get_context_window_for_provider+ from .._helpers import (+ build_httpx_timeout,+ get_context_window_for_provider,+ resolve_request_timeout,+ )context_window_size = get_context_window_for_provider("ollama", settings_snapshot=settings_snapshot@@ -189,6 +188,23 @@ class OllamaProvider(BaseLLMProvider):if isinstance(enable_thinking, bool):ollama_params["reasoning"] = enable_thinking+ # ChatOllama forwards client_kwargs to both the sync and the async+ # ``ollama.Client``, which pass them straight to ``httpx.Client``.+ # Auth headers go here too: ChatOllama has no ``headers`` field, so+ # a top-level ``headers=`` kwarg was silently dropped and the+ # Authorization header never reached the wire.+ client_kwargs = {+ "timeout": build_httpx_timeout(+ resolve_request_timeout(settings_snapshot), Timeout+ )+ }+ auth_headers = cls._get_auth_headers(+ settings_snapshot=settings_snapshot+ )+ if auth_headers:+ client_kwargs["headers"] = auth_headers+ ollama_params["client_kwargs"] = client_kwargs+llm = ChatOllama(**ollama_params)# Log the actual client configuration after creationdiff --git a/src/local_deep_research/llm/providers/implementations/openai.py b/src/local_deep_research/llm/providers/implementations/openai.pyindex f6c4b0f4c3..4b19b60678 100644--- a/src/local_deep_research/llm/providers/implementations/openai.py+++ b/src/local_deep_research/llm/providers/implementations/openai.py@@ -1,6 +1,7 @@"""OpenAI LLM provider for Local Deep Research."""from langchain_openai import ChatOpenAI+from ....security.secure_logging import logger# get_setting_from_snapshot and NoSettingsContextError are imported inside@@ -11,6 +12,11 @@ from ....security.secure_logging import loggerfrom ....security.ssrf_validator import assert_base_url_safefrom ..base import Exposurefrom ..openai_base import OpenAICompatibleProvider+from .._helpers import (+ build_timeout,+ resolve_max_retries,+ resolve_request_timeout,+)class OpenAIProvider(OpenAICompatibleProvider):@@ -114,20 +120,6 @@ class OpenAIProvider(OpenAICompatibleProvider):settings_snapshot,)- _get_optional_setting(- openai_params,- "max_retries",- "llm.max_retries",- settings_snapshot,- )-- _get_optional_setting(- openai_params,- "request_timeout",- "llm.request_timeout",- settings_snapshot,- )-# Apply context-window-aware max_tokens cap (was previously only# applied in dead code in llm_config.get_llm).from .._helpers import (@@ -148,6 +140,15 @@ class OpenAIProvider(OpenAICompatibleProvider):except NoSettingsContextError:pass # Optional parameter+ # Bounded per-operation timeout and retry count (see+ # ``_helpers.build_timeout``). Both are always set: langchain+ # otherwise forwards an explicit ``timeout=None`` that overrides+ # the SDK's own default, leaving reads unbounded.+ openai_params["request_timeout"] = build_timeout(+ resolve_request_timeout(settings_snapshot)+ )+ openai_params["max_retries"] = resolve_max_retries(settings_snapshot)+logger.info(f"Creating {cls.provider_name} LLM with model: {model_name}, "f"temperature: {temperature}"diff --git a/src/local_deep_research/llm/providers/openai_base.py b/src/local_deep_research/llm/providers/openai_base.pyindex edf536e7aa..72a6cc3cae 100644--- a/src/local_deep_research/llm/providers/openai_base.py+++ b/src/local_deep_research/llm/providers/openai_base.py@@ -1,6 +1,7 @@"""Base OpenAI-compatible endpoint provider for Local Deep Research."""from langchain_openai import ChatOpenAI+from ...security.secure_logging import logger# get_setting_from_snapshot and NoSettingsContextError are imported inside@@ -11,6 +12,13 @@ from ...security.secure_logging import loggerfrom ...security.log_sanitizer import redact_secretsfrom ...security.ssrf_validator import assert_base_url_safefrom ...utilities.url_utils import normalize_url+from ._helpers import (+ MODEL_DISCOVERY_TIMEOUT_SECONDS,+ build_httpx_timeout,+ build_timeout,+ resolve_max_retries,+ resolve_request_timeout,+)from .base import BaseLLMProvider@@ -122,21 +130,13 @@ class OpenAICompatibleProvider(BaseLLMProvider):settings_snapshot,)- # Add max_retries if specified- _get_optional_setting(- llm_params,- "max_retries",- "llm.max_retries",- settings_snapshot,- )-- # Add request_timeout if specified- _get_optional_setting(- llm_params,- "request_timeout",- "llm.request_timeout",- settings_snapshot,+ # Bounded per-operation timeout and retry count. Both are always+ # set: langchain otherwise forwards an explicit ``timeout=None``+ # that overrides the SDK's own default, leaving reads unbounded.+ llm_params["request_timeout"] = build_timeout(+ resolve_request_timeout(settings_snapshot))+ llm_params["max_retries"] = resolve_max_retries(settings_snapshot)# Request usage stats on streamed responses (stream_options.# include_usage). Opt-in via subclass kwargs because some@@ -215,6 +215,11 @@ class OpenAICompatibleProvider(BaseLLMProvider):except NoSettingsContextError:pass+ llm_params["request_timeout"] = build_timeout(+ resolve_request_timeout(settings_snapshot)+ )+ llm_params["max_retries"] = resolve_max_retries(settings_snapshot)+return ChatOpenAI(**llm_params)@classmethod@@ -319,7 +324,7 @@ class OpenAICompatibleProvider(BaseLLMProvider):# Use a dummy key for providers that don't require authapi_key = api_key or "dummy-key-for-models-list"- from openai import OpenAI+ from openai import OpenAI, Timeout# Use provided base_url or fall back to class defaultif not base_url:@@ -345,8 +350,20 @@ class OpenAICompatibleProvider(BaseLLMProvider):)return []- # Create OpenAI client (uses library defaults for timeout)- client = OpenAI(api_key=api_key, base_url=base_url)+ # A fresh client per call, never routed through langchain's+ # cached httpx factory, so the hashable tuple buys nothing+ # here — the SDK's own ``Timeout`` object keeps the+ # informational ``x-stainless-read-timeout`` header well+ # formed. ``openai.Timeout`` is the httpx2 flavour the SDK+ # ships; ``httpx.Timeout`` would not be recognised.+ client = OpenAI(+ api_key=api_key,+ base_url=base_url,+ timeout=build_httpx_timeout(+ MODEL_DISCOVERY_TIMEOUT_SECONDS, Timeout+ ),+ max_retries=0,+ )# Fetch modelslogger.debug(diff --git a/tests/embeddings/test_embedding_request_timeout.py b/tests/embeddings/test_embedding_request_timeout.pynew file mode 100644index 0000000000..afcf3c8a00--- /dev/null+++ b/tests/embeddings/test_embedding_request_timeout.py@@ -0,0 +1,232 @@+"""``llm.request_timeout`` / ``llm.max_retries`` also bound embeddings.++The setting documents itself as covering "LLM inference and embedding+requests ... applied to every supported provider". Before this, both+embedding clients were built with ``Timeout(timeout=None)`` — langchain+passes an explicit ``None`` that overrides the SDK's own default — so a+stalled embedding call held an indexing worker forever, and the OpenAI+model-discovery client had no timeout or retry cap at all.+"""++from unittest.mock import MagicMock, patch++import openai+import pytest++from local_deep_research.embeddings.providers.implementations.ollama import (+ OllamaEmbeddingsProvider,+)+from local_deep_research.embeddings.providers.implementations.openai import (+ OpenAIEmbeddingsProvider,+)+from local_deep_research.llm.providers._helpers import (+ MODEL_DISCOVERY_TIMEOUT_SECONDS,+)++_OLLAMA_MODULE = (+ "local_deep_research.embeddings.providers.implementations.ollama"+)+_OPENAI_MODULE = (+ "local_deep_research.embeddings.providers.implementations.openai"+)+++def _settings(overrides=None):+ values = {+ "embeddings.ollama.model": "nomic-embed-text",+ "embeddings.ollama.num_ctx": 8192,+ "embeddings.openai.model": "text-embedding-3-small",+ "embeddings.openai.api_key": "test-key",+ }+ values.update(overrides or {})+ return values+++def _side_effect(values):+ def _get(key, default=None, *args, **kwargs):+ return values.get(key, default)++ return _get+++class TestOllamaEmbeddingsTimeout:+ @pytest.mark.parametrize(+ ("configured", "expected"),+ # (None -> the default; 0 -> clamped up to the 1s floor; 10**9 ->+ # honoured, since the upper bound was removed deliberately.)+ [(None, 1800.0), (37, 37.0), (0, 1.0), (10**9, 1.0e9)],+ )+ def test_client_kwargs_carry_a_bounded_timeout(self, configured, expected):+ # Given+ values = _settings(+ {} if configured is None else {"llm.request_timeout": configured}+ )++ # When+ with patch(+ f"{_OLLAMA_MODULE}.get_setting_from_snapshot",+ side_effect=_side_effect(values),+ ):+ with patch(+ f"{_OLLAMA_MODULE}.get_ollama_base_url",+ return_value="http://localhost:11434",+ ):+ with patch(f"{_OLLAMA_MODULE}.OllamaEmbeddings") as mock_cls:+ OllamaEmbeddingsProvider.create_embeddings(+ settings_snapshot=values+ )++ # Then+ timeout = mock_cls.call_args[1]["client_kwargs"]["timeout"]+ assert timeout.read == expected+ assert timeout.write == expected+ assert timeout.pool == expected+ # Literal, not the imported constant: raising CONNECT_TIMEOUT_SECONDS+ # is a behaviour change and must break a test, not ride along.+ assert timeout.connect == min(5.0, expected)+++class TestOpenAIEmbeddingsTimeout:+ def test_request_timeout_and_retries_are_bounded(self):+ # Given+ values = _settings({"llm.request_timeout": 45, "llm.max_retries": 1})++ # When+ with patch(+ f"{_OPENAI_MODULE}.get_setting_from_snapshot",+ side_effect=_side_effect(values),+ ):+ with patch(+ "langchain_openai.OpenAIEmbeddings", return_value=MagicMock()+ ) as mock_cls:+ OpenAIEmbeddingsProvider.create_embeddings(+ settings_snapshot=values+ )++ # Then+ kwargs = mock_cls.call_args[1]+ # An SDK ``Timeout`` object, not the chat path's hashable tuple:+ # ``OpenAIEmbeddings`` never consults langchain's ``@lru_cache``'d+ # ``_get_default_httpx_client``, so each instance owns its httpx+ # clients whatever the timeout's type, and the object keeps the+ # SDK's ``x-stainless-read-timeout`` header well formed.+ # ``openai.Timeout`` is the httpx flavour the openai SDK ships+ # (httpx2). Reverting this call site to ``build_timeout`` fails+ # the isinstance assertion. See ``_helpers.build_httpx_timeout``.+ timeout = kwargs["request_timeout"]+ assert isinstance(timeout, openai.Timeout)+ assert (timeout.connect, timeout.read, timeout.write, timeout.pool) == (+ 5.0,+ 45,+ 45,+ 45,+ )+ assert kwargs["max_retries"] == 1++ def test_model_discovery_is_bounded_without_sdk_retries(self):+ # Given+ values = _settings({"embeddings.openai.base_url": None})+ fake_client = MagicMock()+ fake_client.models.list.return_value = MagicMock(data=[])++ # When+ with patch(+ f"{_OPENAI_MODULE}.get_setting_from_snapshot",+ side_effect=_side_effect(values),+ ):+ with patch(+ "openai.OpenAI", return_value=fake_client+ ) as mock_openai:+ OpenAIEmbeddingsProvider.get_available_models(+ settings_snapshot=values+ )++ # Then+ kwargs = mock_openai.call_args.kwargs+ # Fresh client per call, outside langchain's cached factory, so+ # the SDK's own ``Timeout`` object rather than the chat path's+ # tuple. Literals, not the imported constant: changing the+ # discovery budget is a behaviour change and must break a test+ # rather than ride along on the import.+ timeout = kwargs["timeout"]+ assert isinstance(timeout, openai.Timeout)+ assert (timeout.connect, timeout.read, timeout.write, timeout.pool) == (+ 5.0,+ 30,+ 30,+ 30,+ )+ assert MODEL_DISCOVERY_TIMEOUT_SECONDS == 30+ assert kwargs["max_retries"] == 0+++class TestOllamaEmbeddingsAuthHeaders:+ """The chat path routes ``llm.ollama.api_key`` through client_kwargs.++ Embeddings talk to the same authenticated Ollama instance through the+ same single key setting, so indexing used to 401 while chat worked.+ ``OllamaEmbeddings`` has no ``headers`` field either, so client_kwargs+ is the only route. Dropping the ``build_bearer_header`` block in+ ``embeddings/providers/implementations/ollama.py`` fails both tests+ below.+ """++ def _create(self, values, *, patch_cls):+ """``build_bearer_header`` reads the snapshot dict directly."""+ with patch(+ f"{_OLLAMA_MODULE}.get_setting_from_snapshot",+ side_effect=_side_effect(values),+ ):+ with patch(+ f"{_OLLAMA_MODULE}.get_ollama_base_url",+ return_value="http://localhost:11434",+ ):+ if not patch_cls:+ return OllamaEmbeddingsProvider.create_embeddings(+ settings_snapshot=values+ )+ with patch(f"{_OLLAMA_MODULE}.OllamaEmbeddings") as mock_cls:+ OllamaEmbeddingsProvider.create_embeddings(+ settings_snapshot=values+ )+ return mock_cls.call_args[1]++ def test_the_configured_key_becomes_a_bearer_header(self):+ # Given+ values = _settings({"llm.ollama.api_key": "secret-token"})++ # When+ kwargs = self._create(values, patch_cls=True)++ # Then+ assert kwargs["client_kwargs"]["headers"] == {+ "Authorization": "Bearer secret-token"+ }+ assert "headers" not in kwargs++ def test_no_key_configured_adds_no_header(self):+ # Given — bare local Ollama needs no auth+ values = _settings()++ # When+ kwargs = self._create(values, patch_cls=True)++ # Then+ assert "headers" not in kwargs["client_kwargs"]++ def test_the_header_reaches_both_real_httpx_clients(self):+ """End-to-end against the real OllamaEmbeddings, no network."""+ # Given+ values = _settings({"llm.ollama.api_key": "secret-token"})++ # When+ instance = self._create(values, patch_cls=False)++ # Then+ for client in (instance._client, instance._async_client):+ assert (+ client._client.headers.get("authorization")+ == "Bearer secret-token"+ )+ assert client._client.timeout.read == 1800+ assert client._client.timeout.connect == 5.0diff --git a/tests/error_handling/test_openai_compat_timeout_hint.py b/tests/error_handling/test_openai_compat_timeout_hint.pynew file mode 100644index 0000000000..567a2b1937--- /dev/null+++ b/tests/error_handling/test_openai_compat_timeout_hint.py@@ -0,0 +1,157 @@+"""The timeout hint must name a real control and not misdirect.++The hint is the only user-facing text that explains what to do about a+timeout, and nothing pinned it before: deleting it, or restoring the+earlier ``Settings -> LLM -> Request timeout`` wording, left every other+test green. These tests fail on either.++No network: ``openai`` and ``httpx`` exceptions are constructed directly.+"""++import httpx+import pytest+from openai import APIConnectionError, APITimeoutError++from local_deep_research.error_handling.openai_compat_errors import (+ friendly_openai_compatible_error,+)+from local_deep_research.llm.providers._helpers import CONNECT_TIMEOUT_SECONDS+from local_deep_research.settings.manager import SettingsManager+++def _req():+ return httpx.Request("POST", "http://localhost:8000/v1/chat/completions")+++def _message(exc):+ return friendly_openai_compatible_error(+ exc,+ provider="vllm",+ base_url="http://localhost:8000/v1",+ model="llama-3-8b",+ )+++def _chain(outer, *causes):+ current = outer+ for cause in causes:+ current.__cause__ = cause+ current = cause+ return outer++[email protected](scope="module")+def ui_label():+ """The label the settings UI actually renders for the knob."""+ defaults = SettingsManager(db_session=None).default_settings+ return defaults["llm.request_timeout"]["name"]+++class TestReadTimeoutHint:+ def test_the_hint_names_the_label_the_ui_renders(self, ui_label):+ """A user has to be able to find the control by that name."""+ # Given+ exc = _chain(+ APITimeoutError(request=_req()),+ httpx.ReadTimeout("timed out"),+ )++ # When+ message = _message(exc)++ # Then+ assert ui_label == "LLM Request Inactivity Timeout (seconds)"+ assert ui_label in message++ def test_the_hint_names_the_setting_key_and_environment_variable(self):+ # Given+ exc = _chain(+ APITimeoutError(request=_req()),+ httpx.ReadTimeout("timed out"),+ )++ # When+ message = _message(exc)++ # Then+ assert "llm.request_timeout" in message+ assert "LDR_LLM_REQUEST_TIMEOUT" in message++ def test_no_settings_navigation_path_is_invented(self):+ """``Settings -> LLM -> Request timeout`` is not a real path.++ Restoring that wording fails here.+ """+ # Given+ exc = _chain(+ APITimeoutError(request=_req()),+ httpx.ReadTimeout("timed out"),+ )++ # When+ message = _message(exc)++ # Then+ assert "Request timeout" not in message+ assert "Settings ->" not in message+++class TestConnectTimeoutIsNotMisdirected:+ """A connect timeout arrives as ``APITimeoutError`` too.++ ``openai._base_client`` wraps every ``httpx.TimeoutException`` —+ ``ConnectTimeout`` included — in ``APITimeoutError``, so it takes the+ same branch as a read timeout. Without the connect sentence the hint+ tells a user whose endpoint is unreachable to raise a setting that+ cannot affect them. Deleting that sentence fails both tests here.+ """++ def _connect_timeout(self):+ return _chain(+ APITimeoutError(request=_req()),+ httpx.ConnectTimeout("timed out"),+ )++ def test_the_user_is_told_the_setting_does_not_move_the_connect_bound(+ self,+ ):+ # Given+ exc = self._connect_timeout()++ # When+ message = _message(exc)++ # Then+ assert "Error type: openai_timeout" in message+ assert "Connection attempts are bounded at most at 5 seconds" in message+ assert "will not help if the endpoint cannot be reached" in message++ def test_the_stated_connect_bound_matches_the_shipped_constant(self):+ """Raising CONNECT_TIMEOUT_SECONDS must break this, not go unsaid."""+ # Given+ exc = self._connect_timeout()++ # When+ message = _message(exc)++ # Then+ assert CONNECT_TIMEOUT_SECONDS == 5.0+ assert f"{CONNECT_TIMEOUT_SECONDS:g} seconds" in message+++class TestNonTimeoutFailures:+ def test_a_refused_connection_gets_no_timeout_hint(self):+ """Nothing about a raisable timeout when nothing timed out."""+ # Given+ exc = _chain(+ APIConnectionError(message="Connection error.", request=_req()),+ httpx.ConnectError("All connection attempts failed"),+ )++ # When+ message = _message(exc)++ # Then+ assert "Error type: openai_connection_refused" in message+ assert "llm.request_timeout" not in message+ assert "LDR_LLM_REQUEST_TIMEOUT" not in messagediff --git a/tests/llm/test_provider_adapter_contracts.py b/tests/llm/test_provider_adapter_contracts.pyindex 4e2c779893..9bcf9c46b8 100644--- a/tests/llm/test_provider_adapter_contracts.py+++ b/tests/llm/test_provider_adapter_contracts.py@@ -218,6 +218,10 @@ class TestCredentialsAtConstruction:as ``Authorization: Bearer <sentinel>``, and the log really didcapture this call (the base_url line is present). Only then is theabsence of the sentinel meaningful.++ ChatOllama forwards client_kwargs to both HTTP clients. Its model+ ignores unsupported top-level headers, so the positive control must+ inspect the shared transport options."""from local_deep_research.llm.providers.implementations.ollama import (OllamaProvider,@@ -238,7 +242,10 @@ class TestCredentialsAtConstruction:kwargs = chat.call_args.kwargs# positive control (a): the credential reached the client.- assert kwargs["headers"] == {"Authorization": f"Bearer {FAKE_KEY}"}+ assert kwargs["client_kwargs"]["headers"] == {+ "Authorization": f"Bearer {FAKE_KEY}"+ }+ assert "headers" not in kwargs# positive control (b): this call's logging reached the sink.assert "http://localhost:11434" in captured_logs.text# the property under test.diff --git a/tests/llm/test_provider_base_url_ssrf.py b/tests/llm/test_provider_base_url_ssrf.pyindex 10be86b2d3..250c17c2c6 100644--- a/tests/llm/test_provider_base_url_ssrf.py+++ b/tests/llm/test_provider_base_url_ssrf.py@@ -353,7 +353,13 @@ class TestOpenAICompatibleListModelsSsrf:return Mock(data=[])class _FakeClient:- def __init__(self, api_key=None, base_url=None):+ def __init__(+ self,+ api_key=None,+ base_url=None,+ timeout=None,+ max_retries=None,+ ):captured["base_url"] = base_urlself.models = _FakeModels()diff --git a/tests/llm_providers/implementations/test_anthropic_provider.py b/tests/llm_providers/implementations/test_anthropic_provider.pyindex 9865fe6c7f..b4743e8004 100644--- a/tests/llm_providers/implementations/test_anthropic_provider.py+++ b/tests/llm_providers/implementations/test_anthropic_provider.py@@ -262,6 +262,31 @@ class TestAnthropicListModels:_, kwargs = mk.call_argsassert kwargs.get("base_url") is None+ def test_model_discovery_uses_fixed_timeout_without_sdk_retries(self):+ # Given+ snapshot = {+ "llm.anthropic.api_key": "sk-test",+ "llm.request_timeout": 1800,+ }+ fake_client = Mock()+ fake_client.models.list.return_value = Mock(data=[])++ # When+ with patch(+ "anthropic.Anthropic", return_value=fake_client+ ) as mock_anthropic:+ AnthropicProvider.list_models(settings_snapshot=snapshot)++ # Then+ call_kwargs = mock_anthropic.call_args.kwargs+ # Per-operation timeouts, with connect capped independently so a+ # blackholed endpoint fails fast instead of holding the worker.+ assert call_kwargs["timeout"].read == 30+ assert call_kwargs["timeout"].write == 30+ assert call_kwargs["timeout"].pool == 30+ assert call_kwargs["timeout"].connect == 5.0+ assert call_kwargs["max_retries"] == 0+def test_cloud_no_base_url_does_not_short_circuit(self):"""Unlike the custom-endpoint subclass, the cloud provider hasurl_setting=None, so a missing base_url must NOT return [] — it mustdiff --git a/tests/llm_providers/implementations/test_openai_provider.py b/tests/llm_providers/implementations/test_openai_provider.pyindex 56a99fa768..2c4ad139a1 100644--- a/tests/llm_providers/implementations/test_openai_provider.py+++ b/tests/llm_providers/implementations/test_openai_provider.py@@ -278,7 +278,10 @@ class TestOpenAICreateLLM:OpenAIProvider.create_llm(model_name="test-model")call_kwargs = mock_chat.call_args[1]- assert call_kwargs["request_timeout"] == 120+ # Per-operation timeouts as a hashable+ # ``(connect, read, write, pool)`` tuple; connect stays+ # short so a blackholed endpoint fails fast.+ assert call_kwargs["request_timeout"] == (5.0, 120, 120, 120)def test_create_llm_passes_api_base_when_set(self):"""Passes custom API base URL when configured."""diff --git a/tests/llm_providers/test_anthropic_high_value.py b/tests/llm_providers/test_anthropic_high_value.pyindex bda3496953..ecfd8b262c 100644--- a/tests/llm_providers/test_anthropic_high_value.py+++ b/tests/llm_providers/test_anthropic_high_value.py@@ -279,11 +279,17 @@ class TestAnthropicChatConstruction:AnthropicProvider.create_llm(model_name="test-model")def test_only_expected_params_passed_without_max_tokens(self):- """Without max_tokens, only model, anthropic_api_key, temperature should be passed."""+ """Without max_tokens, only model, anthropic_api_key, temperature, timeout should be passed."""with patch(GET_SETTING, side_effect=_make_setting_side_effect()):with patch(CHAT_ANTHROPIC) as mock_chat:mock_chat.return_value = Mock()AnthropicProvider.create_llm(model_name="test-model")call_kwargs = mock_chat.call_args[1]- expected_keys = {"model", "anthropic_api_key", "temperature"}+ expected_keys = {+ "model",+ "anthropic_api_key",+ "temperature",+ "timeout",+ "max_retries",+ }assert set(call_kwargs.keys()) == expected_keysdiff --git a/tests/llm_providers/test_openai_base.py b/tests/llm_providers/test_openai_base.pyindex b60133fcf8..e188edd341 100644--- a/tests/llm_providers/test_openai_base.py+++ b/tests/llm_providers/test_openai_base.py@@ -2,6 +2,7 @@Tests for OpenAI-compatible base provider."""+import openaiimport pytestfrom unittest.mock import Mock, patch@@ -267,6 +268,46 @@ class TestOpenAICompatibleListModels:assert result[0]["value"] == "gpt-4"assert result[1]["value"] == "gpt-3.5-turbo"+ def test_model_discovery_uses_fixed_timeout_without_sdk_retries(+ self, mock_openai_client+ ):+ # Given+ snapshot = {+ "llm.openai_endpoint.api_key": "test-key",+ "llm.request_timeout": 1800,+ }++ # When+ with patch(+ "openai.OpenAI", return_value=mock_openai_client+ ) as mock_openai:+ OpenAICompatibleProvider.list_models(settings_snapshot=snapshot)++ # Then+ call_kwargs = mock_openai.call_args.kwargs+ timeout = call_kwargs["timeout"]+ # An SDK ``Timeout`` object, not the chat path's hashable tuple:+ # this client is built fresh per call and never reaches+ # langchain's cached httpx factory, so the tuple would buy+ # nothing and would only put its repr in the SDK's+ # ``x-stainless-read-timeout`` header. Reverting the call site to+ # ``build_timeout`` fails the isinstance assertion below.+ # ``openai.Timeout`` is the httpx flavour the openai SDK ships+ # (httpx2); an ``httpx.Timeout`` would not be recognised by it.+ assert isinstance(timeout, openai.Timeout)+ assert (timeout.connect, timeout.read, timeout.write, timeout.pool) == (+ 5.0,+ 30,+ 30,+ 30,+ )+ # ``MODEL_DISCOVERY_TIMEOUT_SECONDS`` is an int, and+ # ``build_timeout`` is annotated ``tuple[float, ...]``. Ints+ # compare equal to the floats above, so the type is asserted:+ # dropping the ``float()`` coercion fails here.+ assert type(timeout.read) is float+ assert call_kwargs["max_retries"] == 0+def test_list_models_for_api_error_handling(self):"""Returns empty list on error."""with patch("openai.OpenAI") as mock_openai:diff --git a/tests/llm_providers/test_request_timeout.py b/tests/llm_providers/test_request_timeout.pynew file mode 100644index 0000000000..c69ad5e493--- /dev/null+++ b/tests/llm_providers/test_request_timeout.py@@ -0,0 +1,644 @@+"""Tests for ``llm.request_timeout`` / ``llm.max_retries`` wiring.++``llm.request_timeout`` is a *per-operation* bound (connect, write, and+the wait for the next response chunk), not a cap on total response time.+The behavioural consequences of that choice are measured against a real+local server in ``test_request_timeout_behavior.py``; this module pins+resolution and constructor wiring.+"""++from decimal import Decimal+from fractions import Fraction+from unittest.mock import patch++import math++import pytest++from local_deep_research.llm.providers._helpers import (+ CONNECT_TIMEOUT_SECONDS,+ resolve_max_retries,+ resolve_request_timeout,+)+from local_deep_research.llm.providers.implementations.anthropic import (+ AnthropicProvider,+)+from local_deep_research.llm.providers.implementations.custom_anthropic_endpoint import (+ CustomAnthropicEndpointProvider,+)+from local_deep_research.llm.providers.implementations.ollama import (+ OllamaProvider,+)+from local_deep_research.llm.providers.implementations.custom_openai_endpoint import (+ CustomOpenAIEndpointProvider,+)+from local_deep_research.llm.providers.implementations.llamacpp import (+ LlamaCppProvider,+)+from local_deep_research.llm.providers.implementations.lmstudio import (+ LMStudioProvider,+)+from local_deep_research.llm.providers.implementations.openai import (+ OpenAIProvider,+)+++_OPENAI_COMPATIBLE_CHAT_PATCH = (+ "local_deep_research.llm.providers.openai_base.ChatOpenAI"+)+_OPENAI_CHAT_PATCH = (+ "local_deep_research.llm.providers.implementations.openai.ChatOpenAI"+)+_OLLAMA_CHAT_PATCH = (+ "local_deep_research.llm.providers.implementations.ollama.ChatOllama"+)+_ANTHROPIC_CHAT_PATCH = (+ "local_deep_research.llm.providers.implementations.anthropic.ChatAnthropic"+)++_CONSTRUCTOR_CASES = (+ (AnthropicProvider, _ANTHROPIC_CHAT_PATCH),+ (CustomAnthropicEndpointProvider, _ANTHROPIC_CHAT_PATCH),+ (CustomOpenAIEndpointProvider, _OPENAI_COMPATIBLE_CHAT_PATCH),+ (LMStudioProvider, _OPENAI_COMPATIBLE_CHAT_PATCH),+ (LlamaCppProvider, _OPENAI_COMPATIBLE_CHAT_PATCH),+ (OpenAIProvider, _OPENAI_CHAT_PATCH),+ (OllamaProvider, _OLLAMA_CHAT_PATCH),+)++_PROVIDER_SETTINGS = {+ AnthropicProvider: {"llm.anthropic.api_key": "test-key"},+ CustomAnthropicEndpointProvider: {+ "llm.anthropic_endpoint.api_key": "test-key",+ "llm.anthropic_endpoint.url": "http://localhost:9090",+ },+ CustomOpenAIEndpointProvider: {+ "llm.openai_endpoint.api_key": "test-key",+ "llm.openai_endpoint.url": "http://localhost:8000/v1",+ },+ LMStudioProvider: {+ "llm.lmstudio.api_key": "test-key",+ "llm.lmstudio.url": "http://localhost:1234/v1",+ "llm.local_context_window_size": 8192,+ },+ LlamaCppProvider: {+ "llm.llamacpp.api_key": "test-key",+ "llm.llamacpp.url": "http://localhost:8080/v1",+ "llm.local_context_window_size": 8192,+ },+ OpenAIProvider: {"llm.openai.api_key": "test-key"},+ OllamaProvider: {+ "llm.ollama.url": "http://localhost:11434",+ "llm.local_context_window_size": 8192,+ },+}+++def _base_settings():+ """Minimum snapshot for create_llm to reach ChatOllama construction."""+ return {+ "llm.ollama.url": "http://localhost:11434",+ "llm.local_context_window_size": 8192,+ "llm.supports_max_tokens": True,+ "llm.max_tokens": 4096,+ }+++class TestRequestTimeoutResolution:+ @pytest.mark.parametrize(+ "configured_timeout", [1, Fraction(3, 2), Decimal("45"), "60", 1800]+ )+ def test_returns_float_for_finite_values_within_the_configured_range(+ self, configured_timeout+ ):+ # Given+ snapshot = {"llm.request_timeout": configured_timeout}++ # When+ timeout = resolve_request_timeout(snapshot)++ # Then+ assert timeout == float(configured_timeout)+ assert isinstance(timeout, float)++ @pytest.mark.parametrize("snapshot", [{}, {"llm.request_timeout": None}])+ def test_uses_the_default_when_the_setting_is_missing_or_none(+ self, snapshot+ ):+ # Given+ settings_snapshot = snapshot++ # When+ timeout = resolve_request_timeout(settings_snapshot)++ # Then+ assert timeout == 1800+ assert isinstance(timeout, float)++ @pytest.mark.parametrize(+ ("configured_timeout", "expected"),+ [+ (0, 1.0),+ (-5, 1.0),+ (Fraction(1) - Fraction(1, 2**54), 1.0),+ ],+ )+ def test_clamps_out_of_range_values_instead_of_raising(+ self, configured_timeout, expected+ ):+ """An operator typo must not be a self-inflicted LLM outage.++ Before the clamp, every one of these raised ``ValueError`` out of+ each provider's ``create_llm``, so a single bad field killed all+ LLM construction until it was corrected.+ """+ # Given+ snapshot = {"llm.request_timeout": configured_timeout}++ # When+ timeout = resolve_request_timeout(snapshot)++ # Then+ assert timeout == expected++ @pytest.mark.parametrize(+ "configured_timeout", [1801, 10**9, Fraction(1800) + Fraction(1, 2**40)]+ )+ def test_large_values_are_honoured_not_clamped(self, configured_timeout):+ """There is no ceiling: a slow local model may need hours.++ These values were previously clamped to a 1800s maximum. The cap was+ removed because any ceiling generous enough for a large model on+ modest hardware is too generous to catch a typo anyway -- the guard+ that actually prevents an unbounded wait is the finite-number check,+ which still rejects ``inf``.+ """+ settings_snapshot = {"llm.request_timeout": configured_timeout}++ timeout = resolve_request_timeout(settings_snapshot)++ assert timeout == pytest.approx(float(configured_timeout))+ assert math.isfinite(timeout)++ @pytest.mark.parametrize(+ "configured_timeout",+ [+ True,+ False,+ float("nan"),+ float("inf"),+ float("-inf"),+ "not-a-number",+ object(),+ [30],+ ],+ )+ def test_falls_back_to_the_default_for_unusable_shapes(+ self, configured_timeout+ ):+ """Every other reader on the construction path degrades; so does this."""+ # Given+ snapshot = {"llm.request_timeout": configured_timeout}++ # When+ timeout = resolve_request_timeout(snapshot)++ # Then+ assert timeout == 1800.0++ def test_full_format_snapshot_without_ui_element_still_resolves(self):+ """A full-format entry lacking ``ui_element`` resolves to ``"60"``.++ The value arrives as a string; it must still bound the client+ rather than take down LLM construction.+ """+ # Given+ snapshot = {"llm.request_timeout": {"value": 60}}++ # When+ timeout = resolve_request_timeout(snapshot)++ # Then+ assert timeout == 60.0++ def test_full_format_snapshot_with_a_boolean_number_value_uses_default(+ self,+ ):+ # Given — the number parser would coerce ``True`` to ``1``+ snapshot = {+ "llm.request_timeout": {"value": True, "ui_element": "number"}+ }++ # When+ timeout = resolve_request_timeout(snapshot)++ # Then+ assert timeout == 1800.0+++class TestMaxRetriesResolution:+ @pytest.mark.parametrize(+ ("snapshot", "expected"),+ [+ ({}, 2),+ ({"llm.max_retries": None}, 2),+ ({"llm.max_retries": 0}, 0),+ ({"llm.max_retries": 4}, 4),+ ({"llm.max_retries": "3"}, 3),+ # Out of range: a user setting 1000 must not get 1000 attempts.+ ({"llm.max_retries": 1000}, 5),+ ({"llm.max_retries": -1}, 0),+ ({"llm.max_retries": True}, 2),+ ({"llm.max_retries": "nope"}, 2),+ ],+ )+ def test_resolves_within_bounds(self, snapshot, expected):+ # When+ retries = resolve_max_retries(snapshot)++ # Then+ assert retries == expected+ assert isinstance(retries, int)+++class TestOllamaClientKwargs:+ """``llm.request_timeout`` and auth headers flow via client_kwargs."""++ def test_configured_timeout_is_passed_via_client_kwargs(self):+ # Given+ snapshot = _base_settings()+ snapshot["llm.request_timeout"] = 37++ def side_effect(key, default=None, *args, **kwargs):+ return snapshot.get(key, default)++ # When+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.get_setting_from_snapshot",+ side_effect=side_effect,+ ):+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.ChatOllama"+ ) as mock_chat:+ OllamaProvider.create_llm(+ model_name="llama3.1:8b", settings_snapshot=snapshot+ )++ # Then+ timeout = mock_chat.call_args[1]["client_kwargs"]["timeout"]+ assert timeout.read == 37+ assert timeout.write == 37+ assert timeout.pool == 37+ # Literal, not the imported constant: raising CONNECT_TIMEOUT_SECONDS+ # is a behaviour change and must break a test, not ride along.+ assert timeout.connect == 5.0++ def test_missing_setting_falls_back_to_the_default(self):+ # Given — request_timeout deliberately absent+ snapshot = _base_settings()++ def side_effect(key, default=None, *args, **kwargs):+ return snapshot.get(key, default)++ # When+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.get_setting_from_snapshot",+ side_effect=side_effect,+ ):+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.ChatOllama"+ ) as mock_chat:+ OllamaProvider.create_llm(+ model_name="llama3.1:8b", settings_snapshot=snapshot+ )++ # Then+ assert mock_chat.call_args[1]["client_kwargs"]["timeout"].read == 1800++ def test_auth_headers_are_passed_through_client_kwargs(self):+ """ChatOllama has no ``headers`` field, so a top-level kwarg was dropped.++ Routing them through client_kwargs is the only way the+ Authorization header reaches the underlying httpx clients.+ """+ # Given+ snapshot = _base_settings()+ snapshot["llm.ollama.api_key"] = "secret-token"++ def side_effect(key, default=None, *args, **kwargs):+ return snapshot.get(key, default)++ # When+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.get_setting_from_snapshot",+ side_effect=side_effect,+ ):+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.ChatOllama"+ ) as mock_chat:+ OllamaProvider.create_llm(+ model_name="llama3.1:8b", settings_snapshot=snapshot+ )++ # Then+ kwargs = mock_chat.call_args[1]+ assert kwargs["client_kwargs"]["headers"] == {+ "Authorization": "Bearer secret-token"+ }+ assert "headers" not in kwargs++ def test_auth_headers_actually_reach_both_httpx_clients(self):+ """End-to-end: the real ChatOllama must put the header on the wire."""+ # Given+ snapshot = _base_settings()+ snapshot["llm.ollama.api_key"] = "secret-token"++ def side_effect(key, default=None, *args, **kwargs):+ return snapshot.get(key, default)++ # When+ with patch(+ "local_deep_research.llm.providers.implementations.ollama.get_setting_from_snapshot",+ side_effect=side_effect,+ ):+ llm = OllamaProvider.create_llm(+ model_name="llama3.1:8b", settings_snapshot=snapshot+ )++ # Then+ for client in (llm._client, llm._async_client):+ assert (+ client._client.headers.get("authorization")+ == "Bearer secret-token"+ )+ assert client._client.timeout.read == 1800+ assert client._client.timeout.connect == 5.0+++class TestRequestTimeoutConstructorWiring:+ @pytest.mark.parametrize(+ ("timeout_setting", "expected_timeout"),+ (+ ({"llm.request_timeout": 37}, 37),+ ({}, 1800),+ ({"llm.request_timeout": None}, 1800),+ # Below the floor: clamped, never raised.+ ({"llm.request_timeout": 0}, 1),+ # Above the old ceiling: honoured, no longer clamped.+ ({"llm.request_timeout": 10**9}, 10**9),+ # The UI number parser runs before timeout clamping. Its+ # conversion overflow must default and still construct clients.+ (+ {+ "llm.request_timeout": {+ "value": 10**400,+ "ui_element": "number",+ }+ },+ 1800,+ ),+ ),+ ids=(+ "configured",+ "missing",+ "explicit-none",+ "too-low",+ "very-high",+ "full-format-overflow",+ ),+ )+ @pytest.mark.parametrize(+ ("provider", "chat_patch"),+ _CONSTRUCTOR_CASES,+ ids=(+ "anthropic",+ "custom-anthropic-endpoint",+ "custom-openai-endpoint",+ "lmstudio",+ "llamacpp",+ "openai",+ "ollama",+ ),+ )+ def test_passes_the_resolved_timeout_to_each_client_constructor(+ self, provider, chat_patch, timeout_setting, expected_timeout+ ):+ # Given+ snapshot = {+ **_PROVIDER_SETTINGS[provider],+ "llm.supports_max_tokens": False,+ **timeout_setting,+ }++ # When+ with patch(chat_patch) as mock_chat:+ provider.create_llm(+ model_name="test-model", settings_snapshot=snapshot+ )++ # Then+ kwargs = mock_chat.call_args.kwargs+ expected_connect = min(5.0, expected_timeout)+ if provider is OllamaProvider:+ # ollama.Client is per-instance regardless, so the SDK's own+ # Timeout object is used there.+ timeout = kwargs["client_kwargs"]["timeout"]+ assert timeout.read == expected_timeout+ assert timeout.write == expected_timeout+ assert timeout.pool == expected_timeout+ assert timeout.connect == expected_connect+ return+ if issubclass(provider, AnthropicProvider):+ # ChatAnthropic types the field ``float | None`` and builds its+ # httpx client from a cached factory, so only a scalar fits.+ assert kwargs["timeout"] == expected_timeout+ return+ # OpenAI-compatible: a hashable ``(connect, read, write, pool)``+ # tuple, not a Timeout object — see+ # ``TestOpenAIHttpxClientSharing`` for why the shape matters.+ assert kwargs["request_timeout"] == (+ expected_connect,+ expected_timeout,+ expected_timeout,+ expected_timeout,+ )++ @pytest.mark.parametrize(+ ("retries_setting", "expected_retries"),+ (+ ({}, 2),+ ({"llm.max_retries": 0}, 0),+ ({"llm.max_retries": 1000}, 5),+ (+ {"llm.max_retries": {"value": 10**400, "ui_element": "number"}},+ 2,+ ),+ ),+ ids=("default", "disabled", "clamped", "full-format-overflow"),+ )+ @pytest.mark.parametrize(+ ("provider", "chat_patch"),+ [case for case in _CONSTRUCTOR_CASES if case[0] is not OllamaProvider],+ ids=(+ "anthropic",+ "custom-anthropic-endpoint",+ "custom-openai-endpoint",+ "lmstudio",+ "llamacpp",+ "openai",+ ),+ )+ def test_passes_bounded_max_retries_to_each_client_constructor(+ self, provider, chat_patch, retries_setting, expected_retries+ ):+ """Unbounded SDK retries multiply the effective timeout.++ Without an explicit value both SDKs default to 2 internally while+ the langchain field reads ``None``, so the effective wait was+ ~3x the advertised bound with nothing capping a user-set value.+ """+ # Given+ snapshot = {+ **_PROVIDER_SETTINGS[provider],+ "llm.supports_max_tokens": False,+ **retries_setting,+ }++ # When+ with patch(chat_patch) as mock_chat:+ provider.create_llm(+ model_name="test-model", settings_snapshot=snapshot+ )++ # Then+ assert mock_chat.call_args.kwargs["max_retries"] == expected_retries++ @pytest.mark.parametrize(+ ("provider", "chat_patch"),+ _CONSTRUCTOR_CASES,+ ids=(+ "anthropic",+ "custom-anthropic-endpoint",+ "custom-openai-endpoint",+ "lmstudio",+ "llamacpp",+ "openai",+ "ollama",+ ),+ )+ def test_an_unusable_timeout_never_blocks_client_construction(+ self, provider, chat_patch+ ):+ """A bad value must degrade to the default, not raise."""+ # Given+ snapshot = {+ **_PROVIDER_SETTINGS[provider],+ "llm.supports_max_tokens": False,+ "llm.request_timeout": True,+ }++ # When+ with patch(chat_patch) as mock_chat:+ provider.create_llm(+ model_name="test-model", settings_snapshot=snapshot+ )++ # Then+ mock_chat.assert_called_once()+ kwargs = mock_chat.call_args.kwargs+ if provider is OllamaProvider:+ assert kwargs["client_kwargs"]["timeout"].read == 1800+ elif issubclass(provider, AnthropicProvider):+ assert kwargs["timeout"] == 1800+ else:+ assert kwargs["request_timeout"] == (5.0, 1800, 1800, 1800)+++class TestOpenAIHttpxClientSharing:+ """The timeout's *type* decides whether httpx clients are shared.++ ``langchain_openai`` builds ChatOpenAI's httpx clients through an+ ``@lru_cache``'d factory and quietly falls back to a per-instance+ client when the timeout value is unhashable+ (``_client_utils._get_default_httpx_client``). ``httpx.Timeout``+ defines ``__eq__`` without ``__hash__``, so passing a ``Timeout``+ object here would give every LLM in the process its own connection+ pool — and would falsify ``utilities/llm_utils.py``'s stated reason+ for ``_close_base_llm`` skipping these classes, plus the matching+ invariants in ``docs/developing/resource-cleanup.md``.++ Reverting ``_helpers.build_timeout`` to return+ ``openai.Timeout(connect=..., read=..., write=..., pool=...)`` leaves+ the effective per-operation bounds identical — and fails the+ hashability and sharing tests below.++ No network: the base_url is never contacted, only the client wiring+ that the constructor sets up is inspected.+ """++ _SNAPSHOT = {+ **_PROVIDER_SETTINGS[CustomOpenAIEndpointProvider],+ "llm.supports_max_tokens": False,+ "llm.request_timeout": 1800,+ }++ def _build(self):+ return CustomOpenAIEndpointProvider.create_llm(+ model_name="test-model", settings_snapshot=dict(self._SNAPSHOT)+ )++ def test_the_timeout_reaches_the_constructor_in_a_hashable_shape(self):+ # When+ with patch(_OPENAI_COMPATIBLE_CHAT_PATCH) as mock_chat:+ CustomOpenAIEndpointProvider.create_llm(+ model_name="test-model", settings_snapshot=dict(self._SNAPSHOT)+ )++ # Then — a tuple, and hash() must not raise+ timeout = mock_chat.call_args.kwargs["request_timeout"]+ assert timeout == (5.0, 1800, 1800, 1800)+ hash(timeout)+ # ``build_timeout`` is annotated ``tuple[float, float, float,+ # float]``, so the shape is asserted rather than inferred from the+ # equality above, where ints compare equal to floats. This says+ # nothing about the ``float(seconds)`` coercion: ``seconds`` here+ # comes from ``resolve_request_timeout`` and is already a float,+ # so this passes with or without it. Dropping that coercion is+ # caught by ``test_openai_base.py`` (the discovery path, where+ # ``MODEL_DISCOVERY_TIMEOUT_SECONDS`` is a bare int).+ assert all(type(part) is float for part in timeout)++ def test_two_llms_share_one_httpx_client(self):+ """Same settings -> same pooled sync and async httpx client."""+ # Given / When+ first = self._build()+ second = self._build()++ # Then+ assert first is not second+ assert first.root_client._client is second.root_client._client+ assert (+ first.root_async_client._client is second.root_async_client._client+ )++ def test_the_effective_per_operation_bounds_survive_the_tuple(self):+ """The tuple must expand to exactly the bounds the docs promise."""+ # When+ effective = self._build().root_client._client.timeout++ # Then+ assert effective.connect == 5.0+ assert effective.read == 1800+ assert effective.write == 1800+ assert effective.pool == 1800+++class TestConnectTimeoutConstant:+ def test_the_connect_bound_is_five_seconds(self):+ """Pinned as a literal so raising the constant fails here first.++ The user-facing timeout hint in+ ``error_handling/openai_compat_errors.py`` states this number in+ prose; changing it means changing that text too.+ """+ assert CONNECT_TIMEOUT_SECONDS == 5.0diff --git a/tests/llm_providers/test_request_timeout_behavior.py b/tests/llm_providers/test_request_timeout_behavior.pynew file mode 100644index 0000000000..12d41d725e--- /dev/null+++ b/tests/llm_providers/test_request_timeout_behavior.py@@ -0,0 +1,234 @@+"""Measured behaviour of ``llm.request_timeout`` against a local server.++These tests exercise a real ``ChatOpenAI`` built by the provider against a+loopback HTTP server that misbehaves in specific ways, and assert on+elapsed wall-clock time and on the number of request attempts the server+actually saw — not on constructor kwargs.++They pin the semantics the setting documents:++* it bounds each individual network operation, so a silent peer is+ abandoned at the configured value;+* it is *not* a total-response cap, so a peer that keeps dripping bytes+ legitimately outlives it (this is why the setting is named and+ documented as an inactivity bound);+* SDK retries multiply the effective wait, which is why+ ``llm.max_retries`` exists and is bounded.+"""++import json+import threading+import time+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++import pytest+from openai import APITimeoutError++from local_deep_research.llm.providers.implementations.custom_openai_endpoint import (+ CustomOpenAIEndpointProvider,+)++_COMPLETION_BODY = json.dumps(+ {+ "id": "chatcmpl-test",+ "object": "chat.completion",+ "created": 0,+ "model": "test-model",+ "choices": [+ {+ "index": 0,+ "message": {"role": "assistant", "content": "hi"},+ "finish_reason": "stop",+ }+ ],+ "usage": {+ "prompt_tokens": 1,+ "completion_tokens": 1,+ "total_tokens": 2,+ },+ }+).encode()++# Long enough that the client always gives up first, short enough that a+# leaked handler thread cannot outlive the test session by much.+_SERVER_HANG_SECONDS = 30.0++# Small enough that the dripped body is split into ~17 writes, so the drip+# test can keep a wide margin between the inter-chunk gap and the read bound+# and still have the whole exchange outlast that bound.+_DRIP_CHUNK_BYTES = 16+++class _Handler(BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"++ def do_POST(self): # noqa: N802 - stdlib naming+ length = int(self.headers.get("Content-Length") or 0)+ self.rfile.read(length)+ self.server.attempts.append(time.monotonic())++ mode = self.server.mode+ if mode == "silent":+ # Accept the connection, then never answer.+ self.server.release.wait(_SERVER_HANG_SECONDS)+ return++ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(_COMPLETION_BODY)))+ self.end_headers()++ if mode == "drip":+ for start in range(0, len(_COMPLETION_BODY), _DRIP_CHUNK_BYTES):+ self.wfile.write(+ _COMPLETION_BODY[start : start + _DRIP_CHUNK_BYTES]+ )+ self.wfile.flush()+ time.sleep(self.server.gap)+ elif mode == "stall":+ self.wfile.write(_COMPLETION_BODY[:16])+ self.wfile.flush()+ self.server.release.wait(_SERVER_HANG_SECONDS)+ else:+ self.wfile.write(_COMPLETION_BODY)++ def log_message(self, *args): # pragma: no cover - silence stderr+ pass+++def slow_server():+ """A loopback OpenAI-compatible endpoint with configurable badness."""+ server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler)+ server.daemon_threads = True+ server.mode = "ok"+ server.gap = 0.0+ server.attempts = []+ server.release = threading.Event()+ thread = threading.Thread(target=server.serve_forever, daemon=True)+ thread.start()+ try:+ yield server+ finally:+ server.release.set()+ server.shutdown()+ server.server_close()+ thread.join(timeout=5)+++def _make_llm(server, *, timeout, retries):+ snapshot = {+ "llm.openai_endpoint.api_key": "test-key",+ "llm.openai_endpoint.url": (+ f"http://127.0.0.1:{server.server_address[1]}/v1"+ ),+ "llm.supports_max_tokens": False,+ "llm.request_timeout": timeout,+ "llm.max_retries": retries,+ }+ return CustomOpenAIEndpointProvider.create_llm(+ model_name="test-model", settings_snapshot=snapshot+ )+++def _elapsed_failure(llm):+ started = time.monotonic()+ with pytest.raises(APITimeoutError):+ llm.invoke("hello")+ return time.monotonic() - started+++class TestSilentServer:+ def test_aborts_at_the_configured_timeout_when_retries_are_disabled(+ self, slow_server+ ):+ """A peer that accepts and never answers must not hold the worker."""+ # Given+ slow_server.mode = "silent"+ llm = _make_llm(slow_server, timeout=1, retries=0)++ # When+ elapsed = _elapsed_failure(llm)++ # Then+ assert len(slow_server.attempts) == 1+ assert 0.5 <= elapsed < 3.0, elapsed++ def test_sdk_retries_multiply_the_effective_wait(self, slow_server):+ """One logical call costs (max_retries + 1) x the timeout.++ With ``llm.max_retries`` unwired, both SDKs silently used their own+ default of 2, so a "1 second" bound cost ~3 timeouts plus backoff+ with nothing capping a user-supplied value.+ """+ # Given+ slow_server.mode = "silent"+ no_retry = _make_llm(slow_server, timeout=1, retries=0)+ with_retries = _make_llm(slow_server, timeout=1, retries=2)++ # When+ baseline = _elapsed_failure(no_retry)+ attempts_without_retries = len(slow_server.attempts)+ slow_server.attempts.clear()+ retried = _elapsed_failure(with_retries)++ # Then+ assert attempts_without_retries == 1+ assert len(slow_server.attempts) == 3+ assert retried > baseline * 2, (baseline, retried)++ def test_an_out_of_range_timeout_is_clamped_rather_than_fatal(+ self, slow_server+ ):+ """``0`` used to raise ValueError out of every create_llm call."""+ # Given+ slow_server.mode = "silent"+ llm = _make_llm(slow_server, timeout=0, retries=0)++ # When+ elapsed = _elapsed_failure(llm)++ # Then — clamped up to the 1s minimum, not an outage+ assert len(slow_server.attempts) == 1+ assert elapsed < 3.0, elapsed+++class TestResponseBodyPacing:+ def test_a_stalled_response_body_aborts_at_the_timeout(self, slow_server):+ """A gap between chunks larger than the bound ends the request."""+ # Given+ slow_server.mode = "stall"+ llm = _make_llm(slow_server, timeout=1, retries=0)++ # When+ elapsed = _elapsed_failure(llm)++ # Then+ assert 0.5 <= elapsed < 5.0, elapsed++ def test_a_steady_drip_outlives_the_timeout_by_design(self, slow_server):+ """The bound is per-operation, not a cap on total response time.++ This is the documented behaviour: as long as bytes keep arriving+ inside the window, the response completes even though the whole+ exchange takes far longer than ``llm.request_timeout``. The+ setting name, description and docs/CONFIGURATION.md must keep+ saying so — if this test ever starts failing because a total+ deadline was added, those need updating with it.+ """+ # Given — a 10x margin between the inter-chunk gap and the read+ # bound, so a loaded CI runner stalling between writes cannot flip+ # the result into a spurious timeout.+ slow_server.mode = "drip"+ slow_server.gap = 0.2+ llm = _make_llm(slow_server, timeout=2, retries=0)++ # When+ started = time.monotonic()+ result = llm.invoke("hello")+ elapsed = time.monotonic() - started++ # Then — the whole exchange outlasts the 2s bound because no single+ # gap reaches it.+ assert result.content == "hi"+ assert elapsed > 2.0, elapseddiff --git a/tests/security/test_llm_object_not_logged.py b/tests/security/test_llm_object_not_logged.pynew file mode 100644index 0000000000..8767de19c3--- /dev/null+++ b/tests/security/test_llm_object_not_logged.py@@ -0,0 +1,777 @@+"""Nothing in ``src/`` may log or serialise a whole LLM object.++Routing the Ollama bearer token through ``client_kwargs`` is the only way+it reaches the httpx clients (``ChatOllama`` has no ``headers`` field),+but ``client_kwargs`` is a plain pydantic field: langchain_ollama marks+it neither ``exclude`` nor ``repr=False``, so the token is present in+``repr(llm)``, ``llm.model_dump()`` and ``dumpd(llm)`` — the+``serialized`` payload every LangChain callback receives. There is no+supported way to hide it without subclassing a third-party model, so the+exposure is contained by contract instead.++Today the only consumer is ``metrics.token_counter.TokenCountingCallback+.on_llm_start``, which reads named scalar keys (``_type``, ``name``,+``kwargs["model"]``) and neither logs nor persists the dict.++What this contract actually detects+-----------------------------------+It is a syntactic (``ast``) rule over the shipped package, so it sees+*shapes*, not values. It reports, anywhere in a module:++1. a whole LLM object or the whole ``serialized`` dict reaching a logging+ call — every ``logger.<level>(...)`` plus ``logger.bind(...)``, in a+ positional argument, a keyword value, or an f-string placeholder;+2. the same values reaching a serialisation call — ``str``/``repr``/+ ``print``/``format``/``pprint``/``pformat``, ``dumpd``/``dumps``,+ ``json.dumps``/``yaml.dump``/``pickle.dumps``, and the+ ``.model_dump()``/``.model_dump_json()``/``.dict()``/``.json()``/+ ``.to_dict()`` methods on the object itself — whether or not the+ result is then logged, so ``row.payload = llm.model_dump()`` counts;+3. ``<llm>.__dict__`` and the credential-bearing ``*client_kwargs``+ attributes in any of those positions — including a read *into* one:+ ``llm.client_kwargs["headers"]``, the chained+ ``…["headers"]["Authorization"]`` and ``llm.client_kwargs.get(…)`` all+ count, because that mapping *is* the bearer token;+4. a whole LLM object interpolated into *any* f-string, logged or not,+ because the resulting string is one assignment away from a sink;+5. the ``serialized`` dict passed to any call at all, bare or wrapped in+ a container (``audit([serialized])``, ``store(**{"p": serialized})``)+ — the callback may read named keys off it (``serialized["_type"]``,+ ``serialized.get("name")``), never hand the dict on;+6. any of the above inside a comprehension, in either the produced value+ or the iterable it is drawn from (``[m for m in [llm]]``).++An "LLM object" here means a bare name in ``_LLM_NAMES`` or an attribute+read whose final name is in ``_LLM_ATTRS`` (``self.llm``, ``self._llm``,+``agent.chat_model``, …) — and reading one LLM-valued attribute *off*+another (``self.llm.base_llm``, the wrapper unwrap that hands a caller+the raw model) is itself such a read.++What it does not detect+-----------------------+- **A scalar field of an LLM already identified.** ``self.base_llm.model``+ and its private spellings (``_model``, ``llm_model``, ``_chat_model``)+ are langchain's model-*name* string, so the chain stops there. Any other+ attribute name is either credential-bearing, another handle on the+ object, or outside the lists entirely.+- **Renamed or derived values.** ``x = llm`` then ``logger.info(x)``,+ ``payload = llm.model_dump()`` logged three functions later, a dict+ literal built field-by-field from the object, or an LLM stored in a+ container and read back (``llms[0]``, ``self._pool["chat"]``). There is+ no dataflow analysis here. The same limit covers a credential mapping+ reached through a call result (``getattr(llm, "client_kwargs")["headers"]``,+ ``vars(llm)["client_kwargs"]``), its ``.values()``/``.items()``/``.copy()``+ views, and a bare local that holds the mapping before it is attached to+ the model (``client_kwargs = {...}`` in the provider factory).+- **Third-party sinks.** Anything langchain, an SDK, a callback handler+ from another package, or a tracing integration does with the object+ after it leaves ``src/`` — including the ``serialized`` payload+ langchain itself constructs and dispatches to every registered handler.+- **A class's own ``__str__``/``__repr__``/``__format__`` restating the+ object it wraps** — ``repr``/``str``/``format`` of a whole LLM, or one+ in the f-string the method returns. That does not create a sink, it+ restates the exposure the contract already assumes: it does mean+ ``repr(RateLimitedLLMWrapper(...))`` carries the token exactly as+ ``repr(llm)`` does. *Only* that forwarding is exempt — a logger call, a+ ``model_dump``/``dumpd``/``json.dumps``/``yaml.dump``, a persistence+ call, a ``__dict__`` read and a ``serialized`` hand-off are all still+ reported inside such a method, as is anything in a function nested in+ it. The method must be defined directly in a ``class`` body; a+ module-level ``def __repr__`` is an ordinary function.+- **Code outside the shipped package**: tests, scripts and notebooks.+- **Names outside the two lists below.** The sets are literal; an LLM+ held in a variable called something else is invisible to this rule.++So this is a tripwire against the idioms that would leak the token+tomorrow, not a proof that it cannot leak. The underlying exposure is+unfixed and unfixable from here without subclassing a third-party model.++``getattr(llm, "base_url", ...)`` and ``llm.some_attr`` stay allowed —+naming one attribute is not dumping the object, unless that attribute is+a credential mapping or another model — as do the identity and type+predicates ``type(llm)``, ``id(llm)``, ``isinstance(llm, X)`` and+``llm is None``, which reveal nothing.+"""++import ast+from pathlib import Path++import pytest++import local_deep_research++SRC = Path(local_deep_research.__file__).parent++# --- BEGIN PREDICATE ---+_LOG_METHODS = {+ "debug",+ "info",+ "warning",+ "warn",+ "error",+ "exception",+ "critical",+ "success",+ "trace",+ "log",+ # loguru's contextual binder: `logger.bind(llm=llm).info(...)` puts the+ # object in the record's `extra` dict, which every sink formats.+ "bind",+}+# Called as plain functions: `repr(llm)`, `dumpd(llm)`. `json.dumps(x)`+# reaches this rule through _DUMP_METHODS instead.+_DUMP_FUNCS = {+ "str",+ "repr",+ "format",+ "print",+ "pprint",+ "pformat",+ "dumpd",+ "dumps",+}+# Called as methods. Two distinct positions matter: the *receiver*+# (`llm.model_dump()`) and the *arguments* (`json.dumps(serialized)`,+# `yaml.dump(llm)`).+_DUMP_METHODS = {+ "dump",+ "dumps",+ "dumpd",+ "model_dump",+ "model_dump_json",+ "dict",+ "json",+ "to_dict",+ "to_json",+ "serialize",+}+# Bare variable names that hold a whole LLM.+_LLM_NAMES = {+ "llm",+ "_llm",+ "base_llm",+ "raw_llm",+ "chat_model",+ "chat_llm",+ "wrapped_llm",+ "model_instance",+ "llm_instance",+ "provider_llm",+}+# Attribute names that hold langchain's model-*name* string rather than a+# model object: `ChatOllama.model`, `ChatOpenAI.model`, and the private+# spellings this tree uses for the same value. Read off an LLM this rule+# has already identified (`self.base_llm.model`), one of these is a single+# scalar field and the chain stops there; read off anything else+# (`strategy.model`) the owner is unknown and the value is taken to be the+# object.+_SCALAR_LLM_FIELDS = {+ "model",+ "_model",+ "llm_model",+ "_chat_model",+}+# Attribute names that hold a whole LLM: `self.llm`, `handler._llm`,+# `strategy.model`. Checked on the *final* name of the attribute chain,+# so the owner can be anything.+_LLM_ATTRS = _LLM_NAMES | _SCALAR_LLM_FIELDS+# The callback payload. Named keys may be read off it; the dict itself+# may not be logged, serialised, or handed to another call.+_SERIALIZED = "serialized"+_CREDENTIAL_ATTRS = {+ "client_kwargs",+ "sync_client_kwargs",+ "async_client_kwargs",+}+# Mapping reads that are equivalent to a subscript: `d.get("k")` and+# `d["k"]` return the same value, so both are followed to their base.+_MAPPING_READS = {"get", "setdefault", "pop"}+# Reveal nothing about the object's contents, so an LLM reaching one of+# these is not a leak — and neither is whatever the call returns.+_SAFE_FUNCS = {+ "type",+ "id",+ "isinstance",+ "issubclass",+ "callable",+ "hasattr",+ "len",+ "bool",+ "getattr",+}+++def _llm_label(node):+ """Return a printable name if ``node`` is a whole LLM / the payload."""+ if isinstance(node, ast.Name):+ if node.id in _LLM_NAMES:+ return node.id+ if node.id == _SERIALIZED:+ return _SERIALIZED+ return None+ if isinstance(node, ast.Attribute) and node.attr in _LLM_ATTRS:+ # `self.base_llm.model` is langchain's model-*name* string, a+ # scalar field of an LLM we have already identified — not a+ # second LLM — so that one chain stops here. Every other name in+ # the set holds an object, and reading it off an LLM is exactly+ # the wrapper unwrap (`self.llm.base_llm`) that hands a caller+ # the credential-bearing model: those keep the label.+ if (+ node.attr in _SCALAR_LLM_FIELDS+ and _llm_label(node.value) is not None+ ):+ return None+ return f".{node.attr}"+ return None+++def _is_safe_call(node):+ """``type(llm)``/``isinstance(llm, X)``/``getattr(llm, "base_url")``.++ ``getattr`` is exempt only when the attribute it names is not itself+ credential-bearing.+ """+ if not (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)):+ return False+ if node.func.id not in _SAFE_FUNCS:+ return False+ if node.func.id == "getattr" and len(node.args) >= 2:+ wanted = node.args[1]+ if (+ isinstance(wanted, ast.Constant)+ and wanted.value in _CREDENTIAL_ATTRS+ ):+ return False+ return True+++def _is_identity_compare(node):+ """``llm is None`` / ``llm in registry`` — a boolean, not a dump."""+ return isinstance(node, ast.Compare) and all(+ isinstance(op, (ast.Is, ast.IsNot, ast.In, ast.NotIn))+ for op in node.ops+ )+++def _indexed_base(node):+ """Follow a ``[...]``/``.get(...)`` chain down to what is indexed.++ ``llm.client_kwargs["headers"]`` *is* ``{"Authorization": "Bearer …"}``+ — indexing a credential mapping is not reading one named scalar off+ the object, so the chain is followed to its base and reported.+ ``serialized["_type"]`` and ``serialized.get("name")`` bottom out in+ the callback payload, which the contract explicitly allows to be read+ by key, so that base returns nothing.+ """+ depth = 0+ while True:+ if isinstance(node, ast.Subscript):+ node, depth = node.value, depth + 1+ continue+ if (+ isinstance(node, ast.Call)+ and isinstance(node.func, ast.Attribute)+ and node.func.attr in _MAPPING_READS+ ):+ node, depth = node.func.value, depth + 1+ continue+ break+ if depth == 0:+ return None+ if isinstance(node, ast.Attribute) and node.attr in _CREDENTIAL_ATTRS:+ owner = _llm_label(node.value)+ return None if owner is None else f"{owner}.{node.attr}"+ label = _llm_label(node)+ return None if label in (None, _SERIALIZED) else label+++def _payload_leaks(node):+ """Whole-object leaks reachable from a value that *is* being emitted.++ Descent stops at a named scalar attribute read (``llm.model_name`` is+ one string, not the object) and at a subscript of anything but a+ credential mapping or an LLM (``serialized["_type"]``).+ """+ if node is None or _is_safe_call(node) or _is_identity_compare(node):+ return []+ label = _llm_label(node)+ if label is not None:+ return [(node.lineno, f"`{label}`")]+ if isinstance(node, ast.Attribute):+ owner = _llm_label(node.value)+ if owner is not None and (+ node.attr in _CREDENTIAL_ATTRS or node.attr == "__dict__"+ ):+ return [(node.lineno, f"`{owner}.{node.attr}`")]+ # A named attribute of an LLM is allowed; keep looking inside+ # anything that is not itself a plain name/attribute chain.+ if isinstance(node.value, (ast.Name, ast.Attribute)):+ return []+ return _payload_leaks(node.value)+ if isinstance(node, ast.Subscript):+ base = _indexed_base(node)+ if base is not None:+ return [(node.lineno, f"`{base}` is read by key")]+ return []+ if isinstance(node, ast.IfExp):+ # `'yes' if llm else 'no'` — the test is a truthiness guard in the+ # same class as `llm is None`; only the branches are emitted.+ return _payload_leaks(node.body) + _payload_leaks(node.orelse)+ if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.Not):+ return []+ if isinstance(node, ast.Call):+ base = _indexed_base(node)+ if base is not None:+ return [(node.lineno, f"`{base}` is read by key")]+ out = []+ for arg in node.args:+ out += _payload_leaks(arg)+ for kw in node.keywords:+ out += _payload_leaks(kw.value)+ out += _payload_leaks(node.func)+ return out+ if isinstance(node, ast.comprehension):+ # The loop target is a fresh binding, not a read of the object;+ # the iterable and the guards are ordinary reads.+ out = _payload_leaks(node.iter)+ for cond in node.ifs:+ out += _payload_leaks(cond)+ return out+ # No `ast.expr` filter: `ast.comprehension` is not an expression, and+ # skipping it left `[m for m in [llm]]` unvisited below its `elt`.+ out = []+ for child in ast.iter_child_nodes(node):+ out += _payload_leaks(child)+ return out+++def _serialized_reads(node):+ """Bare ``serialized`` reads in a value handed to a call.++ Wrapping it changes nothing: ``audit([serialized])``,+ ``audit({"p": serialized})`` and ``store(**{"p": serialized})`` hand+ the dict on exactly as ``audit(serialized)`` does. A read *by key*+ inside the container (``{serialized["_type"]: 1}``) is still allowed.+ """+ if isinstance(node, ast.Name):+ return [node] if node.id == _SERIALIZED else []+ if isinstance(node, (ast.List, ast.Tuple, ast.Set, ast.Dict)):+ out = []+ for child in ast.iter_child_nodes(node):+ out += _serialized_reads(child)+ return out+ return []+++def _call_leaks(node):+ """Node-local rules for a single ``ast.Call``."""+ func = node.func+ out = []+ args = list(node.args) + [kw.value for kw in node.keywords]++ is_log = isinstance(func, ast.Attribute) and func.attr in _LOG_METHODS+ if is_log:+ for arg in args:+ out += [+ (line, f"{what} reaches `.{func.attr}(...)`")+ for line, what in _payload_leaks(arg)+ ]++ is_dump_func = isinstance(func, ast.Name) and func.id in _DUMP_FUNCS+ is_dump_method = (+ isinstance(func, ast.Attribute) and func.attr in _DUMP_METHODS+ )+ if is_dump_func or is_dump_method:+ sink = func.id if is_dump_func else f".{func.attr}"+ for arg in args:+ out += [+ (line, f"{what} is serialised by `{sink}(...)`")+ for line, what in _payload_leaks(arg)+ ]+ if is_dump_method:+ # The receiver: `llm.model_dump()`, `self.llm.dict()`.+ receiver = _llm_label(func.value)+ if receiver is not None:+ out.append(+ (+ node.lineno,+ f"`{receiver}` is serialised by `.{func.attr}()`",+ )+ )++ # The callback payload may be read by key, never handed on.+ if not (is_log or _is_safe_call(node)):+ for arg in args:+ for name in _serialized_reads(arg):+ out.append(+ (+ name.lineno,+ f"`{_SERIALIZED}` is passed whole to another call",+ )+ )+ return out+++# A wrapper's own ``__repr__`` forwarding to the object it wraps does not+# create a sink: it makes ``repr(wrapper)`` exactly as revealing as+# ``repr(llm)``, which is the exposure this contract already assumes and+# guards. ``RateLimitedLLMWrapper.__str__``/``__repr__``+# (``web_search_engines/rate_limiting/llm/wrapper.py``) is the site this+# carve-out exists for, but these names are common — 62 of them across 22+# files, mostly SQLAlchemy models — so the exemption is kept as narrow as+# the justification: it covers only the forwarding stringify itself+# (``repr``/``str``/``format`` of an LLM, or an LLM in the f-string the+# method returns), only in a method defined directly in a ``ClassDef``+# body, and not in any function nested inside it. A logger call, a+# ``model_dump``, a persistence call or a ``serialized`` hand-off is a+# real sink wherever it is written.+_REPR_DUNDERS = {"__str__", "__repr__", "__format__"}+# Restating the wrapped object's own text is what these methods are for.+_FORWARDING_REPR_FUNCS = {"str", "repr", "format"}+++def _own_scope_nodes(func):+ """Every node of ``func`` that is not inside a nested scope."""+ inside = set()+ pending = list(ast.iter_child_nodes(func))+ while pending:+ node = pending.pop()+ inside.add(id(node))+ if isinstance(+ node,+ (+ ast.FunctionDef,+ ast.AsyncFunctionDef,+ ast.Lambda,+ ast.ClassDef,+ ),+ ):+ # A nested `def helper(): print(llm)` is ordinary code that+ # happens to sit in a dunder; it gets no carve-out.+ continue+ pending += list(ast.iter_child_nodes(node))+ return inside+++def _repr_dunder_nodes(tree):+ """Nodes in a class's own ``__str__``/``__repr__``/``__format__``.++ The method must be a direct child of a ``ClassDef`` body: a+ module-level ``def __repr__`` is an ordinary function, and naming one+ of these must not switch the contract off for a whole module.+ """+ inside = set()+ for owner in ast.walk(tree):+ if not isinstance(owner, ast.ClassDef):+ continue+ for node in owner.body:+ if (+ isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))+ and node.name in _REPR_DUNDERS+ ):+ inside |= _own_scope_nodes(node)+ return inside+++def _is_forwarding_repr(node):+ """``repr(self.base_llm)`` — the wrapped object restating itself.++ Only a whole LLM qualifies: ``repr(llm.client_kwargs)`` is a+ credential read wearing a repr, and ``str(serialized)`` is the+ callback payload. Neither is what a wrapper's dunder is for.+ """+ if not (+ isinstance(node.func, ast.Name)+ and node.func.id in _FORWARDING_REPR_FUNCS+ ):+ return False+ return bool(node.args) and _is_forwarded_value(node.args[0])+++def _is_forwarded_value(node):+ """A whole LLM, or its repr — all a wrapper's dunder may restate."""+ if isinstance(node, ast.Call):+ return _is_forwarding_repr(node)+ label = _llm_label(node)+ return label is not None and label != _SERIALIZED+++def _violations(source: str, label: str):+ """Yield ``"<label>:<line> <reason>"`` for every offending sink."""+ tree = ast.parse(source, filename=label)+ exempt = _repr_dunder_nodes(tree)+ found = []+ for node in ast.walk(tree):+ in_repr = id(node) in exempt+ if isinstance(node, ast.Call):+ # Inside a repr dunder the wrapped object's own text may be+ # restated — and nothing else. Every other sink stays live.+ if in_repr and _is_forwarding_repr(node):+ continue+ found += _call_leaks(node)+ elif isinstance(node, ast.JoinedStr):+ # An f-string is one assignment away from any sink, so the+ # interpolation itself is the violation — except where such a+ # method restates the wrapped object, which is the f-string+ # `f"RateLimitedLLMWrapper({repr(self.base_llm)})"` returns.+ for part in node.values:+ if not isinstance(part, ast.FormattedValue):+ continue+ if in_repr and _is_forwarded_value(part.value):+ continue+ found += [+ (line, f"{what} is interpolated into an f-string")+ for line, what in _payload_leaks(part.value)+ ]+ elif isinstance(node, ast.Attribute) and node.attr == "__dict__":+ owner = _llm_label(node.value)+ if owner is not None:+ found.append((node.lineno, f"`{owner}.__dict__` is read"))+ seen = set()+ out = []+ for line, reason in found:+ key = (line, reason)+ if key in seen:+ continue+ seen.add(key)+ out.append(f"{label}:{line} {reason}")+ return sorted(out)+++# --- END PREDICATE ---+++class TestTheGuardActuallyDetects:+ """A contract test that cannot fail is worthless; prove it fires."""++ @pytest.mark.parametrize(+ "snippet",+ [+ 'logger.info(f"built {llm}")',+ "logger.debug(llm)",+ 'logger.info("cfg %s", llm.client_kwargs)',+ "json.dumps(serialized)",+ "logger.warning(repr(llm))",+ 'logger.info(f"start {serialized}")',+ # Shapes the first round of this contract missed.+ 'logger.info(f"{repr(llm)}")',+ 'logger.info(f"{llm.model_dump()}")',+ 'logger.info("%s", llm.model_dump())',+ "row.payload = llm.model_dump()",+ "store(dumpd(llm))",+ "session.add(Row(payload=serialized))",+ 'logger.info(f"{self.llm}")',+ 'logger.bind(llm=llm).info("made")',+ 'msg = f"llm={llm}"',+ "print(llm)",+ # Neighbours of the above.+ 'logger.info(f"{self._llm}")',+ 'logger.debug(f"{self.model}")',+ "yaml.dump(llm)",+ "pickle.dumps(self.chat_model)",+ "payload = self.llm.__dict__",+ 'logger.info("cfg %s", self.llm.sync_client_kwargs)',+ 'logger.info(getattr(llm, "client_kwargs", None))',+ "audit(serialized)",+ "logger.bind(payload=serialized).debug('x')",+ "return llm.model_dump_json()",+ "blob = self.base_llm.to_dict()",+ # Indexing a credential mapping reads the token itself.+ 'logger.error("cfg %s", llm.client_kwargs["headers"])',+ "logger.error(f\"{llm.client_kwargs['headers']}\")",+ 'logger.info(llm.client_kwargs["headers"]["Authorization"])',+ "logger.info(f\"{llm.sync_client_kwargs['headers']}\")",+ "logger.info(f\"{llm.client_kwargs.get('headers')}\")",+ 'logger.info(f"{llm.async_client_kwargs[0]}")',+ # Unwrapping a wrapper hands over the wrapped model.+ 'logger.info(f"{self.llm.base_llm}")',+ 'logger.info(f"{llm.base_llm}")',+ 'logger.info(f"{self.model.base_llm}")',+ 'logger.info(f"{self.llm.base_llm.client_kwargs}")',+ "self.llm.base_llm.model_dump()",+ "repr(self.llm.base_llm)",+ # A comprehension's iterable is read like any other value.+ 'logger.info(f"{[m for m in [llm]]}")',+ 'logger.info(", ".join(str(m) for m in [llm]))',+ # Wrapping the payload does not stop it being handed on.+ "audit([serialized])",+ 'audit({"p": serialized})',+ 'store(**{"p": serialized})',+ # The repr-dunder exemption covers the forwarded repr and+ # nothing else: not a logger, not a dump, not persistence,+ # not the payload, not a nested `def`, not a bare function.+ "class W:\n"+ " def __repr__(self):\n"+ " logger.info(self.base_llm)\n"+ " return 'W'",+ "class W:\n"+ " def __repr__(self):\n"+ " session.add(Row(p=self.llm.model_dump()))\n"+ " return 'x'",+ "class W:\n"+ " def __repr__(self):\n"+ " audit(serialized)\n"+ " return 'x'",+ "class W:\n"+ " def __format__(self, spec):\n"+ " store(dumpd(llm))\n"+ " return 'x'",+ "class W:\n"+ " def __repr__(self):\n"+ " def helper():\n"+ " return repr(llm)\n"+ " return 'x'",+ "class NotAnLLM:\n"+ " def __repr__(self):\n"+ " yaml.dump(llm)\n"+ " return 'x'",+ "def __repr__(self):\n return repr(self.llm)",+ "class W:\n"+ " def __repr__(self):\n"+ " return repr(llm.client_kwargs)",+ ],+ ids=(+ "fstring-llm",+ "bare-llm",+ "client-kwargs-attr",+ "json-dumps-serialized",+ "repr-llm",+ "fstring-serialized",+ "nested-repr-in-fstring",+ "model-dump-in-fstring",+ "model-dump-printf-arg",+ "model-dump-persisted",+ "dumpd-into-a-store",+ "serialized-persisted",+ "self-llm-in-fstring",+ "logger-bind-kwarg",+ "fstring-outside-a-call",+ "print-llm",+ "self-underscore-llm",+ "self-model-attr",+ "yaml-dump",+ "pickle-dumps-attr",+ "dunder-dict",+ "credential-attr-on-self",+ "getattr-credential-attr",+ "serialized-to-any-call",+ "bind-serialized",+ "model-dump-json-returned",+ "to-dict-on-attr",+ "credential-subscript",+ "credential-subscript-in-an-fstring",+ "credential-subscript-chained",+ "sync-credential-subscript",+ "credential-get",+ "async-credential-index",+ "wrapper-unwrap-off-self-llm",+ "wrapper-unwrap-off-a-bare-llm",+ "wrapper-unwrap-off-self-model",+ "credential-behind-an-unwrap",+ "model-dump-behind-an-unwrap",+ "repr-of-an-unwrap",+ "comprehension-iterable",+ "generator-iterable",+ "serialized-in-a-list",+ "serialized-in-a-dict",+ "serialized-in-a-splat",+ "logger-inside-a-repr-dunder",+ "persistence-inside-a-repr-dunder",+ "serialized-inside-a-repr-dunder",+ "dumpd-inside-a-format-dunder",+ "nested-def-inside-a-repr-dunder",+ "repr-dunder-of-an-unrelated-class",+ "module-level-repr-dunder",+ "credential-repr-inside-a-repr-dunder",+ ),+ )+ def test_offending_shapes_are_reported(self, snippet):+ assert _violations(snippet, "<snippet>")++ @pytest.mark.parametrize(+ "snippet",+ [+ "logger.info(f\"url {getattr(llm, 'base_url', None)}\")",+ 'logger.debug(f"model {llm.model_name}")',+ 'model_name = serialized["_type"]',+ 'logger.info("count %d", len(prompts))',+ # Identity and type predicates reveal nothing (R2.4).+ 'logger.info(f"{type(llm).__name__}")',+ 'logger.info(f"{id(llm)}")',+ 'logger.info(f"{llm is None}")',+ 'logger.info("%s", isinstance(llm, ChatOpenAI))',+ "logger.debug(f\"{serialized.get('name')}\")",+ 'logger.info("%s", serialized["kwargs"]["model"])',+ 'logger.debug(f"{llm.__class__.__name__}")',+ "if llm is not None:\n logger.info('ready')",+ 'json.dumps({"model": llm.model_name})',+ # Truthiness guards and a wrapper restating its own repr.+ "logger.debug(f\"llm={'yes' if self.llm else 'no'}\")",+ 'logger.debug(f"{not self.llm}")',+ "model = str(self.base_llm.model)",+ 'logger.info(f"{self.llm.model}")',+ 'logger.debug(f"{self.base_llm._model}")',+ "class W:\n"+ " def __repr__(self):\n"+ " return f'W({repr(self.base_llm)})'",+ 'class W:\n def __repr__(self):\n return f"W({self.base_llm!r})"',+ "class W:\n def __str__(self):\n return str(self.base_llm)",+ # A comprehension over something that is not an LLM, and the+ # payload still read by key inside a container.+ 'logger.info(f"{[p for p in prompts]}")',+ 'logger.info(f"{[m for m in pool if m is llm]}")',+ 'audit({serialized["_type"]: 1})',+ ],+ ids=(+ "getattr-one-attribute",+ "one-named-attribute",+ "subscript-outside-a-log-call",+ "unrelated-log",+ "type-name",+ "id-of",+ "is-none",+ "isinstance",+ "serialized-get-named-key",+ "serialized-nested-key",+ "class-name",+ "identity-guard",+ "one-attribute-into-json",+ "truthiness-guard-in-a-conditional",+ "negated-truthiness-guard",+ "model-name-field-of-an-llm",+ "model-field-of-an-identified-llm",+ "private-model-field-behind-an-unwrap",+ "wrapper-repr-dunder",+ "wrapper-repr-dunder-conversion",+ "wrapper-str-dunder",+ "comprehension-without-an-llm",+ "identity-guard-in-a-comprehension",+ "serialized-key-inside-a-container",+ ),+ )+ def test_legitimate_shapes_are_not_reported(self, snippet):+ assert not _violations(snippet, "<snippet>")+++class TestNoSinkInSource:+ def test_no_module_logs_or_serialises_a_whole_llm(self):+ """Grep-style contract over the shipped package."""+ # Given+ modules = sorted(SRC.rglob("*.py"))+ assert len(modules) > 100, "source tree not found"++ # When+ offenders = []+ for path in modules:+ offenders += _violations(+ path.read_text(encoding="utf-8"),+ str(path.relative_to(SRC)),+ )++ # Then+ assert offenders == [], (+ "An LLM object or a callback `serialized` payload reaches a log "+ "or serialisation sink. The Ollama bearer token lives in "+ "`client_kwargs`, so that sink would persist a credential:\n"+ + "\n".join(offenders)+ )diff --git a/tests/settings/golden_master_settings.json b/tests/settings/golden_master_settings.jsonindex 454314554f..4145ef4e26 100644--- a/tests/settings/golden_master_settings.json+++ b/tests/settings/golden_master_settings.json@@ -1386,6 +1386,20 @@"value": 20480,"visible": true},+ "llm.max_retries": {+ "category": "llm_parameters",+ "description": "Maximum provider SDK retries per failed inference or embedding request. Applies to OpenAI-compatible and Anthropic SDKs; the Ollama client does not retry. Values outside 0-5 are clamped. Each retry starts another request with its own per-operation timeouts; SDK backoff or server-requested retry delays can further extend elapsed time. This setting does not impose a total request deadline. Model discovery SDK retries are disabled.",+ "editable": true,+ "max_value": 5,+ "min_value": 0,+ "name": "LLM Max Retries",+ "options": null,+ "step": 1,+ "type": "LLM",+ "ui_element": "number",+ "value": 2,+ "visible": true+ },"llm.max_tokens": {"category": "llm_parameters","description": "Maximum tokens in model responses. Automatically capped at 80% of the context window size to leave room for prompt tokens.",@@ -1624,6 +1638,20 @@"value": "ollama","visible": true},+ "llm.request_timeout": {+ "category": "llm_parameters",+ "description": "Per-operation HTTP timeout in seconds for LLM inference and embedding requests, applied to every supported provider. It bounds each connect, write and wait for the next response chunk individually; it is not a cap on the total duration of a response. A peer that keeps sending data can exceed this value, including for a non-streaming response. Connecting is separately capped at 5 seconds for OpenAI-compatible and Ollama clients; Anthropic inference uses this value for connecting too. There is no upper limit; values below 1 second are clamped to 1, and non-numeric or non-finite values use the default. Model discovery is unaffected: OpenAI-compatible providers and Anthropic use 30 seconds, while Google and Ollama retain provider-specific limits of 10 and 5 seconds, respectively.",+ "editable": true,+ "max_value": null,+ "min_value": 1,+ "name": "LLM Request Inactivity Timeout (seconds)",+ "options": null,+ "step": 1,+ "type": "LLM",+ "ui_element": "number",+ "value": 1800,+ "visible": true+ },"llm.requesty.api_key": {"category": "llm_general","description": "API key to use for the Requesty provider.",diff --git a/tests/settings/test_llm_request_timeout_default.py b/tests/settings/test_llm_request_timeout_default.pynew file mode 100644index 0000000000..bd49f73638--- /dev/null+++ b/tests/settings/test_llm_request_timeout_default.py@@ -0,0 +1,100 @@+"""Focused tests for the ``llm.request_timeout`` shared default setting."""++from local_deep_research.settings.manager import SettingsManager+++def test_llm_request_timeout_default_metadata():+ """The default carries the shared editable numeric contract.++ Given the JSON defaults loaded by SettingsManager with no DB,+ when the ``llm.request_timeout`` entry is selected,+ then its projected metadata dict matches the expected contract.+ """+ # Given+ defaults = SettingsManager(db_session=None).default_settings++ # When+ setting = defaults["llm.request_timeout"]++ # Then+ projected_keys = (+ "category",+ "editable",+ "max_value",+ "min_value",+ "name",+ "options",+ "step",+ "type",+ "ui_element",+ "value",+ "visible",+ )+ projected = {key: setting[key] for key in projected_keys}+ assert projected == {+ "category": "llm_parameters",+ "editable": True,+ "max_value": None,+ "min_value": 1,+ "name": "LLM Request Inactivity Timeout (seconds)",+ "options": None,+ "step": 1,+ "type": "LLM",+ "ui_element": "number",+ "value": 1800,+ "visible": True,+ }+++def test_llm_max_retries_default_metadata():+ """``llm.max_retries`` is declared with enforceable bounds.++ Given the JSON defaults loaded by SettingsManager with no DB,+ when the ``llm.max_retries`` entry is selected,+ then it carries numeric bounds on the number of additional attempts.+ Per-operation timeouts and SDK retry delays also affect elapsed time.+ """+ # Given+ defaults = SettingsManager(db_session=None).default_settings++ # When+ setting = defaults["llm.max_retries"]++ # Then+ assert setting["ui_element"] == "number"+ assert setting["min_value"] == 0+ assert setting["max_value"] == 5+ assert setting["value"] == 2+ assert setting["editable"] is True+++def test_llm_request_timeout_description_does_not_promise_a_total_bound():+ """The description must describe a per-operation bound, not a total one."""+ # Given+ defaults = SettingsManager(db_session=None).default_settings++ # When+ description = defaults["llm.request_timeout"]["description"]++ # Then+ assert "Per-operation" in description+ assert "it is not a cap on the total duration of a response" in description+ assert "embedding" in description+++def test_llm_request_timeout_description_scopes_discovery_limits():+ # Given+ defaults = SettingsManager(db_session=None).default_settings++ # When+ description = defaults["llm.request_timeout"]["description"]++ # Then+ assert (+ "OpenAI-compatible providers and Anthropic use 30 seconds"+ in description+ )+ assert (+ "Google and Ollama retain provider-specific limits of 10 and 5 seconds"+ in description+ )diff --git a/tests/settings/test_settings_defaults_integrity.py b/tests/settings/test_settings_defaults_integrity.pyindex 9b32169dde..1c346ac367 100644--- a/tests/settings/test_settings_defaults_integrity.py+++ b/tests/settings/test_settings_defaults_integrity.py@@ -109,8 +109,6 @@ KNOWN_MISSING_DEFAULTS = {"app.api_rate_limit","app.default_theme","app.enable_api",- "llm.max_retries",- "llm.request_timeout","llm.streaming","llm.openai.api_base","llm.openai.organization",@@ -177,6 +175,8 @@ def collect_consumed_setting_keys() -> set:# policy module's _get_setting_value(snapshot, "key", default).r'_get_setting(?:_value)?\([^,]+,\s*["\']([^"\']+)["\']',r'get_setting_from_snapshot\(\s*["\']([^"\']+)["\']',+ # Bounded numeric readers in llm/providers/_helpers.py.+ r'_resolve_bounded_number\([^,]+,\s*["\']([^"\']+)["\']',r'get_bool_setting\(\s*["\']([^"\']+)["\']',r'get_bool_setting_from_snapshot\(\s*["\']([^"\']+)["\']',r'extract_setting_value\([^,]+,\s*["\']([^"\']+)["\']',diff --git a/tests/test_openai_api_key_usage.py b/tests/test_openai_api_key_usage.pyindex 401ad03c06..9b01dae9ef 100644--- a/tests/test_openai_api_key_usage.py+++ b/tests/test_openai_api_key_usage.py@@ -223,7 +223,10 @@ class TestOpenAIAPIKeyUsage:# Verify retry configurationcall_args = mock_openai.call_argsassert call_args[1].get("max_retries") == 3- assert call_args[1].get("request_timeout") == 60+ # A hashable ``(connect, read, write, pool)`` tuple — connect is+ # capped separately so a blackholed endpoint fails fast, and the+ # shape keeps langchain's cached httpx client reachable.+ assert call_args[1].get("request_timeout") == (5.0, 60, 60, 60)def test_openai_organization_id(self, openai_settings_snapshot):"""Test OpenAI organization ID configuration."""@@ -278,7 +281,7 @@ class TestOpenAIAPIKeyUsage:# Additional parametersassert call_kwargs["streaming"] is Trueassert call_kwargs["max_retries"] == 5- assert call_kwargs["request_timeout"] == 120+ assert call_kwargs["request_timeout"] == (5.0, 120, 120, 120)assert call_kwargs["openai_organization"] == "org-123"# Verify max_tokens was set