Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Drop-http-preflight-probe · L-260914-21d3f3 — drop blocking HEAD probe from pre-check (#1200)
Thomas Hebrardb654339369ce
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex def3f1c39..2d786a3ab 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,11 @@- **`pretty_print_mode` in `[runtime.log]` (Breaking)**: `rich` (the default), `poor` or `silent`, applied at boot beside `log_mode`, so a host with no console turns the "Output of pipe" panels off in configuration rather than in code; a silent printer builds no Rich renderable and does not measure the terminal to size one. Teardown returns the mode to whatever the process held before that boot rather than to `rich`, a boot that fails part-way included, so a process that set the mode itself before booting gets its own mode back; a mode assigned after boot is released too. `pipelex doctor` applies the key too. Boot replaces a `PrettyPrinter.mode` assigned in code before it, so set the key instead.+### Changed++- **Remote inputs are checked the way the web checks them (Breaking)**: an http(s) URL on a `Document` or `Image` input is validated for syntax when the inputs are shaped, and refused with `PipelineInputUrlInvalidError` when it does not parse; whether the resource exists is decided by the operator that fetches it, which now raises a caller-facing `RemoteFileFetchError` naming the URL and what the server or the network answered, instead of a raw `httpx` error. The former pre-flight HEAD probe is gone: on the hosted runner it ran as workflow code, so a host that stalled the request tripped Temporal's deadlock detector and the run ended `TIMED_OUT` with no error, and its outcome only ever produced a log line. A local file path is still required to exist before the run starts.+- **The User-Agent is `Pipelex/<version>`**: the `(https://pipelex.com)` suffix is gone. Product and version is the shape a browser or an SDK sends; the crawler convention of a URL in parentheses is what bot walls key on, and the brand sites that stalled the old string until the timeout serve the new one in under a second.+### Fixed- **Poor pretty-print mode**: a panel whose title is wider than the terminal no longer hangs the run in an endless wrapping loop, which every operator pipe's "Output of pipe" panel did on an 80-column headless console, and no longer draws a frame wider than the terminal either — an over-wide title is elided the way Rich's own panel truncates one. Titles print without their Rich markup, including on the branch a bare url takes, which is where every image output lands; a title Rich reads as an unmatched closing tag, such as a path in square brackets, prints as it stands instead of raising on the execution path. An output prints as its plain rendering instead of a Rich object's repr.diff --git a/docs/errors/execution-and-runtime.md b/docs/errors/execution-and-runtime.mdindex 247752515..9e6804845 100644--- a/docs/errors/execution-and-runtime.md+++ b/docs/errors/execution-and-runtime.md@@ -32,6 +32,7 @@ own page. Classes are grouped by subsystem.- [`PipeStackOverflowError`](pipe-stack-overflow-error.md) — Pipe stack overflow- [`PipelineExecutionError`](pipeline-execution-error.md) — Pipeline execution- [`PipelineInputContentError`](pipeline-input-content-error.md) — Pipeline input content+- [`PipelineInputUrlInvalidError`](pipeline-input-url-invalid-error.md) — Pipeline input url invalid- [`PipelineInputUrlMissingError`](pipeline-input-url-missing-error.md) — Pipeline input url missing- [`PipelineManagerAlreadyExistsError`](pipeline-manager-already-exists-error.md) — Pipeline manager already exists- [`PipelineManagerNotFoundError`](pipeline-manager-not-found-error.md) — Pipeline manager not founddiff --git a/docs/errors/pipeline-input-url-invalid-error.md b/docs/errors/pipeline-input-url-invalid-error.mdnew file mode 100644index 000000000..d6297c495--- /dev/null+++ b/docs/errors/pipeline-input-url-invalid-error.md@@ -0,0 +1,22 @@+---+title: "Pipeline input url invalid"+description: "Reference for the `PipelineInputUrlInvalidError` Pipelex error class."+---++<!-- pipelex:generated -->++# Pipeline input url invalid++An Image/Document input carries an http(s) url that does not parse as one.++| Field | Value |+|---|---|+| `error_type` | `PipelineInputUrlInvalidError` |+| `title` | Pipeline input url invalid |+| `type_uri` | `https://docs.pipelex.com/latest/errors/pipeline-input-url-invalid-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.pipeline.exceptions` |+| Parent class | [`PipelineInputContentError`](pipeline-input-content-error.md) |+| `user_action` | `change_input` — Provide a well-formed http(s) URL: a scheme, a host, and no whitespace. |++[Back to Error Reference](index.md)diff --git a/docs/errors/platform-and-tooling.md b/docs/errors/platform-and-tooling.mdindex 9bd913c05..3b63207ee 100644--- a/docs/errors/platform-and-tooling.md+++ b/docs/errors/platform-and-tooling.md@@ -37,6 +37,7 @@ own page. Classes are grouped by subsystem.- [`JsonTypeError`](json-type-error.md) — Json type- [`ModuleFileError`](module-file-error.md) — Module file- [`PyPdfium2RendererError`](py-pdfium2-renderer-error.md) — Py pdfium 2 renderer+- [`RemoteFileFetchError`](remote-file-fetch-error.md) — Remote file could not be fetched- [`SecretNotFoundError`](secret-not-found-error.md) — Secret not found- [`SsrfBlockedError`](ssrf-blocked-error.md) — Outbound request blocked (SSRF guard)- [`StorageConfigError`](storage-config-error.md) — Storage configdiff --git a/docs/errors/remote-file-fetch-error.md b/docs/errors/remote-file-fetch-error.mdnew file mode 100644index 000000000..ca2cfde89--- /dev/null+++ b/docs/errors/remote-file-fetch-error.md@@ -0,0 +1,22 @@+---+title: "Remote file could not be fetched"+description: "Reference for the `RemoteFileFetchError` Pipelex error class."+---++<!-- pipelex:generated -->++# Remote file could not be fetched++A remote file an input or a pipe referenced could not be fetched.++| Field | Value |+|---|---|+| `error_type` | `RemoteFileFetchError` |+| `title` | Remote file could not be fetched |+| `type_uri` | `https://docs.pipelex.com/latest/errors/remote-file-fetch-error/` |+| `error_domain` | `input` |+| Defined in | `pipelex.tools.misc.exceptions` |+| Parent class | [`ToolError`](tool-error.md) |+| `user_action` | `change_input` — Check that the URL serves the file to a plain HTTP client, or upload the file and reference it instead. |++[Back to Error Reference](index.md)diff --git a/pipelex/pipeline/exceptions.py b/pipelex/pipeline/exceptions.pyindex c1be6f79c..c34ef83c2 100644--- a/pipelex/pipeline/exceptions.py+++ b/pipelex/pipeline/exceptions.py@@ -256,6 +256,22 @@ class PipelineInputContentError(PipelexError):)+class PipelineInputUrlInvalidError(PipelineInputContentError):+ """An Image/Document input carries an http(s) url that does not parse as one.++ Caller-facing on purpose: the message repeats the url the caller supplied and+ what is wrong with its shape, both of which are the caller's own facts. Whether+ the resource behind a well-formed url exists is not decided here — the operator+ that consumes the input fetches it and reports a real failure.+ """++ _authors_caller_facing_message = True+ user_action = UserAction(+ kind=UserActionKind.CHANGE_INPUT,+ detail="Provide a well-formed http(s) URL: a scheme, a host, and no whitespace.",+ )++class PipelineInputUrlMissingError(PipelineInputContentError):"""An Image/Document input carries a blank url.diff --git a/pipelex/pipeline/input_normalizer.py b/pipelex/pipeline/input_normalizer.pyindex dc37b854b..03233e758 100644--- a/pipelex/pipeline/input_normalizer.py+++ b/pipelex/pipeline/input_normalizer.py@@ -17,12 +17,13 @@ from pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.pipeline.exceptions import PipelineInputContentError, PipelineInputUrlMissingError+from pipelex.pipeline.exceptions import PipelineInputContentError, PipelineInputUrlInvalidError, PipelineInputUrlMissingErrorfrom pipelex.runtime_hub import get_storage_providerfrom pipelex.tools.misc.file_utils import load_binary_asyncfrom pipelex.tools.misc.filetype_utils import detect_file_type_from_bytes+from pipelex.tools.misc.http_utils import validate_http_url_syntaxfrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract-from pipelex.tools.uri.resolved_uri import ResolvedBase64DataUrl, ResolvedLocalPath+from pipelex.tools.uri.resolved_uri import ResolvedBase64DataUrl, ResolvedHttpUrl, ResolvedLocalPathfrom pipelex.tools.uri.uri_resolver import resolve_uri# Type alias for content types that can have their URLs normalized@@ -224,6 +225,16 @@ async def _normalize_url_content(resolved_uri = resolve_uri(content.url)+ if isinstance(resolved_uri, ResolvedHttpUrl):+ # Syntax only, no network: the operator that consumes the input fetches it and+ # reports a real failure, which is the one honest test of a remote resource.+ try:+ validate_http_url_syntax(url=resolved_uri.url)+ except ValueError as exc:+ msg = f"{type(content).__name__} input: {exc}"+ raise PipelineInputUrlInvalidError(msg) from exc+ return content+if isinstance(resolved_uri, ResolvedBase64DataUrl):# Decode base64 data and storeraw_bytes = base64.b64decode(resolved_uri.base64_data)diff --git a/pipelex/tools/misc/exceptions.py b/pipelex/tools/misc/exceptions.pyindex d73e1a04a..6f64cd780 100644--- a/pipelex/tools/misc/exceptions.py+++ b/pipelex/tools/misc/exceptions.py@@ -2,6 +2,8 @@ from typing import Selfimport tomli+from pipelex.base_exceptions import ErrorDomain+from pipelex.cogt.inference.error_classification import UserAction, UserActionKindfrom pipelex.system.exceptions import ToolError@@ -36,3 +38,22 @@ class TomlError(ToolError):@classmethoddef from_tomli_error(cls, exc: tomli.TOMLDecodeError) -> Self:return cls(message=exc.msg, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno)+++class RemoteFileFetchError(ToolError):+ """A remote file an input or a pipe referenced could not be fetched.++ Raised where the fetch happens — inside the operator that consumes the resource, the+ one honest test that a URL serves what it claims to. The URL is caller-supplied, so+ ``error_domain = INPUT`` and the message is caller-facing: it names the URL and what+ the server or the network said (an HTTP status, a refused connection, a timeout), and+ never a resolved address.+ """++ error_domain = ErrorDomain.INPUT+ _declared_title = "Remote file could not be fetched"+ _authors_caller_facing_message = True+ user_action = UserAction(+ kind=UserActionKind.CHANGE_INPUT,+ detail="Check that the URL serves the file to a plain HTTP client, or upload the file and reference it instead.",+ )diff --git a/pipelex/tools/misc/file_fetch_utils.py b/pipelex/tools/misc/file_fetch_utils.pyindex 4e62c0642..23dc4254f 100644--- a/pipelex/tools/misc/file_fetch_utils.py+++ b/pipelex/tools/misc/file_fetch_utils.py@@ -1,6 +1,7 @@import httpxfrom httpx import Response+from pipelex.tools.misc.exceptions import RemoteFileFetchErrorfrom pipelex.tools.misc.http_utils import get_user_agent@@ -8,14 +9,34 @@ async def fetch_file_from_url_httpx(url: str,*,request_timeout: int | None = None,+ transport: httpx.AsyncBaseTransport | None = None,) -> bytes:+ """Fetch the bytes at ``url``, or raise :class:`RemoteFileFetchError` saying why not.++ Raises:+ RemoteFileFetchError: The server answered with an error status, refused the+ connection, or did not answer in time.+ """user_agent = get_user_agent()- async with httpx.AsyncClient(headers={"User-Agent": user_agent}) as client:- response: Response = await client.get(- url,- timeout=request_timeout,- follow_redirects=True,- )- response.raise_for_status() # Raise exception for 4XX/5XX status codes+ try:+ async with httpx.AsyncClient(headers={"User-Agent": user_agent}, transport=transport) as client:+ response: Response = await client.get(+ url,+ timeout=request_timeout,+ follow_redirects=True,+ )+ response.raise_for_status()+ except httpx.HTTPStatusError as exc:+ msg = f"Could not fetch '{url}': the server answered HTTP {exc.response.status_code}"+ raise RemoteFileFetchError(msg) from exc+ except httpx.TimeoutException as exc:+ msg = f"Could not fetch '{url}': the request timed out"+ raise RemoteFileFetchError(msg) from exc+ except httpx.ConnectError as exc:+ msg = f"Could not fetch '{url}': the host could not be reached"+ raise RemoteFileFetchError(msg) from exc+ except httpx.RequestError as exc:+ msg = f"Could not fetch '{url}': the request failed ({type(exc).__name__})"+ raise RemoteFileFetchError(msg) from exc- return response.content+ return response.contentdiff --git a/pipelex/tools/misc/http_utils.py b/pipelex/tools/misc/http_utils.pyindex 612a67cef..f2a3b5a6b 100644--- a/pipelex/tools/misc/http_utils.py+++ b/pipelex/tools/misc/http_utils.py@@ -1,80 +1,67 @@-import httpx+from pydantic import HttpUrl, TypeAdapter, ValidationError-from pipelex import logfrom pipelex.tools.misc.file_utils import path_existsfrom pipelex.tools.misc.package_utils import get_package_version-from pipelex.urls import URLsURL_MAX_LENGTH = 2048# URI schemes that are handled internally and should not be validated_SKIP_VALIDATION_PREFIXES = ("data:", "pipelex-storage://")-# HTTP status codes where HEAD rejection is a known server misconfiguration-# (CDNs, signed URLs, auth-gated endpoints) and a GET fallback is justified.-_HEAD_REJECTED_CODES = {403, 405}+# The syntax of an http(s) URL, as pydantic reads it: scheme, host, no whitespace, a length bound.+_HTTP_URL_ADAPTER = TypeAdapter(HttpUrl)+++def validate_http_url_syntax(*, url: str) -> None:+ """Check that ``url`` is a well-formed http(s) URL, without touching the network.++ Raises:+ ValueError: If the URL does not parse as an http(s) URL.+ """+ try:+ _HTTP_URL_ADAPTER.validate_python(url)+ except ValidationError as exc:+ first_error = exc.errors()[0]["msg"] if exc.errors() else "not a valid http(s) URL"+ msg = f"URL '{url}' is not a valid http(s) URL: {first_error}"+ raise ValueError(msg) from excdef get_user_agent() -> str:+ """The User-Agent pipelex sends when it fetches a resource on a user's behalf: ``Pipelex/<version>``.++ Product and version only, the shape a browser or an SDK sends. The crawler convention of a+ URL in parentheses is what bot walls key on: the same host that serves ``Pipelex/0.57.0``+ in under a second stalls ``Pipelex/0.57.0 (https://pipelex.com)`` until the timeout.+ """version = get_package_version()- homepage_url = URLs.homepage- return f"Pipelex/{version} ({homepage_url})"+ return f"Pipelex/{version}"def validate_url_resource_exists(url: str) -> None:- """Validate that a URL points to an existing resource.+ """Validate that a URL points to an existing resource, without touching the network.By the time a URL reaches DocumentContent/ImageContent, it should alreadybe resolved (absolute path or fully qualified URL).- For HTTP/HTTPS URLs: performs a HEAD request (falling back to a streaming GET- on 405) as a best-effort reachability probe. Failures are logged as warnings- but do NOT raise — the downstream extractor is the source of truth, and many- sites bot-block HEAD/unknown User-Agents with 403/401/429 while still serving- the actual content fine.For local file paths: checks that the file exists on disk.+ For HTTP/HTTPS URLs: no check at all. The downstream extractor is the source+ of truth for remote resources, and this function runs in the pipe router's+ input pre-check, which under an orchestrator such as Temporal is workflow+ code: a blocking request there trips the deadlock detector and the run is+ retried until it times out. A remote probe never changed the outcome anyway+ (it only logged), so there is nothing to move elsewhere.Skips validation for internal URIs (base64 data URLs, pipelex-storage://).Raises:- ValueError: If a local file path does not exist. HTTP failures only log a warning.+ ValueError: If a local file path does not exist."""if url.startswith(_SKIP_VALIDATION_PREFIXES):returnif url.startswith(("http://", "https://")):- _validate_http_url(url)- else:- _validate_local_path(url)-+ return-def _validate_http_url(url: str) -> None:- user_agent = get_user_agent()- headers = {"User-Agent": user_agent}- try:- response = httpx.head(url, timeout=10, follow_redirects=True, headers=headers)- if response.status_code in _HEAD_REJECTED_CODES:- log.verbose(f"HEAD request to '{url}' returned {response.status_code}, falling back to streaming GET")- with httpx.stream("GET", url, timeout=10, follow_redirects=True, headers=headers) as stream_response:- stream_response.raise_for_status()- else:- response.raise_for_status()- except httpx.HTTPStatusError as exc:- status_code = exc.response.status_code- msg = f"Pre-flight URL check: URL '{url}' returned HTTP {status_code} (continuing — downstream extractor will decide)"- # 401/403/429 are typical bot-block codes when servers reject HEAD/unknown User-Agents while still serving real content — keep these quiet.- if status_code in {401, 403, 429}:- log.debug(msg)- else:- log.warning(msg)- except httpx.ConnectError:- msg = f"Pre-flight URL check: URL '{url}' could not be reached (connection failed) (continuing — downstream extractor will decide)"- log.warning(msg)- except httpx.TimeoutException:- msg = f"Pre-flight URL check: URL '{url}' timed out (continuing — downstream extractor will decide)"- log.warning(msg)- except httpx.HTTPError:- msg = f"Pre-flight URL check: URL '{url}' could not be fetched (continuing — downstream extractor will decide)"- log.warning(msg)+ _validate_local_path(url)def _validate_local_path(url: str) -> None:diff --git a/subject_grants.toml b/subject_grants.tomlindex 1aa972272..84126466f 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -6440,10 +6440,6 @@ rationale = "Transforms the string; single operand."param = "data"rationale = "Transforms the data; single operand."-["pipelex/tools/misc/http_utils.py::_validate_http_url"]-param = "url"-rationale = "Validates/checks the url; single operand."-["pipelex/tools/misc/http_utils.py::_validate_local_path"]param = "url"rationale = "Validates/checks the url; single operand."diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txtindex 201caa11e..0079d38a1 100644--- a/tests/data/errors/error_identity.txt+++ b/tests/data/errors/error_identity.txt@@ -252,6 +252,7 @@ PipelexSetupError | Pipelex setup | https://docs.pipelex.com/latest/errors/pipelPipelexUnexpectedError | Unexpected internal error | https://docs.pipelex.com/latest/errors/pipelex-unexpected-error/PipelineExecutionError | Pipeline execution | https://docs.pipelex.com/latest/errors/pipeline-execution-error/PipelineInputContentError | Pipeline input content | https://docs.pipelex.com/latest/errors/pipeline-input-content-error/+PipelineInputUrlInvalidError | Pipeline input url invalid | https://docs.pipelex.com/latest/errors/pipeline-input-url-invalid-error/PipelineInputUrlMissingError | Pipeline input url missing | https://docs.pipelex.com/latest/errors/pipeline-input-url-missing-error/PipelineManagerAlreadyExistsError | Pipeline manager already exists | https://docs.pipelex.com/latest/errors/pipeline-manager-already-exists-error/PipelineManagerNotFoundError | Pipeline manager not found | https://docs.pipelex.com/latest/errors/pipeline-manager-not-found-error/@@ -270,6 +271,7 @@ ReadinessCheckError | Readiness check | https://docs.pipelex.com/latest/errors/rRemoteConfigFetchError | Remote config fetch | https://docs.pipelex.com/latest/errors/remote-config-fetch-error/RemoteConfigUnavailableError | Remote config unavailable | https://docs.pipelex.com/latest/errors/remote-config-unavailable-error/RemoteConfigValidationError | Remote config validation | https://docs.pipelex.com/latest/errors/remote-config-validation-error/+RemoteFileFetchError | Remote file could not be fetched | https://docs.pipelex.com/latest/errors/remote-file-fetch-error/ReportingManagerError | Reporting manager | https://docs.pipelex.com/latest/errors/reporting-manager-error/RetiredPluginEntryPointGroupError | Retired plugin entry point group | https://docs.pipelex.com/latest/errors/retired-plugin-entry-point-group-error/RoutingProfileBlueprintValueError | Routing profile blueprint value | https://docs.pipelex.com/latest/errors/routing-profile-blueprint-value-error/diff --git a/tests/unit/pipelex/core/stuffs/test_document_content_url_validation.py b/tests/unit/pipelex/core/stuffs/test_document_content_url_validation.pyindex 50b5dfca5..49ab7c1fa 100644--- a/tests/unit/pipelex/core/stuffs/test_document_content_url_validation.py+++ b/tests/unit/pipelex/core/stuffs/test_document_content_url_validation.py@@ -1,8 +1,7 @@"""Unit tests for URL validation in DocumentContent."""-import logging-import pytest+from pytest_mock import MockerFixturefrom pipelex.core.stuffs.document_content import DocumentContent@@ -15,12 +14,12 @@ class TestDocumentContentUrlValidation:doc = DocumentContent(url="https://this-domain-cannot-exist.invalid/file.pdf")assert doc.url == "https://this-domain-cannot-exist.invalid/file.pdf"- def test_validate_resources_unreachable_remote_url(self, caplog: pytest.LogCaptureFixture) -> None:- """validate_resources() does NOT raise for an unreachable HTTP URL: it logs a warning and lets the downstream extractor decide."""+ def test_validate_resources_remote_url_is_not_probed(self, mocker: MockerFixture) -> None:+ """validate_resources() makes no HTTP request for a remote URL: the downstream extractor is the source of truth."""+ mock_head = mocker.patch("httpx.head")doc = DocumentContent(url="https://this-domain-cannot-exist.invalid/file.pdf")- with caplog.at_level(logging.WARNING, logger="pipelex"):- doc.validate_resources()- assert any("could not be reached" in record.message for record in caplog.records)+ doc.validate_resources()+ mock_head.assert_not_called()def test_validate_resources_nonexistent_local_file(self) -> None:"""validate_resources() raises ValueError for a non-existent local file."""diff --git a/tests/unit/pipelex/core/stuffs/test_image_content_url_validation.py b/tests/unit/pipelex/core/stuffs/test_image_content_url_validation.pyindex 58e8ca90e..a342e1903 100644--- a/tests/unit/pipelex/core/stuffs/test_image_content_url_validation.py+++ b/tests/unit/pipelex/core/stuffs/test_image_content_url_validation.py@@ -1,8 +1,7 @@"""Unit tests for URL validation in ImageContent."""-import logging-import pytest+from pytest_mock import MockerFixturefrom pipelex.core.stuffs.image_content import ImageContent@@ -15,12 +14,12 @@ class TestImageContentUrlValidation:img = ImageContent(url="https://this-domain-cannot-exist.invalid/image.png")assert img.url == "https://this-domain-cannot-exist.invalid/image.png"- def test_validate_resources_unreachable_remote_url(self, caplog: pytest.LogCaptureFixture) -> None:- """validate_resources() does NOT raise for an unreachable HTTP URL: it logs a warning and lets the downstream extractor decide."""+ def test_validate_resources_remote_url_is_not_probed(self, mocker: MockerFixture) -> None:+ """validate_resources() makes no HTTP request for a remote URL: the downstream extractor is the source of truth."""+ mock_head = mocker.patch("httpx.head")img = ImageContent(url="https://this-domain-cannot-exist.invalid/image.png")- with caplog.at_level(logging.WARNING, logger="pipelex"):- img.validate_resources()- assert any("could not be reached" in record.message for record in caplog.records)+ img.validate_resources()+ mock_head.assert_not_called()def test_validate_resources_nonexistent_local_file(self) -> None:"""validate_resources() raises ValueError for a non-existent local file."""diff --git a/tests/unit/pipelex/pipeline/test_input_normalizer.py b/tests/unit/pipelex/pipeline/test_input_normalizer.pyindex bf3226235..da11b73cc 100644--- a/tests/unit/pipelex/pipeline/test_input_normalizer.py+++ b/tests/unit/pipelex/pipeline/test_input_normalizer.py@@ -24,7 +24,7 @@ from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.stuff_factory import StuffFactory-from pipelex.pipeline.exceptions import PipelineInputContentError, PipelineInputUrlMissingError+from pipelex.pipeline.exceptions import PipelineInputContentError, PipelineInputUrlInvalidError, PipelineInputUrlMissingErrorfrom pipelex.pipeline.input_normalizer import normalize_data_urls_to_storagefrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract@@ -58,6 +58,29 @@ class TestInputNormalizerUrlGuards:with pytest.raises(PipelineInputUrlMissingError, match="blank url"):await normalize_data_urls_to_storage(_memory_with_document(blank_url), storage_scope="test/scope")+ @pytest.mark.parametrize("bad_url", ["https://", "https://exa mple.com/file.pdf"])+ async def test_malformed_http_url_raises_caller_facing_input_error(self, mocker: MockerFixture, bad_url: str) -> None:+ """A malformed http(s) url is refused at shaping time, with a message STRICT disclosure keeps."""+ _patch_storage_and_config(mocker)++ with pytest.raises(PipelineInputUrlInvalidError) as exc_info:+ await normalize_data_urls_to_storage(_memory_with_document(bad_url), storage_scope="test/scope")++ strict = exc_info.value.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)+ assert "not a valid http(s) URL" in strict["message"]++ async def test_well_formed_http_url_passes_through_unchanged(self, mocker: MockerFixture) -> None:+ """A well-formed http(s) url is not fetched or probed at shaping time."""+ _patch_storage_and_config(mocker)+ mock_get = mocker.patch("httpx.AsyncClient.get")++ memory = await normalize_data_urls_to_storage(_memory_with_document("https://example.com/file.pdf"), storage_scope="test/scope")++ content = memory.get_stuff("document").content+ assert isinstance(content, DocumentContent)+ assert content.url == "https://example.com/file.pdf"+ mock_get.assert_not_called()+async def test_blank_url_message_survives_strict_disclosure(self, mocker: MockerFixture) -> None:"""The blank-url message names only the accepted schemes, so STRICT keeps it.diff --git a/tests/unit/pipelex/tools/misc/test_file_fetch_utils.py b/tests/unit/pipelex/tools/misc/test_file_fetch_utils.pynew file mode 100644index 000000000..a28f06894--- /dev/null+++ b/tests/unit/pipelex/tools/misc/test_file_fetch_utils.py@@ -0,0 +1,48 @@+import httpx+import pytest++from pipelex.base_exceptions import DisclosureMode+from pipelex.tools.misc.exceptions import RemoteFileFetchError+from pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpx++URL = "https://example.com/file.pdf"+++def _transport_answering(status_code: int, content: bytes = b"") -> httpx.MockTransport:+ return httpx.MockTransport(lambda _request: httpx.Response(status_code, content=content))+++def _transport_raising(exc: httpx.RequestError) -> httpx.MockTransport:+ def _handler(_request: httpx.Request) -> httpx.Response:+ raise exc++ return httpx.MockTransport(_handler)++[email protected](loop_scope="class")+class TestFetchFileFromUrl:+ async def test_success_returns_the_bytes(self) -> None:+ raw_bytes = await fetch_file_from_url_httpx(url=URL, transport=_transport_answering(200, b"%PDF-1.4"))+ assert raw_bytes == b"%PDF-1.4"++ @pytest.mark.parametrize("status_code", [404, 403, 500])+ async def test_error_status_raises_caller_facing_error(self, status_code: int) -> None:+ """A 4xx/5xx becomes an INPUT-domain error naming the URL and the status, kept under STRICT disclosure."""+ with pytest.raises(RemoteFileFetchError) as exc_info:+ await fetch_file_from_url_httpx(url=URL, transport=_transport_answering(status_code))++ strict = exc_info.value.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)+ assert f"HTTP {status_code}" in strict["message"]+ assert URL in strict["message"]++ @pytest.mark.parametrize(+ ("exc", "expected"),+ [+ pytest.param(httpx.ConnectError("boom"), "could not be reached", id="connect"),+ pytest.param(httpx.ReadTimeout("slow"), "timed out", id="timeout"),+ pytest.param(httpx.RemoteProtocolError("bad"), "request failed", id="protocol"),+ ],+ )+ async def test_network_failure_raises_caller_facing_error(self, exc: httpx.RequestError, expected: str) -> None:+ with pytest.raises(RemoteFileFetchError, match=expected):+ await fetch_file_from_url_httpx(url=URL, transport=_transport_raising(exc))diff --git a/tests/unit/pipelex/tools/misc/test_http_utils.py b/tests/unit/pipelex/tools/misc/test_http_utils.pyindex 3ed0a82d8..3b23199f3 100644--- a/tests/unit/pipelex/tools/misc/test_http_utils.py+++ b/tests/unit/pipelex/tools/misc/test_http_utils.py@@ -1,94 +1,76 @@-import logging-from collections.abc import Callable-from typing import NoReturn--import httpximport pytestfrom pytest_mock import MockerFixture-from pipelex.tools.misc.http_utils import validate_url_resource_exists---def _make_status_error(status_code: int) -> httpx.HTTPStatusError:- request = httpx.Request("HEAD", "https://example.test/resource")- response = httpx.Response(status_code=status_code, request=request)- return httpx.HTTPStatusError(message=f"HTTP {status_code}", request=request, response=response)---def _raise_status_error(status_code: int) -> Callable[..., NoReturn]:- def _stub(*_args: object, **_kwargs: object) -> NoReturn:- raise _make_status_error(status_code)+from pipelex.tools.misc.http_utils import get_user_agent, validate_http_url_syntax, validate_url_resource_exists- return _stub+class TestValidateUrlResourceExists:+ """The pre-check never touches the network: remote URLs pass through, local paths must exist."""-class TestValidateHttpUrl:- """Tests for HTTP URL validation HEAD/GET fallback logic and warn-only contract."""-- def test_head_success_does_not_fall_back_to_get(self, mocker: MockerFixture) -> None:- """When HEAD returns 200, no GET request is made."""- mock_head_response = mocker.MagicMock()- mock_head_response.status_code = 200- mock_head_response.raise_for_status = mocker.MagicMock()- mocker.patch("pipelex.tools.misc.http_utils.httpx.head", return_value=mock_head_response)- mock_stream = mocker.patch("pipelex.tools.misc.http_utils.httpx.stream")+ @pytest.mark.parametrize(+ "url",+ [+ pytest.param("https://example.com/file.png", id="https"),+ pytest.param("http://example.com/file.png", id="http"),+ pytest.param("https://this-domain-cannot-exist.invalid/file.pdf", id="unresolvable-host"),+ ],+ )+ def test_remote_url_is_not_probed(self, mocker: MockerFixture, url: str) -> None:+ """No HTTP request is made for a remote URL, whatever the host."""+ mock_head = mocker.patch("httpx.head")+ mock_get = mocker.patch("httpx.get")+ mock_stream = mocker.patch("httpx.stream")- validate_url_resource_exists("https://example.com/file.png")+ validate_url_resource_exists(url)- mock_head_response.raise_for_status.assert_called_once()+ mock_head.assert_not_called()+ mock_get.assert_not_called()mock_stream.assert_not_called()@pytest.mark.parametrize(- "status_code",+ "url",[- pytest.param(403, id="forbidden"),- pytest.param(405, id="method-not-allowed"),+ pytest.param("data:image/png;base64,abc123", id="data-url"),+ pytest.param("pipelex-storage://bucket/file.pdf", id="pipelex-storage"),],)- def test_head_rejection_falls_back_to_get(self, mocker: MockerFixture, status_code: int) -> None:- """When HEAD returns a rejection code (403, 405), a streaming GET is attempted."""- mock_head_response = mocker.MagicMock()- mock_head_response.status_code = status_code+ def test_internal_uri_is_skipped(self, url: str) -> None:+ validate_url_resource_exists(url)- mocker.patch("pipelex.tools.misc.http_utils.httpx.head", return_value=mock_head_response)+ def test_existing_local_path_passes(self) -> None:+ validate_url_resource_exists("pyproject.toml")- mock_get_response = mocker.MagicMock()- mock_get_response.raise_for_status = mocker.MagicMock()- mock_get_response.__enter__ = mocker.MagicMock(return_value=mock_get_response)- mock_get_response.__exit__ = mocker.MagicMock(return_value=False)- mocker.patch("pipelex.tools.misc.http_utils.httpx.stream", return_value=mock_get_response)+ def test_missing_local_path_raises(self) -> None:+ with pytest.raises(ValueError, match="does not exist"):+ validate_url_resource_exists("/nonexistent/path/to/file.png")- validate_url_resource_exists("https://example.com/file.png")-- mock_get_response.raise_for_status.assert_called_once()-- @pytest.mark.parametrize("status_code", [401, 403, 429])- def test_bot_block_status_codes_are_debug_only(- self,- status_code: int,- mocker: MockerFixture,- caplog: pytest.LogCaptureFixture,- ) -> None:- """401/403/429 are typical bot-block codes and must NOT log at WARNING."""- mocker.patch("pipelex.tools.misc.http_utils.httpx.head", side_effect=_raise_status_error(status_code))- url = "https://example.test/resource"- with caplog.at_level(logging.DEBUG, logger="pipelex"):- validate_url_resource_exists(url)- warning_records = [record for record in caplog.records if record.levelno >= logging.WARNING and url in record.message]- assert not warning_records, f"Expected no WARNING-level log for status {status_code}, got: {[record.message for record in warning_records]}"+ @pytest.mark.parametrize(+ "url",+ [+ pytest.param("https://example.com/file.pdf", id="https"),+ pytest.param("http://localhost:8000/file.pdf", id="localhost-with-port"),+ pytest.param("https://example.com/a%20b.pdf?x=1#frag", id="encoded-query-fragment"),+ ],+ )+ def test_well_formed_http_url_syntax_passes(self, url: str) -> None:+ validate_http_url_syntax(url=url)- @pytest.mark.parametrize("status_code", [404, 500, 503])- def test_other_status_codes_emit_warning(- self,- status_code: int,- mocker: MockerFixture,- caplog: pytest.LogCaptureFixture,- ) -> None:- """4xx (other than 401/403/429) and 5xx codes must surface as WARNING per the warn-only contract."""- mocker.patch("pipelex.tools.misc.http_utils.httpx.head", side_effect=_raise_status_error(status_code))- url = "https://example.test/resource"- with caplog.at_level(logging.DEBUG, logger="pipelex"):- validate_url_resource_exists(url)- warning_records = [record for record in caplog.records if record.levelno == logging.WARNING and url in record.message]- assert warning_records, f"Expected a WARNING-level log mentioning the URL for status {status_code}"- assert str(status_code) in warning_records[0].message+ @pytest.mark.parametrize(+ "url",+ [+ pytest.param("https://", id="no-host"),+ pytest.param("https://exa mple.com/file.pdf", id="space-in-host"),+ pytest.param("ftp://example.com/file.pdf", id="wrong-scheme"),+ pytest.param("not a url", id="plain-text"),+ ],+ )+ def test_malformed_http_url_syntax_raises(self, url: str) -> None:+ with pytest.raises(ValueError, match="not a valid http\\(s\\) URL"):+ validate_http_url_syntax(url=url)++ def test_user_agent_is_product_and_version_only(self) -> None:+ """No URL in parentheses: that crawler signature is what bot walls stall on."""+ user_agent = get_user_agent()+ assert user_agent.startswith("Pipelex/")+ assert "(" not in user_agent+ assert "http" not in user_agent