Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Generated-image-stored-untyped · L-260920-76086e (#1220)

Louis Choquelba26f8965d0d
diff --git a/.test_durations b/.test_durations
index faa68dd0b..b3711c385 100644
--- a/.test_durations
+++ b/.test_durations
@@ -3207,8 +3207,8 @@
"tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py::TestExtractGenerateDryBranch::test_dry_document_mocks_pages_without_io": 0.8999,
"tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py::TestExtractGenerateDryBranch::test_dry_image_mocks_single_page": 0.0737,
"tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py::TestExtractGenerateDryBranch::test_live_runs_provider_and_stores": 0.118,
- "tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py::TestGeneratedContentFactoryStorageKey::test_storage_key_extension_follows_resolved_mime_type[provider-mime-overrides-requested-format]": 0.4864,
- "tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py::TestGeneratedContentFactoryStorageKey::test_storage_key_extension_follows_resolved_mime_type[provider-silent-requested-format-wins]": 3.5335,
+ "tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py::TestGeneratedContentFactoryStoredObject::test_stored_object_and_key_follow_resolved_mime_type[provider-mime-overrides-requested-format]": 0.4864,
+ "tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py::TestGeneratedContentFactoryStoredObject::test_stored_object_and_key_follow_resolved_mime_type[provider-silent-requested-format-wins]": 3.5335,
"tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py::TestImgGenGenerateDryBranch::test_dry_list_returns_nb_images_mocks_without_io": 0.0427,
"tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py::TestImgGenGenerateDryBranch::test_dry_single_skips_provider_and_storage": 0.8222,
"tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py::TestImgGenGenerateDryBranch::test_dry_single_yields_one_mock_even_with_nb_images_above_one": 0.0598,
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3bcc0e93f..f2ce370e5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,14 @@
### Fixed
+- **A generated image is stored under its own media type, so a store no longer serves it as `binary/octet-stream`**: `GeneratedContentFactory.make_image_content` resolves the effective mime type once and already used it to mint the storage key's extension, but passed no `content_type` to `StorageProviderAbstract.store` on either the direct-bytes path or the fetched-remote one. The S3 provider only sets `ContentType` when it is given one, so every generated image landed untyped and S3 answered `binary/octet-stream` on the way back out — which reached any consumer that trusts the stored type, a same-origin assets route serving the object to a browser among them. Both calls now pass the resolved mime type, so the key's extension, the object's stored type and the reported `mime_type` cannot diverge.
+
+- **A generated image fetched from a remote URL is stored under the media type that URL served**: `GeneratedImageRawDetails` admits a bare URL with no mime type and no image format on the promise that the type is settled when the bytes are downloaded, and nothing kept that promise — the fetch discarded the response's `Content-Type` and the resolution fell through to the requested image format or to a hardcoded `image/jpeg`. The new `fetch_file_and_content_type_from_url_httpx` reports the declared type alongside the bytes, normalized and with its parameters stripped, and the factory stores under any `image/<subtype>` it names — not merely the formats `ImageFormat` can itself produce, since what this runtime generates is a different question from what a remote URL may serve, and a GIF or an AVIF stored as `image/jpeg` is the very error being removed. The storage key's extension follows that subtype so it keeps agreeing with the stored type. A type the caller declared still wins, and a served type that is not an image at all — the `text/html` of an error page dressed as a 200 — is dropped.
+
+- **A remote image that fails to download degrades to its URL instead of failing the pipe**: `make_image_content` guarded its fetch with `except (httpx.HTTPStatusError, httpx.RequestError)`, but `fetch_file_from_url_httpx` converts every httpx failure into `RemoteFileFetchError`, so that clause caught nothing and a 404 or a timeout on a remote image propagated out and failed the whole run. The guard now names the exception the fetch actually raises, and the fallback it was written for — log the warning, keep the remote URL — runs.
+
+- **A remote fetch honours a timeout again**: `fetch_file_from_url_httpx` passed its `request_timeout` straight to httpx, and httpx reads an explicit `timeout=None` — the parameter's own default — as *no timeout at all* rather than as its client default. Every caller that did not name one therefore had no timeout, so a hanging server blocked the caller indefinitely and the `TimeoutException` branch that raises a caller-facing `RemoteFileFetchError` was unreachable. The helper now passes httpx's `USE_CLIENT_DEFAULT` sentinel where no timeout was asked for.
+
- **A manifold error on the native routes keeps its request id under the gateway's pipelex-spelled trace header**: `extract_manifold_metadata` reads the gateway's trace id from `x-pipelex-trace-id` before the inherited `x-portkey-trace-id`, still preferring a provider's own `x-request-id` over both. The gateway emits the two spellings with the same value today, so nothing changes yet; once it drops the vendor one, such an error whose provider sent no `x-request-id` keeps a request id instead of reporting none. `extract_gateway_metadata` is unchanged and still reads the vendor spelling alone — it serves the Portkey cloud and the manifold image path, which travels on `portkey_ai` and keeps that spelling until it is ported off the SDK.
## [v0.60.0] - 2026-09-19
diff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.py
index 124fed9ec..146c46f94 100644
--- a/pipelex/cogt/content_generation/generated_content_factory.py
+++ b/pipelex/cogt/content_generation/generated_content_factory.py
@@ -1,8 +1,6 @@
import base64
import hashlib
-import httpx
-
from pipelex import log
from pipelex.cogt.content_generation.exceptions import NeitherUrlNorDataError
from pipelex.cogt.extract.extract_output import ExtractOutput
@@ -14,7 +12,8 @@ from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.system.storage_scope import GENERATED_CONTENT_LEAF
from pipelex.tools.misc.base64_utils import extract_base64_str_from_base64_url_if_possible
-from pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpx
+from pipelex.tools.misc.exceptions import RemoteFileFetchError
+from pipelex.tools.misc.file_fetch_utils import fetch_file_and_content_type_from_url_httpx
from pipelex.tools.misc.image_utils import ImageFormat
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
@@ -60,7 +59,13 @@ class GeneratedContentFactory:
case "image/webp":
extension = "webp"
case _:
- extension = "jpg"
+ # A media type this runtime does not itself produce can still arrive,
+ # because a remote URL may serve one. The extension follows the subtype
+ # so that the key keeps agreeing with the content type the object is
+ # stored under; a subtype that is not a plain token (`svg+xml`) would
+ # not make a filename, and falls back with the key alone imprecise.
+ subtype = mime_type.removeprefix("image/")
+ extension = subtype if subtype.isalnum() else "jpg"
# The `generated/` leaf is HARDCODED, and `uri_format` renders only the
# filename under it. It used to render the whole key, `{storage_scope}/…`,
# which put generated bytes loose at the root of the scope while the
@@ -76,8 +81,25 @@ class GeneratedContentFactory:
filename = uri_format.format(hash=hash_digest, extension=extension)
return f"{storage_scope}/{GENERATED_CONTENT_LEAF}/{filename}"
- async def _fetch_remote_content(self, url: str) -> bytes:
- return await fetch_file_from_url_httpx(url=url)
+ async def _fetch_remote_content(self, url: str) -> tuple[bytes, str | None]:
+ """Fetch the bytes at `url` and the image media type the server declared for them.
+
+ Any `image/<subtype>` the server names is kept, not merely the three formats
+ `ImageFormat` can itself produce: that enum says what this runtime generates and
+ converts, which is a different question from what a remote URL is allowed to
+ serve. A remote image really can be a GIF, an AVIF or a TIFF — an extract
+ provider surfacing an embedded document image is the ordinary way one arrives —
+ and storing such an object under `image/jpeg` because the enum does not list it
+ is the confident error this whole fix exists to remove.
+
+ What is dropped is everything that is not an image at all: the
+ `application/octet-stream` of a store that never learned better, or the
+ `text/html` of an error page a server dressed as a 200.
+ """
+ raw_bytes, media_type = await fetch_file_and_content_type_from_url_httpx(url)
+ if media_type and media_type.startswith("image/"):
+ return raw_bytes, media_type
+ return raw_bytes, None
async def make_image_content(
self,
@@ -114,15 +136,26 @@ class GeneratedContentFactory:
raise NeitherUrlNorDataError(msg)
# Resolve the effective mime type ONCE, before any storage key is built:
- # the provider's actual answer first (reported mime, then base64-extracted),
- # the REQUESTED image format only as a fallback. The storage-key extension
- # derives from this same resolution, so key and mime cannot diverge when a
- # provider returns a different format than requested.
+ # what the provider actually declared about these bytes first (its reported
+ # mime, then the one a base64 data-URL carries), the REQUESTED image format
+ # only as a fallback, and a hardcoded default only when nothing said anything.
+ # The storage-key extension and the content type the object is stored under
+ # both derive from this same resolution, so key, stored type and mime cannot
+ # diverge when a provider returns a different format than requested. Passing
+ # it to `store` is not optional: a store that receives no content type falls
+ # back to its own default (S3 answers `binary/octet-stream`), and every
+ # consumer that trusts the stored type then serves the wrong one.
+ #
+ # A remote URL is the one case where nothing has declared anything yet, which
+ # `GeneratedImageRawDetails` permits on purpose — the type is settled when the
+ # bytes are downloaded, below, from the response's own `Content-Type`. That
+ # answer beats the requested format and the default, and loses to a declared
+ # one: storing a fetched PNG as the `image/jpeg` somebody asked for is not a
+ # smaller error than storing it untyped, it is the same error made confidently.
+ declared_mime_type: str | None = raw_details.mime_type or base64_extracted_mime_type
mime_type: str
- if raw_details.mime_type:
- mime_type = raw_details.mime_type
- elif base64_extracted_mime_type:
- mime_type = base64_extracted_mime_type
+ if declared_mime_type:
+ mime_type = declared_mime_type
elif image_format:
mime_type = image_format.as_mime_type
else:
@@ -139,7 +172,7 @@ class GeneratedContentFactory:
data=actual_bytes,
mime_type=mime_type,
)
- url = await self.storage_provider.store(data=actual_bytes, key=storage_key)
+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type)
is_remote_url = False
else:
msg = "No URL or bytes found"
@@ -148,17 +181,23 @@ class GeneratedContentFactory:
public_url: str | None
if is_remote_url and get_config().runtime.storage.is_fetch_remote_content_enabled:
try:
- actual_bytes = await self._fetch_remote_content(url=url)
- except (httpx.HTTPStatusError, httpx.RequestError) as exc:
+ # `RemoteFileFetchError` is the ONLY exception this can raise: the fetch
+ # helper converts every httpx failure into it. Catching the httpx classes
+ # here, as this used to, caught nothing — a 404 or a timeout on a remote
+ # image escaped and failed the whole pipe instead of degrading to the URL.
+ actual_bytes, fetched_mime_type = await self._fetch_remote_content(url=url)
+ except RemoteFileFetchError as exc:
log.warning(f"Failed to fetch a remote image: {exc}")
public_url = url
else:
+ if not declared_mime_type and fetched_mime_type:
+ mime_type = fetched_mime_type
storage_key = self._build_storage_key(
storage_scope=storage_scope,
data=actual_bytes,
mime_type=mime_type,
)
- url = await self.storage_provider.store(data=actual_bytes, key=storage_key)
+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type)
public_url = await self.storage_provider.public_url(uri=url)
elif not is_remote_url:
public_url = await self.storage_provider.public_url(uri=url)
diff --git a/pipelex/tools/misc/file_fetch_utils.py b/pipelex/tools/misc/file_fetch_utils.py
index 23dc4254f..a367ee834 100644
--- a/pipelex/tools/misc/file_fetch_utils.py
+++ b/pipelex/tools/misc/file_fetch_utils.py
@@ -1,28 +1,41 @@
import httpx
-from httpx import Response
+from httpx import USE_CLIENT_DEFAULT, Response, Timeout
from pipelex.tools.misc.exceptions import RemoteFileFetchError
from pipelex.tools.misc.http_utils import get_user_agent
-async def fetch_file_from_url_httpx(
+async def fetch_file_and_content_type_from_url_httpx(
url: str,
*,
request_timeout: int | None = None,
transport: httpx.AsyncBaseTransport | None = None,
-) -> bytes:
- """Fetch the bytes at ``url``, or raise :class:`RemoteFileFetchError` saying why not.
+) -> tuple[bytes, str | None]:
+ """Fetch the bytes at ``url`` along with the media type the server declared for them.
+
+ The content type is the response's own ``Content-Type`` with its parameters stripped
+ and lowercased, or ``None`` when the server declared none. It is the only honest
+ answer about what a remote URL actually serves, so a caller that stores those bytes
+ under a media type has to ask for it rather than guess from what it requested.
+
+ Returns:
+ The response body, and the declared media type or ``None``.
Raises:
RemoteFileFetchError: The server answered with an error status, refused the
connection, or did not answer in time.
"""
user_agent = get_user_agent()
+ # httpx reads an explicit `timeout=None` as NO timeout, not as "use the client's
+ # default" — that is what its `USE_CLIENT_DEFAULT` sentinel is for. Passing the
+ # bare `None` this signature defaults to left a hanging server able to block a
+ # caller forever, and the `TimeoutException` branch below unreachable.
+ timeout = Timeout(request_timeout) if request_timeout is not None else USE_CLIENT_DEFAULT
try:
async with httpx.AsyncClient(headers={"User-Agent": user_agent}, transport=transport) as client:
response: Response = await client.get(
url,
- timeout=request_timeout,
+ timeout=timeout,
follow_redirects=True,
)
response.raise_for_status()
@@ -39,4 +52,28 @@ async def fetch_file_from_url_httpx(
msg = f"Could not fetch '{url}': the request failed ({type(exc).__name__})"
raise RemoteFileFetchError(msg) from exc
- return response.content
+ declared_content_type: str | None = response.headers.get("content-type")
+ if declared_content_type is None:
+ return response.content, None
+ media_type = declared_content_type.split(";")[0].strip().lower()
+ return response.content, media_type or None
+
+
+async def fetch_file_from_url_httpx(
+ url: str,
+ *,
+ request_timeout: int | None = None,
+ transport: httpx.AsyncBaseTransport | None = None,
+) -> bytes:
+ """Fetch the bytes at ``url``, or raise :class:`RemoteFileFetchError` saying why not.
+
+ Raises:
+ RemoteFileFetchError: The server answered with an error status, refused the
+ connection, or did not answer in time.
+ """
+ raw_bytes, _ = await fetch_file_and_content_type_from_url_httpx(
+ url,
+ request_timeout=request_timeout,
+ transport=transport,
+ )
+ return raw_bytes
diff --git a/subject_grants.toml b/subject_grants.toml
index d92bf86dd..603c70b53 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -6332,6 +6332,10 @@ rationale = "Inner helper: filters the file list; single operand."
param = "exc"
rationale = "Factory from source: builds the result from the exc; single operand."
+["pipelex/tools/misc/file_fetch_utils.py::fetch_file_and_content_type_from_url_httpx"]
+param = "url"
+rationale = "Fetches the file at the URL; single operand."
+
["pipelex/tools/misc/file_fetch_utils.py::fetch_file_from_url_httpx"]
param = "url"
rationale = "Fetches the file at the URL; single operand."
diff --git a/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py b/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py
index 3c84d29a6..598ce6690 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py
@@ -3,13 +3,15 @@ from pytest_mock import MockerFixture
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
+from pipelex.config import get_config
+from pipelex.tools.misc.exceptions import RemoteFileFetchError
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
FAKE_IMAGE_BYTES = b"fake-image-bytes-for-hashing"
@pytest.mark.asyncio(loop_scope="class")
-class TestGeneratedContentFactoryStorageKey:
+class TestGeneratedContentFactoryStoredObject:
@pytest.mark.parametrize(
("reported_mime_type", "requested_image_format", "expected_extension", "expected_mime_type"),
[
@@ -17,7 +19,7 @@ class TestGeneratedContentFactoryStorageKey:
pytest.param(None, "png", "png", "image/png", id="provider-silent-requested-format-wins"),
],
)
- async def test_storage_key_extension_follows_resolved_mime_type(
+ async def test_stored_object_and_key_follow_resolved_mime_type(
self,
mocker: MockerFixture,
reported_mime_type: str | None,
@@ -25,11 +27,13 @@ class TestGeneratedContentFactoryStorageKey:
expected_extension: str,
expected_mime_type: str,
):
- """The storage-key extension derives from the same mime resolution the result reports.
+ """The stored object's content type and the key's extension follow the reported mime.
The provider's actual mime type wins over the requested image format, and the requested
- format only fills in when the provider is silent — so the minted key's extension and the
- resulting `mime_type` can never diverge (e.g. a `.png` key holding `image/jpeg` bytes).
+ format only fills in when the provider is silent — so the minted key's extension, the
+ object's stored content type and the resulting `mime_type` can never diverge (e.g. a
+ `.png` key holding `image/jpeg` bytes, or an object a store serves as
+ `binary/octet-stream` because it was written without one).
"""
storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)
storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")
@@ -54,5 +58,97 @@ class TestGeneratedContentFactoryStorageKey:
stored_key: str = store_call.kwargs["key"]
assert stored_key.startswith("test/scope/generated/")
assert stored_key.endswith(f".{expected_extension}")
+ assert store_call.kwargs["content_type"] == expected_mime_type
assert image_content.mime_type == expected_mime_type
assert image_content.url == "pipelex-storage://stored-uri"
+
+
[email protected](loop_scope="class")
+class TestGeneratedContentFactoryFetchedRemoteImage:
+ """The fetched-remote path, where nothing has declared a media type yet.
+
+ `GeneratedImageRawDetails` deliberately admits a bare URL with no mime type and no
+ image format, on the promise that the type is settled when the bytes are downloaded.
+ These are the tests that the promise is kept: the response's own `Content-Type` is
+ what the object is stored under, rather than the requested format or a default.
+ """
+
+ @pytest.fixture
+ def _remote_fetch_enabled(self, mocker: MockerFixture) -> None:
+ mocker.patch.object(get_config().runtime.storage, "is_fetch_remote_content_enabled", True)
+
+ @pytest.mark.usefixtures("_remote_fetch_enabled")
+ @pytest.mark.parametrize(
+ ("declared_mime_type", "requested_image_format", "served_content_type", "expected_extension", "expected_mime_type"),
+ [
+ pytest.param(None, None, "image/png", "png", "image/png", id="served-type-settles-an-undeclared-image"),
+ pytest.param(None, "jpeg", "image/png", "png", "image/png", id="served-type-beats-requested-format"),
+ pytest.param(None, None, "image/gif", "gif", "image/gif", id="served-type-we-do-not-generate-is-still-stored"),
+ pytest.param(None, None, "application/octet-stream", "jpg", "image/jpeg", id="non-image-served-type-is-dropped"),
+ pytest.param(None, None, "text/html", "jpg", "image/jpeg", id="error-page-dressed-as-a-200-is-dropped"),
+ pytest.param(None, None, None, "jpg", "image/jpeg", id="silent-server-falls-back-to-default"),
+ pytest.param("image/webp", None, "image/png", "webp", "image/webp", id="declared-type-beats-served-type"),
+ ],
+ )
+ async def test_stored_object_follows_the_served_content_type(
+ self,
+ mocker: MockerFixture,
+ declared_mime_type: str | None,
+ requested_image_format: str | None,
+ served_content_type: str | None,
+ expected_extension: str,
+ expected_mime_type: str,
+ ):
+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)
+ storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")
+ storage_provider.public_url = mocker.AsyncMock(return_value="https://cdn.example/stored")
+ mocker.patch(
+ "pipelex.cogt.content_generation.generated_content_factory.fetch_file_and_content_type_from_url_httpx",
+ new=mocker.AsyncMock(return_value=(FAKE_IMAGE_BYTES, served_content_type)),
+ )
+ factory = GeneratedContentFactory(storage_provider=storage_provider)
+
+ raw_details = GeneratedImageRawDetails(
+ size=None,
+ actual_url="https://example.com/generated-image",
+ mime_type=declared_mime_type,
+ image_format=requested_image_format,
+ )
+
+ image_content = await factory.make_image_content(
+ storage_scope="test/scope",
+ raw_details=raw_details,
+ )
+
+ storage_provider.store.assert_awaited_once()
+ store_call = storage_provider.store.await_args
+ assert store_call is not None
+ stored_key: str = store_call.kwargs["key"]
+ assert stored_key.endswith(f".{expected_extension}")
+ assert store_call.kwargs["content_type"] == expected_mime_type
+ assert image_content.mime_type == expected_mime_type
+
+ @pytest.mark.usefixtures("_remote_fetch_enabled")
+ async def test_a_failed_fetch_degrades_to_the_remote_url(self, mocker: MockerFixture):
+ """A fetch that fails leaves the remote URL standing instead of failing the pipe.
+
+ Every httpx failure reaches this path as `RemoteFileFetchError`, so a fallback
+ written against the httpx classes never fired and a 404 escaped `make_image_content`.
+ """
+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)
+ storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")
+ storage_provider.public_url = mocker.AsyncMock(return_value=None)
+ mocker.patch(
+ "pipelex.cogt.content_generation.generated_content_factory.fetch_file_and_content_type_from_url_httpx",
+ new=mocker.AsyncMock(side_effect=RemoteFileFetchError("the server answered HTTP 404")),
+ )
+ factory = GeneratedContentFactory(storage_provider=storage_provider)
+
+ image_content = await factory.make_image_content(
+ storage_scope="test/scope",
+ raw_details=GeneratedImageRawDetails(size=None, actual_url="https://example.com/gone"),
+ )
+
+ storage_provider.store.assert_not_awaited()
+ assert image_content.url == "https://example.com/gone"
+ assert image_content.public_url == "https://example.com/gone"
diff --git a/tests/unit/pipelex/tools/misc/test_file_fetch_utils.py b/tests/unit/pipelex/tools/misc/test_file_fetch_utils.py
index a28f06894..cb7edc2e2 100644
--- a/tests/unit/pipelex/tools/misc/test_file_fetch_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_file_fetch_utils.py
@@ -3,13 +3,13 @@ import pytest
from pipelex.base_exceptions import DisclosureMode
from pipelex.tools.misc.exceptions import RemoteFileFetchError
-from pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpx
+from pipelex.tools.misc.file_fetch_utils import fetch_file_and_content_type_from_url_httpx, fetch_file_from_url_httpx
URL = "https://example.com/file.pdf"
-def _transport_answering(status_code: int, content: bytes = b"") -> httpx.MockTransport:
- return httpx.MockTransport(lambda _request: httpx.Response(status_code, content=content))
+def _transport_answering(status_code: int, content: bytes = b"", headers: dict[str, str] | None = None) -> httpx.MockTransport:
+ return httpx.MockTransport(lambda _request: httpx.Response(status_code, content=content, headers=headers))
def _transport_raising(exc: httpx.RequestError) -> httpx.MockTransport:
@@ -46,3 +46,53 @@ class TestFetchFileFromUrl:
async def test_network_failure_raises_caller_facing_error(self, exc: httpx.RequestError, expected: str) -> None:
with pytest.raises(RemoteFileFetchError, match=expected):
await fetch_file_from_url_httpx(url=URL, transport=_transport_raising(exc))
+
+
[email protected](loop_scope="class")
+class TestFetchFileAndContentTypeFromUrl:
+ """The declared media type, which is the only honest answer about what a URL serves."""
+
+ @pytest.mark.parametrize(
+ ("served", "expected"),
+ [
+ pytest.param("image/png", "image/png", id="plain"),
+ pytest.param("image/png; charset=binary", "image/png", id="parameters-stripped"),
+ pytest.param(" IMAGE/PNG ; q=1 ", "image/png", id="whitespace-and-case-normalized"),
+ pytest.param(None, None, id="silent-server"),
+ ],
+ )
+ async def test_the_declared_content_type_is_reported_normalized(self, served: str | None, expected: str | None) -> None:
+ headers = {"Content-Type": served} if served is not None else {}
+ raw_bytes, content_type = await fetch_file_and_content_type_from_url_httpx(
+ URL,
+ transport=_transport_answering(200, b"\x89PNG", headers=headers),
+ )
+
+ assert raw_bytes == b"\x89PNG"
+ assert content_type == expected
+
+
[email protected](loop_scope="class")
+class TestFetchTimeout:
+ """The timeout that the caller-facing timeout error depends on actually being set."""
+
+ @staticmethod
+ def _transport_recording(seen: dict[str, object]) -> httpx.MockTransport:
+ def _handler(request: httpx.Request) -> httpx.Response:
+ seen["timeout"] = request.extensions.get("timeout")
+ return httpx.Response(200, content=b"ok")
+
+ return httpx.MockTransport(_handler)
+
+ async def test_no_request_timeout_leaves_the_client_default_in_force(self) -> None:
+ """An explicit `timeout=None` would mean NO timeout in httpx, which is not the intent."""
+ seen: dict[str, object] = {}
+ await fetch_file_from_url_httpx(url=URL, transport=self._transport_recording(seen))
+
+ assert seen["timeout"] == {"connect": 5.0, "read": 5.0, "write": 5.0, "pool": 5.0}
+
+ async def test_an_explicit_request_timeout_is_honoured(self) -> None:
+ seen: dict[str, object] = {}
+ await fetch_file_from_url_httpx(url=URL, request_timeout=30, transport=self._transport_recording(seen))
+
+ assert seen["timeout"] == {"connect": 30, "read": 30, "write": 30, "pool": 30}