Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/S3-checksum-on-store · L-261022-be20e0 (#1475)
Louis Choquel7fcc6dfbdce6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 463073c6b..1f731d40b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -11,6 +11,7 @@### Changed+- **Every S3 write carries its SHA-256 (Breaking)**: the S3 storage provider sends the SHA-256 of every object it writes, so S3 refuses a body that arrived damaged and keeps the checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` returns it, and a generated file's write carries the whole digest of which its key carries the first 16 digits; its client no longer signs the payload, which the signed checksum protects. `StorageProviderAbstract.store` takes the digest as `checksum_sha256_hex` from a caller that hashed the bytes already, and hands it to `_store`, so a storage plugin adds the keyword `checksum_sha256_hex` to its `_store`, sending it when its storage can check a write's checksum and ignoring it otherwise.- **`pipelex-agent validate bundle --graph` writes into an output folder (Breaking)**: the flowchart it draws, `dry_run.html`, and `mermaidflow.html` with `--graph-format mermaidflow` or `both`, go into a new numbered folder, `<pipe_code>_output_<NN>/`, under the `results/` directory a `run bundle` of the same target writes to (beside the bundle file named, beside the bundle defining a folder's entry pipe, or in the folder's own `results/` when `--pipe` names the pipe), holding the `.gitignore` a run's output folder holds, instead of beside the bundle drawn, so validating a method no longer adds files to its tree or changes its digest. Read each file's path from `graph_files` in the envelope, the Markdown listing them under `## Graph files`, and the folder from `graph_output_dir`.- **The API server's examples send a method in `files`**: the server's README and the API Server pages send a method to `/v1/execute` and `/v1/validate` in `files`, a map from each file's path in the method's folder to its content, where they sent `mthds_contents`. The reference pages and the OpenAPI artifact now say that `mthds_contents`, with `bundle_b64`, `resources` and `/v1/validate`'s `mthds_sources`, is being retired and will be refused by a later release, so a client sends `files` now.- **A PipeFunc package-offer factory is never called on a thread running an event loop (Breaking)**: the factory a plugin registers with `PluginRegistrar.add_pipe_func_package_offer` may block for as long as its work takes and must not use an event loop of its caller's, and pipelex enforces it: a load that would call the factory on a thread running an event loop fails with `PipelexUnexpectedError` instead, so a host that calls a load itself from asynchronous code, such as a worker's activity, moves it off the loop with `run_method_load` (`pipelex.libraries.method_load_threads`) or with `acquire_library_async` (`pipelex.pipeline.execution_seams`), the twin of `acquire_library` that loads off the loop. The factory's signature is unchanged, and so is the plugin API version. `run_method_load` runs a blocking call in the method-load pool, a named pool of at most `METHOD_LOAD_MAX_THREADS` threads (four) apart from asyncio's default executor, in a copy of the caller's context, and when its caller is cancelled, the shutdown of `asyncio.run` included, withdraws a call no thread has taken yet or lets the cancellation through only once the running call has ended; the general mechanism is `WorkerThreads` (`pipelex.tools.misc.async_utils`).@@ -21,6 +22,7 @@### Fixed+- **A file name's type is the same on every Python**: `mime_type_of_file_name` answers `text/markdown` for `.md` and `.markdown`, `application/yaml` for `.yaml` and `.yml`, and the OpenDocument types for `.odt`, `.ods` and `.odp` on every supported Python, where Python 3.11's table knew no Markdown and only 3.14's knew the others, so a file of these types has the format key `md`, `yaml`, `odt`, `ods` or `odp` whatever the Python, and a bare zip named `.odt`, `.ods` or `.odp` is identified as that OpenDocument type.- **A method edited between its listing and its load is refused**: a run read its method's files again after listing them, so a file edited in between was loaded under a digest that named other bytes. Every file a load reads from a tree listed on this host, a bundle it parses, the `requirements.txt` it judges and the Python it scans, is now read against the hash the listing recorded, a load reads every file of a tree the run listed before it again, once per run, as it takes the tree, and the bundle text a CLI read for its run is checked against the listed file, so a bundle, a template or any other file of the tree edited, replaced or removed since the listing is refused before anything of the method is parsed, imported or run, with `MethodTreeChangedError` (`pipelex.libraries.exceptions`), coded `method.package_invalid` with the new reason `tree_changed` and the file's `path` in the tree; `run`, `validate` and `fix` report it as an invalid method whose item names that file. A `requirements.txt` that can no longer be read is this refusal too, where it was `requirements_invalid`. The checked reads are `read_tree_file_as_listed`, `read_tree_bundle_as_listed`, `ensure_tree_files_as_listed` and `ensure_bundle_text_as_listed` in `pipelex.libraries.method_trees`, `MethodTreeListing.loading()` marks a load reading a listing and `ensure_unchanged()` checks a tree a caller placed a bundle in, and `MthdsParser.make_pipelex_bundle_blueprint` takes the bytes it parses as `bundle_data`, beside the `bundle_path` that names them.- **Method loads leave the event loop**: every method load pipelex reaches from asynchronous code, a run's, a validation's and those of the commands that run a loop, runs in the method-load pool, where it ran on the event loop, the reading and parsing of the bundles it loads included, so listing and hashing a method's files, a sandbox's judgment of its `requirements.txt` and a fetch of a package it calls by address no longer hold up the other requests or activities the loop serves; the API server's run routes and `/v1/validate` also write a request's method tree to disk and remove it there, and judge its requirements there, so a client that disconnects never has the tree removed while the check still reads it, the crate routes resolve and project there, and the in-process validator reads a tree's entry pipe for its graph there too. A module imported by file while a method loads, such as one defining a structure class, is published and executed under a lock that the load's class discovery also holds, so two loads in parallel never take a module half executed, and a module that fails to execute leaves `sys.modules` rather than stay there half imported. `/v1/validate` of a method tree calling a package nothing shipped still resolves it as a run does, from the methods installed on the server and by a fetch on a miss, so its verdict matches the run's, and the fetch now runs off the event loop.- **Where a method's Python travels, in the docs**: the PipeFunc Requirements, PipeFunc and run pages say that the API server receives a method as a tree holding its bundles and its Python together and judges its `requirements.txt` from the tree's listing, that the runner never imports a method's Python, that a run's crate names each step's method tree by its digest and carries none of its files, that a `PipeFunc` call's files are read through the method file store the worker's host installs, and that a worker a method tree's validation is dispatched to checks the method's requirements again, where they said the server received a method's Python apart from its bundles, the Python was captured as text, the crate carried the method's Python sources and the worker never checked anything.diff --git a/api/tests/unit/test_blob_store.py b/api/tests/unit/test_blob_store.pyindex 0b1ae5548..e47d2acfa 100644--- a/api/tests/unit/test_blob_store.py+++ b/api/tests/unit/test_blob_store.py@@ -78,7 +78,7 @@ class _CountingStorage(StorageProviderAbstract):self._lookups_in_flight -= 1@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:self.unconditional_keys.append(key)self.objects[key] = datadiff --git a/docs/contribute/keyword-only-arguments.md b/docs/contribute/keyword-only-arguments.mdindex 1a48db82e..f6ba24c0e 100644--- a/docs/contribute/keyword-only-arguments.md+++ b/docs/contribute/keyword-only-arguments.md@@ -194,7 +194,7 @@ An override — skipped because of `@override`; fix (and grant) the base signatu```python@override-def _store(self, data, *, key, content_type): ... # base StorageProviderAbstract._store defines the contract and holds any grant+def _store(self, data, *, key, content_type, checksum_sha256_hex): ... # base StorageProviderAbstract._store defines the contract and holds any grant```A dunder — skipped by name; the interpreter owns the calling convention:diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.mdindex bd76c46dc..4c729de70 100644--- a/docs/under-the-hood/distributed-content-generation.md+++ b/docs/under-the-hood/distributed-content-generation.md@@ -154,7 +154,7 @@ async def generate_and_store_images(img_gen_assignment):)```-Each object is stored under `{storage_scope}/generated/<filename>`, the filename rendered from the storage method's `uri_format` (`{hash}.{extension}` by default). The extension follows the MIME type the object is stored under, so the key and the stored content type always agree: `jpg` for `image/jpeg`, `svg` for `image/svg+xml`, and `bin` for a type with no extension of its own. A document a `PipeDocGen` step prints is stored under `{storage_scope}/generated/<hash>/<filename>` instead, so that every download carries its readable name, and that name is held to the grammar every `pipelex-storage://` reader accepts, the hosted platform's included: a stem of `[A-Za-z0-9_-]`, one dot and an alphanumeric extension (`path_safe_file_name` in `pipelex/tools/storage/storage_file_name.py`). A name holding a letter outside ASCII or an inner dot stores without complaint and is then refused by such a reader, so the file could never be downloaded.+Each object is stored under `{storage_scope}/generated/<filename>`, the filename rendered from the storage method's `uri_format` (`{hash}.{extension}` by default). The extension follows the MIME type the object is stored under, so the key and the stored content type always agree: `jpg` for `image/jpeg`, `svg` for `image/svg+xml`, and `bin` for a type with no extension of its own. The `{hash}` is the first 16 hexadecimal digits of the bytes' SHA-256, and the write carries the whole digest, so S3 checks the body against it and keeps it beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` reads it (see [Checksums](storage-provider-plugins.md#checksums)). A document a `PipeDocGen` step prints is stored under `{storage_scope}/generated/<hash>/<filename>` instead, so that every download carries its readable name, and that name is held to the grammar every `pipelex-storage://` reader accepts, the hosted platform's included: a stem of `[A-Za-z0-9_-]`, one dot and an alphanumeric extension (`path_safe_file_name` in `pipelex/tools/storage/storage_file_name.py`). A name holding a letter outside ASCII or an inner dot stores without complaint and is then refused by such a reader, so the file could never be downloaded.!!! info "What crosses the boundary"`ImageContent` carries `url` (storage URI), `public_url`, `mime_type`, paired `width`/`height`, and `caption` — but never raw bytes. The `url` can be an S3 URI, HTTP URL, or local file path depending on storage configuration.diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.mdindex 4dd49c842..6e1e58720 100644--- a/docs/under-the-hood/storage-provider-plugins.md+++ b/docs/under-the-hood/storage-provider-plugins.md@@ -147,7 +147,7 @@ The duplicate detection is the same fail-loud `_add` helper the inference/orchesA third-party storage plugin is a distribution that:-1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object, and one that can create an object only when its key is free also overrides `_store_if_absent` (see [Create-only writes](#create-only-writes));+1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and takes the bytes' SHA-256 as `checksum_sha256_hex`, which it sends when its storage can check a write's checksum and ignores otherwise (see [Checksums](#checksums)), and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object, and one that can create an object only when its key is free also overrides `_store_if_absent` (see [Create-only writes](#create-only-writes));2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;3. advertises itself under the `pipelex.plugins.kernel` entry-point group — a storage provider is kernel-layer, and the group is how a plugin declares that (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means and what happens if you pick the wrong one; the pre-split `pipelex.plugins` group is retired and now fails startup):@@ -167,6 +167,12 @@ Use `pipelex plugins list` to see every discovered plugin, the entry-point groupEach built-in provider makes the condition atomic in its storage: S3 sends `If-None-Match: *` with the SHA-256 as `x-amz-checksum-sha256` and reads a `412` as the key held, and on a `409`, which S3 answers while another write or delete of the key is in flight, sends the write again after a short pause, a bounded number of times, failing once the conflict outlasts them; GCS uploads on generation 0; the local provider writes a temporary file and hard-links it to the key, which fails when the name is taken, and fails loudly when something other than a file is there or a file stands where one of the key's folders should be; the in-memory one checks its dict. A provider that does not override `_store_if_absent`, such as an external plugin written before it existed, keeps working through the default, which looks the key up with `_load_size` and then calls `_store`, so two writers racing on one free key may both write, which a caller keying objects by their content can live with since both write the same bytes, but which a storage refusing unconditional writes would refuse.+### Checksums++`store(data=…, key=…, content_type=…, checksum_sha256_hex=…)` takes the bytes' SHA-256 in lowercase hexadecimal from a caller that hashed them already and hands it to `_store`; the default `_store_if_absent` hands its checksum to `_store` too, so a provider overriding only `_store` receives it on both paths. Unlike `store_if_absent`, `store` does not check it against the bytes: the caller hashed the bytes it hands over, so hashing them again here would only repeat the work. The generated content factory is such a caller: a generated file's key carries the first 16 hexadecimal digits of its SHA-256, and the write carries the whole digest.++The S3 provider sends the SHA-256 with every write it makes, `store` and `store_if_absent` alike, so S3 refuses a body that does not hash to it, answering `BadDigest`, which the provider raises as `StorageS3Error`, and keeps the checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` returns it as `ChecksumSHA256`. A reader of the bucket therefore learns the full hash of any object the provider wrote without reading its bytes. Each write passes `ChecksumAlgorithm="SHA256"`, and with the caller's digest `ChecksumSHA256` as well, which botocore sends as the `x-amz-checksum-sha256` header; without one, botocore hashes the body as it sends it and sends the checksum as a trailer of the upload. The client never signs the payload (`payload_signing_enabled` is off). botocore already leaves it unsigned when it computes the checksum itself; with the caller's digest, the checksum header is among the request's signed headers, sent over TLS, which protects the body as well as a signed payload would, and botocore, which recognises only a checksum of its own, would otherwise hash the whole body a second time just to sign it. The other built-in providers take no SHA-256 checksum: GCS and its client library check a CRC32C or an MD5 of the upload, and a local file and the in-memory store have no checksum of their own.+### A provider is not the tenant boundaryA provider reads any key it is handed with the process's own credentials, and it should: it has no notion of whose run asked. On a host serving several tenants, what bounds a run's reads is the **read scope** the host passes with the run, which the content-generation leaves and the input seam check before they call a provider. A key outside that prefix, and any local path, is refused there, so a provider never sees it. A self-hosted deployment serving one tenant passes no read scope and reads as before. See [Distributed Content Generation](distributed-content-generation.md#what-a-leaf-may-read-the-read-scope) for the rule and where it runs.diff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.pyindex 2a0f5455a..be7d8fd20 100644--- a/pipelex/cogt/content_generation/generated_content_factory.py+++ b/pipelex/cogt/content_generation/generated_content_factory.py@@ -32,10 +32,10 @@ class GeneratedContentFactory:self,storage_scope: str,*,- data: bytes,+ sha256_hex: str,mime_type: str,) -> str:- """Build a storage key using a SHA-256 hash of the data.+ """Build a storage key from the SHA-256 of the data.Args:storage_scope: The run's opaque storage prefix — see@@ -45,7 +45,9 @@ class GeneratedContentFactory:Collapsing them into one host-supplied string is what lets a hostkey generated content by tenant instead of by uploader, withoutthe runtime learning what a tenant is.- data: The binary data to hash+ sha256_hex: The SHA-256 of the data in lowercase hexadecimal, of+ which the key carries the first 16 digits. The caller sends the+ whole digest with the write, so the bytes are hashed once.mime_type: The resolved effective MIME type — the same value theresulting content reports. It alone decides the file extension,so the key can never disagree with the content's `mime_type`@@ -58,7 +60,7 @@ class GeneratedContentFactory:`{storage_scope}/generated/<filename>`, where the filename isrendered from the configured `uri_format` (`{hash}`, `{extension}`)."""- hash_digest = hashlib.sha256(data).hexdigest()[:16]+ hash_digest = sha256_hex[:16]media_type = mime_type.split(";", 1)[0].strip().lower()match media_type:@@ -138,15 +140,17 @@ class GeneratedContentFactory:) -> DocumentContent:"""Store a produced file's bytes and return the `DocumentContent` that points at them.- The key is `{storage_scope}/generated/{hash}/{filename}`: the hash keeps two different files with the- same name apart, and ending in the human filename means every backend's URL, and so every download, is- called `invoice-INV-2026-0142.pdf` rather than a hash. The caller owns the filename's safety: it must be- one name every `pipelex-storage://` reader accepts, as `path_safe_file_name` makes it, or a host that- checks the key, such as the hosted platform, refuses to serve the file.+ The key is `{storage_scope}/generated/{hash}/{filename}`: the hash, the first 16 hexadecimal digits of the+ bytes' SHA-256, keeps two different files with the same name apart, and ending in the human filename means+ every backend's URL, and so every download, is called `invoice-INV-2026-0142.pdf` rather than a hash. The+ whole digest goes with the write, so a storage that checks it, as S3 does, keeps it beside the file. The+ caller owns the filename's safety: it must be one name every `pipelex-storage://` reader accepts, as+ `path_safe_file_name` makes it, or a host that checks the key, such as the hosted platform, refuses to serve+ the file."""- hash_digest = hashlib.sha256(data).hexdigest()[:16]- storage_key = f"{storage_scope}/{GENERATED_CONTENT_LEAF}/{hash_digest}/{filename}"- url = await self.storage_provider.store(data=data, key=storage_key, content_type=mime_type)+ sha256_hex = hashlib.sha256(data).hexdigest()+ storage_key = f"{storage_scope}/{GENERATED_CONTENT_LEAF}/{sha256_hex[:16]}/{filename}"+ url = await self.storage_provider.store(data=data, key=storage_key, content_type=mime_type, checksum_sha256_hex=sha256_hex)public_url = await self.storage_provider.public_url(uri=url)return DocumentContent(url=url, public_url=public_url, mime_type=mime_type, filename=filename)@@ -216,12 +220,13 @@ class GeneratedContentFactory:url = remote_urlis_remote_url = Trueelif actual_bytes:+ sha256_hex = hashlib.sha256(actual_bytes).hexdigest()storage_key = self._build_storage_key(storage_scope=storage_scope,- data=actual_bytes,+ sha256_hex=sha256_hex,mime_type=mime_type,)- url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type)+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type, checksum_sha256_hex=sha256_hex)is_remote_url = Falseelse:msg = "No URL or bytes found"@@ -244,12 +249,13 @@ class GeneratedContentFactory:else:if not declared_mime_type and fetched_mime_type:mime_type = fetched_mime_type+ sha256_hex = hashlib.sha256(actual_bytes).hexdigest()storage_key = self._build_storage_key(storage_scope=storage_scope,- data=actual_bytes,+ sha256_hex=sha256_hex,mime_type=mime_type,)- url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type)+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type, checksum_sha256_hex=sha256_hex)public_url = await self.storage_provider.public_url(uri=url)elif not is_remote_url:public_url = await self.storage_provider.public_url(uri=url)diff --git a/pipelex/tools/misc/filetype_utils.py b/pipelex/tools/misc/filetype_utils.pyindex c50378370..ccd8731ac 100644--- a/pipelex/tools/misc/filetype_utils.py+++ b/pipelex/tools/misc/filetype_utils.py@@ -18,8 +18,6 @@ _MIME_OVERRIDES: Final[dict[str, str]] = {"image/jpeg": "jpeg","image/jpg": "jpeg","text/plain": "txt",- # Python 3.11's default table has no entry for Markdown; 3.12 added `.md`- "text/markdown": "md","application/json": "json",}@@ -31,12 +29,28 @@ OFFICE_MIME_TYPES_BY_EXTENSION: Final[dict[str, str]] = {".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",}+# The types the built-in table of one Python this package supports lacks and a later one knows, named here so that every+# Python answers the same, and the runtime never disagrees on a type with a host running another Python, such as the+# hosted platform, which types an upload from its name with `mime_type_of_file_name`: Markdown entered the table in+# Python 3.12, YAML and the OpenDocument formats in 3.14. The types are the ones those tables give.+_MIME_TYPES_LATER_PYTHONS_KNOW: Final[dict[str, str]] = {+ ".md": "text/markdown",+ ".markdown": "text/markdown",+ ".yaml": "application/yaml",+ ".yml": "application/yaml",+ ".odt": "application/vnd.oasis.opendocument.text",+ ".ods": "application/vnd.oasis.opendocument.spreadsheet",+ ".odp": "application/vnd.oasis.opendocument.presentation",+}+# A table of our own: a `MimeTypes` instance starts from Python's built-in table, never from the host's `mime.types`# files or registry, which only the module-level functions of `mimetypes` read, and it gets the types added here._MIME_DB: Final[mimetypes.MimeTypes] = mimetypes.MimeTypes()_MIME_DB.add_type("application/json", ".json", strict=True)for office_extension, office_mime_type in OFFICE_MIME_TYPES_BY_EXTENSION.items():_MIME_DB.add_type(office_mime_type, office_extension, strict=True)+for later_python_extension, later_python_mime_type in _MIME_TYPES_LATER_PYTHONS_KNOW.items():+ _MIME_DB.add_type(later_python_mime_type, later_python_extension, strict=True)class FileType(BaseModel):@@ -124,7 +138,9 @@ def mime_type_of_file_name(*, file_name: str) -> str | None:"""The MIME type a file's name implies by its extension, or None when the extension is not known.Read from this module's own table, which names the Office Open XML types, so an `.xlsx`, `.docx` or `.pptx` file is- stored under its type whatever the host's `mimetypes` table holds, and never from the host's own files.+ stored under its type whatever the host's `mimetypes` table holds, and never from the host's own files. It also+ names the types only a later Python's table knows, such as `text/markdown` for `.md`, so the answer is the same on+ every Python this package supports.Args:file_name: The file's name, or a path or storage key ending in it.diff --git a/pipelex/tools/storage/gcp_storage_provider.py b/pipelex/tools/storage/gcp_storage_provider.pyindex 917e62e45..3836f5fd7 100644--- a/pipelex/tools/storage/gcp_storage_provider.py+++ b/pipelex/tools/storage/gcp_storage_provider.py@@ -269,13 +269,15 @@ class GcpStorageProvider(StorageProviderAbstract):raise StorageGcpError(msg) from exc@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:"""Store bytes to a GCS object.Args:data: The bytes to store.key: Storage key (without scheme prefix).content_type: Optional MIME type for the object.+ checksum_sha256_hex: Ignored: GCS takes no SHA-256 checksum, and the client library checks the upload's own+ CRC32C or MD5 on the way.Raises:StorageGcpError: If the GCS operation fails.diff --git a/pipelex/tools/storage/in_memory_storage_provider.py b/pipelex/tools/storage/in_memory_storage_provider.pyindex 91e03bdcd..6f3cc7c99 100644--- a/pipelex/tools/storage/in_memory_storage_provider.py+++ b/pipelex/tools/storage/in_memory_storage_provider.py@@ -60,13 +60,14 @@ class InMemoryStorageProvider(RootModel[InMemoryStorageRoot], StorageProviderAbsreturn len(self.root[key])@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:"""Store bytes in memory.Args:data: The bytes to store.key: Storage key (without scheme prefix).content_type: Ignored for in-memory storage.+ checksum_sha256_hex: Ignored: memory has no checksum of its own to send it to."""self.root[key] = datadiff --git a/pipelex/tools/storage/local_storage_provider.py b/pipelex/tools/storage/local_storage_provider.pyindex b382382b8..6dfb9b57d 100644--- a/pipelex/tools/storage/local_storage_provider.py+++ b/pipelex/tools/storage/local_storage_provider.py@@ -146,7 +146,7 @@ class LocalStorageProvider(StorageProviderAbstract):os.fsync(file_handle.fileno())@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:"""Store bytes to a file, published whole or not at all.The bytes are written to a temporary file beside the target, flushed to disk, then renamed onto the target,@@ -158,6 +158,7 @@ class LocalStorageProvider(StorageProviderAbstract):data: The bytes to store.key: Storage key (relative path, without scheme prefix).content_type: Ignored for local storage.+ checksum_sha256_hex: Ignored: a file has no checksum of its own to send it to.Raises:StorageInvalidUriError: If the key is invalid.diff --git a/pipelex/tools/storage/s3_storage_provider.py b/pipelex/tools/storage/s3_storage_provider.pyindex d404c8e41..5ac6849a8 100644--- a/pipelex/tools/storage/s3_storage_provider.py+++ b/pipelex/tools/storage/s3_storage_provider.py@@ -46,6 +46,26 @@ def checksum_sha256_header_value(*, sha256_hex: str) -> str:return base64.b64encode(bytes.fromhex(sha256_hex)).decode("ascii")+def _checksum_sha256_put_params(*, checksum_sha256_hex: str | None) -> dict[str, str]:+ """The `PutObject` parameters that make S3 check a write's SHA-256 and keep it beside the object.++ Every write the provider makes carries them, so S3 refuses a body that does not hash to the checksum and stores the+ checksum, which a `HeadObject` with `ChecksumMode=ENABLED` returns as `ChecksumSHA256`. With the digest, it goes as+ the `x-amz-checksum-sha256` header, and the bytes are not hashed again. Without it, `ChecksumAlgorithm` alone makes+ botocore hash the body as it sends it and send the checksum as a trailer of the upload.++ Args:+ checksum_sha256_hex: The SHA-256 of the body in lowercase hexadecimal, or None for botocore to compute it.++ Returns:+ The parameters to add to a `put_object` call.+ """+ params = {"ChecksumAlgorithm": "SHA256"}+ if checksum_sha256_hex is not None:+ params["ChecksumSHA256"] = checksum_sha256_header_value(sha256_hex=checksum_sha256_hex)+ return params++class S3StorageProvider(StorageProviderAbstract):"""Storage provider implementation for AWS S3 storage.@@ -116,7 +136,13 @@ class S3StorageProvider(StorageProviderAbstract):# shared host. A content security policy can allow a bucket only by its own host, so pin the virtual# style: `<bucket>.s3.<region>.amazonaws.com`. botocore still falls back to path-style for a bucket# name that cannot be a hostname, a dotted one included.- config = AioConfig(signature_version="s3v4", s3={"addressing_style": "virtual"})+ #+ # Every write carries the body's SHA-256, which S3 checks the body against (`_checksum_sha256_put_params`).+ # When botocore computes it, it goes as a trailer of an upload botocore already leaves unsigned. When the+ # caller gave it, it goes as the signed `x-amz-checksum-sha256` header, and botocore, which recognises only+ # a checksum of its own, would hash the whole body once more just to sign it. Over TLS the signed checksum+ # protects the body as well as a signed payload would, so the payload is never signed.+ config = AioConfig(signature_version="s3v4", s3={"addressing_style": "virtual", "payload_signing_enabled": False})return {"service_name": "s3",@@ -274,13 +300,18 @@ class S3StorageProvider(StorageProviderAbstract):return content_length@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:- """Store bytes to an S3 object.+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:+ """Store bytes to an S3 object, with S3 checking their SHA-256.++ The PUT carries the SHA-256, so S3 refuses a body that arrived damaged, as a `BadDigest`, and keeps the+ checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` reads it.Args:data: The bytes to store.key: Storage key (without scheme prefix).content_type: Optional MIME type for the object.+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, or None for botocore to compute it as+ it sends the bytes.Raises:StorageS3Error: If the S3 operation fails (any ClientError or BotoCoreError).@@ -303,6 +334,7 @@ class S3StorageProvider(StorageProviderAbstract):}if content_type:put_params["ContentType"] = content_type+ put_params.update(_checksum_sha256_put_params(checksum_sha256_hex=checksum_sha256_hex))await client.put_object(**put_params)except client.exceptions.NoSuchBucket as exc:msg = f"Bucket not found in S3: '{self._bucket_name}'"@@ -323,8 +355,9 @@ class S3StorageProvider(StorageProviderAbstract):"""Create an S3 object unless one exists at the key, with S3 checking the condition and the checksum.The PUT carries `If-None-Match: *`, so S3 writes it only when the key holds no object and answers `412`- otherwise, which is the key already held. With a checksum, it carries the SHA-256 as `x-amz-checksum-sha256`,- and S3 refuses a body whose hash differs, so the object holds exactly the bytes the checksum names.+ otherwise, which is the key already held. It carries the SHA-256, as `x-amz-checksum-sha256` when the caller+ gave it and computed by botocore otherwise, and S3 refuses a body whose hash differs, so the object holds+ exactly the bytes the checksum names, and keeps the checksum beside it.A `409 ConditionalRequestConflict` means another operation on the key, a conditional write or a delete, wasin flight, and S3's answer to it is to retry: the PUT is sent again after a short pause, a bounded number of@@ -336,7 +369,8 @@ class S3StorageProvider(StorageProviderAbstract):data: The bytes to store, already checked against their checksum.key: Storage key (without scheme prefix).content_type: Optional MIME type for the object.- checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, sent for S3 to check, or None.+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, sent for S3 to check, or None for+ botocore to compute it as it sends the bytes.Returns:True when this call created the object, False when the key already held one.@@ -361,8 +395,7 @@ class S3StorageProvider(StorageProviderAbstract):}if content_type:put_params["ContentType"] = content_type- if checksum_sha256_hex is not None:- put_params["ChecksumSHA256"] = checksum_sha256_header_value(sha256_hex=checksum_sha256_hex)+ put_params.update(_checksum_sha256_put_params(checksum_sha256_hex=checksum_sha256_hex))retry_delays = CONDITIONAL_CONFLICT_RETRY_DELAYS_SECONDSconditional_conflict: ClientError | None = Nonediff --git a/pipelex/tools/storage/storage_provider_abstract.py b/pipelex/tools/storage/storage_provider_abstract.pyindex d31d352dd..72abcc6da 100644--- a/pipelex/tools/storage/storage_provider_abstract.py+++ b/pipelex/tools/storage/storage_provider_abstract.py@@ -28,7 +28,7 @@ class StorageProviderAbstract(ABC):Subclasses must implement:- _load_with_metadata(key): Load data with MIME type by key (without scheme)- - _store(data, key, content_type): Store data and return URI+ - _store(data, key, content_type, checksum_sha256_hex): Store data- public_url(uri): Return human-readable link for URIA provider Pipelex ships names its kind in ``storage_kind``, the ``scheme`` its refusals carry; a plugin's provider@@ -165,29 +165,39 @@ class StorageProviderAbstract(ABC):stored_data = await self._load_with_metadata(key)return len(stored_data.data)- async def store(self, data: bytes, *, key: str, content_type: str | None = None) -> str:+ async def store(self, data: bytes, *, key: str, content_type: str | None = None, checksum_sha256_hex: str | None = None) -> str:"""Store data and return full URI with scheme.+ A provider whose storage can check a write's checksum, as S3 can, sends the bytes' SHA-256 with every write, so+ the storage refuses a body that arrived damaged and keeps the checksum beside the object. A caller that hashed+ the bytes already, for their key, passes the digest, so they are not hashed again. Unlike `store_if_absent`,+ which checks a checksum handed to it, this trusts the caller's, which is the hash of bytes the caller holds, and+ S3 still refuses a body that does not match it.+Args:data: The bytes to store.key: Storage key (without scheme prefix).content_type: Optional MIME type for the data (used by cloud providers).+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, when the caller has it, or None.Returns:Full URI with PIPELEX_STORAGE_SCHEME prefix."""uri = self._add_scheme(key)- await self._store(data=data, key=key, content_type=content_type)+ await self._store(data=data, key=key, content_type=content_type, checksum_sha256_hex=checksum_sha256_hex)return uri@abstractmethod- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:"""Store data in storage.Args:data: The bytes to store.key: Storage key (without scheme prefix).content_type: Optional MIME type for the data.+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, or None when the caller has not hashed+ the bytes. A provider whose storage checks a write's checksum sends it, hashing the bytes itself when it+ is None; another ignores it."""async def store_if_absent(@@ -238,17 +248,15 @@ class StorageProviderAbstract(ABC):data: The bytes to store, already checked against `checksum_sha256_hex`.key: Storage key (without scheme prefix).content_type: Optional MIME type for the data.- checksum_sha256_hex: The SHA-256 of `data`, or None. This default sends it nowhere.+ checksum_sha256_hex: The SHA-256 of `data`, or None, handed on to `_store`.Returns:True when this call stored the object, False when the key already held one."""- # A plain store has nowhere to send a checksum: `store_if_absent` checked the bytes against it already.- _ = checksum_sha256_hextry:await self._load_size(key)except StorageFileNotFoundError:- await self._store(data=data, key=key, content_type=content_type)+ await self._store(data=data, key=key, content_type=content_type, checksum_sha256_hex=checksum_sha256_hex)return Truereturn Falsediff --git a/tests/integration/pipelex/plugins/test_storage_external_plugin.py b/tests/integration/pipelex/plugins/test_storage_external_plugin.pyindex 30919a53c..db9e20f91 100644--- a/tests/integration/pipelex/plugins/test_storage_external_plugin.py+++ b/tests/integration/pipelex/plugins/test_storage_external_plugin.py@@ -43,7 +43,7 @@ class _FakeExternalStorageProvider(StorageProviderAbstract):raise NotImplementedError@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:raise NotImplementedError@overridediff --git a/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py b/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.pyindex 64d2c1d29..f13619b0d 100644--- a/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py+++ b/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py@@ -1,3 +1,5 @@+import hashlib+import pytestfrom mthds.protocol.problem.registry import CoreCodefrom pytest_mock import MockerFixture@@ -10,6 +12,7 @@ from pipelex.tools.network.exceptions import SsrfBlockedErrorfrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstractFAKE_IMAGE_BYTES = b"fake-image-bytes-for-hashing"+FAKE_IMAGE_SHA256 = hashlib.sha256(FAKE_IMAGE_BYTES).hexdigest()@pytest.mark.asyncio(loop_scope="class")@@ -64,6 +67,47 @@ class TestGeneratedContentFactoryStoredObject:assert image_content.mime_type == expected_mime_typeassert image_content.url == "pipelex-storage://stored-uri"+ async def test_the_write_carries_the_whole_digest_the_key_shortens(self, mocker: MockerFixture):+ """The bytes are hashed once: the key carries the first 16 digits, and the write the whole digest.++ A storage that checks a write's SHA-256, as S3 does, keeps it beside the object, so a reader learns the+ file's full hash without reading it, which a key holding only its start cannot give.+ """+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)+ storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")+ storage_provider.public_url = mocker.AsyncMock(return_value=None)+ factory = GeneratedContentFactory(storage_provider=storage_provider)++ await factory.make_image_content(+ storage_scope="test/scope",+ raw_details=GeneratedImageRawDetails(size=None, actual_bytes=FAKE_IMAGE_BYTES, mime_type="image/png"),+ )++ store_call = storage_provider.store.await_args+ assert store_call is not None+ assert store_call.kwargs["key"] == f"test/scope/generated/{FAKE_IMAGE_SHA256[:16]}.png"+ assert store_call.kwargs["checksum_sha256_hex"] == FAKE_IMAGE_SHA256++ async def test_a_document_write_carries_the_whole_digest_its_key_shortens(self, mocker: MockerFixture):+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)+ storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")+ storage_provider.public_url = mocker.AsyncMock(return_value=None)+ factory = GeneratedContentFactory(storage_provider=storage_provider)+ document_bytes = b"%PDF-1.7 a generated document"+ document_sha256 = hashlib.sha256(document_bytes).hexdigest()++ document_content = await factory.make_document_content(+ storage_scope="test/scope", data=document_bytes, mime_type="application/pdf", filename="invoice.pdf"+ )++ storage_provider.store.assert_awaited_once_with(+ data=document_bytes,+ key=f"test/scope/generated/{document_sha256[:16]}/invoice.pdf",+ content_type="application/pdf",+ checksum_sha256_hex=document_sha256,+ )+ assert document_content.url == "pipelex-storage://stored-uri"+class TestGeneratedContentFactoryStorageKey:"""The key builder on its own, whatever its caller passes.@@ -97,7 +141,7 @@ class TestGeneratedContentFactoryStorageKey:storage_key = factory._build_storage_key( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]storage_scope="test/scope",- data=FAKE_IMAGE_BYTES,+ sha256_hex=FAKE_IMAGE_SHA256,mime_type=mime_type,)@@ -172,6 +216,7 @@ class TestGeneratedContentFactoryFetchedRemoteImage:stored_key: str = store_call.kwargs["key"]assert stored_key.endswith(f".{expected_extension}")assert store_call.kwargs["content_type"] == expected_mime_type+ assert store_call.kwargs["checksum_sha256_hex"] == FAKE_IMAGE_SHA256assert image_content.mime_type == expected_mime_type@pytest.mark.usefixtures("_remote_fetch_enabled")diff --git a/tests/unit/pipelex/tools/misc/test_file_format_keys.py b/tests/unit/pipelex/tools/misc/test_file_format_keys.pyindex e19e8f2f6..f4ba6c51f 100644--- a/tests/unit/pipelex/tools/misc/test_file_format_keys.py+++ b/tests/unit/pipelex/tools/misc/test_file_format_keys.py@@ -64,6 +64,8 @@ class TestFileFormatKeys:("image/svg+xml", "image"),("text/html", "html"),("text/markdown", "md"),+ ("application/yaml", "yaml"),+ ("application/vnd.oasis.opendocument.spreadsheet", "ods"),("text/plain; charset=utf-8", "txt"),("APPLICATION/PDF", "pdf"),],@@ -124,6 +126,7 @@ class TestFileFormatKeys:(None, "report.docx", DOCX_MIME),(None, "deck.pptx", PPTX_MIME),("application/octet-stream", "budget.xlsx", XLSX_MIME),+ (None, "minutes.odt", "application/vnd.oasis.opendocument.text"),(None, "notes.txt", None),(PPTX_MIME, "report.docx", PPTX_MIME),("application/zip", "report.docx", DOCX_MIME),diff --git a/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py b/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.pyindex 4d9421e55..fa6276666 100644--- a/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py+++ b/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py@@ -27,6 +27,27 @@ class TestMimeTypeOfFileName:def test_the_type_follows_the_extension(self, file_name: str, expected: str | None) -> None:assert mime_type_of_file_name(file_name=file_name) == expected+ @pytest.mark.parametrize(+ ("file_name", "expected"),+ [+ ("notes.md", "text/markdown"),+ ("org/assets/README.MD", "text/markdown"),+ ("notes.markdown", "text/markdown"),+ ("config.yaml", "application/yaml"),+ ("config.yml", "application/yaml"),+ ("minutes.odt", "application/vnd.oasis.opendocument.text"),+ ("budget.ods", "application/vnd.oasis.opendocument.spreadsheet"),+ ("deck.odp", "application/vnd.oasis.opendocument.presentation"),+ ],+ )+ def test_a_type_only_a_later_python_knows_is_known_on_every_python(self, file_name: str, expected: str) -> None:+ """Python 3.11's table has no Markdown, and only 3.14's has YAML and OpenDocument, yet every Python answers alike.++ The hosted platform types an upload from its name with this function on its own Python, so a runtime on another+ one must give the same answer. The suite runs on each supported Python, which is what proves it.+ """+ assert mime_type_of_file_name(file_name=file_name) == expected+@pytest.mark.parametrize(("file_name", "expected"), [("report.docx", _DOCX), ("budget.xlsx", _XLSX), ("deck.pptx", _PPTX)])def test_the_office_types_never_depend_on_the_hosts_mimetypes_table(self, mocker: MockerFixture, file_name: str, expected: str) -> None:"""A host whose `mime.types` knows no office type, as a slim server image's does not, still gets them."""diff --git a/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py b/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.pyindex 52e1c5602..4dc872c07 100644--- a/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py+++ b/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py@@ -11,6 +11,8 @@ class TestMimeTypeToExtension:("text/markdown", "md"),("text/markdown; charset=utf-8", "md"),("TEXT/Markdown", "md"),+ ("application/yaml", "yaml"),+ ("application/vnd.oasis.opendocument.text", "odt"),("text/plain", "txt"),("text/csv", "csv"),("text/html", "html"),diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.pyindex ad226c813..425c16501 100644--- a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py@@ -106,7 +106,7 @@ class TestS3StorageProvider:s3_provider_no_signed_urls: S3StorageProvider,mock_aiobotocore: dict[str, Any],) -> None:- """Test that store() correctly calls the S3 put_object method."""+ """store() writes with put_object and asks for the SHA-256, which botocore computes when the caller has no digest."""test_data = b"Test data"key = "test/file.bin"content_type = "application/octet-stream"@@ -118,8 +118,42 @@ class TestS3StorageProvider:Key=key,Body=test_data,ContentType=content_type,+ ChecksumAlgorithm="SHA256",)+ async def test_store_sends_the_callers_digest_as_the_checksum(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aiobotocore: dict[str, Any],+ ) -> None:+ """A caller that hashed the bytes already hands the digest over, and it goes as `x-amz-checksum-sha256`."""+ data = b"generated bytes"+ sha256 = hashlib.sha256(data).hexdigest()++ await s3_provider_no_signed_urls.store(+ data=data, key="scope/generated/abc/file.pdf", content_type="application/pdf", checksum_sha256_hex=sha256+ )++ mock_aiobotocore["client"].put_object.assert_called_once_with(+ Bucket=S3_TEST_BUCKET,+ Key="scope/generated/abc/file.pdf",+ Body=data,+ ContentType="application/pdf",+ ChecksumAlgorithm="SHA256",+ ChecksumSHA256=base64.b64encode(hashlib.sha256(data).digest()).decode("ascii"),+ )++ async def test_the_client_never_signs_the_payload(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aiobotocore: dict[str, Any],+ ) -> None:+ """The signed checksum protects the body, so botocore is not asked to hash it again to sign it."""+ await s3_provider_no_signed_urls.store(data=b"bytes", key="scope/file.bin")++ client_config = mock_aiobotocore["session"].create_client.call_args.kwargs["config"]+ assert client_config.s3 == {"addressing_style": "virtual", "payload_signing_enabled": False}+async def test_load_returns_data(self,mocker: MockerFixture,@@ -231,6 +265,7 @@ class TestS3StorageProvider:Bucket=S3_TEST_BUCKET,Key=key,Body=b"",+ ChecksumAlgorithm="SHA256",)async def test_store_large_binary_data(@@ -578,17 +613,20 @@ class TestS3StorageProvider:Body=data,IfNoneMatch="*",ContentType="application/octet-stream",+ ChecksumAlgorithm="SHA256",ChecksumSHA256=base64.b64encode(hashlib.sha256(data).digest()).decode("ascii"),)- async def test_store_if_absent_without_a_checksum_sends_none(+ async def test_store_if_absent_without_a_checksum_lets_botocore_compute_it(self,s3_provider_no_signed_urls: S3StorageProvider,mock_aiobotocore: dict[str, Any],) -> None:assert await s3_provider_no_signed_urls.store_if_absent(data=b"bytes", key="scope/blobs/one") is True- mock_aiobotocore["client"].put_object.assert_called_once_with(Bucket=S3_TEST_BUCKET, Key="scope/blobs/one", Body=b"bytes", IfNoneMatch="*")+ mock_aiobotocore["client"].put_object.assert_called_once_with(+ Bucket=S3_TEST_BUCKET, Key="scope/blobs/one", Body=b"bytes", IfNoneMatch="*", ChecksumAlgorithm="SHA256"+ )@pytest.mark.parametrize("error_code", ["412", "PreconditionFailed"])async def test_store_if_absent_over_a_held_object_is_no_error(diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider_checksum.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider_checksum.pynew file mode 100644index 000000000..f77e61542--- /dev/null+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider_checksum.py@@ -0,0 +1,126 @@+import base64+import hashlib+import inspect+from pathlib import Path+from typing import Any, NamedTuple++import pytest+from aiobotocore.session import AioSession, get_session+from pytest_mock import MockerFixture++from pipelex.tools.storage.s3_storage_provider import S3StorageProvider++# botocore reads these at session creation; any of them set on the developer's machine could change the request+_AWS_ENV_VARS_TO_CLEAR = (+ "AWS_PROFILE",+ "AWS_DEFAULT_PROFILE",+ "AWS_SESSION_TOKEN",+ "AWS_SECURITY_TOKEN",+ "AWS_DEFAULT_REGION",+ "AWS_REGION",+ "AWS_ENDPOINT_URL",+ "AWS_ENDPOINT_URL_S3",+ "AWS_REQUEST_CHECKSUM_CALCULATION",+ "AWS_RESPONSE_CHECKSUM_VALIDATION",+)+CHECKSUM_TEST_BUCKET = "pipelex-app-dev"+CHECKSUM_TEST_REGION = "eu-west-3"+PAYLOAD = b"the bytes a run generated"+PAYLOAD_SHA256_BASE64 = base64.b64encode(hashlib.sha256(PAYLOAD).digest()).decode("ascii")+++class _RequestCapturedError(Exception):+ """Raised by the capturing hook, so the request is read as botocore would send it and never sent."""+++class _CapturedRequest(NamedTuple):+ headers: dict[str, str]+ body: bytes+++async def _read_body(body: Any) -> bytes:+ if body is None:+ return b""+ if isinstance(body, bytes | bytearray):+ return bytes(body)+ read_result = body.read()+ if inspect.isawaitable(read_result):+ read_result = await read_result+ return bytes(read_result)++[email protected](loop_scope="class")+class TestS3StorageProviderChecksumOnTheWire:+ @pytest.fixture+ def isolated_aws_environment(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:+ """Give botocore dummy credentials and keep the developer's own AWS configuration out of the test."""+ for env_var in _AWS_ENV_VARS_TO_CLEAR:+ monkeypatch.delenv(env_var, raising=False)+ empty_aws_file = tmp_path / "empty_aws_file"+ empty_aws_file.write_text("")+ monkeypatch.setenv("AWS_CONFIG_FILE", str(empty_aws_file))+ monkeypatch.setenv("AWS_SHARED_CREDENTIALS_FILE", str(empty_aws_file))+ monkeypatch.setenv("AWS_ACCESS_KEY_ID", "AKIDEXAMPLE")+ monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "dummy-secret-for-local-signing")++ @pytest.fixture+ def captured_requests(self, mocker: MockerFixture, isolated_aws_environment: None) -> list[_CapturedRequest]: # ruff: ignore[unused-method-argument]+ """The `PutObject` requests the provider's clients would send, signed by real botocore and stopped before they leave.++ Mocking `put_object` shows only the parameters the provider passes; whether botocore turns them into a checksum S3+ checks, and whether it hashes the body once more to sign it, is botocore's doing, so these tests read the request+ itself. Nothing reaches the network.+ """+ captured: list[_CapturedRequest] = []+ session: AioSession = get_session()++ async def capture(request: Any, **_kwargs: Any) -> None:+ headers = {+ str(name).lower(): value.decode("ascii") if isinstance(value, bytes) else str(value) for name, value in request.headers.items()+ }+ captured.append(_CapturedRequest(headers=headers, body=await _read_body(request.body)))+ raise _RequestCapturedError++ session.register("before-send.s3.PutObject", capture)+ mocker.patch("aiobotocore.session.get_session", return_value=session)+ return captured++ @staticmethod+ def _make_provider() -> S3StorageProvider:+ return S3StorageProvider(bucket_name=CHECKSUM_TEST_BUCKET, region=CHECKSUM_TEST_REGION, signed_urls_lifespan=None)++ async def test_a_callers_digest_goes_as_a_signed_header_and_the_body_is_not_hashed_again(self, captured_requests: list[_CapturedRequest]) -> None:+ with pytest.raises(_RequestCapturedError):+ await self._make_provider().store(+ data=PAYLOAD,+ key="scope/generated/abc/report.pdf",+ content_type="application/pdf",+ checksum_sha256_hex=hashlib.sha256(PAYLOAD).hexdigest(),+ )++ request = captured_requests[-1]+ assert request.headers["x-amz-checksum-sha256"] == PAYLOAD_SHA256_BASE64+ assert request.headers["x-amz-sdk-checksum-algorithm"] == "SHA256"+ # The signature covers the checksum, and botocore did not hash the body to sign it.+ assert "x-amz-checksum-sha256" in request.headers["authorization"]+ assert request.headers["x-amz-content-sha256"] == "UNSIGNED-PAYLOAD"+ assert "x-amz-trailer" not in request.headers+ assert request.body == PAYLOAD++ async def test_without_a_digest_botocore_sends_the_sha256_as_a_trailer(self, captured_requests: list[_CapturedRequest]) -> None:+ with pytest.raises(_RequestCapturedError):+ await self._make_provider().store(data=PAYLOAD, key="scope/assets/upload.txt", content_type="text/plain")++ request = captured_requests[-1]+ assert request.headers["x-amz-sdk-checksum-algorithm"] == "SHA256"+ assert request.headers["x-amz-trailer"] == "x-amz-checksum-sha256"+ assert f"x-amz-checksum-sha256:{PAYLOAD_SHA256_BASE64}".encode("ascii") in request.body++ async def test_a_create_only_write_carries_the_checksum_beside_its_condition(self, captured_requests: list[_CapturedRequest]) -> None:+ with pytest.raises(_RequestCapturedError):+ await self._make_provider().store_if_absent(data=PAYLOAD, key="scope/blobs/one", checksum_sha256_hex=hashlib.sha256(PAYLOAD).hexdigest())++ request = captured_requests[-1]+ assert request.headers["if-none-match"] == "*"+ assert request.headers["x-amz-checksum-sha256"] == PAYLOAD_SHA256_BASE64+ assert request.headers["x-amz-content-sha256"] == "UNSIGNED-PAYLOAD"diff --git a/tests/unit/pipelex/tools/storage/test_storage_load_size.py b/tests/unit/pipelex/tools/storage/test_storage_load_size.pyindex c7af6a120..7689bdb22 100644--- a/tests/unit/pipelex/tools/storage/test_storage_load_size.py+++ b/tests/unit/pipelex/tools/storage/test_storage_load_size.py@@ -34,7 +34,7 @@ class _LoadOnlyProvider(StorageProviderAbstract):return StoredData(data=self.objects[key], mime_type=None)@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:self.objects[key] = data@overridediff --git a/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py b/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.pyindex cbf186745..d6856d561 100644--- a/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py+++ b/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py@@ -22,6 +22,7 @@ class _PlainProvider(StorageProviderAbstract):def __init__(self) -> None:self.objects: dict[str, bytes] = {}+ self.checksums: dict[str, str | None] = {}@overrideasync def _load_with_metadata(self, key: str) -> StoredData:@@ -31,8 +32,9 @@ class _PlainProvider(StorageProviderAbstract):return StoredData(data=self.objects[key], mime_type=None)@override- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:self.objects[key] = data+ self.checksums[key] = checksum_sha256_hex@overrideasync def public_url(self, uri: str) -> str | None:@@ -59,6 +61,15 @@ class TestStorageStoreIfAbsent:assert created is Trueassert await provider.load(uri=f"{PIPELEX_STORAGE_SCHEME}{KEY}") == PAYLOAD+ async def test_the_default_hands_the_checksum_on_to_the_plain_store(self) -> None:+ """A provider overriding only `_store` still receives the checksum, for a storage that can check it."""+ provider = _PlainProvider()+ sha256 = hashlib.sha256(PAYLOAD).hexdigest()++ assert await provider.store_if_absent(data=PAYLOAD, key=KEY, checksum_sha256_hex=sha256) is True++ assert provider.checksums == {KEY: sha256}+async def test_bytes_not_matching_their_checksum_are_refused_before_any_write(self, tmp_path: Path) -> None:for provider in _providers(tmp_path=tmp_path):with pytest.raises(StorageChecksumMismatchError, match="do not hash to the checksum"):