Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/S3-checksum-on-store · L-261022-be20e0 (#1475)

Louis Choquel7fcc6dfbdce6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 463073c6b..1f731d40b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,7 @@
### Changed
+- **Every S3 write carries its SHA-256 (Breaking)**: the S3 storage provider sends the SHA-256 of every object it writes, so S3 refuses a body that arrived damaged and keeps the checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` returns it, and a generated file's write carries the whole digest of which its key carries the first 16 digits; its client no longer signs the payload, which the signed checksum protects. `StorageProviderAbstract.store` takes the digest as `checksum_sha256_hex` from a caller that hashed the bytes already, and hands it to `_store`, so a storage plugin adds the keyword `checksum_sha256_hex` to its `_store`, sending it when its storage can check a write's checksum and ignoring it otherwise.
- **`pipelex-agent validate bundle --graph` writes into an output folder (Breaking)**: the flowchart it draws, `dry_run.html`, and `mermaidflow.html` with `--graph-format mermaidflow` or `both`, go into a new numbered folder, `<pipe_code>_output_<NN>/`, under the `results/` directory a `run bundle` of the same target writes to (beside the bundle file named, beside the bundle defining a folder's entry pipe, or in the folder's own `results/` when `--pipe` names the pipe), holding the `.gitignore` a run's output folder holds, instead of beside the bundle drawn, so validating a method no longer adds files to its tree or changes its digest. Read each file's path from `graph_files` in the envelope, the Markdown listing them under `## Graph files`, and the folder from `graph_output_dir`.
- **The API server's examples send a method in `files`**: the server's README and the API Server pages send a method to `/v1/execute` and `/v1/validate` in `files`, a map from each file's path in the method's folder to its content, where they sent `mthds_contents`. The reference pages and the OpenAPI artifact now say that `mthds_contents`, with `bundle_b64`, `resources` and `/v1/validate`'s `mthds_sources`, is being retired and will be refused by a later release, so a client sends `files` now.
- **A PipeFunc package-offer factory is never called on a thread running an event loop (Breaking)**: the factory a plugin registers with `PluginRegistrar.add_pipe_func_package_offer` may block for as long as its work takes and must not use an event loop of its caller's, and pipelex enforces it: a load that would call the factory on a thread running an event loop fails with `PipelexUnexpectedError` instead, so a host that calls a load itself from asynchronous code, such as a worker's activity, moves it off the loop with `run_method_load` (`pipelex.libraries.method_load_threads`) or with `acquire_library_async` (`pipelex.pipeline.execution_seams`), the twin of `acquire_library` that loads off the loop. The factory's signature is unchanged, and so is the plugin API version. `run_method_load` runs a blocking call in the method-load pool, a named pool of at most `METHOD_LOAD_MAX_THREADS` threads (four) apart from asyncio's default executor, in a copy of the caller's context, and when its caller is cancelled, the shutdown of `asyncio.run` included, withdraws a call no thread has taken yet or lets the cancellation through only once the running call has ended; the general mechanism is `WorkerThreads` (`pipelex.tools.misc.async_utils`).
@@ -21,6 +22,7 @@
### Fixed
+- **A file name's type is the same on every Python**: `mime_type_of_file_name` answers `text/markdown` for `.md` and `.markdown`, `application/yaml` for `.yaml` and `.yml`, and the OpenDocument types for `.odt`, `.ods` and `.odp` on every supported Python, where Python 3.11's table knew no Markdown and only 3.14's knew the others, so a file of these types has the format key `md`, `yaml`, `odt`, `ods` or `odp` whatever the Python, and a bare zip named `.odt`, `.ods` or `.odp` is identified as that OpenDocument type.
- **A method edited between its listing and its load is refused**: a run read its method's files again after listing them, so a file edited in between was loaded under a digest that named other bytes. Every file a load reads from a tree listed on this host, a bundle it parses, the `requirements.txt` it judges and the Python it scans, is now read against the hash the listing recorded, a load reads every file of a tree the run listed before it again, once per run, as it takes the tree, and the bundle text a CLI read for its run is checked against the listed file, so a bundle, a template or any other file of the tree edited, replaced or removed since the listing is refused before anything of the method is parsed, imported or run, with `MethodTreeChangedError` (`pipelex.libraries.exceptions`), coded `method.package_invalid` with the new reason `tree_changed` and the file's `path` in the tree; `run`, `validate` and `fix` report it as an invalid method whose item names that file. A `requirements.txt` that can no longer be read is this refusal too, where it was `requirements_invalid`. The checked reads are `read_tree_file_as_listed`, `read_tree_bundle_as_listed`, `ensure_tree_files_as_listed` and `ensure_bundle_text_as_listed` in `pipelex.libraries.method_trees`, `MethodTreeListing.loading()` marks a load reading a listing and `ensure_unchanged()` checks a tree a caller placed a bundle in, and `MthdsParser.make_pipelex_bundle_blueprint` takes the bytes it parses as `bundle_data`, beside the `bundle_path` that names them.
- **Method loads leave the event loop**: every method load pipelex reaches from asynchronous code, a run's, a validation's and those of the commands that run a loop, runs in the method-load pool, where it ran on the event loop, the reading and parsing of the bundles it loads included, so listing and hashing a method's files, a sandbox's judgment of its `requirements.txt` and a fetch of a package it calls by address no longer hold up the other requests or activities the loop serves; the API server's run routes and `/v1/validate` also write a request's method tree to disk and remove it there, and judge its requirements there, so a client that disconnects never has the tree removed while the check still reads it, the crate routes resolve and project there, and the in-process validator reads a tree's entry pipe for its graph there too. A module imported by file while a method loads, such as one defining a structure class, is published and executed under a lock that the load's class discovery also holds, so two loads in parallel never take a module half executed, and a module that fails to execute leaves `sys.modules` rather than stay there half imported. `/v1/validate` of a method tree calling a package nothing shipped still resolves it as a run does, from the methods installed on the server and by a fetch on a miss, so its verdict matches the run's, and the fetch now runs off the event loop.
- **Where a method's Python travels, in the docs**: the PipeFunc Requirements, PipeFunc and run pages say that the API server receives a method as a tree holding its bundles and its Python together and judges its `requirements.txt` from the tree's listing, that the runner never imports a method's Python, that a run's crate names each step's method tree by its digest and carries none of its files, that a `PipeFunc` call's files are read through the method file store the worker's host installs, and that a worker a method tree's validation is dispatched to checks the method's requirements again, where they said the server received a method's Python apart from its bundles, the Python was captured as text, the crate carried the method's Python sources and the worker never checked anything.
diff --git a/api/tests/unit/test_blob_store.py b/api/tests/unit/test_blob_store.py
index 0b1ae5548..e47d2acfa 100644
--- a/api/tests/unit/test_blob_store.py
+++ b/api/tests/unit/test_blob_store.py
@@ -78,7 +78,7 @@ class _CountingStorage(StorageProviderAbstract):
self._lookups_in_flight -= 1
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
self.unconditional_keys.append(key)
self.objects[key] = data
diff --git a/docs/contribute/keyword-only-arguments.md b/docs/contribute/keyword-only-arguments.md
index 1a48db82e..f6ba24c0e 100644
--- a/docs/contribute/keyword-only-arguments.md
+++ b/docs/contribute/keyword-only-arguments.md
@@ -194,7 +194,7 @@ An override — skipped because of `@override`; fix (and grant) the base signatu
```python
@override
-def _store(self, data, *, key, content_type): ... # base StorageProviderAbstract._store defines the contract and holds any grant
+def _store(self, data, *, key, content_type, checksum_sha256_hex): ... # base StorageProviderAbstract._store defines the contract and holds any grant
```
A dunder — skipped by name; the interpreter owns the calling convention:
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
index bd76c46dc..4c729de70 100644
--- a/docs/under-the-hood/distributed-content-generation.md
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -154,7 +154,7 @@ async def generate_and_store_images(img_gen_assignment):
)
```
-Each object is stored under `{storage_scope}/generated/<filename>`, the filename rendered from the storage method's `uri_format` (`{hash}.{extension}` by default). The extension follows the MIME type the object is stored under, so the key and the stored content type always agree: `jpg` for `image/jpeg`, `svg` for `image/svg+xml`, and `bin` for a type with no extension of its own. A document a `PipeDocGen` step prints is stored under `{storage_scope}/generated/<hash>/<filename>` instead, so that every download carries its readable name, and that name is held to the grammar every `pipelex-storage://` reader accepts, the hosted platform's included: a stem of `[A-Za-z0-9_-]`, one dot and an alphanumeric extension (`path_safe_file_name` in `pipelex/tools/storage/storage_file_name.py`). A name holding a letter outside ASCII or an inner dot stores without complaint and is then refused by such a reader, so the file could never be downloaded.
+Each object is stored under `{storage_scope}/generated/<filename>`, the filename rendered from the storage method's `uri_format` (`{hash}.{extension}` by default). The extension follows the MIME type the object is stored under, so the key and the stored content type always agree: `jpg` for `image/jpeg`, `svg` for `image/svg+xml`, and `bin` for a type with no extension of its own. The `{hash}` is the first 16 hexadecimal digits of the bytes' SHA-256, and the write carries the whole digest, so S3 checks the body against it and keeps it beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` reads it (see [Checksums](storage-provider-plugins.md#checksums)). A document a `PipeDocGen` step prints is stored under `{storage_scope}/generated/<hash>/<filename>` instead, so that every download carries its readable name, and that name is held to the grammar every `pipelex-storage://` reader accepts, the hosted platform's included: a stem of `[A-Za-z0-9_-]`, one dot and an alphanumeric extension (`path_safe_file_name` in `pipelex/tools/storage/storage_file_name.py`). A name holding a letter outside ASCII or an inner dot stores without complaint and is then refused by such a reader, so the file could never be downloaded.
!!! info "What crosses the boundary"
`ImageContent` carries `url` (storage URI), `public_url`, `mime_type`, paired `width`/`height`, and `caption` — but never raw bytes. The `url` can be an S3 URI, HTTP URL, or local file path depending on storage configuration.
diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.md
index 4dd49c842..6e1e58720 100644
--- a/docs/under-the-hood/storage-provider-plugins.md
+++ b/docs/under-the-hood/storage-provider-plugins.md
@@ -147,7 +147,7 @@ The duplicate detection is the same fail-loud `_add` helper the inference/orches
A third-party storage plugin is a distribution that:
-1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object, and one that can create an object only when its key is free also overrides `_store_if_absent` (see [Create-only writes](#create-only-writes));
+1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and takes the bytes' SHA-256 as `checksum_sha256_hex`, which it sends when its storage can check a write's checksum and ignores otherwise (see [Checksums](#checksums)), and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object, and one that can create an object only when its key is free also overrides `_store_if_absent` (see [Create-only writes](#create-only-writes));
2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;
3. advertises itself under the `pipelex.plugins.kernel` entry-point group — a storage provider is kernel-layer, and the group is how a plugin declares that (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means and what happens if you pick the wrong one; the pre-split `pipelex.plugins` group is retired and now fails startup):
@@ -167,6 +167,12 @@ Use `pipelex plugins list` to see every discovered plugin, the entry-point group
Each built-in provider makes the condition atomic in its storage: S3 sends `If-None-Match: *` with the SHA-256 as `x-amz-checksum-sha256` and reads a `412` as the key held, and on a `409`, which S3 answers while another write or delete of the key is in flight, sends the write again after a short pause, a bounded number of times, failing once the conflict outlasts them; GCS uploads on generation 0; the local provider writes a temporary file and hard-links it to the key, which fails when the name is taken, and fails loudly when something other than a file is there or a file stands where one of the key's folders should be; the in-memory one checks its dict. A provider that does not override `_store_if_absent`, such as an external plugin written before it existed, keeps working through the default, which looks the key up with `_load_size` and then calls `_store`, so two writers racing on one free key may both write, which a caller keying objects by their content can live with since both write the same bytes, but which a storage refusing unconditional writes would refuse.
+### Checksums
+
+`store(data=…, key=…, content_type=…, checksum_sha256_hex=…)` takes the bytes' SHA-256 in lowercase hexadecimal from a caller that hashed them already and hands it to `_store`; the default `_store_if_absent` hands its checksum to `_store` too, so a provider overriding only `_store` receives it on both paths. Unlike `store_if_absent`, `store` does not check it against the bytes: the caller hashed the bytes it hands over, so hashing them again here would only repeat the work. The generated content factory is such a caller: a generated file's key carries the first 16 hexadecimal digits of its SHA-256, and the write carries the whole digest.
+
+The S3 provider sends the SHA-256 with every write it makes, `store` and `store_if_absent` alike, so S3 refuses a body that does not hash to it, answering `BadDigest`, which the provider raises as `StorageS3Error`, and keeps the checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` returns it as `ChecksumSHA256`. A reader of the bucket therefore learns the full hash of any object the provider wrote without reading its bytes. Each write passes `ChecksumAlgorithm="SHA256"`, and with the caller's digest `ChecksumSHA256` as well, which botocore sends as the `x-amz-checksum-sha256` header; without one, botocore hashes the body as it sends it and sends the checksum as a trailer of the upload. The client never signs the payload (`payload_signing_enabled` is off). botocore already leaves it unsigned when it computes the checksum itself; with the caller's digest, the checksum header is among the request's signed headers, sent over TLS, which protects the body as well as a signed payload would, and botocore, which recognises only a checksum of its own, would otherwise hash the whole body a second time just to sign it. The other built-in providers take no SHA-256 checksum: GCS and its client library check a CRC32C or an MD5 of the upload, and a local file and the in-memory store have no checksum of their own.
+
### A provider is not the tenant boundary
A provider reads any key it is handed with the process's own credentials, and it should: it has no notion of whose run asked. On a host serving several tenants, what bounds a run's reads is the **read scope** the host passes with the run, which the content-generation leaves and the input seam check before they call a provider. A key outside that prefix, and any local path, is refused there, so a provider never sees it. A self-hosted deployment serving one tenant passes no read scope and reads as before. See [Distributed Content Generation](distributed-content-generation.md#what-a-leaf-may-read-the-read-scope) for the rule and where it runs.
diff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.py
index 2a0f5455a..be7d8fd20 100644
--- a/pipelex/cogt/content_generation/generated_content_factory.py
+++ b/pipelex/cogt/content_generation/generated_content_factory.py
@@ -32,10 +32,10 @@ class GeneratedContentFactory:
self,
storage_scope: str,
*,
- data: bytes,
+ sha256_hex: str,
mime_type: str,
) -> str:
- """Build a storage key using a SHA-256 hash of the data.
+ """Build a storage key from the SHA-256 of the data.
Args:
storage_scope: The run's opaque storage prefix — see
@@ -45,7 +45,9 @@ class GeneratedContentFactory:
Collapsing them into one host-supplied string is what lets a host
key generated content by tenant instead of by uploader, without
the runtime learning what a tenant is.
- data: The binary data to hash
+ sha256_hex: The SHA-256 of the data in lowercase hexadecimal, of
+ which the key carries the first 16 digits. The caller sends the
+ whole digest with the write, so the bytes are hashed once.
mime_type: The resolved effective MIME type — the same value the
resulting content reports. It alone decides the file extension,
so the key can never disagree with the content's `mime_type`
@@ -58,7 +60,7 @@ class GeneratedContentFactory:
`{storage_scope}/generated/<filename>`, where the filename is
rendered from the configured `uri_format` (`{hash}`, `{extension}`).
"""
- hash_digest = hashlib.sha256(data).hexdigest()[:16]
+ hash_digest = sha256_hex[:16]
media_type = mime_type.split(";", 1)[0].strip().lower()
match media_type:
@@ -138,15 +140,17 @@ class GeneratedContentFactory:
) -> DocumentContent:
"""Store a produced file's bytes and return the `DocumentContent` that points at them.
- The key is `{storage_scope}/generated/{hash}/{filename}`: the hash keeps two different files with the
- same name apart, and ending in the human filename means every backend's URL, and so every download, is
- called `invoice-INV-2026-0142.pdf` rather than a hash. The caller owns the filename's safety: it must be
- one name every `pipelex-storage://` reader accepts, as `path_safe_file_name` makes it, or a host that
- checks the key, such as the hosted platform, refuses to serve the file.
+ The key is `{storage_scope}/generated/{hash}/{filename}`: the hash, the first 16 hexadecimal digits of the
+ bytes' SHA-256, keeps two different files with the same name apart, and ending in the human filename means
+ every backend's URL, and so every download, is called `invoice-INV-2026-0142.pdf` rather than a hash. The
+ whole digest goes with the write, so a storage that checks it, as S3 does, keeps it beside the file. The
+ caller owns the filename's safety: it must be one name every `pipelex-storage://` reader accepts, as
+ `path_safe_file_name` makes it, or a host that checks the key, such as the hosted platform, refuses to serve
+ the file.
"""
- hash_digest = hashlib.sha256(data).hexdigest()[:16]
- storage_key = f"{storage_scope}/{GENERATED_CONTENT_LEAF}/{hash_digest}/{filename}"
- url = await self.storage_provider.store(data=data, key=storage_key, content_type=mime_type)
+ sha256_hex = hashlib.sha256(data).hexdigest()
+ storage_key = f"{storage_scope}/{GENERATED_CONTENT_LEAF}/{sha256_hex[:16]}/{filename}"
+ url = await self.storage_provider.store(data=data, key=storage_key, content_type=mime_type, checksum_sha256_hex=sha256_hex)
public_url = await self.storage_provider.public_url(uri=url)
return DocumentContent(url=url, public_url=public_url, mime_type=mime_type, filename=filename)
@@ -216,12 +220,13 @@ class GeneratedContentFactory:
url = remote_url
is_remote_url = True
elif actual_bytes:
+ sha256_hex = hashlib.sha256(actual_bytes).hexdigest()
storage_key = self._build_storage_key(
storage_scope=storage_scope,
- data=actual_bytes,
+ sha256_hex=sha256_hex,
mime_type=mime_type,
)
- url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type)
+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type, checksum_sha256_hex=sha256_hex)
is_remote_url = False
else:
msg = "No URL or bytes found"
@@ -244,12 +249,13 @@ class GeneratedContentFactory:
else:
if not declared_mime_type and fetched_mime_type:
mime_type = fetched_mime_type
+ sha256_hex = hashlib.sha256(actual_bytes).hexdigest()
storage_key = self._build_storage_key(
storage_scope=storage_scope,
- data=actual_bytes,
+ sha256_hex=sha256_hex,
mime_type=mime_type,
)
- url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type)
+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key, content_type=mime_type, checksum_sha256_hex=sha256_hex)
public_url = await self.storage_provider.public_url(uri=url)
elif not is_remote_url:
public_url = await self.storage_provider.public_url(uri=url)
diff --git a/pipelex/tools/misc/filetype_utils.py b/pipelex/tools/misc/filetype_utils.py
index c50378370..ccd8731ac 100644
--- a/pipelex/tools/misc/filetype_utils.py
+++ b/pipelex/tools/misc/filetype_utils.py
@@ -18,8 +18,6 @@ _MIME_OVERRIDES: Final[dict[str, str]] = {
"image/jpeg": "jpeg",
"image/jpg": "jpeg",
"text/plain": "txt",
- # Python 3.11's default table has no entry for Markdown; 3.12 added `.md`
- "text/markdown": "md",
"application/json": "json",
}
@@ -31,12 +29,28 @@ OFFICE_MIME_TYPES_BY_EXTENSION: Final[dict[str, str]] = {
".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
}
+# The types the built-in table of one Python this package supports lacks and a later one knows, named here so that every
+# Python answers the same, and the runtime never disagrees on a type with a host running another Python, such as the
+# hosted platform, which types an upload from its name with `mime_type_of_file_name`: Markdown entered the table in
+# Python 3.12, YAML and the OpenDocument formats in 3.14. The types are the ones those tables give.
+_MIME_TYPES_LATER_PYTHONS_KNOW: Final[dict[str, str]] = {
+ ".md": "text/markdown",
+ ".markdown": "text/markdown",
+ ".yaml": "application/yaml",
+ ".yml": "application/yaml",
+ ".odt": "application/vnd.oasis.opendocument.text",
+ ".ods": "application/vnd.oasis.opendocument.spreadsheet",
+ ".odp": "application/vnd.oasis.opendocument.presentation",
+}
+
# A table of our own: a `MimeTypes` instance starts from Python's built-in table, never from the host's `mime.types`
# files or registry, which only the module-level functions of `mimetypes` read, and it gets the types added here.
_MIME_DB: Final[mimetypes.MimeTypes] = mimetypes.MimeTypes()
_MIME_DB.add_type("application/json", ".json", strict=True)
for office_extension, office_mime_type in OFFICE_MIME_TYPES_BY_EXTENSION.items():
_MIME_DB.add_type(office_mime_type, office_extension, strict=True)
+for later_python_extension, later_python_mime_type in _MIME_TYPES_LATER_PYTHONS_KNOW.items():
+ _MIME_DB.add_type(later_python_mime_type, later_python_extension, strict=True)
class FileType(BaseModel):
@@ -124,7 +138,9 @@ def mime_type_of_file_name(*, file_name: str) -> str | None:
"""The MIME type a file's name implies by its extension, or None when the extension is not known.
Read from this module's own table, which names the Office Open XML types, so an `.xlsx`, `.docx` or `.pptx` file is
- stored under its type whatever the host's `mimetypes` table holds, and never from the host's own files.
+ stored under its type whatever the host's `mimetypes` table holds, and never from the host's own files. It also
+ names the types only a later Python's table knows, such as `text/markdown` for `.md`, so the answer is the same on
+ every Python this package supports.
Args:
file_name: The file's name, or a path or storage key ending in it.
diff --git a/pipelex/tools/storage/gcp_storage_provider.py b/pipelex/tools/storage/gcp_storage_provider.py
index 917e62e45..3836f5fd7 100644
--- a/pipelex/tools/storage/gcp_storage_provider.py
+++ b/pipelex/tools/storage/gcp_storage_provider.py
@@ -269,13 +269,15 @@ class GcpStorageProvider(StorageProviderAbstract):
raise StorageGcpError(msg) from exc
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
"""Store bytes to a GCS object.
Args:
data: The bytes to store.
key: Storage key (without scheme prefix).
content_type: Optional MIME type for the object.
+ checksum_sha256_hex: Ignored: GCS takes no SHA-256 checksum, and the client library checks the upload's own
+ CRC32C or MD5 on the way.
Raises:
StorageGcpError: If the GCS operation fails.
diff --git a/pipelex/tools/storage/in_memory_storage_provider.py b/pipelex/tools/storage/in_memory_storage_provider.py
index 91e03bdcd..6f3cc7c99 100644
--- a/pipelex/tools/storage/in_memory_storage_provider.py
+++ b/pipelex/tools/storage/in_memory_storage_provider.py
@@ -60,13 +60,14 @@ class InMemoryStorageProvider(RootModel[InMemoryStorageRoot], StorageProviderAbs
return len(self.root[key])
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
"""Store bytes in memory.
Args:
data: The bytes to store.
key: Storage key (without scheme prefix).
content_type: Ignored for in-memory storage.
+ checksum_sha256_hex: Ignored: memory has no checksum of its own to send it to.
"""
self.root[key] = data
diff --git a/pipelex/tools/storage/local_storage_provider.py b/pipelex/tools/storage/local_storage_provider.py
index b382382b8..6dfb9b57d 100644
--- a/pipelex/tools/storage/local_storage_provider.py
+++ b/pipelex/tools/storage/local_storage_provider.py
@@ -146,7 +146,7 @@ class LocalStorageProvider(StorageProviderAbstract):
os.fsync(file_handle.fileno())
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
"""Store bytes to a file, published whole or not at all.
The bytes are written to a temporary file beside the target, flushed to disk, then renamed onto the target,
@@ -158,6 +158,7 @@ class LocalStorageProvider(StorageProviderAbstract):
data: The bytes to store.
key: Storage key (relative path, without scheme prefix).
content_type: Ignored for local storage.
+ checksum_sha256_hex: Ignored: a file has no checksum of its own to send it to.
Raises:
StorageInvalidUriError: If the key is invalid.
diff --git a/pipelex/tools/storage/s3_storage_provider.py b/pipelex/tools/storage/s3_storage_provider.py
index d404c8e41..5ac6849a8 100644
--- a/pipelex/tools/storage/s3_storage_provider.py
+++ b/pipelex/tools/storage/s3_storage_provider.py
@@ -46,6 +46,26 @@ def checksum_sha256_header_value(*, sha256_hex: str) -> str:
return base64.b64encode(bytes.fromhex(sha256_hex)).decode("ascii")
+def _checksum_sha256_put_params(*, checksum_sha256_hex: str | None) -> dict[str, str]:
+ """The `PutObject` parameters that make S3 check a write's SHA-256 and keep it beside the object.
+
+ Every write the provider makes carries them, so S3 refuses a body that does not hash to the checksum and stores the
+ checksum, which a `HeadObject` with `ChecksumMode=ENABLED` returns as `ChecksumSHA256`. With the digest, it goes as
+ the `x-amz-checksum-sha256` header, and the bytes are not hashed again. Without it, `ChecksumAlgorithm` alone makes
+ botocore hash the body as it sends it and send the checksum as a trailer of the upload.
+
+ Args:
+ checksum_sha256_hex: The SHA-256 of the body in lowercase hexadecimal, or None for botocore to compute it.
+
+ Returns:
+ The parameters to add to a `put_object` call.
+ """
+ params = {"ChecksumAlgorithm": "SHA256"}
+ if checksum_sha256_hex is not None:
+ params["ChecksumSHA256"] = checksum_sha256_header_value(sha256_hex=checksum_sha256_hex)
+ return params
+
+
class S3StorageProvider(StorageProviderAbstract):
"""Storage provider implementation for AWS S3 storage.
@@ -116,7 +136,13 @@ class S3StorageProvider(StorageProviderAbstract):
# shared host. A content security policy can allow a bucket only by its own host, so pin the virtual
# style: `<bucket>.s3.<region>.amazonaws.com`. botocore still falls back to path-style for a bucket
# name that cannot be a hostname, a dotted one included.
- config = AioConfig(signature_version="s3v4", s3={"addressing_style": "virtual"})
+ #
+ # Every write carries the body's SHA-256, which S3 checks the body against (`_checksum_sha256_put_params`).
+ # When botocore computes it, it goes as a trailer of an upload botocore already leaves unsigned. When the
+ # caller gave it, it goes as the signed `x-amz-checksum-sha256` header, and botocore, which recognises only
+ # a checksum of its own, would hash the whole body once more just to sign it. Over TLS the signed checksum
+ # protects the body as well as a signed payload would, so the payload is never signed.
+ config = AioConfig(signature_version="s3v4", s3={"addressing_style": "virtual", "payload_signing_enabled": False})
return {
"service_name": "s3",
@@ -274,13 +300,18 @@ class S3StorageProvider(StorageProviderAbstract):
return content_length
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
- """Store bytes to an S3 object.
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
+ """Store bytes to an S3 object, with S3 checking their SHA-256.
+
+ The PUT carries the SHA-256, so S3 refuses a body that arrived damaged, as a `BadDigest`, and keeps the
+ checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` reads it.
Args:
data: The bytes to store.
key: Storage key (without scheme prefix).
content_type: Optional MIME type for the object.
+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, or None for botocore to compute it as
+ it sends the bytes.
Raises:
StorageS3Error: If the S3 operation fails (any ClientError or BotoCoreError).
@@ -303,6 +334,7 @@ class S3StorageProvider(StorageProviderAbstract):
}
if content_type:
put_params["ContentType"] = content_type
+ put_params.update(_checksum_sha256_put_params(checksum_sha256_hex=checksum_sha256_hex))
await client.put_object(**put_params)
except client.exceptions.NoSuchBucket as exc:
msg = f"Bucket not found in S3: '{self._bucket_name}'"
@@ -323,8 +355,9 @@ class S3StorageProvider(StorageProviderAbstract):
"""Create an S3 object unless one exists at the key, with S3 checking the condition and the checksum.
The PUT carries `If-None-Match: *`, so S3 writes it only when the key holds no object and answers `412`
- otherwise, which is the key already held. With a checksum, it carries the SHA-256 as `x-amz-checksum-sha256`,
- and S3 refuses a body whose hash differs, so the object holds exactly the bytes the checksum names.
+ otherwise, which is the key already held. It carries the SHA-256, as `x-amz-checksum-sha256` when the caller
+ gave it and computed by botocore otherwise, and S3 refuses a body whose hash differs, so the object holds
+ exactly the bytes the checksum names, and keeps the checksum beside it.
A `409 ConditionalRequestConflict` means another operation on the key, a conditional write or a delete, was
in flight, and S3's answer to it is to retry: the PUT is sent again after a short pause, a bounded number of
@@ -336,7 +369,8 @@ class S3StorageProvider(StorageProviderAbstract):
data: The bytes to store, already checked against their checksum.
key: Storage key (without scheme prefix).
content_type: Optional MIME type for the object.
- checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, sent for S3 to check, or None.
+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, sent for S3 to check, or None for
+ botocore to compute it as it sends the bytes.
Returns:
True when this call created the object, False when the key already held one.
@@ -361,8 +395,7 @@ class S3StorageProvider(StorageProviderAbstract):
}
if content_type:
put_params["ContentType"] = content_type
- if checksum_sha256_hex is not None:
- put_params["ChecksumSHA256"] = checksum_sha256_header_value(sha256_hex=checksum_sha256_hex)
+ put_params.update(_checksum_sha256_put_params(checksum_sha256_hex=checksum_sha256_hex))
retry_delays = CONDITIONAL_CONFLICT_RETRY_DELAYS_SECONDS
conditional_conflict: ClientError | None = None
diff --git a/pipelex/tools/storage/storage_provider_abstract.py b/pipelex/tools/storage/storage_provider_abstract.py
index d31d352dd..72abcc6da 100644
--- a/pipelex/tools/storage/storage_provider_abstract.py
+++ b/pipelex/tools/storage/storage_provider_abstract.py
@@ -28,7 +28,7 @@ class StorageProviderAbstract(ABC):
Subclasses must implement:
- _load_with_metadata(key): Load data with MIME type by key (without scheme)
- - _store(data, key, content_type): Store data and return URI
+ - _store(data, key, content_type, checksum_sha256_hex): Store data
- public_url(uri): Return human-readable link for URI
A provider Pipelex ships names its kind in ``storage_kind``, the ``scheme`` its refusals carry; a plugin's provider
@@ -165,29 +165,39 @@ class StorageProviderAbstract(ABC):
stored_data = await self._load_with_metadata(key)
return len(stored_data.data)
- async def store(self, data: bytes, *, key: str, content_type: str | None = None) -> str:
+ async def store(self, data: bytes, *, key: str, content_type: str | None = None, checksum_sha256_hex: str | None = None) -> str:
"""Store data and return full URI with scheme.
+ A provider whose storage can check a write's checksum, as S3 can, sends the bytes' SHA-256 with every write, so
+ the storage refuses a body that arrived damaged and keeps the checksum beside the object. A caller that hashed
+ the bytes already, for their key, passes the digest, so they are not hashed again. Unlike `store_if_absent`,
+ which checks a checksum handed to it, this trusts the caller's, which is the hash of bytes the caller holds, and
+ S3 still refuses a body that does not match it.
+
Args:
data: The bytes to store.
key: Storage key (without scheme prefix).
content_type: Optional MIME type for the data (used by cloud providers).
+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, when the caller has it, or None.
Returns:
Full URI with PIPELEX_STORAGE_SCHEME prefix.
"""
uri = self._add_scheme(key)
- await self._store(data=data, key=key, content_type=content_type)
+ await self._store(data=data, key=key, content_type=content_type, checksum_sha256_hex=checksum_sha256_hex)
return uri
@abstractmethod
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
"""Store data in storage.
Args:
data: The bytes to store.
key: Storage key (without scheme prefix).
content_type: Optional MIME type for the data.
+ checksum_sha256_hex: The SHA-256 of `data` in lowercase hexadecimal, or None when the caller has not hashed
+ the bytes. A provider whose storage checks a write's checksum sends it, hashing the bytes itself when it
+ is None; another ignores it.
"""
async def store_if_absent(
@@ -238,17 +248,15 @@ class StorageProviderAbstract(ABC):
data: The bytes to store, already checked against `checksum_sha256_hex`.
key: Storage key (without scheme prefix).
content_type: Optional MIME type for the data.
- checksum_sha256_hex: The SHA-256 of `data`, or None. This default sends it nowhere.
+ checksum_sha256_hex: The SHA-256 of `data`, or None, handed on to `_store`.
Returns:
True when this call stored the object, False when the key already held one.
"""
- # A plain store has nowhere to send a checksum: `store_if_absent` checked the bytes against it already.
- _ = checksum_sha256_hex
try:
await self._load_size(key)
except StorageFileNotFoundError:
- await self._store(data=data, key=key, content_type=content_type)
+ await self._store(data=data, key=key, content_type=content_type, checksum_sha256_hex=checksum_sha256_hex)
return True
return False
diff --git a/tests/integration/pipelex/plugins/test_storage_external_plugin.py b/tests/integration/pipelex/plugins/test_storage_external_plugin.py
index 30919a53c..db9e20f91 100644
--- a/tests/integration/pipelex/plugins/test_storage_external_plugin.py
+++ b/tests/integration/pipelex/plugins/test_storage_external_plugin.py
@@ -43,7 +43,7 @@ class _FakeExternalStorageProvider(StorageProviderAbstract):
raise NotImplementedError
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
raise NotImplementedError
@override
diff --git a/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py b/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py
index 64d2c1d29..f13619b0d 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_generated_content_factory.py
@@ -1,3 +1,5 @@
+import hashlib
+
import pytest
from mthds.protocol.problem.registry import CoreCode
from pytest_mock import MockerFixture
@@ -10,6 +12,7 @@ from pipelex.tools.network.exceptions import SsrfBlockedError
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
FAKE_IMAGE_BYTES = b"fake-image-bytes-for-hashing"
+FAKE_IMAGE_SHA256 = hashlib.sha256(FAKE_IMAGE_BYTES).hexdigest()
@pytest.mark.asyncio(loop_scope="class")
@@ -64,6 +67,47 @@ class TestGeneratedContentFactoryStoredObject:
assert image_content.mime_type == expected_mime_type
assert image_content.url == "pipelex-storage://stored-uri"
+ async def test_the_write_carries_the_whole_digest_the_key_shortens(self, mocker: MockerFixture):
+ """The bytes are hashed once: the key carries the first 16 digits, and the write the whole digest.
+
+ A storage that checks a write's SHA-256, as S3 does, keeps it beside the object, so a reader learns the
+ file's full hash without reading it, which a key holding only its start cannot give.
+ """
+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)
+ storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")
+ storage_provider.public_url = mocker.AsyncMock(return_value=None)
+ factory = GeneratedContentFactory(storage_provider=storage_provider)
+
+ await factory.make_image_content(
+ storage_scope="test/scope",
+ raw_details=GeneratedImageRawDetails(size=None, actual_bytes=FAKE_IMAGE_BYTES, mime_type="image/png"),
+ )
+
+ store_call = storage_provider.store.await_args
+ assert store_call is not None
+ assert store_call.kwargs["key"] == f"test/scope/generated/{FAKE_IMAGE_SHA256[:16]}.png"
+ assert store_call.kwargs["checksum_sha256_hex"] == FAKE_IMAGE_SHA256
+
+ async def test_a_document_write_carries_the_whole_digest_its_key_shortens(self, mocker: MockerFixture):
+ storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)
+ storage_provider.store = mocker.AsyncMock(return_value="pipelex-storage://stored-uri")
+ storage_provider.public_url = mocker.AsyncMock(return_value=None)
+ factory = GeneratedContentFactory(storage_provider=storage_provider)
+ document_bytes = b"%PDF-1.7 a generated document"
+ document_sha256 = hashlib.sha256(document_bytes).hexdigest()
+
+ document_content = await factory.make_document_content(
+ storage_scope="test/scope", data=document_bytes, mime_type="application/pdf", filename="invoice.pdf"
+ )
+
+ storage_provider.store.assert_awaited_once_with(
+ data=document_bytes,
+ key=f"test/scope/generated/{document_sha256[:16]}/invoice.pdf",
+ content_type="application/pdf",
+ checksum_sha256_hex=document_sha256,
+ )
+ assert document_content.url == "pipelex-storage://stored-uri"
+
class TestGeneratedContentFactoryStorageKey:
"""The key builder on its own, whatever its caller passes.
@@ -97,7 +141,7 @@ class TestGeneratedContentFactoryStorageKey:
storage_key = factory._build_storage_key( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
storage_scope="test/scope",
- data=FAKE_IMAGE_BYTES,
+ sha256_hex=FAKE_IMAGE_SHA256,
mime_type=mime_type,
)
@@ -172,6 +216,7 @@ class TestGeneratedContentFactoryFetchedRemoteImage:
stored_key: str = store_call.kwargs["key"]
assert stored_key.endswith(f".{expected_extension}")
assert store_call.kwargs["content_type"] == expected_mime_type
+ assert store_call.kwargs["checksum_sha256_hex"] == FAKE_IMAGE_SHA256
assert image_content.mime_type == expected_mime_type
@pytest.mark.usefixtures("_remote_fetch_enabled")
diff --git a/tests/unit/pipelex/tools/misc/test_file_format_keys.py b/tests/unit/pipelex/tools/misc/test_file_format_keys.py
index e19e8f2f6..f4ba6c51f 100644
--- a/tests/unit/pipelex/tools/misc/test_file_format_keys.py
+++ b/tests/unit/pipelex/tools/misc/test_file_format_keys.py
@@ -64,6 +64,8 @@ class TestFileFormatKeys:
("image/svg+xml", "image"),
("text/html", "html"),
("text/markdown", "md"),
+ ("application/yaml", "yaml"),
+ ("application/vnd.oasis.opendocument.spreadsheet", "ods"),
("text/plain; charset=utf-8", "txt"),
("APPLICATION/PDF", "pdf"),
],
@@ -124,6 +126,7 @@ class TestFileFormatKeys:
(None, "report.docx", DOCX_MIME),
(None, "deck.pptx", PPTX_MIME),
("application/octet-stream", "budget.xlsx", XLSX_MIME),
+ (None, "minutes.odt", "application/vnd.oasis.opendocument.text"),
(None, "notes.txt", None),
(PPTX_MIME, "report.docx", PPTX_MIME),
("application/zip", "report.docx", DOCX_MIME),
diff --git a/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py b/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py
index 4d9421e55..fa6276666 100644
--- a/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py
+++ b/tests/unit/pipelex/tools/misc/test_mime_type_of_file_name.py
@@ -27,6 +27,27 @@ class TestMimeTypeOfFileName:
def test_the_type_follows_the_extension(self, file_name: str, expected: str | None) -> None:
assert mime_type_of_file_name(file_name=file_name) == expected
+ @pytest.mark.parametrize(
+ ("file_name", "expected"),
+ [
+ ("notes.md", "text/markdown"),
+ ("org/assets/README.MD", "text/markdown"),
+ ("notes.markdown", "text/markdown"),
+ ("config.yaml", "application/yaml"),
+ ("config.yml", "application/yaml"),
+ ("minutes.odt", "application/vnd.oasis.opendocument.text"),
+ ("budget.ods", "application/vnd.oasis.opendocument.spreadsheet"),
+ ("deck.odp", "application/vnd.oasis.opendocument.presentation"),
+ ],
+ )
+ def test_a_type_only_a_later_python_knows_is_known_on_every_python(self, file_name: str, expected: str) -> None:
+ """Python 3.11's table has no Markdown, and only 3.14's has YAML and OpenDocument, yet every Python answers alike.
+
+ The hosted platform types an upload from its name with this function on its own Python, so a runtime on another
+ one must give the same answer. The suite runs on each supported Python, which is what proves it.
+ """
+ assert mime_type_of_file_name(file_name=file_name) == expected
+
@pytest.mark.parametrize(("file_name", "expected"), [("report.docx", _DOCX), ("budget.xlsx", _XLSX), ("deck.pptx", _PPTX)])
def test_the_office_types_never_depend_on_the_hosts_mimetypes_table(self, mocker: MockerFixture, file_name: str, expected: str) -> None:
"""A host whose `mime.types` knows no office type, as a slim server image's does not, still gets them."""
diff --git a/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py b/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py
index 52e1c5602..4dc872c07 100644
--- a/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py
+++ b/tests/unit/pipelex/tools/misc/test_mime_type_to_extension.py
@@ -11,6 +11,8 @@ class TestMimeTypeToExtension:
("text/markdown", "md"),
("text/markdown; charset=utf-8", "md"),
("TEXT/Markdown", "md"),
+ ("application/yaml", "yaml"),
+ ("application/vnd.oasis.opendocument.text", "odt"),
("text/plain", "txt"),
("text/csv", "csv"),
("text/html", "html"),
diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
index ad226c813..425c16501 100644
--- a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
@@ -106,7 +106,7 @@ class TestS3StorageProvider:
s3_provider_no_signed_urls: S3StorageProvider,
mock_aiobotocore: dict[str, Any],
) -> None:
- """Test that store() correctly calls the S3 put_object method."""
+ """store() writes with put_object and asks for the SHA-256, which botocore computes when the caller has no digest."""
test_data = b"Test data"
key = "test/file.bin"
content_type = "application/octet-stream"
@@ -118,8 +118,42 @@ class TestS3StorageProvider:
Key=key,
Body=test_data,
ContentType=content_type,
+ ChecksumAlgorithm="SHA256",
)
+ async def test_store_sends_the_callers_digest_as_the_checksum(
+ self,
+ s3_provider_no_signed_urls: S3StorageProvider,
+ mock_aiobotocore: dict[str, Any],
+ ) -> None:
+ """A caller that hashed the bytes already hands the digest over, and it goes as `x-amz-checksum-sha256`."""
+ data = b"generated bytes"
+ sha256 = hashlib.sha256(data).hexdigest()
+
+ await s3_provider_no_signed_urls.store(
+ data=data, key="scope/generated/abc/file.pdf", content_type="application/pdf", checksum_sha256_hex=sha256
+ )
+
+ mock_aiobotocore["client"].put_object.assert_called_once_with(
+ Bucket=S3_TEST_BUCKET,
+ Key="scope/generated/abc/file.pdf",
+ Body=data,
+ ContentType="application/pdf",
+ ChecksumAlgorithm="SHA256",
+ ChecksumSHA256=base64.b64encode(hashlib.sha256(data).digest()).decode("ascii"),
+ )
+
+ async def test_the_client_never_signs_the_payload(
+ self,
+ s3_provider_no_signed_urls: S3StorageProvider,
+ mock_aiobotocore: dict[str, Any],
+ ) -> None:
+ """The signed checksum protects the body, so botocore is not asked to hash it again to sign it."""
+ await s3_provider_no_signed_urls.store(data=b"bytes", key="scope/file.bin")
+
+ client_config = mock_aiobotocore["session"].create_client.call_args.kwargs["config"]
+ assert client_config.s3 == {"addressing_style": "virtual", "payload_signing_enabled": False}
+
async def test_load_returns_data(
self,
mocker: MockerFixture,
@@ -231,6 +265,7 @@ class TestS3StorageProvider:
Bucket=S3_TEST_BUCKET,
Key=key,
Body=b"",
+ ChecksumAlgorithm="SHA256",
)
async def test_store_large_binary_data(
@@ -578,17 +613,20 @@ class TestS3StorageProvider:
Body=data,
IfNoneMatch="*",
ContentType="application/octet-stream",
+ ChecksumAlgorithm="SHA256",
ChecksumSHA256=base64.b64encode(hashlib.sha256(data).digest()).decode("ascii"),
)
- async def test_store_if_absent_without_a_checksum_sends_none(
+ async def test_store_if_absent_without_a_checksum_lets_botocore_compute_it(
self,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aiobotocore: dict[str, Any],
) -> None:
assert await s3_provider_no_signed_urls.store_if_absent(data=b"bytes", key="scope/blobs/one") is True
- mock_aiobotocore["client"].put_object.assert_called_once_with(Bucket=S3_TEST_BUCKET, Key="scope/blobs/one", Body=b"bytes", IfNoneMatch="*")
+ mock_aiobotocore["client"].put_object.assert_called_once_with(
+ Bucket=S3_TEST_BUCKET, Key="scope/blobs/one", Body=b"bytes", IfNoneMatch="*", ChecksumAlgorithm="SHA256"
+ )
@pytest.mark.parametrize("error_code", ["412", "PreconditionFailed"])
async def test_store_if_absent_over_a_held_object_is_no_error(
diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider_checksum.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider_checksum.py
new file mode 100644
index 000000000..f77e61542
--- /dev/null
+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider_checksum.py
@@ -0,0 +1,126 @@
+import base64
+import hashlib
+import inspect
+from pathlib import Path
+from typing import Any, NamedTuple
+
+import pytest
+from aiobotocore.session import AioSession, get_session
+from pytest_mock import MockerFixture
+
+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider
+
+# botocore reads these at session creation; any of them set on the developer's machine could change the request
+_AWS_ENV_VARS_TO_CLEAR = (
+ "AWS_PROFILE",
+ "AWS_DEFAULT_PROFILE",
+ "AWS_SESSION_TOKEN",
+ "AWS_SECURITY_TOKEN",
+ "AWS_DEFAULT_REGION",
+ "AWS_REGION",
+ "AWS_ENDPOINT_URL",
+ "AWS_ENDPOINT_URL_S3",
+ "AWS_REQUEST_CHECKSUM_CALCULATION",
+ "AWS_RESPONSE_CHECKSUM_VALIDATION",
+)
+CHECKSUM_TEST_BUCKET = "pipelex-app-dev"
+CHECKSUM_TEST_REGION = "eu-west-3"
+PAYLOAD = b"the bytes a run generated"
+PAYLOAD_SHA256_BASE64 = base64.b64encode(hashlib.sha256(PAYLOAD).digest()).decode("ascii")
+
+
+class _RequestCapturedError(Exception):
+ """Raised by the capturing hook, so the request is read as botocore would send it and never sent."""
+
+
+class _CapturedRequest(NamedTuple):
+ headers: dict[str, str]
+ body: bytes
+
+
+async def _read_body(body: Any) -> bytes:
+ if body is None:
+ return b""
+ if isinstance(body, bytes | bytearray):
+ return bytes(body)
+ read_result = body.read()
+ if inspect.isawaitable(read_result):
+ read_result = await read_result
+ return bytes(read_result)
+
+
[email protected](loop_scope="class")
+class TestS3StorageProviderChecksumOnTheWire:
+ @pytest.fixture
+ def isolated_aws_environment(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
+ """Give botocore dummy credentials and keep the developer's own AWS configuration out of the test."""
+ for env_var in _AWS_ENV_VARS_TO_CLEAR:
+ monkeypatch.delenv(env_var, raising=False)
+ empty_aws_file = tmp_path / "empty_aws_file"
+ empty_aws_file.write_text("")
+ monkeypatch.setenv("AWS_CONFIG_FILE", str(empty_aws_file))
+ monkeypatch.setenv("AWS_SHARED_CREDENTIALS_FILE", str(empty_aws_file))
+ monkeypatch.setenv("AWS_ACCESS_KEY_ID", "AKIDEXAMPLE")
+ monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "dummy-secret-for-local-signing")
+
+ @pytest.fixture
+ def captured_requests(self, mocker: MockerFixture, isolated_aws_environment: None) -> list[_CapturedRequest]: # ruff: ignore[unused-method-argument]
+ """The `PutObject` requests the provider's clients would send, signed by real botocore and stopped before they leave.
+
+ Mocking `put_object` shows only the parameters the provider passes; whether botocore turns them into a checksum S3
+ checks, and whether it hashes the body once more to sign it, is botocore's doing, so these tests read the request
+ itself. Nothing reaches the network.
+ """
+ captured: list[_CapturedRequest] = []
+ session: AioSession = get_session()
+
+ async def capture(request: Any, **_kwargs: Any) -> None:
+ headers = {
+ str(name).lower(): value.decode("ascii") if isinstance(value, bytes) else str(value) for name, value in request.headers.items()
+ }
+ captured.append(_CapturedRequest(headers=headers, body=await _read_body(request.body)))
+ raise _RequestCapturedError
+
+ session.register("before-send.s3.PutObject", capture)
+ mocker.patch("aiobotocore.session.get_session", return_value=session)
+ return captured
+
+ @staticmethod
+ def _make_provider() -> S3StorageProvider:
+ return S3StorageProvider(bucket_name=CHECKSUM_TEST_BUCKET, region=CHECKSUM_TEST_REGION, signed_urls_lifespan=None)
+
+ async def test_a_callers_digest_goes_as_a_signed_header_and_the_body_is_not_hashed_again(self, captured_requests: list[_CapturedRequest]) -> None:
+ with pytest.raises(_RequestCapturedError):
+ await self._make_provider().store(
+ data=PAYLOAD,
+ key="scope/generated/abc/report.pdf",
+ content_type="application/pdf",
+ checksum_sha256_hex=hashlib.sha256(PAYLOAD).hexdigest(),
+ )
+
+ request = captured_requests[-1]
+ assert request.headers["x-amz-checksum-sha256"] == PAYLOAD_SHA256_BASE64
+ assert request.headers["x-amz-sdk-checksum-algorithm"] == "SHA256"
+ # The signature covers the checksum, and botocore did not hash the body to sign it.
+ assert "x-amz-checksum-sha256" in request.headers["authorization"]
+ assert request.headers["x-amz-content-sha256"] == "UNSIGNED-PAYLOAD"
+ assert "x-amz-trailer" not in request.headers
+ assert request.body == PAYLOAD
+
+ async def test_without_a_digest_botocore_sends_the_sha256_as_a_trailer(self, captured_requests: list[_CapturedRequest]) -> None:
+ with pytest.raises(_RequestCapturedError):
+ await self._make_provider().store(data=PAYLOAD, key="scope/assets/upload.txt", content_type="text/plain")
+
+ request = captured_requests[-1]
+ assert request.headers["x-amz-sdk-checksum-algorithm"] == "SHA256"
+ assert request.headers["x-amz-trailer"] == "x-amz-checksum-sha256"
+ assert f"x-amz-checksum-sha256:{PAYLOAD_SHA256_BASE64}".encode("ascii") in request.body
+
+ async def test_a_create_only_write_carries_the_checksum_beside_its_condition(self, captured_requests: list[_CapturedRequest]) -> None:
+ with pytest.raises(_RequestCapturedError):
+ await self._make_provider().store_if_absent(data=PAYLOAD, key="scope/blobs/one", checksum_sha256_hex=hashlib.sha256(PAYLOAD).hexdigest())
+
+ request = captured_requests[-1]
+ assert request.headers["if-none-match"] == "*"
+ assert request.headers["x-amz-checksum-sha256"] == PAYLOAD_SHA256_BASE64
+ assert request.headers["x-amz-content-sha256"] == "UNSIGNED-PAYLOAD"
diff --git a/tests/unit/pipelex/tools/storage/test_storage_load_size.py b/tests/unit/pipelex/tools/storage/test_storage_load_size.py
index c7af6a120..7689bdb22 100644
--- a/tests/unit/pipelex/tools/storage/test_storage_load_size.py
+++ b/tests/unit/pipelex/tools/storage/test_storage_load_size.py
@@ -34,7 +34,7 @@ class _LoadOnlyProvider(StorageProviderAbstract):
return StoredData(data=self.objects[key], mime_type=None)
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
self.objects[key] = data
@override
diff --git a/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py b/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py
index cbf186745..d6856d561 100644
--- a/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py
+++ b/tests/unit/pipelex/tools/storage/test_storage_store_if_absent.py
@@ -22,6 +22,7 @@ class _PlainProvider(StorageProviderAbstract):
def __init__(self) -> None:
self.objects: dict[str, bytes] = {}
+ self.checksums: dict[str, str | None] = {}
@override
async def _load_with_metadata(self, key: str) -> StoredData:
@@ -31,8 +32,9 @@ class _PlainProvider(StorageProviderAbstract):
return StoredData(data=self.objects[key], mime_type=None)
@override
- async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
self.objects[key] = data
+ self.checksums[key] = checksum_sha256_hex
@override
async def public_url(self, uri: str) -> str | None:
@@ -59,6 +61,15 @@ class TestStorageStoreIfAbsent:
assert created is True
assert await provider.load(uri=f"{PIPELEX_STORAGE_SCHEME}{KEY}") == PAYLOAD
+ async def test_the_default_hands_the_checksum_on_to_the_plain_store(self) -> None:
+ """A provider overriding only `_store` still receives the checksum, for a storage that can check it."""
+ provider = _PlainProvider()
+ sha256 = hashlib.sha256(PAYLOAD).hexdigest()
+
+ assert await provider.store_if_absent(data=PAYLOAD, key=KEY, checksum_sha256_hex=sha256) is True
+
+ assert provider.checksums == {KEY: sha256}
+
async def test_bytes_not_matching_their_checksum_are_refused_before_any_write(self, tmp_path: Path) -> None:
for provider in _providers(tmp_path=tmp_path):
with pytest.raises(StorageChecksumMismatchError, match="do not hash to the checksum"):