Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/S3-client-reuse · L-261026-729dbb — S3 client per call, HEAD before GET (#1489)

Louis Choquel6eabd43b6596
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 64e208f8a..2db872b23 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,12 @@
# Changelog
+## [Unreleased]
+
+### Fixed
+
+- **The S3 storage provider reuses one client per event loop**: each event loop opens one aiobotocore client on its first S3 operation, and every later operation on that loop shares it and its pool of `S3_MAX_POOL_CONNECTIONS` connections, where every operation, a presigned URL included, built a client and opened a TLS connection of its own; the client is closed as its loop shuts down or at the runtime's teardown, which now calls `StorageProviderAbstract.teardown()`, a no-op that a storage plugin keeping a client open between operations overrides to close it. Measured from a laptop against a bucket in another region, the API server's read of a method tree of 20 blobs went from 4,261 ms of wall time and 425 ms of CPU to 1,582 ms and 32 ms with this change alone.
+- **The API server reads each blob a request names in one request**: a run, a validation and the tooling routes read each hash entry of `files` and `dependencies` with one read of at most one byte past the size the entry states, 16 at a time, where they looked each blob's size up first and then read the blobs 4 at a time; a hash held at another size still answers `422` `request.invalid`, its size looked up only for that refusal's message, and damaged bytes still answer `500` `pipelex.storage.damaged`. Measured as above with both changes, reading a tree of 1, 20 and 50 blobs takes 194, 419 and 838 ms of wall time and 1.3, 14 and 31 ms of CPU, where it took 1,142, 4,261 and 10,253 ms and 26, 425 and 1,000 ms. The local storage provider now reads such a leading range of a file without loading the whole file.
+
## [v0.92.0] - 2026-10-25
### Changed
diff --git a/api/pipelex_api/blob_store.py b/api/pipelex_api/blob_store.py
index b1f3fc8d5..3d7dc23e5 100644
--- a/api/pipelex_api/blob_store.py
+++ b/api/pipelex_api/blob_store.py
@@ -38,21 +38,24 @@ From this the blob store answers without reading what it does not need:
ones held, and nothing is written, while bytes of another size mean the object held is damaged, which only a storage
that does not publish writes whole could leave, a 500 `pipelex.storage.damaged`, since nothing writes over a blob
and its operator must remove the object.
-- A run looks up each hash's stored size first. A hash held at another size than an entry states is a 422
- `request.invalid`, never `request.blobs_missing`, since uploading those bytes again would change nothing: the entry
- is wrong.
- Only a blob held at the size its entries state is read, so the bytes read are bounded by the sizes the request
- states, which its census already held to the tree's limits, and each is checked against its hash once read. Bytes
- that do not hash to their key mean the object stored under it is damaged, which no caller can repair and no retry
- cures: a 500 `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, and an error
- record, "Stored blob bytes do not match their hash", naming the hash for the operator to remove the object, after
- which the blob is uploaded again.
+- A run reads each hash's bytes in one request, a read of at most one byte more than the largest size its entries
+ state, so the bytes read are bounded by the sizes the request states, which its census already held to the tree's
+ limits, whatever size the object is held at. A hash held at another size than an entry states, which the length read
+ shows, is a 422 `request.invalid`, never `request.blobs_missing`, since uploading those bytes again would change
+ nothing: the entry is wrong. Only that refusal looks the size held up, for the operator's message, when the read
+ stopped short of the object's end.
+ Bytes read at the size their entries state are checked against their hash. Bytes that do not hash to their key mean
+ the object stored under it is damaged, which no caller can repair and no retry cures: a 500
+ `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, and an error record, "Stored blob
+ bytes do not match their hash", naming the hash for the operator to remove the object, after which the blob is
+ uploaded again.
Lookups and reads run a bounded number at a time, so one request naming many blobs never opens them all at once.
"""
from collections.abc import Iterable, Mapping
from functools import partial
+from typing import NamedTuple
from mthds.method_tree.digest import DIGEST_PREFIX, sha256_hex, tree_listing
from mthds.method_tree.file_entry import BlobStore, InMemoryBlobStore
@@ -73,9 +76,11 @@ BLOBS_PREFIX = "blobs"
TREES_PREFIX = "trees"
_BLOB_CONTENT_TYPE = "application/octet-stream"
_LISTING_CONTENT_TYPE = "text/plain; charset=utf-8"
-# How many lookups of a blob's size, which read metadata alone, and how many reads or writes of a blob's bytes, one
-# request runs at a time.
+# How many lookups of a blob's size, which read metadata alone, how many reads of the blobs a request names, each
+# bounded by the size the request states for it, and how many writes of a blob's bytes one request runs at a time. The
+# S3 provider's pool of connections holds at least as many as the first two (`S3_MAX_POOL_CONNECTIONS`).
MAX_CONCURRENT_BLOB_LOOKUPS = 16
+MAX_CONCURRENT_BLOB_READS = 16
MAX_CONCURRENT_BLOB_TRANSFERS = 4
# The message of the error record a blob read whose bytes do not match their hash writes, for the operator to search.
BLOB_DAMAGED_MESSAGE = "Stored blob bytes do not match their hash"
@@ -104,6 +109,16 @@ def problem_scheme_of(*, storage_method: str) -> str:
return PLUGIN_STORAGE_SCHEME
+class StatedBlobs(NamedTuple):
+ """The blobs a map of file entries names by hash, as the blob store holds them."""
+
+ held: BlobStore
+ """The bytes of each hash held at the size its entries state, checked against the hash."""
+
+ mismatched: dict[str, int]
+ """Each hash held at another size than an entry states, with the size it is held at, in hash order."""
+
+
class RunnerBlobStore:
"""The blobs and tree listings of one caller's scope, over the deployment's storage provider."""
@@ -155,37 +170,68 @@ class RunnerBlobStore:
sizes = await gather_bounded([partial(self.stored_size, sha256=sha256) for sha256 in hashes], max_concurrency=MAX_CONCURRENT_BLOB_LOOKUPS)
return dict(zip(hashes, sizes, strict=True))
- async def _read_verified(self, *, sha256: str, size: int) -> bytes:
- """The stored bytes of `sha256`, held at `size`, checked against their hash.
+ async def _read_bounded(self, *, sha256: str, nb_bytes: int) -> bytes | None:
+ """At most `nb_bytes` leading bytes of the blob `sha256`, read in one request, or None when the hash is not held."""
+ try:
+ return await self._storage.load_head(f"{PIPELEX_STORAGE_SCHEME}{self._blob_key(sha256=sha256)}", nb_bytes=nb_bytes)
+ except StorageFileNotFoundError:
+ return None
+
+ async def read_stated(self, *, stated_sizes: Mapping[str, set[int]]) -> StatedBlobs:
+ """Read the blobs a map of file entries names, each in one request bounded by the sizes its entries state.
+
+ Each hash is read for at most one byte more than the largest size its entries state, so a hash held at a larger
+ size shows it without its bytes being read, and the bytes read stay bounded by the sizes stated. A hash not
+ held is left out, for the reader to report among the blobs this runner does not hold.
- Raises (as a problem document): a 500 `pipelex.storage.damaged` naming the kind of storage, for bytes that
- are not `size` bytes hashing to `sha256`, after an error record naming the hash, the one place the operator
- reads which damaged object to remove.
+ Args:
+ stated_sizes: Each hash the map names, with every size its entries state for it, which the map's census
+ held to the tree's limits.
+
+ Raises (as a problem document): a 500 `pipelex.storage.damaged` naming the kind of storage, when no hash is
+ held at another size and bytes read at the size their entries state do not hash to their key, after an error
+ record naming each such hash, the one place the operator reads which damaged object to remove.
"""
- data = await self._storage.load(f"{PIPELEX_STORAGE_SCHEME}{self._blob_key(sha256=sha256)}")
- if len(data) != size or sha256_hex(data) != sha256:
+ hashes = sorted(stated_sizes)
+ reads = await gather_bounded(
+ [partial(self._read_bounded, sha256=sha256, nb_bytes=max(stated_sizes[sha256]) + 1) for sha256 in hashes],
+ max_concurrency=MAX_CONCURRENT_BLOB_READS,
+ )
+ blobs: dict[str, bytes] = {}
+ mismatched: dict[str, int] = {}
+ for sha256, data in zip(hashes, reads, strict=True):
+ if data is None:
+ continue
+ sizes = stated_sizes[sha256]
+ if sizes == {len(data)}:
+ blobs[sha256] = data
+ elif len(data) <= max(sizes):
+ mismatched[sha256] = len(data)
+ else:
+ # The read stopped a byte past the largest size stated: the object is larger, and its size is looked up.
+ held_size = await self.stored_size(sha256=sha256)
+ if held_size is not None:
+ mismatched[sha256] = held_size
+ if mismatched:
+ return StatedBlobs(held=InMemoryBlobStore(), mismatched=mismatched)
+ damaged = [sha256 for sha256, data in blobs.items() if sha256_hex(data) != sha256]
+ for sha256 in damaged:
log.error(
BLOB_DAMAGED_MESSAGE,
- fields={"blob_sha256": sha256, "blob_size_bytes": size, "read_size_bytes": len(data), "storage_scheme": self._scheme},
+ fields={
+ "blob_sha256": sha256,
+ "blob_size_bytes": len(blobs[sha256]),
+ "read_size_bytes": len(blobs[sha256]),
+ "storage_scheme": self._scheme,
+ },
)
+ if damaged:
msg = (
- f"The bytes this runner's blob store holds under {sha256} do not hash to it: the storage holds a damaged "
+ f"The bytes this runner's blob store holds under {damaged[0]} do not hash to it: the storage holds a damaged "
"object, which its operator removes for the blob to be uploaded again."
)
raise BlobStoreDamagedError(msg, scheme=self._scheme)
- return data
-
- async def read_held(self, *, sizes: Mapping[str, int]) -> BlobStore:
- """Read these held blobs, each at its stored size, as the synchronous blob store a map of file entries is read with.
-
- Args:
- sizes: Each hash to read, with the size it is held at, which the caller bounded before asking.
- """
- blobs = await gather_bounded(
- [partial(self._read_verified, sha256=sha256, size=size) for sha256, size in sizes.items()],
- max_concurrency=MAX_CONCURRENT_BLOB_TRANSFERS,
- )
- return InMemoryBlobStore(blobs)
+ return StatedBlobs(held=InMemoryBlobStore(list(blobs.values())), mismatched={})
async def store_blob(self, *, data: bytes) -> str:
"""Hold `data` under its SHA-256, creating the blob unless it is held already, and return the hash.
diff --git a/api/pipelex_api/method_tree_request.py b/api/pipelex_api/method_tree_request.py
index 89888dc68..1b14bc4f5 100644
--- a/api/pipelex_api/method_tree_request.py
+++ b/api/pipelex_api/method_tree_request.py
@@ -419,26 +419,27 @@ def _refuse_map(*, refusal_error: MethodTreeRefusedError, label: str, field: str
async def _held_blobs(*, blob_store: RunnerBlobStore, stated_sizes: dict[str, set[int]], label: str, field: str) -> BlobStore:
- """The bytes a map's hash entries name that the blob store holds, looked up by size before any is read.
+ """The bytes a map's hash entries name that the blob store holds, each read in one request bounded by its stated size.
A hash held at another size than one of its entries states is refused, since uploading its bytes again would not
- change the size the store holds them at. Every other held hash is held at the one size its entries state, which the
- census bounded, so reading them reads no more than the census allowed. A hash not held is left out, for the reader
- to report among the blobs this runner does not hold.
-
- Raises (as a problem document): a 422 `request.invalid` naming the field the map rides; its message, in the operator's
- diagnostics, lists each hash held at another size, with that size, which no parameter of the code carries.
+ change the size the store holds them at. Every hash is read for at most one byte past the largest size its entries
+ state, which the census bounded, so reading them reads no more than the census allowed. A hash not held is left
+ out, for the reader to report among the blobs this runner does not hold.
+
+ Raises (as problem documents): a 422 `request.invalid` naming the field the map rides, whose message, in the
+ operator's diagnostics, lists each hash held at another size, with that size, which no parameter of the code
+ carries; and, when no hash is, the 500 `pipelex.storage.damaged` of the blob store for bytes that do not hash to
+ their key.
"""
- stored = await blob_store.stored_sizes(sha256s=sorted(stated_sizes))
- mismatched = {sha256: size for sha256, size in stored.items() if size is not None and stated_sizes[sha256] != {size}}
- if mismatched:
- described = ", ".join(f"{sha256} (held at {size} bytes)" for sha256, size in mismatched.items())
+ stated = await blob_store.read_stated(stated_sizes=stated_sizes)
+ if stated.mismatched:
+ described = ", ".join(f"{sha256} (held at {size} bytes)" for sha256, size in stated.mismatched.items())
msg = (
f"{label} states a size for a hash other than the size of the bytes this runner holds under it: {described}. "
"A hash names one set of bytes, so state the size of those bytes."
)
raise RunnerRequestError(msg, field=field)
- return await blob_store.read_held(sizes={sha256: size for sha256, size in stored.items() if size is not None})
+ return stated.held
async def read_request_tree(
diff --git a/api/tests/unit/test_blob_store.py b/api/tests/unit/test_blob_store.py
index e47d2acfa..a8b482885 100644
--- a/api/tests/unit/test_blob_store.py
+++ b/api/tests/unit/test_blob_store.py
@@ -1,9 +1,10 @@
"""The runner's blob store over a storage provider: what it looks up by size, what it reads, and how many at once.
-The storage here counts its calls and the ones in flight, so these pin what the routes cannot show: a blob held at
-another size than an entry states is never read, reads and lookups run a bounded number at a time, every write is
-create-only with its checksum, a short object is never written over, and a tree whose listing is held is recorded
-without a lookup per file.
+The storage here counts its calls and the ones in flight, so these pin what the routes cannot show: a request's blobs
+are read in one request each, never past a byte more than the size stated, a blob held at another size than an entry
+states is refused without its bytes read, reads and lookups run a bounded number at a time, every write is create-only
+with its checksum, a short object is never written over, and a tree whose listing is held is recorded without a lookup
+per file.
"""
import asyncio
@@ -14,6 +15,7 @@ import pytest
from mthds.method_tree.digest import digest_of_contents, sha256_hex, tree_files_from_contents, tree_listing
from pipelex.tools.log.log_fields import carried_attributes
from pipelex.tools.storage.exceptions import StorageFileNotFoundError
+from pipelex.tools.storage.s3_storage_provider import S3_MAX_POOL_CONNECTIONS
from pipelex.tools.storage.storage_config import StorageMethod
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract, StoredData
from pytest_mock import MockerFixture
@@ -22,7 +24,7 @@ from typing_extensions import override
from pipelex_api.blob_store import (
BLOB_DAMAGED_MESSAGE,
MAX_CONCURRENT_BLOB_LOOKUPS,
- MAX_CONCURRENT_BLOB_TRANSFERS,
+ MAX_CONCURRENT_BLOB_READS,
RunnerBlobStore,
problem_scheme_of,
)
@@ -39,6 +41,7 @@ class _CountingStorage(StorageProviderAbstract):
def __init__(self) -> None:
self.objects: dict[str, bytes] = {}
self.loaded_keys: list[str] = []
+ self.head_reads: list[tuple[str, int]] = []
self.sized_keys: list[str] = []
self.created: list[tuple[str, str | None]] = []
self.create_attempts: list[str] = []
@@ -63,6 +66,20 @@ class _CountingStorage(StorageProviderAbstract):
finally:
self._loads_in_flight -= 1
+ @override
+ async def _load_head(self, key: str, *, nb_bytes: int) -> bytes:
+ self._loads_in_flight += 1
+ self.most_loads_at_once = max(self.most_loads_at_once, self._loads_in_flight)
+ try:
+ await asyncio.sleep(0)
+ self.head_reads.append((key, nb_bytes))
+ if key not in self.objects:
+ msg = f"File not found: '{key}'"
+ raise StorageFileNotFoundError(msg)
+ return self.objects[key][:nb_bytes]
+ finally:
+ self._loads_in_flight -= 1
+
@override
async def _load_size(self, key: str) -> int:
self._lookups_in_flight += 1
@@ -109,22 +126,65 @@ def _hash_map(entries: dict[str, tuple[bytes, int]]) -> str:
@pytest.mark.asyncio(loop_scope="class")
class TestRunnerBlobStore:
- async def test_a_blob_held_at_another_size_is_refused_before_any_byte_is_read(self) -> None:
- """An entry stating a size other than the held bytes' is a `request.invalid`, and the census bounds every read."""
+ async def test_a_blob_held_larger_than_stated_is_refused_reading_one_byte_past_the_size_stated(self) -> None:
+ """An entry stating a size other than the held bytes' is a `request.invalid`, and the census bounds every read.
+
+ The read asks for one byte more than the size stated, which shows the object is larger without reading it, and
+ only this refusal looks the size held up, for the operator's message.
+ """
storage = _CountingStorage()
blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage, scheme="memory")
large = b"x" * 4096
await blob_store.store_blob(data=large)
+ storage.sized_keys.clear()
with pytest.raises(RunnerRequestError) as exc_info:
- await read_request_tree(document=_hash_map({"large.bin": (large, 0)}), blob_store=blob_store, label="The `files` map", field="files")
+ await read_request_tree(document=_hash_map({"large.bin": (large, 10)}), blob_store=blob_store, label="The `files` map", field="files")
problem = problem_of(exc_info.value)
assert problem["status"] == 422
assert problem["code"] == "request.invalid"
assert problem["params"] == {"field": "files"}
# The hash and the size held ride the operator's message alone.
- assert f"{sha256_hex(large)}" in problem["diagnostics"]["message"]
+ assert f"{sha256_hex(large)} (held at 4096 bytes)" in problem["diagnostics"]["message"]
+ assert storage.head_reads == [(_blob_key(large), 11)]
+ assert storage.sized_keys == [_blob_key(large)]
+ assert storage.loaded_keys == []
+
+ async def test_a_blob_held_smaller_than_stated_is_refused_with_no_lookup(self) -> None:
+ storage = _CountingStorage()
+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage, scheme="memory")
+ small = b"a few bytes"
+ await blob_store.store_blob(data=small)
+ storage.sized_keys.clear()
+
+ with pytest.raises(RunnerRequestError) as exc_info:
+ await read_request_tree(document=_hash_map({"small.txt": (small, 100)}), blob_store=blob_store, label="The `files` map", field="files")
+
+ problem = problem_of(exc_info.value)
+ assert problem["code"] == "request.invalid"
+ assert f"{sha256_hex(small)} (held at {len(small)} bytes)" in problem["diagnostics"]["message"]
+ assert storage.head_reads == [(_blob_key(small), 101)]
+ assert storage.sized_keys == []
+
+ async def test_each_blob_a_request_names_is_read_in_one_request(self) -> None:
+ """No size is looked up before a read: the read itself shows the length, which the hash then checks."""
+ storage = _CountingStorage()
+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage, scheme="memory")
+ contents = {f"file_{index_file}.txt": f"content {index_file}".encode() for index_file in range(5)}
+ for data in contents.values():
+ storage.objects[_blob_key(data)] = data
+
+ tree = await read_request_tree(
+ document=_hash_map({tree_path: (data, len(data)) for tree_path, data in contents.items()}),
+ blob_store=blob_store,
+ label="The map",
+ field="files",
+ )
+
+ assert tree == RequestTree.make_from_contents(contents=contents)
+ assert sorted(storage.head_reads) == sorted((_blob_key(data), len(data) + 1) for data in contents.values())
+ assert storage.sized_keys == []
assert storage.loaded_keys == []
async def test_a_blob_not_held_is_missing_and_a_held_one_is_read_and_checked(self) -> None:
@@ -182,16 +242,22 @@ class TestRunnerBlobStore:
async def test_lookups_and_reads_run_a_bounded_number_at_a_time(self) -> None:
storage = _CountingStorage()
blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage, scheme="memory")
- blobs = [f"file number {index_blob}".encode() for index_blob in range(3 * MAX_CONCURRENT_BLOB_LOOKUPS)]
+ blobs = [f"file number {index_blob}".encode() for index_blob in range(3 * max(MAX_CONCURRENT_BLOB_LOOKUPS, MAX_CONCURRENT_BLOB_READS))]
for data in blobs:
storage.objects[_blob_key(data)] = data
stored = await blob_store.stored_sizes(sha256s=[sha256_hex(data) for data in blobs])
- held = await blob_store.read_held(sizes={sha256: size for sha256, size in stored.items() if size is not None})
+ stated = await blob_store.read_stated(stated_sizes={sha256_hex(data): {len(data)} for data in blobs})
- assert [held.read_blob(sha256_hex(data)) for data in blobs] == blobs
+ assert stored == {sha256_hex(data): len(data) for data in blobs}
+ assert [stated.held.read_blob(sha256_hex(data)) for data in blobs] == blobs
+ assert stated.mismatched == {}
assert storage.most_lookups_at_once == MAX_CONCURRENT_BLOB_LOOKUPS
- assert storage.most_loads_at_once == MAX_CONCURRENT_BLOB_TRANSFERS
+ assert storage.most_loads_at_once == MAX_CONCURRENT_BLOB_READS
+
+ async def test_the_s3_client_pool_holds_every_request_a_blob_store_runs_at_once(self) -> None:
+ """The S3 provider shares one client per loop, so a pool smaller than these would queue a request's own reads."""
+ assert max(MAX_CONCURRENT_BLOB_LOOKUPS, MAX_CONCURRENT_BLOB_READS) <= S3_MAX_POOL_CONNECTIONS
async def test_a_short_object_is_asked_for_and_its_upload_is_a_server_fault_that_writes_nothing(self) -> None:
"""A blob cut short is damaged storage, which no write repairs: nothing writes over a blob, so its operator must."""
diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.md
index 1fbf4c674..5924b1799 100644
--- a/docs/api-server/pipe-run.md
+++ b/docs/api-server/pipe-run.md
@@ -299,7 +299,7 @@ The answer is `{"missing": [{"sha256": "…", "size": n, "upload": {"method": "P
**Writes are create-only.** The runner creates a blob, or a tree's listing, only when its key holds nothing, and never replaces an object already there: on S3 the write carries `If-None-Match: *` and its SHA-256 as `x-amz-checksum-sha256`, which S3 checks against the body, on GCS it is conditional on generation 0, and on the local filesystem the file is created through a hard link, so the local storage's folder must sit on a filesystem that supports them. A key already holding an object is the object already held, the same bytes since the key is their hash, and the write succeeds without changing it. A deployment's bucket can therefore refuse every unconditional write under `blobs/` and `trees/` without refusing any of the runner's, as the hosted platform's does.
-**Sizes.** A blob is held at a size, which the runner reads from the storage's metadata without reading the bytes. `POST /v1/blobs/missing` lists a blob the runner does not hold at the size the sender states, an object held at another size included. Its upload then tells the two cases apart: bytes of the size held are the ones held, and change nothing, while bytes of another size mean the object held is damaged, which only a storage that does not publish a write whole could leave, a `500` `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, since nothing writes over a blob and the deployment's operator removes the object for it to be uploaded again. On a run or a validation, every hash entry is looked up by size before any byte is read: a hash the runner holds at another size than the entry states is a `422` `request.invalid` naming the field the entry is in, since the bytes under a hash have one size and uploading them again would change nothing, so the entry's size is the one to correct; the operator view's `diagnostics.message` names each such hash with the size it is held at. Only bytes held at the size their entry states are read, a bounded number at a time, and checked against their hash; bytes that do not hash to their key mean the storage holds a damaged object, a `500` `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, since a retry reads the same object. In both cases the operator repairs it by removing the object the line `Stored blob bytes do not match their hash` names under `blob_sha256`, for it to be uploaded again.
+**Sizes.** A blob is held at a size, which the runner reads from the storage's metadata without reading the bytes. `POST /v1/blobs/missing` lists a blob the runner does not hold at the size the sender states, an object held at another size included. Its upload then tells the two cases apart: bytes of the size held are the ones held, and change nothing, while bytes of another size mean the object held is damaged, which only a storage that does not publish a write whole could leave, a `500` `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, since nothing writes over a blob and the deployment's operator removes the object for it to be uploaded again. On a run or a validation, every hash entry is read in one request, a bounded number at a time, which reads at most one byte more than the size the entry states, so the bytes read stay within the sizes the request states however large the object held is. A hash the runner holds at another size than the entry states, which that read shows, is a `422` `request.invalid` naming the field the entry is in, since the bytes under a hash have one size and uploading them again would change nothing, so the entry's size is the one to correct; the operator view's `diagnostics.message` names each such hash with the size it is held at, which the runner looks up from the storage's metadata when the read stopped short of the object's end. Bytes read at the size their entry states are checked against their hash; bytes that do not hash to their key mean the storage holds a damaged object, a `500` `pipelex.storage.damaged` naming the kind of storage, whose verdict is not to retry, since a retry reads the same object. In both cases the operator repairs it by removing the object the line `Stored blob bytes do not match their hash` names under `blob_sha256`, for it to be uploaded again.
**The digest shortcut.** Once the runner has read a request's tree in full, it records the tree's listing beside its blobs, after every file it names is held, and an upload never records one. A validation records it only when its request carries a `read_scope`: without one, its scope would be the caller's own id, where a dry run must leave nothing stored. Asking `POST /v1/blobs/missing` with the digest of a recorded tree then answers an empty list in one lookup, however many files the tree has, when every blob listed is a file of that tree; a blob outside it is looked up as if no digest were sent.
diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.md
index 6e1e58720..94341484c 100644
--- a/docs/under-the-hood/storage-provider-plugins.md
+++ b/docs/under-the-hood/storage-provider-plugins.md
@@ -147,7 +147,7 @@ The duplicate detection is the same fail-loud `_add` helper the inference/orches
A third-party storage plugin is a distribution that:
-1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and takes the bytes' SHA-256 as `checksum_sha256_hex`, which it sends when its storage can check a write's checksum and ignores otherwise (see [Checksums](#checksums)), and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object, and one that can create an object only when its key is free also overrides `_store_if_absent` (see [Create-only writes](#create-only-writes));
+1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and takes the bytes' SHA-256 as `checksum_sha256_hex`, which it sends when its storage can check a write's checksum and ignores otherwise (see [Checksums](#checksums)), and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object, one that can create an object only when its key is free also overrides `_store_if_absent` (see [Create-only writes](#create-only-writes)), and one that keeps a client open between operations closes it in `teardown` (see [Clients and teardown](#clients-and-teardown));
2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;
3. advertises itself under the `pipelex.plugins.kernel` entry-point group — a storage provider is kernel-layer, and the group is how a plugin declares that (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means and what happens if you pick the wrong one; the pre-split `pipelex.plugins` group is retired and now fails startup):
@@ -173,6 +173,16 @@ Each built-in provider makes the condition atomic in its storage: S3 sends `If-N
The S3 provider sends the SHA-256 with every write it makes, `store` and `store_if_absent` alike, so S3 refuses a body that does not hash to it, answering `BadDigest`, which the provider raises as `StorageS3Error`, and keeps the checksum beside the object, where a `HeadObject` with `ChecksumMode=ENABLED` returns it as `ChecksumSHA256`. A reader of the bucket therefore learns the full hash of any object the provider wrote without reading its bytes. Each write passes `ChecksumAlgorithm="SHA256"`, and with the caller's digest `ChecksumSHA256` as well, which botocore sends as the `x-amz-checksum-sha256` header; without one, botocore hashes the body as it sends it and sends the checksum as a trailer of the upload. The client never signs the payload (`payload_signing_enabled` is off). botocore already leaves it unsigned when it computes the checksum itself; with the caller's digest, the checksum header is among the request's signed headers, sent over TLS, which protects the body as well as a signed payload would, and botocore, which recognises only a checksum of its own, would otherwise hash the whole body a second time just to sign it. The other built-in providers take no SHA-256 checksum: GCS and its client library check a CRC32C or an MD5 of the upload, and a local file and the in-memory store have no checksum of their own.
+### Clients and teardown
+
+The S3 provider keeps one aiobotocore client per event loop. The first operation a loop runs opens it, and every later operation on that loop shares it and its pool of connections, `S3_MAX_POOL_CONNECTIONS` of them, so an operation pays neither a client build nor a new TLS connection to S3; an operation finding every connection busy waits for one. It must be one per loop, because an aiobotocore client and the aiohttp session under it belong to the loop that opened them, and the runtime reaches storage from more than one: a command line runs each step under its own `asyncio.run`, and a test suite runs a loop per class.
+
+A client is closed on its own loop, at whichever of two moments comes first. A loop that shuts its asynchronous generators down, as `asyncio.run` does once its tasks have ended, closes the client as it shuts down, through a generator the provider started on that loop with the client. The runtime's teardown calls `teardown()` on the provider it booted with, which schedules the close on a loop still running, the server's own loop when an API server shuts down, and runs a loop no longer running until the close is done. A loop closed without shutting its generators down leaves its client open, with nothing left to close it on, which aiohttp reports as an unclosed session. A close that fails is logged at DEBUG and never raised out of the teardown, so the other loops' clients are still closed and the rest of the runtime's teardown still runs. An operation after a teardown opens a new client.
+
+The GCS provider needs none of this: its client is the synchronous `google-cloud-storage` one, built once and called in worker threads, whose connections belong to no loop.
+
+`StorageProviderAbstract.teardown()` holds nothing by default. A plugin's provider that does its own I/O through a client it keeps between operations closes that client there. The teardown is synchronous and may run on the very thread running the loop the client belongs to, so a client bound to a loop is closed by scheduling its close on that loop, never by waiting for it, and the provider still works afterwards, opening a new client on its next operation. It should not raise: a raise skips the releases the runtime's teardown makes after it.
+
### A provider is not the tenant boundary
A provider reads any key it is handed with the process's own credentials, and it should: it has no notion of whose run asked. On a host serving several tenants, what bounds a run's reads is the **read scope** the host passes with the run, which the content-generation leaves and the input seam check before they call a provider. A key outside that prefix, and any local path, is refused there, so a provider never sees it. A self-hosted deployment serving one tenant passes no read scope and reads as before. See [Distributed Content Generation](distributed-content-generation.md#what-a-leaf-may-read-the-read-scope) for the rule and where it runs.
diff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.py
index 8894a4da7..1099a2116 100644
--- a/pipelex/runtime_boot.py
+++ b/pipelex/runtime_boot.py
@@ -753,6 +753,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.sdk_client_manager.teardown()
# tools
+ self.runtime_hub.get_storage_provider().teardown()
if self.class_registry:
self.class_registry.teardown()
diff --git a/pipelex/tools/storage/local_storage_provider.py b/pipelex/tools/storage/local_storage_provider.py
index 6dfb9b57d..5c909b6e6 100644
--- a/pipelex/tools/storage/local_storage_provider.py
+++ b/pipelex/tools/storage/local_storage_provider.py
@@ -105,6 +105,34 @@ class LocalStorageProvider(StorageProviderAbstract):
return StoredData(data=data, mime_type=mime_type)
+ @override
+ async def _load_head(self, key: str, *, nb_bytes: int) -> bytes:
+ """Read the first bytes of a file, instead of the whole file.
+
+ Args:
+ key: Storage key (relative path, without scheme prefix).
+ nb_bytes: How many leading bytes to read. A file shorter than this is read whole.
+
+ Returns:
+ At most `nb_bytes` leading bytes of the file.
+
+ Raises:
+ StorageFileNotFoundError: If the file does not exist.
+ StorageInvalidUriError: If the key is invalid.
+ StorageLocalError: If the filesystem read fails (e.g. permission denied, I/O error).
+ """
+ file_path = self._validate_key(key)
+ try:
+ async with aiofiles.open(file_path, "rb") as file_handle: # pyright: ignore[reportUnknownMemberType]
+ data: bytes = await file_handle.read(nb_bytes)
+ except FileNotFoundError as exc:
+ msg = f"File not found: '{key}'"
+ raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
+ except OSError as exc:
+ msg = f"Failed to read file '{key}': {exc}"
+ raise StorageLocalError(msg) from exc
+ return data
+
@override
async def _load_size(self, key: str) -> int:
"""The size of a file, read from its metadata without opening it.
diff --git a/pipelex/tools/storage/s3_storage_provider.py b/pipelex/tools/storage/s3_storage_provider.py
index 5ac6849a8..67d67bc87 100644
--- a/pipelex/tools/storage/s3_storage_provider.py
+++ b/pipelex/tools/storage/s3_storage_provider.py
@@ -1,20 +1,34 @@
+import asyncio
import base64
import importlib.util
+import threading
from asyncio import sleep
+from collections.abc import AsyncGenerator, Callable
+from concurrent.futures import Future
+from contextlib import AbstractAsyncContextManager, AsyncExitStack
from typing import Any, ClassVar
from urllib.parse import quote
from typing_extensions import override
+from pipelex import log
from pipelex.errors.error_codes import ErrorCode
from pipelex.errors.registry.pipelex_codes import PipelexErrorCode
from pipelex.system.exceptions import MissingDependencyError
+from pipelex.tools.log.error_fields import error_fields
+from pipelex.tools.misc.async_utils import is_event_loop_running_on_this_thread
from pipelex.tools.storage.exceptions import (
StorageFileNotFoundError,
StorageS3Error,
)
from pipelex.tools.storage.storage_provider_abstract import StorageKind, StorageProviderAbstract, StoredData
+# How many connections the S3 client of one event loop keeps open at most. Every operation on the loop shares the
+# client, so its pool serves every request the loop runs at once: the API server's blob store runs its lookups and its
+# reads for one request a bounded number at a time, which its tests hold to this pool, and the pool leaves room for a
+# second request doing the same. An operation finding every connection busy waits for one, it is not refused.
+S3_MAX_POOL_CONNECTIONS = 32
+
# The S3 error codes by which S3 refuses the credentials themselves, rather than failing to serve: an operator fixes these.
_S3_CREDENTIAL_REFUSAL_CODES = frozenset({"AccessDenied", "InvalidAccessKeyId", "SignatureDoesNotMatch", "ExpiredToken", "InvalidToken"})
@@ -66,11 +80,96 @@ def _checksum_sha256_put_params(*, checksum_sha256_hex: str | None) -> dict[str,
return params
+class _LoopClient:
+ """The S3 client of one event loop, opened on the loop's first operation and shared by every later one.
+
+ An aiobotocore client and the aiohttp session under it belong to the loop that opened them, so each loop gets a
+ client of its own. Its closer is an asynchronous generator iterated once on that loop, which registers it with the
+ loop: a loop shutting its asynchronous generators down, as `asyncio.run` does once its tasks have ended, closes the
+ closer, which closes the client on its own loop before the loop closes. The provider's teardown closes it sooner, on
+ a loop still running.
+ """
+
+ def __init__(self, *, loop: asyncio.AbstractEventLoop) -> None:
+ self.loop = loop
+ self._opening = asyncio.Lock()
+ self._releases = AsyncExitStack()
+ self._client: Any = None
+ self._closer: AsyncGenerator[None] | None = None
+ self.is_closed = False
+
+ async def client(self, *, open_client: Callable[[], AbstractAsyncContextManager[Any]]) -> Any:
+ """The loop's client, opened with `open_client` by the first caller, the others on the loop waiting for it.
+
+ A client that failed to open is opened again by the next caller.
+ """
+ if self._client is None:
+ async with self._opening:
+ if self._client is None:
+ client = await self._releases.enter_async_context(open_client())
+ closer = self._close_at_loop_shutdown()
+ # The first iteration runs on the loop, which then holds the closer and closes it as it shuts down.
+ await anext(closer)
+ self._closer = closer
+ self._client = client
+ return self._client
+
+ async def _close_at_loop_shutdown(self) -> AsyncGenerator[None]:
+ try:
+ yield
+ finally:
+ self.is_closed = True
+ self._client = None
+ await self._releases.aclose()
+
+ async def close(self) -> None:
+ """Close the client, on its own loop. Closing it again does nothing."""
+ if self._closer is not None:
+ await self._closer.aclose()
+
+ def close_soon(self) -> None:
+ """Close the client from synchronous code on any thread, as far as its loop allows, without waiting for it.
+
+ On a running loop the close is scheduled there. A loop no longer running is run until the close is done, unless
+ this thread runs another loop, which cannot run a second one: that client is left to its loop's own shutdown. A
+ closed loop has nothing left to close with, and holds no connection a client of it could still use.
+
+ It never raises, as it runs in the runtime's teardown: a close that fails is only logged, either way.
+ """
+ if self._closer is None or self.loop.is_closed():
+ return
+ if self.loop.is_running():
+ close_future = asyncio.run_coroutine_threadsafe(self.close(), self.loop)
+ close_future.add_done_callback(lambda done: self.log_close_failure(close_future=done))
+ elif not is_event_loop_running_on_this_thread():
+ try:
+ self.loop.run_until_complete(self.close())
+ except Exception as exc: # ruff: ignore[blind-except]
+ # (2) closing runs aiobotocore's and aiohttp's own teardown, whose exception surface is not documented;
+ # the runtime's teardown must never fail, and the other loops' clients must still be closed.
+ log.debug("An S3 client could not be closed", fields=error_fields(exc=exc))
+
+ @classmethod
+ def log_close_failure(cls, *, close_future: Future[None]) -> None:
+ """Say at DEBUG why a close scheduled on a running loop failed, as that close's done callback.
+
+ A cancelled close holds no exception to read: `Future.exception()` raises `CancelledError` for it, which would
+ escape the callback.
+ """
+ if close_future.cancelled():
+ return
+ close_exc = close_future.exception()
+ if close_exc is not None:
+ log.debug("An S3 client could not be closed", fields=error_fields(exc=close_exc))
+
+
class S3StorageProvider(StorageProviderAbstract):
"""Storage provider implementation for AWS S3 storage.
- Files are stored in an S3 bucket with keys being path strings.
- Uses aiobotocore for async S3 operations.
+ Files are stored in an S3 bucket with keys being path strings, through aiobotocore. Each event loop that reaches the
+ provider opens one S3 client on its first operation, and every operation on that loop shares it, with its pool of
+ connections, so an operation pays neither a client build nor a new TLS connection. The client is closed as its loop
+ shuts down, or by `teardown`, whichever comes first (`_LoopClient`).
"""
storage_kind: ClassVar[StorageKind] = "s3"
@@ -92,6 +191,41 @@ class S3StorageProvider(StorageProviderAbstract):
self._region = region
self._signed_urls_lifespan = signed_urls_lifespan
self._session: Any = None
+ # Several threads may each run a loop of their own, so the map of each loop's client is guarded.
+ self._loop_clients: dict[asyncio.AbstractEventLoop, _LoopClient] = {}
+ self._loop_clients_lock = threading.Lock()
+
+ async def _client(self) -> Any:
+ """The S3 client of the running event loop, opened on the loop's first operation."""
+ loop = asyncio.get_running_loop()
+ with self._loop_clients_lock:
+ loop_client = self._loop_clients.get(loop)
+ if loop_client is None or loop_client.is_closed:
+ # A loop that has closed never runs again: its entry is dropped as a new loop arrives, so a process
+ # running loop after loop, as a command line running each step under `asyncio.run` does, holds one.
+ for closed_loop in [known_loop for known_loop in self._loop_clients if known_loop.is_closed()]:
+ del self._loop_clients[closed_loop]
+ loop_client = _LoopClient(loop=loop)
+ self._loop_clients[loop] = loop_client
+ return await loop_client.client(open_client=self._open_client)
+
+ def _open_client(self) -> AbstractAsyncContextManager[Any]:
+ """The context that opens an S3 client on entry and closes it on exit."""
+ context: AbstractAsyncContextManager[Any] = self._get_session().create_client(**self._get_client_config())
+ return context
+
+ @override
+ def teardown(self) -> None:
+ """Close the S3 client of every loop still running, or still open, on that loop.
+
+ A client whose loop is running is closed there soon after, without this waiting for it, since this runs as the
+ runtime's synchronous teardown, on that very loop when a server shuts down. Any later operation opens a new one.
+ """
+ with self._loop_clients_lock:
+ loop_clients = list(self._loop_clients.values())
+ self._loop_clients = {}
+ for loop_client in loop_clients:
+ loop_client.close_soon()
def _check_dependency(self) -> None:
"""Check if aiobotocore is installed.
@@ -142,7 +276,11 @@ class S3StorageProvider(StorageProviderAbstract):
# caller gave it, it goes as the signed `x-amz-checksum-sha256` header, and botocore, which recognises only
# a checksum of its own, would hash the whole body once more just to sign it. Over TLS the signed checksum
# protects the body as well as a signed payload would, so the payload is never signed.
- config = AioConfig(signature_version="s3v4", s3={"addressing_style": "virtual", "payload_signing_enabled": False})
+ config = AioConfig(
+ signature_version="s3v4",
+ s3={"addressing_style": "virtual", "payload_signing_enabled": False},
+ max_pool_connections=S3_MAX_POOL_CONNECTIONS,
+ )
return {
"service_name": "s3",
@@ -171,36 +309,33 @@ class S3StorageProvider(StorageProviderAbstract):
NoCredentialsError,
)
- session = self._get_session()
- client_config = self._get_client_config()
-
- async with session.create_client(**client_config) as client:
- try:
- response = await client.get_object(Bucket=self._bucket_name, Key=key)
- async with response["Body"] as stream:
- data: bytes = await stream.read()
- # Extract ContentType from S3 response
- content_type: str | None = response.get("ContentType")
- return StoredData(data=data, mime_type=content_type)
- except client.exceptions.NoSuchKey as exc:
+ client = await self._client()
+ try:
+ response = await client.get_object(Bucket=self._bucket_name, Key=key)
+ async with response["Body"] as stream:
+ data: bytes = await stream.read()
+ # Extract ContentType from S3 response
+ content_type: str | None = response.get("ContentType")
+ return StoredData(data=data, mime_type=content_type)
+ except client.exceptions.NoSuchKey as exc:
+ msg = f"Object not found in S3: '{key}'"
+ raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
+ except client.exceptions.NoSuchBucket as exc:
+ msg = f"Bucket not found in S3: '{self._bucket_name}'"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
+ except ClientError as exc:
+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
+ if error_code == "NoSuchKey":
msg = f"Object not found in S3: '{key}'"
raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
- except client.exceptions.NoSuchBucket as exc:
- msg = f"Bucket not found in S3: '{self._bucket_name}'"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
- except ClientError as exc:
- error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
- if error_code == "NoSuchKey":
- msg = f"Object not found in S3: '{key}'"
- raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
- msg = f"S3 ClientError ({error_code}) for key '{key}'"
- raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
- except NoCredentialsError as exc:
- msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
- except BotoCoreError as exc:
- msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg) from exc
+ msg = f"S3 ClientError ({error_code}) for key '{key}'"
+ raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
+ except NoCredentialsError as exc:
+ msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
+ except BotoCoreError as exc:
+ msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg) from exc
@override
async def _load_head(self, key: str, *, nb_bytes: int) -> bytes:
@@ -223,38 +358,35 @@ class S3StorageProvider(StorageProviderAbstract):
NoCredentialsError,
)
- session = self._get_session()
- client_config = self._get_client_config()
-
- async with session.create_client(**client_config) as client:
- try:
- # The Range header's end offset is inclusive.
- response = await client.get_object(Bucket=self._bucket_name, Key=key, Range=f"bytes=0-{nb_bytes - 1}")
- async with response["Body"] as stream:
- data: bytes = await stream.read()
- return data[:nb_bytes]
- except client.exceptions.NoSuchKey as exc:
+ client = await self._client()
+ try:
+ # The Range header's end offset is inclusive.
+ response = await client.get_object(Bucket=self._bucket_name, Key=key, Range=f"bytes=0-{nb_bytes - 1}")
+ async with response["Body"] as stream:
+ data: bytes = await stream.read()
+ return data[:nb_bytes]
+ except client.exceptions.NoSuchKey as exc:
+ msg = f"Object not found in S3: '{key}'"
+ raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
+ except client.exceptions.NoSuchBucket as exc:
+ msg = f"Bucket not found in S3: '{self._bucket_name}'"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
+ except ClientError as exc:
+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
+ if error_code == "InvalidRange":
+ # S3 refuses any range over an empty object: its head is empty.
+ return b""
+ if error_code == "NoSuchKey":
msg = f"Object not found in S3: '{key}'"
raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
- except client.exceptions.NoSuchBucket as exc:
- msg = f"Bucket not found in S3: '{self._bucket_name}'"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
- except ClientError as exc:
- error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
- if error_code == "InvalidRange":
- # S3 refuses any range over an empty object: its head is empty.
- return b""
- if error_code == "NoSuchKey":
- msg = f"Object not found in S3: '{key}'"
- raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
- msg = f"S3 ClientError ({error_code}) for key '{key}'"
- raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
- except NoCredentialsError as exc:
- msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
- except BotoCoreError as exc:
- msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg) from exc
+ msg = f"S3 ClientError ({error_code}) for key '{key}'"
+ raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
+ except NoCredentialsError as exc:
+ msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
+ except BotoCoreError as exc:
+ msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg) from exc
@override
async def _load_size(self, key: str) -> int:
@@ -276,28 +408,25 @@ class S3StorageProvider(StorageProviderAbstract):
NoCredentialsError,
)
- session = self._get_session()
- client_config = self._get_client_config()
-
- async with session.create_client(**client_config) as client:
- try:
- response = await client.head_object(Bucket=self._bucket_name, Key=key)
- except ClientError as exc:
- error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
- # A HEAD answer has no body, so a missing object comes back as the bare status code.
- if error_code in {"404", "NoSuchKey", "NotFound"}:
- msg = f"Object not found in S3: '{key}'"
- raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
- msg = f"S3 ClientError ({error_code}) for key '{key}'"
- raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
- except NoCredentialsError as exc:
- msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
- except BotoCoreError as exc:
- msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg) from exc
- content_length: int = response["ContentLength"]
- return content_length
+ client = await self._client()
+ try:
+ response = await client.head_object(Bucket=self._bucket_name, Key=key)
+ except ClientError as exc:
+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
+ # A HEAD answer has no body, so a missing object comes back as the bare status code.
+ if error_code in {"404", "NoSuchKey", "NotFound"}:
+ msg = f"Object not found in S3: '{key}'"
+ raise StorageFileNotFoundError(msg).with_params(scheme=self.storage_kind) from exc
+ msg = f"S3 ClientError ({error_code}) for key '{key}'"
+ raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
+ except NoCredentialsError as exc:
+ msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
+ except BotoCoreError as exc:
+ msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg) from exc
+ content_length: int = response["ContentLength"]
+ return content_length
@override
async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
@@ -322,33 +451,30 @@ class S3StorageProvider(StorageProviderAbstract):
NoCredentialsError,
)
- session = self._get_session()
- client_config = self._get_client_config()
-
- async with session.create_client(**client_config) as client:
- try:
- put_params: dict[str, Any] = {
- "Bucket": self._bucket_name,
- "Key": key,
- "Body": data,
- }
- if content_type:
- put_params["ContentType"] = content_type
- put_params.update(_checksum_sha256_put_params(checksum_sha256_hex=checksum_sha256_hex))
- await client.put_object(**put_params)
- except client.exceptions.NoSuchBucket as exc:
- msg = f"Bucket not found in S3: '{self._bucket_name}'"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
- except ClientError as exc:
- error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
- msg = f"S3 ClientError ({error_code}) for key '{key}'"
- raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
- except NoCredentialsError as exc:
- msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
- except BotoCoreError as exc:
- msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg) from exc
+ client = await self._client()
+ try:
+ put_params: dict[str, Any] = {
+ "Bucket": self._bucket_name,
+ "Key": key,
+ "Body": data,
+ }
+ if content_type:
+ put_params["ContentType"] = content_type
+ put_params.update(_checksum_sha256_put_params(checksum_sha256_hex=checksum_sha256_hex))
+ await client.put_object(**put_params)
+ except client.exceptions.NoSuchBucket as exc:
+ msg = f"Bucket not found in S3: '{self._bucket_name}'"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
+ except ClientError as exc:
+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
+ msg = f"S3 ClientError ({error_code}) for key '{key}'"
+ raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
+ except NoCredentialsError as exc:
+ msg = f"S3 credentials not found for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_CREDENTIAL_MISSING) from exc
+ except BotoCoreError as exc:
+ msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg) from exc
@override
async def _store_if_absent(self, *, data: bytes, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> bool:
@@ -384,9 +510,6 @@ class S3StorageProvider(StorageProviderAbstract):
ClientError,
)
- session = self._get_session()
- client_config = self._get_client_config()
-
put_params: dict[str, Any] = {
"Bucket": self._bucket_name,
"Key": key,
@@ -399,28 +522,28 @@ class S3StorageProvider(StorageProviderAbstract):
retry_delays = CONDITIONAL_CONFLICT_RETRY_DELAYS_SECONDS
conditional_conflict: ClientError | None = None
- async with session.create_client(**client_config) as client:
- for index_attempt in range(len(retry_delays) + 1):
- if index_attempt > 0:
- await sleep(retry_delays[index_attempt - 1])
- try:
- await client.put_object(**put_params)
- except client.exceptions.NoSuchBucket as exc:
- msg = f"Bucket not found in S3: '{self._bucket_name}'"
- raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
- except ClientError as exc:
- error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
- if error_code in _PRECONDITION_FAILED_CODES:
- return False
- if error_code not in _CONDITIONAL_CONFLICT_CODES:
- msg = f"S3 ClientError ({error_code}) for key '{key}'"
- raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
- conditional_conflict = exc
- continue
- except BotoCoreError as exc:
- msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
- raise StorageS3Error(msg) from exc
- return True
+ client = await self._client()
+ for index_attempt in range(len(retry_delays) + 1):
+ if index_attempt > 0:
+ await sleep(retry_delays[index_attempt - 1])
+ try:
+ await client.put_object(**put_params)
+ except client.exceptions.NoSuchBucket as exc:
+ msg = f"Bucket not found in S3: '{self._bucket_name}'"
+ raise StorageS3Error(msg).with_error_code(PipelexErrorCode.CONFIG_INVALID) from exc
+ except ClientError as exc:
+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")
+ if error_code in _PRECONDITION_FAILED_CODES:
+ return False
+ if error_code not in _CONDITIONAL_CONFLICT_CODES:
+ msg = f"S3 ClientError ({error_code}) for key '{key}'"
+ raise StorageS3Error(msg).with_error_code(_code_for_s3_client_error(s3_error_code=error_code)) from exc
+ conditional_conflict = exc
+ continue
+ except BotoCoreError as exc:
+ msg = f"S3 backend error for key '{key}': {type(exc).__name__}"
+ raise StorageS3Error(msg) from exc
+ return True
msg = f"S3 ClientError (ConditionalRequestConflict) for key '{key}' on each of {len(retry_delays) + 1} attempts"
raise StorageS3Error(msg) from conditional_conflict
@@ -467,18 +590,15 @@ class S3StorageProvider(StorageProviderAbstract):
if self._signed_urls_lifespan is None:
return self._make_public_url(key)
- session = self._get_session()
- client_config = self._get_client_config()
-
- async with session.create_client(**client_config) as client:
- try:
- presigned_url: str = await client.generate_presigned_url(
- "get_object",
- Params={"Bucket": self._bucket_name, "Key": key},
- ExpiresIn=self._signed_urls_lifespan,
- )
- return presigned_url
- except (BotoCoreError, ClientError):
- # ClientError (signing rejected) and BotoCoreError (transport failure) both
- # fall back to the public URL — same "if signing fails, serve unsigned" semantics.
- return self._make_public_url(key)
+ client = await self._client()
+ try:
+ presigned_url: str = await client.generate_presigned_url(
+ "get_object",
+ Params={"Bucket": self._bucket_name, "Key": key},
+ ExpiresIn=self._signed_urls_lifespan,
+ )
+ return presigned_url
+ except (BotoCoreError, ClientError):
+ # ClientError (signing rejected) and BotoCoreError (transport failure) both
+ # fall back to the public URL — same "if signing fails, serve unsigned" semantics.
+ return self._make_public_url(key)
diff --git a/pipelex/tools/storage/storage_provider_abstract.py b/pipelex/tools/storage/storage_provider_abstract.py
index 72abcc6da..7c77ca161 100644
--- a/pipelex/tools/storage/storage_provider_abstract.py
+++ b/pipelex/tools/storage/storage_provider_abstract.py
@@ -31,6 +31,8 @@ class StorageProviderAbstract(ABC):
- _store(data, key, content_type, checksum_sha256_hex): Store data
- public_url(uri): Return human-readable link for URI
+ A subclass holding a client open between operations overrides ``teardown`` to close it.
+
A provider Pipelex ships names its kind in ``storage_kind``, the ``scheme`` its refusals carry; a plugin's provider
keeps the default, ``plugin``.
"""
@@ -260,6 +262,15 @@ class StorageProviderAbstract(ABC):
return True
return False
+ def teardown(self) -> None:
+ """Release what the provider keeps open between operations, such as a client and its connections.
+
+ The runtime's teardown calls it on the provider it booted with, from synchronous code, possibly on the thread
+ running the event loop the provider served. This default holds nothing. A provider that keeps a client
+ bound to an event loop closes it on that loop, scheduling the close on a loop still running rather than
+ waiting for it, and must still work afterwards, opening a new client on its next operation.
+ """
+
@abstractmethod
async def public_url(self, uri: str) -> str | None:
"""Return human-readable link for this URI.
diff --git a/tests/unit/pipelex/test_runtime_boot_storage_teardown.py b/tests/unit/pipelex/test_runtime_boot_storage_teardown.py
new file mode 100644
index 000000000..33b4598c3
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_storage_teardown.py
@@ -0,0 +1,53 @@
+from typing_extensions import override
+
+from pipelex.pipelex import Pipelex
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.tools.storage.exceptions import StorageFileNotFoundError
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract, StoredData
+
+
+def _test_integration_mode() -> IntegrationMode:
+ """The boot mode the session conftest uses, so a re-boot here matches the one it replaces."""
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+class _TeardownCountingStorage(StorageProviderAbstract):
+ """A storage holding nothing, counting its teardowns."""
+
+ def __init__(self) -> None:
+ self.teardown_count = 0
+
+ @override
+ async def _load_with_metadata(self, key: str) -> StoredData:
+ msg = f"File not found: '{key}'"
+ raise StorageFileNotFoundError(msg)
+
+ @override
+ async def _store(self, data: bytes, *, key: str, content_type: str | None, checksum_sha256_hex: str | None) -> None:
+ pass
+
+ @override
+ async def public_url(self, uri: str) -> str | None:
+ return None
+
+ @override
+ def teardown(self) -> None:
+ self.teardown_count += 1
+
+
+class TestRuntimeTeardownReleasesTheStorageProvider:
+ def test_the_runtime_teardown_tears_down_the_storage_provider_it_booted_with(self) -> None:
+ """A provider keeping a client open between operations, as the S3 one does, closes it when the runtime goes."""
+ storage_provider = _TeardownCountingStorage()
+
+ Pipelex.teardown_if_needed()
+ try:
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, storage_provider=storage_provider)
+ assert storage_provider.teardown_count == 0
+
+ Pipelex.teardown_if_needed()
+
+ assert storage_provider.teardown_count == 1
+ finally:
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
index 425c16501..cee479642 100644
--- a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
@@ -9,7 +9,12 @@ from pytest_mock import MockerFixture
from pipelex.errors.problem import ProblemView
from pipelex.errors.registry.pipelex_codes import PipelexErrorCode
from pipelex.tools.storage.exceptions import StorageFileNotFoundError, StorageInvalidKeyError, StorageS3Error
-from pipelex.tools.storage.s3_storage_provider import CONDITIONAL_CONFLICT_RETRY_DELAYS_SECONDS, S3StorageProvider, checksum_sha256_header_value
+from pipelex.tools.storage.s3_storage_provider import (
+ CONDITIONAL_CONFLICT_RETRY_DELAYS_SECONDS,
+ S3_MAX_POOL_CONNECTIONS,
+ S3StorageProvider,
+ checksum_sha256_header_value,
+)
from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME
# Test constants
@@ -154,6 +159,17 @@ class TestS3StorageProvider:
client_config = mock_aiobotocore["session"].create_client.call_args.kwargs["config"]
assert client_config.s3 == {"addressing_style": "virtual", "payload_signing_enabled": False}
+ async def test_the_client_pool_is_sized_for_operations_running_at_once(
+ self,
+ s3_provider_no_signed_urls: S3StorageProvider,
+ mock_aiobotocore: dict[str, Any],
+ ) -> None:
+ """Every operation on a loop shares one client, so its pool, not botocore's default of 10, bounds them."""
+ await s3_provider_no_signed_urls.store(data=b"bytes", key="scope/file.bin")
+
+ client_config = mock_aiobotocore["session"].create_client.call_args.kwargs["config"]
+ assert client_config.max_pool_connections == S3_MAX_POOL_CONNECTIONS
+
async def test_load_returns_data(
self,
mocker: MockerFixture,
@@ -478,8 +494,8 @@ class TestS3StorageProvider:
expected_public_url = f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/{key}"
assert display == expected_public_url
- async def test_session_is_reused(self, mock_aiobotocore: dict[str, Any]) -> None:
- """Test that the session is lazily initialized and reused."""
+ async def test_session_and_client_are_reused(self, mock_aiobotocore: dict[str, Any]) -> None:
+ """The session and the loop's client are opened on the first operation, and every later one shares them."""
provider = S3StorageProvider(
bucket_name=S3_TEST_BUCKET,
region=S3_TEST_REGION,
@@ -496,6 +512,7 @@ class TestS3StorageProvider:
assert first_call_count == 1
assert second_call_count == 1 # Session should only be created once
+ assert mock_aiobotocore["session"].create_client.call_count == 1
async def test_load_head_sends_a_ranged_get(
self,
diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider_client_reuse.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider_client_reuse.py
new file mode 100644
index 000000000..a788351a5
--- /dev/null
+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider_client_reuse.py
@@ -0,0 +1,214 @@
+"""The S3 client each event loop opens once and shares, and when it is closed.
+
+These tests run their own event loops, as a command line running each step under `asyncio.run` does, so they can watch
+a client outlive one operation, stay out of another loop, and close as its loop shuts down or the provider tears down.
+"""
+
+import asyncio
+from typing import Any, NamedTuple
+
+import pytest
+from botocore.exceptions import NoRegionError
+from pytest_mock import MockerFixture
+
+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider
+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME
+
+_URI = f"{PIPELEX_STORAGE_SCHEME}scope/blobs/abc"
+
+
+class _ClientEvent(NamedTuple):
+ kind: str
+ loop: asyncio.AbstractEventLoop
+
+
+class _RecordingSession:
+ """An aiobotocore session whose clients record, on the loop they run on, when they open and when they close."""
+
+ def __init__(self, *, mocker: MockerFixture, failing_opens: int = 0, failing_closes: int = 0) -> None:
+ self._mocker = mocker
+ self.failing_opens = failing_opens
+ self.failing_closes = failing_closes
+ self.events: list[_ClientEvent] = []
+
+ def kinds(self) -> list[str]:
+ return [event.kind for event in self.events]
+
+ def create_client(self, **_config: Any) -> Any:
+ session = self
+ client = self._mocker.AsyncMock()
+ client.head_object = self._mocker.AsyncMock(return_value={"ContentLength": 3})
+
+ class _ClientContext:
+ async def __aenter__(self) -> Any:
+ # Yield, so operations starting together all reach the opening before it ends.
+ await asyncio.sleep(0)
+ if session.failing_opens > 0:
+ session.failing_opens -= 1
+ raise NoRegionError
+ session.events.append(_ClientEvent(kind="open", loop=asyncio.get_running_loop()))
+ return client
+
+ async def __aexit__(self, *_exc_info: object) -> None:
+ if session.failing_closes > 0:
+ session.failing_closes -= 1
+ msg = "The connection could not be closed"
+ raise OSError(msg)
+ session.events.append(_ClientEvent(kind="close", loop=asyncio.get_running_loop()))
+
+ return _ClientContext()
+
+
+def _make_provider() -> S3StorageProvider:
+ return S3StorageProvider(bucket_name="test-bucket", region="us-east-1", signed_urls_lifespan=None)
+
+
+class TestS3StorageProviderClientReuse:
+ @pytest.fixture
+ def session(self, mocker: MockerFixture) -> _RecordingSession:
+ recording_session = _RecordingSession(mocker=mocker)
+ mocker.patch("aiobotocore.session.get_session", return_value=recording_session)
+ return recording_session
+
+ def test_operations_on_one_loop_share_one_client_closed_as_the_loop_shuts_down(self, session: _RecordingSession) -> None:
+ provider = _make_provider()
+ loops: list[asyncio.AbstractEventLoop] = []
+
+ async def _operate() -> None:
+ loops.append(asyncio.get_running_loop())
+ for _ in range(3):
+ assert await provider.load_size(uri=_URI) == 3
+ assert session.kinds() == ["open"]
+
+ asyncio.run(_operate())
+
+ assert session.events == [_ClientEvent(kind="open", loop=loops[0]), _ClientEvent(kind="close", loop=loops[0])]
+
+ def test_operations_starting_together_open_one_client(self, session: _RecordingSession) -> None:
+ provider = _make_provider()
+
+ async def _operate() -> None:
+ sizes = await asyncio.gather(*[provider.load_size(uri=_URI) for _ in range(8)])
+ assert sizes == [3] * 8
+
+ asyncio.run(_operate())
+
+ assert session.kinds() == ["open", "close"]
+
+ def test_each_loop_opens_its_own_client_and_closes_it_on_itself(self, session: _RecordingSession) -> None:
+ """An aiobotocore client belongs to the loop that opened it, so a second loop never reuses the first one's."""
+ provider = _make_provider()
+ loops: list[asyncio.AbstractEventLoop] = []
+
+ async def _operate() -> None:
+ loops.append(asyncio.get_running_loop())
+ await provider.load_size(uri=_URI)
+
+ asyncio.run(_operate())
+ asyncio.run(_operate())
+
+ assert session.events == [
+ _ClientEvent(kind="open", loop=loops[0]),
+ _ClientEvent(kind="close", loop=loops[0]),
+ _ClientEvent(kind="open", loop=loops[1]),
+ _ClientEvent(kind="close", loop=loops[1]),
+ ]
+
+ def test_teardown_on_the_running_loop_closes_its_client_soon_after(self, session: _RecordingSession) -> None:
+ """A server tears the runtime down from its own loop: the close is scheduled there, and a later operation reopens."""
+ provider = _make_provider()
+
+ async def _operate() -> None:
+ await provider.load_size(uri=_URI)
+ provider.teardown()
+ assert session.kinds() == ["open"]
+ for _ in range(5):
+ await asyncio.sleep(0)
+ assert session.kinds() == ["open", "close"]
+ await provider.load_size(uri=_URI)
+ assert session.kinds() == ["open", "close", "open"]
+
+ asyncio.run(_operate())
+
+ assert session.kinds() == ["open", "close", "open", "close"]
+
+ def test_teardown_closes_the_client_of_a_loop_no_longer_running(self, session: _RecordingSession) -> None:
+ provider = _make_provider()
+ loop = asyncio.new_event_loop()
+ try:
+ loop.run_until_complete(provider.load_size(uri=_URI))
+ assert session.kinds() == ["open"]
+
+ provider.teardown()
+
+ assert session.events == [_ClientEvent(kind="open", loop=loop), _ClientEvent(kind="close", loop=loop)]
+ finally:
+ loop.close()
+
+ def test_a_close_failing_on_a_stopped_loop_is_logged_and_the_other_loops_clients_still_close(self, mocker: MockerFixture) -> None:
+ """The runtime's teardown must never fail, so a client that cannot be closed does not keep the next one open."""
+ failing_session = _RecordingSession(mocker=mocker, failing_closes=1)
+ mocker.patch("aiobotocore.session.get_session", return_value=failing_session)
+ log_debug = mocker.patch("pipelex.tools.storage.s3_storage_provider.log.debug")
+ provider = _make_provider()
+ first_loop = asyncio.new_event_loop()
+ second_loop = asyncio.new_event_loop()
+ try:
+ first_loop.run_until_complete(provider.load_size(uri=_URI))
+ second_loop.run_until_complete(provider.load_size(uri=_URI))
+
+ provider.teardown()
+
+ assert failing_session.events == [
+ _ClientEvent(kind="open", loop=first_loop),
+ _ClientEvent(kind="open", loop=second_loop),
+ _ClientEvent(kind="close", loop=second_loop),
+ ]
+ log_debug.assert_called_once()
+ assert log_debug.call_args.args == ("An S3 client could not be closed",)
+ finally:
+ first_loop.close()
+ second_loop.close()
+
+ def test_a_close_failing_on_the_running_loop_is_logged(self, mocker: MockerFixture) -> None:
+ failing_session = _RecordingSession(mocker=mocker, failing_closes=1)
+ mocker.patch("aiobotocore.session.get_session", return_value=failing_session)
+ log_debug = mocker.patch("pipelex.tools.storage.s3_storage_provider.log.debug")
+ provider = _make_provider()
+
+ async def _operate() -> None:
+ await provider.load_size(uri=_URI)
+ provider.teardown()
+ for _ in range(5):
+ await asyncio.sleep(0)
+ log_debug.assert_called_once()
+ assert log_debug.call_args.args == ("An S3 client could not be closed",)
+
+ asyncio.run(_operate())
+
+ assert failing_session.kinds() == ["open"]
+
+ def test_teardown_leaves_a_closed_loop_s_client_alone(self, session: _RecordingSession) -> None:
+ """A loop closed without shutting its generators down left its client open, and nothing can run its close any more."""
+ provider = _make_provider()
+ loop = asyncio.new_event_loop()
+ loop.run_until_complete(provider.load_size(uri=_URI))
+ loop.close()
+
+ provider.teardown()
+
+ assert session.kinds() == ["open"]
+
+ def test_a_client_that_failed_to_open_is_opened_again_by_the_next_operation(self, mocker: MockerFixture) -> None:
+ failing_session = _RecordingSession(mocker=mocker, failing_opens=1)
+ mocker.patch("aiobotocore.session.get_session", return_value=failing_session)
+ provider = _make_provider()
+
+ async def _operate() -> None:
+ with pytest.raises(NoRegionError):
+ await provider.load_size(uri=_URI)
+ assert await provider.load_size(uri=_URI) == 3
+
+ asyncio.run(_operate())
+
+ assert failing_session.kinds() == ["open", "close"]
diff --git a/tests/unit/pipelex/tools/storage/test_storage_load_head.py b/tests/unit/pipelex/tools/storage/test_storage_load_head.py
index 0397435f0..0c87d458d 100644
--- a/tests/unit/pipelex/tools/storage/test_storage_load_head.py
+++ b/tests/unit/pipelex/tools/storage/test_storage_load_head.py
@@ -1,6 +1,6 @@
"""The default head read every storage provider inherits: the whole object, sliced.
-A provider that can read a byte range (S3, GCS) overrides it; one that cannot, including an external
+A provider that can read a byte range (local, S3, GCS) overrides it; one that cannot, including an external
storage plugin written before the head read existed, keeps working through this default.
"""