Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feat(delivery): persist token usage as a tokens_usages.json result artifact (#1058)
Louis Choquel27864c8470fe
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2cf0d010b..c74c6eedb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,10 @@## [Unreleased]+### Added++- **Durable runs now persist token usage: the delivery executor writes a `tokens_usages.json` result artifact** (`{"tokens_usages": [...], "usage_assembly_error": null}`) alongside `working_memory.json`, the `main_stuff.*` renders, and the graph outputs. The records use the same per-record `model_dump(mode="json")` wire shape the `/execute` response carries on `pipe_output.tokens_usages` (token counts by category, `unit_costs` in $/1M, model id — for LLM, img-gen, extract, and search calls), so a client polling a durable run's result files can compute costs the same way a sync `/execute` caller does. The artifact is written unconditionally: a run with usage assembly off yields explicit nulls, distinguishable from a pre-artifact run (file absent).+### Fixed- **Validation errors stay domain-qualified: every raise site that names a `pipe_code` now also carries `domain_code`.** The presentation chain (VS Code extension + mthds-ui) identifies pipes by full pipe ref (`domain_code.pipe_code`); a handful of `PipeValidationError` raise sites (PipeLLM static input checks, PipeStructure input-mismatch, the pipe sorter's circular-dependency error) omitted `domain_code`, degrading node decorations and click-to-navigate for those errors in multi-domain bundles. The sorter gains a `domain_code` parameter threaded from the bundle spec's domain.diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.mdindex a8716cda1..f8d12e031 100644--- a/docs/under-the-hood/execution-graph-tracing.md+++ b/docs/under-the-hood/execution-graph-tracing.md@@ -262,6 +262,8 @@ graph_spec = manager.close_tracer(pipeline_run_id)Trace events travel through an `EventLogProtocol` backend (`pipelex/tracing/`): the tracer emits events into it during the run (write side, wired in `pipeline_run_setup`), and `assemble_tracing` reads them back after the run to build the `GraphSpec` and usage aggregates (read side, triggered from `PipeRun.run`). Both sides normally build their backend instance independently from `tracing_config` via `make_event_log` — NDJSON files or DynamoDB bridge the two instances through external storage.+The assembled usage rides back on `pipe_output.tokens_usages` (with any assembly failure on `usage_assembly_error`), which the sync `/execute` response returns directly. For delivery-enabled runs (a storage target set), the delivery executor (`pipelex/pipe_run/delivery_executor.py`) also persists it as a `tokens_usages.json` result artifact — `{"tokens_usages": [...], "usage_assembly_error": null}` — next to `working_memory.json`, the `main_stuff.*` renders, and the graph outputs, so a durable client polling result files gets the same usage records a sync caller does. The artifact is written unconditionally: explicit nulls mean usage assembly was off for that run, while an absent file means the run was delivered before the artifact existed.+For fully in-process runs, `pipelex.hub.scoped_event_log` pins one shared instance for both sides instead:```pythondiff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.pyindex 20173627b..74c9f88a8 100644--- a/pipelex/pipe_run/delivery_executor.py+++ b/pipelex/pipe_run/delivery_executor.py@@ -80,7 +80,7 @@ class DeliveryExecutor:"""Generate the full set of result files from a PipeOutput.Produces: working_memory.json, main_stuff (json/md/html/viewer),- graph outputs (mermaidflow, reactflow, graphspec).+ tokens_usages.json, graph outputs (mermaidflow, reactflow, graphspec).Supports two `pipe_output` shapes:- Typed: `working_memory` populated (in-process / same-worker path).@@ -133,12 +133,33 @@ class DeliveryExecutor:else:await self._generate_main_stuff_files(main_resolved, files=files)+ files["tokens_usages.json"] = self._generate_usage_file(pipe_output)+graph_spec = pipe_output.graph_specif graph_spec:await self._generate_graph_files(graph_spec, files=files)return files+ @classmethod+ def _generate_usage_file(cls, pipe_output: PipeOutput) -> ResultFile:+ """Serialize the run's assembled usage onto the tokens_usages.json artifact.++ Written unconditionally, so a durable client polling the result files can tell+ "usage assembly was off for this run" (file present, ``tokens_usages`` null) from+ "run delivered before the artifact existed" (file absent). The records use the same+ per-usage ``model_dump(mode="json")`` wire shape the ``/execute`` response carries+ on ``pipe_output.tokens_usages``.+ """+ tokens_usages_dump: list[dict[str, Any]] | None = None+ if pipe_output.tokens_usages is not None:+ tokens_usages_dump = [tokens_usage.model_dump(mode="json") for tokens_usage in pipe_output.tokens_usages]+ usage_doc: dict[str, Any] = {+ "tokens_usages": tokens_usages_dump,+ "usage_assembly_error": pipe_output.usage_assembly_error,+ }+ return ResultFile(data=clean_json_dumps(usage_doc, indent=2).encode("utf-8"), content_type="application/json")+@classmethoddef _get_raw_main_stuff_dict(cls, *, working_memory_raw: dict[str, Any]) -> dict[str, Any] | None:"""Extract the main stuff dict from a raw working_memory, following aliases."""diff --git a/subject_grants.toml b/subject_grants.tomlindex 64c57444b..15eeeb747 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -4324,6 +4324,10 @@ rationale = "Verb-object: renders files from the main stuff; options stay keyworparam = "raw_main_stuff"rationale = "Verb-object: renders files from the raw main stuff; options stay keyword."+["pipelex/pipe_run/delivery_executor.py::DeliveryExecutor._generate_usage_file"]+param = "pipe_output"+rationale = "Verb-object helper mirroring its sibling generators: serialize this pipe_output's usage into the tokens_usages.json artifact."+["pipelex/pipe_run/delivery_executor.py::DeliveryExecutor._store_results"]param = "pipe_output"rationale = "Verb-object: stores the pipe output; options stay keyword."diff --git a/tests/integration/pipelex/tools/storage/conftest.py b/tests/integration/pipelex/tools/storage/conftest.pyindex 7adb64a06..187869c81 100644--- a/tests/integration/pipelex/tools/storage/conftest.py+++ b/tests/integration/pipelex/tools/storage/conftest.py@@ -1,9 +1,8 @@from pathlib import Pathfrom typing import Any, Literal, cast-from unittest.mock import AsyncMock, MagicMockimport pytest-from pytest_mock import MockerFixture+from pytest_mock import MockerFixture, MockTypefrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.config import get_config@@ -39,16 +38,16 @@ def s3_mock(mocker: MockerFixture) -> dict[str, Any]:# Storage for mock datamock_data_storage: dict[str, bytes] = {}- def create_mock_stream(key: str) -> AsyncMock:+ def create_mock_stream(key: str) -> MockType:"""Create a mock async stream for response body."""- stream = AsyncMock()- stream.read = AsyncMock(side_effect=lambda: mock_data_storage.get(key, b""))- stream.__aenter__ = AsyncMock(return_value=stream)- stream.__aexit__ = AsyncMock(return_value=None)+ stream: MockType = mocker.AsyncMock()+ stream.read = mocker.AsyncMock(side_effect=lambda: mock_data_storage.get(key, b""))+ stream.__aenter__ = mocker.AsyncMock(return_value=stream)+ stream.__aexit__ = mocker.AsyncMock(return_value=None)return stream# Create mock exceptions- mock_exceptions = MagicMock()+ mock_exceptions = mocker.MagicMock()mock_exceptions.NoSuchKey = type("NoSuchKey", (Exception,), {})mock_exceptions.NoSuchBucket = type("NoSuchBucket", (Exception,), {})mock_exceptions.ClientError = type("ClientError", (Exception,), {})@@ -72,20 +71,20 @@ def s3_mock(mocker: MockerFixture) -> dict[str, Any]:return f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/{key}?X-Amz-Signature=mock"# Create mock client with sync functions wrapped in AsyncMock- mock_client = AsyncMock()- mock_client.get_object = AsyncMock(side_effect=mock_get_object)- mock_client.put_object = AsyncMock(side_effect=mock_put_object)- mock_client.generate_presigned_url = AsyncMock(side_effect=mock_generate_presigned_url)+ mock_client = mocker.AsyncMock()+ mock_client.get_object = mocker.AsyncMock(side_effect=mock_get_object)+ mock_client.put_object = mocker.AsyncMock(side_effect=mock_put_object)+ mock_client.generate_presigned_url = mocker.AsyncMock(side_effect=mock_generate_presigned_url)mock_client.exceptions = mock_exceptions# Create async context manager for client- mock_client_context = AsyncMock()- mock_client_context.__aenter__ = AsyncMock(return_value=mock_client)- mock_client_context.__aexit__ = AsyncMock(return_value=None)+ mock_client_context = mocker.AsyncMock()+ mock_client_context.__aenter__ = mocker.AsyncMock(return_value=mock_client)+ mock_client_context.__aexit__ = mocker.AsyncMock(return_value=None)# Create mock session- mock_session = MagicMock()- mock_session.client = MagicMock(return_value=mock_client_context)+ mock_session = mocker.MagicMock()+ mock_session.client = mocker.MagicMock(return_value=mock_client_context)# Patch aioboto3.Sessionmocker.patch("aioboto3.Session", return_value=mock_session)@@ -114,8 +113,8 @@ def gcp_mock(tmp_path: Path, mocker: MockerFixture) -> dict[str, Any]:mock_data_storage: dict[str, bytes] = {}# Create mock blob that behaves like a real blob- def create_mock_blob(key: str) -> MagicMock:- blob = MagicMock()+ def create_mock_blob(key: str) -> MockType:+ blob: MockType = mocker.MagicMock()blob.exists.side_effect = lambda: key in mock_data_storageblob.download_as_bytes.side_effect = lambda: mock_data_storage[key]blob.upload_from_string.side_effect = lambda data, content_type=None: mock_data_storage.__setitem__(key, data) # noqa: ARG005 # pyright: ignore[reportUnknownLambdaType,reportUnknownArgumentType]@@ -123,11 +122,11 @@ def gcp_mock(tmp_path: Path, mocker: MockerFixture) -> dict[str, Any]:return blob# Create mock bucket- mock_bucket = MagicMock()+ mock_bucket = mocker.MagicMock()mock_bucket.blob.side_effect = create_mock_blob# Create mock client- mock_client = MagicMock()+ mock_client = mocker.MagicMock()mock_client.bucket.return_value = mock_bucket# Patch the Client.from_service_account_json method directlydiff --git a/tests/unit/pipelex/builder/test_runner_generator.py b/tests/unit/pipelex/builder/test_runner_generator.pyindex 17d1fa5fa..98993b852 100644--- a/tests/unit/pipelex/builder/test_runner_generator.py+++ b/tests/unit/pipelex/builder/test_runner_generator.py@@ -3,8 +3,7 @@from __future__ import annotationsimport json-from typing import Any, cast-from unittest.mock import MagicMock+from typing import TYPE_CHECKING, Any, castimport pytest@@ -15,6 +14,9 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecsfrom pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec+if TYPE_CHECKING:+ from pytest_mock import MockerFixture, MockType+class TestConceptGenerateInputRepresentationJson:"""Test Concept.render_concept_representation for JSON format."""@@ -197,75 +199,75 @@ class TestGenerateRunnerCode:"""Test generate_runner_code function."""@pytest.fixture- def mock_pipe_single_output(self) -> MagicMock:+ def mock_pipe_single_output(self, mocker: MockerFixture) -> MockType:"""Create a mock pipe with a single output."""text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)document_concept = ConceptFactory.make_native_concept(NativeConceptCode.DOCUMENT)- mock_pipe = MagicMock()+ mock_pipe: MockType = mocker.MagicMock()mock_pipe.code = "test_pipe"mock_pipe.output = StuffSpec(concept=text_concept)mock_pipe.inputs = InputStuffSpecs(root={"document": StuffSpec(concept=document_concept)})return mock_pipe@pytest.fixture- def mock_pipe_list_output(self) -> MagicMock:+ def mock_pipe_list_output(self, mocker: MockerFixture) -> MockType:"""Create a mock pipe with a list output."""text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)document_concept = ConceptFactory.make_native_concept(NativeConceptCode.DOCUMENT)- mock_pipe = MagicMock()+ mock_pipe: MockType = mocker.MagicMock()mock_pipe.code = "test_pipe_list"mock_pipe.output = StuffSpec(concept=text_concept)mock_pipe.inputs = InputStuffSpecs(root={"documents": StuffSpec(concept=document_concept, multiplicity=True)})return mock_pipe- def test_runner_code_includes_imports(self, mock_pipe_single_output: MagicMock) -> None:+ def test_runner_code_includes_imports(self, mock_pipe_single_output: MockType) -> None:"""Test that generated runner code includes necessary imports."""runner_code = generate_runner_code(mock_pipe_single_output)assert "import asyncio" in runner_codeassert "from pipelex.pipelex import Pipelex" in runner_codeassert "from pipelex.pipeline.runner import PipelexMTHDSProtocol" in runner_code- def test_runner_code_includes_structure_imports(self, mock_pipe_single_output: MagicMock) -> None:+ def test_runner_code_includes_structure_imports(self, mock_pipe_single_output: MockType) -> None:"""Test that generated runner code includes structure class imports."""runner_code = generate_runner_code(mock_pipe_single_output)assert "from pipelex.core.stuffs.document_content import DocumentContent" in runner_codeassert "from pipelex.core.stuffs.text_content import TextContent" in runner_code- def test_runner_code_single_output_return_type(self, mock_pipe_single_output: MagicMock) -> None:+ def test_runner_code_single_output_return_type(self, mock_pipe_single_output: MockType) -> None:"""Test that generated runner code has correct return type for single output."""runner_code = generate_runner_code(mock_pipe_single_output, output_multiplicity=False)assert "async def run_test_pipe() -> TextContent:" in runner_codeassert "pipe_output.main_stuff_as(content_type=TextContent)" in runner_code- def test_runner_code_list_output_return_type(self, mock_pipe_list_output: MagicMock) -> None:+ def test_runner_code_list_output_return_type(self, mock_pipe_list_output: MockType) -> None:"""Test that generated runner code has correct return type for list output."""runner_code = generate_runner_code(mock_pipe_list_output, output_multiplicity=True)assert "async def run_test_pipe_list() -> list[TextContent]:" in runner_codeassert "pipe_output.main_stuff_as_items(item_type=TextContent)" in runner_code- def test_runner_code_includes_inputs(self, mock_pipe_single_output: MagicMock) -> None:+ def test_runner_code_includes_inputs(self, mock_pipe_single_output: MockType) -> None:"""Test that generated runner code includes input values."""runner_code = generate_runner_code(mock_pipe_single_output)assert '"document":' in runner_codeassert '"concept": "native.Document"' in runner_codeassert "DocumentContent(" in runner_code- def test_runner_code_includes_main_block(self, mock_pipe_single_output: MagicMock) -> None:+ def test_runner_code_includes_main_block(self, mock_pipe_single_output: MockType) -> None:"""Test that generated runner code includes main block."""runner_code = generate_runner_code(mock_pipe_single_output)assert 'if __name__ == "__main__":' in runner_codeassert "Pipelex.make()" in runner_codeassert "asyncio.run(run_test_pipe())" in runner_code- def test_runner_code_multiplicity_input(self, mock_pipe_list_output: MagicMock) -> None:+ def test_runner_code_multiplicity_input(self, mock_pipe_list_output: MockType) -> None:"""Test that generated runner code handles input multiplicity correctly."""runner_code = generate_runner_code(mock_pipe_list_output)# Should have list-wrapped content for multiplicity inputassert "[DocumentContent(" in runner_code- def test_runner_code_custom_class_import_format(self) -> None:+ def test_runner_code_custom_class_import_format(self, mocker: MockerFixture) -> None:"""Custom classes import from the single-module types projection (structures/structures.py)."""# Create a custom concept (non-native)custom_concept = ConceptFactory.make(@@ -276,7 +278,7 @@ class TestGenerateRunnerCode:)document_concept = ConceptFactory.make_native_concept(NativeConceptCode.DOCUMENT)- mock_pipe = MagicMock()+ mock_pipe = mocker.MagicMock()mock_pipe.code = "custom_pipe"mock_pipe.output = StuffSpec(concept=custom_concept)mock_pipe.inputs = InputStuffSpecs(root={"document": StuffSpec(concept=document_concept)})@@ -286,7 +288,7 @@ class TestGenerateRunnerCode:assert "from structures.structures import CustomOutput" in runner_codeassert "from .structures" not in runner_code- def test_runner_code_class_name_overrides_spell_emitted_names(self) -> None:+ def test_runner_code_class_name_overrides_spell_emitted_names(self, mocker: MockerFixture) -> None:"""The overrides map re-spells runtime-qualified class names to the emitted (bare) names."""custom_concept = ConceptFactory.make(domain_code="test_domain",@@ -296,7 +298,7 @@ class TestGenerateRunnerCode:)text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)- mock_pipe = MagicMock()+ mock_pipe = mocker.MagicMock()mock_pipe.code = "custom_pipe"mock_pipe.output = StuffSpec(concept=custom_concept)mock_pipe.inputs = InputStuffSpecs(root={"message": StuffSpec(concept=text_concept)})@@ -307,12 +309,12 @@ class TestGenerateRunnerCode:assert "pipe_output.main_stuff_as(content_type=CustomOutput)" in runner_codeassert "test_domain__CustomOutput" not in runner_code- def test_runner_code_anything_output_uses_any_return_type(self) -> None:+ def test_runner_code_anything_output_uses_any_return_type(self, mocker: MockerFixture) -> None:"""Test that Anything output concept uses Any return type and main_stuff instead of main_stuff_as."""anything_concept = ConceptFactory.make_native_concept(NativeConceptCode.ANYTHING)text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)- mock_pipe = MagicMock()+ mock_pipe = mocker.MagicMock()mock_pipe.code = "anything_output_pipe"mock_pipe.output = StuffSpec(concept=anything_concept)mock_pipe.inputs = InputStuffSpecs(root={"message": StuffSpec(concept=text_concept)})diff --git a/tests/unit/pipelex/cli/test_codegen_cli.py b/tests/unit/pipelex/cli/test_codegen_cli.pyindex 05dab7727..3eab8c950 100644--- a/tests/unit/pipelex/cli/test_codegen_cli.py+++ b/tests/unit/pipelex/cli/test_codegen_cli.py@@ -25,9 +25,7 @@ from pipelex.libraries.library_crate import LibraryCratefrom pipelex.libraries.pipe.exceptions import PipeLibraryErrorif TYPE_CHECKING:- from unittest.mock import MagicMock-- from pytest_mock import MockerFixture+ from pytest_mock import MockerFixture, MockTypeTYPES = "pipelex.cli.commands.codegen.types_cmd"INPUTS = "pipelex.cli.commands.codegen.inputs_cmd"@@ -36,7 +34,7 @@ INPUTS = "pipelex.cli.commands.codegen.inputs_cmd"class TestCodegenCli:"""The two Phase-1 codegen commands, with boot/teardown and the crate loader mocked out."""- def _neutralize_boot(self, mocker: MockerFixture, *, module: str) -> MagicMock:+ def _neutralize_boot(self, mocker: MockerFixture, *, module: str) -> MockType:boot = mocker.patch(f"{module}.make_pipelex_for_cli")mocker.patch(f"{module}.Pipelex.teardown_if_needed")mocker.patch(f"{module}.tag")diff --git a/tests/unit/pipelex/cli/test_init_credentials.py b/tests/unit/pipelex/cli/test_init_credentials.pyindex d15316a2b..8c97af22c 100644--- a/tests/unit/pipelex/cli/test_init_credentials.py+++ b/tests/unit/pipelex/cli/test_init_credentials.py@@ -15,9 +15,8 @@ from pipelex.cli.commands.init.credentials import (if TYPE_CHECKING:from pathlib import Path- from unittest.mock import MagicMock- from pytest_mock import MockerFixture+ from pytest_mock import MockerFixture, MockTypeclass TestInitCredentials:@@ -129,7 +128,7 @@ class TestInitCredentials:global_config_dir=tmp_path,)mock_prompt = mocker.patch("pipelex.cli.commands.init.credentials.Prompt.ask")- mock_console: MagicMock = mocker.MagicMock()+ mock_console: MockType = mocker.MagicMock()prompt_credentials(console=mock_console, backends_toml_path=backends_toml)# Should print "already set" message, no Prompt.ask callsmock_console.print.assert_called()@@ -151,7 +150,7 @@ class TestInitCredentials:"pipelex.cli.commands.init.credentials.Prompt.ask",return_value="sk-test-value",)- mock_console: MagicMock = mocker.MagicMock()+ mock_console: MockType = mocker.MagicMock()prompt_credentials(console=mock_console, backends_toml_path=backends_toml)# Verify .env was written@@ -181,7 +180,7 @@ class TestInitCredentials:"pipelex.cli.commands.init.credentials.Prompt.ask",return_value="",)- mock_console: MagicMock = mocker.MagicMock()+ mock_console: MockType = mocker.MagicMock()prompt_credentials(console=mock_console, backends_toml_path=backends_toml)# Verify .env was NOT written (no values entered)@@ -206,7 +205,7 @@ class TestInitCredentials:"pipelex.cli.commands.init.credentials.Prompt.ask",return_value="sk-new",)- mock_console: MagicMock = mocker.MagicMock()+ mock_console: MockType = mocker.MagicMock()prompt_credentials(console=mock_console, backends_toml_path=backends_toml)result = read_env_file(env_path)@@ -225,6 +224,6 @@ class TestInitCredentials:global_config_dir=tmp_path,)mock_prompt = mocker.patch("pipelex.cli.commands.init.credentials.Prompt.ask")- mock_console: MagicMock = mocker.MagicMock()+ mock_console: MockType = mocker.MagicMock()prompt_credentials(console=mock_console, backends_toml_path=backends_toml)mock_prompt.assert_not_called()diff --git a/tests/unit/pipelex/cli/test_resolve_exit_codes.py b/tests/unit/pipelex/cli/test_resolve_exit_codes.pyindex 1bcf30ab3..fef030fcc 100644--- a/tests/unit/pipelex/cli/test_resolve_exit_codes.py+++ b/tests/unit/pipelex/cli/test_resolve_exit_codes.py@@ -22,9 +22,7 @@ from pipelex.libraries.exceptions import LibraryLoadingErrorfrom pipelex.libraries.pipe.exceptions import PipeLibraryErrorif TYPE_CHECKING:- from unittest.mock import MagicMock-- from pytest_mock import MockerFixture+ from pytest_mock import MockerFixture, MockTypeMODULE = "pipelex.cli.commands.resolve_cmd"# The load/normalize/library-manager seams now live in the shared crate-loading helper, so the@@ -35,7 +33,7 @@ CRATE_LOADING = "pipelex.cli.commands.crate_loading"class TestResolveExitCodes:"""The resolve 0/1/2 exit-code policy, with boot/teardown mocked out."""- def _neutralize_boot(self, mocker: MockerFixture) -> MagicMock:+ def _neutralize_boot(self, mocker: MockerFixture) -> MockType:boot = mocker.patch(f"{MODULE}.make_pipelex_for_cli")mocker.patch(f"{MODULE}.Pipelex.teardown_if_needed")mocker.patch(f"{MODULE}.tag")diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.pyindex 4e8ff3fc4..6201c6a66 100644--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py@@ -1,11 +1,15 @@+import jsonimport socketimport pytest-from pytest_mock import MockerFixture+from pytest_mock import MockerFixture, MockTypefrom pipelex.base_exceptions import ErrorDomain, ErrorReportfrom pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction, UserActionKindfrom pipelex.cogt.inference.provider_name import ProviderName+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.memory.absence import AbsenceKind, AbsenceRecordfrom pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError@@ -20,6 +24,7 @@ from pipelex.pipe_run.delivery_assignment import ()from pipelex.pipe_run.delivery_executor import DeliveryExecutorfrom pipelex.pipe_run.exceptions import PipeJobError, StorageDeliveryError, WebhookDeliveryError+from pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.network.exceptions import SsrfBlockedError@@ -37,6 +42,14 @@ def _make_main_stuff() -> Stuff:)+def _make_output_mock(mocker: MockerFixture) -> MockType:+ """A PipeOutput stand-in with the usage fields defaulted to None, like a run with usage assembly off."""+ mock_output: MockType = mocker.MagicMock()+ mock_output.tokens_usages = None+ mock_output.usage_assembly_error = None+ return mock_output++@pytest.mark.asyncio(loop_scope="class")class TestDeliveryExecutor:async def test_execute_storage_only(self, mocker: MockerFixture) -> None:@@ -45,7 +58,7 @@ class TestDeliveryExecutor:mock_storage.public_url = mocker.Mock(return_value="file:///tmp/results/plr-123")mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = Nonemock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()@@ -69,6 +82,7 @@ class TestDeliveryExecutor:assert any("test-user/plr-123/main_stuff.json" in key for key in stored_keys)assert any("test-user/plr-123/main_stuff.md" in key for key in stored_keys)assert any("test-user/plr-123/main_stuff.html" in key for key in stored_keys)+ assert any("test-user/plr-123/tokens_usages.json" in key for key in stored_keys)async def test_execute_webhook_only(self, mocker: MockerFixture) -> None:mock_client = mocker.AsyncMock()@@ -163,7 +177,7 @@ class TestDeliveryExecutor:mock_storage.store = mocker.AsyncMock(side_effect=Exception("S3 down"))mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = Nonemock_output.working_memory.smart_dump.return_value = {}mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()@@ -185,7 +199,7 @@ class TestDeliveryExecutor:"""A completed run always resolves its declared output — a typed working memory with neithera main stuff nor a recorded absence is a contract violation, not an empty envelope."""- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = Nonemock_output.working_memory = WorkingMemory()mock_output.graph_spec = None@@ -195,7 +209,7 @@ class TestDeliveryExecutor:async def test_generate_result_files_raises_without_main_stuff_raw(self, mocker: MockerFixture) -> None:"""Same contract on the raw (cross-process) path: neither a raw main stuff nor a recorded absence fails loudly."""- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = {"root": {}, "aliases": {}}mock_output.graph_spec = None@@ -215,7 +229,7 @@ class TestDeliveryExecutor:producing_pipe="check_penalty_clause",),)- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = Nonemock_output.working_memory = working_memorymock_output.graph_spec = None@@ -236,7 +250,7 @@ class TestDeliveryExecutor:"""Same on the raw (cross-process) path: a recorded main absence in the raw ledger deliversthe absence artifact instead of the contract-violation error."""- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = {"root": {},"aliases": {},@@ -261,6 +275,70 @@ class TestDeliveryExecutor:assert "summarize_penalty" in md_textassert "main_stuff.html" in files+ async def test_generate_result_files_writes_usage_artifact_with_records(self, mocker: MockerFixture) -> None:+ """The tokens_usages.json artifact carries the run's usage records in the same+ per-record ``model_dump(mode="json")`` wire shape the /execute response uses, so a+ durable client can compute costs from the polled result files alone.+ """+ mock_output = _make_output_mock(mocker)+ mock_output.working_memory_raw = None+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()+ mock_output.graph_spec = None+ mock_output.tokens_usages = [+ LLMTokensUsage(+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="plr-usage"),+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ nb_tokens_by_category={TokenCategory.INPUT: 15, TokenCategory.OUTPUT: 4},+ unit_costs={CostCategory.INPUT: 3.0, CostCategory.OUTPUT: 15.0},+ )+ ]++ files = await DeliveryExecutor().generate_result_files(mock_output)++ assert files["tokens_usages.json"].content_type == "application/json"+ usage_doc = json.loads(files["tokens_usages.json"].data.decode("utf-8"))+ assert usage_doc["usage_assembly_error"] is None+ records = usage_doc["tokens_usages"]+ assert len(records) == 1+ record = records[0]+ assert record["model_type"] == "llm"+ assert record["inference_model_name"] == "test-model"+ assert record["inference_model_id"] == "test-model-id"+ assert record["nb_tokens_by_category"] == {"input": 15, "output": 4}+ assert record["unit_costs"] == {"input": 3.0, "output": 15.0}++ async def test_generate_result_files_usage_artifact_null_when_usage_off(self, mocker: MockerFixture) -> None:+ """A run with usage assembly off still writes the artifact with explicit nulls, so a+ durable client can tell "usage off for this run" (file present, null) from "run+ delivered before the artifact existed" (file absent).+ """+ mock_output = _make_output_mock(mocker)+ mock_output.working_memory_raw = None+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()+ mock_output.graph_spec = None++ files = await DeliveryExecutor().generate_result_files(mock_output)++ usage_doc = json.loads(files["tokens_usages.json"].data.decode("utf-8"))+ assert usage_doc == {"tokens_usages": None, "usage_assembly_error": None}++ async def test_generate_result_files_usage_artifact_carries_assembly_error(self, mocker: MockerFixture) -> None:+ """A failed usage assembly surfaces on the artifact instead of silently reading as "usage off"."""+ mock_output = _make_output_mock(mocker)+ mock_output.working_memory_raw = None+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()+ mock_output.graph_spec = None+ mock_output.usage_assembly_error = "usage event read failed"++ files = await DeliveryExecutor().generate_result_files(mock_output)++ usage_doc = json.loads(files["tokens_usages.json"].data.decode("utf-8"))+ assert usage_doc == {"tokens_usages": None, "usage_assembly_error": "usage event read failed"}+async def test_try_local_hydrate_stuff_returns_typed_for_builtin(self) -> None:from pipelex.core.stuffs.text_content import TextContent # noqa: PLC0415from pipelex.hub import get_class_registry # noqa: PLC0415@@ -329,7 +407,7 @@ class TestDeliveryExecutor:"</pre><script>..." would break out of the <pre> wrapper and executeas live HTML when the stored result file is fetched."""- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = {"root": {"main_stuff": {@@ -387,7 +465,7 @@ class TestDeliveryExecutor:),)- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = {"root": {"cv_pages": {@@ -527,7 +605,7 @@ class TestDeliveryExecutor:mock_storage.store = mocker.AsyncMock(return_value="pipelex-storage://test-key")mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)- mock_output = mocker.MagicMock()+ mock_output = _make_output_mock(mocker)mock_output.working_memory_raw = Nonemock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()diff --git a/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py b/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.pyindex 8d370a41c..1d93fd682 100644--- a/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py+++ b/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py@@ -23,9 +23,7 @@ from pipelex.system.pipelex_service.remote_config_cache import ()if TYPE_CHECKING:- from unittest.mock import MagicMock-- from pytest_mock import MockerFixture+ from pytest_mock import MockerFixture, MockTypedef _valid_remote_config_payload(extra: dict[str, object] | None = None) -> dict[str, object]:@@ -87,7 +85,7 @@ class TestRemoteConfigCache:def test_read_corrupted_json_returns_none_and_logs(self,isolated_cache_dir: Path,- mock_log: MagicMock,+ mock_log: MockType,) -> None:cache_path = isolated_cache_dir / "cache" / "remote_config.json"cache_path.parent.mkdir(parents=True, exist_ok=True)diff --git a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.pyindex f9a1a967f..842803aa2 100644--- a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py@@ -1,7 +1,6 @@from datetime import timedeltafrom pathlib import Pathfrom typing import Any-from unittest.mock import MagicMockimport pytestfrom pytest_mock import MockerFixture@@ -47,16 +46,16 @@ class TestGcpStorageProvider:from google.cloud import storage # type: ignore[import-untyped] # noqa: PLC0415# Create mock objects- mock_blob = MagicMock()+ mock_blob = mocker.MagicMock()mock_blob.exists.return_value = Truemock_blob.download_as_bytes.return_value = b""- mock_blob.upload_from_string = MagicMock()+ mock_blob.upload_from_string = mocker.MagicMock()mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"- mock_bucket = MagicMock()+ mock_bucket = mocker.MagicMock()mock_bucket.blob.return_value = mock_blob- mock_client = MagicMock()+ mock_client = mocker.MagicMock()mock_client.bucket.return_value = mock_bucket# Patch the Client.from_service_account_json method directlydiff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.pyindex 38c008d11..e227ec05d 100644--- a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py@@ -1,5 +1,4 @@from typing import Any-from unittest.mock import AsyncMock, MagicMockimport pytestfrom botocore.exceptions import BotoCoreError, ClientError, EndpointConnectionError, NoCredentialsError, ReadTimeoutError@@ -26,33 +25,33 @@ class TestS3StorageProvider:for test assertions."""# Create mock stream for response body- mock_stream = AsyncMock()- mock_stream.read = AsyncMock(return_value=b"")- mock_stream.__aenter__ = AsyncMock(return_value=mock_stream)- mock_stream.__aexit__ = AsyncMock(return_value=None)+ mock_stream = mocker.AsyncMock()+ mock_stream.read = mocker.AsyncMock(return_value=b"")+ mock_stream.__aenter__ = mocker.AsyncMock(return_value=mock_stream)+ mock_stream.__aexit__ = mocker.AsyncMock(return_value=None)# Create mock client- mock_client = AsyncMock()- mock_client.get_object = AsyncMock(return_value={"Body": mock_stream})- mock_client.put_object = AsyncMock(return_value={})- mock_client.generate_presigned_url = AsyncMock(+ mock_client = mocker.AsyncMock()+ mock_client.get_object = mocker.AsyncMock(return_value={"Body": mock_stream})+ mock_client.put_object = mocker.AsyncMock(return_value={})+ mock_client.generate_presigned_url = mocker.AsyncMock(return_value=f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/test?signature=abc123")# Create mock exceptions (modeled exceptions on the client)- mock_exceptions = MagicMock()+ mock_exceptions = mocker.MagicMock()mock_exceptions.NoSuchKey = type("NoSuchKey", (Exception,), {})mock_exceptions.NoSuchBucket = type("NoSuchBucket", (Exception,), {})mock_client.exceptions = mock_exceptions# Create async context manager for client- mock_client_context = AsyncMock()- mock_client_context.__aenter__ = AsyncMock(return_value=mock_client)- mock_client_context.__aexit__ = AsyncMock(return_value=None)+ mock_client_context = mocker.AsyncMock()+ mock_client_context.__aenter__ = mocker.AsyncMock(return_value=mock_client)+ mock_client_context.__aexit__ = mocker.AsyncMock(return_value=None)# Create mock session- mock_session = MagicMock()- mock_session.client = MagicMock(return_value=mock_client_context)+ mock_session = mocker.MagicMock()+ mock_session.client = mocker.MagicMock(return_value=mock_client_context)# Patch aioboto3.Sessionmocker.patch("aioboto3.Session", return_value=mock_session)@@ -118,12 +117,13 @@ class TestS3StorageProvider:async def test_load_returns_data(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that load() returns the data from S3."""expected_data = b"Test data \x00\x01\x02\xff"- mock_aioboto3["stream"].read = AsyncMock(return_value=expected_data)+ mock_aioboto3["stream"].read = mocker.AsyncMock(return_value=expected_data)uri = f"{PIPELEX_STORAGE_SCHEME}test/file.bin"loaded_data = await s3_provider_no_signed_urls.load(uri=uri)@@ -132,11 +132,12 @@ class TestS3StorageProvider:async def test_load_with_nonexistent_key_raises_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that loading a non-existent object raises StorageFileNotFoundError."""- mock_aioboto3["client"].get_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchKey("Key not found"))+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchKey("Key not found"))nonexistent_uri = f"{PIPELEX_STORAGE_SCHEME}nonexistent/file.bin"with pytest.raises(StorageFileNotFoundError) as exc_info:@@ -173,6 +174,7 @@ class TestS3StorageProvider:async def test_public_url_returns_presigned_url_when_signed_urls_enabled(self,+ mocker: MockerFixture,s3_provider_with_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:@@ -180,7 +182,7 @@ class TestS3StorageProvider:key = "presigned/test.bin"uri = f"{PIPELEX_STORAGE_SCHEME}{key}"expected_presigned = "https://test-bucket.s3.amazonaws.com/presigned/test.bin?X-Amz-Signature=xyz"- mock_aioboto3["client"].generate_presigned_url = AsyncMock(return_value=expected_presigned)+ mock_aioboto3["client"].generate_presigned_url = mocker.AsyncMock(return_value=expected_presigned)display = await s3_provider_with_signed_urls.public_url(uri=uri)@@ -193,6 +195,7 @@ class TestS3StorageProvider:async def test_public_url_handles_sync_presigned_url(self,+ mocker: MockerFixture,s3_provider_with_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:@@ -201,7 +204,7 @@ class TestS3StorageProvider:uri = f"{PIPELEX_STORAGE_SCHEME}{key}"expected_presigned = "https://test-bucket.s3.amazonaws.com/sync-presign/test.bin?X-Amz-Signature=sync"# Return a plain string (non-awaitable) to simulate sync behavior in some aioboto3 versions- mock_aioboto3["client"].generate_presigned_url = MagicMock(return_value=expected_presigned)+ mock_aioboto3["client"].generate_presigned_url = mocker.MagicMock(return_value=expected_presigned)display = await s3_provider_with_signed_urls.public_url(uri=uri)@@ -254,11 +257,12 @@ class TestS3StorageProvider:async def test_store_bucket_not_found_raises_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that storing to a non-existent bucket raises StorageS3Error."""- mock_aioboto3["client"].put_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))with pytest.raises(StorageS3Error) as exc_info:await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")@@ -267,11 +271,12 @@ class TestS3StorageProvider:async def test_load_bucket_not_found_raises_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that loading from a non-existent bucket raises StorageS3Error."""- mock_aioboto3["client"].get_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"with pytest.raises(StorageS3Error) as exc_info:@@ -281,12 +286,13 @@ class TestS3StorageProvider:async def test_load_client_error_with_no_such_key_code_raises_file_not_found(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that ClientError with NoSuchKey code raises StorageFileNotFoundError."""error_response: Any = {"Error": {"Code": "NoSuchKey", "Message": "The specified key does not exist."}}- mock_aioboto3["client"].get_object = AsyncMock(side_effect=ClientError(error_response, "GetObject"))+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=ClientError(error_response, "GetObject"))uri = f"{PIPELEX_STORAGE_SCHEME}missing/file.bin"with pytest.raises(StorageFileNotFoundError) as exc_info:@@ -296,12 +302,13 @@ class TestS3StorageProvider:async def test_load_client_error_with_access_denied_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that ClientError with AccessDenied code raises StorageS3Error."""error_response: Any = {"Error": {"Code": "AccessDenied", "Message": "Access Denied"}}- mock_aioboto3["client"].get_object = AsyncMock(side_effect=ClientError(error_response, "GetObject"))+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=ClientError(error_response, "GetObject"))uri = f"{PIPELEX_STORAGE_SCHEME}forbidden/file.bin"with pytest.raises(StorageS3Error) as exc_info:@@ -311,12 +318,13 @@ class TestS3StorageProvider:async def test_store_client_error_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that ClientError during store raises StorageS3Error."""error_response: Any = {"Error": {"Code": "InvalidAccessKeyId", "Message": "Invalid access key"}}- mock_aioboto3["client"].put_object = AsyncMock(side_effect=ClientError(error_response, "PutObject"))+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=ClientError(error_response, "PutObject"))with pytest.raises(StorageS3Error) as exc_info:await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")@@ -325,11 +333,12 @@ class TestS3StorageProvider:async def test_load_no_credentials_error_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that NoCredentialsError (a BotoCoreError subclass) raises StorageS3Error."""- mock_aioboto3["client"].get_object = AsyncMock(side_effect=NoCredentialsError())+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=NoCredentialsError())uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"with pytest.raises(StorageS3Error) as exc_info:@@ -340,11 +349,12 @@ class TestS3StorageProvider:async def test_store_no_credentials_error_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that NoCredentialsError (a BotoCoreError subclass) during store raises StorageS3Error."""- mock_aioboto3["client"].put_object = AsyncMock(side_effect=NoCredentialsError())+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=NoCredentialsError())with pytest.raises(StorageS3Error) as exc_info:await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")@@ -354,11 +364,12 @@ class TestS3StorageProvider:async def test_load_endpoint_connection_error_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that EndpointConnectionError (a BotoCoreError subclass) raises StorageS3Error."""- mock_aioboto3["client"].get_object = AsyncMock(side_effect=EndpointConnectionError(endpoint_url="https://s3.amazonaws.com"))+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=EndpointConnectionError(endpoint_url="https://s3.amazonaws.com"))uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"with pytest.raises(StorageS3Error) as exc_info:@@ -369,11 +380,12 @@ class TestS3StorageProvider:async def test_load_read_timeout_error_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that ReadTimeoutError — the canonical transient-network BotoCoreError that used to escape — raises StorageS3Error."""- mock_aioboto3["client"].get_object = AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))uri = f"{PIPELEX_STORAGE_SCHEME}slow/file.bin"with pytest.raises(StorageS3Error) as exc_info:@@ -385,11 +397,12 @@ class TestS3StorageProvider:async def test_store_read_timeout_error_raises_s3_error(self,+ mocker: MockerFixture,s3_provider_no_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that ReadTimeoutError during store (slow upload, transient blip) raises StorageS3Error."""- mock_aioboto3["client"].put_object = AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))with pytest.raises(StorageS3Error) as exc_info:await s3_provider_no_signed_urls.store(data=b"test", key="slow/upload.bin")@@ -400,13 +413,14 @@ class TestS3StorageProvider:async def test_public_url_falls_back_to_public_url_on_botocore_error(self,+ mocker: MockerFixture,s3_provider_with_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:"""Test that public_url() falls back to public URL on a transport BotoCoreError, not just ClientError."""key = "fallback/timeout.bin"uri = f"{PIPELEX_STORAGE_SCHEME}{key}"- mock_aioboto3["client"].generate_presigned_url = MagicMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))+ mock_aioboto3["client"].generate_presigned_url = mocker.MagicMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))display = await s3_provider_with_signed_urls.public_url(uri=uri)@@ -415,6 +429,7 @@ class TestS3StorageProvider:async def test_public_url_falls_back_to_public_url_on_client_error(self,+ mocker: MockerFixture,s3_provider_with_signed_urls: S3StorageProvider,mock_aioboto3: dict[str, Any],) -> None:@@ -422,7 +437,7 @@ class TestS3StorageProvider:key = "fallback/test.bin"uri = f"{PIPELEX_STORAGE_SCHEME}{key}"error_response: Any = {"Error": {"Code": "SignatureDoesNotMatch", "Message": "Signature error"}}- mock_aioboto3["client"].generate_presigned_url = MagicMock(side_effect=ClientError(error_response, "GeneratePresignedUrl"))+ mock_aioboto3["client"].generate_presigned_url = mocker.MagicMock(side_effect=ClientError(error_response, "GeneratePresignedUrl"))display = await s3_provider_with_signed_urls.public_url(uri=uri)diff --git a/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py b/tests/unit/pipelex/tools/test_jinja2_with_images_filter.pyindex 9f39f159b..1b7751b91 100644--- a/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py+++ b/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py@@ -5,10 +5,10 @@ Note: Core functionality tests using real Stuff classes are in integration tests"""from typing import Any-from unittest.mock import MagicMockimport pytestfrom jinja2.runtime import Context, Undefined+from pytest_mock import MockerFixture, MockTypefrom pipelex.cogt.templating.text_format import TextFormatfrom pipelex.tools.jinja2.exceptions import Jinja2ContextError@@ -23,76 +23,76 @@ from pipelex.tools.jinja2.jinja2_with_images_filter import (class TestWithImagesFilterValidation:"""Tests for with_images filter error handling and edge cases."""- def _make_context(self, registry: ImageRegistry | None = None, text_format: TextFormat = TextFormat.PLAIN) -> Context:+ def _make_context(self, mocker: MockerFixture, registry: ImageRegistry | None = None, text_format: TextFormat = TextFormat.PLAIN) -> Context:"""Create a mock Jinja2 context."""context_dict: dict[str, Any] = {}if registry is not None:context_dict[Jinja2ContextKey.IMAGE_REGISTRY] = registrycontext_dict[Jinja2ContextKey.TEXT_FORMAT] = text_format- mock_env = MagicMock()+ mock_env = mocker.MagicMock()mock_env.undefined = Undefined- context = MagicMock(spec=Context)+ context: MockType = mocker.MagicMock(spec=Context)context.get = lambda key, default=None: context_dict.get(key, default) # pyright: ignore[reportUnknownLambdaType, reportUnknownArgumentType]context.environment = mock_envreturn context- def test_with_images_raises_on_undefined(self) -> None:+ def test_with_images_raises_on_undefined(self, mocker: MockerFixture) -> None:"""Test with_images raises error for undefined value."""registry = ImageRegistry()- context = self._make_context(registry=registry)+ context = self._make_context(mocker, registry=registry)with pytest.raises(Jinja2ContextError, match="undefined"):with_images(context, value=Undefined())- def test_with_images_raises_on_wrong_registry_type(self) -> None:+ def test_with_images_raises_on_wrong_registry_type(self, mocker: MockerFixture) -> None:"""Test with_images raises if registry is wrong type."""context_dict: dict[str, Any] = {Jinja2ContextKey.IMAGE_REGISTRY: "not_a_registry",Jinja2ContextKey.TEXT_FORMAT: TextFormat.PLAIN,}- context = MagicMock(spec=Context)+ context = mocker.MagicMock(spec=Context)context.get = lambda key, default=None: context_dict.get(key, default) # pyright: ignore[reportUnknownLambdaType, reportUnknownArgumentType]# Pass a list (which is accepted by the filter) to trigger the registry type checkwith pytest.raises(Jinja2ContextError, match="Expected ImageRegistry"):with_images(context, value=[])- def test_with_images_raises_on_string_input(self) -> None:+ def test_with_images_raises_on_string_input(self, mocker: MockerFixture) -> None:"""Test with_images raises error when receiving a plain string.This catches the common mistake of chaining filters in wrong order,e.g., {{ pages | tag | with_images }} where tag converts to string first."""registry = ImageRegistry()- context = self._make_context(registry=registry)+ context = self._make_context(mocker, registry=registry)with pytest.raises(Jinja2ContextError, match="does not implement the ImageRenderable protocol"):with_images(context, value="This is a plain string")- def test_with_images_raises_on_number_input(self) -> None:+ def test_with_images_raises_on_number_input(self, mocker: MockerFixture) -> None:"""Test with_images raises error when receiving a number."""registry = ImageRegistry()- context = self._make_context(registry=registry)+ context = self._make_context(mocker, registry=registry)with pytest.raises(Jinja2ContextError, match="does not implement the ImageRenderable protocol"):with_images(context, value=42)- def test_with_images_raises_on_dict_input(self) -> None:+ def test_with_images_raises_on_dict_input(self, mocker: MockerFixture) -> None:"""Test with_images raises error when receiving a plain dict."""registry = ImageRegistry()- context = self._make_context(registry=registry)+ context = self._make_context(mocker, registry=registry)with pytest.raises(Jinja2ContextError, match="does not implement the ImageRenderable protocol"):with_images(context, value={"key": "value"})- def test_with_images_accepts_empty_list(self) -> None:+ def test_with_images_accepts_empty_list(self, mocker: MockerFixture) -> None:"""Test with_images handles empty list gracefully."""registry = ImageRegistry()- context = self._make_context(registry=registry)+ context = self._make_context(mocker, registry=registry)result = with_images(context, value=[])diff --git a/wip/durable-results-tokens-usage-exposure.md b/wip/durable-results-tokens-usage-exposure.mdnew file mode 100644index 000000000..d719be5ed--- /dev/null+++ b/wip/durable-results-tokens-usage-exposure.md@@ -0,0 +1,46 @@+# Durable-run results drop `tokens_usages` — expose usage on the start+poll path++**Status:** Feature ask from the Fenix workspace (delivery-plan task A6, cost-measurement+exploration, 2026-07-18). Live-verified against hosted api-dev. Full findings:+`~/repos/Fenix/project/designs/a6-cost-measurement.md`.++## The gap++The engine assembles per-call usage onto `PipeOutput.tokens_usages` (token counts by category ++`unit_costs` $/1M + model id, for LLM **and** img-gen/extract/search), and the sync path returns+it: `POST /v1/execute` → `pipe_output.tokens_usages` — confirmed live (probe run `d020718b…`:+claude-4.6-sonnet, `{input: 15, output: 4}`, `unit_costs {input: 3.0, output: 15.0}` → $0.000105+computable client-side).++But a durable client (`start` + `GET /v1/runs/{id}/results`) never sees it:++- **pipelex** — `pipe_run/delivery_executor.py` `generate_result_files(pipe_output)` receives the+ full `PipeOutput` (so `tokens_usages` + `usage_assembly_error` are in hand) but persists only+ `graphspec.json`, `main_stuff.json` (+ md/html/viewer renders), `working_memory.json`.+- **pipelex-platform** — `src/pipelex_platform/routers/v1/runs.py`+ `_fetch_run_result_artifacts()` reads exactly those three files; `RunResultsResponse` has no+ usage field. Per-node `metrics: {}` in the graphspec sits empty; no+ `/runs/{id}/usage`-style route exists (probed, 404).++Sync `/execute` is not a workaround for durable clients: the ~30s gateway ceiling rules it out+for img-gen and parallel-heavy pipes — exactly the runs whose cost matters most.++## Proposed change (smallest that closes it)++1. **pipelex** `delivery_executor.generate_result_files`: also write `tokens_usages.json` —+ `{"tokens_usages": [...], "usage_assembly_error": null}`, same serialization the `/execute`+ response already uses.+2. **pipelex-platform** `runs.py`: fetch the fourth file in `_fetch_run_result_artifacts()`+ (tolerating its absence for runs delivered before the change) and add+ `tokens_usages` / `usage_assembly_error` to `RunResultsResponse`.++No SDK release needed to unblock: `pipelex-sdk` 0.4.0's `RunResults` is `extra="allow"`, so the+new fields ride along via `model_extra`; typing them in the SDK can follow.++## Notes++- Keep it raw records, not a computed $ total — consumers (e.g. `fenix-pipelex`) do the+ multiply-and-sum deterministically, and the existing `CostRegistry.build_cost_summary` shape is+ available if a summary is wanted later.+- Known caveat carried over from the engine: models with no `costs` in their backend TOML price+ at 0 — consumers should flag zero-cost records rather than trust them.