Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feat(delivery): persist token usage as a tokens_usages.json result artifact (#1058)

Louis Choquel27864c8470fe
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2cf0d010b..c74c6eedb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
## [Unreleased]
+### Added
+
+- **Durable runs now persist token usage: the delivery executor writes a `tokens_usages.json` result artifact** (`{"tokens_usages": [...], "usage_assembly_error": null}`) alongside `working_memory.json`, the `main_stuff.*` renders, and the graph outputs. The records use the same per-record `model_dump(mode="json")` wire shape the `/execute` response carries on `pipe_output.tokens_usages` (token counts by category, `unit_costs` in $/1M, model id — for LLM, img-gen, extract, and search calls), so a client polling a durable run's result files can compute costs the same way a sync `/execute` caller does. The artifact is written unconditionally: a run with usage assembly off yields explicit nulls, distinguishable from a pre-artifact run (file absent).
+
### Fixed
- **Validation errors stay domain-qualified: every raise site that names a `pipe_code` now also carries `domain_code`.** The presentation chain (VS Code extension + mthds-ui) identifies pipes by full pipe ref (`domain_code.pipe_code`); a handful of `PipeValidationError` raise sites (PipeLLM static input checks, PipeStructure input-mismatch, the pipe sorter's circular-dependency error) omitted `domain_code`, degrading node decorations and click-to-navigate for those errors in multi-domain bundles. The sorter gains a `domain_code` parameter threaded from the bundle spec's domain.
diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.md
index a8716cda1..f8d12e031 100644
--- a/docs/under-the-hood/execution-graph-tracing.md
+++ b/docs/under-the-hood/execution-graph-tracing.md
@@ -262,6 +262,8 @@ graph_spec = manager.close_tracer(pipeline_run_id)
Trace events travel through an `EventLogProtocol` backend (`pipelex/tracing/`): the tracer emits events into it during the run (write side, wired in `pipeline_run_setup`), and `assemble_tracing` reads them back after the run to build the `GraphSpec` and usage aggregates (read side, triggered from `PipeRun.run`). Both sides normally build their backend instance independently from `tracing_config` via `make_event_log` — NDJSON files or DynamoDB bridge the two instances through external storage.
+The assembled usage rides back on `pipe_output.tokens_usages` (with any assembly failure on `usage_assembly_error`), which the sync `/execute` response returns directly. For delivery-enabled runs (a storage target set), the delivery executor (`pipelex/pipe_run/delivery_executor.py`) also persists it as a `tokens_usages.json` result artifact — `{"tokens_usages": [...], "usage_assembly_error": null}` — next to `working_memory.json`, the `main_stuff.*` renders, and the graph outputs, so a durable client polling result files gets the same usage records a sync caller does. The artifact is written unconditionally: explicit nulls mean usage assembly was off for that run, while an absent file means the run was delivered before the artifact existed.
+
For fully in-process runs, `pipelex.hub.scoped_event_log` pins one shared instance for both sides instead:
```python
diff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.py
index 20173627b..74c9f88a8 100644
--- a/pipelex/pipe_run/delivery_executor.py
+++ b/pipelex/pipe_run/delivery_executor.py
@@ -80,7 +80,7 @@ class DeliveryExecutor:
"""Generate the full set of result files from a PipeOutput.
Produces: working_memory.json, main_stuff (json/md/html/viewer),
- graph outputs (mermaidflow, reactflow, graphspec).
+ tokens_usages.json, graph outputs (mermaidflow, reactflow, graphspec).
Supports two `pipe_output` shapes:
- Typed: `working_memory` populated (in-process / same-worker path).
@@ -133,12 +133,33 @@ class DeliveryExecutor:
else:
await self._generate_main_stuff_files(main_resolved, files=files)
+ files["tokens_usages.json"] = self._generate_usage_file(pipe_output)
+
graph_spec = pipe_output.graph_spec
if graph_spec:
await self._generate_graph_files(graph_spec, files=files)
return files
+ @classmethod
+ def _generate_usage_file(cls, pipe_output: PipeOutput) -> ResultFile:
+ """Serialize the run's assembled usage onto the tokens_usages.json artifact.
+
+ Written unconditionally, so a durable client polling the result files can tell
+ "usage assembly was off for this run" (file present, ``tokens_usages`` null) from
+ "run delivered before the artifact existed" (file absent). The records use the same
+ per-usage ``model_dump(mode="json")`` wire shape the ``/execute`` response carries
+ on ``pipe_output.tokens_usages``.
+ """
+ tokens_usages_dump: list[dict[str, Any]] | None = None
+ if pipe_output.tokens_usages is not None:
+ tokens_usages_dump = [tokens_usage.model_dump(mode="json") for tokens_usage in pipe_output.tokens_usages]
+ usage_doc: dict[str, Any] = {
+ "tokens_usages": tokens_usages_dump,
+ "usage_assembly_error": pipe_output.usage_assembly_error,
+ }
+ return ResultFile(data=clean_json_dumps(usage_doc, indent=2).encode("utf-8"), content_type="application/json")
+
@classmethod
def _get_raw_main_stuff_dict(cls, *, working_memory_raw: dict[str, Any]) -> dict[str, Any] | None:
"""Extract the main stuff dict from a raw working_memory, following aliases."""
diff --git a/subject_grants.toml b/subject_grants.toml
index 64c57444b..15eeeb747 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -4324,6 +4324,10 @@ rationale = "Verb-object: renders files from the main stuff; options stay keywor
param = "raw_main_stuff"
rationale = "Verb-object: renders files from the raw main stuff; options stay keyword."
+["pipelex/pipe_run/delivery_executor.py::DeliveryExecutor._generate_usage_file"]
+param = "pipe_output"
+rationale = "Verb-object helper mirroring its sibling generators: serialize this pipe_output's usage into the tokens_usages.json artifact."
+
["pipelex/pipe_run/delivery_executor.py::DeliveryExecutor._store_results"]
param = "pipe_output"
rationale = "Verb-object: stores the pipe output; options stay keyword."
diff --git a/tests/integration/pipelex/tools/storage/conftest.py b/tests/integration/pipelex/tools/storage/conftest.py
index 7adb64a06..187869c81 100644
--- a/tests/integration/pipelex/tools/storage/conftest.py
+++ b/tests/integration/pipelex/tools/storage/conftest.py
@@ -1,9 +1,8 @@
from pathlib import Path
from typing import Any, Literal, cast
-from unittest.mock import AsyncMock, MagicMock
import pytest
-from pytest_mock import MockerFixture
+from pytest_mock import MockerFixture, MockType
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
from pipelex.config import get_config
@@ -39,16 +38,16 @@ def s3_mock(mocker: MockerFixture) -> dict[str, Any]:
# Storage for mock data
mock_data_storage: dict[str, bytes] = {}
- def create_mock_stream(key: str) -> AsyncMock:
+ def create_mock_stream(key: str) -> MockType:
"""Create a mock async stream for response body."""
- stream = AsyncMock()
- stream.read = AsyncMock(side_effect=lambda: mock_data_storage.get(key, b""))
- stream.__aenter__ = AsyncMock(return_value=stream)
- stream.__aexit__ = AsyncMock(return_value=None)
+ stream: MockType = mocker.AsyncMock()
+ stream.read = mocker.AsyncMock(side_effect=lambda: mock_data_storage.get(key, b""))
+ stream.__aenter__ = mocker.AsyncMock(return_value=stream)
+ stream.__aexit__ = mocker.AsyncMock(return_value=None)
return stream
# Create mock exceptions
- mock_exceptions = MagicMock()
+ mock_exceptions = mocker.MagicMock()
mock_exceptions.NoSuchKey = type("NoSuchKey", (Exception,), {})
mock_exceptions.NoSuchBucket = type("NoSuchBucket", (Exception,), {})
mock_exceptions.ClientError = type("ClientError", (Exception,), {})
@@ -72,20 +71,20 @@ def s3_mock(mocker: MockerFixture) -> dict[str, Any]:
return f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/{key}?X-Amz-Signature=mock"
# Create mock client with sync functions wrapped in AsyncMock
- mock_client = AsyncMock()
- mock_client.get_object = AsyncMock(side_effect=mock_get_object)
- mock_client.put_object = AsyncMock(side_effect=mock_put_object)
- mock_client.generate_presigned_url = AsyncMock(side_effect=mock_generate_presigned_url)
+ mock_client = mocker.AsyncMock()
+ mock_client.get_object = mocker.AsyncMock(side_effect=mock_get_object)
+ mock_client.put_object = mocker.AsyncMock(side_effect=mock_put_object)
+ mock_client.generate_presigned_url = mocker.AsyncMock(side_effect=mock_generate_presigned_url)
mock_client.exceptions = mock_exceptions
# Create async context manager for client
- mock_client_context = AsyncMock()
- mock_client_context.__aenter__ = AsyncMock(return_value=mock_client)
- mock_client_context.__aexit__ = AsyncMock(return_value=None)
+ mock_client_context = mocker.AsyncMock()
+ mock_client_context.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client_context.__aexit__ = mocker.AsyncMock(return_value=None)
# Create mock session
- mock_session = MagicMock()
- mock_session.client = MagicMock(return_value=mock_client_context)
+ mock_session = mocker.MagicMock()
+ mock_session.client = mocker.MagicMock(return_value=mock_client_context)
# Patch aioboto3.Session
mocker.patch("aioboto3.Session", return_value=mock_session)
@@ -114,8 +113,8 @@ def gcp_mock(tmp_path: Path, mocker: MockerFixture) -> dict[str, Any]:
mock_data_storage: dict[str, bytes] = {}
# Create mock blob that behaves like a real blob
- def create_mock_blob(key: str) -> MagicMock:
- blob = MagicMock()
+ def create_mock_blob(key: str) -> MockType:
+ blob: MockType = mocker.MagicMock()
blob.exists.side_effect = lambda: key in mock_data_storage
blob.download_as_bytes.side_effect = lambda: mock_data_storage[key]
blob.upload_from_string.side_effect = lambda data, content_type=None: mock_data_storage.__setitem__(key, data) # noqa: ARG005 # pyright: ignore[reportUnknownLambdaType,reportUnknownArgumentType]
@@ -123,11 +122,11 @@ def gcp_mock(tmp_path: Path, mocker: MockerFixture) -> dict[str, Any]:
return blob
# Create mock bucket
- mock_bucket = MagicMock()
+ mock_bucket = mocker.MagicMock()
mock_bucket.blob.side_effect = create_mock_blob
# Create mock client
- mock_client = MagicMock()
+ mock_client = mocker.MagicMock()
mock_client.bucket.return_value = mock_bucket
# Patch the Client.from_service_account_json method directly
diff --git a/tests/unit/pipelex/builder/test_runner_generator.py b/tests/unit/pipelex/builder/test_runner_generator.py
index 17d1fa5fa..98993b852 100644
--- a/tests/unit/pipelex/builder/test_runner_generator.py
+++ b/tests/unit/pipelex/builder/test_runner_generator.py
@@ -3,8 +3,7 @@
from __future__ import annotations
import json
-from typing import Any, cast
-from unittest.mock import MagicMock
+from typing import TYPE_CHECKING, Any, cast
import pytest
@@ -15,6 +14,9 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture, MockType
+
class TestConceptGenerateInputRepresentationJson:
"""Test Concept.render_concept_representation for JSON format."""
@@ -197,75 +199,75 @@ class TestGenerateRunnerCode:
"""Test generate_runner_code function."""
@pytest.fixture
- def mock_pipe_single_output(self) -> MagicMock:
+ def mock_pipe_single_output(self, mocker: MockerFixture) -> MockType:
"""Create a mock pipe with a single output."""
text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)
document_concept = ConceptFactory.make_native_concept(NativeConceptCode.DOCUMENT)
- mock_pipe = MagicMock()
+ mock_pipe: MockType = mocker.MagicMock()
mock_pipe.code = "test_pipe"
mock_pipe.output = StuffSpec(concept=text_concept)
mock_pipe.inputs = InputStuffSpecs(root={"document": StuffSpec(concept=document_concept)})
return mock_pipe
@pytest.fixture
- def mock_pipe_list_output(self) -> MagicMock:
+ def mock_pipe_list_output(self, mocker: MockerFixture) -> MockType:
"""Create a mock pipe with a list output."""
text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)
document_concept = ConceptFactory.make_native_concept(NativeConceptCode.DOCUMENT)
- mock_pipe = MagicMock()
+ mock_pipe: MockType = mocker.MagicMock()
mock_pipe.code = "test_pipe_list"
mock_pipe.output = StuffSpec(concept=text_concept)
mock_pipe.inputs = InputStuffSpecs(root={"documents": StuffSpec(concept=document_concept, multiplicity=True)})
return mock_pipe
- def test_runner_code_includes_imports(self, mock_pipe_single_output: MagicMock) -> None:
+ def test_runner_code_includes_imports(self, mock_pipe_single_output: MockType) -> None:
"""Test that generated runner code includes necessary imports."""
runner_code = generate_runner_code(mock_pipe_single_output)
assert "import asyncio" in runner_code
assert "from pipelex.pipelex import Pipelex" in runner_code
assert "from pipelex.pipeline.runner import PipelexMTHDSProtocol" in runner_code
- def test_runner_code_includes_structure_imports(self, mock_pipe_single_output: MagicMock) -> None:
+ def test_runner_code_includes_structure_imports(self, mock_pipe_single_output: MockType) -> None:
"""Test that generated runner code includes structure class imports."""
runner_code = generate_runner_code(mock_pipe_single_output)
assert "from pipelex.core.stuffs.document_content import DocumentContent" in runner_code
assert "from pipelex.core.stuffs.text_content import TextContent" in runner_code
- def test_runner_code_single_output_return_type(self, mock_pipe_single_output: MagicMock) -> None:
+ def test_runner_code_single_output_return_type(self, mock_pipe_single_output: MockType) -> None:
"""Test that generated runner code has correct return type for single output."""
runner_code = generate_runner_code(mock_pipe_single_output, output_multiplicity=False)
assert "async def run_test_pipe() -> TextContent:" in runner_code
assert "pipe_output.main_stuff_as(content_type=TextContent)" in runner_code
- def test_runner_code_list_output_return_type(self, mock_pipe_list_output: MagicMock) -> None:
+ def test_runner_code_list_output_return_type(self, mock_pipe_list_output: MockType) -> None:
"""Test that generated runner code has correct return type for list output."""
runner_code = generate_runner_code(mock_pipe_list_output, output_multiplicity=True)
assert "async def run_test_pipe_list() -> list[TextContent]:" in runner_code
assert "pipe_output.main_stuff_as_items(item_type=TextContent)" in runner_code
- def test_runner_code_includes_inputs(self, mock_pipe_single_output: MagicMock) -> None:
+ def test_runner_code_includes_inputs(self, mock_pipe_single_output: MockType) -> None:
"""Test that generated runner code includes input values."""
runner_code = generate_runner_code(mock_pipe_single_output)
assert '"document":' in runner_code
assert '"concept": "native.Document"' in runner_code
assert "DocumentContent(" in runner_code
- def test_runner_code_includes_main_block(self, mock_pipe_single_output: MagicMock) -> None:
+ def test_runner_code_includes_main_block(self, mock_pipe_single_output: MockType) -> None:
"""Test that generated runner code includes main block."""
runner_code = generate_runner_code(mock_pipe_single_output)
assert 'if __name__ == "__main__":' in runner_code
assert "Pipelex.make()" in runner_code
assert "asyncio.run(run_test_pipe())" in runner_code
- def test_runner_code_multiplicity_input(self, mock_pipe_list_output: MagicMock) -> None:
+ def test_runner_code_multiplicity_input(self, mock_pipe_list_output: MockType) -> None:
"""Test that generated runner code handles input multiplicity correctly."""
runner_code = generate_runner_code(mock_pipe_list_output)
# Should have list-wrapped content for multiplicity input
assert "[DocumentContent(" in runner_code
- def test_runner_code_custom_class_import_format(self) -> None:
+ def test_runner_code_custom_class_import_format(self, mocker: MockerFixture) -> None:
"""Custom classes import from the single-module types projection (structures/structures.py)."""
# Create a custom concept (non-native)
custom_concept = ConceptFactory.make(
@@ -276,7 +278,7 @@ class TestGenerateRunnerCode:
)
document_concept = ConceptFactory.make_native_concept(NativeConceptCode.DOCUMENT)
- mock_pipe = MagicMock()
+ mock_pipe = mocker.MagicMock()
mock_pipe.code = "custom_pipe"
mock_pipe.output = StuffSpec(concept=custom_concept)
mock_pipe.inputs = InputStuffSpecs(root={"document": StuffSpec(concept=document_concept)})
@@ -286,7 +288,7 @@ class TestGenerateRunnerCode:
assert "from structures.structures import CustomOutput" in runner_code
assert "from .structures" not in runner_code
- def test_runner_code_class_name_overrides_spell_emitted_names(self) -> None:
+ def test_runner_code_class_name_overrides_spell_emitted_names(self, mocker: MockerFixture) -> None:
"""The overrides map re-spells runtime-qualified class names to the emitted (bare) names."""
custom_concept = ConceptFactory.make(
domain_code="test_domain",
@@ -296,7 +298,7 @@ class TestGenerateRunnerCode:
)
text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)
- mock_pipe = MagicMock()
+ mock_pipe = mocker.MagicMock()
mock_pipe.code = "custom_pipe"
mock_pipe.output = StuffSpec(concept=custom_concept)
mock_pipe.inputs = InputStuffSpecs(root={"message": StuffSpec(concept=text_concept)})
@@ -307,12 +309,12 @@ class TestGenerateRunnerCode:
assert "pipe_output.main_stuff_as(content_type=CustomOutput)" in runner_code
assert "test_domain__CustomOutput" not in runner_code
- def test_runner_code_anything_output_uses_any_return_type(self) -> None:
+ def test_runner_code_anything_output_uses_any_return_type(self, mocker: MockerFixture) -> None:
"""Test that Anything output concept uses Any return type and main_stuff instead of main_stuff_as."""
anything_concept = ConceptFactory.make_native_concept(NativeConceptCode.ANYTHING)
text_concept = ConceptFactory.make_native_concept(NativeConceptCode.TEXT)
- mock_pipe = MagicMock()
+ mock_pipe = mocker.MagicMock()
mock_pipe.code = "anything_output_pipe"
mock_pipe.output = StuffSpec(concept=anything_concept)
mock_pipe.inputs = InputStuffSpecs(root={"message": StuffSpec(concept=text_concept)})
diff --git a/tests/unit/pipelex/cli/test_codegen_cli.py b/tests/unit/pipelex/cli/test_codegen_cli.py
index 05dab7727..3eab8c950 100644
--- a/tests/unit/pipelex/cli/test_codegen_cli.py
+++ b/tests/unit/pipelex/cli/test_codegen_cli.py
@@ -25,9 +25,7 @@ from pipelex.libraries.library_crate import LibraryCrate
from pipelex.libraries.pipe.exceptions import PipeLibraryError
if TYPE_CHECKING:
- from unittest.mock import MagicMock
-
- from pytest_mock import MockerFixture
+ from pytest_mock import MockerFixture, MockType
TYPES = "pipelex.cli.commands.codegen.types_cmd"
INPUTS = "pipelex.cli.commands.codegen.inputs_cmd"
@@ -36,7 +34,7 @@ INPUTS = "pipelex.cli.commands.codegen.inputs_cmd"
class TestCodegenCli:
"""The two Phase-1 codegen commands, with boot/teardown and the crate loader mocked out."""
- def _neutralize_boot(self, mocker: MockerFixture, *, module: str) -> MagicMock:
+ def _neutralize_boot(self, mocker: MockerFixture, *, module: str) -> MockType:
boot = mocker.patch(f"{module}.make_pipelex_for_cli")
mocker.patch(f"{module}.Pipelex.teardown_if_needed")
mocker.patch(f"{module}.tag")
diff --git a/tests/unit/pipelex/cli/test_init_credentials.py b/tests/unit/pipelex/cli/test_init_credentials.py
index d15316a2b..8c97af22c 100644
--- a/tests/unit/pipelex/cli/test_init_credentials.py
+++ b/tests/unit/pipelex/cli/test_init_credentials.py
@@ -15,9 +15,8 @@ from pipelex.cli.commands.init.credentials import (
if TYPE_CHECKING:
from pathlib import Path
- from unittest.mock import MagicMock
- from pytest_mock import MockerFixture
+ from pytest_mock import MockerFixture, MockType
class TestInitCredentials:
@@ -129,7 +128,7 @@ class TestInitCredentials:
global_config_dir=tmp_path,
)
mock_prompt = mocker.patch("pipelex.cli.commands.init.credentials.Prompt.ask")
- mock_console: MagicMock = mocker.MagicMock()
+ mock_console: MockType = mocker.MagicMock()
prompt_credentials(console=mock_console, backends_toml_path=backends_toml)
# Should print "already set" message, no Prompt.ask calls
mock_console.print.assert_called()
@@ -151,7 +150,7 @@ class TestInitCredentials:
"pipelex.cli.commands.init.credentials.Prompt.ask",
return_value="sk-test-value",
)
- mock_console: MagicMock = mocker.MagicMock()
+ mock_console: MockType = mocker.MagicMock()
prompt_credentials(console=mock_console, backends_toml_path=backends_toml)
# Verify .env was written
@@ -181,7 +180,7 @@ class TestInitCredentials:
"pipelex.cli.commands.init.credentials.Prompt.ask",
return_value="",
)
- mock_console: MagicMock = mocker.MagicMock()
+ mock_console: MockType = mocker.MagicMock()
prompt_credentials(console=mock_console, backends_toml_path=backends_toml)
# Verify .env was NOT written (no values entered)
@@ -206,7 +205,7 @@ class TestInitCredentials:
"pipelex.cli.commands.init.credentials.Prompt.ask",
return_value="sk-new",
)
- mock_console: MagicMock = mocker.MagicMock()
+ mock_console: MockType = mocker.MagicMock()
prompt_credentials(console=mock_console, backends_toml_path=backends_toml)
result = read_env_file(env_path)
@@ -225,6 +224,6 @@ class TestInitCredentials:
global_config_dir=tmp_path,
)
mock_prompt = mocker.patch("pipelex.cli.commands.init.credentials.Prompt.ask")
- mock_console: MagicMock = mocker.MagicMock()
+ mock_console: MockType = mocker.MagicMock()
prompt_credentials(console=mock_console, backends_toml_path=backends_toml)
mock_prompt.assert_not_called()
diff --git a/tests/unit/pipelex/cli/test_resolve_exit_codes.py b/tests/unit/pipelex/cli/test_resolve_exit_codes.py
index 1bcf30ab3..fef030fcc 100644
--- a/tests/unit/pipelex/cli/test_resolve_exit_codes.py
+++ b/tests/unit/pipelex/cli/test_resolve_exit_codes.py
@@ -22,9 +22,7 @@ from pipelex.libraries.exceptions import LibraryLoadingError
from pipelex.libraries.pipe.exceptions import PipeLibraryError
if TYPE_CHECKING:
- from unittest.mock import MagicMock
-
- from pytest_mock import MockerFixture
+ from pytest_mock import MockerFixture, MockType
MODULE = "pipelex.cli.commands.resolve_cmd"
# The load/normalize/library-manager seams now live in the shared crate-loading helper, so the
@@ -35,7 +33,7 @@ CRATE_LOADING = "pipelex.cli.commands.crate_loading"
class TestResolveExitCodes:
"""The resolve 0/1/2 exit-code policy, with boot/teardown mocked out."""
- def _neutralize_boot(self, mocker: MockerFixture) -> MagicMock:
+ def _neutralize_boot(self, mocker: MockerFixture) -> MockType:
boot = mocker.patch(f"{MODULE}.make_pipelex_for_cli")
mocker.patch(f"{MODULE}.Pipelex.teardown_if_needed")
mocker.patch(f"{MODULE}.tag")
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
index 4e8ff3fc4..6201c6a66 100644
--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py
+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
@@ -1,11 +1,15 @@
+import json
import socket
import pytest
-from pytest_mock import MockerFixture
+from pytest_mock import MockerFixture, MockType
from pipelex.base_exceptions import ErrorDomain, ErrorReport
from pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction, UserActionKind
from pipelex.cogt.inference.provider_name import ProviderName
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.core.concepts.concept import Concept
from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
@@ -20,6 +24,7 @@ from pipelex.pipe_run.delivery_assignment import (
)
from pipelex.pipe_run.delivery_executor import DeliveryExecutor
from pipelex.pipe_run.exceptions import PipeJobError, StorageDeliveryError, WebhookDeliveryError
+from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.network.exceptions import SsrfBlockedError
@@ -37,6 +42,14 @@ def _make_main_stuff() -> Stuff:
)
+def _make_output_mock(mocker: MockerFixture) -> MockType:
+ """A PipeOutput stand-in with the usage fields defaulted to None, like a run with usage assembly off."""
+ mock_output: MockType = mocker.MagicMock()
+ mock_output.tokens_usages = None
+ mock_output.usage_assembly_error = None
+ return mock_output
+
+
@pytest.mark.asyncio(loop_scope="class")
class TestDeliveryExecutor:
async def test_execute_storage_only(self, mocker: MockerFixture) -> None:
@@ -45,7 +58,7 @@ class TestDeliveryExecutor:
mock_storage.public_url = mocker.Mock(return_value="file:///tmp/results/plr-123")
mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = None
mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
@@ -69,6 +82,7 @@ class TestDeliveryExecutor:
assert any("test-user/plr-123/main_stuff.json" in key for key in stored_keys)
assert any("test-user/plr-123/main_stuff.md" in key for key in stored_keys)
assert any("test-user/plr-123/main_stuff.html" in key for key in stored_keys)
+ assert any("test-user/plr-123/tokens_usages.json" in key for key in stored_keys)
async def test_execute_webhook_only(self, mocker: MockerFixture) -> None:
mock_client = mocker.AsyncMock()
@@ -163,7 +177,7 @@ class TestDeliveryExecutor:
mock_storage.store = mocker.AsyncMock(side_effect=Exception("S3 down"))
mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = None
mock_output.working_memory.smart_dump.return_value = {}
mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
@@ -185,7 +199,7 @@ class TestDeliveryExecutor:
"""A completed run always resolves its declared output — a typed working memory with neither
a main stuff nor a recorded absence is a contract violation, not an empty envelope.
"""
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = None
mock_output.working_memory = WorkingMemory()
mock_output.graph_spec = None
@@ -195,7 +209,7 @@ class TestDeliveryExecutor:
async def test_generate_result_files_raises_without_main_stuff_raw(self, mocker: MockerFixture) -> None:
"""Same contract on the raw (cross-process) path: neither a raw main stuff nor a recorded absence fails loudly."""
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = {"root": {}, "aliases": {}}
mock_output.graph_spec = None
@@ -215,7 +229,7 @@ class TestDeliveryExecutor:
producing_pipe="check_penalty_clause",
),
)
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = None
mock_output.working_memory = working_memory
mock_output.graph_spec = None
@@ -236,7 +250,7 @@ class TestDeliveryExecutor:
"""Same on the raw (cross-process) path: a recorded main absence in the raw ledger delivers
the absence artifact instead of the contract-violation error.
"""
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = {
"root": {},
"aliases": {},
@@ -261,6 +275,70 @@ class TestDeliveryExecutor:
assert "summarize_penalty" in md_text
assert "main_stuff.html" in files
+ async def test_generate_result_files_writes_usage_artifact_with_records(self, mocker: MockerFixture) -> None:
+ """The tokens_usages.json artifact carries the run's usage records in the same
+ per-record ``model_dump(mode="json")`` wire shape the /execute response uses, so a
+ durable client can compute costs from the polled result files alone.
+ """
+ mock_output = _make_output_mock(mocker)
+ mock_output.working_memory_raw = None
+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
+ mock_output.graph_spec = None
+ mock_output.tokens_usages = [
+ LLMTokensUsage(
+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="plr-usage"),
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ nb_tokens_by_category={TokenCategory.INPUT: 15, TokenCategory.OUTPUT: 4},
+ unit_costs={CostCategory.INPUT: 3.0, CostCategory.OUTPUT: 15.0},
+ )
+ ]
+
+ files = await DeliveryExecutor().generate_result_files(mock_output)
+
+ assert files["tokens_usages.json"].content_type == "application/json"
+ usage_doc = json.loads(files["tokens_usages.json"].data.decode("utf-8"))
+ assert usage_doc["usage_assembly_error"] is None
+ records = usage_doc["tokens_usages"]
+ assert len(records) == 1
+ record = records[0]
+ assert record["model_type"] == "llm"
+ assert record["inference_model_name"] == "test-model"
+ assert record["inference_model_id"] == "test-model-id"
+ assert record["nb_tokens_by_category"] == {"input": 15, "output": 4}
+ assert record["unit_costs"] == {"input": 3.0, "output": 15.0}
+
+ async def test_generate_result_files_usage_artifact_null_when_usage_off(self, mocker: MockerFixture) -> None:
+ """A run with usage assembly off still writes the artifact with explicit nulls, so a
+ durable client can tell "usage off for this run" (file present, null) from "run
+ delivered before the artifact existed" (file absent).
+ """
+ mock_output = _make_output_mock(mocker)
+ mock_output.working_memory_raw = None
+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
+ mock_output.graph_spec = None
+
+ files = await DeliveryExecutor().generate_result_files(mock_output)
+
+ usage_doc = json.loads(files["tokens_usages.json"].data.decode("utf-8"))
+ assert usage_doc == {"tokens_usages": None, "usage_assembly_error": None}
+
+ async def test_generate_result_files_usage_artifact_carries_assembly_error(self, mocker: MockerFixture) -> None:
+ """A failed usage assembly surfaces on the artifact instead of silently reading as "usage off"."""
+ mock_output = _make_output_mock(mocker)
+ mock_output.working_memory_raw = None
+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
+ mock_output.graph_spec = None
+ mock_output.usage_assembly_error = "usage event read failed"
+
+ files = await DeliveryExecutor().generate_result_files(mock_output)
+
+ usage_doc = json.loads(files["tokens_usages.json"].data.decode("utf-8"))
+ assert usage_doc == {"tokens_usages": None, "usage_assembly_error": "usage event read failed"}
+
async def test_try_local_hydrate_stuff_returns_typed_for_builtin(self) -> None:
from pipelex.core.stuffs.text_content import TextContent # noqa: PLC0415
from pipelex.hub import get_class_registry # noqa: PLC0415
@@ -329,7 +407,7 @@ class TestDeliveryExecutor:
"</pre><script>..." would break out of the <pre> wrapper and execute
as live HTML when the stored result file is fetched.
"""
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = {
"root": {
"main_stuff": {
@@ -387,7 +465,7 @@ class TestDeliveryExecutor:
),
)
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = {
"root": {
"cv_pages": {
@@ -527,7 +605,7 @@ class TestDeliveryExecutor:
mock_storage.store = mocker.AsyncMock(return_value="pipelex-storage://test-key")
mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)
- mock_output = mocker.MagicMock()
+ mock_output = _make_output_mock(mocker)
mock_output.working_memory_raw = None
mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
diff --git a/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py b/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py
index 8d370a41c..1d93fd682 100644
--- a/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py
+++ b/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py
@@ -23,9 +23,7 @@ from pipelex.system.pipelex_service.remote_config_cache import (
)
if TYPE_CHECKING:
- from unittest.mock import MagicMock
-
- from pytest_mock import MockerFixture
+ from pytest_mock import MockerFixture, MockType
def _valid_remote_config_payload(extra: dict[str, object] | None = None) -> dict[str, object]:
@@ -87,7 +85,7 @@ class TestRemoteConfigCache:
def test_read_corrupted_json_returns_none_and_logs(
self,
isolated_cache_dir: Path,
- mock_log: MagicMock,
+ mock_log: MockType,
) -> None:
cache_path = isolated_cache_dir / "cache" / "remote_config.json"
cache_path.parent.mkdir(parents=True, exist_ok=True)
diff --git a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py
index f9a1a967f..842803aa2 100644
--- a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py
+++ b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py
@@ -1,7 +1,6 @@
from datetime import timedelta
from pathlib import Path
from typing import Any
-from unittest.mock import MagicMock
import pytest
from pytest_mock import MockerFixture
@@ -47,16 +46,16 @@ class TestGcpStorageProvider:
from google.cloud import storage # type: ignore[import-untyped] # noqa: PLC0415
# Create mock objects
- mock_blob = MagicMock()
+ mock_blob = mocker.MagicMock()
mock_blob.exists.return_value = True
mock_blob.download_as_bytes.return_value = b""
- mock_blob.upload_from_string = MagicMock()
+ mock_blob.upload_from_string = mocker.MagicMock()
mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"
- mock_bucket = MagicMock()
+ mock_bucket = mocker.MagicMock()
mock_bucket.blob.return_value = mock_blob
- mock_client = MagicMock()
+ mock_client = mocker.MagicMock()
mock_client.bucket.return_value = mock_bucket
# Patch the Client.from_service_account_json method directly
diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
index 38c008d11..e227ec05d 100644
--- a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py
@@ -1,5 +1,4 @@
from typing import Any
-from unittest.mock import AsyncMock, MagicMock
import pytest
from botocore.exceptions import BotoCoreError, ClientError, EndpointConnectionError, NoCredentialsError, ReadTimeoutError
@@ -26,33 +25,33 @@ class TestS3StorageProvider:
for test assertions.
"""
# Create mock stream for response body
- mock_stream = AsyncMock()
- mock_stream.read = AsyncMock(return_value=b"")
- mock_stream.__aenter__ = AsyncMock(return_value=mock_stream)
- mock_stream.__aexit__ = AsyncMock(return_value=None)
+ mock_stream = mocker.AsyncMock()
+ mock_stream.read = mocker.AsyncMock(return_value=b"")
+ mock_stream.__aenter__ = mocker.AsyncMock(return_value=mock_stream)
+ mock_stream.__aexit__ = mocker.AsyncMock(return_value=None)
# Create mock client
- mock_client = AsyncMock()
- mock_client.get_object = AsyncMock(return_value={"Body": mock_stream})
- mock_client.put_object = AsyncMock(return_value={})
- mock_client.generate_presigned_url = AsyncMock(
+ mock_client = mocker.AsyncMock()
+ mock_client.get_object = mocker.AsyncMock(return_value={"Body": mock_stream})
+ mock_client.put_object = mocker.AsyncMock(return_value={})
+ mock_client.generate_presigned_url = mocker.AsyncMock(
return_value=f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/test?signature=abc123"
)
# Create mock exceptions (modeled exceptions on the client)
- mock_exceptions = MagicMock()
+ mock_exceptions = mocker.MagicMock()
mock_exceptions.NoSuchKey = type("NoSuchKey", (Exception,), {})
mock_exceptions.NoSuchBucket = type("NoSuchBucket", (Exception,), {})
mock_client.exceptions = mock_exceptions
# Create async context manager for client
- mock_client_context = AsyncMock()
- mock_client_context.__aenter__ = AsyncMock(return_value=mock_client)
- mock_client_context.__aexit__ = AsyncMock(return_value=None)
+ mock_client_context = mocker.AsyncMock()
+ mock_client_context.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client_context.__aexit__ = mocker.AsyncMock(return_value=None)
# Create mock session
- mock_session = MagicMock()
- mock_session.client = MagicMock(return_value=mock_client_context)
+ mock_session = mocker.MagicMock()
+ mock_session.client = mocker.MagicMock(return_value=mock_client_context)
# Patch aioboto3.Session
mocker.patch("aioboto3.Session", return_value=mock_session)
@@ -118,12 +117,13 @@ class TestS3StorageProvider:
async def test_load_returns_data(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that load() returns the data from S3."""
expected_data = b"Test data \x00\x01\x02\xff"
- mock_aioboto3["stream"].read = AsyncMock(return_value=expected_data)
+ mock_aioboto3["stream"].read = mocker.AsyncMock(return_value=expected_data)
uri = f"{PIPELEX_STORAGE_SCHEME}test/file.bin"
loaded_data = await s3_provider_no_signed_urls.load(uri=uri)
@@ -132,11 +132,12 @@ class TestS3StorageProvider:
async def test_load_with_nonexistent_key_raises_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that loading a non-existent object raises StorageFileNotFoundError."""
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchKey("Key not found"))
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchKey("Key not found"))
nonexistent_uri = f"{PIPELEX_STORAGE_SCHEME}nonexistent/file.bin"
with pytest.raises(StorageFileNotFoundError) as exc_info:
@@ -173,6 +174,7 @@ class TestS3StorageProvider:
async def test_public_url_returns_presigned_url_when_signed_urls_enabled(
self,
+ mocker: MockerFixture,
s3_provider_with_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
@@ -180,7 +182,7 @@ class TestS3StorageProvider:
key = "presigned/test.bin"
uri = f"{PIPELEX_STORAGE_SCHEME}{key}"
expected_presigned = "https://test-bucket.s3.amazonaws.com/presigned/test.bin?X-Amz-Signature=xyz"
- mock_aioboto3["client"].generate_presigned_url = AsyncMock(return_value=expected_presigned)
+ mock_aioboto3["client"].generate_presigned_url = mocker.AsyncMock(return_value=expected_presigned)
display = await s3_provider_with_signed_urls.public_url(uri=uri)
@@ -193,6 +195,7 @@ class TestS3StorageProvider:
async def test_public_url_handles_sync_presigned_url(
self,
+ mocker: MockerFixture,
s3_provider_with_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
@@ -201,7 +204,7 @@ class TestS3StorageProvider:
uri = f"{PIPELEX_STORAGE_SCHEME}{key}"
expected_presigned = "https://test-bucket.s3.amazonaws.com/sync-presign/test.bin?X-Amz-Signature=sync"
# Return a plain string (non-awaitable) to simulate sync behavior in some aioboto3 versions
- mock_aioboto3["client"].generate_presigned_url = MagicMock(return_value=expected_presigned)
+ mock_aioboto3["client"].generate_presigned_url = mocker.MagicMock(return_value=expected_presigned)
display = await s3_provider_with_signed_urls.public_url(uri=uri)
@@ -254,11 +257,12 @@ class TestS3StorageProvider:
async def test_store_bucket_not_found_raises_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that storing to a non-existent bucket raises StorageS3Error."""
- mock_aioboto3["client"].put_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))
+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))
with pytest.raises(StorageS3Error) as exc_info:
await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")
@@ -267,11 +271,12 @@ class TestS3StorageProvider:
async def test_load_bucket_not_found_raises_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that loading from a non-existent bucket raises StorageS3Error."""
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))
uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"
with pytest.raises(StorageS3Error) as exc_info:
@@ -281,12 +286,13 @@ class TestS3StorageProvider:
async def test_load_client_error_with_no_such_key_code_raises_file_not_found(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that ClientError with NoSuchKey code raises StorageFileNotFoundError."""
error_response: Any = {"Error": {"Code": "NoSuchKey", "Message": "The specified key does not exist."}}
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=ClientError(error_response, "GetObject"))
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=ClientError(error_response, "GetObject"))
uri = f"{PIPELEX_STORAGE_SCHEME}missing/file.bin"
with pytest.raises(StorageFileNotFoundError) as exc_info:
@@ -296,12 +302,13 @@ class TestS3StorageProvider:
async def test_load_client_error_with_access_denied_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that ClientError with AccessDenied code raises StorageS3Error."""
error_response: Any = {"Error": {"Code": "AccessDenied", "Message": "Access Denied"}}
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=ClientError(error_response, "GetObject"))
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=ClientError(error_response, "GetObject"))
uri = f"{PIPELEX_STORAGE_SCHEME}forbidden/file.bin"
with pytest.raises(StorageS3Error) as exc_info:
@@ -311,12 +318,13 @@ class TestS3StorageProvider:
async def test_store_client_error_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that ClientError during store raises StorageS3Error."""
error_response: Any = {"Error": {"Code": "InvalidAccessKeyId", "Message": "Invalid access key"}}
- mock_aioboto3["client"].put_object = AsyncMock(side_effect=ClientError(error_response, "PutObject"))
+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=ClientError(error_response, "PutObject"))
with pytest.raises(StorageS3Error) as exc_info:
await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")
@@ -325,11 +333,12 @@ class TestS3StorageProvider:
async def test_load_no_credentials_error_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that NoCredentialsError (a BotoCoreError subclass) raises StorageS3Error."""
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=NoCredentialsError())
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=NoCredentialsError())
uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"
with pytest.raises(StorageS3Error) as exc_info:
@@ -340,11 +349,12 @@ class TestS3StorageProvider:
async def test_store_no_credentials_error_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that NoCredentialsError (a BotoCoreError subclass) during store raises StorageS3Error."""
- mock_aioboto3["client"].put_object = AsyncMock(side_effect=NoCredentialsError())
+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=NoCredentialsError())
with pytest.raises(StorageS3Error) as exc_info:
await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")
@@ -354,11 +364,12 @@ class TestS3StorageProvider:
async def test_load_endpoint_connection_error_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that EndpointConnectionError (a BotoCoreError subclass) raises StorageS3Error."""
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=EndpointConnectionError(endpoint_url="https://s3.amazonaws.com"))
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=EndpointConnectionError(endpoint_url="https://s3.amazonaws.com"))
uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"
with pytest.raises(StorageS3Error) as exc_info:
@@ -369,11 +380,12 @@ class TestS3StorageProvider:
async def test_load_read_timeout_error_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that ReadTimeoutError — the canonical transient-network BotoCoreError that used to escape — raises StorageS3Error."""
- mock_aioboto3["client"].get_object = AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))
+ mock_aioboto3["client"].get_object = mocker.AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))
uri = f"{PIPELEX_STORAGE_SCHEME}slow/file.bin"
with pytest.raises(StorageS3Error) as exc_info:
@@ -385,11 +397,12 @@ class TestS3StorageProvider:
async def test_store_read_timeout_error_raises_s3_error(
self,
+ mocker: MockerFixture,
s3_provider_no_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that ReadTimeoutError during store (slow upload, transient blip) raises StorageS3Error."""
- mock_aioboto3["client"].put_object = AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))
+ mock_aioboto3["client"].put_object = mocker.AsyncMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))
with pytest.raises(StorageS3Error) as exc_info:
await s3_provider_no_signed_urls.store(data=b"test", key="slow/upload.bin")
@@ -400,13 +413,14 @@ class TestS3StorageProvider:
async def test_public_url_falls_back_to_public_url_on_botocore_error(
self,
+ mocker: MockerFixture,
s3_provider_with_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
"""Test that public_url() falls back to public URL on a transport BotoCoreError, not just ClientError."""
key = "fallback/timeout.bin"
uri = f"{PIPELEX_STORAGE_SCHEME}{key}"
- mock_aioboto3["client"].generate_presigned_url = MagicMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))
+ mock_aioboto3["client"].generate_presigned_url = mocker.MagicMock(side_effect=ReadTimeoutError(endpoint_url="https://s3.amazonaws.com"))
display = await s3_provider_with_signed_urls.public_url(uri=uri)
@@ -415,6 +429,7 @@ class TestS3StorageProvider:
async def test_public_url_falls_back_to_public_url_on_client_error(
self,
+ mocker: MockerFixture,
s3_provider_with_signed_urls: S3StorageProvider,
mock_aioboto3: dict[str, Any],
) -> None:
@@ -422,7 +437,7 @@ class TestS3StorageProvider:
key = "fallback/test.bin"
uri = f"{PIPELEX_STORAGE_SCHEME}{key}"
error_response: Any = {"Error": {"Code": "SignatureDoesNotMatch", "Message": "Signature error"}}
- mock_aioboto3["client"].generate_presigned_url = MagicMock(side_effect=ClientError(error_response, "GeneratePresignedUrl"))
+ mock_aioboto3["client"].generate_presigned_url = mocker.MagicMock(side_effect=ClientError(error_response, "GeneratePresignedUrl"))
display = await s3_provider_with_signed_urls.public_url(uri=uri)
diff --git a/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py b/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py
index 9f39f159b..1b7751b91 100644
--- a/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py
+++ b/tests/unit/pipelex/tools/test_jinja2_with_images_filter.py
@@ -5,10 +5,10 @@ Note: Core functionality tests using real Stuff classes are in integration tests
"""
from typing import Any
-from unittest.mock import MagicMock
import pytest
from jinja2.runtime import Context, Undefined
+from pytest_mock import MockerFixture, MockType
from pipelex.cogt.templating.text_format import TextFormat
from pipelex.tools.jinja2.exceptions import Jinja2ContextError
@@ -23,76 +23,76 @@ from pipelex.tools.jinja2.jinja2_with_images_filter import (
class TestWithImagesFilterValidation:
"""Tests for with_images filter error handling and edge cases."""
- def _make_context(self, registry: ImageRegistry | None = None, text_format: TextFormat = TextFormat.PLAIN) -> Context:
+ def _make_context(self, mocker: MockerFixture, registry: ImageRegistry | None = None, text_format: TextFormat = TextFormat.PLAIN) -> Context:
"""Create a mock Jinja2 context."""
context_dict: dict[str, Any] = {}
if registry is not None:
context_dict[Jinja2ContextKey.IMAGE_REGISTRY] = registry
context_dict[Jinja2ContextKey.TEXT_FORMAT] = text_format
- mock_env = MagicMock()
+ mock_env = mocker.MagicMock()
mock_env.undefined = Undefined
- context = MagicMock(spec=Context)
+ context: MockType = mocker.MagicMock(spec=Context)
context.get = lambda key, default=None: context_dict.get(key, default) # pyright: ignore[reportUnknownLambdaType, reportUnknownArgumentType]
context.environment = mock_env
return context
- def test_with_images_raises_on_undefined(self) -> None:
+ def test_with_images_raises_on_undefined(self, mocker: MockerFixture) -> None:
"""Test with_images raises error for undefined value."""
registry = ImageRegistry()
- context = self._make_context(registry=registry)
+ context = self._make_context(mocker, registry=registry)
with pytest.raises(Jinja2ContextError, match="undefined"):
with_images(context, value=Undefined())
- def test_with_images_raises_on_wrong_registry_type(self) -> None:
+ def test_with_images_raises_on_wrong_registry_type(self, mocker: MockerFixture) -> None:
"""Test with_images raises if registry is wrong type."""
context_dict: dict[str, Any] = {
Jinja2ContextKey.IMAGE_REGISTRY: "not_a_registry",
Jinja2ContextKey.TEXT_FORMAT: TextFormat.PLAIN,
}
- context = MagicMock(spec=Context)
+ context = mocker.MagicMock(spec=Context)
context.get = lambda key, default=None: context_dict.get(key, default) # pyright: ignore[reportUnknownLambdaType, reportUnknownArgumentType]
# Pass a list (which is accepted by the filter) to trigger the registry type check
with pytest.raises(Jinja2ContextError, match="Expected ImageRegistry"):
with_images(context, value=[])
- def test_with_images_raises_on_string_input(self) -> None:
+ def test_with_images_raises_on_string_input(self, mocker: MockerFixture) -> None:
"""Test with_images raises error when receiving a plain string.
This catches the common mistake of chaining filters in wrong order,
e.g., {{ pages | tag | with_images }} where tag converts to string first.
"""
registry = ImageRegistry()
- context = self._make_context(registry=registry)
+ context = self._make_context(mocker, registry=registry)
with pytest.raises(Jinja2ContextError, match="does not implement the ImageRenderable protocol"):
with_images(context, value="This is a plain string")
- def test_with_images_raises_on_number_input(self) -> None:
+ def test_with_images_raises_on_number_input(self, mocker: MockerFixture) -> None:
"""Test with_images raises error when receiving a number."""
registry = ImageRegistry()
- context = self._make_context(registry=registry)
+ context = self._make_context(mocker, registry=registry)
with pytest.raises(Jinja2ContextError, match="does not implement the ImageRenderable protocol"):
with_images(context, value=42)
- def test_with_images_raises_on_dict_input(self) -> None:
+ def test_with_images_raises_on_dict_input(self, mocker: MockerFixture) -> None:
"""Test with_images raises error when receiving a plain dict."""
registry = ImageRegistry()
- context = self._make_context(registry=registry)
+ context = self._make_context(mocker, registry=registry)
with pytest.raises(Jinja2ContextError, match="does not implement the ImageRenderable protocol"):
with_images(context, value={"key": "value"})
- def test_with_images_accepts_empty_list(self) -> None:
+ def test_with_images_accepts_empty_list(self, mocker: MockerFixture) -> None:
"""Test with_images handles empty list gracefully."""
registry = ImageRegistry()
- context = self._make_context(registry=registry)
+ context = self._make_context(mocker, registry=registry)
result = with_images(context, value=[])
diff --git a/wip/durable-results-tokens-usage-exposure.md b/wip/durable-results-tokens-usage-exposure.md
new file mode 100644
index 000000000..d719be5ed
--- /dev/null
+++ b/wip/durable-results-tokens-usage-exposure.md
@@ -0,0 +1,46 @@
+# Durable-run results drop `tokens_usages` — expose usage on the start+poll path
+
+**Status:** Feature ask from the Fenix workspace (delivery-plan task A6, cost-measurement
+exploration, 2026-07-18). Live-verified against hosted api-dev. Full findings:
+`~/repos/Fenix/project/designs/a6-cost-measurement.md`.
+
+## The gap
+
+The engine assembles per-call usage onto `PipeOutput.tokens_usages` (token counts by category +
+`unit_costs` $/1M + model id, for LLM **and** img-gen/extract/search), and the sync path returns
+it: `POST /v1/execute` → `pipe_output.tokens_usages` — confirmed live (probe run `d020718b…`:
+claude-4.6-sonnet, `{input: 15, output: 4}`, `unit_costs {input: 3.0, output: 15.0}` → $0.000105
+computable client-side).
+
+But a durable client (`start` + `GET /v1/runs/{id}/results`) never sees it:
+
+- **pipelex** — `pipe_run/delivery_executor.py` `generate_result_files(pipe_output)` receives the
+ full `PipeOutput` (so `tokens_usages` + `usage_assembly_error` are in hand) but persists only
+ `graphspec.json`, `main_stuff.json` (+ md/html/viewer renders), `working_memory.json`.
+- **pipelex-platform** — `src/pipelex_platform/routers/v1/runs.py`
+ `_fetch_run_result_artifacts()` reads exactly those three files; `RunResultsResponse` has no
+ usage field. Per-node `metrics: {}` in the graphspec sits empty; no
+ `/runs/{id}/usage`-style route exists (probed, 404).
+
+Sync `/execute` is not a workaround for durable clients: the ~30s gateway ceiling rules it out
+for img-gen and parallel-heavy pipes — exactly the runs whose cost matters most.
+
+## Proposed change (smallest that closes it)
+
+1. **pipelex** `delivery_executor.generate_result_files`: also write `tokens_usages.json` —
+ `{"tokens_usages": [...], "usage_assembly_error": null}`, same serialization the `/execute`
+ response already uses.
+2. **pipelex-platform** `runs.py`: fetch the fourth file in `_fetch_run_result_artifacts()`
+ (tolerating its absence for runs delivered before the change) and add
+ `tokens_usages` / `usage_assembly_error` to `RunResultsResponse`.
+
+No SDK release needed to unblock: `pipelex-sdk` 0.4.0's `RunResults` is `extra="allow"`, so the
+new fields ride along via `model_extra`; typing them in the SDK can follow.
+
+## Notes
+
+- Keep it raw records, not a computed $ total — consumers (e.g. `fenix-pipelex`) do the
+ multiply-and-sum deterministically, and the existing `CostRegistry.build_cost_summary` shape is
+ available if a summary is wanted later.
+- Known caveat carried over from the engine: models with no `costs` in their backend TOML price
+ at 0 — consumers should flag zero-cost records rather than trust them.