Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Address PR #1059 review findings: docs contract-accuracy + test-assertion pins

Louis Choquel8331e86686da
diff --git a/docs/features/cost-tracking.md b/docs/features/cost-tracking.md
index d863c90ec..42e1d3385 100644
--- a/docs/features/cost-tracking.md
+++ b/docs/features/cost-tracking.md
@@ -39,4 +39,4 @@ For configuration, see [Reporting Configuration](../configuration/config-practic
## Usage on the API Wire
-API clients (the `/execute` response and the durable `tokens_usages.json` result artifact) receive token usage in a trimmed, client-facing record shape with a computed per-call `cost` — not the full internal usage models this page describes. See [TokensUsage Wire Records](../under-the-hood/tokens-usage-wire-records.md).
+API clients (the `/execute` response and the durable `tokens_usages.json` result artifact) receive token usage in a trimmed, client-facing record shape with a computed per-call `cost` (`null` for unrated models) — not the full internal usage models this page describes. See [TokensUsage Wire Records](../under-the-hood/tokens-usage-wire-records.md).
diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.md
index f8d12e031..4ef6d53fe 100644
--- a/docs/under-the-hood/execution-graph-tracing.md
+++ b/docs/under-the-hood/execution-graph-tracing.md
@@ -262,7 +262,7 @@ graph_spec = manager.close_tracer(pipeline_run_id)
Trace events travel through an `EventLogProtocol` backend (`pipelex/tracing/`): the tracer emits events into it during the run (write side, wired in `pipeline_run_setup`), and `assemble_tracing` reads them back after the run to build the `GraphSpec` and usage aggregates (read side, triggered from `PipeRun.run`). Both sides normally build their backend instance independently from `tracing_config` via `make_event_log` — NDJSON files or DynamoDB bridge the two instances through external storage.
-The assembled usage rides back on `pipe_output.tokens_usages` (with any assembly failure on `usage_assembly_error`), which the sync `/execute` response returns directly. For delivery-enabled runs (a storage target set), the delivery executor (`pipelex/pipe_run/delivery_executor.py`) also persists it as a `tokens_usages.json` result artifact — `{"tokens_usages": [...], "usage_assembly_error": null}` — next to `working_memory.json`, the `main_stuff.*` renders, and the graph outputs, so a durable client polling result files gets the same usage records a sync caller does. The artifact is written unconditionally: explicit nulls mean usage assembly was off for that run, while an absent file means the run was delivered before the artifact existed.
+The assembled usage rides back on `pipe_output.tokens_usages` (with any assembly failure on `usage_assembly_error`), which the sync `/execute` response returns directly. For delivery-enabled runs (a storage target set), the delivery executor (`pipelex/pipe_run/delivery_executor.py`) also persists it as a `tokens_usages.json` result artifact — `{"tokens_usages": [...], "usage_assembly_error": null}` — next to `working_memory.json`, the `main_stuff.*` renders, and the graph outputs, so a durable client polling result files gets the same usage records a sync caller does. The artifact is written unconditionally on every successful result delivery: explicit nulls mean usage assembly was off for that run. A failed run stores no result files at all, so an absent `tokens_usages.json` means either the run failed or it was delivered before the artifact existed — tell those apart from the delivery status, not from the file's presence.
For fully in-process runs, `pipelex.hub.scoped_event_log` pins one shared instance for both sides instead:
diff --git a/docs/under-the-hood/tokens-usage-wire-records.md b/docs/under-the-hood/tokens-usage-wire-records.md
index 1d2488cef..86724bf11 100644
--- a/docs/under-the-hood/tokens-usage-wire-records.md
+++ b/docs/under-the-hood/tokens-usage-wire-records.md
@@ -47,7 +47,7 @@ The record shape is also invariant to server-side observability settings: teleme
## Null semantics
-- `tokens_usages: null` — usage assembly was off for the run (or, hosted, the run was delivered before the artifact existed, in which case the durable file is absent entirely).
+- `tokens_usages: null` — usage assembly was off for the run. (The durable file is absent entirely when the run failed — a failed run stores no result files — or, hosted, when it was delivered before the artifact existed.)
- `tokens_usages: []` — assembly on, no inference happened.
- `usage_assembly_error` non-null — assembly *broke* (event-read failure), distinct from "off". It is a bare diagnostic string, mirroring `graph_assembly_error`.
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
index f41eb0c94..c4c3b158a 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
@@ -40,7 +40,7 @@ class TestPipeFuncExecutionTransport:
def test_request_rejects_non_finite_timeout(self) -> None:
"""A non-finite timeout would disable the runaway-code guard, so the wire contract rejects it."""
- with pytest.raises(ValidationError):
+ with pytest.raises(ValidationError, match="timeout_seconds"):
PipeFuncExecutionRequest(
crate=LibraryCrate(),
working_memory_raw={},
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
index 62b965b41..4517d0f1f 100644
--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py
+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
@@ -287,6 +287,7 @@ class TestDeliveryExecutor:
mock_output.graph_spec = None
mock_output.tokens_usages = [
LLMTokensUsage(
+ model_type="llm",
job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="plr-usage"),
inference_model_name="test-model",
inference_model_id="test-model-id",
diff --git a/wip/durable-results-tokens-usage-exposure.md b/wip/durable-results-tokens-usage-exposure.md
index c5bbad2b2..2143b0b26 100644
--- a/wip/durable-results-tokens-usage-exposure.md
+++ b/wip/durable-results-tokens-usage-exposure.md
@@ -27,7 +27,9 @@ But a durable client (`start` + `GET /v1/runs/{id}/results`) never sees it:
Sync `/execute` is not a workaround for durable clients: the ~30s gateway ceiling rules it out
for img-gen and parallel-heavy pipes — exactly the runs whose cost matters most.
-## Proposed change (smallest that closes it)
+## Proposed change (smallest that closes it) — ✅ BOTH SHIPPED in v0.40
+
+Both halves landed: pipelex writes the artifact (`delivery_executor.generate_result_files`), pipelex-platform serves it (`runs.py`).
1. **pipelex** `delivery_executor.generate_result_files`: also write `tokens_usages.json` —
`{"tokens_usages": [...], "usage_assembly_error": null}`, same serialization the `/execute`
diff --git a/wip/pipe-compose-whole-stuff-fix.md b/wip/pipe-compose-whole-stuff-fix.md
index fa34220e7..c6109e3df 100644
--- a/wip/pipe-compose-whole-stuff-fix.md
+++ b/wip/pipe-compose-whole-stuff-fix.md
@@ -114,4 +114,4 @@ All changes in `pipelex/pipe_operators/compose/structured_content_composer.py` (
- [x] `mthds-plugins/templates/skills/shared/mthds-reference.md.j2` (the generated per-target copies come from this template): construct `from` row now reads "Reference a whole input or a nested field" + a note on the whole-native-stuff → native-field automatic conversion; `make build` propagated to all target copies
- [x] `mcp-demos/wip/pipe-compose-issue/README.md`: annotated — update banner at top, correction blockquote in §5, update blockquote in §8 with the stale takeaways marked inline; root cause = implementation gaps fixed in pipelex v0.39.2, compose-based design recommended again
-**Phase 4 executed 2026-07-18 — TRACK COMPLETE.** Edits uncommitted in `pipelex-plugins/`, `mthds-plugins/`, `mcp-demos/` (commit per-repo).
+**Phase 4 executed 2026-07-18 — TRACK COMPLETE.** Committed per-repo: `pipelex-plugins` 2f97d2d, `mthds-plugins` 4e6288c, `mcp-demos` d422861.
diff --git a/wip/pr-1059-review-notes.md b/wip/pr-1059-review-notes.md
new file mode 100644
index 000000000..343466145
--- /dev/null
+++ b/wip/pr-1059-review-notes.md
@@ -0,0 +1,28 @@
+# PR #1059 review notes — deferred items
+
+Triage record for the SWE-agent review threads on [PR #1059](https://github.com/Pipelex/pipelex/pull/1059) (release v0.40.0, carrying the `TokensUsageRecord` client wire shape, the durable `tokens_usages.json` artifact, and PipeFunc transport hardening). Each of the 14 unresolved threads was verified against the code by a dedicated Opus agent. Confirmed important items were fixed in-place on the release branch (docs contract-accuracy sentence, two test-assertion tightenings, two wip-tracker staleness corrections); false positives were replied to and resolved. This file captures the items deliberately **deferred** — each is real (or partially real) but is a design decision, a diagnostics-quality tradeoff, or test-only fragility that a release-branch review pass should not resolve unilaterally. Their threads are left **open**.
+
+## 1. Wire-cost / CLI-total parity — fixture-coverage gap (cubic P2)
+
+- **Reporter:** cubic (thread `PRRT_kwDOOwmMFc6SEzQN`), `pipelex/cogt/usage/cost_registry.py` (`compute_tokens_usage_cost`).
+- **Verdict:** false positive *as filed*, with a narrow real residual. The finding claims `compute_tokens_usage_cost` "duplicates the logic in `CostRegistry.complete_cost_report`" and can diverge. In fact the two paths already share the pieces that carry the cost *policy* — the rate lookup + cached-discount fallback (`model_cost_per_token`) and the total formula (`compute_total_cost`). Only the structural component derivation is duplicated. Parity is **explicitly asserted**: `tests/unit/pipelex/cogt/usage/test_usage_cost.py:37-46` compares the summed wire `cost` against `CostRegistry.aggregate_costs(...).total_cost` — the actual number the CLI cost table prints — plus per-record parity across all four usage variants.
+- **Why deferred:** the only divergence that escapes the parity test is *widening which token categories count toward the table total* (e.g. starting to charge reasoning/audio), because the parity fixture `RATED_NB_TOKENS` (`tests/unit/pipelex/cogt/usage/test_data.py`) only carries `INPUT`/`INPUT_CACHED`/`OUTPUT`. That is a fixture-coverage hole, not the architectural drift the finding describes — and closing it by folding `compute_tokens_usage_cost` into `complete_cost_report` reshapes a client-facing response path, which is not worth destabilizing a release branch for.
+- **Recommendation:** cheap, test-only follow-up — add an `OUTPUT_REASONING`/`INPUT_AUDIO` fixture variant to `all_variants()` so the parity test also guards the excluded-category boundary. The delegation refactor (make parity structural instead of asserted) is optional and belongs off a release branch.
+
+## 2. Transported PipeFunc failures lose traceback source-line text (cubic P2)
+
+- **Reporter:** cubic (thread `PRRT_kwDOOwmMFc6SEzQb`), `pipelex/pipe_operators/func/direct_pipe_func_executor.py` (`run_pipe_func_transported`, `finally` cleanup).
+- **Verdict:** confirmed as a code fact, and PR-introduced. The new deterministic cleanup (`shutil.rmtree` of the materialized workdir in the `finally`, on every path) runs *during exception unwinding*, before the default excepthook formats the traceback. Because Python resolves frame source *text* lazily via `linecache` at format time (and the `SourceFileLoader` fallback also reads from the now-deleted path), the customer-code frames in the stderr diagnostic show `file:line:function` **without the source-line text**. Verified empirically: the subprocess entrypoint (`pipe_func_transported_entrypoint.py`) does not capture `traceback.format_exc()` into a shipped-back string, so nothing materializes the source before cleanup. Before this PR there was no cleanup at all (the workdir leaked), so those source lines used to survive — this is a genuine diagnostics regression traded for deterministic cleanup + `sys.path` pruning.
+- **Why deferred:** what is lost is only the source-line *text* of customer frames — the exception *type*, *message*, and every frame's *file + line-number + function* all survive, and the filename was a throwaway temp path anyway. The reviewer's "retain the workdir until diagnostics are captured" directly conflicts with the PR's goal (clean up on every path, which is needed for long-lived reuse — in-process sandbox / Temporal worker). The correct fix (eagerly materialize `traceback.TracebackException.from_exception(exc, lookup_lines=True)` before `rmtree`, then surface that captured text since the excepthook would re-look-up and fail again) is a non-trivial change to error-path behavior with excepthook interplay.
+- **Recommendation:** implement the eager-capture fix on a follow-up branch off `dev`, with a regression test (a crate whose `@pipe_func` raises at a known line; run transported; assert the emitted traceback contains the source-line text). Not worth landing on the release branch — it only affects failure-diagnostic richness, not correctness.
+
+## 3. `finally` teardown in the workdir test (cubic P2)
+
+- **Reporter:** cubic (thread `PRRT_kwDOOwmMFc6SEzQa`), `tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py` (`finally: get_library_manager().teardown(...)`).
+- **Verdict:** false positive on the concrete claim, minor latent nit. The finding says the unconditional teardown can raise `LibraryError` ("library not found") and mask the original test exception. Traced against the flow, that cannot happen: rehydration calls `open_fresh_library(library_id=_TRANSPORTED_LIBRARY_ID)` → `open_library` (which **registers** the library in `_libraries`) at step 2 of `rehydrate_library_and_memory`, *before* crate load, working-memory hydration, and the function lookup where a failure could occur. The test's trivial crate (`{"funcs.py": "x = 1\n"}`) cannot fail during source-materialization/import/registration, so every reachable failure point is *after* registration — teardown always finds the library and succeeds. The reviewer's suggested "just rely on `scoped_current_library`" is not viable alone: that context manager only restores the `_library_id` ContextVar, it does **not** unregister the process-global library, so some teardown is genuinely required to avoid leaking `_TRANSPORTED_LIBRARY_ID` into later tests.
+- **Why deferred:** the teardown is coupled to rehydration's internal ordering. A *future* change that made the transported run fail before `open_fresh_library` (e.g. an import-time error in the crate) would let the `finally` raise `LibraryError` that supersedes (chains over) the real failure — a latent robustness nit, not a current bug.
+- **Recommendation:** when convenient, make the teardown best-effort (guard on membership, or `try/except LibraryError: pass`) or move library lifecycle into a dedicated autouse fixture. Test-only and safe to land any time; no urgency.
+
+## Out of scope for this PR (other repo)
+
+- **`pipelex-platform` stale API-response docstring.** `pipelex-platform/src/pipelex_platform/routers/v1/runs.py:101-102` still describes `tokens_usages` as "token counts by category + **unit costs** + model id" — stale against the v0.40 `TokensUsageRecord` wire shape (which drops `unit_costs` and carries a server-computed `cost`). Unlike the wip trackers above, that one is a live API-response docstring. Fix belongs in the `pipelex-platform` repo; flagged here so it is not lost.