Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Distributed cost reporting (#967)
Louis Choquel8fe1eda53d40
diff --git a/.claude/skills/temporal-e2e-validate/SKILL.md b/.claude/skills/temporal-e2e-validate/SKILL.mdindex 52d53a7a6..002d98ec4 100644--- a/.claude/skills/temporal-e2e-validate/SKILL.md+++ b/.claude/skills/temporal-e2e-validate/SKILL.md@@ -7,8 +7,10 @@ description: >test cases, and (2) true 3-process setup (server + separate worker process+ submitter) that validates the actual deployment topology includingcross-process serialization, LibraryCrate propagation, deferred hydration,- concurrent isolation, image payload storage, and cross-worker graph- tracing with GraphSpec assembly. Step 8 validates v1 per-activity routing.+ concurrent isolation, image payload storage, cross-worker graph+ tracing with GraphSpec assembly, and cross-worker cost-report assembly+ (runner-side usage aggregated into a single submitter cost report, validated+ cheaply via `--mock-inference`). Step 8 validates v1 per-activity routing.Step 9 validates v2 per-queue submitter options (timeouts, retry,rate-limit), per-handle option overrides, named worker-runtime profilesselected via `--profile`, and the strict `--task-queue` CLI typo check@@ -22,10 +24,12 @@ description: >"temporal regression", "temporal validation", "3-process test", "full temporaltest", "validate phases", "image temporal", "queue options", "runtime profile","validate error handling", "error report", "error propagation",+ "cost report", "distributed cost", "mock inference",or wants comprehensive verification that distributed execution worksend-to-end. Also use proactively after changes to LibraryCrate, deferredhydration, ClassRegistry scoping, graph tracing, image generation activities,content storage, queue options resolution, worker-runtime profiles,+ usage / cost reporting, the `--costs` / `--mock-inference` flags,error classification / `ErrorReport` / the Temporal error boundary, orTemporal workflow code.allowed-tools:@@ -293,7 +297,7 @@ loaded only for the work at hand. Read them as needed:| Reference file | Covers | Read it when ||---|---|---|| `references/mode-2-setup.md` | Steps 1–2: Temporal server + worker processes; run-mode (dry vs live) | **Always — start here.** Every other Mode 2 file assumes the server + worker(s) are already up. |-| `references/mode-2-tiers.md` | Steps 3–7: Tiers 1–14 (sequence, hydration, parallel, image gen/flow, isolation, object gen, extract, CV batch, error propagation), codec, final report | Validating core distributed execution, or running the full suite. |+| `references/mode-2-tiers.md` | Steps 3–7: Tiers 1–16 (sequence, hydration, parallel, image gen/flow, isolation, cross-worker usage + cost report, object gen, extract, CV batch, error propagation), codec, final report | Validating core distributed execution, cost reporting, or running the full suite. || `references/routing-battery.md` | Step 8: v1 `activity_queues` per-activity / per-handle routing (Tiers 10a–10c) | Validating that `activity_queues` isolates workers. Opt-in. || `references/queue-options-battery.md` | Step 9: v2 per-queue options, per-handle overrides, rate limits, worker-runtime profiles, CLI typo check (Scenarios A–F) | Validating queue options / worker-runtime profiles. Opt-in. |@@ -301,6 +305,7 @@ loaded only for the work at hand. Read them as needed:- "validate temporal" / "full temporal test" / a broad regression check → `mode-2-setup.md`, then `mode-2-tiers.md`; offer the two batteries as opt-in extras.- "validate temporal error handling" / "error report" / "error propagation" → Mode 1 Step 2b above (the precise pytest assertions) **plus** `mode-2-setup.md` + `mode-2-tiers.md` Step 5f (Tiers 13–16, cross-process).+- "cost report" / "distributed cost" / "mock inference" → `mode-2-setup.md` (split workers), then `mode-2-tiers.md` Step 5b' (Tier 8b) — `--mock-inference` makes it free and deterministic; no live spend needed.- "queue options" / "runtime profile" / "routing" → `mode-2-setup.md`, then `routing-battery.md` and/or `queue-options-battery.md`.- Anything image-related → `mode-2-setup.md` + `mode-2-tiers.md` Tiers 4–5, run **live** — dry-run cannot surface payload-size bugs.diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-setup.md b/.claude/skills/temporal-e2e-validate/references/mode-2-setup.mdindex 1b4311179..5a90ad063 100644--- a/.claude/skills/temporal-e2e-validate/references/mode-2-setup.md+++ b/.claude/skills/temporal-e2e-validate/references/mode-2-setup.md@@ -78,7 +78,7 @@ Splits responsibilities so workflows and activities run in different processes:- `router` worker registers all workflows (`WfPipeRouter`, `WfPipeRun`, …),`disable_all_activities = true`.- `runner` worker registers all activities (`act_deliver`, `act_llm_*`,- `act_assemble_graph`, `act_flush_trace_events`, …), `disable_all_workflows = true`.+ `act_assemble_tracing`, `act_flush_trace_events`, …), `disable_all_workflows = true`.This forces every activity to be picked up by a *different* Python process than theworkflow that scheduled it. The runner process never executes `WfPipeRouter`, so it@@ -91,7 +91,7 @@ dynamic concept defined in the bundle.> - `WfPipeRouter` dehydrates `pipe_output` via `prepare_for_temporal()` before> returning, so workflow-level transit carries raw dicts (no class lookup).> - `WfPipeRun` rehydrates back on the *router* (same process that loaded the crate).-> - Activities the runner actually executes in dry-run (`act_assemble_graph`,+> - Activities the runner actually executes in dry-run (`act_assemble_tracing`,> `act_flush_trace_events`) operate on raw event records — no dynamic class needed.> - `act_deliver` is **only scheduled when `delivery_assignment is not None`**> (`wf_pipe_run.py:79`), and `pipelex run bundle` does not pass one.diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.mdindex 733371f27..fb00bdace 100644--- a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md+++ b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md@@ -10,7 +10,7 @@- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)- **Step 4** — Verify graph output- **Step 5** — Concurrent isolation tests (concept / pipe / multi-concept)-- **Step 5b** — Tier 8: cross-worker usage emission+- **Step 5b** — Tier 8: cross-worker usage emission; Tier 8b: cross-worker cost report assembly (`--mock-inference`, free)- **Step 5c** — Tier 9: object generation cross-process- **Step 5d** — Tier 11: `make_extract_pages` two-activity cross-process- **Step 5e** — Tier 12: deeply-nested controller stack (CV batch screening)@@ -348,9 +348,17 @@ counterpart to the Phase 4 integration testbody — on the router process — and never dispatch `act_llm_gen_text` to therunner. So a vanilla `pipelex run bundle --temporal --dry-run` againstrouter+runner workers will emit ALL `usage_report` events with-`writer_id="primary"`, never `act_*`. To deterministically observe runner-side-fallback in dry-run, run the Phase 4 integration test, which substitutes the-inference activity with a wrapper that synthesizes a real `LLMJob` server-side:+`writer_id="primary"`, never `act_*`.++**Cheap deterministic CLI way: `--mock-inference` (Tier 8b below).** To observe+runner-side `act_*` writer files from the CLI *without LLM spend*, use+`--mock-inference`: a LIVE run (operators dispatch `act_llm_gen_text` to the+runner exactly as a real run does) whose AI calls are faked at the inference leaf+with reportable synthetic usage. This is the deterministic, free path — prefer it+over live mode for writer-id observation, and see **Tier 8b** for the full+cross-worker cost-report assertion. The two other paths below remain valid: the+Phase 4 integration test (which substitutes the inference activity with a wrapper+that synthesizes a real `LLMJob` server-side), and live mode (real spend).```bashtimeout 180 .venv/bin/pytest -x -v \@@ -397,6 +405,124 @@ own `task_queue`) so that `act_llm_gen_text` resolves to a queue the runnerlistens on, and confirm both workers are running with the latest code(restart them if they predate the Phase 2 runner-side fallback commit).+### Step 5b': Tier 8b — Cross-worker cost report assembly (`--mock-inference`, free + deterministic)++Tier 8 proves a runner-side `UsageReportEvent` *lands* in the NDJSON partition.+Tier 8b proves the next link end-to-end: those cross-worker usage events+*assemble* onto `PipeOutput.tokens_usages` (via `act_assemble_tracing` with+`assemble_usage=True`) and the submitter renders a **single** end-of-run cost+report covering usage from all workers — with **no LLM spend**.++**Why `--mock-inference` and not `--dry-run`.** Dry-run instantiates+`ContentGeneratorDry` inside the workflow body *on the router* and never+dispatches `act_llm_gen_text` to the runner (Tier 8 note), and its usage is+zero-token → the cost report is *suppressed*. `--mock-inference` keeps+`run_mode=LIVE` so operators dispatch the real `act_llm_gen_text` to the runner+exactly like a paid run; only the inference *leaf* is faked, emitting reportable+non-zero synthetic usage (model `mock_inference`). So the runner emits `act_*`+usage events AND the report is non-suppressed — the one thing dry-run can't+validate cheaply.++**Requires split router+runner workers** (Step 2 in `mode-2-setup.md`). The point+is that the usage events originate on the *runner* process and must aggregate+into the report assembled for the *router*/submitter. `--mock-inference` is on the+main `run` subcommands (pipe/method/bundle) and is mutually exclusive with+`--dry-run`.++**Primary check — both artifacts assemble from one event read (D3):**++```bash+timeout 180 .venv/bin/pipelex run bundle \+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \+ --pipe native_text_sequence \+ --temporal --mock-inference --no-logo --graph --costs 2>&1 | tail -30+echo "EXIT=$?"+```++Expect, at the end of the submitter output:++- A Rich **cost table** with model `mock_inference`, non-zero input/output token+ counts, and a run total (the synthetic per-call counts are+ `MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}`, so a+ 2-LLM-step sequence totals 200 input / 100 output).+- A `reactflow.html` for the run (graph assembles from the *same* event read).+- Exit 0.++Then confirm the usage came cross-worker, and that there is exactly **one**+report (no double-count between the fast path and the runner fallback):++```bash+RUN_ID=$(ls -t .pipelex/traces/ | head -1)+ls -la .pipelex/traces/$RUN_ID/+grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson+grep -hoE '"writer_id":"[^"]+"' .pipelex/traces/$RUN_ID/*.ndjson | sort -u+grep -hc '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson | paste -sd+ - | bc+```++Expect:++- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file (runner-side) carrying+ `usage_report` events with `writer_id` starting `act_`, alongside router-side+ `wf_*.ndjson` with `writer_id="primary"`.+- The total `usage_report` event count equals the number of LLM steps (one per+ mocked `act_llm_gen_text`) — and the rendered table's row count / total match+ it. No duplicate usage rows.++**Cross-child aggregation (A+B+C → one report).** Run a fan-out bundle so usage+from multiple child workflows aggregates into a single submitter report:++```bash+timeout 180 .venv/bin/pipelex run bundle \+ tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \+ --pipe temporal_parallel_sequence \+ --temporal --mock-inference --no-logo --graph --costs 2>&1 | tail -30+echo "EXIT=$?"+```++Expect a single end-of-run cost table whose token totals sum the parent and both+child-workflow branches — not one table per branch.++**Negative check — `--no-costs` gates costs only, graph unaffected:**++```bash+timeout 180 .venv/bin/pipelex run bundle \+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \+ --pipe native_text_sequence \+ --temporal --mock-inference --no-logo --graph --no-costs 2>&1 | tail -30+echo "EXIT=$?"+RUN_ID=$(ls -t .pipelex/traces/ | head -1)+grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson || echo "No usage_report events (expected with --no-costs)"+ls results/*/reactflow.html 2>/dev/null && echo "graph still rendered (expected)"+```++Expect: **no** cost table rendered, **no** `usage_report` events in the NDJSON+(the usage event-log isn't wired when `--no-costs`), but the `reactflow.html`+still assembles — proving `--costs` gates only the cost channel, independent of+`--graph`.++**Live arm (real spend, opt-in)** — mirrors Tier 8's live block for the+real-payload case. Drop `--mock-inference` so the real `act_llm_gen_text` runs on+the runner and bills real tokens:++```bash+timeout 600 .venv/bin/pipelex run bundle \+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \+ --pipe native_text_sequence \+ --temporal --no-logo --graph --costs 2>&1 | tail -30+```++Expect the same single end-of-run cost table, now with the real model handle and+real token counts/cost. Use this only to validate the real-payload path; the mock+arm above is the regression-friendly default.++After each run, tell the user: PASS/FAIL, whether the cost table rendered, the+distinct `writer_id` set (must include an `act_*`), the usage-event count vs the+rendered total, and the graph file path. The pytest counterpart with no spend is+`tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py`+(LIVE `native_text_sequence` with `is_mock_inference=True` through `WfPipeRun`:+the real `act_llm_gen_text` mocks the leaf and `tokens_usages` assembles back+reportable under the `mock_inference` model).+### Step 5c: Tier 9 — Object generation through Temporal cross-processValidates that `act_llm_gen_object` and `act_llm_gen_object_list` survive a@@ -865,6 +991,7 @@ ls results/*/reactflow.html| Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — || Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — || Tier 8: Cross-worker usage | Runner-side `UsageReportEvent` lands in same NDJSON dir with `act_*` writer_id (live mode or integration test) | PASS/FAIL | — | — |+| Tier 8b: Cross-worker cost report | `--mock-inference` (free): runner-side usage assembles onto `PipeOutput.tokens_usages` and the submitter renders a single non-suppressed cost report (model `mock_inference`); `--no-costs` renders none while `--graph` still assembles | PASS/FAIL | path | — || Tier 9: Object gen cross-process | `act_llm_gen_object` / `act_llm_gen_object_list` survive the JSON round-trip with nested fields intact | PASS/FAIL | path | — || Tier 10a: Multi-activity routing | `activity_queues.default` routes both `act_llm_gen_text` and `act_img_gen_images` to their dedicated worker pools; default runner sees 0 hits for either | PASS/FAIL/SKIPPED | — | — || Tier 10b: Per-handle routing | `activity_queues.by_handle` overrides the activity default per model handle — two distinct handles in one workflow land on two distinct workers | PASS/FAIL/SKIPPED | path | — |diff --git a/.claude/skills/temporal-e2e-validate/references/queue-options-battery.md b/.claude/skills/temporal-e2e-validate/references/queue-options-battery.mdindex 971635c17..4fe12da75 100644--- a/.claude/skills/temporal-e2e-validate/references/queue-options-battery.md+++ b/.claude/skills/temporal-e2e-validate/references/queue-options-battery.md@@ -106,7 +106,7 @@ sleep 4tmux capture-pane -t temporal-worker-router -p -S -30 | grep "Temporal Worker started"```-**Also keep a general `runner` on the default task queue.** Un-routed activities — tracing (`act_flush_trace_events`, `act_assemble_graph`) and `act_deliver` — are not in `activity_queues`, so they fall through to `default_task_queue` (`temporal_task_queue`). The specialized runners below only poll their own named queues, so without a general runner those activities have no poller and any `--graph` run (Scenarios A–B use `--graph`) **hangs** on the first trace flush (the activity sits `PENDING_SCHEDULED` forever). Spawn it:+**Also keep a general `runner` on the default task queue.** Un-routed activities — tracing (`act_flush_trace_events`, `act_assemble_tracing`) and `act_deliver` — are not in `activity_queues`, so they fall through to `default_task_queue` (`temporal_task_queue`). The specialized runners below only poll their own named queues, so without a general runner those activities have no poller and any `--graph` run (Scenarios A–B use `--graph`) **hangs** on the first trace flush (the activity sits `PENDING_SCHEDULED` forever). Spawn it:```bashtmux kill-session -t temporal-worker-runner 2>/dev/nulldiff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex 1c5b1ffe0..e4a2ff815 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -35,6 +35,8 @@is_normalize_data_urls_to_storage = true# Set to false to disable generation of execution graphsis_generate_graph = true+# Set to false to stop emitting usage (cost) tracing events+is_generate_usage = true# Bounded fan-out concurrency for PipeBatch: at most this many branches run at once.# Set max_concurrency = "unbounded" to disable the bound.@@ -192,7 +194,7 @@ is_reporting_enabled = true[pipelex.reporting_config]# Cost reporting settings-is_log_costs_to_console = false+is_log_costs_to_console = trueis_generate_cost_report_file_enabled = falsecost_report_dir_path = "reports"cost_report_base_name = "cost_report"diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 993e7bacf..0aa2182e3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,37 @@# Changelog+## [Unreleased]++This cycle reworks **cost reporting** so it survives distributed execution and stops leaking. Cost now rides on the run result instead of a side buffer: a multi-worker Temporal run aggregates usage from every worker into a single end-of-run report, the success-path registry leak is gone by removal, and a new `--costs` switch decouples cost collection from `--graph`.++### Added++- **`--costs` / `--no-costs` (default on).** `pipelex run pipe|method|bundle` gains a dedicated cost switch that emits usage tracing events and renders the end-of-run cost report. It rides the shared trace-event transport independently of `--graph`, so `--no-graph --costs` reports cost without building a graph and `--graph --no-costs` builds a graph with no cost report. It replaces the removed `--cost-report` (see Changed).++- **Distributed cost reporting.** In Temporal mode, usage emitted from inference activities running on separate worker processes is now aggregated into a single cost report at the submitter. Usage is assembled from the trace-event stream onto `PipeOutput` in both direct and Temporal execution, so a cross-worker run reports its true total — previously cross-worker usage was emitted but never rendered into a report.++- **`tokens_usages` on `PipeOutput`.** New `tokens_usages: list[AnyTokensUsage] | None` and `usage_assembly_error: str | None` fields, mirroring `graph_spec` / `graph_assembly_error`. Cost is now part of the run result and is exposed automatically wherever a `PipeOutput` is returned, including the Pipelex API response. Render it with `CostRegistry.generate_report(tokens_usages=...)`.++- **`--mock-inference`.** A LIVE run that fakes AI calls at the inference leaf with reportable synthetic usage: operators dispatch real activities (exercising the true distributed path) but no tokens are billed, so distributed cost reporting can be validated cheaply and deterministically. Mutually exclusive with `--dry-run`. It covers the LLM leaf; image-generation / extract / search under `--mock-inference` fail loud with `MockInferenceUnsupportedError` (pointing at `--dry-run`) rather than silently calling the real provider.++- **Cost report in the agent CLI JSON.** `pipelex-agent run ... --with-memory` attaches a best-effort structured `cost_report` (`{total_cost, by_model}`, real USD) to its JSON envelope when the run did reportable work and summary aggregation succeeds — treat it as optional (absent for dry runs, `--no-costs`, the API-runner path, or an aggregation failure). The agent surface stays JSON-only — no Rich table on stderr; compact mode is unchanged.++### Changed++- **`--cost-report` removed, folded into `--costs`.** Breaking: `--cost-report/--no-cost-report` is gone from `run pipe|method|bundle`. Use `--costs` (default on) instead.++- **Cost reporting is event-sourced; the submitter-side `UsageRegistry` is removed.** The cost report is rendered from `PipeOutput.tokens_usages`, not from an in-process registry. `ReportingProtocol` (and `ReportingNoOp`) no longer expose `open_registry` / `close_registry` / `generate_report` / `inject_tokens_usages`, and the `UsageRegistry` model is gone. Embedders that rendered cost via `get_report_delegate().generate_report()` must render from the returned `pipe_output.tokens_usages` instead.++- **`is_log_costs_to_console` now defaults `true`.** With `--costs` on, the CLI prints the cost table at end of run by default (parity with `--graph` producing visible output). Per-inference-job console logging is removed — the report renders once at the end, not per call. Library embedders who don't want console output set it `false`.++- **Dry runs never emit a cost report; free-model runs still do.** Suppression keys on whether the run did reportable work (any tokens or any cost), not on total cost alone: a dry run (zero tokens, zero cost) is suppressed, while a real run on a free / zero-price model (e.g. Ollama) still reports its token usage with a zero total.++- **Temporal `act_assemble_graph` renamed to `act_assemble_tracing`.** It now assembles both the graph (`GraphSpec`) and usage (`tokens_usages`) from a single trace-event read, gated per concern by the run's `--graph` / `--costs` flags.++### Fixed++- **`UsageRegistry` success-path leak.** The per-run cost registry was opened during run setup but closed only on the failure path, so every successful run leaked its registry — and in a long-lived process (e.g. the Pipelex API) reusing a `pipeline_run_id` could collide on the orphaned registry. The registry is removed entirely, so the leak is structurally impossible.+## [v0.31.0] - 2026-06-04This release hardens Pipelex at its edges. The headliners: a full **error-handling overhaul** that gives every error a stable, RFC 7807-shaped identity and carries it all the way out to webhooks and external surfaces; **lenient validation with pipe signatures**, so you can sketch and dry-run a whole pipeline top-down before a single pipe is implemented; and a first, **experimental cut of CSV tabular support** for reading and writing typed lists straight from `.csv`.diff --git a/docs/configuration/config-practical/reporting-config.md b/docs/configuration/config-practical/reporting-config.mdindex df1b76372..40ed31125 100644--- a/docs/configuration/config-practical/reporting-config.md+++ b/docs/configuration/config-practical/reporting-config.md@@ -15,12 +15,12 @@ The reporting configuration controls how Pipelex generates and manages cost repo### Console Logging```toml-is_log_costs_to_console = false+is_log_costs_to_console = true```-- Controls whether costs are logged to the console in **real-time**-- When enabled, displays token cost reports after each LLM operation (and other operations with costs)-- Default: `false`+- Controls whether the end-of-run cost table is printed to the console+- When enabled (and `--costs` is on), prints a single cost-by-model table after the pipeline finishes — but only when the run had reportable usage; it is suppressed for dry runs (zero tokens and zero cost) and on a usage-assembly/aggregation failure (which never fails the run)+- Default: `true` — library embedders who don't want console output can set it to `false`### Cost Report File Generationdiff --git a/docs/errors/index.md b/docs/errors/index.mdindex e2bd1ae06..1c4dd8471 100644--- a/docs/errors/index.md+++ b/docs/errors/index.md@@ -202,6 +202,14 @@ classification rules, and the cross-boundary Temporal bridge.- [`MissingDependencyError`](missing-dependency-error.md) — Missing dependency+## MockInferenceObjectFidelityError++- [`MockInferenceObjectFidelityError`](mock-inference-object-fidelity-error.md) — Mock inference object fidelity++## MockInferenceUnsupportedError++- [`MockInferenceUnsupportedError`](mock-inference-unsupported-error.md) — Mock inference unsupported+## ModelReferenceParseError- [`ModelReferenceParseError`](model-reference-parse-error.md) — Model reference parsediff --git a/docs/errors/mock-inference-object-fidelity-error.md b/docs/errors/mock-inference-object-fidelity-error.mdnew file mode 100644index 000000000..ee276318e--- /dev/null+++ b/docs/errors/mock-inference-object-fidelity-error.md@@ -0,0 +1,21 @@+---+title: "Mock inference object fidelity"+description: "Reference for the `MockInferenceObjectFidelityError` Pipelex error class."+---++<!-- pipelex:generated -->++# Mock inference object fidelity++Raised when a ``--mock-inference`` synthetic object fails re-validation against its original class.++| Field | Value |+|---|---|+| `error_type` | `MockInferenceObjectFidelityError` |+| `title` | Mock inference object fidelity |+| `type_uri` | `https://docs.pipelex.com/latest/errors/mock-inference-object-fidelity-error/` |+| `error_domain` | `input` |+| Defined in | `pipelex.cogt.content_generation.exceptions` |+| Parent class | [`PipelexError`](pipelex-error.md) |++[Back to Error Model overview](../under-the-hood/error-model.md)diff --git a/docs/errors/mock-inference-unsupported-error.md b/docs/errors/mock-inference-unsupported-error.mdnew file mode 100644index 000000000..c57e1eb2e--- /dev/null+++ b/docs/errors/mock-inference-unsupported-error.md@@ -0,0 +1,21 @@+---+title: "Mock inference unsupported"+description: "Reference for the `MockInferenceUnsupportedError` Pipelex error class."+---++<!-- pipelex:generated -->++# Mock inference unsupported++Raised when ``--mock-inference`` reaches an inference operation that has no leaf-level mock.++| Field | Value |+|---|---|+| `error_type` | `MockInferenceUnsupportedError` |+| `title` | Mock inference unsupported |+| `type_uri` | `https://docs.pipelex.com/latest/errors/mock-inference-unsupported-error/` |+| `error_domain` | `input` |+| Defined in | `pipelex.cogt.content_generation.exceptions` |+| Parent class | [`PipelexError`](pipelex-error.md) |++[Back to Error Model overview](../under-the-hood/error-model.md)diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.mdindex 694f24863..2f83b1e76 100644--- a/docs/under-the-hood/execution-graph-tracing.md+++ b/docs/under-the-hood/execution-graph-tracing.md@@ -22,7 +22,7 @@ Pipe Execution → GraphTracer → GraphSpec → Renderers → HTML/Mermaid```!!! info "Non-Intrusive Design"- Graph tracing is opt-in. When disabled, a no-op tracer is used with zero overhead. The tracer is injected via `GraphContext` in `JobMetadata`, not global state.+ Graph tracing is opt-in. When disabled, a no-op tracer is used with zero overhead. The tracer is injected via `TraceContext` in `JobMetadata`, not global state.---@@ -66,7 +66,7 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunMode# Execute with graph tracing via configrunner = PipelexRunner(- execution_config=config.with_graph_config_overrides(generate_graph=True),+ execution_config=config.with_execution_overrides(generate_graph=True),)response = await runner.execute_pipeline(pipe_code="my_pipe",@@ -76,7 +76,7 @@ pipe_output = response.pipe_output# Dry run with graph: the same runner in DRY mode with mock inputs — no separate code path.dry_runner = PipelexRunner(pipe_run_mode=PipeRunMode.DRY,- execution_config=config.with_graph_config_overrides(generate_graph=True, mock_inputs=True),+ execution_config=config.with_execution_overrides(generate_graph=True, mock_inputs=True),)response = await dry_runner.execute_pipeline(pipe_code="my_pipe")graph_spec = response.pipe_output.graph_spec@@ -111,7 +111,7 @@ flowchart TBdirection TBMGR["GraphTracerManager<br/>(singleton)"]TRACER["GraphTracer"]- CTX["GraphContext"]+ CTX["TraceContext"]MGR --> TRACERTRACER --> CTXend@@ -214,7 +214,7 @@ class GraphSpec(BaseModel):```python# 1. Manager opens tracer for pipeline runmanager = GraphTracerManager.get_or_create_instance()-graph_context = manager.open_tracer(+trace_context = manager.open_tracer(graph_id=pipeline_run_id,data_inclusion=config.data_inclusion,pipeline_ref_domain="my_domain",@@ -224,12 +224,12 @@ graph_context = manager.open_tracer(# 2. Context flows through JobMetadata to each pipejob_metadata = JobMetadata(pipeline_run_id=pipeline_run_id,- graph_context=graph_context,+ trace_context=trace_context,)# 3. Each pipe reports start/end to tracernode_id, child_context = manager.on_pipe_start(- graph_context=graph_context,+ trace_context=trace_context,pipe_code="extract_text",pipe_type="PipeExtract",node_kind=NodeKind.OPERATOR,@@ -239,7 +239,7 @@ node_id, child_context = manager.on_pipe_start(# 4. On completion, report success with outputmanager.on_pipe_end_success(- graph_id=graph_context.graph_id,+ graph_id=trace_context.graph_id,node_id=node_id,ended_at=datetime.now(timezone.utc),output_spec=IOSpec(...),@@ -249,20 +249,20 @@ manager.on_pipe_end_success(graph_spec = manager.close_tracer(pipeline_run_id)```-### GraphContext Propagation+### TraceContext Propagation-GraphContext is a serializable context that flows through pipe execution:+TraceContext is a serializable context that flows through pipe execution:```python-class GraphContext(BaseModel):+class TraceContext(BaseModel):graph_id: str # Unique graph identifierparent_node_id: str | None # Parent pipe's node IDnode_sequence: int # Counter for generating node IDsdata_inclusion: DataInclusionConfig # What data to capture- def copy_for_child(self, child_node_id: str, next_sequence: int) -> GraphContext:+ def copy_for_child(self, child_node_id: str, next_sequence: int) -> TraceContext:"""Create context for nested pipe execution."""- return GraphContext(+ return TraceContext(graph_id=self.graph_id,parent_node_id=child_node_id,node_sequence=next_sequence,@@ -479,7 +479,7 @@ validate_graphspec(graph_spec)| `pipelex/graph/graph_tracer.py` | GraphTracer implementation || `pipelex/graph/graph_tracer_manager.py` | Singleton manager for tracers || `pipelex/graph/graph_tracer_protocol.py` | Protocol + NoOp implementation |-| `pipelex/graph/graph_context.py` | Serializable tracing context |+| `pipelex/graph/trace_context.py` | Serializable tracing context || `pipelex/graph/graph_analysis.py` | Pre-computed graph analysis || `pipelex/graph/graph_factory.py` | Output generation factory || `pipelex/graph/graph_config.py` | Configuration models |diff --git a/pipelex/cli/agent_cli/CLAUDE.md b/pipelex/cli/agent_cli/CLAUDE.mdindex 36d40bbe8..e6c7975a0 100644--- a/pipelex/cli/agent_cli/CLAUDE.md+++ b/pipelex/cli/agent_cli/CLAUDE.md@@ -15,7 +15,7 @@ Only the error format is backed by a module-level `ContextVar` in `agent_output.Markdown structure per command:-- `run`: `# Pipeline run complete`, a `## Result` section (the rendered `main_stuff` markdown with `--with-memory`, otherwise the concept JSON in a fenced block), and output / graph file paths.+- `run`: `# Pipeline run complete`, a `## Result` section (the rendered `main_stuff` markdown with `--with-memory`, otherwise the concept JSON in a fenced block), and output / graph file paths. Cost is machine-first: on the local runner, when costs are on (`--costs`, the default) and the run did reportable work (any tokens or any cost), a best-effort `cost_report` object (`{total_cost, by_model: [...]}`, real USD) rides the JSON `--with-memory` envelope — there is **no** Rich cost table on stderr (unlike the human `pipelex run` CLI). The markdown renderer ignores the `cost_report` key, so it is JSON-only. A free/zero-price model still reports its token usage (with `total_cost` 0). The `cost_report` is **absent** when the run did no reportable work (dry runs), when `--no-costs` is passed, on the API-runner path (which does not assemble local usage), or if cost-summary aggregation fails (caught and skipped so it never fails an otherwise-successful run) — so consumers must treat it as optional.- `validate`: `# Validation passed`, the bundle path when relevant, and a list of validated pipes with their status. Every entry in `validated_pipes` (markdown and JSON) identifies its pipe by the namespaced `pipe_ref` (`domain.code`) — the same unambiguous identity across `validate all`, `validate bundle`, and `validate pipe`, so the same pipe is never reported under two identifiers.- `init`: `# Pipelex initialized` with target directory, enabled backends, and routing profile.- error path: `# Error: <error_type>`, the message, the hint as a `> 💡` callout, and a `## Details` section. `error_source` (internal stack frames) is omitted from markdown — it remains in the JSON envelope (`--error-format json`) for programmatic consumers.diff --git a/pipelex/cli/agent_cli/commands/run/_run_core.py b/pipelex/cli/agent_cli/commands/run/_run_core.pyindex bca0c99c4..3f629bfc4 100644--- a/pipelex/cli/agent_cli/commands/run/_run_core.py+++ b/pipelex/cli/agent_cli/commands/run/_run_core.py@@ -7,7 +7,10 @@ import shutilfrom pathlib import Pathfrom typing import Any+from pipelex import log+from pipelex.base_exceptions import PipelexErrorfrom pipelex.cli.agent_cli.commands.run._output_helpers import build_run_output+from pipelex.cogt.usage.cost_registry import CostRegistryfrom pipelex.config import get_configfrom pipelex.graph.graph_factory import generate_graph_outputs, save_graph_outputs_to_dirfrom pipelex.pipe_run.pipe_run_mode import PipeRunMode@@ -24,6 +27,7 @@ async def run_pipeline_core(mock_inputs: bool = False,library_dirs: list[str] | None = None,graph: bool = False,+ costs: bool = True,with_memory: bool = False,) -> dict[str, Any]:"""Core logic for running a pipeline and returning JSON-serializable output.@@ -37,6 +41,7 @@ async def run_pipeline_core(mock_inputs: Whether to generate mock data for missing inputs.library_dirs: List of library directories to search for pipe definitions.graph: Whether to generate execution graph visualizations.+ costs: Whether to emit usage (cost) tracing events. Default True.with_memory: Whether to include full working memory in output (True) orreturn compact concept JSON only (False, default).@@ -48,8 +53,9 @@ async def run_pipeline_core("""pipe_run_mode = PipeRunMode.DRY if dry_run else None- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=graph,+ generate_usage=costs,mock_inputs=mock_inputs or None,)@@ -99,6 +105,25 @@ async def run_pipeline_core(# JSON file and, when with_memory is True, included in the returned dict.side_effects: dict[str, Any] = {}+ # Cost report: best-effort structured `cost_report` object in the JSON output (machine-first; no Rich+ # table on stderr like the human CLI). build_cost_summary returns None for runs that did no reportable+ # work (dry runs: zero tokens and zero cost), so a free/zero-price real run yields a summary (total_cost+ # 0) while a dry run yields None. When a summary is produced it rides the side-effect envelope: on disk+ # always, and in the stdout result under --with-memory; the markdown renderer ignores the key, so it is+ # JSON-only. It is therefore absent for dry runs, --no-costs (the gate below), or an aggregation failure+ # (the guard below skips it without failing the run) — consumers treat it as optional.+ if costs and pipe_output.tokens_usages:+ # A cost-summary failure must never fail an otherwise-successful run (mirrors the main CLI's+ # render_run_cost_report guard): CostRegistry can raise CostRegistryError (a PipelexError) during+ # aggregation. Catch it, log, and skip the cost_report rather than propagate.+ try:+ cost_summary = CostRegistry.build_cost_summary(tokens_usages=pipe_output.tokens_usages)+ except PipelexError as cost_summary_error:+ log.warning(f"Cost summary generation failed (run succeeded): {cost_summary_error}")+ else:+ if cost_summary is not None:+ side_effects["cost_report"] = cost_summary+# Generate and save graph visualizations if requestedif graph and pipe_output.graph_spec:graph_config = execution_config.graph_configdiff --git a/pipelex/cli/agent_cli/commands/run/bundle_cmd.py b/pipelex/cli/agent_cli/commands/run/bundle_cmd.pyindex e4227ad85..c4ccabb7e 100644--- a/pipelex/cli/agent_cli/commands/run/bundle_cmd.py+++ b/pipelex/cli/agent_cli/commands/run/bundle_cmd.py@@ -52,6 +52,10 @@ def run_bundle_cmd(bool,typer.Option("--graph/--no-graph", help="Generate execution graph visualizations (saved alongside output)"),] = True,+ costs: Annotated[+ bool,+ typer.Option("--costs/--no-costs", help="Emit usage (cost) tracing events. Default on."),+ ] = True,library_dir: Annotated[list[str] | None,typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files)"),@@ -200,6 +204,7 @@ def run_bundle_cmd(mock_inputs=mock_inputs,library_dirs=library_dir,graph=graph,+ costs=costs,with_memory=with_memory,))diff --git a/pipelex/cli/agent_cli/commands/run/method_cmd.py b/pipelex/cli/agent_cli/commands/run/method_cmd.pyindex baf865654..4d664a2ef 100644--- a/pipelex/cli/agent_cli/commands/run/method_cmd.py+++ b/pipelex/cli/agent_cli/commands/run/method_cmd.py@@ -50,6 +50,10 @@ def run_method_cmd(bool,typer.Option("--graph/--no-graph", help="Generate execution graph visualizations (saved alongside output)"),] = True,+ costs: Annotated[+ bool,+ typer.Option("--costs/--no-costs", help="Emit usage (cost) tracing events. Default on."),+ ] = True,library_dir: Annotated[list[str] | None,typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files)"),@@ -157,6 +161,7 @@ def run_method_cmd(mock_inputs=mock_inputs,library_dirs=all_library_dirs,graph=graph,+ costs=costs,with_memory=with_memory,))diff --git a/pipelex/cli/agent_cli/commands/run/pipe_cmd.py b/pipelex/cli/agent_cli/commands/run/pipe_cmd.pyindex 58ba84ae3..80b56720c 100644--- a/pipelex/cli/agent_cli/commands/run/pipe_cmd.py+++ b/pipelex/cli/agent_cli/commands/run/pipe_cmd.py@@ -46,6 +46,10 @@ def run_pipe_cmd(bool,typer.Option("--graph/--no-graph", help="Generate execution graph visualizations (saved alongside output)"),] = True,+ costs: Annotated[+ bool,+ typer.Option("--costs/--no-costs", help="Emit usage (cost) tracing events. Default on."),+ ] = True,library_dir: Annotated[list[str] | None,typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files)"),@@ -152,6 +156,7 @@ def run_pipe_cmd(mock_inputs=mock_inputs,library_dirs=library_dir,graph=graph,+ costs=costs,with_memory=with_memory,))diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.pyindex e9b00e02e..e0feb6620 100644--- a/pipelex/cli/commands/run/_run_core.py+++ b/pipelex/cli/commands/run/_run_core.py@@ -26,12 +26,13 @@ from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceErrorfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff_viewer import render_stuff_viewerfrom pipelex.graph.graph_factory import generate_graph_outputs, save_graph_outputs_to_dir-from pipelex.hub import get_console, get_report_delegate, get_telemetry_manager+from pipelex.hub import get_console, get_telemetry_managerfrom pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityErrorfrom pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.exceptions import PipelineExecutionErrorfrom pipelex.pipeline.runner import PipelexRunner+from pipelex.reporting.cost_report_renderer import render_run_cost_reportfrom pipelex.system.runtime import IntegrationModefrom pipelex.system.telemetry.events import EventPropertyfrom pipelex.tools.misc.exceptions import JsonTypeError@@ -47,6 +48,28 @@ if TYPE_CHECKING:COMMAND = "run"+def validate_run_flag_combination(*, dry_run: bool, mock_inference: bool, mock_inputs: bool) -> None:+ """Reject illegal ``--dry-run`` / ``--mock-inference`` / ``--mock-inputs`` combinations.++ Single owner of which run-flag combinations are legal, shared by the ``pipe`` / ``method`` / ``bundle``+ run subcommands so they can't drift:++ - ``--mock-inputs`` requires ``--dry-run`` — it fills missing required inputs for the dry generator+ that ``--dry-run`` swaps in pre-dispatch; without ``--dry-run`` there is nothing for it to feed.+ - ``--mock-inference`` cannot be combined with ``--dry-run`` — ``--dry-run`` swaps the generator+ pre-dispatch so the leaf is never reached, which would silently ignore ``--mock-inference``.++ Prints the offending combination to stderr and raises ``typer.Exit(1)``; returns ``None`` when the+ combination is legal.+ """+ if mock_inputs and not dry_run:+ typer.secho("Failed to run: --mock-inputs requires --dry-run", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)+ if mock_inference and dry_run:+ typer.secho("Failed to run: --mock-inference cannot be combined with --dry-run", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)++async def _execute_run(pipe_code: str | None,bundle_path: str | None,@@ -59,9 +82,10 @@ async def _execute_run(graph_full_data: bool | None,output_dir: str,dry_run: bool,+ mock_inference: bool,mock_inputs: bool,library_dir: list[str] | None,- cost_report: bool | None,+ costs: bool | None = None,dynamic_output_concept_ref: str | None = None,*,save_csv: str | None = None,@@ -142,8 +166,9 @@ async def _execute_run(pipe_run_mode = PipeRunMode.DRY if dry_run else None# Build effective execution config with CLI overrides- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=graph,+ generate_usage=costs,force_include_full_data=graph_full_data,mock_inputs=mock_inputs or None,)@@ -152,6 +177,7 @@ async def _execute_run(runner = PipelexRunner(bundle_uris=[bundle_path] if bundle_path else None,pipe_run_mode=pipe_run_mode,+ is_mock_inference=mock_inference,execution_config=execution_config,library_dirs=library_dir,)@@ -286,19 +312,16 @@ async def _execute_run(raise typer.Exit(1) from csv_exclog.verbose(f"Main stuff CSV saved to: {save_csv}")- reporting_config = get_config().pipelex.reporting_config- # --no-cost-report (cost_report is False) skips the report entirely: no table, no CSV.- # Otherwise: console follows the flag (if given) or config; CSV follows config.- if cost_report is not False:- print_to_console = cost_report or reporting_config.is_log_costs_to_console- if print_to_console or reporting_config.is_generate_cost_report_file_enabled:- try:- get_report_delegate().generate_report(- pipeline_run_id=response.pipeline_run_id,- print_to_console=print_to_console,- )- except (OSError, PipelexError) as cost_report_error:- log.warning(f"Cost report generation failed (run succeeded): {cost_report_error}")+ # Render the end-of-run cost report from the usage assembled onto pipe_output (event-sourced),+ # gated by the resolved --costs. Channels (console / CSV) follow the reporting config (D6).+ # The renderer keeps its cost-domain name (`is_generate_costs`) on purpose; it is fed the usage+ # gate (`is_generate_usage`) — "if usage was generated, render the cost view". Do not "align" the+ # keyword to the field: the cost report is a view over usage, not the same switch.+ render_run_cost_report(+ pipeline_run_id=response.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=execution_config.is_generate_usage,+ )# Print completion recapconsole = get_console()@@ -334,9 +357,10 @@ def execute_run(graph_full_data: bool | None,output_dir: str,dry_run: bool,+ mock_inference: bool,mock_inputs: bool,library_dir: list[str] | None,- cost_report: bool | None = None,+ costs: bool | None = None,telemetry_command_label: str = COMMAND,temporal: bool | None = None,dynamic_output_concept_ref: str | None = None,@@ -367,9 +391,10 @@ def execute_run(graph_full_data=graph_full_data,output_dir=output_dir,dry_run=dry_run,+ mock_inference=mock_inference,mock_inputs=mock_inputs,library_dir=library_dir,- cost_report=cost_report,+ costs=costs,dynamic_output_concept_ref=dynamic_output_concept_ref,save_csv=save_csv,)diff --git a/pipelex/cli/commands/run/bundle_cmd.py b/pipelex/cli/commands/run/bundle_cmd.pyindex 96b30aca6..bc5bf39c5 100644--- a/pipelex/cli/commands/run/bundle_cmd.py+++ b/pipelex/cli/commands/run/bundle_cmd.py@@ -6,7 +6,7 @@ from typing import Annotatedimport typerfrom pipelex.builder.conventions import DEFAULT_BUNDLE_FILE_NAME, DEFAULT_INPUTS_FILE_NAME-from pipelex.cli.commands.run._run_core import COMMAND, execute_run+from pipelex.cli.commands.run._run_core import COMMAND, execute_run, validate_run_flag_combinationfrom pipelex.core.interpreter.helpers import MTHDS_EXTENSION, is_pipelex_file@@ -61,6 +61,13 @@ def run_bundle_cmd(bool,typer.Option("--dry-run", help="Run pipeline in dry mode (no actual inference calls)"),] = False,+ mock_inference: Annotated[+ bool,+ typer.Option(+ "--mock-inference",+ help="Live run that fakes AI calls at the inference leaf with reportable synthetic usage. Mutually exclusive with --dry-run.",+ ),+ ] = False,mock_inputs: Annotated[bool,typer.Option("--mock-inputs", help="Generate mock data for missing required inputs (requires --dry-run)"),@@ -81,11 +88,11 @@ def run_bundle_cmd(help="Concept ref (e.g. 'document_qa.ReferenceCount') used to resolve a pipe whose output is declared as 'Dynamic'.",),] = None,- cost_report: Annotated[+ costs: Annotated[bool | None,typer.Option(- "--cost-report/--no-cost-report",- help="Override config: --cost-report forces the cost table on; --no-cost-report skips reporting entirely (no table and no CSV file).",+ "--costs/--no-costs",+ help="Override config: emit usage (cost) tracing events and render the end-of-run cost report. Default on.",),] = None,save_csv: Annotated[@@ -105,14 +112,7 @@ def run_bundle_cmd(pipelex run bundle my_bundle.mthds --pipe my_pipe --inputs data.jsonpipelex run bundle pipeline_01/ --dry-run"""- # Validate --mock-inputs requires --dry-run- if mock_inputs and not dry_run:- typer.secho(- "Failed to run: --mock-inputs requires --dry-run",- fg=typer.colors.RED,- err=True,- )- raise typer.Exit(1)+ validate_run_flag_combination(dry_run=dry_run, mock_inference=mock_inference, mock_inputs=mock_inputs)pipe_code: str | None = pipebundle_path: str | None = None@@ -183,9 +183,10 @@ def run_bundle_cmd(graph_full_data=graph_full_data,output_dir=output_dir,dry_run=dry_run,+ mock_inference=mock_inference,mock_inputs=mock_inputs,library_dir=library_dir,- cost_report=cost_report,+ costs=costs,telemetry_command_label=f"{COMMAND} bundle",temporal=temporal,dynamic_output_concept_ref=dynamic_output_concept_ref,diff --git a/pipelex/cli/commands/run/method_cmd.py b/pipelex/cli/commands/run/method_cmd.pyindex 715673658..a9f901e17 100644--- a/pipelex/cli/commands/run/method_cmd.py+++ b/pipelex/cli/commands/run/method_cmd.py@@ -5,7 +5,7 @@ from typing import Annotatedimport typer-from pipelex.cli.commands.run._run_core import COMMAND, execute_run+from pipelex.cli.commands.run._run_core import COMMAND, execute_run, validate_run_flag_combinationfrom pipelex.cli.method_resolver import resolve_method_target@@ -60,6 +60,13 @@ def run_method_cmd(bool,typer.Option("--dry-run", help="Run pipeline in dry mode (no actual inference calls)"),] = False,+ mock_inference: Annotated[+ bool,+ typer.Option(+ "--mock-inference",+ help="Live run that fakes AI calls at the inference leaf with reportable synthetic usage. Mutually exclusive with --dry-run.",+ ),+ ] = False,mock_inputs: Annotated[bool,typer.Option("--mock-inputs", help="Generate mock data for missing required inputs (requires --dry-run)"),@@ -80,11 +87,11 @@ def run_method_cmd(help="Concept ref (e.g. 'document_qa.ReferenceCount') used to resolve a pipe whose output is declared as 'Dynamic'.",),] = None,- cost_report: Annotated[+ costs: Annotated[bool | None,typer.Option(- "--cost-report/--no-cost-report",- help="Override config: --cost-report forces the cost table on; --no-cost-report skips reporting entirely (no table and no CSV file).",+ "--costs/--no-costs",+ help="Override config: emit usage (cost) tracing events and render the end-of-run cost report. Default on.",),] = None,save_csv: Annotated[@@ -107,14 +114,7 @@ def run_method_cmd(pipelex run method my-method --inputs data.jsonpipelex run method my-method --dry-run"""- # Validate --mock-inputs requires --dry-run- if mock_inputs and not dry_run:- typer.secho(- "Failed to run: --mock-inputs requires --dry-run",- fg=typer.colors.RED,- err=True,- )- raise typer.Exit(1)+ validate_run_flag_combination(dry_run=dry_run, mock_inference=mock_inference, mock_inputs=mock_inputs)# Resolve method name to pipe_code and library dirspipe_code, method_library_dirs, _ = resolve_method_target(@@ -153,9 +153,10 @@ def run_method_cmd(graph_full_data=graph_full_data,output_dir=effective_output_dir,dry_run=dry_run,+ mock_inference=mock_inference,mock_inputs=mock_inputs,library_dir=effective_library_dir,- cost_report=cost_report,+ costs=costs,telemetry_command_label=f"{COMMAND} method",temporal=temporal,dynamic_output_concept_ref=dynamic_output_concept_ref,diff --git a/pipelex/cli/commands/run/pipe_cmd.py b/pipelex/cli/commands/run/pipe_cmd.pyindex fd95ae73b..db645f638 100644--- a/pipelex/cli/commands/run/pipe_cmd.py+++ b/pipelex/cli/commands/run/pipe_cmd.py@@ -5,7 +5,7 @@ from typing import Annotatedimport typer-from pipelex.cli.commands.run._run_core import COMMAND, execute_run+from pipelex.cli.commands.run._run_core import COMMAND, execute_run, validate_run_flag_combinationfrom pipelex.cli.method_resolver import resolve_pipe_from_exportsfrom pipelex.core.interpreter.helpers import MTHDS_EXTENSION, is_pipelex_file@@ -57,6 +57,13 @@ def run_pipe_cmd(bool,typer.Option("--dry-run", help="Run pipeline in dry mode (no actual inference calls)"),] = False,+ mock_inference: Annotated[+ bool,+ typer.Option(+ "--mock-inference",+ help="Live run that fakes AI calls at the inference leaf with reportable synthetic usage. Mutually exclusive with --dry-run.",+ ),+ ] = False,mock_inputs: Annotated[bool,typer.Option("--mock-inputs", help="Generate mock data for missing required inputs (requires --dry-run)"),@@ -77,11 +84,11 @@ def run_pipe_cmd(help="Concept ref (e.g. 'document_qa.ReferenceCount') used to resolve a pipe whose output is declared as 'Dynamic'.",),] = None,- cost_report: Annotated[+ costs: Annotated[bool | None,typer.Option(- "--cost-report/--no-cost-report",- help="Override config: --cost-report forces the cost table on; --no-cost-report skips reporting entirely (no table and no CSV file).",+ "--costs/--no-costs",+ help="Override config: emit usage (cost) tracing events and render the end-of-run cost report. Default on.",),] = None,save_csv: Annotated[@@ -111,14 +118,7 @@ def run_pipe_cmd()raise typer.Exit(1)- # Validate --mock-inputs requires --dry-run- if mock_inputs and not dry_run:- typer.secho(- "Failed to run: --mock-inputs requires --dry-run",- fg=typer.colors.RED,- err=True,- )- raise typer.Exit(1)+ validate_run_flag_combination(dry_run=dry_run, mock_inference=mock_inference, mock_inputs=mock_inputs)# Check installed methods' exports for additional library dirstry:@@ -148,9 +148,10 @@ def run_pipe_cmd(graph_full_data=graph_full_data,output_dir=output_dir,dry_run=dry_run,+ mock_inference=mock_inference,mock_inputs=mock_inputs,library_dir=library_dir,- cost_report=cost_report,+ costs=costs,telemetry_command_label=f"{COMMAND} pipe",temporal=temporal,dynamic_output_concept_ref=dynamic_output_concept_ref,diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.pyindex 9db7d7d43..6ad0de243 100644--- a/pipelex/cogt/content_generation/content_generator.py+++ b/pipelex/cogt/content_generation/content_generator.py@@ -1,5 +1,6 @@from typing import Any+from pydantic import BaseModel, ValidationErrorfrom typing_extensions import overridefrom pipelex import log@@ -11,6 +12,7 @@ from pipelex.cogt.content_generation.assignment_models import (TemplatingAssignment,)from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata+from pipelex.cogt.content_generation.exceptions import MockInferenceObjectFidelityErrorfrom pipelex.cogt.content_generation.extract_generate import extract_gen_pagesfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list, img_gen_single_image@@ -34,6 +36,34 @@ from pipelex.tools.misc.image_utils import ImageFormatfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar+def _revalidate_against_object_class(+ raw_obj: BaseModel,+ object_class: type[BaseModelTypeVar],+ *,+ is_mock_inference: bool,+) -> BaseModelTypeVar:+ """Re-validate a leaf-generated object's data against the original ``object_class``.++ ``llm_gen_object`` / ``llm_gen_object_list`` return plain ``BaseModel``s reconstructed from the JSON+ schema; re-validating their data against the original class makes the result the proper subtype (e.g.+ ``StructuredContent``) the caller expects.++ Under ``--mock-inference`` the object was built by polyfactory from the schema-reconstructed class, which+ can drop invariants the original class enforces (custom validators, ``json_schema_extra`` format/pattern+ hints datamodel-code-generator omits on round-trip). A re-validation failure there is the known object-mock+ fidelity gap, so the ``ValidationError`` is re-raised as a clear typed+ :class:`MockInferenceObjectFidelityError` that names the class and points at ``--dry-run``. The catch is+ scoped to the mock path only — a LIVE provider's invalid output keeps its existing ``ValidationError``.+ """+ raw_data = raw_obj.model_dump(serialize_as_any=True)+ if not is_mock_inference:+ return object_class.model_validate(raw_data)+ try:+ return object_class.model_validate(raw_data)+ except ValidationError as exc:+ raise MockInferenceObjectFidelityError.for_object_class(object_class.__name__) from exc++class ContentGenerator(ContentGeneratorProtocol):def __init__(self, generated_content_factory: GeneratedContentFactory) -> None:self._generated_content_factory = generated_content_factory@@ -79,10 +109,7 @@ class ContentGenerator(ContentGeneratorProtocol):)raw_obj = await llm_gen_object(object_assignment=object_assignment)log.verbose(f"{self.__class__.__name__} generated object direct: {raw_obj}")- # llm_gen_object returns a plain BaseModel reconstructed from the JSON schema.- # Validate its data against the original class so the result is a proper subtype- # (e.g. StructuredContent) expected by the caller.- return object_class.model_validate(raw_obj.model_dump(serialize_as_any=True))+ return _revalidate_against_object_class(raw_obj, object_class, is_mock_inference=job_metadata.is_mock_inference)@override@update_job_metadata@@ -105,7 +132,7 @@ class ContentGenerator(ContentGeneratorProtocol):)raw_list = await llm_gen_object_list(object_assignment=object_assignment)log.verbose(f"{self.__class__.__name__} generated object list direct: {raw_list}")- return [object_class.model_validate(raw_obj.model_dump(serialize_as_any=True)) for raw_obj in raw_list]+ return [_revalidate_against_object_class(raw_obj, object_class, is_mock_inference=job_metadata.is_mock_inference) for raw_obj in raw_list]@overrideasync def make_image_content(diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.pyindex a9316bf11..eea614c22 100644--- a/pipelex/cogt/content_generation/content_generator_dry.py+++ b/pipelex/cogt/content_generation/content_generator_dry.py@@ -1,11 +1,10 @@-from datetime import datetimefrom typing import Anyfrom typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata-from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory+from pipelex.cogt.content_generation.dry_mock import build_mock_object, report_dry_llm_jobfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParamsfrom pipelex.cogt.extract.extract_output import ExtractOutput, Page@@ -13,10 +12,7 @@ from pipelex.cogt.image.generated_image import GeneratedImageRawDetailsfrom pipelex.cogt.image.image_size import ImageSizefrom pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParamsfrom pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt-from pipelex.cogt.llm.llm_job import LLMJob-from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReportfrom pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TemplatingStyle@@ -25,56 +21,11 @@ from pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContentfrom pipelex.core.stuffs.text_and_images_content import TextAndImagesContentfrom pipelex.core.stuffs.text_content import TextContent-from pipelex.hub import get_report_delegatefrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsingfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVarfrom pipelex.urls import URLs-_DRY_RUN_INFERENCE_MODEL_NAME = "dry_run"-_DRY_RUN_INFERENCE_MODEL_ID = "dry_run"---def _report_dry_llm_job(job_metadata: JobMetadata, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None:- """Build a synthetic LLMJob with zero-token usage and report it.-- This makes the runner-side cross-worker emission path observable via the- existing dry-run e2e harness without forcing assertions to run live LLM.- The synthetic LLMTokensUsage carries zero counts and a "dry_run" model id- so consumers can distinguish dry-run usage from real-inference usage.-- The synthetic job_metadata copy gets completed_at set so report_inference_job- can format the duration without crashing on None — matching what- LLMJob.llm_job_after_complete does in the live path. JobMetadata.started_at- can arrive either offset-naive (from the default factory) or offset-aware- (e.g. callers in pipe_abstract.py use datetime.now(timezone.utc)), so we- build the synthetic ``now`` with the same tzinfo as the incoming started_at- — mixing aware/naive raises TypeError when JobMetadata.duration subtracts.- """- tz = job_metadata.started_at.tzinfo if job_metadata.started_at is not None else None- now = datetime.now(tz)- synthetic_metadata = job_metadata.model_copy(- update={- "started_at": job_metadata.started_at or now,- "completed_at": now,- },- )- tokens_usage = LLMTokensUsage(- job_metadata=synthetic_metadata,- inference_model_name=_DRY_RUN_INFERENCE_MODEL_NAME,- inference_model_id=_DRY_RUN_INFERENCE_MODEL_ID,- unit_costs={},- nb_tokens_by_category={},- )- synthetic_job = LLMJob(- job_metadata=synthetic_metadata,- llm_prompt=llm_prompt,- job_params=llm_setting.make_llm_job_params(),- job_config=LLMJobConfig(schema_reask_max_attempts=1),- job_report=LLMJobReport(llm_tokens_usage=tokens_usage),- )- get_report_delegate().report_inference_job(inference_job=synthetic_job)-class ContentGeneratorDry(ContentGeneratorProtocol):"""This class is used to generate mock content for testing purposes.@@ -107,7 +58,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):func_name = "make_llm_text"log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")prompt_truncated = llm_prompt_for_text.desc(truncate_text_length=self._text_gen_truncate_length)- _report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_main, llm_prompt=llm_prompt_for_text)+ report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_main, llm_prompt=llm_prompt_for_text)return f"DRY RUN: {func_name} • llm_setting={llm_setting_main.desc()} • prompt={prompt_truncated}"@override@@ -119,12 +70,10 @@ class ContentGeneratorDry(ContentGeneratorProtocol):llm_setting_for_object: LLMSetting,llm_prompt_for_object: LLMPrompt,) -> BaseModelTypeVar:- object_factory = DryRunFactory.make_dry_run_factory(object_class)- # We run validators to ensure mock data is valid. Fields with format constraints- # (snake_case, PascalCase, etc.) should have `examples` defined in their Field()- # so polyfactory uses those instead of random strings.- _report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_for_object, llm_prompt=llm_prompt_for_object)- return object_factory.build()+ report_dry_llm_job(job_metadata=job_metadata, llm_setting=llm_setting_for_object, llm_prompt=llm_prompt_for_object)+ # build_mock_object runs validators so the mock is valid; fields with format constraints+ # (snake_case, PascalCase, ...) should declare examples / mock_format in their Field().+ return build_mock_object(object_class)@override@update_job_metadatadiff --git a/pipelex/cogt/content_generation/dry_mock.py b/pipelex/cogt/content_generation/dry_mock.pynew file mode 100644index 000000000..512bb3050--- /dev/null+++ b/pipelex/cogt/content_generation/dry_mock.py@@ -0,0 +1,207 @@+"""Leaf-level inference mocking shared by ``--dry-run`` and ``--mock-inference``.++Two triggers, one mechanism. Both fake the AI call *at the cogt leaf* — the+lowest point where ``ContentGenerator`` (direct) and the Temporal activities+(``act_llm_gen_*``) converge — so a single branch covers both execution modes:++- ``--dry-run`` (``run_mode=DRY``): ``ContentGeneratorDry`` is swapped in+ pre-dispatch by each operator; it reports a **zero-token** synthetic LLM job+ via :func:`report_dry_llm_job`. Zero tokens ⇒ ``AggregatedCosts.has_reportable_usage``+ is False ⇒ the end-of-run cost report is suppressed (correct: a dry run did no+ real work).+- ``--mock-inference`` (``run_mode=LIVE`` + ``JobMetadata.is_mock_inference``):+ operators dispatch normally and the leaf itself routes to the+ :func:`mock_llm_gen_text` / :func:`mock_llm_gen_object` /+ :func:`mock_llm_gen_object_list` helpers here, which report **non-zero**+ synthetic usage via :func:`report_mock_inference_llm_job`. Non-zero tokens ⇒ a+ cost report *renders*. This is the durable reason the two modes differ at the+ reporting layer: only a non-zero mock can validate cross-worker cost-report+ rendering cheaply and deterministically (no provider spend).++This module is the single home for "what a mocked inference produces". The+follow-up that makes ``run_mode=DRY`` honor the backend+(``wip/dry-run-refactor/followup-leaf-run-mode-mock.md``) re-keys the leaf branch+from ``is_mock_inference`` to ``run_mode`` and folds ``ContentGeneratorDry`` into+these helpers — so the helpers, not the call sites, are the load-bearing piece.+"""++from datetime import datetime++from pydantic import BaseModel++from pipelex import log+from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment+from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReport+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory+from pipelex.config import get_config+from pipelex.hub import get_report_delegate+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar++# Sentinel model identifiers so a synthetic usage record is never confused with real inference.+DRY_RUN_INFERENCE_MODEL_NAME = "dry_run"+DRY_RUN_INFERENCE_MODEL_ID = "dry_run"+MOCK_INFERENCE_MODEL_NAME = "mock_inference"+MOCK_INFERENCE_MODEL_ID = "mock_inference"++# Synthetic, deterministic, clearly non-real token counts for ``--mock-inference``. Non-zero so the+# assembled usage is reportable (``AggregatedCosts.has_reportable_usage`` True → the cost report renders),+# which is exactly what ``--dry-run``'s zero-token usage deliberately suppresses. Input/output differ so+# the rendered table distinguishes the two columns. Cost stays 0 (``unit_costs={}``): a mocked run has+# token usage but no real spend — the "free model" reporting case.+MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY: NbTokensByCategoryDict = {+ TokenCategory.INPUT: 100,+ TokenCategory.OUTPUT: 50,+}+++def _report_synthetic_llm_job(+ *,+ job_metadata: JobMetadata,+ llm_setting: LLMSetting,+ llm_prompt: LLMPrompt,+ inference_model_name: str,+ inference_model_id: str,+ nb_tokens_by_category: NbTokensByCategoryDict,+) -> None:+ """Build a synthetic ``LLMJob`` and report it through ``get_report_delegate()``.++ Shared core of :func:`report_dry_llm_job` (zero tokens) and+ :func:`report_mock_inference_llm_job` (non-zero tokens). Reporting it makes the runner-side+ cross-worker emission path observable without a real LLM call.++ The synthetic ``job_metadata`` copy gets ``completed_at`` set so ``report_inference_job`` can+ format the duration without crashing on ``None`` — matching what ``LLMJob.llm_job_after_complete``+ does in the live path. ``JobMetadata.started_at`` can arrive either offset-naive (from the default+ factory) or offset-aware (e.g. ``pipe_abstract.py`` uses ``datetime.now(timezone.utc)``), so the+ synthetic ``now`` is built with the same tzinfo as the incoming ``started_at`` — mixing aware/naive+ raises ``TypeError`` when ``JobMetadata.duration`` subtracts.+ """+ tz = job_metadata.started_at.tzinfo if job_metadata.started_at is not None else None+ now = datetime.now(tz)+ synthetic_metadata = job_metadata.model_copy(+ update={+ "started_at": job_metadata.started_at or now,+ "completed_at": now,+ },+ )+ tokens_usage = LLMTokensUsage(+ job_metadata=synthetic_metadata,+ inference_model_name=inference_model_name,+ inference_model_id=inference_model_id,+ unit_costs={},+ nb_tokens_by_category=nb_tokens_by_category,+ )+ synthetic_job = LLMJob(+ job_metadata=synthetic_metadata,+ llm_prompt=llm_prompt,+ job_params=llm_setting.make_llm_job_params(),+ job_config=LLMJobConfig(schema_reask_max_attempts=1),+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),+ )+ get_report_delegate().report_inference_job(inference_job=synthetic_job)+++def report_dry_llm_job(job_metadata: JobMetadata, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None:+ """Report a zero-token synthetic LLM job for a ``--dry-run`` inference (cost report suppressed)."""+ _report_synthetic_llm_job(+ job_metadata=job_metadata,+ llm_setting=llm_setting,+ llm_prompt=llm_prompt,+ inference_model_name=DRY_RUN_INFERENCE_MODEL_NAME,+ inference_model_id=DRY_RUN_INFERENCE_MODEL_ID,+ nb_tokens_by_category={},+ )+++def report_mock_inference_llm_job(job_metadata: JobMetadata, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None:+ """Report a non-zero synthetic LLM job for a ``--mock-inference`` call (cost report renders)."""+ _report_synthetic_llm_job(+ job_metadata=job_metadata,+ llm_setting=llm_setting,+ llm_prompt=llm_prompt,+ inference_model_name=MOCK_INFERENCE_MODEL_NAME,+ inference_model_id=MOCK_INFERENCE_MODEL_ID,+ nb_tokens_by_category=dict(MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY),+ )+++def build_mock_object(model_class: type[BaseModelTypeVar]) -> BaseModelTypeVar:+ """Build one mock instance of ``model_class`` via the dry-run polyfactory.++ Runs validators so the mock is valid; fields with format constraints+ (snake_case, PascalCase, ...) should declare ``examples`` / ``mock_format`` so+ polyfactory uses those instead of random strings.+ """+ return DryRunFactory.make_dry_run_factory(model_class).build()+++def _mock_text(*, llm_prompt: LLMPrompt, llm_setting: LLMSetting) -> str:+ truncate_length = get_config().pipelex.dry_run_config.text_gen_truncate_length+ prompt_truncated = llm_prompt.desc(truncate_text_length=truncate_length)+ return f"MOCK INFERENCE • llm_setting={llm_setting.desc()} • prompt={prompt_truncated}"+++def mock_llm_gen_text(llm_assignment: LLMAssignment) -> str:+ """Leaf mock for ``llm_gen_text``: synthetic text + reportable usage, no provider call."""+ job_metadata = llm_assignment.job_metadata+ log.verbose(f"🤡 MOCK INFERENCE: llm_gen_text for '{job_metadata.pipeline_run_id}'")+ report_mock_inference_llm_job(+ job_metadata=job_metadata,+ llm_setting=llm_assignment.llm_setting,+ llm_prompt=llm_assignment.llm_prompt,+ )+ return _mock_text(llm_prompt=llm_assignment.llm_prompt, llm_setting=llm_assignment.llm_setting)+++def _reconstruct_class_and_report(object_assignment: ObjectAssignment) -> type[BaseModel]:+ """Reconstruct the object's model class from its JSON schema and report one synthetic usage event.++ Shared by both object mocks. The leaf carries only the JSON schema (not the original class), so the+ class is rebuilt via :class:`SchemaToModelFactory`. Reports exactly once — the live leaf makes one+ ``gen_object`` call (a list is one call against a list-wrapper schema), so a single ``UsageReportEvent``+ matches the real one-call topology that cross-worker assertions count.+ """+ llm_assignment = object_assignment.llm_assignment_for_object+ content_class = SchemaToModelFactory.make_from_json_schema(+ schema=object_assignment.object_class_schema,+ class_name=object_assignment.object_class_name,+ )+ report_mock_inference_llm_job(+ job_metadata=llm_assignment.job_metadata,+ llm_setting=llm_assignment.llm_setting,+ llm_prompt=llm_assignment.llm_prompt,+ )+ return content_class+++def mock_llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel:+ """Leaf mock for ``llm_gen_object``: a polyfactory-built instance of the schema model + reportable usage.++ Built from the schema-reconstructed class (the leaf carries only the JSON schema, not the original+ class), so format hints encoded via ``json_schema_extra`` that datamodel-code-generator drops on+ round-trip are not honored — exotic-format schemas may yield mock data the original class would+ reject. ``ContentGenerator.make_object`` re-validates against the original class and re-raises that+ failure as ``MockInferenceObjectFidelityError`` (review F2); the direct ``ContentGeneratorDry``+ dry-run path keeps full fidelity by building the original class. Durable fix:+ followup-leaf-run-mode-mock.md §8.+ """+ return build_mock_object(_reconstruct_class_and_report(object_assignment))+++def mock_llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseModel]:+ """Leaf mock for ``llm_gen_object_list``: ``nb_list_items`` builds + one reportable usage event."""+ item_class = _reconstruct_class_and_report(object_assignment)+ nb_list_items = get_config().pipelex.dry_run_config.nb_list_items+ # Unlike ``ContentGeneratorDry.make_object_list``, the first item's ``pipe_code`` is intentionally NOT+ # set to ``"mock_main"``: that coordination satisfies ``BundleHeaderSpec.main_pipe`` during bundle+ # dry-validation, which runs under ``run_mode=DRY`` (``ContentGeneratorDry``, never this leaf mock).+ # ``--mock-inference`` is a LIVE run and never drives bundle dry-validation, so there is no main_pipe+ # check to satisfy here.+ return [build_mock_object(item_class) for _ in range(nb_list_items)]diff --git a/pipelex/cogt/content_generation/exceptions.py b/pipelex/cogt/content_generation/exceptions.pyindex 6eafdbac0..c20877ce5 100644--- a/pipelex/cogt/content_generation/exceptions.py+++ b/pipelex/cogt/content_generation/exceptions.py@@ -1,4 +1,4 @@-from pipelex.base_exceptions import PipelexError, SecurityError+from pipelex.base_exceptions import ErrorDomain, PipelexError, SecurityErrorclass NeitherUrlNorDataError(PipelexError):@@ -7,3 +7,65 @@ class NeitherUrlNorDataError(PipelexError):class UnsafeSchemaError(SecurityError):pass+++class MockInferenceUnsupportedError(PipelexError):+ """Raised when ``--mock-inference`` reaches an inference operation that has no leaf-level mock.++ ``--mock-inference`` is honored only at the LLM leaf, which returns synthetic text/objects with+ reportable usage and never calls a provider. Image generation, document extraction, and web search+ have no such leaf mock yet, so under ``--mock-inference`` they would dispatch to the **real** provider+ and spend real money — the exact opposite of what the flag promises. This guard fails loud at those+ leaves instead of silently spending. Full no-spend coverage of every operator is available today via+ ``--dry-run``; per-operator leaf mocks land later (``wip/dry-run-refactor/followup-leaf-run-mode-mock.md``,+ via ``run_mode=DRY``).+ """++ # The caller picked an unsupported flag/operation combination they can fix (drop --mock-inference+ # for this pipe, or use --dry-run) — caller-fixable input, not a server-side fault.+ error_domain = ErrorDomain.INPUT+ # The message is pure caller-facing guidance (names the operation, points at --dry-run) with no+ # internal paths or secrets, so it should survive STRICT disclosure intact.+ _authors_caller_facing_message = True++ @classmethod+ def for_operation(cls, operation: str) -> "MockInferenceUnsupportedError":+ """Build the guard error for a named inference operation (e.g. ``"image generation (PipeImgGen)"``)."""+ return cls(+ f"--mock-inference does not support {operation}: it has no leaf-level mock, so the run would "+ "call the real provider and spend. Use --dry-run for full no-spend coverage."+ )+++class MockInferenceObjectFidelityError(PipelexError):+ """Raised when a ``--mock-inference`` synthetic object fails re-validation against its original class.++ The object leaf mock builds its instance from the **schema-reconstructed** class (the leaf carries+ only the JSON schema, not the original class). ``ContentGenerator.make_object`` then re-validates that+ data against the **original** class. Invariants the JSON-schema round-trip cannot capture — custom+ ``@field_validator`` / ``@model_validator`` logic, or format / pattern hints encoded via+ ``json_schema_extra`` that datamodel-code-generator drops — are absent from the reconstructed class, so+ polyfactory can fill a value the reconstructed class accepts but the original rejects. This turns an+ opaque mid-run ``pydantic.ValidationError`` into a clear, actionable message that names the class and+ points at the no-spend alternative. Scoped to the mock path only: a LIVE provider's invalid output keeps+ its existing ``ValidationError`` behavior. The durable fix (build against the original class in-process)+ lands with the leaf run-mode follow-up (``wip/dry-run-refactor/followup-leaf-run-mode-mock.md`` §8).+ """++ # The caller picked --mock-inference for a pipe whose output concept enforces an invariant the schema+ # round-trip drops; they can fix it by using --dry-run (full-fidelity mock) for this pipe — caller-fixable+ # input, not a server-side fault.+ error_domain = ErrorDomain.INPUT+ # The message is pure caller-facing guidance (names the caller's own output concept class, points at+ # --dry-run) with no internal paths or secrets, so it should survive STRICT disclosure intact.+ _authors_caller_facing_message = True++ @classmethod+ def for_object_class(cls, object_class_name: str) -> "MockInferenceObjectFidelityError":+ """Build the fidelity error for a named output object class (e.g. ``"InvoiceLine"``)."""+ return cls(+ f"--mock-inference built a synthetic '{object_class_name}' that failed re-validation against the "+ "original class: the mock is generated from the JSON schema, which can drop invariants the class "+ "enforces (custom validators, json_schema_extra format/pattern hints). Use --dry-run for "+ "full-fidelity mock objects (it builds the original class directly)."+ )diff --git a/pipelex/cogt/content_generation/extract_generate.py b/pipelex/cogt/content_generation/extract_generate.pyindex fd0846105..f230033a2 100644--- a/pipelex/cogt/content_generation/extract_generate.py+++ b/pipelex/cogt/content_generation/extract_generate.py@@ -1,4 +1,5 @@from pipelex.cogt.content_generation.assignment_models import ExtractAssignment+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedErrorfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.extract.extract_job_factory import ExtractJobFactoryfrom pipelex.cogt.extract.extract_output import ExtractOutput@@ -7,6 +8,9 @@ from pipelex.hub import get_extract_workerasync def extract_gen_pages(extract_assignment: ExtractAssignment) -> ExtractOutput:+ if extract_assignment.job_metadata.is_mock_inference:+ error = MockInferenceUnsupportedError.for_operation("document extraction (PipeExtract)")+ raise errorextract_worker = get_extract_worker(extract_handle=extract_assignment.extract_handle)extract_job = ExtractJobFactory.make_extract_job(extract_input=extract_assignment.extract_input,diff --git a/pipelex/cogt/content_generation/img_gen_generate.py b/pipelex/cogt/content_generation/img_gen_generate.pyindex a45ab01f4..192366d44 100644--- a/pipelex/cogt/content_generation/img_gen_generate.py+++ b/pipelex/cogt/content_generation/img_gen_generate.py@@ -1,5 +1,6 @@from pipelex import logfrom pipelex.cogt.content_generation.assignment_models import ImgGenAssignment+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedErrorfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.image.generated_image import GeneratedImageRawDetailsfrom pipelex.cogt.img_gen.img_gen_job_factory import ImgGenJobFactory@@ -8,6 +9,9 @@ from pipelex.hub import get_img_gen_workerasync def img_gen_single_image(img_gen_assignment: ImgGenAssignment) -> GeneratedImageRawDetails:+ if img_gen_assignment.job_metadata.is_mock_inference:+ error = MockInferenceUnsupportedError.for_operation("image generation (PipeImgGen)")+ raise errorimg_gen_worker = get_img_gen_worker(img_gen_handle=img_gen_assignment.img_gen_handle)img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt(img_gen_prompt=img_gen_assignment.img_gen_prompt,@@ -21,6 +25,9 @@ async def img_gen_single_image(img_gen_assignment: ImgGenAssignment) -> Generateasync def img_gen_image_list(img_gen_assignment: ImgGenAssignment) -> list[GeneratedImageRawDetails]:+ if img_gen_assignment.job_metadata.is_mock_inference:+ error = MockInferenceUnsupportedError.for_operation("image generation (PipeImgGen)")+ raise errorimg_gen_worker = get_img_gen_worker(img_gen_handle=img_gen_assignment.img_gen_handle)img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt(img_gen_prompt=img_gen_assignment.img_gen_prompt,diff --git a/pipelex/cogt/content_generation/llm_generate.py b/pipelex/cogt/content_generation/llm_generate.pyindex cddddd719..059e9e046 100644--- a/pipelex/cogt/content_generation/llm_generate.py+++ b/pipelex/cogt/content_generation/llm_generate.py@@ -4,12 +4,15 @@ from pydantic import BaseModelfrom pipelex import logfrom pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment+from pipelex.cogt.content_generation.dry_mock import mock_llm_gen_object, mock_llm_gen_object_list, mock_llm_gen_textfrom pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactoryfrom pipelex.cogt.llm.llm_job_factory import LLMJobFactoryfrom pipelex.hub import get_llm_workerasync def llm_gen_text(llm_assignment: LLMAssignment) -> str:+ if llm_assignment.job_metadata.is_mock_inference:+ return mock_llm_gen_text(llm_assignment)llm_worker = get_llm_worker(llm_handle=llm_assignment.llm_handle)llm_job = LLMJobFactory.make_llm_job(job_metadata=llm_assignment.job_metadata,@@ -23,6 +26,8 @@ async def llm_gen_text(llm_assignment: LLMAssignment) -> str:async def llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel:llm_assignment = object_assignment.llm_assignment_for_object+ if llm_assignment.job_metadata.is_mock_inference:+ return mock_llm_gen_object(object_assignment)llm_worker = get_llm_worker(llm_handle=llm_assignment.llm_handle)llm_job = LLMJobFactory.make_llm_job(job_metadata=llm_assignment.job_metadata,@@ -42,6 +47,8 @@ async def llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel:async def llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseModel]:llm_assignment = object_assignment.llm_assignment_for_object+ if llm_assignment.job_metadata.is_mock_inference:+ return mock_llm_gen_object_list(object_assignment)log.verbose(f"llm_gen_object_list to generate a list of '{object_assignment.object_class_name}'")llm_worker = get_llm_worker(llm_handle=llm_assignment.llm_handle)llm_job = LLMJobFactory.make_llm_job(diff --git a/pipelex/cogt/usage/cost_registry.py b/pipelex/cogt/usage/cost_registry.pyindex a731c6205..972b7e52d 100644--- a/pipelex/cogt/usage/cost_registry.py+++ b/pipelex/cogt/usage/cost_registry.py@@ -1,7 +1,7 @@import csvfrom collections.abc import Sequencefrom pathlib import Path-from typing import Any+from typing import Any, NamedTuplefrom pydantic import Field, RootModelfrom rich import box@@ -24,6 +24,31 @@ TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostRCostRegistryRoot = list[TokenCostReport]+class AggregatedCosts(NamedTuple):+ """One run's token usage aggregated for reporting: flat records, per-model groups, and run totals.++ ``total_cost`` is the real/unscaled USD total; ``total_nb_tokens`` is input-joined + output across all+ usages. Computing both here, once, makes this the single source of truth the table footer, the agent JSON+ summary, and the dry-run suppression gate all read from — so they can never disagree.+ """++ records: list[dict[str, Any]]+ grouped_by_model: dict[str, dict[str, float]]+ model_types: dict[str, str]+ total_cost: float+ total_nb_tokens: int++ @property+ def has_reportable_usage(self) -> bool:+ """True when the run did real work worth reporting: any tokens OR any cost.++ Dry runs emit zero-token, zero-cost synthetic usage, so this is False for them (the cost report is+ suppressed). A real run on a free/zero-price model has tokens (cost 0), so this stays True and its+ token usage is still reported — gating on cost alone would wrongly hide it.+ """+ return self.total_nb_tokens > 0 or self.total_cost > 0++class CostRegistry(RootModel[CostRegistryRoot]):root: CostRegistryRoot = Field(default_factory=empty_list_factory_of(LLMTokenCostReport))@@ -50,17 +75,39 @@ class CostRegistry(RootModel[CostRegistryRoot]):else:log.verbose(f"No report to generate for pipeline '{pipeline_run_id}'")return- cost_registry = CostRegistry()- for tokens_usage in tokens_usages:- cost_report = cls.complete_cost_report(tokens_usage=tokens_usage)- cost_registry.root.append(cost_report)+ cls.render_report(+ cls.aggregate_costs(tokens_usages=tokens_usages),+ pipeline_run_id=pipeline_run_id,+ unit_scale=unit_scale,+ cost_report_file_path=cost_report_file_path,+ print_to_console=print_to_console,+ )- records = cost_registry.to_records()+ @classmethod+ def render_report(+ cls,+ aggregated: AggregatedCosts,+ *,+ pipeline_run_id: str,+ unit_scale: float,+ cost_report_file_path: Path | None = None,+ print_to_console: bool = True,+ ) -> None:+ """Render a pre-aggregated cost report to the console (Rich table) and/or a CSV file.++ Split out of ``generate_report`` so a caller that already aggregated — the CLI renderer, which needs+ the totals to decide whether to report at all — renders from a single aggregation pass instead of+ aggregating a second time just to display.+ """+ records = aggregated.records+ grouped_by_model = aggregated.grouped_by_model+ model_types = aggregated.model_types# Use LLMTokenCostReportField for field names (same string values as ImgGenTokenCostReportField)report_field = LLMTokenCostReportField- # Calculate total costs overall+ # Per-category footer subtotals (display detail). The headline total comes from the single aggregation+ # (aggregated.total_cost) so the footer, the agent JSON, and the suppression gate share one definition.total_nb_tokens_input_cached = sum(record.get(report_field.NB_TOKENS_INPUT_CACHED, 0) for record in records)total_nb_tokens_input_non_cached = sum(record.get(report_field.NB_TOKENS_INPUT_NON_CACHED, 0) for record in records)total_nb_tokens_input_joined = sum(record.get(report_field.NB_TOKENS_INPUT_JOINED, 0) for record in records)@@ -69,48 +116,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):total_cost_input_non_cached = sum(record.get(report_field.COST_INPUT_NON_CACHED, 0) for record in records)total_cost_input_joined = sum(record.get(report_field.COST_INPUT_JOINED, 0) for record in records)total_cost_output = sum(record.get(report_field.COST_OUTPUT, 0) for record in records)- total_cost = cls.compute_total_cost(- input_non_cached_cost=total_cost_input_non_cached,- input_cached_cost=total_cost_input_cached,- output_cost=total_cost_output,- )-- # Calculate costs per model - group by model name- grouped_by_model: dict[str, dict[str, float]] = {}- model_types: dict[str, str] = {}- for record in records:- model_name = (- record.get(report_field.LLM_NAME)- or record.get(ImgGenTokenCostReportField.IMG_GEN_NAME)- or record.get(ExtractTokenCostReportField.EXTRACT_NAME)- or record.get(SearchTokenCostReportField.SEARCH_NAME, "unknown")- )- model_type = record.get(report_field.MODEL_TYPE, "llm")- model_types[model_name] = model_type- if model_name not in grouped_by_model:- grouped_by_model[model_name] = {- report_field.NB_TOKENS_INPUT_CACHED: 0,- report_field.NB_TOKENS_INPUT_NON_CACHED: 0,- report_field.NB_TOKENS_INPUT_JOINED: 0,- report_field.NB_TOKENS_OUTPUT: 0,- report_field.COST_INPUT_CACHED: 0.0,- report_field.COST_INPUT_NON_CACHED: 0.0,- report_field.COST_INPUT_JOINED: 0.0,- report_field.COST_OUTPUT: 0.0,- }-- # Aggregate values- for field in [- report_field.NB_TOKENS_INPUT_CACHED,- report_field.NB_TOKENS_INPUT_NON_CACHED,- report_field.NB_TOKENS_INPUT_JOINED,- report_field.NB_TOKENS_OUTPUT,- report_field.COST_INPUT_CACHED,- report_field.COST_INPUT_NON_CACHED,- report_field.COST_INPUT_JOINED,- report_field.COST_OUTPUT,- ]:- grouped_by_model[model_name][field] += record.get(field, 0)+ total_cost = aggregated.total_costif not grouped_by_model:msg = "Empty report aggregation by model name"@@ -207,6 +213,102 @@ class CostRegistry(RootModel[CostRegistryRoot]):def compute_total_cost(cls, input_non_cached_cost: float, input_cached_cost: float, output_cost: float) -> float:return input_non_cached_cost + input_cached_cost + output_cost+ @classmethod+ def aggregate_costs(cls, tokens_usages: Sequence[TokensUsage]) -> AggregatedCosts:+ """Aggregate token usages into flat records, per-model groups, and run totals — in one pass.++ The single source of truth for a run's reporting data. Shared by ``render_report`` (Rich table / CSV)+ and ``build_cost_summary`` (agent CLI JSON), and consulted by the CLI renderer's suppression gate via+ :attr:`AggregatedCosts.has_reportable_usage` — so the table, the JSON, and the gate never disagree on+ what a run cost or whether it is worth reporting.+ """+ cost_registry = cls()+ for tokens_usage in tokens_usages:+ cost_registry.root.append(cls.complete_cost_report(tokens_usage=tokens_usage))+ records = cost_registry.to_records()++ report_field = LLMTokenCostReportField+ grouped_by_model: dict[str, dict[str, float]] = {}+ model_types: dict[str, str] = {}+ for record in records:+ model_name = (+ record.get(report_field.LLM_NAME)+ or record.get(ImgGenTokenCostReportField.IMG_GEN_NAME)+ or record.get(ExtractTokenCostReportField.EXTRACT_NAME)+ or record.get(SearchTokenCostReportField.SEARCH_NAME, "unknown")+ )+ model_types[model_name] = record.get(report_field.MODEL_TYPE, "llm")+ if model_name not in grouped_by_model:+ grouped_by_model[model_name] = {+ report_field.NB_TOKENS_INPUT_CACHED: 0,+ report_field.NB_TOKENS_INPUT_NON_CACHED: 0,+ report_field.NB_TOKENS_INPUT_JOINED: 0,+ report_field.NB_TOKENS_OUTPUT: 0,+ report_field.COST_INPUT_CACHED: 0.0,+ report_field.COST_INPUT_NON_CACHED: 0.0,+ report_field.COST_INPUT_JOINED: 0.0,+ report_field.COST_OUTPUT: 0.0,+ }+ for field in [+ report_field.NB_TOKENS_INPUT_CACHED,+ report_field.NB_TOKENS_INPUT_NON_CACHED,+ report_field.NB_TOKENS_INPUT_JOINED,+ report_field.NB_TOKENS_OUTPUT,+ report_field.COST_INPUT_CACHED,+ report_field.COST_INPUT_NON_CACHED,+ report_field.COST_INPUT_JOINED,+ report_field.COST_OUTPUT,+ ]:+ grouped_by_model[model_name][field] += record.get(field, 0)++ total_cost = cls.compute_total_cost(+ input_non_cached_cost=sum(record.get(report_field.COST_INPUT_NON_CACHED, 0) for record in records),+ input_cached_cost=sum(record.get(report_field.COST_INPUT_CACHED, 0) for record in records),+ output_cost=sum(record.get(report_field.COST_OUTPUT, 0) for record in records),+ )+ total_nb_tokens = int(+ sum(record.get(report_field.NB_TOKENS_INPUT_JOINED, 0) for record in records)+ + sum(record.get(report_field.NB_TOKENS_OUTPUT, 0) for record in records)+ )+ return AggregatedCosts(+ records=records,+ grouped_by_model=grouped_by_model,+ model_types=model_types,+ total_cost=total_cost,+ total_nb_tokens=total_nb_tokens,+ )++ @classmethod+ def build_cost_summary(cls, tokens_usages: Sequence[TokensUsage]) -> dict[str, Any] | None:+ """Build a JSON-serializable cost summary (real/unscaled USD) for machine consumers.++ Returns ``None`` only when the run did no reportable work (a dry run: zero tokens and zero cost), so a+ real run on a free/zero-price model still reports its token usage with ``total_cost`` 0. Otherwise+ returns ``{"total_cost": float, "by_model": [{model, model_type, nb_tokens_input, nb_tokens_output, cost}, ...]}``.+ """+ aggregated = cls.aggregate_costs(tokens_usages=tokens_usages)+ if not aggregated.has_reportable_usage:+ return None++ report_field = LLMTokenCostReportField+ by_model: list[dict[str, Any]] = []+ for model_name, aggregated_data in aggregated.grouped_by_model.items():+ model_cost = cls.compute_total_cost(+ input_non_cached_cost=aggregated_data[report_field.COST_INPUT_NON_CACHED],+ input_cached_cost=aggregated_data[report_field.COST_INPUT_CACHED],+ output_cost=aggregated_data[report_field.COST_OUTPUT],+ )+ by_model.append(+ {+ "model": model_name,+ "model_type": aggregated.model_types.get(model_name, "llm"),+ "nb_tokens_input": int(aggregated_data[report_field.NB_TOKENS_INPUT_JOINED]),+ "nb_tokens_output": int(aggregated_data[report_field.NB_TOKENS_OUTPUT]),+ "cost": model_cost,+ }+ )+ return {"total_cost": aggregated.total_cost, "by_model": by_model}+@classmethoddef compute_cost_report(cls, tokens_usage: TokensUsage) -> TokenCostReport:costs_by_token_category: CostsByCategoryDict = {}diff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.pyindex e0b2ed11c..fdb55e719 100644--- a/pipelex/core/pipes/pipe_abstract.py+++ b/pipelex/core/pipes/pipe_abstract.py@@ -38,7 +38,7 @@ from pipelex.tools.misc.string_utils import is_snake_casefrom pipelex.types import Selfif TYPE_CHECKING:- from pipelex.graph.graph_context import GraphContext+ from pipelex.graph.trace_context import TraceContextfrom pipelex.libraries.library_crate import LibraryCratePipeAbstractType = type["PipeAbstract"]@@ -104,15 +104,15 @@ class PipeAbstract(ABC, BaseModel):Called by pipe subclasses during execution to capture runtime-resolved data(rendered prompts, resolved models, etc.) for the GraphSpec."""- graph_context = job_metadata.graph_context- if graph_context is None:+ trace_context = job_metadata.trace_context+ if trace_context is None:returntracer_manager = GraphTracerManager.get_instance()- if tracer_manager is None or graph_context.parent_node_id is None:+ if tracer_manager is None or trace_context.parent_node_id is None:returntracer_manager.register_execution_data(- lookup_key=graph_context.lookup_key,- node_id=graph_context.parent_node_id,+ lookup_key=trace_context.lookup_key,+ node_id=trace_context.parent_node_id,execution_data=execution_data,)@@ -464,11 +464,11 @@ class PipeAbstract(ABC, BaseModel):"""# Handle graph tracing if enabledgraph_node_id: str | None = None- child_graph_context: GraphContext | None = None+ child_trace_context: TraceContext | None = Nonetracer_manager = None- parent_graph_context = job_metadata.graph_context- if parent_graph_context is not None:+ parent_trace_context = job_metadata.trace_context+ if parent_trace_context is not None:tracer_manager = GraphTracerManager.get_instance()if tracer_manager is not None:started_at = datetime.now(timezone.utc)@@ -479,26 +479,38 @@ class PipeAbstract(ABC, BaseModel):for var_name in self.needed_inputs().required_names:stuff = working_memory.get_optional_stuff(var_name)if stuff is not None:+ # E1: gate the expensive payload serialization on emit_graph_events too — in+ # costs-only mode the GraphSpec is never assembled, so these dumps would be built+ # then discarded. The lightweight IOSpec (name/concept/content_type/digest) is kept+ # so node ids and usage-event correlation are unaffected.+ include_graph_data = parent_trace_context.emit_graph_eventsinput_spec = IOSpec(name=var_name,concept=stuff.concept.code,content_type=stuff.content.content_type,digest=stuff.stuff_code,- data=stuff.content.smart_dump() if parent_graph_context.data_inclusion.stuff_json_content else None,- data_text=stuff.content.rendered_pretty_text() if parent_graph_context.data_inclusion.stuff_text_content else None,- data_html=stuff.content.rendered_pretty_html() if parent_graph_context.data_inclusion.stuff_html_content else None,+ data=stuff.content.smart_dump()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content)+ else None,+ data_text=stuff.content.rendered_pretty_text()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_text_content)+ else None,+ data_html=stuff.content.rendered_pretty_html()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_html_content)+ else None,)input_specs.append(input_spec)- # Serialize pipe and concept data for registries if enabled+ # Serialize pipe and concept data for registries if enabled (E1: also gated on+ # emit_graph_events — the registries only feed the GraphSpec).pipe_data: dict[str, Any] | None = Noneconcept_data: list[dict[str, Any]] | None = None- if parent_graph_context.data_inclusion.pipe_and_concept_registry:+ if parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry:pipe_data = self.model_dump(mode="json")concept_data = self._make_concept_data_for_registry()- graph_node_id, child_graph_context = tracer_manager.on_pipe_start(- graph_context=parent_graph_context,+ graph_node_id, child_trace_context = tracer_manager.on_pipe_start(+ trace_context=parent_trace_context,pipe_code=self.code,pipe_type=self.type,node_kind=node_kind,@@ -509,11 +521,11 @@ class PipeAbstract(ABC, BaseModel):description=self.description,domain_code=self.domain_code,)- # Update job metadata with child graph context for nested pipes- if child_graph_context is not None:+ # Update job metadata with child trace context for nested pipes+ if child_trace_context is not None:job_metadata = job_metadata.copy_with_update(otel_context=job_metadata.otel_context,- graph_context=child_graph_context,+ trace_context=child_trace_context,)try:await self.validate_before_run(@@ -546,12 +558,12 @@ class PipeAbstract(ABC, BaseModel):# Can't be a `finally`: the success/error paths record different things and# the error path needs the exception object.# Record graph tracing error- if tracer_manager is not None and parent_graph_context is not None:+ if tracer_manager is not None and parent_trace_context is not None:error_stack: str | None = None- if parent_graph_context.data_inclusion.error_stack_traces:+ if parent_trace_context.data_inclusion.error_stack_traces:error_stack = traceback.format_exc()tracer_manager.on_pipe_end_error(- lookup_key=parent_graph_context.lookup_key,+ lookup_key=parent_trace_context.lookup_key,node_id=graph_node_id,ended_at=datetime.now(timezone.utc),error_type=type(exc).__name__,@@ -561,29 +573,35 @@ class PipeAbstract(ABC, BaseModel):raise# Record graph tracing success- if tracer_manager is not None and parent_graph_context is not None:+ if tracer_manager is not None and parent_trace_context is not None:# Capture output spec for data flow tracking# Note: main_stuff may not exist for pipes like PipeParallel with add_each_output=truemain_stuff = pipe_output.working_memory.get_optional_main_stuff()output_spec: IOSpec | None = Noneif main_stuff is not None:+ # E1: same gating as the input block — skip the discarded payload dumps in costs-only mode.+ include_graph_data = parent_trace_context.emit_graph_eventsoutput_spec = IOSpec(name=output_name or main_stuff.stuff_name or "main_stuff",concept=main_stuff.concept.code,content_type=main_stuff.content.content_type,digest=main_stuff.stuff_code,- data=main_stuff.content.smart_dump() if parent_graph_context.data_inclusion.stuff_json_content else None,- data_text=main_stuff.content.rendered_pretty_text() if parent_graph_context.data_inclusion.stuff_text_content else None,- data_html=main_stuff.content.rendered_pretty_html() if parent_graph_context.data_inclusion.stuff_html_content else None,+ data=main_stuff.content.smart_dump() if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content) else None,+ data_text=main_stuff.content.rendered_pretty_text()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_text_content)+ else None,+ data_html=main_stuff.content.rendered_pretty_html()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_html_content)+ else None,)- # Serialize output concept for registry if enabled+ # Serialize output concept for registry if enabled (E1: also gated on emit_graph_events).output_concept_data: dict[str, Any] | None = None- if parent_graph_context.data_inclusion.pipe_and_concept_registry and main_stuff is not None:+ if parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry and main_stuff is not None:output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept)tracer_manager.on_pipe_end_success(- lookup_key=parent_graph_context.lookup_key,+ lookup_key=parent_trace_context.lookup_key,node_id=graph_node_id,ended_at=datetime.now(timezone.utc),output_spec=output_spec,diff --git a/pipelex/core/pipes/pipe_output.py b/pipelex/core/pipes/pipe_output.pyindex 3c976feee..a68248f51 100644--- a/pipelex/core/pipes/pipe_output.py+++ b/pipelex/core/pipes/pipe_output.py@@ -16,6 +16,7 @@ from pipelex.core.stuffs.text_content import TextContentfrom pipelex.graph.graphspec import GraphSpecfrom pipelex.libraries.library_crate import LibraryCratefrom pipelex.pipeline.pipeline_models import SpecialPipelineId+from pipelex.reporting.reporting_types import AnyTokensUsageclass PipeOutput(PipeOutputAbstract[WorkingMemory]):@@ -24,6 +25,12 @@ class PipeOutput(PipeOutputAbstract[WorkingMemory]):pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)graph_spec: GraphSpec | None = Nonegraph_assembly_error: str | None = None+ # Token usage assembled from the trace-event stream at the end of the run (mirrors graph_spec):+ # the submitter renders the cost report from this field. None when cost reporting was off or the run+ # emitted no trace events at all; an empty list when on with events present but no inference happened.+ # usage_assembly_error mirrors graph_assembly_error.+ tokens_usages: list[AnyTokensUsage] | None = None+ usage_assembly_error: str | None = Nonedef prepare_for_temporal(self, library_crate: LibraryCrate | None = None) -> "PipeOutput":"""Dehydrate WorkingMemory to raw dict for Temporal transit.diff --git a/pipelex/graph/graph_context.py b/pipelex/graph/graph_context.pydeleted file mode 100644index 84e1b80ab..000000000--- a/pipelex/graph/graph_context.py+++ /dev/null@@ -1,63 +0,0 @@-"""GraphContext model for passing graph tracing state through JobMetadata.--This is the serializable context that flows through pipe execution,-similar to OtelContext for OpenTelemetry tracing.-"""--from pydantic import BaseModel, ConfigDict, Field--from pipelex.graph.graph_config import DataInclusionConfig---class GraphContext(BaseModel):- """Serializable context for graph tracing passed through JobMetadata.-- This context enables building a GraphSpec by tracking parent-child- relationships as pipes execute. It's designed to be serializable- for distributed environments where contextvars don't work.-- Attributes:- graph_id: Unique identifier for this execution graph (typically pipeline_run_id).- parent_node_id: The node ID of the parent pipe (None for root).- node_sequence: Monotonic counter for generating unique node IDs within this graph.- data_inclusion: Configuration controlling which data formats to capture in IOSpec fields.- """-- model_config = ConfigDict(strict=True, extra="forbid")-- graph_id: str = Field(description="Unique identifier for the execution graph")- tracer_key: str | None = Field(default=None, description="Lookup key in GraphTracerManager. Defaults to graph_id when None.")- parent_node_id: str | None = Field(default=None, description="Node ID of the parent pipe, None for root")- node_sequence: int = Field(default=0, description="Monotonic counter for generating node IDs")- data_inclusion: DataInclusionConfig = Field(description="Controls which data formats to capture")-- @property- def lookup_key(self) -> str:- """Key for looking up the tracer in GraphTracerManager."""- return self.tracer_key or self.graph_id-- def make_node_id(self) -> str:- """Generate a unique node ID within this graph.-- Returns:- A unique node ID in format "{graph_id}:node_{sequence}".- """- return f"{self.graph_id}:node_{self.node_sequence}"-- def copy_for_child(self, child_node_id: str, next_sequence: int) -> "GraphContext":- """Create a child context for a nested pipe execution.-- Args:- child_node_id: The node ID assigned to the child pipe.- next_sequence: The next sequence number for the child context.-- Returns:- A new GraphContext with updated parent_node_id and node_sequence.- """- return GraphContext(- graph_id=self.graph_id,- tracer_key=self.tracer_key,- parent_node_id=child_node_id,- node_sequence=next_sequence,- data_inclusion=self.data_inclusion,- )diff --git a/pipelex/graph/graph_rendering.py b/pipelex/graph/graph_rendering.pyindex 86b238a72..c971aabfe 100644--- a/pipelex/graph/graph_rendering.py+++ b/pipelex/graph/graph_rendering.py@@ -165,7 +165,7 @@ async def generate_graph_for_bundle("""graph_spec, pipe_code = await _dry_run_bundle(bundle_path, library_dirs)- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,mock_inputs=True,)@@ -235,7 +235,7 @@ async def generate_view_for_bundle("""graph_spec, pipe_code = await _dry_run_bundle(bundle_path, library_dirs)- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,mock_inputs=True,)diff --git a/pipelex/graph/graph_tracer.py b/pipelex/graph/graph_tracer.pyindex 0f74f8ef0..8602770a0 100644--- a/pipelex/graph/graph_tracer.py+++ b/pipelex/graph/graph_tracer.py@@ -6,7 +6,6 @@ from typing import Anyfrom typing_extensions import overridefrom pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graph_context import GraphContextfrom pipelex.graph.graph_tracer_protocol import GraphTracerProtocolfrom pipelex.graph.graphspec import (EdgeKind,@@ -21,6 +20,7 @@ from pipelex.graph.graphspec import (PipelineRef,TimingSpec,)+from pipelex.graph.trace_context import TraceContextfrom pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in __init__ annotationsfrom pipelex.tracing.trace_events import (BatchAggregateEvent,@@ -135,6 +135,10 @@ class GraphTracer(GraphTracerProtocol):# The branch_producer_node_id is snapshotted at registration time, before register_controller_output# overrides _stuff_producer_map to point branch stuff codes to the controller nodeself._parallel_combine_map: dict[str, tuple[str, list[tuple[str, str]]]] = {}+ # Whether this run emits graph (node/edge) events and assembles a GraphSpec on teardown.+ # In costs-only mode this is False: the tracer still mints node ids for the in-memory graph+ # (so usage-event node_id correlation stays valid) but teardown skips the discarded spec build.+ self._emit_graph_events: bool = True# Event log for distributed tracing (None = no event emission, direct mode)self._event_log: EventLogProtocol | None = None# "direct" for single-process mode, full Temporal workflow ID otherwise@@ -216,7 +220,9 @@ class GraphTracer(GraphTracerProtocol):event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",pipeline_run_id: str | None = None,- ) -> GraphContext:+ emit_graph_events: bool = True,+ emit_usage_events: bool = True,+ ) -> TraceContext:"""Initialize tracing for a new pipeline run.Args:@@ -229,8 +235,13 @@ class GraphTracer(GraphTracerProtocol):workflow_id: Temporal workflow ID or "direct" for single-process mode.When not "direct", node/edge IDs include the workflow_id segment.pipeline_run_id: Pipeline run ID for event emission. Required when event_log is set.+ emit_graph_events: Whether this run assembles a GraphSpec. When False (costs-only mode),+ teardown skips the discarded spec build; the returned TraceContext carries the flag.+ emit_usage_events: Whether this run emits usage (cost) events. Stamped onto the returned+ TraceContext so it is born with the correct flag."""self._is_active = True+ self._emit_graph_events = emit_graph_eventsself._graph_id = graph_idself._pipeline_ref = PipelineRef(domain=pipeline_ref_domain,@@ -252,47 +263,58 @@ class GraphTracer(GraphTracerProtocol):self._pipe_registry = {}self._concept_registry = {}- return GraphContext(+ return TraceContext(graph_id=graph_id,parent_node_id=None,node_sequence=0,data_inclusion=data_inclusion,+ emit_graph_events=emit_graph_events,+ emit_usage_events=emit_usage_events,)@overridedef teardown(self) -> GraphSpec | None:- """Finalize tracing and return the built GraphSpec."""+ """Finalize tracing and return the built GraphSpec.++ In costs-only mode (``emit_graph_events`` False) the GraphSpec is never consumed, so this+ skips the edge-correlation passes and node-spec construction entirely — close-as-cleanup — and+ returns None. The in-memory nodes accumulated during the run are simply discarded on reset.+ """if not self._is_active:return None- # Mark any still-running nodes as canceled (shouldn't happen in normal flow)- for node_data in self._nodes.values():- if node_data.status == NodeStatus.RUNNING:- node_data.status = NodeStatus.CANCELED- node_data.ended_at = datetime.now(timezone.utc)-- # Generate DATA edges by correlating input stuff_codes with producer nodes- # (must happen before setting _is_active = False since add_edge checks it)- self._generate_data_edges()- self._generate_batch_item_edges()- self._generate_batch_aggregate_edges()- self._generate_parallel_combine_edges()+ graph: GraphSpec | None = None+ if self._emit_graph_events:+ # Mark any still-running nodes as canceled (shouldn't happen in normal flow)+ for node_data in self._nodes.values():+ if node_data.status == NodeStatus.RUNNING:+ node_data.status = NodeStatus.CANCELED+ node_data.ended_at = datetime.now(timezone.utc)++ # Generate DATA edges by correlating input stuff_codes with producer nodes+ # (must happen before setting _is_active = False since add_edge checks it)+ self._generate_data_edges()+ self._generate_batch_item_edges()+ self._generate_batch_aggregate_edges()+ self._generate_parallel_combine_edges()++ self._is_active = False++ # Build the final GraphSpec+ nodes = [node_data.to_node_spec() for node_data in self._nodes.values()]++ graph = GraphSpec(+ graph_id=self._graph_id or "unknown",+ created_at=self._created_at or datetime.now(timezone.utc),+ pipeline_ref=self._pipeline_ref or PipelineRef(),+ nodes=nodes,+ edges=self._edges,+ pipe_registry=dict(self._pipe_registry),+ concept_registry=dict(self._concept_registry),+ )self._is_active = False- # Build the final GraphSpec- nodes = [node_data.to_node_spec() for node_data in self._nodes.values()]-- graph = GraphSpec(- graph_id=self._graph_id or "unknown",- created_at=self._created_at or datetime.now(timezone.utc),- pipeline_ref=self._pipeline_ref or PipelineRef(),- nodes=nodes,- edges=self._edges,- pipe_registry=dict(self._pipe_registry),- concept_registry=dict(self._concept_registry),- )-# Reset internal stateself._graph_id = Noneself._pipeline_ref = None@@ -569,7 +591,7 @@ class GraphTracer(GraphTracerProtocol):@overridedef on_pipe_start(self,- graph_context: GraphContext,+ trace_context: TraceContext,pipe_code: str,pipe_type: str,node_kind: NodeKind,@@ -579,12 +601,12 @@ class GraphTracer(GraphTracerProtocol):concept_data: list[dict[str, Any]] | None = None,description: str | None = None,domain_code: str | None = None,- ) -> tuple[str, GraphContext]:+ ) -> tuple[str, TraceContext]:"""Record the start of a pipe execution."""if not self._is_active:# Return dummy values when not active- node_id = graph_context.make_node_id()- child_context = graph_context.copy_for_child(node_id, graph_context.node_sequence + 1)+ node_id = trace_context.make_node_id()+ child_context = trace_context.copy_for_child(node_id, trace_context.node_sequence + 1)return node_id, child_context# Generate node ID (includes workflow_id in Temporal mode)@@ -597,7 +619,7 @@ class GraphTracer(GraphTracerProtocol):pipe_type=pipe_type,node_kind=node_kind,started_at=started_at,- parent_node_id=graph_context.parent_node_id,+ parent_node_id=trace_context.parent_node_id,input_specs=input_specs,description=description,domain_code=domain_code,@@ -605,7 +627,7 @@ class GraphTracer(GraphTracerProtocol):self._nodes[node_id] = node_data# Accumulate pipe and concept registry data (deduplicated)- if graph_context.data_inclusion.pipe_and_concept_registry:+ if trace_context.data_inclusion.pipe_and_concept_registry:if pipe_data is not None:pipe_ref = f"{pipe_data.get('domain_code', '')}.{pipe_data.get('code', '')}"if pipe_ref not in self._pipe_registry:@@ -626,7 +648,7 @@ class GraphTracer(GraphTracerProtocol):timestamp=started_at,sequence=self._next_event_sequence(),node_id=node_id,- parent_node_id=graph_context.parent_node_id,+ parent_node_id=trace_context.parent_node_id,pipe_code=pipe_code,pipe_type=pipe_type,node_kind=node_kind,@@ -639,15 +661,15 @@ class GraphTracer(GraphTracerProtocol):)# Add containment edge from parent if this is a child pipe- if graph_context.parent_node_id is not None:+ if trace_context.parent_node_id is not None:self.add_edge(- source_node_id=graph_context.parent_node_id,+ source_node_id=trace_context.parent_node_id,target_node_id=node_id,edge_kind=EdgeKind.CONTAINS,)# Create child context - use copy_for_child to preserve include_full_data- child_context = graph_context.copy_for_child(+ child_context = trace_context.copy_for_child(child_node_id=node_id,next_sequence=self._node_sequence,)diff --git a/pipelex/graph/graph_tracer_manager.py b/pipelex/graph/graph_tracer_manager.pyindex 10e0f30d1..b45e24786 100644--- a/pipelex/graph/graph_tracer_manager.py+++ b/pipelex/graph/graph_tracer_manager.py@@ -4,10 +4,10 @@ from datetime import datetimefrom typing import Anyfrom pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graph_context import GraphContextfrom pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graph_tracer_protocol import GraphTracerProtocolfrom pipelex.graph.graphspec import EdgeKind, GraphSpec, IOSpec, NodeKind+from pipelex.graph.trace_context import TraceContextfrom pipelex.system.registries.singleton import ABCSingletonMeta, MetaSingletonfrom pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in open_tracer signature@@ -100,7 +100,9 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):workflow_id: str = "direct",pipeline_run_id: str | None = None,tracer_key: str | None = None,- ) -> GraphContext:+ emit_graph_events: bool = True,+ emit_usage_events: bool = True,+ ) -> TraceContext:"""Create and initialize a new tracer for a pipeline run.Args:@@ -115,9 +117,14 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):tracer_key: Lookup key for the tracer in the manager's dict. Defaults to graph_id.In Temporal mode, use the workflow_id to avoid collisions when multipleworkflows share the same graph_id on the same process.+ emit_graph_events: Whether this run assembles a GraphSpec / emits graph events.+ Threaded into setup so the returned TraceContext is born with the correct flag+ (no post-hoc model_copy needed at the call site).+ emit_usage_events: Whether this run emits usage (cost) events. Threaded into setup+ so the returned TraceContext is born with the correct flag.Returns:- Initial GraphContext to pass through JobMetadata.+ Initial TraceContext to pass through JobMetadata.Raises:ValueError: If a tracer for this key already exists.@@ -130,7 +137,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):tracer = GraphTracer()self._tracers[key] = tracer- graph_context = tracer.setup(+ trace_context = tracer.setup(graph_id=graph_id,data_inclusion=data_inclusion,pipeline_ref_domain=pipeline_ref_domain,@@ -138,11 +145,13 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):event_log=event_log,workflow_id=workflow_id,pipeline_run_id=pipeline_run_id,+ emit_graph_events=emit_graph_events,+ emit_usage_events=emit_usage_events,)- # Set the tracer_key on the GraphContext so downstream lookups use the same key+ # Set the tracer_key on the TraceContext so downstream lookups use the same keyif tracer_key is not None:- graph_context = graph_context.model_copy(update={"tracer_key": tracer_key})- return graph_context+ trace_context = trace_context.model_copy(update={"tracer_key": tracer_key})+ return trace_contextdef close_tracer(self, tracer_key: str) -> GraphSpec | None:"""Finalize tracing for a specific pipeline run and return its GraphSpec.@@ -190,7 +199,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):def on_pipe_start(self,- graph_context: GraphContext,+ trace_context: TraceContext,pipe_code: str,pipe_type: str,node_kind: NodeKind,@@ -200,11 +209,11 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):concept_data: list[dict[str, Any]] | None = None,description: str | None = None,domain_code: str | None = None,- ) -> tuple[str | None, GraphContext | None]:+ ) -> tuple[str | None, TraceContext | None]:"""Record the start of a pipe execution.Args:- graph_context: Current graph context containing graph_id.+ trace_context: Current trace context containing graph_id.pipe_code: The pipe code being executed.pipe_type: The pipe type (e.g., "PipeLLM", "PipeSequence").node_kind: The kind of node (controller, operator, etc.).@@ -216,14 +225,14 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):domain_code: Optional domain code of the pipe (mirrored onto NodeSpec).Returns:- Tuple of (node_id, child_graph_context) if tracing is active, (None, None) otherwise.+ Tuple of (node_id, child_trace_context) if tracing is active, (None, None) otherwise."""- tracer = self._get_tracer(graph_context.lookup_key)+ tracer = self._get_tracer(trace_context.lookup_key)if tracer is None:return None, Nonereturn tracer.on_pipe_start(- graph_context=graph_context,+ trace_context=trace_context,pipe_code=pipe_code,pipe_type=pipe_type,node_kind=node_kind,diff --git a/pipelex/graph/graph_tracer_protocol.py b/pipelex/graph/graph_tracer_protocol.pyindex 4eb69fc90..ebdbf80e2 100644--- a/pipelex/graph/graph_tracer_protocol.py+++ b/pipelex/graph/graph_tracer_protocol.py@@ -4,8 +4,8 @@ from typing import Any, Protocolfrom typing_extensions import overridefrom pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graph_context import GraphContextfrom pipelex.graph.graphspec import EdgeKind, GraphSpec, IOSpec, NodeKind+from pipelex.graph.trace_context import TraceContextfrom pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in setup signature@@ -25,7 +25,9 @@ class GraphTracerProtocol(Protocol):event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",pipeline_run_id: str | None = None,- ) -> GraphContext:+ emit_graph_events: bool = True,+ emit_usage_events: bool = True,+ ) -> TraceContext:"""Initialize tracing for a new pipeline run.Args:@@ -36,9 +38,13 @@ class GraphTracerProtocol(Protocol):event_log: Optional event log for distributed tracing.workflow_id: Temporal workflow ID or "direct" for single-process mode.pipeline_run_id: Pipeline run ID for event emission.+ emit_graph_events: Whether graph (node/edge) trace events should be emitted for this run.+ Stamped onto the returned TraceContext so it is born with the correct flag.+ emit_usage_events: Whether usage (cost) trace events should be emitted for this run.+ Stamped onto the returned TraceContext so it is born with the correct flag.Returns:- Initial GraphContext to pass through JobMetadata.+ Initial TraceContext to pass through JobMetadata."""...@@ -52,7 +58,7 @@ class GraphTracerProtocol(Protocol):def on_pipe_start(self,- graph_context: GraphContext,+ trace_context: TraceContext,pipe_code: str,pipe_type: str,node_kind: NodeKind,@@ -62,11 +68,11 @@ class GraphTracerProtocol(Protocol):concept_data: list[dict[str, Any]] | None = None,description: str | None = None,domain_code: str | None = None,- ) -> tuple[str, GraphContext]:+ ) -> tuple[str, TraceContext]:"""Record the start of a pipe execution.Args:- graph_context: Current graph context from JobMetadata.+ trace_context: Current trace context from JobMetadata.pipe_code: The pipe code being executed.pipe_type: The pipe type (e.g., "PipeLLM", "PipeSequence").node_kind: The kind of node (controller, operator, etc.).@@ -78,7 +84,7 @@ class GraphTracerProtocol(Protocol):domain_code: Optional domain code of the pipe (mirrored onto NodeSpec).Returns:- Tuple of (node_id for this pipe, updated GraphContext for children).+ Tuple of (node_id for this pipe, updated TraceContext for children)."""...@@ -244,10 +250,14 @@ class GraphTracerNoOp(GraphTracerProtocol):event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",pipeline_run_id: str | None = None,- ) -> GraphContext:- return GraphContext(+ emit_graph_events: bool = True,+ emit_usage_events: bool = True,+ ) -> TraceContext:+ return TraceContext(graph_id=graph_id,data_inclusion=data_inclusion,+ emit_graph_events=emit_graph_events,+ emit_usage_events=emit_usage_events,)@override@@ -257,7 +267,7 @@ class GraphTracerNoOp(GraphTracerProtocol):@overridedef on_pipe_start(self,- graph_context: GraphContext,+ trace_context: TraceContext,pipe_code: str,pipe_type: str,node_kind: NodeKind,@@ -267,9 +277,9 @@ class GraphTracerNoOp(GraphTracerProtocol):concept_data: list[dict[str, Any]] | None = None,description: str | None = None,domain_code: str | None = None,- ) -> tuple[str, GraphContext]:- node_id = graph_context.make_node_id()- child_context = graph_context.copy_for_child(node_id, graph_context.node_sequence + 1)+ ) -> tuple[str, TraceContext]:+ node_id = trace_context.make_node_id()+ child_context = trace_context.copy_for_child(node_id, trace_context.node_sequence + 1)return node_id, child_context@overridediff --git a/pipelex/graph/trace_context.py b/pipelex/graph/trace_context.pynew file mode 100644index 000000000..cb6b4312b--- /dev/null+++ b/pipelex/graph/trace_context.py@@ -0,0 +1,84 @@+"""TraceContext model for passing per-execution trace state through JobMetadata.++This is the serializable context that flows through pipe execution, similar to+OtelContext for OpenTelemetry tracing. It is the shared transport for two event+streams that ride on one substrate — the per-execution node tree:++- **graph** (node/edge events) → assembled into a ``GraphSpec`` for visualization;+- **usage** (cost events) → attributed to a node via ``parent_node_id`` and rendered+ into a cost report.++Both streams hang off the same node tree, so the node-minting machinery+(``make_node_id`` / ``copy_for_child`` / ``node_sequence``) is shared, not graph-only.+The node tree is always built when either stream is on — including costs-only+(``--no-graph --costs``) mode, where usage still needs ``parent_node_id`` to attribute to.+"""++from pydantic import BaseModel, ConfigDict, Field++from pipelex.graph.graph_config import DataInclusionConfig+++class TraceContext(BaseModel):+ """Serializable per-execution trace context passed through JobMetadata.++ Tracks parent-child relationships as pipes execute, building the node tree that+ both the graph stream (edges) and the usage stream (cost attribution) read from.+ Designed to be serializable for distributed environments where contextvars don't work.++ Attributes:+ graph_id: Unique identifier for this execution trace (typically pipeline_run_id).+ parent_node_id: The node ID of the parent pipe (None for root). The shared anchor —+ read by the graph stream (edges) and the usage stream (cost attribution).+ node_sequence: Monotonic counter for generating unique node IDs within this trace.+ data_inclusion: Configuration controlling which data formats to capture in IOSpec fields.+ emit_graph_events: Whether graph (node/edge) trace events should be emitted for this run+ (driven by ``is_generate_graph``). The in-memory tracer always accumulates; this only+ gates event emission onto the shared event-log transport.+ emit_usage_events: Whether usage (cost) trace events should be emitted for this run+ (driven by ``is_generate_usage``). Independent of ``emit_graph_events`` so cost reporting+ survives ``--no-graph``.+ """++ model_config = ConfigDict(strict=True, extra="forbid")++ graph_id: str = Field(description="Unique identifier for this execution trace (typically the pipeline run id)")+ tracer_key: str | None = Field(default=None, description="Lookup key in GraphTracerManager. Defaults to graph_id when None.")+ parent_node_id: str | None = Field(default=None, description="Node ID of the parent pipe, None for root")+ node_sequence: int = Field(default=0, description="Monotonic counter for generating node IDs")+ data_inclusion: DataInclusionConfig = Field(description="Controls which data formats to capture")+ emit_graph_events: bool = Field(default=True, description="Whether to emit graph (node/edge) trace events")+ emit_usage_events: bool = Field(default=True, description="Whether to emit usage (cost) trace events")++ @property+ def lookup_key(self) -> str:+ """Key for looking up the tracer in GraphTracerManager."""+ return self.tracer_key or self.graph_id++ def make_node_id(self) -> str:+ """Generate a unique node ID within this trace.++ Returns:+ A unique node ID in format "{graph_id}:node_{sequence}".+ """+ return f"{self.graph_id}:node_{self.node_sequence}"++ def copy_for_child(self, child_node_id: str, next_sequence: int) -> "TraceContext":+ """Create a child context for a nested pipe execution.++ Args:+ child_node_id: The node ID assigned to the child pipe.+ next_sequence: The next sequence number for the child context.++ Returns:+ A new TraceContext with updated parent_node_id and node_sequence.+ """+ return TraceContext(+ graph_id=self.graph_id,+ tracer_key=self.tracer_key,+ parent_node_id=child_node_id,+ node_sequence=next_sequence,+ data_inclusion=self.data_inclusion,+ emit_graph_events=self.emit_graph_events,+ emit_usage_events=self.emit_usage_events,+ )diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex 1c5b1ffe0..e4a2ff815 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -35,6 +35,8 @@is_normalize_data_urls_to_storage = true# Set to false to disable generation of execution graphsis_generate_graph = true+# Set to false to stop emitting usage (cost) tracing events+is_generate_usage = true# Bounded fan-out concurrency for PipeBatch: at most this many branches run at once.# Set max_concurrency = "unbounded" to disable the bound.@@ -192,7 +194,7 @@ is_reporting_enabled = true[pipelex.reporting_config]# Cost reporting settings-is_log_costs_to_console = false+is_log_costs_to_console = trueis_generate_cost_report_file_enabled = falsecost_report_dir_path = "reports"cost_report_base_name = "cost_report"diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.pyindex be325a645..8d39a381b 100644--- a/pipelex/pipe_controllers/batch/pipe_batch.py+++ b/pipelex/pipe_controllers/batch/pipe_batch.py@@ -189,13 +189,13 @@ class PipeBatch(PipeController):)# Register batch item extraction with graph tracer- if job_metadata.graph_context is not None:+ if job_metadata.trace_context is not None:tracer_manager = GraphTracerManager.get_instance()if tracer_manager is not None:# Pass this PipeBatch's node_id so BATCH_ITEM edges can source from the controller- batch_controller_node_id = job_metadata.graph_context.parent_node_id+ batch_controller_node_id = job_metadata.trace_context.parent_node_idtracer_manager.register_batch_item_extraction(- lookup_key=job_metadata.graph_context.lookup_key,+ lookup_key=job_metadata.trace_context.lookup_key,list_stuff_code=input_stuff.stuff_code,item_stuff_code=branch_input_item_code,item_index=branch_index,@@ -222,16 +222,16 @@ class PipeBatch(PipeController):)# Register batch aggregation with graph tracer- if job_metadata.graph_context is not None:+ if job_metadata.trace_context is not None:tracer_manager = GraphTracerManager.get_instance()if tracer_manager is not None:- # Pass the PipeBatch's node_id (from graph_context.parent_node_id) so that+ # Pass the PipeBatch's node_id (from trace_context.parent_node_id) so that# BATCH_AGGREGATE edges correctly target this PipeBatch node, not a parent# controller that may later finish and register as producer of the same output- batch_controller_node_id = job_metadata.graph_context.parent_node_id+ batch_controller_node_id = job_metadata.trace_context.parent_node_idfor agg_index, item_stuff_code in enumerate(branch_output_stuff_codes):tracer_manager.register_batch_aggregation(- lookup_key=job_metadata.graph_context.lookup_key,+ lookup_key=job_metadata.trace_context.lookup_key,output_list_stuff_code=output_stuff.stuff_code,item_stuff_code=item_stuff_code,item_index=agg_index,diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex 323889e89..a821c0c78 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -333,14 +333,19 @@ class PipeParallel(PipeController):PipeParallel to downstream consumers.Args:- job_metadata: The job metadata containing graph context.+ job_metadata: The job metadata containing trace context.output_stuffs: Mapping of output_name to the branch output Stuff."""- graph_context = job_metadata.graph_context- if graph_context is None:+ trace_context = job_metadata.trace_context+ if trace_context is None:+ return+ # E1: in costs-only mode the GraphSpec is never assembled, so skip the controller-output+ # registration entirely — its IOSpec payloads (smart_dump / rendered_pretty_*) would be built+ # then discarded. The node ids minted by on_pipe_start are unaffected.+ if not trace_context.emit_graph_events:returntracer_manager = GraphTracerManager.get_instance()- if tracer_manager is None or graph_context.parent_node_id is None:+ if tracer_manager is None or trace_context.parent_node_id is None:returnfor output_name_key, output_stuff in output_stuffs.items():output_spec = IOSpec(@@ -348,13 +353,13 @@ class PipeParallel(PipeController):concept=output_stuff.concept.code,content_type=output_stuff.content.content_type,digest=output_stuff.stuff_code,- data=output_stuff.content.smart_dump() if graph_context.data_inclusion.stuff_json_content else None,- data_text=output_stuff.content.rendered_pretty_text() if graph_context.data_inclusion.stuff_text_content else None,- data_html=output_stuff.content.rendered_pretty_html() if graph_context.data_inclusion.stuff_html_content else None,+ data=output_stuff.content.smart_dump() if trace_context.data_inclusion.stuff_json_content else None,+ data_text=output_stuff.content.rendered_pretty_text() if trace_context.data_inclusion.stuff_text_content else None,+ data_html=output_stuff.content.rendered_pretty_html() if trace_context.data_inclusion.stuff_html_content else None,)tracer_manager.register_controller_output(- lookup_key=graph_context.lookup_key,- node_id=graph_context.parent_node_id,+ lookup_key=trace_context.lookup_key,+ node_id=trace_context.parent_node_id,output_spec=output_spec,)@@ -370,22 +375,22 @@ class PipeParallel(PipeController):are merged into the combined output.Args:- job_metadata: The job metadata containing graph context.+ job_metadata: The job metadata containing trace context.combined_stuff: The combined output Stuff.branch_stuffs: Mapping of output_name to the branch output Stuff."""- graph_context = job_metadata.graph_context- if graph_context is None:+ trace_context = job_metadata.trace_context+ if trace_context is None:returntracer_manager = GraphTracerManager.get_instance()- if tracer_manager is None or graph_context.parent_node_id is None:+ if tracer_manager is None or trace_context.parent_node_id is None:returnbranch_stuff_codes = [stuff.stuff_code for stuff in branch_stuffs.values()]tracer_manager.register_parallel_combine(- lookup_key=graph_context.lookup_key,+ lookup_key=trace_context.lookup_key,combined_stuff_code=combined_stuff.stuff_code,branch_stuff_codes=branch_stuff_codes,- parallel_controller_node_id=graph_context.parent_node_id,+ parallel_controller_node_id=trace_context.parent_node_id,)@overridediff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.pyindex 132e2be50..7c79fba07 100644--- a/pipelex/pipe_operators/search/pipe_search.py+++ b/pipelex/pipe_operators/search/pipe_search.py@@ -4,6 +4,7 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.content_generation.dry_run_factory import DryRunFactory+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedErrorfrom pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.models.model_deck_check import check_search_choice_with_deck@@ -82,6 +83,13 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeSearchOutput:+ # Web search has no leaf-level mock and (unlike LLM/img-gen/extract) no cogt content_generation+ # leaf — its provider spend happens right here. Under --mock-inference (run_mode stays LIVE) this+ # live path runs, so guard here: fail loud rather than silently hit the real search provider.+ if job_metadata.is_mock_inference:+ error = MockInferenceUnsupportedError.for_operation("web search (PipeSearch)")+ raise error+# 0. Log the search runsearch_choice_desc = self.search_choice or "default"log.dev(f"✨ PipeSearch '{self.code}' running with search choice '{search_choice_desc}' ✨")diff --git a/pipelex/pipe_run/dry_run_pipeline.py b/pipelex/pipe_run/dry_run_pipeline.pyindex ed07f16d3..2b378f0f9 100644--- a/pipelex/pipe_run/dry_run_pipeline.py+++ b/pipelex/pipe_run/dry_run_pipeline.py@@ -62,7 +62,7 @@ async def dry_run_pipeline(pipe_code: str = main_pipe_code- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,mock_inputs=True,)diff --git a/pipelex/pipe_run/graph_assembly.py b/pipelex/pipe_run/graph_assembly.pydeleted file mode 100644index 14dd5db54..000000000--- a/pipelex/pipe_run/graph_assembly.py+++ /dev/null@@ -1,67 +0,0 @@-"""Shared graph assembly logic for PipeRun implementations.--After pipe execution, assembles the full GraphSpec from trace events.-Works with any event log backend (NDJSON, DynamoDB, etc.) — the backend-is determined by the tracing config, not by the caller.-"""--import json--from pydantic import ValidationError--from pipelex import log-from pipelex.base_exceptions import PipelexConfigError-from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.graph.graphspec import PipelineRef-from pipelex.system.exceptions import MissingDependencyError-from pipelex.tracing.event_log_factory import make_event_log-from pipelex.tracing.graphspec_assembler import GraphSpecAssembler---def assemble_graph_on_output(- pipe_output: PipeOutput,- pipeline_run_id: str,- domain_code: str | None = None,- main_pipe_code: str | None = None,-) -> None:- """Assemble the full graph from trace events and set it on pipe_output.-- Reads all events for the pipeline_run_id from the configured event log- backend and uses GraphSpecAssembler to build the complete cross-worker graph.- Falls back to whatever graph_spec is already on pipe_output if assembly fails.-- Tracing is observability and treated as best-effort: runtime I/O issues,- malformed event data, and broken tracing infrastructure (config errors,- missing optional dependencies) are caught and warned about, not propagated.- Programming bugs (KeyError, AttributeError, etc.) propagate so they surface- during development.-- Args:- pipe_output: The pipe output to set graph_spec on.- pipeline_run_id: The pipeline run ID to query events for.- domain_code: Domain code for the pipeline ref.- main_pipe_code: Main pipe code for the pipeline ref.- """- tracing_config = get_config().pipelex.tracing_config- if not tracing_config.is_enabled:- return-- try:- event_log = make_event_log(tracing_config)- try:- events = event_log.read_events(pipeline_run_id)- if events:- pipe_output.graph_spec = GraphSpecAssembler.assemble(- events=events,- graph_id=pipeline_run_id,- pipeline_ref=PipelineRef(- domain=domain_code,- main_pipe=main_pipe_code,- ),- )- log.debug(f"Graph assembled from {len(events)} events for pipeline_run_id={pipeline_run_id}")- finally:- event_log.close()- except (OSError, json.JSONDecodeError, ValidationError, PipelexConfigError, MissingDependencyError) as graph_assembly_error:- log.warning(f"Graph assembly failed, using existing graph: {graph_assembly_error}")diff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.pyindex b1810df8c..906888a70 100644--- a/pipelex/pipe_run/pipe_run.py+++ b/pipelex/pipe_run/pipe_run.py@@ -10,8 +10,8 @@ from pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatusfrom pipelex.pipe_run.delivery_executor import DeliveryExecutorfrom pipelex.pipe_run.exceptions import DeliveryError-from pipelex.pipe_run.graph_assembly import assemble_graph_on_outputfrom pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol+from pipelex.pipe_run.tracing_assembly import assemble_tracing_on_outputif TYPE_CHECKING:from pipelex.core.pipes.pipe_output import PipeOutput@@ -67,10 +67,17 @@ class PipeRun(PipeRunProtocol):f"Suppressed tracer close error: {tracer_close_error}")- if pipe_output is not None:- assemble_graph_on_output(+ # Assemble graph and/or usage onto pipe_output from the single trace-event read. The two+ # concerns are gated independently: graph events feed the GraphSpec (so a costs-only run does+ # not set an empty GraphSpec, preserving the --no-graph contract), usage events feed the+ # tokens_usages the submitter renders the cost report from.+ trace_context = pipe_job.job_metadata.trace_context+ if pipe_output is not None and trace_context is not None and (trace_context.emit_graph_events or trace_context.emit_usage_events):+ assemble_tracing_on_output(pipe_output=pipe_output,pipeline_run_id=pipeline_run_id,+ assemble_graph=trace_context.emit_graph_events,+ assemble_usage=trace_context.emit_usage_events,domain_code=pipe_job.pipe.domain_code,main_pipe_code=pipe_job.pipe.code,)diff --git a/pipelex/pipe_run/tracing_assembly.py b/pipelex/pipe_run/tracing_assembly.pynew file mode 100644index 000000000..5b6bef963--- /dev/null+++ b/pipelex/pipe_run/tracing_assembly.py@@ -0,0 +1,183 @@+"""Shared tracing-assembly logic for PipeRun implementations.++After pipe execution, reads the trace-event stream once and assembles both the+full GraphSpec (for ``--graph``) and the aggregated token usage (for ``--costs``)+from it. Works with any event log backend (NDJSON, DynamoDB, etc.) — the backend+is determined by the tracing config, not by the caller.++The same artifacts ride back on :class:`pipelex.core.pipes.pipe_output.PipeOutput`+in both DIRECT and TEMPORAL modes; :class:`TracingAssembly` is the small carrier+that crosses the Temporal activity boundary (returned by ``act_assemble_tracing``).+"""++import json+from typing import cast++from pydantic import BaseModel, ConfigDict, ValidationError++from pipelex import log+from pipelex.base_exceptions import PipelexConfigError+from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.graph.graphspec import GraphSpec, PipelineRef+from pipelex.reporting.reporting_types import AnyTokensUsage+from pipelex.system.exceptions import MissingDependencyError+from pipelex.tracing.event_log_factory import make_event_log+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler+from pipelex.tracing.usage_aggregator import UsageAggregator++# DynamoDB-backend read failures come in two sibling botocore base classes (neither subclasses the other):+# ClientError (service-side throttle / auth) and BotoCoreError (transport / credential / timeout, e.g.+# EndpointConnectionError, ReadTimeoutError, NoCredentialsError). Both are imported together (present or+# absent together) and join the best-effort read catch when boto3 is installed — matching s3_storage_provider.+try:+ from botocore.exceptions import BotoCoreError as _BotoCoreError # type: ignore[import-untyped]+ from botocore.exceptions import ClientError as _BotoClientError # type: ignore[import-untyped]+except ImportError:+ _BotoCoreError = None # type: ignore[assignment, misc]+ _BotoClientError = None # type: ignore[assignment, misc]+++class TracingAssembly(BaseModel):+ """Artifacts assembled from a single read of the trace-event stream.++ Mirrors the tracing fields on :class:`PipeOutput`. ``graph_spec`` /+ ``tokens_usages`` are populated only for the concerns that were requested and+ succeeded; the ``*_error`` fields carry a best-effort failure note otherwise.+ Returned by both the DIRECT assembly helper and the Temporal+ ``act_assemble_tracing`` activity, so it must stay serializable across the+ Temporal boundary (it is — ``AnyTokensUsage`` already crosses it via+ ``UsageReportEvent``).+ """++ model_config = ConfigDict(extra="forbid")++ graph_spec: GraphSpec | None = None+ graph_assembly_error: str | None = None+ tokens_usages: list[AnyTokensUsage] | None = None+ usage_assembly_error: str | None = None+++def assemble_tracing(+ pipeline_run_id: str,+ *,+ assemble_graph: bool,+ assemble_usage: bool,+ domain_code: str | None = None,+ main_pipe_code: str | None = None,+) -> TracingAssembly:+ """Read the trace events for ``pipeline_run_id`` once and assemble the requested artifacts.++ Feeds the single event read into ``GraphSpecAssembler`` (when ``assemble_graph``)+ and ``UsageAggregator`` (when ``assemble_usage``).++ Tracing is observability and treated as best-effort: runtime I/O issues+ (including the DynamoDB backend's botocore ``ClientError`` — service-side+ throttle / auth — and ``BotoCoreError`` — transport / credential / timeout —+ when boto3 is installed), malformed event data, and broken tracing+ infrastructure (config errors, missing optional dependencies) are caught and+ recorded in the ``*_error`` fields, not propagated. Programming bugs (KeyError,+ AttributeError, etc.) propagate so they surface during development.++ Args:+ pipeline_run_id: The pipeline run ID to query events for.+ assemble_graph: Whether to assemble a GraphSpec from the events.+ assemble_usage: Whether to aggregate token usage from the events.+ domain_code: Domain code for the graph's pipeline ref.+ main_pipe_code: Main pipe code for the graph's pipeline ref.++ Returns:+ A TracingAssembly carrying whichever artifacts were requested and succeeded.+ """+ result = TracingAssembly()+ tracing_config = get_config().pipelex.tracing_config+ if not tracing_config.is_enabled:+ return result+ if not (assemble_graph or assemble_usage):+ return result++ # Best-effort read: a backend failure degrades to an *_assembly_error note rather than failing the run.+ # Both botocore base classes (ClientError + BotoCoreError) join the catch together when boto3 is installed.+ read_errors: tuple[type[BaseException], ...] = (OSError, json.JSONDecodeError, ValidationError, PipelexConfigError, MissingDependencyError)+ if _BotoClientError is not None and _BotoCoreError is not None:+ # cast: botocore is untyped, so the imported classes are Unknown; we know they are BaseException types.+ read_errors = (*read_errors, cast("type[BaseException]", _BotoClientError), cast("type[BaseException]", _BotoCoreError))++ try:+ event_log = make_event_log(tracing_config)+ try:+ events = event_log.read_events(pipeline_run_id)+ finally:+ event_log.close()+ except read_errors as read_error:+ message = f"Tracing assembly failed to read events for pipeline_run_id={pipeline_run_id}: {read_error}"+ log.warning(message)+ if assemble_graph:+ result.graph_assembly_error = message+ if assemble_usage:+ result.usage_assembly_error = message+ return result++ if not events:+ return result++ if assemble_usage:+ result.tokens_usages = UsageAggregator.aggregate(events)++ if assemble_graph:+ try:+ result.graph_spec = GraphSpecAssembler.assemble(+ events=events,+ graph_id=pipeline_run_id,+ pipeline_ref=PipelineRef(+ domain=domain_code,+ main_pipe=main_pipe_code,+ ),+ )+ log.debug(f"Graph assembled from {len(events)} events for pipeline_run_id={pipeline_run_id}")+ except ValidationError as validation_error:+ message = f"Graph assembly failed for pipeline_run_id={pipeline_run_id}: {validation_error}"+ log.warning(message)+ result.graph_assembly_error = message++ return result+++def assemble_tracing_on_output(+ pipe_output: PipeOutput,+ pipeline_run_id: str,+ *,+ assemble_graph: bool,+ assemble_usage: bool,+ domain_code: str | None = None,+ main_pipe_code: str | None = None,+) -> None:+ """Assemble graph and/or usage from trace events and set them on pipe_output (DIRECT mode).++ Thin wrapper over :func:`assemble_tracing` that applies the assembled artifacts onto the+ PipeOutput. Only sets a field when its artifact was produced, so existing values are left+ untouched on a no-op / failed read.++ Args:+ pipe_output: The pipe output to set graph_spec / tokens_usages on.+ pipeline_run_id: The pipeline run ID to query events for.+ assemble_graph: Whether to assemble and set the GraphSpec.+ assemble_usage: Whether to aggregate and set token usage.+ domain_code: Domain code for the graph's pipeline ref.+ main_pipe_code: Main pipe code for the graph's pipeline ref.+ """+ result = assemble_tracing(+ pipeline_run_id=pipeline_run_id,+ assemble_graph=assemble_graph,+ assemble_usage=assemble_usage,+ domain_code=domain_code,+ main_pipe_code=main_pipe_code,+ )+ if result.graph_spec is not None:+ pipe_output.graph_spec = result.graph_spec+ if result.graph_assembly_error is not None:+ pipe_output.graph_assembly_error = result.graph_assembly_error+ if result.tokens_usages is not None:+ pipe_output.tokens_usages = result.tokens_usages+ if result.usage_assembly_error is not None:+ pipe_output.usage_assembly_error = result.usage_assembly_errordiff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 61c5290ba..5f416afff 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -57,7 +57,7 @@ default_directory_base_name = "pipeline"is_reporting_enabled = true[pipelex.reporting_config]-is_log_costs_to_console = false+is_log_costs_to_console = trueis_generate_cost_report_file_enabled = falsecost_report_dir_path = "reports"cost_report_base_name = "cost_report"@@ -294,6 +294,7 @@ pipe_stack_limit = 20is_normalize_data_urls_to_storage = trueis_mock_inputs = falseis_generate_graph = true+is_generate_usage = true# Bounded fan-out concurrency for PipeBatch: at most this many branches run at once.# A modest default keeps a large workload (one pipe over many items) from overwhelmingdiff --git a/pipelex/pipeline/bundle_validator.py b/pipelex/pipeline/bundle_validator.pyindex 42773ec34..67a89302e 100644--- a/pipelex/pipeline/bundle_validator.py+++ b/pipelex/pipeline/bundle_validator.py@@ -42,7 +42,6 @@ from pipelex.hub import (get_current_library_id_or_none,get_library_manager,get_pipe_library,- get_report_delegate,get_telemetry_manager,set_current_library,)@@ -55,7 +54,7 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipe_signature.exceptions import SignaturesNotAllowedErrorfrom pipelex.pipe_signature.signature_walk import collect_signature_paths, collect_signature_refsfrom pipelex.pipeline.execution_seams import acquire_library, prepare_pipe_job-from pipelex.pipeline.pipeline_models import SpecialPipelineId+from pipelex.pipeline.pipeline_factory import PipelineFactoryfrom pipelex.system.configuration.configs import PipelineExecutionConfigfrom pipelex.system.telemetry.events import EventName, EventPropertyfrom pipelex.system.telemetry.otel_constants import OTelConstants@@ -225,20 +224,21 @@ class BundleValidator:# 3. One validation telemetry event per sweep (relocated from the CLI's _validate_core).get_telemetry_manager().track_event(event_name=EventName.PIPE_DRY_RUN, properties={EventProperty.NB_PIPES: len(sweepable_pipes)})- # 4. The dry-run sweep. The DRY leaf emits a synthetic zero-token LLM report, so open ONE- # report registry for the whole sweep and close it in `finally`: the registry is keyed by- # the constant DRY_RUN_UNTITLED id, which would collide ("already exists") on a second sweep- # if left open. Mock inputs are built by prepare_pipe_job from this DRY + is_mock_inputs config.- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ # 4. The dry-run sweep. Each pipe is dry-run under a UNIQUE per-sweep pipeline run id (a+ # `dry_run_`-prefixed uuid, not a constant — self-describing if it ever surfaces in a log).+ # The DRY leaf emits a synthetic zero-token LLM report; with the live registry gone (usage now+ # rides on PipeOutput) the sweep accumulates no per-run state on the process-global reporting+ # manager, so overlapping sweeps (e.g. concurrent `/validate` API requests) cannot collide.+ # Mock inputs are built by prepare_pipe_job from this DRY + is_mock_inputs config.+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,mock_inputs=True,)- get_report_delegate().open_registry(pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)- try:- for pipe in sweepable_pipes:- results[pipe.pipe_ref] = await self._classify_pipe(pipe=pipe, library_id=library_id, execution_config=execution_config)- finally:- get_report_delegate().close_registry(pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)+ dry_run_pipeline_id = f"dry_run_{PipelineFactory.make_pipeline_run_id()}"+ for pipe in sweepable_pipes:+ results[pipe.pipe_ref] = await self._classify_pipe(+ pipe=pipe, library_id=library_id, execution_config=execution_config, dry_run_pipeline_id=dry_run_pipeline_id+ )# 5. Aggregate + report.return self._aggregate(results=results, start_time=start_time)@@ -274,7 +274,9 @@ class BundleValidator:dep_paths=all_dep_paths,)- async def _classify_pipe(self, *, pipe: PipeAbstract, library_id: str, execution_config: PipelineExecutionConfig) -> DryRunOutput:+ async def _classify_pipe(+ self, *, pipe: PipeAbstract, library_id: str, execution_config: PipelineExecutionConfig, dry_run_pipeline_id: str+ ) -> DryRunOutput:"""Build the mock job and run the pipe DRY through the direct primitive; classify the outcome.Wraps **both** the mock-input build (``prepare_pipe_job``) and the run in one try, catching the@@ -289,7 +291,7 @@ class BundleValidator:library_id=library_id,execution_config=execution_config,pipe_run_mode=PipeRunMode.DRY,- pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED,+ pipeline_run_id=dry_run_pipeline_id,user_id=OTelConstants.DEFAULT_USER_ID,)await self._pipe_run.run(pipe_job)diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.pyindex ca8ccef40..841702a8d 100644--- a/pipelex/pipeline/execution_seams.py+++ b/pipelex/pipeline/execution_seams.py@@ -49,7 +49,7 @@ from pipelex.tools.misc.file_utils import reject_bare_str_or_pathif TYPE_CHECKING:from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint- from pipelex.graph.graph_context import GraphContext+ from pipelex.graph.trace_context import TraceContextdef acquire_library(@@ -164,12 +164,13 @@ async def prepare_pipe_job(user_id: str,inputs: PipelineInputs | WorkingMemory | None = None,search_domain_codes: list[str] | None = None,- graph_context: "GraphContext | None" = None,+ trace_context: "TraceContext | None" = None,otel_context: OtelContext | None = None,output_name: str | None = None,output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_ref: str | None = None,request_id: str | None = None,+ is_mock_inference: bool = False,) -> PipeJob:"""Build a :class:`PipeJob` for ``pipe`` against an already-open library.@@ -177,8 +178,11 @@ async def prepare_pipe_job(``execution_config.is_mock_inputs``, optional data-url normalization), runparams, job metadata, and the library crate. Performs no pipeline-managerregistration, no report-registry open, no telemetry, no graph-tracer open,- and no library mutation. ``graph_context`` / ``otel_context`` are created by- the caller and threaded onto the job metadata.+ and no library mutation. ``trace_context`` / ``otel_context`` are created by+ the caller and threaded onto the job metadata. ``is_mock_inference`` (the+ ``--mock-inference`` trigger) is the single-writer point onto+ :attr:`JobMetadata.is_mock_inference` — a LIVE run whose LLM inference-leaf calls are+ faked (non-LLM leaves — image-gen / extract / search — raise ``MockInferenceUnsupportedError``)."""working_memory: WorkingMemory | None = None@@ -218,8 +222,9 @@ async def prepare_pipe_job(user_id=user_id,pipeline_run_id=pipeline_run_id,otel_context=otel_context,- graph_context=graph_context,+ trace_context=trace_context,request_id=request_id,+ is_mock_inference=is_mock_inference,)pipe_run_params = PipeRunParamsFactory.make_run_params(diff --git a/pipelex/pipeline/job_metadata.py b/pipelex/pipeline/job_metadata.pyindex f266e8293..5cbeacb5b 100644--- a/pipelex/pipeline/job_metadata.py+++ b/pipelex/pipeline/job_metadata.py@@ -3,7 +3,7 @@ from typing import Anyfrom pydantic import BaseModel, Field-from pipelex.graph.graph_context import GraphContext+from pipelex.graph.trace_context import TraceContextfrom pipelex.system.telemetry.otel_context import OtelContextfrom pipelex.types import StrEnum@@ -69,13 +69,25 @@ class JobMetadata(BaseModel):# OTel context with precomputed trace/span IDs. None when telemetry is disabled.otel_context: OtelContext | None = None- # GraphSpec tracing context. None when graph tracing is disabled.- graph_context: GraphContext | None = None+ # Per-execution trace context carrying the shared node tree for both the graph+ # (node/edge) and usage (cost) event streams. None when neither stream is enabled.+ trace_context: TraceContext | None = Nonecontent_generation_job_id: str | None = Noneunit_job_id: UnitJobId | None = Nonejob_category: JobCategory | None = None+ # Interim ``--mock-inference`` trigger: a LIVE run (``run_mode`` stays LIVE, so operators+ # dispatch normally) whose AI calls are faked at the cogt leaf instead of hitting a provider.+ # Lives here — not on ``PipeRunParams`` — because ``PipeRunParams.run_mode`` does not reach the+ # leaf, whereas ``JobMetadata`` rides into every assignment and crosses the Temporal boundary, so+ # the leaf branch (``llm_generate.py``) and the activity body both see it. Single-writer:+ # ``prepare_pipe_job`` off the CLI flag. The leaf emits synthetic *non-zero* usage so a cost+ # report renders (unlike ``run_mode=DRY`` whose zero-token usage is suppressed) — this is the+ # cheap, deterministic cross-worker cost-report validation affordance. See+ # ``cogt/content_generation/dry_mock.py`` and ``wip/dry-run-refactor/followup-leaf-run-mode-mock.md``.+ is_mock_inference: bool = False+started_at: datetime | None = Field(default_factory=datetime.now)completed_at: datetime | None = None@@ -88,7 +100,7 @@ class JobMetadata(BaseModel):def copy_with_update(self,otel_context: OtelContext | None,- graph_context: GraphContext | None = None,+ trace_context: TraceContext | None = None,**updates: Any,) -> "JobMetadata":"""Create a copy of this metadata with updates applied.@@ -97,17 +109,17 @@ class JobMetadata(BaseModel):otel_context: OTel context to set on the copy. Always set explicitlybecause it's computed fresh per pipe run and should replace the parent's context(even when None, e.g. in dry mode or when tracing is disabled).- graph_context: GraphSpec tracing context to set on the copy. If None,- inherits from the current context (unlike otel_context).+ trace_context: Per-execution trace context (graph + usage streams) to set on+ the copy. If None, inherits from the current context (unlike otel_context).**updates: Fields to update on the copy."""- # graph_context defaults to current value if not provided (inheritance)- effective_graph_context = graph_context if graph_context is not None else self.graph_context+ # trace_context defaults to current value if not provided (inheritance)+ effective_trace_context = trace_context if trace_context is not None else self.trace_contextreturn self.model_copy(deep=True,update={"otel_context": otel_context,- "graph_context": effective_graph_context,+ "trace_context": effective_trace_context,**updates,},)diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.pyindex b08973446..98828d501 100644--- a/pipelex/pipeline/pipeline_run_setup.py+++ b/pipelex/pipeline/pipeline_run_setup.py@@ -34,7 +34,7 @@ from pipelex.tracing.event_log_factory import make_event_logif TYPE_CHECKING:from pipelex.core.pipes.pipe_abstract import PipeAbstract- from pipelex.graph.graph_context import GraphContext+ from pipelex.graph.trace_context import TraceContextfrom pipelex.tracing.event_log_protocol import EventLogProtocol@@ -50,6 +50,7 @@ async def pipeline_run_setup(output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_ref: str | None = None,pipe_run_mode: PipeRunMode | None = None,+ is_mock_inference: bool = False,search_domain_codes: list[str] | None = None,user_id: str | None = None,pipeline_run_id: str | None = None,@@ -103,6 +104,12 @@ async def pipeline_run_setup(Pipe run mode: ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``. If not specified,inferred from the environment variable ``PIPELEX_FORCE_DRY_RUN_MODE``. Defaultsto ``PipeRunMode.LIVE`` if the environment variable is not set.+ is_mock_inference:+ The ``--mock-inference`` trigger: keep ``run_mode`` LIVE (operators dispatch+ normally) but fake every AI call at the cogt leaf. Threaded onto+ :attr:`JobMetadata.is_mock_inference`, which reaches the leaf in both direct and+ Temporal modes. Distinct from ``--dry-run``: it exercises the live control flow and+ emits reportable (non-zero) synthetic usage so a cost report renders.search_domain_codes:List of domain codes to search for pipes. The executed pipe's domain is automaticallyadded if not already present.@@ -151,9 +158,8 @@ async def pipeline_run_setup()library_manager = get_library_manager()- graph_context: GraphContext | None = None+ trace_context: TraceContext | None = Noneevent_log: EventLogProtocol | None = None- registry_opened = Falsesuccess = Falsetry:# Resolve the pipe to execute against the now-open library.@@ -180,23 +186,35 @@ async def pipeline_run_setup(if pipe.domain_code not in search_domain_codes:search_domain_codes.insert(0, pipe.domain_code)- # Initialize graph tracing if requested (after pipe is loaded so we have domain info)- if execution_config.is_generate_graph:- # Create event log when tracing is enabled+ # Initialize the tracing context if graph OR cost reporting is requested (after pipe is loaded so we+ # have domain info). The two concerns share one event-log transport and one in-memory tracer; the+ # booleans on TraceContext (emit_graph_events / emit_usage_events) say which event stream to emit.+ is_generate_graph = execution_config.is_generate_graph+ is_generate_usage = execution_config.is_generate_usage+ if is_generate_graph or is_generate_usage:+ # Create the event log when tracing is enabled — it is the shared transport for both graph+ # (node/edge) events and usage (cost) events.config = get_config()tracing_config = config.pipelex.tracing_configif tracing_config.is_enabled:event_log = make_event_log(tracing_config)graph_tracer_manager = GraphTracerManager.get_or_create_instance()- graph_context = graph_tracer_manager.open_tracer(+ # The emit flags (D4) are threaded into open_tracer so the returned TraceContext is born with+ # the correct values — no post-hoc model_copy. Propagated to child contexts via copy_for_child.+ trace_context = graph_tracer_manager.open_tracer(graph_id=pipeline_run_id,data_inclusion=execution_config.graph_config.data_inclusion,pipeline_ref_domain=pipe.domain_code,pipeline_ref_main_pipe=pipe_code,- event_log=event_log,+ # D5: in costs-only mode (--no-graph --costs) pass event_log=None so the tracer accumulates+ # the in-memory graph and keeps minting node ids, but emits NO graph events. Usage events+ # are wired separately via set_event_log below.+ event_log=event_log if is_generate_graph else None,workflow_id="direct",pipeline_run_id=pipeline_run_id,+ emit_graph_events=is_generate_graph,+ emit_usage_events=is_generate_usage,)# TODO: rethink this, it's not forcing@@ -206,11 +224,11 @@ async def pipeline_run_setup(else:pipe_run_mode = PipeRunMode.LIVE- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)- registry_opened = True-- # Set event log on the report delegate for distributed usage event emission- if event_log is not None:+ # Register the event log on the report delegate for usage event emission — only when cost reporting+ # is on. In graph-only mode (--graph --no-costs) the tracer owns the event_log for graph events, but+ # no usage-event context is registered, so usage events are suppressed (the runner fallback also+ # gates on emit_usage_events).+ if is_generate_usage and event_log is not None:get_report_delegate().set_event_log(context_key=pipeline_run_id,event_log=event_log,@@ -240,11 +258,12 @@ async def pipeline_run_setup(library_id=library_id,execution_config=execution_config,pipe_run_mode=pipe_run_mode,+ is_mock_inference=is_mock_inference,pipeline_run_id=pipeline_run_id,user_id=user_id,inputs=inputs,search_domain_codes=search_domain_codes,- graph_context=graph_context,+ trace_context=trace_context,otel_context=otel_context,output_name=output_name,output_multiplicity=output_multiplicity,@@ -263,22 +282,16 @@ async def pipeline_run_setup(finally:if not success:# Error-path-only cleanup for failures after the library was acquired: tear down the graph- # tracer, event-log state, the report registry, and the library, and restore the outer- # current-library, then let the exception propagate. Uses try/finally (not except) so a- # BaseException — e.g. asyncio.CancelledError — is covered too. acquire_library owns teardown- # for load-time failures (before this try); this block owns the post-acquire window.- if graph_context is not None:+ # tracer, event-log state, and the library, and restore the outer current-library, then let+ # the exception propagate. Uses try/finally (not except) so a BaseException — e.g.+ # asyncio.CancelledError — is covered too. acquire_library owns teardown for load-time+ # failures (before this try); this block owns the post-acquire window.+ if trace_context is not None:tracer_manager = GraphTracerManager.get_instance()if tracer_manager is not None:tracer_manager.close_tracer(pipeline_run_id)if event_log is not None:get_report_delegate().clear_event_log(context_key=pipeline_run_id)- # Close the per-run registry only if open_registry actually ran. This finally also runs for- # failures *before* open_registry; the registry_opened guard is a clean semantic gate (close- # only the registry we opened). close_registry is itself idempotent via pop(..., None), so the- # guard is about intent, not KeyError avoidance.- if registry_opened:- get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)# Restore the caller's outer current-library FIRST so the guarantee survives a teardown raise,# then tear the library down — mirroring validate_bundle. set_current_library cannot take None,# so route the "no outer was set" case through clear_current_library. The `!= library_id` guarddiff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex cdb4149ee..17ef3791c 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -54,10 +54,12 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):def __init__(self,+ *,library_id: str | None = None,library_dirs: list[str] | None = None,bundle_uris: list[str] | None = None,pipe_run_mode: PipeRunMode | None = None,+ is_mock_inference: bool = False,search_domain_codes: list[str] | None = None,user_id: str | None = None,execution_config: PipelineExecutionConfig | None = None,@@ -67,6 +69,7 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):self.library_dirs = library_dirsself.bundle_uris = bundle_urisself.pipe_run_mode = pipe_run_mode+ self.is_mock_inference = is_mock_inferenceself.search_domain_codes = search_domain_codesself.user_id = user_idself.execution_config = execution_config@@ -152,6 +155,7 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):output_multiplicity=output_multiplicity,dynamic_output_concept_ref=dynamic_output_concept_ref,pipe_run_mode=self.pipe_run_mode,+ is_mock_inference=self.is_mock_inference,search_domain_codes=self.search_domain_codes,user_id=self.user_id,)@@ -200,9 +204,10 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):msg = f"Input validation failed for '{model_name}': {formatted_error}"raise PipeExecutionError(message=msg) from excfinally:- # Close graph tracer if it was opened (cleanup only — the PipeRun is- # responsible for capturing the graph spec on pipe_output)- if execution_config.is_generate_graph and pipeline_run_id is not None:+ # Close the tracer if it was opened (cleanup only — the PipeRun is responsible for capturing+ # the graph spec on pipe_output). The tracer is opened whenever graph OR cost reporting is on+ # (costs-only mode opens it with event_log=None), so the close gate must match that condition.+ if (execution_config.is_generate_graph or execution_config.is_generate_usage) and pipeline_run_id is not None:tracer_manager = GraphTracerManager.get_instance()if tracer_manager is not None:tracer_manager.close_tracer(pipeline_run_id)diff --git a/pipelex/reporting/cost_report_renderer.py b/pipelex/reporting/cost_report_renderer.pynew file mode 100644index 000000000..3269f776c--- /dev/null+++ b/pipelex/reporting/cost_report_renderer.py@@ -0,0 +1,82 @@+"""Render the end-of-run cost report from usage assembled on the pipeline output.++The cost report is sourced from the ``tokens_usages`` list that rides back on ``PipeOutput``+(assembled from the trace-event stream at the end of the run), not from a live per-process+registry. The **main** CLI (``pipelex run``) calls this to render the console table / CSV; the+**agent** CLI does not — it builds a machine-readable ``cost_report`` object via+``CostRegistry.build_cost_summary`` for its JSON envelope instead (no Rich table on its surface).+"""++from collections.abc import Sequence+from pathlib import Path++from pipelex import log+from pipelex.base_exceptions import PipelexError+from pipelex.cogt.usage.cost_registry import CostRegistry+from pipelex.config import get_config+from pipelex.reporting.reporting_types import AnyTokensUsage+from pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path+++def render_run_cost_report(+ *,+ pipeline_run_id: str,+ tokens_usages: Sequence[AnyTokensUsage] | None,+ is_generate_costs: bool,+) -> None:+ """Render the cost report from a run's assembled token usage.++ No-op unless cost collection was on (``is_generate_costs``), the run actually assembled usage+ (``tokens_usages`` is a non-empty list), AND the run did real work+ (:attr:`~pipelex.cogt.usage.cost_registry.AggregatedCosts.has_reportable_usage` — any tokens or any cost).+ The work guard suppresses dry runs (which emit zero-token, zero-cost synthetic usage) while still+ reporting real runs on free/zero-price models, whose tokens matter even though they cost nothing —+ gating on cost alone would wrongly hide them. The two channels follow D6, each applied only when costs+ are on:++ - console (Rich table): gated by ``reporting_config.is_log_costs_to_console``+ - CSV file: gated by ``reporting_config.is_generate_cost_report_file_enabled``++ A cost-report failure never fails an otherwise-successful run: the aggregation, the CSV directory/path+ setup, and the render all run inside one try, so ``OSError`` (directory create / CSV write),+ ``UnicodeEncodeError`` (a record string the CSV's UTF-8 encoder can't represent, e.g. a lone surrogate),+ and ``PipelexError`` (e.g. ``CostRegistryError``) are caught and logged as a warning rather than propagating.++ Args:+ pipeline_run_id: The run whose usage is being reported (titles the table, names nothing else).+ tokens_usages: The usage records assembled onto ``pipe_output.tokens_usages``.+ is_generate_costs: The resolved ``--costs`` gate for this run.+ """+ if not is_generate_costs or not tokens_usages:+ return++ reporting_config = get_config().pipelex.reporting_config+ print_to_console = reporting_config.is_log_costs_to_console+ is_csv_enabled = reporting_config.is_generate_cost_report_file_enabled+ if not (print_to_console or is_csv_enabled):+ return++ try:+ # One aggregation pass feeds both the suppression decision and the render (no double aggregation).+ aggregated = CostRegistry.aggregate_costs(tokens_usages)+ if not aggregated.has_reportable_usage:+ return++ cost_report_file_path: Path | None = None+ if is_csv_enabled:+ ensure_path(Path(reporting_config.cost_report_dir_path))+ cost_report_file_path = get_incremental_file_path(+ base_path=Path(reporting_config.cost_report_dir_path),+ base_name=reporting_config.cost_report_base_name,+ extension=reporting_config.cost_report_extension,+ )++ CostRegistry.render_report(+ aggregated,+ pipeline_run_id=pipeline_run_id,+ unit_scale=reporting_config.cost_report_unit_scale,+ cost_report_file_path=cost_report_file_path,+ print_to_console=print_to_console,+ )+ except (OSError, UnicodeEncodeError, PipelexError) as cost_report_error:+ log.warning(f"Cost report generation failed (run succeeded): {cost_report_error}")diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 30c3ffeca..50e6607de 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -1,39 +1,54 @@-from collections.abc import Sequencefrom datetime import datetime, timezone-from pathlib import Path-from typing import NamedTuple+from typing import NamedTuple, cast-from pydantic import Field, RootModelfrom typing_extensions import overridefrom pipelex import logfrom pipelex.base_exceptions import PipelexConfigError-from pipelex.cogt.exceptions import ReportingManagerErrorfrom pipelex.cogt.extract.extract_job import ExtractJobfrom pipelex.cogt.img_gen.img_gen_job import ImgGenJobfrom pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.llm.llm_job import LLMJob-from pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.search.search_job import SearchJob-from pipelex.cogt.usage.cost_registry import CostRegistryfrom pipelex.config import get_config-from pipelex.graph.graph_context import GraphContext-from pipelex.pipeline.pipeline_models import SpecialPipelineId+from pipelex.graph.trace_context import TraceContextfrom pipelex.reporting.reporting_protocol import ReportingProtocol-from pipelex.reporting.reporting_types import AnyTokensUsage, TokensUsage+from pipelex.reporting.reporting_types import AnyTokensUsagefrom pipelex.system.exceptions import MissingDependencyError-from pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path-from pipelex.tools.typing.pydantic_utils import empty_list_factory_offrom pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.event_log_protocol import EventLogProtocolfrom pipelex.tracing.trace_events import UsageReportEvent+# DynamoDB PutItem failures come in two sibling botocore base classes (neither subclasses the other):+# ClientError (service-side throttle / auth) and BotoCoreError (transport / credential / timeout, e.g.+# EndpointConnectionError, ReadTimeoutError, NoCredentialsError). Both are imported together (present or+# absent together) and join the best-effort emit catch when boto3 is installed — matching s3_storage_provider.try:+ from botocore.exceptions import BotoCoreError as _BotoCoreError # type: ignore[import-untyped]from botocore.exceptions import ClientError as _BotoClientError # type: ignore[import-untyped]except ImportError:+ _BotoCoreError = None # type: ignore[assignment, misc]_BotoClientError = None # type: ignore[assignment, misc]+# Infra-level failures an event-log backend can raise from emit(). Usage/cost reporting is a side+# concern: by the time report_inference_job runs, the inference has already succeeded (and been+# billed), so a transient event-log write failure must never propagate and turn a successful+# inference into a failed pipeline. Both emit paths — the registered-context fast path and the+# runner fallback — drop these with a WARNING. OSError covers the NDJSON backend (dir/file write+# errors); ClientError (service-side throttle / auth) and BotoCoreError (transport / credential /+# timeout, e.g. EndpointConnectionError, ReadTimeoutError, NoCredentialsError) cover the DynamoDB+# backend when boto3 is installed. ClientError and BotoCoreError are sibling botocore base classes+# (neither subclasses the other), so both must be listed.+# cast: botocore is untyped, so the imported classes are Unknown; we know they are BaseException types.+# Single assignment (no reassignment) so the uppercase constant satisfies reportConstantRedefinition.+_EMIT_BEST_EFFORT_EXCEPTIONS: tuple[type[BaseException], ...] = (+ (OSError, cast("type[BaseException]", _BotoClientError), cast("type[BaseException]", _BotoCoreError))+ if _BotoClientError is not None and _BotoCoreError is not None+ else (OSError,)+)++class _EventLogContext(NamedTuple):"""Per-workflow/run event log state. Private to ReportingManager."""@@ -42,24 +57,9 @@ class _EventLogContext(NamedTuple):pipeline_run_id: str-UsageRegistryRoot = list[TokensUsage]---class UsageRegistry(RootModel[UsageRegistryRoot]):- root: UsageRegistryRoot = Field(default_factory=empty_list_factory_of(LLMTokensUsage))-- def get_current_tokens_usage(self) -> UsageRegistryRoot:- return self.root-- def add_tokens_usage(self, tokens_usage: TokensUsage):- self.root.append(tokens_usage)--class ReportingManager(ReportingProtocol):def __init__(self):- self._reporting_config = get_config().pipelex.reporting_config- self._usage_registries: dict[str, UsageRegistry] = {}- # Per-context event log state, keyed by graph_context.lookup_key.+ # Per-context event log state, keyed by trace_context.lookup_key.# Each concurrent workflow/run gets its own isolated context.self._event_log_contexts: dict[str, _EventLogContext] = {}@@ -78,7 +78,7 @@ class ReportingManager(ReportingProtocol):"""Configure event log for a specific workflow/run context.Args:- context_key: Unique key for this context (graph_context.lookup_key).+ context_key: Unique key for this context (trace_context.lookup_key).event_log: The event log backend for emitting UsageReportEvents.workflow_id: Temporal workflow ID or "direct".pipeline_run_id: Pipeline run ID for event correlation.@@ -100,55 +100,16 @@ class ReportingManager(ReportingProtocol):@overridedef setup(self):- self._usage_registries.clear()- self._usage_registries[SpecialPipelineId.UNTITLED] = UsageRegistry()+ self._event_log_contexts.clear()@overridedef teardown(self):- self._usage_registries.clear()self._event_log_contexts.clear()############################################################# Private methods############################################################- def _get_registry_strict(self, pipeline_run_id: str) -> UsageRegistry:- """Return the registry for ``pipeline_run_id`` or raise ``KeyError`` on miss.-- Used by ``_report_*_job``: on the runner, the registry is never opened- because ``open_registry`` runs on the API process. Callers swallow- ``KeyError`` and skip the local-add — the runner-side- ``UsageReportEvent`` emission (Phase 2 fallback) is independent of- the registry, so distributed reassembly still gets the data.- """- return self._usage_registries[pipeline_run_id]-- def _get_or_create_registry(self, pipeline_run_id: str) -> UsageRegistry:- """Return the registry for ``pipeline_run_id``, creating it on miss.-- Used by ``inject_tokens_usages`` (the P1 cross-worker assembly path),- the console cost-report path, and ``generate_report`` when called for- a specific run that wasn't opened on this process.- """- if pipeline_run_id not in self._usage_registries:- self._usage_registries[pipeline_run_id] = UsageRegistry()- return self._usage_registries[pipeline_run_id]-- def _try_add_to_registry(self, pipeline_run_id: str, tokens_usage: AnyTokensUsage) -> None:- """Add to the registry if it exists; silently skip if not.-- Skipping on miss is the runner-process behavior — the registry was- never opened here. The local-skip is intentional: the- ``UsageReportEvent`` emitted by ``_emit_usage_event`` carries the- data into the shared backend partition, and the P1 readback path- will assemble it on the process that owns ``inject_tokens_usages``.- """- try:- registry = self._get_registry_strict(pipeline_run_id)- except KeyError:- return- registry.add_tokens_usage(tokens_usage)-def _report_llm_job(self, llm_job: LLMJob):llm_tokens_usage = llm_job.job_report.llm_tokens_usage@@ -156,14 +117,8 @@ class ReportingManager(ReportingProtocol):log.warning("LLM job has no llm_tokens_usage")return- pipeline_run_id = llm_job.job_metadata.pipeline_run_id- self._try_add_to_registry(pipeline_run_id, llm_tokens_usage)self._emit_usage_event(llm_job, llm_tokens_usage)- if self._reporting_config.is_log_costs_to_console:- llm_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=llm_tokens_usage)- log.verbose(llm_token_cost_report, title="Token Cost report")-def _report_img_gen_job(self, img_gen_job: ImgGenJob):img_gen_tokens_usage = img_gen_job.job_report.img_gen_tokens_usage@@ -171,14 +126,8 @@ class ReportingManager(ReportingProtocol):log.warning("ImgGen job has no img_gen_tokens_usage")return- pipeline_run_id = img_gen_job.job_metadata.pipeline_run_id- self._try_add_to_registry(pipeline_run_id, img_gen_tokens_usage)self._emit_usage_event(img_gen_job, img_gen_tokens_usage)- if self._reporting_config.is_log_costs_to_console:- img_gen_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=img_gen_tokens_usage)- log.verbose(img_gen_token_cost_report, title="Token Cost report")-def _report_extract_job(self, extract_job: ExtractJob):extract_tokens_usage = extract_job.job_report.extract_tokens_usage@@ -186,14 +135,8 @@ class ReportingManager(ReportingProtocol):log.warning("Extract job has no extract_tokens_usage")return- pipeline_run_id = extract_job.job_metadata.pipeline_run_id- self._try_add_to_registry(pipeline_run_id, extract_tokens_usage)self._emit_usage_event(extract_job, extract_tokens_usage)- if self._reporting_config.is_log_costs_to_console:- extract_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=extract_tokens_usage)- log.verbose(extract_token_cost_report, title="Token Cost report")-def _report_search_job(self, search_job: SearchJob):search_tokens_usage = search_job.job_report.search_tokens_usage@@ -201,43 +144,12 @@ class ReportingManager(ReportingProtocol):log.warning("Search job has no search_tokens_usage")return- pipeline_run_id = search_job.job_metadata.pipeline_run_id- self._try_add_to_registry(pipeline_run_id, search_tokens_usage)self._emit_usage_event(search_job, search_tokens_usage)- if self._reporting_config.is_log_costs_to_console:- search_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=search_tokens_usage)- log.verbose(search_token_cost_report, title="Token Cost report")-- ############################################################- # ReportingProtocol- ############################################################-- @override- def open_registry(self, pipeline_run_id: str):- if pipeline_run_id in self._usage_registries:- msg = f"Registry for pipeline '{pipeline_run_id}' already exists"- raise ReportingManagerError(msg)- self._usage_registries[pipeline_run_id] = UsageRegistry()-- def inject_tokens_usages(self, pipeline_run_id: str, tokens_usages: Sequence[AnyTokensUsage]) -> None:- """Inject externally-collected token usage records into a pipeline's registry.-- Used after assembling usage data from distributed trace events, so that- generate_report() can produce a complete cost report across all workers.-- Args:- pipeline_run_id: The pipeline run to add usage data to.- tokens_usages: Token usage records to inject.- """- registry = self._get_or_create_registry(pipeline_run_id)- for tokens_usage in tokens_usages:- registry.add_tokens_usage(tokens_usage)-def _emit_usage_event(self, inference_job: InferenceJobAbstract, tokens_usage: AnyTokensUsage) -> None:"""Emit a UsageReportEvent for this job.- Fast path: when set_event_log was registered for this graph context's+ Fast path: when set_event_log was registered for this trace context'slookup_key (router process or direct mode), emit through the cachedper-context event log.@@ -246,29 +158,40 @@ class ReportingManager(ReportingProtocol):log so the event still lands in the same backend partition as therest of the run. See _emit_usage_event_runner_fallback for details."""- graph_context = inference_job.job_metadata.graph_context- if graph_context is None:+ trace_context = inference_job.job_metadata.trace_context+ if trace_context is None:+ return++ # Gate cost emission on emit_usage_events BEFORE the context lookup, so both the fast path+ # (_emit_via_registered_context) and the runner fallback are guarded by the same check. This+ # keeps correctness from resting on the cross-file invariant "a context is registered (via+ # set_event_log) only when costs are on": _event_log_contexts is a process-global singleton+ # and clear_event_log is best-effort in finally blocks, so a leaked context from a prior+ # costs-enabled run could otherwise let a later graph-only run (emit_usage_events=False) emit+ # usage events through the fast path on a colliding lookup_key (reused pipeline_run_id /+ # workflow_id).+ if not trace_context.emit_usage_events:return- context = self._event_log_contexts.get(graph_context.lookup_key)+ context = self._event_log_contexts.get(trace_context.lookup_key)if context is not None:- self._emit_via_registered_context(context, graph_context, tokens_usage)+ self._emit_via_registered_context(context, trace_context, tokens_usage)returnself._emit_usage_event_runner_fallback(inference_job=inference_job,tokens_usage=tokens_usage,- graph_context=graph_context,+ trace_context=trace_context,)@staticmethoddef _emit_via_registered_context(context: _EventLogContext,- graph_context: GraphContext,+ trace_context: TraceContext,tokens_usage: AnyTokensUsage,) -> None:"""Fast-path emit through a context registered via set_event_log."""- node_id = graph_context.parent_node_id or "unknown"+ node_id = trace_context.parent_node_id or "unknown"seq = context.event_log.next_sequence()event = UsageReportEvent(@@ -280,13 +203,28 @@ class ReportingManager(ReportingProtocol):node_id=node_id,tokens_usage=tokens_usage,)- context.event_log.emit(event)+ ReportingManager._emit_best_effort(event_log=context.event_log, event=event)++ @staticmethod+ def _emit_best_effort(event_log: EventLogProtocol, event: UsageReportEvent) -> None:+ """Emit a usage event, dropping infra-level backend failures with a WARNING.++ Shared by both emit paths (the registered-context fast path and the runner fallback).+ report_inference_job runs synchronously after the inference has already succeeded, so an+ event-log write failure must not propagate and fail the pipeline. We catch the specific+ infra exception classes the backends raise at emit() time (see+ ``_EMIT_BEST_EFFORT_EXCEPTIONS``) and drop the event. Other exceptions propagate.+ """+ try:+ event_log.emit(event)+ except _EMIT_BEST_EFFORT_EXCEPTIONS as exc:+ log.warning(f"Usage event emit failed; dropping: {exc}")def _emit_usage_event_runner_fallback(self,inference_job: InferenceJobAbstract,tokens_usage: AnyTokensUsage,- graph_context: GraphContext,+ trace_context: TraceContext,) -> None:"""Emit through a per-process activity event log when no context was registered.@@ -301,12 +239,12 @@ class ReportingManager(ReportingProtocol):Suppression of retried-emit duplicates is a separate, harder problem(deferred follow-up).- Specific exceptions caught and dropped with WARNING:+ Construction-time failures (lazy event-log build) caught and dropped with WARNING:- ``OSError``: NDJSON dir unwritable, file system errors.- ``MissingDependencyError``: ``boto3`` missing for the DynamoDB backend.- ``PipelexConfigError``: factory misconfigured.- - ``botocore.exceptions.ClientError`` (when boto3 is installed):- DynamoDB throttle / auth fail at PutItem time.+ The emit() itself is delegated to ``_emit_best_effort``, which drops the+ backend's emit-time infra failures (see ``_EMIT_BEST_EFFORT_EXCEPTIONS``).Other exceptions propagate."""tracing_config = get_config().pipelex.tracing_config@@ -322,8 +260,8 @@ class ReportingManager(ReportingProtocol):if process_event_log is None:return- workflow_id = graph_context.tracer_key or graph_context.graph_id- node_id = graph_context.parent_node_id or "unknown"+ workflow_id = trace_context.tracer_key or trace_context.graph_id+ node_id = trace_context.parent_node_id or "unknown"ActivityEventLogCache.warn_once_runner_fallback_engaged(workflow_id=workflow_id, writer_id=process_event_log.writer_id)@@ -338,14 +276,7 @@ class ReportingManager(ReportingProtocol):tokens_usage=tokens_usage,)- emit_exceptions: tuple[type[BaseException], ...] = (OSError,)- if _BotoClientError is not None:- emit_exceptions = (*emit_exceptions, _BotoClientError)-- try:- process_event_log.emit(event)- except emit_exceptions as exc:- log.warning(f"Runner-side usage event emit failed; dropping: {exc}")+ self._emit_best_effort(event_log=process_event_log, event=event)@overridedef report_inference_job(self, inference_job: InferenceJobAbstract):@@ -360,37 +291,3 @@ class ReportingManager(ReportingProtocol):self._report_search_job(search_job=inference_job)else:log.warning(f"ReportingManager does not support reporting for inference job type: {type(inference_job).__name__}")-- @override- def generate_report(self, pipeline_run_id: str | None = None, print_to_console: bool = True):- is_csv_enabled = self._reporting_config.is_generate_cost_report_file_enabled- if is_csv_enabled:- ensure_path(Path(self._reporting_config.cost_report_dir_path))-- registries_to_process: dict[str, UsageRegistry] = {}- if pipeline_run_id:- registries_to_process = {pipeline_run_id: self._get_or_create_registry(pipeline_run_id)}- else:- registries_to_process = self._usage_registries-- for run_id, registry in registries_to_process.items():- cost_report_file_path: Path | None = None- if is_csv_enabled:- cost_report_file_path = get_incremental_file_path(- base_path=Path(self._reporting_config.cost_report_dir_path),- base_name=self._reporting_config.cost_report_base_name,- extension=self._reporting_config.cost_report_extension,- )- CostRegistry.generate_report(- pipeline_run_id=run_id,- tokens_usages=registry.get_current_tokens_usage(),- unit_scale=self._reporting_config.cost_report_unit_scale,- cost_report_file_path=cost_report_file_path,- print_to_console=print_to_console,- )-- @override- def close_registry(self, pipeline_run_id: str):- # Idempotent on miss (mirrors clear_event_log): close_registry runs from sweep / runner- # `finally` blocks, where a KeyError from a bare `pop` would mask the in-flight exception.- self._usage_registries.pop(pipeline_run_id, None)diff --git a/pipelex/reporting/reporting_protocol.py b/pipelex/reporting/reporting_protocol.pyindex 451c056a5..72995df8a 100644--- a/pipelex/reporting/reporting_protocol.py+++ b/pipelex/reporting/reporting_protocol.py@@ -7,14 +7,8 @@ from pipelex.tracing.event_log_protocol import EventLogProtocolclass ReportingProtocol(Protocol):- def open_registry(self, pipeline_run_id: str): ...-def report_inference_job(self, inference_job: InferenceJobAbstract): ...- def generate_report(self, pipeline_run_id: str | None = None, print_to_console: bool = True): ...-- def close_registry(self, pipeline_run_id: str): ...-def setup(self): ...def teardown(self): ...@@ -31,22 +25,10 @@ class ReportingProtocol(Protocol):class ReportingNoOp(ReportingProtocol):- @override- def open_registry(self, pipeline_run_id: str):- pass-@overridedef report_inference_job(self, inference_job: InferenceJobAbstract):pass- @override- def generate_report(self, pipeline_run_id: str | None = None, print_to_console: bool = True):- pass-- @override- def close_registry(self, pipeline_run_id: str):- pass-@overridedef setup(self):passdiff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.pyindex dab8c7ad5..d0a0a6654 100644--- a/pipelex/system/configuration/configs.py+++ b/pipelex/system/configuration/configs.py@@ -154,22 +154,25 @@ class PipelineExecutionConfig(ConfigModel):is_normalize_data_urls_to_storage: boolis_mock_inputs: boolis_generate_graph: bool+ is_generate_usage: boolgraph_config: GraphConfig# Bounded fan-out concurrency for PipeBatch (the resilience-without-Temporal pillar).# An integer caps the number of branches executed at once; the literal "unbounded" disables the bound.max_concurrency: Annotated[int, Field(ge=1)] | Literal["unbounded"]- def with_graph_config_overrides(+ def with_execution_overrides(self,generate_graph: bool | None = None,+ generate_usage: bool | None = None,force_include_full_data: bool | None = None,mock_inputs: bool | None = None,) -> Self:"""Create a copy of this config with optional overrides.Args:- generate_graph: If not None, overrides is_generate_graph.+ generate_graph: If not None, overrides is_generate_graph (graph node/edge events + GraphSpec assembly).+ generate_usage: If not None, overrides is_generate_usage (emit usage/cost tracing events).force_include_full_data: If not None, overrides all graph_config.data_inclusion flags(stuff_json_content, stuff_text_content, stuff_html_content, error_stack_traces).mock_inputs: If not None, overrides is_mock_inputs. When True, generates mock@@ -183,6 +186,9 @@ class PipelineExecutionConfig(ConfigModel):if generate_graph is not None:updates["is_generate_graph"] = generate_graph+ if generate_usage is not None:+ updates["is_generate_usage"] = generate_usage+if mock_inputs is not None:updates["is_mock_inputs"] = mock_inputsdiff --git a/pipelex/temporal/tasks.py b/pipelex/temporal/tasks.pyindex e6e5844c8..3aeabb6fa 100644--- a/pipelex/temporal/tasks.py+++ b/pipelex/temporal/tasks.py@@ -6,7 +6,7 @@ from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_imfrom pipelex.temporal.tprl_content_generation.act_jinja2_generate import act_jinja2_gen_textfrom pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_object, act_llm_gen_object_list, act_llm_gen_textfrom pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views-from pipelex.temporal.tprl_pipe.act_assemble_graph import act_assemble_graph+from pipelex.temporal.tprl_pipe.act_assemble_tracing import act_assemble_tracingfrom pipelex.temporal.tprl_pipe.act_deliver import act_deliverfrom pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_eventsfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter@@ -39,7 +39,7 @@ class Tasks:WfPipeRun,],activity_list=[- act_assemble_graph,+ act_assemble_tracing,act_deliver,act_flush_trace_events,],diff --git a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.pyindex 332329067..88b30529d 100644--- a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py+++ b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py@@ -1,6 +1,6 @@from typing import Any-from pydantic import BaseModel # noqa: TC002+from pydantic import BaseModel, ValidationErrorfrom temporalio import workflowfrom temporalio.exceptions import ActivityError, ApplicationErrorfrom typing_extensions import override@@ -15,6 +15,7 @@ from pipelex.cogt.content_generation.assignment_models import (TemplatingAssignment,)from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata+from pipelex.cogt.content_generation.exceptions import MockInferenceObjectFidelityErrorfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams@@ -45,6 +46,34 @@ from pipelex.temporal.tprl_content_generation.act_render_page_views import act_rfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar+def _revalidate_against_object_class(+ raw_obj: BaseModel,+ object_class: type[BaseModelTypeVar],+ *,+ is_mock_inference: bool,+) -> BaseModelTypeVar:+ """Re-validate an activity-boundary object against the original ``object_class``.++ The Temporal arm of ``ContentGenerator._revalidate_against_object_class``: round-trips the+ activity-boundary ``BaseModel`` through json (``mode="json"`` is required for fields that need json-mode+ serialization to round-trip cleanly) into the caller's original concrete class (e.g. ``StructuredContent``).++ Under ``--mock-inference`` the object was built from the schema-reconstructed class inside+ ``act_llm_gen_object*``, which can drop invariants the original class enforces (custom validators,+ ``json_schema_extra`` format/pattern hints datamodel-code-generator omits on round-trip). That+ re-validation failure is re-raised as a clear :class:`MockInferenceObjectFidelityError` rather than an+ opaque pydantic crash mid-workflow (review F2). Scoped to the mock path only — a LIVE provider's invalid+ output keeps its existing ``ValidationError``.+ """+ raw_data = raw_obj.model_dump(mode="json", serialize_as_any=True)+ if not is_mock_inference:+ return object_class.model_validate(raw_data)+ try:+ return object_class.model_validate(raw_data)+ except ValidationError as exc:+ raise MockInferenceObjectFidelityError.for_object_class(object_class.__name__) from exc++class ContentGeneratorInWorkflow(ContentGeneratorProtocol):"""ContentGenerator that dispatches each operation as a Temporal activity directly.@@ -137,10 +166,7 @@ class ContentGeneratorInWorkflow(ContentGeneratorProtocol):raise TemporalError.from_app_error(exc=exc.cause) from excraiselog.verbose(f"ContentGeneratorInWorkflow generated object direct: {obj}")- # Round-trip through json so the activity-boundary BaseModel becomes the caller's- # original concrete class (e.g. StructuredContent). ``mode="json"`` is required- # for fields that need json-mode serialization to round-trip cleanly.- return object_class.model_validate(obj.model_dump(mode="json", serialize_as_any=True))+ return _revalidate_against_object_class(obj, object_class, is_mock_inference=job_metadata.is_mock_inference)@override@update_job_metadata@@ -181,7 +207,7 @@ class ContentGeneratorInWorkflow(ContentGeneratorProtocol):raise TemporalError.from_app_error(exc=exc.cause) from excraiselog.verbose(f"ContentGeneratorInWorkflow generated object list direct: {obj_list}")- return [object_class.model_validate(raw_obj.model_dump(mode="json", serialize_as_any=True)) for raw_obj in obj_list]+ return [_revalidate_against_object_class(raw_obj, object_class, is_mock_inference=job_metadata.is_mock_inference) for raw_obj in obj_list]@overrideasync def make_image_content(diff --git a/pipelex/temporal/tprl_pipe/act_assemble_graph.py b/pipelex/temporal/tprl_pipe/act_assemble_graph.pydeleted file mode 100644index 4cf9b38e2..000000000--- a/pipelex/temporal/tprl_pipe/act_assemble_graph.py+++ /dev/null@@ -1,58 +0,0 @@-"""Temporal activity for assembling the graph spec from trace events.--Reads trace events from the configured backend (DynamoDB) and assembles-the full GraphSpec. Runs as an activity because DynamoDB reads are I/O-that cannot happen inside a workflow thread.-"""--from pydantic import BaseModel-from temporalio import activity--from pipelex import log-from pipelex.config import get_config-from pipelex.graph.graphspec import GraphSpec, PipelineRef-from pipelex.tracing.event_log_factory import make_event_log-from pipelex.tracing.graphspec_assembler import GraphSpecAssembler---class AssembleGraphArg(BaseModel):- """Input for the graph assembly activity."""-- pipeline_run_id: str- domain_code: str | None = None- main_pipe_code: str | None = None---# Deliberately NOT decorated with @convert_pipelex_errors: this activity is best-effort-# observability — it swallows every failure and degrades to None, so no error ever-# crosses the boundary for the decorator to convert.[email protected](name="act_assemble_graph")-async def act_assemble_graph(arg: AssembleGraphArg) -> GraphSpec | None: # noqa: RUF029- """Read trace events and assemble GraphSpec. Returns None if no events found."""- tracing_config = get_config().pipelex.tracing_config- if not tracing_config.is_enabled:- return None-- try:- event_log = make_event_log(tracing_config)- try:- events = event_log.read_events(arg.pipeline_run_id)- if not events:- log.debug(f"No trace events found for pipeline_run_id={arg.pipeline_run_id}")- return None- graph_spec = GraphSpecAssembler.assemble(- events=events,- graph_id=arg.pipeline_run_id,- pipeline_ref=PipelineRef(- domain=arg.domain_code,- main_pipe=arg.main_pipe_code,- ),- )- log.debug(f"Graph assembled from {len(events)} events for pipeline_run_id={arg.pipeline_run_id}")- return graph_spec- finally:- event_log.close()- except Exception as exc: # noqa: BLE001- # Temporal activity root: graph assembly is best-effort observability — any failure degrades to None rather than failing the workflow.- log.warning(f"Graph assembly activity failed: {exc}")- return Nonediff --git a/pipelex/temporal/tprl_pipe/act_assemble_tracing.py b/pipelex/temporal/tprl_pipe/act_assemble_tracing.pynew file mode 100644index 000000000..d3fdd3d8a--- /dev/null+++ b/pipelex/temporal/tprl_pipe/act_assemble_tracing.py@@ -0,0 +1,50 @@+"""Temporal activity for assembling tracing artifacts from the trace-event stream.++Reads trace events from the configured backend (DynamoDB / NDJSON) once and+assembles the full GraphSpec (for ``--graph``) and the aggregated token usage+(for ``--costs``). Runs as an activity because backend reads are I/O that cannot+happen inside a workflow thread.+"""++from pydantic import BaseModel+from temporalio import activity++from pipelex.pipe_run.tracing_assembly import TracingAssembly, assemble_tracing+++class AssembleTracingArg(BaseModel):+ """Input for the tracing assembly activity."""++ pipeline_run_id: str+ domain_code: str | None = None+ main_pipe_code: str | None = None+ # Which artifacts to assemble (mirror of the run's TraceContext emit flags). The submitter gates+ # the dispatch on these too; passing them through keeps the activity a pure function of its input.+ assemble_graph: bool = True+ assemble_usage: bool = True+++# NOT decorated with @convert_pipelex_errors: tracing assembly is observability, not a pipe step.+# `assemble_tracing` already catches the EXPECTED best-effort failures (backend read errors, malformed+# events, broken tracing infra) and returns them on the *_assembly_error fields — those never raise here.+# A genuinely unexpected error (a programming bug in assembly) is deliberately allowed to propagate so the+# activity fails and `WfPipeRun` records it on usage_assembly_error / graph_assembly_error (its existing+# `except ActivityError` branch), mirroring DIRECT mode where assemble_tracing_on_output lets the same+# errors surface. Swallowing them here would make the workflow see success and silently produce no cost+# report and no diagnostic. The plain ActivityError carries the message WfPipeRun stringifies; no+# ErrorReport classification is needed for an observability artifact.[email protected](name="act_assemble_tracing")+async def act_assemble_tracing(arg: AssembleTracingArg) -> TracingAssembly: # noqa: RUF029+ """Read trace events and assemble graph + usage.++ Returns an empty result when no events are found or tracing is disabled. Expected best-effort+ failures are caught inside :func:`assemble_tracing` and returned on the ``*_assembly_error`` fields;+ unexpected errors propagate so the workflow surfaces them (see the module-level note above).+ """+ return assemble_tracing(+ pipeline_run_id=arg.pipeline_run_id,+ assemble_graph=arg.assemble_graph,+ assemble_usage=arg.assemble_usage,+ domain_code=arg.domain_code,+ main_pipe_code=arg.main_pipe_code,+ )diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.pyindex 2f61c314f..cfd2cebfd 100644--- a/pipelex/temporal/tprl_pipe/wf_pipe_router.py+++ b/pipelex/temporal/tprl_pipe/wf_pipe_router.py@@ -46,7 +46,7 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):event_log = Nonewf_graph_tracer_manager: GraphTracerManager | None = Nonewf_tracer_key: str | None = None- graph_context = workflow_arg.job_metadata.graph_context+ trace_context = workflow_arg.job_metadata.trace_contextpipe_output: PipeOutput | None = None@@ -77,37 +77,47 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):wf_workflow_id = workflow.info().workflow_idtracing_config = get_config().pipelex.tracing_config- if tracing_config.is_enabled and graph_context is not None:+ if tracing_config.is_enabled and trace_context is not None:try:# Use BufferingEventLog inside workflows (no I/O allowed).# Events are flushed to the real backend via act_flush_trace_events.event_log = BufferingEventLog()wf_graph_tracer_manager = GraphTracerManager.get_or_create_instance()wf_tracer_key = wf_workflow_id- wf_graph_context = wf_graph_tracer_manager.open_tracer(- graph_id=graph_context.graph_id,- data_inclusion=graph_context.data_inclusion,- event_log=event_log,+ wf_trace_context = wf_graph_tracer_manager.open_tracer(+ graph_id=trace_context.graph_id,+ data_inclusion=trace_context.data_inclusion,+ # D5: only feed the event log to the tracer when graph events are wanted. In+ # costs-only mode the tracer still mints node ids but emits no graph events;+ # usage events flow via the report delegate's set_event_log below.+ event_log=event_log if trace_context.emit_graph_events else None,workflow_id=wf_workflow_id,pipeline_run_id=pipeline_run_id,tracer_key=wf_tracer_key,+ # Threaded in so the returned context is born with the correct flags (no emit-flag+ # footgun in the model_copy below).+ emit_graph_events=trace_context.emit_graph_events,+ emit_usage_events=trace_context.emit_usage_events,)- # Update job_metadata with the per-workflow graph_context (carries tracer_key),- # but preserve parent_node_id from the incoming context so CONTAINS edges- # link back to the parent workflow's controller node.- wf_graph_context = wf_graph_context.model_copy(- update={"parent_node_id": graph_context.parent_node_id},+ # Update job_metadata with the per-workflow trace_context (carries tracer_key + emit+ # flags), but preserve parent_node_id from the incoming context so CONTAINS edges link+ # back to the parent workflow's controller node.+ wf_trace_context = wf_trace_context.model_copy(+ update={"parent_node_id": trace_context.parent_node_id},)workflow_arg.job_metadata = workflow_arg.job_metadata.model_copy(- update={"graph_context": wf_graph_context},- )- # Configure the report delegate for usage event emission- get_report_delegate().set_event_log(- context_key=wf_workflow_id,- event_log=event_log,- workflow_id=wf_workflow_id,- pipeline_run_id=pipeline_run_id,+ update={"trace_context": wf_trace_context},)+ # Configure the report delegate for usage event emission — only when cost reporting+ # is on. In graph-only mode no usage context is registered, so usage events are+ # suppressed (the runner fallback also gates on emit_usage_events).+ if trace_context.emit_usage_events:+ get_report_delegate().set_event_log(+ context_key=wf_workflow_id,+ event_log=event_log,+ workflow_id=wf_workflow_id,+ pipeline_run_id=pipeline_run_id,+ )except Exception as exc: # noqa: BLE001# Best-effort: per-workflow tracing setup must never fail the workflow — log and continue without it.workflow_log.warning(f"Failed to set up per-workflow tracing, continuing without: {exc}")@@ -138,11 +148,13 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):raise TemporalError.from_app_error(exc=exc.cause) from excraisefinally:- # Close per-workflow graph tracer (collects in-memory graph spec)+ # Close per-workflow graph tracer (collects in-memory graph spec). F1: only assign the spec+ # when graph events were requested — in costs-only mode close_tracer returns None (teardown+ # skips the spec build), and this guard keeps the contract explicit even if that changes.if wf_graph_tracer_manager is not None and wf_tracer_key is not None:try:graph_spec = wf_graph_tracer_manager.close_tracer(wf_tracer_key)- if graph_spec is not None and pipe_output is not None:+ if graph_spec is not None and pipe_output is not None and trace_context is not None and trace_context.emit_graph_events:pipe_output.graph_spec = graph_specexcept Exception as tracer_exc: # noqa: BLE001# Best-effort: tracer close in the finally block must never fail the workflow — log and continue.diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_run.py b/pipelex/temporal/tprl_pipe/wf_pipe_run.pyindex 9eeb491b5..55147a99a 100644--- a/pipelex/temporal/tprl_pipe/wf_pipe_run.py+++ b/pipelex/temporal/tprl_pipe/wf_pipe_run.py@@ -14,7 +14,7 @@ with workflow.unsafe.imports_passed_through():from pipelex.temporal.tprl.observability import build_search_attributes, build_static_summaryfrom pipelex.temporal.tprl.temporal_error import recover_error_reportfrom pipelex.temporal.tprl.workflow_caller import WorkflowClass- from pipelex.temporal.tprl_pipe.act_assemble_graph import AssembleGraphArg, act_assemble_graph+ from pipelex.temporal.tprl_pipe.act_assemble_tracing import AssembleTracingArg, act_assemble_tracingfrom pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg, act_deliverfrom pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArgfrom pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter@@ -73,25 +73,45 @@ class WfPipeRun(WorkflowClass[PipeRunArg, PipeOutput]):execution_error.__cause__ = excworkflow_log.error(f"WfPipeRouter failed: {exc}")- # Step 2: Assemble full graph from trace events (cross-worker)- # Runs as an activity because DynamoDB reads are I/O forbidden in workflows.- if pipe_output is not None:+ # Step 2: Assemble full graph + usage from trace events (cross-worker)+ # Runs as an activity because DynamoDB reads are I/O forbidden in workflows. The dispatch is+ # gated on the run's emit flags (F1): a costs-only run assembles usage but no graph, so+ # graph_spec stays None — matching DIRECT mode and preserving the --no-graph contract.+ trace_context = pipe_job.job_metadata.trace_context+ if pipe_output is not None and trace_context is not None and (trace_context.emit_graph_events or trace_context.emit_usage_events):try:- graph_spec = await workflow.execute_activity(- act_assemble_graph,- arg=AssembleGraphArg(+ tracing_assembly = await workflow.execute_activity(+ act_assemble_tracing,+ arg=AssembleTracingArg(pipeline_run_id=pipeline_run_id,domain_code=pipe_job.pipe.domain_code,main_pipe_code=pipe_job.pipe.code,+ assemble_graph=trace_context.emit_graph_events,+ assemble_usage=trace_context.emit_usage_events,),start_to_close_timeout=timedelta(seconds=30),retry_policy=RetryPolicy(maximum_attempts=3),)- if graph_spec is not None:- pipe_output.graph_spec = graph_spec- except ActivityError as graph_exc:- workflow_log.warning(f"Graph assembly failed, continuing with delivery: {graph_exc}")- pipe_output.graph_assembly_error = str(graph_exc)+ # Mirror DIRECT's assemble_tracing_on_output: copy every populated field, including the+ # best-effort *_assembly_error fields. assemble_tracing catches the expected read/assemble+ # failures internally and returns them on these fields (no exception raised), so without+ # this the failure would be observable in DIRECT but silently lost in TEMPORAL.+ if tracing_assembly.graph_spec is not None:+ pipe_output.graph_spec = tracing_assembly.graph_spec+ if tracing_assembly.graph_assembly_error is not None:+ pipe_output.graph_assembly_error = tracing_assembly.graph_assembly_error+ if tracing_assembly.tokens_usages is not None:+ pipe_output.tokens_usages = tracing_assembly.tokens_usages+ if tracing_assembly.usage_assembly_error is not None:+ pipe_output.usage_assembly_error = tracing_assembly.usage_assembly_error+ except ActivityError as assembly_exc:+ workflow_log.warning(f"Tracing assembly failed, continuing with delivery: {assembly_exc}")+ # Record the failure only on the concern(s) actually requested, so a costs-only run never+ # surfaces a graph_assembly_error (and vice versa).+ if trace_context.emit_graph_events:+ pipe_output.graph_assembly_error = str(assembly_exc)+ if trace_context.emit_usage_events:+ pipe_output.usage_assembly_error = str(assembly_exc)# Step 3: Run delivery activity if requested — notifies the completion# Lambda of success or failure when a delivery_assignment was provided.diff --git a/pipelex/tools/log/log_dispatch.py b/pipelex/tools/log/log_dispatch.pyindex be42a241f..1353ad3b5 100644--- a/pipelex/tools/log/log_dispatch.py+++ b/pipelex/tools/log/log_dispatch.py@@ -152,7 +152,7 @@ class LogDispatch:message_for_console = f"{caller_info_str}: {message}"if include_exception:- message += f"\n{traceback.format_exc()}"+ message_for_console += f"\n{traceback.format_exc()}"self._log_to_console(message=message_for_console, severity=severity)def _log_data(diff --git a/tests/conftest.py b/tests/conftest.pyindex 5a46936d8..c143d3cf9 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -7,7 +7,7 @@ import shortuuidfrom pytest_mock import MockerFixturefrom pipelex import log-from pipelex.hub import get_library_manager, get_report_delegate, set_current_library+from pipelex.hub import get_library_manager, set_current_libraryfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.system.pipelex_service.pipelex_service_config import (@@ -169,21 +169,16 @@ def load_empty_library() -> Generator[Callable[[], str], None, None]:@pytest.fixture-def job_metadata(request: pytest.FixtureRequest) -> Generator[JobMetadata, None, None]:+def job_metadata(request: pytest.FixtureRequest) -> JobMetadata:"""Provide a JobMetadata instance with test-specific values.Uses the test node ID as pipeline_run_id for better traceability in logs.- Opens a registry for the pipeline run ID before the test and closes it after."""test_id: str = request.node.nodeid # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]random_code: str = shortuuid.uuid()[:5]pipeline_run_id: str = f"{test_id}-{random_code}"- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)-- yield JobMetadata(+ return JobMetadata(user_id="pytest",pipeline_run_id=pipeline_run_id,)-- get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)diff --git a/tests/e2e/pipelex/graph/test_graph_with_full_data.py b/tests/e2e/pipelex/graph/test_graph_with_full_data.pyindex a17f8d1a2..5d490b28f 100644--- a/tests/e2e/pipelex/graph/test_graph_with_full_data.py+++ b/tests/e2e/pipelex/graph/test_graph_with_full_data.py@@ -106,7 +106,7 @@ class TestGraphWithFullData:4. Verifies interactive rendering works with the captured data"""# Build effective config with graph tracing and full data capture enabled- exec_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ exec_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,force_include_full_data=True,)diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.pyindex 02caca6fb..12b663fca 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.py@@ -43,7 +43,7 @@ class TestPipeBatchGraph:4. Saves the GraphSpec for inspection"""# Build effective config with graph tracing enabled- exec_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ exec_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,force_include_full_data=False,)@@ -167,7 +167,7 @@ class TestPipeBatchGraph:"""# Build config with graph tracing and all graph outputs enabledbase_config = get_config().pipelex.pipeline_execution_config- exec_config = base_config.with_graph_config_overrides(+ exec_config = base_config.with_execution_overrides(generate_graph=True,force_include_full_data=False,)diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.pyindex ca9af5e59..695fa6036 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.py@@ -46,7 +46,7 @@ class TestPipeParallelGraph:"""# Build config with graph tracing and all graph outputs enabledbase_config = get_config().pipelex.pipeline_execution_config- exec_config = base_config.with_graph_config_overrides(+ exec_config = base_config.with_execution_overrides(generate_graph=True,force_include_full_data=False,)@@ -186,7 +186,7 @@ class TestPipeParallelGraph:"""# Build config with graph tracingbase_config = get_config().pipelex.pipeline_execution_config- exec_config = base_config.with_graph_config_overrides(+ exec_config = base_config.with_execution_overrides(generate_graph=True,force_include_full_data=False,)diff --git a/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.py b/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.pynew file mode 100644index 000000000..9fe8f2815--- /dev/null+++ b/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.py@@ -0,0 +1,118 @@+"""Integration test: the agent CLI surfaces the cost report in its JSON output (not a stderr table).++``run_pipeline_core`` attaches a ``cost_report`` object to the ``--with-memory`` envelope when the run+has non-zero cost, and omits it for zero-cost (dry-run-shaped) usage. The runner is mocked so the test+crafts the assembled ``tokens_usages`` directly and spends nothing.+"""++from pathlib import Path++import pytest+from pytest_mock import MockerFixture++from pipelex.cli.agent_cli.commands.run._run_core import run_pipeline_core # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipeline.job_metadata import JobMetadata++_RUN_CORE_MODULE = "pipelex.cli.agent_cli.commands.run._run_core"+++def _non_zero_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="claude-x",+ inference_model_id="claude-x-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},+ )+++def _zero_cost_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="dry",+ inference_model_id="dry-run",+ nb_tokens_by_category={},+ unit_costs={},+ )+++def _free_model_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ """A free/zero-price model: real tokens, no unit costs -> tokens but zero cost."""+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="ollama-x",+ inference_model_id="ollama-x-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={},+ )++[email protected](loop_scope="class")+class TestAgentRunCostReport:+ def _patch_runner(self, mocker: MockerFixture, pipe_output: PipeOutput) -> None:+ response = mocker.MagicMock()+ response.pipe_output = pipe_output+ runner = mocker.MagicMock()+ runner.execute_pipeline = mocker.AsyncMock(return_value=response)+ mocker.patch(f"{_RUN_CORE_MODULE}.PipelexRunner", return_value=runner)++ async def test_cost_report_in_with_memory_json(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:+ pipe_output = PipeOutput(+ working_memory=WorkingMemory(),+ pipeline_run_id="agent-run",+ tokens_usages=[_non_zero_usage(job_metadata)],+ )+ self._patch_runner(mocker, pipe_output)++ result = await run_pipeline_core(+ pipe_code="my_pipe",+ bundle_uris=[str(tmp_path / "bundle.mthds")],+ costs=True,+ with_memory=True,+ )++ assert "cost_report" in result+ assert result["cost_report"]["total_cost"] == 0.2+ assert result["cost_report"]["by_model"][0]["model"] == "claude-x"++ async def test_cost_report_for_free_model_run(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:+ """A free/zero-price model with real tokens still surfaces a cost_report (total_cost 0)."""+ pipe_output = PipeOutput(+ working_memory=WorkingMemory(),+ pipeline_run_id="agent-free",+ tokens_usages=[_free_model_usage(job_metadata)],+ )+ self._patch_runner(mocker, pipe_output)++ result = await run_pipeline_core(+ pipe_code="my_pipe",+ bundle_uris=[str(tmp_path / "bundle.mthds")],+ costs=True,+ with_memory=True,+ )++ assert "cost_report" in result+ assert result["cost_report"]["total_cost"] == 0.0+ assert result["cost_report"]["by_model"][0]["nb_tokens_input"] == 100++ async def test_no_cost_report_for_dry_run(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:+ pipe_output = PipeOutput(+ working_memory=WorkingMemory(),+ pipeline_run_id="agent-dry",+ tokens_usages=[_zero_cost_usage(job_metadata)],+ )+ self._patch_runner(mocker, pipe_output)++ result = await run_pipeline_core(+ pipe_code="my_pipe",+ bundle_uris=[str(tmp_path / "bundle.mthds")],+ costs=True,+ with_memory=True,+ )++ assert "cost_report" not in resultdiff --git a/tests/integration/pipelex/cogt/content_generation/test_mock_inference_object_fidelity.py b/tests/integration/pipelex/cogt/content_generation/test_mock_inference_object_fidelity.pynew file mode 100644index 000000000..23f73df03--- /dev/null+++ b/tests/integration/pipelex/cogt/content_generation/test_mock_inference_object_fidelity.py@@ -0,0 +1,88 @@+"""Regression test for the ``--mock-inference`` object-mock fidelity gap (review F2).++A ``--mock-inference`` object mock is built from the schema-reconstructed class, then re-validated by+``ContentGenerator.make_object`` against the **original** class. An invariant the JSON-schema round-trip+cannot capture — here a custom ``@field_validator`` — is absent from the reconstructed class, so the mock+carries a value the original class rejects. The contract under test: that failure surfaces as a clear typed+``MockInferenceObjectFidelityError`` (naming the class, pointing at ``--dry-run``), NOT an opaque+``pydantic.ValidationError`` mid-run. A class without such a hidden invariant round-trips fine and the mock+validates — proving the guard is scoped to genuine fidelity failures, not every mock object.++No provider is ever called (the leaf mock short-circuits before ``get_llm_worker``), so this needs no+inference marker.+"""++import pytest+from pydantic import field_validator++from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol+from pipelex.cogt.content_generation.exceptions import MockInferenceObjectFidelityError+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.pipeline.job_metadata import JobMetadata+++class ConstrainedName(StructuredContent):+ """Original class with an invariant the JSON-schema round-trip cannot capture."""++ name: str++ @field_validator("name")+ @classmethod+ def _require_prefix(cls, value: str) -> str:+ if not value.startswith("PFX_"):+ msg = "name must start with 'PFX_'"+ raise ValueError(msg)+ return value+++class PlainName(StructuredContent):+ """Control class whose only constraint (a plain string field) survives the round-trip."""++ name: str++[email protected](loop_scope="class")+class TestMockInferenceObjectFidelity:+ def _mock_job_metadata(self, job_metadata: JobMetadata) -> JobMetadata:+ return job_metadata.model_copy(update={"is_mock_inference": True})++ async def test_make_object_raises_typed_error_on_fidelity_gap(+ self, job_metadata: JobMetadata, content_generator: ContentGeneratorProtocol+ ) -> None:+ """make_object re-raises the dropped-invariant ValidationError as MockInferenceObjectFidelityError."""+ with pytest.raises(MockInferenceObjectFidelityError) as exc_info:+ await content_generator.make_object(+ job_metadata=self._mock_job_metadata(job_metadata),+ object_class=ConstrainedName,+ llm_prompt_for_object=LLMPrompt(user_text="make a prefixed name"),+ llm_setting_for_object=LLMSetting(model="gpt-4o", temperature=0.5),+ )+ assert ConstrainedName.__name__ in str(exc_info.value)+ assert "--dry-run" in str(exc_info.value)++ async def test_make_object_list_raises_typed_error_on_fidelity_gap(+ self, job_metadata: JobMetadata, content_generator: ContentGeneratorProtocol+ ) -> None:+ """make_object_list re-raises the per-item fidelity failure as the same typed error."""+ with pytest.raises(MockInferenceObjectFidelityError):+ await content_generator.make_object_list(+ job_metadata=self._mock_job_metadata(job_metadata),+ object_class=ConstrainedName,+ llm_prompt_for_object_list=LLMPrompt(user_text="make prefixed names"),+ llm_setting_for_object_list=LLMSetting(model="gpt-4o", temperature=0.5),+ )++ async def test_make_object_succeeds_when_no_hidden_invariant(+ self, job_metadata: JobMetadata, content_generator: ContentGeneratorProtocol+ ) -> None:+ """A class whose constraints survive the round-trip mocks and re-validates without the guard firing."""+ result = await content_generator.make_object(+ job_metadata=self._mock_job_metadata(job_metadata),+ object_class=PlainName,+ llm_prompt_for_object=LLMPrompt(user_text="make a plain name"),+ llm_setting_for_object=LLMSetting(model="gpt-4o", temperature=0.5),+ )+ assert isinstance(result, PlainName)+ assert isinstance(result.name, str)diff --git a/tests/integration/pipelex/cogt/test_extract.py b/tests/integration/pipelex/cogt/test_extract.pyindex 193ee9c17..7dc5f26ea 100644--- a/tests/integration/pipelex/cogt/test_extract.py+++ b/tests/integration/pipelex/cogt/test_extract.py@@ -5,7 +5,7 @@ from pipelex.cogt.content_generation.generated_content_factory import GeneratedCfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobParamsfrom pipelex.cogt.extract.extract_job_factory import ExtractJobFactory-from pipelex.hub import get_extract_worker, get_report_delegate+from pipelex.hub import get_extract_workerfrom pipelex.pipeline.job_metadata import JobMetadatafrom tests.cases import DocumentTestCases, ImageTestCasesfrom tests.integration.pipelex.fixtures.model_combo import ModelCombo@@ -48,7 +48,6 @@ class TestExtract:assert page_contentsfor page_index, page_content in enumerate(page_contents):pretty_print(page_content, title=f"Page {page_index}")- get_report_delegate().generate_report()@pytest.mark.parametrize("url", DocumentTestCases.DOCUMENT_URLS)async def test_extract_pdf_url(@@ -165,7 +164,6 @@ class TestExtract:assert page_contentsfor page_index, page_content in enumerate(page_contents):pretty_print(page_content, title=f"Page {page_index}")- get_report_delegate().generate_report()@pytest.mark.parametrize("file_path", DocumentTestCases.DOCUMENT_FILE_PATHS)async def test_extract_image_save(diff --git a/tests/integration/pipelex/cogt/test_img_gen.py b/tests/integration/pipelex/cogt/test_img_gen.pyindex 28830c777..5dfeb1abe 100644--- a/tests/integration/pipelex/cogt/test_img_gen.py+++ b/tests/integration/pipelex/cogt/test_img_gen.py@@ -4,7 +4,7 @@ from pipelex import pretty_print, pretty_print_urlfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobParamsfrom pipelex.cogt.img_gen.img_gen_job_factory import ImgGenJobFactory-from pipelex.hub import get_img_gen_worker, get_report_delegate+from pipelex.hub import get_img_gen_workerfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.misc.image_utils import ImageFormatfrom tests.integration.pipelex.fixtures.img_gen_fixtures import skip_if_img_gen_params_unsupported@@ -52,7 +52,6 @@ class TestImageGeneration:pretty_print(image_content, title=f"Image content for topic '{topic}'")assert image_content.public_url is not Nonepretty_print_url(image_content.public_url, title=f"Image URL for topic '{topic}'")- get_report_delegate().generate_report()@pytest.mark.parametrize(("topic", "positive_text", "negative_text"), ImageGenTestCases.IMAGE_GEN_PROMPT_CONTENTS)async def test_img_gen_single_transparent(diff --git a/tests/integration/pipelex/cogt/test_llm_inference.py b/tests/integration/pipelex/cogt/test_llm_inference.pyindex 82b93a809..e6379c40d 100644--- a/tests/integration/pipelex/cogt/test_llm_inference.py+++ b/tests/integration/pipelex/cogt/test_llm_inference.py@@ -33,7 +33,6 @@ class TestLLMInference:generated_text = await llm_worker.gen_text(llm_job=llm_job)assert generated_textpretty_print(generated_text)- # get_report_delegate().generate_report()async def test_simple_gen_object_from_text(self, job_metadata: JobMetadata, llm_job_params: LLMJobParams, llm_combo: ModelCombo):log.info(f"test_simple_gen_object_from_text: Testing llm_handle '{llm_combo.handle}'")@@ -53,7 +52,6 @@ class TestLLMInference:generated_object = await llm_worker.gen_object(llm_job=llm_job, schema=Person)assert generated_objectpretty_print(generated_object)- # get_report_delegate().generate_report()async def test_simple_gen_object_list_from_text(self, job_metadata: JobMetadata, llm_job_params: LLMJobParams, llm_combo: ModelCombo):"""Generate a list of structured objects from a text prompt."""@@ -108,7 +106,6 @@ class TestLLMInference:pretty_print(generated_text, title=f"Vision of {image_path}")except PromptImageFormatError as exc:pytest.skip(f"Prompt Image format not supported for this LLM: {llm_combo.handle} because {exc}")- # get_report_delegate().generate_report()@pytest.mark.parametrize("image_path", [LLMVisionTestCases.PATH_IMG_PNG_1])async def test_gen_object_from_image(self, job_metadata: JobMetadata, llm_job_params: LLMJobParams, llm_combo: ModelCombo, image_path: str):@@ -143,4 +140,3 @@ class TestLLMInference:pretty_print(generated_object, title=f"Image Description of {image_path}")except PromptImageFormatError as exc:pytest.skip(f"Prompt Image format not supported for this LLM: {llm_combo.handle} because {exc}")- # get_report_delegate().generate_report()diff --git a/tests/integration/pipelex/cogt/test_llm_report.py b/tests/integration/pipelex/cogt/test_llm_report.pyindex ce245815f..a232b67ab 100644--- a/tests/integration/pipelex/cogt/test_llm_report.py+++ b/tests/integration/pipelex/cogt/test_llm_report.py@@ -6,15 +6,11 @@ from pipelex import pretty_printfrom pipelex.cogt.llm.llm_job_components import LLMJobConfigfrom pipelex.cogt.llm.llm_job_factory import LLMJobFactoryfrom pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.hub import get_llm_worker, get_model_deck, get_report_delegate+from pipelex.hub import get_llm_worker, get_model_deckfrom pipelex.pipeline.job_metadata import JobMetadatafrom tests.integration.pipelex.cogt.test_data import LLMTestCases-def test_llm_report_without_running_anything():- get_report_delegate().generate_report()--@pytest.mark.llm@pytest.mark.inference@pytest.mark.asyncio(loop_scope="class")diff --git a/tests/integration/pipelex/cogt/test_search.py b/tests/integration/pipelex/cogt/test_search.pyindex 26a6ca154..22e0b7cdc 100644--- a/tests/integration/pipelex/cogt/test_search.py+++ b/tests/integration/pipelex/cogt/test_search.py@@ -6,7 +6,7 @@ from pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.search.search_job_factory import SearchJobFactoryfrom pipelex.cogt.search.search_setting import SearchSettingfrom pipelex.cogt.search.search_worker_factory import SearchWorkerFactory-from pipelex.hub import get_model_deck, get_report_delegate+from pipelex.hub import get_model_deckfrom pipelex.pipeline.job_metadata import JobMetadatafrom tests.integration.pipelex.cogt.test_data import SearchTestCasesfrom tests.integration.pipelex.fixtures.model_combo import ModelCombo@@ -47,7 +47,6 @@ class TestSearch:for source in result.sources:assert source.title, "Expected source to have a title"assert source.url, "Expected source to have a URL"- get_report_delegate().generate_report()@pytest.mark.parametrize(("topic", "query"),@@ -76,4 +75,3 @@ class TestSearch:assert "summary" in data, "Expected 'summary' key in data"assert "key_points" in data, "Expected 'key_points' key in data"assert len(result["sources"]) > 0, "Expected at least one source"- get_report_delegate().generate_report()diff --git a/tests/integration/pipelex/fixtures/pipe_job_helpers.py b/tests/integration/pipelex/fixtures/pipe_job_helpers.pyindex 74337e8c6..0994c90d0 100644--- a/tests/integration/pipelex/fixtures/pipe_job_helpers.py+++ b/tests/integration/pipelex/fixtures/pipe_job_helpers.py@@ -6,7 +6,7 @@ from pipelex.core.interpreter.interpreter import PipelexInterpreterfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.pipes.pipe_abstract import PipeAbstract-from pipelex.hub import clear_current_library, get_library_manager, get_report_delegate, get_required_pipe, set_current_library+from pipelex.hub import clear_current_library, get_library_manager, get_required_pipe, set_current_libraryfrom pipelex.libraries.library_crate import LibraryCratefrom pipelex.pipe_run.pipe_job import PipeJobfrom pipelex.pipe_run.pipe_job_factory import PipeJobFactory@@ -77,12 +77,7 @@ def pipe_job_from_library(library.set_class_registry(scoped_registry)set_current_library(library_id=library_id)- # Open a reporting registry for live mode (required by the reporting manager)- needs_registry = not pipe_run_mode.is_drypipeline_run_id = _make_unique_pipeline_run_id()- if needs_registry:- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)-try:load_fn(library_id)@@ -94,8 +89,6 @@ def pipe_job_from_library(yield pipe_jobfinally:- if needs_registry:- get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)library_manager.teardown(library_id=library_id)clear_current_library()diff --git a/tests/integration/pipelex/pipeline/test_bundle_validator.py b/tests/integration/pipelex/pipeline/test_bundle_validator.pyindex 9a88f91ae..e8b8e4dd0 100644--- a/tests/integration/pipelex/pipeline/test_bundle_validator.py+++ b/tests/integration/pipelex/pipeline/test_bundle_validator.py@@ -1,13 +1,17 @@"""Integration tests for BundleValidator's real composition (Phase 2).Unlike the unit suite (which mocks the seams + the run primitive), these run a real pipe DRY-through the composed stack — ``prepare_pipe_job`` + a direct in-process ``PipeRun`` + the real-report registry + real telemetry — to pin the contracts mocks cannot: that a real DRY run-classifies SUCCESS, that the per-sweep registry is closed in ``finally`` (so the constant-``DRY_RUN_UNTITLED`` id does not collide on a second sweep), and that the sweep emits exactly one+through the composed stack — ``prepare_pipe_job`` + a direct in-process ``PipeRun`` + real+telemetry — to pin the contracts mocks cannot: that a real DRY run classifies SUCCESS, that each+sweep dry-runs its pipes under a UNIQUE per-sweep pipeline run id (and that concurrent sweeps+genuinely overlap and both succeed — the live registry that once made a constant id collide is+gone, so this is now a plain concurrency guarantee), and that the sweep emits exactly one``PIPE_DRY_RUN`` event with no stray runner ``PIPELINE_EXECUTE`` / ``PIPELINE_COMPLETE`` events."""+import asyncio+from typing import Any+import pytestfrom pytest_mock import MockerFixture@@ -17,6 +21,7 @@ from pipelex.hub import (get_required_pipe,get_telemetry_manager,)+from pipelex.pipeline import bundle_validatorfrom pipelex.pipeline.bundle_validator import BundleValidator, DryRunStatusfrom pipelex.pipeline.execution_seams import acquire_libraryfrom pipelex.system.telemetry.events import EventName@@ -89,10 +94,13 @@ class TestBundleValidatorIntegration:library_manager.teardown(library_id=library_id)clear_current_library()- async def test_two_consecutive_sweeps_no_registry_collision(self) -> None:- # The sweep opens a report registry keyed by the constant DRY_RUN_UNTITLED id. Without- # close_registry in `finally`, the second sweep's open_registry would raise "already exists"- # against the REAL report delegate — this asserts the close-in-finally actually fires.+ async def test_consecutive_sweeps_use_distinct_run_ids(self, mocker: MockerFixture) -> None:+ # Each sweep dry-runs its pipes under a UNIQUE per-sweep pipeline run id (a `dry_run_`-prefixed+ # uuid) threaded into prepare_pipe_job. With the live registry removed (usage rides on PipeOutput),+ # nothing accumulates per-run state, but the per-sweep id stays distinct so each run is individually+ # identifiable. Spy on the REAL prepare_pipe_job (mocker.spy still calls through) and assert the two+ # sweeps used DIFFERENT `dry_run_`-prefixed ids and both classified SUCCESS.+ prepare_spy = mocker.spy(bundle_validator, "prepare_pipe_job")library_manager = get_library_manager()library_id = "bv_two_sweeps_lib"_, qualified_main_pipe = acquire_library(library_id=library_id, mthds_contents=[_BV_MTHDS])@@ -108,6 +116,64 @@ class TestBundleValidatorIntegration:library_manager.teardown(library_id=library_id)clear_current_library()+ run_ids = [call.kwargs["pipeline_run_id"] for call in prepare_spy.call_args_list]+ assert len(run_ids) == 2+ assert all(run_id.startswith("dry_run_") for run_id in run_ids)+ assert run_ids[0] != run_ids[1]++ async def test_concurrent_sweeps_run_with_genuine_overlap(self, mocker: MockerFixture) -> None:+ # Two OVERLAPPING sweeps (concurrent `/validate` API requests) must both run to completion and+ # classify SUCCESS. A DRY sweep runs to completion without yielding to the event loop, so `gather`+ # alone would run the two sweeps sequentially and never overlap (in production the overlap comes+ # from yield points elsewhere in the request handler). We force the overlap deterministically: each+ # sweep PARKS inside prepare_pipe_job until both are in flight at once. We release only when two are+ # parked simultaneously and assert the PEAK simultaneous occupancy reached 2 — that is what proves+ # genuine overlap. It is robust to the sweep's pipe count: a sweep blocks on its FIRST prepare (it+ # cannot start a second pipe until the first returns), so each sweep contributes at most one to the+ # count; the gate cannot be tripped by one sweep alone. (Before Phase 4, the sweep keyed a report+ # registry on the process-global ReportingManager, and a constant id made the second sweep collide+ # with "already exists"; the live registry is now gone, so concurrent sweeps share no per-run state+ # at all.) The 5s wait_for is a safety net: if one sweep never reaches prepare (peak stays 1), it+ # times out into a clear failure rather than hanging forever.+ real_prepare = bundle_validator.prepare_pipe_job+ both_parked = asyncio.Event()+ parked = 0+ peak_parked = 0++ async def barrier_prepare(*args: Any, **kwargs: Any) -> Any:+ nonlocal parked, peak_parked+ parked += 1+ peak_parked = max(peak_parked, parked)+ if parked >= 2:+ both_parked.set()+ try:+ await asyncio.wait_for(both_parked.wait(), timeout=5)+ finally:+ parked -= 1+ return await real_prepare(*args, **kwargs)++ mocker.patch.object(bundle_validator, "prepare_pipe_job", side_effect=barrier_prepare)++ library_manager = get_library_manager()+ library_id = "bv_concurrent_sweeps_lib"+ _, qualified_main_pipe = acquire_library(library_id=library_id, mthds_contents=[_BV_MTHDS])+ try:+ assert qualified_main_pipe is not None+ pipe = get_required_pipe(pipe_code=qualified_main_pipe)+ first, second = await asyncio.gather(+ BundleValidator().validate_pipes([pipe], library_id=library_id),+ BundleValidator().validate_pipes([pipe], library_id=library_id),+ )+ assert first[pipe.pipe_ref].status.is_success+ assert second[pipe.pipe_ref].status.is_success+ finally:+ library_manager.teardown(library_id=library_id)+ clear_current_library()++ # Proof of genuine overlap: both sweeps were parked inside prepare_pipe_job at the same instant.+ # If they had run sequentially (the failure this test guards against), peak_parked would be 1.+ assert peak_parked == 2+async def test_cross_package_controller_skipped_not_aborting_sweep(self) -> None:# Finding #1 / D7: a controller whose branch references an UNLOADED cross-package sub-pipe# raises PipeNotFoundError in the step-1 validate_with_libraries pass. It must be recordeddiff --git a/tests/integration/pipelex/pipeline/test_cost_report_rendering.py b/tests/integration/pipelex/pipeline/test_cost_report_rendering.pynew file mode 100644index 000000000..f191ce4ca--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_cost_report_rendering.py@@ -0,0 +1,186 @@+"""Integration tests for the Phase-3 event-sourced cost-report rendering.++These cover the cutover from the live registry to ``PipeOutput`` as the cost source:++- A real (dry, no-spend) DIRECT run renders the cost table from ``pipe_output.tokens_usages``,+ and the usage assembled onto ``PipeOutput`` matches the still-present registry for the same run+ (numeric-fidelity guard during the transition; the registry is removed in Phase 4).+- ``--no-costs`` assembles no usage, so the renderer is a no-op (nothing printed).+- Hand-built usage proves the table and CSV channels render correct *non-zero* totals (the dry path+ reports zero-token synthetic usage, so non-zero totals can't come from a dry run).+"""++import io+from pathlib import Path++import pytest+from pytest_mock import MockerFixture+from rich.console import Console++from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.pipeline.runner import PipelexRunner+from pipelex.reporting.cost_report_renderer import render_run_cost_report+from pipelex.system.configuration.configs import NdjsonTracingConfig, PipelineExecutionConfig, TracingBackend++_DOMAIN = "cost_report_rendering"+_MTHDS = f"""+domain = "{_DOMAIN}"+description = "Minimal bundle for cost-report rendering tests"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = {{ type = "text", description = "Topic name" }}++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used to exercise cost-report rendering"+inputs = {{ subject = "Text" }}+output = "Topic"+prompt = "Echo the $subject as a topic"+"""+++def _recording_console() -> Console:+ return Console(record=True, file=io.StringIO(), width=400)+++def _non_zero_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ # INPUT 100 @ $1/M + OUTPUT 50 @ $2/M -> total cost 0.2000.+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},+ )+++def _free_model_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ # Real tokens, no unit costs (a local/free model) -> token counts but total cost 0.+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="free-model",+ inference_model_id="free-model-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={},+ )++[email protected](loop_scope="class")+class TestCostReportRendering:+ def _enable_ndjson_tracing(self, mocker: MockerFixture, traces_dir: str) -> None:+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", True)+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=traces_dir))++ def _config(self, *, generate_usage: bool) -> PipelineExecutionConfig:+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=False,+ generate_usage=generate_usage,+ mock_inputs=True,+ )++ async def _run(self, execution_config: PipelineExecutionConfig) -> PipeOutput:+ runner = PipelexRunner(pipe_run_mode=PipeRunMode.DRY, execution_config=execution_config)+ response = await runner.execute_pipeline(pipe_code="echo_topic", mthds_contents=[_MTHDS])+ return response.pipe_output++ async def test_renders_correct_non_zero_totals(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ """Hand-built usage: the console channel renders the model, totals, and total cost."""+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", True)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", False)+ console = _recording_console()+ mocker.patch("pipelex.cogt.usage.cost_registry.get_console", return_value=console)++ render_run_cost_report(pipeline_run_id="hand-built", tokens_usages=[_non_zero_usage(job_metadata)], is_generate_costs=True)++ rendered = console.export_text()+ assert "test-model" in rendered+ assert "Total" in rendered+ assert "0.2000" in rendered # 100 input @ $1/M + 50 output @ $2/M++ async def test_renders_free_model_tokens_with_zero_cost(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ """A free/zero-price model with real tokens IS rendered (token counts shown, cost 0.0000) — NOT+ suppressed like a dry run. Gating on zero cost alone would wrongly hide a real local-model run.+ """+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", True)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", False)+ console = _recording_console()+ mocker.patch("pipelex.cogt.usage.cost_registry.get_console", return_value=console)++ render_run_cost_report(pipeline_run_id="free-run", tokens_usages=[_free_model_usage(job_metadata)], is_generate_costs=True)++ rendered = console.export_text()+ assert "free-model" in rendered+ assert "Total" in rendered # the table is rendered, not suppressed+ assert "100" in rendered # input-joined token count still reported+ assert "0.0000" in rendered # cost is zero, but the table is shown anyway++ async def test_csv_channel_writes_file(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:+ """The CSV channel writes a populated report file under the configured directory."""+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", False)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", True)+ mocker.patch.object(reporting_config, "cost_report_dir_path", str(tmp_path))++ render_run_cost_report(pipeline_run_id="csv-run", tokens_usages=[_non_zero_usage(job_metadata)], is_generate_costs=True)++ csv_files = list(tmp_path.glob("*.csv"))+ assert len(csv_files) == 1+ contents = csv_files[0].read_text(encoding="utf-8")+ assert "test-model" in contents++ async def test_direct_dry_run_suppresses_table_but_populates_usage(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """A DIRECT dry run assembles (zero-token) usage onto PipeOutput, but the renderer prints NOTHING+ because the run did no reportable work (zero tokens and zero cost) — only dry runs are suppressed;+ a free model with real tokens would still render.+ """+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_render")))+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", True)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", False)+ console = _recording_console()+ mocker.patch("pipelex.cogt.usage.cost_registry.get_console", return_value=console)++ pipe_output = await self._run(self._config(generate_usage=True))++ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1++ render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=True,+ )+ # Dry runs emit zero-token synthetic usage -> no tokens and no cost -> no table printed.+ assert console.export_text() == ""++ async def test_no_costs_renders_nothing(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """--no-costs assembles no usage, so the renderer prints nothing."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_no_costs")))+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", True)+ console = _recording_console()+ mocker.patch("pipelex.cogt.usage.cost_registry.get_console", return_value=console)++ pipe_output = await self._run(self._config(generate_usage=False))++ assert pipe_output.tokens_usages is None++ render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=False,+ )+ assert console.export_text() == ""diff --git a/tests/integration/pipelex/pipeline/test_direct_tracing_assembly.py b/tests/integration/pipelex/pipeline/test_direct_tracing_assembly.pynew file mode 100644index 000000000..a524a177d--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_direct_tracing_assembly.py@@ -0,0 +1,115 @@+"""Integration tests for the Phase-2 DIRECT tracing assembly onto PipeOutput.++A real (dry, no-spend) run through :meth:`PipelexRunner.execute_pipeline` must ride the assembled+artifacts back on ``pipe_output``:++- **costs on:** ``pipe_output.tokens_usages`` is populated from the run's emitted usage events.+- **costs-only (``--no-graph --costs``):** usage is populated but ``graph_spec`` stays None (F1 — the+ decoupled event stream must not produce a node-less GraphSpec under ``--no-graph``).+- **graph-only (``--graph --no-costs``):** ``graph_spec`` is populated but ``tokens_usages`` stays None.+- **E1:** costs-only mode does not pay the per-pipe stuff serialization that only feeds the GraphSpec.++These run fully dry (``PipeRunMode.DRY`` + ``mock_inputs``) so no inference happens; the dry content+generator still reports token usage inline, which is what populates the usage stream.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.stuffs.stuff_content import StuffContent+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipeline.runner import PipelexRunner+from pipelex.system.configuration.configs import NdjsonTracingConfig, PipelineExecutionConfig, TracingBackend++_DIRECT_DOMAIN = "direct_tracing_assembly"+_DIRECT_MTHDS = f"""+domain = "{_DIRECT_DOMAIN}"+description = "Minimal bundle for DIRECT tracing assembly tests"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = {{ type = "text", description = "Topic name" }}++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used to exercise DIRECT tracing assembly"+inputs = {{ subject = "Text" }}+output = "Topic"+prompt = "Echo the $subject as a topic"+"""+++def _config(*, generate_graph: bool, generate_usage: bool, full_data: bool = False) -> PipelineExecutionConfig:+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=generate_graph,+ generate_usage=generate_usage,+ mock_inputs=True,+ force_include_full_data=full_data or None,+ )++[email protected](loop_scope="class")+class TestDirectTracingAssembly:+ def _enable_ndjson_tracing(self, mocker: MockerFixture, traces_dir: str) -> None:+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", True)+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=traces_dir))++ async def _run(self, execution_config: PipelineExecutionConfig) -> PipeOutput:+ runner = PipelexRunner(pipe_run_mode=PipeRunMode.DRY, execution_config=execution_config)+ response = await runner.execute_pipeline(pipe_code="echo_topic", mthds_contents=[_DIRECT_MTHDS])+ return response.pipe_output++ async def test_costs_on_populates_tokens_usages(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_costs_on")))++ pipe_output = await self._run(_config(generate_graph=True, generate_usage=True))++ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1+ assert pipe_output.graph_spec is not None+ assert pipe_output.usage_assembly_error is None++ async def test_costs_only_leaves_graph_spec_none(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """F1 (DIRECT): a costs-only run must not produce a node-less GraphSpec under --no-graph."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_costs_only")))++ pipe_output = await self._run(_config(generate_graph=False, generate_usage=True))++ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1+ assert pipe_output.graph_spec is None++ async def test_graph_only_leaves_tokens_usages_none(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_graph_only")))++ pipe_output = await self._run(_config(generate_graph=True, generate_usage=False))++ assert pipe_output.graph_spec is not None+ assert pipe_output.tokens_usages is None++ async def test_graph_mode_serializes_stuff_payloads(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """Baseline for E1: with graph events + full data inclusion, the per-pipe HTML payload IS built."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_graph_full")))+ render_spy = mocker.spy(StuffContent, "rendered_pretty_html")++ await self._run(_config(generate_graph=True, generate_usage=True, full_data=True))++ assert render_spy.call_count >= 1++ async def test_costs_only_skips_stuff_serialization(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """E1: in costs-only mode the per-pipe HTML payload (graph-only) is never built, even with full data inclusion."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_costs_full")))+ render_spy = mocker.spy(StuffContent, "rendered_pretty_html")++ pipe_output = await self._run(_config(generate_graph=False, generate_usage=True, full_data=True))++ assert render_spy.call_count == 0+ # And the usage still rode back — the optimization didn't break cost reporting.+ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1diff --git a/tests/integration/pipelex/pipeline/test_execution_seams.py b/tests/integration/pipelex/pipeline/test_execution_seams.pyindex 4bcc4ea5c..88a65143b 100644--- a/tests/integration/pipelex/pipeline/test_execution_seams.py+++ b/tests/integration/pipelex/pipeline/test_execution_seams.py@@ -45,7 +45,7 @@ prompt = "Echo the $subject as a topic"def _dry_mock_config() -> PipelineExecutionConfig:- return get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,mock_inputs=True,)@@ -182,7 +182,7 @@ class TestExecutionSeams:normalize_spy = mocker.spy(execution_seams_module, "normalize_data_urls_to_storage")normalize_config = (get_config()- .pipelex.pipeline_execution_config.with_graph_config_overrides(+ .pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,mock_inputs=False,)diff --git a/tests/integration/pipelex/pipeline/test_mock_inference_direct.py b/tests/integration/pipelex/pipeline/test_mock_inference_direct.pynew file mode 100644index 000000000..40897c954--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_mock_inference_direct.py@@ -0,0 +1,136 @@+"""Integration tests for ``--mock-inference`` in DIRECT mode (Phase 5).++A ``--mock-inference`` run stays LIVE (real control flow, real templating) but fakes every AI call+at the cogt leaf, emitting *reportable* (non-zero) synthetic usage. These tests prove the end-to-end+contract without any provider spend:++- The leaf never reaches the LLM worker (``get_llm_worker`` is not called).+- Usage rides back on ``PipeOutput.tokens_usages`` under the ``mock_inference`` sentinel model, with+ non-zero tokens — so ``AggregatedCosts.has_reportable_usage`` is True and the cost report renders+ (unlike ``--dry-run``, whose zero-token usage Phase 3 deliberately suppresses).++Both the text path (``mock_llm_gen_text``) and the structured-object path (``mock_llm_gen_object``)+are covered. NDJSON tracing is enabled so the usage events assemble onto ``PipeOutput``.+"""++import io++import pytest+from pytest_mock import MockerFixture+from rich.console import Console++from pipelex.cogt.content_generation.dry_mock import MOCK_INFERENCE_MODEL_NAME+from pipelex.cogt.usage.cost_registry import CostRegistry+from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipeline.runner import PipelexRunner+from pipelex.reporting.cost_report_renderer import render_run_cost_report+from pipelex.system.configuration.configs import NdjsonTracingConfig, PipelineExecutionConfig, TracingBackend++_DOMAIN = "mock_inference_direct"+_MTHDS = f"""+domain = "{_DOMAIN}"+description = "Minimal bundle for --mock-inference DIRECT tests"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = {{ type = "text", description = "Topic name" }}++[pipe.write_text]+type = "PipeLLM"+description = "Pipe that outputs plain text"+inputs = {{ subject = "Text" }}+output = "Text"+prompt = "Write about $subject"++[pipe.write_topic]+type = "PipeLLM"+description = "Pipe that outputs a structured Topic"+inputs = {{ subject = "Text" }}+output = "Topic"+prompt = "Make a topic from $subject"+"""+++def _recording_console() -> Console:+ return Console(record=True, file=io.StringIO(), width=400)++[email protected](loop_scope="class")+class TestMockInferenceDirect:+ def _enable_ndjson_tracing(self, mocker: MockerFixture, traces_dir: str) -> None:+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", True)+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=traces_dir))++ def _config(self) -> PipelineExecutionConfig:+ # Costs on (default) so usage assembles; mock_inputs fills the `subject` input for the LIVE run.+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=False,+ generate_usage=True,+ mock_inputs=True,+ )++ async def _run_mock_inference(self, pipe_code: str) -> PipeOutput:+ # pipe_run_mode left as None -> LIVE; is_mock_inference fakes only the AI leaf.+ runner = PipelexRunner(is_mock_inference=True, execution_config=self._config())+ response = await runner.execute_pipeline(pipe_code=pipe_code, mthds_contents=[_MTHDS])+ return response.pipe_output++ async def test_text_pipe_mocks_leaf_and_assembles_reportable_usage(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """A LIVE text pipe under --mock-inference never calls the worker, yet assembles non-zero usage."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_mock_text")))+ worker_spy = mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")++ pipe_output = await self._run_mock_inference("write_text")++ worker_spy.assert_not_called() # no provider call -> no spend+ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1+ assert all(usage.inference_model_name == MOCK_INFERENCE_MODEL_NAME for usage in pipe_output.tokens_usages)++ aggregated = CostRegistry.aggregate_costs(pipe_output.tokens_usages)+ assert aggregated.has_reportable_usage # non-zero tokens -> the report is NOT suppressed+ assert aggregated.total_nb_tokens > 0++ async def test_object_pipe_mocks_leaf_and_assembles_reportable_usage(+ self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture+ ) -> None:+ """The structured-object path (mock_llm_gen_object) also fakes the leaf and reports non-zero usage."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_mock_object")))+ worker_spy = mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")++ pipe_output = await self._run_mock_inference("write_topic")++ worker_spy.assert_not_called()+ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1+ aggregated = CostRegistry.aggregate_costs(pipe_output.tokens_usages)+ assert aggregated.has_reportable_usage+ # The structured output was produced by the mock object builder (a Topic with a name).+ main_stuff = pipe_output.working_memory.get_optional_main_stuff()+ assert main_stuff is not None++ async def test_cost_report_renders_for_mock_inference(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """The end-of-run cost report RENDERS for a --mock-inference run (non-suppressed), under the sentinel model."""+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_mock_render")))+ mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", True)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", False)+ console = _recording_console()+ mocker.patch("pipelex.cogt.usage.cost_registry.get_console", return_value=console)++ pipe_output = await self._run_mock_inference("write_text")++ render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=True,+ )+ rendered = console.export_text()+ assert MOCK_INFERENCE_MODEL_NAME in rendered # the table is shown (NOT suppressed like a dry run)+ assert "Total" in rendereddiff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.pyindex 571261d23..14dcdc9b6 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_characterization.py@@ -34,13 +34,11 @@ What is pinned here:a ``library_id``, which it never did). ``acquire_library`` now owns load-timeteardown and the recomposed ``try``/``finally`` owns the post-acquire window,matching the already-hardened ``validate_bundle``.-- **Failure after ``open_registry`` closes the registry:** a failure between- ``open_registry`` and a successful return (here ``prepare_pipe_job`` raising)- closes the per-run ``UsageRegistry`` the wrapper opened, so a long-lived runner- does not leak one registry per failed run. A failure *before* ``open_registry``- must NOT call ``close_registry`` (a ``registry_opened`` guard keeps the finally from- closing a registry it never opened; ``close_registry`` is idempotent via- ``pop(..., None)``, so this is a semantic guard, not ``KeyError`` avoidance).+- **Success path leaves no per-run reporting state (Phase 4 — leak fixed by removal):** a+ successful run through ``execute_pipeline`` leaves the ``ReportingManager`` with no accumulated+ per-run state — the per-run event-log context is cleared on the way out, and the live usage+ registry has been removed entirely, so the success-path leak it used to suffer (a per-run+ registry opened in setup and never closed) is structurally impossible.- **Failure after acquire restores the outer current-library:** when the callerhad an outer current-library set, a post-acquire failure restores it ratherthan clobbering it to ``None`` — mirroring ``acquire_library``'s own load-failure@@ -65,9 +63,10 @@ from pipelex.hub import ()from pipelex.libraries.pipe.exceptions import PipeNotFoundErrorfrom pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline import pipeline_run_setup as pipeline_run_setup_modulefrom pipelex.pipeline.pipeline_run_setup import pipeline_run_setup-from pipelex.system.configuration.configs import PipelineExecutionConfig+from pipelex.pipeline.runner import PipelexRunner+from pipelex.reporting.reporting_manager import ReportingManager+from pipelex.system.configuration.configs import NdjsonTracingConfig, PipelineExecutionConfig, TracingBackendfrom pipelex.system.telemetry.events import EventName_CHAR_DOMAIN = "prs_char"@@ -91,14 +90,14 @@ prompt = "Echo the $subject as a topic"def _dry_mock_config() -> PipelineExecutionConfig:- return get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,mock_inputs=True,)def _no_mock_config() -> PipelineExecutionConfig:- return get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,mock_inputs=False,)@@ -211,45 +210,36 @@ class TestPipelineRunSetupCharacterization:assert teardown_spy.call_count == teardown_before + 1assert get_current_library_id_or_none() == prev_library_id- async def test_failure_after_open_registry_closes_registry(self, mocker: MockerFixture) -> None:- # The seam reorder put working-memory assembly (prepare_pipe_job) AFTER open_registry. A- # failure there must close the registry the wrapper opened, or a long-lived runner leaks one- # UsageRegistry per failed run. Patch the module-level prepare_pipe_job to raise (it runs after- # open_registry) and assert the registry opened for this run is closed exactly once.- open_spy = mocker.spy(get_report_delegate(), "open_registry")- close_spy = mocker.spy(get_report_delegate(), "close_registry")- mocker.patch.object(pipeline_run_setup_module, "prepare_pipe_job", side_effect=RuntimeError("boom in prepare_pipe_job"))-- with pytest.raises(RuntimeError):- await pipeline_run_setup(- execution_config=_dry_mock_config(),- mthds_contents=[_CHAR_MTHDS],- pipe_code="echo_topic",- pipe_run_mode=PipeRunMode.DRY,- )-- assert open_spy.call_count == 1- opened_run_id = open_spy.call_args.kwargs["pipeline_run_id"]- assert close_spy.call_count == 1- assert close_spy.call_args.kwargs["pipeline_run_id"] == opened_run_id-- async def test_failure_before_open_registry_does_not_close_registry(self, mocker: MockerFixture) -> None:- # The wrapper finally also runs for failures BEFORE open_registry (pipe resolution here, which- # precedes it). The registry_opened guard ensures close_registry is only called for a registry we- # actually opened, so it must NOT be called on this pre-open failure, and the original error must- # propagate intact. (close_registry is idempotent via pop(..., None); the guard is about not- # touching state we never created, not KeyError avoidance.)- close_spy = mocker.spy(get_report_delegate(), "close_registry")-- with pytest.raises(PipeNotFoundError):- await pipeline_run_setup(- execution_config=_dry_mock_config(),- mthds_contents=[_CHAR_MTHDS],- pipe_code="absent_pipe",- pipe_run_mode=PipeRunMode.DRY,- )-- assert close_spy.call_count == 0+ async def test_success_path_leaves_no_per_run_reporting_state(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ # Phase 4: the live usage registry is removed entirely, so the success-path leak (a per-run+ # registry opened in pipeline_run_setup and never closed) is structurally impossible. Run a real+ # DRY pipeline through execute_pipeline with tracing + costs on (so a usage event-log context IS+ # registered during the run), then assert the ReportingManager carries NO per-run state afterward:+ # the registry concept no longer exists, and the event-log context was cleared on the success path.+ traces_dir = str(tmp_path_factory.mktemp("char_no_leak"))+ tracing_config = get_config().pipelex.tracing_config+ mocker.patch.object(tracing_config, "is_enabled", True)+ mocker.patch.object(tracing_config, "backend", TracingBackend.NDJSON)+ mocker.patch.object(tracing_config, "ndjson", NdjsonTracingConfig(traces_dir=traces_dir))++ delegate = get_report_delegate()+ # The registry attribute is gone for good — the leak cannot recur.+ assert not hasattr(delegate, "_usage_registries")++ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=False,+ generate_usage=True,+ mock_inputs=True,+ )+ runner = PipelexRunner(pipe_run_mode=PipeRunMode.DRY, execution_config=execution_config)+ response = await runner.execute_pipeline(pipe_code="echo_topic", mthds_contents=[_CHAR_MTHDS])++ assert not hasattr(delegate, "_usage_registries")+ # This run registered a usage event-log context (tracing + costs on); the success path must have+ # cleared it. Assert against THIS run's key rather than global emptiness — _event_log_contexts is a+ # process-global singleton that sibling tests on the same xdist worker can populate.+ if isinstance(delegate, ReportingManager):+ assert response.pipeline_run_id not in delegate._event_log_contexts # noqa: SLF001 # pyright: ignore[reportPrivateUsage]async def test_failure_after_acquire_restores_outer_current_library(self, load_empty_library: Callable[[], str]) -> None:# acquire_library restores the outer current-library on its own load failure; the wrapper'sdiff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.pynew file mode 100644index 000000000..d44695bab--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py@@ -0,0 +1,117 @@+"""Integration tests for the Phase-1 emit-gate wiring in :func:`pipeline_run_setup`.++Pins that the two concerns are decoupled at setup time over the shared event-log transport:++- **costs-only (``--no-graph --costs``):** the tracer is still opened (so node ids are minted), the+ returned trace_context carries ``emit_graph_events=False`` / ``emit_usage_events=True``, and the+ usage event-log context IS registered on the report delegate (``set_event_log``).+- **graph-only (``--graph --no-costs``):** the returned trace_context carries+ ``emit_graph_events=True`` / ``emit_usage_events=False`` and the usage event-log context is NOT+ registered (so usage events are suppressed).++These assert the gating decisions D4/D5/D6 land on the run's TraceContext and the report delegate,+without needing an actual inference run.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.hub import clear_current_library, get_library_manager, get_report_delegate+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup+from pipelex.system.configuration.configs import NdjsonTracingConfig, PipelineExecutionConfig, TracingBackend++_GATE_DOMAIN = "prs_emit_gates"+_GATE_MTHDS = f"""+domain = "{_GATE_DOMAIN}"+description = "Minimal bundle for pipeline_run_setup emit-gate tests"++[concept.Topic]+description = "A topic"++[concept.Topic.structure]+name = {{ type = "text", description = "Topic name" }}++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used only to set up a PipeJob"+inputs = {{ subject = "Text" }}+output = "Topic"+prompt = "Echo the $subject as a topic"+"""+++def _config(*, generate_graph: bool, generate_usage: bool) -> PipelineExecutionConfig:+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=generate_graph,+ generate_usage=generate_usage,+ mock_inputs=True,+ )+++def _cleanup(pipeline_run_id: str, library_id: str) -> None:+ """Tear down the per-run state pipeline_run_setup leaves open on the success path.++ The success path intentionally does NOT close the tracer / event-log context; execute_pipeline+ normally owns this teardown. Since this test calls pipeline_run_setup directly, it cleans up itself.+ """+ get_report_delegate().clear_event_log(context_key=pipeline_run_id)+ tracer_manager = GraphTracerManager.get_instance()+ if tracer_manager is not None:+ tracer_manager.close_tracer(pipeline_run_id)+ get_library_manager().teardown(library_id=library_id)+ clear_current_library()++[email protected](loop_scope="class")+class TestPipelineRunSetupEmitGates:+ def _enable_ndjson_tracing(self, mocker: MockerFixture, traces_dir: str) -> None:+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", True)+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=traces_dir))++ async def test_costs_only_registers_usage_log_and_sets_flags(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_costs_only")))+ set_event_log_spy = mocker.spy(get_report_delegate(), "set_event_log")++ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ execution_config=_config(generate_graph=False, generate_usage=True),+ mthds_contents=[_GATE_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ )+ try:+ trace_context = pipe_job.job_metadata.trace_context+ # The tracer is opened in costs-only mode (D5), so the context exists and mints node ids.+ assert trace_context is not None+ assert trace_context.emit_graph_events is False+ assert trace_context.emit_usage_events is True+ # The usage event-log context IS registered for this run (costs on).+ registered_keys = [call.kwargs.get("context_key") for call in set_event_log_spy.call_args_list]+ assert pipeline_run_id in registered_keys+ finally:+ _cleanup(pipeline_run_id, library_id)++ async def test_graph_only_does_not_register_usage_log(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ self._enable_ndjson_tracing(mocker, str(tmp_path_factory.mktemp("traces_graph_only")))+ set_event_log_spy = mocker.spy(get_report_delegate(), "set_event_log")++ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(+ execution_config=_config(generate_graph=True, generate_usage=False),+ mthds_contents=[_GATE_MTHDS],+ pipe_code="echo_topic",+ pipe_run_mode=PipeRunMode.DRY,+ )+ try:+ trace_context = pipe_job.job_metadata.trace_context+ assert trace_context is not None+ assert trace_context.emit_graph_events is True+ assert trace_context.emit_usage_events is False+ # No usage event-log context registered for this run (costs off) -> usage events suppressed.+ registered_keys = [call.kwargs.get("context_key") for call in set_event_log_spy.call_args_list]+ assert pipeline_run_id not in registered_keys+ finally:+ _cleanup(pipeline_run_id, library_id)diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.pyindex 9c3267c44..d1a967131 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py@@ -36,7 +36,7 @@ prompt = "Echo the $subject as a topic"class TestPipelineRunSetupRequestId:async def test_request_id_threads_onto_job_metadata(self) -> None:"""``pipeline_run_setup(..., request_id=...)`` puts the value on ``job_metadata.request_id``."""- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,)pipe_job, _, _ = await pipeline_run_setup(@@ -49,7 +49,7 @@ class TestPipelineRunSetupRequestId:async def test_request_id_absent_defaults_to_none(self) -> None:"""Omitting ``request_id`` leaves ``job_metadata.request_id`` as ``None``."""- execution_config = get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,)pipe_job, _, _ = await pipeline_run_setup(diff --git a/tests/integration/pipelex/pipeline/test_runner_library_lifecycle.py b/tests/integration/pipelex/pipeline/test_runner_library_lifecycle.pyindex ddcf59040..1bd6490b5 100644--- a/tests/integration/pipelex/pipeline/test_runner_library_lifecycle.py+++ b/tests/integration/pipelex/pipeline/test_runner_library_lifecycle.py@@ -63,7 +63,7 @@ prompt = "Echo the $subject as a topic"def _dry_mock_config() -> PipelineExecutionConfig:- return get_config().pipelex.pipeline_execution_config.with_graph_config_overrides(+ return get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=False,mock_inputs=True,)diff --git a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.pynew file mode 100644index 000000000..dd5c750df--- /dev/null+++ b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py@@ -0,0 +1,62 @@+"""Integration test for the ``is_mock_inference`` hard guard in PipeSearch.++Web search has no leaf-level mock, and unlike the LLM/img-gen/extract operators it has no+``cogt/content_generation`` leaf — its provider spend happens directly in+``PipeSearch._live_run_operator_pipe``. Under ``--mock-inference`` (``run_mode=LIVE`` ++``JobMetadata.is_mock_inference``) the operator takes that live path, so the guard lives there and must+fail loud before the search worker is built. ``MockInferenceUnsupportedError`` is a plain+``PipelexError`` (not a model-availability error), so ``PipeOperator._live_run_pipe`` does not re-wrap+it — it propagates unchanged through the router.+"""++from typing import Callable++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.hub import get_pipe_library, get_pipe_router+from pipelex.pipe_operators.search.pipe_search import PipeSearch+from pipelex.pipe_operators.search.pipe_search_blueprint import PipeSearchBlueprint+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory+from pipelex.pipe_run.pipe_run_params import PipeRunMode+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.pipeline.job_metadata import JobMetadata++[email protected](loop_scope="class")+class TestPipeSearchMockInferenceGuard:+ async def test_mock_inference_raises_before_building_search_worker(+ self,+ job_metadata: JobMetadata,+ load_empty_library: Callable[[], None],+ mocker: MockerFixture,+ ) -> None:+ """A LIVE PipeSearch run with is_mock_inference=True raises the guard before any provider call."""+ load_empty_library()++ worker_factory_spy = mocker.patch("pipelex.pipe_operators.search.pipe_search.SearchWorkerFactory.make_search_worker")++ pipe = PipeFactory[PipeSearch].make_from_blueprint(+ domain_code="generic",+ pipe_code="adhoc_for_test_search_mock_inference_guard",+ blueprint=PipeSearchBlueprint(+ description="Search mock-inference guard test",+ output=NativeConceptCode.SEARCH_RESULT,+ prompt="Search for recent technology news.",+ ),+ )+ get_pipe_library().add_new_pipe(pipe)++ pipe_job = PipeJobFactory.make_pipe_job(+ pipe=pipe,+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),+ job_metadata=job_metadata.model_copy(update={"is_mock_inference": True}),+ )++ with pytest.raises(MockInferenceUnsupportedError):+ await get_pipe_router().run(pipe_job=pipe_job)++ worker_factory_spy.assert_not_called() # no provider call -> no spenddiff --git a/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_e2e.py b/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_e2e.pyindex c9123287c..06d9b9af6 100644--- a/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_e2e.py+++ b/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_e2e.py@@ -2,9 +2,10 @@ from pathlib import Pathfrom typing import Callableimport pytest+from pytest_mock import MockerFixture, MockTypefrom pipelex import pretty_print-from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.stuffs.stuff_factory import StuffFactory@@ -16,22 +17,17 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import PipeRunModefrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadata-from pipelex.reporting.reporting_manager import ReportingManagerfrom tests.integration.pipelex.pipes.operator.pipe_structure.test_structures_basic import RestaurantReview-def _count_llm_calls_for(pipeline_run_id: str) -> int:- """Count LLM completion records the reporting manager has accumulated for a run.+def _count_llm_report_calls(report_spy: MockType) -> int:+ """Count the LLM inference jobs reported during a run.- Each successful PipeLLM/PipeStructure completion appends one LLMTokensUsage to the- registry, so this gives us a direct count of LLM calls. Skipped under non-ReportingManager- delegates (e.g. ReportingNoOp), which never accumulate records.+ Each successful PipeLLM/PipeStructure completion reports exactly one LLMJob to the reporting+ delegate (the canonical per-call hook), so spying on report_inference_job gives a direct count+ of LLM calls — the observation point now that the per-run usage registry has been removed."""- delegate = get_report_delegate()- if not isinstance(delegate, ReportingManager):- return 0- registry = delegate._get_or_create_registry(pipeline_run_id) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- return sum(1 for usage in registry.get_current_tokens_usage() if isinstance(usage, LLMTokensUsage))+ return sum(1 for call in report_spy.call_args_list if isinstance(call.kwargs.get("inference_job"), LLMJob))def _assert_restaurant_review_shape(review: RestaurantReview) -> None:@@ -85,6 +81,7 @@ class TestPreliminaryTextE2E:)async def test_preliminary_text_runs_through_elaborated_pipes(self,+ mocker: MockerFixture,job_metadata: JobMetadata,pipe_run_mode: PipeRunMode,load_test_library: Callable[[list[Path]], None],@@ -117,6 +114,7 @@ class TestPreliminaryTextE2E:job_metadata=job_metadata,working_memory=working_memory,)+ report_spy = mocker.spy(get_report_delegate(), "report_inference_job")pipe_output = await get_pipe_router().run(pipe_job=pipe_job)pretty_print(pipe_output, title=f"preliminary_text e2e output ({pipe_code})")@@ -125,7 +123,7 @@ class TestPreliminaryTextE2E:assert pipe_output.working_memory.get_stuff("draft_text") is not Noneif pipe_run_mode.is_live:# Exactly two LLM calls: one for the draft text, one for the structuring step.- assert _count_llm_calls_for(job_metadata.pipeline_run_id) == 2+ assert _count_llm_report_calls(report_spy) == 2assert pipe_output.main_stuff.concept.code == "RestaurantReview"if expected_is_list:items = pipe_output.main_stuff_as_items(item_type=RestaurantReview)diff --git a/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_inline_e2e.py b/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_inline_e2e.pyindex 2b31c0aa5..05215d10b 100644--- a/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_inline_e2e.py+++ b/tests/integration/pipelex/pipes/operator/pipe_structure/test_preliminary_text_inline_e2e.py@@ -2,9 +2,10 @@ from pathlib import Pathfrom typing import Any, Callableimport pytest+from pytest_mock import MockerFixture, MockTypefrom pipelex import pretty_print-from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.stuffs.structured_content import StructuredContent@@ -17,15 +18,10 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import PipeRunModefrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadata-from pipelex.reporting.reporting_manager import ReportingManager-def _count_llm_calls_for(pipeline_run_id: str) -> int:- delegate = get_report_delegate()- if not isinstance(delegate, ReportingManager):- return 0- registry = delegate._get_or_create_registry(pipeline_run_id) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- return sum(1 for usage in registry.get_current_tokens_usage() if isinstance(usage, LLMTokensUsage))+def _count_llm_report_calls(report_spy: MockType) -> int:+ return sum(1 for call in report_spy.call_args_list if isinstance(call.kwargs.get("inference_job"), LLMJob))def _assert_hiking_trip_report_shape(content: Any) -> None: # pyright: ignore[reportExplicitAny, reportAny]@@ -84,6 +80,7 @@ class TestPreliminaryTextInlineE2E:)async def test_preliminary_text_with_inline_structure(self,+ mocker: MockerFixture,job_metadata: JobMetadata,pipe_run_mode: PipeRunMode,load_test_library: Callable[[list[Path]], None],@@ -115,6 +112,7 @@ class TestPreliminaryTextInlineE2E:job_metadata=job_metadata,working_memory=working_memory,)+ report_spy = mocker.spy(get_report_delegate(), "report_inference_job")pipe_output = await get_pipe_router().run(pipe_job=pipe_job)pretty_print(pipe_output, title=f"preliminary_text inline-structure output ({pipe_code})")@@ -123,7 +121,7 @@ class TestPreliminaryTextInlineE2E:assert pipe_output.working_memory.get_stuff("draft_text") is not Noneif pipe_run_mode.is_live:# Exactly two LLM calls: one for the draft text, one for the structuring step.- assert _count_llm_calls_for(job_metadata.pipeline_run_id) == 2+ assert _count_llm_report_calls(report_spy) == 2assert pipe_output.main_stuff.concept.code == "HikingTripReport"if expected_is_list:items = pipe_output.main_stuff_as_items(item_type=StructuredContent)diff --git a/tests/integration/pipelex/temporal/content_generation/conftest.py b/tests/integration/pipelex/temporal/content_generation/conftest.pyindex b894903fc..71df3dd0d 100644--- a/tests/integration/pipelex/temporal/content_generation/conftest.py+++ b/tests/integration/pipelex/temporal/content_generation/conftest.py@@ -1,11 +1,10 @@-from collections.abc import Generatorfrom typing import castimport pytestfrom pytest import FixtureRequestfrom pipelex.config import get_config-from pipelex.hub import get_class_registry, get_report_delegate+from pipelex.hub import get_class_registryfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.temporal.test_extras.temporal_registry_test_models import Personfrom tests.integration.pipelex.temporal.test_utils import rprint@@ -33,15 +32,13 @@ def register_test_temporal_models():@pytest.fixture-def tprl_job_metadata(request: FixtureRequest) -> Generator[JobMetadata, None, None]:- """Provide a JobMetadata instance with an open reporting registry.+def tprl_job_metadata(request: FixtureRequest) -> JobMetadata:+ """Provide a JobMetadata instance.Uses the test function name as pipeline_run_id for stable test identification."""pipeline_run_id = cast("str", request.node.originalname) # pyright: ignore[reportUnknownMemberType]- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)- yield JobMetadata(+ return JobMetadata(user_id="test",pipeline_run_id=pipeline_run_id,)- get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)diff --git a/tests/integration/pipelex/temporal/tracing/conftest.py b/tests/integration/pipelex/temporal/tracing/conftest.pyindex a3dd60339..23804c8d8 100644--- a/tests/integration/pipelex/temporal/tracing/conftest.py+++ b/tests/integration/pipelex/temporal/tracing/conftest.py@@ -72,7 +72,7 @@ def enable_tracing(tracing_tmp_dir: Path) -> Generator[None, None, None]:@pytest.fixture(scope="class")def sequence_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for native_text_sequence (graph_context injected per-execution by execute_and_assemble)."""+ """PipeJob for native_text_sequence (trace_context injected per-execution by execute_and_assemble)."""yield from pipe_job_from_bundle(bundle_file=SequenceTracingTestData.BUNDLE_FILE,pipe_code=SequenceTracingTestData.PIPE_CODE,@@ -82,7 +82,7 @@ def sequence_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob,@pytest.fixture(scope="class")def parallel_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for temporal_parallel (graph_context injected per-execution by execute_and_assemble)."""+ """PipeJob for temporal_parallel (trace_context injected per-execution by execute_and_assemble)."""yield from pipe_job_from_bundle(bundle_file=ParallelTracingTestData.BUNDLE_FILE,pipe_code=ParallelTracingTestData.PIPE_CODE,@@ -92,7 +92,7 @@ def parallel_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob,@pytest.fixture(scope="class")def batch_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for temporal_batch (graph_context injected per-execution by execute_and_assemble)."""+ """PipeJob for temporal_batch (trace_context injected per-execution by execute_and_assemble)."""yield from pipe_job_from_bundle(bundle_file=BatchTracingTestData.BUNDLE_FILE,pipe_code=BatchTracingTestData.PIPE_CODE,diff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.pyindex 903fce25c..1061cec10 100644--- a/tests/integration/pipelex/temporal/tracing/helpers.py+++ b/tests/integration/pipelex/temporal/tracing/helpers.py@@ -19,8 +19,8 @@ from pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.config import get_configfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graph_context import GraphContextfrom pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, NodeStatus, PipelineRef+from pipelex.graph.trace_context import TraceContextfrom pipelex.hub import get_report_delegatefrom pipelex.pipe_run.pipe_job import PipeJobfrom pipelex.reporting.reporting_manager import ReportingManager@@ -59,13 +59,22 @@ def route_activities_to(queue: str, activity_names: Iterable[str]) -> Generator[worker_config.activity_queues[activity_name] = original-def inject_graph_context(pipe_job: PipeJob, pipeline_run_id: str) -> PipeJob:- """Deep-copy a PipeJob and inject a GraphContext onto its JobMetadata.+def inject_trace_context(+ pipe_job: PipeJob,+ pipeline_run_id: str,+ *,+ emit_graph_events: bool = True,+ emit_usage_events: bool = True,+) -> PipeJob:+ """Deep-copy a PipeJob and inject a TraceContext onto its JobMetadata.Also overrides pipeline_run_id (must not be the dry-run sentinel,since NdjsonEventLog uses it as a directory name).++ The emit flags default to True (the legacy "context present → emit both"+ behavior); pass ``emit_graph_events=False`` to exercise costs-only mode."""- graph_context = GraphContext(+ trace_context = TraceContext(graph_id=pipeline_run_id,parent_node_id=None,node_sequence=0,@@ -76,10 +85,12 @@ def inject_graph_context(pipe_job: PipeJob, pipeline_run_id: str) -> PipeJob:stuff_html_content=False,error_stack_traces=False,),+ emit_graph_events=emit_graph_events,+ emit_usage_events=emit_usage_events,)new_metadata = pipe_job.job_metadata.model_copy(update={- "graph_context": graph_context,+ "trace_context": trace_context,"pipeline_run_id": pipeline_run_id,},)@@ -109,7 +120,7 @@ async def execute_and_assemble("""# Give each execution its own pipeline_run_id to avoid event accumulationexecution_run_id = f"tracing_exec_{uuid.uuid4().hex[:12]}"- execution_job = inject_graph_context(pipe_job, execution_run_id)+ execution_job = inject_trace_context(pipe_job, execution_run_id)task_queue = str(uuid.uuid4())workflow_id = str(uuid.uuid4())@@ -279,7 +290,7 @@ async def make_split_workers("act_jinja2_gen_text","act_extract_gen_extract_pages","act_render_page_views",- "act_assemble_graph",+ "act_assemble_tracing","act_deliver",],},diff --git a/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py b/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.pynew file mode 100644index 000000000..0fb69872c--- /dev/null+++ b/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py@@ -0,0 +1,96 @@+"""Temporal (cross-process) coverage for ``--mock-inference`` (Phase 5).++The DIRECT counterpart lives in+``tests/integration/pipelex/pipeline/test_mock_inference_direct.py``. This arm proves the part DIRECT+cannot: that ``JobMetadata.is_mock_inference`` survives the Temporal serialization boundary and the+**real** ``act_llm_gen_text`` activity honors it — faking the LLM at the cogt leaf inside the activity+body (no provider call), with the synthetic, reportable usage assembling back onto+``PipeOutput.tokens_usages`` via Step 2's ``act_assemble_tracing``.++Run LIVE (``PipeRunMode.LIVE``) so the workflow actually dispatches ``act_llm_gen_text`` (DRY would+swap ``ContentGeneratorDry`` in pre-dispatch and never reach the activity). The only thing faked is the+inference leaf, keyed on the per-run flag.++Like its sibling tracing tests this is ``gha_disabled`` by the directory conftest (the in-process+WorkflowEnvironment + concurrent pipe execution hangs under CI xdist); verify locally/serially.+"""++import uuid+from collections.abc import Generator+from typing import TYPE_CHECKING++import pytest+from temporalio.client import Client as TemporalClient++from pipelex.cogt.content_generation.dry_mock import MOCK_INFERENCE_MODEL_NAME+from pipelex.cogt.usage.cost_registry import CostRegistry+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg+from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output+from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_output import PipeOutput++[email protected](loop_scope="class")+class TestTemporalMockInference:+ @pytest.fixture+ def mock_inference_sequence_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:+ """A LIVE native_text_sequence job carrying is_mock_inference=True.++ LIVE forces the workflow to dispatch the real ``act_llm_gen_text``; the flag makes that+ activity's leaf fake the LLM. The same bundle is run LIVE (with a fake activity substitute)+ by ``test_split_worker_usage`` — here we run the *real* activity and let the flag do the mocking.+ """+ for pipe_job in pipe_job_from_bundle(+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,+ pipe_code=SequenceTracingTestData.PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ isolated_registry=is_class_registry_isolated,+ ):+ yield pipe_job.model_copy(+ update={"job_metadata": pipe_job.job_metadata.model_copy(update={"is_mock_inference": True})},+ )++ async def test_mock_inference_assembles_reportable_usage_cross_process(+ self,+ mock_inference_sequence_job: PipeJob,+ temporal_client: TemporalClient,+ ) -> None:+ """The flag crosses the boundary, the real activity mocks the leaf, and reportable usage rides back."""+ execution_run_id = f"mock_inf_{uuid.uuid4().hex[:12]}"+ # costs-only (no graph) mirrors the realistic --no-graph --costs default the skill validates.+ execution_job = inject_trace_context(+ mock_inference_sequence_job,+ execution_run_id,+ emit_graph_events=False,+ emit_usage_events=True,+ )+ assert execution_job.job_metadata.is_mock_inference, "inject_trace_context must preserve is_mock_inference"++ pipe_run_arg = PipeRunArg(pipe_job=execution_job).prepare_for_temporal()+ task_queue = str(uuid.uuid4())+ workflow_id = str(uuid.uuid4())++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRun.run,+ arg=pipe_run_arg,+ id=workflow_id,+ task_queue=task_queue,+ )+ rehydrate_pipe_output(pipe_output, pipe_job=mock_inference_sequence_job)++ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1+ # All usage came from the mocked leaf (the sentinel model) — proof no real provider ran.+ assert all(usage.inference_model_name == MOCK_INFERENCE_MODEL_NAME for usage in pipe_output.tokens_usages)+ # Non-zero synthetic usage -> a cost report would render (not suppressed like a dry run).+ assert CostRegistry.aggregate_costs(pipe_output.tokens_usages).has_reportable_usagediff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.pyindex e8b34e1f4..826f8a8c2 100644--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py@@ -33,10 +33,11 @@ from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouterfrom pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.ndjson_event_log import NdjsonEventLogfrom pipelex.tracing.trace_events import PipeStartEvent, UsageReportEvent+from pipelex.tracing.usage_aggregator import UsageAggregatorfrom tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundlefrom tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_outputfrom tests.integration.pipelex.temporal.tracing.helpers import (- inject_graph_context,+ inject_trace_context,make_split_workers,ndjson_files_for_run,)@@ -125,7 +126,7 @@ class TestSplitWorkerUsageEmission:"""Run the workflow on q_router; activities dispatch to q_runner."""q_router, q_runner = split_queuesexecution_run_id = f"split_exec_{uuid.uuid4().hex[:12]}"- execution_job = inject_graph_context(live_sequence_tracing_job, execution_run_id)+ execution_job = inject_trace_context(live_sequence_tracing_job, execution_run_id)workflow_id = f"wf_{uuid.uuid4().hex[:12]}"async with make_split_workers(temporal_client, q_router=q_router, q_runner=q_runner):@@ -203,3 +204,29 @@ class TestSplitWorkerUsageEmission:for key, count in counts.items():assert count == 1, f"Duplicate UsageReportEvent for (node_id, workflow_id)={key}: count={count}"++ async def test_runner_usage_aggregates_to_tokens_usages(+ self,+ live_sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ tracing_tmp_dir: Path,+ split_queues: tuple[str, str],+ route_llm_text_to_runner_queue: None, # noqa: ARG002+ ) -> None:+ """The cross-worker usage events feed the production ``UsageAggregator`` (the same path that+ rides ``tokens_usages`` back on PipeOutput): aggregating the read-back stream yields one usage+ record per emitted ``UsageReportEvent``, none dropped.+ """+ run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)++ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))+ try:+ events = reader.read_events(run_id)+ finally:+ reader.close()++ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]+ tokens_usages = UsageAggregator.aggregate(events)+ assert tokens_usages, "Expected the aggregator to surface at least one token-usage record"+ assert len(tokens_usages) == len(usage_events)+ assert tokens_usages == [evt.tokens_usage for evt in usage_events]diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py b/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.pynew file mode 100644index 000000000..6d040a779--- /dev/null+++ b/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py@@ -0,0 +1,124 @@+"""F1 invariant for the Temporal ``WfPipeRun`` assembly path (Step 2).++Mirrors the DIRECT-mode coverage in+``tests/integration/pipelex/pipeline/test_direct_tracing_assembly.py`` for the+TEMPORAL arm. A full ``WfPipeRun`` run wraps the real ``WfPipeRouter`` as a child+and then runs Step 2's ``act_assemble_tracing``; the assembled artifacts must ride+back on ``pipe_output`` per the run's emit flags:++- **costs-only (``emit_graph_events=False, emit_usage_events=True``):** usage rides+ back on ``tokens_usages`` but ``graph_spec`` stays None. This is the exact+ DIRECT/TEMPORAL symmetry F1 guards — both the ``WfPipeRun`` Step-2 dispatch+ (``assemble_graph=False`` → activity returns ``graph_spec=None``) and the+ ``WfPipeRouter`` finally assignment must avoid producing a GraphSpec under+ ``--no-graph``.+- **graph+costs:** both ``graph_spec`` and ``tokens_usages`` populated.+- **graph-only (``emit_usage_events=False``):** ``graph_spec`` populated,+ ``tokens_usages`` stays None.++Runs fully dry (``PipeRunMode.DRY``, the default for ``sequence_tracing_job``) so no+inference happens; the dry content generator reports token usage inline, which is+what populates the usage stream that ``act_assemble_tracing`` aggregates.+"""++import uuid++import pytest+from temporalio.client import Client as TemporalClient++from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg+from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output+from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context++[email protected](loop_scope="class")+class TestWfPipeRunTracingAssembly:+ async def _run(+ self,+ sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ *,+ emit_graph_events: bool,+ emit_usage_events: bool,+ ) -> PipeOutput:+ """Run a full ``WfPipeRun`` (router child + Step-2 assembly) on a single worker."""+ execution_run_id = f"wfrun_assembly_{uuid.uuid4().hex[:12]}"+ execution_job = inject_trace_context(+ sequence_tracing_job,+ execution_run_id,+ emit_graph_events=emit_graph_events,+ emit_usage_events=emit_usage_events,+ )+ pipe_run_arg = PipeRunArg(pipe_job=execution_job).prepare_for_temporal()+ task_queue = str(uuid.uuid4())+ workflow_id = str(uuid.uuid4())++ async with get_task_manager().make_worker(+ temporal_client,+ task_queue=task_queue,+ is_not_sandboxed=True,+ ):+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]+ workflow=WfPipeRun.run,+ arg=pipe_run_arg,+ id=workflow_id,+ task_queue=task_queue,+ )++ rehydrate_pipe_output(pipe_output, pipe_job=sequence_tracing_job)+ return pipe_output++ async def test_costs_only_leaves_graph_spec_none(+ self,+ sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ ) -> None:+ """F1 (TEMPORAL): a costs-only run rides usage back but must leave graph_spec None."""+ pipe_output = await self._run(+ sequence_tracing_job,+ temporal_client,+ emit_graph_events=False,+ emit_usage_events=True,+ )++ assert pipe_output.graph_spec is None, "costs-only WfPipeRun must not produce a GraphSpec under --no-graph"+ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1++ async def test_graph_and_costs_populate_both(+ self,+ sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ ) -> None:+ """Positive control: with both concerns on, both artifacts ride back (so the costs-only assertion is not vacuous)."""+ pipe_output = await self._run(+ sequence_tracing_job,+ temporal_client,+ emit_graph_events=True,+ emit_usage_events=True,+ )++ assert pipe_output.graph_spec is not None+ assert pipe_output.tokens_usages is not None+ assert len(pipe_output.tokens_usages) >= 1++ async def test_graph_only_leaves_tokens_usages_none(+ self,+ sequence_tracing_job: PipeJob,+ temporal_client: TemporalClient,+ ) -> None:+ """The mirror asymmetry: graph-only assembles a GraphSpec but never sets tokens_usages."""+ pipe_output = await self._run(+ sequence_tracing_job,+ temporal_client,+ emit_graph_events=True,+ emit_usage_events=False,+ )++ assert pipe_output.graph_spec is not None+ assert pipe_output.tokens_usages is Nonediff --git a/tests/integration/pipelex/temporal/workflows/conftest.py b/tests/integration/pipelex/temporal/workflows/conftest.pyindex 56a7a2514..88ce5103b 100644--- a/tests/integration/pipelex/temporal/workflows/conftest.py+++ b/tests/integration/pipelex/temporal/workflows/conftest.py@@ -1,19 +1,9 @@import uuid-from collections.abc import Generatorimport pytest-from pipelex.hub import get_report_delegate-@pytest.fixture-def workflow_run_id() -> Generator[str, None, None]:- """Provide a workflow run ID with an open reporting registry.-- Opens a registry for the run ID before the test and closes it after,- mirroring the job_metadata fixture pattern.- """- run_id = str(uuid.uuid4())- get_report_delegate().open_registry(pipeline_run_id=run_id)- yield run_id- get_report_delegate().close_registry(pipeline_run_id=run_id)+def workflow_run_id() -> str:+ """Provide a workflow run ID, mirroring the job_metadata fixture pattern."""+ return str(uuid.uuid4())diff --git a/tests/unit/pipelex/cli/test_mock_inference_cli_guard.py b/tests/unit/pipelex/cli/test_mock_inference_cli_guard.pynew file mode 100644index 000000000..b292b23ef--- /dev/null+++ b/tests/unit/pipelex/cli/test_mock_inference_cli_guard.py@@ -0,0 +1,35 @@+"""Unit tests for the ``--mock-inference`` CLI surface on the run subcommands.++Two cheap, no-boot checks: every run subcommand declares the option, and each rejects combining+``--mock-inference`` with ``--dry-run`` (the two are mutually exclusive — dry-run swaps the generator+pre-dispatch, so the leaf mock would be silently ignored). The guard runs at the top of each command,+before any filesystem access or Pipelex boot, so calling the command function directly is enough.+"""++from collections.abc import Callable+from inspect import signature++import pytest+import typer++from pipelex.cli.commands.run.bundle_cmd import run_bundle_cmd+from pipelex.cli.commands.run.method_cmd import run_method_cmd+from pipelex.cli.commands.run.pipe_cmd import run_pipe_cmd+++class TestMockInferenceCliGuard:+ @pytest.mark.parametrize("command", [run_pipe_cmd, run_bundle_cmd, run_method_cmd])+ def test_run_command_declares_mock_inference(self, command: Callable[..., None]) -> None:+ assert "mock_inference" in signature(command).parameters, f"{command.__name__} must declare a --mock-inference option"++ @pytest.mark.parametrize(+ ("command", "required_kwargs"),+ [+ (run_pipe_cmd, {"pipe_code": "some_pipe"}),+ (run_bundle_cmd, {"path": "some_path"}),+ (run_method_cmd, {"name": "some_method"}),+ ],+ )+ def test_rejects_mock_inference_combined_with_dry_run(self, command: Callable[..., None], required_kwargs: dict[str, str]) -> None:+ with pytest.raises(typer.Exit):+ command(dry_run=True, mock_inference=True, **required_kwargs)diff --git a/tests/unit/pipelex/cli/test_run_flag_combination.py b/tests/unit/pipelex/cli/test_run_flag_combination.pynew file mode 100644index 000000000..f35c546d1--- /dev/null+++ b/tests/unit/pipelex/cli/test_run_flag_combination.py@@ -0,0 +1,35 @@+"""Unit tests for ``validate_run_flag_combination`` — the single owner of which run-flag combinations are legal.++The ``pipe`` / ``method`` / ``bundle`` run subcommands all delegate their ``--dry-run`` / ``--mock-inference`` /+``--mock-inputs`` guarding to this one function, so the three can't drift (review F3). The CLI-level wiring —+that each subcommand actually calls it — is pinned separately in ``test_mock_inference_cli_guard.py``; here we+pin the full truth table of the validator itself.+"""++import pytest+import typer++from pipelex.cli.commands.run._run_core import validate_run_flag_combination # noqa: PLC2701+++class TestRunFlagCombination:+ def test_mock_inputs_without_dry_run_is_rejected(self) -> None:+ with pytest.raises(typer.Exit):+ validate_run_flag_combination(dry_run=False, mock_inference=False, mock_inputs=True)++ def test_mock_inference_with_dry_run_is_rejected(self) -> None:+ with pytest.raises(typer.Exit):+ validate_run_flag_combination(dry_run=True, mock_inference=True, mock_inputs=False)++ @pytest.mark.parametrize(+ ("dry_run", "mock_inference", "mock_inputs"),+ [+ (False, False, False), # plain live run+ (True, False, False), # plain dry run+ (False, True, False), # mock-inference (live)+ (True, False, True), # dry run with mock inputs+ ],+ )+ def test_legal_combinations_pass(self, dry_run: bool, mock_inference: bool, mock_inputs: bool) -> None:+ # No raise == legal (the function returns None and exits only on an illegal combination).+ validate_run_flag_combination(dry_run=dry_run, mock_inference=mock_inference, mock_inputs=mock_inputs)diff --git a/tests/unit/pipelex/cli/test_traceback_flag_run_core.py b/tests/unit/pipelex/cli/test_traceback_flag_run_core.pyindex b9c8931ce..50f341666 100644--- a/tests/unit/pipelex/cli/test_traceback_flag_run_core.py+++ b/tests/unit/pipelex/cli/test_traceback_flag_run_core.py@@ -60,7 +60,7 @@ class TestTracebackFlagInExecuteRunInnerExcepts:# Patch get_config to provide a minimal execution configmock_config = mocker.MagicMock()- mock_config.pipelex.pipeline_execution_config.with_graph_config_overrides.return_value = mocker.MagicMock()+ mock_config.pipelex.pipeline_execution_config.with_execution_overrides.return_value = mocker.MagicMock()mocker.patch("pipelex.cli.commands.run._run_core.get_config", return_value=mock_config)ctx = click.Context(click.Command("run"), obj={"traceback": True})@@ -83,9 +83,9 @@ class TestTracebackFlagInExecuteRunInnerExcepts:graph_full_data=None,output_dir=OUTPUT_DIR,dry_run=False,+ mock_inference=False,mock_inputs=False,library_dir=None,- cost_report=None,))@@ -113,9 +113,9 @@ class TestTracebackFlagInExecuteRunInnerExcepts:graph_full_data=None,output_dir=OUTPUT_DIR,dry_run=False,+ mock_inference=False,mock_inputs=False,library_dir=None,- cost_report=None,))@@ -142,9 +142,9 @@ class TestTracebackFlagInExecuteRunInnerExcepts:graph_full_data=None,output_dir=OUTPUT_DIR,dry_run=False,+ mock_inference=False,mock_inputs=False,library_dir=None,- cost_report=None,))@@ -171,9 +171,9 @@ class TestTracebackFlagInExecuteRunInnerExcepts:graph_full_data=None,output_dir=OUTPUT_DIR,dry_run=False,+ mock_inference=False,mock_inputs=False,library_dir=None,- cost_report=None,))diff --git a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.pyindex 8796bb314..f85466e01 100644--- a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py+++ b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_reporting.py@@ -29,8 +29,10 @@ class TestContentGeneratorDryReporting:mock_delegate = mocker.MagicMock()mock_delegate.report_inference_job.side_effect = _capture+ # ContentGeneratorDry now delegates synthetic-job reporting to dry_mock.report_dry_llm_job,+ # so patch get_report_delegate where it is looked up at call time (dry_mock, not this module).mocker.patch(- "pipelex.cogt.content_generation.content_generator_dry.get_report_delegate",+ "pipelex.cogt.content_generation.dry_mock.get_report_delegate",return_value=mock_delegate,)diff --git a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.pyindex d919e22f7..9246c21d0 100644--- a/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py+++ b/tests/unit/pipelex/cogt/content_generation/test_content_generator_dry_tz.py@@ -41,8 +41,10 @@ class TestContentGeneratorDryTimezone:mock_delegate = mocker.MagicMock()mock_delegate.report_inference_job.side_effect = _capture+ # The tz-handling logic now lives in dry_mock; ContentGeneratorDry delegates to it, so patch+ # get_report_delegate where it is looked up at call time (dry_mock, not this module).mocker.patch(- "pipelex.cogt.content_generation.content_generator_dry.get_report_delegate",+ "pipelex.cogt.content_generation.dry_mock.get_report_delegate",return_value=mock_delegate,)diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.pynew file mode 100644index 000000000..46d294ea1--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py@@ -0,0 +1,88 @@+"""Unit tests for the shared leaf-level mock helpers (dry_mock).++The load-bearing contract: a ``--mock-inference`` synthetic LLM job carries *non-zero* tokens so it is+reportable (``AggregatedCosts.has_reportable_usage`` True → a cost report renders), while a ``--dry-run``+synthetic job carries zero tokens and stays non-reportable (the report is suppressed). This is the durable+reason the two modes diverge at the reporting layer.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.assignment_models import LLMAssignment+from pipelex.cogt.content_generation.dry_mock import (+ DRY_RUN_INFERENCE_MODEL_NAME,+ MOCK_INFERENCE_MODEL_NAME,+ mock_llm_gen_text,+ report_dry_llm_job,+ report_mock_inference_llm_job,+)+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.cogt.usage.cost_registry import CostRegistry+from pipelex.pipeline.job_metadata import JobMetadata+++class TestDryMock:+ def _capture_reported_job(self, mocker: MockerFixture) -> list[LLMJob]:+ """Patch get_report_delegate in dry_mock and capture every reported LLMJob."""+ captured: list[LLMJob] = []++ def _capture(inference_job: LLMJob) -> None:+ captured.append(inference_job)++ mock_delegate = mocker.MagicMock()+ mock_delegate.report_inference_job.side_effect = _capture+ mocker.patch("pipelex.cogt.content_generation.dry_mock.get_report_delegate", return_value=mock_delegate)+ return captured++ def test_mock_inference_job_is_reportable_non_zero(self, mocker: MockerFixture) -> None:+ """report_mock_inference_llm_job emits non-zero usage under the mock_inference sentinel -> reportable."""+ captured = self._capture_reported_job(mocker)+ report_mock_inference_llm_job(+ job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_mock"),+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),+ llm_prompt=LLMPrompt(),+ )++ assert len(captured) == 1+ usage = captured[0].job_report.llm_tokens_usage+ assert usage is not None+ assert usage.inference_model_name == MOCK_INFERENCE_MODEL_NAME+ aggregated = CostRegistry.aggregate_costs([usage])+ assert aggregated.has_reportable_usage # the whole point: a cost report will render+ assert aggregated.total_nb_tokens > 0++ def test_dry_job_is_zero_and_not_reportable(self, mocker: MockerFixture) -> None:+ """report_dry_llm_job stays zero-token under the dry_run sentinel -> NOT reportable (suppressed)."""+ captured = self._capture_reported_job(mocker)+ report_dry_llm_job(+ job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_dry"),+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),+ llm_prompt=LLMPrompt(),+ )++ assert len(captured) == 1+ usage = captured[0].job_report.llm_tokens_usage+ assert usage is not None+ assert usage.inference_model_name == DRY_RUN_INFERENCE_MODEL_NAME+ aggregated = CostRegistry.aggregate_costs([usage])+ assert not aggregated.has_reportable_usage # dry runs are deliberately suppressed+ assert aggregated.total_nb_tokens == 0++ @pytest.mark.asyncio+ async def test_mock_llm_gen_text_returns_synthetic_text_and_reports(self, mocker: MockerFixture) -> None:+ """mock_llm_gen_text returns a synthetic string and reports one reportable usage event."""+ captured = self._capture_reported_job(mocker)+ assignment = LLMAssignment(+ job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_text", is_mock_inference=True),+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),+ llm_prompt=LLMPrompt(),+ )++ result = mock_llm_gen_text(assignment)++ assert result.startswith("MOCK INFERENCE")+ assert len(captured) == 1+ assert captured[0].job_report.llm_tokens_usage is not Nonediff --git a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_mock_guard.py b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_mock_guard.pynew file mode 100644index 000000000..4b6be31b4--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_mock_guard.py@@ -0,0 +1,58 @@+"""Unit tests for the ``is_mock_inference`` hard guard in the extract leaf (extract_generate).++``--mock-inference`` has no leaf-level mock for document extraction, so reaching the extract leaf under+the flag would dispatch to the real provider and spend. The leaf must fail loud instead: raise+``MockInferenceUnsupportedError`` before ``get_extract_worker`` is ever called. With the flag off, the+real worker path runs unchanged — proving the guard is keyed strictly on ``job_metadata.is_mock_inference``.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.assignment_models import ExtractAssignment+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError+from pipelex.cogt.content_generation.extract_generate import extract_gen_pages+from pipelex.cogt.extract.extract_input import ExtractInput+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams+from pipelex.pipeline.job_metadata import JobMetadata+++class TestExtractGenerateMockGuard:+ def _assignment(self, *, is_mock_inference: bool) -> ExtractAssignment:+ return ExtractAssignment(+ job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_extract_guard", is_mock_inference=is_mock_inference),+ extract_handle="mock-extract-handle",+ extract_input=ExtractInput(document_uri="file:///tmp/doc.pdf"),+ extract_job_params=ExtractJobParams(+ max_nb_images=None,+ image_min_size=None,+ should_caption_images=False,+ should_include_page_views=False,+ page_views_dpi=None,+ ),+ extract_job_config=ExtractJobConfig(),+ )++ @pytest.mark.asyncio+ async def test_mock_flag_raises_and_skips_worker(self, mocker: MockerFixture) -> None:+ """is_mock_inference=True -> the leaf raises before any provider call (get_extract_worker untouched)."""+ worker_spy = mocker.patch("pipelex.cogt.content_generation.extract_generate.get_extract_worker")++ with pytest.raises(MockInferenceUnsupportedError):+ await extract_gen_pages(self._assignment(is_mock_inference=True))++ worker_spy.assert_not_called() # no provider call -> no spend++ @pytest.mark.asyncio+ async def test_no_flag_uses_real_worker(self, mocker: MockerFixture) -> None:+ """is_mock_inference=False -> the real worker path runs (get_extract_worker is called)."""+ sentinel = mocker.MagicMock()+ worker = mocker.MagicMock()+ worker.extract_pages = mocker.AsyncMock(return_value=sentinel)+ worker_spy = mocker.patch("pipelex.cogt.content_generation.extract_generate.get_extract_worker", return_value=worker)+ mocker.patch("pipelex.cogt.content_generation.extract_generate.ExtractJobFactory.make_extract_job", return_value=mocker.MagicMock())++ result = await extract_gen_pages(self._assignment(is_mock_inference=False))++ worker_spy.assert_called_once()+ assert result is sentineldiff --git a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_mock_guard.py b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_mock_guard.pynew file mode 100644index 000000000..ff58502dd--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_mock_guard.py@@ -0,0 +1,58 @@+"""Unit tests for the ``is_mock_inference`` hard guard in the img-gen leaf (img_gen_generate).++``--mock-inference`` has no leaf-level mock for image generation, so reaching the img-gen leaf under+the flag would dispatch to the real provider and spend. The leaf must fail loud instead: raise+``MockInferenceUnsupportedError`` before ``get_img_gen_worker`` is ever called. With the flag off, the+real worker path runs unchanged — proving the guard is keyed strictly on ``job_metadata.is_mock_inference``.+"""++from collections.abc import Awaitable, Callable++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError+from pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list, img_gen_single_image+from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobConfig, ImgGenJobParams+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt+from pipelex.pipeline.job_metadata import JobMetadata+++class TestImgGenGenerateMockGuard:+ def _assignment(self, *, is_mock_inference: bool) -> ImgGenAssignment:+ return ImgGenAssignment(+ job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_img_guard", is_mock_inference=is_mock_inference),+ img_gen_handle="mock-img-handle",+ img_gen_prompt=ImgGenPrompt(positive_text="a red apple"),+ img_gen_job_params=ImgGenJobParams(aspect_ratio=AspectRatio.SQUARE, background=Background.AUTO),+ img_gen_job_config=ImgGenJobConfig(is_sync_mode=True),+ nb_images=1,+ )++ @pytest.mark.parametrize("leaf", [img_gen_single_image, img_gen_image_list])+ @pytest.mark.asyncio+ async def test_mock_flag_raises_and_skips_worker(self, mocker: MockerFixture, leaf: Callable[[ImgGenAssignment], Awaitable[object]]) -> None:+ """is_mock_inference=True -> the leaf raises before any provider call (get_img_gen_worker untouched)."""+ worker_spy = mocker.patch("pipelex.cogt.content_generation.img_gen_generate.get_img_gen_worker")++ with pytest.raises(MockInferenceUnsupportedError):+ await leaf(self._assignment(is_mock_inference=True))++ worker_spy.assert_not_called() # no provider call -> no spend++ @pytest.mark.asyncio+ async def test_no_flag_uses_real_worker(self, mocker: MockerFixture) -> None:+ """is_mock_inference=False -> the real worker path runs (get_img_gen_worker is called)."""+ sentinel = mocker.MagicMock()+ worker = mocker.MagicMock()+ worker.gen_image = mocker.AsyncMock(return_value=sentinel)+ worker_spy = mocker.patch("pipelex.cogt.content_generation.img_gen_generate.get_img_gen_worker", return_value=worker)+ mocker.patch(+ "pipelex.cogt.content_generation.img_gen_generate.ImgGenJobFactory.make_img_gen_job_from_prompt", return_value=mocker.MagicMock()+ )++ result = await img_gen_single_image(self._assignment(is_mock_inference=False))++ worker_spy.assert_called_once()+ assert result is sentineldiff --git a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_mock_branch.py b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_mock_branch.pynew file mode 100644index 000000000..c2ed7d2bd--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_mock_branch.py@@ -0,0 +1,50 @@+"""Unit tests for the ``is_mock_inference`` branch in the LLM leaf (llm_generate).++The leaf is the single point where direct execution and the Temporal activities converge, so branching+there covers both modes. These tests pin that the branch is keyed strictly on+``job_metadata.is_mock_inference``: when set, the worker is never touched and the mock helper answers;+when unset, the real worker path runs.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.assignment_models import LLMAssignment+from pipelex.cogt.content_generation.llm_generate import llm_gen_text+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.pipeline.job_metadata import JobMetadata+++class TestLlmGenerateMockBranch:+ def _assignment(self, *, is_mock_inference: bool) -> LLMAssignment:+ return LLMAssignment(+ job_metadata=JobMetadata(user_id="u", pipeline_run_id="run_branch", is_mock_inference=is_mock_inference),+ llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),+ llm_prompt=LLMPrompt(),+ )++ @pytest.mark.asyncio+ async def test_mock_flag_routes_to_mock_and_skips_worker(self, mocker: MockerFixture) -> None:+ """is_mock_inference=True -> the mock helper answers and get_llm_worker is never called (no spend)."""+ worker_spy = mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")+ mocker.patch("pipelex.cogt.content_generation.dry_mock.get_report_delegate", return_value=mocker.MagicMock())++ result = await llm_gen_text(self._assignment(is_mock_inference=True))++ worker_spy.assert_not_called()+ assert result.startswith("MOCK INFERENCE")++ @pytest.mark.asyncio+ async def test_no_flag_uses_real_worker(self, mocker: MockerFixture) -> None:+ """is_mock_inference=False -> the real worker path runs (get_llm_worker is called)."""+ worker = mocker.MagicMock()+ worker.gen_text = mocker.AsyncMock(return_value="real generated text")+ worker_spy = mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker", return_value=worker)+ report_spy = mocker.patch("pipelex.cogt.content_generation.dry_mock.get_report_delegate", return_value=mocker.MagicMock())++ result = await llm_gen_text(self._assignment(is_mock_inference=False))++ worker_spy.assert_called_once()+ assert result == "real generated text"+ report_spy.assert_not_called() # the mock reporting path is not taken on a real rundiff --git a/tests/unit/pipelex/cogt/usage/test_cost_registry.py b/tests/unit/pipelex/cogt/usage/test_cost_registry.pyindex 869a4e13a..91c4aa53c 100644--- a/tests/unit/pipelex/cogt/usage/test_cost_registry.py+++ b/tests/unit/pipelex/cogt/usage/test_cost_registry.py@@ -498,3 +498,100 @@ class TestCostRegistry:rows = list(csv.DictReader(file))assert len(rows) == 1assert rows[0][LLMTokenCostReportField.LLM_NAME] == "test-model"++ def test_aggregate_costs_totals_and_reportability(self, job_metadata: JobMetadata):+ """One aggregation pass yields the run total and the reportable-work flag.++ ``has_reportable_usage`` is True for a priced run AND for a free/zero-price run with real tokens+ (cost 0), but False for a dry run (zero tokens, zero cost) — so dry runs are the only thing suppressed.+ """+ priced = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="m",+ inference_model_id="m-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000}, # 0.1 + 0.1 = 0.2+ )+ free = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="ollama-x",+ inference_model_id="ollama-x-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={}, # real tokens, no price -> cost 0+ )+ dry = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="dry",+ inference_model_id="dry-run",+ nb_tokens_by_category={},+ unit_costs={},+ )++ priced_aggregated = CostRegistry.aggregate_costs([priced])+ assert priced_aggregated.total_cost == 0.2+ assert priced_aggregated.total_nb_tokens == 150+ assert priced_aggregated.has_reportable_usage is True++ free_aggregated = CostRegistry.aggregate_costs([free])+ assert free_aggregated.total_cost == 0.0+ assert free_aggregated.total_nb_tokens == 150+ assert free_aggregated.has_reportable_usage is True # tokens but no cost -> still reportable++ dry_aggregated = CostRegistry.aggregate_costs([dry])+ assert dry_aggregated.total_cost == 0.0+ assert dry_aggregated.total_nb_tokens == 0+ assert dry_aggregated.has_reportable_usage is False # the only case we suppress++ def test_build_cost_summary_non_zero(self, job_metadata: JobMetadata):+ """A priced run yields a JSON-serializable summary with a total and per-model rows."""+ usage = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="claude-x",+ inference_model_id="claude-x-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},+ )++ summary = CostRegistry.build_cost_summary([usage])++ assert summary is not None+ assert summary["total_cost"] == 0.2+ assert len(summary["by_model"]) == 1+ row = summary["by_model"][0]+ assert row["model"] == "claude-x"+ assert row["model_type"] == "llm"+ assert row["nb_tokens_input"] == 100+ assert row["nb_tokens_output"] == 50+ assert row["cost"] == 0.2++ def test_build_cost_summary_free_model_returns_summary(self, job_metadata: JobMetadata):+ """A free/zero-price model with real tokens still reports its usage, with total_cost 0."""+ free = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="ollama-x",+ inference_model_id="ollama-x-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={},+ )++ summary = CostRegistry.build_cost_summary([free])++ assert summary is not None+ assert summary["total_cost"] == 0.0+ row = summary["by_model"][0]+ assert row["model"] == "ollama-x"+ assert row["nb_tokens_input"] == 100+ assert row["nb_tokens_output"] == 50+ assert row["cost"] == 0.0++ def test_build_cost_summary_dry_run_returns_none(self, job_metadata: JobMetadata):+ """A dry-run-shaped usage (zero tokens, zero cost) does no reportable work, so no summary."""+ dry = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="dry",+ inference_model_id="dry-run",+ nb_tokens_by_category={},+ unit_costs={},+ )++ assert CostRegistry.build_cost_summary([dry]) is Nonediff --git a/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py b/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.pyindex eb3699046..8d6d6bef7 100644--- a/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py+++ b/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py@@ -11,7 +11,7 @@ from pipelex.pipe_operators.llm.pipe_llm import PipeLLMfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprintfrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadata-from tests.unit.pipelex.graph.conftest import make_graph_context+from tests.unit.pipelex.graph.conftest import make_trace_context@pytest.mark.asyncio(loop_scope="class")@@ -51,7 +51,7 @@ class TestRunPipeForwardsTracerMetadata:job_metadata = JobMetadata(user_id="pytest",pipeline_run_id="test-run-meta",- graph_context=make_graph_context(graph_id="test-run-meta"),+ trace_context=make_trace_context(graph_id="test-run-meta"),)await pipe.run_pipe(diff --git a/tests/unit/pipelex/graph/conftest.py b/tests/unit/pipelex/graph/conftest.pyindex b3fee0fb7..993267ef3 100644--- a/tests/unit/pipelex/graph/conftest.py+++ b/tests/unit/pipelex/graph/conftest.py@@ -4,7 +4,7 @@ import pytestfrom pipelex.config import get_configfrom pipelex.graph.graph_config import DataInclusionConfig, GraphConfig-from pipelex.graph.graph_context import GraphContext+from pipelex.graph.trace_context import TraceContextdef make_defaulted_data_inclusion_config(@@ -35,7 +35,7 @@ def make_defaulted_data_inclusion_config()-def make_graph_context(+def make_trace_context(graph_id: str = "test-graph",parent_node_id: str | None = None,node_sequence: int = 0,@@ -43,8 +43,8 @@ def make_graph_context(stuff_text_content: bool = False,stuff_html_content: bool = False,error_stack_traces: bool = False,-) -> GraphContext:- """Create a GraphContext for testing.+) -> TraceContext:+ """Create a TraceContext for testing.Args:graph_id: The graph identifier.@@ -56,9 +56,9 @@ def make_graph_context(error_stack_traces: Whether to include error stack traces.Returns:- A GraphContext configured for testing.+ A TraceContext configured for testing."""- return GraphContext(+ return TraceContext(graph_id=graph_id,parent_node_id=parent_node_id,node_sequence=node_sequence,diff --git a/tests/unit/pipelex/graph/test_graph_tracer.py b/tests/unit/pipelex/graph/test_graph_tracer.pyindex 82fb0b975..1745c6cb6 100644--- a/tests/unit/pipelex/graph/test_graph_tracer.py+++ b/tests/unit/pipelex/graph/test_graph_tracer.py@@ -11,7 +11,7 @@ class TestGraphTracer:"""Tests for GraphTracer implementation."""def test_setup_returns_initial_context(self) -> None:- """Test that setup returns a properly initialized GraphContext."""+ """Test that setup returns a properly initialized TraceContext."""tracer = GraphTracer()context = tracer.setup(graph_id="test-graph-001", data_inclusion=make_defaulted_data_inclusion_config())@@ -50,7 +50,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)node_id, child_context = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -93,7 +93,7 @@ class TestGraphTracer:# Start parent (sequence controller)started_at = datetime.now(timezone.utc)parent_id, parent_child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_sequence",pipe_type="PipeSequence",node_kind=NodeKind.CONTROLLER,@@ -102,7 +102,7 @@ class TestGraphTracer:# Start child 1 (operator)child1_id, _child1_ctx = tracer.on_pipe_start(- graph_context=parent_child_ctx,+ trace_context=parent_child_ctx,pipe_code="llm_step_1",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -116,7 +116,7 @@ class TestGraphTracer:# Start child 2 (operator)child2_id, _child2_ctx = tracer.on_pipe_start(- graph_context=parent_child_ctx,+ trace_context=parent_child_ctx,pipe_code="llm_step_2",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -153,7 +153,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="failing_pipe",pipe_type="PipeFunc",node_kind=NodeKind.OPERATOR,@@ -185,7 +185,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)_node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="stuck_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -208,7 +208,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)node1_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="pipe_1",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -217,7 +217,7 @@ class TestGraphTracer:tracer.on_pipe_end_success(node_id=node1_id, ended_at=started_at + timedelta(milliseconds=10))node2_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="pipe_2",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -250,7 +250,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)condition_id, cond_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_condition",pipe_type="PipeCondition",node_kind=NodeKind.CONTROLLER,@@ -258,7 +258,7 @@ class TestGraphTracer:)outcome_id, _ = tracer.on_pipe_start(- graph_context=cond_ctx,+ trace_context=cond_ctx,pipe_code="outcome_true",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -294,7 +294,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -327,7 +327,7 @@ class TestGraphTracer:started_at = datetime.now(timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="summarize",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -357,7 +357,7 @@ class TestGraphTracer:# Pipe 1: produces stuff with digest "stuff_001"node1_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="producer_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -371,7 +371,7 @@ class TestGraphTracer:# Pipe 2: consumes stuff with digest "stuff_001"node2_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -401,7 +401,7 @@ class TestGraphTracer:# Pipe consumes stuff that wasn't produced by any tracked pipenode_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_only",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -428,7 +428,7 @@ class TestGraphTracer:# Pipe produces and consumes the same stuff (shouldn't happen, but guard against it)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="self_ref_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -457,7 +457,7 @@ class TestGraphTracer:# Producer pipeproducer_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="producer",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -471,7 +471,7 @@ class TestGraphTracer:# Consumer 1consumer1_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_1",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -482,7 +482,7 @@ class TestGraphTracer:# Consumer 2consumer2_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_2",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -515,7 +515,7 @@ class TestGraphTracer:# PipeBatch that consumes the listbatch_id, batch_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -525,7 +525,7 @@ class TestGraphTracer:# Branch 0: processes item 0branch0_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -540,7 +540,7 @@ class TestGraphTracer:# Branch 1: processes item 1branch1_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -599,7 +599,7 @@ class TestGraphTracer:# PipeBatch that produces the output listbatch_id, batch_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -609,7 +609,7 @@ class TestGraphTracer:# Branch 0: produces item 0branch0_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -623,7 +623,7 @@ class TestGraphTracer:# Branch 1: produces item 1branch1_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -677,7 +677,7 @@ class TestGraphTracer:# PipeBatchbatch_id, batch_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -687,7 +687,7 @@ class TestGraphTracer:# Branch 0branch0_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="process_item",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -771,7 +771,7 @@ class TestGraphTracer:# PipeBatch that consumes the listbatch_id, batch_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -781,7 +781,7 @@ class TestGraphTracer:# Branch: processes itembranch_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -828,7 +828,7 @@ class TestGraphTracer:# PipeBatchbatch_id, batch_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -838,7 +838,7 @@ class TestGraphTracer:# Branch: produces itembranch_id, _ = tracer.on_pipe_start(- graph_context=batch_ctx,+ trace_context=batch_ctx,pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -886,7 +886,7 @@ class TestGraphTracer:# Controller node (e.g., PipeParallel)controller_id, ctrl_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_parallel",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -896,7 +896,7 @@ class TestGraphTracer:# Branch 1: produces output with digest "branch_output_1"branch1_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="branch_pipe_1",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -911,7 +911,7 @@ class TestGraphTracer:# Branch 2: produces output with digest "branch_output_2"branch2_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="branch_pipe_2",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -936,7 +936,7 @@ class TestGraphTracer:# Consumer pipe that uses branch_output_1consumer_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -983,7 +983,7 @@ class TestGraphTracer:# Producer pipe creates stuff with digest "original_stuff"producer_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="producer",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -997,7 +997,7 @@ class TestGraphTracer:# Controller consumes "original_stuff" and its main_stuff is the samecontroller_id, _ctrl_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="my_parallel",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -1014,7 +1014,7 @@ class TestGraphTracer:# Consumer should still get the edge from the original producer, not the controllerconsumer_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -1054,7 +1054,7 @@ class TestGraphTracer:# Controller with multiple outputscontroller_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="multi_output_pipe",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -1082,7 +1082,7 @@ class TestGraphTracer:# Consumer A reads digest_aconsumer_a_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -1093,7 +1093,7 @@ class TestGraphTracer:# Consumer B reads digest_bconsumer_b_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -1104,7 +1104,7 @@ class TestGraphTracer:# Consumer C reads digest_cconsumer_c_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer_c",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,diff --git a/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.py b/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.pynew file mode 100644index 000000000..4990ad41e--- /dev/null+++ b/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.py@@ -0,0 +1,60 @@+from datetime import datetime, timezone++from pipelex.graph.graph_tracer import GraphTracer+from pipelex.graph.graphspec import NodeKind+from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config+++class TestGraphTracerEmitGating:+ """The emit flags threaded into setup are born onto the TraceContext and gate teardown (E1)."""++ def _start_a_node(self, tracer: GraphTracer, context: object) -> None:+ tracer.on_pipe_start(+ trace_context=context, # type: ignore[arg-type]+ pipe_code="some_pipe",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ started_at=datetime.now(timezone.utc),+ )++ def test_setup_stamps_emit_flags_onto_context(self) -> None:+ tracer = GraphTracer()+ context = tracer.setup(+ graph_id="emit-flags",+ data_inclusion=make_defaulted_data_inclusion_config(),+ emit_graph_events=False,+ emit_usage_events=True,+ )+ assert context.emit_graph_events is False+ assert context.emit_usage_events is True++ def test_setup_defaults_emit_flags_true(self) -> None:+ tracer = GraphTracer()+ context = tracer.setup(graph_id="emit-defaults", data_inclusion=make_defaulted_data_inclusion_config())+ assert context.emit_graph_events is True+ assert context.emit_usage_events is True++ def test_teardown_returns_none_when_graph_events_off(self) -> None:+ """Costs-only mode: the tracer still mints node ids, but teardown skips the discarded spec build."""+ tracer = GraphTracer()+ context = tracer.setup(+ graph_id="costs-only",+ data_inclusion=make_defaulted_data_inclusion_config(),+ emit_graph_events=False,+ )+ self._start_a_node(tracer, context)+ assert tracer.teardown() is None+ assert tracer.is_active is False++ def test_teardown_builds_spec_when_graph_events_on(self) -> None:+ tracer = GraphTracer()+ context = tracer.setup(+ graph_id="graph-on",+ data_inclusion=make_defaulted_data_inclusion_config(),+ emit_graph_events=True,+ )+ self._start_a_node(tracer, context)+ graph_spec = tracer.teardown()+ assert graph_spec is not None+ assert graph_spec.graph_id == "graph-on"+ assert len(graph_spec.nodes) == 1diff --git a/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py b/tests/unit/pipelex/graph/test_graph_tracer_event_emission.pyindex d9810773f..2f60a7070 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py@@ -59,7 +59,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -93,7 +93,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)parent_id, child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="parent_pipe",pipe_type="PipeSequence",node_kind=NodeKind.CONTROLLER,@@ -101,7 +101,7 @@ class TestGraphTracerEventEmission:)child_id, _ = tracer.on_pipe_start(- graph_context=child_ctx,+ trace_context=child_ctx,pipe_code="child_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -129,7 +129,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -170,7 +170,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -210,14 +210,14 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_a, _ctx_a = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="pipe_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,started_at=started_at,)node_b, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="pipe_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -258,7 +258,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="parallel_pipe",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -350,7 +350,7 @@ class TestGraphTracerEventEmission:# Create controller nodectrl_id, ctrl_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="parallel_ctrl",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -359,7 +359,7 @@ class TestGraphTracerEventEmission:# Create branch nodes and end them with outputsbranch_a_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="branch_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -372,7 +372,7 @@ class TestGraphTracerEventEmission:)branch_b_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="branch_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -413,7 +413,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -446,7 +446,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -470,14 +470,14 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_a, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="pipe_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,started_at=started_at,)node_b, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="pipe_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -510,7 +510,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)for index in range(5):node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code=f"pipe_{index}",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -536,7 +536,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)node_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,diff --git a/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.py b/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.pyindex fd135ce64..f1d6782b2 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.py@@ -14,7 +14,7 @@ class TestGraphTracerNodeMetadata:started_at = datetime.now(timezone.utc)node_id, _child = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="summarize",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -42,7 +42,7 @@ class TestGraphTracerNodeMetadata:started_at = datetime.now(timezone.utc)node_id, _child = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="noop",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,diff --git a/tests/unit/pipelex/graph/test_graph_tracer_noop.py b/tests/unit/pipelex/graph/test_graph_tracer_noop.pyindex b1566a4ad..e708a4ef0 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_noop.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_noop.py@@ -32,7 +32,7 @@ class TestGraphTracerNoOp:started_at = datetime.now(timezone.utc)node_id, child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,diff --git a/tests/unit/pipelex/graph/test_graph_context.py b/tests/unit/pipelex/graph/test_trace_context.pysimilarity index 78%rename from tests/unit/pipelex/graph/test_graph_context.pyrename to tests/unit/pipelex/graph/test_trace_context.pyindex 6e2679cda..c437b2d11 100644--- a/tests/unit/pipelex/graph/test_graph_context.py+++ b/tests/unit/pipelex/graph/test_trace_context.py@@ -1,21 +1,21 @@-"""Unit tests for GraphContext."""+"""Unit tests for TraceContext."""-from tests.unit.pipelex.graph.conftest import make_graph_context+from tests.unit.pipelex.graph.conftest import make_trace_context-class TestGraphContext:- """Tests for GraphContext model."""+class TestTraceContext:+ """Tests for TraceContext model."""def test_make_node_id(self) -> None:"""Test node ID generation."""- context = make_graph_context(graph_id="ctx-test", node_sequence=5)+ context = make_trace_context(graph_id="ctx-test", node_sequence=5)node_id = context.make_node_id()assert node_id == "ctx-test:node_5"def test_copy_for_child(self) -> None:"""Test creating child context."""- parent = make_graph_context(graph_id="ctx-test", parent_node_id=None, node_sequence=0)+ parent = make_trace_context(graph_id="ctx-test", parent_node_id=None, node_sequence=0)child = parent.copy_for_child(child_node_id="ctx-test:node_0", next_sequence=1)assert child.graph_id == "ctx-test"@@ -27,7 +27,7 @@ class TestGraphContext:def test_copy_for_child_preserves_data_inclusion(self) -> None:"""Test that copy_for_child preserves data_inclusion config."""- parent = make_graph_context(+ parent = make_trace_context(graph_id="ctx-test",stuff_json_content=True,stuff_text_content=True,diff --git a/tests/unit/pipelex/graph/test_trace_context_emit_flags.py b/tests/unit/pipelex/graph/test_trace_context_emit_flags.pynew file mode 100644index 000000000..989fb8bfe--- /dev/null+++ b/tests/unit/pipelex/graph/test_trace_context_emit_flags.py@@ -0,0 +1,51 @@+"""Unit tests for the TraceContext emit-stream flags (D4).++Pins that ``emit_graph_events`` / ``emit_usage_events`` default to True (backward-compatible+"context present means emit both") and that ``copy_for_child`` propagates whatever the parent+carried — so a costs-only or graph-only run keeps its gating as nested pipes spawn children.+"""++from pipelex.graph.graph_config import DataInclusionConfig+from pipelex.graph.trace_context import TraceContext++DATA_INCLUSION_OFF = DataInclusionConfig(+ pipe_and_concept_registry=False,+ stuff_json_content=False,+ stuff_text_content=False,+ stuff_html_content=False,+ error_stack_traces=False,+)+++class TestTraceContextEmitFlags:+ def test_defaults_emit_both_streams(self) -> None:+ """A TraceContext built without explicit flags emits both streams (legacy behavior)."""+ context = TraceContext(graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF)+ assert context.emit_graph_events is True+ assert context.emit_usage_events is True++ def test_copy_for_child_preserves_costs_only_flags(self) -> None:+ """costs-only flags survive into a child context (graph off, costs on)."""+ parent = TraceContext(+ graph_id="run_1",+ data_inclusion=DATA_INCLUSION_OFF,+ emit_graph_events=False,+ emit_usage_events=True,+ )+ child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=1)+ assert child.parent_node_id == "run_1:node_0"+ assert child.node_sequence == 1+ assert child.emit_graph_events is False+ assert child.emit_usage_events is True++ def test_copy_for_child_preserves_graph_only_flags(self) -> None:+ """graph-only flags survive into a child context (graph on, costs off)."""+ parent = TraceContext(+ graph_id="run_1",+ data_inclusion=DATA_INCLUSION_OFF,+ emit_graph_events=True,+ emit_usage_events=False,+ )+ child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=2)+ assert child.emit_graph_events is True+ assert child.emit_usage_events is Falsediff --git a/tests/unit/pipelex/pipe_run/test_graph_assembly.py b/tests/unit/pipelex/pipe_run/test_graph_assembly.pydeleted file mode 100644index 1cab6375f..000000000--- a/tests/unit/pipelex/pipe_run/test_graph_assembly.py+++ /dev/null@@ -1,143 +0,0 @@-import json--import pytest-from pydantic import ValidationError-from pytest_mock import MockerFixture--from pipelex.base_exceptions import PipelexConfigError-from pipelex.pipe_run.graph_assembly import assemble_graph_on_output-from pipelex.system.exceptions import MissingDependencyError---class TestAssembleGraphOnOutput:- def test_returns_when_tracing_disabled(self, mocker: MockerFixture) -> None:- """When tracing is disabled, the function returns without touching pipe_output."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = False-- mock_make_event_log = mocker.patch("pipelex.pipe_run.graph_assembly.make_event_log")-- mock_pipe_output = mocker.MagicMock()- original_graph_spec = mock_pipe_output.graph_spec-- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-disabled")-- mock_make_event_log.assert_not_called()- assert mock_pipe_output.graph_spec is original_graph_spec-- def test_swallows_oserror_with_warning(self, mocker: MockerFixture) -> None:- """OSError from event log read is caught; pipe_output.graph_spec is left unchanged."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = True-- mock_event_log = mocker.MagicMock()- mock_event_log.read_events = mocker.MagicMock(side_effect=OSError("file vanished"))- mocker.patch(- "pipelex.pipe_run.graph_assembly.make_event_log",- return_value=mock_event_log,- )-- mock_pipe_output = mocker.MagicMock()- original_graph_spec = mock_pipe_output.graph_spec-- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-oserror")-- assert mock_pipe_output.graph_spec is original_graph_spec- mock_event_log.close.assert_called_once()-- def test_swallows_pipelex_config_error(self, mocker: MockerFixture) -> None:- """PipelexConfigError from make_event_log is caught (broken tracing config doesn't fail the pipe)."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = True-- mocker.patch(- "pipelex.pipe_run.graph_assembly.make_event_log",- side_effect=PipelexConfigError("ndjson section missing"),- )-- mock_pipe_output = mocker.MagicMock()- original_graph_spec = mock_pipe_output.graph_spec-- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-config")-- assert mock_pipe_output.graph_spec is original_graph_spec-- def test_swallows_missing_dependency_error(self, mocker: MockerFixture) -> None:- """MissingDependencyError from make_event_log is caught (e.g., boto3 missing for DynamoDB)."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = True-- mocker.patch(- "pipelex.pipe_run.graph_assembly.make_event_log",- side_effect=MissingDependencyError(dependency_name="boto3", extra_name="aws"),- )-- mock_pipe_output = mocker.MagicMock()- original_graph_spec = mock_pipe_output.graph_spec-- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-missing-dep")-- assert mock_pipe_output.graph_spec is original_graph_spec-- def test_swallows_validation_error(self, mocker: MockerFixture) -> None:- """Pydantic ValidationError from GraphSpec construction is caught."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = True-- mock_event_log = mocker.MagicMock()- mock_event_log.read_events = mocker.MagicMock(return_value=[mocker.MagicMock()])- mocker.patch(- "pipelex.pipe_run.graph_assembly.make_event_log",- return_value=mock_event_log,- )-- validation_error = ValidationError.from_exception_data(title="GraphSpec", line_errors=[])- mocker.patch(- "pipelex.pipe_run.graph_assembly.GraphSpecAssembler.assemble",- side_effect=validation_error,- )-- mock_pipe_output = mocker.MagicMock()- original_graph_spec = mock_pipe_output.graph_spec-- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-validation")-- assert mock_pipe_output.graph_spec is original_graph_spec- mock_event_log.close.assert_called_once()-- def test_swallows_json_decode_error(self, mocker: MockerFixture) -> None:- """json.JSONDecodeError from event log read is caught."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = True-- mock_event_log = mocker.MagicMock()- mock_event_log.read_events = mocker.MagicMock(- side_effect=json.JSONDecodeError("corrupt", doc="{", pos=0),- )- mocker.patch(- "pipelex.pipe_run.graph_assembly.make_event_log",- return_value=mock_event_log,- )-- mock_pipe_output = mocker.MagicMock()- original_graph_spec = mock_pipe_output.graph_spec-- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-jsondecode")-- assert mock_pipe_output.graph_spec is original_graph_spec- mock_event_log.close.assert_called_once()-- def test_propagates_unexpected_keyerror(self, mocker: MockerFixture) -> None:- """Programming bugs (KeyError) propagate — proves tightening from blanket except Exception."""- mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")- mock_config.return_value.pipelex.tracing_config.is_enabled = True-- mocker.patch(- "pipelex.pipe_run.graph_assembly.make_event_log",- side_effect=KeyError("unexpected_key"),- )-- mock_pipe_output = mocker.MagicMock()-- with pytest.raises(KeyError):- assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-keyerror")diff --git a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py b/tests/unit/pipelex/pipe_run/test_tracing_assembly.pynew file mode 100644index 000000000..56ec40255--- /dev/null+++ b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py@@ -0,0 +1,276 @@+import json+from datetime import datetime, timezone++import pytest+from pydantic import ValidationError+from pytest_mock import MockerFixture++from pipelex.base_exceptions import PipelexConfigError+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipe_run.tracing_assembly import TracingAssembly, assemble_tracing, assemble_tracing_on_output+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.system.exceptions import MissingDependencyError+from pipelex.tracing.trace_events import UsageReportEvent++_MODULE = "pipelex.pipe_run.tracing_assembly"+++def _make_usage_event(pipeline_run_id: str, node_id: str) -> UsageReportEvent:+ """A UsageReportEvent carrying a real LLMTokensUsage (so the aggregator returns it)."""+ metadata = JobMetadata(user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id)+ tokens_usage = LLMTokensUsage(+ job_metadata=metadata,+ inference_model_name="fake_model",+ inference_model_id="fake_model_id",+ unit_costs={CostCategory.INPUT: 0.0, CostCategory.OUTPUT: 0.0},+ nb_tokens_by_category={TokenCategory.INPUT: 3, TokenCategory.OUTPUT: 5},+ )+ return UsageReportEvent(+ pipeline_run_id=pipeline_run_id,+ workflow_id="direct",+ timestamp=datetime.now(timezone.utc),+ sequence=0,+ node_id=node_id,+ tokens_usage=tokens_usage,+ )+++class TestTracingAssembly:+ def _enable_tracing(self, mocker: MockerFixture) -> None:+ mock_config = mocker.patch(f"{_MODULE}.get_config")+ mock_config.return_value.pipelex.tracing_config.is_enabled = True++ def _disable_tracing(self, mocker: MockerFixture) -> None:+ mock_config = mocker.patch(f"{_MODULE}.get_config")+ mock_config.return_value.pipelex.tracing_config.is_enabled = False++ def test_returns_empty_when_tracing_disabled(self, mocker: MockerFixture) -> None:+ self._disable_tracing(mocker)+ make_event_log = mocker.patch(f"{_MODULE}.make_event_log")++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)++ make_event_log.assert_not_called()+ assert result == TracingAssembly()++ def test_returns_empty_when_neither_concern_requested(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ make_event_log = mocker.patch(f"{_MODULE}.make_event_log")++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=False, assemble_usage=False)++ make_event_log.assert_not_called()+ assert result == TracingAssembly()++ def test_empty_events_returns_empty(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)+ assemble = mocker.patch(f"{_MODULE}.GraphSpecAssembler.assemble")++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)++ assert result.graph_spec is None+ assert result.tokens_usages is None+ assemble.assert_not_called()+ event_log.close.assert_called_once()++ def test_usage_only_aggregates_and_skips_graph(self, mocker: MockerFixture) -> None:+ """Costs-only: usage is aggregated, graph_spec stays None even with events present, assembler untouched."""+ self._enable_tracing(mocker)+ usage_event = _make_usage_event("plr-usage", "plr-usage:node_0")+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[usage_event])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)+ assemble = mocker.patch(f"{_MODULE}.GraphSpecAssembler.assemble")++ result = assemble_tracing(pipeline_run_id="plr-usage", assemble_graph=False, assemble_usage=True)++ assert result.graph_spec is None+ assert result.tokens_usages == [usage_event.tokens_usage]+ assemble.assert_not_called()++ def test_graph_only_assembles_and_skips_usage(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ usage_event = _make_usage_event("plr-graph", "plr-graph:node_0")+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[usage_event])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)+ sentinel_graph = mocker.MagicMock()+ assemble = mocker.patch(f"{_MODULE}.GraphSpecAssembler.assemble", return_value=sentinel_graph)++ result = assemble_tracing(+ pipeline_run_id="plr-graph",+ assemble_graph=True,+ assemble_usage=False,+ domain_code="dom",+ main_pipe_code="main",+ )++ assert result.graph_spec is sentinel_graph+ assert result.tokens_usages is None+ assemble.assert_called_once()++ def test_graph_and_usage_both_assembled(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ usage_event = _make_usage_event("plr-both", "plr-both:node_0")+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[usage_event])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)+ sentinel_graph = mocker.MagicMock()+ mocker.patch(f"{_MODULE}.GraphSpecAssembler.assemble", return_value=sentinel_graph)++ result = assemble_tracing(pipeline_run_id="plr-both", assemble_graph=True, assemble_usage=True)++ assert result.graph_spec is sentinel_graph+ assert result.tokens_usages == [usage_event.tokens_usage]++ def test_read_oserror_sets_error_on_requested_concerns(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(side_effect=OSError("file vanished"))+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)++ assert result.graph_spec is None+ assert result.tokens_usages is None+ assert result.graph_assembly_error is not None+ assert result.usage_assembly_error is not None+ event_log.close.assert_called_once()++ def test_read_botocore_errors_are_caught(self, mocker: MockerFixture) -> None:+ """DynamoDB backend failures degrade to an assembly error, never failing the run. Covers BOTH botocore+ base classes (siblings, neither a subclass of the other): ClientError (service-side throttle / auth)+ and BotoCoreError subclasses (transport / credential / timeout, e.g. EndpointConnectionError).+ """+ botocore_exceptions = pytest.importorskip("botocore.exceptions")+ client_error = botocore_exceptions.ClientError(+ error_response={"Error": {"Code": "ProvisionedThroughputExceededException", "Message": "throttled"}},+ operation_name="Query",+ )+ transport_error = botocore_exceptions.EndpointConnectionError(endpoint_url="https://dynamodb.test")+ for backend_error in (client_error, transport_error):+ self._enable_tracing(mocker)+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(side_effect=backend_error)+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)++ assert result.graph_spec is None, backend_error+ assert result.tokens_usages is None, backend_error+ assert result.graph_assembly_error is not None, backend_error+ assert result.usage_assembly_error is not None, backend_error+ event_log.close.assert_called_once()++ def test_read_error_only_marks_requested_concern(self, mocker: MockerFixture) -> None:+ """A costs-only read failure sets usage_assembly_error but never graph_assembly_error."""+ self._enable_tracing(mocker)+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(side_effect=json.JSONDecodeError("corrupt", doc="{", pos=0))+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=False, assemble_usage=True)++ assert result.usage_assembly_error is not None+ assert result.graph_assembly_error is None++ def test_read_config_error_is_caught(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ mocker.patch(f"{_MODULE}.make_event_log", side_effect=PipelexConfigError("ndjson section missing"))++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=False)++ assert result.graph_spec is None+ assert result.graph_assembly_error is not None++ def test_read_missing_dependency_is_caught(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ mocker.patch(f"{_MODULE}.make_event_log", side_effect=MissingDependencyError(dependency_name="boto3", extra_name="aws"))++ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)++ assert result.graph_assembly_error is not None+ assert result.usage_assembly_error is not None++ def test_graph_validation_error_still_aggregates_usage(self, mocker: MockerFixture) -> None:+ """A GraphSpec build failure records graph_assembly_error but leaves the usage aggregation intact."""+ self._enable_tracing(mocker)+ usage_event = _make_usage_event("plr-mixed", "plr-mixed:node_0")+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[usage_event])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)+ validation_error = ValidationError.from_exception_data(title="GraphSpec", line_errors=[])+ mocker.patch(f"{_MODULE}.GraphSpecAssembler.assemble", side_effect=validation_error)++ result = assemble_tracing(pipeline_run_id="plr-mixed", assemble_graph=True, assemble_usage=True)++ assert result.graph_spec is None+ assert result.graph_assembly_error is not None+ assert result.tokens_usages == [usage_event.tokens_usage]++ def test_unexpected_keyerror_propagates(self, mocker: MockerFixture) -> None:+ """Programming bugs (KeyError) propagate — proves tightening from a blanket except Exception."""+ self._enable_tracing(mocker)+ mocker.patch(f"{_MODULE}.make_event_log", side_effect=KeyError("unexpected_key"))++ with pytest.raises(KeyError):+ assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)++ def test_tracing_assembly_round_trips_through_json(self) -> None:+ """TracingAssembly is the Temporal activity return type; its tokens_usages discriminated union+ must survive a model_dump_json → model_validate_json round-trip (proxy for the pydantic data+ converter that carries it across the activity boundary).+ """+ usage_event = _make_usage_event("plr-roundtrip", "plr-roundtrip:node_0")+ original = TracingAssembly(tokens_usages=[usage_event.tokens_usage])++ restored = TracingAssembly.model_validate_json(original.model_dump_json())++ assert restored.graph_spec is None+ assert restored.tokens_usages == [usage_event.tokens_usage]++ def test_on_output_applies_assembled_fields(self, mocker: MockerFixture) -> None:+ self._enable_tracing(mocker)+ usage_event = _make_usage_event("plr-out", "plr-out:node_0")+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[usage_event])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)+ sentinel_graph = mocker.MagicMock()+ mocker.patch(f"{_MODULE}.GraphSpecAssembler.assemble", return_value=sentinel_graph)++ pipe_output = PipeOutput(pipeline_run_id="plr-out")+ assemble_tracing_on_output(+ pipe_output=pipe_output,+ pipeline_run_id="plr-out",+ assemble_graph=True,+ assemble_usage=True,+ )++ assert pipe_output.graph_spec is sentinel_graph+ assert pipe_output.tokens_usages == [usage_event.tokens_usage]++ def test_on_output_leaves_fields_when_nothing_assembled(self, mocker: MockerFixture) -> None:+ """A no-op read leaves pipe_output's tracing fields untouched (no clobbering with None)."""+ self._enable_tracing(mocker)+ event_log = mocker.MagicMock()+ event_log.read_events = mocker.MagicMock(return_value=[])+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)++ pipe_output = PipeOutput(pipeline_run_id="plr-noop")+ assemble_tracing_on_output(+ pipe_output=pipe_output,+ pipeline_run_id="plr-noop",+ assemble_graph=True,+ assemble_usage=True,+ )++ assert pipe_output.graph_spec is None+ assert pipe_output.tokens_usages is None+ assert pipe_output.graph_assembly_error is None+ assert pipe_output.usage_assembly_error is Nonediff --git a/tests/unit/pipelex/pipeline/test_bundle_validator.py b/tests/unit/pipelex/pipeline/test_bundle_validator.pyindex b7ccf1dc2..f0969a5c4 100644--- a/tests/unit/pipelex/pipeline/test_bundle_validator.py+++ b/tests/unit/pipelex/pipeline/test_bundle_validator.py@@ -2,9 +2,9 @@These pin the tolerant per-pipe classification, the D3 union catch + recursive SKIPPEDcause-walk, the D7 wiring-before-signature precedence, the single collect-all aggregate-(no per-pipe early abort), and the per-sweep telemetry + report-registry lifecycle. The-seams (``prepare_pipe_job``) and the run primitive (``PipeRun``) are mocked here; the real-composition is exercised by the integration suite.+(no per-pipe early abort), and the per-sweep telemetry. The seams (``prepare_pipe_job``)+and the run primitive (``PipeRun``) are mocked here; the real composition is exercised by+the integration suite."""import pytest@@ -18,7 +18,6 @@ from pipelex.pipe_run.exceptions import DryRunError, PipeRunErrorfrom pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipe_signature.exceptions import SignaturesNotAllowedErrorfrom pipelex.pipeline.bundle_validator import BundleValidator, DryRunStatus-from pipelex.pipeline.pipeline_models import SpecialPipelineIdfrom pipelex.system.telemetry.events import EventName, EventProperty@@ -35,13 +34,12 @@ class TestBundleValidator:def _patch_env(self, mocker: MockerFixture, *, allowed_to_fail: list[str] | None = None):"""Patch the hub getters, ``prepare_pipe_job``, and the ``PipeRun`` class in bundle_validator.- Returns ``(validator, report_delegate, telemetry_manager, prepare_mock, pipe_run)`` so callers can- assert on the registry / telemetry / seam / run interactions. ``pipe_run.run`` defaults to a- success-returning ``AsyncMock`` — individual tests override its ``return_value`` / ``side_effect``.- Patching the ``PipeRun`` symbol (rather than reaching into the instance's protected ``_pipe_run``)- keeps the test off the validator's private surface.+ Returns ``(validator, telemetry_manager, prepare_mock, pipe_run)`` so callers can assert on the+ telemetry / seam / run interactions. ``pipe_run.run`` defaults to a success-returning+ ``AsyncMock`` — individual tests override its ``return_value`` / ``side_effect``. Patching the+ ``PipeRun`` symbol (rather than reaching into the instance's protected ``_pipe_run``) keeps the+ test off the validator's private surface."""- report_delegate = mocker.patch("pipelex.pipeline.bundle_validator.get_report_delegate").return_valuetelemetry_manager = mocker.patch("pipelex.pipeline.bundle_validator.get_telemetry_manager").return_valuemock_get_config = mocker.patch("pipelex.pipeline.bundle_validator.get_config")mock_get_config.return_value.pipelex.dry_run_config.allowed_to_fail_pipes = allowed_to_fail or []@@ -50,11 +48,11 @@ class TestBundleValidator:pipe_run = mocker.MagicMock(name="pipe_run")pipe_run.run = mocker.AsyncMock(return_value=mocker.MagicMock(name="pipe_output"))mocker.patch("pipelex.pipeline.bundle_validator.PipeRun", return_value=pipe_run)- return BundleValidator(), report_delegate, telemetry_manager, prepare_mock, pipe_run+ return BundleValidator(), telemetry_manager, prepare_mock, pipe_run@pytest.mark.asyncioasync def test_success_classification(self, mocker: MockerFixture) -> None:- validator, _report, _telemetry, _prepare, _pipe_run = self._patch_env(mocker)+ validator, _telemetry, _prepare, _pipe_run = self._patch_env(mocker)pipe = self._make_pipe(mocker, code="ok_pipe", pipe_ref="dom.ok_pipe")results = await validator.validate_pipes([pipe], library_id="lib-1")@@ -67,7 +65,7 @@ class TestBundleValidator:async def test_skipped_when_prepare_raises_bare_pipe_not_found(self, mocker: MockerFixture) -> None:# A cross-package unresolved dependency surfaces during mock-input build (prepare_pipe_job)# as a bare PipeNotFoundError — reclassify as SKIPPED, never SUCCESS, never a sweep abort.- validator, _report, _telemetry, prepare_mock, _pipe_run = self._patch_env(mocker)+ validator, _telemetry, prepare_mock, _pipe_run = self._patch_env(mocker)prepare_mock.side_effect = PipeNotFoundError("dep->other.missing not found")pipe = self._make_pipe(mocker, code="dep_pipe", pipe_ref="dom.dep_pipe")@@ -82,7 +80,7 @@ class TestBundleValidator:async def test_skipped_when_run_raises_wrapped_pipe_not_found(self, mocker: MockerFixture) -> None:# Routing through PipeRun.run no longer surfaces a bare PipeNotFoundError: the run layer# re-raises and the router wraps it. The recursive __cause__ walk must still reclassify SKIPPED.- validator, _report, _telemetry, _prepare, pipe_run = self._patch_env(mocker)+ validator, _telemetry, _prepare, pipe_run = self._patch_env(mocker)cause = PipeNotFoundError("dep->other.missing not found")wrapped = PipeRunError(message="run failed", run_mode=PipeRunMode.DRY, pipe_code="dep_pipe")wrapped.__cause__ = cause@@ -97,7 +95,7 @@ class TestBundleValidator:async def test_allowed_failure_returned_in_map_without_raising(self, mocker: MockerFixture) -> None:# A FactoryException (e.g. a PipeSignature mint) on an allowed-to-fail pipe is a handled FAILURE# carried in the returned map — the sweep does NOT raise because it is not an unexpected failure.- validator, _report, _telemetry, _prepare, pipe_run = self._patch_env(mocker, allowed_to_fail=["dom.allowed_pipe"])+ validator, _telemetry, _prepare, pipe_run = self._patch_env(mocker, allowed_to_fail=["dom.allowed_pipe"])pipe_run.run = mocker.AsyncMock(side_effect=FactoryException("polyfactory could not build mock content"))pipe = self._make_pipe(mocker, code="allowed_pipe", pipe_ref="dom.allowed_pipe")@@ -113,7 +111,7 @@ class TestBundleValidator:# only the bare code ("allowed_pipe") must NOT tolerate the namespaced pipe ("dom.allowed_pipe") —# the failure is unexpected and aggregates into a DryRunError. (The positive namespaced match is# pinned by test_allowed_failure_returned_in_map_without_raising.)- validator, _report, _telemetry, _prepare, pipe_run = self._patch_env(mocker, allowed_to_fail=["allowed_pipe"])+ validator, _telemetry, _prepare, pipe_run = self._patch_env(mocker, allowed_to_fail=["allowed_pipe"])pipe_run.run = mocker.AsyncMock(side_effect=FactoryException("polyfactory could not build mock content"))pipe = self._make_pipe(mocker, code="allowed_pipe", pipe_ref="dom.allowed_pipe")@@ -125,7 +123,7 @@ class TestBundleValidator:async def test_unexpected_validation_error_raises_dry_run_error(self, mocker: MockerFixture) -> None:# A pydantic ValidationError on a non-allowed pipe classifies FAILURE and is aggregated into a# single DryRunError (third-party-exception classify — FAILURE, not an escaping traceback).- validator, _report, _telemetry, _prepare, pipe_run = self._patch_env(mocker)+ validator, _telemetry, _prepare, pipe_run = self._patch_env(mocker)class _Tiny(BaseModel):value: int@@ -144,7 +142,7 @@ class TestBundleValidator:# D3 widening: a non-dependency PipelexError raised mid-run is a per-pipe FAILURE, NOT a sweep# abort. The narrow tuple the old path used would let it escape and abort before the next pipe;# the base-PipelexError catch keeps the loop going, so the second pipe still runs.- validator, _report, _telemetry, _prepare, pipe_run = self._patch_env(mocker)+ validator, _telemetry, _prepare, pipe_run = self._patch_env(mocker)pipe_run.run = mocker.AsyncMock(side_effect=[PipelexError("unexpected domain failure"), mocker.MagicMock(name="ok_output")])failing_pipe = self._make_pipe(mocker, code="boom_pipe", pipe_ref="dom.boom_pipe")ok_pipe = self._make_pipe(mocker, code="ok_pipe", pipe_ref="dom.ok_pipe")@@ -161,7 +159,7 @@ class TestBundleValidator:async def test_collect_all_unexpected_failures_reported(self, mocker: MockerFixture) -> None:# Collect-all, not first-failure-abort: a sweep with >= 2 non-allowed failures reports BOTH in one# aggregated error.- validator, _report, _telemetry, _prepare, pipe_run = self._patch_env(mocker)+ validator, _telemetry, _prepare, pipe_run = self._patch_env(mocker)pipe_run.run = mocker.AsyncMock(side_effect=[PipelexError("fail one"), PipelexError("fail two")])pipe_a = self._make_pipe(mocker, code="a_pipe", pipe_ref="dom.a_pipe")pipe_b = self._make_pipe(mocker, code="b_pipe", pipe_ref="dom.b_pipe")@@ -176,7 +174,7 @@ class TestBundleValidator:async def test_strict_signature_pre_pass_raises_before_sweep(self, mocker: MockerFixture) -> None:# In strict mode a reached signature raises the aggregated SignaturesNotAllowedError before any# pipe is dry-run — so the seam (prepare_pipe_job) is never invoked.- validator, _report, _telemetry, prepare_mock, _pipe_run = self._patch_env(mocker)+ validator, _telemetry, prepare_mock, _pipe_run = self._patch_env(mocker)signature_pipe = self._make_pipe(mocker, code="sig_pipe", pipe_ref="dom.sig_pipe", is_signature=True)with pytest.raises(SignaturesNotAllowedError) as exc_info:@@ -188,7 +186,7 @@ class TestBundleValidator:async def test_validate_with_libraries_error_precedes_signature_pre_pass(self, mocker: MockerFixture) -> None:# D7 precedence: the static wiring pass runs (and can raise) BEFORE the signature pre-pass, so a# wiring failure surfaces ahead of a signature error even when both are present.- validator, _report, _telemetry, _prepare, _pipe_run = self._patch_env(mocker)+ validator, _telemetry, _prepare, _pipe_run = self._patch_env(mocker)spy_collect = mocker.patch("pipelex.pipeline.bundle_validator.collect_signature_refs")wiring_pipe = self._make_pipe(mocker, code="wiring_pipe", pipe_ref="dom.wiring_pipe")# Stand-in for a wiring error (real validate_with_libraries raises PipeValidationError).@@ -206,7 +204,7 @@ class TestBundleValidator:# signature-checked: the strict pre-pass runs over the FULL pipe list, not just the wiring# survivors. Otherwise an unimplemented PipeSignature hidden behind a wiring gap would silently# pass strict validation (the old dry_run_pipes ran the pre-pass over all pipes first).- validator, _report, _telemetry, prepare_mock, _pipe_run = self._patch_env(mocker)+ validator, _telemetry, prepare_mock, _pipe_run = self._patch_env(mocker)# Step 1 drops this pipe to SKIPPED — its wiring references an unloaded cross-package sub-pipe...pipe = self._make_pipe(mocker, code="hybrid_pipe", pipe_ref="dom.hybrid_pipe")pipe.validate_with_libraries.side_effect = PipeNotFoundError("hybrid->other.missing not found")@@ -223,7 +221,7 @@ class TestBundleValidator:@pytest.mark.asyncioasync def test_one_pipe_dry_run_event_emitted_for_the_sweep(self, mocker: MockerFixture) -> None:- validator, _report, telemetry_manager, _prepare, _pipe_run = self._patch_env(mocker)+ validator, telemetry_manager, _prepare, _pipe_run = self._patch_env(mocker)pipes = [self._make_pipe(mocker, code=f"p{idx}", pipe_ref=f"dom.p{idx}") for idx in range(3)]await validator.validate_pipes(pipes, library_id="lib-1")@@ -234,16 +232,17 @@ class TestBundleValidator:)@pytest.mark.asyncio- async def test_registry_opened_once_and_closed_in_finally(self, mocker: MockerFixture) -> None:- # One report registry per sweep, keyed by the constant DRY_RUN_UNTITLED id, closed in `finally`- # even when a pipe run raises an uncaught (non-classified) exception — so the constant id cannot- # collide on the next sweep.- validator, report_delegate, _telemetry, _prepare, pipe_run = self._patch_env(mocker)+ async def test_sweep_threads_unique_dry_run_id_into_prepare(self, mocker: MockerFixture) -> None:+ # Each sweep dry-runs its pipes under a UNIQUE per-sweep pipeline run id (a `dry_run_`-prefixed+ # uuid) threaded into prepare_pipe_job. The live registry is gone (usage rides on PipeOutput), so+ # there is no per-sweep cleanup to verify — but the id threading survives. Pin that prepare is+ # called with a `dry_run_`-prefixed id even when a pipe run raises an uncaught exception.+ validator, _telemetry, prepare, pipe_run = self._patch_env(mocker)pipe_run.run = mocker.AsyncMock(side_effect=KeyError("uncaught programming bug"))pipe = self._make_pipe(mocker, code="p", pipe_ref="dom.p")with pytest.raises(KeyError):await validator.validate_pipes([pipe], library_id="lib-1")- report_delegate.open_registry.assert_called_once_with(pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)- report_delegate.close_registry.assert_called_once_with(pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)+ threaded_id = prepare.call_args.kwargs["pipeline_run_id"]+ assert threaded_id.startswith("dry_run_")diff --git a/tests/unit/pipelex/reporting/test_cost_report_renderer.py b/tests/unit/pipelex/reporting/test_cost_report_renderer.pynew file mode 100644index 000000000..a093a6312--- /dev/null+++ b/tests/unit/pipelex/reporting/test_cost_report_renderer.py@@ -0,0 +1,157 @@+"""Unit tests for the Phase-3 end-of-run cost-report renderer.++``render_run_cost_report`` is the single event-sourced renderer both CLIs call: it reads the+usage assembled onto ``pipe_output.tokens_usages`` and renders it through the two D6 channels+(console + CSV), each gated by the reporting config and the resolved ``--costs`` switch.+"""++from pathlib import Path++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.exceptions import CostRegistryError+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_config+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.reporting.cost_report_renderer import render_run_cost_report++_RUN_ID = "test-run"+++def _usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},+ )+++def _zero_cost_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ """A dry-run-shaped usage: no tokens -> zero tokens and zero cost."""+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="dry-model",+ inference_model_id="dry-run",+ nb_tokens_by_category={},+ unit_costs={},+ )+++def _free_model_usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ """A free/zero-price model: real tokens, no unit costs -> tokens but zero cost."""+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="ollama-x",+ inference_model_id="ollama-x-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={},+ )+++class TestRenderRunCostReport:+ def _set_channels(self, mocker: MockerFixture, *, console: bool, csv: bool, csv_dir: Path | None = None) -> None:+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", console)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", csv)+ if csv_dir is not None:+ mocker.patch.object(reporting_config, "cost_report_dir_path", str(csv_dir))++ def test_no_op_when_costs_off(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=True, csv=True)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_usage(job_metadata)], is_generate_costs=False)++ spy.assert_not_called()++ def test_no_op_when_tokens_none(self, mocker: MockerFixture) -> None:+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=True, csv=True)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=None, is_generate_costs=True)++ spy.assert_not_called()++ def test_no_op_when_tokens_empty(self, mocker: MockerFixture) -> None:+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=True, csv=True)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[], is_generate_costs=True)++ spy.assert_not_called()++ def test_no_op_when_no_reportable_usage(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ """A dry-run-shaped usage (zero tokens, zero cost) does no reportable work -> no report, channels on."""+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=True, csv=True)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_zero_cost_usage(job_metadata)], is_generate_costs=True)++ spy.assert_not_called()++ def test_renders_free_model_with_tokens(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ """A free/zero-price model with real tokens IS reported (cost 0) — not suppressed like a dry run."""+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=True, csv=False)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_free_model_usage(job_metadata)], is_generate_costs=True)++ spy.assert_called_once()+ aggregated = spy.call_args.args[0]+ assert aggregated.total_cost == 0.0+ assert aggregated.total_nb_tokens == 150++ def test_no_op_when_both_channels_off(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=False, csv=False)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_usage(job_metadata)], is_generate_costs=True)++ spy.assert_not_called()++ def test_console_channel_only(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=True, csv=False)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_usage(job_metadata)], is_generate_costs=True)++ spy.assert_called_once()+ assert spy.call_args.args[0].total_cost == 0.2 # the aggregated total, computed once+ kwargs = spy.call_args.kwargs+ assert kwargs["pipeline_run_id"] == _RUN_ID+ assert kwargs["print_to_console"] is True+ assert kwargs["cost_report_file_path"] is None++ def test_csv_channel_only(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._set_channels(mocker, console=False, csv=True, csv_dir=tmp_path)++ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_usage(job_metadata)], is_generate_costs=True)++ spy.assert_called_once()+ kwargs = spy.call_args.kwargs+ assert kwargs["print_to_console"] is False+ cost_report_file_path = kwargs["cost_report_file_path"]+ assert cost_report_file_path is not None+ assert Path(cost_report_file_path).parent == tmp_path++ @pytest.mark.parametrize(+ "render_error",+ [+ UnicodeEncodeError("utf-8", "\ud800", 0, 1, "surrogates not allowed"),+ CostRegistryError("aggregation blew up"),+ OSError("disk full while writing CSV"),+ ],+ )+ def test_render_failure_never_fails_the_run(self, mocker: MockerFixture, job_metadata: JobMetadata, render_error: Exception) -> None:+ """A reporting-side failure (UTF-8 CSV encode, cost-registry, or I/O) is caught — the run still succeeds."""+ mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report", side_effect=render_error)+ self._set_channels(mocker, console=True, csv=False)++ # Must not raise: the guard degrades the failure to a warning rather than failing the successful run.+ render_run_cost_report(pipeline_run_id=_RUN_ID, tokens_usages=[_usage(job_metadata)], is_generate_costs=True)diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.pyindex cd3bdd771..4da138ab2 100644--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py@@ -25,7 +25,7 @@ from pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.config import get_configfrom pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graph_context import GraphContext+from pipelex.graph.trace_context import TraceContextfrom pipelex.pipeline.job_metadata import JobMetadata, UnitJobIdfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend@@ -47,7 +47,7 @@ DATA_INCLUSION_OFF = DataInclusionConfig(def _make_llm_job(pipeline_run_id: str,- graph_context: GraphContext | None,+ trace_context: TraceContext | None,nb_input_tokens: int = 100,nb_output_tokens: int = 50,) -> LLMJob:@@ -55,7 +55,7 @@ def _make_llm_job(job_metadata = JobMetadata(user_id="test_user",pipeline_run_id=pipeline_run_id,- graph_context=graph_context,+ trace_context=trace_context,started_at=now,completed_at=now + timedelta(seconds=1),unit_job_id=UnitJobId.LLM_GEN_TEXT,@@ -76,12 +76,12 @@ def _make_llm_job()-def _make_graph_context(+def _make_trace_context(graph_id: str,parent_node_id: str | None = "g:node_0",tracer_key: str | None = None,-) -> GraphContext:- return GraphContext(+) -> TraceContext:+ return TraceContext(graph_id=graph_id,tracer_key=tracer_key,parent_node_id=parent_node_id,@@ -119,8 +119,8 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz", parent_node_id="g:node_2")- llm_job = _make_llm_job("run_abc", graph_context=graph_context)+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz", parent_node_id="g:node_2")+ llm_job = _make_llm_job("run_abc", trace_context=trace_context)manager.report_inference_job(llm_job)@@ -143,19 +143,19 @@ class TestEmitRunnerFallback:tmp_path: Path,mocker: MockerFixture,) -> None:- """workflow_id on the emitted event is graph_context.tracer_key when set, else graph_id."""+ """workflow_id on the emitted event is trace_context.tracer_key when set, else graph_id."""_enable_ndjson_tracing(mocker, tmp_path)manager = ReportingManager()manager.setup()# Case 1: tracer_key set → workflow_id == tracer_key- ctx_with_tracer = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")- manager.report_inference_job(_make_llm_job("run_abc", graph_context=ctx_with_tracer))+ ctx_with_tracer = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=ctx_with_tracer))# Case 2: tracer_key None → workflow_id == graph_id- ctx_no_tracer = _make_graph_context(graph_id="run_def", tracer_key=None)- manager.report_inference_job(_make_llm_job("run_def", graph_context=ctx_no_tracer))+ ctx_no_tracer = _make_trace_context(graph_id="run_def", tracer_key=None)+ manager.report_inference_job(_make_llm_job("run_def", trace_context=ctx_no_tracer))reader = NdjsonEventLog(traces_dir=str(tmp_path))events_abc = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]@@ -177,10 +177,10 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))assert spy.call_count == 1@@ -232,25 +232,25 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_silent", tracer_key="wf_silent")+ trace_context = _make_trace_context(graph_id="run_silent", tracer_key="wf_silent")- manager.report_inference_job(_make_llm_job("run_silent", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_silent", trace_context=trace_context))# No NDJSON files should have been created.assert list(tmp_path.glob("**/*.ndjson")) == []- def test_no_graph_context_skips_emit(+ def test_no_trace_context_skips_emit(self,tmp_path: Path,mocker: MockerFixture,) -> None:- """When job_metadata.graph_context is None, neither fast path nor fallback engages."""+ """When job_metadata.trace_context is None, neither fast path nor fallback engages."""_enable_ndjson_tracing(mocker, tmp_path)manager = ReportingManager()manager.setup()- manager.report_inference_job(_make_llm_job("run_abc", graph_context=None))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=None))assert list(tmp_path.glob("**/*.ndjson")) == []@@ -283,11 +283,11 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")with caplog.at_level(logging.WARNING):# Must not raise — the failure is logged and dropped.- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))assert any(record.levelno == logging.WARNING for record in caplog.records)@@ -314,10 +314,10 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")with caplog.at_level(logging.WARNING):- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))assert any(record.levelno == logging.WARNING for record in caplog.records)@@ -332,11 +332,11 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")with caplog.at_level(logging.WARNING):for _ in range(100):- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]assert len(engaged_records) == 1@@ -350,16 +350,16 @@ class TestEmitRunnerFallback:"""Module-level once-flag survives multiple ReportingManager instances."""_enable_ndjson_tracing(mocker, tmp_path)- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")with caplog.at_level(logging.WARNING):manager_a = ReportingManager()manager_a.setup()- manager_a.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager_a.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))manager_b = ReportingManager()manager_b.setup()- manager_b.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager_b.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))engaged_records = [record for record in caplog.records if "runner-side" in record.message.lower()]assert len(engaged_records) == 1@@ -377,10 +377,10 @@ class TestEmitRunnerFallback:manager = ReportingManager()manager.setup()- graph_context = _make_graph_context(graph_id="run_abc", tracer_key="wf_xyz")+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))- manager.report_inference_job(_make_llm_job("run_abc", graph_context=graph_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))reader = NdjsonEventLog(traces_dir=str(tmp_path))events = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]diff --git a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.pynew file mode 100644index 000000000..d48ee6b66--- /dev/null+++ b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py@@ -0,0 +1,178 @@+"""Unit tests for the emit_usage_events gate in ReportingManager (Phase 1 decoupling).++Pins the load-bearing decoupling: usage emission is gated by ``trace_context.emit_usage_events``,+independent of graph events. In graph-only mode (``--graph --no-costs``) no usage event-log context+is registered, so reporting lands on the runner fallback — which must suppress the usage event. In+costs-only / default mode the same fallback emits. This is what lets ``--no-graph`` keep cost data+and ``--no-costs`` drop it while both share one event-log transport.+"""++from datetime import datetime, timedelta, timezone+from pathlib import Path++from pytest_mock import MockerFixture++from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_config+from pipelex.graph.graph_config import DataInclusionConfig+from pipelex.graph.trace_context import TraceContext+from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId+from pipelex.reporting.reporting_manager import ReportingManager+from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend+from pipelex.tracing.activity_event_log import ActivityEventLogCache+from pipelex.tracing.ndjson_event_log import NdjsonEventLog+from pipelex.tracing.trace_events import UsageReportEvent++DATA_INCLUSION_OFF = DataInclusionConfig(+ pipe_and_concept_registry=False,+ stuff_json_content=False,+ stuff_text_content=False,+ stuff_html_content=False,+ error_stack_traces=False,+)+++def _make_llm_job(pipeline_run_id: str, trace_context: TraceContext | None) -> LLMJob:+ now = datetime.now(timezone.utc)+ job_metadata = JobMetadata(+ user_id="test_user",+ pipeline_run_id=pipeline_run_id,+ trace_context=trace_context,+ started_at=now,+ completed_at=now + timedelta(seconds=1),+ unit_job_id=UnitJobId.LLM_GEN_TEXT,+ )+ tokens_usage = LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ )+ return LLMJob(+ job_metadata=job_metadata,+ llm_prompt=LLMPrompt(),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(schema_reask_max_attempts=1),+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),+ )+++def _make_trace_context(graph_id: str, *, emit_usage_events: bool, emit_graph_events: bool = True) -> TraceContext:+ return TraceContext(+ graph_id=graph_id,+ tracer_key=f"wf_{graph_id}",+ parent_node_id=f"{graph_id}:node_0",+ node_sequence=0,+ data_inclusion=DATA_INCLUSION_OFF,+ emit_graph_events=emit_graph_events,+ emit_usage_events=emit_usage_events,+ )+++def _enable_ndjson_tracing(mocker: MockerFixture, traces_dir: Path) -> None:+ cfg = get_config().pipelex.tracing_config+ mocker.patch.object(cfg, "is_enabled", True)+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=str(traces_dir)))+++class TestEmitUsageEventGating:+ """Pins that usage emission honors trace_context.emit_usage_events on BOTH paths.++ The gate lives in the dispatcher (_emit_usage_event), before the context lookup, so it guards the+ fast path (a registered set_event_log context) and the runner fallback alike — correctness no+ longer rests on the cross-file invariant "a context is registered only when costs are on".+ """++ def test_graph_only_suppresses_usage_event(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """--graph --no-costs: emit_usage_events=False -> the runner fallback writes no usage event."""+ ActivityEventLogCache.reset_for_tests()+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()+ trace_context = _make_trace_context("run_graph_only", emit_usage_events=False, emit_graph_events=True)++ manager.report_inference_job(_make_llm_job("run_graph_only", trace_context=trace_context))++ # Nothing emitted anywhere: no usage context was registered (set_event_log skipped) and the+ # fallback returns early on emit_usage_events=False.+ assert list(tmp_path.glob("**/*.ndjson")) == []+ ActivityEventLogCache.reset_for_tests()++ def test_costs_enabled_emits_usage_event_via_fallback(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """Costs on: emit_usage_events=True -> the runner fallback emits the usage event as before."""+ ActivityEventLogCache.reset_for_tests()+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()+ trace_context = _make_trace_context("run_costs", emit_usage_events=True, emit_graph_events=False)++ manager.report_inference_job(_make_llm_job("run_costs", trace_context=trace_context))++ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ usage_events = [evt for evt in reader.read_events("run_costs") if isinstance(evt, UsageReportEvent)]+ assert len(usage_events) == 1+ assert usage_events[0].pipeline_run_id == "run_costs"+ assert usage_events[0].writer_id.startswith("act_")+ ActivityEventLogCache.reset_for_tests()++ def test_registered_context_suppresses_usage_when_costs_off(self, mocker: MockerFixture) -> None:+ """Fast path: a REGISTERED context with emit_usage_events=False must emit nothing.++ Simulates the leak scenario the hoisted gate defends against — a usage event-log context that+ outlived its run (clear_event_log skipped) and collides on lookup_key with a later graph-only+ run. Without the gate the fast path would emit; the dispatcher's early-return suppresses it.+ """+ manager = ReportingManager()+ manager.setup()+ trace_context = _make_trace_context("run_leaked", emit_usage_events=False, emit_graph_events=True)++ event_log_spy = mocker.MagicMock()+ event_log_spy.writer_id = "leaked_writer"+ event_log_spy.next_sequence.return_value = 0+ manager.set_event_log(+ context_key=trace_context.lookup_key,+ event_log=event_log_spy,+ workflow_id="direct",+ pipeline_run_id="run_leaked",+ )++ manager.report_inference_job(_make_llm_job("run_leaked", trace_context=trace_context))++ event_log_spy.emit.assert_not_called()++ def test_registered_context_emits_when_costs_on(self, mocker: MockerFixture) -> None:+ """Control: with the SAME registration but emit_usage_events=True, the fast path DOES emit.++ Proves the suppression above is the gate's doing, not a mis-wired registration that never+ reaches the fast path.+ """+ manager = ReportingManager()+ manager.setup()+ trace_context = _make_trace_context("run_fastpath", emit_usage_events=True, emit_graph_events=True)++ event_log_spy = mocker.MagicMock()+ event_log_spy.writer_id = "fastpath_writer"+ event_log_spy.next_sequence.return_value = 0+ manager.set_event_log(+ context_key=trace_context.lookup_key,+ event_log=event_log_spy,+ workflow_id="direct",+ pipeline_run_id="run_fastpath",+ )++ manager.report_inference_job(_make_llm_job("run_fastpath", trace_context=trace_context))++ event_log_spy.emit.assert_called_once()+ emitted_event = event_log_spy.emit.call_args.args[0]+ assert isinstance(emitted_event, UsageReportEvent)+ assert emitted_event.pipeline_run_id == "run_fastpath"+ assert emitted_event.writer_id == "fastpath_writer"diff --git a/tests/unit/pipelex/reporting/test_no_orphan_registries.py b/tests/unit/pipelex/reporting/test_no_orphan_registries.pydeleted file mode 100644index 3c32733bb..000000000--- a/tests/unit/pipelex/reporting/test_no_orphan_registries.py+++ /dev/null@@ -1,141 +0,0 @@-"""Pin the three-method registry split contract.--Phase 3 of the cross-worker tracing P0 plan replaces the single auto-creating-``_get_registry`` with two explicit methods:--- ``_get_registry_strict`` raises ``KeyError`` when the registry was never- opened. Used by the ``_report_*_job`` paths so the runner-process case- (no ``open_registry`` call) does not silently accumulate orphan registries.-- ``_get_or_create_registry`` creates on miss. Used by ``inject_tokens_usages``- (the P1 cross-worker assembly path) and ``generate_report`` for runs not- opened on this process.-"""--from datetime import datetime, timedelta, timezone--import pytest--from pipelex.cogt.llm.llm_job import LLMJob-from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.cogt.llm.llm_report import LLMTokensUsage-from pipelex.cogt.usage.cost_category import CostCategory-from pipelex.cogt.usage.token_category import TokenCategory-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId-from pipelex.reporting.reporting_manager import ReportingManager---def _make_llm_job(pipeline_run_id: str) -> LLMJob:- now = datetime.now(timezone.utc)- job_metadata = JobMetadata(- user_id="test_user",- pipeline_run_id=pipeline_run_id,- started_at=now,- completed_at=now + timedelta(seconds=1),- unit_job_id=UnitJobId.LLM_GEN_TEXT,- )- tokens_usage = LLMTokensUsage(- job_metadata=job_metadata,- inference_model_name="test-model",- inference_model_id="test-model-id",- unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},- nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},- )- return LLMJob(- job_metadata=job_metadata,- llm_prompt=LLMPrompt(),- job_params=LLMJobParams(temperature=0.5),- job_config=LLMJobConfig(schema_reask_max_attempts=1),- job_report=LLMJobReport(llm_tokens_usage=tokens_usage),- )---class TestNoOrphanRegistries:- """Phase 3 contract: runner-process reporting must not silently auto-create registries."""-- def test_runner_does_not_accumulate_registries(self) -> None:- """When open_registry was never called, _report_llm_job must not insert a registry.-- The runner process gets activities reported with pipeline_run_ids it has never seen.- Pre-Phase-3 behavior auto-created an empty registry on every miss, accumulating- memory for every distinct pipeline_run_id. Phase 3 makes _report_*_job silently skip- the add (the usage event still emits via the runner fallback path).- """- manager = ReportingManager()- manager.setup()-- manager.report_inference_job(_make_llm_job("never_opened"))-- assert "never_opened" not in manager._usage_registries # noqa: SLF001 # pyright: ignore[reportPrivateUsage]-- def test_direct_mode_still_accumulates_registry(self) -> None:- """When open_registry was called for this run, _report_llm_job must add to it."""- manager = ReportingManager()- manager.setup()- manager.open_registry("run_x")-- manager.report_inference_job(_make_llm_job("run_x"))-- registry = manager._get_or_create_registry("run_x") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- usages = registry.get_current_tokens_usage()- assert len(usages) == 1- assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 100-- def test_inject_tokens_usages_still_creates_on_miss(self) -> None:- """inject_tokens_usages preserves auto-create behavior via _get_or_create_registry.-- This is the P1 cross-worker assembly path: usage records collected from trace- events get injected into a registry that may not have been opened on this- process — the registry must be created on demand.- """- manager = ReportingManager()- manager.setup()-- tokens_usage = LLMTokensUsage(- job_metadata=_make_llm_job("new_run").job_metadata,- inference_model_name="test-model",- inference_model_id="test-model-id",- unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},- nb_tokens_by_category={TokenCategory.INPUT: 50, TokenCategory.OUTPUT: 25},- )- manager.inject_tokens_usages(pipeline_run_id="new_run", tokens_usages=[tokens_usage])-- registry = manager._get_or_create_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- assert len(registry.get_current_tokens_usage()) == 1-- def test_get_registry_strict_raises_when_missing(self) -> None:- """_get_registry_strict raises KeyError on miss — distinct from _get_or_create_registry."""- manager = ReportingManager()- manager.setup()-- with pytest.raises(KeyError):- manager._get_registry_strict("never_opened") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]-- def test_close_registry_is_idempotent_on_missing(self) -> None:- """close_registry on an unopened (or already-closed) run is a no-op, never a KeyError.-- close_registry runs from sweep / runner ``finally`` blocks (e.g. BundleValidator's- per-sweep registry). A bare ``pop`` would raise KeyError there and mask the in-flight- exception. Pin the defaulted-pop hardening: closing a run that was never opened, and- double-closing an opened run, must both be silent.- """- manager = ReportingManager()- manager.setup()-- # Never opened → no-op.- manager.close_registry("never_opened")-- # Opened then double-closed → first removes it, second is a silent no-op.- manager.open_registry("run_once")- manager.close_registry("run_once")- manager.close_registry("run_once")-- assert "run_once" not in manager._usage_registries # noqa: SLF001 # pyright: ignore[reportPrivateUsage]-- def test_generate_report_for_unopened_run_creates_empty_registry_and_renders(self) -> None:- """generate_report on a never-opened run does not crash; renders empty cost report."""- manager = ReportingManager()- manager.setup()-- # Must not raise.- manager.generate_report(pipeline_run_id="never_opened")diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.pyindex da74d1b35..8efa6c992 100644--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py@@ -1,12 +1,14 @@"""Unit tests for ReportingManager event emission.Validates that when an EventLogProtocol is provided, ReportingManager emits-UsageReportEvent alongside existing local accumulation, with per-context-isolation for concurrent workflows.+UsageReportEvent with per-context isolation for concurrent workflows."""+import loggingfrom datetime import datetime, timedelta, timezone+import pytest+from botocore.exceptions import ClientErrorfrom pytest_mock import MockerFixturefrom pipelex.cogt.llm.llm_job import LLMJob@@ -17,7 +19,7 @@ from pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.config import get_configfrom pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graph_context import GraphContext+from pipelex.graph.trace_context import TraceContextfrom pipelex.pipeline.job_metadata import JobMetadata, UnitJobIdfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.tracing.in_memory_event_log import InMemoryEventLog@@ -34,14 +36,14 @@ DATA_INCLUSION_OFF = DataInclusionConfig(def _make_test_llm_job(pipeline_run_id: str,- graph_context: GraphContext | None = None,+ trace_context: TraceContext | None = None,) -> LLMJob:"""Create a minimal LLMJob for testing event emission."""now = datetime.now(timezone.utc)job_metadata = JobMetadata(user_id="test_user",pipeline_run_id=pipeline_run_id,- graph_context=graph_context,+ trace_context=trace_context,started_at=now,completed_at=now + timedelta(seconds=1),unit_job_id=UnitJobId.LLM_GEN_TEXT,@@ -62,14 +64,14 @@ def _make_test_llm_job()-def _make_graph_context(+def _make_trace_context(graph_id: str,parent_node_id: str | None = None,tracer_key: str | None = None,node_sequence: int = 0,-) -> GraphContext:- """Create a GraphContext for testing with lookup_key resolution."""- return GraphContext(+) -> TraceContext:+ """Create a TraceContext for testing with lookup_key resolution."""+ return TraceContext(graph_id=graph_id,tracer_key=tracer_key,parent_node_id=parent_node_id,@@ -87,7 +89,7 @@ class TestReportingEventEmission:def _make_reporting_manager_with_event_log(self) -> tuple[ReportingManager, InMemoryEventLog]:"""Create a ReportingManager configured with an event log.- The context_key is PIPELINE_RUN_ID, matching the lookup_key of GraphContexts+ The context_key is PIPELINE_RUN_ID, matching the lookup_key of TraceContextscreated with graph_id=PIPELINE_RUN_ID and no tracer_key."""event_log = InMemoryEventLog()@@ -99,7 +101,6 @@ class TestReportingEventEmission:workflow_id=self.WORKFLOW_ID,pipeline_run_id=self.PIPELINE_RUN_ID,)- manager.open_registry(self.PIPELINE_RUN_ID)return manager, event_log# ------------------------------------------------------------------@@ -110,27 +111,27 @@ class TestReportingEventEmission:"""UsageReportEvent is emitted when reporting an LLM inference job with event_log set."""manager, event_log = self._make_reporting_manager_with_event_log()- graph_context = _make_graph_context(graph_id=self.PIPELINE_RUN_ID)- llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, graph_context=graph_context)+ trace_context = _make_trace_context(graph_id=self.PIPELINE_RUN_ID)+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, trace_context=trace_context)manager.report_inference_job(llm_job)events = event_log.read_events(self.PIPELINE_RUN_ID)usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]assert len(usage_events) == 1assert usage_events[0].tokens_usage.nb_tokens_by_category[TokenCategory.INPUT] == 100- assert usage_events[0].node_id == "unknown" # No graph_context parent set+ assert usage_events[0].node_id == "unknown" # No trace_context parent set- def test_report_inference_job_captures_node_id_from_graph_context(self) -> None:+ def test_report_inference_job_captures_node_id_from_trace_context(self) -> None:"""UsageReportEvent captures the parent node_id from graph context."""manager, event_log = self._make_reporting_manager_with_event_log()- graph_context = _make_graph_context(+ trace_context = _make_trace_context(graph_id=self.PIPELINE_RUN_ID,parent_node_id="test-graph:node_42",node_sequence=43,)- llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, graph_context=graph_context)+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, trace_context=trace_context)manager.report_inference_job(llm_job)events = event_log.read_events(self.PIPELINE_RUN_ID)@@ -139,63 +140,52 @@ class TestReportingEventEmission:assert usage_events[0].node_id == "test-graph:node_42"assert usage_events[0].workflow_id == self.WORKFLOW_ID- # ------------------------------------------------------------------- # Registry tests (unchanged behavior)- # ------------------------------------------------------------------+ @pytest.mark.parametrize(+ "emit_error",+ [+ ClientError(+ error_response={"Error": {"Code": "ProvisionedThroughputExceededException"}},+ operation_name="PutItem",+ ),+ OSError("disk full"),+ ],+ ids=["client_error", "os_error"],+ )+ def test_registered_context_emit_failure_is_best_effort(+ self,+ mocker: MockerFixture,+ caplog: pytest.LogCaptureFixture,+ emit_error: Exception,+ ) -> None:+ """A backend emit failure on the registered-context fast path is dropped with a WARNING, never raised.++ report_inference_job runs synchronously after the inference has already succeeded (and been+ billed), so a transient event-log write failure — DynamoDB throttle/auth/timeout (ClientError)+ or NDJSON disk error (OSError) — must not propagate and turn a successful inference into a+ failed pipeline. Mirrors the runner-fallback contract in test_emit_runner_fallback.+ """+ manager, event_log = self._make_reporting_manager_with_event_log()+ mocker.patch.object(event_log, "emit", side_effect=emit_error)- def test_inject_tokens_usages_adds_to_registry(self) -> None:- """inject_tokens_usages adds externally-collected usage data to the pipeline's registry."""- manager = ReportingManager()- manager.setup()- manager.open_registry(self.PIPELINE_RUN_ID)-- tokens_usage = LLMTokensUsage(- job_metadata=_make_test_llm_job(self.PIPELINE_RUN_ID).job_metadata,- inference_model_name="test-model",- inference_model_id="test-model-id",- unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},- nb_tokens_by_category={TokenCategory.INPUT: 200, TokenCategory.OUTPUT: 100},- )+ trace_context = _make_trace_context(graph_id=self.PIPELINE_RUN_ID)+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, trace_context=trace_context)- manager.inject_tokens_usages(- pipeline_run_id=self.PIPELINE_RUN_ID,- tokens_usages=[tokens_usage],- )+ with caplog.at_level(logging.WARNING):+ # Must not raise — the failure is logged and dropped.+ manager.report_inference_job(llm_job)- registry = manager._get_or_create_registry(self.PIPELINE_RUN_ID) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- usages = registry.get_current_tokens_usage()- assert len(usages) == 1- assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 200+ assert any(record.levelno == logging.WARNING for record in caplog.records)- def test_inject_tokens_usages_auto_creates_registry(self) -> None:- """inject_tokens_usages auto-creates registry if it doesn't exist yet."""+ def test_setup_and_teardown_without_event_log(self) -> None:+ """Without any event_log registered, setup/teardown are inert and report_inference_job no-ops."""manager = ReportingManager()manager.setup()- tokens_usage = LLMTokensUsage(- job_metadata=_make_test_llm_job("new_run").job_metadata,- inference_model_name="test-model",- inference_model_id="test-model-id",- unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},- nb_tokens_by_category={TokenCategory.INPUT: 50, TokenCategory.OUTPUT: 25},- )-- manager.inject_tokens_usages(- pipeline_run_id="new_run",- tokens_usages=[tokens_usage],- )-- registry = manager._get_or_create_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- usages = registry.get_current_tokens_usage()- assert len(usages) == 1+ # A job whose trace_context is None must not emit anything and must not raise.+ manager.report_inference_job(_make_test_llm_job("direct_run", trace_context=None))- def test_no_event_log_works_as_before(self) -> None:- """ReportingManager without event_log works exactly as before."""- manager = ReportingManager()- manager.setup()- manager.open_registry("direct_run")- manager.close_registry("direct_run")manager.teardown()+ assert len(manager._event_log_contexts) == 0 # noqa: SLF001 # pyright: ignore[reportPrivateUsage]# ------------------------------------------------------------------# Per-context isolation regression tests@@ -210,14 +200,12 @@ class TestReportingEventEmission:manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")- manager.open_registry("run_a")- manager.open_registry("run_b")- ctx_a = _make_graph_context(graph_id="run_a", tracer_key="wf_a")- ctx_b = _make_graph_context(graph_id="run_b", tracer_key="wf_b")+ ctx_a = _make_trace_context(graph_id="run_a", tracer_key="wf_a")+ ctx_b = _make_trace_context(graph_id="run_b", tracer_key="wf_b")- job_a = _make_test_llm_job("run_a", graph_context=ctx_a)- job_b = _make_test_llm_job("run_b", graph_context=ctx_b)+ job_a = _make_test_llm_job("run_a", trace_context=ctx_a)+ job_b = _make_test_llm_job("run_b", trace_context=ctx_b)manager.report_inference_job(job_a)manager.report_inference_job(job_b)@@ -240,17 +228,15 @@ class TestReportingEventEmission:manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")- manager.open_registry("run_a")- manager.open_registry("run_b")# Clear context A before any emissionmanager.clear_event_log(context_key="wf_a")- ctx_a = _make_graph_context(graph_id="run_a", tracer_key="wf_a")- ctx_b = _make_graph_context(graph_id="run_b", tracer_key="wf_b")+ ctx_a = _make_trace_context(graph_id="run_a", tracer_key="wf_a")+ ctx_b = _make_trace_context(graph_id="run_b", tracer_key="wf_b")- job_a = _make_test_llm_job("run_a", graph_context=ctx_a)- job_b = _make_test_llm_job("run_b", graph_context=ctx_b)+ job_a = _make_test_llm_job("run_a", trace_context=ctx_a)+ job_b = _make_test_llm_job("run_b", trace_context=ctx_b)manager.report_inference_job(job_a) # Should NOT emit (context cleared)manager.report_inference_job(job_b) # Should emit@@ -260,11 +246,11 @@ class TestReportingEventEmission:assert len(events_a) == 0assert len(events_b) == 1- def test_no_graph_context_skips_emission(self) -> None:- """Jobs without graph_context skip event emission even when event logs are configured."""+ def test_no_trace_context_skips_emission(self) -> None:+ """Jobs without trace_context skip event emission even when event logs are configured."""manager, event_log = self._make_reporting_manager_with_event_log()- llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, graph_context=None)+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, trace_context=None)manager.report_inference_job(llm_job)events = event_log.read_events(self.PIPELINE_RUN_ID)@@ -292,16 +278,14 @@ class TestReportingEventEmission:manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")- manager.open_registry("run_a")- manager.open_registry("run_b")- ctx_a = _make_graph_context(graph_id="run_a", tracer_key="wf_a")- ctx_b = _make_graph_context(graph_id="run_b", tracer_key="wf_b")+ ctx_a = _make_trace_context(graph_id="run_a", tracer_key="wf_a")+ ctx_b = _make_trace_context(graph_id="run_b", tracer_key="wf_b")# Emit 3 events for context A, 1 for context Bfor _index in range(3):- manager.report_inference_job(_make_test_llm_job("run_a", graph_context=ctx_a))- manager.report_inference_job(_make_test_llm_job("run_b", graph_context=ctx_b))+ manager.report_inference_job(_make_test_llm_job("run_a", trace_context=ctx_a))+ manager.report_inference_job(_make_test_llm_job("run_b", trace_context=ctx_b))events_a = [evt for evt in event_log_a.read_events("run_a") if isinstance(evt, UsageReportEvent)]events_b = [evt for evt in event_log_b.read_events("run_b") if isinstance(evt, UsageReportEvent)]@@ -321,10 +305,9 @@ class TestReportingEventEmission:manager = ReportingManager()manager.setup()- manager.open_registry("run_silent")- graph_context = _make_graph_context(graph_id="run_silent", tracer_key="wf_silent", parent_node_id="g:node_0")- llm_job = _make_test_llm_job("run_silent", graph_context=graph_context)+ trace_context = _make_trace_context(graph_id="run_silent", tracer_key="wf_silent", parent_node_id="g:node_0")+ llm_job = _make_test_llm_job("run_silent", trace_context=trace_context)manager.report_inference_job(llm_job)# No event log was ever set; no event should have been emitted anywhere.diff --git a/tests/unit/pipelex/temporal/test_act_assemble_tracing_error_propagation.py b/tests/unit/pipelex/temporal/test_act_assemble_tracing_error_propagation.pynew file mode 100644index 000000000..dd32442ad--- /dev/null+++ b/tests/unit/pipelex/temporal/test_act_assemble_tracing_error_propagation.py@@ -0,0 +1,44 @@+"""Pins the error-propagation contract of ``act_assemble_tracing``.++Tracing assembly is best-effort observability: ``assemble_tracing`` catches the EXPECTED+read/assemble failures and returns them on the ``*_assembly_error`` fields (no raise). A+genuinely unexpected error (a programming bug in assembly) must instead PROPAGATE out of the+activity so that ``WfPipeRun``'s ``except ActivityError`` branch records it on the pipe output —+mirroring DIRECT mode, where ``assemble_tracing_on_output`` lets the same errors surface.+Swallowing it into an empty result would make the workflow see success and silently produce no+cost report and no diagnostic. These tests pin that the activity does not swallow unexpected+errors and otherwise passes the assembled result through unchanged.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.pipe_run.tracing_assembly import TracingAssembly+from pipelex.temporal.tprl_pipe.act_assemble_tracing import AssembleTracingArg, act_assemble_tracing++[email protected](loop_scope="class")+class TestActAssembleTracingErrorPropagation:+ async def test_unexpected_error_propagates(self, mocker: MockerFixture) -> None:+ """An unexpected error from assemble_tracing propagates instead of degrading to an empty result."""+ mocker.patch(+ "pipelex.temporal.tprl_pipe.act_assemble_tracing.assemble_tracing",+ side_effect=KeyError("unexpected programming bug in assembly"),+ )+ arg = AssembleTracingArg(pipeline_run_id="plr-boom", assemble_graph=True, assemble_usage=True)++ with pytest.raises(KeyError):+ await act_assemble_tracing(arg)++ async def test_assembled_result_passes_through(self, mocker: MockerFixture) -> None:+ """On success the activity returns assemble_tracing's result unchanged (including *_error fields)."""+ expected = TracingAssembly(usage_assembly_error="recorded read failure")+ mocker.patch(+ "pipelex.temporal.tprl_pipe.act_assemble_tracing.assemble_tracing",+ return_value=expected,+ )+ arg = AssembleTracingArg(pipeline_run_id="plr-ok", assemble_graph=False, assemble_usage=True)++ result = await act_assemble_tracing(arg)++ assert result == expecteddiff --git a/tests/unit/pipelex/temporal/test_content_generator_in_workflow_object_fidelity.py b/tests/unit/pipelex/temporal/test_content_generator_in_workflow_object_fidelity.pynew file mode 100644index 000000000..d2903d4ee--- /dev/null+++ b/tests/unit/pipelex/temporal/test_content_generator_in_workflow_object_fidelity.py@@ -0,0 +1,95 @@+"""Unit tests for the ``--mock-inference`` object-mock fidelity guard on the Temporal arm (review F2).++The activity (``act_llm_gen_object*``) builds the mock from the schema-reconstructed class, then+``ContentGeneratorInWorkflow.make_object`` re-validates it against the original class. When the original+class enforces an invariant the JSON-schema round-trip cannot capture (here a custom ``@field_validator``),+that re-validation fails — and must surface as a clear typed ``MockInferenceObjectFidelityError`` rather than+an opaque ``pydantic.ValidationError`` mid-workflow. The guard is scoped to ``is_mock_inference`` only: a LIVE+provider's invalid output keeps its existing ``ValidationError``. ``execute_activity`` is mocked to return the+synthetic object, so no Temporal server (or real provider) is involved.+"""++import pytest+from pydantic import BaseModel, ValidationError, field_validator+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.exceptions import MockInferenceObjectFidelityError+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+++class ConstrainedName(BaseModel):+ """Original output class with an invariant the JSON-schema round-trip cannot capture."""++ name: str++ @field_validator("name")+ @classmethod+ def _require_prefix(cls, value: str) -> str:+ if not value.startswith("PFX_"):+ msg = "name must start with 'PFX_'"+ raise ValueError(msg)+ return value+++class RawName(BaseModel):+ """Stand-in for what the (mocked) activity returns: a permissive reconstructed-class instance."""++ name: str+++def _make_generator() -> ContentGeneratorInWorkflow:+ return ContentGeneratorInWorkflow(generated_content_factory=GeneratedContentFactory(storage_provider=InMemoryStorageProvider()))+++def _mock_job_metadata(*, is_mock_inference: bool) -> JobMetadata:+ return JobMetadata(user_id="u", pipeline_run_id="run", pipe_code="my_pipe", is_mock_inference=is_mock_inference)++[email protected](loop_scope="class")[email protected]("patch_workflow_runtime")+class TestContentGeneratorInWorkflowObjectFidelity:+ @pytest.fixture+ def patch_workflow_runtime(self, mocker: MockerFixture) -> None:+ mocker.patch("temporalio.workflow.unsafe.is_replaying_history_events", return_value=False)++ async def test_make_object_raises_typed_error_on_fidelity_gap(self, mocker: MockerFixture) -> None:+ """A mock-inference object whose re-validation fails surfaces MockInferenceObjectFidelityError."""+ mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock, return_value=RawName(name="bad"))++ with pytest.raises(MockInferenceObjectFidelityError) as exc_info:+ await _make_generator().make_object(+ job_metadata=_mock_job_metadata(is_mock_inference=True),+ object_class=ConstrainedName,+ llm_setting_for_object=LLMSetting(model="test-llm", temperature=0.5),+ llm_prompt_for_object=LLMPrompt(user_text="hello"),+ )+ assert ConstrainedName.__name__ in str(exc_info.value)++ async def test_make_object_list_raises_typed_error_on_fidelity_gap(self, mocker: MockerFixture) -> None:+ """The list path applies the same per-item guard."""+ mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock, return_value=[RawName(name="bad")])++ with pytest.raises(MockInferenceObjectFidelityError):+ await _make_generator().make_object_list(+ job_metadata=_mock_job_metadata(is_mock_inference=True),+ object_class=ConstrainedName,+ llm_setting_for_object_list=LLMSetting(model="test-llm", temperature=0.5),+ llm_prompt_for_object_list=LLMPrompt(user_text="hello"),+ )++ async def test_non_mock_run_keeps_raw_validation_error(self, mocker: MockerFixture) -> None:+ """Outside --mock-inference the guard is inert: a re-validation failure stays a raw ValidationError."""+ mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock, return_value=RawName(name="bad"))++ with pytest.raises(ValidationError):+ await _make_generator().make_object(+ job_metadata=_mock_job_metadata(is_mock_inference=False),+ object_class=ConstrainedName,+ llm_setting_for_object=LLMSetting(model="test-llm", temperature=0.5),+ llm_prompt_for_object=LLMPrompt(user_text="hello"),+ )diff --git a/tests/unit/pipelex/tools/test_log_dispatch.py b/tests/unit/pipelex/tools/test_log_dispatch.pynew file mode 100644index 000000000..88a445415--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_dispatch.py@@ -0,0 +1,42 @@+from pytest_mock import MockerFixture++from pipelex import log+++class TestLogDispatchIncludeException:+ def test_include_exception_appends_traceback_to_console_message(self, mocker: MockerFixture) -> None:+ # Regression guard: `include_exception=True` must append the traceback to the string actually sent+ # to the console. The bug appended it to a dead local while the console was fed the pre-append+ # string, so server-side errors (e.g. the API's `log.error(..., include_exception=True)`) logged+ # only their summary line and never the underlying cause.+ to_console = mocker.patch.object(log.log_dispatch, "_log_to_console")++ cause = "boom-cause"+ try:+ raise ValueError(cause)+ except ValueError:+ log.error("something failed", include_exception=True)++ to_console.assert_called_once()+ call = to_console.call_args+ assert call is not None+ logged_message = call.kwargs["message"]+ assert "something failed" in logged_message+ assert "Traceback (most recent call last)" in logged_message+ assert "ValueError: boom-cause" in logged_message++ def test_no_traceback_when_include_exception_false(self, mocker: MockerFixture) -> None:+ to_console = mocker.patch.object(log.log_dispatch, "_log_to_console")++ cause = "boom-cause"+ try:+ raise ValueError(cause)+ except ValueError:+ log.error("something failed", include_exception=False)++ to_console.assert_called_once()+ call = to_console.call_args+ assert call is not None+ logged_message = call.kwargs["message"]+ assert "something failed" in logged_message+ assert "Traceback" not in logged_messagediff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.pyindex 7a3c5a191..769c0c7ca 100644--- a/tests/unit/pipelex/tracing/test_assembler_equivalence.py+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py@@ -14,9 +14,9 @@ from typing import Any, Callableimport pytest-from pipelex.graph.graph_context import GraphContextfrom pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, IOSpec, NodeKind, NodeSpec+from pipelex.graph.trace_context import TraceContextfrom pipelex.tracing.graphspec_assembler import GraphSpecAssemblerfrom pipelex.tracing.in_memory_event_log import InMemoryEventLogfrom tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config@@ -109,9 +109,9 @@ def _assert_graphs_equivalent(direct_spec: GraphSpec, assembled_spec: GraphSpec)# Scenario runners: execute the same scenario through both paths# ----------------------------------------------------------------------------# A scenario is a callable that takes a GraphTracer and its initial GraphContext,+# A scenario is a callable that takes a GraphTracer and its initial TraceContext,# and exercises a tracing scenario on it.-ScenarioFn = Callable[[GraphTracer, GraphContext], None]+ScenarioFn = Callable[[GraphTracer, TraceContext], None]def _run_both_paths(scenario: ScenarioFn) -> tuple[GraphSpec, GraphSpec]:@@ -151,13 +151,13 @@ def _run_both_paths(scenario: ScenarioFn) -> tuple[GraphSpec, GraphSpec]:# ----------------------------------------------------------------------------def _scenario_simple_sequence(tracer: GraphTracer, context: GraphContext) -> None:+def _scenario_simple_sequence(tracer: GraphTracer, context: TraceContext) -> None:"""3 child pipes in sequence under a parent controller."""started_at = _T0# Parent controllerparent_id, child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="sequence",pipe_type="PipeSequence",node_kind=NodeKind.CONTROLLER,@@ -166,7 +166,7 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: GraphContext) -> Non# Child 1: produces digest_achild1_id, _ = tracer.on_pipe_start(- graph_context=child_ctx,+ trace_context=child_ctx,pipe_code="gen_text",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -180,7 +180,7 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: GraphContext) -> Non# Child 2: consumes digest_a, produces digest_bchild2_id, _ = tracer.on_pipe_start(- graph_context=child_ctx,+ trace_context=child_ctx,pipe_code="refine",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -195,7 +195,7 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: GraphContext) -> Non# Child 3: consumes digest_bchild3_id, _ = tracer.on_pipe_start(- graph_context=child_ctx,+ trace_context=child_ctx,pipe_code="format",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -215,12 +215,12 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: GraphContext) -> Non)-def _scenario_parallel_branches(tracer: GraphTracer, context: GraphContext) -> None:+def _scenario_parallel_branches(tracer: GraphTracer, context: TraceContext) -> None:"""Controller with 2 branch pipes producing PARALLEL_COMBINE edges."""started_at = _T0ctrl_id, ctrl_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="parallel",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -229,7 +229,7 @@ def _scenario_parallel_branches(tracer: GraphTracer, context: GraphContext) -> N# Branch Abranch_a_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="branch_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -243,7 +243,7 @@ def _scenario_parallel_branches(tracer: GraphTracer, context: GraphContext) -> N# Branch Bbranch_b_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="branch_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -280,12 +280,12 @@ def _scenario_parallel_branches(tracer: GraphTracer, context: GraphContext) -> N)-def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: GraphContext) -> None:+def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: TraceContext) -> None:"""Batch controller with item extraction and aggregation."""started_at = _T0ctrl_id, ctrl_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="batch",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -309,7 +309,7 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: GraphContext) -# Branch 0: processes item 0branch0_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="process",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -324,7 +324,7 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: GraphContext) -# Branch 1: processes item 1branch1_id, _ = tracer.on_pipe_start(- graph_context=ctrl_ctx,+ trace_context=ctrl_ctx,pipe_code="process",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -359,12 +359,12 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: GraphContext) -)-def _scenario_partial_failure(tracer: GraphTracer, context: GraphContext) -> None:+def _scenario_partial_failure(tracer: GraphTracer, context: TraceContext) -> None:"""Child pipe fails, parent has no end event → FAILED + CANCELED nodes."""started_at = _T0_parent_id, child_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="sequence",pipe_type="PipeSequence",node_kind=NodeKind.CONTROLLER,@@ -372,7 +372,7 @@ def _scenario_partial_failure(tracer: GraphTracer, context: GraphContext) -> Non)child_id, _ = tracer.on_pipe_start(- graph_context=child_ctx,+ trace_context=child_ctx,pipe_code="gen_text",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -388,13 +388,13 @@ def _scenario_partial_failure(tracer: GraphTracer, context: GraphContext) -> Non# Parent has no end event → will be marked CANCELED-def _scenario_pass_through(tracer: GraphTracer, context: GraphContext) -> None:+def _scenario_pass_through(tracer: GraphTracer, context: TraceContext) -> None:"""Pass-through output (same digest as input) is not registered as producer."""started_at = _T0# Pass-through pipe: input digest_a → output digest_a (same)pt_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="passthrough",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -409,7 +409,7 @@ def _scenario_pass_through(tracer: GraphTracer, context: GraphContext) -> None:# Downstream pipe consumes digest_adown_id, _ = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="consumer",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -422,12 +422,12 @@ def _scenario_pass_through(tracer: GraphTracer, context: GraphContext) -> None:)-def _scenario_condition_selected_outcome(tracer: GraphTracer, context: GraphContext) -> None:+def _scenario_condition_selected_outcome(tracer: GraphTracer, context: TraceContext) -> None:"""Condition pipe with SELECTED_OUTCOME edge."""started_at = _T0cond_id, cond_ctx = tracer.on_pipe_start(- graph_context=context,+ trace_context=context,pipe_code="check",pipe_type="PipeCondition",node_kind=NodeKind.CONTROLLER,@@ -435,7 +435,7 @@ def _scenario_condition_selected_outcome(tracer: GraphTracer, context: GraphCont)outcome_id, _ = tracer.on_pipe_start(- graph_context=cond_ctx,+ trace_context=cond_ctx,pipe_code="branch_true",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,diff --git a/wip/distributed-execution/README.md b/wip/distributed-execution/README.mdindex b19d5762d..a5941cadf 100644--- a/wip/distributed-execution/README.md+++ b/wip/distributed-execution/README.md@@ -22,13 +22,13 @@ This folder is the **distributed-execution track**: running MTHDS methods as Tem| # | Item | Detail ||---|---|---|| **P0 ✓** | Tracing & cost reporting across separate-process workers | shipped — [tracing-cost-reporting.md](tracing-cost-reporting.md) |-| **P0.1** | Dry-run through activity dispatch (testing affordance) | below |+| **P0.1 ✓** | Dry-run through activity dispatch (testing affordance) | shipped as `--mock-inference` — below || **P0.2** | Deferred follow-ons from P0 | below |-| **P1** | Cross-worker cost report assembly wiring | depends on P0 — below |+| **P1 ✓** | Cross-worker cost report assembly | shipped (Option B — usage rides on `PipeOutput`) — below || **P2** | Local cross-package dependencies in the crate | [local-cross-package-deps.md](local-cross-package-deps.md) || **P3** | Remote dependencies from GitHub | needs P2 — [remote-deps-from-github.md](remote-deps-from-github.md) |-P0/P0.1/P0.2/P1 form one chain (P0 unlocks the rest); P2/P3 are an independent chain (P2 blocks P3). [tracing-cost-reporting.md](tracing-cost-reporting.md) carries the as-built design and the open T2/T3 gaps that motivate P1.+P0/P0.1/P0.2/P1 formed one chain (P0 unlocked the rest); **P0, P0.1, and P1 are now shipped** (see [the registry tracker](../registry/README.md) for the as-built sequencing). P0.2 is still open (independent follow-ons). P2/P3 are an independent chain (P2 blocks P3). [tracing-cost-reporting.md](tracing-cost-reporting.md) carries the as-built design; T1 and T2 are fixed, T3 (request-scoped tracing state) remains open.Sibling track (separate branch, not in this plan): the error-handling work — see [`error-handling/README.md`](../error-handling/README.md).@@ -40,13 +40,17 @@ Shipped. The as-built design, what works today, and the open T2/T3 gaps are in [----## P0.1 — Dry-run through activity dispatch (testing affordance)+## P0.1 — Dry-run through activity dispatch (testing affordance) — DONE-Today `--dry-run` instantiates `ContentGeneratorDry()` directly inside the workflow body (`pipe_llm.py:515`, and similar sites in `pipe_extract.py`, `pipe_compose.py`, `pipe_img_gen.py`). The activity (`act_llm_gen_text` etc.) is never dispatched, so the cross-worker `UsageReportEvent` emission path cannot be exercised in dry-run against router+runner workers — the runner-side fallback only fires when the activity actually dispatches across processes. (Observed validating P0: `pipelex run bundle --temporal --dry-run --mock-inputs` against a router+runner topology produces only `writer_id="primary"` events; no `wf_*__w_act_*.ndjson` files.)+Shipped as **`--mock-inference`** (distinct from `--dry-run`): a LIVE run (`run_mode` stays LIVE so operators dispatch normally) whose AI call is faked at the cogt inference *leaf*, inside the dispatched activity. So the cross-process surfaces — `_event_log_contexts` lookup miss, runner-side fallback, `writer_id="act_*"` emission — all fire, with no LLM spend.-What's needed: a run mode that keeps activity dispatch but mocks inference *inside* the activity (use `ContentGeneratorDry` inside the activity body, not in place of dispatching it), so the cross-process surfaces — `_event_log_contexts` lookup miss, runner-side fallback, `writer_id="act_*"` emission — all fire. Concretely: a `--mock-inference` flag distinct from `--dry-run` (or redefine `--dry-run` to dispatch-and-mock-inside and rename the current behavior); equivalent mock sites in `pipe_extract.py` / `pipe_compose.py` / `pipe_img_gen.py`; and `temporal-e2e-validate` Tier 8 able to run in this mode and surface `wf_*__w_act_*.ndjson` deterministically.+As built:-Beyond P0, the same affordance unlocks dry-run e2e for any cross-process work (P1 cost report assembly, distributed graph assembly, payload codec stress testing) without real LLM spend.+- `JobMetadata.is_mock_inference: bool` carries the signal into every activity (it crosses the Temporal boundary), written single-source from `prepare_pipe_job`. The leaf (`cogt/content_generation/llm_generate.py`) branches to a shared mock in `cogt/content_generation/dry_mock.py` when the flag is set.+- The synthetic usage is **reportable non-zero** (`MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}`, model `mock_inference`) — distinct from `--dry-run`'s zero-token, suppressed usage — so the cross-worker cost report actually renders. This is the durable reason `--mock-inference` ≠ `--dry-run` at the reporting layer.+- `temporal-e2e-validate` Tier 8b runs in this mode and surfaces `wf_*__w_act_*.ndjson` + a rendered cost report deterministically.++**Scope deviations (deliberate):** the leaf mock covers the **LLM leaf only**; image-gen / extract / search under `--mock-inference` fail loud with `MockInferenceUnsupportedError` (their output is stored *above* the leaf, so a leaf mock would push synthetic data through storage). `--mock-inference` is on the main `run` subcommands only (not the agent CLI). Full per-operator coverage and the eventual `is_mock_inference → run_mode=DRY` re-keying are tracked in [`../dry-run-refactor/followup-leaf-run-mode-mock.md`](../dry-run-refactor/followup-leaf-run-mode-mock.md) (B2). See [the registry feature](../registry/README.md) and its [`cost-reporting-overview.html`](../registry/cost-reporting-overview.html) for the full as-built.---@@ -62,21 +66,18 @@ Each is independently scoped and not blocking. Pick up individually as signals d----## P1 — Cross-worker cost report assembly wiring--### Why this is second, not first--Even with P0 shipped, the events still need to be turned into a cost report. The read-back path exists for the **graph** (`assemble_graph_on_output` / `act_assemble_graph` → `GraphSpecAssembler`) but not for **usage**. The pieces all exist on the manager side:+## P1 — Cross-worker cost report assembly — DONE-- `UsageAggregator.aggregate(events) → list[AnyTokensUsage]` (`pipelex/tracing/usage_aggregator.py`).-- `ReportingManager.inject_tokens_usages(pipeline_run_id, tokens_usages)` (`reporting_manager.py:223`) — docstring: *"Used after assembling usage data from distributed trace events, so that generate_report() can produce a complete cost report across all workers."*-- `ReportingManager.generate_report(pipeline_run_id)` (`reporting_manager.py:365`).+Shipped via **Option B** (the locked design; see [the registry overview](../registry/cost-reporting-overview.html)): rather than wire `read events → inject_tokens_usages → generate_report` onto the process-singleton manager, usage rides on `PipeOutput` exactly like the graph spec, and the submitter renders the report from that field. This resolved P1 **and** the `UsageRegistry` success-path leak in one move — with nothing populating a submitter-side registry, the registry was removed outright.-`generate_report()` has a runtime caller — the CLI run path invokes it at `_run_core.py:224` — but that path reports only from the **local, in-process** usage registries; it never reads back the distributed trace events. `UsageAggregator.aggregate()` and `inject_tokens_usages()` still have zero runtime callers, so `UsageReportEvent`s emitted by runner/worker processes sit in the backend and never get assembled into the cost report. The missing piece is the cross-worker read-back, not `generate_report` itself.+As built:-### The work+- The single trace-event read in `assemble_tracing` / `act_assemble_tracing` (renamed from `graph_assembly` / `act_assemble_graph`) feeds `UsageAggregator.aggregate()` when `--costs` is on, setting `PipeOutput.tokens_usages` in both direct and Temporal modes (the same hook that builds the `GraphSpec`).+- The submitter renders via `CostRegistry.generate_report(tokens_usages=...)` fed from `pipe_output.tokens_usages` (main CLI: `pipelex/reporting/cost_report_renderer.py::render_run_cost_report`).+- A dedicated `--costs` / `is_generate_costs` switch (default on) gates usage events + `UsageAggregator` over the shared event-log transport, independent of `--graph`.+- Retired: `ReportingManager.inject_tokens_usages` / `generate_report` / `open_registry` / `close_registry`, the `UsageRegistry` model, and the local-only registry fallback (`_get_registry_strict` / `_try_add_to_registry`) are all gone.-Read events for `pipeline_run_id` → `UsageAggregator.aggregate()` → `ReportingManager.inject_tokens_usages()` → `generate_report()`, wired in both direct mode (`pipelex/pipeline/runner.py`) and Temporal mode (alongside or inside the existing graph-assembly hook) — the same place that calls the graph assembler can call the usage aggregator and inject the result before `generate_report` runs. Verify the cross-worker case: parent workflow on Worker A, child on Worker B, activity on Worker C → one end-of-run cost report carrying usage from all three. Once the read-back lands, revisit the local-only registry fallback (`_get_registry_strict` at `reporting_manager.py:115`, `_try_add_to_registry` at `:137`), whose local-only behaviour stands until the cross-worker path is wired.+Cross-worker case verified without spend: parent + child + activity usage aggregates into one end-of-run cost report — `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py`, `test_mock_inference_temporal.py`, and the `temporal-e2e-validate` skill's Tier 8b. The full phased as-built is in [the registry tracker](../registry/README.md).---@@ -113,4 +114,4 @@ P2 (Local cross-package deps)P3 (Remote deps from GitHub)```-P0/P0.1/P0.2/P1 and P2/P3 are independent tracks; P0 unlocks P0.1, P0.2, and P1; P2 blocks P3. P0.1 is a parallel testing affordance that raises confidence in P0 / P1 / future cross-process work.+P0/P0.1/P0.2/P1 and P2/P3 are independent tracks; P0 unlocked P0.1, P0.2, and P1; P2 blocks P3. **P0, P0.1, and P1 are shipped**; P0.2 (independent follow-ons) and P2/P3 remain. P0.1 (`--mock-inference`) is a testing affordance that raised confidence in P0 / P1 and unlocks future cross-process work.diff --git a/wip/distributed-execution/tracing-cost-reporting.md b/wip/distributed-execution/tracing-cost-reporting.mdindex cfe24c13b..c79b9de0d 100644--- a/wip/distributed-execution/tracing-cost-reporting.md+++ b/wip/distributed-execution/tracing-cost-reporting.md@@ -1,8 +1,8 @@# Tracing & Cost Reporting — As Built-> **Status**: Shipped — the current tracing & cost-reporting implementation (supersedes an earlier distributed-tracing design, now retired).-> **Last updated**: 2026-05-04-> **Related**: [the distributed-execution plan](README.md).+> **Status**: Shipped — the current tracing & cost-reporting implementation (supersedes an earlier distributed-tracing design, now retired). T1 **and** T2 are now fixed; cross-worker cost reporting works end to end.+> **Last updated**: 2026-06-07+> **Related**: [the distributed-execution plan](README.md), [the registry overview](../registry/cost-reporting-overview.html), and [the registry feature](../registry/README.md).This file documents what actually shipped, plus the known gaps that the plan addresses.@@ -21,7 +21,7 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set- **Workflow → activity flush, not interceptor.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) wires a `BufferingEventLog` into the per-workflow `GraphTracerManager` and into the `ReportingManager` via `set_event_log(context_key=workflow_id, ...)`. After pipe execution, buffered events are drained and persisted by `act_flush_trace_events` (`pipelex/temporal/tprl_pipe/act_flush_trace_events.py`), which runs the synchronous boto3 / file writes off the workflow thread.- **Per-context keying** — `ReportingManager` holds a dict of `_event_log_contexts` keyed by `graph_context.lookup_key`, so concurrent workflows don't trample each other.- **Direct mode unchanged** — `pipeline_run_setup.py:282` calls the same `set_event_log` path, so the same `EventLogProtocol` machinery runs in-process for non-Temporal execution.-- **Graph assembly across workers** — `pipelex/pipe_run/graph_assembly.py` (`assemble_graph_on_output`) and `pipelex/temporal/tprl_pipe/act_assemble_graph.py` read all events for a `pipeline_run_id` from the configured backend and feed them into `GraphSpecAssembler` to build the cross-worker `GraphSpec`.+- **Tracing assembly across workers (graph + cost)** — `pipelex/pipe_run/tracing_assembly.py` (`assemble_tracing` / `assemble_tracing_on_output`) and `pipelex/temporal/tprl_pipe/act_assemble_tracing.py` read all events for a `pipeline_run_id` from the configured backend **once** and feed them into both `GraphSpecAssembler` (when `--graph`) and `UsageAggregator` (when `--costs`), setting `PipeOutput.graph_spec` and `PipeOutput.tokens_usages` respectively. (Renamed from `graph_assembly.py` / `act_assemble_graph` when T2 landed — see below.)## What works today@@ -30,7 +30,8 @@ The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set| Direct mode (single process) | ✅ | `pipeline_run_setup` configures event log on `ReportingManager`; events emitted in-process. || Single-worker Temporal (workflows + activities on same Worker bundle) | ✅ | Activities share the workflow's process; usage events emit through the singleton `ReportingManager` whose `set_event_log` was configured by `WfPipeRouter`. || Parent + child workflows on different worker processes | ✅ | Each workflow's process buffers its own events and flushes its own slice. All slices land in the same NDJSON dir / DynamoDB partition keyed by `pipeline_run_id`. |-| Cross-worker **graph** assembly | ✅ | `assemble_graph_on_output` / `act_assemble_graph` reads all events for the `pipeline_run_id` and runs `GraphSpecAssembler`. |+| Cross-worker **graph** assembly | ✅ | `assemble_tracing_on_output` / `act_assemble_tracing` reads all events for the `pipeline_run_id` and runs `GraphSpecAssembler`. |+| Cross-worker **cost** assembly + report | ✅ | Same single event read feeds `UsageAggregator` → `PipeOutput.tokens_usages`; the submitter renders one cost report from it. (T2 — fixed.) |## What is broken or missing@@ -45,25 +46,24 @@ The fix has two independent parts that ship together:- **Runner-side fallback in `_emit_usage_event`.** When the activity's `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`, it now falls back to a process-local event log built from `tracing_config`. The event log is cached per process via `pipelex.tracing.activity_event_log.get_or_create_activity_event_log` (guarded by a module-level `threading.Lock` so concurrent first-emitters from multiple activity threads agree on a single writer_id and instance). The fallback uses `graph_context.tracer_key or graph_context.graph_id` as `workflow_id` (no new `JobMetadata` field needed — `tracer_key` is already populated by `WfPipeRouter.run`). Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught at WARNING — never silently dropped, never `except Exception`. A one-shot warning per process records that the fallback engaged.- **Writer-id disambiguation in `TraceEvent`.** A new `writer_id: str = "primary"` field makes `(workflow_id, sequence)` collisions impossible across writers. NDJSON file naming becomes `wf_{workflow_id}__w_{writer_id}.ndjson` for non-primary writers; the `(pipeline_run_id, workflow_id, writer_id)` cache key prevents two writers from sharing a stale file handle. Read-side dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence primary so a runner-side `UsageReportEvent` does not sort before earlier router events. DynamoDB SK becomes `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.-The `_get_registry` orphan-accumulation TODO at `reporting_manager.py` is gone: the method was split into `_get_registry_strict` (used by `_report_*_job` — runner processes silently skip the registry add) and `_get_or_create_registry` (used by `inject_tokens_usages`, the console cost-report path, and `generate_report`).+The `_get_registry` orphan-accumulation TODO at `reporting_manager.py` is moot: T2 removed the submitter-side `UsageRegistry` entirely (see below), so there is no longer any per-run registry to accumulate or leak. The four `_report_*_job` methods now only emit a `UsageReportEvent` onto the event log; nothing buffers usage in the manager.The known retry-related over-counting case (R2) is documented and pinned by `test_retried_activity_emits_duplicate_usage_event_documenting_r2`. A `tracing_config.strict_mode` flag (raise instead of WARNING + drop) is deferred — see [the plan, §P0.2](README.md).-### Issue T2 — Cross-worker cost report assembly is not wired+### Issue T2 — Cross-worker cost report assembly — **FIXED (P1)**-**Status**: Still open. P0 (T1 above) ensures the events now land in the same backend partition from every worker; P1 is the remaining plumbing change to read them back into a cost report.+**Status**: Resolved. Usage now rides on `PipeOutput` exactly like the graph spec, and the submitter renders the cost report from that field — so a cross-worker run produces a single, complete report.-**Severity**: High — events are persisted but never turned into a report.+> **Design note (as resolved):** T2/P1 was the same lifecycle as the `UsageRegistry` success-path leak — the leak was the missing *close*, T2 was the missing *replay-populate*, of one per-run cost-aggregation buffer. **Option B (locked) resolved both by removal:** usage is assembled from the trace-event stream at the end of the run and rides back on `PipeOutput.tokens_usages`; with nothing populating a submitter-side registry, the registry was deleted and the leak became structurally impossible. The unified model and the locked decision are explained in [the registry overview](../registry/cost-reporting-overview.html); the feature index is [`../registry/README.md`](../registry/README.md).-The pieces all exist:+**As built:**-- `UsageAggregator.aggregate(events) → list[AnyTokensUsage]` (`pipelex/tracing/usage_aggregator.py`).-- `ReportingManager.inject_tokens_usages(pipeline_run_id, tokens_usages)` whose docstring literally says: *"Used after assembling usage data from distributed trace events, so that generate_report() can produce a complete cost report across all workers."* (`reporting_manager.py:223`).-- `ReportingManager.generate_report(pipeline_run_id)` (`reporting_manager.py:365`).+- The single event read in `assemble_tracing` / `act_assemble_tracing` (see "What shipped" above) now feeds `UsageAggregator.aggregate(events) → list[AnyTokensUsage]` (`pipelex/tracing/usage_aggregator.py`) when `--costs` is on, setting `PipeOutput.tokens_usages` in both direct and Temporal modes.+- The submitter renders it via `CostRegistry.generate_report(tokens_usages=...)` — fed from `pipe_output.tokens_usages`, never from a process-local registry. The main CLI path goes through `pipelex/reporting/cost_report_renderer.py::render_run_cost_report`.+- The decoupling that made this safe: a dedicated `--costs` / `is_generate_costs` switch (default on) gates usage events + `UsageAggregator` over the shared event-log transport, independent of `--graph`. Previously usage events only emitted when `--graph` was on.+- The retired surface: `ReportingManager.inject_tokens_usages` / `generate_report` / `open_registry` / `close_registry` and the `UsageRegistry` model are **gone** — there is no submitter-side registry left to populate.-But **the cross-worker assembly chain is not wired up**. The graph assembly path (`graph_assembly.py`, `act_assemble_graph.py`) reads events back, but only feeds them into `GraphSpecAssembler` — never into `UsageAggregator`. `generate_report()` does have a runtime caller (the CLI run path, `_run_core.py:224`), but it reports only the local, in-process usage registries; `UsageAggregator.aggregate()` and `inject_tokens_usages()` still have zero runtime callers, so the events emitted by runner/worker processes never get assembled into the cost report.--Net effect: even when usage events ARE captured (i.e., the same-worker case in the table above), the captured events sit in NDJSON / DynamoDB and never become a cross-worker cost report.+Net effect: usage events emitted by runner/worker processes are now aggregated into one cross-worker cost report. Validated without spend by `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py` (read-back aggregation), `test_mock_inference_temporal.py` (`--mock-inference` cross-process), and the direct-mode `test_direct_tracing_assembly.py` / `test_cost_report_rendering.py`.### Issue T3 — `set_event_log` is on a process-singleton `ReportingManager`@@ -77,9 +77,9 @@ Net effect: even when usage events ARE captured (i.e., the same-worker case in t## How this connects to the plan-- T1 was the **top priority** in the [distributed-execution plan](README.md) — now resolved.-- T2 is the **next priority** — events now land in the same partition from every worker; the small remaining step is wiring `read events → UsageAggregator → inject_tokens_usages → generate_report`.-- T3 is the architectural shape; the T1 fix mitigates the immediate impact, the deeper refactor will land alongside T2 / further follow-ups.+- T1 was the **top priority** in the [distributed-execution plan](README.md) — resolved.+- T2 is **resolved** — usage rides on `PipeOutput` (assembled from the same event read as the graph spec) and the submitter renders one cross-worker report. The wiring chosen was *not* `read events → inject_tokens_usages → generate_report` but the leaner Option B: `read events → UsageAggregator → PipeOutput.tokens_usages → CostRegistry.generate_report(tokens_usages=...)`, which let the submitter-side registry be deleted outright.+- T3 is the architectural shape; the T1 fix mitigated the immediate impact and T2's removal of the submitter-side registry shrank the singleton's surface further. The deeper "request-scoped tracing state instead of process-singleton" refactor remains the open follow-up (see Deferred / [the registry tracker](../registry/README.md)).---diff --git a/wip/dry-run-refactor/followup-leaf-run-mode-mock.md b/wip/dry-run-refactor/followup-leaf-run-mode-mock.mdindex 17b9c366f..d00eb2988 100644--- a/wip/dry-run-refactor/followup-leaf-run-mode-mock.md+++ b/wip/dry-run-refactor/followup-leaf-run-mode-mock.md@@ -5,6 +5,8 @@> **Design rationale:** [`D-plan.md`](./D-plan.md) §3.5 (run mode ⟂ backend) and §4.8 (leaf-level mock). **Risks:** D-plan §8 (object-mock fidelity, req-1 fidelity regressions).>> **Depends on:** the consolidation ([`consolidation-as-built.md`](./consolidation-as-built.md)) is *not* a hard prerequisite (this is a separable cogt/operator refactor), but the Temporal-validation follow-up builds on **both**, so sequence this after the consolidation lands. **Branch off the same D-plan.**+>+> **⚠️ Coordinate with the registry branch's Phase 5 (`fix/For-API-update`, decided 2026-06-06).** That branch ships `--mock-inference` / `is_mock_inference` as an *interim* trigger, deliberately built **leaf-first so it lands B1's core ahead of time**: a shared `cogt/content_generation/dry_mock.py` + a per-leaf dry branch keyed on a per-run flag carried on `JobMetadata`. So when this follow-up runs, **B1 collapses to "re-key that helper from `is_mock_inference` → `run_mode`"** (carrier + helper already exist — verify before building from scratch), and **B2 settles the fate of `is_mock_inference`** (retire vs keep a thin reportable-mock — see B2). One distinction to preserve when re-keying: `--mock-inference` emits *non-zero* synthetic usage so a cost report renders; `--dry-run` stays zero-token and its report is *suppressed*. See the registry feature's [`../registry/deferred-followups.md`](../registry/deferred-followups.md) (`--mock-inference` coverage + the `is_mock_inference` fate decision).Goal: move the LIVE/DRY decision **down to the cogt leaf** so DRY honors the configured backend — DRY-on-Temporal dispatches `act_llm_gen_*` and mocks **inside** the activity, retiring the "DRY → local in-process" shortcut. Separable cogt/operator refactor (§4.8). **Resolve the Pre-flight items before starting.**@@ -31,6 +33,10 @@ Resolve **before** the phase that depends on it. Record the answer in the releva## Phase B1 — Thread `run_mode` to the leaf + add the leaf DRY branch (§4.8)+> **B1's core LANDED on the registry branch's Phase 5 (`fix/For-API-update`).** Already built: `JobMetadata.is_mock_inference`, the shared `cogt/content_generation/dry_mock.py` (synthetic-job reporting parameterized zero-vs-non-zero, plus the leaf mocks + `build_mock_object`), and the per-leaf branch in `llm_generate.py` keyed on `is_mock_inference`. So B1 reduces to: (1) carry `run_mode` on `JobMetadata` alongside `is_mock_inference` (same single-writer, `prepare_pipe_job`), and (2) re-key the leaf branch from `is_mock_inference` → `run_mode==DRY`. **Don't rebuild the helper.** Two caveats to finish what Phase 5 deliberately scoped out: it covers the **LLM leaf only** — `img_gen_*` / `extract_gen_pages` / `templating_gen_text` leaf branches are still TODO here (Phase 5 deferred them because their output is stored above the leaf; see that branch's §7); and `ContentGeneratorDry` still owns its method bodies (only its *reporting* delegates to `dry_mock`), so B2's collapse is still real work.++> **Interim hard guard already landed (F1, registry branch).** `img_gen_single_image` / `img_gen_image_list` / `extract_gen_pages`, plus `PipeSearch._live_run_operator_pipe` (web search has no `content_generation` leaf — it spends in the operator's live path), now **raise `MockInferenceUnsupportedError`** under `is_mock_inference` instead of silently calling the real provider. So when B1 adds the real synthetic-output leaf branch for img-gen/extract here, it is **replacing a fail-loud guard, not patching a silent-spend path** — delete the guard at that leaf as you add the DRY branch. Search has no leaf to thread `run_mode` into; decide whether it gains a real mock or keeps the guard when B1/B2 settle the operator dry paths. See `wip/registry/deferred-followups.md` (`--mock-inference` coverage).+- [ ] Confirm the funnel by grep: `ContentGenerator.make_llm_text` and `ContentGeneratorInWorkflow.make_llm_text` both build `LLMAssignment` and converge on `llm_gen_text(assignment)` (inline vs via `act_llm_gen_text`).- [ ] Carry `run_mode` on `JobMetadata` (or the confirmed carrier); single-writer from `prepare_pipe_job` / `PipeRunParams`.- [ ] *Tests first:* per-leaf DRY-branch unit tests; the object-mock-from-schema fidelity test on a representative `StructuredContent`.@@ -39,8 +45,9 @@ Resolve **before** the phase that depends on it. Record the answer in the releva## Phase B2 — Collapse the operator dry path (§4.8)-- [ ] Remove the pipe-level `ContentGeneratorDry()` swap from each operator's `_dry_run_pipe` (`PipeLLM`, `PipeImgGen`, `PipeExtract`/`PipeOcr`, templating operators — grep for `ContentGeneratorDry()` and `_dry_run_pipe`). Route DRY through the hub content generator with `run_mode` threaded.-- [ ] Re-express the boot `not needs_inference` fallback as force-`run_mode=DRY`; apply the confirmed `ContentGeneratorDry` disposition.+- [ ] Remove the pipe-level `ContentGeneratorDry()` swap from each operator's `_dry_run_operator_pipe` (`PipeLLM`, `PipeCompose`, `PipeImgGen`, `PipeExtract`, `PipeStructure` — grep for `ContentGeneratorDry()`). Route DRY through the hub content generator with `run_mode` threaded.+- [ ] Re-express the boot `not needs_inference` fallback (`pipelex.py:368`) as force-`run_mode=DRY`; apply the confirmed `ContentGeneratorDry` disposition.+- [ ] **Settle `is_mock_inference` / `--mock-inference`** (the registry branch's interim trigger). Now that `run_mode=DRY` honors the backend, **either** (a) *fully retire* the flag, its `JobMetadata` field, its CLI option, and the leaf branch's `is_mock_inference` arm — `run_mode=DRY` becomes the sole trigger — and re-key the registry branch's `temporal-e2e-validate` Tier 8b to assert on assembled `tokens_usages` (plain DRY emits zero-token usage, which Phase 3 *suppresses*, so the rendered table can't be asserted under DRY); **or** (b) *keep a thin reportable-mock* (the non-zero-synthetic-usage arm) precisely so Tier 8b can still validate the rendered cost report cheaply. Pick based on whether cheap rendered-report validation is worth one surviving flag. (Cross-ref: registry feature's [`../registry/deferred-followups.md`](../registry/deferred-followups.md).)- [ ] `PipeSignature._dry_run_pipe` and controller dry behavior unchanged (no leaf op).- [ ] Verify **direct + DRY** outcomes unchanged: full `make agent-test` green + spot-check a dry `validate --all` and a dry single-pipe run.diff --git a/wip/init-yes-mode-handoff.md b/wip/init-yes-mode-handoff.mdnew file mode 100644index 000000000..747ca4a79--- /dev/null+++ b/wip/init-yes-mode-handoff.md@@ -0,0 +1,89 @@+# Handoff: non-interactive `pipelex init --yes` mode++Untracked scratch note (in `wip/`, not gitignored — delete when done). Written from a cocode session that hit the downstream symptom. pipelex version in play: 0.31.0.++## Why we want this++Downstream repos need CI to initialize Pipelex **without** interactive prompts and **without** vendoring a full `.pipelex/` config tree into their repo. Today that's impossible: `pipelex init` is interactive end-to-end.++The concrete trigger: cocode CI started failing with `Pipelex must be initialized before running the tests` after cocode de-vendored its `.pipelex/` config (cocode commit `1dd8831`). cocode's `tests/conftest.py` gates the session on Pipelex's `check_is_initialized()`. On CI there's no global `~/.pipelex/` and (after de-vendoring) no project `.pipelex/`, so the gate aborts the run. Locally it passed only because the dev had a global config.++Important: `--disable-inference` does **not** help here. It skips the *gateway terms* check and swaps in a mock content generator — it never bypassed *initialization* (the presence of config files). Pipelex's own CI sidesteps the problem only because pipelex **vendors its full `.pipelex/` tree in-repo**. We don't want every downstream repo to carry (and keep in sync) a vendored config just to run CI.++**Stopgap already applied in cocode** (to be reverted once this lands): re-vendored a minimal `.pipelex/` CI config (pipelex.toml, plxt.toml, `inference/` backends+deck+routing), force-added past the gitignore. See the "Downstream follow-up" section.++## Goal++Add a non-interactive `pipelex init --yes` (`-y`) that materializes a complete, valid config with zero prompts and exit 0 — such that `check_is_initialized()` returns True — suitable for CI and automation.++## What already exists (good news — it's mostly plumbing)++- `init` CLI command: `pipelex/cli/_cli.py:158` (`init_command`) → calls `init_cmd(focus=focus, local=local)`. No `--yes` exposed yet.+- `init_cmd`: `pipelex/cli/commands/init/command.py:466`. Already takes `skip_confirmation: bool` — but that only gates the **final** "Continue with initialization?" confirm (`command.py:545-563`). It does **not** gate the backend/routing/credentials prompts.+- Pattern to mirror: `update_command` already has `--yes/-y` (`_cli.py:209`) threaded into `update_cmd(yes=...)`. Copy that shape.+- `init_cmd(focus=InitFocus.CONFIG, skip_confirmation=True)` is already called non-interactively from doctor (`doctor_cmd.py:1028,1048`).++**Key insight:** every interactive prompt site already carries a sensible `default=`. So `--yes` is well-bounded: short-circuit each prompt to its existing default instead of asking. The defaults already encode the "yes" answer.++## Prompt sites to short-circuit under `--yes`++Each must return its `default` (or computed default) without prompting:++- `command.py:249` — `Confirm.ask("Continue with initialization?", default=True)`. Verify it isn't reached when `assume_yes` (today it's the `skip_confirmation` path).+- `ui/backends_ui.py:188` — `Prompt.ask("Enter your choices", default=default_str)`. First-time default is `"1"` = `pipelex_gateway` (`backends_ui.py:177-184`). Under `--yes`: accept `default_str` (and the computed `default_indices`) directly.+- `ui/routing_ui.py:63` and `:150` — `Prompt.ask(..., default=...)`. Accept defaults.+- `routing.py:74` — `Confirm.ask("Would you like to create it?", default=True)`. Accept True.+- `credentials.py:140` — `Prompt.ask(var, default="", password=True)`. Under `--yes`: **skip the prompt loop entirely** (default `""` already means "skip, set later" — the UI even says so at `credentials.py:128`). Creds come from env in CI.+- `ide_extension.py:109` — `Confirm.ask(... install IDE extension ...)`. Under `--yes`: default to **No** (don't install in CI). Confirm its current default.+- `ui/gateway_ui.py:107` — `Confirm.ask(...)` gateway terms. **Legal decision — do not silently accept ToS.** See below.++## Recommended implementation++1. Add `--yes/-y` typer option on `init_command` (`_cli.py:158`), mirroring `update_command:209`. Pass it as `assume_yes` into `init_cmd`.+2. Thread `assume_yes` into `init_cmd` and down to the UI helpers (`backends_ui`, `routing_ui`, `routing`, `credentials`, `ide_extension`, `gateway_ui`). Centralize with a tiny helper, e.g. `ask_or_default(assume_yes, ask_fn, default)`, so each site reads cleanly.+3. Reconcile with `skip_confirmation`: cleanest is to introduce `assume_yes` and have it **subsume** `skip_confirmation` (assume_yes ⇒ skip the final confirm too). Update the doctor callsites (`doctor_cmd.py:1028,1048`) accordingly.+4. Credentials under `--yes`: skip the loop; don't write blank entries to `.env`.+5. **High-value optional:** auto-enable `assume_yes` when non-interactive — `not sys.stdin.isatty()` or a CI env var (`GITHUB_ACTIONS`/`CI`, already detected via `runtime_manager`/`shared_pytest_plugins.py`). Then CI "just works" even without the flag, and a closed stdin never hangs.++## Gateway terms — handle deliberately++Do **not** auto-accept the Pipelex Gateway ToS just because `--yes` was passed. Options, pick one:+- In CI mode (`runtime_manager.is_ci_testing` / `IntegrationMode.CI`) terms are already skipped for the *test run*; for `init --yes` itself, leave the gateway **enabled but unaccepted** and let the CI test path (which uses mock inference) proceed — it never calls the gateway.+- Or require a separate explicit opt-in for acceptance: `--accept-terms` flag or `PIPELEX_ACCEPT_TERMS=1` env. `--yes` alone should not constitute legal acceptance.++## Resulting config must satisfy `check_is_initialized()`++`pipelex/system/configuration/config_check.py:12-18` requires all of: `pipelex.toml`, `plxt.toml`, the backends file (`config_manager.backends_file_path`), and the routing-profiles file (`config_manager.routing_profiles_file_path`) — resolvable in project or global dir. `--yes` must produce all of them. Default backend = `pipelex_gateway`; default routing profile; telemetry default (decide: accept default vs. disable in non-interactive — check the telemetry default).++## Acceptance criteria++In an environment with **no** global `~/.pipelex/` and **no** project `.pipelex/`:+- `pipelex init --yes --local` → exit 0, **zero prompts**, creates `.pipelex/` such that `python -c "from pipelex.system.configuration.config_check import check_is_initialized; assert check_is_initialized()"` passes.+- `pipelex init --yes` (global) → same for `~/.pipelex/`.+- Runs with stdin closed / non-tty must not hang.+- Gateway ToS is **not** silently accepted (assert).+- The existing interactive `pipelex init` flow is unchanged.++## Tests / conformance++- Unit-test `init_cmd(assume_yes=True)` in a tmp config dir (monkeypatch the config dirs). Patch `Confirm.ask`/`Prompt.ask` to **raise** if called, proving no prompts fire; assert the files exist and `check_is_initialized()` is True.+- The init CLI is a cross-repo surface — there's likely a spec in the workspace `docs/specs/` with a paired test in the `conformance/` repo (see workspace CLAUDE.md). Add the `--yes` flag to that spec section, add a conformance test, and run `conformance/scripts/check-spec-links.py`.++## Downstream follow-up (don't lose this)++Once `init --yes` ships and pipelex is re-released + re-pinned:+- **cocode:** revert the stopgap — remove the re-vendored `.pipelex/` config (currently staged on branch `release/v0.7.0`), and instead either add a CI step `pipelex init --yes --local` before `make gha-tests` in `.github/workflows/tests-check.yml`, **or** rely on auto-init-when-non-tty/CI if you implement option 5 (then cocode's conftest can call an init helper rather than just gating).+- Audit other repos that vendor `.pipelex/` purely for CI (e.g. `methods/`, `test-bed/`, `pipelex-api-deploy/`) — they can adopt the same `init --yes` pattern.++## Files likely to touch++- `pipelex/cli/_cli.py` — `init_command`: add `--yes/-y`, pass through.+- `pipelex/cli/commands/init/command.py` — `init_cmd`: accept `assume_yes`, thread it, reconcile `skip_confirmation`.+- `pipelex/cli/commands/init/ui/backends_ui.py` — short-circuit to default selection.+- `pipelex/cli/commands/init/ui/routing_ui.py` — short-circuit to defaults.+- `pipelex/cli/commands/init/routing.py` — `Confirm` default.+- `pipelex/cli/commands/init/credentials.py` — skip the loop under `assume_yes`.+- `pipelex/cli/commands/init/ide_extension.py` — default No under `assume_yes`.+- `pipelex/cli/commands/init/ui/gateway_ui.py` — terms handling (deliberate).+- `pipelex/cli/commands/doctor_cmd.py` — update callsites if `skip_confirmation` is folded into `assume_yes`.+- Tests + `docs/specs/` + `conformance/`.diff --git a/wip/registry/README.md b/wip/registry/README.mdnew file mode 100644index 000000000..499ff8ae3--- /dev/null+++ b/wip/registry/README.md@@ -0,0 +1,24 @@+# `UsageRegistry` leak fix + distributed cost reporting++Cost reporting was reworked from a leaky per-run, in-process `UsageRegistry` buffer into an **event-sourced artifact that rides back on `PipeOutput`**. The success-path registry leak is gone by removal; a multi-worker (Temporal) run now aggregates usage from every worker into a single end-of-run cost report. A default-on `--costs` switch decouples cost collection from `--graph` over a shared event-log transport, and `--mock-inference` validates the distributed path cheaply.++Shipped in **PR #967** on `fix/For-API-update`, with the companion rename **#968** (`GraphContext → TraceContext`, `is_generate_costs → is_generate_usage`).++> **Scope — NDJSON only.** The cost-reporting event log is implemented and validated through **NDJSON**. The end-of-run read-back is backend-agnostic, so the code path covers DynamoDB for free — but making distributed cost reporting actually work on DynamoDB is **deferred** and is not a validation target of this work.++## What we built++- [`cost-reporting-overview.html`](cost-reporting-overview.html) — **the illustrated overview. Start here.** TL;DR, the two-bugs-one-lifecycle framing, before/after architecture, the shared-transport `--costs` switch, the cross-worker flow, the critical diffs, and the deferred items.++## Open follow-ups++- [`deferred-followups.md`](deferred-followups.md) — the deliberate non-goals: the costs-only in-memory tracer skip (E1 deep half), cost-per-node correlation, the A1 `TraceContext.from_execution_config` factory, T5, T3 request-scoped tracing state, `--mock-inference` coverage, and the leftover `graph_id → run_id` rename + shelved `TraceContext` split.+- [`cost-report-deferred-decisions.md`](cost-report-deferred-decisions.md) — two deferred **design decisions**: (#3) cost reporting is coupled to `tracing_config.is_enabled` (costs-on + tracing-off ⇒ no report); (#6) the agent CLI gates costs on the raw `costs` param while the main CLI gates on the resolved `is_generate_usage`. Both want a deliberate call.+- [`usage-reporting-without-cost.md`](usage-reporting-without-cost.md) — capability idea (low priority): make usage (token counts) a separately-requestable output, with cost as one view layered on top.++## Companion track++The cost-reporting half mirrors graph-spec assembly, documented in the distributed-execution feature:++- [`../distributed-execution/tracing-cost-reporting.md`](../distributed-execution/tracing-cost-reporting.md) — as-built tracing & cost reporting (T2 cross-worker cost-report assembly = **FIXED** by this work).+- [`../distributed-execution/README.md`](../distributed-execution/README.md) — the priority plan (P0.1 `--mock-inference` and P1 cost-report assembly both **shipped** here).diff --git a/wip/registry/cost-report-deferred-decisions.md b/wip/registry/cost-report-deferred-decisions.mdnew file mode 100644index 000000000..9f9b8b477--- /dev/null+++ b/wip/registry/cost-report-deferred-decisions.md@@ -0,0 +1,64 @@+# Cost reporting — deferred design decisions (Phase 3 review)++Two findings from the Phase 3 code review (event-sourced cost-report rendering) are **design decisions, not bugs to reflexively patch**. They are deferred here so they are not lost. The review's correctness/cleanup findings (#1 free-model suppression, #2 try-scope, #4/#7 aggregation consolidation) were fixed in the Phase 3 change itself; these two need a deliberate call.++> **Naming (post-#968):** the internal gate was renamed `is_generate_costs` → `is_generate_usage` (config field on `PipelineExecutionConfig`) and the override param `generate_costs` → `generate_usage` (`with_execution_overrides`); the symbol names below are current. Unchanged: the public `--costs` / `--no-costs` flag and the renderer's `render_run_cost_report(is_generate_costs=…)` param (the cost domain keeps its cost name).++Both touch the same question the registry track is converging on: *what is the single, intentional trigger for "collect and report costs"?* Right now that intent is spread across three switches — `--costs` / `is_generate_usage`, `tracing_config.is_enabled`, and the agent CLI's own `costs` param default — and they don't fully agree.++---++## #3 — Cost reporting is coupled to `tracing_config.is_enabled`++**What.** After the Phase 3 cutover, the *only* cost renderer reads `pipe_output.tokens_usages`, which is assembled from the trace-event stream. That stream only exists when tracing is enabled. So with `[pipelex.tracing_config] is_enabled = false` but `--costs` on (the default), **no cost report is produced at all** — silently.++**Evidence.**++- `pipelex/pipeline/pipeline_run_setup.py:193` — the event log is created only `if tracing_config.is_enabled:`; otherwise `event_log` stays `None`.+- `pipelex/pipeline/pipeline_run_setup.py:231` — usage events are emitted only `if is_generate_usage and event_log is not None:` → no event log ⇒ no usage events.+- `pipelex/pipe_run/tracing_assembly.py:80` — `assemble_tracing` early-returns `if not tracing_config.is_enabled:`, leaving `tokens_usages = None`.+- `pipelex/reporting/cost_report_renderer.py` — `render_run_cost_report` no-ops on `not tokens_usages`.++**Why it's a regression.** Before Phase 3, the cost report was rendered from the live `UsageRegistry`, which `_report_*_job` populated directly — independent of tracing. So `pipelex run --costs` printed a table even with tracing disabled. Now it doesn't. Default config has `is_enabled = true`, so the default experience is fine; the gap is for embedders who turn tracing off (e.g. to avoid NDJSON files) and reasonably expect `--costs` to keep working.++**The design question.** Is `--costs` *meant* to require the event-log transport?++- **Option A — accept the coupling, make it honest.** Treat tracing as the cost transport (which is the Phase 3/4 architecture). Then: when `is_generate_usage` is on but `tracing_config.is_enabled` is off, emit a one-time `log.warning` ("costs requested but tracing is disabled; no cost report will be produced — enable tracing or disable --costs"), and document the dependency in `reporting-config.md` / the `--costs` help text. Cheapest, keeps one transport.+- **Option B — decouple at setup.** When `is_generate_usage` is on, force the event log into existence even if `tracing_config.is_enabled` is off (a costs-only, in-memory event log that never writes NDJSON), and let `assemble_tracing` run for the usage concern regardless of `is_enabled`. Restores the pre-Phase-3 "costs work without tracing" guarantee at the cost of a second reason the event log can exist.+- **Option C — separate enable flags.** Split "emit trace events to a backend (NDJSON/DDB)" from "assemble in-memory usage for reporting." `is_enabled` governs the former; `is_generate_usage` governs the latter. Cleanest conceptually, most config surface.++**Lean:** A for now (it matches the locked Option-B-events-as-single-source direction and is a few lines), with C as the principled end state if embedders hit it. Decide before Phase 4 removes the registry — after that there is no fallback renderer, so the coupling becomes the only behavior.++---++## #6 — Agent CLI gates costs on the raw `costs` param; main CLI gates on the resolved config++**What.** The two CLIs resolve "should we collect/report costs" differently:++- **Main CLI** (`pipelex run`): the renderer is gated on `execution_config.is_generate_usage` — the value *resolved* from the `--costs/--no-costs` tri-state over the config default. A user who sets `is_generate_usage = false` in config and passes no flag gets no costs.+- **Agent CLI** (`pipelex-agent run`): `costs` is a hard-coded `bool = True` parameter (see `run_pipeline_core` / the `pipe_cmd`/`bundle_cmd`/`method_cmd` defaults). It is forced into the config via `with_execution_overrides(generate_usage=costs)` and also gates `build_cost_summary` directly (`pipelex/cli/agent_cli/commands/run/_run_core.py:111`). It **never consults the config default** `is_generate_usage`.++**Consequence.** An embedder who disables cost collection globally (`[pipeline_execution_config] is_generate_usage = false`, e.g. to cut event-log overhead) gets it honored by `pipelex run` but **not** by `pipelex-agent run` — the agent keeps emitting usage events and attaching a `cost_report` to every `--with-memory` envelope / on-disk JSON unless `--no-costs` is passed explicitly.++**History.** This predates Phase 3 (the agent `--costs` default-True override landed in Phase 1). Phase 3 only made it *observable*, because it surfaces a `cost_report` object in the agent output where before nothing was shown.++**The design question.** Should the agent CLI honor the config `is_generate_usage` default like the main CLI, or is "agent always collects unless told not to" intentional?++- **Option A — make the agent tri-state like the main CLI.** Change the agent `costs` param to `bool | None = None` and resolve it through `with_execution_overrides` (None ⇒ use config default). One-line-ish per command + the `_run_core` gate reads the resolved `execution_config.is_generate_usage`. Makes the two surfaces consistent.+- **Option B — keep agent default-on by design, document it.** If the agent is meant to be maximally informative for machine consumers regardless of an embedder's human-CLI preference, leave it but document the divergence in `agent_cli/CLAUDE.md` so it's a choice, not an accident.++**Lean:** A — consistency across the two surfaces is worth more than the agent being unconditionally chatty, and `--no-costs` still lets a caller force it off. But it's a behavior change for agent consumers, so it needs a deliberate sign-off rather than being folded silently into the Phase 3 fix.++### #6b — Agent `--costs` is a silent no-op in API mode (same root)++**What.** The agent `run` subcommands declare `--costs/--no-costs` unconditionally, but the agent has two runner paths: `run_pipeline_core` (local) and `run_pipeline_core_api` (remote API runner). Only the local path builds an `execution_config` and renders/attaches a cost summary. In **API mode** the flag is accepted and then **silently ignored** — execution happens remotely and the local `costs` value does nothing. Flagged by cubic (P2, `pipe_cmd.py:51`) and Greptile's summary ("the agent API runner accepts cost flags but does not apply or report them", `_run_core_api.py`) on PR #967.++**Why it's a design decision, not a reflexive patch.** "Rejecting `--costs` as unsupported in API mode" needs to distinguish an *explicitly-passed* flag from the `True` default — but the agent uses `bool = True`, so a user passing `--costs` is indistinguishable from the default (the exact same ambiguity #6 Option A would resolve by going tri-state `bool | None`). And whether API-mode cost reporting should instead surface costs *from the API response* is a product call, not a lint fix. So #6b is a strict consequence of #6: making the agent `costs` tri-state (#6 Option A) is the prerequisite for any honest API-mode handling (warn / reject / surface-from-response). Resolve them together; do not bolt a one-off rejection onto the `True`-default param.++**Broader framing (found at PR #967 final review): API-mode flag handling is inconsistent across the board.** The `RunnerType.API` branch in the agent `run` subcommands *rejects* `--dry-run` and `--mock-inputs` (`agent_error("... not supported with --runner api")`) but *silently ignores* `--graph` AND `--costs` (neither is threaded into `run_pipeline_core_api`). So a one-off `--no-costs` rejection would deepen the inconsistency (why reject `--no-costs` but not `--graph`?). The right fix is one holistic decision about how the agent CLI handles local-runner-only flags under `--runner api` — reject them all uniformly, or document them all as no-ops — covering `--graph` and `--costs` together. That is the deliberate call #6/#6b is waiting on; a partial patch on just `--costs` is explicitly the wrong move.++---++## Cross-cutting note++Both findings are facets of the same gap: there is no *single owner* of "is cost collection/reporting on for this run?" The value is currently derived in three places (config default, the two CLIs' flag resolution, and the tracing toggle) and they disagree at the edges. Whoever takes the Phase 4 registry removal should consider collapsing this into one resolved predicate on `PipelineExecutionConfig` that both CLIs and the assembly path read — at which point #3 and #6 both fall out as consequences of one decision.diff --git a/wip/registry/cost-reporting-overview.html b/wip/registry/cost-reporting-overview.htmlnew file mode 100644index 000000000..0bc46ecda--- /dev/null+++ b/wip/registry/cost-reporting-overview.html@@ -0,0 +1,498 @@+<!DOCTYPE html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1.0">+<title>Cost Reporting Rework — UsageRegistry leak fix + distributed cost reporting</title>+<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>+<style>+ :root{+ --bg:#0f1220; --panel:#171a2b; --panel-2:#1d2136; --ink:#e7e9f3; --muted:#9aa0bd;+ --line:#2a2f4a; --accent:#6c7bff; --accent-2:#3ad1c4; --warn:#ffb454; --danger:#ff6b6b;+ --good:#56d364; --add-bg:#10301c; --add-ink:#9be9a8; --del-bg:#3a1620; --del-ink:#ff9aa6;+ --code:#c9d1ff;+ --maxw:1080px;+ }+ *{box-sizing:border-box}+ html{scroll-behavior:smooth}+ body{+ margin:0; background:linear-gradient(180deg,#0c0e1a,#0f1220 240px); color:var(--ink);+ font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;+ -webkit-font-smoothing:antialiased;+ }+ a{color:var(--accent-2);text-decoration:none}+ a:hover{text-decoration:underline}+ .wrap{max-width:var(--maxw);margin:0 auto;padding:0 22px 96px}+ header.hero{+ max-width:var(--maxw);margin:0 auto;padding:54px 22px 26px;+ }+ .eyebrow{letter-spacing:.16em;text-transform:uppercase;font-size:12px;color:var(--accent);font-weight:700}+ h1{font-size:34px;line-height:1.15;margin:.35em 0 .25em;letter-spacing:-.01em}+ h1 .sub{display:block;font-size:18px;color:var(--muted);font-weight:500;margin-top:.5em}+ .badges{display:flex;flex-wrap:wrap;gap:8px;margin-top:18px}+ .badge{font-size:12.5px;font-weight:600;padding:5px 11px;border-radius:999px;border:1px solid var(--line);background:var(--panel);color:var(--ink)}+ .badge.green{border-color:#1f5132;background:#10271a;color:var(--good)}+ .badge.blue{border-color:#28346b;background:#141a36;color:#aeb8ff}+ .badge.amber{border-color:#5a4420;background:#2a2012;color:var(--warn)}++ h2{font-size:24px;margin:54px 0 14px;padding-top:10px;letter-spacing:-.01em;border-top:1px solid var(--line)}+ h2 .n{color:var(--accent);font-variant-numeric:tabular-nums;margin-right:.5ch}+ h3{font-size:18px;margin:28px 0 10px}+ p{margin:.7em 0}+ ul,ol{margin:.6em 0;padding-left:1.3em}+ li{margin:.3em 0}+ code{font-family:"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;font-size:.88em;background:#0c1024;border:1px solid var(--line);border-radius:5px;padding:1px 5px;color:var(--code)}+ strong{color:#fff}+ .lead{font-size:17.5px;color:#cfd3ee}++ .card{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:20px 22px;margin:18px 0}+ .card.tldr{background:linear-gradient(160deg,#191d33,#15182a);border-color:#33407e}+ .tldr h2{border:0;margin:0 0 8px;padding:0;font-size:21px}+ .tldr ul{margin:.4em 0}+ .kpis{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin-top:16px}+ .kpi{background:var(--panel-2);border:1px solid var(--line);border-radius:10px;padding:13px 14px}+ .kpi b{display:block;font-size:21px;color:#fff;line-height:1.1}+ .kpi span{font-size:12.5px;color:var(--muted)}++ .callout{border-radius:12px;padding:16px 18px 16px 18px;margin:20px 0;border:1px solid;position:relative}+ .callout .tag{font-size:11.5px;font-weight:800;letter-spacing:.08em;text-transform:uppercase;display:inline-block;margin-bottom:6px}+ .callout.scope{background:linear-gradient(120deg,#2a2012,#241b10);border-color:#6a5223}+ .callout.scope .tag{color:var(--warn)}+ .callout.info{background:#10182f;border-color:#274073}+ .callout.info .tag{color:#7fb0ff}+ .callout.good{background:#0f2519;border-color:#1f5132}+ .callout.good .tag{color:var(--good)}+ .callout p:first-of-type{margin-top:0}+ .callout p:last-child{margin-bottom:0}++ .mermaid{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px;margin:18px 0;overflow-x:auto;text-align:center}++ /* tabs */+ .tabs{margin:18px 0}+ .tabs .tabbar{display:flex;gap:6px;flex-wrap:wrap;border-bottom:1px solid var(--line);margin-bottom:0}+ .tabs .tabbtn{appearance:none;border:1px solid var(--line);border-bottom:none;background:var(--panel);color:var(--muted);+ font:inherit;font-size:13.5px;font-weight:600;padding:8px 14px;border-radius:9px 9px 0 0;cursor:pointer;margin-bottom:-1px}+ .tabs .tabbtn[aria-selected="true"]{background:var(--panel-2);color:#fff;border-color:var(--accent)}+ .tabs .tabpane{display:none;background:var(--panel-2);border:1px solid var(--line);border-radius:0 10px 10px 10px;padding:4px 0 0}+ .tabs .tabpane.active{display:block}+ .tabpane .mermaid{border:0;background:transparent;margin:6px 0}++ /* code / diff */+ pre{margin:0;background:#0b0e1f;border:1px solid var(--line);border-radius:10px;overflow-x:auto}+ pre.code{padding:14px 16px}+ pre code{background:none;border:none;padding:0;color:#cdd3f5;font-size:13px;line-height:1.7;display:block}+ .filename{font-family:"SF Mono",monospace;font-size:12px;color:var(--muted);background:#0c1024;border:1px solid var(--line);border-bottom:none;border-radius:9px 9px 0 0;padding:7px 13px;display:inline-block;margin-bottom:-1px}+ .filename + pre{border-radius:0 10px 10px 10px}+ .diff{padding:12px 0;font-size:13px;line-height:1.65}+ .diff .ln{display:block;padding:0 16px;white-space:pre;font-family:"SF Mono",monospace;color:#aeb4d6}+ .diff .add{background:var(--add-bg);color:var(--add-ink)}+ .diff .del{background:var(--del-bg);color:var(--del-ink)}+ .diff .ctx{color:#8b91b4}+ .diff .cmt{color:#6f76a0;font-style:italic}++ details{background:var(--panel);border:1px solid var(--line);border-radius:11px;margin:14px 0;overflow:hidden}+ details[open]{background:var(--panel)}+ summary{cursor:pointer;padding:14px 18px;font-weight:600;list-style:none;display:flex;align-items:center;gap:10px}+ summary::-webkit-details-marker{display:none}+ summary::before{content:"▸";color:var(--accent);font-size:13px;transition:transform .15s}+ details[open] summary::before{transform:rotate(90deg)}+ summary:hover{background:var(--panel-2)}+ .details-body{padding:2px 18px 18px}++ table{border-collapse:collapse;width:100%;margin:16px 0;font-size:14.5px;overflow:hidden;border-radius:10px}+ th,td{border:1px solid var(--line);padding:9px 12px;text-align:left;vertical-align:top}+ th{background:var(--panel-2);color:#fff;font-weight:600}+ td code{font-size:.85em}+ tr:nth-child(even) td{background:#13162600}+ .pill{font-size:11px;font-weight:700;padding:2px 8px;border-radius:999px;white-space:nowrap}+ .pill.add{background:#10271a;color:var(--good);border:1px solid #1f5132}+ .pill.rm{background:#2c1620;color:#ff9aa6;border:1px solid #5a2233}+ .pill.chg{background:#241c12;color:var(--warn);border:1px solid #5a4420}++ .two{display:grid;grid-template-columns:1fr 1fr;gap:16px}+ @media(max-width:760px){.two{grid-template-columns:1fr}}+ .mini{font-size:13.5px;color:var(--muted)}+ hr.soft{border:0;border-top:1px solid var(--line);margin:30px 0}+ .footer{margin-top:60px;padding-top:18px;border-top:1px solid var(--line);color:var(--muted);font-size:13px}+ .anchor-note{color:var(--muted);font-size:13.5px}+ kbd{font-family:"SF Mono",monospace;font-size:.82em;background:#0c1024;border:1px solid var(--line);border-bottom-width:2px;border-radius:5px;padding:1px 6px;color:#dfe3ff}+</style>+</head>+<body>++<header class="hero">+ <div class="eyebrow">Pipelex · cost-reporting rework</div>+ <h1>From a leaky in-process registry to event-sourced cost on the run result+ <span class="sub">The <code>UsageRegistry</code> success-path leak fix & distributed cost reporting — one piece of work</span>+ </h1>+ <div class="badges">+ <span class="badge green">Status: shipped on <code>fix/For-API-update</code></span>+ <span class="badge blue">PR #967 (core) · #968 (rename)</span>+ <span class="badge amber">Validated backend: NDJSON only — DynamoDB deferred</span>+ </div>+</header>++<div class="wrap">++ <!-- ============ TL;DR ============ -->+ <div class="card tldr">+ <h2>TL;DR</h2>+ <p class="lead">Cost reporting used to ride on a per-run, in-process <code>UsageRegistry</code> buffer. That buffer <strong>leaked on every successful run</strong> (it was only closed on the failure path) and it <strong>could not see usage from other worker processes</strong>, so distributed (Temporal) runs never reported their true cost. We removed the registry entirely and made cost an <strong>event-sourced artifact that rides back on <code>PipeOutput</code></strong>, exactly like <code>graph_spec</code> already does.</p>+ <ul>+ <li><strong>The leak is gone by removal.</strong> No submitter-side registry is opened anymore, so there is nothing to leak — the bug is structurally impossible, not just patched.</li>+ <li><strong>Cost now rides on the run result.</strong> New <code>PipeOutput.tokens_usages</code> is assembled from the trace-event stream at end-of-run, in <em>both</em> direct and Temporal modes. A multi-worker run aggregates usage from every worker into a <strong>single end-of-run cost report</strong>.</li>+ <li><strong>New <code>--costs / --no-costs</code> switch (default on)</strong> decouples cost collection from <code>--graph</code> over a shared event-log transport. The old <code>--cost-report</code> flag is removed (no back-compat).</li>+ <li><strong><code>--mock-inference</code></strong> fakes the LLM leaf with reportable synthetic usage so the cross-worker path can be validated cheaply and deterministically — no tokens billed.</li>+ <li><strong>Companion rename (#968):</strong> <code>GraphContext → TraceContext</code> and the internal gate <code>is_generate_costs → is_generate_usage</code> — the object carries two streams (graph + usage), so it's named for what it is.</li>+ </ul>++ <div class="callout scope" style="margin-bottom:0">+ <span class="tag">⚠ Read this — scope of what shipped</span>+ <p style="margin-bottom:0"><strong>We only implemented & validated the cost-reporting event log through NDJSON.</strong> The read-back is backend-agnostic, so the <em>code path</em> covers DynamoDB for free — but <strong>making distributed cost reporting actually work on DynamoDB is deferred</strong> and is <em>not</em> a validation target of this work. Local NDJSON is the one backend we care about and test here.</p>+ </div>++ <div class="kpis">+ <div class="kpi"><b>0</b><span>submitter registries opened (was 1/run, leaked)</span></div>+ <div class="kpi"><b>1</b><span>aggregated cost report per run, even cross-worker</span></div>+ <div class="kpi"><b>NDJSON</b><span>validated transport · DynamoDB deferred</span></div>+ <div class="kpi"><b>−175</b><span>lines deleted from <code>reporting_manager.py</code></span></div>+ </div>+ </div>++ <!-- ============ THE PROBLEM ============ -->+ <h2><span class="n">1.</span>Two bugs that were really one lifecycle</h2>+ <p>The investigation started as a narrow "the registry leaks on success" bug. On closer look it was the <em>same</em> object lifecycle as the long-deferred <strong>distributed cost-report aggregation</strong>. The <code>UsageRegistry</code> is a per-run, in-process cost buffer whose full lifecycle is <strong>open → populate → report → close</strong>. Two different steps were broken:</p>+ <ul>+ <li><strong>The leak</strong> = the <em>close</em> step never ran on the success path. The registry was opened in <code>pipeline_run_setup</code> but closed only inside <code>if not success:</code>. In a long-lived process (e.g. the Pipelex API) this orphaned a registry on every request, and reusing a <code>pipeline_run_id</code> could even collide on the leftover.</li>+ <li><strong>Distributed cost reporting</strong> = the cross-worker <em>replay-populate</em> step was never wired. Usage events from inference activities on separate worker processes landed in the backend but were never read back into a report. Cross-worker runs reported nothing.</li>+ </ul>+ <p>Because they're the same lifecycle, fixing them separately would have meant baking in registry-placement decisions for the leak that the cost-reporting design then had to unwind. So they were designed and built together.</p>++ <div class="mermaid">+flowchart LR+ O([open]):::ok --> P([populate]):::ok --> R([report]):::ok --> C([close]):::bad+ P -. "replay-populate from<br/>other workers — never wired" .-> X[/distributed cost<br/>reporting missing/]:::bad+ C -. "never runs on success" .-> L[/registry leak/]:::bad+ classDef ok fill:#10271a,stroke:#1f5132,color:#9be9a8;+ classDef bad fill:#3a1620,stroke:#5a2233,color:#ff9aa6;+ </div>+ <p class="anchor-note">Same buffer, same lifecycle, two missing steps — the unifying insight behind treating the leak fix and distributed cost reporting as one piece of work.</p>++ <!-- ============ THE FIX ============ -->+ <h2><span class="n">2.</span>The fix — mirror graph-spec assembly (Option B)</h2>+ <p>Graph tracing had already solved the exact cross-worker problem: "analyze the event stream at the end of the run and ride the artifact back on <code>PipeOutput</code>." Cost reporting now does the literal same thing. We <strong>removed the submitter-side registry</strong> and instead assemble usage from the trace-event stream at the same hook that assembles the graph, attaching it to <code>PipeOutput.tokens_usages</code>. The submitter renders the cost report from that field.</p>++ <div class="callout good">+ <span class="tag">✓ Why removal beats "remember to close"</span>+ <p style="margin-bottom:0">With nothing populating a submitter-side registry, there is no registry to leak — on every entry point and both backends. The live <code>_try_add_to_registry</code> path and the dead <code>inject_tokens_usages</code> retired with it. The leak fix is a <strong>structural guarantee</strong>, not a discipline you can forget.</p>+ </div>++ <h3>Before vs after</h3>+ <div class="tabs" id="arch">+ <div class="tabbar" role="tablist">+ <button class="tabbtn" role="tab" aria-selected="true" data-tab="arch-before">Before — side buffer (leaks)</button>+ <button class="tabbtn" role="tab" aria-selected="false" data-tab="arch-after">After — rides on PipeOutput</button>+ </div>+ <div class="tabpane active" id="arch-before" role="tabpanel">+ <div class="mermaid">+flowchart TB+ subgraph SUB["submitter process"]+ RS["pipeline_run_setup<br/>open_registry()"] --> REG[("UsageRegistry<br/>per-run buffer")]+ INF["in-process inference<br/>_try_add_to_registry"] --> REG+ REND["generate_report()"] --> REG+ RS -. "close_registry()<br/><b>only if not success</b>" .-> REG+ end+ REG ==> LEAK["💧 success path:<br/>never closed = LEAK"]:::bad+ WK["usage from other<br/>worker processes"] -. "emitted, never read back" .-> NOPE["✗ no cross-worker report"]:::bad+ classDef bad fill:#3a1620,stroke:#5a2233,color:#ff9aa6;+ </div>+ </div>+ <div class="tabpane" id="arch-after" role="tabpanel">+ <div class="mermaid">+flowchart TB+ INF["inference anywhere<br/>(DIRECT or any worker)"] -->|"UsageReportEvent"| EL[("event log<br/>NDJSON partition")]+ EL -->|"read once at end of run"| AS["assemble_tracing"]+ AS -->|"UsageAggregator"| TU["tokens_usages"]+ AS -->|"GraphSpecAssembler"| GS["graph_spec"]+ TU --> PO[["PipeOutput"]]+ GS --> PO+ PO -->|"flows back to submitter"| RR["render_run_cost_report()"]:::ok+ classDef ok fill:#10271a,stroke:#1f5132,color:#9be9a8;+ </div>+ </div>+ </div>+ <p class="mini">Direct mode assembles in the submitter process (<code>PipeRun.run</code> <code>finally</code>); Temporal mode assembles in the <code>act_assemble_tracing</code> activity (renamed from <code>act_assemble_graph</code>) and the workflow sets both artifacts on <code>PipeOutput</code> before returning. Same hook, same flow, one mental model.</p>++ <!-- ============ SHARED TRANSPORT ============ -->+ <h2><span class="n">3.</span>The load-bearing decoupling — <code>--costs</code> over a shared transport</h2>+ <p>This is the one piece that's <strong>not cosmetic</strong>. Previously usage events were only emitted when <code>--graph</code> + tracing were on (the event log was created <em>inside</em> <code>if is_generate_graph</code>). Once the live registry is gone, <code>--no-graph</code> would lose cost entirely. So we added a dedicated, default-on <strong><code>--costs</code></strong> switch that mirrors <code>--graph</code>: the event log is the <strong>shared transport</strong>; each switch gates its own event type and its own assembler; each rides its artifact back on <code>PipeOutput</code>.</p>++ <div class="mermaid">+flowchart TB+ G["--graph<br/>(emit_graph_events)"]:::g -->|gates| GE["graph node / edge events"]:::g+ C["--costs<br/>(emit_usage_events)"]:::c -->|gates| UE["UsageReportEvent"]:::c+ GE --> EL[("shared event log<br/>NDJSON")]+ UE --> EL+ EL --> GA["GraphSpecAssembler"]:::g --> GS["graph_spec"]:::g+ EL --> UA["UsageAggregator"]:::c --> TU["tokens_usages"]:::c+ GS --> PO[["PipeOutput"]]+ TU --> PO+ classDef g fill:#141a36,stroke:#28346b,color:#aeb8ff;+ classDef c fill:#0f2519,stroke:#1f5132,color:#9be9a8;+ </div>++ <table>+ <thead><tr><th>Invocation</th><th>Graph built?</th><th>Cost reported?</th></tr></thead>+ <tbody>+ <tr><td><code>--graph --costs</code> (defaults)</td><td>yes</td><td>yes</td></tr>+ <tr><td><code>--no-graph --costs</code></td><td>no</td><td><strong>yes</strong> — this is what the switch buys</td></tr>+ <tr><td><code>--graph --no-costs</code></td><td>yes</td><td>no</td></tr>+ <tr><td><code>--no-graph --no-costs</code></td><td>no</td><td>no</td></tr>+ </tbody>+ </table>+ <p class="mini"><strong>D5 detail:</strong> in costs-only mode the graph tracer is still opened with <code>event_log=None</code> — it keeps minting node ids (so each <code>UsageReportEvent</code> can attribute to its <code>node_id</code>) but emits no graph events. The node tree is the shared substrate both streams hang off; that's the deeper reason the rename to <code>TraceContext</code> is the right framing.</p>++ <!-- ============ DISTRIBUTED ============ -->+ <h2><span class="n">4.</span>Distributed cost reporting — cross-worker → one report</h2>+ <p>In Temporal mode a single logical run can span several processes: a parent workflow on one worker, a child on another, the inference activity on a runner. Every <code>UsageReportEvent</code> lands in <strong>one partition</strong> with a <code>writer_id</code> to disambiguate the producer. At end-of-run, a single event read feeds both assemblers, and the aggregated usage rides home on <code>PipeOutput</code>.</p>++ <div class="mermaid">+sequenceDiagram+ autonumber+ participant S as Submitter+ participant WA as Worker A<br/>(parent workflow)+ participant WC as Runner<br/>(inference activity)+ participant L as NDJSON log<br/>(one partition)+ participant AT as act_assemble_tracing++ S->>WA: dispatch WfPipeRun (--costs on)+ WA->>WC: inference activity+ WC-->>L: UsageReportEvent (writer_id = act_*)+ WA-->>L: UsageReportEvent (parent steps)+ Note over L: every worker's usage<br/>lands in one partition+ WA->>AT: Step 2 — assemble (single read)+ L-->>AT: read_events()+ AT-->>WA: TracingAssembly{ graph_spec, tokens_usages }+ WA-->>S: PipeOutput.tokens_usages+ S->>S: render_run_cost_report(tokens_usages=…)+ </div>++ <div class="callout info">+ <span class="tag">ℹ Single source of truth = events</span>+ <p style="margin-bottom:0">In direct mode, inference no longer <em>also</em> adds to a live registry — that path was removed. Cost always comes from the aggregated event stream, so there is <strong>no double-counting</strong> to guard against. One run, one source.</p>+ </div>++ <!-- ============ MOCK INFERENCE ============ -->+ <h2><span class="n">5.</span><code>--mock-inference</code> — validate the distributed path for free</h2>+ <p>Proving cross-worker cost reporting used to need real LLM spend in a real multi-worker topology. <code>--mock-inference</code> is a <strong>LIVE</strong> run (operators dispatch real activities — the true distributed path is exercised) whose AI call is faked <em>at the inference leaf</em> with <strong>reportable, non-zero synthetic usage</strong>. So a cost report actually renders from cross-worker usage, deterministically, with nothing billed.</p>+ <ul>+ <li>It is <strong>not</strong> <code>--dry-run</code>: dry-run swaps the generator <em>before</em> dispatch, so it never reaches the activity and emits zero-token (suppressed) usage. Mock-inference keeps <code>run_mode = LIVE</code> and mocks only the leaf — the one thing dry-run can't validate cheaply is a <em>rendered</em> cost report.</li>+ <li><strong>LLM leaf only.</strong> Image-gen / extract / search under <code>--mock-inference</code> <strong>fail loud</strong> with <code>MockInferenceUnsupportedError</code> (pointing at <code>--dry-run</code>) rather than silently calling the real provider — their output is stored <em>above</em> the leaf, so a leaf-level mock would push synthetic data through the storage path.</li>+ <li>Mutually exclusive with <code>--dry-run</code>; main <code>run</code> subcommands only (agent CLI deferred).</li>+ </ul>++ <!-- ============ CRITICAL CODE ============ -->+ <h2><span class="n">6.</span>The critical changes, in code</h2>+ <p class="mini">Minimal, representative diffs for the load-bearing edits. Symbol names are the durable anchors; line numbers drift.</p>++ <details open>+ <summary>① The leak fix — stop opening a registry at all <span class="pill rm" style="margin-left:auto">removal</span></summary>+ <div class="details-body">+ <p>The asymmetry that caused the leak: <code>open_registry()</code> ran unconditionally during setup, but <code>close_registry()</code> sat behind <code>if not success:</code>. Both are simply gone now.</p>+ <span class="filename">pipelex/pipeline/pipeline_run_setup.py</span>+ <pre class="diff">+<span class="ln del">- registry_opened = False</span>+<span class="ln ctx"> ...</span>+<span class="ln del">- if execution_config.is_generate_graph:</span>+<span class="ln add">+ is_generate_graph = execution_config.is_generate_graph</span>+<span class="ln add">+ is_generate_usage = execution_config.is_generate_usage</span>+<span class="ln add">+ if is_generate_graph or is_generate_usage:</span>+<span class="ln ctx"> ...</span>+<span class="ln cmt"> # D5: costs-only (--no-graph --costs) → tracer still mints node ids,</span>+<span class="ln cmt"> # but emits no graph events; usage rides via set_event_log below.</span>+<span class="ln del">- event_log=event_log,</span>+<span class="ln add">+ event_log=event_log if is_generate_graph else None,</span>+<span class="ln add">+ emit_graph_events=is_generate_graph,</span>+<span class="ln add">+ emit_usage_events=is_generate_usage,</span>+<span class="ln ctx"> ...</span>+<span class="ln del">- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)</span>+<span class="ln del">- registry_opened = True</span>+<span class="ln del">- if event_log is not None:</span>+<span class="ln add">+ if is_generate_usage and event_log is not None:</span>+<span class="ln ctx"> get_report_delegate().set_event_log(...) # the surviving usage transport</span>+<span class="ln ctx"> ...</span>+<span class="ln ctx"> finally:</span>+<span class="ln del">- if registry_opened:</span>+<span class="ln del">- get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)</span>+ </pre>+ </div>+ </details>++ <details>+ <summary>② Usage rides on the run result — new fields on <code>PipeOutput</code> <span class="pill add" style="margin-left:auto">added</span></summary>+ <div class="details-body">+ <p>A literal mirror of <code>graph_spec</code> / <code>graph_assembly_error</code>. <code>AnyTokensUsage</code> is already a discriminated union of pydantic models, already serialized through the event-log backend — so it crosses the Temporal boundary safely.</p>+ <span class="filename">pipelex/core/pipes/pipe_output.py</span>+ <pre class="diff">+<span class="ln ctx"> graph_spec: GraphSpec | None = None</span>+<span class="ln ctx"> graph_assembly_error: str | None = None</span>+<span class="ln add">+ # Token usage assembled from the trace-event stream at end of run (mirrors graph_spec):</span>+<span class="ln add">+ # the submitter renders the cost report from this field. None when cost reporting was off</span>+<span class="ln add">+ # or no trace events emitted; an empty list when on but no inference happened.</span>+<span class="ln add">+ tokens_usages: list[AnyTokensUsage] | None = None</span>+<span class="ln add">+ usage_assembly_error: str | None = None</span>+ </pre>+ <p class="mini">Because <code>PipelexPipelineExecuteResponse</code> wraps <code>pipe_output</code>, the field is auto-exposed on the Pipelex API response — no extra plumbing.</p>+ </div>+ </details>++ <details>+ <summary>③ The reporting surface shrinks — registry methods leave the protocol <span class="pill rm" style="margin-left:auto">removal</span></summary>+ <div class="details-body">+ <p><code>ReportingProtocol</code> (and <code>ReportingNoOp</code>) no longer expose registry lifecycle. The <code>UsageRegistry</code> model is deleted outright. Embedders that rendered cost via <code>get_report_delegate().generate_report()</code> now render from the returned <code>pipe_output.tokens_usages</code> instead.</p>+ <span class="filename">pipelex/reporting/reporting_protocol.py</span>+ <pre class="diff">+<span class="ln ctx"> class ReportingProtocol(Protocol):</span>+<span class="ln del">- def open_registry(self, pipeline_run_id: str): ...</span>+<span class="ln ctx"> def report_inference_job(self, inference_job: InferenceJobAbstract): ...</span>+<span class="ln del">- def generate_report(self, pipeline_run_id: str | None = None, print_to_console: bool = True): ...</span>+<span class="ln del">- def close_registry(self, pipeline_run_id: str): ...</span>+<span class="ln ctx"> def setup(self): ...</span>+<span class="ln ctx"> def teardown(self): ...</span>+<span class="ln ctx"> def set_event_log(self, ...): ... # kept — the usage transport</span>+ </pre>+ <p class="mini"><code>reporting_manager.py</code> lost <strong>175 lines</strong>: <code>_usage_registries</code>, <code>open_registry</code>, <code>close_registry</code>, the <code>_get_registry_strict</code> / <code>_get_or_create_registry</code> / <code>_try_add_to_registry</code> trio, <code>inject_tokens_usages</code>, <code>generate_report</code>, and the <code>UsageRegistry</code> / <code>UsageRegistryRoot</code> models. The emit core (<code>_emit_usage_event</code> + fast path + runner fallback) is unchanged — that's the surviving transport.</p>+ </div>+ </details>++ <details>+ <summary>④ The only cost renderer is now event-sourced <span class="pill add" style="margin-left:auto">new module</span></summary>+ <div class="details-body">+ <p>The main CLI renders from <code>pipe_output.tokens_usages</code>. One aggregation pass feeds both the suppression decision and the render. Note the gate: <strong>dry runs are suppressed, but real runs on free/zero-price models still report</strong> their token usage (with a zero total) — gating on cost alone would wrongly hide them.</p>+ <span class="filename">pipelex/reporting/cost_report_renderer.py</span>+ <pre class="code"><code>def render_run_cost_report(+ *, pipeline_run_id: str,+ tokens_usages: Sequence[AnyTokensUsage] | None,+ is_generate_costs: bool, # public flag name kept; cost domain stays cost-named+) -> None:+ if not is_generate_costs or not tokens_usages:+ return+ reporting_config = get_config().pipelex.reporting_config+ print_to_console = reporting_config.is_log_costs_to_console # default now TRUE+ is_csv_enabled = reporting_config.is_generate_cost_report_file_enabled+ if not (print_to_console or is_csv_enabled):+ return+ try:+ aggregated = CostRegistry.aggregate_costs(tokens_usages) # single pass+ if not aggregated.has_reportable_usage: # any tokens OR any cost+ return # → suppresses dry runs only+ ...+ CostRegistry.render_report(aggregated, pipeline_run_id=pipeline_run_id, ...)+ except (OSError, UnicodeEncodeError, PipelexError) as e:+ log.warning(f"Cost report generation failed (run succeeded): {e}")</code></pre>+ <p class="mini">A cost-report failure never fails an otherwise-successful run — aggregation, CSV setup, and render all sit inside one run-succeeded <code>try</code>. The agent CLI takes a different path: no Rich table, a machine-readable <code>cost_report</code> object on its <code>--with-memory</code> JSON envelope instead.</p>+ </div>+ </details>++ <!-- ============ PUBLIC SURFACE ============ -->+ <h2><span class="n">7.</span>What changed on the public surface</h2>+ <table>+ <thead><tr><th>Change</th><th>Kind</th><th>Notes</th></tr></thead>+ <tbody>+ <tr><td><code>--costs / --no-costs</code> on <code>run pipe|method|bundle</code></td><td><span class="pill add">added</span></td><td>Default on. Reports cost independently of <code>--graph</code>.</td></tr>+ <tr><td><code>--cost-report / --no-cost-report</code></td><td><span class="pill rm">removed</span></td><td><strong>Breaking.</strong> Folded into <code>--costs</code>. No deprecation period.</td></tr>+ <tr><td><code>PipeOutput.tokens_usages</code> + <code>usage_assembly_error</code></td><td><span class="pill add">added</span></td><td>Cost is part of the run result; exposed on the API response automatically.</td></tr>+ <tr><td><code>--mock-inference</code></td><td><span class="pill add">added</span></td><td>LIVE run, faked LLM leaf, reportable synthetic usage. Excl. with <code>--dry-run</code>.</td></tr>+ <tr><td>agent CLI <code>cost_report</code> in JSON (<code>--with-memory</code>)</td><td><span class="pill add">added</span></td><td>Best-effort <code>{total_cost, by_model}</code>, real USD. JSON-only, no stderr table.</td></tr>+ <tr><td><code>is_log_costs_to_console</code> default <code>false → true</code></td><td><span class="pill chg">changed</span></td><td>CLI prints the cost table by default; per-inference-job console logging removed.</td></tr>+ <tr><td><code>UsageRegistry</code> + registry methods on <code>ReportingProtocol</code></td><td><span class="pill rm">removed</span></td><td>Render from <code>pipe_output.tokens_usages</code> instead.</td></tr>+ <tr><td><code>act_assemble_graph → act_assemble_tracing</code></td><td><span class="pill chg">changed</span></td><td>Now assembles graph <em>and</em> usage from one event read.</td></tr>+ <tr><td>Dry runs never report; free-model runs still do</td><td><span class="pill chg">changed</span></td><td>Gate keys on "did real work" (any tokens or any cost), not total cost.</td></tr>+ </tbody>+ </table>++ <!-- ============ RENAME ============ -->+ <h2><span class="n">8.</span>Companion rename (#968) — <code>GraphContext → TraceContext</code></h2>+ <p>Once usage rides on the same per-run object as the graph, the name <code>GraphContext</code> under-described it — a cost concern was riding on something called "Graph." But usage isn't a free-rider: each <code>UsageReportEvent.node_id</code> is stamped from the context's <code>parent_node_id</code>, so <strong>graph events and cost events are two projections of one substrate</strong> — the per-execution node tree. The rename names the object for what it is.</p>+ <div class="two">+ <div class="card" style="margin:0">+ <h3 style="margin-top:0">Renamed</h3>+ <ul class="mini" style="margin-bottom:0">+ <li><code>GraphContext</code> → <code>TraceContext</code> (file, type, the <code>graph_context</code> attribute incl. <code>JobMetadata.trace_context</code>)</li>+ <li><code>is_generate_costs</code> → <code>is_generate_usage</code> (config field; the gate feeds <code>emit_usage_events</code>, so "usage gates usage")</li>+ <li><code>with_execution_overrides(generate_costs=…)</code> → <code>generate_usage=…</code></li>+ </ul>+ </div>+ <div class="card" style="margin:0">+ <h3 style="margin-top:0">Deliberately kept</h3>+ <ul class="mini" style="margin-bottom:0">+ <li>The public <code>--costs / --no-costs</code> flag — "costs" is the deliverable the user asked for.</li>+ <li>The whole cost domain: <code>CostRegistry</code>, <code>cost_report_renderer.py</code>, <code>render_run_cost_report(is_generate_costs=…)</code> — they genuinely compute money.</li>+ <li>The <code>graph_id → run_id</code> nit — deferred to a focused follow-up.</li>+ </ul>+ </div>+ </div>++ <!-- ============ DEFERRED ============ -->+ <h2><span class="n">9.</span>Explicitly deferred / non-goals</h2>++ <div class="callout scope">+ <span class="tag">⚠ The headline deferral — DynamoDB</span>+ <p style="margin-bottom:0"><strong>Distributed cost reporting is validated on NDJSON only.</strong> The end-of-run read-back is backend-agnostic, so DynamoDB rides along in the <em>code path</em> for free — but the DynamoDB-specific concerns (<code>BatchWriteItem</code> batching, the shared-filesystem invariant, per-thread <code>writer_id</code>) are <strong>not</strong> exercised or guaranteed by this work. Treat DynamoDB-backed distributed cost reporting as unproven until it's a deliberate validation target.</p>+ </div>++ <details>+ <summary>Other deliberate non-goals (none block the shipped work)</summary>+ <div class="details-body">+ <ul>+ <li><strong>Cost-per-node breakdown.</strong> Each <code>UsageReportEvent</code> already carries the producing pipe's <code>node_id</code>, but the report aggregates by model only. A per-node view is a future projection over the same stream — data's already there.</li>+ <li><strong>Costs-only in-memory tracer skip (deep half of E1).</strong> Under <code>--no-graph --costs</code> the tracer still accumulates the in-memory graph (cheap; never assembled). The cheap half — gating per-pipe serialization on <code>emit_graph_events</code> — landed; fully skipping the tracer is deferred (must still mint node ids for usage attribution).</li>+ <li><strong><code>--mock-inference</code> coverage.</strong> LLM leaf only; img-gen / extract / search fail loud. Agent CLI doesn't expose it yet. Full per-operator synthetic-output coverage comes via the dry-run-refactor follow-up (<code>run_mode=DRY</code>).</li>+ <li><strong>Two design decisions (see <code>cost-report-deferred-decisions.md</code>).</strong> #3 — cost reporting is coupled to <code>tracing_config.is_enabled</code> (costs-on + tracing-off ⇒ no report). #6 — the agent CLI gates on the raw <code>costs</code> param while the main CLI gates on the resolved <code>is_generate_usage</code> config. Both want a deliberate call, not a reflexive patch.</li>+ <li><strong>T3 — request-scoped tracing state</strong> instead of the process-singleton <code>ReportingManager</code>. Removing the registry shrank the singleton's surface but the per-context dict remains. Open follow-up.</li>+ <li><strong>Pre-existing bugs flagged, not fixed here:</strong> the cost report ignores reasoning/audio/prediction token costs; DIRECT vs Temporal object re-validation use different dump modes. Both predate this work.</li>+ </ul>+ </div>+ </details>++ <hr class="soft">+ <p class="mini">Forward-looking records live alongside this file in <code>wip/registry/</code> — the deferred non-goals in <code>deferred-followups.md</code>, the two deferred design decisions in <code>cost-report-deferred-decisions.md</code>, and the usage-without-cost capability idea in <code>usage-reporting-without-cost.md</code>. The full build/review history is in PR #967 and git.</p>++ <div class="footer">+ Pipelex · cost-reporting rework · branch <code>fix/For-API-update</code> (PR #967) + rename PR #968. Generated for co-developer onboarding — diagrams render via mermaid (CDN).+ </div>++</div>++<script>+ mermaid.initialize({+ startOnLoad:true,+ theme:'dark',+ themeVariables:{+ fontFamily:'-apple-system,Segoe UI,Roboto,sans-serif',+ primaryColor:'#1d2136', primaryBorderColor:'#39406e', primaryTextColor:'#e7e9f3',+ lineColor:'#6c7bff', secondaryColor:'#141a36', tertiaryColor:'#10182f',+ actorBkg:'#1d2136', actorBorder:'#39406e', actorTextColor:'#e7e9f3',+ noteBkgColor:'#241b10', noteBorderColor:'#5a4420', noteTextColor:'#ffd9a0',+ sequenceNumberColor:'#0f1220'+ },+ flowchart:{curve:'basis',useMaxWidth:true},+ sequence:{useMaxWidth:true,actorMargin:38}+ });++ // tabs+ document.querySelectorAll('.tabs').forEach(function(group){+ var btns = group.querySelectorAll('.tabbtn');+ btns.forEach(function(btn){+ btn.addEventListener('click', function(){+ var id = btn.getAttribute('data-tab');+ btns.forEach(function(b){ b.setAttribute('aria-selected', b===btn ? 'true':'false'); });+ group.querySelectorAll('.tabpane').forEach(function(p){+ p.classList.toggle('active', p.id===id);+ });+ });+ });+ });+</script>+</body>+</html>diff --git a/wip/registry/deferred-followups.md b/wip/registry/deferred-followups.mdnew file mode 100644index 000000000..504592e92--- /dev/null+++ b/wip/registry/deferred-followups.md@@ -0,0 +1,39 @@+# Registry / cost-reporting — deferred follow-ups++Durable home for the follow-ups this feature deliberately left out of scope. The leak fix + distributed cost reporting shipped in **PR #967** (+ companion rename **#968**); see [`cost-reporting-overview.html`](cost-reporting-overview.html) for the as-built overview. The items below are the known, intentional non-goals — none blocks the shipped work.++Two further deferred items — both *design decisions* rather than follow-up work — live in their own doc: [`cost-report-deferred-decisions.md`](cost-report-deferred-decisions.md) (#3 cost reporting coupled to `tracing_config.is_enabled`; #6 agent-CLI vs main-CLI "is cost reporting on?" gating divergence). Those want a deliberate call; the items here are scoped work to pick up when signals warrant.++## Optimizations++- **D5 — costs-only in-memory tracer skip (deep half of review item E1).** In `--no-graph --costs` mode the graph tracer still runs in memory (minting node ids, accumulating the in-memory graph) even though no `GraphSpec` is ever assembled or output. Phase 2 landed the *cheap* half: per-pipe graph serialization (`pipe_abstract.py` IOSpec dumps, pipe/concept registry, `pipe_parallel.py` controller-output registration, `graph_tracer.teardown` spec build) is gated on `emit_graph_events`, so costs-only no longer pays for serialization. The *deep* half — skipping the in-memory tracer entirely in costs-only mode — is deferred. Watch-item: `on_pipe_start` still mints node ids so usage-event `node_id` correlation is unaffected; any skip must preserve that.++- **Cost-per-node correlation via `UsageReportEvent.node_id`.** Each `UsageReportEvent` already carries the `node_id` of the pipe that produced it, but the assembled cost report aggregates by model only. A future report could break cost down per graph node (which pipe/step cost what), correlating usage events to graph nodes via `node_id`. Not built; the data is already in the event stream.++## Altitude / architecture++- **Review item A1 — `TraceContext.from_execution_config(...)` factory.** The `(graph, costs) → (event_log routing, set_event_log, flags)` decision is currently made at each tracing arm (DIRECT `pipeline_run_setup`, TEMPORAL `wf_pipe_router`). A2 captured most of the value by threading the emit flags into `open_tracer`/`setup` (so the returned `TraceContext` is born correct and the post-hoc `model_copy` footguns are gone). A single factory that owns the whole decision for both arms is the remaining polish — deferred as low-value-now.++- **`graph_id → run_id` rename + the shelved `TraceContext` split (left over from the merged #968 rename).** #968 renamed `GraphContext → TraceContext` and the gate `is_generate_costs → is_generate_usage`, but deliberately parked two follow-ons: (1) the `graph_id → run_id` field rename — it really is the run id, but the rename ripples into the graph-named `GraphTracerManager.open_tracer(graph_id=…)` plumbing and the `make_node_id` format, so it wants a focused pass; (2) a *type* split — a `TraceContext` base with a `GraphContext(TraceContext)` subclass (Option 2), or extracting the `emit_graph_events` / `emit_usage_events` toggles into an `EmitPolicy` value object (Option 3). Both splits are low-value now: the node tree is **always** built (even under `--no-graph --costs`, per D5), so the runtime object is always the full graph-capable context — a split would only narrow what the reporting seam is allowed to *see*, never drop machinery. Revisit Option 2 only if reporting grows graph-independent consumers; Option 3 only if a third trace stream appears. Do **not** touch `UsageReportEvent.node_id` — cost-per-node attribution to the tree is a deliberate feature.++- **Review item T5 — `wf_pipe_router.py` setup-failure `except` doesn't restore `job_metadata.trace_context`.** On a tracer-setup failure the pipe can run with a `tracer_key` pointing at a closed tracer. Pre-existing latent issue (not introduced by this work; Temporal is unshipped — see [[project_temporal_not_shipped]]). Flag-and-fix candidate, parked.++- **T3 — request-scoped tracing state instead of the process-singleton `ReportingManager`.** The deeper refactor in the distributed-execution track ([`../distributed-execution/tracing-cost-reporting.md`](../distributed-execution/tracing-cost-reporting.md) T3). Removing the submitter-side `UsageRegistry` shrank the singleton's surface, but the per-context `_event_log_contexts` dict on a process-global manager remains. Open follow-up.++ **Facet flagged on PR #967 (codex, P1) — activity usage writes can mutate a workflow-registered buffer.** In a *single-process* Temporal deployment (one worker hosting both `WfPipeRouter` workflows and the inference activities), the workflow registers its `BufferingEventLog` in the process-global `ReportingManager._event_log_contexts`. An inference activity in the same process then hits the fast path (`_emit_via_registered_context`) and appends to the workflow's in-memory buffer — an activity side effect that is **not** part of Temporal's deterministic replay (the activity *result* replays from history, but the buffer append does not), so on replay the later drain/flush can diverge or drop usage. **Why this is deferred, not patched on this PR:** (1) it is *pre-existing* — the fast path + per-context registration predate this work (the registry track kept the emit core unchanged; it only removed the submitter-side registry), so PR #967 neither introduced nor worsened it; (2) the **recommended split router+runner topology does not hit it** — activities run on the runner process, which never registered the workflow's buffer, so they correctly take the per-process activity-log fallback; (3) the clean fix is request-scoped tracing state (this T3 refactor) or making activities always bypass workflow-registered buffers — bolting Temporal-awareness (`activity.in_activity()`) into the generic, non-Temporal `ReportingManager` would be a layering violation; (4) Temporal is not shipped to prod. Pick this up with the T3 refactor.++## Pre-existing gaps surfaced during the PR #967 final review (flag, not fix-here)++These were found by the gstack `/review` adversarial pass (Codex + Claude). They are **pre-existing** (not introduced by the registry/cost work) and out of scope for this PR's purpose, so they are flagged here rather than expanding the PR. Per the repo "flag and fix existing bugs" rule, each is a candidate follow-up.++- **Cost report ignores reasoning / audio / prediction token costs.** `CostRegistry.complete_cost_report` only computes cost for `INPUT` (→ joined/cached/non-cached) and `OUTPUT`; `aggregate_costs` / `compute_total_cost` only sum those. The `TokenCategory` enum also has `OUTPUT_REASONING`, `OUTPUT_AUDIO`, `OUTPUT_ACCEPTED_PREDICTION`, `OUTPUT_REJECTED_PREDICTION`, `INPUT_AUDIO` — providers (e.g. the OpenAI Responses factory) populate some of these, but their cost is never computed or summed, so a run with reasoning/audio tokens **under-reports total cost**. Pre-existing (this PR only refactored `compute_total_cost`; it did not change the category set). Fixing it properly means extending `compute_cost_report` / `complete_cost_report` / `aggregate_costs` / the render columns / the agent summary to handle the extra categories, with per-provider unit-cost wiring — a cost-model change, separate from the leak/distributed work.++- **DIRECT vs TEMPORAL object re-validation use different dump modes.** `ContentGenerator._revalidate_against_object_class` (DIRECT) dumps with `model_dump(serialize_as_any=True)` (python mode) while `ContentGeneratorInWorkflow` (Temporal) uses `model_dump(mode="json", serialize_as_any=True)`. Same logical re-validation, two serialization modes — so a `--mock-inference` synthetic object (or any object) can round-trip cleanly in one execution mode and raise `MockInferenceObjectFidelityError` in the other. The divergence predates this PR (each arm kept its existing behavior), but the shared-helper framing makes it load-bearing. Aligning the two on one canonical mode is the durable fix; which mode is canonical is the judgment call.++## `--mock-inference` coverage++- **LLM leaf only.** `--mock-inference` mocks the LLM inference leaf (`llm_gen_text`/`object`/`object_list`). Image-gen / extract / search **fail loud** with `MockInferenceUnsupportedError` (their output is stored *above* the leaf by `generated_content_factory`, so a leaf-level mock would push synthetic data through the storage path — whereas `ContentGeneratorDry` short-circuits storage). Full per-operator synthetic-output coverage comes via [`../dry-run-refactor/followup-leaf-run-mode-mock.md`](../dry-run-refactor/followup-leaf-run-mode-mock.md) B2, which replaces each hard guard with a real synthetic-output branch.++- **Main `run` subcommands only.** The flag is on `pipelex run pipe|method|bundle`, not the agent CLI. The mechanism (`JobMetadata.is_mock_inference` + leaf branch + `PipelexRunner`) is surface-agnostic, so the agent CLI can adopt it trivially when needed.++- **`is_mock_inference` fate after the dry-run-refactor follow-up.** Decided 2026-06-06: build the interim flag now (Phase 5 is the follow-up's B1 core — a reusable leaf-level dry helper), settle retirement at B2. Open sub-decision: *fully retire* `--mock-inference` (DRY-on-Temporal covers cross-worker dispatch + `tokens_usages` assembly once it dispatches) vs *keep a thin reportable-mock* (DRY emits zero-token usage → suppressed report, so the cheap *rendered* cost-report validation — Tier 8b — is unique to a non-zero mock). Decide at B2 time. See [`../dry-run-refactor/followup-leaf-run-mode-mock.md`](../dry-run-refactor/followup-leaf-run-mode-mock.md).diff --git a/wip/registry/usage-reporting-without-cost.md b/wip/registry/usage-reporting-without-cost.mdnew file mode 100644index 000000000..979a5d8a8--- /dev/null+++ b/wip/registry/usage-reporting-without-cost.md@@ -0,0 +1,31 @@+# Capability: usage reporting independent of cost++**Status: IDEA, LOW PRIORITY.** Not decided, not scheduled. Captured because it is interesting in its own right; revisit when a concrete signal appears (a user wanting token counts without dollars, or a model/backend with no cost data).++## The capability gap++Today usage-emission and cost-rendering are **bundled under one switch**. The `--costs` flag (→ internal gate `is_generate_usage`, renamed from `is_generate_costs` in #968) turns on usage-event emission *and* the cost report is the only deliverable. There is **no mode that gives you token counts without dollar figures.**++But usage (token counts) is valuable on its own, not just as an input to a cost number:++- **Consumption monitoring / profiling** — how many tokens a run burns, where, regardless of price. Useful for prompt-size regressions and context-budget work even when you don't care about (or can't price) the spend.+- **Models / backends without cost data.** A handle with no per-token unit cost configured produces a meaningless or zero cost report, but its token counts are still real and worth surfacing.+- **Budget-agnostic dashboards.** Token usage is a stable signal; prices drift. Reporting usage separately lets a downstream consumer apply its own pricing.++The data is already there — `UsageReportEvent` carries `TokensUsage` (`nb_tokens_by_category`); cost is computed *from* it by `CostRegistry`. So "usage without cost" is dropping a derivation step, not capturing new data.++## Why this is the deeper question behind the naming cleanup++The companion rename merged in #968 (`is_generate_costs → is_generate_usage`, `GraphContext → TraceContext`) handled *consistency* — it aligned the internal gate name to the usage stream it feeds, and deliberately left the public `--costs` flag and the cost domain alone. It did **not** add any capability.++This doc is the *capability* question: should usage be a first-class, separately-requestable output, with cost as one **view** layered on top? If yes, the public `--costs`-vs-`--usage` naming falls out of it naturally (usage = the measured data, cost = a priced projection). If no, keeping "cost" as the single user-facing word is correct and the naming work stops at the internal gate. **Decide the capability first; the public naming is downstream of it.**++## Rough shape if pursued (not designed)++- Split the single gate into two intents: *emit usage* and *render cost*. Cost-render becomes gated on usage-present (it already is, post-companion-rename) plus a separate "show costs" choice.+- Surface a **usage-only report**: token counts by category / by model, no dollar columns — a sibling renderer (or a mode of `cost_report_renderer.py`) that omits the priced columns.+- Public surface: likely a `--usage` / `--costs` pair (or `--report usage|cost|both`), where `--costs` implies usage. Exact ergonomics TBD.++## Related, already-captured++- **Cost-per-node breakdown** ([`deferred-followups.md`](deferred-followups.md), "Cost-per-node correlation") — also a *view* over the same `UsageReportEvent` stream (aggregate by `node_id` instead of by model). A usage-only-per-node report is the intersection of that item and this one; if both get built, they share the same "projection over the usage event stream" machinery.