Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feat(reporting): one-arg render_cost_report_for_output submitter helper (#978)
Louis Choquel942579ed8dbc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c5f6e67ab..97527b819 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,11 @@# Changelog+## [Unreleased]++### Added++- **`render_cost_report_for_output(pipe_output)` submitter helper.** A one-arg convenience over `render_run_cost_report` that unpacks `pipeline_run_id` and `tokens_usages` from a finished `PipeOutput` and derives the `--costs` gate from the output itself — `pipe_output.tokens_usages is None` is exactly the signal that cost reporting was off for the run, the decision the runner already resolved (with all `--costs/--no-costs` overrides applied) and recorded on the output. Embedders and the `pipelex run` CLI no longer re-derive the three primitive arguments by hand or re-read global config to reconstruct the gate. `render_run_cost_report` is unchanged as the low-level primitive for paths where the three values come from different places (the agent CLI's `build_cost_summary` JSON envelope, distributed reassembly).+## [v0.32.0] - 2026-06-09This cycle reworks **cost reporting** so it survives distributed execution and stops leaking. Cost now rides on the run result instead of a side buffer: a multi-worker Temporal run aggregates usage from every worker into a single end-of-run report, the success-path registry leak is gone by removal, and a new `--costs` switch decouples cost collection from `--graph`. It also extracts the **framework-agnostic runtime bridge** so any host runtime — not just Mistral Workflows — can embed Pipelex through one boundary.diff --git a/docs/features/cost-tracking.md b/docs/features/cost-tracking.mdindex 228bebf40..ce29aa182 100644--- a/docs/features/cost-tracking.md+++ b/docs/features/cost-tracking.md@@ -19,6 +19,18 @@ Real-time cost information displayed in the console during pipeline execution. EExport detailed cost reports as CSV files for analysis, billing, and audit trails. Each row captures the pipe, model, token counts (input/output), and cost for a single operation.+## Rendering Reports When Embedding++Cost is part of the run result: a finished run carries its assembled usage on `pipe_output.tokens_usages`. If you embed Pipelex and want the same end-of-run console table / CSV that the CLI prints, render it in one call from the output:++```python+from pipelex.reporting.cost_report_renderer import render_cost_report_for_output++render_cost_report_for_output(pipe_output)+```++The `--costs` gate is read off the output itself — `tokens_usages` is `None` exactly when cost reporting was off for the run — so you never re-read global config to reconstruct whether costs were on.+## Custom ReportingImplement custom reporting backends via the [Reporting Delegate](advanced-customizations.md) injection point. The reporting protocol receives structured cost events during execution, allowing you to push data to your own analytics, dashboards, or billing systems.diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.pyindex e0feb6620..97d7553d2 100644--- a/pipelex/cli/commands/run/_run_core.py+++ b/pipelex/cli/commands/run/_run_core.py@@ -32,7 +32,7 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.exceptions import PipelineExecutionErrorfrom pipelex.pipeline.runner import PipelexRunner-from pipelex.reporting.cost_report_renderer import render_run_cost_report+from pipelex.reporting.cost_report_renderer import render_cost_report_for_outputfrom pipelex.system.runtime import IntegrationModefrom pipelex.system.telemetry.events import EventPropertyfrom pipelex.tools.misc.exceptions import JsonTypeError@@ -312,16 +312,11 @@ async def _execute_run(raise typer.Exit(1) from csv_exclog.verbose(f"Main stuff CSV saved to: {save_csv}")- # Render the end-of-run cost report from the usage assembled onto pipe_output (event-sourced),- # gated by the resolved --costs. Channels (console / CSV) follow the reporting config (D6).- # The renderer keeps its cost-domain name (`is_generate_costs`) on purpose; it is fed the usage- # gate (`is_generate_usage`) — "if usage was generated, render the cost view". Do not "align" the- # keyword to the field: the cost report is a view over usage, not the same switch.- render_run_cost_report(- pipeline_run_id=response.pipeline_run_id,- tokens_usages=pipe_output.tokens_usages,- is_generate_costs=execution_config.is_generate_usage,- )+ # Render the end-of-run cost report from the usage assembled onto pipe_output (event-sourced).+ # The gate is read off the output itself — pipe_output.tokens_usages is None exactly when cost+ # reporting was off for this run — so the submitter no longer re-derives it from config. Channels+ # (console / CSV) follow the reporting config (D6).+ render_cost_report_for_output(pipe_output)# Print completion recapconsole = get_console()diff --git a/pipelex/reporting/cost_report_renderer.py b/pipelex/reporting/cost_report_renderer.pyindex 3269f776c..cb34d84f1 100644--- a/pipelex/reporting/cost_report_renderer.py+++ b/pipelex/reporting/cost_report_renderer.py@@ -14,6 +14,7 @@ from pipelex import logfrom pipelex.base_exceptions import PipelexErrorfrom pipelex.cogt.usage.cost_registry import CostRegistryfrom pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.reporting.reporting_types import AnyTokensUsagefrom pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path@@ -80,3 +81,31 @@ def render_run_cost_report()except (OSError, UnicodeEncodeError, PipelexError) as cost_report_error:log.warning(f"Cost report generation failed (run succeeded): {cost_report_error}")+++def render_cost_report_for_output(pipe_output: PipeOutput) -> None:+ """Render the end-of-run cost report from a finished output (self-contained, one-arg submitter helper).++ The standard submitter convenience over :func:`render_run_cost_report`: a finished+ :class:`~pipelex.core.pipes.pipe_output.PipeOutput` already carries the three values the primitive+ needs, so callers (the ``pipelex run`` CLI, external embedders) no longer re-derive them by hand.++ Crucially, the ``--costs`` gate is read **off the output, not config**:+ :attr:`~pipelex.core.pipes.pipe_output.PipeOutput.tokens_usages` is ``None`` exactly when cost+ reporting was off for this run. The runner already resolved that decision at run time (with all+ ``--costs/--no-costs`` overrides applied) and recorded it on the output. Re-reading global config to+ reconstruct it would only be correct when config wasn't overridden or mutated between run and render —+ the output is authoritative, config is not.++ Outcome-identical to the historical CLI gate (``execution_config.is_generate_usage``): the primitive+ ANDs both guards (``not is_generate_costs or not tokens_usages``), so the cases where the two sources+ disagree (usage on but tracing off → ``tokens_usages is None``) no-op either way.++ Args:+ pipe_output: The finished run output carrying ``pipeline_run_id`` and ``tokens_usages``.+ """+ render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=pipe_output.tokens_usages is not None,+ )diff --git a/tests/unit/pipelex/reporting/test_render_cost_report_for_output.py b/tests/unit/pipelex/reporting/test_render_cost_report_for_output.pynew file mode 100644index 000000000..5ddec0b03--- /dev/null+++ b/tests/unit/pipelex/reporting/test_render_cost_report_for_output.py@@ -0,0 +1,66 @@+"""Unit tests for the one-arg submitter helper ``render_cost_report_for_output``.++The helper is a thin convenience over ``render_run_cost_report``: it unpacks the three primitive+arguments from a finished ``PipeOutput`` and — crucially — derives the ``--costs`` gate from the+output itself (``tokens_usages is None`` exactly when cost reporting was off), not from global config.+These tests pin that derivation and the no-op/render outcomes it produces.+"""++from pytest_mock import MockerFixture++from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.reporting.cost_report_renderer import render_cost_report_for_output++_RUN_ID = "output-helper-run"+++def _usage(job_metadata: JobMetadata) -> LLMTokensUsage:+ return LLMTokensUsage(+ job_metadata=job_metadata,+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},+ )+++class TestRenderCostReportForOutput:+ def _enable_channels(self, mocker: MockerFixture) -> None:+ reporting_config = get_config().pipelex.reporting_config+ mocker.patch.object(reporting_config, "is_log_costs_to_console", True)+ mocker.patch.object(reporting_config, "is_generate_cost_report_file_enabled", False)++ def test_no_op_when_tokens_usages_none(self, mocker: MockerFixture) -> None:+ """tokens_usages is None encodes 'cost reporting was off' -> the derived gate is False -> no render."""+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._enable_channels(mocker)++ render_cost_report_for_output(PipeOutput(pipeline_run_id=_RUN_ID, tokens_usages=None))++ spy.assert_not_called()++ def test_no_op_when_tokens_usages_empty(self, mocker: MockerFixture) -> None:+ """An empty list (costs on, no inference) is not None, but the primitive's `not tokens_usages` guard no-ops."""+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._enable_channels(mocker)++ render_cost_report_for_output(PipeOutput(pipeline_run_id=_RUN_ID, tokens_usages=[]))++ spy.assert_not_called()++ def test_renders_with_output_run_id_when_usage_present(self, mocker: MockerFixture, job_metadata: JobMetadata) -> None:+ """A non-empty tokens_usages derives the gate True and renders, titled with the output's own run id."""+ spy = mocker.patch("pipelex.reporting.cost_report_renderer.CostRegistry.render_report")+ self._enable_channels(mocker)++ render_cost_report_for_output(PipeOutput(pipeline_run_id=_RUN_ID, tokens_usages=[_usage(job_metadata)]))++ spy.assert_called_once()+ kwargs = spy.call_args.kwargs+ assert kwargs["pipeline_run_id"] == _RUN_ID+ assert kwargs["cost_report_file_path"] is Nonediff --git a/wip/registry/README.md b/wip/registry/README.mdindex 9a06f5a93..1ee07a0f8 100644--- a/wip/registry/README.md+++ b/wip/registry/README.md@@ -10,6 +10,10 @@ Shipped in **PR #967** on `fix/For-API-update`, with the companion rename **#968- [`cost-reporting-overview.html`](cost-reporting-overview.html) — **the illustrated overview. Start here.** TL;DR, the two-bugs-one-lifecycle framing, before/after architecture, the shared-transport `--costs` switch, the cross-worker flow, the critical diffs, and the deferred items.+## Shipped follow-ups++- [`cost-report-submitter-helper-handoff.md`](cost-report-submitter-helper-handoff.md) — ✅ the one-arg `render_cost_report_for_output(pipe_output)` submitter helper (the embedder-ergonomics facet): submitters render the standard cost report from a finished output in one call, with the `--costs` gate read off the output (`tokens_usages is None`) instead of re-read from config. Landed on `feature/Cost-report-helper`; the downstream cocode delegate collapse is still pending. Composes with — does not resolve — decisions #3/#6 below.+## Open follow-ups- [`deferred-followups.md`](deferred-followups.md) — the deliberate non-goals: the costs-only in-memory tracer skip (E1 deep half), cost-per-node correlation, the A1 `TraceContext.from_execution_config` factory, T5, T3 request-scoped tracing state, the costs-only event-log explicit-close cleanup (PR #977 re-review), `--mock-inference` coverage, and the leftover `graph_id → run_id` rename + shelved `TraceContext` split.diff --git a/wip/registry/cost-report-submitter-helper-handoff.md b/wip/registry/cost-report-submitter-helper-handoff.mdnew file mode 100644index 000000000..2a894f5c2--- /dev/null+++ b/wip/registry/cost-report-submitter-helper-handoff.md@@ -0,0 +1,86 @@+# Handoff — one-arg `render_cost_report_for_output(pipe_output)` submitter helper++**Status: ✅ Shipped** (`feature/Cost-report-helper`). `render_cost_report_for_output` is in `pipelex/reporting/cost_report_renderer.py`, the `pipelex run` CLI (`_run_core.py`) uses it, `render_run_cost_report` is unchanged as the primitive, unit tests pin the gate-from-output derivation (`tests/unit/pipelex/reporting/test_render_cost_report_for_output.py`), and the as-built overview + `docs/features/cost-tracking.md` document the surface. Downstream cocode collapse is still pending (re-pin `pipelex`, then delete cocode's local `_render_cost_report` gate). The rationale below is kept because it composes with the still-open decisions #3/#6 in `cost-report-deferred-decisions.md`.++**Task:** add a thin, self-contained cost-report rendering helper to `pipelex/reporting/cost_report_renderer.py` that takes only a finished `PipeOutput`, so every "submitter" (the `pipelex run` CLI, and external embedders like cocode) stops re-deriving the same three arguments — and stops re-reading global config to reconstruct a decision the run already recorded on the output.++This is the **submitter-ergonomics facet** of the same theme as the deferred decisions in this registry (`cost-report-deferred-decisions.md` #3/#6, and its cross-cutting note: "there is no single owner of *is cost reporting on for this run?*"). It does not resolve #3 or #6 — it is a smaller, additive convenience that should land on its own, and it composes cleanly with whatever those decisions settle.++## Where this came from++Bumping the `pipelex` dependency in the sibling `../cocode/` repo broke its build: the new `ReportingProtocol` (post the event-sourced cost-report rework, PR #967) no longer has `generate_report()`. cocode migrated its call sites to `render_run_cost_report(...)`, and to do so it had to write this glue (`../cocode/cocode/swe/swe_cmd.py`, `_render_cost_report`):++```python+render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=get_config().pipelex.pipeline_execution_config.is_generate_usage,+)+```++Two smells fall out of that:++1. **Duplication.** All three arguments are derivable from a finished run. The `pipelex run` CLI writes the same unpacking by hand (`pipelex/cli/commands/run/_run_core.py:320`). Every embedder that wants the standard cost report has to reproduce it.+2. **Wrong source of truth for the gate.** cocode re-reads global config (`is_generate_usage`) to reconstruct whether costs were on — but the runner already made that decision at run time, with all `--costs/--no-costs` override resolution applied, and **recorded it on the output**: `PipeOutput.tokens_usages` is `None` precisely when cost reporting was off. See the field docstring at `pipelex/core/pipes/pipe_output.py:28-32`:++ > the submitter renders the cost report from this field. None when cost reporting was off or the run emitted no trace events at all; an empty list when on with events present but no inference happened.++ Reading config afterwards re-derives a fact the output already carries — correct only when config wasn't overridden and wasn't mutated between run and render. The output is authoritative; config is not.++## Proposed change++Add a free function next to `render_run_cost_report` (do **not** add a method on `PipeOutput` — that would couple the core model to the renderer's deps: config, `CostRegistry`, Rich; reporting may depend on core, not the reverse):++```python+def render_cost_report_for_output(pipe_output: PipeOutput) -> None:+ """Render the end-of-run cost report from a finished output (self-contained).++ The gate is read off the output, not config: pipe_output.tokens_usages is None+ exactly when cost reporting was off for this run (see PipeOutput.tokens_usages).+ """+ render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=pipe_output.tokens_usages is not None,+ )+```++Keep `render_run_cost_report(*, pipeline_run_id, tokens_usages, is_generate_costs)` exactly as-is — it stays the **low-level primitive** for the cases where the three values genuinely come from different places: the agent CLI's machine-readable envelope path (`build_cost_summary`, `pipelex/cli/agent_cli/commands/run/_run_core.py`) and any distributed/Temporal reassembly where the output isn't the single carrier.++Then collapse the `pipelex run` call site (`_run_core.py:320`) to `render_cost_report_for_output(pipe_output)`.++## Why deriving the gate from the output is behaviorally identical to the current CLI gate++`render_run_cost_report` already ANDs both guards (`if not is_generate_costs or not tokens_usages: return`, `cost_report_renderer.py:50`), so for every run produced by the standard runner the *rendered outcome* is unchanged when the gate is read from `tokens_usages` instead of from `execution_config.is_generate_usage`:++- usage on, inference happened → `tokens_usages` non-empty → render (both agree).+- usage on, dry run (no inference) → `tokens_usages == []` → `is not None` True, then `has_reportable_usage` suppresses (both agree).+- usage on but tracing disabled (decision **#3**) → `tokens_usages is None` → no render. The current CLI gate is `True` here but the renderer no-ops on `not tokens_usages` anyway → same outcome. **The helper inherits #3, it does not worsen it.**+- usage off → `tokens_usages is None` → no render (both agree).++So this is a pure de-duplication + correct-source-of-truth change, not a behavior change.++## Scope / non-goals++- **Do not** fold in decisions #3 (cost reporting coupled to `tracing_config.is_enabled`) or #6 (agent-CLI vs main-CLI gate divergence). Those are separate deliberate calls tracked in `cost-report-deferred-decisions.md`. This helper is render-surface only (console/CSV) and is orthogonal to the agent CLI's `build_cost_summary` JSON path.+- **Do not** change the signature of `render_run_cost_report`.+- **Do not** add a `PipeOutput` method (layering — see above).+- This is complementary to the cross-cutting note's "one resolved predicate on `PipelineExecutionConfig`": even after that lands, the *render-time* consumer should read the resolved decision off the output, which already records it.++## Read first++- `cost-reporting-overview.html` — as-built architecture (start here for the event-sourced model).+- `cost-report-deferred-decisions.md` — #3 and #6, the adjacent gate-source decisions; this handoff is the third, smaller facet.+- `deferred-followups.md` — T3 request-scoped tracing state (the broader "single owner" refactor this rolls toward).++## Acceptance criteria++- `render_cost_report_for_output(pipe_output)` added to `pipelex/reporting/cost_report_renderer.py`; `render_run_cost_report` unchanged.+- `pipelex run` (`_run_core.py`) uses the new helper; no behavior change to its console/CSV output (verify with a real run under `--costs`, `--no-costs`, `--dry-run`, and a free/zero-price model).+- No new import cycle (`reporting` → `core.pipe_output` is fine; `core.pipe_output` must not import the renderer).+- Docs updated in `pipelex/docs/` where the cost-report rendering surface is described.+- `make check` / `make test` green.++## Downstream (sibling repo, not part of this change)++Once shipped and pipelex is re-pinned in `../cocode/`, cocode's `_render_cost_report` collapses to a one-line delegate to `render_cost_report_for_output(...)` and then deletes — its current config-reading gate goes away. No action needed in pipelex; noting it so the API shape stays embedder-friendly (the one-arg form is the whole point).diff --git a/wip/registry/cost-reporting-overview.html b/wip/registry/cost-reporting-overview.htmlindex 0bc46ecda..729e03dee 100644--- a/wip/registry/cost-reporting-overview.html+++ b/wip/registry/cost-reporting-overview.html@@ -391,6 +391,15 @@ sequenceDiagramexcept (OSError, UnicodeEncodeError, PipelexError) as e:log.warning(f"Cost report generation failed (run succeeded): {e}")</code></pre><p class="mini">A cost-report failure never fails an otherwise-successful run — aggregation, CSV setup, and render all sit inside one run-succeeded <code>try</code>. The agent CLI takes a different path: no Rich table, a machine-readable <code>cost_report</code> object on its <code>--with-memory</code> JSON envelope instead.</p>+ <p>Submitters don't call <code>render_run_cost_report</code> directly. They call the one-arg convenience <code>render_cost_report_for_output(pipe_output)</code>, which unpacks the three primitive arguments from a finished <code>PipeOutput</code> — and, crucially, <strong>derives the <code>--costs</code> gate from the output itself</strong>: <code>pipe_output.tokens_usages is None</code> is exactly the signal that cost reporting was off for this run. The runner already resolved that decision at run time (with all <code>--costs/--no-costs</code> overrides applied) and recorded it on the output, so re-reading global config afterwards would only re-derive — incorrectly, if config was overridden or mutated between run and render — a fact the output already carries authoritatively.</p>+ <span class="filename">pipelex/reporting/cost_report_renderer.py</span>+ <pre class="code"><code>def render_cost_report_for_output(pipe_output: PipeOutput) -> None:+ render_run_cost_report(+ pipeline_run_id=pipe_output.pipeline_run_id,+ tokens_usages=pipe_output.tokens_usages,+ is_generate_costs=pipe_output.tokens_usages is not None, # gate read off the output, not config+ )</code></pre>+ <p class="mini">This is a pure de-dup + correct-source-of-truth change, not a behavior change: the primitive ANDs both guards (<code>not is_generate_costs or not tokens_usages</code>), so the one case where output and config disagree (costs on but tracing off ⇒ <code>tokens_usages is None</code> — decision #3) no-ops either way. <code>render_run_cost_report</code> stays the low-level primitive for paths where the three values genuinely come from different places: the agent CLI's <code>build_cost_summary</code> JSON envelope, and any distributed reassembly where the output isn't the single carrier. The <code>pipelex run</code> CLI and external embedders (e.g. cocode) collapse to the one-arg form.</p></div></details>