Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Summary-events · L-260927-c49be2 (#1394)
Louis Choquelffa947e10cdc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0d8ef44ba..755cc909f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,11 +6,14 @@- **The console shows a log call's fields and renders named layouts**: the `console` sink prints a record's fields after its message as a `key=value` suffix, `Scanned the inputs files=7`, each value on one line and cut short past 80 characters, a handled exception's `error.message` only past 2000, leaving out the run identifiers, and colours a value by its field's name, a pipe code red and a concept bold green, with any other field dimmed. A call can name a console layout with `layout=`, a Rich template over its fields in which every value is escaped, starting with `LogLayout.PIPE_RUN` for the pipe-run tree; every other sink writes the plain message and the fields.- **Log-call conventions and their guard**: the logging documentation states how Pipelex's own log calls are written, a fixed message with its values in `fields` named from a shared vocabulary, an exception carried by the record rather than spliced into the text, no run identifier, payload, markup or emoji, and what each level promises. `make check-log-calls` (`pipelex-dev check-log-calls`), part of `make agent-check` and CI, refuses an interpolated or non-literal message at `INFO` and above and a Rich markup tag in a message at any level, against a committed baseline of the existing calls that only shrinks: `--against <ref>` and `--against-merge-base <ref>` refuse an entry the baseline at the base revision does not list, which CI runs against a pull request's base and `make` against the merge base with `origin/dev`. `pipelex.tools.log.error_fields` gives a handled exception's class and text as the `error.type` and `error.message` fields of a warning, which carries no traceback.+- **Summary events for inference calls and pipe runs**: every inference call, an LLM call, an image generation, an extraction, a search, a judgment or a document print, a call its checks refuse included, ends with one `INFO` event, `Inference call ends`, carrying the operation under `gen_ai.operation.name`, the model under `model_handle`, `backend_name`, `sdk`, `gen_ai.request.model` and `gen_ai.response.model`, the tokens under `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` and the price under `cost_usd` where the call recorded them, `duration_ms`, and `outcome`, `success`, `error` with `error.type`, or `cancelled` for work stopped from outside, logged by the worker bases so every provider's and plugin's worker gets it. Every live pipe run ends with `Pipe run ends`, logged as the pipe's own span closes and carrying its ids, with the announcement's fields, `duration_ms`, `outcome` and `error.type`, so a run's `json` output alone gives latency and cost per operation and model; an event that cannot be built is replaced by the warning `A summary event could not be logged` and never changes the work's outcome. The console draws each on one line through `LogLayout.INFERENCE_CALL_END` and `LogLayout.PIPE_RUN_END`; a document engine overrides `render` alone, and the print stage that calls it logs the event.+- **Request- and page-priced usage**: a usage says what its counts are in `pricing_unit` (`PricingUnit` in `pipelex.cogt.usage.pricing_unit`), `token`, `request` or `page`, and a worker billed by the request or by the page records it with `record_unit_priced_usage` (`pipelex.cogt.usage.usage_cost`), as the Linkup workers and the extraction page fallback now do, so the summary event carries the call's cost and no token counts.### Changed- **Library-loading warnings carry their values as fields**: the warnings logged while a library loads, for an unparseable `METHODS.toml`, an unsatisfied or unparseable `mthds_version`, a dependency with no valid bundle or a pipe that would not load, an unresolvable method package and the teardown of a stale library, each have one fixed message and carry the package address, the dependency alias, the pipe code and the error as fields, where they used to splice those values into the text.- **The pipe announcement is a plain message with fields**: every live pipe run logs `Pipe run starts` with `pipe_type`, `pipe_code`, `output_concept` and `pipe_depth` as fields, where the `json`, `otlp` and `gcp` sinks used to receive Rich markup inside the message, and the console draws the same coloured pipe tree from those fields through `LogLayout.PIPE_RUN`. The warning that a model's fixed temperature replaced the requested one carries its values as fields in the same way: `model_handle`, `backend_name`, `sdk`, the model's handle under `gen_ai.request.model` and the provider's model id under `gen_ai.response.model` as on the LLM call's span, the requested temperature under `gen_ai.request.temperature`, and `fixed_temperature`; the verbose line saying a model's settings were not checked, since its backend's SDK is not installed, carries the same model fields.+- **The default output logs every inference call and every live pipe run**: at the default `INFO` level an application embedding pipelex now gets an `Inference call ends` line after each inference call and a `Pipe run ends` line after each live pipe run. To quiet either, raise its logger's level in `[runtime.log.package_log_levels]`: `pipelex-cogt-inference-inference_call_summary = "WARNING"` for the inference event, and `pipelex-pipe_machinery-pipe_abstract = "WARNING"` for the pipe-run end, which quiets the pipe announcement too, since the two share that logger.### Fixed@@ -19,6 +22,7 @@### Removed+- **Cost helpers outside `pipelex.cogt.usage.usage_cost` (Breaking)**: `compute_tokens_usage_cost` is imported from `pipelex.cogt.usage.usage_cost` rather than `pipelex.cogt.usage.cost_registry`, `CostRegistry.compute_total_cost` is gone for `compute_total_cost` in the same module, and the `TokensUsage` alias of `pipelex.reporting.reporting_types` is gone: import `TokensUsage` from `pipelex.cogt.usage.usage_cost`, where it is a plain union of the usage types without the old discriminator, or use the discriminated `AnyTokensUsage`, which stays in `pipelex.reporting.reporting_types`.- **`is_markup_enabled` (Breaking)**: the console reads no log message as Rich markup, so the `[runtime.log.rich_log]` key that switched it is gone, along with `VERBATIM_MARK` in `pipelex.tools.log.log_fields`. A message now prints exactly as written, so a `list[int]`, a bracketed path, the `[myapp.jobs.nightly]: ` prefix of a logger named in lowercase and the `[cycle]` marker of a circular content no longer vanish, while a message of your own that relied on markup for its colour shows its tags and should put its values in `fields`. **Migration:** run `pipelex migrate`, which deletes the key from the file on disk (ledger entry `pipelex-config@7`).## [v0.80.0] - 2026-10-09diff --git a/docs/tools/logging.md b/docs/tools/logging.mdindex ce6f22954..70a96628c 100644--- a/docs/tools/logging.md+++ b/docs/tools/logging.md@@ -17,7 +17,7 @@ Pipelex uses the standard Python log levels and adds two of its own, `VERBOSE` b|-------|-------|---------------------------------|| VERBOSE | 5 | Step-level tracing for Pipelex's own developers, following the runtime through its steps || DEBUG | 10 | What you would need to diagnose a problem from a log a user sends |-| INFO | 20 | A lifecycle milestone, and there are few of them |+| INFO | 20 | A lifecycle milestone, and there are few of them, besides the [summary event](#summary-events) that ends each inference call and each live pipe run || WARNING | 30 | A handled degradation, or a user misconfiguration worth fixing; a warning is actionable || ERROR | 40 | A run failed or data was lost || CRITICAL | 50 | The process cannot go on |@@ -280,6 +280,8 @@ The registry holds these layouts:| Layout | Fields it presents | What it draws || --- | --- | --- || `LogLayout.PIPE_RUN` | `pipe_type`, `pipe_code`, `output_concept`, `pipe_depth` (an integer, `0` at the top level and at most `100`) | `PipeCompose: compose_company → Company`, indented three spaces per level of depth and behind `↳` when nested, in the style map's colours. A depth of another type, or out of range, falls back to the message. The pipe announcement names it, and only a live run announces itself, so the line carries no run mode |+| `LogLayout.PIPE_RUN_END` | The pipe-run layout's fields, `duration_ms` and `outcome` | `PipeLLM: describe_company done in 1.25 s`, at the depth and in the colours of the line that announced the run, or `failed after 1.25 s` in red with `error.type` following as the suffix, or `cancelled after 1.25 s` in yellow. A duration is written in milliseconds under a second and in seconds from a second on. An unknown outcome, or a duration that is no number or is negative, falls back to the message. The [pipe-run end event](#the-pipe-run-end) names it |+| `LogLayout.INFERENCE_CALL_END` | Every field of the [inference summary event](#the-inference-call-end) but `error.type` | `claude-5.5-sonnet chat · 2,048 → 512 tokens · $0.009216 · done in 1.23 s`: the model's handle, the operation, the tokens in and out, the cost and the duration, or `failed after` in red with `error.type` following as the suffix, or `cancelled after` in yellow. A count or a cost the call did not report is left out with its separator, and a cost under a millionth of a dollar is written `<$0.000001`, so it never reads as a free call's `$0`. The keys naming the model a second way, `gen_ai.request.model`, `gen_ai.response.model`, `backend_name` and `sdk`, are presented and not drawn, so the line stays short and the other sinks carry them. A count, a cost or a duration of the wrong type, or an unknown outcome, falls back to the message |A layout is a `ConsoleLayout`: a `template` of Rich markup whose placeholders are bare field names (`{pipe_code}`, never an attribute, an index or a format spec, which registration refuses), the `presented_fields` the suffix leaves out, and, for a shape a template cannot express alone, a subclass whose `derived_values` computes presentation values such as an indentation from the fields.@@ -308,6 +310,96 @@ Optional inclusion of caller information in logs, prefixed to the console line:- Module name- Customizable format templates+## Summary events++Two kinds of work end with one event each, whichever way they end: an inference call and a live pipe run. Each event has a fixed message, carries what the work was, how long it took and how it ended as fields, and is logged once, when the work returns or raises, so a log store selects every one by its message and groups them by their fields. The event is the work's measurement and never part of its result: if its own fields cannot be built or logged, it is replaced by one WARNING, `A summary event could not be logged`, naming the event in `summary_event` with the failure's `error.type` and `error.message`, and the work returns or raises exactly as it would have. A handler that raises on the event, as a host's might on a full disk, gets that warning next, and when it refuses the warning too nothing more is logged: the failure of a log line never reaches the work. They are what an operator's dashboards are built from: a run's `json` output alone gives the latency and the cost of each operation on each model, and the failures by their class, with no span store and no text to parse.++Both are logged at **INFO**. The dashboards read the events at the default production level, and a query over a run's `json` output computes latency and cost per operation and model from these events alone only if they are logged at that level. The pipe-run end pairs with the pipe-run announcement, `Pipe run starts`, which is logged at INFO already. Each event has a console layout that draws it on one short line, so a terminal stays readable. So an application that embeds Pipelex at the default level gets one line per inference call and one per live pipe run. A host that wants either one quiet raises its logger's level with `package_log_levels`: `pipelex-cogt-inference-inference_call_summary = "WARNING"` for the inference event, and `pipelex-pipe_machinery-pipe_abstract = "WARNING"` for the pipe-run end, which quiets the announcement too, since the two share that logger.++The fields the two events share:++| Field | Type | What it says |+| --- | --- | --- |+| `duration_ms` | number, in milliseconds | How long the work took, read off a monotonic clock, to the microsecond |+| `outcome` | `success`, `error` or `cancelled` | Whether the work returned, failed, or was stopped from outside. An exception that is not an `Exception` stops work without the work failing, so it ends the work `cancelled`: a task cancelled, as a batch or a parallel controller cancels its siblings when one fails and as a client cancels a run, an interrupt, a generator closed or the interpreter exiting. So the siblings a failure cancels are not counted as failures of their own |+| `error.type` | string | The class of the exception the work failed with, when `outcome` is `error` only. The exception's text stays off the event: the exception propagates to whoever handles it, which logs it there |++### The inference call end++Every inference call ends with `Inference call ends`, on the logger `pipelex.cogt.inference.inference_call_summary`: an LLM call, the generation of an image, an extraction, a search, a judgment or the print of a document. It is logged by the worker bases, `LLMWorkerAbstract`, `ImgGenWorkerAbstract`, `ExtractWorkerAbstract`, `SearchWorkerAbstract` and `JudgmentWorkerAbstract`, around the call their public method makes, the checks that may refuse the call before the provider is reached included, so every worker that subclasses them, a provider's or a plugin's, ends its calls with the event, a refused call too, and logs nothing for it itself. A document engine, a `DocGenWorkerAbstract`, overrides `render` alone, which the print stage, `render_document_and_store`, runs on a thread of the print pool: the stage logs the event around the print and the checks before it, the run's read scope, the engine's model and its installation, so a refused print ends with it too, and it logs it on the coroutine that awaits the thread, so a print cancelled from outside ends `cancelled` once, while the thread, which nothing stops, finishes its render and logs nothing more. An LLM call's event is logged as the call's span closes, just after the span has ended but while it is still the active Pipelex span, so it carries that span under `pipelex.trace_id` and `pipelex.span_id`; a call the checks refuse ends before its span starts and carries none.++| Field | Type | What it says |+| --- | --- | --- |+| `gen_ai.operation.name` | string | The operation: `chat` for an LLM call, OpenTelemetry's name for it, and the family's own name where the conventions define none, `img_gen`, `extract`, `search`, `judgment` and `doc_gen` |+| `model_handle` | string | The model's handle, as the model deck names it |+| `backend_name` | string | The backend serving the model |+| `sdk` | string | The SDK the backend reaches the model through |+| `gen_ai.request.model` | string | The model requested, its handle, as on the LLM span |+| `gen_ai.response.model` | string | The provider's id of the model serving the call, as on the LLM span |+| `gen_ai.usage.input_tokens` | integer | The tokens the call read, when it reported them |+| `gen_ai.usage.output_tokens` | integer | The tokens the call wrote, when it reported them |+| `cost_usd` | number, in US dollars | What the call cost, when it reported usage and the model has rates |+| `duration_ms`, `outcome`, `error.type` | | As above |++A handle names one model per model type, so a query groups by `gen_ai.operation.name` and `model_handle` together. The model keys are read when the call ends, so a worker that names its model in the checks before the call is named on the event as on the call's span, and one that learns from the provider which model served the call is named after that model. A print that ended before its handle was resolved, refused by the read scope, or whose handle resolves to no model served here, names the model by its handle alone, under `model_handle` and `gen_ai.request.model`, and leaves off `backend_name`, `sdk` and `gen_ai.response.model`, which only a served model has. The usage is the one the call recorded on its job's report, the object the base hands to `ReportingProtocol.report_inference_job` when it reports the job and the reporting manager turns into a usage event, and the event prices it with `compute_tokens_usage_cost`, the cost engine the usage records and the run's cost report read, so a usage the cost report counts has the same price there as on the event. The event carries what the job recorded, though, and the cost report only what was reported: the search and judgment bases report a job whichever way it ends, while the LLM, image-generation and extraction bases report it only when it succeeds, so a failure of theirs the provider billed carries a cost on the event that the run's cost report does not count. A token count or a cost the call did not record is left off, never written as zero: a call that failed before the provider answered, or that the checks before it refused, recorded no usage, a model with no rates has no price, and a document engine reports no usage at all, so a print carries neither. A failure the provider billed, a search or a judgment whose answer a guard refused, carries the usage it recorded.++Rates are per million tokens, so a call billed by the request or by the page records each unit as a million tokens in and out, and the rate table prices one unit: a Linkup search or fetch is one request, and an extraction whose provider reports no usage is priced by its pages. Its usage says so in its `pricing_unit`, `request` or `page` rather than `token`, and the event carries the call's cost and no token counts, since a request or a page is not a token. A worker records such a usage with `record_unit_priced_usage` (`pipelex.cogt.usage.usage_cost`), a plugin's worker as much as a provider's, and never writes the counts itself.++```text+INFO 🧠: claude-5.5-sonnet chat · 2,048 → 512 tokens · $0.009216 · done in 1.23 s+INFO 🧠: claude-5.5-sonnet chat · failed after 2.40 s error.type=LLMCompletionError+```++```json+{"time": "2026-10-09T23:08:37.099Z", "severity": "INFO", "logger": "pipelex.cogt.inference.inference_call_summary", "message": "Inference call ends", "pipeline_run_id": "0b2319a6-c511-4e1f-80f6-914cbab5f28b", "pipe_run_id": "0b63ab15-ff93-4c4f-9a2d-4f1c4f9901d4", "gen_ai.operation.name": "chat", "model_handle": "claude-5.5-sonnet", "backend_name": "anthropic", "sdk": "anthropic", "gen_ai.request.model": "claude-5.5-sonnet", "gen_ai.response.model": "claude-sonnet-5-5", "gen_ai.usage.input_tokens": 2048, "gen_ai.usage.output_tokens": 512, "cost_usd": 0.009216, "duration_ms": 1234.512, "outcome": "success"}+```++On Pipelex's LLM span, `gen_ai.operation.name` still holds Pipelex's unit-job id, `llm_gen_text` or `llm_gen_object`, which the PostHog exporter reads back to name the span; the event carries OpenTelemetry's operation name under that key, and the span is to follow it. Every other `gen_ai.*` key means the same on both.++### The pipe-run end++Every live pipe run ends with `Pipe run ends`, on the logger `pipelex.pipe_machinery.pipe_abstract`, the announcement's. `PipeAbstract.live_run_pipe` enters it with the announcement, under the run's `pipe_run_id`, and logs it when the run returns or raises, so every announced run ends once, a nested one before the pipe it runs under, and a dry run, which announces nothing, ends with nothing. It is logged as the pipe's own span closes, just after the span has ended but while it is still the active Pipelex span, so with telemetry on it carries the span of the pipe that ended under `pipelex.trace_id` and `pipelex.span_id`, as the lines of the run do, a nested pipe's its own and never its parent's; a run whose setup fails before its span is active still ends with the event, outside any span of its own.++| Field | Type | What it says |+| --- | --- | --- |+| `pipe_type` | string | The pipe's class |+| `pipe_code` | string | The pipe |+| `output_concept` | string | The concept the pipe produces |+| `pipe_depth` | integer | How deep the run is nested, `0` at the top level, as in the announcement |+| `duration_ms`, `outcome`, `error.type` | | As above |++```text+INFO 🧠: PipeSequence: brief_company → Text+INFO 🧠: ↳ PipeLLM: describe_company → Text+INFO 🧠: claude-5.5-sonnet chat · 1,000 → 200 tokens · $0.004 · done in 1.23 s+INFO 🧠: ↳ PipeLLM: describe_company done in 1.25 s+INFO 🧠: PipeSequence: brief_company done in 1.31 s+```++```json+{"time": "2026-10-09T23:08:37.127Z", "severity": "INFO", "logger": "pipelex.pipe_machinery.pipe_abstract", "message": "Pipe run ends", "pipeline_run_id": "0b2319a6-c511-4e1f-80f6-914cbab5f28b", "pipe_run_id": "0b63ab15-ff93-4c4f-9a2d-4f1c4f9901d4", "pipe_type": "PipeLLM", "pipe_code": "describe_company", "output_concept": "Text", "pipe_depth": 1, "duration_ms": 1250.663, "outcome": "success"}+```++### Pricing a run from its events++A dashboard reads a run's `json` output, keeps the inference events and adds them up by operation and model:++```python+import json+from collections import defaultdict++calls = [record for record in map(json.loads, open("run.jsonl")) if record["message"] == "Inference call ends"]+groups = defaultdict(list)+for call in calls:+ groups[call["gen_ai.operation.name"], call["model_handle"]].append(call)+for (operation, model_handle), group in groups.items():+ cost_usd = sum(call.get("cost_usd", 0) for call in group)+ latency_ms = sum(call["duration_ms"] for call in group) / len(group)+ print(operation, model_handle, len(group), f"${cost_usd:.4f}", f"{latency_ms:.0f} ms")+```++A test runs a small method live, with stand-in workers answering every model call on two models, and computes the cost and the latency of each operation and model from the `json` output alone.+## Log-call conventionsPipelex's own log calls, in `pipelex/` and in the API server's `api/pipelex_api/`, follow these conventions, so that a line written today can be grouped, counted and filtered in a log store tomorrow. The [log-call guard](../contribute/log-calls.md), `make check-log-calls`, holds every call to the ones that can be read off the source, the fixed message at INFO and above and the absence of markup; the rest is for review. Code of your own that logs through `log` is welcome to follow them too.@@ -331,7 +423,7 @@ log.warning(### Values in fields-What varies goes in `fields`, named by the [naming convention](#naming-convention), and one concept has one name across the codebase, so that a query written for one line finds every line about the same thing. Pipelex's own concepts take the names in the table below. A concept the OpenTelemetry semantic conventions define takes their key verbatim: `file.path` for a path on disk, `url.full` for a URL, `user.id` for a user, `error.type` for the class of an error, and the `gen_ai.*` keys for inference, such as `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.request.temperature`, `gen_ai.operation.name`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. A `gen_ai.*` key means on a log line what it means on Pipelex's LLM span: `gen_ai.request.model` is the model's handle and `gen_ai.response.model` the provider's id of the model serving it. A line about an inference call carries them beside `model_handle`, `backend_name` and `sdk`, so it joins its span in a log store, while `model_handle` names the handle on every line about a model, inference or not. A count is named `<what>_count` (`concept_count`) and a duration `duration_ms`, in milliseconds.+What varies goes in `fields`, named by the [naming convention](#naming-convention), and one concept has one name across the codebase, so that a query written for one line finds every line about the same thing. Pipelex's own concepts take the names in the table below. A concept the OpenTelemetry semantic conventions define takes their key verbatim: `file.path` for a path on disk, `url.full` for a URL, `user.id` for a user, `error.type` for the class of an error, and the `gen_ai.*` keys for inference, such as `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.request.temperature`, `gen_ai.operation.name`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. A `gen_ai.*` key means on a log line what it means on Pipelex's LLM span: `gen_ai.request.model` is the model's handle and `gen_ai.response.model` the provider's id of the model serving it. `gen_ai.operation.name` is the one exception for now, as [the inference call end](#the-inference-call-end) explains. A line about an inference call carries them beside `model_handle`, `backend_name` and `sdk`, so it joins its span in a log store, while `model_handle` names the handle on every line about a model, inference or not. A count is named `<what>_count` (`concept_count`) and a duration `duration_ms`, in milliseconds.| Field | What it names || --- | --- |@@ -358,6 +450,10 @@ What varies goes in `fields`, named by the [naming convention](#naming-conventio| `backend_name` | The inference backend serving a model, as the backends configuration names it: `anthropic`, `bedrock` || `sdk` | The SDK a backend reaches a model through, as the backends configuration names it: `openai`, `bedrock_anthropic` || `fixed_temperature` | The one temperature a model accepts, as its constraints declare it, used in place of the one requested |+| `cost_usd` | What an inference call cost, in US dollars, as a number, as the cost engine prices the usage it reported |+| `duration_ms` | How long a unit of work took, in milliseconds, as a number |+| `outcome` | How a unit of work ended, `success`, `error` or `cancelled`, on the [summary events](#summary-events) |+| `summary_event` | The [summary event](#summary-events) a line is about, by its message: `Inference call ends` || `node_id` | A node of a run's execution graph, as the graph tracer names it || `env_var` | The name of an environment variable, never its value || `error.type`, `error.message` | A handled exception's class name and its text, as [Exceptions](#exceptions) describes |@@ -385,7 +481,7 @@ The [level table](#log-levels) states what a line at each level means, and choos- **ERROR** says a run failed or data was lost, and that somebody should look; it carries the exception with `include_exception=True` when there is one.- **WARNING** must be actionable: a degradation that was handled, or a misconfiguration the user can fix. A warning nobody can act on moves down to DEBUG, or up to ERROR when something was in fact lost.-- **INFO** marks a lifecycle milestone, and there are few of them, because Pipelex is also a library and a library is quiet at INFO.+- **INFO** marks a lifecycle milestone, and there are few of them, because Pipelex is also a library and a library says little at INFO. The [summary events](#summary-events) are the one exception in number, an event for each inference call and each live pipe run: they are INFO because the dashboards built from them read the default production level, and a host that wants Pipelex quieter at INFO raises their loggers' level, as that section shows.- **DEBUG** answers a question you would ask on reading a log a user sent: which file was read, which fallback was taken, why a model was left out.- **VERBOSE** follows the runtime step by step, for Pipelex's own developers.diff --git a/docs/under-the-hood/document-engine-plugins.md b/docs/under-the-hood/document-engine-plugins.mdindex 52f4625eb..629ee3c38 100644--- a/docs/under-the-hood/document-engine-plugins.md+++ b/docs/under-the-hood/document-engine-plugins.md@@ -122,7 +122,7 @@ def render(self, *, job: RenderJob, resources: RenderResources) -> RenderedDocum- **`RenderResources.load(uri=…, position=…)`** is how an engine reads a file its document names, such as an image or a stylesheet. It resolves `pipelex-storage://` keys through the run's storage provider, decodes `data:` URLs, fetches `https://` through the SSRF guard, and refuses what the run's read scope does not allow, a local path included, with `UriReadRefusedError`. An engine reads nothing any other way. It returns a **`LoadedResource`**: the file's bytes as `data`, and as `mime_type` the media type the file's source gives it, described below.- **`RenderedDocument`** holds the bytes. Their MIME type and suffix are the format's.-`render` is synchronous and runs on a thread of a print pool of its own, never on the event loop's default executor, which the engine's reads need, while `RenderResources.load` hands each read back to the event loop the print started from. An engine keeps no state from one print into the next. It signals a document it cannot print by raising `DocGenRenderError`, which passes through; anything else it raises, beyond a Pipelex error, is reported as a `DocGenRenderError` naming the engine and the file.+`render` is synchronous and runs on a thread of a print pool of its own, never on the event loop's default executor, which the engine's reads need, while `RenderResources.load` hands each read back to the event loop the print started from. The print stage that calls it ends the print with the event every inference call ends with, `Inference call ends`, carrying no tokens and no cost since an engine reports none (see [Summary events](../tools/logging.md#summary-events)), logged on the coroutine that awaits the thread, so a print cancelled from outside ends `cancelled` while the thread finishes its render; an engine overrides `render` alone and logs nothing for the print itself. An engine keeps no state from one print into the next. It signals a document it cannot print by raising `DocGenRenderError`, which passes through; anything else it raises, beyond a Pipelex error, is reported as a `DocGenRenderError` naming the engine and the file.The contract is what an engine uses in `pipelex.cogt.doc_gen`: the worker; the render job, its resources, the `LoadedResource` they return and the rendered document (`render_job.py`); the formats and sources (`doc_gen_format.py`); `DocGenRenderError`, the one error of `exceptions.py` an engine raises; the layout tree (`layout_tree.py`); the template check request and the findings a checker returns (`template_check.py`), with the input shapes the request carries (`InputShape` and `InputShapeKind` in `input_shape.py`); and the two modules of the next section. The rest of the package is Pipelex's own side of the print. An engine reaches Pipelex only through the contract and the registration seam every plugin uses, which also hands a worker its model spec, its backend, its SDK clients and its reporting delegate: anything else in Pipelex may move or change in an ordinary release, while a breaking change to the contract moves `PLUGIN_API_VERSION`. A piece an engine is missing is added to the contract, rather than imported from elsewhere in Pipelex.diff --git a/docs/under-the-hood/inference-backend-plugins.md b/docs/under-the-hood/inference-backend-plugins.mdindex c22934bbd..e7096ac85 100644--- a/docs/under-the-hood/inference-backend-plugins.md+++ b/docs/under-the-hood/inference-backend-plugins.md@@ -271,10 +271,14 @@ What an out-of-tree backend plugin imports *is* the contract. The published surf| `DocGenWorkerAbstract` | `pipelex.cogt.doc_gen.doc_gen_worker_abstract` | the document-generation worker contract || `RenderJob`, `RenderedDocument`, `RenderResources`, `LoadedResource` | `pipelex.cogt.doc_gen.render_job` | the job a document engine prints, the file it returns, and how it reads a file the document names, with that file's media type || `NbTokensByCategoryDict`, `TokenCategory` | `pipelex.cogt.usage.token_category` | the token usage a worker reports |+| `PricingUnit` | `pipelex.cogt.usage.pricing_unit` | what a usage counts: tokens, or requests or pages priced as tokens |+| `record_unit_priced_usage` | `pipelex.cogt.usage.usage_cost` | records a call billed by the request or by the page on its usage || `BaseModelTypeVar` | `pipelex.tools.typing.pydantic_utils` | the schema type variable of a structured-output signature || `CogtError`, `SdkTypeError`, `ImgGenGenerationError`, `ImgGenParameterError`, `InferenceErrorCategory` | `pipelex.cogt.exceptions` | the inference error bases and categories a worker raises with || `LLMCapabilityError` | `pipelex.cogt.exceptions` | a request the model refuses, raised by an LLM request check |+A worker implements the provider half of a call, `_gen_text` or `_extract_pages` for instance, and records the usage the provider answered with on its job's report. The base around it logs the event every call ends with, `Inference call ends`, with that usage, its cost, its duration and its outcome, so a worker logs nothing for the call itself (see [Summary events](../tools/logging.md#summary-events)). A provider that bills by the request or by the page rather than by the token is recorded with `record_unit_priced_usage(tokens_usage=…, pricing_unit=PricingUnit.REQUEST, nb_units=1)`, which writes each unit as a million tokens in and out so the per-million rates price one unit, and marks the usage so the event keeps the call's cost and writes no token counts; a worker never writes those counts itself.+**Building on an OpenAI- or Anthropic-compatible service**| Symbol | Module | Role |diff --git a/docs/under-the-hood/per-node-usage-attribution.md b/docs/under-the-hood/per-node-usage-attribution.mdindex 2797e55d8..cde8de131 100644--- a/docs/under-the-hood/per-node-usage-attribution.md+++ b/docs/under-the-hood/per-node-usage-attribution.md@@ -14,7 +14,7 @@ Every inference call emits a `UsageReportEvent` carrying `node_id` (from `trace_- **`UsageAggregator`** (`pipelex/tracing/usage_aggregator.py`) projects it into a flat list — one record per call, `node_id` dropped. That list becomes `pipe_output.tokens_usages`, trimmed to [`TokensUsageRecord`](tokens-usage-wire-records.md) at the client boundary, and feeds the console cost table and the CSV.- **`usage_attribution`** (`pipelex/tracing/usage_attribution.py`) projects it into per-node totals — `node_id` kept, calls folded together. Those totals become `NodeSpec.usage` and `GraphSpec.usage`, assembled by `GraphSpecAssembler`.-The two projections are deliberately separate — `tokens_usages` is a shipped, `extra="forbid"` client contract, and a graph is not the place to re-litigate its record shape. What they are *not* allowed to do is disagree on arithmetic, so both compute every dollar through the same `compute_tokens_usage_cost` (`pipelex/cogt/usage/cost_registry.py`) and define a token total the same way `AggregatedCosts.total_nb_tokens` does.+The two projections are deliberately separate — `tokens_usages` is a shipped, `extra="forbid"` client contract, and a graph is not the place to re-litigate its record shape. What they are *not* allowed to do is disagree on arithmetic, so both compute every dollar through the same `compute_tokens_usage_cost` (`pipelex/cogt/usage/usage_cost.py`) and define a token total the same way `AggregatedCosts.total_nb_tokens` does.```mermaidflowchart TB@@ -141,7 +141,7 @@ So usage is deliberately assembler-only. That gap is *asserted* rather than norm| `pipelex/tracing/usage_attribution.py` | `UsageAccumulator`, `roll_up`, `attribute_usage` || `pipelex/tracing/graphspec_assembler.py` | Folds `UsageReportEvent`, attributes in pass 2 || `pipelex/tracing/usage_aggregator.py` | The other projection — the flat `tokens_usages` list |-| `pipelex/cogt/usage/cost_registry.py` | `compute_tokens_usage_cost` — the one cost engine |+| `pipelex/cogt/usage/usage_cost.py` | `compute_tokens_usage_cost` — the one cost engine |## Related pagesdiff --git a/docs/under-the-hood/tokens-usage-wire-records.md b/docs/under-the-hood/tokens-usage-wire-records.mdindex 2b39d9431..f969faaca 100644--- a/docs/under-the-hood/tokens-usage-wire-records.md+++ b/docs/under-the-hood/tokens-usage-wire-records.md@@ -41,7 +41,7 @@ The record shape is also invariant to server-side observability settings: teleme## Cost semantics-`cost` is computed by `compute_tokens_usage_cost` (`pipelex/cogt/usage/cost_registry.py`) from the rates stamped on the internal record at inference time — the same cost engine and the same canonical total as the CLI cost table: input_non_cached + input_cached + output component costs, with the cached-discount fallback (cached input at 50% of the input rate when no explicit cached rate exists). Categories the cost engine excludes from totals (audio, reasoning, prediction) are excluded here too. A run-level total is the sum of per-record `cost` values.+`cost` is computed by `compute_tokens_usage_cost` (`pipelex/cogt/usage/usage_cost.py`) from the rates stamped on the internal record at inference time — the same cost engine and the same canonical total as the CLI cost table: input_non_cached + input_cached + output component costs, with the cached-discount fallback (cached input at 50% of the input rate when no explicit cached rate exists). Categories the cost engine excludes from totals (audio, reasoning, prediction) are excluded here too. A run-level total is the sum of per-record `cost` values.`cost: null` means "unrated" (no rate table: own-GPU models, mock/dry runs) — distinct from `0.0`, which is a rated call that cost nothing.diff --git a/pipelex/cogt/content_generation/doc_gen_generate.py b/pipelex/cogt/content_generation/doc_gen_generate.pyindex b0e7a84d9..aa13bcbb4 100644--- a/pipelex/cogt/content_generation/doc_gen_generate.py+++ b/pipelex/cogt/content_generation/doc_gen_generate.py@@ -18,6 +18,7 @@ import asyncioimport contextvarsimport threadingfrom concurrent.futures import ThreadPoolExecutor+from typing import TYPE_CHECKINGfrom typing_extensions import override@@ -30,12 +31,16 @@ from pipelex.cogt.doc_gen.doc_gen_engine import get_doc_gen_inference_model, reqfrom pipelex.cogt.doc_gen.doc_gen_worker_factory import DocGenWorkerFactoryfrom pipelex.cogt.doc_gen.exceptions import DocGenRenderErrorfrom pipelex.cogt.doc_gen.render_job import LoadedResource, RenderedDocument, RenderResources+from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperationfrom pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.runtime_hub import get_report_delegatefrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstractfrom pipelex.tools.uri.uri_bytes import load_bytes_and_mime_type_from_any_urifrom pipelex.tools.uri.uri_read_scope import authorize_uri_read+if TYPE_CHECKING:+ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+# How long an engine may wait for one file it reads, such as an image fetched over https._RESOURCE_LOAD_TIMEOUT_SECONDS = 120# How many documents print at once; more wait their turn. A print is CPU-bound apart from its reads.@@ -83,8 +88,8 @@ async def render_document_and_store() -> DocumentContent:"""Print the composed document with its engine, store the file, and return the `DocumentContent` pointing at it.- The read scope is authorized first, then the DRY branch, which prints and stores nothing, so a dry run- refuses what a live one would.+ The read scope is authorized first in either mode, so a dry run, which prints and stores nothing, refuses what+ a live one would. A live print ends with the event every inference call ends with, as `_print_document` says.Raises:UriReadRefusedError: an image the document names is outside the run's read scope.@@ -93,48 +98,82 @@ async def render_document_and_store(no installed plugin registers it.DocGenRenderError: the engine could not print it, or failed in a way it did not report."""- authorize_assignment_reads(job_metadata=render_assignment.job_metadata, uri_references=render_assignment.referenced_uris())if render_assignment.cogt_run_params.run_mode.is_dry:+ authorize_assignment_reads(job_metadata=render_assignment.job_metadata, uri_references=render_assignment.referenced_uris())return dry_render_document(render_assignment)+ rendered = await _print_document(render_assignment=render_assignment, storage_provider=generated_content_factory.storage_provider)composition = render_assignment.composition- inference_model = get_doc_gen_inference_model(- model_handle=render_assignment.doc_gen_setting.model,- doc_gen_format=composition.format,- source=composition.source,- pipe_code=render_assignment.job_metadata.pipe_code,- )- require_doc_gen_engine_installed(- inference_model=inference_model,- doc_gen_format=composition.format,- source=composition.source,- pipe_code=render_assignment.job_metadata.pipe_code,- )- worker = DocGenWorkerFactory.make_doc_gen_worker(inference_model=inference_model, reporting_delegate=get_report_delegate())- render_job = composition.make_render_job()- resources = RunRenderResources(- storage_provider=generated_content_factory.storage_provider,- read_scope=render_assignment.job_metadata.run_metadata.read_scope,- loop=asyncio.get_running_loop(),- )- # The context is carried into the thread, as `asyncio.to_thread` would, so the engine logs under the run.- run_context = contextvars.copy_context()-- def _print() -> RenderedDocument:- return run_context.run(worker.render, job=render_job, resources=resources)-- try:- rendered = await asyncio.get_running_loop().run_in_executor(_PRINT_EXECUTOR, _print)- except PipelexError:- # Already classified: a refused read, or a failure the engine reported as a `DocGenRenderError`.- raise- except Exception as exc:- # Dynamic plugin dispatch: an engine is plugin code whose exceptions cannot be enumerated, and anything- # else it raises is a failure to print this document.- msg = f"The engine '{inference_model.name}' could not print '{composition.filename}': {exc}"- raise DocGenRenderError(msg) from excreturn await generated_content_factory.make_document_content(storage_scope=render_assignment.job_metadata.run_metadata.storage_scope,data=rendered.data,mime_type=composition.format.mime_type,filename=composition.filename,)+++async def _print_document(*, render_assignment: RenderDocumentAssignment, storage_provider: StorageProviderAbstract) -> RenderedDocument:+ """Print the composed document with its engine, ending the print with the event every inference call ends with.++ The event is entered before the checks that may refuse the print, the read scope, the engine's model and its+ installation, so a refused print ends with it too, as a refused call of any other family does. It is logged on+ this coroutine rather than on the engine's thread, so its outcome is the awaiting coroutine's: a print cancelled+ from outside ends `cancelled`, once, while the thread, which nothing can stop, finishes its render and logs+ nothing more. An engine reports no usage, so the event carries no tokens and no cost, never a zero standing for+ either.++ Raises:+ UriReadRefusedError: an image the document names is outside the run's read scope.+ ModelNotFoundError: the assignment's engine is not a model served here.+ DocGenEngineMissingError: the engine is the built-in one or one of the plugin's and is no longer declared here, or+ no installed plugin registers it.+ DocGenRenderError: the engine could not print it, or failed in a way it did not report.+ """+ composition = render_assignment.composition+ model_handle = render_assignment.doc_gen_setting.model+ pipe_code = render_assignment.job_metadata.pipe_code+ # The event reads it when the print ends, so it names the model the handle resolved to, or the handle alone when+ # the print ended before it was resolved or because it resolved to none.+ inference_model: InferenceModelSpec | None = None+ with InferenceCallSummary(+ operation=InferenceOperation.DOC_GEN,+ model_handle=model_handle,+ read_inference_model=lambda: inference_model,+ read_tokens_usage=None,+ ):+ authorize_assignment_reads(job_metadata=render_assignment.job_metadata, uri_references=render_assignment.referenced_uris())+ inference_model = get_doc_gen_inference_model(+ model_handle=model_handle,+ doc_gen_format=composition.format,+ source=composition.source,+ pipe_code=pipe_code,+ )+ require_doc_gen_engine_installed(+ inference_model=inference_model,+ doc_gen_format=composition.format,+ source=composition.source,+ pipe_code=pipe_code,+ )+ engine_name = inference_model.name+ worker = DocGenWorkerFactory.make_doc_gen_worker(inference_model=inference_model, reporting_delegate=get_report_delegate())+ render_job = composition.make_render_job()+ resources = RunRenderResources(+ storage_provider=storage_provider,+ read_scope=render_assignment.job_metadata.run_metadata.read_scope,+ loop=asyncio.get_running_loop(),+ )+ # The context is carried into the thread, as `asyncio.to_thread` would, so the engine logs under the run.+ run_context = contextvars.copy_context()++ def _render() -> RenderedDocument:+ return run_context.run(worker.render, job=render_job, resources=resources)++ try:+ return await asyncio.get_running_loop().run_in_executor(_PRINT_EXECUTOR, _render)+ except PipelexError:+ # Already classified: a refused read, or a failure the engine reported as a `DocGenRenderError`.+ raise+ except Exception as exc:+ # Dynamic plugin dispatch: an engine is plugin code whose exceptions cannot be enumerated, and anything+ # else it raises is a failure to print this document.+ msg = f"The engine '{engine_name}' could not print '{composition.filename}': {exc}"+ raise DocGenRenderError(msg) from excdiff --git a/pipelex/cogt/content_generation/read_authorization.py b/pipelex/cogt/content_generation/read_authorization.pyindex a3a6ae0cf..2caa2a300 100644--- a/pipelex/cogt/content_generation/read_authorization.py+++ b/pipelex/cogt/content_generation/read_authorization.py@@ -5,8 +5,9 @@ assignment, and both orchestration modes call the same leaves — the in-process``ContentGenerator`` and the Temporal activities — so the leaves are the one place every read andevery hand-off to another reader share. Each leaf authorizes the URLs its assignment declares(``referenced_uris()``) as its first statement, before the dry-run branch, so a dry run refuses the-method a live run would, and before any worker is built. See :mod:`pipelex.tools.uri.uri_read_scope`-for the rule.+method a live run would, and before any worker is built. The document print authorizes first in each+branch instead, its live branch inside the summary event the print ends with, so a print the scope+refuses ends with that event too. See :mod:`pipelex.tools.uri.uri_read_scope` for the rule."""from pipelex.system.job_metadata import JobMetadatadiff --git a/pipelex/cogt/doc_gen/doc_gen_worker_abstract.py b/pipelex/cogt/doc_gen/doc_gen_worker_abstract.pyindex b3ca64b6e..b4c9c8d86 100644--- a/pipelex/cogt/doc_gen/doc_gen_worker_abstract.py+++ b/pipelex/cogt/doc_gen/doc_gen_worker_abstract.py@@ -18,7 +18,10 @@ class DocGenWorkerAbstract(InferenceWorkerAbstract):registers `reportlab`; the Pipelex document generation plugin registers the rest from outside this repository.A worker is made for each print, and `render` runs on a thread of the print pool, never on the event loop, so- an engine loads its library once per process, in the module that holds its worker, rather than per worker.+ an engine loads its library once per process, in the module that holds its worker, rather than per worker. An+ engine overrides `render` alone: the print stage that calls it, `render_document_and_store`, ends each print with+ the event every inference call ends with, on the coroutine that awaits the engine's thread, so an engine logs+ nothing for the print itself."""def __init__(self, *, inference_model: InferenceModelSpec, reporting_delegate: ReportingProtocol | None = None):diff --git a/pipelex/cogt/extract/extract_report.py b/pipelex/cogt/extract/extract_report.pyindex 2466aa8ab..a71c5d769 100644--- a/pipelex/cogt/extract/extract_report.py+++ b/pipelex/cogt/extract/extract_report.py@@ -4,6 +4,7 @@ from typing import Any, Literalfrom pydantic import BaseModelfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict+from pipelex.cogt.usage.pricing_unit import PricingUnitfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.system.job_metadata import JobMetadata@@ -69,3 +70,5 @@ class ExtractTokensUsage(BaseModel):unit_costs: CostsByCategoryDictinference_model_id: strnb_tokens_by_category: NbTokensByCategoryDict+ # What the counts are: tokens, or requests or pages recorded as a million tokens each so the rates price one.+ pricing_unit: PricingUnit = PricingUnit.TOKENdiff --git a/pipelex/cogt/extract/extract_worker_abstract.py b/pipelex/cogt/extract/extract_worker_abstract.pyindex 8bf3d4a47..926e73e54 100644--- a/pipelex/cogt/extract/extract_worker_abstract.py+++ b/pipelex/cogt/extract/extract_worker_abstract.py@@ -7,9 +7,11 @@ from pipelex.cogt.exceptions import CogtError, ExtractCapabilityError, ExtractInfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job import ExtractJobfrom pipelex.cogt.extract.extract_output import ExtractOutput+from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperationfrom pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory+from pipelex.cogt.usage.pricing_unit import PricingUnit+from pipelex.cogt.usage.usage_cost import record_unit_priced_usagefrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.system.job_metadata import UnitJobIdfrom pipelex.tools.misc.filetype_utils import IMAGE_FORMAT_KEY, describe_file_format, describe_format_keys, format_key_from_mime_type@@ -108,42 +110,49 @@ class ExtractWorkerAbstract(InferenceWorkerAbstract):)raise ExtractInputFormatError(msg)+ def _call_summary(self, *, extract_job: ExtractJob) -> InferenceCallSummary:+ """The event the call ends with, its model and its usage read off the worker and the job it reports when it ends."""+ return InferenceCallSummary(+ operation=InferenceOperation.EXTRACT,+ model_handle=self.inference_model.name,+ read_inference_model=lambda: self.inference_model,+ read_tokens_usage=lambda: extract_job.job_report.extract_tokens_usage,+ )+async def extract_pages(self,extract_job: ExtractJob,) -> ExtractOutput:- # Verify that the job is valid- extract_job.validate_before_execution()-- # Verify feasibility- self._check_can_perform_job(extract_job=extract_job)- # TODO: check can generate object (where it will be appropriate)-- # metadata- extract_job.job_metadata.unit_job_id = UnitJobId.EXTRACT_PAGES-- # Prepare job- extract_job.extract_job_before_start(inference_model=self.inference_model)-- # Execute job- try:- result = await self._extract_pages(extract_job=extract_job)- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)- raise-- # Populate page count as fallback usage (only if no real usage was reported)- if (extract_tokens_usage := extract_job.job_report.extract_tokens_usage) and not extract_tokens_usage.nb_tokens_by_category:- nb_tokens: NbTokensByCategoryDict = {- TokenCategory.INPUT: len(result.pages) * 1_000_000,- TokenCategory.OUTPUT: len(result.pages) * 1_000_000,- }- extract_tokens_usage.nb_tokens_by_category = nb_tokens-- # Report job- extract_job.extract_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=extract_job)+ # The call ends with its summary event whichever way it ends, a refusal by the checks below included+ with self._call_summary(extract_job=extract_job):+ # Verify that the job is valid+ extract_job.validate_before_execution()++ # Verify feasibility+ self._check_can_perform_job(extract_job=extract_job)+ # TODO: check can generate object (where it will be appropriate)++ # metadata+ extract_job.job_metadata.unit_job_id = UnitJobId.EXTRACT_PAGES++ # Prepare job+ extract_job.extract_job_before_start(inference_model=self.inference_model)++ # Execute job+ try:+ result = await self._extract_pages(extract_job=extract_job)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise++ # Price the call by its pages when the provider reported no usage+ if (extract_tokens_usage := extract_job.job_report.extract_tokens_usage) and not extract_tokens_usage.nb_tokens_by_category:+ record_unit_priced_usage(tokens_usage=extract_tokens_usage, pricing_unit=PricingUnit.PAGE, nb_units=len(result.pages))++ # Report job+ extract_job.extract_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=extract_job)return resultdiff --git a/pipelex/cogt/img_gen/img_gen_report.py b/pipelex/cogt/img_gen/img_gen_report.pyindex 65b424f83..e6dae170f 100644--- a/pipelex/cogt/img_gen/img_gen_report.py+++ b/pipelex/cogt/img_gen/img_gen_report.py@@ -4,6 +4,7 @@ from typing import Any, Literalfrom pydantic import BaseModelfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict+from pipelex.cogt.usage.pricing_unit import PricingUnitfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.system.job_metadata import JobMetadata@@ -69,3 +70,5 @@ class ImgGenTokensUsage(BaseModel):unit_costs: CostsByCategoryDictinference_model_id: strnb_tokens_by_category: NbTokensByCategoryDict+ # What the counts are: tokens, or requests or pages recorded as a million tokens each so the rates price one.+ pricing_unit: PricingUnit = PricingUnit.TOKENdiff --git a/pipelex/cogt/img_gen/img_gen_worker_abstract.py b/pipelex/cogt/img_gen/img_gen_worker_abstract.pyindex da6c9b83f..13057db41 100644--- a/pipelex/cogt/img_gen/img_gen_worker_abstract.py+++ b/pipelex/cogt/img_gen/img_gen_worker_abstract.py@@ -5,6 +5,7 @@ from typing_extensions import overridefrom pipelex.cogt.exceptions import CogtError, ImgGenParameterErrorfrom pipelex.cogt.image.generated_image import GeneratedImageRawDetailsfrom pipelex.cogt.img_gen.img_gen_job import ImgGenJob+from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperationfrom pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.reporting.reporting_protocol import ReportingProtocol@@ -48,33 +49,44 @@ class ImgGenWorkerAbstract(InferenceWorkerAbstract):)raise ImgGenParameterError(msg)+ def _call_summary(self, *, img_gen_job: ImgGenJob) -> InferenceCallSummary:+ """The event the call ends with, its model and its usage read off the worker and the job it reports when it ends."""+ return InferenceCallSummary(+ operation=InferenceOperation.IMG_GEN,+ model_handle=self.inference_model.name,+ read_inference_model=lambda: self.inference_model,+ read_tokens_usage=lambda: img_gen_job.job_report.img_gen_tokens_usage,+ )+async def gen_image(self,img_gen_job: ImgGenJob,) -> GeneratedImageRawDetails:- # Verify that the job is valid- img_gen_job.validate_before_execution()+ # The call ends with its summary event whichever way it ends, a refusal by the checks below included+ with self._call_summary(img_gen_job=img_gen_job):+ # Verify that the job is valid+ img_gen_job.validate_before_execution()- # Verify feasibility- self._check_can_perform_job(img_gen_job=img_gen_job)+ # Verify feasibility+ self._check_can_perform_job(img_gen_job=img_gen_job)- # metadata- img_gen_job.job_metadata.unit_job_id = UnitJobId.IMG_GEN_TEXT_TO_IMAGE+ # metadata+ img_gen_job.job_metadata.unit_job_id = UnitJobId.IMG_GEN_TEXT_TO_IMAGE- # Prepare job- img_gen_job.img_gen_job_before_start(inference_model=self.inference_model)+ # Prepare job+ img_gen_job.img_gen_job_before_start(inference_model=self.inference_model)- # Execute job- try:- result = await self._gen_image(img_gen_job=img_gen_job)- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)- raise+ # Execute job+ try:+ result = await self._gen_image(img_gen_job=img_gen_job)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise- # Report job- img_gen_job.img_gen_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=img_gen_job)+ # Report job+ img_gen_job.img_gen_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=img_gen_job)return result@@ -91,29 +103,31 @@ class ImgGenWorkerAbstract(InferenceWorkerAbstract):*,nb_images: int,) -> list[GeneratedImageRawDetails]:- # Verify that the job is valid- img_gen_job.validate_before_execution()-- # Verify feasibility- self._check_can_perform_job(img_gen_job=img_gen_job)-- # metadata- img_gen_job.job_metadata.unit_job_id = UnitJobId.IMG_GEN_TEXT_TO_IMAGE-- # Prepare job- img_gen_job.img_gen_job_before_start(inference_model=self.inference_model)-- # Execute job- try:- result = await self._gen_image_list(img_gen_job=img_gen_job, nb_images=nb_images)- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)- raise-- # Report job- img_gen_job.img_gen_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=img_gen_job)+ # The call ends with its summary event whichever way it ends, a refusal by the checks below included+ with self._call_summary(img_gen_job=img_gen_job):+ # Verify that the job is valid+ img_gen_job.validate_before_execution()++ # Verify feasibility+ self._check_can_perform_job(img_gen_job=img_gen_job)++ # metadata+ img_gen_job.job_metadata.unit_job_id = UnitJobId.IMG_GEN_TEXT_TO_IMAGE++ # Prepare job+ img_gen_job.img_gen_job_before_start(inference_model=self.inference_model)++ # Execute job+ try:+ result = await self._gen_image_list(img_gen_job=img_gen_job, nb_images=nb_images)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise++ # Report job+ img_gen_job.img_gen_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=img_gen_job)return resultdiff --git a/pipelex/cogt/inference/inference_call_summary.py b/pipelex/cogt/inference/inference_call_summary.pynew file mode 100644index 000000000..c4506638a--- /dev/null+++ b/pipelex/cogt/inference/inference_call_summary.py@@ -0,0 +1,161 @@+"""The one event an inference call ends with, whichever way it ends: what a dashboard of inference is built from.++Every worker base wraps the call its public method makes in an ``InferenceCallSummary``, the LLM, image-generation,+extraction, search and judgment bases alike, so a provider's worker and a plugin's, which subclass those bases and+implement only the provider half, end every call with the event and log nothing for it themselves. A document engine+is a synchronous worker that overrides ``render`` alone, so its print is wrapped by the print stage that calls it,+``render_document_and_store``, on the coroutine that awaits the engine's thread. The block starts before the checks+that may refuse the call, so a refused call ends with the event too. The event is logged once, when the call returns or+raises, at INFO, under the fixed message ``Inference call ends``, so a log store selects every call by that message+and groups them by its fields.++The fields are the call's dimensions. ``gen_ai.operation.name`` is the operation, OpenTelemetry's name where its+semantic conventions define one (``chat`` for an LLM call) and the inference family's own name otherwise. The model+rides under the ``gen_ai.*`` keys with the meaning Pipelex's LLM span gives them, the handle requested under+``gen_ai.request.model`` and the provider's id of the model serving it under ``gen_ai.response.model``, beside+``model_handle``, ``backend_name`` and ``sdk``. A handle names one model per model type, so a query groups by the+operation and the model together. Then the tokens in and out, the cost in US dollars, the duration in milliseconds+and the outcome, with ``error.type`` on failure, and ``cancelled`` for a call stopped from outside.++The usage is the one the call recorded on its job report, the ``*TokensUsage`` object the base hands to+``ReportingProtocol.report_inference_job`` when it reports the job, priced with ``compute_tokens_usage_cost``, the+cost engine the client-facing usage records and the run graph read, so a usage gets the same price on the event and+in the cost report. The event reads it whether or not the job was reported: a failure the provider billed carries the+cost it recorded even where the base reports only a success, so the event may price a call the cost report never+sees. A token count or a cost the call did not record is left off the event, never written as zero: a family that+reports no usage, document generation, carries neither, and a call that failed before the provider answered recorded+none. A usage billed by the request or by the page, a Linkup search or fetch, or an extraction whose provider reports+no usage and is priced by its pages, records each unit as a million tokens in and out so the rate table prices one,+and says so in its ``pricing_unit``: that call carries its cost and no token counts, since a request or a page is not+a token.+"""++from __future__ import annotations++from enum import StrEnum+from typing import TYPE_CHECKING, Any++from typing_extensions import override++from pipelex import log+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.cogt.usage.usage_cost import compute_tokens_usage_cost+from pipelex.system.telemetry.otel_constants import GenAISpanAttr+from pipelex.tools.log.console_layouts import LogLayout+from pipelex.tools.log.summary_event import SummaryEvent+from pipelex.tools.log.summary_fields import COST_USD_FIELD++if TYPE_CHECKING:+ from collections.abc import Callable++ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+ from pipelex.cogt.usage.usage_cost import TokensUsage++#: The message every inference call ends with; the call is in the fields.+INFERENCE_CALL_ENDS_MESSAGE = "Inference call ends"+++class InferenceOperation(StrEnum):+ """The values of ``gen_ai.operation.name`` on the event an inference call ends with.++ OpenTelemetry's operation name where its GenAI conventions define one for the call, and the inference family's+ own name otherwise. An LLM call is a chat completion. The conventions name no operation for generating an image+ outside a multimodal chat, for reading a document's text, for a search that answers with its sources, for a+ judgment or for printing a document, and their ``retrieval`` is a search of a vector store, so those take the+ family's name.+ """++ CHAT = "chat"+ IMG_GEN = "img_gen"+ EXTRACT = "extract"+ SEARCH = "search"+ JUDGMENT = "judgment"+ DOC_GEN = "doc_gen"+++class InferenceCallSummary(SummaryEvent):+ """Times one inference call and logs the event it ends with, once, when it ends, however it ends.++ A context manager around the call, the checks that may refuse it included: entering it starts the clock, and+ leaving it logs the event, the outcome read off the exception leaving the block, if any, with ``cancelled`` for a+ call stopped from outside. It never handles that exception, which goes on as it came, and a failure to build or log+ the event is logged as a warning in its place, never raised. A worker base whose call runs under a span enters+ ``ends_here`` around the block the span is active in, so the event is logged as the span closes and names it.++ The model is read when the call ends, as everything else on the event is, so the event names it as the call left+ it: a worker that routes the call in its checks, or learns from the provider which model served it, names that model.+ A call whose handle resolved to no model served here, a document print refused before its model was resolved or for+ want of one, names the model by its handle alone, and the keys only a served model has are left off.++ Args:+ operation: The call's operation, ``gen_ai.operation.name``.+ model_handle: The handle the call asks for its model by, which names the model on the event when it resolves+ to none served here.+ read_inference_model: Reads the model serving the call when it ends; ``None`` when the handle resolved to none.+ read_tokens_usage: Reads the usage the call recorded, from the job report the worker hands to the reporting+ path, when the call ends; ``None`` for a family that reports no usage.+ read_request_model: Reads ``gen_ai.request.model`` when the call ends, the model's handle unless the worker names+ it otherwise.+ read_response_model: Reads ``gen_ai.response.model`` when the call ends, the provider's id of the model unless the+ worker names it otherwise.+ """++ def __init__(+ self,+ *,+ operation: InferenceOperation,+ model_handle: str,+ read_inference_model: Callable[[], InferenceModelSpec | None],+ read_tokens_usage: Callable[[], TokensUsage | None] | None,+ read_request_model: Callable[[], str] | None = None,+ read_response_model: Callable[[], str] | None = None,+ ) -> None:+ super().__init__(message=INFERENCE_CALL_ENDS_MESSAGE)+ self._operation = operation+ self._model_handle = model_handle+ self._read_inference_model = read_inference_model+ self._read_tokens_usage = read_tokens_usage+ self._read_request_model = read_request_model+ self._read_response_model = read_response_model++ @override+ def _work_fields(self) -> dict[str, Any]:+ """The call's operation, its model under every key that names it, and the usage it recorded."""+ return {GenAISpanAttr.OPERATION_NAME: self._operation, **self._model_fields(), **self._usage_fields()}++ @override+ def _log_event(self, *, fields: dict[str, Any]) -> None:+ log.info(INFERENCE_CALL_ENDS_MESSAGE, fields=fields, layout=LogLayout.INFERENCE_CALL_END)++ def _model_fields(self) -> dict[str, Any]:+ """The model under every key that names it, read now; the handle alone for a model that resolved to none."""+ inference_model = self._read_inference_model()+ if inference_model is None:+ return {"model_handle": self._model_handle, GenAISpanAttr.REQUEST_MODEL: self._model_handle}+ return {+ "model_handle": inference_model.name,+ "backend_name": inference_model.backend_name,+ "sdk": inference_model.sdk,+ GenAISpanAttr.REQUEST_MODEL: self._read_request_model() if self._read_request_model is not None else inference_model.name,+ GenAISpanAttr.RESPONSE_MODEL: self._read_response_model() if self._read_response_model is not None else inference_model.model_id,+ }++ def _usage_fields(self) -> dict[str, Any]:+ """The tokens in and out and the cost the call recorded, each left out when it recorded none.++ A usage priced by the request or by the page counts units, not tokens, so it gives its cost and no counts.+ """+ tokens_usage = self._read_tokens_usage() if self._read_tokens_usage is not None else None+ if tokens_usage is None or not tokens_usage.nb_tokens_by_category:+ return {}+ nb_tokens_by_category = tokens_usage.nb_tokens_by_category+ usage_fields: dict[str, Any] = {}+ if tokens_usage.pricing_unit.counts_tokens:+ if TokenCategory.INPUT in nb_tokens_by_category:+ usage_fields[GenAISpanAttr.USAGE_INPUT_TOKENS] = nb_tokens_by_category[TokenCategory.INPUT]+ if TokenCategory.OUTPUT in nb_tokens_by_category:+ usage_fields[GenAISpanAttr.USAGE_OUTPUT_TOKENS] = nb_tokens_by_category[TokenCategory.OUTPUT]+ cost = compute_tokens_usage_cost(tokens_usage)+ if cost is not None:+ usage_fields[COST_USD_FIELD] = cost+ return usage_fieldsdiff --git a/pipelex/cogt/judgment/judgment_report.py b/pipelex/cogt/judgment/judgment_report.pyindex ca9496408..a4a3292ce 100644--- a/pipelex/cogt/judgment/judgment_report.py+++ b/pipelex/cogt/judgment/judgment_report.py@@ -4,6 +4,7 @@ from typing import Any, Literalfrom pydantic import BaseModelfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict+from pipelex.cogt.usage.pricing_unit import PricingUnitfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.system.job_metadata import JobMetadata@@ -77,3 +78,5 @@ class JudgmentTokensUsage(BaseModel):unit_costs: CostsByCategoryDictinference_model_id: strnb_tokens_by_category: NbTokensByCategoryDict+ # What the counts are: tokens, or requests or pages recorded as a million tokens each so the rates price one.+ pricing_unit: PricingUnit = PricingUnit.TOKENdiff --git a/pipelex/cogt/judgment/judgment_worker_abstract.py b/pipelex/cogt/judgment/judgment_worker_abstract.pyindex 046c3886d..29e4dff18 100644--- a/pipelex/cogt/judgment/judgment_worker_abstract.py+++ b/pipelex/cogt/judgment/judgment_worker_abstract.py@@ -5,6 +5,7 @@ from collections.abc import Mappingfrom typing_extensions import overridefrom pipelex.cogt.exceptions import CogtError, JudgmentAnswerMismatchError, JudgmentCapabilityError+from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperationfrom pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstractfrom pipelex.cogt.inference.prompt_file_checks import check_prompt_documents_are_read, check_prompt_images_are_imagesfrom pipelex.cogt.judgment.judgment_job import JudgmentJob@@ -37,32 +38,44 @@ class JudgmentWorkerAbstract(InferenceWorkerAbstract):def desc(self) -> str:return f"Judgment using {self.inference_model.desc}"+ def _call_summary(self, *, judgment_job: JudgmentJob) -> InferenceCallSummary:+ """The event the call ends with, its model and its usage read off the worker and the job it reports when it ends."""+ return InferenceCallSummary(+ operation=InferenceOperation.JUDGMENT,+ model_handle=self.inference_model.name,+ read_inference_model=lambda: self.inference_model,+ read_tokens_usage=lambda: judgment_job.job_report.judgment_tokens_usage,+ )+async def judge(self,judgment_job: JudgmentJob,) -> dict[str, JudgmentAnswer]:"""Answer every question in the job over its state, keyed as the questions were."""- judgment_job.validate_before_execution()- judgment_job.job_metadata.unit_job_id = UnitJobId.JUDGMENT_ANSWER- judgment_job.judgment_job_before_start(inference_model=self.inference_model)-- try:- self._check_can_read_files(judgment_job=judgment_job)- answers = await self._judge(judgment_job=judgment_job)- _check_answers_match_questions(judgment_job=judgment_job, answers=answers)- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)- raise- finally:- # Completion and reporting belong on *every* way out, not just the happy one. The- # answer-shape guard above raises after the provider already answered and after usage was- # recorded, so that call is billed — reporting only on success would make the spend vanish- # from the run's cost report. A failure that never reached the provider recorded no tokens- # (`judgment_job_before_start` initialises `nb_tokens_by_category` empty), so it reports as- # the zero-cost attempt it was rather than inventing a charge.- judgment_job.judgment_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=judgment_job)+ # The call ends with its summary event whichever way it ends, a refusal by the checks included+ with self._call_summary(judgment_job=judgment_job):+ judgment_job.validate_before_execution()+ judgment_job.job_metadata.unit_job_id = UnitJobId.JUDGMENT_ANSWER+ judgment_job.judgment_job_before_start(inference_model=self.inference_model)++ try:+ self._check_can_read_files(judgment_job=judgment_job)+ answers = await self._judge(judgment_job=judgment_job)+ _check_answers_match_questions(judgment_job=judgment_job, answers=answers)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise+ finally:+ # Completion and reporting belong on *every* way out, not just the happy one. The+ # answer-shape guard above raises after the provider already answered and after usage was+ # recorded, so that call is billed — reporting only on success would make the spend vanish+ # from the run's cost report. A failure that never reached the provider recorded no tokens+ # (`judgment_job_before_start` initialises `nb_tokens_by_category` empty), so it reports as+ # the zero-cost attempt it was rather than inventing a charge. The summary event the call+ # ends with reads the same usage, so a billed failure carries its tokens and cost there too.+ judgment_job.judgment_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=judgment_job)return answersdiff --git a/pipelex/cogt/llm/llm_report.py b/pipelex/cogt/llm/llm_report.pyindex 35fa503be..6c5872579 100644--- a/pipelex/cogt/llm/llm_report.py+++ b/pipelex/cogt/llm/llm_report.py@@ -4,6 +4,7 @@ from typing import Any, Literalfrom pydantic import BaseModelfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict+from pipelex.cogt.usage.pricing_unit import PricingUnitfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.system.job_metadata import JobMetadata@@ -69,3 +70,5 @@ class LLMTokensUsage(BaseModel):unit_costs: CostsByCategoryDictinference_model_id: strnb_tokens_by_category: NbTokensByCategoryDict+ # What the counts are: tokens, or requests or pages recorded as a million tokens each so the rates price one.+ pricing_unit: PricingUnit = PricingUnit.TOKENdiff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.pyindex 4c01b1afa..ee4fd94bc 100644--- a/pipelex/cogt/llm/llm_worker_abstract.py+++ b/pipelex/cogt/llm/llm_worker_abstract.py@@ -10,6 +10,7 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.exceptions import CogtError, LLMCapabilityError+from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperationfrom pipelex.cogt.inference.inference_constants import InferenceOutputTypefrom pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstractfrom pipelex.cogt.inference.prompt_file_checks import check_prompt_documents_are_read, check_prompt_images_are_images@@ -171,6 +172,21 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):"""Get the response model name from the inference model."""return self.inference_model.model_id+ def _call_summary(self, *, llm_job: LLMJob) -> InferenceCallSummary:+ """The event the call ends with, its model named as the span names it and its usage read off the reported job.++ The model names are read when the call ends, never here, before `_before_job` and the provider's answer, so a+ worker that names its model in either is named the same way on the event and on the span.+ """+ return InferenceCallSummary(+ operation=InferenceOperation.CHAT,+ model_handle=self.inference_model.name,+ read_inference_model=lambda: self.inference_model,+ read_tokens_usage=lambda: llm_job.job_report.llm_tokens_usage,+ read_request_model=self._get_request_model_name,+ read_response_model=self._get_response_model_name,+ )+def _start_otel_span_llm(self, llm_job: LLMJob, *, output_type: InferenceOutputType, output_class_name: str | None = None) -> Span | None:"""Start an OTel span for the LLM job and return it.@@ -486,30 +502,35 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):# metadatallm_job.job_metadata.unit_job_id = UnitJobId.LLM_GEN_TEXT- await self._before_job(llm_job=llm_job)-- # Start OTel span after _before_job (which may set model info)- span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.TEXT)-- # The span is the Pipelex span active here until it ends, whichever way it ends, so a log line- # during the call, a provider SDK's included, names it under `pipelex.*`; OpenTelemetry's current- # context is left alone, and it is what the line's standard trace fields name.- with pipelex_span_active(span=span):- try:- self.check_request(inference_model=self.inference_model, job_params=self._sent_job_params(llm_job=llm_job), is_structured=False)- text_result = await self._gen_text(llm_job=llm_job)- await self._after_text_job(span=span, llm_job=llm_job, result_text=text_result)- return text_result- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())- raise- finally:- # `_gen_text` / `_after_text_job` raised before the span was ended — close it with the- # in-flight error (a CogtError, or any unexpected failure) so no span leaks and the- # failure stays in telemetry. On success `_after_text_job` already ended the span.- pending_error = sys.exc_info()[1]- if pending_error is not None and span is not None and span.is_recording():- self._end_otel_span_with_error(span=span, error=pending_error)+ # The call ends with its summary event whichever way it ends, a refusal by the checks of `_before_job` included+ with self._call_summary(llm_job=llm_job) as call_summary:+ await self._before_job(llm_job=llm_job)++ # Start OTel span after _before_job (which may set model info)+ span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.TEXT)++ # The span is the Pipelex span active here until it ends, whichever way it ends, so a log line+ # during the call, a provider SDK's included, names it under `pipelex.*`; OpenTelemetry's current+ # context is left alone, and it is what the line's standard trace fields name. The summary event+ # is logged at the end of this block, as the span closes: the span has ended by then, on success in+ # the job's after-step and on failure in the `finally` below, but it is still the active Pipelex span,+ # so the event carries its ids.+ with pipelex_span_active(span=span), call_summary.ends_here():+ try:+ self.check_request(inference_model=self.inference_model, job_params=self._sent_job_params(llm_job=llm_job), is_structured=False)+ text_result = await self._gen_text(llm_job=llm_job)+ await self._after_text_job(span=span, llm_job=llm_job, result_text=text_result)+ return text_result+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())+ raise+ finally:+ # `_gen_text` / `_after_text_job` raised before the span was ended — close it with the+ # in-flight error (a CogtError, or any unexpected failure) so no span leaks and the+ # failure stays in telemetry. On success `_after_text_job` already ended the span.+ pending_error = sys.exc_info()[1]+ if pending_error is not None and span is not None and span.is_recording():+ self._end_otel_span_with_error(span=span, error=pending_error)@abstractmethodasync def _gen_text(@@ -527,35 +548,40 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):# metadatallm_job.job_metadata.unit_job_id = UnitJobId.LLM_GEN_OBJECT- await self._before_job(llm_job=llm_job)-- # Start OTel span after _before_job (which may set model info)- span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.OBJECT, output_class_name=schema.__name__)-- # The span is the Pipelex span active here until it ends, whichever way it ends, so a log line- # during the call, a provider SDK's included, names it under `pipelex.*`; OpenTelemetry's current- # context is left alone, and it is what the line's standard trace fields name.- with pipelex_span_active(span=span):- try:- self.check_request(inference_model=self.inference_model, job_params=self._sent_job_params(llm_job=llm_job), is_structured=True)- object_result = await self._gen_object(llm_job=llm_job, schema=schema)-- # Cleanup result- if hasattr(object_result, "_raw_response"):- delattr(object_result, "_raw_response") # ruff: ignore[del-attr-with-constant] - not a declared model field, so `del obj._attr` cannot type-check-- await self._after_object_job(span=span, llm_job=llm_job, result_object=object_result)- return object_result- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())- raise- finally:- # `_gen_object` / `_after_object_job` raised before the span was ended — close it with- # the in-flight error (a CogtError, or any unexpected failure) so no span leaks and the- # failure stays in telemetry. On success `_after_object_job` already ended the span.- pending_error = sys.exc_info()[1]- if pending_error is not None and span is not None and span.is_recording():- self._end_otel_span_with_error(span=span, error=pending_error)+ # The call ends with its summary event whichever way it ends, a refusal by the checks of `_before_job` included+ with self._call_summary(llm_job=llm_job) as call_summary:+ await self._before_job(llm_job=llm_job)++ # Start OTel span after _before_job (which may set model info)+ span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.OBJECT, output_class_name=schema.__name__)++ # The span is the Pipelex span active here until it ends, whichever way it ends, so a log line+ # during the call, a provider SDK's included, names it under `pipelex.*`; OpenTelemetry's current+ # context is left alone, and it is what the line's standard trace fields name. The summary event+ # is logged at the end of this block, as the span closes: the span has ended by then, on success in+ # the job's after-step and on failure in the `finally` below, but it is still the active Pipelex span,+ # so the event carries its ids.+ with pipelex_span_active(span=span), call_summary.ends_here():+ try:+ self.check_request(inference_model=self.inference_model, job_params=self._sent_job_params(llm_job=llm_job), is_structured=True)+ object_result = await self._gen_object(llm_job=llm_job, schema=schema)++ # Cleanup result+ if hasattr(object_result, "_raw_response"):+ delattr(object_result, "_raw_response") # ruff: ignore[del-attr-with-constant] - not a declared model field, so `del obj._attr` cannot type-check++ await self._after_object_job(span=span, llm_job=llm_job, result_object=object_result)+ return object_result+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())+ raise+ finally:+ # `_gen_object` / `_after_object_job` raised before the span was ended — close it with+ # the in-flight error (a CogtError, or any unexpected failure) so no span leaks and the+ # failure stays in telemetry. On success `_after_object_job` already ended the span.+ pending_error = sys.exc_info()[1]+ if pending_error is not None and span is not None and span.is_recording():+ self._end_otel_span_with_error(span=span, error=pending_error)@abstractmethodasync def _gen_object(diff --git a/pipelex/cogt/search/search_report.py b/pipelex/cogt/search/search_report.pyindex bc167002d..b18eefacb 100644--- a/pipelex/cogt/search/search_report.py+++ b/pipelex/cogt/search/search_report.py@@ -4,6 +4,7 @@ from typing import Any, Literalfrom pydantic import BaseModelfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict+from pipelex.cogt.usage.pricing_unit import PricingUnitfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.system.job_metadata import JobMetadata@@ -69,3 +70,5 @@ class SearchTokensUsage(BaseModel):unit_costs: CostsByCategoryDictinference_model_id: strnb_tokens_by_category: NbTokensByCategoryDict+ # What the counts are: tokens, or requests or pages recorded as a million tokens each so the rates price one.+ pricing_unit: PricingUnit = PricingUnit.TOKENdiff --git a/pipelex/cogt/search/search_worker_abstract.py b/pipelex/cogt/search/search_worker_abstract.pyindex 4b43ad72e..bf237fabd 100644--- a/pipelex/cogt/search/search_worker_abstract.py+++ b/pipelex/cogt/search/search_worker_abstract.py@@ -4,6 +4,7 @@ from typing import Anyfrom typing_extensions import overridefrom pipelex.cogt.exceptions import CogtError+from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperationfrom pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.search.search_job import SearchJob@@ -27,30 +28,42 @@ class SearchWorkerAbstract(InferenceWorkerAbstract):def desc(self) -> str:return f"Search using {self.inference_model.desc}"+ def _call_summary(self, *, search_job: SearchJob) -> InferenceCallSummary:+ """The event the call ends with, its model and its usage read off the worker and the job it reports when it ends."""+ return InferenceCallSummary(+ operation=InferenceOperation.SEARCH,+ model_handle=self.inference_model.name,+ read_inference_model=lambda: self.inference_model,+ read_tokens_usage=lambda: search_job.job_report.search_tokens_usage,+ )+async def search_sourced_answer(self,search_job: SearchJob,) -> SearchResultContent:"""Execute a search query and return a sourced answer with sources."""- search_job.validate_before_execution()- search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER- search_job.search_job_before_start(inference_model=self.inference_model)+ # The call ends with its summary event whichever way it ends, a refusal by the job's validation included+ with self._call_summary(search_job=search_job):+ search_job.validate_before_execution()+ search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER+ search_job.search_job_before_start(inference_model=self.inference_model)- try:- result = await self._search_sourced_answer(search_job=search_job)- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)- raise- finally:- # Completion and reporting belong on *every* way out, not just the happy one. A backend's- # response-shape guard raises after the provider already answered and after usage was- # recorded, so that call is billed — reporting only on success would make the spend vanish- # from the run's cost report. A failure that never reached the provider recorded no tokens- # (`search_job_before_start` initialises `nb_tokens_by_category` empty), so it reports as the- # zero-cost attempt it was rather than inventing a charge.- search_job.search_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=search_job)+ try:+ result = await self._search_sourced_answer(search_job=search_job)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise+ finally:+ # Completion and reporting belong on *every* way out, not just the happy one. A backend's+ # response-shape guard raises after the provider already answered and after usage was+ # recorded, so that call is billed — reporting only on success would make the spend vanish+ # from the run's cost report. A failure that never reached the provider recorded no tokens+ # (`search_job_before_start` initialises `nb_tokens_by_category` empty), so it reports as the+ # zero-cost attempt it was rather than inventing a charge. The summary event the call ends+ # with reads the same usage, so a billed failure carries its tokens and cost there too.+ search_job.search_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=search_job)return result@@ -61,22 +74,24 @@ class SearchWorkerAbstract(InferenceWorkerAbstract):schema: type[BaseModelTypeVar],) -> dict[str, Any]:"""Execute a search query and return structured data matching the schema."""- search_job.validate_before_execution()- search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED- search_job.search_job_before_start(inference_model=self.inference_model)+ # The call ends with its summary event whichever way it ends, a refusal by the job's validation included+ with self._call_summary(search_job=search_job):+ search_job.validate_before_execution()+ search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED+ search_job.search_job_before_start(inference_model=self.inference_model)- try:- result = await self._search_structured(search_job=search_job, schema=schema)- except CogtError as exc:- exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)- raise- finally:- # Reported on every way out, for the reason spelled out in `search_sourced_answer` — and it- # bites hardest here, because this arm is the one whose response-shape guards reject a- # payload the provider already charged for.- search_job.search_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=search_job)+ try:+ result = await self._search_structured(search_job=search_job, schema=schema)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise+ finally:+ # Reported on every way out, for the reason spelled out in `search_sourced_answer` — and it+ # bites hardest here, because this arm is the one whose response-shape guards reject a+ # payload the provider already charged for.+ search_job.search_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=search_job)return resultdiff --git a/pipelex/cogt/usage/cost_registry.py b/pipelex/cogt/usage/cost_registry.pyindex 54858c949..b2423c175 100644--- a/pipelex/cogt/usage/cost_registry.py+++ b/pipelex/cogt/usage/cost_registry.py@@ -15,49 +15,14 @@ from pipelex.cogt.search.search_report import SearchTokenCostReport, SearchTokenfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDictfrom pipelex.cogt.usage.costs_per_token import model_cost_per_tokenfrom pipelex.cogt.usage.token_category import TokenCategory+from pipelex.cogt.usage.usage_cost import TokensUsage, compute_total_costfrom pipelex.runtime_hub import get_consolefrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | JudgmentTokensUsageTokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReport | JudgmentTokenCostReportCostRegistryRoot = list[TokenCostReport]-def compute_tokens_usage_cost(tokens_usage: TokensUsage) -> float | None:- """Compute the canonical USD cost of a single inference call, or None when unrated.-- Returns ``None`` when the usage carries no rate table (``unit_costs`` is empty:- own-GPU models, dry/mock runs). Otherwise returns the same canonical total the cost- table reports for the call — input_non_cached + input_cached + output component- costs, with the cached-discount fallback from ``model_cost_per_token``. Categories- the cost engine excludes from totals (audio, reasoning, prediction) are excluded- here too: one cost engine, one total.- """- if not tokens_usage.unit_costs:- return None- nb_tokens_input_joined = tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0)- nb_tokens_input_cached = tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT_CACHED, 0)- nb_tokens_input_non_cached = nb_tokens_input_joined - nb_tokens_input_cached- nb_tokens_output = tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0)- input_non_cached_cost = nb_tokens_input_non_cached * model_cost_per_token(- costs=tokens_usage.unit_costs,- cost_category=CostCategory.INPUT_NON_CACHED,- )- input_cached_cost = nb_tokens_input_cached * model_cost_per_token(- costs=tokens_usage.unit_costs,- cost_category=CostCategory.INPUT_CACHED,- )- output_cost = nb_tokens_output * model_cost_per_token(- costs=tokens_usage.unit_costs,- cost_category=CostCategory.OUTPUT,- )- return CostRegistry.compute_total_cost(- input_non_cached_cost=input_non_cached_cost,- input_cached_cost=input_cached_cost,- output_cost=output_cost,- )--class ModelUsageKey(NamedTuple):"""One model a run called, as its usage is grouped: the kind of call, then the model's name.@@ -199,7 +164,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):# Add rows for each modelfor model_key, aggregated_data in grouped_by_model.items():- row_total_cost = cls.compute_total_cost(+ row_total_cost = compute_total_cost(input_non_cached_cost=aggregated_data[report_field.COST_INPUT_NON_CACHED],input_cached_cost=aggregated_data[report_field.COST_INPUT_CACHED],output_cost=aggregated_data[report_field.COST_OUTPUT],@@ -258,10 +223,6 @@ class CostRegistry(RootModel[CostRegistryRoot]):writer.writeheader()writer.writerows(records)- @classmethod- def compute_total_cost(cls, *, input_non_cached_cost: float, input_cached_cost: float, output_cost: float) -> float:- return input_non_cached_cost + input_cached_cost + output_cost-@classmethoddef aggregate_costs(cls, tokens_usages: Sequence[TokensUsage]) -> AggregatedCosts:"""Aggregate token usages into flat records, per-model groups, and run totals — in one pass.@@ -310,7 +271,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):]:grouped_by_model[model_key][field] += record.get(field, 0)- total_cost = cls.compute_total_cost(+ total_cost = compute_total_cost(input_non_cached_cost=sum(record.get(report_field.COST_INPUT_NON_CACHED, 0) for record in records),input_cached_cost=sum(record.get(report_field.COST_INPUT_CACHED, 0) for record in records),output_cost=sum(record.get(report_field.COST_OUTPUT, 0) for record in records),@@ -341,7 +302,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):report_field = LLMTokenCostReportFieldby_model: list[dict[str, Any]] = []for model_key, aggregated_data in aggregated.grouped_by_model.items():- model_cost = cls.compute_total_cost(+ model_cost = compute_total_cost(input_non_cached_cost=aggregated_data[report_field.COST_INPUT_NON_CACHED],input_cached_cost=aggregated_data[report_field.COST_INPUT_CACHED],output_cost=aggregated_data[report_field.COST_OUTPUT],diff --git a/pipelex/cogt/usage/pricing_unit.py b/pipelex/cogt/usage/pricing_unit.pynew file mode 100644index 000000000..3db474bb7--- /dev/null+++ b/pipelex/cogt/usage/pricing_unit.py@@ -0,0 +1,31 @@+"""What an inference call's usage counts: tokens, or requests or pages priced as tokens.++Rates are per million tokens, so a provider that bills by the request, or an extraction priced by its pages, records+each unit as a million tokens in and out, and the rate table prices one unit. The cost is right, and the counts are not+tokens: a dashboard of tokens must not add them up. The usage says which it holds in its ``pricing_unit`` so a reader+of the counts can tell them apart. Today one reader does, the event an inference call ends with, which keeps the cost+and leaves the token counts off. The run's cost table and the run graph's usage still add a unit-priced call's counts+to their token totals, and the client-facing ``TokensUsageRecord`` still carries them as tokens.+"""++from enum import StrEnum++#: How many tokens one request or one page is recorded as, so a rate per million tokens prices one unit.+TOKENS_PER_PRICED_UNIT = 1_000_000+++class PricingUnit(StrEnum):+ """What a usage's ``nb_tokens_by_category`` counts."""++ TOKEN = "token"+ REQUEST = "request"+ PAGE = "page"++ @property+ def counts_tokens(self) -> bool:+ """Whether the counts are tokens the provider read and wrote, rather than requests or pages priced as tokens."""+ match self:+ case PricingUnit.TOKEN:+ return True+ case PricingUnit.REQUEST | PricingUnit.PAGE:+ return Falsediff --git a/pipelex/cogt/usage/usage_cost.py b/pipelex/cogt/usage/usage_cost.pynew file mode 100644index 000000000..0718100a2--- /dev/null+++ b/pipelex/cogt/usage/usage_cost.py@@ -0,0 +1,83 @@+"""The cost of one inference call's usage, in US dollars: the one cost engine every report and event prices with.++The run's cost table, the client-facing usage records, the run graph's attribution and the event an inference call+ends with all price a usage here, so a usage gets the same price whichever of them reads it. The module holds the+arithmetic alone and imports nothing that renders, so a worker base prices the call it just made without importing the+console that the cost table prints through.++It is also where a worker whose provider bills by the request, or an extraction priced by its pages, records that+usage, with ``record_unit_priced_usage``, so the rates price each unit and the usage says its counts are no tokens.+"""++from typing import Literal++from pipelex.cogt.extract.extract_report import ExtractTokensUsage+from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage+from pipelex.cogt.judgment.judgment_report import JudgmentTokensUsage+from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.search.search_report import SearchTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.costs_per_token import model_cost_per_token+from pipelex.cogt.usage.pricing_unit import TOKENS_PER_PRICED_UNIT, PricingUnit+from pipelex.cogt.usage.token_category import TokenCategory++TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | JudgmentTokensUsage+++def compute_total_cost(*, input_non_cached_cost: float, input_cached_cost: float, output_cost: float) -> float:+ """The total of a call's component costs: what it read uncached, what it read from the cache, and what it wrote."""+ return input_non_cached_cost + input_cached_cost + output_cost+++def compute_tokens_usage_cost(tokens_usage: TokensUsage) -> float | None:+ """Compute the canonical USD cost of a single inference call, or None when unrated.++ Returns ``None`` when the usage carries no rate table (``unit_costs`` is empty:+ own-GPU models, dry/mock runs). Otherwise returns the same canonical total the cost+ table reports for the call — input_non_cached + input_cached + output component+ costs, with the cached-discount fallback from ``model_cost_per_token``. Categories+ the cost engine excludes from totals (audio, reasoning, prediction) are excluded+ here too: one cost engine, one total.+ """+ if not tokens_usage.unit_costs:+ return None+ nb_tokens_input_joined = tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0)+ nb_tokens_input_cached = tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT_CACHED, 0)+ nb_tokens_input_non_cached = nb_tokens_input_joined - nb_tokens_input_cached+ nb_tokens_output = tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0)+ input_non_cached_cost = nb_tokens_input_non_cached * model_cost_per_token(+ costs=tokens_usage.unit_costs,+ cost_category=CostCategory.INPUT_NON_CACHED,+ )+ input_cached_cost = nb_tokens_input_cached * model_cost_per_token(+ costs=tokens_usage.unit_costs,+ cost_category=CostCategory.INPUT_CACHED,+ )+ output_cost = nb_tokens_output * model_cost_per_token(+ costs=tokens_usage.unit_costs,+ cost_category=CostCategory.OUTPUT,+ )+ return compute_total_cost(+ input_non_cached_cost=input_non_cached_cost,+ input_cached_cost=input_cached_cost,+ output_cost=output_cost,+ )+++def record_unit_priced_usage(*, tokens_usage: TokensUsage, pricing_unit: Literal[PricingUnit.REQUEST, PricingUnit.PAGE], nb_units: int) -> None:+ """Record a call billed by the request or by the page, rather than by the token, on its usage.++ The rates are per million tokens, so each unit is recorded as a million tokens in and out, and the rate table+ prices one unit; the usage says it counts ``pricing_unit``, which the event an inference call ends with reads, so it+ keeps the call's cost and writes no token counts. The other readers of the counts do not read it yet, as+ ``pipelex.cogt.usage.pricing_unit`` says. Every worker that prices this way, a provider's or a plugin's, records its+ usage through here.++ Args:+ tokens_usage: The usage on the job's report, which the worker base hands to the reporting path.+ pricing_unit: What the provider billed: requests, or pages.+ nb_units: How many requests or pages the call was billed for.+ """+ nb_tokens = nb_units * TOKENS_PER_PRICED_UNIT+ tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: nb_tokens, TokenCategory.OUTPUT: nb_tokens}+ tokens_usage.pricing_unit = pricing_unitdiff --git a/pipelex/graph/graphspec.py b/pipelex/graph/graphspec.pyindex 8cd0da058..88c918858 100644--- a/pipelex/graph/graphspec.py+++ b/pipelex/graph/graphspec.py@@ -311,11 +311,13 @@ class ModelUsageSpec(BaseModel):inference_model_name: strinference_model_id: str- # Kind of inference: "llm", "img_gen", "extract", "search", "judgment". The discriminator a- # consumer needs before displaying token counts: extract/search/img_gen are billed- # PER REQUEST, and that price is encoded by putting 1_000_000 in each token- # category (rates are per-million), so their "tokens" are a scaled request counter.- # llm and judgment report the real tokens the provider read and wrote.+ # Kind of inference: "llm", "img_gen", "extract", "search", "judgment". It does not tell a+ # consumer whether the token counts are tokens: most calls of every kind report the real+ # tokens the provider read and wrote, while a call billed by the request or by the page, a+ # Linkup search or fetch or an extraction priced by its pages, records each unit as+ # 1_000_000 in each token category so the per-million rates price one unit. Its usage says+ # so in `pricing_unit` (`pipelex.cogt.usage.pricing_unit`), which this spec does not carry+ # yet, so the token totals here still add those scaled counts.model_type: strinference_calls: int = 0rated_inference_calls: int = 0diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.pyindex 16d3c289c..7dc94362d 100644--- a/pipelex/pipe_machinery/pipe_abstract.py+++ b/pipelex/pipe_machinery/pipe_abstract.py@@ -7,6 +7,7 @@ import shortuuidfrom opentelemetry import tracefrom opentelemetry.trace import NonRecordingSpan, Span, SpanContext, SpanKind, Status, StatusCode, TraceFlagsfrom pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator+from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.inference.error_classification import UserAction, UserActionKind@@ -47,6 +48,7 @@ from pipelex.system.telemetry.otel_factory import OtelFactoryfrom pipelex.system.telemetry.telemetry_identity import make_run_identity_span_attributesfrom pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstractfrom pipelex.tools.log.console_layouts import LogLayout+from pipelex.tools.log.summary_event import SummaryEventfrom pipelex.tools.misc.package_utils import get_package_versionfrom pipelex.tools.misc.string_utils import get_root_from_dotted_path, is_snake_casefrom pipelex.validation_error_types import PipeValidationErrorType@@ -59,6 +61,37 @@ PipeAbstractType = type["PipeAbstract"]#: The message every live pipe run announces itself with; the pipe is in the fields, and the console draws it as the pipe tree.PIPE_RUN_STARTS_MESSAGE = "Pipe run starts"+#: The message every live pipe run ends with, whichever way it ends; the pipe, its duration and its outcome are in the fields.+PIPE_RUN_ENDS_MESSAGE = "Pipe run ends"+++class PipeRunEnd(SummaryEvent):+ """Times a live pipe run and logs the event it ends with, once, when it ends, however it ends.++ The pair of the run's announcement: entered with it, under the run's `pipe_run_id`, and left once the run returns+ or raises, so every announced run ends with one event and a dry run, which announces nothing, with none. The run+ enters `ends_here` around the block its span is active in, so the event is logged as the pipe's own span closes,+ just after the block has ended it, while it is still the active Pipelex span, and carries that span's ids; a+ failure before that block still ends the run with its event. The event carries the announcement's fields, then+ `duration_ms`, `outcome`, `error.type` on failure, and `cancelled` for a run stopped from outside. It never handles+ the exception it reads the outcome from, which goes on as it came.++ Args:+ pipe_fields: The announcement's fields, the same mapping the announcement was logged with.+ """++ def __init__(self, *, pipe_fields: dict[str, Any]) -> None:+ super().__init__(message=PIPE_RUN_ENDS_MESSAGE)+ self._pipe_fields = pipe_fields++ @override+ def _work_fields(self) -> dict[str, Any]:+ return self._pipe_fields++ @override+ def _log_event(self, *, fields: dict[str, Any]) -> None:+ log.info(PIPE_RUN_ENDS_MESSAGE, fields=fields, layout=LogLayout.PIPE_RUN_END)+class AbsentInput(NamedTuple):"""One needed input that is absent from working memory, with its ledger record when one exists."""@@ -997,10 +1030,13 @@ class PipeAbstract(ABC, BaseModel):# names the pipe run it belongs to, a nested pipe's until it binds its own. The outer binding# comes back when the pipe returns, however it returns. A pipe lifted for absent optional# inputs never gets here: it has no run and no id, and its skip line carries the enclosing binding.- with log.context(pipe_run_id=this_pipe_run_id):+ # The run ends with its summary event, under the same binding, whichever way it ends: it is logged+ # as the pipe's own span closes when the run reaches it, and here when the setup before it fails.+ pipe_run_fields = self._pipe_run_fields(pipe_run_params=pipe_run_params)+ with log.context(pipe_run_id=this_pipe_run_id), PipeRunEnd(pipe_fields=pipe_run_fields) as run_end:# A fixed message, the pipe in the fields: the console draws the pipe tree from them through# the pipe-run layout, and every other sink writes them as keys a query can select on.- log.info(PIPE_RUN_STARTS_MESSAGE, fields=self._pipe_run_fields(pipe_run_params=pipe_run_params), layout=LogLayout.PIPE_RUN)+ log.info(PIPE_RUN_STARTS_MESSAGE, fields=pipe_run_fields, layout=LogLayout.PIPE_RUN)# Handle telemetry ------------------------------------------------------------@@ -1045,8 +1081,9 @@ class PipeAbstract(ABC, BaseModel):# line inside the run names it under `pipelex.*`. OpenTelemetry's current context is left# alone, so a host's own instrumentation is never re-parented and a line's standard trace# fields keep naming the host's span, and the span's children still take their parent from- # `child_metadata`.- with pipelex_span_active(span=span):+ # `child_metadata`. The run's summary event is logged at the end of this block, as the span closes:+ # the span has ended by then, but it is still the active Pipelex span, so the event carries its ids.+ with pipelex_span_active(span=span), run_end.ends_here():try:pipe_output = await self._live_run_pipe(job_metadata=child_metadata,diff --git a/pipelex/providers/linkup/linkup_extract_worker.py b/pipelex/providers/linkup/linkup_extract_worker.pyindex ca981f318..fac8fd2c5 100644--- a/pipelex/providers/linkup/linkup_extract_worker.py+++ b/pipelex/providers/linkup/linkup_extract_worker.py@@ -23,7 +23,8 @@ from pipelex.cogt.inference.error_classification import extract_linkup_metadatafrom pipelex.cogt.inference.error_classify import classify_inference_errorfrom pipelex.cogt.inference.error_render import InferenceErrorFamily, render_inference_errorfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.cogt.usage.pricing_unit import PricingUnit+from pipelex.cogt.usage.usage_cost import record_unit_priced_usagefrom pipelex.reporting.reporting_protocol import ReportingProtocol@@ -85,12 +86,9 @@ class LinkupExtractWorker(ExtractWorkerAbstract):model_handle=self.inference_model.name,) from sdk_exc- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ # Linkup bills by the request, which the usage records as one request priced at the model's rateif extract_tokens_usage := extract_job.job_report.extract_tokens_usage:- extract_tokens_usage.nb_tokens_by_category = {- TokenCategory.INPUT: 1_000_000,- TokenCategory.OUTPUT: 1_000_000,- }+ record_unit_priced_usage(tokens_usage=extract_tokens_usage, pricing_unit=PricingUnit.REQUEST, nb_units=1)max_images = job_params.max_nb_imagesextracted_images: list[ExtractedImageFromPage] = []diff --git a/pipelex/providers/linkup/linkup_search_worker.py b/pipelex/providers/linkup/linkup_search_worker.pyindex 847e05c13..b65b39660 100644--- a/pipelex/providers/linkup/linkup_search_worker.py+++ b/pipelex/providers/linkup/linkup_search_worker.py@@ -24,7 +24,8 @@ from pipelex.cogt.search.search_depth import SearchDepthfrom pipelex.cogt.search.search_job import SearchJobfrom pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstractfrom pipelex.cogt.search.structured_search_payload import extract_structured_search_payload-from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.cogt.usage.pricing_unit import PricingUnit+from pipelex.cogt.usage.usage_cost import record_unit_priced_usagefrom pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.search_result_content import SearchResultContentfrom pipelex.providers.linkup.linkup_exceptions import LinkupSearchEmptyResultError, LinkupSearchResponseError@@ -88,9 +89,9 @@ class LinkupSearchWorker(SearchWorkerAbstract):model_handle=self.inference_model.name,) from sdk_exc- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ # Linkup bills by the request, which the usage records as one request priced at the model's rateif search_tokens_usage := search_job.job_report.search_tokens_usage:- search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}+ record_unit_priced_usage(tokens_usage=search_tokens_usage, pricing_unit=PricingUnit.REQUEST, nb_units=1)sources: list[DocumentContent] = []for source in response.sources:@@ -160,9 +161,9 @@ class LinkupSearchWorker(SearchWorkerAbstract):model_handle=self.inference_model.name,) from sdk_exc- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ # Linkup bills by the request, which the usage records as one request priced at the model's rateif search_tokens_usage := search_job.job_report.search_tokens_usage:- search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}+ record_unit_priced_usage(tokens_usage=search_tokens_usage, pricing_unit=PricingUnit.REQUEST, nb_units=1)# The contract of a structured search is the structured payload itself — it is validated against# the caller's output structure class, which has nowhere to put sources. Asking for them wraps thediff --git a/pipelex/reporting/reporting_types.py b/pipelex/reporting/reporting_types.pyindex 4c0d89131..7f9502880 100644--- a/pipelex/reporting/reporting_types.py+++ b/pipelex/reporting/reporting_types.py@@ -14,4 +14,3 @@ AnyTokensUsage = Annotated[LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | JudgmentTokensUsage,Field(discriminator="model_type"),]-TokensUsage = AnyTokensUsagediff --git a/pipelex/reporting/usage_records.py b/pipelex/reporting/usage_records.pyindex 9d9691ea1..a1674286b 100644--- a/pipelex/reporting/usage_records.py+++ b/pipelex/reporting/usage_records.py@@ -19,7 +19,7 @@ from typing import TYPE_CHECKING, Any, castfrom pydantic import BaseModel, ConfigDict-from pipelex.cogt.usage.cost_registry import compute_tokens_usage_cost+from pipelex.cogt.usage.usage_cost import compute_tokens_usage_costfrom pipelex.reporting.reporting_types import AnyTokensUsageif TYPE_CHECKING:diff --git a/pipelex/tools/log/console_layouts.py b/pipelex/tools/log/console_layouts.pyindex 6044c9fea..af00e663f 100644--- a/pipelex/tools/log/console_layouts.py+++ b/pipelex/tools/log/console_layouts.py@@ -22,6 +22,7 @@ The registry is one table in code. Rich is imported only where a layout is rendefrom __future__ import annotations+import mathimport stringfrom enum import StrEnumfrom typing import TYPE_CHECKING, Any@@ -29,7 +30,9 @@ from typing import TYPE_CHECKING, Anyfrom pydantic import BaseModel, ConfigDict, field_validatorfrom typing_extensions import override+from pipelex.system.telemetry.otel_constants import GenAISpanAttrfrom pipelex.tools.log.console_fields import FIELD_STYLES, format_layout_value, one_line_text+from pipelex.tools.log.summary_fields import COST_USD_FIELD, DURATION_MS_FIELD, OUTCOME_FIELD, Outcomefrom pipelex.tools.misc.rich_extra import require_richif TYPE_CHECKING:@@ -43,6 +46,8 @@ class LogLayout(StrEnum):"""The layouts a log call can name with ``layout=``."""PIPE_RUN = "pipe_run"+ PIPE_RUN_END = "pipe_run_end"+ INFERENCE_CALL_END = "inference_call_end"class ConsoleLayout(BaseModel):@@ -170,9 +175,183 @@ PIPE_RUN_LAYOUT = PipeRunLayout(presented_fields=frozenset({"pipe_type", "pipe_code", "output_concept", "pipe_depth"}),)+#: What a summary event says before its duration, by how the work ended.+SUCCESS_ENDING = "done in"+ERROR_ENDING = "failed after"+CANCELLED_ENDING = "cancelled after"+#: Below this many milliseconds a duration is written in milliseconds, and in seconds from it on.+MILLISECONDS_PER_SECOND = 1000+#: How many decimals of a dollar a cost is written to, before its trailing zeros are dropped.+COST_DECIMALS = 6+++def _endings(*, outcome: Any) -> dict[str, str]:+ """The success, error and cancelled endings of a summary event, all but one of them empty, so each takes its own style.++ Raises:+ ValueError: If the outcome is not one of ``success``, ``error`` and ``cancelled``.+ """+ match Outcome(outcome):+ case Outcome.SUCCESS:+ return {"success_ending": SUCCESS_ENDING, "error_ending": "", "cancelled_ending": ""}+ case Outcome.ERROR:+ return {"success_ending": "", "error_ending": ERROR_ENDING, "cancelled_ending": ""}+ case Outcome.CANCELLED:+ return {"success_ending": "", "error_ending": "", "cancelled_ending": CANCELLED_ENDING}+++def _is_number(*, value: Any) -> bool:+ """Whether a value is a finite number, a boolean, which is an integer to Python, excluded."""+ return not isinstance(value, bool) and isinstance(value, (int, float)) and math.isfinite(value)+++def format_duration(*, duration_ms: Any) -> str:+ """A duration in milliseconds as a person reads it: ``840 ms`` under a second, ``1.25 s`` from a second on.++ Raises:+ TypeError: If the duration is not a number.+ ValueError: If the duration is negative or not finite.+ """+ if not _is_number(value=duration_ms):+ msg = f"A duration must be a finite number of milliseconds, not {type(duration_ms).__name__}"+ raise TypeError(msg)+ if duration_ms < 0:+ msg = "A duration cannot be negative"+ raise ValueError(msg)+ if round(duration_ms) < MILLISECONDS_PER_SECOND:+ return f"{duration_ms:.0f} ms"+ return f"{duration_ms / MILLISECONDS_PER_SECOND:.2f} s"+++def format_cost(*, cost_usd: Any) -> str:+ """A cost in US dollars to the millionth of a dollar, its trailing zeros dropped: ``$0.0123``, ``$0``.++ A cost that rounds to nothing at that precision is written as under the millionth, ``<$0.000001``, so a call that+ cost a little is never read as a free one, which alone is written ``$0``.++ Raises:+ TypeError: If the cost is not a number.+ """+ if not _is_number(value=cost_usd):+ msg = f"A cost must be a finite number of dollars, not {type(cost_usd).__name__}"+ raise TypeError(msg)+ formatted = f"${cost_usd:.{COST_DECIMALS}f}".rstrip("0").rstrip(".")+ if formatted == "$0" and cost_usd > 0:+ return f"<${10**-COST_DECIMALS:.{COST_DECIMALS}f}"+ return formatted+++def _format_token_count(*, nb_tokens: Any) -> str:+ """A token count with its thousands separated: ``12,288``.++ Raises:+ TypeError: If the count is not an integer.+ """+ if isinstance(nb_tokens, bool) or not isinstance(nb_tokens, int):+ msg = f"A token count must be an integer, not {type(nb_tokens).__name__}"+ raise TypeError(msg)+ return f"{nb_tokens:,}"+++class PipeRunEndLayout(PipeRunLayout):+ """The end of a pipe run, drawn in the pipe tree under the line that announced it.++ ``PipeLLM: describe_company done in 1.25 s``, indented and behind ``↳`` at the depth of its announcement. It reads+ the pipe-run layout's depth, ``duration_ms`` and ``outcome``; a failure says ``failed after`` in red, and its+ ``error.type`` follows as the suffix, and a cancelled run says ``cancelled after`` in yellow.+ """++ @override+ def derived_values(self, *, fields: Mapping[str, Any]) -> dict[str, Any]:+ """The tree's indentation and branch mark, the ending the outcome calls for, and the duration as a person reads it.++ Raises:+ TypeError: If the depth or the duration is of the wrong type.+ ValueError: If the depth is out of range, the duration negative or the outcome unknown.+ """+ return {+ **super().derived_values(fields=fields),+ **_endings(outcome=fields[OUTCOME_FIELD]),+ "duration": format_duration(duration_ms=fields[DURATION_MS_FIELD]),+ }+++PIPE_RUN_END_LAYOUT = PipeRunEndLayout(+ template=(+ "{indent}[yellow]{branch}[/yellow]{branch_gap}"+ f"{_styled_placeholder(field='pipe_type', suffix=':')} {_styled_placeholder(field='pipe_code')} "+ "[dim]{success_ending}[/dim][bold red]{error_ending}[/bold red][yellow]{cancelled_ending}[/yellow] {duration}"+ ),+ presented_fields=frozenset({"pipe_type", "pipe_code", "output_concept", "pipe_depth", DURATION_MS_FIELD, OUTCOME_FIELD}),+)+++class InferenceCallEndLayout(ConsoleLayout):+ """The end of an inference call, on one compact line: the model, the operation, the tokens, the cost and the duration.++ ``claude-5.5-sonnet chat · 2,048 → 512 tokens · $0.009216 · done in 1.23 s``, the tokens in before the arrow and out+ after it. A count or a cost the call did not report is left out with its separator. The model is drawn by its+ handle, and the keys that name the same model otherwise, ``gen_ai.request.model``, ``gen_ai.response.model``,+ ``backend_name`` and ``sdk``, are left off the line, which the ``json`` sink writes in full. A failure says+ ``failed after`` in red, and its ``error.type`` follows as the suffix; a cancelled call says ``cancelled after`` in+ yellow.+ """++ @override+ def derived_values(self, *, fields: Mapping[str, Any]) -> dict[str, Any]:+ """The operation, the usage with its separators, the ending the outcome calls for and the duration.++ Raises:+ TypeError: If a token count, the cost or the duration is of the wrong type.+ ValueError: If the duration is negative or the outcome unknown.+ """+ input_tokens = fields.get(GenAISpanAttr.USAGE_INPUT_TOKENS)+ output_tokens = fields.get(GenAISpanAttr.USAGE_OUTPUT_TOKENS)+ usage_parts: list[str] = []+ if input_tokens is not None and output_tokens is not None:+ usage_parts.append(f"{_format_token_count(nb_tokens=input_tokens)} → {_format_token_count(nb_tokens=output_tokens)} tokens")+ elif input_tokens is not None:+ usage_parts.append(f"{_format_token_count(nb_tokens=input_tokens)} tokens in")+ elif output_tokens is not None:+ usage_parts.append(f"{_format_token_count(nb_tokens=output_tokens)} tokens out")+ cost_usd = fields.get(COST_USD_FIELD)+ if cost_usd is not None:+ usage_parts.append(format_cost(cost_usd=cost_usd))+ return {+ "operation": fields[GenAISpanAttr.OPERATION_NAME],+ "usage": "".join(f" · {part}" for part in usage_parts),+ **_endings(outcome=fields[OUTCOME_FIELD]),+ "duration": format_duration(duration_ms=fields[DURATION_MS_FIELD]),+ }+++INFERENCE_CALL_END_LAYOUT = InferenceCallEndLayout(+ template=(+ "[cyan]{model_handle}[/cyan] [dim]{operation}[/dim]{usage} · "+ "[dim]{success_ending}[/dim][bold red]{error_ending}[/bold red][yellow]{cancelled_ending}[/yellow] {duration}"+ ),+ presented_fields=frozenset(+ {+ GenAISpanAttr.OPERATION_NAME,+ "model_handle",+ "backend_name",+ "sdk",+ GenAISpanAttr.REQUEST_MODEL,+ GenAISpanAttr.RESPONSE_MODEL,+ GenAISpanAttr.USAGE_INPUT_TOKENS,+ GenAISpanAttr.USAGE_OUTPUT_TOKENS,+ COST_USD_FIELD,+ DURATION_MS_FIELD,+ OUTCOME_FIELD,+ }+ ),+)+#: Every layout a call can name, by its name.CONSOLE_LAYOUTS: dict[LogLayout, ConsoleLayout] = {LogLayout.PIPE_RUN: PIPE_RUN_LAYOUT,+ LogLayout.PIPE_RUN_END: PIPE_RUN_END_LAYOUT,+ LogLayout.INFERENCE_CALL_END: INFERENCE_CALL_END_LAYOUT,}diff --git a/pipelex/tools/log/summary_event.py b/pipelex/tools/log/summary_event.pynew file mode 100644index 000000000..730532795--- /dev/null+++ b/pipelex/tools/log/summary_event.py@@ -0,0 +1,110 @@+"""The timed context manager a unit of work ends with its summary event through, once, however it ends.++``SummaryEvent`` is what the event an inference call ends with and the event a pipe run ends with share: it starts the+clock when the work starts, logs the event once when the work ends, its outcome read off the exception leaving the+block, and never lets a failure of the event's own change the work's outcome. The fields it writes are named in+``pipelex.tools.log.summary_fields``, which the console layouts read without importing this module and the logger it+logs through.+"""++from __future__ import annotations++import contextlib+from abc import ABC, abstractmethod+from typing import TYPE_CHECKING, Any, Self++from pipelex.tools.log.error_fields import error_fields+from pipelex.tools.log.log import log+from pipelex.tools.log.summary_fields import DURATION_MS_FIELD, SUMMARY_EVENT_FIELD, elapsed_ms, outcome_fields, start_clock++if TYPE_CHECKING:+ from types import TracebackType++#: The message of the warning a summary event that could not be logged is replaced with.+SUMMARY_EVENT_FAILED_MESSAGE = "A summary event could not be logged"+++class SummaryEvent(ABC):+ """Times a unit of work and logs the one event it ends with, once, when it ends, however it ends.++ A context manager around the work. Entering it starts the clock, and leaving it logs the event: the work's own+ fields, then ``duration_ms``, then ``outcome``, read off the exception leaving the block, with ``error.type`` on+ failure, and ``cancelled`` for work stopped from outside. It never handles that exception, which goes on as it came.++ The event is the work's measurement, never part of its result: if its fields cannot be built or logged, a call+ that succeeded must still succeed and one that failed must fail with its own exception. So a failure of the+ event's own is logged once in its place, as a warning, and the work's outcome goes on unchanged; a warning the+ handlers refuse in their turn is dropped, since nothing is left to say it with.++ Some work runs under a context that only exists part of the way, a span that is started after checks that may+ refuse the work. ``ends_here`` enters the event again for the inner block, so the event is logged when that block+ ends, still under the context the block sets, while a refusal before it still ends the work with its event, logged+ when the outer block ends. The clock starts at the first entry, and the event is logged at the first exit, so+ nothing that can fail may follow the inner block inside the outer one.++ A subclass gives the work's fields in ``_work_fields`` and logs the event in ``_log_event``, from its own module,+ with its fixed message and its console layout, so the event is on the logger of the code it measures and a host+ quietens one event without the other.++ Args:+ message: The event's message, which names it in the warning a failure of its own is replaced with.+ """++ _started_at: float++ def __init__(self, *, message: str) -> None:+ self._message = message+ self._has_started = False+ self._has_ended = False++ def __enter__(self) -> Self:+ if not self._has_started:+ self._has_started = True+ self._started_at = start_clock()+ return self++ def __exit__(+ self,+ exc_type: type[BaseException] | None,+ exc_value: BaseException | None,+ exc_traceback: TracebackType | None,+ ) -> None:+ if self._has_ended:+ return+ self._has_ended = True+ self._end(error=exc_value)++ def ends_here(self) -> Self:+ """The event again, to enter around the inner block whose end it is logged at.++ ``with pipelex_span_active(span=span), call_summary.ends_here():`` logs the event while the span is still the+ active Pipelex span, so the line names it, even where the block has already ended the span itself, and the+ outer ``with call_summary:`` logs it only when the inner block was never reached.+ """+ return self++ @abstractmethod+ def _work_fields(self) -> dict[str, Any]:+ """The fields that say what the work was, read when it ends, before its duration and its outcome."""++ @abstractmethod+ def _log_event(self, *, fields: dict[str, Any]) -> None:+ """Log the event with ``fields``, at INFO, under its fixed message and its console layout."""++ def _end(self, *, error: BaseException | None) -> None:+ """Log the event the work ends with, having raised ``error`` or not, or the warning a failure of its own is logged as.++ Nothing raised here leaves it, the warning's own failure included, so the work's result or exception goes on as it came.+ """+ try:+ duration_ms = elapsed_ms(started_at=self._started_at)+ fields = {**self._work_fields(), DURATION_MS_FIELD: duration_ms, **outcome_fields(error=error)}+ self._log_event(fields=fields)+ except Exception as event_error: # ruff: ignore[blind-except]+ # (2) the work's fields are read from whatever the work recorded, a provider worker's usage included, a plugin's+ # among them, and the event goes through whatever handlers the host installed, so what building and logging it can+ # raise cannot be enumerated, and the work's outcome must not change.+ # (2) the warning goes through the same handlers, which may refuse it as they refused the event; then there is+ # nowhere left to say so, and the work's outcome still must not change.+ with contextlib.suppress(Exception):+ log.warning(SUMMARY_EVENT_FAILED_MESSAGE, fields={SUMMARY_EVENT_FIELD: self._message, **error_fields(exc=event_error)})diff --git a/pipelex/tools/log/summary_fields.py b/pipelex/tools/log/summary_fields.pynew file mode 100644index 000000000..48a555e6c--- /dev/null+++ b/pipelex/tools/log/summary_fields.py@@ -0,0 +1,72 @@+"""The fields every summary event shares, the event a unit of work ends with: how long it took, what it cost and how it ended.++A summary event, the one an inference call ends with or the one a pipe run ends with, says how long the work took in+``duration_ms``, and how it ended in ``outcome``: ``success`` when it returned, ``error`` when it raised, naming the+class of the exception in ``error.type``, the OpenTelemetry semantic-convention key for the class of error an operation+ended with, and ``cancelled`` when it was stopped from outside rather than failing, by a cancellation, an interrupt or+the interpreter exiting. A query counts failures by ``outcome`` and groups them by ``error.type`` without parsing any+text, and the siblings a failure in a batch cancels are not counted as failures of their own. The+exception's own text stays off the event: the exception propagates to whoever handles it, which logs it there with its+traceback. An inference call's event also says what the call cost, in ``cost_usd``.++The names live here, beside the console layouts that draw the events, so a layout reads them without importing the+code that logs them, ``SummaryEvent`` (``pipelex.tools.log.summary_event``) among it.+"""++from enum import StrEnum+from time import perf_counter++from pipelex.tools.log.error_fields import ERROR_TYPE_FIELD++#: How the unit of work ended.+OUTCOME_FIELD = "outcome"++#: How long the unit of work took, in milliseconds, as a number.+DURATION_MS_FIELD = "duration_ms"++#: What an inference call cost, in US dollars, as a number.+COST_USD_FIELD = "cost_usd"++#: The summary event a line is about, by its message.+SUMMARY_EVENT_FIELD = "summary_event"+++class Outcome(StrEnum):+ """The values of ``outcome``."""++ SUCCESS = "success"+ ERROR = "error"+ CANCELLED = "cancelled"+++def outcome_fields(*, error: BaseException | None) -> dict[str, str]:+ """The ``outcome`` field of a unit of work, and its ``error.type`` when it failed.++ An exception that is not an ``Exception`` is not a failure of the work: Python keeps that branch of the hierarchy+ for what stops work from outside, a task cancelled (``asyncio.CancelledError``), as a batch or a parallel+ controller cancels its siblings and a client cancels a run, an interrupt (``KeyboardInterrupt``), a generator+ closed (``GeneratorExit``) or the interpreter exiting (``SystemExit``). That work was cancelled, and its event+ names no ``error.type``.++ Args:+ error: The exception the work ended with, or ``None`` when it returned.++ Returns:+ ``{"outcome": "success"}``, ``{"outcome": "cancelled"}``, or+ ``{"outcome": "error", "error.type": <the exception's class name>}``.+ """+ if error is None:+ return {OUTCOME_FIELD: Outcome.SUCCESS}+ if not isinstance(error, Exception):+ return {OUTCOME_FIELD: Outcome.CANCELLED}+ return {OUTCOME_FIELD: Outcome.ERROR, ERROR_TYPE_FIELD: type(error).__name__}+++def start_clock() -> float:+ """A reading of the monotonic clock a summary event's duration is measured from, for ``elapsed_ms``."""+ return perf_counter()+++def elapsed_ms(*, started_at: float) -> float:+ """The milliseconds since ``started_at``, a ``start_clock`` reading, to the microsecond."""+ return round((perf_counter() - started_at) * 1000, 3)diff --git a/pipelex/tracing/usage_attribution.py b/pipelex/tracing/usage_attribution.pyindex 2e22d5792..ff085ede7 100644--- a/pipelex/tracing/usage_attribution.py+++ b/pipelex/tracing/usage_attribution.py@@ -23,6 +23,7 @@ from collections.abc import Mapping, Sequencefrom pipelex import logfrom pipelex.cogt.llm.llm_report import LLMTokenCostReportFieldfrom pipelex.cogt.usage.cost_registry import AggregatedCosts, CostRegistry, ModelUsageKey+from pipelex.cogt.usage.usage_cost import compute_total_costfrom pipelex.graph.graphspec import ModelUsageSpec, NodeUsageSpecfrom pipelex.reporting.reporting_types import AnyTokensUsage@@ -78,7 +79,7 @@ def _model_specs(*, usages: Sequence[AnyTokensUsage], aggregated: AggregatedCostspecs: list[ModelUsageSpec] = []for model_key, model_data in aggregated.grouped_by_model.items():rated_calls = rated_by_model.get(model_key, 0)- model_cost = CostRegistry.compute_total_cost(+ model_cost = compute_total_cost(input_non_cached_cost=model_data[LLMTokenCostReportField.COST_INPUT_NON_CACHED],input_cached_cost=model_data[LLMTokenCostReportField.COST_INPUT_CACHED],output_cost=model_data[LLMTokenCostReportField.COST_OUTPUT],diff --git a/subject_grants.toml b/subject_grants.tomlindex 369054e86..7bc38acf7 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -2288,7 +2288,7 @@ rationale = "Transforms/renders the aggregated; single operand."param = "records"rationale = "Cost report op on the records; single operand."-["pipelex/cogt/usage/cost_registry.py::compute_tokens_usage_cost"]+["pipelex/cogt/usage/usage_cost.py::compute_tokens_usage_cost"]param = "tokens_usage"rationale = "Verb-object: compute the cost of one usage record; the single obvious operand is the usage."diff --git a/tests/integration/pipelex/pipeline/test_data.py b/tests/integration/pipelex/pipeline/test_data.pyindex b3ca9c472..56b44f6e0 100644--- a/tests/integration/pipelex/pipeline/test_data.py+++ b/tests/integration/pipelex/pipeline/test_data.py@@ -511,3 +511,59 @@ inputs = { company = "Company", short_summary = "ShortSummary" }output = "Text"template = "{{ company.name }} works in {{ company.sector }}: {{ short_summary.text }}""""+++class SummaryEventsTestData:+ """A small method run live on two models, every model call answered by a stand-in worker, to read the events it ends with."""++ MAIN_PIPE: ClassVar[str] = "brief_company"++ INPUTS: ClassVar[PipelineInputs] = {"brief": "Acme builds reusable rockets in Toulouse and sells launches to research labs."}++ MTHDS: ClassVar[str] = """+domain = "summary_events_check"+description = "Describe a company, structure it and pitch it, on two models"+main_pipe = "brief_company"++[concept.Company]+description = "A company"++[concept.Company.structure]+name = { type = "text", description = "The company's name" }+sector = { type = "text", description = "The company's sector" }++[pipe.brief_company]+type = "PipeSequence"+description = "Describe the company, structure it and pitch it"+inputs = { brief = "Text" }+output = "Text"+steps = [+ { pipe = "describe_company", result = "description" },+ { pipe = "structure_company", result = "company" },+ { pipe = "pitch_company", result = "pitch" },+]++[pipe.describe_company]+type = "PipeLLM"+description = "Describe the company the brief is about"+inputs = { brief = "Text" }+output = "Text"+model = "$testing-text"+prompt = "Describe the company this brief is about: $brief"++[pipe.structure_company]+type = "PipeLLM"+description = "Structure the description as a company"+inputs = { description = "Text" }+output = "Company"+model = "$testing-text"+prompt = "Read the company out of this description: $description"++[pipe.pitch_company]+type = "PipeLLM"+description = "Pitch the company in one sentence"+inputs = { description = "Text" }+output = "Text"+model = "$writing-creative"+prompt = "Pitch this company in one sentence: $description"+"""diff --git a/tests/integration/pipelex/pipeline/test_summary_events_price_a_run.py b/tests/integration/pipelex/pipeline/test_summary_events_price_a_run.pynew file mode 100644index 000000000..48fec601b--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_summary_events_price_a_run.py@@ -0,0 +1,143 @@+from __future__ import annotations++import io+import json+import logging+import math+from collections import defaultdict+from contextlib import contextmanager+from typing import TYPE_CHECKING, Any++import pytest+from typing_extensions import override++from pipelex.cogt.inference.inference_call_summary import INFERENCE_CALL_ENDS_MESSAGE+from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract+from pipelex.cogt.llm.thinking_mode import ThinkingMode+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.pipe_machinery.pipe_abstract import PIPE_RUN_ENDS_MESSAGE+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.telemetry.otel_constants import GenAISpanAttr+from pipelex.tools.log.json_log_sink import MESSAGE_KEY, JsonLogSink+from tests.integration.pipelex.pipeline.test_data import SummaryEventsTestData++if TYPE_CHECKING:+ from collections.abc import Generator++ from pytest_mock import MockerFixture++ from pipelex.cogt.llm.llm_job import LLMJob+ from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar++# The models the method's presets resolve to here, as the stand-in workers name them, with their rates per million+# tokens in and out. Each call reads 1,000 tokens and writes 200.+STAND_IN_MODELS: dict[str, tuple[str, dict[CostCategory, float]]] = {+ "@default-small": ("small-test-model", {CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 4.0}),+ "@default-premium": ("premium-test-model", {CostCategory.INPUT: 5.0, CostCategory.OUTPUT: 20.0}),+}+INPUT_TOKENS = 1000+OUTPUT_TOKENS = 200+++class StandInLLMWorker(LLMWorkerAbstract):+ """Answers every generation without a provider, recording the usage a provider would have answered with."""++ @override+ async def _gen_text(self, llm_job: LLMJob) -> str:+ self._record_usage(llm_job=llm_job)+ return "Acme builds reusable rockets for research labs."++ @override+ async def _gen_object(self, llm_job: LLMJob, *, schema: type[BaseModelTypeVar]) -> BaseModelTypeVar:+ self._record_usage(llm_job=llm_job)+ return schema.model_validate({field_name: f"Stand-in {field_name}" for field_name in schema.model_fields})++ @staticmethod+ def _record_usage(*, llm_job: LLMJob) -> None:+ tokens_usage = llm_job.job_report.llm_tokens_usage+ assert tokens_usage is not None+ tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: INPUT_TOKENS, TokenCategory.OUTPUT: OUTPUT_TOKENS}+++def _stand_in_worker(*, llm_handle: str) -> StandInLLMWorker:+ """The worker the handle resolves to here, a stand-in for the model it would have reached."""+ model_name, rates = STAND_IN_MODELS[llm_handle]+ inference_model = InferenceModelSpec(+ backend_name="stand_in",+ name=model_name,+ sdk="stand_in",+ model_type=ModelType.LLM,+ model_id=f"{model_name}-2026",+ inputs=["text"],+ outputs=["text", "structured"],+ costs=rates,+ thinking_mode=ThinkingMode.NONE,+ max_tokens=None,+ max_prompt_images=None,+ )+ return StandInLLMWorker(inference_model=inference_model, reporting_delegate=None)+++@contextmanager+def _json_sink_on_root(*, buffer: io.StringIO) -> Generator[None]:+ """The ``json`` sink's handler on the root logger, beside whatever the session installed, for as long as the block runs."""+ handler = JsonLogSink(stream=buffer).handler+ root_logger = logging.getLogger()+ root_logger.addHandler(handler)+ try:+ yield+ finally:+ root_logger.removeHandler(handler)+ handler.close()++[email protected](loop_scope="class")+class TestSummaryEventsPriceARun:+ async def test_the_json_output_alone_gives_latency_and_cost_per_operation_and_model(+ self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture+ ) -> None:+ """What a dashboard does with a run's ``json`` output: group the inference events by operation and model, and add them up."""+ mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker", side_effect=_stand_in_worker)+ # The default production level.+ caplog.set_level(logging.INFO, logger="pipelex")+ runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE)+ buffer = io.StringIO()++ with _json_sink_on_root(buffer=buffer):+ await runner.execute(+ pipe_code=SummaryEventsTestData.MAIN_PIPE, mthds_contents=[SummaryEventsTestData.MTHDS], inputs=SummaryEventsTestData.INPUTS+ )++ lines: list[dict[str, Any]] = [json.loads(line) for line in buffer.getvalue().splitlines() if line]+ calls = [line for line in lines if line[MESSAGE_KEY] == INFERENCE_CALL_ENDS_MESSAGE]+ groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)+ for call in calls:+ groups[call[GenAISpanAttr.OPERATION_NAME], call["model_handle"]].append(call)+ cost_by_group = {key: sum(call["cost_usd"] for call in group) for key, group in groups.items()}+ latency_by_group = {key: sum(call["duration_ms"] for call in group) / len(group) for key, group in groups.items()}+ tokens_by_group = {+ key: (sum(call[GenAISpanAttr.USAGE_INPUT_TOKENS] for call in group), sum(call[GenAISpanAttr.USAGE_OUTPUT_TOKENS] for call in group))+ for key, group in groups.items()+ }++ assert {key: len(group) for key, group in groups.items()} == {("chat", "small-test-model"): 2, ("chat", "premium-test-model"): 1}+ # 1,000 tokens in and 200 out: $0.0018 a call at $1 and $4 a million, $0.009 at $5 and $20.+ assert math.isclose(cost_by_group["chat", "small-test-model"], 0.0036)+ assert math.isclose(cost_by_group["chat", "premium-test-model"], 0.009)+ assert tokens_by_group == {("chat", "small-test-model"): (2000, 400), ("chat", "premium-test-model"): (1000, 200)}+ for key, latency_ms in latency_by_group.items():+ assert isinstance(latency_ms, float), key+ assert latency_ms >= 0, key+ assert all(call["outcome"] == "success" for call in calls)+ # Every pipe the run announced ended once, at the depth it announced itself.+ ended = sorted((line["pipe_code"], line["pipe_depth"], line["outcome"]) for line in lines if line[MESSAGE_KEY] == PIPE_RUN_ENDS_MESSAGE)+ assert ended == [+ ("brief_company", 0, "success"),+ ("describe_company", 1, "success"),+ ("pitch_company", 1, "success"),+ ("structure_company", 1, "success"),+ ]diff --git a/tests/unit/pipelex/cogt/inference/test_inference_call_summary.py b/tests/unit/pipelex/cogt/inference/test_inference_call_summary.pynew file mode 100644index 000000000..eeb477b15--- /dev/null+++ b/tests/unit/pipelex/cogt/inference/test_inference_call_summary.py@@ -0,0 +1,778 @@+from __future__ import annotations++import asyncio+import io+import json+import logging+import math+import threading+from concurrent.futures import ThreadPoolExecutor+from typing import TYPE_CHECKING, Any, cast++import pytest+from opentelemetry.sdk.trace import TracerProvider+from opentelemetry.sdk.trace.export import SimpleSpanProcessor+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter+from typing_extensions import override++from pipelex.cogt.content_generation import doc_gen_generate+from pipelex.cogt.content_generation.assignment_models import RenderDocumentAssignment+from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams+from pipelex.cogt.content_generation.doc_gen_generate import render_document_and_store+from pipelex.cogt.doc_gen import doc_gen_engine+from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource+from pipelex.cogt.doc_gen.doc_gen_setting import DocGenSetting+from pipelex.cogt.doc_gen.doc_gen_worker_abstract import DocGenWorkerAbstract+from pipelex.cogt.doc_gen.doc_gen_worker_factory import DocGenWorkerFactory+from pipelex.cogt.doc_gen.document_composition import DocumentComposition+from pipelex.cogt.doc_gen.exceptions import DocGenEngineMissingError+from pipelex.cogt.doc_gen.layout_tree import ImageBlock, LayoutBlock, LayoutDocument, MarkdownBlock+from pipelex.cogt.doc_gen.render_job import RenderedDocument+from pipelex.cogt.exceptions import (+ ExtractCapabilityError,+ ImgGenParameterError,+ LLMCapabilityError,+ LLMCompletionError,+ ModelNotFoundError,+ SearchJobFailureError,+)+from pipelex.cogt.extract.extract_input import ExtractInput+from pipelex.cogt.extract.extract_job import ExtractJob+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams, ExtractJobReport+from pipelex.cogt.extract.extract_output import ExtractOutput, Page+from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract+from pipelex.cogt.image.generated_image import GeneratedImageRawDetails+from pipelex.cogt.image.prompt_image import PromptImageUri+from pipelex.cogt.img_gen.img_gen_job_factory import ImgGenJobFactory+from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract+from pipelex.cogt.inference.inference_call_summary import INFERENCE_CALL_ENDS_MESSAGE, InferenceOperation+from pipelex.cogt.judgment.judgment_models import JudgmentAnswer, YesNoAnswer, YesNoQuestion+from pipelex.cogt.judgment.judgment_worker_abstract import JudgmentWorkerAbstract+from pipelex.cogt.llm.llm_job import LLMJob+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams+from pipelex.cogt.llm.llm_prompt import LLMPrompt+from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract+from pipelex.cogt.llm.thinking_mode import ThinkingMode+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.search.search_job_factory import SearchJobFactory+from pipelex.cogt.search.search_setting import SearchSetting+from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.pricing_unit import PricingUnit+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory+from pipelex.cogt.usage.usage_cost import compute_tokens_usage_cost+from pipelex.core.stuffs.search_result_content import SearchResultContent+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY+from pipelex.system.telemetry.otel_constants import GenAISpanAttr+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract+from pipelex.tools.log.console_layouts import LogLayout+from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, JsonLogSink+from pipelex.tools.log.log_fields import LAYOUT_MARK, attached_field_names+from pipelex.tools.log.summary_event import SUMMARY_EVENT_FAILED_MESSAGE+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+from pipelex.tools.uri.exceptions import UriReadRefusedError+from tests.helpers.console_log_rendering import rendered_text+from tests.unit.pipelex.cogt.judgment.fake_judgment_worker import make_fake_judgment_job+from tests.unit.pipelex.providers.reportlab.reportlab_test_helpers import reportlab_pdf_model++if TYPE_CHECKING:+ from collections.abc import Awaitable, Callable, Iterator++ from pytest_mock import MockerFixture++ from pipelex.cogt.doc_gen.render_job import RenderJob, RenderResources+ from pipelex.cogt.img_gen.img_gen_job import ImgGenJob+ from pipelex.cogt.judgment.judgment_job import JudgmentJob+ from pipelex.cogt.search.search_job import SearchJob+ from pipelex.cogt.usage.usage_cost import TokensUsage+ from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar++SUMMARY_LOGGER = "pipelex.cogt.inference.inference_call_summary"+LLM_USAGE: NbTokensByCategoryDict = {TokenCategory.INPUT: 1200, TokenCategory.OUTPUT: 300}+OTHER_USAGE: NbTokensByCategoryDict = {TokenCategory.INPUT: 40, TokenCategory.OUTPUT: 8}+# The rates per million tokens every stand-in model is priced at: $1 in and $2 out.+RATES = {CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0}+STARTED_AT = 10.0+ENDED_AT = 11.25+DURATION_MS = 1250.0+# The model a routing LLM worker names once `_before_job` has routed the call, and the one the provider says served it.+ROUTED_MODEL = "gpt-test-routed"+SERVED_MODEL = "gpt-test-2026-10-01"+# The document engine's model keys, as the kit's `internal.toml` declares `reportlab-pdf`.+REPORTLAB_MODEL_FIELDS: dict[str, Any] = {+ "model_handle": "reportlab-pdf",+ "backend_name": "internal",+ "sdk": "reportlab",+ GenAISpanAttr.REQUEST_MODEL: "reportlab-pdf",+ GenAISpanAttr.RESPONSE_MODEL: "print-pdf",+}+RUN_TRACE_ID = 0x0123456789ABCDEF0123456789ABCDEF+PIPE_SPAN_ID = 0x00000000000000AB+++def _model(+ *, model_type: ModelType, name: str, model_id: str, costs: dict[CostCategory, float] | None = None, inputs: list[str] | None = None+) -> InferenceModelSpec:+ return InferenceModelSpec(+ backend_name="stand_in_backend",+ name=name,+ sdk="stand_in_sdk",+ model_type=model_type,+ model_id=model_id,+ inputs=inputs or ["text"],+ outputs=["text", "structured"],+ costs=RATES if costs is None else costs,+ thinking_mode=ThinkingMode.NONE,+ max_tokens=None,+ max_prompt_images=None,+ )+++def _job_metadata(*, otel_context: OtelContext | None = None) -> JobMetadata:+ return JobMetadata(+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id="plr-summary"),+ pipe_code="some_pipe",+ otel_context=otel_context,+ )+++def _record_usage(*, tokens_usage: TokensUsage | None, usage: NbTokensByCategoryDict | None) -> None:+ """What a provider's half of a worker does with the usage the provider answered with."""+ if tokens_usage is not None and usage is not None:+ tokens_usage.nb_tokens_by_category = dict(usage)+++class _StandInLLMWorker(LLMWorkerAbstract):+ def __init__(self, *, usage: NbTokensByCategoryDict | None = None, error: BaseException | None = None, reporting_delegate: Any = None) -> None:+ LLMWorkerAbstract.__init__(+ self, inference_model=_model(model_type=ModelType.LLM, name="gpt-test", model_id="gpt-test-2026"), reporting_delegate=reporting_delegate+ )+ self._usage = usage+ self._error = error++ @override+ async def _gen_text(self, llm_job: LLMJob) -> str:+ _record_usage(tokens_usage=llm_job.job_report.llm_tokens_usage, usage=self._usage)+ if self._error is not None:+ raise self._error+ return "answer"++ @override+ async def _gen_object(self, llm_job: LLMJob, *, schema: type[BaseModelTypeVar]) -> BaseModelTypeVar:+ raise NotImplementedError+++class _StandInImgGenWorker(ImgGenWorkerAbstract):+ def __init__(self) -> None:+ ImgGenWorkerAbstract.__init__(self, inference_model=_model(model_type=ModelType.IMG_GEN, name="img-test", model_id="img-test-2026"))++ @override+ async def _gen_image(self, img_gen_job: ImgGenJob) -> GeneratedImageRawDetails:+ _record_usage(tokens_usage=img_gen_job.job_report.img_gen_tokens_usage, usage=OTHER_USAGE)+ return GeneratedImageRawDetails(size=None, actual_url="https://example.com/image.png")++ @override+ async def _gen_image_list(self, img_gen_job: ImgGenJob, *, nb_images: int) -> list[GeneratedImageRawDetails]:+ raise NotImplementedError+++class _StandInExtractWorker(ExtractWorkerAbstract):+ def __init__(self, *, usage: NbTokensByCategoryDict | None) -> None:+ ExtractWorkerAbstract.__init__(+ self,+ extra_config={},+ inference_model=_model(model_type=ModelType.TEXT_EXTRACTOR, name="extract-test", model_id="extract-test-2026", inputs=["pdf"]),+ )+ self._usage = usage++ @override+ async def _extract_pages(self, extract_job: ExtractJob) -> ExtractOutput:+ _record_usage(tokens_usage=extract_job.job_report.extract_tokens_usage, usage=self._usage)+ return ExtractOutput(pages={index: Page(text="page text") for index in range(3)})+++class _StandInSearchWorker(SearchWorkerAbstract):+ def __init__(self, *, error: Exception | None = None) -> None:+ SearchWorkerAbstract.__init__(self, inference_model=_model(model_type=ModelType.SEARCH, name="search-test", model_id="search-test-2026"))+ self._error = error++ @override+ async def _search_sourced_answer(self, search_job: SearchJob) -> SearchResultContent:+ _record_usage(tokens_usage=search_job.job_report.search_tokens_usage, usage=OTHER_USAGE)+ if self._error is not None:+ raise self._error+ return SearchResultContent(answer="an answer")++ @override+ async def _search_structured(self, search_job: SearchJob, *, schema: type[BaseModelTypeVar]) -> dict[str, Any]:+ raise NotImplementedError+++class _StandInJudgmentWorker(JudgmentWorkerAbstract):+ def __init__(self) -> None:+ JudgmentWorkerAbstract.__init__(self, inference_model=_model(model_type=ModelType.JUDGMENT, name="judge-test", model_id="judge-test-2026"))++ @override+ async def _judge(self, judgment_job: JudgmentJob) -> dict[str, JudgmentAnswer]:+ _record_usage(tokens_usage=judgment_job.job_report.judgment_tokens_usage, usage=OTHER_USAGE)+ return {"is_urgent": YesNoAnswer(yes_no=True)}+++class _RoutingLLMWorker(_StandInLLMWorker):+ """Names the model it routes the call to in `_before_job`, and learns from the provider which model served the call."""++ def __init__(self) -> None:+ super().__init__(usage=LLM_USAGE)+ self._routed_model = "gpt-test"+ self._served_model = "gpt-test-2026"++ @override+ async def _before_job(self, llm_job: LLMJob) -> None:+ await super()._before_job(llm_job=llm_job)+ self._routed_model = ROUTED_MODEL++ @override+ async def _gen_text(self, llm_job: LLMJob) -> str:+ self._served_model = SERVED_MODEL+ return await super()._gen_text(llm_job=llm_job)++ @override+ def _get_request_model_name(self) -> str:+ return self._routed_model++ @override+ def _get_response_model_name(self) -> str:+ return self._served_model+++class _BlockingEngine(DocGenWorkerAbstract):+ """A document engine whose render waits until the test releases it, as a long print does."""++ def __init__(self) -> None:+ super().__init__(inference_model=reportlab_pdf_model())+ self.has_started = threading.Event()+ self.release = threading.Event()++ @override+ def render(self, *, job: RenderJob, resources: RenderResources) -> RenderedDocument:+ self.has_started.set()+ self.release.wait(timeout=10)+ return RenderedDocument(data=b"%PDF-1.4")+++def _render_assignment(+ *, model_handle: str = "reportlab-pdf", read_scope: str | None = None, image_url: str | None = None+) -> RenderDocumentAssignment:+ blocks: list[LayoutBlock] = [MarkdownBlock(markdown="Hello")]+ if image_url is not None:+ blocks.append(ImageBlock(url=image_url))+ return RenderDocumentAssignment(+ job_metadata=JobMetadata(+ run_metadata=RunMetadata(user_id="pytest", storage_scope="org_abc/run", read_scope=read_scope, pipeline_run_id="plr-summary"),+ pipe_code="print_summary",+ ),+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),+ composition=DocumentComposition(+ format=DocGenFormat.PDF,+ source=DocGenSource.LAYOUT,+ filename="summary.pdf",+ title="Summary",+ layout=LayoutDocument(title="Summary", blocks=blocks),+ ),+ doc_gen_setting=DocGenSetting(model=model_handle),+ )+++def _generated_content_factory(mocker: MockerFixture) -> Any:+ """The factory a print stores its file through, standing in for the run's storage."""+ factory = mocker.MagicMock()+ factory.storage_provider = mocker.MagicMock(spec=StorageProviderAbstract)+ factory.make_document_content = mocker.AsyncMock(return_value="the stored document")+ return factory+++def _llm_job(*, otel_context: OtelContext | None = None, llm_prompt: LLMPrompt | None = None) -> LLMJob:+ return LLMJob(+ job_metadata=_job_metadata(otel_context=otel_context),+ llm_prompt=llm_prompt or LLMPrompt(user_text="hello"),+ job_params=LLMJobParams(temperature=0.5),+ job_config=LLMJobConfig(schema_reask_max_attempts=1),+ )+++def _extract_job(*, extract_input: ExtractInput | None = None) -> ExtractJob:+ return ExtractJob(+ extract_input=extract_input or ExtractInput(document_uri="https://example.com/report.pdf"),+ job_params=ExtractJobParams.make_default_extract_job_params(),+ job_config=ExtractJobConfig(),+ job_report=ExtractJobReport(),+ job_metadata=_job_metadata(),+ )+++def _search_job() -> SearchJob:+ return SearchJobFactory.make_search_job("what is new", search_setting=SearchSetting(model="search-test"), job_metadata=_job_metadata())+++async def _run_img_gen() -> None:+ img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt_contents("a cat", negative_text=None, job_metadata=_job_metadata())+ await _StandInImgGenWorker().gen_image(img_gen_job=img_gen_job)+++async def _run_extract() -> None:+ await _StandInExtractWorker(usage=OTHER_USAGE).extract_pages(extract_job=_extract_job())+++async def _run_search() -> None:+ await _StandInSearchWorker().search_sourced_answer(search_job=_search_job())+++async def _run_judgment() -> None:+ await _StandInJudgmentWorker().judge(judgment_job=make_fake_judgment_job({"is_urgent": YesNoQuestion(instructions="Is it urgent?")}))+++# Each family's call, its operation and its model keys. Each reports 40 tokens in and 8 out, which the stand-ins' rates+# price at $0.000056.+OTHER_TOKEN_FIELDS: dict[str, Any] = {GenAISpanAttr.USAGE_INPUT_TOKENS: 40, GenAISpanAttr.USAGE_OUTPUT_TOKENS: 8}+OTHER_COST_USD = 0.000056+FAMILY_CASES: list[tuple[Callable[[], Awaitable[None]], InferenceOperation, tuple[str, str]]] = [+ (_run_img_gen, InferenceOperation.IMG_GEN, ("img-test", "img-test-2026")),+ (_run_extract, InferenceOperation.EXTRACT, ("extract-test", "extract-test-2026")),+ (_run_search, InferenceOperation.SEARCH, ("search-test", "search-test-2026")),+ (_run_judgment, InferenceOperation.JUDGMENT, ("judge-test", "judge-test-2026")),+]+FAMILY_IDS = ["image generation", "extraction", "search", "judgment"]++# A picture handed to models that read only text or only PDFs, which the checks before the call refuse.+CAT_PICTURE = PromptImageUri(uri="https://example.com/cat.png")+++async def _refuse_llm() -> None:+ await _StandInLLMWorker(usage=LLM_USAGE).gen_text(llm_job=_llm_job(llm_prompt=LLMPrompt(user_text="describe it", user_images=[CAT_PICTURE])))+++async def _refuse_img_gen() -> None:+ img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt_contents("a cat", negative_text=None, job_metadata=_job_metadata())+ img_gen_job.img_gen_prompt.input_images = [CAT_PICTURE]+ await _StandInImgGenWorker().gen_image(img_gen_job=img_gen_job)+++async def _refuse_extract() -> None:+ extract_job = _extract_job(extract_input=ExtractInput(image_uri="https://example.com/scan.png"))+ await _StandInExtractWorker(usage=OTHER_USAGE).extract_pages(extract_job=extract_job)+++REFUSAL_CASES: list[tuple[Callable[[], Awaitable[None]], type[Exception], InferenceOperation]] = [+ (_refuse_llm, LLMCapabilityError, InferenceOperation.CHAT),+ (_refuse_img_gen, ImgGenParameterError, InferenceOperation.IMG_GEN),+ (_refuse_extract, ExtractCapabilityError, InferenceOperation.EXTRACT),+]+REFUSAL_IDS = ["an LLM without vision given a picture", "an image model given an input image", "a PDF extractor given an image"]+# A print each check before the engine refuses: what the assignment asks, whether the engine's plugin is installed, the+# refusal, and the model keys the event names it by, the handle alone where the print ended before the model resolved.+FOREIGN_IMAGE = "pipelex-storage://org_other/assets/secret.png"+DOC_GEN_REFUSAL_CASES: list[tuple[dict[str, Any], bool, type[Exception], dict[str, Any]]] = [+ (+ {"read_scope": "org_abc", "image_url": FOREIGN_IMAGE},+ True,+ UriReadRefusedError,+ {"model_handle": "reportlab-pdf", GenAISpanAttr.REQUEST_MODEL: "reportlab-pdf"},+ ),+ ({"model_handle": "no-such-engine"}, True, ModelNotFoundError, {"model_handle": "no-such-engine", GenAISpanAttr.REQUEST_MODEL: "no-such-engine"}),+ ({}, False, DocGenEngineMissingError, REPORTLAB_MODEL_FIELDS),+]+DOC_GEN_REFUSAL_IDS = ["an image outside the read scope", "an engine no model here serves", "an engine no plugin registers"]+# A count the provider half recorded as nothing at all, which the cost engine cannot subtract from.+BROKEN_USAGE = cast("NbTokensByCategoryDict", {TokenCategory.INPUT: None, TokenCategory.OUTPUT: 300})+++def _summaries(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:+ return [record for record in caplog.records if record.name == SUMMARY_LOGGER and record.getMessage() == INFERENCE_CALL_ENDS_MESSAGE]+++def _fields(record: logging.LogRecord) -> dict[str, Any]:+ """The fields the call attached to the record, read the way a sink reads them."""+ return {name: getattr(record, name) for name in attached_field_names(record=record)}+++def fixed_clock(mocker: MockerFixture) -> None:+ """Every call starts at the same reading and ends 1.25 s later, so its duration is known."""+ mocker.patch("pipelex.tools.log.summary_event.start_clock", return_value=STARTED_AT)+ mocker.patch("pipelex.tools.log.summary_fields.perf_counter", return_value=ENDED_AT)+++def span_exporter(mocker: MockerFixture) -> InMemorySpanExporter:+ """The runtime's tracer, replaced by one exporting to memory, so the test reads back the span the worker started."""+ exporter = InMemorySpanExporter()+ provider = TracerProvider()+ provider.add_span_processor(SimpleSpanProcessor(exporter))+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=provider.get_tracer(__name__))+ return exporter+++def json_lines() -> Iterator[io.StringIO]:+ """The `json` sink's lines for every event the summary logger writes, written as it logs them, with the span active then."""+ buffer = io.StringIO()+ handler = JsonLogSink(stream=buffer).handler+ summary_logger = logging.getLogger(SUMMARY_LOGGER)+ summary_logger.addHandler(handler)+ try:+ yield buffer+ finally:+ summary_logger.removeHandler(handler)+++def _read_json_lines(buffer: io.StringIO) -> list[dict[str, Any]]:+ return [json.loads(line) for line in buffer.getvalue().splitlines() if line]++[email protected]("fixed_clock")+class TestInferenceCallSummary:+ @pytest.mark.asyncio+ async def test_a_successful_call_ends_with_the_event_and_the_usage_the_reporting_path_is_handed(+ self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture+ ) -> None:+ reporting_delegate = mocker.MagicMock()+ worker = _StandInLLMWorker(usage=LLM_USAGE, reporting_delegate=reporting_delegate)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await worker.gen_text(llm_job=_llm_job())++ (record,) = _summaries(caplog)+ reported_job: LLMJob = reporting_delegate.report_inference_job.call_args.kwargs["inference_job"]+ reported_usage = reported_job.job_report.llm_tokens_usage+ assert reported_usage is not None+ assert record.levelno == logging.INFO+ assert record.__dict__[LAYOUT_MARK] == LogLayout.INFERENCE_CALL_END+ assert _fields(record) == {+ GenAISpanAttr.OPERATION_NAME: "chat",+ "model_handle": "gpt-test",+ "backend_name": "stand_in_backend",+ "sdk": "stand_in_sdk",+ GenAISpanAttr.REQUEST_MODEL: "gpt-test",+ GenAISpanAttr.RESPONSE_MODEL: "gpt-test-2026",+ GenAISpanAttr.USAGE_INPUT_TOKENS: reported_usage.nb_tokens_by_category[TokenCategory.INPUT],+ GenAISpanAttr.USAGE_OUTPUT_TOKENS: reported_usage.nb_tokens_by_category[TokenCategory.OUTPUT],+ "cost_usd": compute_tokens_usage_cost(reported_usage),+ "duration_ms": DURATION_MS,+ "outcome": "success",+ }+ assert math.isclose(_fields(record)["cost_usd"], 0.0018)++ @pytest.mark.asyncio+ async def test_the_model_keys_mean_what_the_llm_span_makes_them_mean(+ self, caplog: pytest.LogCaptureFixture, span_exporter: InMemorySpanExporter+ ) -> None:+ otel_context = OtelContext(trace_id=RUN_TRACE_ID, trace_name="some_pipe", trace_name_redacted="some_pipe", span_id=PIPE_SPAN_ID)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await _StandInLLMWorker(usage=LLM_USAGE).gen_text(llm_job=_llm_job(otel_context=otel_context))++ (record,) = _summaries(caplog)+ (span,) = span_exporter.get_finished_spans()+ assert span.attributes is not None+ for key in (GenAISpanAttr.REQUEST_MODEL, GenAISpanAttr.RESPONSE_MODEL, GenAISpanAttr.USAGE_INPUT_TOKENS, GenAISpanAttr.USAGE_OUTPUT_TOKENS):+ assert getattr(record, key) == span.attributes[key], key++ @pytest.mark.asyncio+ async def test_a_call_failing_before_the_provider_answers_says_so_and_carries_no_usage(self, caplog: pytest.LogCaptureFixture) -> None:+ worker = _StandInLLMWorker(error=LLMCompletionError(message="provider refused"))++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER), pytest.raises(LLMCompletionError, match="provider refused"):+ await worker.gen_text(llm_job=_llm_job())++ (record,) = _summaries(caplog)+ fields = _fields(record)+ assert (fields["outcome"], fields["error.type"], fields["duration_ms"]) == ("error", "LLMCompletionError", DURATION_MS)+ assert GenAISpanAttr.USAGE_INPUT_TOKENS not in fields+ assert GenAISpanAttr.USAGE_OUTPUT_TOKENS not in fields+ assert "cost_usd" not in fields++ @pytest.mark.asyncio+ async def test_a_failure_the_provider_billed_carries_its_usage_and_cost(self, caplog: pytest.LogCaptureFixture) -> None:+ """A search whose answer a guard refuses was answered and billed, and the reporting path reports it: so does the event."""+ worker = _StandInSearchWorker(error=SearchJobFailureError(message="the answer had no sources"))++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER), pytest.raises(SearchJobFailureError):+ await worker.search_sourced_answer(search_job=_search_job())++ (record,) = _summaries(caplog)+ fields = _fields(record)+ assert math.isclose(fields.pop("cost_usd"), OTHER_COST_USD)+ assert fields == {+ GenAISpanAttr.OPERATION_NAME: "search",+ "model_handle": "search-test",+ "backend_name": "stand_in_backend",+ "sdk": "stand_in_sdk",+ GenAISpanAttr.REQUEST_MODEL: "search-test",+ GenAISpanAttr.RESPONSE_MODEL: "search-test-2026",+ **OTHER_TOKEN_FIELDS,+ "duration_ms": DURATION_MS,+ "outcome": "error",+ "error.type": "SearchJobFailureError",+ }++ @pytest.mark.parametrize(("run_call", "operation", "model_names"), FAMILY_CASES, ids=FAMILY_IDS)+ @pytest.mark.asyncio+ async def test_every_family_ends_its_call_with_the_event_under_its_operation(+ self,+ caplog: pytest.LogCaptureFixture,+ run_call: Callable[[], Awaitable[None]],+ operation: InferenceOperation,+ model_names: tuple[str, str],+ ) -> None:+ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await run_call()++ (record,) = _summaries(caplog)+ handle, model_id = model_names+ fields = _fields(record)+ assert math.isclose(fields.pop("cost_usd"), OTHER_COST_USD)+ assert fields == {+ GenAISpanAttr.OPERATION_NAME: operation,+ "model_handle": handle,+ "backend_name": "stand_in_backend",+ "sdk": "stand_in_sdk",+ GenAISpanAttr.REQUEST_MODEL: handle,+ GenAISpanAttr.RESPONSE_MODEL: model_id,+ **OTHER_TOKEN_FIELDS,+ "duration_ms": DURATION_MS,+ "outcome": "success",+ }++ @pytest.mark.asyncio+ async def test_a_document_print_ends_with_the_event_and_no_usage_at_all(self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture) -> None:+ """A document engine reports no usage, so its event carries no tokens and no cost, rather than zeros."""+ generated_content_factory = _generated_content_factory(mocker)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ stored = await render_document_and_store(_render_assignment(), generated_content_factory=generated_content_factory)++ assert stored == "the stored document"+ assert generated_content_factory.make_document_content.await_args.kwargs["data"].startswith(b"%PDF")+ (record,) = _summaries(caplog)+ assert _fields(record) == {+ GenAISpanAttr.OPERATION_NAME: "doc_gen",+ **REPORTLAB_MODEL_FIELDS,+ "duration_ms": DURATION_MS,+ "outcome": "success",+ }++ @pytest.mark.parametrize(+ ("assignment_kwargs", "is_engine_installed", "refusal_type", "model_fields"), DOC_GEN_REFUSAL_CASES, ids=DOC_GEN_REFUSAL_IDS+ )+ @pytest.mark.asyncio+ async def test_a_document_print_the_checks_refuse_ends_with_the_event_too(+ self,+ caplog: pytest.LogCaptureFixture,+ mocker: MockerFixture,+ assignment_kwargs: dict[str, Any],+ is_engine_installed: bool,+ refusal_type: type[Exception],+ model_fields: dict[str, Any],+ ) -> None:+ """The read scope, the engine's model and its installation are checked inside the event, as every other family's checks are."""+ if not is_engine_installed:+ mocker.patch.object(doc_gen_engine, "get_inference_backend_registry").return_value.has.return_value = False+ generated_content_factory = _generated_content_factory(mocker)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER), pytest.raises(refusal_type):+ await render_document_and_store(_render_assignment(**assignment_kwargs), generated_content_factory=generated_content_factory)++ generated_content_factory.make_document_content.assert_not_awaited()+ (record,) = _summaries(caplog)+ assert _fields(record) == {+ GenAISpanAttr.OPERATION_NAME: "doc_gen",+ **model_fields,+ "duration_ms": DURATION_MS,+ "outcome": "error",+ "error.type": refusal_type.__name__,+ }++ @pytest.mark.asyncio+ async def test_a_document_print_cancelled_from_outside_ends_cancelled_once_whatever_its_thread_does_after(+ self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture+ ) -> None:+ """The event follows the coroutine awaiting the print: the engine's thread, which nothing stops, ends its render unheard."""+ engine = _BlockingEngine()+ mocker.patch.object(DocGenWorkerFactory, "make_doc_gen_worker", return_value=engine)+ print_pool = ThreadPoolExecutor(max_workers=1)+ mocker.patch.object(doc_gen_generate, "_PRINT_EXECUTOR", print_pool)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ print_task = asyncio.create_task(+ render_document_and_store(_render_assignment(), generated_content_factory=_generated_content_factory(mocker))+ )+ assert await asyncio.to_thread(engine.has_started.wait, 10)+ print_task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await print_task+ engine.release.set()+ # The engine's thread has returned from its render once the pool it ran on has shut down.+ await asyncio.to_thread(print_pool.shutdown, wait=True)++ (record,) = _summaries(caplog)+ fields = _fields(record)+ assert (fields["outcome"], fields["duration_ms"]) == ("cancelled", DURATION_MS)+ assert "error.type" not in fields++ @pytest.mark.asyncio+ async def test_an_extraction_priced_by_its_pages_carries_its_cost_and_no_page_counts_as_tokens(self, caplog: pytest.LogCaptureFixture) -> None:+ """The base records three pages as three million tokens in and out so the rate table prices a page; a page is not a token."""+ extract_job = _extract_job()++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await _StandInExtractWorker(usage=None).extract_pages(extract_job=extract_job)++ (record,) = _summaries(caplog)+ fields = _fields(record)+ reported_usage = extract_job.job_report.extract_tokens_usage+ assert reported_usage is not None+ assert reported_usage.nb_tokens_by_category == {TokenCategory.INPUT: 3_000_000, TokenCategory.OUTPUT: 3_000_000}+ assert reported_usage.pricing_unit is PricingUnit.PAGE+ assert fields["cost_usd"] == compute_tokens_usage_cost(reported_usage)+ assert math.isclose(fields["cost_usd"], 9.0)+ assert GenAISpanAttr.USAGE_INPUT_TOKENS not in fields+ assert GenAISpanAttr.USAGE_OUTPUT_TOKENS not in fields++ @pytest.mark.asyncio+ async def test_an_unrated_model_carries_its_tokens_and_no_cost(self, caplog: pytest.LogCaptureFixture) -> None:+ """A model with no rate table, one run on its own GPUs, has no price: the cost is left off rather than written as zero."""+ worker = _StandInLLMWorker(usage=LLM_USAGE)+ worker.inference_model = _model(model_type=ModelType.LLM, name="local-test", model_id="local-test-1", costs={})++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await worker.gen_text(llm_job=_llm_job())++ (record,) = _summaries(caplog)+ fields = _fields(record)+ assert (fields[GenAISpanAttr.USAGE_INPUT_TOKENS], fields[GenAISpanAttr.USAGE_OUTPUT_TOKENS]) == (1200, 300)+ assert "cost_usd" not in fields++ @pytest.mark.asyncio+ async def test_the_json_sink_writes_the_tokens_the_cost_and_the_duration_as_numbers(self, caplog: pytest.LogCaptureFixture) -> None:+ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await _StandInLLMWorker(usage=LLM_USAGE).gen_text(llm_job=_llm_job())+ (record,) = _summaries(caplog)+ buffer = io.StringIO()++ JsonLogSink(stream=buffer).handler.handle(record)++ line: dict[str, Any] = json.loads(buffer.getvalue())+ assert (line[MESSAGE_KEY], line[LOGGER_KEY]) == (INFERENCE_CALL_ENDS_MESSAGE, SUMMARY_LOGGER)+ assert line[GenAISpanAttr.USAGE_INPUT_TOKENS] == 1200+ assert line[GenAISpanAttr.USAGE_OUTPUT_TOKENS] == 300+ assert isinstance(line[GenAISpanAttr.USAGE_INPUT_TOKENS], int)+ assert isinstance(line[GenAISpanAttr.USAGE_OUTPUT_TOKENS], int)+ assert isinstance(line["cost_usd"], float)+ assert math.isclose(line["cost_usd"], 0.0018)+ assert isinstance(line["duration_ms"], float)+ assert line["duration_ms"] == DURATION_MS+ assert (line[GenAISpanAttr.OPERATION_NAME], line["outcome"]) == ("chat", "success")+ assert LogLayout.INFERENCE_CALL_END not in line.values()++ @pytest.mark.asyncio+ async def test_the_console_draws_the_call_on_one_compact_line(self, caplog: pytest.LogCaptureFixture) -> None:+ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await _StandInLLMWorker(usage=LLM_USAGE).gen_text(llm_job=_llm_job())+ with pytest.raises(LLMCompletionError):+ await _StandInLLMWorker(error=LLMCompletionError(message="provider refused")).gen_text(llm_job=_llm_job())++ succeeded, failed = _summaries(caplog)+ assert rendered_text(record=succeeded).plain == "🧠: gpt-test chat · 1,200 → 300 tokens · $0.0018 · done in 1.25 s"+ assert rendered_text(record=failed).plain == "🧠: gpt-test chat · failed after 1.25 s error.type=LLMCompletionError"++ @pytest.mark.parametrize(("refuse_call", "refusal_type", "operation"), REFUSAL_CASES, ids=REFUSAL_IDS)+ @pytest.mark.asyncio+ async def test_a_call_the_checks_refuse_ends_with_the_event_too(+ self,+ caplog: pytest.LogCaptureFixture,+ refuse_call: Callable[[], Awaitable[None]],+ refusal_type: type[Exception],+ operation: InferenceOperation,+ ) -> None:+ """A refusal before the provider is called is a call that failed: it ends with the event, its error type and no usage."""+ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER), pytest.raises(refusal_type):+ await refuse_call()++ (record,) = _summaries(caplog)+ fields = _fields(record)+ assert fields[GenAISpanAttr.OPERATION_NAME] == operation+ assert (fields["outcome"], fields["error.type"], fields["duration_ms"]) == ("error", refusal_type.__name__, DURATION_MS)+ assert {GenAISpanAttr.USAGE_INPUT_TOKENS, GenAISpanAttr.USAGE_OUTPUT_TOKENS, "cost_usd"}.isdisjoint(fields)++ @pytest.mark.asyncio+ async def test_the_llm_event_carries_the_calls_span_and_a_refused_call_names_none(+ self, caplog: pytest.LogCaptureFixture, span_exporter: InMemorySpanExporter, json_lines: io.StringIO+ ) -> None:+ otel_context = OtelContext(trace_id=RUN_TRACE_ID, trace_name="some_pipe", trace_name_redacted="some_pipe", span_id=PIPE_SPAN_ID)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await _StandInLLMWorker(usage=LLM_USAGE).gen_text(llm_job=_llm_job(otel_context=otel_context))+ with pytest.raises(LLMCapabilityError):+ await _StandInLLMWorker(usage=LLM_USAGE).gen_text(+ llm_job=_llm_job(otel_context=otel_context, llm_prompt=LLMPrompt(user_text="describe it", user_images=[CAT_PICTURE]))+ )++ (span,) = span_exporter.get_finished_spans()+ span_context = span.get_span_context()+ assert span_context is not None+ succeeded, refused = _read_json_lines(json_lines)+ assert (succeeded["outcome"], succeeded[PIPELEX_SPAN_ID_KEY]) == ("success", f"{span_context.span_id:016x}")+ assert refused["outcome"] == "error"+ assert PIPELEX_SPAN_ID_KEY not in refused++ @pytest.mark.asyncio+ async def test_a_cancelled_call_ends_with_a_cancelled_outcome_and_no_error_type(self, caplog: pytest.LogCaptureFixture) -> None:+ """A call cancelled from outside, as a batch cancels its siblings when one fails, is not a failure of the call."""+ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER), pytest.raises(asyncio.CancelledError):+ await _StandInLLMWorker(error=asyncio.CancelledError()).gen_text(llm_job=_llm_job())++ (record,) = _summaries(caplog)+ fields = _fields(record)+ assert (fields["outcome"], fields["duration_ms"]) == ("cancelled", DURATION_MS)+ assert "error.type" not in fields+ assert rendered_text(record=record).plain == "🧠: gpt-test chat · cancelled after 1.25 s"++ @pytest.mark.parametrize("provider_error", [None, LLMCompletionError(message="provider refused")], ids=["the call succeeds", "the call fails"])+ @pytest.mark.asyncio+ async def test_an_event_that_cannot_be_priced_never_changes_the_calls_outcome(+ self, caplog: pytest.LogCaptureFixture, provider_error: LLMCompletionError | None+ ) -> None:+ """A usage the cost engine cannot price is logged once as a warning; the call's result or its own error goes on."""+ worker = _StandInLLMWorker(usage=BROKEN_USAGE, error=provider_error)++ with caplog.at_level(logging.INFO):+ if provider_error is None:+ assert await worker.gen_text(llm_job=_llm_job()) == "answer"+ else:+ with pytest.raises(LLMCompletionError, match="provider refused") as raised:+ await worker.gen_text(llm_job=_llm_job())+ assert raised.value is provider_error++ assert not _summaries(caplog)+ (failure,) = [record for record in caplog.records if record.getMessage() == SUMMARY_EVENT_FAILED_MESSAGE]+ failure_fields = _fields(failure)+ assert failure.levelno == logging.WARNING+ assert (failure_fields["summary_event"], failure_fields["error.type"]) == (INFERENCE_CALL_ENDS_MESSAGE, "TypeError")++ @pytest.mark.asyncio+ async def test_the_model_names_are_read_when_the_call_ends(self, caplog: pytest.LogCaptureFixture, span_exporter: InMemorySpanExporter) -> None:+ """A worker routing the call in `_before_job` is named as its span names it, and the model the provider served is the one named."""+ otel_context = OtelContext(trace_id=RUN_TRACE_ID, trace_name="some_pipe", trace_name_redacted="some_pipe", span_id=PIPE_SPAN_ID)++ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ await _RoutingLLMWorker().gen_text(llm_job=_llm_job(otel_context=otel_context))++ (record,) = _summaries(caplog)+ (span,) = span_exporter.get_finished_spans()+ assert span.attributes is not None+ fields = _fields(record)+ assert fields[GenAISpanAttr.REQUEST_MODEL] == span.attributes[GenAISpanAttr.REQUEST_MODEL] == ROUTED_MODEL+ assert fields[GenAISpanAttr.RESPONSE_MODEL] == SERVED_MODELdiff --git a/tests/unit/pipelex/cogt/test_worker_error_enrichment.py b/tests/unit/pipelex/cogt/test_worker_error_enrichment.pyindex fe3ce4268..b86777967 100644--- a/tests/unit/pipelex/cogt/test_worker_error_enrichment.py+++ b/tests/unit/pipelex/cogt/test_worker_error_enrichment.py@@ -135,6 +135,7 @@ def _make_inference_model(backend_name: str = WORKER_PROVIDER) -> SimpleNamespacreturn SimpleNamespace(name=WORKER_MODEL,backend_name=backend_name,+ sdk="stub-sdk",model_id=WORKER_MODEL,tag="stub-tag",desc="stub-desc",diff --git a/tests/unit/pipelex/cogt/usage/test_cost_registry.py b/tests/unit/pipelex/cogt/usage/test_cost_registry.pyindex 3d2a7c7b4..cf3f71e20 100644--- a/tests/unit/pipelex/cogt/usage/test_cost_registry.py+++ b/tests/unit/pipelex/cogt/usage/test_cost_registry.py@@ -430,15 +430,6 @@ class TestCostRegistry:actual_cost = float(cost_str)assert abs(actual_cost - expected_scaled_cost) < 0.0001- def test_compute_total_cost(self):- """Test total cost computation."""- total = CostRegistry.compute_total_cost(- input_non_cached_cost=1.0,- input_cached_cost=0.5,- output_cost=2.0,- )- assert total == 3.5-def test_generate_report_print_to_console_false_skips_console(self, job_metadata: JobMetadata, mocker: MockerFixture):"""When print_to_console=False, the console.print is not called."""mock_console = mocker.MagicMock()diff --git a/tests/unit/pipelex/cogt/usage/test_usage_cost.py b/tests/unit/pipelex/cogt/usage/test_usage_cost.pyindex 337021529..7be8d224a 100644--- a/tests/unit/pipelex/cogt/usage/test_usage_cost.py+++ b/tests/unit/pipelex/cogt/usage/test_usage_cost.py@@ -1,12 +1,17 @@import pytestfrom pipelex.cogt.usage.cost_category import CostCategory-from pipelex.cogt.usage.cost_registry import CostRegistry, compute_tokens_usage_cost+from pipelex.cogt.usage.cost_registry import CostRegistry+from pipelex.cogt.usage.usage_cost import compute_tokens_usage_cost, compute_total_costfrom pipelex.reporting.reporting_types import AnyTokensUsagefrom tests.unit.pipelex.cogt.usage.test_data import RATED_EXPECTED_COST, UsageFixturesclass TestComputeTokensUsageCost:+ def test_total_cost_adds_up_the_components(self):+ """A call's total is what it read uncached, plus what it read from the cache, plus what it wrote."""+ assert compute_total_cost(input_non_cached_cost=1.0, input_cached_cost=0.5, output_cost=2.0) == 3.5+def test_none_when_unrated(self):"""No rate table (own-GPU, dry/mock run) means no cost claim — None, not 0.0."""assert compute_tokens_usage_cost(UsageFixtures.unrated_usage()) is None@@ -27,7 +32,7 @@ class TestComputeTokensUsageCost:def test_parity_with_cost_registry_per_record(self, tokens_usage: AnyTokensUsage):"""The wire cost equals the canonical CostRegistry total for the same record — one cost engine, no drift."""cost_report = CostRegistry.complete_cost_report(tokens_usage=tokens_usage)- registry_total = CostRegistry.compute_total_cost(+ registry_total = compute_total_cost(input_non_cached_cost=cost_report.costs_by_token_category[CostCategory.INPUT_NON_CACHED],input_cached_cost=cost_report.costs_by_token_category[CostCategory.INPUT_CACHED],output_cost=cost_report.costs_by_token_category.get(CostCategory.OUTPUT, 0.0),diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_end.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_end.pynew file mode 100644index 000000000..1740347c5--- /dev/null+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_end.py@@ -0,0 +1,373 @@+from __future__ import annotations++import asyncio+import io+import json+import logging+from typing import TYPE_CHECKING, Any++import pytest+from opentelemetry.sdk.trace import TracerProvider+from opentelemetry.sdk.trace.export import SimpleSpanProcessor+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter+from typing_extensions import override++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.pipes.exceptions import PipeRunError+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec+from pipelex.pipe_machinery.pipe_abstract import PIPE_RUN_ENDS_MESSAGE, PIPE_RUN_STARTS_MESSAGE, PipeAbstract+from pipelex.pipe_run.pipe_run_params import PipeRunParams+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY+from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract+from pipelex.tools.log.console_layouts import LogLayout+from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, JsonLogSink+from pipelex.tools.log.log_fields import LAYOUT_MARK, attached_field_names+from tests.helpers.console_log_rendering import rendered_text++if TYPE_CHECKING:+ from collections.abc import Iterator++ from opentelemetry.sdk.trace import ReadableSpan+ from pytest_mock import MockerFixture++ from pipelex.core.memory.working_memory import WorkingMemory+ from pipelex.libraries.library_crate import LibraryCrate++PIPE_LOGGER = PipeAbstract.__module__+STARTED_AT = 10.0+ENDED_AT = 11.25+DURATION_MS = 1250.0+# The trace the submission derived for the run, under the virtual root parent every root pipe span takes.+RUN_OTEL_CONTEXT = OtelContext(+ trace_id=0x0123456789ABCDEF0123456789ABCDEF,+ trace_name="pipe_run_end_test",+ trace_name_redacted="pipe_run_end_test",+ span_id=OTelConstants.OTEL_VIRTUAL_ROOT_PARENT_SPAN_ID,+)+++class EndingPipe(PipeAbstract):+ """A pipe whose live run runs its child pipe, if it has one, then returns or fails; its dry run returns."""++ pipe_category: Any = "PipeOperator"+ type: Any = "PipeFunc"+ child: EndingPipe | None = None+ fails: bool = False+ is_cancelled: bool = False++ @override+ def validate_inputs_with_library(self) -> None: ...++ @override+ def validate_inputs_static(self) -> None: ...++ @override+ def validate_output_with_library(self) -> None: ...++ @override+ def validate_output_static(self) -> None: ...++ @override+ async def _validate_before_run(self, **kwargs: Any) -> None: ...++ @override+ async def _validate_after_run(self, **kwargs: Any) -> None: ...++ @override+ def required_variables(self) -> set[str]:+ return set()++ @override+ def needed_inputs(self, *, visited_pipes: set[str] | None = None) -> Any:+ return self.inputs++ @override+ async def _live_run_pipe(+ self,+ *,+ job_metadata: JobMetadata,+ working_memory: WorkingMemory,+ pipe_run_params: PipeRunParams,+ output_name: str | None = None,+ library_crate: LibraryCrate | None = None,+ ) -> PipeOutput:+ if self.child is not None:+ await self.child.run_pipe(job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params)+ if self.fails:+ msg = f"{self.code} failed"+ raise PipeRunError(msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)+ if self.is_cancelled:+ raise asyncio.CancelledError+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)++ @override+ async def _dry_run_pipe(+ self,+ *,+ job_metadata: JobMetadata,+ working_memory: WorkingMemory,+ pipe_run_params: PipeRunParams,+ output_name: str | None = None,+ library_crate: LibraryCrate | None = None,+ ) -> PipeOutput:+ if self.child is not None:+ await self.child.run_pipe(job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params)+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)+++def _make_pipe(*, code: str, child: EndingPipe | None = None, fails: bool = False, is_cancelled: bool = False) -> EndingPipe:+ return EndingPipe(+ code=code,+ domain_code="test_pipe_run_end",+ description=code,+ output=StuffSpec(concept=ConceptFactory.make_native_concept(NativeConceptCode.TEXT)),+ child=child,+ fails=fails,+ is_cancelled=is_cancelled,+ )+++def _job_metadata(*, otel_context: OtelContext | None = None) -> JobMetadata:+ return JobMetadata(+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id="plr-end"),+ otel_context=otel_context,+ )+++def _pipe_run_params(*, run_mode: PipeRunMode) -> PipeRunParams:+ return PipeRunParams(run_mode=run_mode, batch_max_concurrency=1, pipe_stack_limit=10)+++async def _run(*, pipe: EndingPipe, run_mode: PipeRunMode = PipeRunMode.LIVE, otel_context: OtelContext | None = None) -> None:+ await pipe.run_pipe(+ job_metadata=_job_metadata(otel_context=otel_context),+ working_memory=WorkingMemoryFactory.make_empty(),+ pipe_run_params=_pipe_run_params(run_mode=run_mode),+ )+++def _records(caplog: pytest.LogCaptureFixture, *, message: str) -> list[logging.LogRecord]:+ return [record for record in caplog.records if record.name == PIPE_LOGGER and record.getMessage() == message]+++def _field(record: logging.LogRecord, *, name: str) -> Any:+ """A field carried on the record, read the way a sink reads it."""+ return getattr(record, name)+++def _fields(record: logging.LogRecord) -> dict[str, Any]:+ """The fields the call attached to the record, without the context identifiers the run bound."""+ return {name: getattr(record, name) for name in attached_field_names(record=record) if name != "pipe_run_id"}+++def _span_id_of(spans: tuple[ReadableSpan, ...], *, code: str) -> str:+ """The span id of the pipe span named after `code`, as the `json` sink writes it."""+ (span,) = [span for span in spans if span.name.endswith(f": {code}")]+ span_context = span.get_span_context()+ assert span_context is not None+ return f"{span_context.span_id:016x}"+++def fixed_clock(mocker: MockerFixture) -> None:+ """Every run starts at the same reading and ends 1.25 s later, so its duration is known."""+ mocker.patch("pipelex.tools.log.summary_event.start_clock", return_value=STARTED_AT)+ mocker.patch("pipelex.tools.log.summary_fields.perf_counter", return_value=ENDED_AT)+++def span_exporter(mocker: MockerFixture) -> InMemorySpanExporter:+ """The runtime's tracer, replaced by one exporting to memory, so the test reads back the spans the pipes started."""+ exporter = InMemorySpanExporter()+ provider = TracerProvider()+ provider.add_span_processor(SimpleSpanProcessor(exporter))+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=provider.get_tracer(__name__))+ return exporter+++def json_lines() -> Iterator[io.StringIO]:+ """The `json` sink's lines for every record the pipe logger writes, written as it logs them, with the span active then."""+ buffer = io.StringIO()+ handler = JsonLogSink(stream=buffer).handler+ pipe_logger = logging.getLogger(PIPE_LOGGER)+ pipe_logger.addHandler(handler)+ try:+ yield buffer+ finally:+ pipe_logger.removeHandler(handler)+++def _run_end_lines(buffer: io.StringIO) -> list[dict[str, Any]]:+ """The `json` lines of the events the pipe runs ended with, in the order they ended."""+ lines = [json.loads(line) for line in buffer.getvalue().splitlines() if line]+ return [line for line in lines if line[MESSAGE_KEY] == PIPE_RUN_ENDS_MESSAGE]++[email protected]("fixed_clock")+class TestLiveRunPipeEnd:+ async def test_a_run_that_succeeds_ends_with_the_event_under_its_announcement_id(self, caplog: pytest.LogCaptureFixture) -> None:+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ await _run(pipe=_make_pipe(code="describe_company"))++ (announcement,) = _records(caplog, message=PIPE_RUN_STARTS_MESSAGE)+ (run_end,) = _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)+ assert run_end.levelno == logging.INFO+ assert run_end.__dict__[LAYOUT_MARK] == LogLayout.PIPE_RUN_END+ assert _field(run_end, name="pipe_run_id") == _field(announcement, name="pipe_run_id")+ assert _fields(run_end) == {+ "pipe_type": "EndingPipe",+ "pipe_code": "describe_company",+ "output_concept": "Text",+ "pipe_depth": 0,+ "duration_ms": DURATION_MS,+ "outcome": "success",+ }++ async def test_a_run_that_fails_ends_with_its_outcome_and_error_type(self, caplog: pytest.LogCaptureFixture) -> None:+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER), pytest.raises(PipeRunError, match="describe_company failed"):+ await _run(pipe=_make_pipe(code="describe_company", fails=True))++ (run_end,) = _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)+ assert _fields(run_end) == {+ "pipe_type": "EndingPipe",+ "pipe_code": "describe_company",+ "output_concept": "Text",+ "pipe_depth": 0,+ "duration_ms": DURATION_MS,+ "outcome": "error",+ "error.type": "PipeRunError",+ }++ @pytest.mark.parametrize("child_fails", [False, True], ids=["the nested run succeeds", "the nested run fails"])+ async def test_every_announced_run_ends_once_the_nested_one_first_at_its_own_depth(+ self, caplog: pytest.LogCaptureFixture, child_fails: bool+ ) -> None:+ pipe = _make_pipe(code="build_profile", child=_make_pipe(code="describe_company", fails=child_fails))++ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ if child_fails:+ with pytest.raises(PipeRunError):+ await _run(pipe=pipe)+ else:+ await _run(pipe=pipe)++ announcements = _records(caplog, message=PIPE_RUN_STARTS_MESSAGE)+ run_ends = _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)+ outcome = "error" if child_fails else "success"+ assert [(_field(record, name="pipe_code"), _field(record, name="pipe_depth")) for record in announcements] == [+ ("build_profile", 0),+ ("describe_company", 1),+ ]+ assert [(_field(record, name="pipe_code"), _field(record, name="pipe_depth"), _field(record, name="outcome")) for record in run_ends] == [+ ("describe_company", 1, outcome),+ ("build_profile", 0, outcome),+ ]++ async def test_a_dry_run_ends_with_no_event_as_it_starts_with_no_announcement(self, caplog: pytest.LogCaptureFixture) -> None:+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ await _run(pipe=_make_pipe(code="build_profile", child=_make_pipe(code="describe_company")), run_mode=PipeRunMode.DRY)++ assert not _records(caplog, message=PIPE_RUN_STARTS_MESSAGE)+ assert not _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)++ async def test_the_console_draws_the_end_in_the_pipe_tree_at_its_depth(self, caplog: pytest.LogCaptureFixture) -> None:+ pipe = _make_pipe(code="build_profile", child=_make_pipe(code="describe_company"))+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ await _run(pipe=pipe)+ with pytest.raises(PipeRunError):+ await _run(pipe=_make_pipe(code="describe_company", fails=True))++ lines = [rendered_text(record=record).plain for record in _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)]+ assert lines == [+ "🧠: ↳ EndingPipe: describe_company done in 1.25 s",+ "🧠: EndingPipe: build_profile done in 1.25 s",+ "🧠: EndingPipe: describe_company failed after 1.25 s error.type=PipeRunError",+ ]++ async def test_the_json_sink_writes_the_duration_and_the_depth_as_numbers(self, caplog: pytest.LogCaptureFixture) -> None:+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ await _run(pipe=_make_pipe(code="describe_company"))+ (run_end,) = _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)+ buffer = io.StringIO()++ JsonLogSink(stream=buffer).handler.handle(run_end)++ line: dict[str, Any] = json.loads(buffer.getvalue())+ assert (line[MESSAGE_KEY], line[LOGGER_KEY]) == (PIPE_RUN_ENDS_MESSAGE, PIPE_LOGGER)+ assert isinstance(line["duration_ms"], float)+ assert line["duration_ms"] == DURATION_MS+ assert isinstance(line["pipe_depth"], int)+ assert (line["pipe_code"], line["outcome"]) == ("describe_company", "success")+ assert "error.type" not in line+ assert LogLayout.PIPE_RUN_END not in line.values()++ @pytest.mark.parametrize("fails", [False, True], ids=["the run succeeds", "the run fails"])+ async def test_a_root_runs_end_carries_its_own_span(+ self, caplog: pytest.LogCaptureFixture, span_exporter: InMemorySpanExporter, json_lines: io.StringIO, fails: bool+ ) -> None:+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ if fails:+ with pytest.raises(PipeRunError):+ await _run(pipe=_make_pipe(code="describe_company", fails=True), otel_context=RUN_OTEL_CONTEXT)+ else:+ await _run(pipe=_make_pipe(code="describe_company"), otel_context=RUN_OTEL_CONTEXT)++ (run_end,) = _run_end_lines(json_lines)+ assert (run_end["pipe_code"], run_end["outcome"]) == ("describe_company", "error" if fails else "success")+ assert run_end[PIPELEX_SPAN_ID_KEY] == _span_id_of(span_exporter.get_finished_spans(), code="describe_company")++ @pytest.mark.parametrize("child_fails", [False, True], ids=["the nested run succeeds", "the nested run fails"])+ async def test_a_nested_runs_end_carries_its_own_span_and_its_parents_end_the_parents(+ self, caplog: pytest.LogCaptureFixture, span_exporter: InMemorySpanExporter, json_lines: io.StringIO, child_fails: bool+ ) -> None:+ pipe = _make_pipe(code="build_profile", child=_make_pipe(code="describe_company", fails=child_fails))++ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER):+ if child_fails:+ with pytest.raises(PipeRunError):+ await _run(pipe=pipe, otel_context=RUN_OTEL_CONTEXT)+ else:+ await _run(pipe=pipe, otel_context=RUN_OTEL_CONTEXT)++ spans = span_exporter.get_finished_spans()+ child_end, parent_end = _run_end_lines(json_lines)+ assert (child_end["pipe_code"], child_end[PIPELEX_SPAN_ID_KEY]) == ("describe_company", _span_id_of(spans, code="describe_company"))+ assert (parent_end["pipe_code"], parent_end[PIPELEX_SPAN_ID_KEY]) == ("build_profile", _span_id_of(spans, code="build_profile"))++ async def test_a_failure_before_the_span_still_ends_the_run_once(+ self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture, json_lines: io.StringIO+ ) -> None:+ """The setup between the announcement and the span can fail; the run still ends with one event, outside any span."""+ mocker.patch.object(EndingPipe, "_start_pipe_span", side_effect=RuntimeError("the tracer broke"))++ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER), pytest.raises(RuntimeError, match="the tracer broke"):+ await _run(pipe=_make_pipe(code="describe_company"), otel_context=RUN_OTEL_CONTEXT)++ (run_end,) = _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)+ assert (_field(run_end, name="outcome"), _field(run_end, name="error.type")) == ("error", "RuntimeError")+ (run_end_line,) = _run_end_lines(json_lines)+ assert PIPELEX_SPAN_ID_KEY not in run_end_line++ async def test_a_cancelled_run_ends_with_a_cancelled_outcome_and_no_error_type(self, caplog: pytest.LogCaptureFixture) -> None:+ """A run cancelled from outside, a sibling of a failed batch item or a run its client cancelled, did not fail."""+ with caplog.at_level(logging.INFO, logger=PIPE_LOGGER), pytest.raises(asyncio.CancelledError):+ await _run(pipe=_make_pipe(code="describe_company", is_cancelled=True))++ (run_end,) = _records(caplog, message=PIPE_RUN_ENDS_MESSAGE)+ assert _fields(run_end) == {+ "pipe_type": "EndingPipe",+ "pipe_code": "describe_company",+ "output_concept": "Text",+ "pipe_depth": 0,+ "duration_ms": DURATION_MS,+ "outcome": "cancelled",+ }+ assert rendered_text(record=run_end).plain == "🧠: EndingPipe: describe_company cancelled after 1.25 s"diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.pyindex 3f1915b31..ca70e32a2 100644--- a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py@@ -23,7 +23,7 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec-from pipelex.pipe_machinery.pipe_abstract import PipeAbstract+from pipelex.pipe_machinery.pipe_abstract import PIPE_RUN_ENDS_MESSAGE, PIPE_RUN_STARTS_MESSAGE, PipeAbstractfrom pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.system.job_metadata import JobMetadata, RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunMode@@ -158,7 +158,12 @@ def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:def _announcements(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:"""The lines ``live_run_pipe`` itself emits to announce each pipe, on the pipe machinery's own logger."""- return [record for record in caplog.records if record.name == PipeAbstract.__module__ and record.levelno == logging.INFO]+ return [record for record in caplog.records if record.name == PipeAbstract.__module__ and record.getMessage() == PIPE_RUN_STARTS_MESSAGE]+++def _run_ends(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:+ """The events ``live_run_pipe`` ends each pipe run with, on the same logger, in the order the runs ended."""+ return [record for record in caplog.records if record.name == PipeAbstract.__module__ and record.getMessage() == PIPE_RUN_ENDS_MESSAGE]def _field(record: logging.LogRecord, *, name: str) -> Any:@@ -189,6 +194,8 @@ class TestLiveRunPipeLogContext:assert succeeded.context == running.context(announcement,) = _announcements(caplog)assert _field(announcement, name="pipe_run_id") == minted_id+ (run_end,) = _run_ends(caplog)+ assert _field(run_end, name="pipe_run_id") == minted_id(record,) = _own_records(caplog)assert _field(record, name="pipe_run_id") == minted_idassert _field(record, name="request_id") == "req-live"@@ -215,6 +222,9 @@ class TestLiveRunPipeLogContext:outer_announcement, inner_announcement = _announcements(caplog)assert _field(outer_announcement, name="pipe_run_id") == outer_idassert _field(inner_announcement, name="pipe_run_id") == inner_id+ inner_end, outer_end = _run_ends(caplog)+ assert _field(inner_end, name="pipe_run_id") == inner_id+ assert _field(outer_end, name="pipe_run_id") == outer_idouter_record, inner_record, back_record = _own_records(caplog)assert _field(outer_record, name="pipe_run_id") == outer_idassert _field(inner_record, name="pipe_run_id") == inner_id@@ -246,3 +256,6 @@ class TestLiveRunPipeLogContext:assert _field(inner_error, name="pipe_run_id") == inner_idassert outer_error.getMessage() == "outer span ends in error"assert _field(outer_error, name="pipe_run_id") == outer_id+ inner_end, outer_end = _run_ends(caplog)+ assert (_field(inner_end, name="pipe_run_id"), _field(inner_end, name="outcome")) == (inner_id, "error")+ assert (_field(outer_end, name="pipe_run_id"), _field(outer_end, name="outcome")) == (outer_id, "error")diff --git a/tests/unit/pipelex/providers/linkup/test_linkup_request_pricing.py b/tests/unit/pipelex/providers/linkup/test_linkup_request_pricing.pynew file mode 100644index 000000000..312f0555a--- /dev/null+++ b/tests/unit/pipelex/providers/linkup/test_linkup_request_pricing.py@@ -0,0 +1,142 @@+"""Linkup bills by the request, so its workers record each call as one request priced at the model's rate.++The rates are per million tokens, so a request is recorded as a million tokens in and out, and the usage says it counts+requests. The event every inference call ends with keeps the request's cost and writes no token counts, which would+otherwise add a million tokens to a dashboard of tokens for every search. No provider is called: the SDK client is+mocked, so this needs no inference marker.+"""++from __future__ import annotations++import logging+import math+from typing import TYPE_CHECKING, Any++import pytest+from linkup import LinkupFetchResponse, LinkupSourcedAnswer+from pydantic import BaseModel++from pipelex.cogt.extract.extract_input import ExtractInput+from pipelex.cogt.extract.extract_job import ExtractJob+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams, ExtractJobReport+from pipelex.cogt.inference.inference_call_summary import INFERENCE_CALL_ENDS_MESSAGE+from pipelex.cogt.llm.thinking_mode import ThinkingMode+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.search.search_job_factory import SearchJobFactory+from pipelex.cogt.search.search_setting import SearchSetting+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.pricing_unit import PricingUnit+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.providers.linkup.linkup_extract_worker import LinkupExtractWorker+from pipelex.providers.linkup.linkup_search_worker import LinkupSearchWorker+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.telemetry.otel_constants import GenAISpanAttr+from pipelex.tools.log.log_fields import attached_field_names++if TYPE_CHECKING:+ from collections.abc import Awaitable, Callable++ from pytest_mock import MockerFixture++ from pipelex.cogt.usage.usage_cost import TokensUsage++SUMMARY_LOGGER = "pipelex.cogt.inference.inference_call_summary"+# Linkup's rate, per million tokens, which prices one request since a request is recorded as a million tokens.+REQUEST_RATE = 0.005+ONE_REQUEST = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}+++class CompanyBrief(BaseModel):+ name: str+ summary: str+++def _model(*, model_type: ModelType, name: str, model_id: str, inputs: list[str], outputs: list[str]) -> InferenceModelSpec:+ return InferenceModelSpec(+ backend_name="linkup",+ name=name,+ sdk="linkup",+ model_type=model_type,+ model_id=model_id,+ inputs=inputs,+ outputs=outputs,+ costs={CostCategory.INPUT: REQUEST_RATE, CostCategory.OUTPUT: 0.0},+ thinking_mode=ThinkingMode.NONE,+ max_tokens=None,+ max_prompt_images=None,+ )+++def _job_metadata() -> JobMetadata:+ return JobMetadata(+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", read_scope=None, pipeline_run_id="plr-linkup"),+ pipe_code="some_pipe",+ )+++def _search_worker(mocker: MockerFixture, *, response: Any) -> LinkupSearchWorker:+ linkup_client = mocker.patch("pipelex.providers.linkup.linkup_search_worker.LinkupClient").return_value+ linkup_client.async_search = mocker.AsyncMock(return_value=response)+ model = _model(+ model_type=ModelType.SEARCH, name="linkup-standard", model_id="standard", inputs=["text"], outputs=["sourced-answers", "structured"]+ )+ return LinkupSearchWorker(inference_model=model, api_key="test-key")+++async def _search_sourced_answer(mocker: MockerFixture) -> TokensUsage | None:+ worker = _search_worker(mocker, response=LinkupSourcedAnswer(answer="Acme makes anvils.", sources=[]))+ search_job = SearchJobFactory.make_search_job(+ "what does Acme make", search_setting=SearchSetting(model="linkup-standard"), job_metadata=_job_metadata()+ )+ await worker.search_sourced_answer(search_job=search_job)+ return search_job.job_report.search_tokens_usage+++async def _search_structured(mocker: MockerFixture) -> TokensUsage | None:+ worker = _search_worker(mocker, response={"name": "Acme", "summary": "Acme makes anvils."})+ search_job = SearchJobFactory.make_search_job(+ "what does Acme make", search_setting=SearchSetting(model="linkup-standard"), job_metadata=_job_metadata()+ )+ await worker.search_structured(search_job=search_job, schema=CompanyBrief)+ return search_job.job_report.search_tokens_usage+++async def _fetch_page(mocker: MockerFixture) -> TokensUsage | None:+ linkup_client = mocker.patch("pipelex.providers.linkup.linkup_extract_worker.LinkupClient").return_value+ linkup_client.async_fetch = mocker.AsyncMock(return_value=LinkupFetchResponse(markdown="# Acme"))+ model = _model(model_type=ModelType.TEXT_EXTRACTOR, name="linkup-fetch", model_id="fetch", inputs=["web_page"], outputs=["pages"])+ worker = LinkupExtractWorker(extra_config={}, inference_model=model, api_key="test-key")+ extract_job = ExtractJob(+ extract_input=ExtractInput(document_uri="https://example.com/acme"),+ job_params=ExtractJobParams.make_default_extract_job_params(),+ job_config=ExtractJobConfig(),+ job_report=ExtractJobReport(),+ job_metadata=_job_metadata(),+ )+ await worker.extract_pages(extract_job=extract_job)+ return extract_job.job_report.extract_tokens_usage++[email protected](loop_scope="class")+class TestLinkupRequestPricing:+ @pytest.mark.parametrize(+ "run_call",+ [_search_sourced_answer, _search_structured, _fetch_page],+ ids=["a sourced-answer search", "a structured search", "a page fetch"],+ )+ async def test_a_linkup_call_is_priced_as_one_request_and_its_event_writes_no_token_counts(+ self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture, run_call: Callable[[MockerFixture], Awaitable[TokensUsage | None]]+ ) -> None:+ with caplog.at_level(logging.INFO, logger=SUMMARY_LOGGER):+ tokens_usage = await run_call(mocker)++ assert tokens_usage is not None+ assert tokens_usage.nb_tokens_by_category == ONE_REQUEST+ assert tokens_usage.pricing_unit is PricingUnit.REQUEST+ (record,) = [record for record in caplog.records if record.name == SUMMARY_LOGGER and record.getMessage() == INFERENCE_CALL_ENDS_MESSAGE]+ fields = {name: getattr(record, name) for name in attached_field_names(record=record)}+ assert fields["outcome"] == "success"+ assert math.isclose(fields["cost_usd"], REQUEST_RATE)+ assert GenAISpanAttr.USAGE_INPUT_TOKENS not in fields+ assert GenAISpanAttr.USAGE_OUTPUT_TOKENS not in fieldsdiff --git a/tests/unit/pipelex/tools/test_log_console_layouts.py b/tests/unit/pipelex/tools/test_log_console_layouts.pyindex 33da4674e..f60c01dde 100644--- a/tests/unit/pipelex/tools/test_log_console_layouts.py+++ b/tests/unit/pipelex/tools/test_log_console_layouts.py@@ -24,6 +24,24 @@ from tests.helpers.console_log_rendering import (if TYPE_CHECKING:from pytest_mock import MockerFixture+# The fields of the event an inference call ends with, for a call that reported its usage and its cost.+INFERENCE_CALL_END_FIELDS: dict[str, Any] = {+ "gen_ai.operation.name": "chat",+ "model_handle": "gpt-5.6-terra",+ "backend_name": "openai",+ "sdk": "openai_responses",+ "gen_ai.request.model": "gpt-5.6-terra",+ "gen_ai.response.model": "gpt-5.6-terra-2026-08-01",+ "gen_ai.usage.input_tokens": 12288,+ "gen_ai.usage.output_tokens": 567,+ "cost_usd": 0.0123,+ "duration_ms": 1234.5,+ "outcome": "success",+}+# The fields of the event a pipe run ends with, for a top-level run that succeeded.+PIPE_RUN_END_FIELDS: dict[str, Any] = {**PIPE_RUN_FIELDS, "duration_ms": 840.0, "outcome": "success"}+USAGE_KEYS = ("gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens", "cost_usd")+class TestConsoleLayouts:@pytest.mark.parametrize(@@ -42,6 +60,87 @@ class TestConsoleLayouts:assert styles_of(text=text, fragment="compose_company") == [FIELD_STYLES["pipe_code"]]assert styles_of(text=text, fragment="Company") == [FIELD_STYLES["output_concept"]]+ @pytest.mark.parametrize(+ ("overrides", "dropped", "expected"),+ [+ ({}, (), "gpt-5.6-terra chat · 12,288 → 567 tokens · $0.0123 · done in 1.23 s"),+ ({"duration_ms": 840.4}, (), "gpt-5.6-terra chat · 12,288 → 567 tokens · $0.0123 · done in 840 ms"),+ ({"duration_ms": 999.6}, (), "gpt-5.6-terra chat · 12,288 → 567 tokens · $0.0123 · done in 1.00 s"),+ ({}, ("gen_ai.usage.output_tokens",), "gpt-5.6-terra chat · 12,288 tokens in · $0.0123 · done in 1.23 s"),+ ({}, ("gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens"), "gpt-5.6-terra chat · $0.0123 · done in 1.23 s"),+ ({}, ("cost_usd",), "gpt-5.6-terra chat · 12,288 → 567 tokens · done in 1.23 s"),+ ({"gen_ai.operation.name": "doc_gen", "model_handle": "reportlab-pdf"}, USAGE_KEYS, "reportlab-pdf doc_gen · done in 1.23 s"),+ ({"cost_usd": 0.0}, (), "gpt-5.6-terra chat · 12,288 → 567 tokens · $0 · done in 1.23 s"),+ ({"cost_usd": 0.0000004}, (), "gpt-5.6-terra chat · 12,288 → 567 tokens · <$0.000001 · done in 1.23 s"),+ ({"cost_usd": 0.0000006}, (), "gpt-5.6-terra chat · 12,288 → 567 tokens · $0.000001 · done in 1.23 s"),+ (+ {"outcome": "error", "error.type": "LLMCompletionError"},+ USAGE_KEYS,+ "gpt-5.6-terra chat · failed after 1.23 s error.type=LLMCompletionError",+ ),+ ({"outcome": "cancelled"}, USAGE_KEYS, "gpt-5.6-terra chat · cancelled after 1.23 s"),+ ],+ ids=[+ "tokens and cost",+ "under a second",+ "a second once rounded",+ "tokens in alone",+ "a cost alone",+ "tokens alone",+ "no usage at all",+ "a free call",+ "a cost below the last digit shown",+ "a cost rounding up to the last digit shown",+ "a failure",+ "a cancellation",+ ],+ )+ def test_the_inference_call_end_layout_draws_the_call_on_one_compact_line(+ self, overrides: dict[str, Any], dropped: tuple[str, ...], expected: str+ ) -> None:+ fields = {key: value for key, value in {**INFERENCE_CALL_END_FIELDS, **overrides}.items() if key not in dropped}++ text = rendered_text(record=record_with_fields(message="Inference call ends", extra=fields, layout=LogLayout.INFERENCE_CALL_END))++ assert text.plain == f"🧠: {expected}"++ @pytest.mark.parametrize(+ ("overrides", "expected"),+ [+ ({}, "EndingPipe: compose_company done in 840 ms"),+ ({"pipe_depth": 2, "duration_ms": 61_250.0}, " ↳ EndingPipe: compose_company done in 61.25 s"),+ ({"outcome": "error", "error.type": "PipeRunError"}, "EndingPipe: compose_company failed after 840 ms error.type=PipeRunError"),+ ({"outcome": "cancelled"}, "EndingPipe: compose_company cancelled after 840 ms"),+ ],+ ids=["top level", "nested twice", "a failure", "a cancellation"],+ )+ def test_the_pipe_run_end_layout_draws_the_end_in_the_pipe_tree(self, overrides: dict[str, Any], expected: str) -> None:+ fields = {**PIPE_RUN_END_FIELDS, "pipe_type": "EndingPipe", **overrides}++ text = rendered_text(record=record_with_fields(message="Pipe run ends", extra=fields, layout=LogLayout.PIPE_RUN_END))++ assert text.plain == f"🧠: {expected}"+ assert styles_of(text=text, fragment="compose_company") == [FIELD_STYLES["pipe_code"]]++ @pytest.mark.parametrize(+ ("layout", "fields", "overrides", "expected_values"),+ [+ (LogLayout.PIPE_RUN_END, PIPE_RUN_END_FIELDS, {"outcome": "partial"}, "outcome=partial"),+ (LogLayout.PIPE_RUN_END, PIPE_RUN_END_FIELDS, {"duration_ms": -1.0}, "duration_ms=-1.0"),+ (LogLayout.PIPE_RUN_END, PIPE_RUN_END_FIELDS, {"duration_ms": "slow"}, "duration_ms=slow"),+ (LogLayout.INFERENCE_CALL_END, INFERENCE_CALL_END_FIELDS, {"cost_usd": "cheap"}, "cost_usd=cheap"),+ (LogLayout.INFERENCE_CALL_END, INFERENCE_CALL_END_FIELDS, {"gen_ai.usage.input_tokens": 1.5}, "gen_ai.usage.input_tokens=1.5"),+ ],+ ids=["an unknown outcome", "a negative duration", "a duration that is no number", "a cost that is no number", "a fractional token count"],+ )+ def test_a_value_a_summary_layout_refuses_falls_back_to_the_message_and_every_field(+ self, layout: LogLayout, fields: dict[str, Any], overrides: dict[str, Any], expected_values: str+ ) -> None:+ text = rendered_text(record=record_with_fields(message="Work ends", extra={**fields, **overrides}, layout=layout))++ assert text.plain.startswith("🧠: Work ends ")+ assert expected_values in text.plain+def test_a_value_carrying_markup_or_an_emoji_code_prints_as_written_inside_a_layout(self) -> None:record = record_with_fields(message="Pipe run starts",diff --git a/tests/unit/pipelex/tools/test_log_summary_event.py b/tests/unit/pipelex/tools/test_log_summary_event.pynew file mode 100644index 000000000..0ebc42d26--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_summary_event.py@@ -0,0 +1,229 @@+from __future__ import annotations++import asyncio+import logging+from contextlib import contextmanager+from contextvars import ContextVar+from typing import TYPE_CHECKING, Any++import pytest+from typing_extensions import override++from pipelex import log+from pipelex.tools.log.log_fields import attached_field_names+from pipelex.tools.log.summary_event import SUMMARY_EVENT_FAILED_MESSAGE, SummaryEvent+from pipelex.tools.log.summary_fields import outcome_fields++if TYPE_CHECKING:+ from collections.abc import Generator, Iterator++ from pytest_mock import MockerFixture++WORK_ENDS_MESSAGE = "Work ends"+STARTED_AT = 10.0+ENDED_AT = 10.5+DURATION_MS = 500.0++# The block a line is logged in, standing for a context such as a span that only the inner block runs under.+_CURRENT_BLOCK: ContextVar[str] = ContextVar("current_block", default="outer")+++@contextmanager+def _inner_block() -> Generator[None]:+ token = _CURRENT_BLOCK.set("inner")+ try:+ yield+ finally:+ _CURRENT_BLOCK.reset(token)+++class WorkEnds(SummaryEvent):+ """A summary event whose work fields are given, or whose reading of them fails; it notes the block it is logged in."""++ def __init__(self, *, fields_error: Exception | None = None) -> None:+ super().__init__(message=WORK_ENDS_MESSAGE)+ self._fields_error = fields_error+ self.logged_in: list[str] = []++ @override+ def _work_fields(self) -> dict[str, Any]:+ if self._fields_error is not None:+ raise self._fields_error+ return {"work": "compose"}++ @override+ def _log_event(self, *, fields: dict[str, Any]) -> None:+ self.logged_in.append(_CURRENT_BLOCK.get())+ log.info(WORK_ENDS_MESSAGE, fields=fields)+++class _RefusingHandler(logging.Handler):+ """A host's handler that raises on every record it is handed, as one writing to a full disk would, noting each message."""++ def __init__(self) -> None:+ super().__init__(level=logging.INFO)+ self.refused: list[str] = []++ @override+ def emit(self, record: logging.LogRecord) -> None:+ self.refused.append(record.getMessage())+ msg = "No space left on device"+ raise OSError(msg)+++def _refuse_before_the_call() -> None:+ msg = "refused before the call"+ raise RuntimeError(msg)+++def _fields(record: logging.LogRecord) -> dict[str, Any]:+ return {name: getattr(record, name) for name in attached_field_names(record=record)}+++def _events(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:+ return [record for record in caplog.records if record.getMessage() == WORK_ENDS_MESSAGE]+++def _failures(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:+ return [record for record in caplog.records if record.getMessage() == SUMMARY_EVENT_FAILED_MESSAGE]+++def fixed_clock(mocker: MockerFixture) -> None:+ """Every unit of work starts at the same reading and ends half a second later."""+ mocker.patch("pipelex.tools.log.summary_event.start_clock", return_value=STARTED_AT)+ mocker.patch("pipelex.tools.log.summary_fields.perf_counter", return_value=ENDED_AT)+++def refusing_handler() -> Iterator[_RefusingHandler]:+ """A handler on the root logger that refuses every record at INFO and above, the event and its warning alike."""+ handler = _RefusingHandler()+ root_logger = logging.getLogger()+ root_logger.addHandler(handler)+ try:+ yield handler+ finally:+ root_logger.removeHandler(handler)++[email protected]("fixed_clock")+class TestLogSummaryEvent:+ @pytest.mark.parametrize(+ ("error", "expected"),+ [+ (None, {"outcome": "success"}),+ (ValueError("bad input"), {"outcome": "error", "error.type": "ValueError"}),+ (asyncio.CancelledError(), {"outcome": "cancelled"}),+ (KeyboardInterrupt(), {"outcome": "cancelled"}),+ (GeneratorExit(), {"outcome": "cancelled"}),+ (SystemExit(1), {"outcome": "cancelled"}),+ ],+ ids=["returned", "raised", "a task cancelled", "an interrupt", "a generator closed", "the interpreter exiting"],+ )+ def test_the_outcome_says_whether_the_work_returned_failed_or_was_cancelled(self, error: BaseException | None, expected: dict[str, str]) -> None:+ """Only an `Exception` is a failure; the rest of `BaseException` stops work from outside, and names no error type."""+ assert outcome_fields(error=error) == expected++ @pytest.mark.parametrize(+ "error",+ [None, ValueError("bad input"), asyncio.CancelledError()],+ ids=["returned", "raised", "cancelled"],+ )+ def test_the_event_is_logged_once_with_the_work_fields_its_duration_and_its_outcome(+ self, caplog: pytest.LogCaptureFixture, error: BaseException | None+ ) -> None:+ with caplog.at_level(logging.INFO):+ if error is None:+ with WorkEnds():+ pass+ else:+ with pytest.raises(type(error)), WorkEnds():+ raise error++ (event,) = _events(caplog)+ assert event.name == __name__+ assert _fields(event) == {"work": "compose", "duration_ms": DURATION_MS, **outcome_fields(error=error)}+ assert not _failures(caplog)++ @pytest.mark.parametrize("raises_inside", [False, True], ids=["the inner block returns", "the inner block raises"])+ def test_ends_here_logs_the_event_at_the_end_of_the_inner_block_and_only_there(+ self, caplog: pytest.LogCaptureFixture, raises_inside: bool+ ) -> None:+ """The event is logged inside whatever the inner block runs under, and the outer block's end logs nothing more."""+ work_ends = WorkEnds()++ with caplog.at_level(logging.INFO), work_ends:+ try:+ with _inner_block(), work_ends.ends_here():+ if raises_inside:+ msg = "inner failure"+ raise ValueError(msg)+ except ValueError:+ pass++ (event,) = _events(caplog)+ assert work_ends.logged_in == ["inner"]+ assert _fields(event)["outcome"] == ("error" if raises_inside else "success")++ def test_a_failure_before_the_inner_block_ends_the_work_with_its_event_from_the_outer_block(self, caplog: pytest.LogCaptureFixture) -> None:+ work_ends = WorkEnds()++ with caplog.at_level(logging.INFO), pytest.raises(RuntimeError, match="refused before the call"), work_ends:+ _refuse_before_the_call()++ (event,) = _events(caplog)+ assert work_ends.logged_in == ["outer"]+ assert _fields(event) == {"work": "compose", "duration_ms": DURATION_MS, "outcome": "error", "error.type": "RuntimeError"}++ def test_the_clock_starts_at_the_first_entry_only(self, mocker: MockerFixture) -> None:+ start_clock = mocker.patch("pipelex.tools.log.summary_event.start_clock", return_value=STARTED_AT)+ work_ends = WorkEnds()++ with work_ends, work_ends.ends_here():+ pass++ start_clock.assert_called_once_with()++ @pytest.mark.parametrize("work_error", [None, ValueError("the provider refused")], ids=["the work returned", "the work raised"])+ def test_an_event_that_cannot_be_built_never_changes_the_works_outcome(+ self, caplog: pytest.LogCaptureFixture, work_error: ValueError | None+ ) -> None:+ """A failure of the event's own is logged once as a warning; the work's result or exception goes on unchanged."""+ broken_event = WorkEnds(fields_error=TypeError("unsupported operand type(s) for -: 'NoneType' and 'int'"))++ with caplog.at_level(logging.INFO):+ if work_error is None:+ with broken_event:+ result = "the work's result"+ assert result == "the work's result"+ else:+ with pytest.raises(ValueError, match="the provider refused") as raised, broken_event:+ raise work_error+ assert raised.value is work_error++ assert not _events(caplog)+ (failure,) = _failures(caplog)+ assert failure.levelno == logging.WARNING+ assert _fields(failure) == {+ "summary_event": WORK_ENDS_MESSAGE,+ "error.type": "TypeError",+ "error.message": "unsupported operand type(s) for -: 'NoneType' and 'int'",+ }++ @pytest.mark.parametrize("work_error", [None, ValueError("the provider refused")], ids=["the work returned", "the work raised"])+ def test_a_handler_refusing_the_event_and_its_warning_never_changes_the_works_outcome(+ self, caplog: pytest.LogCaptureFixture, refusing_handler: _RefusingHandler, work_error: ValueError | None+ ) -> None:+ """The warning goes through the handlers that refused the event; when they refuse it too, the work still goes on as it came."""+ with caplog.at_level(logging.INFO):+ if work_error is None:+ with WorkEnds():+ result = "the work's result"+ assert result == "the work's result"+ else:+ with pytest.raises(ValueError, match="the provider refused") as raised, WorkEnds():+ raise work_error+ assert raised.value is work_error++ assert refusing_handler.refused == [WORK_ENDS_MESSAGE, SUMMARY_EVENT_FAILED_MESSAGE]