Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Usage-scoped-event-log · L-261008-e2acca (#1430)

Louis Choquel08e0710784c2
diff --git a/.drift/acks/hub-layering-convention.toml b/.drift/acks/hub-layering-convention.toml
index a5ac63e3b..d015409ec 100644
--- a/.drift/acks/hub-layering-convention.toml
+++ b/.drift/acks/hub-layering-convention.toml
@@ -1,10 +1,10 @@
contract = "hub-layering-convention"
-digest = "sha256:558d2a12672e159edf2827791bcb5370781df2b576b958e0ea48989833087f8b"
+digest = "sha256:8ad6535ebd13f9a08da05c84f082e11f9dec80718a9666d9bcaba5c3c4a55747"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-10-13T02:26:31Z"
-rationale = "real-catch: runtime_hub.py gains the module-level resolve_effective_run_mode, moved from the pipe tier so the kernel layer can reach it; the run-mode row of the partition table now names it and the resolve_run_mode_for_boot paragraph says why it sits beside it. No import crosses the boundary: runtime_hub imports cogt_run_params, which is kernel-layer, and KERNEL_LAYER_PACKAGES is unchanged."
+reviewed_at = "2026-10-15T23:18:44Z"
+rationale = "clean-pass: runtime_hub.py changes only the scoped_event_log docstring, which now says usage events honour the override too; no symbol moves between the hubs, no import crosses the boundary, and the tracing row of the partition table already names scoped_event_log and get_event_log_override."
[trigger_files]
"pipelex/cli/dev_cli/commands/hub_layering_guard.py" = "blob:fda2237457fe0b6830ff2ae8f720698735bec4c3"
"pipelex/interpreter_hub.py" = "blob:023f8452daf76dde409afc7c0966b149b11edabb"
-"pipelex/runtime_hub.py" = "blob:8269c5d9d549dd06eb8329e3763efed403f95fed"
+"pipelex/runtime_hub.py" = "blob:2af9f62d93b2005daa55a7c61e0390601e430c00"
diff --git a/.drift/acks/pipelex-kernel-docs.toml b/.drift/acks/pipelex-kernel-docs.toml
index bc914050b..ce0360753 100644
--- a/.drift/acks/pipelex-kernel-docs.toml
+++ b/.drift/acks/pipelex-kernel-docs.toml
@@ -1,8 +1,8 @@
contract = "pipelex-kernel-docs"
-digest = "sha256:1ed250b7f7895e5a94b70b4329c99cef9f1f2c41803c69494deeacbce044392c"
+digest = "sha256:3ac351addba0c0ef338f06edc502ccbfb51ab30a145a97a2a5c911ad874451d7"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-10-13T02:26:32Z"
-rationale = "real-catch: PipelexKernel.make now resolves its mode through runtime_hub.resolve_effective_run_mode and refuses a trace_context stating another mode with the new TraceContextRunModeError in kernel/exceptions.py. pipelex-kernel.md's usage example built a TraceContext without the now-required run_mode; it now states the effective mode on the context and on make, a paragraph explains the refusal and why the kernel refuses rather than corrects its copy, and the keyless-boot sentence names resolve_effective_run_mode."
+reviewed_at = "2026-10-15T23:42:58Z"
+rationale = "clean-pass: the PipelexKernel.make docstring's parenthetical now names the scoped event log beside the runner fallback as stamping the metadata's pipeline_run_id; pipelex-kernel.md already says the same since this branch, so the doc needs no change."
[trigger_files]
"pipelex/kernel/__init__.py" = "blob:c4f1b0fe2ccc17db9110733f71b612f4ac1ba4ce"
@@ -22,7 +22,7 @@ rationale = "real-catch: PipelexKernel.make now resolves its mode through runtim
"pipelex/kernel/llm_prompt_content.py" = "blob:f996cead6e7789494f01f69c128ae71d6dcf8466"
"pipelex/kernel/llm_results.py" = "blob:6e3b8a2878b5bb284ece5fac10262fb59ac2029d"
"pipelex/kernel/memory_ops.py" = "blob:7fbbfaaec99f9712d8e215520cb02e26aad3fb40"
-"pipelex/kernel/pipelex_kernel.py" = "blob:a923ec3c37769d0eac1b37be075971464c27c06c"
+"pipelex/kernel/pipelex_kernel.py" = "blob:c7d40db1572aaca00d1c04a1039f5e3e564d6f87"
"pipelex/kernel/prompt_assembly.py" = "blob:7559725bc0d813cd136bf258b8178d96a8f49194"
"pipelex/kernel/prompt_references.py" = "blob:a4a8c5535c7e51a16ac6c8d1296439b331192a7f"
"pipelex/kernel/search_ops.py" = "blob:b7de7ff62c45dde7f7aa8a9f409aecf3b664cce5"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7bcc6f119..810bd6113 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,11 @@
# Changelog
+## [Unreleased]
+
+### Changed
+
+- **A scoped event log captures usage events**: a usage event emitted while `runtime_hub.scoped_event_log` is set now goes into the scoped event log and nowhere else, ahead of an event log registered with `ReportingManager.set_event_log` and of the per-process runner fallback, including inside an isolated sub-execution such as a Temporal activity, where it always went to the fallback. Code running a leaf inside an activity can therefore scope a fresh `InMemoryEventLog` around it and return the leaf's usage events with its result, which is how a backend with no shared tracing store, such as Mistral Workflows, collects a run's usage; with no scoped event log set, usage events are routed as before.
+
## [v0.86.0] - 2026-10-13
### Highlights
diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.md
index 3385f9c00..1d8f290ca 100644
--- a/docs/under-the-hood/execution-graph-tracing.md
+++ b/docs/under-the-hood/execution-graph-tracing.md
@@ -301,6 +301,7 @@ Semantics:
- The override is ContextVar-scoped (mirrors `scoped_pipe_router`), so concurrent runs with separate scopes never cross-contaminate, and the prior value is restored on exit.
- A set override **implies tracing-enabled**: it is honored even when `runtime.tracing.is_enabled` is `False`, on both the write side and the read side's early-return.
+- **Usage events honor it too, ahead of every other route.** `ReportingManager` writes a usage event emitted in the scope into the scoped instance and nowhere else: not into an event log registered with `set_event_log`, and not into the per-process runner fallback, even inside an isolated sub-execution (an activity, as the [isolated-execution probe](orchestrator-plugins.md#the-isolated-execution-probe-slot) reports it), where an emission otherwise always takes the fallback. The event names the run of the job's own metadata, as on the fallback, and the gate on the trace context's `emit_usage_events` still applies. This is what lets code running a leaf inside an activity capture that leaf's usage: it scopes a fresh `InMemoryEventLog` around the leaf, reads the usage events back and returns them with the activity's result. Each leaf's log needs a `writer_id` of its own, such as the activity's id: the leaves of one workflow attempt share its chain and each fresh log numbers its events from zero, so two leaves writing as `primary` would collide on the store key once their events meet in one log, which keeps only the first. It is how a backend with no shared tracing store, such as Mistral Workflows, can collect a run's usage: the run adds each leaf's returned events to its own in-memory log, so the run's usage is a function of the results its history records, and a replay rebuilds it. With no override set, a usage event goes to the event log registered for its run outside an isolated sub-execution, and to the runner fallback inside one or when none is registered.
- Lifecycle: the read side does not `close()` the scoped instance and the machinery never calls `cleanup()` on it — but the write-side tracer DOES call `close()` on its event log at teardown, before the read side assembles. A scoped event log's `close()` must therefore be idempotent or a no-op (as `InMemoryEventLog`'s is); scoping a backend whose `close()` releases a real resource would break its own assembly read.
This is what lets a graph-producing dry-run trace entirely in memory (no NDJSON file, no DynamoDB round-trip). Both dry-run entrypoints rely on it: `dry_run_pipe_in_process` (`pipelex/pipe_run/dry_run_in_process.py` — the graph arm of protocol `validate` and of the single Temporal validation activity) and `dry_run_pipeline` (`pipelex/pipeline/dry_run_pipeline.py`) itself — these functions exist to produce a graph, so they install their own scoped `InMemoryEventLog` rather than depending on the host having tracing configured (a host with `runtime.tracing.is_enabled = false`, like pipelex-api's `/validate` in direct mode, still gets its graph). A run that nonetheless finishes without a graph raises the typed `DryRunGraphNotProducedError`.
diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.md
index 24da1c672..30d91cfcb 100644
--- a/docs/under-the-hood/orchestrator-plugins.md
+++ b/docs/under-the-hood/orchestrator-plugins.md
@@ -151,7 +151,7 @@ Each `claim_*` takes a **thunk** (a zero-arg factory), never a constructed insta
### The isolated-execution-probe slot
-The first four slots swap in the runtime's implementations of core execution services. The fifth, `claim_isolated_execution_probe`, is different in kind: its thunk resolves not a service but an **ambient predicate** (`Callable[[], bool]`) reporting whether the current call runs inside an isolated sub-execution — a Temporal activity — whose side-effecting emissions must not be written into the parent run's replay-deterministic buffer. `ReportingManager` consults it (`hub.is_in_isolated_execution()`) to route an activity-side usage emission to the per-process log instead of the workflow's registered buffer. It exists as a hub slot for the same reason the others do: only a boot-orchestrator plugin whose runtime has a replay/activity split knows how to answer, and core names no such runtime. Unclaimed (any in-process boot), the hub's core default answers "never isolated" — the in-process orchestrator has no such split.
+The first four slots swap in the runtime's implementations of core execution services. The fifth, `claim_isolated_execution_probe`, is different in kind: its thunk resolves not a service but an **ambient predicate** (`Callable[[], bool]`) reporting whether the current call runs inside an isolated sub-execution — a Temporal activity — whose side-effecting emissions must not be written into the parent run's replay-deterministic buffer. `ReportingManager` consults it (`hub.is_in_isolated_execution()`) to route an activity-side usage emission to the per-process log instead of the workflow's registered buffer. A scoped event log (`runtime_hub.scoped_event_log`) set by the activity's own code comes first: the emission goes into it and nowhere else, which is how an activity captures its leaf's usage to return it with its result (see [the scoped override](execution-graph-tracing.md#event-log-transport-and-the-scoped-override)). The override is a ContextVar, so a workflow's scope never reaches an activity, co-located or not. It exists as a hub slot for the same reason the others do: only a boot-orchestrator plugin whose runtime has a replay/activity split knows how to answer, and core names no such runtime. Unclaimed (any in-process boot), the hub's core default answers "never isolated" — the in-process orchestrator has no such split.
Because the gate is a name-match, `boot_orchestrator` must name a plugin that actually registered. After discovery, `Pipelex.setup` rejects a `boot_orchestrator` that no registered plugin carries — a typo or a missing plugin (e.g. `--orchestrator temporal` without the Temporal plugin installed) raises `UnknownBootOrchestratorError` instead of silently running in-process: nothing would claim the hub slots, so the process would otherwise fall through to the core defaults and execute on the wrong runtime. The check matches against **plugin names** (the same namespace the gate uses), not the `orchestration_mode` registry — a plugin's name and the token(s) it serves are separate namespaces that *may* differ, even where a shipped plugin keeps them identical (our Temporal plugin is named `temporal` and serves `temporal`). The error names no specific plugin, keeping core decoupled.
diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.md
index 64fa65c7a..3453f2ce1 100644
--- a/docs/under-the-hood/pipelex-kernel.md
+++ b/docs/under-the-hood/pipelex-kernel.md
@@ -171,11 +171,13 @@ finally:
get_report_delegate().clear_event_log(context_key=trace_context.lookup_key)
```
+A caller can scope the event log instead of registering it: under `with scoped_event_log(event_log):` (`pipelex.runtime_hub`), every usage event a step emits goes into that instance and nowhere else, ahead of any registered event log and of the runner fallback, with nothing to clear afterwards. That is the form a host must use when it runs a step inside an isolated sub-execution, such as an activity, to capture the step's usage and return it with its result, since an emission there never takes a registered event log (see [the scoped override](execution-graph-tracing.md#event-log-transport-and-the-scoped-override)).
+
The trace context also states the chain of execution attempts its events belong to, required as the mode is, since every usage event takes its execution, its attempt and its lineage from it (see [Executions and Their Attempts](execution-graph-tracing.md#executions-and-their-attempts)). A host running the kernel in its own process states `make_direct_chain()`, the direct mode's one execution in its first attempt. A host running it inside a durable workflow states that workflow's own chain, its lineage followed by its id and the attempt its engine runs, so that the usage of a new attempt supersedes the earlier one's, where both would otherwise be written as `direct@1`.
The trace context states the mode the run takes, which `resolve_effective_run_mode` gives: every usage event a step writes states the context's `run_mode`, so a context saying live for a run whose steps run dry would stamp the whole run live. `PipelexKernel.make` refuses a context stating another mode with `TraceContextRunModeError` (`pipelex/kernel/exceptions.py`), a `ValueError` raised before any step runs. It refuses rather than correct its own copy because the trace is the caller's, possibly shared with writers the kernel never sees, which a corrected copy would leave disagreeing with the kernel's events.
-Passing a `trace_context` adopts its `graph_id` as the run's `pipeline_run_id`, and passing a `pipeline_run_id` that differs from it raises `ValueError` rather than letting either win. The two are one identity: letting them diverge would scatter a single run's usage events across two ids, because the registered-context emit path stamps the event log's id while the runner fallback stamps the metadata's — and a read-back keyed on either would silently miss the other's.
+Passing a `trace_context` adopts its `graph_id` as the run's `pipeline_run_id`, and passing a `pipeline_run_id` that differs from it raises `ValueError` rather than letting either win. The two are one identity: letting them diverge would scatter a single run's usage events across two ids, because the registered-context emit path stamps the event log's id while the scoped event log and the runner fallback stamp the metadata's — and a read-back keyed on either would silently miss the other's.
`pipelex.tracing` holds both halves a caller needs (`make_event_log` for a configured backend, `UsageAggregator` for the read-back) and is kernel-layer, so none of this costs the boot contract. The records that come out are the same [`TokensUsage` wire records](./tokens-usage-wire-records.md) an `/execute` response carries — pinned by an integration test that runs the same step through both callers and compares them.
diff --git a/pipelex/kernel/pipelex_kernel.py b/pipelex/kernel/pipelex_kernel.py
index a923ec3c3..c7d40db15 100644
--- a/pipelex/kernel/pipelex_kernel.py
+++ b/pipelex/kernel/pipelex_kernel.py
@@ -129,8 +129,9 @@ class PipelexKernel:
the cogt leaf emits a usage event only when the metadata it is handed carries one. Passing it
adopts its ``graph_id`` as this run's ``pipeline_run_id`` rather than minting a fresh id —
the two are one identity, and letting them diverge would scatter a single run's usage events
- across two ids (the registered-context emit path stamps the event log's id, the runner
- fallback stamps the metadata's), so the read-back would silently miss half of them.
+ across two ids (the registered-context emit path stamps the event log's id, the scoped
+ event log and the runner fallback stamp the metadata's), so the read-back would silently
+ miss half of them.
``pipeline_run_id`` names the run when no trace context does. The run id is a trace
context's ``graph_id`` when one is given, else this value, else a fresh ``uuid4``. A trace
diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.py
index e5fa39378..4a52524fa 100644
--- a/pipelex/reporting/reporting_manager.py
+++ b/pipelex/reporting/reporting_manager.py
@@ -14,7 +14,7 @@ from pipelex.cogt.search.search_job import SearchJob
from pipelex.config import get_config
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.reporting.reporting_types import AnyTokensUsage
-from pipelex.runtime_hub import is_in_isolated_execution
+from pipelex.runtime_hub import get_event_log_override, is_in_isolated_execution
from pipelex.system.exceptions import MissingDependencyError
from pipelex.system.trace_context import TraceContext
from pipelex.tools.log.error_fields import error_fields
@@ -37,11 +37,11 @@ except ImportError:
# Infra-level failures an event-log backend can raise from emit(). Usage/cost reporting is a side
# concern: by the time report_inference_job runs, the inference has already succeeded (and been
# billed), so a transient event-log write failure must never propagate and turn a successful
-# inference into a failed pipeline. Both emit paths — the registered-context fast path and the
-# runner fallback — drop these with a WARNING. OSError covers the NDJSON backend (dir/file write
-# errors); ClientError (service-side throttle / auth) and BotoCoreError (transport / credential /
-# timeout, e.g. EndpointConnectionError, ReadTimeoutError, NoCredentialsError) cover the DynamoDB
-# backend when boto3 is installed. ClientError and BotoCoreError are sibling botocore base classes
+# inference into a failed pipeline. Every emit path — the scoped override, the registered-context
+# fast path and the runner fallback — drops these with a WARNING. OSError covers the NDJSON backend
+# (dir/file write errors); ClientError (service-side throttle / auth) and BotoCoreError (transport /
+# credential / timeout, e.g. EndpointConnectionError, ReadTimeoutError, NoCredentialsError) cover the
+# DynamoDB backend when boto3 is installed. ClientError and BotoCoreError are sibling botocore base classes
# (neither subclasses the other), so both must be listed.
# cast: botocore is untyped, so the imported classes are Unknown; we know they are BaseException types.
# Single assignment (no reassignment) so the uppercase constant satisfies reportConstantRedefinition.
@@ -176,6 +176,20 @@ class ReportingManager(ReportingProtocol):
def _emit_usage_event(self, inference_job: InferenceJobAbstract, *, tokens_usage: AnyTokensUsage) -> None:
"""Emit a UsageReportEvent for this job.
+ Scoped path, ahead of every other: when a scoped event log is set
+ (``runtime_hub.scoped_event_log``), the event goes into it and nowhere else —
+ not into a registered context, not into the runner fallback — whether or not
+ the code runs in an isolated sub-execution, and whether or not tracing is
+ enabled, since a set override implies tracing-enabled, as it does for the
+ tracer setup in ``pipeline_run_setup`` and for ``assemble_tracing``. This is
+ what lets code running a leaf inside an isolated sub-execution capture the
+ usage that leaf emits and return it with its result, for a backend such as
+ Mistral Workflows that has no shared store for the run to read usage back from.
+ It is safe inside an isolated sub-execution, unlike the registered fast path
+ below: the override is a ContextVar, so it reaches only code running in the
+ context that set it and the tasks started from it, and an isolated
+ sub-execution sees one only when its own code scoped it.
+
Fast path: when set_event_log was registered for this trace context's
lookup_key (workflow-thread or direct mode), emit through the cached
per-context event log.
@@ -200,8 +214,8 @@ class ReportingManager(ReportingProtocol):
if trace_context is None:
return
- # Gate cost emission on emit_usage_events BEFORE the context lookup, so both the fast path
- # (_emit_via_registered_context) and the runner fallback are guarded by the same check. This
+ # Gate cost emission on emit_usage_events BEFORE any routing, so the scoped override, the fast
+ # path (a registered context) and the runner fallback are all guarded by the same check. This
# keeps correctness from resting on the cross-file invariant "a context is registered (via
# set_event_log) only when costs are on": _event_log_contexts is a process-global singleton
# and clear_event_log is best-effort in finally blocks, so a leaked context from a prior
@@ -211,10 +225,29 @@ class ReportingManager(ReportingProtocol):
if not trace_context.emit_usage_events:
return
+ # A scoped override (see hub.scoped_event_log) is the transport of whatever runs in its scope and
+ # implies tracing-enabled: the event goes into it and nowhere else, ahead of the registered contexts
+ # and the runner fallback, inside an isolated sub-execution too (see the docstring). The run is the
+ # job's own, as on the runner fallback, since no registration names one here.
+ event_log_override = get_event_log_override()
+ if event_log_override is not None:
+ self._emit_into_event_log(
+ event_log=event_log_override,
+ pipeline_run_id=inference_job.job_metadata.run_metadata.pipeline_run_id,
+ trace_context=trace_context,
+ tokens_usage=tokens_usage,
+ )
+ return
+
if not is_in_isolated_execution():
context = self._event_log_contexts.get(trace_context.lookup_key)
if context is not None:
- self._emit_via_registered_context(context=context, trace_context=trace_context, tokens_usage=tokens_usage)
+ self._emit_into_event_log(
+ event_log=context.event_log,
+ pipeline_run_id=context.pipeline_run_id,
+ trace_context=trace_context,
+ tokens_usage=tokens_usage,
+ )
return
self._emit_usage_event_runner_fallback(
@@ -224,36 +257,40 @@ class ReportingManager(ReportingProtocol):
)
@staticmethod
- def _emit_via_registered_context(
+ def _emit_into_event_log(
*,
- context: _EventLogContext,
+ event_log: EventLogProtocol,
+ pipeline_run_id: str,
trace_context: TraceContext,
tokens_usage: AnyTokensUsage,
) -> None:
- """Fast-path emit through a context registered via set_event_log."""
- node_id = trace_context.parent_node_id or UNATTRIBUTED_NODE_ID
- seq = context.event_log.next_sequence()
+ """Build the usage event a trace context states and emit it into ``event_log``, best-effort.
+ Shared by every emit path — the scoped override, a context registered via set_event_log and the
+ runner fallback — which differ only in the event log they write into and the run they name. The
+ execution, its attempt and its lineage are the trace context's on every path: a usage event belongs
+ to the attempt of the workflow that ran it, which a new attempt of that workflow supersedes.
+ """
event = UsageReportEvent(
- pipeline_run_id=context.pipeline_run_id,
+ pipeline_run_id=pipeline_run_id,
workflow_id=trace_context.own_attempt.execution,
attempt=trace_context.own_attempt.attempt,
lineage=trace_context.lineage,
path=trace_context.parent_path,
- writer_id=context.event_log.writer_id,
+ writer_id=event_log.writer_id,
mode=trace_context.run_mode,
timestamp=datetime.now(UTC),
- sequence=seq,
- node_id=node_id,
+ sequence=event_log.next_sequence(),
+ node_id=trace_context.parent_node_id or UNATTRIBUTED_NODE_ID,
tokens_usage=tokens_usage,
)
- ReportingManager._emit_best_effort(event_log=context.event_log, event=event)
+ ReportingManager._emit_best_effort(event_log=event_log, event=event)
@staticmethod
def _emit_best_effort(*, event_log: EventLogProtocol, event: UsageReportEvent) -> None:
"""Emit a usage event, dropping infra-level backend failures with a WARNING.
- Shared by both emit paths (the registered-context fast path and the runner fallback).
+ Shared by every emit path (the scoped override, the registered-context fast path and the runner fallback).
report_inference_job runs synchronously after the inference has already succeeded, so an
event-log write failure must not propagate and fail the pipeline. We catch the specific
infra exception classes the backends raise at emit() time (see
@@ -276,7 +313,8 @@ class ReportingManager(ReportingProtocol):
This is the universal path for activity-side usage emissions (audit finding
H1): every emission from inside a Temporal activity routes here, co-located
or remote alike, so the workflow's in-sandbox buffer stays a pure function
- of inline execution. It also covers the original fallback case — a runner
+ of inline execution — unless the activity's own code set a scoped event log,
+ which takes the emission instead. It also covers the original fallback case — a runner
process where ``set_event_log`` was never called. The event log is
process-local, built from ``tracing_config``, stamped with a stable
per-process writer_id of the form ``act_{pid}_{uuid8}``.
@@ -308,34 +346,19 @@ class ReportingManager(ReportingProtocol):
if process_event_log is None:
return
- # The execution, its attempt and its lineage are the trace context's, as on the fast path: an activity's usage
- # belongs to the attempt of the workflow that ran it, which a new attempt of that workflow supersedes.
- workflow_id = trace_context.own_attempt.execution
- node_id = trace_context.parent_node_id or UNATTRIBUTED_NODE_ID
-
# The line names the run's tracer key beside the execution: the execution is `direct` for every run in direct
# mode and under the kernel, so it alone would not say which run engaged the fallback.
ActivityEventLogCache.log_once_runner_fallback_engaged(
- workflow_id=workflow_id, tracer_key=trace_context.lookup_key, writer_id=process_event_log.writer_id
+ workflow_id=trace_context.own_attempt.execution, tracer_key=trace_context.lookup_key, writer_id=process_event_log.writer_id
)
- seq = process_event_log.next_sequence()
- event = UsageReportEvent(
+ self._emit_into_event_log(
+ event_log=process_event_log,
pipeline_run_id=inference_job.job_metadata.run_metadata.pipeline_run_id,
- workflow_id=workflow_id,
- attempt=trace_context.own_attempt.attempt,
- lineage=trace_context.lineage,
- path=trace_context.parent_path,
- writer_id=process_event_log.writer_id,
- mode=trace_context.run_mode,
- timestamp=datetime.now(UTC),
- sequence=seq,
- node_id=node_id,
+ trace_context=trace_context,
tokens_usage=tokens_usage,
)
- self._emit_best_effort(event_log=process_event_log, event=event)
-
@override
def report_inference_job(self, inference_job: InferenceJobAbstract):
if isinstance(inference_job, LLMJob):
diff --git a/pipelex/runtime_hub.py b/pipelex/runtime_hub.py
index 8269c5d9d..2af9f62d9 100644
--- a/pipelex/runtime_hub.py
+++ b/pipelex/runtime_hub.py
@@ -722,6 +722,19 @@ def scoped_event_log(event_log: "EventLogProtocol") -> Generator[None, None, Non
store bridges the two sides). A set override implies tracing-enabled: it is honored
even when ``runtime.tracing.is_enabled`` is False.
+ Usage events honor it too, ahead of everything else: ``ReportingManager`` writes a
+ usage event emitted in the scope into this instance and nowhere else — not into an
+ event log registered with ``set_event_log``, not into the per-process runner fallback —
+ including inside an isolated sub-execution (``is_in_isolated_execution()`` true). So
+ code running a leaf inside an activity can scope a fresh log around it, read the
+ leaf's usage events back and return them with its result (each leaf's log with a
+ ``writer_id`` of its own, since the leaves of one workflow attempt share its chain and
+ would otherwise collide on the store key once gathered), which is how a backend
+ with no shared tracing store, such as Mistral Workflows, gets a run's usage into the
+ run's own log, where a replay rebuilds it from the recorded results. Being a
+ ContextVar, the override reaches only code running in the context that set it and the
+ tasks started from it, so an activity sees one only when its own code scoped it.
+
Lifecycle: the machinery never calls ``cleanup`` on the instance and the read side does
not ``close`` it — but the write-side tracer DOES call ``close()`` on its event log at
teardown (``GraphTracer._reset``), which happens BEFORE the read side assembles. A scoped
diff --git a/tests/integration/pipelex/pipeline/test_usage_scoped_event_log.py b/tests/integration/pipelex/pipeline/test_usage_scoped_event_log.py
new file mode 100644
index 000000000..63bb7ff98
--- /dev/null
+++ b/tests/integration/pipelex/pipeline/test_usage_scoped_event_log.py
@@ -0,0 +1,61 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.dry_mock import MOCK_USAGE_MODEL_NAME
+from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.config import get_config
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.runtime_hub import scoped_event_log
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+
+_DOMAIN = "usage_scoped_event_log"
+_MTHDS = f"""
+domain = "{_DOMAIN}"
+description = "Minimal bundle for usage capture through a scoped event log"
+
+[pipe.write_text]
+type = "PipeLLM"
+description = "Pipe that outputs plain text"
+inputs = {{ subject = "Text" }}
+output = "Text"
+prompt = "Write about $subject"
+"""
+
+
[email protected](loop_scope="class")
+class TestUsageScopedEventLog:
+ async def test_isolated_leaf_usage_assembles_from_the_scoped_event_log(self, mocker: MockerFixture) -> None:
+ """A run whose leaves report as if inside an isolated execution gets its usage from its scoped log alone.
+
+ Tracing is off and the isolation probe says every emission comes from an isolated sub-execution, so before
+ usage honoured the override the leaf's usage went to the runner fallback, which tracing-off drops, and the
+ run assembled none. Now the leaf's usage lands in the scoped log, the run's assembly reads it from there, and
+ the per-process fallback is never built.
+ """
+ mocker.patch.object(get_config().runtime.tracing, "is_enabled", False)
+ mocker.patch("pipelex.reporting.reporting_manager.is_in_isolated_execution", return_value=True)
+ fallback_spy = mocker.spy(ActivityEventLogCache, "get_or_create")
+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(
+ generate_graph=False,
+ generate_usage=True,
+ mock_inputs=True,
+ )
+ runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, is_mock_usage=True, execution_config=execution_config)
+ run_log = InMemoryEventLog()
+
+ with scoped_event_log(run_log):
+ response = await runner.execute(pipe_code="write_text", mthds_contents=[_MTHDS])
+
+ pipe_output = response.pipe_output
+ assert pipe_output.usage_assembly_error is None
+ assert pipe_output.tokens_usages is not None
+ assert len(pipe_output.tokens_usages) >= 1
+ assert all(usage.inference_model_name == MOCK_USAGE_MODEL_NAME for usage in pipe_output.tokens_usages)
+ assert CostRegistry.aggregate_costs(pipe_output.tokens_usages).total_nb_tokens > 0
+ pipeline_run_id = pipe_output.pipeline_run_id
+ captured = [event for event in run_log.read_events(pipeline_run_id) if isinstance(event, UsageReportEvent)]
+ assert len(captured) == len(pipe_output.tokens_usages)
+ fallback_spy.assert_not_called()
diff --git a/tests/unit/pipelex/reporting/test_emit_usage_scoped_event_log.py b/tests/unit/pipelex/reporting/test_emit_usage_scoped_event_log.py
new file mode 100644
index 000000000..a1e9ee1b5
--- /dev/null
+++ b/tests/unit/pipelex/reporting/test_emit_usage_scoped_event_log.py
@@ -0,0 +1,341 @@
+import asyncio
+import logging
+from collections.abc import Callable
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+from typing import Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job import ExtractJob
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams, ExtractJobReport
+from pipelex.cogt.extract.extract_report import ExtractTokensUsage
+from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
+from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobConfig, ImgGenJobParams, ImgGenJobReport
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
+from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
+from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.runtime_hub import get_event_log_override, scoped_event_log
+from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend
+from pipelex.system.data_inclusion_config import DataInclusionConfig
+from pipelex.system.execution_chain import ExecutionAttempt
+from pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobId
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.system.trace_context import TraceContext
+from pipelex.tools.misc.image_utils import ImageFormat
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+
+_DATA_INCLUSION_OFF = DataInclusionConfig(
+ pipe_and_concept_registry=False,
+ stuff_json_content=False,
+ error_stack_traces=False,
+)
+
+_ISOLATION_PROBE_TARGET = "pipelex.reporting.reporting_manager.is_in_isolated_execution"
+
+
+def _trace_context(run_id: str, *, emit_usage_events: bool = True) -> TraceContext:
+ """A trace context for a leaf of a dispatched workflow: its own attempt under the run's, at a known path."""
+ return TraceContext(
+ graph_id=run_id,
+ tracer_key=f"wf_{run_id}",
+ parent_node_id=f"{run_id}:node_3",
+ node_sequence=3,
+ data_inclusion=_DATA_INCLUSION_OFF,
+ emit_usage_events=emit_usage_events,
+ run_mode=PipeRunMode.LIVE,
+ chain=[ExecutionAttempt(execution=f"wf_{run_id}", attempt=1), ExecutionAttempt(execution=f"wf_{run_id}--branch", attempt=2)],
+ parent_path="method.main/step_2",
+ )
+
+
+def _job_metadata(run_id: str, *, emit_usage_events: bool = True, unit_job_id: UnitJobId | None = None) -> JobMetadata:
+ now = datetime.now(UTC)
+ return JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test_user", pipeline_run_id=run_id),
+ trace_context=_trace_context(run_id, emit_usage_events=emit_usage_events),
+ started_at=now,
+ completed_at=now + timedelta(seconds=1),
+ unit_job_id=unit_job_id,
+ )
+
+
+def _make_llm_job(run_id: str, *, emit_usage_events: bool = True) -> LLMJob:
+ job_metadata = _job_metadata(run_id, emit_usage_events=emit_usage_events, unit_job_id=UnitJobId.LLM_GEN_TEXT)
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(schema_reask_max_attempts=1),
+ job_report=LLMJobReport(
+ llm_tokens_usage=LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="llm-model",
+ inference_model_id="llm-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},
+ )
+ ),
+ )
+
+
+def _make_img_gen_job(run_id: str) -> ImgGenJob:
+ job_metadata = _job_metadata(run_id)
+ return ImgGenJob(
+ img_gen_prompt=ImgGenPrompt(positive_text="a prompt"),
+ job_params=ImgGenJobParams(
+ aspect_ratio=AspectRatio.SQUARE,
+ size=None,
+ background=Background.OPAQUE,
+ input_fidelity=None,
+ output_format=ImageFormat.PNG,
+ ),
+ job_config=ImgGenJobConfig(is_sync_mode=False),
+ job_report=ImgGenJobReport(
+ img_gen_tokens_usage=ImgGenTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="img-model",
+ inference_model_id="img-model-id",
+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 200},
+ )
+ ),
+ job_metadata=job_metadata,
+ )
+
+
+def _make_extract_job(run_id: str) -> ExtractJob:
+ job_metadata = _job_metadata(run_id)
+ return ExtractJob(
+ extract_input=ExtractInput(document_uri="/tmp/test.pdf"), # ruff: ignore[hardcoded-temp-file]
+ job_params=ExtractJobParams.make_default_extract_job_params(),
+ job_config=ExtractJobConfig(),
+ job_report=ExtractJobReport(
+ extract_tokens_usage=ExtractTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="extract-model",
+ inference_model_id="extract-model-id",
+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 1000},
+ nb_tokens_by_category={TokenCategory.INPUT: 300, TokenCategory.OUTPUT: 400},
+ )
+ ),
+ job_metadata=job_metadata,
+ )
+
+
+def _enable_ndjson_tracing(mocker: MockerFixture, traces_dir: Path) -> None:
+ """Turn tracing on with an NDJSON backend in ``traces_dir``, so the runner fallback has somewhere to write."""
+ cfg = get_config().runtime.tracing
+ mocker.patch.object(cfg, "is_enabled", True)
+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)
+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=str(traces_dir)))
+
+
+def _usage_events(event_log: InMemoryEventLog, run_id: str) -> list[UsageReportEvent]:
+ return [evt for evt in event_log.read_events(run_id) if isinstance(evt, UsageReportEvent)]
+
+
+def _fallback_usage_events(traces_dir: Path, run_id: str) -> list[UsageReportEvent]:
+ return [evt for evt in NdjsonEventLog(traces_dir=str(traces_dir)).read_events(run_id) if isinstance(evt, UsageReportEvent)]
+
+
+def _register_spy_context(manager: ReportingManager, mocker: MockerFixture, *, run_id: str) -> Any:
+ """Register a spy event log for the run's lookup key, as ``pipeline_run_setup`` or a workflow would."""
+ event_log_spy = mocker.MagicMock()
+ event_log_spy.writer_id = "registered_writer"
+ event_log_spy.next_sequence.return_value = 0
+ manager.set_event_log(context_key=_trace_context(run_id).lookup_key, event_log=event_log_spy, pipeline_run_id=run_id)
+ return event_log_spy
+
+
[email protected](autouse=True)
+def _reset_activity_event_log_state() -> Any: # pyright: ignore[reportUnusedFunction]
+ """Class-level state on ActivityEventLogCache persists across tests; reset it."""
+ ActivityEventLogCache.reset_for_tests()
+ yield
+ ActivityEventLogCache.reset_for_tests()
+
+
+class TestEmitUsageScopedEventLog:
+ @pytest.mark.parametrize("is_isolated", [False, True])
+ def test_scoped_event_log_takes_the_usage_event_and_nothing_else_does(self, tmp_path: Path, mocker: MockerFixture, is_isolated: bool) -> None:
+ """Under a scoped log, the event lands in it alone: the registered context and the fallback get nothing."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=is_isolated)
+ manager = ReportingManager()
+ manager.setup()
+ registered_spy = _register_spy_context(manager, mocker, run_id="run_scoped")
+ scoped_log = InMemoryEventLog(writer_id="leaf_capture")
+
+ with scoped_event_log(scoped_log):
+ manager.report_inference_job(_make_llm_job("run_scoped"))
+
+ captured = _usage_events(scoped_log, "run_scoped")
+ assert len(captured) == 1
+ event = captured[0]
+ assert event.pipeline_run_id == "run_scoped"
+ assert event.writer_id == "leaf_capture"
+ assert event.sequence == 0
+ assert event.workflow_id == "wf_run_scoped--branch"
+ assert event.attempt == 2
+ assert event.lineage == [ExecutionAttempt(execution="wf_run_scoped", attempt=1)]
+ assert event.path == "method.main/step_2"
+ assert event.node_id == "run_scoped:node_3"
+ assert event.mode.is_live
+ assert event.tokens_usage.nb_tokens_by_category == {TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50}
+ registered_spy.emit.assert_not_called()
+ assert list(tmp_path.glob("**/*.ndjson")) == []
+
+ @pytest.mark.parametrize(
+ ("make_job", "model_name"),
+ [
+ (_make_llm_job, "llm-model"),
+ (_make_img_gen_job, "img-model"),
+ (_make_extract_job, "extract-model"),
+ ],
+ )
+ def test_leaf_in_an_isolated_execution_captures_its_usage(
+ self, tmp_path: Path, mocker: MockerFixture, make_job: Callable[[str], InferenceJobAbstract], model_name: str
+ ) -> None:
+ """The plugin's case: code running a leaf in an activity scopes a fresh log and reads the leaf's usage back."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=True)
+ fallback_spy = mocker.spy(ActivityEventLogCache, "get_or_create")
+ manager = ReportingManager()
+ manager.setup()
+
+ leaf_log = InMemoryEventLog(writer_id="activity_leaf")
+ with scoped_event_log(leaf_log):
+ manager.report_inference_job(make_job("run_activity"))
+
+ captured = _usage_events(leaf_log, "run_activity")
+ assert [event.tokens_usage.inference_model_name for event in captured] == [model_name]
+ fallback_spy.assert_not_called()
+ assert _fallback_usage_events(tmp_path, "run_activity") == []
+
+ def test_scoped_event_log_is_honoured_with_tracing_disabled(self, mocker: MockerFixture) -> None:
+ """A set override implies tracing-enabled, for usage as for the tracer and the assembly."""
+ mocker.patch.object(get_config().runtime.tracing, "is_enabled", False)
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=True)
+ manager = ReportingManager()
+ manager.setup()
+ scoped_log = InMemoryEventLog()
+
+ with scoped_event_log(scoped_log):
+ manager.report_inference_job(_make_llm_job("run_tracing_off"))
+
+ assert len(_usage_events(scoped_log, "run_tracing_off")) == 1
+
+ def test_scoped_event_log_still_honours_the_costs_off_gate(self, mocker: MockerFixture) -> None:
+ """A trace context with usage events off emits nothing, into the scoped log as anywhere else."""
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=True)
+ manager = ReportingManager()
+ manager.setup()
+ scoped_log = InMemoryEventLog()
+
+ with scoped_event_log(scoped_log):
+ manager.report_inference_job(_make_llm_job("run_costs_off", emit_usage_events=False))
+
+ assert scoped_log.read_events("run_costs_off") == []
+
+ def test_scoped_event_log_emit_failure_is_best_effort(self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> None:
+ """A scoped log whose emit fails with an infra error drops the event with a warning, as every path does."""
+ manager = ReportingManager()
+ manager.setup()
+ failing_log = InMemoryEventLog()
+ emit_spy = mocker.patch.object(failing_log, "emit", side_effect=OSError("disk full"))
+
+ with caplog.at_level(logging.WARNING), scoped_event_log(failing_log):
+ # Must not raise: the failure is logged and dropped.
+ manager.report_inference_job(_make_llm_job("run_failing"))
+
+ emit_spy.assert_called_once()
+ assert any(record.levelno == logging.WARNING for record in caplog.records)
+
+ def test_no_override_outside_isolation_takes_the_registered_context(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """With no scoped log, a registered context takes the event as before, and the fallback gets nothing."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=False)
+ manager = ReportingManager()
+ manager.setup()
+ registered_spy = _register_spy_context(manager, mocker, run_id="run_registered")
+
+ assert get_event_log_override() is None
+ manager.report_inference_job(_make_llm_job("run_registered"))
+
+ registered_spy.emit.assert_called_once()
+ emitted_event = registered_spy.emit.call_args.args[0]
+ assert isinstance(emitted_event, UsageReportEvent)
+ assert emitted_event.pipeline_run_id == "run_registered"
+ assert emitted_event.writer_id == "registered_writer"
+ assert emitted_event.workflow_id == "wf_run_registered--branch"
+ assert emitted_event.path == "method.main/step_2"
+ assert list(tmp_path.glob("**/*.ndjson")) == []
+
+ def test_no_override_inside_isolation_takes_the_runner_fallback(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """With no scoped log, an isolated sub-execution skips the registered context for the fallback, as before."""
+ _enable_ndjson_tracing(mocker, tmp_path)
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=True)
+ manager = ReportingManager()
+ manager.setup()
+ registered_spy = _register_spy_context(manager, mocker, run_id="run_fallback")
+
+ assert get_event_log_override() is None
+ manager.report_inference_job(_make_llm_job("run_fallback"))
+
+ registered_spy.emit.assert_not_called()
+ fallback_events = _fallback_usage_events(tmp_path, "run_fallback")
+ assert len(fallback_events) == 1
+ assert fallback_events[0].writer_id.startswith("act_")
+ assert fallback_events[0].workflow_id == "wf_run_fallback--branch"
+
+ def test_leaving_the_scope_restores_the_previous_routing(self, mocker: MockerFixture) -> None:
+ """Once the scope exits, the next event goes to the registered context again, and the scoped log keeps only its own."""
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=False)
+ manager = ReportingManager()
+ manager.setup()
+ registered_spy = _register_spy_context(manager, mocker, run_id="run_restore")
+ scoped_log = InMemoryEventLog()
+
+ with scoped_event_log(scoped_log):
+ manager.report_inference_job(_make_llm_job("run_restore"))
+ manager.report_inference_job(_make_llm_job("run_restore"))
+
+ assert len(_usage_events(scoped_log, "run_restore")) == 1
+ registered_spy.emit.assert_called_once()
+
+ @pytest.mark.asyncio
+ async def test_concurrent_scopes_capture_only_their_own_usage(self, mocker: MockerFixture) -> None:
+ """Two leaves running concurrently in one process, each in its own scope, never see each other's usage."""
+ mocker.patch(_ISOLATION_PROBE_TARGET, return_value=True)
+ manager = ReportingManager()
+ manager.setup()
+
+ async def run_leaf(run_id: str) -> InMemoryEventLog:
+ leaf_log = InMemoryEventLog(writer_id=f"leaf_{run_id}")
+ with scoped_event_log(leaf_log):
+ # Yield so the two leaves interleave inside their scopes before either reports.
+ await asyncio.sleep(0)
+ manager.report_inference_job(_make_llm_job(run_id))
+ await asyncio.sleep(0)
+ return leaf_log
+
+ first_log, second_log = await asyncio.gather(run_leaf("run_first"), run_leaf("run_second"))
+
+ assert [event.pipeline_run_id for event in first_log.read_events("run_first")] == ["run_first"]
+ assert first_log.read_events("run_second") == []
+ assert [event.pipeline_run_id for event in second_log.read_events("run_second")] == ["run_second"]
+ assert second_log.read_events("run_first") == []