Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Stamp-invocation-paths · L-260927-d30355 (#1428)
Louis Choquelf05b618cdd75
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex de86ddb9d..03e5d6aa0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,7 +10,9 @@### Changed- **Every trace event states its `mode` (Breaking)**: each trace event carries `mode`, `live` or `dry`, the mode the run's pipes run in, which the run's `TraceContext` carries as the new `run_mode`, so a usage event an activity writes states it too, and a live run a keyless boot (`needs_inference=False`) forces dry states `dry` on its trace context, its events and its graph, and, like every dry run, opens no OpenTelemetry trace; an event stored without `mode` by an earlier version is refused with `EventLogSchemaMismatchError` rather than read as live, and a consumer that builds trace events passes it. Callers follow new signatures: `TraceContext.run_mode` is required, with no default, and so is `run_mode` on `GraphTracerManager.open_tracer` and on the tracers' `setup` (`GraphTracerProtocol`, `GraphTracer` and `GraphTracerNoOp`), where it replaces `mode: GraphSpecMode`, so a workflow opening a tracer of its own for part of a run passes the `run_mode` of the context it received, and `PipelexKernel.make` refuses, with `TraceContextRunModeError`, a `trace_context` stating another mode than the one its run takes, which the new `runtime_hub.resolve_effective_run_mode` gives, dry on a keyless boot. `GraphSpecAssembler.assemble` no longer takes `mode` and raises the new `TraceEventModeError` on a set of events that mixes modes or holds none, and `assemble_tracing` and `assemble_tracing_on_output` no longer take `run_mode`: the graph states the mode its events state, so a dry run assembled from its events in a durable workflow gets a dry graph where it got a live one, and `assemble_tracing` reports a `TraceEventModeError` as the run's `graph_assembly_error`.-- **Controllers register their execution data as they decide (Breaking)**: a `PipeSequence` registers `step_count` as it starts, now the number of steps its author wrote, and the new `current_step` as each step starts; a `PipeParallel` registers its `branch_count`, `add_each_output` and `combined_output_concept` before its branches run, a `PipeBatch` its `item_count` and `branch_pipe_code` before its items run, and a live `PipeCondition` its `evaluated_expression` and `selected_outcome` once its choice is final, before the chosen pipe runs, where each used to register them once it had finished. `GraphTracerProtocol` adds `register_batch_fan_out` and `register_condition_resolution`, and `add_selected_outcome_edge` takes `selected_outcome: str | None` in place of `outcome_value`.+- **Controllers register their execution data as they decide (Breaking)**: a `PipeSequence` registers `step_count` as it starts, now the number of steps its author wrote; a `PipeParallel` registers its `branch_count`, `add_each_output` and `combined_output_concept` before its branches run, a `PipeBatch` its `item_count` and `branch_pipe_code` before its items run, and a live `PipeCondition` its `evaluated_expression` and `selected_outcome` once its choice is final, before the chosen pipe runs, where each used to register them once it had finished. `GraphTracerProtocol` adds `register_batch_fan_out` and `register_condition_resolution`, and `add_selected_outcome_edge` takes `selected_outcome: str | None` in place of `outcome_value`.+- **Trace events and graph nodes name their invocation (Breaking)**: every trace event carries `path`, the invocation path the hosted progress contract defines, which names a run of a pipe the same way in every run of the method, live or dry, on every engine: the entry pipe's `pipe_ref`, then one segment per controller level, `step_<n>` as the author numbered the steps, binding steps included, `branch_<n>`, `item_<n>` counting the items from 1, and `outcome_<key>` or `default` for a condition's outcome group, the outcomes that run one pipe as the library resolves it, however they spell it, named by the first key that runs it, percent-encoded, so `filings.review_filings/step_3/item_2/step_1`, and a dry run runs each group's pipe once; a binding step's events carry `null`. Every `NodeSpec` carries the same `path`, `pipe_start` carries `pipe_ref`, `pipe_end_success`, `pipe_end_error` and `pipe_end_skipped` carry `pipe_code`, `pipe_ref` and `pipe_type`, and a failed node's `ErrorSpec` carries `caller_facing`, true when its message was written for the caller, or the error's text with `caller_facing` false when the error's report cannot be built, a fault the tracer logs while the run still fails with the pipe's own error. A controller hands each pipe its place through `JobMetadata.for_child_at`; since tracing never fails a run, a pipe started without one is traced with a `null` path and the tracer logs the fault, the new `InvocationPathError`, as an error, while an inactive tracer and the no-op tracer derive no path; `GraphTracerManager.on_pipe_start` and the tracers' `on_pipe_start` require `pipe_ref`, `on_pipe_end_error` takes `caller_facing`, and `TraceContext.copy_for_child` requires `invocation_path`. An event stored without these fields by an earlier version is refused with `EventLogSchemaMismatchError`, and a consumer that builds trace events passes them.+- **Trace events name the attempt of the execution that wrote them (Breaking)**: every trace event carries `attempt`, the attempt of its execution counting from 1, and `lineage`, the `{execution, attempt}` entries it was dispatched under, empty for the run's own execution, beside `workflow_id`, which names the execution, `direct` in direct mode. `GraphTracerManager.open_tracer` takes `attempt` and `received_context`, the trace context a dispatched workflow received, which it continues: its parent node, its path and, when it names the workflow's place as a batch item's does, its chain as the workflow's lineage; `workflow_id` must be the id the engine keeps across attempts, and `TraceContext` carries the new `parent_path`, `child_segment` and `chain`, the last required with no default, so a host building a context of its own states it, `make_direct_chain()` in a single process. Node and edge ids name the chain that minted them, `<graph_id>:<workflow_id>:attempt_<n>:node_<k>`, followed after the attempt by a digest of the lineage for a dispatched workflow, so two attempts of one workflow never mint the same id; direct mode keeps `<graph_id>:node_<k>`. `ReportingManager.set_event_log` no longer takes `workflow_id`: a usage event takes its execution, attempt and lineage from the chain of its trace context, on the registered path as on the runner fallback. A trace event is frozen, and one whose `workflow_id` is empty is refused with `EventLogSchemaMismatchError`. The event logs keep every attempt's events, and every one of them, the DynamoDB and buffering logs included, returns a run's events deduplicated on the event's chain, writer, kind and sequence and ordered by chain, sequence and writer; the DynamoDB log's sort key becomes `EVENT#<chain digest>#<writer_id>#<sequence>`, the chain digest being the SHA-256 of the chain key, so the key stays within DynamoDB's limit however deep the lineage, and the NDJSON log writes one file per attempt of each execution, `wf_<workflow_id>__c_<chain digest>.ndjson`, `__w_<writer_id>` added for a writer other than `primary`. `GraphSpecAssembler` builds the graph, and dates it, from current events only: a higher attempt of an execution supersedes the earlier one and everything it dispatched, so a retried invocation appears once at its path, a lost attempt leaves no failed node, and a superseded attempt's usage counts in the run's total as unattributed usage.### Fixeddiff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yamlindex 4edb88038..a4bbf9bf2 100644--- a/docs/api-server/openapi/pipelex-api.openapi.yaml+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml@@ -2590,6 +2590,10 @@ components:- type: string- type: 'null'title: Stack+ caller_facing:+ type: boolean+ title: Caller Facing+ default: falseadditionalProperties: falsetype: objectrequired:@@ -4015,6 +4019,11 @@ components:id:type: stringtitle: Id+ path:+ anyOf:+ - type: string+ - type: 'null'+ title: Pathkind:$ref: '#/components/schemas/NodeKind'pipe_code:diff --git a/docs/errors/invocation-path-error.md b/docs/errors/invocation-path-error.mdnew file mode 100644index 000000000..3a1f4b462--- /dev/null+++ b/docs/errors/invocation-path-error.md@@ -0,0 +1,21 @@+---+title: "Invocation path not derivable"+description: "Reference for the `InvocationPathError` Pipelex error class."+---++<!-- pipelex:generated -->++# Invocation path not derivable++Raised when an invocation's path cannot be derived: a controller dispatched a pipe without naming its segment.++| Field | Value |+|---|---|+| `error_type` | `InvocationPathError` |+| `title` | Invocation path not derivable |+| `type_uri` | `https://docs.pipelex.com/latest/errors/invocation-path-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.system.exceptions` |+| Parent class | [`PipelexUnexpectedError`](pipelex-unexpected-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/platform-and-tooling.md b/docs/errors/platform-and-tooling.mdindex 7ed448331..cc600e69e 100644--- a/docs/errors/platform-and-tooling.md+++ b/docs/errors/platform-and-tooling.md@@ -72,6 +72,7 @@ own page. Classes are grouped by subsystem.- [`EnvVarNotFoundError`](env-var-not-found-error.md) — Environment variable not set- [`FatalError`](fatal-error.md) — Fatal error- [`FuncRegistryError`](func-registry-error.md) — Func registry+- [`InvocationPathError`](invocation-path-error.md) — Invocation path not derivable- [`JobMetadataError`](job-metadata-error.md) — Job metadata- [`LangfuseCredentialsError`](langfuse-credentials-error.md) — Langfuse credentials- [`MissingDependencyError`](missing-dependency-error.md) — Missing dependencydiff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.mdindex b92979b1b..3385f9c00 100644--- a/docs/under-the-hood/execution-graph-tracing.md+++ b/docs/under-the-hood/execution-graph-tracing.md@@ -225,6 +225,14 @@ Every `EdgeSpec` also carries an `optional` boolean (default `false`). On a `DATA `SKIPPED` node also carries `skip_reason` (a human-readable sentence naming the absent input, e.g. `skipped because input 'source' is absent`). A lifted pipe with a plural output still writes a real empty-list value, so its node registers that output and downstream `DATA` edges resolve normally; a lifted singular output is a recorded absence with no output spec.+A failed node's `error` carries `caller_facing`, true when its `message` was written for the caller: the error report of the error the pipe raised says so in its `caller_facing_message`, as the run's own failure report does. A reader that shows failures to a caller under the strict disclosure shows the message only when it is true. Tracing never changes how a run fails: the report is built while the pipe's error is on its way to the caller (`pipelex/graph/traced_failure.py`), so when building it raises, the fault is logged as an error, the node records the error's text with `caller_facing` false, and the run fails with the pipe's own error.++### Node Paths++Every node that runs a pipe carries its `path`, the [invocation path](#invocation-paths) of that run of the pipe, which names it the same way in every run of the method, live or dry, on every engine. A binding step's node runs no pipe and has no path. The `node_id` stays the graph's own key, unique within one graph and different from one run to the next; the path is the name a reader joins the views of a run on.++A `node_id`, like an `edge_id`, names the attempt that minted it (`make_trace_id_prefix`, `pipelex/system/trace_context.py`). Each attempt of an execution numbers its nodes and edges from 0, so the id carries the whole [chain](#executions-and-their-attempts): `<graph_id>:<workflow_id>:attempt_<n>:node_<k>` for the run's own workflow, followed after the attempt by `:lineage_<16 hex digits>`, the start of the SHA-256 digest of its lineage's key, for a workflow another one dispatched. Two attempts of one workflow, and a workflow that two attempts of its dispatcher dispatched under the same name, therefore mint different ids, and a reader keyed on `node_id` never merges a lost attempt's failed node with the one its retry ran. The direct mode's one execution, in its first attempt, mints `<graph_id>:node_<k>`.+---## Implementation@@ -252,6 +260,7 @@ job_metadata = JobMetadata(node_id, child_context = manager.on_pipe_start(trace_context=trace_context,pipe_code="extract_text",+ pipe_ref="my_domain.extract_text", # names the entry pipe in every path; None for a binding steppipe_type="PipeExtract",node_kind=NodeKind.OPERATOR,started_at=datetime.now(timezone.utc),@@ -274,7 +283,7 @@ graph_spec = manager.close_tracer(pipeline_run_id)Trace events travel through an `EventLogProtocol` backend (`pipelex/tracing/`): the tracer emits events into it during the run (write side, wired in `pipeline_run_setup`), and `assemble_tracing` reads them back after the run to build the `GraphSpec` and usage aggregates (read side, triggered from `PipeRun.run`). Both sides normally build their backend instance independently from `tracing_config` via `make_event_log` — NDJSON files or DynamoDB bridge the two instances through external storage.-The DynamoDB backend (`pipelex/tracing/dynamodb_event_log.py`) writes each event as one item keyed by the run (`PK = PIPELINE_RUN#{pipeline_run_id}`, `SK = EVENT#{workflow_id}#{writer_id}#{sequence}`), and every item carries a `ttl` in epoch seconds, set to the time of the write plus `runtime.tracing.dynamodb.retention_days` (30 days by default). DynamoDB deletes an item once its `ttl` has passed, provided the table names `ttl` as its TTL attribute, so a run's events, which by default hold its full output data, are not kept forever. The retention counts from the write rather than from the event's own `timestamp`, and a Temporal replay that re-emits an event overwrites the same item, pushing its expiry back by no more than the time between the two writes. A reader that needs a run's events, such as the graph and usage assembly, must therefore read them within the retention. Only the items written by a version that sets `ttl` expire: an item an earlier version wrote has no `ttl`, so it stays in the table until it is deleted or given one.+The DynamoDB backend (`pipelex/tracing/dynamodb_event_log.py`) writes each event as one item keyed by the run (`PK = PIPELINE_RUN#{pipeline_run_id}`, `SK = EVENT#{chain_digest}#{writer_id}#{sequence}`, see [Executions and Their Attempts](#executions-and-their-attempts)), and every item carries a `ttl` in epoch seconds, set to the time of the write plus `runtime.tracing.dynamodb.retention_days` (30 days by default). DynamoDB deletes an item once its `ttl` has passed, provided the table names `ttl` as its TTL attribute, so a run's events, which by default hold its full output data, are not kept forever. The retention counts from the write rather than from the event's own `timestamp`, and a Temporal replay that re-emits an event overwrites the same item, pushing its expiry back by no more than the time between the two writes. A reader that needs a run's events, such as the graph and usage assembly, must therefore read them within the retention. Only the items written by a version that sets `ttl` expire: an item an earlier version wrote has no `ttl`, so it stays in the table until it is deleted or given one.The assembled usage rides back on `pipe_output.tokens_usages` (with any assembly failure on `usage_assembly_error`), which the sync `/execute` response returns directly. For delivery-enabled runs (a storage target set), the delivery executor (`pipelex/pipe_run/delivery_executor.py`) also persists it as a `tokens_usages.json` result artifact — `{"tokens_usages": [...], "usage_assembly_error": null}` — next to `working_memory.json`, the `main_stuff.*` renders, and the graph outputs (`graphspec.json` with its three companions `pipe_io_contracts.json`, `input_form.json` and `output_form.json`, see [Outputs](#outputs)), so a durable client polling result files gets the same usage records a sync caller does. The artifact is written unconditionally on every successful result delivery: explicit nulls mean usage assembly was off for that run. A failed run stores no result files at all, so an absent `tokens_usages.json` means either the run failed or it was delivered before the artifact existed — tell those apart from the delivery status, not from the file's presence.@@ -300,21 +309,27 @@ This is what lets a graph-producing dry-run trace entirely in memory (no NDJSONThe tracer writes one event for each thing that happens in the run, in the order it happens (`pipelex/tracing/trace_events.py`). Every backend carries the events whole: the NDJSON and DynamoDB logs store each one as its JSON and read it back through the discriminated union `AnyTraceEvent`, and the in-memory and buffering logs hold the objects themselves. Every event carries these fields:-- `pipeline_run_id`, `workflow_id` and `writer_id`, which name the run, the workflow and the process that wrote it;+- `pipeline_run_id`, `workflow_id` and `writer_id`, which name the run, the execution that wrote it, never empty and `direct` in direct mode, and the process that wrote it;+- `attempt`, the attempt of that execution, counting from 1, and `lineage`, the attempts it was dispatched under (see [Executions and Their Attempts](#executions-and-their-attempts));+- `path`, the [invocation path](#invocation-paths) of the invocation the event is about, `null` for a binding step, which runs no pipe;- `timestamp`, and `sequence`, which rises with each event one writer writes;- `event_kind`, which tells the event's kind;- `mode`, `live` or `dry`, the mode of the run that wrote it.+An event is frozen, since it records what happened, and it computes its chain, its chain key and its store key once, the first time they are read, since every read and every assembly of a run reads them for each of its events; a copy made with `model_copy(update=...)` computes them again from its own fields. An event whose `workflow_id` is empty is refused when it is read, like any event in a shape this version does not accept, with `EventLogSchemaMismatchError`.++Each event kind that the hosted progress contract, which defines the run progress the hosted API reports, maps to a progress event carries that progress event's fields: the contract's `execution` is `workflow_id`, its `at` is `timestamp`, and its `kind` is read off `event_kind`.+The run's `TraceContext` carries the mode as `run_mode`, so a usage event that an activity writes from the context alone states it too. `mode` is required: an event stored by a version that did not write it is refused with `EventLogSchemaMismatchError` rather than read as live, and running the pipeline again writes a log in the current shape. `run_mode` is required too, with no default, on `TraceContext`, on `GraphTracerManager.open_tracer` and on the tracer's `setup`, since a default would stamp a dry run's events as live: whoever opens a trace states the mode, and a workflow that opens a tracer of its own for part of a run passes the `run_mode` of the context it received.The mode a trace states is the one the run's pipes run in, which is not always the one requested: a keyless boot (`needs_inference=False`) runs a requested live run dry. `pipeline_run_setup` therefore resolves the run's mode once, before it opens the tracer, through `resolve_effective_run_mode` (`pipelex/runtime_hub.py`), which first refuses `is_mock_usage` on a live request and then applies the keyless coercion. The tracer, the trace context, the OpenTelemetry context and the pipe job all take that one value, so a forced dry run states `dry` on every event and on its graph, and opens no OpenTelemetry trace, as no dry run does. The kernel tier holds to the same rule from the other side: its caller opens the trace, so `PipelexKernel.make` refuses, with `TraceContextRunModeError`, a trace context stating another mode than the one `resolve_effective_run_mode` gives the run (see [The Pipelex Kernel](pipelex-kernel.md)).| `event_kind` | Written when | What it carries ||---|---|---|-| `pipe_start` | A pipe starts | The node, its parent, the pipe's code, type, description and domain, its node kind and its inputs; a `PipeSequence` also states `step_count`, the number of steps its author wrote, and a `PipeParallel` states `branch_count`, the number of its branches, both `null` on every other pipe |-| `pipe_end_success` | A pipe ends with an output | The node, its end time, its output and its metrics |-| `pipe_end_error` | A pipe fails | The node, its end time and the error |-| `pipe_end_skipped` | A pipe is lifted because a plain input is absent | The node, its end time, the reason, and the empty list a plural output still writes |+| `pipe_start` | A pipe starts | The node, its parent, the pipe's code, its `pipe_ref`, its domain-qualified reference (`null` for a binding step), its type, description and domain, its node kind and its inputs; a `PipeSequence` also states `step_count`, the number of steps its author wrote, and a `PipeParallel` states `branch_count`, the number of its branches, both `null` on every other pipe |+| `pipe_end_success` | A pipe ends with an output | The node, the pipe's `pipe_code`, `pipe_ref` and `pipe_type`, as its start states them, its end time, its output and its metrics |+| `pipe_end_error` | A pipe fails | The node, the pipe's code, reference and type, its end time and the error, whose `caller_facing` says whether its message was written for the caller |+| `pipe_end_skipped` | A pipe is lifted because a plain input is absent | The node, the pipe's code, reference and type, its end time, the reason, and the empty list a plural output still writes || `edge` | The tracer draws an edge | Its source, target, kind, label and stuff digests || `controller_output` | A controller registers an output it produced | The node and the output || `batch_item` | A `PipeBatch` extracts an item from its input list | The list and item digests, the item index and the batch's node |@@ -332,7 +347,7 @@ A controller records each decision the moment it is final, so a reader followingThe controllers register their execution data at the same moments:-- A `PipeSequence` registers `step_count` as it starts, and `current_step`, the step's number among the steps its author wrote, as each step starts. The binding written for a dotted `batch_over` shares the number of the batch step after it.+- A `PipeSequence` registers `step_count` as it starts. It records no step number as each step starts: the number of a step is in the path of the pipe the step runs, `<sequence path>/step_<n>`.- A `PipeParallel` registers `branch_count`, `add_each_output` and `combined_output_concept` before any branch runs.- A `PipeBatch` registers `item_count` and `branch_pipe_code` with its fan-out. The two spellings of `branch_pipe_code` differ: the execution data carries the blueprint's field as its author wrote it, which may be domain-qualified, such as `my_domain.summarize`, while the `batch_fanned_out` event's `branch_pipe_code` is always the bare code, `summarize`, with `branch_pipe_ref` the qualified reference, `my_domain.summarize`.- A live `PipeCondition` registers `evaluated_expression` and `selected_outcome` with its resolution. Here `selected_outcome` is the outcome itself, the code of the pipe it runs or `continue` or `fail`, where the `condition_resolved` event names the outcome-map key.@@ -350,18 +365,56 @@ class TraceContext(BaseModel):node_sequence: int # Counter for generating node IDsdata_inclusion: DataInclusionConfig # What data to capturerun_mode: PipeRunMode # Live or dry, stated on every trace event+ parent_path: str | None # Invocation path of the pipe whose children run under this context+ child_segment: str | None # Segment a controller gives the next pipe it runs+ chain: list[ExecutionAttempt] # The lineage of the writing execution, then its own {execution, attempt}; required- def copy_for_child(self, child_node_id: str, *, next_sequence: int) -> TraceContext:- """Create context for nested pipe execution."""+ def copy_for_child(self, child_node_id: str, *, next_sequence: int, invocation_path: str | None) -> TraceContext:+ """Create context for nested pipe execution, under the child's own invocation path."""return TraceContext(graph_id=self.graph_id,parent_node_id=child_node_id,node_sequence=next_sequence,data_inclusion=self.data_inclusion,run_mode=self.run_mode,+ parent_path=invocation_path,+ child_segment=None,+ chain=list(self.chain),)```+A controller names the place of each pipe it runs before it runs it: it hands the pipe `job_metadata.for_child_at(segment=...)`, a copy whose trace context carries the segment, and the pipe's start derives its path from the context's `parent_path` and that segment (`TraceContext.invocation_path`). A pipe started under a parent path without a segment, or with a segment and no parent path, cannot be placed: `TraceContext.invocation_path` raises `InvocationPathError`, a fault of the runtime, never of the method. Tracing never fails a run, so the tracer catches it, traces the pipe with a `null` path, which leaves every pipe under it unplaceable too, and logs the fault as an error, `A pipe was traced without its invocation path`, with the pipe's `pipe_ref` and `node_id` in its fields. An inactive tracer and the no-op tracer record nothing and derive no path at all. The invariant that every invocation has a path is enforced by the test suite instead: an autouse fixture in `tests/conftest.py` fails any test during which a tracer logged that fault.++### Invocation Paths++The hosted progress contract names every invocation of a run by its path (`pipelex/system/invocation_path.py`): the run's entry pipe's `pipe_ref`, then one segment per controller level, such as `filings.review_filings/step_3/item_2/step_1`. A path holds no counter, no run-scoped id and no engine's name, so the same invocation has the same path in every run of the same method, live or dry, on every engine.++| Segment | Added by | Names |+|---|---|---|+| `step_<n>` | `PipeSequence` | The pipe its step `n` runs, numbered from 1 as the author wrote the steps, binding steps included. The binding written for a dotted `batch_over` is no step of the author's, takes no number and has no path |+| `branch_<n>` | `PipeParallel` | The pipe of its branch `n`, numbered from 1 in declaration order |+| `item_<n>` | `PipeBatch` | Its branch pipe, run for item `n` of the list: the item at index 0 is `item_1` |+| `outcome_<key>` | `PipeCondition` | The pipe of one outcome group: the outcomes that run one pipe, the default included, form one group, named by the first key of the outcome map that runs it, percent-encoded (`high risk` gives `outcome_high%20risk`). Outcomes are grouped by the pipe the library resolves each to, however they spell it, so a dependency's pipe written `alias->domain.code` by one outcome and `alias->code` by another is one group |+| `default` | `PipeCondition` | The pipe of its default outcome, when no key runs that pipe |++A step or a branch that batches runs as the `PipeBatch` the runtime makes for it, at the step's or the branch's segment, with its items under it. A live condition, which takes one group, and a dry one, which runs the pipe of every group once (`PipeCondition.outcome_group_pipes`), name each pipe by the same path.++A path is absolute whichever execution emits it. A batch item that runs as an execution of its own receives its item's place with its input: the item's `JobMetadata` carries the trace context whose `parent_path` is the batch's path and whose `child_segment` is `item_<n>`, so the execution's tracer, opened with `received_context`, stamps every path under the item's.++### Executions and Their Attempts++An execution is a part of a run that emits events and that its engine runs, and may run again, as a whole: the run's own workflow, and each batch item a distributed engine runs as a child workflow. In direct mode the whole run is one execution, `direct`. Each run of an execution is an attempt, numbered from 1.++`GraphTracerManager.open_tracer` takes the execution as `workflow_id`, which must be the id the engine keeps across attempts, the attempt as `attempt`, and, for an execution another one dispatched, the trace context it received as `received_context`. A received context naming a place hands the new execution the dispatcher's chain as its `lineage`; the run's own workflow receives a context naming no place, and its lineage is empty. Every event the tracer writes stamps `workflow_id`, `attempt` and `lineage`, and so does every usage event, which takes them from the chain of the trace context it is reported under: `ReportingManager.set_event_log` takes no workflow id.++`TraceContext.chain` is therefore required, with no default, as `run_mode` is: a context that defaulted to the direct mode's chain would write the usage of a durable workflow's second attempt under `direct@1`, a chain no attempt of that workflow has: no later attempt would supersede it, and the store keys would no longer tell the two attempts' usage apart. A tracer's `setup` states the chain on the context it returns; a host that builds a context of its own, as one driving the [kernel](pipelex-kernel.md) does, states it too, `make_direct_chain()` in a single process.++A higher attempt of an execution, dispatched under the same attempts, supersedes the earlier one and everything it dispatched, at any depth (`pipelex/system/execution_chain.py`, `current_chain_keys`). The assembler builds the graph from current events only, so a retried invocation appears once, at its path, and a lost attempt leaves no failed node behind; the graph's `created_at` is the timestamp of its earliest current event, so an attempt lost long before its retry does not date it either. A superseded attempt's inference was spent all the same, so its usage still counts in the run's total, as unattributed usage.++The event logs keep every attempt's events: their dedup key is the event's chain, its writer, its kind and its sequence (`TraceEvent.store_key`), since attempts of one execution number their events from the same start. The chain key reads `<execution>@<attempt>` for each entry of the chain, percent-encoded and joined by `/`. The DynamoDB log's sort key is `EVENT#<chain digest>#<writer_id>#<sequence>`, the chain digest being the SHA-256 hex digest of the chain key and the sequence zero-padded to ten digits: the chain key grows with the lineage and with the length of its workflow ids, while DynamoDB refuses a sort key over 1,024 bytes, so the digest keeps the key one length at any depth, and, being the same for the same chain, lets a replayed write overwrite its own row. The row's chain is read back from its payload, which holds the whole event. A row an earlier version wrote, whose event carries no `attempt`, `lineage` or `path`, is refused with `EventLogSchemaMismatchError` when it is read, as in every event log, rather than deduplicated against a current row; the schema is the backend's own, and no other writer's is promised to match it. Every event log returns a run's events by one rule, `deduplicate_and_order_events` (`pipelex/tracing/event_log_protocol.py`): one event per store key, ordered by chain, then sequence, then writer, whatever order its store holds them in, which for DynamoDB is the order of the digests. The deduplication drops the second write of an event that a replay wrote again, which a log that appends holds twice; the DynamoDB log's key already keeps one row per event.++The NDJSON log writes one file per attempt of each execution and per writer, `<pipeline_run_id>/wf_<workflow_id>__c_<chain digest>.ndjson`, with `__w_<writer_id>` before the extension for a writer other than `primary`, the chain digest being the one the DynamoDB sort key carries. Two attempts of one workflow, or a workflow that a new attempt dispatched again under the same name while its predecessor still drains, may write at once through handles of their own; naming the file by the chain keeps them from appending to one file, where their partial lines could interleave and leave the whole run's log unreadable. The reader reads every `.ndjson` file of the run's directory.+### Producer/Consumer ParadigmData flow in the graph follows a producer/consumer model:diff --git a/docs/under-the-hood/per-node-usage-attribution.md b/docs/under-the-hood/per-node-usage-attribution.mdindex cde8de131..6067d30c2 100644--- a/docs/under-the-hood/per-node-usage-attribution.md+++ b/docs/under-the-hood/per-node-usage-attribution.md@@ -21,7 +21,7 @@ flowchart TBCALL["inference call completes"]EVENT["UsageReportEvent<br/>{ node_id, tokens_usage }"]- subgraph LOG["event log — one stream, ordered by (workflow_id, sequence)"]+ subgraph LOG["event log — one stream, ordered by (chain, sequence)"]STREAM["PipeStart · PipeEndSuccess · Edge · UsageReport"]end@@ -49,7 +49,9 @@ flowchart TB## When attribution happens, and why it waits-`GraphSpecAssembler` folds usage events during pass 1, keyed by the `node_id` the event *named* — with no node lookup. Resolution against the real node set happens in pass 2. This is not an optimisation: a `UsageReportEvent` can legitimately be read **before** the `PipeStartEvent` of the node it names, because the two can come from different workers and the stream is ordered by `(workflow_id, sequence)`. "Is this node real?" is only answerable once every event has been seen.+`GraphSpecAssembler` folds usage events during pass 1, keyed by the `node_id` the event *named* — with no node lookup. Resolution against the real node set happens in pass 2. This is not an optimisation: a `UsageReportEvent` can legitimately be read **before** the `PipeStartEvent` of the node it names, because the two can come from different workers and the stream is ordered by `(chain, sequence)`, the chain being the attempts of the executions that wrote the event. "Is this node real?" is only answerable once every event has been seen.++A usage event of a superseded attempt, one a later attempt of its execution or of an execution that dispatched it replaced, names a node the graph does not show, since the graph is built from current events only (see [Executions and Their Attempts](execution-graph-tracing.md#executions-and-their-attempts)). Its inference was spent all the same, so the assembler counts it in the run's total, as unattributed usage.Pass 2 then does three things: resolves each accumulated total onto its node (or into the unattributed bucket), rolls subtrees up the `parent_node_id` chain, and computes the run total.diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.mdindex f080d3d63..64fa65c7a 100644--- a/docs/under-the-hood/pipelex-kernel.md+++ b/docs/under-the-hood/pipelex-kernel.md@@ -150,6 +150,7 @@ Everything else is yours:```pythonfrom pipelex.runtime_hub import get_report_delegate, resolve_effective_run_mode+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContextfrom pipelex.tracing.in_memory_event_log import InMemoryEventLog@@ -158,8 +159,10 @@ from pipelex.tracing.usage_aggregator import UsageAggregator# The mode the run takes on this boot: a boot without inference runs a live request dry.run_mode = resolve_effective_run_mode(requested=PipeRunMode.LIVE, is_mock_usage=False)event_log = InMemoryEventLog()-trace_context = TraceContext(graph_id=run_id, data_inclusion=data_inclusion, emit_graph_events=False, emit_usage_events=True, run_mode=run_mode)-get_report_delegate().set_event_log(context_key=trace_context.lookup_key, event_log=event_log, workflow_id="direct", pipeline_run_id=run_id)+trace_context = TraceContext(+ graph_id=run_id, data_inclusion=data_inclusion, emit_graph_events=False, emit_usage_events=True, run_mode=run_mode, chain=make_direct_chain()+)+get_report_delegate().set_event_log(context_key=trace_context.lookup_key, event_log=event_log, pipeline_run_id=run_id)try:kernel = PipelexKernel.make(user_id="my-service", storage_scope="my-service", read_scope=None, run_mode=run_mode, trace_context=trace_context)...@@ -168,6 +171,8 @@ finally:get_report_delegate().clear_event_log(context_key=trace_context.lookup_key)```+The trace context also states the chain of execution attempts its events belong to, required as the mode is, since every usage event takes its execution, its attempt and its lineage from it (see [Executions and Their Attempts](execution-graph-tracing.md#executions-and-their-attempts)). A host running the kernel in its own process states `make_direct_chain()`, the direct mode's one execution in its first attempt. A host running it inside a durable workflow states that workflow's own chain, its lineage followed by its id and the attempt its engine runs, so that the usage of a new attempt supersedes the earlier one's, where both would otherwise be written as `direct@1`.+The trace context states the mode the run takes, which `resolve_effective_run_mode` gives: every usage event a step writes states the context's `run_mode`, so a context saying live for a run whose steps run dry would stamp the whole run live. `PipelexKernel.make` refuses a context stating another mode with `TraceContextRunModeError` (`pipelex/kernel/exceptions.py`), a `ValueError` raised before any step runs. It refuses rather than correct its own copy because the trace is the caller's, possibly shared with writers the kernel never sees, which a corrected copy would leave disagreeing with the kernel's events.Passing a `trace_context` adopts its `graph_id` as the run's `pipeline_run_id`, and passing a `pipeline_run_id` that differs from it raises `ValueError` rather than letting either win. The two are one identity: letting them diverge would scatter a single run's usage events across two ids, because the registered-context emit path stamps the event log's id while the runner fallback stamps the metadata's — and a read-back keyed on either would silently miss the other's.diff --git a/pipelex/graph/graph_tracer.py b/pipelex/graph/graph_tracer.pyindex a79ce6f7f..13820ddab 100644--- a/pipelex/graph/graph_tracer.py+++ b/pipelex/graph/graph_tracer.py@@ -5,8 +5,9 @@ from typing import Anyfrom typing_extensions import override+from pipelex import logfrom pipelex.graph.condition_output_merge import ConditionOutputMerge, apply_condition_output_merges-from pipelex.graph.graph_tracer_protocol import GraphTracerProtocol+from pipelex.graph.graph_tracer_protocol import GraphTracerProtocol, invocation_path_of_startfrom pipelex.graph.graphspec import (EdgeKind,EdgeSpec,@@ -24,8 +25,10 @@ from pipelex.graph.graphspec import ()from pipelex.graph.selected_outcome import derive_selected_outcome_edgesfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.exceptions import InvocationPathError+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.pipe_run_mode import PipeRunMode-from pipelex.system.trace_context import TraceContext+from pipelex.system.trace_context import TraceContext, make_trace_id_prefixfrom pipelex.tracing.event_log_protocol import EventLogProtocol # ruff: ignore[typing-only-first-party-import] - used in __init__ annotationsfrom pipelex.tracing.trace_events import (BatchAggregateEvent,@@ -44,6 +47,29 @@ from pipelex.tracing.trace_events import (TraceEvent,)+INVOCATION_PATH_FAULT_MESSAGE = "A pipe was traced without its invocation path"+"""The error a tracer logs when it cannot derive an invocation's path; the test suite fails any test that logs it."""+++def _derive_invocation_path(*, trace_context: TraceContext, pipe_ref: str | None, pipe_type: str, node_kind: NodeKind, node_id: str) -> str | None:+ """The invocation path of a node started under `trace_context`, or None when it has none.++ A binding step runs no pipe, so it has no path. Every invocation has one, given by the segment its controller hands+ it through `JobMetadata.for_child_at`; a controller that hands it none is a fault of the runtime, never of the+ method, and tracing never fails a run for it: the node is traced without a path, its children cannot be placed+ either, and the fault is logged as an error. The test suite fails any test that logs it (`tests/conftest.py`), which+ is where the invariant that every invocation has a path is enforced.+ """+ try:+ return invocation_path_of_start(trace_context=trace_context, pipe_ref=pipe_ref, node_kind=node_kind)+ except InvocationPathError:+ log.error(+ INVOCATION_PATH_FAULT_MESSAGE,+ fields={"pipe_ref": pipe_ref, "pipe_type": pipe_type, "node_id": node_id},+ include_exception=True,+ )+ return None+class _MutableNodeData:"""Internal mutable data structure for building a node before finalization."""@@ -51,7 +77,9 @@ class _MutableNodeData:def __init__(self,node_id: str,+ path: str | None,pipe_code: str,+ pipe_ref: str | None,pipe_type: str,node_kind: NodeKind,started_at: datetime,@@ -61,7 +89,9 @@ class _MutableNodeData:domain_code: str | None = None,) -> None:self.node_id = node_id+ self.path = pathself.pipe_code = pipe_code+ self.pipe_ref = pipe_refself.pipe_type = pipe_typeself.node_kind = node_kindself.started_at = started_at@@ -97,6 +127,7 @@ class _MutableNodeData:return NodeSpec(node_id=self.node_id,+ path=self.path,kind=self.node_kind,pipe_code=self.pipe_code,pipe_type=self.pipe_type,@@ -155,8 +186,16 @@ class GraphTracer(GraphTracerProtocol):self._emit_graph_events: bool = True# Event log for distributed tracing (None = no event emission, direct mode)self._event_log: EventLogProtocol | None = None- # "direct" for single-process mode, full Temporal workflow ID otherwise+ # "direct" for single-process mode, the workflow's id otherwise: it names the events' execution, the same in+ # each of its attemptsself._workflow_id: str = "direct"+ # The attempt of that workflow its engine runs, and the attempts it was dispatched under (empty for the run's own)+ self._attempt: int = 1+ self._lineage: list[ExecutionAttempt] = []+ # The prefix of every node and edge id this attempt mints, which names the trace and the chain, so that no two+ # attempts of one execution, and no two executions dispatched under the same name by different attempts, share+ # an id (`make_trace_id_prefix`)+ self._id_prefix: str = ""# Pipeline run ID for event emissionself._pipeline_run_id: str | None = None# Per-writer monotonic counter for event sequencing@@ -210,21 +249,24 @@ class GraphTracer(GraphTracerProtocol):assert self._event_log is not Nonereturn self._event_log.writer_id+ def _path_of(self, *, node_id: str | None) -> str | None:+ """The invocation path of a node this tracer started, or None for any other node and for one that runs no pipe."""+ if node_id is None:+ return None+ node_data = self._nodes.get(node_id)+ if node_data is None:+ return None+ return node_data.path+def _make_node_id(self) -> str:- """Generate a node ID, including workflow_id when in Temporal mode."""- if self._workflow_id != "direct":- node_id = f"{self._graph_id}:{self._workflow_id}:node_{self._node_sequence}"- else:- node_id = f"{self._graph_id}:node_{self._node_sequence}"+ """Generate a node ID, unique to this attempt's chain: the prefix names the trace and the chain."""+ node_id = f"{self._id_prefix}:node_{self._node_sequence}"self._node_sequence += 1return node_iddef _make_edge_id(self) -> str:- """Generate an edge ID, including workflow_id when in Temporal mode."""- if self._workflow_id != "direct":- edge_id = f"{self._graph_id}:{self._workflow_id}:edge_{self._edge_sequence}"- else:- edge_id = f"{self._graph_id}:edge_{self._edge_sequence}"+ """Generate an edge ID, unique to this attempt's chain: the prefix names the trace and the chain."""+ edge_id = f"{self._id_prefix}:edge_{self._edge_sequence}"self._edge_sequence += 1return edge_id@@ -238,6 +280,8 @@ class GraphTracer(GraphTracerProtocol):pipeline_ref_main_pipe: str | None = None,event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",+ attempt: int = 1,+ lineage: list[ExecutionAttempt] | None = None,pipeline_run_id: str | None = None,emit_graph_events: bool = True,emit_usage_events: bool = True,@@ -252,8 +296,14 @@ class GraphTracer(GraphTracerProtocol):pipeline_ref_main_pipe: Optional main pipe name.event_log: Optional event log for distributed tracing. When set,every tracing method emits a TraceEvent as a side effect.- workflow_id: Temporal workflow ID or "direct" for single-process mode.- When not "direct", node/edge IDs include the workflow_id segment.+ workflow_id: The id of the workflow whose code emits the events, the same in each of its attempts, or+ "direct" for single-process mode. It names the events' execution. Outside the direct mode's first+ attempt, node and edge IDs name the chain: the workflow, its attempt and a digest of its lineage+ (`make_trace_id_prefix`).+ attempt: The attempt of that workflow its engine runs, counting from 1, stamped on every event; 1 in+ single-process mode.+ lineage: The attempts the workflow was dispatched under, from the run's own down to the one that+ dispatched it, stamped on every event; None or empty for the run's own workflow.pipeline_run_id: Pipeline run ID for event emission. Required when event_log is set.emit_graph_events: Whether this run assembles a GraphSpec. When False (costs-only mode),teardown skips the discarded spec build; the returned TraceContext carries the flag.@@ -281,6 +331,10 @@ class GraphTracer(GraphTracerProtocol):self._condition_output_merges = []self._event_log = event_logself._workflow_id = workflow_id+ self._attempt = attempt+ self._lineage = list(lineage or [])+ chain = [*self._lineage, ExecutionAttempt(execution=workflow_id, attempt=attempt)]+ self._id_prefix = make_trace_id_prefix(graph_id=graph_id, chain=chain)self._pipeline_run_id = pipeline_run_idself._event_sequence = 0self._pipe_registry = {}@@ -296,6 +350,7 @@ class GraphTracer(GraphTracerProtocol):emit_graph_events=emit_graph_events,emit_usage_events=emit_usage_events,run_mode=run_mode,+ chain=chain,)@override@@ -359,6 +414,9 @@ class GraphTracer(GraphTracerProtocol):self._event_log.close()self._event_log = Noneself._workflow_id = "direct"+ self._attempt = 1+ self._lineage = []+ self._id_prefix = ""self._pipeline_run_id = Noneself._event_sequence = 0self._pipe_registry = {}@@ -554,6 +612,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=batch_controller_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -599,6 +660,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=batch_controller_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -643,6 +707,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=parallel_controller_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -675,6 +742,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=merge.condition_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -715,6 +785,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=batch_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -759,6 +832,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=condition_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -774,6 +850,7 @@ class GraphTracer(GraphTracerProtocol):trace_context: TraceContext,*,pipe_code: str,+ pipe_ref: str | None,pipe_type: str,node_kind: NodeKind,started_at: datetime,@@ -785,20 +862,31 @@ class GraphTracer(GraphTracerProtocol):step_count: int | None = None,branch_count: int | None = None,) -> tuple[str, TraceContext]:- """Record the start of a pipe execution."""+ """Record the start of a pipe execution.++ The node's invocation path is derived here, from the path of the invocation it runs under and the segment its+ controller gave it, or from its own `pipe_ref` for the run's entry pipe; the pipes it runs continue it. Tracing+ never fails a run: an inactive tracer records nothing and derives no path, and a node whose path cannot be+ derived is traced without one (see `_derive_invocation_path`).+ """if not self._is_active:# Return dummy values when not activenode_id = trace_context.make_node_id()- child_context = trace_context.copy_for_child(node_id, next_sequence=trace_context.node_sequence + 1)+ child_context = trace_context.copy_for_child(node_id, next_sequence=trace_context.node_sequence + 1, invocation_path=None)return node_id, child_context- # Generate node ID (includes workflow_id in Temporal mode)+ # Generate node ID (names the chain outside the direct mode's first attempt)node_id = self._make_node_id()+ invocation_path = _derive_invocation_path(+ trace_context=trace_context, pipe_ref=pipe_ref, pipe_type=pipe_type, node_kind=node_kind, node_id=node_id+ )# Create mutable node datanode_data = _MutableNodeData(node_id=node_id,+ path=invocation_path,pipe_code=pipe_code,+ pipe_ref=pipe_ref,pipe_type=pipe_type,node_kind=node_kind,started_at=started_at,@@ -828,12 +916,16 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=node_data.path,mode=self._run_mode,timestamp=started_at,sequence=self._next_event_sequence(),node_id=node_id,parent_node_id=trace_context.parent_node_id,pipe_code=pipe_code,+ pipe_ref=pipe_ref,pipe_type=pipe_type,node_kind=node_kind,description=description,@@ -858,6 +950,7 @@ class GraphTracer(GraphTracerProtocol):child_context = trace_context.copy_for_child(child_node_id=node_id,next_sequence=self._node_sequence,+ invocation_path=invocation_path,)return node_id, child_context@@ -884,6 +977,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=node_data.path,mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -945,10 +1041,16 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=node_data.path,mode=self._run_mode,timestamp=ended_at,sequence=self._next_event_sequence(),node_id=node_id,+ pipe_code=node_data.pipe_code,+ pipe_ref=node_data.pipe_ref,+ pipe_type=node_data.pipe_type,ended_at=ended_at,output_spec=output_spec,metrics=metrics or {},@@ -991,6 +1093,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=node_data.path,mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),@@ -1008,6 +1113,7 @@ class GraphTracer(GraphTracerProtocol):error_type: str,error_message: str,error_stack: str | None = None,+ caller_facing: bool = False,) -> None:"""Record failed completion of a pipe execution."""if not self._is_active:@@ -1023,6 +1129,7 @@ class GraphTracer(GraphTracerProtocol):error_type=error_type,message=error_message,stack=error_stack,+ caller_facing=caller_facing,)node_data.error = error@@ -1033,10 +1140,16 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=node_data.path,mode=self._run_mode,timestamp=ended_at,sequence=self._next_event_sequence(),node_id=node_id,+ pipe_code=node_data.pipe_code,+ pipe_ref=node_data.pipe_ref,+ pipe_type=node_data.pipe_type,ended_at=ended_at,error=error,)@@ -1091,10 +1204,16 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=node_data.path,mode=self._run_mode,timestamp=ended_at,sequence=self._next_event_sequence(),node_id=node_id,+ pipe_code=node_data.pipe_code,+ pipe_ref=node_data.pipe_ref,+ pipe_type=node_data.pipe_type,ended_at=ended_at,skip_reason=skip_reason,output_spec=output_spec,@@ -1139,6 +1258,9 @@ class GraphTracer(GraphTracerProtocol):pipeline_run_id=self._event_pipeline_run_id,writer_id=self._event_writer_id(),workflow_id=self._workflow_id,+ attempt=self._attempt,+ lineage=list(self._lineage),+ path=self._path_of(node_id=target_node_id),mode=self._run_mode,timestamp=datetime.now(UTC),sequence=self._next_event_sequence(),diff --git a/pipelex/graph/graph_tracer_manager.py b/pipelex/graph/graph_tracer_manager.pyindex 3312e7553..9d3e838ce 100644--- a/pipelex/graph/graph_tracer_manager.py+++ b/pipelex/graph/graph_tracer_manager.py@@ -1,7 +1,7 @@"""Graph tracer manager with singleton pattern for access from PipeAbstract without hub imports."""from datetime import datetime-from typing import Any+from typing import TYPE_CHECKING, Anyfrom pipelex import logfrom pipelex.graph.condition_output_merge import ConditionOutputMerge@@ -15,6 +15,9 @@ from pipelex.system.trace_context import TraceContextfrom pipelex.tools.log.error_fields import error_fieldsfrom pipelex.tracing.event_log_protocol import EventLogProtocol # ruff: ignore[typing-only-first-party-import] - used in open_tracer signature+if TYPE_CHECKING:+ from pipelex.system.execution_chain import ExecutionAttempt+class GraphTracerManager(metaclass=ABCSingletonMeta):"""Singleton manager for graph tracing supporting multiple concurrent pipeline runs.@@ -103,6 +106,8 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):pipeline_ref_main_pipe: str | None = None,event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",+ attempt: int = 1,+ received_context: TraceContext | None = None,pipeline_run_id: str | None = None,tracer_key: str | None = None,emit_graph_events: bool = True,@@ -119,7 +124,16 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):pipeline_ref_main_pipe: Optional main pipe name.event_log: Optional event log for distributed tracing. When set,the tracer emits trace events as a side effect alongside in-memory accumulation.- workflow_id: Temporal workflow ID or "direct" for single-process mode.+ workflow_id: The id of the workflow whose code emits the events, the same in each of its attempts, or+ "direct" for single-process mode. It names the events' execution, so it must never be an id the+ engine changes from one attempt to the next.+ attempt: The attempt of that workflow its engine runs, counting from 1; 1 in single-process mode.+ received_context: The trace context a workflow received with its input, when it runs part of a run that+ another workflow dispatched to it, such as a batch item. The returned context continues it: the+ parent node, so that CONTAINS edges link back to the dispatching workflow's node, the parent path and+ the segment the dispatcher gave, so that every path the workflow stamps continues the one it was handed,+ and, when the dispatcher named the workflow's place, the received chain as the workflow's lineage.+ A received context naming no place, as the run's own workflow receives, leaves the lineage empty.pipeline_run_id: Pipeline run ID for event emission.tracer_key: Lookup key for the tracer in the manager's dict. Defaults to graph_id.In Temporal mode, use the workflow_id to avoid collisions when multiple@@ -165,6 +179,12 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):tracer = GraphTracer()+ # The dispatching attempt hands a dispatched workflow its own chain beside the path prefix: that chain is the+ # lineage of every event the dispatched workflow writes. The run's own workflow was handed no place, and has none.+ lineage: list[ExecutionAttempt] = []+ if received_context is not None and received_context.child_segment is not None:+ lineage = list(received_context.chain)+trace_context = tracer.setup(graph_id=graph_id,data_inclusion=data_inclusion,@@ -172,6 +192,8 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):pipeline_ref_main_pipe=pipeline_ref_main_pipe,event_log=event_log,workflow_id=workflow_id,+ attempt=attempt,+ lineage=lineage,pipeline_run_id=pipeline_run_id,emit_graph_events=emit_graph_events,emit_usage_events=emit_usage_events,@@ -182,6 +204,14 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):trace_context = trace_context.model_copy(update={"tracer_key": tracer_key})if not describe_pipe_io:trace_context = trace_context.model_copy(update={"describe_pipe_io": False})+ if received_context is not None:+ trace_context = trace_context.model_copy(+ update={+ "parent_node_id": received_context.parent_node_id,+ "parent_path": received_context.parent_path,+ "child_segment": received_context.child_segment,+ }+ )# Register only once setup has fully succeeded: a failure above must not leave# an unusable tracer entry in the process-wide manager.@@ -237,6 +267,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):trace_context: TraceContext,*,pipe_code: str,+ pipe_ref: str | None,pipe_type: str,node_kind: NodeKind,started_at: datetime,@@ -253,6 +284,8 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):Args:trace_context: Current trace context containing graph_id.pipe_code: The pipe code being executed.+ pipe_ref: The `domain.code` reference of the pipe, which names the run's entry pipe in every invocation+ path; None for a node that runs no pipe, such as a binding step.pipe_type: The pipe type (e.g., "PipeLLM", "PipeSequence").node_kind: The kind of node (controller, operator, etc.).started_at: When the pipe started executing.@@ -274,6 +307,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):return tracer.on_pipe_start(trace_context=trace_context,pipe_code=pipe_code,+ pipe_ref=pipe_ref,pipe_type=pipe_type,node_kind=node_kind,started_at=started_at,@@ -390,6 +424,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):error_type: str,error_message: str,error_stack: str | None = None,+ caller_facing: bool = False,) -> None:"""Record failed completion of a pipe execution.@@ -400,6 +435,8 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):error_type: The exception type name.error_message: The error message.error_stack: Optional stack trace.+ caller_facing: Whether the message was written for the caller, as the error report of the error the+ pipe raised says in its `caller_facing_message`."""if node_id is None:return@@ -414,6 +451,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):error_type=error_type,error_message=error_message,error_stack=error_stack,+ caller_facing=caller_facing,)def register_controller_output(diff --git a/pipelex/graph/graph_tracer_protocol.py b/pipelex/graph/graph_tracer_protocol.pyindex 0f3b7fc35..1598092d2 100644--- a/pipelex/graph/graph_tracer_protocol.py+++ b/pipelex/graph/graph_tracer_protocol.py@@ -6,11 +6,33 @@ from typing_extensions import overridefrom pipelex.graph.condition_output_merge import ConditionOutputMergefrom pipelex.graph.graphspec import EdgeKind, GraphSpec, IOSpec, NodeKindfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.exceptions import InvocationPathError+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContextfrom pipelex.tracing.event_log_protocol import EventLogProtocol # ruff: ignore[typing-only-first-party-import] - used in setup signature+def invocation_path_of_start(*, trace_context: TraceContext, pipe_ref: str | None, node_kind: NodeKind) -> str | None:+ """The invocation path of a node started under `trace_context`, or None for a node that runs no pipe.++ Args:+ trace_context: The context the node starts under, carrying its parent's path and the segment its controller+ gave it.+ pipe_ref: The `domain.code` reference of the pipe the node runs, which names the run's entry pipe.+ node_kind: The node's kind: a binding step runs no pipe, so no invocation has its path.++ Raises:+ InvocationPathError: The node runs a pipe, and either no `pipe_ref` was given or the context cannot place it.+ """+ if not node_kind.is_invocation:+ return None+ if pipe_ref is None:+ msg = f"A {node_kind} node was started without the reference of the pipe it runs, so its invocation path cannot be derived."+ raise InvocationPathError(msg)+ return trace_context.invocation_path(pipe_ref=pipe_ref)++class GraphTracerProtocol(Protocol):"""Protocol for building GraphSpec during pipe execution.@@ -27,6 +49,8 @@ class GraphTracerProtocol(Protocol):pipeline_ref_main_pipe: str | None = None,event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",+ attempt: int = 1,+ lineage: list[ExecutionAttempt] | None = None,pipeline_run_id: str | None = None,emit_graph_events: bool = True,emit_usage_events: bool = True,@@ -40,7 +64,11 @@ class GraphTracerProtocol(Protocol):pipeline_ref_domain: Optional domain name for the pipeline.pipeline_ref_main_pipe: Optional main pipe name.event_log: Optional event log for distributed tracing.- workflow_id: Temporal workflow ID or "direct" for single-process mode.+ workflow_id: The id of the workflow whose code emits the events, the same in each of its attempts, or+ "direct" for single-process mode. It names the events' execution.+ attempt: The attempt of that workflow its engine runs, counting from 1; 1 in single-process mode.+ lineage: The attempts the workflow was dispatched under, from the run's own down to the one that+ dispatched it; None or empty for the run's own workflow.pipeline_run_id: Pipeline run ID for event emission.emit_graph_events: Whether graph (node/edge) trace events should be emitted for this run.Stamped onto the returned TraceContext so it is born with the correct flag.@@ -67,6 +95,7 @@ class GraphTracerProtocol(Protocol):trace_context: TraceContext,*,pipe_code: str,+ pipe_ref: str | None,pipe_type: str,node_kind: NodeKind,started_at: datetime,@@ -83,6 +112,8 @@ class GraphTracerProtocol(Protocol):Args:trace_context: Current trace context from JobMetadata.pipe_code: The pipe code being executed.+ pipe_ref: The `domain.code` reference of the pipe, which names the run's entry pipe in every invocation+ path; None for a node that runs no pipe, such as a binding step.pipe_type: The pipe type (e.g., "PipeLLM", "PipeSequence").node_kind: The kind of node (controller, operator, etc.).started_at: When the pipe started executing.@@ -96,6 +127,9 @@ class GraphTracerProtocol(Protocol):Returns:Tuple of (node_id for this pipe, updated TraceContext for children).++ Tracing never fails a run, so a tracer raises nothing when it cannot derive the pipe's invocation path: it+ traces the pipe without one and logs the fault."""...@@ -129,6 +163,7 @@ class GraphTracerProtocol(Protocol):error_type: str,error_message: str,error_stack: str | None = None,+ caller_facing: bool = False,) -> None:"""Record failed completion of a pipe execution.@@ -138,6 +173,8 @@ class GraphTracerProtocol(Protocol):error_type: The exception type name.error_message: The error message.error_stack: Optional stack trace.+ caller_facing: Whether the message was written for the caller, as an error report's+ `caller_facing_message` says."""...@@ -352,6 +389,8 @@ class GraphTracerNoOp(GraphTracerProtocol):pipeline_ref_main_pipe: str | None = None,event_log: "EventLogProtocol | None" = None,workflow_id: str = "direct",+ attempt: int = 1,+ lineage: list[ExecutionAttempt] | None = None,pipeline_run_id: str | None = None,emit_graph_events: bool = True,emit_usage_events: bool = True,@@ -363,6 +402,7 @@ class GraphTracerNoOp(GraphTracerProtocol):emit_graph_events=emit_graph_events,emit_usage_events=emit_usage_events,run_mode=run_mode,+ chain=[*(lineage or []), ExecutionAttempt(execution=workflow_id, attempt=attempt)],)@override@@ -375,6 +415,7 @@ class GraphTracerNoOp(GraphTracerProtocol):trace_context: TraceContext,*,pipe_code: str,+ pipe_ref: str | None,pipe_type: str,node_kind: NodeKind,started_at: datetime,@@ -386,8 +427,9 @@ class GraphTracerNoOp(GraphTracerProtocol):step_count: int | None = None,branch_count: int | None = None,) -> tuple[str, TraceContext]:+ # The no-op tracer records nothing, so it derives no invocation path either: tracing never fails a run.node_id = trace_context.make_node_id()- child_context = trace_context.copy_for_child(node_id, next_sequence=trace_context.node_sequence + 1)+ child_context = trace_context.copy_for_child(node_id, next_sequence=trace_context.node_sequence + 1, invocation_path=None)return node_id, child_context@override@@ -421,6 +463,7 @@ class GraphTracerNoOp(GraphTracerProtocol):error_type: str,error_message: str,error_stack: str | None = None,+ caller_facing: bool = False,) -> None:passdiff --git a/pipelex/graph/graphspec.py b/pipelex/graph/graphspec.pyindex 88c918858..2ab0a2c79 100644--- a/pipelex/graph/graphspec.py+++ b/pipelex/graph/graphspec.py@@ -51,6 +51,15 @@ class NodeKind(StrEnum):# A PipeSequence binding step: it runs no pipe, and produces the stuff it binds from a path in working memory.BINDING = "binding"+ @property+ def is_invocation(self) -> bool:+ """Whether a node of this kind is the run of a pipe, which an invocation path names."""+ match self:+ case NodeKind.PIPE_CALL | NodeKind.CONTROLLER | NodeKind.OPERATOR:+ return True+ case NodeKind.INPUT | NodeKind.OUTPUT | NodeKind.ARTIFACT | NodeKind.ERROR | NodeKind.BINDING:+ return False+class NodeStatus(StrEnum):"""Execution status of a node."""@@ -279,6 +288,10 @@ class ErrorSpec(BaseModel):error_type: strmessage: strstack: str | None = None+ # Whether `message` was written for the caller, as the `caller_facing_message` of the error report of the error the+ # pipe raised says: a reader that discloses failures to the caller under the strict disclosure shows the message+ # only when this is true.+ caller_facing: bool = False@field_validator("stack", mode="after")@classmethod@@ -424,6 +437,10 @@ class NodeSpec(BaseModel):model_config = ConfigDict(extra="forbid", strict=True, populate_by_name=True)node_id: str = Field(validation_alias="id", serialization_alias="id")+ # The invocation path naming the pipe run this node is, the same in every run of the method and on every engine+ # (`pipelex.system.invocation_path`), beside the run-scoped `node_id`; None for a node that runs no pipe, such as a+ # binding step.+ path: str | None = Nonekind: NodeKind = Field(strict=False)pipe_code: str | None = Nonepipe_type: str | None = Nonediff --git a/pipelex/graph/traced_failure.py b/pipelex/graph/traced_failure.pynew file mode 100644index 000000000..6b36b9b4c--- /dev/null+++ b/pipelex/graph/traced_failure.py@@ -0,0 +1,44 @@+"""What the execution graph records of the error that ended a node: its message, and whether it was written for the caller."""++from typing import NamedTuple++from pipelex import log+from pipelex.base_exceptions import PipelexError+from pipelex.tools.log.error_fields import error_fields++ERROR_REPORT_FAULT_MESSAGE = "The error report of a failed pipe could not be built for its trace"+"""The error logged when a failed pipe's error report raises while its node's failure is traced."""+++class TracedFailure(NamedTuple):+ """The message a failed node records, and whether that message was written for the caller."""++ message: str+ caller_facing: bool+++def make_traced_failure(*, error: BaseException) -> TracedFailure:+ """What a node ended by `error` records of it.++ Whether the message was written for the caller is the error report's to say, as it says it of the run's own failure:+ a Pipelex error records its report's message and `caller_facing_message`, and any other exception records its text,+ which is never written for the caller.++ Tracing never changes how a run fails. This runs while the pipe's own error is being handled, on its way to the+ caller, so building the report must not raise in its place: a report that cannot be built is logged as an error,+ and the node records the error's text, as for any other exception.+ """+ if isinstance(error, PipelexError):+ try:+ error_report = error.to_error_report()+ except Exception as report_fault: # ruff: ignore[blind-except]+ # Unbounded code: `to_error_report` is overridden across the whole error hierarchy, the errors plugins+ # define included, and walks the error's cause chain, so what it raises cannot be enumerated. The pipe's own+ # error is the one the run fails with, so the fault is logged and the node records the error's text.+ log.error(+ ERROR_REPORT_FAULT_MESSAGE,+ fields={"failed_error_type": type(error).__name__, **error_fields(exc=report_fault)},+ )+ return TracedFailure(message=str(error), caller_facing=False)+ return TracedFailure(message=error_report.message, caller_facing=error_report.caller_facing_message)+ return TracedFailure(message=str(error), caller_facing=False)diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.pyindex 6408b100f..e8130dcc0 100644--- a/pipelex/pipe_controllers/batch/pipe_batch.py+++ b/pipelex/pipe_controllers/batch/pipe_batch.py@@ -16,6 +16,7 @@ from pipelex.interpreter_hub import get_pipe_router, get_required_pipefrom pipelex.pipe_controllers.pipe_controller import PipeControllerfrom pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams+from pipelex.system.invocation_path import item_segmentfrom pipelex.system.job_metadata import JobMetadatafrom pipelex.system.telemetry.otel_constants import OTelLogAttrfrom pipelex.tools.log.log_fields import USER_ACTION_FIELD@@ -189,10 +190,13 @@ class PipeBatch(PipeController):# `run_batch_branch`, not `run`: this is the one dispatch in the pipe tree whose# semantics are "a per-item fan-out branch". In-process routers treat it as a plain# run (the protocol's default body); a distributed router may isolate it.+ # The item's pipe is named by its invocation path, whose `item_<n>` segment numbers the items from 1: the+ # item at index 0 of the list is `item_1`. A router that runs the item as an execution of its own hands it+ # this metadata, so every path that execution stamps continues the item's.return await get_pipe_router().run_batch_branch(pipe_job=PipeJobFactory.make_pipe_job(pipe=sub_pipe,- job_metadata=job_metadata,+ job_metadata=job_metadata.for_child_at(segment=item_segment(item_number=branch_index + 1)),working_memory=branch_memory,pipe_run_params=branch_pipe_run_params,output_name=None,diff --git a/pipelex/pipe_controllers/binding/binding_step.py b/pipelex/pipe_controllers/binding/binding_step.pyindex c4994f14b..6f51fe705 100644--- a/pipelex/pipe_controllers/binding/binding_step.py+++ b/pipelex/pipe_controllers/binding/binding_step.py@@ -29,6 +29,7 @@ from pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.graph.graphspec import NodeKindfrom pipelex.graph.stuff_io_spec import make_stuff_io_spec+from pipelex.graph.traced_failure import make_traced_failurefrom pipelex.pipe_controllers.binding.binding_derivation import BindingDerivationfrom pipelex.pipe_controllers.binding.binding_value import FoundNothing, bind_contentfrom pipelex.system.job_metadata import JobMetadata@@ -234,9 +235,11 @@ class BindingStep(BaseModel):if root_stuff is not Noneelse None)+ # A binding runs no pipe, so it has no pipe reference and no invocation path.node_id, _ = tracer_manager.on_pipe_start(trace_context=trace_context,pipe_code=self.from_path,+ pipe_ref=None,pipe_type=BINDING_NODE_PIPE_TYPE,node_kind=NodeKind.BINDING,started_at=datetime.now(UTC),@@ -285,10 +288,23 @@ class BindingStep(BaseModel):tracer_manager = GraphTracerManager.get_instance()if tracer_manager is None:return+ error_type: str+ error_message: str+ caller_facing: bool+ if error is None:+ error_type = "UnknownError"+ error_message = "the binding step stopped without binding a value"+ caller_facing = False+ else:+ traced_failure = make_traced_failure(error=error)+ error_type = type(error).__name__+ error_message = traced_failure.message+ caller_facing = traced_failure.caller_facingtracer_manager.on_pipe_end_error(lookup_key=trace_context.lookup_key,node_id=node_id,ended_at=datetime.now(UTC),- error_type=type(error).__name__ if error is not None else "UnknownError",- error_message=str(error) if error is not None else "the binding step stopped without binding a value",+ error_type=error_type,+ error_message=error_message,+ caller_facing=caller_facing,)diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.pyindex c067a898a..7821ac15d 100644--- a/pipelex/pipe_controllers/condition/pipe_condition.py+++ b/pipelex/pipe_controllers/condition/pipe_condition.py@@ -33,6 +33,7 @@ from pipelex.pipe_machinery.pipe_abstract import PipeAbstractfrom pipelex.pipe_machinery.template_guard_lint import lint_authored_templatefrom pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_apply+from pipelex.system.invocation_path import DEFAULT_OUTCOME_SEGMENT, outcome_segmentfrom pipelex.system.job_metadata import JobMetadatafrom pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesErrorfrom pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables@@ -542,7 +543,7 @@ class PipeCondition(PipeController):return await get_pipe_router().run(pipe_job=PipeJobFactory.make_pipe_job(pipe=chosen_pipe,- job_metadata=job_metadata,+ job_metadata=job_metadata.for_child_at(segment=self.outcome_group_segment(pipe=chosen_pipe)),working_memory=working_memory,pipe_run_params=pipe_run_params,output_name=output_name,@@ -550,6 +551,48 @@ class PipeCondition(PipeController):),)+ def outcome_group_segment(self, *, pipe: PipeAbstract) -> str:+ """The invocation-path segment of the outcome group that runs `pipe`.++ The outcomes that run one pipe form one group, the default included: the group is named by the first key, in the+ order the outcome map declares them, that runs its pipe, and a group only the default outcome runs is `default`.+ A live run, which takes one group, and a dry run, which runs the pipe of every group, so name each pipe by the+ same path. Outcomes are told apart by the pipe the library resolves each to, never by how they spell it: a+ dependency's pipe may be written `alias->domain.code` in one outcome and `alias->code` in another, and both run+ the one pipe. An outcome naming no pipe the library holds runs nothing here, so it joins no group.++ Args:+ pipe: The pipe an outcome runs, as the library resolves it.+ """+ for outcome_key, outcome_value in self.outcome_map.items():+ if outcome_value in SpecialOutcome.value_list():+ continue+ if get_optional_pipe(pipe_code=outcome_value) is pipe:+ return outcome_segment(outcome_key=outcome_key)+ return DEFAULT_OUTCOME_SEGMENT++ def outcome_group_pipes(self) -> list[PipeAbstract]:+ """The pipe of each outcome group, once per group, in the order a dry run runs them.++ A dry run cannot know which outcome a live run takes, so it runs the pipe of every group, and runs it once+ however many outcomes run it and however they spell it: the outcomes are grouped by the pipe the library+ resolves each to, as `outcome_group_segment` groups them. The groups run in the order of the sorted spellings+ of their pipes, the default outcome's pipe last, so that the steps after the condition read the value a live+ run falls back to when nothing matches, and the order is a property of the method rather than of set iteration.++ Raises:+ PipeNotFoundError: An outcome names a pipe the library does not hold.+ """+ pipes_by_visit_key: dict[str, PipeAbstract] = {}+ for pipe_code in sorted(self.pipe_dependencies()):+ outcome_pipe = get_required_pipe(pipe_code=pipe_code)+ pipes_by_visit_key.setdefault(outcome_pipe.visit_key, outcome_pipe)+ group_pipes = list(pipes_by_visit_key.values())+ if not self.default_outcome or self.default_outcome in SpecialOutcome.value_list():+ return group_pipes+ default_pipe = get_required_pipe(pipe_code=self.default_outcome)+ return [outcome_pipe for outcome_pipe in group_pipes if outcome_pipe is not default_pipe] + [default_pipe]+def _record_resolution(self, *, job_metadata: JobMetadata, evaluated_expression: str, outcome: str) -> None:"""Record the outcome a live run chose, in the execution data and as the condition's resolution trace event.@@ -627,22 +670,21 @@ class PipeCondition(PipeController):))- # A dry run cannot know which outcome a live run takes, so every pipe outcome runs. Each- # outcome but the last runs on a copy of the memory and run params this condition received,- # as PipeParallel runs its branches: no outcome sees what a sibling wrote, and the memory- # this condition leaves is the one a live run choosing the last outcome would leave. The- # last outcome is the default outcome's pipe, so the steps after the condition read the- # value a live run falls back to when nothing matches, and the order is a property of the- # method rather than of set iteration order. The graph then shows the condition's result- # as one stuff that every outcome produces, which the merge registered below records.+ # A dry run cannot know which outcome a live run takes, so the pipe of every outcome group+ # runs, once per group (`outcome_group_pipes`). Each group but the last runs on a copy of the+ # memory and run params this condition received, as PipeParallel runs its branches: no group+ # sees what a sibling wrote, and the memory this condition leaves is the one a live run+ # choosing the last group would leave. The last group is the default outcome's, so the steps+ # after the condition read the value a live run falls back to when nothing matches. The graph+ # then shows the condition's result as one stuff that every outcome produces, which the merge+ # registered below records.received_stuff_codes = {stuff.stuff_code for stuff in working_memory.root.values()}outcome_slots: list[DryRunOutcomeSlot] = []- outcome_pipe_codes = self._dry_run_outcome_order()- for outcome_index, pipe_code in enumerate(outcome_pipe_codes):- is_last_outcome = outcome_index == len(outcome_pipe_codes) - 1- outcome_pipe = get_required_pipe(pipe_code=pipe_code)+ outcome_pipes = self.outcome_group_pipes()+ for outcome_index, outcome_pipe in enumerate(outcome_pipes):+ is_last_outcome = outcome_index == len(outcome_pipes) - 1outcome_output = await outcome_pipe.run_pipe(- job_metadata=job_metadata,+ job_metadata=job_metadata.for_child_at(segment=self.outcome_group_segment(pipe=outcome_pipe)),working_memory=working_memory if is_last_outcome else working_memory.make_deep_copy(),pipe_run_params=pipe_run_params if is_last_outcome else pipe_run_params.make_deep_copy(),output_name=output_name,@@ -692,14 +734,6 @@ class PipeCondition(PipeController):)return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)- def _dry_run_outcome_order(self) -> list[str]:- """The pipe outcomes in the order a dry run runs them: sorted, with the default outcome's pipe last."""- pipe_codes = sorted(self.pipe_dependencies())- if self.default_outcome in pipe_codes:- pipe_codes.remove(self.default_outcome)- pipe_codes.append(self.default_outcome)- return pipe_codes-def _register_dry_run_output_merge(self,*,diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex c0ea18729..54edf8a1b 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -41,6 +41,7 @@ from pipelex.pipe_controllers.sub_pipe import SubPipefrom pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaintfrom pipelex.pipe_machinery.pipe_abstract import CompanionSlotfrom pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_apply+from pipelex.system.invocation_path import branch_segmentfrom pipelex.system.job_metadata import JobMetadatafrom pipelex.validation_error_types import PipeValidationErrorType@@ -477,11 +478,13 @@ class PipeParallel(PipeController):tasks: list[Coroutine[Any, Any, PipeOutput]] = []- for sub_pipe in self.parallel_sub_pipes:+ # Each branch's pipe is named by its invocation path, whose `branch_<n>` segment numbers the branches from 1 in+ # the order they are declared; a branch that batches runs as the PipeBatch made for it, at that segment.+ for branch_index, sub_pipe in enumerate(self.parallel_sub_pipes):tasks.append(sub_pipe.run_pipe(calling_pipe_code=self.code,- job_metadata=job_metadata,+ job_metadata=job_metadata.for_child_at(segment=branch_segment(branch_number=branch_index + 1)),working_memory=working_memory.make_deep_copy(),sub_pipe_run_params=pipe_run_params.make_deep_copy(),library_crate=library_crate,diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.pyindex 926f67c59..cd47a4b58 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py@@ -47,6 +47,7 @@ from pipelex.pipe_controllers.sub_pipe import SubPipefrom pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaint, is_same_value_spec, taint_after_write, write_after_liftfrom pipelex.pipe_machinery.validation import is_valid_input_name, suggest_plain_namefrom pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams, output_multiplicity_to_apply+from pipelex.system.invocation_path import step_segmentfrom pipelex.system.job_metadata import JobMetadatafrom pipelex.validation_error_types import PipeValidationErrorType@@ -1038,17 +1039,12 @@ class PipeSequence(PipeController):typed_flow: SequenceTypedFlow | None = None# By step index, each binding the run derived from the value its root held, the flow having no spec for that root.runtime_bindings: dict[int, DerivedBinding] = {}- # The sequence records how many steps it runs as it starts, and the number of each step as that step starts.+ # The sequence records how many steps it runs as it starts. Each step's pipe is named by its invocation path,+ # whose `step_<n>` segment is the step's number among the steps the author wrote.self._register_execution_data(job_metadata=job_metadata, execution_data={"step_count": self.traced_step_count()})- current_step: int | None = Nonefor sub_pipe_index, sub_pipe in enumerate(self.sequential_sub_pipes):is_last_step = sub_pipe_index == len(self.sequential_sub_pipes) - 1- step_number = self._step_number(step_index=sub_pipe_index)- # The binding written for a dotted `batch_over` shares the number of the batch step after it: one record each.- if step_number != current_step:- current_step = step_number- self._register_execution_data(job_metadata=job_metadata, execution_data={"current_step": step_number})if isinstance(sub_pipe, BindingStep):if typed_flow is None:typed_flow = self.build_typed_flow()@@ -1072,10 +1068,11 @@ class PipeSequence(PipeController):sub_pipe_run_params = pipe_run_params.model_copy()else:sub_pipe_run_params = pipe_run_params.model_copy(update=({"final_stuff_code": None}))+ # A step that batches runs as the PipeBatch made for it, at the step's own segment.pipe_output = await sub_pipe.run_pipe(calling_pipe_code=self.code,working_memory=evolving_memory,- job_metadata=job_metadata,+ job_metadata=job_metadata.for_child_at(segment=step_segment(step_number=self._step_number(step_index=sub_pipe_index))),sub_pipe_run_params=sub_pipe_run_params,library_crate=library_crate,)diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.pyindex 9a64351ca..d7c16d708 100644--- a/pipelex/pipe_machinery/pipe_abstract.py+++ b/pipelex/pipe_machinery/pipe_abstract.py@@ -26,6 +26,7 @@ from pipelex.core.stuffs.stuff import Stufffrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.graph.graph_tracer_manager import GraphTracerManager, IOSpec, NodeKindfrom pipelex.graph.stuff_io_spec import make_stuff_io_spec+from pipelex.graph.traced_failure import make_traced_failurefrom pipelex.libraries.library_crate import LibraryCratefrom pipelex.pipe_machinery.memory_writes import MemoryWritefrom pipelex.pipe_machinery.pipe_blueprint import PipeCategory, PipeType, valid_pipe_type_tags@@ -812,6 +813,7 @@ class PipeAbstract(ABC, BaseModel):graph_node_id, child_trace_context = tracer_manager.on_pipe_start(trace_context=parent_trace_context,pipe_code=self.code,+ pipe_ref=self.pipe_ref,pipe_type=self.type,node_kind=node_kind,started_at=started_at,@@ -875,13 +877,15 @@ class PipeAbstract(ABC, BaseModel):error_stack: str | None = Noneif parent_trace_context.data_inclusion.error_stack_traces:error_stack = traceback.format_exc()+ traced_failure = make_traced_failure(error=exc)tracer_manager.on_pipe_end_error(lookup_key=parent_trace_context.lookup_key,node_id=graph_node_id,ended_at=datetime.now(UTC),error_type=type(exc).__name__,- error_message=str(exc),+ error_message=traced_failure.message,error_stack=error_stack,+ caller_facing=traced_failure.caller_facing,)raisediff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.pyindex d7ef4e23f..3229d023d 100644--- a/pipelex/pipeline/pipeline_run_setup.py+++ b/pipelex/pipeline/pipeline_run_setup.py@@ -415,7 +415,6 @@ async def pipeline_run_setup(get_report_delegate().set_event_log(context_key=pipeline_run_id,event_log=event_log,- workflow_id="direct",pipeline_run_id=pipeline_run_id,)diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 8930abf6d..e5fa39378 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -53,10 +53,13 @@ _EMIT_BEST_EFFORT_EXCEPTIONS: tuple[type[BaseException], ...] = (class _EventLogContext(NamedTuple):- """Per-workflow/run event log state. Private to ReportingManager."""+ """Per-workflow/run event log state. Private to ReportingManager.++ It holds no workflow id: a usage event names its execution, its attempt and its lineage from the trace context it was+ reported under, whose chain the workflow's tracer opened, so the event's chain is always the one its context states.+ """event_log: EventLogProtocol- workflow_id: strpipeline_run_id: str@@ -76,7 +79,6 @@ class ReportingManager(ReportingProtocol):*,context_key: str,event_log: EventLogProtocol,- workflow_id: str,pipeline_run_id: str,) -> None:"""Configure event log for a specific workflow/run context.@@ -84,8 +86,6 @@ class ReportingManager(ReportingProtocol):Args:context_key: Unique key for this context (trace_context.lookup_key).event_log: The event log backend for emitting UsageReportEvents.- workflow_id: Run-scoped execution identity stamped into events- (Temporal run ID, or "direct" outside Temporal).pipeline_run_id: Pipeline run ID for event correlation."""# The silent overwrite-on-existing-key below is intentional and load-bearing.@@ -99,7 +99,6 @@ class ReportingManager(ReportingProtocol):# the eviction-poison class on the cost path (pre-M1 open_tracer did exactly that).self._event_log_contexts[context_key] = _EventLogContext(event_log=event_log,- workflow_id=workflow_id,pipeline_run_id=pipeline_run_id,)@@ -237,7 +236,10 @@ class ReportingManager(ReportingProtocol):event = UsageReportEvent(pipeline_run_id=context.pipeline_run_id,- workflow_id=context.workflow_id,+ workflow_id=trace_context.own_attempt.execution,+ attempt=trace_context.own_attempt.attempt,+ lineage=trace_context.lineage,+ path=trace_context.parent_path,writer_id=context.event_log.writer_id,mode=trace_context.run_mode,timestamp=datetime.now(UTC),@@ -306,15 +308,24 @@ class ReportingManager(ReportingProtocol):if process_event_log is None:return- workflow_id = trace_context.tracer_key or trace_context.graph_id+ # The execution, its attempt and its lineage are the trace context's, as on the fast path: an activity's usage+ # belongs to the attempt of the workflow that ran it, which a new attempt of that workflow supersedes.+ workflow_id = trace_context.own_attempt.executionnode_id = trace_context.parent_node_id or UNATTRIBUTED_NODE_ID- ActivityEventLogCache.log_once_runner_fallback_engaged(workflow_id=workflow_id, writer_id=process_event_log.writer_id)+ # The line names the run's tracer key beside the execution: the execution is `direct` for every run in direct+ # mode and under the kernel, so it alone would not say which run engaged the fallback.+ ActivityEventLogCache.log_once_runner_fallback_engaged(+ workflow_id=workflow_id, tracer_key=trace_context.lookup_key, writer_id=process_event_log.writer_id+ )seq = process_event_log.next_sequence()event = UsageReportEvent(pipeline_run_id=inference_job.job_metadata.run_metadata.pipeline_run_id,workflow_id=workflow_id,+ attempt=trace_context.own_attempt.attempt,+ lineage=trace_context.lineage,+ path=trace_context.parent_path,writer_id=process_event_log.writer_id,mode=trace_context.run_mode,timestamp=datetime.now(UTC),diff --git a/pipelex/reporting/reporting_protocol.py b/pipelex/reporting/reporting_protocol.pyindex 467bdb521..89b5c9caa 100644--- a/pipelex/reporting/reporting_protocol.py+++ b/pipelex/reporting/reporting_protocol.py@@ -18,7 +18,6 @@ class ReportingProtocol(Protocol):*,context_key: str,event_log: EventLogProtocol,- workflow_id: str,pipeline_run_id: str,) -> None: ...@@ -44,7 +43,6 @@ class ReportingNoOp(ReportingProtocol):*,context_key: str,event_log: EventLogProtocol,- workflow_id: str,pipeline_run_id: str,) -> None:passdiff --git a/pipelex/system/exceptions.py b/pipelex/system/exceptions.pyindex d913d8bf2..b36cba67e 100644--- a/pipelex/system/exceptions.py+++ b/pipelex/system/exceptions.py@@ -13,6 +13,16 @@ class JobMetadataError(PipelexUnexpectedError):pass+class InvocationPathError(PipelexUnexpectedError):+ """Raised when an invocation's path cannot be derived: a controller dispatched a pipe without naming its segment.++ Every controller names the place of each pipe it runs before it runs it, so this is a fault of the runtime, never+ of the method.+ """++ _declared_title = "Invocation path not derivable"++class NestedKeyConflictError(ToolError):"""Raised when attempting to create nested keys under a non-dict value."""diff --git a/pipelex/system/execution_chain.py b/pipelex/system/execution_chain.pynew file mode 100644index 000000000..1ab40b414--- /dev/null+++ b/pipelex/system/execution_chain.py@@ -0,0 +1,96 @@+"""Executions, their attempts, and the chain that ties a trace event to the attempts it ran under.++The hosted progress contract (`conformance/specs/pipelex-platform-api.md`, "Executions and their attempts") calls an+**execution** a part of a run that emits events and that its engine runs, and may run again, as a whole: the run's own+workflow, and every workflow it dispatches, which on every engine Pipelex runs on today is each batch item's. Each run+of an execution is an **attempt**, numbered from 1. In the runtime's direct mode the whole run is one execution,+`direct`, in its first attempt.++An execution that dispatches another hands it its own **lineage** followed by its own `{execution, attempt}`, so an+event's **chain**, its lineage followed by its own entry, runs from the run's own execution down to the one that+emitted it. A higher attempt of an execution, dispatched under the same attempts, **supersedes** the earlier one and+everything it dispatched, at any depth: an event is current when no attempt in its chain is superseded.+"""++import hashlib+from collections.abc import Iterable, Sequence+from urllib.parse import quote++from pydantic import BaseModel, ConfigDict, Field++DIRECT_EXECUTION = "direct"+"""The one execution of a run in the runtime's direct mode, which runs the whole run in one process."""+++class ExecutionAttempt(BaseModel):+ """One attempt of one execution: the workflow whose code emits events, and which run of it by its engine.++ Attributes:+ execution: Names the execution, the same in each of its attempts: the id of the workflow, or `direct`.+ attempt: The attempt of that execution, counting from 1.+ """++ model_config = ConfigDict(extra="forbid", frozen=True, strict=True)++ execution: str = Field(min_length=1)+ attempt: int = Field(ge=1)+++def make_direct_chain() -> list[ExecutionAttempt]:+ """The chain of a run in the runtime's direct mode: its one execution, in its first attempt."""+ return [ExecutionAttempt(execution=DIRECT_EXECUTION, attempt=1)]+++def execution_chain_key(*, chain: Sequence[ExecutionAttempt]) -> str:+ """One string naming a chain, distinct for every two chains that differ.++ Each entry reads `<execution>@<attempt>`, the execution percent-encoded so that no name holds a separator, and the+ entries are joined by `/`. The empty chain, the lineage of the run's own execution, gives the empty string.+ """+ return "/".join(f"{quote(entry.execution, safe='')}@{entry.attempt}" for entry in chain)+++def execution_chain_digest(*, chain_key: str) -> str:+ """The SHA-256 hexadecimal digest of a chain's key, which names the chain in one length however deep it is.++ A chain key grows with the lineage and with the length of its execution names, while the places that key on a chain+ bound their keys, such as a DynamoDB sort key or a file name. The digest is the same for the same chain, so a+ replayed write lands where its first write did, and differs for every two chains that differ.+ """+ return hashlib.sha256(chain_key.encode("utf-8")).hexdigest()+++def current_chain_keys(*, chains: Iterable[Sequence[ExecutionAttempt]]) -> set[str]:+ """The keys of the chains, among those given, that no attempt of the run supersedes.++ An attempt in a chain is superseded when some chain of the run holds a higher attempt of the same execution after+ the same entries, that is, dispatched under the same attempts. A chain is current when no attempt in it is+ superseded, its own included. The rule compares attempt numbers within one execution and never a clock.++ Args:+ chains: The chains of every event of the run, in any order, repeats included.++ Returns:+ The `execution_chain_key` of every chain given that is current.+ """+ distinct_chains: dict[str, Sequence[ExecutionAttempt]] = {}+ for chain in chains:+ distinct_chains.setdefault(execution_chain_key(chain=chain), chain)++ # The highest attempt of each execution, per the entries it was dispatched under.+ latest_attempts: dict[tuple[str, str], int] = {}+ for chain in distinct_chains.values():+ for entry_index, entry in enumerate(chain):+ slot = (execution_chain_key(chain=chain[:entry_index]), entry.execution)+ latest_attempts[slot] = max(latest_attempts.get(slot, 0), entry.attempt)++ current_keys: set[str] = set()+ for chain_key, chain in distinct_chains.items():+ is_current = True+ for entry_index, entry in enumerate(chain):+ if entry.attempt < latest_attempts[execution_chain_key(chain=chain[:entry_index]), entry.execution]:+ is_current = False+ break+ if is_current:+ current_keys.add(chain_key)+ return current_keysdiff --git a/pipelex/system/invocation_path.py b/pipelex/system/invocation_path.pynew file mode 100644index 000000000..aca6c48e2--- /dev/null+++ b/pipelex/system/invocation_path.py@@ -0,0 +1,69 @@+"""The invocation path: the name of one run of one pipe inside a run.++The hosted progress contract (`conformance/specs/pipelex-platform-api.md`, "Invocation paths") names every invocation+of a run by its path: the run's entry pipe's `pipe_ref`, then one segment per controller level down to the invocation,+such as `filings.review_filings/step_3/item_2/step_1`. A path holds no counter, no run-scoped id and no engine's name,+so the same invocation has the same path in every run of the same method, on every engine, live or dry.++The segments, under the controller that adds each:++- `step_<n>` under a `PipeSequence`: its step `n`, numbered from 1 as the author wrote the steps, binding steps+ included. A binding step runs no pipe, so no invocation has its path.+- `branch_<n>` under a `PipeParallel`: its branch `n`, numbered from 1 in declaration order.+- `item_<n>` under a `PipeBatch`: the branch pipe run for item `n` of the list, numbered from 1.+- `outcome_<key>` under a `PipeCondition`: the pipe of one outcome group, named by the first key of the outcome map+ that runs it, percent-encoded.+- `default` under a `PipeCondition`: the pipe of its default outcome, when no key of its outcome map runs that pipe.++A step or a branch that batches runs as the `PipeBatch` the runtime makes for it, which is the invocation at the step's+or the branch's segment, its items under it.+"""++from urllib.parse import quote++from pipelex.system.exceptions import InvocationPathError++PATH_SEPARATOR = "/"+DEFAULT_OUTCOME_SEGMENT = "default"+++def _ordinal(*, segment_kind: str, number: int) -> int:+ """The number of a segment, which counts from 1.++ Raises:+ InvocationPathError: The number is below 1.+ """+ if number < 1:+ msg = f"An invocation path's '{segment_kind}' segment counts from 1, and was given {number}."+ raise InvocationPathError(msg)+ return number+++def step_segment(*, step_number: int) -> str:+ """The segment of the pipe a sequence's step runs, its number counting from 1 as the author wrote the steps."""+ return f"step_{_ordinal(segment_kind='step', number=step_number)}"+++def branch_segment(*, branch_number: int) -> str:+ """The segment of the pipe of a parallel's branch, numbered from 1 in the order the parallel declares them."""+ return f"branch_{_ordinal(segment_kind='branch', number=branch_number)}"+++def item_segment(*, item_number: int) -> str:+ """The segment of a batch's branch pipe run for one item of its list: the item at index 0 is `item_1`."""+ return f"item_{_ordinal(segment_kind='item', number=item_number)}"+++def outcome_segment(*, outcome_key: str) -> str:+ """The segment of a condition's outcome group, named by a key of its outcome map.++ Every byte of the key's UTF-8 form outside the unreserved set (letters, digits, `-`, `.`, `_` and `~`) is written as+ `%` and two upper-case hexadecimal digits, so `high risk` gives `outcome_high%20risk` and a key holding a `/` cannot+ split a segment.+ """+ return f"outcome_{quote(outcome_key, safe='')}"+++def child_path(*, parent_path: str, segment: str) -> str:+ """The path of the invocation a controller runs at `segment`, under its own path."""+ return f"{parent_path}{PATH_SEPARATOR}{segment}"diff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.pyindex b546526f4..b8da2f1f7 100644--- a/pipelex/system/job_metadata.py+++ b/pipelex/system/job_metadata.py@@ -295,6 +295,20 @@ class JobMetadata(BaseModel):pipe_run_id=self.pipe_run_id,)+ def for_child_at(self, *, segment: str) -> "JobMetadata":+ """A copy of this metadata for a pipe a controller runs, naming that pipe's place under the controller.++ The segment rides on the trace context, beside the controller's own invocation path, and the pipe's start+ derives its path from the two (see ``TraceContext.invocation_path``). Without a trace context there is no+ path to derive, so the metadata is returned as it is.++ Args:+ segment: The pipe's segment under its controller, from ``pipelex.system.invocation_path``.+ """+ if self.trace_context is None:+ return self+ return self.model_copy(update={"trace_context": self.trace_context.with_child_segment(segment=segment)})+def copy_with_update(self,*,diff --git a/pipelex/system/trace_context.py b/pipelex/system/trace_context.pyindex f16265e47..ec4c6918b 100644--- a/pipelex/system/trace_context.py+++ b/pipelex/system/trace_context.py@@ -12,13 +12,56 @@ Both streams hang off the same node tree, so the node-minting machinery(``make_node_id`` / ``copy_for_child`` / ``node_sequence``) is shared, not graph-only.The node tree is always built when either stream is on — including costs-only(``--no-graph --costs``) mode, where usage still needs ``parent_node_id`` to attribute to.++Beside the run-scoped node ids, the context carries what names an invocation in every view of a+run (``pipelex.system.invocation_path``): the path of the invocation its pipes run under, the+segment a controller gives the next pipe it runs, and the chain of execution attempts the events+written under it belong to (``pipelex.system.execution_chain``)."""+from collections.abc import Sequence+from pydantic import BaseModel, ConfigDict, Fieldfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.exceptions import InvocationPathError+from pipelex.system.execution_chain import ExecutionAttempt, execution_chain_digest, execution_chain_key, make_direct_chain+from pipelex.system.invocation_path import child_pathfrom pipelex.system.pipe_run_mode import PipeRunMode+LINEAGE_DIGEST_LENGTH = 16+"""How many hexadecimal digits of its lineage's digest an id minted by a dispatched execution carries."""+++def make_trace_id_prefix(*, graph_id: str, chain: Sequence[ExecutionAttempt]) -> str:+ """The prefix of every node and edge id that one attempt of one execution mints in the trace `graph_id`.++ An attempt numbers its nodes and edges from 0, and an execution keeps its name in each of its attempts, so ids made+ of the trace, the execution and that number alone would give a retried attempt the ids of the attempt it replaces,+ and an execution that a new attempt dispatched again under the same name the ids of its predecessor: a reader keyed+ on `node_id` would then merge a failed node and a succeeded one. The prefix therefore names the whole chain: the+ execution, its attempt, and, for an execution another one dispatched, the first `LINEAGE_DIGEST_LENGTH` digits of its+ lineage's digest, which keeps the id one length however deep the lineage. The prefix is a function of the chain+ alone, so every replay of an attempt mints the same ids. The direct mode's one execution, in its first attempt, is+ the only chain of its run and keeps the bare `graph_id`.++ Examples: `run_1` in direct mode, `run_1:exec-run:attempt_2` for the second attempt of the run's own workflow, and+ `run_1:exec-item-2:attempt_1:lineage_<16 hexadecimal digits>` for a batch item's workflow.++ Args:+ graph_id: The trace the ids belong to.+ chain: The chain of the attempt minting the ids: its lineage followed by its own `{execution, attempt}`.+ """+ if list(chain) == make_direct_chain():+ return graph_id+ own_attempt = chain[-1]+ prefix = f"{graph_id}:{own_attempt.execution}:attempt_{own_attempt.attempt}"+ lineage = chain[:-1]+ if lineage:+ lineage_digest = execution_chain_digest(chain_key=execution_chain_key(chain=lineage))+ prefix += f":lineage_{lineage_digest[:LINEAGE_DIGEST_LENGTH]}"+ return prefix+class TraceContext(BaseModel):"""Serializable per-execution trace context passed through JobMetadata.@@ -45,6 +88,17 @@ class TraceContext(BaseModel):run_mode: Whether the run is live or a dry run. Every trace event the run emits states it, theusage events an activity emits included, which have this context and nothing else of the run.Required, with no default: a context built without it would stamp a dry run's events as live.+ parent_path: The invocation path of the pipe whose children run under this context, None above the+ run's entry pipe. It is absolute from the entry pipe, whichever execution reads it: a batch item's+ own execution receives its item's path as the prefix of every path it stamps.+ child_segment: The segment, under ``parent_path``, that a controller gives the next pipe it runs. Set+ on the copy of the context the controller hands that pipe, and never carried further down.+ chain: The execution attempts the events written under this context belong to, from the run's own+ execution down to the one writing them: its lineage followed by its own ``{execution, attempt}``. Required,+ with no default: a usage event takes its execution, its attempt and its lineage from it, so a context built+ without it would write a retried workflow's usage as the direct mode's first attempt, which no later attempt+ supersedes. A tracer's ``setup`` states it; a host building a context of its own states it too, the direct+ mode's being ``make_direct_chain()``."""model_config = ConfigDict(strict=True, extra="forbid")@@ -58,29 +112,77 @@ class TraceContext(BaseModel):emit_usage_events: bool = Field(default=True, description="Whether to emit usage (cost) trace events")describe_pipe_io: bool = Field(default=True, description="Whether to build the I/O artifacts describing the graphspec's data")run_mode: PipeRunMode = Field(strict=False, description="Whether the run is live or a dry run, stated by whoever opens the trace")+ parent_path: str | None = Field(default=None, description="Invocation path of the pipe whose children run under this context")+ child_segment: str | None = Field(default=None, description="Segment a controller gives the next pipe it runs, under parent_path")+ chain: list[ExecutionAttempt] = Field(+ min_length=1,+ description="The execution attempts this context's events belong to, ending with the one writing them, stated by whoever builds it",+ )@propertydef lookup_key(self) -> str:"""Key for looking up the tracer in GraphTracerManager."""return self.tracer_key or self.graph_id+ @property+ def own_attempt(self) -> ExecutionAttempt:+ """The attempt of the execution that writes the events under this context."""+ return self.chain[-1]++ @property+ def lineage(self) -> list[ExecutionAttempt]:+ """The attempts the writing execution was dispatched under, empty for the run's own execution."""+ return self.chain[:-1]++ def with_child_segment(self, *, segment: str) -> "TraceContext":+ """A copy of this context naming the place, under its parent path, of the next pipe a controller runs."""+ return self.model_copy(update={"child_segment": segment})++ def invocation_path(self, *, pipe_ref: str) -> str:+ """The path of the invocation of a pipe started under this context.++ The run's entry pipe, started above any parent path, is named by its `pipe_ref`. Any other pipe is named by+ the parent path followed by the segment its controller gave it.++ Args:+ pipe_ref: The `domain.code` reference of the pipe started, which names the run's entry pipe.++ Raises:+ InvocationPathError: A pipe started under a parent path without a segment, or with a segment and no+ parent path to put it under.+ """+ if self.parent_path is None:+ if self.child_segment is not None:+ msg = f"The pipe '{pipe_ref}' was given the segment '{self.child_segment}' with no parent invocation to place it under."+ raise InvocationPathError(msg)+ return pipe_ref+ if self.child_segment is None:+ msg = (+ f"The pipe '{pipe_ref}' started under the invocation '{self.parent_path}' without the segment its "+ "controller gives it, so its invocation path cannot be derived."+ )+ raise InvocationPathError(msg)+ return child_path(parent_path=self.parent_path, segment=self.child_segment)+def make_node_id(self) -> str:- """Generate a unique node ID within this trace.+ """Generate a node ID unique within this trace, as the tracer does (see `make_trace_id_prefix`).Returns:- A unique node ID in format "{graph_id}:node_{sequence}".+ A node ID in format "{prefix}:node_{sequence}", the prefix naming the trace and the chain."""- return f"{self.graph_id}:node_{self.node_sequence}"+ return f"{make_trace_id_prefix(graph_id=self.graph_id, chain=self.chain)}:node_{self.node_sequence}"- def copy_for_child(self, child_node_id: str, *, next_sequence: int) -> "TraceContext":+ def copy_for_child(self, child_node_id: str, *, next_sequence: int, invocation_path: str | None) -> "TraceContext":"""Create a child context for a nested pipe execution.Args:child_node_id: The node ID assigned to the child pipe.next_sequence: The next sequence number for the child context.+ invocation_path: The child pipe's invocation path, which the pipes it runs continue; None for a node that+ is no invocation, such as a binding step, under which no pipe runs.Returns:- A new TraceContext with updated parent_node_id and node_sequence.+ A new TraceContext with updated parent_node_id, node_sequence and parent_path, and no child segment."""return TraceContext(graph_id=self.graph_id,@@ -92,4 +194,7 @@ class TraceContext(BaseModel):emit_usage_events=self.emit_usage_events,describe_pipe_io=self.describe_pipe_io,run_mode=self.run_mode,+ parent_path=invocation_path,+ child_segment=None,+ chain=list(self.chain),)diff --git a/pipelex/tracing/activity_event_log.py b/pipelex/tracing/activity_event_log.pyindex d85452b80..d0ae0ed20 100644--- a/pipelex/tracing/activity_event_log.py+++ b/pipelex/tracing/activity_event_log.py@@ -77,7 +77,7 @@ class ActivityEventLogCache:return cls._cached_event_log@classmethod- def log_once_runner_fallback_engaged(cls, *, workflow_id: str, writer_id: str) -> None:+ def log_once_runner_fallback_engaged(cls, *, workflow_id: str, tracer_key: str, writer_id: str) -> None:"""Log a single INFO the first time activity-side emission engages.The log is per-process; subsequent emissions from the same process are@@ -85,6 +85,11 @@ class ActivityEventLogCache:(co-located or split deployment alike) — INFO level, one line, sooperators can see the writer_id in the worker logs without log spam athigh activity throughput.++ Args:+ workflow_id: The execution whose usage engaged it, `direct` in direct mode and under the kernel.+ tracer_key: The lookup key of the run's tracer, its tracer key or its graph id, which names the run.+ writer_id: The process's writer id, stamped on every event it emits."""if cls._fallback_logged:return@@ -97,7 +102,7 @@ class ActivityEventLogCache:# the workflow router or on a separate worker pool.log.info("Usage events are emitted through this process's activity event log",- fields={"writer_id": writer_id, "workflow_id": workflow_id},+ fields={"writer_id": writer_id, "workflow_id": workflow_id, "tracer_key": tracer_key},)@classmethoddiff --git a/pipelex/tracing/buffering_event_log.py b/pipelex/tracing/buffering_event_log.pyindex ff55bd1d8..97bc889f8 100644--- a/pipelex/tracing/buffering_event_log.py+++ b/pipelex/tracing/buffering_event_log.py@@ -8,7 +8,7 @@ import threadingfrom typing_extensions import override-from pipelex.tracing.event_log_protocol import EventLogProtocol+from pipelex.tracing.event_log_protocol import EventLogProtocol, deduplicate_and_order_eventsfrom pipelex.tracing.trace_events import TraceEvent@@ -58,8 +58,8 @@ class BufferingEventLog(EventLogProtocol):@overridedef read_events(self, pipeline_run_id: str) -> list[TraceEvent]:- """Return buffered events matching the pipeline_run_id."""- return [event for event in self._buffer if event.pipeline_run_id == pipeline_run_id]+ """Return the buffered events of the run, deduplicated and ordered as every event log returns them."""+ return deduplicate_and_order_events(events=[event for event in self._buffer if event.pipeline_run_id == pipeline_run_id])@overridedef close(self) -> None:diff --git a/pipelex/tracing/dynamodb_event_log.py b/pipelex/tracing/dynamodb_event_log.pyindex d07ccc5b7..4df718636 100644--- a/pipelex/tracing/dynamodb_event_log.py+++ b/pipelex/tracing/dynamodb_event_log.py@@ -2,13 +2,23 @@Stores trace events in a DynamoDB table with the schema:PK: PIPELINE_RUN#{pipeline_run_id}- SK: EVENT#{workflow_id}#{writer_id}#{sequence:010d}- payload: dict (full event via model_dump)+ SK: EVENT#{chain_digest}#{writer_id}#{sequence:010d}+ payload: the full event as JSON, its lineage, workflow and attempt includedttl: epoch seconds at which DynamoDB may delete the item, the write time plus the configured retention-Legacy emissions write writer_id="primary" so existing rows remain valid.+The chain digest is the SHA-256 of the event's chain key, the event's lineage+followed by its own workflow and attempt (`TraceEvent.chain_key`), so a+workflow's new attempt, whose sequence starts again, never overwrites the rows+of its earlier one, while the sort key keeps one length however deep the+lineage and however long its workflow ids: DynamoDB refuses a sort key over+1,024 bytes. The row's chain itself is read back from its payload.++The schema is this backend's own, and no other writer's is promised to match+it. A row written by a version whose sort key named the workflow and no chain+holds an event without `attempt`, `lineage` or `path`, which the read refuses+with `EventLogSchemaMismatchError` rather than deduplicating it against a+current row.-Compatible with the hosted trace-event DynamoDB adapter schema.Requires: pip install "pipelex[dynamodb]""""@@ -21,9 +31,10 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.system.exceptions import MissingDependencyError+from pipelex.system.execution_chain import execution_chain_digestfrom pipelex.tools.log.error_fields import error_fieldsfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error-from pipelex.tracing.event_log_protocol import EventLogProtocol+from pipelex.tracing.event_log_protocol import EventLogProtocol, deduplicate_and_order_eventsfrom pipelex.tracing.exceptions import EventLogReadError, EventLogSchemaMismatchError, EventLogSetupErrorfrom pipelex.tracing.trace_events import AnyTraceEvent, TraceEvent, json_invalid_message@@ -43,9 +54,12 @@ class DynamoDBEventLog(EventLogProtocol):"""Event log backed by AWS DynamoDB.Write path: PutItem per event, synchronous. DynamoDB's PK+SK uniqueness- provides natural deduplication for Temporal replay re-emissions.- Read path: Query on PK, returns events sorted by SK which encodes- (workflow_id, sequence) — matching the NDJSON ordering contract.+ provides natural deduplication for Temporal replay re-emissions, since a+ replayed event has the same chain, writer and sequence, and so the same SK.+ Read path: Query on PK, then the events ordered as every event log returns+ them (`deduplicate_and_order_events`): the SK's chain digest orders the+ rows by no chain. The rule's deduplication finds nothing to drop here,+ the key having already kept one row per event.Retention: every item carries a `ttl` in epoch seconds, `retention_days`past its write, which is the attribute the events table expires items on,so a run's events are deleted by DynamoDB instead of being kept forever.@@ -110,8 +124,14 @@ class DynamoDBEventLog(EventLogProtocol):return f"PIPELINE_RUN#{pipeline_run_id}"@staticmethod- def _make_sk(workflow_id: str, *, writer_id: str, sequence: int) -> str:- return f"EVENT#{workflow_id}#{writer_id}#{sequence:010d}"+ def _make_sk(*, chain_key: str, writer_id: str, sequence: int) -> str:+ """The sort key of one event: its chain, its writer and its sequence, which tell it apart from every other event+ of its run.++ The chain is written as the SHA-256 hex digest of its key, so the sort key's length does not grow with the+ lineage, and the same chain always gives the same digest, so a replayed write still overwrites its own row.+ """+ return f"EVENT#{execution_chain_digest(chain_key=chain_key)}#{writer_id}#{sequence:010d}"def _key_condition(self, pipeline_run_id: str) -> Any:"""Build a KeyConditionExpression for querying by pipeline_run_id."""@@ -125,9 +145,10 @@ class DynamoDBEventLog(EventLogProtocol):ttl = int(time.time()) + self._retention_secondsitem: dict[str, Any] = {"PK": self._make_pk(event.pipeline_run_id),- "SK": self._make_sk(event.workflow_id, writer_id=event.writer_id, sequence=event.sequence),+ "SK": self._make_sk(chain_key=event.chain_key, writer_id=event.writer_id, sequence=event.sequence),"pipeline_run_id": event.pipeline_run_id,"workflow_id": event.workflow_id,+ "attempt": event.attempt,"writer_id": event.writer_id,"sequence": event.sequence,"payload": event.model_dump_json(),@@ -139,6 +160,12 @@ class DynamoDBEventLog(EventLogProtocol):def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:"""Read all events for a pipeline run from DynamoDB.+ The events are ordered by (chain, sequence, writer_id), as every event+ log returns them, whatever order the sort keys give the rows. A row in+ the shape an earlier version wrote, without `attempt`, `lineage` or+ `path`, is refused with ``EventLogSchemaMismatchError``, never read as+ the direct mode's first attempt.+Store-level failures (throttling, auth, network) are translated into``EventLogReadError`` so best-effort callers (graph assembly) can degradegracefully without catching raw botocore exceptions. Individual@@ -208,7 +235,7 @@ class DynamoDBEventLog(EventLogProtocol):)raise EventLogSchemaMismatchError(msg)- return events+ return deduplicate_and_order_events(events=events)@overridedef close(self) -> None:diff --git a/pipelex/tracing/event_log_protocol.py b/pipelex/tracing/event_log_protocol.pyindex 1006f3cfb..da3a80895 100644--- a/pipelex/tracing/event_log_protocol.py+++ b/pipelex/tracing/event_log_protocol.py@@ -1,15 +1,39 @@"""Protocol for trace event storage backends.-Defines the interface that all event log implementations must satisfy.-The NDJSON backend is the local implementation; cloud backends (e.g. DynamoDB)-will implement the same protocol later.+Defines the interface that all event log implementations must satisfy, and the one rule by which every one of them+returns the events of a run (`deduplicate_and_order_events`): the in-memory, NDJSON, DynamoDB and buffering logs."""+from collections.abc import Iterablefrom typing import Protocolfrom pipelex.tracing.trace_events import TraceEvent+def deduplicate_and_order_events(*, events: Iterable[TraceEvent]) -> list[TraceEvent]:+ """The events of a run as an event log returns them, whatever order its store holds them in.++ One event is kept per `TraceEvent.store_key` (chain, writer_id, event type, sequence), the first one met: a replay+ writes an event again under the same key, which a log that appends, such as the NDJSON and in-memory ones, holds+ twice, while a log keyed on the event, such as the DynamoDB one, overwrites its row and so holds it once. An event+ stored by a version that wrote no `attempt`, `lineage` or `path` never reaches this rule: the log's read refuses it+ with `EventLogSchemaMismatchError`. The events are ordered by (chain, sequence, writer_id):+ a dispatching attempt's events before those of the workflows it dispatched, whose chains continue its own, and,+ within one attempt, by sequence before writer, so that two writers emitting into the same attempt are not reordered+ by the lexical order of their ids.++ Args:+ events: The events read from the store, in any order, repeats included.++ Returns:+ The distinct events, ordered.+ """+ kept: dict[tuple[str, str, str, int], TraceEvent] = {}+ for event in events:+ kept.setdefault(event.store_key, event)+ return sorted(kept.values(), key=lambda event: (event.chain_key, event.sequence, event.writer_id))++class EventLogProtocol(Protocol):"""Storage backend for trace events.@@ -20,8 +44,13 @@ class EventLogProtocol(Protocol):Each event log instance carries an immutable `writer_id` that emittersstamp on every TraceEvent they construct via this log. Two writerssharing the same (pipeline_run_id, workflow_id) partition must use- distinct writer_ids so the read-side dedup key- (workflow_id, writer_id, type, sequence) keeps their events apart.+ distinct writer_ids so the read-side dedup key, `TraceEvent.store_key`+ (chain, writer_id, type, sequence), keeps their events apart.++ The chain, an event's lineage followed by its own workflow and attempt, is+ in the key so that a workflow's new attempt, which restarts its sequence,+ never overwrites the events of its earlier one, and neither does a workflow+ that a new attempt dispatched again under the same name."""@property@@ -39,8 +68,8 @@ class EventLogProtocol(Protocol):All emitters sharing the same event log must use this method toobtain sequence numbers. Combined with writer_id, this guarantees- events never collide on the (workflow_id, writer_id, sequence) key- regardless of storage backend.+ events of one attempt never collide on the (chain, writer_id, sequence)+ key regardless of storage backend.Implementations MUST be safe to call from multiple threadsconcurrently — the runner-side activity event log is cached@@ -63,8 +92,12 @@ class EventLogProtocol(Protocol):def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:"""Read all events for a pipeline run.- Returns events deduplicated by (workflow_id, event_type, sequence) and sorted- by (workflow_id, sequence) for deterministic ordering.+ Returns events deduplicated by `TraceEvent.store_key` (chain, writer_id,+ event type, sequence) and sorted by (chain, sequence, writer_id) for+ deterministic ordering: a dispatching attempt's events before those of+ the workflows it dispatched, whose chains continue its own. Every+ implementation applies `deduplicate_and_order_events` to what its store+ holds, so none returns its store's own order.Returns an empty list if no events exist for the given run."""...diff --git a/pipelex/tracing/graphspec_assembler.py b/pipelex/tracing/graphspec_assembler.pyindex 12e14add9..4e688bee5 100644--- a/pipelex/tracing/graphspec_assembler.py+++ b/pipelex/tracing/graphspec_assembler.py@@ -5,6 +5,12 @@ GraphTracer.teardown() but working from serialized events instead ofin-memory state. Used after Temporal execution to assemble cross-workergraphs from NDJSON event files. The graph's provenance mode is the run modeits events state, never one the caller passes.++A run's record holds the events of every attempt of every workflow, and the+graph is built from the current ones alone: a higher attempt of a workflow+supersedes the earlier one and everything it dispatched+(`pipelex.system.execution_chain`), so a retried invocation is one node, the+one its latest attempt ran, and never a failed node beside a succeeded one."""from collections.abc import Sequence@@ -32,6 +38,7 @@ from pipelex.graph.graphspec import (output_digest_is_optional,)from pipelex.graph.selected_outcome import derive_selected_outcome_edges+from pipelex.system.execution_chain import current_chain_keysfrom pipelex.tracing.exceptions import TraceEventModeErrorfrom pipelex.tracing.trace_events import (UNATTRIBUTED_NODE_ID,@@ -63,6 +70,7 @@ class _AssemblerNodeData:def __init__(self,node_id: str,+ path: str | None,pipe_code: str,pipe_type: str,node_kind: NodeKind,@@ -73,6 +81,7 @@ class _AssemblerNodeData:domain_code: str | None = None,) -> None:self.node_id = node_id+ self.path = pathself.pipe_code = pipe_codeself.pipe_type = pipe_typeself.node_kind = node_kind@@ -106,6 +115,7 @@ class _AssemblerNodeData:return NodeSpec(node_id=self.node_id,+ path=self.path,kind=self.node_kind,pipe_code=self.pipe_code,pipe_type=self.pipe_type,@@ -137,8 +147,14 @@ class GraphSpecAssembler:Usage attribution deliberately waits for pass 2: a ``UsageReportEvent`` can be readbefore the ``PipeStartEvent`` of the node it names (they can come from different- workers, and the stream is ordered by (workflow_id, sequence)), so "is this node+ workers, and the stream is ordered by (chain, sequence)), so "is this nodereal?" is only answerable once every event has been seen.++ Only current events build the graph, and the earliest of them dates it: an event whose+ chain holds a superseded attempt is dropped, since a new attempt of a workflow replaces+ what its earlier attempt, and every workflow that attempt dispatched, did. Its usage was+ spent all the same, so a superseded usage report still counts in the run's total, as+ unattributed usage: the node it named belongs to an attempt the graph does not show."""@staticmethod@@ -153,7 +169,8 @@ class GraphSpecAssembler:The GraphSpec's provenance mode is the run mode every event states.Args:- events: Flat list of trace events, pre-sorted by (workflow_id, sequence).+ events: Flat list of trace events, pre-sorted by (chain, sequence), the events of+ superseded attempts included.graph_id: The graph identifier for the assembled GraphSpec.pipeline_ref: Optional pipeline reference metadata.@@ -164,7 +181,12 @@ class GraphSpecAssembler:TraceEventModeError: When the events mix modes, or there are none, and so state no single run mode."""mode = _graphspec_mode_of_events(events=events, graph_id=graph_id)- assembler = _AssemblerState(graph_id=graph_id, pipeline_ref=pipeline_ref or PipelineRef(), mode=mode)+ assembler = _AssemblerState(+ graph_id=graph_id,+ pipeline_ref=pipeline_ref or PipelineRef(),+ mode=mode,+ current_chain_keys=current_chain_keys(chains=[event.chain for event in events]),+ )assembler.pass_one(events)assembler.pass_two()return assembler.build_graph_spec()@@ -190,10 +212,12 @@ def _graphspec_mode_of_events(*, events: Sequence[TraceEvent], graph_id: str) ->class _AssemblerState:"""Mutable state for a single assembler invocation."""- def __init__(self, graph_id: str, pipeline_ref: PipelineRef, mode: GraphSpecMode) -> None:+ def __init__(self, graph_id: str, pipeline_ref: PipelineRef, mode: GraphSpecMode, current_chain_keys: set[str]) -> None:self._graph_id = graph_idself._pipeline_ref = pipeline_refself._mode = mode+ # The chains no attempt of the run supersedes: only their events build the graph+ self._current_chain_keys = current_chain_keysself._earliest_timestamp: datetime | None = None# Pass 1 accumulation@@ -226,6 +250,14 @@ class _AssemblerState:def pass_one(self, events: Sequence[TraceEvent]) -> None:"""Pass 1: Build nodes, producer map, and collect metadata."""for event in events:+ if event.chain_key not in self._current_chain_keys:+ # A superseded attempt's event: what it recorded was done again, or undone, by a later attempt, so it+ # neither builds the graph nor dates it. Its usage was spent all the same, so it counts in the run's+ # total, unattributed.+ if isinstance(event, UsageReportEvent):+ self._usage_by_reported_node.setdefault(UNATTRIBUTED_NODE_ID, []).append(event.tokens_usage)+ continue+self._track_earliest_timestamp(event)if isinstance(event, PipeStartEvent):@@ -303,6 +335,7 @@ class _AssemblerState:def _handle_pipe_start(self, event: PipeStartEvent) -> None:node_data = _AssemblerNodeData(node_id=event.node_id,+ path=event.path,pipe_code=event.pipe_code,pipe_type=event.pipe_type,node_kind=event.node_kind,diff --git a/pipelex/tracing/in_memory_event_log.py b/pipelex/tracing/in_memory_event_log.pyindex 2fcf3bfa9..3bdfd6548 100644--- a/pipelex/tracing/in_memory_event_log.py+++ b/pipelex/tracing/in_memory_event_log.py@@ -4,7 +4,7 @@ import threadingfrom typing_extensions import override-from pipelex.tracing.event_log_protocol import EventLogProtocol+from pipelex.tracing.event_log_protocol import EventLogProtocol, deduplicate_and_order_eventsfrom pipelex.tracing.trace_events import TraceEvent@@ -47,23 +47,13 @@ class InMemoryEventLog(EventLogProtocol):def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:"""Return events for a pipeline run, deduplicated and sorted.- Deduplicates by (workflow_id, writer_id, type, sequence), keeping- the first occurrence. Sorts by (workflow_id, sequence, writer_id) —- sequence is primary so two writers emitting into the same workflow- partition do not get reordered by writer-id lexicographic sort.+ Deduplicates by `TraceEvent.store_key` (chain, writer_id, type,+ sequence), keeping the first occurrence. Sorts by (chain, sequence,+ writer_id) — sequence before writer so two writers emitting into the+ same attempt's partition do not get reordered by writer-id+ lexicographic sort."""- filtered = [evt for evt in self._events if evt.pipeline_run_id == pipeline_run_id]-- seen: set[tuple[str, str, str, int]] = set()- deduped: list[TraceEvent] = []- for event in filtered:- dedup_key = (event.workflow_id, event.writer_id, type(event).__name__, event.sequence)- if dedup_key not in seen:- seen.add(dedup_key)- deduped.append(event)-- deduped.sort(key=lambda evt: (evt.workflow_id, evt.sequence, evt.writer_id))- return deduped+ return deduplicate_and_order_events(events=[evt for evt in self._events if evt.pipeline_run_id == pipeline_run_id])@overridedef close(self) -> None:diff --git a/pipelex/tracing/ndjson_event_log.py b/pipelex/tracing/ndjson_event_log.pyindex 77132792e..1318e8106 100644--- a/pipelex/tracing/ndjson_event_log.py+++ b/pipelex/tracing/ndjson_event_log.py@@ -1,7 +1,12 @@"""NDJSON file-based event log implementation.-One JSON event per line, one file per workflow, organized by pipeline run:- {traces_dir}/{pipeline_run_id}/wf_{workflow_id}.ndjson+One JSON event per line, one file per attempt of each execution and per writer, organized by pipeline run:+ {traces_dir}/{pipeline_run_id}/wf_{workflow_id}__c_{chain_digest}.ndjson+ {traces_dir}/{pipeline_run_id}/wf_{workflow_id}__c_{chain_digest}__w_{writer_id}.ndjson++The chain digest is the SHA-256 of the events' chain key, their lineage followed by their own workflow and attempt+(`TraceEvent.chain_key`), as the DynamoDB backend's sort key writes it. The reader reads every file of the run's+directory."""import shutil@@ -13,10 +18,11 @@ from pydantic import TypeAdapter, ValidationErrorfrom typing_extensions import overridefrom pipelex import log+from pipelex.system.execution_chain import execution_chain_digestfrom pipelex.system.telemetry.otel_constants import OTelLogAttrfrom pipelex.tools.log.error_fields import error_fieldsfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error-from pipelex.tracing.event_log_protocol import EventLogProtocol+from pipelex.tracing.event_log_protocol import EventLogProtocol, deduplicate_and_order_eventsfrom pipelex.tracing.exceptions import EventLogSchemaMismatchErrorfrom pipelex.tracing.trace_events import AnyTraceEvent, TraceEvent, json_invalid_message@@ -28,14 +34,19 @@ class NdjsonEventLog(EventLogProtocol):Write path: appends one JSON line per event, flushed immediately.Read path: globs all .ndjson files in the run directory, parses,- deduplicates by (workflow_id, writer_id, type, sequence), and sorts- by (workflow_id, sequence, writer_id) — sequence primary so a+ deduplicates by `TraceEvent.store_key` (chain, writer_id, type, sequence),+ and sorts by (chain, sequence, writer_id) — sequence before writer so arunner-side writer's events do not sort before earlier router events.- Multi-writer file naming: events from writer_id="primary" land in the- legacy file `wf_{workflow_id}.ndjson`; events from any other writer- land in `wf_{workflow_id}__w_{writer_id}.ndjson`. The file-handle cache- key is `(pipeline_run_id, workflow_id, writer_id)` so two writers+ File naming: each attempt of each execution writes its own file, named by+ its chain, so two attempts of one workflow, or a workflow that a new+ attempt dispatched again under the same name while its predecessor still+ drains, never append to one file through two handles, whose writes could+ interleave partial lines and leave the run's whole log unreadable. Events+ from writer_id="primary" land in `wf_{workflow_id}__c_{chain_digest}.ndjson`;+ events from any other writer land in+ `wf_{workflow_id}__c_{chain_digest}__w_{writer_id}.ndjson`. The file-handle+ cache key is `(pipeline_run_id, chain_key, writer_id)` so two writersemitting concurrently never share a stale handle.For multi-process / multi-host deployments, traces_dir must be a@@ -64,7 +75,7 @@ class NdjsonEventLog(EventLogProtocol):threads sharing this backend (via ``get_or_create_activity_event_log``)cannot read the same value before either increments — duplicatesequence numbers would collide on the- ``(workflow_id, writer_id, type, sequence)`` dedup key and silently+ ``(chain, writer_id, type, sequence)`` dedup key and silentlydrop one event."""with self._sequence_lock:@@ -73,21 +84,25 @@ class NdjsonEventLog(EventLogProtocol):return seq@staticmethod- def _file_name_for(workflow_id: str, *, writer_id: str) -> str:- """File name for a (workflow_id, writer_id) pair.+ def _file_name_for(*, workflow_id: str, chain_key: str, writer_id: str) -> str:+ """File name for the events one writer writes for one attempt of one execution.++ The chain is written as its digest (`execution_chain_digest`), as the+ DynamoDB sort key writes it, so the name does not grow with the+ lineage, and the same chain always names the same file, so a replayed+ attempt appends to its own. The workflow id leads the name so that a+ person reading the run directory sees whose file it is.Workflow ids use ``_`` as their separator and so should not contain``/`` (e.g. ``ut-{uuid}_step_two-9a262f1f``). The ``replace`` below is adefensive guard that keeps the derived file name flat inside the rundirectory even if a ``/`` ever slips into an id.-- The legacy single-writer name is preserved when writer_id="primary"- so existing files continue to be written and read correctly."""safe_id = workflow_id.replace("/", "__")+ chain_digest = execution_chain_digest(chain_key=chain_key)if writer_id == "primary":- return f"wf_{safe_id}.ndjson"- return f"wf_{safe_id}__w_{writer_id}.ndjson"+ return f"wf_{safe_id}__c_{chain_digest}.ndjson"+ return f"wf_{safe_id}__c_{chain_digest}__w_{writer_id}.ndjson"# ------------------------------------------------------------------# Write@@ -98,10 +113,10 @@ class NdjsonEventLog(EventLogProtocol):"""Append event as a JSON line, flush immediately.Creates the run directory on first write. Caches file handles- keyed by (pipeline_run_id, workflow_id, writer_id) to avoid+ keyed by (pipeline_run_id, chain_key, writer_id) to avoidrepeated open() calls for high-frequency events."""- cache_key = (event.pipeline_run_id, event.workflow_id, event.writer_id)+ cache_key = (event.pipeline_run_id, event.chain_key, event.writer_id)handle = self._file_handles.get(cache_key)if handle is None:@@ -110,7 +125,8 @@ class NdjsonEventLog(EventLogProtocol):if handle is None:run_dir = self._traces_dir / event.pipeline_run_idrun_dir.mkdir(parents=True, exist_ok=True)- file_path = run_dir / self._file_name_for(event.workflow_id, writer_id=event.writer_id)+ file_name = self._file_name_for(workflow_id=event.workflow_id, chain_key=event.chain_key, writer_id=event.writer_id)+ file_path = run_dir / file_namehandle = open(file_path, "a", encoding="utf-8") # ruff: ignore[open-file-with-context-handler]self._file_handles[cache_key] = handle@@ -125,8 +141,10 @@ class NdjsonEventLog(EventLogProtocol):def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:"""Read all events for a pipeline run from NDJSON files.- Deduplicates by (workflow_id, sequence) to handle Temporal replay- re-emission. Sorts by (workflow_id, sequence) for deterministic ordering.+ Deduplicates by `TraceEvent.store_key` (chain, writer_id, type,+ sequence) to handle Temporal replay re-emission, while keeping the+ events of every attempt. Sorts by (chain, sequence, writer_id) for+ deterministic ordering.Corrupt lines (truncated JSON from crash-mid-write) are skipped witha warning log.@@ -141,7 +159,6 @@ class NdjsonEventLog(EventLogProtocol):ndjson_files = sorted(run_dir.glob("*.ndjson"))- seen: set[tuple[str, str, str, int]] = set()events: list[TraceEvent] = []refused_count = 0first_refusal: str | None = None@@ -178,10 +195,7 @@ class NdjsonEventLog(EventLogProtocol):first_refusal = f"{ndjson_path}:{line_number} — {format_pydantic_validation_error(validation_error)}"continue- dedup_key = (event.workflow_id, event.writer_id, type(event).__name__, event.sequence)- if dedup_key not in seen:- seen.add(dedup_key)- events.append(event)+ events.append(event)if refused_count:msg = (@@ -191,12 +205,12 @@ class NdjsonEventLog(EventLogProtocol):)raise EventLogSchemaMismatchError(msg)- # TODO: causal ordering — sorting by (workflow_id, sequence) groups by lexicographic- # workflow ID, not execution order. In parent/child workflow topologies this can cause- # incorrect producer map overwrites in GraphSpecAssembler. Consider timestamp-based- # or topology-aware ordering.- events.sort(key=lambda evt: (evt.workflow_id, evt.sequence, evt.writer_id))- return events+ # TODO: causal ordering — sorting by (chain, sequence) puts a dispatching attempt's events+ # before those of the workflows it dispatched, whose chains continue its own, but orders+ # sibling workflows by their names, not by execution order. In parent/child workflow+ # topologies this can cause incorrect producer map overwrites in GraphSpecAssembler.+ # Consider timestamp-based or topology-aware ordering.+ return deduplicate_and_order_events(events=events)# ------------------------------------------------------------------# Cleanupdiff --git a/pipelex/tracing/trace_events.py b/pipelex/tracing/trace_events.pyindex 1135d80de..cf447a3e4 100644--- a/pipelex/tracing/trace_events.py+++ b/pipelex/tracing/trace_events.py@@ -5,15 +5,19 @@ GraphTracer's in-memory accumulation, but as immutable records suitablefor NDJSON file storage and cross-worker graph assembly."""+from collections.abc import Mappingfrom datetime import datetimefrom enum import StrEnum-from typing import Annotated, Any, Literal+from functools import cached_property+from typing import Annotated, Any, Literal, Self-from pydantic import BaseModel, Field, ValidationError+from pydantic import BaseModel, ConfigDict, Field, ValidationError+from typing_extensions import overridefrom pipelex.graph.condition_output_merge import ConditionOutputMergefrom pipelex.graph.graphspec import EdgeKind, ErrorSpec, IOSpec, NodeKindfrom pipelex.reporting.reporting_types import AnyTokensUsage+from pipelex.system.execution_chain import ExecutionAttempt, execution_chain_keyfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of@@ -40,29 +44,84 @@ class TraceEventKind(StrEnum):class TraceEvent(BaseModel):"""Base model for all trace events.+ The fields of the hosted progress contract (`conformance/specs/pipelex-platform-api.md`, "How the runtime+ carries them") are read off these: `execution` is `workflow_id`, `attempt` and `lineage` are their own, `at` is+ `timestamp` and `sequence` is `sequence`.+Shared fields:- pipeline_run_id: identifies the pipeline execution- - workflow_id: Temporal workflow ID or "direct" for in-process mode+ - workflow_id: the id of the workflow whose code emitted the event, the same in each of its attempts, or+ "direct" in the runtime's direct mode. It names the event's execution.+ - attempt: the attempt of that workflow which emitted the event, counting from 1; 1 in direct mode.+ - lineage: the attempts that workflow was dispatched under, from the run's own workflow down to the one that+ dispatched it; empty for the run's own workflow. An event's chain is its lineage followed by its own+ `{execution, attempt}`, and a higher attempt of an execution supersedes the earlier one and everything it+ dispatched (`pipelex.system.execution_chain`).+ - path: the invocation path of the pipe run the event is about (`pipelex.system.invocation_path`): the batch's for+ a fan-out, the condition's for a resolution, the target's for an edge, the pipe whose work an inference+ served for a usage report. None for an event about no invocation, such as a binding step's.- writer_id: identifies the emitting event-log instance, so two writerssharing the same (pipeline_run_id, workflow_id) partition can emitindependent sequence streams without colliding. Defaults to "primary",- which preserves the legacy single-writer NDJSON file naming and DDB- sort-key shape; non-default values come from the per-process activity- event log used by separate-process workers.+ which preserves the legacy single-writer NDJSON file naming; non-default values+ come from the per-process activity event log used by separate-process workers.+ It names no execution: one writer may carry the events of several at once.- event_kind: discriminator for the event subclass (defined on each subclass as Literal)- mode: whether the run that emitted the event is live or a dry run. Every event of one runcarries the same mode, so a reader tells dry events from live ones by this field alone.- - timestamp: UTC, for display/debugging only+ - timestamp: UTC, when the event happened: a start's start, an end's end, a decision's moment. It is recorded+ once, so every write of one event carries the same value; ordering and deduplication never read it.- sequence: per-writer monotonic counter, for ordering and deduplication++ A store keeps apart every two events whose chain, writer or sequence differ (`store_key`): keyed on the workflow,+ the writer and the sequence alone, a workflow's new attempt would overwrite the events of its earlier one.++ An event is frozen, since it records what happened: its chain, its chain key and its store key are computed from its+ fields once, the first time they are read, and kept, as every read and every assembly of a run reads them for each+ of its events."""+ model_config = ConfigDict(frozen=True)+pipeline_run_id: str- workflow_id: str+ workflow_id: str = Field(min_length=1)+ attempt: int = Field(ge=1)+ lineage: list[ExecutionAttempt]+ path: str | Nonewriter_id: str = "primary"mode: PipeRunModetimestamp: datetimesequence: int+ @cached_property+ def chain(self) -> list[ExecutionAttempt]:+ """The attempts from the run's own workflow down to the one that emitted this event."""+ return [*self.lineage, ExecutionAttempt(execution=self.workflow_id, attempt=self.attempt)]++ @cached_property+ def chain_key(self) -> str:+ """One string naming this event's chain, distinct for every two chains that differ."""+ return execution_chain_key(chain=self.chain)++ @cached_property+ def store_key(self) -> tuple[str, str, str, int]:+ """What tells this event apart from every other event of its run in a store: its chain, its writer, its kind+ and its sequence. Two writes with the same key are one event, written again by a replay.+ """+ return (self.chain_key, self.writer_id, type(self).__name__, self.sequence)++ @override+ def model_copy(self, *, update: Mapping[str, Any] | None = None, deep: bool = False) -> Self:+ """A copy of this event, which computes its keys again: an update can change the fields they are read from."""+ copied = super().model_copy(update=update, deep=deep)+ for cached_name in _CACHED_EVENT_KEYS:+ vars(copied).pop(cached_name, None)+ return copied+++# The names under which a trace event keeps the keys it computed from its fields.+_CACHED_EVENT_KEYS: tuple[str, ...] = ("chain", "chain_key", "store_key")+# ---------------------------------------------------------------------------# Event subclasses@@ -76,6 +135,8 @@ class PipeStartEvent(TraceEvent):node_id: strparent_node_id: str | None = Nonepipe_code: str+ pipe_ref: str | None+ """The `domain.code` reference of the pipe; None on a binding step's node, which runs no pipe."""pipe_type: strnode_kind: NodeKinddescription: str | None = None@@ -95,6 +156,9 @@ class PipeEndSuccessEvent(TraceEvent):event_kind: Literal[TraceEventKind.PIPE_END_SUCCESS] = TraceEventKind.PIPE_END_SUCCESSnode_id: str+ pipe_code: str+ pipe_ref: str | None+ pipe_type: strended_at: datetimeoutput_spec: IOSpec | None = Nonemetrics: dict[str, float] = Field(default_factory=dict)@@ -102,10 +166,17 @@ class PipeEndSuccessEvent(TraceEvent):class PipeEndErrorEvent(TraceEvent):- """Emitted when a pipe fails."""+ """Emitted when a pipe fails.++ Its `error` gives the progress contract's `failure`: `message`, and `caller_facing`, true when that message was+ written for the caller.+ """event_kind: Literal[TraceEventKind.PIPE_END_ERROR] = TraceEventKind.PIPE_END_ERRORnode_id: str+ pipe_code: str+ pipe_ref: str | None+ pipe_type: strended_at: datetimeerror: ErrorSpec@@ -119,6 +190,9 @@ class PipeEndSkippedEvent(TraceEvent):event_kind: Literal[TraceEventKind.PIPE_END_SKIPPED] = TraceEventKind.PIPE_END_SKIPPEDnode_id: str+ pipe_code: str+ pipe_ref: str | None+ pipe_type: strended_at: datetimeskip_reason: stroutput_spec: IOSpec | None = Nonediff --git a/subject_grants.toml b/subject_grants.tomlindex d1d6a0413..9a0f96f69 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -6896,10 +6896,6 @@ rationale = "Derivation: encodes the run id into a key condition; single operandparam = "pipeline_run_id"rationale = "Derivation: encodes the run id into the partition key; single operand."-["pipelex/tracing/dynamodb_event_log.py::DynamoDBEventLog._make_sk"]-param = "workflow_id"-rationale = "Derivation: encodes the workflow id into the sort key; writer/sequence stay keyword."-["pipelex/tracing/event_log_factory.py::make_event_log"]param = "tracing_config"rationale = "Factory from source: builds the event log from the tracing config; writer_id stays keyword."@@ -6980,10 +6976,6 @@ rationale = "Verb-object: reads the event to track its timestamp; single operandparam = "events"rationale = "First pass over the event sequence; single operand."-["pipelex/tracing/ndjson_event_log.py::NdjsonEventLog._file_name_for"]-param = "workflow_id"-rationale = "Derivation: encodes the workflow id into a file name; writer_id stays keyword."-["pipelex/tracing/usage_aggregator.py::UsageAggregator.aggregate"]param = "events"rationale = "Verb-object: aggregates the events; single operand."diff --git a/tests/conftest.py b/tests/conftest.pyindex d8bf7812d..9be89c38e 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -1,3 +1,4 @@+import loggingfrom collections.abc import Callable, Generatorfrom pathlib import Pathfrom typing import TYPE_CHECKING@@ -5,8 +6,10 @@ from typing import TYPE_CHECKINGimport pytestimport shortuuidfrom pytest_mock import MockerFixture+from typing_extensions import overridefrom pipelex import log+from pipelex.graph.graph_tracer import INVOCATION_PATH_FAULT_MESSAGEfrom pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_libraryfrom pipelex.pipelex import Pipelexfrom pipelex.system.environment import PIPELEX_HOME_ENV_KEY@@ -71,6 +74,39 @@ def isolate_usage_stats() -> Generator[None, None, None]:yield+class _InvocationPathFaultCollector(logging.Handler):+ """Collects the errors a tracer logs when it traces an invocation without its path."""++ def __init__(self) -> None:+ super().__init__(level=logging.ERROR)+ self.records: list[logging.LogRecord] = []++ @override+ def emit(self, record: logging.LogRecord) -> None:+ # A configured caller prefix may precede the message, so the message is matched at the end of the line.+ if record.getMessage().endswith(INVOCATION_PATH_FAULT_MESSAGE):+ self.records.append(record)++[email protected](autouse=True)+def invocation_path_faults() -> Generator[list[logging.LogRecord], None, None]:+ """Fail any test during which a tracer traced an invocation without its invocation path.++ Tracing never fails a run, so a controller that hands a pipe no place only makes the tracer log the fault+ (`pipelex/graph/graph_tracer.py`): this is where the invariant that every invocation gets a path is enforced, over+ every test that runs a pipe. A test that provokes the fault on purpose requests this fixture by name, checks the+ records it holds, and empties the list.+ """+ collector = _InvocationPathFaultCollector()+ root_logger = logging.getLogger()+ root_logger.addHandler(collector)+ yield collector.records+ root_logger.removeHandler(collector)+ if collector.records:+ fields = [{"pipe_ref": getattr(record, "pipe_ref", None), "node_id": getattr(record, "node_id", None)} for record in collector.records]+ pytest.fail(f"An invocation was traced without its invocation path, which every invocation must have: {fields}")++def _get_test_integration_mode() -> IntegrationMode:"""Return the appropriate integration mode for tests.diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txtindex 700782a19..5a41aea20 100644--- a/tests/data/errors/error_identity.txt+++ b/tests/data/errors/error_identity.txt@@ -141,6 +141,7 @@ InputShapingError | Input shaping | https://docs.pipelex.com/latest/errors/inputInputStuffSpecNotFoundError | Input stuff spec not found | https://docs.pipelex.com/latest/errors/input-stuff-spec-not-found-error/InputStuffSpecsError | Input stuff specs | https://docs.pipelex.com/latest/errors/input-stuff-specs-error/InputStuffSpecsFactoryError | Input stuff specs factory | https://docs.pipelex.com/latest/errors/input-stuff-specs-factory-error/+InvocationPathError | Invocation path not derivable | https://docs.pipelex.com/latest/errors/invocation-path-error/Jinja2ContextError | Jinja 2 context | https://docs.pipelex.com/latest/errors/jinja2-context-error/Jinja2DetectVariablesError | Jinja 2 detect variables | https://docs.pipelex.com/latest/errors/jinja2-detect-variables-error/Jinja2StuffError | Jinja 2 stuff | https://docs.pipelex.com/latest/errors/jinja2-stuff-error/diff --git a/tests/helpers/usage_stats_run_events.py b/tests/helpers/usage_stats_run_events.pyindex a545d53d3..fc110eafd 100644--- a/tests/helpers/usage_stats_run_events.py+++ b/tests/helpers/usage_stats_run_events.py@@ -47,6 +47,9 @@ def llm_usage_event(},)return UsageReportEvent(+ attempt=1,+ lineage=[],+ path=None,pipeline_run_id=USAGE_STATS_RUN_ID,workflow_id="direct",timestamp=datetime.now(UTC),@@ -59,6 +62,10 @@ def llm_usage_event(def pipe_start_event(*, pipe_type: str, sequence: int) -> PipeStartEvent:return PipeStartEvent(+ attempt=1,+ lineage=[],+ path=None,+ pipe_ref=None,pipeline_run_id=USAGE_STATS_RUN_ID,workflow_id="direct",timestamp=datetime.now(UTC),diff --git a/tests/integration/pipelex/kernel/test_usage_parity.py b/tests/integration/pipelex/kernel/test_usage_parity.pyindex 3b728d33c..e9b32e986 100644--- a/tests/integration/pipelex/kernel/test_usage_parity.py+++ b/tests/integration/pipelex/kernel/test_usage_parity.py@@ -54,6 +54,7 @@ from pipelex.reporting.reporting_types import AnyTokensUsagefrom pipelex.reporting.usage_records import TokensUsageRecord, make_tokens_usage_recordfrom pipelex.runtime_hub import get_report_delegatefrom pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContextfrom pipelex.tracing.in_memory_event_log import InMemoryEventLog@@ -119,11 +120,11 @@ class TestKernelUsageParity:emit_graph_events=False,emit_usage_events=True,run_mode=PipeRunMode.DRY,+ chain=make_direct_chain(),)get_report_delegate().set_event_log(context_key=trace_context.lookup_key,event_log=event_log,- workflow_id="direct",pipeline_run_id=trace_context.graph_id,)try:diff --git a/tests/integration/pipelex/pipeline/invocation_paths/conftest.py b/tests/integration/pipelex/pipeline/invocation_paths/conftest.pynew file mode 100644index 000000000..d3d441330--- /dev/null+++ b/tests/integration/pipelex/pipeline/invocation_paths/conftest.py@@ -0,0 +1,15 @@+from collections.abc import Iterator++import pytest++from pipelex.system.registries.func_registry import func_registry+from tests.integration.pipelex.pipeline.invocation_paths.invocation_paths_helpers import invocation_paths_condense_filing++[email protected](autouse=True)+def register_condense_filing() -> Iterator[None]:+ """Register the function the worked example's `condense_filing` runs, for the test's duration."""+ func_registry.register_function(invocation_paths_condense_filing)+ yield+ if func_registry.has_function(invocation_paths_condense_filing.__name__):+ func_registry.unregister_function_by_name(invocation_paths_condense_filing.__name__)diff --git a/tests/integration/pipelex/pipeline/invocation_paths/invocation_paths_helpers.py b/tests/integration/pipelex/pipeline/invocation_paths/invocation_paths_helpers.pynew file mode 100644index 000000000..bc25b8be2--- /dev/null+++ b/tests/integration/pipelex/pipeline/invocation_paths/invocation_paths_helpers.py@@ -0,0 +1,103 @@+"""A router running each batch item as an execution of its own, as the engines Pipelex runs on do, and what it needs."""++from contextvars import ContextVar+from typing import TYPE_CHECKING++import kajson+from typing_extensions import override++from pipelex.base_exceptions import PipelexError+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.stuffs.text_content import TextContent+from pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_router import PipeRouter+from pipelex.runtime_hub import get_event_log_override+from pipelex.system.job_metadata import JobMetadata++if TYPE_CHECKING:+ from pipelex.system.trace_context import TraceContext++# Set while an attempt the engine is about to lose runs: the attempt fails at the step that condenses its filing.+_IS_LOST_ATTEMPT: ContextVar[bool] = ContextVar("invocation_paths_is_lost_attempt", default=False)+++def invocation_paths_condense_filing(working_memory: WorkingMemory) -> TextContent:+ reading = working_memory.get_stuff_as_str(name="reading")+ if _IS_LOST_ATTEMPT.get():+ msg = f"The worker condensing '{reading}' was lost"+ raise RuntimeError(msg)+ return TextContent(text=f"Condensed: {reading}")+++class ItemExecutionRouter(PipeRouter):+ """Runs each batch item as an execution of its own, which a distributed engine may run again.++ The item's job crosses a serialization boundary first, as a child workflow's input does, and the item's execution+ opens its own tracer on the context it received, as the hosted run workflow does. The item named by+ `retried_item_segment` runs twice: its first attempt fails, and the engine runs a second, whose output the batch+ receives.++ Attributes:+ retried_item_segment: The segment of the item whose execution runs a second attempt.+ received_contexts: By item segment, the trace context each item's execution received.+ """++ def __init__(self, *, retried_item_segment: str) -> None:+ super().__init__()+ self.retried_item_segment = retried_item_segment+ self.received_contexts: dict[str, TraceContext] = {}++ @override+ async def run_batch_branch(self, pipe_job: PipeJob) -> PipeOutput:+ received_metadata = kajson.loads(kajson.dumps(pipe_job.job_metadata))+ assert isinstance(received_metadata, JobMetadata)+ received_context = received_metadata.trace_context+ assert received_context is not None+ assert received_context.child_segment is not None+ item_segment = received_context.child_segment+ self.received_contexts[item_segment] = received_context+ execution = f"exec-{item_segment}"+ if item_segment != self.retried_item_segment:+ return await self._run_attempt(pipe_job=pipe_job, received_metadata=received_metadata, execution=execution, attempt=1)++ lost_attempt_token = _IS_LOST_ATTEMPT.set(True)+ try:+ await self._run_attempt(pipe_job=pipe_job, received_metadata=received_metadata, execution=execution, attempt=1)+ except PipelexError:+ pass+ else:+ msg = f"The first attempt of {execution} was meant to fail"+ raise AssertionError(msg)+ finally:+ _IS_LOST_ATTEMPT.reset(lost_attempt_token)+ return await self._run_attempt(pipe_job=pipe_job, received_metadata=received_metadata, execution=execution, attempt=2)++ async def _run_attempt(self, *, pipe_job: PipeJob, received_metadata: JobMetadata, execution: str, attempt: int) -> PipeOutput:+ received_context = received_metadata.trace_context+ assert received_context is not None+ tracer_manager = GraphTracerManager.get_or_create_instance()+ tracer_key = f"{execution}@{attempt}"+ attempt_context = tracer_manager.open_tracer(+ graph_id=received_context.graph_id,+ data_inclusion=received_context.data_inclusion,+ event_log=get_event_log_override(),+ workflow_id=execution,+ attempt=attempt,+ received_context=received_context,+ pipeline_run_id=received_metadata.run_metadata.pipeline_run_id,+ tracer_key=tracer_key,+ run_mode=received_context.run_mode,+ )+ # Each attempt starts from the input the engine delivered, never from what an earlier attempt left behind.+ attempt_job = pipe_job.model_copy(+ update={+ "job_metadata": received_metadata.model_copy(update={"trace_context": attempt_context}),+ "working_memory": pipe_job.get_working_memory().make_deep_copy(),+ }+ )+ try:+ return await super().run_batch_branch(attempt_job)+ finally:+ tracer_manager.close_tracer(tracer_key)diff --git a/tests/integration/pipelex/pipeline/invocation_paths/test_data.py b/tests/integration/pipelex/pipeline/invocation_paths/test_data.pynew file mode 100644index 000000000..5c5ccc1ac--- /dev/null+++ b/tests/integration/pipelex/pipeline/invocation_paths/test_data.py@@ -0,0 +1,189 @@+from typing import Any, ClassVar+++class InvocationPathsTestData:+ """The worked example of the hosted progress contract, with leaves that run without a model.++ `filings.review_filings` is a sequence of four steps: a condition whose `detailed` outcome runs a parallel of two+ branches, then a step batched over the filings, whose items each run a sequence of two steps. Beyond the worked+ example, the condition maps a second key to `deep_checks`, which joins that group, and a key holding a space and a+ `/` to `legal_check`, whose segment percent-encodes it.+ """++ DOMAIN: ClassVar[str] = "filings"+ ENTRY_PATH: ClassVar[str] = "filings.review_filings"+ BUNDLE: ClassVar[str] = """+domain = "filings"+description = "Review a set of filings"+main_pipe = "review_filings"++[concept.Filing]+description = "A filing"++[concept.Filing.structure]+title = { type = "text", description = "The filing's title", required = true }++[concept.Dossier]+description = "The filings under review"++[concept.Dossier.structure]+filings = { type = "list", item_type = "concept", item_concept_ref = "Filing", description = "The filings", required = true }++[pipe.review_filings]+type = "PipeSequence"+description = "Extract the filings, check them at the depth asked, summarize each, report"+inputs = { dossier = "Dossier", depth = "Text" }+output = "Text"+steps = [+ { pipe = "extract_filings", result = "extracted" },+ { pipe = "choose_depth", result = "checks" },+ { pipe = "summarize_filing", batch_over = "dossier.filings", batch_as = "filing", result = "summaries" },+ { pipe = "assemble_report", result = "report" },+]++[pipe.extract_filings]+type = "PipeCompose"+description = "Extract the filings"+inputs = { depth = "Text" }+output = "Text"+template = "Extracted for a {{ depth }} review"++[pipe.choose_depth]+type = "PipeCondition"+description = "Check the filings in depth, quickly, or for the legal and tax questions"+inputs = { depth = "Text" }+output = "Anything"+expression = "depth"+outcomes = { detailed = "deep_checks", quick = "quick_check", "high risk" = "deep_checks", "legal/tax review" = "legal_check" }+default_outcome = "quick_check"++[pipe.deep_checks]+type = "PipeParallel"+description = "Check the figures and the wording at once"+inputs = { depth = "Text" }+output = "Composite"+branches = [+ { pipe = "check_figures", result = "figures" },+ { pipe = "check_wording", result = "wording" },+]++[pipe.check_figures]+type = "PipeCompose"+description = "Check the figures"+inputs = { depth = "Text" }+output = "Text"+template = "Figures checked for a {{ depth }} review"++[pipe.check_wording]+type = "PipeCompose"+description = "Check the wording"+inputs = { depth = "Text" }+output = "Text"+template = "Wording checked for a {{ depth }} review"++[pipe.quick_check]+type = "PipeCompose"+description = "Skim the filings"+inputs = { depth = "Text" }+output = "Text"+template = "Skimmed for a {{ depth }} review"++[pipe.legal_check]+type = "PipeCompose"+description = "Check the legal and tax questions"+inputs = { depth = "Text" }+output = "Text"+template = "Legal and tax questions checked for a {{ depth }} review"++[pipe.summarize_filing]+type = "PipeSequence"+description = "Read a filing, then condense it"+inputs = { filing = "Filing" }+output = "Text"+steps = [+ { pipe = "read_filing", result = "reading" },+ { pipe = "condense_filing", result = "summary" },+]++[pipe.read_filing]+type = "PipeCompose"+description = "Read a filing"+inputs = { filing = "Filing" }+output = "Text"+template = "Read {{ filing.title }}"++[pipe.condense_filing]+type = "PipeFunc"+description = "Condense the reading of a filing"+inputs = { reading = "Text" }+output = "Text"+function_name = "invocation_paths_condense_filing"++[pipe.assemble_report]+type = "PipeCompose"+description = "Assemble the report"+inputs = { extracted = "Text", summaries = "Text[]" }+output = "Text"+template = "{{ extracted }} / {{ summaries }}"+"""++ DOSSIER: ClassVar[dict[str, Any]] = {"filings": [{"title": "Annual accounts"}, {"title": "Board minutes"}, {"title": "Tax return"}]}++ # The paths of a live run over the three filings whose condition took `detailed`, as the worked example gives+ # them: by path, the code and the type of the pipe run there.+ LIVE_DETAILED_PATHS: ClassVar[dict[str, tuple[str, str]]] = {+ "filings.review_filings": ("review_filings", "PipeSequence"),+ "filings.review_filings/step_1": ("extract_filings", "PipeCompose"),+ "filings.review_filings/step_2": ("choose_depth", "PipeCondition"),+ "filings.review_filings/step_2/outcome_detailed": ("deep_checks", "PipeParallel"),+ "filings.review_filings/step_2/outcome_detailed/branch_1": ("check_figures", "PipeCompose"),+ "filings.review_filings/step_2/outcome_detailed/branch_2": ("check_wording", "PipeCompose"),+ "filings.review_filings/step_3": ("summarize_filing_batch", "PipeBatch"),+ "filings.review_filings/step_3/item_1": ("summarize_filing", "PipeSequence"),+ "filings.review_filings/step_3/item_1/step_1": ("read_filing", "PipeCompose"),+ "filings.review_filings/step_3/item_1/step_2": ("condense_filing", "PipeFunc"),+ "filings.review_filings/step_3/item_2": ("summarize_filing", "PipeSequence"),+ "filings.review_filings/step_3/item_2/step_1": ("read_filing", "PipeCompose"),+ "filings.review_filings/step_3/item_2/step_2": ("condense_filing", "PipeFunc"),+ "filings.review_filings/step_3/item_3": ("summarize_filing", "PipeSequence"),+ "filings.review_filings/step_3/item_3/step_1": ("read_filing", "PipeCompose"),+ "filings.review_filings/step_3/item_3/step_2": ("condense_filing", "PipeFunc"),+ "filings.review_filings/step_4": ("assemble_report", "PipeCompose"),+ }++ # Where each pipe of the condition runs, whichever outcome a live run takes and in a dry run, which runs them all.+ OUTCOME_GROUP_PATHS: ClassVar[dict[str, str]] = {+ "deep_checks": "filings.review_filings/step_2/outcome_detailed",+ "quick_check": "filings.review_filings/step_2/outcome_quick",+ "legal_check": "filings.review_filings/step_2/outcome_legal%2Ftax%20review",+ }++ # A sequence whose author wrote a binding step: the binding runs no pipe, and takes the number of the step it is.+ BOUND_STEP_BUNDLE: ClassVar[str] = """+domain = "bound_steps"+description = "Label a filing by its title"+main_pipe = "label_filing"++[concept.Filing]+description = "A filing"++[concept.Filing.structure]+title = { type = "text", description = "The filing's title", required = true }++[pipe.label_filing]+type = "PipeSequence"+description = "Bind the filing's title, then label the filing with it"+inputs = { filing = "Filing" }+output = "Text"+steps = [+ { from = "filing.title", result = "title" },+ { pipe = "write_label", result = "label" },+]++[pipe.write_label]+type = "PipeCompose"+description = "Write the label"+inputs = { title = "Text" }+output = "Text"+template = "Label: {{ title }}"+"""diff --git a/tests/integration/pipelex/pipeline/invocation_paths/test_invocation_paths.py b/tests/integration/pipelex/pipeline/invocation_paths/test_invocation_paths.pynew file mode 100644index 000000000..f73b7cecb--- /dev/null+++ b/tests/integration/pipelex/pipeline/invocation_paths/test_invocation_paths.py@@ -0,0 +1,287 @@+import json+import logging+from pathlib import Path+from typing import Any++import pytest+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.graph.graphspec import GraphSpec, NodeStatus+from pipelex.interpreter_hub import scoped_pipe_router+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.runtime_hub import scoped_event_log+from pipelex.system.execution_chain import ExecutionAttempt+from pipelex.system.job_metadata import JobMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.tracing.ndjson_event_log import NdjsonEventLog+from pipelex.tracing.trace_events import PipeEndErrorEvent, PipeStartEvent, TraceEvent+from tests.integration.pipelex.pipeline.invocation_paths.invocation_paths_helpers import ItemExecutionRouter+from tests.integration.pipelex.pipeline.invocation_paths.test_data import InvocationPathsTestData++_BINDING_PIPE_TYPE = "BindingStep"+++def _read_trace_lines(traces_dir: Path) -> list[dict[str, Any]]:+ """Every line the run wrote, file by file, each file in the order it was written."""+ lines: list[dict[str, Any]] = []+ for ndjson_file in sorted(traces_dir.rglob("*.ndjson")):+ lines.extend(json.loads(line) for line in ndjson_file.read_text(encoding="utf-8").splitlines() if line.strip())+ return lines+++def _concerned_node_id(line: dict[str, Any]) -> str:+ """The node an event is about, whose path it carries."""+ node_id: str+ match line["event_kind"]:+ case "edge":+ node_id = line["target_node_id"]+ case "batch_item" | "batch_aggregate":+ node_id = line["batch_controller_node_id"]+ case "parallel_combine":+ node_id = line["parallel_controller_node_id"]+ case "condition_output_merge":+ node_id = line["merge"]["condition_node_id"]+ case _:+ node_id = line["node_id"]+ return node_id+++def _invocation_paths(lines: list[dict[str, Any]]) -> dict[str, tuple[str, str]]:+ """By path, the code and type of the pipe each invocation start names, refusing two starts at one path and an+ invocation started without a path: only a binding step, which runs no pipe, has none.+ """+ paths: dict[str, tuple[str, str]] = {}+ for line in lines:+ if line["event_kind"] != "pipe_start":+ continue+ if line["path"] is None:+ assert line["pipe_type"] == _BINDING_PIPE_TYPE, f"an invocation started without a path: {line['pipe_code']}"+ continue+ assert line["path"] not in paths, f"two starts at {line['path']}"+ paths[line["path"]] = (line["pipe_code"], line["pipe_type"])+ return paths+++def _graph_paths(graph_spec: GraphSpec | None) -> list[str]:+ assert graph_spec is not None+ return sorted(node.path for node in graph_spec.nodes if node.path is not None)+++async def _run(*, depth: str, run_mode: PipeRunMode, traces_dir: Path) -> PipeOutput:+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=True, mock_inputs=run_mode.is_dry)+ runner = PipelexMTHDSProtocol(pipe_run_mode=run_mode, execution_config=execution_config)+ inputs: dict[str, Any] | None = None+ if not run_mode.is_dry:+ inputs = {"depth": depth, "dossier": {"concept": f"{InvocationPathsTestData.DOMAIN}.Dossier", "content": InvocationPathsTestData.DOSSIER}}+ with scoped_event_log(NdjsonEventLog(traces_dir=str(traces_dir))):+ response = await runner.execute(mthds_contents=[InvocationPathsTestData.BUNDLE], inputs=inputs)+ return response.pipe_output++[email protected](loop_scope="class")+class TestInvocationPaths:+ async def test_every_event_of_a_live_run_carries_the_path_of_the_worked_example(self, tmp_path: Path) -> None:+ pipe_output = await _run(depth="detailed", run_mode=PipeRunMode.LIVE, traces_dir=tmp_path)++ lines = _read_trace_lines(tmp_path)+ assert _invocation_paths(lines) == InvocationPathsTestData.LIVE_DETAILED_PATHS+ # One execution in its first attempt wrote them all.+ assert {(line["workflow_id"], line["attempt"], json.dumps(line["lineage"])) for line in lines} == {("direct", 1, "[]")}++ # The binding the runtime writes for the dotted `batch_over` runs no pipe and takes no path, and leaves the+ # batch it was written for the number of the author's step.+ binding_starts = [line for line in lines if line["event_kind"] == "pipe_start" and line["pipe_type"] == _BINDING_PIPE_TYPE]+ assert [(start["pipe_code"], start["path"], start["pipe_ref"]) for start in binding_starts] == [("dossier.filings", None, None)]++ # Every event carries the path of the invocation it is about, and an invocation's start names its pipe.+ path_by_node_id = {line["node_id"]: line["path"] for line in lines if line["event_kind"] == "pipe_start"}+ for line in lines:+ assert line["path"] == path_by_node_id[_concerned_node_id(line)], line+ if line["event_kind"] == "pipe_start" and line["path"] is not None:+ assert line["pipe_ref"] == f"{InvocationPathsTestData.DOMAIN}.{line['pipe_code']}"+ if line["event_kind"] == "pipe_end_success":+ ended_path: str | None = line["path"]+ expected_pipe = (+ (line["pipe_code"], _BINDING_PIPE_TYPE) if ended_path is None else InvocationPathsTestData.LIVE_DETAILED_PATHS[ended_path]+ )+ assert (line["pipe_code"], line["pipe_type"]) == expected_pipe++ # Item `n` of the batch is the filing at index `n - 1`.+ for item_number, filing in enumerate(InvocationPathsTestData.DOSSIER["filings"], start=1):+ read_end = next(+ line+ for line in lines+ if line["event_kind"] == "pipe_end_success"+ and line["path"] == f"{InvocationPathsTestData.ENTRY_PATH}/step_3/item_{item_number}/step_1"+ )+ assert filing["title"] in json.dumps(read_end), read_end++ # The graph names each invocation by the same path.+ assert _graph_paths(pipe_output.graph_spec) == sorted(InvocationPathsTestData.LIVE_DETAILED_PATHS)++ @pytest.mark.parametrize(+ ("depth", "chosen_pipe_code"),+ [+ ("detailed", "deep_checks"),+ ("high risk", "deep_checks"),+ ("quick", "quick_check"),+ ("routine", "quick_check"),+ ("legal/tax review", "legal_check"),+ ],+ ids=["first_key_of_a_group", "second_key_of_a_group", "key_sharing_the_default_pipe", "default_outcome", "key_needing_encoding"],+ )+ async def test_a_condition_runs_each_outcome_group_at_one_path(self, tmp_path: Path, depth: str, chosen_pipe_code: str) -> None:+ """The outcomes running one pipe form one group, named by its first key, whichever of them a live run takes."""+ await _run(depth=depth, run_mode=PipeRunMode.LIVE, traces_dir=tmp_path)++ outcome_starts = [+ (path, pipe_code)+ for path, (pipe_code, _) in _invocation_paths(_read_trace_lines(tmp_path)).items()+ if path.startswith(f"{InvocationPathsTestData.ENTRY_PATH}/step_2/") and path.count("/") == 2+ ]+ assert outcome_starts == [(InvocationPathsTestData.OUTCOME_GROUP_PATHS[chosen_pipe_code], chosen_pipe_code)]++ async def test_a_dry_run_names_each_pipe_by_the_path_a_live_run_gives_it(self, tmp_path: Path) -> None:+ """A dry run runs every outcome group of the condition, each at the path a live run taking it gives."""+ pipe_output = await _run(depth="detailed", run_mode=PipeRunMode.DRY, traces_dir=tmp_path)++ lines = _read_trace_lines(tmp_path)+ assert {line["mode"] for line in lines} == {"dry"}+ dry_paths = _invocation_paths(lines)+ for pipe_code, group_path in InvocationPathsTestData.OUTCOME_GROUP_PATHS.items():+ assert dry_paths[group_path][0] == pipe_code+ live_paths_outside_the_items = {path: pipe for path, pipe in InvocationPathsTestData.LIVE_DETAILED_PATHS.items() if "/item_" not in path}+ assert live_paths_outside_the_items.items() <= dry_paths.items()+ # The dry batch's items are numbered from 1 as the live batch's are, each running the same pipes.+ item_numbers = sorted(int(path.split("/item_")[1]) for path in dry_paths if path.count("/") == 2 and "/item_" in path)+ assert item_numbers == list(range(1, len(item_numbers) + 1))+ assert item_numbers+ assert dry_paths[f"{InvocationPathsTestData.ENTRY_PATH}/step_3/item_1/step_2"] == ("condense_filing", "PipeFunc")+ assert _graph_paths(pipe_output.graph_spec) == sorted(dry_paths)++ async def test_an_item_run_as_an_execution_of_its_own_continues_the_path_it_received(self, tmp_path: Path) -> None:+ """Each item runs as an execution of its own, from a job read back from its serialized form, and the engine runs+ one of them twice: both attempts stamp the same paths under the prefix the item received, the graph keeps the+ later one, and no node of the lost attempt shows as failed.+ """+ router = ItemExecutionRouter(retried_item_segment="item_2")+ with scoped_pipe_router(router):+ pipe_output = await _run(depth="detailed", run_mode=PipeRunMode.LIVE, traces_dir=tmp_path)++ batch_path = f"{InvocationPathsTestData.ENTRY_PATH}/step_3"+ run_attempt = ExecutionAttempt(execution="direct", attempt=1)+ # The job each item's execution read back carries the item's place and the attempt that dispatched it.+ assert sorted(router.received_contexts) == ["item_1", "item_2", "item_3"]+ for item_segment, received_context in router.received_contexts.items():+ assert (received_context.parent_path, received_context.child_segment) == (batch_path, item_segment)+ assert received_context.chain == [run_attempt]++ events: list[TraceEvent] = NdjsonEventLog(traces_dir=str(tmp_path)).read_events(pipe_output.pipeline_run_id)+ item_starts: dict[tuple[str, int], set[str]] = {}+ for event in events:+ if event.workflow_id == "direct":+ assert (event.attempt, event.lineage) == (1, [])+ continue+ # An item's execution stamps the lineage it was dispatched under, and only paths under the one it received.+ item_segment = event.workflow_id.removeprefix("exec-")+ assert event.lineage == [run_attempt]+ assert event.path is not None+ assert event.path == f"{batch_path}/{item_segment}" or event.path.startswith(f"{batch_path}/{item_segment}/")+ if isinstance(event, PipeStartEvent):+ item_starts.setdefault((item_segment, event.attempt), set()).add(event.path)+ # The two attempts of item 2 share their paths.+ assert sorted(item_starts) == [("item_1", 1), ("item_2", 1), ("item_2", 2), ("item_3", 1)]+ assert (+ item_starts["item_2", 1]+ == item_starts["item_2", 2]+ == {+ f"{batch_path}/item_2",+ f"{batch_path}/item_2/step_1",+ f"{batch_path}/item_2/step_2",+ }+ )+ lost_failures = [event for event in events if isinstance(event, PipeEndErrorEvent) and event.workflow_id == "exec-item_2"]+ assert {(event.attempt, event.path) for event in lost_failures} >= {(1, f"{batch_path}/item_2/step_2")}++ # Each attempt of each execution mints its own node ids, the two attempts of item 2 included, and each item's+ # root node names as its parent the batch node that the run's execution minted.+ starts = [event for event in events if isinstance(event, PipeStartEvent)]+ assert len({event.node_id for event in starts}) == len(starts)+ batch_node_id = next(event.node_id for event in starts if event.path == batch_path)+ item_roots = [event for event in starts if event.workflow_id != "direct" and event.path is not None and event.path.count("/") == 2]+ assert sorted((event.workflow_id, event.attempt) for event in item_roots) == [+ ("exec-item_1", 1),+ ("exec-item_2", 1),+ ("exec-item_2", 2),+ ("exec-item_3", 1),+ ]+ assert {event.parent_node_id for event in item_roots} == {batch_node_id}++ # Each attempt of each execution wrote its own file, which the reader found with the others.+ ndjson_files = sorted((tmp_path / pipe_output.pipeline_run_id).glob("*.ndjson"))+ for ndjson_file in ndjson_files:+ file_lines = [json.loads(line) for line in ndjson_file.read_text(encoding="utf-8").splitlines() if line.strip()]+ assert len({json.dumps([line["lineage"], line["workflow_id"], line["attempt"]]) for line in file_lines}) == 1, ndjson_file.name+ assert len(ndjson_files) == len({(event.chain_key, event.writer_id) for event in events})++ # The graph holds every invocation once, from the current attempts, and nothing of the lost attempt failed in it.+ graph_spec = pipe_output.graph_spec+ assert _graph_paths(graph_spec) == sorted(InvocationPathsTestData.LIVE_DETAILED_PATHS)+ assert graph_spec is not None+ assert [node.path for node in graph_spec.nodes if node.status == NodeStatus.FAILED] == []+ # The parent references across the dispatch still resolve: the batch node contains each item's current node.+ item_node_ids = {+ node.node_id+ for node in graph_spec.nodes+ if node.path is not None and node.path.startswith(f"{batch_path}/item_") and node.path.count("/") == 2+ }+ assert len(item_node_ids) == len(InvocationPathsTestData.DOSSIER["filings"])+ contained_by_batch = {edge.target for edge in graph_spec.edges if edge.kind.is_contains and edge.source == batch_node_id}+ assert item_node_ids <= contained_by_batch+ main_text = pipe_output.main_stuff.as_text.text+ for filing in InvocationPathsTestData.DOSSIER["filings"]:+ assert f"Condensed: Read {filing['title']}" in main_text++ async def test_a_binding_step_the_author_wrote_takes_its_number(self, tmp_path: Path) -> None:+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=True)+ runner = PipelexMTHDSProtocol(execution_config=execution_config)+ with scoped_event_log(NdjsonEventLog(traces_dir=str(tmp_path))):+ await runner.execute(+ mthds_contents=[InvocationPathsTestData.BOUND_STEP_BUNDLE],+ inputs={"filing": {"concept": "bound_steps.Filing", "content": {"title": "Annual accounts"}}},+ )++ lines = _read_trace_lines(tmp_path)+ assert _invocation_paths(lines) == {+ "bound_steps.label_filing": ("label_filing", "PipeSequence"),+ "bound_steps.label_filing/step_2": ("write_label", "PipeCompose"),+ }+ binding_starts = [line for line in lines if line["event_kind"] == "pipe_start" and line["pipe_type"] == _BINDING_PIPE_TYPE]+ assert [(start["pipe_code"], start["path"]) for start in binding_starts] == [("filing.title", None)]++ async def test_a_controller_that_hands_no_place_does_not_fail_the_run(+ self, tmp_path: Path, mocker: MockerFixture, invocation_path_faults: list[logging.LogRecord]+ ) -> None:+ """Tracing never fails a run: a dispatch site that forgets to hand its pipe a place leaves that pipe without a+ path, and the tracer logs the fault, which fails every other test (`tests/conftest.py`), while the run completes.+ """++ def _forget_the_place(self: JobMetadata, *, segment: str) -> JobMetadata: # ruff: ignore[unused-function-argument]+ return self++ mocker.patch.object(JobMetadata, "for_child_at", _forget_the_place)+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=True)+ runner = PipelexMTHDSProtocol(execution_config=execution_config)+ with scoped_event_log(NdjsonEventLog(traces_dir=str(tmp_path))):+ response = await runner.execute(+ mthds_contents=[InvocationPathsTestData.BOUND_STEP_BUNDLE],+ inputs={"filing": {"concept": "bound_steps.Filing", "content": {"title": "Annual accounts"}}},+ )++ assert response.pipe_output.main_stuff.as_text.text == "Label: Annual accounts"+ starts = [(line["pipe_code"], line["path"]) for line in _read_trace_lines(tmp_path) if line["event_kind"] == "pipe_start"]+ assert starts == [("label_filing", "bound_steps.label_filing"), ("filing.title", None), ("write_label", None)]+ assert [getattr(record, "pipe_ref", None) for record in invocation_path_faults] == ["bound_steps.write_label"]+ invocation_path_faults.clear()diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_method_faults.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_method_faults.pyindex 66a3f8349..c2a09278d 100644--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_method_faults.py+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_method_faults.py@@ -124,6 +124,7 @@ class TestPipeConditionMethodFaults:)try:condition_node_id, condition_context = tracer_manager.on_pipe_start(+ pipe_ref=f"tests.{pipe_condition.code}",trace_context=root_context,pipe_code=pipe_condition.code,pipe_type=pipe_condition.type,diff --git a/tests/unit/pipelex/graph/conftest.py b/tests/unit/pipelex/graph/conftest.pyindex 3f3551bc6..8484f6de4 100644--- a/tests/unit/pipelex/graph/conftest.py+++ b/tests/unit/pipelex/graph/conftest.py@@ -5,6 +5,7 @@ import pytestfrom pipelex.config import get_configfrom pipelex.graph.graph_config import GraphConfigfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContext@@ -59,6 +60,7 @@ def make_trace_context(error_stack_traces=error_stack_traces,),run_mode=PipeRunMode.LIVE,+ chain=make_direct_chain(),)diff --git a/tests/unit/pipelex/graph/test_graph_tracer.py b/tests/unit/pipelex/graph/test_graph_tracer.pyindex b7f35d480..c3aa50d8b 100644--- a/tests/unit/pipelex/graph/test_graph_tracer.py+++ b/tests/unit/pipelex/graph/test_graph_tracer.py@@ -1,11 +1,12 @@"""Unit tests for GraphTracer."""+import loggingfrom datetime import UTC, datetime, timedeltafrom pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graphspec import EdgeKind, GraphSpecMode, IOSpec, NodeKind, NodeStatusfrom pipelex.system.pipe_run_mode import PipeRunMode-from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config+from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config, make_trace_contextclass TestGraphTracer:@@ -68,6 +69,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)node_id, child_context = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -111,6 +113,7 @@ class TestGraphTracer:# Start parent (sequence controller)started_at = datetime.now(UTC)parent_id, parent_child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_sequence",trace_context=context,pipe_code="my_sequence",pipe_type="PipeSequence",@@ -120,7 +123,8 @@ class TestGraphTracer:# Start child 1 (operator)child1_id, _child1_ctx = tracer.on_pipe_start(- trace_context=parent_child_ctx,+ pipe_ref="tests.llm_step_1",+ trace_context=parent_child_ctx.with_child_segment(segment="step_1"),pipe_code="llm_step_1",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -134,7 +138,8 @@ class TestGraphTracer:# Start child 2 (operator)child2_id, _child2_ctx = tracer.on_pipe_start(- trace_context=parent_child_ctx,+ pipe_ref="tests.llm_step_2",+ trace_context=parent_child_ctx.with_child_segment(segment="step_2"),pipe_code="llm_step_2",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -164,6 +169,79 @@ class TestGraphTracer:assert {edge.source for edge in edges} == {parent_id}assert {edge.target for edge in edges} == {child1_id, child2_id}+ def test_a_pipe_started_without_its_place_is_traced_without_a_path(self, invocation_path_faults: list[logging.LogRecord]) -> None:+ """Tracing never fails a run: a pipe whose controller handed it no segment is traced without a path, and so is+ every pipe under it, which cannot be placed either, and the tracer logs each fault as an error.+ """+ tracer = GraphTracer()+ context = tracer.setup(graph_id="unplaced-test", data_inclusion=make_defaulted_data_inclusion_config(), run_mode=PipeRunMode.LIVE)+ started_at = datetime.now(UTC)++ entry_id, entry_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_sequence",+ trace_context=context,+ pipe_code="my_sequence",+ pipe_type="PipeSequence",+ node_kind=NodeKind.CONTROLLER,+ started_at=started_at,+ )+ placed_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.step_one",+ trace_context=entry_ctx.with_child_segment(segment="step_1"),+ pipe_code="step_one",+ pipe_type="PipeCompose",+ node_kind=NodeKind.OPERATOR,+ started_at=started_at,+ )+ unplaced_id, unplaced_ctx = tracer.on_pipe_start(+ pipe_ref="tests.step_two",+ trace_context=entry_ctx,+ pipe_code="step_two",+ pipe_type="PipeSequence",+ node_kind=NodeKind.CONTROLLER,+ started_at=started_at,+ )+ under_unplaced_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.leaf",+ trace_context=unplaced_ctx.with_child_segment(segment="step_1"),+ pipe_code="leaf",+ pipe_type="PipeCompose",+ node_kind=NodeKind.OPERATOR,+ started_at=started_at,+ )+ for node_id in (under_unplaced_id, unplaced_id, placed_id, entry_id):+ tracer.on_pipe_end_success(node_id=node_id, ended_at=started_at + timedelta(milliseconds=10))+ graph_spec = tracer.teardown()++ assert graph_spec is not None+ assert {node.node_id: node.path for node in graph_spec.nodes} == {+ entry_id: "tests.my_sequence",+ placed_id: "tests.my_sequence/step_1",+ unplaced_id: None,+ under_unplaced_id: None,+ }+ faults = [(record.levelno, getattr(record, "pipe_ref", None), getattr(record, "node_id", None)) for record in invocation_path_faults]+ assert faults == [(logging.ERROR, "tests.step_two", unplaced_id), (logging.ERROR, "tests.leaf", under_unplaced_id)]+ invocation_path_faults.clear()++ def test_an_inactive_tracer_derives_no_path(self) -> None:+ """An inactive tracer records nothing, so it derives nothing that could fail the run, even from a context that+ could place no pipe.+ """+ unplaceable_context = make_trace_context().with_child_segment(segment="step_1")++ node_id, child_context = GraphTracer().on_pipe_start(+ pipe_ref="tests.step_one",+ trace_context=unplaceable_context,+ pipe_code="step_one",+ pipe_type="PipeCompose",+ node_kind=NodeKind.OPERATOR,+ started_at=datetime.now(UTC),+ )++ assert node_id == "test-graph:node_0"+ assert (child_context.parent_node_id, child_context.parent_path, child_context.child_segment) == (node_id, None, None)+def test_error_tracking(self) -> None:"""Test tracking failed pipe execution."""tracer = GraphTracer()@@ -171,6 +249,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.failing_pipe",trace_context=context,pipe_code="failing_pipe",pipe_type="PipeFunc",@@ -203,6 +282,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.lifted_pipe",trace_context=context,pipe_code="lifted_pipe",pipe_type="PipeFunc",@@ -234,6 +314,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)producer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.lifted_plural_pipe",trace_context=context,pipe_code="lifted_plural_pipe",pipe_type="PipeFunc",@@ -248,6 +329,7 @@ class TestGraphTracer:)consumer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.items_consumer",trace_context=context,pipe_code="items_consumer",pipe_type="PipeFunc",@@ -276,6 +358,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)producer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.maybe_producer",trace_context=context,pipe_code="maybe_producer",pipe_type="PipeCondition",@@ -289,6 +372,7 @@ class TestGraphTracer:)consumer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.verdict_consumer",trace_context=context,pipe_code="verdict_consumer",pipe_type="PipeLLM",@@ -314,6 +398,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)_node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.stuck_pipe",trace_context=context,pipe_code="stuck_pipe",pipe_type="PipeLLM",@@ -337,6 +422,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)node1_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_1",trace_context=context,pipe_code="pipe_1",pipe_type="PipeLLM",@@ -346,6 +432,7 @@ class TestGraphTracer:tracer.on_pipe_end_success(node_id=node1_id, ended_at=started_at + timedelta(milliseconds=10))node2_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_2",trace_context=context,pipe_code="pipe_2",pipe_type="PipeLLM",@@ -379,6 +466,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)condition_id, cond_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_condition",trace_context=context,pipe_code="my_condition",pipe_type="PipeCondition",@@ -387,7 +475,8 @@ class TestGraphTracer:)outcome_id, _ = tracer.on_pipe_start(- trace_context=cond_ctx,+ pipe_ref="tests.outcome_true",+ trace_context=cond_ctx.with_child_segment(segment="step_1"),pipe_code="outcome_true",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -423,6 +512,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -456,6 +546,7 @@ class TestGraphTracer:started_at = datetime.now(UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.summarize",trace_context=context,pipe_code="summarize",pipe_type="PipeLLM",@@ -486,6 +577,7 @@ class TestGraphTracer:# Pipe 1: produces stuff with digest "stuff_001"node1_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.producer_pipe",trace_context=context,pipe_code="producer_pipe",pipe_type="PipeLLM",@@ -500,6 +592,7 @@ class TestGraphTracer:# Pipe 2: consumes stuff with digest "stuff_001"node2_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_pipe",trace_context=context,pipe_code="consumer_pipe",pipe_type="PipeLLM",@@ -530,6 +623,7 @@ class TestGraphTracer:# Pipe consumes stuff that wasn't produced by any tracked pipenode_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_only",trace_context=context,pipe_code="consumer_only",pipe_type="PipeLLM",@@ -557,6 +651,7 @@ class TestGraphTracer:# Pipe produces and consumes the same stuff (shouldn't happen, but guard against it)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.self_ref_pipe",trace_context=context,pipe_code="self_ref_pipe",pipe_type="PipeLLM",@@ -586,6 +681,7 @@ class TestGraphTracer:# Producer pipeproducer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.producer",trace_context=context,pipe_code="producer",pipe_type="PipeLLM",@@ -600,6 +696,7 @@ class TestGraphTracer:# Consumer 1consumer1_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_1",trace_context=context,pipe_code="consumer_1",pipe_type="PipeLLM",@@ -611,6 +708,7 @@ class TestGraphTracer:# Consumer 2consumer2_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_2",trace_context=context,pipe_code="consumer_2",pipe_type="PipeLLM",@@ -644,6 +742,7 @@ class TestGraphTracer:# PipeBatch that consumes the listbatch_id, batch_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_batch",trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",@@ -654,7 +753,8 @@ class TestGraphTracer:# Branch 0: processes item 0branch0_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.branch_pipe",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -669,7 +769,8 @@ class TestGraphTracer:# Branch 1: processes item 1branch1_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.branch_pipe",+ trace_context=batch_ctx.with_child_segment(segment="step_2"),pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -728,6 +829,7 @@ class TestGraphTracer:# PipeBatch that produces the output listbatch_id, batch_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_batch",trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",@@ -738,7 +840,8 @@ class TestGraphTracer:# Branch 0: produces item 0branch0_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.branch_pipe",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -752,7 +855,8 @@ class TestGraphTracer:# Branch 1: produces item 1branch1_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.branch_pipe",+ trace_context=batch_ctx.with_child_segment(segment="step_2"),pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -806,6 +910,7 @@ class TestGraphTracer:# PipeBatchbatch_id, batch_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_batch",trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",@@ -816,7 +921,8 @@ class TestGraphTracer:# Branch 0branch0_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.process_item",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="process_item",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -900,6 +1006,7 @@ class TestGraphTracer:# PipeBatch that consumes the listbatch_id, batch_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_batch",trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",@@ -910,7 +1017,8 @@ class TestGraphTracer:# Branch: processes itembranch_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.branch_pipe",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -957,6 +1065,7 @@ class TestGraphTracer:# PipeBatchbatch_id, batch_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_batch",trace_context=context,pipe_code="my_batch",pipe_type="PipeBatch",@@ -967,7 +1076,8 @@ class TestGraphTracer:# Branch: produces itembranch_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.branch_pipe",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="branch_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -1015,6 +1125,7 @@ class TestGraphTracer:# Controller node (e.g., PipeParallel)controller_id, ctrl_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_parallel",trace_context=context,pipe_code="my_parallel",pipe_type="PipeParallel",@@ -1025,7 +1136,8 @@ class TestGraphTracer:# Branch 1: produces output with digest "branch_output_1"branch1_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.branch_pipe_1",+ trace_context=ctrl_ctx.with_child_segment(segment="step_1"),pipe_code="branch_pipe_1",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -1040,7 +1152,8 @@ class TestGraphTracer:# Branch 2: produces output with digest "branch_output_2"branch2_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.branch_pipe_2",+ trace_context=ctrl_ctx.with_child_segment(segment="step_2"),pipe_code="branch_pipe_2",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -1065,6 +1178,7 @@ class TestGraphTracer:# Consumer pipe that uses branch_output_1consumer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_pipe",trace_context=context,pipe_code="consumer_pipe",pipe_type="PipeLLM",@@ -1112,6 +1226,7 @@ class TestGraphTracer:# Producer pipe creates stuff with digest "original_stuff"producer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.producer",trace_context=context,pipe_code="producer",pipe_type="PipeLLM",@@ -1126,6 +1241,7 @@ class TestGraphTracer:# Controller consumes "original_stuff" and its main_stuff is the samecontroller_id, _ctrl_ctx = tracer.on_pipe_start(+ pipe_ref="tests.my_parallel",trace_context=context,pipe_code="my_parallel",pipe_type="PipeParallel",@@ -1143,6 +1259,7 @@ class TestGraphTracer:# Consumer should still get the edge from the original producer, not the controllerconsumer_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer",trace_context=context,pipe_code="consumer",pipe_type="PipeLLM",@@ -1183,6 +1300,7 @@ class TestGraphTracer:# Controller with multiple outputscontroller_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.multi_output_pipe",trace_context=context,pipe_code="multi_output_pipe",pipe_type="PipeParallel",@@ -1211,6 +1329,7 @@ class TestGraphTracer:# Consumer A reads digest_aconsumer_a_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_a",trace_context=context,pipe_code="consumer_a",pipe_type="PipeLLM",@@ -1222,6 +1341,7 @@ class TestGraphTracer:# Consumer B reads digest_bconsumer_b_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_b",trace_context=context,pipe_code="consumer_b",pipe_type="PipeLLM",@@ -1233,6 +1353,7 @@ class TestGraphTracer:# Consumer C reads digest_cconsumer_c_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer_c",trace_context=context,pipe_code="consumer_c",pipe_type="PipeLLM",diff --git a/tests/unit/pipelex/graph/test_graph_tracer_decision_events.py b/tests/unit/pipelex/graph/test_graph_tracer_decision_events.pyindex 738669a6d..75fb76b00 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_decision_events.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_decision_events.py@@ -9,6 +9,7 @@ from pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.graph.graph_tracer_protocol import GraphTracerNoOpfrom pipelex.graph.graphspec import IOSpec, NodeKind+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContextfrom pipelex.tracing.in_memory_event_log import InMemoryEventLog@@ -52,12 +53,13 @@ class TestGraphTracerDecisionEvents:"""The resolution is recorded before the chosen pipe starts, and the graph's SELECTED_OUTCOME edge to it is labelled with the key."""tracer, context, event_log = _make_tracer(run_mode=PipeRunMode.LIVE)condition_id, condition_ctx = tracer.on_pipe_start(- trace_context=context, pipe_code="route", pipe_type="PipeCondition", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.route", trace_context=context, pipe_code="route", pipe_type="PipeCondition", node_kind=NodeKind.CONTROLLER, started_at=_T0)tracer.register_condition_resolution(condition_node_id=condition_id, rendered_value=rendered_value, selected_outcome=selected_outcome)chosen_id, _ = tracer.on_pipe_start(- trace_context=condition_ctx,+ pipe_ref="tests.handle",+ trace_context=condition_ctx.with_child_segment(segment="step_1"),pipe_code="handle",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,@@ -84,17 +86,29 @@ class TestGraphTracerDecisionEvents:"""An outcome mapped to `continue` or `fail` starts no pipe: the resolution is recorded, and the graph has no edge from it."""tracer, context, event_log = _make_tracer(run_mode=PipeRunMode.LIVE)sequence_id, sequence_ctx = tracer.on_pipe_start(- trace_context=context, pipe_code="flow", pipe_type="PipeSequence", node_kind=NodeKind.CONTROLLER, started_at=_T0, step_count=2+ pipe_ref="tests.flow",+ trace_context=context,+ pipe_code="flow",+ pipe_type="PipeSequence",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,+ step_count=2,)condition_id, _ = tracer.on_pipe_start(- trace_context=sequence_ctx, pipe_code="route", pipe_type="PipeCondition", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.route",+ trace_context=sequence_ctx.with_child_segment(segment="step_1"),+ pipe_code="route",+ pipe_type="PipeCondition",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,)tracer.register_condition_resolution(condition_node_id=condition_id, rendered_value="skip", selected_outcome="skip")tracer.on_pipe_end_success(node_id=condition_id, ended_at=_T0 + timedelta(seconds=1))# The next step starts under the sequence, never under the conditionnext_step_id, _ = tracer.on_pipe_start(- trace_context=sequence_ctx,+ pipe_ref="tests.later",+ trace_context=sequence_ctx.with_child_segment(segment="step_2"),pipe_code="later",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,@@ -112,6 +126,7 @@ class TestGraphTracerDecisionEvents:def test_batch_fan_out_precedes_the_first_item(self) -> None:tracer, context, event_log = _make_tracer(run_mode=PipeRunMode.LIVE)batch_id, batch_ctx = tracer.on_pipe_start(+ pipe_ref="tests.per_item",trace_context=context,pipe_code="per_item",pipe_type="PipeBatch",@@ -126,7 +141,8 @@ class TestGraphTracerDecisionEvents:item_ids: list[str] = []for index_item in range(2):item_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.process",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="process",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,@@ -149,10 +165,17 @@ class TestGraphTracerDecisionEvents:def test_controllers_state_their_width_as_they_start(self) -> None:tracer, context, event_log = _make_tracer(run_mode=PipeRunMode.LIVE)sequence_id, sequence_ctx = tracer.on_pipe_start(- trace_context=context, pipe_code="flow", pipe_type="PipeSequence", node_kind=NodeKind.CONTROLLER, started_at=_T0, step_count=3+ pipe_ref="tests.flow",+ trace_context=context,+ pipe_code="flow",+ pipe_type="PipeSequence",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,+ step_count=3,)parallel_id, _ = tracer.on_pipe_start(- trace_context=sequence_ctx,+ pipe_ref="tests.fan",+ trace_context=sequence_ctx.with_child_segment(segment="step_1"),pipe_code="fan",pipe_type="PipeParallel",node_kind=NodeKind.CONTROLLER,@@ -173,7 +196,12 @@ class TestGraphTracerDecisionEvents:assert context.run_mode == run_modebatch_id, batch_ctx = tracer.on_pipe_start(- trace_context=context, pipe_code="per_item", pipe_type="PipeBatch", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.per_item",+ trace_context=context,+ pipe_code="per_item",+ pipe_type="PipeBatch",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,)assert batch_ctx.run_mode == run_modetracer.register_batch_fan_out(@@ -181,7 +209,8 @@ class TestGraphTracerDecisionEvents:)tracer.register_batch_item_extraction(list_stuff_code="list", item_stuff_code="item", item_index=0, batch_controller_node_id=batch_id)condition_id, condition_ctx = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.route",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="route",pipe_type="PipeCondition",node_kind=NodeKind.CONTROLLER,@@ -189,7 +218,8 @@ class TestGraphTracerDecisionEvents:)tracer.register_condition_resolution(condition_node_id=condition_id, rendered_value="yes", selected_outcome="yes")chosen_id, _ = tracer.on_pipe_start(- trace_context=condition_ctx,+ pipe_ref="tests.handle",+ trace_context=condition_ctx.with_child_segment(segment="step_1"),pipe_code="handle",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,@@ -231,13 +261,19 @@ class TestGraphTracerDecisionEvents:run_mode=PipeRunMode.LIVE,)batch_id, batch_ctx = tracer.on_pipe_start(- trace_context=context, pipe_code="per_item", pipe_type="PipeBatch", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.per_item",+ trace_context=context,+ pipe_code="per_item",+ pipe_type="PipeBatch",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,)tracer.register_batch_fan_out(batch_node_id=batch_id, item_count=1, branch_pipe_code="route", branch_pipe_ref="flows.route", branch_pipe_type="PipeCondition")condition_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.route",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="route",pipe_type="PipeCondition",node_kind=NodeKind.CONTROLLER,@@ -257,7 +293,9 @@ class TestGraphTracerDecisionEvents:"""tracer, context, event_log = _make_tracer(run_mode=PipeRunMode.LIVE)foreign_node_id = "graph-decisions:other_workflow:node_0"- foreign_ctx = context.copy_for_child(child_node_id=foreign_node_id, next_sequence=0)+ foreign_ctx = context.copy_for_child(child_node_id=foreign_node_id, next_sequence=0, invocation_path="flows.route/step_1").with_child_segment(+ segment="outcome_yes"+ )tracer.register_batch_fan_out(batch_node_id=foreign_node_id,@@ -268,7 +306,12 @@ class TestGraphTracerDecisionEvents:)tracer.register_condition_resolution(condition_node_id=foreign_node_id, rendered_value="yes", selected_outcome="yes")chosen_id, _ = tracer.on_pipe_start(- trace_context=foreign_ctx, pipe_code="handle", pipe_type="PipeCompose", node_kind=NodeKind.OPERATOR, started_at=_T0+ pipe_ref="tests.handle",+ trace_context=foreign_ctx,+ pipe_code="handle",+ pipe_type="PipeCompose",+ node_kind=NodeKind.OPERATOR,+ started_at=_T0,)tracer.on_pipe_end_success(node_id=chosen_id, ended_at=_T0 + timedelta(seconds=1))graph_spec = tracer.teardown()@@ -281,7 +324,7 @@ class TestGraphTracerDecisionEvents:def test_a_trace_context_states_its_run_mode(self) -> None:"""A context has no default mode: one built without it would stamp a dry run's events as live."""with pytest.raises(ValidationError, match="run_mode"):- TraceContext(graph_id=_GRAPH_ID, data_inclusion=make_defaulted_data_inclusion_config()) # type: ignore[call-arg] # pyright: ignore[reportCallIssue] # the static error is the runtime contract under test+ TraceContext(graph_id=_GRAPH_ID, data_inclusion=make_defaulted_data_inclusion_config(), chain=make_direct_chain()) # type: ignore[call-arg] # pyright: ignore[reportCallIssue] # the static error is the runtime contract under testdef test_the_noop_tracer_accepts_decisions_and_carries_the_mode(self) -> None:tracer = GraphTracerNoOp()@@ -308,7 +351,12 @@ class TestGraphTracerDecisionEvents:run_mode=PipeRunMode.DRY,)batch_id, _ = manager.on_pipe_start(- trace_context=context, pipe_code="per_item", pipe_type="PipeBatch", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.per_item",+ trace_context=context,+ pipe_code="per_item",+ pipe_type="PipeBatch",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,)assert batch_id is not Nonemanager.register_batch_fan_out(diff --git a/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.py b/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.pyindex 0c39dfb6d..6a26b5e2c 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_emit_gating.py@@ -11,6 +11,7 @@ class TestGraphTracerEmitGating:def _start_a_node(self, tracer: GraphTracer, context: object) -> None:tracer.on_pipe_start(+ pipe_ref="tests.some_pipe",trace_context=context, # type: ignore[arg-type]pipe_code="some_pipe",pipe_type="PipeLLM",diff --git a/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py b/tests/unit/pipelex/graph/test_graph_tracer_event_emission.pyindex c10780b5b..67df0b1c6 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py@@ -8,6 +8,7 @@ from datetime import UTC, datetime, timedeltafrom pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graphspec import EdgeKind, IOSpec, NodeKind, NodeStatus+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.tracing.in_memory_event_log import InMemoryEventLogfrom pipelex.tracing.trace_events import (@@ -62,6 +63,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -97,6 +99,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)parent_id, child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.parent_pipe",trace_context=context,pipe_code="parent_pipe",pipe_type="PipeSequence",@@ -105,7 +108,8 @@ class TestGraphTracerEventEmission:)child_id, _ = tracer.on_pipe_start(- trace_context=child_ctx,+ pipe_ref="tests.child_pipe",+ trace_context=child_ctx.with_child_segment(segment="step_1"),pipe_code="child_pipe",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -134,6 +138,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -176,6 +181,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -201,6 +207,47 @@ class TestGraphTracerEventEmission:assert evt.ended_at == ended_atassert evt.error.error_type == "ValueError"assert evt.error.message == "Something went wrong"+ assert evt.error.caller_facing is False+ # An end event names the pipe it ends, as its start does, and the invocation's path.+ assert (evt.pipe_code, evt.pipe_ref, evt.pipe_type, evt.path) == ("test_pipe", "tests.test_pipe", "PipeLLM", "tests.test_pipe")++ def test_pipe_end_error_states_whether_its_message_was_written_for_the_caller(self) -> None:+ """The failure's `caller_facing` reaches the event and the node of the graph."""+ tracer, event_log = self._make_tracer_with_event_log()+ context = tracer.setup(+ graph_id=self.GRAPH_ID,+ data_inclusion=make_defaulted_data_inclusion_config(),+ event_log=event_log,+ workflow_id=self.WORKFLOW_ID,+ pipeline_run_id=self.PIPELINE_RUN_ID,+ run_mode=PipeRunMode.LIVE,+ )+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)+ node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",+ trace_context=context,+ pipe_code="test_pipe",+ pipe_type="PipeParallel",+ node_kind=NodeKind.CONTROLLER,+ started_at=started_at,+ )++ tracer.on_pipe_end_error(+ node_id=node_id,+ ended_at=started_at + timedelta(milliseconds=50),+ error_type="StuffFactoryError",+ error_message="PipeParallel 'test_pipe' cannot combine its branch results into its output 'Report'.",+ caller_facing=True,+ )+ graph_spec = tracer.teardown()++ error_events = [evt for evt in event_log.read_events(self.PIPELINE_RUN_ID) if isinstance(evt, PipeEndErrorEvent)]+ assert [evt.error.caller_facing for evt in error_events] == [True]+ assert graph_spec is not None+ failed_node = next(node for node in graph_spec.nodes if node.node_id == node_id)+ assert failed_node.error is not None+ assert failed_node.error.caller_facing is True+ assert failed_node.path == "tests.test_pipe"def test_add_edge_emits_event(self) -> None:"""EdgeEvent is emitted on add_edge."""@@ -217,6 +264,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_a, _ctx_a = tracer.on_pipe_start(+ pipe_ref="tests.pipe_a",trace_context=context,pipe_code="pipe_a",pipe_type="PipeLLM",@@ -224,6 +272,7 @@ class TestGraphTracerEventEmission:started_at=started_at,)node_b, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_b",trace_context=context,pipe_code="pipe_b",pipe_type="PipeLLM",@@ -266,6 +315,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.parallel_pipe",trace_context=context,pipe_code="parallel_pipe",pipe_type="PipeParallel",@@ -361,6 +411,7 @@ class TestGraphTracerEventEmission:# Create controller nodectrl_id, ctrl_ctx = tracer.on_pipe_start(+ pipe_ref="tests.parallel_ctrl",trace_context=context,pipe_code="parallel_ctrl",pipe_type="PipeParallel",@@ -370,7 +421,8 @@ class TestGraphTracerEventEmission:# Create branch nodes and end them with outputsbranch_a_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.branch_a",+ trace_context=ctrl_ctx.with_child_segment(segment="step_1"),pipe_code="branch_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -383,7 +435,8 @@ class TestGraphTracerEventEmission:)branch_b_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.branch_b",+ trace_context=ctrl_ctx.with_child_segment(segment="step_2"),pipe_code="branch_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -421,6 +474,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -441,7 +495,7 @@ class TestGraphTracerEventEmission:assert graph_spec.nodes[0].status == NodeStatus.SUCCEEDEDdef test_node_id_includes_workflow_id_in_temporal_mode(self) -> None:- """Node IDs include workflow_id segment when not in direct mode."""+ """Node IDs name the workflow and its attempt when not in direct mode."""tracer, event_log = self._make_tracer_with_event_log(workflow_id="wf_run_123")context = tracer.setup(@@ -455,6 +509,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",@@ -463,7 +518,61 @@ class TestGraphTracerEventEmission:)assert "wf_run_123" in node_id- assert node_id == f"{self.GRAPH_ID}:wf_run_123:node_0"+ assert node_id == f"{self.GRAPH_ID}:wf_run_123:attempt_1:node_0"++ def test_each_attempt_and_each_dispatch_mints_its_own_ids(self) -> None:+ """Two attempts of one workflow, and a workflow two attempts of its parent dispatched under one name, number+ their nodes and edges from 0 alike: their ids differ by the chain alone, so no reader keyed on an id merges the+ node a lost attempt failed with the one its retry ran.+ """+ run_1 = ExecutionAttempt(execution="wf_run", attempt=1)+ run_2 = ExecutionAttempt(execution="wf_run", attempt=2)+ setups: list[tuple[str, int, list[ExecutionAttempt]]] = [+ ("wf_run", 1, []),+ ("wf_run", 2, []),+ ("wf_item", 1, [run_1]),+ ("wf_item", 2, [run_1]),+ ("wf_item", 1, [run_2]),+ ]+ ids_by_chain: list[set[str]] = []+ for workflow_id, attempt, lineage in setups:+ event_log = InMemoryEventLog()+ tracer = GraphTracer()+ context = tracer.setup(+ graph_id=self.GRAPH_ID,+ data_inclusion=make_defaulted_data_inclusion_config(),+ event_log=event_log,+ workflow_id=workflow_id,+ attempt=attempt,+ lineage=lineage,+ pipeline_run_id=self.PIPELINE_RUN_ID,+ run_mode=PipeRunMode.LIVE,+ )+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)+ node_a, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_a",+ trace_context=context,+ pipe_code="pipe_a",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ started_at=started_at,+ )+ node_b, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_b",+ trace_context=context,+ pipe_code="pipe_b",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ started_at=started_at,+ )+ tracer.add_edge(source_node_id=node_a, target_node_id=node_b, edge_kind=EdgeKind.DATA)+ edge_ids = {evt.edge_id for evt in event_log.read_events(self.PIPELINE_RUN_ID) if isinstance(evt, EdgeEvent)}+ assert len(edge_ids) == 1+ ids_by_chain.append({node_a, node_b, *edge_ids})+ tracer.teardown()++ all_ids = [minted_id for chain_ids in ids_by_chain for minted_id in chain_ids]+ assert len(all_ids) == len(set(all_ids)) == 3 * len(setups)def test_edge_id_includes_workflow_id_in_temporal_mode(self) -> None:"""Edge IDs include workflow_id segment when not in direct mode."""@@ -480,6 +589,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_a, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_a",trace_context=context,pipe_code="pipe_a",pipe_type="PipeLLM",@@ -487,6 +597,7 @@ class TestGraphTracerEventEmission:started_at=started_at,)node_b, _ = tracer.on_pipe_start(+ pipe_ref="tests.pipe_b",trace_context=context,pipe_code="pipe_b",pipe_type="PipeLLM",@@ -503,7 +614,7 @@ class TestGraphTracerEventEmission:events = event_log.read_events(self.PIPELINE_RUN_ID)edge_events = [evt for evt in events if isinstance(evt, EdgeEvent) and evt.edge_kind == EdgeKind.DATA]assert len(edge_events) == 1- assert "wf_run_456" in edge_events[0].edge_id+ assert edge_events[0].edge_id == f"{self.GRAPH_ID}:wf_run_456:attempt_1:edge_0"def test_event_sequence_is_monotonic(self) -> None:"""Events have monotonically increasing sequence numbers."""@@ -521,6 +632,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)for index in range(5):node_id, _ = tracer.on_pipe_start(+ pipe_ref=f"tests.{f'pipe_{index}'}",trace_context=context,pipe_code=f"pipe_{index}",pipe_type="PipeLLM",@@ -544,6 +656,7 @@ class TestGraphTracerEventEmission:started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)node_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",diff --git a/tests/unit/pipelex/graph/test_graph_tracer_manager.py b/tests/unit/pipelex/graph/test_graph_tracer_manager.pyindex f71600a38..76751874b 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_manager.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_manager.py@@ -1,12 +1,18 @@"""Unit tests for GraphTracerManager's per-run tracer lifecycle (open/close registration)."""+from datetime import UTC, datetime+import pytestfrom pytest_mock import MockerFixturefrom pipelex.graph.graph_tracer import GraphTracerfrom pipelex.graph.graph_tracer_manager import GraphTracerManager+from pipelex.graph.graphspec import NodeKindfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.tracing.trace_events import PipeStartEventdef _make_data_inclusion() -> DataInclusionConfig:@@ -45,3 +51,100 @@ class TestGraphTracerManagerOpenTracer:assert manager.get_tracer("leak_check_graph") is Nonefinally:GraphTracerManager.clear_instance()++ def test_a_dispatched_execution_continues_the_place_and_the_chain_it_received(self) -> None:+ """A batch item run as an execution of its own stamps paths under its item's, and the dispatching attempt as its lineage."""+ GraphTracerManager.clear_instance()+ manager = GraphTracerManager.get_or_create_instance()+ event_log = InMemoryEventLog()+ try:+ run_context = manager.open_tracer(+ graph_id="run_7Qd2kV",+ data_inclusion=_make_data_inclusion(),+ event_log=event_log,+ workflow_id="exec-run",+ attempt=2,+ pipeline_run_id="run_7Qd2kV",+ tracer_key="exec-run-key",+ run_mode=PipeRunMode.LIVE,+ )+ assert (run_context.chain, run_context.lineage) == ([ExecutionAttempt(execution="exec-run", attempt=2)], [])+ _, entry_context = manager.on_pipe_start(+ trace_context=run_context,+ pipe_code="review_filings",+ pipe_ref="filings.review_filings",+ pipe_type="PipeSequence",+ node_kind=NodeKind.CONTROLLER,+ started_at=datetime.now(UTC),+ )+ assert entry_context is not None+ batch_node_id, batch_context = manager.on_pipe_start(+ trace_context=entry_context.with_child_segment(segment="step_3"),+ pipe_code="summarize_filing_batch",+ pipe_ref="filings.summarize_filing_batch",+ pipe_type="PipeBatch",+ node_kind=NodeKind.CONTROLLER,+ started_at=datetime.now(UTC),+ )+ assert batch_context is not None++ item_context = manager.open_tracer(+ graph_id="run_7Qd2kV",+ data_inclusion=_make_data_inclusion(),+ event_log=event_log,+ workflow_id="exec-item-2",+ received_context=batch_context.with_child_segment(segment="item_2"),+ pipeline_run_id="run_7Qd2kV",+ tracer_key="exec-item-2-key",+ run_mode=PipeRunMode.LIVE,+ )+ assert item_context.chain == [ExecutionAttempt(execution="exec-run", attempt=2), ExecutionAttempt(execution="exec-item-2", attempt=1)]+ assert (item_context.parent_node_id, item_context.parent_path, item_context.child_segment) == (+ batch_node_id,+ "filings.review_filings/step_3",+ "item_2",+ )+ manager.on_pipe_start(+ trace_context=item_context,+ pipe_code="summarize_filing",+ pipe_ref="filings.summarize_filing",+ pipe_type="PipeSequence",+ node_kind=NodeKind.CONTROLLER,+ started_at=datetime.now(UTC),+ )+ finally:+ manager.close_tracer("exec-item-2-key")+ manager.close_tracer("exec-run-key")+ GraphTracerManager.clear_instance()++ starts = [event for event in event_log.read_events("run_7Qd2kV") if isinstance(event, PipeStartEvent)]+ assert sorted((event.workflow_id, event.attempt, event.path, len(event.lineage)) for event in starts) == [+ ("exec-item-2", 1, "filings.review_filings/step_3/item_2", 1),+ ("exec-run", 2, "filings.review_filings", 0),+ ("exec-run", 2, "filings.review_filings/step_3", 0),+ ]+ item_start = next(event for event in starts if event.workflow_id == "exec-item-2")+ assert item_start.lineage == [ExecutionAttempt(execution="exec-run", attempt=2)]++ def test_a_received_context_naming_no_place_leaves_the_lineage_empty(self) -> None:+ """The run's own execution receives the submitter's context, which names no place: it was dispatched by no attempt."""+ GraphTracerManager.clear_instance()+ manager = GraphTracerManager.get_or_create_instance()+ try:+ submitted_context = manager.open_tracer(+ graph_id="submitted_graph", data_inclusion=_make_data_inclusion(), tracer_key="submitter", run_mode=PipeRunMode.LIVE+ )+ run_context = manager.open_tracer(+ graph_id="submitted_graph",+ data_inclusion=_make_data_inclusion(),+ workflow_id="exec-run",+ attempt=3,+ received_context=submitted_context,+ tracer_key="exec-run-key",+ run_mode=PipeRunMode.LIVE,+ )+ assert run_context.chain == [ExecutionAttempt(execution="exec-run", attempt=3)]+ assert (run_context.parent_path, run_context.child_segment) == (None, None)+ assert run_context.invocation_path(pipe_ref="filings.review_filings") == "filings.review_filings"+ finally:+ GraphTracerManager.clear_instance()diff --git a/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.py b/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.pyindex 91ab0948c..8509db97f 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_node_metadata.py@@ -15,6 +15,7 @@ class TestGraphTracerNodeMetadata:started_at = datetime.now(UTC)node_id, _child = tracer.on_pipe_start(+ pipe_ref="tests.summarize",trace_context=context,pipe_code="summarize",pipe_type="PipeLLM",@@ -43,6 +44,7 @@ class TestGraphTracerNodeMetadata:started_at = datetime.now(UTC)node_id, _child = tracer.on_pipe_start(+ pipe_ref="tests.noop",trace_context=context,pipe_code="noop",pipe_type="PipeLLM",diff --git a/tests/unit/pipelex/graph/test_graph_tracer_noop.py b/tests/unit/pipelex/graph/test_graph_tracer_noop.pyindex 5df846054..00a3832ac 100644--- a/tests/unit/pipelex/graph/test_graph_tracer_noop.py+++ b/tests/unit/pipelex/graph/test_graph_tracer_noop.py@@ -5,7 +5,7 @@ from datetime import UTC, datetime, timedeltafrom pipelex.graph.graph_tracer_protocol import GraphTracerNoOpfrom pipelex.graph.graphspec import EdgeKind, NodeKindfrom pipelex.system.pipe_run_mode import PipeRunMode-from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config+from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config, make_trace_contextclass TestGraphTracerNoOp:@@ -26,6 +26,23 @@ class TestGraphTracerNoOp:assert result is None+ def test_noop_derives_no_path(self) -> None:+ """The no-op tracer records nothing, so it derives nothing that could fail the run, even from a context that+ could place no pipe.+ """+ unplaceable_context = make_trace_context().with_child_segment(segment="step_1")++ node_id, child_context = GraphTracerNoOp().on_pipe_start(+ pipe_ref="tests.step_one",+ trace_context=unplaceable_context,+ pipe_code="step_one",+ pipe_type="PipeCompose",+ node_kind=NodeKind.OPERATOR,+ started_at=datetime.now(UTC),+ )++ assert (child_context.parent_node_id, child_context.parent_path, child_context.child_segment) == (node_id, None, None)+def test_noop_pipe_lifecycle(self) -> None:"""Test that no-op tracer handles pipe lifecycle without errors."""tracer = GraphTracerNoOp()@@ -33,6 +50,7 @@ class TestGraphTracerNoOp:started_at = datetime.now(UTC)node_id, child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.test_pipe",trace_context=context,pipe_code="test_pipe",pipe_type="PipeLLM",diff --git a/tests/unit/pipelex/graph/test_trace_context.py b/tests/unit/pipelex/graph/test_trace_context.pyindex a3c2a4e74..ed790eada 100644--- a/tests/unit/pipelex/graph/test_trace_context.py+++ b/tests/unit/pipelex/graph/test_trace_context.py@@ -1,6 +1,18 @@"""Unit tests for TraceContext."""-from tests.unit.pipelex.graph.conftest import make_trace_context+import pytest+from pydantic import ValidationError++from pipelex.system.exceptions import InvocationPathError+from pipelex.system.execution_chain import ExecutionAttempt, execution_chain_digest, execution_chain_key+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.trace_context import TraceContext, make_trace_id_prefix+from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config, make_trace_context++_RUN_1 = ExecutionAttempt(execution="exec-run", attempt=1)+_RUN_2 = ExecutionAttempt(execution="exec-run", attempt=2)+_ITEM_1 = ExecutionAttempt(execution="exec-item-2", attempt=1)+_ITEM_2 = ExecutionAttempt(execution="exec-item-2", attempt=2)class TestTraceContext:@@ -13,17 +25,58 @@ class TestTraceContext:assert node_id == "ctx-test:node_5"+ def test_a_trace_context_states_its_chain(self) -> None:+ """A context has no default chain: one built without it would write a retried workflow's usage as `direct@1`."""+ with pytest.raises(ValidationError, match="chain"):+ TraceContext(graph_id="ctx-test", data_inclusion=make_defaulted_data_inclusion_config(), run_mode=PipeRunMode.LIVE) # type: ignore[call-arg] # pyright: ignore[reportCallIssue] # the static error is the runtime contract under test++ @pytest.mark.parametrize(+ ("chain", "expected_prefix"),+ [+ ([ExecutionAttempt(execution="direct", attempt=1)], "run_1"),+ ([_RUN_2], "run_1:exec-run:attempt_2"),+ (+ [_RUN_1, _ITEM_2],+ f"run_1:exec-item-2:attempt_2:lineage_{execution_chain_digest(chain_key=execution_chain_key(chain=[_RUN_1]))[:16]}",+ ),+ ],+ ids=["direct_mode", "run_workflow_retried", "dispatched_workflow"],+ )+ def test_node_ids_name_the_chain_that_mints_them(self, chain: list[ExecutionAttempt], expected_prefix: str) -> None:+ context = make_trace_context(graph_id="run_1", node_sequence=3).model_copy(update={"chain": chain})++ assert make_trace_id_prefix(graph_id="run_1", chain=chain) == expected_prefix+ assert context.make_node_id() == f"{expected_prefix}:node_3"++ def test_every_chain_of_a_run_mints_its_own_node_ids(self) -> None:+ """Each attempt numbers its nodes from 0 and keeps its execution's name, so the ids name the whole chain: a retry+ of the run's workflow, a retry of an item's, and an item dispatched again under the same name by a new attempt+ of the run's workflow each mint ids no other chain mints.+ """+ chains = [[_RUN_1], [_RUN_2], [_RUN_1, _ITEM_1], [_RUN_1, _ITEM_2], [_RUN_2, _ITEM_1]]+ first_node_ids = [make_trace_context(graph_id="run_1").model_copy(update={"chain": chain}).make_node_id() for chain in chains]++ assert len(set(first_node_ids)) == len(chains)+ # The same chain, as a replay of an attempt has, mints the same ids.+ replayed = make_trace_context(graph_id="run_1").model_copy(update={"chain": [_RUN_2, _ITEM_1]})+ assert replayed.make_node_id() == first_node_ids[-1]+def test_copy_for_child(self) -> None:"""Test creating child context."""parent = make_trace_context(graph_id="ctx-test", parent_node_id=None, node_sequence=0)- child = parent.copy_for_child(child_node_id="ctx-test:node_0", next_sequence=1)+ child = parent.copy_for_child(child_node_id="ctx-test:node_0", next_sequence=1, invocation_path="tests.ctx_pipe")assert child.graph_id == "ctx-test"assert child.parent_node_id == "ctx-test:node_0"assert child.node_sequence == 1+ # The child continues the parent's invocation path, names no segment yet, and keeps the parent's chain+ assert child.parent_path == "tests.ctx_pipe"+ assert child.child_segment is None+ assert child.chain == parent.chain# Parent should be unchangedassert parent.parent_node_id is Noneassert parent.node_sequence == 0+ assert parent.parent_path is Nonedef test_copy_for_child_preserves_data_inclusion(self) -> None:"""Test that copy_for_child preserves data_inclusion config."""@@ -31,6 +84,59 @@ class TestTraceContext:graph_id="ctx-test",stuff_json_content=True,)- child = parent.copy_for_child(child_node_id="ctx-test:node_0", next_sequence=1)+ child = parent.copy_for_child(child_node_id="ctx-test:node_0", next_sequence=1, invocation_path="tests.ctx_pipe")assert child.data_inclusion.stuff_json_content is True++ def test_the_entry_pipe_is_named_by_its_pipe_ref(self) -> None:+ assert make_trace_context().invocation_path(pipe_ref="filings.review_filings") == "filings.review_filings"++ def test_a_child_pipe_continues_its_parent_path_with_the_segment_its_controller_gave(self) -> None:+ parent = make_trace_context().copy_for_child(child_node_id="test-graph:node_0", next_sequence=1, invocation_path="filings.review_filings")+ placed = parent.with_child_segment(segment="step_3")++ assert placed.invocation_path(pipe_ref="filings.summarize_filing_batch") == "filings.review_filings/step_3"+ # The segment rides on the copy handed to that one pipe, never on the controller's own context.+ assert parent.child_segment is None++ @pytest.mark.parametrize(+ ("parent_path", "child_segment", "message"),+ [+ ("filings.review_filings", None, "without the segment its controller gives it"),+ (None, "step_1", "with no parent invocation to place it under"),+ ],+ ids=["segment_missing", "parent_missing"],+ )+ def test_a_path_that_cannot_be_derived_is_refused(self, parent_path: str | None, child_segment: str | None, message: str) -> None:+ context = make_trace_context().model_copy(update={"parent_path": parent_path, "child_segment": child_segment})++ with pytest.raises(InvocationPathError, match=message):+ context.invocation_path(pipe_ref="filings.read_filing")++ def test_the_chain_names_the_writing_attempt_and_its_lineage(self) -> None:+ run_attempt = ExecutionAttempt(execution="exec-run", attempt=1)+ item_attempt = ExecutionAttempt(execution="exec-item-2", attempt=2)+ context = make_trace_context().model_copy(update={"chain": [run_attempt, item_attempt]})++ assert (context.own_attempt, context.lineage) == (item_attempt, [run_attempt])+ assert make_trace_context().chain == [ExecutionAttempt(execution="direct", attempt=1)]+ assert make_trace_context().lineage == []++ def test_the_path_prefix_and_the_chain_survive_serialization(self) -> None:+ """What a dispatched execution reads back from its input is what the dispatcher sent."""+ context = (+ make_trace_context()+ .copy_for_child(child_node_id="test-graph:node_4", next_sequence=5, invocation_path="filings.review_filings/step_3")+ .with_child_segment(segment="item_2")+ .model_copy(update={"chain": [ExecutionAttempt(execution="exec-run", attempt=3)]})+ )++ recovered = TraceContext.model_validate_json(context.model_dump_json())++ assert recovered == context+ assert (recovered.parent_path, recovered.child_segment, recovered.chain) == (+ "filings.review_filings/step_3",+ "item_2",+ [ExecutionAttempt(execution="exec-run", attempt=3)],+ )+ assert recovered.invocation_path(pipe_ref="filings.summarize_filing") == "filings.review_filings/step_3/item_2"diff --git a/tests/unit/pipelex/graph/test_trace_context_emit_flags.py b/tests/unit/pipelex/graph/test_trace_context_emit_flags.pyindex c64110895..239ef4317 100644--- a/tests/unit/pipelex/graph/test_trace_context_emit_flags.py+++ b/tests/unit/pipelex/graph/test_trace_context_emit_flags.py@@ -6,6 +6,7 @@ carried — so a costs-only or graph-only run keeps its gating as nested pipes s"""from pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContext@@ -19,7 +20,7 @@ DATA_INCLUSION_OFF = DataInclusionConfig(class TestTraceContextEmitFlags:def test_defaults_emit_both_streams(self) -> None:"""A TraceContext built without explicit flags emits both streams (legacy behavior)."""- context = TraceContext(graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF, run_mode=PipeRunMode.LIVE)+ context = TraceContext(graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF, run_mode=PipeRunMode.LIVE, chain=make_direct_chain())assert context.emit_graph_events is Trueassert context.emit_usage_events is Trueassert context.describe_pipe_io is True@@ -27,9 +28,14 @@ class TestTraceContextEmitFlags:def test_copy_for_child_preserves_costs_only_flags(self) -> None:"""costs-only flags survive into a child context (graph off, costs on)."""parent = TraceContext(- graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF, emit_graph_events=False, emit_usage_events=True, run_mode=PipeRunMode.LIVE+ graph_id="run_1",+ data_inclusion=DATA_INCLUSION_OFF,+ emit_graph_events=False,+ emit_usage_events=True,+ run_mode=PipeRunMode.LIVE,+ chain=make_direct_chain(),)- child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=1)+ child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=1, invocation_path="tests.ctx_pipe")assert child.parent_node_id == "run_1:node_0"assert child.node_sequence == 1assert child.emit_graph_events is False@@ -38,14 +44,21 @@ class TestTraceContextEmitFlags:def test_copy_for_child_preserves_graph_only_flags(self) -> None:"""graph-only flags survive into a child context (graph on, costs off)."""parent = TraceContext(- graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF, emit_graph_events=True, emit_usage_events=False, run_mode=PipeRunMode.LIVE+ graph_id="run_1",+ data_inclusion=DATA_INCLUSION_OFF,+ emit_graph_events=True,+ emit_usage_events=False,+ run_mode=PipeRunMode.LIVE,+ chain=make_direct_chain(),)- child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=2)+ child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=2, invocation_path="tests.ctx_pipe")assert child.emit_graph_events is Trueassert child.emit_usage_events is Falsedef test_copy_for_child_preserves_describe_pipe_io_off(self) -> None:"""A run that will write no graphspec keeps its I/O description off in every child context."""- parent = TraceContext(graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF, describe_pipe_io=False, run_mode=PipeRunMode.LIVE)- child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=1)+ parent = TraceContext(+ graph_id="run_1", data_inclusion=DATA_INCLUSION_OFF, describe_pipe_io=False, run_mode=PipeRunMode.LIVE, chain=make_direct_chain()+ )+ child = parent.copy_for_child(child_node_id="run_1:node_0", next_sequence=1, invocation_path="tests.ctx_pipe")assert child.describe_pipe_io is Falsediff --git a/tests/unit/pipelex/graph/test_traced_failure.py b/tests/unit/pipelex/graph/test_traced_failure.pynew file mode 100644index 000000000..f93432b61--- /dev/null+++ b/tests/unit/pipelex/graph/test_traced_failure.py@@ -0,0 +1,46 @@+import logging++import pytest+from pytest_mock import MockerFixture++from pipelex.base_exceptions import PipelexError+from pipelex.core.stuffs.exceptions import StuffFactoryError+from pipelex.graph.traced_failure import ERROR_REPORT_FAULT_MESSAGE, TracedFailure, make_traced_failure+++class TestTracedFailure:+ @pytest.mark.parametrize(+ ("error", "expected"),+ [+ (+ StuffFactoryError("Input 'topic' is missing").as_caller_fault(),+ TracedFailure(message="Input 'topic' is missing", caller_facing=True),+ ),+ (+ StuffFactoryError("A factory refusal that is not the caller's"),+ TracedFailure(message="A factory refusal that is not the caller's", caller_facing=False),+ ),+ (RuntimeError("The worker was lost"), TracedFailure(message="The worker was lost", caller_facing=False)),+ ],+ ids=["caller_fault", "pipelex_error", "foreign_exception"],+ )+ def test_a_failure_records_its_message_and_whether_it_was_written_for_the_caller(self, error: BaseException, expected: TracedFailure) -> None:+ assert make_traced_failure(error=error) == expected++ def test_a_pipelex_error_records_its_report_message(self) -> None:+ error = StuffFactoryError("Error combining stuffs for concept Report").as_caller_fault()+ assert isinstance(error, PipelexError)+ assert make_traced_failure(error=error).message == error.to_error_report().message++ def test_a_report_that_cannot_be_built_records_the_errors_text(self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> None:+ """Tracing never changes how a run fails: a report that raises is logged, and the node records the error's text."""+ error = StuffFactoryError("Input 'topic' is missing").as_caller_fault()+ mocker.patch.object(error, "to_error_report", side_effect=RuntimeError("The report could not be built"))++ with caplog.at_level(logging.ERROR):+ traced_failure = make_traced_failure(error=error)++ assert traced_failure == TracedFailure(message=str(error), caller_facing=False)+ (record,) = [record for record in caplog.records if record.getMessage().endswith(ERROR_REPORT_FAULT_MESSAGE)]+ assert record.levelno == logging.ERROR+ assert (vars(record)["failed_error_type"], vars(record)["error.type"]) == ("StuffFactoryError", "RuntimeError")diff --git a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.pyindex b17645248..0cde2ea52 100644--- a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py+++ b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py@@ -30,6 +30,7 @@ from pipelex.kernel.memory_ops import ()from pipelex.kernel.pipelex_kernel import PipelexKernelfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContextfrom pipelex.tools.log.log_context import LogContext, get_log_context@@ -51,6 +52,7 @@ def _trace_context(*, run_mode: PipeRunMode = PipeRunMode.DRY) -> TraceContext:emit_graph_events=False,emit_usage_events=True,run_mode=run_mode,+ chain=make_direct_chain(),)diff --git a/tests/unit/pipelex/pipe_controllers/binding/test_binding_step.py b/tests/unit/pipelex/pipe_controllers/binding/test_binding_step.pyindex 6a4ee5e4a..d14d958f8 100644--- a/tests/unit/pipelex/pipe_controllers/binding/test_binding_step.py+++ b/tests/unit/pipelex/pipe_controllers/binding/test_binding_step.py@@ -1,4 +1,5 @@import pytest+from pytest_mock import MockerFixturefrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.memory.absence import AbsenceKind, AbsenceRecord@@ -8,9 +9,12 @@ from pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.number_content import NumberContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.pipe_controllers.binding import binding_step as binding_step_modulefrom pipelex.pipe_controllers.binding.binding_derivation import BindingDerivation, BindingRoot, derive_bindingfrom pipelex.pipe_controllers.binding.binding_step import BindingStep+from pipelex.system.job_metadata import JobMetadata, RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunMode+from tests.unit.pipelex.graph.conftest import make_trace_contextfrom tests.unit.pipelex.pipe_controllers.binding.test_data import INVOICE, RESOLVER, SupplierRecord, make_invoice_record_INVOICE_CONCEPT = Concept(domain_code="billing", code="Invoice", description="An invoice", structure_class_name="InvoiceRecord")@@ -239,6 +243,33 @@ class TestBindingStep:with pytest.raises(PipeRunInputsError, match="reads 'invoice', which is not in working memory"):step.skip_untyped_root(working_memory=WorkingMemory(), calling_pipe_code=_SEQUENCE_CODE, run_mode=PipeRunMode.LIVE)+ @pytest.mark.parametrize(+ ("error", "expected_failure"),+ [+ (None, ("UnknownError", "the binding step stopped without binding a value", False)),+ (RuntimeError("the root could not be read"), ("RuntimeError", "the root could not be read", False)),+ ],+ ids=["stopped_without_an_error", "stopped_by_an_error"],+ )+ def test_a_stopped_binding_closes_its_node_as_failed_once(+ self, mocker: MockerFixture, error: BaseException | None, expected_failure: tuple[str, str, bool]+ ) -> None:+ tracer_manager = mocker.MagicMock()+ mocker.patch.object(binding_step_module.GraphTracerManager, "get_instance", return_value=tracer_manager)+ job_metadata = JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test_user", pipeline_run_id="run_binding"),+ trace_context=make_trace_context(graph_id="run_binding"),+ )++ BindingStep(from_path="invoice.supplier", output_name="supplier").trace_error(+ job_metadata=job_metadata, node_id="run_binding:node_3", error=error+ )++ tracer_manager.on_pipe_end_error.assert_called_once()+ call_kwargs = tracer_manager.on_pipe_end_error.call_args.kwargs+ assert (call_kwargs["lookup_key"], call_kwargs["node_id"]) == ("run_binding", "run_binding:node_3")+ assert (call_kwargs["error_type"], call_kwargs["error_message"], call_kwargs["caller_facing"]) == expected_failure+def test_the_step_is_written_back_as_mthds_writes_it(self) -> None:step = BindingStep(from_path="invoice.supplier.address.city", output_name="city")diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_outcome_groups.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_outcome_groups.pynew file mode 100644index 000000000..e47df2629--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_outcome_groups.py@@ -0,0 +1,148 @@+from collections.abc import Callable+from typing import Any++import pytest+from pytest_mock import MockerFixture++from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.interpreter_hub import get_pipe_library, get_required_pipe+from pipelex.libraries.pipe.pipe_library import PipeLibrary+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome+from pipelex.pipe_machinery.pipe_factory import PipeFactory+from pipelex.pipe_operators.compose.pipe_compose import PipeCompose+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.system.execution_chain import make_direct_chain+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+from tests.unit.pipelex.graph.conftest import make_trace_context++_DEPENDENCY_ALIAS = "risk_pack"+++def _make_outcome_pipe(*, domain_code: str, pipe_code: str) -> PipeCompose:+ return PipeFactory[PipeCompose].make_from_blueprint(+ domain_code=domain_code,+ pipe_code=pipe_code,+ blueprint=PipeComposeBlueprint(description=f"The {pipe_code} outcome", inputs={}, output="native.Text", template=pipe_code),+ )+++def _load_outcome_pipes() -> None:+ """The filings domain's own outcome pipes, and a dependency package's `risk.deep_checks`."""+ pipe_library = get_pipe_library()+ # The library a test opens is the concrete one, which holds a dependency package's pipes under their alias.+ assert isinstance(pipe_library, PipeLibrary)+ for pipe_code in ("deep_checks", "quick_check", "legal_check", "file_away"):+ pipe_library.add_new_pipe(pipe=_make_outcome_pipe(domain_code="filings", pipe_code=pipe_code))+ pipe_library.add_dependency_pipe(alias=_DEPENDENCY_ALIAS, pipe=_make_outcome_pipe(domain_code="risk", pipe_code="deep_checks"), is_exported=True)+++def _make_condition(*, outcomes: dict[str, str | SpecialOutcome], default_outcome: str | SpecialOutcome) -> PipeCondition:+ return PipeFactory[PipeCondition].make_from_blueprint(+ domain_code="filings",+ pipe_code="choose_depth",+ blueprint=PipeConditionBlueprint(+ description="Check the filings in depth, quickly, or not at all",+ inputs={"depth": "native.Text"},+ output="native.Text?",+ expression="depth",+ outcomes=outcomes,+ default_outcome=default_outcome,+ ),+ )+++def _make_filings_condition(*, default_outcome: str | SpecialOutcome) -> PipeCondition:+ return _make_condition(+ outcomes={+ "detailed": "filings.deep_checks",+ "quick": "filings.quick_check",+ "high risk": "filings.deep_checks",+ "legal/tax review": "filings.legal_check",+ "skip": SpecialOutcome.CONTINUE,+ },+ default_outcome=default_outcome,+ )+++def _make_condition_spelling_one_pipe_two_ways() -> PipeCondition:+ """The dependency's `risk.deep_checks`, written with its domain by a key and without it by the default outcome."""+ return _make_condition(+ outcomes={"quick": "filings.quick_check", "detailed": f"{_DEPENDENCY_ALIAS}->risk.deep_checks"},+ default_outcome=f"{_DEPENDENCY_ALIAS}->deep_checks",+ )+++class TestPipeConditionOutcomeGroups:+ @pytest.mark.parametrize(+ ("default_outcome", "pipe_ref", "expected_segment"),+ [+ ("filings.file_away", "filings.deep_checks", "outcome_detailed"),+ ("filings.file_away", "filings.quick_check", "outcome_quick"),+ ("filings.file_away", "filings.legal_check", "outcome_legal%2Ftax%20review"),+ ("filings.file_away", "filings.file_away", "default"),+ ("filings.quick_check", "filings.quick_check", "outcome_quick"),+ ],+ ids=["group_of_two_keys", "group_of_one_key", "encoded_key", "group_only_the_default_runs", "default_joins_a_key_group"],+ )+ def test_each_pipe_runs_at_the_segment_of_its_outcome_group(+ self, load_empty_library: Callable[[], str], default_outcome: str, pipe_ref: str, expected_segment: str+ ) -> None:+ load_empty_library()+ _load_outcome_pipes()+ condition = _make_filings_condition(default_outcome=default_outcome)++ assert condition.outcome_group_segment(pipe=get_required_pipe(pipe_code=pipe_ref)) == expected_segment++ def test_outcomes_spelling_one_pipe_differently_form_one_group(self, load_empty_library: Callable[[], str]) -> None:+ """Outcomes are grouped by the pipe the library resolves them to, not by how they spell it."""+ load_empty_library()+ _load_outcome_pipes()+ condition = _make_condition_spelling_one_pipe_two_ways()+ dependency_pipe = get_required_pipe(pipe_code=f"{_DEPENDENCY_ALIAS}->risk.deep_checks")+ assert get_required_pipe(pipe_code=f"{_DEPENDENCY_ALIAS}->deep_checks") is dependency_pipe++ assert condition.outcome_group_segment(pipe=dependency_pipe) == "outcome_detailed"+ # The host's own `filings.deep_checks` shares the dependency's code and is another pipe, in no group here.+ assert condition.outcome_group_segment(pipe=get_required_pipe(pipe_code="filings.deep_checks")) == "default"+ assert [pipe.pipe_ref for pipe in condition.outcome_group_pipes()] == ["filings.quick_check", "risk.deep_checks"]+ assert condition.outcome_group_pipes()[-1] is dependency_pipe++ @pytest.mark.asyncio(loop_scope="function")+ async def test_a_dry_run_runs_each_group_once_at_its_path(self, load_empty_library: Callable[[], str], mocker: MockerFixture) -> None:+ """A dry run runs the pipe of every group, once however many outcomes spell it, at the segment a live run gives it."""+ load_empty_library()+ _load_outcome_pipes()+ condition = _make_condition_spelling_one_pipe_two_ways()+ runs: list[tuple[str, str | None]] = []++ def _record_run(+ self: PipeCompose,+ *,+ job_metadata: JobMetadata,+ working_memory: WorkingMemory,+ **_kwargs: Any,+ ) -> PipeOutput:+ assert job_metadata.trace_context is not None+ runs.append((self.pipe_ref, job_metadata.trace_context.child_segment))+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)++ mocker.patch.object(PipeCompose, "run_pipe", autospec=True, side_effect=_record_run)+ trace_context = make_trace_context(graph_id="run_1").model_copy(update={"parent_path": "filings.choose_depth", "chain": make_direct_chain()})+ job_metadata = JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="pytest", pipeline_run_id="run_1"),+ trace_context=trace_context,+ )++ await condition._dry_run_controller_pipe( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ job_metadata=job_metadata,+ working_memory=WorkingMemoryFactory.make_empty(),+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),+ )++ assert runs == [("filings.quick_check", "outcome_quick"), ("risk.deep_checks", "outcome_detailed")]diff --git a/tests/unit/pipelex/pipe_machinery/test_run_pipe_traced_failure.py b/tests/unit/pipelex/pipe_machinery/test_run_pipe_traced_failure.pynew file mode 100644index 000000000..09461d9e5--- /dev/null+++ b/tests/unit/pipelex/pipe_machinery/test_run_pipe_traced_failure.py@@ -0,0 +1,63 @@+"""Verify that tracing a pipe's failure never changes the error the run fails with."""++from typing import Callable++import pytest+from pytest_mock import MockerFixture++from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.exceptions import StuffFactoryError+from pipelex.pipe_machinery.pipe_abstract import InputPresenceScan+from pipelex.pipe_machinery.pipe_factory import PipeFactory+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from tests.unit.pipelex.graph.conftest import make_trace_context++[email protected](loop_scope="class")+class TestRunPipeTracedFailure:+ async def test_a_failure_whose_report_cannot_be_built_fails_the_run_with_the_pipes_own_error(+ self,+ mocker: MockerFixture,+ load_empty_library: Callable[[], str],+ ) -> None:+ """Building the failure's report runs while the pipe's error is on its way out: when it raises, the node records+ the error's text, and the run still fails with the pipe's own error, never with the report's.+ """+ load_empty_library()+ pipe = PipeFactory[PipeLLM].make_from_blueprint(+ domain_code="greeting_domain",+ pipe_code="say_hello",+ blueprint=PipeLLMBlueprint(description="Greet the user.", output="native.Text", prompt="Say hello."),+ )+ pipe_error = StuffFactoryError("The pipe's own failure")+ mocker.patch.object(pipe_error, "to_error_report", side_effect=RuntimeError("The report could not be built"))+ empty_presence_scan = InputPresenceScan(missing_names=[], forced_absent=[], liftable=[])+ mocker.patch.object(PipeLLM, "validate_before_run", mocker.AsyncMock(return_value=empty_presence_scan))+ mocker.patch.object(PipeLLM, "live_run_pipe", mocker.AsyncMock(side_effect=pipe_error))+ mock_manager = mocker.MagicMock()+ mock_manager.on_pipe_start = mocker.MagicMock(return_value=("test-run:node_0", None))+ mocker.patch("pipelex.pipe_machinery.pipe_abstract.GraphTracerManager.get_instance", return_value=mock_manager)+ job_metadata = JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="pytest", pipeline_run_id="test-run"),+ trace_context=make_trace_context(graph_id="test-run"),+ )++ with pytest.raises(StuffFactoryError) as exc_info:+ await pipe.run_pipe(+ job_metadata=job_metadata,+ working_memory=WorkingMemoryFactory.make_empty(),+ pipe_run_params=PipeRunParamsFactory.make_run_params(),+ )++ assert exc_info.value is pipe_error+ mock_manager.on_pipe_end_error.assert_called_once()+ end_kwargs = mock_manager.on_pipe_end_error.call_args.kwargs+ assert (end_kwargs["node_id"], end_kwargs["error_type"], end_kwargs["error_message"], end_kwargs["caller_facing"]) == (+ "test-run:node_0",+ "StuffFactoryError",+ str(pipe_error),+ False,+ )diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run_usage_stats.py b/tests/unit/pipelex/pipe_run/test_pipe_run_usage_stats.pyindex 3ef809b7e..0d8bd1ac3 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_run_usage_stats.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_run_usage_stats.py@@ -11,6 +11,7 @@ from pipelex.pipe_run import pipe_run as pipe_run_modulefrom pipelex.pipe_run.pipe_run import PipeRunfrom pipelex.pipe_run.tracing_assembly import RunTraceEventsfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.job_metadata import JobMetadata, RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.runtime import IntegrationMode@@ -36,6 +37,7 @@ def _job(*, mocker: MockerFixture, is_traced: bool) -> Any:emit_graph_events=False,emit_usage_events=True,run_mode=PipeRunMode.LIVE,+ chain=make_direct_chain(),)job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id=USAGE_STATS_RUN_ID, storage_scope="test/scope", read_scope=None),diff --git a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py b/tests/unit/pipelex/pipe_run/test_tracing_assembly.pyindex 948622b46..b76aa0991 100644--- a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py+++ b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py@@ -35,6 +35,9 @@ def _make_usage_event(pipeline_run_id: str, node_id: str, run_mode: PipeRunModenb_tokens_by_category={TokenCategory.INPUT: 3, TokenCategory.OUTPUT: 5},)return UsageReportEvent(+ attempt=1,+ lineage=[],+ path=None,pipeline_run_id=pipeline_run_id,workflow_id="direct",timestamp=datetime.now(UTC),diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.pyindex d8a8a77df..346abedef 100644--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py@@ -29,6 +29,7 @@ from pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackendfrom pipelex.system.data_inclusion_config import DataInclusionConfigfrom pipelex.system.exceptions import MissingDependencyError+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobIdfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContext@@ -83,6 +84,8 @@ def _make_trace_context(parent_node_id: str | None = "g:node_0",tracer_key: str | None = None,run_mode: PipeRunMode = PipeRunMode.LIVE,+ chain: list[ExecutionAttempt] | None = None,+ parent_path: str | None = None,) -> TraceContext:return TraceContext(graph_id=graph_id,@@ -91,6 +94,8 @@ def _make_trace_context(node_sequence=0,data_inclusion=DATA_INCLUSION_OFF,run_mode=run_mode,+ chain=chain or [ExecutionAttempt(execution=tracer_key or "direct", attempt=1)],+ parent_path=parent_path,)@@ -131,7 +136,8 @@ class TestEmitRunnerFallback:run_dir = tmp_path / "run_abc"ndjson_files = list(run_dir.glob("*.ndjson"))assert len(ndjson_files) == 1- assert ndjson_files[0].name.startswith("wf_wf_xyz__w_act_"), ndjson_files[0].name+ assert ndjson_files[0].name.startswith("wf_wf_xyz__c_"), ndjson_files[0].name+ assert "__w_act_" in ndjson_files[0].name, ndjson_files[0].namereader = NdjsonEventLog(traces_dir=str(tmp_path))events = reader.read_events("run_abc")@@ -158,33 +164,33 @@ class TestEmitRunnerFallback:usage_events = [evt for evt in NdjsonEventLog(traces_dir=str(tmp_path)).read_events("run_dry") if isinstance(evt, UsageReportEvent)]assert [evt.mode for evt in usage_events] == [PipeRunMode.DRY]- def test_runner_fallback_uses_tracer_key_when_set(+ def test_runner_fallback_stamps_the_chain_and_path_of_its_trace_context(self,tmp_path: Path,mocker: MockerFixture,) -> None:- """workflow_id on the emitted event is trace_context.tracer_key when set, else graph_id."""+ """An activity's usage event names the workflow attempt its trace context says it ran under, never the tracer key.++ A new attempt of that workflow supersedes it, so the event must carry the attempt and the lineage, and the path of+ the invocation whose work the inference served.+ """_enable_ndjson_tracing(mocker, tmp_path)manager = ReportingManager()manager.setup()+ chain = [ExecutionAttempt(execution="wf_run", attempt=1), ExecutionAttempt(execution="wf_item_2", attempt=3)]+ trace_context = _make_trace_context(+ graph_id="run_abc",+ tracer_key="temporal_run_id_of_attempt_3",+ chain=chain,+ parent_path="flows.review/step_3/item_2/step_1",+ )+ manager.report_inference_job(_make_llm_job("run_abc", trace_context=trace_context))- # Case 1: tracer_key set → workflow_id == tracer_key- ctx_with_tracer = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")- manager.report_inference_job(_make_llm_job("run_abc", trace_context=ctx_with_tracer))-- # Case 2: tracer_key None → workflow_id == graph_id- ctx_no_tracer = _make_trace_context(graph_id="run_def", tracer_key=None)- manager.report_inference_job(_make_llm_job("run_def", trace_context=ctx_no_tracer))-- reader = NdjsonEventLog(traces_dir=str(tmp_path))- events_abc = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]- events_def = [evt for evt in reader.read_events("run_def") if isinstance(evt, UsageReportEvent)]-- assert len(events_abc) == 1- assert events_abc[0].workflow_id == "wf_xyz"- assert len(events_def) == 1- assert events_def[0].workflow_id == "run_def"+ usage_events = [evt for evt in NdjsonEventLog(traces_dir=str(tmp_path)).read_events("run_abc") if isinstance(evt, UsageReportEvent)]+ assert [(evt.workflow_id, evt.attempt, evt.lineage, evt.path) for evt in usage_events] == [+ ("wf_item_2", 3, [ExecutionAttempt(execution="wf_run", attempt=1)], "flows.review/step_3/item_2/step_1")+ ]def test_fallback_caches_event_log_per_process(self,@@ -361,6 +367,32 @@ class TestEmitRunnerFallback:engaged_records = [record for record in caplog.records if record.getMessage() == _ENGAGED_MESSAGE]assert len(engaged_records) == 1+ def test_engaged_log_names_the_run_beside_its_execution(+ self,+ tmp_path: Path,+ mocker: MockerFixture,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ """Two direct runs in one process: the line fires once, for the first, and names that run by its tracer key,+ since the execution of every direct run is `direct`.+ """+ _enable_ndjson_tracing(mocker, tmp_path)++ manager = ReportingManager()+ manager.setup()+ with caplog.at_level(logging.INFO):+ for run_id in ("run_first", "run_second"):+ manager.report_inference_job(_make_llm_job(run_id, trace_context=_make_trace_context(graph_id=run_id)))++ engaged_records = [record for record in caplog.records if record.getMessage() == _ENGAGED_MESSAGE]+ assert [(getattr(record, "workflow_id", None), getattr(record, "tracer_key", None)) for record in engaged_records] == [+ ("direct", "run_first")+ ]+ # Both runs' usage was emitted all the same, each under the direct execution.+ for run_id in ("run_first", "run_second"):+ usage_events = [evt for evt in NdjsonEventLog(traces_dir=str(tmp_path)).read_events(run_id) if isinstance(evt, UsageReportEvent)]+ assert [evt.workflow_id for evt in usage_events] == ["direct"]+def test_engaged_log_emitted_once_even_with_multiple_managers(self,tmp_path: Path,diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.pyindex 15098d35a..6d06a253e 100644--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py@@ -36,6 +36,7 @@ from pipelex.config import get_configfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackendfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.job_metadata import JobMetadata, RunMetadatafrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContext@@ -74,6 +75,7 @@ def _trace_context(run_id: str, tracer_key: str) -> TraceContext:node_sequence=0,data_inclusion=_DATA_INCLUSION_OFF,run_mode=PipeRunMode.LIVE,+ chain=make_direct_chain(),)diff --git a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.pyindex 482ebed37..305e06483 100644--- a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py+++ b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py@@ -22,6 +22,7 @@ from pipelex.config import get_configfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackendfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import make_direct_chainfrom pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobIdfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContext@@ -71,6 +72,7 @@ def _make_trace_context(graph_id: str, *, emit_usage_events: bool, emit_graph_evemit_graph_events=emit_graph_events,emit_usage_events=emit_usage_events,run_mode=PipeRunMode.LIVE,+ chain=make_direct_chain(),)@@ -140,7 +142,6 @@ class TestEmitUsageEventGating:manager.set_event_log(context_key=trace_context.lookup_key,event_log=event_log_spy,- workflow_id="direct",pipeline_run_id="run_leaked",)@@ -164,7 +165,6 @@ class TestEmitUsageEventGating:manager.set_event_log(context_key=trace_context.lookup_key,event_log=event_log_spy,- workflow_id="direct",pipeline_run_id="run_fastpath",)@@ -198,7 +198,6 @@ class TestEmitUsageEventGating:manager.set_event_log(context_key=trace_context.lookup_key,event_log=event_log_spy,- workflow_id="wf_run_isolated",pipeline_run_id="run_isolated",)diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.pyindex 504411b4e..ea53f64d8 100644--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py@@ -20,6 +20,7 @@ from pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.config import get_configfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobIdfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.system.trace_context import TraceContext@@ -68,8 +69,10 @@ def _make_trace_context(tracer_key: str | None = None,node_sequence: int = 0,run_mode: PipeRunMode = PipeRunMode.LIVE,+ chain: list[ExecutionAttempt] | None = None,+ parent_path: str | None = None,) -> TraceContext:- """Create a TraceContext for testing with lookup_key resolution."""+ """Create a TraceContext for testing with lookup_key resolution, its execution named after its tracer key."""return TraceContext(graph_id=graph_id,tracer_key=tracer_key,@@ -77,6 +80,8 @@ def _make_trace_context(node_sequence=node_sequence,data_inclusion=DATA_INCLUSION_OFF,run_mode=run_mode,+ chain=chain or [ExecutionAttempt(execution=tracer_key or "direct", attempt=1)],+ parent_path=parent_path,)@@ -98,7 +103,6 @@ class TestReportingEventEmission:manager.set_event_log(context_key=self.PIPELINE_RUN_ID,event_log=event_log,- workflow_id=self.WORKFLOW_ID,pipeline_run_id=self.PIPELINE_RUN_ID,)return manager, event_log@@ -136,10 +140,13 @@ class TestReportingEventEmission:"""UsageReportEvent captures the parent node_id from graph context."""manager, event_log = self._make_reporting_manager_with_event_log()+ lineage = [ExecutionAttempt(execution="wf_rpt_run", attempt=1)]trace_context = _make_trace_context(graph_id=self.PIPELINE_RUN_ID,parent_node_id="test-graph:node_42",node_sequence=43,+ chain=[*lineage, ExecutionAttempt(execution=self.WORKFLOW_ID, attempt=2)],+ parent_path="flows.review/step_3/item_1/step_2",)llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, trace_context=trace_context)@@ -149,7 +156,9 @@ class TestReportingEventEmission:usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]assert len(usage_events) == 1assert usage_events[0].node_id == "test-graph:node_42"- assert usage_events[0].workflow_id == self.WORKFLOW_ID+ # The execution, its attempt and its lineage are the trace context's, and so is the path of the invocation+ assert (usage_events[0].workflow_id, usage_events[0].attempt, usage_events[0].lineage) == (self.WORKFLOW_ID, 2, lineage)+ assert usage_events[0].path == "flows.review/step_3/item_1/step_2"@pytest.mark.parametrize("emit_error",@@ -209,8 +218,8 @@ class TestReportingEventEmission:manager = ReportingManager()manager.setup()- manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")- manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")+ manager.set_event_log(context_key="wf_a", event_log=event_log_a, pipeline_run_id="run_a")+ manager.set_event_log(context_key="wf_b", event_log=event_log_b, pipeline_run_id="run_b")ctx_a = _make_trace_context(graph_id="run_a", tracer_key="wf_a")ctx_b = _make_trace_context(graph_id="run_b", tracer_key="wf_b")@@ -240,9 +249,9 @@ class TestReportingEventEmission:manager = ReportingManager()manager.setup()- manager.set_event_log(context_key="wf_a", event_log=stale_log, workflow_id="wf_a", pipeline_run_id="run_stale")+ manager.set_event_log(context_key="wf_a", event_log=stale_log, pipeline_run_id="run_stale")# Same key again — must not raise, must replace.- manager.set_event_log(context_key="wf_a", event_log=fresh_log, workflow_id="wf_a", pipeline_run_id="run_fresh")+ manager.set_event_log(context_key="wf_a", event_log=fresh_log, pipeline_run_id="run_fresh")ctx = _make_trace_context(graph_id="run_fresh", tracer_key="wf_a")manager.report_inference_job(_make_test_llm_job("run_fresh", trace_context=ctx))@@ -260,8 +269,8 @@ class TestReportingEventEmission:manager = ReportingManager()manager.setup()- manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")- manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")+ manager.set_event_log(context_key="wf_a", event_log=event_log_a, pipeline_run_id="run_a")+ manager.set_event_log(context_key="wf_b", event_log=event_log_b, pipeline_run_id="run_b")# Clear context A before any emissionmanager.clear_event_log(context_key="wf_a")@@ -310,8 +319,8 @@ class TestReportingEventEmission:manager = ReportingManager()manager.setup()- manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")- manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")+ manager.set_event_log(context_key="wf_a", event_log=event_log_a, pipeline_run_id="run_a")+ manager.set_event_log(context_key="wf_b", event_log=event_log_b, pipeline_run_id="run_b")ctx_a = _make_trace_context(graph_id="run_a", tracer_key="wf_a")ctx_b = _make_trace_context(graph_id="run_b", tracer_key="wf_b")diff --git a/tests/unit/pipelex/system/test_execution_chain.py b/tests/unit/pipelex/system/test_execution_chain.pynew file mode 100644index 000000000..9bcb4bd29--- /dev/null+++ b/tests/unit/pipelex/system/test_execution_chain.py@@ -0,0 +1,65 @@+import pytest++from pipelex.system.execution_chain import (+ DIRECT_EXECUTION,+ ExecutionAttempt,+ current_chain_keys,+ execution_chain_key,+ make_direct_chain,+)+++def _chain(*entries: tuple[str, int]) -> list[ExecutionAttempt]:+ return [ExecutionAttempt(execution=execution, attempt=attempt) for execution, attempt in entries]+++class TestExecutionChain:+ def test_the_direct_chain_is_one_execution_in_its_first_attempt(self) -> None:+ assert make_direct_chain() == [ExecutionAttempt(execution=DIRECT_EXECUTION, attempt=1)]++ def test_a_chain_key_tells_every_two_chains_apart(self) -> None:+ assert execution_chain_key(chain=[]) == ""+ assert execution_chain_key(chain=_chain(("exec-run", 1), ("exec-item-2", 3))) == "exec-run@1/exec-item-2@3"+ # An execution's name cannot forge a separator.+ assert execution_chain_key(chain=_chain(("a/b@1", 1))) != execution_chain_key(chain=_chain(("a", 1), ("b", 1)))+ assert execution_chain_key(chain=_chain(("a/b@1", 1))) == "a%2Fb%401@1"++ def test_an_attempt_and_its_lineage_are_validated(self) -> None:+ with pytest.raises(ValueError, match="greater than or equal to 1"):+ ExecutionAttempt(execution="exec-run", attempt=0)+ with pytest.raises(ValueError, match="at least 1 character"):+ ExecutionAttempt(execution="", attempt=1)++ def test_a_new_attempt_of_the_run_supersedes_everything_the_earlier_one_dispatched(self) -> None:+ """The worked example, run again: attempt 2 of the run's execution supersedes attempt 1 and every item it dispatched."""+ run_1 = _chain(("exec-run", 1))+ item_2_under_run_1 = _chain(("exec-run", 1), ("exec-item-2", 1))+ item_3_under_run_1 = _chain(("exec-run", 1), ("exec-item-3", 2))+ run_2 = _chain(("exec-run", 2))+ item_1_under_run_2 = _chain(("exec-run", 2), ("exec-item-1", 1))++ current = current_chain_keys(chains=[run_1, item_2_under_run_1, item_3_under_run_1, run_2, item_1_under_run_2, run_1])++ assert current == {execution_chain_key(chain=run_2), execution_chain_key(chain=item_1_under_run_2)}++ def test_a_retried_item_supersedes_only_its_own_earlier_attempts(self) -> None:+ run_1 = _chain(("exec-run", 1))+ item_1 = _chain(("exec-run", 1), ("exec-item-1", 1))+ item_2_first = _chain(("exec-run", 1), ("exec-item-2", 1))+ item_2_second = _chain(("exec-run", 1), ("exec-item-2", 2))+ nested_under_item_2_first = _chain(("exec-run", 1), ("exec-item-2", 1), ("exec-nested", 4))++ current = current_chain_keys(chains=[run_1, item_1, item_2_first, item_2_second, nested_under_item_2_first])++ assert current == {execution_chain_key(chain=chain) for chain in (run_1, item_1, item_2_second)}++ def test_attempts_are_compared_under_the_same_dispatching_attempts_only(self) -> None:+ """An item's attempt 1 under the run's attempt 2 is not superseded by its attempt 3 under the run's attempt 1."""+ item_under_run_1 = _chain(("exec-run", 1), ("exec-item", 3))+ run_2 = _chain(("exec-run", 2))+ item_under_run_2 = _chain(("exec-run", 2), ("exec-item", 1))++ assert current_chain_keys(chains=[item_under_run_1, run_2, item_under_run_2]) == {+ execution_chain_key(chain=run_2),+ execution_chain_key(chain=item_under_run_2),+ }diff --git a/tests/unit/pipelex/system/test_invocation_path.py b/tests/unit/pipelex/system/test_invocation_path.pynew file mode 100644index 000000000..82ea1b04f--- /dev/null+++ b/tests/unit/pipelex/system/test_invocation_path.py@@ -0,0 +1,53 @@+from collections.abc import Callable++import pytest++from pipelex.system.exceptions import InvocationPathError+from pipelex.system.invocation_path import (+ DEFAULT_OUTCOME_SEGMENT,+ branch_segment,+ child_path,+ item_segment,+ outcome_segment,+ step_segment,+)+++class TestInvocationPath:+ def test_ordinal_segments_count_from_one(self) -> None:+ assert (step_segment(step_number=3), branch_segment(branch_number=1), item_segment(item_number=12)) == ("step_3", "branch_1", "item_12")+ assert DEFAULT_OUTCOME_SEGMENT == "default"++ @pytest.mark.parametrize(+ ("build", "segment_kind"),+ [+ (lambda: step_segment(step_number=0), "step"),+ (lambda: branch_segment(branch_number=-1), "branch"),+ (lambda: item_segment(item_number=0), "item"),+ ],+ ids=["step_zero", "negative_branch", "item_zero"],+ )+ def test_an_ordinal_below_one_names_no_segment(self, build: Callable[[], str], segment_kind: str) -> None:+ with pytest.raises(InvocationPathError, match=f"'{segment_kind}' segment counts from 1"):+ build()++ @pytest.mark.parametrize(+ ("outcome_key", "expected_segment"),+ [+ ("detailed", "outcome_detailed"),+ ("high risk", "outcome_high%20risk"),+ ("legal/tax review", "outcome_legal%2Ftax%20review"),+ ("a-b.c_d~e", "outcome_a-b.c_d~e"),+ ("décision", "outcome_d%C3%A9cision"),+ ("50%", "outcome_50%25"),+ ],+ ids=["plain", "space", "slash", "unreserved_kept", "utf8_upper_hex", "percent"],+ )+ def test_an_outcome_key_is_percent_encoded(self, outcome_key: str, expected_segment: str) -> None:+ segment = outcome_segment(outcome_key=outcome_key)+ assert segment == expected_segment+ assert "/" not in segment++ def test_a_child_path_adds_one_segment(self) -> None:+ batch_path = child_path(parent_path="filings.review_filings", segment=step_segment(step_number=3))+ assert child_path(parent_path=batch_path, segment=item_segment(item_number=2)) == "filings.review_filings/step_3/item_2"diff --git a/tests/unit/pipelex/system/test_job_metadata_child_place.py b/tests/unit/pipelex/system/test_job_metadata_child_place.pynew file mode 100644index 000000000..00e064806--- /dev/null+++ b/tests/unit/pipelex/system/test_job_metadata_child_place.py@@ -0,0 +1,58 @@+import kajson++from pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.execution_chain import ExecutionAttempt+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.trace_context import TraceContext+++def _make_job_metadata(*, trace_context: TraceContext | None) -> JobMetadata:+ return JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_7Qd2kV"),+ trace_context=trace_context,+ )+++def _make_batch_context() -> TraceContext:+ """The context a batch at `filings.review_filings/step_3` hands the pipes it runs, in attempt 1 of the run."""+ return TraceContext(+ graph_id="run_7Qd2kV",+ parent_node_id="run_7Qd2kV:node_9",+ node_sequence=10,+ data_inclusion=DataInclusionConfig(stuff_json_content=False, error_stack_traces=False, pipe_and_concept_registry=False),+ run_mode=PipeRunMode.LIVE,+ parent_path="filings.review_filings/step_3",+ chain=[ExecutionAttempt(execution="exec-run", attempt=1)],+ )+++class TestJobMetadataChildPlace:+ def test_a_child_place_rides_on_a_copy_of_the_trace_context(self) -> None:+ batch_metadata = _make_job_metadata(trace_context=_make_batch_context())++ item_metadata = batch_metadata.for_child_at(segment="item_2")++ assert item_metadata.trace_context is not None+ assert (item_metadata.trace_context.parent_path, item_metadata.trace_context.child_segment) == ("filings.review_filings/step_3", "item_2")+ assert batch_metadata.trace_context is not None+ assert batch_metadata.trace_context.child_segment is None++ def test_without_a_trace_context_there_is_no_place_to_name(self) -> None:+ untraced_metadata = _make_job_metadata(trace_context=None)+ assert untraced_metadata.for_child_at(segment="item_2") is untraced_metadata++ def test_a_batch_items_metadata_read_back_keeps_its_path_prefix_and_lineage(self) -> None:+ """The item's metadata crosses to the execution running the item as a workflow's input does, and keeps its place."""+ item_metadata = _make_job_metadata(trace_context=_make_batch_context()).for_child_at(segment="item_2")++ for recovered in (JobMetadata.model_validate_json(item_metadata.model_dump_json()), kajson.loads(kajson.dumps(item_metadata))):+ assert isinstance(recovered, JobMetadata)+ recovered_context = recovered.trace_context+ assert recovered_context is not None+ assert (recovered_context.parent_path, recovered_context.child_segment, recovered_context.chain) == (+ "filings.review_filings/step_3",+ "item_2",+ [ExecutionAttempt(execution="exec-run", attempt=1)],+ )+ assert recovered_context.invocation_path(pipe_ref="filings.summarize_filing") == "filings.review_filings/step_3/item_2"diff --git a/tests/unit/pipelex/tracing/conftest.py b/tests/unit/pipelex/tracing/conftest.pyindex 1169e8392..497d3c9dc 100644--- a/tests/unit/pipelex/tracing/conftest.py+++ b/tests/unit/pipelex/tracing/conftest.py@@ -3,6 +3,7 @@from datetime import UTC, datetimefrom pipelex.graph.graphspec import EdgeKind, NodeKind+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.tracing.trace_events import EdgeEvent, PipeStartEvent@@ -11,9 +12,15 @@ def make_trace_event(workflow_id: str = "wf_abc",sequence: int = 0,pipeline_run_id: str = "run_001",+ attempt: int = 1,+ lineage: list[ExecutionAttempt] | None = None,) -> PipeStartEvent:"""Create a minimal PipeStartEvent for testing."""return PipeStartEvent(+ attempt=attempt,+ lineage=lineage or [],+ path=None,+ pipe_ref=None,pipeline_run_id=pipeline_run_id,workflow_id=workflow_id,timestamp=datetime(2025, 6, 15, 10, 30, 0, tzinfo=UTC),@@ -33,6 +40,9 @@ def make_edge_event() -> EdgeEvent:"""Create a minimal EdgeEvent for testing."""return EdgeEvent(+ attempt=1,+ lineage=[],+ path=None,pipeline_run_id=pipeline_run_id,workflow_id=workflow_id,timestamp=datetime(2025, 6, 15, 10, 30, 0, tzinfo=UTC),diff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.pyindex 9ccdd19c9..796363f13 100644--- a/tests/unit/pipelex/tracing/test_assembler_equivalence.py+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py@@ -5,7 +5,7 @@ Runs identical tracing scenarios through both paths:- Path B: GraphTracer with InMemoryEventLog → read events → GraphSpecAssembler.assemble() → GraphSpecThe two GraphSpecs must be structurally identical (same nodes, edges, statuses, data flow),-allowing for normalized IDs (workflow_id segment stripped) and ignored TimingSpec.+allowing for normalized IDs (chain segment stripped) and ignored TimingSpec."""import re@@ -43,14 +43,14 @@ _T0 = datetime(2025, 6, 15, 10, 0, 0, tzinfo=UTC)# Structural comparison helpers# ----------------------------------------------------------------------------# Pattern to strip the workflow_id segment from node/edge IDs:-# "graph_id:wf_xxx:node_0" → "graph_id:node_0"-_WF_SEGMENT_RE = re.compile(r":" + re.escape(_WORKFLOW_ID) + r":")+# Pattern to strip the chain segment, the workflow and its attempt, from node/edge IDs:+# "graph_id:wf_xxx:attempt_1:node_0" → "graph_id:node_0"+_CHAIN_SEGMENT_RE = re.compile(r":" + re.escape(_WORKFLOW_ID) + r":attempt_\d+(:lineage_[0-9a-f]+)?:")def _normalize_id(identifier: str) -> str:- """Strip the workflow_id segment from a node or edge ID."""- return _WF_SEGMENT_RE.sub(":", identifier)+ """Strip the chain segment from a node or edge ID."""+ return _CHAIN_SEGMENT_RE.sub(":", identifier)def _normalize_node(node: NodeSpec) -> dict[str, Any]:@@ -170,6 +170,7 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: TraceContext) -> Non# Parent controllerparent_id, child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.sequence",trace_context=context,pipe_code="sequence",pipe_type="PipeSequence",@@ -179,7 +180,8 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: TraceContext) -> Non# Child 1: produces digest_achild1_id, _ = tracer.on_pipe_start(- trace_context=child_ctx,+ pipe_ref="tests.gen_text",+ trace_context=child_ctx.with_child_segment(segment="step_1"),pipe_code="gen_text",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -193,7 +195,8 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: TraceContext) -> Non# Child 2: consumes digest_a, produces digest_bchild2_id, _ = tracer.on_pipe_start(- trace_context=child_ctx,+ pipe_ref="tests.refine",+ trace_context=child_ctx.with_child_segment(segment="step_2"),pipe_code="refine",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -208,7 +211,8 @@ def _scenario_simple_sequence(tracer: GraphTracer, context: TraceContext) -> Non# Child 3: consumes digest_bchild3_id, _ = tracer.on_pipe_start(- trace_context=child_ctx,+ pipe_ref="tests.format",+ trace_context=child_ctx.with_child_segment(segment="step_3"),pipe_code="format",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -233,6 +237,7 @@ def _scenario_parallel_branches(tracer: GraphTracer, context: TraceContext) -> Nstarted_at = _T0ctrl_id, ctrl_ctx = tracer.on_pipe_start(+ pipe_ref="tests.parallel",trace_context=context,pipe_code="parallel",pipe_type="PipeParallel",@@ -242,7 +247,8 @@ def _scenario_parallel_branches(tracer: GraphTracer, context: TraceContext) -> N# Branch Abranch_a_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.branch_a",+ trace_context=ctrl_ctx.with_child_segment(segment="step_1"),pipe_code="branch_a",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -256,7 +262,8 @@ def _scenario_parallel_branches(tracer: GraphTracer, context: TraceContext) -> N# Branch Bbranch_b_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.branch_b",+ trace_context=ctrl_ctx.with_child_segment(segment="step_2"),pipe_code="branch_b",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -298,6 +305,7 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: TraceContext) -started_at = _T0ctrl_id, ctrl_ctx = tracer.on_pipe_start(+ pipe_ref="tests.batch",trace_context=context,pipe_code="batch",pipe_type="PipeBatch",@@ -322,7 +330,8 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: TraceContext) -# Branch 0: processes item 0branch0_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.process",+ trace_context=ctrl_ctx.with_child_segment(segment="step_1"),pipe_code="process",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -337,7 +346,8 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: TraceContext) -# Branch 1: processes item 1branch1_id, _ = tracer.on_pipe_start(- trace_context=ctrl_ctx,+ pipe_ref="tests.process",+ trace_context=ctrl_ctx.with_child_segment(segment="step_2"),pipe_code="process",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -377,6 +387,7 @@ def _scenario_partial_failure(tracer: GraphTracer, context: TraceContext) -> Nonstarted_at = _T0_parent_id, child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.sequence",trace_context=context,pipe_code="sequence",pipe_type="PipeSequence",@@ -385,7 +396,8 @@ def _scenario_partial_failure(tracer: GraphTracer, context: TraceContext) -> Non)child_id, _ = tracer.on_pipe_start(- trace_context=child_ctx,+ pipe_ref="tests.gen_text",+ trace_context=child_ctx.with_child_segment(segment="step_1"),pipe_code="gen_text",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -407,6 +419,7 @@ def _scenario_pass_through(tracer: GraphTracer, context: TraceContext) -> None:# Pass-through pipe: input digest_a → output digest_a (same)pt_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.passthrough",trace_context=context,pipe_code="passthrough",pipe_type="PipeParallel",@@ -422,6 +435,7 @@ def _scenario_pass_through(tracer: GraphTracer, context: TraceContext) -> None:# Downstream pipe consumes digest_adown_id, _ = tracer.on_pipe_start(+ pipe_ref="tests.consumer",trace_context=context,pipe_code="consumer",pipe_type="PipeLLM",@@ -444,6 +458,7 @@ def _scenario_with_usage(tracer: GraphTracer, context: TraceContext) -> None:started_at = _T0parent_id, child_ctx = tracer.on_pipe_start(+ pipe_ref="tests.sequence",trace_context=context,pipe_code="sequence",pipe_type="PipeSequence",@@ -452,7 +467,8 @@ def _scenario_with_usage(tracer: GraphTracer, context: TraceContext) -> None:)child_id, _ = tracer.on_pipe_start(- trace_context=child_ctx,+ pipe_ref="tests.gen_text",+ trace_context=child_ctx.with_child_segment(segment="step_1"),pipe_code="gen_text",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -473,6 +489,9 @@ def _usage_events_for(*, assembled_child_node_id: str, event_log: InMemoryEventL"""One rated LLM call attributed to ``assembled_child_node_id``."""return [UsageReportEvent(+ attempt=1,+ lineage=[],+ path=None,pipeline_run_id=_PIPELINE_RUN_ID,workflow_id=_WORKFLOW_ID,writer_id=event_log.writer_id,@@ -501,6 +520,7 @@ def _scenario_condition_selected_outcome(tracer: GraphTracer, context: TraceContstarted_at = _T0cond_id, cond_ctx = tracer.on_pipe_start(+ pipe_ref="tests.check",trace_context=context,pipe_code="check",pipe_type="PipeCondition",@@ -510,7 +530,8 @@ def _scenario_condition_selected_outcome(tracer: GraphTracer, context: TraceConttracer.register_condition_resolution(condition_node_id=cond_id, rendered_value="true", selected_outcome="true")outcome_id, _ = tracer.on_pipe_start(- trace_context=cond_ctx,+ pipe_ref="tests.branch_true",+ trace_context=cond_ctx.with_child_segment(segment="step_1"),pipe_code="branch_true",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -536,6 +557,7 @@ def _scenario_decisions_at_decision_time(tracer: GraphTracer, context: TraceContstarted_at = _T0seq_id, seq_ctx = tracer.on_pipe_start(+ pipe_ref="tests.flow",trace_context=context,pipe_code="flow",pipe_type="PipeSequence",@@ -545,7 +567,8 @@ def _scenario_decisions_at_decision_time(tracer: GraphTracer, context: TraceCont)cond_id, cond_ctx = tracer.on_pipe_start(- trace_context=seq_ctx,+ pipe_ref="tests.route",+ trace_context=seq_ctx.with_child_segment(segment="step_1"),pipe_code="route",pipe_type="PipeCondition",node_kind=NodeKind.CONTROLLER,@@ -553,7 +576,8 @@ def _scenario_decisions_at_decision_time(tracer: GraphTracer, context: TraceCont)tracer.register_condition_resolution(condition_node_id=cond_id, rendered_value="unmatched", selected_outcome=None)outcome_id, _ = tracer.on_pipe_start(- trace_context=cond_ctx,+ pipe_ref="tests.fallback",+ trace_context=cond_ctx.with_child_segment(segment="step_1"),pipe_code="fallback",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,@@ -565,7 +589,8 @@ def _scenario_decisions_at_decision_time(tracer: GraphTracer, context: TraceConttracer.on_pipe_end_success(node_id=cond_id, ended_at=started_at + timedelta(seconds=4))batch_id, batch_ctx = tracer.on_pipe_start(- trace_context=seq_ctx,+ pipe_ref="tests.per_item",+ trace_context=seq_ctx.with_child_segment(segment="step_2"),pipe_code="per_item",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,@@ -579,7 +604,8 @@ def _scenario_decisions_at_decision_time(tracer: GraphTracer, context: TraceContlist_stuff_code="digest_items", item_stuff_code="digest_item_0", item_index=0, batch_controller_node_id=batch_id)item_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.process",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="process",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,@@ -603,6 +629,7 @@ def _scenario_dry_condition_merge(tracer: GraphTracer, context: TraceContext) ->started_at = _T0seq_id, seq_ctx = tracer.on_pipe_start(+ pipe_ref="tests.screen",trace_context=context,pipe_code="screen",pipe_type="PipeSequence",@@ -610,14 +637,16 @@ def _scenario_dry_condition_merge(tracer: GraphTracer, context: TraceContext) ->started_at=started_at,)flow_id, flow_ctx = tracer.on_pipe_start(- trace_context=seq_ctx,+ pipe_ref="tests.follow_up_flow",+ trace_context=seq_ctx.with_child_segment(segment="step_1"),pipe_code="follow_up_flow",pipe_type="PipeSequence",node_kind=NodeKind.CONTROLLER,started_at=started_at + timedelta(milliseconds=500),)cond_id, cond_ctx = tracer.on_pipe_start(- trace_context=flow_ctx,+ pipe_ref="tests.route",+ trace_context=flow_ctx.with_child_segment(segment="step_1"),pipe_code="route",pipe_type="PipeCondition",node_kind=NodeKind.CONTROLLER,@@ -626,14 +655,16 @@ def _scenario_dry_condition_merge(tracer: GraphTracer, context: TraceContext) ->)batch_id, batch_ctx = tracer.on_pipe_start(- trace_context=cond_ctx,+ pipe_ref="tests.write_questions",+ trace_context=cond_ctx.with_child_segment(segment="step_1"),pipe_code="write_questions",pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,started_at=started_at + timedelta(seconds=2),)item_id, _ = tracer.on_pipe_start(- trace_context=batch_ctx,+ pipe_ref="tests.write_question",+ trace_context=batch_ctx.with_child_segment(segment="step_1"),pipe_code="write_question",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -657,7 +688,8 @@ def _scenario_dry_condition_merge(tracer: GraphTracer, context: TraceContext) ->)rejection_id, _ = tracer.on_pipe_start(- trace_context=cond_ctx,+ pipe_ref="tests.write_rejection",+ trace_context=cond_ctx.with_child_segment(segment="step_2"),pipe_code="write_rejection",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -688,7 +720,8 @@ def _scenario_dry_condition_merge(tracer: GraphTracer, context: TraceContext) ->)assemble_id, _ = tracer.on_pipe_start(- trace_context=seq_ctx,+ pipe_ref="tests.assemble",+ trace_context=seq_ctx.with_child_segment(segment="step_2"),pipe_code="assemble",pipe_type="PipeLLM",node_kind=NodeKind.OPERATOR,@@ -766,7 +799,12 @@ class TestAssemblerEquivalence:run_mode=PipeRunMode.LIVE,)cond_id, cond_ctx = parent_tracer.on_pipe_start(- trace_context=parent_ctx, pipe_code="route", pipe_type="PipeCondition", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.route",+ trace_context=parent_ctx,+ pipe_code="route",+ pipe_type="PipeCondition",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,)parent_tracer.register_condition_resolution(condition_node_id=cond_id, rendered_value="urgent", selected_outcome="urgent")@@ -781,6 +819,7 @@ class TestAssemblerEquivalence:run_mode=cond_ctx.run_mode,).model_copy(update={"parent_node_id": cond_ctx.parent_node_id})chosen_id, _ = child_tracer.on_pipe_start(+ pipe_ref="tests.escalate",trace_context=child_ctx,pipe_code="escalate",pipe_type="PipeCompose",@@ -797,11 +836,17 @@ class TestAssemblerEquivalence:direct_tracer = GraphTracer()direct_ctx = direct_tracer.setup(graph_id=_GRAPH_ID, data_inclusion=data_inclusion, run_mode=PipeRunMode.LIVE)direct_cond_id, direct_cond_ctx = direct_tracer.on_pipe_start(- trace_context=direct_ctx, pipe_code="route", pipe_type="PipeCondition", node_kind=NodeKind.CONTROLLER, started_at=_T0+ pipe_ref="tests.route",+ trace_context=direct_ctx,+ pipe_code="route",+ pipe_type="PipeCondition",+ node_kind=NodeKind.CONTROLLER,+ started_at=_T0,)direct_tracer.register_condition_resolution(condition_node_id=direct_cond_id, rendered_value="urgent", selected_outcome="urgent")direct_chosen_id, _ = direct_tracer.on_pipe_start(- trace_context=direct_cond_ctx,+ pipe_ref="tests.escalate",+ trace_context=direct_cond_ctx.with_child_segment(segment="step_1"),pipe_code="escalate",pipe_type="PipeCompose",node_kind=NodeKind.OPERATOR,diff --git a/tests/unit/pipelex/tracing/test_assembler_supersession.py b/tests/unit/pipelex/tracing/test_assembler_supersession.pynew file mode 100644index 000000000..03743e900--- /dev/null+++ b/tests/unit/pipelex/tracing/test_assembler_supersession.py@@ -0,0 +1,213 @@+from datetime import UTC, datetime, timedelta+from typing import Any++import pytest++from pipelex.cogt.llm.llm_report import LLMTokensUsage+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.graph.graphspec import ErrorSpec, GraphSpec, NodeKind, NodeStatus+from pipelex.system.execution_chain import ExecutionAttempt+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler+from pipelex.tracing.trace_events import PipeEndErrorEvent, PipeEndSuccessEvent, PipeStartEvent, TraceEvent, UsageReportEvent++_GRAPH_ID = "run_7Qd2kV"+_T0 = datetime(2026, 10, 12, 9, 30, 0, tzinfo=UTC)+_RUN = "exec-run"+_ITEM = "exec-item-2"+_ENTRY_PATH = "filings.review_filings"+_BATCH_PATH = f"{_ENTRY_PATH}/step_3"+_ITEM_PATH = f"{_BATCH_PATH}/item_2"+_LEAF_PATH = f"{_ITEM_PATH}/step_2"+_ENTRY_NODE_ID = f"{_GRAPH_ID}:{_RUN}:node_0"+_BATCH_NODE_ID = f"{_GRAPH_ID}:{_RUN}:node_1"+++class _Writer:+ """Writes the events of one attempt of one execution, numbering them as its tracer does."""++ def __init__(self, *, execution: str, attempt: int, lineage: list[ExecutionAttempt]) -> None:+ self.execution = execution+ self.attempt = attempt+ self.lineage = lineage+ self.sequence = 0++ def _base(self, *, path: str) -> dict[str, Any]:+ base: dict[str, Any] = {+ "pipeline_run_id": _GRAPH_ID,+ "workflow_id": self.execution,+ "attempt": self.attempt,+ "lineage": self.lineage,+ "path": path,+ "timestamp": _T0 + timedelta(seconds=self.attempt * 100 + self.sequence),+ "sequence": self.sequence,+ "mode": PipeRunMode.LIVE,+ }+ self.sequence += 1+ return base++ def start(self, *, node_id: str, parent_node_id: str | None, path: str, pipe_code: str, pipe_type: str) -> PipeStartEvent:+ node_kind = NodeKind.CONTROLLER if pipe_type in {"PipeSequence", "PipeBatch"} else NodeKind.OPERATOR+ return PipeStartEvent(+ **self._base(path=path),+ node_id=node_id,+ parent_node_id=parent_node_id,+ pipe_code=pipe_code,+ pipe_ref=f"filings.{pipe_code}",+ pipe_type=pipe_type,+ node_kind=node_kind,+ )++ def succeed(self, *, node_id: str, path: str, pipe_code: str, pipe_type: str) -> PipeEndSuccessEvent:+ base = self._base(path=path)+ return PipeEndSuccessEvent(+ **base, node_id=node_id, pipe_code=pipe_code, pipe_ref=f"filings.{pipe_code}", pipe_type=pipe_type, ended_at=base["timestamp"]+ )++ def fail(self, *, node_id: str, path: str, pipe_code: str, pipe_type: str) -> PipeEndErrorEvent:+ base = self._base(path=path)+ return PipeEndErrorEvent(+ **base,+ node_id=node_id,+ pipe_code=pipe_code,+ pipe_ref=f"filings.{pipe_code}",+ pipe_type=pipe_type,+ ended_at=base["timestamp"],+ error=ErrorSpec(error_type="WorkerLostError", message="An internal error occurred."),+ )++ def usage(self, *, node_id: str, path: str) -> UsageReportEvent:+ return UsageReportEvent(+ **self._base(path=path),+ node_id=node_id,+ tokens_usage=LLMTokensUsage(+ job_metadata=JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user_test", pipeline_run_id=_GRAPH_ID),+ pipe_code="condense_filing",+ unit_job_id=UnitJobId.LLM_GEN_TEXT,+ job_category=JobCategory.LLM_JOB,+ ),+ inference_model_name="test-model",+ inference_model_id="test-model-id",+ unit_costs={CostCategory.INPUT: 3.0, CostCategory.OUTPUT: 15.0},+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},+ ),+ )+++def _item_attempt_events(*, attempt: int, run_attempt: int, node_prefix: str, is_lost: bool) -> list[TraceEvent]:+ """One attempt of item 2's execution: the item's sequence, the step that condenses its filing, and that step's usage."""+ writer = _Writer(execution=_ITEM, attempt=attempt, lineage=[ExecutionAttempt(execution=_RUN, attempt=run_attempt)])+ item_node_id = f"{node_prefix}:node_0"+ leaf_node_id = f"{node_prefix}:node_1"+ events: list[TraceEvent] = [+ writer.start(node_id=item_node_id, parent_node_id=_BATCH_NODE_ID, path=_ITEM_PATH, pipe_code="summarize_filing", pipe_type="PipeSequence"),+ writer.start(node_id=leaf_node_id, parent_node_id=item_node_id, path=_LEAF_PATH, pipe_code="condense_filing", pipe_type="PipeLLM"),+ writer.usage(node_id=leaf_node_id, path=_ITEM_PATH),+ ]+ if is_lost:+ events.append(writer.fail(node_id=leaf_node_id, path=_LEAF_PATH, pipe_code="condense_filing", pipe_type="PipeLLM"))+ events.append(writer.fail(node_id=item_node_id, path=_ITEM_PATH, pipe_code="summarize_filing", pipe_type="PipeSequence"))+ else:+ events.append(writer.succeed(node_id=leaf_node_id, path=_LEAF_PATH, pipe_code="condense_filing", pipe_type="PipeLLM"))+ events.append(writer.succeed(node_id=item_node_id, path=_ITEM_PATH, pipe_code="summarize_filing", pipe_type="PipeSequence"))+ return events+++def _run_attempt_events(*, attempt: int, is_ended: bool) -> list[TraceEvent]:+ """One attempt of the run's own execution: the entry sequence and its batch."""+ writer = _Writer(execution=_RUN, attempt=attempt, lineage=[])+ events: list[TraceEvent] = [+ writer.start(node_id=_ENTRY_NODE_ID, parent_node_id=None, path=_ENTRY_PATH, pipe_code="review_filings", pipe_type="PipeSequence"),+ writer.start(+ node_id=_BATCH_NODE_ID, parent_node_id=_ENTRY_NODE_ID, path=_BATCH_PATH, pipe_code="summarize_filing_batch", pipe_type="PipeBatch"+ ),+ ]+ if is_ended:+ events.append(writer.succeed(node_id=_BATCH_NODE_ID, path=_BATCH_PATH, pipe_code="summarize_filing_batch", pipe_type="PipeBatch"))+ events.append(writer.succeed(node_id=_ENTRY_NODE_ID, path=_ENTRY_PATH, pipe_code="review_filings", pipe_type="PipeSequence"))+ return events+++def _nodes_by_path(graph_spec: GraphSpec) -> dict[str, list[NodeStatus]]:+ nodes_by_path: dict[str, list[NodeStatus]] = {}+ for node in graph_spec.nodes:+ assert node.path is not None+ nodes_by_path.setdefault(node.path, []).append(node.status)+ return nodes_by_path+++class TestAssemblerSupersession:+ @pytest.mark.parametrize(+ "lost_node_prefix",+ [f"{_GRAPH_ID}:{_ITEM}", f"{_GRAPH_ID}:{_ITEM}-first-run"],+ ids=["attempts_share_node_ids", "attempts_mint_their_own_node_ids"],+ )+ def test_a_retried_item_shows_its_current_attempt_alone(self, lost_node_prefix: str) -> None:+ """Both attempts of the item name one invocation at each path; the graph keeps the later, with no phantom failure."""+ events = [+ *_run_attempt_events(attempt=1, is_ended=True),+ *_item_attempt_events(attempt=1, run_attempt=1, node_prefix=lost_node_prefix, is_lost=True),+ *_item_attempt_events(attempt=2, run_attempt=1, node_prefix=f"{_GRAPH_ID}:{_ITEM}", is_lost=False),+ ]++ graph_spec = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)++ assert _nodes_by_path(graph_spec) == {+ _ENTRY_PATH: [NodeStatus.SUCCEEDED],+ _BATCH_PATH: [NodeStatus.SUCCEEDED],+ _ITEM_PATH: [NodeStatus.SUCCEEDED],+ _LEAF_PATH: [NodeStatus.SUCCEEDED],+ }+ # The lost attempt's inference was spent all the same: the run's total counts it, as usage no node shown holds.+ assert graph_spec.usage is not None+ assert (graph_spec.usage.total.inference_calls, graph_spec.usage.unattributed.inference_calls) == (2, 1)+ leaf_node = next(node for node in graph_spec.nodes if node.path == _LEAF_PATH)+ assert leaf_node.usage is not None+ assert leaf_node.usage.inference_calls == 1++ def test_a_new_attempt_of_the_run_supersedes_every_item_the_earlier_one_dispatched(self) -> None:+ """Attempt 2 of the run's execution runs the batch again: the item attempt 1 dispatched is gone, though it ended well."""+ events = [+ *_run_attempt_events(attempt=1, is_ended=False),+ *_item_attempt_events(attempt=1, run_attempt=1, node_prefix=f"{_GRAPH_ID}:{_ITEM}-under-run-1", is_lost=False),+ *_run_attempt_events(attempt=2, is_ended=True),+ *_item_attempt_events(attempt=1, run_attempt=2, node_prefix=f"{_GRAPH_ID}:{_ITEM}", is_lost=False),+ ]++ graph_spec = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)++ assert _nodes_by_path(graph_spec) == {+ _ENTRY_PATH: [NodeStatus.SUCCEEDED],+ _BATCH_PATH: [NodeStatus.SUCCEEDED],+ _ITEM_PATH: [NodeStatus.SUCCEEDED],+ _LEAF_PATH: [NodeStatus.SUCCEEDED],+ }+ assert {node.node_id for node in graph_spec.nodes if node.path == _LEAF_PATH} == {f"{_GRAPH_ID}:{_ITEM}:node_1"}+ assert graph_spec.usage is not None+ assert (graph_spec.usage.total.inference_calls, graph_spec.usage.unattributed.inference_calls) == (2, 1)++ def test_a_superseded_attempt_does_not_date_the_graph(self) -> None:+ """The graph is dated by its earliest current event: the run's first attempt, lost an hour before the second+ started, is no part of the graph, so it does not date it either.+ """+ one_hour = timedelta(hours=1)+ second_attempt_events = [+ event.model_copy(update={"timestamp": event.timestamp + one_hour})+ for event in [+ *_run_attempt_events(attempt=2, is_ended=True),+ *_item_attempt_events(attempt=1, run_attempt=2, node_prefix=f"{_GRAPH_ID}:{_ITEM}", is_lost=False),+ ]+ ]+ first_attempt_events = [+ *_run_attempt_events(attempt=1, is_ended=False),+ *_item_attempt_events(attempt=1, run_attempt=1, node_prefix=f"{_GRAPH_ID}:{_ITEM}-under-run-1", is_lost=True),+ ]++ graph_spec = GraphSpecAssembler.assemble(events=[*first_attempt_events, *second_attempt_events], graph_id=_GRAPH_ID)++ second_attempt_start = min(event.timestamp for event in second_attempt_events)+ assert second_attempt_start - max(event.timestamp for event in first_attempt_events) > one_hour - timedelta(minutes=5)+ assert graph_spec.created_at == second_attempt_startdiff --git a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.pyindex 3fa374ece..7cca1b83e 100644--- a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py+++ b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py@@ -38,6 +38,9 @@ def _base(sequence: int) -> dict[str, Any]:return {"pipeline_run_id": _PIPELINE_RUN_ID,"workflow_id": _WORKFLOW_ID,+ "attempt": 1,+ "lineage": [],+ "path": None,"timestamp": _T0 + timedelta(seconds=sequence),"sequence": sequence,"mode": PipeRunMode.LIVE,@@ -61,6 +64,7 @@ def _make_usage(*, unit_costs: CostsByCategoryDict, model_name: str = "test-modedef _pipe_start(*, sequence: int, node_id: str, parent_node_id: str | None, node_kind: NodeKind, pipe_type: str) -> PipeStartEvent:return PipeStartEvent(+ pipe_ref=None,**_base(sequence),node_id=node_id,parent_node_id=parent_node_id,@@ -71,7 +75,9 @@ def _pipe_start(*, sequence: int, node_id: str, parent_node_id: str | None, nodedef _pipe_end(*, sequence: int, node_id: str) -> PipeEndSuccessEvent:- return PipeEndSuccessEvent(**_base(sequence), node_id=node_id, ended_at=_T0 + timedelta(seconds=sequence))+ return PipeEndSuccessEvent(+ pipe_code="test_pipe", pipe_ref=None, pipe_type="PipeFunc", **_base(sequence), node_id=node_id, ended_at=_T0 + timedelta(seconds=sequence)+ )def _usage_event(*, sequence: int, node_id: str, unit_costs: CostsByCategoryDict, model_name: str = "test-model") -> UsageReportEvent:diff --git a/tests/unit/pipelex/tracing/test_dynamodb_event_log.py b/tests/unit/pipelex/tracing/test_dynamodb_event_log.pyindex 05122105f..727d7a84e 100644--- a/tests/unit/pipelex/tracing/test_dynamodb_event_log.py+++ b/tests/unit/pipelex/tracing/test_dynamodb_event_log.py@@ -10,24 +10,38 @@ from pytest_mock import MockerFixturefrom pipelex.graph.graphspec import NodeKindfrom pipelex.system.configuration.configs import DynamoDBTracingConfig, TracingConfig+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.tracing.buffering_event_log import BufferingEventLogfrom pipelex.tracing.dynamodb_event_log import DynamoDBEventLogfrom pipelex.tracing.event_log_factory import make_event_logfrom pipelex.tracing.event_log_protocol import EventLogProtocolfrom pipelex.tracing.exceptions import EventLogReadError, EventLogSchemaMismatchError, EventLogSetupError+from pipelex.tracing.in_memory_event_log import InMemoryEventLogfrom pipelex.tracing.trace_events import BatchFannedOutEvent, ConditionResolvedEvent, PipeStartEvent, TraceEventfrom tests.unit.pipelex.tracing.conftest import make_trace_event+# DynamoDB refuses a sort key longer than this many bytes.+_DYNAMODB_MAX_SORT_KEY_BYTES = 1024+_SECONDS_PER_DAY = 86_400_RUN_ID = "plr-decisions"_T0 = datetime(2026, 10, 13, 9, 0, 0, tzinfo=UTC)def _decision_events(*, mode: PipeRunMode) -> list[TraceEvent]:- base: dict[str, Any] = {"pipeline_run_id": _RUN_ID, "workflow_id": "wf_flow", "timestamp": _T0, "mode": mode}+ base: dict[str, Any] = {+ "pipeline_run_id": _RUN_ID,+ "workflow_id": "wf_flow",+ "attempt": 1,+ "lineage": [],+ "path": None,+ "timestamp": _T0,+ "mode": mode,+ }return [PipeStartEvent(+ pipe_ref=None,**base,sequence=0,node_id="g:wf_flow:node_0",@@ -49,6 +63,39 @@ def _decision_events(*, mode: PipeRunMode) -> list[TraceEvent]:]+def _chain_event(*, lineage: list[ExecutionAttempt], workflow_id: str, attempt: int, sequence: int, writer_id: str = "primary") -> PipeStartEvent:+ return PipeStartEvent(+ pipeline_run_id=_RUN_ID,+ workflow_id=workflow_id,+ attempt=attempt,+ lineage=lineage,+ path=None,+ writer_id=writer_id,+ timestamp=_T0,+ mode=PipeRunMode.LIVE,+ sequence=sequence,+ pipe_ref=None,+ node_id=f"g:{workflow_id}:node_{sequence}",+ pipe_code="step",+ pipe_type="PipeCompose",+ node_kind=NodeKind.OPERATOR,+ )+++class _FakeEventsTable:+ """A DynamoDB table holding one item per (PK, SK), which a query returns in sort-key order, as DynamoDB does."""++ def __init__(self) -> None:+ self.items: dict[tuple[str, str], dict[str, Any]] = {}++ def put_item(self, *, Item: dict[str, Any]) -> None: # ruff: ignore[invalid-argument-name] # boto3 kwarg name+ self.items[Item["PK"], Item["SK"]] = Item++ def query(self, *, KeyConditionExpression: Any, **_: Any) -> dict[str, Any]: # ruff: ignore[invalid-argument-name] # boto3 kwarg name+ partition_key = KeyConditionExpression.get_expression()["values"][1]+ return {"Items": [item for (pk, _sk), item in sorted(self.items.items()) if pk == partition_key]}++class TestDynamoDBEventLog:def _make_event_log_with_table(self, mocker: MockerFixture, *, retention_days: int = 30) -> tuple[DynamoDBEventLog, Any]:# boto3.resource(...) does not perform any network call at construction.@@ -169,6 +216,69 @@ class TestDynamoDBEventLog:with pytest.raises(EventLogSchemaMismatchError):event_log.read_events(_RUN_ID)+ def _make_event_log_with_fake_table(self, mocker: MockerFixture) -> tuple[DynamoDBEventLog, _FakeEventsTable]:+ event_log = DynamoDBEventLog(table_name="trace-events-test", region="us-east-1", retention_days=30)+ fake_table = _FakeEventsTable()+ mocker.patch.object(event_log, "_table", new=fake_table)+ return event_log, fake_table++ def test_the_sort_key_stays_bounded_however_deep_and_long_the_lineage(self, mocker: MockerFixture) -> None:+ """Nested batches dispatched under long workflow ids give a chain key far past DynamoDB's sort-key limit: the sort+ key carries a digest of the chain, so its length does not grow with it, while a replayed write still overwrites+ its own row and every attempt keeps its own.+ """+ event_log, fake_table = self._make_event_log_with_fake_table(mocker)+ lineage = [ExecutionAttempt(execution=f"run-{depth:02d}-" + "x" * 200, attempt=depth + 1) for depth in range(12)]+ first_attempt = _chain_event(lineage=lineage, workflow_id="item-" + "y" * 300, attempt=1, sequence=7, writer_id="act_4242_abcdef12")+ second_attempt = first_attempt.model_copy(update={"attempt": 2})+ assert len(first_attempt.chain_key.encode()) > 2 * _DYNAMODB_MAX_SORT_KEY_BYTES++ event_log.emit(first_attempt)+ event_log.emit(first_attempt)+ event_log.emit(second_attempt)++ sort_keys = [sort_key for (_partition_key, sort_key) in fake_table.items]+ assert len(sort_keys) == 2+ for sort_key in sort_keys:+ assert len(sort_key.encode()) <= _DYNAMODB_MAX_SORT_KEY_BYTES+ assert sort_key.startswith("EVENT#")+ assert sort_key.endswith("#act_4242_abcdef12#0000000007")+ # The chain itself is read back whole from the row's payload.+ assert event_log.read_events(_RUN_ID) == [first_attempt, second_attempt]++ def test_read_events_dedupes_and_orders_as_every_event_log_does(self, mocker: MockerFixture) -> None:+ """The sort keys order the rows by the digest of their chains, which is no order of the run: the read returns+ the events deduplicated and ordered by (chain, sequence, writer), as the in-memory and NDJSON logs do, and a+ row written twice under two sort keys, as before and after the sort key changed shape, is read once.+ """+ event_log, fake_table = self._make_event_log_with_fake_table(mocker)+ run_attempt = ExecutionAttempt(execution="exec-run", attempt=1)+ events = [+ _chain_event(lineage=[run_attempt], workflow_id="exec-item-2", attempt=1, sequence=0),+ _chain_event(lineage=[], workflow_id="exec-run", attempt=1, sequence=1, writer_id="act_9_0000"),+ _chain_event(lineage=[run_attempt], workflow_id="exec-item-1", attempt=1, sequence=0),+ _chain_event(lineage=[], workflow_id="exec-run", attempt=1, sequence=0),+ _chain_event(lineage=[], workflow_id="exec-run", attempt=1, sequence=2),+ ]+ for event in events:+ event_log.emit(event)+ legacy_row = {**fake_table.items[next(iter(fake_table.items))], "SK": "EVENT#exec-run@1/exec-item-2@1#primary#0000000000"}+ fake_table.put_item(Item=legacy_row)+ reference_log = InMemoryEventLog()+ for event in events:+ reference_log.emit(event)++ read_back = event_log.read_events(_RUN_ID)++ assert [(event.chain_key, event.sequence, event.writer_id) for event in read_back] == [+ ("exec-run@1", 0, "primary"),+ ("exec-run@1", 1, "act_9_0000"),+ ("exec-run@1", 2, "primary"),+ ("exec-run@1/exec-item-1@1", 0, "primary"),+ ("exec-run@1/exec-item-2@1", 0, "primary"),+ ]+ assert read_back == reference_log.read_events(_RUN_ID)+def test_init_translates_construction_error(self, mocker: MockerFixture) -> None:# A botocore failure while building the boto3 client (e.g. a misconfigured region) must surface as# the domain EventLogSetupError so best-effort callers (tracing assembly) degrade rather than abort.diff --git a/tests/unit/pipelex/tracing/test_graphspec_assembler.py b/tests/unit/pipelex/tracing/test_graphspec_assembler.pyindex c243fd638..dca19d52e 100644--- a/tests/unit/pipelex/tracing/test_graphspec_assembler.py+++ b/tests/unit/pipelex/tracing/test_graphspec_assembler.py@@ -51,6 +51,9 @@ def _base(workflow_id: str, sequence: int, timestamp_offset: int = 0, run_mode:return {"pipeline_run_id": _PIPELINE_RUN_ID,"workflow_id": workflow_id,+ "attempt": 1,+ "lineage": [],+ "path": None,"timestamp": _time_at(timestamp_offset or sequence),"sequence": sequence,"mode": run_mode,@@ -79,9 +82,21 @@ class TestGraphSpecAssembler:result = GraphSpecAssembler.assemble(events=[PipeStartEvent(- **_base(_WF_A, 0, run_mode=run_mode), node_id=node_id, pipe_code="only", pipe_type="PipeLLM", node_kind=NodeKind.OPERATOR+ pipe_ref=None,+ **_base(_WF_A, 0, run_mode=run_mode),+ node_id=node_id,+ pipe_code="only",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ ),+ PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",+ **_base(_WF_A, 1, run_mode=run_mode),+ node_id=node_id,+ ended_at=_time_at(1),),- PipeEndSuccessEvent(**_base(_WF_A, 1, run_mode=run_mode), node_id=node_id, ended_at=_time_at(1)),],graph_id=_GRAPH_ID,)@@ -100,6 +115,7 @@ class TestGraphSpecAssembler:"""A set of events that mixes modes holds more than one run, and an empty one states no mode: neither gets a graph."""events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, index_event, run_mode=run_mode),node_id=_node_id(_WF_A, index_event),pipe_code=f"pipe_{index_event}",@@ -122,6 +138,7 @@ class TestGraphSpecAssembler:result = GraphSpecAssembler.assemble(events=[PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=node_id,pipe_code="sourceful_pipe",@@ -143,6 +160,9 @@ class TestGraphSpecAssembler:],),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=node_id,ended_at=_time_at(1),@@ -171,6 +191,7 @@ class TestGraphSpecAssembler:result = GraphSpecAssembler.assemble(events=[PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=producer_node,pipe_code="lifted_plural_pipe",@@ -178,6 +199,9 @@ class TestGraphSpecAssembler:node_kind=NodeKind.OPERATOR,),PipeEndSkippedEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=producer_node,ended_at=_time_at(1),@@ -185,6 +209,7 @@ class TestGraphSpecAssembler:output_spec=_io_spec("items", "stuff-empty-list"),),PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 2),node_id=consumer_node,pipe_code="items_consumer",@@ -193,6 +218,9 @@ class TestGraphSpecAssembler:input_specs=[_io_spec("items", "stuff-empty-list")],),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 3),node_id=consumer_node,ended_at=_time_at(3),@@ -226,6 +254,7 @@ class TestGraphSpecAssembler:events = [# Parent startsPipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=parent,pipe_code="sequence",@@ -234,6 +263,7 @@ class TestGraphSpecAssembler:),# Child 1 startsPipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 1),node_id=child_1,parent_node_id=parent,@@ -251,6 +281,9 @@ class TestGraphSpecAssembler:),# Child 1 ends with output digest_aPipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 3),node_id=child_1,ended_at=_time_at(3),@@ -258,6 +291,7 @@ class TestGraphSpecAssembler:),# Child 2 starts with input digest_aPipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 4),node_id=child_2,parent_node_id=parent,@@ -276,6 +310,9 @@ class TestGraphSpecAssembler:),# Child 2 ends with output digest_bPipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 6),node_id=child_2,ended_at=_time_at(6),@@ -283,6 +320,7 @@ class TestGraphSpecAssembler:),# Child 3 starts with input digest_bPipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 7),node_id=child_3,parent_node_id=parent,@@ -301,6 +339,9 @@ class TestGraphSpecAssembler:),# Child 3 endsPipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 9),node_id=child_3,ended_at=_time_at(9),@@ -308,6 +349,9 @@ class TestGraphSpecAssembler:),# Parent endsPipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 10),node_id=parent,ended_at=_time_at(10),@@ -341,6 +385,7 @@ class TestGraphSpecAssembler:events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=controller,pipe_code="parallel",@@ -348,6 +393,7 @@ class TestGraphSpecAssembler:node_kind=NodeKind.CONTROLLER,),PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 1),node_id=branch_1,parent_node_id=controller,@@ -363,12 +409,16 @@ class TestGraphSpecAssembler:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 3),node_id=branch_1,ended_at=_time_at(3),output_spec=_io_spec("branch_out", "digest_branch_1"),),PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 4),node_id=branch_2,parent_node_id=controller,@@ -384,6 +434,9 @@ class TestGraphSpecAssembler:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 6),node_id=branch_2,ended_at=_time_at(6),@@ -407,6 +460,9 @@ class TestGraphSpecAssembler:output_spec=_io_spec("combined_out", "digest_combined"),),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 9),node_id=controller,ended_at=_time_at(9),@@ -430,6 +486,7 @@ class TestGraphSpecAssembler:events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=controller,pipe_code="batch",@@ -454,6 +511,7 @@ class TestGraphSpecAssembler:),# Branch 0 processes item 0PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 3),node_id=branch_0,parent_node_id=controller,@@ -470,6 +528,9 @@ class TestGraphSpecAssembler:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 5),node_id=branch_0,ended_at=_time_at(5),@@ -477,6 +538,7 @@ class TestGraphSpecAssembler:),# Branch 1 processes item 1PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 6),node_id=branch_1,parent_node_id=controller,@@ -493,6 +555,9 @@ class TestGraphSpecAssembler:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 8),node_id=branch_1,ended_at=_time_at(8),@@ -515,6 +580,9 @@ class TestGraphSpecAssembler:),# Controller endsPipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 11),node_id=controller,ended_at=_time_at(11),@@ -550,6 +618,7 @@ class TestGraphSpecAssembler:events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=outcome,parent_node_id=condition,@@ -565,11 +634,15 @@ class TestGraphSpecAssembler:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 2),node_id=outcome,ended_at=_time_at(4),),PipeStartEvent(+ pipe_ref=None,**_base(_WF_B, 0),node_id=condition,pipe_code="check",@@ -591,6 +664,9 @@ class TestGraphSpecAssembler:label="true",),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_B, 3),node_id=condition,ended_at=_time_at(5),@@ -609,6 +685,7 @@ class TestGraphSpecAssembler:events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=outcome,parent_node_id=absent_condition,@@ -617,6 +694,9 @@ class TestGraphSpecAssembler:node_kind=NodeKind.OPERATOR,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=outcome,ended_at=_time_at(4),@@ -641,6 +721,7 @@ class TestGraphSpecAssembler:events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=parent,pipe_code="sequence",@@ -648,6 +729,7 @@ class TestGraphSpecAssembler:node_kind=NodeKind.CONTROLLER,),PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 1),node_id=child,parent_node_id=parent,@@ -663,6 +745,9 @@ class TestGraphSpecAssembler:edge_kind=EdgeKind.CONTAINS,),PipeEndErrorEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 3),node_id=child,ended_at=_time_at(3),@@ -691,6 +776,7 @@ class TestGraphSpecAssembler:events = [# wf_a events (sorted first alphabetically)PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=node_a0,pipe_code="pipe_a",@@ -698,6 +784,9 @@ class TestGraphSpecAssembler:node_kind=NodeKind.OPERATOR,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=node_a0,ended_at=_time_at(1),@@ -705,6 +794,7 @@ class TestGraphSpecAssembler:),# wf_b events (sorted second)PipeStartEvent(+ pipe_ref=None,**_base(_WF_B, 0),node_id=node_b0,pipe_code="pipe_b",@@ -713,6 +803,9 @@ class TestGraphSpecAssembler:input_specs=[_io_spec("input", "digest_from_a")],),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_B, 1),node_id=node_b0,ended_at=_time_at(3),@@ -741,6 +834,7 @@ class TestGraphSpecAssembler:events = [# wf_a: consumer (sorted first)PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=consumer,pipe_code="consumer_pipe",@@ -749,12 +843,16 @@ class TestGraphSpecAssembler:input_specs=[_io_spec("input", "digest_x")],),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=consumer,ended_at=_time_at(5),),# wf_b: producer (sorted second)PipeStartEvent(+ pipe_ref=None,**_base(_WF_B, 0),node_id=producer,pipe_code="producer_pipe",@@ -762,6 +860,9 @@ class TestGraphSpecAssembler:node_kind=NodeKind.OPERATOR,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_B, 1),node_id=producer,ended_at=_time_at(3),@@ -784,6 +885,7 @@ class TestGraphSpecAssembler:events = [# Pass-through pipe: input digest_a → output digest_a (same)PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=passthrough,pipe_code="passthrough",@@ -792,6 +894,9 @@ class TestGraphSpecAssembler:input_specs=[_io_spec("main_input", "digest_a")],),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=passthrough,ended_at=_time_at(1),@@ -799,6 +904,7 @@ class TestGraphSpecAssembler:),# Downstream pipe consumes digest_aPipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 2),node_id=downstream,pipe_code="consumer",@@ -807,6 +913,9 @@ class TestGraphSpecAssembler:input_specs=[_io_spec("input", "digest_a")],),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 3),node_id=downstream,ended_at=_time_at(3),@@ -827,6 +936,7 @@ class TestGraphSpecAssembler:node = _node_id(_WF_A, 0)events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=node,pipe_code="summarize",@@ -836,6 +946,9 @@ class TestGraphSpecAssembler:domain_code="summarization",),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=node,ended_at=_time_at(1),@@ -853,6 +966,7 @@ class TestGraphSpecAssembler:node = _node_id(_WF_A, 0)events = [PipeStartEvent(+ pipe_ref=None,**_base(_WF_A, 0),node_id=node,pipe_code="noop",@@ -860,6 +974,9 @@ class TestGraphSpecAssembler:node_kind=NodeKind.OPERATOR,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(_WF_A, 1),node_id=node,ended_at=_time_at(1),diff --git a/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.py b/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.pyindex cf8b2bdc3..0e9b68118 100644--- a/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.py+++ b/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.py@@ -34,6 +34,9 @@ def _base(sequence: int) -> dict[str, Any]:return {"pipeline_run_id": _PIPELINE_RUN_ID,"workflow_id": _WF_ID,+ "attempt": 1,+ "lineage": [],+ "path": None,"timestamp": _time_at(sequence),"sequence": sequence,"mode": PipeRunMode.LIVE,@@ -48,6 +51,7 @@ def _make_sequence_events() -> list[TraceEvent]:return [PipeStartEvent(+ pipe_ref=None,**_base(0),node_id=parent,pipe_code="sequence",@@ -55,6 +59,7 @@ def _make_sequence_events() -> list[TraceEvent]:node_kind=NodeKind.CONTROLLER,),PipeStartEvent(+ pipe_ref=None,**_base(1),node_id=child_1,parent_node_id=parent,@@ -70,12 +75,16 @@ def _make_sequence_events() -> list[TraceEvent]:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(3),node_id=child_1,ended_at=_time_at(3),output_spec=IOSpec(name="output", digest="digest_a"),),PipeStartEvent(+ pipe_ref=None,**_base(4),node_id=child_2,parent_node_id=parent,@@ -92,12 +101,18 @@ def _make_sequence_events() -> list[TraceEvent]:edge_kind=EdgeKind.CONTAINS,),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(6),node_id=child_2,ended_at=_time_at(6),output_spec=IOSpec(name="output", digest="digest_b"),),PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_base(7),node_id=parent,ended_at=_time_at(7),diff --git a/tests/unit/pipelex/tracing/test_ndjson_event_log.py b/tests/unit/pipelex/tracing/test_ndjson_event_log.pyindex a07dc0495..fd8d16d59 100644--- a/tests/unit/pipelex/tracing/test_ndjson_event_log.py+++ b/tests/unit/pipelex/tracing/test_ndjson_event_log.py@@ -6,6 +6,7 @@ from pathlib import Pathimport pytest+from pipelex.system.execution_chain import ExecutionAttempt, execution_chain_digestfrom pipelex.tracing.exceptions import EventLogSchemaMismatchErrorfrom pipelex.tracing.ndjson_event_log import NdjsonEventLogfrom pipelex.tracing.trace_events import EdgeEvent, PipeStartEvent@@ -25,6 +26,12 @@ def _worker_fn(traces_dir: str, pipeline_run_id: str, workflow_id: str, num_evenevent_log.close()+def _only_run_file(run_dir: Path) -> Path:+ """The one file a run written by one attempt of one execution, through one writer, holds."""+ (ndjson_file,) = run_dir.glob("*.ndjson")+ return ndjson_file++class TestNdjsonEventLog:"""Tests for the NDJSON file-based event log backend."""@@ -45,7 +52,7 @@ class TestNdjsonEventLog:assert read_event.pipe_code == "test_pipe"def test_deduplication(self, tmp_path: Path) -> None:- """Duplicate (workflow_id, event_kind, sequence) triples are deduplicated."""+ """Events of one chain, writer, kind and sequence are one event written twice, and are deduplicated."""event_log = NdjsonEventLog(traces_dir=str(tmp_path))event_log.emit(make_trace_event(sequence=0))event_log.emit(make_trace_event(sequence=0))@@ -67,6 +74,22 @@ class TestNdjsonEventLog:assert PipeStartEvent in event_kindsassert EdgeEvent in event_kinds+ def test_each_attempt_keeps_its_own_events(self, tmp_path: Path) -> None:+ """Attempts of one execution number their events from the same start, and every attempt's events are kept.++ They are read back attempt by attempt, an execution's earlier attempt first, and an execution dispatched under+ another attempt of its dispatcher is another chain.+ """+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))+ event_log.emit(make_trace_event(workflow_id="exec-item", sequence=0, attempt=2, lineage=[ExecutionAttempt(execution="exec-run", attempt=1)]))+ event_log.emit(make_trace_event(workflow_id="exec-item", sequence=0, attempt=1, lineage=[ExecutionAttempt(execution="exec-run", attempt=1)]))+ event_log.emit(make_trace_event(workflow_id="exec-item", sequence=0, attempt=1, lineage=[ExecutionAttempt(execution="exec-run", attempt=2)]))+ event_log.emit(make_trace_event(workflow_id="exec-item", sequence=0, attempt=2, lineage=[ExecutionAttempt(execution="exec-run", attempt=1)]))++ result = event_log.read_events("run_001")++ assert [event.chain_key for event in result] == ["exec-run@1/exec-item@1", "exec-run@1/exec-item@2", "exec-run@2/exec-item@1"]+def test_multiple_workflows(self, tmp_path: Path) -> None:"""Events from multiple workflows are returned sorted by (workflow_id, sequence)."""event_log = NdjsonEventLog(traces_dir=str(tmp_path))@@ -86,7 +109,7 @@ class TestNdjsonEventLog:assert result[3].workflow_id == "wf_zzz"def test_separate_files_per_workflow(self, tmp_path: Path) -> None:- """Each workflow_id gets its own NDJSON file."""+ """Each workflow gets its own NDJSON file, named by the workflow and by its chain's digest."""event_log = NdjsonEventLog(traces_dir=str(tmp_path))event_log.emit(make_trace_event(workflow_id="wf_alpha", sequence=0))event_log.emit(make_trace_event(workflow_id="wf_beta", sequence=0))@@ -94,8 +117,40 @@ class TestNdjsonEventLog:run_dir = tmp_path / "run_001"ndjson_files = sorted(run_dir.glob("*.ndjson"))assert len(ndjson_files) == 2- assert ndjson_files[0].name == "wf_wf_alpha.ndjson"- assert ndjson_files[1].name == "wf_wf_beta.ndjson"+ assert ndjson_files[0].name == f"wf_wf_alpha__c_{execution_chain_digest(chain_key='wf_alpha@1')}.ndjson"+ assert ndjson_files[1].name == f"wf_wf_beta__c_{execution_chain_digest(chain_key='wf_beta@1')}.ndjson"++ def test_each_attempt_of_each_execution_writes_its_own_file(self, tmp_path: Path) -> None:+ """Two attempts of one workflow, and one workflow two attempts of its dispatcher dispatched under one name, may+ write at once, each through a handle of its own: they never append to one file, where their partial lines could+ interleave and leave the whole run's log unreadable. The reader finds every file of the run.+ """+ run_1 = ExecutionAttempt(execution="exec-run", attempt=1)+ run_2 = ExecutionAttempt(execution="exec-run", attempt=2)+ chains: list[tuple[int, list[ExecutionAttempt]]] = [(1, [run_1]), (2, [run_1]), (1, [run_2])]+ writers = [NdjsonEventLog(traces_dir=str(tmp_path)) for _ in chains]+ for sequence in range(2):+ for writer, (attempt, lineage) in zip(writers, chains, strict=True):+ writer.emit(make_trace_event(workflow_id="exec-item", sequence=sequence, attempt=attempt, lineage=lineage))+ for writer in writers:+ writer.close()++ run_dir = tmp_path / "run_001"+ ndjson_files = sorted(run_dir.glob("*.ndjson"))+ assert len(ndjson_files) == len(chains)+ for ndjson_file in ndjson_files:+ assert ndjson_file.name.startswith("wf_exec-item__c_")+ assert len(ndjson_file.read_text(encoding="utf-8").splitlines()) == 2++ result = NdjsonEventLog(traces_dir=str(tmp_path)).read_events("run_001")+ assert [(event.chain_key, event.sequence) for event in result] == [+ ("exec-run@1/exec-item@1", 0),+ ("exec-run@1/exec-item@1", 1),+ ("exec-run@1/exec-item@2", 0),+ ("exec-run@1/exec-item@2", 1),+ ("exec-run@2/exec-item@1", 0),+ ("exec-run@2/exec-item@1", 1),+ ]def test_cleanup_removes_directory(self, tmp_path: Path) -> None:"""Cleanup removes the pipeline run directory and closes file handles."""@@ -126,7 +181,7 @@ class TestNdjsonEventLog:event_log.emit(make_trace_event(sequence=1))event_log.close()- ndjson_file = tmp_path / "run_001" / "wf_wf_abc.ndjson"+ ndjson_file = _only_run_file(tmp_path / "run_001")with open(ndjson_file, "a", encoding="utf-8") as fhandle:fhandle.write("not valid json\n")@@ -146,7 +201,7 @@ class TestNdjsonEventLog:event_log = NdjsonEventLog(traces_dir=str(tmp_path))event_log.emit(make_trace_event(sequence=0))event_log.close()- ndjson_file = tmp_path / "run_001" / "wf_wf_abc.ndjson"+ ndjson_file = _only_run_file(tmp_path / "run_001")with open(ndjson_file, "a", encoding="utf-8") as fhandle:fhandle.write('{"event_kind": "pipe_start", "traced_value": "a_value_from_the_run\n')@@ -171,7 +226,7 @@ class TestNdjsonEventLog:event_log.emit(make_trace_event(sequence=0))event_log.close()- ndjson_file = tmp_path / "run_001" / "wf_wf_abc.ndjson"+ ndjson_file = _only_run_file(tmp_path / "run_001")with open(ndjson_file, "a", encoding="utf-8") as fhandle:fhandle.write('{"event_kind": "pipe_start", "bad": "traced-value"}\n')@@ -189,7 +244,7 @@ class TestNdjsonEventLog:event_log.emit(make_trace_event(sequence=0))event_log.close()- ndjson_file = tmp_path / "run_001" / "wf_wf_abc.ndjson"+ ndjson_file = _only_run_file(tmp_path / "run_001")with open(ndjson_file, "a", encoding="utf-8") as fhandle:fhandle.write('{"event_kind": "pipe_start", "bad": "traced-value"}\n')@@ -198,6 +253,23 @@ class TestNdjsonEventLog:assert "traced-value" not in str(refusal.value)+ def test_a_line_naming_no_execution_is_refused_with_the_others(self, tmp_path: Path) -> None:+ """A line whose `workflow_id` is empty names no execution: it is counted with the lines the models refuse,+ rather than read and then failing whoever first asks for its chain.+ """+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))+ event_log.emit(make_trace_event(sequence=0))+ event_log.close()++ ndjson_file = _only_run_file(tmp_path / "run_001")+ nameless_line = make_trace_event(sequence=1).model_dump_json().replace('"workflow_id":"wf_abc"', '"workflow_id":""')+ assert '"workflow_id":""' in nameless_line+ with open(ndjson_file, "a", encoding="utf-8") as fhandle:+ fhandle.write(nameless_line + "\n")++ with pytest.raises(EventLogSchemaMismatchError, match="1 trace event"):+ event_log.read_events("run_001")+def test_multiprocess_concurrent_writes(self, tmp_path: Path) -> None:"""Multiple processes writing to different workflow files produces no corruption."""traces_dir = str(tmp_path)diff --git a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.pyindex 943643eea..357703645 100644--- a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py+++ b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py@@ -32,6 +32,9 @@ def _job_metadata() -> JobMetadata:def _usage_event(tokens_usage: AnyTokensUsage, sequence: int) -> UsageReportEvent:return UsageReportEvent(+ attempt=1,+ lineage=[],+ path=None,pipeline_run_id="run-non-llm",workflow_id="wf-non-llm",writer_id="act_test",diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.pyindex f3565deb1..75bb25f29 100644--- a/tests/unit/pipelex/tracing/test_trace_events.py+++ b/tests/unit/pipelex/tracing/test_trace_events.py@@ -5,14 +5,17 @@ from typing import Any, ClassVarimport pytestfrom pydantic import TypeAdapter, ValidationError+from pytest_mock import MockerFixturefrom pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.graph.condition_output_merge import ConditionOutputMerge, ConditionOutputTypingfrom pipelex.graph.graphspec import EdgeKind, ErrorSpec, IOSpec, NodeKind+from pipelex.system.execution_chain import ExecutionAttemptfrom pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobIdfrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.tracing import trace_events as trace_events_modulefrom pipelex.tracing.trace_events import (AnyTraceEvent,BatchAggregateEvent,@@ -42,12 +45,16 @@ class _Shared:TIMESTAMP = datetime(2025, 6, 15, 10, 30, 0, tzinfo=UTC)NODE_ID = "graph_1:wf_pipe_router_xyz:node_0"PARENT_NODE_ID = "graph_1:wf_pipe_router_xyz:node_1"+ PATH = "records.review_records/step_2/item_3"@staticmethoddef make_base_fields(sequence: int = 0) -> dict[str, Any]:return {"pipeline_run_id": _Shared.PIPELINE_RUN_ID,"workflow_id": _Shared.WORKFLOW_ID,+ "attempt": 2,+ "lineage": [ExecutionAttempt(execution="wf_run", attempt=1)],+ "path": _Shared.PATH,"timestamp": _Shared.TIMESTAMP,"sequence": sequence,"mode": PipeRunMode.LIVE,@@ -70,6 +77,7 @@ class _Shared:error_type="LLMError",message="Model returned an error",stack="Traceback (most recent call last):\n File ...",+ caller_facing=True,)@staticmethod@@ -107,6 +115,7 @@ class TestTraceEvents:"node_id": _Shared.NODE_ID,"parent_node_id": _Shared.PARENT_NODE_ID,"pipe_code": "gen_summary",+ "pipe_ref": "records.gen_summary","pipe_type": "PipeLLM","node_kind": NodeKind.OPERATOR,"input_specs": [_Shared.make_io_spec()],@@ -117,6 +126,9 @@ class TestTraceEvents:TraceEventKind.PIPE_END_SUCCESS,{"node_id": _Shared.NODE_ID,+ "pipe_code": "gen_summary",+ "pipe_ref": "records.gen_summary",+ "pipe_type": "PipeLLM","ended_at": _Shared.TIMESTAMP,"output_spec": _Shared.make_io_spec(name="output_text", digest="def456"),"metrics": {"tokens": 150.0},@@ -127,6 +139,9 @@ class TestTraceEvents:TraceEventKind.PIPE_END_ERROR,{"node_id": _Shared.NODE_ID,+ "pipe_code": "gen_summary",+ "pipe_ref": "records.gen_summary",+ "pipe_type": "PipeLLM","ended_at": _Shared.TIMESTAMP,"error": _Shared.make_error_spec(),},@@ -136,6 +151,9 @@ class TestTraceEvents:TraceEventKind.PIPE_END_SKIPPED,{"node_id": _Shared.NODE_ID,+ "pipe_code": "gen_summary",+ "pipe_ref": "records.gen_summary",+ "pipe_type": "PipeLLM","ended_at": _Shared.TIMESTAMP,"skip_reason": "skipped because input 'source' is absent",},@@ -293,6 +311,7 @@ class TestTraceEvents:def test_pipe_start_with_empty_inputs(self) -> None:"""PipeStartEvent works with no input specs."""event = PipeStartEvent(+ pipe_ref=None,**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,pipe_code="noop_pipe",@@ -309,6 +328,9 @@ class TestTraceEvents:def test_pipe_end_success_with_no_output(self) -> None:"""PipeEndSuccessEvent works with no output spec."""event = PipeEndSuccessEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,ended_at=_Shared.TIMESTAMP,@@ -333,6 +355,7 @@ class TestTraceEvents:extra={"custom_field": "custom_value"},)event = PipeStartEvent(+ pipe_ref=None,**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,pipe_code="test_pipe",@@ -359,6 +382,9 @@ class TestTraceEvents:stack="long stack trace here...",)event = PipeEndErrorEvent(+ pipe_code="test_pipe",+ pipe_ref=None,+ pipe_type="PipeFunc",**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,ended_at=_Shared.TIMESTAMP,@@ -392,6 +418,7 @@ class TestTraceEvents:def test_pipe_start_carries_description_and_domain_code(self) -> None:event = PipeStartEvent(+ pipe_ref=None,**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,pipe_code="gen_summary",@@ -406,6 +433,7 @@ class TestTraceEvents:def test_pipe_start_description_and_domain_code_default_to_none(self) -> None:event = PipeStartEvent(+ pipe_ref=None,**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,pipe_code="noop_pipe",@@ -418,6 +446,7 @@ class TestTraceEvents:def test_pipe_start_description_and_domain_code_round_trip(self) -> None:event = PipeStartEvent(+ pipe_ref=None,**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,pipe_code="gen_summary",@@ -441,6 +470,7 @@ class TestTraceEvents:def test_pipe_start_carries_step_and_branch_counts(self, step_count: int | None, branch_count: int | None) -> None:"""A sequence states its authored step count and a parallel its branch count as they start; any other pipe states neither."""event = PipeStartEvent(+ pipe_ref=None,**_Shared.make_base_fields(),node_id=_Shared.NODE_ID,pipe_code="controller",@@ -506,6 +536,20 @@ class TestTraceEvents:del base_fields["mode"]with pytest.raises(ValidationError):PipeStartEvent(+ pipe_ref=None,+ **base_fields,+ node_id=_Shared.NODE_ID,+ pipe_code="noop_pipe",+ pipe_type="PipeNoop",+ node_kind=NodeKind.OPERATOR,+ )++ def test_an_event_naming_no_execution_is_refused(self) -> None:+ """An empty `workflow_id` names no execution: the event is refused when it is read, never when its chain is."""+ base_fields: dict[str, Any] = {**_Shared.make_base_fields(), "workflow_id": ""}+ with pytest.raises(ValidationError, match="workflow_id"):+ PipeStartEvent(+ pipe_ref=None,**base_fields,node_id=_Shared.NODE_ID,pipe_code="noop_pipe",@@ -513,6 +557,34 @@ class TestTraceEvents:node_kind=NodeKind.OPERATOR,)+ def test_an_event_computes_its_keys_once_and_a_copy_computes_them_again(self, mocker: MockerFixture) -> None:+ """Every read and every assembly reads an event's keys, so they are computed once; an event is frozen, and a copy+ made with an update reads its keys from its own fields.+ """+ chain_key_spy = mocker.spy(trace_events_module, "execution_chain_key")+ event = PipeStartEvent(+ pipe_ref=None,+ **_Shared.make_base_fields(sequence=4),+ node_id=_Shared.NODE_ID,+ pipe_code="noop_pipe",+ pipe_type="PipeNoop",+ node_kind=NodeKind.OPERATOR,+ )++ for _ in range(3):+ assert event.chain_key == f"wf_run@1/{_Shared.WORKFLOW_ID}@2"+ assert event.store_key == (f"wf_run@1/{_Shared.WORKFLOW_ID}@2", "primary", "PipeStartEvent", 4)+ assert chain_key_spy.call_count == 1++ with pytest.raises(ValidationError, match="frozen"):+ event.attempt = 3 # type: ignore[misc] # the static read-only error is the runtime contract under test+ retried = event.model_copy(update={"attempt": 3})+ assert retried.chain == [ExecutionAttempt(execution="wf_run", attempt=1), ExecutionAttempt(execution=_Shared.WORKFLOW_ID, attempt=3)]+ assert retried.store_key == (f"wf_run@1/{_Shared.WORKFLOW_ID}@3", "primary", "PipeStartEvent", 4)+ # The keys are no field: they are neither serialized nor compared.+ assert "chain_key" not in event.model_dump()+ assert _any_trace_event_adapter.validate_json(event.model_dump_json()) == event+def test_all_event_kinds_covered(self) -> None:"""Every TraceEventKind has a corresponding test case and event class."""tested_kinds = {entry[1] for entry in self.EVENTS_AND_KINDS}diff --git a/tests/unit/pipelex/tracing/test_usage_aggregator.py b/tests/unit/pipelex/tracing/test_usage_aggregator.pyindex 79076e8c2..fae2d6dd5 100644--- a/tests/unit/pipelex/tracing/test_usage_aggregator.py+++ b/tests/unit/pipelex/tracing/test_usage_aggregator.py@@ -36,6 +36,9 @@ def _make_base_fields(sequence: int = 0) -> dict[str, Any]:return {"pipeline_run_id": _PIPELINE_RUN_ID,"workflow_id": _WORKFLOW_ID,+ "attempt": 1,+ "lineage": [],+ "path": None,"timestamp": _TIMESTAMP,"sequence": sequence,"mode": PipeRunMode.LIVE,@@ -52,6 +55,7 @@ def _make_usage_event(sequence: int, model_name: str = "claude-sonnet") -> Usagedef _make_pipe_start_event(sequence: int) -> PipeStartEvent:return PipeStartEvent(+ pipe_ref=None,**_make_base_fields(sequence),node_id=f"graph_1:{_WORKFLOW_ID}:node_{sequence}",pipe_code="test_pipe",diff --git a/tests/unit/pipelex/tracing/test_writer_id_schema.py b/tests/unit/pipelex/tracing/test_writer_id_schema.pyindex c532b46a8..72c775af4 100644--- a/tests/unit/pipelex/tracing/test_writer_id_schema.py+++ b/tests/unit/pipelex/tracing/test_writer_id_schema.py@@ -5,6 +5,7 @@ expose a writer_id property, and that the read-side dedup/sort/file-routing keysall incorporate writer_id correctly."""+import hashlibimport jsonfrom datetime import UTC, datetimefrom pathlib import Path@@ -13,6 +14,7 @@ from typing import Anyfrom pytest_mock import MockerFixturefrom pipelex.graph.graphspec import NodeKind+from pipelex.system.execution_chain import execution_chain_digestfrom pipelex.system.pipe_run_mode import PipeRunModefrom pipelex.tracing.buffering_event_log import BufferingEventLogfrom pipelex.tracing.dynamodb_event_log import DynamoDBEventLog@@ -23,10 +25,16 @@ from pipelex.tracing.trace_events import PipeStartEvent_TS = datetime(2025, 6, 15, 10, 30, 0, tzinfo=UTC)_PIPELINE_RUN_ID = "run_writer_id_001"_WORKFLOW_ID = "wf_writer_id_xyz"+# The digest naming the chain of an event written by the first attempt of the run's own workflow+_CHAIN_DIGEST = execution_chain_digest(chain_key=f"{_WORKFLOW_ID}@1")def _make_pipe_start(sequence: int, writer_id: str = "primary", workflow_id: str = _WORKFLOW_ID) -> PipeStartEvent:return PipeStartEvent(+ attempt=1,+ lineage=[],+ path=None,+ pipe_ref=None,pipeline_run_id=_PIPELINE_RUN_ID,workflow_id=workflow_id,writer_id=writer_id,@@ -46,6 +54,10 @@ class TestWriterIdSchema:def test_trace_event_has_writer_id_default_primary(self) -> None:"""A TraceEvent built without writer_id round-trips with writer_id="primary"."""event = PipeStartEvent(+ attempt=1,+ lineage=[],+ path=None,+ pipe_ref=None,pipeline_run_id=_PIPELINE_RUN_ID,workflow_id=_WORKFLOW_ID,timestamp=_TS,@@ -76,12 +88,16 @@ class TestWriterIdSchema:legacy_payload: dict[str, Any] = {"pipeline_run_id": _PIPELINE_RUN_ID,"workflow_id": _WORKFLOW_ID,+ "attempt": 1,+ "lineage": [],+ "path": None,"timestamp": _TS.isoformat(),"sequence": 0,"mode": "live","event_kind": "pipe_start","node_id": f"g:{_WORKFLOW_ID}:node_0","pipe_code": "pipe_x",+ "pipe_ref": "tests.pipe_x","pipe_type": "PipeLLM","node_kind": NodeKind.OPERATOR,}@@ -110,7 +126,7 @@ class TestWriterIdSchema:assert writer_ids == {"a", "b"}def test_ndjson_writer_id_in_filename_for_non_primary(self, tmp_path: Path) -> None:- """NDJSON file naming includes writer_id when non-primary; legacy name when primary."""+ """NDJSON file naming includes writer_id when non-primary, after the workflow and its chain's digest."""primary_log = NdjsonEventLog(traces_dir=str(tmp_path))runner_log = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="act_pid42")@@ -120,12 +136,12 @@ class TestWriterIdSchema:run_dir = tmp_path / _PIPELINE_RUN_IDndjson_files = sorted(path.name for path in run_dir.glob("*.ndjson"))- assert ndjson_files == [f"wf_{_WORKFLOW_ID}.ndjson", f"wf_{_WORKFLOW_ID}__w_act_pid42.ndjson"]+ assert ndjson_files == [f"wf_{_WORKFLOW_ID}__c_{_CHAIN_DIGEST}.ndjson", f"wf_{_WORKFLOW_ID}__c_{_CHAIN_DIGEST}__w_act_pid42.ndjson"]def test_two_writers_same_workflow_get_separate_handles(self, tmp_path: Path) -> None:"""Two NdjsonEventLogs with different writer_ids must not share the file-handle cache slot.- Pins the (pipeline_run_id, workflow_id, writer_id) cache key. With a 2-tuple key,+ Pins the (pipeline_run_id, chain_key, writer_id) cache key. With a 2-tuple key,the second writer would steal the first writer's file handle and corrupt routing."""log_a = NdjsonEventLog(traces_dir=str(tmp_path), writer_id="writer_a")@@ -137,8 +153,8 @@ class TestWriterIdSchema:log_b.close()run_dir = tmp_path / _PIPELINE_RUN_ID- a_lines = (run_dir / f"wf_{_WORKFLOW_ID}__w_writer_a.ndjson").read_text(encoding="utf-8").splitlines()- b_lines = (run_dir / f"wf_{_WORKFLOW_ID}__w_writer_b.ndjson").read_text(encoding="utf-8").splitlines()+ a_lines = (run_dir / f"wf_{_WORKFLOW_ID}__c_{_CHAIN_DIGEST}__w_writer_a.ndjson").read_text(encoding="utf-8").splitlines()+ b_lines = (run_dir / f"wf_{_WORKFLOW_ID}__c_{_CHAIN_DIGEST}__w_writer_b.ndjson").read_text(encoding="utf-8").splitlines()assert len(a_lines) == 1assert len(b_lines) == 1@@ -187,7 +203,7 @@ class TestWriterIdSchema:assert events[2].writer_id == "primary"def test_dynamodb_sk_includes_writer_id(self, mocker: MockerFixture) -> None:- """DynamoDB SK encodes writer_id between workflow_id and sequence."""+ """DynamoDB SK encodes writer_id between the digest of the event's chain and its sequence."""captured_items: list[dict[str, Any]] = []class _StubTable:@@ -208,7 +224,8 @@ class TestWriterIdSchema:event_log.emit(_make_pipe_start(sequence=0, writer_id="act_pid42"))assert len(captured_items) == 1- assert captured_items[0]["SK"] == f"EVENT#{_WORKFLOW_ID}#act_pid42#0000000000"+ chain_digest = hashlib.sha256(f"{_WORKFLOW_ID}@1".encode()).hexdigest()+ assert captured_items[0]["SK"] == f"EVENT#{chain_digest}#act_pid42#0000000000"def test_event_log_exposes_writer_id_property(self, tmp_path: Path) -> None:"""Every event log backend exposes its writer_id via a read-only property.