Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Json-sink-trace-context · L-260925-c9ab0d (#1259)

Louis Choquel6154387c9896
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d6ace9365..b535be2e8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
### Added
+- **Log lines carry the trace context of the span they were logged in**: the `json` sink writes `trace_id`, `span_id` and `trace_flags` in lowercase hex, the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, on every line logged inside a valid span and none of them outside one, reserving the three names so a field spelling one rides under the `field_` prefix, and the `otlp` sink files each record under that span. The span is the Pipelex span active at the log call, a pipe's or an LLM call's, which the runtime holds in a context variable of its own from the span's start to its end whenever it traces a run, under the Pipelex Gateway's telemetry or AI span tracing on your own PostHog; outside one it is OpenTelemetry's current span, only read, so a line your code logs under its own span joins your trace. Pipelex never makes its spans current in the process's OpenTelemetry context, so a host's own instrumentation is never re-parented under them, and the spans' parents still travel in the job metadata. A boot line held until the sink arrives keeps the span it was logged in, and the opt-in boto3 Bedrock client runs its blocking call through `asyncio.to_thread`, which carries the context into the thread, as [Logging](tools/logging.md#the-trace-context) describes.
- **Named fields and a run-scoped context on the log calls**: the seven `log.*` methods take a keyword-only `fields={...}`, carried as attributes of the stdlib `LogRecord` and never rendered into the message, and `with log.context(request_id=..., pipeline_run_id=..., pipe_run_id=...)` stamps the run's identifiers onto every record emitted in scope, nested bindings merging and an absent identifier staying absent rather than reading `None`. `PipeRun.run` binds the request and pipeline identifiers from the job's metadata for a direct-mode run, and every pipe binds its own `pipe_run_id` around the whole of its run, its announcement and its failure included. A `dict` or `list` content is carried as the record's `data` attribute beside its console rendering, a JSON-ready snapshot of the call rather than the caller's live object, and an entry named like an attribute the record already owns, a record factory's included, or like one of the names set after the fields are attached — the formatter's, and the internal marker that tells a sink a record reached it already — is carried under a `field_` prefix instead of raising or steering delivery, whatever the order the names arrive in — a field spelling `data` beside structured content included, so the content keeps the name a sink reads it under and the field keeps its value beside it, as [Logging](tools/logging.md) describes.
- **Log sinks are a plugin capability selected by `[runtime.log] sink` (Breaking)**: where the records go is a registered sink token, `console` (the Rich handler, the default so the CLI keeps its rendering), `json` (one JSON object per line carrying `time`, `severity`, `logger`, `message`, the fields, the context identifiers and the exception, with no ANSI ever, for a server behind a log agent) or `otlp` (the OpenTelemetry logs signal, configured under `[runtime.log.otlp]`, so a collector receives the logs beside the spans), each of the two reserving the keys it writes itself so that a field named like one of them rides under a `field_` prefix rather than overwriting it, or a token an installed plugin registers through `add_log_sink`, and a token nobody provides stops the boot naming the registered ones, while `pipelex doctor` reports it in a row of its own and carries on through the console sink. `log_mode`, `poor_loggers`, `generic_poor_logger` and `is_console_logging_enabled` are gone with the rich-or-poor mode, and `set_log_mode`, `set_poor_log_formatter` and `LevelAndEmojiLogFormatter` with them, so a server that wants plain output selects `json` and a host that wants Pipelex silent sets `pipelex = "OFF"` under `[runtime.log.package_log_levels]`. **Migration:** run `pipelex migrate`; ledger entry `pipelex-config@5` deletes the four retired keys from `~/.pipelex/` and every project `.pipelex/` file, after which the defaults supply `sink = "console"`. A file that had `log_mode = "poor"` or `is_console_logging_enabled = false` chose a behaviour that deletion undoes and no migration may write back, so such a file sets `sink = "json"` or `pipelex = "OFF"` under `[runtime.log.package_log_levels]` by hand, as the entry's title says. [Log Sink Plugins](under-the-hood/log-sink-plugins.md) documents the seam.
- **A `gcp` log sink writing to Google Cloud Logging**: `sink = "gcp"` sends each record to Cloud Logging through the `google-cloud-logging` client library, which the new `gcp-logging` extra installs; selecting it without the extra stops the boot with the install hint and the `json` alternative named. Each record becomes one struct entry — the level mapped onto the Cloud Logging severity scale, the message, the logger, the exception and the fields as the JSON payload, the run-scoped identifiers as the entry's labels, which is what Cloud Logging indexes, and the run's own OpenTelemetry trace id, project-qualified, in the entry's `trace` field, so a line and the spans of its run are filed together. The entries leave through the library's background-thread transport, so no record costs an API round trip on the thread that logged it, with the export path rejected before the handler's lock so a refused batch is reported on stderr, rate-limited, rather than through the pipeline that refused it, and the teardown's flush carrying a deadline the library's own flush does not. `[runtime.log.gcp]` carries the log name, the project when it is not the ambient one, and a service-account credentials path for a process that should not use Application Default Credentials. **Most processes on Google Cloud want `json` instead**: Cloud Run, GKE and any platform running a logging agent over the container's stdout ingest what that sink writes and need no client at all, which [Logging Configuration](configuration/config-practical/logging-config.md) spells out.
@@ -25,6 +26,7 @@
- **Image and document inputs carry a `public_url` whatever form their url takes**: an input given as a `pipelex-storage://` reference now gets a link signed through the storage provider, replacing any link it carried, which may have expired, and an input given as an `http(s)` URL gets that URL as its `public_url` unless it names another. This holds for an image or document passed alone, in a list or in a structured field, so a template writing `{{ image.public_url }}` no longer renders `None` for one uploaded to storage or passed by URL, and a reference the storage provider refuses as a key is reported as an input error before the run starts.
- **A NUL byte in an input's path or storage reference is an input error**: a local file path, or a `pipelex-storage://` reference on local storage, holding a NUL byte used to escape as an internal error; it is now reported as an input error before the run starts.
- **Unsigned storage links percent-encode their key**: the S3 and GCS providers' unsigned links now encode a key holding a space, `+`, `#` or `?` as a signed link does, instead of naming a path the URL cuts short.
+- **A cancelled pipe run ends its span**: the pipe's span was closed only when the run raised an `Exception`, so a run cancelled by a timeout, a disconnected client or a cancelled orchestration task never ended its span and no exporter received it. It now ends with an error status like any other failure.
- **A log call before `log.configure` no longer raises**: it goes to the stdlib's default handling at the stdlib's default level, so a library that logs before Pipelex boots, or a boot that logs while it configures, cannot crash on the log line itself. Between `log.configure` and the sink's arrival at boot, every record is held and replayed through the selected sink in order, so a boot's own lines are rendered by the sink the configuration chose, and a boot that dies before the sink arrives sends all of them to stderr rather than only a warning or worse: each one passed the level the configuration set, and a failed boot is when that trail is worth most.
- **A log call never raises on a value the record or JSON refuses**: a field named like an attribute the `LogRecord` class owns (`__class__`, `__dict__`, `getMessage`) is prefixed like any other collision instead of raising `TypeError` or shadowing the method every formatter calls, and a content `json` refuses, a circular reference or a mapping with a non-string key, is rendered as its `repr` with no `data` instead of raising `ValueError` or `TypeError` out of `log.info`. A non-finite float nested inside a pydantic model is spelled as text like one at the top level, so a single `NaN` costs that one float its type and no other value on the line its structure. On the `console` sink the same promise needed the stdlib's own guard put back: Rich overrides `emit` without it, so a message carrying a tag-shaped span — `list[int]` in a type complaint, or a bracketed path, which Rich reads as a closing tag — raised `MarkupError` out of the log call and replaced the failure being reported with itself. Such a line now costs at most its own rendering, and an error assembled from an exception is rendered verbatim rather than read as markup, so the brackets and the type names in it reach the terminal as they were written.
- **A field named `_pipelex_forwarded` no longer deletes its own record**: the name the holding handler stamps on a record it has already forwarded was treated as unspellable because it starts with an underscore, but a fresh record does not carry it, so a caller's `fields={"_pipelex_forwarded": ...}` landed unprefixed and had the record rejected from every sink by the guard that exists to stop a double delivery. Such an entry is now carried under the `field_` prefix like any other collision, and the line is delivered.
diff --git a/docs/configuration/config-practical/logging-config.md b/docs/configuration/config-practical/logging-config.md
index b5c0d9086..f80d1bccc 100644
--- a/docs/configuration/config-practical/logging-config.md
+++ b/docs/configuration/config-practical/logging-config.md
@@ -198,10 +198,10 @@ keywords_to_hilight = []
## The `json` Sink
-No section of its own: the sink writes to `console_log_target`, one JSON object per line and no ANSI ever. Each line carries `time` (ISO 8601, UTC, milliseconds, `Z`), `severity` (the level name), `logger` (the module-named logger), `message`, `exception` when the record carries one, and then every field, the run-scoped identifiers and the `data` attribute flat beside them, under their own names. Those keys are reserved on every line, so a field named like one of them is carried under a `field_` prefix whether or not the line carries an exception, and a non-finite float is written as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser.
+No section of its own: the sink writes to `console_log_target`, one JSON object per line and no ANSI ever. Each line carries `time` (ISO 8601, UTC, milliseconds, `Z`), `severity` (the level name), `logger` (the module-named logger), `message`, `exception` when the record carries one, `trace_id`, `span_id` and `trace_flags` when the record was logged inside a span, and then every field, the run-scoped identifiers and the `data` attribute flat beside them, under their own names. The three trace keys are the ones OpenTelemetry specifies for trace context in a JSON log that is not OTLP, lowercase hex at their full widths (32, 16 and 2 digits), so a collector or an error tracker joins the line to the span it was logged in; the span is the Pipelex span active at the log call, the pipe's own or the LLM call's during a run, and outside one OpenTelemetry's current span, your own code's, which Pipelex reads and never changes (see [Logging](../../tools/logging.md#the-trace-context)); a line logged outside any span carries none of the three. Those keys are reserved on every line, so a field named like one of them is carried under a `field_` prefix whether or not the line carries an exception or a span, and a non-finite float is written as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser.
```json
-{"time": "2026-09-16T10:12:03.417Z", "severity": "INFO", "logger": "pipelex.pipe_operators.pipe_llm", "message": "Running the pipe", "request_id": "req-7f3a", "pipeline_run_id": "plr-01", "pipe_run_id": "pr-9c", "model": "gpt-5"}
+{"time": "2026-09-16T10:12:03.417Z", "severity": "INFO", "logger": "pipelex.pipe_operators.pipe_llm", "message": "Running the pipe", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "trace_flags": "01", "request_id": "req-7f3a", "pipeline_run_id": "plr-01", "pipe_run_id": "pr-9c", "model": "gpt-5"}
```
## The `otlp` Sink
@@ -217,6 +217,7 @@ headers = { Authorization = "Bearer ..." }
- `endpoint`: The collector's logs URL. Left unset, the exporter follows the OpenTelemetry environment conventions: `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT` with the `/v1/logs` path, then the collector default on localhost
- `headers`: Headers sent with every export, an authorization header typically. Empty, the default, leaves `OTEL_EXPORTER_OTLP_HEADERS` in charge
- The records are exported in batches on the OTLP HTTP protocol, with the same service identity as the spans the runtime already exports, so a collector files the two together
+- Each record carries the trace context of the span it was logged in, the Pipelex span active at the log call, the pipe's own or the LLM call's during a run, and outside one OpenTelemetry's current span, your own code's, so a collector files the record under that span; a record logged outside any span carries none. Pipelex never makes its own spans current in the process's OpenTelemetry context, so your own instrumentation is never re-parented under them
- The semantic-convention keys the sink writes itself — the source location `code.file.path`, `code.function.name` and `code.line.number`, and the `exception.*` keys — are reserved on every record whether or not it carries an exception, exactly as the `json` sink reserves its own keys: a field named like one of them is carried under the same `field_` prefix, so nothing is overwritten and a field keeps one wire name whichever sink is selected
- The sink never exports its own export path: a record the SDK or the transport emits while an export is in flight is rejected before the handler's lock is taken, so an unreachable collector costs a warning on the export thread and never a loop or a hang at exit. A flush at teardown is bounded by the exporter's own timeout, `OTEL_EXPORTER_OTLP_TIMEOUT`, and whatever the sink raises while it flushes or closes is said on stderr rather than left to interrupt the teardown
@@ -246,7 +247,7 @@ credentials_file_path = "gcp_credentials.json"
Each record becomes one Cloud Logging entry with a JSON payload:
- The level maps onto the Cloud Logging severity scale. That scale has nothing below `DEBUG`, so Pipelex's two custom levels, `VERBOSE` and `DEV`, both land there
-- The payload carries `message`, `logger` and `exception` when the record carries one, then every field and the `data` attribute flat beside them. The keys the `json` sink reserves are reserved here too, `time` and `severity` included although the payload carries neither — the client library takes both out of band — so a field named like one of them is carried under a `field_` prefix under either sink rather than under one and not the other; a value JSON cannot carry — a non-finite float, a model, a circular structure — is written as text rather than costing the line
+- The payload carries `message`, `logger` and `exception` when the record carries one, then every field and the `data` attribute flat beside them. The keys the `json` sink reserves are reserved here too, `time`, `severity`, `trace_id`, `span_id` and `trace_flags` included although the payload carries none of them — the client library takes the time and the severity out of band, and the entry's own `trace` field carries the trace — so a field named like one of them is carried under a `field_` prefix under either sink rather than under one and not the other; a value JSON cannot carry — a non-finite float, a model, a circular structure — is written as text rather than costing the line
- The run-scoped identifiers become the entry's **labels** rather than payload keys: `request_id`, `pipeline_run_id` and `pipe_run_id`, whichever of them the record carries. Cloud Logging indexes labels, so these are what a query filters a run by
- The entry's `trace` field carries the run's own OpenTelemetry trace id, project-qualified as `projects/<project>/traces/<trace-id>`. The id is derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of the run it belongs to agree on it and Cloud Logging files them together
- The entries leave through the client library's background-thread transport, which batches them off the thread that logged, so no record costs an API round trip on the calling thread. The teardown flushes and closes it, and the flush carries a deadline of its own because the library's does not
diff --git a/docs/setup/telemetry.md b/docs/setup/telemetry.md
index 688100983..7c247cfee 100644
--- a/docs/setup/telemetry.md
+++ b/docs/setup/telemetry.md
@@ -49,6 +49,14 @@ Custom telemetry is configured in `.pipelex/telemetry.toml` and allows you to se
Custom telemetry is completely independent from Gateway telemetry—you can use both, either, or neither.
+### Pipelex's spans in your process
+
+Either stream makes Pipelex trace a run: the Gateway stream whenever it is on, and yours when AI span tracing is enabled on your PostHog. Pipelex's tracer stays its own and never becomes the global one, so your own spans never reach Pipelex's exporters, and **Pipelex never makes its spans current in your process's OpenTelemetry context**:
+
+- Your own instrumentation, an HTTP client's or a provider SDK's, keeps opening its spans under your own current span, in your own trace, while a Pipelex run goes. It is never re-parented under a pipe or an LLM call, nor kept by a sampler that follows its parent because a Pipelex span was sampled.
+- An error tracker that reads OpenTelemetry's current span sees yours, not Pipelex's.
+- The `json` and `otlp` log sinks still join each line to the Pipelex span it was logged in, the pipe's or the LLM call's, which Pipelex keeps in a context variable of its own, and outside a Pipelex run to your current span, as [Logging](../tools/logging.md#the-trace-context) describes.
+
## Quick Setup
When you run `pipelex init`, a default `telemetry.toml` configuration file is created:
diff --git a/docs/tools/logging.md b/docs/tools/logging.md
index e52d0147e..f6420efe9 100644
--- a/docs/tools/logging.md
+++ b/docs/tools/logging.md
@@ -112,6 +112,23 @@ The identifiers travel in the payload, and the contextvar is in-process plumbing
The last two are the runner's and the orchestration plugin's to bind, beside the payload they read; the runtime only provides `log.context`.
+### The trace context
+
+Beside the identifiers, a record is joined to the span it was logged in, and the structured sinks write that on their own, with nothing for a call site to pass: the `json` sink as `trace_id`, `span_id` and `trace_flags`, lowercase hex, the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, and the `otlp` sink as the record's own trace context, so a collector files the record under the span. The span is found in this order:
+
+1. **The Pipelex span active at the log call**, the pipe's or the LLM call's, while Pipelex runs one.
+2. **Otherwise, OpenTelemetry's current span**, read and never changed, so a line your own code logs under its own span, outside a Pipelex run, joins your trace.
+3. **Otherwise, none**: the `json` line carries none of the three keys, and the `otlp` record no trace context. The `console` sink writes none in any case.
+
+The runtime holds each span it starts as the active Pipelex span from its start to its end, however it ends:
+
+- **A pipe run's span**: `live_run_pipe` starts the span, then runs the pipe with it held, so a line logged by the pipe's own work, a controller's or an operator's and the lines of the extract, image-generation, search or function call an operator makes, names the pipe's span, and a nested pipe's lines name the nested pipe's span until it returns. The line announcing the run is logged before the span starts, under the enclosing span.
+- **An LLM call's span**: the LLM worker runs the provider call with its generation span held, so the lines of the call and of its failure name that span, a provider SDK's own lines included.
+
+The runtime starts these spans only when it traces a live run, which it does once telemetry has created its tracer: with the Pipelex Gateway's telemetry on, which is automatic when the Gateway serves your inference, or with AI span tracing enabled on your own PostHog, as [Telemetry](../setup/telemetry.md#pipelexs-spans-in-your-process) describes. Otherwise it holds no span, as it does when its tracer is a no-op one that starts spans naming no trace, under `OTEL_SDK_DISABLED` for instance, and a line names whatever span your own code runs under, if any. A boot line held until the sink arrives is replayed in the context it was logged in, so it keeps its span too, and a thread started with `asyncio.to_thread` or `contextvars.copy_context` inherits the span of the code that started it.
+
+Holding a span is in-process plumbing, like binding the log context: it lives in a context variable of Pipelex's own. **Pipelex never makes its spans current in your process's OpenTelemetry context**, so your own instrumentation, an HTTP client's or a provider SDK's, is never re-parented under a Pipelex span, and an error tracker that reads OpenTelemetry's current span sees yours, not Pipelex's. A span's children take their parent from the job metadata, so a step running in another process gets the parent it always did, and a line that process logs names a span only while the process runs a span of its own, the LLM call's for instance.
+
## Structured content
When the content is not a string, it is rendered as JSON for the message, indented by `json_logs_indent`, and the rendering is read back as the record's `data` attribute for a structured sink. `data` is therefore a snapshot of the call, JSON-ready whatever the content held, and the caller may mutate the object afterwards without changing what was emitted:
diff --git a/docs/under-the-hood/log-sink-plugins.md b/docs/under-the-hood/log-sink-plugins.md
index 710ff648a..7fff7a15c 100644
--- a/docs/under-the-hood/log-sink-plugins.md
+++ b/docs/under-the-hood/log-sink-plugins.md
@@ -93,13 +93,13 @@ class LogSink(ABC):
### What a sink reads off a record
-Every sink reads the same record, shaped by the [logging facade](../tools/logging.md): the message is `record.getMessage()`, and everything the call, the bound context or a record factory attached rides as attributes. `carried_attributes(record=...)` in `pipelex/tools/log/log_fields.py` hands a sink exactly that set, in the order the attributes were attached, by subtracting the attributes the stdlib gives every record and the ones this package stamps on a record itself. An exception a call carried with `include_exception=True` is the record's `exc_info`, the stdlib's own channel, which each sink renders its own way.
+Every sink reads the same record, shaped by the [logging facade](../tools/logging.md): the message is `record.getMessage()`, and everything the call, the bound context or a record factory attached rides as attributes. `carried_attributes(record=...)` in `pipelex/tools/log/log_fields.py` hands a sink exactly that set, in the order the attributes were attached, by subtracting the attributes the stdlib gives every record and the ones this package stamps on a record itself. An exception a call carried with `include_exception=True` is the record's `exc_info`, the stdlib's own channel, which each sink renders its own way. The trace context is not on the record: it is read from the context when the sink's handler runs, which is inside the log call, or, for a boot line held until the sink arrives, from the context the holding handler copied when the line was logged and replays it under. The runtime holds the pipe's span or the LLM call's in a context variable of its own while it runs, and never makes it OpenTelemetry's current span, so a sink that joins its records to their traces reads it at emit through `pipelex/system/telemetry/current_span.py`: `span_context_for_logs()` returns the active Pipelex span's context, else the valid current OpenTelemetry span's, else `None`, and `otel_context_for_logs()` returns the current context with the active Pipelex span set in it, for an OpenTelemetry API that takes a context. The `json` sink reads the first, the `otlp` sink the second.
Three rules follow for a sink author, and the shipped sinks honour all of them:
- **A sink serializes on emit, synchronously, on the calling thread.** A field's value rides the record by reference, so a value the caller mutates after the call is not what the record said; reading it at emit is what makes the record a faithful account of the call. A sink that queues, the `otlp` sink on its batching exporter for instance, translates the record into its wire form at emit and queues the translation. The `data` attribute needs no such care: it is already a JSON-ready snapshot of the call.
- **A sink renders every record it is handed and never raises.** A value the wire cannot carry is written as text, a JSON rendering for a mapping, a `repr` for what JSON refuses; a serialization failure is a fact about the value, never a reason to lose the line. A handler failure goes through the stdlib's `handleError`, as any handler's does.
-- **A sink reserves the keys it writes itself, on every record.** A carried attribute named like one of them is written under the `COLLIDING_FIELD_PREFIX` — `field_` — applied until the name lands free, which is the same treatment the record gives a field named after a stdlib attribute. Reserved unconditionally, not only when the sink happens to write the key on this record: a field's wire name must not depend on whether an exception was attached, and a field must not survive one sink and vanish on another. The `json` sink reserves `time`, `severity`, `logger`, `message` and `exception`; the `gcp` sink reserves that same set even though only `message`, `logger` and `exception` are payload keys there, the client library carrying the time and the severity out of band, so a field named like one of them is prefixed under either sink rather than under one only; the `otlp` sink reserves its `code.*` and `exception.*` semantic-convention keys.
+- **A sink reserves the keys it writes itself, on every record.** A carried attribute named like one of them is written under the `COLLIDING_FIELD_PREFIX` — `field_` — applied until the name lands free, which is the same treatment the record gives a field named after a stdlib attribute. Reserved unconditionally, not only when the sink happens to write the key on this record: a field's wire name must not depend on whether an exception was attached, and a field must not survive one sink and vanish on another. The `json` sink reserves `time`, `severity`, `logger`, `message`, `exception`, `trace_id`, `span_id` and `trace_flags`; the `gcp` sink reserves that same set even though only `message`, `logger` and `exception` are payload keys there, the client library carrying the time and the severity out of band and the entry's own `trace` field carrying the trace, so a field named like one of them is prefixed under either sink rather than under one only; the `otlp` sink reserves its `code.*` and `exception.*` semantic-convention keys.
---
@@ -121,9 +121,9 @@ class LogSinkPlugin:
| Sink | What it does | Where it reads its settings |
|------|--------------|-----------------------------|
-| `json` | One JSON object per line on the configured stream: `time`, `severity`, `logger`, `message`, `exception` when there is one, then the fields, the context identifiers and `data` flat beside them, a field named like one of those keys under a `field_` prefix on every line, a non-finite float as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser, and no ANSI ever. What the hosted plane's runner and worker select. | `console_log_target` |
+| `json` | One JSON object per line on the configured stream: `time`, `severity`, `logger`, `message`, `exception` when there is one, `trace_id`, `span_id` and `trace_flags` in hex when the record was logged inside a span, the active Pipelex span or else the current OpenTelemetry one, under the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, then the fields, the context identifiers and `data` flat beside them, a field named like one of those keys under a `field_` prefix on every line, a non-finite float as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser, and no ANSI ever. What the hosted plane's runner and worker select. | `console_log_target` |
| `console` | The Rich handler with the emoji formatter and every `[runtime.log.rich_log]` setting, byte for byte what the console showed before sinks existed. Rich, the `cli` extra, is imported when the handler is built, and a process that selects another sink never loads it through this path. | `console_log_target`, `[runtime.log.rich_log]` |
-| `otlp` | The OpenTelemetry logs signal: a `LoggerProvider` carrying the same service identity as the tracer, a `BatchLogRecordProcessor` and the OTLP HTTP log exporter. The message is the body, the level maps onto the OTel severity scale, the fields, identifiers and `data` ride as attributes (a mapping as JSON text), an exception lands under the `exception.*` semantic-convention keys, and a collector receives the logs beside the spans the runtime already exports. The keys the sink writes itself, the `code.*` source location and the `exception.*` set, are reserved on every record whether or not it carries an exception, so a field named like one of them rides under a `field_` prefix rather than being overwritten. A filter on the handler rejects the records of the sink's own export path, the SDK's by logger name and the transport's by the context value the SDK sets around an export, before the handler's lock is taken, so an export failure never re-enters the pipeline and a shutdown never waits on itself. | `[runtime.log.otlp]` |
+| `otlp` | The OpenTelemetry logs signal: a `LoggerProvider` carrying the same service identity as the tracer, a `BatchLogRecordProcessor` and the OTLP HTTP log exporter. The message is the body, the level maps onto the OTel severity scale, the fields, identifiers and `data` ride as attributes (a mapping as JSON text), an exception lands under the `exception.*` semantic-convention keys, each record is filed under the span it was logged in, the active Pipelex span or else the current OpenTelemetry one, and a collector receives the logs beside the spans the runtime already exports. The keys the sink writes itself, the `code.*` source location and the `exception.*` set, are reserved on every record whether or not it carries an exception, so a field named like one of them rides under a `field_` prefix rather than being overwritten. A filter on the handler rejects the records of the sink's own export path, the SDK's by logger name and the transport's by the context value the SDK sets around an export, before the handler's lock is taken, so an export failure never re-enters the pipeline and a shutdown never waits on itself. | `[runtime.log.otlp]` |
| `gcp` | Google Cloud Logging through the `google-cloud-logging` client library, behind the `gcp-logging` extra. Each record becomes one struct entry: the level maps onto the Cloud Logging severity scale (which has nothing below `DEBUG`, so `VERBOSE` and `DEV` both land there), the message, the logger, the exception and the fields become the JSON payload, the run-scoped identifiers become the entry's labels, which is what Cloud Logging indexes, and the entry's `trace` field carries the run's own OpenTelemetry trace id project-qualified, derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of its run agree on it. The entries leave through the client library's background-thread transport, which batches them off the calling thread. **Most processes on Google Cloud want `json` instead**: a platform whose logging agent reads the container's stdout, Cloud Run and GKE among them, ingests what that sink writes and needs no client at all; `gcp` is for a process with no such agent in front of it, or one writing to a log or a project that is not the ambient one. | `[runtime.log.gcp]` |
The `otlp` factory imports the OpenTelemetry logs SDK when it runs, and the `gcp` factory the Google Cloud Logging client library, never at register, so registering the built-ins imports neither, which the import-light guard pins.
diff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.py
index 44b1e46b1..b4bc39424 100644
--- a/pipelex/cogt/llm/llm_worker_abstract.py
+++ b/pipelex/cogt/llm/llm_worker_abstract.py
@@ -16,6 +16,7 @@ from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedCons
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.system.exceptions import JobMetadataError
from pipelex.system.job_metadata import UnitJobId
+from pipelex.system.telemetry.current_span import pipelex_span_active
from pipelex.system.telemetry.otel_constants import (
GenAISpanAttr,
LangfuseSpanAttr,
@@ -227,16 +228,17 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
attributes=span_attributes,
)
- # Debug logging
+ # Debug logging, under the span it announces, so the line's trace context names that span
span_ctx = span.get_span_context()
- log.verbose(
- f"[OTel] LLM SPAN STARTED:\n"
- f" pipe_code='{pipe_code}'\n"
- f" pipeline_run_id='{pipeline_run_id}'\n"
- f" trace_id={span_ctx.trace_id:032x}\n"
- f" span_id={span_ctx.span_id:016x}\n"
- f" parent_span_id={parent_span_id:016x}"
- )
+ with pipelex_span_active(span=span):
+ log.verbose(
+ f"[OTel] LLM SPAN STARTED:\n"
+ f" pipe_code='{pipe_code}'\n"
+ f" pipeline_run_id='{pipeline_run_id}'\n"
+ f" trace_id={span_ctx.trace_id:032x}\n"
+ f" span_id={span_ctx.span_id:016x}\n"
+ f" parent_span_id={parent_span_id:016x}"
+ )
return span
@@ -462,20 +464,24 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
# Start OTel span after _before_job (which may set model info)
span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.TEXT)
- try:
- text_result = await self._gen_text(llm_job=llm_job)
- await self._after_text_job(span=span, llm_job=llm_job, result_text=text_result)
- return text_result
- except CogtError as exc:
- exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())
- raise
- finally:
- # `_gen_text` / `_after_text_job` raised before the span was ended — close it with the
- # in-flight error (a CogtError, or any unexpected failure) so no span leaks and the
- # failure stays in telemetry. On success `_after_text_job` already ended the span.
- pending_error = sys.exc_info()[1]
- if pending_error is not None and span is not None and span.is_recording():
- self._end_otel_span_with_error(span=span, llm_job=llm_job, error=pending_error)
+ # The span is the Pipelex span active here until it ends, whichever way it ends, so a log line
+ # during the call, a provider SDK's included, is joined to it; OpenTelemetry's current context
+ # is left alone.
+ with pipelex_span_active(span=span):
+ try:
+ text_result = await self._gen_text(llm_job=llm_job)
+ await self._after_text_job(span=span, llm_job=llm_job, result_text=text_result)
+ return text_result
+ except CogtError as exc:
+ exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())
+ raise
+ finally:
+ # `_gen_text` / `_after_text_job` raised before the span was ended — close it with the
+ # in-flight error (a CogtError, or any unexpected failure) so no span leaks and the
+ # failure stays in telemetry. On success `_after_text_job` already ended the span.
+ pending_error = sys.exc_info()[1]
+ if pending_error is not None and span is not None and span.is_recording():
+ self._end_otel_span_with_error(span=span, llm_job=llm_job, error=pending_error)
@abstractmethod
async def _gen_text(
@@ -501,25 +507,29 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
# Start OTel span after _before_job (which may set model info)
span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.OBJECT, output_class_name=schema.__name__)
- try:
- object_result = await self._gen_object(llm_job=llm_job, schema=schema)
-
- # Cleanup result
- if hasattr(object_result, "_raw_response"):
- delattr(object_result, "_raw_response") # ruff: ignore[del-attr-with-constant] - not a declared model field, so `del obj._attr` cannot type-check
-
- await self._after_object_job(span=span, llm_job=llm_job, result_object=object_result)
- return object_result
- except CogtError as exc:
- exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())
- raise
- finally:
- # `_gen_object` / `_after_object_job` raised before the span was ended — close it with
- # the in-flight error (a CogtError, or any unexpected failure) so no span leaks and the
- # failure stays in telemetry. On success `_after_object_job` already ended the span.
- pending_error = sys.exc_info()[1]
- if pending_error is not None and span is not None and span.is_recording():
- self._end_otel_span_with_error(span=span, llm_job=llm_job, error=pending_error)
+ # The span is the Pipelex span active here until it ends, whichever way it ends, so a log line
+ # during the call, a provider SDK's included, is joined to it; OpenTelemetry's current context
+ # is left alone.
+ with pipelex_span_active(span=span):
+ try:
+ object_result = await self._gen_object(llm_job=llm_job, schema=schema)
+
+ # Cleanup result
+ if hasattr(object_result, "_raw_response"):
+ delattr(object_result, "_raw_response") # ruff: ignore[del-attr-with-constant] - not a declared model field, so `del obj._attr` cannot type-check
+
+ await self._after_object_job(span=span, llm_job=llm_job, result_object=object_result)
+ return object_result
+ except CogtError as exc:
+ exc.fill_model_and_provider(model_handle=self._get_request_model_name(), backend_name=self._get_provider_name())
+ raise
+ finally:
+ # `_gen_object` / `_after_object_job` raised before the span was ended — close it with
+ # the in-flight error (a CogtError, or any unexpected failure) so no span leaks and the
+ # failure stays in telemetry. On success `_after_object_job` already ended the span.
+ pending_error = sys.exc_info()[1]
+ if pending_error is not None and span is not None and span.is_recording():
+ self._end_otel_span_with_error(span=span, llm_job=llm_job, error=pending_error)
@abstractmethod
async def _gen_object(
diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.py
index af0feea20..8db607a4e 100644
--- a/pipelex/pipe_machinery/pipe_abstract.py
+++ b/pipelex/pipe_machinery/pipe_abstract.py
@@ -33,6 +33,7 @@ from pipelex.system.caller_identity import CallerIdentity, scoped_caller_identit
from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.registries.class_registry_access import get_class_registry
+from pipelex.system.telemetry.current_span import pipelex_span_active
from pipelex.system.telemetry.otel_constants import (
LangfuseSpanAttr,
OTelConstants,
@@ -972,23 +973,29 @@ class PipeAbstract(ABC, BaseModel):
# Run pipe ------------------------------------------------------------
- try:
- pipe_output = await self._live_run_pipe(
- job_metadata=child_metadata,
- working_memory=working_memory,
- pipe_run_params=pipe_run_params,
- output_name=output_name,
- library_crate=library_crate,
- )
- except Exception as exc:
- # Broad catch is intentional: the OTel span must be closed with ERROR status
- # on any failure. Observes-and-re-raises — see note on the catch in _run_pipe_traced.
- self._end_pipe_span_error(span, error=exc, is_root_span=is_root_span)
- raise
+ # The span is the Pipelex span active here until it ends, whichever way it ends, so a log
+ # line inside the run is joined to it. OpenTelemetry's current context is left alone, so a
+ # host's own instrumentation is never re-parented, and the span's children still take their
+ # parent from `child_metadata`.
+ with pipelex_span_active(span=span):
+ try:
+ pipe_output = await self._live_run_pipe(
+ job_metadata=child_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
+ )
+ except BaseException as exc:
+ # Broad catch is intentional: the OTel span must be closed with ERROR status
+ # on any failure, a cancellation included, which is not an Exception.
+ # Observes-and-re-raises — see note on the catch in _run_pipe_traced.
+ self._end_pipe_span_error(span, error=exc, is_root_span=is_root_span)
+ raise
- # Handle telemetry ------------------------------------------------------------
+ # Handle telemetry ------------------------------------------------------------
- self._end_pipe_span_success(span=span, pipe_output=pipe_output, is_root_span=is_root_span)
+ self._end_pipe_span_success(span=span, pipe_output=pipe_output, is_root_span=is_root_span)
return pipe_output
@@ -1155,17 +1162,18 @@ class PipeAbstract(ABC, BaseModel):
attributes=span_attributes,
)
- # Debug logging
+ # Debug logging, under the span it announces, so the line's trace context names that span
span_ctx = span.get_span_context()
- log.verbose(
- f"[OTel] PIPE SPAN STARTED:\n"
- f" pipe_code='{self.code}'\n"
- f" pipeline_run_id='{pipeline_run_id}'\n"
- f" trace_id={span_ctx.trace_id:032x}\n"
- f" span_id={span_ctx.span_id:016x}\n"
- f" parent_span_id={parent_span_id:016x}\n"
- f" is_root_span={is_root_span}"
- )
+ with pipelex_span_active(span=span):
+ log.verbose(
+ f"[OTel] PIPE SPAN STARTED:\n"
+ f" pipe_code='{self.code}'\n"
+ f" pipeline_run_id='{pipeline_run_id}'\n"
+ f" trace_id={span_ctx.trace_id:032x}\n"
+ f" span_id={span_ctx.span_id:016x}\n"
+ f" parent_span_id={parent_span_id:016x}\n"
+ f" is_root_span={is_root_span}"
+ )
return span, is_root_span
@@ -1203,7 +1211,7 @@ class PipeAbstract(ABC, BaseModel):
span.set_attribute(LangfuseSpanAttr.TRACE_OUTCOME, SpanOutcome.SUCCESS)
span.end()
- def _end_pipe_span_error(self, span: Span | None, *, error: Exception, is_root_span: bool = False) -> None:
+ def _end_pipe_span_error(self, span: Span | None, *, error: BaseException, is_root_span: bool = False) -> None:
"""End the pipe's OTel span with error status. Safe to call if span is None.
Args:
diff --git a/pipelex/providers/bedrock/bedrock_client_boto3.py b/pipelex/providers/bedrock/bedrock_client_boto3.py
index 6bc5ed687..925254d02 100644
--- a/pipelex/providers/bedrock/bedrock_client_boto3.py
+++ b/pipelex/providers/bedrock/bedrock_client_boto3.py
@@ -36,8 +36,10 @@ class BedrockClientBoto3(BedrockClientProtocol):
if system_text:
params["system"] = [{"text": system_text}]
- loop: asyncio.AbstractEventLoop = asyncio.get_event_loop()
- resp_dict: dict[str, Any] = await loop.run_in_executor(None, lambda: self.boto3_client.converse(**params)) # pyright: ignore[reportUnknownLambdaType, reportUnknownMemberType]
+ # ``to_thread`` rather than ``run_in_executor``: it carries the context over, so the SDK's own log
+ # lines in the thread are joined to the LLM span, and a span its instrumentation opens has the
+ # caller's current span as its parent.
+ resp_dict: dict[str, Any] = await asyncio.to_thread(self.boto3_client.converse, **params) # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType]
usage_dict: dict[str, Any] = resp_dict["usage"]
nb_tokens_by_category: NbTokensByCategoryDict = {
diff --git a/pipelex/system/telemetry/current_span.py b/pipelex/system/telemetry/current_span.py
new file mode 100644
index 000000000..743057edf
--- /dev/null
+++ b/pipelex/system/telemetry/current_span.py
@@ -0,0 +1,80 @@
+"""The Pipelex span active in this task, which a log line is joined to.
+
+The runtime starts its spans with an explicit parent carried in the job metadata, because a run's trace
+is derived from its business identifiers and has to survive a process boundary, where an in-process
+context does not travel. A span started that way is current nowhere, and the runtime never makes it
+OpenTelemetry's current span: Pipelex's tracer is its own and never the process's global one, so one of
+its spans made current there would re-parent the host application's own instrumentation under a trace
+only Pipelex's exporters receive, and have a sampler that follows its parent keep it. Instead, the
+runtime holds the span in a context variable of its own while the work it measures runs, in this task
+and in whatever copies the task's context, a thread started by ``asyncio.to_thread`` or a boot line the
+holding handler replays, and restores the previous value on the way out. The log sinks read it to write
+a line's trace context, and nothing else does.
+
+Holding a span never ends it, records an exception on it or sets its status, which stay with the code
+that started it, and it never becomes the way a parent reaches a child: the job metadata still carries
+that, so a child running in another process gets the same parent it always did.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import TYPE_CHECKING
+
+from opentelemetry import trace
+from opentelemetry.context import get_current
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from opentelemetry.context import Context
+ from opentelemetry.trace import Span, SpanContext
+
+# The Pipelex span active here; only ever a span naming a trace, since ``pipelex_span_active`` sets no other.
+_ACTIVE_PIPELEX_SPAN: ContextVar[Span | None] = ContextVar("pipelex_active_span", default=None)
+
+
+@contextmanager
+def pipelex_span_active(*, span: Span | None) -> Generator[None]:
+ """Hold ``span`` as the Pipelex span active here while the block runs; no span, or one naming no trace, holds nothing.
+
+ ``None`` is what the runtime holds with no tracer or in a dry run. A span naming no trace is what a
+ no-op tracer starts, under ``OTEL_SDK_DISABLED`` for one, and holding it would hide the enclosing
+ Pipelex span, or the host's current one, from every line of the block. OpenTelemetry's own context is
+ left as it is either way.
+ """
+ if span is None or not span.get_span_context().is_valid:
+ yield
+ return
+ token = _ACTIVE_PIPELEX_SPAN.set(span)
+ try:
+ yield
+ finally:
+ _ACTIVE_PIPELEX_SPAN.reset(token)
+
+
+def span_context_for_logs() -> SpanContext | None:
+ """The span a line logged here is joined to: the active Pipelex span, else OpenTelemetry's current span, else none.
+
+ The current span is only read, so a line logged under the host's own span, outside any Pipelex run,
+ still joins the host's trace.
+ """
+ pipelex_span = _ACTIVE_PIPELEX_SPAN.get()
+ if pipelex_span is not None:
+ return pipelex_span.get_span_context()
+ current_span_context = trace.get_current_span().get_span_context()
+ if current_span_context.is_valid:
+ return current_span_context
+ return None
+
+
+def otel_context_for_logs() -> Context:
+ """The context a log record is emitted in: the current one with the active Pipelex span in it, or the current one as is.
+
+ Built for the one record and never attached, so OpenTelemetry's current context stays what it was.
+ """
+ pipelex_span = _ACTIVE_PIPELEX_SPAN.get()
+ if pipelex_span is None:
+ return get_current()
+ return trace.set_span_in_context(pipelex_span)
diff --git a/pipelex/tools/log/gcp_log_sink.py b/pipelex/tools/log/gcp_log_sink.py
index 8ad03c6e4..303abd3c4 100644
--- a/pipelex/tools/log/gcp_log_sink.py
+++ b/pipelex/tools/log/gcp_log_sink.py
@@ -50,8 +50,9 @@ GCP_LOGGING_EXTRA_NAME = "gcp-logging"
# The sink writes three of the ``json`` sink's keys into the payload — its ``message``, ``logger`` and
# ``exception``, imported rather than respelled — and reserves that sink's whole set against a carried
# attribute, so one field keeps one wire name whichever of the two a process selects. ``time`` and
-# ``severity`` are not payload keys here, the client library carrying both out of band, but a field
-# named like one is ``field_time`` and ``field_severity`` under either sink rather than under one only.
+# ``severity`` are not payload keys here, the client library carrying both out of band, and neither are
+# ``trace_id``, ``span_id`` and ``trace_flags``, the entry's own ``trace`` field carrying the trace, but a
+# field named like one is ``field_time`` or ``field_trace_id`` under either sink rather than under one only.
FIXED_PAYLOAD_KEYS = FIXED_KEYS
# The stdlib formatter the ``json`` sink renders a traceback through, so both sinks spell one exception
diff --git a/pipelex/tools/log/json_log_sink.py b/pipelex/tools/log/json_log_sink.py
index 3c8dc6a7f..b5a942307 100644
--- a/pipelex/tools/log/json_log_sink.py
+++ b/pipelex/tools/log/json_log_sink.py
@@ -2,12 +2,19 @@
The keys are the ones a log agent ingests without a parser, the CloudWatch agent, the Google Cloud
Logging agent and any OTLP collector among them: ``time``, ``severity``, ``logger``, ``message``, then
-``exception`` when the record carries one, then the fields, the context identifiers and the ``data``
-attribute under their own names. The sink's own keys are reserved whether or not the line carries
-them: a field named like one is carried under the same ``field_`` prefix the record uses for a name
-the stdlib owns, on every line and not only the ones with an exception, so no value is lost and a
-field keeps one wire name. A non-finite float is written as the string ``"NaN"``, ``"Infinity"`` or
-``"-Infinity"``, since JSON has no token for it that a strict parser accepts.
+``exception`` when the record carries one, then ``trace_id``, ``span_id`` and ``trace_flags`` when the
+record was logged inside a valid span, then the fields, the context identifiers and the ``data``
+attribute under their own names. The trace keys are the ones OpenTelemetry specifies for trace context
+in a JSON log that is not OTLP, hex-encoded, so a collector or an error tracker joins the line to its
+trace without a parser. They name the Pipelex span active when the record is formatted, a pipe's or an
+LLM call's, or, outside one, OpenTelemetry's current span, which is only read; the stream handler formats
+inside the log call, in the calling task, and a boot line held until the sink arrives is replayed in the
+context it was logged in, so it keeps its span. The sink's own keys, the trace keys among them, are
+reserved whether or not the line carries them: a field named like one is carried under the same
+``field_`` prefix the record uses for a name the stdlib owns, on every line and not only the ones with
+an exception or a span, so no value is lost and a field keeps one wire name. A non-finite float is
+written as the string ``"NaN"``, ``"Infinity"`` or ``"-Infinity"``, since JSON has no token for it that
+a strict parser accepts.
"""
from __future__ import annotations
@@ -18,8 +25,10 @@ import sys
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
+from opentelemetry import trace
from typing_extensions import override
+from pipelex.system.telemetry.current_span import span_context_for_logs
from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes
from pipelex.tools.log.log_sink import LogSink, json_fallback, spell_non_finite
@@ -31,8 +40,11 @@ SEVERITY_KEY = "severity"
LOGGER_KEY = "logger"
MESSAGE_KEY = "message"
EXCEPTION_KEY = "exception"
+TRACE_ID_KEY = "trace_id"
+SPAN_ID_KEY = "span_id"
+TRACE_FLAGS_KEY = "trace_flags"
# The keys the sink writes itself, reserved on every line; a carried attribute of the same name is prefixed.
-FIXED_KEYS = frozenset({TIME_KEY, SEVERITY_KEY, LOGGER_KEY, MESSAGE_KEY, EXCEPTION_KEY})
+FIXED_KEYS = frozenset({TIME_KEY, SEVERITY_KEY, LOGGER_KEY, MESSAGE_KEY, EXCEPTION_KEY, TRACE_ID_KEY, SPAN_ID_KEY, TRACE_FLAGS_KEY})
def _json_line(*, payload: dict[str, Any]) -> str:
@@ -51,6 +63,18 @@ def _json_line(*, payload: dict[str, Any]) -> str:
return json.dumps(safe_payload, ensure_ascii=False, default=str)
+def _trace_context() -> dict[str, str]:
+ """The trace context of the span a line logged here is joined to, under the OpenTelemetry JSON keys, or nothing outside one."""
+ span_context = span_context_for_logs()
+ if span_context is None:
+ return {}
+ return {
+ TRACE_ID_KEY: trace.format_trace_id(span_context.trace_id),
+ SPAN_ID_KEY: trace.format_span_id(span_context.span_id),
+ TRACE_FLAGS_KEY: f"{span_context.trace_flags:02x}",
+ }
+
+
def _iso_utc(*, created: float) -> str:
"""The record's creation time as ISO 8601 in UTC with millisecond precision and a ``Z`` suffix."""
return datetime.fromtimestamp(created, tz=UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")
@@ -71,6 +95,7 @@ class JsonLogFormatter(logging.Formatter):
payload[EXCEPTION_KEY] = record.exc_text or self.formatException(record.exc_info)
elif record.exc_text:
payload[EXCEPTION_KEY] = record.exc_text
+ payload.update(_trace_context())
for name, value in carried_attributes(record=record).items():
key = name
while key in payload or key in FIXED_KEYS:
diff --git a/pipelex/tools/log/log_holding.py b/pipelex/tools/log/log_holding.py
index ad3924b82..597d77907 100644
--- a/pipelex/tools/log/log_holding.py
+++ b/pipelex/tools/log/log_holding.py
@@ -13,6 +13,7 @@ over each of those records runs over it on that path too.
from __future__ import annotations
+import contextvars
import logging
from typing_extensions import override
@@ -23,6 +24,9 @@ from pipelex.tools.log.log_fields import FORWARDED_MARK
# logging and never installs a sink cannot grow without bound. A boot holds a few dozen lines.
HOLDING_CAPACITY = 1000
+# A held record and a copy of the context it was emitted in, which its replay runs under.
+HeldRecord = tuple[logging.LogRecord, contextvars.Context]
+
class ForwardedRecordFilter(logging.Filter):
"""Rejects a record the holding handler already forwarded to this handler.
@@ -72,7 +76,7 @@ class HoldingLogHandler(logging.Handler):
def __init__(self, *, last_resort_filter: logging.Filter | None = None) -> None:
super().__init__(level=logging.NOTSET)
- self._held: list[logging.LogRecord] = []
+ self._held: list[HeldRecord] = []
self._released_to: logging.Handler | None = None
self._last_resort_filter = last_resort_filter
@@ -93,7 +97,7 @@ class HoldingLogHandler(logging.Handler):
return
if len(self._held) >= HOLDING_CAPACITY:
del self._held[0]
- self._held.append(record)
+ self._held.append((record, contextvars.copy_context()))
def release_to(self, *, handler: logging.Handler) -> None:
"""Hand every held record to the handler, in the order they were emitted, and forward to it whatever arrives after.
@@ -105,13 +109,18 @@ class HoldingLogHandler(logging.Handler):
reason: a thread that read the root logger's handler list before the handoff can still reach the
sink's handler carrying a record this drain has already delivered, and the mark is the only thing
that tells the handler's guard so.
+
+ Each record is delivered in the context it was emitted in, copied when it was held, so a sink that
+ reads the context at emit, the span a trace-joining sink writes, Pipelex's own or OpenTelemetry's
+ current one, reads the one the record was logged under rather than the one the boot holds at the
+ handoff, and a record another thread logged inside a span of its own keeps that span too.
"""
self.acquire()
try:
held, self._held = self._held, []
self._released_to = handler
- for record in held:
- _deliver(handler=handler, record=record)
+ for record, emitted_in in held:
+ emitted_in.run(_deliver, handler=handler, record=record)
setattr(record, FORWARDED_MARK, True)
finally:
self.release()
@@ -136,7 +145,7 @@ class HoldingLogHandler(logging.Handler):
self.release()
last_resort = logging.lastResort
if last_resort is not None:
- for record in held:
+ for record, _emitted_in in held:
if self._last_resort_filter is not None and not self._last_resort_filter.filter(record):
continue
last_resort.handle(record)
diff --git a/pipelex/tools/log/otlp_log_sink.py b/pipelex/tools/log/otlp_log_sink.py
index fd33ecf53..83d12bfe1 100644
--- a/pipelex/tools/log/otlp_log_sink.py
+++ b/pipelex/tools/log/otlp_log_sink.py
@@ -8,9 +8,11 @@ rendered exception text so that what the redaction processor scrubbed is what le
semantic-convention keys the sink writes itself — the source location and the exception — are reserved
whether or not the record carries an exception, exactly as the ``json`` sink reserves its own keys: a
field named like one is carried under the same ``field_`` prefix, so the same field survives a change of
-sink. The records the sink's own export path emits, the SDK's and the transport's, are rejected by a
-filter on the handler and never exported. This module imports the OpenTelemetry SDK at load, which is why
-the built-in plugin imports it inside the ``otlp`` factory and nowhere else.
+sink. Each record is filed under the Pipelex span active when it was logged, a pipe's or an LLM
+call's, or, outside one, under OpenTelemetry's current span, which is only read. The records the sink's
+own export path emits, the SDK's and the transport's, are rejected by a filter on the handler and never
+exported. This module imports the OpenTelemetry SDK at load, which is why the built-in plugin imports it
+inside the ``otlp`` factory and nowhere else.
"""
from __future__ import annotations
@@ -23,7 +25,6 @@ from typing import TYPE_CHECKING, Any, cast
from opentelemetry._logs import SeverityNumber # ruff: ignore[import-private-name]
from opentelemetry.context import (
_SUPPRESS_INSTRUMENTATION_KEY, # ruff: ignore[import-private-name] # pyright: ignore[reportPrivateUsage]
- get_current,
get_value,
)
from opentelemetry.sdk._logs import LoggerProvider # ruff: ignore[import-private-name]
@@ -31,6 +32,7 @@ from opentelemetry.semconv._incubating.attributes import code_attributes # ruff
from opentelemetry.semconv.attributes import exception_attributes
from typing_extensions import override
+from pipelex.system.telemetry.current_span import otel_context_for_logs
from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes
from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE
from pipelex.tools.log.log_sink import LogSink, render_json
@@ -144,7 +146,7 @@ class OtlpLogHandler(logging.Handler):
logger.emit(
timestamp=int(record.created * 1e9),
observed_timestamp=time_ns(),
- context=get_current() or None,
+ context=otel_context_for_logs(),
severity_text=_severity_text(levelname=record.levelname),
severity_number=_severity_number(levelno=record.levelno),
body=record.getMessage(),
diff --git a/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py b/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
new file mode 100644
index 000000000..e0094d290
--- /dev/null
+++ b/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
@@ -0,0 +1,227 @@
+"""The LLM worker holds its generation span as the Pipelex span active from its start to its end.
+
+The span is started with the pipe's span as its explicit parent, read off the job metadata, and that
+stays as it was; what changes is that the provider call runs with the span held, so a log line during
+the call is joined to it, and the previous one comes back when the span ends, whichever way it ends.
+OpenTelemetry's current span is never touched: inside the call it is whatever it was outside, the host's
+or none, so a provider SDK's own instrumentation is never re-parented under the LLM span. Driven through
+the public ``gen_text`` and ``gen_object`` on a worker whose provider half records the spans it runs under.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+import pytest
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
+from opentelemetry.trace import INVALID_SPAN_ID, StatusCode
+from pydantic import BaseModel
+from typing_extensions import override
+
+from pipelex.cogt.exceptions import LLMCompletionError
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
+from pipelex.system.telemetry.current_span import span_context_for_logs
+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opentelemetry.sdk.trace import ReadableSpan
+ from pytest_mock import MockerFixture
+
+ from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+
+# The logger the worker's own span lines are written on.
+LLM_WORKER_LOGGER = LLMWorkerAbstract.__module__
+
+RUN_TRACE_ID = 0x0123456789ABCDEF0123456789ABCDEF
+PIPE_SPAN_ID = 0x00000000000000AB
+
+
+class _Answer(BaseModel):
+ text: str
+
+
+class _RecordingLLMWorker(LLMWorkerAbstract):
+ """A worker whose provider half records the span a log line is joined to and OpenTelemetry's current span, and fails when asked to."""
+
+ def __init__(self, *, inference_model: InferenceModelSpec, fails: bool) -> None:
+ LLMWorkerAbstract.__init__(self, inference_model=inference_model, reporting_delegate=None)
+ self.fails = fails
+ self.seen_log_span_ids: list[int] = []
+ self.seen_current_spans: list[object] = []
+
+ def _record(self) -> None:
+ log_span_context = span_context_for_logs()
+ self.seen_log_span_ids.append(INVALID_SPAN_ID if log_span_context is None else log_span_context.span_id)
+ self.seen_current_spans.append(trace.get_current_span())
+ if self.fails:
+ msg = "provider refused"
+ raise LLMCompletionError(message=msg)
+
+ @override
+ async def _gen_text(self, llm_job: LLMJob) -> str:
+ self._record()
+ return "answer"
+
+ @override
+ async def _gen_object(self, llm_job: LLMJob, *, schema: type[BaseModelTypeVar]) -> BaseModelTypeVar:
+ self._record()
+ return schema.model_validate({"text": "answer"})
+
+
+class _SpanRecordingHandler(logging.Handler):
+ """Records each message with the span id a sink would write for it, read at emit as a sink reads it."""
+
+ def __init__(self) -> None:
+ super().__init__(level=logging.NOTSET)
+ self.span_ids_by_message: dict[str, int] = {}
+
+ @override
+ def emit(self, record: logging.LogRecord) -> None:
+ span_context = span_context_for_logs()
+ self.span_ids_by_message[record.getMessage()] = INVALID_SPAN_ID if span_context is None else span_context.span_id
+
+
+def _make_worker(*, fails: bool = False) -> _RecordingLLMWorker:
+ inference_model = InferenceModelSpec(
+ backend_name="openai",
+ name="gpt-test",
+ sdk="openai",
+ model_type=ModelType.LLM,
+ model_id="gpt-test-id",
+ inputs=["text"],
+ outputs=["text", "structured"],
+ costs={CostCategory.INPUT: 1, CostCategory.OUTPUT: 2},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+ return _RecordingLLMWorker(inference_model=inference_model, fails=fails)
+
+
+def _make_llm_job() -> LLMJob:
+ """A job run under a pipe whose span the metadata names as the parent."""
+ job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-llm", storage_scope="test/scope"),
+ pipe_code="some_pipe",
+ otel_context=OtelContext(trace_id=RUN_TRACE_ID, trace_name="some_pipe", trace_name_redacted="some_pipe", span_id=PIPE_SPAN_ID),
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(user_text="hello"),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(schema_reask_max_attempts=1),
+ )
+
+
+def _only_span(exporter: InMemorySpanExporter) -> ReadableSpan:
+ (span,) = exporter.get_finished_spans()
+ return span
+
+
+def _span_id(span: ReadableSpan) -> int:
+ span_context = span.get_span_context()
+ assert span_context is not None
+ return int(span_context.span_id)
+
+
+def span_exporter(mocker: MockerFixture) -> InMemorySpanExporter:
+ """The runtime's tracer, replaced by one exporting to memory, so the test reads back the span the worker started."""
+ exporter = InMemorySpanExporter()
+ provider = TracerProvider()
+ provider.add_span_processor(SimpleSpanProcessor(exporter))
+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=provider.get_tracer(__name__))
+ return exporter
+
+
+def worker_lines(caplog: pytest.LogCaptureFixture) -> Iterator[_SpanRecordingHandler]:
+ """The worker's own lines at VERBOSE, each with the span a sink would join it to."""
+ caplog.set_level(LOGGING_LEVEL_VERBOSE, logger=LLM_WORKER_LOGGER)
+ handler = _SpanRecordingHandler()
+ logger = logging.getLogger(LLM_WORKER_LOGGER)
+ logger.addHandler(handler)
+ try:
+ yield handler
+ finally:
+ logger.removeHandler(handler)
+
+
+class TestLLMWorkerCurrentSpan:
+ async def test_gen_text_runs_the_provider_call_under_its_span(self, span_exporter: InMemorySpanExporter) -> None:
+ worker = _make_worker()
+
+ assert await worker.gen_text(llm_job=_make_llm_job()) == "answer"
+
+ llm_span = _only_span(span_exporter)
+ assert worker.seen_log_span_ids == [_span_id(llm_span)]
+ assert llm_span.parent is not None
+ assert llm_span.parent.span_id == PIPE_SPAN_ID
+ assert span_context_for_logs() is None
+
+ async def test_gen_object_runs_the_provider_call_under_its_span(self, span_exporter: InMemorySpanExporter) -> None:
+ worker = _make_worker()
+
+ assert await worker.gen_object(llm_job=_make_llm_job(), schema=_Answer) == _Answer(text="answer")
+
+ llm_span = _only_span(span_exporter)
+ assert worker.seen_log_span_ids == [_span_id(llm_span)]
+ assert span_context_for_logs() is None
+
+ async def test_the_call_never_makes_its_span_opentelemetrys_current_one(self, span_exporter: InMemorySpanExporter) -> None:
+ """Inside the call the current span is whatever it was outside, so the provider SDK's instrumentation is never re-parented."""
+ host_tracer = TracerProvider().get_tracer(__name__)
+ without_host = _make_worker()
+ under_host = _make_worker()
+
+ await without_host.gen_text(llm_job=_make_llm_job())
+ with host_tracer.start_as_current_span("host") as host_span:
+ await under_host.gen_object(llm_job=_make_llm_job(), schema=_Answer)
+
+ assert without_host.seen_current_spans == [trace.INVALID_SPAN]
+ assert under_host.seen_current_spans == [host_span]
+ # A line logged in the call still names the LLM span, not the host's.
+ assert under_host.seen_log_span_ids == [_span_id(span_exporter.get_finished_spans()[-1])]
+
+ async def test_a_failed_call_restores_the_previous_span_and_is_recorded_once(self, span_exporter: InMemorySpanExporter) -> None:
+ worker = _make_worker(fails=True)
+
+ with pytest.raises(LLMCompletionError, match="provider refused"):
+ await worker.gen_text(llm_job=_make_llm_job())
+
+ llm_span = _only_span(span_exporter)
+ assert worker.seen_log_span_ids == [_span_id(llm_span)]
+ assert span_context_for_logs() is None
+ # The worker's own error path records the failure; holding the span records nothing more.
+ assert llm_span.status.status_code is StatusCode.ERROR
+ assert [event.name for event in llm_span.events] == ["exception"]
+
+ async def test_the_span_started_line_names_the_span_it_announces(
+ self, span_exporter: InMemorySpanExporter, worker_lines: _SpanRecordingHandler
+ ) -> None:
+ """The verbose line the worker logs once its span has started carries that span's id, not the one enclosing the call."""
+ worker = _make_worker()
+
+ await worker.gen_text(llm_job=_make_llm_job())
+
+ llm_span_id = _span_id(_only_span(span_exporter))
+ started = {message: span_id for message, span_id in worker_lines.span_ids_by_message.items() if message.startswith("[OTel] LLM SPAN STARTED")}
+ assert list(started.values()) == [llm_span_id]
+ (message,) = started
+ assert f"\n span_id={llm_span_id:016x}\n" in message
diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
new file mode 100644
index 000000000..469cc1564
--- /dev/null
+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
@@ -0,0 +1,401 @@
+"""``live_run_pipe`` holds the pipe's span as the Pipelex span active from its start to its end, so the run's log lines carry it.
+
+The span is started with an explicit parent read off the job metadata, which is how a child running in
+another process reaches it, and that stays as it was: the pipe only holds its already-started span for
+the in-process work, and restores the previous one when the span ends, however it ends. Both wire sinks
+read the active Pipelex span, so a line logged during the run names the pipe's span: the ``json`` sink
+under ``trace_id`` and ``span_id``, the ``otlp`` sink as the record's own trace context. OpenTelemetry's
+current span is never touched: inside the run it is whatever it was outside, the host's or none, so the
+host's own instrumentation is never re-parented under the pipe. Each test runs once per sink.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import io
+import json
+import logging
+from typing import TYPE_CHECKING, Any, NamedTuple
+
+import pytest
+from opentelemetry import trace
+from opentelemetry.sdk._logs.export import InMemoryLogExporter, SimpleLogRecordProcessor
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
+from opentelemetry.trace import INVALID_SPAN_ID, StatusCode
+from pydantic import Field, PrivateAttr
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.system.telemetry.current_span import span_context_for_logs
+from pipelex.system.telemetry.otel_constants import OTelConstants
+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
+from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, SPAN_ID_KEY, TRACE_ID_KEY, JsonLogSink
+from pipelex.tools.log.log import Log
+from pipelex.tools.log.log_config import LogConfig
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
+from pipelex.tools.log.otlp_log_sink import OtlpLogSink
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+if TYPE_CHECKING:
+ from collections.abc import Callable, Iterator
+
+ from opentelemetry.sdk.trace import ReadableSpan
+ from pytest_mock import MockerFixture
+
+ from pipelex.core.memory.working_memory import WorkingMemory
+ from pipelex.libraries.library_crate import LibraryCrate
+
+# The logger the pipe's own span lines are written on.
+PIPE_ABSTRACT_LOGGER = PipeAbstract.__module__
+
+# The trace the submission derived for the run, and the virtual root parent every root pipe span takes.
+RUN_TRACE_ID = 0x0123456789ABCDEF0123456789ABCDEF
+RUN_OTEL_CONTEXT = OtelContext(
+ trace_id=RUN_TRACE_ID,
+ trace_name="current_span_test",
+ trace_name_redacted="current_span_test",
+ span_id=OTelConstants.OTEL_VIRTUAL_ROOT_PARENT_SPAN_ID,
+)
+
+
+class LineTrace(NamedTuple):
+ """The trace context a sink wrote for one line, as hex, or ``None`` where the line carries none."""
+
+ trace_id: str | None
+ span_id: str | None
+
+
+class Sunk(NamedTuple):
+ """A fresh ``Log`` with a sink installed, and how to read back the trace context of each line a logger wrote, this module's by default."""
+
+ log: Log
+ read_lines_of: Callable[[str], dict[str, LineTrace]]
+
+ def read_lines(self) -> dict[str, LineTrace]:
+ return self.read_lines_of(__name__)
+
+
+class CurrentSpanPipe(PipeAbstract):
+ """A pipe whose live run logs a line and records the spans it runs under; it may nest a pipe, and it may fail.
+
+ It records the span a line logged there is joined to, and OpenTelemetry's current span, which is what the
+ host's own instrumentation would open its spans under.
+ """
+
+ pipe_category: Any = "PipeOperator"
+ type: Any = "PipeFunc"
+ nested: CurrentSpanPipe | None = None
+ fails: bool = False
+ hangs: bool = False
+ seen_log_span_ids: list[int] = Field(default_factory=empty_list_factory_of(int))
+ seen_current_spans: list[object] = Field(default_factory=empty_list_factory_of(object))
+ _hanging: asyncio.Event = PrivateAttr(default_factory=asyncio.Event)
+
+ @property
+ def hanging(self) -> asyncio.Event:
+ """Set once a hanging pipe has logged its line and started to wait."""
+ return self._hanging
+
+ @override
+ def validate_inputs_with_library(self) -> None: ...
+
+ @override
+ def validate_inputs_static(self) -> None: ...
+
+ @override
+ def validate_output_with_library(self) -> None: ...
+
+ @override
+ def validate_output_static(self) -> None: ...
+
+ @override
+ async def _validate_before_run(self, **kwargs: Any) -> None: ...
+
+ @override
+ async def _validate_after_run(self, **kwargs: Any) -> None: ...
+
+ @override
+ def required_variables(self) -> set[str]:
+ return set()
+
+ @override
+ def needed_inputs(self, *, visited_pipes: set[str] | None = None) -> Any:
+ return self.inputs
+
+ @override
+ async def _live_run_pipe(
+ self,
+ *,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ library_crate: LibraryCrate | None = None,
+ ) -> PipeOutput:
+ log_span_context = span_context_for_logs()
+ self.seen_log_span_ids.append(INVALID_SPAN_ID if log_span_context is None else log_span_context.span_id)
+ self.seen_current_spans.append(trace.get_current_span())
+ log.info(f"inside {self.code}")
+ if self.nested is not None:
+ await self.nested.live_run_pipe(job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params)
+ log.info(f"back in {self.code}")
+ if self.fails:
+ msg = f"{self.code} failed"
+ raise RuntimeError(msg)
+ if self.hangs:
+ self._hanging.set()
+ await asyncio.Event().wait()
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)
+
+ @override
+ async def _dry_run_pipe(
+ self,
+ *,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ library_crate: LibraryCrate | None = None,
+ ) -> PipeOutput:
+ raise NotImplementedError
+
+
+def _make_pipe(*, code: str, nested: CurrentSpanPipe | None = None, fails: bool = False, hangs: bool = False) -> CurrentSpanPipe:
+ return CurrentSpanPipe(
+ code=code,
+ domain_code="test_current_span",
+ description=code,
+ output=StuffSpec(concept=ConceptFactory.make_native_concept(NativeConceptCode.TEXT)),
+ nested=nested,
+ fails=fails,
+ hangs=hangs,
+ )
+
+
+def _traced_metadata() -> JobMetadata:
+ """What a submission builds when the runtime traces: the run's identifiers and its derived trace."""
+ return JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", pipeline_run_id="plr-span"),
+ otel_context=RUN_OTEL_CONTEXT,
+ )
+
+
+def _live_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, batch_max_concurrency=1, pipe_stack_limit=10)
+
+
+async def _run(pipe: CurrentSpanPipe) -> None:
+ await pipe.live_run_pipe(job_metadata=_traced_metadata(), working_memory=WorkingMemoryFactory.make_empty(), pipe_run_params=_live_params())
+
+
+def _package_log_config() -> LogConfig:
+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")
+ return LogConfig.model_validate(config_dict["runtime"]["log"])
+
+
+def _json_lines(buffer: io.StringIO, *, logger_name: str) -> dict[str, LineTrace]:
+ lines = [json.loads(line) for line in buffer.getvalue().splitlines() if line]
+ return {
+ line[MESSAGE_KEY]: LineTrace(trace_id=line.get(TRACE_ID_KEY), span_id=line.get(SPAN_ID_KEY))
+ for line in lines
+ if line[LOGGER_KEY] == logger_name
+ }
+
+
+def _otlp_lines(exporter: InMemoryLogExporter, *, logger_name: str) -> dict[str, LineTrace]:
+ lines: dict[str, LineTrace] = {}
+ for log_data in exporter.get_finished_logs():
+ if log_data.instrumentation_scope.name != logger_name:
+ continue
+ record = log_data.log_record
+ trace_id = f"{record.trace_id:032x}" if record.trace_id else None
+ span_id = f"{record.span_id:016x}" if record.span_id else None
+ lines[str(record.body)] = LineTrace(trace_id=trace_id, span_id=span_id)
+ return lines
+
+
+def _span_id(span: ReadableSpan) -> int:
+ span_context = span.get_span_context()
+ assert span_context is not None
+ return int(span_context.span_id)
+
+
+def _hex_span_id(span: ReadableSpan) -> str:
+ return f"{_span_id(span):016x}"
+
+
+def _span_named(spans: tuple[ReadableSpan, ...], *, code: str) -> ReadableSpan:
+ (span,) = [span for span in spans if span.name.endswith(f": {code}")]
+ return span
+
+
+class TestLiveRunPipeCurrentSpan:
+ @pytest.fixture(params=["json", "otlp"])
+ def sunk(self, request: pytest.FixtureRequest, caplog: pytest.LogCaptureFixture) -> Iterator[Sunk]:
+ """A fresh ``Log`` with the sink under test installed, torn down so the root logger is left as found."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ fresh = Log()
+ fresh.configure(log_config=_package_log_config())
+ read_lines_of: Callable[[str], dict[str, LineTrace]]
+ if request.param == "json":
+ buffer = io.StringIO()
+ fresh.install_sink(JsonLogSink(stream=buffer))
+
+ def read_json_lines(logger_name: str) -> dict[str, LineTrace]:
+ return _json_lines(buffer, logger_name=logger_name)
+
+ read_lines_of = read_json_lines
+ else:
+ exporter = InMemoryLogExporter()
+ fresh.install_sink(OtlpLogSink(processor=SimpleLogRecordProcessor(exporter)))
+
+ def read_otlp_lines(logger_name: str) -> dict[str, LineTrace]:
+ return _otlp_lines(exporter, logger_name=logger_name)
+
+ read_lines_of = read_otlp_lines
+
+ try:
+ yield Sunk(log=fresh, read_lines_of=read_lines_of)
+ finally:
+ fresh.reset()
+
+ @pytest.fixture
+ def span_exporter(self, mocker: MockerFixture) -> InMemorySpanExporter:
+ """The runtime's tracer, replaced by one exporting to memory, so the test reads back the spans the pipes started."""
+ exporter = InMemorySpanExporter()
+ provider = TracerProvider()
+ provider.add_span_processor(SimpleSpanProcessor(exporter))
+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=provider.get_tracer(__name__))
+ return exporter
+
+ async def test_a_line_logged_during_the_run_carries_the_pipes_span(self, sunk: Sunk, span_exporter: InMemorySpanExporter) -> None:
+ pipe = _make_pipe(code="solo")
+
+ await _run(pipe)
+
+ pipe_span = _span_named(span_exporter.get_finished_spans(), code="solo")
+ assert sunk.read_lines()["inside solo"] == LineTrace(trace_id=f"{RUN_TRACE_ID:032x}", span_id=_hex_span_id(pipe_span))
+ assert pipe.seen_log_span_ids == [_span_id(pipe_span)]
+ assert span_context_for_logs() is None
+ log.info("after the run")
+ assert sunk.read_lines()["after the run"] == LineTrace(trace_id=None, span_id=None)
+
+ async def test_the_run_never_makes_its_span_opentelemetrys_current_one(self, sunk: Sunk, span_exporter: InMemorySpanExporter) -> None:
+ """Inside the run the current span is whatever it was outside, so the host's instrumentation is never re-parented."""
+ host_tracer = TracerProvider().get_tracer(__name__)
+ without_host = _make_pipe(code="hostless")
+ under_host = _make_pipe(code="hosted")
+
+ await _run(without_host)
+ with host_tracer.start_as_current_span("host") as host_span:
+ await _run(under_host)
+
+ assert without_host.seen_current_spans == [trace.INVALID_SPAN]
+ assert under_host.seen_current_spans == [host_span]
+ # The line still names the pipe's span, not the host's.
+ pipe_span = _span_named(span_exporter.get_finished_spans(), code="hosted")
+ assert sunk.read_lines()["inside hosted"] == LineTrace(trace_id=f"{RUN_TRACE_ID:032x}", span_id=_hex_span_id(pipe_span))
+
+ async def test_a_span_started_line_names_the_span_it_announces(
+ self, sunk: Sunk, span_exporter: InMemorySpanExporter, caplog: pytest.LogCaptureFixture
+ ) -> None:
+ """The verbose line a pipe logs once its span has started carries that span's ids, not the enclosing pipe's or none."""
+ caplog.set_level(LOGGING_LEVEL_VERBOSE, logger=PIPE_ABSTRACT_LOGGER)
+ outer = _make_pipe(code="outer", nested=_make_pipe(code="inner"))
+
+ await _run(outer)
+
+ spans = span_exporter.get_finished_spans()
+ started = {
+ message: line for message, line in sunk.read_lines_of(PIPE_ABSTRACT_LOGGER).items() if message.startswith("[OTel] PIPE SPAN STARTED")
+ }
+ assert sorted(line.span_id or "" for line in started.values()) == sorted(
+ _hex_span_id(_span_named(spans, code=code)) for code in ("outer", "inner")
+ )
+ for message, line in started.items():
+ assert f"\n span_id={line.span_id}\n" in message
+
+ async def test_a_nested_pipe_holds_its_own_span_and_the_outer_one_comes_back(self, sunk: Sunk, span_exporter: InMemorySpanExporter) -> None:
+ outer = _make_pipe(code="outer", nested=_make_pipe(code="inner"))
+
+ await _run(outer)
+
+ spans = span_exporter.get_finished_spans()
+ outer_span = _span_named(spans, code="outer")
+ inner_span = _span_named(spans, code="inner")
+ lines = sunk.read_lines()
+ run_trace = f"{RUN_TRACE_ID:032x}"
+ assert lines["inside outer"] == LineTrace(trace_id=run_trace, span_id=_hex_span_id(outer_span))
+ assert lines["inside inner"] == LineTrace(trace_id=run_trace, span_id=_hex_span_id(inner_span))
+ assert lines["back in outer"] == LineTrace(trace_id=run_trace, span_id=_hex_span_id(outer_span))
+ # The nested span's parent still arrives through the job metadata, exactly as it did.
+ assert inner_span.parent is not None
+ assert inner_span.parent.span_id == _span_id(outer_span)
+
+ async def test_a_failing_pipe_keeps_its_span_through_the_failure_and_records_it_once(
+ self, sunk: Sunk, span_exporter: InMemorySpanExporter
+ ) -> None:
+ pipe = _make_pipe(code="failing", fails=True)
+
+ with pytest.raises(RuntimeError, match="failing failed"):
+ await _run(pipe)
+
+ pipe_span = _span_named(span_exporter.get_finished_spans(), code="failing")
+ assert sunk.read_lines()["inside failing"].span_id == _hex_span_id(pipe_span)
+ assert span_context_for_logs() is None
+ # The pipe's own error hook records the failure; holding the span records nothing more.
+ assert pipe_span.status.status_code is StatusCode.ERROR
+ assert [event.name for event in pipe_span.events] == ["exception"]
+
+ async def test_without_a_tracer_a_line_names_the_callers_span(self, sunk: Sunk, mocker: MockerFixture) -> None:
+ """No tracer, no pipe span: nothing is held, so a line names whatever span the caller runs under."""
+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=None)
+ caller_tracer = TracerProvider().get_tracer(__name__)
+ pipe = _make_pipe(code="untraced")
+
+ with caller_tracer.start_as_current_span("caller") as caller_span:
+ await _run(pipe)
+ caller_context = caller_span.get_span_context()
+
+ assert sunk.read_lines()["inside untraced"] == LineTrace(trace_id=f"{caller_context.trace_id:032x}", span_id=f"{caller_context.span_id:016x}")
+ assert pipe.seen_current_spans == [caller_span]
+
+ async def test_a_cancelled_pipe_ends_its_span(self, sunk: Sunk, span_exporter: InMemorySpanExporter) -> None:
+ """A cancellation is not an ``Exception``; the span still ends, so the lines of the run name a span the backend receives."""
+ pipe = _make_pipe(code="cancelled", hangs=True)
+ task = asyncio.create_task(_run(pipe))
+ await pipe.hanging.wait()
+
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+
+ pipe_span = _span_named(span_exporter.get_finished_spans(), code="cancelled")
+ assert pipe_span.status.status_code is StatusCode.ERROR
+ assert sunk.read_lines()["inside cancelled"].span_id == _hex_span_id(pipe_span)
+
+ async def test_a_no_op_tracer_leaves_the_callers_span_to_the_line(self, sunk: Sunk, mocker: MockerFixture) -> None:
+ """A no-op tracer, what the SDK hands out under ``OTEL_SDK_DISABLED``, starts spans naming no trace, which must not hide the caller's."""
+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=trace.NoOpTracer())
+ caller_tracer = TracerProvider().get_tracer(__name__)
+ pipe = _make_pipe(code="no_op")
+
+ with caller_tracer.start_as_current_span("caller") as caller_span:
+ await _run(pipe)
+ caller_context = caller_span.get_span_context()
+
+ assert sunk.read_lines()["inside no_op"] == LineTrace(trace_id=f"{caller_context.trace_id:032x}", span_id=f"{caller_context.span_id:016x}")
diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
index fa2a2f30f..cb0081957 100644
--- a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
@@ -124,7 +124,7 @@ class RecordingPipe(PipeAbstract):
self._see(moment="succeeded")
@override
- def _end_pipe_span_error(self, span: Any, *, error: Exception, is_root_span: bool = False) -> None:
+ def _end_pipe_span_error(self, span: Any, *, error: BaseException, is_root_span: bool = False) -> None:
self._see(moment="failed")
log.info(f"{self.code} span ends in error")
diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
new file mode 100644
index 000000000..80e01f4f5
--- /dev/null
+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
@@ -0,0 +1,50 @@
+"""The boto3 Bedrock client runs its blocking call in a thread that keeps the caller's context.
+
+The LLM worker holds its generation span as the Pipelex span active around the provider call, and the
+promise is that the SDK's own log lines during the call are joined to it, while a span the SDK's
+instrumentation opens has the caller's current span as its parent. The boto3 client blocks, so it runs in
+a thread, and only a thread that carries the context over keeps both there.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.trace import INVALID_SPAN_ID
+
+from pipelex.providers.bedrock.bedrock_client_boto3 import BedrockClientBoto3
+from pipelex.system.telemetry.current_span import pipelex_span_active, span_context_for_logs
+
+
+class _RecordingConverse:
+ """Stands in for the boto3 runtime client: records the spans the thread that runs ``converse`` sees."""
+
+ def __init__(self) -> None:
+ self.seen_log_span_ids: list[int] = []
+ self.seen_current_spans: list[object] = []
+
+ def converse(self, **_params: Any) -> dict[str, Any]:
+ log_span_context = span_context_for_logs()
+ self.seen_log_span_ids.append(INVALID_SPAN_ID if log_span_context is None else log_span_context.span_id)
+ self.seen_current_spans.append(trace.get_current_span())
+ return {"usage": {"inputTokens": 1, "outputTokens": 2}, "output": {"message": {"content": [{"text": "answer"}]}}}
+
+
+class TestBedrockBoto3CurrentSpan:
+ async def test_the_blocking_call_runs_under_the_callers_spans(self) -> None:
+ client = BedrockClientBoto3(aws_region="us-east-1")
+ recording = _RecordingConverse()
+ client.boto3_client = recording
+ tracer = TracerProvider().get_tracer(__name__)
+ llm_span = tracer.start_span("llm call")
+
+ with tracer.start_as_current_span("host") as host_span, pipelex_span_active(span=llm_span):
+ text, _ = await client.chat(messages=[], system_text=None, model="some-model", temperature=0.5)
+
+ assert text == "answer"
+ assert recording.seen_log_span_ids == [llm_span.get_span_context().span_id]
+ assert recording.seen_current_spans == [host_span]
diff --git a/tests/unit/pipelex/system/telemetry/test_current_span.py b/tests/unit/pipelex/system/telemetry/test_current_span.py
new file mode 100644
index 000000000..9875e2879
--- /dev/null
+++ b/tests/unit/pipelex/system/telemetry/test_current_span.py
@@ -0,0 +1,106 @@
+"""``pipelex_span_active`` holds a Pipelex span for a block, and the log readers join a line to it.
+
+A line logged in the block is joined to the Pipelex span, and outside any Pipelex span to OpenTelemetry's
+current span, which is only read. OpenTelemetry's current context is never touched: inside the block the
+current span is whatever it was outside, the host's or none. The span stays owned by the code that
+started it: the block never ends it, never records an exception on it and never sets its status, even
+when an exception crosses the block while the span is still recording, and the previous Pipelex span
+comes back on the way out, however the block exits. A ``None`` span, which is what the runtime holds
+when it has no tracer, and a span naming no trace, which is what a no-op tracer starts, hold nothing.
+"""
+
+from __future__ import annotations
+
+import pytest
+from opentelemetry import trace
+from opentelemetry.sdk.trace import Span as SdkSpan
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.trace import StatusCode
+
+from pipelex.system.telemetry.current_span import otel_context_for_logs, pipelex_span_active, span_context_for_logs
+
+
+def _started_span(*, name: str) -> SdkSpan:
+ span = TracerProvider().get_tracer(__name__).start_span(name)
+ assert isinstance(span, SdkSpan)
+ return span
+
+
+def _fail_under(*, span: SdkSpan) -> None:
+ with pipelex_span_active(span=span):
+ msg = "boom"
+ raise ValueError(msg)
+
+
+class TestPipelexSpanActive:
+ def test_a_line_joins_the_pipelex_span_inside_the_block_and_the_previous_one_after(self) -> None:
+ outer = _started_span(name="outer")
+ inner = _started_span(name="inner")
+
+ with pipelex_span_active(span=outer):
+ with pipelex_span_active(span=inner):
+ assert span_context_for_logs() == inner.get_span_context()
+ assert span_context_for_logs() == outer.get_span_context()
+ assert span_context_for_logs() is None
+ assert inner.is_recording()
+
+ def test_the_pipelex_span_takes_precedence_over_the_hosts_current_span(self) -> None:
+ host = _started_span(name="host")
+ pipelex_span = _started_span(name="pipe")
+
+ with trace.use_span(host), pipelex_span_active(span=pipelex_span):
+ assert span_context_for_logs() == pipelex_span.get_span_context()
+ assert trace.get_current_span(otel_context_for_logs()) is pipelex_span
+
+ def test_opentelemetrys_current_span_is_never_touched(self) -> None:
+ host = _started_span(name="host")
+ pipelex_span = _started_span(name="pipe")
+
+ with pipelex_span_active(span=pipelex_span):
+ assert trace.get_current_span() is trace.INVALID_SPAN
+ with trace.use_span(host), pipelex_span_active(span=pipelex_span):
+ assert trace.get_current_span() is host
+
+ def test_outside_any_pipelex_span_a_line_joins_the_hosts_current_span(self) -> None:
+ host = _started_span(name="host")
+
+ with trace.use_span(host):
+ assert span_context_for_logs() == host.get_span_context()
+ assert trace.get_current_span(otel_context_for_logs()) is host
+
+ def test_outside_any_span_a_line_joins_none(self) -> None:
+ assert span_context_for_logs() is None
+ with trace.use_span(trace.INVALID_SPAN):
+ assert span_context_for_logs() is None
+ assert not trace.get_current_span(otel_context_for_logs()).get_span_context().is_valid
+
+ def test_an_exception_crossing_the_block_leaves_the_span_as_it_was(self) -> None:
+ span = _started_span(name="work")
+
+ with pytest.raises(ValueError, match="boom"):
+ _fail_under(span=span)
+
+ assert span_context_for_logs() is None
+ assert span.is_recording()
+ assert span.status.status_code is StatusCode.UNSET
+ assert list(span.events) == []
+
+ def test_a_none_span_holds_nothing(self) -> None:
+ enclosing = _started_span(name="enclosing")
+ host = _started_span(name="host")
+
+ with pipelex_span_active(span=enclosing), pipelex_span_active(span=None):
+ assert span_context_for_logs() == enclosing.get_span_context()
+ with trace.use_span(host), pipelex_span_active(span=None):
+ assert span_context_for_logs() == host.get_span_context()
+
+ def test_a_span_naming_no_trace_hides_neither_the_enclosing_pipelex_span_nor_the_hosts(self) -> None:
+ """What a no-op tracer starts, under ``OTEL_SDK_DISABLED`` for one, must not hide a valid span."""
+ enclosing = _started_span(name="enclosing")
+ host = _started_span(name="host")
+
+ with pipelex_span_active(span=enclosing), pipelex_span_active(span=trace.INVALID_SPAN):
+ assert span_context_for_logs() == enclosing.get_span_context()
+ with trace.use_span(host), pipelex_span_active(span=trace.INVALID_SPAN):
+ assert span_context_for_logs() == host.get_span_context()
+ assert trace.get_current_span(otel_context_for_logs()) is host
diff --git a/tests/unit/pipelex/tools/test_log_json_trace_context.py b/tests/unit/pipelex/tools/test_log_json_trace_context.py
new file mode 100644
index 000000000..c7915bae4
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_json_trace_context.py
@@ -0,0 +1,195 @@
+"""The ``json`` sink writes the trace context of the span a record is logged in.
+
+That span is the Pipelex span active at the log call, and outside one OpenTelemetry's current span, the
+host's own. The keys are the ones OpenTelemetry specifies for trace context in a JSON log that is not
+OTLP, ``trace_id``, ``span_id`` and ``trace_flags``, lowercase hex at their full widths, and a record
+logged outside any valid span carries none of them. A boot line held until the sink arrives keeps the
+span it was logged in. The records go the whole way, from the facade through the module-named logger to the
+sink's handler, on a fresh ``Log`` so the installed sink is the one under test and the teardown leaves
+the root logger as it found it.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import logging
+import re
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.trace import NonRecordingSpan, SpanContext, TraceFlags
+
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.telemetry.current_span import pipelex_span_active
+from pipelex.tools.log.json_log_sink import (
+ LOGGER_KEY,
+ MESSAGE_KEY,
+ SEVERITY_KEY,
+ SPAN_ID_KEY,
+ TIME_KEY,
+ TRACE_FLAGS_KEY,
+ TRACE_ID_KEY,
+ JsonLogSink,
+)
+from pipelex.tools.log.log import Log
+from pipelex.tools.log.log_config import LogConfig
+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+TRACE_KEYS: tuple[str, ...] = (TRACE_ID_KEY, SPAN_ID_KEY, TRACE_FLAGS_KEY)
+
+# A span context whose ids are small enough that only a full-width, zero-padded spelling gets them right,
+# and which is not sampled, so its flags are written as they are rather than as the sampled default.
+_UNSAMPLED_SPAN_CONTEXT = SpanContext(trace_id=0xAB, span_id=0xCD, is_remote=False, trace_flags=TraceFlags(TraceFlags.DEFAULT))
+
+
+def _package_log_config() -> LogConfig:
+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")
+ return LogConfig.model_validate(config_dict["runtime"]["log"])
+
+
+def _own_lines(buffer: io.StringIO) -> list[dict[str, Any]]:
+ """Every line this module emitted, each parsed as one JSON object, whatever else the process logged meanwhile."""
+ lines = [json.loads(line) for line in buffer.getvalue().splitlines() if line]
+ return [line for line in lines if line[LOGGER_KEY] == __name__]
+
+
+class TestJsonSinkTraceContext:
+ @pytest.fixture
+ def json_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, io.StringIO]]:
+ """A fresh ``Log`` with the json sink installed on a buffer, torn down so the root logger is left as found."""
+ # pytest's ``log_level`` option restores the root logger's level at every phase boundary, undoing the
+ # level ``configure`` sets from inside a fixture; the module's own logger is enabled explicitly and
+ # ``caplog`` restores it at teardown.
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ fresh = Log()
+ fresh.configure(log_config=_package_log_config())
+ fresh.install_sink(JsonLogSink(stream=buffer))
+ try:
+ yield fresh, buffer
+ finally:
+ fresh.reset()
+
+ def test_a_line_logged_inside_a_span_carries_its_trace_context_in_hex(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ tracer = TracerProvider().get_tracer(__name__)
+ with tracer.start_as_current_span("work") as span:
+ fresh.info("inside")
+ span_context = span.get_span_context()
+
+ (line,) = _own_lines(buffer)
+ assert line[TRACE_ID_KEY] == f"{span_context.trace_id:032x}"
+ assert line[SPAN_ID_KEY] == f"{span_context.span_id:016x}"
+ assert line[TRACE_FLAGS_KEY] == "01"
+ assert re.fullmatch(r"[0-9a-f]{32}", line[TRACE_ID_KEY])
+ assert re.fullmatch(r"[0-9a-f]{16}", line[SPAN_ID_KEY])
+
+ def test_a_line_logged_in_a_pipelex_span_carries_it_rather_than_the_hosts(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ tracer = TracerProvider().get_tracer(__name__)
+ pipelex_span = tracer.start_span("pipe")
+ with tracer.start_as_current_span("host"):
+ with pipelex_span_active(span=pipelex_span):
+ fresh.info("in the pipe")
+ fresh.info("back in the host")
+ span_context = pipelex_span.get_span_context()
+
+ in_pipe, in_host = _own_lines(buffer)
+ assert in_pipe[TRACE_ID_KEY] == f"{span_context.trace_id:032x}"
+ assert in_pipe[SPAN_ID_KEY] == f"{span_context.span_id:016x}"
+ assert in_host[TRACE_ID_KEY] != in_pipe[TRACE_ID_KEY]
+
+ def test_a_line_logged_outside_any_span_carries_none_of_the_keys(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ fresh.info("outside")
+
+ (line,) = _own_lines(buffer)
+ assert not set(TRACE_KEYS) & set(line)
+
+ def test_an_invalid_current_span_writes_nothing(self, json_log: tuple[Log, io.StringIO]) -> None:
+ """A context can hold a span that names no trace; a line under it is a line outside any span."""
+ fresh, buffer = json_log
+ with trace.use_span(trace.INVALID_SPAN):
+ fresh.info("under the invalid span")
+
+ (line,) = _own_lines(buffer)
+ assert not set(TRACE_KEYS) & set(line)
+
+ def test_the_ids_are_zero_padded_and_the_flags_written_as_they_are(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)):
+ fresh.info("unsampled")
+
+ (line,) = _own_lines(buffer)
+ assert line[TRACE_ID_KEY] == "000000000000000000000000000000ab"
+ assert line[SPAN_ID_KEY] == "00000000000000cd"
+ assert line[TRACE_FLAGS_KEY] == "00"
+
+ def test_the_line_leaving_the_span_no_longer_carries_it(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ tracer = TracerProvider().get_tracer(__name__)
+ with tracer.start_as_current_span("outer") as outer:
+ with tracer.start_as_current_span("inner") as inner:
+ fresh.info("in the inner span")
+ fresh.info("back in the outer span")
+ fresh.info("after both")
+
+ in_inner, in_outer, after = _own_lines(buffer)
+ assert in_inner[SPAN_ID_KEY] == f"{inner.get_span_context().span_id:016x}"
+ assert in_outer[SPAN_ID_KEY] == f"{outer.get_span_context().span_id:016x}"
+ assert in_inner[TRACE_ID_KEY] == in_outer[TRACE_ID_KEY]
+ assert not set(TRACE_KEYS) & set(after)
+
+ def test_the_trace_keys_follow_the_fixed_keys_and_precede_the_fields(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)):
+ fresh.info("ordered", fields={"files": 7})
+
+ (line,) = _own_lines(buffer)
+ assert list(line) == [TIME_KEY, SEVERITY_KEY, LOGGER_KEY, MESSAGE_KEY, TRACE_ID_KEY, SPAN_ID_KEY, TRACE_FLAGS_KEY, "files"]
+
+ def test_the_trace_keys_are_reserved_with_or_without_a_span(self, json_log: tuple[Log, io.StringIO]) -> None:
+ """A field named like a trace key is prefixed on every line, so its wire name never depends on a span being current."""
+ fresh, buffer = json_log
+ supplied = dict.fromkeys(TRACE_KEYS, "supplied")
+ fresh.info("no span", fields=supplied)
+ with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)):
+ fresh.info("in a span", fields=supplied)
+
+ without_span, with_span = _own_lines(buffer)
+ for key in TRACE_KEYS:
+ assert key not in without_span
+ assert without_span[f"{COLLIDING_FIELD_PREFIX}{key}"] == "supplied"
+ assert with_span[f"{COLLIDING_FIELD_PREFIX}{key}"] == "supplied"
+ assert with_span[TRACE_ID_KEY] == "000000000000000000000000000000ab"
+
+ def test_a_boot_line_held_until_the_sink_arrives_keeps_the_span_it_was_logged_in(self, caplog: pytest.LogCaptureFixture) -> None:
+ """The holding handler replays each record in the context it was emitted in, not in the one the install runs under."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ fresh = Log()
+ fresh.configure(log_config=_package_log_config())
+ tracer = TracerProvider().get_tracer(__name__)
+ pipelex_span = tracer.start_span("pipe at boot")
+ try:
+ with tracer.start_as_current_span("boot step") as boot_step:
+ fresh.info("held inside a span")
+ with pipelex_span_active(span=pipelex_span):
+ fresh.info("held inside a pipelex span")
+ fresh.info("held outside any span")
+ with tracer.start_as_current_span("install"), pipelex_span_active(span=tracer.start_span("pipe at install")):
+ fresh.install_sink(JsonLogSink(stream=buffer))
+ finally:
+ fresh.reset()
+
+ inside, inside_pipelex, outside = _own_lines(buffer)
+ assert inside[SPAN_ID_KEY] == f"{boot_step.get_span_context().span_id:016x}"
+ assert inside_pipelex[SPAN_ID_KEY] == f"{pipelex_span.get_span_context().span_id:016x}"
+ assert not set(TRACE_KEYS) & set(outside)
diff --git a/tests/unit/pipelex/tools/test_log_otlp_trace_context.py b/tests/unit/pipelex/tools/test_log_otlp_trace_context.py
new file mode 100644
index 000000000..c11217222
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_otlp_trace_context.py
@@ -0,0 +1,109 @@
+"""The ``otlp`` sink files each record under the span it was logged in.
+
+That span is the Pipelex span active at the log call, and outside one OpenTelemetry's current span, the
+host's own. The sink hands the SDK a context holding that span, from which the SDK takes the record's
+trace id, span id and flags, so a collector files the record under it; a record logged outside any span
+carries none. A boot line held until the sink arrives keeps the span it was logged in. The sink
+is built on a synchronous processor so each record is exported before the assertion reads it.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+import pytest
+from opentelemetry.sdk._logs.export import InMemoryLogExporter, SimpleLogRecordProcessor
+from opentelemetry.sdk.trace import TracerProvider
+
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.telemetry.current_span import pipelex_span_active
+from pipelex.tools.log.log import Log
+from pipelex.tools.log.log_config import LogConfig
+from pipelex.tools.log.otlp_log_sink import OtlpLogSink
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opentelemetry.sdk._logs import LogRecord
+
+
+def _package_log_config() -> LogConfig:
+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")
+ return LogConfig.model_validate(config_dict["runtime"]["log"])
+
+
+def _own_records(exporter: InMemoryLogExporter) -> list[LogRecord]:
+ """The records this module emitted, whatever else the process logged meanwhile."""
+ return [log_data.log_record for log_data in exporter.get_finished_logs() if log_data.instrumentation_scope.name == __name__]
+
+
+class TestOtlpSinkTraceContext:
+ @pytest.fixture
+ def otlp_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, InMemoryLogExporter]]:
+ """A fresh ``Log`` with the otlp sink installed on an in-memory exporter, torn down so the root logger is left as found."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ exporter = InMemoryLogExporter()
+ fresh = Log()
+ fresh.configure(log_config=_package_log_config())
+ fresh.install_sink(OtlpLogSink(processor=SimpleLogRecordProcessor(exporter)))
+ try:
+ yield fresh, exporter
+ finally:
+ fresh.reset()
+
+ def test_a_record_logged_inside_a_span_is_filed_under_it(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:
+ fresh, exporter = otlp_log
+ tracer = TracerProvider().get_tracer(__name__)
+ with tracer.start_as_current_span("work") as span:
+ fresh.info("inside")
+ fresh.info("outside")
+ span_context = span.get_span_context()
+
+ inside, outside = _own_records(exporter)
+ assert inside.trace_id == span_context.trace_id
+ assert inside.span_id == span_context.span_id
+ assert inside.trace_flags == span_context.trace_flags
+ assert outside.trace_id == 0
+ assert outside.span_id == 0
+
+ def test_a_record_logged_in_a_pipelex_span_is_filed_under_it_rather_than_the_hosts(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:
+ fresh, exporter = otlp_log
+ tracer = TracerProvider().get_tracer(__name__)
+ pipelex_span = tracer.start_span("pipe")
+ with tracer.start_as_current_span("host") as host_span:
+ with pipelex_span_active(span=pipelex_span):
+ fresh.info("in the pipe")
+ fresh.info("back in the host")
+ span_context = pipelex_span.get_span_context()
+
+ in_pipe, in_host = _own_records(exporter)
+ assert in_pipe.trace_id == span_context.trace_id
+ assert in_pipe.span_id == span_context.span_id
+ assert in_pipe.trace_flags == span_context.trace_flags
+ assert in_host.span_id == host_span.get_span_context().span_id
+
+ def test_a_boot_line_held_until_the_sink_arrives_keeps_the_span_it_was_logged_in(self, caplog: pytest.LogCaptureFixture) -> None:
+ """The holding handler replays each record in the context it was emitted in, not in the one the install runs under."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ exporter = InMemoryLogExporter()
+ fresh = Log()
+ fresh.configure(log_config=_package_log_config())
+ tracer = TracerProvider().get_tracer(__name__)
+ pipelex_span = tracer.start_span("pipe at boot")
+ try:
+ with tracer.start_as_current_span("boot step") as boot_step:
+ fresh.info("held inside a span")
+ with pipelex_span_active(span=pipelex_span):
+ fresh.info("held inside a pipelex span")
+ fresh.info("held outside any span")
+ with tracer.start_as_current_span("install"), pipelex_span_active(span=tracer.start_span("pipe at install")):
+ fresh.install_sink(OtlpLogSink(processor=SimpleLogRecordProcessor(exporter)))
+ finally:
+ fresh.reset()
+
+ inside, inside_pipelex, outside = _own_records(exporter)
+ assert inside.span_id == boot_step.get_span_context().span_id
+ assert inside_pipelex.span_id == pipelex_span.get_span_context().span_id
+ assert outside.span_id == 0
diff --git a/wip/structured-logs/review-deferrals.md b/wip/structured-logs/review-deferrals.md
index faf31dd3d..d3742ea76 100644
--- a/wip/structured-logs/review-deferrals.md
+++ b/wip/structured-logs/review-deferrals.md
@@ -312,3 +312,29 @@ The fixes of the pass above had not been reviewed, so the ladder read stalled. T
- **The empty-match guard on `extra_patterns` misses zero-width matches in context** (*unverified*, code-review, restating the pass above). L-260920-44f6ff.
- **The `console` sink's Rich traceback renders from the exception object** (*unverified*, codex:adversarial, restating every round). L-260916-8bcd3f.
- **A mapping key that is itself a secret is carried through** (*unverified*, codex:adversarial, restating round 2). L-260916-715f62.
+
+## L-260925-c9ab0d, round 1 (bar open)
+
+The json sink's trace context, with the pipe and LLM spans made current. The pass fixed a held boot line taking the install's span rather than its own, which the verifier reproduced with a record logged on a second thread inside its own span: the holding handler now copies the context a record was emitted in and replays the record under it, for every sink. It also documented that the Gateway stream alone makes the spans current, and split the test module to one class each. What follows was read as real and left.
+
+- **Under the Gateway stream alone, making the spans current gives the host the side effects without the join** (*verified*, cubic, with the verifier's experiments). The tracer exists whenever `custom_posthog.tracing.enabled or pipelex_telemetry_enabled` (`pipelex/system/telemetry/telemetry_manager.py`), and the Gateway stream is on automatically for a Gateway user outside a test run, so for an operator with no exporter of their own the spans go only to Pipelex's PostHog. Their json lines then name spans they never receive, their own instrumentation's spans are re-parented under a parent their backend never gets, and a `ParentBased` host sampler keeps those children whatever its ratio, since the run's parent is hard-coded `SAMPLED` (`pipe_abstract.py`, `llm_worker_abstract.py`). The claim that the host's spans are exported to Pipelex's PostHog was refuted: `make_ai_tracer` never registers its provider globally. The alternative is to make the spans current only when the operator has an exporter of their own (custom PostHog tracing, OTLP or Langfuse); it would also take the join away from any deployment, the hosted plane included, whose lines are meant to meet the traces on Pipelex's own stream. That is a product decision, not a defect, and this pass documented the behaviour in `docs/setup/telemetry.md` under "Pipelex's spans in your process" and in `docs/tools/logging.md` instead of choosing.
+- **A Langfuse-only or OTLP-only telemetry configuration appears to create no tracer** (*verified by reading, not by running*, the verifier, outside the diff). The OTLP and Langfuse exporters are wired only inside `make_ai_tracer` (`pipelex/system/telemetry/otel_factory.py`), which runs only under the condition above, and `PostHogTracingConfig.enabled` defaults to `false`, so the Langfuse example in `docs/setup/telemetry.md` would export nothing without the Gateway stream or PostHog tracing. Pre-existing and outside this member; the orchestrator files it.
+
+## L-260925-c9ab0d, round 2 (bar defects)
+
+The verifier confirmed three defects and the pass fixed them. A no-op tracer, which the SDK hands out under `OTEL_SDK_DISABLED`, starts spans naming no trace, and making one current hid the caller's span for the whole pipe or LLM call; `span_made_current` now leaves the context alone for such a span. The opt-in `bedrock_boto3` client ran its blocking call through `run_in_executor`, which does not carry the context, so the LLM span was not current inside the call; it now uses `asyncio.to_thread`. A cancelled pipe run never ended its span, because the catch took `Exception` only; that predates this branch, and the catch now takes `BaseException`. What follows was left.
+
+- **`HeldRecord` is a bare tuple where the standards ask for a `NamedTuple`** (*unverified*, cubic). `pipelex/tools/log/log_holding.py` unpacks it in two places, `release_to` and `close`. This is an improvement, not a defect, at this bar. It also conflicts with the redaction member's `close()` edit, so it is better done after the merge.
+- **The spans made current re-parent and force-sample host instrumentation** (codex:adversarial, reproduced with `ParentBased(TraceIdRatioBased(0.0))`). This is the round-1 entry above, re-raised with the recommendation to carry the span in a Pipelex-private context variable for log correlation and make OpenTelemetry current-ness opt-in. The round-1 entry stands: it is Louis's decision.
+
+## L-260925-c9ab0d, the design change after round 2
+
+The two entries above on re-parenting and force-sampling host instrumentation no longer describe the branch. The sprint's orchestrator ruled that Pipelex must not touch the process's global OpenTelemetry context, since doing so is a side effect on a client's infrastructure: Pipelex now holds its pipe and LLM spans in a context variable of its own (`pipelex/system/telemetry/current_span.py`), the `json` and `otlp` sinks read that span first and OpenTelemetry's current span, read only, after it, and a test asserts the current span inside a pipe run and an LLM call is whatever it was outside. Whether Pipelex's spans should ever become OpenTelemetry-current, always, when the operator's own exporter receives them, or by an explicit opt-in, is an open decision for Louis on its own ledger item, which also carries what error trackers reading the current span would gain.
+
+## L-260925-c9ab0d, round 2 after the design change (bar defects)
+
+The first pass after the design change fixed one defect: the pipe's and the LLM worker's verbose "SPAN STARTED" lines were logged before their span was held, so their trace ids named the enclosing span; they are now logged with it held. It rejected the graph-tracing catch in `_run_pipe_traced` taking `Exception` only, which the verifier refuted: the graph tracer's teardown marks a node still running as `CANCELED`, which is the right status for a cancelled pipe. The second pass fixed nothing and left what follows.
+
+- **Inside a Pipelex run, a line joins the Pipelex span even where the host has a valid span of its own** (codex:adversarial, "do not ship", and cubic; verified by reading, since it is the precedence the sprint's orchestrator specified). `span_context_for_logs` and `otel_context_for_logs` in `pipelex/system/telemetry/current_span.py` prefer the held Pipelex span. Under the Gateway stream alone, Pipelex's spans reach only Pipelex's exporters, so an operator running their own OpenTelemetry tracing with the `json` or `otlp` sink gets lines inside a run tagged with a trace their backend never receives, while their own instrumentation's spans for the same work stay in their trace. On `dev` before this branch the `otlp` sink filed those records under the host's current span. The alternatives the reviewers named: prefer a valid host span unless Pipelex's spans reach a destination the operator owns, carry the Pipelex ids as namespaced attributes beside the host's standard ones, or make the precedence configurable. This is the same product question as whether Pipelex's spans should ever be OpenTelemetry-current, seen from the log side, and it belongs with that decision rather than in this branch.
+- **The pipe span's catch was widened to `BaseException` where the Python standards prescribe `try`/`finally` for required cleanup** (*unverified*, cubic). `live_run_pipe` in `pipelex/pipe_machinery/pipe_abstract.py` observes and re-raises, which is correct in behaviour; the LLM worker in the same diff ends its span in a `finally` that reads `sys.exc_info()`. Moving the pipe to that shape is a code-shape improvement, to be done together with `_run_pipe_traced`, whose own comment explains why it is not a `finally`.
+- **`current_span.py` and the `*_current_span.py` tests keep their names from the design that made the spans current** (*unverified*, cubic). Renaming the module to `active_span.py` and the tests to match would line the names up with `pipelex_span_active`; an improvement, not a defect.