Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Log-trace-fields-current · L-260927-6b356b (#1286)

Louis Choquel5d2a58d18ad8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 68b929613..b76aa877a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,7 @@
### Changed
- **The `gcp` log sink stops the boot when Google refuses its credentials (Breaking)**: the sink refreshes its credentials once when it is built, within five seconds, and credentials Google refuses (a revoked or expired refresh token, a deleted service-account key, a Google Cloud machine with no service account) or that cannot be loaded (no Application Default Credentials, a key file that is missing or holds no valid key) raise the new `GcpLogSinkCredentialsError`, a `config`-domain `CredentialsError` naming the credentials, Google's answer and what renews them, which `pipelex doctor` reports in its log-sink row. Such a process used to boot and lose every record in silence, since over gRPC the failed refresh is retried for a minute before anything reports it; a refresh that cannot reach the token endpoint or the metadata server, gets an answer that says to retry (a `5xx`, `408` or `429`), or does not answer in time, is said on stderr and the boot goes on.
+- **A log line's standard trace fields name OpenTelemetry's current span, and the Pipelex span rides under `pipelex.*` (Breaking)**: in the `json`, `otlp` and `gcp` sinks, `trace_id`, `span_id` and `trace_flags` (the `otlp` record's own trace context, the `gcp` entry's `trace`) now always name your process's current span and are absent when there is none, inside a Pipelex run too, where they used to name the pipe's or the LLM call's span; that span is written beside them as `pipelex.trace_id` and `pipelex.span_id`, keys in the `json` line and the `gcp` payload and attributes on the `otlp` record, reserved on every line like the sinks' other keys. The `gcp` entry's `trace` is no longer derived from `pipeline_run_id` and is set only under a current span, with the new `spanId` and `traceSampled` beside it, and the `pipeline_run_id` label still selects a run's lines. If you export Pipelex's spans to your own backend, join a line inside a run to them on `pipelex.trace_id` and `pipelex.span_id`. For a sink plugin, `span_context_for_logs()` and `otel_context_for_logs()` are replaced by `current_span_context_for_logs()`, `pipelex_span_context_for_logs()` and `pipelex_trace_fields_for_logs()` in `pipelex.system.telemetry.current_span`.
### Fixed
diff --git a/docs/configuration/config-practical/logging-config.md b/docs/configuration/config-practical/logging-config.md
index 55c8f682b..55ec89ab7 100644
--- a/docs/configuration/config-practical/logging-config.md
+++ b/docs/configuration/config-practical/logging-config.md
@@ -198,10 +198,10 @@ keywords_to_hilight = []
## The `json` Sink
-No section of its own: the sink writes to `console_log_target`, one JSON object per line and no ANSI ever. Each line carries `time` (ISO 8601, UTC, milliseconds, `Z`), `severity` (the level name), `logger` (the module-named logger), `message`, `exception` when the record carries one, `trace_id`, `span_id` and `trace_flags` when the record was logged inside a span, and then every field, the run-scoped identifiers and the `data` attribute flat beside them, under their own names. The three trace keys are the ones OpenTelemetry specifies for trace context in a JSON log that is not OTLP, lowercase hex at their full widths (32, 16 and 2 digits), so a collector or an error tracker joins the line to the span it was logged in; the span is the Pipelex span active at the log call, the pipe's own or the LLM call's during a run, and outside one OpenTelemetry's current span, your own code's, which Pipelex reads and never changes (see [Logging](../../tools/logging.md#the-trace-context)); a line logged outside any span carries none of the three. Those keys are reserved on every line, so a field named like one of them is carried under a `field_` prefix whether or not the line carries an exception or a span, and a non-finite float is written as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser.
+No section of its own: the sink writes to `console_log_target`, one JSON object per line and no ANSI ever. Each line carries `time` (ISO 8601, UTC, milliseconds, `Z`), `severity` (the level name), `logger` (the module-named logger), `message`, `exception` when the record carries one, `trace_id`, `span_id` and `trace_flags` when OpenTelemetry's current span names a trace, `pipelex.trace_id` and `pipelex.span_id` when a Pipelex span is active, and then every field, the run-scoped identifiers and the `data` attribute flat beside them, under their own names. The three standard trace keys are the ones OpenTelemetry specifies for trace context in a JSON log that is not OTLP, lowercase hex at their full widths (32, 16 and 2 digits), and they name OpenTelemetry's current span, your own code's, which Pipelex reads and never changes, so a collector or an error tracker joins the line to your trace. The two `pipelex.*` keys, 32 and 16 hex digits, name the Pipelex span active at the log call, the pipe's own or the LLM call's during a run. The two sets are independent, so a line in a run under no span of yours carries only the `pipelex.*` keys (see [Logging](../../tools/logging.md#the-trace-context)). Every key the sink writes, the trace keys included, is reserved on every line, so a field named like one of them is carried under a `field_` prefix whether or not the line carries an exception or a span, and a non-finite float is written as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser.
```json
-{"time": "2026-09-16T10:12:03.417Z", "severity": "INFO", "logger": "pipelex.pipe_operators.pipe_llm", "message": "Running the pipe", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "trace_flags": "01", "request_id": "req-7f3a", "pipeline_run_id": "plr-01", "pipe_run_id": "pr-9c", "model": "gpt-5"}
+{"time": "2026-09-16T10:12:03.417Z", "severity": "INFO", "logger": "pipelex.pipe_operators.pipe_llm", "message": "Running the pipe", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "trace_flags": "01", "pipelex.trace_id": "0af7651916cd43dd8448eb211c80319c", "pipelex.span_id": "b7ad6b7169203331", "request_id": "req-7f3a", "pipeline_run_id": "plr-01", "pipe_run_id": "pr-9c", "model": "gpt-5"}
```
## The `otlp` Sink
@@ -217,8 +217,8 @@ headers = { Authorization = "Bearer ..." }
- `endpoint`: The collector's logs URL. Left unset, the exporter follows the OpenTelemetry environment conventions: `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT` with the `/v1/logs` path, then the collector default on localhost
- `headers`: Headers sent with every export, an authorization header typically. Empty, the default, leaves `OTEL_EXPORTER_OTLP_HEADERS` in charge
- The records are exported in batches on the OTLP HTTP protocol, with the same service identity as the spans the runtime already exports, so a collector files the two together
-- Each record carries the trace context of the span it was logged in, the Pipelex span active at the log call, the pipe's own or the LLM call's during a run, and outside one OpenTelemetry's current span, your own code's, so a collector files the record under that span; a record logged outside any span carries none. Pipelex never makes its own spans current in the process's OpenTelemetry context, so your own instrumentation is never re-parented under them
-- The semantic-convention keys the sink writes itself — the source location `code.file.path`, `code.function.name` and `code.line.number`, and the `exception.*` keys — are reserved on every record whether or not it carries an exception, exactly as the `json` sink reserves its own keys: a field named like one of them is carried under the same `field_` prefix, so nothing is overwritten and a field keeps one wire name whichever sink is selected
+- Each record carries the trace context of OpenTelemetry's current span, your own code's, so a collector files the record under your span; a record logged where no current span names a trace carries none. The Pipelex span active at the log call, the pipe's own or the LLM call's during a run, rides as the `pipelex.trace_id` and `pipelex.span_id` attributes, in hex. Pipelex never makes its own spans current in the process's OpenTelemetry context, so your own instrumentation is never re-parented under them
+- The keys the sink writes itself — the source location `code.file.path`, `code.function.name` and `code.line.number`, `pipelex.trace_id` and `pipelex.span_id`, and the `exception.*` keys — are reserved on every record whether or not it carries an exception or a Pipelex span, exactly as the `json` sink reserves its own keys: a field named like one of them is carried under the same `field_` prefix, so nothing is overwritten and a field keeps one wire name whichever sink is selected
- The sink never exports its own export path: a record the SDK or the transport emits while an export is in flight is rejected before the handler's lock is taken, so an unreachable collector costs a warning on the export thread and never a loop or a hang at exit. A flush at teardown is bounded by the exporter's own timeout, `OTEL_EXPORTER_OTLP_TIMEOUT`, and whatever the sink raises while it flushes or closes is said on stderr rather than left to interrupt the teardown
## The `gcp` Sink
@@ -255,9 +255,9 @@ The sink refreshes its credentials once when it is built, before the transport s
Each record becomes one Cloud Logging entry with a JSON payload:
- The level maps onto the Cloud Logging severity scale. That scale has nothing below `DEBUG`, so Pipelex's two custom levels, `VERBOSE` and `DEV`, both land there
-- The payload carries `message`, `logger` and `exception` when the record carries one, then every field and the `data` attribute flat beside them. The keys the `json` sink reserves are reserved here too, `time`, `severity`, `trace_id`, `span_id` and `trace_flags` included although the payload carries none of them — the client library takes the time and the severity out of band, and the entry's own `trace` field carries the trace — so a field named like one of them is carried under a `field_` prefix under either sink rather than under one and not the other; a value JSON cannot carry — a non-finite float, a model, a circular structure — is written as text rather than costing the line
+- The payload carries `message`, `logger` and `exception` when the record carries one, `pipelex.trace_id` and `pipelex.span_id` when a Pipelex span is active, the pipe's own or the LLM call's, written exactly as the `json` sink writes them, then every field and the `data` attribute flat beside them. The keys the `json` sink reserves are reserved here too, `time`, `severity`, `trace_id`, `span_id` and `trace_flags` included although the payload carries none of them — the client library takes the time and the severity out of band, and the entry's own `trace`, `spanId` and `traceSampled` carry the current span — so a field named like one of them is carried under a `field_` prefix under either sink rather than under one and not the other; a value JSON cannot carry — a non-finite float, a model, a circular structure — is written as text rather than costing the line
- The run-scoped identifiers become the entry's **labels** rather than payload keys: `request_id`, `pipeline_run_id` and `pipe_run_id`, whichever of them the record carries. Cloud Logging indexes labels, so these are what a query filters a run by
-- The entry's `trace` field carries the run's own OpenTelemetry trace id, project-qualified as `projects/<project>/traces/<trace-id>`. The id is derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of the run it belongs to agree on it and Cloud Logging files them together
+- The entry's `trace`, `spanId` and `traceSampled` name OpenTelemetry's current span, your own code's, the trace project-qualified as `projects/<project>/traces/<trace-id>`, so Cloud Logging files the line under your trace. A line logged where no current span names a trace carries none of the three, inside a Pipelex run too: the `pipeline_run_id` label is what selects a run's lines, and the `pipelex.*` payload keys name its spans
- The entries leave through the client library's background-thread transport, which batches them off the thread that logged, so no record costs an API round trip on the calling thread. The teardown flushes and closes it, and the flush carries a deadline of its own, five seconds, because the library's does not. A flush that runs out of time says so on stderr, that the transport was still writing and that what it still holds is lost unless it sends it while it closes, followed by what a fresh refresh of the credentials answers, a refresh given five seconds of its own, so such a flush returns within ten and the close then gives the transport the client library's own grace period: refused, with Google's answer, which is why the write is held; unreachable; or still accepted, in which case the Cloud Logging API or the network is what holds it. A handler that has closed does not flush again when the process exits
- What the export path logs never leaves through the sink: the library reports a refused batch through a logger of its own, and a report exported through the pipeline it reports on would fail with it and be reported again. The handler rejects those records, and every record emitted on the library's export thread, before its lock is taken — the same guard the `otlp` sink carries, against the same deadlock at exit. What the sink's own credentials check logs while a teardown's flush waits on it is rejected the same way and printed on stderr, so the check still answers when the process's exit holds the handler's lock
- A refused write is still said: the library marks a refused batch done, so the flush succeeds and nothing else would report it, and the entries in it are lost. The export path's warnings and errors are printed on stderr instead, the first one in each window of time with its traceback and the ones after it counted in a single line once that window has passed — with the next failure, on the next line the process logs, or at the teardown at the latest — so a process whose writes Cloud Logging refuses says so without printing once per line it logs
diff --git a/docs/setup/telemetry.md b/docs/setup/telemetry.md
index 7c247cfee..28877b418 100644
--- a/docs/setup/telemetry.md
+++ b/docs/setup/telemetry.md
@@ -55,7 +55,8 @@ Either stream makes Pipelex trace a run: the Gateway stream whenever it is on, a
- Your own instrumentation, an HTTP client's or a provider SDK's, keeps opening its spans under your own current span, in your own trace, while a Pipelex run goes. It is never re-parented under a pipe or an LLM call, nor kept by a sampler that follows its parent because a Pipelex span was sampled.
- An error tracker that reads OpenTelemetry's current span sees yours, not Pipelex's.
-- The `json` and `otlp` log sinks still join each line to the Pipelex span it was logged in, the pipe's or the LLM call's, which Pipelex keeps in a context variable of its own, and outside a Pipelex run to your current span, as [Logging](../tools/logging.md#the-trace-context) describes.
+- The standard trace fields of a log line, `trace_id` and `span_id` in the `json` sink, the record's trace context in the `otlp` sink and the entry's `trace` in the `gcp` sink, name your current span, inside a Pipelex run as outside one, so your log backend files a line under your own trace. The Pipelex span a line was logged in, the pipe's or the LLM call's, which Pipelex keeps in a context variable of its own, rides beside them as `pipelex.trace_id` and `pipelex.span_id`, as [Logging](../tools/logging.md#the-trace-context) describes.
+- If you export Pipelex's spans to your own backend, through your PostHog or an OTLP destination, they arrive in a trace of their own, not yours, so a line inside a run joins them on `pipelex.trace_id` and `pipelex.span_id` rather than on the standard fields, which a backend's trace view follows by itself.
## Quick Setup
diff --git a/docs/tools/logging.md b/docs/tools/logging.md
index 66e8bda81..281a53ba1 100644
--- a/docs/tools/logging.md
+++ b/docs/tools/logging.md
@@ -116,20 +116,30 @@ The last two are the runner's and the orchestration plugin's to bind, beside the
### The trace context
-Beside the identifiers, a record is joined to the span it was logged in, and the structured sinks write that on their own, with nothing for a call site to pass: the `json` sink as `trace_id`, `span_id` and `trace_flags`, lowercase hex, the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, and the `otlp` sink as the record's own trace context, so a collector files the record under the span. The span is found in this order:
+Beside the identifiers, a record names the spans it was logged in, and the structured sinks write them on their own, with nothing for a call site to pass. There are two, read independently of each other:
-1. **The Pipelex span active at the log call**, the pipe's or the LLM call's, while Pipelex runs one.
-2. **Otherwise, OpenTelemetry's current span**, read and never changed, so a line your own code logs under its own span, outside a Pipelex run, joins your trace.
-3. **Otherwise, none**: the `json` line carries none of the three keys, and the `otlp` record no trace context. The `console` sink writes none in any case.
+- **The standard trace fields name OpenTelemetry's current span** in your process, which Pipelex reads and never changes, so your log backend files a line under the same trace as your own spans. The `json` sink writes them as `trace_id`, `span_id` and `trace_flags`, lowercase hex, the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP; the `otlp` sink as the record's own trace context, so a collector files the record under the span; and the `gcp` sink as the entry's `trace`, `spanId` and `traceSampled`. A line logged where no current span names a trace carries none of them.
+- **The Pipelex fields name the Pipelex span active at the log call**, the pipe's or the LLM call's, while Pipelex runs one: `pipelex.trace_id` and `pipelex.span_id`, lowercase hex, as keys in the `json` line and the `gcp` payload and as attributes on the `otlp` record. They are the fields that say which pipe or which LLM call a line belongs to. A line logged outside a Pipelex span carries neither.
+
+So a line carries both, either or neither:
+
+| Where the line is logged | Standard fields | `pipelex.*` fields |
+| --- | --- | --- |
+| In a Pipelex run, under a span of your own | Your span | The pipe's or the LLM call's span |
+| In a Pipelex run, under no span of yours | Absent | The pipe's or the LLM call's span |
+| Outside a Pipelex run, under a span of your own | Your span | Absent |
+| Outside both | Absent | Absent |
+
+The two sets name different traces inside a run: Pipelex's spans belong to a trace of their own, which only Pipelex's exporters receive. If you export Pipelex's spans to your own backend, join a line to them on `pipelex.trace_id` and `pipelex.span_id`. Every line of a run also carries its `pipeline_run_id`, which selects the run's lines whatever the spans. The `console` sink writes no trace context in any case.
The runtime holds each span it starts as the active Pipelex span from its start to its end, however it ends:
-- **A pipe run's span**: `live_run_pipe` starts the span, then runs the pipe with it held, so a line logged by the pipe's own work, a controller's or an operator's and the lines of the extract, image-generation, search or function call an operator makes, names the pipe's span, and a nested pipe's lines name the nested pipe's span until it returns. The line announcing the run is logged before the span starts, under the enclosing span.
-- **An LLM call's span**: the LLM worker runs the provider call with its generation span held, so the lines of the call and of its failure name that span, a provider SDK's own lines included.
+- **A pipe run's span**: `live_run_pipe` starts the span, then runs the pipe with it held, so a line logged by the pipe's own work, a controller's or an operator's and the lines of the extract, image-generation, search or function call an operator makes, names the pipe's span under `pipelex.*`, and a nested pipe's lines name the nested pipe's span until it returns. The line announcing the run is logged before the span starts, under the enclosing span.
+- **An LLM call's span**: the LLM worker runs the provider call with its generation span held, so the lines of the call and of its failure name that span under `pipelex.*`, a provider SDK's own lines included.
-The runtime starts these spans only when it traces a live run, which it does once telemetry has created its tracer: with the Pipelex Gateway's telemetry on, which is automatic when the Gateway serves your inference, or with AI span tracing enabled on your own PostHog, as [Telemetry](../setup/telemetry.md#pipelexs-spans-in-your-process) describes. Otherwise it holds no span, as it does when its tracer is a no-op one that starts spans naming no trace, under `OTEL_SDK_DISABLED` for instance, and a line names whatever span your own code runs under, if any. A boot line held until the sink arrives is replayed in the context it was logged in, so it keeps its span too, and a thread started with `asyncio.to_thread` or `contextvars.copy_context` inherits the span of the code that started it.
+The runtime starts these spans only when it traces a live run, which it does once telemetry has created its tracer: with the Pipelex Gateway's telemetry on, which is automatic when the Gateway serves your inference, or with AI span tracing enabled on your own PostHog, as [Telemetry](../setup/telemetry.md#pipelexs-spans-in-your-process) describes. Otherwise it holds no span, as it does when its tracer is a no-op one that starts spans naming no trace, under `OTEL_SDK_DISABLED` for instance, and a line carries no `pipelex.*` fields. A boot line held until the sink arrives is replayed in the context it was logged in, so it keeps both its spans, and a thread started with `asyncio.to_thread` or `contextvars.copy_context` inherits both from the code that started it.
-Holding a span is in-process plumbing, like binding the log context: it lives in a context variable of Pipelex's own. **Pipelex never makes its spans current in your process's OpenTelemetry context**, so your own instrumentation, an HTTP client's or a provider SDK's, is never re-parented under a Pipelex span, and an error tracker that reads OpenTelemetry's current span sees yours, not Pipelex's. A span's children take their parent from the job metadata, so a step running in another process gets the parent it always did, and a line that process logs names a span only while the process runs a span of its own, the LLM call's for instance.
+Holding a span is in-process plumbing, like binding the log context: it lives in a context variable of Pipelex's own. **Pipelex never makes its spans current in your process's OpenTelemetry context**, so your own instrumentation, an HTTP client's or a provider SDK's, is never re-parented under a Pipelex span, and an error tracker that reads OpenTelemetry's current span sees yours, not Pipelex's. A span's children take their parent from the job metadata, so a step running in another process gets the parent it always did, and a line that process logs carries `pipelex.*` fields only while the process runs a span of its own, the LLM call's for instance.
## Structured content
diff --git a/docs/under-the-hood/log-sink-plugins.md b/docs/under-the-hood/log-sink-plugins.md
index 60fe6d881..9def7a71c 100644
--- a/docs/under-the-hood/log-sink-plugins.md
+++ b/docs/under-the-hood/log-sink-plugins.md
@@ -93,13 +93,13 @@ class LogSink(ABC):
### What a sink reads off a record
-Every sink reads the same record, shaped by the [logging facade](../tools/logging.md): the message is `record.getMessage()`, and everything the call, the bound context or a record factory attached rides as attributes. `carried_attributes(record=...)` in `pipelex/tools/log/log_fields.py` hands a sink exactly that set, in the order the attributes were attached, by subtracting the attributes the stdlib gives every record and the ones this package stamps on a record itself. An exception a call carried with `include_exception=True` is the record's `exc_info`, the stdlib's own channel, which each sink renders its own way. The trace context is not on the record: it is read from the context when the sink's handler runs, which is inside the log call, or, for a boot line held until the sink arrives, from the context the holding handler copied when the line was logged and replays it under. The runtime holds the pipe's span or the LLM call's in a context variable of its own while it runs, and never makes it OpenTelemetry's current span, so a sink that joins its records to their traces reads it at emit through `pipelex/system/telemetry/current_span.py`: `span_context_for_logs()` returns the active Pipelex span's context, else the valid current OpenTelemetry span's, else `None`, and `otel_context_for_logs()` returns the current context with the active Pipelex span set in it, for an OpenTelemetry API that takes a context. The `json` sink reads the first, the `otlp` sink the second.
+Every sink reads the same record, shaped by the [logging facade](../tools/logging.md): the message is `record.getMessage()`, and everything the call, the bound context or a record factory attached rides as attributes. `carried_attributes(record=...)` in `pipelex/tools/log/log_fields.py` hands a sink exactly that set, in the order the attributes were attached, by subtracting the attributes the stdlib gives every record and the ones this package stamps on a record itself. An exception a call carried with `include_exception=True` is the record's `exc_info`, the stdlib's own channel, which each sink renders its own way. The trace context is not on the record: it is read from the context when the sink's handler runs, which is inside the log call, or, for a boot line held until the sink arrives, from the context the holding handler copied when the line was logged and replays it under. A record names two spans, read independently at emit through `pipelex/system/telemetry/current_span.py`. Its standard trace fields name OpenTelemetry's current span, which a sink only reads: `current_span_context_for_logs()` returns its context when it names a trace, else `None`, and a sink handing a context to an OpenTelemetry API passes `get_current()` as it is. The runtime holds the pipe's span or the LLM call's in a context variable of its own while it runs, and never makes it OpenTelemetry's current span, so it rides beside them: `pipelex_span_context_for_logs()` returns the held Pipelex span's context, else `None`, and `pipelex_trace_fields_for_logs()` returns it already spelled, a dict of `pipelex.trace_id` and `pipelex.span_id` in lowercase hex at their full widths, or an empty dict outside a Pipelex span, under the key names `PIPELEX_TRACE_ID_KEY` and `PIPELEX_SPAN_ID_KEY`. The `json` sink writes the current span under the OpenTelemetry JSON keys and the Pipelex fields after them, the `otlp` sink files the record under `get_current()` and writes the Pipelex fields as attributes, and the `gcp` sink passes the current span as the entry's `trace`, `span_id` and `trace_sampled` and writes the Pipelex fields into the payload.
Three rules follow for a sink author, and the shipped sinks honour all of them:
- **A sink serializes on emit, synchronously, on the calling thread.** A field's value rides the record by reference, so a value the caller mutates after the call is not what the record said; reading it at emit is what makes the record a faithful account of the call. A sink that queues, the `otlp` sink on its batching exporter for instance, translates the record into its wire form at emit and queues the translation. The `data` attribute needs no such care: it is already a JSON-ready snapshot of the call.
- **A sink renders every record it is handed and never raises.** A value the wire cannot carry is written as text, a JSON rendering for a mapping, a `repr` for what JSON refuses; a serialization failure is a fact about the value, never a reason to lose the line. A handler failure goes through the stdlib's `handleError`, as any handler's does.
-- **A sink reserves the keys it writes itself, on every record.** A carried attribute named like one of them is written under the `COLLIDING_FIELD_PREFIX` — `field_` — applied until the name lands free, which is the same treatment the record gives a field named after a stdlib attribute. Reserved unconditionally, not only when the sink happens to write the key on this record: a field's wire name must not depend on whether an exception was attached, and a field must not survive one sink and vanish on another. The `json` sink reserves `time`, `severity`, `logger`, `message`, `exception`, `trace_id`, `span_id` and `trace_flags`; the `gcp` sink reserves that same set even though only `message`, `logger` and `exception` are payload keys there, the client library carrying the time and the severity out of band and the entry's own `trace` field carrying the trace, so a field named like one of them is prefixed under either sink rather than under one only; the `otlp` sink reserves its `code.*` and `exception.*` semantic-convention keys.
+- **A sink reserves the keys it writes itself, on every record.** A carried attribute named like one of them is written under the `COLLIDING_FIELD_PREFIX` — `field_` — applied until the name lands free, which is the same treatment the record gives a field named after a stdlib attribute. Reserved unconditionally, not only when the sink happens to write the key on this record: a field's wire name must not depend on whether an exception was attached, and a field must not survive one sink and vanish on another. The `json` sink reserves `time`, `severity`, `logger`, `message`, `exception`, `trace_id`, `span_id`, `trace_flags`, `pipelex.trace_id` and `pipelex.span_id`; the `gcp` sink reserves that same set even though only `message`, `logger`, `exception` and the two `pipelex.*` keys are payload keys there, the client library carrying the time and the severity out of band and the entry's own `trace`, `spanId` and `traceSampled` carrying the current span, so a field named like one of them is prefixed under either sink rather than under one only; the `otlp` sink reserves its `code.*` and `exception.*` semantic-convention keys and the two `pipelex.*` ones.
---
@@ -121,10 +121,10 @@ class LogSinkPlugin:
| Sink | What it does | Where it reads its settings |
|------|--------------|-----------------------------|
-| `json` | One JSON object per line on the configured stream: `time`, `severity`, `logger`, `message`, `exception` when there is one, `trace_id`, `span_id` and `trace_flags` in hex when the record was logged inside a span, the active Pipelex span or else the current OpenTelemetry one, under the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, then the fields, the context identifiers and `data` flat beside them, a field named like one of those keys under a `field_` prefix on every line, a non-finite float as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser, and no ANSI ever. What the hosted plane's runner and worker select. | `console_log_target` |
+| `json` | One JSON object per line on the configured stream: `time`, `severity`, `logger`, `message`, `exception` when there is one, `trace_id`, `span_id` and `trace_flags` in hex when OpenTelemetry's current span names a trace, under the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, `pipelex.trace_id` and `pipelex.span_id` in hex when a Pipelex span is active, then the fields, the context identifiers and `data` flat beside them, a field named like one of those keys under a `field_` prefix on every line, a non-finite float as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser, and no ANSI ever. What the hosted plane's runner and worker select. | `console_log_target` |
| `console` | The Rich handler with the emoji formatter and every `[runtime.log.rich_log]` setting, byte for byte what the console showed before sinks existed. Rich, the `cli` extra, is imported when the handler is built, and a process that selects another sink never loads it through this path. | `console_log_target`, `[runtime.log.rich_log]` |
-| `otlp` | The OpenTelemetry logs signal: a `LoggerProvider` carrying the same service identity as the tracer, a `BatchLogRecordProcessor` and the OTLP HTTP log exporter. The message is the body, the level maps onto the OTel severity scale, the fields, identifiers and `data` ride as attributes (a mapping as JSON text), an exception lands under the `exception.*` semantic-convention keys, each record is filed under the span it was logged in, the active Pipelex span or else the current OpenTelemetry one, and a collector receives the logs beside the spans the runtime already exports. The keys the sink writes itself, the `code.*` source location and the `exception.*` set, are reserved on every record whether or not it carries an exception, so a field named like one of them rides under a `field_` prefix rather than being overwritten. A filter on the handler rejects the records of the sink's own export path, the SDK's by logger name and the transport's by the context value the SDK sets around an export, before the handler's lock is taken, so an export failure never re-enters the pipeline and a shutdown never waits on itself. | `[runtime.log.otlp]` |
-| `gcp` | Google Cloud Logging through the `google-cloud-logging` client library, behind the `gcp-logging` extra. Each record becomes one struct entry: the level maps onto the Cloud Logging severity scale (which has nothing below `DEBUG`, so `VERBOSE` and `DEV` both land there), the message, the logger, the exception and the fields become the JSON payload, the run-scoped identifiers become the entry's labels, which is what Cloud Logging indexes, and the entry's `trace` field carries the run's own OpenTelemetry trace id project-qualified, derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of its run agree on it. The entries leave through the client library's background-thread transport, which batches them off the calling thread. The factory refreshes the credentials once before that transport starts, so credentials Google refuses stop the boot with `GcpLogSinkCredentialsError` rather than losing every record in silence. **Most processes on Google Cloud want `json` instead**: a platform whose logging agent reads the container's stdout, Cloud Run and GKE among them, ingests what that sink writes and needs no client at all; `gcp` is for a process with no such agent in front of it, or one writing to a log or a project that is not the ambient one. | `[runtime.log.gcp]` |
+| `otlp` | The OpenTelemetry logs signal: a `LoggerProvider` carrying the same service identity as the tracer, a `BatchLogRecordProcessor` and the OTLP HTTP log exporter. The message is the body, the level maps onto the OTel severity scale, the fields, identifiers and `data` ride as attributes (a mapping as JSON text), an exception lands under the `exception.*` semantic-convention keys, each record is filed under OpenTelemetry's current span, with the active Pipelex span as the `pipelex.trace_id` and `pipelex.span_id` attributes, and a collector receives the logs beside the spans the runtime already exports. The keys the sink writes itself, the `code.*` source location, the two `pipelex.*` keys and the `exception.*` set, are reserved on every record whether or not it carries an exception, so a field named like one of them rides under a `field_` prefix rather than being overwritten. A filter on the handler rejects the records of the sink's own export path, the SDK's by logger name and the transport's by the context value the SDK sets around an export, before the handler's lock is taken, so an export failure never re-enters the pipeline and a shutdown never waits on itself. | `[runtime.log.otlp]` |
+| `gcp` | Google Cloud Logging through the `google-cloud-logging` client library, behind the `gcp-logging` extra. Each record becomes one struct entry: the level maps onto the Cloud Logging severity scale (which has nothing below `DEBUG`, so `VERBOSE` and `DEV` both land there), the message, the logger, the exception and the fields become the JSON payload, the run-scoped identifiers become the entry's labels, which is what Cloud Logging indexes, the entry's `trace`, `spanId` and `traceSampled` name OpenTelemetry's current span, the trace project-qualified, and the active Pipelex span rides in the payload as `pipelex.trace_id` and `pipelex.span_id`. The entries leave through the client library's background-thread transport, which batches them off the calling thread. The factory refreshes the credentials once before that transport starts, so credentials Google refuses stop the boot with `GcpLogSinkCredentialsError` rather than losing every record in silence. **Most processes on Google Cloud want `json` instead**: a platform whose logging agent reads the container's stdout, Cloud Run and GKE among them, ingests what that sink writes and needs no client at all; `gcp` is for a process with no such agent in front of it, or one writing to a log or a project that is not the ambient one. | `[runtime.log.gcp]` |
The `otlp` factory imports the OpenTelemetry logs SDK when it runs, and the `gcp` factory the Google Cloud Logging client library, never at register, so registering the built-ins imports neither, which the import-light guard pins.
diff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.py
index b4bc39424..c5d1daecc 100644
--- a/pipelex/cogt/llm/llm_worker_abstract.py
+++ b/pipelex/cogt/llm/llm_worker_abstract.py
@@ -228,7 +228,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
attributes=span_attributes,
)
- # Debug logging, under the span it announces, so the line's trace context names that span
+ # Debug logging, under the span it announces, so the line's `pipelex.*` fields name that span
span_ctx = span.get_span_context()
with pipelex_span_active(span=span):
log.verbose(
@@ -465,8 +465,8 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.TEXT)
# The span is the Pipelex span active here until it ends, whichever way it ends, so a log line
- # during the call, a provider SDK's included, is joined to it; OpenTelemetry's current context
- # is left alone.
+ # during the call, a provider SDK's included, names it under `pipelex.*`; OpenTelemetry's current
+ # context is left alone, and it is what the line's standard trace fields name.
with pipelex_span_active(span=span):
try:
text_result = await self._gen_text(llm_job=llm_job)
@@ -508,8 +508,8 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
span = self._start_otel_span_llm(llm_job=llm_job, output_type=InferenceOutputType.OBJECT, output_class_name=schema.__name__)
# The span is the Pipelex span active here until it ends, whichever way it ends, so a log line
- # during the call, a provider SDK's included, is joined to it; OpenTelemetry's current context
- # is left alone.
+ # during the call, a provider SDK's included, names it under `pipelex.*`; OpenTelemetry's current
+ # context is left alone, and it is what the line's standard trace fields name.
with pipelex_span_active(span=span):
try:
object_result = await self._gen_object(llm_job=llm_job, schema=schema)
diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.py
index 675f4e524..ae465d45c 100644
--- a/pipelex/pipe_machinery/pipe_abstract.py
+++ b/pipelex/pipe_machinery/pipe_abstract.py
@@ -980,9 +980,10 @@ class PipeAbstract(ABC, BaseModel):
# Run pipe ------------------------------------------------------------
# The span is the Pipelex span active here until it ends, whichever way it ends, so a log
- # line inside the run is joined to it. OpenTelemetry's current context is left alone, so a
- # host's own instrumentation is never re-parented, and the span's children still take their
- # parent from `child_metadata`.
+ # line inside the run names it under `pipelex.*`. OpenTelemetry's current context is left
+ # alone, so a host's own instrumentation is never re-parented and a line's standard trace
+ # fields keep naming the host's span, and the span's children still take their parent from
+ # `child_metadata`.
with pipelex_span_active(span=span):
try:
pipe_output = await self._live_run_pipe(
@@ -1168,7 +1169,7 @@ class PipeAbstract(ABC, BaseModel):
attributes=span_attributes,
)
- # Debug logging, under the span it announces, so the line's trace context names that span
+ # Debug logging, under the span it announces, so the line's `pipelex.*` fields name that span
span_ctx = span.get_span_context()
with pipelex_span_active(span=span):
log.verbose(
diff --git a/pipelex/providers/bedrock/bedrock_client_boto3.py b/pipelex/providers/bedrock/bedrock_client_boto3.py
index 925254d02..69d2f817b 100644
--- a/pipelex/providers/bedrock/bedrock_client_boto3.py
+++ b/pipelex/providers/bedrock/bedrock_client_boto3.py
@@ -37,8 +37,8 @@ class BedrockClientBoto3(BedrockClientProtocol):
params["system"] = [{"text": system_text}]
# ``to_thread`` rather than ``run_in_executor``: it carries the context over, so the SDK's own log
- # lines in the thread are joined to the LLM span, and a span its instrumentation opens has the
- # caller's current span as its parent.
+ # lines in the thread name the LLM span under `pipelex.*` and the caller's current span in their
+ # standard trace fields, and a span its instrumentation opens has that current span as its parent.
resp_dict: dict[str, Any] = await asyncio.to_thread(self.boto3_client.converse, **params) # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType]
usage_dict: dict[str, Any] = resp_dict["usage"]
diff --git a/pipelex/system/telemetry/current_span.py b/pipelex/system/telemetry/current_span.py
index 743057edf..4b38a81fe 100644
--- a/pipelex/system/telemetry/current_span.py
+++ b/pipelex/system/telemetry/current_span.py
@@ -1,4 +1,10 @@
-"""The Pipelex span active in this task, which a log line is joined to.
+"""The two spans a log line names: OpenTelemetry's current span, and the Pipelex span active in this task.
+
+A line's standard trace fields name OpenTelemetry's current span in the process, which Pipelex only
+reads and never sets, so an operator's log backend files a line under the trace of the operator's own
+spans. The Pipelex span rides beside them, under the ``pipelex.trace_id`` and ``pipelex.span_id`` keys,
+so a line inside a run still names the pipe or the LLM call it was logged in. The two are read
+independently: a line can carry both, either, or neither.
The runtime starts its spans with an explicit parent carried in the job metadata, because a run's trace
is derived from its business identifiers and has to survive a process boundary, where an in-process
@@ -9,7 +15,7 @@ only Pipelex's exporters receive, and have a sampler that follows its parent kee
runtime holds the span in a context variable of its own while the work it measures runs, in this task
and in whatever copies the task's context, a thread started by ``asyncio.to_thread`` or a boot line the
holding handler replays, and restores the previous value on the way out. The log sinks read it to write
-a line's trace context, and nothing else does.
+a line's ``pipelex.*`` fields, and nothing else does.
Holding a span never ends it, records an exception on it or sets its status, which stay with the code
that started it, and it never becomes the way a parent reaches a child: the job metadata still carries
@@ -23,14 +29,17 @@ from contextvars import ContextVar
from typing import TYPE_CHECKING
from opentelemetry import trace
-from opentelemetry.context import get_current
if TYPE_CHECKING:
from collections.abc import Generator
- from opentelemetry.context import Context
from opentelemetry.trace import Span, SpanContext
+# The keys a log line names the held Pipelex span under, in every sink that writes trace context: the
+# OpenTelemetry attribute-namespace spelling, which the ``otlp`` sink needs as an attribute name anyway.
+PIPELEX_TRACE_ID_KEY = "pipelex.trace_id"
+PIPELEX_SPAN_ID_KEY = "pipelex.span_id"
+
# The Pipelex span active here; only ever a span naming a trace, since ``pipelex_span_active`` sets no other.
_ACTIVE_PIPELEX_SPAN: ContextVar[Span | None] = ContextVar("pipelex_active_span", default=None)
@@ -41,8 +50,7 @@ def pipelex_span_active(*, span: Span | None) -> Generator[None]:
``None`` is what the runtime holds with no tracer or in a dry run. A span naming no trace is what a
no-op tracer starts, under ``OTEL_SDK_DISABLED`` for one, and holding it would hide the enclosing
- Pipelex span, or the host's current one, from every line of the block. OpenTelemetry's own context is
- left as it is either way.
+ Pipelex span from every line of the block. OpenTelemetry's own context is left as it is either way.
"""
if span is None or not span.get_span_context().is_valid:
yield
@@ -54,27 +62,40 @@ def pipelex_span_active(*, span: Span | None) -> Generator[None]:
_ACTIVE_PIPELEX_SPAN.reset(token)
-def span_context_for_logs() -> SpanContext | None:
- """The span a line logged here is joined to: the active Pipelex span, else OpenTelemetry's current span, else none.
+def current_span_context_for_logs() -> SpanContext | None:
+ """OpenTelemetry's current span in the process, which a line's standard trace fields name, or ``None`` when it names no trace.
- The current span is only read, so a line logged under the host's own span, outside any Pipelex run,
- still joins the host's trace.
+ The span is only read. A Pipelex span held here never stands in for it: inside a run with no current
+ span of the host's, the standard fields are absent and the Pipelex span rides only under ``pipelex.*``.
"""
- pipelex_span = _ACTIVE_PIPELEX_SPAN.get()
- if pipelex_span is not None:
- return pipelex_span.get_span_context()
current_span_context = trace.get_current_span().get_span_context()
if current_span_context.is_valid:
return current_span_context
return None
-def otel_context_for_logs() -> Context:
- """The context a log record is emitted in: the current one with the active Pipelex span in it, or the current one as is.
+def pipelex_span_context_for_logs() -> SpanContext | None:
+ """The Pipelex span held here by ``pipelex_span_active``, the innermost one, or ``None`` outside any.
- Built for the one record and never attached, so OpenTelemetry's current context stays what it was.
+ OpenTelemetry's current span never stands in for it: outside a Pipelex span, a line carries no
+ ``pipelex.*`` fields whatever the host's span is.
"""
pipelex_span = _ACTIVE_PIPELEX_SPAN.get()
if pipelex_span is None:
- return get_current()
- return trace.set_span_in_context(pipelex_span)
+ return None
+ return pipelex_span.get_span_context()
+
+
+def pipelex_trace_fields_for_logs() -> dict[str, str]:
+ """The ``pipelex.trace_id`` and ``pipelex.span_id`` of the Pipelex span held here, in lowercase hex at their full widths, or nothing.
+
+ Written whenever a Pipelex span is held, even when it is also OpenTelemetry's current span, so a
+ reader joining on these keys finds them on every line of a traced run whatever the process's setup.
+ """
+ span_context = pipelex_span_context_for_logs()
+ if span_context is None:
+ return {}
+ return {
+ PIPELEX_TRACE_ID_KEY: trace.format_trace_id(span_context.trace_id),
+ PIPELEX_SPAN_ID_KEY: trace.format_span_id(span_context.span_id),
+ }
diff --git a/pipelex/tools/log/gcp_log_sink.py b/pipelex/tools/log/gcp_log_sink.py
index 238b457c8..6c0239e23 100644
--- a/pipelex/tools/log/gcp_log_sink.py
+++ b/pipelex/tools/log/gcp_log_sink.py
@@ -2,8 +2,11 @@
Each record becomes one struct entry: the level maps onto the Cloud Logging severity scale, the
message and the call's fields become the JSON payload, the run-scoped identifiers become labels, and
-the run's OpenTelemetry trace id becomes the entry's ``trace`` field, so Cloud Logging files the line
-under the same trace as the spans the runtime exports. The entries leave through the client library's
+OpenTelemetry's current span, the host's own, which is only read, becomes the entry's ``trace``,
+``spanId`` and ``traceSampled``, so Cloud Logging files the line under the trace of the host's spans; a
+line logged under no current span has none of the three. The Pipelex span held when the record was
+logged, a pipe's or an LLM call's, rides in the payload as ``pipelex.trace_id`` and ``pipelex.span_id``,
+in hex, exactly as the ``json`` sink writes it. The entries leave through the client library's
transport, a batching background thread in production, so no record costs an API round trip on the
thread that logged it. What that thread itself logs never leaves through the sink: the library reports
a refused batch through a logger of its own, and a report exported through the pipeline it reports on
@@ -39,15 +42,16 @@ from enum import StrEnum
from http import HTTPStatus
from typing import TYPE_CHECKING, Any, NamedTuple, Protocol, cast
+from opentelemetry import trace
from typing_extensions import override
from pipelex.system.exceptions import MissingDependencyError
+from pipelex.system.telemetry.current_span import current_span_context_for_logs, pipelex_trace_fields_for_logs
from pipelex.tools.log.exceptions import GcpLogSinkCredentialsError
from pipelex.tools.log.json_log_sink import EXCEPTION_KEY, FIXED_KEYS, LOGGER_KEY, MESSAGE_KEY
from pipelex.tools.log.log_context import PIPE_RUN_ID_FIELD, PIPELINE_RUN_ID_FIELD, REQUEST_ID_FIELD
from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes
from pipelex.tools.log.log_sink import LogSink, LogSinkMethod, render_json
-from pipelex.tools.misc.hash_utils import hash_md5_to_int
if TYPE_CHECKING:
from collections.abc import Callable
@@ -58,12 +62,14 @@ if TYPE_CHECKING:
GCP_LOGGING_DEPENDENCY_NAME = "google-cloud-logging"
GCP_LOGGING_EXTRA_NAME = "gcp-logging"
-# The sink writes three of the ``json`` sink's keys into the payload — its ``message``, ``logger`` and
-# ``exception``, imported rather than respelled — and reserves that sink's whole set against a carried
-# attribute, so one field keeps one wire name whichever of the two a process selects. ``time`` and
-# ``severity`` are not payload keys here, the client library carrying both out of band, and neither are
-# ``trace_id``, ``span_id`` and ``trace_flags``, the entry's own ``trace`` field carrying the trace, but a
-# field named like one is ``field_time`` or ``field_trace_id`` under either sink rather than under one only.
+# The sink writes some of the ``json`` sink's keys into the payload — its ``message``, ``logger`` and
+# ``exception``, and the ``pipelex.trace_id`` and ``pipelex.span_id`` of the held Pipelex span, all
+# imported rather than respelled — and reserves that sink's whole set against a carried attribute, so one
+# field keeps one wire name whichever of the two a process selects. ``time`` and ``severity`` are not
+# payload keys here, the client library carrying both out of band, and neither are ``trace_id``,
+# ``span_id`` and ``trace_flags``, the entry's own ``trace``, ``spanId`` and ``traceSampled`` carrying the
+# current span, but a field named like one is ``field_time`` or ``field_trace_id`` under either sink
+# rather than under one only.
FIXED_PAYLOAD_KEYS = FIXED_KEYS
# The stdlib formatter the ``json`` sink renders a traceback through, so both sinks spell one exception
@@ -286,14 +292,9 @@ def severity_for_level(*, levelno: int) -> GcpLogSeverity:
return GcpLogSeverity.CRITICAL
-def trace_name_for_run(*, project: str, pipeline_run_id: str) -> str:
- """The fully qualified Cloud Logging trace name for one pipeline run.
-
- The trace id is derived from the pipeline run id exactly as the tracer derives it, by the same
- hash, so a line and the spans of the run it belongs to carry one id without either having to
- reach the other. Cloud Logging wants that id project-qualified and as 32 hex digits.
- """
- return f"projects/{project}/traces/{hash_md5_to_int(pipeline_run_id):032x}"
+def trace_name(*, project: str, trace_id: int) -> str:
+ """The fully qualified Cloud Logging trace name for an OpenTelemetry trace id: project-qualified, as 32 hex digits."""
+ return f"projects/{project}/traces/{trace.format_trace_id(trace_id)}"
def _payload_value(*, value: Any) -> Any:
@@ -327,7 +328,7 @@ def _exception_text(*, record: logging.LogRecord) -> str | None:
def _entry_payload(*, record: logging.LogRecord) -> dict[str, Any]:
- """The struct payload for one record: the message, the logger, the exception, then the fields.
+ """The struct payload for one record: the message, the logger, the exception, the held Pipelex span, then the fields.
The run-scoped identifiers are left out: they are the entry's labels. Everything else the call,
the record factory or the structured content attached rides flat beside the fixed keys, under a
@@ -340,6 +341,7 @@ def _entry_payload(*, record: logging.LogRecord) -> dict[str, Any]:
exception_text = _exception_text(record=record)
if exception_text is not None:
payload[EXCEPTION_KEY] = exception_text
+ payload.update(pipelex_trace_fields_for_logs())
for name, value in carried_attributes(record=record).items():
if name in LABEL_ATTRIBUTES:
continue
@@ -350,6 +352,23 @@ def _entry_payload(*, record: logging.LogRecord) -> dict[str, Any]:
return payload
+def _entry_trace_context(*, project: str) -> dict[str, Any]:
+ """The entry's ``trace``, ``span_id`` and ``trace_sampled`` for OpenTelemetry's current span, or nothing when it names no trace.
+
+ Passed to the transport as keyword arguments, which the client library's transport forwards to the
+ entry it builds. Only ever the current span: a Pipelex span held here is the payload's, and a run
+ with no current span is filed under no trace, its ``pipeline_run_id`` label being what selects it.
+ """
+ span_context = current_span_context_for_logs()
+ if span_context is None:
+ return {}
+ return {
+ "trace": trace_name(project=project, trace_id=span_context.trace_id),
+ "span_id": trace.format_span_id(span_context.span_id),
+ "trace_sampled": span_context.trace_flags.sampled,
+ }
+
+
def _entry_labels(*, record: logging.LogRecord) -> dict[str, str]:
"""The entry's labels: the run-scoped identifiers the record carries, as text, and only the ones it has."""
labels: dict[str, str] = {}
@@ -523,14 +542,12 @@ class GcpLogHandler(logging.Handler):
@override
def emit(self, record: logging.LogRecord) -> None:
try:
- pipeline_run_id = getattr(record, PIPELINE_RUN_ID_FIELD, None)
- trace = trace_name_for_run(project=self._project, pipeline_run_id=pipeline_run_id) if isinstance(pipeline_run_id, str) else None
self._transport.send(
record,
_entry_payload(record=record),
severity=severity_for_level(levelno=record.levelno).value,
labels=_entry_labels(record=record),
- trace=trace,
+ **_entry_trace_context(project=self._project),
)
except Exception: # ruff: ignore[blind-except]
# Unbounded code: a third-party SDK with no documented exception types, over values the caller attached.
diff --git a/pipelex/tools/log/json_log_sink.py b/pipelex/tools/log/json_log_sink.py
index b5a942307..5e643d0dd 100644
--- a/pipelex/tools/log/json_log_sink.py
+++ b/pipelex/tools/log/json_log_sink.py
@@ -2,19 +2,20 @@
The keys are the ones a log agent ingests without a parser, the CloudWatch agent, the Google Cloud
Logging agent and any OTLP collector among them: ``time``, ``severity``, ``logger``, ``message``, then
-``exception`` when the record carries one, then ``trace_id``, ``span_id`` and ``trace_flags`` when the
-record was logged inside a valid span, then the fields, the context identifiers and the ``data``
-attribute under their own names. The trace keys are the ones OpenTelemetry specifies for trace context
-in a JSON log that is not OTLP, hex-encoded, so a collector or an error tracker joins the line to its
-trace without a parser. They name the Pipelex span active when the record is formatted, a pipe's or an
-LLM call's, or, outside one, OpenTelemetry's current span, which is only read; the stream handler formats
-inside the log call, in the calling task, and a boot line held until the sink arrives is replayed in the
-context it was logged in, so it keeps its span. The sink's own keys, the trace keys among them, are
-reserved whether or not the line carries them: a field named like one is carried under the same
-``field_`` prefix the record uses for a name the stdlib owns, on every line and not only the ones with
-an exception or a span, so no value is lost and a field keeps one wire name. A non-finite float is
-written as the string ``"NaN"``, ``"Infinity"`` or ``"-Infinity"``, since JSON has no token for it that
-a strict parser accepts.
+``exception`` when the record carries one, then ``trace_id``, ``span_id`` and ``trace_flags`` when
+OpenTelemetry's current span names a trace, then ``pipelex.trace_id`` and ``pipelex.span_id`` when a
+Pipelex span is held, then the fields, the context identifiers and the ``data`` attribute under their
+own names. The standard trace keys are the ones OpenTelemetry specifies for trace context in a JSON log
+that is not OTLP, hex-encoded, so a collector or an error tracker joins the line to the trace of the
+host's own spans without a parser; Pipelex only reads that span. The ``pipelex.*`` keys name the Pipelex
+span held when the record is formatted, a pipe's or an LLM call's, whether or not it is also the current
+span. The stream handler formats inside the log call, in the calling task, and a boot line held until
+the sink arrives is replayed in the context it was logged in, so it keeps both its spans. The sink's own
+keys, the trace keys among them, are reserved whether or not the line carries them: a field named like
+one is carried under the same ``field_`` prefix the record uses for a name the stdlib owns, on every line
+and not only the ones with an exception or a span, so no value is lost and a field keeps one wire name.
+A non-finite float is written as the string ``"NaN"``, ``"Infinity"`` or ``"-Infinity"``, since JSON has
+no token for it that a strict parser accepts.
"""
from __future__ import annotations
@@ -28,7 +29,12 @@ from typing import TYPE_CHECKING, Any
from opentelemetry import trace
from typing_extensions import override
-from pipelex.system.telemetry.current_span import span_context_for_logs
+from pipelex.system.telemetry.current_span import (
+ PIPELEX_SPAN_ID_KEY,
+ PIPELEX_TRACE_ID_KEY,
+ current_span_context_for_logs,
+ pipelex_trace_fields_for_logs,
+)
from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes
from pipelex.tools.log.log_sink import LogSink, json_fallback, spell_non_finite
@@ -44,7 +50,20 @@ TRACE_ID_KEY = "trace_id"
SPAN_ID_KEY = "span_id"
TRACE_FLAGS_KEY = "trace_flags"
# The keys the sink writes itself, reserved on every line; a carried attribute of the same name is prefixed.
-FIXED_KEYS = frozenset({TIME_KEY, SEVERITY_KEY, LOGGER_KEY, MESSAGE_KEY, EXCEPTION_KEY, TRACE_ID_KEY, SPAN_ID_KEY, TRACE_FLAGS_KEY})
+FIXED_KEYS = frozenset(
+ {
+ TIME_KEY,
+ SEVERITY_KEY,
+ LOGGER_KEY,
+ MESSAGE_KEY,
+ EXCEPTION_KEY,
+ TRACE_ID_KEY,
+ SPAN_ID_KEY,
+ TRACE_FLAGS_KEY,
+ PIPELEX_TRACE_ID_KEY,
+ PIPELEX_SPAN_ID_KEY,
+ }
+)
def _json_line(*, payload: dict[str, Any]) -> str:
@@ -64,15 +83,15 @@ def _json_line(*, payload: dict[str, Any]) -> str:
def _trace_context() -> dict[str, str]:
- """The trace context of the span a line logged here is joined to, under the OpenTelemetry JSON keys, or nothing outside one."""
- span_context = span_context_for_logs()
- if span_context is None:
- return {}
- return {
- TRACE_ID_KEY: trace.format_trace_id(span_context.trace_id),
- SPAN_ID_KEY: trace.format_span_id(span_context.span_id),
- TRACE_FLAGS_KEY: f"{span_context.trace_flags:02x}",
- }
+ """The current span under the OpenTelemetry JSON keys, then the held Pipelex span under the ``pipelex.*`` ones, each only if present."""
+ trace_context: dict[str, str] = {}
+ span_context = current_span_context_for_logs()
+ if span_context is not None:
+ trace_context[TRACE_ID_KEY] = trace.format_trace_id(span_context.trace_id)
+ trace_context[SPAN_ID_KEY] = trace.format_span_id(span_context.span_id)
+ trace_context[TRACE_FLAGS_KEY] = f"{span_context.trace_flags:02x}"
+ trace_context.update(pipelex_trace_fields_for_logs())
+ return trace_context
def _iso_utc(*, created: float) -> str:
diff --git a/pipelex/tools/log/otlp_log_sink.py b/pipelex/tools/log/otlp_log_sink.py
index 83d12bfe1..7eaa4ff19 100644
--- a/pipelex/tools/log/otlp_log_sink.py
+++ b/pipelex/tools/log/otlp_log_sink.py
@@ -4,14 +4,16 @@ Each record becomes one OTel log record on the logger named after the emitting m
is the body, the level maps onto the OTel severity scale, the record's fields, context identifiers and
``data`` ride as attributes, with a value the wire cannot carry as is written as JSON text, and an
exception lands under the ``exception.*`` semantic-convention keys, the stacktrace being the record's
-rendered exception text so that what the redaction processor scrubbed is what leaves. The
-semantic-convention keys the sink writes itself — the source location and the exception — are reserved
-whether or not the record carries an exception, exactly as the ``json`` sink reserves its own keys: a
-field named like one is carried under the same ``field_`` prefix, so the same field survives a change of
-sink. Each record is filed under the Pipelex span active when it was logged, a pipe's or an LLM
-call's, or, outside one, under OpenTelemetry's current span, which is only read. The records the sink's
-own export path emits, the SDK's and the transport's, are rejected by a filter on the handler and never
-exported. This module imports the OpenTelemetry SDK at load, which is why the built-in plugin imports it
+rendered exception text so that what the redaction processor scrubbed is what leaves. The keys the
+sink writes itself — the source location and the exception under their semantic-convention names, and
+Pipelex's own ``pipelex.trace_id`` and ``pipelex.span_id`` — are reserved whether or not the record
+carries them, exactly as the ``json`` sink reserves its own keys: a field named like one is carried under
+the same ``field_`` prefix, so the same field survives a change of sink. Each record is filed under
+OpenTelemetry's current span when it was logged, the host's own, which is only read, and under no span
+when it names no trace; the Pipelex span held when it was logged, a pipe's or an LLM call's, rides
+beside it as the two ``pipelex.*`` attributes, in hex, whether or not it is also the current span. The
+records the sink's own export path emits, the SDK's and the transport's, are rejected by a filter on the
+handler and never exported. This module imports the OpenTelemetry SDK at load, which is why the built-in plugin imports it
inside the ``otlp`` factory and nowhere else.
"""
@@ -25,6 +27,7 @@ from typing import TYPE_CHECKING, Any, cast
from opentelemetry._logs import SeverityNumber # ruff: ignore[import-private-name]
from opentelemetry.context import (
_SUPPRESS_INSTRUMENTATION_KEY, # ruff: ignore[import-private-name] # pyright: ignore[reportPrivateUsage]
+ get_current,
get_value,
)
from opentelemetry.sdk._logs import LoggerProvider # ruff: ignore[import-private-name]
@@ -32,7 +35,7 @@ from opentelemetry.semconv._incubating.attributes import code_attributes # ruff
from opentelemetry.semconv.attributes import exception_attributes
from typing_extensions import override
-from pipelex.system.telemetry.current_span import otel_context_for_logs
+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY, PIPELEX_TRACE_ID_KEY, pipelex_trace_fields_for_logs
from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes
from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE
from pipelex.tools.log.log_sink import LogSink, render_json
@@ -55,15 +58,18 @@ FLUSH_TIMEOUT_MILLIS = 5000
_ATTRIBUTE_SCALAR_TYPES = (str, bool, int, float)
-# The semantic-convention keys the sink writes itself, reserved on every record whether or not it carries
-# an exception: a field named like one is carried under the same ``field_`` prefix the record uses for a
-# name the stdlib owns, so no value is lost, a field keeps one wire name whatever the record holds, and the
-# same field survives this sink as it survives the ``json`` one.
+# The keys the sink writes itself, the semantic-convention ones and Pipelex's own ``pipelex.*`` pair,
+# reserved on every record whether or not it carries an exception or a span: a field named like one is
+# carried under the same ``field_`` prefix the record uses for a name the stdlib owns, so no value is
+# lost, a field keeps one wire name whatever the record holds, and the same field survives this sink as it
+# survives the ``json`` one.
RESERVED_ATTRIBUTE_KEYS = frozenset(
{
code_attributes.CODE_FILE_PATH,
code_attributes.CODE_FUNCTION_NAME,
code_attributes.CODE_LINE_NUMBER,
+ PIPELEX_TRACE_ID_KEY,
+ PIPELEX_SPAN_ID_KEY,
exception_attributes.EXCEPTION_TYPE,
exception_attributes.EXCEPTION_MESSAGE,
exception_attributes.EXCEPTION_STACKTRACE,
@@ -146,7 +152,7 @@ class OtlpLogHandler(logging.Handler):
logger.emit(
timestamp=int(record.created * 1e9),
observed_timestamp=time_ns(),
- context=otel_context_for_logs(),
+ context=get_current(),
severity_text=_severity_text(levelname=record.levelname),
severity_number=_severity_number(levelno=record.levelno),
body=record.getMessage(),
@@ -168,7 +174,7 @@ class OtlpLogHandler(logging.Handler):
@staticmethod
def _attributes(*, record: logging.LogRecord) -> dict[str, Any]:
- """The record's source location, its exception and everything it carries, the sink's own keys first.
+ """The record's source location, the held Pipelex span, its exception and everything it carries, the sink's own keys first.
The sink's keys go down before the carried attributes so that a field named like one is prefixed
rather than silently overwriting the location or the exception it names — and the reservation covers
@@ -180,6 +186,7 @@ class OtlpLogHandler(logging.Handler):
code_attributes.CODE_FUNCTION_NAME: record.funcName,
code_attributes.CODE_LINE_NUMBER: record.lineno,
}
+ attributes.update(pipelex_trace_fields_for_logs())
if record.exc_info:
exc_type, exc_value, exc_traceback = record.exc_info
if exc_type is not None:
diff --git a/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py b/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
index e0094d290..714cbe0a3 100644
--- a/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
+++ b/tests/unit/pipelex/cogt/llm/test_llm_worker_current_span.py
@@ -2,7 +2,7 @@
The span is started with the pipe's span as its explicit parent, read off the job metadata, and that
stays as it was; what changes is that the provider call runs with the span held, so a log line during
-the call is joined to it, and the previous one comes back when the span ends, whichever way it ends.
+the call names it under ``pipelex.*``, and the previous one comes back when the span ends, whichever way it ends.
OpenTelemetry's current span is never touched: inside the call it is whatever it was outside, the host's
or none, so a provider SDK's own instrumentation is never re-parented under the LLM span. Driven through
the public ``gen_text`` and ``gen_object`` on a worker whose provider half records the spans it runs under.
@@ -32,7 +32,7 @@ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
-from pipelex.system.telemetry.current_span import span_context_for_logs
+from pipelex.system.telemetry.current_span import pipelex_span_context_for_logs
from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
@@ -56,7 +56,7 @@ class _Answer(BaseModel):
class _RecordingLLMWorker(LLMWorkerAbstract):
- """A worker whose provider half records the span a log line is joined to and OpenTelemetry's current span, and fails when asked to."""
+ """A worker whose provider half records the Pipelex span a log line names and OpenTelemetry's current span, and fails when asked to."""
def __init__(self, *, inference_model: InferenceModelSpec, fails: bool) -> None:
LLMWorkerAbstract.__init__(self, inference_model=inference_model, reporting_delegate=None)
@@ -65,7 +65,7 @@ class _RecordingLLMWorker(LLMWorkerAbstract):
self.seen_current_spans: list[object] = []
def _record(self) -> None:
- log_span_context = span_context_for_logs()
+ log_span_context = pipelex_span_context_for_logs()
self.seen_log_span_ids.append(INVALID_SPAN_ID if log_span_context is None else log_span_context.span_id)
self.seen_current_spans.append(trace.get_current_span())
if self.fails:
@@ -84,7 +84,7 @@ class _RecordingLLMWorker(LLMWorkerAbstract):
class _SpanRecordingHandler(logging.Handler):
- """Records each message with the span id a sink would write for it, read at emit as a sink reads it."""
+ """Records each message with the Pipelex span id a sink would write for it under ``pipelex.span_id``, read at emit as a sink reads it."""
def __init__(self) -> None:
super().__init__(level=logging.NOTSET)
@@ -92,7 +92,7 @@ class _SpanRecordingHandler(logging.Handler):
@override
def emit(self, record: logging.LogRecord) -> None:
- span_context = span_context_for_logs()
+ span_context = pipelex_span_context_for_logs()
self.span_ids_by_message[record.getMessage()] = INVALID_SPAN_ID if span_context is None else span_context.span_id
@@ -173,7 +173,7 @@ class TestLLMWorkerCurrentSpan:
assert worker.seen_log_span_ids == [_span_id(llm_span)]
assert llm_span.parent is not None
assert llm_span.parent.span_id == PIPE_SPAN_ID
- assert span_context_for_logs() is None
+ assert pipelex_span_context_for_logs() is None
async def test_gen_object_runs_the_provider_call_under_its_span(self, span_exporter: InMemorySpanExporter) -> None:
worker = _make_worker()
@@ -182,7 +182,7 @@ class TestLLMWorkerCurrentSpan:
llm_span = _only_span(span_exporter)
assert worker.seen_log_span_ids == [_span_id(llm_span)]
- assert span_context_for_logs() is None
+ assert pipelex_span_context_for_logs() is None
async def test_the_call_never_makes_its_span_opentelemetrys_current_one(self, span_exporter: InMemorySpanExporter) -> None:
"""Inside the call the current span is whatever it was outside, so the provider SDK's instrumentation is never re-parented."""
@@ -207,7 +207,7 @@ class TestLLMWorkerCurrentSpan:
llm_span = _only_span(span_exporter)
assert worker.seen_log_span_ids == [_span_id(llm_span)]
- assert span_context_for_logs() is None
+ assert pipelex_span_context_for_logs() is None
# The worker's own error path records the failure; holding the span records nothing more.
assert llm_span.status.status_code is StatusCode.ERROR
assert [event.name for event in llm_span.events] == ["exception"]
diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
index 469cc1564..6a0d0751a 100644
--- a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_current_span.py
@@ -3,10 +3,12 @@
The span is started with an explicit parent read off the job metadata, which is how a child running in
another process reaches it, and that stays as it was: the pipe only holds its already-started span for
the in-process work, and restores the previous one when the span ends, however it ends. Both wire sinks
-read the active Pipelex span, so a line logged during the run names the pipe's span: the ``json`` sink
-under ``trace_id`` and ``span_id``, the ``otlp`` sink as the record's own trace context. OpenTelemetry's
-current span is never touched: inside the run it is whatever it was outside, the host's or none, so the
-host's own instrumentation is never re-parented under the pipe. Each test runs once per sink.
+read the active Pipelex span, so a line logged during the run names the pipe's span under
+``pipelex.trace_id`` and ``pipelex.span_id``, as keys in the ``json`` sink and as attributes in the
+``otlp`` one, while the line's standard trace fields name OpenTelemetry's current span. That span is
+never touched: inside the run it is whatever it was outside, the host's or none, so the host's own
+instrumentation is never re-parented under the pipe, and a line in a run under no host span carries no
+standard trace fields at all. Each test runs once per sink.
"""
from __future__ import annotations
@@ -38,7 +40,7 @@ from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.system.configuration.config_loader import ConfigLoader
from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
-from pipelex.system.telemetry.current_span import span_context_for_logs
+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY, PIPELEX_TRACE_ID_KEY, pipelex_span_context_for_logs
from pipelex.system.telemetry.otel_constants import OTelConstants
from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, SPAN_ID_KEY, TRACE_ID_KEY, JsonLogSink
@@ -72,10 +74,15 @@ RUN_OTEL_CONTEXT = OtelContext(
class LineTrace(NamedTuple):
- """The trace context a sink wrote for one line, as hex, or ``None`` where the line carries none."""
+ """The trace context a sink wrote for one line, as hex, or ``None`` where the line carries none.
- trace_id: str | None
- span_id: str | None
+ The standard fields name OpenTelemetry's current span; the ``pipelex_*`` ones the Pipelex span held.
+ """
+
+ trace_id: str | None = None
+ span_id: str | None = None
+ pipelex_trace_id: str | None = None
+ pipelex_span_id: str | None = None
class Sunk(NamedTuple):
@@ -91,7 +98,7 @@ class Sunk(NamedTuple):
class CurrentSpanPipe(PipeAbstract):
"""A pipe whose live run logs a line and records the spans it runs under; it may nest a pipe, and it may fail.
- It records the span a line logged there is joined to, and OpenTelemetry's current span, which is what the
+ It records the Pipelex span a line logged there names, and OpenTelemetry's current span, which is what the
host's own instrumentation would open its spans under.
"""
@@ -145,7 +152,7 @@ class CurrentSpanPipe(PipeAbstract):
output_name: str | None = None,
library_crate: LibraryCrate | None = None,
) -> PipeOutput:
- log_span_context = span_context_for_logs()
+ log_span_context = pipelex_span_context_for_logs()
self.seen_log_span_ids.append(INVALID_SPAN_ID if log_span_context is None else log_span_context.span_id)
self.seen_current_spans.append(trace.get_current_span())
log.info(f"inside {self.code}")
@@ -209,7 +216,12 @@ def _package_log_config() -> LogConfig:
def _json_lines(buffer: io.StringIO, *, logger_name: str) -> dict[str, LineTrace]:
lines = [json.loads(line) for line in buffer.getvalue().splitlines() if line]
return {
- line[MESSAGE_KEY]: LineTrace(trace_id=line.get(TRACE_ID_KEY), span_id=line.get(SPAN_ID_KEY))
+ line[MESSAGE_KEY]: LineTrace(
+ trace_id=line.get(TRACE_ID_KEY),
+ span_id=line.get(SPAN_ID_KEY),
+ pipelex_trace_id=line.get(PIPELEX_TRACE_ID_KEY),
+ pipelex_span_id=line.get(PIPELEX_SPAN_ID_KEY),
+ )
for line in lines
if line[LOGGER_KEY] == logger_name
}
@@ -223,7 +235,15 @@ def _otlp_lines(exporter: InMemoryLogExporter, *, logger_name: str) -> dict[str,
record = log_data.log_record
trace_id = f"{record.trace_id:032x}" if record.trace_id else None
span_id = f"{record.span_id:016x}" if record.span_id else None
- lines[str(record.body)] = LineTrace(trace_id=trace_id, span_id=span_id)
+ attributes = dict(record.attributes or {})
+ pipelex_trace_id = attributes.get(PIPELEX_TRACE_ID_KEY)
+ pipelex_span_id = attributes.get(PIPELEX_SPAN_ID_KEY)
+ lines[str(record.body)] = LineTrace(
+ trace_id=trace_id,
+ span_id=span_id,
+ pipelex_trace_id=None if pipelex_trace_id is None else str(pipelex_trace_id),
+ pipelex_span_id=None if pipelex_span_id is None else str(pipelex_span_id),
+ )
return lines
@@ -288,11 +308,12 @@ class TestLiveRunPipeCurrentSpan:
await _run(pipe)
pipe_span = _span_named(span_exporter.get_finished_spans(), code="solo")
- assert sunk.read_lines()["inside solo"] == LineTrace(trace_id=f"{RUN_TRACE_ID:032x}", span_id=_hex_span_id(pipe_span))
+ # No host span: the standard fields are absent, and the pipe's span rides under ``pipelex.*``.
+ assert sunk.read_lines()["inside solo"] == LineTrace(pipelex_trace_id=f"{RUN_TRACE_ID:032x}", pipelex_span_id=_hex_span_id(pipe_span))
assert pipe.seen_log_span_ids == [_span_id(pipe_span)]
- assert span_context_for_logs() is None
+ assert pipelex_span_context_for_logs() is None
log.info("after the run")
- assert sunk.read_lines()["after the run"] == LineTrace(trace_id=None, span_id=None)
+ assert sunk.read_lines()["after the run"] == LineTrace()
async def test_the_run_never_makes_its_span_opentelemetrys_current_one(self, sunk: Sunk, span_exporter: InMemorySpanExporter) -> None:
"""Inside the run the current span is whatever it was outside, so the host's instrumentation is never re-parented."""
@@ -306,9 +327,15 @@ class TestLiveRunPipeCurrentSpan:
assert without_host.seen_current_spans == [trace.INVALID_SPAN]
assert under_host.seen_current_spans == [host_span]
- # The line still names the pipe's span, not the host's.
+ # The line's standard fields name the host's span, and the pipe's span rides beside it under ``pipelex.*``.
pipe_span = _span_named(span_exporter.get_finished_spans(), code="hosted")
- assert sunk.read_lines()["inside hosted"] == LineTrace(trace_id=f"{RUN_TRACE_ID:032x}", span_id=_hex_span_id(pipe_span))
+ host_context = host_span.get_span_context()
+ assert sunk.read_lines()["inside hosted"] == LineTrace(
+ trace_id=f"{host_context.trace_id:032x}",
+ span_id=f"{host_context.span_id:016x}",
+ pipelex_trace_id=f"{RUN_TRACE_ID:032x}",
+ pipelex_span_id=_hex_span_id(pipe_span),
+ )
async def test_a_span_started_line_names_the_span_it_announces(
self, sunk: Sunk, span_exporter: InMemorySpanExporter, caplog: pytest.LogCaptureFixture
@@ -323,11 +350,11 @@ class TestLiveRunPipeCurrentSpan:
started = {
message: line for message, line in sunk.read_lines_of(PIPE_ABSTRACT_LOGGER).items() if message.startswith("[OTel] PIPE SPAN STARTED")
}
- assert sorted(line.span_id or "" for line in started.values()) == sorted(
+ assert sorted(line.pipelex_span_id or "" for line in started.values()) == sorted(
_hex_span_id(_span_named(spans, code=code)) for code in ("outer", "inner")
)
for message, line in started.items():
- assert f"\n span_id={line.span_id}\n" in message
+ assert f"\n span_id={line.pipelex_span_id}\n" in message
async def test_a_nested_pipe_holds_its_own_span_and_the_outer_one_comes_back(self, sunk: Sunk, span_exporter: InMemorySpanExporter) -> None:
outer = _make_pipe(code="outer", nested=_make_pipe(code="inner"))
@@ -339,9 +366,9 @@ class TestLiveRunPipeCurrentSpan:
inner_span = _span_named(spans, code="inner")
lines = sunk.read_lines()
run_trace = f"{RUN_TRACE_ID:032x}"
- assert lines["inside outer"] == LineTrace(trace_id=run_trace, span_id=_hex_span_id(outer_span))
- assert lines["inside inner"] == LineTrace(trace_id=run_trace, span_id=_hex_span_id(inner_span))
- assert lines["back in outer"] == LineTrace(trace_id=run_trace, span_id=_hex_span_id(outer_span))
+ assert lines["inside outer"] == LineTrace(pipelex_trace_id=run_trace, pipelex_span_id=_hex_span_id(outer_span))
+ assert lines["inside inner"] == LineTrace(pipelex_trace_id=run_trace, pipelex_span_id=_hex_span_id(inner_span))
+ assert lines["back in outer"] == LineTrace(pipelex_trace_id=run_trace, pipelex_span_id=_hex_span_id(outer_span))
# The nested span's parent still arrives through the job metadata, exactly as it did.
assert inner_span.parent is not None
assert inner_span.parent.span_id == _span_id(outer_span)
@@ -355,8 +382,8 @@ class TestLiveRunPipeCurrentSpan:
await _run(pipe)
pipe_span = _span_named(span_exporter.get_finished_spans(), code="failing")
- assert sunk.read_lines()["inside failing"].span_id == _hex_span_id(pipe_span)
- assert span_context_for_logs() is None
+ assert sunk.read_lines()["inside failing"].pipelex_span_id == _hex_span_id(pipe_span)
+ assert pipelex_span_context_for_logs() is None
# The pipe's own error hook records the failure; holding the span records nothing more.
assert pipe_span.status.status_code is StatusCode.ERROR
assert [event.name for event in pipe_span.events] == ["exception"]
@@ -386,7 +413,7 @@ class TestLiveRunPipeCurrentSpan:
pipe_span = _span_named(span_exporter.get_finished_spans(), code="cancelled")
assert pipe_span.status.status_code is StatusCode.ERROR
- assert sunk.read_lines()["inside cancelled"].span_id == _hex_span_id(pipe_span)
+ assert sunk.read_lines()["inside cancelled"].pipelex_span_id == _hex_span_id(pipe_span)
async def test_a_no_op_tracer_leaves_the_callers_span_to_the_line(self, sunk: Sunk, mocker: MockerFixture) -> None:
"""A no-op tracer, what the SDK hands out under ``OTEL_SDK_DISABLED``, starts spans naming no trace, which must not hide the caller's."""
diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
index 80e01f4f5..e3f619564 100644
--- a/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
@@ -1,9 +1,9 @@
"""The boto3 Bedrock client runs its blocking call in a thread that keeps the caller's context.
The LLM worker holds its generation span as the Pipelex span active around the provider call, and the
-promise is that the SDK's own log lines during the call are joined to it, while a span the SDK's
-instrumentation opens has the caller's current span as its parent. The boto3 client blocks, so it runs in
-a thread, and only a thread that carries the context over keeps both there.
+promise is that the SDK's own log lines during the call name it under ``pipelex.*``, while their standard
+trace fields, and any span the SDK's instrumentation opens, follow the caller's current span. The boto3
+client blocks, so it runs in a thread, and only a thread that carries the context over keeps both there.
"""
from __future__ import annotations
@@ -16,7 +16,7 @@ from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import INVALID_SPAN_ID
from pipelex.providers.bedrock.bedrock_client_boto3 import BedrockClientBoto3
-from pipelex.system.telemetry.current_span import pipelex_span_active, span_context_for_logs
+from pipelex.system.telemetry.current_span import current_span_context_for_logs, pipelex_span_active, pipelex_span_context_for_logs
class _RecordingConverse:
@@ -24,11 +24,14 @@ class _RecordingConverse:
def __init__(self) -> None:
self.seen_log_span_ids: list[int] = []
+ self.seen_log_current_span_ids: list[int] = []
self.seen_current_spans: list[object] = []
def converse(self, **_params: Any) -> dict[str, Any]:
- log_span_context = span_context_for_logs()
+ log_span_context = pipelex_span_context_for_logs()
self.seen_log_span_ids.append(INVALID_SPAN_ID if log_span_context is None else log_span_context.span_id)
+ log_current_span_context = current_span_context_for_logs()
+ self.seen_log_current_span_ids.append(INVALID_SPAN_ID if log_current_span_context is None else log_current_span_context.span_id)
self.seen_current_spans.append(trace.get_current_span())
return {"usage": {"inputTokens": 1, "outputTokens": 2}, "output": {"message": {"content": [{"text": "answer"}]}}}
@@ -47,4 +50,5 @@ class TestBedrockBoto3CurrentSpan:
assert text == "answer"
assert recording.seen_log_span_ids == [llm_span.get_span_context().span_id]
+ assert recording.seen_log_current_span_ids == [host_span.get_span_context().span_id]
assert recording.seen_current_spans == [host_span]
diff --git a/tests/unit/pipelex/system/telemetry/test_current_span.py b/tests/unit/pipelex/system/telemetry/test_current_span.py
index 9875e2879..cac8b723f 100644
--- a/tests/unit/pipelex/system/telemetry/test_current_span.py
+++ b/tests/unit/pipelex/system/telemetry/test_current_span.py
@@ -1,12 +1,11 @@
-"""``pipelex_span_active`` holds a Pipelex span for a block, and the log readers join a line to it.
-
-A line logged in the block is joined to the Pipelex span, and outside any Pipelex span to OpenTelemetry's
-current span, which is only read. OpenTelemetry's current context is never touched: inside the block the
-current span is whatever it was outside, the host's or none. The span stays owned by the code that
-started it: the block never ends it, never records an exception on it and never sets its status, even
-when an exception crosses the block while the span is still recording, and the previous Pipelex span
-comes back on the way out, however the block exits. A ``None`` span, which is what the runtime holds
-when it has no tracer, and a span naming no trace, which is what a no-op tracer starts, hold nothing.
+"""``pipelex_span_active`` holds a Pipelex span for a block, where the ``pipelex.*`` log fields read it.
+
+OpenTelemetry's current context is never touched: inside the block the current span is whatever it was
+outside, the host's or none. The span stays owned by the code that started it: the block never ends it,
+never records an exception on it and never sets its status, even when an exception crosses the block
+while the span is still recording, and the previous Pipelex span comes back on the way out, however the
+block exits. A ``None`` span, which is what the runtime holds when it has no tracer, and a span naming
+no trace, which is what a no-op tracer starts, hold nothing.
"""
from __future__ import annotations
@@ -17,7 +16,7 @@ from opentelemetry.sdk.trace import Span as SdkSpan
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import StatusCode
-from pipelex.system.telemetry.current_span import otel_context_for_logs, pipelex_span_active, span_context_for_logs
+from pipelex.system.telemetry.current_span import pipelex_span_active, pipelex_span_context_for_logs
def _started_span(*, name: str) -> SdkSpan:
@@ -33,25 +32,17 @@ def _fail_under(*, span: SdkSpan) -> None:
class TestPipelexSpanActive:
- def test_a_line_joins_the_pipelex_span_inside_the_block_and_the_previous_one_after(self) -> None:
+ def test_the_innermost_pipelex_span_is_held_inside_the_block_and_the_previous_one_after(self) -> None:
outer = _started_span(name="outer")
inner = _started_span(name="inner")
with pipelex_span_active(span=outer):
with pipelex_span_active(span=inner):
- assert span_context_for_logs() == inner.get_span_context()
- assert span_context_for_logs() == outer.get_span_context()
- assert span_context_for_logs() is None
+ assert pipelex_span_context_for_logs() == inner.get_span_context()
+ assert pipelex_span_context_for_logs() == outer.get_span_context()
+ assert pipelex_span_context_for_logs() is None
assert inner.is_recording()
- def test_the_pipelex_span_takes_precedence_over_the_hosts_current_span(self) -> None:
- host = _started_span(name="host")
- pipelex_span = _started_span(name="pipe")
-
- with trace.use_span(host), pipelex_span_active(span=pipelex_span):
- assert span_context_for_logs() == pipelex_span.get_span_context()
- assert trace.get_current_span(otel_context_for_logs()) is pipelex_span
-
def test_opentelemetrys_current_span_is_never_touched(self) -> None:
host = _started_span(name="host")
pipelex_span = _started_span(name="pipe")
@@ -61,46 +52,30 @@ class TestPipelexSpanActive:
with trace.use_span(host), pipelex_span_active(span=pipelex_span):
assert trace.get_current_span() is host
- def test_outside_any_pipelex_span_a_line_joins_the_hosts_current_span(self) -> None:
- host = _started_span(name="host")
-
- with trace.use_span(host):
- assert span_context_for_logs() == host.get_span_context()
- assert trace.get_current_span(otel_context_for_logs()) is host
-
- def test_outside_any_span_a_line_joins_none(self) -> None:
- assert span_context_for_logs() is None
- with trace.use_span(trace.INVALID_SPAN):
- assert span_context_for_logs() is None
- assert not trace.get_current_span(otel_context_for_logs()).get_span_context().is_valid
-
def test_an_exception_crossing_the_block_leaves_the_span_as_it_was(self) -> None:
span = _started_span(name="work")
with pytest.raises(ValueError, match="boom"):
_fail_under(span=span)
- assert span_context_for_logs() is None
+ assert pipelex_span_context_for_logs() is None
assert span.is_recording()
assert span.status.status_code is StatusCode.UNSET
assert list(span.events) == []
def test_a_none_span_holds_nothing(self) -> None:
enclosing = _started_span(name="enclosing")
- host = _started_span(name="host")
+ with pipelex_span_active(span=None):
+ assert pipelex_span_context_for_logs() is None
with pipelex_span_active(span=enclosing), pipelex_span_active(span=None):
- assert span_context_for_logs() == enclosing.get_span_context()
- with trace.use_span(host), pipelex_span_active(span=None):
- assert span_context_for_logs() == host.get_span_context()
+ assert pipelex_span_context_for_logs() == enclosing.get_span_context()
- def test_a_span_naming_no_trace_hides_neither_the_enclosing_pipelex_span_nor_the_hosts(self) -> None:
+ def test_a_span_naming_no_trace_does_not_hide_the_enclosing_pipelex_span(self) -> None:
"""What a no-op tracer starts, under ``OTEL_SDK_DISABLED`` for one, must not hide a valid span."""
enclosing = _started_span(name="enclosing")
- host = _started_span(name="host")
+ with pipelex_span_active(span=trace.INVALID_SPAN):
+ assert pipelex_span_context_for_logs() is None
with pipelex_span_active(span=enclosing), pipelex_span_active(span=trace.INVALID_SPAN):
- assert span_context_for_logs() == enclosing.get_span_context()
- with trace.use_span(host), pipelex_span_active(span=trace.INVALID_SPAN):
- assert span_context_for_logs() == host.get_span_context()
- assert trace.get_current_span(otel_context_for_logs()) is host
+ assert pipelex_span_context_for_logs() == enclosing.get_span_context()
diff --git a/tests/unit/pipelex/system/telemetry/test_current_span_log_readers.py b/tests/unit/pipelex/system/telemetry/test_current_span_log_readers.py
new file mode 100644
index 000000000..97d3ba3a6
--- /dev/null
+++ b/tests/unit/pipelex/system/telemetry/test_current_span_log_readers.py
@@ -0,0 +1,91 @@
+"""The log readers read the two spans a line names, each on its own.
+
+A line's standard trace fields name OpenTelemetry's current span, which is only read, and its
+``pipelex.*`` fields name the Pipelex span held in the task. The two are independent: a Pipelex span
+held never hides the current span, and the current span never stands in for a Pipelex one. A span naming
+no trace reads as no span at all, and the ``pipelex.*`` ids are written in lowercase hex at their full
+widths.
+"""
+
+from __future__ import annotations
+
+from opentelemetry import trace
+from opentelemetry.sdk.trace import Span as SdkSpan
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.trace import NonRecordingSpan, SpanContext, TraceFlags
+
+from pipelex.system.telemetry.current_span import (
+ PIPELEX_SPAN_ID_KEY,
+ PIPELEX_TRACE_ID_KEY,
+ current_span_context_for_logs,
+ pipelex_span_active,
+ pipelex_span_context_for_logs,
+ pipelex_trace_fields_for_logs,
+)
+
+# A span whose ids are small enough that only a full-width, zero-padded spelling gets them right.
+_SMALL_IDS_SPAN = NonRecordingSpan(SpanContext(trace_id=0xAB, span_id=0xCD, is_remote=False, trace_flags=TraceFlags(TraceFlags.SAMPLED)))
+
+
+def _started_span(*, name: str) -> SdkSpan:
+ span = TracerProvider().get_tracer(__name__).start_span(name)
+ assert isinstance(span, SdkSpan)
+ return span
+
+
+class TestCurrentSpanLogReaders:
+ def test_the_hosts_current_span_is_read_inside_and_outside_a_pipelex_span(self) -> None:
+ host = _started_span(name="host")
+ pipelex_span = _started_span(name="pipe")
+
+ with trace.use_span(host):
+ assert current_span_context_for_logs() == host.get_span_context()
+ with pipelex_span_active(span=pipelex_span):
+ assert current_span_context_for_logs() == host.get_span_context()
+
+ def test_a_pipelex_span_never_stands_in_for_a_missing_current_span(self) -> None:
+ with pipelex_span_active(span=_started_span(name="pipe")):
+ assert current_span_context_for_logs() is None
+
+ def test_a_current_span_naming_no_trace_reads_as_none(self) -> None:
+ assert current_span_context_for_logs() is None
+ with trace.use_span(trace.INVALID_SPAN):
+ assert current_span_context_for_logs() is None
+
+ def test_the_hosts_current_span_never_stands_in_for_a_missing_pipelex_span(self) -> None:
+ with trace.use_span(_started_span(name="host")):
+ assert pipelex_span_context_for_logs() is None
+
+ def test_the_held_pipelex_span_is_read_under_the_hosts_current_span(self) -> None:
+ pipelex_span = _started_span(name="pipe")
+
+ with trace.use_span(_started_span(name="host")), pipelex_span_active(span=pipelex_span):
+ assert pipelex_span_context_for_logs() == pipelex_span.get_span_context()
+
+ def test_the_pipelex_fields_write_the_held_span_as_both_ids_in_lowercase_hex(self) -> None:
+ pipelex_span = _started_span(name="pipe")
+ span_context = pipelex_span.get_span_context()
+ assert span_context is not None
+
+ with pipelex_span_active(span=pipelex_span):
+ fields = pipelex_trace_fields_for_logs()
+
+ assert fields == {
+ PIPELEX_TRACE_ID_KEY: f"{span_context.trace_id:032x}",
+ PIPELEX_SPAN_ID_KEY: f"{span_context.span_id:016x}",
+ }
+
+ def test_the_pipelex_fields_zero_pad_the_ids_to_their_full_widths(self) -> None:
+ with pipelex_span_active(span=_SMALL_IDS_SPAN):
+ fields = pipelex_trace_fields_for_logs()
+
+ assert fields == {PIPELEX_TRACE_ID_KEY: "000000000000000000000000000000ab", PIPELEX_SPAN_ID_KEY: "00000000000000cd"}
+
+ def test_no_held_span_writes_no_pipelex_fields_even_under_the_hosts_current_span(self) -> None:
+ assert pipelex_trace_fields_for_logs() == {}
+ with trace.use_span(_started_span(name="host")):
+ assert pipelex_trace_fields_for_logs() == {}
+
+ def test_the_pipelex_field_keys_are_namespaced_under_pipelex(self) -> None:
+ assert PIPELEX_TRACE_ID_KEY == "pipelex.trace_id"
+ assert PIPELEX_SPAN_ID_KEY == "pipelex.span_id"
diff --git a/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py b/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py
index deee96d2f..5b86b7cad 100644
--- a/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py
+++ b/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py
@@ -23,13 +23,14 @@ from google.cloud.logging_v2.handlers.transports.background_thread import (
_Worker, # pyright: ignore[reportPrivateUsage]
)
from google.cloud.logging_v2.handlers.transports.base import Transport
+from google.cloud.logging_v2.logger import Batch
from pipelex.tools.log.gcp_log_sink import (
GCP_LOGGING_LOGGER_PREFIXES,
GCP_WORKER_THREAD_NAME,
MESSAGE_KEY,
GcpLogSeverity,
- trace_name_for_run,
+ trace_name,
)
@@ -51,13 +52,30 @@ class TestTheClientLibraryContract:
assert queued["labels"]["python_logger"] == "pipelex"
def test_a_struct_entry_renders_the_severity_name_and_the_trace_the_sink_passes(self) -> None:
- trace_name = trace_name_for_run(project="p", pipeline_run_id="plr-1")
+ entry_trace = trace_name(project="p", trace_id=0xAB)
# The library builds its entries from a namedtuple whose fields mypy does not see.
- entry: Any = StructEntry(payload={MESSAGE_KEY: "m"}, severity=GcpLogSeverity.WARNING.value, trace=trace_name) # type: ignore[call-arg]
+ entry: Any = StructEntry(payload={MESSAGE_KEY: "m"}, severity=GcpLogSeverity.WARNING.value, trace=entry_trace) # type: ignore[call-arg]
api_repr = cast("dict[str, Any]", entry.to_api_repr())
assert api_repr["severity"] == GcpLogSeverity.WARNING.value
- assert api_repr["trace"] == trace_name
+ assert api_repr["trace"] == entry_trace
+
+ def test_the_trace_the_span_and_the_sampled_bit_the_sink_passes_reach_the_entry_through_the_queue(self) -> None:
+ """The path the transport really takes: ``send`` enqueues the keyword arguments, and the batch builds the entry from them."""
+ entry_trace = trace_name(project="p", trace_id=0xAB)
+ worker: Any = _Worker(object())
+ record = logging.LogRecord(name="pipelex", level=logging.INFO, pathname="", lineno=0, msg="m", args=(), exc_info=None)
+ worker.enqueue(record, {MESSAGE_KEY: "m"}, labels={}, trace=entry_trace, span_id="00000000000000cd", trace_sampled=False)
+ queued = cast("dict[str, Any]", worker._queue.get_nowait()) # ruff: ignore[private-member-access]
+ # The library's batch is untyped; it is the one the worker builds its entries in.
+ batch: Any = Batch(logger=None, client=None)
+ batch.log(**queued)
+
+ (entry,) = cast("list[Any]", batch.entries)
+ api_repr = cast("dict[str, Any]", entry.to_api_repr())
+ assert api_repr["trace"] == entry_trace
+ assert api_repr["spanId"] == "00000000000000cd"
+ assert api_repr["traceSampled"] is False
def test_the_export_thread_still_carries_the_name_the_sinks_guard_rejects(self) -> None:
assert GCP_WORKER_THREAD_NAME == _WORKER_THREAD_NAME
diff --git a/tests/unit/pipelex/tools/test_log_gcp_json_payload_parity.py b/tests/unit/pipelex/tools/test_log_gcp_json_payload_parity.py
index 17a26401a..728b942a3 100644
--- a/tests/unit/pipelex/tools/test_log_gcp_json_payload_parity.py
+++ b/tests/unit/pipelex/tools/test_log_gcp_json_payload_parity.py
@@ -15,8 +15,10 @@ import sys
from typing import Any
from google.cloud.logging_v2.handlers.transports.base import Transport
+from opentelemetry.sdk.trace import TracerProvider
from typing_extensions import override
+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY, PIPELEX_TRACE_ID_KEY, pipelex_span_active
from pipelex.tools.log.gcp_log_sink import EXCEPTION_KEY, GcpLogHandler
from pipelex.tools.log.json_log_sink import JsonLogFormatter
from pipelex.tools.log.log_fields import attach_log_record_extra
@@ -103,3 +105,18 @@ class TestTheGcpAndJsonPayloadsAgree:
json_payload, gcp_payload = self._both_payloads(self._record(exc_info=(None, None, None)))
assert gcp_payload[EXCEPTION_KEY] == json_payload[EXCEPTION_KEY]
+
+ def test_the_held_pipelex_span_is_spelled_the_same_way_by_both_sinks(self) -> None:
+ with pipelex_span_active(span=TracerProvider().get_tracer(__name__).start_span("pipe")):
+ json_payload, gcp_payload = self._both_payloads(self._record())
+
+ for key in (PIPELEX_TRACE_ID_KEY, PIPELEX_SPAN_ID_KEY):
+ assert gcp_payload[key] == json_payload[key]
+
+ def test_a_field_named_like_a_pipelex_key_is_prefixed_under_both(self) -> None:
+ json_payload, gcp_payload = self._both_payloads(self._record(fields={PIPELEX_TRACE_ID_KEY: "mine", PIPELEX_SPAN_ID_KEY: "mine"}))
+
+ for payload in (json_payload, gcp_payload):
+ assert payload[f"field_{PIPELEX_TRACE_ID_KEY}"] == "mine"
+ assert payload[f"field_{PIPELEX_SPAN_ID_KEY}"] == "mine"
+ assert PIPELEX_TRACE_ID_KEY not in payload
diff --git a/tests/unit/pipelex/tools/test_log_gcp_sink.py b/tests/unit/pipelex/tools/test_log_gcp_sink.py
index 9fe5c7fa9..81baae174 100644
--- a/tests/unit/pipelex/tools/test_log_gcp_sink.py
+++ b/tests/unit/pipelex/tools/test_log_gcp_sink.py
@@ -18,9 +18,13 @@ from typing import TYPE_CHECKING, Any
import pytest
from google.cloud.logging_v2.handlers.transports.base import Transport
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.trace import NonRecordingSpan, SpanContext, TraceFlags
from typing_extensions import override
from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY, PIPELEX_TRACE_ID_KEY, pipelex_span_active
from pipelex.tools.log.gcp_log_sink import (
EXCEPTION_KEY,
GCP_WORKER_THREAD_NAME,
@@ -34,7 +38,6 @@ from pipelex.tools.log.gcp_log_sink import (
from pipelex.tools.log.log import Log
from pipelex.tools.log.log_config import LogConfig
from pipelex.tools.log.log_redaction import CYCLE_TEXT
-from pipelex.tools.misc.hash_utils import hash_md5_to_int
from pipelex.tools.misc.toml_utils import load_toml_from_path
if TYPE_CHECKING:
@@ -44,6 +47,10 @@ if TYPE_CHECKING:
PROJECT = "a-test-project"
+# The keyword arguments the sink passes the transport for OpenTelemetry's current span, and only when there is one.
+ENTRY_TRACE_KWARGS = frozenset({"trace", "span_id", "trace_sampled"})
+PIPELEX_TRACE_KEYS: tuple[str, ...] = (PIPELEX_TRACE_ID_KEY, PIPELEX_SPAN_ID_KEY)
+
class CapturedEntry:
"""One entry the sink handed the transport, as the transport received it."""
@@ -54,6 +61,9 @@ class CapturedEntry:
self.severity: Any = kwargs.get("severity")
self.labels: dict[str, str] = kwargs.get("labels") or {}
self.trace: str | None = kwargs.get("trace")
+ self.span_id: str | None = kwargs.get("span_id")
+ self.trace_sampled: bool | None = kwargs.get("trace_sampled")
+ self.kwarg_names = frozenset(kwargs)
class FakeTransport(Transport): # pyright: ignore[reportUntypedBaseClass]
@@ -211,23 +221,108 @@ class TestGcpLogSink:
assert "request_id" not in entry.payload
assert "pipe_run_id" not in entry.payload
- def test_the_trace_field_carries_the_runs_own_trace_id_project_qualified(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ def test_a_record_in_a_run_under_a_host_span_is_filed_under_the_host_span_and_names_the_pipelex_one(
+ self, gcp_log: tuple[Log, FakeTransport]
+ ) -> None:
+ fresh, transport = gcp_log
+ tracer = TracerProvider().get_tracer(__name__)
+ pipelex_span = tracer.start_span("pipe")
+ with tracer.start_as_current_span("host") as host_span, pipelex_span_active(span=pipelex_span), fresh.context(pipeline_run_id="plr-01"):
+ fresh.info("running")
+ host_context = host_span.get_span_context()
+ pipelex_context = pipelex_span.get_span_context()
+
+ (entry,) = _own_entries(transport)
+ assert entry.trace == f"projects/{PROJECT}/traces/{host_context.trace_id:032x}"
+ assert entry.span_id == f"{host_context.span_id:016x}"
+ assert entry.trace_sampled is True
+ assert entry.payload[PIPELEX_TRACE_ID_KEY] == f"{pipelex_context.trace_id:032x}"
+ assert entry.payload[PIPELEX_SPAN_ID_KEY] == f"{pipelex_context.span_id:016x}"
+ assert entry.labels == {"pipeline_run_id": "plr-01"}
+
+ def test_a_record_in_a_run_with_no_host_span_is_filed_under_no_trace_and_names_the_pipelex_span(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ """The run's id no longer stands in for a trace: the ``pipeline_run_id`` label is what selects the run's lines."""
+ fresh, transport = gcp_log
+ pipelex_span = TracerProvider().get_tracer(__name__).start_span("pipe")
+ with pipelex_span_active(span=pipelex_span), fresh.context(pipeline_run_id="plr-01"):
+ fresh.info("running")
+ pipelex_context = pipelex_span.get_span_context()
+
+ (entry,) = _own_entries(transport)
+ assert not ENTRY_TRACE_KWARGS & entry.kwarg_names
+ assert entry.payload[PIPELEX_TRACE_ID_KEY] == f"{pipelex_context.trace_id:032x}"
+ assert entry.payload[PIPELEX_SPAN_ID_KEY] == f"{pipelex_context.span_id:016x}"
+ assert entry.labels == {"pipeline_run_id": "plr-01"}
+
+ def test_a_record_carrying_a_run_id_with_no_span_at_all_is_filed_under_no_trace(self, gcp_log: tuple[Log, FakeTransport]) -> None:
fresh, transport = gcp_log
with fresh.context(pipeline_run_id="plr-01"):
fresh.info("running")
(entry,) = _own_entries(transport)
- assert entry.trace == f"projects/{PROJECT}/traces/{hash_md5_to_int('plr-01'):032x}"
+ assert not ENTRY_TRACE_KWARGS & entry.kwarg_names
+ assert not set(PIPELEX_TRACE_KEYS) & set(entry.payload)
assert entry.labels == {"pipeline_run_id": "plr-01"}
- def test_a_record_outside_any_run_carries_no_trace_and_no_labels(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ def test_a_record_outside_a_run_under_a_host_span_is_filed_under_it_with_no_pipelex_keys(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ fresh, transport = gcp_log
+ with TracerProvider().get_tracer(__name__).start_as_current_span("host") as host_span:
+ fresh.info("in the host")
+ host_context = host_span.get_span_context()
+
+ (entry,) = _own_entries(transport)
+ assert entry.trace == f"projects/{PROJECT}/traces/{host_context.trace_id:032x}"
+ assert entry.span_id == f"{host_context.span_id:016x}"
+ assert entry.trace_sampled is True
+ assert not set(PIPELEX_TRACE_KEYS) & set(entry.payload)
+
+ def test_an_unsampled_host_span_is_passed_as_unsampled_with_zero_padded_ids(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ fresh, transport = gcp_log
+ unsampled = NonRecordingSpan(SpanContext(trace_id=0xAB, span_id=0xCD, is_remote=False, trace_flags=TraceFlags(TraceFlags.DEFAULT)))
+ with trace.use_span(unsampled):
+ fresh.info("unsampled")
+
+ (entry,) = _own_entries(transport)
+ assert entry.trace == f"projects/{PROJECT}/traces/000000000000000000000000000000ab"
+ assert entry.span_id == "00000000000000cd"
+ assert entry.trace_sampled is False
+
+ def test_a_record_outside_any_run_and_any_span_carries_no_trace_and_no_labels(self, gcp_log: tuple[Log, FakeTransport]) -> None:
fresh, transport = gcp_log
fresh.info("unbound")
(entry,) = _own_entries(transport)
- assert entry.trace is None
+ assert not ENTRY_TRACE_KWARGS & entry.kwarg_names
+ assert not set(PIPELEX_TRACE_KEYS) & set(entry.payload)
assert entry.labels == {}
+ def test_the_pipelex_keys_follow_the_exception_and_precede_the_fields(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ fresh, transport = gcp_log
+ with pipelex_span_active(span=TracerProvider().get_tracer(__name__).start_span("pipe")):
+ try:
+ message = "kaput"
+ raise ValueError(message)
+ except ValueError:
+ fresh.error("failed", fields={"files": 7}, include_exception=True)
+
+ (entry,) = _own_entries(transport)
+ assert list(entry.payload) == [MESSAGE_KEY, LOGGER_KEY, EXCEPTION_KEY, PIPELEX_TRACE_ID_KEY, PIPELEX_SPAN_ID_KEY, "files"]
+
+ def test_the_pipelex_keys_are_reserved_with_or_without_a_pipelex_span(self, gcp_log: tuple[Log, FakeTransport]) -> None:
+ """A field named like a ``pipelex.*`` key is prefixed on every entry, so its wire name never depends on a span being held."""
+ fresh, transport = gcp_log
+ supplied = dict.fromkeys(PIPELEX_TRACE_KEYS, "supplied")
+ fresh.info("no span", fields=supplied)
+ with pipelex_span_active(span=TracerProvider().get_tracer(__name__).start_span("pipe")):
+ fresh.info("in a pipe", fields=supplied)
+
+ without_span, with_span = (entry.payload for entry in _own_entries(transport))
+ for key in PIPELEX_TRACE_KEYS:
+ assert key not in without_span
+ assert without_span[f"field_{key}"] == "supplied"
+ assert with_span[f"field_{key}"] == "supplied"
+ assert with_span[key] != "supplied"
+
def test_structured_content_rides_in_the_data_key_of_the_payload(self, gcp_log: tuple[Log, FakeTransport]) -> None:
fresh, transport = gcp_log
fresh.info({"key": "value", "nested": {"flag": True}}, title="Config")
diff --git a/tests/unit/pipelex/tools/test_log_gcp_sink_mapping.py b/tests/unit/pipelex/tools/test_log_gcp_sink_mapping.py
index 997c3c82e..cef2f7c4e 100644
--- a/tests/unit/pipelex/tools/test_log_gcp_sink_mapping.py
+++ b/tests/unit/pipelex/tools/test_log_gcp_sink_mapping.py
@@ -1,7 +1,7 @@
"""The two mappings the ``gcp`` sink owns, read without a handler in the way.
-The severity scale and the trace name are pure functions of a level and of a run id, so they are
-asserted directly rather than through an entry: what the sink does with them is
+The severity scale and the trace name are pure functions of a level and of a project and a trace id, so
+they are asserted directly rather than through an entry: what the sink does with them is
``test_log_gcp_sink.py``'s subject.
"""
@@ -9,8 +9,7 @@ from __future__ import annotations
import logging
-from pipelex.tools.log.gcp_log_sink import GcpLogSeverity, severity_for_level, trace_name_for_run
-from pipelex.tools.misc.hash_utils import hash_md5_to_int
+from pipelex.tools.log.gcp_log_sink import GcpLogSeverity, severity_for_level, trace_name
class TestGcpLogSinkMapping:
@@ -21,9 +20,5 @@ class TestGcpLogSinkMapping:
def test_a_level_above_critical_still_maps_to_critical(self) -> None:
assert severity_for_level(levelno=logging.CRITICAL + 10) is GcpLogSeverity.CRITICAL
- def test_the_trace_name_is_the_tracers_own_id_in_32_hex_digits(self) -> None:
- trace_name = trace_name_for_run(project="p", pipeline_run_id="plr-42")
- prefix, _, trace_id = trace_name.rpartition("/")
- assert prefix == "projects/p/traces"
- assert len(trace_id) == 32
- assert int(trace_id, 16) == hash_md5_to_int("plr-42")
+ def test_the_trace_name_is_the_trace_id_project_qualified_in_32_hex_digits(self) -> None:
+ assert trace_name(project="p", trace_id=0xAB) == "projects/p/traces/000000000000000000000000000000ab"
diff --git a/tests/unit/pipelex/tools/test_log_json_trace_context.py b/tests/unit/pipelex/tools/test_log_json_trace_context.py
index c7915bae4..b07e80fac 100644
--- a/tests/unit/pipelex/tools/test_log_json_trace_context.py
+++ b/tests/unit/pipelex/tools/test_log_json_trace_context.py
@@ -1,12 +1,13 @@
-"""The ``json`` sink writes the trace context of the span a record is logged in.
-
-That span is the Pipelex span active at the log call, and outside one OpenTelemetry's current span, the
-host's own. The keys are the ones OpenTelemetry specifies for trace context in a JSON log that is not
-OTLP, ``trace_id``, ``span_id`` and ``trace_flags``, lowercase hex at their full widths, and a record
-logged outside any valid span carries none of them. A boot line held until the sink arrives keeps the
-span it was logged in. The records go the whole way, from the facade through the module-named logger to the
-sink's handler, on a fresh ``Log`` so the installed sink is the one under test and the teardown leaves
-the root logger as it found it.
+"""The ``json`` sink writes the trace context of the two spans a record is logged in.
+
+The standard keys, the ones OpenTelemetry specifies for trace context in a JSON log that is not OTLP,
+``trace_id``, ``span_id`` and ``trace_flags``, name OpenTelemetry's current span at the log call, the
+host's own, and are absent when it names no trace. The ``pipelex.trace_id`` and ``pipelex.span_id``
+keys name the Pipelex span held at the log call, a pipe's or an LLM call's, and are absent outside one.
+The two sets are independent, all ids are lowercase hex at their full widths, and a boot line held until
+the sink arrives keeps both the spans it was logged in. The records go the whole way, from the facade
+through the module-named logger to the sink's handler, on a fresh ``Log`` so the installed sink is the
+one under test and the teardown leaves the root logger as it found it.
"""
from __future__ import annotations
@@ -23,7 +24,7 @@ from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import NonRecordingSpan, SpanContext, TraceFlags
from pipelex.system.configuration.config_loader import ConfigLoader
-from pipelex.system.telemetry.current_span import pipelex_span_active
+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY, PIPELEX_TRACE_ID_KEY, pipelex_span_active
from pipelex.tools.log.json_log_sink import (
LOGGER_KEY,
MESSAGE_KEY,
@@ -43,6 +44,7 @@ if TYPE_CHECKING:
from collections.abc import Iterator
TRACE_KEYS: tuple[str, ...] = (TRACE_ID_KEY, SPAN_ID_KEY, TRACE_FLAGS_KEY)
+PIPELEX_TRACE_KEYS: tuple[str, ...] = (PIPELEX_TRACE_ID_KEY, PIPELEX_SPAN_ID_KEY)
# A span context whose ids are small enough that only a full-width, zero-padded spelling gets them right,
# and which is not sampled, so its flags are written as they are rather than as the sampled default.
@@ -91,27 +93,50 @@ class TestJsonSinkTraceContext:
assert re.fullmatch(r"[0-9a-f]{32}", line[TRACE_ID_KEY])
assert re.fullmatch(r"[0-9a-f]{16}", line[SPAN_ID_KEY])
- def test_a_line_logged_in_a_pipelex_span_carries_it_rather_than_the_hosts(self, json_log: tuple[Log, io.StringIO]) -> None:
+ def test_a_line_in_a_run_under_a_host_span_names_the_host_span_and_the_pipelex_one_beside_it(self, json_log: tuple[Log, io.StringIO]) -> None:
fresh, buffer = json_log
tracer = TracerProvider().get_tracer(__name__)
pipelex_span = tracer.start_span("pipe")
- with tracer.start_as_current_span("host"):
- with pipelex_span_active(span=pipelex_span):
- fresh.info("in the pipe")
- fresh.info("back in the host")
- span_context = pipelex_span.get_span_context()
+ with tracer.start_as_current_span("host") as host_span, pipelex_span_active(span=pipelex_span):
+ fresh.info("in the pipe")
+ host_context = host_span.get_span_context()
+ pipelex_context = pipelex_span.get_span_context()
+
+ (line,) = _own_lines(buffer)
+ assert line[TRACE_ID_KEY] == f"{host_context.trace_id:032x}"
+ assert line[SPAN_ID_KEY] == f"{host_context.span_id:016x}"
+ assert line[TRACE_FLAGS_KEY] == "01"
+ assert line[PIPELEX_TRACE_ID_KEY] == f"{pipelex_context.trace_id:032x}"
+ assert line[PIPELEX_SPAN_ID_KEY] == f"{pipelex_context.span_id:016x}"
+ assert line[PIPELEX_TRACE_ID_KEY] != line[TRACE_ID_KEY]
- in_pipe, in_host = _own_lines(buffer)
- assert in_pipe[TRACE_ID_KEY] == f"{span_context.trace_id:032x}"
- assert in_pipe[SPAN_ID_KEY] == f"{span_context.span_id:016x}"
- assert in_host[TRACE_ID_KEY] != in_pipe[TRACE_ID_KEY]
+ def test_a_line_in_a_run_with_no_host_span_carries_only_the_pipelex_keys(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ pipelex_span = TracerProvider().get_tracer(__name__).start_span("pipe")
+ with pipelex_span_active(span=pipelex_span):
+ fresh.info("in the pipe")
+ pipelex_context = pipelex_span.get_span_context()
+
+ (line,) = _own_lines(buffer)
+ assert not set(TRACE_KEYS) & set(line)
+ assert line[PIPELEX_TRACE_ID_KEY] == f"{pipelex_context.trace_id:032x}"
+ assert line[PIPELEX_SPAN_ID_KEY] == f"{pipelex_context.span_id:016x}"
+
+ def test_a_line_outside_a_run_under_a_host_span_carries_only_the_standard_keys(self, json_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = json_log
+ with TracerProvider().get_tracer(__name__).start_as_current_span("host") as host_span:
+ fresh.info("in the host")
+
+ (line,) = _own_lines(buffer)
+ assert line[SPAN_ID_KEY] == f"{host_span.get_span_context().span_id:016x}"
+ assert not set(PIPELEX_TRACE_KEYS) & set(line)
def test_a_line_logged_outside_any_span_carries_none_of_the_keys(self, json_log: tuple[Log, io.StringIO]) -> None:
fresh, buffer = json_log
fresh.info("outside")
(line,) = _own_lines(buffer)
- assert not set(TRACE_KEYS) & set(line)
+ assert not set(TRACE_KEYS + PIPELEX_TRACE_KEYS) & set(line)
def test_an_invalid_current_span_writes_nothing(self, json_log: tuple[Log, io.StringIO]) -> None:
"""A context can hold a span that names no trace; a line under it is a line outside any span."""
@@ -147,30 +172,45 @@ class TestJsonSinkTraceContext:
assert in_inner[TRACE_ID_KEY] == in_outer[TRACE_ID_KEY]
assert not set(TRACE_KEYS) & set(after)
- def test_the_trace_keys_follow_the_fixed_keys_and_precede_the_fields(self, json_log: tuple[Log, io.StringIO]) -> None:
+ def test_the_trace_keys_follow_the_fixed_keys_the_pipelex_ones_follow_them_and_the_fields_come_last(
+ self, json_log: tuple[Log, io.StringIO]
+ ) -> None:
fresh, buffer = json_log
- with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)):
+ pipelex_span = TracerProvider().get_tracer(__name__).start_span("pipe")
+ with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)), pipelex_span_active(span=pipelex_span):
fresh.info("ordered", fields={"files": 7})
(line,) = _own_lines(buffer)
- assert list(line) == [TIME_KEY, SEVERITY_KEY, LOGGER_KEY, MESSAGE_KEY, TRACE_ID_KEY, SPAN_ID_KEY, TRACE_FLAGS_KEY, "files"]
+ assert list(line) == [
+ TIME_KEY,
+ SEVERITY_KEY,
+ LOGGER_KEY,
+ MESSAGE_KEY,
+ TRACE_ID_KEY,
+ SPAN_ID_KEY,
+ TRACE_FLAGS_KEY,
+ PIPELEX_TRACE_ID_KEY,
+ PIPELEX_SPAN_ID_KEY,
+ "files",
+ ]
def test_the_trace_keys_are_reserved_with_or_without_a_span(self, json_log: tuple[Log, io.StringIO]) -> None:
- """A field named like a trace key is prefixed on every line, so its wire name never depends on a span being current."""
+ """A field named like a trace key is prefixed on every line, so its wire name never depends on a span being current or held."""
fresh, buffer = json_log
- supplied = dict.fromkeys(TRACE_KEYS, "supplied")
+ supplied = dict.fromkeys(TRACE_KEYS + PIPELEX_TRACE_KEYS, "supplied")
fresh.info("no span", fields=supplied)
- with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)):
- fresh.info("in a span", fields=supplied)
+ with trace.use_span(NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)), pipelex_span_active(span=NonRecordingSpan(_UNSAMPLED_SPAN_CONTEXT)):
+ fresh.info("in both spans", fields=supplied)
without_span, with_span = _own_lines(buffer)
- for key in TRACE_KEYS:
+ for key in TRACE_KEYS + PIPELEX_TRACE_KEYS:
assert key not in without_span
assert without_span[f"{COLLIDING_FIELD_PREFIX}{key}"] == "supplied"
assert with_span[f"{COLLIDING_FIELD_PREFIX}{key}"] == "supplied"
assert with_span[TRACE_ID_KEY] == "000000000000000000000000000000ab"
+ assert with_span[PIPELEX_TRACE_ID_KEY] == "000000000000000000000000000000ab"
- def test_a_boot_line_held_until_the_sink_arrives_keeps_the_span_it_was_logged_in(self, caplog: pytest.LogCaptureFixture) -> None:
+ def test_a_boot_line_held_until_the_sink_arrives_keeps_the_spans_it_was_logged_in(self, caplog: pytest.LogCaptureFixture) -> None:
"""The holding handler replays each record in the context it was emitted in, not in the one the install runs under."""
caplog.set_level(logging.INFO, logger=__name__)
buffer = io.StringIO()
@@ -191,5 +231,7 @@ class TestJsonSinkTraceContext:
inside, inside_pipelex, outside = _own_lines(buffer)
assert inside[SPAN_ID_KEY] == f"{boot_step.get_span_context().span_id:016x}"
- assert inside_pipelex[SPAN_ID_KEY] == f"{pipelex_span.get_span_context().span_id:016x}"
- assert not set(TRACE_KEYS) & set(outside)
+ assert not set(PIPELEX_TRACE_KEYS) & set(inside)
+ assert inside_pipelex[PIPELEX_SPAN_ID_KEY] == f"{pipelex_span.get_span_context().span_id:016x}"
+ assert not set(TRACE_KEYS) & set(inside_pipelex)
+ assert not set(TRACE_KEYS + PIPELEX_TRACE_KEYS) & set(outside)
diff --git a/tests/unit/pipelex/tools/test_log_otlp_trace_context.py b/tests/unit/pipelex/tools/test_log_otlp_trace_context.py
index c11217222..990afcd8d 100644
--- a/tests/unit/pipelex/tools/test_log_otlp_trace_context.py
+++ b/tests/unit/pipelex/tools/test_log_otlp_trace_context.py
@@ -1,10 +1,11 @@
-"""The ``otlp`` sink files each record under the span it was logged in.
-
-That span is the Pipelex span active at the log call, and outside one OpenTelemetry's current span, the
-host's own. The sink hands the SDK a context holding that span, from which the SDK takes the record's
-trace id, span id and flags, so a collector files the record under it; a record logged outside any span
-carries none. A boot line held until the sink arrives keeps the span it was logged in. The sink
-is built on a synchronous processor so each record is exported before the assertion reads it.
+"""The ``otlp`` sink files each record under OpenTelemetry's current span, and names the Pipelex span beside it.
+
+The sink hands the SDK the current context, from which the SDK takes the record's trace id, span id and
+flags, so a collector files the record under the host's own span, or under none when it names no trace.
+The Pipelex span held at the log call, a pipe's or an LLM call's, rides as the ``pipelex.trace_id`` and
+``pipelex.span_id`` attributes, in hex, and is absent outside one. A boot line held until the sink
+arrives keeps both the spans it was logged in. The sink is built on a synchronous processor so each
+record is exported before the assertion reads it.
"""
from __future__ import annotations
@@ -15,11 +16,13 @@ from typing import TYPE_CHECKING
import pytest
from opentelemetry.sdk._logs.export import InMemoryLogExporter, SimpleLogRecordProcessor
from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.trace import INVALID_SPAN_ID, INVALID_TRACE_ID
from pipelex.system.configuration.config_loader import ConfigLoader
-from pipelex.system.telemetry.current_span import pipelex_span_active
+from pipelex.system.telemetry.current_span import PIPELEX_SPAN_ID_KEY, PIPELEX_TRACE_ID_KEY, pipelex_span_active
from pipelex.tools.log.log import Log
from pipelex.tools.log.log_config import LogConfig
+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX
from pipelex.tools.log.otlp_log_sink import OtlpLogSink
from pipelex.tools.misc.toml_utils import load_toml_from_path
@@ -28,6 +31,8 @@ if TYPE_CHECKING:
from opentelemetry.sdk._logs import LogRecord
+PIPELEX_TRACE_KEYS: tuple[str, ...] = (PIPELEX_TRACE_ID_KEY, PIPELEX_SPAN_ID_KEY)
+
def _package_log_config() -> LogConfig:
config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")
@@ -39,6 +44,10 @@ def _own_records(exporter: InMemoryLogExporter) -> list[LogRecord]:
return [log_data.log_record for log_data in exporter.get_finished_logs() if log_data.instrumentation_scope.name == __name__]
+def _attributes(record: LogRecord) -> dict[str, object]:
+ return dict(record.attributes or {})
+
+
class TestOtlpSinkTraceContext:
@pytest.fixture
def otlp_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, InMemoryLogExporter]]:
@@ -53,38 +62,81 @@ class TestOtlpSinkTraceContext:
finally:
fresh.reset()
- def test_a_record_logged_inside_a_span_is_filed_under_it(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:
+ def test_a_record_outside_a_run_under_a_host_span_is_filed_under_it_with_no_pipelex_attributes(
+ self, otlp_log: tuple[Log, InMemoryLogExporter]
+ ) -> None:
fresh, exporter = otlp_log
tracer = TracerProvider().get_tracer(__name__)
with tracer.start_as_current_span("work") as span:
fresh.info("inside")
- fresh.info("outside")
span_context = span.get_span_context()
- inside, outside = _own_records(exporter)
+ (inside,) = _own_records(exporter)
assert inside.trace_id == span_context.trace_id
assert inside.span_id == span_context.span_id
assert inside.trace_flags == span_context.trace_flags
- assert outside.trace_id == 0
- assert outside.span_id == 0
+ assert not set(PIPELEX_TRACE_KEYS) & set(_attributes(inside))
- def test_a_record_logged_in_a_pipelex_span_is_filed_under_it_rather_than_the_hosts(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:
+ def test_a_record_in_a_run_under_a_host_span_is_filed_under_the_host_span_and_names_the_pipelex_one(
+ self, otlp_log: tuple[Log, InMemoryLogExporter]
+ ) -> None:
fresh, exporter = otlp_log
tracer = TracerProvider().get_tracer(__name__)
pipelex_span = tracer.start_span("pipe")
- with tracer.start_as_current_span("host") as host_span:
- with pipelex_span_active(span=pipelex_span):
- fresh.info("in the pipe")
- fresh.info("back in the host")
- span_context = pipelex_span.get_span_context()
+ with tracer.start_as_current_span("host") as host_span, pipelex_span_active(span=pipelex_span):
+ fresh.info("in the pipe")
+ host_context = host_span.get_span_context()
+ pipelex_context = pipelex_span.get_span_context()
+
+ (in_pipe,) = _own_records(exporter)
+ assert in_pipe.trace_id == host_context.trace_id
+ assert in_pipe.span_id == host_context.span_id
+ assert in_pipe.trace_flags == host_context.trace_flags
+ attributes = _attributes(in_pipe)
+ assert attributes[PIPELEX_TRACE_ID_KEY] == f"{pipelex_context.trace_id:032x}"
+ assert attributes[PIPELEX_SPAN_ID_KEY] == f"{pipelex_context.span_id:016x}"
+
+ def test_a_record_in_a_run_with_no_host_span_is_filed_under_none_and_names_the_pipelex_span(
+ self, otlp_log: tuple[Log, InMemoryLogExporter]
+ ) -> None:
+ fresh, exporter = otlp_log
+ pipelex_span = TracerProvider().get_tracer(__name__).start_span("pipe")
+ with pipelex_span_active(span=pipelex_span):
+ fresh.info("in the pipe")
+ pipelex_context = pipelex_span.get_span_context()
+
+ (in_pipe,) = _own_records(exporter)
+ assert in_pipe.trace_id == INVALID_TRACE_ID
+ assert in_pipe.span_id == INVALID_SPAN_ID
+ attributes = _attributes(in_pipe)
+ assert attributes[PIPELEX_TRACE_ID_KEY] == f"{pipelex_context.trace_id:032x}"
+ assert attributes[PIPELEX_SPAN_ID_KEY] == f"{pipelex_context.span_id:016x}"
+
+ def test_a_record_outside_any_span_carries_neither(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:
+ fresh, exporter = otlp_log
+ fresh.info("outside")
- in_pipe, in_host = _own_records(exporter)
- assert in_pipe.trace_id == span_context.trace_id
- assert in_pipe.span_id == span_context.span_id
- assert in_pipe.trace_flags == span_context.trace_flags
- assert in_host.span_id == host_span.get_span_context().span_id
+ (outside,) = _own_records(exporter)
+ assert outside.trace_id == INVALID_TRACE_ID
+ assert outside.span_id == INVALID_SPAN_ID
+ assert not set(PIPELEX_TRACE_KEYS) & set(_attributes(outside))
- def test_a_boot_line_held_until_the_sink_arrives_keeps_the_span_it_was_logged_in(self, caplog: pytest.LogCaptureFixture) -> None:
+ def test_the_pipelex_keys_are_reserved_with_or_without_a_pipelex_span(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:
+ """A field named like a ``pipelex.*`` key is prefixed on every record, so its wire name never depends on a span being held."""
+ fresh, exporter = otlp_log
+ supplied = dict.fromkeys(PIPELEX_TRACE_KEYS, "supplied")
+ fresh.info("no span", fields=supplied)
+ with pipelex_span_active(span=TracerProvider().get_tracer(__name__).start_span("pipe")):
+ fresh.info("in a pipe", fields=supplied)
+
+ without_span, with_span = (_attributes(record) for record in _own_records(exporter))
+ for key in PIPELEX_TRACE_KEYS:
+ assert key not in without_span
+ assert without_span[f"{COLLIDING_FIELD_PREFIX}{key}"] == "supplied"
+ assert with_span[f"{COLLIDING_FIELD_PREFIX}{key}"] == "supplied"
+ assert with_span[key] != "supplied"
+
+ def test_a_boot_line_held_until_the_sink_arrives_keeps_the_spans_it_was_logged_in(self, caplog: pytest.LogCaptureFixture) -> None:
"""The holding handler replays each record in the context it was emitted in, not in the one the install runs under."""
caplog.set_level(logging.INFO, logger=__name__)
exporter = InMemoryLogExporter()
@@ -105,5 +157,8 @@ class TestOtlpSinkTraceContext:
inside, inside_pipelex, outside = _own_records(exporter)
assert inside.span_id == boot_step.get_span_context().span_id
- assert inside_pipelex.span_id == pipelex_span.get_span_context().span_id
- assert outside.span_id == 0
+ assert not set(PIPELEX_TRACE_KEYS) & set(_attributes(inside))
+ assert inside_pipelex.span_id == INVALID_SPAN_ID
+ assert _attributes(inside_pipelex)[PIPELEX_SPAN_ID_KEY] == f"{pipelex_span.get_span_context().span_id:016x}"
+ assert outside.span_id == INVALID_SPAN_ID
+ assert not set(PIPELEX_TRACE_KEYS) & set(_attributes(outside))
diff --git a/wip/log-trace-fields-current/design.md b/wip/log-trace-fields-current/design.md
new file mode 100644
index 000000000..e6ca940d9
--- /dev/null
+++ b/wip/log-trace-fields-current/design.md
@@ -0,0 +1,69 @@
+---
+status: landed
+item: L-260927-6b356b
+---
+
+# A log line's standard trace fields name OpenTelemetry's current span
+
+This is the log-line half of the ruling Louis took on L-260925-247280 on 2026-09-27, option L2: the standard trace fields of a log line always name OpenTelemetry's current span in the process, and the Pipelex span active in the task travels beside them as `pipelex.trace_id` and `pipelex.span_id`. The span half of the same ruling, the `join_host_trace` opt-in that makes Pipelex's spans current, is L-260927-f30a84 and is out of scope here.
+
+## What the code does today
+
+The item's claims were checked against `dev` at `4520b42a2` and hold.
+
+- `pipelex/system/telemetry/current_span.py`: `span_context_for_logs()` returns the held Pipelex span's context whenever one is held, and falls back to OpenTelemetry's current span only outside a Pipelex span. `otel_context_for_logs()` does the same for the OpenTelemetry API that takes a context, by building a context with the Pipelex span set in it.
+- `pipelex/tools/log/json_log_sink.py`: `_trace_context()` writes `trace_id`, `span_id` and `trace_flags` from `span_context_for_logs()`.
+- `pipelex/tools/log/otlp_log_sink.py`: `OtlpLogHandler.emit` files each record under `otel_context_for_logs()`, so the OTLP record's own trace context names the Pipelex span.
+- `pipelex/tools/log/gcp_log_sink.py`: the entry's `trace` field is `trace_name_for_run(...)`, a trace id derived from the record's `pipeline_run_id` by the tracer's own hash, whether or not any tracer exists, and the entry carries no `spanId` and no `traceSampled`.
+- Every line already carries `pipeline_run_id` (`pipelex/tools/log/log_context.py`), and the `gcp` sink carries it as a label, so a run stays joinable without the standard fields.
+- The behaviour shipped in v0.66.0 (the CHANGELOG entry "Log lines carry the trace context of the span they were logged in"), so this is a change to a released wire shape.
+- Neither `pipelex-server` nor `pipelex-api` reads these helpers, registers a global tracer provider, or instruments FastAPI; nothing in the hosted plane consumes `trace_id` from a log line. Two docs describe the current behaviour: `pipelex-api/docs/logging.md:28` says the trace fields are present "for a line the runtime emits from inside a traced run", which this change makes false, and `pipelex-server/docs/logging.md:14`.
+
+## The rule
+
+For every sink that writes trace context:
+
+- **The standard fields** (`trace_id`, `span_id`, `trace_flags`, or the sink's own equivalent) name OpenTelemetry's current span in the process when it is valid, and are absent otherwise. Pipelex only reads that span; it never sets it.
+- **The Pipelex fields** `pipelex.trace_id` and `pipelex.span_id` name the Pipelex span held in the task by `pipelex_span_active`, when one is held, and are absent otherwise.
+
+The two are independent: a line can carry both, either, or neither. Inside a run with a host span it carries both, naming different traces; inside a run with no host span only the Pipelex fields; outside a run under a host span only the standard fields; outside everything, neither. Once L-260927-f30a84 makes the Pipelex span current under its opt-in, the two sets coincide inside a run, and nothing on the log side has to change for that.
+
+## Wire shape per sink
+
+| Sink | Standard fields, from OpenTelemetry's current span | Pipelex fields, from the held Pipelex span |
+| --- | --- | --- |
+| `json` | `trace_id` (32 hex), `span_id` (16 hex), `trace_flags` (2 hex), as today | `pipelex.trace_id` (32 hex), `pipelex.span_id` (16 hex), right after the standard ones and before the carried fields |
+| `otlp` | The log record's own trace context, from `get_current()` | Attributes `pipelex.trace_id` and `pipelex.span_id`, hex |
+| `gcp` | The entry's `trace` (`projects/<project>/traces/<32 hex>`), `span_id` (16 hex) and `trace_sampled` (the sampled bit) | Payload keys `pipelex.trace_id` and `pipelex.span_id`, hex |
+
+The `console` sink writes no trace context and is untouched.
+
+## Decisions
+
+- **Flat dotted keys, not a nested `pipelex` object.** `pipelex.trace_id` is the OpenTelemetry attribute-namespace spelling, which the `otlp` sink needs anyway, and the span attributes Pipelex already exports use the same prefix (`pipelex.run.user_id`). One spelling across the three sinks keeps a field's wire name independent of the sink, which is the rule the sinks already follow for every other key. A backend that flattens nested JSON into dotted names, CloudWatch Logs Insights among them, addresses the flat key the way it would a nested one.
+- **The Pipelex fields are written whenever a Pipelex span is held, even when they equal the standard ones.** A consumer that joins on `pipelex.trace_id` must find it on every line of a traced run, whatever the operator's configuration. Omitting it when it duplicates the standard fields would save a few bytes and make the key conditional on a setting the reader cannot see.
+- **No `pipelex.trace_flags`.** The ruling names the trace and span ids only, and the Pipelex span's flags are the hard-coded `SAMPLED` of its synthetic parent (`pipe_abstract.py`, `llm_worker_abstract.py`), which says nothing a reader can use. L-260927-f30a84 replaces that flag with the host's decision; if the flag ever carries information on its own, adding the key is additive.
+- **The `gcp` sink stops deriving a trace from `pipeline_run_id`.** `trace_name_for_run` goes; the `trace` field is built from the current span's trace id, and `span_id` and `trace_sampled` are passed beside it, which the client library's `BackgroundThreadTransport.send` forwards as keyword arguments to the entry (checked against the installed `google-cloud-logging`: `send(record, message, **kwargs)` enqueues the kwargs, and `Batch.log(message=None, **kw)` builds the entry from them). The derivation was also wrong in a way the item does not mention: it filed a line under a trace even when no tracer existed, and under the opt-in the Pipelex trace id will no longer be the hash of `pipeline_run_id` at all. The Pipelex fields in the payload come from the held span, never from a derivation, for the same reason.
+- **The new keys are reserved like the others.** `pipelex.trace_id` and `pipelex.span_id` join the `json` sink's `FIXED_KEYS`, which the `gcp` sink reuses as its reserved payload keys, and the `otlp` sink's `RESERVED_ATTRIBUTE_KEYS`, so a carried field of the same name is written under the `field_` prefix on every line, with or without a span.
+- **The readers are renamed, not re-pointed.** `span_context_for_logs()` keeps a name while changing its meaning, and every caller of it, the sinks and the tests that assert which Pipelex span a pipe, an LLM call or the Bedrock thread holds, needs to decide which of the two spans it means. `current_span.py` exposes instead:
+ - `current_span_context_for_logs() -> SpanContext | None`: OpenTelemetry's current span when valid.
+ - `pipelex_span_context_for_logs() -> SpanContext | None`: the held Pipelex span's context.
+ - `pipelex_trace_fields_for_logs() -> dict[str, str]`: the two Pipelex fields, hex-encoded, or an empty dict, which all three sinks write.
+ - `PIPELEX_TRACE_ID_KEY` and `PIPELEX_SPAN_ID_KEY`, the one place the two names are spelled.
+
+ `otel_context_for_logs()` is deleted: the `otlp` sink passes `get_current()`. The old names disappear rather than stay as aliases, per the workspace's no-backward-compatibility rule; `docs/under-the-hood/log-sink-plugins.md` documents them for sink-plugin authors, so the CHANGELOG entry names the rename.
+- **The held span keeps being held.** `pipelex_span_active` and its call sites in `pipe_abstract.py` and `llm_worker_abstract.py` are unchanged: the private context variable is now read only for the `pipelex.*` fields, and it is still what L-260927-f30a84 builds on.
+
+## What it costs
+
+- **An operator who exports Pipelex's spans to their own backend loses the standard-field join inside a run until the opt-in exists.** With an OTLP, Langfuse or custom-PostHog destination and the `json` or `otlp` log sink, a line inside a run today carries the Pipelex span in `trace_id`, so the backend links it to the Pipelex span it received. After this change that line carries the host's span, or none, in `trace_id`, and the Pipelex span only under `pipelex.trace_id`, which a backend's trace view does not follow on its own. The ruling accepted this: the opt-in L-260927-f30a84 is where the two coincide. If both are close to done, releasing them together removes the window; if not, the CHANGELOG entry tells such an operator to join on `pipelex.trace_id` until the opt-in ships.
+- **The hosted plane's lines inside a run lose `trace_id` until it opts in.** The runner and the worker register no host tracer, so after this change their JSON lines carry only `pipelex.trace_id` and `pipelex.span_id` inside a run. Nothing there consumes `trace_id` today; L-260927-f30932 turns the opt-in on.
+- **A `gcp` line inside a run with no host span is no longer grouped under a trace in Cloud Logging.** The derived trace grouped a run's lines in the Logs Explorer's trace view even with no spans exported anywhere. The `pipeline_run_id` label still selects the run's lines, which is the join the ruling relies on.
+- **A released wire shape and a documented plugin API change.** The field semantics shipped in v0.66.0, and the reader names are in the sink-plugin page. Both go in the CHANGELOG as breaking.
+
+## Out of scope
+
+- Making Pipelex's spans current, following the host's sampler, and seeding the run's trace from the host span: L-260927-f30a84.
+- Turning the opt-in on in the hosted plane: L-260927-f30932.
+- Emitting Pipelex's spans through the host's global provider: L-260927-0355ff.
+- The `pipelex-api` and `pipelex-server` logging docs, which describe the fields as they are today: L-260927-a12b10 and L-260927-4e4777, owned by those repos and blocked by this item.
diff --git a/wip/log-trace-fields-current/plan.md b/wip/log-trace-fields-current/plan.md
new file mode 100644
index 000000000..7ed72b813
--- /dev/null
+++ b/wip/log-trace-fields-current/plan.md
@@ -0,0 +1,34 @@
+---
+status: landed
+item: L-260927-6b356b
+---
+
+# Plan: log trace fields follow OpenTelemetry's current span
+
+The design is [`design.md`](design.md). The work is one pull request on `feature/Log-trace-fields-current`, in the order below, tests first at each step. Its size does not call for checkpoints between steps; the one review gate is before the PR opens.
+
+## Steps
+
+1. **The readers** (`pipelex/system/telemetry/current_span.py`). Rewrite `tests/unit/pipelex/system/telemetry/test_current_span.py` first: the current-span reader returns the host's valid span inside and outside a held Pipelex span and `None` with no valid current span; the Pipelex reader returns the innermost held span and `None` outside one; the fields helper renders both ids zero-padded, or nothing; the existing guarantees keep their tests (an exception crossing the block restores the previous span, `None` and a span naming no trace hold nothing, OpenTelemetry's current span is never touched). Then replace `span_context_for_logs` and `otel_context_for_logs` with `current_span_context_for_logs`, `pipelex_span_context_for_logs`, `pipelex_trace_fields_for_logs` and the two key constants, and rewrite the module docstring around the two-span rule.
+2. **The `json` sink.** In `tests/unit/pipelex/tools/test_log_json_trace_context.py`, replace the "Pipelex span rather than the host's" test with the four cases of the rule (inside a run with a host span, inside a run without one, outside a run under a host span, outside everything), add the key order (standard trace keys, then the Pipelex ones, then the fields), the reservation of the two new keys with and without a span, and keep the held-boot-line test, now asserting both sets. Then `_trace_context()` reads the current span, the payload adds `pipelex_trace_fields_for_logs()`, and `FIXED_KEYS` gains the two keys.
+3. **The `otlp` sink.** In `tests/unit/pipelex/tools/test_log_otlp_trace_context.py`, the same four cases against the in-memory exporter: the record's trace context names the host span or nothing, and the `pipelex.*` attributes name the held span or are absent; plus the reservation. Then `emit` passes `get_current()`, `_attributes` writes the Pipelex fields right after the source location, and `RESERVED_ATTRIBUTE_KEYS` gains the two keys.
+4. **The `gcp` sink.** In `tests/unit/pipelex/tools/test_log_gcp_sink.py`, replace the "run's own trace id" test with the four cases on the fake transport: `trace`, `span_id` and `trace_sampled` from the current span or absent, the Pipelex fields in the payload or absent, and a record carrying `pipeline_run_id` with no span getting no `trace`. Extend `test_log_gcp_client_library_contract.py` so the installed library renders the `trace`, `span_id` and `trace_sampled` the sink passes into the entry, and drop the `trace_name_for_run` test from `test_log_gcp_sink_mapping.py`. Then replace `trace_name_for_run` with a builder taking the project and a trace id, pass the three kwargs only when a current span exists, write the Pipelex fields into the payload, and drop the `hash_md5_to_int` import. Check `test_log_gcp_json_payload_parity.py` still holds with the new reserved keys.
+5. **The span-holding tests.** `test_live_run_pipe_current_span.py`, `test_llm_worker_current_span.py` and `test_bedrock_boto3_current_span.py` assert which Pipelex span the runtime holds; move them to `pipelex_span_context_for_logs()`, and in the Bedrock test assert the thread sees the host span through `current_span_context_for_logs()` as well.
+6. **The docs.**
+ - `docs/tools/logging.md`, "The trace context": replace the three-step precedence with the two independent sets and the four cases, keep the paragraph on which spans the runtime holds, and say the `pipelex.*` fields are the ones that name the pipe or the LLM call.
+ - `docs/setup/telemetry.md`, "Pipelex's spans in your process": the third bullet says the standard fields name your current span and the Pipelex span rides under `pipelex.*`, and what an operator exporting Pipelex's spans joins on until the opt-in exists.
+ - `docs/configuration/config-practical/logging-config.md`: the `json` line description and its example line, and the `gcp` entry's `trace`, `span_id` and `trace_sampled` and payload keys.
+ - `docs/under-the-hood/log-sink-plugins.md`: the reader functions a sink plugin calls, the reserved keys, and the `json` row of the sink table.
+ - The module docstrings of the three sinks.
+7. **The CHANGELOG**, under `[Unreleased]`, one breaking entry: the standard trace fields name OpenTelemetry's current span, the Pipelex span rides as `pipelex.trace_id` and `pipelex.span_id` in the `json`, `otlp` and `gcp` sinks, the `gcp` sink's `trace` is no longer derived from `pipeline_run_id` and now carries `span_id` and `trace_sampled`, and `span_context_for_logs` and `otel_context_for_logs` are replaced by the new readers.
+8. **The gates.** `make agent-check`, then `make agent-test`, then `/rev` at the depth `ledger review-profile` derives, then the PR `feature/Log-trace-fields-current · L-260927-6b356b` with `Closes L-260927-6b356b`. Flip this plan and the design to `landed` inside the PR.
+
+## Follow-ups filed with this plan
+
+- `pipelex-api/docs/logging.md:28` describes the trace fields as present on any line inside a traced run, which stops being true: L-260927-a12b10, owned by `pipelex-api`.
+- `pipelex-server/docs/logging.md:14` describes the trace fields without the Pipelex ones: L-260927-4e4777, owned by `pipelex-server` and related to L-260927-f30932, which changes the same paragraph again when the hosted plane opts in.
+
+## Decided while building
+
+- **Landed independently of L-260927-f30a84**, as the plan recommended: the change is small and the opt-in is not. Whether they ship in one release is the release's call; the CHANGELOG entry tells an operator exporting Pipelex's spans to join on `pipelex.trace_id` meanwhile.
+- **Beyond the steps above**, the `json`/`gcp` payload-parity test gained the two `pipelex.*` keys (same spelling under both sinks, prefixed under both when a field collides), the client-library contract test drives `trace`, `span_id` and `trace_sampled` through the real worker queue and `Batch.log` rather than a hand-built entry, and the comments at the `pipelex_span_active` call sites (`pipe_abstract.py`, `llm_worker_abstract.py`, `bedrock_client_boto3.py`) say which fields the held span now reaches.