Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Log-fields-context · L-260916-a1dbd5 — fields, context, no stack walk (#1205)

Louis Choquelee9d8132a18f
diff --git a/.drift/acks/config-docs.toml b/.drift/acks/config-docs.toml
index c50fea85a..899411a4f 100644
--- a/.drift/acks/config-docs.toml
+++ b/.drift/acks/config-docs.toml
@@ -1,8 +1,8 @@
contract = "config-docs"
-digest = "sha256:253467d8e548ab963307b4efa6209306a83860076cc8b6116b20e7f74cd5db4e"
+digest = "sha256:7855e5a0a09388da54a6365f26e2f0febef02310b309b613a562a969748d7f2e"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-09-15T23:02:03Z"
-rationale = "The two rendering keys are deleted from DataInclusionConfig and from every shipped pipelex.toml, and the deletion is accounted for by ledger entry pipelex-config@4 with its regenerated goldens. Reviewed docs/configuration: no page there names the data_inclusion keys, and the logging page's pretty-print section is unaffected. The tracing page under docs/under-the-hood, which did name them, is updated in this same change."
+reviewed_at = "2026-09-16T14:58:00Z"
+rationale = "Docstring-only wording fix in ConfigLoader.take_stale_configuration_warning: the review pass found the earlier text claimed a pre-configure warning misses the console, while it lands on stderr unformatted through logging.lastResort, and the docstring now says so. No config model, key, default or validator moved, so the migration ledgers owe nothing and docs/configuration stays as the previous ack left it."
[trigger_files]
"pipelex/cogt/config_cogt.py" = "blob:02f71cb4a6b01aebda59c2e31f262ef9c865e17b"
@@ -10,7 +10,7 @@ rationale = "The two rendering keys are deleted from DataInclusionConfig and fro
"pipelex/pipelex.toml" = "blob:a4197193c4e9df0f09ef795149a505c0a149fdda"
"pipelex/system/configuration/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"
"pipelex/system/configuration/config_check.py" = "blob:88e15a92b186088a5ea1c7c4ebcedac830707d3a"
-"pipelex/system/configuration/config_loader.py" = "blob:cbd3c9a54d5f9410c65b08702d19560a06f83086"
+"pipelex/system/configuration/config_loader.py" = "blob:ac384ce993933656682999c26afffb4c9f9b1975"
"pipelex/system/configuration/config_model.py" = "blob:83b29dc6e93393c9b98c4353636caf0135c80339"
"pipelex/system/configuration/config_root.py" = "blob:c7d43851270912d17413cfc6a511247d3c7cc160"
"pipelex/system/configuration/config_surface.py" = "blob:96f685656a3f4355e8a073e8bf6184e020fea8f5"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 08000f42b..a322565af 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,12 +2,21 @@
## [Unreleased]
+### Added
+
+- **Named fields and a run-scoped context on the log calls**: the seven `log.*` methods take a keyword-only `fields={...}`, carried as attributes of the stdlib `LogRecord` and never rendered into the message, and `with log.context(request_id=..., pipeline_run_id=..., pipe_run_id=...)` stamps the run's identifiers onto every record emitted in scope, nested bindings merging and an absent identifier staying absent rather than reading `None`. `PipeRun.run` binds the request and pipeline identifiers from the job's metadata for a direct-mode run, and every pipe binds its own `pipe_run_id` around the whole of its run, its announcement and its failure included. A `dict` or `list` content is carried as the record's `data` attribute beside its console rendering, a JSON-ready snapshot of the call rather than the caller's live object, and an entry named like an attribute the record already owns, a record factory's included, is carried under a `field_` prefix instead of raising, whatever the order the names arrive in. See [Logging](tools/logging.md).
+
### Changed
+- **Loggers are named by module (Breaking)**: a record from `pipelex/pipe_operators/pipe_llm.py` is emitted on the `pipelex.pipe_operators.pipe_llm` logger instead of `pipelex`, so a handler or filter keyed on the bare package name must key on the prefix, and a `package_log_levels` key now works at module depth too, `pipelex-pipe_operators-pipe_llm = "DEBUG"` for instance. The stack walk that named the logger on every line is gone, replaced by one frame lookup, and the caller-info templates that name the module now render instead of raising.
- **The pitch names who each way of running a method is for**: the README, the documentation's front page and **Quick Start**, and the site description now say a method runs "as an MCP for chatbots, as a webapp for people, or via API for your software", instead of "from your agent or your chatbot via MCP, as a webapp, or via API in any software".
- **The quick start says the plugin also puts a method in your software**: the README and the documentation's **Quick Start** now say the plugin's skills build methods, run them and put them in your software, name `/pipelex-scaffold` for a webapp and `/pipelex-integrate` for a typed call from TypeScript or Python code, and say that `@pipelex/sdk` is the TypeScript SDK and `pipelex-sdk` the Python one.
- **The documentation's cookbook is one page that points to the cookbook**: the cookbook now holds Pipelex's example methods as packages you run by their address on the hosted API, each with its own page showing every way to use it, so the documentation no longer keeps a page per example. **The Pipelex Cookbook** says what the cookbook is and links to it, the pages that linked an example now link its page in the cookbook or in the method library, and every old example address redirects to the new page. The two pages that documented the runtime itself move into the runtime's own sections: **Writing an Inference Plugin**, under **Under the Hood**, now carries the whole sample plugin and the configuration that routes a model to it, and **CV Batch Screening, Step by Step** joins **Get Started**. The README's cookbook line says what the cookbook now is.
+### Fixed
+
+- **A log call before `log.configure` no longer raises**: it goes to the stdlib's default handling at the stdlib's default level, so a library that logs before Pipelex boots, or a boot that logs while it configures, cannot crash on the log line itself.
+
## [v0.65.0] - 2026-09-25
### Changed
diff --git a/docs/configuration/config-practical/logging-config.md b/docs/configuration/config-practical/logging-config.md
index f538d27ea..95e1e8d64 100644
--- a/docs/configuration/config-practical/logging-config.md
+++ b/docs/configuration/config-practical/logging-config.md
@@ -34,6 +34,7 @@ pipelex = "INFO"
- Override log levels for specific packages
- Use `-` instead of `.` in package names (e.g., `urllib3-connectionpool`)
+- A key works at any depth of the logger hierarchy, because Pipelex names every logger after the emitting module: `pipelex` governs the whole runtime, and `pipelex-pipe_operators-pipe_llm = "DEBUG"` opens one module while the rest of `pipelex` stays at its level
### Console Output
diff --git a/docs/tools/logging.md b/docs/tools/logging.md
index f1a7bdce6..04ae25791 100644
--- a/docs/tools/logging.md
+++ b/docs/tools/logging.md
@@ -1,13 +1,13 @@
---
title: "Logging"
-description: "Explore Pipelex logging: custom log levels, Rich console formatting, caller information, and structured data logging for debugging."
+description: "Explore Pipelex logging: named fields, the run-scoped context, module-named loggers, custom log levels, Rich console formatting and structured data logging."
---
# Pipelex Logging System
## Overview
-Pipelex provides a sophisticated logging system that extends Python's standard logging with additional features like emoji support, rich formatting, and custom log levels.
+Pipelex logs through one facade, `from pipelex import log`, built on Python's standard `logging`. A call takes a message, optional named fields and the usual presentation options; the run-scoped identifiers are bound once at a process entry and stamped onto every record emitted in scope. Fields and identifiers ride the stdlib `LogRecord` as attributes, never spliced into the message text, so a structured sink renders them as fields while the console keeps a narrative line.
## Log Levels
@@ -24,53 +24,6 @@ In addition to standard Python log levels, Pipelex introduces custom levels:
| CRITICAL | 50 | Critical errors |
| OFF | 999 | Disable logging |
-## Logging Features
-
-### 1. Rich Formatting
-
-- Beautiful console output with color coding
-- Syntax highlighting for JSON and other data structures
-- Support for clickable file paths
-- Word wrapping for better readability
-
-### 2. Emoji Support
-
-Built-in emoji indicators for different components:
-
-- 🧠 Pipelex core messages
-- ⚪️ OpenAI-related logs
-- 🌀 Google-related logs
-- ⚡️ Network connections
-- *️⃣ JSON processing
-- 🧿 Poor-log channel (`#poor-log`, the simplified fallback logger)
-
-### 3. Caller Information
-
-Optional inclusion of caller information in logs:
-
-- File name and line number
-- Function name
-- Module name
-- Customizable format templates
-
-### 4. Structured Data Logging
-
-Intelligent handling of different data types:
-
-- Pretty-printing for dictionaries and lists
-- JSON formatting with customizable indentation
-- Special handling for None values
-- Exception traceback integration
-
-### 5. Log Dispatch System
-
-Smart routing of log messages:
-
-- Automatic detection of log origin
-- Separate handlers for different logging needs
-- Configurable rich or poor log modes
-- Support for poor loggers (simplified logging)
-
## Using the Logger
```python
@@ -79,13 +32,16 @@ from pipelex import log
# Basic logging
log.info("Simple message")
+# Named fields ride the record as attributes; the message stays narrative
+log.info("Scanned the inputs", fields={"files": 7, "bytes": 12_288})
+
# Logging with title
log.info("Detailed message", title="Process Status")
# Logging with inline title
log.info("Quick update", inline="Status")
-# Logging structured data
+# Logging structured data: rendered as JSON for the console, carried as `data` for a structured sink
data = {"key": "value", "nested": {"data": True}}
log.verbose(data, title="Configuration")
@@ -102,6 +58,116 @@ log.dev("Testing new feature")
log.verbose("Detailed debug information")
```
+Every one of the seven methods (`verbose`, `debug`, `dev`, `info`, `warning`, `error`, `critical`) takes the same keyword-only `fields`. `title`, `inline`, `problem_id` and `include_exception` keep their meaning beside it.
+
+## Fields
+
+`fields` is a mapping of named values. Each entry becomes an attribute of the emitted `LogRecord`, which is where a formatter or a sink reads it: `record.files` for the example above, or `%(files)s` in a stdlib format string. Nothing from `fields` is written into the message, so `record.getMessage()` is exactly the text you passed.
+
+A value can be anything; the console ignores it and a structured sink serializes it, so prefer plain JSON-ready values (strings, numbers, booleans, lists and dictionaries of those) for anything meant to leave the process.
+
+### Naming convention
+
+- A field name is a `snake_case` identifier, spelled the way the value is spelled where it comes from: a field carrying a payload's `pipeline_run_id` is `pipeline_run_id`, not `pipelineRunId` or `run`.
+- Where the [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/logs/) define a key for the concept, use that key verbatim, dots included (`http.response.status_code`, `code.function.name`), so an OTLP sink emits it without translation. The three run identifiers have no such key and keep their payload names.
+- `request_id`, `pipeline_run_id` and `pipe_run_id` are reserved for the [run-scoped context](#the-run-scoped-context), and `data` is reserved for [structured content](#structured-content). A field of one of those names is accepted and takes precedence as described below, but nothing else should use them.
+
+### Names the stdlib owns
+
+The stdlib refuses an `extra` key that would overwrite one of the record's own attributes (`name`, `message`, `lineno`, `module`, `args`, `asctime` and the rest), and a library's log call never raises. An entry of such a name is therefore carried under the prefix `field_`: `fields={"name": "alpha"}` lands as `record.field_name`. What counts as owned is read off the record actually built, through whatever record factory is installed, so an attribute an OpenTelemetry or tracing instrumentation stamps on every record is a collision too, for a field, a context identifier and `data` alike. The prefix is applied until the name lands on an attribute nobody owns, and entries attach in order, so a call that gives both `name` and `field_name` keeps both values whatever their order: the one that arrives second lands on `field_field_name`.
+
+## The run-scoped context
+
+The identifiers that correlate a record with the run it belongs to are bound once and stamped onto every record emitted while the binding holds:
+
+```python
+with log.context(request_id="req-7f3a", pipeline_run_id="plr-01"):
+ log.info("Loading the bundle") # carries request_id and pipeline_run_id
+ with log.context(pipe_run_id="pr-9c"):
+ log.info("Running the pipe") # carries all three
+ log.info("Delivering") # pipe_run_id is gone again
+log.info("Outside") # carries none of them
+```
+
+The context is a `LogContext` with three optional identifiers, `request_id`, `pipeline_run_id` and `pipe_run_id`. The rules:
+
+- **Absent means absent.** An identifier that is not bound is not on the record; it is never the string `"None"`.
+- **Nested bindings merge, the inner overriding the outer** for the identifiers it gives. Passing `None` inherits the outer value rather than clearing it.
+- **Exit restores the previous binding**, whether the block returns or raises.
+- **The binding is task-local.** It lives in a `contextvars.ContextVar`, so concurrent asyncio tasks each keep their own and a task started inside the block inherits it.
+- **A field overrides the context for its own record**: `log.info("...", fields={"request_id": "other"})` inside a bound context stamps `other`, for a call site that speaks about a request it is not running under.
+
+`pipelex.tools.log.log_context.get_log_context()` returns the current `LogContext`, or `None` outside any binding.
+
+### Where the context is bound
+
+The identifiers travel in the payload, and the contextvar is in-process plumbing bound after deserialization and nothing else; it never crosses a process boundary. Each process entry binds from the payload it received:
+
+- **A direct-mode run**: `PipeRun.run` binds `request_id` and `pipeline_run_id` from the job's `JobMetadata` for the whole run, delivery included, and releases the binding when the run returns. The metadata a submission builds carries no `pipe_run_id` yet.
+- **Every pipe run**: `live_run_pipe` mints the pipe run's id and binds `pipe_run_id` around the whole of the run, the line announcing it, its span lines and its failure included, so every record emitted during a pipe's run names the run it belongs to, a nested pipe rebinding its own and the outer id coming back when it returns, however it returns. A pipe lifted for absent optional inputs does not run and has no id: its skip line carries the enclosing binding, the parent pipe's or none. This is the binding every orchestration shares, direct or distributed.
+- **An API request**: the runner's request middleware is where `request_id` is bound from the inbound request, for the request's duration.
+- **A durable-execution activity or workflow**: the entry is where the identifiers are bound from the payload the orchestrator handed it.
+
+The last two are the runner's and the orchestration plugin's to bind, beside the payload they read; the runtime only provides `log.context`.
+
+## Structured content
+
+When the content is not a string, it is rendered as JSON for the console, indented by `json_logs_indent`, and the JSON-ready form is carried as the record's `data` attribute for a structured sink:
+
+- a `dict` is carried as a dictionary,
+- a `list` is carried as a list,
+- any other object, a pydantic model for instance, is carried as the dictionary or list its serialization produces,
+- `None` is rendered as the word `None` and carries no `data`.
+
+Structured content owns `data` outright: a `data` entry in `fields` beside a non-string content is overridden.
+
+## Logger names and levels
+
+Every record is emitted on the stdlib logger named after the module that made the call: a line from `pipelex/pipe_operators/pipe_llm.py` goes to `pipelex.pipe_operators.pipe_llm`, a line from your own `myapp.jobs.nightly` module goes to `myapp.jobs.nightly`. The module is read from one frame lookup at a fixed depth, so a log line costs a dictionary and a frame lookup; nothing walks the stack.
+
+Because the stdlib logger hierarchy inherits levels, a `package_log_levels` key at any depth works, with `-` standing for `.`:
+
+```toml
+[runtime.log.package_log_levels]
+pipelex = "INFO"
+pipelex-pipe_operators-pipe_llm = "DEBUG"
+httpx = "WARNING"
+```
+
+The record's `pathname`, `lineno` and `funcName` point at the calling line, which is what the Rich handler links to and what the caller-info templates render when `is_caller_info_enabled` is on; all of it comes from that same frame, with no source file read.
+
+## Before configuration
+
+`log.configure` runs at boot, and a call before it never raises: the record goes to the stdlib's default handling at the stdlib's default level, so an `INFO` is dropped and a warning reaches stderr through `logging.lastResort`. No handler is installed as a side effect, and nothing is swallowed. A library that logs before Pipelex boots, and the boot itself while it configures, are both safe.
+
+## Console rendering
+
+### Rich Formatting
+
+- Color coding and syntax highlighting for JSON and other data structures
+- Clickable file paths pointing at the calling line
+- Word wrapping for better readability
+
+### Emoji Support
+
+Built-in emoji indicators for different components, keyed on the logger name's prefix:
+
+- 🧠 Pipelex core messages
+- ⚪️ OpenAI-related logs
+- 🌀 Google-related logs
+- ⚡️ Network connections
+- *️⃣ JSON processing
+- 🧿 Poor-log channel (`#poor-log`, the simplified fallback logger)
+
+### Caller Information
+
+Optional inclusion of caller information in logs, prefixed to the console line:
+
+- File name and line number
+- Function name
+- Module name
+- Customizable format templates
+
## Best Practices
1. **Log Level Selection**:
@@ -114,13 +180,19 @@ log.verbose("Detailed debug information")
- Use ERROR for actual errors
- Use CRITICAL for system-critical issues
-2. **Structured Data**:
+2. **Fields over interpolation**:
+
+ - Put a value a reader might filter or aggregate on in `fields`, and keep the message a sentence
+ - Name a field after the thing it carries, in `snake_case`, or after its OpenTelemetry key when one exists
+ - Leave the run identifiers to the context; bind them at the process entry, never per call
+
+3. **Structured Data**:
- - Log complex data structures directly
+ - Log complex data structures directly; they reach the console as JSON and a sink as `data`
- Use titles for context
- Include problem IDs for trackable issues
-3. **Exception Handling**:
+4. **Exception Handling**:
- Use `include_exception=True` for error context
- Include relevant data in error logs
diff --git a/pipelex/cogt/model_backends/gateway_config.py b/pipelex/cogt/model_backends/gateway_config.py
index 5bf6dc803..2084803a8 100644
--- a/pipelex/cogt/model_backends/gateway_config.py
+++ b/pipelex/cogt/model_backends/gateway_config.py
@@ -42,8 +42,11 @@ def drop_unknown_gateway_defaults(*, gateway_model_specs: BackendModelSpecs) ->
Deliberately pure, and deliberately silent: it runs on the success path of every gateway-backend
load, including loads that happen before `runtime_hub.set_config()` has configured the log
- dispatch. A `log` call here would turn a plain data transform into a boot-order dependency and
- crash the caller with `LogConfig is not set`.
+ dispatch. A `log` call here would turn a plain data transform into a boot-order dependency: a line
+ emitted before the dispatch is configured goes to the stdlib's default handling, where an `info`
+ is dropped and a warning lands on stderr unformatted, ahead of the console the boot is about to
+ configure, so a warning about the served payload would read differently depending on when the
+ load happened to run.
"""
known_fields = InferenceModelSpecBlueprint.model_fields.keys()
defaults = gateway_model_specs.get("defaults")
diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.py
index 2c4ee1571..af0feea20 100644
--- a/pipelex/pipe_machinery/pipe_abstract.py
+++ b/pipelex/pipe_machinery/pipe_abstract.py
@@ -923,66 +923,72 @@ class PipeAbstract(ABC, BaseModel):
output_name: str | None = None,
library_crate: LibraryCrate | None = None,
) -> PipeOutput:
- log.info(self._format_pipe_run_info(pipe_run_params=pipe_run_params))
-
- # Handle telemetry ------------------------------------------------------------
-
# Generate pipe_run_id (business ID, always set)
this_pipe_run_id = PipelineFactory.make_pipe_run_id()
- # Derive OtelContext if telemetry is enabled (not dry mode and tracer available)
- # The trace_id comes from parent's otel_context (already computed at pipeline start)
- this_otel_context: OtelContext | None = None
- span: Span | None = None
- is_root_span: bool = False
-
- parent_otel_context = job_metadata.otel_context
- if not pipe_run_params.run_mode.is_dry and parent_otel_context is not None:
- # Start OTel span first
- span, is_root_span = self._start_pipe_span(
- parent_otel_context=parent_otel_context,
- run_metadata=job_metadata.run_metadata,
- working_memory=working_memory,
- )
- # Get the actual span_id from OTel (OTel generates its own span_id)
- if span:
- span_context = span.get_span_context()
- this_otel_context = OtelContext(
- trace_id=parent_otel_context.trace_id,
- trace_name=parent_otel_context.trace_name,
- trace_name_redacted=parent_otel_context.trace_name_redacted,
- span_id=span_context.span_id,
+ # The pipe run's id exists from here on, so this is where the log context takes it: every
+ # record emitted during the run, its announcement, its span lines and its failure included,
+ # names the pipe run it belongs to, a nested pipe's until it binds its own. The outer binding
+ # comes back when the pipe returns, however it returns. A pipe lifted for absent optional
+ # inputs never gets here: it has no run and no id, and its skip line carries the enclosing binding.
+ with log.context(pipe_run_id=this_pipe_run_id):
+ log.info(self._format_pipe_run_info(pipe_run_params=pipe_run_params))
+
+ # Handle telemetry ------------------------------------------------------------
+
+ # Derive OtelContext if telemetry is enabled (not dry mode and tracer available)
+ # The trace_id comes from parent's otel_context (already computed at pipeline start)
+ this_otel_context: OtelContext | None = None
+ span: Span | None = None
+ is_root_span: bool = False
+
+ parent_otel_context = job_metadata.otel_context
+ if not pipe_run_params.run_mode.is_dry and parent_otel_context is not None:
+ # Start OTel span first
+ span, is_root_span = self._start_pipe_span(
+ parent_otel_context=parent_otel_context,
+ run_metadata=job_metadata.run_metadata,
+ working_memory=working_memory,
)
+ # Get the actual span_id from OTel (OTel generates its own span_id)
+ if span:
+ span_context = span.get_span_context()
+ this_otel_context = OtelContext(
+ trace_id=parent_otel_context.trace_id,
+ trace_name=parent_otel_context.trace_name,
+ trace_name_redacted=parent_otel_context.trace_name_redacted,
+ span_id=span_context.span_id,
+ )
- # Create child metadata with updated pipe_code and pipe_run_id
- # This passes down a modified copy rather than mutating the original
- # otel_context is passed separately because it must always be set explicitly
- # (even when None in dry mode) to avoid inheriting stale parent context
- child_metadata = job_metadata.copy_with_update(
- otel_context=this_otel_context,
- pipe_code=self.code,
- pipe_run_id=this_pipe_run_id,
- )
+ # Create child metadata with updated pipe_code and pipe_run_id
+ # This passes down a modified copy rather than mutating the original
+ # otel_context is passed separately because it must always be set explicitly
+ # (even when None in dry mode) to avoid inheriting stale parent context
+ child_metadata = job_metadata.copy_with_update(
+ otel_context=this_otel_context,
+ pipe_code=self.code,
+ pipe_run_id=this_pipe_run_id,
+ )
- # Run pipe ------------------------------------------------------------
+ # Run pipe ------------------------------------------------------------
- try:
- pipe_output = await self._live_run_pipe(
- job_metadata=child_metadata,
- working_memory=working_memory,
- pipe_run_params=pipe_run_params,
- output_name=output_name,
- library_crate=library_crate,
- )
- except Exception as exc:
- # Broad catch is intentional: the OTel span must be closed with ERROR status
- # on any failure. Observes-and-re-raises — see note on the catch in _run_pipe_traced.
- self._end_pipe_span_error(span, error=exc, is_root_span=is_root_span)
- raise
+ try:
+ pipe_output = await self._live_run_pipe(
+ job_metadata=child_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
+ )
+ except Exception as exc:
+ # Broad catch is intentional: the OTel span must be closed with ERROR status
+ # on any failure. Observes-and-re-raises — see note on the catch in _run_pipe_traced.
+ self._end_pipe_span_error(span, error=exc, is_root_span=is_root_span)
+ raise
- # Handle telemetry ------------------------------------------------------------
+ # Handle telemetry ------------------------------------------------------------
- self._end_pipe_span_success(span=span, pipe_output=pipe_output, is_root_span=is_root_span)
+ self._end_pipe_span_success(span=span, pipe_output=pipe_output, is_root_span=is_root_span)
return pipe_output
diff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.py
index c5fe53382..c202599e7 100644
--- a/pipelex/pipe_run/pipe_run.py
+++ b/pipelex/pipe_run/pipe_run.py
@@ -34,6 +34,23 @@ class PipeRun(PipeRunProtocol):
pipe_job: PipeJob,
*,
delivery_assignment: DeliveryAssignment | None = None,
+ ) -> PipeOutput:
+ job_metadata = pipe_job.job_metadata
+ pipeline_run_id: str = job_metadata.run_metadata.pipeline_run_id
+ # The run's identifiers ride the payload; here is where a direct-mode run binds them onto every
+ # record it emits, the delivery's included. The binding is released when the run returns.
+ with log.context(
+ request_id=job_metadata.run_metadata.request_id,
+ pipeline_run_id=pipeline_run_id,
+ pipe_run_id=job_metadata.pipe_run_id,
+ ):
+ return await self._run_bound(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+
+ async def _run_bound(
+ self,
+ *,
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None,
) -> PipeOutput:
pipeline_run_id: str = pipe_job.job_metadata.run_metadata.pipeline_run_id
status: DeliveryStatus = DeliveryStatus.COMPLETED
diff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.py
index cbd3c9a54..ac384ce99 100644
--- a/pipelex/system/configuration/config_loader.py
+++ b/pipelex/system/configuration/config_loader.py
@@ -65,8 +65,10 @@ class ConfigLoader:
"""The warning a tolerated boot owes the user, once — or ``None`` when the load was clean.
The loader parks it rather than logging it because the main configuration is what
- *configures logging*: at the moment the retry succeeds there is no logger yet, and the
- dispatch raises on any attempt. The boot emits it right after ``log.configure``.
+ *configures logging*: at the moment the retry succeeds no handler is installed yet, and a
+ line emitted then goes to the stdlib's default handling, where an ``INFO`` is dropped and a
+ warning lands on stderr unformatted, ahead of the console about to be configured. The boot
+ emits it right after ``log.configure``, where it reads like every other line.
"""
warning, self._stale_warning = self._stale_warning, None
return warning
diff --git a/pipelex/tools/log/log.py b/pipelex/tools/log/log.py
index 599861801..2121b40ff 100644
--- a/pipelex/tools/log/log.py
+++ b/pipelex/tools/log/log.py
@@ -1,15 +1,24 @@
+from __future__ import annotations
+
import logging
import sys
-from typing import Any
+from typing import TYPE_CHECKING, Any
from rich.console import Console
from rich.logging import RichHandler
-from pipelex.tools.log.log_config import LogConfig, LogMode
+from pipelex.tools.log.log_context import bind_log_context
from pipelex.tools.log.log_dispatch import LogDispatch
from pipelex.tools.log.log_formatter import EmojiLogFormatter, LevelAndEmojiLogFormatter
from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_OFF, LOGGING_LEVEL_VERBOSE, LogLevel
+if TYPE_CHECKING:
+ from collections.abc import Mapping
+ from contextlib import AbstractContextManager
+
+ from pipelex.tools.log.log_config import LogConfig, LogMode
+ from pipelex.tools.log.log_context import LogContext
+
class Log:
"""A class for managing logging configurations and operations."""
@@ -28,22 +37,6 @@ class Log:
def set_log_mode(self, mode: LogMode):
self.log_dispatch.set_log_mode(mode=mode)
- @property
- def _log_config(self) -> LogConfig:
- """Get the log configuration, raising an error if it's not set.
-
- Returns:
- LogConfig: The current log configuration.
-
- Raises:
- RuntimeError: If the log configuration is not set.
-
- """
- if self._log_config_instance is None:
- msg = "LogConfig is not set. You must initialize Pipelex first."
- raise RuntimeError(msg)
- return self._log_config_instance
-
def reset(self):
"""Reset the logging system."""
# Remove all handlers from the root logger
@@ -150,7 +143,9 @@ class Log:
bool: True if the message should be ignored, False otherwise.
"""
- return bool(problem_id) and problem_id in self._log_config.silenced_problem_ids
+ if self._log_config_instance is None:
+ return False
+ return bool(problem_id) and problem_id in self._log_config_instance.silenced_problem_ids
########################################################
# Public methods
@@ -221,12 +216,28 @@ class Log:
for package_name, level in package_log_levels.items():
self.set_level_for_package(package_name=package_name, level=level)
+ def context(
+ self,
+ *,
+ request_id: str | None = None,
+ pipeline_run_id: str | None = None,
+ pipe_run_id: str | None = None,
+ ) -> AbstractContextManager[LogContext]:
+ """Bind the run-scoped identifiers onto every record emitted inside the block.
+
+ Bound at a process entry from the payload it received — ``PipeRun.run`` binds from the job's
+ metadata — and released when the block exits. Nested blocks merge, the inner overriding the
+ outer for the identifiers it gives; a ``None`` inherits rather than clears.
+ """
+ return bind_log_context(request_id=request_id, pipeline_run_id=pipeline_run_id, pipe_run_id=pipe_run_id)
+
def verbose(
self,
content: str | Any,
*,
title: str | None = None,
inline: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log a verbose message.
@@ -235,10 +246,11 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
"""
severity = LOGGING_LEVEL_VERBOSE
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
def debug(
self,
@@ -246,6 +258,7 @@ class Log:
*,
title: str | None = None,
inline: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log a debug message.
@@ -254,10 +267,11 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
"""
severity = logging.DEBUG
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
def dev(
self,
@@ -265,6 +279,7 @@ class Log:
*,
title: str | None = None,
inline: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log a development message.
@@ -273,10 +288,11 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
"""
severity = LOGGING_LEVEL_DEV
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
def info(
self,
@@ -284,6 +300,7 @@ class Log:
*,
title: str | None = None,
inline: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log an info message.
@@ -292,10 +309,11 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
"""
severity = logging.INFO
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
def warning(
self,
@@ -304,6 +322,7 @@ class Log:
title: str | None = None,
inline: str | None = None,
problem_id: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log a warning message.
@@ -312,13 +331,14 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
problem_id (str | None, optional): A problem ID to associate with the warning. Defaults to None.
"""
if self._should_ignore(problem_id=problem_id):
return
severity = logging.WARNING
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
def error(
self,
@@ -328,6 +348,7 @@ class Log:
inline: str | None = None,
include_exception: bool = False,
problem_id: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log an error message.
@@ -336,6 +357,7 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
include_exception (bool, optional): Whether to include exception information. Defaults to False.
problem_id (str | None, optional): A problem ID to associate with the error. Defaults to None.
@@ -349,6 +371,7 @@ class Log:
title=title,
inline=inline,
include_exception=include_exception,
+ fields=fields,
)
def critical(
@@ -359,6 +382,7 @@ class Log:
inline: str | None = None,
include_exception: bool = False,
problem_id: str | None = None,
+ fields: Mapping[str, Any] | None = None,
):
"""Log a critical message.
@@ -367,6 +391,7 @@ class Log:
title (str | None, optional): The title of the log message. Defaults to None.
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
+ fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
include_exception (bool, optional): Whether to include exception information. Defaults to False.
problem_id (str | None, optional): A problem ID to associate with the critical message. Defaults to None.
@@ -380,6 +405,7 @@ class Log:
title=title,
inline=inline,
include_exception=include_exception,
+ fields=fields,
)
diff --git a/pipelex/tools/log/log_context.py b/pipelex/tools/log/log_context.py
new file mode 100644
index 000000000..edac11501
--- /dev/null
+++ b/pipelex/tools/log/log_context.py
@@ -0,0 +1,91 @@
+"""The run-scoped log context: the identifiers every record emitted in scope carries.
+
+The identifiers travel in the payload (``JobMetadata`` for a direct-mode run, the request or the
+activity argument at a process entry elsewhere); a process entry binds them here with
+:func:`bind_log_context`, and the dispatch stamps them onto every record until the block exits. The
+contextvar is in-process plumbing after deserialization and nothing else: it never crosses a process
+boundary, and a record emitted outside any bound context carries none of the identifiers.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel, ConfigDict
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+# The reserved identifier names. They are spelled exactly as the payload fields they come from, so a
+# record and the ``ErrorReport`` or the run result it relates to agree on the key.
+REQUEST_ID_FIELD = "request_id"
+PIPELINE_RUN_ID_FIELD = "pipeline_run_id"
+PIPE_RUN_ID_FIELD = "pipe_run_id"
+
+
+class LogContext(BaseModel):
+ """The identifiers bound for a scope. An absent identifier is ``None`` here and absent from the record."""
+
+ model_config = ConfigDict(frozen=True)
+
+ request_id: str | None = None
+ pipeline_run_id: str | None = None
+ pipe_run_id: str | None = None
+
+ @property
+ def fields(self) -> dict[str, str]:
+ """The bound identifiers as record attributes: only the ones that are set, never the string ``"None"``."""
+ fields: dict[str, str] = {}
+ if self.request_id is not None:
+ fields[REQUEST_ID_FIELD] = self.request_id
+ if self.pipeline_run_id is not None:
+ fields[PIPELINE_RUN_ID_FIELD] = self.pipeline_run_id
+ if self.pipe_run_id is not None:
+ fields[PIPE_RUN_ID_FIELD] = self.pipe_run_id
+ return fields
+
+ def merged_with(
+ self,
+ *,
+ request_id: str | None = None,
+ pipeline_run_id: str | None = None,
+ pipe_run_id: str | None = None,
+ ) -> LogContext:
+ """A copy where each given identifier overrides this one's and a ``None`` inherits it."""
+ return LogContext(
+ request_id=request_id if request_id is not None else self.request_id,
+ pipeline_run_id=pipeline_run_id if pipeline_run_id is not None else self.pipeline_run_id,
+ pipe_run_id=pipe_run_id if pipe_run_id is not None else self.pipe_run_id,
+ )
+
+
+_log_context_var: ContextVar[LogContext | None] = ContextVar("pipelex_log_context", default=None)
+
+
+def get_log_context() -> LogContext | None:
+ """The context bound for the current task, or ``None`` outside any ``bind_log_context`` block."""
+ return _log_context_var.get()
+
+
+@contextmanager
+def bind_log_context(
+ *,
+ request_id: str | None = None,
+ pipeline_run_id: str | None = None,
+ pipe_run_id: str | None = None,
+) -> Generator[LogContext]:
+ """Bind the identifiers for the duration of the block, merged over whatever is already bound.
+
+ Nested blocks merge, the inner one overriding the outer for the identifiers it gives, and the exit
+ restores the previous binding whether the block returns or raises. A contextvar is task-local, so
+ concurrent asyncio tasks each keep their own binding.
+ """
+ current = _log_context_var.get() or LogContext()
+ bound = current.merged_with(request_id=request_id, pipeline_run_id=pipeline_run_id, pipe_run_id=pipe_run_id)
+ token = _log_context_var.set(bound)
+ try:
+ yield bound
+ finally:
+ _log_context_var.reset(token)
diff --git a/pipelex/tools/log/log_dispatch.py b/pipelex/tools/log/log_dispatch.py
index ee83c9eaf..a6be0e8d7 100644
--- a/pipelex/tools/log/log_dispatch.py
+++ b/pipelex/tools/log/log_dispatch.py
@@ -1,16 +1,60 @@
+"""Where a ``log.<level>(...)`` call becomes a stdlib ``LogRecord``.
+
+One frame lookup at a fixed depth names the emitting module and locates the call; the logger is named
+after that module; the bound context, the call's fields and any structured content ride the record as
+attributes through ``extra``; and the console keeps the narrative line it always had. Nothing here
+walks the stack, and nothing here raises for want of a configuration: before ``configure`` the record
+goes to the stdlib's default handling.
+"""
+
+from __future__ import annotations
+
import inspect
+import json
import logging
-import os
import traceback
from pathlib import Path
-from typing import Any, cast
+from typing import TYPE_CHECKING, Any, cast
from pipelex.tools.log.log_config import CallerInfoTemplate, LogConfig, LogMode
+from pipelex.tools.log.log_context import get_log_context
+from pipelex.tools.log.log_fields import attach_log_record_extra, build_log_record_extra
from pipelex.tools.misc.json_utils import purify_json, purify_json_dict, purify_json_list
+if TYPE_CHECKING:
+ from collections.abc import Mapping
+ from types import FrameType
+
+# The frames between ``_caller_frame`` and the code that called ``log.<level>(...)``: the helper's own
+# frame, then ``LogDispatch.dispatch``, then the ``Log`` method. A change to that call chain moves this number.
+CALLER_FRAME_DEPTH = 3
+UNKNOWN_MODULE_NAME = "unknown"
+UNKNOWN_FILE_NAME = "(unknown file)"
+UNKNOWN_FUNCTION_NAME = "(unknown function)"
+
+
+def _caller_frame() -> FrameType | None:
+ """The frame of the code that called ``log.<level>(...)``, or ``None`` when the stack is shallower than that.
+
+ A fixed number of ``f_back`` hops from this frame: no stack is materialised and no module is looked up.
+ """
+ frame = inspect.currentframe()
+ for _ in range(CALLER_FRAME_DEPTH):
+ if frame is None:
+ return None
+ frame = frame.f_back
+ return frame
+
+
+def _module_name(*, frame: FrameType | None) -> str:
+ if frame is None:
+ return UNKNOWN_MODULE_NAME
+ module_name = cast("str | None", frame.f_globals.get("__name__"))
+ return module_name or UNKNOWN_MODULE_NAME
+
class LogDispatch:
- """A class for handling log dispatching to both console and Google Cloud."""
+ """Turns the facade's calls into records on module-named stdlib loggers."""
########################################################
# Init and Configure
@@ -27,22 +71,6 @@ class LogDispatch:
"""Reset the log dispatch."""
self._log_config_instance = None
- @property
- def _log_config(self) -> LogConfig:
- """Retrieves the log configuration.
-
- Returns:
- LogConfig: The current log configuration.
-
- Raises:
- RuntimeError: If LogConfig is not set.
-
- """
- if self._log_config_instance is None:
- msg = "LogConfig is not set. You must call runtime_hub.set_config()."
- raise RuntimeError(msg)
- return self._log_config_instance
-
def configure(self, log_config: LogConfig):
"""Configures the LogDispatch with log configuration.
@@ -60,7 +88,7 @@ class LogDispatch:
self.log_mode = log_config.log_mode
########################################################
- # Private methods
+ # Dispatch
########################################################
def dispatch(
@@ -71,214 +99,125 @@ class LogDispatch:
title: str | None = None,
inline: str | None = None,
include_exception: bool = False,
+ fields: Mapping[str, Any] | None = None,
):
- """Dispatches a log message to appropriate logging methods based on content type.
+ """Emit one record for the call, on the logger named after the calling module.
Args:
- content (Union[str, Any]): The content to be logged.
- severity (int): The severity level of the log message.
- title (str | None, optional): The title of the log message. Defaults to None.
- inline (str | None, optional): Inline title for the log message. Defaults to None.
- Used to display the title inline, only if the title arg is None.
- include_exception (bool, optional): Whether to include exception traceback. Defaults to False.
+ content: The content to log. A string is the message; anything else is rendered as JSON for
+ the console and carried as the ``data`` attribute for structured sinks.
+ severity: The severity level of the log message.
+ title: A title rendered above the content. Defaults to None.
+ inline: A title rendered inline before a string content, used only when ``title`` is None.
+ include_exception: Whether to append the current exception's traceback to the message.
+ fields: Named values carried as attributes of the record, never rendered into the message.
"""
- caller_info_str: str | None = None
- if (
- (self._log_config.is_caller_info_enabled)
- and (frame0 := inspect.currentframe())
- and (frame1 := frame0.f_back)
- # and (frame2 := frame1.f_back)
- and (caller_frame := frame1.f_back)
- ):
- caller_info = inspect.getframeinfo(caller_frame)
- caller_path = Path(caller_info.filename)
- try:
- caller_path = caller_path.relative_to(Path.cwd())
- except ValueError:
- # This can happen if the file is on a different drive (on Windows)
- # In this case, we'll keep the absolute path
- pass
- caller_line = caller_info.lineno
- caller_func = caller_info.function
- template_str = CallerInfoTemplate.for_template_key(key=self._log_config.caller_info_template)
- caller_info_str = template_str.format(file=str(caller_path), line=caller_line, func=caller_func)
-
- if isinstance(content, str):
- self._log_message(
- message=content,
- severity=severity,
- caller_info_str=caller_info_str,
- title=title,
- inline=inline,
- include_exception=include_exception,
- )
- else:
- self._log_data(
- data=content,
- severity=severity,
- caller_info_str=caller_info_str,
- title=title,
- include_exception=include_exception,
- )
-
- def _log_message(
- self,
- message: str,
- *,
- severity: int,
- caller_info_str: str | None,
- title: str | None = None,
- inline: str | None = None,
- include_exception: bool = False,
- ):
- """Logs a message to both console and Google Cloud.
+ caller_frame = _caller_frame()
+ module_name = _module_name(frame=caller_frame)
+ log_config = self._log_config_instance
+ if log_config is not None and not log_config.is_console_logging_enabled:
+ return
+ logger = self._logger_for(module_name=module_name, log_config=log_config)
+ if not logger.isEnabledFor(severity):
+ return
- Args:
- message (str): The message to be logged.
- severity (int): The severity level of the log message.
- caller_info_str (str | None): Information about the caller.
- title (str | None, optional): The title of the log message. Defaults to None.
- inline (str | None, optional): Inline title for the log message. Defaults to None.
- Used to display the title inline, only if the title arg is None.
- include_exception (bool, optional): Whether to include exception traceback. Defaults to False.
+ message, data = self._render_content(content=content, title=title, inline=inline, log_config=log_config)
+ if log_config is not None and log_config.is_caller_info_enabled and caller_frame is not None:
+ caller_info_str = self._caller_info(frame=caller_frame, module_name=module_name, log_config=log_config)
+ message = f"{caller_info_str}: {message}"
+ if include_exception:
+ message += f"\n{traceback.format_exc()}"
- """
- if title is not None:
- message = f"{title}:\n{message}"
- elif inline is not None:
- message = f"{inline}: {message}"
+ extra = build_log_record_extra(context=get_log_context(), fields=fields, data=data)
+ self._log_to_console(message=message, severity=severity, logger=logger, caller_frame=caller_frame, extra=extra)
- message_for_console = message
- if caller_info_str is not None:
- message_for_console = f"{caller_info_str}: {message}"
+ ########################################################
+ # Private methods
+ ########################################################
- if include_exception:
- message_for_console += f"\n{traceback.format_exc()}"
- self._log_to_console(message=message_for_console, severity=severity)
+ def _logger_for(self, *, module_name: str, log_config: LogConfig | None) -> logging.Logger:
+ """The module-named logger, or the generic poor logger when the dispatch runs in poor mode."""
+ if self.log_mode == LogMode.POOR and log_config is not None:
+ return logging.getLogger(log_config.generic_poor_logger)
+ return logging.getLogger(module_name)
- def _log_data(
+ def _render_content(
self,
- data: Any,
*,
- severity: int,
- caller_info_str: str | None,
- title: str | None = None,
- include_exception: bool = False,
- ):
- """Logs potentially structured data (maybe it's a dict or a list) to both console and Google Cloud.
-
- Args:
- data (Any): The data to be logged.
- severity (int): The severity level of the log message.
- caller_info_str (str | None): Information about the caller.
- title (str | None, optional): The title of the log message. Defaults to None.
- include_exception (bool, optional): Whether to include exception traceback. Defaults to False.
-
- """
- if data is None:
- message = "None"
- if title is not None:
- message = f"{title}:\n{message}"
- if caller_info_str is not None:
- message = f"{caller_info_str}: {message}"
- if include_exception:
- message += f"\n{traceback.format_exc()}"
- self._log_to_console(message=message, severity=severity)
- elif isinstance(data, dict):
- dict_string: str
- _, dict_string = purify_json_dict(
- data=data,
- indent=self._log_config.json_logs_indent,
- is_warning_enabled=True,
- )
- message = f"\n{dict_string}"
+ content: str | Any,
+ title: str | None,
+ inline: str | None,
+ log_config: LogConfig | None,
+ ) -> tuple[str, Any | None]:
+ """The console line for the content and, for structured content, its JSON-ready form."""
+ if isinstance(content, str):
if title is not None:
- message = f"{title}:{message}"
- if caller_info_str is not None:
- message = f"{caller_info_str}: {message}"
- if include_exception:
- message += f"\n{traceback.format_exc()}"
- self._log_to_console(message=message, severity=severity)
- elif isinstance(data, list):
- list_data = cast("list[Any]", data)
- _, list_string = purify_json_list(
- data=list_data,
- indent=self._log_config.json_logs_indent,
- is_truncate_bytes_enabled=True,
- )
- message = f"\n{list_string}"
+ return f"{title}:\n{content}", None
+ if inline is not None:
+ return f"{inline}: {content}", None
+ return content, None
+
+ if content is None:
if title is not None:
- message = f"{title}:{message}"
- if caller_info_str is not None:
- message = f"{caller_info_str}: {message}"
- if include_exception:
- message += f"\n{traceback.format_exc()}"
- self._log_to_console(message=message, severity=severity)
+ return f"{title}:\nNone", None
+ return "None", None
+
+ indent = log_config.json_logs_indent if log_config is not None else None
+ if isinstance(content, dict):
+ _, rendered = purify_json_dict(data=content, indent=indent, is_warning_enabled=True)
+ elif isinstance(content, list):
+ _, rendered = purify_json_list(data=cast("list[Any]", content), indent=indent, is_truncate_bytes_enabled=True)
else:
- _, dict_string = purify_json(
- data=data,
- indent=self._log_config.json_logs_indent,
- is_truncate_bytes_enabled=True,
- is_warning_enabled=False,
- )
- message = f"\n{dict_string}"
- if title is not None:
- message = f"{title}:{message}"
- if caller_info_str is not None:
- message = f"{caller_info_str}: {message}"
- if include_exception:
- message += f"\n{traceback.format_exc()}"
- self._log_to_console(message=message, severity=severity)
-
- def _log_to_console(self, message: str, *, severity: int):
- """Logs a message to the console.
+ _, rendered = purify_json(data=content, indent=indent, is_truncate_bytes_enabled=True, is_warning_enabled=False)
+ # The structure the helpers hand back is the caller's own object whenever it was JSON-clean as
+ # given, and a sink that serializes later would read whatever the caller did to it since. The
+ # data is therefore the rendering re-read: a snapshot of the call, JSON-ready whatever it held.
+ data: Any = json.loads(rendered)
+ message = f"\n{rendered}"
+ if title is not None:
+ message = f"{title}:{message}"
+ return message, data
- Args:
- message (str): The message to be logged.
- severity (int): The severity level of the log message.
+ def _caller_info(self, *, frame: FrameType, module_name: str, log_config: LogConfig) -> str:
+ """The caller's location in the configured template, read off the frame without touching the source file."""
+ caller_path = Path(frame.f_code.co_filename)
+ try:
+ caller_path = caller_path.relative_to(Path.cwd())
+ except ValueError:
+ # This can happen if the file is on a different drive (on Windows)
+ # In this case, we'll keep the absolute path
+ pass
+ template_str = CallerInfoTemplate.for_template_key(key=log_config.caller_info_template)
+ return template_str.format(file=str(caller_path), line=frame.f_lineno, func=frame.f_code.co_name, module=module_name)
+
+ def _log_to_console(
+ self,
+ message: str,
+ *,
+ severity: int,
+ logger: logging.Logger,
+ caller_frame: FrameType | None,
+ extra: dict[str, Any],
+ ):
+ """Build the record at the caller's location, attach what it carries, and hand it to the logger's handlers.
+ The record is built first and the extra attached afterwards: ``makeRecord`` raises on a key the
+ record already owns, and what it owns is only known once the installed record factory has run.
"""
- if not self._log_config.is_console_logging_enabled:
- return
-
- match self.log_mode:
- case LogMode.RICH:
- pass
- case LogMode.POOR:
- logger = logging.getLogger(self._log_config.generic_poor_logger)
- logger.log(level=severity, msg=message, stacklevel=6)
- return
-
- stack = inspect.stack()
- try:
- logging_module_path = Path(__file__).absolute()
- log_origin_name = "unknown"
-
- for frame_info in stack[1:]:
- try:
- frame = frame_info.frame
- module = inspect.getmodule(frame)
-
- if module is None:
- continue
-
- if hasattr(module, "__file__") and module.__file__ is not None:
- module_file = os.path.abspath(module.__file__)
- else:
- continue
-
- if module_file == logging_module_path or module_file.endswith("/log.py"):
- continue
- if module.__name__ == "__main__":
- log_origin_name = "Unknown"
- else:
- log_origin_name = module.__name__.split(sep=".", maxsplit=1)[0]
- break
- finally:
- del frame_info
- logger = logging.getLogger(log_origin_name)
- logger.log(level=severity, msg=message, stacklevel=5)
- finally:
- if stack:
- del stack
+ if caller_frame is None:
+ pathname, lineno, func_name = UNKNOWN_FILE_NAME, 0, UNKNOWN_FUNCTION_NAME
+ else:
+ pathname, lineno, func_name = caller_frame.f_code.co_filename, caller_frame.f_lineno or 0, caller_frame.f_code.co_name
+ record = logger.makeRecord(
+ name=logger.name,
+ level=severity,
+ fn=pathname,
+ lno=lineno,
+ msg=message,
+ args=(),
+ exc_info=None,
+ func=func_name,
+ )
+ attach_log_record_extra(record=record, extra=extra)
+ logger.handle(record)
diff --git a/pipelex/tools/log/log_fields.py b/pipelex/tools/log/log_fields.py
new file mode 100644
index 000000000..868e4b9d0
--- /dev/null
+++ b/pipelex/tools/log/log_fields.py
@@ -0,0 +1,65 @@
+"""What a record carries beyond its message: the bound context, the call's fields and the structured content.
+
+All three become attributes of the stdlib ``LogRecord``, which is where a sink reads them. The stdlib
+refuses an ``extra`` key that would overwrite one of the record's own attributes, and a library's log
+call never raises, so a colliding entry is carried under a prefixed name instead. The collision is read
+off the record actually built, through whatever record factory is installed, so an attribute a factory
+added is a collision too rather than the stdlib's ``KeyError``.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ import logging
+ from collections.abc import Mapping
+
+ from pipelex.tools.log.log_context import LogContext
+
+# The attribute a ``dict`` or ``list`` content is carried under, JSON-ready, beside its console rendering.
+DATA_FIELD = "data"
+
+# The prefix an entry takes when its name is one the record already owns.
+COLLIDING_FIELD_PREFIX = "field_"
+
+# Set by the formatter rather than the constructor, so a fresh record does not carry them yet and the stdlib refuses them all the same.
+FORMATTER_OWNED_ATTRIBUTES = frozenset({"message", "asctime"})
+
+
+def build_log_record_extra(
+ *,
+ context: LogContext | None,
+ fields: Mapping[str, Any] | None,
+ data: Any | None,
+) -> dict[str, Any]:
+ """The ``extra`` for one record, in precedence order: the bound context, then the call's fields, then the content.
+
+ A field overrides the context for its record, so a call site that names a request it is not
+ running under can say so; structured content owns ``data`` outright.
+ """
+ extra: dict[str, Any] = {}
+ if context is not None:
+ extra.update(context.fields)
+ if fields:
+ extra.update(fields)
+ if data is not None:
+ extra[DATA_FIELD] = data
+ return extra
+
+
+def attach_log_record_extra(*, record: logging.LogRecord, extra: Mapping[str, Any]) -> None:
+ """Set each entry as an attribute of the record, under a prefixed name when the record already owns that name.
+
+ The record was built by the logger, through the installed record factory, so what it owns is exactly
+ what the stdlib's own ``makeRecord`` would refuse: its declared attributes and anything a factory
+ stamped on it. The prefix is applied until the name lands on an attribute nobody owns, and entries
+ are attached in order, so an entry attached earlier under a prefixed name is owned for the entries
+ after it: whatever the order of the mapping, no value is lost.
+ """
+ owned = vars(record)
+ for name, value in extra.items():
+ attribute = name
+ while attribute in owned or attribute in FORMATTER_OWNED_ATTRIBUTES:
+ attribute = f"{COLLIDING_FIELD_PREFIX}{attribute}"
+ setattr(record, attribute, value)
diff --git a/subject_grants.toml b/subject_grants.toml
index b7dd51238..2bf168e8c 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -6136,14 +6136,6 @@ rationale = "Factory from source: builds the result from the key; single operand
param = "target"
rationale = "Factory from source: builds the result from the target; single operand."
-["pipelex/tools/log/log_dispatch.py::LogDispatch._log_data"]
-param = "data"
-rationale = "Logs the content; single operand (formatting keyword-only)."
-
-["pipelex/tools/log/log_dispatch.py::LogDispatch._log_message"]
-param = "message"
-rationale = "Logs the content; single operand (formatting keyword-only)."
-
["pipelex/tools/log/log_dispatch.py::LogDispatch._log_to_console"]
param = "message"
rationale = "Transforms the message; single operand."
diff --git a/tests/unit/pipelex/cli/test_agent_output.py b/tests/unit/pipelex/cli/test_agent_output.py
index b5704be80..e7799afb1 100644
--- a/tests/unit/pipelex/cli/test_agent_output.py
+++ b/tests/unit/pipelex/cli/test_agent_output.py
@@ -4,6 +4,7 @@ from __future__ import annotations
import datetime
import json
+import logging
from typing import TYPE_CHECKING, Any
import pytest
@@ -53,6 +54,17 @@ class TestAgentOutput:
yield
consume_setup_warnings()
+ @pytest.fixture(autouse=True)
+ def _restore_logging_cutoff(self) -> Iterator[None]:
+ """Restore the process-global ``logging.disable`` threshold after a test that runs the app.
+
+ The app callback arms the agent CLI cutoff through ``silence_logging_for_agent_cli``;
+ left in place it silences every log call of every test that runs after this module.
+ """
+ original_disable = logging.root.manager.disable
+ yield
+ logging.disable(original_disable)
+
def test_agent_error_outputs_json_to_stderr(self, capsys: pytest.CaptureFixture[str]) -> None:
"""agent_error should print valid JSON to stderr and exit with code 1."""
with pytest.raises(typer.Exit) as exc_info:
diff --git a/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py b/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py
index f0cd633f6..5ecfadd7c 100644
--- a/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py
+++ b/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py
@@ -12,9 +12,6 @@ rules — see `test_backend_library_leniency.py`.
from typing import Any, cast
-from pytest_mock import MockerFixture
-
-from pipelex import log
from pipelex.cogt.model_backends.gateway_config import drop_unknown_gateway_defaults
from pipelex.cogt.model_backends.model_spec_factory import BackendModelSpecs, InferenceModelSpecBlueprint
@@ -50,17 +47,6 @@ class TestGatewayUnknownDefaults:
assert drop_unknown_gateway_defaults(gateway_model_specs=specs) is specs
- def test_pruning_does_not_need_the_log_hub(self, mocker: MockerFixture) -> None:
- """It runs on the success path of every gateway load, including ones that precede runtime_hub.set_config().
-
- The served config really does carry an unknown key today, so this is the ordinary path, not a corner.
- """
- mocker.patch.object(log.log_dispatch, "_log_config_instance", None)
-
- pruned = drop_unknown_gateway_defaults(gateway_model_specs=self._specs(defaults={"max_tokens": 4096, "a_field_we_removed": "anthropic"}))
-
- assert pruned["defaults"] == {"max_tokens": 4096}
-
def test_per_model_unknown_keys_are_untouched(self) -> None:
"""This function is scoped to `defaults`; the per-model rule is the loader's, tested in `test_gateway_unknown_per_model_keys.py`."""
# The unknown key in `defaults` is what forces the deep-copy prune to run at all: without it
diff --git a/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_per_model_keys.py b/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_per_model_keys.py
index 9e51720c0..8a6baa1e6 100644
--- a/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_per_model_keys.py
+++ b/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_per_model_keys.py
@@ -13,7 +13,6 @@ from typing import Any, cast
from pytest_mock import MockerFixture
-from pipelex import log
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.model_backends.backend import PipelexBackend
from pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary
@@ -139,11 +138,3 @@ class TestGatewayUnknownPerModelKeys:
extra_headers, _ = GatewayFactory.make_extras(model_spec, inference_job=mocker.MagicMock(spec=LLMJob), output_desc="text")
assert extra_headers["x-portkey-config"] == "pc-openai-6e7576"
-
- def test_pruning_a_per_model_key_does_not_need_the_log_hub(self, tmp_path: Path, mocker: MockerFixture) -> None:
- """Same constraint as the `defaults` prune: this runs on gateway loads that precede runtime_hub.set_config()."""
- mocker.patch.object(log.log_dispatch, "_log_config_instance", None)
-
- library = self._load(tmp_path, model_specs=self._remote_specs(per_model_extras={"a_field_we_removed": "openai"}))
-
- assert library.get_inference_backend(backend_name="pipelex_gateway") is not None
diff --git a/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
new file mode 100644
index 000000000..fa2a2f30f
--- /dev/null
+++ b/tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py
@@ -0,0 +1,246 @@
+"""``live_run_pipe`` binds the pipe run's own id onto the log context for the whole of the pipe's run.
+
+The job metadata a submission builds carries no ``pipe_run_id``; the pipe mints one as it starts and
+hands it down on a metadata copy. The binding here is what makes every record emitted during a pipe's
+run name the run it belongs to: the line announcing it, the records of its body, its span hooks and
+its failure alike. A nested pipe rebinds its own and the outer one comes back when it returns, however
+it returns. A pipe lifted for absent optional inputs has no run and no id, so its skip line is not in
+scope here. It is the binding every orchestration shares, so it lives on the pipe and not on the runner.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING, Any, NamedTuple
+
+import pytest
+from pydantic import Field
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.tools.log.log_context import LogContext, get_log_context
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+if TYPE_CHECKING:
+ from pipelex.core.memory.working_memory import WorkingMemory
+ from pipelex.libraries.library_crate import LibraryCrate
+
+
+class Seen(NamedTuple):
+ """What a recording pipe saw at one moment of its run: the moment, the metadata's pipe_run_id if any, and the bound context."""
+
+ moment: str
+ pipe_run_id: str | None
+ context: LogContext | None
+
+
+class RecordingPipe(PipeAbstract):
+ """A pipe whose live run and span hooks record the context they see; it may run a nested pipe, and it may fail."""
+
+ pipe_category: Any = "PipeOperator"
+ type: Any = "PipeFunc"
+ nested: RecordingPipe | None = None
+ fails: bool = False
+ seen: list[Seen] = Field(default_factory=empty_list_factory_of(Seen))
+
+ def _see(self, *, moment: str, job_metadata: JobMetadata | None = None) -> None:
+ pipe_run_id = job_metadata.pipe_run_id if job_metadata is not None else None
+ self.seen.append(Seen(moment=f"{self.code} {moment}", pipe_run_id=pipe_run_id, context=get_log_context()))
+
+ @override
+ def validate_inputs_with_library(self) -> None: ...
+
+ @override
+ def validate_inputs_static(self) -> None: ...
+
+ @override
+ def validate_output_with_library(self) -> None: ...
+
+ @override
+ def validate_output_static(self) -> None: ...
+
+ @override
+ async def _validate_before_run(self, **kwargs: Any) -> None: ...
+
+ @override
+ async def _validate_after_run(self, **kwargs: Any) -> None: ...
+
+ @override
+ def required_variables(self) -> set[str]:
+ return set()
+
+ @override
+ def needed_inputs(self, *, visited_pipes: set[str] | None = None) -> Any:
+ return self.inputs
+
+ @override
+ async def _live_run_pipe(
+ self,
+ *,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ library_crate: LibraryCrate | None = None,
+ ) -> PipeOutput:
+ self._see(moment="running", job_metadata=job_metadata)
+ log.info(f"inside {self.code}")
+ if self.fails:
+ msg = f"{self.code} failed"
+ raise RuntimeError(msg)
+ if self.nested is not None:
+ await self.nested.live_run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ )
+ self._see(moment="after nested", job_metadata=job_metadata)
+ log.info(f"back in {self.code}")
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)
+
+ @override
+ async def _dry_run_pipe(
+ self,
+ *,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ library_crate: LibraryCrate | None = None,
+ ) -> PipeOutput:
+ raise NotImplementedError
+
+ @override
+ def _end_pipe_span_success(self, span: Any, *, pipe_output: PipeOutput, is_root_span: bool) -> None:
+ self._see(moment="succeeded")
+
+ @override
+ def _end_pipe_span_error(self, span: Any, *, error: Exception, is_root_span: bool = False) -> None:
+ self._see(moment="failed")
+ log.info(f"{self.code} span ends in error")
+
+
+def _make_pipe(*, code: str, nested: RecordingPipe | None = None, fails: bool = False) -> RecordingPipe:
+ return RecordingPipe(
+ code=code,
+ domain_code="test_log_context",
+ description=code,
+ output=StuffSpec(concept=ConceptFactory.make_native_concept(NativeConceptCode.TEXT)),
+ nested=nested,
+ fails=fails,
+ seen=[],
+ )
+
+
+def _live_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, batch_max_concurrency=1, pipe_stack_limit=10)
+
+
+def _seam_shaped_metadata() -> JobMetadata:
+ """What a submission builds: the run's identifiers, and no pipe_run_id yet."""
+ return JobMetadata(run_metadata=RunMetadata(user_id="pytest", storage_scope="test/scope", pipeline_run_id="plr-live", request_id="req-live"))
+
+
+def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ return [record for record in caplog.records if record.name == __name__]
+
+
+def _announcements(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ """The lines ``live_run_pipe`` itself emits to announce each pipe, on the pipe machinery's own logger."""
+ return [record for record in caplog.records if record.name == PipeAbstract.__module__ and record.levelno == logging.INFO]
+
+
+def _field(record: logging.LogRecord, *, name: str) -> Any:
+ """A field carried on the record: an attribute the stdlib does not declare, read the way a sink reads it."""
+ return getattr(record, name)
+
+
+def _pipe_run_ids(seen: list[Seen]) -> list[str | None]:
+ return [entry.context.pipe_run_id if entry.context is not None else None for entry in seen]
+
+
+class TestLiveRunPipeLogContext:
+ async def test_the_minted_pipe_run_id_is_bound_for_the_whole_pipe_and_released_after(self, caplog: pytest.LogCaptureFixture) -> None:
+ pipe = _make_pipe(code="outer")
+
+ with caplog.at_level(logging.INFO), log.context(request_id="req-live", pipeline_run_id="plr-live"):
+ await pipe.live_run_pipe(
+ job_metadata=_seam_shaped_metadata(), working_memory=WorkingMemoryFactory.make_empty(), pipe_run_params=_live_params()
+ )
+ assert get_log_context() == LogContext(request_id="req-live", pipeline_run_id="plr-live")
+
+ running, succeeded = pipe.seen
+ assert (running.moment, succeeded.moment) == ("outer running", "outer succeeded")
+ minted_id = running.pipe_run_id
+ assert minted_id is not None
+ assert running.context == LogContext(request_id="req-live", pipeline_run_id="plr-live", pipe_run_id=minted_id)
+ assert succeeded.context == running.context
+ (announcement,) = _announcements(caplog)
+ assert _field(announcement, name="pipe_run_id") == minted_id
+ (record,) = _own_records(caplog)
+ assert _field(record, name="pipe_run_id") == minted_id
+ assert _field(record, name="request_id") == "req-live"
+ assert _field(record, name="pipeline_run_id") == "plr-live"
+
+ async def test_a_nested_pipe_binds_its_own_id_and_the_outer_one_comes_back(self, caplog: pytest.LogCaptureFixture) -> None:
+ inner = _make_pipe(code="inner")
+ outer = _make_pipe(code="outer", nested=inner)
+
+ with caplog.at_level(logging.INFO), log.context(pipeline_run_id="plr-live"):
+ await outer.live_run_pipe(
+ job_metadata=_seam_shaped_metadata(), working_memory=WorkingMemoryFactory.make_empty(), pipe_run_params=_live_params()
+ )
+
+ outer_id = outer.seen[0].pipe_run_id
+ inner_id = inner.seen[0].pipe_run_id
+ assert outer_id is not None
+ assert inner_id is not None
+ assert inner_id != outer_id
+ assert [entry.moment for entry in outer.seen] == ["outer running", "outer after nested", "outer succeeded"]
+ assert _pipe_run_ids(outer.seen) == [outer_id, outer_id, outer_id]
+ assert [entry.moment for entry in inner.seen] == ["inner running", "inner succeeded"]
+ assert _pipe_run_ids(inner.seen) == [inner_id, inner_id]
+ outer_announcement, inner_announcement = _announcements(caplog)
+ assert _field(outer_announcement, name="pipe_run_id") == outer_id
+ assert _field(inner_announcement, name="pipe_run_id") == inner_id
+ outer_record, inner_record, back_record = _own_records(caplog)
+ assert _field(outer_record, name="pipe_run_id") == outer_id
+ assert _field(inner_record, name="pipe_run_id") == inner_id
+ assert _field(back_record, name="pipe_run_id") == outer_id
+
+ async def test_a_failing_pipe_keeps_its_own_id_through_its_failure(self, caplog: pytest.LogCaptureFixture) -> None:
+ inner = _make_pipe(code="inner", fails=True)
+ outer = _make_pipe(code="outer", nested=inner)
+
+ with caplog.at_level(logging.INFO), log.context(pipeline_run_id="plr-live"):
+ with pytest.raises(RuntimeError, match="inner failed"):
+ await outer.live_run_pipe(
+ job_metadata=_seam_shaped_metadata(), working_memory=WorkingMemoryFactory.make_empty(), pipe_run_params=_live_params()
+ )
+ assert get_log_context() == LogContext(pipeline_run_id="plr-live")
+
+ outer_id = outer.seen[0].pipe_run_id
+ inner_id = inner.seen[0].pipe_run_id
+ assert outer_id is not None
+ assert inner_id is not None
+ assert [entry.moment for entry in inner.seen] == ["inner running", "inner failed"]
+ assert _pipe_run_ids(inner.seen) == [inner_id, inner_id]
+ assert [entry.moment for entry in outer.seen] == ["outer running", "outer failed"]
+ assert _pipe_run_ids(outer.seen) == [outer_id, outer_id]
+ outer_record, inner_record, inner_error, outer_error = _own_records(caplog)
+ assert _field(outer_record, name="pipe_run_id") == outer_id
+ assert _field(inner_record, name="pipe_run_id") == inner_id
+ assert inner_error.getMessage() == "inner span ends in error"
+ assert _field(inner_error, name="pipe_run_id") == inner_id
+ assert outer_error.getMessage() == "outer span ends in error"
+ assert _field(outer_error, name="pipe_run_id") == outer_id
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py b/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py
new file mode 100644
index 000000000..8615c4993
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_pipe_run_log_context.py
@@ -0,0 +1,91 @@
+"""Direct-mode ``PipeRun.run`` binds the log context from the job's own metadata.
+
+The run-scoped identifiers travel in the payload; the contextvar is in-process plumbing bound at the
+entry and released when the run returns, so a record emitted by anything the run calls carries them
+and a record emitted after the run does not. The job metadata a submission builds carries no
+``pipe_run_id``: the pipe's own is minted by ``live_run_pipe``, which binds it there, tested in
+``tests/unit/pipelex/pipe_machinery/test_live_run_pipe_log_context.py``.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+from pipelex import log
+from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
+from pipelex.tools.log.log_context import LogContext, get_log_context
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+def _field(record: logging.LogRecord, *, name: str) -> Any:
+ """A field carried on the record: an attribute the stdlib does not declare, read the way a sink reads it."""
+ return getattr(record, name)
+
+
[email protected](loop_scope="class")
+class TestPipeRunLogContext:
+ async def test_run_binds_the_context_from_the_job_metadata(self, mocker: MockerFixture, caplog: pytest.LogCaptureFixture) -> None:
+ seen: dict[str, Any] = {}
+ mock_output = mocker.MagicMock()
+
+ async def observe(pipe_job: Any) -> Any: # ruff: ignore[unused-function-argument]
+ await asyncio.sleep(0)
+ seen["context"] = get_log_context()
+ log.info("inside the run")
+ return mock_output
+
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(side_effect=observe)
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-ctx", storage_scope="test/scope", request_id="req-ctx"),
+ pipe_run_id="pr-ctx",
+ )
+
+ assert get_log_context() is None
+ with caplog.at_level(logging.INFO):
+ result = await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job)
+
+ assert result == mock_output
+ assert seen["context"] == LogContext(request_id="req-ctx", pipeline_run_id="plr-ctx", pipe_run_id="pr-ctx")
+ assert get_log_context() is None
+ (record,) = [record for record in caplog.records if record.name == __name__]
+ assert _field(record, name="request_id") == "req-ctx"
+ assert _field(record, name="pipeline_run_id") == "plr-ctx"
+ assert _field(record, name="pipe_run_id") == "pr-ctx"
+
+ async def test_run_binds_only_what_the_metadata_carries(self, mocker: MockerFixture) -> None:
+ seen: dict[str, Any] = {}
+
+ async def observe(pipe_job: Any) -> Any: # ruff: ignore[unused-function-argument]
+ await asyncio.sleep(0)
+ seen["context"] = get_log_context()
+ return mocker.MagicMock()
+
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(side_effect=observe)
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-only", storage_scope="test/scope"))
+
+ await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job)
+
+ assert seen["context"] == LogContext(pipeline_run_id="plr-only")
+ assert seen["context"].fields == {"pipeline_run_id": "plr-only"}
+
+ async def test_the_context_is_released_when_the_run_fails(self, mocker: MockerFixture) -> None:
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(side_effect=RuntimeError("router blew up"))
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-fail", storage_scope="test/scope"))
+
+ with pytest.raises(RuntimeError):
+ await PipeRun(pipe_router=mock_router).run(pipe_job=mock_job)
+
+ assert get_log_context() is None
diff --git a/tests/unit/pipelex/system/configuration/test_boot_tolerance.py b/tests/unit/pipelex/system/configuration/test_boot_tolerance.py
index 3adeb585d..9159ae8ed 100644
--- a/tests/unit/pipelex/system/configuration/test_boot_tolerance.py
+++ b/tests/unit/pipelex/system/configuration/test_boot_tolerance.py
@@ -531,9 +531,9 @@ class TestTheMainConfigurationLoader:
) -> None:
"""And the *migrated* value is the one that lands, not merely a configuration that parses.
- The main configuration is what *configures logging*, so at this point in a boot no logger
- exists yet: the warning is parked on the loader for the boot to emit once it does. The
- unconfigured dispatch here is a cold boot's, and it raises on any attempt to log through it.
+ The main configuration is what *configures logging*, so at this point in a boot no handler
+ is installed yet: the warning is parked on the loader for the boot to emit once one is. The
+ unconfigured dispatch here is a cold boot's, whose lines go to the stdlib's default handling.
"""
global_dir, _ = fake_dirs
write_synthetic_ledger(
diff --git a/tests/unit/pipelex/tools/test_log_before_configure.py b/tests/unit/pipelex/tools/test_log_before_configure.py
new file mode 100644
index 000000000..ec5495828
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_before_configure.py
@@ -0,0 +1,67 @@
+"""A log call before ``log.configure`` never raises.
+
+A library's log call is never the thing that crashes the process. Before configuration the record
+goes to the stdlib's default handling at the stdlib's default level: no handler is installed as a
+side effect, nothing is swallowed, and a warning still reaches stderr through ``logging.lastResort``.
+"""
+
+from __future__ import annotations
+
+import logging
+import subprocess # ruff: ignore[suspicious-subprocess-import]
+import sys
+from typing import TYPE_CHECKING
+
+from pipelex.tools.log.log import Log
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
+
+if TYPE_CHECKING:
+ import pytest
+
+
+def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ return [record for record in caplog.records if record.name == __name__]
+
+
+class TestLogBeforeConfigure:
+ def test_every_level_emits_without_raising_and_installs_no_handler(self, caplog: pytest.LogCaptureFixture) -> None:
+ fresh = Log()
+ root_handlers_before = list(logging.getLogger().handlers)
+
+ with caplog.at_level(LOGGING_LEVEL_VERBOSE):
+ fresh.verbose("verbose")
+ fresh.debug("debug")
+ fresh.dev("dev")
+ fresh.info("info", fields={"files": 1})
+ fresh.warning("warning", problem_id="some_problem")
+ try:
+ msg = "cause"
+ raise ValueError(msg)
+ except ValueError:
+ fresh.error("error", include_exception=True)
+ fresh.critical("critical")
+ fresh.info({"structured": "content"}, title="Data")
+
+ assert logging.getLogger().handlers == root_handlers_before
+ messages = [record.getMessage() for record in _own_records(caplog)]
+ assert messages[:5] == ["verbose", "debug", "dev", "info", "warning"]
+ assert messages[5].startswith("error\nTraceback")
+ assert messages[6] == "critical"
+ assert messages[7].startswith("Data:")
+ assert all(record.name == __name__ for record in _own_records(caplog))
+
+ def test_a_cold_process_routes_to_the_stdlib_default_handling(self) -> None:
+ """Run it for real: a fresh interpreter, no configure, a warning on stderr and an info dropped at the default level."""
+ script = "from pipelex import log\nlog.info('an info line')\nlog.warning('a warning line')\nprint('survived')\n"
+ completed = subprocess.run( # ruff: ignore[subprocess-without-shell-equals-true]
+ [sys.executable, "-c", script],
+ capture_output=True,
+ text=True,
+ check=False,
+ timeout=120,
+ )
+
+ assert completed.returncode == 0, completed.stderr
+ assert completed.stdout.strip() == "survived"
+ assert "a warning line" in completed.stderr
+ assert "an info line" not in completed.stderr
diff --git a/tests/unit/pipelex/tools/test_log_caller_info.py b/tests/unit/pipelex/tools/test_log_caller_info.py
new file mode 100644
index 000000000..adabcdc01
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_caller_info.py
@@ -0,0 +1,54 @@
+"""Caller information is read off the frame, never off the source file."""
+
+from __future__ import annotations
+
+import inspect
+import logging
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex.tools.log.log import Log
+from pipelex.tools.log.log_config import CallerInfoTemplate, LogConfig
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from pytest_mock import MockerFixture
+
+
+def _own_records(caplog: pytest.LogCaptureFixture, *, name: str) -> list[logging.LogRecord]:
+ return [record for record in caplog.records if record.name == name]
+
+
+class TestCallerInfo:
+ @pytest.fixture
+ def log_with_caller_info(self) -> Iterator[Log]:
+ """A fresh ``Log`` configured with caller info on, torn down so its handler leaves the root logger."""
+ from pipelex.system.configuration.config_loader import ConfigLoader # ruff: ignore[import-outside-top-level]
+ from pipelex.tools.misc.toml_utils import load_toml_from_path # ruff: ignore[import-outside-top-level]
+
+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")
+ log_config = LogConfig.model_validate(config_dict["runtime"]["log"]).model_copy(
+ update={"is_caller_info_enabled": True, "caller_info_template": CallerInfoTemplate.FUNC_MODULE_LINE},
+ )
+ fresh = Log()
+ fresh.configure(log_config=log_config)
+ try:
+ yield fresh
+ finally:
+ fresh.reset()
+
+ def test_caller_info_names_the_function_module_and_line_without_reading_source(
+ self,
+ log_with_caller_info: Log,
+ caplog: pytest.LogCaptureFixture,
+ mocker: MockerFixture,
+ ) -> None:
+ mocker.patch.object(inspect, "getframeinfo", side_effect=AssertionError("no source lookup"))
+ with caplog.at_level(logging.INFO):
+ log_with_caller_info.info("located")
+
+ (record,) = _own_records(caplog, name=__name__)
+ assert record.funcName == "test_caller_info_names_the_function_module_and_line_without_reading_source"
+ assert record.getMessage() == f"{record.funcName} {__name__} {record.lineno}: located"
diff --git a/tests/unit/pipelex/tools/test_log_caller_module.py b/tests/unit/pipelex/tools/test_log_caller_module.py
new file mode 100644
index 000000000..0e5f84186
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_caller_module.py
@@ -0,0 +1,82 @@
+"""The emitting module is derived from one frame lookup, and loggers are named by module.
+
+The old dispatch walked ``inspect.stack()`` and called ``inspect.getmodule`` on every frame for every
+line, which on a slow container filesystem cost more than a second per line. The module now comes
+from a frame at a fixed depth and the logger is named after it.
+"""
+
+from __future__ import annotations
+
+import inspect
+import logging
+import types
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from pipelex import log
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
+
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
+ import pytest
+ from pytest_mock import MockerFixture
+
+SYNTHETIC_MODULE = "synthetic_logpkg.sub.mod"
+
+
+def _emitter_in_module(*, module_name: str) -> Callable[[str], None]:
+ """A function whose globals say it lives in ``module_name``, so a log call from it is attributed there."""
+
+ def emit(message: str) -> None:
+ log.info(message)
+
+ return types.FunctionType(emit.__code__, {"__name__": module_name, "log": log}, "emit")
+
+
+def _own_records(caplog: pytest.LogCaptureFixture, *, name: str) -> list[logging.LogRecord]:
+ return [record for record in caplog.records if record.name == name]
+
+
+class TestLoggerNames:
+ def test_the_logger_is_named_after_the_emitting_module(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info("from the test module")
+
+ (record,) = _own_records(caplog, name=__name__)
+ assert record.name == __name__
+ assert Path(record.pathname).resolve() == Path(__file__).resolve()
+ assert record.funcName == "test_the_logger_is_named_after_the_emitting_module"
+ assert record.lineno > 0
+
+ def test_a_nested_module_gets_its_qualified_name(self, caplog: pytest.LogCaptureFixture) -> None:
+ emit = _emitter_in_module(module_name=SYNTHETIC_MODULE)
+ with caplog.at_level(logging.INFO):
+ emit("from a synthetic module")
+
+ (record,) = _own_records(caplog, name=SYNTHETIC_MODULE)
+ assert record.getMessage() == "from a synthetic module"
+
+ def test_no_stack_walk_happens_on_emission(self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture) -> None:
+ """Every level emits with ``inspect.stack`` and ``inspect.getmodule`` patched to raise."""
+
+ def refuse(*_args: object, **_kwargs: object) -> object:
+ msg = "the log dispatch must not walk the stack"
+ raise AssertionError(msg)
+
+ mocker.patch.object(inspect, "stack", refuse)
+ mocker.patch.object(inspect, "getmodule", refuse)
+
+ with caplog.at_level(LOGGING_LEVEL_VERBOSE):
+ log.verbose("verbose")
+ log.debug("debug")
+ log.dev("dev")
+ log.info("info")
+ log.warning("warning")
+ log.error("error")
+ log.critical("critical")
+ log.info({"structured": True})
+
+ messages = [record.getMessage() for record in _own_records(caplog, name=__name__)]
+ assert messages[:7] == ["verbose", "debug", "dev", "info", "warning", "error", "critical"]
+ assert len(messages) == 8
diff --git a/tests/unit/pipelex/tools/test_log_context.py b/tests/unit/pipelex/tools/test_log_context.py
new file mode 100644
index 000000000..ccc13b86e
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_context.py
@@ -0,0 +1,85 @@
+"""The run-scoped log context.
+
+``with log.context(...)`` binds the run's identifiers onto a task-local context: nested bindings merge,
+an absent identifier inherits rather than clears, the binding is released on exit however the block
+ends, and two concurrent tasks never see each other's.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any
+
+import pytest
+
+from pipelex import log
+from pipelex.tools.log.log_context import LogContext, get_log_context
+
+
+def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ """The records this test module emitted, whatever else the session's handlers saw."""
+ return [record for record in caplog.records if record.name == __name__]
+
+
+def _field(record: logging.LogRecord, *, name: str) -> Any:
+ """A field carried on the record: an attribute the stdlib does not declare, read the way a sink reads it."""
+ return getattr(record, name)
+
+
+class TestLogContext:
+ def test_context_yields_the_bound_context_and_restores_on_exit(self) -> None:
+ assert get_log_context() is None
+ with log.context(request_id="r1", pipeline_run_id="p1") as bound:
+ assert bound == LogContext(request_id="r1", pipeline_run_id="p1")
+ assert get_log_context() is bound
+ assert get_log_context() is None
+
+ def test_nested_contexts_merge_and_the_inner_overrides(self) -> None:
+ with log.context(request_id="r1", pipeline_run_id="p1"):
+ with log.context(pipeline_run_id="p2", pipe_run_id="pr1") as inner:
+ assert inner == LogContext(request_id="r1", pipeline_run_id="p2", pipe_run_id="pr1")
+ outer = get_log_context()
+ assert outer == LogContext(request_id="r1", pipeline_run_id="p1")
+
+ def test_a_none_identifier_inherits_rather_than_clears(self) -> None:
+ with log.context(request_id="r1"), log.context(request_id=None, pipe_run_id="pr1") as inner:
+ assert inner == LogContext(request_id="r1", pipe_run_id="pr1")
+
+ def test_context_is_restored_when_the_block_raises(self) -> None:
+ def explode() -> None:
+ msg = "boom"
+ raise RuntimeError(msg)
+
+ with pytest.raises(RuntimeError), log.context(request_id="r1"):
+ explode()
+ assert get_log_context() is None
+
+ def test_fields_of_a_context_omit_absent_identifiers(self) -> None:
+ assert LogContext().fields == {}
+ assert LogContext(pipeline_run_id="p1").fields == {"pipeline_run_id": "p1"}
+ assert LogContext(request_id="r1", pipeline_run_id="p1", pipe_run_id="pr1").fields == {
+ "request_id": "r1",
+ "pipeline_run_id": "p1",
+ "pipe_run_id": "pr1",
+ }
+
+ @pytest.mark.asyncio
+ async def test_context_is_task_local(self, caplog: pytest.LogCaptureFixture) -> None:
+ """Two concurrent tasks each see their own binding; neither leaks into the other or into the caller."""
+
+ async def emit(request_id: str) -> None:
+ with log.context(request_id=request_id):
+ await asyncio.sleep(0)
+ log.info("in task", fields={"tag": request_id})
+ await asyncio.sleep(0)
+ assert get_log_context() == LogContext(request_id=request_id)
+
+ with caplog.at_level(logging.INFO):
+ await asyncio.gather(emit("a"), emit("b"))
+
+ assert get_log_context() is None
+ records = _own_records(caplog)
+ assert len(records) == 2
+ for record in records:
+ assert _field(record, name="request_id") == _field(record, name="tag")
diff --git a/tests/unit/pipelex/tools/test_log_fields.py b/tests/unit/pipelex/tools/test_log_fields.py
new file mode 100644
index 000000000..eb2b6d05f
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_fields.py
@@ -0,0 +1,201 @@
+"""The fields channel of a log call.
+
+A log call takes named ``fields``, and every record emitted inside ``with log.context(...)`` carries the
+bound identifiers beside them. Both ride the stdlib ``LogRecord`` as attributes, never spliced into the
+message, so a structured sink renders them as fields while the console keeps its narrative line. A name
+the record already owns is carried under a prefix rather than raising.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+from pipelex import log
+from pipelex.tools.log.log_context import get_log_context
+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
+
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
+
+def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ """The records this test module emitted, whatever else the session's handlers saw."""
+ return [record for record in caplog.records if record.name == __name__]
+
+
+def _field(record: logging.LogRecord, *, name: str) -> Any:
+ """A field carried on the record: an attribute the stdlib does not declare, read the way a sink reads it."""
+ return getattr(record, name)
+
+
+class TestLogFields:
+ def test_fields_and_context_ride_the_record_as_attributes(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO), log.context(request_id="r1", pipeline_run_id="p1"):
+ log.info("scanned the inputs", fields={"files": 7})
+
+ (record,) = _own_records(caplog)
+ assert _field(record, name="files") == 7
+ assert _field(record, name="request_id") == "r1"
+ assert _field(record, name="pipeline_run_id") == "p1"
+ assert not hasattr(record, "pipe_run_id")
+ message = record.getMessage()
+ assert message == "scanned the inputs"
+ assert "r1" not in message
+ assert "p1" not in message
+ assert "7" not in message
+
+ def test_a_record_outside_any_context_carries_no_identifier(self, caplog: pytest.LogCaptureFixture) -> None:
+ assert get_log_context() is None
+ with caplog.at_level(logging.INFO):
+ log.info("outside", fields={"files": 7})
+
+ (record,) = _own_records(caplog)
+ assert _field(record, name="files") == 7
+ assert not hasattr(record, "request_id")
+ assert not hasattr(record, "pipeline_run_id")
+ assert not hasattr(record, "pipe_run_id")
+ assert "None" not in record.getMessage()
+
+ @pytest.mark.parametrize(
+ ("level", "method_name"),
+ [
+ (LOGGING_LEVEL_VERBOSE, "verbose"),
+ (logging.DEBUG, "debug"),
+ (15, "dev"),
+ (logging.INFO, "info"),
+ (logging.WARNING, "warning"),
+ (logging.ERROR, "error"),
+ (logging.CRITICAL, "critical"),
+ ],
+ )
+ def test_every_level_takes_fields(self, caplog: pytest.LogCaptureFixture, level: int, method_name: str) -> None:
+ method: Callable[..., None] = getattr(log, method_name)
+ with caplog.at_level(LOGGING_LEVEL_VERBOSE), log.context(pipe_run_id="pr1"):
+ method("at every level", fields={"attempt": 2})
+
+ (record,) = _own_records(caplog)
+ assert record.levelno == level
+ assert _field(record, name="attempt") == 2
+ assert _field(record, name="pipe_run_id") == "pr1"
+ assert record.getMessage() == "at every level"
+
+ def test_title_and_inline_keep_their_meaning_beside_fields(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info("body", title="Heading", fields={"k": "v"})
+ log.info("body", inline="Tag", fields={"k": "v"})
+
+ titled, inlined = _own_records(caplog)
+ assert titled.getMessage() == "Heading:\nbody"
+ assert inlined.getMessage() == "Tag: body"
+ assert _field(titled, name="k") == "v"
+ assert _field(inlined, name="k") == "v"
+
+ def test_a_field_named_like_a_stdlib_record_attribute_is_prefixed_not_raised(self, caplog: pytest.LogCaptureFixture) -> None:
+ """The stdlib raises ``KeyError`` when ``extra`` overwrites a record attribute; a log call never raises."""
+ with caplog.at_level(logging.INFO):
+ log.info("collision", fields={"name": "not-the-logger", "message": "not-the-message", "lineno": -1, "safe": 1})
+
+ (record,) = _own_records(caplog)
+ assert record.name == __name__
+ assert record.getMessage() == "collision"
+ assert record.lineno > 0
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}name") == "not-the-logger"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}message") == "not-the-message"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}lineno") == -1
+ assert _field(record, name="safe") == 1
+
+ def test_an_attribute_a_record_factory_added_is_a_collision_too(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A record factory (an OpenTelemetry or tracing instrumentation) stamps attributes the stdlib never declared.
+
+ The collision is read off the record actually built, so a field, a context identifier or the
+ ``data`` attribute named like one is prefixed rather than raising ``KeyError`` out of the log call.
+ """
+ previous_factory = logging.getLogRecordFactory()
+
+ def stamping_factory(*args: Any, **kwargs: Any) -> logging.LogRecord:
+ record = previous_factory(*args, **kwargs)
+ record.otelTraceID = "trace-from-factory"
+ record.request_id = "request-from-factory"
+ record.data = "data-from-factory"
+ return record
+
+ logging.setLogRecordFactory(stamping_factory)
+ try:
+ with caplog.at_level(logging.INFO), log.context(request_id="r1"):
+ log.info({"key": "value"}, fields={"otelTraceID": "trace-from-call"})
+ finally:
+ logging.setLogRecordFactory(previous_factory)
+
+ (record,) = _own_records(caplog)
+ assert _field(record, name="otelTraceID") == "trace-from-factory"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}otelTraceID") == "trace-from-call"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}request_id") == "r1"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{DATA_FIELD}") == {"key": "value"}
+
+ @pytest.mark.parametrize(
+ ("fields", "expected"),
+ [
+ pytest.param(
+ {"name": "alpha", "field_name": "beta"},
+ {f"{COLLIDING_FIELD_PREFIX}name": "alpha", f"{COLLIDING_FIELD_PREFIX}{COLLIDING_FIELD_PREFIX}name": "beta"},
+ id="colliding-name-first",
+ ),
+ pytest.param(
+ {"field_name": "beta", "name": "alpha"},
+ {f"{COLLIDING_FIELD_PREFIX}name": "beta", f"{COLLIDING_FIELD_PREFIX}{COLLIDING_FIELD_PREFIX}name": "alpha"},
+ id="prefixed-name-first",
+ ),
+ pytest.param(
+ {"field_message": "given", "message": "colliding"},
+ {f"{COLLIDING_FIELD_PREFIX}message": "given", f"{COLLIDING_FIELD_PREFIX}{COLLIDING_FIELD_PREFIX}message": "colliding"},
+ id="formatter-owned-prefixed-first",
+ ),
+ ],
+ )
+ def test_a_prefixed_name_that_is_itself_given_loses_no_value(
+ self, caplog: pytest.LogCaptureFixture, fields: dict[str, str], expected: dict[str, str]
+ ) -> None:
+ """Whichever of ``name`` and ``field_name`` arrives first keeps ``field_name``; the other lands on ``field_field_name``."""
+ with caplog.at_level(logging.INFO):
+ log.info("both given", fields=fields)
+
+ (record,) = _own_records(caplog)
+ assert record.name == __name__
+ assert record.getMessage() == "both given"
+ for attribute, value in expected.items():
+ assert getattr(record, attribute) == value
+
+ def test_a_factory_owning_the_prefixed_name_too_loses_neither_value(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A factory that stamps both ``request_id`` and ``field_request_id`` keeps both; the bound identifier lands one prefix further."""
+ previous_factory = logging.getLogRecordFactory()
+
+ def stamping_factory(*args: Any, **kwargs: Any) -> logging.LogRecord:
+ record = previous_factory(*args, **kwargs)
+ record.request_id = "request-from-factory"
+ record.field_request_id = "prefixed-from-factory"
+ return record
+
+ logging.setLogRecordFactory(stamping_factory)
+ try:
+ with caplog.at_level(logging.INFO), log.context(request_id="r1"):
+ log.info("bound under a factory")
+ finally:
+ logging.setLogRecordFactory(previous_factory)
+
+ (record,) = _own_records(caplog)
+ assert _field(record, name="request_id") == "request-from-factory"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}request_id") == "prefixed-from-factory"
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{COLLIDING_FIELD_PREFIX}request_id") == "r1"
+
+ def test_a_field_overrides_the_context_for_that_record(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO), log.context(request_id="from-context"):
+ log.info("about another request", fields={"request_id": "from-call-site"})
+ log.info("about this request")
+
+ override, plain = _own_records(caplog)
+ assert _field(override, name="request_id") == "from-call-site"
+ assert _field(plain, name="request_id") == "from-context"
diff --git a/tests/unit/pipelex/tools/test_log_package_levels.py b/tests/unit/pipelex/tools/test_log_package_levels.py
new file mode 100644
index 000000000..47d790087
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_package_levels.py
@@ -0,0 +1,84 @@
+"""A ``package_log_levels`` key is honoured through the stdlib logger hierarchy.
+
+Loggers are named by module, so a package key governs every module under it and a module key,
+spelled with dashes in the configuration, is mapped to the dotted logger name.
+"""
+
+from __future__ import annotations
+
+import logging
+import types
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex import log
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE, LogLevel
+
+if TYPE_CHECKING:
+ from collections.abc import Callable, Iterator
+
+SYNTHETIC_PACKAGE = "synthetic_logpkg"
+SYNTHETIC_MODULE = f"{SYNTHETIC_PACKAGE}.sub.mod"
+SYNTHETIC_SIBLING = f"{SYNTHETIC_PACKAGE}.other"
+
+
+def _emitter_in_module(*, module_name: str) -> Callable[[str], None]:
+ """A function whose globals say it lives in ``module_name``, so a log call from it is attributed there."""
+
+ def emit(message: str) -> None:
+ log.info(message)
+
+ return types.FunctionType(emit.__code__, {"__name__": module_name, "log": log}, "emit")
+
+
+def _own_records(caplog: pytest.LogCaptureFixture, *, name: str) -> list[logging.LogRecord]:
+ return [record for record in caplog.records if record.name == name]
+
+
+def clean_synthetic_levels() -> Iterator[None]:
+ """Whatever level a test sets on the synthetic loggers is undone afterwards."""
+ yield
+ for logger_name in (SYNTHETIC_PACKAGE, SYNTHETIC_MODULE, SYNTHETIC_SIBLING, __name__):
+ logging.getLogger(logger_name).setLevel(logging.NOTSET)
+
+
[email protected]("clean_synthetic_levels")
+class TestPackageLogLevels:
+ def test_a_package_level_key_still_governs_its_modules(self, caplog: pytest.LogCaptureFixture) -> None:
+ log.set_levels_for_packages({SYNTHETIC_PACKAGE: LogLevel.WARNING})
+ emit = _emitter_in_module(module_name=SYNTHETIC_MODULE)
+
+ with caplog.at_level(LOGGING_LEVEL_VERBOSE):
+ emit("dropped by the package level")
+
+ assert _own_records(caplog, name=SYNTHETIC_MODULE) == []
+
+ def test_a_module_level_key_is_honoured_through_the_dash_to_dot_mapping(self, caplog: pytest.LogCaptureFixture) -> None:
+ log.set_levels_for_packages(
+ {
+ SYNTHETIC_PACKAGE: LogLevel.WARNING,
+ SYNTHETIC_MODULE.replace(".", "-"): LogLevel.DEBUG,
+ }
+ )
+ emit_module = _emitter_in_module(module_name=SYNTHETIC_MODULE)
+ emit_sibling = _emitter_in_module(module_name=SYNTHETIC_SIBLING)
+
+ with caplog.at_level(LOGGING_LEVEL_VERBOSE):
+ emit_module("kept by the module level")
+ emit_sibling("dropped by the package level")
+
+ (kept,) = _own_records(caplog, name=SYNTHETIC_MODULE)
+ assert kept.getMessage() == "kept by the module level"
+ assert _own_records(caplog, name=SYNTHETIC_SIBLING) == []
+
+ def test_a_module_level_key_on_the_emitting_test_module(self, caplog: pytest.LogCaptureFixture) -> None:
+ log.set_levels_for_packages({__name__.replace(".", "-"): LogLevel.WARNING})
+
+ with caplog.at_level(LOGGING_LEVEL_VERBOSE):
+ log.info("dropped")
+ log.warning("kept")
+
+ messages = [record.getMessage() for record in _own_records(caplog, name=__name__)]
+ assert messages == ["kept"]
diff --git a/tests/unit/pipelex/tools/test_log_structured_content.py b/tests/unit/pipelex/tools/test_log_structured_content.py
new file mode 100644
index 000000000..d2736a8e5
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_structured_content.py
@@ -0,0 +1,107 @@
+"""Structured content on a log call.
+
+A ``dict``, a ``list`` or a model given as the content is rendered as JSON for the console and carried
+as the record's ``data`` attribute for a structured sink: a JSON-ready snapshot of the call, never the
+caller's live object.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+from datetime import UTC, datetime
+from decimal import Decimal
+from typing import TYPE_CHECKING, Any
+from uuid import UUID
+
+from pydantic import BaseModel
+
+from pipelex import log
+from pipelex.tools.log.log_fields import DATA_FIELD
+
+if TYPE_CHECKING:
+ import pytest
+
+
+def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:
+ """The records this test module emitted, whatever else the session's handlers saw."""
+ return [record for record in caplog.records if record.name == __name__]
+
+
+class TestStructuredContent:
+ def test_dict_content_is_carried_as_data_and_rendered_for_the_console(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info({"key": "value", "nested": {"flag": True}}, title="Config")
+
+ (record,) = _own_records(caplog)
+ assert getattr(record, DATA_FIELD) == {"key": "value", "nested": {"flag": True}}
+ rendered = record.getMessage()
+ assert rendered.startswith("Config:")
+ assert '"key": "value"' in rendered
+
+ def test_list_content_is_carried_as_data(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info([1, "two", {"three": 3}])
+
+ (record,) = _own_records(caplog)
+ assert getattr(record, DATA_FIELD) == [1, "two", {"three": 3}]
+
+ def test_data_is_a_snapshot_of_the_call_not_the_callers_live_object(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A sink that serializes later, on a queue or a batching exporter, reads what the call said, not what the caller did since."""
+ counters: dict[str, Any] = {"seen": 1, "inner": {"flag": False}}
+ items: list[Any] = [{"n": 1}]
+ with caplog.at_level(logging.INFO):
+ log.info(counters)
+ log.info(items)
+ counters["seen"] = 999
+ counters["inner"]["flag"] = True
+ items[0]["n"] = 999
+
+ dict_record, list_record = _own_records(caplog)
+ assert getattr(dict_record, DATA_FIELD) is not counters
+ assert getattr(dict_record, DATA_FIELD) == {"seen": 1, "inner": {"flag": False}}
+ assert getattr(list_record, DATA_FIELD) is not items
+ assert getattr(list_record, DATA_FIELD) == [{"n": 1}]
+ assert json.loads(dict_record.getMessage()) == getattr(dict_record, DATA_FIELD)
+
+ def test_a_list_of_models_is_carried_as_json_ready_data(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A python-mode model dump keeps datetimes and the like as objects; ``data`` is what the console rendered, re-read as JSON."""
+
+ class Event(BaseModel):
+ when: datetime
+ price: Decimal
+ ref: UUID
+
+ event = Event(when=datetime(2020, 1, 2, 3, 4, 5, tzinfo=UTC), price=Decimal("1.50"), ref=UUID(int=7))
+ with caplog.at_level(logging.INFO):
+ log.info([event])
+
+ (record,) = _own_records(caplog)
+ data = getattr(record, DATA_FIELD)
+ assert json.loads(json.dumps(data)) == data
+ assert data == json.loads(record.getMessage())
+ assert data[0]["when"] == "2020-01-02 03:04:05+00:00"
+ assert data[0]["price"] == "1.50"
+ assert data[0]["ref"] == str(UUID(int=7))
+
+ def test_string_content_carries_no_data(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info("plain")
+
+ (record,) = _own_records(caplog)
+ assert not hasattr(record, DATA_FIELD)
+
+ def test_structured_content_wins_over_a_data_field(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info({"from": "content"}, fields={DATA_FIELD: "from-field"})
+
+ (record,) = _own_records(caplog)
+ assert getattr(record, DATA_FIELD) == {"from": "content"}
+
+ def test_none_content_is_rendered_as_the_word_none_without_data(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info(None, title="Empty")
+
+ (record,) = _own_records(caplog)
+ assert record.getMessage() == "Empty:\nNone"
+ assert not hasattr(record, DATA_FIELD)
diff --git a/wip/structured-logs/review-deferrals.md b/wip/structured-logs/review-deferrals.md
new file mode 100644
index 000000000..14d20b025
--- /dev/null
+++ b/wip/structured-logs/review-deferrals.md
@@ -0,0 +1,40 @@
+# Structured logs: findings deferred by review passes
+
+Traces for what a review pass on a structured-logs member read as real but not worth fixing at that pass's bar. The sprint is L-260916-a8cdd2; the campaign documents live at the workspace root under `wip/structured-logs/`. An entry marked *unverified* rests on the reviewer's word alone: nobody reproduced it.
+
+## L-260916-a1dbd5, round 1 (fields, context, no stack walk)
+
+- **Caller attribution rides a fixed frame depth** (*unverified*, cubic). `CALLER_FRAME_DEPTH` in `pipelex/tools/log/log_dispatch.py` counts the frames between the frame lookup and the code that called `log.<level>(...)`, so a helper inserted into or removed from that call chain would attribute records to the wrong module and apply the wrong logger level. The tests in `tests/unit/pipelex/tools/test_log_caller_module.py` emit through every level and assert the caller's module, path and function, which is what catches such a change. A walk up the frames until one leaves the log package would remove the constant at the cost of a loop per line; the fixed depth stays until a second facade entry point makes the constant wrong for one of them.
+- **A given `field_<name>` beside a colliding `<name>`** (*unverified*, cubic). Prefixing a colliding entry could overwrite an entry the caller gave under the prefixed name. Deferred here on the reading that the ordered attach kept both values; round 2 verified that it did so in one order only, and fixed it: the prefix is now applied until the destination is free.
+- **The traceback rides the message rather than `exc_info`** (verified by inspection, code-review and the implementer). `include_exception` appends `traceback.format_exc()` to the message text in the dispatch. A structured sink wants the exception as `exc_info` or `exc_text`; setting it today would make the Rich handler render the traceback twice. The sink member L-260916-aa6e58 owns that decision, and the note on it carries the detail.
+
+## L-260916-a1dbd5, round 2 (bar defects)
+
+- **`is_console_logging_enabled = false` silences every handler, not the console** (*unverified*, code-review). The early return in `LogDispatch.dispatch` precedes `logger.handle(record)`, so a deployment that turns the console off to attach its own handler to the root logger gets no Pipelex record at all. The gate is the one the old console-only dispatch had, and it is the sink member's to replace: L-260916-aa6e58 selects sinks by configuration and retires the rich-or-poor mode, and the note on it carries this.
+- **The formatter mutates `record.msg` in place** (*unverified*, code-review, outside the diff). `LevelAndEmojiLogFormatter.format` in `pipelex/tools/log/log_formatter.py` rewrites the record's message for WARNING and above, so a record formatted by more than one handler accumulates colour escapes. Pre-existing; it starts to matter when a second handler exists, which is the sink member's work, and the note on L-260916-aa6e58 carries it.
+- **A `data` field is overwritten by structured content rather than prefixed** (*unverified*, code-review). `build_log_record_extra` lets the content's `data` win over `fields={"data": ...}` outright, the one place a value is discarded rather than carried under a prefix. Documented as deliberate in `docs/tools/logging.md`; no call site gives such a field, and a rule that prefixes it would be a change of contract rather than a fix.
+
+## L-260916-a1dbd5, round 3 (bar necessity)
+
+The bar admitted only what the round-2 pass itself introduced or what the branch could not ship without. The three entries marked *verified* were confirmed by reproduction and deferred because they predate that pass, no runtime caller reaches them, and the sink member L-260916-aa6e58 owns the same modules before the sprint's release; the note on that item carries each with its fix.
+
+- **The collision guard reads only the record's instance dict** (*verified*, codex:review and codex:adversarial). `attach_log_record_extra` in `pipelex/tools/log/log_fields.py` tests `name in vars(record)`, which misses the attributes the `LogRecord` class owns: `fields={"__class__": ...}` and `{"__dict__": ...}` raise `TypeError` out of the log call, and `{"getMessage": ...}` shadows the method so the formatter fails and the stdlib's `handleError` drops the record. The fix is `hasattr(record, attribute)` in place of the `vars` lookup, keeping the `FORMATTER_OWNED_ATTRIBUTES` clause since `message` is not on a fresh record, with a regression test per key. No caller in `pipelex/` passes `fields=` yet; the exposure is a consumer forwarding a mapping whose keys it does not control, a kajson-encoded object among them.
+- **A circular-reference content raises `ValueError` out of the facade** (*verified*, code-review). `_render_content` hands a `dict` or `list` to the `purify_json*` helpers, which catch `TypeError` only, and `json.dumps` raises `ValueError` on a cycle. Byte-identical on dev, where the raise is unconditional; the branch's docstring in `log_fields.py` is what promises a log call never raises. The fix is a guard on the log path, a `ValueError` caught beside the `TypeError` in the helpers or a fallback rendering in `_render_content`, and the promise narrowed if neither lands.
+- **The changelog bullet is four sentences against the rule's three** (*verified*, code-review). The closing cross-reference is its own sentence since the branch's first commit; it folds into the previous one when the bullet is next touched.
+- **A process entry cannot clear an inherited identifier** (*unverified*, cubic and code-review). `LogContext.merged_with` reads `None` as inherit, so `PipeRun.run` invoked inside an existing binding keeps the ambient `pipe_run_id` until `live_run_pipe` mints one; `request_id` and `pipeline_run_id` are always set by a run and cannot leak. No call site runs `PipeRun.run` inside a binding today; a replace-or-clear semantics on `bind_log_context` is the fix if an embedder or a nested run ever does.
+- **`fields` values ride by reference while `data` is a snapshot** (*unverified*, code-review). A field given a live `dict` or `list` reads whatever the caller did to it by the time a queued sink serializes it. `docs/tools/logging.md` advises JSON-ready values as guidance rather than guarantee; whether the sink snapshots on emit or the contract states the requirement is the sink member's call.
+- **`data` is re-parsed on every structured call before any sink reads it** (*unverified*, code-review). `json.loads(rendered)` in `_render_content` builds a second object graph per record at an enabled level; a lazy attribute or a sink-side parse is the sink member's design choice.
+- **The reserved-name comment claims `ErrorReport` carries the identifiers** (*unverified*, code-review). The comment above the constants in `pipelex/tools/log/log_context.py` says a record and the `ErrorReport` agree on the key; `ErrorReport` carries none of the three, and the only `request_id` reachable from it is the provider's, a different identifier. The comment should name the run result and the payload fields alone.
+- **The deleted gateway tests were the only mechanical guard on the no-log-on-load-path invariant** (*unverified*, code-review). Round 1 removed two tests that asserted the absence of a `RuntimeError` that no longer exists; the invariant they stood beside, that nothing logs on the gateway load path because a pre-configure line lands on stderr unformatted, is now prose only in `gateway_config.py`. A test that asserts no record is emitted on that path restores the guard.
+- **Nits** (*unverified*, code-review): `pipe_run.py` still interpolates `pipeline_run_id` into five message strings while the same value rides the record as a field; `test_log_fields.py` hardcodes the DEV level as a number beside an imported constant; `bedrock_client_boto3.py` uses `run_in_executor`, which does not propagate contextvars, so a log call from that thread would lose the identifiers, though only a boto3 call runs there.
+
+## L-260916-a1dbd5, round 4 (bar freeze)
+
+The bar changed no code. Every reviewer raised the class-owned attribute collision again, and code-review the circular-reference raise, the console flag and the round-3 nits; those keep their entries above. What follows is new.
+
+- **A filter attached to the `pipelex` logger no longer sees the records** (*unverified*, cubic). Python consults a logger's filters only for records originating on that logger, never while a descendant's record propagates, so a filter an integration attached to `logging.getLogger("pipelex")` stops running now that records originate on module loggers. No repository in the workspace attaches one. The changelog's "Loggers are named by module" bullet says a filter keyed on the bare package name must key on the prefix, which does not help a filter attached to the logger: such a filter must move to the handler, and the bullet and `docs/tools/logging.md` should say so.
+- **A dict with a non-string key raises `TypeError` out of the facade** (*unverified*, code-review). `json.dumps` applies its `default` to values, not keys, so `log.info({(1, 2): "a"})` raises from the `purify_json*` fallback too. Byte-identical on dev; the same guard as the circular-reference entry above covers it.
+- **The logging doc overstates what `data` is** (*unverified*, code-review). The "Structured content" section says any other object is carried as the dictionary or list its serialization produces; a scalar content lands as itself, an object that defeats both `json` and `kajson` lands as the `purify_json` warning wrapper, and a `float("nan")` survives the `json.loads` round trip as a float standard JSON cannot re-emit. A sink author iterating `record.data` meets a scalar. The sink member decides the shape and the page follows.
+- **The caller-info test fixture's teardown is process-global** (*unverified*, code-review). `log_with_caller_info` in `tests/unit/pipelex/tools/test_log_caller_info.py` calls `Log.reset()`, which removes and closes every root-logger handler and calls `logging.shutdown()`. `Log.reset()` is pre-existing; the unit suite is green under random and fixed ordering, so this is a latent isolation hazard rather than a failure. A fixture that removes only the handler it added would remove the hazard.
+- **The dev-side caller-info fix is filed under Changed** (*unverified*, code-review). `FUNC_MODULE` and `FUNC_MODULE_LINE` raised `KeyError: 'module'` on dev at the first log line after boot; the branch passes the module and covers it in `test_log_caller_info.py`, and the changelog folds that into the logger-naming bullet rather than a Fixed entry.
+- **Out of scope, unverified: seven findings on the `main..dev` delta** (code-review, whose skill diffed `main...HEAD`). Files this branch does not touch, listed here so they can be chased and never as this branch's: `pipelex/pipeline/input_normalizer.py:232` (`HttpUrl`'s length cap rejects long signed URLs); `pipelex/tracing/ndjson_event_log.py:163` and `pipelex/tracing/dynamodb_event_log.py:170` (corrupt-vs-refused decided without checking `loc == ()`); `pipelex/tracing/ndjson_event_log.py:164` (raw `validation_error` interpolation leaks traced content); `pipelex/cli/commands/graph_cmd.py:59` (`json_invalid` misreported as a stale-file error); `pipelex/graph/mermaidflow/templates/_interactive_scripts.js.jinja2:275` (the JSON tab falls back without consulting `disabled`); `pipelex/tools/misc/http_utils.py:22` (whitespace accepted against the documented rule); `pipelex/runtime_hub.py:210` (`reset_boot_state` reverts the agent CLI's SILENT printer).