Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Log-sink-seam · L-260916-aa6e58 — sink registry, json otlp console (#1206)
Louis Choquelfc20e0ea06ba
diff --git a/.drift/acks/cli-docs.toml b/.drift/acks/cli-docs.tomlindex d41ebf6c9..f4686d8a3 100644--- a/.drift/acks/cli-docs.toml+++ b/.drift/acks/cli-docs.toml@@ -1,8 +1,8 @@contract = "cli-docs"-digest = "sha256:3bbf5f40906a0b8af13238f3f78a01fdeb90eed363620f7f881dfd7ec46b59c0"-reviewed_by = "Louis Choquel"-reviewed_at = "2026-09-20T20:19:19Z"-rationale = "Reviewed docs/tools/cli/ and pipelex/cli/agent_cli/CLAUDE.md against the trigger. The only change to check_model_cmd.py is the argument-help example, which moves from the retired @best-claude alias to @best-gpt now that the shipped deck is Azure-only. Neither review target quotes that example or any deck alias: agent-cli.md and the agent-CLI contract doc describe check-model's flags and output modes generically, and update.md describes the managed/override file split without naming a single alias or model handle. No doc change needed."+digest = "sha256:28dfaac2f38ffb7bebd4e7aaf1e872dcf8a1ff0401e91b846006b6a001a28d89"+reviewed_by = "Claude (Opus 5.5)"+reviewed_at = "2026-09-25T12:38:13Z"+rationale = "Merge of origin/feature/Log-fields-context (carrying dev's releases v0.61.0 through v0.65.0) into feature/Log-sink-seam: the merged trigger set is the union of two changes each acked on its own side, with no trigger file changed on both. This branch's side (agent_cli_factory.py and both doctor_cmd.py) is covered by the checks.log_sink row in pipelex/cli/agent_cli/CLAUDE.md, present in the merged tree. The parent's side is check_model_cmd.py's argument-help example moving from the retired @best-claude alias to @best-gpt; neither docs/tools/cli/ nor the agent-CLI contract doc quotes that example or any deck alias (grep for best-claude, best-gemini and best-mistral finds nothing in either target). The ack file's line-wise auto-merge kept each side's trigger snapshot, so this re-ack records the merged tree as reviewed as a whole. No doc change needed."[trigger_files]"pipelex/cli/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"@@ -11,7 +11,7 @@ rationale = "Reviewed docs/tools/cli/ and pipelex/cli/agent_cli/CLAUDE.md agains"pipelex/cli/agent_cli/_agent_cli.py" = "blob:e49ea3d36aa0e964e6d2cc66e901f8362c9ae76b""pipelex/cli/agent_cli/commands/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/agent_cli/commands/accept_gateway_terms_cmd.py" = "blob:cb8338c09a7b5dee6088551c156d69dab613213a"-"pipelex/cli/agent_cli/commands/agent_cli_factory.py" = "blob:ba63ff317613e9be35677fca949f0991a6f6c47e"+"pipelex/cli/agent_cli/commands/agent_cli_factory.py" = "blob:49b8bab889d8673e7d4f1d683efe0928e43914b9""pipelex/cli/agent_cli/commands/agent_output.py" = "blob:3dc0283ef1daee4e17183b01f8a5c9d92f2688af""pipelex/cli/agent_cli/commands/bundle_path_resolver.py" = "blob:30339c5d91048f4e67312d5691ed9a65d57472e0""pipelex/cli/agent_cli/commands/check_model_cmd.py" = "blob:91589e517509248a4639a37e82f46a69538f8071"@@ -20,7 +20,7 @@ rationale = "Reviewed docs/tools/cli/ and pipelex/cli/agent_cli/CLAUDE.md agains"pipelex/cli/agent_cli/commands/codegen/check_cmd.py" = "blob:9e8beb084a9f04f39e8acca9e6ae2554d40cb667""pipelex/cli/agent_cli/commands/codegen/types_cmd.py" = "blob:dea775a8a469816f2c84ea4f6a5c9234c4af6705""pipelex/cli/agent_cli/commands/concept_cmd.py" = "blob:8e5c1881805cda39a1dae06be247a4fc257ccf69"-"pipelex/cli/agent_cli/commands/doctor_cmd.py" = "blob:d19554769b75dd2f096ccadb5e2cedc3acb86925"+"pipelex/cli/agent_cli/commands/doctor_cmd.py" = "blob:33d142c3ecdeb10c4ac83bd43f4949afcdd1d455""pipelex/cli/agent_cli/commands/fix/__init__.py" = "blob:65662c86f17f610ad55ecfc3e27a022b9aa26e8b""pipelex/cli/agent_cli/commands/fix/app.py" = "blob:e6190d9bde6a05755c04bd21806f699461038ac0""pipelex/cli/agent_cli/commands/fix/bundle_cmd.py" = "blob:37b13a1ead7b55d601f53b23647c285fc85418cd"@@ -82,7 +82,7 @@ rationale = "Reviewed docs/tools/cli/ and pipelex/cli/agent_cli/CLAUDE.md agains"pipelex/cli/commands/codegen/inputs_cmd.py" = "blob:20738a9e42f0e5ef7ffc489093348af3cb4643a0""pipelex/cli/commands/codegen/types_cmd.py" = "blob:b7c9c7a3664c9fbe66f9b4c3c5ebf9ea1322445c""pipelex/cli/commands/crate_loading.py" = "blob:3d7d651931ecf1a15a3e2d72411b3212b55d74ef"-"pipelex/cli/commands/doctor_cmd.py" = "blob:d800566d0f69b766131baa0f64ee9f9fbb41cf57"+"pipelex/cli/commands/doctor_cmd.py" = "blob:7ea6e1b0f260a0de8cceae4e5c2deede7b0abbd7""pipelex/cli/commands/fix/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/commands/fix/_diff_sandbox.py" = "blob:5ce264f5b0ab4ff4172f4d8789450916947f5c0e""pipelex/cli/commands/fix/_fix_core.py" = "blob:8f6a187a0fb42c498629b177753d81620800b0da"diff --git a/.drift/acks/config-docs.toml b/.drift/acks/config-docs.tomlindex 899411a4f..1e9468137 100644--- a/.drift/acks/config-docs.toml+++ b/.drift/acks/config-docs.toml@@ -1,13 +1,13 @@contract = "config-docs"-digest = "sha256:7855e5a0a09388da54a6365f26e2f0febef02310b309b613a562a969748d7f2e"+digest = "sha256:6c7a74ebfa3059f589421b8f984df95feb1aabc705950eb48b0ff2322e4141c6"reviewed_by = "Louis Choquel"-reviewed_at = "2026-09-16T14:58:00Z"-rationale = "Docstring-only wording fix in ConfigLoader.take_stale_configuration_warning: the review pass found the earlier text claimed a pre-configure warning misses the console, while it lands on stderr unformatted through logging.lastResort, and the docstring now says so. No config model, key, default or validator moved, so the migration ledgers owe nothing and docs/configuration stays as the previous ack left it."+reviewed_at = "2026-09-16T18:14:03Z"+rationale = "Log sinks replace the log mode: log_config.py loses LogMode, log_mode, poor_loggers, generic_poor_logger and is_console_logging_enabled and gains the sink key and the [runtime.log.otlp] table (endpoint, headers); pipelex.toml ships sink = console and the otlp table. docs/configuration/config-practical/logging-config.md is rewritten around the sink key with a section per shipped sink and a migration paragraph, and migration ledger entry pipelex-config@5 deletes the four retired keys with the goldens regenerated by make umig. No validator moved."[trigger_files]"pipelex/cogt/config_cogt.py" = "blob:02f71cb4a6b01aebda59c2e31f262ef9c865e17b""pipelex/cogt/img_gen/img_gen_job_components.py" = "blob:084985080413fd8d6c3b1b9b0747020cb0161858"-"pipelex/pipelex.toml" = "blob:a4197193c4e9df0f09ef795149a505c0a149fdda"+"pipelex/pipelex.toml" = "blob:3fb0bd5076a7d2993502159a121a69fe03210086""pipelex/system/configuration/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc""pipelex/system/configuration/config_check.py" = "blob:88e15a92b186088a5ea1c7c4ebcedac830707d3a""pipelex/system/configuration/config_loader.py" = "blob:ac384ce993933656682999c26afffb4c9f9b1975"@@ -18,4 +18,4 @@ rationale = "Docstring-only wording fix in ConfigLoader.take_stale_configuration"pipelex/system/configuration/pipe_func_config.py" = "blob:dc8890e914fbcdb611147e42eaa0cd87b32288a5""pipelex/system/telemetry/telemetry_config.py" = "blob:7070951a6d3e4c10cb7d59822ce168c6590c53be""pipelex/tools/aws/aws_config.py" = "blob:c1aeafeae08d3921464db33795f11a29ff3892ba"-"pipelex/tools/log/log_config.py" = "blob:288f992a399a106c0aa1129c85aae048e402cb6d"+"pipelex/tools/log/log_config.py" = "blob:a32629f52056364493e96f071e7b6814cefaeeb9"diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex aa4fbe29e..dd73e4c41 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -96,6 +96,10 @@ console_log_target = "stderr"# targets to stderr internally to keep its stdout strictly reserved for the success# envelope.console_print_target = "stdout"+# sink names where the records go: "console" (the Rich handler, for a terminal), "json" (one JSON+# object per line, for a server behind a log agent), "otlp" (the OpenTelemetry logs signal, for a+# collector), or a token an installed plugin registers.+sink = "console"[runtime.log.package_log_levels]# Log levels for specific packages (use "-" instead of "." in package names)diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a322565af..4b552b0fa 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,18 +4,22 @@### Added-- **Named fields and a run-scoped context on the log calls**: the seven `log.*` methods take a keyword-only `fields={...}`, carried as attributes of the stdlib `LogRecord` and never rendered into the message, and `with log.context(request_id=..., pipeline_run_id=..., pipe_run_id=...)` stamps the run's identifiers onto every record emitted in scope, nested bindings merging and an absent identifier staying absent rather than reading `None`. `PipeRun.run` binds the request and pipeline identifiers from the job's metadata for a direct-mode run, and every pipe binds its own `pipe_run_id` around the whole of its run, its announcement and its failure included. A `dict` or `list` content is carried as the record's `data` attribute beside its console rendering, a JSON-ready snapshot of the call rather than the caller's live object, and an entry named like an attribute the record already owns, a record factory's included, is carried under a `field_` prefix instead of raising, whatever the order the names arrive in. See [Logging](tools/logging.md).+- **Named fields and a run-scoped context on the log calls**: the seven `log.*` methods take a keyword-only `fields={...}`, carried as attributes of the stdlib `LogRecord` and never rendered into the message, and `with log.context(request_id=..., pipeline_run_id=..., pipe_run_id=...)` stamps the run's identifiers onto every record emitted in scope, nested bindings merging and an absent identifier staying absent rather than reading `None`. `PipeRun.run` binds the request and pipeline identifiers from the job's metadata for a direct-mode run, and every pipe binds its own `pipe_run_id` around the whole of its run, its announcement and its failure included. A `dict` or `list` content is carried as the record's `data` attribute beside its console rendering, a JSON-ready snapshot of the call rather than the caller's live object, and an entry named like an attribute the record already owns, a record factory's included, or like one of the names set after the fields are attached — the formatter's, and the internal marker that tells a sink a record reached it already — is carried under a `field_` prefix instead of raising or steering delivery, whatever the order the names arrive in — a field spelling `data` beside structured content included, so the content keeps the name a sink reads it under and the field keeps its value beside it, as [Logging](tools/logging.md) describes.+- **Log sinks are a plugin capability selected by `[runtime.log] sink` (Breaking)**: where the records go is a registered sink token, `console` (the Rich handler, the default so the CLI keeps its rendering), `json` (one JSON object per line carrying `time`, `severity`, `logger`, `message`, the fields, the context identifiers and the exception, with no ANSI ever, for a server behind a log agent) or `otlp` (the OpenTelemetry logs signal, configured under `[runtime.log.otlp]`, so a collector receives the logs beside the spans), each of the two reserving the keys it writes itself so that a field named like one of them rides under a `field_` prefix rather than overwriting it, or a token an installed plugin registers through `add_log_sink`, and a token nobody provides stops the boot naming the registered ones, while `pipelex doctor` reports it in a row of its own and carries on through the console sink. `log_mode`, `poor_loggers`, `generic_poor_logger` and `is_console_logging_enabled` are gone with the rich-or-poor mode, and `set_log_mode`, `set_poor_log_formatter` and `LevelAndEmojiLogFormatter` with them, so a server that wants plain output selects `json` and a host that wants Pipelex silent sets `pipelex = "OFF"` under `[runtime.log.package_log_levels]`. **Migration:** run `pipelex migrate`; ledger entry `pipelex-config@5` deletes the four retired keys from `~/.pipelex/` and every project `.pipelex/` file, after which the defaults supply `sink = "console"`. A file that had `log_mode = "poor"` or `is_console_logging_enabled = false` chose a behaviour that deletion undoes and no migration may write back, so such a file sets `sink = "json"` or `pipelex = "OFF"` under `[runtime.log.package_log_levels]` by hand, as the entry's title says. [Log Sink Plugins](under-the-hood/log-sink-plugins.md) documents the seam.### Changed-- **Loggers are named by module (Breaking)**: a record from `pipelex/pipe_operators/pipe_llm.py` is emitted on the `pipelex.pipe_operators.pipe_llm` logger instead of `pipelex`, so a handler or filter keyed on the bare package name must key on the prefix, and a `package_log_levels` key now works at module depth too, `pipelex-pipe_operators-pipe_llm = "DEBUG"` for instance. The stack walk that named the logger on every line is gone, replaced by one frame lookup, and the caller-info templates that name the module now render instead of raising.+- **Loggers are named by module (Breaking)**: a record from `pipelex/pipe_operators/pipe_llm.py` is emitted on the `pipelex.pipe_operators.pipe_llm` logger instead of `pipelex`, so a handler keyed on the bare package name must key on the prefix, a filter an integration attached to the `pipelex` logger must move to its handler since Python never consults an ancestor logger's filters for a record propagating up from a module logger, and a `package_log_levels` key now works at module depth too, `pipelex-pipe_operators-pipe_llm = "DEBUG"` for instance. The stack walk that named the logger on every line is gone, replaced by one frame lookup, and the caller-info templates that name the module now render instead of raising.+- **`include_exception` carries the traceback as the record's `exc_info`**: instead of splicing `traceback.format_exc()` into the message, so the console renders one Rich traceback, the `json` sink an `exception` field and the `otlp` sink the `exception.*` attributes, and a call made outside an `except` block carries nothing where a `NoneType: None` line used to be appended.+- **Teardown removes only the handlers Pipelex installed**: `log.reset()` no longer strips every handler off the root logger nor calls `logging.shutdown()`, so a host's or a test's handler survives a Pipelex teardown; the sink's handler is flushed and closed, which is where a batching sink ships what it still holds.- **The pitch names who each way of running a method is for**: the README, the documentation's front page and **Quick Start**, and the site description now say a method runs "as an MCP for chatbots, as a webapp for people, or via API for your software", instead of "from your agent or your chatbot via MCP, as a webapp, or via API in any software".- **The quick start says the plugin also puts a method in your software**: the README and the documentation's **Quick Start** now say the plugin's skills build methods, run them and put them in your software, name `/pipelex-scaffold` for a webapp and `/pipelex-integrate` for a typed call from TypeScript or Python code, and say that `@pipelex/sdk` is the TypeScript SDK and `pipelex-sdk` the Python one.- **The documentation's cookbook is one page that points to the cookbook**: the cookbook now holds Pipelex's example methods as packages you run by their address on the hosted API, each with its own page showing every way to use it, so the documentation no longer keeps a page per example. **The Pipelex Cookbook** says what the cookbook is and links to it, the pages that linked an example now link its page in the cookbook or in the method library, and every old example address redirects to the new page. The two pages that documented the runtime itself move into the runtime's own sections: **Writing an Inference Plugin**, under **Under the Hood**, now carries the whole sample plugin and the configuration that routes a model to it, and **CV Batch Screening, Step by Step** joins **Get Started**. The README's cookbook line says what the cookbook now is.### Fixed-- **A log call before `log.configure` no longer raises**: it goes to the stdlib's default handling at the stdlib's default level, so a library that logs before Pipelex boots, or a boot that logs while it configures, cannot crash on the log line itself.+- **A log call before `log.configure` no longer raises**: it goes to the stdlib's default handling at the stdlib's default level, so a library that logs before Pipelex boots, or a boot that logs while it configures, cannot crash on the log line itself. Between `log.configure` and the sink's arrival at boot, every record is held and replayed through the selected sink in order, so a boot's own lines are rendered by the sink the configuration chose, and a boot that dies before the sink arrives sends all of them to stderr rather than only a warning or worse: each one passed the level the configuration set, and a failed boot is when that trail is worth most.+- **A log call never raises on a value the record or JSON refuses**: a field named like an attribute the `LogRecord` class owns (`__class__`, `__dict__`, `getMessage`) is prefixed like any other collision instead of raising `TypeError` or shadowing the method every formatter calls, and a content `json` refuses, a circular reference or a mapping with a non-string key, is rendered as its `repr` with no `data` instead of raising `ValueError` or `TypeError` out of `log.info`. A non-finite float nested inside a pydantic model is spelled as text like one at the top level, so a single `NaN` costs that one float its type and no other value on the line its structure. On the `console` sink the same promise needed the stdlib's own guard put back: Rich overrides `emit` without it, so a message carrying a tag-shaped span — `list[int]` in a type complaint, or a bracketed path, which Rich reads as a closing tag — raised `MarkupError` out of the log call and replaced the failure being reported with itself. Such a line now costs at most its own rendering, and an error assembled from an exception is rendered verbatim rather than read as markup, so the brackets and the type names in it reach the terminal as they were written.## [v0.65.0] - 2026-09-25diff --git a/docs/configuration/config-practical/logging-config.md b/docs/configuration/config-practical/logging-config.mdindex 95e1e8d64..934b3f986 100644--- a/docs/configuration/config-practical/logging-config.md+++ b/docs/configuration/config-practical/logging-config.md@@ -1,5 +1,5 @@----description: "Fine-tune Pipelex logging levels, Rich console output, and message formatting through TOML configuration settings in pipelex.toml."+description: "Select where Pipelex log records go with the sink key, and fine-tune logging levels, the console rendering, the JSON and OTLP sinks and message formatting through TOML configuration settings in pipelex.toml."---# Logging Configuration@@ -8,7 +8,7 @@ Configuration section: `[runtime.log]`## Overview-The logging configuration controls how Pipelex handles log messages, their formatting, and their display. The configuration is split into general logging settings and Rich console output settings.+The logging configuration controls the levels, where the records go, and how they are rendered. Where they go is one key, `sink`, naming a registered log sink; each shipped sink then reads its own settings from the section, and the rest of the section applies to every sink.## General Settings@@ -20,6 +20,7 @@ default_log_level = "INFO"- Sets the default logging level for all loggers- Valid values: `"VERBOSE"`, `"DEBUG"`, `"DEV"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`, `"OFF"`+- `"OFF"` silences the loggers `[runtime.log.package_log_levels]` does not pin; Pipelex's own loggers are pinned there at `INFO`, so silencing them, whichever sink is selected, is `pipelex = "OFF"` in that section### Package-Specific Log Levels@@ -36,14 +37,28 @@ pipelex = "INFO"- Use `-` instead of `.` in package names (e.g., `urllib3-connectionpool`)- A key works at any depth of the logger hierarchy, because Pipelex names every logger after the emitting module: `pipelex` governs the whole runtime, and `pipelex-pipe_operators-pipe_llm = "DEBUG"` opens one module while the rest of `pipelex` stays at its level-### Console Output+### The Sink```toml-is_console_logging_enabled = true+sink = "console"```-- Enable/disable console logging-- Default: `true`+- Names the registered log sink boot installs on the root logger, the way `runtime.storage.method` names a storage backend+- `"console"`: the Rich handler, for a terminal. The default, so the CLI keeps its rendering+- `"json"`: one JSON object per line, for a server behind a log agent+- `"otlp"`: the OpenTelemetry logs signal, for a collector+- Any other token an installed plugin registers; a token nobody provides stops the boot naming the registered ones+- The sink is a plugin capability: how it is discovered, selected and written is in [Log Sink Plugins](../../under-the-hood/log-sink-plugins.md)++### Console Log Target++```toml+console_log_target = "stderr"+```++- The process stream the `console` and `json` sinks write to: `"stdout"` or `"stderr"`+- Default: `"stderr"`, because logs are diagnostics and must stay off the data channel+- `console_print_target`, beside it, is a different knob: the Rich `Console` used for banners and the main CLI's data tables, which default to `"stdout"` so they can be piped to a file### Pretty-Print Mode@@ -65,7 +80,7 @@ json_logs_indent = 4presentation_line_width = 120```-- `json_logs_indent`: Indentation for JSON log output+- `json_logs_indent`: Indentation of the JSON rendering a structured content (a `dict`, a `list`, a model) gets in the message- `presentation_line_width`: Maximum line width for formatted output### Caller Information@@ -94,23 +109,9 @@ silenced_problem_ids = ["azure_openai_no_stream_options"]- List of problem IDs to silence- Prevents specific warnings from being logged-### Poor Loggers+## The `console` Sink-```toml-generic_poor_logger = "#poor-log"-poor_loggers = [- "kajson.decoder.sandbox",- "kajson.encoder.sandbox",- "class_registry.sandbox"-]-```--- Loggers that use simplified formatting without fancy features-- Useful for sandbox environments or when simple logging is needed--## Rich Console Configuration--Configuration section: `[runtime.log.rich_log]`+Configuration section: `[runtime.log.rich_log]`, read only when `sink = "console"`. The sink writes to `console_log_target`.### Display Options@@ -143,7 +144,7 @@ is_tracebacks_show_locals = falsetracebacks_suppress = []```-- Control how Python tracebacks are displayed+- Control how Python tracebacks are displayed; an exception a call carries with `include_exception=True` is rendered once, as a Rich traceback- Enable/disable word wrapping and local variable display- Suppress specific traceback patterns@@ -156,22 +157,44 @@ keywords_to_hilight = []- List of keywords to highlight in log messages- Useful for emphasizing important terms+## The `json` Sink++No section of its own: the sink writes to `console_log_target`, one JSON object per line and no ANSI ever. Each line carries `time` (ISO 8601, UTC, milliseconds, `Z`), `severity` (the level name), `logger` (the module-named logger), `message`, `exception` when the record carries one, and then every field, the run-scoped identifiers and the `data` attribute flat beside them, under their own names. Those keys are reserved on every line, so a field named like one of them is carried under a `field_` prefix whether or not the line carries an exception, and a non-finite float is written as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser.++```json+{"time": "2026-09-16T10:12:03.417Z", "severity": "INFO", "logger": "pipelex.pipe_operators.pipe_llm", "message": "Running the pipe", "request_id": "req-7f3a", "pipeline_run_id": "plr-01", "pipe_run_id": "pr-9c", "model": "gpt-5"}+```++## The `otlp` Sink++Configuration section: `[runtime.log.otlp]`, read only when `sink = "otlp"`.++```toml+[runtime.log.otlp]+endpoint = "http://collector:4318/v1/logs"+headers = { Authorization = "Bearer ..." }+```++- `endpoint`: The collector's logs URL. Left unset, the exporter follows the OpenTelemetry environment conventions: `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT` with the `/v1/logs` path, then the collector default on localhost+- `headers`: Headers sent with every export, an authorization header typically. Empty, the default, leaves `OTEL_EXPORTER_OTLP_HEADERS` in charge+- The records are exported in batches on the OTLP HTTP protocol, with the same service identity as the spans the runtime already exports, so a collector files the two together+- The semantic-convention keys the sink writes itself — the source location `code.file.path`, `code.function.name` and `code.line.number`, and the `exception.*` keys — are reserved on every record whether or not it carries an exception, exactly as the `json` sink reserves its own keys: a field named like one of them is carried under the same `field_` prefix, so nothing is overwritten and a field keeps one wire name whichever sink is selected+- The sink never exports its own export path: a record the SDK or the transport emits while an export is in flight is rejected before the handler's lock is taken, so an unreachable collector costs a warning on the export thread and never a loop or a hang at exit. A flush at teardown is bounded by the exporter's own timeout, `OTEL_EXPORTER_OTLP_TIMEOUT`, and whatever the sink raises while it flushes or closes is said on stderr rather than left to interrupt the teardown+## Example Configuration```toml[runtime.log]default_log_level = "INFO"-log_mode = "rich"+sink = "console"+console_log_target = "stderr"+console_print_target = "stdout"pretty_print_mode = "rich"-is_console_logging_enabled = truejson_logs_indent = 4presentation_line_width = 120is_caller_info_enabled = truecaller_info_template = "file_line_func"-silenced_problem_ids = []-generic_poor_logger = "#poor-log"-poor_loggers = [][runtime.log.package_log_levels]pipelex = "INFO"@@ -189,21 +212,34 @@ is_tracebacks_word_wrap = trueis_tracebacks_show_locals = falsetracebacks_suppress = []keywords_to_hilight = ["error", "warning", "failed"]++[runtime.log.otlp]+headers = {}```+## Migrating From the Log Mode++`log_mode` (`rich` / `poor`), `poor_loggers`, `generic_poor_logger` and `is_console_logging_enabled` are gone from the schema, and the defaults layer supplies `sink = "console"` in their place. Run `pipelex migrate`: ledger entry `pipelex-config@5` deletes the four keys from an existing file, and the defaults then take over.++Two of those keys chose a behaviour their deletion undoes, and no operation in the migration vocabulary can write the replacement — so where a choice was made, writing the replacement is yours:++- `is_console_logging_enabled = false` suppressed every record Pipelex's own log calls emitted. Its equivalent is `pipelex = "OFF"` under `[runtime.log.package_log_levels]`, which deep-merges over the base's `INFO` and leaves the third-party levels alone; `default_log_level` governs only the loggers that section does not pin, so on its own it silences none of Pipelex's records. Silence for everything takes `default_log_level = "OFF"` and every entry of that section at `OFF`, and sending the records elsewhere instead means selecting the sink that goes there. `is_console_logging_enabled = true` was the default and asks for nothing: the migration deletes it and the new default renders the same console.+- `log_mode = "poor"` chose a plain handler for a process with no terminal; that process now sets `sink = "json"`. `log_mode = "rich"` chose what `console` renders, and asks for nothing either.+- `poor_loggers` and `generic_poor_logger` have nothing to carry over: every record goes to the one selected sink.+## Best Practices1. **Development Environment**:- - Enable caller info for better debugging+ - Keep the `console` sink, and enable caller info for better debugging- Use verbose logging levels- Enable local variables in tracebacks2. **Production Environment**:+ - Select the `json` sink behind a log agent, or the `otlp` sink in front of a collector- Disable caller info for performance- Use INFO or higher log levels- - Disable local variables in tracebacks3. **Package Log Levels**:@@ -214,3 +250,4 @@ keywords_to_hilight = ["error", "warning", "failed"]## Related Documentation- [Logging Tool](../../tools/logging.md) - Using Pipelex logging in your code+- [Log Sink Plugins](../../under-the-hood/log-sink-plugins.md) - The seam behind the `sink` key, and how to write a sinkdiff --git a/docs/errors/duplicate-log-sink-error.md b/docs/errors/duplicate-log-sink-error.mdnew file mode 100644index 000000000..689f85ab8--- /dev/null+++ b/docs/errors/duplicate-log-sink-error.md@@ -0,0 +1,21 @@+---+title: "Duplicate log sink"+description: "Reference for the `DuplicateLogSinkError` Pipelex error class."+---++<!-- pipelex:generated -->++# Duplicate log sink++Two plugins registered a log sink for the same method.++| Field | Value |+|---|---|+| `error_type` | `DuplicateLogSinkError` |+| `title` | Duplicate log sink |+| `type_uri` | `https://docs.pipelex.com/latest/errors/duplicate-log-sink-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/inference-and-providers.md b/docs/errors/inference-and-providers.mdindex e9e43dbd6..d6b72aaba 100644--- a/docs/errors/inference-and-providers.md+++ b/docs/errors/inference-and-providers.md@@ -120,6 +120,7 @@ own page. Classes are grouped by subsystem.- [`DuplicateBundleValidatorError`](duplicate-bundle-validator-error.md) — Duplicate bundle validator- [`DuplicateHttpErrorMapperError`](duplicate-http-error-mapper-error.md) — Duplicate http error mapper- [`DuplicateInferenceBackendError`](duplicate-inference-backend-error.md) — Duplicate inference backend+- [`DuplicateLogSinkError`](duplicate-log-sink-error.md) — Duplicate log sink- [`DuplicateModelListerError`](duplicate-model-lister-error.md) — Duplicate model lister- [`DuplicateOrchestratorError`](duplicate-orchestrator-error.md) — Duplicate orchestrator- [`DuplicatePipeFuncExecutorError`](duplicate-pipe-func-executor-error.md) — Duplicate pipe func executor@@ -133,6 +134,7 @@ own page. Classes are grouped by subsystem.- [`PluginLayerViolationError`](plugin-layer-violation-error.md) — Plugin layer violation- [`RetiredPluginEntryPointGroupError`](retired-plugin-entry-point-group-error.md) — Retired plugin entry point group- [`UnknownBootOrchestratorError`](unknown-boot-orchestrator-error.md) — Unknown boot orchestrator+- [`UnknownLogSinkError`](unknown-log-sink-error.md) — Unknown log sink- [`UnknownPipeFuncExecutionModeError`](unknown-pipe-func-execution-mode-error.md) — Unknown pipe func execution mode- [`UnknownSecretsMethodError`](unknown-secrets-method-error.md) — Unknown secrets method- [`UnknownStorageMethodError`](unknown-storage-method-error.md) — Unknown storage methoddiff --git a/docs/errors/unknown-log-sink-error.md b/docs/errors/unknown-log-sink-error.mdnew file mode 100644index 000000000..5dde9926e--- /dev/null+++ b/docs/errors/unknown-log-sink-error.md@@ -0,0 +1,21 @@+---+title: "Unknown log sink"+description: "Reference for the `UnknownLogSinkError` Pipelex error class."+---++<!-- pipelex:generated -->++# Unknown log sink++A configured log sink has no registered factory.++| Field | Value |+|---|---|+| `error_type` | `UnknownLogSinkError` |+| `title` | Unknown log sink |+| `type_uri` | `https://docs.pipelex.com/latest/errors/unknown-log-sink-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/tools/logging.md b/docs/tools/logging.mdindex 04ae25791..33563b85d 100644--- a/docs/tools/logging.md+++ b/docs/tools/logging.md@@ -1,13 +1,13 @@---title: "Logging"-description: "Explore Pipelex logging: named fields, the run-scoped context, module-named loggers, custom log levels, Rich console formatting and structured data logging."+description: "Explore Pipelex logging: named fields, the run-scoped context, module-named loggers, custom log levels, the console, json and otlp sinks and structured data logging."---# Pipelex Logging System## Overview-Pipelex logs through one facade, `from pipelex import log`, built on Python's standard `logging`. A call takes a message, optional named fields and the usual presentation options; the run-scoped identifiers are bound once at a process entry and stamped onto every record emitted in scope. Fields and identifiers ride the stdlib `LogRecord` as attributes, never spliced into the message text, so a structured sink renders them as fields while the console keeps a narrative line.+Pipelex logs through one facade, `from pipelex import log`, built on Python's standard `logging`. A call takes a message, optional named fields and the usual presentation options; the run-scoped identifiers are bound once at a process entry and stamped onto every record emitted in scope. Fields and identifiers ride the stdlib `LogRecord` as attributes, never spliced into the message text, so a structured sink renders them as fields while the console keeps a narrative line. Where the records go is one config key, `sink` under `[runtime.log]`: `console` renders them through Rich, `json` writes one JSON object per line, `otlp` ships them to an OpenTelemetry collector, and a plugin can register another. The keys are in [Logging Configuration](../configuration/config-practical/logging-config.md) and the seam itself in [Log Sink Plugins](../under-the-hood/log-sink-plugins.md).## Log Levels@@ -48,7 +48,7 @@ log.verbose(data, title="Configuration")# Warning with problem IDlog.warning("API rate limit approaching", problem_id="rate_limit_warning")-# Error with exception traceback+# Error carrying the exception being handled, for the sink to renderlog.error("Failed to process", include_exception=True)# Development logging@@ -58,24 +58,26 @@ log.dev("Testing new feature")log.verbose("Detailed debug information")```-Every one of the seven methods (`verbose`, `debug`, `dev`, `info`, `warning`, `error`, `critical`) takes the same keyword-only `fields`. `title`, `inline`, `problem_id` and `include_exception` keep their meaning beside it.+Every one of the seven methods (`verbose`, `debug`, `dev`, `info`, `warning`, `error`, `critical`) takes the same keyword-only `fields`. `title`, `inline`, `problem_id` and `include_exception` keep their meaning beside it. `include_exception=True` carries the exception being handled as the record's `exc_info`, with nothing spliced into the message: the `console` sink renders the traceback under the line, the `json` sink writes it under the `exception` key and the `otlp` sink under the `exception.*` attributes. Outside an `except` block it carries nothing.## Fields`fields` is a mapping of named values. Each entry becomes an attribute of the emitted `LogRecord`, which is where a formatter or a sink reads it: `record.files` for the example above, or `%(files)s` in a stdlib format string. Nothing from `fields` is written into the message, so `record.getMessage()` is exactly the text you passed.-A value can be anything; the console ignores it and a structured sink serializes it, so prefer plain JSON-ready values (strings, numbers, booleans, lists and dictionaries of those) for anything meant to leave the process.+A value can be anything. It rides the record by reference and the sink serializes it on emit, on the calling thread, so what leaves the process is the value as it was at the call, and mutating it afterwards changes nothing already emitted. The `console` sink ignores it, the `json` sink dumps a pydantic model in JSON mode and falls back to `str` for a value JSON does not know, and the `otlp` sink carries a scalar or a sequence of scalars as an attribute and anything else as JSON text; prefer plain JSON-ready values (strings, numbers, booleans, lists and dictionaries of those) for anything meant to be queried later.### Naming convention- A field name is a `snake_case` identifier, spelled the way the value is spelled where it comes from: a field carrying a payload's `pipeline_run_id` is `pipeline_run_id`, not `pipelineRunId` or `run`.- Where the [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/logs/) define a key for the concept, use that key verbatim, dots included (`http.response.status_code`, `code.function.name`), so an OTLP sink emits it without translation. The three run identifiers have no such key and keep their payload names.-- `request_id`, `pipeline_run_id` and `pipe_run_id` are reserved for the [run-scoped context](#the-run-scoped-context), and `data` is reserved for [structured content](#structured-content). A field of one of those names is accepted and takes precedence as described below, but nothing else should use them.+- `request_id`, `pipeline_run_id` and `pipe_run_id` are reserved for the [run-scoped context](#the-run-scoped-context), and `data` is reserved for [structured content](#structured-content). A field of one of those names is accepted and never dropped: it takes precedence over the context for the three identifiers, and it rides under `field_data` beside structured content. Nothing else should use them.-### Names the stdlib owns+### Names that are not yours to giveThe stdlib refuses an `extra` key that would overwrite one of the record's own attributes (`name`, `message`, `lineno`, `module`, `args`, `asctime` and the rest), and a library's log call never raises. An entry of such a name is therefore carried under the prefix `field_`: `fields={"name": "alpha"}` lands as `record.field_name`. What counts as owned is read off the record actually built, through whatever record factory is installed, so an attribute an OpenTelemetry or tracing instrumentation stamps on every record is a collision too, for a field, a context identifier and `data` alike. The prefix is applied until the name lands on an attribute nobody owns, and entries attach in order, so a call that gives both `name` and `field_name` keeps both values whatever their order: the one that arrives second lands on `field_field_name`.+Two kinds of name are reserved though nothing on a fresh record owns them yet, because both are stamped after the entries are attached and the stdlib's refusal therefore cannot cover them: what the formatter sets (`message`, `asctime`) and what Pipelex's own logging machinery sets — `_pipelex_forwarded`, the marker that tells the sink's handler a record reached it through the boot's holding handler already. Both take the same `field_` prefix, and the marker is never handed to a sink as something the record carries. Without that reservation, `fields={"_pipelex_forwarded": True}` would have the sink's own filter read the record as one already delivered and drop it whole.+## The run-scoped contextThe identifiers that correlate a record with the run it belongs to are bound once and stamped onto every record emitted while the binding holds:@@ -112,14 +114,18 @@ The last two are the runner's and the orchestration plugin's to bind, beside the## Structured content-When the content is not a string, it is rendered as JSON for the console, indented by `json_logs_indent`, and the JSON-ready form is carried as the record's `data` attribute for a structured sink:+When the content is not a string, it is rendered as JSON for the message, indented by `json_logs_indent`, and the rendering is read back as the record's `data` attribute for a structured sink. `data` is therefore a snapshot of the call, JSON-ready whatever the content held, and the caller may mutate the object afterwards without changing what was emitted:++- a `dict` is carried as a dictionary and a `list` as a list, their values as JSON reads them back: a datetime as its text, a `Decimal` as a string,+- a pydantic model, or a list of models, is carried as the dictionary or list its serialization produces,+- a number or a boolean is carried as itself, `log.info(42)` giving `data == 42`,+- a value JSON cannot serialize goes through the JSON helpers' fallbacks, kajson first and `str` last, and a dictionary that reaches the last fallback is wrapped as `{"!": ...}` to flag it,+- `None` is rendered as the word `None` and carries no `data`,+- content `json` refuses outright, a circular reference or a mapping with a non-string key, is rendered as its `repr` and carries no `data`; a log call never raises.-- a `dict` is carried as a dictionary,-- a `list` is carried as a list,-- any other object, a pydantic model for instance, is carried as the dictionary or list its serialization produces,-- `None` is rendered as the word `None` and carries no `data`.+A `NaN` or an infinity survives the round trip as a float; a wire sink writes it as the string `"NaN"`, `"Infinity"` or `"-Infinity"`, since JSON has no token for it that a strict parser accepts.-Structured content owns `data` outright: a `data` entry in `fields` beside a non-string content is overridden.+Structured content owns the `data` name: a `data` entry in `fields` beside a non-string content keeps its value and is carried under `field_data`, the same prefix a field named like a record attribute or like a sink's reserved key gets. Nothing passed in `fields` is dropped for a name collision.## Logger names and levels@@ -134,14 +140,18 @@ pipelex-pipe_operators-pipe_llm = "DEBUG"httpx = "WARNING"```-The record's `pathname`, `lineno` and `funcName` point at the calling line, which is what the Rich handler links to and what the caller-info templates render when `is_caller_info_enabled` is on; all of it comes from that same frame, with no source file read.+The record's `pathname`, `lineno` and `funcName` point at the calling line, which is what the `console` sink links to and what the caller-info templates render when `is_caller_info_enabled` is on; all of it comes from that same frame, with no source file read.++A filter attached to `logging.getLogger("pipelex")` never sees these records: the stdlib runs a logger's filters only for records emitted on that very logger, and a record is emitted on the module-named one. Attach the filter to the handler instead, which the stdlib runs for every record it handles, or, to reach whichever sink is installed, append a processor to the sink's `processors` list, as described in [Log Sink Plugins](../under-the-hood/log-sink-plugins.md#logsink).## Before configuration-`log.configure` runs at boot, and a call before it never raises: the record goes to the stdlib's default handling at the stdlib's default level, so an `INFO` is dropped and a warning reaches stderr through `logging.lastResort`. No handler is installed as a side effect, and nothing is swallowed. A library that logs before Pipelex boots, and the boot itself while it configures, are both safe.+Logging is configured in two steps at boot. `log.configure` sets the levels and installs a holding handler on the root logger; then, once the plugin registrar is built, the configured sink is looked up, its handler is installed and the held records are replayed to it in order, so the boot's own lines reach the sink selected for them. A record emitted during the handoff reaches the sink either way, replayed or forwarded, and one held record the sink cannot render gets the stdlib's own recovery and costs none of the others. A call before `configure` never raises: the record goes to the stdlib's default handling at the stdlib's default level, so an `INFO` is dropped and a warning reaches stderr through `logging.lastResort`. A boot that fails between the two steps closes the holding handler, which hands what it held at `WARNING` and above to `logging.lastResort` and drops the rest. No handler is installed before `configure`, and nothing is swallowed. A library that logs before Pipelex boots, and the boot itself while it configures, are both safe.## Console rendering+The default sink, `console`, renders through Rich with every `[runtime.log.rich_log]` setting. Rich is imported when the sink is built and nowhere else, so a process that selects `json` or `otlp` never loads it, and one that selects `console` without Rich installed stops at boot naming the extra to install and the `json` alternative.+### Rich Formatting- Color coding and syntax highlighting for JSON and other data structures@@ -157,7 +167,6 @@ Built-in emoji indicators for different components, keyed on the logger name's p- 🌀 Google-related logs- ⚡️ Network connections- *️⃣ JSON processing-- 🧿 Poor-log channel (`#poor-log`, the simplified fallback logger)### Caller Information@@ -194,11 +203,12 @@ Optional inclusion of caller information in logs, prefixed to the console line:4. **Exception Handling**:- - Use `include_exception=True` for error context+ - Use `include_exception=True` inside the `except` block, and let the sink render the traceback its own way- Include relevant data in error logs- Use appropriate log levels for exceptions## Related Documentation-- [Logging Configuration](../configuration/config-practical/logging-config.md) - Configure log behavior in `pipelex.toml`+- [Logging Configuration](../configuration/config-practical/logging-config.md) - Configure log behavior and select the sink in `pipelex.toml`+- [Log Sink Plugins](../under-the-hood/log-sink-plugins.md) - The sink seam, the built-in sinks and how to write one- [CLI](./cli/index.md) - Commands that surface runtime logs during developmentdiff --git a/docs/under-the-hood/log-sink-plugins.md b/docs/under-the-hood/log-sink-plugins.mdnew file mode 100644index 000000000..20d1e5d14--- /dev/null+++ b/docs/under-the-hood/log-sink-plugins.md@@ -0,0 +1,192 @@+---+title: "Log Sink Plugins"+description: "How Pipelex selects where its log records go by config, the keyed-registry-plus-config-selected-singleton seam a log sink plugin rides, the shipped json, console and otlp sinks, and how to author an out-of-tree sink."+---++# Log Sink Plugins++Every record the runtime emits, a `log.info(...)` in a pipe operator, a warning from a provider, the failure line of a run, leaves the process through a single **log sink** installed on the root logger at boot. Which sink that is comes entirely from data: one config field, `runtime.log.sink`, names a sink, and a **log sink plugin** is what teaches Pipelex how to build it.++Core names no sink by import or by string. The built-in sinks (`json`, `console`, `otlp`) are a plugin too, the always-on `LogSinkPlugin`, riding the exact same seam an out-of-tree package would. This page documents that seam, the contract a plugin registers, what a sink reads off a record, and how to write one.++This is the third application of the mechanism the [storage provider](storage-provider-plugins.md) seam introduced and the [secrets provider](secrets-provider-plugins.md) seam reused; the three pages describe the same shape with the nouns swapped, and this one adds what is particular to logging: the sink is resolved after logging is already configured, and every sink shares one record contract.++---++## The same shape as storage and secrets++A sink is a **process-global singleton selected by its own config key**, independent of the orchestrator and of any call, so it rides the **keyed registry + config-selected singleton** mechanism rather than a per-call registry or a hub slot:++> Plugins register N sink factories into a registry keyed by an open `method` token. At boot, core reads `runtime.log.sink`, looks that token up in the registry, calls the factory to produce the one sink, and installs that sink's handler on the root logger.++There is no `match runtime.log.sink:` anywhere in boot: the token set is open, so validation *is* the registry lookup. Adding a sink means registering a factory for its token; nothing in core changes. Every log call in the runtime and in your own code keeps going through `from pipelex import log`, and is unaffected by which sink was selected.++---++## The seam in one view++```+RuntimeBoot.__init__+ └─ log.configure(runtime.log) # levels set; a holding handler takes every record+RuntimeBoot.setup+ └─ build_registrar(config, builtin_plugins=…, …) # pure, import-light+ ├─ for each built-in plugin (LogSinkPlugin is one)+ └─ for each installed entry point in the requested groups+ └─ plugin.register(registrar) # side-effect-free+ └─ registrar.add_log_sink(method=…, factory=…)+ └─ LogSinkRegistry(registrar.log_sinks)+ └─ sink = registry.get_required(method=runtime.log.sink)(runtime.log)+ └─ log.install_sink(sink) # the sink's handler replaces the holding handler,+ # which replays what it held through it, in order+teardown+ └─ log.reset() # flushes and closes the sink's handler, removes it; nothing it raises escapes+```++### Configured before it is selected++Logging is configured as soon as the configuration is read, in `RuntimeBoot.__init__`, so that everything the boot says afterwards is a record on a module-named logger at the configured level. The sink, though, is a plugin capability, and plugin discovery runs a little later, in `setup()`, once the boot knows which built-ins and which entry-point groups it reads. Rather than choose a default renderer for that window, `configure` installs a **holding handler** that keeps every record, and `install_sink` replays them through the selected sink's handler the moment it is installed, in the order they were emitted. A boot's own lines are therefore rendered by the sink the configuration chose, never dropped and never written in a shape nothing chose.++The sink is the first capability resolved out of the registrar, ahead of secrets and storage, precisely so that every later line of the boot goes through it. A boot that dies before that point, on the gateway terms gate for instance, releases its process globals through `log.reset()`, which closes the holding handler: what it still holds gets the stdlib's last-resort treatment, a warning or worse reaches stderr and the rest is dropped, exactly as a record emitted before `configure` would be.++`pipelex doctor`, which deliberately bypasses `Pipelex.make` to diagnose a broken configuration, selects its sink the same way: the pure `build_registrar` discovery, then the `runtime.log.sink` lookup. Where boot stops on a token nobody registered, the doctor installs the console sink instead and reports the token in a row of its own, naming the registered sinks, so the report it exists to produce is still produced.++---++## The contract++### `LogSinkFactoryFn`++A sink is produced by a typed **callable**, whole log config in, sink out (`pipelex/plugins/log_sink_registry.py`):++```python+LogSinkFactoryFn = Callable[[LogConfig], LogSink]+```++Passing the whole `LogConfig` is deliberate: a factory reads whatever it needs, the stream target, its own settings section, at the boot apply-point, never at registration. A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:++```python+registrar.add_log_sink(method="gcp", factory=_make_gcp_log_sink)+```++`register` is **side-effect-free**: it may call the registrar menu and nothing else. That is what keeps `build_registrar` safe to run more than once, at boot and again for the `pipelex plugins list` diagnostic, whose rows show the sinks each plugin contributed as `log sink <method>`.++### `LogSink`++The factory returns a `LogSink` (`pipelex/tools/log/log_sink.py`), an abstract base with one method to implement and two things every sink gets from the base:++```python+class LogSink(ABC):+ processors: list[LogRecordProcessor] # the slot every sink shares++ @abstractmethod+ def make_handler(self) -> logging.Handler: ... # built once, at install++ @property+ def handler(self) -> logging.Handler: ... # the same handler on every read++ def redirect_to_stderr(self) -> None: ... # a no-op unless the sink writes to a process stream+```++- **`make_handler`** builds the stdlib handler the sink installs on the root logger, and is where a heavy import belongs. It is called once, when boot installs the sink, so selecting a sink is what pays for its dependency and a sink whose package is missing fails there, at boot, with `MissingDependencyError` naming the extra.+- **`processors`** is the slot a cross-cutting transformation hooks into. A processor is a callable that edits a record in place, `Callable[[logging.LogRecord], None]`, and the base runs every processor over each record before the handler formats it, through a filter on the sink's own handler. Redaction is the intended one: appended to the base's list, it reaches every sink, built-in or external, with no sink knowing about it. A processor sees the record the root logger shares with every other handler, so a handler an integration attached to the root logger sees the record as the processors left it only if it runs after this one.+- **`redirect_to_stderr`** exists for the agent CLI, whose stdout is reserved for its structured envelope: a sink that writes to a process stream moves it to stderr, and one that writes to a collector ignores the call.++### What a sink reads off a record++Every sink reads the same record, shaped by the [logging facade](../tools/logging.md): the message is `record.getMessage()`, and everything the call, the bound context or a record factory attached rides as attributes. `carried_attributes(record=...)` in `pipelex/tools/log/log_fields.py` hands a sink exactly that set, in the order the attributes were attached, by subtracting the attributes the stdlib gives every record and the ones this package stamps on a record itself. An exception a call carried with `include_exception=True` is the record's `exc_info`, the stdlib's own channel, which each sink renders its own way.++Three rules follow for a sink author, and the shipped sinks honour all of them:++- **A sink serializes on emit, synchronously, on the calling thread.** A field's value rides the record by reference, so a value the caller mutates after the call is not what the record said; reading it at emit is what makes the record a faithful account of the call. A sink that queues, the `otlp` sink on its batching exporter for instance, translates the record into its wire form at emit and queues the translation. The `data` attribute needs no such care: it is already a JSON-ready snapshot of the call.+- **A sink renders every record it is handed and never raises.** A value the wire cannot carry is written as text, a JSON rendering for a mapping, a `repr` for what JSON refuses; a serialization failure is a fact about the value, never a reason to lose the line. A handler failure goes through the stdlib's `handleError`, as any handler's does.+- **A sink reserves the keys it writes itself, on every record.** A carried attribute named like one of them is written under the `COLLIDING_FIELD_PREFIX` — `field_` — applied until the name lands free, which is the same treatment the record gives a field named after a stdlib attribute. Reserved unconditionally, not only when the sink happens to write the key on this record: a field's wire name must not depend on whether an exception was attached, and a field must not survive one sink and vanish on another. The `json` sink reserves `time`, `severity`, `logger`, `message` and `exception`; the `otlp` sink reserves its `code.*` and `exception.*` semantic-convention keys.++---++## The built-in `LogSinkPlugin`++`pipelex/providers/log_sinks/log_sink_plugin.py` is the reference sink plugin. It is **core-unconditional**, a process needs somewhere for its records to go, so it joins `KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES` and cannot be disabled into a boot with no sink:++```python+class LogSinkPlugin:+ name = "log_sinks"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_log_sink(method=LogSinkMethod.JSON, factory=_make_json_log_sink)+ registrar.add_log_sink(method=LogSinkMethod.CONSOLE, factory=_make_console_log_sink)+ registrar.add_log_sink(method=LogSinkMethod.OTLP, factory=_make_otlp_log_sink)+```++| Sink | What it does | Where it reads its settings |+|------|--------------|-----------------------------|+| `json` | One JSON object per line on the configured stream: `time`, `severity`, `logger`, `message`, `exception` when there is one, then the fields, the context identifiers and `data` flat beside them, a field named like one of those keys under a `field_` prefix on every line, a non-finite float as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser, and no ANSI ever. What the hosted plane's runner and worker select. | `console_log_target` |+| `console` | The Rich handler with the emoji formatter and every `[runtime.log.rich_log]` setting, byte for byte what the console showed before sinks existed. Rich is imported when the handler is built, and a process that selects another sink never loads it through this path. | `console_log_target`, `[runtime.log.rich_log]` |+| `otlp` | The OpenTelemetry logs signal: a `LoggerProvider` carrying the same service identity as the tracer, a `BatchLogRecordProcessor` and the OTLP HTTP log exporter. The message is the body, the level maps onto the OTel severity scale, the fields, identifiers and `data` ride as attributes (a mapping as JSON text), an exception lands under the `exception.*` semantic-convention keys, and a collector receives the logs beside the spans the runtime already exports. The keys the sink writes itself, the `code.*` source location and the `exception.*` set, are reserved on every record whether or not it carries an exception, so a field named like one of them rides under a `field_` prefix rather than being overwritten. A filter on the handler rejects the records of the sink's own export path, the SDK's by logger name and the transport's by the context value the SDK sets around an export, before the handler's lock is taken, so an export failure never re-enters the pipeline and a shutdown never waits on itself. | `[runtime.log.otlp]` |++The `otlp` factory imports the OpenTelemetry logs SDK when it runs, never at register, so registering the built-ins imports none of it, which the import-light guard pins.++### Why `console` is the default++`pipelex/pipelex.toml` ships `sink = "console"`. A user who installs the CLI expects the rendering the CLI always had, the emoji, the colours, the clickable paths, and a default that changed it would be a regression for every terminal. A server is the process that must choose: it selects `json`, and the fail-loud factory is what makes the choice hard to forget. Rich is a hard dependency today, so a server that forgot to select `json` still boots with the console sink; once Rich moves behind the CLI extra, the same server refuses to boot with the extra and the `json` alternative named, rather than shipping ANSI to a log agent.++---++## Selecting a sink by config++`runtime.log.sink` is an **open `str` token**, not a closed enum. The built-ins use `"json"`, `"console"` and `"otlp"`; an external plugin registers its own. A config naming an external token **parses fine**, the token is stored verbatim and its installability is validated later, at registry lookup:++```toml+# .pipelex/pipelex.toml+[runtime.log]+sink = "gcp" # an out-of-tree sink, selected iff its plugin is installed+```++Whether that token names an *installed* sink is validated at **registry lookup**, not at parse: an unknown token surfaces as `UnknownLogSinkError` at boot, which lists the registered tokens so the fix is obvious.++An external sink has no settings section of its own in `LogConfig`, whose sub-models are the shipped sinks'; until a generic passthrough for external sinks lands, an out-of-tree sink reads its own configuration from the environment or its own file, as an out-of-tree storage provider does.++---++## Fail-loud guarantees++| Condition | Error |+|-----------|-------|+| `runtime.log.sink` names no registered sink | `UnknownLogSinkError` (lists the registered tokens) |+| the selected sink's package is not installed | `MissingDependencyError` (package + `pipelex[<extra>]` hint), raised at install, at boot |+| two plugins register the same `method` | `DuplicateLogSinkError` (names both plugins) |+| `name` (`"log_sinks"`) in `runtime.plugins.disabled` | `CoreUnconditionalPluginDisabledError` |+| published under the retired `pipelex.plugins` group | `RetiredPluginEntryPointGroupError` |+| entry point raises while loading/registering | `BrokenPluginError` |++The duplicate detection is the same fail-loud `_add` helper the storage and secrets menus use.++---++## Authoring an out-of-tree log sink plugin++A third-party log sink plugin is a distribution that:++1. implements a `LogSink` subclass whose `make_handler` builds the handler and performs whatever SDK import the handler needs, so the module stays import-light; the handler reads the record through `carried_attributes`, serializes on emit, and never raises;+2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_log_sink(method="<token>", factory=...)` and nothing else;+3. advertises itself under the `pipelex.plugins.kernel` entry-point group, a log sink being a kernel-layer capability (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means):++```toml+# pyproject.toml of your plugin package+[project.entry-points."pipelex.plugins.kernel"]+gcp_log_sink = "pipelex_log_gcp.plugin:GcpLogSinkPlugin"+```++Installing the distribution makes the token selectable (`runtime.log.sink = "gcp"`); uninstalling removes it. No core change, no central registration list: *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `runtime.plugins.disabled` denylist.++Use `pipelex plugins list` to see every discovered plugin, what each contributed (`log sink <method>` among the rows), and its denylist state.++---++## Related++- [Logging](../tools/logging.md): the facade, the fields, the context and what a record carries+- [Logging Configuration](../configuration/config-practical/logging-config.md): the `sink` key and each shipped sink's settings+- [Storage Provider Plugins](storage-provider-plugins.md) and [Secrets Provider Plugins](secrets-provider-plugins.md): the sibling config-selected-singleton seams+- [Inference Backend Plugins](inference-backend-plugins.md): the shared discovery and denylist machinery+- [Error Model](error-model.md): how these errors render and dereferencediff --git a/mkdocs.yml b/mkdocs.ymlindex f6d36fc24..4dddc3aa5 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -351,6 +351,7 @@ plugins:- under-the-hood/orchestrator-plugins.md: "Orchestrator Plugins"- under-the-hood/storage-provider-plugins.md: "Storage Provider Plugins"- under-the-hood/secrets-provider-plugins.md: "Secrets Provider Plugins"+ - under-the-hood/log-sink-plugins.md: "Log Sink Plugins"- under-the-hood/error-model.md: "Error Model"- under-the-hood/runtime-bridge-and-transport.md: "Runtime Bridge & Transport"- under-the-hood/distributed-content-generation.md: "Content Generation Across Boundaries"@@ -584,6 +585,7 @@ nav:- Orchestrator Plugins: under-the-hood/orchestrator-plugins.md- Storage Provider Plugins: under-the-hood/storage-provider-plugins.md- Secrets Provider Plugins: under-the-hood/secrets-provider-plugins.md+ - Log Sink Plugins: under-the-hood/log-sink-plugins.md- Error Model: under-the-hood/error-model.md- Runtime Bridge & Transport: under-the-hood/runtime-bridge-and-transport.md- Content Generation Across Boundaries: under-the-hood/distributed-content-generation.mddiff --git a/pipelex/cli/agent_cli/CLAUDE.md b/pipelex/cli/agent_cli/CLAUDE.mdindex 927a48099..1de00b1aa 100644--- a/pipelex/cli/agent_cli/CLAUDE.md+++ b/pipelex/cli/agent_cli/CLAUDE.md@@ -97,7 +97,7 @@ commands/| `models` | Lists available model presets, aliases, and waterfalls. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value) || `check-model` | Validates a model reference and suggests alternatives if invalid. `--type`/`-t` for model category, `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value) || `migrate` | Migrates this machine's configuration files (global `~/.pipelex/` and project `.pipelex/`, non-recursively) to the current schema by replaying each surface's ledger. Does **not** boot — it is the command for a configuration that cannot load. Writes only with `--yes`; `--dry-run` plans and is the default; the two together are refused with exit 2. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json`. The verdict is the structured `needs_attention`, not the exit code. |-| `doctor` | Checks config, credentials, models health. `--global`/`-g` for global dir. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value). The telemetry check carries a `finding` (`healthy`/`not_found`/`unparseable`/`out_of_date`/`invalid`) beside its `healthy` and `message`; branch on that, and note `out_of_date` is `pipelex migrate`'s and never `pipelex init telemetry`'s — a reset would discard the settings the migration keeps — and that it means the migration would actually rewrite that file, so a file it would only report on is `invalid`. `checks.pending_migrations` is the machine-wide half: a `finding` (`up_to_date`/`pending`/`needs_attention`/`unavailable`) plus `migratable_files` and `attention_files`. It is the only channel a machine has for a pending migration — a tolerated boot warns a person and this CLI silences logging — and it is **not** scoped by `--global`, because it answers for `pipelex migrate`, which walks both directories. |+| `doctor` | Checks config, credentials, models health. `--global`/`-g` for global dir. `--format markdown\|json` (success, default: markdown) + `--error-format markdown\|json` (errors, defaults to `--format`'s value). The telemetry check carries a `finding` (`healthy`/`not_found`/`unparseable`/`out_of_date`/`invalid`) beside its `healthy` and `message`; branch on that, and note `out_of_date` is `pipelex migrate`'s and never `pipelex init telemetry`'s — a reset would discard the settings the migration keeps — and that it means the migration would actually rewrite that file, so a file it would only report on is `invalid`. `checks.pending_migrations` is the machine-wide half: a `finding` (`up_to_date`/`pending`/`needs_attention`/`unavailable`) plus `migratable_files` and `attention_files`. It is the only channel a machine has for a pending migration — a tolerated boot warns a person and this CLI silences logging — and it is **not** scoped by `--global`, because it answers for `pipelex migrate`, which walks both directories. `checks.plugins` is present once the runtime setup discovered the plugins the way boot does: `healthy` and `message`, not healthy when the plugin registry did not build, in which case the message names the plugin error. `checks.log_sink` is present once the runtime setup ran: `healthy` and `message`, not healthy when `[runtime.log] sink` names no registered sink, when the configured sink failed to install or when the registry did not build. The message says why, and says where the report's own lines went: onto the console sink on stderr, which the doctor installs in place of the one that failed, or onto a sink already recorded, which it leaves alone rather than installing over — a sink is recorded before the held records are replayed, so a failed installation does not always leave nothing behind. |## Key Patternsdiff --git a/pipelex/cli/agent_cli/commands/agent_cli_factory.py b/pipelex/cli/agent_cli/commands/agent_cli_factory.pyindex ba63ff317..49b8bab88 100644--- a/pipelex/cli/agent_cli/commands/agent_cli_factory.py+++ b/pipelex/cli/agent_cli/commands/agent_cli_factory.py@@ -146,16 +146,17 @@ def apply_agent_cli_output_discipline() -> None:start of every agent CLI entry point); this helper handles the channels that areINDEPENDENT of Python's logging system:- 1. ``log.redirect_to_stderr`` keeps the RichHandler's console on stderr — defense- in case ``logging.disable`` is ever cleared.+ 1. ``log.redirect_to_stderr`` points the installed sink at stderr when it writes to a+ process stream — defense in case ``logging.disable`` is ever cleared.2. ``PrettyPrinter.mode = SILENT`` neutralizes ``pretty_print(...)`` entirely(Rich-based, not logging-based).3. Hub-level ``set_console_print_target(STDERR)`` for the Rich ``Console`` used bybanners / tables (also Rich-based, not logging-based).Safe to call from the broken-config doctor path where ``setup_doctor_runtime`` was- skipped: ``log.redirect_to_stderr`` no-ops when no rich_handler is registered, and- the hub print-target call is gated on a hub being installed.+ skipped: ``log.redirect_to_stderr`` no-ops before a sink is installed, a sink that+ writes to no process stream ignores it, and the hub print-target call is gated on a+ hub being installed."""log.redirect_to_stderr()PrettyPrinter.mode = PrettyPrintMode.SILENTdiff --git a/pipelex/cli/agent_cli/commands/doctor_cmd.py b/pipelex/cli/agent_cli/commands/doctor_cmd.pyindex d19554769..33d142c3e 100644--- a/pipelex/cli/agent_cli/commands/doctor_cmd.py+++ b/pipelex/cli/agent_cli/commands/doctor_cmd.py@@ -14,8 +14,10 @@ from pipelex.cli.agent_cli.commands.agent_output import CliOutputFormat, agent_efrom pipelex.cli.commands.doctor_cmd import (BackendFileReport,ConfigLocationInfo,+ LogSinkCheck,PendingMigrationsCheck,PendingMigrationsFinding,+ PluginsCheck,TelemetryConfigCheck,TelemetryConfigFinding,check_backend_credentials,@@ -28,6 +30,7 @@ from pipelex.cli.commands.doctor_cmd import ()from pipelex.core.validation import MIGRATE_COMMANDfrom pipelex.system.configuration.config_loader import config_manager+from pipelex.tools.log.log import logdef _status_icon(*, healthy: bool) -> str:@@ -125,6 +128,18 @@ def _format_doctor_markdown(result: dict[str, Any]) -> str:lines.append(f"\n## Telemetry \u2014 {_status_icon(healthy=telemetry_check['healthy'])}\n")lines.append(telemetry_check["message"])+ # Plugins, present only when the runtime setup discovered them+ plugins_check = checks.get("plugins")+ if plugins_check is not None:+ lines.append(f"\n## Plugins \u2014 {_status_icon(healthy=plugins_check['healthy'])}\n")+ lines.append(plugins_check["message"])++ # Log Sink, present only when the runtime setup ran+ log_sink_check = checks.get("log_sink")+ if log_sink_check is not None:+ lines.append(f"\n## Log Sink \u2014 {_status_icon(healthy=log_sink_check['healthy'])}\n")+ lines.append(log_sink_check["message"])+# Backend Credentialscreds_check = checks["backend_credentials"]lines.append(f"\n## Backend Credentials \u2014 {_status_icon(healthy=creds_check['healthy'])}\n")@@ -198,6 +213,19 @@ def agent_doctor_cmd(Target directory: auto-detects project .pipelex/ if present, else ~/.pipelex/.Use --global/-g to force checking the global ~/.pipelex/ directory."""+ # The doctor configures logging and installs a sink for its own report; once the report is out,+ # it releases them, so an ``otlp`` sink's exporter and the hook it registered at exit go with the+ # command rather than outliving it. Logging an embedder configured before calling in is left alone.+ logging_was_configured = log.is_configured+ try:+ _do_agent_doctor_cmd(global_=global_, output_format=output_format, error_format=error_format)+ finally:+ if not logging_was_configured:+ log.reset()+++def _do_agent_doctor_cmd(*, global_: bool, output_format: CliOutputFormat, error_format: CliOutputFormat | None) -> None:+ """The report itself; ``agent_doctor_cmd`` is the entry point that releases what this configured."""set_agent_cli_error_format(error_format or output_format)# Process-global logging cutoff, BEFORE setup_doctor_runtime / check_* can trigger# any third-party log line.@@ -239,9 +267,13 @@ def agent_doctor_cmd(models_healthy: boolmodels_message: strbackend_file_reports: dict[str, BackendFileReport]+ log_sink_check: LogSinkCheck | None = None+ plugins_check: PluginsCheck | None = Noneif config_healthy:try:- setup_doctor_runtime(log_config_overrides=AGENT_CLI_STDERR_LOG_FIELDS, config_dir=config_dir)+ runtime_setup = setup_doctor_runtime(log_config_overrides=AGENT_CLI_STDERR_LOG_FIELDS, config_dir=config_dir)+ log_sink_check = runtime_setup.log_sink+ plugins_check = runtime_setup.plugins# Pin discipline BEFORE check_models. setup_doctor_runtime uses# log.configure_if_unset(), which no-ops when a prior process already configured# logging (embedded reuse, interleaved tests) — in that case AGENT_CLI_STDERR_LOG_FIELDS@@ -273,7 +305,15 @@ def agent_doctor_cmd(# installed a hub or configured log — the helper guards both internally).apply_agent_cli_output_discipline()- all_healthy = config_healthy and pending_migrations_check.is_healthy and telemetry_check.is_healthy and backends_healthy and models_healthy+ all_healthy = (+ config_healthy+ and pending_migrations_check.is_healthy+ and telemetry_check.is_healthy+ and backends_healthy+ and models_healthy+ and (log_sink_check is None or log_sink_check.is_healthy)+ and (plugins_check is None or plugins_check.is_healthy)+ )# Build backend credential detailsbackends_list: list[dict[str, Any]] = []@@ -332,6 +372,12 @@ def agent_doctor_cmd(recommended_actions.append(f"Manually fix backend configuration in {config_location.config_dir}/inference/backends/{file_report.backend_name}.toml")+ if plugins_check is not None and not plugins_check.is_healthy:+ recommended_actions.append(+ "Fix, upgrade or uninstall the plugin the plugins check names, or take a core plugin out of runtime.plugins.disabled"+ )+ if log_sink_check is not None and not log_sink_check.is_healthy:+ recommended_actions.append("Set 'sink' in [runtime.log] to a registered log sink, or fix what the log_sink check says stopped it")result: dict[str, Any] = {"success": True,@@ -368,6 +414,10 @@ def agent_doctor_cmd(},},}+ if plugins_check is not None:+ result["checks"]["plugins"] = {"healthy": plugins_check.is_healthy, "message": plugins_check.message}+ if log_sink_check is not None:+ result["checks"]["log_sink"] = {"healthy": log_sink_check.is_healthy, "message": log_sink_check.message}if recommended_actions:result["recommended_actions"] = recommended_actionsdiff --git a/pipelex/cli/commands/doctor_cmd.py b/pipelex/cli/commands/doctor_cmd.pyindex d800566d0..7ea6e1b0f 100644--- a/pipelex/cli/commands/doctor_cmd.py+++ b/pipelex/cli/commands/doctor_cmd.py@@ -42,13 +42,18 @@ from pipelex.cogt.models.deck_manifest import DeckFileStatus, DeckSyncReport, cofrom pipelex.cogt.models.model_manager import ModelManagerfrom pipelex.config import get_configfrom pipelex.core.validation import MIGRATE_COMMAND, raise_config_setup_error, report_validation_error+from pipelex.interpreter_plugins.builtins import BUILTIN_PLUGINS, CORE_UNCONDITIONAL_PLUGIN_NAMES, ENTRY_POINT_GROUPSfrom pipelex.kit.paths import get_kit_configs_dirfrom pipelex.migration.exceptions import MigrationErrorfrom pipelex.migration.run import config_directories_to_migrate, migrate_config_directories, scan_config_surface+from pipelex.plugins.discovery import build_registrar+from pipelex.plugins.exceptions import PluginError+from pipelex.plugins.log_sink_registry import LogSinkRegistryfrom pipelex.runtime_hub import RuntimeHub, get_console, set_runtime_hubfrom pipelex.system.configuration.config_loader import CONFIG_REFUSED, config_manager, pydantic_error_behindfrom pipelex.system.configuration.config_surface import PIPELEX_CONFIG_SURFACE_ID, TELEMETRY_CONFIG_SURFACE_ID, strip_reserved_metafrom pipelex.system.configuration.configs import PipelexConfig+from pipelex.system.console_target import ConsoleTargetfrom pipelex.system.environment import get_optional_envfrom pipelex.system.pipelex_service.exceptions import (RemoteConfigUnavailableError,@@ -61,7 +66,9 @@ from pipelex.system.pipelex_service.pipelex_service_config import ()from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcherfrom pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_FILE_NAME, TelemetryConfig+from pipelex.tools.log.console_log_sink import ConsoleLogSinkfrom pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_sink import LogSinkMethodfrom pipelex.tools.misc.dict_utils import extract_vars_from_strings_recursivefrom pipelex.tools.misc.exceptions import TomlErrorfrom pipelex.tools.misc.json_utils import deep_update@@ -672,6 +679,8 @@ def display_health_report(config_location: ConfigLocationInfo,fix_mode: bool = False,models_skipped: bool = False,+ log_sink_check: LogSinkCheck | None = None,+ plugins_check: PluginsCheck | None = None,) -> None:"""Display a comprehensive health report.@@ -695,9 +704,22 @@ def display_health_report(models_skipped: True when check_models was bypassed because config is broken — renderthe Models row as a yellow advisory and suppress its standalone Solutions entry,since the Config Files row already steers the user.+ log_sink_check: What the runtime setup found about ``[runtime.log] sink``; None when the+ setup never ran, in which case the row is not rendered.+ plugins_check: What the runtime setup found when it discovered the plugins; None when it+ never did, in which case the row is not rendered."""+ log_sink_healthy = log_sink_check is None or log_sink_check.is_healthy+ plugins_healthy = plugins_check is None or plugins_check.is_healthyall_healthy = (- config_healthy and pending_migrations_check.is_healthy and telemetry_check.is_healthy and backends_healthy and models_healthy and deck_healthy+ config_healthy+ and pending_migrations_check.is_healthy+ and telemetry_check.is_healthy+ and backends_healthy+ and models_healthy+ and deck_healthy+ and log_sink_healthy+ and plugins_healthy)# Overall status panel@@ -750,6 +772,26 @@ def display_health_report(console.print(f" [red]✗[/red] {escape(telemetry_check.message)}")console.print()+ # Plugins section: the discovery boot runs, so a plugin that stops it is a row rather than the+ # end of the report.+ if plugins_check is not None:+ console.print("[bold]Plugins[/bold]")+ if plugins_check.is_healthy:+ console.print(f" [green]✓[/green] {escape(plugins_check.message)}")+ else:+ console.print(f" [red]✗[/red] {escape(plugins_check.message)}")+ console.print()++ # Log Sink section: where this very report's lines go, so a token nobody registered is a row+ # rather than the end of the report.+ if log_sink_check is not None:+ console.print("[bold]Log Sink[/bold]")+ if log_sink_check.is_healthy:+ console.print(f" [green]✓[/green] {escape(log_sink_check.message)}")+ else:+ console.print(f" [red]✗[/red] {escape(log_sink_check.message)}")+ console.print()+# Backend Credentials sectionconsole.print("[bold]Backend Credentials[/bold]")if backends_healthy:@@ -856,6 +898,8 @@ def display_health_report(has_deck_drift = not deck_healthyhas_recommendations = (can_auto_fix_config+ or not log_sink_healthy+ or not plugins_healthyor can_auto_fix_telemetryor can_migrateor migrations_need_a_look@@ -902,6 +946,16 @@ def display_health_report(if has_deck_drift:console.print(" • Run [cyan]pipelex update[/cyan] to refresh the model deck from the current kit")+ if not plugins_healthy:+ console.print(+ " • Fix, upgrade or uninstall the plugin named in the Plugins row, or take a core plugin out of runtime.plugins.disabled"+ )++ if not log_sink_healthy:+ console.print(+ f" • Set [cyan]sink[/cyan] in {escape('[runtime.log]')} to a registered log sink, or fix what the Log Sink row says stopped it"+ )+# Backend file issuesif can_auto_fix_backends:if fix_mode:@@ -1012,7 +1066,123 @@ def check_deck_sync(*, config_dir: Path | None = None) -> tuple[bool, DeckSyncRereturn False, report, f"{len(actionable)} deck file(s) need action"-def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = None, config_dir: Path | None = None) -> None:+class LogSinkCheck(BaseModel):+ """What the doctor found about ``[runtime.log] sink``: the configured sink installed, or the fallback and why."""++ model_config = ConfigDict(frozen=True)++ is_healthy: bool+ message: str+++class PluginsCheck(BaseModel):+ """What the doctor found when it discovered the plugins the way boot does: the registry built, or what stopped it."""++ model_config = ConfigDict(frozen=True)++ is_healthy: bool+ message: str+++class DoctorRuntimeSetup(BaseModel):+ """What ``setup_doctor_runtime`` found on its way to a configured logger, one row each.++ ``plugins`` is ``None`` when logging was already configured in this process, since the doctor then+ keeps the sink in place and never discovers the plugins.+ """++ model_config = ConfigDict(frozen=True)++ log_sink: LogSinkCheck+ plugins: PluginsCheck | None+++FALLBACK_LOG_SINK_NOTE = f"this report goes through the '{LogSinkMethod.CONSOLE}' sink on stderr instead"+FALLBACK_LOG_SINK_REFUSED_NOTE = (+ "a log sink is already recorded, so the console fallback could not stand in and this report's own lines go wherever that one sends them"+)+++def _install_fallback_log_sink(*, log_config: LogConfig) -> str:+ """Install the sink the doctor's own lines fall back to, the console sink pinned to stderr, and say what happened.++ Pinned rather than built from the configuration, because the configuration may be exactly what+ made the configured sink fail: a console target no sink can write to fails the console sink too+ when it reads the same field.++ Refused when a sink is already recorded, because ``log.install_sink`` refuses a second one and the+ doctor would then raise in place of the failure it was called to report. That is reachable rather+ than theoretical: ``install_sink`` records the sink *before* replaying what the holding handler+ held, deliberately, so that a replay which raises still leaves the sink findable by ``reset`` — so+ a failed installation does not always leave nothing behind. The returned note is what the caller+ interpolates, so a row never promises a fallback that did not stand in.+ """+ if log.sink is not None:+ return FALLBACK_LOG_SINK_REFUSED_NOTE+ log.install_sink(ConsoleLogSink(rich_log_config=log_config.rich_log, target=ConsoleTarget.STDERR))+ return FALLBACK_LOG_SINK_NOTE+++def install_doctor_log_sink(*, registry: LogSinkRegistry | None, log_config: LogConfig) -> LogSinkCheck:+ """Install the configured sink, or the console sink on stderr with a finding when it cannot be.++ Boot stops on an unregistered token, on a factory that raises and on a registry that did not+ build. The doctor must not, since where its own lines go is one of the things it diagnoses: it+ installs the console sink on stderr instead and reports why in a row of its own, so the rest of+ the report is still produced. The retry is usually safe because ``log.install_sink`` builds the+ handler before it touches the root logger, so a sink that failed to build left nothing installed —+ but not always, which is why the fallback is the one that decides whether it can stand in and+ hands back the note the row carries.+ """+ if registry is None:+ note = _install_fallback_log_sink(log_config=log_config)+ return LogSinkCheck(+ is_healthy=False,+ message=f"The log sink '{log_config.sink}' could not be resolved because the plugin registry did not build; {note}",+ )+ if not registry.has(method=log_config.sink):+ note = _install_fallback_log_sink(log_config=log_config)+ return LogSinkCheck(+ is_healthy=False,+ message=(f"No log sink is registered for '{log_config.sink}' in [runtime.log]; {note}. Registered sinks: {', '.join(registry.methods)}"),+ )+ try:+ log.install_sink(registry.get_required(method=log_config.sink)(log_config))+ except Exception as exc: # ruff: ignore[blind-except]+ # A factory or a handler that raises on this configuration, a console target no sink writes+ # to or a dependency the sink needs: the row says so, and the report goes on.+ note = _install_fallback_log_sink(log_config=log_config)+ return LogSinkCheck(is_healthy=False, message=f"The log sink '{log_config.sink}' could not be installed: {exc}; {note}")+ return LogSinkCheck(is_healthy=True, message=f"Log sink '{log_config.sink}' installed")+++def discover_plugins_and_install_doctor_log_sink(*, log_config: LogConfig) -> DoctorRuntimeSetup:+ """Discover the plugins the way boot does, then install the configured sink, each a row rather than the end of the report.++ ``build_registrar`` is fail-loud: a plugin built against another plugin API, a broken third-party+ module or a core plugin named in ``runtime.plugins.disabled`` raises a ``PluginError``, which is not+ a configuration error and which the doctor exists to diagnose rather than die on.+ """+ try:+ registrar = build_registrar(+ config=get_config(),+ boot_orchestrator=None,+ builtin_plugins=BUILTIN_PLUGINS,+ core_unconditional_plugin_names=CORE_UNCONDITIONAL_PLUGIN_NAMES,+ entry_point_groups=ENTRY_POINT_GROUPS,+ )+ except PluginError as exc:+ return DoctorRuntimeSetup(+ plugins=PluginsCheck(is_healthy=False, message=f"The plugin registry did not build: {exc.message}"),+ log_sink=install_doctor_log_sink(registry=None, log_config=log_config),+ )+ return DoctorRuntimeSetup(+ plugins=PluginsCheck(is_healthy=True, message="Plugins discovered and registered"),+ log_sink=install_doctor_log_sink(registry=LogSinkRegistry(registrar.log_sinks), log_config=log_config),+ )+++def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = None, config_dir: Path | None = None) -> DoctorRuntimeSetup:"""Spin up a fresh RuntimeHub and configure logging for doctor checks.Doctor intentionally bypasses ``Pipelex.make`` so it can diagnose a broken config@@ -1033,7 +1203,13 @@ def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = Non``log.configure`` is invoked through ``configure_if_unset`` so that if a libraryembedding or interleaved test has already configured logging, this call no-ops- instead of raising the once-per-process ``RuntimeError``.+ instead of raising the once-per-process ``RuntimeError``. When it does apply, the+ sink is selected the way boot selects it: the pure ``build_registrar`` discovery,+ then the ``runtime.log.sink`` lookup — the doctor bypasses ``Pipelex.make`` but not+ the configuration's choice of where its own lines go. Where boot would stop, on a+ registry that does not build, a token nobody registered or a sink that fails to+ install, the doctor installs the console sink on stderr and says so in the rows it+ returns.Args:log_config_overrides: Optional mapping of ``LogConfig`` field names → values to@@ -1041,6 +1217,12 @@ def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = Nonconfig_dir: Optional explicit config dir (e.g. for ``--global``). When provided,project/global layering is bypassed and only this directory is read.+ Returns:+ The plugins row and the log sink row. The log sink row is healthy when the configured+ sink was installed, or when logging was already configured and its sink stays; not+ healthy when the console sink on stderr stands in, and the message says why. The+ plugins row is ``None`` when logging was already configured.+Raises:PipelexConfigError: If config validation fails. Translation of``pydantic.ValidationError`` to keep doctor's error surface stable.@@ -1062,10 +1244,16 @@ def setup_doctor_runtime(*, log_config_overrides: Mapping[str, Any] | None = Nondeep_update(merged, updates=log_config_overrides)log_config = LogConfig.model_validate(merged)runtime_hub.set_console_print_target(target=log_config.console_print_target)- log.configure_if_unset(log_config=log_config)+ runtime_setup = DoctorRuntimeSetup(+ log_sink=LogSinkCheck(is_healthy=True, message="Logging was already configured in this process, and its sink stays"),+ plugins=None,+ )+ if log.configure_if_unset(log_config=log_config):+ runtime_setup = discover_plugins_and_install_doctor_log_sink(log_config=log_config)runtime_hub.set_pretty_print_mode(mode=log_config.pretty_print_mode)if (stale_warning := config_manager.take_stale_configuration_warning()) is not None:log.warning(stale_warning)+ return runtime_setupdef check_models(*, config_dir: Path | None = None) -> tuple[bool, str, dict[str, BackendFileReport]]:@@ -1179,6 +1367,10 @@ def doctor_cmd(fix: If True, offer to fix detected issues interactively"""console = get_console()+ # The doctor configures logging and installs a sink for its own report; once the report is out,+ # it releases them, so an ``otlp`` sink's exporter and the hook it registered at exit go with the+ # command rather than outliving it. Logging an embedder configured before calling in is left alone.+ logging_was_configured = log.is_configuredtry:do_doctor_cmd(fix=fix)@@ -1192,6 +1384,9 @@ def doctor_cmd(console.print(" [cyan]https://go.pipelex.com/discord[/cyan] - Discord Community")console.print()sys.exit(1)+ finally:+ if not logging_was_configured:+ log.reset()def do_doctor_cmd(@@ -1227,9 +1422,13 @@ def do_doctor_cmd(models_healthy: boolmodels_message: strbackend_file_reports: dict[str, BackendFileReport]+ log_sink_check: LogSinkCheck | None = None+ plugins_check: PluginsCheck | None = Noneif config_healthy:try:- setup_doctor_runtime()+ runtime_setup = setup_doctor_runtime()+ log_sink_check = runtime_setup.log_sink+ plugins_check = runtime_setup.pluginsmodels_healthy, models_message, backend_file_reports = check_models()models_skipped = Falseexcept PipelexConfigError as exc:@@ -1264,10 +1463,19 @@ def do_doctor_cmd(deck_report=deck_report,config_location=config_location,fix_mode=fix,+ log_sink_check=log_sink_check,+ plugins_check=plugins_check,)all_healthy = (- config_healthy and pending_migrations_check.is_healthy and telemetry_check.is_healthy and backends_healthy and models_healthy and deck_healthy+ config_healthy+ and pending_migrations_check.is_healthy+ and telemetry_check.is_healthy+ and backends_healthy+ and models_healthy+ and deck_healthy+ and (log_sink_check is None or log_sink_check.is_healthy)+ and (plugins_check is None or plugins_check.is_healthy))# Exit code: 0 if healthy, 1 if issues founddiff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex aa4fbe29e..dd73e4c41 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -96,6 +96,10 @@ console_log_target = "stderr"# targets to stderr internally to keep its stdout strictly reserved for the success# envelope.console_print_target = "stdout"+# sink names where the records go: "console" (the Rich handler, for a terminal), "json" (one JSON+# object per line, for a server behind a log agent), "otlp" (the OpenTelemetry logs signal, for a+# collector), or a token an installed plugin registers.+sink = "console"[runtime.log.package_log_levels]# Log levels for specific packages (use "-" instead of "." in package names)diff --git a/pipelex/migration/goldens/pipelex-config/[email protected] b/pipelex/migration/goldens/pipelex-config/[email protected]new file mode 100644index 000000000..3fb0bd507--- /dev/null+++ b/pipelex/migration/goldens/pipelex-config/[email protected]@@ -0,0 +1,455 @@+####################################################################################################+# Runtime — kernel-layer, process-scoped infrastructure (what runtime_hub brokers)+####################################################################################################++[runtime]++[runtime.storage]+is_fetch_remote_content_enabled = true+is_upload_local_content_enabled = true+method = "local" # local, in_memory, s3, gcp++[runtime.storage.local]+uri_format = "{hash}.{extension}"+local_storage_path = ".pipelex/storage"++[runtime.storage.in_memory]+uri_format = "{hash}.{extension}"++[runtime.storage.s3]+uri_format = "{hash}.{extension}"+bucket_name = ""+region = ""+signed_urls_lifespan_seconds = 3600++[runtime.storage.gcp]+uri_format = "{hash}.{extension}"+bucket_name = ""+project_id = ""+signed_urls_lifespan_seconds = 3600++[runtime.secrets]+method = "env" # env (built-in); an external plugin may add e.g. vault++####################################################################################################+# Log config+####################################################################################################++[runtime.log]+default_log_level = "INFO"+# sink names the registered log sink boot installs on the root logger, the way runtime.storage.method+# names a storage backend: "console" (the Rich handler, for a terminal), "json" (one JSON object per+# line, for a server behind a log agent), "otlp" (the OpenTelemetry logs signal, for a collector), or a+# token an installed plugin registers. "console" is the default so the CLI keeps its rendering; a+# server selects "json". An unregistered token stops the boot naming the registered ones.+sink = "console"+# console_log_target controls the stream the console and json sinks write to (log.debug/info/...).+# Logs are diagnostics → stderr.+console_log_target = "stderr"+# console_print_target controls the Console returned by get_console(), used for banners,+# deck notices, and the main `pipelex` CLI's data tables (`show backends`, `show models`,+# `which`, `doctor`). Tables are data the user pipes to a file → stdout.+# Note: the agent CLI (`pipelex-agent`) emits its actual JSON/markdown results via bare+# builtin `print(...)` to sys.stdout, bypassing Rich entirely — so this knob does NOT+# affect the agent CLI data channel. The agent CLI factory additionally forces both+# targets to stderr internally to keep its stdout strictly reserved for the success+# envelope (see `_AGENT_CLI_STDERR_CONSOLE_OVERRIDES` in `agent_cli_factory.py`).+console_print_target = "stdout"++json_logs_indent = 4+presentation_line_width = 120+silenced_problem_ids = ["azure_openai_no_stream_options"]+caller_info_template = "file_line"+is_caller_info_enabled = false++# pretty_print_mode controls the panels `pretty_print(...)` renders, the "Output of pipe" panel after+# every operator pipe among them: "rich" renders Rich tables, "poor" prints plain text, "silent" prints+# nothing and builds no renderable at all. A host with no console, or one that runs pipes on a thread+# that must not stall, sets "silent"; the agent CLI forces it.+pretty_print_mode = "rich"++[runtime.log.package_log_levels]+# for logger names, the dots "." have been replaced by "-" to avoid toml parsing issues+# TODO: use quotes to re-enable names with dots+anthropic = "INFO"+asyncio = "INFO"+botocore = "INFO"+botocore-credentials = "WARNING"+google = "INFO"+httpx = "WARNING"+httpcore = "INFO"+openai = "INFO"+instructor = "INFO"+urllib3-connectionpool = "INFO"+urllib3-util-retry = "INFO"++pipelex = "INFO"+++[runtime.log.rich_log]+is_show_time = false+is_show_level = true+is_link_path_enabled = true+is_markup_enabled = true+highlighter_name = "json"+is_rich_tracebacks = true+is_tracebacks_word_wrap = true+is_tracebacks_show_locals = false+tracebacks_suppress = []+keywords_to_hilight = []++# The settings of the "otlp" sink. An endpoint key, when set, names the collector's logs URL+# (for instance "http://localhost:4318/v1/logs"); left unset, the exporter follows the OpenTelemetry+# environment conventions: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, then OTEL_EXPORTER_OTLP_ENDPOINT, then+# the collector default on localhost. Empty headers likewise leave OTEL_EXPORTER_OTLP_HEADERS in charge.+[runtime.log.otlp]+headers = {}++[runtime.aws]+api_key_method = "env"+bedrock_access_variant = "aws_access"++[runtime.reporting]+is_log_costs_to_console = true+is_generate_cost_report_file_enabled = false+cost_report_dir_path = "reports"+cost_report_base_name = "cost_report"+cost_report_extension = "csv"+cost_report_unit_scale = 1.0++####################################################################################################+# Tracing config (event log for distributed tracing across workers)+####################################################################################################++[runtime.tracing]+is_enabled = true+backend = "ndjson"++[runtime.tracing.ndjson]+traces_dir = ".pipelex/traces"++[runtime.tracing.dynamodb]+table_name = ""+region = ""++[runtime.observer]+observer_dir = "results/observer"++[runtime.plugins]+# Names of discovered plugins to skip at startup (a denylist). Discovery is the+# source of truth for which plugins are present; this only turns one off.+disabled = []++####################################################################################################+# Inference — the model-calling seam (was [cogt])+####################################################################################################++[inference]+transport_max_retries = 2++[inference.model_deck]+is_model_fallback_enabled = true+missing_presets_reaction = "log"++[inference.llm]+default_max_images = 100+is_structure_prompt_enabled = true+is_dump_text_prompts_enabled = false+is_dump_response_text_enabled = false+schema_reask_max_attempts = 3++[inference.llm.instructor]+is_dump_kwargs_enabled = false+is_dump_response_enabled = false+is_dump_error_enabled = false++[inference.llm.openai.effort_to_level_map]+none = "none"+minimal = "minimal"+low = "low"+medium = "medium"+high = "high"+xhigh = "xhigh"+max = "xhigh"++[inference.llm.anthropic]+structured_output_timeout_seconds = 1200++[inference.llm.anthropic.effort_to_level_map]+none = "disabled"+minimal = "low"+low = "low"+medium = "medium"+high = "high"+xhigh = "xhigh"+max = "max"++[inference.llm.google.effort_to_level_map]+none = "disabled"+minimal = "minimal"+low = "low"+medium = "medium"+high = "high"+xhigh = "high"+max = "high"++[inference.llm.mistral.effort_to_level_map]+none = "disabled"+minimal = "reasoning"+low = "reasoning"+medium = "reasoning"+high = "reasoning"+xhigh = "reasoning"+max = "reasoning"++[inference.llm.effort_to_budget_maps.anthropic]+none = 0 # Required by validator; unreachable at runtime (level map gates NONE as disabled before budget lookup)+minimal = 512+low = 1024+medium = 5000+high = 16384+xhigh = 32768+max = 65536++[inference.llm.effort_to_budget_maps.gemini]+none = 0 # Required by validator; unreachable at runtime (level map gates NONE as disabled before budget lookup)+minimal = 512+low = 1024+medium = 5000+high = 16384+xhigh = 32768+max = 65536++[inference.llm.generic_templates]+output_structure_prompt = """+++---+The instance we want to generate will be for the following class:+{{ class_structure_str }}++DO NOT create information.+If some information is not present for an attribute, output the default value or None according to the attribute definition.+"""++structuring_prompt = """+Read the following text carefully and produce the requested structured output from it.++---+{{ text }}+"""++####################################################################################################+# Image generation config+####################################################################################################++[inference.img_gen.img_gen_job]+is_sync_mode = false++[inference.img_gen.img_gen_param_defaults]+nb_steps = 28 # 28 is a good default for Flux, possible values are [1,2,4,8] for SDXL Lightning+quality = "low"+aspect_ratio = "square" # "square", "landscape_4_3", "landscape_3_2", "landscape_16_9", "landscape_21_9", "landscape_4_1", "landscape_8_1", "portrait_3_4", "portrait_2_3", "portrait_9_16", "portrait_9_21", "portrait_1_4", "portrait_1_8"+guidance_scale = 2.5+background = "auto"+is_moderated = true+safety_tolerance = 5+is_raw = false+seed = "auto"++[inference.img_gen.quality_to_steps_maps]+flux = { "low" = 14, "medium" = 28, "high" = 56 }+sdxl_lightning = { "low" = 2, "medium" = 4, "high" = 8 }+qwen_image = { "low" = 10, "medium" = 20, "high" = 28 }++####################################################################################################+# Extract config+####################################################################################################++[inference.extract]+default_page_views_dpi = 72++####################################################################################################+# Gateway test config+####################################################################################################++[inference.gateway_test.config_id_substitutions]+# this is an empty dict, it's only used by Pipelex staff to override the ¨Pipelex Gateway¨ config+# in order to test new model integrations or debug issues with the Gateway++####################################################################################################+# Templating config+####################################################################################################++[inference.templating]+default_templating_style = { tag_style = "xml" }++####################################################################################################+# Dry run config+####################################################################################################++[inference.dry_run]+text_gen_truncate_length = 256+nb_list_items = 3+nb_extract_pages = 4+allowed_to_fail_pipes = [+ "failing_pipelines.infinite_loop_1", # Loop but only for testing purposes — matched on the namespaced pipe_ref+]+image_urls = [+ "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_1.jpg",+ "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_2.png",+]++####################################################################################################+# Interpreter — library-scoped method machinery (what interpreter_hub brokers)+####################################################################################################++[interpreter]++####################################################################################################+# MTHDS config+####################################################################################################++[interpreter.mthds.inline_tables]+spaces_inside_curly_braces = true++[interpreter.mthds.strings]+prefer_literal = false+force_multiline = false+length_limit_to_multiline = 100+ensure_trailing_newline = true+ensure_leading_blank_line = true++[interpreter.mthds.concepts]+structure_field_ordering = [+ "type",+ "concept_ref",+ "item_type",+ "item_concept_ref",+ "description",+ "choices",+ "required",+]++[interpreter.mthds.pipes]+field_ordering = ["type", "description", "inputs", "output"]++####################################################################################################+# Methods config+####################################################################################################++[interpreter.methods]+# When a bundle references another method by address (github.com/...->domain.pipe) and no installed+# method matches, fetch the package by address and install it into ~/.mthds/methods/ so the load can+# proceed. Set to false to forbid network fetches at load time — a miss then raises a diagnostic+# telling you the address and how to install it manually. The PIPELEX_METHODS_FETCH_ON_MISS+# environment variable (1/0, true/false) overrides this setting when set.+fetch_on_miss = true++####################################################################################################+# Pipe run config+####################################################################################################++[interpreter.pipe_run]+pipe_stack_limit = 20++####################################################################################################+# PipeFunc config+####################################################################################################++[interpreter.pipe_func]+# PipeFunc execution mode, selected from the PipeFuncExecutorRegistry the plugins populate. "direct"+# (core) imports and runs customer functions in this process; "daytona" (our Daytona sandbox plugin)+# runs them in a remote box. Any non-"direct" mode carries customer PipeFunc code as source text and+# executes it only inside an isolated sandbox, never imported here.+execution_mode = "direct"++####################################################################################################+# Pipeline execution config+####################################################################################################++[interpreter.pipeline_execution]+is_normalize_data_urls_to_storage = true+is_mock_inputs = false+is_generate_graph = true+is_generate_usage = true++# Bounded fan-out concurrency for PipeBatch: at most this many branches run at once.+# A modest default keeps a large workload (one pipe over many items) from overwhelming+# asyncio, memory, and provider rate limits. Set to "unbounded" to disable the bound.+max_concurrency = 8++[interpreter.pipeline_execution.graph]++[interpreter.pipeline_execution.graph.data_inclusion]+stuff_json_content = true+error_stack_traces = true+pipe_and_concept_registry = true++[interpreter.pipeline_execution.graph.graphs_inclusion]+graphspec_json = true+mermaidflow_mmd = true+mermaidflow_html = true+reactflow_html = true++[interpreter.pipeline_execution.graph.mermaid]+direction = "left_to_right"+is_include_data_edges = true+is_include_contains_edges = false+is_include_selected_outcome_edges = true+is_show_stuff_codes = false++[interpreter.pipeline_execution.graph.mermaid.style]+theme = "dark"++[interpreter.pipeline_execution.graph.reactflow]+is_use_cdn = true+layout_direction = "left_to_right"+nodesep = 50+ranksep = 30+edge_type = "bezier"+initial_zoom = 1.0+pan_to_top = true+default_title = "Pipelex Graph"+# Forwarded to mthds-ui as `showControllers`: when true, controller pipes (PipeSequence,+# PipeParallel, PipeCondition, PipeBatch) render as grouping containers around their children;+# when false, they collapse out of the graph and edges connect stuff nodes directly.+show_batch_controller = true++[interpreter.pipeline_execution.graph.reactflow.style]+theme = "dark"+palette = "dracula"++[interpreter.scan]+excluded_dirs = [+ ".venv",+ "venv",+ "env",+ ".env",+ "virtualenv",+ ".virtualenv",+ ".git",+ "__pycache__",+ ".pytest_cache",+ ".mypy_cache",+ ".ruff_cache",+ "node_modules",+ "results",+]++[interpreter.builder]+fix_loop_max_attempts = 10+default_output_dir = "."+default_bundle_file_name = "bundle"+default_directory_base_name = "pipeline"++####################################################################################################+# Kit Config+####################################################################################################++[kit]+# Agent rule files to install (used by `pipelex-dev kit rules`)+# Each entry is one of: "cursor", "agents", "claude"+# - "cursor" installs .cursor/rules/*.mdc files (cannot be combined with others)+# - "agents" / "claude" install AGENTS.md / CLAUDE.md respectively — both can be listed together+preferred_agent_targets = ["claude", "agents"]diff --git a/pipelex/migration/goldens/pipelex-config/[email protected] b/pipelex/migration/goldens/pipelex-config/[email protected]new file mode 100644index 000000000..cfe308d3e--- /dev/null+++ b/pipelex/migration/goldens/pipelex-config/[email protected]@@ -0,0 +1,1456 @@+{+ "surface_id": "pipelex-config",+ "schema_version": 5,+ "document_root_is_open": false,+ "paths": {+ "inference": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.dry_run": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.dry_run.allowed_to_fail_pipes": {+ "value_type": "list[str]",+ "required": false,+ "default": [+ "failing_pipelines.infinite_loop_1"+ ],+ "open_node": false+ },+ "inference.dry_run.image_urls": {+ "value_type": "list[str]",+ "required": true,+ "default": [+ "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_1.jpg",+ "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_2.png"+ ],+ "open_node": false+ },+ "inference.dry_run.nb_extract_pages": {+ "value_type": "int",+ "required": true,+ "default": 4,+ "open_node": false+ },+ "inference.dry_run.nb_list_items": {+ "value_type": "int",+ "required": true,+ "default": 3,+ "open_node": false+ },+ "inference.dry_run.text_gen_truncate_length": {+ "value_type": "int",+ "required": true,+ "default": 256,+ "open_node": false+ },+ "inference.extract": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.extract.default_page_views_dpi": {+ "value_type": "int",+ "required": true,+ "default": 72,+ "open_node": false+ },+ "inference.gateway_test": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.gateway_test.config_id_substitutions": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {},+ "open_node": true+ },+ "inference.gateway_test.config_id_substitutions.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "inference.img_gen": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.img_gen.img_gen_job": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.img_gen.img_gen_job.is_sync_mode": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.aspect_ratio": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "landscape_16_9",+ "landscape_21_9",+ "landscape_3_2",+ "landscape_4_1",+ "landscape_4_3",+ "landscape_8_1",+ "portrait_1_4",+ "portrait_1_8",+ "portrait_2_3",+ "portrait_3_4",+ "portrait_9_16",+ "portrait_9_21",+ "square"+ ],+ "default": "square",+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.background": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "auto",+ "opaque",+ "transparent"+ ],+ "default": "auto",+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.guidance_scale": {+ "value_type": "float",+ "required": true,+ "default": 2.5,+ "open_node": false,+ "constraints": {+ "gt": 0+ }+ },+ "inference.img_gen.img_gen_param_defaults.is_moderated": {+ "value_type": "bool",+ "required": false,+ "default": true,+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.is_raw": {+ "value_type": "bool",+ "required": false,+ "default": false,+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.nb_steps": {+ "value_type": "int",+ "required": false,+ "default": 28,+ "open_node": false,+ "constraints": {+ "gt": 0+ }+ },+ "inference.img_gen.img_gen_param_defaults.quality": {+ "value_type": "enum",+ "required": false,+ "enum_members": [+ "high",+ "low",+ "medium"+ ],+ "default": "low",+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.safety_tolerance": {+ "value_type": "int",+ "required": true,+ "default": 5,+ "open_node": false,+ "constraints": {+ "ge": 1,+ "le": 6+ }+ },+ "inference.img_gen.img_gen_param_defaults.seed": {+ "value_type": "int | literal",+ "required": true,+ "enum_members": [+ "auto"+ ],+ "default": "auto",+ "open_node": false+ },+ "inference.img_gen.img_gen_param_defaults.size": {+ "value_type": "enum | table",+ "required": false,+ "enum_members": [+ "0.5k",+ "1k",+ "2k",+ "4k"+ ],+ "open_node": false+ },+ "inference.img_gen.quality_to_steps_maps": {+ "value_type": "dict[str, dict[str, int]]",+ "required": true,+ "default": {+ "flux": {+ "low": 14,+ "medium": 28,+ "high": 56+ },+ "sdxl_lightning": {+ "low": 2,+ "medium": 4,+ "high": 8+ },+ "qwen_image": {+ "low": 10,+ "medium": 20,+ "high": 28+ }+ },+ "open_node": true+ },+ "inference.img_gen.quality_to_steps_maps.*": {+ "value_type": "dict[str, int]",+ "required": true,+ "open_node": true+ },+ "inference.img_gen.quality_to_steps_maps.*.*": {+ "value_type": "int",+ "required": true,+ "open_node": false+ },+ "inference.llm": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.llm.anthropic": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.llm.anthropic.effort_to_level_map": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {+ "none": "disabled",+ "minimal": "low",+ "low": "low",+ "medium": "medium",+ "high": "high",+ "xhigh": "xhigh",+ "max": "max"+ },+ "open_node": true+ },+ "inference.llm.anthropic.effort_to_level_map.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "inference.llm.anthropic.structured_output_timeout_seconds": {+ "value_type": "int",+ "required": true,+ "default": 1200,+ "open_node": false+ },+ "inference.llm.default_max_images": {+ "value_type": "int",+ "required": true,+ "default": 100,+ "open_node": false+ },+ "inference.llm.effort_to_budget_maps": {+ "value_type": "dict[str, dict[str, int]]",+ "required": true,+ "default": {+ "anthropic": {+ "none": 0,+ "minimal": 512,+ "low": 1024,+ "medium": 5000,+ "high": 16384,+ "xhigh": 32768,+ "max": 65536+ },+ "gemini": {+ "none": 0,+ "minimal": 512,+ "low": 1024,+ "medium": 5000,+ "high": 16384,+ "xhigh": 32768,+ "max": 65536+ }+ },+ "open_node": true+ },+ "inference.llm.effort_to_budget_maps.*": {+ "value_type": "dict[str, int]",+ "required": true,+ "open_node": true+ },+ "inference.llm.effort_to_budget_maps.*.*": {+ "value_type": "int",+ "required": true,+ "open_node": false+ },+ "inference.llm.generic_templates": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {+ "output_structure_prompt": "\n\n---\nThe instance we want to generate will be for the following class:\n{{ class_structure_str }}\n\nDO NOT create information.\nIf some information is not present for an attribute, output the default value or None according to the attribute definition.\n",+ "structuring_prompt": "Read the following text carefully and produce the requested structured output from it.\n\n---\n{{ text }}\n"+ },+ "open_node": true+ },+ "inference.llm.generic_templates.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "inference.llm.google": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.llm.google.effort_to_level_map": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {+ "none": "disabled",+ "minimal": "minimal",+ "low": "low",+ "medium": "medium",+ "high": "high",+ "xhigh": "high",+ "max": "high"+ },+ "open_node": true+ },+ "inference.llm.google.effort_to_level_map.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "inference.llm.instructor": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.llm.instructor.is_dump_error_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "inference.llm.instructor.is_dump_kwargs_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "inference.llm.instructor.is_dump_response_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "inference.llm.is_dump_response_text_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "inference.llm.is_dump_text_prompts_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "inference.llm.is_structure_prompt_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "inference.llm.mistral": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.llm.mistral.effort_to_level_map": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {+ "none": "disabled",+ "minimal": "reasoning",+ "low": "reasoning",+ "medium": "reasoning",+ "high": "reasoning",+ "xhigh": "reasoning",+ "max": "reasoning"+ },+ "open_node": true+ },+ "inference.llm.mistral.effort_to_level_map.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "inference.llm.openai": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.llm.openai.effort_to_level_map": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {+ "none": "none",+ "minimal": "minimal",+ "low": "low",+ "medium": "medium",+ "high": "high",+ "xhigh": "xhigh",+ "max": "xhigh"+ },+ "open_node": true+ },+ "inference.llm.openai.effort_to_level_map.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "inference.llm.schema_reask_max_attempts": {+ "value_type": "int",+ "required": true,+ "default": 3,+ "open_node": false,+ "constraints": {+ "ge": 1,+ "le": 10+ }+ },+ "inference.model_deck": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.model_deck.is_model_fallback_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "inference.model_deck.missing_presets_reaction": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "log",+ "none",+ "raise"+ ],+ "default": "log",+ "open_node": false+ },+ "inference.templating": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.templating.default_templating_style": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "inference.templating.default_templating_style.tag_style": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "no_tag",+ "square_brackets",+ "ticks",+ "xml"+ ],+ "default": "xml",+ "open_node": false+ },+ "inference.templating.default_templating_style.text_format": {+ "value_type": "enum",+ "required": false,+ "enum_members": [+ "html",+ "json",+ "markdown",+ "plain"+ ],+ "open_node": false+ },+ "inference.transport_max_retries": {+ "value_type": "int",+ "required": true,+ "default": 2,+ "open_node": false,+ "constraints": {+ "ge": 0,+ "le": 10+ }+ },+ "interpreter": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.builder": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.builder.default_bundle_file_name": {+ "value_type": "str",+ "required": true,+ "default": "bundle",+ "open_node": false+ },+ "interpreter.builder.default_directory_base_name": {+ "value_type": "str",+ "required": true,+ "default": "pipeline",+ "open_node": false+ },+ "interpreter.builder.default_output_dir": {+ "value_type": "str",+ "required": true,+ "default": ".",+ "open_node": false+ },+ "interpreter.builder.fix_loop_max_attempts": {+ "value_type": "int",+ "required": true,+ "default": 10,+ "open_node": false+ },+ "interpreter.methods": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.methods.fetch_on_miss": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.mthds": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.mthds.concepts": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.mthds.concepts.structure_field_ordering": {+ "value_type": "list[str]",+ "required": true,+ "default": [+ "type",+ "concept_ref",+ "item_type",+ "item_concept_ref",+ "description",+ "choices",+ "required"+ ],+ "open_node": false+ },+ "interpreter.mthds.inline_tables": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.mthds.inline_tables.spaces_inside_curly_braces": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.mthds.pipes": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.mthds.pipes.field_ordering": {+ "value_type": "list[str]",+ "required": true,+ "default": [+ "type",+ "description",+ "inputs",+ "output"+ ],+ "open_node": false+ },+ "interpreter.mthds.strings": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.mthds.strings.ensure_leading_blank_line": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.mthds.strings.ensure_trailing_newline": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.mthds.strings.force_multiline": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "interpreter.mthds.strings.length_limit_to_multiline": {+ "value_type": "int",+ "required": true,+ "default": 100,+ "open_node": false+ },+ "interpreter.mthds.strings.prefer_literal": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "interpreter.pipe_func": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipe_func.execution_mode": {+ "value_type": "str",+ "required": true,+ "default": "direct",+ "open_node": false+ },+ "interpreter.pipe_run": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipe_run.pipe_stack_limit": {+ "value_type": "int",+ "required": true,+ "default": 20,+ "open_node": false+ },+ "interpreter.pipeline_execution": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.data_inclusion": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.data_inclusion.error_stack_traces": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.data_inclusion.pipe_and_concept_registry": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.data_inclusion.stuff_json_content": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.graphs_inclusion": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.graphs_inclusion.graphspec_json": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.graphs_inclusion.mermaidflow_html": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.graphs_inclusion.mermaidflow_mmd": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.graphs_inclusion.reactflow_html": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.direction": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "left_to_right",+ "top_down"+ ],+ "default": "left_to_right",+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.is_include_contains_edges": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.is_include_data_edges": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.is_include_selected_outcome_edges": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.is_show_stuff_codes": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.style": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.mermaid.style.theme": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "base",+ "dark",+ "default",+ "forest",+ "neutral"+ ],+ "default": "dark",+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.default_title": {+ "value_type": "str",+ "required": true,+ "default": "Pipelex Graph",+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.edge_type": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "bezier",+ "smoothstep",+ "step",+ "straight"+ ],+ "default": "bezier",+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.initial_zoom": {+ "value_type": "float",+ "required": true,+ "default": 1.0,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.is_use_cdn": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.layout_direction": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "left_to_right",+ "top_down"+ ],+ "default": "left_to_right",+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.nodesep": {+ "value_type": "int",+ "required": true,+ "default": 50,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.pan_to_top": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.ranksep": {+ "value_type": "int",+ "required": true,+ "default": 30,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.show_batch_controller": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.style": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.style.palette": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "dracula",+ "yellow_blue"+ ],+ "default": "dracula",+ "open_node": false+ },+ "interpreter.pipeline_execution.graph.reactflow.style.theme": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "dark",+ "light",+ "system"+ ],+ "default": "dark",+ "open_node": false+ },+ "interpreter.pipeline_execution.is_generate_graph": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.is_generate_usage": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.is_mock_inputs": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "interpreter.pipeline_execution.is_normalize_data_urls_to_storage": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "interpreter.pipeline_execution.max_concurrency": {+ "value_type": "int | literal",+ "required": true,+ "enum_members": [+ "unbounded"+ ],+ "default": 8,+ "open_node": false,+ "constraints": {+ "ge": 1+ }+ },+ "interpreter.scan": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "interpreter.scan.excluded_dirs": {+ "value_type": "frozenset[str]",+ "required": true,+ "default": [+ ".venv",+ "venv",+ "env",+ ".env",+ "virtualenv",+ ".virtualenv",+ ".git",+ "__pycache__",+ ".pytest_cache",+ ".mypy_cache",+ ".ruff_cache",+ "node_modules",+ "results"+ ],+ "open_node": false+ },+ "kit": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "kit.preferred_agent_targets": {+ "value_type": "list[enum]",+ "required": true,+ "enum_members": [+ "agents",+ "claude",+ "cursor"+ ],+ "default": [+ "claude",+ "agents"+ ],+ "open_node": false+ },+ "runtime": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.aws": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.aws.api_key_method": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "env",+ "secret_provider"+ ],+ "default": "env",+ "open_node": false+ },+ "runtime.aws.bedrock_access_variant": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "aws_access",+ "bedrock_token"+ ],+ "default": "aws_access",+ "open_node": false+ },+ "runtime.log": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.log.caller_info_template": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "file_func",+ "file_line",+ "file_line_func",+ "func",+ "func_line",+ "func_module",+ "func_module_line"+ ],+ "default": "file_line",+ "open_node": false+ },+ "runtime.log.console_log_target": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "file",+ "stderr",+ "stdout"+ ],+ "default": "stderr",+ "open_node": false+ },+ "runtime.log.console_print_target": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "file",+ "stderr",+ "stdout"+ ],+ "default": "stdout",+ "open_node": false+ },+ "runtime.log.default_log_level": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "CRITICAL",+ "DEBUG",+ "DEV",+ "ERROR",+ "INFO",+ "OFF",+ "VERBOSE",+ "WARNING"+ ],+ "default": "INFO",+ "open_node": false+ },+ "runtime.log.is_caller_info_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "runtime.log.json_logs_indent": {+ "value_type": "int",+ "required": true,+ "default": 4,+ "open_node": false+ },+ "runtime.log.otlp": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.log.otlp.endpoint": {+ "value_type": "str",+ "required": false,+ "open_node": false+ },+ "runtime.log.otlp.headers": {+ "value_type": "dict[str, str]",+ "required": true,+ "default": {},+ "open_node": true+ },+ "runtime.log.otlp.headers.*": {+ "value_type": "str",+ "required": true,+ "open_node": false+ },+ "runtime.log.package_log_levels": {+ "value_type": "dict[str, enum]",+ "required": true,+ "default": {+ "anthropic": "INFO",+ "asyncio": "INFO",+ "botocore": "INFO",+ "botocore-credentials": "WARNING",+ "google": "INFO",+ "httpx": "WARNING",+ "httpcore": "INFO",+ "openai": "INFO",+ "instructor": "INFO",+ "urllib3-connectionpool": "INFO",+ "urllib3-util-retry": "INFO",+ "pipelex": "INFO"+ },+ "open_node": true+ },+ "runtime.log.package_log_levels.*": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "CRITICAL",+ "DEBUG",+ "DEV",+ "ERROR",+ "INFO",+ "OFF",+ "VERBOSE",+ "WARNING"+ ],+ "open_node": false+ },+ "runtime.log.presentation_line_width": {+ "value_type": "int",+ "required": true,+ "default": 120,+ "open_node": false+ },+ "runtime.log.pretty_print_mode": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "poor",+ "rich",+ "silent"+ ],+ "default": "rich",+ "open_node": false+ },+ "runtime.log.rich_log": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.log.rich_log.highlighter_name": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "json",+ "repr"+ ],+ "default": "json",+ "open_node": false+ },+ "runtime.log.rich_log.is_link_path_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.log.rich_log.is_markup_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.log.rich_log.is_rich_tracebacks": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.log.rich_log.is_show_level": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.log.rich_log.is_show_time": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "runtime.log.rich_log.is_tracebacks_show_locals": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "runtime.log.rich_log.is_tracebacks_word_wrap": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.log.rich_log.keywords_to_hilight": {+ "value_type": "list[str]",+ "required": true,+ "default": [],+ "open_node": false+ },+ "runtime.log.rich_log.tracebacks_suppress": {+ "value_type": "list[str]",+ "required": true,+ "default": [],+ "open_node": false+ },+ "runtime.log.silenced_problem_ids": {+ "value_type": "list[str]",+ "required": true,+ "default": [+ "azure_openai_no_stream_options"+ ],+ "open_node": false+ },+ "runtime.log.sink": {+ "value_type": "str",+ "required": true,+ "default": "console",+ "open_node": false+ },+ "runtime.observer": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.observer.observer_dir": {+ "value_type": "str",+ "required": true,+ "default": "results/observer",+ "open_node": false+ },+ "runtime.plugins": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.plugins.disabled": {+ "value_type": "list[str]",+ "required": true,+ "default": [],+ "open_node": false+ },+ "runtime.reporting": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.reporting.cost_report_base_name": {+ "value_type": "str",+ "required": true,+ "default": "cost_report",+ "open_node": false+ },+ "runtime.reporting.cost_report_dir_path": {+ "value_type": "str",+ "required": true,+ "default": "reports",+ "open_node": false+ },+ "runtime.reporting.cost_report_extension": {+ "value_type": "str",+ "required": true,+ "default": "csv",+ "open_node": false+ },+ "runtime.reporting.cost_report_unit_scale": {+ "value_type": "float",+ "required": true,+ "default": 1.0,+ "open_node": false,+ "constraints": {+ "gt": 0+ }+ },+ "runtime.reporting.is_generate_cost_report_file_enabled": {+ "value_type": "bool",+ "required": true,+ "default": false,+ "open_node": false+ },+ "runtime.reporting.is_log_costs_to_console": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.secrets": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.secrets.method": {+ "value_type": "str",+ "required": true,+ "default": "env",+ "open_node": false+ },+ "runtime.storage": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.storage.gcp": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "runtime.storage.gcp.bucket_name": {+ "value_type": "str",+ "required": true,+ "default": "",+ "open_node": false+ },+ "runtime.storage.gcp.project_id": {+ "value_type": "str",+ "required": true,+ "default": "",+ "open_node": false+ },+ "runtime.storage.gcp.signed_urls_lifespan_seconds": {+ "value_type": "int | literal",+ "required": true,+ "enum_members": [+ "disabled"+ ],+ "default": 3600,+ "open_node": false+ },+ "runtime.storage.gcp.uri_format": {+ "value_type": "str",+ "required": true,+ "default": "{hash}.{extension}",+ "open_node": false+ },+ "runtime.storage.in_memory": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "runtime.storage.in_memory.uri_format": {+ "value_type": "str",+ "required": true,+ "default": "{hash}.{extension}",+ "open_node": false+ },+ "runtime.storage.is_fetch_remote_content_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.storage.is_upload_local_content_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.storage.local": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "runtime.storage.local.local_storage_path": {+ "value_type": "str",+ "required": true,+ "default": ".pipelex/storage",+ "open_node": false+ },+ "runtime.storage.local.uri_format": {+ "value_type": "str",+ "required": true,+ "default": "{hash}.{extension}",+ "open_node": false+ },+ "runtime.storage.method": {+ "value_type": "str",+ "required": true,+ "default": "local",+ "open_node": false+ },+ "runtime.storage.s3": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "runtime.storage.s3.bucket_name": {+ "value_type": "str",+ "required": true,+ "default": "",+ "open_node": false+ },+ "runtime.storage.s3.region": {+ "value_type": "str",+ "required": true,+ "default": "",+ "open_node": false+ },+ "runtime.storage.s3.signed_urls_lifespan_seconds": {+ "value_type": "int | literal",+ "required": true,+ "enum_members": [+ "disabled"+ ],+ "default": 3600,+ "open_node": false+ },+ "runtime.storage.s3.uri_format": {+ "value_type": "str",+ "required": true,+ "default": "{hash}.{extension}",+ "open_node": false+ },+ "runtime.tracing": {+ "value_type": "table",+ "required": true,+ "open_node": false+ },+ "runtime.tracing.backend": {+ "value_type": "enum",+ "required": true,+ "enum_members": [+ "dynamodb",+ "ndjson"+ ],+ "default": "ndjson",+ "open_node": false+ },+ "runtime.tracing.dynamodb": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "runtime.tracing.dynamodb.region": {+ "value_type": "str",+ "required": true,+ "default": "",+ "open_node": false+ },+ "runtime.tracing.dynamodb.table_name": {+ "value_type": "str",+ "required": true,+ "default": "",+ "open_node": false+ },+ "runtime.tracing.is_enabled": {+ "value_type": "bool",+ "required": true,+ "default": true,+ "open_node": false+ },+ "runtime.tracing.ndjson": {+ "value_type": "table",+ "required": false,+ "open_node": false+ },+ "runtime.tracing.ndjson.traces_dir": {+ "value_type": "str",+ "required": true,+ "default": ".pipelex/traces",+ "open_node": false+ }+ }+}diff --git a/pipelex/migration/ledgers/pipelex-config.toml b/pipelex/migration/ledgers/pipelex-config.tomlindex 77e6ec9f3..ad19f922f 100644--- a/pipelex/migration/ledgers/pipelex-config.toml+++ b/pipelex/migration/ledgers/pipelex-config.toml@@ -12,7 +12,7 @@ id = "pipelex-config"title = "The main Pipelex configuration"base_file = "pipelex.toml"tier_glob = "pipelex_*.toml"-current_schema_version = 4+current_schema_version = 5min_supported_schema_version = 0# This entry is inserted *below* the reshape rather than appended above it, and the order is what@@ -375,3 +375,75 @@ key = "stuff_text_content"kind = "delete_key"table_path = ["interpreter", "pipeline_execution", "graph", "data_inclusion"]key = "stuff_html_content"++# Safe, with one thing it cannot do, said in the title because the title is what a person sees. Two of+# the retired keys chose a behaviour their deletion undoes: `log_mode = "poor"` chose plain output for a+# process with no terminal, whose replacement is `sink = "json"`, and `is_console_logging_enabled = false`+# chose silence, whose replacement is `pipelex = "OFF"` under `[runtime.log.package_log_levels]`. Neither+# replacement can be translated by a migration: no operation in this vocabulary writes a value at all, a+# `remap_value` reaches a string alone where that key holds a bool and is refused as `safe` onto `sink`,+# an open token rather than an enumeration, and the package level is a concrete key beneath an open node,+# which no operation may address. So the only lever left is `unsafe`, and `unsafe` is the wrong one here:+# safety is a property of an entry and this version's link owns all four removals, so it cannot be+# narrowed to the case that matters, and it would stop the boot of every file still carrying any of these+# keys at their old defaults — every full copy of an older base file — for the sake of the few that chose+# otherwise. Nothing chose otherwise. That is the decision, recorded here rather than argued again having+# been raised and settled on each review round of the branch that retired these keys: no deployment set+# these keys, nothing depends on the flag outliving its removal, and this project keeps no backward+# compatibility. So the keys are deleted, and a file that did choose otherwise sets its replacement by+# hand, which is what the title and the guidance are for.+[[migration]]+id = "pipelex-config@5"+to_schema_version = 5+introduced_in = "0.60.0"+breaking = true+safety = "safe"+title = "Log sinks replace the rich-or-poor log mode: by hand, a poor-mode file then selects sink = \"json\" and a silenced one sets pipelex = \"OFF\" under package_log_levels"+description = """+Where log records go is now a sink selected by `[runtime.log] sink`, a registered token such as \+`console`, `json` or `otlp`, in place of the `rich` / `poor` log mode. `log_mode`, the poor-logger \+settings `poor_loggers` and `generic_poor_logger`, and `is_console_logging_enabled` are deleted from \+files that still set them, and the defaults layer supplies `sink = "console"`, the Rich rendering. A \+file that had set `log_mode = "poor"` or `is_console_logging_enabled = false` chose a behaviour that \+this deletion undoes, and no migration may write the replacement: such a file sets `sink = "json"` or \+`pipelex = "OFF"` under `[runtime.log.package_log_levels]` by hand.+"""+guidance = """+A file that set `log_mode = "poor"` chose a plain handler for a process with no terminal; that process \+now sets `sink = "json"` in `[runtime.log]`, one JSON object per line, which is what a log agent ingests \+without a parser. A file that set `log_mode = "rich"` chose what the default, `console`, renders, and \+needs nothing.++`is_console_logging_enabled = false` suppressed every record Pipelex's own log calls emitted, third-party \+loggers still reaching the handlers. To suppress them now set `pipelex = "OFF"` under \+`[runtime.log.package_log_levels]` in the project override: that section pins the `pipelex` logger at \+`INFO` in the base configuration, the override deep-merges over it and leaves the third-party levels \+alone, and `default_log_level` governs only the loggers the section does not pin, so on its own it \+silences none of Pipelex's records. Silence for everything, third-party loggers included, takes \+`default_log_level = "OFF"` and every entry of that section at `OFF`. To send the records somewhere \+else instead, select the sink that goes there. `is_console_logging_enabled = true` was the default and \+asks for nothing beyond the deletion the migration performs.++`poor_loggers` and `generic_poor_logger` routed some records to a plain stream handler beside the \+console; every record now goes to the one selected sink, so there is nothing to carry over.+"""++[[migration.ops]]+kind = "delete_key"+table_path = ["runtime", "log"]+key = "log_mode"++[[migration.ops]]+kind = "delete_key"+table_path = ["runtime", "log"]+key = "poor_loggers"++[[migration.ops]]+kind = "delete_key"+table_path = ["runtime", "log"]+key = "generic_poor_logger"++[[migration.ops]]+kind = "delete_key"+table_path = ["runtime", "log"]+key = "is_console_logging_enabled"diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex a4197193c..3fb0bd507 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -37,7 +37,13 @@ method = "env" # env (built-in); an external plugin may add e.g. vault[runtime.log]default_log_level = "INFO"-# console_log_target controls the RichHandler used by the logging system (log.debug/info/...).+# sink names the registered log sink boot installs on the root logger, the way runtime.storage.method+# names a storage backend: "console" (the Rich handler, for a terminal), "json" (one JSON object per+# line, for a server behind a log agent), "otlp" (the OpenTelemetry logs signal, for a collector), or a+# token an installed plugin registers. "console" is the default so the CLI keeps its rendering; a+# server selects "json". An unregistered token stops the boot naming the registered ones.+sink = "console"+# console_log_target controls the stream the console and json sinks write to (log.debug/info/...).# Logs are diagnostics → stderr.console_log_target = "stderr"# console_print_target controls the Console returned by get_console(), used for banners,@@ -49,7 +55,6 @@ console_log_target = "stderr"# targets to stderr internally to keep its stdout strictly reserved for the success# envelope (see `_AGENT_CLI_STDERR_CONSOLE_OVERRIDES` in `agent_cli_factory.py`).console_print_target = "stdout"-is_console_logging_enabled = truejson_logs_indent = 4presentation_line_width = 120@@ -57,16 +62,12 @@ silenced_problem_ids = ["azure_openai_no_stream_options"]caller_info_template = "file_line"is_caller_info_enabled = false-log_mode = "rich"# pretty_print_mode controls the panels `pretty_print(...)` renders, the "Output of pipe" panel after# every operator pipe among them: "rich" renders Rich tables, "poor" prints plain text, "silent" prints# nothing and builds no renderable at all. A host with no console, or one that runs pipes on a thread# that must not stall, sets "silent"; the agent CLI forces it.pretty_print_mode = "rich"-poor_loggers = []-generic_poor_logger = "#poor-log"-[runtime.log.package_log_levels]# for logger names, the dots "." have been replaced by "-" to avoid toml parsing issues# TODO: use quotes to re-enable names with dots@@ -97,6 +98,13 @@ is_tracebacks_show_locals = falsetracebacks_suppress = []keywords_to_hilight = []+# The settings of the "otlp" sink. An endpoint key, when set, names the collector's logs URL+# (for instance "http://localhost:4318/v1/logs"); left unset, the exporter follows the OpenTelemetry+# environment conventions: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, then OTEL_EXPORTER_OTLP_ENDPOINT, then+# the collector default on localhost. Empty headers likewise leave OTEL_EXPORTER_OTLP_HEADERS in charge.+[runtime.log.otlp]+headers = {}+[runtime.aws]api_key_method = "env"bedrock_access_variant = "aws_access"diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.pyindex 34cf6d7ae..49a49e01f 100644--- a/pipelex/plugins/contract.py+++ b/pipelex/plugins/contract.py@@ -20,6 +20,10 @@ if TYPE_CHECKING:# v4 split the single ``pipelex.plugins`` entry-point group into the two ``PluginGroup`` groups# below: a plugin now declares its layer by the group it publishes under, and a kernel-group plugin# may no longer reach the interpreter tier of the menu.+#+# ``add_log_sink`` — a third config-selected, process-global registry (``runtime.log.sink`` picks the+# factory at boot) — joined the menu under v4 without a bump: a menu addition breaks no plugin that+# targets v4, and a bump would have made every installed plugin re-declare ``targets_api`` for nothing.PLUGIN_API_VERSION: int = 4diff --git a/pipelex/plugins/exceptions.py b/pipelex/plugins/exceptions.pyindex efe921f56..511ce920c 100644--- a/pipelex/plugins/exceptions.py+++ b/pipelex/plugins/exceptions.py@@ -123,6 +123,20 @@ class DuplicateSecretsProviderError(PluginError):super().__init__(message)+class DuplicateLogSinkError(PluginError):+ """Two plugins registered a log sink for the same method."""++ def __init__(self, *, method: str, first_plugin: str, second_plugin: str):+ self.method = method+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Log sink for method '{method}' is registered by both plugin '{first_plugin}' and plugin '{second_plugin}'. "+ "Each method must have a single sink."+ )+ super().__init__(message)++class DuplicatePipeFuncExecutorError(PluginError):"""Two plugins registered a PipeFunc executor for the same execution mode."""@@ -371,3 +385,28 @@ class UnknownSecretsMethodError(PluginError):"Check runtime.secrets.method, or install/enable the plugin that provides that method.")super().__init__(message)+++class UnknownLogSinkError(PluginError):+ """A configured log sink has no registered factory.++ ``runtime.log.sink`` selects a sink from the registry the built-in ``LogSinkPlugin`` (and any+ external plugin contributing one) populates. When the token names no registered factory — a typo,+ or an external plugin that is not installed or was disabled via ``runtime.plugins.disabled`` — boot+ fails loud here rather than starting with its records going nowhere. The message lists the+ registered sinks so the fix is obvious.+ """++ # The message describes the caller's own input (the configured sink) and lists the registered+ # ones; it is fully actionable, so keep it verbatim under STRICT disclosure.+ _authors_caller_facing_message = True++ def __init__(self, *, method: str, registered_methods: list[str]):+ self.method = method+ self.registered_methods = registered_methods+ available = ", ".join(sorted(registered_methods)) or "(none)"+ message = (+ f"No log sink is registered for '{method}'. Registered sinks: {available}. "+ "Check runtime.log.sink, or install/enable the plugin that provides that sink."+ )+ super().__init__(message)diff --git a/pipelex/plugins/log_sink_registry.py b/pipelex/plugins/log_sink_registry.pynew file mode 100644index 000000000..0b571ae7d--- /dev/null+++ b/pipelex/plugins/log_sink_registry.py@@ -0,0 +1,37 @@+from collections.abc import Callable++from pipelex.plugins.exceptions import UnknownLogSinkError+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_sink import LogSink++# A plugin's factory for one log sink: whole log config in, sink out. The whole config (not a+# pre-resolved sub-config) is passed so a factory can read whatever it needs — the stream target, its+# own settings section — at the boot apply-point, never at registration. The sink's handler is built+# later still, when boot installs it, which is where a missing dependency fails loud.+LogSinkFactoryFn = Callable[[LogConfig], LogSink]+++class LogSinkRegistry:+ """Read view over the log-sink factories contributed by discovered plugins.++ Keyed by the open sink ``method`` token (a ``str``; the built-in ``LogSinkPlugin`` registers the+ ``LogSinkMethod`` values, an external plugin registers e.g. ``"gcp"``). Built once at boot from the+ registrar's accumulated ``log_sinks``; boot reads ``runtime.log.sink`` and calls the looked-up+ factory to produce the one sink installed on the root logger. Mirrors ``StorageProviderRegistry``.+ """++ def __init__(self, log_sinks: dict[str, LogSinkFactoryFn]):+ self._log_sinks: dict[str, LogSinkFactoryFn] = dict(log_sinks)++ def get_required(self, *, method: str) -> LogSinkFactoryFn:+ factory = self._log_sinks.get(method)+ if factory is None:+ raise UnknownLogSinkError(method=method, registered_methods=self.methods)+ return factory++ def has(self, *, method: str) -> bool:+ return method in self._log_sinks++ @property+ def methods(self) -> list[str]:+ return list(self._log_sinks)diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.pyindex 7928a4aef..a3baf8150 100644--- a/pipelex/plugins/registrar.py+++ b/pipelex/plugins/registrar.py@@ -10,6 +10,7 @@ from pipelex.plugins.exceptions import (DuplicateBundleValidatorError,DuplicateHttpErrorMapperError,DuplicateInferenceBackendError,+ DuplicateLogSinkError,DuplicateModelListerError,DuplicateOrchestratorError,DuplicatePipeFuncExecutorError,@@ -19,6 +20,7 @@ from pipelex.plugins.exceptions import (PluginLayerViolationError,)from pipelex.plugins.inference_backend_registry import InferenceFamily, MakeWorkerFn+from pipelex.plugins.log_sink_registry import LogSinkFactoryFnfrom pipelex.plugins.model_lister_registry import ListModelsFnfrom pipelex.plugins.orchestrator_registry import OrchestratorProtocolfrom pipelex.plugins.pipe_func_executor_registry import PipeFuncExecutorFactoryFn@@ -158,6 +160,7 @@ class PluginRegistrar:self.bundle_validators: dict[OrchestrationMode, BundleValidatorProtocol] = {}self.storage_providers: dict[str, StorageProviderFactoryFn] = {}self.secrets_providers: dict[str, SecretsProviderFactoryFn] = {}+ self.log_sinks: dict[str, LogSinkFactoryFn] = {}self.pipe_func_executors: dict[str, PipeFuncExecutorFactoryFn] = {}# Ordered list (not a type-keyed dict) because the exception types are# resolved lazily — only ``get_http_error_mappers`` invokes the providers,@@ -172,6 +175,7 @@ class PluginRegistrar:self._bundle_validator_sources: dict[OrchestrationMode, str] = {}self._storage_provider_sources: dict[str, str] = {}self._secrets_provider_sources: dict[str, str] = {}+ self._log_sink_sources: dict[str, str] = {}self._pipe_func_executor_sources: dict[str, str] = {}self._slot_sources: dict[HubSlot, str] = {}# Reassigned per plugin by build_registrar; the floating default keeps the@@ -284,6 +288,27 @@ class PluginRegistrar:),)+ def add_log_sink(self, *, method: str, factory: LogSinkFactoryFn) -> None:+ """Contribute a factory for one log sink, keyed by an open ``method`` token.++ The built-in ``LogSinkPlugin`` registers the ``json`` / ``console`` / ``otlp`` methods; an+ external plugin registers its own token (e.g. ``"gcp"``). Boot reads ``runtime.log.sink`` and+ calls the looked-up factory to produce the one sink whose handler goes on the root logger.+ ``factory`` is invoked at that boot apply-point, never here, and the sink builds its handler+ later still, at install — so a factory or a sink may import a heavy SDK (Rich, the OpenTelemetry+ logs SDK) while ``register`` stays import-light. Fail-loud on a duplicate method, naming both plugins.+ """+ self._add(+ store=self.log_sinks,+ sources=self._log_sink_sources,+ key=method,+ value=factory,+ contribution=f"log sink {method}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateLogSinkError(+ method=method, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )+def add_pipe_func_executor(self, *, mode: str, factory: PipeFuncExecutorFactoryFn) -> None:"""Contribute a factory for one PipeFunc execution mode, keyed by an open ``mode`` token.diff --git a/pipelex/providers/builtins.py b/pipelex/providers/builtins.pyindex ab781fe36..384ab1b35 100644--- a/pipelex/providers/builtins.py+++ b/pipelex/providers/builtins.py@@ -10,6 +10,7 @@ from pipelex.providers.gateway.gateway_plugin import GatewayPluginfrom pipelex.providers.google.google_plugin import GooglePluginfrom pipelex.providers.huggingface.huggingface_plugin import HuggingFacePluginfrom pipelex.providers.linkup.linkup_plugin import LinkupPlugin+from pipelex.providers.log_sinks.log_sink_plugin import LogSinkPluginfrom pipelex.providers.manifold.manifold_plugin import ManifoldPluginfrom pipelex.providers.mistral.mistral_plugin import MistralPluginfrom pipelex.providers.openai.openai_plugin import OpenAIPlugin@@ -34,6 +35,7 @@ from pipelex.providers.storage.storage_plugin import StoragePlugin# Each is import-light: importing this module imports no backend SDK (the SDKs load lazily inside the# make_worker closures).KERNEL_BUILTIN_PLUGINS: list[PipelexPlugin] = [+ LogSinkPlugin(),StoragePlugin(),SecretsPlugin(),OpenAIPlugin(),@@ -55,13 +57,14 @@ KERNEL_BUILTIN_PLUGINS: list[PipelexPlugin] = []# Kernel-layer built-ins that core requires unconditionally — naming one in ``runtime.plugins.disabled`` is a-# configuration error, not a no-op. ``storage`` supplies every built-in storage backend-# (``runtime.storage.method`` must resolve to a registered factory or boot fails loud); ``secrets``-# supplies the built-in ``env`` secrets backend (``runtime.secrets.method`` must likewise resolve or boot-# fails loud); ``openai`` is the always-on default inference driver (no optional SDK to avoid), so-# disabling it would only break the out-of-the-box experience. The interpreter-layer half of this set-# lives beside its plugins, in ``pipelex.interpreter_plugins.builtins``.-KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"storage", "secrets", "openai"})+# configuration error, not a no-op. ``log_sinks`` supplies every built-in log sink+# (``runtime.log.sink`` must resolve to a registered factory or boot fails loud); ``storage`` supplies+# every built-in storage backend (``runtime.storage.method`` must likewise resolve or boot fails loud);+# ``secrets`` supplies the built-in ``env`` secrets backend (``runtime.secrets.method`` must likewise+# resolve or boot fails loud); ``openai`` is the always-on default inference driver (no optional SDK to+# avoid), so disabling it would only break the out-of-the-box experience. The interpreter-layer half of+# this set lives beside its plugins, in ``pipelex.interpreter_plugins.builtins``.+KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"log_sinks", "storage", "secrets", "openai"})# The entry-point groups a kernel-only boot reads: its own, and only its own. Sits beside the# built-in manifest because it answers the same question for the other half of discovery — what andiff --git a/pipelex/providers/log_sinks/__init__.py b/pipelex/providers/log_sinks/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/pipelex/providers/log_sinks/log_sink_plugin.py b/pipelex/providers/log_sinks/log_sink_plugin.pynew file mode 100644index 000000000..6d695f31a--- /dev/null+++ b/pipelex/providers/log_sinks/log_sink_plugin.py@@ -0,0 +1,64 @@+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.system.runtime import RunEnvironment+from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.tools.log.console_log_sink import ConsoleLogSink+from pipelex.tools.log.json_log_sink import JsonLogSink+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_sink import LogSink, LogSinkMethod, stream_for_target+from pipelex.tools.misc.package_utils import get_package_version+++def _make_json_log_sink(config: LogConfig) -> LogSink:+ return JsonLogSink(stream=stream_for_target(target=config.console_log_target))+++def _make_console_log_sink(config: LogConfig) -> LogSink:+ # Rich is imported when the sink builds its handler, at install, never here.+ return ConsoleLogSink(rich_log_config=config.rich_log, target=config.console_log_target)+++def _make_otlp_log_sink(config: LogConfig) -> LogSink:+ # Deferred imports: the OpenTelemetry logs SDK and the exporter load only when this sink is+ # selected, so registering the built-ins stays import-light and a process on another sink never+ # pays for them. The sink module imports the SDK at load, which is why it is imported here too.+ from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter # ruff: ignore[import-outside-top-level, import-private-name]+ from opentelemetry.sdk._logs.export import BatchLogRecordProcessor # ruff: ignore[import-outside-top-level, import-private-name]+ from opentelemetry.sdk.resources import Resource as OTelResource # ruff: ignore[import-outside-top-level]+ from opentelemetry.semconv._incubating.attributes import deployment_attributes # ruff: ignore[import-outside-top-level, import-private-name]+ from opentelemetry.semconv.attributes import service_attributes # ruff: ignore[import-outside-top-level]++ from pipelex.tools.log.otlp_log_sink import OtlpLogSink # ruff: ignore[import-outside-top-level]++ # The same identity the tracer provider declares, so a collector files the logs beside the spans.+ resource = OTelResource.create(+ attributes={+ service_attributes.SERVICE_NAME: OTelConstants.SERVICE_NAME,+ service_attributes.SERVICE_VERSION: get_package_version(),+ OTelConstants.SERVICE_NAMESPACE_KEY: OTelConstants.SERVICE_NAMESPACE,+ deployment_attributes.DEPLOYMENT_ENVIRONMENT: RunEnvironment.get_from_env_var().value,+ }+ )+ # ``None`` for an absent endpoint or empty headers leaves the exporter to the OTEL_EXPORTER_OTLP_*+ # environment conventions, the way a collector-side deployment is configured.+ exporter = OTLPLogExporter(endpoint=config.otlp.endpoint, headers=config.otlp.headers or None)+ return OtlpLogSink(processor=BatchLogRecordProcessor(exporter), resource=resource)+++class LogSinkPlugin:+ """Always-on built-in provider of the ``json`` / ``console`` / ``otlp`` log sinks.++ Core-unconditional: a process needs somewhere for its records to go, so this plugin cannot be+ disabled into a boot with no sink (see ``KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES``). It registers+ one factory per built-in method; ``runtime.log.sink`` selects which one boot invokes. Importing+ this module is import-light: Rich loads when the console sink builds its handler, and the+ OpenTelemetry SDK when the ``otlp`` factory runs, never at register.+ """++ name = "log_sinks"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_log_sink(method=LogSinkMethod.JSON, factory=_make_json_log_sink)+ registrar.add_log_sink(method=LogSinkMethod.CONSOLE, factory=_make_console_log_sink)+ registrar.add_log_sink(method=LogSinkMethod.OTLP, factory=_make_otlp_log_sink)diff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.pyindex 048d87921..798e349d5 100644--- a/pipelex/runtime_boot.py+++ b/pipelex/runtime_boot.py@@ -82,6 +82,7 @@ from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistryfrom pipelex.plugins.discovery import build_registrarfrom pipelex.plugins.exceptions import UnknownBootOrchestratorErrorfrom pipelex.plugins.inference_backend_registry import InferenceBackendRegistry+from pipelex.plugins.log_sink_registry import LogSinkRegistryfrom pipelex.plugins.model_lister_registry import ModelListerRegistryfrom pipelex.plugins.orchestrator_registry import OrchestratorRegistryfrom pipelex.plugins.registrar import HubSlot, PluginRegistrar@@ -547,6 +548,15 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.if boot_orchestrator is not None and boot_orchestrator not in plugin_registrar.registered_plugin_names:raise UnknownBootOrchestratorError(requested=boot_orchestrator)+ # The log sink: the first capability resolved out of the registrar, because every line the rest+ # of this boot emits should be rendered by the sink the configuration chose. ``log.configure``+ # ran in ``__init__``, before discovery could, and has held every record since; installing the+ # sink replays them through it. The built-in LogSinkPlugin supplies every shipped sink, so there+ # is no separate core default, and an unknown token fails loud here listing the registered ones.+ log_config = get_config().runtime.log+ log_sink_registry = LogSinkRegistry(plugin_registrar.log_sinks)+ log.install_sink(log_sink_registry.get_required(method=log_config.sink)(log_config))+# Secrets provider precedence: explicit setup() param > config-selected registry factory.# The built-in SecretsPlugin supplies the "env" method, so there is no separate core default.# Resolved here because the telemetry factory just below (and the model setup further down) consume it.diff --git a/pipelex/system/exceptions.py b/pipelex/system/exceptions.pyindex 0710191ac..592b48356 100644--- a/pipelex/system/exceptions.py+++ b/pipelex/system/exceptions.py@@ -3,6 +3,7 @@ from enum import StrEnumfrom typing import ClassVarfrom pipelex.base_exceptions import ErrorDomain, PipelexError, PipelexUnexpectedError+from pipelex.tools.log.log_fields import VERBATIM_MARKclass ToolError(PipelexError):@@ -52,14 +53,17 @@ class TracebackMessageError(PipelexError):def __init__(self, message: str):super().__init__(message)logger_name = __name__+ # The message is assembled from whatever raised, so it carries text nobody chose: `[Errno 2]`, a+ # `list[int]` in a type complaint, a fragment of a traceback. On a console that interprets markup+ # that is a tag, and Rich either swallows the bracketed span or raises on an unbalanced one — an+ # error message is the last line that may be silently altered. The mark asks for it verbatim.+ verbatim = {VERBATIM_MARK: False}match self.__class__.error_mode:case TracebackMessageErrorMode.ERROR:- generic_poor_logger = "#poor-log"- logger = logging.getLogger(generic_poor_logger)- logger.error(message)+ logging.getLogger(logger_name).error(message, extra=verbatim)case TracebackMessageErrorMode.EXCEPTION:self.logger = logging.getLogger(logger_name)- self.logger.exception(message)+ self.logger.exception(message, extra=verbatim)class FatalError(TracebackMessageError):diff --git a/pipelex/tools/log/console_log_sink.py b/pipelex/tools/log/console_log_sink.pynew file mode 100644index 000000000..a9618acd7--- /dev/null+++ b/pipelex/tools/log/console_log_sink.py@@ -0,0 +1,101 @@+"""The ``console`` sink: the Rich handler with the emoji formatter and every ``[runtime.log.rich_log]`` setting.++Rich is imported when the handler is built and nowhere else in this module, so a process that selects+another sink never loads it, and one that selects this sink without Rich installed fails at boot with+the extra to install and the ``json`` alternative named, per the plugin system's fail-at-use rule.+"""++from __future__ import annotations++import sys+from typing import TYPE_CHECKING++from typing_extensions import override++from pipelex.system.exceptions import MissingDependencyError+from pipelex.tools.log.log_config import HighlighterName+from pipelex.tools.log.log_formatter import EmojiLogFormatter+from pipelex.tools.log.log_sink import LogSink, LogSinkMethod, stream_for_target++if TYPE_CHECKING:+ import logging++ from rich.logging import RichHandler++ from pipelex.system.console_target import ConsoleTarget+ from pipelex.tools.log.log_config import RichLogConfig++# The extra that installs Rich. Rich is a hard dependency today; this is the name the extra takes when it moves.+RICH_EXTRA_NAME = "cli"+++class ConsoleLogSink(LogSink):+ """Today's console rendering, byte for byte: a ``RichHandler`` on the configured stream."""++ def __init__(self, *, rich_log_config: RichLogConfig, target: ConsoleTarget) -> None:+ super().__init__()+ self._rich_log_config = rich_log_config+ self._target = target+ self._rich_handler: RichHandler | None = None++ @override+ def make_handler(self) -> logging.Handler:+ try:+ from rich.console import Console # ruff: ignore[import-outside-top-level]+ from rich.highlighter import Highlighter, JSONHighlighter, ReprHighlighter # ruff: ignore[import-outside-top-level]+ from rich.logging import RichHandler # ruff: ignore[import-outside-top-level]+ except ImportError as exc:+ msg = (+ f"The '{LogSinkMethod.CONSOLE}' log sink renders through Rich. Install the extra, "+ f"or select the '{LogSinkMethod.JSON}' sink in [runtime.log] for a process with no terminal."+ )+ raise MissingDependencyError(dependency_name="rich", extra_name=RICH_EXTRA_NAME, message=msg) from exc++ # Declared here because ``RichHandler`` is imported here, which is what keeps Rich off the import+ # path of a process that selected another sink. ``RichHandler`` overrides ``emit`` and does not+ # restore the ``try``/``handleError`` the stdlib's own handlers put around theirs, so anything+ # raised while rendering leaves the log call: a message carrying a tag-shaped span — `list[int]` in+ # a type complaint, or a bracketed path, both of which an error message is made of — raises+ # ``MarkupError`` out of `log.error` and replaces whatever was being reported with itself. A log+ # call never raises, so the guard goes back on and a line Rich cannot render gets the stdlib's own+ # recovery instead.+ class GuardedRichHandler(RichHandler):+ @override+ def emit(self, record: logging.LogRecord) -> None:+ try:+ super().emit(record)+ except Exception: # ruff: ignore[blind-except]+ self.handleError(record)++ config = self._rich_log_config+ highlighter: Highlighter+ match config.highlighter_name:+ case HighlighterName.JSON:+ highlighter = JSONHighlighter()+ case HighlighterName.REPR:+ highlighter = ReprHighlighter()+ handler = GuardedRichHandler(+ console=Console(file=stream_for_target(target=self._target)),+ show_time=config.is_show_time,+ show_level=config.is_show_level,+ enable_link_path=config.is_link_path_enabled,+ highlighter=highlighter,+ markup=config.is_markup_enabled,+ rich_tracebacks=config.is_rich_tracebacks,+ tracebacks_word_wrap=config.is_tracebacks_word_wrap,+ tracebacks_show_locals=config.is_tracebacks_show_locals,+ tracebacks_suppress=config.tracebacks_suppress,+ keywords=config.keywords_to_hilight,+ )+ handler.setFormatter(EmojiLogFormatter())+ self._rich_handler = handler+ return handler++ @override+ def redirect_to_stderr(self) -> None:+ from rich.console import Console # ruff: ignore[import-outside-top-level]++ if self._rich_handler is None:+ _ = self.handler+ if self._rich_handler is not None:+ self._rich_handler.console = Console(file=sys.stderr)diff --git a/pipelex/tools/log/json_log_sink.py b/pipelex/tools/log/json_log_sink.pynew file mode 100644index 000000000..3c8dc6a7f--- /dev/null+++ b/pipelex/tools/log/json_log_sink.py@@ -0,0 +1,102 @@+"""The ``json`` sink: one JSON object per line on the configured stream, and no ANSI ever.++The keys are the ones a log agent ingests without a parser, the CloudWatch agent, the Google Cloud+Logging agent and any OTLP collector among them: ``time``, ``severity``, ``logger``, ``message``, then+``exception`` when the record carries one, then the fields, the context identifiers and the ``data``+attribute under their own names. The sink's own keys are reserved whether or not the line carries+them: a field named like one is carried under the same ``field_`` prefix the record uses for a name+the stdlib owns, on every line and not only the ones with an exception, so no value is lost and a+field keeps one wire name. A non-finite float is written as the string ``"NaN"``, ``"Infinity"`` or+``"-Infinity"``, since JSON has no token for it that a strict parser accepts.+"""++from __future__ import annotations++import json+import logging+import sys+from datetime import UTC, datetime+from typing import TYPE_CHECKING, Any++from typing_extensions import override++from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes+from pipelex.tools.log.log_sink import LogSink, json_fallback, spell_non_finite++if TYPE_CHECKING:+ from typing import TextIO++TIME_KEY = "time"+SEVERITY_KEY = "severity"+LOGGER_KEY = "logger"+MESSAGE_KEY = "message"+EXCEPTION_KEY = "exception"+# The keys the sink writes itself, reserved on every line; a carried attribute of the same name is prefixed.+FIXED_KEYS = frozenset({TIME_KEY, SEVERITY_KEY, LOGGER_KEY, MESSAGE_KEY, EXCEPTION_KEY})+++def _json_line(*, payload: dict[str, Any]) -> str:+ """One JSON object for the payload, never raising and never anything but an object.++ A model dumps in JSON mode and an unknown object becomes its text. When ``json`` refuses the+ payload outright, a circular reference or a mapping with a non-string key inside a carried value,+ every carried value is written as its ``repr`` and the line keeps its shape: a sink must render+ every record it is handed, and a serialization failure is a fact about the value, never a reason to+ lose the line or to break the one-object-per-line contract.+ """+ try:+ return json.dumps(payload, ensure_ascii=False, allow_nan=False, default=json_fallback)+ except (TypeError, ValueError):+ safe_payload = {key: value if key in FIXED_KEYS else repr(value) for key, value in payload.items()}+ return json.dumps(safe_payload, ensure_ascii=False, default=str)+++def _iso_utc(*, created: float) -> str:+ """The record's creation time as ISO 8601 in UTC with millisecond precision and a ``Z`` suffix."""+ return datetime.fromtimestamp(created, tz=UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")+++class JsonLogFormatter(logging.Formatter):+ """Renders a record as one JSON object, its carried attributes flat beside the fixed keys."""++ @override+ def format(self, record: logging.LogRecord) -> str:+ payload: dict[str, Any] = {+ TIME_KEY: _iso_utc(created=record.created),+ SEVERITY_KEY: record.levelname,+ LOGGER_KEY: record.name,+ MESSAGE_KEY: record.getMessage(),+ }+ if record.exc_info:+ payload[EXCEPTION_KEY] = record.exc_text or self.formatException(record.exc_info)+ elif record.exc_text:+ payload[EXCEPTION_KEY] = record.exc_text+ for name, value in carried_attributes(record=record).items():+ key = name+ while key in payload or key in FIXED_KEYS:+ key = f"{COLLIDING_FIELD_PREFIX}{key}"+ payload[key] = spell_non_finite(value=value)+ return _json_line(payload=payload)+++class JsonLogSink(LogSink):+ """One JSON object per line on a text stream."""++ def __init__(self, *, stream: TextIO) -> None:+ super().__init__()+ self._stream = stream+ self._stream_handler: logging.StreamHandler[TextIO] | None = None++ @override+ def make_handler(self) -> logging.Handler:+ handler: logging.StreamHandler[TextIO] = logging.StreamHandler(self._stream)+ handler.setFormatter(JsonLogFormatter())+ self._stream_handler = handler+ return handler++ @override+ def redirect_to_stderr(self) -> None:+ if self._stream_handler is None:+ _ = self.handler+ if self._stream_handler is not None:+ self._stream_handler.setStream(sys.stderr)diff --git a/pipelex/tools/log/log.py b/pipelex/tools/log/log.pyindex 2121b40ff..554eb604a 100644--- a/pipelex/tools/log/log.py+++ b/pipelex/tools/log/log.py@@ -1,27 +1,52 @@from __future__ import annotations+import contextlibimport loggingimport sysfrom typing import TYPE_CHECKING, Any-from rich.console import Console-from rich.logging import RichHandler-from pipelex.tools.log.log_context import bind_log_contextfrom pipelex.tools.log.log_dispatch import LogDispatch-from pipelex.tools.log.log_formatter import EmojiLogFormatter, LevelAndEmojiLogFormatter+from pipelex.tools.log.log_holding import ForwardedRecordFilter, HoldingLogHandlerfrom pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_OFF, LOGGING_LEVEL_VERBOSE, LogLevelif TYPE_CHECKING:- from collections.abc import Mapping+ from collections.abc import Callable, Mappingfrom contextlib import AbstractContextManager- from pipelex.tools.log.log_config import LogConfig, LogMode+ from pipelex.tools.log.log_config import LogConfigfrom pipelex.tools.log.log_context import LogContext+ from pipelex.tools.log.log_sink import LogSink+++def _finish_teardown_step(*, sink: LogSink, verb: str, step: Callable[[], None]) -> None:+ """Run one step of a sink's teardown and let nothing it raises escape.++ A closed stream, a test capture or a redirected process stream torn down first, stays silent: the+ stdlib's own shutdown tolerates the same two. Anything else, an exporter that cannot reach its+ collector or a processor whose shutdown raises, is a fact about the sink, said on stderr with the+ step that failed, and never a reason to leave the teardown half done.+ """+ try:+ step()+ except (OSError, ValueError):+ pass+ except Exception as exc: # ruff: ignore[blind-except]+ # A closed or disconnected stderr is the one place the diagnostic can go, and a teardown that+ # stopped to say it could not be said would be the failure it was written to avoid.+ with contextlib.suppress(Exception):+ sys.stderr.write(f"The log sink {type(sink).__name__} failed to {verb} at reset: {exc!r}\n")class Log:- """A class for managing logging configurations and operations."""+ """A class for managing logging configurations and operations.++ Configuration is two steps, because the sink is a plugin capability and plugin discovery runs a+ little after the configuration is read: ``configure`` sets the levels and holds every record on the+ root logger, then ``install_sink`` puts the selected sink's handler in its place and replays what was+ held through it. ``reset`` removes exactly what those two installed and leaves every other handler,+ a host's or a test's, where it was.+ """######################################################### Init and Configure@@ -30,38 +55,50 @@ class Log:def __init__(self):"""Initialize the Log class with default attributes."""self._log_config_instance: LogConfig | None = None- self.rich_handler: logging.Handler | None = None- self.poor_handler: logging.Handler | None = None+ self._holding_handler: HoldingLogHandler | None = None+ self._sink: LogSink | None = Noneself.log_dispatch: LogDispatch = LogDispatch()- def set_log_mode(self, mode: LogMode):- self.log_dispatch.set_log_mode(mode=mode)+ @property+ def sink(self) -> LogSink | None:+ """The installed sink, or ``None`` before ``install_sink`` and after ``reset``."""+ return self._sink++ @property+ def is_configured(self) -> bool:+ """Whether ``configure`` has run and ``reset`` has not, which is what makes a later ``configure`` refuse."""+ return self._log_config_instance is not Nonedef reset(self):- """Reset the logging system."""- # Remove all handlers from the root logger+ """Remove what ``configure`` and ``install_sink`` put on the root logger, and forget the configuration.++ The sink's handler is flushed and closed, which is where a batching sink ships what it still+ holds, and nothing that flush or close raises escapes: this runs first in the runtime's release+ of its process globals, and an error out of it would skip the rest and leave the process+ unbootable. The close runs whatever the flush did, since the close is what stops an exporter's+ thread, ships its last batch and unregisters what it put at exit. A holding handler still in+ place, because the boot died before its sink arrived, is closed too, and what it holds gets the+ stdlib's last-resort handling.+ """root_logger = logging.getLogger()- for handler in root_logger.handlers[:]:- root_logger.removeHandler(handler)- handler.close()-- # Remove handlers from poor loggers- if self._log_config_instance:- poor_loggers = {- *self._log_config_instance.poor_loggers,- self._log_config_instance.generic_poor_logger,- }- for logger_name in poor_loggers:- logger = logging.getLogger(logger_name)- for handler in logger.handlers[:]:- logger.removeHandler(handler)- handler.close()-- logging.shutdown()- self._log_config_instance = None- self.rich_handler = None- self.poor_handler = None- self.log_dispatch.reset()+ try:+ if self._sink is not None:+ sink, self._sink = self._sink, None+ handler = sink.handler+ root_logger.removeHandler(handler)+ try:+ _finish_teardown_step(sink=sink, verb="flush", step=handler.flush)+ finally:+ _finish_teardown_step(sink=sink, verb="close", step=handler.close)+ if self._holding_handler is not None:+ holding, self._holding_handler = self._holding_handler, None+ root_logger.removeHandler(holding)+ holding.close()+ finally:+ # Whatever the handlers did, the configuration is forgotten: a reset that left it in place+ # would make the next ``configure`` refuse, which is the unbootable process this guards.+ self._log_config_instance = None+ self.log_dispatch.reset()def configure_if_unset(self, log_config: LogConfig) -> bool:"""Configure logging unless already configured.@@ -100,38 +137,58 @@ class Log:self._log_config_instance = log_config- self.rich_handler = log_config.rich_log.make_rich_handler(target=log_config.console_log_target)- self.rich_handler.setFormatter(EmojiLogFormatter())-- # Configure the root logger+ # Configure the root logger: the levels now, the sink when discovery hands it over. Until then+ # every record is held, so nothing a boot says is lost or written in a shape nothing chose.root_logger = logging.getLogger()root_logger.setLevel(log_config.default_log_level.int_logging_level)- root_logger.addHandler(self.rich_handler)-- self.poor_handler = logging.StreamHandler()- self.poor_handler.setFormatter(LevelAndEmojiLogFormatter())- poor_loggers = {*log_config.poor_loggers, log_config.generic_poor_logger}- for logger_name in poor_loggers:- logger = logging.getLogger(logger_name)- logger.setLevel(log_config.default_log_level.int_logging_level)- logger.addHandler(self.poor_handler)- logger.propagate = False+ self._holding_handler = HoldingLogHandler()+ root_logger.addHandler(self._holding_handler)self.set_levels_for_packages(package_log_levels=log_config.package_log_levels)self.verbose("Logs configured and config set")- def set_poor_log_formatter(self, formatter: logging.Formatter):- """Set the formatter for the poor log handler.+ def install_sink(self, sink: LogSink):+ """Put the sink's handler on the root logger and replay through it every record held since ``configure``.- Args:- formatter (logging.Formatter): The formatter to use for poor logging.+ The handler is built here, so a sink whose dependency is missing fails at this call, at boot,+ with the extra named. Once per configuration: a second sink is refused, as a second ``configure`` is.++ Raises:+ RuntimeError: If logging is not configured, or a sink is already installed.+ MissingDependencyError: If the sink's handler needs a package that is not installed."""- if self.poor_handler is None:- msg = "Poor log handler is not set."+ if self._log_config_instance is None:+ msg = "Logging is not configured. Call log.configure() before log.install_sink()."raise RuntimeError(msg)- self.poor_handler.setFormatter(formatter)+ if self._sink is not None:+ msg = "A log sink is already installed. You can only call log.install_sink() once per configuration."+ raise RuntimeError(msg)++ handler = sink.handler+ # Ahead of every other filter, so the sink's processors never run on a record it rejects.+ handler.filters.insert(0, ForwardedRecordFilter())+ root_logger = logging.getLogger()+ # Recorded before the replay: a handler that raises on one held record leaves the sink+ # installed all the same, so ``reset`` finds it and removes it rather than leaking it into+ # the next boot.+ self._sink = sink+ if self._holding_handler is None:+ root_logger.addHandler(handler)+ return+ # The held records drain first, so every one of them precedes whatever is emitted from now+ # on. A record emitted meanwhile reaches the holding handler, which forwards it once and+ # marks it; once the sink's handler is on the root logger too, the guard on it rejects the+ # root's own delivery of that same record, so nothing is delivered twice, and nothing reaches+ # neither, since one of the two handlers is on the root at every instant.+ holding, self._holding_handler = self._holding_handler, None+ try:+ holding.release_to(handler=handler)+ finally:+ root_logger.addHandler(handler)+ root_logger.removeHandler(holding)+ holding.close()def _should_ignore(self, problem_id: str | None = None) -> bool:"""Check if a log message should be ignored based on the problem ID.@@ -152,15 +209,13 @@ class Log:########################################################def redirect_to_stderr(self):- """Redirect the root logger's rich handler to stderr.+ """Point the installed sink at stderr when it writes to a process stream.- Used by the agent CLI to ensure no log output pollutes stdout.+ Used by the agent CLI to ensure no log output pollutes stdout. A no-op before a sink is+ installed, and for a sink that writes to no process stream."""- if self.rich_handler is not None:- if not isinstance(self.rich_handler, RichHandler):- msg = "Rich handler is not properly set."- raise RuntimeError(msg)- self.rich_handler.console = Console(file=sys.stderr)+ if self._sink is not None:+ self._sink.redirect_to_stderr()def set_level_by_int(self, *, level_int: int):"""Set the log level using an integer value.diff --git a/pipelex/tools/log/log_config.py b/pipelex/tools/log/log_config.pyindex 288f992a3..a32629f52 100644--- a/pipelex/tools/log/log_config.py+++ b/pipelex/tools/log/log_config.py@@ -1,13 +1,9 @@from __future__ import annotations-import sysfrom enum import StrEnumfrom typing import castfrom pydantic import Field, field_validator-from rich.console import Console-from rich.highlighter import Highlighter, JSONHighlighter, ReprHighlighter-from rich.logging import RichHandlerfrom pipelex.system.configuration.config_model import ConfigModelfrom pipelex.system.console_target import ConsoleTarget@@ -15,11 +11,6 @@ from pipelex.tools.log.log_levels import LogLevelfrom pipelex.tools.misc.pretty import PrettyPrintMode-class LogMode(StrEnum):- RICH = "rich"- POOR = "poor"--class HighlighterName(StrEnum):JSON = "json"REPR = "repr"@@ -58,6 +49,8 @@ class CallerInfoTemplate(StrEnum):class RichLogConfig(ConfigModel):+ """The settings of the ``console`` sink's Rich handler. Read by the sink; this module imports no Rich."""+is_show_time: boolis_show_level: boolis_link_path_enabled: bool@@ -69,49 +62,29 @@ class RichLogConfig(ConfigModel):tracebacks_suppress: list[str]keywords_to_hilight: list[str]- def make_rich_handler(self, target: ConsoleTarget) -> RichHandler:- match target:- case ConsoleTarget.STDOUT:- console = Console(file=sys.stdout)- case ConsoleTarget.STDERR:- console = Console(file=sys.stderr)- # case ConsoleTarget.FILE:- # console = Console(file=target.file_path)- case _:- msg = f"Invalid console target: {target}"- raise ValueError(msg)- highlighter: Highlighter- match self.highlighter_name:- case HighlighterName.JSON:- highlighter = JSONHighlighter()- case HighlighterName.REPR:- highlighter = ReprHighlighter()-- return RichHandler(- console=console,- show_time=self.is_show_time,- show_level=self.is_show_level,- enable_link_path=self.is_link_path_enabled,- highlighter=highlighter,- markup=self.is_markup_enabled,- rich_tracebacks=self.is_rich_tracebacks,- tracebacks_word_wrap=self.is_tracebacks_word_wrap,- tracebacks_show_locals=self.is_tracebacks_show_locals,- tracebacks_suppress=self.tracebacks_suppress,- keywords=self.keywords_to_hilight,- )++class OtlpLogSinkConfig(ConfigModel):+ """The settings of the ``otlp`` sink.++ An absent ``endpoint`` leaves the exporter to the OpenTelemetry environment conventions:+ ``OTEL_EXPORTER_OTLP_LOGS_ENDPOINT``, then ``OTEL_EXPORTER_OTLP_ENDPOINT`` with the ``/v1/logs``+ path, then the collector default on localhost. Empty ``headers`` likewise leave+ ``OTEL_EXPORTER_OTLP_HEADERS`` in charge.+ """++ endpoint: str | None = None+ headers: dict[str, str]class LogConfig(ConfigModel):default_log_level: LogLevel = Field(strict=False)package_log_levels: dict[str, LogLevel]- log_mode: LogMode = Field(strict=False)+ # The registered log-sink token boot selects: an open string, validated at the registry lookup.+ sink: strpretty_print_mode: PrettyPrintMode = Field(strict=False)console_log_target: ConsoleTarget = Field(strict=False)console_print_target: ConsoleTarget = Field(strict=False)- is_console_logging_enabled: bool-json_logs_indent: intpresentation_line_width: intis_caller_info_enabled: bool@@ -120,10 +93,7 @@ class LogConfig(ConfigModel):silenced_problem_ids: list[str]rich_log: RichLogConfig-- # logger name to use for safe logging without fancy features like code filepath and stuff- generic_poor_logger: str- poor_loggers: list[str]+ otlp: OtlpLogSinkConfig@field_validator("package_log_levels", mode="before")@classmethoddiff --git a/pipelex/tools/log/log_context.py b/pipelex/tools/log/log_context.pyindex edac11501..9ab46d8ef 100644--- a/pipelex/tools/log/log_context.py+++ b/pipelex/tools/log/log_context.py@@ -18,8 +18,8 @@ from pydantic import BaseModel, ConfigDictif TYPE_CHECKING:from collections.abc import Generator-# The reserved identifier names. They are spelled exactly as the payload fields they come from, so a-# record and the ``ErrorReport`` or the run result it relates to agree on the key.+# The reserved identifier names. They are spelled exactly as the payload fields they come from, the+# job metadata's and the run result's, so a record and the run it relates to agree on the key.REQUEST_ID_FIELD = "request_id"PIPELINE_RUN_ID_FIELD = "pipeline_run_id"PIPE_RUN_ID_FIELD = "pipe_run_id"diff --git a/pipelex/tools/log/log_dispatch.py b/pipelex/tools/log/log_dispatch.pyindex a6be0e8d7..561dc11fa 100644--- a/pipelex/tools/log/log_dispatch.py+++ b/pipelex/tools/log/log_dispatch.py@@ -12,11 +12,11 @@ from __future__ import annotationsimport inspectimport jsonimport logging-import traceback+import sysfrom pathlib import Pathfrom typing import TYPE_CHECKING, Any, cast-from pipelex.tools.log.log_config import CallerInfoTemplate, LogConfig, LogMode+from pipelex.tools.log.log_config import CallerInfoTemplate, LogConfigfrom pipelex.tools.log.log_context import get_log_contextfrom pipelex.tools.log.log_fields import attach_log_record_extra, build_log_record_extrafrom pipelex.tools.misc.json_utils import purify_json, purify_json_dict, purify_json_list@@ -25,6 +25,9 @@ if TYPE_CHECKING:from collections.abc import Mappingfrom types import FrameType+ # The triple ``sys.exc_info`` returns while an exception is being handled.+ ExcInfo = tuple[type[BaseException], BaseException, Any]+# The frames between ``_caller_frame`` and the code that called ``log.<level>(...)``: the helper's own# frame, then ``LogDispatch.dispatch``, then the ``Log`` method. A change to that call chain moves this number.CALLER_FRAME_DEPTH = 3@@ -53,6 +56,14 @@ def _module_name(*, frame: FrameType | None) -> str:return module_name or UNKNOWN_MODULE_NAME+def _active_exc_info() -> ExcInfo | None:+ """The exception being handled, as the record's ``exc_info``, or ``None`` outside any ``except`` block."""+ exc_type, exc_value, exc_traceback = sys.exc_info()+ if exc_type is None or exc_value is None:+ return None+ return exc_type, exc_value, exc_traceback++class LogDispatch:"""Turns the facade's calls into records on module-named stdlib loggers."""@@ -62,10 +73,6 @@ class LogDispatch:# TODO: more elegant init for log_dispatch / logdef __init__(self):self._log_config_instance: LogConfig | None = None- self.log_mode: LogMode = LogMode.RICH-- def set_log_mode(self, mode: LogMode):- self.log_mode = modedef reset(self):"""Reset the log dispatch."""@@ -85,7 +92,6 @@ class LogDispatch:msg = "LogConfig is already set. You can only call log.configure() once."raise RuntimeError(msg)self._log_config_instance = log_config- self.log_mode = log_config.log_mode######################################################### Dispatch@@ -109,16 +115,15 @@ class LogDispatch:severity: The severity level of the log message.title: A title rendered above the content. Defaults to None.inline: A title rendered inline before a string content, used only when ``title`` is None.- include_exception: Whether to append the current exception's traceback to the message.+ include_exception: Whether to carry the exception being handled on the record, as its+ ``exc_info``, for every sink to render its own way. Nothing is spliced into the message.fields: Named values carried as attributes of the record, never rendered into the message."""caller_frame = _caller_frame()module_name = _module_name(frame=caller_frame)log_config = self._log_config_instance- if log_config is not None and not log_config.is_console_logging_enabled:- return- logger = self._logger_for(module_name=module_name, log_config=log_config)+ logger = logging.getLogger(module_name)if not logger.isEnabledFor(severity):return@@ -126,22 +131,15 @@ class LogDispatch:if log_config is not None and log_config.is_caller_info_enabled and caller_frame is not None:caller_info_str = self._caller_info(frame=caller_frame, module_name=module_name, log_config=log_config)message = f"{caller_info_str}: {message}"- if include_exception:- message += f"\n{traceback.format_exc()}"+ exc_info = _active_exc_info() if include_exception else Noneextra = build_log_record_extra(context=get_log_context(), fields=fields, data=data)- self._log_to_console(message=message, severity=severity, logger=logger, caller_frame=caller_frame, extra=extra)+ self._emit_record(message=message, severity=severity, logger=logger, caller_frame=caller_frame, exc_info=exc_info, extra=extra)######################################################### Private methods########################################################- def _logger_for(self, *, module_name: str, log_config: LogConfig | None) -> logging.Logger:- """The module-named logger, or the generic poor logger when the dispatch runs in poor mode."""- if self.log_mode == LogMode.POOR and log_config is not None:- return logging.getLogger(log_config.generic_poor_logger)- return logging.getLogger(module_name)-def _render_content(self,*,@@ -164,16 +162,23 @@ class LogDispatch:return "None", Noneindent = log_config.json_logs_indent if log_config is not None else None- if isinstance(content, dict):- _, rendered = purify_json_dict(data=content, indent=indent, is_warning_enabled=True)- elif isinstance(content, list):- _, rendered = purify_json_list(data=cast("list[Any]", content), indent=indent, is_truncate_bytes_enabled=True)- else:- _, rendered = purify_json(data=content, indent=indent, is_truncate_bytes_enabled=True, is_warning_enabled=False)- # The structure the helpers hand back is the caller's own object whenever it was JSON-clean as- # given, and a sink that serializes later would read whatever the caller did to it since. The- # data is therefore the rendering re-read: a snapshot of the call, JSON-ready whatever it held.- data: Any = json.loads(rendered)+ data: Any | None+ try:+ if isinstance(content, dict):+ _, rendered = purify_json_dict(data=content, indent=indent, is_warning_enabled=True)+ elif isinstance(content, list):+ _, rendered = purify_json_list(data=cast("list[Any]", content), indent=indent, is_truncate_bytes_enabled=True)+ else:+ _, rendered = purify_json(data=content, indent=indent, is_truncate_bytes_enabled=True, is_warning_enabled=False)+ # The structure the helpers hand back is the caller's own object whenever it was JSON-clean as+ # given, and a sink that serializes later would read whatever the caller did to it since. The+ # data is therefore the rendering re-read: a snapshot of the call, JSON-ready whatever it held.+ data = json.loads(rendered)+ except (TypeError, ValueError):+ # What ``json`` refuses outright, a circular reference or a mapping with a non-string key,+ # is rendered as its ``repr`` and carries no ``data``: a log call never raises.+ rendered = repr(cast("object", content))+ data = Nonemessage = f"\n{rendered}"if title is not None:message = f"{title}:{message}"@@ -191,13 +196,14 @@ class LogDispatch:template_str = CallerInfoTemplate.for_template_key(key=log_config.caller_info_template)return template_str.format(file=str(caller_path), line=frame.f_lineno, func=frame.f_code.co_name, module=module_name)- def _log_to_console(+ def _emit_record(self,- message: str,*,+ message: str,severity: int,logger: logging.Logger,caller_frame: FrameType | None,+ exc_info: ExcInfo | None,extra: dict[str, Any],):"""Build the record at the caller's location, attach what it carries, and hand it to the logger's handlers.@@ -216,7 +222,7 @@ class LogDispatch:lno=lineno,msg=message,args=(),- exc_info=None,+ exc_info=exc_info,func=func_name,)attach_log_record_extra(record=record, extra=extra)diff --git a/pipelex/tools/log/log_fields.py b/pipelex/tools/log/log_fields.pyindex 868e4b9d0..a5d288763 100644--- a/pipelex/tools/log/log_fields.py+++ b/pipelex/tools/log/log_fields.py@@ -5,14 +5,19 @@ refuses an ``extra`` key that would overwrite one of the record's own attributescall never raises, so a colliding entry is carried under a prefixed name instead. The collision is readoff the record actually built, through whatever record factory is installed, so an attribute a factoryadded is a collision too rather than the stdlib's ``KeyError``.++Pipelex's own machinery owns a name on the record too, and it is reserved here for the same reason the+``json`` sink reserves its own keys: a name nobody owns *yet* is a name a caller's field lands on freely,+and this one steers delivery. So the reserved set spans what the formatter sets later and what this+package stamps later, and neither is a field a sink reads back."""from __future__ import annotations+import loggingfrom typing import TYPE_CHECKING, Anyif TYPE_CHECKING:- import loggingfrom collections.abc import Mappingfrom pipelex.tools.log.log_context import LogContext@@ -26,6 +31,39 @@ COLLIDING_FIELD_PREFIX = "field_"# Set by the formatter rather than the constructor, so a fresh record does not carry them yet and the stdlib refuses them all the same.FORMATTER_OWNED_ATTRIBUTES = frozenset({"message", "asctime"})+# The attribute the holding handler stamps on a record it has already forwarded to the sink's handler, and+# which that handler's forwarding filter reads to reject the root logger's own second delivery of it. The+# name lives here rather than beside the handler because reserving it is what makes it safe: it is stamped+# after the forward, so a fresh record does not own it, and a caller's field spelling it would be read as+# the marker and cost the whole record its delivery.+FORWARDED_MARK = "_pipelex_forwarded"++# Rich's own per-record override of its handler's markup setting, which it reads as+# ``getattr(record, "markup", self.markup)``, and so the one way to put one record on the console verbatim+# while the handler goes on interpreting markup for every other. A message assembled from an exception is+# what needs it: text like ``[Errno 2]`` is a markup tag to Rich, which swallows the span, or raises on an+# unbalanced one. The name is Rich's rather than ours, but stamping it is a console concern that rides no+# wire, and reserving it is what stops a caller steering the console through a field of that name.+VERBATIM_MARK = "markup"++# The names this package stamps on a record itself. Never a field: a caller's entry of the same name is+# prefixed on the way on, and a record carrying one does not hand it to a sink as something it carries.+PIPELEX_OWNED_ATTRIBUTES = frozenset({FORWARDED_MARK, VERBATIM_MARK})++# Reserved whether or not the record carries the name yet, which is exactly what the stdlib's own refusal+# cannot cover: both sets are stamped after the entries are attached.+RESERVED_ATTRIBUTES = FORMATTER_OWNED_ATTRIBUTES | PIPELEX_OWNED_ATTRIBUTES++# The attributes the stdlib gives every record, read off one built by the stdlib's own constructor on+# this interpreter rather than listed by hand, so a version that adds one (``taskName`` arrived with+# 3.12) is covered. Everything else on a record is what a call, a context or a record factory put there.+STDLIB_RECORD_ATTRIBUTES: frozenset[str] = (+ frozenset(+ vars(logging.LogRecord(name="", level=logging.NOTSET, pathname="", lineno=0, msg="", args=(), exc_info=None)),+ )+ | FORMATTER_OWNED_ATTRIBUTES+)+def build_log_record_extra(*,@@ -35,8 +73,11 @@ def build_log_record_extra() -> dict[str, Any]:"""The ``extra`` for one record, in precedence order: the bound context, then the call's fields, then the content.- A field overrides the context for its record, so a call site that names a request it is not- running under can say so; structured content owns ``data`` outright.+ A field overrides the context for its record, so a call site that names a request it is not running+ under can say so. Structured content owns the ``data`` name, and a field of that name is moved aside+ under the ``field_`` prefix rather than destroyed — the same discipline the record's own attributes+ and every wire sink's reserved keys follow, applied until the name lands where nothing sits, so a call+ passing both ``data`` and ``field_data`` beside structured content loses neither."""extra: dict[str, Any] = {}if context is not None:@@ -44,6 +85,11 @@ def build_log_record_extra(if fields:extra.update(fields)if data is not None:+ if DATA_FIELD in extra:+ displaced = f"{COLLIDING_FIELD_PREFIX}{DATA_FIELD}"+ while displaced in extra:+ displaced = f"{COLLIDING_FIELD_PREFIX}{displaced}"+ extra[displaced] = extra[DATA_FIELD]extra[DATA_FIELD] = datareturn extra@@ -52,14 +98,28 @@ def attach_log_record_extra(*, record: logging.LogRecord, extra: Mapping[str, An"""Set each entry as an attribute of the record, under a prefixed name when the record already owns that name.The record was built by the logger, through the installed record factory, so what it owns is exactly- what the stdlib's own ``makeRecord`` would refuse: its declared attributes and anything a factory- stamped on it. The prefix is applied until the name lands on an attribute nobody owns, and entries- are attached in order, so an entry attached earlier under a prefixed name is owned for the entries- after it: whatever the order of the mapping, no value is lost.+ what the stdlib's own ``makeRecord`` would refuse: its declared attributes, anything a factory stamped+ on it, and what the ``LogRecord`` class itself owns, ``getMessage`` and the dunders among them, which+ a lookup in the instance dict alone would miss. Beside those, the names set later than this — the+ formatter's and this package's own — are reserved though nothing owns them yet, because a record that+ accepted one would go on to be read as having been formatted or forwarded. The prefix is applied until+ the name lands on an attribute nobody owns, and entries are attached in order, so an entry attached+ earlier under a prefixed name is owned for the entries after it: whatever the order of the mapping, no+ value is lost."""- owned = vars(record)for name, value in extra.items():attribute = name- while attribute in owned or attribute in FORMATTER_OWNED_ATTRIBUTES:+ while hasattr(record, attribute) or attribute in RESERVED_ATTRIBUTES:attribute = f"{COLLIDING_FIELD_PREFIX}{attribute}"setattr(record, attribute, value)+++def carried_attributes(*, record: logging.LogRecord) -> dict[str, Any]:+ """Everything on the record that is not the stdlib's or ours: the fields, the context identifiers, ``data``, a factory's stamps.++ Read the way a structured sink reads a record, in the order the attributes were attached. A value is+ handed back as the call gave it: a sink serializes it when it emits, on the calling thread. What this+ package stamps on a record itself is machinery and belongs on no wire, so it is left out here as it is+ reserved on the way in.+ """+ return {name: value for name, value in vars(record).items() if name not in STDLIB_RECORD_ATTRIBUTES and name not in PIPELEX_OWNED_ATTRIBUTES}diff --git a/pipelex/tools/log/log_formatter.py b/pipelex/tools/log/log_formatter.pyindex 0f8302bde..a783fdb46 100644--- a/pipelex/tools/log/log_formatter.py+++ b/pipelex/tools/log/log_formatter.py@@ -2,9 +2,6 @@ import loggingfrom typing_extensions import override-from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE-from pipelex.tools.misc.terminal_utils import BOLD_FONT, RESET_FONT, TerminalColor-# TODO: move these to the configdef emoji_for_channel(channel_name: str) -> str | None:@@ -27,9 +24,6 @@ def emoji_for_channel(channel_name: str) -> str | None:elif channel_name.startswith("kajson"):# space added to make it look betterreturn "*️⃣ "- elif channel_name.startswith("#poor-log"):- # space added to make it look better- return "🧿 "elif channel_name.startswith("pipelex"):return "🧠"else:@@ -37,56 +31,19 @@ def emoji_for_channel(channel_name: str) -> str | None:class EmojiLogFormatter(logging.Formatter):- @override- def format(self, record: logging.LogRecord):- log_fmt: str- emoji = emoji_for_channel(record.name)- if emoji == "":- log_fmt = "%(message)s"- elif emoji:- log_fmt = f"{emoji}: %(message)s"- else:- log_fmt = "[%(name)s]: %(message)s"- formatter = logging.Formatter(log_fmt)-- return formatter.format(record)+ """The console sink's formatter: an emoji for the logger's prefix, then the message, and the record left as it was.+ The prefix goes on in ``formatMessage`` rather than ``format``: the Rich handler renders a record+ that carries ``exc_info`` from ``formatMessage`` alone and discards what ``format`` returned, so a+ prefix built anywhere else vanished from exactly the lines that carry a traceback. The base class's+ ``format`` sets ``record.message``, calls this, and appends the exception text.+ """-log_level_color: dict[int, TerminalColor] = {- LOGGING_LEVEL_VERBOSE: TerminalColor.WHITE,- logging.DEBUG: TerminalColor.GREEN,- LOGGING_LEVEL_DEV: TerminalColor.CYAN,- logging.INFO: TerminalColor.BLUE,- logging.WARNING: TerminalColor.YELLOW,- logging.ERROR: TerminalColor.RED,- logging.CRITICAL: TerminalColor.MAGENTA,-}--# added spaces and truncated to 5 characters to make it look better-log_level_tag: dict[int, str] = {- LOGGING_LEVEL_VERBOSE: "VERBO",- logging.DEBUG: "DEBUG",- LOGGING_LEVEL_DEV: "DEV ",- logging.INFO: "INFO ",- logging.WARNING: f"{BOLD_FONT}WARNING{RESET_FONT}",- logging.ERROR: f"{BOLD_FONT}/ERROR\\{RESET_FONT}",- logging.CRITICAL: f"{BOLD_FONT}CRITICAL{RESET_FONT}",-}---class LevelAndEmojiLogFormatter(logging.Formatter):@override- def format(self, record: logging.LogRecord):+ def formatMessage(self, record: logging.LogRecord) -> str:emoji = emoji_for_channel(record.name)- color = log_level_color.get(record.levelno, RESET_FONT)- tag = log_level_tag.get(record.levelno, "?????")+ if emoji == "":+ return record.messageif emoji:- log_fmt = f"{emoji} {color}{tag}:{RESET_FONT} %(message)s"- else:- log_fmt = f"{color}{tag}:%(name)s{RESET_FONT} %(message)s"- formatter = logging.Formatter(log_fmt)-- if record.levelno in {logging.WARNING, logging.ERROR, logging.CRITICAL}:- record.msg = f"{color}{BOLD_FONT}{record.msg}{RESET_FONT}"-- return formatter.format(record)+ return f"{emoji}: {record.message}"+ return f"[{record.name}]: {record.message}"diff --git a/pipelex/tools/log/log_holding.py b/pipelex/tools/log/log_holding.pynew file mode 100644index 000000000..5ba8c6745--- /dev/null+++ b/pipelex/tools/log/log_holding.py@@ -0,0 +1,130 @@+"""The handler that holds records between ``log.configure`` and the sink's arrival.++Logging is configured as soon as the configuration is read, and the sink is a capability the plugin+discovery hands over a little later in the boot. The records emitted in between are held here, in+order, and replayed through the sink's handler the moment it is installed, so a boot's own lines are+rendered by the sink the configuration chose rather than dropped or written in a shape nothing chose.+A boot that dies before a sink arrives closes this handler, and what it still holds reaches stderr+through the stdlib's last resort — every record of it, since the root logger's level already admitted+them and that level came from the configuration, so each held line is one this process was asked to+show. A boot that failed is when that trail is worth most.+"""++from __future__ import annotations++import logging++from typing_extensions import override++from pipelex.tools.log.log_fields import FORWARDED_MARK++# The most records held at once; beyond it the oldest are dropped, so a process that configures+# logging and never installs a sink cannot grow without bound. A boot holds a few dozen lines.+HOLDING_CAPACITY = 1000+++class ForwardedRecordFilter(logging.Filter):+ """Rejects a record the holding handler already forwarded to this handler.++ While the sink's handler and the holding handler are both on the root logger, a record reaches the+ holding handler first, which forwards it and marks it; this filter is what stops the root logger's+ own delivery of the same record right after. It sits ahead of every other filter on the handler,+ so the sink's processors never run on a record it rejects, and it stays installed: after the+ handoff no record is marked again, and a thread that read the root's handler list mid-handoff+ can still be on its way.++ The mark is only ever ours, and that is a property of the attachment rather than of the spelling:+ the name is reserved in ``log_fields``, so a caller's field spelling it is carried under a prefix+ and cannot make this filter drop a record nobody delivered.+ """++ @override+ def filter(self, record: logging.LogRecord) -> bool:+ return not getattr(record, FORWARDED_MARK, False)+++def _deliver(*, handler: logging.Handler, record: logging.LogRecord) -> None:+ """Hand one record to the sink's handler the way the root logger would have.++ ``Logger.callHandlers`` is what checks a handler's level; ``Handler.handle`` does not, so a replay+ or a forward that called it alone would deliver what the live path withholds. And a record the+ handler cannot render gets the stdlib's own recovery, ``handleError``, rather than raising out of+ the log call that emitted it.+ """+ if record.levelno < handler.level:+ return+ try:+ handler.handle(record)+ except Exception: # ruff: ignore[blind-except]+ handler.handleError(record)+++class HoldingLogHandler(logging.Handler):+ """Holds every record it is handed until a sink's handler takes them over, then forwards to that handler."""++ def __init__(self) -> None:+ super().__init__(level=logging.NOTSET)+ self._held: list[logging.LogRecord] = []+ self._released_to: logging.Handler | None = None++ @property+ def held_count(self) -> int:+ return len(self._held)++ @override+ def emit(self, record: logging.LogRecord) -> None:+ # ``handle`` holds this handler's lock here, the lock ``release_to`` drains under, so a record+ # arrives either before the drain and is held, or after it and is forwarded to the handler+ # that took the held ones: never into a list nobody reads again. The mark goes on after the+ # forward, so the forward passes the handler's own guard and the root logger's delivery of+ # the same record, when the sink's handler is on the root too, does not.+ if self._released_to is not None:+ _deliver(handler=self._released_to, record=record)+ setattr(record, FORWARDED_MARK, True)+ return+ if len(self._held) >= HOLDING_CAPACITY:+ del self._held[0]+ self._held.append(record)++ def release_to(self, *, handler: logging.Handler) -> None:+ """Hand every held record to the handler, in the order they were emitted, and forward to it whatever arrives after.++ One record the handler cannot render, a line Rich reads as unbalanced markup for one, gets the+ stdlib's own recovery, ``handleError``, and costs none of the records after it.++ A drained record is marked after its delivery exactly as a forwarded one is, and for the same+ reason: a thread that read the root logger's handler list before the handoff can still reach the+ sink's handler carrying a record this drain has already delivered, and the mark is the only thing+ that tells the handler's guard so.+ """+ self.acquire()+ try:+ held, self._held = self._held, []+ self._released_to = handler+ for record in held:+ _deliver(handler=handler, record=record)+ setattr(record, FORWARDED_MARK, True)+ finally:+ self.release()++ @override+ def close(self) -> None:+ """Give what is still held the stdlib's last-resort handling: every held record reaches stderr.++ Every one of them, and not only a warning or worse. The last resort's own level is meant for a+ record emitted before anything was configured, where nobody has said what is worth seeing; these+ records passed the root logger's level, which ``configure`` set from the configuration, so each one+ is a line this process was asked to show. The only moment this runs with anything still held is a+ boot that died before its sink arrived, which is exactly when the trail a verbose run was turned on+ to produce is the thing being looked for.+ """+ self.acquire()+ try:+ held, self._held = self._held, []+ finally:+ self.release()+ last_resort = logging.lastResort+ if last_resort is not None:+ for record in held:+ last_resort.handle(record)+ super().close()diff --git a/pipelex/tools/log/log_sink.py b/pipelex/tools/log/log_sink.pynew file mode 100644index 000000000..46e38bc4a--- /dev/null+++ b/pipelex/tools/log/log_sink.py@@ -0,0 +1,172 @@+"""Where the records go: the sink seam.++A sink owns one stdlib handler on the root logger and everything about how a record leaves the+process: the stream, the wire shape, the exporter. Which sink a process runs is a registrar+capability selected by ``[runtime.log] sink``, exactly as the storage and secrets providers are: the+built-in plugin registers ``json``, ``console`` and ``otlp`` under open string tokens, and an external+plugin registers its own. This module holds what every sink shares, the base class with its processor+slot and the stream resolution, and names no sink.+"""++from __future__ import annotations++import json+import logging+import math+import sys+from abc import ABC, abstractmethod+from collections.abc import Callable+from enum import StrEnum+from typing import TYPE_CHECKING, Any, cast++from pydantic import BaseModel+from typing_extensions import override++from pipelex.system.console_target import ConsoleTarget++if TYPE_CHECKING:+ from collections.abc import Sequence+ from typing import TextIO++# JSON has no spelling for a float that is not a number: ``json.dumps`` writes the bare tokens ``NaN``+# and ``Infinity`` by default, which a strict parser refuses line and all. A wire sink writes these+# strings in their place.+NAN_TEXT = "NaN"+POSITIVE_INFINITY_TEXT = "Infinity"+NEGATIVE_INFINITY_TEXT = "-Infinity"++# A processor edits a record in place before the sink renders it; redaction is the intended one. It+# runs on the sink's own handler, once per record, so a handler another integration attached to the+# root logger sees the record as the processors left it only if it runs after this one.+LogRecordProcessor = Callable[[logging.LogRecord], None]+++class LogSinkMethod(StrEnum):+ """The tokens the built-in plugin registers. An external plugin registers its own string."""++ JSON = "json"+ CONSOLE = "console"+ OTLP = "otlp"+++class _ProcessorFilter(logging.Filter):+ """Runs the sink's processors over each record before the handler formats it, and never drops one."""++ def __init__(self, processors: list[LogRecordProcessor]):+ super().__init__()+ self._processors = processors++ @override+ def filter(self, record: logging.LogRecord) -> bool:+ for processor in self._processors:+ processor(record)+ return True+++class LogSink(ABC):+ """One destination for the records the root logger emits.++ A subclass builds the stdlib handler in ``make_handler``, and does whatever heavy import it needs+ there rather than at module load, so that selecting a sink is what pays for its dependency. The+ base attaches the processor slot to that handler and hands the same handler back on every read.+ """++ def __init__(self) -> None:+ self.processors: list[LogRecordProcessor] = []+ self._handler: logging.Handler | None = None++ @abstractmethod+ def make_handler(self) -> logging.Handler:+ """Build the handler this sink installs on the root logger. Called once, at install."""++ @property+ def handler(self) -> logging.Handler:+ """The sink's handler, built on first read with the processors wired in front of it."""+ if self._handler is None:+ handler = self.make_handler()+ handler.addFilter(_ProcessorFilter(self.processors))+ self._handler = handler+ return self._handler++ def redirect_to_stderr(self) -> None:+ """Point a sink that writes to a process stream at stderr; a sink that does not ignores the call."""+ return+++def stream_for_target(*, target: ConsoleTarget) -> TextIO:+ """The process stream a target names, read at call time so a redirected stream is the one used."""+ match target:+ case ConsoleTarget.STDOUT:+ return sys.stdout+ case ConsoleTarget.STDERR:+ return sys.stderr+ case ConsoleTarget.FILE:+ msg = f"A log sink cannot write to the console target '{target}': choose stdout or stderr."+ raise ValueError(msg)+++def json_fallback(value: Any) -> Any: # kw-only: ignore — json.dumps calls its ``default`` positionally+ """What ``json.dumps`` writes for a value it does not know: a model's JSON dump, anything else as text.++ A model's dump is spelled the way a value handed in directly is. ``model_dump(mode="json")`` returns a+ plain mapping that still holds the Python float, pydantic's own spelling of a non-finite one applying+ only where pydantic writes the JSON text itself, so a ``NaN`` nested in a model would reach ``json``+ under ``allow_nan=False`` and cost the whole payload its structure. Spelled here, it costs that one+ float its type and nothing else anything.+ """+ if isinstance(value, BaseModel):+ return spell_non_finite(value=value.model_dump(mode="json"))+ return str(value)+++def spell_non_finite(*, value: Any) -> Any:+ """The value with every non-finite float, at any depth, replaced by the string JSON can carry.++ A ``NaN`` becomes ``"NaN"``, an infinity ``"Infinity"`` or ``"-Infinity"``, inside a mapping or a+ sequence as at the top, and everything else is returned as it was, a bool included since a bool+ is never a float here. A tuple comes back as a list, which is what JSON would have made of it. A+ container that contains itself is returned as it is, for ``json`` to refuse the way it always did.+ """+ return _spell_non_finite(value=value, open_containers=set())+++def _spell_non_finite(*, value: Any, open_containers: set[int]) -> Any:+ if isinstance(value, float):+ if math.isnan(value):+ return NAN_TEXT+ if math.isinf(value):+ return POSITIVE_INFINITY_TEXT if value > 0 else NEGATIVE_INFINITY_TEXT+ return value+ container_id = id(value)+ if container_id in open_containers:+ return value+ if isinstance(value, dict):+ mapping = cast("dict[Any, Any]", value)+ open_containers.add(container_id)+ try:+ return {key: _spell_non_finite(value=item, open_containers=open_containers) for key, item in mapping.items()}+ finally:+ open_containers.discard(container_id)+ if isinstance(value, (list, tuple)):+ sequence = cast("Sequence[Any]", value)+ open_containers.add(container_id)+ try:+ return [_spell_non_finite(value=item, open_containers=open_containers) for item in sequence]+ finally:+ open_containers.discard(container_id)+ return value+++def render_json(*, value: Any) -> str:+ """Serialize one value for a wire sink without ever raising, and never as anything but JSON.++ A model dumps in JSON mode and an unknown object becomes its text; a non-finite float is spelled+ as the string ``"NaN"``, ``"Infinity"`` or ``"-Infinity"`` rather than the bare token ``json``+ would write. A value ``json`` refuses outright, a circular reference or a mapping with a non-string+ key, is written as its ``repr`` instead: a sink must render every record it is handed, and a+ serialization failure is a fact about the value, never a reason to lose the line.+ """+ try:+ return json.dumps(spell_non_finite(value=value), ensure_ascii=False, allow_nan=False, default=json_fallback)+ except (TypeError, ValueError):+ return json.dumps(repr(value), ensure_ascii=False)diff --git a/pipelex/tools/log/otlp_log_sink.py b/pipelex/tools/log/otlp_log_sink.pynew file mode 100644index 000000000..1ab148a55--- /dev/null+++ b/pipelex/tools/log/otlp_log_sink.py@@ -0,0 +1,212 @@+"""The ``otlp`` sink: the OpenTelemetry logs signal, so a collector receives logs beside the spans.++Each record becomes one OTel log record on the logger named after the emitting module: the message+is the body, the level maps onto the OTel severity scale, the record's fields, context identifiers and+``data`` ride as attributes, with a value the wire cannot carry as is written as JSON text, and an+exception lands under the ``exception.*`` semantic-convention keys. The semantic-convention keys the+sink writes itself — the source location and the exception — are reserved whether or not the record+carries an exception, exactly as the ``json`` sink reserves its own keys: a field named like one is+carried under the same ``field_`` prefix, so the same field survives a change of sink. The records the+sink's own export path emits, the SDK's and the transport's, are rejected by a filter on the handler and+never exported. This module imports the OpenTelemetry SDK at load, which is why the built-in plugin+imports it inside the ``otlp`` factory and nowhere else.+"""++from __future__ import annotations++import logging+import traceback+from time import time_ns+from typing import TYPE_CHECKING, Any, cast++from opentelemetry._logs import SeverityNumber # ruff: ignore[import-private-name]+from opentelemetry.context import (+ _SUPPRESS_INSTRUMENTATION_KEY, # ruff: ignore[import-private-name] # pyright: ignore[reportPrivateUsage]+ get_current,+ get_value,+)+from opentelemetry.sdk._logs import LoggerProvider # ruff: ignore[import-private-name]+from opentelemetry.semconv._incubating.attributes import code_attributes # ruff: ignore[import-private-name]+from opentelemetry.semconv.attributes import exception_attributes+from typing_extensions import override++from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes+from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE+from pipelex.tools.log.log_sink import LogSink, render_json++if TYPE_CHECKING:+ from collections.abc import Sequence++ from opentelemetry.sdk._logs import LogRecordProcessor as OTelLogRecordProcessor+ from opentelemetry.sdk.resources import Resource++# The loggers the OpenTelemetry SDK and its exporters write to. A record from one of them must not be+# exported through the pipeline that emitted it: an export failure would log, be exported, fail, and+# log again.+OTEL_LOGGER_PREFIX = "opentelemetry"++# The deadline a flush hands the provider. The SDK's batch processor discards it today and waits on+# its export lock unbounded, so the bound a flush actually has is the exporter's own timeout,+# ``OTEL_EXPORTER_OTLP_TIMEOUT``; the value is passed for the SDK version that honours it.+FLUSH_TIMEOUT_MILLIS = 5000++_ATTRIBUTE_SCALAR_TYPES = (str, bool, int, float)++# The semantic-convention keys the sink writes itself, reserved on every record whether or not it carries+# an exception: a field named like one is carried under the same ``field_`` prefix the record uses for a+# name the stdlib owns, so no value is lost, a field keeps one wire name whatever the record holds, and the+# same field survives this sink as it survives the ``json`` one.+RESERVED_ATTRIBUTE_KEYS = frozenset(+ {+ code_attributes.CODE_FILE_PATH,+ code_attributes.CODE_FUNCTION_NAME,+ code_attributes.CODE_LINE_NUMBER,+ exception_attributes.EXCEPTION_TYPE,+ exception_attributes.EXCEPTION_MESSAGE,+ exception_attributes.EXCEPTION_STACKTRACE,+ },+)+++class ExportPathFilter(logging.Filter):+ """Rejects the records the sink's own export path emits, before the handler's lock is taken.++ Two guards. The SDK's and the exporters' loggers are named ``opentelemetry.*`` and are rejected by+ name. The transport beneath the exporter, ``requests`` and ``urllib3`` for the HTTP one, logs under+ its own names, so the second guard reads the context value the SDK's batch processor attaches+ around ``exporter.export`` and rejects any record emitted while it is set: the records of an export+ in flight, on the exporting thread, whatever the transport is called. That key is a private SDK+ name; the test against the installed SDK is what pins it.++ A filter rather than a check inside ``emit``, because ``Handler.handle`` runs the filters before it+ takes the handler's lock and ``emit`` after. A guard in ``emit`` still let an exporting thread block+ on this lock to report its failure while ``logging.shutdown`` held it and waited for the export+ lock: a deadlock at exit, reproduced with the installed SDK.+ """++ @override+ def filter(self, record: logging.LogRecord) -> bool:+ if record.name == OTEL_LOGGER_PREFIX or record.name.startswith(f"{OTEL_LOGGER_PREFIX}."):+ return False+ return not get_value(_SUPPRESS_INSTRUMENTATION_KEY)+++def _severity_number(*, levelno: int) -> SeverityNumber:+ """The OTel severity for a stdlib level, our two custom levels placed where they sit on ours."""+ if levelno <= LOGGING_LEVEL_VERBOSE:+ return SeverityNumber.TRACE+ if levelno < LOGGING_LEVEL_DEV:+ return SeverityNumber.DEBUG+ if levelno < logging.INFO:+ return SeverityNumber.DEBUG4+ if levelno < logging.WARNING:+ return SeverityNumber.INFO+ if levelno < logging.ERROR:+ return SeverityNumber.WARN+ if levelno < logging.CRITICAL:+ return SeverityNumber.ERROR+ return SeverityNumber.FATAL+++def _severity_text(*, levelname: str) -> str:+ """The OTel display spelling: ``WARN`` for the stdlib's ``WARNING``, every other name as is."""+ return "WARN" if levelname == "WARNING" else levelname+++def _attribute_value(*, value: Any) -> Any:+ """A value the OTel attribute model carries as is, or its JSON text when it does not.++ The SDK drops a sequence whose elements are not all of one scalar type, and it compares types by+ equality, so ``[1, True]`` is mixed to it: the same test here, and JSON text for what would be dropped.+ """+ if isinstance(value, _ATTRIBUTE_SCALAR_TYPES):+ return value+ if isinstance(value, (list, tuple)):+ items: list[Any] = list(cast("Sequence[Any]", value))+ element_types: set[type[Any]] = {type(item) for item in items}+ if not items or (len(element_types) == 1 and next(iter(element_types)) in _ATTRIBUTE_SCALAR_TYPES):+ return items+ return render_json(value=value)+++class OtlpLogHandler(logging.Handler):+ """Translates each stdlib record into an OTel log record and emits it on the provider's logger."""++ def __init__(self, *, logger_provider: LoggerProvider) -> None:+ super().__init__(level=logging.NOTSET)+ self._logger_provider = logger_provider++ @override+ def emit(self, record: logging.LogRecord) -> None:+ try:+ logger = self._logger_provider.get_logger(record.name)+ logger.emit(+ timestamp=int(record.created * 1e9),+ observed_timestamp=time_ns(),+ context=get_current() or None,+ severity_text=_severity_text(levelname=record.levelname),+ severity_number=_severity_number(levelno=record.levelno),+ body=record.getMessage(),+ attributes=self._attributes(record=record),+ )+ except Exception: # ruff: ignore[blind-except]+ # The stdlib's own contract for a handler: report through ``handleError`` and never raise+ # out of the log call.+ self.handleError(record)++ @override+ def flush(self) -> None:+ self._logger_provider.force_flush(timeout_millis=FLUSH_TIMEOUT_MILLIS)++ @override+ def close(self) -> None:+ self._logger_provider.shutdown()+ super().close()++ @staticmethod+ def _attributes(*, record: logging.LogRecord) -> dict[str, Any]:+ """The record's source location, its exception and everything it carries, the sink's own keys first.++ The sink's keys go down before the carried attributes so that a field named like one is prefixed+ rather than silently overwriting the location or the exception it names — and the reservation covers+ every one of them whether or not this record has an exception, so a field's wire name never depends+ on that.+ """+ attributes: dict[str, Any] = {+ code_attributes.CODE_FILE_PATH: record.pathname,+ code_attributes.CODE_FUNCTION_NAME: record.funcName,+ code_attributes.CODE_LINE_NUMBER: record.lineno,+ }+ if record.exc_info:+ exc_type, exc_value, exc_traceback = record.exc_info+ if exc_type is not None:+ attributes[exception_attributes.EXCEPTION_TYPE] = exc_type.__name__+ if exc_value is not None:+ attributes[exception_attributes.EXCEPTION_MESSAGE] = str(exc_value)+ if exc_traceback is not None:+ attributes[exception_attributes.EXCEPTION_STACKTRACE] = "".join(traceback.format_exception(exc_type, exc_value, exc_traceback))+ for name, value in carried_attributes(record=record).items():+ key = name+ while key in attributes or key in RESERVED_ATTRIBUTE_KEYS:+ key = f"{COLLIDING_FIELD_PREFIX}{key}"+ attributes[key] = _attribute_value(value=value)+ return attributes+++class OtlpLogSink(LogSink):+ """The OpenTelemetry logs signal behind one log record processor, a batching exporter in production."""++ def __init__(self, *, processor: OTelLogRecordProcessor, resource: Resource | None = None) -> None:+ super().__init__()+ self._logger_provider = LoggerProvider(resource=resource)+ self._logger_provider.add_log_record_processor(processor)++ @property+ def logger_provider(self) -> LoggerProvider:+ return self._logger_provider++ @override+ def make_handler(self) -> logging.Handler:+ handler = OtlpLogHandler(logger_provider=self._logger_provider)+ handler.addFilter(ExportPathFilter())+ return handlerdiff --git a/pyproject.toml b/pyproject.tomlindex 885e2180a..d0252dd7e 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -26,7 +26,7 @@ dependencies = ["filetype>=1.2.0","httpx>=0.23.0,<1.0.0","idna>=3.15",- "instructor>=1.13", # 1.13+ exposes instructor.core; 1.11/1.12 had mypy typing errors+ "instructor>=1.13", # 1.13+ exposes instructor.core; 1.11/1.12 had mypy typing errors"jinja2>=3.1.4","json2html>=1.3.0","kajson==0.7.1",@@ -34,10 +34,10 @@ dependencies = ["mthds==0.16.0","networkx>=3.4.2","openai>=2.0.0",- "opentelemetry-api",- "opentelemetry-exporter-otlp-proto-http",- "opentelemetry-semantic-conventions",- "opentelemetry-sdk",+ "opentelemetry-api>=1.38.0",+ "opentelemetry-exporter-otlp-proto-http>=1.38.0",+ "opentelemetry-semantic-conventions>=0.59b0",+ "opentelemetry-sdk>=1.38.0", # 1.38+ takes the keyword-only emit() arguments the otlp sink passes"pillow>=12.1.1","pipelex-tools-py>=0.1.3","polyfactory>=2.21.0",@@ -48,7 +48,7 @@ dependencies = ["pydantic>=2.10.6,<3.0.0","python-dotenv>=1.2.2","PyYAML>=6.0.2",- "reportlab>=4.0,<5", # for generating synthetic PDFs+ "reportlab>=4.0,<5", # for generating synthetic PDFs"requests>=2.33.0","rich>=13.8.1","semantic-version>=2.10.0",diff --git a/subject_grants.toml b/subject_grants.tomlindex 2bf168e8c..d292614bf 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -5084,6 +5084,22 @@ rationale = "Plugin protocol register: the registrar is the sole sink the pluginparam = "date_str"rationale = "Parses/consumes the date_str; single operand."+["pipelex/providers/log_sinks/log_sink_plugin.py::LogSinkPlugin.register"]+param = "registrar"+rationale = "Plugin protocol: registers into the registrar; single operand, consistent with sibling plugins."++["pipelex/providers/log_sinks/log_sink_plugin.py::_make_console_log_sink"]+param = "config"+rationale = "Factory from source: builds the sink from the config; single operand, the LogSinkFactoryFn shape."++["pipelex/providers/log_sinks/log_sink_plugin.py::_make_json_log_sink"]+param = "config"+rationale = "Factory from source: builds the sink from the config; single operand, the LogSinkFactoryFn shape."++["pipelex/providers/log_sinks/log_sink_plugin.py::_make_otlp_log_sink"]+param = "config"+rationale = "Factory from source: builds the sink from the config; single operand, the LogSinkFactoryFn shape."+["pipelex/providers/manifold/manifold_completions_factory.py::ManifoldCompletionsFactory.make_openai_client_for_completions"]param = "model_handle"rationale = "Factory from source: builds the result from the model_handle; single operand."@@ -6096,6 +6112,10 @@ rationale = "Logs the content; single operand (formatting keyword-only)."param = "content"rationale = "Logs the content; single operand (formatting keyword-only)."+["pipelex/tools/log/log.py::Log.install_sink"]+param = "sink"+rationale = "Setter: installs the sink; single operand."+["pipelex/tools/log/log.py::Log.set_level"]param = "level"rationale = "Setter: installs the level; single operand."@@ -6112,14 +6132,6 @@ rationale = "Setter: installs the package_name; single operand."param = "package_log_levels"rationale = "Setter: installs the package_log_levels; single operand."-["pipelex/tools/log/log.py::Log.set_log_mode"]-param = "mode"-rationale = "Setter: installs the mode; single operand."--["pipelex/tools/log/log.py::Log.set_poor_log_formatter"]-param = "formatter"-rationale = "Setter: installs the formatter; single operand."-["pipelex/tools/log/log.py::Log.verbose"]param = "content"rationale = "Logs the content; single operand (formatting keyword-only)."@@ -6132,14 +6144,6 @@ rationale = "Logs the content; single operand (formatting keyword-only)."param = "key"rationale = "Factory from source: builds the result from the key; single operand."-["pipelex/tools/log/log_config.py::RichLogConfig.make_rich_handler"]-param = "target"-rationale = "Factory from source: builds the result from the target; single operand."--["pipelex/tools/log/log_dispatch.py::LogDispatch._log_to_console"]-param = "message"-rationale = "Transforms the message; single operand."-["pipelex/tools/log/log_dispatch.py::LogDispatch.configure"]param = "log_config"rationale = "Setter: installs the log_config; single operand."@@ -6148,10 +6152,6 @@ rationale = "Setter: installs the log_config; single operand."param = "content"rationale = "Logs the content; single operand (formatting keyword-only)."-["pipelex/tools/log/log_dispatch.py::LogDispatch.set_log_mode"]-param = "mode"-rationale = "Setter: installs the mode; single operand."-["pipelex/tools/log/log_formatter.py::emoji_for_channel"]param = "channel_name"rationale = "Renders the channel_name; single operand."diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txtindex f95916d68..34d857991 100644--- a/tests/data/errors/error_identity.txt+++ b/tests/data/errors/error_identity.txt@@ -69,6 +69,7 @@ DryRunObjectFidelityError | Dry run object fidelity | https://docs.pipelex.com/lDuplicateBundleValidatorError | Duplicate bundle validator | https://docs.pipelex.com/latest/errors/duplicate-bundle-validator-error/DuplicateHttpErrorMapperError | Duplicate http error mapper | https://docs.pipelex.com/latest/errors/duplicate-http-error-mapper-error/DuplicateInferenceBackendError | Duplicate inference backend | https://docs.pipelex.com/latest/errors/duplicate-inference-backend-error/+DuplicateLogSinkError | Duplicate log sink | https://docs.pipelex.com/latest/errors/duplicate-log-sink-error/DuplicateModelListerError | Duplicate model lister | https://docs.pipelex.com/latest/errors/duplicate-model-lister-error/DuplicateOrchestratorError | Duplicate orchestrator | https://docs.pipelex.com/latest/errors/duplicate-orchestrator-error/DuplicatePipeFuncExecutorError | Duplicate pipe func executor | https://docs.pipelex.com/latest/errors/duplicate-pipe-func-executor-error/@@ -318,6 +319,7 @@ ToolError | Tool error | https://docs.pipelex.com/latest/errors/tool-error/TracebackMessageError | Traceback message | https://docs.pipelex.com/latest/errors/traceback-message-error/UnknownBootOrchestratorError | Unknown boot orchestrator | https://docs.pipelex.com/latest/errors/unknown-boot-orchestrator-error/UnknownInputNameError | Unknown input name | https://docs.pipelex.com/latest/errors/unknown-input-name-error/+UnknownLogSinkError | Unknown log sink | https://docs.pipelex.com/latest/errors/unknown-log-sink-error/UnknownPipeFuncExecutionModeError | Unknown pipe func execution mode | https://docs.pipelex.com/latest/errors/unknown-pipe-func-execution-mode-error/UnknownSecretsMethodError | Unknown secrets method | https://docs.pipelex.com/latest/errors/unknown-secrets-method-error/UnknownStorageMethodError | Unknown storage method | https://docs.pipelex.com/latest/errors/unknown-storage-method-error/diff --git a/tests/integration/pipelex/plugins/test_log_sink_external_plugin.py b/tests/integration/pipelex/plugins/test_log_sink_external_plugin.pynew file mode 100644index 000000000..fd1fcba7d--- /dev/null+++ b/tests/integration/pipelex/plugins/test_log_sink_external_plugin.py@@ -0,0 +1,128 @@+"""End-to-end: an out-of-tree log sink discovered through a ``pipelex.plugins.kernel`` entry point is+selectable via ``runtime.log.sink`` and installed on the root logger, and a token nobody provides stops+the boot naming the registered ones.++This exercises the whole discovery → selection → install chain the built-in sinks ride, with a fake+external token: a fake entry point loads a plugin registering ``method="test_sink"``, the boot config+selects that token, and the installed sink is the one on the root logger, the boot's own lines replayed+through it.+"""++from __future__ import annotations++import logging+from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest+from typing_extensions import override++from pipelex import log+from pipelex.pipelex import Pipelex+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.discovery import GroupedEntryPoint+from pipelex.plugins.exceptions import UnknownLogSinkError+from pipelex.plugins.plugin_group import PluginGroup+from pipelex.system.runtime import IntegrationMode, runtime_manager+from pipelex.tools.log.log_sink import LogSink, LogSinkMethod++if TYPE_CHECKING:+ from collections.abc import Generator+ from importlib.metadata import EntryPoint++ from pytest_mock import MockerFixture++ from pipelex.plugins.registrar import PluginRegistrar+ from pipelex.tools.log.log_config import LogConfig++EXTERNAL_SINK_METHOD = "test_sink"+EXTERNAL_PLUGIN_NAME = "test_log_sink_ext"+++class _RecordingHandler(logging.Handler):+ def __init__(self) -> None:+ super().__init__()+ self.records: list[logging.LogRecord] = []++ @override+ def emit(self, record: logging.LogRecord) -> None:+ self.records.append(record)+++class _RecordingLogSink(LogSink):+ """A minimal out-of-tree sink: its distinct type proves the selection, its records prove the install."""++ def __init__(self) -> None:+ super().__init__()+ self.recording_handler = _RecordingHandler()++ @override+ def make_handler(self) -> logging.Handler:+ return self.recording_handler+++def _make_fake_external_sink(_config: LogConfig) -> LogSink:+ return _RecordingLogSink()+++class _FakeLogSinkPlugin:+ """An external plugin contributing one log sink under the ``test_sink`` token."""++ name = EXTERNAL_PLUGIN_NAME+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_log_sink(method=EXTERNAL_SINK_METHOD, factory=_make_fake_external_sink)++[email protected](autouse=True)+def reset_pipelex_config_fixture() -> Generator[None, None, None]:+ """Override the global module fixture: this module boots per test and tears down."""+ Pipelex.teardown_if_needed()+ yield+ Pipelex.teardown_if_needed()+++def _test_integration_mode() -> IntegrationMode:+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST+++class TestLogSinkExternalPlugin:+ def test_external_sink_is_discovered_selected_and_installed_on_the_root_logger(self, mocker: MockerFixture) -> None:+ """A fake entry point + a config naming its token boots with that external sink's handler on the root logger."""+ fake_entry_point = SimpleNamespace(name=EXTERNAL_PLUGIN_NAME, load=lambda: _FakeLogSinkPlugin)+ # A log sink is a kernel-layer capability, so its dist publishes under the kernel group.+ grouped = GroupedEntryPoint(group=PluginGroup.KERNEL, entry_point=cast("EntryPoint", fake_entry_point))+ mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[grouped])++ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ config_overrides={"runtime": {"log": {"sink": EXTERNAL_SINK_METHOD}}},+ )+ log.info("after the boot")++ sink = log.sink+ assert isinstance(sink, _RecordingLogSink)+ assert sink.handler in logging.getLogger().handlers+ messages = [record.getMessage() for record in sink.recording_handler.records]+ assert "after the boot" in messages++ Pipelex.teardown_if_needed()+ assert log.sink is None+ assert sink.handler not in logging.getLogger().handlers++ def test_a_token_nobody_provides_stops_the_boot_naming_the_registered_sinks(self) -> None:+ with pytest.raises(UnknownLogSinkError) as exc_info:+ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ config_overrides={"runtime": {"log": {"sink": "nobody_provides_this"}}},+ )++ assert exc_info.value.method == "nobody_provides_this"+ assert set(exc_info.value.registered_methods) == {method.value for method in LogSinkMethod}+ assert "runtime.log.sink" in str(exc_info.value)+ # The failed boot released the process globals: logging is unconfigured again and no sink is left behind.+ assert log.sink is None+ assert Pipelex.get_optional_instance() is Nonediff --git a/tests/integration/pipelex/plugins/test_log_sink_json_boot.py b/tests/integration/pipelex/plugins/test_log_sink_json_boot.pynew file mode 100644index 000000000..e198fb721--- /dev/null+++ b/tests/integration/pipelex/plugins/test_log_sink_json_boot.py@@ -0,0 +1,62 @@+"""A boot with ``sink = "json"`` carries no Rich handler on the root logger, and each line it writes is one JSON object.++The hosted plane's runner and worker select this sink; this is the shape a log agent ingests.+"""++from __future__ import annotations++import json+import logging+from typing import TYPE_CHECKING++import pytest+from rich.logging import RichHandler++from pipelex import log+from pipelex.pipelex import Pipelex+from pipelex.system.runtime import IntegrationMode, runtime_manager+from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, SEVERITY_KEY, JsonLogSink+from pipelex.tools.log.log_sink import LogSinkMethod++if TYPE_CHECKING:+ from collections.abc import Generator++[email protected](autouse=True)+def reset_pipelex_config_fixture() -> Generator[None, None, None]:+ """Override the global module fixture: this module boots per test and tears down."""+ Pipelex.teardown_if_needed()+ yield+ Pipelex.teardown_if_needed()+++def _test_integration_mode() -> IntegrationMode:+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST+++class TestLogSinkJsonBoot:+ def test_json_boot_has_no_rich_handler_and_writes_one_json_object_per_line(self, capsys: pytest.CaptureFixture[str]) -> None:+ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ config_overrides={"runtime": {"log": {"sink": LogSinkMethod.JSON.value}}},+ )+ with log.context(request_id="req-json"):+ log.info("a line for the agent", fields={"files": 3})++ assert isinstance(log.sink, JsonLogSink)+ root_handlers = logging.getLogger().handlers+ assert log.sink.handler in root_handlers+ assert not any(isinstance(handler, RichHandler) for handler in root_handlers)++ captured = capsys.readouterr()+ assert captured.out == ""+ lines = [line for line in captured.err.splitlines() if line]+ parsed = [json.loads(line) for line in lines]+ assert all(isinstance(line, dict) for line in parsed)+ (own,) = [line for line in parsed if line[LOGGER_KEY] == __name__]+ assert own[MESSAGE_KEY] == "a line for the agent"+ assert own[SEVERITY_KEY] == "INFO"+ assert own["files"] == 3+ assert own["request_id"] == "req-json"+ assert "\x1b" not in captured.errdiff --git a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.pyindex efa075d12..9446f15fe 100644--- a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py+++ b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py@@ -19,6 +19,7 @@ from pipelex.interpreter_plugins.pipe_func.pipe_func_plugin import PipeFuncPlugifrom pipelex.pipelex import Pipelexfrom pipelex.plugins.contract import PLUGIN_API_VERSIONfrom pipelex.plugins.registrar import HubSlot, PluginOrigin, PluginRegistrar+from pipelex.providers.log_sinks.log_sink_plugin import LogSinkPluginfrom pipelex.runtime_hub import get_content_generatorfrom pipelex.system.runtime import IntegrationMode, runtime_managerfrom pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider@@ -43,13 +44,15 @@ def _test_integration_mode() -> IntegrationMode:def _fake_registrar(mocker: MockerFixture) -> PluginRegistrar:"""A registrar with empty registries (boot stores them but doesn't consume them during setup).- Storage, secrets, and pipe_func are the exceptions: boot *does* select each from the registrar- during setup. Every boot below passes an explicit ``storage_provider`` and ``secrets_provider``- (each wins ahead of the empty registry), and the built-in ``PipeFuncPlugin`` is registered here so- the default ``direct`` execution mode resolves — keeping this suite focused on hub slots rather- than provider/executor selection.+ Storage, secrets, the log sink and pipe_func are the exceptions: boot *does* select each from the+ registrar during setup. Every boot below passes an explicit ``storage_provider`` and+ ``secrets_provider`` (each wins ahead of the empty registry), and the built-in ``LogSinkPlugin`` and+ ``PipeFuncPlugin`` are registered here so the configured sink and the default ``direct`` execution+ mode resolve — keeping this suite focused on hub slots rather than provider/executor selection."""registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(temporal=SimpleNamespace(is_enabled=False))))+ registrar.begin_plugin(name="log_sinks", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION, group=None)+ LogSinkPlugin().register(registrar)registrar.begin_plugin(name="pipe_func", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION, group=None)PipeFuncPlugin().register(registrar)mocker.patch("pipelex.runtime_boot.build_registrar", return_value=registrar)diff --git a/tests/unit/pipelex/cli/test_agent_doctor_cmd.py b/tests/unit/pipelex/cli/test_agent_doctor_cmd.pyindex 3b8dce956..5a794e691 100644--- a/tests/unit/pipelex/cli/test_agent_doctor_cmd.py+++ b/tests/unit/pipelex/cli/test_agent_doctor_cmd.py@@ -3,26 +3,37 @@from __future__ import annotationsimport json-from typing import TYPE_CHECKING+import logging+from typing import TYPE_CHECKING, Anyimport pytestimport typer+from typing_extensions import overrideif TYPE_CHECKING:from pytest_mock import MockerFixture+from pipelex.cli.agent_cli.commands import doctor_cmd as agent_doctor_modulefrom pipelex.cli.agent_cli.commands.agent_cli_factory import AGENT_CLI_STDERR_LOG_FIELDSfrom pipelex.cli.agent_cli.commands.agent_output import CliOutputFormatfrom pipelex.cli.agent_cli.commands.doctor_cmd import agent_doctor_cmdfrom pipelex.cli.commands.doctor_cmd import (+ DoctorRuntimeSetup,+ LogSinkCheck,PendingMigrationsCheck,PendingMigrationsFinding,+ PluginsCheck,TelemetryConfigCheck,TelemetryConfigFinding,)from pipelex.cogt.model_backends.backend_credentials import BackendCredentialsReportfrom pipelex.core.validation import MIGRATE_COMMAND+from pipelex.system.configuration.config_loader import ConfigLoaderfrom pipelex.system.console_target import ConsoleTarget+from pipelex.tools.log.log import log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_sink import LogSink+from pipelex.tools.misc.toml_utils import load_toml_from_pathNO_PENDING_MIGRATIONS = PendingMigrationsCheck(finding=PendingMigrationsFinding.UP_TO_DATE,@@ -30,9 +41,43 @@ NO_PENDING_MIGRATIONS = PendingMigrationsCheck()+HEALTHY_LOG_SINK = LogSinkCheck(is_healthy=True, message="Log sink 'console' installed")+HEALTHY_PLUGINS = PluginsCheck(is_healthy=True, message="Plugins discovered and registered")+HEALTHY_RUNTIME_SETUP = DoctorRuntimeSetup(log_sink=HEALTHY_LOG_SINK, plugins=HEALTHY_PLUGINS)+++class _NullSink(LogSink):+ @override+ def make_handler(self) -> logging.Handler:+ return logging.NullHandler()+++def _package_log_config() -> LogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate(config_dict["runtime"]["log"])++class TestAgentDoctorCmd:"""Tests for agent_doctor_cmd JSON output."""+ def test_the_doctor_releases_the_logging_it_configured_once_the_report_is_out(self, mocker: MockerFixture) -> None:+ log.reset()+ sink = _NullSink()++ def report_through_a_sink(**_options: Any) -> None:+ log.configure(log_config=_package_log_config())+ log.install_sink(sink)++ mocker.patch.object(agent_doctor_module, "_do_agent_doctor_cmd", side_effect=report_through_a_sink)+ try:+ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ assert log.sink is None+ assert not log.is_configured+ assert sink.handler not in logging.getLogger().handlers+ finally:+ log.reset()+@pytest.fixture(autouse=True)def _mock_doctor_bootstrap(self, mocker: MockerFixture) -> None:"""Stub the runtime bootstrap so unit tests don't load real config or mutate@@ -46,7 +91,7 @@ class TestAgentDoctorCmd:out of scope for these tests — they cover the command's output shape, not theruntime bootstrap mechanics."""- mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime")+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime", return_value=HEALTHY_RUNTIME_SETUP)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.apply_agent_cli_output_discipline")mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.silence_logging_for_agent_cli")# And the migration row, for a fourth reason: unlike every other check it takes no@@ -84,6 +129,59 @@ class TestAgentDoctorCmd:assert parsed["all_healthy"] is Trueassert "recommended_actions" not in parsed+ def test_an_unregistered_log_sink_rides_the_envelope_with_its_action(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:+ """The doctor ran on the console sink; the envelope says why the report is not healthy and what to set."""+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime",+ return_value=DoctorRuntimeSetup(+ log_sink=LogSinkCheck(is_healthy=False, message="No log sink is registered for 'jsn' in [runtime.log]; registered: console, json"),+ plugins=HEALTHY_PLUGINS,+ ),+ )+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files", return_value=(True, 0, "All config files present"))+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="Telemetry configured"),+ )+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials", return_value=(True, {}, "All backends healthy"))+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_models", return_value=(True, "Models valid", {}))++ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ parsed = json.loads(capsys.readouterr().out)+ assert parsed["all_healthy"] is False+ assert parsed["checks"]["log_sink"]["healthy"] is False+ assert "'jsn'" in parsed["checks"]["log_sink"]["message"]+ assert any("[runtime.log]" in action for action in parsed["recommended_actions"])++ def test_a_plugin_registry_that_did_not_build_rides_the_envelope_with_its_action(+ self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]+ ) -> None:+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime",+ return_value=DoctorRuntimeSetup(+ log_sink=LogSinkCheck(+ is_healthy=False, message="The log sink 'json' could not be resolved because the plugin registry did not build"+ ),+ plugins=PluginsCheck(is_healthy=False, message="The plugin registry did not build: Plugin 'storage' is required by core"),+ ),+ )+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files", return_value=(True, 0, "All config files present"))+ mocker.patch(+ "pipelex.cli.agent_cli.commands.doctor_cmd.check_telemetry_config",+ return_value=TelemetryConfigCheck(finding=TelemetryConfigFinding.HEALTHY, message="Telemetry configured"),+ )+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_backend_credentials", return_value=(True, {}, "All backends healthy"))+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_models", return_value=(True, "Models valid", {}))++ agent_doctor_cmd(output_format=CliOutputFormat.JSON)++ parsed = json.loads(capsys.readouterr().out)+ assert parsed["all_healthy"] is False+ assert parsed["checks"]["plugins"]["healthy"] is False+ assert "'storage'" in parsed["checks"]["plugins"]["message"]+ assert any("plugins check" in action for action in parsed["recommended_actions"])+@pytest.mark.parametrize(("finding", "expected_action"),[@@ -352,7 +450,7 @@ class TestAgentDoctorCmd:translation. The fix short-circuits to a "skipped — fix configuration errorsfirst" model section so the full triage report still reaches stdout."""- mock_setup = mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime")+ mock_setup = mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime", return_value=HEALTHY_RUNTIME_SETUP)mock_check_models = mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_models")mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files",@@ -448,7 +546,7 @@ class TestAgentDoctorCmd:# Re-mock setup_doctor_runtime and apply_agent_cli_output_discipline so we can# capture their call args (the autouse fixture also mocks them, but we need the# fresh handles here).- mock_setup = mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime")+ mock_setup = mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime", return_value=HEALTHY_RUNTIME_SETUP)mock_discipline = mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.apply_agent_cli_output_discipline")mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.check_config_files",@@ -496,7 +594,7 @@ class TestAgentDoctorCmd:call_order.append("check_models")return True, "OK", {}- mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime")+ mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.setup_doctor_runtime", return_value=HEALTHY_RUNTIME_SETUP)mocker.patch("pipelex.cli.agent_cli.commands.doctor_cmd.apply_agent_cli_output_discipline",side_effect=record_discipline,diff --git a/tests/unit/pipelex/cli/test_doctor_cmd.py b/tests/unit/pipelex/cli/test_doctor_cmd.pyindex bfaf70d2c..fa487aa4a 100644--- a/tests/unit/pipelex/cli/test_doctor_cmd.py+++ b/tests/unit/pipelex/cli/test_doctor_cmd.py@@ -12,8 +12,11 @@ if TYPE_CHECKING:from pytest_mock import MockerFixturefrom pipelex.cli.commands.doctor_cmd import (+ DoctorRuntimeSetup,+ LogSinkCheck,PendingMigrationsCheck,PendingMigrationsFinding,+ PluginsCheck,TelemetryConfigCheck,TelemetryConfigFinding,check_backend_credentials,@@ -36,6 +39,11 @@ NO_PENDING_MIGRATIONS = PendingMigrationsCheck()+HEALTHY_LOG_SINK = LogSinkCheck(is_healthy=True, message="Log sink 'console' installed")+HEALTHY_PLUGINS = PluginsCheck(is_healthy=True, message="Plugins discovered and registered")+HEALTHY_RUNTIME_SETUP = DoctorRuntimeSetup(log_sink=HEALTHY_LOG_SINK, plugins=HEALTHY_PLUGINS)++class TestDoctorLayeredResolution:"""Verify that doctor checks use layered config resolution:project .pipelex/ first, fall back to global ~/.pipelex/.@@ -52,7 +60,7 @@ class TestDoctorLayeredResolution:"""# Stub the runtime bootstrap so the test doesn't load real config or call log.configure# (once-per-process). The bootstrap call itself is exercised separately.- mock_setup = mocker.patch("pipelex.cli.commands.doctor_cmd.setup_doctor_runtime")+ mock_setup = mocker.patch("pipelex.cli.commands.doctor_cmd.setup_doctor_runtime", return_value=HEALTHY_RUNTIME_SETUP)mock_check_config = mocker.patch("pipelex.cli.commands.doctor_cmd.check_config_files",return_value=(True, 0, "OK"),diff --git a/tests/unit/pipelex/cli/test_doctor_display_report.py b/tests/unit/pipelex/cli/test_doctor_display_report.pyindex 4f373f1d9..79ddddb08 100644--- a/tests/unit/pipelex/cli/test_doctor_display_report.py+++ b/tests/unit/pipelex/cli/test_doctor_display_report.py@@ -10,8 +10,10 @@ from rich.console import Consolefrom pipelex.cli.commands.doctor_cmd import (BackendFileReport,ConfigLocationInfo,+ LogSinkCheck,PendingMigrationsCheck,PendingMigrationsFinding,+ PluginsCheck,TelemetryConfigCheck,TelemetryConfigFinding,display_health_report,@@ -95,6 +97,41 @@ class TestDoctorDisplayReport:assert "Deck is up to date with pipelex 1.2.0" in outputassert "Possible Solutions" not in output+ def test_an_unregistered_log_sink_is_a_row_of_its_own_and_flags_the_report(self, console: Console) -> None:+ """The report still renders, on the console sink, and says which token was set and which sinks exist."""+ kwargs = _healthy_report_kwargs()+ kwargs["log_sink_check"] = LogSinkCheck(is_healthy=False, message="No log sink is registered for 'jsn'; registered: console, json")++ display_health_report(**kwargs)++ output = console.export_text()+ assert "Overall Status: ⚠️ Issues Found" in output+ assert "Log Sink" in output+ assert "No log sink is registered for 'jsn'" in output+ assert "Set sink in [runtime.log]" in output++ def test_without_a_runtime_setup_there_is_no_log_sink_row(self, console: Console) -> None:+ display_health_report(**_healthy_report_kwargs())++ assert "Log Sink" not in console.export_text()++ def test_a_plugin_registry_that_did_not_build_is_a_row_of_its_own_and_flags_the_report(self, console: Console) -> None:+ kwargs = _healthy_report_kwargs()+ kwargs["plugins_check"] = PluginsCheck(is_healthy=False, message="The plugin registry did not build: Plugin 'storage' is required by core")++ display_health_report(**kwargs)++ output = console.export_text()+ assert "Overall Status: ⚠️ Issues Found" in output+ assert "Plugins" in output+ assert "Plugin 'storage' is required by core" in output+ assert "plugin named in the Plugins row" in output++ def test_without_a_plugin_discovery_there_is_no_plugins_row(self, console: Console) -> None:+ display_health_report(**_healthy_report_kwargs())++ assert "Plugins" not in console.export_text()+def test_global_location_rendered(self, console: Console) -> None:"""Without a project .pipelex/, the global location line is rendered."""kwargs = _healthy_report_kwargs()diff --git a/tests/unit/pipelex/cli/test_doctor_log_sink.py b/tests/unit/pipelex/cli/test_doctor_log_sink.pynew file mode 100644index 000000000..ad0080b7d--- /dev/null+++ b/tests/unit/pipelex/cli/test_doctor_log_sink.py@@ -0,0 +1,188 @@+"""The doctor installs the configured log sink, or the console sink on stderr with a finding when it cannot.++Boot stops on an unregistered token, on a sink that fails to install and on a plugin registry that does+not build. The doctor exists to diagnose exactly that kind of misconfiguration, so it must not die on+where its own lines go: the rows say what was set and what stopped it. Once the report is out, the+doctor releases the logging it configured, and only that.+"""++from __future__ import annotations++import logging+import sys+from typing import TYPE_CHECKING, Any++import pytest+from rich.logging import RichHandler+from typing_extensions import override++from pipelex.cli.commands import doctor_cmd+from pipelex.cli.commands.doctor_cmd import FALLBACK_LOG_SINK_NOTE, discover_plugins_and_install_doctor_log_sink, install_doctor_log_sink+from pipelex.plugins.exceptions import CoreUnconditionalPluginDisabledError+from pipelex.plugins.log_sink_registry import LogSinkRegistry+from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.system.console_target import ConsoleTarget+from pipelex.tools.log.console_log_sink import ConsoleLogSink+from pipelex.tools.log.log import log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_sink import LogSink, LogSinkMethod+from pipelex.tools.misc.toml_utils import load_toml_from_path++if TYPE_CHECKING:+ from collections.abc import Iterator++ from pytest_mock import MockerFixture+++def _log_config(*, sink: str, console_log_target: ConsoleTarget = ConsoleTarget.STDERR) -> LogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate({**config_dict["runtime"]["log"], "sink": sink, "console_log_target": console_log_target})+++def _assert_the_fallback_console_sink_is_installed_on_stderr() -> None:+ sink = log.sink+ assert isinstance(sink, ConsoleLogSink)+ handler = sink.handler+ assert isinstance(handler, RichHandler)+ assert handler.console.file is sys.stderr+++class _NamedSink(LogSink):+ def __init__(self, *, name: str) -> None:+ super().__init__()+ self.name = name++ @override+ def make_handler(self) -> logging.Handler:+ raise NotImplementedError+++class _NullSink(LogSink):+ @override+ def make_handler(self) -> logging.Handler:+ return logging.NullHandler()+++def _registry() -> LogSinkRegistry:+ return LogSinkRegistry(+ {+ LogSinkMethod.CONSOLE: lambda _config: _NamedSink(name=LogSinkMethod.CONSOLE),+ LogSinkMethod.JSON: lambda _config: _NamedSink(name=LogSinkMethod.JSON),+ }+ )+++class TestDoctorLogSink:+ @pytest.fixture+ def released_log(self) -> Iterator[None]:+ """The process-wide facade, released before and after, since the doctor command works on it and not on a fresh instance."""+ log.reset()+ try:+ yield+ finally:+ log.reset()++ @pytest.mark.usefixtures("released_log")+ def test_the_doctor_releases_the_logging_it_configured_once_the_report_is_out(self, mocker: MockerFixture) -> None:+ sink = _NullSink()++ def report_through_a_sink(**_options: Any) -> None:+ log.configure(log_config=_log_config(sink=LogSinkMethod.CONSOLE))+ log.install_sink(sink)+ assert log.sink is sink++ mocker.patch.object(doctor_cmd, "do_doctor_cmd", side_effect=report_through_a_sink)++ doctor_cmd.doctor_cmd(fix=False)++ assert log.sink is None+ assert not log.is_configured+ assert sink.handler not in logging.getLogger().handlers++ @pytest.mark.usefixtures("released_log")+ def test_the_doctor_leaves_logging_an_embedder_configured_before_calling_in(self, mocker: MockerFixture) -> None:+ sink = _NullSink()+ log.configure(log_config=_log_config(sink=LogSinkMethod.CONSOLE))+ log.install_sink(sink)+ mocker.patch.object(doctor_cmd, "do_doctor_cmd")++ doctor_cmd.doctor_cmd(fix=False)++ assert log.sink is sink+ assert log.is_configured++ @pytest.mark.usefixtures("released_log")+ def test_a_registered_token_installs_that_sink_and_the_row_is_healthy(self, mocker: MockerFixture) -> None:+ install_sink = mocker.patch.object(doctor_cmd.log, "install_sink")++ check = install_doctor_log_sink(registry=_registry(), log_config=_log_config(sink=LogSinkMethod.JSON))++ assert check.is_healthy+ assert LogSinkMethod.JSON in check.message+ (installed,) = install_sink.call_args.args+ assert isinstance(installed, _NamedSink)+ assert installed.name == LogSinkMethod.JSON++ @pytest.mark.usefixtures("released_log")+ def test_an_unregistered_token_installs_the_console_sink_and_names_the_registered_ones(self, mocker: MockerFixture) -> None:+ install_sink = mocker.patch.object(doctor_cmd.log, "install_sink")++ check = install_doctor_log_sink(registry=_registry(), log_config=_log_config(sink="jsn"))++ assert not check.is_healthy+ assert "'jsn'" in check.message+ assert f"{LogSinkMethod.CONSOLE}, {LogSinkMethod.JSON}" in check.message+ (installed,) = install_sink.call_args.args+ assert isinstance(installed, ConsoleLogSink)++ @pytest.mark.usefixtures("released_log")+ def test_a_sink_that_fails_to_install_on_this_config_is_a_row_and_the_report_goes_on_through_stderr(self) -> None:+ """The shipped default, the console sink, on a console target no sink writes to: the fallback cannot read the same field."""+ log_config = _log_config(sink=LogSinkMethod.CONSOLE, console_log_target=ConsoleTarget.FILE)+ log.configure(log_config=log_config)+ registry = LogSinkRegistry(+ {LogSinkMethod.CONSOLE: lambda config: ConsoleLogSink(rich_log_config=config.rich_log, target=config.console_log_target)}+ )++ check = install_doctor_log_sink(registry=registry, log_config=log_config)++ assert not check.is_healthy+ assert "could not be installed" in check.message+ assert "choose stdout or stderr" in check.message+ _assert_the_fallback_console_sink_is_installed_on_stderr()++ @pytest.mark.usefixtures("released_log")+ def test_a_plugin_registry_that_does_not_build_is_a_row_and_the_report_goes_on_through_stderr(self, mocker: MockerFixture) -> None:+ log_config = _log_config(sink=LogSinkMethod.JSON)+ log.configure(log_config=log_config)+ mocker.patch.object(doctor_cmd, "get_config")+ mocker.patch.object(doctor_cmd, "build_registrar", side_effect=CoreUnconditionalPluginDisabledError(plugin_name="storage"))++ runtime_setup = discover_plugins_and_install_doctor_log_sink(log_config=log_config)++ assert runtime_setup.plugins is not None+ assert not runtime_setup.plugins.is_healthy+ assert "'storage'" in runtime_setup.plugins.message+ assert not runtime_setup.log_sink.is_healthy+ assert "registry did not build" in runtime_setup.log_sink.message+ _assert_the_fallback_console_sink_is_installed_on_stderr()++ @pytest.mark.usefixtures("released_log")+ def test_a_fallback_refused_because_a_sink_is_already_recorded_says_so_and_keeps_that_sink(self) -> None:+ """``install_sink`` records the sink before it replays what the holding handler held, deliberately, so that a replay+ which raises still leaves the sink findable by ``reset``. The fallback therefore cannot assume that a failed+ installation left nothing behind: installing on top would raise in place of the failure it was called to report, and+ a row promising the console fallback would name a sink that never stood in.+ """+ log_config = _log_config(sink=LogSinkMethod.JSON)+ log.configure(log_config=log_config)+ already_recorded = _NullSink()+ log.install_sink(already_recorded)++ check = install_doctor_log_sink(registry=None, log_config=log_config)++ assert not check.is_healthy+ assert "registry did not build" in check.message+ assert "already recorded" in check.message+ assert FALLBACK_LOG_SINK_NOTE not in check.message+ assert log.sink is already_recordeddiff --git a/tests/unit/pipelex/cli/test_format_doctor_markdown.py b/tests/unit/pipelex/cli/test_format_doctor_markdown.pyindex e4ada1877..15ca169b5 100644--- a/tests/unit/pipelex/cli/test_format_doctor_markdown.py+++ b/tests/unit/pipelex/cli/test_format_doctor_markdown.py@@ -86,6 +86,26 @@ class TestFormatDoctorMarkdown:output = _format_doctor_markdown(self._HEALTHY_RESULT)assert "/home/user/.pipelex (global)" in output+ def test_log_sink_section_renders_only_when_the_check_ran(self) -> None:+ with_sink: dict[str, Any] = {+ **self._HEALTHY_RESULT,+ "checks": {**self._HEALTHY_RESULT["checks"], "log_sink": {"healthy": False, "message": "No log sink is registered for 'jsn'"}},+ }++ assert "## Log Sink" in _format_doctor_markdown(with_sink)+ assert "No log sink is registered for 'jsn'" in _format_doctor_markdown(with_sink)+ assert "## Log Sink" not in _format_doctor_markdown(self._HEALTHY_RESULT)++ def test_plugins_section_renders_only_when_the_discovery_ran(self) -> None:+ with_plugins: dict[str, Any] = {+ **self._HEALTHY_RESULT,+ "checks": {**self._HEALTHY_RESULT["checks"], "plugins": {"healthy": False, "message": "The plugin registry did not build"}},+ }++ assert "## Plugins" in _format_doctor_markdown(with_plugins)+ assert "The plugin registry did not build" in _format_doctor_markdown(with_plugins)+ assert "## Plugins" not in _format_doctor_markdown(self._HEALTHY_RESULT)+def test_healthy_output_has_no_recommended_actions(self) -> None:"""Healthy result should not include a Recommended Actions section."""output = _format_doctor_markdown(self._HEALTHY_RESULT)diff --git a/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py b/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.pyindex 5ecfadd7c..88bca9908 100644--- a/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py+++ b/tests/unit/pipelex/cogt/model_backends/test_gateway_unknown_defaults.py@@ -10,8 +10,11 @@ see `test_gateway_unknown_per_model_keys.py`. A local backend file gets the strirules — see `test_backend_library_leniency.py`."""+import loggingfrom typing import Any, cast+import pytest+from pipelex.cogt.model_backends.gateway_config import drop_unknown_gateway_defaultsfrom pipelex.cogt.model_backends.model_spec_factory import BackendModelSpecs, InferenceModelSpecBlueprint@@ -47,6 +50,25 @@ class TestGatewayUnknownDefaults:assert drop_unknown_gateway_defaults(gateway_model_specs=specs) is specs+ def test_pruning_emits_no_record(self, caplog: pytest.LogCaptureFixture) -> None:+ """`drop_unknown_gateway_defaults` is documented as deliberately silent, and this is what holds it to that.++ It runs on the success path of every gateway-backend load, including loads that precede+ `runtime_hub.set_config()`. A log call here would turn a plain data transform into a boot-order+ dependency: before the dispatch is configured a line takes the stdlib's default handling, so a+ warning about the served payload would read differently depending on when the load happened to run.+ The guard this replaces asserted that an unconfigured dispatch raised, which is no longer true — a+ log call before `configure` is now handled rather than fatal — so nothing fails loudly of its own+ accord if a line appears here. Asserting the silence directly is what survives that change.+ """+ specs = self._specs(defaults={"max_tokens": 4096, "a_field_we_removed": "anthropic"})++ with caplog.at_level(logging.NOTSET):+ pruned = drop_unknown_gateway_defaults(gateway_model_specs=specs)++ assert pruned["defaults"] == {"max_tokens": 4096}+ assert caplog.records == []+def test_per_model_unknown_keys_are_untouched(self) -> None:"""This function is scoped to `defaults`; the per-model rule is the loader's, tested in `test_gateway_unknown_per_model_keys.py`."""# The unknown key in `defaults` is what forces the deep-copy prune to run at all: without itdiff --git a/tests/unit/pipelex/migration/test_log_mode_retirement.py b/tests/unit/pipelex/migration/test_log_mode_retirement.pynew file mode 100644index 000000000..04e7d875c--- /dev/null+++ b/tests/unit/pipelex/migration/test_log_mode_retirement.py@@ -0,0 +1,92 @@+"""The entry that retires the rich-or-poor log mode, exercised on the files it is about.++It deletes the four retired keys and is `safe`, so a full copy of an older base file, which carries every+one of them at its old default, migrates in one run. What it cannot do is written where a person reads+it: a file that had set `log_mode = "poor"` or `is_console_logging_enabled = false` chose a behaviour the+deletion undoes, no operation in the vocabulary writes the replacement, and the title names it. The recipe+it names for silence is checked against the code, since the base pins the `pipelex` logger and the root+level alone would silence none of Pipelex's records.+"""++import logging+from copy import deepcopy+from typing import Any++from pipelex.migration.engine import replay_ledger_over_text+from pipelex.migration.ledger import load_ledger, packaged_migration_dir+from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.system.configuration.config_surface import PIPELEX_CONFIG_SURFACE_ID+from pipelex.tools.log.log import Log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.misc.json_utils import deep_update+from pipelex.tools.misc.toml_utils import load_toml_from_content, load_toml_from_path++ENTRY_ID = "pipelex-config@5"++FILE_IN_POOR_MODE = """\+[runtime.log]+default_log_level = "INFO"+log_mode = "poor"+is_console_logging_enabled = false+poor_loggers = ["pipelex"]+generic_poor_logger = "#poor-log"+"""++FILE_AT_THE_CURRENT_SHAPE = """\+[runtime.log]+default_log_level = "INFO"+sink = "json"+"""+++class TestTheLogModeRetirement:+ def test_the_retired_keys_are_deleted_and_the_rest_of_the_section_is_kept(self) -> None:+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=PIPELEX_CONFIG_SURFACE_ID)+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_IN_POOR_MODE)++ assert replay.blocked == []+ assert [step.entry_id for step in replay.steps] == [ENTRY_ID]+ migrated: dict[str, Any] = load_toml_from_content(replay.text)+ assert migrated == {"runtime": {"log": {"default_log_level": "INFO"}}}++ def test_the_step_a_person_sees_names_what_a_poor_mode_file_sets_by_hand(self) -> None:+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=PIPELEX_CONFIG_SURFACE_ID)+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_IN_POOR_MODE)++ (step,) = replay.steps+ assert 'sink = "json"' in step.title+ assert 'pipelex = "OFF"' in step.title+ assert 'sink = "json"' in step.description+ assert 'pipelex = "OFF"' in step.description++ def test_the_silencing_recipe_the_entry_names_silences_every_pipelex_logger(self) -> None:+ """``default_log_level`` governs only what the package levels do not pin, and the base pins ``pipelex``: the recipe is the package level."""+ base: dict[str, Any] = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ merged = deepcopy(base["runtime"]["log"])+ deep_update(merged, updates={"package_log_levels": {"pipelex": "OFF"}})+ fresh = Log()+ fresh.configure(log_config=LogConfig.model_validate(merged))+ try:+ assert not logging.getLogger("pipelex").isEnabledFor(logging.CRITICAL)+ assert not logging.getLogger("pipelex.pipe_operators.pipe_llm").isEnabledFor(logging.CRITICAL)+ assert logging.getLogger("httpx").isEnabledFor(logging.WARNING)+ finally:+ fresh.reset()++ def test_the_migrated_file_is_accepted_by_the_current_model_and_lands_on_the_console_sink(self) -> None:+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=PIPELEX_CONFIG_SURFACE_ID)+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_IN_POOR_MODE)+ migrated: dict[str, Any] = load_toml_from_content(replay.text)+ base: dict[str, Any] = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")++ log_config = LogConfig.model_validate({**base["runtime"]["log"], **migrated["runtime"]["log"]})++ assert log_config.sink == "console"++ def test_a_file_at_the_current_shape_hears_nothing(self) -> None:+ ledger = load_ledger(migration_dir=packaged_migration_dir(), surface_id=PIPELEX_CONFIG_SURFACE_ID)+ replay = replay_ledger_over_text(ledger=ledger, text=FILE_AT_THE_CURRENT_SHAPE)++ assert replay.blocked == []+ assert replay.steps == []+ assert replay.text is FILE_AT_THE_CURRENT_SHAPEdiff --git a/tests/unit/pipelex/plugins/test_import_light_boot.py b/tests/unit/pipelex/plugins/test_import_light_boot.pyindex 222fb8df2..f6c6846eb 100644--- a/tests/unit/pipelex/plugins/test_import_light_boot.py+++ b/tests/unit/pipelex/plugins/test_import_light_boot.py@@ -33,6 +33,10 @@ _GUARD_SCRIPT = textwrap.dedent(# discovery/registration in core must never pull one in."fastapi","starlette",+ # Nor the OpenTelemetry SDK and its exporters: the built-in ``otlp`` log sink defers them into+ # its factory, so registering the sinks must not load them.+ "opentelemetry.sdk",+ "opentelemetry.exporter",)class _Blocker(importlib.abc.MetaPathFinder):@@ -73,6 +77,9 @@ _GUARD_SCRIPT = textwrap.dedent(# google-cloud-storage: the s3/gcp SDK guards live inside the providers' I/O methods, so# registration stays import-light even though those SDKs are BLOCKED above.assert registrar.storage_providers, "expected the built-in storage providers to be registered import-light"+ # The built-in LogSinkPlugin registers the three sinks without importing the OpenTelemetry SDK:+ # the ``otlp`` factory imports it when the sink is selected, never at register.+ assert registrar.log_sinks, "expected the built-in log sinks to be registered import-light"print("import-light OK")""")diff --git a/tests/unit/pipelex/plugins/test_log_sink_registry.py b/tests/unit/pipelex/plugins/test_log_sink_registry.pynew file mode 100644index 000000000..7216b7c6b--- /dev/null+++ b/tests/unit/pipelex/plugins/test_log_sink_registry.py@@ -0,0 +1,129 @@+"""The log-sink seam: the registrar collects per-method sink factories into a registry.++Pins the seam independent of the real built-in sinks: a factory registered for a method is the one the+registry hands back; a miss raises ``UnknownLogSinkError`` listing the registered methods; a second+registration for the same method fails loud naming both plugins; an empty registry misses every+method; and the built-in plugin contributes its three sinks under the rows ``pipelex plugins list`` shows.+"""++from __future__ import annotations++import logging+from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest+from typing_extensions import override++from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.discovery import build_registrar+from pipelex.plugins.exceptions import DuplicateLogSinkError, UnknownLogSinkError+from pipelex.plugins.log_sink_registry import LogSinkRegistry+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar+from pipelex.providers.builtins import KERNEL_BUILTIN_PLUGINS, KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES, KERNEL_ENTRY_POINT_GROUPS+from pipelex.tools.log.log_sink import LogSink, LogSinkMethod++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++ from pipelex.system.configuration.configs import PipelexConfig+ from pipelex.tools.log.log_config import LogConfig+++def _fake_config() -> PipelexConfig:+ return cast("PipelexConfig", SimpleNamespace(runtime=SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+++def _make_registrar() -> PluginRegistrar:+ return PluginRegistrar(config=_fake_config())+++class _NullLogSink(LogSink):+ @override+ def make_handler(self) -> logging.Handler:+ return logging.NullHandler()+++def _fake_factory(_config: LogConfig) -> LogSink:+ """Stand-in factory: identity is all the registry tests assert (never actually invoked here)."""+ return _NullLogSink()+++class TestLogSinkRegistry:+ def test_registered_factory_is_retrievable_by_method(self) -> None:+ """A factory registered for a method is the exact callable the built registry returns for it."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=None)+ registrar.add_log_sink(method="gcp", factory=_fake_factory)++ registry = LogSinkRegistry(registrar.log_sinks)+ assert registry.get_required(method="gcp") is _fake_factory+ assert registry.has(method="gcp")+ assert registry.methods == ["gcp"]++ def test_contribution_recorded_on_the_active_plugin(self) -> None:+ """Registering a sink records a ``log sink <method>`` contribution on the plugin's discovery."""+ registrar = _make_registrar()+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=None)++ registrar.add_log_sink(method="gcp", factory=_fake_factory)++ assert "log sink gcp" in discovery.contributions++ def test_get_required_miss_raises_listing_registered_methods(self) -> None:+ """A miss names the requested method and lists the registered ones (the boot-time actionable error)."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=None)+ registrar.add_log_sink(method="json", factory=_fake_factory)+ registrar.add_log_sink(method="console", factory=_fake_factory)+ registry = LogSinkRegistry(registrar.log_sinks)++ with pytest.raises(UnknownLogSinkError) as exc_info:+ registry.get_required(method="gcp")++ assert exc_info.value.method == "gcp"+ assert set(exc_info.value.registered_methods) == {"json", "console"}+ message = str(exc_info.value)+ assert "json" in message+ assert "console" in message+ assert "runtime.log.sink" in message++ def test_empty_registry_misses_every_method(self) -> None:+ """A registry with no factories misses every method — soft via has, loud via get_required."""+ registry = LogSinkRegistry({})++ assert not registry.has(method="json")+ assert registry.methods == []+ with pytest.raises(UnknownLogSinkError):+ registry.get_required(method="json")++ def test_duplicate_method_fails_loud_naming_both_plugins(self) -> None:+ """Two plugins registering a sink for the same method is a fail-loud conflict naming both."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=None)+ registrar.add_log_sink(method="gcp", factory=_fake_factory)+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=None)++ with pytest.raises(DuplicateLogSinkError) as exc_info:+ registrar.add_log_sink(method="gcp", factory=_fake_factory)++ assert exc_info.value.first_plugin == "alpha"+ assert exc_info.value.second_plugin == "beta"+ assert exc_info.value.method == "gcp"++ def test_the_builtin_plugin_contributes_the_three_sinks(self, mocker: MockerFixture) -> None:+ """The built-in ``log_sinks`` plugin is discovered with one contribution per shipped sink, the rows ``pipelex plugins list`` shows."""+ mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[])+ registrar = build_registrar(+ config=_fake_config(),+ boot_orchestrator=None,+ builtin_plugins=KERNEL_BUILTIN_PLUGINS,+ core_unconditional_plugin_names=KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES,+ entry_point_groups=KERNEL_ENTRY_POINT_GROUPS,+ )++ by_name = {discovery.name: discovery for discovery in registrar.discoveries}+ contributions = by_name["log_sinks"].contributions+ for method in LogSinkMethod:+ assert f"log sink {method}" in contributions+ assert set(LogSinkRegistry(registrar.log_sinks).methods) == {method.value for method in LogSinkMethod}diff --git a/tests/unit/pipelex/plugins/test_plugin_discovery.py b/tests/unit/pipelex/plugins/test_plugin_discovery.pyindex 83a839466..f50021d51 100644--- a/tests/unit/pipelex/plugins/test_plugin_discovery.py+++ b/tests/unit/pipelex/plugins/test_plugin_discovery.py@@ -251,9 +251,9 @@ class TestPluginDiscovery:disabled_discovery = next(discovery for discovery in registrar.discoveries if discovery.name == "optional")assert disabled_discovery.status == PluginStatus.DISABLED- @pytest.mark.parametrize("plugin_name", ["direct", "pipe_func", "storage", "secrets", "openai"])+ @pytest.mark.parametrize("plugin_name", ["direct", "pipe_func", "log_sinks", "storage", "secrets", "openai"])def test_disabling_core_unconditional_plugin_raises(self, plugin_name: str) -> None:- """Denylisting any plugin core requires unconditionally (pipe_func/storage/secrets included) is a startup error."""+ """Denylisting any plugin core requires unconditionally (pipe_func/log_sinks/storage/secrets included) is a startup error."""from pipelex.interpreter_plugins.builtins import BUILTIN_PLUGINS, CORE_UNCONDITIONAL_PLUGIN_NAMES # ruff: ignore[import-outside-top-level]with pytest.raises(CoreUnconditionalPluginDisabledError) as exc_info:diff --git a/tests/unit/pipelex/tools/test_log_before_configure.py b/tests/unit/pipelex/tools/test_log_before_configure.pyindex ec5495828..53d6081f5 100644--- a/tests/unit/pipelex/tools/test_log_before_configure.py+++ b/tests/unit/pipelex/tools/test_log_before_configure.py@@ -43,9 +43,12 @@ class TestLogBeforeConfigure:fresh.info({"structured": "content"}, title="Data")assert logging.getLogger().handlers == root_handlers_before- messages = [record.getMessage() for record in _own_records(caplog)]+ records = _own_records(caplog)+ messages = [record.getMessage() for record in records]assert messages[:5] == ["verbose", "debug", "dev", "info", "warning"]- assert messages[5].startswith("error\nTraceback")+ assert messages[5] == "error"+ assert records[5].exc_info is not None+ assert records[5].exc_info[0] is ValueErrorassert messages[6] == "critical"assert messages[7].startswith("Data:")assert all(record.name == __name__ for record in _own_records(caplog))diff --git a/tests/unit/pipelex/tools/test_log_console_sink.py b/tests/unit/pipelex/tools/test_log_console_sink.pynew file mode 100644index 000000000..9bc16f6e0--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_console_sink.py@@ -0,0 +1,190 @@+"""The ``console`` sink renders byte for byte what the Rich handler rendered before it existed.++The reference is the handler ``log.configure`` used to build inline: a ``RichHandler`` fed every+``[runtime.log.rich_log]`` setting and the emoji formatter. Both handlers are pointed at the same kind+of non-terminal console and handed the same fixed record set, and their output must be identical.+"""++from __future__ import annotations++import io+import logging+from typing import Any++import pytest+from rich.console import Console+from rich.highlighter import Highlighter, JSONHighlighter, ReprHighlighter+from rich.logging import RichHandler++from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.system.console_target import ConsoleTarget+from pipelex.tools.log.console_log_sink import ConsoleLogSink+from pipelex.tools.log.log_config import HighlighterName, LogConfig, RichLogConfig+from pipelex.tools.log.log_fields import VERBATIM_MARK, attach_log_record_extra+from pipelex.tools.log.log_formatter import EmojiLogFormatter+from pipelex.tools.misc.toml_utils import load_toml_from_path++CONSOLE_WIDTH = 100+++def _package_rich_log_config() -> RichLogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate(config_dict["runtime"]["log"]).rich_log+++def _reference_handler(*, config: RichLogConfig) -> logging.Handler:+ """The handler ``log.configure`` built before the sink seam, setting for setting."""+ highlighter: Highlighter+ match config.highlighter_name:+ case HighlighterName.JSON:+ highlighter = JSONHighlighter()+ case HighlighterName.REPR:+ highlighter = ReprHighlighter()+ handler = RichHandler(+ console=Console(file=io.StringIO()),+ show_time=config.is_show_time,+ show_level=config.is_show_level,+ enable_link_path=config.is_link_path_enabled,+ highlighter=highlighter,+ markup=config.is_markup_enabled,+ rich_tracebacks=config.is_rich_tracebacks,+ tracebacks_word_wrap=config.is_tracebacks_word_wrap,+ tracebacks_show_locals=config.is_tracebacks_show_locals,+ tracebacks_suppress=config.tracebacks_suppress,+ keywords=config.keywords_to_hilight,+ )+ handler.setFormatter(EmojiLogFormatter())+ return handler+++def _fixed_record_set() -> list[logging.LogRecord]:+ """Every shape a record takes: a bare line, a warning with fields, structured content, an error with a traceback, a foreign logger."""+ records: list[logging.LogRecord] = []++ def record(*, name: str, level: int, message: str, exc_info: Any = None, extra: dict[str, Any] | None = None) -> None:+ built = logging.LogRecord(name=name, level=level, pathname="/repo/pipelex/module.py", lineno=42, msg=message, args=(), exc_info=exc_info)+ built.created = 1_700_000_000.0+ built.msecs = 0.0+ if extra:+ attach_log_record_extra(record=built, extra=extra)+ records.append(built)++ record(name="pipelex.pipe_operators.pipe_llm", level=logging.INFO, message="Running the pipe")+ record(name="pipelex.pipe_operators.pipe_llm", level=logging.WARNING, message="Slow backend", extra={"files": 7, "request_id": "r1"})+ record(name="pipelex.pipeline.pipe_run", level=logging.INFO, message='Config:\n{\n "key": "value"\n}', extra={"data": {"key": "value"}})+ try:+ msg = "boom"+ raise ValueError(msg)+ except ValueError as exc:+ record(name="pipelex.pipeline.pipe_run", level=logging.ERROR, message="Failed", exc_info=(type(exc), exc, exc.__traceback__))+ record(name="openai._base_client", level=logging.INFO, message="Retrying request")+ record(name="myapp.jobs.nightly", level=logging.INFO, message="A logger with no emoji")+ return records+++def _render_one(*, config: RichLogConfig, message: str, extra: dict[str, Any] | None) -> str:+ """One record through a fresh console sink, so a per-record attribute can be read off the rendering."""+ buffer = io.StringIO()+ handler = ConsoleLogSink(rich_log_config=config, target=ConsoleTarget.STDERR).handler+ assert isinstance(handler, RichHandler)+ handler.console = Console(file=buffer, width=CONSOLE_WIDTH, force_terminal=False, color_system=None, legacy_windows=False)+ record = logging.LogRecord(+ name="pipelex.system.exceptions", level=logging.ERROR, pathname="/repo/pipelex/module.py", lineno=42, msg=message, args=(), exc_info=None+ )+ for name, value in (extra or {}).items():+ setattr(record, name, value)+ handler.handle(record)+ return buffer.getvalue()+++def _render(handler: logging.Handler) -> str:+ buffer = io.StringIO()+ rich_handler = handler+ assert isinstance(rich_handler, RichHandler)+ rich_handler.console = Console(file=buffer, width=CONSOLE_WIDTH, force_terminal=False, color_system=None, legacy_windows=False)+ for record in _fixed_record_set():+ handler.handle(record)+ return buffer.getvalue()+++class TestConsoleLogSink:+ def test_output_is_byte_identical_to_the_handler_configure_used_to_build(self) -> None:+ config = _package_rich_log_config()+ reference = _render(_reference_handler(config=config))+ sink = ConsoleLogSink(rich_log_config=config, target=ConsoleTarget.STDERR)++ rendered = _render(sink.handler)++ assert rendered+ assert "🧠: Running the pipe" in rendered+ assert "ValueError" in rendered+ assert rendered == reference++ def test_the_prefix_stays_on_a_line_that_carries_a_traceback(self) -> None:+ """The Rich handler renders such a line from ``formatMessage`` alone, and the emoji must survive that path too."""+ config = _package_rich_log_config()+ assert config.is_rich_tracebacks, "the shipped default is what the regression rode on"+ sink = ConsoleLogSink(rich_log_config=config, target=ConsoleTarget.STDERR)++ rendered = _render(sink.handler)++ assert "🧠: Failed" in rendered+ assert "ValueError: boom" in rendered++ @pytest.mark.parametrize(+ "message",+ [+ "expected list[int], got str",+ "no such file: [/etc/pipelex.toml]",+ ],+ ids=["a type complaint", "a bracketed path"],+ )+ def test_a_message_shaped_like_markup_renders_intact_only_when_the_record_asks_for_verbatim(self, message: str) -> None:+ """An error message carries text nobody chose, and Rich reads a tag-shaped span in it as markup.++ Rich's tag has to start with a lowercase letter, `#`, `/` or `@`, so `[Errno 2]` is safe and the+ two shapes this codebase's messages are actually made of are not: `list[int]` loses the bracketed+ span, and a bracketed path opens what Rich reads as a closing tag and raises inside the handler,+ where `handleError` costs the whole line. The configuration asks for markup and Pipelex's own+ lines use it, so the opt-out is per record: Rich reads `markup` off the record ahead of its+ handler's setting, which is what `TracebackMessageError` stamps through `VERBATIM_MARK`.+ """+ config = _package_rich_log_config()+ assert config.is_markup_enabled, "the mark is an opt-out, so an enabled setting is what it opts out of"++ interpreted = _render_one(config=config, message=message, extra=None)+ verbatim = _render_one(config=config, message=message, extra={VERBATIM_MARK: False})++ assert message not in interpreted+ assert message in verbatim++ @pytest.mark.parametrize(+ "message",+ [+ "expected list[int], got str",+ "no such file: [/etc/pipelex.toml]",+ ],+ ids=["a type complaint", "a bracketed path"],+ )+ def test_a_message_rich_refuses_costs_its_rendering_and_never_the_log_call(self, message: str) -> None:+ """Rich overrides ``emit`` without the stdlib's ``handleError`` guard, so a refusal left the log call.++ A bracketed path reads as a closing tag with nothing open and Rich raises ``MarkupError`` from+ inside the handler: unguarded, that propagated out of `log.error` and replaced the failure being+ reported with itself. The sink's handler restores the guard, so the worst a line Rich refuses costs+ is its own rendering.+ """+ config = _package_rich_log_config()++ rendered = _render_one(config=config, message=message, extra=None)++ assert message not in rendered++ def test_the_handler_is_a_rich_handler_with_the_emoji_formatter_and_the_same_object_on_every_read(self) -> None:+ sink = ConsoleLogSink(rich_log_config=_package_rich_log_config(), target=ConsoleTarget.STDERR)++ handler = sink.handler++ assert isinstance(handler, RichHandler)+ assert isinstance(handler.formatter, EmojiLogFormatter)+ assert sink.handler is handlerdiff --git a/tests/unit/pipelex/tools/test_log_console_sink_without_rich.py b/tests/unit/pipelex/tools/test_log_console_sink_without_rich.pynew file mode 100644index 000000000..49c3562e0--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_console_sink_without_rich.py@@ -0,0 +1,75 @@+"""Selecting the ``console`` sink without Rich installed fails at boot naming the extra and the ``json`` alternative.++Rich is a hard dependency today and the runtime's own modules import it, so the subprocess evicts every+``rich`` module and installs a meta-path finder that refuses to import it again: the next import, the one+the sink's ``make_handler`` performs, is exactly what an uninstalled Rich would be. A subprocess, because+evicting a package from ``sys.modules`` in the test process would leave duplicate class objects behind+for every later test. Modelled on ``test_import_light_boot.py``.+"""++import subprocess # ruff: ignore[suspicious-subprocess-import]+import sys+import textwrap++_GUARD_SCRIPT = textwrap.dedent(+ """+ import importlib.abc+ import sys++ from pipelex.system.console_target import ConsoleTarget+ from pipelex.system.exceptions import MissingDependencyError+ from pipelex.tools.log.console_log_sink import RICH_EXTRA_NAME, ConsoleLogSink+ from pipelex.tools.log.log_config import HighlighterName, RichLogConfig++ for name in [module for module in sys.modules if module == "rich" or module.startswith("rich.")]:+ del sys.modules[name]++ class _Blocker(importlib.abc.MetaPathFinder):+ def find_spec(self, fullname, path, target=None):+ if fullname == "rich" or fullname.startswith("rich."):+ raise ImportError(f"no-rich guard blocked '{fullname}'")+ return None++ sys.meta_path.insert(0, _Blocker())++ sink = ConsoleLogSink(+ rich_log_config=RichLogConfig(+ is_show_time=False,+ is_show_level=True,+ is_link_path_enabled=True,+ highlighter_name=HighlighterName.JSON,+ is_markup_enabled=True,+ is_rich_tracebacks=True,+ is_tracebacks_word_wrap=True,+ is_tracebacks_show_locals=False,+ tracebacks_suppress=[],+ keywords_to_hilight=[],+ ),+ target=ConsoleTarget.STDERR,+ )+ try:+ sink.handler+ except MissingDependencyError as exc:+ message = str(exc)+ assert exc.dependency_name == "rich", message+ assert exc.extra_name == RICH_EXTRA_NAME, message+ assert f"pipelex[{RICH_EXTRA_NAME}]" in message, message+ assert "json" in message, message+ print("fails loud OK")+ else:+ raise SystemExit("expected MissingDependencyError from the console sink without Rich")+ """+)+++class TestConsoleLogSinkWithoutRich:+ def test_console_sink_fails_loud_naming_the_extra_and_the_json_alternative(self) -> None:+ result = subprocess.run( # ruff: ignore[subprocess-without-shell-equals-true]+ [sys.executable, "-c", _GUARD_SCRIPT],+ capture_output=True,+ text=True,+ check=False,+ timeout=120,+ )+ assert result.returncode == 0, f"no-rich guard failed:\nstdout={result.stdout}\nstderr={result.stderr}"+ assert "fails loud OK" in result.stdoutdiff --git a/tests/unit/pipelex/tools/test_log_dispatch.py b/tests/unit/pipelex/tools/test_log_dispatch.pyindex 88a445415..42406cf98 100644--- a/tests/unit/pipelex/tools/test_log_dispatch.py+++ b/tests/unit/pipelex/tools/test_log_dispatch.py@@ -1,42 +1,65 @@-from pytest_mock import MockerFixture+"""``include_exception`` carries the exception being handled on the record, and never in the message.++The traceback used to be spliced into the message text, which a structured sink would have carried as+a message and the Rich handler would have rendered twice once the record's ``exc_info`` was set. It now+rides the record as ``exc_info``, the stdlib's own channel, so every sink renders it its own way: the+console as a Rich traceback, the JSON sink as an ``exception`` field, the OTLP sink as the+``exception.*`` attributes.+"""++from __future__ import annotations++import logging+from typing import TYPE_CHECKINGfrom pipelex import log+if TYPE_CHECKING:+ import pytest+++def _own_records(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:+ return [record for record in caplog.records if record.name == __name__]-class TestLogDispatchIncludeException:- def test_include_exception_appends_traceback_to_console_message(self, mocker: MockerFixture) -> None:- # Regression guard: `include_exception=True` must append the traceback to the string actually sent- # to the console. The bug appended it to a dead local while the console was fed the pre-append- # string, so server-side errors (e.g. the API's `log.error(..., include_exception=True)`) logged- # only their summary line and never the underlying cause.- to_console = mocker.patch.object(log.log_dispatch, "_log_to_console")+class TestLogDispatchIncludeException:+ def test_include_exception_carries_the_active_exception_as_exc_info(self, caplog: pytest.LogCaptureFixture) -> None:cause = "boom-cause"- try:- raise ValueError(cause)- except ValueError:- log.error("something failed", include_exception=True)-- to_console.assert_called_once()- call = to_console.call_args- assert call is not None- logged_message = call.kwargs["message"]- assert "something failed" in logged_message- assert "Traceback (most recent call last)" in logged_message- assert "ValueError: boom-cause" in logged_message-- def test_no_traceback_when_include_exception_false(self, mocker: MockerFixture) -> None:- to_console = mocker.patch.object(log.log_dispatch, "_log_to_console")+ with caplog.at_level(logging.ERROR):+ try:+ raise ValueError(cause)+ except ValueError:+ log.error("something failed", include_exception=True)++ (record,) = _own_records(caplog)+ assert record.getMessage() == "something failed"+ assert "Traceback" not in record.getMessage()+ assert record.exc_info is not None+ exc_type, exc_value, exc_traceback = record.exc_info+ assert exc_type is ValueError+ assert str(exc_value) == cause+ assert exc_traceback is not None+ # The stdlib formatter renders it once, from the record, the way any sink can.+ assert "ValueError: boom-cause" in logging.Formatter().format(record)+ def test_no_exc_info_when_include_exception_is_false(self, caplog: pytest.LogCaptureFixture) -> None:cause = "boom-cause"- try:- raise ValueError(cause)- except ValueError:- log.error("something failed", include_exception=False)-- to_console.assert_called_once()- call = to_console.call_args- assert call is not None- logged_message = call.kwargs["message"]- assert "something failed" in logged_message- assert "Traceback" not in logged_message+ with caplog.at_level(logging.ERROR):+ try:+ raise ValueError(cause)+ except ValueError:+ log.error("something failed", include_exception=False)++ (record,) = _own_records(caplog)+ assert record.getMessage() == "something failed"+ assert record.exc_info is None++ def test_include_exception_outside_an_except_block_carries_nothing(self, caplog: pytest.LogCaptureFixture) -> None:+ """No exception is being handled, so there is nothing to carry: no ``NoneType: None`` anywhere."""+ with caplog.at_level(logging.ERROR):+ log.error("nothing active", include_exception=True)++ (record,) = _own_records(caplog)+ assert record.getMessage() == "nothing active"+ assert record.exc_info is None+ assert "NoneType" not in logging.Formatter().format(record)diff --git a/tests/unit/pipelex/tools/test_log_fields.py b/tests/unit/pipelex/tools/test_log_fields.pyindex eb2b6d05f..7c836fb79 100644--- a/tests/unit/pipelex/tools/test_log_fields.py+++ b/tests/unit/pipelex/tools/test_log_fields.py@@ -15,7 +15,7 @@ import pytestfrom pipelex import logfrom pipelex.tools.log.log_context import get_log_context-from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD, FORWARDED_MARK, VERBATIM_MARK, carried_attributesfrom pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSEif TYPE_CHECKING:@@ -199,3 +199,93 @@ class TestLogFields:override, plain = _own_records(caplog)assert _field(override, name="request_id") == "from-call-site"assert _field(plain, name="request_id") == "from-context"++ def test_a_data_field_beside_structured_content_is_carried_under_the_prefix_not_destroyed(self, caplog: pytest.LogCaptureFixture) -> None:+ """``data`` is the content's name, and a field spelling it used to be overwritten where every other collision is prefixed.++ The content keeps the name a sink reads it under, and the field keeps its value beside it. A call+ that spells both ``data`` and ``field_data`` loses neither, the prefix being applied until the name+ lands where nothing sits.+ """+ with caplog.at_level(logging.INFO):+ log.info({"from": "content"}, title="Data", fields={DATA_FIELD: "from-fields", "safe": 1})+ log.info({"from": "content"}, title="Data", fields={DATA_FIELD: "from-fields", f"{COLLIDING_FIELD_PREFIX}{DATA_FIELD}": "also-given"})++ one, both = _own_records(caplog)+ assert _field(one, name=DATA_FIELD) == {"from": "content"}+ assert _field(one, name=f"{COLLIDING_FIELD_PREFIX}{DATA_FIELD}") == "from-fields"+ assert _field(one, name="safe") == 1+ assert _field(both, name=DATA_FIELD) == {"from": "content"}+ assert _field(both, name=f"{COLLIDING_FIELD_PREFIX}{DATA_FIELD}") == "also-given"+ assert _field(both, name=f"{COLLIDING_FIELD_PREFIX}{COLLIDING_FIELD_PREFIX}{DATA_FIELD}") == "from-fields"++ def test_a_field_named_like_the_forwarding_marker_is_prefixed_and_never_reaches_a_sink(self, caplog: pytest.LogCaptureFixture) -> None:+ """The marker is Pipelex's own, so a fresh record does not carry the name yet and the stdlib would not refuse it.++ Left unreserved, a caller naming it hands the sink's handler a record its forwarding filter reads as+ one already delivered, and the whole record is dropped. The name is reserved, so the value is carried+ under the prefix, the record does not claim to have been forwarded, and the marker is not a field.+ """+ with caplog.at_level(logging.INFO):+ log.info("impersonation", fields={FORWARDED_MARK: True, "safe": 1})++ (record,) = _own_records(caplog)+ assert record.getMessage() == "impersonation"+ assert not getattr(record, FORWARDED_MARK, False)+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{FORWARDED_MARK}") is True+ assert _field(record, name="safe") == 1+ carried = carried_attributes(record=record)+ assert FORWARDED_MARK not in carried+ assert carried[f"{COLLIDING_FIELD_PREFIX}{FORWARDED_MARK}"] is True++ def test_a_field_named_like_the_verbatim_mark_is_prefixed_and_reaches_no_sink(self, caplog: pytest.LogCaptureFixture) -> None:+ """The mark is Rich's per-record markup override, which the console handler reads ahead of its own setting.++ Left unreserved, a caller naming it would decide whether the console interprets markup for that+ line, and the value would ride into the `json` and `otlp` payloads as a field, where it means+ nothing. Reserved, the caller's value is carried under the prefix and is not a carried attribute.+ """+ with caplog.at_level(logging.INFO):+ log.info("not the console's business", fields={VERBATIM_MARK: True, "safe": 1})++ (record,) = _own_records(caplog)+ assert not hasattr(record, VERBATIM_MARK)+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{VERBATIM_MARK}") is True+ carried = carried_attributes(record=record)+ assert VERBATIM_MARK not in carried+ assert carried[f"{COLLIDING_FIELD_PREFIX}{VERBATIM_MARK}"] is True+ assert carried["safe"] == 1++ def test_a_record_the_error_path_stamped_verbatim_hands_no_such_field_to_a_sink(self, caplog: pytest.LogCaptureFixture) -> None:+ """`TracebackMessageError` stamps the mark itself, and a structured sink must not write it as a field."""+ with caplog.at_level(logging.INFO):+ log.info("a line", fields={"safe": 1})++ (record,) = _own_records(caplog)+ setattr(record, VERBATIM_MARK, False)+ assert carried_attributes(record=record) == {"safe": 1}++ def test_the_forwarding_marker_a_record_already_carries_is_not_a_carried_field(self, caplog: pytest.LogCaptureFixture) -> None:+ """A structured sink reads the record's own fields; the marker is machinery and belongs on no wire."""+ with caplog.at_level(logging.INFO):+ log.info("already forwarded", fields={"safe": 1})++ (record,) = _own_records(caplog)+ setattr(record, FORWARDED_MARK, True)+ assert carried_attributes(record=record) == {"safe": 1}++ @pytest.mark.parametrize("name", ["__class__", "__dict__", "getMessage"])+ def test_a_name_the_record_class_owns_is_prefixed_and_the_record_survives(self, caplog: pytest.LogCaptureFixture, name: str) -> None:+ """A name owned by the ``LogRecord`` class rather than the instance is a collision too.++ Set on the instance, ``__class__`` and ``__dict__`` raise ``TypeError`` out of the log call and+ ``getMessage`` shadows the method so every formatter fails and the stdlib drops the record. Each is+ carried under the prefix instead, and the record still formats.+ """+ with caplog.at_level(logging.INFO):+ log.info("class-owned", fields={name: "given"})++ (record,) = _own_records(caplog)+ assert record.getMessage() == "class-owned"+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{name}") == "given"+ assert logging.Formatter("%(message)s").format(record) == "class-owned"diff --git a/tests/unit/pipelex/tools/test_log_holding.py b/tests/unit/pipelex/tools/test_log_holding.pynew file mode 100644index 000000000..c7220b123--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_holding.py@@ -0,0 +1,295 @@+"""Records emitted between ``log.configure`` and ``log.install_sink`` are held and replayed, in order, through the sink.++A boot that dies before its sink arrives closes the holding handler, and every record it held reaches+stderr through the stdlib's last resort: each one passed the level the configuration set, so each one is+a line the process was asked to show.+"""++from __future__ import annotations++import logging+from typing import TYPE_CHECKING++import pytest+from typing_extensions import override++from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.tools.log.log import Log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_fields import FORWARDED_MARK+from pipelex.tools.log.log_holding import HOLDING_CAPACITY, ForwardedRecordFilter, HoldingLogHandler+from pipelex.tools.log.log_sink import LogSink+from pipelex.tools.misc.toml_utils import load_toml_from_path++if TYPE_CHECKING:+ from collections.abc import Iterator++ from pytest_mock import MockerFixture+++def _package_log_config() -> LogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate(config_dict["runtime"]["log"])+++class _ListHandler(logging.Handler):+ def __init__(self) -> None:+ super().__init__()+ self.records: list[logging.LogRecord] = []++ @override+ def emit(self, record: logging.LogRecord) -> None:+ self.records.append(record)+++class _ListSink(LogSink):+ def __init__(self) -> None:+ super().__init__()+ self.list_handler = _ListHandler()++ @override+ def make_handler(self) -> logging.Handler:+ return self.list_handler++ def own_messages(self) -> list[str]:+ return [record.getMessage() for record in self.list_handler.records if record.name == __name__]+++class _RaisingOnPoisonHandler(_ListHandler):+ """Raises out of ``emit`` on one message, the way Rich raises on a line it reads as unbalanced markup."""++ @override+ def emit(self, record: logging.LogRecord) -> None:+ if record.getMessage() == "poison":+ msg = "closing tag '[/pipe]' doesn't match any open tag"+ raise RuntimeError(msg)+ super().emit(record)+++class _RaisingOnPoisonSink(_ListSink):+ def __init__(self) -> None:+ super().__init__()+ self.list_handler = _RaisingOnPoisonHandler()+++class TestHoldingLogHandler:+ @pytest.fixture+ def fresh_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[Log]:+ # pytest's ``log_level`` option restores the root logger's level at every phase boundary, undoing the+ # level ``configure`` sets from inside a fixture; the module's own logger is enabled explicitly and+ # ``caplog`` restores it at teardown.+ caplog.set_level(logging.INFO, logger=__name__)+ fresh = Log()+ fresh.configure(log_config=_package_log_config())+ try:+ yield fresh+ finally:+ fresh.reset()++ def test_records_held_since_configure_are_replayed_through_the_sink_in_order(self, fresh_log: Log) -> None:+ fresh_log.info("held first")+ fresh_log.warning("held second")+ sink = _ListSink()++ fresh_log.install_sink(sink)+ fresh_log.info("live third")++ assert sink.own_messages() == ["held first", "held second", "live third"]+ assert fresh_log.sink is sink+ assert sink.handler in logging.getLogger().handlers+ assert not any(isinstance(handler, HoldingLogHandler) for handler in logging.getLogger().handlers)++ def test_a_field_named_like_the_forwarding_marker_does_not_cost_the_record(self, fresh_log: Log) -> None:+ """The marker is internal, and a caller's field must never be read as one: the record reaches the sink all the same."""+ sink = _ListSink()+ fresh_log.install_sink(sink)++ fresh_log.warning("not lost", fields={FORWARDED_MARK: True})+ fresh_log.info("nor the next one")++ assert sink.own_messages() == ["not lost", "nor the next one"]++ def test_a_processor_edits_the_replayed_and_the_live_records_alike(self, fresh_log: Log) -> None:+ fresh_log.info("held")+ sink = _ListSink()++ def stamp(record: logging.LogRecord) -> None:+ record.stamped = True++ sink.processors.append(stamp)+ fresh_log.install_sink(sink)+ fresh_log.info("live")++ own = [record for record in sink.list_handler.records if record.name == __name__]+ assert [getattr(record, "stamped", None) for record in own] == [True, True]++ def test_without_a_sink_every_held_record_reaches_stderr_at_reset(self, fresh_log: Log, capsys: pytest.CaptureFixture[str]) -> None:+ """The failed-boot path: ``reset`` closes the holding handler and the stdlib's last resort speaks for all of them.++ Not only a warning or worse. The last resort's own level is meant for a record emitted before+ anything was configured, where nobody has said what is worth seeing; these passed the root logger's+ level, which came from the configuration, and a boot that died is exactly when the trail a verbose+ run was turned on to produce is what is being looked for.+ """+ fresh_log.info("an info the configuration asked for")+ fresh_log.warning("a warning that must be seen")++ fresh_log.reset()++ captured = capsys.readouterr()+ assert "a warning that must be seen" in captured.err+ assert "an info the configuration asked for" in captured.err++ def test_install_sink_before_configure_and_a_second_sink_are_refused(self) -> None:+ fresh = Log()+ with pytest.raises(RuntimeError, match="not configured"):+ fresh.install_sink(_ListSink())+ fresh.configure(log_config=_package_log_config())+ try:+ fresh.install_sink(_ListSink())+ with pytest.raises(RuntimeError, match="already installed"):+ fresh.install_sink(_ListSink())+ finally:+ fresh.reset()++ def test_a_held_record_the_sink_cannot_render_costs_neither_the_others_nor_the_installation(+ self, fresh_log: Log, capsys: pytest.CaptureFixture[str]+ ) -> None:+ """The sink stays installed so ``reset`` removes it, and the records after the poisoned one are replayed."""+ fresh_log.info("held first")+ fresh_log.info("poison")+ fresh_log.info("held third")+ sink = _RaisingOnPoisonSink()++ fresh_log.install_sink(sink)+ fresh_log.info("live fourth")++ assert sink.own_messages() == ["held first", "held third", "live fourth"]+ assert fresh_log.sink is sink+ assert sink.handler in logging.getLogger().handlers+ assert "Logging error" in capsys.readouterr().err+ fresh_log.reset()+ assert fresh_log.sink is None+ assert sink.handler not in logging.getLogger().handlers++ @staticmethod+ def _emit_while_both_handlers_are_on_the_root(*, mocker: MockerFixture, fresh_log: Log, message: str) -> None:+ """Widen the handoff's window deterministically: a record is emitted right before the holding handler leaves the root.++ ``install_sink`` puts the sink's handler on the root logger and then removes the holding handler; wrapping+ the removal emits a record while both are on the root, which is the interleaving a concurrent thread produces.+ """+ original_remove = logging.Logger.removeHandler++ def remove_after_emitting(logger: logging.Logger, handler: logging.Handler) -> None:+ if isinstance(handler, HoldingLogHandler):+ fresh_log.info(message)+ original_remove(logger, handler)++ mocker.patch.object(logging.Logger, "removeHandler", autospec=True, side_effect=remove_after_emitting)++ def test_a_record_emitted_while_both_handlers_are_on_the_root_is_delivered_exactly_once(self, fresh_log: Log, mocker: MockerFixture) -> None:+ fresh_log.info("held")+ sink = _ListSink()+ self._emit_while_both_handlers_are_on_the_root(mocker=mocker, fresh_log=fresh_log, message="meanwhile")++ fresh_log.install_sink(sink)+ fresh_log.info("live")++ assert sink.own_messages().count("meanwhile") == 1+ assert not any(isinstance(handler, HoldingLogHandler) for handler in logging.getLogger().handlers)++ def test_the_held_records_precede_a_record_emitted_during_the_handoff(self, fresh_log: Log, mocker: MockerFixture) -> None:+ fresh_log.info("held first")+ fresh_log.warning("held second")+ sink = _ListSink()+ self._emit_while_both_handlers_are_on_the_root(mocker=mocker, fresh_log=fresh_log, message="meanwhile")++ fresh_log.install_sink(sink)+ fresh_log.info("live")++ assert sink.own_messages() == ["held first", "held second", "meanwhile", "live"]++ def test_a_processor_runs_once_on_a_record_forwarded_during_the_handoff(self, fresh_log: Log, mocker: MockerFixture) -> None:+ sink = _ListSink()++ def count(record: logging.LogRecord) -> None:+ record.processed = getattr(record, "processed", 0) + 1++ sink.processors.append(count)+ self._emit_while_both_handlers_are_on_the_root(mocker=mocker, fresh_log=fresh_log, message="meanwhile")++ fresh_log.install_sink(sink)++ own = [record for record in sink.list_handler.records if record.name == __name__]+ assert [(record.getMessage(), getattr(record, "processed", None)) for record in own] == [("meanwhile", 1)]++ def test_a_record_that_reaches_the_holding_handler_after_the_drain_is_forwarded_to_the_sink(self) -> None:+ """A thread that picked the holding handler off the root logger just before its removal must lose nothing."""+ holding = HoldingLogHandler()+ target = _ListHandler()+ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="before", args=(), exc_info=None))+ holding.release_to(handler=target)++ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="late", args=(), exc_info=None))+ holding.close()+ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="later still", args=(), exc_info=None))++ assert [record.getMessage() for record in target.records] == ["before", "late", "later still"]+ assert holding.held_count == 0++ def test_a_drained_record_delivered_again_to_the_sinks_handler_is_refused(self) -> None:+ """The handoff's other side: a thread that picked the sink's handler up mid-handoff must not re-deliver a drained record.++ ``install_sink`` puts the sink's handler on the root logger between the drain and the holding+ handler's removal, so a thread descheduled inside ``callHandlers`` can resume on the sink's handler+ carrying a record the drain has already delivered. The forward path marked its records and the drain+ did not, which left the guard on the handler nothing to read.+ """+ holding = HoldingLogHandler()+ target = _ListHandler()+ target.filters.insert(0, ForwardedRecordFilter())+ record = logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="drained once", args=(), exc_info=None)+ holding.handle(record)++ holding.release_to(handler=target)+ target.handle(record)++ assert [delivered.getMessage() for delivered in target.records] == ["drained once"]+ assert getattr(record, FORWARDED_MARK, False) is True++ def test_a_record_below_the_handlers_level_is_neither_replayed_nor_forwarded(self) -> None:+ """``Handler.handle`` checks no level where the root logger's dispatch does, so the replay and the forward check it."""+ holding = HoldingLogHandler()+ target = _ListHandler()+ target.setLevel(logging.WARNING)+ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="held info", args=(), exc_info=None))+ holding.handle(logging.LogRecord(name=__name__, level=logging.WARNING, pathname="", lineno=0, msg="held warning", args=(), exc_info=None))+ holding.release_to(handler=target)++ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="late info", args=(), exc_info=None))+ holding.handle(logging.LogRecord(name=__name__, level=logging.ERROR, pathname="", lineno=0, msg="late error", args=(), exc_info=None))++ assert [record.getMessage() for record in target.records] == ["held warning", "late error"]++ def test_a_forwarded_record_the_sink_cannot_render_costs_neither_the_log_call_nor_the_next_record(self) -> None:+ holding = HoldingLogHandler()+ target = _RaisingOnPoisonHandler()+ holding.release_to(handler=target)++ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="poison", args=(), exc_info=None))+ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg="after", args=(), exc_info=None))++ assert [record.getMessage() for record in target.records] == ["after"]++ def test_the_holding_handler_keeps_the_newest_records_up_to_its_capacity(self) -> None:+ holding = HoldingLogHandler()+ for index in range(HOLDING_CAPACITY + 5):+ holding.handle(logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg=str(index), args=(), exc_info=None))++ assert holding.held_count == HOLDING_CAPACITY+ target = _ListHandler()+ holding.release_to(handler=target)+ assert holding.held_count == 0+ assert target.records[0].getMessage() == "5"+ assert target.records[-1].getMessage() == str(HOLDING_CAPACITY + 4)diff --git a/tests/unit/pipelex/tools/test_log_idempotence.py b/tests/unit/pipelex/tools/test_log_idempotence.pyindex 4735dd935..f65755e67 100644--- a/tests/unit/pipelex/tools/test_log_idempotence.py+++ b/tests/unit/pipelex/tools/test_log_idempotence.py@@ -5,13 +5,14 @@ with library embedders or interleaved tests that may already have called``log.configure`` earlier in the same process — instead of raising the once-per-processRuntimeError, ``configure_if_unset`` returns False and no-ops.-Each test uses the ``fresh_log`` fixture so the RichHandler installed by ``configure``-is removed from the global root logger on teardown; otherwise handlers accumulate-across the suite and pollute later test output.+Each test uses the ``fresh_log`` fixture so the handler ``configure`` installs on the+global root logger is removed on teardown; otherwise handlers accumulate across the+suite and pollute later test output."""from __future__ import annotations+import loggingfrom typing import TYPE_CHECKINGimport pytest@@ -38,10 +39,10 @@ def log_config() -> LogConfig:def fresh_log() -> Iterator[Log]:"""Yield a fresh Log() instance and reset it on teardown.- ``Log.configure`` attaches a RichHandler to the global root logger- (``logging.getLogger()``) and StreamHandlers to every named poor_logger. Without- explicit cleanup these handlers leak across tests in the same process — later tests- inherit duplicated handlers and stray output. ``reset()`` removes them.+ ``Log.configure`` attaches a holding handler to the global root logger+ (``logging.getLogger()``), which a sink's handler later replaces. Without explicit+ cleanup these handlers leak across tests in the same process — later tests inherit+ duplicated handlers and stray output. ``reset()`` removes them."""log_instance = Log()try:@@ -53,12 +54,14 @@ def fresh_log() -> Iterator[Log]:class TestLogConfigureIfUnset:def test_returns_true_and_applies_on_fresh_instance(self, fresh_log: Log, log_config: LogConfig) -> None:"""A fresh Log() instance has no config; configure_if_unset must apply it."""+ root_handlers_before = list(logging.getLogger().handlers)+applied = fresh_log.configure_if_unset(log_config=log_config)assert applied is True- # rich_handler is only built inside configure(); seeing it set proves we- # didn't take the no-op branch.- assert fresh_log.rich_handler is not None+ # The holding handler is only installed inside configure(); seeing one more root handler+ # proves we didn't take the no-op branch.+ assert len(logging.getLogger().handlers) == len(root_handlers_before) + 1def test_returns_false_after_prior_configure_without_raising(self, fresh_log: Log, log_config: LogConfig) -> None:"""A second call must no-op and return False instead of raising RuntimeError.@@ -80,3 +83,16 @@ class TestLogConfigureIfUnset:applied = fresh_log.configure_if_unset(log_config=log_config)assert applied is True++ def test_reset_leaves_every_other_root_handler_in_place(self, fresh_log: Log, log_config: LogConfig) -> None:+ """Teardown removes what configure installed and nothing else: a host's handler survives a Pipelex reset."""+ root_logger = logging.getLogger()+ foreign = logging.NullHandler()+ root_logger.addHandler(foreign)+ try:+ fresh_log.configure(log_config=log_config)+ fresh_log.reset()++ assert foreign in root_logger.handlers+ finally:+ root_logger.removeHandler(foreign)diff --git a/tests/unit/pipelex/tools/test_log_json_sink.py b/tests/unit/pipelex/tools/test_log_json_sink.pynew file mode 100644index 000000000..65d7dfee0--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_json_sink.py@@ -0,0 +1,218 @@+"""The ``json`` sink: one JSON object per line, parsed back, with the fields, the context and the exception.++The records go the whole way, from the facade through the module-named logger to the sink's handler,+on a fresh ``Log`` so the installed sink is the one under test and the teardown leaves the root logger+as it found it.+"""++from __future__ import annotations++import io+import json+import logging+import sys+from datetime import UTC, datetime+from typing import TYPE_CHECKING, Any++import pytest+from pydantic import BaseModel++from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.tools.log.json_log_sink import EXCEPTION_KEY, LOGGER_KEY, MESSAGE_KEY, SEVERITY_KEY, TIME_KEY, JsonLogSink+from pipelex.tools.log.log import Log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD+from pipelex.tools.misc.toml_utils import load_toml_from_path++if TYPE_CHECKING:+ from collections.abc import Iterator+++def _package_log_config() -> LogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate(config_dict["runtime"]["log"])+++def _own_lines(buffer: io.StringIO) -> list[dict[str, Any]]:+ """Every line this module emitted, each parsed as one JSON object, whatever else the process logged meanwhile."""+ lines = [json.loads(line) for line in buffer.getvalue().splitlines() if line]+ return [line for line in lines if line[LOGGER_KEY] == __name__]+++class TestJsonLogSink:+ @pytest.fixture+ def json_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, io.StringIO]]:+ """A fresh ``Log`` with the json sink installed on a buffer, torn down so the root logger is left as found."""+ # pytest's ``log_level`` option restores the root logger's level at every phase boundary, undoing the+ # level ``configure`` sets from inside a fixture; the module's own logger is enabled explicitly and+ # ``caplog`` restores it at teardown.+ caplog.set_level(logging.INFO, logger=__name__)+ buffer = io.StringIO()+ fresh = Log()+ fresh.configure(log_config=_package_log_config())+ fresh.install_sink(JsonLogSink(stream=buffer))+ try:+ yield fresh, buffer+ finally:+ fresh.reset()++ def test_each_record_is_one_json_object_with_the_fixed_keys(self, json_log: tuple[Log, io.StringIO]) -> None:+ fresh, buffer = json_log+ before = datetime.now(tz=UTC)+ fresh.info("first")+ fresh.warning("second")++ first, second = _own_lines(buffer)+ assert first[MESSAGE_KEY] == "first"+ assert first[SEVERITY_KEY] == "INFO"+ assert first[LOGGER_KEY] == __name__+ assert second[MESSAGE_KEY] == "second"+ assert second[SEVERITY_KEY] == "WARNING"+ assert first[TIME_KEY].endswith("Z")+ stamped = datetime.fromisoformat(first[TIME_KEY])+ assert stamped >= before.replace(microsecond=(before.microsecond // 1000) * 1000)+ assert EXCEPTION_KEY not in first++ def test_fields_context_and_data_ride_flat_beside_the_fixed_keys(self, json_log: tuple[Log, io.StringIO]) -> None:+ fresh, buffer = json_log+ with fresh.context(request_id="r1", pipeline_run_id="p1"):+ fresh.info("scanned", fields={"files": 7, "ratio": 0.5, "ok": True, "tags": ["a", "b"]})+ fresh.info({"key": "value", "nested": {"flag": True}}, title="Config")++ scanned, structured = _own_lines(buffer)+ assert scanned["files"] == 7+ assert scanned["ratio"] == 0.5+ assert scanned["ok"] is True+ assert scanned["tags"] == ["a", "b"]+ assert scanned["request_id"] == "r1"+ assert scanned["pipeline_run_id"] == "p1"+ assert "pipe_run_id" not in scanned+ assert structured[DATA_FIELD] == {"key": "value", "nested": {"flag": True}}+ assert structured[MESSAGE_KEY].startswith("Config:")++ def test_the_exception_is_a_field_and_not_part_of_the_message(self, json_log: tuple[Log, io.StringIO]) -> None:+ fresh, buffer = json_log+ try:+ msg = "boom"+ raise ValueError(msg)+ except ValueError:+ fresh.error("failed", include_exception=True)++ (line,) = _own_lines(buffer)+ assert line[MESSAGE_KEY] == "failed"+ assert "Traceback (most recent call last)" in line[EXCEPTION_KEY]+ assert "ValueError: boom" in line[EXCEPTION_KEY]++ def test_no_ansi_ever_even_for_a_warning_or_an_error(self, json_log: tuple[Log, io.StringIO]) -> None:+ fresh, buffer = json_log+ fresh.warning("careful")+ fresh.error("broken")+ fresh.critical("down")++ assert "\x1b" not in buffer.getvalue()+ assert [line[SEVERITY_KEY] for line in _own_lines(buffer)] == ["WARNING", "ERROR", "CRITICAL"]++ def test_a_field_named_like_a_fixed_key_is_prefixed_not_lost(self, json_log: tuple[Log, io.StringIO]) -> None:+ fresh, buffer = json_log+ fresh.info("collide", fields={SEVERITY_KEY: "custom", TIME_KEY: "then", "logger": "other"})++ (line,) = _own_lines(buffer)+ assert line[SEVERITY_KEY] == "INFO"+ assert line[f"{COLLIDING_FIELD_PREFIX}{SEVERITY_KEY}"] == "custom"+ assert line[f"{COLLIDING_FIELD_PREFIX}{TIME_KEY}"] == "then"+ assert line[f"{COLLIDING_FIELD_PREFIX}logger"] == "other"++ def test_a_value_json_does_not_know_is_serialized_and_never_lost(self, json_log: tuple[Log, io.StringIO]) -> None:+ """A model dumps in JSON mode, a datetime becomes text, a cycle keeps the line an object."""++ class Item(BaseModel):+ name: str+ when: datetime++ cyclic: dict[str, Any] = {}+ cyclic["me"] = cyclic+ fresh, buffer = json_log+ fresh.info("odd values", fields={"item": Item(name="x", when=datetime(2020, 1, 2, tzinfo=UTC)), "when": datetime(2021, 3, 4, tzinfo=UTC)})+ fresh.info("cyclic value", fields={"loop": cyclic})++ odd, cyclic_line = _own_lines(buffer)+ assert odd["item"] == {"name": "x", "when": "2020-01-02T00:00:00Z"}+ assert odd["when"] == "2021-03-04 00:00:00+00:00"+ assert cyclic_line[MESSAGE_KEY] == "cyclic value"+ assert isinstance(cyclic_line["loop"], str)+ assert "{...}" in cyclic_line["loop"]++ def test_a_non_finite_float_is_written_as_text_a_strict_parser_accepts(self, json_log: tuple[Log, io.StringIO]) -> None:+ """JSON has no NaN: the bare tokens Python writes by default would cost the whole line, so they are strings."""++ def refuse(token: str) -> None:+ msg = f"bare token {token} on the wire"+ raise ValueError(msg)++ fresh, buffer = json_log+ fresh.info(+ "readings", fields={"reading": float("nan"), "up": float("inf"), "nested": {"deep": [float("-inf"), 1]}, "ok": "kept", "flag": True}+ )+ fresh.info({"structured": float("inf")}, title="Data")++ strict = [json.loads(line, parse_constant=refuse) for line in buffer.getvalue().splitlines() if line]+ readings, structured = [line for line in strict if line[LOGGER_KEY] == __name__]+ assert readings["reading"] == "NaN"+ assert readings["up"] == "Infinity"+ assert readings["nested"] == {"deep": ["-Infinity", 1]}+ assert readings["ok"] == "kept"+ assert readings["flag"] is True+ assert structured[DATA_FIELD] == {"structured": "Infinity"}++ def test_a_non_finite_float_inside_a_model_costs_no_other_value_its_type(self, json_log: tuple[Log, io.StringIO]) -> None:+ """A model's dump is a plain mapping that still holds the Python float, which ``json`` refuses under ``allow_nan=False``.++ Unspelled, that refusal reached the line's own recovery, which reprs every value the line carries:+ one ``NaN`` nested in one field turned an integer into ``"3"`` and a list into its ``repr``, so the+ whole line lost its structure for it. The fallback spells a dump the way a value given directly is.+ """++ class Reading(BaseModel):+ score: float+ label: str++ fresh, buffer = json_log+ fresh.info("one bad float", fields={"reading": Reading(score=float("nan"), label="x"), "attempt": 3, "tags": ["a", "b"]})++ (line,) = _own_lines(buffer)+ assert line["reading"] == {"score": "NaN", "label": "x"}+ assert line["attempt"] == 3+ assert line["tags"] == ["a", "b"]++ def test_the_sinks_own_keys_are_reserved_with_or_without_an_exception(self, json_log: tuple[Log, io.StringIO]) -> None:+ """A field's wire name must not depend on an exception being active, and a cycle under a reserved name must not cost the line."""+ cyclic: dict[str, Any] = {}+ cyclic["me"] = cyclic+ fresh, buffer = json_log+ fresh.info("no exception", fields={EXCEPTION_KEY: "supplied"})+ try:+ msg = "boom"+ raise ValueError(msg)+ except ValueError:+ fresh.error("with exception", include_exception=True, fields={EXCEPTION_KEY: "supplied"})+ fresh.info("cycle under a reserved name", fields={EXCEPTION_KEY: cyclic})++ without, with_exception, cycle = _own_lines(buffer)+ assert EXCEPTION_KEY not in without+ assert without[f"{COLLIDING_FIELD_PREFIX}{EXCEPTION_KEY}"] == "supplied"+ assert "ValueError: boom" in with_exception[EXCEPTION_KEY]+ assert with_exception[f"{COLLIDING_FIELD_PREFIX}{EXCEPTION_KEY}"] == "supplied"+ assert cycle[MESSAGE_KEY] == "cycle under a reserved name"+ assert "{...}" in cycle[f"{COLLIDING_FIELD_PREFIX}{EXCEPTION_KEY}"]++ def test_redirect_to_stderr_moves_the_stream(self, json_log: tuple[Log, io.StringIO], capsys: pytest.CaptureFixture[str]) -> None:+ fresh, buffer = json_log+ fresh.redirect_to_stderr()+ fresh.info("on stderr now")++ assert _own_lines(buffer) == []+ captured = capsys.readouterr()+ assert any(json.loads(line)[MESSAGE_KEY] == "on stderr now" for line in captured.err.splitlines() if line.startswith("{"))+ assert fresh.sink is not None+ assert isinstance(fresh.sink, JsonLogSink)+ assert fresh.sink.handler.stream is sys.stderr # type: ignore[attr-defined]diff --git a/tests/unit/pipelex/tools/test_log_otlp_sink.py b/tests/unit/pipelex/tools/test_log_otlp_sink.pynew file mode 100644index 000000000..dd974c7dc--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_otlp_sink.py@@ -0,0 +1,253 @@+"""The ``otlp`` sink: a record reaches an in-memory log exporter with the fields as attributes.++The sink is built on a synchronous processor so each record is exported before the assertion reads it;+production builds it on the batching processor and the OTLP HTTP exporter instead.+"""++from __future__ import annotations++import logging+import threading+from typing import TYPE_CHECKING, Any++import pytest+from opentelemetry._logs import SeverityNumber+from opentelemetry.context import _SUPPRESS_INSTRUMENTATION_KEY, attach, detach, set_value # pyright: ignore[reportPrivateUsage]+from opentelemetry.sdk._logs.export import InMemoryLogExporter, SimpleLogRecordProcessor+from opentelemetry.semconv._incubating.attributes import code_attributes+from opentelemetry.semconv.attributes import exception_attributes++from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.tools.log.log import Log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX+from pipelex.tools.log.otlp_log_sink import FLUSH_TIMEOUT_MILLIS, ExportPathFilter, OtlpLogSink+from pipelex.tools.misc.toml_utils import load_toml_from_path++if TYPE_CHECKING:+ from collections.abc import Iterator++ from opentelemetry.sdk._logs import LogData+ from pytest_mock import MockerFixture+++def _package_log_config() -> LogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate(config_dict["runtime"]["log"])+++def _own_logs(exporter: InMemoryLogExporter) -> list[LogData]:+ """The records this module emitted, whatever else the process logged meanwhile."""+ return [log_data for log_data in exporter.get_finished_logs() if log_data.instrumentation_scope.name == __name__]+++def _attributes(log_data: LogData) -> dict[str, Any]:+ attributes = log_data.log_record.attributes+ assert attributes is not None+ return dict(attributes)+++class TestOtlpLogSink:+ @pytest.fixture+ def otlp_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, InMemoryLogExporter]]:+ # pytest's ``log_level`` option restores the root logger's level at every phase boundary, undoing the+ # level ``configure`` sets from inside a fixture; the module's own logger is enabled explicitly and+ # ``caplog`` restores it at teardown.+ caplog.set_level(logging.INFO, logger=__name__)+ exporter = InMemoryLogExporter()+ fresh = Log()+ fresh.configure(log_config=_package_log_config())+ fresh.install_sink(OtlpLogSink(processor=SimpleLogRecordProcessor(exporter)))+ try:+ yield fresh, exporter+ finally:+ fresh.reset()++ def test_a_record_reaches_the_exporter_with_the_fields_and_the_context_as_attributes(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ fresh, exporter = otlp_log+ with fresh.context(request_id="r1", pipe_run_id="pr1"):+ fresh.info("scanned", fields={"files": 7, "ratio": 0.5, "tags": ["a", "b"], "meta": {"k": "v"}})++ (log_data,) = _own_logs(exporter)+ record = log_data.log_record+ assert record.body == "scanned"+ assert record.severity_text == "INFO"+ assert record.severity_number is SeverityNumber.INFO+ attributes = _attributes(log_data)+ assert attributes["files"] == 7+ assert attributes["ratio"] == 0.5+ assert list(attributes["tags"]) == ["a", "b"]+ assert attributes["meta"] == '{"k": "v"}'+ assert attributes["request_id"] == "r1"+ assert attributes["pipe_run_id"] == "pr1"+ assert "pipeline_run_id" not in attributes+ assert attributes[code_attributes.CODE_FILE_PATH] == __file__+ assert attributes[code_attributes.CODE_FUNCTION_NAME] == "test_a_record_reaches_the_exporter_with_the_fields_and_the_context_as_attributes"+ assert attributes[code_attributes.CODE_LINE_NUMBER] > 0++ def test_structured_content_rides_as_json_text_in_the_data_attribute(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ fresh, exporter = otlp_log+ fresh.info({"key": "value", "nested": {"flag": True}}, title="Config")++ (log_data,) = _own_logs(exporter)+ assert _attributes(log_data)["data"] == '{"key": "value", "nested": {"flag": true}}'++ def test_the_exception_lands_under_the_semantic_convention_keys(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ fresh, exporter = otlp_log+ try:+ msg = "boom"+ raise ValueError(msg)+ except ValueError:+ fresh.error("failed", include_exception=True)++ (log_data,) = _own_logs(exporter)+ assert log_data.log_record.body == "failed"+ assert log_data.log_record.severity_number is SeverityNumber.ERROR+ attributes = _attributes(log_data)+ assert attributes[exception_attributes.EXCEPTION_TYPE] == "ValueError"+ assert attributes[exception_attributes.EXCEPTION_MESSAGE] == "boom"+ assert "Traceback (most recent call last)" in attributes[exception_attributes.EXCEPTION_STACKTRACE]++ def test_the_semantic_convention_keys_are_reserved_with_or_without_an_exception(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ """The sink writes these keys itself, so a field named like one is prefixed rather than overwritten.++ Reserved whether or not the record carries an exception, exactly as the json sink reserves its own+ keys on every line: a field's wire name must not depend on an exception being active, and the same+ field must not survive one sink and vanish on another.+ """+ fresh, exporter = otlp_log+ supplied = {+ code_attributes.CODE_FILE_PATH: "supplied/path.py",+ code_attributes.CODE_LINE_NUMBER: 999,+ code_attributes.CODE_FUNCTION_NAME: "supplied_function",+ exception_attributes.EXCEPTION_TYPE: "SuppliedError",+ exception_attributes.EXCEPTION_MESSAGE: "supplied message",+ exception_attributes.EXCEPTION_STACKTRACE: "supplied stacktrace",+ }+ fresh.info("no exception", fields=supplied)+ try:+ msg = "boom"+ raise ValueError(msg)+ except ValueError:+ fresh.error("with exception", include_exception=True, fields=supplied)++ without, with_exception = _own_logs(exporter)+ for log_data in (without, with_exception):+ attributes = _attributes(log_data)+ for name, value in supplied.items():+ assert attributes[f"{COLLIDING_FIELD_PREFIX}{name}"] == value+ assert attributes[code_attributes.CODE_FILE_PATH] == __file__+ assert attributes[code_attributes.CODE_LINE_NUMBER] > 0+ assert attributes[code_attributes.CODE_FUNCTION_NAME] == "test_the_semantic_convention_keys_are_reserved_with_or_without_an_exception"+ assert _attributes(with_exception)[exception_attributes.EXCEPTION_TYPE] == "ValueError"+ assert _attributes(with_exception)[exception_attributes.EXCEPTION_MESSAGE] == "boom"+ assert exception_attributes.EXCEPTION_TYPE not in _attributes(without)++ @pytest.mark.parametrize(+ ("method_name", "severity_text", "severity_number"),+ [+ ("verbose", "VERBOSE", SeverityNumber.TRACE),+ ("debug", "DEBUG", SeverityNumber.DEBUG),+ ("dev", "DEV", SeverityNumber.DEBUG4),+ ("info", "INFO", SeverityNumber.INFO),+ ("warning", "WARN", SeverityNumber.WARN),+ ("error", "ERROR", SeverityNumber.ERROR),+ ("critical", "CRITICAL", SeverityNumber.FATAL),+ ],+ )+ def test_every_level_maps_onto_the_otel_severity_scale(+ self,+ otlp_log: tuple[Log, InMemoryLogExporter],+ caplog: pytest.LogCaptureFixture,+ method_name: str,+ severity_text: str,+ severity_number: SeverityNumber,+ ) -> None:+ fresh, exporter = otlp_log+ caplog.set_level(logging.NOTSET + 1, logger=__name__)+ getattr(fresh, method_name)("at every level")++ (log_data,) = _own_logs(exporter)+ assert log_data.log_record.severity_text == severity_text+ assert log_data.log_record.severity_number is severity_number++ def test_a_record_from_the_opentelemetry_sdk_itself_is_not_exported(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ """An exporter's own failure line must not re-enter the pipeline that failed."""+ _, exporter = otlp_log+ logging.getLogger("opentelemetry.exporter.otlp").error("Failed to export logs batch")+ logging.getLogger("opentelemetry").warning("dropped")++ scopes = {log_data.instrumentation_scope.name for log_data in exporter.get_finished_logs()}+ assert not any(scope == "opentelemetry" or scope.startswith("opentelemetry.") for scope in scopes)++ def test_the_sdks_own_record_is_rejected_before_the_handler_lock_is_taken(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ """At exit the stdlib's shutdown holds this lock while it flushes, and an exporting thread reporting its failure must not wait on it."""+ fresh, _ = otlp_log+ assert fresh.sink is not None+ handler = fresh.sink.handler+ assert any(isinstance(handler_filter, ExportPathFilter) for handler_filter in handler.filters)+ sdk_record = logging.LogRecord(+ name="opentelemetry.sdk", level=logging.ERROR, pathname="", lineno=0, msg="export failed", args=(), exc_info=None+ )+ lock_taken = threading.Event()+ let_go = threading.Event()++ def hold_the_lock() -> None:+ handler.acquire()+ lock_taken.set()+ let_go.wait()+ handler.release()++ holder = threading.Thread(target=hold_the_lock)+ holder.start()+ lock_taken.wait()+ handled = threading.Event()++ def handle_the_sdks_record() -> None:+ handler.handle(sdk_record)+ handled.set()++ try:+ threading.Thread(target=handle_the_sdks_record, daemon=True).start()+ assert handled.wait(timeout=2), "the guard ran after the lock was taken"+ finally:+ let_go.set()+ holder.join()++ def test_a_record_emitted_during_an_export_is_rejected_whatever_its_logger_is_called(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ """The SDK sets a context value around ``exporter.export``; the transport logs under it, and the sink reads it. A private key, pinned here."""+ fresh, exporter = otlp_log+ token = attach(set_value(_SUPPRESS_INSTRUMENTATION_KEY, True))+ try:+ logging.getLogger("urllib3.connectionpool").warning("Retrying after connection broken")+ fresh.info("mine, during an export")+ finally:+ detach(token)+ fresh.info("mine, after the export")++ assert [+ log_data.log_record.body for log_data in exporter.get_finished_logs() if log_data.instrumentation_scope.name.startswith("urllib3")+ ] == []+ assert [log_data.log_record.body for log_data in _own_logs(exporter)] == ["mine, after the export"]++ def test_a_mixed_type_sequence_rides_as_json_text_rather_than_being_dropped(self, otlp_log: tuple[Log, InMemoryLogExporter]) -> None:+ """The SDK keeps a sequence of one scalar type, compared by type equality, and drops the rest to ``None``."""+ fresh, exporter = otlp_log+ fresh.info("sequences", fields={"mixed": [1, "foo"], "int_bool": [1, True], "homogeneous": ["a", "b"], "empty": []})++ (log_data,) = _own_logs(exporter)+ attributes = _attributes(log_data)+ assert attributes["mixed"] == '[1, "foo"]'+ assert attributes["int_bool"] == "[1, true]"+ assert list(attributes["homogeneous"]) == ["a", "b"]+ assert list(attributes["empty"]) == []++ def test_flush_hands_the_provider_a_deadline(self, otlp_log: tuple[Log, InMemoryLogExporter], mocker: MockerFixture) -> None:+ fresh, _ = otlp_log+ sink = fresh.sink+ assert isinstance(sink, OtlpLogSink)+ force_flush = mocker.patch.object(sink.logger_provider, "force_flush")++ sink.handler.flush()++ force_flush.assert_called_once_with(timeout_millis=FLUSH_TIMEOUT_MILLIS)diff --git a/tests/unit/pipelex/tools/test_log_reset.py b/tests/unit/pipelex/tools/test_log_reset.pynew file mode 100644index 000000000..bbd6ee645--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_reset.py@@ -0,0 +1,163 @@+"""``Log.reset`` finishes whatever the sink's handler does when it is flushed and closed.++It runs first in the runtime's release of its process globals, so an error escaping it would skip the+rest of that release and leave the process unbootable. A closed stream stays silent, as it does for+the stdlib's own shutdown; anything else the sink raises is said on stderr and swallowed.+"""++from __future__ import annotations++import io+import logging+from typing import TYPE_CHECKING++import pytest+from typing_extensions import override++from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.tools.log.log import Log+from pipelex.tools.log.log_config import LogConfig+from pipelex.tools.log.log_sink import LogSink+from pipelex.tools.misc.toml_utils import load_toml_from_path++if TYPE_CHECKING:+ from collections.abc import Iterator++ from pytest_mock import MockerFixture+++def _package_log_config() -> LogConfig:+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")+ return LogConfig.model_validate(config_dict["runtime"]["log"])+++class _RaisingAtCloseHandler(logging.Handler):+ def __init__(self, *, error: Exception) -> None:+ super().__init__()+ self._error = error++ @override+ def emit(self, record: logging.LogRecord) -> None:+ return++ @override+ def close(self) -> None:+ raise self._error+++class _RaisingAtCloseSink(LogSink):+ def __init__(self, *, error: Exception) -> None:+ super().__init__()+ self._error = error++ @override+ def make_handler(self) -> logging.Handler:+ return _RaisingAtCloseHandler(error=self._error)+++class _RaisingAtFlushHandler(logging.Handler):+ def __init__(self, *, error: Exception) -> None:+ super().__init__()+ self._error = error+ self.is_closed = False++ @override+ def emit(self, record: logging.LogRecord) -> None:+ return++ @override+ def flush(self) -> None:+ raise self._error++ @override+ def close(self) -> None:+ self.is_closed = True+ super().close()+++class _RaisingAtFlushSink(LogSink):+ def __init__(self, *, error: Exception) -> None:+ super().__init__()+ self._error = error++ @override+ def make_handler(self) -> logging.Handler:+ return _RaisingAtFlushHandler(error=self._error)+++class TestLogReset:+ @pytest.fixture+ def configured_log(self) -> Iterator[Log]:+ fresh = Log()+ fresh.configure(log_config=_package_log_config())+ try:+ yield fresh+ finally:+ fresh.reset()++ def test_an_error_out_of_the_sinks_close_is_said_on_stderr_and_the_reset_finishes(+ self, configured_log: Log, capsys: pytest.CaptureFixture[str]+ ) -> None:+ sink = _RaisingAtCloseSink(error=RuntimeError("the collector is gone"))+ configured_log.install_sink(sink)++ configured_log.reset()++ assert configured_log.sink is None+ assert sink.handler not in logging.getLogger().handlers+ captured = capsys.readouterr()+ assert "_RaisingAtCloseSink" in captured.err+ assert "the collector is gone" in captured.err++ def test_a_closed_stream_stays_silent_as_it_does_for_the_stdlibs_own_shutdown(+ self, configured_log: Log, capsys: pytest.CaptureFixture[str]+ ) -> None:+ sink = _RaisingAtCloseSink(error=ValueError("I/O operation on closed file"))+ configured_log.install_sink(sink)++ configured_log.reset()++ assert configured_log.sink is None+ assert capsys.readouterr().err == ""++ def test_a_flush_that_raises_is_said_on_stderr_and_the_close_still_runs(self, configured_log: Log, capsys: pytest.CaptureFixture[str]) -> None:+ """The close is what stops an exporter's thread and unregisters its exit hook; a failed flush must not skip it."""+ sink = _RaisingAtFlushSink(error=RuntimeError("the collector is unreachable"))+ configured_log.install_sink(sink)+ handler = sink.handler+ assert isinstance(handler, _RaisingAtFlushHandler)++ configured_log.reset()++ assert handler.is_closed+ assert configured_log.sink is None+ assert handler not in logging.getLogger().handlers+ captured = capsys.readouterr()+ assert "failed to flush" in captured.err+ assert "the collector is unreachable" in captured.err++ def test_a_flush_on_a_closed_stream_stays_silent_and_the_close_still_runs(self, configured_log: Log, capsys: pytest.CaptureFixture[str]) -> None:+ sink = _RaisingAtFlushSink(error=ValueError("I/O operation on closed file"))+ configured_log.install_sink(sink)+ handler = sink.handler+ assert isinstance(handler, _RaisingAtFlushHandler)++ configured_log.reset()++ assert handler.is_closed+ assert configured_log.sink is None+ assert capsys.readouterr().err == ""++ def test_a_closed_stderr_does_not_stop_the_reset_and_the_configuration_is_forgotten(self, configured_log: Log, mocker: MockerFixture) -> None:+ """The diagnostic has one place to go; when that place is gone the teardown still finishes and the next configure is not refused."""+ sink = _RaisingAtCloseSink(error=RuntimeError("the collector is gone"))+ configured_log.install_sink(sink)+ closed_stderr = io.StringIO()+ closed_stderr.close()+ mocker.patch("sys.stderr", closed_stderr)++ configured_log.reset()++ assert configured_log.sink is None+ assert not configured_log.is_configured+ assert sink.handler not in logging.getLogger().handlersdiff --git a/tests/unit/pipelex/tools/test_log_sink.py b/tests/unit/pipelex/tools/test_log_sink.pynew file mode 100644index 000000000..2cc8e7f21--- /dev/null+++ b/tests/unit/pipelex/tools/test_log_sink.py@@ -0,0 +1,89 @@+"""The sink base: the processor slot runs before the handler formats, the handler is built once, and a stream is resolved at call time."""++from __future__ import annotations++import io+import logging+import sys++import pytest+from typing_extensions import override++from pipelex.system.console_target import ConsoleTarget+from pipelex.tools.log.log_sink import LogSink, render_json, spell_non_finite, stream_for_target+++class _RecordingHandler(logging.Handler):+ def __init__(self) -> None:+ super().__init__()+ self.seen: list[str] = []++ @override+ def emit(self, record: logging.LogRecord) -> None:+ self.seen.append(f"{record.getMessage()}|{getattr(record, 'redacted', '-')}")+++class _RecordingSink(LogSink):+ def __init__(self) -> None:+ super().__init__()+ self.built = 0++ @override+ def make_handler(self) -> logging.Handler:+ self.built += 1+ return _RecordingHandler()+++def _record(*, message: str) -> logging.LogRecord:+ return logging.LogRecord(name=__name__, level=logging.INFO, pathname="", lineno=0, msg=message, args=(), exc_info=None)+++class TestLogSink:+ def test_the_handler_is_built_once_and_the_processors_run_before_it_emits(self) -> None:+ sink = _RecordingSink()++ def redact(record: logging.LogRecord) -> None:+ record.redacted = "yes"++ sink.processors.append(redact)+ handler = sink.handler+ handler.handle(_record(message="one"))+ sink.handler.handle(_record(message="two"))++ assert sink.built == 1+ assert sink.handler is handler+ assert isinstance(handler, _RecordingHandler)+ assert handler.seen == ["one|yes", "two|yes"]++ def test_redirect_to_stderr_is_a_no_op_on_a_sink_that_writes_to_no_stream(self) -> None:+ sink = _RecordingSink()+ sink.redirect_to_stderr()+ assert sink.built == 0++ def test_stream_for_target_resolves_the_live_process_stream(self, monkeypatch: pytest.MonkeyPatch) -> None:+ replaced_err = io.StringIO()+ monkeypatch.setattr(sys, "stderr", replaced_err)++ assert stream_for_target(target=ConsoleTarget.STDERR) is replaced_err+ assert stream_for_target(target=ConsoleTarget.STDOUT) is sys.stdout+ with pytest.raises(ValueError, match="file"):+ stream_for_target(target=ConsoleTarget.FILE)++ def test_render_json_never_raises(self) -> None:+ cyclic: dict[str, object] = {}+ cyclic["me"] = cyclic++ assert render_json(value={"a": 1}) == '{"a": 1}'+ assert render_json(value=object()).startswith('"<object object at')+ assert "{...}" in render_json(value=cyclic)+ assert render_json(value={(1, 2): "a"}) == "\"{(1, 2): 'a'}\""++ def test_a_non_finite_float_is_spelled_as_text_at_any_depth_and_nothing_else_moves(self) -> None:+ value = {"nan": float("nan"), "inf": float("inf"), "deep": [float("-inf"), 1, True, "x"], "pair": (2.5, float("nan"))}++ spelled = spell_non_finite(value=value)++ assert spelled == {"nan": "NaN", "inf": "Infinity", "deep": ["-Infinity", 1, True, "x"], "pair": [2.5, "NaN"]}+ assert spelled["deep"][2] is True+ assert render_json(value=[float("nan")]) == '["NaN"]'+ assert "NaN" not in render_json(value=[1.5])diff --git a/tests/unit/pipelex/tools/test_log_structured_content.py b/tests/unit/pipelex/tools/test_log_structured_content.pyindex d2736a8e5..519c33809 100644--- a/tests/unit/pipelex/tools/test_log_structured_content.py+++ b/tests/unit/pipelex/tools/test_log_structured_content.py@@ -105,3 +105,33 @@ class TestStructuredContent:(record,) = _own_records(caplog)assert record.getMessage() == "Empty:\nNone"assert not hasattr(record, DATA_FIELD)++ def test_a_circular_reference_is_rendered_as_its_repr_and_never_raises(self, caplog: pytest.LogCaptureFixture) -> None:+ """``json.dumps`` raises ``ValueError`` on a cycle, which the JSON helpers do not catch; a log call never raises."""+ cyclic: dict[str, Any] = {"name": "loop"}+ cyclic["self"] = cyclic+ with caplog.at_level(logging.INFO):+ log.info(cyclic, title="Cycle")++ (record,) = _own_records(caplog)+ assert record.getMessage().startswith("Cycle:\n")+ assert "loop" in record.getMessage()+ assert not hasattr(record, DATA_FIELD)++ def test_a_non_string_key_is_rendered_as_its_repr_and_never_raises(self, caplog: pytest.LogCaptureFixture) -> None:+ """``json.dumps`` applies ``default`` to values and not to keys, so a tuple key raises ``TypeError`` from every fallback."""+ keyed: dict[Any, Any] = {(1, 2): "a"}+ with caplog.at_level(logging.INFO):+ log.info(keyed)++ (record,) = _own_records(caplog)+ assert "(1, 2)" in record.getMessage()+ assert not hasattr(record, DATA_FIELD)++ def test_a_scalar_content_is_carried_as_itself(self, caplog: pytest.LogCaptureFixture) -> None:+ """``data`` is whatever the JSON rendering parses back to, a scalar for a scalar content."""+ with caplog.at_level(logging.INFO):+ log.info(42)++ (record,) = _own_records(caplog)+ assert getattr(record, DATA_FIELD) == 42diff --git a/uv.lock b/uv.lockindex eab09bc2f..93db0070a 100644--- a/uv.lock+++ b/uv.lock@@ -3476,10 +3476,10 @@ requires-dist = [{ name = "mypy", marker = "extra == 'dev'", specifier = "==1.19.1" },{ name = "networkx", specifier = ">=3.4.2" },{ name = "openai", specifier = ">=2.0.0" },- { name = "opentelemetry-api" },- { name = "opentelemetry-exporter-otlp-proto-http" },- { name = "opentelemetry-sdk" },- { name = "opentelemetry-semantic-conventions" },+ { name = "opentelemetry-api", specifier = ">=1.38.0" },+ { name = "opentelemetry-exporter-otlp-proto-http", specifier = ">=1.38.0" },+ { name = "opentelemetry-sdk", specifier = ">=1.38.0" },+ { name = "opentelemetry-semantic-conventions", specifier = ">=0.59b0" },{ name = "pillow", specifier = ">=12.1.1" },{ name = "pipelex-tools", marker = "extra == 'dev'", specifier = ">=0.7.2" },{ name = "pipelex-tools-py", specifier = ">=0.1.3" },diff --git a/wip/structured-logs/review-deferrals.md b/wip/structured-logs/review-deferrals.mdindex 14d20b025..dca18c8b9 100644--- a/wip/structured-logs/review-deferrals.md+++ b/wip/structured-logs/review-deferrals.md@@ -2,6 +2,8 @@Traces for what a review pass on a structured-logs member read as real but not worth fixing at that pass's bar. The sprint is L-260916-a8cdd2; the campaign documents live at the workspace root under `wip/structured-logs/`. An entry marked *unverified* rests on the reviewer's word alone: nobody reproduced it.+The sink member L-260916-aa6e58 took every entry below that named it: the traceback now rides the record's `exc_info`, the console flag and the poor branch retired with the log mode, the mutating formatter went with the poor handler, the class-owned attribute collision and the circular-reference and non-string-key raises are guarded and tested, the changelog bullet is folded, the fields-by-reference and `data` contracts are written down in the logging page, the `data` parse stays eager and the reason is recorded there, the `ErrorReport` comment names the payload fields alone, and the logger-attached filter advice is corrected in the changelog and the page. The entries stay here as the trace of the rounds that found them.+## L-260916-a1dbd5, round 1 (fields, context, no stack walk)- **Caller attribution rides a fixed frame depth** (*unverified*, cubic). `CALLER_FRAME_DEPTH` in `pipelex/tools/log/log_dispatch.py` counts the frames between the frame lookup and the code that called `log.<level>(...)`, so a helper inserted into or removed from that call chain would attribute records to the wrong module and apply the wrong logger level. The tests in `tests/unit/pipelex/tools/test_log_caller_module.py` emit through every level and assert the caller's module, path and function, which is what catches such a change. A walk up the frames until one leaves the log package would remove the constant at the cost of a loop per line; the fixed depth stays until a second facade entry point makes the constant wrong for one of them.@@ -38,3 +40,133 @@ The bar changed no code. Every reviewer raised the class-owned attribute collisi- **The caller-info test fixture's teardown is process-global** (*unverified*, code-review). `log_with_caller_info` in `tests/unit/pipelex/tools/test_log_caller_info.py` calls `Log.reset()`, which removes and closes every root-logger handler and calls `logging.shutdown()`. `Log.reset()` is pre-existing; the unit suite is green under random and fixed ordering, so this is a latent isolation hazard rather than a failure. A fixture that removes only the handler it added would remove the hazard.- **The dev-side caller-info fix is filed under Changed** (*unverified*, code-review). `FUNC_MODULE` and `FUNC_MODULE_LINE` raised `KeyError: 'module'` on dev at the first log line after boot; the branch passes the module and covers it in `test_log_caller_info.py`, and the changelog folds that into the logger-naming bullet rather than a Fixed entry.- **Out of scope, unverified: seven findings on the `main..dev` delta** (code-review, whose skill diffed `main...HEAD`). Files this branch does not touch, listed here so they can be chased and never as this branch's: `pipelex/pipeline/input_normalizer.py:232` (`HttpUrl`'s length cap rejects long signed URLs); `pipelex/tracing/ndjson_event_log.py:163` and `pipelex/tracing/dynamodb_event_log.py:170` (corrupt-vs-refused decided without checking `loc == ()`); `pipelex/tracing/ndjson_event_log.py:164` (raw `validation_error` interpolation leaks traced content); `pipelex/cli/commands/graph_cmd.py:59` (`json_invalid` misreported as a stale-file error); `pipelex/graph/mermaidflow/templates/_interactive_scripts.js.jinja2:275` (the JSON tab falls back without consulting `disabled`); `pipelex/tools/misc/http_utils.py:22` (whitespace accepted against the documented rule); `pipelex/runtime_hub.py:210` (`reset_boot_state` reverts the agent CLI's SILENT printer).++## L-260916-aa6e58, round 1 (bar open)++The pass fixed the nine confirmed findings: the OTLP handler's re-entrancy filter and its teardown guard, the mixed-type sequence, the JSON sink's non-finite floats and its reserved keys, the holding handoff, the console formatter's prefix under `exc_info`, the doctor's fallback sink and the migration entry's account of what a poor-mode or silenced file loses. The refuted finding (the override and kit configs carrying `sink` without `[runtime.log.otlp]`) was rejected: the kit carries the user-facing subset by design and the deep merge supplies the rest. What follows was read as real and left.++- **The held-record window spans the remote config fetch** (*unverified*, code-review). Between `Log.configure` and `log.install_sink` in `RuntimeBoot.setup`, the registrar is built and the remote config may be fetched, so a boot that logs verbosely on that path holds those records in memory up to the holding handler's capacity; beyond it the oldest are dropped. The capacity is generous for a boot and the window is documented in `docs/under-the-hood/log-sink-plugins.md`; a sink installed earlier would need the registrar earlier, which is the boot order's decision and not this member's.+- **The JSON sink binds its stream at factory time** (*unverified*, code-review). `pipelex/tools/log/json_log_sink.py` resolves `sys.stdout` or `sys.stderr` when the factory runs, so a consumer that swaps `sys.stderr` after boot, pytest's capture among them, keeps writing to the stream the sink captured. Resolving the stream per emit would follow the swap at the cost of a lookup per line; deliberate for now, and the test module patches the stream before installing.+- **A `console_log_target = "file"` raises a bare `ValueError`** (*unverified*, code-review, pre-existing). `stream_for_target` in `pipelex/tools/log/log_sink.py` inherits dev's handling of the third target value, which has no file path to open. The enum member predates this branch and no caller sets it; retiring the member or giving it a path is a config change of its own.+- **`redirect_to_stderr` imports Rich unguarded** (*unverified*, code-review). A server install without the CLI extra fails on that path. L-260916-fe7969 moves Rich behind the extra and adds the import guard, and the note on it carries this.+- **The Rich handler raises `MarkupError` out of a log call** (*verified by the round-1 verifier*, pre-existing). With `is_markup_enabled = true`, `RichHandler.emit` renders the message outside its own `try`, so a message whose text reads as a stray closing tag, `[/pipe]` for one, raises out of `log.info`. The holding drain now contains it for held records; a live record still raises. Dev's Rich mode did the same. A console handler subclass whose `emit` routes the error to `handleError` is the fix, owned by the console sink's next pass or by L-260916-fe7969 when it reworks the Rich path.+- **The doctor duplicates the log-sink composition root** (*unverified*, cubic). `doctor_cmd.py` performs discovery, registry construction, selection and installation that `RuntimeBoot.setup` also performs, and the fallback added this round widens the difference. Extracting one function both call is a refactor below this bar; it is worth doing before a third caller appears.+- **The migration deletes `log_mode` and `is_console_logging_enabled` without translating them** (*verified by the round-1 verifier*, cubic). A `remap_value` onto `sink` is refused by the ledger check because `sink` is an open token, an intermediate schema for a split into a safe and an unsafe entry has no golden a model ever had, and an `unsafe` entry blocks every full copy of an older base file at its defaults for the sake of the few that chose `poor` or `false`. The entry stays `safe`; what a person running `pipelex migrate` sees is the step's title, so the title names what a poor-mode file then sets by hand, and the description and guidance say what the deletion undoes for both keys. The engine gap underneath, that a safe entry's guidance is rendered nowhere, is L-260916-5d142f.+- **The migration coverage gate is blind to a deleted enum key** (*verified by the round-1 verifier*, tooling). `_check_enum_members_remapped` in `pipelex/migration/coverage.py` continues when a path does not survive into the new schema, which is how a `delete_key` on `log_mode` passed the gate. Filed as L-260916-0e1435.++## L-260916-aa6e58, round 2 (bar defects)++The pull request opened after round 1, so this round measured the sink seam alone against `feature/Log-fields-context`. Every reviewer raised the handoff's duplicate window and the flush that skips the close, and the adversarial reviewer the migration's silencing recipe; those were fixed. The concurrent `install_sink` race (cubic) was rejected: the boot configures logging once per process and `Log.configure` refuses a second call, so two installers is a state the runtime does not reach. What follows was read as real and left.++- **The structured-content guard is wider than the failure it catches** (*unverified*, code-review). `_render_content` in `pipelex/tools/log/log_dispatch.py` wraps both the `purify_json*` call and the `json.loads` re-read in one `except (TypeError, ValueError)`, so a re-read that failed after a successful purification would discard the truncated rendering for `repr(content)`, whole. No trigger is shown: the truncation happens inside the helpers on byte values, so what they hand back is loadable JSON. Narrowing the `try` to the helpers, or keeping the rendering when only the re-read failed, removes the question when the function is next touched.+- **The doctor keeps an already-configured sink without resolving the configured token** (*unverified*, cubic). When logging was configured before `setup_doctor_runtime` runs, an embedder or an interleaved test, the log-sink row reports healthy without asking the registry about `sink`. The CLI configures nothing before the command, so `pipelex doctor` always takes the other arm; an embedder calling the doctor in-process after its own boot keeps its own sink, which is the intent. Resolving the token in both arms is a small extension of the row, below this bar.+- **The migration's safe deletion of a silencing file** (adversarial, restating round 1). The round-1 entry above stands; this round corrected the recipe the entry names so that a person who follows it is silenced in fact.++## L-260916-aa6e58, round 3 (bar necessity)++The bar admitted what the round-2 pass's own changes introduced: the holding handler's replay and forward delivered records below the sink handler's level, since `Handler.handle` checks no level where `Logger.callHandlers` does, and the teardown's stderr diagnostic could itself raise on a closed stream and leave logging configured. Both were reproduced by the driver and fixed. What follows was read as real and left.++- **A sink whose handler build fails after its construction allocated resources is dropped without teardown** (*unverified*, cubic). `install_sink` reads `sink.handler` before recording the sink, deliberately, so that a failed build leaves the runtime untouched and the doctor's fallback can install; the cost is that a plugin sink whose factory opened a connection or started a thread and whose `make_handler` then raised has nothing that closes it, since the sink contract carries no teardown hook of its own. The built-in sinks allocate in the handler or, for `otlp`, in a provider that registers its own shutdown at exit. A `close` hook on the sink contract is the fix, and a contract change wants its own pass.+- **The migration's safe deletion of a silencing file, at boot** (adversarial, restating rounds 1 and 2 with one new fact). Boot tolerance replays the ledger in memory over a stale file, so a deployment that set `is_console_logging_enabled = false` resumes output on upgrade without running `pipelex migrate`. The boot's stale-configuration warning prints the titles it carried forward, so that deployment sees, on the console the boot re-enabled, the entry's title naming the by-hand edit; no repository in the workspace sets the flag. The round-1 entry stands, with L-260916-5d142f and L-260916-0e1435 carrying the engine gaps.+- **The doctor duplicates the plugin and sink composition root** (cubic, restating rounds 1 and 2). The fallback added in round 2 widened the difference; the entry above stands.+- **A processor that raises escapes the log call** (*verified by code-review's reproduction*). `_ProcessorFilter.filter` in `pipelex/tools/log/log_sink.py` runs each processor unguarded, and the stdlib runs a handler's filters outside every `try`, so a processor raising on a live record raises out of `log.info`; the same processor raising during the holding replay or forward is contained. Latent: nothing in `pipelex/` registers a processor yet. The scrub member L-260916-4837d4 registers the first one and owns the guard, a per-processor `except` routed to `handleError`; the note on it carries this.+- **The OTLP sink overwrites a carried `code.*` or `exception.*` field** (*verified by code-review's reproduction*). `_attributes` in `pipelex/tools/log/otlp_log_sink.py` assigns the semantic-convention keys after seeding from the carried attributes, so a field named `code.line.number` is discarded where every other sink re-prefixes; the JSON sink's prefix loop with those keys as its reserved set is the fix, below this bar.+- **Nits** (code-review): `OtlpLogHandler.close` calls the provider's shutdown before `super().close()`, so a shutdown that raises leaves the handler in the stdlib's handler list, contained by the teardown guard; the holding handler's drop at capacity is linear per record where a bounded deque would be constant, reached only by a process that never installs a sink.++## L-260916-aa6e58, round 4 (bar freeze)++The bar changed no code. The adversarial reviewer raised the migration's safe deletion of a silencing file a fourth time, and code-review the OTLP close order, the linear drop and the kit configs without `[runtime.log.otlp]`; the first three keep their entries above and the last was refuted in round 1. What follows is new. The two entries marked *verified* were reproduced by the driver and are filed as ledger bugs, since each is a one-line fix with a test that the branch below this one ships without.++- **A caller field named `_pipelex_forwarded` is read as the handoff mark** (*verified*, codex:review and code-review). `attach_log_record_extra` prefixes a name only when the record already owns it, and a fresh record does not own the mark, so `log.info("x", fields={"_pipelex_forwarded": True})` lands the attribute unprefixed and `ForwardedRecordFilter` drops the record from every sink, after the handoff too. The comment on `FORWARDED_MARK` claims the underscore keeps a caller from spelling it, which held while records went through `makeRecord(extra=...)` and no longer does. Filed as L-260916-40bf74: treat the mark as owned beside `FORMATTER_OWNED_ATTRIBUTES`, correct the comment, add the test.+- **The doctor's fallback install can itself raise** (*verified*, code-review). `install_doctor_log_sink` retries on any exception out of `install_sink` on the claim that a failed install leaves nothing installed; `install_sink` records the sink before the replay, deliberately, so a held record the handler cannot render whose `handleError` write meets a closed stderr raises `ValueError` with the sink recorded, and the fallback then raises `RuntimeError("already installed")`, which nothing in the doctor catches. Filed as L-260916-7e131d: gate the fallback on `log.sink is None`.+- **A caller's `fields={"data": ...}` is overwritten by structured content** (*unverified*, code-review). `build_log_record_extra` documents that structured content owns `data` outright, which makes it the one collision in the module that is not preserved under the `field_` prefix. A documented choice; prefixing the caller's entry instead would make the module's no-value-lost claim hold for every key, when the module is next touched.+- **Sink factories take the whole `LogConfig`** (*unverified*, cubic). The plugin contract hands every external factory the core configuration, which has no place for a plugin's own settings, so `docs/under-the-hood/log-sink-plugins.md` directs plugins to environment variables or private files. A narrow factory context carrying a plugin-owned payload is a contract change and wants its own pass, with the teardown hook the round-3 entry asks for.+- **Processors mutate the record every root handler shares** (*unverified*, cubic). `_ProcessorFilter` edits the original record, so a host-installed root handler sees it processed or not by its position in the handler list; a scrub that redacts reaches one handler and misses another. The scrub member L-260916-4837d4 registers the first processor and decides whether it runs on a copy or at one canonical point; the note on it carries this.+- **Held records pin live tracebacks** (*unverified*, code-review). `include_exception` now rides the record as `exc_info` rather than a formatted string, so a record held between `configure` and `install_sink` keeps its traceback, every frame and its locals, until the sink arrives or the boot dies. Bounded by the holding capacity and by a window of a few dozen lines; formatting `exc_text` eagerly when holding would cap it.+- **Nit** (code-review): `spell_non_finite` in `pipelex/tools/log/log_sink.py` catches `TypeError` and `ValueError` only, so a value nested deeper than the interpreter's recursion limit raises `RecursionError` past the docstring's never-raising claim; contained by `StreamHandler.emit`'s `handleError`, so the line is lost and not the process.++## L-260916-aa6e58, the migration finding, settled by the owner on 2026-09-17++**`pipelex-config@5` stays `safe`, and this finding is closed rather than deferred.** It was raised on every round of this member and once more by a Codex pass reading `origin/dev` during the Rich member's round 1, each time as the same reading: the entry deletes `is_console_logging_enabled = false` without translating it, and boot tolerance replays the ledger in memory, so an upgrade alone hands a silenced deployment its console back.++An implementer cured it on 2026-09-17 by making the entry `unsafe`, and that change was reverted the same day on the owner's decision. What the block cost is what decided it. `safety` is a property of an entry, entries are one per schema version and contiguous (`pipelex/migration/ledger.py:197`), and this version's link owns all four key removals, so the block cannot be narrowed to the case that matters. An `unsafe` entry is rehearsed and never applied (`pipelex/migration/engine.py:118`), so the retired keys survive the boot-tolerance replay, the re-validation fails again against a model that is `extra="forbid"` (`pipelex/system/configuration/config_model.py:12`), and `load_config_and_validate` re-raises (`pipelex/system/configuration/config_loader.py:530`). The block therefore converts "boots with a warning" into "fails to boot" for **every** file still carrying any of the four keys at their old defaults — every full copy of an older base file — for the sake of the few that chose otherwise.++The owner's reading is that nothing chose otherwise: no deployment set these keys, nothing depends on the flag outliving its removal, and this project keeps no backward compatibility. Round 3 had already recorded the corroborating fact, that no repository in the workspace sets the flag.++**Reopening this needs a new fact, not a new reading** — a configuration that actually sets one of the retired keys and a reason it may not be edited by hand. The rationale is recorded in the entry's own comment in `pipelex/migration/ledgers/pipelex-config.toml` so that it travels with the code rather than with this document.++Two corrections outlived the revert, both independent of the safety decision and both kept: the entry's guidance and `docs/configuration/config-practical/logging-config.md` had said `is_console_logging_enabled = false` "silenced every handler on the root logger", where `LogDispatch.dispatch` returned early and third-party loggers still reached the handlers, and the entry's comment had named `default_log_level = "OFF"` as the replacement where the base pins `pipelex` in `package_log_levels` and the root level alone silences none of Pipelex's records.++One finding from the reverted work is real and stays: `BlockedEntry` carries no `title` or `description` (`pipelex/migration/plan.py:72`), so a blocked entry reaches its reader through `guidance` and `detail` alone. It is latent — no shipped ledger entry is `unsafe` — and it is the mirror of L-260916-5d142f, which carries a safe step's unrendered guidance; the note on that item records it.++## L-260916-aa6e58, round 1 after the growth reset (bar open)++The branch grew past the growth line after round 4 — the forwarding-marker reservation and the OTLP reserved keys — so the ladder reset to round 1 at bar `open` and all four reviewers read the whole branch against `origin/dev` again. Twelve findings; eight went to a verifier against the checkout at `84c231c82`, and four were sorted as standing deferrals whose traces are above. **Not one of the twelve is about the lines the reset was triggered by**, which is worth recording: the new code is the part of this branch the rounds have now read most and found least in.++Five were confirmed and fixed. Each fix was mutation-tested — the fix reverted, its own guard watched going red, and no other test moving:++- **A non-finite float nested in a pydantic model cost the whole JSON line its structure.** `json_fallback` returned `model_dump(mode="json")`, which keeps the Python `nan` because pydantic's own spelling applies only where pydantic writes the text itself, so `json.dumps` refused the payload under `allow_nan=False` and `_json_line`'s recovery reprd every non-fixed key: an integer field became a string, a list its `repr`. The fallback now spells the dump the way a value handed in directly is spelled. Reachable with `sink = "json"`, which the hosted plane's runner and worker select, and the `otlp` sink's `render_json` took the same route.+- **`release_to` drained held records without marking them**, where the forward path marks. A thread descheduled inside `callHandlers` before the handoff can resume on the sink's handler carrying a record the drain already delivered, and the guard on that handler had nothing to read — a window distinct from the one the forwarding-marker commit closed, on the other side of the drain.+- **`close` dropped every held record below the last resort's own WARNING**, so a boot that died between `configure` and `install_sink` lost exactly the trail a verbose run was turned on to produce. The last resort's level is meant for a record emitted before anything was configured; these passed the level the configuration set.+- **A field spelling `data` beside structured content was destroyed** rather than moved aside under the `field_` prefix — the round-4 deferral above, taken rather than carried again, since the discipline it breaks is the one the module states everywhere else. The content keeps the name a sink reads it under and the field keeps its value, the prefix applied until the name lands where nothing sits.+- **The four `opentelemetry-*` dependencies carried no version specifier at all**, and no job resolves the floor. The SDK took a positional `emit(record)` in the versions the declaration permitted while the `otlp` sink passes the keyword form, so such a resolution made every record a `TypeError` that `handleError` turned into a traceback on stderr — nothing reaching the collector while the process reported success. Floors are now declared at the locked versions. A lowest-resolution job is still absent and would be the thing that keeps them honest.++Deferred, with what the verification established:++- **A processor that raises kills the live log call** (*verified*, codex:adversarial). `_ProcessorFilter.filter` runs the callbacks inside `Handler.handle`, which the stdlib calls before its own `try`, so the exception leaves `log.info` itself; the replay path swallows the same exception through `_deliver`'s `handleError`, so the two paths disagree about the same processor. No processor is registered anywhere in the tree, but `docs/under-the-hood/log-sink-plugins.md` tells readers to append a redaction one, so it is reachable for anyone following the documentation. It stays with the scrub member L-260916-4837d4, together with the shared-record mutation above, because the remedy is that member's contract choice rather than a review round's: a redaction processor that raises means the record is *unredacted*, so rendering it anyway leaks what it was meant to hide, while the class currently promises it never drops one.+- **The OTLP suppression guard rejects far more than the export path** (*verified*, code-review). `ExportPathFilter` returns on the context value alone after the name guard, so any record emitted while the SDK's batch processor holds `_SUPPRESS_INSTRUMENTATION_KEY` is dropped whatever its origin. The concrete loss is in this repo and the finding did not name it: `PostHogSpanExporter._do_export` logs its own failures on a `pipelex.*` logger inside a `BatchSpanProcessor` scope, so Pipelex's span-export diagnostics vanish from the `otlp` sink without trace. Deferred because the guard's breadth is what fixed a reproduced exit deadlock and the transport's logger names are open-ended, so narrowing it is design work — a diagnostics path that bypasses the export queue — and not a flip. Needs `sink = "otlp"` and telemetry both.+- **A boot that fails inside `__init__` after `configure` never closes the holding handler at all** (*verified*, found by the verifier rather than by a reviewer). `_release_after_failed_boot` is what reaches `log.reset()`, and `runtime_boot.py`'s own docstring records that both `make()`s construct outside their `try`, so on that path nothing is emitted anywhere — not even the stale-configuration warning. That is `runtime_boot`'s structure and not this module's, and the `close` fix above does not reach it.++One earlier entry is refined rather than repeated. **The doctor's fallback install** (filed as L-260916-7e131d) was confirmed as to ordering, and the verification moved where its fix belongs: `install_sink` records the sink before the replay *deliberately*, so `reset` can find and remove a handler already on the root logger, and the comment at the assignment says so. The defect is the doctor's `except` arm installing without clearing first, so the cure is in `doctor_cmd.py` and the ordering in `log.py` stays. The verifier also narrowed the trigger: every failure the arm was written for — a missing dependency, a console target no sink writes to, a factory that raises — happens at `sink.handler` before the assignment, so the fallback works for all of them, and reaching the double install needs the replay's own recovery to raise, which in practice means an unusable stderr or a third-party handler overriding `handleError`.++## L-260916-aa6e58, round 2 after the growth reset (bar defects)++Three reviewers of the four found only the standing deferrals — cubic four, `codex:review` two, `codex:adversarial` one, every distinct claim already traced above and none in the code the round's focus named. `code-review` was the one that found new ground, and what it found was the most serious thing six rounds on this member have turned up.++**A log call could raise on the default sink, which is the one thing the branch promises it never does.** The finding arrived as a claim about one error path; verifying it made it larger and moved its cause. `RichHandler` overrides `emit` and does not restore the `try`/`handleError` the stdlib's own handlers put around theirs, so anything raised while rendering leaves the log call — and Rich's markup parser raises on ordinary text. Its tag has to open with a lowercase letter, `#`, `/` or `@`, which the finding's own example got wrong: `[Errno 2]` is safe. What is not safe is the two shapes an error message is actually made of. `list[int]` in a type complaint loses the bracketed span silently. A bracketed path reads as a closing tag with nothing open and raises `MarkupError`. Confirmed end to end against the branch before the fix: `log.error("no such file: [/etc/pipelex.toml]")` raised out of the log call, and constructing a `FatalError` carrying such a message raised `MarkupError` **in place of** the failure it was reporting — the diagnostic destroyed and replaced by an unrelated exception. Fixed at both levels: the sink's handler restores the stdlib's guard, so a line Rich refuses costs at most its own rendering; and `TracebackMessageError` asks for its record verbatim through the per-record override Rich reads ahead of its handler's setting, so brackets and type names in an error reach the terminal as written. The mark is reserved in `log_fields` beside the forwarding marker, for the same two reasons: it is a console concern that rides no wire, and a caller's field spelling it must not decide how the console reads that line. Both arms of the error mode are marked — the `error` one this branch moved onto a propagating logger, and the `exception` one, which `origin/dev` already exposed.++**A guard the branch deleted is restored in a form that survives what retired it.** Two tests pinned that the gateway prune runs before the log hub is configured, by asserting that an unconfigured dispatch raised. A log call before `configure` is now handled rather than fatal, so they stopped failing for their own reason and were removed — while `drop_unknown_gateway_defaults` goes on stating the invariant in its docstring, "deliberately pure, and deliberately silent", with nothing holding it to it. The `defaults` guard now asserts the silence directly. Its per-model twin is deliberately **not** restored: nothing in that path's source claims silence, so pinning it would invent a contract rather than hold one, and a test asserting no record across a whole backend-library load would be brittle for the wrong reason.++Deferred, and filed where a trace alone would not be enough:++- **The doctor emits its own diagnostics through the configured sink** (L-260917-0be44e). With `sink = "otlp"` the doctor's stale-configuration warning is shipped to a collector and the terminal stays silent, while the Log Sink row reports healthy. Deferred because the remedy is contested *within this same round*: this finding wants the doctor to prefer a user-visible sink and merely report the configured one's resolvability, while cubic's standing finding wants it to duplicate less of `RuntimeBoot.setup` and reproduce the production path more faithfully. That tension is the useful part, and the narrower cure that needs neither is to carry the warning in the report rather than emit it through `log.*`.+- **The `json` sink writes ANSI escapes when a third-party library decorates a record** (L-260917-398eae, *unverified*: no uvicorn in this checkout, so the attribute name is the reviewer's own claim). `carried_attributes` makes any foreign decoration a field, and the module promises "no ANSI ever". It is the hosted plane's own configuration, so it is the case most likely to be seen. Deferred as a contract question: stripping escapes from every value is a scan on the hot path, a denylist of attribute names rots, and softening the promise gives it up — the reviewer offered all three and chose none.++Rejected, with the reason, so it is not re-raised as new:++- **`_render_content` discards a good rendering when its re-parse fails** (code-review). The claim is that a content whose purified rendering is not re-parseable falls to `repr` and loses a rendering that succeeded. That state cannot occur: `rendered` is `json.dumps` output of an already-purified structure, and the truncation the helpers apply runs on the structure before the dump rather than on its text, so the re-parse cannot fail where the dump succeeded. The `except` arm's `repr` fallback is reached only when `purify_json*` itself raised, which is the documented path and the correct outcome. The other half of the same finding — that the serialize-then-reparse round trip is paid on every structured call for a value the `console` sink discards — is a real cost and an improvement rather than a defect, so it is deferred at this bar and recorded here.+- **The holding handler drops the oldest record on overflow** (code-review). Offered as wrong for a failure-diagnosis buffer, on the argument that the earliest lines describe what first went wrong. The argument is not made out: the lines nearest a failure are at least as often the useful ones, the constant's own comment states the choice and its reason (bounded growth for a process that never installs a sink), and the same comment notes a boot holds a few dozen lines against a capacity of a thousand, so the path is not one a real boot reaches.++One thing noticed while verifying and left alone: `TracebackMessageError` in its `exception` mode calls `logger.exception` outside any `except` block, so a record built there appends `NoneType: None`. It predates the branch, `include_exception`'s own version of the same thing was fixed separately, and it is cosmetic noise rather than a loss.++## L-260916-aa6e58, round 3 after the growth reset (bar necessity)++All four reviewers ran over the whole branch against `origin/dev` at `2e81b2c68`, and the round changed no code. Nothing the bar admits was confirmed: the two findings Codex raised are standing deferrals already traced above, and the one new claim that would have been fixed if true turned out to be pre-existing.++**The processor seam came back for the fourth consecutive round, and this time from two reviewers.** `codex:adversarial` raised it at confidence 1.0 and `codex:review` at P2, both naming `pipelex/tools/log/log_sink.py:60-63` and both describing the mechanism the earlier rounds already verified: `_ProcessorFilter.filter` runs each callback where the stdlib runs a handler's filters, outside every `try`, so a processor raising on a live record raises out of `log.info` itself, while the same processor raising during the holding replay is contained by `_deliver`'s `handleError`. Nothing about the claim has changed and neither has the reason it stays here: the remedy is the scrub member L-260916-4837d4's contract choice rather than a review round's, because a redaction processor that raises means the record is *unredacted*, so rendering it anyway leaks exactly what it was meant to hide, while the class as written promises it never drops a record. No processor is registered anywhere in `pipelex/`, and the note on that member carries the finding. What is worth recording is the repetition itself: four rounds and two independent reviewers now agree it is real, so whoever merges this member should decide it deliberately rather than let it default through.++**The OTLP suppression guard was raised again, with the mechanism this document had already supplied.** `codex:review` named `pipelex/tools/log/otlp_log_sink.py:88-91` and reached PostHog by its own route — `BatchSpanProcessor` also sets `_SUPPRESS_INSTRUMENTATION_KEY`, so `PostHogSpanExporter._do_export`'s failure logs are rejected although they never came from this sink's export pipeline. That is the concrete loss round 2 recorded above and filed as L-260917-472a3e; the finding adds independent confirmation and no new ground.++**The one new claim the bar would have admitted was verified and refuted on attribution.** cubic raised at P1 that `configure()` overwrites the root level and every configured package logger's level while `reset()` restores none of them, so an embedder's logging stays changed after teardown. The mechanism is real and now carries a verified failing scenario, but it is not this branch's: `origin/dev` overwrites the same two families and additionally a third (the poor loggers), restores nothing either, and strips the host's own root handlers on top — so this branch strictly shrinks the residue. The new doctor `finally: log.reset()` sites do not widen the exposure, being guarded by `if not logging_was_configured`, which fires only where the doctor itself wrote those levels. Filed with the whole verification as L-260918-aee78b, together with the precedent that makes it cheap to specify: `reset_boot_state` already remembers and restores the pre-boot pretty-print mode.++Deferred, each new at this round and none of them admitted by `necessity`:++- **`pipelex doctor` can block for a minute or more on a machine configured for `otlp`** (*unverified*, code-review). `pipelex/tools/log/otlp_log_sink.py:158`: the doctor now installs the configured sink, both doctor commands reach `log.reset()` in a `finally`, and the flush is `force_flush(timeout_millis=5000)` whose own comment concedes the SDK discards that deadline — the real bound being `OTEL_EXPORTER_OTLP_TIMEOUT` plus the HTTP exporter's retry backoff. The case that stings is an operator running the doctor precisely because the collector is down. A doctor-specific short deadline, or skipping the flush when the doctor installed the sink itself, is the shape of the cure.+- **`stream_for_target`'s docstring promises a lazily-read stream that no caller reads lazily** (*unverified*, code-review). `pipelex/tools/log/log_sink.py:96` says the stream is "read at call time so a redirected stream is the one used", while `_make_json_log_sink` resolves it at boot and `ConsoleLogSink` binds it into `Console(file=...)` at handler build. An embedder swapping `sys.stderr` after `Pipelex.make()` gets records on the original stream. Either the docstring states build-time capture or the sinks resolve in `emit`; both are decisions rather than corrections, which is why it waits.+- **A `LogSink` instance is single-use and nothing says so** (*unverified*, code-review). `pipelex/tools/log/log_sink.py:82` caches the handler forever while `Log.reset()` closes it, so re-installing the same sink instance would hand back a closed handler, insert a second `ForwardedRecordFilter` into its filters and attach a dead handler to the root logger. No in-tree caller does this — boot and the doctor each build a fresh sink — but `LogSink` is the documented extension point, so either a `close()` hook clearing `_handler` or a line in the base class's docstring closes it.+- **A sink that fails to install is dropped with its thread still running** (*unverified*, code-review). `pipelex/cli/commands/doctor_cmd.py:1129`: for `sink = "otlp"` the factory constructs a `LoggerProvider` and a `BatchLogRecordProcessor` — an exporter thread and an `atexit` hook — before `log.install_sink` is reached, so an install that raises leaves them live while the `except` installs the console fallback and the later `reset()` tears down only that fallback. The window is narrow today because `OtlpLogHandler.__init__` is hard to make fail, but the shape applies to any plugin-contributed sink whose factory allocates.++Rejected, with the reason:++- **The holding handler drops the oldest record on overflow** (code-review), offered again and rejected again on the argument recorded under round 2 above: the lines nearest a failure are at least as often the useful ones, the constant's own comment states the choice and its reason, and a boot holds a few dozen lines against a capacity of a thousand. Nothing new was offered with it.+- **Caller attribution rides a fixed frame depth** (cubic), `pipelex/tools/log/log_dispatch.py:33`, already traced under the fields member's round 1 and belonging to that member's code rather than this one's. No wrapper exists today, so the claim is about a change nobody has made.++## L-260916-aa6e58, the processor seam, settled by the owner on 2026-09-18++Four consecutive rounds on this member raised the same finding, the last of them from two independent reviewers at confidence 1.0, and each round deferred it with a reasoned rationale rather than a shrug. Louis settled it instead of letting a fifth round take it: **the redaction member `L-260916-4837d4` owns the processor failure policy, and this member ships the seam as it stands.**++The defect is not in dispute and never was. `_ProcessorFilter.filter` runs each registered processor where the stdlib runs a handler's filters, which is outside every `try`, so a processor raising on a live record raises out of `log.info` itself and can abort the application work that made the call; the same processor raising during the holding replay is contained by `_deliver`'s `handleError`, so the two paths disagree about the same failure. Nothing in `pipelex/` registers a processor today, which is what makes shipping possible, but `docs/under-the-hood/log-sink-plugins.md` tells readers to append a redaction one, so the path is reachable for anyone following the documentation.++The reason it belongs there rather than here is that the cure is a contract choice and not a guard. A redaction processor that raises means the record is *unredacted*: rendering it anyway leaks exactly what the processor existed to hide, while dropping it contradicts `LogSink`'s promise that it never drops a record. Only the first registrant of a processor is in a position to take that decision, and it arrives with that member. What it owes is recorded on its item: decide the policy and say why, with fail-closed — drop the record and emit a minimal diagnostic carrying none of the unprocessed payload — as the shape the reviewers converged on; implement the guard at the processor boundary, routed to the handler's error path; and cover both the live and replay paths so they can no longer disagree.++A later round that re-raises this should be closed against this entry rather than argued again.