Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Console-log-fields · L-260927-a9d922 (#1388)

Louis Choquel19f505f59dd8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 84bff2f9e..2f42120ba 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
## [Unreleased]
+### Added
+
+- **The console shows a log call's fields and renders named layouts**: the `console` sink prints a record's fields after its message as a `key=value` suffix, `Scanned the inputs files=7`, each value on one line and cut short when long, leaving out the run identifiers, and colours a value by its field's name, a pipe code red and a concept bold green, with any other field dimmed. A call can name a console layout with `layout=`, a Rich template over its fields in which every value is escaped, starting with `LogLayout.PIPE_RUN` for the pipe-run tree; every other sink writes the plain message and the fields.
+
### Changed
- **MTHDS standard 4.0.0 (Breaking)**: pipelex now requires `mthds==0.20.0` and `pipelex-sdk==0.37.0`, the releases that report MTHDS standard `4.0.0`, the language this runtime has implemented since v0.75.0, so everyone downstream inherits those exact versions and a project that pins either package itself has to move with them. `pipelex --version` prints `mthds-standard 4.0.0`, every crate pipelex normalizes is stamped with `mthds_version` `4.0.0` while crate fingerprints stay the same, and a package whose `METHODS.toml` constrains `mthds_version` to `^3.0.0`, `~3.0.0` or exactly `3.0.0` now logs a warning when it loads, which `>=3.0.0` or `^4.0.0` does not.
diff --git a/docs/api-server/logging.md b/docs/api-server/logging.md
index 3e11aceed..290fcc252 100644
--- a/docs/api-server/logging.md
+++ b/docs/api-server/logging.md
@@ -53,7 +53,7 @@ A key whose value is not set for this request is **absent** from the line rather
`detail` is the one field whose absence follows the failure's origin rather than the request's shape, and it is worth knowing which way round. A failure this API authored itself — a validation error, an unknown model category — carries `detail` on the record. A failure that arrives as a Pipelex `ErrorReport`, which is most `5xx` and every domain error, does not: the response body still carries a `detail`, but the record does not, because the body's text has been through disclosure redaction and the cause has not. So a `4xx` from that path logs as `API error 422: SomeError` with no explanation and no traceback, and the response is where the explanation is. Build an operator query on `error_type` and `route`, which every line carries, rather than on `detail`.
-Every field in that table is a **record attribute**, which is a different thing from the message. A structured sink — `json` here, and the OTLP sink — writes them beside the message as keys. The Rich console sink renders the message alone and ignores them entirely, so a deployment that switches the sink to `"console"` does not get a more readable version of these lines: it gets `API error 500: PipelexConfigError` and nothing else. That is a reason to keep `sink = "json"` wherever the lines are being read by anything other than a person at a terminal.
+Every field in that table is a **record attribute**, which is a different thing from the message. A structured sink — `json` here, and the OTLP sink — writes them beside the message as keys. The Rich console sink shows them after the message as a `key=value` suffix, each value on one line and cut short when long, and leaves out the run identifiers (`request_id`, `pipeline_run_id`), which a person at a terminal does not need on every line, so `API error 500: PipelexConfigError` is followed by the route, the status and the rest of the fields above. That rendering is for a person reading a terminal; keep `sink = "json"` wherever the lines are read by anything else.
Disposition follows the HTTP status, not the error domain: a `4xx` is a caller mistake and logs at `WARNING` without a traceback; a `5xx` is a server fault and logs at `ERROR` with one. See [Error Responses](error-responses.md) for the response side of the same failure.
diff --git a/docs/configuration/config-practical/logging-config.md b/docs/configuration/config-practical/logging-config.md
index f1397aa85..36ec6b242 100644
--- a/docs/configuration/config-practical/logging-config.md
+++ b/docs/configuration/config-practical/logging-config.md
@@ -150,7 +150,7 @@ silenced_problem_ids = ["azure_openai_no_stream_options"]
## The `console` Sink
-Configuration section: `[runtime.log.rich_log]`, read only when `sink = "console"`. The sink writes to `console_log_target`.
+Configuration section: `[runtime.log.rich_log]`, read only when `sink = "console"`. The sink writes to `console_log_target`. It shows a record's fields after its message, coloured by name, and renders the layouts a call names; neither has a setting (see [Console rendering](../../tools/logging.md#console-rendering)).
### Display Options
diff --git a/docs/tools/logging.md b/docs/tools/logging.md
index 2cc4838c2..83cbc4df9 100644
--- a/docs/tools/logging.md
+++ b/docs/tools/logging.md
@@ -7,7 +7,7 @@ description: "Explore Pipelex logging: named fields, the run-scoped context, mod
## Overview
-Pipelex logs through one facade, `from pipelex import log`, built on Python's standard `logging`. A call takes a message, optional named fields and the usual presentation options; the run-scoped identifiers are bound once at a process entry and stamped onto every record emitted in scope. Fields and identifiers ride the stdlib `LogRecord` as attributes, never spliced into the message text, so a structured sink renders them as fields while the console keeps a narrative line. Where the records go is one config key, `sink` under `[runtime.log]`: `console` renders them through Rich, `json` writes one JSON object per line, `otlp` ships them to an OpenTelemetry collector, and a plugin can register another. The keys are in [Logging Configuration](../configuration/config-practical/logging-config.md) and the seam itself in [Log Sink Plugins](../under-the-hood/log-sink-plugins.md).
+Pipelex logs through one facade, `from pipelex import log`, built on Python's standard `logging`. A call takes a message, optional named fields and the usual presentation options; the run-scoped identifiers are bound once at a process entry and stamped onto every record emitted in scope. Fields and identifiers ride the stdlib `LogRecord` as attributes, never spliced into the message text, so a structured sink writes them as keys while the console shows the fields after the message, coloured by name. Where the records go is one config key, `sink` under `[runtime.log]`: `console` renders them through Rich, `json` writes one JSON object per line, `otlp` ships them to an OpenTelemetry collector, and a plugin can register another. The keys are in [Logging Configuration](../configuration/config-practical/logging-config.md) and the seam itself in [Log Sink Plugins](../under-the-hood/log-sink-plugins.md).
## Log Levels
@@ -35,6 +35,15 @@ log.info("Simple message")
# Named fields ride the record as attributes; the message stays narrative
log.info("Scanned the inputs", fields={"files": 7, "bytes": 12_288})
+# A named console layout; every sink but the console writes the plain message and the fields
+from pipelex.tools.log.console_layouts import LogLayout
+
+log.info(
+ "Pipe run starts",
+ fields={"pipe_type": "PipeCompose", "pipe_code": "compose_company", "output_concept": "Company", "pipe_depth": 1, "is_dry_run": False},
+ layout=LogLayout.PIPE_RUN,
+)
+
# Logging with title
log.info("Detailed message", title="Process Status")
@@ -58,13 +67,13 @@ log.dev("Testing new feature")
log.verbose("Detailed debug information")
```
-Every one of the seven methods (`verbose`, `debug`, `dev`, `info`, `warning`, `error`, `critical`) takes the same keyword-only `fields`. `title`, `inline`, `problem_id` and `include_exception` keep their meaning beside it. `include_exception=True` carries the exception being handled as the record's `exc_info`, with nothing spliced into the message: the `console` sink renders the traceback under the line, the `json` sink writes it under the `exception` key and the `otlp` sink under the `exception.*` attributes. Outside an `except` block it carries nothing.
+Every one of the seven methods (`verbose`, `debug`, `dev`, `info`, `warning`, `error`, `critical`) takes the same keyword-only `fields` and `layout`, the second naming a [console layout](#layouts). `title`, `inline`, `problem_id` and `include_exception` keep their meaning beside them. `include_exception=True` carries the exception being handled as the record's `exc_info`, with nothing spliced into the message: the `console` sink renders the traceback under the line, the `json` sink writes it under the `exception` key and the `otlp` sink under the `exception.*` attributes. Outside an `except` block it carries nothing.
## Fields
`fields` is a mapping of named values. Each entry becomes an attribute of the emitted `LogRecord`, which is where a formatter or a sink reads it: `record.files` for the example above, or `%(files)s` in a stdlib format string. Nothing from `fields` is written into the message, so `record.getMessage()` is exactly the text you passed.
-A value can be anything. It rides the record by reference and the sink serializes it on emit, on the calling thread, so what leaves the process is the value as it was at the call, and mutating it afterwards changes nothing already emitted. The `console` sink ignores it, the `json` sink dumps a pydantic model in JSON mode and falls back to `str` for a value JSON does not know, and the `otlp` sink carries a scalar or a sequence of scalars as an attribute and anything else as JSON text; prefer plain JSON-ready values (strings, numbers, booleans, lists and dictionaries of those) for anything meant to be queried later. A value JSON refuses outright, a circular reference or a mapping with a non-string key, is written by the `json` sink as its `repr`, and that costs the value alone: every other field on the line keeps its JSON type, so a number or a boolean beside it is still queried as one.
+A value can be anything. It rides the record by reference and the sink serializes it on emit, on the calling thread, so what leaves the process is the value as it was at the call, and mutating it afterwards changes nothing already emitted. The `console` sink shows it after the message on one line, as [Fields after the message](#fields-after-the-message) describes, the `json` sink dumps a pydantic model in JSON mode and falls back to `str` for a value JSON does not know, and the `otlp` sink carries a scalar or a sequence of scalars as an attribute and anything else as JSON text; prefer plain JSON-ready values (strings, numbers, booleans, lists and dictionaries of those) for anything meant to be queried later. A value JSON refuses outright, a circular reference or a mapping with a non-string key, is written by the `json` sink as its `repr`, and that costs the value alone: every other field on the line keeps its JSON type, so a number or a boolean beside it is still queried as one.
### Naming convention
@@ -76,7 +85,9 @@ A value can be anything. It rides the record by reference and the sink serialize
The stdlib refuses an `extra` key that would overwrite one of the record's own attributes (`name`, `message`, `lineno`, `module`, `args`, `asctime` and the rest), and a library's log call never raises. An entry of such a name is therefore carried under the prefix `field_`: `fields={"name": "alpha"}` lands as `record.field_name`. What counts as owned is read off the record actually built, through whatever record factory is installed, so an attribute an OpenTelemetry or tracing instrumentation stamps on every record is a collision too, for a field, a context identifier and `data` alike. The prefix is applied until the name lands on an attribute nobody owns, and entries attach in order, so a call that gives both `name` and `field_name` keeps both values whatever their order: the one that arrives second lands on `field_field_name`.
-Two kinds of name are reserved though nothing on a fresh record owns them yet, because both are stamped after the entries are attached and the stdlib's refusal therefore cannot cover them: what the formatter sets (`message`, `asctime`) and what Pipelex's own logging machinery sets — `_pipelex_forwarded`, the marker that tells the sink's handler a record reached it through the boot's holding handler already. Both take the same `field_` prefix, and the marker is never handed to a sink as something the record carries. Without that reservation, `fields={"_pipelex_forwarded": True}` would have the sink's own filter read the record as one already delivered and drop it whole.
+Two kinds of name are reserved though nothing on a fresh record owns them yet, because both are stamped after the entries are attached and the stdlib's refusal therefore cannot cover them: what the formatter sets (`message`, `asctime`) and what Pipelex's own logging machinery sets — `_pipelex_forwarded`, the marker that tells the sink's handler a record reached it through the boot's holding handler already, `_pipelex_field_names`, the names the call's entries landed on, which is what the console renders, and `_pipelex_layout`, the [layout](#layouts) a call named. Both kinds take the same `field_` prefix, and no mark is ever handed to a sink as something the record carries. Without that reservation, `fields={"_pipelex_forwarded": True}` would have the sink's own filter read the record as one already delivered and drop it whole, and `fields={"_pipelex_layout": "pipe_run"}` would pick the line's layout.
+
+The names Rich's console handler reads off each record ahead of its own settings, `markup` and `highlighter`, are reserved the same way and for the same reason: a field landing on `markup` would decide whether the console reads that line's markup, and one landing on `highlighter` would be called in the highlighter's place, so `fields={"highlighter": "pygments"}` would raise inside the handler and lose the whole line. Such a field rides under `field_markup` or `field_highlighter`, and neither name is handed to a sink as something the record carries.
## The run-scoped context
@@ -172,7 +183,7 @@ In place, because what the processor does is *remove* something. A record that h
`log.install_sink` is what puts the processor in front of the sink's own, so every sink gets it — the built-in ones, an out-of-tree one, and the console sink `pipelex doctor` falls back to — and no sink knows about it. A boot that dies before its sink arrives runs the same processor, behind the same guard, over every record it held before handing them to the stdlib's last resort on stderr, so the trail a failed boot leaves is scrubbed too. `[runtime.log.redaction] is_enabled = false` installs nothing at all.
-The processor fails closed. A sink processor that raises is reported on stderr and the record is handed on, which is the right rule for a processor that enriches; for one that removes, it would mean the secret ships. So when the scrub itself fails, a value nested past the interpreter's recursion limit for one, the record is stripped before the failure is reported: its message becomes `[REDACTION FAILED: <exception type>]`, every field and the `data` attribute become `[REDACTED]`, and the exception text is dropped. The line says the scrub failed, and nothing of the call leaves with it.
+The processor fails closed. A sink processor that raises is reported on stderr and the record is handed on, which is the right rule for a processor that enriches; for one that removes, it would mean the secret ships. So when the scrub itself fails, a value nested past the interpreter's recursion limit for one, the record is stripped before the failure is reported: its message becomes `[REDACTION FAILED: <exception type>]`, every field and the `data` attribute become `[REDACTED]`, the exception text is dropped, and so is any [console layout](#layouts) the call named, which would otherwise draw the redacted fields in place of the notice. The line says the scrub failed, and nothing of the call leaves with it.
The stripping can fail in its turn, and that case is closed too. The commonest reason the scrub fails is a stack that has run out — a `log.<level>(...)` call made a few frames from the limit, by a recursive walker or by an `except RecursionError:` handler — and a stack that could not take the scrub cannot always take the stripping of what it left behind a line later. So the record is marked before the stripping starts, by a plain assignment into its own dictionary that pushes no frame, and unmarked only once the stripping has gone all the way through; a record still carrying the mark when the processors are done is **dropped** rather than emitted. A notice saying the record was quarantined while it still carried the field the scrub never read would be worse than no line at all.
@@ -203,6 +214,61 @@ Logging is configured in two steps at boot. `log.configure` sets the levels and
The default sink, `console`, renders through Rich with every `[runtime.log.rich_log]` setting. Rich is the `cli` extra (`pipelex[cli]`), which the command-line tools install. The sink imports Rich when it is built, so a process that selects `json` or `otlp` never loads it through the sink, and one that selects `console` without Rich installed stops at boot naming the extra to install and the `json` alternative. That refusal is not what keeps a server off the console: `typer` and `instructor`, both core dependencies, require Rich, so it is installed even where the extra is not, and a server that forgot to select `json` boots on the `console` sink. A server therefore selects `json` in its own configuration, and sets `pretty_print_mode` to `"poor"` or `"silent"` beside it, since the boot makes neither choice for it (see [Pretty-Print Mode](../configuration/config-practical/logging-config.md#pretty-print-mode)).
+### Fields after the message
+
+The console shows the fields a call gave after its message, as a `key=value` suffix, the way structlog's console renderer and pino-pretty do:
+
+```python
+log.info("Scanned the inputs", fields={"files": 7, "source": "inbox/march"})
+```
+
+```text
+INFO 🧠: Scanned the inputs files=7 source=inbox/march
+```
+
+- **What is shown.** The fields the call attached, in the order it gave them, under the name each landed on (`field_name` for a field called `name`). The run identifiers (`request_id`, `pipeline_run_id`, `pipe_run_id`) are left out, since they are the same on every line of a run and would drown the message, and so is `data`, the structured content the message already renders. Nothing else on the record is shown: neither the stdlib's own attributes, nor Pipelex's marks, nor what a record factory or a third-party library stamped on it.
+- **How a value is written.** On one line: a string bare, or quoted when it is empty or holds a space, an `=`, a `"`, a `\` or a character a terminal would act on. Inside the quotes a backslash is written `\\`, a quote `\"` and any character a terminal would act on as its escape (`"line\nbreak"`), so `fields={"a": "b=c"}` prints `a="b=c"` rather than a second pair. A number, a boolean, `None`, a mapping, a list or a pydantic model is written as compact JSON (`true`, `null`, `{"key":"value"}`, `["a","b"]`), and anything else as its text. A value longer than 80 characters is cut short and ends with `…`. A key is written the same way, so a field whose name holds a line break, a space, an `=` or an escape sequence is quoted and escaped rather than forging a line or a second pair: `fields={"x=1": 2}` prints `"x=1"=2`. The [redaction](#redaction) has run before the console renders, so a secret is already `[REDACTED]`.
+- **Never markup.** The suffix is built as styled Rich `Text`, not as a markup string, so a value carrying `[red]x[/red]` prints exactly that, whatever `is_markup_enabled` says about the message.
+
+There is no setting to hide the suffix: it carries the values a message used to interpolate, so hiding it would hide what the line is about.
+
+### The style map
+
+A field's value is coloured by the field's name, wherever it appears, from one table in `pipelex/tools/log/console_fields.py` (`FIELD_STYLES`): a pipe code red, a concept (`output_concept`, `concept_ref`, `concept_code`) bold green, a pipe type white, a stuff name cyan and a domain bold magenta, the colours the pipe announcement and the stuff renderings have always used. A field outside the map, and every key, is dimmed. The colour follows the name the field was given, so a `pipe_code` that landed on `field_pipe_code` because a record factory owns `pipe_code` is still red. So a call is coloured by naming its fields, never by writing markup into its message. The map lives in code, with no configuration key; extending it is a one-line change beside the sink.
+
+### Layouts
+
+A few lines have a shape that matters on a terminal, the pipe-run tree above all. For those, a call names a layout, a Rich template over the record's fields registered in `pipelex/tools/log/console_layouts.py`:
+
+```python
+from pipelex.tools.log.console_layouts import LogLayout
+
+log.info(
+ "Pipe run starts",
+ fields={"pipe_type": "PipeCompose", "pipe_code": "compose_company", "output_concept": "Company", "pipe_depth": 1, "is_dry_run": False},
+ layout=LogLayout.PIPE_RUN,
+)
+```
+
+```text
+INFO 🧠: ↳ PipeCompose: compose_company → Company
+```
+
+- **The console renders the layout in place of the message**, after the logger's emoji, and every other sink ignores it: the `json`, `otlp` and `gcp` sinks write the plain message, `Pipe run starts`, and the fields, and never the layout's name, which rides the record under a reserved mark. The name is explicit at the call, so rewording the message cannot lose the layout, and it is a `LogLayout` member, so a misspelt one is a type error.
+- **Every value is escaped** with Rich's own escape before it is substituted, the values the layout derives included, so a field carrying `[red]x[/red]` prints literally. A value is written on one line and cut short past 80 characters, as in the suffix, except that a string is never quoted.
+- **The fields a layout presents are not repeated after it**; any other field the call gave still follows as the suffix, and the run identifiers stay hidden.
+- **A layout that cannot be filled costs nothing but itself.** A field it needs is missing, a value its derivation refuses, or anything else the layout raises: the console falls back to the message and shows every field after it.
+- **A layout is for a string message.** A call whose content is anything else, a mapping, a list or a model, keeps its layout off the record, and the console renders its message: the message is the only place that content renders, since the suffix never repeats `data`, and a content JSON refuses, a circular one for instance, lives only in the message's `repr`. A record whose [redaction](#redaction) failed loses its layout too, so the notice saying the scrub failed is what prints.
+- **A layout replaces the whole message.** A line rendered through a layout shows neither the title nor the inline text the call gave, nor any [caller information](#caller-information), all of which belong to the message it replaces. A traceback the record carries still prints under the line, as it does under a message, and the fields stay on the line above it, whether Rich renders the traceback or `is_rich_tracebacks = false` has it printed as plain text.
+
+The registry holds these layouts:
+
+| Layout | Fields it presents | What it draws |
+| --- | --- | --- |
+| `LogLayout.PIPE_RUN` | `pipe_type`, `pipe_code`, `output_concept`, `pipe_depth` (an integer, `0` at the top level and at most `100`), `is_dry_run` (a boolean) | `PipeCompose: compose_company → Company`, indented three spaces per level of depth and behind `↳` when nested, with `Dry run:` before the pipe type for a dry run, in the style map's colours. A depth or a flag of another type, or a depth out of range, falls back to the message |
+
+A layout is a `ConsoleLayout`: a `template` of Rich markup whose placeholders are bare field names (`{pipe_code}`, never an attribute, an index or a format spec, which registration refuses), the `presented_fields` the suffix leaves out, and, for a shape a template cannot express alone, a subclass whose `derived_values` computes presentation values such as an indentation from the fields.
+
### Rich Formatting
- Color coding and syntax highlighting for JSON and other data structures
@@ -242,7 +308,7 @@ Optional inclusion of caller information in logs, prefixed to the console line:
2. **Fields over interpolation**:
- - Put a value a reader might filter or aggregate on in `fields`, and keep the message a sentence
+ - Put a value a reader might filter or aggregate on in `fields`, and keep the message a sentence; the console shows it after the message, coloured by its name
- Name a field after the thing it carries, in `snake_case`, or after its OpenTelemetry key when one exists
- Leave the run identifiers to the context; bind them at the process entry, never per call
diff --git a/pipelex/tools/log/console_fields.py b/pipelex/tools/log/console_fields.py
new file mode 100644
index 000000000..832962e2a
--- /dev/null
+++ b/pipelex/tools/log/console_fields.py
@@ -0,0 +1,179 @@
+"""What the ``console`` sink shows of a record's fields: the suffix after the message and the style map.
+
+The console renders the fields a call attached as a ``key=value`` suffix after the message, the way
+structlog's console renderer and pino-pretty do, so a value moved out of a message into ``fields=`` still
+reaches the person at the terminal. It renders the fields ``attach_log_record_extra`` recorded and no other
+attribute, so neither the stdlib's own attributes nor Pipelex's marks nor what a record factory or a
+third-party library stamped ever shows. The run identifiers and ``data`` are left out as well: the
+identifiers are the same on every line of a run and would drown the message, and ``data`` is the structured
+content the message already renders, which is why the dispatch never stamps a layout on a call carrying it.
+
+A value renders on one line: a string bare unless it is empty or holds a space, an equals sign, a quote, a
+backslash or a character a terminal would act on, in which case it is quoted, with a backslash and a quote
+escaped by a backslash and that character written as its escape; anything else as compact JSON; and the
+whole cut short past ``FIELD_VALUE_MAX_LENGTH``. A key is written the same way, so a field name holding a
+line break, a space, an equals sign or an escape sequence can forge neither a line nor a pair. Redaction has
+already run when the console renders a record, so what is rendered is what the scrub left.
+
+Colour follows the name the field was given, from ``FIELD_STYLES``, wherever the field appears, and whatever
+collision prefix it landed under; a field outside the map renders dimmed. The map is one table in code, the
+colours the pipe announcement has always used. Nothing here imports Rich: the suffix is a list of
+``(text, style)`` segments the sink turns into a Rich ``Text``, never a markup string, so a value carrying
+``[red]`` prints as written whatever the handler's markup setting.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+from collections.abc import Mapping
+from typing import TYPE_CHECKING, Any
+
+from pydantic import BaseModel
+
+from pipelex.tools.log.log_context import PIPE_RUN_ID_FIELD, PIPELINE_RUN_ID_FIELD, REQUEST_ID_FIELD
+from pipelex.tools.log.log_fields import DATA_FIELD, attached_field_names, given_field_name
+from pipelex.tools.log.log_sink import json_fallback, spell_non_finite
+
+if TYPE_CHECKING:
+ import logging
+
+# The style of a field's value, by the field's name. A concept is bold green, a pipe code red, a pipe type
+# white, a stuff name cyan and a domain bold magenta: the colours the pipe announcement and the stuff
+# renderings have always used, so a value keeps its colour when it moves from a message into a field.
+FIELD_STYLES: dict[str, str] = {
+ "pipe_code": "red",
+ "pipe_type": "white",
+ "output_concept": "bold green",
+ "concept_ref": "bold green",
+ "concept_code": "bold green",
+ "stuff_name": "cyan",
+ "domain_code": "bold magenta",
+}
+
+# The style of a value whose field is not in the map, and of every key.
+UNMAPPED_FIELD_STYLE = "dim"
+FIELD_KEY_STYLE = "dim"
+
+# What the console never repeats after a message: the run identifiers, bound once for a whole run, and
+# ``data``, the structured content the message already renders. The dispatch stamps a layout only on a
+# call whose content is a string, so a layout never hides the message that renders that content.
+CONSOLE_HIDDEN_FIELDS = frozenset({REQUEST_ID_FIELD, PIPELINE_RUN_ID_FIELD, PIPE_RUN_ID_FIELD, DATA_FIELD})
+
+# The longest a rendered value gets, the truncation mark included.
+FIELD_VALUE_MAX_LENGTH = 80
+TRUNCATION_MARK = "…"
+
+FIELD_SEPARATOR = " "
+KEY_VALUE_SEPARATOR = "="
+QUOTE = '"'
+ESCAPE = "\\"
+
+# What makes a printable string quoted rather than bare: the empty string aside, a character that would
+# otherwise read as the end of a pair, the start of the next or an escape, so the suffix cannot be forged.
+QUOTED_CHARACTERS = frozenset({FIELD_SEPARATOR, KEY_VALUE_SEPARATOR, QUOTE, ESCAPE})
+
+# A segment of the suffix: its text, and the Rich style it is printed in.
+StyledSegment = tuple[str, str]
+
+
+def attached_fields(*, record: logging.LogRecord) -> dict[str, Any]:
+ """Everything the call attached to the record, under the names it landed on and in the order it was attached.
+
+ The fields, the context identifiers and ``data`` alike: what a layout may read. The suffix leaves some of
+ them out, which ``field_suffix_segments`` decides.
+ """
+ fields: dict[str, Any] = {}
+ for name in attached_field_names(record=record):
+ if name in record.__dict__:
+ fields[name] = record.__dict__[name]
+ return fields
+
+
+def field_style(*, name: str) -> str:
+ """The style a field's value is printed in: the entry in ``FIELD_STYLES`` for the name it was given, else dimmed.
+
+ Read off the name the caller gave rather than the one it landed on, so a ``pipe_code`` that a record
+ factory pushed to ``field_pipe_code`` keeps its colour.
+ """
+ return FIELD_STYLES.get(given_field_name(name=name), UNMAPPED_FIELD_STYLE)
+
+
+def field_suffix_segments(*, fields: Mapping[str, Any], presented_fields: frozenset[str]) -> list[StyledSegment]:
+ """The suffix for these fields, a space before each ``key=value``, as styled segments; none when nothing is left to show.
+
+ The run identifiers, ``data`` and the fields a layout already presented are left out.
+ """
+ segments: list[StyledSegment] = []
+ for name, value in fields.items():
+ if name in CONSOLE_HIDDEN_FIELDS or name in presented_fields:
+ continue
+ segments.append((FIELD_SEPARATOR, ""))
+ # The key is written like a value: a name a caller chose can hold a line break that would forge a
+ # line, a space that would read as two pairs or an escape sequence the terminal would act on.
+ segments.append((f"{format_field_value(value=name)}{KEY_VALUE_SEPARATOR}", FIELD_KEY_STYLE))
+ segments.append((format_field_value(value=value), field_style(name=name)))
+ return segments
+
+
+def format_field_value(*, value: Any) -> str:
+ """A field's value as the suffix prints it: one line, a string quoted when it could forge a pair, cut short when long."""
+ return _truncated(text=_one_line(value=value, is_quoting_strings=True))
+
+
+def format_layout_value(*, value: Any) -> str:
+ """A field's value as a layout substitutes it: one line, a string as itself, cut short when long."""
+ return _truncated(text=_one_line(value=value, is_quoting_strings=False))
+
+
+def one_line_text(*, text: str) -> str:
+ """The text with every character a terminal would act on written as its escape, so it stays on one line."""
+ return "".join(character if character.isprintable() else _escaped_character(character=character) for character in text)
+
+
+def _one_line(*, value: Any, is_quoting_strings: bool) -> str:
+ if isinstance(value, str):
+ return _string_text(text=value, is_quoting_strings=is_quoting_strings)
+ if isinstance(value, float) and not math.isfinite(value):
+ # ``NaN``, ``Infinity`` or ``-Infinity``, bare, as the wire sinks spell it.
+ return str(spell_non_finite(value=value))
+ if _renders_as_json(value=value):
+ return one_line_text(text=_compact_json(value=value))
+ return _string_text(text=str(value), is_quoting_strings=is_quoting_strings)
+
+
+def _renders_as_json(*, value: Any) -> bool:
+ """Whether a value renders as compact JSON: ``None``, a boolean, a number, a mapping, a list, a tuple or a model."""
+ return value is None or isinstance(value, (bool, int, float, Mapping, list, tuple, BaseModel))
+
+
+def _string_text(*, text: str, is_quoting_strings: bool) -> str:
+ if not is_quoting_strings:
+ return one_line_text(text=text)
+ if text and text.isprintable() and QUOTED_CHARACTERS.isdisjoint(text):
+ return text
+ # The backslashes first, so the ones escaping a quote, and the ones the terminal escapes bring, are
+ # never doubled: a backslash in the text reads ``\\`` and a quote ``\"``, and neither ends the value.
+ escaped = one_line_text(text=text.replace(ESCAPE, f"{ESCAPE}{ESCAPE}").replace(QUOTE, f"{ESCAPE}{QUOTE}"))
+ return f"{QUOTE}{escaped}{QUOTE}"
+
+
+def _compact_json(*, value: object) -> str:
+ """Compact JSON, a model in JSON mode, an unknown object as its text and a non-finite float as its spelling.
+
+ What ``json`` refuses outright, a circular reference or a mapping with a non-string key, renders as its ``repr``.
+ """
+ try:
+ return json.dumps(spell_non_finite(value=value), ensure_ascii=False, allow_nan=False, separators=(",", ":"), default=json_fallback)
+ except (TypeError, ValueError):
+ return repr(value)
+
+
+def _escaped_character(*, character: str) -> str:
+ return character.encode("unicode_escape").decode("ascii")
+
+
+def _truncated(*, text: str) -> str:
+ if len(text) <= FIELD_VALUE_MAX_LENGTH:
+ return text
+ return f"{text[: FIELD_VALUE_MAX_LENGTH - len(TRUNCATION_MARK)]}{TRUNCATION_MARK}"
diff --git a/pipelex/tools/log/console_layouts.py b/pipelex/tools/log/console_layouts.py
new file mode 100644
index 000000000..2de52291a
--- /dev/null
+++ b/pipelex/tools/log/console_layouts.py
@@ -0,0 +1,187 @@
+"""The console layouts: Rich templates over a record's fields, for the few lines whose shape matters on a terminal.
+
+A call names its layout with ``layout=`` (``log.info("Pipe run starts", fields={...}, layout=LogLayout.PIPE_RUN)``),
+which the dispatch stamps on the record under ``LAYOUT_MARK`` when the call's content is a string, and drops
+for any other content, which only the message renders. The ``console`` sink renders such a record
+through the layout's template in place of its message, and every other sink ignores the layout and writes
+the plain message and the fields: the mark is reserved, so it reaches no wire. The name is explicit at the
+call, so rewording the message can never silently lose the layout.
+
+A template is Rich markup with ``{name}`` placeholders. Each placeholder is a field of the record, or a value
+the layout derives from the fields, such as an indentation computed from a depth. Every value is rendered on
+one line and escaped with Rich's own escape before it is substituted, the derived ones included, so a value
+can never be read as markup: a field carrying ``[red]`` prints as written. The fields a layout presents are
+left out of the suffix that follows it; any other field the call gave still renders there. A layout whose
+fields are missing, whose derivation refuses a value or whose template Rich refuses, costs nothing but
+itself: whatever it raised, the console falls back to the message and the suffix. A traceback the record
+carries prints under the line either way, and a record the redaction quarantined loses its layout, so the
+notice saying why its fields are redacted is what prints.
+
+The registry is one table in code. Rich is imported only where a layout is rendered, after asking for it.
+"""
+
+from __future__ import annotations
+
+import string
+from enum import StrEnum
+from typing import TYPE_CHECKING, Any
+
+from pydantic import BaseModel, ConfigDict, field_validator
+from typing_extensions import override
+
+from pipelex.tools.log.console_fields import FIELD_STYLES, format_layout_value, one_line_text
+from pipelex.tools.misc.rich_extra import require_rich
+
+if TYPE_CHECKING:
+ from collections.abc import Mapping
+
+#: What a layout says when the extra is missing; only the console sink renders one, and it asked already.
+CONSOLE_LAYOUT_MISSING_MESSAGE = "A console layout renders through Rich."
+
+
+class LogLayout(StrEnum):
+ """The layouts a log call can name with ``layout=``."""
+
+ PIPE_RUN = "pipe_run"
+
+
+class ConsoleLayout(BaseModel):
+ """A Rich template over a record's fields, and the fields it presents.
+
+ ``template`` is Rich markup with ``{name}`` placeholders, and nothing but a bare name in a placeholder.
+ ``presented_fields`` are the record's fields the layout shows, which the suffix after it does not
+ repeat; they are the placeholders that name a field and those ``derived_values`` reads. A subclass
+ overrides ``derived_values`` to compute presentation values from the fields, which are escaped like the
+ fields themselves.
+ """
+
+ model_config = ConfigDict(frozen=True)
+
+ template: str
+ presented_fields: frozenset[str]
+
+ @field_validator("template")
+ @classmethod
+ def validate_template(cls, template: str) -> str:
+ for _, placeholder, format_spec, conversion in string.Formatter().parse(template):
+ if placeholder is None:
+ continue
+ if not placeholder.isidentifier() or format_spec or conversion:
+ msg = f"A console layout's placeholder must be a bare name, with no attribute, index, format spec or conversion: '{{{placeholder}}}'"
+ raise ValueError(msg)
+ return template
+
+ @property
+ def placeholders(self) -> tuple[str, ...]:
+ """The names the template substitutes, in order of first appearance."""
+ names: dict[str, None] = {}
+ for _, placeholder, _, _ in string.Formatter().parse(self.template):
+ if placeholder is not None:
+ names[placeholder] = None
+ return tuple(names)
+
+ def derived_values(self, *, fields: Mapping[str, Any]) -> dict[str, Any]:
+ """Presentation values the template substitutes beside the fields. None by default."""
+ del fields
+ return {}
+
+ def render_markup(self, *, fields: Mapping[str, Any]) -> str:
+ """The template filled with the fields and the derived values, each on one line and escaped.
+
+ A derivation is each layout's own code, so the errors below are what a well-behaved one raises rather
+ than all it can: the console sink falls back to the message whatever a layout raises.
+
+ Raises:
+ KeyError: If a placeholder names neither a field of the record nor a derived value.
+ ValueError: If a derivation refuses a field's value.
+ TypeError: If a derivation refuses a field's type.
+ """
+ require_rich(message=CONSOLE_LAYOUT_MISSING_MESSAGE)
+ from rich.markup import escape
+
+ derived = self.derived_values(fields=fields)
+ substitutions: dict[str, str] = {}
+ for name in self.placeholders:
+ if name in derived:
+ text = one_line_text(text=str(derived[name]))
+ else:
+ text = format_layout_value(value=fields[name])
+ substitutions[name] = escape(text)
+ return self.template.format_map(substitutions)
+
+
+def _styled_placeholder(*, field: str, suffix: str = "") -> str:
+ """A field's placeholder in the style ``FIELD_STYLES`` gives it, so a layout and the suffix colour it alike."""
+ return f"[{FIELD_STYLES[field]}]{{{field}}}{suffix}[/]"
+
+
+#: How far a nested pipe run is indented per level of depth.
+PIPE_RUN_INDENT = " "
+#: What marks a nested pipe run, after its indentation.
+PIPE_RUN_BRANCH = "↳ "
+#: What precedes a dry run's pipe type.
+PIPE_RUN_DRY_RUN_LABEL = "Dry run: "
+#: The deepest nesting the pipe-run layout indents. The indentation is built from the depth, so an unbounded
+#: one builds a string as long as the caller likes, or raises ``OverflowError``; a deeper run than this,
+#: which is far past any pipe stack a run reaches, falls back to its message.
+PIPE_RUN_MAX_DEPTH = 100
+
+
+class PipeRunLayout(ConsoleLayout):
+ """The pipe-run tree: a nested run indented under its parent, behind a branch mark, a dry run labelled.
+
+ ``PipeCompose: compose_company → Company`` at the top level, the same line indented and behind ``↳`` for
+ a nested run, ``Dry run:`` before the pipe type for a dry one. It reads ``pipe_depth``, an integer from
+ ``0`` for a top-level run up to ``PIPE_RUN_MAX_DEPTH``, and ``is_dry_run``, a boolean.
+ """
+
+ @override
+ def derived_values(self, *, fields: Mapping[str, Any]) -> dict[str, Any]:
+ """The indentation, the branch mark and the dry-run label, from a depth and a flag this layout checks first.
+
+ Raises:
+ TypeError: If the depth is not an integer, or the flag not a boolean.
+ ValueError: If the depth is negative or deeper than ``PIPE_RUN_MAX_DEPTH``.
+ """
+ depth = fields["pipe_depth"]
+ is_dry_run = fields["is_dry_run"]
+ # A boolean is an integer to Python, and ``True`` would indent one level.
+ if isinstance(depth, bool) or not isinstance(depth, int):
+ msg = f"The pipe-run layout's depth must be an integer, not {type(depth).__name__}"
+ raise TypeError(msg)
+ if not 0 <= depth <= PIPE_RUN_MAX_DEPTH:
+ msg = f"The pipe-run layout's depth must be between 0 and {PIPE_RUN_MAX_DEPTH}"
+ raise ValueError(msg)
+ # Judged by truthiness, the string ``"false"`` would label a live run a dry one.
+ if not isinstance(is_dry_run, bool):
+ msg = f"The pipe-run layout's dry-run flag must be a boolean, not {type(is_dry_run).__name__}"
+ raise TypeError(msg)
+ return {
+ "indent": PIPE_RUN_INDENT * depth,
+ "branch": PIPE_RUN_BRANCH if depth > 0 else "",
+ "dry_run_label": PIPE_RUN_DRY_RUN_LABEL if is_dry_run else "",
+ }
+
+
+PIPE_RUN_LAYOUT = PipeRunLayout(
+ template=(
+ "{indent}[yellow]{branch}[/yellow][dim]{dry_run_label}[/dim]"
+ f"{_styled_placeholder(field='pipe_type', suffix=':')} {_styled_placeholder(field='pipe_code')} "
+ f"[yellow]→[/yellow] {_styled_placeholder(field='output_concept')}"
+ ),
+ presented_fields=frozenset({"pipe_type", "pipe_code", "output_concept", "pipe_depth", "is_dry_run"}),
+)
+
+#: Every layout a call can name, by its name.
+CONSOLE_LAYOUTS: dict[LogLayout, ConsoleLayout] = {
+ LogLayout.PIPE_RUN: PIPE_RUN_LAYOUT,
+}
+
+
+def console_layout(*, name: str) -> ConsoleLayout | None:
+ """The layout registered under the name, or ``None`` for a name no layout has."""
+ try:
+ layout_name = LogLayout(name)
+ except ValueError:
+ return None
+ return CONSOLE_LAYOUTS.get(layout_name)
diff --git a/pipelex/tools/log/console_log_sink.py b/pipelex/tools/log/console_log_sink.py
index bb04b281f..ab44babe2 100644
--- a/pipelex/tools/log/console_log_sink.py
+++ b/pipelex/tools/log/console_log_sink.py
@@ -1,5 +1,10 @@
"""The ``console`` sink: the Rich handler with the emoji formatter and every ``[runtime.log.rich_log]`` setting.
+The handler renders a record's fields after its message as a styled ``key=value`` suffix, and a record that
+names a layout through that layout's template; what is shown and how it is coloured is in
+``console_fields`` and ``console_layouts``. A traceback printed as text, with Rich tracebacks off, goes
+under the line, after the suffix.
+
Rich is the ``cli`` extra. It is imported when the handler is built and nowhere else in this module, so
this module asks for Rich only where this sink is the one selected; a process that selects another sink
never reaches that import. One that selects this sink without Rich installed fails at boot with the extra
@@ -17,14 +22,18 @@ from typing import TYPE_CHECKING
from typing_extensions import override
+from pipelex.tools.log.console_fields import attached_fields, field_suffix_segments
+from pipelex.tools.log.console_layouts import console_layout
from pipelex.tools.log.log_config import HighlighterName
-from pipelex.tools.log.log_formatter import EmojiLogFormatter
+from pipelex.tools.log.log_fields import LAYOUT_MARK
+from pipelex.tools.log.log_formatter import EmojiLogFormatter, channel_prefix
from pipelex.tools.log.log_sink import LogSink, LogSinkMethod, stream_for_target
from pipelex.tools.misc.rich_extra import require_rich
if TYPE_CHECKING:
import logging
+ from rich.console import ConsoleRenderable
from rich.logging import RichHandler
from pipelex.system.console_target import ConsoleTarget
@@ -38,7 +47,7 @@ CONSOLE_SINK_MISSING_MESSAGE = (
class ConsoleLogSink(LogSink):
- """Today's console rendering, byte for byte: a ``RichHandler`` on the configured stream."""
+ """A ``RichHandler`` on the configured stream, rendering the message, or its layout, and the fields after it."""
def __init__(self, *, rich_log_config: RichLogConfig, target: ConsoleTarget) -> None:
super().__init__()
@@ -52,6 +61,7 @@ class ConsoleLogSink(LogSink):
from rich.console import Console
from rich.highlighter import Highlighter, JSONHighlighter, ReprHighlighter
from rich.logging import RichHandler
+ from rich.text import Text
# Declared here because ``RichHandler`` is imported here, which is what keeps Rich off the import
# path of a process that selected another sink. ``RichHandler`` overrides ``emit`` and does not
@@ -69,6 +79,57 @@ class ConsoleLogSink(LogSink):
except Exception: # ruff: ignore[blind-except]
self.handleError(record)
+ @override
+ def render_message(self, record: logging.LogRecord, message: str) -> ConsoleRenderable:
+ """The message, or the layout the call named, then the fields as a styled ``key=value`` suffix, then any traceback text.
+
+ The suffix is assembled as ``Text`` from styled segments and appended after the highlighter has
+ run on the message, so neither the markup setting nor the highlighter ever reads a field's value.
+ A layout that cannot be filled, whatever it raises, falls back to the message, and every field
+ then goes to the suffix. The dispatch stamps a layout only on a call whose content is a string,
+ so a structured content, which only its message renders, never meets one.
+
+ With Rich tracebacks off, what Rich hands over is the formatter's whole output: the message line,
+ then the exception's text and any stack text. That tail is split off the line and printed under
+ it, after the suffix, so the fields stay on the line they describe and a layout keeps the
+ traceback. It is printed as plain text, so nothing in a traceback is ever read as markup. A record
+ with no such text renders exactly as before.
+ """
+ formatter = self.formatter
+ formatted_line = formatter.formatMessage(record) if formatter is not None else message
+ message_line = formatted_line if message.startswith(formatted_line) else message
+ appended_text = message[len(message_line) :]
+ fields = attached_fields(record=record)
+ presented_fields: frozenset[str] = frozenset()
+ message_text: ConsoleRenderable | None = None
+ layout_name = record.__dict__.get(LAYOUT_MARK)
+ layout = console_layout(name=layout_name) if isinstance(layout_name, str) else None
+ if layout is not None:
+ try:
+ layout_markup = layout.render_markup(fields=fields)
+ # No emoji codes either: a value spelled `:fire:` is a value, not a picture.
+ message_text = Text.assemble(channel_prefix(logger_name=record.name), Text.from_markup(layout_markup, emoji=False))
+ presented_fields = layout.presented_fields
+ except Exception: # ruff: ignore[blind-except]
+ # A layout's derivation is code each layout brings, filled with values each call
+ # chooses, so what it can raise is open-ended: a missing field, a refused value, a
+ # template Rich refuses, or arithmetic on a value nobody bounded. A layout costs
+ # nothing but itself, so whatever it raised, the line falls back to its message.
+ message_text = None
+ if message_text is None:
+ message_text = super().render_message(record, message_line)
+ if isinstance(message_text, Text):
+ segments = field_suffix_segments(fields=fields, presented_fields=presented_fields)
+ if segments:
+ message_text.append_text(Text.assemble(*segments))
+ if appended_text:
+ # The formatter puts a line break before the exception text only when the message does
+ # not already end in one, and the suffix or a layout now stands between the two.
+ if not message_text.plain.endswith("\n") and not appended_text.startswith("\n"):
+ message_text.append("\n")
+ message_text.append_text(Text(appended_text))
+ return message_text
+
config = self._rich_log_config
highlighter: Highlighter
match config.highlighter_name:
diff --git a/pipelex/tools/log/log.py b/pipelex/tools/log/log.py
index bd2f43835..aa1dc8503 100644
--- a/pipelex/tools/log/log.py
+++ b/pipelex/tools/log/log.py
@@ -16,6 +16,7 @@ if TYPE_CHECKING:
from collections.abc import Callable, Mapping
from contextlib import AbstractContextManager
+ from pipelex.tools.log.console_layouts import LogLayout
from pipelex.tools.log.log_config import LogConfig
from pipelex.tools.log.log_context import LogContext
from pipelex.tools.log.log_sink import LogRecordProcessor, LogSink
@@ -339,6 +340,7 @@ class Log:
title: str | None = None,
inline: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log a verbose message.
@@ -348,10 +350,11 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
"""
severity = LOGGING_LEVEL_VERBOSE
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields, layout=layout)
def debug(
self,
@@ -360,6 +363,7 @@ class Log:
title: str | None = None,
inline: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log a debug message.
@@ -369,10 +373,11 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
"""
severity = logging.DEBUG
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields, layout=layout)
def dev(
self,
@@ -381,6 +386,7 @@ class Log:
title: str | None = None,
inline: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log a development message.
@@ -390,10 +396,11 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
"""
severity = LOGGING_LEVEL_DEV
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields, layout=layout)
def info(
self,
@@ -402,6 +409,7 @@ class Log:
title: str | None = None,
inline: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log an info message.
@@ -411,10 +419,11 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
"""
severity = logging.INFO
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields, layout=layout)
def warning(
self,
@@ -424,6 +433,7 @@ class Log:
inline: str | None = None,
problem_id: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log a warning message.
@@ -433,13 +443,14 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
problem_id (str | None, optional): A problem ID to associate with the warning. Defaults to None.
"""
if self._should_ignore(problem_id=problem_id):
return
severity = logging.WARNING
- self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields)
+ self.log_dispatch.dispatch(content=content, severity=severity, title=title, inline=inline, fields=fields, layout=layout)
def error(
self,
@@ -450,6 +461,7 @@ class Log:
include_exception: bool = False,
problem_id: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log an error message.
@@ -459,6 +471,7 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
include_exception (bool, optional): Whether to include exception information. Defaults to False.
problem_id (str | None, optional): A problem ID to associate with the error. Defaults to None.
@@ -473,6 +486,7 @@ class Log:
inline=inline,
include_exception=include_exception,
fields=fields,
+ layout=layout,
)
def critical(
@@ -484,6 +498,7 @@ class Log:
include_exception: bool = False,
problem_id: str | None = None,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Log a critical message.
@@ -493,6 +508,7 @@ class Log:
inline (str | None, optional): Inline title for the log message. Defaults to None.
Used to display the title inline, only if the title arg is None.
fields (Mapping[str, Any] | None, optional): Named values carried as attributes of the record, never rendered into the message.
+ layout (LogLayout | None, optional): The console layout to render a string content through; every other sink ignores it.
include_exception (bool, optional): Whether to include exception information. Defaults to False.
problem_id (str | None, optional): A problem ID to associate with the critical message. Defaults to None.
@@ -507,6 +523,7 @@ class Log:
inline=inline,
include_exception=include_exception,
fields=fields,
+ layout=layout,
)
diff --git a/pipelex/tools/log/log_dispatch.py b/pipelex/tools/log/log_dispatch.py
index 805849d2a..3bd3afb17 100644
--- a/pipelex/tools/log/log_dispatch.py
+++ b/pipelex/tools/log/log_dispatch.py
@@ -18,7 +18,7 @@ from typing import TYPE_CHECKING, Any, cast
from pipelex.tools.log.log_config import CallerInfoTemplate, LogConfig
from pipelex.tools.log.log_context import get_log_context
-from pipelex.tools.log.log_fields import attach_log_record_extra, build_log_record_extra
+from pipelex.tools.log.log_fields import LAYOUT_MARK, attach_log_record_extra, build_log_record_extra
from pipelex.tools.log.log_redaction import redact_secret_entries
from pipelex.tools.misc.json_utils import purify_json, purify_json_dict, purify_json_list
@@ -26,6 +26,8 @@ if TYPE_CHECKING:
from collections.abc import Mapping
from types import FrameType
+ from pipelex.tools.log.console_layouts import LogLayout
+
# The triple ``sys.exc_info`` returns while an exception is being handled.
ExcInfo = tuple[type[BaseException], BaseException, Any]
@@ -107,6 +109,7 @@ class LogDispatch:
inline: str | None = None,
include_exception: bool = False,
fields: Mapping[str, Any] | None = None,
+ layout: LogLayout | None = None,
):
"""Emit one record for the call, on the logger named after the calling module.
@@ -119,6 +122,9 @@ class LogDispatch:
include_exception: Whether to carry the exception being handled on the record, as its
``exc_info``, for every sink to render its own way. Nothing is spliced into the message.
fields: Named values carried as attributes of the record, never rendered into the message.
+ layout: The console layout the record is rendered through, carried under ``LAYOUT_MARK``,
+ which only the console sink reads. It is carried only for a string content; any other
+ content renders in the message alone, which a layout would replace.
"""
caller_frame = _caller_frame()
@@ -133,9 +139,18 @@ class LogDispatch:
caller_info_str = self._caller_info(frame=caller_frame, module_name=module_name, log_config=log_config)
message = f"{caller_info_str}: {message}"
exc_info = _active_exc_info() if include_exception else None
+ # A layout renders in place of the message, so it is kept only where the message is a string the
+ # call gave. Any other content renders in the message alone: the console's suffix never repeats
+ # ``data``, a content JSON refused carries no ``data`` and lives only in the message's ``repr``, and
+ # an installed record factory that owns ``data`` sends the content to a prefixed name the suffix cuts
+ # short. Only here is the content's kind known, so the layout is dropped here rather than guessed at
+ # by the sink from what the record happens to carry.
+ record_layout = layout if isinstance(content, str) else None
extra = build_log_record_extra(context=get_log_context(), fields=fields, data=data)
- self._emit_record(message=message, severity=severity, logger=logger, caller_frame=caller_frame, exc_info=exc_info, extra=extra)
+ self._emit_record(
+ message=message, severity=severity, logger=logger, caller_frame=caller_frame, exc_info=exc_info, extra=extra, layout=record_layout
+ )
########################################################
# Private methods
@@ -221,11 +236,13 @@ class LogDispatch:
caller_frame: FrameType | None,
exc_info: ExcInfo | None,
extra: dict[str, Any],
+ layout: LogLayout | None,
):
"""Build the record at the caller's location, attach what it carries, and hand it to the logger's handlers.
The record is built first and the extra attached afterwards: ``makeRecord`` raises on a key the
- record already owns, and what it owns is only known once the installed record factory has run.
+ record already owns, and what it owns is only known once the installed record factory has run. The
+ layout is stamped last, under a name no field can take.
"""
if caller_frame is None:
pathname, lineno, func_name = UNKNOWN_FILE_NAME, 0, UNKNOWN_FUNCTION_NAME
@@ -242,4 +259,6 @@ class LogDispatch:
func=func_name,
)
attach_log_record_extra(record=record, extra=extra)
+ if layout is not None:
+ record.__dict__[LAYOUT_MARK] = layout
logger.handle(record)
diff --git a/pipelex/tools/log/log_fields.py b/pipelex/tools/log/log_fields.py
index 016df9cd1..9e1247a87 100644
--- a/pipelex/tools/log/log_fields.py
+++ b/pipelex/tools/log/log_fields.py
@@ -8,8 +8,9 @@ added is a collision too rather than the stdlib's ``KeyError``.
Pipelex's own machinery owns a name on the record too, and it is reserved here for the same reason the
``json`` sink reserves its own keys: a name nobody owns *yet* is a name a caller's field lands on freely,
-and this one steers delivery. So the reserved set spans what the formatter sets later and what this
-package stamps later, and neither is a field a sink reads back.
+and this one steers delivery. So the reserved set spans what the formatter sets later, what this
+package stamps later and what Rich's console handler reads off a record ahead of its own settings, and
+none of them is a field a sink reads back.
"""
from __future__ import annotations
@@ -46,6 +47,17 @@ FORWARDED_MARK = "_pipelex_forwarded"
# wire, and reserving it is what stops a caller steering the console through a field of that name.
VERBATIM_MARK = "markup"
+# Rich's per-record override of its handler's highlighter, which ``RichHandler.render_message`` reads as
+# ``getattr(record, "highlighter", self.highlighter)`` and then calls on the line. A caller's field landing
+# on it would be called in the highlighter's place: a string there raises, and the whole line is lost.
+RICH_HIGHLIGHTER_ATTRIBUTE = "highlighter"
+
+# Every attribute Rich's handler reads off a record ahead of its own setting: ``render_message`` reads these
+# two, and nothing else ``emit`` or ``render`` reads is outside the stdlib's own attributes. Each steers the
+# console alone and means nothing on a wire, so neither is a field a caller can land on nor one a sink is
+# handed as something the record carries.
+RICH_RECORD_OVERRIDES = frozenset({VERBATIM_MARK, RICH_HIGHLIGHTER_ATTRIBUTE})
+
# The attribute the redaction stamps on a record it could not strip, and takes back off the moment it
# has. It is the fail-closed half of the scrub: a record still carrying it reaches no sink, because what
# it carries is whatever the call put there and the scrub never read. It is set before the stripping
@@ -53,17 +65,29 @@ VERBATIM_MARK = "markup"
# second thing to go right at the moment the first one went wrong.
UNSCRUBBED_MARK = "_pipelex_unscrubbed"
+# The attribute naming, in order, the attributes ``attach_log_record_extra`` set on the record: the fields,
+# the context identifiers and ``data`` under the names they landed on. It is how the console tells what the
+# call attached from what a record factory or a third-party library stamped, which it does not render. A
+# caller's field spelling it would hand the console a list of names the call never gave.
+FIELD_NAMES_MARK = "_pipelex_field_names"
+
+# The attribute carrying the console layout a call named with ``layout=``. The layout is console
+# presentation: the console sink reads it, and reserving it is what keeps it off every wire, where the
+# message already says what the line is. A caller's field spelling it would pick the line's layout.
+LAYOUT_MARK = "_pipelex_layout"
+
# The names this package stamps on a record itself. Never a field: a caller's entry of the same name is
# prefixed on the way on, and a record carrying one does not hand it to a sink as something it carries.
# Reserving them is what makes each one safe, because each is stamped later than the entries are attached
# and so a fresh record owns none of them: a caller's field spelling ``FORWARDED_MARK`` would be read as
# the forwarding marker and cost the whole record its delivery, and one spelling ``UNSCRUBBED_MARK`` would
# have a perfectly ordinary record read as one the scrub could not strip, and dropped.
-PIPELEX_OWNED_ATTRIBUTES = frozenset({FORWARDED_MARK, VERBATIM_MARK, UNSCRUBBED_MARK})
+PIPELEX_OWNED_ATTRIBUTES = frozenset({FORWARDED_MARK, VERBATIM_MARK, UNSCRUBBED_MARK, FIELD_NAMES_MARK, LAYOUT_MARK})
# Reserved whether or not the record carries the name yet, which is exactly what the stdlib's own refusal
-# cannot cover: both sets are stamped after the entries are attached.
-RESERVED_ATTRIBUTES = FORMATTER_OWNED_ATTRIBUTES | PIPELEX_OWNED_ATTRIBUTES
+# cannot cover: the formatter's and Pipelex's are stamped after the entries are attached, and Rich's are
+# read by the console handler whoever set them.
+RESERVED_ATTRIBUTES = FORMATTER_OWNED_ATTRIBUTES | PIPELEX_OWNED_ATTRIBUTES | RICH_RECORD_OVERRIDES
# The attributes the stdlib gives every record, read off one built by the stdlib's own constructor on
# this interpreter rather than listed by hand, so a version that adds one (``taskName`` arrived with
@@ -122,12 +146,36 @@ def attach_log_record_extra(*, record: logging.LogRecord, extra: Mapping[str, An
the name lands on an attribute nobody owns, and entries are attached in order, so an entry attached
earlier under a prefixed name is owned for the entries after it: whatever the order of the mapping, no
value is lost.
+
+ The names the entries landed on are recorded, in order, under ``FIELD_NAMES_MARK``, which is how the
+ console tells what this call attached from what anything else stamped on the record.
"""
+ attached: list[str] = []
for name, value in extra.items():
attribute = name
while hasattr(record, attribute) or attribute in RESERVED_ATTRIBUTES:
attribute = f"{COLLIDING_FIELD_PREFIX}{attribute}"
setattr(record, attribute, value)
+ attached.append(attribute)
+ already_attached: tuple[str, ...] = record.__dict__.get(FIELD_NAMES_MARK, ())
+ record.__dict__[FIELD_NAMES_MARK] = (*already_attached, *attached)
+
+
+def given_field_name(*, name: str) -> str:
+ """The name a field was given, read off the name it landed on with every collision prefix taken off.
+
+ A field lands under a prefixed name whenever the record already owns the one the caller used, so
+ whatever judges a field by its meaning, a secret's name or a colour, reads it here rather than as carried.
+ """
+ while name.startswith(COLLIDING_FIELD_PREFIX):
+ name = name[len(COLLIDING_FIELD_PREFIX) :]
+ return name
+
+
+def attached_field_names(*, record: logging.LogRecord) -> tuple[str, ...]:
+ """The attributes ``attach_log_record_extra`` set on the record, in order, or none for a record it never saw."""
+ names: tuple[str, ...] = record.__dict__.get(FIELD_NAMES_MARK, ())
+ return names
def carried_attributes(*, record: logging.LogRecord) -> dict[str, Any]:
@@ -135,7 +183,11 @@ def carried_attributes(*, record: logging.LogRecord) -> dict[str, Any]:
Read the way a structured sink reads a record, in the order the attributes were attached. A value is
handed back as the call gave it: a sink serializes it when it emits, on the calling thread. What this
- package stamps on a record itself is machinery and belongs on no wire, so it is left out here as it is
- reserved on the way in.
+ package stamps on a record itself is machinery and Rich's per-record overrides steer the console alone,
+ so neither belongs on a wire, and both are left out here as they are reserved on the way in.
"""
- return {name: value for name, value in vars(record).items() if name not in STDLIB_RECORD_ATTRIBUTES and name not in PIPELEX_OWNED_ATTRIBUTES}
+ return {
+ name: value
+ for name, value in vars(record).items()
+ if name not in STDLIB_RECORD_ATTRIBUTES and name not in PIPELEX_OWNED_ATTRIBUTES and name not in RICH_RECORD_OVERRIDES
+ }
diff --git a/pipelex/tools/log/log_formatter.py b/pipelex/tools/log/log_formatter.py
index a783fdb46..a27e41071 100644
--- a/pipelex/tools/log/log_formatter.py
+++ b/pipelex/tools/log/log_formatter.py
@@ -41,9 +41,14 @@ class EmojiLogFormatter(logging.Formatter):
@override
def formatMessage(self, record: logging.LogRecord) -> str:
- emoji = emoji_for_channel(record.name)
- if emoji == "":
- return record.message
- if emoji:
- return f"{emoji}: {record.message}"
- return f"[{record.name}]: {record.message}"
+ return f"{channel_prefix(logger_name=record.name)}{record.message}"
+
+
+def channel_prefix(*, logger_name: str) -> str:
+ """What the console puts before a record's message: the logger's emoji, nothing for the root, else the logger's name in brackets."""
+ emoji = emoji_for_channel(logger_name)
+ if emoji == "":
+ return ""
+ if emoji:
+ return f"{emoji}: "
+ return f"[{logger_name}]: "
diff --git a/pipelex/tools/log/log_redaction.py b/pipelex/tools/log/log_redaction.py
index cb3d7424b..086782210 100644
--- a/pipelex/tools/log/log_redaction.py
+++ b/pipelex/tools/log/log_redaction.py
@@ -35,7 +35,7 @@ from typing import TYPE_CHECKING, Any, cast
from pydantic import BaseModel
-from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD, UNSCRUBBED_MARK, carried_attributes
+from pipelex.tools.log.log_fields import DATA_FIELD, LAYOUT_MARK, UNSCRUBBED_MARK, carried_attributes, given_field_name
if TYPE_CHECKING:
from collections.abc import Sequence
@@ -234,6 +234,9 @@ def make_redaction_processor(*, config: LogRedactionConfig) -> LogRecordProcesso
def _quarantine(*, record: logging.LogRecord, exc: Exception) -> None:
"""Strip a record whose scrub failed down to a notice naming the failure, so nothing unscrubbed leaves with it.
+ The console layout the call named is taken off too: a layout renders in place of the message, so it
+ would draw the redacted fields and hide the notice that says why they are redacted.
+
The mark ``redact`` set is lifted at the end and only there, so a strip that gets partway leaves the
record marked and the record is dropped rather than emitted as a notice that says it was stripped
while still carrying what the scrub never read.
@@ -243,6 +246,7 @@ def _quarantine(*, record: logging.LogRecord, exc: Exception) -> None:
record.exc_info = None
record.exc_text = None
record.stack_info = None
+ record.__dict__.pop(LAYOUT_MARK, None)
for name in carried_attributes(record=record):
setattr(record, name, REDACTED_TEXT)
record.__dict__.pop(UNSCRUBBED_MARK, None)
@@ -284,9 +288,7 @@ def _is_secret_field_name(*, name: str) -> bool:
Judging the name as it was spelled rather than as it was carried covers that, and covers a caller who
spelled the prefix themselves, which is the safe direction to be wrong in.
"""
- while name.startswith(COLLIDING_FIELD_PREFIX):
- name = name[len(COLLIDING_FIELD_PREFIX) :]
- return _is_secret_key(key=name)
+ return _is_secret_key(key=given_field_name(name=name))
def _redact_exception_text(*, record: logging.LogRecord, patterns: tuple[RedactionPattern, ...]) -> None:
diff --git a/tests/helpers/console_log_rendering.py b/tests/helpers/console_log_rendering.py
new file mode 100644
index 000000000..be7911488
--- /dev/null
+++ b/tests/helpers/console_log_rendering.py
@@ -0,0 +1,105 @@
+"""Builders and readings shared by the tests of the console sink's field suffix, style map and layouts."""
+
+from __future__ import annotations
+
+import logging
+from contextlib import contextmanager
+from typing import TYPE_CHECKING, Any
+
+from rich.console import Console
+from rich.logging import RichHandler
+from rich.text import Text
+
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.console_target import ConsoleTarget
+from pipelex.tools.log.console_log_sink import ConsoleLogSink
+from pipelex.tools.log.log import Log
+from pipelex.tools.log.log_config import LogConfig, RichLogConfig
+from pipelex.tools.log.log_fields import LAYOUT_MARK, attach_log_record_extra
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+
+if TYPE_CHECKING:
+ import io
+ from collections.abc import Generator
+
+ from pipelex.tools.log.console_layouts import LogLayout
+ from pipelex.tools.log.log_sink import LogSink
+
+CONSOLE_WIDTH = 200
+
+# The fields the pipe-run layout presents, for a top-level run that is not a dry run.
+PIPE_RUN_FIELDS: dict[str, Any] = {
+ "pipe_type": "PipeCompose",
+ "pipe_code": "compose_company",
+ "output_concept": "Company",
+ "pipe_depth": 0,
+ "is_dry_run": False,
+}
+
+
+def package_log_config() -> LogConfig:
+ config_dict = load_toml_from_path(ConfigLoader().pipelex_root_dir / "pipelex.toml")
+ return LogConfig.model_validate(config_dict["runtime"]["log"])
+
+
+def console_handler() -> RichHandler:
+ handler = ConsoleLogSink(rich_log_config=package_log_config().rich_log, target=ConsoleTarget.STDERR).handler
+ assert isinstance(handler, RichHandler)
+ return handler
+
+
+def record_with_fields(*, message: str, extra: dict[str, Any] | None = None, layout: LogLayout | str | None = None) -> logging.LogRecord:
+ """A record as the dispatch builds it: the extra attached through the fields channel, the layout under its mark."""
+ record = logging.LogRecord(
+ name="pipelex.tools.demo", level=logging.INFO, pathname="/repo/pipelex/module.py", lineno=42, msg=message, args=(), exc_info=None
+ )
+ attach_log_record_extra(record=record, extra=extra or {})
+ if layout is not None:
+ record.__dict__[LAYOUT_MARK] = layout
+ return record
+
+
+def rendered_text(*, record: logging.LogRecord) -> Text:
+ """The ``Text`` the console sink's handler renders for the record's message line, styles and all."""
+ handler = console_handler()
+ record.message = record.getMessage()
+ assert handler.formatter is not None
+ rendered = handler.render_message(record, handler.formatter.formatMessage(record))
+ assert isinstance(rendered, Text)
+ return rendered
+
+
+def styles_of(*, text: Text, fragment: str) -> list[str]:
+ """The styles of the spans covering exactly the fragment, where it first appears in the text."""
+ start = text.plain.index(fragment)
+ end = start + len(fragment)
+ return [str(span.style) for span in text.spans if span.start == start and span.end == end]
+
+
+@contextmanager
+def installed_log(*, sink: LogSink) -> Generator[Log]:
+ """A fresh ``Log`` with the sink installed, reset afterwards so the root logger is left as found."""
+ fresh = Log()
+ fresh.configure(log_config=package_log_config())
+ fresh.install_sink(sink)
+ try:
+ yield fresh
+ finally:
+ fresh.reset()
+
+
+def package_rich_log_config_without_rich_tracebacks() -> RichLogConfig:
+ """The package's console settings with Rich tracebacks off, so a traceback reaches the handler as text after the message."""
+ return package_log_config().rich_log.model_copy(update={"is_rich_tracebacks": False})
+
+
+def console_sink_on_buffer(*, buffer: io.StringIO, rich_log_config: RichLogConfig | None = None) -> ConsoleLogSink:
+ """A console sink whose handler writes to the buffer, wide and colourless, so a line is read as plain text.
+
+ The handler takes the package's console settings unless others are given.
+ """
+ sink = ConsoleLogSink(rich_log_config=rich_log_config or package_log_config().rich_log, target=ConsoleTarget.STDERR)
+ handler = sink.handler
+ assert isinstance(handler, RichHandler)
+ handler.console = Console(file=buffer, width=CONSOLE_WIDTH, force_terminal=False, color_system=None, legacy_windows=False)
+ return sink
diff --git a/tests/unit/pipelex/tools/test_log_console_field_values.py b/tests/unit/pipelex/tools/test_log_console_field_values.py
new file mode 100644
index 000000000..2329a10f3
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_console_field_values.py
@@ -0,0 +1,89 @@
+from __future__ import annotations
+
+import math
+from pathlib import Path
+from typing import Any
+
+import pytest
+from pydantic import BaseModel
+
+from pipelex.tools.log.console_fields import FIELD_VALUE_MAX_LENGTH, TRUNCATION_MARK, format_field_value, format_layout_value
+
+
+class _Sample(BaseModel):
+ code: str
+ count: int
+
+
+class TestConsoleFieldValues:
+ @pytest.mark.parametrize(
+ ("value", "expected"),
+ [
+ (7, "7"),
+ (0.5, "0.5"),
+ (True, "true"),
+ (None, "null"),
+ ("alpha", "alpha"),
+ ("two words", '"two words"'),
+ ("", '""'),
+ ('say "hi" now', '"say \\"hi\\" now"'),
+ ("b=c", '"b=c"'),
+ ('a"b', '"a\\"b"'),
+ ("a\\b", '"a\\\\b"'),
+ ('ends in \\"', '"ends in \\\\\\""'),
+ ("tab\\tliteral\ttab", '"tab\\\\tliteral\\ttab"'),
+ ("line\nbreak", '"line\\nbreak"'),
+ ("\x1b[31mred", '"\\x1b[31mred"'),
+ ({"key": "value", "nested": [1, 2]}, '{"key":"value","nested":[1,2]}'),
+ (["a", "b"], '["a","b"]'),
+ (("a", 1), '["a",1]'),
+ (math.nan, "NaN"),
+ ({"ratio": math.inf}, '{"ratio":"Infinity"}'),
+ (_Sample(code="x", count=2), '{"code":"x","count":2}'),
+ (Path("reports/march.csv"), "reports/march.csv"),
+ ],
+ ids=[
+ "int",
+ "float",
+ "bool",
+ "none",
+ "bare string",
+ "spaced string",
+ "empty string",
+ "quotes",
+ "equals sign",
+ "bare quote",
+ "backslash",
+ "backslash before a quote",
+ "typed escape beside a real one",
+ "newline",
+ "escape",
+ "mapping",
+ "list",
+ "tuple",
+ "nan",
+ "nested infinity",
+ "model",
+ "path",
+ ],
+ )
+ def test_a_value_renders_compactly_on_one_line(self, value: Any, expected: str) -> None:
+ assert format_field_value(value=value) == expected
+
+ def test_a_layout_substitutes_a_string_as_itself_and_still_on_one_line(self) -> None:
+ assert format_layout_value(value="two words") == "two words"
+ assert format_layout_value(value="line\nbreak") == "line\\nbreak"
+ assert format_layout_value(value=7) == "7"
+
+ def test_a_value_json_refuses_renders_as_its_repr(self) -> None:
+ circular: dict[str, Any] = {}
+ circular["self"] = circular
+
+ assert format_field_value(value=circular) == "{'self': {...}}"
+
+ def test_a_long_value_is_cut_short_at_the_limit_with_a_mark(self) -> None:
+ rendered = format_field_value(value=list(range(200)))
+
+ assert len(rendered) == FIELD_VALUE_MAX_LENGTH
+ assert rendered.endswith(TRUNCATION_MARK)
+ assert rendered.startswith("[0,1,2,")
diff --git a/tests/unit/pipelex/tools/test_log_console_fields.py b/tests/unit/pipelex/tools/test_log_console_fields.py
new file mode 100644
index 000000000..55b2992c3
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_console_fields.py
@@ -0,0 +1,187 @@
+from __future__ import annotations
+
+import io
+import logging
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+from pipelex.tools.log.console_fields import FIELD_KEY_STYLE, FIELD_STYLES, FIELD_VALUE_MAX_LENGTH, TRUNCATION_MARK, UNMAPPED_FIELD_STYLE
+from pipelex.tools.log.log_fields import (
+ COLLIDING_FIELD_PREFIX,
+ FIELD_NAMES_MARK,
+ RICH_HIGHLIGHTER_ATTRIBUTE,
+ VERBATIM_MARK,
+ attach_log_record_extra,
+)
+from tests.helpers.console_log_rendering import (
+ console_sink_on_buffer,
+ installed_log,
+ package_log_config,
+ record_with_fields,
+ rendered_text,
+ styles_of,
+)
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from pipelex.tools.log.log import Log
+
+
+class TestConsoleFields:
+ @pytest.fixture
+ def console_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, io.StringIO]]:
+ """A fresh ``Log`` with the console sink on a buffer, the module's own logger enabled for the test."""
+ # pytest's ``log_level`` option restores the root logger's level at every phase boundary, so the
+ # module's own logger is enabled explicitly, and ``caplog`` restores it at teardown.
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ with installed_log(sink=console_sink_on_buffer(buffer=buffer)) as fresh:
+ yield fresh, buffer
+
+ def test_the_fields_follow_the_message_as_key_value(self, console_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = console_log
+
+ fresh.info("Scanned the inputs", fields={"files": 7})
+
+ assert "Scanned the inputs files=7" in buffer.getvalue()
+
+ def test_the_run_identifiers_and_the_structured_content_are_never_repeated(self, console_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = console_log
+
+ with fresh.context(request_id="req-1", pipeline_run_id="plr-1", pipe_run_id="pr-1"):
+ fresh.info("Inside a run", fields={"files": 7})
+ fresh.info({"key": "value"}, title="Config", fields={"attempt": 2})
+
+ rendered = buffer.getvalue()
+ assert "Inside a run files=7" in rendered
+ assert "} attempt=2" in rendered
+ for hidden in ("request_id", "pipeline_run_id", "pipe_run_id", "req-1", "plr-1", "pr-1", "data="):
+ assert hidden not in rendered
+
+ def test_a_field_named_like_a_record_attribute_or_a_mark_shows_under_the_name_it_landed_on(self, console_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = console_log
+
+ fresh.info("Renamed", fields={"name": "alpha", VERBATIM_MARK: True})
+
+ assert "Renamed field_name=alpha field_markup=true" in buffer.getvalue()
+
+ def test_a_field_named_like_rich_s_highlighter_override_keeps_the_line(self, console_log: tuple[Log, io.StringIO]) -> None:
+ """Rich calls whatever the record carries as ``highlighter``, so a string field of that name raised inside the handler and lost the line."""
+ fresh, buffer = console_log
+
+ fresh.info("Highlighted", fields={RICH_HIGHLIGHTER_ATTRIBUTE: "pygments"})
+
+ assert f"Highlighted {COLLIDING_FIELD_PREFIX}{RICH_HIGHLIGHTER_ATTRIBUTE}=pygments" in buffer.getvalue()
+
+ @pytest.mark.parametrize(
+ ("fields", "expected"),
+ [
+ ({"x=1": 2}, 'Paired "x=1"=2'),
+ ({"a": "b=c"}, 'Paired a="b=c"'),
+ ({'say"': 'it"s'}, 'Paired "say\\""="it\\"s"'),
+ ({"path": "C:\\dir"}, 'Paired path="C:\\\\dir"'),
+ ],
+ ids=["an equals sign in a key", "an equals sign in a value", "a quote", "a backslash"],
+ )
+ def test_a_key_or_value_that_could_forge_a_pair_is_quoted(
+ self, console_log: tuple[Log, io.StringIO], fields: dict[str, Any], expected: str
+ ) -> None:
+ """Printed bare, ``{"x=1": 2}`` read as ``x=1=2`` and ``{"a": "b=c"}`` as ``a=b=c``.
+
+ An unescaped backslash could also end a quoted value early.
+ """
+ fresh, buffer = console_log
+
+ fresh.info("Paired", fields=fields)
+
+ assert expected in buffer.getvalue()
+
+ @pytest.mark.parametrize(
+ ("name", "expected"),
+ [
+ ("hello\nFORGED", 'Keyed "hello\\nFORGED"=1'),
+ ("two words", 'Keyed "two words"=1'),
+ ("\x1b[2J", 'Keyed "\\x1b[2J"=1'),
+ ],
+ ids=["a line break", "a space", "an escape sequence"],
+ )
+ def test_a_key_is_written_like_a_value_so_it_forges_neither_a_line_nor_a_pair(
+ self, console_log: tuple[Log, io.StringIO], name: str, expected: str
+ ) -> None:
+ """A field's name is the caller's text too: unescaped, a line break printed a forged ``FORGED=1`` line of its own."""
+ fresh, buffer = console_log
+
+ fresh.info("Keyed", fields={name: 1})
+
+ rendered = buffer.getvalue()
+ assert expected in rendered
+ assert "\x1b" not in rendered
+ assert not any(line.lstrip().startswith("FORGED") for line in rendered.splitlines())
+
+ def test_a_value_carrying_markup_prints_as_written_while_the_message_still_reads_markup(self, console_log: tuple[Log, io.StringIO]) -> None:
+ assert package_log_config().rich_log.is_markup_enabled, "the suffix must hold whatever the setting it runs beside"
+ fresh, buffer = console_log
+
+ fresh.info("[bold]Received[/bold]", fields={"detail": "[red]x[/red]"})
+
+ rendered = buffer.getvalue()
+ assert "Received detail=[red]x[/red]" in rendered
+ assert "[bold]" not in rendered
+
+ def test_a_long_value_is_cut_short(self, console_log: tuple[Log, io.StringIO]) -> None:
+ fresh, buffer = console_log
+
+ fresh.info("Long", fields={"excerpt": "x" * 500})
+
+ rendered = buffer.getvalue()
+ assert f"excerpt={'x' * (FIELD_VALUE_MAX_LENGTH - len(TRUNCATION_MARK))}{TRUNCATION_MARK}" in rendered
+ assert "x" * FIELD_VALUE_MAX_LENGTH not in rendered
+
+ def test_a_field_is_styled_by_its_name_and_a_field_outside_the_map_is_dimmed(self) -> None:
+ text = rendered_text(
+ record=record_with_fields(
+ message="Running",
+ extra={"pipe_code": "compose_company", "output_concept": "Company", "pipe_type": "PipeLLM", "attempt": 2},
+ ),
+ )
+
+ assert text.plain == "🧠: Running pipe_code=compose_company output_concept=Company pipe_type=PipeLLM attempt=2"
+ assert styles_of(text=text, fragment="compose_company") == [FIELD_STYLES["pipe_code"]] == ["red"]
+ assert styles_of(text=text, fragment="Company") == [FIELD_STYLES["output_concept"]] == ["bold green"]
+ assert styles_of(text=text, fragment="PipeLLM") == [FIELD_STYLES["pipe_type"]] == ["white"]
+ assert styles_of(text=text, fragment="2") == [UNMAPPED_FIELD_STYLE]
+ assert styles_of(text=text, fragment="pipe_code=") == [FIELD_KEY_STYLE]
+
+ def test_a_field_a_record_factory_pushed_under_the_prefix_keeps_its_colour(self) -> None:
+ """The style used to be looked up by the name the field landed on, so ``field_pipe_code`` printed dimmed."""
+ record = logging.LogRecord(
+ name="pipelex.tools.demo", level=logging.INFO, pathname="/repo/pipelex/module.py", lineno=42, msg="Running", args=(), exc_info=None
+ )
+ record.pipe_code = "from-factory"
+ attach_log_record_extra(record=record, extra={"pipe_code": "compose_company"})
+
+ text = rendered_text(record=record)
+
+ assert text.plain == f"🧠: Running {COLLIDING_FIELD_PREFIX}pipe_code=compose_company"
+ assert styles_of(text=text, fragment="compose_company") == [FIELD_STYLES["pipe_code"]]
+
+ def test_what_a_record_factory_or_the_runtime_stamped_is_not_shown(self) -> None:
+ record = record_with_fields(message="Stamped", extra={"files": 7})
+ record.otelSpanID = "0"
+ record.__dict__[VERBATIM_MARK] = False
+
+ text = rendered_text(record=record)
+
+ assert text.plain == "🧠: Stamped files=7"
+ assert FIELD_NAMES_MARK not in text.plain
+
+ def test_a_record_the_fields_channel_never_saw_renders_its_message_alone(self) -> None:
+ record = logging.LogRecord(name="httpx", level=logging.INFO, pathname="/x.py", lineno=1, msg="HTTP Request", args=(), exc_info=None)
+ record.custom = "a library's own extra"
+
+ text = rendered_text(record=record)
+
+ assert text.plain.endswith("HTTP Request")
+ assert "custom" not in text.plain
diff --git a/tests/unit/pipelex/tools/test_log_console_layouts.py b/tests/unit/pipelex/tools/test_log_console_layouts.py
new file mode 100644
index 000000000..c4584a862
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_console_layouts.py
@@ -0,0 +1,211 @@
+from __future__ import annotations
+
+import io
+import json
+import logging
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+from pipelex.tools.log.console_fields import FIELD_STYLES, FIELD_VALUE_MAX_LENGTH
+from pipelex.tools.log.console_layouts import CONSOLE_LAYOUTS, PIPE_RUN_MAX_DEPTH, ConsoleLayout, LogLayout, PipeRunLayout
+from pipelex.tools.log.json_log_sink import LOGGER_KEY, MESSAGE_KEY, JsonLogSink
+from pipelex.tools.log.log_fields import FIELD_NAMES_MARK, LAYOUT_MARK
+from pipelex.tools.log.log_redaction import QUARANTINE_PREFIX, REDACTED_TEXT
+from tests.helpers.console_log_rendering import (
+ PIPE_RUN_FIELDS,
+ console_sink_on_buffer,
+ installed_log,
+ record_with_fields,
+ rendered_text,
+ styles_of,
+)
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+class TestConsoleLayouts:
+ @pytest.mark.parametrize(
+ ("overrides", "expected"),
+ [
+ ({}, "PipeCompose: compose_company → Company"),
+ ({"pipe_depth": 1}, " ↳ PipeCompose: compose_company → Company"),
+ ({"pipe_depth": 2, "is_dry_run": True}, " ↳ Dry run: PipeCompose: compose_company → Company"),
+ ],
+ ids=["top level", "nested", "nested dry run"],
+ )
+ def test_the_pipe_run_layout_draws_the_tree_in_the_style_map_colours(self, overrides: dict[str, Any], expected: str) -> None:
+ text = rendered_text(record=record_with_fields(message="Pipe run starts", extra={**PIPE_RUN_FIELDS, **overrides}, layout=LogLayout.PIPE_RUN))
+
+ assert text.plain == f"🧠: {expected}"
+ assert styles_of(text=text, fragment="compose_company") == [FIELD_STYLES["pipe_code"]]
+ assert styles_of(text=text, fragment="Company") == [FIELD_STYLES["output_concept"]]
+
+ def test_a_value_carrying_markup_or_an_emoji_code_prints_as_written_inside_a_layout(self) -> None:
+ record = record_with_fields(
+ message="Pipe run starts",
+ extra={**PIPE_RUN_FIELDS, "pipe_code": "[red]x[/red]", "output_concept": ":fire:"},
+ layout=LogLayout.PIPE_RUN,
+ )
+
+ text = rendered_text(record=record)
+
+ assert text.plain == "🧠: PipeCompose: [red]x[/red] → :fire:"
+ assert styles_of(text=text, fragment="[red]x[/red]") == [FIELD_STYLES["pipe_code"]]
+
+ def test_the_console_renders_a_layout_end_to_end_and_the_other_fields_follow_it(self, caplog: pytest.LogCaptureFixture) -> None:
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ with installed_log(sink=console_sink_on_buffer(buffer=buffer)) as fresh, fresh.context(request_id="req-1"):
+ fresh.info("Pipe run starts", fields={**PIPE_RUN_FIELDS, "pipe_depth": 1, "attempt": 2}, layout=LogLayout.PIPE_RUN)
+
+ rendered = buffer.getvalue()
+ assert " ↳ PipeCompose: compose_company → Company attempt=2" in rendered
+ assert "Pipe run starts" not in rendered
+ assert "req-1" not in rendered
+
+ @pytest.mark.parametrize(
+ ("overrides", "expected_values"),
+ [
+ ({"pipe_depth": "deep"}, "pipe_depth=deep is_dry_run=false"),
+ ({"pipe_depth": 10**20}, "pipe_depth=100000000000000000000 is_dry_run=false"),
+ ({"pipe_depth": PIPE_RUN_MAX_DEPTH + 1}, f"pipe_depth={PIPE_RUN_MAX_DEPTH + 1} is_dry_run=false"),
+ ({"pipe_depth": -1}, "pipe_depth=-1 is_dry_run=false"),
+ ({"pipe_depth": True}, "pipe_depth=true is_dry_run=false"),
+ ({"is_dry_run": "false"}, "pipe_depth=0 is_dry_run=false"),
+ ({"is_dry_run": 1}, "pipe_depth=0 is_dry_run=1"),
+ ],
+ ids=[
+ "a depth that is no integer",
+ "a depth no index can hold",
+ "a depth past the bound",
+ "a negative depth",
+ "a boolean depth",
+ "a dry-run flag spelled as a string",
+ "a dry-run flag given as an integer",
+ ],
+ )
+ def test_a_value_the_pipe_run_derivation_refuses_falls_back_to_the_message_and_every_field(
+ self, overrides: dict[str, Any], expected_values: str
+ ) -> None:
+ """A depth used to be repeated as given, so a huge one raised ``OverflowError`` past the fallback and lost the line, or built a huge string.
+
+ The flag used to be judged by truthiness, so ``"false"`` labelled a live run ``Dry run:``.
+ """
+ record = record_with_fields(message="Pipe run starts", extra={**PIPE_RUN_FIELDS, **overrides}, layout=LogLayout.PIPE_RUN)
+
+ text = rendered_text(record=record)
+
+ assert text.plain == f"🧠: Pipe run starts pipe_type=PipeCompose pipe_code=compose_company output_concept=Company {expected_values}"
+
+ def test_a_layout_that_cannot_be_filled_falls_back_to_the_message_and_every_field(self) -> None:
+ text = rendered_text(record=record_with_fields(message="Pipe run starts", extra={"pipe_code": "compose_company"}, layout=LogLayout.PIPE_RUN))
+
+ assert text.plain == "🧠: Pipe run starts pipe_code=compose_company"
+
+ def test_a_layout_that_raises_anything_at_all_costs_only_itself(self, mocker: MockerFixture) -> None:
+ """What a layout's derivation can raise is open-ended, and an error outside the ones the fallback named used to lose the line."""
+ mocker.patch.object(PipeRunLayout, "derived_values", side_effect=OverflowError("cannot fit 'int' into an index-sized integer"))
+
+ text = rendered_text(record=record_with_fields(message="Pipe run starts", extra={"pipe_code": "compose_company"}, layout=LogLayout.PIPE_RUN))
+
+ assert text.plain == "🧠: Pipe run starts pipe_code=compose_company"
+
+ def test_a_record_carrying_structured_content_keeps_its_message_and_every_field_follows_it(self, caplog: pytest.LogCaptureFixture) -> None:
+ """The console never repeats ``data`` in the suffix because the message renders it, so a layout must not replace that message."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ with installed_log(sink=console_sink_on_buffer(buffer=buffer)) as fresh:
+ fresh.info({"the_content_key": 1}, title="Pipe run starts", fields={**PIPE_RUN_FIELDS, "attempt": 2}, layout=LogLayout.PIPE_RUN)
+
+ rendered = buffer.getvalue()
+ assert "Pipe run starts:" in rendered
+ assert '"the_content_key": 1' in rendered
+ assert "} pipe_type=PipeCompose pipe_code=compose_company output_concept=Company pipe_depth=0 is_dry_run=false attempt=2" in rendered
+ assert "→" not in rendered
+
+ def test_a_content_json_refuses_keeps_its_message_rather_than_a_layout(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A circular content carries no ``data`` and lives only in the message's ``repr``, which a layout used to replace whole."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ circular: dict[str, Any] = {"the_content_key": 1}
+ circular["self"] = circular
+ buffer = io.StringIO()
+ with installed_log(sink=console_sink_on_buffer(buffer=buffer)) as fresh:
+ fresh.info(circular, fields=PIPE_RUN_FIELDS, layout=LogLayout.PIPE_RUN)
+
+ rendered = buffer.getvalue()
+ assert "'the_content_key': 1" in rendered
+ assert "pipe_code=compose_company" in rendered
+ assert "→" not in rendered
+
+ def test_a_content_a_record_factory_pushed_off_data_keeps_its_message_rather_than_a_layout(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A factory that owns ``data`` sends the content to ``field_data``, which a layout used to cut down to the suffix's short value."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ long_value = "x" * (FIELD_VALUE_MAX_LENGTH + 20)
+ previous_factory = logging.getLogRecordFactory()
+
+ def stamping_factory(*args: Any, **kwargs: Any) -> logging.LogRecord:
+ record = previous_factory(*args, **kwargs)
+ record.data = "data-from-factory"
+ return record
+
+ buffer = io.StringIO()
+ with installed_log(sink=console_sink_on_buffer(buffer=buffer)) as fresh:
+ logging.setLogRecordFactory(stamping_factory)
+ try:
+ fresh.info({"the_content_key": long_value}, fields=PIPE_RUN_FIELDS, layout=LogLayout.PIPE_RUN)
+ finally:
+ logging.setLogRecordFactory(previous_factory)
+
+ rendered = buffer.getvalue()
+ assert f'"the_content_key": "{long_value}"' in rendered
+ assert "→" not in rendered
+
+ def test_a_record_whose_redaction_failed_shows_the_notice_rather_than_its_layout(
+ self, caplog: pytest.LogCaptureFixture, mocker: MockerFixture
+ ) -> None:
+ """The quarantine turns the message into a notice and every field into ``[REDACTED]``.
+
+ A layout left on the record drew those fields and hid the notice. The shipped layout refuses a redacted
+ depth and would fall back by accident, so a layout that derives nothing stands in for it.
+ """
+ caplog.set_level(logging.INFO, logger=__name__)
+ mocker.patch.dict(
+ CONSOLE_LAYOUTS, {LogLayout.PIPE_RUN: ConsoleLayout(template="[red]{pipe_code}[/]", presented_fields=frozenset({"pipe_code"}))}
+ )
+ mocker.patch("pipelex.tools.log.log_redaction._redact_record", side_effect=RecursionError("too deep"))
+ buffer = io.StringIO()
+ with installed_log(sink=console_sink_on_buffer(buffer=buffer)) as fresh:
+ fresh.info("Pipe run starts", fields={"pipe_code": "compose_company"}, layout=LogLayout.PIPE_RUN)
+
+ rendered = buffer.getvalue()
+ assert f"{QUARANTINE_PREFIX}RecursionError] pipe_code={REDACTED_TEXT}" in rendered
+ assert "compose_company" not in rendered
+
+ def test_a_layout_name_nobody_registered_renders_the_message(self) -> None:
+ text = rendered_text(record=record_with_fields(message="Plain", extra={"files": 7}, layout="no_such_layout"))
+
+ assert text.plain == "🧠: Plain files=7"
+
+ def test_every_layout_a_call_can_name_is_registered(self) -> None:
+ assert set(CONSOLE_LAYOUTS) == set(LogLayout)
+
+ @pytest.mark.parametrize("template", ["{pipe.code}", "{pipe_code:>10}", "{pipe_code!r}", "{0}", "{codes[0]}"])
+ def test_a_placeholder_other_than_a_bare_name_is_refused_at_registration(self, template: str) -> None:
+ with pytest.raises(ValueError, match="bare name"):
+ ConsoleLayout(template=template, presented_fields=frozenset())
+
+ def test_the_json_sink_writes_the_plain_message_and_the_fields_and_never_the_layout(self, caplog: pytest.LogCaptureFixture) -> None:
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ with installed_log(sink=JsonLogSink(stream=buffer)) as fresh:
+ fresh.info("Pipe run starts", fields={**PIPE_RUN_FIELDS, "pipe_code": "[red]x[/red]"}, layout=LogLayout.PIPE_RUN)
+
+ lines: list[dict[str, Any]] = [json.loads(line) for line in buffer.getvalue().splitlines() if line]
+ (line,) = [line for line in lines if line[LOGGER_KEY] == __name__]
+ assert line[MESSAGE_KEY] == "Pipe run starts"
+ assert {key: line[key] for key in PIPE_RUN_FIELDS} == {**PIPE_RUN_FIELDS, "pipe_code": "[red]x[/red]"}
+ assert LAYOUT_MARK not in line
+ assert FIELD_NAMES_MARK not in line
+ assert LogLayout.PIPE_RUN not in line.values()
diff --git a/tests/unit/pipelex/tools/test_log_console_sink.py b/tests/unit/pipelex/tools/test_log_console_sink.py
index 9bc16f6e0..efc9a7f45 100644
--- a/tests/unit/pipelex/tools/test_log_console_sink.py
+++ b/tests/unit/pipelex/tools/test_log_console_sink.py
@@ -1,8 +1,9 @@
-"""The ``console`` sink renders byte for byte what the Rich handler rendered before it existed.
+"""The ``console`` sink renders byte for byte what the Rich handler rendered before it existed, plus the fields.
The reference is the handler ``log.configure`` used to build inline: a ``RichHandler`` fed every
``[runtime.log.rich_log]`` setting and the emoji formatter. Both handlers are pointed at the same kind
-of non-terminal console and handed the same fixed record set, and their output must be identical.
+of non-terminal console and handed the same fixed record set. A record whose call gave no field renders
+identically through both; a record with fields renders the same line with the fields after the message.
"""
from __future__ import annotations
@@ -18,11 +19,18 @@ from rich.logging import RichHandler
from pipelex.system.configuration.config_loader import ConfigLoader
from pipelex.system.console_target import ConsoleTarget
+from pipelex.tools.log.console_layouts import LogLayout
from pipelex.tools.log.console_log_sink import ConsoleLogSink
from pipelex.tools.log.log_config import HighlighterName, LogConfig, RichLogConfig
from pipelex.tools.log.log_fields import VERBATIM_MARK, attach_log_record_extra
from pipelex.tools.log.log_formatter import EmojiLogFormatter
from pipelex.tools.misc.toml_utils import load_toml_from_path
+from tests.helpers.console_log_rendering import (
+ PIPE_RUN_FIELDS,
+ console_sink_on_buffer,
+ installed_log,
+ package_rich_log_config_without_rich_tracebacks,
+)
CONSOLE_WIDTH = 100
@@ -57,9 +65,13 @@ def _reference_handler(*, config: RichLogConfig) -> logging.Handler:
return handler
-def _fixed_record_set() -> list[logging.LogRecord]:
- """Every shape a record takes: a bare line, a warning with fields, structured content, an error with a traceback, a foreign logger."""
+def _fixed_record_set(*, is_with_call_fields: bool = True) -> list[logging.LogRecord]:
+ """Every shape a record takes: a bare line, a warning with fields, structured content, an error with a traceback, a foreign logger.
+
+ Without call fields, the warning carries the run identifier alone, which the console never shows.
+ """
records: list[logging.LogRecord] = []
+ warning_extra: dict[str, Any] = {"files": 7, "request_id": "r1"} if is_with_call_fields else {"request_id": "r1"}
def record(*, name: str, level: int, message: str, exc_info: Any = None, extra: dict[str, Any] | None = None) -> None:
built = logging.LogRecord(name=name, level=level, pathname="/repo/pipelex/module.py", lineno=42, msg=message, args=(), exc_info=exc_info)
@@ -70,7 +82,7 @@ def _fixed_record_set() -> list[logging.LogRecord]:
records.append(built)
record(name="pipelex.pipe_operators.pipe_llm", level=logging.INFO, message="Running the pipe")
- record(name="pipelex.pipe_operators.pipe_llm", level=logging.WARNING, message="Slow backend", extra={"files": 7, "request_id": "r1"})
+ record(name="pipelex.pipe_operators.pipe_llm", level=logging.WARNING, message="Slow backend", extra=warning_extra)
record(name="pipelex.pipeline.pipe_run", level=logging.INFO, message='Config:\n{\n "key": "value"\n}', extra={"data": {"key": "value"}})
try:
msg = "boom"
@@ -97,29 +109,48 @@ def _render_one(*, config: RichLogConfig, message: str, extra: dict[str, Any] |
return buffer.getvalue()
-def _render(handler: logging.Handler) -> str:
+def _render(handler: logging.Handler, *, is_with_call_fields: bool = True) -> str:
buffer = io.StringIO()
rich_handler = handler
assert isinstance(rich_handler, RichHandler)
rich_handler.console = Console(file=buffer, width=CONSOLE_WIDTH, force_terminal=False, color_system=None, legacy_windows=False)
- for record in _fixed_record_set():
+ for record in _fixed_record_set(is_with_call_fields=is_with_call_fields):
handler.handle(record)
return buffer.getvalue()
class TestConsoleLogSink:
- def test_output_is_byte_identical_to_the_handler_configure_used_to_build(self) -> None:
- config = _package_rich_log_config()
- reference = _render(_reference_handler(config=config))
+ @pytest.mark.parametrize("is_rich_tracebacks", [True, False], ids=["rich tracebacks", "tracebacks as text"])
+ def test_output_is_byte_identical_to_the_handler_configure_used_to_build_when_no_call_gave_a_field(self, is_rich_tracebacks: bool) -> None:
+ """The run identifier and the structured content's ``data`` are attached too, and neither adds anything to the line.
+
+ With Rich tracebacks off, the traceback reaches the handler as text after the message, which the sink
+ splits off and prints under the line: a traceback with nothing Rich would read as markup prints as it did.
+ """
+ config = _package_rich_log_config().model_copy(update={"is_rich_tracebacks": is_rich_tracebacks})
+ reference = _render(_reference_handler(config=config), is_with_call_fields=False)
sink = ConsoleLogSink(rich_log_config=config, target=ConsoleTarget.STDERR)
- rendered = _render(sink.handler)
+ rendered = _render(sink.handler, is_with_call_fields=False)
assert rendered
assert "🧠: Running the pipe" in rendered
assert "ValueError" in rendered
assert rendered == reference
+ def test_a_record_with_fields_gets_them_after_the_message_and_every_other_line_is_unchanged(self) -> None:
+ config = _package_rich_log_config()
+ reference = _render(_reference_handler(config=config))
+ sink = ConsoleLogSink(rich_log_config=config, target=ConsoleTarget.STDERR)
+
+ rendered = _render(sink.handler)
+
+ assert "🧠: Slow backend files=7" in rendered
+ assert "r1" not in rendered
+ # The suffix takes the padding the message column had after the message, so the line keeps its width.
+ suffix = " files=7"
+ assert rendered == reference.replace(f"🧠: Slow backend{' ' * len(suffix)}", f"🧠: Slow backend{suffix}")
+
def test_the_prefix_stays_on_a_line_that_carries_a_traceback(self) -> None:
"""The Rich handler renders such a line from ``formatMessage`` alone, and the emoji must survive that path too."""
config = _package_rich_log_config()
@@ -131,6 +162,74 @@ class TestConsoleLogSink:
assert "🧠: Failed" in rendered
assert "ValueError: boom" in rendered
+ @pytest.mark.parametrize(
+ ("layout", "fields", "expected_line"),
+ [
+ (None, {"attempt": 2}, "Pipe run failed attempt=2"),
+ (LogLayout.PIPE_RUN, {**PIPE_RUN_FIELDS, "attempt": 2}, "PipeCompose: compose_company → Company attempt=2"),
+ ],
+ ids=["the message", "a layout"],
+ )
+ def test_with_tracebacks_as_text_the_fields_stay_on_the_line_and_the_traceback_prints_under_it(
+ self,
+ caplog: pytest.LogCaptureFixture,
+ layout: LogLayout | None,
+ fields: dict[str, Any],
+ expected_line: str,
+ ) -> None:
+ """With Rich tracebacks off, the formatter appends the traceback to the message the handler receives.
+
+ The suffix used to follow that text, so the fields read as part of the exception, and a layout, which
+ replaces the message, dropped the traceback whole. The exception's message is shaped like markup,
+ which the traceback text must never be read as.
+ """
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ sink = console_sink_on_buffer(buffer=buffer, rich_log_config=package_rich_log_config_without_rich_tracebacks())
+ with installed_log(sink=sink) as fresh:
+ try:
+ msg = "no such file: [/etc/pipelex.toml]"
+ raise ValueError(msg)
+ except ValueError:
+ fresh.error("Pipe run failed", include_exception=True, fields=fields, layout=layout)
+
+ lines = buffer.getvalue().splitlines()
+ (index_line,) = [index_candidate for index_candidate, line in enumerate(lines) if expected_line in line]
+ lines_under = lines[index_line + 1 :]
+ assert any("Traceback (most recent call last):" in line for line in lines_under)
+ assert any("ValueError: no such file: [/etc/pipelex.toml]" in line for line in lines_under)
+ assert not any("attempt=" in line for line in lines_under)
+
+ @pytest.mark.parametrize(
+ ("layout", "fields"),
+ [
+ (None, {"files": 7}),
+ (LogLayout.PIPE_RUN, {**PIPE_RUN_FIELDS, "files": 7}),
+ ],
+ ids=["the message", "a layout"],
+ )
+ def test_with_tracebacks_as_text_a_message_ending_in_a_line_break_still_has_its_traceback_start_a_line(
+ self,
+ caplog: pytest.LogCaptureFixture,
+ layout: LogLayout | None,
+ fields: dict[str, Any],
+ ) -> None:
+ """The formatter adds a line break before the exception text only when the message lacks one, so it ran on straight after the suffix."""
+ caplog.set_level(logging.INFO, logger=__name__)
+ buffer = io.StringIO()
+ sink = console_sink_on_buffer(buffer=buffer, rich_log_config=package_rich_log_config_without_rich_tracebacks())
+ with installed_log(sink=sink) as fresh:
+ try:
+ msg = "boom"
+ raise ValueError(msg)
+ except ValueError:
+ fresh.error("Pipe run failed\n", include_exception=True, fields=fields, layout=layout)
+
+ lines = buffer.getvalue().splitlines()
+ (suffix_line,) = [line for line in lines if "files=7" in line]
+ assert "Traceback" not in suffix_line
+ assert any(line.strip() == "Traceback (most recent call last):" for line in lines)
+
@pytest.mark.parametrize(
"message",
[
diff --git a/tests/unit/pipelex/tools/test_log_fields.py b/tests/unit/pipelex/tools/test_log_fields.py
index 7c836fb79..f2ec57212 100644
--- a/tests/unit/pipelex/tools/test_log_fields.py
+++ b/tests/unit/pipelex/tools/test_log_fields.py
@@ -14,8 +14,19 @@ from typing import TYPE_CHECKING, Any
import pytest
from pipelex import log
+from pipelex.tools.log.console_layouts import LogLayout
from pipelex.tools.log.log_context import get_log_context
-from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD, FORWARDED_MARK, VERBATIM_MARK, carried_attributes
+from pipelex.tools.log.log_fields import (
+ COLLIDING_FIELD_PREFIX,
+ DATA_FIELD,
+ FIELD_NAMES_MARK,
+ FORWARDED_MARK,
+ LAYOUT_MARK,
+ RICH_HIGHLIGHTER_ATTRIBUTE,
+ VERBATIM_MARK,
+ attached_field_names,
+ carried_attributes,
+)
from pipelex.tools.log.log_levels import LOGGING_LEVEL_VERBOSE
if TYPE_CHECKING:
@@ -32,6 +43,13 @@ def _field(record: logging.LogRecord, *, name: str) -> Any:
return getattr(record, name)
+def _circular_mapping() -> dict[str, Any]:
+ """A mapping that contains itself, which JSON refuses outright."""
+ circular: dict[str, Any] = {}
+ circular["self"] = circular
+ return circular
+
+
class TestLogFields:
def test_fields_and_context_ride_the_record_as_attributes(self, caplog: pytest.LogCaptureFixture) -> None:
with caplog.at_level(logging.INFO), log.context(request_id="r1", pipeline_run_id="p1"):
@@ -238,24 +256,39 @@ class TestLogFields:
assert FORWARDED_MARK not in carried
assert carried[f"{COLLIDING_FIELD_PREFIX}{FORWARDED_MARK}"] is True
- def test_a_field_named_like_the_verbatim_mark_is_prefixed_and_reaches_no_sink(self, caplog: pytest.LogCaptureFixture) -> None:
- """The mark is Rich's per-record markup override, which the console handler reads ahead of its own setting.
+ @pytest.mark.parametrize("name", [VERBATIM_MARK, RICH_HIGHLIGHTER_ATTRIBUTE])
+ def test_a_field_named_like_a_rich_per_record_override_is_prefixed_and_reaches_no_sink(self, caplog: pytest.LogCaptureFixture, name: str) -> None:
+ """``markup`` and ``highlighter`` are what Rich's console handler reads off a record ahead of its own settings.
- Left unreserved, a caller naming it would decide whether the console interprets markup for that
- line, and the value would ride into the `json` and `otlp` payloads as a field, where it means
- nothing. Reserved, the caller's value is carried under the prefix and is not a carried attribute.
+ Left unreserved, a caller naming ``markup`` would decide whether the console interprets markup for
+ that line, and one naming ``highlighter`` would have its value called as the highlighter, which
+ raises for a string and loses the line. Either value would also ride into the `json` and `otlp`
+ payloads as a field, where it means nothing. Reserved, the caller's value is carried under the
+ prefix and is not a carried attribute.
"""
with caplog.at_level(logging.INFO):
- log.info("not the console's business", fields={VERBATIM_MARK: True, "safe": 1})
+ log.info("not the console's business", fields={name: True, "safe": 1})
(record,) = _own_records(caplog)
- assert not hasattr(record, VERBATIM_MARK)
- assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{VERBATIM_MARK}") is True
+ assert not hasattr(record, name)
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{name}") is True
carried = carried_attributes(record=record)
- assert VERBATIM_MARK not in carried
- assert carried[f"{COLLIDING_FIELD_PREFIX}{VERBATIM_MARK}"] is True
+ assert name not in carried
+ assert carried[f"{COLLIDING_FIELD_PREFIX}{name}"] is True
assert carried["safe"] == 1
+ def test_what_a_third_party_stamped_as_rich_s_highlighter_is_no_carried_field(self, caplog: pytest.LogCaptureFixture) -> None:
+ """A library steering the console through ``extra`` puts an object there that means nothing on a wire.
+
+ Carried, the scrub would also turn that object into text, which Rich would then fail to call.
+ """
+ with caplog.at_level(logging.INFO):
+ log.info("a line", fields={"safe": 1})
+
+ (record,) = _own_records(caplog)
+ setattr(record, RICH_HIGHLIGHTER_ATTRIBUTE, object())
+ assert carried_attributes(record=record) == {"safe": 1}
+
def test_a_record_the_error_path_stamped_verbatim_hands_no_such_field_to_a_sink(self, caplog: pytest.LogCaptureFixture) -> None:
"""`TracebackMessageError` stamps the mark itself, and a structured sink must not write it as a field."""
with caplog.at_level(logging.INFO):
@@ -265,6 +298,51 @@ class TestLogFields:
setattr(record, VERBATIM_MARK, False)
assert carried_attributes(record=record) == {"safe": 1}
+ def test_the_names_the_call_attached_are_recorded_in_order_and_are_no_field(self, caplog: pytest.LogCaptureFixture) -> None:
+ """The console renders what the call attached and nothing else on the record, so the names are recorded where they landed."""
+ with caplog.at_level(logging.INFO), log.context(request_id="r1"):
+ log.info("recorded", fields={"files": 7, "name": "alpha"})
+
+ (record,) = _own_records(caplog)
+ assert attached_field_names(record=record) == ("request_id", "files", f"{COLLIDING_FIELD_PREFIX}name")
+ assert FIELD_NAMES_MARK not in carried_attributes(record=record)
+
+ def test_the_layout_a_call_names_rides_the_record_under_its_mark_and_is_no_field(self, caplog: pytest.LogCaptureFixture) -> None:
+ with caplog.at_level(logging.INFO):
+ log.info("laid out", fields={"pipe_code": "main"}, layout=LogLayout.PIPE_RUN)
+ log.info("not laid out", fields={"pipe_code": "main"})
+
+ laid_out, plain = _own_records(caplog)
+ assert getattr(laid_out, LAYOUT_MARK) == LogLayout.PIPE_RUN
+ assert not hasattr(plain, LAYOUT_MARK)
+ assert carried_attributes(record=laid_out) == {"pipe_code": "main"}
+
+ @pytest.mark.parametrize(
+ "make_content",
+ [lambda: {"key": "value"}, lambda: ["a", "b"], lambda: None, _circular_mapping],
+ ids=["a mapping", "a list", "none", "a mapping JSON refuses"],
+ )
+ def test_a_layout_rides_a_string_content_only(self, caplog: pytest.LogCaptureFixture, make_content: Callable[[], Any]) -> None:
+ """Any other content renders in the message alone, which a layout replaces, so the dispatch drops the layout there."""
+ with caplog.at_level(logging.INFO):
+ log.info(make_content(), fields={"pipe_code": "main"}, layout=LogLayout.PIPE_RUN)
+
+ (record,) = _own_records(caplog)
+ assert not hasattr(record, LAYOUT_MARK)
+ assert _field(record, name="pipe_code") == "main"
+
+ @pytest.mark.parametrize("mark", [FIELD_NAMES_MARK, LAYOUT_MARK])
+ def test_a_field_named_like_a_console_mark_is_prefixed_and_steers_nothing(self, caplog: pytest.LogCaptureFixture, mark: str) -> None:
+ """Left unreserved, a caller's field would hand the console a list of names or pick the line's layout."""
+ with caplog.at_level(logging.INFO):
+ log.info("impersonation", fields={mark: "pipe_run", "safe": 1})
+
+ (record,) = _own_records(caplog)
+ assert getattr(record, f"{COLLIDING_FIELD_PREFIX}{mark}") == "pipe_run"
+ assert not hasattr(record, LAYOUT_MARK)
+ assert attached_field_names(record=record) == (f"{COLLIDING_FIELD_PREFIX}{mark}", "safe")
+ assert carried_attributes(record=record) == {f"{COLLIDING_FIELD_PREFIX}{mark}": "pipe_run", "safe": 1}
+
def test_the_forwarding_marker_a_record_already_carries_is_not_a_carried_field(self, caplog: pytest.LogCaptureFixture) -> None:
"""A structured sink reads the record's own fields; the marker is machinery and belongs on no wire."""
with caplog.at_level(logging.INFO):
diff --git a/tests/unit/pipelex/tools/test_log_redaction.py b/tests/unit/pipelex/tools/test_log_redaction.py
index 2235dfb05..399986fbf 100644
--- a/tests/unit/pipelex/tools/test_log_redaction.py
+++ b/tests/unit/pipelex/tools/test_log_redaction.py
@@ -18,9 +18,10 @@ from pydantic import BaseModel, field_serializer
from typing_extensions import override
from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.tools.log.console_layouts import LogLayout
from pipelex.tools.log.log import Log
from pipelex.tools.log.log_config import LogConfig, LogRedactionConfig
-from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD
+from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, DATA_FIELD, LAYOUT_MARK
from pipelex.tools.log.log_redaction import (
ARGUMENTS_WITHHELD_TEXT,
CYCLE_TEXT,
@@ -367,6 +368,7 @@ class TestLogRedaction:
)
record.payload = {"nested": "sk_live_0123456789abcdef"}
setattr(record, DATA_FIELD, ["sk_live_0123456789abcdef"])
+ record.__dict__[LAYOUT_MARK] = LogLayout.PIPE_RUN
processor = make_redaction_processor(config=LogRedactionConfig(is_enabled=True, extra_patterns=[]))
with pytest.raises(RecursionError):
@@ -376,6 +378,8 @@ class TestLogRedaction:
assert record.args == ()
assert getattr(record, FIELD_NAME) == REDACTED_TEXT
assert getattr(record, DATA_FIELD) == REDACTED_TEXT
+ # A layout renders in place of the message, so one left on the record would hide the notice.
+ assert LAYOUT_MARK not in record.__dict__
assert record.exc_info is None
assert record.exc_text is None
assert "sk_live" not in logging.Formatter().format(record)