Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

chore(agent-cli): finalize Phase 3 (tomlkit + stderr leniency) and fold into v0.30.1 CHANGELOG

Louis Choquela392a1188254
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e0c8ebaac..e6318fa8a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,10 +1,14 @@
# Changelog
-## [v0.30.1] - 2026-05-25
+## [v0.30.1] - 2026-05-26
### Fixed
-- **`pipelex-agent` now silences all pipelex logs on stderr regardless of user TOML.** The agent CLI is machine-consumed: stdout is reserved for the structured success envelope (JSON / markdown) and stderr for the structured error envelope. Free-floating `log.*` calls — even a single `log.debug` from `telemetry_factory.py` or a `log.warning` from `validation_error_categorizer.py` — would corrupt the stderr channel for downstream parsers (`mthds-js`'s `PipelexRunner` doing `JSON.parse(stderr)` on the validate hook). `make_pipelex_for_agent_cli` now injects `config_overrides` into `Pipelex.make()` that pin `default_log_level = OFF` and `package_log_levels.pipelex = OFF` from the very first `log.configure` call — so a user setting `[pipelex.log_config.package_log_levels] pipelex = "DEBUG"` in `~/.pipelex/pipelex.toml` can no longer leak DEBUG/INFO/WARNING lines onto the agent CLI's stderr channel. The DEBUG log line in `telemetry_factory.py:77` itself is unchanged — it's still useful diagnostic info for the human `pipelex` CLI.
+- **`pipelex-agent` now silences every Python logger on stderr regardless of user TOML.** The agent CLI is machine-consumed: stdout is reserved for the structured success envelope (JSON / markdown) and stderr for the structured error envelope. Free-floating `log.*` calls — a `log.debug` from `telemetry_factory.py`, a `log.warning` from `validation_error_categorizer.py`, or an INFO/WARNING line from any third-party dep (`anthropic`, `httpx`, `botocore`, `openai`, anything a transitive dep configures) — would corrupt the stderr channel for downstream parsers (`mthds-js`'s `PipelexRunner` doing `JSON.parse(stderr)` on the validate hook). Two layers, defense-in-depth:
+ - **Layer 1 — pin pipelex's own logs off via config.** `make_pipelex_for_agent_cli` injects `config_overrides` into `Pipelex.make()` that pin `default_log_level = OFF` and `package_log_levels.pipelex = OFF` from the very first `log.configure` call. A user setting `[pipelex.log_config.package_log_levels] pipelex = "DEBUG"` in `~/.pipelex/pipelex.toml` can no longer leak its own DEBUG/INFO/WARNING lines.
+ - **Layer 2 — process-global cutoff that covers every logger.** New `silence_logging_for_agent_cli()` calls `logging.disable(sys.maxsize)` — a process-global threshold checked inside `Logger.isEnabledFor` BEFORE any per-logger level. No record gets created for any logger at any level (including custom levels above `CRITICAL`), regardless of which package emits or what level the user configured. Wired into the Typer `app_callback` so every subcommand — including `init` and `accept-gateway-terms`, which bypass `make_pipelex_for_agent_cli` — silences logging before any command body runs; idempotent per-call invocations remain inside `make_pipelex_for_agent_cli` and `agent_doctor_cmd` as belt-and-braces for direct library callers.
+
+ A new e2e regression test pins the contract by setting `anthropic`, `httpx`, `botocore`, `openai` to `DEBUG` in the user TOML and asserting both stdout and stderr stay clean.
### Changed
diff --git a/TODOS.md b/TODOS.md
index f8ba6e989..5df466da3 100644
--- a/TODOS.md
+++ b/TODOS.md
@@ -87,23 +87,23 @@ Production-code changes are done. Test-only work remains.
## Phase 3 — Test/helper robustness (independent improvements)
-- [ ] **Step 3.1 (fixes #13)** — In `tests/e2e/agent_cli/test_stdout_is_clean_json.py::test_models_json_stdout_and_stderr_stay_clean_under_third_party_logger_enabled`, replace the `assert result.stderr == ""` with: stderr must be either empty OR parse as a structured error envelope (a dict with `error` field). This catches log-line leaks (which would not be valid JSON) while tolerating environmental noise like ResourceWarnings. Helper: try `json.loads(result.stderr)` and accept either parse-success-with-envelope-shape or empty.
-- [ ] **Step 3.2 (fixes #7, #8, #15)** — Rewrite `_set_package_log_level` in `tests/e2e/agent_cli/test_stdout_is_clean_json.py` using `tomlkit` (already a project dep — see `init_cmd.py`'s usage). Pseudo:
+- [x] **Step 3.1 (fixes #13)** — In `tests/e2e/agent_cli/test_stdout_is_clean_json.py::test_models_json_stdout_and_stderr_stay_clean_under_third_party_logger_enabled`, replace the `assert result.stderr == ""` with: stderr must be either empty OR parse as a structured error envelope (a dict with `error` field). This catches log-line leaks (which would not be valid JSON) while tolerating environmental noise like ResourceWarnings. Helper: try `json.loads(result.stderr)` and accept either parse-success-with-envelope-shape or empty. **Done:** added `_assert_stderr_is_clean_or_structured_envelope` helper; the test now calls it in place of the strict empty-string assertion.
+- [x] **Step 3.2 (fixes #7, #8, #15)** — Rewrite `_set_package_log_level` in `tests/e2e/agent_cli/test_stdout_is_clean_json.py` using `tomlkit` (already a project dep — see `init_cmd.py`'s usage). Pseudo:
```python
import tomlkit
doc = tomlkit.parse(pipelex_toml_path.read_text())
doc["pipelex"]["log_config"]["package_log_levels"][package_name] = level
pipelex_toml_path.write_text(tomlkit.dumps(doc))
```
- This eliminates all three line-based-rewriter edges (duplicate section, no-space matcher, no-trailing-newline) in one pass. Keep the `_set_pipelex_package_log_level_to_debug` compat shim so the original two callers don't change.
-- [ ] **Step 3.3** — Sanity check: read the rewritten test file to confirm the section path `["pipelex"]["log_config"]["package_log_levels"]` exists in the kit's `pipelex.toml` (it does, line 177 — `[pipelex.log_config.package_log_levels]`).
+ This eliminates all three line-based-rewriter edges (duplicate section, no-space matcher, no-trailing-newline) in one pass. Keep the `_set_pipelex_package_log_level_to_debug` compat shim so the original two callers don't change. **Done:** used the project's `load_toml_with_tomlkit` + `save_toml_to_path` helpers from `pipelex.tools.misc.toml_utils` (proper type stubs, same pair `init_cmd` uses) instead of raw tomlkit — pyright was unhappy with the raw `tomlkit.dumps` overload. Compat shim preserved.
+- [x] **Step 3.3** — Sanity check: read the rewritten test file to confirm the section path `["pipelex"]["log_config"]["package_log_levels"]` exists in the kit's `pipelex.toml` (it does, line 177 — `[pipelex.log_config.package_log_levels]`). **Confirmed via grep on both `pipelex/pipelex.toml:98` and `pipelex/kit/configs/pipelex.toml:177`.**
### ✅ CHECKPOINT 3 — final verification
-- [ ] Run `make agent-check`. Must be clean.
-- [ ] Run targeted: `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/cli/ tests/unit/pipelex/tools/ tests/integration/pipelex/cli/ tests/e2e/agent_cli/`. All pass.
-- [ ] Run `make tb`. Pass.
-- [ ] Skim `git diff HEAD --stat` — the diff should now cover: `_agent_cli.py` (new silence wire), `agent_cli_factory.py` (sys.maxsize), `test_agent_cli_factory.py` (fixture), `test_agent_cli_factory_suppression.py` + `test_agent_cli_output_discipline.py` (per-logger restore), `test_stdout_is_clean_json.py` (relaxed assertion + tomlkit rewrite).
+- [x] Run `make agent-check`. Must be clean. **Result:** 0 pyright errors, mypy success on 1893 files.
+- [x] Run targeted: `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/cli/ tests/unit/pipelex/tools/ tests/integration/pipelex/cli/ tests/e2e/agent_cli/`. All pass. **Result:** 1541 passed, 1 skipped (pre-existing `test_toml_utils.py:61` skip — unrelated) in 33.33s.
+- [x] Run `make tb`. Pass. **Result:** 2 passed, 6708 deselected in 6.13s.
+- [x] Skim `git diff HEAD --stat` — the diff should now cover: `_agent_cli.py` (new silence wire), `agent_cli_factory.py` (sys.maxsize), `test_agent_cli_factory.py` (fixture), `test_agent_cli_factory_suppression.py` + `test_agent_cli_output_discipline.py` (per-logger restore), `test_stdout_is_clean_json.py` (relaxed assertion + tomlkit rewrite). **Note:** Phase 1 (commit `d98eb0e2`) and Phase 2 (commit `9563915f`) are already landed on the branch, so `git diff HEAD --stat` now shows only the Phase 3 file (`test_stdout_is_clean_json.py`) plus `TODOS.md`. The cumulative range vs `main`/release base covers every file listed.
---
@@ -122,7 +122,7 @@ The `/code-review` surfaced 15 findings; these 9 were explicitly skipped after t
When all phases are checked off:
-- [ ] `git diff HEAD --stat` shows the expected files (see Checkpoint 3 list).
-- [ ] All three e2e tests in `test_stdout_is_clean_json.py` pass.
-- [ ] No regression in `make agent-test` (full suite — run before commit per CLAUDE.md release/commit conventions).
-- [ ] Decide whether this work folds into the existing `release/v0.30.1` branch or warrants a follow-up release entry in CHANGELOG. (Note: per memory `feedback_no_unreleased_header.md`, use `[Unreleased]` on work branches — replace with versioned entry only at release time. The current branch is already named `release/v0.30.1` so check with user before bumping.)
+- [x] `git diff HEAD --stat` shows the expected files (see Checkpoint 3 list). **Cumulative diff across the 3 commits on this branch since `Release v0.30.1` (6902952c): `_agent_cli.py`, `agent_cli_factory.py`, `doctor_cmd.py`, `test_stdout_is_clean_json.py`, `test_agent_cli_factory.py`, `test_agent_cli_factory_init_overrides.py`, `test_agent_cli_factory_suppression.py`, `test_agent_cli_output_discipline.py`, `test_agent_doctor_cmd.py`, `TODOS.md`, `CHANGELOG.md`.**
+- [x] All three e2e tests in `test_stdout_is_clean_json.py` pass. **Result:** 3 passed in 3.73s.
+- [x] No regression in `make agent-test` (full suite — run before commit per CLAUDE.md release/commit conventions). **Result:** "All tests passed." — full suite clean.
+- [x] Decide whether this work folds into the existing `release/v0.30.1` branch or warrants a follow-up release entry in CHANGELOG. **Decision (user-confirmed):** fold into the existing v0.30.1 CHANGELOG entry. v0.30.1 is not yet tagged (only v0.30.0 exists), so the entry was still safely editable. Appended a second `### Fixed` bullet covering the Phase 1-3 work (process-global `logging.disable(sys.maxsize)`, `app_callback` wiring, third-party-logger leak regression test) and bumped the v0.30.1 date from 2026-05-25 → 2026-05-26.
diff --git a/tests/e2e/agent_cli/test_stdout_is_clean_json.py b/tests/e2e/agent_cli/test_stdout_is_clean_json.py
index 652270e57..c6905d22d 100644
--- a/tests/e2e/agent_cli/test_stdout_is_clean_json.py
+++ b/tests/e2e/agent_cli/test_stdout_is_clean_json.py
@@ -35,6 +35,7 @@ from typing import TYPE_CHECKING, cast
import pytest
+from pipelex.tools.misc.toml_utils import load_toml_with_tomlkit, save_toml_to_path
from tests.e2e.agent_cli.conftest import PIPELEX_AGENT_BIN
if TYPE_CHECKING:
@@ -44,47 +45,16 @@ if TYPE_CHECKING:
def _set_package_log_level(pipelex_toml_path: Path, *, package_name: str, level: str) -> None:
"""Set ``<package_name> = "<level>"`` inside ``[pipelex.log_config.package_log_levels]``.
- If the package key already exists in the section, rewrites its value. Otherwise
- appends a new line at the end of the section. Either way the user's TOML ends up
- explicitly carrying the level, which is what we need to exercise the
- "user-configured third-party logger" leak scenario.
-
- Asserts on missing-section so a future kit refactor surfaces here instead of
- silently neutering the test.
+ Uses the project's ``tomlkit``-based helpers (the same pair ``init_cmd`` relies
+ on) to parse-edit-dump the file. This handles every edge case the previous
+ line-based rewriter had to special-case (duplicate section headers, whitespace
+ variants in key lines, missing trailing newlines) — tomlkit's parser is the
+ source of truth for what the agent CLI ultimately reads back via Pipelex's
+ config loader.
"""
- original_text = pipelex_toml_path.read_text(encoding="utf-8")
- lines = original_text.splitlines(keepends=True)
- target_section_start: int | None = None
- target_section_end: int | None = None
- in_target_section = False
- for index, line in enumerate(lines):
- stripped = line.strip()
- if stripped.startswith("[") and stripped.endswith("]"):
- if in_target_section:
- target_section_end = index
- in_target_section = False
- if stripped == "[pipelex.log_config.package_log_levels]":
- in_target_section = True
- target_section_start = index
- if in_target_section:
- target_section_end = len(lines)
- if target_section_start is None or target_section_end is None:
- msg = f"Could not find [pipelex.log_config.package_log_levels] in {pipelex_toml_path}"
- raise AssertionError(msg)
-
- new_line = f'{package_name} = "{level}"\n'
- for index in range(target_section_start + 1, target_section_end):
- if lines[index].strip().startswith(f"{package_name} "):
- lines[index] = new_line
- break
- else:
- # Insert before the first trailing blank line so the section's contents stay grouped.
- insert_index = target_section_end
- while insert_index > target_section_start + 1 and lines[insert_index - 1].strip() == "":
- insert_index -= 1
- lines.insert(insert_index, new_line)
-
- pipelex_toml_path.write_text("".join(lines), encoding="utf-8")
+ doc = load_toml_with_tomlkit(pipelex_toml_path)
+ doc["pipelex"]["log_config"]["package_log_levels"][package_name] = level # type: ignore[index]
+ save_toml_to_path(doc, pipelex_toml_path)
def _set_pipelex_package_log_level_to_debug(pipelex_toml_path: Path) -> None:
@@ -92,6 +62,36 @@ def _set_pipelex_package_log_level_to_debug(pipelex_toml_path: Path) -> None:
_set_package_log_level(pipelex_toml_path, package_name="pipelex", level="DEBUG")
+def _assert_stderr_is_clean_or_structured_envelope(stderr: str) -> None:
+ """Assert that ``stderr`` is either empty or a parseable structured error envelope.
+
+ The agent CLI's contract is: stderr is reserved for the structured error envelope
+ (a JSON object with an ``error`` field). The strictest catch on a third-party
+ logger leak is: any free-floating log line on stderr would NOT be valid JSON, so
+ ``json.loads`` fails. This relaxation lets the test tolerate the edge case where
+ the envelope itself legitimately appears on stderr (which would never happen on a
+ successful run, but it absorbs unrelated environmental noise that still parses
+ cleanly), while still failing loudly on the log-leak shape we actually guard
+ against.
+ """
+ if stderr == "":
+ return
+ try:
+ parsed = json.loads(stderr)
+ except json.JSONDecodeError as exc:
+ msg = (
+ f"stderr must be either empty or a parseable JSON error envelope — any other "
+ f"content (log line, warning, print) means something leaked through the agent "
+ f"CLI's stdout/stderr discipline.\nstderr={stderr!r}\nparse error={exc!s}"
+ )
+ raise AssertionError(msg) from exc
+ if not (isinstance(parsed, dict) and "error" in parsed):
+ msg = (
+ f"stderr parsed as JSON but is not the structured error envelope shape (expected a dict containing an ``error`` key).\nstderr={stderr!r}"
+ )
+ raise AssertionError(msg)
+
+
def _set_console_targets(pipelex_toml_path: Path, *, log_target: str, print_target: str) -> None:
"""Rewrite ``console_log_target`` and ``console_print_target`` directly under
``[pipelex.log_config]`` to the given values.
@@ -221,14 +221,10 @@ class TestAgentCliStdoutIsCleanJson:
# stdout must parse as JSON in one shot — no log line preceding/following the envelope.
parsed = json.loads(result.stdout)
assert isinstance(parsed, dict)
- # stderr must stay empty — any line here proves a third-party logger leaked through
- # the root RichHandler. Empty-stderr is the strictest possible assertion on the
- # "no log corrupts the error envelope" contract.
- assert result.stderr == "", (
- f"stderr must stay clean even when third-party loggers are bumped to DEBUG — "
- f"otherwise their records leak through the root's RichHandler and corrupt the "
- f"JSON error envelope.\nstderr={result.stderr!r}"
- )
+ # stderr must be either empty (the expected case for a successful run) or a
+ # parseable structured error envelope — anything else (log line, warning, print)
+ # proves a third-party logger leaked through the root RichHandler.
+ _assert_stderr_is_clean_or_structured_envelope(result.stderr)
def test_models_json_stdout_resists_user_targets_override_to_stdout(
self,