Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
chore(agent-cli): finalize Phase 3 (tomlkit + stderr leniency) and fold into v0.30.1 CHANGELOG
Louis Choquela392a1188254
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e0c8ebaac..e6318fa8a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,10 +1,14 @@# Changelog-## [v0.30.1] - 2026-05-25+## [v0.30.1] - 2026-05-26### Fixed-- **`pipelex-agent` now silences all pipelex logs on stderr regardless of user TOML.** The agent CLI is machine-consumed: stdout is reserved for the structured success envelope (JSON / markdown) and stderr for the structured error envelope. Free-floating `log.*` calls — even a single `log.debug` from `telemetry_factory.py` or a `log.warning` from `validation_error_categorizer.py` — would corrupt the stderr channel for downstream parsers (`mthds-js`'s `PipelexRunner` doing `JSON.parse(stderr)` on the validate hook). `make_pipelex_for_agent_cli` now injects `config_overrides` into `Pipelex.make()` that pin `default_log_level = OFF` and `package_log_levels.pipelex = OFF` from the very first `log.configure` call — so a user setting `[pipelex.log_config.package_log_levels] pipelex = "DEBUG"` in `~/.pipelex/pipelex.toml` can no longer leak DEBUG/INFO/WARNING lines onto the agent CLI's stderr channel. The DEBUG log line in `telemetry_factory.py:77` itself is unchanged — it's still useful diagnostic info for the human `pipelex` CLI.+- **`pipelex-agent` now silences every Python logger on stderr regardless of user TOML.** The agent CLI is machine-consumed: stdout is reserved for the structured success envelope (JSON / markdown) and stderr for the structured error envelope. Free-floating `log.*` calls — a `log.debug` from `telemetry_factory.py`, a `log.warning` from `validation_error_categorizer.py`, or an INFO/WARNING line from any third-party dep (`anthropic`, `httpx`, `botocore`, `openai`, anything a transitive dep configures) — would corrupt the stderr channel for downstream parsers (`mthds-js`'s `PipelexRunner` doing `JSON.parse(stderr)` on the validate hook). Two layers, defense-in-depth:+ - **Layer 1 — pin pipelex's own logs off via config.** `make_pipelex_for_agent_cli` injects `config_overrides` into `Pipelex.make()` that pin `default_log_level = OFF` and `package_log_levels.pipelex = OFF` from the very first `log.configure` call. A user setting `[pipelex.log_config.package_log_levels] pipelex = "DEBUG"` in `~/.pipelex/pipelex.toml` can no longer leak its own DEBUG/INFO/WARNING lines.+ - **Layer 2 — process-global cutoff that covers every logger.** New `silence_logging_for_agent_cli()` calls `logging.disable(sys.maxsize)` — a process-global threshold checked inside `Logger.isEnabledFor` BEFORE any per-logger level. No record gets created for any logger at any level (including custom levels above `CRITICAL`), regardless of which package emits or what level the user configured. Wired into the Typer `app_callback` so every subcommand — including `init` and `accept-gateway-terms`, which bypass `make_pipelex_for_agent_cli` — silences logging before any command body runs; idempotent per-call invocations remain inside `make_pipelex_for_agent_cli` and `agent_doctor_cmd` as belt-and-braces for direct library callers.++ A new e2e regression test pins the contract by setting `anthropic`, `httpx`, `botocore`, `openai` to `DEBUG` in the user TOML and asserting both stdout and stderr stay clean.### Changeddiff --git a/TODOS.md b/TODOS.mdindex f8ba6e989..5df466da3 100644--- a/TODOS.md+++ b/TODOS.md@@ -87,23 +87,23 @@ Production-code changes are done. Test-only work remains.## Phase 3 — Test/helper robustness (independent improvements)-- [ ] **Step 3.1 (fixes #13)** — In `tests/e2e/agent_cli/test_stdout_is_clean_json.py::test_models_json_stdout_and_stderr_stay_clean_under_third_party_logger_enabled`, replace the `assert result.stderr == ""` with: stderr must be either empty OR parse as a structured error envelope (a dict with `error` field). This catches log-line leaks (which would not be valid JSON) while tolerating environmental noise like ResourceWarnings. Helper: try `json.loads(result.stderr)` and accept either parse-success-with-envelope-shape or empty.-- [ ] **Step 3.2 (fixes #7, #8, #15)** — Rewrite `_set_package_log_level` in `tests/e2e/agent_cli/test_stdout_is_clean_json.py` using `tomlkit` (already a project dep — see `init_cmd.py`'s usage). Pseudo:+- [x] **Step 3.1 (fixes #13)** — In `tests/e2e/agent_cli/test_stdout_is_clean_json.py::test_models_json_stdout_and_stderr_stay_clean_under_third_party_logger_enabled`, replace the `assert result.stderr == ""` with: stderr must be either empty OR parse as a structured error envelope (a dict with `error` field). This catches log-line leaks (which would not be valid JSON) while tolerating environmental noise like ResourceWarnings. Helper: try `json.loads(result.stderr)` and accept either parse-success-with-envelope-shape or empty. **Done:** added `_assert_stderr_is_clean_or_structured_envelope` helper; the test now calls it in place of the strict empty-string assertion.+- [x] **Step 3.2 (fixes #7, #8, #15)** — Rewrite `_set_package_log_level` in `tests/e2e/agent_cli/test_stdout_is_clean_json.py` using `tomlkit` (already a project dep — see `init_cmd.py`'s usage). Pseudo:```pythonimport tomlkitdoc = tomlkit.parse(pipelex_toml_path.read_text())doc["pipelex"]["log_config"]["package_log_levels"][package_name] = levelpipelex_toml_path.write_text(tomlkit.dumps(doc))```- This eliminates all three line-based-rewriter edges (duplicate section, no-space matcher, no-trailing-newline) in one pass. Keep the `_set_pipelex_package_log_level_to_debug` compat shim so the original two callers don't change.-- [ ] **Step 3.3** — Sanity check: read the rewritten test file to confirm the section path `["pipelex"]["log_config"]["package_log_levels"]` exists in the kit's `pipelex.toml` (it does, line 177 — `[pipelex.log_config.package_log_levels]`).+ This eliminates all three line-based-rewriter edges (duplicate section, no-space matcher, no-trailing-newline) in one pass. Keep the `_set_pipelex_package_log_level_to_debug` compat shim so the original two callers don't change. **Done:** used the project's `load_toml_with_tomlkit` + `save_toml_to_path` helpers from `pipelex.tools.misc.toml_utils` (proper type stubs, same pair `init_cmd` uses) instead of raw tomlkit — pyright was unhappy with the raw `tomlkit.dumps` overload. Compat shim preserved.+- [x] **Step 3.3** — Sanity check: read the rewritten test file to confirm the section path `["pipelex"]["log_config"]["package_log_levels"]` exists in the kit's `pipelex.toml` (it does, line 177 — `[pipelex.log_config.package_log_levels]`). **Confirmed via grep on both `pipelex/pipelex.toml:98` and `pipelex/kit/configs/pipelex.toml:177`.**### ✅ CHECKPOINT 3 — final verification-- [ ] Run `make agent-check`. Must be clean.-- [ ] Run targeted: `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/cli/ tests/unit/pipelex/tools/ tests/integration/pipelex/cli/ tests/e2e/agent_cli/`. All pass.-- [ ] Run `make tb`. Pass.-- [ ] Skim `git diff HEAD --stat` — the diff should now cover: `_agent_cli.py` (new silence wire), `agent_cli_factory.py` (sys.maxsize), `test_agent_cli_factory.py` (fixture), `test_agent_cli_factory_suppression.py` + `test_agent_cli_output_discipline.py` (per-logger restore), `test_stdout_is_clean_json.py` (relaxed assertion + tomlkit rewrite).+- [x] Run `make agent-check`. Must be clean. **Result:** 0 pyright errors, mypy success on 1893 files.+- [x] Run targeted: `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/cli/ tests/unit/pipelex/tools/ tests/integration/pipelex/cli/ tests/e2e/agent_cli/`. All pass. **Result:** 1541 passed, 1 skipped (pre-existing `test_toml_utils.py:61` skip — unrelated) in 33.33s.+- [x] Run `make tb`. Pass. **Result:** 2 passed, 6708 deselected in 6.13s.+- [x] Skim `git diff HEAD --stat` — the diff should now cover: `_agent_cli.py` (new silence wire), `agent_cli_factory.py` (sys.maxsize), `test_agent_cli_factory.py` (fixture), `test_agent_cli_factory_suppression.py` + `test_agent_cli_output_discipline.py` (per-logger restore), `test_stdout_is_clean_json.py` (relaxed assertion + tomlkit rewrite). **Note:** Phase 1 (commit `d98eb0e2`) and Phase 2 (commit `9563915f`) are already landed on the branch, so `git diff HEAD --stat` now shows only the Phase 3 file (`test_stdout_is_clean_json.py`) plus `TODOS.md`. The cumulative range vs `main`/release base covers every file listed.---@@ -122,7 +122,7 @@ The `/code-review` surfaced 15 findings; these 9 were explicitly skipped after tWhen all phases are checked off:-- [ ] `git diff HEAD --stat` shows the expected files (see Checkpoint 3 list).-- [ ] All three e2e tests in `test_stdout_is_clean_json.py` pass.-- [ ] No regression in `make agent-test` (full suite — run before commit per CLAUDE.md release/commit conventions).-- [ ] Decide whether this work folds into the existing `release/v0.30.1` branch or warrants a follow-up release entry in CHANGELOG. (Note: per memory `feedback_no_unreleased_header.md`, use `[Unreleased]` on work branches — replace with versioned entry only at release time. The current branch is already named `release/v0.30.1` so check with user before bumping.)+- [x] `git diff HEAD --stat` shows the expected files (see Checkpoint 3 list). **Cumulative diff across the 3 commits on this branch since `Release v0.30.1` (6902952c): `_agent_cli.py`, `agent_cli_factory.py`, `doctor_cmd.py`, `test_stdout_is_clean_json.py`, `test_agent_cli_factory.py`, `test_agent_cli_factory_init_overrides.py`, `test_agent_cli_factory_suppression.py`, `test_agent_cli_output_discipline.py`, `test_agent_doctor_cmd.py`, `TODOS.md`, `CHANGELOG.md`.**+- [x] All three e2e tests in `test_stdout_is_clean_json.py` pass. **Result:** 3 passed in 3.73s.+- [x] No regression in `make agent-test` (full suite — run before commit per CLAUDE.md release/commit conventions). **Result:** "All tests passed." — full suite clean.+- [x] Decide whether this work folds into the existing `release/v0.30.1` branch or warrants a follow-up release entry in CHANGELOG. **Decision (user-confirmed):** fold into the existing v0.30.1 CHANGELOG entry. v0.30.1 is not yet tagged (only v0.30.0 exists), so the entry was still safely editable. Appended a second `### Fixed` bullet covering the Phase 1-3 work (process-global `logging.disable(sys.maxsize)`, `app_callback` wiring, third-party-logger leak regression test) and bumped the v0.30.1 date from 2026-05-25 → 2026-05-26.diff --git a/tests/e2e/agent_cli/test_stdout_is_clean_json.py b/tests/e2e/agent_cli/test_stdout_is_clean_json.pyindex 652270e57..c6905d22d 100644--- a/tests/e2e/agent_cli/test_stdout_is_clean_json.py+++ b/tests/e2e/agent_cli/test_stdout_is_clean_json.py@@ -35,6 +35,7 @@ from typing import TYPE_CHECKING, castimport pytest+from pipelex.tools.misc.toml_utils import load_toml_with_tomlkit, save_toml_to_pathfrom tests.e2e.agent_cli.conftest import PIPELEX_AGENT_BINif TYPE_CHECKING:@@ -44,47 +45,16 @@ if TYPE_CHECKING:def _set_package_log_level(pipelex_toml_path: Path, *, package_name: str, level: str) -> None:"""Set ``<package_name> = "<level>"`` inside ``[pipelex.log_config.package_log_levels]``.- If the package key already exists in the section, rewrites its value. Otherwise- appends a new line at the end of the section. Either way the user's TOML ends up- explicitly carrying the level, which is what we need to exercise the- "user-configured third-party logger" leak scenario.-- Asserts on missing-section so a future kit refactor surfaces here instead of- silently neutering the test.+ Uses the project's ``tomlkit``-based helpers (the same pair ``init_cmd`` relies+ on) to parse-edit-dump the file. This handles every edge case the previous+ line-based rewriter had to special-case (duplicate section headers, whitespace+ variants in key lines, missing trailing newlines) — tomlkit's parser is the+ source of truth for what the agent CLI ultimately reads back via Pipelex's+ config loader."""- original_text = pipelex_toml_path.read_text(encoding="utf-8")- lines = original_text.splitlines(keepends=True)- target_section_start: int | None = None- target_section_end: int | None = None- in_target_section = False- for index, line in enumerate(lines):- stripped = line.strip()- if stripped.startswith("[") and stripped.endswith("]"):- if in_target_section:- target_section_end = index- in_target_section = False- if stripped == "[pipelex.log_config.package_log_levels]":- in_target_section = True- target_section_start = index- if in_target_section:- target_section_end = len(lines)- if target_section_start is None or target_section_end is None:- msg = f"Could not find [pipelex.log_config.package_log_levels] in {pipelex_toml_path}"- raise AssertionError(msg)-- new_line = f'{package_name} = "{level}"\n'- for index in range(target_section_start + 1, target_section_end):- if lines[index].strip().startswith(f"{package_name} "):- lines[index] = new_line- break- else:- # Insert before the first trailing blank line so the section's contents stay grouped.- insert_index = target_section_end- while insert_index > target_section_start + 1 and lines[insert_index - 1].strip() == "":- insert_index -= 1- lines.insert(insert_index, new_line)-- pipelex_toml_path.write_text("".join(lines), encoding="utf-8")+ doc = load_toml_with_tomlkit(pipelex_toml_path)+ doc["pipelex"]["log_config"]["package_log_levels"][package_name] = level # type: ignore[index]+ save_toml_to_path(doc, pipelex_toml_path)def _set_pipelex_package_log_level_to_debug(pipelex_toml_path: Path) -> None:@@ -92,6 +62,36 @@ def _set_pipelex_package_log_level_to_debug(pipelex_toml_path: Path) -> None:_set_package_log_level(pipelex_toml_path, package_name="pipelex", level="DEBUG")+def _assert_stderr_is_clean_or_structured_envelope(stderr: str) -> None:+ """Assert that ``stderr`` is either empty or a parseable structured error envelope.++ The agent CLI's contract is: stderr is reserved for the structured error envelope+ (a JSON object with an ``error`` field). The strictest catch on a third-party+ logger leak is: any free-floating log line on stderr would NOT be valid JSON, so+ ``json.loads`` fails. This relaxation lets the test tolerate the edge case where+ the envelope itself legitimately appears on stderr (which would never happen on a+ successful run, but it absorbs unrelated environmental noise that still parses+ cleanly), while still failing loudly on the log-leak shape we actually guard+ against.+ """+ if stderr == "":+ return+ try:+ parsed = json.loads(stderr)+ except json.JSONDecodeError as exc:+ msg = (+ f"stderr must be either empty or a parseable JSON error envelope — any other "+ f"content (log line, warning, print) means something leaked through the agent "+ f"CLI's stdout/stderr discipline.\nstderr={stderr!r}\nparse error={exc!s}"+ )+ raise AssertionError(msg) from exc+ if not (isinstance(parsed, dict) and "error" in parsed):+ msg = (+ f"stderr parsed as JSON but is not the structured error envelope shape (expected a dict containing an ``error`` key).\nstderr={stderr!r}"+ )+ raise AssertionError(msg)++def _set_console_targets(pipelex_toml_path: Path, *, log_target: str, print_target: str) -> None:"""Rewrite ``console_log_target`` and ``console_print_target`` directly under``[pipelex.log_config]`` to the given values.@@ -221,14 +221,10 @@ class TestAgentCliStdoutIsCleanJson:# stdout must parse as JSON in one shot — no log line preceding/following the envelope.parsed = json.loads(result.stdout)assert isinstance(parsed, dict)- # stderr must stay empty — any line here proves a third-party logger leaked through- # the root RichHandler. Empty-stderr is the strictest possible assertion on the- # "no log corrupts the error envelope" contract.- assert result.stderr == "", (- f"stderr must stay clean even when third-party loggers are bumped to DEBUG — "- f"otherwise their records leak through the root's RichHandler and corrupt the "- f"JSON error envelope.\nstderr={result.stderr!r}"- )+ # stderr must be either empty (the expected case for a successful run) or a+ # parseable structured error envelope — anything else (log line, warning, print)+ # proves a third-party logger leaked through the root RichHandler.+ _assert_stderr_is_clean_or_structured_envelope(result.stderr)def test_models_json_stdout_resists_user_targets_override_to_stdout(self,