Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix(cli): correct build output format help and validation docs (#1068)

Louis Choquel1d8a2f6ce901
diff --git a/.drift/acks/cli-docs.toml b/.drift/acks/cli-docs.toml
index 0192884b1..4709b9d60 100644
--- a/.drift/acks/cli-docs.toml
+++ b/.drift/acks/cli-docs.toml
@@ -1,8 +1,8 @@
contract = "cli-docs"
-digest = "sha256:951772e532e83bafefb42f75214b9c65098a722c290ec1e36f943a4b3e23ac6e"
-reviewed_by = "Claude (Opus 5)"
-reviewed_at = "2026-07-26T09:35:23Z"
-rationale = "All 28 triggers changed by hub import lines only — the runtime/interpreter rename (service_hub -> runtime_hub, method_hub -> interpreter_hub). Verified mechanically: diffed all 28 trigger files together and filtered out every changed line naming a hub; nothing remained, so no CLI argument, flag, format value, default, or command behavior moved. The vocabulary and identifier edits of this rename (LOW_LAYER_PACKAGES -> RUNTIME_LAYER_PACKAGES, the guard's success panel) live under pipelex/cli/dev_cli/**, which this contract excludes. Opened both review targets: docs/tools/cli/ and pipelex/cli/agent_cli/CLAUDE.md contain zero references to pipelex.hub, service_hub, method_hub, or either new hub module — they document the user-facing command surface, not Python import paths, so the rename is invisible to them. Nothing to update."
+digest = "sha256:22604d4351dd102437cb162b4e4a3e577b32cd43b3a1e6ee66891b8587785003"
+reviewed_by = "Louis Choquel"
+reviewed_at = "2026-07-28T08:47:52Z"
+rationale = "Two build-output help strings changed: the 'output' group help in build/app.py advertised a 'typescript' --format value that does not exist (accepted: json|python|schema, and 'typescript' errors out), and method_cmd.py's --format help lacked the TypeScript/Zod parenthetical its pipe/bundle siblings carry. Reviewed both targets. docs/tools/cli/ WAS stale: build/index.md described the command as 'Generate example output JSON for a pipe', predating the python and schema formats — fixed to match the corrected group help. build/output.md was already accurate (all three subcommands list json/python/schema; TypeScript consistently framed as a downstream use of --format schema, not a format value) — no change needed. pipelex/cli/agent_cli/CLAUDE.md is out of scope: the agent CLI exposes no 'build' command at all, so no help text, flag, or format value there could have moved. Real catch, not a clean pass."
[trigger_files]
"pipelex/cli/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"
@@ -55,7 +55,7 @@ rationale = "All 28 triggers changed by hub import lines only — the runtime/in
"pipelex/cli/cli_factory.py" = "blob:1f859ab3b995e1a67e5fdc812a168d56c338e970"
"pipelex/cli/commands/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"
"pipelex/cli/commands/build/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"
-"pipelex/cli/commands/build/app.py" = "blob:fabe555870e61d147d0bf01e49de6e5cc9719d34"
+"pipelex/cli/commands/build/app.py" = "blob:0a96edab273c838aa9e0fae23646805c3a775e9d"
"pipelex/cli/commands/build/inputs/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"
"pipelex/cli/commands/build/inputs/_inputs_core.py" = "blob:2e91d50b6e547e76b7d429085d584e4cbc77b0df"
"pipelex/cli/commands/build/inputs/app.py" = "blob:2a7770c8294b8c2c1b4420ac00d6ffdfa50609bf"
@@ -66,7 +66,7 @@ rationale = "All 28 triggers changed by hub import lines only — the runtime/in
"pipelex/cli/commands/build/output/_output_core.py" = "blob:2df512eeaea4e87c60997612848139d63cfd7d77"
"pipelex/cli/commands/build/output/app.py" = "blob:67996650b99551fbd58be30e692688204d16935a"
"pipelex/cli/commands/build/output/bundle_cmd.py" = "blob:a4a04ce555e0bbf2178206269fe769c13a0bd0f7"
-"pipelex/cli/commands/build/output/method_cmd.py" = "blob:f43bd91fd9007358f725dc489212bec35a1e223d"
+"pipelex/cli/commands/build/output/method_cmd.py" = "blob:2625da8d562ba4f66cf17890772c7a1a674c3bb9"
"pipelex/cli/commands/build/output/pipe_cmd.py" = "blob:eb0db8279c93fcff9d5fe4546f593d753282acb5"
"pipelex/cli/commands/build/runner/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"
"pipelex/cli/commands/build/runner/_runner_core.py" = "blob:c33eec4a86bc33b799e441cc205ec895194c07bb"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b941e39d1..58e3779b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,8 @@
- **`hub.set_observer()` was documented as the way to register a custom observer, and never worked.** The value was written to the hub and never read — the live path is `Pipelex.make(observers={...})` → `MultiObserver` → `PipeRouter`, so anyone following that page got a silently ignored observer. The write-only state is deleted and both `docs/advanced/observer-provider-injection.md` and `docs/advanced/index.md` are rewritten to the mechanism that actually works, with a warning against hand-constructing a hub.
- **A duplicate `set_library_manager` pair in `Pipelex.setup` built, registered, then silently discarded a first `LibraryManager()`** when the caller passed none. Removed.
+- **The validation feature page promised a check that does not exist and cannot exist.** "Structure validation — Verify concept compatibility between pipes, ensuring inputs and outputs match" read as an inter-step contract: step N's output concept matched against step N+1's declared input concept. `PipeSequence` steps do not chain that way — each resolves its inputs by name from working memory, sourced from the pipeline's inputs or any earlier step, so there is no consecutive-step contract to verify. [Validation & Dry Run](features/validation-dry-run.md) now describes the checks that actually run (per-pipe input concepts, controller output boundary, variable binding) and states the working-memory resolution model explicitly.
+- **`pipelex build --help` advertised a `typescript` output format that does not exist.** The `output` group help read "(JSON, Python, or TypeScript)" while the accepted `--format` values are `json`, `python`, `schema` — passing `typescript` errored out. The group help now names the real formats; TypeScript remains what it always was, a downstream use of `--format schema` (JSON Schema → TypeScript interfaces or Zod schemas), which the per-subcommand help now states consistently across `pipe`, `bundle`, and `method`.
## [v0.40.0] - 2026-07-19
diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.md
index 821675a1b..a8fb0c7ab 100644
--- a/docs/features/validation-dry-run.md
+++ b/docs/features/validation-dry-run.md
@@ -16,8 +16,13 @@ Pipelex provides multiple layers of validation to catch issues before they cost
Check pipeline syntax, structure, and compatibility without execution:
- **Syntax validation** — Catch MTHDS language errors via plxt linting
-- **Structure validation** — Verify concept compatibility between pipes, ensuring inputs and outputs match
-- **Input validation** — Ensure required inputs are provided and correctly typed
+- **Structure validation** — Resolve pipe and concept references in the loaded library, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)
+- **Input validation** — Ensure required inputs are provided and correctly typed, and that every variable a step reads is bound by the time that step runs
+
+A pipe that references a sub-pipe from a package that isn't loaded is reported as **skipped** rather than validated, so a passing run is not proof that every cross-package dependency resolves.
+
+!!! note "Steps read from working memory, not from the previous step"
+ A `PipeSequence` step resolves its inputs by name from working memory. They may come from the pipeline's inputs or from any earlier step — not necessarily the one immediately before it. Validation therefore checks that each name is bound when the step runs, rather than matching one step's output concept to the next step's declared input concept. There is no output-to-input chaining contract between consecutive steps to verify.
## Dry Run Mode
diff --git a/docs/tools/cli/build/index.md b/docs/tools/cli/build/index.md
index cf6697b5f..9e470fad9 100644
--- a/docs/tools/cli/build/index.md
+++ b/docs/tools/cli/build/index.md
@@ -13,4 +13,4 @@ Generate pipelines, runner code, Python structures, and JSON templates from natu
| [**build runner**](runner.md) | Generate Python code to run a pipe |
| [**build structures**](structures.md) | Generate the structures (Pydantic models) of your concepts |
| [**build inputs**](inputs.md) | Generate example input JSON for a pipe |
-| [**build output**](output.md) | Generate example output JSON for a pipe |
+| [**build output**](output.md) | Generate example output representation for a pipe (JSON, Python, or JSON Schema) |
diff --git a/pipelex/cli/commands/build/app.py b/pipelex/cli/commands/build/app.py
index fabe55587..0a96edab2 100644
--- a/pipelex/cli/commands/build/app.py
+++ b/pipelex/cli/commands/build/app.py
@@ -9,6 +9,6 @@ build_app = typer.Typer(help="Build working pipelines from natural language requ
# inputs, output, runner are now Typer groups with method/pipe subcommands
build_app.add_typer(build_inputs_app, name="inputs", help="Generate example input JSON for a pipe")
-build_app.add_typer(build_output_app, name="output", help="Generate example output representation for a pipe (JSON, Python, or TypeScript)")
+build_app.add_typer(build_output_app, name="output", help="Generate example output representation for a pipe (JSON, Python, or JSON Schema)")
build_app.add_typer(build_runner_app, name="runner", help="Build the Python code to run a pipe with the necessary inputs")
build_app.command("structures", help="Generate Python structure files from concept definitions in MTHDS files")(build_structures_command)
diff --git a/pipelex/cli/commands/build/output/method_cmd.py b/pipelex/cli/commands/build/output/method_cmd.py
index f43bd91fd..2625da8d5 100644
--- a/pipelex/cli/commands/build/output/method_cmd.py
+++ b/pipelex/cli/commands/build/output/method_cmd.py
@@ -36,7 +36,7 @@ def build_output_method_cmd(
typer.Option(
"--format",
"-f",
- help="Output format: 'json' for JSON example, 'python' for Python code, 'schema' for JSON Schema",
+ help="Output format: 'json' for JSON example, 'python' for Python code, 'schema' for JSON Schema (useful for TypeScript/Zod generation)",
),
] = "json",
) -> None:
diff --git a/wip/drift-contracts/dogfood-log.md b/wip/drift-contracts/dogfood-log.md
index 8093c7d0b..a1f3c5101 100644
--- a/wip/drift-contracts/dogfood-log.md
+++ b/wip/drift-contracts/dogfood-log.md
@@ -17,3 +17,4 @@ One entry per ack. Verdicts: **real-catch** (the review found actual staleness),
- **2026-07-26 · hub-layering-convention · clean-pass (initial)** — **new contract**, added on Louis' explicit say-so after being built, verified, and deliberately reverted at H-2. Triggers are the layering guard plus both hub modules; review target is `docs/contribute/hub-layering.md`. No verify command: `check-hub-layering` already gates `make check` and CI, so re-running it pre-ack would add the unstaged-file hard error without adding a guarantee. Initial review was mechanical rather than a read-through — extracted all 65 public module-level symbols from the two hubs and confirmed every one appears in the doc's partition tables, then imported `RUNTIME_LAYER_PACKAGES` from the guard and confirmed all 11 declared packages are named in Enforcement. Both checks are cheap and repeatable; whoever reviews this contract next should run them rather than re-read the tables. Deliberately **not** made a trigger: `tests/unit/pipelex/test_runtime_layer_import_closure.py`, even though the doc has a section on it — that stays within the scope Louis approved, and it is the obvious first candidate if this contract ever proves under-triggered. Watch item for the pilot verdict: both hub modules are type-annotated with most of the tree, so a *type move* elsewhere edits their import blocks and opens this contract — the same import-churn shape logged four times above. If that is what its first few openings look like, this contract is evidence for the content-aware digest, not against it.
- **2026-07-27 · hub-layering-convention · real-catch** — first opening after the initial baseline, and it **answers that entry's watch item in the negative**: this was not the import-churn shape. Both triggers changed semantically — the guard's declared runtime layer gained `pipelex.runtime_hub` (closing a gap where the module at the centre of the rule was exempt from it), and `runtime_hub.py` shed write-only class-registry state. The review found four stale claims in `docs/contribute/hub-layering.md`, none of which the contract's two prescribed mechanical checks can see: they verify *coverage* (every hub symbol appears, every declared package is named) and are blind to a covered claim being **false**. Three of the four were exactly that — a compliance claim that read as a claim about the property when the guard only checks direct imports; a "the one module not covered" sentence wrong on both halves; and a leaf-import rule whose stated criterion contradicts two of its three in-tree importers. That is the case *for* a prose review target with a human-or-agent reader, and the sharpest evidence in this log that the mechanical checks are a floor, not the review. Note also what this says about the content-aware-digest recommendation running through the entries above: it would have opened this contract too (the trigger diffs are code, not imports), so the recommendation costs nothing here — the two are not in tension.
- **2026-07-27 · hub-layering-convention · real-catch (second)** — opened by a *behavioral* guard change, the second consecutive non-import-churn opening for this contract: `_is_type_checking_test` matched `ast.Attribute(attr=...)` with an unconstrained receiver, so any `settings.TYPE_CHECKING:` block claimed the layer-rule exemption; the attributed form is now rooted at `typing`. The doc's carve-out bullet was under-specified rather than false, and needed the constraint stated. **The more interesting half is what the review caught that the triggers cannot see.** The same commit hardened both test modules the review target has sections on — the closure test gained a negative control and a subprocess timeout, the lifecycle test now pins the real `Pipelex.teardown` instead of a stand-in — and *neither test file is a trigger*, deliberately, per the initial baseline entry above. So the contract fired on the guard and the review then found staleness caused by changes it is structurally blind to. Two readings, and the honest one matters: this is **not** an argument for adding the test modules as triggers (the initial entry's scoping decision was Louis', and the spillover was caught anyway), but it *is* evidence that a review target's value is bounded by what a reviewer reads, not by what opened the contract. A narrower trigger set that had stayed closed here would have shipped two stale doc sentences.
+- **2026-07-28 · cli-docs · real-catch** — first opening for this contract that was *not* a mechanical sweep, and the first that caught real staleness. Triggers were two `build output` help strings: the group help in `build/app.py` advertised "(JSON, Python, or TypeScript)" while `--format` accepts `json|python|schema` (passing `typescript` errors out), and `method_cmd.py`'s `--format` help lacked the TypeScript/Zod parenthetical its `pipe`/`bundle` siblings carry. Reviewing `docs/tools/cli/` against them found `docs/tools/cli/build/index.md` describing the command as "Generate example output JSON for a pipe" — written before `python` and `schema` existed, and stale in the same direction as the help string that opened the contract. `build/output.md` was already accurate (all three subcommands list the three formats; TypeScript is consistently framed as a downstream use of `--format schema`), and `agent_cli/CLAUDE.md` has no `build` command at all, so it was structurally out of scope. Both stale strings fixed before the ack. Worth recording against the four consecutive clean-passes above: this contract's value shows up precisely when a trigger diff is a *user-visible string* rather than an import line — which is the same distinction the content-aware-digest recommendation draws, and further evidence it would preserve the openings that matter while dropping the ones that don't. Note the finding arrived from outside the contract (a developer reported the `typescript` help lie); the contract's contribution was pulling the neighbouring doc table into the same fix instead of leaving it to rot another cycle.
diff --git a/wip/pr-1059-review-notes.md b/wip/pr-1059-review-notes.md
index 343466145..fd4315c21 100644
--- a/wip/pr-1059-review-notes.md
+++ b/wip/pr-1059-review-notes.md
@@ -7,7 +7,7 @@ Triage record for the SWE-agent review threads on [PR #1059](https://github.com/
- **Reporter:** cubic (thread `PRRT_kwDOOwmMFc6SEzQN`), `pipelex/cogt/usage/cost_registry.py` (`compute_tokens_usage_cost`).
- **Verdict:** false positive *as filed*, with a narrow real residual. The finding claims `compute_tokens_usage_cost` "duplicates the logic in `CostRegistry.complete_cost_report`" and can diverge. In fact the two paths already share the pieces that carry the cost *policy* — the rate lookup + cached-discount fallback (`model_cost_per_token`) and the total formula (`compute_total_cost`). Only the structural component derivation is duplicated. Parity is **explicitly asserted**: `tests/unit/pipelex/cogt/usage/test_usage_cost.py:37-46` compares the summed wire `cost` against `CostRegistry.aggregate_costs(...).total_cost` — the actual number the CLI cost table prints — plus per-record parity across all four usage variants.
- **Why deferred:** the only divergence that escapes the parity test is *widening which token categories count toward the table total* (e.g. starting to charge reasoning/audio), because the parity fixture `RATED_NB_TOKENS` (`tests/unit/pipelex/cogt/usage/test_data.py`) only carries `INPUT`/`INPUT_CACHED`/`OUTPUT`. That is a fixture-coverage hole, not the architectural drift the finding describes — and closing it by folding `compute_tokens_usage_cost` into `complete_cost_report` reshapes a client-facing response path, which is not worth destabilizing a release branch for.
-- **Recommendation:** cheap, test-only follow-up — add an `OUTPUT_REASONING`/`INPUT_AUDIO` fixture variant to `all_variants()` so the parity test also guards the excluded-category boundary. The delegation refactor (make parity structural instead of asserted) is optional and belongs off a release branch.
+- **Recommendation:** cheap, test-only follow-up — add a *dedicated* single-record parity test for an excluded category, not a new entry in `all_variants()`. Add a `UsageFixtures` method (e.g. `reasoning_usage()`) carrying an `OUTPUT_REASONING` (and/or `INPUT_AUDIO`) token count plus a matching rate, then assert that both `compute_tokens_usage_cost(reasoning_usage())` and `CostRegistry.aggregate_costs([reasoning_usage()]).total_cost` equal the three-component total (the excluded category contributing 0 to both) — so a future change that starts counting the extra category in one path but not the other breaks the test. Two footguns to avoid, both verified against the current tests: (1) do **not** mutate the shared `RATED_NB_TOKENS` constant (`tests/unit/pipelex/cogt/usage/test_data.py`) in place — `tests/unit/pipelex/reporting/test_usage_records.py` hard-asserts the exact `{"input":1000,"input_cached":400,"output":250}` dict, so mutating it breaks those; (2) do **not** fold the new heterogeneous variant into `all_variants()` — `test_parity_with_aggregate_run_total` asserts exact-float `==` and holds only because all four current variants are token-identical, so a heterogeneous variant risks an ULP mismatch from float-summation order. The delegation refactor (make parity structural instead of asserted) is optional and belongs off a release branch.
## 2. Transported PipeFunc failures lose traceback source-line text (cubic P2)