Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Document the --with-memory envelope chaining regression (D8)

Louis Choquel8f6ace553b73
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 241d27fd8..e0a6354c6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -55,7 +55,7 @@
- **Clean validation summary message on every surface:** The top-level error `message` (agent JSON, the API `/validate` body) is now a clean, author-facing summary derived from the structured errors, instead of the raw pydantic dump (`Validation error(s): … Value errors: '<field>': Value error, …`). The API's opt-in invalid-verdict `rendered_markdown` now carries the same per-error `💡 Suggested fix` prose as the CLI. A still-invalid fix that stops with no progress now names the re-proposed fix in author terms rather than dumping an internal fingerprint string.
- **Input templates default to "light" shape (Breaking):** `pipelex build inputs` and `pipelex-agent inputs` now generate light templates by default, outputting the bare values expected by the signature rather than the verbose `{"concept", "content"}` envelopes. TOML templates include the declared concept as a `# concept: ...` comment.
- **Structure-field `type = "date"` behavior (Breaking):** A `date` field now generates a `datetime.date` (JSON schema `format: date`), preventing LLMs from hallucinating times for calendar dates. A new `type = "datetime"` field handles full timestamps (`format: date-time`), and default-value validation now rejects a `datetime` default on a `date` field.
-- **Strict input validation (Breaking):** Undeclared input names are now a hard error (previously silently ignored), and top-level `null` values are rejected — absence must be expressed by omitting the key.
+- **Strict input validation (Breaking):** Undeclared input names are now a hard error (previously silently ignored), and top-level `null` values are rejected — absence must be expressed by omitting the key. Known limitation: this also rejects the *extra* stuffs carried by a `pipelex-agent run --with-memory` envelope, so piping one run's full working memory into a downstream pipe that declares only a subset of it now fails with `UnknownInputNameError`. Until envelope-derived inputs get their own leniency, narrow the envelope to the declared inputs before piping, or pass the inputs explicitly with `--inputs`.
- **Explicit envelope validation (Breaking):** The `{"concept", "content"}` escape hatch is now compatibility-checked against the pipe's declared signature; incompatible concepts raise a typed error.
- **TOML date/time handling (Breaking):** TOML date and datetime literals are natively supported and map directly to the `Date` concept; a bare time-of-day literal maps to the new `Time` concept (it never silently becomes a `Date` — shaping a `Time` into a `Date` slot fails the compatibility check).
- **`ImageContent` flattens its pixel dimensions (Breaking):** `size: ImageSize | None` is replaced by paired optional `width` / `height` integer fields (both-or-neither, enforced by a validator) — a named nested shape in the crate is a concept, and the size grouping did not earn that citizenship. `ImageSize` remains as the internal img-gen parameter type. Image wire payloads change shape accordingly.
diff --git a/wip/inputs/README.md b/wip/inputs/README.md
index 18f90acd7..be7dbc6f9 100644
--- a/wip/inputs/README.md
+++ b/wip/inputs/README.md
@@ -37,6 +37,10 @@ Each deferred note was re-verified against the landed shaper and given an outcom
- **Decided leave-it** — `d4-hint-still-envelope.md` (the D4 error hint keeps rendering the envelope shape; option 1, it is a valid unambiguous fallback and making it light is a layer/cycle refactor unjustified by a fallback string).
- **Still out of scope (structure-field codegen / URI-tools / release-wave authoring)** — `container-default-temporal-codegen-gap.md`, `structure-field-fidelity-guard.md` (both structure-field codegen, untouched by Smart Inputs), `uri-scheme-classification-stopgap.md` (a `tools/uri` `resolve_uri` refactor; no live bug, tested), `refines-hint-native-list-drift.md` (the release-wave authoring-guidance sweep).
+## Open regression — `envelope-chaining-vs-d8.md` (2026-07-14)
+
+Unlike the notes above, this one is **not a tradeoff — it is a live regression that v0.39.0 ships**, found by Codex on the release PR #1048 and knowingly deferred out of the release. D8 makes the agent CLI reject its own `--with-memory` output: the stdin envelope carries every `working_memory.root` entry, so chaining into a downstream pipe that declares only a subset now dies with `UnknownInputNameError`. It worked in v0.38.0 (no name check existed), and the chained `run … --with-memory | run …` pattern is taught in the shipped `mthds-plugins` skills. The note carries the repro, the three fix options (recommended: an ignore-extras policy for envelope-derived inputs only, threaded like `inputs_base_dir`), and the open `--runner api` parity question. **Pick this up first when the inputs track reopens.**
+
## Documents
**Design (source of truth for decisions):**
@@ -55,3 +59,5 @@ Each deferred note was re-verified against the landed shaper and given an outcom
- `yesno-pr-reviewers-guide.md` — the former YesNo PR reading guide. Historical.
**Deferred design notes** (triage outcomes above): `case1-bare-date-arm-gap.md`, `bare-file-path-cli-resolution-gap.md`, `scalar-envelope-arm-asymmetry.md`, `loader-vs-factory-date-split-duplication.md`, `input-shaper-multiplicity-gaps.md`, `d4-hint-still-envelope.md`, `container-default-temporal-codegen-gap.md`, `structure-field-fidelity-guard.md`, `uri-scheme-classification-stopgap.md`, `refines-hint-native-list-drift.md`.
+
+**Open regression** (section above): `envelope-chaining-vs-d8.md` — the one item here that is a bug rather than a tradeoff.
diff --git a/wip/inputs/envelope-chaining-vs-d8.md b/wip/inputs/envelope-chaining-vs-d8.md
new file mode 100644
index 000000000..e00a00d3d
--- /dev/null
+++ b/wip/inputs/envelope-chaining-vs-d8.md
@@ -0,0 +1,71 @@
+# Envelope chaining vs D8 — `--with-memory` output is no longer valid input
+
+**Status:** confirmed regression in v0.39.0, **deliberately deferred** out of the release (decision taken 2026-07-14 during the PR #1048 review).
+**Reported by:** Codex (P2) on the [v0.39.0 release PR #1048](https://github.com/Pipelex/pipelex/pull/1048#pullrequestreview-4698103093).
+**Area:** D8 (strict unknown input names) × the agent CLI's stdin working-memory envelope.
+
+## The bug
+
+The agent CLI can no longer consume its own documented output. Piping an upstream `--with-memory` envelope into a downstream pipe that declares only *some* of the carried stuffs fails with `UnknownInputNameError`.
+
+Three hops:
+
+1. `resolve_stdin_inputs` (`pipelex/cli/agent_cli/commands/run/stdin_resolver.py:107-126`) explodes **every** entry of `working_memory.root` into an input entry. Its own docstring names the producer: *"Full envelope: a dict with a `working_memory` key at the top level (from upstream `--with-memory` output)"*.
+2. Those inputs reach the shaper unchanged: `run_pipeline_core` → `PipelexMTHDSProtocol.execute(inputs=…)` → `prepare_pipe_job` (`pipelex/pipeline/execution_seams.py:195-207`, which always passes `input_specs=pipe.inputs`) → `WorkingMemoryFactory.make_from_pipeline_inputs` → `InputShaper.shape`.
+3. `InputShaper._check_input_names` (`pipelex/core/memory/input_shaper.py:136-142`) raises on the **first** provided name absent from `input_specs.declared_names`. There is no envelope-vs-flat distinction, no ignore-extras path, and the only way to skip the shaper entirely is handing `execute` an already-built `WorkingMemory` — which the CLI never does.
+
+An upstream sequence carries its own inputs in the envelope alongside its results, so the very first extra name (`text`) kills the run even though the declared input (`summary`) is present.
+
+## Reproduction
+
+Bundle with a `prepare` sequence (`inputs = { text = "Text" }`, step `result = "summary"`) and a `classify` pipe (`inputs = { summary = "Summary" }`):
+
+```bash
+pipelex-agent run bundle chain.mthds --pipe prepare \
+ --inputs '{"text":"The quick brown fox jumps over the lazy dog."}' \
+ --dry-run --no-graph --with-memory --format json > up.json
+# working_memory.root keys: ['text', 'summary']; aliases: {'main_stuff': 'summary'}
+
+cat up.json | pipelex-agent run bundle chain.mthds --pipe classify --dry-run --no-graph --format json
+```
+
+```json
+{
+ "error": true,
+ "error_type": "UnknownInputNameError",
+ "message": "Input 'text' is not declared by this pipe. Declared inputs: 'summary'."
+}
+```
+
+## Why it is a regression
+
+`git show v0.38.0:pipelex/core/memory/working_memory_factory.py` — `make_from_pipeline_inputs` took no `input_specs` and performed no name check, so extra envelope entries were carried harmlessly into the working memory. `input_shaper.py` does not exist in v0.38.0; it arrived with Smart Inputs (#1028). v0.39.0 is therefore the first release where the envelope the CLI emits is not an envelope the CLI accepts.
+
+The design anticipated the *class* of breakage but not this instance. `smart-inputs-design.md` (D8, "Risk to weigh") says: *"any workflow deliberately over-providing inputs … would break — if that pattern matters, downgrade to a warning."* What was never considered is that **the CLI itself is a first-party over-providing producer**: `--with-memory` exists precisely to carry the whole memory forward.
+
+## Blast radius
+
+- **The chaining pattern is taught in shipped agent skills.** `mthds-plugins/mthds/skills/mthds-run/SKILL.md` and `mthds-plugins/mthds/skills/shared/mthds-agent-guide.md` both instruct `run … --with-memory | run … --with-memory | run …`, with `--with-memory` on intermediate steps and a bare final step. Those examples are broken until this is fixed — **fixing them (or updating them) is part of the follow-up, and lives in the `mthds-plugins` repo, not here.**
+- **It fails loudly**, with a precise, actionable message — not silent corruption. That is what made deferral acceptable.
+- **Workarounds:** narrow the envelope before piping (`jq '{working_memory: {root: {summary: .working_memory.root.summary}, aliases: .working_memory.aliases}}'`), or drop `--with-memory` and pass `--inputs`.
+- **Narrower than it looks for single-pipe upstreams:** when the upstream is a single pipe, its result lands under the literal key `main_stuff` with empty aliases, so a downstream declaring `summary` finds nothing to bind even with extras ignored. Full-envelope chaining is genuinely useful when the upstream is a *sequence* whose step `result` names match the downstream's declared inputs (the repro above). The fix restores a real but narrow capability.
+
+## Fix options
+
+**1. Core policy param — recommended.** Keep D8 strict for flat/file/inline inputs; make *envelope-derived* inputs lenient about extras. Thread the signal along the rails `inputs_base_dir` already uses (same shape, same call sites): `ParsedCliInputs` (`stdin_resolver.py:20-31`, gains a memory-derived marker set when `resolve_stdin_inputs` saw a `working_memory` key) → `run_pipeline_core` → `PipelexMTHDSProtocol.execute` → `prepare_pipe_job` (`execution_seams.py:172,202`) → `make_from_pipeline_inputs` (`working_memory_factory.py:71-107`) → `InputShaper.shape`. Carry it as a `StrEnum` (`UnknownInputNamePolicy.ERROR | IGNORE`, defaulting to `ERROR` everywhere — project standards ban `bool` params). Under `IGNORE`, filter `pipeline_inputs` down to `input_specs.declared_names` **before both** `_check_input_names` and the build loop — the loop calls `get_required_stuff_spec`, so undeclared names must be *dropped*, not merely un-raised.
+
+**2. CLI-local filter.** After boot the CLI can resolve the target pipe, so it could filter the envelope-derived dict against the pipe's declared names itself. Zero core churn — but it puts the leniency outside the shaper that owns input semantics, and every other envelope consumer (the hosted runner) stays broken.
+
+**3. Downgrade D8 to a warning.** The design doc's own escape hatch. One function, no plumbing, un-breaks every path including `--runner api`. Cost: gives up the misspelled-input safety net D8 was deliberately added to provide. Only worth it if we conclude the typo-catching value was overrated.
+
+## Open question to settle with the fix
+
+**Is envelope chaining a local-runner-only capability?** `--runner api` ships `inputs` as a plain dict to the hosted runner (`_run_core_api.py:22,45`), which applies the same D8 check server-side. A CLI-local policy signal (options 1 and 2) does not reach it, so `--runner api` + envelope chaining stays broken unless the MTHDS protocol carries the same signal — a cross-repo change (`mthds/protocol`, `pipelex-api`). Decide this explicitly rather than letting the local path quietly diverge from the hosted one.
+
+A more principled alternative worth weighing at that point: treat a `working_memory` envelope as a **`WorkingMemory`**, not as inputs — hydrate it CLI-side and hand it to `execute`, which already bypasses the shaper (`execution_seams.py:196-197`). Semantically right ("memory in → memory continues", intermediates stay readable) and needs no shaper change; the cost is a hydrator from `smart_dump()` that needs the class registry loaded, which the CLI does not have before it builds the runner. That is why it was not done, not a reason it is wrong.
+
+## Test approach for whoever fixes it
+
+- **Shaper unit:** the lenient twin of the D8 test at `tests/unit/pipelex/core/memory/input_shaper/test_errors.py` — extras dropped, declared slots still shaped, strict policy still raising.
+- **CLI unit:** `tests/unit/pipelex/cli/test_stdin_resolver.py` — pin that an envelope parse is flagged memory-derived and a flat parse is not.
+- **e2e (the real chain):** `tests/e2e/pipelex/pipes/smart_inputs/` — a fixture bundle with an upstream sequence + downstream consumer; dry-run the upstream, feed the resolved envelope into the downstream, assert success. (`tests/e2e/pipelex/cli/test_toml_inputs_build.py` is the precedent if you want it to shell the real binary.)