Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Fix PipeCompose construct whole-stuff copies into native fields (#1051)
Louis Choquelc389fa51edfa
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b5f2778e1..93a91227c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,10 @@- **`LibraryManagerAbstract.is_crate_loaded(*, library_id, fingerprint)`** — public query over the per-library crate-fingerprint bookkeeping that already backs `load_from_crate`'s idempotency. A `True` answer means the library's ClassRegistry holds the crate's dynamic classes, so callers (e.g. `pipelex-transport`'s `scoped_library_for_crate`) can hydrate within an existing scope instead of opening a fresh one — preserving dynamic-class identity with instances that scope already produced. This is the core half of the fix for the PipeParallel same-concept combine failing over Temporal with `expected X, got X` pydantic model-type errors.+### Fixed++- **PipeCompose construct: whole-stuff copies into native fields now convert in every promised case.** `{ from = "..." }` referencing a whole native stuff used to hand the content wrapper (`TextContent`, `ListContent`, ...) to the composed field in several cases, failing the dry-run runnable gate on correct methods. Three gaps fixed: optional (non-required) fields never converted (the `Optional[X]` annotation defeated the type detection), list-of-text targets dumped each item as a `{"text": ...}` dict instead of extracting the string, and scalar wrappers other than `Text` were not handled at all. The conversion matrix now covers `Text → text`, `Number → number`, `YesNo → boolean`, and `Date → date`, for both required and optional target fields, scalar and list-item positions — including nullable list items (`list[str | None]`), which normalize like their non-nullable counterparts. Unconvertible list items now raise a clear `StructuredContentComposerTypeError` instead of surfacing as a cryptic pydantic error downstream. One fidelity guard: copying a `Date` that carries a time of day into a bare `date` field raises instead of silently dropping the time and its UTC offset (breaking: this case previously truncated silently).+## [v0.39.1] - 2026-07-15### Changeddiff --git a/TODOS.md b/TODOS.mdindex 205d87ff9..d25ff9cf6 100644--- a/TODOS.md+++ b/TODOS.md@@ -1,173 +1,116 @@-# Drift Hunt — campaign tracker+# PipeCompose construct: whole-stuff → native-field conversion fixes-A campaign, not a feature: sweep every hand-written doc page for code↔doc drift, fix what's stale, and come out with per-section defect-density evidence. The drift-contracts system (`docs/contribute/drift-contracts.md`, design at `wip/drift-contracts-design.md`) enforces the floor *going forward*; this hunt backfills the floor once. Promoted from `wip/kick-off-drift-hunt.md` (now a pointer stub) — this file is the live tracker.+**Status: GREEN — CHECKPOINT 2 reached (2026-07-17). Phases 1+2+3 complete: fix landed and verified, docs + changelog written. Remaining: Phase 4 cross-repo follow-ups, release-gated — do NOT do them before a pipelex release ships the fix.**-## Cold-start context (update at every checkpoint)+## Cold-start state (as of Checkpoint 2, 2026-07-17)-- **Status:** **✅ CAMPAIGN DONE — CHECKPOINT 2 CLOSE-OUT EXECUTED (2026-07-12), STOPPED for Louis' final review.** The record: `wip/drift-hunt/findings/SUMMARY.md` (close-out section: coverage disclosure, Stage-2 resolution table, ranked drift-contract shortlist for the Phase 3 verdict, lessons) + `inventory.md`; the handoff: `deferred-code-findings.md` (10 items). Full `make agent-test` GREEN on the reconciled branch. `wip/drift-hunt/` pruned (removed files at `git show drift-hunt-archive:wip/drift-hunt/<path>` — pushed tag pinning `9a6201a5e`). ⚠ Record correction: the observer-provider-injection.md "partially fixed" note below was a **stale-base artifact** — the page is fully fixed on dev (S2-3/#1042 fixed Part-2 F38–F42; re-verified against live code at close-out), no follow-up needed. Everything below this point is historical narrative. — Historical: **ALL FOUR STAGE-2 PRs MERGED TO DEV (2026-07-12)** — squash SHAs: **#1042 (S2-3) `9814e2f4c`, #1043 (S2-1) `126616a88`, #1044 (S2-2) `46511ce9c`, #1045 (S2-4) `cb13e5b33`**. dev tip = `cb13e5b33`. All four PR branches deleted, remote AND local (local tips verified ancestors of the merged `refs/pull/N/head` before force-delete — squash merges always fail `git branch -d`'s ancestry check, that refusal is expected, not a danger signal). Merges were forced sequential by branch protection (head must be up to date with dev): after each squash the next PR went BEHIND → `gh pr update-branch` + squash auto-merge + wait for CI, three times. The main checkout `../pipelex` is back on `dev` at the tip; this `_drift` worktree stays on `docs/Drift-hunt`. Pre-merge, PR #1045 was finalized: Louis's review edits + bot-triage fixes landed as PR-branch commit `1b776eca5` (configure-ai-providers.md local-overrides/init/telemetry.toml notes; `generate-mthds-schema` added to the kit-rules no-make fallback list — verified against `Makefile:882` `plxt-lint: env generate-mthds-schema-quiet`, regenerated via `make rules`; data-extraction.md explicit `storage_dir` for pre-boot `LocalObserver` — verified against `local_observer.py:19`; observer-provider-injection.md JSONL filenames corrected to the `LocalObserverEventType` values); all 8 bot threads (greptile/cubic/codex) confirmed-and-fixed or answered, replied + resolved; CI green on that head. ⚠ observer-provider-injection.md is now only PARTIALLY fixed out-of-batch (JSONL names done; the broken `from pipelex import Pipelex` import at line 209 and pre-rename hook prose remain — follow-up stands, smaller). **Campaign branch reconciled with dev** (merge `4094fc050`, Phase-H style): 8 files conflicted (the two `CLAUDE.md`/`AGENTS.md` generated pair, kit agent_rules ×2, data-extraction, python-classes, PipeSequence, configure-ai-providers), all resolved by taking dev's reviewed superset — residual diff vs dev is campaign artifacts only (TODOS.md + `wip/drift-hunt/` + the kick-off stub). Earlier same day: **S2-4 DONE (2026-07-12)** (branch `docs/Drift-hunt-s2-4`, off dev `15e0a94ea`; campaign-branch mirror `5e15bec08` + records commit). All 23 features+tail findings fixed, **0 rejected-at-fix-time**; all cross-cutting sweeps landed (PipeStructure roster ×3 incl. the landing page — count-free phrasing per the no-hardcoded-counts rule; `agent-cli.md` `--mock-inputs` carry-over; D13 heartbeat everywhere). **+F11 (features file) and F14–F22 (tail file) fix-time drift found & fixed (per D16, adversarially verified), 3 candidates refuted** (Linkup/pypdfium2 extract-list "omissions" — correctly SCOPED lists, the scope qualifier is part of the claim; the debugging-page marker expression — provably selects the identical test set). **Two lessons with teeth:** (1) repo-root `CLAUDE.md`/`AGENTS.md` are GENERATED from `pipelex/kit/agent_rules/` — a direct edit failed `make check-rules` mid-batch; the D13 fixes were applied at the source templates + `make rules` (AGENTS.md picked up the fixes for free). (2) The batch's own new `LocalObserver` snippet was written with a broken import (`from pipelex import Pipelex`; real: `from pipelex.pipelex import Pipelex`) — caught pre-commit by import-checking every new snippet (the S2-1 lesson applied). The workspace-root `CLAUDE.md:160` heartbeat claim was fixed in ITS OWN repo (single-file commit `cc2b68c` on `feature/Follow-ups`). Empirical re-verification throughout (`pipelex init config` re-run sandboxed; the `uv` commands run live; pytest `-k` semantics proven by collection). 1 new code-side deferral (`deferred-code-findings.md` item 10: shipped `plxt.toml` "built-in default" comment contradicts its own `array_auto_collapse` value). 1 out-of-batch record for Louis: **`docs/advanced/observer-provider-injection.md` (+ `advanced/index.md:18`) still carry pre-rename observer hooks, wrong JSONL filenames, and the broken `Pipelex` import** — survived Part 2's stub classification and S2-3; one-page follow-up. Gates green (docs-check strict + full `make check`). **Blind post-fix review (2 no-context Sonnet reviewers on `66b1d6ebd`): ZERO defects.** `docs/setup/telemetry.md` untouched as ordered. Resolution records: banners + end sections of `findings/features-reliability.md` and `findings/the-tail.md`. Burn: ~1.5M tokens (13 fix + 6 verify + 2 review agents). ⟶ **NEXT = CHECKPOINT 2 (campaign close-out — STOP for Louis): full `make agent-test` on final state, `findings/SUMMARY.md` close-out (incl. the cookbook-unswept D12 disclosure + contract-candidate shortlist), hand off `deferred-code-findings.md` (10 items), prune `wip/drift-hunt/`.** Earlier: **S2-2 DONE (2026-07-12), PR #1044 → dev OPEN** (branch `docs/Drift-hunt-s2-2`, off dev, docs-only, 10 pages; campaign-branch commits `8f03eb6f2` + `1824f2db4`). All 14 concepts + top-level findings fixed, **0 rejected-at-fix-time**; **+7 fix-time drift found & fixed (F55–F61, per D16, adversarially verified) + 1 stale section pointer, and 6 candidates refuted** — notably the multi-line inline TOML tables candidate (the shipped tomli 2.4.1 accepts TOML 1.1; testing against the ACTUAL loader, not a generic parser, decided it) and the domain "three properties" claim (`main_pipe` is bundle/crate metadata the runtime `Domain` drops). Every changed example re-extracted from the pages and re-validated (cross-bundle via `-L` scaffolds); the libraries.md `execute()` pattern and the F3 Document Extraction sequence executed live in dry-run. **Blind post-fix review (2 no-context Sonnet reviewers on `8f03eb6f2`): ZERO defects — first Stage-2 batch to survive clean.** 2 new code-side deferrals (`deferred-code-findings.md` items 8–9: `tomli>=2.3.0` floor predates the TOML 1.1 syntax the docs/kit rely on; `build structures` opaque-placeholder shadowing after a class migration). 1 out-of-batch defect recorded for Louis: **`pipes/pipe-operators/PipeLLM.md:126-135` inline `@document` survived S2-1** (adversarially confirmed; one-line fix, could ride PR #1043's branch). Convention proposal for Louis (D11-adjacent): elided sequence sketches are elision-by-design iff the snippet validates once placeholder sub-pipes are scaffolded. Also: the globally-installed `pipelex-agent` shim backing the marketplace validate-mthds hook is broken in this workspace (`ModuleNotFoundError: pipelex_tools`) — hit independently by multiple agents. The PR carries this page set's unshipped Stage-0 fixes (native-concepts triple-quote + adapt-to-llm-prompting-style), self-contained on dev. Gates green (docs-check strict on both branches; `make check` green — check-urls flaked once, passed on retry). Resolution record: banner + end sections of `findings/building-methods.md`. Burn: ~1.5M tokens (8 fix + 4 verify + 2 review agents). ⟶ **NEXT = S2-4 (the last part), ⛔ needs Louis' per-part greenlight (D18).** Earlier: **S2-1 DONE (2026-07-12), PR #1043 → dev OPEN** (branch `docs/Drift-hunt-s2-1`, off dev, docs-only, 17 pages; campaign-branch commits `7942339cc` + `dd6174ee6`). All 30 `pipes/` findings fixed, **0 rejected-at-fix-time**; **+10 fix-time drift found & fixed (F45–F54, per D16, adversarially verified) and 2 candidates refuted** (the PipeCondition fallback fragment — fragment standard; the PipeExtract Overview one-liner — the code's own taxonomy covers web pages under "document processing"). Every changed bundle example re-validated against the real parser incl. dry-run; the blind post-fix review caught **2 defects in the batch's own text** — the "run a bundle" Python snippets on TWO pages never ran (`bundle_uris` is only a dedup key; `execute()` needs `pipe_code` or `mthds_contents` — inherited from the original docs, the Stage-1 kwarg-rename finding masked the broken call pattern) and the F45 dedup claim missed its `bundle_uris` gate. Corrected snippets executed live in dry-run. 2 new code-side deferrals (`deferred-code-findings.md` items 6–7: CSV codec datetime inconsistency; concept-factory `refines` asymmetry). Gates green (docs-check strict on both branches + `make check`). Resolution record: banner + end sections of `findings/building-methods.md`. Burn: ~1.7M tokens (10 fix + 3 verify + 2 review agents). ⚠ S2-2 lesson: **a Stage-1 finding scoped to a kwarg can hide a non-functional surrounding example — validate the whole snippet, not just the finding's clause.** ⟶ **NEXT = S2-2 (recommended), ⛔ needs Louis' per-part greenlight (D18).** Earlier: **S2-3 DONE (2026-07-12), PR #1042 → dev OPEN** (branch `docs/Drift-hunt-s2-3`, off dev, docs-only). All 42 under-the-hood+advanced findings fixed, **0 rejected-at-fix-time**; +4 fix-time drift found & fixed (F43–F46, per D16, adversarially verified) and **2 fix-time candidates REFUTED — the campaign's first refutations** (fix-time finders over-flag; the D6 verify gate is what keeps D16 honest). 2 code-side items deferred (`deferred-code-findings.md` items 4–5), F24 fixed at its real site + class label corrected, `setup/telemetry.md` untouched. The blind post-fix review (2 no-context Sonnet reviewers on the commit SHA) caught 3 defects in the batch's own new text — including one **inherited from the Stage-1 finding's Reality text itself** (F36's "packaged kit configs" fallback; real fallback is global `~/.pipelex/`) — all folded in before the PR. ⚠ Lesson for S2-1/2/4: D6's "code is the authority" applies to the finding's *Reality* text too, and the blind post-fix review earns its keep — keep both. Gates green (docs-check strict on both branches + `make check`). Resolution record: banner + end section of `findings/under-the-hood-advanced.md`. Burn: ~2.4M tokens (14 fix + 2 verify + 2 review agents). ⟶ **NEXT = S2-1 (recommended), ⛔ needs Louis' per-part greenlight (D18).** Earlier: **🎉 STAGE 1 COMPLETE — PART 4 DONE (2026-07-12).** The tail (18 pages: get-started 3, contribute 3, tools non-CLI 2, setup 1, viewpoint 1, root index/license 2, D9 additions 6) swept via the wave Workflow (`wf_0c7af057-a6e`): 244 claims checked, **12 confirmed / 0 rejected**, **+1 more found by the post-run false-negative check** = **13 drift**, 0 reclassified. Severity: 2 breaks-a-user / 9 wrong / 2 misleading / 0 stale; **density 0.72 (9/18 dirty)**. Findings in `wip/drift-hunt/findings/the-tail.md`; `SUMMARY.md` now carries the closed Stage-1 scoreboard. No fixes applied (fixes = Stage 2). **STAGE 1 TOTAL: 104 pages, 1,439 claims, 109 drift, 3 convention, 0 rejected, density 1.05.** ⟶ **NEXT = CHECKPOINT 1 (STOP for Louis).** Earlier: Parts 1–3 recaps below; CHECKPOINT 0 CLOSED (D7–D10), Phase H DONE, Phase B DONE, Stage 0 DONE, gates green.-- **Part 4 recap + what it taught (read before Stage 2):** the tail is mostly **contributor / process / meta** pages, so its dominant claim type was one the campaign had never met — **repo tooling** (make targets, generated file trees, `mkdocs.yml`, `robots.txt`, the Python floor). Four new review rules were added on top of the reused Part-3 block: **H** make-targets-and-tooling (grep the Makefile *and read the recipe*), **I** process/policy prose is out of scope (zero findings on a code-of-conduct is the *correct* outcome), **J** version requirements (`requires-python = ">=3.11,<3.15"`; 3.10 dropped — fired zero times, `CONTRIBUTING.md` is correct), **K** docs-site meta-claims (vs `mkdocs.yml` / workflows / Makefile). **Three headline results:** (1) **the drift correlates with audience, not complexity** — every maintainer-facing page came back clean (drift-contracts, keyword-only, telemetry setup), every page explaining tooling *to someone else* drifted; (2) **docs edited in the same commit as their code do not drift** (proven: `d3550330a`, `15e0a94ea` each touched doc+code together) — direct evidence for the drift-contracts mechanism; (3) **the tooling class wants a CONTRACT, not a check** — every documented make target *exists*, so an existence script scores zero; what drifted is *behavior under a stable name* (`agent-test` grew a heartbeat 3.5 weeks after the page was written on the premise it was silent).-- **⚠ False-negative discipline — IT FINALLY CAUGHT SOMETHING (Part 4).** The rule (a *clean* result on a page seeded as high-risk is where the campaign most likely fools itself) held in Parts 1–3 every time. In Part 4 it **broke a clean verdict**: the language tutorial's four bundles all pass `pipelex validate bundle` *and* dry-run — and the page still teaches a **false rule** about the `@` vs `$` sigils (F13: it claims they differ by *data provenance*, input-vs-working-memory; they actually differ by *rendering shape*, block-on-its-own-line vs inline, and both resolve against the same `declared_inputs`). **Lesson with teeth for Stage 2 and for the contract design: a snippet validator verifies the snippet, never the sentence beside it.** With 109 confirmed and **0 ever rejected**, the pipeline's error mode is proven to be **false negatives, not false positives** — so keep the FN spot-check as a standing step.-- **⚠ Correction to the campaign's own record (fix at Stage 2):** Part 2's **F24 is right as written** (`pipelex-agent init --config` doesn't accept a `telemetry_mode` field) but its **class label — "fabricated `telemetry_mode` values" — is WRONG**. The values are real (`PostHogMode` StrEnum = `off`/`anonymous`/`identified`, `telemetry_config.py:27-32`) and `docs/setup/telemetry.md` documents them **correctly** (it FN-checked clean, decisively). A Stage-2 fixer trusting the class label could gut an accurate page. Fix F24 at `init-cli-flows.md:404`; leave `setup/telemetry.md` alone; correct the label in `under-the-hood-advanced.md`. **✅ DONE in S2-3 (all three parts honored).**-- **Part 2 recap (DONE 2026-07-12):** under-the-hood (19) + advanced (7) = 26 pages, 435 claims, 42 confirmed / 0 rejected / 0 reclassified (10 breaks / 23 wrong / 4 misleading / 5 stale; density 1.62, or 2.0 vs the 21 substantive pages — 5 `advanced/*-injection.md` are Under-Construction stubs). Findings in `wip/drift-hunt/findings/under-the-hood-advanced.md`. Dominant class: renamed Python API never propagated into docs.-- **Part 1 recap (DONE 2026-07-12):** building-methods (35 pages), 452 claims, 47 confirmed / 0 rejected → **3 reclassified as authoring convention (D11) → 44 drift** (25 breaks / 11 wrong / 6 misleading / 2 stale; density 1.26). Findings in `wip/drift-hunt/findings/building-methods.md`.-- **Next:** **NOTHING — campaign closed at Checkpoint 2 (2026-07-12).** Awaiting Louis' final review of the close-out. Campaign branch: do NOT push unless Louis asks (origin copy is stale at `2cb9c5957`, branch is ahead — expected). Post-campaign follow-ups live outside this tracker: the drift-contracts Phase 3 verdict consumes the shortlist in `findings/SUMMARY.md`; the 10 deferred code findings are ordinary follow-ups from `deferred-code-findings.md`.-- **Carry-overs into Stage 2:** ✅ ALL DONE in S2-4 — `docs/tools/cli/agent-cli.md:38` fixed; the `PipeStructure` roster fixed on all 3 pages (incl. a new PipeStructure section on `features/pipe-operators.md`). (The F24 class-label correction was ✅ done in S2-3.)-- **Reusable wave script (for Stage C, the last fan-out):** `/Users/lchoquel/.claude/projects/-Users-lchoquel-repos-Pipelex--drift/268d8cee-6e55-4669-9dff-33187cf92158/workflows/scripts/drift-hunt-tail-part4-wf_0c7af057-a6e.js` — start from **this** one (not Part 3's): it carries the full rule stack (Part-3's A–G + Part-4's H–L) and the D11 carve-out. Swap `PAGES` + seeds. For Stage C, add the D3 ruling (cookbook ground truth is `../pipelex-cookbook`, resolve references there) and **budget for the FN spot-check as a first-class step, not an afterthought**.-- **Phase H carved out (2026-07-12, Louis-approved):** the drift-tool hardening shipped separately as **PR #1041 → dev MERGED** (`15e0a94ea`, branch `fix/Drift-ack-hardening`). **dev merged into this branch 2026-07-12** (`19f991300`): the 4 drift-ack files conflicted (not the "content-identical, no conflict" originally predicted — #1041 landed as a *reviewed superset* of the branch's early `8c7413afc`, adding the post-verify recheck + staging-failure rollback), all resolved by taking dev's superset, so the drift area is now byte-identical to dev. Gates green (agent-check clean, `test_drift_ack_cmd.py` 18 passed incl. #1041's 2 new tests). Branch is now 0 behind dev.-- **Branch / worktree:** `docs/Drift-hunt` off `dev`, in the `_drift` worktree (treat as repo root). Do NOT push unless Louis asks. Commits so far: tracker `5e3699c64`, hardening `8c7413afc`, backfill `4093b1628`, then Stage 0 artifacts + fixes, then **dev merge `19f991300`** + a `ruff-format` normalization `86b569acf`, then **Stage 1 Part 1 findings** (see git log).-- **Working artifacts (post-prune):** `wip/drift-hunt/` — `findings/SUMMARY.md` (the campaign record + close-out), `inventory.md` (the denominator), `deferred-code-findings.md` (the handoff). Everything else pruned at Checkpoint 2; recover via `git show drift-hunt-archive:wip/drift-hunt/<path>` (pushed tag pinning `9a6201a5e`).-- **Decisions taken:** D1–D10 below, plus H-A2 implemented (narrow warn→fail escalation — see Phase H).-- **Key learning (D6 vindicated):** `pipelex validate --all` was initially "confirmed dead" from `--help` (no group-level `--all`) and a fix was drafted — live execution proved a shortcut layer forwards it to `validate pipe`; fix reverted. Never conclude a CLI form is dead from `--help` alone; run it.-- **Open questions:** none — all rulings taken (D1–D19), campaign closed. The only pending act is Louis' final review.+- **Branch:** `fix/PipeCompose-bug` (off `dev`-lineage tip `e660283bf`), committed and PR'd to `dev` at Checkpoint 2.+- **The change** (all verified green together — `make agent-check` clean, full `make agent-test` passed):+ - Modified: `pipelex/pipe_operators/compose/structured_content_composer.py` (the whole fix: `_extract_native_scalar` matrix + `NATIVE_SCALAR_TARGET_TYPES` + `NativeScalarExtraction` NamedTuple at module top; Optional unwrap in `_get_field_expected_type`/`_get_nested_field_class`; `_convert_list_items_as_scalars`; loud `_validate_item_compatibility`)+ - Modified: `subject_grants.toml` (grant for `unwrap_optional`'s positional `annotation` param — do not let `make fko` demote it)+ - Modified (test support): `tests/integration/pipelex/pipes/operator/pipe_compose_structured/{compose_structured_models.mthds,models_for_pipe_compose.py,test_data.py}` (new holder concepts/models/constructs for native-scalar cases)+ - New: `pipelex/tools/typing/annotation_utils.py` (`unwrap_optional`)+ - New tests: `tests/e2e/test_pipe_compose_whole_stuff_mthds.py` + `tests/e2e/fixtures/compose_whole_stuff/`, `tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_pipe_compose_native_scalar_conversions.py`, `tests/unit/pipelex/tools/test_annotation_utils.py`+ - Docs (Phase 3): `docs/building-methods/pipes/pipe-operators/PipeCompose.md` gained a "Copying Whole Inputs Into Native Fields" section (conversion matrix + worked example with optional fields); `CHANGELOG.md` `[Unreleased]` gained the Fixed entry; docs sweep found no other page describing construct `from` semantics, and the `StructuredContentComposerTypeError` error page is generated boilerplate with no raising-condition prose — nothing else to align.+ - New: this `TODOS.md`+- **Resume with:** Phase 4 below — ONLY after a pipelex release ships the fix.+- **Quick re-verify:** the e2e command in "Red test" below (expect all tests passing), or the targeted suites in Phase 2's checklist.-## Decisions+## The problem in one paragraph-- **D1 — cocode is out of scope.** Not ready for this task (user call, 2026-07-05). The whole hunt runs on the pre-screen script + Claude agent fan-outs; do not reach for cocode even for sub-steps.-- **D2 — scope = hand-written pages under `docs/`.** Excluded as *generated*: `docs/errors/` (regenerated by `make gep`) and the generated gateway-models doc(s). Excluded as *freshly reviewed*: `docs/configuration/` and the CLI docs (`docs/tools/cli/` + `pipelex/cli/agent_cli/CLAUDE.md`) — they got a genuine review as the drift-contracts seed acks; re-sweeping them now would measure nothing. Everything else under `docs/` is in: building-methods, features, cookbook, under-the-hood, tools (non-CLI), advanced, reliability, get-started, contribute, setup, viewpoint, index/license.-- **D3 — cookbook ground truth is a sibling repo.** `docs/cookbook/` pages reference examples that live in `../pipelex-cookbook` (read-only ground truth), not in this tree. Reviewers and the pre-screen's path-existence check must resolve cookbook references there; never conclude "referenced example missing" from this tree alone.-- **D4 — fixes land in reviewable batches.** One commit per docs section at minimum; separate PRs per section if a batch grows beyond comfortable review size. Default base: `dev`.-- **D5 — no manifest growth during the hunt.** The drift-contracts dogfood freeze stays in force: the hunt produces *evidence* for which sections might earn a contract later; it does not add contracts to `drift.toml`.-- **D6 — findings must survive adversarial verification before anything is fixed.** No finding is acted on unless a second, independent agent confirms it against the code with `file:line` evidence. Fix agents re-verify at fix time; a finding that doesn't hold up gets recorded as rejected, not silently dropped.-- **D7 — cookbook cross-check is its own greenlight-gated stage (Louis, 2026-07-12).** Comparing `docs/cookbook/` pages against `../pipelex-cookbook`'s actual files runs as Stage C, separate from Stage 1 and ONLY on Louis' explicit greenlight — the cookbook may change soon. Cookbook pages are out of the Stage 1 fleet.-- **D8 — cross-repo references: minimal only; code defects get deferred, not fixed (Louis, 2026-07-12).** A short, named pointer to another repo's own docs is fine; workspace-relative paths a docs-site reader can't resolve, or restating another repo's content at length, is drift — inline the short essence here, leave the detail in its home repo. Code defects surfaced by the hunt are not fixed mid-campaign: record them in `wip/drift-hunt/deferred-code-findings.md`.-- **D9 — scope edges: all in (Louis, 2026-07-12).** The four pages D2 didn't name (`docs/agents/debugging-hanging-pytest-runs.md`, `docs/analytics/data-extraction.md`, `docs/distributed-execution/index.md`, `docs/CLAUDE.md`) AND the root-level `CONTRIBUTING.md` + `CODE_OF_CONDUCT.md` (the include-targets of the docs stubs) join Stage 1. Note: the two root files got no Stage 0 pre-screen (it scanned `docs/` only) — their review agents start without seed rows.-- **D11 — scaffolding omissions in teaching examples are authoring convention, not drift (Louis, 2026-07-12).** Omitting a top-level `domain`, a pipe `description`, or the `prompt` body in a focused example is NOT drift — a reader fills it in (with a placeholder) before running. **Still drift** (survives scaffolding): a written prompt that ignores its own declared input; a functionally-required input that IS the subject (e.g. PipeExtract's Image/Document); wrong field names / output types / enum values / defaults. Applies to ALL Stage-1 parts and Stage C — **seed every future review agent with this carve-out** alongside the FP do-not-flag list. Consequence for the derived-check candidate: `validate-doc-bundle-examples` must inject placeholder `description`/`domain` and tolerate an omitted prompt before running `pipelex validate bundle`, so it only flags defects that survive scaffolding.-- **D12 — Stage C (cookbook) is DEFERRED (Louis, 2026-07-12).** Not part of this campaign's remaining arc; supersedes D7's greenlight gate. The 21 cookbook pages stay unswept — say so plainly in the close-out (Stage 1 covered 104 of the 125 in-scope pages).-- **D13 — the `make agent-test` heartbeat claim gets fixed EVERYWHERE, in Stage 2 (Louis, 2026-07-12).** `agent-test` is no longer silent on success (start line + 20 s heartbeat + completion line — `Makefile:732-739`, `:49-65`, `:27`). Fix Part-4 F8's doc page **and** the two agent-facing instruction files that are its source: `CLAUDE.md:52` ("Zero output on success") and the **workspace-root `CLAUDE.md:160`** ("silent on success"). ⚠ **The workspace-root file is in a DIFFERENT GIT REPO** (`/Users/lchoquel/repos/Pipelex` is its own repo; this worktree is `pipelex`) — it cannot ride this branch. It needs its own commit over there, in the same S2-4 session.-- **D14 — Stage 2 = 4 parts, ONE PR PER PART to `dev`, one cold-start session each (Louis, 2026-07-12).** Independent, not stacked: the parts touch disjoint page sets, so they can land in any order. Each part resumes cold from its own findings file alone.-- **D15 — fixers MAY restructure a broken teaching example (Louis, 2026-07-12).** When a correct fix needs more than a one-line patch (e.g. a prompt that ignores its own declared input), the fixer may rewrite the example so it is correct and actually runs. **Bounded:** no new content, no editorial rewrites, and the example must **re-validate against the real parser** before the fix counts. This is what many of the ~18 building-methods bundle-example findings actually require.-- **D16 — drift spotted DURING a fix: fix in place, if it clears the same bar (Louis, 2026-07-12).** Fix agents re-verify against the code (D6), so they will find drift Stage 1 missed — F13 proves the sweep is not exhaustive. If the new drift is **on a page already in the batch** and clears the same evidence bar (`file:line` on both sides, adversarially verified), fix it in the same batch and **append it to the findings file** (the audit trail must stay complete). Drift on an out-of-scope page is **recorded, not fixed**.-- **D17 — Stage 2 stays DOCS-ONLY (Louis, 2026-07-12).** No derived checks, no new `drift.toml` contracts (D5 holds). The contract/check shortlist is handed to the drift-contracts Phase 3 verdict at Checkpoint 2. Keeps every Stage-2 PR pure-docs and the campaign's evidence honest — and Part 4 is the cautionary tale: a green snippet check reads as a clean page when it is not (F13).-- **D18 — ⛔ NO Stage-2 part starts without Louis' explicit greenlight (Louis, 2026-07-12).** Same gate as D10 governed Stage 1: per part, he says go. Do not begin fixing because the plan looks settled.-- **D19 — elided sequence sketches are authoring convention, not drift (Louis, 2026-07-12).** A sequence sketch whose step pipes are absent from the snippet is elision-by-design **iff the snippet validates once placeholder sub-pipes are scaffolded** (and no prose claims it is complete or runnable). Ruled during S2-2 (the native-concepts "In Complex Methods" refutation; same pattern at `working-memory.md:24-27`). D11-adjacent: seed every future review/fix/verify agent with this carve-out alongside D11's, so the pattern is ruled uniformly instead of re-litigated per page. Still drift: a sketch that fails even with placeholder sub-pipes scaffolded (wrong step syntax, bad concept refs, sequence-output mismatch).-- **D10 — Stage 1 runs as 4 parts, one cold-start session each (Louis, 2026-07-12).** Full fan-out, worst-first, but split so context is cleared between parts and each part resumes cold from this tracker: Part 1 building-methods (35) → Part 2 under-the-hood + advanced (26) → Part 3 features + reliability (25) → Part 4 the tail: get-started, contribute, tools non-CLI, setup, viewpoint, root index/license, the D9 additions (18). Per part: run the wave Workflow, commit `findings/`, update Cold-start context (findings count + token burn), report to Louis — he greenlights the next part (or stops) before a new session picks it up.+A user designing a method with `/pipelex-design` used PipeCompose construct mode to copy whole native stuffs into native-typed fields of a structured output — `rejection_email = { from = "email" }` where `email` is a whole `Text` stuff, and `interview_questions = { from = "questions" }` where `questions` is a whole `Text[5]` stuff. Structural validation accepted it; the dry-run runnable gate rejected it because the composed field received the content wrapper object (`TextContent`, `ListContent[...]`) instead of the native value (`str`, `list[str]`). The user's usage was correct: the PipeCompose reference (`docs/building-methods/pipes/pipe-operators/PipeCompose.md`, "Copy value from input variable or nested field") and the composer's own design (`_resolve_from_var` docstring: "TextContent -> str: extract .text", "ListContent -> list[X]: extract items") both promise exactly this conversion. The implementation has gaps. **Verdict: fix pipelex, do not change the language or warn users off the pattern.**-## What "drift" means here (and what it doesn't)+Full analysis with the user's original bundle and reasoning: `/Users/lchoquel/repos/Pipelex/mcp-demos/wip/pipe-compose-issue/README.md` (+ runnable repro under `repro/`). Note: that README's §5/§8 conclude "whole-stuff `from` hands over the wrapper **by design**" — that is a misdiagnosis of these bugs as a language rule; this plan is the correction.-In scope per page: statements the code can contradict — commands, flags, config keys, defaults, file paths, API/class/function names, described behaviors and sequences, code snippets that should run or parse. Out of scope: editorial quality, tone, structure, SEO, completeness wishes. This is a drift hunt, not a rewrite; standing repo rules apply to fixes (docs = current reality, no hardcoded counts, MkDocs conventions).+## Diagnosis — three gaps, all in `pipelex/pipe_operators/compose/structured_content_composer.py`-## Checkpoint protocol (applies to every checkpoint below)+All verified empirically against the dev tree (v0.39.1 era, branch `dev`):-1. **Gates:** `make agent-check` (any code change), `make docs-check` (mkdocs strict — any doc change), full `make agent-test` at checkpoints.-2. **Commit** the phase's work (per D4 granularity for doc fixes). No push.-3. **Update the Cold-start context** section above (status, commit SHAs, decisions, open questions) so a fresh session can resume cold.-4. **Fan out `/code-review`** on the phase's changes: Sonnet-5 sub-agent(s), spawned with **no inherited context** — hand each one only a pointer to the changes (commit SHA or `git diff <base>..HEAD` range), never the plan, the rationale, or our own conclusions. Triage findings for clean-solid-not-over-engineered; apply what survives.-5. **STOP for Louis** where marked — Checkpoints 0, 1, and 2 are user-review gates.+1. **`Optional[...]` target fields are never converted.** Non-required structure fields generate `Optional[X]` annotations (`pipelex/core/concepts/structure_generation/generator.py:302-304`). `_get_field_expected_type` (`structured_content_composer.py:377`) returns the raw union annotation, and every `_expects_*` helper returns False on it (`issubclass(Optional[str], str)` raises TypeError → False at `:392-408`; `get_origin(Optional[List[str]])` is `Union`, not `list`, at `:474-484`), so `_convert_for_target_type` falls into its "unknown target type, return as-is" fallback and pydantic then rejects the wrapper. **Proof of oversight, not decision:** the NESTED path already unwraps Optional in `_get_nested_field_class` (`:704-717`); the FROM_VAR path never got the same treatment. And flipping the field to `required = true` makes the same whole-Text copy convert cleanly to `str`.+2. **List items that are native scalars are not extracted.** Even with a required `list`/`item_type = "text"` target (`List[str]`), `_convert_list_items_as_dicts` (`:501`) dumps each `TextContent` item as a dict `{"text": "..."}` via `model_dump` instead of extracting `.text` → `questions.N: string_type` pydantic errors. Related: `_validate_item_compatibility` (`:555`) silently no-ops when the expected item type has no `model_validate` (e.g. `str`), so the mismatch surfaces as a confusing downstream pydantic error rather than a clear conversion error.+3. **Scalar wrappers other than `TextContent` are not handled at all.** `_convert_for_target_type` (`:298`) special-cases only `TextContent` and `ListContent`. A whole `Number` stuff into a required `number` field fails with `NumberContent (expected float)`. Same family presumably applies to `YesNo` → `boolean` and `Date` → `date` fields.-## Deferred drift-contracts items (track here, do NOT do during the hunt)+Reference matrix of what works vs. breaks today (from the repro):-- [ ] **Phase 3 — pilot verdict:** per-contract keep / narrow / mechanize / drop decision, driven by `wip/drift-contracts/dogfood-log.md`. Gated on the dogfood period verdict; the hunt feeds it evidence (Stage 1 logs observations; Checkpoint 2 hands over the contract-candidate shortlist). **Working doc drafted 2026-07-13: `wip/drift-contracts/phase-3-verdict.md` — awaiting Louis' rulings on its decision boxes.** (Original phase breakdown: the `docs/Update` branch is gone; the surviving source is `wip/drift-contracts-design.md` → Rollout plan.)-- [ ] **lint-drift promotion into lint-all / required CI check:** gated on the open-PR backlog landing (a required check would fail every open PR that predates its acks). Track only.-- [x] **PR #1040 hardening findings (A1/A2, `wip/drift-contracts/pr-1040-review-notes.md`):** resolved in Phase H below.+| Construct source | Target field | Today |+|---|---|---|+| `{ from = "score.candidate_name" }` (dotted path to scalar leaf) | any matching native | ✅ works (raw Python value, skips conversion) |+| literal (`is_fit = true`) | matching native | ✅ works |+| `{ from = "<whole Text stuff>" }` | required `text` | ✅ works (designed conversion fires) |+| `{ from = "<whole Text stuff>" }` | optional `text` | ❌ gap 1 |+| `{ from = "<whole Text[] stuff>" }` | required `list`/`item_type="text"` | ❌ gap 2 |+| `{ from = "<whole Text[] stuff>" }` | optional `list`/`item_type="text"` | ❌ gaps 1+2 (the user's exact case) |+| `{ from = "<whole Number stuff>" }` | required `number` | ❌ gap 3 |----+## Red test (already written, this is the gate)-## Phase H — hardening pass (drift tool, from PR #1040 review notes)+- `tests/e2e/test_pipe_compose_whole_stuff_mthds.py` — one dry-run test (`validate_bundle` must pass the runnable gate on every pipe) + one parametrized live test (deterministic PipeCompose execution, no inference, asserts composed values AND exact native types).+- Fixture bundle: `tests/e2e/fixtures/compose_whole_stuff/compose_whole_stuff.mthds` — one holder concept + one compose pipe per row of the matrix above.+- Current state: `whole_text_to_required_text_field` (control) passes; the dry-run test and the other live cases fail with the wrapper-mismatch errors quoted in the diagnosis.+- Run it: `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/e2e/test_pipe_compose_whole_stuff_mthds.py`-- [x] **H-A1 — `drift ack` auto-stages the ack file it writes.** `git add .drift/acks/<id>.toml` right after `save_ack`, via a new `stage_file` git-adapter function (subject grant recorded BEFORE running checks). Makes the local gate honest: the ack lands in the same index `drift check` reads. Temp-repo test: ack path appears in `git diff --cached --name-only` after `drift ack`.-- [x] **H-A2 — DECISION: implement the narrow warn→fail escalation.** When a contract has `verify_commands` AND an unstaged-modified or untracked file matches its triggers, `drift ack` fails instead of warning — that is the one case where the verify guarantee is meaningfully weakened (verify ran on working-tree content the digest does not cover). Check runs BEFORE the verify commands (fail fast, don't burn a `make tb`). No-verify contracts keep the warn-only behavior. Tests: verify-contract + dirty trigger → `DriftAckError`, ack not written, verify not run; no-verify contract keeps warning + writing.-- [x] Update `docs/contribute/drift-contracts.md` + `.claude/skills/drift-review/SKILL.md` (ack now auto-stages; warn→fail rule) + `CHANGELOG.md` [Unreleased].-- [x] Gates: `make agent-check`, targeted tests (`tests/unit/pipelex/cli/dev/`), `make docs-check`.+## Fix design-## Phase B — dogfood-log backfill+All changes in `pipelex/pipe_operators/compose/structured_content_composer.py` (+ possibly a small typing util):-- [x] Backfill the missing mandatory entry in `wip/drift-contracts/dogfood-log.md` for the `config-docs` ack recorded 2026-07-11 during the subject-grants regrind (ack rationale: Python-3.10 drop / StrEnum swap in `img_gen_job_components.py` via the Phase-M dev merge — clean-pass). Mark it honestly as a backfill.+- **Optional unwrap (gap 1):** add a helper that unwraps `Optional[X]`/`X | None` to `X` when there is exactly one non-`None` arm (handle both `typing.Union` and `types.UnionType` origins — mirror the logic already in `_get_nested_field_class:704-717`). Apply it in `_get_field_expected_type`, and refactor `_get_nested_field_class` to use the same helper. Candidate home if made reusable: `pipelex/tools/typing/`.+- **Scalar wrapper matrix (gap 3):** extend `_convert_for_target_type` so each native scalar wrapper converts to its native target: `NumberContent.number` → `float`/`int`, `YesNoContent.yes_no` → `bool`, and (decision D1) `DateContent` → `date`. Keep the existing "target expects the wrapper class → keep object" behavior symmetrical for each (like `_convert_text_content` does for `TextContent`).+- **List item scalar extraction (gap 2):** in the `list[X]` target path, when the expected item type is a native scalar (`str`/`float`/`int`/`bool`/`date`), extract the scalar from each item wrapper using the same per-wrapper conversion as above, instead of `model_dump`. Keep dict-dumping for structured item types (that path is correct and tested).+- **Loud failure for unconvertible items:** make `_validate_item_compatibility` raise `StructuredContentComposerTypeError` when the expected item type is not a model and no scalar extraction applies, instead of silently passing the item through to a cryptic pydantic error.+- Dotted-path resolution (`_resolve_dotted_path`) already funnels StuffContent leaves through `_convert_for_target_type`, so gaps 1 and 3 fixes benefit it automatically; raw leaves are untouched.-### CHECKPOINT H — verify + review (no user stop)+### Decisions to arbitrate (ask Louis if unsure)-- [x] Checkpoint protocol steps 1–4 (commit Phase H + B; `/code-review` fan-out on the hardening commit only — Phase B is a log entry). Apply surviving findings, then proceed.+- **D1 — matrix scope:** `Text` + `Number` + `YesNo` are clearly in. Is `DateContent` → `date` in scope now (`DateContent` carries `date` + optional `time`; target `date` fields would take `.date`)? Recommendation: include `Date`, skip `Time`/`datetime` until a use case shows up.+- **D2 — true multi-arm unions** (e.g. a field typed `str | int`): recommendation: unwrap ONLY the Optional shape (single non-None arm); leave genuine unions on the current as-is fallback and revisit on demand.+- **D3 — static (pre-dry-run) detection:** a structurally impossible construct source (e.g. whole `Image` stuff → `text` field) could be caught at blueprint validation instead of dry-run. Out of scope here; if judged worthwhile, record as a deferred follow-up in `wip/` per the deferred-items convention.----+### Rulings taken (Checkpoint 1)-## Stage 0 — inventory + mechanical pre-screen+- **D1 ruling:** followed the recommendation — matrix is `TextContent → str`, `NumberContent → float/int` (never `bool`, an `int` subclass), `YesNoContent → bool`, `DateContent → date` (exact `date` target only, never `datetime`, a `date` subclass). `Time`/`datetime` targets deferred until a use case shows up.+- **D2 ruling:** followed the recommendation — `unwrap_optional` unwraps ONLY the single-non-None-arm shape (both `Optional[X]` and `X | None` origins); genuine multi-arm unions are returned unchanged. Side effect: `_get_nested_field_class` previously picked the *first* non-None arm of a multi-arm union arbitrarily; it now leaves such unions untouched (same as the FROM_VAR path).+- **D3:** untouched, remains a candidate deferred follow-up.-Deterministic checks that need no judgment, to make the semantic pass sharper and cheaper. The seed-ack dogfood showed this class is rich (dead config section, phantom CLI flag, dead config field — all mechanically detectable).+### Deviations from the fix design (Checkpoint 1)-- [x] Build the inventory: every in-scope `.md` page (per D2), grouped by section, committed as `wip/drift-hunt/inventory.md` with a page count per section — the denominator for defect density later. **Scope note for Checkpoint 0:** D2's list doesn't name `docs/agents/`, `docs/analytics/`, `docs/distributed-execution/`, nor root-level `changelog.md` / `contributing.md` / `CODE_OF_CONDUCT.md` — inventory them as in-scope-pending-confirmation (hand-written, not generated, not freshly reviewed) and let Louis rule at Checkpoint 0.-- [x] Write the pre-screen script at `wip/drift-hunt/prescreen.py` (throwaway campaign tooling — self-contained, stdlib + repo venv only). Checks, each producing `(page, claim, verdict, evidence)` rows:- - Referenced repo file paths exist in the tree (special-case `docs/cookbook/` → resolve against `../pipelex-cookbook`, per D3).- - Mentioned config keys exist in `pipelex/pipelex.toml` / the `configs.py` model tree.- - Mentioned `pipelex` / `pipelex-dev` CLI commands and flags exist in the actual `--help` surfaces.- - Fenced `toml` blocks parse; fenced `python` blocks compile (`ast.parse`).- - Internal doc links resolve (complementing, not re-implementing, the mkdocs strict build).-- [x] Run it; triage the raw hits into `wip/drift-hunt/suspects.md` — per section: confirmed-mechanical defects vs needs-judgment flags vs false positives (note false-positive patterns; they tune Stage 1 prompts).-- [x] Fix the confirmed mechanical defects immediately (they need no Stage 1) — one commit, gate: `make docs-check` green.+- Instead of one `_convert_*_content` method per wrapper, a single `_extract_native_scalar` conversion-matrix helper runs first in `_convert_for_target_type` (shared by the field path and the list-item path); wrapper → wrapper-class targets keep flowing through the generic `_convert_content_for_field` path, so the "target expects the wrapper class → keep object" symmetry is preserved for every wrapper. `_convert_text_content`'s str-extraction branch moved into that shared helper.+- `_validate_item_compatibility` raises only when the expected item type is a **native scalar** (str/float/int/bool/date), not for every non-model type: non-model item types such as `dict[...]` legitimately validate from the `model_dump` output, so a blanket raise would have regressed them. The list[native-scalar] path normally diverts to `_convert_list_items_as_scalars` (which raises its own clear error) before reaching that guard.+- The unwrap helper landed as `pipelex/tools/typing/annotation_utils.py::unwrap_optional` (subject grant recorded for the positional `annotation` param), unit-tested in `tests/unit/pipelex/tools/test_annotation_utils.py` (flat dir, matching the existing tools test layout).+- Perimeter tests went into a new sibling module `test_pipe_compose_native_scalar_conversions.py` (1 TestClass per module rule) rather than extending `test_pipe_compose_content_conversions.py`; coverage includes a falsy `YesNoContent(yes_no=False) → bool` case and `DateContent → date`.-### CHECKPOINT 0 — STOP (suspect list in hand)+## Checklist-- [x] Checkpoint protocol steps 1–4: full `make agent-test` green, Stage 0 outputs + fixes committed, Cold-start context updated, `/code-review` fan-out on the fix commit came back clean (all five fixes verified accurate; one adjacent staleness found in the same table — image_reference.py → shared/ — applied as a follow-up commit).-- [x] **Louis' triage rulings (2026-07-12):** judgment seeds resolved and FP patterns confirmed — recorded as D7/D8, in `suspects.md`, and in `deferred-code-findings.md`.-- [x] **Louis' scope + shape rulings (2026-07-12):** scope edges all in (D9); Stage 1 = full fan-out, worst-first, in 4 cold-start parts (D10). **Checkpoint 0 CLOSED — each Stage 1 part still needs its own explicit go.**+### Phase 1 — core fix (red → green)----+- [x] Diagnose the failure and validate the three gaps empirically (scratchpad repros: required-vs-optional discriminator, required-list failure, Number failure)+- [x] Red e2e test: fixture bundle + test module (see "Red test" above), verified red, `make agent-check` green+- [x] Optional-unwrap helper + use it in `_get_field_expected_type` and `_get_nested_field_class`+- [x] Scalar wrapper conversion matrix in `_convert_for_target_type` (D1 scope)+- [x] List-item native-scalar extraction in the `list[X]` path+- [x] `_validate_item_compatibility` raises clearly on unconvertible item types+- [x] Red e2e module fully green (dry-run + all live cases)-## Stage 1 — semantic review fan-out — ✅ **COMPLETE (all 4 parts, 2026-07-12)**+### Phase 2 — perimeter tests-Four parts, one cold-start session each (D10), 104 pages total. Cookbook pages excluded (Stage C, per D7). **Final: 1,439 claims, 109 drift, 3 convention, 0 rejected, density 1.05.** Burn: Part 1 ≈ 4.5M + Part 2 ≈ 4.8M + Part 3 ≈ 2.7M + Part 4 ≈ 2.7M tokens.+- [x] Sibling module `test_pipe_compose_native_scalar_conversions.py` with: TextContent → `Optional[str]` field, ListContent → `Optional[list[str]]` field, ListContent[TextContent] → `list[str]`, NumberContent → `float`, YesNo (falsy) → `bool`, Date → `date`, using registered Python models with Optional annotations+- [x] Unwrap helper unit-tested (`tests/unit/pipelex/tools/test_annotation_utils.py`)+- [x] `make agent-check` green+- [x] Targeted suites green: `tests/unit/pipelex/pipe_operators/ tests/integration/pipelex/pipes/` + the e2e module+- [x] Full `make agent-test` green-- [x] **Part 1 — building-methods (35 pages). DONE 2026-07-12.** 47 confirmed / 0 rejected → **44 drift** after 3 convention reclassifications (D11). → `findings/building-methods.md`, scoreboard `findings/SUMMARY.md`. Dominant class: doc bundle examples that fail `pipelex validate bundle` even after scaffolding (~18). Strongest drift-contract candidate: `validate-doc-bundle-examples` (with placeholder injection, per D11).-- [x] **Part 2 — under-the-hood + advanced (26 pages). DONE 2026-07-12.** 42 confirmed / 0 rejected / 0 convention-reclassified → 42 drift (10 breaks / 23 wrong / 4 misleading / 5 stale; density 1.62, or 2.0 vs 21 substantive pages — 5 `advanced/*-injection.md` are Under-Construction stubs). → `findings/under-the-hood-advanced.md`. Dominant class: renamed Python API/method never propagated into docs (the #1 cross-part class now). Strongest new drift-contract candidates: `doc-python-snippets-import` (import+getattr on doc symbols), `doc-python-signatures-vs-live` (keyword-only drift), `doc-toml-config-vs-shipped`.-- [x] **Part 3 — features + reliability (25 pages). DONE 2026-07-12.** 10 confirmed / 0 rejected / 0 convention-reclassified → 10 drift (2 breaks / 7 wrong / 1 misleading / 0 stale; **density 0.40 — the cleanest section**, 8/25 dirty; all 4 reliability pages clean). → `findings/features-reliability.md`. Dominant class: stale CLOSED enumerations (a list written once, the registry grew, the list never revisited). Strongest new drift-contract candidate: `doc-enumerations-vs-registry`. Two new review rules (exhaustive-vs-illustrative lists; floor counts) — reuse in Part 4. Carry-over → `docs/tools/cli/agent-cli.md:38`.-- [x] **Part 4 — the tail (18 pages). DONE 2026-07-12.** 12 confirmed by the wave + 1 by the post-run FN check / 0 rejected / 0 reclassified → **13 drift** (2 breaks / 9 wrong / 2 misleading / 0 stale; density 0.72, 9/18 dirty). → `findings/the-tail.md`. Dominant class: **repo-tooling drift** (make targets, generated trees, `mkdocs.yml`, `robots.txt`) — and crucially **not** a phantom-target class: every documented target exists; what drifted is *behavior under a stable name*. Strongest new drift-contract candidate: **`make-targets-and-tooling` — a contract, NOT a derived check** (nothing mechanizable would have caught any of it). Two new rules added (H make-targets, K docs-site meta) plus two scope rules (I process prose out of scope, J version floors). **STAGE 1 COMPLETE ⟶ Checkpoint 1.**+**CHECKPOINT 1 — fix landed and verified.** Update this file: check off the boxes, note any deviation from the fix design, record D1/D2 rulings. Good handoff point before docs.-Per part (the wave protocol) — **all four parts complete:**+### Phase 3 — docs + changelog (this repo)-- [x] Per page, a review agent: read the page, extract its checkable claims, verify each against the current code, report findings with severity (`breaks-a-user` / `wrong` / `misleading` / `stale-detail`) and `file:line` evidence for both the doc claim and the contradicting code. Seed each agent with the page's Stage 0 suspect rows and the Checkpoint-0-blessed do-not-flag list (`suspects.md` FP patterns).-- [x] Adversarial verify pass (D6): every finding independently re-derived by a verifier agent prompted to refute it; only confirmed findings survive. Run as a pipeline (page-review → verify per finding), not a barrier. **Result across Stage 1: 109 confirmed, 0 ever rejected.**-- [x] **FN spot-check (added in Part 3, promoted to a standing step in Part 4):** hand-verify the *clean* verdicts on the pages seeded as highest-risk — that is where the campaign is most likely to fool itself. It held in Parts 1–3 and **broke a verdict in Part 4** (F13). With 0 rejections across the whole of Stage 1, the pipeline's error mode is **false negatives, not false positives**. Keep this step in Stage C.-- [x] Aggregate into `wip/drift-hunt/findings/<section>.md` (findings ranked by severity), building toward `findings/SUMMARY.md`: per-section counts, defect density (findings / pages), rejected-findings list with reasons.-- [x] Log observations relevant to the drift-contracts dogfood — they feed the Phase 3 verdict, per D5. **Headline: drift correlates with *audience*, not complexity** (author-is-the-reader pages stay current; explain-it-to-someone-else pages drift), and **docs edited in the same commit as their code do not drift at all** — which is the drift-contracts thesis, observed in the wild.-- [x] End of part: commit the findings, update the Cold-start context, report to Louis.+- [x] `docs/building-methods/pipes/pipe-operators/PipeCompose.md`: document the whole-stuff copy in construct mode — worked example copying a whole `Text` and a whole `Text[]` input into native fields, including an optional field; state the conversion matrix (wrapper → native target, wrapper → wrapper target)+- [x] `CHANGELOG.md` `[Unreleased]` → Fixed: PipeCompose construct `{ from = "..." }` now converts whole native stuffs into optional native fields, native-scalar list items, and non-Text scalar wrappers (Number/YesNo/…)+- [x] Grep `docs/` for other PipeCompose construct mentions that describe or constrain `from` semantics; align them (no other page describes construct `from`; error pages are generated boilerplate — nothing to change)-### CHECKPOINT 1 — ✅ CLOSED (Louis, 2026-07-12)+**CHECKPOINT 2 — ready for PR.** Commit, run the pre-push gates, open PR to `dev`. The cross-repo items below are release-gated — do NOT do them before a pipelex release ships the fix.-- [x] Checkpoint protocol steps 1–4 (findings committed `32792618d`; full `make agent-test` green; review fan-out n/a — no code/tooling changed).-- [x] **Louis' rulings — recorded as D12–D17 below.** Stage C deferred; the `agent-test` heartbeat gets fixed everywhere (incl. both `CLAUDE.md`s) in Stage 2; Stage 2 = 4 parts, one PR each; fixers may restructure a broken example; mid-fix drift is fixed in place on the same page; Stage 2 stays docs-only.+### Phase 4 — cross-repo follow-ups (release-gated)------## Stage C — cookbook cross-check — ⏸️ **DEFERRED (Louis, 2026-07-12 — D12)**--Not part of this campaign's remaining arc. The 21 `docs/cookbook/` pages stay **unswept**; note that in the close-out so the campaign's coverage claim stays honest (Stage 1 covered 104 of the 125 in-scope pages). If it is ever revived: compare each page against `../pipelex-cookbook` (read-only ground truth, D3), reuse the Part-4 wave script (it carries the full rule stack), and budget the FN spot-check as a first-class step.-------## Stage 2 — fix in batches (shape settled by D14; **4 parts, one cold-start session each**)--Mirrors Stage 1's session-sharding: each part resumes cold from **its own findings file alone**, lands as **one PR to `dev`** (D14), and touches a page set disjoint from the other parts — so the four can land in any order without conflicts.--**Recommended order: S2-3 → S2-1 → S2-2 → S2-4.** S2-3 first because it is the highest volume but the *lowest effort per fix* (mostly one-line renamed-API substitutions) — it shakes out the fix loop cheaply. S2-1 second, while attention is freshest, because it is the hardest. S2-4 last because it absorbs every cross-cutting sweep.--- [x] **S2-1 — building-methods / `pipes/`** (30 findings, 17 pages). **DONE 2026-07-12 — PR #1043 MERGED to dev (squash `126616a88`)** (branch `docs/Drift-hunt-s2-1`, deleted; campaign-branch commits `7942339cc` + `dd6174ee6`). All 30 fixed, 0 rejected-at-fix-time; every changed bundle example re-validated against the real parser (incl. dry-run), broken examples restructured per D15 (PipeSequence, PipeBatch, PipeParallel combine, PipeCondition, multiplicity, pipes/index). +F45–F54 fixed per D16 (11 candidates → 10 confirmed, 2 refuted). Blind post-fix review caught 2 defects in the batch's own text (the non-running `bundle_uris` run-a-bundle snippets on two pages; the F45 dedup gate) — fixed, snippets executed live in dry-run. 2 code-side deferrals recorded (items 6–7). The PR carries this page set's previously-unshipped Stage-0 fix, self-contained on dev. Full fix-time record in `findings/building-methods.md`.-- [x] **S2-2 — building-methods / concepts + top-level** (14 findings, 9 pages). **DONE 2026-07-12 — PR #1044 MERGED to dev (squash `46511ce9c`)** (branch `docs/Drift-hunt-s2-2`, deleted; campaign-branch commits `8f03eb6f2` + `1824f2db4`). All 14 fixed, 0 rejected-at-fix-time; every changed example re-validated against the real parser (libraries.md `execute()` + F3 sequence live in dry-run). +F55–F61 fixed per D16 (12 candidates → 7 confirmed + 1 out-of-definition stale pointer, 6 refuted). Blind post-fix review: zero defects. 2 code-side deferrals (items 8–9); 1 out-of-batch defect recorded (PipeLLM.md inline `@document`, S2-1 page). The PR carries this page set's previously-unshipped Stage-0 fixes (incl. `adapt-to-llm-prompting-style-*.md`), self-contained on dev. Full fix-time record in `findings/building-methods.md`.-- [x] **S2-3 — under-the-hood + advanced** (42 findings, 26 pages). **DONE 2026-07-12 — PR #1042 MERGED to dev (squash `9814e2f4c`)** (branch `docs/Drift-hunt-s2-3`, deleted; campaign-branch commits `10b4bae0a` + `af7106342`). All 42 fixed, 0 rejected-at-fix-time; +F43–F46 fixed per D16 (2 more candidates refuted — first refutations of the campaign); F24 handled per the Cold-start note; 2 code-side deferrals recorded. The PR also carries these pages' previously-unshipped Stage-0 fixes, keeping it self-contained on dev. Full fix-time record in `findings/under-the-hood-advanced.md`. **Post-PR bot-review triage (2026-07-12):** greptile+cubic left 6 unresolved threads → deduped to 4 issues, all 4 confirmed against the code (2 incomplete config tables now presented-as-exhaustive, 1 table row contradicting the corrected `--graph-no-data` info block, 1 "copied once" vs full-reset wording), fixed in PR-branch commit `532725251`, mirrored here via cherry-pick; all 6 threads replied + resolved.-- [x] **S2-4 — features + reliability + the tail + ALL cross-cutting sweeps** (23 findings, 25 pages + the sweeps). **DONE 2026-07-12 — PR #1045 MERGED to dev (squash `cb13e5b33`)** (branch `docs/Drift-hunt-s2-4`, deleted; finalization commit `1b776eca5` = Louis's review edits + 8 bot threads triaged; campaign-branch mirror `5e15bec08`). All 23 fixed, 0 rejected-at-fix-time; all four cross-cutting items landed (PipeStructure roster ×3, `agent-cli.md:38`, D13 heartbeat everywhere — incl. the `pipelex/kit/agent_rules/` SOURCE templates + `make rules` regeneration of `CLAUDE.md`/`AGENTS.md` after `check-rules` caught the direct edit, and the workspace-root `CLAUDE.md` in its own repo, commit `cc2b68c` on `feature/Follow-ups`; `docs/setup/telemetry.md` untouched). +F11 + F14–F22 fixed per D16 (13 candidates → 10 confirmed, 3 refuted). Blind post-fix review: zero defects; one snippet-import defect in the batch's own text caught pre-commit. 1 code-side deferral (item 10); 1 out-of-batch record (observer-provider-injection.md + advanced/index.md, S2-3 pages). Full fix-time record split across `findings/features-reliability.md` and `findings/the-tail.md`.--Per batch:--- [ ] Fix highest severity first within the part. Every fix **re-verified against the code at fix time** (D6) — the finding is the pointer, **the code is the authority**. A finding that no longer reproduces is marked rejected-at-fix-time, not silently patched.-- [ ] Gates: `make docs-check` (mkdocs strict) + `make check`. ⚠ **`docs-check` does NOT cover `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` or the `CLAUDE.md`s** — they are not mkdocs pages. Verify those by hand.-- [ ] Fixing docs does NOT reopen drift contracts (docs are review targets, not triggers). If a fix touches a trigger file, resolve via the `drift-review` skill.-- [ ] Mark resolved findings in the findings files as batches land (mark, don't delete — audit trail until close-out).-- [ ] Per-batch `/code-review` fan-out (checkpoint protocol step 4) before the PR is finalized.-- [ ] Open the PR to `dev` (D14), then report to Louis. Session ends; the next part runs cold.--### CHECKPOINT 2 — ✅ EXECUTED (2026-07-12) — STOPPED for Louis' final review--- [x] All batches landed (all four PRs merged to dev 2026-07-12, campaign branch reconciled, `make docs-check` + `make check` green on the reconciled branch) — **full `make agent-test` on this branch: GREEN** (and the run displayed the start-line + heartbeat lines the D13 fixes document).-- [x] Close-out written in `findings/SUMMARY.md`: coverage disclosure (**104 of 125 in-scope pages swept; the 21 cookbook pages were NEVER swept, D12**), Stage-2 resolution table (109/109 fixed, 0 rejected-at-fix-time; +31 fix-time findings + 1 stale pointer fixed, 13 refuted), the ranked drift-contract shortlist handed to the Phase 3 verdict (per D5/D17), and the campaign's lessons.-- [x] `wip/drift-hunt/deferred-code-findings.md` (10 items) handed off — kept in place, header updated. **The "partial out-of-batch observer page" turned out to be a stale-base artifact, NOT a real remainder:** verified at close-out, `docs/advanced/observer-provider-injection.md` (and `advanced/index.md`) is fully fixed and byte-identical to dev — Part 2's F38–F42 covered those exact defects and S2-3 (PR #1042, squash `9814e2f4c`) fixed them; S2-4's session recorded them as "remaining" because its base predated #1042's merge. Verified against live code: hooks `observer_protocol.py:17-31`, payload keys `pipe_router_protocol.py:18-47`, JSONL names `local_observer.py:11-14`, import + `Pipelex.make` kwargs + `hub.set_observer` (`hub.py:263`). No follow-up needed.-- [x] `wip/drift-hunt/` pruned: kept `findings/SUMMARY.md` + `inventory.md` (the record) + `deferred-code-findings.md` (the handoff); removed the four per-section findings files, `prescreen.py`, `prescreen-raw.md`, `suspects.md`, `checkpoint-0-decision-brief.md` — all recoverable at `git show drift-hunt-archive:wip/drift-hunt/<path>` (pushed tag pinning `9a6201a5e`).-- [x] Cold-start context updated to DONE.-- [ ] **STOP for Louis:** final review of the close-out; decide what happens to the campaign branch (do NOT push unless asked).-------## Out of scope (explicit)--- cocode, in any role (D1).-- Growing `drift.toml` (D5 — evidence now, contracts later).-- Editorial/style/structure rewrites, new content, SEO.-- Generated docs (`docs/errors/`, gateway models) — fix the generator if one of those is wrong.-- `docs/configuration/` + CLI docs (just reviewed via seed acks) — unless a Stage 1 finding elsewhere points back into them.-- Other repos' docs (read `../pipelex-cookbook` as ground truth; don't fix it — hand defects off via SUMMARY.md).+- [ ] `pipelex-plugins/skills/pipelex-design/references/writing-mthds.md` (source of truth; per-target copies under `pipelex-plugins/{pipelex,pipelex-vibe,pipelex-codex}/skills/...`): add a worked whole-stuff copy example to the PipeCompose construct section. Do NOT add a "from must reference a structured field" warning — that would enshrine the bug+- [ ] `mthds-plugins/mthds-dev/skills/shared/mthds-reference.md`: same check — align the construct `from` description if needed+- [ ] `mcp-demos/wip/pipe-compose-issue/README.md`: annotate §5/§8 — root cause was implementation gaps (fixed in pipelex <version>), not a language rule; the compose-based design the user abandoned is the recommended shape again+- [ ] Optional (user-facing): the shipped `recruitment_screening` method (`mcp-demos/pipelex-wip/recruitment_screening/`) can revert to the generate → compose two-step design to restore the structural carry-over guarantee — Louis/user's calldiff --git a/docs/building-methods/pipes/pipe-operators/PipeCompose.md b/docs/building-methods/pipes/pipe-operators/PipeCompose.mdindex 51e3e5ad8..a26803c06 100644--- a/docs/building-methods/pipes/pipe-operators/PipeCompose.md+++ b/docs/building-methods/pipes/pipe-operators/PipeCompose.md@@ -137,6 +137,49 @@ Each field in the `[pipe.name.construct]` section can use one of these methods:| Template | `{ template = "text with $var" }` | Generate string using template interpolation || Fixed | `"value"` or `123` or `true` | Use a static value directly |+### Copying Whole Inputs Into Native Fields++The `from` reference is not limited to dotted paths like `"customer.name"` — it can name a whole input variable. When the referenced input is a native stuff (`Text`, `Number`, `YesNo`, `Date`, or a list of them) and the target field is native-typed, the composer automatically converts the content wrapper into the field's native value. This works for required and optional fields alike.++Conversion matrix:++| Source input | Native target field | Composed value |+|---|---|---|+| `Text` | `type = "text"` | the text string |+| `Number` | `type = "number"` | the number |+| `YesNo` | `type = "boolean"` | the boolean |+| `Date` | `type = "date"` | the date |+| `Text[]` | `type = "list"`, `item_type = "text"` | the list of strings |++When the target field expects a content object rather than a native value (e.g. a field typed with a concept), the object is kept as-is — the conversion only fires when the field expects the native type.++One fidelity guard: a `Date` stuff that carries a time of day cannot be copied into a bare `date` field — that would silently drop the time and its UTC offset, so the composer raises an error instead. Target a `Date`-typed field to keep the full timestamp.++Worked example — assembling a report from whole stuffs produced by earlier steps:++```toml+[concept.ScreeningReport]+description = "The final screening report"++[concept.ScreeningReport.structure]+match_score = { type = "number", description = "The match score", required = true }+rejection_email = { type = "text", description = "The rejection email — optional" }+interview_questions = { type = "list", item_type = "text", description = "Questions to ask — optional" }++[pipe.assemble_report]+type = "PipeCompose"+description = "Assemble the screening report from previously generated pieces"+inputs = { score = "Number", email = "Text", questions = "Text[]" }+output = "ScreeningReport"++[pipe.assemble_report.construct]+match_score = { from = "score" }+rejection_email = { from = "email" }+interview_questions = { from = "questions" }+```++Here `email` is a whole `Text` stuff copied into an optional `text` field, `questions` is a whole `Text[]` stuff copied into an optional `list` of `text` items, and `score` is a whole `Number` stuff copied into a required `number` field. Each lands as its native value (`str`, `list[str]`, `float`).+### Construct Mode Example```tomldiff --git a/pipelex/pipe_operators/compose/structured_content_composer.py b/pipelex/pipe_operators/compose/structured_content_composer.pyindex 1908aebec..d68c9895d 100644--- a/pipelex/pipe_operators/compose/structured_content_composer.py+++ b/pipelex/pipe_operators/compose/structured_content_composer.py@@ -1,6 +1,6 @@+import datetimeimport inspect-import types-from typing import Any, Union, cast, get_args, get_origin+from typing import Any, NamedTuple, TypeAlias, cast, get_args, get_originfrom pydantic import ValidationError@@ -8,9 +8,12 @@ from pipelex import logfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.template_rendering import render_templatefrom pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.date_content import DateContentfrom pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.number_content import NumberContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.core.stuffs.yes_no_content import YesNoContentfrom pipelex.pipe_operators.compose.construct_blueprint import ConstructBlueprint, ConstructFieldBlueprint, ConstructFieldMethodfrom pipelex.pipe_operators.compose.exceptions import (StructuredContentComposerTypeError,@@ -18,9 +21,28 @@ from pipelex.pipe_operators.compose.exceptions import (StructuredContentComposerValueError,)from pipelex.pipe_run.pipe_run_params import PipeRunParams+from pipelex.tools.typing.annotation_utils import unwrap_optionalfrom pipelex.tools.typing.class_utils import are_classes_equivalentfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error+NativeScalarValue: TypeAlias = str | float | int | bool | datetime.date+"""The native Python scalar types a content wrapper can be unwrapped into."""++# The exact native scalar target types (no subclass matching: datetime is a date subclass+# and bool is an int subclass, and neither cross-kind conversion is wanted).+NATIVE_SCALAR_TARGET_TYPES: tuple[type[Any], ...] = (str, float, int, bool, datetime.date)+++class NativeScalarExtraction(NamedTuple):+ """Result of attempting to extract a native scalar from a content wrapper.++ The explicit `matched` flag distinguishes a genuine extraction from a no-match,+ because legitimately extracted values can be falsy (False, 0, empty string).+ """++ matched: bool+ value: NativeScalarValue | None+class StructuredContentComposer:"""Composes a StructuredContent instance from a ConstructBlueprint.@@ -146,9 +168,13 @@ class StructuredContentComposer:"""Resolve a FROM_VAR field by getting value from working memory.The resolution is type-aware: it checks what the target field expects- and converts content accordingly:+ (unwrapping Optional annotations) and converts content accordingly:- TextContent -> str: extract .text- - TextContent -> TextContent/subclass: keep object+ - NumberContent -> float/int: extract .number+ - YesNoContent -> bool: extract .yes_no+ - DateContent -> date: extract .date+ - any wrapper -> its own wrapper class/subclass: keep object+ - ListContent -> list[native scalar]: extract the scalar from each item- ListContent -> list[X]: extract items as dicts- ListContent -> ListContent: keep object@@ -280,7 +306,9 @@ class StructuredContentComposer:else:return current_value # pyright: ignore[reportUnknownVariableType]- def _resolve_from_stuff_name(self, name: str, *, expected_type: type[Any] | None) -> StuffContent | list[dict[str, Any]] | str:+ def _resolve_from_stuff_name(+ self, name: str, *, expected_type: type[Any] | None+ ) -> StuffContent | list[dict[str, Any]] | list[NativeScalarValue] | NativeScalarValue:"""Resolve a simple (non-dotted) path and convert content based on expected type.Args:@@ -295,11 +323,15 @@ class StructuredContentComposer:log.verbose(f" Stuff '{name}' content type: {type(stuff_content).__name__}")return self._convert_for_target_type(stuff_content=stuff_content, expected_type=expected_type)- def _convert_for_target_type(self, stuff_content: StuffContent, *, expected_type: type[Any] | None) -> StuffContent | list[dict[str, Any]] | str:+ def _convert_for_target_type(+ self, stuff_content: StuffContent, *, expected_type: type[Any] | None+ ) -> StuffContent | list[dict[str, Any]] | list[NativeScalarValue] | NativeScalarValue:"""Convert content based on the expected target field type.- Central dispatcher for type-aware conversion. Routes to specific- conversion methods based on content type.+ Central dispatcher for type-aware conversion. Native scalar wrappers are unwrapped+ first (TextContent -> str, NumberContent -> float/int, YesNoContent -> bool,+ DateContent -> date) when the target expects the native type; otherwise routes to+ specific conversion methods based on content type.Args:stuff_content: The content to convert@@ -308,6 +340,11 @@ class StructuredContentComposer:Returns:The content, converted as appropriate for the target field type"""+ if expected_type is not None:+ extraction = self._extract_native_scalar(stuff_content=stuff_content, expected_type=expected_type)+ if extraction.matched and extraction.value is not None:+ log.verbose(f" -> Target expects a native scalar, extracted {type(extraction.value).__name__} from {type(stuff_content).__name__}")+ return extraction.valueif isinstance(stuff_content, TextContent):return self._convert_text_content(text_content=stuff_content, expected_type=expected_type)elif isinstance(stuff_content, ListContent):@@ -321,21 +358,20 @@ class StructuredContentComposer:log.verbose(f" -> Unknown target type, returning {type(stuff_content).__name__} object")return stuff_content- def _convert_text_content(self, text_content: TextContent, *, expected_type: Any) -> TextContent | str:- """Convert TextContent based on expected type (str, TextContent, or subclass).+ def _convert_text_content(self, text_content: TextContent, *, expected_type: Any) -> TextContent:+ """Convert TextContent based on expected type (TextContent or subclass).++ The str target case is handled upstream by the native scalar extraction+ in _convert_for_target_type.Args:text_content: The TextContent to convertexpected_type: The expected type annotation for the target fieldReturns:- Either the text string, or the TextContent object (potentially converted)+ The TextContent object (potentially converted to the expected subclass)"""- if self._expects_type(expected_type=expected_type, target_type=str):- # Target field expects str, extract the text- log.verbose(" -> Target expects str, returning TextContent.text")- return text_content.text- elif self._expects_type(expected_type=expected_type, target_type=TextContent):+ if self._expects_type(expected_type=expected_type, target_type=TextContent):# Target field expects TextContent or subclassconverted_text_content = self._convert_content_for_field(stuff_content=text_content, expected_type=expected_type)assert isinstance(converted_text_content, TextContent)@@ -345,9 +381,56 @@ class StructuredContentComposer:log.verbose(f" -> Unknown target type, returning {type(text_content).__name__} object")return text_content+ def _extract_native_scalar(self, *, stuff_content: StuffContent, expected_type: type[Any]) -> NativeScalarExtraction:+ """Extract the native scalar value from a scalar content wrapper when the target expects it.++ Conversion matrix (wrapper -> native target):+ - TextContent -> str (or str subclass): extract .text+ - NumberContent -> float or int (never bool, an int subclass): extract .number+ - YesNoContent -> bool: extract .yes_no+ - DateContent -> date (exactly, never datetime, a date subclass): extract .date,+ but only when the Date carries no time of day — truncating a timestamped Date+ to a bare date would silently drop the time and its UTC offset, so it raises++ A wrapper-typed target (e.g. a NumberContent field) deliberately does not match here:+ it is handled by the generic StuffContent conversion path so the object is kept.++ Args:+ stuff_content: The content wrapper to extract from+ expected_type: The expected type annotation for the target field or list item++ Returns:+ A NativeScalarExtraction with matched=True and the scalar value when the+ wrapper/target pair is in the matrix, matched=False otherwise++ Raises:+ StructuredContentComposerTypeError: If a Date carrying a time of day targets a bare `date`+ """+ if isinstance(stuff_content, TextContent):+ if self._expects_type(expected_type=expected_type, target_type=str):+ return NativeScalarExtraction(matched=True, value=stuff_content.text)+ elif isinstance(stuff_content, NumberContent):+ if expected_type is not bool and (+ self._expects_type(expected_type=expected_type, target_type=float) or self._expects_type(expected_type=expected_type, target_type=int)+ ):+ return NativeScalarExtraction(matched=True, value=stuff_content.number)+ elif isinstance(stuff_content, YesNoContent):+ if self._expects_type(expected_type=expected_type, target_type=bool):+ return NativeScalarExtraction(matched=True, value=stuff_content.yes_no)+ elif isinstance(stuff_content, DateContent):+ if expected_type is datetime.date:+ if stuff_content.time is not None:+ msg = (+ f"Cannot copy a Date carrying a time of day ({stuff_content.rendered_plain()}) into a bare `date` field: "+ "it would silently drop the time and its UTC offset. Target a Date-typed field to keep the time."+ )+ raise StructuredContentComposerTypeError(msg)+ return NativeScalarExtraction(matched=True, value=stuff_content.date)+ return NativeScalarExtraction(matched=False, value=None)+def _convert_list_content(self, list_content: ListContent[StuffContent], *, expected_type: type[Any] | None- ) -> ListContent[StuffContent] | list[dict[str, Any]]:+ ) -> ListContent[StuffContent] | list[dict[str, Any]] | list[NativeScalarValue]:"""Convert ListContent based on expected type (list[X] or ListContent[X]).Args:@@ -355,7 +438,7 @@ class StructuredContentComposer:expected_type: The expected type annotation for the target fieldReturns:- Either a list of dicts, or a ListContent object with converted items+ Either a list of dicts, a list of native scalars, or a ListContent object with converted items"""expected_item_type: type[Any] | Noneif expected_type and self._expects_list_content_type(expected_type=expected_type):@@ -365,8 +448,12 @@ class StructuredContentComposer:converted_items = self._convert_list_items_as_objects(items=list_content.items, expected_item_type=expected_item_type)return ListContent(items=converted_items)elif expected_type and self._expects_list_type(expected_type=expected_type):- # Target expects list[X], extract items as dicts for Pydantic reconstructionexpected_item_type = self._get_list_item_type(expected_type=expected_type)+ if expected_item_type is not None and expected_item_type in NATIVE_SCALAR_TARGET_TYPES:+ # Target expects list of native scalars, extract the scalar from each item wrapper+ log.verbose(f" -> Target expects list[{expected_item_type}], extracting native scalars from ListContent items")+ return self._convert_list_items_as_scalars(items=list_content.items, expected_item_type=expected_item_type)+ # Target expects list[X] with structured items, extract items as dicts for Pydantic reconstructionlog.verbose(f" -> Target expects list[{expected_item_type}], extracting items from ListContent")return self._convert_list_items_as_dicts(items=list_content.items, expected_item_type=expected_item_type)else:@@ -377,6 +464,10 @@ class StructuredContentComposer:def _get_field_expected_type(self, field_name: str) -> type[Any] | None:"""Get the expected type annotation for a field from the output class.+ Optional annotations (`Optional[X]` / `X | None`) are unwrapped to their non-None+ arm, so an optional field converts exactly like its required counterpart. Genuine+ multi-arm unions are returned unchanged.+Args:field_name: The name of the field@@ -385,7 +476,7 @@ class StructuredContentComposer:"""field_info = self.output_class.model_fields.get(field_name)if field_info and field_info.annotation:- return field_info.annotation+ return cast("type[Any]", unwrap_optional(field_info.annotation))else:return None@@ -486,6 +577,9 @@ class StructuredContentComposer:def _get_list_item_type(self, expected_type: type[Any]) -> type[Any] | None:"""Extract the item type from list[X] or ListContent[X].+ A nullable item annotation (`list[X | None]`) is normalized to its non-None arm,+ so nullable items convert exactly like their non-nullable counterparts.+Args:expected_type: The type annotation (e.g., list[Address] or ListContent[TeamMember])@@ -494,7 +588,7 @@ class StructuredContentComposer:"""args = get_args(expected_type)if args:- return args[0] # type: ignore[return-value, no-any-return]+ return cast("type[Any]", unwrap_optional(args[0]))else:return None@@ -525,6 +619,36 @@ class StructuredContentComposer:log.verbose(f" Returning {len(converted_items)} items as dicts")return converted_items+ def _convert_list_items_as_scalars(self, *, items: list[StuffContent], expected_item_type: type[Any]) -> list[NativeScalarValue]:+ """Extract native scalar values from list item wrappers.++ Used when target is list[X] with X a native scalar type (str/float/int/bool/date):+ each item wrapper is unwrapped to its scalar value (e.g. TextContent -> str).++ Args:+ items: The list of item wrappers to convert+ expected_item_type: The native scalar type expected for each item++ Returns:+ List of native scalar values++ Raises:+ StructuredContentComposerTypeError: If an item is not a wrapper of the expected scalar type+ """+ log.verbose(f" Extracting {len(items)} items as native scalars, expected item type: {expected_item_type}")++ converted_items: list[NativeScalarValue] = []+ for idx, item in enumerate(items):+ extraction = self._extract_native_scalar(stuff_content=item, expected_type=expected_item_type)+ if not extraction.matched or extraction.value is None:+ expected_type_name = getattr(expected_item_type, "__name__", str(expected_item_type))+ msg = f"Cannot convert item[{idx}] from {type(item).__name__} to {expected_type_name}: no native scalar extraction applies"+ raise StructuredContentComposerTypeError(msg)+ converted_items.append(extraction.value)++ log.verbose(f" Returning {len(converted_items)} items as native scalars")+ return converted_items+def _convert_list_items_as_objects(self, items: list[StuffContent], *, expected_item_type: type[Any] | None) -> list[StuffContent]:"""Convert list items while keeping them as objects (for ListContent target).@@ -584,7 +708,17 @@ class StructuredContentComposer:log.verbose(f" Item[{idx}]: {actual_type.__name__} is structurally equivalent to {expected_type_name}")return- # Case 3: Try to validate via dict+ # Case 3: native scalar item type - a wrapper dump can never validate into a bare scalar,+ # so fail loudly here instead of letting Pydantic produce a cryptic downstream error.+ # (The list[native-scalar] path normally extracts scalars before reaching this method.)+ if expected_type in NATIVE_SCALAR_TARGET_TYPES:+ msg = (+ f"Cannot convert item[{idx}] from {actual_type.__name__} to {expected_type_name}: "+ f"a native scalar item cannot be built from a {actual_type.__name__} dump"+ )+ raise StructuredContentComposerTypeError(msg)++ # Case 4: Try to validate via dictlog.verbose(f" Item[{idx}]: Validating conversion {actual_type.__name__} -> {expected_type_name}")item_dict = item.model_dump(exclude_none=False, serialize_as_any=True)@@ -700,21 +834,7 @@ class StructuredContentComposer:# Get the field info from the Pydantic modelfield_info = self.output_class.model_fields.get(field_name)if field_info and field_info.annotation:- annotation = field_info.annotation- # Handle Optional types (both `Optional[X]` and Python 3.10+ `X | None` syntax)- # We need to specifically detect Optional types, not all generic types.- # - Optional[X] has origin Union and type(None) in args- # - X | None (Python 3.10+) has origin types.UnionType and type(None) in args- # Other generic types like list[X] should NOT be unwrapped.- origin = get_origin(annotation)- if origin is Union or origin is types.UnionType:- args = get_args(annotation)- if type(None) in args:- # It's an Optional type, get the non-None type- non_none_args = [arg for arg in args if arg is not type(None)]- if non_none_args:- annotation = non_none_args[0]- return annotation # type: ignore[return-value]+ return cast("type[StuffContent]", unwrap_optional(field_info.annotation))else:msg = f"Cannot determine class for nested field '{field_name}' in {self.output_class.__name__}"raise StructuredContentComposerTypeError(msg)diff --git a/pipelex/tools/typing/annotation_utils.py b/pipelex/tools/typing/annotation_utils.pynew file mode 100644index 000000000..01378ff05--- /dev/null+++ b/pipelex/tools/typing/annotation_utils.py@@ -0,0 +1,27 @@+import types+from typing import Any, Union, get_args, get_origin+++def unwrap_optional(annotation: Any) -> Any:+ """Unwrap an Optional annotation to its single non-None arm.++ Handles both `Optional[X]` / `Union[X, None]` (origin `typing.Union`) and the+ PEP 604 form `X | None` (origin `types.UnionType`). Only the Optional shape is+ unwrapped: a union with two or more non-None arms (e.g. `str | int | None`) is+ returned unchanged, as are non-union annotations.++ Args:+ annotation: The type annotation to unwrap++ Returns:+ The single non-None arm when the annotation is Optional-shaped, otherwise+ the annotation unchanged+ """+ origin = get_origin(annotation)+ if origin is not Union and origin is not types.UnionType:+ return annotation+ args = get_args(annotation)+ non_none_args = [arg for arg in args if arg is not type(None)]+ if len(non_none_args) < len(args) and len(non_none_args) == 1:+ return non_none_args[0]+ return annotationdiff --git a/subject_grants.toml b/subject_grants.tomlindex 2e31695fd..64c57444b 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -6756,6 +6756,10 @@ rationale = "Transforms the path; single operand."param = "path"rationale = "Parses/reads the path; single operand."+["pipelex/tools/typing/annotation_utils.py::unwrap_optional"]+param = "annotation"+rationale = "Reads as verb-object: unwrap_optional(annotation) takes the one obvious operand, the annotation to unwrap, and call sites pass self-labelling annotation expressions."+["pipelex/tools/typing/class_utils.py::has_compatible_field"]param = "model_cls"rationale = "Predicate on the model_cls; single operand."diff --git a/tests/e2e/fixtures/compose_whole_stuff/compose_whole_stuff.mthds b/tests/e2e/fixtures/compose_whole_stuff/compose_whole_stuff.mthdsnew file mode 100644index 000000000..0ce9ac426--- /dev/null+++ b/tests/e2e/fixtures/compose_whole_stuff/compose_whole_stuff.mthds@@ -0,0 +1,82 @@+domain = "compose_whole_stuff"+description = "PipeCompose construct pipes that copy a whole native stuff (Text, Text[], Number) into a native-typed field of a structured concept, via { from = \"<stuff_name>\" }"+main_pipe = "fill_required_text"++# One holder concept per target-field shape, so each pipe isolates exactly one conversion case.++[concept.RequiredTextHolder]+description = "A holder with a single required text field"++[concept.RequiredTextHolder.structure]+title = { type = "text", description = "The title", required = true }++[concept.OptionalTextHolder]+description = "A holder with a single optional text field"++[concept.OptionalTextHolder.structure]+note = { type = "text", description = "The note — optional" }++[concept.RequiredTextListHolder]+description = "A holder with a single required list-of-text field"++[concept.RequiredTextListHolder.structure]+tags = { type = "list", item_type = "text", description = "The tags", required = true }++[concept.OptionalTextListHolder]+description = "A holder with a single optional list-of-text field"++[concept.OptionalTextListHolder.structure]+tags = { type = "list", item_type = "text", description = "The tags — optional" }++[concept.RequiredNumberHolder]+description = "A holder with a single required number field"++[concept.RequiredNumberHolder.structure]+score = { type = "number", description = "The score", required = true }++# --- Compose pipes: one per conversion case ---++[pipe.fill_required_text]+type = "PipeCompose"+description = "Copy a whole Text stuff into a required text field (control case: works)"+inputs = { text_input = "Text" }+output = "RequiredTextHolder"++[pipe.fill_required_text.construct]+title = { from = "text_input" }++[pipe.fill_optional_text]+type = "PipeCompose"+description = "Copy a whole Text stuff into an optional text field"+inputs = { text_input = "Text" }+output = "OptionalTextHolder"++[pipe.fill_optional_text.construct]+note = { from = "text_input" }++[pipe.fill_required_text_list]+type = "PipeCompose"+description = "Copy a whole Text[] stuff into a required list-of-text field"+inputs = { texts_input = "Text[]" }+output = "RequiredTextListHolder"++[pipe.fill_required_text_list.construct]+tags = { from = "texts_input" }++[pipe.fill_optional_text_list]+type = "PipeCompose"+description = "Copy a whole Text[] stuff into an optional list-of-text field"+inputs = { texts_input = "Text[]" }+output = "OptionalTextListHolder"++[pipe.fill_optional_text_list.construct]+tags = { from = "texts_input" }++[pipe.fill_required_number]+type = "PipeCompose"+description = "Copy a whole Number stuff into a required number field"+inputs = { number_input = "Number" }+output = "RequiredNumberHolder"++[pipe.fill_required_number.construct]+score = { from = "number_input" }diff --git a/tests/e2e/test_pipe_compose_whole_stuff_mthds.py b/tests/e2e/test_pipe_compose_whole_stuff_mthds.pynew file mode 100644index 000000000..8f6da545a--- /dev/null+++ b/tests/e2e/test_pipe_compose_whole_stuff_mthds.py@@ -0,0 +1,97 @@+from __future__ import annotations++from pathlib import Path+from typing import Any++import pytest++from pipelex import pretty_print+from pipelex.core.stuffs.number_content import NumberContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipeline.bundle_validator import DryRunStatus+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.pipeline.validate_bundle import validate_bundle++_FIXTURE_DIR = Path(__file__).parent / "fixtures" / "compose_whole_stuff"+_BUNDLE = _FIXTURE_DIR / "compose_whole_stuff.mthds"++[email protected](loop_scope="class")+class TestPipeComposeWholeStuffConstruct:+ """PipeCompose construct `{ from = "<stuff_name>" }` must copy a whole native stuff into a native-typed target field.++ The documented contract (PipeCompose reference, "Copy value from input variable or nested field") is that+ referencing a whole input stuff converts its content wrapper to the target field's native type:+ TextContent -> str, ListContent[TextContent] -> list[str], NumberContent -> float. This must hold for+ required AND optional (non-required, hence Optional[...]-annotated) target fields, both at the dry-run+ runnable gate and in live execution.+ """++ async def test_validate_bundle_dry_run_passes(self) -> None:+ """The whole-stuff construct bundle must pass the runnable gate: dry-run success on every pipe."""+ result = await validate_bundle(mthds_file_path=_BUNDLE)+ assert result.pending_signatures == []+ for pipe_ref, dry_run_output in result.dry_run_result.items():+ assert dry_run_output.status is DryRunStatus.SUCCESS, f"dry-run failed for '{pipe_ref}': {dry_run_output.error_message}"++ @pytest.mark.parametrize(+ ("pipe_code", "inputs", "field_name", "expected_value"),+ [+ pytest.param(+ "fill_required_text",+ {"text_input": TextContent(text="Alpha title")},+ "title",+ "Alpha title",+ id="whole_text_to_required_text_field",+ ),+ pytest.param(+ "fill_optional_text",+ {"text_input": TextContent(text="Sorry, not this time")},+ "note",+ "Sorry, not this time",+ id="whole_text_to_optional_text_field",+ ),+ pytest.param(+ "fill_required_text_list",+ {"texts_input": [TextContent(text="alpha"), TextContent(text="bravo"), TextContent(text="charlie")]},+ "tags",+ ["alpha", "bravo", "charlie"],+ id="whole_text_list_to_required_list_field",+ ),+ pytest.param(+ "fill_optional_text_list",+ {"texts_input": [TextContent(text="alpha"), TextContent(text="bravo"), TextContent(text="charlie")]},+ "tags",+ ["alpha", "bravo", "charlie"],+ id="whole_text_list_to_optional_list_field",+ ),+ pytest.param(+ "fill_required_number",+ {"number_input": NumberContent(number=42.5)},+ "score",+ 42.5,+ id="whole_number_to_required_number_field",+ ),+ ],+ )+ async def test_live_compose_whole_stuff_into_native_field(+ self,+ pipe_code: str,+ inputs: dict[str, Any],+ field_name: str,+ expected_value: Any,+ ) -> None:+ """Live-run each compose pipe with real inputs and check the composed native field value."""+ runner = PipelexMTHDSProtocol(+ library_dirs=[str(_FIXTURE_DIR)],+ pipe_run_mode=PipeRunMode.LIVE,+ )+ response = await runner.execute(pipe_code=pipe_code, inputs=inputs)+ main_stuff = response.pipe_output.main_stuff+ assert main_stuff is not None+ pretty_print(main_stuff.content, title=f"Composed output of '{pipe_code}'")++ composed_value = getattr(main_stuff.content, field_name)+ assert composed_value == expected_value+ assert type(composed_value) is type(expected_value)diff --git a/tests/e2e/test_signature_validation_mthds.py b/tests/e2e/test_signature_validation_mthds.pyindex af01d8f9d..698470e18 100644--- a/tests/e2e/test_signature_validation_mthds.py+++ b/tests/e2e/test_signature_validation_mthds.py@@ -19,7 +19,7 @@ _MULTIPLICITY = _FIXTURE_DIR / "multi_input_multiplicity.mthds"_STRUCTURED = _FIXTURE_DIR / "signature_with_structured_output.mthds"[email protected](loop_scope="class")class TestSignatureValidationE2E:async def test_signature_only_bundle_strict_reports_pending(self) -> None:# Signatures are never an error (D-B): strict validation does not raise — it reports thediff --git a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/compose_structured_models.mthds b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/compose_structured_models.mthdsindex fc219bc0a..51585956b 100644--- a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/compose_structured_models.mthds+++ b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/compose_structured_models.mthds@@ -51,3 +51,12 @@ DataPayload = "Container for JSONContent"MixedMediaReport = "Report with multiple StuffContent types"ImageListGallery = "Gallery with a list of ImageContent"DocumentBundle = "Bundle with a list of DocumentContent"++# Native scalar conversion testing concepts (whole-stuff copy into native fields)+NoteHolder = "Holder with an optional str field"+TagsHolder = "Holder with a required list of str field"+OptionalTagsHolder = "Holder with an optional list of str field"+NullableTagsHolder = "Holder with a list of nullable str items"+ScoreHolder = "Holder with a float field"+ApprovalHolder = "Holder with a bool field"+DeadlineHolder = "Holder with a date field"diff --git a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/models_for_pipe_compose.py b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/models_for_pipe_compose.pyindex f692bf9fa..a2bcff43f 100644--- a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/models_for_pipe_compose.py+++ b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/models_for_pipe_compose.py@@ -1,5 +1,7 @@"""Test StructuredContent models for PipeCompose construct testing."""+from datetime import date+from pydantic import Fieldfrom pipelex.core.stuffs.document_content import DocumentContent@@ -329,3 +331,53 @@ class DocumentBundle(StructuredContent):bundle_name: str = Field(description="Bundle name")documents: list[DocumentContent] = Field(description="List of documents")+++# ============================================================================+# Models for native scalar conversion testing+# These test whole-stuff copies into native-typed fields: the content wrapper+# must be unwrapped to the native value (TextContent -> str, NumberContent -> float,+# YesNoContent -> bool, DateContent -> date), including into Optional fields.+# ============================================================================+++class NoteHolder(StructuredContent):+ """Holder with an optional str field - should receive TextContent.text extracted."""++ note: str | None = Field(default=None, description="The note, optional")+++class TagsHolder(StructuredContent):+ """Holder with a required list[str] field - should receive item texts extracted."""++ tags: list[str] = Field(description="The tags")+++class OptionalTagsHolder(StructuredContent):+ """Holder with an optional list[str] field - should receive item texts extracted."""++ tags: list[str] | None = Field(default=None, description="The tags, optional")+++class NullableTagsHolder(StructuredContent):+ """Holder with a list of nullable str items - should receive item texts extracted."""++ tags: list[str | None] = Field(description="The tags, each possibly null")+++class ScoreHolder(StructuredContent):+ """Holder with a float field - should receive NumberContent.number extracted."""++ score: float = Field(description="The score")+++class ApprovalHolder(StructuredContent):+ """Holder with a bool field - should receive YesNoContent.yes_no extracted."""++ approved: bool = Field(description="Whether approved")+++class DeadlineHolder(StructuredContent):+ """Holder with a date field - should receive DateContent.date extracted."""++ deadline: date = Field(description="The deadline date")diff --git a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_data.py b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_data.pyindex 188f34b37..e3100477d 100644--- a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_data.py+++ b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_data.py@@ -69,6 +69,35 @@ class ContentConversionTestData:}+class NativeScalarConversionTestData:+ """Test data for native scalar conversion tests (whole-stuff copy into native fields)."""++ # Test case: TextContent to Optional[str] field+ TEXT_TO_OPTIONAL_STR_CONSTRUCT: ClassVar[dict[str, Any]] = {+ "note": {"from": "note_text"},+ }++ # Test case: ListContent[TextContent] to list[str] field+ TEXT_LIST_TO_STR_LIST_CONSTRUCT: ClassVar[dict[str, Any]] = {+ "tags": {"from": "tag_texts"},+ }++ # Test case: NumberContent to float field+ NUMBER_TO_FLOAT_CONSTRUCT: ClassVar[dict[str, Any]] = {+ "score": {"from": "score_number"},+ }++ # Test case: YesNoContent to bool field+ YES_NO_TO_BOOL_CONSTRUCT: ClassVar[dict[str, Any]] = {+ "approved": {"from": "approval_flag"},+ }++ # Test case: DateContent to date field+ DATE_TO_DATE_CONSTRUCT: ClassVar[dict[str, Any]] = {+ "deadline": {"from": "deadline_date"},+ }++class ClassCompatibilityTestData:"""Test data for class compatibility tests."""diff --git a/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_pipe_compose_native_scalar_conversions.py b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_pipe_compose_native_scalar_conversions.pynew file mode 100644index 000000000..8f9dbe61e--- /dev/null+++ b/tests/integration/pipelex/pipes/operator/pipe_compose_structured/test_pipe_compose_native_scalar_conversions.py@@ -0,0 +1,244 @@+"""Integration tests for PipeCompose native scalar conversions.++These tests verify that a whole-stuff `{ from = "..." }` construct source unwraps native+content wrappers into native-typed target fields:+- TextContent to Optional[str] field (Optional unwrap)+- ListContent[TextContent] to list[str] and Optional[list[str]] fields (item scalar extraction)+- NumberContent to float field+- YesNoContent to bool field (including a falsy value)+- DateContent to date field+"""++import datetime+from pathlib import Path+from typing import Any, Callable, cast++import pytest++from pipelex import pretty_print+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.date_content import DateContent+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.number_content import NumberContent+from pipelex.core.stuffs.stuff_content import StuffContent+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_content import TextContent+from pipelex.core.stuffs.yes_no_content import YesNoContent+from pipelex.hub import get_native_concept, get_pipe_router+from pipelex.pipe_operators.compose.exceptions import PipeComposeError+from pipelex.pipe_operators.compose.pipe_compose import PipeCompose+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.pipeline.job_metadata import JobMetadata+from tests.integration.pipelex.pipes.operator.pipe_compose_structured.test_data import NativeScalarConversionTestData+++def _make_tag_texts() -> ListContent[TextContent]:+ return ListContent[TextContent](items=[TextContent(text="alpha"), TextContent(text="bravo"), TextContent(text="charlie")])++[email protected]_runnable[email protected](loop_scope="class")+class TestPipeComposeNativeScalarConversions:+ """Integration tests for native scalar conversions in PipeCompose construct mode."""++ @pytest.fixture+ def test_library_path(self) -> list[Path]:+ """Path to the test library for these tests."""+ return [Path("tests/integration/pipelex/pipes/operator/pipe_compose_structured")]++ @pytest.mark.parametrize(+ ("pipe_code", "input_name", "input_concept_code", "input_content", "construct", "output_concept", "field_name", "expected_value"),+ [+ pytest.param(+ "compose_text_to_optional_str",+ "note_text",+ NativeConceptCode.TEXT,+ TextContent(text="A short note"),+ NativeScalarConversionTestData.TEXT_TO_OPTIONAL_STR_CONSTRUCT,+ "NoteHolder",+ "note",+ "A short note",+ id="text_content_to_optional_str_field",+ ),+ pytest.param(+ "compose_text_list_to_str_list",+ "tag_texts",+ NativeConceptCode.TEXT,+ _make_tag_texts(),+ NativeScalarConversionTestData.TEXT_LIST_TO_STR_LIST_CONSTRUCT,+ "TagsHolder",+ "tags",+ ["alpha", "bravo", "charlie"],+ id="text_list_to_required_str_list_field",+ ),+ pytest.param(+ "compose_text_list_to_optional_str_list",+ "tag_texts",+ NativeConceptCode.TEXT,+ _make_tag_texts(),+ NativeScalarConversionTestData.TEXT_LIST_TO_STR_LIST_CONSTRUCT,+ "OptionalTagsHolder",+ "tags",+ ["alpha", "bravo", "charlie"],+ id="text_list_to_optional_str_list_field",+ ),+ pytest.param(+ "compose_text_list_to_nullable_str_list",+ "tag_texts",+ NativeConceptCode.TEXT,+ _make_tag_texts(),+ NativeScalarConversionTestData.TEXT_LIST_TO_STR_LIST_CONSTRUCT,+ "NullableTagsHolder",+ "tags",+ ["alpha", "bravo", "charlie"],+ id="text_list_to_nullable_item_str_list_field",+ ),+ pytest.param(+ "compose_number_to_float",+ "score_number",+ NativeConceptCode.NUMBER,+ NumberContent(number=42.5),+ NativeScalarConversionTestData.NUMBER_TO_FLOAT_CONSTRUCT,+ "ScoreHolder",+ "score",+ 42.5,+ id="number_content_to_float_field",+ ),+ pytest.param(+ "compose_yes_no_to_bool",+ "approval_flag",+ NativeConceptCode.YES_NO,+ YesNoContent(yes_no=False),+ NativeScalarConversionTestData.YES_NO_TO_BOOL_CONSTRUCT,+ "ApprovalHolder",+ "approved",+ False,+ id="yes_no_content_to_bool_field_falsy",+ ),+ pytest.param(+ "compose_date_to_date",+ "deadline_date",+ NativeConceptCode.DATE,+ DateContent(date=datetime.date(2026, 3, 14)),+ NativeScalarConversionTestData.DATE_TO_DATE_CONSTRUCT,+ "DeadlineHolder",+ "deadline",+ datetime.date(2026, 3, 14),+ id="date_content_to_date_field",+ ),+ ],+ )+ async def test_compose_native_scalar_conversion(+ self,+ job_metadata: JobMetadata,+ pipe_run_mode: PipeRunMode,+ load_test_library: Callable[[list[Path]], None],+ test_library_path: list[Path],+ pipe_code: str,+ input_name: str,+ input_concept_code: NativeConceptCode,+ input_content: StuffContent,+ construct: dict[str, Any],+ output_concept: str,+ field_name: str,+ expected_value: Any,+ ):+ """A whole native stuff copied via construct must land in the target field as the native value with the exact native type."""+ load_test_library(test_library_path)++ input_stuff = StuffFactory.make_stuff(+ concept=get_native_concept(input_concept_code),+ content=input_content,+ name=input_name,+ )+ working_memory = WorkingMemory()+ working_memory.add_new_stuff(name=input_name, stuff=input_stuff)++ pipe_compose_blueprint = PipeComposeBlueprint.model_validate(+ {+ "description": f"Compose {output_concept} from a whole {input_concept_code} stuff",+ "inputs": {input_name: input_concept_code},+ "construct": construct,+ "output": f"compose_structured_test.{output_concept}",+ }+ )++ pipe = PipeFactory[PipeCompose].make_from_blueprint(+ domain_code="compose_structured_test",+ pipe_code=pipe_code,+ blueprint=pipe_compose_blueprint,+ )++ pipe_job = PipeJobFactory.make_pipe_job(+ pipe=pipe,+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),+ )+ pipe_output = await get_pipe_router().run(pipe_job=pipe_job)++ main_stuff = pipe_output.main_stuff+ assert main_stuff is not None+ assert type(main_stuff.content).__name__ == output_concept++ composed_value = getattr(main_stuff.content, field_name)+ assert composed_value == expected_value+ assert type(composed_value) is type(expected_value)+ if isinstance(expected_value, list):+ expected_items = cast("list[Any]", expected_value)+ for composed_item, expected_item in zip(composed_value, expected_items, strict=True):+ assert type(composed_item) is type(expected_item)++ pretty_print(main_stuff.content, title=f"{output_concept} - native scalar conversion")++ async def test_compose_timestamped_date_to_date_field_raises(+ self,+ job_metadata: JobMetadata,+ pipe_run_mode: PipeRunMode,+ load_test_library: Callable[[list[Path]], None],+ test_library_path: list[Path],+ ):+ """A Date carrying a time of day must be rejected by a bare `date` target field instead of silently dropping the time."""+ load_test_library(test_library_path)++ timestamped_date = DateContent(+ date=datetime.date(2026, 3, 14),+ time=datetime.time(15, 40, tzinfo=datetime.timezone(datetime.timedelta(hours=2))),+ )+ input_stuff = StuffFactory.make_stuff(+ concept=get_native_concept(NativeConceptCode.DATE),+ content=timestamped_date,+ name="deadline_date",+ )+ working_memory = WorkingMemory()+ working_memory.add_new_stuff(name="deadline_date", stuff=input_stuff)++ pipe_compose_blueprint = PipeComposeBlueprint.model_validate(+ {+ "description": "Compose DeadlineHolder from a whole timestamped Date stuff",+ "inputs": {"deadline_date": NativeConceptCode.DATE},+ "construct": NativeScalarConversionTestData.DATE_TO_DATE_CONSTRUCT,+ "output": "compose_structured_test.DeadlineHolder",+ }+ )++ pipe = PipeFactory[PipeCompose].make_from_blueprint(+ domain_code="compose_structured_test",+ pipe_code="compose_timestamped_date_to_date",+ blueprint=pipe_compose_blueprint,+ )++ pipe_job = PipeJobFactory.make_pipe_job(+ pipe=pipe,+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),+ )++ with pytest.raises(PipeComposeError, match="time of day"):+ await get_pipe_router().run(pipe_job=pipe_job)diff --git a/tests/unit/pipelex/tools/test_annotation_utils.py b/tests/unit/pipelex/tools/test_annotation_utils.pynew file mode 100644index 000000000..2a2656e0a--- /dev/null+++ b/tests/unit/pipelex/tools/test_annotation_utils.py@@ -0,0 +1,43 @@+from typing import Any, Optional, Union++import pytest++from pipelex.core.stuffs.text_content import TextContent+from pipelex.tools.typing.annotation_utils import unwrap_optional+++class TestUnwrapOptional:+ """unwrap_optional must unwrap the Optional shape (single non-None arm) in both the+ typing.Union and PEP 604 spellings, and leave every other annotation unchanged.+ """++ @pytest.mark.parametrize(+ ("annotation", "expected"),+ [+ pytest.param(Optional[str], str, id="typing_optional_str"), # noqa: UP045 — the typing.Optional spelling is the test subject+ pytest.param(Union[str, None], str, id="typing_union_str_none"),+ pytest.param(str | None, str, id="pep604_str_none"),+ pytest.param(Optional[list[str]], list[str], id="typing_optional_list_str"), # noqa: UP045 — the typing.Optional spelling is the test subject+ pytest.param(list[str] | None, list[str], id="pep604_list_str_none"),+ pytest.param(Optional[TextContent], TextContent, id="typing_optional_text_content"), # noqa: UP045 — the typing.Optional spelling is the test subject+ pytest.param(TextContent | None, TextContent, id="pep604_text_content_none"),+ ],+ )+ def test_unwraps_optional_shape(self, annotation: Any, expected: Any):+ assert unwrap_optional(annotation) == expected++ @pytest.mark.parametrize(+ "annotation",+ [+ pytest.param(str, id="bare_class"),+ pytest.param(list[str], id="generic_list"),+ pytest.param(dict[str, int], id="generic_dict"),+ pytest.param(Union[str, int], id="typing_union_multi_arm"),+ pytest.param(str | int, id="pep604_union_multi_arm"),+ pytest.param(Union[str, int, None], id="typing_union_multi_arm_with_none"),+ pytest.param(str | int | None, id="pep604_union_multi_arm_with_none"),+ pytest.param(None, id="none_annotation"),+ ],+ )+ def test_leaves_non_optional_shapes_unchanged(self, annotation: Any):+ assert unwrap_optional(annotation) == annotationdiff --git a/wip/pr-1051-review-notes.md b/wip/pr-1051-review-notes.mdnew file mode 100644index 000000000..dcc1db514--- /dev/null+++ b/wip/pr-1051-review-notes.md@@ -0,0 +1,11 @@+# PR #1051 review notes — deferred items++Triage record for the SWE-agent review threads on [PR #1051](https://github.com/Pipelex/pipelex/pull/1051) (PipeCompose whole-stuff copies into native fields). Confirmed issues were fixed on the branch; this file captures the one item deliberately deferred.++## Defensive guard for genuine-union nested fields (greptile, deferred)++- **Reporter:** greptile (P1), review thread `PRRT_kwDOOwmMFc6RvbW7` on [PR #1051](https://github.com/Pipelex/pipelex/pull/1051), file `pipelex/pipe_operators/compose/structured_content_composer.py` (`_get_nested_field_class`).+- **Verdict:** the reported regression is a false positive — pre-PR code also passed genuine unions (`Address | Contact`, no `None` arm) through unchanged, since its unwrap was gated on `type(None) in args`. The only behavior change is for `X | Y | None`: pre-PR silently picked the first arm (worse), post-PR returns the union unchanged (honest passthrough, locked in by `tests/unit/pipelex/tools/test_annotation_utils.py`).+- **The real (pre-existing, latent) gap:** a hand-authored `StructuredContent` subclass with a genuine multi-arm union nested field, targeted by a NESTED construct, would flow the union into the nested composer's `output_class` and die downstream with a cryptic `AttributeError` on `model_fields` / `model_validate`. Unreachable through the supported authoring path: `StructureGenerator` has no UNION resolved-type kind, so generated models cannot carry genuine union fields.+- **Why deferred:** pre-existing, unreachable from generated structures, and out of scope for the PR. Whether to support hand-authored union fields at all is a design decision.+- **Recommendation if addressed:** raise `StructuredContentComposerTypeError` in `_get_nested_field_class` when the post-`unwrap_optional` annotation is still a union (`get_origin(...) in (Union, types.UnionType)`), naming the field, class, and union arms. Never resolve an arbitrary arm — silent first-arm picking is the anti-pattern this PR removed. Same reasoning applies to `_get_field_expected_type`, though a union there degrades more gracefully (falls through to keep-object).