Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Optionals phase 1: presence markers `?`/`!`, absence ledger, and static optionality validation (#1021)

Louis Choquel8b4c60a7cea6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d9dc73c23..113168b28 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,12 +8,32 @@
- **Secrets provider is now a plugin seam:** The secrets backend (`env`) is selected at boot from a `SecretsProviderRegistry` keyed by the new open `secrets_config.method` token (default `"env"`), populated by the always-on built-in `SecretsPlugin`. A third party can now ship a `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, …) that advertises itself under the `pipelex.plugins` entry-point group and is selected via `secrets_config.method` — the same mechanism the storage seam uses. Out-of-the-box behavior is unchanged: `env` stays the default. An unknown method fails loudly at boot with `UnknownSecretsMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/secrets-provider-plugins.md`.
- **TOML pipeline inputs:** The `--inputs` file passed to `pipelex run` and `pipelex-agent run` (pipe/bundle/method) can now be TOML in addition to JSON, discriminated by file extension — a `.toml` suffix is parsed as TOML, every other value stays JSON. Both formats produce the same input dictionary; TOML's multi-line strings make text-heavy inputs much easier to author. `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json` when `--inputs` is omitted, erroring if both exist. Inline JSON (`{...}`) and agent-CLI stdin inputs stay JSON-only. A bare TOML datetime/date/time literal is rejected with an explicit error (quote it as a string) — native datetime concept support is a separate track.
- **`--format toml` for inputs-template generation:** `pipelex build inputs` and `pipelex-agent inputs` (pipe/bundle/method) accept `--format json|toml` (default `json`) to emit the generated inputs template as TOML. On the main CLI, `toml` defaults the output filename to `inputs.toml`; on the agent CLI, `toml` prints the raw TOML template to stdout (the no-inputs case prints a TOML comment line that loads back as an empty dict).
+- **Absent results are first-class on the wire and every surface (optionals Step E):** A run whose declared `?` output resolves absent is a success everywhere. `main_stuff_name` on `PipelexPipeRunOutput` and the `/execute` response now names the declared output slot even when it resolved absent — consumers branch on the absence record carried in the serialized working memory (`absences`), never on transport. The absence ledger round-trips cross-process (`hydrate_working_memory` reconstructs it, so a recorded absence no longer degrades to a hard miss after transit). The CLI run surfaces render an explicit absence document instead of crashing: `pipelex run` prints the absence and `--save-main-stuff` writes the `"absent": true` artifact (json + md, no viewer), the agent CLI's `run` returns the absence document as its result payload, and OTel/Langfuse output capture serializes the absence for both inputs and outputs.
+- **IO contracts carry optionality (`optional` on `pipe_io_contracts`):** `PipeInputContract` and `PipeOutputContract` gain an `optional: bool` — `true` for `?`-declared inputs (may be omitted; the pipe handles absence) and `?`-declared outputs (may resolve absent).
+- **Advisory `warnings` on the validation report + the useless-`!` lint:** The validation report gains a general `warnings` array — same item shape as `validation_errors`, but advisory: it never flips `is_valid`. Its first occupant is `optional_force_redundant`: a `!` (force) input whose slot is guaranteed present in every analyzed flow — the assertion can never fire, so the marker is dead weight. The lint aggregates across flows before warning (a `!` that is meaningful in one flow is never flagged because another flow guarantees the slot). Every whole-bundle validate surface carries it: the protocol report, the agent-CLI validate envelopes (rendered as a "Warnings" section in markdown), the human `pipelex validate` (yellow advisory lines — never the exit code), and the builder validate operations.
+- **Execution graphs show skips and optional data edges:** `GraphSpec` nodes gain the `skipped` status with a `skip_reason` (a lifted pipe ends its node as skipped — "why did my workflow produce nothing?" is now answerable from the graph), and data edges fed by a declared-optional (`?`) output carry `optional: true`. A skipped pipe with a plural output still registers the real empty list it wrote, so downstream data edges resolve like any produced value. Both flow through the in-process tracer and the distributed event-replay assembler (new `pipe_end_skipped` trace event).
+- **Static optionality validation — the absence-taint pass (breaking):** Validation now proves the absence safety theorem at author time: every maybe-absent slot must reach an explicit sink. A taint pass over each controller's dataflow computes per-slot presence (`guaranteed` / `maybe-absent`) and rejects escapes with new typed validation errors: `optional_not_handled` (a maybe-absent slot ends a sequence — or feeds a condition through a mapped `?`-output pipe — without the controller declaring a `?` output; the message names the absence origin, the propagation path, and the three fixes), `optional_output_required` (**breaking**: a `continue`-reachable `PipeCondition` must declare its output optional, e.g. `output = "Constraint?"` — `continue` resolves the output absent, so a plain output would hide the no-output path), and `optional_branch_required_field` (a required structure field of a `PipeParallel` output fed by a maybe-absent branch — make the field non-required or sink the absence upstream). Plural slots are never tainted: a lifted plural output is the empty list and a batched step compacts, so taint stops at `[]`.
+- **Template guard-lint for optional inputs (`optional_input_unguarded`):** Every template reference to a declared-optional (`?`) input must be guarded — `@?var`, a `{% if var %}...{% endif %}` block, or an inline presence conditional (`... if var is defined else ...`). An unguarded reference (including a plain `@var` sigil on an optional input, a deep `{{ var.attr }}`, or use in an `{% else %}` arm) fails validation with the precise fix. Applies to `PipeLLM` prompts and system prompts, `PipeCompose` templates, and `PipeCondition` expressions. In the same motion, `@?` finally means what it says: a variable declared optional on a pipe is no longer presence-required by the controller miss-gates — the pipe runs and its guarded templates take the absent arm.
+- **Liftable-pipe inventory on the valid report (`liftable_pipes`):** The validation report now lists every pipe that may be skipped (lifted) when an optional slot resolves absent — the liftable pipe, the controller it lifts within, the trigger slots, and the absence source. This is the build-time visibility that makes implicit lifting acceptable: authors see "may be skipped when X is absent" at `/validate` time, not just in the run report.
+- **Absence at runtime — the ledger and the trichotomy (`?` / `!` become live):** Working memory gains an absence ledger: an `AbsenceRecord` (variable, producing pipe, kind = declared-absent / skipped / not-provided, reason, upstream chain) records *why* a slot holds no value, with provenance captured at the moment absence is produced. At run time, a pipe whose plain input is fed a recorded absence is **skipped** (implicit lifting): its own output is recorded absent, chaining back to the origin — write `?` once where absence enters and the chain short-circuits like Swift's `a?.b.c`. A `?` input absorbs absence (the pipe runs and handles both arms); a `!` input fed absence raises the new typed `OptionalValueAbsentError`, whose report names the variable, the consuming pipe, the producing pipe, and the original reason. A skipped pipe with a plural output normalizes to an empty list (with a ledger note) instead of an absent slot — `[]`-emptiness stays the absence story for plurals. Optional **method** inputs may now be omitted from the pipeline inputs: the slot starts as a recorded not-provided absence instead of raising a missing-inputs error, and the missing-required-inputs message now names the optional inputs a caller may omit (it also no longer claims "Dry run" on live runs). Post-run reads get a tri-state resolved accessor (`WorkingMemory.resolve_stuff` / `resolve_main_stuff`: a value or a recorded absence); the run's working memory enumerates all recorded absences. A slot with neither a value nor a record remains a hard error — never-produced is a bug, not an absence. Controller boundaries stay authoritative: a sequence declaring `X?` over a step that needs `X` plain runs and lets its steps lift (the declared-vs-needed input check no longer compares presence markers).
+- **Presence markers `?` and `!` on pipe inputs/outputs — grammar and static carriers:** The io-ref grammar gains presence markers after the multiplicity suffix: `inputs = { clause = "PenaltyClause?" }` (optional input), `clause = "PenaltyClause!"` (force), `output = "PenaltyClause?"` (optional output). Markers parse, are carried through blueprints, builder specs, and IO spec carriers, round-trip through bundle representations, and signature-vs-definition contract matching compares them. Grammar misuse is rejected at blueprint parse with the new `optional_marker_invalid` validation error type: markers never combine with multiplicity (`X[]?` is invalid — an absent plural is just the empty list), `!` is inputs-only, and markers stay illegal on concept definitions, `refines`, structure-field refs, and package refs. No runtime behavior change yet: absence semantics (skip/absorb/force at run time, taint validation, wire surfaces) land in the following steps of the optionals feature.
### Changed
- **Breaking: dropped Python 3.10 support — `requires-python` is now `>=3.11,<3.15`.** The 3.10 compatibility bridges are gone: `StrEnum`, `Self`, and `Traversable` are now imported directly from the stdlib (`enum` / `typing` / `importlib.resources.abc`) at every call site, the `pipelex.types` re-export module that existed only to paper over 3.10 is deleted, and the `backports.strenum` conditional dependency (and its type stub) is dropped. CI test/lint matrices, the `package-check` `requires-python` floor gate, and the contributor docs now start at 3.11. Install on Python 3.11 or newer.
- **Plugin API version bumped to 3 (Breaking):** `PLUGIN_API_VERSION` is now `3` (was `2`) to add the `add_storage_provider` and `add_secrets_provider` registrar menu methods. Plugin discovery version-checks with strict equality, so every external plugin must re-declare `targets_api = 3`. `pipelex-mistralai-workflows` registers no storage or secrets provider and imports none of the removed symbols, so the `targets_api` bump is its only change. `pipelex-temporal` needs the `targets_api` bump **and** a code migration: its payload-codec factory imports the now-removed `make_storage_provider_from_config` (see Removed) and must switch to resolving the provider via the storage-provider registry (`get_storage_provider_registry().get_required(method=...)`).
- **`PipeSignature` is not a pipe type (Breaking):** A signature is now declared by **omitting** `type` — a `[pipe.x]` section with no `type` and nothing but the contract (`description`, `output`, optional `inputs`, optional `signature_for`) *is* a `PipeSignature`. Writing `type = "PipeSignature"` explicitly is rejected with a migration error ("`PipeSignature` is no longer a pipe type — delete the `type` line"), and a typeless section that declares any non-contract field is a hard error naming the field. The `pipelex-agent pipe` authoring command mirrors this: a typeless JSON spec (no `type`) renders a `[pipe.x]` section with no type line, and `--type PipeSignature` is rejected. **MTHDS JSON Schema shape change:** the signature arm no longer carries a `type` property (an explicit tag now fails schema validation), so the downstream schema copies (`mthds`, `vscode-pipelex`, `mthds-ui`) must be re-synced on the next release via the `mthds-schema-sync` skill.
+- **`PipeCondition` `continue` now resolves the output as absent (breaking):** The `continue` special outcome no longer passes the current main stuff through (and no longer errors when there is none). It records a declared-absent `AbsenceRecord` for the condition's declared output — with the evaluated expression as the reason — and returns success, memory otherwise unchanged. Dry-run parity: an all-special-outcomes condition records the same absence instead of raising when `continue` is reachable; a fail-only condition (every outcome and the default map to `fail`) still raises in dry run, since every live path can only fail. Migration: the previous value stays in the working memory under its own name, so downstream pipes consume it explicitly by that name (declared `?` when it may be absent); a coalescing operator is the planned ergonomic replacement for pass-through.
+- **Controllers combine under absence:** `PipeParallel` no longer crashes when a branch result resolves absent (lifted branch or `continue`): with a `Composite` output the absent component is omitted (ledger note kept for observability); with a structured output a non-required field absorbs the absence as its field default (`None` unless the author declared another default), while a required field fed an absent branch raises a typed error naming the branch, the field, and the fixes. `PipeBatch` compacts: absent branch results are dropped from the aggregated list, so batching a "keep or skip" condition over items yields only the kept results.
+- **Result delivery of an absent output:** An absent main output is a first-class success at the delivery boundary — the `main_stuff.json/md/html` artifact files become an explicit absence artifact (JSON carries `"absent": true` plus the absence record with its provenance chain) on both the typed and the raw (cross-process) delivery paths. A working memory with neither a value nor a recorded absence still fails delivery loudly.
+
+### Fixed
+
+- **Template truth-tests on singular values no longer crash:** `{% if var %}` (and `@?var`, which expands to it) on a present singular value used to raise `TypeError: '...' content does not support len().` — Jinja2's truth test fell through to the artefact's list-only `__len__`. `StuffArtefact` now defines `__bool__`: a present non-list artefact is truthy, a list artefact follows list emptiness (an empty list is falsy). This makes the optionals guard idiom safe on both arms.
+
+### Documentation
+
+- **Optionality guide:** New "Understanding Optionality" page (`docs/building-methods/pipes/understanding-optionality.md`) beside the multiplicity guide — presence markers, the runtime trichotomy (skip / run / fail), absence records and provenance, template guards, controllers under absence, and the static safety net. PipeBatch documents compaction under absence, PipeLLM documents the `@?` optional block sigil, and the run CLI page documents the absence artifact an absent main output produces.
### Removed
diff --git a/TODOS.md b/TODOS.md
deleted file mode 100644
index 9d20e33e2..000000000
--- a/TODOS.md
+++ /dev/null
@@ -1,234 +0,0 @@
-# Master plan — Provider plugins: storage & secrets
-
-Status: **TRACK COMPLETE. All 5 phases DONE + all 5 Checkpoints CLEARED.** Both provider verticals (storage + secrets) shipped as config-selected plugin seams; Phase 5 (release gating / cross-repo bookkeeping) done and Checkpoint 5 (final whole-branch clean-room review) cleared with its triage applied. Branch: `feature/More-plugins-2` (worktree `/Users/lchoquel/repos/Pipelex/_plugins`). **Nothing pushed** (no upstream) — the PR to the intended base is Louis's call. This file is the **conductor**; the granular per-seam procedures live in linked docs and should not be duplicated here.
-
-Commits on the branch (newest substantive first; run `git log` for the exact tip — later `docs(plugins)` bookkeeping commits may sit on top):
-
-- `0bbc39474` — fix: Checkpoint-5 whole-branch review triage (**Phase 5 / Checkpoint 5 — the review-triage commit**)
-- `65603e912` — test: secrets-provider seam tests + docs (**Phase 4 — the review target**)
-- `fdf3e114b` — fix: Phase-3 review triage (kit-config sync + stale boot comment)
-- `8a0b32668` — feat: secrets provider → config-selected plugin seam (**Phase 3 code — the review target**)
-- `3940eb7a2` — docs: Phase 2 checkpoint bookkeeping — storage vertical done
-- `37a3b72f1` — fix: Phase-2 review triage (doc tense + stronger gcp secret-wiring assert)
-- `f08193e81` — test: storage-provider seam tests + docs (**Phase 2**)
-- `977c73811` — docs: this plan + Phase 1 checkpoint bookkeeping
-- `58961848a` — fix: contract.py v3 comment (Checkpoint-1 review triage)
-- `8268ff08f` — feat: storage provider → config-selected plugin seam (**Phase 1 code**)
-- `04434f785` — branch base (release v0.37.0 merge). Whole-branch diff for the final Checkpoint 5 = `git diff 04434f785..HEAD`.
-
-**Cold-start (resume here):** **The whole track is COMPLETE — nothing left to build.** Both provider verticals shipped (storage Phases 1–2, secrets Phases 3–4) and Phase 5 (release gating) + Checkpoint 5 (final whole-branch clean-room review + triage) are done. All gates green on the final tip: `agent-check`, `tb`, `check-config-sync`, full `agent-test`. The **only remaining action is Louis's**: open the PR `feature/More-plugins-2` → intended base, and (when the pipelex version carrying this lands) execute the cross-repo obligations recorded in the CHANGELOG — `pipelex-mistralai-workflows` bumps `targets_api` to 3 (bump-only); `pipelex-temporal` bumps `targets_api` to 3 **and** migrates its payload-codec factory off the removed `make_storage_provider_from_config` to `get_storage_provider_registry().get_required(...)`. Deferred design follow-ups (D3 external config passthrough incl. the external-method `uri_format` gap, first real external provider, cookbook example) are captured in `wip/plugins/`. `PLUGIN_API_VERSION` is 3.
-
-> ⚠️ **Phase-4 heads-up (learned in Phase 2):** the s3/gcp "MissingDependencyError when *selected*" phrasing in the plan is imprecise — the SDK guard is deferred to *use*, not *selection*. Expect the secrets detail doc to carry the same imprecision for SDK-backed secrets providers (Vault/AWS) and treat it the same way (pin *import-light registration*, not a select-time raise). See "Phase 2 — as-built".
-
-**Environment notes for the next session (avoid rediscovering these):**
-
-- **Targeted tests for the storage seam:** `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/plugins/ tests/unit/pipelex/tools/storage/ tests/integration/pipelex/tools/storage/` (Phase 2 also needs `tests/integration/pipelex/plugins/` for the external-plugin harness). But Checkpoint 2 requires the **full** `make agent-test`.
-- **This machine's global `~/.pipelex/pipelex.toml` sets `storage_config.method = "s3"`**, so a plain `Pipelex.make(needs_inference=False)` boot smoke test yields an `S3StorageProvider`, *not* `local` (base + repo `.pipelex/` both say `local`; the global override wins). To smoke a specific method faithfully, pass `config_overrides={"pipelex": {"storage_config": {"method": "local"}}}` in a fresh interpreter. This is a machine/global-config condition, **not** a bug in the seam — don't chase it.
-- **Empirically confirmed** (so Phase 2 boot tests can rely on it): `StorageProviderConfig.method: str = Field(strict=False)` coerces a `StrEnum` input to a plain `str`, accepts an external token (e.g. `"azure"`) at parse, and `StrEnum` registry keys resolve against a plain-str `config.method` in dict lookup.
-
-> Replaces the retired cookbook hello-plugin tracker (complete; recoverable in git history).
-
-## What & why (cold-start summary)
-
-Promote the two dependency-injection seams that pass all plugin criteria — **storage provider** and **secrets provider** — to formal `pipelex.plugins` entry-point plugins, so third parties can ship `pipelex-storage-<backend>` / `pipelex-secrets-<backend>` packages selected at deploy time. Chosen after a full boot-audit of DI seams; everything else is already a plugin, a boot-orchestrator-owned hub slot, or a population/config point that should stay one.
-
-**Read these three before starting** (they hold the detailed steps this master plan sequences):
-
-- `wip/plugins/README.md` — the track's shared decisions + boot-audit rationale + the new mechanism.
-- `wip/plugins/storage-provider-plugin.md` — storage detail (defines the shared mechanism). **Lands first.**
-- `wip/plugins/secrets-provider-plugin.md` — secrets detail (reuses the mechanism; **Decision W already RESOLVED = W-A**).
-
-**The new mechanism (both seams share it):** a *keyed registry + config-selected singleton*. Plugins register N provider factories keyed by an open method token (mirroring `OrchestrationMode`); at boot, core reads one config field (`storage_config.method` / `secrets_config.method`), looks the token up in the registry, and calls the factory to produce the one provider set on the hub. Deliberately **not** a `HubSlot` (those are orchestrator-coupled).
-
-## Locked decisions (do not re-litigate)
-
-- **DX-1 — one API bump.** `PLUGIN_API_VERSION` 2→3, done once on this branch. Both seams land before any release, so external plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`) re-declare `targets_api=3` exactly once. Discovery uses strict-equality, so any menu addition is a breaking bump.
-- **DX-2 — unconditional builtins.** `StoragePlugin` and `SecretsPlugin` join `CORE_UNCONDITIONAL_PLUGIN_NAMES` — required infra can't be disabled into a broken boot.
-- **DX-3 — external-provider config surface is a follow-up**, not built here (fixed typed sub-configs only in Phase scope).
-- **D1 / S1 — open method token.** `method` config fields accept an open `str`, validated at registry lookup (unknown → `UnknownStorageMethodError` / `UnknownSecretsMethodError`), not at parse.
-- **W-A (secrets boot ordering) — RESOLVED, refined in Phase 3.** The `# needed for gateway check` comment was stale; the gateway path is secrets-free, so the hardcoded `EnvSecretsProvider()` is gone and secrets is now config-selected from the registry. **Placement deviation (deliberate, see "Phase 3 — as-built"):** the plan's literal W-A said move the `build_registrar` call *above the gateway block* (after line 209 / before old 212). Phase 3 instead placed it *after* the gateway service/terms-check block and *just before the telemetry factory* (secrets' true first consumer). Both satisfy W-A's only hard constraint (secrets resolved before telemetry); the after-the-gate placement additionally preserves precondition-gate-first semantics (an unaccepted-terms / first-run boot fails fast before any discovery work) — a contract two `__new__`-based gateway-terms unit tests encode, which the above-the-gate placement broke.
-
-## Sequencing
-
-Storage vertical (Phases 1–2) → Secrets vertical (Phases 3–4) → Release gating (Phase 5). One coherent commit per phase; a mandatory checkpoint after each.
-
----
-
-## ⛔ Checkpoint protocol (MANDATORY at every checkpoint — do not skip, do not merge phases)
-
-At each `CHECKPOINT`, the agent **must stop** and do all three, in order:
-
-1. **Verify progress.** Run the gates for the phase and paste real results:
- - Always: `make agent-check` (pyright/ruff/mypy/plxt/keyword-only) + `make tb` (boot sequence — critical whenever config/registry wiring changed).
- - Test phases (2, 4) and the final phase: full `make agent-test`. Intermediate phases may scope to the phase's tests but must still pass `agent-check` + `tb`.
- - If a gate fails: fix before proceeding; a checkpoint is not cleared with a red gate.
-
-2. **Commit, then update this file for cold start.** Commit the phase as one commit. Then edit `TODOS.md` (and/or the linked wip doc) so a brand-new session could resume with no lost context: tick the phase's checkboxes, record the **commit SHA**, any decisions taken or deviations from the linked plan, the current state of the code, and the exact next action. Treat this as a handoff you won't be present to explain.
-
-3. **Fan-out a Sonnet-5 `/code-review` sub-agent — with NO inherited context.** Spawn a **fresh** sub-agent (never a fork) whose entire input is a *pointer to the phase's changes* — the commit SHA / `git diff <base>..HEAD` / the working-tree file list — and nothing else. Do **not** hand it the plan, the rationale, the decisions, or your conclusions; a clean-room review is the point (we want clean solid software, not over-engineering). Spawn template:
-
- ```
- Agent(
- subagent_type: "general-purpose", # fresh context — NOT "fork"
- model: "sonnet", # Sonnet-5
- description: "code-review phase N",
- prompt: "Run the /code-review skill on the changes in commit <SHA> "
- "(inspect via `git diff <SHA>^..<SHA>` in /Users/lchoquel/repos/Pipelex/_plugins). "
- "Review ONLY those changes. You have no prior context and should assume none. "
- "Focus: correctness bugs, and over-engineering / unnecessary abstraction / "
- "speculative generality. Report findings ranked most-severe first; "
- "if nothing substantive, say so plainly."
- )
- ```
-
- Then **triage** the findings: apply genuine bug/simplification fixes in a follow-up commit; capture design-tradeoff findings (not silent bugs) as a deferred note under `wip/plugins/` rather than reflexively applying the convenient fix. Record the triage outcome in this file. Only then move to the next phase.
-
----
-
-## Phase 1 — Storage: mechanism + builtin plugin + boot wiring
-*(Detail: `wip/plugins/storage-provider-plugin.md` Phases 1–3. Delivers a working config-selected storage plugin.)*
-
-- [x] `PLUGIN_API_VERSION` 2→3 in `pipelex/plugins/contract.py` (the one batched bump — DX-1).
-- [x] Registrar (`pipelex/plugins/registrar.py`): `storage_providers` dict field + `add_storage_provider(*, method, factory)` via the `_add` helper; `StorageProviderFactoryFn` alias.
-- [x] New `pipelex/plugins/storage_provider_registry.py`: `StorageProviderRegistry` (mirror `OrchestratorRegistry`: `get_optional`/`get_required`/`has`/`methods`).
-- [x] New `UnknownStorageMethodError` in `pipelex/plugins/exceptions.py`.
-- [x] Hub (`pipelex/hub.py`): `set_/get_storage_provider_registry` + module accessor.
-- [x] New `pipelex/plugins/storage/storage_plugin.py`: `StoragePlugin` (`name="storage"`) + module-level factory closures (`_make_local/_in_memory/_s3/_gcp_storage_provider`) holding the exact bodies moved from the deleted `make_storage_provider_from_config` `match` arms (keep `lazy_validate()`; keep GCP's `get_secrets_provider()` read — legal at the boot apply-point).
-- [x] `pipelex/plugins/builtins.py`: register `StoragePlugin()`; add `"storage"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.
-- [x] Boot (`pipelex.py`): build + `set_storage_provider_registry(...)` alongside the other registries; replace the `make_storage_provider_from_config` block with registry selection (explicit param > `registry.get_required(method=config.method)(config)`).
-- [x] Delete `pipelex/tools/storage/storage_provider_factory.py` (grep for other importers first — and re-home them).
-- [x] D1: relax `StorageProviderConfig.method` to open `str`.
-
-### Phase 1 — as-built (deviations & decisions, for cold start)
-
-- **Boot ordering (resolved a plan ambiguity).** The old factory ran at `pipelex.py:307-310`, *before* `build_registrar` (~393), but the registry only exists after `build_registrar`. So storage selection **moved down** to sit right beside the other registry constructions (after the 4 `set_*_registry` calls): build `StorageProviderRegistry(plugin_registrar.storage_providers)` → `set_storage_provider_registry` → `if storage_provider is None: select via registry` → `set_storage_provider`. Verified safe: no `get_storage_provider()` consumer runs during `setup()` before that point (all consumers are run-time), and secrets is on the hub (line 306) so the GCP factory's secret read works. This is byte-equivalent to the old factory and forward-compatible with the Phase 3 W-A move (which relocates only the `build_registrar` *call*, not the storage registry construction).
-- **`DuplicateStorageProviderError` added** (not in the plan checklist but required by the `_add` `on_duplicate` contract; mirrors `DuplicateOrchestratorError`).
-- **D1 blast radius.** `method: StorageMethod` → `method: str = Field(strict=False)` forced a `case _` in `storage_path` + `uri_format` (and the model-validator) because `reportMatchNotExhaustive` fires on open `str`. Empirically confirmed the field coerces a `StrEnum` input to a plain `str` and accepts an external token (e.g. `"azure"`) at parse. Same `case _` added to the two test-side matches (`test_storage_config.py`, `generator_fixtures.py`).
-- **Test re-homing.** `test_storage_provider_factory.py` **deleted** (its SUT is gone; Phase 2 adds the registry + parametrized-boot seam tests that supersede it). `generator_fixtures.py` now selects through `get_storage_provider_registry().get_required(...)`. `test_storage_provider_config.py`'s strict-coercion test rewritten to the D1 open-token behavior (`method == "local"` as plain str; `"azure"` accepted at parse).
-- **Built-in tokens** registered as the `StorageMethod` enum values (`StrEnum` keys are interchangeable with their plain-str form in dict lookup, so boot's plain-str `config.method` resolves them).
-
-### ⛔ CHECKPOINT 1 — run the protocol above (verify → commit + update → clean-room /code-review → triage)
-- [x] Gates green: `make agent-check` (ruff/plxt/pyright/mypy/keyword-only all pass), `make tb` (9 passed), `pipelex plugins list` shows `storage` (builtin, API 3, 4 contributions), boot `method="local"`→`LocalStorageProvider` / `method="in_memory"`→`InMemoryStorageProvider`, unknown token → `UnknownStorageMethodError`. Targeted `tests/unit/pipelex/plugins/ tests/unit+integration/.../tools/storage/` = 1036 passed; re-homed fixture consumers = 4 passed.
-- [x] Commit SHA recorded here: `8268ff08f`
-- [x] Cold-start state updated in this file.
-- [x] Sonnet-5 clean-room `/code-review` fanned out on `8268ff08f`; findings triaged. **Outcome:** review verified correctness (byte-identical factory bodies, safe `StorageMethod→str` D1 relaxation, correct boot ordering, keyword-only compliant, no bare `except`). 3 findings:
- 1. *contract.py v3 comment implied secrets registry already exists* → **FIXED** in follow-up `58961848a` (reworded to "pre-reserved / lands in a follow-up").
- 2. *storage-selection path has no direct automated test after deleting the factory test* → **deferred to Phase 2 as already planned** (Phase 2 checklist adds registry hit/miss + duplicate + parametrized-boot). Manually smoke-verified this phase.
- 3. *`StorageProviderRegistry.get_optional`/`has` have no caller* → **kept (no change).** Confirmed by grep that all sibling read-views (orchestrator/bundle_validator/model_lister) expose the identical shape; `has` is family-wide unused and `get_optional` used by exactly one. It's an established, plan-mandated registry-family convention, not speculative surface — trimming would make storage the odd one out.
- Reviewer also noted a benign boot-ordering nuance: a bad storage config now fails *after* `models_manager.setup()` rather than immediately after secrets (necessary consequence of needing the registrar built first; behavior otherwise byte-equivalent).
-
----
-
-## Phase 2 — Storage: tests + docs
-*(Detail: `wip/plugins/storage-provider-plugin.md` Phases 4–5.)*
-
-- [x] Unit: `StorageProviderRegistry` hit/miss (`UnknownStorageMethodError`) + duplicate-registration fail-loud via `_add`. → `test_storage_provider_registry.py`.
-- [x] Built-in factories: each built-in `method` yields the right provider (registry selection); gcp reads its credentials from the hub secrets provider. → `test_storage_plugin.py`. *(Reframed from "parametrized boot" to direct factory-selection — see as-built; the boot-select line is covered end-to-end by the integration test.)*
-- [x] Integration: an external test plugin registering a fake `method` (entry-point discovered) is selectable via config, and lands on the hub. → `test_storage_external_plugin.py`.
-- [x] Import-light registration pinned: s3/gcp factories register without importing their SDK. → `test_import_light_boot.py` (added `google.cloud.storage` to BLOCKED + assert `storage_providers`).
-- [x] New `docs/under-the-hood/storage-provider-plugins.md` (mirrors `orchestrator-plugins.md`); mkdocs nav (both blocks); cross-links from `inference-backend-plugins.md` Related. *(No "the plugin seams are …" enumeration exists to update — grep-verified.)*
-- [x] CHANGELOG `[Unreleased]`: "breaking" `PLUGIN_API_VERSION` 2→3; storage is now a plugin seam. *(Added a fresh `[Unreleased]` — none existed, branch base was v0.37.0.)*
-
-### Phase 2 — as-built (deviations & decisions, for cold start)
-
-- **The s3/gcp SDK guard is deferred to *use*, not *selection* (plan text was imprecise).** The plan/detail-doc said "s3/gcp arms raise `MissingDependencyError` only when *selected* with the SDK absent." Not so: `S3StorageProvider.__init__` / `GcpStorageProvider.__init__` only store fields; the `find_spec` guard lives in `_get_session` / `_get_bucket`, invoked inside the I/O methods. So *selecting* (constructing) s3/gcp NEVER raises `MissingDependencyError` even with the SDK absent — the error is deferred to first load/store. The correct invariant Phase 2 pins is therefore **import-light registration** (the s3/gcp factories register without importing their SDK), done by extending `test_import_light_boot.py`. The `MissingDependencyError`-on-use path is already covered by the dedicated `test_{s3,gcp}_storage_provider.py`, so it is not re-tested in the seam tests (avoids redundancy). Both aioboto3 and google-cloud-storage happen to be installed in this dev venv, so an in-process "SDK absent" test would need to mock `find_spec` anyway.
-- **Phase-1 latent breakage found + fixed (Phase-1 escapee).** `tests/integration/pipelex/system/test_hub_slot_injection_precedence.py` booted with an empty fake registrar and no explicit `storage_provider`; Phase 1's new selection path (`get_required(method=config.method)`) now rejects that with `UnknownStorageMethodError`. Checkpoint 1 only ran the plugins/storage dirs, not `system/`, so it escaped — the full `make agent-test` at Checkpoint 2 caught it. Fixed by injecting a dependency-free `InMemoryStorageProvider()` in the four boots (this suite tests hub slots, not storage). Grep-verified it was the *only* test with the empty-registrar-boot pattern (`test_plugin_discovery` / `test_import_light_boot` call `build_registrar` directly, never `Pipelex.make`, so they're unaffected).
-- **Test layout** (one TestClass per module): registry read-view + duplicate (`test_storage_provider_registry.py`, mirrors `test_bundle_validator_registry.py`); built-in factories + gcp secret-wiring (`test_storage_plugin.py`); integration discovery→selection→hub (`test_storage_external_plugin.py`). Config helpers reused from `tests/unit/pipelex/tools/storage/test_storage_provider_config.py` (`make_{local,in_memory,s3,gcp}_config`) — local construction touches no filesystem (no tmp_path needed).
-- **Integration harness** = `mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[SimpleNamespace(name=..., load=lambda: FakePluginClass)])` (discovery instantiates a callable/class), keep the real `BUILTIN_PLUGINS`, boot `Pipelex.make(needs_inference=False, config_overrides={"pipelex": {"storage_config": {"method": "test_mem"}}})`, assert `get_storage_provider()` is the fake + registry `.has(method="test_mem")`. An external method token loads fine (the config validator's `case _: pass`). Per-module autouse `reset_pipelex_config_fixture` (teardown-first) overrides the global module fixture (mirror of the two existing `system/` boot-test modules). This is also the harness template for the Phase 4 secrets external-plugin test.
-- **GCP secret-wiring assertion (Phase-2 review triage).** The first cut only asserted the secret was *requested*; the clean-room reviewer flagged it didn't assert the *value* flowed. Strengthened via `mocker.patch(..., wraps=GcpStorageProvider)` + `assert_called_once_with(credentials_file_path=fake_secrets.credentials_path, …)` — the repo-idiomatic way (`reportPrivateUsage`/`SLF001` both hard-block a `provider._credentials_file_path` read, so no private access / suppressions).
-
-### ⛔ CHECKPOINT 2 — CLEARED
-- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), **full `make agent-test` (exit 0, "All tests passed")**. Targeted storage-seam + at-risk boot test also green in isolation.
-- [x] Commit SHAs: `f08193e81` (Phase 2 code/tests/docs/changelog) + `37a3b72f1` (review triage).
-- [x] Cold-start state updated in this file.
-- [x] Sonnet clean-room `/code-review` fanned out on `f08193e81`; findings triaged. **Outcome:** 2 findings, **both genuine and applied** in `37a3b72f1`. (1) *High — doc/code contradiction:* the new page reasserted "the same mechanism backs the secrets provider seam" as present tense, but secrets is not yet registry-selected (and commit `58961848a` had just retracted the same claim in `contract.py`) → reworded to planned/future. (2) *Medium — test gap:* the gcp test checked the secret was requested but not that its value reached the provider → added the wraps-ctor `assert_called_once_with` above. Reviewer otherwise verified: no correctness bugs, no over-engineering, mocks faithful to the real contracts, no tautological assertions, no redundancy across the three new files, all doc snippets match source, MkDocs/pytest conventions respected.
-
----
-
-## Phase 3 — Secrets: mechanism + config + builtin plugin + boot wiring (W-A)
-*(Detail: `wip/plugins/secrets-provider-plugin.md` Phases 1–3. W-A is locked — this is a build instruction, not an investigation.)*
-
-- [x] Registrar: `secrets_providers` dict + `add_secrets_provider(*, method, factory)` via `_add`; `SecretsProviderFactoryFn` alias. *(API already at 3 from Phase 1 — no second bump.)*
-- [x] New `pipelex/plugins/secrets_provider_registry.py`: `SecretsProviderRegistry`.
-- [x] New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`. *(Also added `DuplicateSecretsProviderError` — required by the `_add` `on_duplicate` contract, mirroring storage.)*
-- [x] Hub: `set_/get_secrets_provider_registry` + module accessor.
-- [x] New config `SecretsProviderConfig` (`pipelex/tools/secrets/secrets_config.py`), `method: str = Field(strict=False)`, **no class default**. Placed as sibling of `storage_config` under `pipelex.*` (single flat model, no `SecretsConfig` subclass needed — secrets has no is_fetch/is_upload analogue).
-- [x] `pipelex/pipelex.toml`: added `[pipelex.secrets_config] method = "env"`; wired `secrets_config: SecretsProviderConfig` into the `Pipelex` config model. Same real-valued block added to `.pipelex/pipelex.toml` override. `make tb` green.
-- [x] New `pipelex/plugins/secrets/secrets_plugin.py`: `SecretsPlugin` + `_make_env_secrets_provider(config)` → `EnvSecretsProvider()` (unused `config` param carries `# noqa: ARG001`, the plugins-dir convention for a callback-signature-conforming factory).
-- [x] `builtins.py`: registered `SecretsPlugin()` in `BUILTIN_PLUGINS`; added `"secrets"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.
-- [x] **W-A boot edit** (`pipelex.py`): `build_registrar` + boot-orchestrator gate + secrets registry/selection now sit **just before the telemetry factory** (not above the gateway block — see as-built). Downstream registry constructions still reference the same `plugin_registrar`.
-- [x] **Deleted** the stale `# needed for gateway check` + hardcoded `EnvSecretsProvider()`; replaced with registry selection (explicit `setup(secrets_provider=...)` still wins). Dropped the now-unused `EnvSecretsProvider` import from `pipelex.py`.
-
-### Phase 3 — as-built (deviations & decisions, for cold start)
-
-- **W-A boot placement refined — after the precondition gate, not above the gateway block (the one real deviation).** The plan's literal W-A said move `build_registrar(config=get_config())` up to *before the gateway service block* (after line 209 / old line 212). Doing exactly that broke two unit tests — `tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py::{test_first_run_raises_inference_setup_required, test_needs_inference_true_raises_when_terms_not_accepted}`. Those tests exercise the gateway terms/first-run gate in isolation via `Pipelex.__new__(Pipelex)` (skipping `__init__`, so no config/hub is set) and rely on `setup()` reaching the terms-check `raise` **before** touching `get_config()` / `self.pipelex_hub`. Moving `build_registrar(config=get_config())` above the gate made `setup()` call `get_config()` first → `RuntimeError: Config instance is not set`. **Resolution:** placed the whole plugin-discovery + boot-orchestrator gate + secrets-registry/selection block **after** the gateway service/terms-check block and **immediately before the telemetry factory** — telemetry (`make_telemetry_manager`) is the *true first consumer* of `secrets_provider`, which is W-A's only hard constraint. This is strictly better than the plan's literal placement: the terms/first-run precondition gate fails fast before any discovery work, and no `__new__` test needed changing for the gate. (The boot-orchestrator gate also moved to this spot — still far earlier than its original post-`models_manager.setup` position, so fail-fast is preserved.)
-- **`DuplicateSecretsProviderError` added** (not in the plan checklist) — required by the `_add` `on_duplicate` contract; mirrors `DuplicateStorageProviderError` / `DuplicateOrchestratorError`.
-- **`SecretsProviderConfig` is a single flat model**, not a base+subclass like storage's `StorageProviderConfig`/`StorageConfig`. Storage's subclass exists only to add `is_fetch_remote_content_enabled`/`is_upload_local_content_enabled`; secrets has no such extra fields, so a lone `SecretsProviderConfig(ConfigModel)` with just `method` is the clean-solid shape (no speculative base).
-- **Phase-1 escapee proactively found + fixed (same class as storage's).** `test_hub_slot_injection_precedence.py` boots a fake **empty** registrar (patched into `build_registrar`) and passed an explicit `storage_provider` but no `secrets_provider`; the new secrets selection path (`get_required(method="env")`) rejects the empty registry with `UnknownSecretsMethodError`. Fixed by injecting `EnvSecretsProvider()` in the four boots (+ docstring updated). Grep-verified this was the only `Pipelex.make` boot with a mocked empty/partial registrar (the plugins-dir tests call `build_registrar` directly; `test_storage_external_plugin` keeps the real `BUILTIN_PLUGINS`, so its `env` secrets provider is present). Caught here via the full `make agent-test` rather than escaping to a later checkpoint.
-- **The env factory ignores config.** `_make_env_secrets_provider(config)` returns a bare `EnvSecretsProvider()`; the `config` param exists only to conform to `SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], ...]` (an SDK-backed external provider would read its own settings from it). Marked `# noqa: ARG001`, the established plugins-dir convention (see linkup/bedrock/azure_rest workers).
-
-### ⛔ CHECKPOINT 3 — CLEARED
-- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), **full `make agent-test` (exit 0, "All tests passed")**, `check-config-sync` PASSED (after the triage fix). Manual smokes: `pipelex plugins list` shows `secrets` (builtin, API 3, `secrets provider env`); default `method="env"`→`EnvSecretsProvider`; unknown token→`UnknownSecretsMethodError`.
-- [x] Commit SHAs: `8a0b32668` (Phase 3 code) + `fdf3e114b` (review triage).
-- [x] Cold-start state updated in this file.
-- [x] Sonnet clean-room `/code-review` fanned out on `8a0b32668` (fresh general-purpose agent, no context, pointed only at the commit); findings triaged. **Outcome:** reviewer confirmed **no correctness bug in the boot-order move** (traced every consumer: `build_registrar` is pure, runs after the gateway/terms gate and before the telemetry factory, its only pre-hub consumer; `UnknownBootOrchestratorError` moving earlier is a correct fail-fast improvement). 4 findings:
- 1. *High — kit-config drift:* `pipelex/kit/configs/pipelex.toml` (the `pipelex init` template) missing the new `[pipelex.secrets_config]` block ⇒ `make check-config-sync` fails (CI + `make check` gate; **not** in `agent-check`, which is why Checkpoint-3 gates missed it). → **FIXED** in `fdf3e114b` via `make up-kit-configs`. *(Reconciled against the `worker_scopes` "not in kit copies" false-positive memory: that gate compares kit-configs ↔ `.pipelex` only, not the base `pipelex.toml`; a net-new section in `.pipelex` genuinely diverges — real, not a false positive.)*
- 2. *Medium — stale comment:* the secrets-hub-set comment still said the registrar is "built below," but the reorder moved that build above it. → **FIXED** in `fdf3e114b` (reworded).
- 3. *Over-engineering flag (registry+plugin+config+exceptions for a one-impl seam)* → **kept, no change.** Reviewer explicitly called it "consistent architecture applied deliberately across provider verticals, not a bug" — it is the plan-mandated byte-for-byte mirror of the accepted storage vertical.
- 4. *Docs + dedicated tests missing* → **deferred to Phase 4 as planned** (Phase 3 = mechanism/boot; Phase 4 = tests + docs, exactly as storage split 1↔2).
-
----
-
-## Phase 4 — Secrets: tests + docs
-*(Detail: `wip/plugins/secrets-provider-plugin.md` Phases 4–5.)*
-
-- [x] Unit: `SecretsProviderRegistry` hit/miss (`UnknownSecretsMethodError`) + duplicate fail-loud. → `test_secrets_provider_registry.py` (byte-for-byte mirror of `test_storage_provider_registry.py`).
-- [x] Boot: default config yields `EnvSecretsProvider` on the hub; explicit `setup(secrets_provider=...)` overrides. → `test_secrets_external_plugin.py` (first two methods). Plus a `test_secrets_plugin.py` unit (plugin registers the `env` factory; factory constructs `EnvSecretsProvider`), mirroring `test_storage_plugin.py` but lean (single builtin, no optional-dep matrix).
-- [x] Integration: external test plugin registering a fake secrets `method` (`"test_secret"`, entry-point discovered) is selectable via config. → `test_secrets_external_plugin.py`.
-- [x] Ordering guard: a boot with a non-env secrets method has secrets on the hub before storage's GCP arm reads it. → `test_secrets_external_plugin.py::test_config_selected_secrets_provider_is_read_by_storage_gcp_arm_at_boot`.
-- [x] New `docs/under-the-hood/secrets-provider-plugins.md` (emphasizes the lazy optional-dep closure for SDK-backed Vault/AWS providers + the boot-ordering relationship); mkdocs nav (both blocks); cross-links (storage↔secrets both directions, inference-backend Related, secrets-provider-injection); updated the storage page's stale "planned" forward-ref.
-- [x] CHANGELOG `[Unreleased]`: secrets is now a plugin seam; folded `add_secrets_provider` into the existing `PLUGIN_API_VERSION` 2→3 note.
-
-### Phase 4 — as-built (deviations & decisions, for cold start)
-
-- **The ordering guard is stronger than "assert boot order" implied — telemetry ALSO consumes the config-selected secrets provider, earlier.** First cut asserted `requested_secret_ids == ["GCP_CREDENTIALS_FILE_PATH"]` (exact). It failed: the recorded list led with `POSTHOG_ENDPOINT`, `POSTHOG_API_KEY`, … — the **telemetry factory** reads secrets from the same provider *before* storage is selected (secrets resolves right before the telemetry factory; storage much later). This is extra confirmation of correct ordering, not a bug. Fixed to a **membership** check: `"GCP_CREDENTIALS_FILE_PATH" in requested_secret_ids`. That token is requested **only** by storage's gcp factory (grep-confirmed), so its presence uniquely proves the gcp arm read the external provider from the hub. (Telemetry reads secrets even with `needs_inference=False` — it resolves the POSTHOG keys to build the manager, then the disabled path ignores them; the fake returning a canned gcp path for POSTHOG_* doesn't break boot.)
-- **The gcp override needs BOTH `bucket_name` and `project_id`.** Base `pipelex.toml` ships `[pipelex.storage_config.gcp]` with `bucket_name = ""` **and** `project_id = ""`; `StorageGcpConfig.lazy_validate()` (run inside the gcp factory) rejects either empty. So the ordering-guard boot passes `config_overrides={"pipelex": {"storage_config": {"method": "gcp", "gcp": {"bucket_name": "test-bucket", "project_id": "test-project"}}}}` (deep-merge fills the rest). `GcpStorageProvider.__init__` only stores fields (the `find_spec`/creds-file check lives in `_get_bucket`), so constructing it at boot with a fake creds path is import-light and does no I/O — safe even though `google-cloud-storage` is installed here.
-- **Default-env boot is deterministic on this branch.** `test_default_config_yields_the_env_provider_on_the_hub` boots with no `secrets_config` override and asserts `env`. Verified the machine global `~/.pipelex/pipelex.toml` has **no** `secrets_config` block (it's net-new here), so unlike storage's `method=s3` global-override gotcha, secrets has no machine override to fight — default→`env` holds locally and in CI.
-- **Import-light test NOT extended for secrets (deliberate, unlike storage Phase 2).** `test_import_light_boot.py` gained a storage assert because s3/gcp are SDK-backed and BLOCKED there — a real import-light risk. The only builtin secrets method is `env` (no SDK), so an added `assert registrar.secrets_providers` would restate the registry unit test, not pin an import-light invariant. Left untouched to avoid over-engineering; an external SDK-backed secrets plugin would carry its own import-light guarantee.
-- **Test module layout** (one TestClass per module): `test_secrets_provider_registry.py` (registry read-view + duplicate), `test_secrets_plugin.py` (builtin plugin + env factory), `test_secrets_external_plugin.py` (integration: default/override/external/ordering — one class, four methods). Integration harness identical to `test_storage_external_plugin.py`: `mocker.patch("pipelex.plugins.discovery._external_entry_points", …)` keeps the real `BUILTIN_PLUGINS`, per-module autouse teardown-first `reset_pipelex_config_fixture`. The fake external provider is a `SecretsProviderAbstract` subclass recording requested secret ids.
-- **The Phase-3 escapee stayed fixed.** `test_hub_slot_injection_precedence.py` (fixed in Phase 3 to inject `EnvSecretsProvider()` in its four boots) passed under the full suite — no new escapee surfaced.
-
-### ⛔ CHECKPOINT 4 — CLEARED
-- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), `check-config-sync` PASSED, **full `make agent-test` (exit 0, "All tests passed")**. Targeted secrets-seam trio green in isolation (11 passed).
-- [x] Commit SHA recorded here: `65603e912`
-- [x] Cold-start state updated in this file.
-- [x] Sonnet-5 clean-room `/code-review` fanned out on `65603e912` (fresh general-purpose agent, no context, pointed only at the commit). **Outcome: no substantive findings — nothing to fix, nothing to defer.** The reviewer ran all three test files against the tree (11 passed) and traced every factual claim to source: confirmed the secrets-before-telemetry-before-storage boot ordering in `pipelex.py`, the `gcp` factory's lazy credential check (fake creds path never touches disk / needs no GCP SDK at boot), telemetry's always-on `${POSTHOG_*}` secret resolution (validating the ordering-guard's membership-check rationale), `PLUGIN_API_VERSION` still `3` (changelog correctly folds in, no second bump), and identical nav placement in both mkdocs blocks. Verdict: assertions are concrete and non-tautological (the ordering guard is a genuine regression guard — reordering secrets-after-storage would raise or read the wrong provider inside the gcp factory), no over-engineering (secrets tests deliberately leaner than storage's — no optional-dep matrix, accurate to the single-impl `env` seam), no doc/code contradictions, all cross-links resolve.
-
----
-
-## Phase 5 — Release gating & cross-repo (documentation in this branch; execution is release-gated)
-
-- [x] Record in the CHANGELOG / release notes the cross-repo `targets_api=3` obligation. **Corrected during Checkpoint-5 triage (F2):** the original wording ("they register no provider; the bump is the only change") was wrong for `pipelex-temporal`. `pipelex-mistralai-workflows` is bump-only, but `pipelex-temporal`'s payload-codec factory imports the now-removed `make_storage_provider_from_config`, so it needs the `targets_api` bump **and** a code migration to `get_storage_provider_registry().get_required(...)`. CHANGELOG now says this + carries a `### Removed` bullet for the deleted factory. Did NOT touch those repos here.
-- [x] Confirmed the whole branch is one clean per-phase commit sequence (feat/test → triage → bookkeeping per phase; `git log 04434f785..HEAD`). Opening the PR to the intended base is **left to Louis** (nothing pushed; no upstream).
-
-### ⛔ CHECKPOINT 5 (final) — CLEARED
-- [x] Gates green on the final tip: `make agent-check` (ruff/plxt/pyright 0 errors/mypy 2084 files/keyword-only), `make tb` (9 passed), `make check-config-sync` (passed), **full `make agent-test` (exit 0, "All tests passed")** — run both before the triage (tip `65603e912`+bookkeeping) and again after the triage commit `0bbc39474`.
-- [x] Final Sonnet-5 clean-room `/code-review` over the **whole-branch diff** (`git diff 04434f785..HEAD`) — fresh general-purpose agent, no inherited context, 10 finder angles + line-by-line + a test-quality pass. **Outcome: migration verified largely clean and faithful** (behavior-preserving factory→plugin split, correct secrets→hub→telemetry→storage→gcp-reads-secrets boot ordering, safe `StorageMethod`→`str` widening, conventions honored). 9 findings, all triaged in `0bbc39474`:
- - **F2 [MED-HIGH] — FIXED (genuine, was actively misleading):** deleting `make_storage_provider_from_config` breaks `pipelex-temporal` at import (its codec factory calls it); CHANGELOG claimed a `targets_api` bump was temporal's only change. Corrected CHANGELOG + wip doc + added Removed bullet.
- - **F6 [LOW] — FIXED (simplification):** removed dead `get_optional` from both registries (no production consumer; `get_required` doesn't delegate to it; only its own unit tests used it). This reverses Checkpoint-1's keep-decision, which rested on a factual misread — siblings expose `get_optional`+`has`, while storage/secrets hard-fail via `get_required`+`methods`, so `get_optional` was a vestigial mechanical copy, not a convention match.
- - **F3 [MED] — FIXED:** added symmetric storage injection-precedence test (was tested for secrets only).
- - **F5 [LOW] — FIXED:** parametrized `test_disabling_core_unconditional_plugin_raises` over `direct/storage/secrets/openai`.
- - **F8 [LOW] — FIXED:** `contract.py` v3 comment no longer says `add_secrets_provider` is "in a follow-up commit" (it landed in Phase 3).
- - **F9 [LOW] — FIXED:** `orchestrator-plugins.md` no longer claims `PLUGIN_API_VERSION` "is now 2".
- - **F1 [HIGH per reviewer] — DEFERRED to D3 (design tradeoff, not a silent bug):** external storage method boots/selects cleanly but crashes on the first *generated-content* store because `uri_format` is defined only for the built-in methods (raises loudly on `case _:`). External config surface is explicitly the D3 follow-up; strengthened the D3 note + added a docs-page caveat. External providers are usable today for their own storage API, not yet as the generated-content backing store.
- - **F7 [LOW] — KEPT:** `Field(strict=False)` on the `method` fields is cosmetically vestigial but removal risks a config-parse regression at enum-member call sites for zero benefit.
- - **F4 [MED] — SKIPPED:** a `lazy_validate` negative-path factory test (trivial 3-line wiring; per-config validation already covered) — avoided test bloat.
-- [x] This file updated to DONE with all SHAs; deferred follow-ups (D3 external config passthrough incl. the `uri_format` gap, first real external provider, cookbook example) captured under `wip/plugins/`.
-
----
-
-## Deferred follow-ups (not in this plan — see wip docs)
-
-- DX-3 external-provider config passthrough (storage D3 / secrets S4).
-- First real external provider (e.g. Vault / AWS Secrets Manager) as proof-of-seam — likely a private package if hosted-platform-targeted.
-- Cookbook `pipelex-storage-hello` external-plugin example, mirroring the hello-inference-plugin track.
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeBatch.md b/docs/building-methods/pipes/pipe-controllers/PipeBatch.md
index 36aa71488..f44aa7d3a 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeBatch.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeBatch.md
@@ -28,6 +28,12 @@ Results always preserve input order regardless of the concurrency bound. If a br
For durable, rate-limited execution of very large batches, run the pipeline on the Temporal track.
+## Compaction Under Absence
+
+When a branch's result resolves as a recorded absence — typically because the branch pipe is a `PipeCondition` whose outcome was `continue`, or the branch was skipped (lifted) on an absent optional value — that branch contributes **no item** to the aggregated output list. The list is compacted: a list cannot hold a hole, and absent results are dropped rather than replaced by placeholders. Order is preserved among the results that did produce a value, and each dropped branch is logged with its absence reason.
+
+This is the batch arm of the "route or skip" pattern: batch over items with a condition branch pipe that builds a result for the items that qualify and `continue`s past the rest — the output is the compacted list of qualifying results. See [Understanding Optionality](../understanding-optionality.md) for the full absence model.
+
## Configuration
`PipeBatch` is configured in your pipeline's `.mthds` file.
@@ -84,4 +90,5 @@ How this works:
## Related Documentation
-- [Understanding Multiplicity](../understanding-multiplicity.md) - How Pipelex handles lists and batch processing
\ No newline at end of file
+- [Understanding Multiplicity](../understanding-multiplicity.md) - How Pipelex handles lists and batch processing
+- [Understanding Optionality](../understanding-optionality.md) - Presence markers, absence records, and compaction
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeCondition.md b/docs/building-methods/pipes/pipe-controllers/PipeCondition.md
index 78d2232b6..362b15c61 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeCondition.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeCondition.md
@@ -11,8 +11,8 @@ The `PipeCondition` controller adds branching logic to your pipelines. It evalua
`PipeCondition` is a routing mechanism. Its execution flow is as follows:
1. **Evaluate an Expression**: It takes an expression and renders it using Jinja2, with the full `WorkingMemory` available as context. This evaluation results in a simple string.
-2. **Look Up in Pipe Map**: The resulting string is used as a key to find a corresponding pipe name in the `pipe_map`.
-3. **Use Default (Optional)**: If the key is not found in the `pipe_map`, it will use the `default_pipe_code` if one is provided. If there's no match and no default, an error is raised.
+2. **Look Up in the Outcome Map**: The resulting string is used as a key to find a corresponding outcome in `outcomes` — the name of a pipe to execute, or a special outcome (`continue` / `fail`).
+3. **Use Default**: If the key is not found in `outcomes`, the `default_outcome` is used.
4. **Execute Chosen Pipe**: The chosen pipe is then executed. It receives the exact same `WorkingMemory` and inputs that were passed to the `PipeCondition` operator. The output of the chosen pipe becomes the output of the `PipeCondition` itself.
## Configuration
@@ -29,12 +29,12 @@ The `PipeCondition` controller adds branching logic to your pipelines. It evalua
| `output` | string | The output concept produced by the selected pipe. | Yes |
| `expression` | string | A simple Jinja2 expression. `{{ ... }}` are automatically added. Good for simple variable access like `"my_var.category"`. | Yes (or `expression_template`) |
| `expression_template` | string | A full Jinja2 template string. Use this for more complex logic, like `{% if my_var.value > 10 %}high{% else %}low{% endif %}`. | Yes (or `expression`) |
-| `pipe_map` | table (dict) | A mapping where keys are the possible string results of the expression, and values are the names of the pipes to execute. | Yes |
-| `default_pipe_code` | string | The name of a pipe to execute if the expression result does not match any key in `pipe_map`. | No |
+| `outcomes` | table (dict) | A mapping where keys are the possible string results of the expression, and values are the names of the pipes to execute — or the special outcomes `continue` / `fail`. | Yes |
+| `default_outcome` | string | The outcome used when the expression result does not match any key in `outcomes`: a pipe name, `continue`, or `fail`. | Yes |
| `add_alias_from_expression_to` | string | An advanced feature. If provided, the string result of the expression evaluation is added to the working memory as an alias with this name. | No |
!!! important "Output Concept Matching"
- The output concept of the `PipeCondition` has to match the output of all the pipes in the `pipe_map`.
+ The output concept of the `PipeCondition` has to match the output of all the pipes in the `outcomes`.
### Example: Simple routing based on category
@@ -54,8 +54,9 @@ description = "Route based on category field"
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression = "input_data.category"
+default_outcome = "fail"
-[pipe.route_by_category.pipe_map]
+[pipe.route_by_category.outcomes]
small = "process_small"
medium = "process_medium"
large = "process_large"
@@ -89,7 +90,7 @@ Output this only: "large"
How this works:
1. `PipeCondition` receives input data with a `category` field (e.g., `{category: "small"}`)
2. It evaluates the expression `"input_data.category"` which results in the string `"small"`
-3. It looks up `"small"` in the `pipe_map` and finds the corresponding pipe: `"process_small"`
+3. It looks up `"small"` in the `outcomes` and finds the corresponding pipe: `"process_small"`
4. The `process_small` pipe is executed with the same working memory
5. The output from `process_small` becomes the output of the entire `PipeCondition`
@@ -102,9 +103,9 @@ description = "Route with default handling"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
expression = "classification.type"
-default_pipe_code = "process_unknown"
+default_outcome = "process_unknown"
-[pipe.route_with_fallback.pipe_map]
+[pipe.route_with_fallback.outcomes]
invoice = "process_invoice"
receipt = "process_receipt"
@@ -136,6 +137,42 @@ Process this unknown document type...
"""
```
+## Special outcomes: `continue` and `fail`
+
+Beside pipe names, an outcome (or the `default_outcome`) can be one of two special outcomes:
+
+- **`fail`**: the run fails loudly with an error naming the pipe and the evaluated expression. Use it to make "this should never happen" branches explicit.
+- **`continue`**: the condition declares that it produced **no output**. The runtime records an absence for the declared output (with the evaluated expression as the reason) and the run continues as a success — the rest of the working memory is unchanged.
+
+Because `continue` resolves the declared output as absent, a `continue`-reachable condition (any outcome mapped to `continue`, or `default_outcome = "continue"`) **must declare its output optional** — `output = "Constraint?"`. Validation enforces this statically (`optional_output_required`), so the no-output path is always visible to the type system. The same visibility rule applies at the condition's boundary: if a mapped outcome pipe declares an optional output, the condition must declare `?` too (`optional_not_handled`).
+
+### What happens downstream of `continue`
+
+An absent output propagates like any recorded absence:
+
+- A downstream pipe consuming it as a **plain** input is skipped (lifted), and its own output is recorded absent in turn.
+- A downstream pipe declaring the input **optional** (`?`) runs and handles both arms.
+- A downstream pipe declaring the input **forced** (`!`) fails with a typed error carrying the full provenance chain.
+- Inside a `PipeBatch`, an absent branch result is dropped from the aggregated list (compaction): batching a "keep or skip" condition over items yields a list of only the kept results.
+- Inside a `PipeParallel`, an absent branch result is omitted from the combined output (a non-required structured field absorbs it as its default — `null` unless the field declares another default).
+
+!!! warning "Breaking change: `continue` no longer passes the previous output through"
+ `continue` used to deliver the current main stuff by passing it through. It now resolves the condition's output as **absent** instead. The previous value is not lost — it stays in the working memory under its own name, so a downstream pipe consumes it explicitly by that name (declared `?` when it may be absent). If you need "use the new value if produced, otherwise the previous one", that coalescing idiom is planned as a follow-up; today, model it with an explicit downstream pipe that declares both inputs.
+
+```toml
+[pipe.build_or_skip]
+type = "PipeCondition"
+description = "Builds a constraint for approved links, skips rejected ones"
+inputs = { verified_link = "VerifiedLink" }
+output = "Constraint?"
+expression = "verified_link.verdict"
+default_outcome = "continue"
+
+[pipe.build_or_skip.outcomes]
+approved = "build_single_constraint"
+rejected = "continue"
+```
+
## Expression Types
### Simple Expression
@@ -162,7 +199,7 @@ expression_template = "{% if input_data.score >= 70 %}high{% else %}low{% endif
### Default Routing
```python
-default_pipe_code = "process_unknown"
+default_outcome = "process_unknown"
```
- Fallback pipe when no match is found
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md b/docs/building-methods/pipes/pipe-controllers/PipeParallel.md
index 3cbd45cb9..e2d454c3f 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeParallel.md
@@ -24,6 +24,14 @@ The declared `output` of a `PipeParallel` is the concept the branch results are
Anything else is rejected at validation time: scalar native concepts (`Text`, `Image`, ...), `Dynamic`, `Anything`, and multiplicity suffixes (`Foo[]`, `Foo[3]`) are all invalid — a parallel combination is a named composite, never a scalar or a list (a list aggregation is [`PipeBatch`](PipeBatch.md)'s shape).
+### Combining under absence
+
+A branch result may legitimately be **absent** at run time: the branch pipe declares an optional output (`?`), or the branch is lifted (skipped) because a plain input is fed by one of the parallel's own optional (`?`) inputs. The combine handles absence by output kind:
+
+- **`Composite` output**: the absent component is simply omitted from the combined object (an absence note is kept in the ledger for observability).
+- **Structured output, non-required field**: the absence is absorbed as the field's default (`null` unless the field declares another default).
+- **Structured output, required field**: this is rejected **statically** at validation time (`optional_branch_required_field`) — a required field cannot be fed by a maybe-absent branch. Make the field non-required, or sink the absence upstream with a `?` input on the branch path.
+
## Configuration
`PipeParallel` is configured in your pipeline's `.mthds` file.
diff --git a/docs/building-methods/pipes/pipe-operators/PipeLLM.md b/docs/building-methods/pipes/pipe-operators/PipeLLM.md
index 5d3545b7f..1999d4662 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeLLM.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeLLM.md
@@ -222,7 +222,7 @@ Analyze the document and explain how it relates to the context: $reference_doc
| `model` | string or table | Specifies the LLM choice by name, setting, or preset to use. | No |
| `model_to_structure` | string or table | LLM choice used whenever this `PipeLLM` produces a structured output. Applies both to direct structured generation and to the structuring step when `structuring_method = "preliminary_text"` is set. | No |
| `system_prompt` | string | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Supports the same variable syntax as `prompt`, including image and document references. | No |
-| `prompt` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`). Use `@input_name` as a block sigil on its own line to insert the content of an entire input — inline `@var` raises `TemplateSigilSyntaxError` when `var` collides with a declared input. Image and document variables follow the same rules. Escape with `@@` or `$$` to emit a literal `@` or `$` (e.g., `@@font-face` inside a `<style>` block). | No |
+| `prompt` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`). Use `@input_name` as a block sigil on its own line to insert the content of an entire input — inline `@var` raises `TemplateSigilSyntaxError` when `var` collides with a declared input. For a declared-optional input (`?`), use `@?input_name` — it inserts the content when present and renders nothing when absent, and counts as a guard for the optionality lint (a bare `$var`/`@var` reference to an optional input is rejected as `optional_input_unguarded`; see [Understanding Optionality](../understanding-optionality.md)). Image and document variables follow the same rules. Escape with `@@` or `$$` to emit a literal `@` or `$` (e.g., `@@font-face` inside a `<style>` block). | No |
| `structuring_method` | string | `"direct"` (default) generates the structured output in one LLM call. `"preliminary_text"` expands the pipe at load time into a `PipeLLM` (text) + [`PipeStructure`](./PipeStructure.md) sequence — two LLM calls. Cannot be combined with a `Text` output. | No |
### LLM Setting Fields
diff --git a/docs/building-methods/pipes/understanding-optionality.md b/docs/building-methods/pipes/understanding-optionality.md
new file mode 100644
index 000000000..fddde15ee
--- /dev/null
+++ b/docs/building-methods/pipes/understanding-optionality.md
@@ -0,0 +1,120 @@
+---
+description: "Optionality lets a Pipelex method declare that a value may legitimately be absent — with `?` and `!` presence markers, recorded absences, and static validation that proves every absence is handled."
+---
+
+# Understanding Optionality
+
+Real methods meet values that may legitimately not exist: a nickname that was never provided, a document section that wasn't found, a branch of a condition that decided to produce nothing. Optionality makes that a first-class, declared part of your method's contract — instead of a crash, a fabricated placeholder, or a silent `null`.
+
+This guide explains the presence markers (`?` and `!`), what happens at run time when a value is absent, and the static validation that proves your method handles every possible absence before it ever runs.
+
+## The Presence Markers
+
+A concept reference in a pipe's `inputs` or `output` can carry a presence marker:
+
+| Declaration | Meaning |
+|---|---|
+| `notes = "Note"` | **Plain** (default): the pipe needs the value. If it is absent, the pipe is skipped (see [lifting](#the-runtime-trichotomy-skip-run-or-fail)). |
+| `notes = "Note?"` | **Optional** (`?`): the pipe runs even when the value is absent and handles absence itself. On an output, the pipe may resolve it as a *recorded absence* instead of a value. |
+| `notes = "Note!"` | **Force** (`!`, inputs only): the pipe asserts the value must be present — an absent value fails the run loudly with `OptionalValueAbsentError`, carrying the full provenance of where absence came from. |
+
+Two grammar rules keep the model coherent:
+
+- **Markers never combine with multiplicity.** `Note[]?` is invalid (`optional_marker_invalid`): a plural slot is never absent — when nothing is found, it is the **empty list**. Absence is a singular-value notion.
+- **Outputs take only `?`.** A force marker on an output has no meaning (a pipe cannot assert its own output into existence).
+
+Concept definitions, `refines`, and structure-field references never carry markers — presence is a property of a pipe's contract, not of a concept.
+
+## The Runtime Trichotomy: Skip, Run, or Fail
+
+At run time, a value slot resolves to a value **or a recorded absence** — never a silent hole. When a pipe is about to run and one of its inputs is absent, the marker decides what happens:
+
+1. **Plain input absent → the pipe is skipped (lifted).** The pipe never runs; a `skipped` absence is recorded for its output, chaining to the upstream absence that caused the skip. Downstream pipes then see that output as absent and the same rule applies — absence propagates until something handles it.
+2. **Optional (`?`) input absent → the pipe runs.** The pipe declared it can handle the absence — for template-rendering pipes that means guarding the reference (see [Guarding templates](#guarding-templates-over-optional-inputs)).
+3. **Force (`!`) input absent → the run fails loudly.** `OptionalValueAbsentError` names the variable, the pipe, and the provenance chain back to where absence entered the flow.
+
+A skipped pipe with a **plural** output resolves it as a real empty list (per the plural rule above), with a ledger note for observability — so downstream list consumers keep working.
+
+### Absence records and provenance
+
+Absence is carried as data: an `AbsenceRecord` with the variable name, a `kind` (`declared_absent` — a `?` output resolved absent; `skipped` — the producer was lifted; `not_provided` — the caller omitted an optional method input), a human-readable reason, the producing pipe, and the upstream record it chains to. Provenance is captured when absence is produced, not reconstructed at failure time — so a `!` failure or a run report can always tell you *why* a value is missing.
+
+The invariant everywhere: **a pipe run always resolves its declared output — a value, or a recorded absence.** An absent main output is a successful run; the CLI and the API deliver an explicit absence document (`{"absent": true, ...}`), never an empty result.
+
+## Optional Method Inputs
+
+A `?` on your method's entry-pipe input makes it omittable by the caller: leaving it out of `inputs.json` (or the execute request) records a `not_provided` absence instead of raising the missing-input error. Plain and `!` inputs stay required at the boundary.
+
+```toml
+[pipe.brief_flow]
+type = "PipeSequence"
+description = "Write a brief; enrich it when a hint is provided"
+inputs = { topic = "Topic", hint = "Hint?" }
+output = "Brief"
+steps = [
+ { pipe = "enrich_hint", result = "enriched" },
+ { pipe = "write_brief", result = "brief" },
+]
+```
+
+Here a caller may provide only `topic`. The `enrich_hint` step (which consumes `hint` plainly) is skipped, and `write_brief` — which takes the enrichment as `Enriched?` — runs with it absent.
+
+## Guarding Templates Over Optional Inputs
+
+A template that renders an optional input must guard the reference — otherwise validation rejects it (`optional_input_unguarded`). Three guard idioms are recognized:
+
+- **The optional block sigil `@?`** — inserts the input's content when present, renders nothing when absent:
+
+```toml
+prompt = """
+Write a brief about this topic: $topic
+
+@?enriched
+"""
+```
+
+- **A presence block** — `{% if enriched %} ... {% endif %}` (any reference inside the block is guarded, including `$enriched` and attribute access).
+- **An inline presence conditional** — `{{ enriched.text if enriched else "no enrichment" }}`.
+
+The guard-lint covers every pipe that renders authored templates over its inputs: PipeLLM (prompt and system prompt), PipeCompose (template mode), PipeCondition (expression), PipeSearch, and PipeImgGen.
+
+## Controllers Under Absence
+
+Each controller has a defined behavior when a value inside its flow resolves absent:
+
+- **PipeSequence** — steps consuming an absent value plainly are lifted (skipped), and the absence propagates step to step. The sequence's own declared output must account for it: if the last producing step can be lifted, the sequence output must be declared `?` (otherwise `optional_not_handled`).
+- **PipeCondition** — the `continue` outcome resolves the declared output as a **declared-absent** record (memory otherwise unchanged). A condition that can reach `continue` must declare its output `?` (`optional_output_required`). See [PipeCondition](pipe-controllers/PipeCondition.md).
+- **PipeParallel** — combining under absence: a `Composite` output omits the absent component (with a ledger note); a structured output absorbs an absent branch into a non-required field as that field's declared default; a **required** field fed by a maybe-absent branch is rejected statically (`optional_branch_required_field`). See [PipeParallel](pipe-controllers/PipeParallel.md).
+- **PipeBatch** — absent branch results are **compacted** out of the aggregated list (a list cannot hold a hole). See [PipeBatch](pipe-controllers/PipeBatch.md).
+
+## The Static Safety Net
+
+You never discover an unhandled absence at run time. Validation walks every controller flow and computes, per slot, whether it is guaranteed present or maybe-absent — and requires every possible absence to reach an explicit sink (a `?` boundary, a guard, an absorbing field). The optionality error types:
+
+| Error | Meaning |
+|---|---|
+| `optional_marker_invalid` | Grammar misuse: a marker combined with multiplicity, or `!` on an output. |
+| `optional_not_handled` | A maybe-absent value escapes through a non-optional boundary. The message names the absence origin, the propagation path, and the fixes. |
+| `optional_output_required` | A `continue`-reachable PipeCondition without a `?` output. |
+| `optional_input_unguarded` | An unguarded template reference to a declared-optional input. |
+| `optional_branch_required_field` | A maybe-absent parallel branch feeding a required structure field. |
+
+Two more optionality facts surface on the **valid** report (`pipelex validate`, the agent CLI, and the API):
+
+- **`liftable_pipes`** — every pipe that may be silently skipped at run time, with the slots whose absence triggers the skip and where that absence can come from. Implicit lifting is acceptable *because* it is visible at build time.
+- **`warnings`** — advisory lints that never flip the verdict. The first is `optional_force_redundant`: a `!` whose slot is guaranteed present in every analyzed flow — the assertion can never fire, so the marker is dead weight.
+
+## Best Practices
+
+- **Declare `?` at the boundary you mean it.** An optional method input is a statement to your callers; an optional output is a statement to your consumers. Inside the flow, prefer letting plain pipes lift over sprinkling `?` everywhere — the skip semantics usually express "no input, no output" exactly.
+- **Guard every optional reference in templates** with `@?` when you just want the content inserted, or `{% if %}` when the surrounding prose changes with presence.
+- **Reserve `!` for genuine assertions** — places where your flow analysis says the value could be absent but you know better and want a loud failure rather than a skip. If the lint says it's redundant, remove it.
+- **Check `liftable_pipes` when validating**: it is the list of "this may silently not run" — confirm each entry is intended.
+
+## Related Documentation
+
+- [Understanding Multiplicity](understanding-multiplicity.md) — the other axis of a slot's shape; plural absence is the empty list
+- [PipeCondition](pipe-controllers/PipeCondition.md) — the `continue` outcome and its migration notes
+- [PipeParallel](pipe-controllers/PipeParallel.md) — combining under absence
+- [PipeBatch](pipe-controllers/PipeBatch.md) — compaction of absent branch results
+- [Execution Graph Tracing](../../under-the-hood/execution-graph-tracing.md) — skipped nodes, `skip_reason`, and optional data edges
diff --git a/docs/errors/authoring-and-language.md b/docs/errors/authoring-and-language.md
index 56ca6c96e..c10832d09 100644
--- a/docs/errors/authoring-and-language.md
+++ b/docs/errors/authoring-and-language.md
@@ -27,6 +27,7 @@ own page. Classes are grouped by subsystem.
- [`InputStuffSpecsError`](input-stuff-specs-error.md) — Input stuff specs
- [`InputStuffSpecsFactoryError`](input-stuff-specs-factory-error.md) — Input stuff specs factory
- [`NativeConceptDefinitionError`](native-concept-definition-error.md) — Native concept definition
+- [`OptionalValueAbsentError`](optional-value-absent-error.md) — Optional value absent
- [`PipeFactoryError`](pipe-factory-error.md) — Pipe factory
- [`PipeInputError`](pipe-input-error.md) — Pipe input
- [`PipeInputsFactoryError`](pipe-inputs-factory-error.md) — Pipe inputs factory
diff --git a/docs/errors/optional-value-absent-error.md b/docs/errors/optional-value-absent-error.md
new file mode 100644
index 000000000..0d6ef22da
--- /dev/null
+++ b/docs/errors/optional-value-absent-error.md
@@ -0,0 +1,21 @@
+---
+title: "Optional value absent"
+description: "Reference for the `OptionalValueAbsentError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Optional value absent
+
+A force-marked (`!`) input was fed a recorded absence at run time (D9).
+
+| Field | Value |
+|---|---|
+| `error_type` | `OptionalValueAbsentError` |
+| `title` | Optional value absent |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/optional-value-absent-error/` |
+| `error_domain` | `runtime` |
+| Defined in | `pipelex.core.pipes.inputs.exceptions` |
+| Parent class | [`PipeRunError`](pipe-run-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/tools/cli/agent-cli.md b/docs/tools/cli/agent-cli.md
index eefbcbf6c..ff54158b6 100644
--- a/docs/tools/cli/agent-cli.md
+++ b/docs/tools/cli/agent-cli.md
@@ -76,6 +76,9 @@ For `bundle`, additional options are available:
On a successful `validate bundle` run, the envelope also carries `pending_signatures` — the library-wide list of pipes still declared as `PipeSignature` (unimplemented forward declarations), each namespaced by `pipe_ref` (`domain.code`). It is a non-blocking nudge: in JSON it is a `pending_signatures` array, in markdown a "Pending signatures" section. A top-down build reads it to see exactly which headers remain to implement. The envelope also carries a derived `is_runnable` boolean (`true` ⇔ `pending_signatures` is empty), and the markdown states the runnability verdict in plain English — runnable when complete, NOT yet runnable above the "Pending signatures" section otherwise. Both are scoped to `validate bundle`; `validate all` / `validate pipe` omit them.
+!!! note "Advisory warnings on validate"
+ Whole-bundle validate surfaces (`validate bundle`, `validate method`) also carry a `warnings` array — advisory optionality lints on a VALID bundle that never flip the verdict or the exit code. Each entry has the **same shape as a validation error item** (`category`, `error_type`, `pipe_code`, `domain_code`, `variable_names`, `message`) — this is a different shape from the `init`/`doctor` setup `warnings` (`{type, message}`) documented under Output Contract below. The first occupant is the useless-`!` lint (`optional_force_redundant`): a `!` (force) input whose slot is guaranteed present in every analyzed flow, so the assertion can never fire. In markdown, warnings render as a "Warnings" section. The array is empty when there is nothing to report; single-pipe `validate pipe` omits it (no flow context to lint in).
+
### Inputs
Generate an example inputs template for a pipe, bundle, or method.
@@ -144,6 +147,8 @@ JSON commands return the result object directly. They are not wrapped in a `stat
`RemoteConfigStale` is emitted when the gateway is enabled but the remote config service is unreachable and Pipelex falls back to its on-disk cache (offline mode).
+Do not confuse this setup-warning shape with the `warnings` array on the `validate` envelope — validate warnings are advisory lint items that reuse the validation-error item shape (see "Advisory warnings on validate" above).
+
**Error** — written to stderr:
```json
diff --git a/docs/tools/cli/run.md b/docs/tools/cli/run.md
index 8358cf4da..802829c2d 100644
--- a/docs/tools/cli/run.md
+++ b/docs/tools/cli/run.md
@@ -163,6 +163,8 @@ An inputs file is a dictionary whose keys are input variable names. Pipelex acce
There is no content sniffing: the extension alone decides. Inline JSON passed to `--inputs` (a value starting with `{`) stays JSON-only.
+An input declared optional (`?`) on the entry pipe may be omitted — the run records a `not_provided` absence for it instead of failing (see [Understanding Optionality](../../building-methods/pipes/understanding-optionality.md)).
+
Both formats produce the same input dictionary for the value types they share, so the [PipelineInputs shapes](../../building-methods/pipes/provide-inputs.md) apply to both — with one structural exception: JSON `null` has no TOML equivalent (TOML has no null type).
### JSON
@@ -207,6 +209,10 @@ field = "value"
The output JSON contains the complete working memory after pipeline execution, including all intermediate results and the final output.
+### Absent main output
+
+A run whose main output resolves as a recorded absence (an optional `?` output that produced nothing — e.g. a `PipeCondition` `continue` outcome, or a skipped producer) is a **successful** run. The CLI prints the absence with its reason, and `--save-main-stuff` writes an explicit absence artifact instead of a value dump: `main_stuff.json` is `{"absent": true, ...}` with the absence record, and `main_stuff.md` is a human-readable summary including the provenance chain (no HTML render or interactive viewer is produced — there is nothing to view). `--save-csv` fails with an explicit error and a non-zero exit code, the same way it does for any main output that is not a flat list — there is no tabular value to save.
+
## Related Documentation
- [Executing Pipelines](../../building-methods/pipes/executing-pipelines.md)
diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.md
index 9d5b56549..95d197c1f 100644
--- a/docs/under-the-hood/execution-graph-tracing.md
+++ b/docs/under-the-hood/execution-graph-tracing.md
@@ -194,6 +194,8 @@ class GraphSpec(BaseModel):
| `CONTAINS` | Parent-child containment (controller → children) |
| `SELECTED_OUTCOME` | Condition outcome selection |
+Every `EdgeSpec` also carries an `optional` boolean (default `false`). On a `DATA` edge it marks that the producer's output is declared optional (`?` presence marker) — the value flowed on this run but may be absent on others. Renderers can use it to draw the edge distinctly (e.g. dashed).
+
### Node Status
| NodeStatus | Description |
@@ -202,9 +204,11 @@ class GraphSpec(BaseModel):
| `RUNNING` | Currently executing |
| `SUCCEEDED` | Completed successfully |
| `FAILED` | Execution failed |
-| `SKIPPED` | Skipped during execution |
+| `SKIPPED` | Lifted (skipped) because a plain input resolved absent — a successful outcome, not an error |
| `CANCELED` | Canceled before completion |
+A `SKIPPED` node also carries `skip_reason` (a human-readable sentence naming the absent input, e.g. `skipped because input 'source' is absent`). A lifted pipe with a plural output still writes a real empty-list value, so its node registers that output and downstream `DATA` edges resolve normally; a lifted singular output is a recorded absence with no output spec.
+
---
## Implementation
diff --git a/mkdocs.yml b/mkdocs.yml
index d84b6140e..9e74b74f9 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -249,6 +249,7 @@ plugins:
- building-methods/pipes/working-memory.md: "Working Memory"
- building-methods/pipes/pipe-output.md: "Pipe Output"
- building-methods/pipes/understanding-multiplicity.md: "Understanding Multiplicity"
+ - building-methods/pipes/understanding-optionality.md: "Understanding Optionality"
- building-methods/pipes/pipe-operators/index.md: "Pipe Operators Overview"
- building-methods/pipes/pipe-operators/PipeLLM.md: "PipeLLM"
- building-methods/pipes/pipe-operators/PipeStructure.md: "PipeStructure"
@@ -459,6 +460,7 @@ nav:
- Pipe Output: building-methods/pipes/pipe-output.md
- CSV Input & Output: building-methods/pipes/csv-input-and-output.md
- Understanding Multiplicity: building-methods/pipes/understanding-multiplicity.md
+ - Understanding Optionality: building-methods/pipes/understanding-optionality.md
- Pipe Operators:
- Overview: building-methods/pipes/pipe-operators/index.md
- PipeLLM: building-methods/pipes/pipe-operators/PipeLLM.md
diff --git a/pipelex/builder/operations/validate_ops.py b/pipelex/builder/operations/validate_ops.py
index 9181971a3..9a81f18ac 100644
--- a/pipelex/builder/operations/validate_ops.py
+++ b/pipelex/builder/operations/validate_ops.py
@@ -13,6 +13,8 @@ from pipelex.hub import (
set_current_library,
)
from pipelex.pipeline.bundle_validator import BundleValidator
+from pipelex.pipeline.controller_taint import collect_controller_taint_analyses
+from pipelex.pipeline.optionality_warnings import build_optionality_warnings
from pipelex.pipeline.validate_bundle import build_validated_pipes, validate_bundle
if TYPE_CHECKING:
@@ -70,6 +72,8 @@ async def validate_bundle_file(
"""
result = await validate_bundle(mthds_file_path=bundle_path, library_dirs=library_dirs)
+ # warnings is the advisory lint channel (e.g. the useless-`!` lint) — same item shape as
+ # validation errors, never flips the verdict. Mirrors the agent-CLI validate_bundle_core twin.
return {
"success": True,
"is_valid": True,
@@ -78,6 +82,9 @@ async def validate_bundle_file(
"total_pipes": len(result.dry_run_result),
"pending_signatures": result.pending_signatures,
"is_runnable": not result.pending_signatures,
+ "warnings": [
+ warning.model_dump(exclude_none=True) for warning in build_optionality_warnings(collect_controller_taint_analyses(result.pipes))
+ ],
}
@@ -107,6 +114,10 @@ async def validate_bundle_content(
"total_pipes": len(validate_bundle_result.dry_run_result),
"pending_signatures": validate_bundle_result.pending_signatures,
"is_runnable": not validate_bundle_result.pending_signatures,
+ "warnings": [
+ warning.model_dump(exclude_none=True)
+ for warning in build_optionality_warnings(collect_controller_taint_analyses(validate_bundle_result.pipes))
+ ],
}
@@ -188,6 +199,8 @@ async def validate_pipe_in_bundle(
"""
result = await validate_bundle(mthds_file_path=bundle_path, library_dirs=library_dirs, dry_run_pipe_codes=[pipe_code])
+ # warnings aggregates over the whole loaded bundle (the lint's cross-flow aggregation needs
+ # every flow, not just the sliced pipe). Mirrors the agent-CLI validate_pipe_in_bundle_core twin.
return {
"success": True,
"is_valid": True,
@@ -196,4 +209,7 @@ async def validate_pipe_in_bundle(
"total_pipes": len(result.dry_run_result),
"pending_signatures": result.pending_signatures,
"is_runnable": not result.pending_signatures,
+ "warnings": [
+ warning.model_dump(exclude_none=True) for warning in build_optionality_warnings(collect_controller_taint_analyses(result.pipes))
+ ],
}
diff --git a/pipelex/builder/pipe/pipe_spec.py b/pipelex/builder/pipe/pipe_spec.py
index b11768d44..b98eec919 100644
--- a/pipelex/builder/pipe/pipe_spec.py
+++ b/pipelex/builder/pipe/pipe_spec.py
@@ -12,7 +12,7 @@ from pipelex.cogt.content_generation.dry_run_factory import MockFormat
from pipelex.core.concepts.exceptions import ConceptStringError
from pipelex.core.concepts.validation import validate_concept_ref_or_code
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint, PipeCategory, PipeType, valid_pipe_type_tags
-from pipelex.core.pipes.variable_multiplicity import MUTLIPLICITY_PATTERN, parse_concept_with_multiplicity
+from pipelex.core.pipes.variable_multiplicity import MULTIPLICITY_PATTERN, PresenceMarker, parse_concept_with_multiplicity
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.tools.misc.pretty import PrettyPrintable
from pipelex.tools.misc.string_utils import is_snake_case, normalize_to_ascii
@@ -31,10 +31,17 @@ class PipeSpec(StructuredContent):
- []: variable-length list (e.g., "Text[]")
- [N]: exactly N items (e.g., "Image[3]" for 3 images)
+ Presence Notation:
+ Singular inputs and outputs may carry a presence marker after the concept name:
+ - ?: optional (e.g., "PenaltyClause?" — the slot may legitimately hold no value)
+ - !: force (inputs only, e.g., "PenaltyClause!" — assert the value is present, error if absent)
+ Presence markers never combine with multiplicity brackets: an absent plural is the empty list.
+
Examples:
inputs = {"document": "Document", "queries": "Text[]"} # single document, multiple texts
output = "Article[]" # produces a list of articles
output = "Image[5]" # produces exactly 5 images
+ output = "PenaltyClause?" # may produce no value
"""
model_config = ConfigDict(extra="forbid")
@@ -55,17 +62,20 @@ class PipeSpec(StructuredContent):
description: str = Field(description="Natural language description of the pipe's purpose and functionality.")
inputs: dict[str, str] = Field(
description=(
- "Input specifications mapping variable names to concept codes with optional multiplicity. "
+ "Input specifications mapping variable names to concept codes with optional multiplicity or presence marker. "
"Keys: input names in snake_case. "
- "Values: ConceptCodes in PascalCase with optional brackets. "
- "Examples: 'Text' (single), 'Text[]' (variable list), 'Image[2]' (exactly 2 images), 'domain.Concept[]' (domain-qualified list)."
+ "Values: ConceptCodes in PascalCase with optional brackets, or a presence marker on singular forms. "
+ "Examples: 'Text' (single), 'Text[]' (variable list), 'Image[2]' (exactly 2 images), "
+ "'domain.Concept[]' (domain-qualified list), 'Clause?' (optional input), 'Clause!' (force: error if absent)."
),
json_schema_extra={"mock_format": MockFormat.DICT_SNAKE_KEY_PASCAL_VALUE},
)
output: str = Field(
description=(
- "Output concept code in PascalCase with optional multiplicity brackets. "
- "Examples: 'Text' (single text), 'Article[]' (list of articles), 'Image[3]' (exactly 3 images). "
+ "Output concept code in PascalCase with optional multiplicity brackets, or '?' on singular forms "
+ "to declare the pipe may produce no value ('!' is not allowed on outputs). "
+ "Examples: 'Text' (single text), 'Article[]' (list of articles), 'Image[3]' (exactly 3 images), "
+ "'PenaltyClause?' (may produce no value). "
"IMPORTANT: Always use PascalCase for the concept name."
),
json_schema_extra={"mock_format": MockFormat.PASCAL_CASE},
@@ -88,8 +98,21 @@ class PipeSpec(StructuredContent):
@field_validator("output", mode="after")
@classmethod
def validate_output(cls, output: str) -> str:
- # Extract concept without multiplicity for validation
+ # Extract concept without multiplicity/presence markers for validation
parse_result = parse_concept_with_multiplicity(output)
+ # Mirror the blueprint grammar rules (D1, D4): `!` never on outputs, `?` never with multiplicity
+ if parse_result.presence.is_force:
+ msg = (
+ f"Invalid output: '{output}'. The force marker '!' is not allowed on outputs — "
+ f"it is a use-site assertion for inputs. To declare that this pipe may produce no value, use '?'."
+ )
+ raise ValueError(msg)
+ if parse_result.presence.is_optional and parse_result.multiplicity is not None:
+ msg = (
+ f"Invalid output: '{output}'. The optional marker '?' cannot be combined with multiplicity: "
+ f"a plural slot is never absent — when nothing is found, it is the empty list."
+ )
+ raise ValueError(msg)
try:
validate_concept_ref_or_code(concept_ref_or_code=parse_result.concept_ref_or_code)
except ConceptStringError as exc:
@@ -109,17 +132,30 @@ class PipeSpec(StructuredContent):
msg = f"Invalid input name syntax '{input_name}'. Must be in snake_case."
raise ValueError(msg)
- # Validate the concept spec format with optional multiplicity brackets
- # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]
- match = re.match(MUTLIPLICITY_PATTERN, concept_spec)
+ # Validate the concept spec format: optional multiplicity brackets then optional
+ # presence marker. Pattern allows: ConceptName, domain.ConceptName, ConceptName[],
+ # ConceptName[N], ConceptName?, ConceptName!
+ match = re.match(MULTIPLICITY_PATTERN, concept_spec)
if not match:
msg = (
f"Invalid input syntax for '{input_name}': '{concept_spec}'. "
- f"Expected format: 'ConceptName', 'ConceptName[]', or 'ConceptName[N]' where N is an integer."
+ f"Expected format: 'ConceptName', 'ConceptName[]', 'ConceptName[N]' where N is an integer, "
+ f"with an optional presence marker '?' or '!' on singular forms (e.g. 'ConceptName?')."
+ )
+ raise ValueError(msg)
+
+ # Mirror the blueprint grammar rule (D1, D4): presence markers never combine with multiplicity
+ bracket_content = match.group(2)
+ presence = PresenceMarker.from_symbol(match.group(3))
+ if not presence.is_plain and bracket_content is not None:
+ msg = (
+ f"Invalid input '{input_name}': '{concept_spec}'. "
+ f"The presence marker '{presence.symbol}' cannot be combined with multiplicity: "
+ f"a plural slot is never absent — when nothing is found, it is the empty list."
)
raise ValueError(msg)
- # Extract the concept part (without multiplicity) and validate it
+ # Extract the concept part (without markers) and validate it
concept_ref_or_code = match.group(1)
try:
validate_concept_ref_or_code(concept_ref_or_code=concept_ref_or_code)
diff --git a/pipelex/cli/agent_cli/commands/run/_run_core.py b/pipelex/cli/agent_cli/commands/run/_run_core.py
index df4ca9182..6a9e8ccc7 100644
--- a/pipelex/cli/agent_cli/commands/run/_run_core.py
+++ b/pipelex/cli/agent_cli/commands/run/_run_core.py
@@ -12,6 +12,8 @@ from pipelex.base_exceptions import PipelexError
from pipelex.cli.agent_cli.commands.run._output_helpers import build_run_output
from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceRecord
+from pipelex.core.memory.absence_render import build_absence_html, build_absence_json, build_absence_markdown, build_absence_payload
from pipelex.graph.graph_factory import generate_graph_outputs, save_graph_outputs_to_dir
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipeline.runner import PipelexMTHDSProtocol
@@ -73,13 +75,26 @@ async def run_pipeline_core(
)
pipe_output = response.pipe_output
- main_stuff = pipe_output.working_memory.get_main_stuff()
- main_stuff_json: dict[str, Any] = {
- "json": await main_stuff.content.rendered_json_async(),
- "markdown": await main_stuff.content.rendered_markdown_async(),
- "html": await main_stuff.content.rendered_html_async(),
- }
- compact_result: dict[str, Any] = json.loads(main_stuff_json["json"])
+ # A completed run always resolves its declared output: a value or a recorded absence. An
+ # absent main output renders the explicit absence document on every arm of the envelope.
+ main_resolved = pipe_output.working_memory.resolve_main_stuff()
+ main_stuff_json: dict[str, Any]
+ compact_result: dict[str, Any]
+ if isinstance(main_resolved, AbsenceRecord):
+ main_stuff_json = {
+ "json": build_absence_json(main_resolved),
+ "markdown": build_absence_markdown(main_resolved),
+ "html": build_absence_html(main_resolved),
+ }
+ compact_result = build_absence_payload(main_resolved)
+ else:
+ main_stuff = main_resolved
+ main_stuff_json = {
+ "json": await main_stuff.content.rendered_json_async(),
+ "markdown": await main_stuff.content.rendered_markdown_async(),
+ "html": await main_stuff.content.rendered_html_async(),
+ }
+ compact_result = json.loads(main_stuff_json["json"])
result = build_run_output(
with_memory=with_memory,
diff --git a/pipelex/cli/agent_cli/commands/run/_run_core_api.py b/pipelex/cli/agent_cli/commands/run/_run_core_api.py
index 24db2f790..78a27e0ba 100644
--- a/pipelex/cli/agent_cli/commands/run/_run_core_api.py
+++ b/pipelex/cli/agent_cli/commands/run/_run_core_api.py
@@ -51,15 +51,18 @@ async def run_pipeline_core_api(
# extension-open RunResult — they ride model_extra, never named by the SDK.
extensions: dict[str, Any] = response.model_extra or {}
- # Extract the main stuff content from the working memory. A completed run always delivers
- # a main stuff — a response without one under the announced key is a runner contract violation.
+ # Extract the main stuff content from the working memory. A completed run always resolves its
+ # declared output: a value or a recorded absence — but the mthds SDK wire models do not carry
+ # the absences ledger yet (root + aliases only, extra=forbid), so the resolved-as-absent arm on
+ # this API path is gated on the cross-repo mthds protocol bump. Until then a response with no
+ # main stuff under the announced key is reported as a runner contract violation.
raw_main_stuff_name = extensions.get("main_stuff_name")
main_stuff_name = raw_main_stuff_name if isinstance(raw_main_stuff_name, str) else MAIN_STUFF_NAME
main_stuff = pipe_output.working_memory.root.get(main_stuff_name)
if main_stuff is None:
msg = (
f"Completed run '{response.pipeline_run_id}' response has no main stuff under key '{main_stuff_name}' — "
- "a completed run always delivers a main stuff."
+ "a completed run always resolves its declared output."
)
raise PipeExecutionError(msg)
diff --git a/pipelex/cli/agent_cli/commands/validate/_validate_core.py b/pipelex/cli/agent_cli/commands/validate/_validate_core.py
index 9bead12fe..b03696b03 100644
--- a/pipelex/cli/agent_cli/commands/validate/_validate_core.py
+++ b/pipelex/cli/agent_cli/commands/validate/_validate_core.py
@@ -14,7 +14,9 @@ from pipelex.hub import (
set_current_library,
)
from pipelex.pipeline.bundle_validator import BundleValidator
+from pipelex.pipeline.controller_taint import collect_controller_taint_analyses
from pipelex.pipeline.execution_seams import acquire_library
+from pipelex.pipeline.optionality_warnings import build_optionality_warnings
from pipelex.pipeline.validate_bundle import build_pending_signatures, build_validated_pipes, validate_bundle
if TYPE_CHECKING:
@@ -57,6 +59,9 @@ async def validate_all_core(
# is_runnable = not pending. `validate all` now makes a strict runnability claim (the consumer
# gates the exit code on it unless --allow-signatures), so both keys ride the envelope.
pending_signatures = build_pending_signatures(get_pipe_library().get_pipes_dict())
+ # warnings: same advisory lint channel as the bundle surfaces — `validate all` has the
+ # whole library loaded, which is all the flow context the cross-flow aggregation needs.
+ library_pipes = list(get_pipe_library().get_pipes_dict().values())
return {
"success": True,
"is_valid": True,
@@ -64,6 +69,9 @@ async def validate_all_core(
"total_pipes": len(dry_run_results),
"pending_signatures": pending_signatures,
"is_runnable": not pending_signatures,
+ "warnings": [
+ warning.model_dump(exclude_none=True) for warning in build_optionality_warnings(collect_controller_taint_analyses(library_pipes))
+ ],
}
finally:
# Restore the caller's outer current-library FIRST (so the guarantee survives a teardown
@@ -104,7 +112,8 @@ async def validate_bundle_core(
# Consume the real per-pipe status from the sweep — a fixed all-"SUCCESS" list would hide allowed
# failures and SKIPPED cross-package pipes the dry-run actually recorded (C-8). pending_signatures
# is the library-wide set of still-unimplemented forward declarations — a non-blocking nudge on a
- # successful lenient (--allow-signatures) run.
+ # successful lenient (--allow-signatures) run. warnings is the advisory lint channel (e.g. the
+ # useless-`!` lint) — same item shape as validation errors, never flips the verdict.
return {
"success": True,
"is_valid": True,
@@ -113,6 +122,9 @@ async def validate_bundle_core(
"total_pipes": len(result.dry_run_result),
"pending_signatures": result.pending_signatures,
"is_runnable": not result.pending_signatures,
+ "warnings": [
+ warning.model_dump(exclude_none=True) for warning in build_optionality_warnings(collect_controller_taint_analyses(result.pipes))
+ ],
}
@@ -211,7 +223,8 @@ async def validate_pipe_in_bundle_core(
# Consume the real per-pipe status from the sliced sweep — a fixed "SUCCESS" would hide an allowed
# failure or a cross-package SKIPPED the dry-run actually recorded (C-8). pending_signatures is the
- # library-wide set of still-unimplemented forward declarations.
+ # library-wide set of still-unimplemented forward declarations. warnings aggregates over the whole
+ # loaded bundle (the lint's cross-flow aggregation needs every flow, not just the sliced pipe).
return {
"success": True,
"is_valid": True,
@@ -220,4 +233,7 @@ async def validate_pipe_in_bundle_core(
"total_pipes": len(result.dry_run_result),
"pending_signatures": result.pending_signatures,
"is_runnable": not result.pending_signatures,
+ "warnings": [
+ warning.model_dump(exclude_none=True) for warning in build_optionality_warnings(collect_controller_taint_analyses(result.pipes))
+ ],
}
diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.py
index 4c88df03d..d00fe945f 100644
--- a/pipelex/cli/commands/run/_run_core.py
+++ b/pipelex/cli/commands/run/_run_core.py
@@ -24,6 +24,8 @@ from pipelex.config import get_config
from pipelex.core.concepts.exceptions import ConceptValueError
from pipelex.core.interpreter.exceptions import PipelexInterpreterError
from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.core.memory.absence import AbsenceRecord
+from pipelex.core.memory.absence_render import build_absence_json, build_absence_markdown
from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_viewer import render_stuff_viewer
@@ -212,10 +214,17 @@ async def _execute_run(
typer.secho(f"Failed to execute pipeline '{pipe_code or bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- # Pretty print main_stuff unless disabled or in dry run mode
+ # Pretty print the resolved main result unless disabled or in dry run mode. A completed run
+ # always resolves its declared output: a value or a recorded absence.
+ main_resolved = pipe_output.working_memory.resolve_main_stuff()
if not no_pretty_print and not dry_run:
- title = f"Final output of pipe [red]{pipe_code}[/red]"
- pipe_output.main_stuff.pretty_print_stuff(title=title)
+ if isinstance(main_resolved, AbsenceRecord):
+ get_console().print(
+ f"Final output of pipe [red]{pipe_code}[/red] resolved absent: {main_resolved.reason}",
+ )
+ else:
+ title = f"Final output of pipe [red]{pipe_code}[/red]"
+ main_resolved.pretty_print_stuff(title=title)
# Determine if we need an output directory
output_path: Path | None = None
@@ -246,33 +255,45 @@ async def _execute_run(
elif "reactflow" in output_type and "reactflow" not in saved_graphs:
saved_graphs.append("reactflow")
- # Save main_stuff files if enabled
+ # Save main_stuff files if enabled. An absent main output saves the explicit absence
+ # artifact (json + md, no interactive viewer — nothing to view), never value renders.
saved_main_stuff_formats: list[str] = []
if save_main_stuff and output_path:
- main_stuff = pipe_output.working_memory.get_main_stuff()
- main_stuff_json = await main_stuff.content.rendered_json_async()
- main_stuff_json_path = output_path / "main_stuff.json"
- main_stuff_json_path.write_text(main_stuff_json, encoding="utf-8")
- log.verbose(f"Main stuff JSON saved to: {main_stuff_json_path}")
- saved_main_stuff_formats.append("json")
-
- main_stuff_md = await main_stuff.content.rendered_markdown_async()
- main_stuff_md_path = output_path / "main_stuff.md"
- main_stuff_md_path.write_text(main_stuff_md, encoding="utf-8")
- log.verbose(f"Main stuff Markdown saved to: {main_stuff_md_path}")
- saved_main_stuff_formats.append("md")
-
- main_stuff_html = await main_stuff.content.rendered_html_async()
- main_stuff_html_path = output_path / "main_stuff.html"
- main_stuff_html_path.write_text(main_stuff_html, encoding="utf-8")
- log.verbose(f"Main stuff HTML saved to: {main_stuff_html_path}")
- saved_main_stuff_formats.append("html")
-
- main_stuff_viewer = await render_stuff_viewer(main_stuff)
- main_stuff_viewer_path = output_path / "main_stuff_viewer.html"
- main_stuff_viewer_path.write_text(main_stuff_viewer, encoding="utf-8")
- log.verbose(f"Main stuff HTML viewer saved to: {main_stuff_viewer_path}")
- saved_main_stuff_formats.append("html_viewer")
+ if isinstance(main_resolved, AbsenceRecord):
+ absence_json_path = output_path / "main_stuff.json"
+ absence_json_path.write_text(build_absence_json(main_resolved), encoding="utf-8")
+ log.verbose(f"Main stuff absence JSON saved to: {absence_json_path}")
+ saved_main_stuff_formats.append("json")
+
+ absence_md_path = output_path / "main_stuff.md"
+ absence_md_path.write_text(build_absence_markdown(main_resolved), encoding="utf-8")
+ log.verbose(f"Main stuff absence Markdown saved to: {absence_md_path}")
+ saved_main_stuff_formats.append("md")
+ else:
+ main_stuff = main_resolved
+ main_stuff_json = await main_stuff.content.rendered_json_async()
+ main_stuff_json_path = output_path / "main_stuff.json"
+ main_stuff_json_path.write_text(main_stuff_json, encoding="utf-8")
+ log.verbose(f"Main stuff JSON saved to: {main_stuff_json_path}")
+ saved_main_stuff_formats.append("json")
+
+ main_stuff_md = await main_stuff.content.rendered_markdown_async()
+ main_stuff_md_path = output_path / "main_stuff.md"
+ main_stuff_md_path.write_text(main_stuff_md, encoding="utf-8")
+ log.verbose(f"Main stuff Markdown saved to: {main_stuff_md_path}")
+ saved_main_stuff_formats.append("md")
+
+ main_stuff_html = await main_stuff.content.rendered_html_async()
+ main_stuff_html_path = output_path / "main_stuff.html"
+ main_stuff_html_path.write_text(main_stuff_html, encoding="utf-8")
+ log.verbose(f"Main stuff HTML saved to: {main_stuff_html_path}")
+ saved_main_stuff_formats.append("html")
+
+ main_stuff_viewer = await render_stuff_viewer(main_stuff)
+ main_stuff_viewer_path = output_path / "main_stuff_viewer.html"
+ main_stuff_viewer_path.write_text(main_stuff_viewer, encoding="utf-8")
+ log.verbose(f"Main stuff HTML viewer saved to: {main_stuff_viewer_path}")
+ saved_main_stuff_formats.append("html_viewer")
# Save working memory to JSON if enabled
working_memory_output_path: str | None = None
@@ -290,7 +311,15 @@ async def _execute_run(
# the codec rejects a non-flat row concept with a clear CsvFlatnessError.
if save_csv is not None:
# The path string and its suffix were already validated up front (fail-fast, before the run).
- csv_main_stuff = pipe_output.working_memory.get_main_stuff()
+ if isinstance(main_resolved, AbsenceRecord):
+ typer.secho(
+ f"Failed to --save-csv: the main output resolved absent ({main_resolved.reason}). "
+ "CSV output requires the pipe to produce a flat list (e.g. 'PersonSummary[]').",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
+ csv_main_stuff = main_resolved
csv_content = csv_main_stuff.content
if not isinstance(csv_content, ListContent):
typer.secho(
diff --git a/pipelex/cli/commands/validate/_validate_core.py b/pipelex/cli/commands/validate/_validate_core.py
index b3cd10479..8e1bceb84 100644
--- a/pipelex/cli/commands/validate/_validate_core.py
+++ b/pipelex/cli/commands/validate/_validate_core.py
@@ -30,8 +30,10 @@ from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.pipe_signature.signature_walk import collect_signature_refs
from pipelex.pipelex import Pipelex
from pipelex.pipeline.bundle_validator import BundleValidator
+from pipelex.pipeline.controller_taint import collect_controller_taint_analyses
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.execution_seams import load_libraries_and_activate
+from pipelex.pipeline.optionality_warnings import build_optionality_warnings
from pipelex.pipeline.validate_bundle import build_pending_signatures, validate_bundle
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventProperty
@@ -41,9 +43,21 @@ from pipelex.tools.misc.string_utils import count_with_noun
if TYPE_CHECKING:
from pathlib import Path
+ from pipelex.core.pipes.pipe_abstract import PipeAbstract
+
COMMAND = "validate"
+def _echo_optionality_warnings(pipes: list[PipeAbstract]) -> None:
+ """Render the advisory optionality lints (e.g. the useless-`!` lint) in yellow.
+
+ Advisory only — a warning never changes the validation verdict or the exit code. Requires
+ the validation library to still be loaded (the taint walk resolves child pipes via the hub).
+ """
+ for warning in build_optionality_warnings(collect_controller_taint_analyses(pipes)):
+ typer.secho(f"Warning: {warning.message}", fg=typer.colors.YELLOW)
+
+
def _format_signatures_summary_suffix(signature_count: int) -> str:
"""Return the suffix appended to lenient-mode validation summaries.
@@ -85,6 +99,10 @@ def do_validate_all_libraries_and_dry_run(
# event — sweeping the library we just loaded, without teardown.
asyncio.run(BundleValidator().validate_current_library(allow_signatures=allow_signatures))
+ # Advisory optionality lints over the whole loaded library (the lint's cross-flow
+ # aggregation needs every flow) — printed even when the signature gate below exits non-zero.
+ _echo_optionality_warnings(all_pipes)
+
# Gate-from-report (D-B consumer-decides): signatures are never a validation error, but a
# library with unsatisfied PipeSignature placeholders is valid yet NOT runnable. `validate
# --all` is strict by default — it exits non-zero on pending signatures unless
@@ -141,6 +159,9 @@ async def _validate_pipe_or_bundle(
f"Successfully validated bundle '{bundle_path}'{_format_signatures_summary_suffix(signature_count)}",
fg=typer.colors.GREEN,
)
+ # validate_bundle leaves its validation library open on success, so the taint walk
+ # behind the advisory lints can still resolve the bundle's pipes.
+ _echo_optionality_warnings(bundle_result.pipes)
except FileNotFoundError as exc:
get_console().print(Traceback())
typer.secho(
diff --git a/pipelex/core/concepts/helpers.py b/pipelex/core/concepts/helpers.py
index b2c5ab6ca..0e306c2fc 100644
--- a/pipelex/core/concepts/helpers.py
+++ b/pipelex/core/concepts/helpers.py
@@ -49,18 +49,22 @@ def get_structure_class_name_from_blueprint(
return concept_code
-def strip_multiplicity_from_concept_ref_or_code(concept_ref_or_code: str) -> str:
- """Strip multiplicity from a concept string or code.
+def strip_markers_from_concept_ref_or_code(concept_ref_or_code: str) -> str:
+ """Strip multiplicity and presence markers from a concept string or code.
+
+ Handles the full io-ref suffix grammar: an optional multiplicity suffix ("[]" or "[N]")
+ followed by an optional presence marker ("?" or "!").
Args:
- concept_ref_or_code: The concept string or code to strip multiplicity from
+ concept_ref_or_code: The concept string or code to strip markers from
Returns:
- The concept string or code without multiplicity
+ The concept string or code without multiplicity or presence markers
"""
- if "[" in concept_ref_or_code:
- return concept_ref_or_code.split("[", maxsplit=1)[0]
- return concept_ref_or_code
+ stripped = concept_ref_or_code.removesuffix("?").removesuffix("!")
+ if "[" in stripped:
+ return stripped.split("[", maxsplit=1)[0]
+ return stripped
def normalize_structure_blueprint(structure_dict: dict[str, str | ConceptStructureBlueprint]) -> dict[str, ConceptStructureBlueprint]:
diff --git a/pipelex/core/memory/absence.py b/pipelex/core/memory/absence.py
new file mode 100644
index 000000000..616f0e6c2
--- /dev/null
+++ b/pipelex/core/memory/absence.py
@@ -0,0 +1,49 @@
+from __future__ import annotations
+
+from enum import StrEnum
+
+from pydantic import BaseModel, ConfigDict, Field
+
+
+class AbsenceKind(StrEnum):
+ """How a slot came to hold no value.
+
+ - DECLARED_ABSENT: a producer with an optional (`?`) output declared it produced nothing
+ (e.g. a PipeCondition `continue` outcome, or — phase 2 — an LLM maybe-wrapper).
+ - SKIPPED: the producing pipe was lifted (skipped) because one of its plain inputs was absent.
+ - NOT_PROVIDED: the caller omitted an optional method input from the pipeline inputs.
+ """
+
+ DECLARED_ABSENT = "declared_absent"
+ SKIPPED = "skipped"
+ NOT_PROVIDED = "not_provided"
+
+
+class AbsenceRecord(BaseModel):
+ """A recorded fact that a named slot holds no value, with provenance (D2).
+
+ Absence stays what it mechanically is — no Stuff under the name — but becomes a recorded
+ fact: who produced it, why, and which upstream absence it chains to. Provenance is captured
+ at the moment absence is produced, not reconstructed at failure time.
+ """
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
+
+ variable_name: str
+ kind: AbsenceKind = Field(strict=False)
+ reason: str
+ producing_pipe: str | None = None
+ upstream: AbsenceRecord | None = None
+
+ def provenance_chain(self) -> list[AbsenceRecord]:
+ """This record followed by its upstream chain, ending at the origin absence."""
+ chain: list[AbsenceRecord] = []
+ node: AbsenceRecord | None = self
+ while node is not None:
+ chain.append(node)
+ node = node.upstream
+ return chain
+
+ def origin(self) -> AbsenceRecord:
+ """The first absence in the chain — where absence entered the flow."""
+ return self.provenance_chain()[-1]
diff --git a/pipelex/core/memory/absence_render.py b/pipelex/core/memory/absence_render.py
new file mode 100644
index 000000000..15f2e4ff0
--- /dev/null
+++ b/pipelex/core/memory/absence_render.py
@@ -0,0 +1,63 @@
+"""Shared renderers for the explicit absence artifact.
+
+A run whose declared output resolved absent is a first-class success, and every surface that
+would render the main value renders an explicit absence document instead — never an error,
+never a silently empty envelope. These helpers are the one source of that document's shape so
+the delivery executor, the CLI run cores, and telemetry cannot drift:
+
+- the JSON payload carries an ``"absent": true`` discriminator beside the record fields, so a
+ consumer can tell an absence document from a value dump;
+- the markdown is a human-readable summary with the provenance chain;
+- the HTML is the JSON payload escaped inside a ``<pre>`` block.
+"""
+
+import html
+from typing import Any
+
+from pipelex.core.memory.absence import AbsenceRecord
+from pipelex.tools.misc.json_utils import clean_json_dumps
+
+
+def build_absence_payload(absence_record: AbsenceRecord) -> dict[str, Any]:
+ """The JSON-safe absence document: ``{"absent": true, ...record fields...}``."""
+ return {"absent": True, **absence_record.model_dump()}
+
+
+def build_absence_json(absence_record: AbsenceRecord) -> str:
+ """The absence payload serialized as indented JSON (one serializer for every surface)."""
+ return clean_json_dumps(build_absence_payload(absence_record), indent=2)
+
+
+def build_absence_html(absence_record: AbsenceRecord) -> str:
+ """The HTML absence document: the JSON payload escaped inside a ``<pre>`` block.
+
+ Escaping before embedding is the same XSS-hardening as the raw-JSON value fallback —
+ record fields carry user-controlled strings.
+ """
+ return f"<pre>{html.escape(build_absence_json(absence_record))}</pre>"
+
+
+def build_absence_markdown(absence_record: AbsenceRecord) -> str:
+ """A human-readable absence summary with the provenance chain, as markdown.
+
+ Headings start at H2: the agent CLI embeds this document under its run markdown's
+ ``## Result`` section, and an H1 there would give the composed document two titles.
+ """
+ md_lines = [
+ "## Output absent",
+ "",
+ "The run completed with no value for its declared output.",
+ "",
+ f"- **Slot:** `{absence_record.variable_name}`",
+ f"- **Kind:** {absence_record.kind}",
+ f"- **Reason:** {absence_record.reason}",
+ ]
+ if absence_record.producing_pipe:
+ md_lines.append(f"- **Produced by:** pipe `{absence_record.producing_pipe}`")
+ chain = absence_record.provenance_chain()
+ if len(chain) > 1:
+ md_lines.extend(["", "### Provenance", ""])
+ for chain_index, chained_record in enumerate(chain):
+ producer_suffix = f" (pipe `{chained_record.producing_pipe}`)" if chained_record.producing_pipe else ""
+ md_lines.append(f"{chain_index + 1}. `{chained_record.variable_name}` — {chained_record.kind}: {chained_record.reason}{producer_suffix}")
+ return "\n".join(md_lines) + "\n"
diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.py
index 0aa9dee02..9206a02d9 100644
--- a/pipelex/core/memory/working_memory.py
+++ b/pipelex/core/memory/working_memory.py
@@ -6,6 +6,7 @@ from pydantic import Field, model_validator
from typing_extensions import override
from pipelex import log, pretty_print
+from pipelex.core.memory.absence import AbsenceRecord
from pipelex.core.memory.exceptions import (
WorkingMemoryConsistencyError,
WorkingMemoryStuffAttributeNotFoundError,
@@ -38,6 +39,11 @@ StuffArtefactDict = dict[str, StuffArtefact]
class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):
root: StuffDict = Field(default_factory=dict)
aliases: dict[str, str] = Field(default_factory=dict)
+ # The absence ledger (D2): recorded facts that a named slot holds no value, with provenance.
+ # A name may carry both a value and a record (D4 plural normalization note); the value wins
+ # for consumers — records are consulted only when no Stuff is present, and enumerated whole
+ # by the run report.
+ absences: dict[str, AbsenceRecord] = Field(default_factory=dict)
@model_validator(mode="after")
def validate_stuff_names(self) -> Self:
@@ -93,6 +99,74 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):
message=f"Stuff '{name}' not found in working memory, valid keys are: {self.list_keys()}",
)
+ def record_absence(self, record: AbsenceRecord) -> None:
+ """Write a ledger NOTE: the record is keyed by variable name, any value under that name
+ is left in place (and wins in `resolve_stuff`). This is the observability-note arm (e.g.
+ the D4 plural empty-list note); an absence that RESOLVES a slot goes through
+ `record_resolved_absence` instead.
+ """
+ self.absences[record.variable_name] = record
+
+ def record_resolved_absence(self, record: AbsenceRecord) -> None:
+ """Record an absence as the slot's RESOLUTION: a stale value (or alias) under the same
+ name is removed so it cannot outrank the fresh absence — the mirror of `set_stuff`'s
+ value-supersedes-record invariant.
+ """
+ self.remove_stuff(name=record.variable_name)
+ self.remove_alias(alias=record.variable_name)
+ self.record_absence(record)
+
+ def get_optional_absence(self, name: str) -> AbsenceRecord | None:
+ """Alias-aware, mirroring `get_optional_stuff`: an alias to a resolved-absent slot must
+ surface the record, not degrade to a hard miss.
+ """
+ if record := self.absences.get(name):
+ return record
+ if alias := self.aliases.get(name):
+ return self.absences.get(alias)
+ return None
+
+ def record_new_main_absence(self, record: AbsenceRecord) -> None:
+ """Record a pipe-output absence as the resolved main result.
+
+ Marks both the named slot and the main-stuff position, and removes any stale value under
+ either name so a previous output cannot masquerade as this pipe's. Other names stay
+ untouched — memory is otherwise unchanged.
+ """
+ self.remove_main_stuff()
+ self.remove_alias_to_main_stuff()
+ self.record_resolved_absence(record)
+ self.absences[MAIN_STUFF_NAME] = record
+
+ def resolve_stuff(self, name: str) -> Stuff | AbsenceRecord:
+ """Tri-state resolved accessor for post-run reads: a value or a recorded absence.
+
+ A present Stuff wins over a ledger note under the same name. A slot with neither a value
+ nor a record is a hard miss — never produced, which is a bug, not an absence.
+ """
+ if stuff := self.get_optional_stuff(name=name):
+ return stuff
+ if record := self.get_optional_absence(name=name):
+ return record
+ raise WorkingMemoryStuffNotFoundError(
+ variable_name=name,
+ message=(
+ f"Stuff '{name}' not found in working memory and no absence is recorded for it. "
+ f"Valid keys are: {self.list_keys()}; recorded absences: {list(self.absences.keys())}"
+ ),
+ )
+
+ def resolve_main_stuff(self) -> Stuff | AbsenceRecord:
+ return self.resolve_stuff(name=MAIN_STUFF_NAME)
+
+ def list_missing_names(self, names: set[str]) -> list[str]:
+ """The names (sorted) that hold neither a value nor a recorded absence — genuine misses.
+
+ The shared miss-gate check for controllers: a recorded absence is not a miss (the
+ consuming pipe's own gate applies the trichotomy — skip / run / force).
+ """
+ return [name for name in sorted(names) if self.get_optional_stuff(name=name) is None and self.get_optional_absence(name=name) is None]
+
def get_stuffs(self, names: set[str]) -> list[Stuff]:
the_stuffs: list[Stuff] = []
for name in names:
@@ -118,6 +192,8 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):
def set_stuff(self, *, name: str, stuff: Stuff):
self.root[name] = stuff
+ # A value written under a name supersedes its absence record.
+ self.absences.pop(name, None)
def is_stuff_exists(self, name: str) -> bool:
return name in self.root or name in self.aliases
@@ -145,6 +221,9 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):
def set_new_main_stuff(self, stuff: Stuff, *, name: str | None = None):
# TODO: Add unit tests for this method
+ # A real main output supersedes any positional main-stuff absence record; named-slot
+ # records for other variables stay (they remain genuinely absent).
+ self.absences.pop(MAIN_STUFF_NAME, None)
if name:
self.remove_main_stuff()
self.add_new_stuff(name=name, stuff=stuff, aliases=[MAIN_STUFF_NAME])
diff --git a/pipelex/core/pipes/exceptions.py b/pipelex/core/pipes/exceptions.py
index 65450c59e..3b39b5137 100644
--- a/pipelex/core/pipes/exceptions.py
+++ b/pipelex/core/pipes/exceptions.py
@@ -118,6 +118,25 @@ class PipeValidationErrorType(StrEnum):
MISSING_PIPE_TYPE = "missing_pipe_type"
BATCH_ITEM_NAME_COLLISION = "batch_item_name_collision"
+ # Presence-marker grammar misuse, detected at blueprint parse time: a presence marker
+ # (`?` or `!`) combined with a multiplicity suffix, or `!` on an output (D1, D4 of the
+ # optionals design).
+ OPTIONAL_MARKER_INVALID = "optional_marker_invalid"
+
+ # Static absence-taint violations (D6/D7/D11 of the optionals design): a maybe-absent slot
+ # escaping through a non-optional controller boundary; a `continue`-reachable PipeCondition
+ # without a `?` output; an unguarded template reference to a declared-optional input; a
+ # required structure field fed by a maybe-absent PipeParallel branch.
+ OPTIONAL_NOT_HANDLED = "optional_not_handled"
+ OPTIONAL_OUTPUT_REQUIRED = "optional_output_required"
+ OPTIONAL_INPUT_UNGUARDED = "optional_input_unguarded"
+ OPTIONAL_BRANCH_REQUIRED_FIELD = "optional_branch_required_field"
+
+ # Advisory-only (rides the validation report's `warnings` array, never raised as an error):
+ # a `!` (force) input whose slot is guaranteed present in every analyzed flow — the
+ # assertion can never fire.
+ OPTIONAL_FORCE_REDUNDANT = "optional_force_redundant"
+
# Wiring / reference-resolution failures, detected when validating a pipe's contract against the
# merged library (a referenced concept or dependency pipe does not resolve).
UNRESOLVED_CONCEPT = "unresolved_concept"
diff --git a/pipelex/core/pipes/inputs/exceptions.py b/pipelex/core/pipes/inputs/exceptions.py
index 6942fe2fc..4649aa66f 100644
--- a/pipelex/core/pipes/inputs/exceptions.py
+++ b/pipelex/core/pipes/inputs/exceptions.py
@@ -1,4 +1,6 @@
-from pipelex.base_exceptions import PipelexError
+from pipelex.base_exceptions import ErrorDomain, PipelexError
+from pipelex.cogt.inference.error_classification import UserAction, UserActionKind
+from pipelex.core.memory.absence import AbsenceRecord
from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
@@ -45,5 +47,75 @@ class PipeRunInputsError(PipeRunError):
super().__init__(message, run_mode, pipe_code)
+class OptionalValueAbsentError(PipeRunError):
+ """A force-marked (`!`) input was fed a recorded absence at run time (D9).
+
+ The method force-unwraps a value that this input data does not contain. Carries the
+ variable name, the consuming pipe, and the full provenance chain from the absence ledger,
+ so the report can say where the absence entered the flow and why.
+
+ ``error_domain = RUNTIME``: the failure is data-dependent — the method's assumption failed
+ on this input data, not on the caller's request shape.
+
+ ``_authors_caller_facing_message``: the message and user action describe the caller's own
+ method and data (pipe codes, variable names, ledger reasons — no server internals), and the
+ caller is the only one who can act on them; they must survive STRICT disclosure.
+ """
+
+ error_domain = ErrorDomain.RUNTIME
+ _authors_caller_facing_message = True
+
+ def __init__(
+ self,
+ message: str,
+ *,
+ run_mode: PipeRunMode,
+ pipe_code: str,
+ variable_name: str,
+ absence_record: AbsenceRecord,
+ ):
+ self.variable_name = variable_name
+ self.absence_record = absence_record
+ self.user_action = UserAction(
+ kind=UserActionKind.CHANGE_INPUT,
+ detail=(
+ f"The method force-unwraps ('!') the input '{variable_name}', which is absent for this input data. "
+ f"Provide input data that contains this value, or change the method to tolerate absence "
+ f"(declare the input optional with '?', or let the step skip by removing '!')."
+ ),
+ )
+ super().__init__(message, run_mode, pipe_code)
+
+ @classmethod
+ def make(
+ cls,
+ *,
+ run_mode: PipeRunMode,
+ pipe_code: str,
+ variable_name: str,
+ concept_ref: str,
+ absence_record: AbsenceRecord,
+ ) -> "OptionalValueAbsentError":
+ """Compose the D9 failure message from the provenance chain."""
+ origin = absence_record.origin()
+ if origin.producing_pipe:
+ origin_clause = f"pipe '{origin.producing_pipe}' produced no value — \"{origin.reason}\""
+ else:
+ origin_clause = f'"{origin.reason}"'
+ message = (
+ f"Pipe '{pipe_code}' requires '{variable_name}' to be present (declared '{concept_ref}!'), but it is absent.\n"
+ f"Absence origin: {origin_clause}\n"
+ f"Fix in the method: declare the input '{concept_ref}?' and guard its uses, or let the step skip "
+ f"by removing '!'. Fix in the data: provide input data that contains this value."
+ )
+ return cls(
+ message,
+ run_mode=run_mode,
+ pipe_code=pipe_code,
+ variable_name=variable_name,
+ absence_record=absence_record,
+ )
+
+
class NoInputsRequiredError(Exception):
"""Raised when a pipe has no inputs."""
diff --git a/pipelex/core/pipes/inputs/input_stuff_specs.py b/pipelex/core/pipes/inputs/input_stuff_specs.py
index ef08d77af..7fec305f4 100644
--- a/pipelex/core/pipes/inputs/input_stuff_specs.py
+++ b/pipelex/core/pipes/inputs/input_stuff_specs.py
@@ -9,7 +9,7 @@ from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_representation_generator import ConceptRepresentationFormat
from pipelex.core.pipes.inputs.exceptions import InputStuffSpecNotFoundError
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
-from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker, VariableMultiplicity
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
@@ -60,7 +60,11 @@ class InputStuffSpecs(RootModel[PipeInputsRoot]):
log.verbose(
f"Variable {transformed_key} already exists with a different concept code: {transformed_dict[transformed_key]} -> {stuff_spec}",
)
- transformed_dict[transformed_key] = StuffSpec(concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ transformed_dict[transformed_key] = StuffSpec(
+ concept=stuff_spec.concept,
+ multiplicity=stuff_spec.multiplicity,
+ presence=stuff_spec.presence,
+ )
return transformed_dict
@@ -81,8 +85,15 @@ class InputStuffSpecs(RootModel[PipeInputsRoot]):
def is_variable_existing(self, variable_name: str) -> bool:
return variable_name in self.root
- def add_stuff_spec(self, variable_name: str, *, concept: Concept, multiplicity: VariableMultiplicity | None = None):
- self.root[variable_name] = StuffSpec(concept=concept, multiplicity=multiplicity)
+ def add_stuff_spec(
+ self,
+ variable_name: str,
+ *,
+ concept: Concept,
+ multiplicity: VariableMultiplicity | None = None,
+ presence: PresenceMarker = PresenceMarker.PLAIN,
+ ):
+ self.root[variable_name] = StuffSpec(concept=concept, multiplicity=multiplicity, presence=presence)
@property
def items(self) -> list[tuple[str, StuffSpec]]:
@@ -106,10 +117,24 @@ class InputStuffSpecs(RootModel[PipeInputsRoot]):
def variables(self) -> list[str]:
return list(self.root.keys())
+ @property
+ def declared_names(self) -> list[str]:
+ """Every declared input name, regardless of presence marker."""
+ the_declared_names: list[str] = []
+ for requirement_expression in self.root:
+ declared_variable_name = get_root_from_dotted_path(requirement_expression)
+ the_declared_names.append(declared_variable_name)
+ return the_declared_names
+
@property
def required_names(self) -> list[str]:
+ """Declared input names whose value is required at run time: plain and forced (`!`)
+ inputs. Optional (`?`) inputs are declared but may legitimately be absent.
+ """
the_required_names: list[str] = []
- for requirement_expression in self.root:
+ for requirement_expression, stuff_spec in self.root.items():
+ if stuff_spec.presence.is_optional:
+ continue
required_variable_name = get_root_from_dotted_path(requirement_expression)
the_required_names.append(required_variable_name)
return the_required_names
@@ -125,6 +150,7 @@ class InputStuffSpecs(RootModel[PipeInputsRoot]):
requirement_expression=requirement_expression,
concept=stuff_spec.concept,
multiplicity=stuff_spec.multiplicity,
+ presence=stuff_spec.presence,
),
)
return the_named_stuff_spec
diff --git a/pipelex/core/pipes/inputs/input_stuff_specs_factory.py b/pipelex/core/pipes/inputs/input_stuff_specs_factory.py
index 614f5ecd3..bd717f639 100644
--- a/pipelex/core/pipes/inputs/input_stuff_specs_factory.py
+++ b/pipelex/core/pipes/inputs/input_stuff_specs_factory.py
@@ -7,6 +7,7 @@ from pipelex.core.concepts.validation import validate_concept_ref_or_code
from pipelex.core.pipes.inputs.exceptions import InputStuffSpecsFactoryError
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs, PipeInputsRoot
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker
from pipelex.hub import get_required_concept
if TYPE_CHECKING:
@@ -62,10 +63,11 @@ class InputStuffSpecsFactory:
Raises:
InputStuffSpecsFactoryError: If the stuff spec string format is invalid
"""
- # Pattern to match concept string and optional multiplicity brackets
+ # Pattern to match concept string, optional multiplicity brackets, and optional presence marker
# Group 1: concept string (everything before brackets)
# Group 2: content inside brackets (empty string for [], digits for [5])
- pattern = r"^(.+?)(?:\[(\d*)\])?$"
+ # Group 3: presence marker symbol (None, "?", or "!")
+ pattern = r"^(.+?)(?:\[(\d*)\])?([?!])?$"
match = re.match(pattern, stuff_spec_str)
if not match:
@@ -74,6 +76,7 @@ class InputStuffSpecsFactory:
concept_ref_or_code = match.group(1)
multiplicity_str = match.group(2)
+ marker_symbol = match.group(3)
# Validate and resolve concept string with domain
try:
@@ -98,4 +101,4 @@ class InputStuffSpecsFactory:
# else: No brackets, multiplicity stays None
concept = get_required_concept(concept_ref=concept_ref_with_domain)
- return StuffSpec(concept=concept, multiplicity=multiplicity)
+ return StuffSpec(concept=concept, multiplicity=multiplicity, presence=PresenceMarker.from_symbol(marker_symbol))
diff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.py
index a7c12cb56..1c4119438 100644
--- a/pipelex/core/pipes/pipe_abstract.py
+++ b/pipelex/core/pipes/pipe_abstract.py
@@ -1,8 +1,9 @@
import traceback
from abc import ABC, abstractmethod
from datetime import UTC, datetime
-from typing import TYPE_CHECKING, Any, Self, final
+from typing import TYPE_CHECKING, Any, NamedTuple, Self, final
+import shortuuid
from opentelemetry import trace
from opentelemetry.trace import NonRecordingSpan, Span, SpanContext, SpanKind, Status, StatusCode, TraceFlags
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
@@ -10,18 +11,23 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator, model_valida
from pipelex import log
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.native.concept_native import NativeConceptCode
-from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory
from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
-from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
-from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
+from pipelex.core.pipes.inputs.exceptions import OptionalValueAbsentError, PipeRunInputsError
+from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs, NamedStuffSpec
from pipelex.core.pipes.pipe_blueprint import PipeCategory, PipeType, valid_pipe_type_tags
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
from pipelex.core.pipes.validation import is_variable_satisfied_by_inputs
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.graph.graph_tracer_manager import GraphTracerManager, IOSpec, NodeKind
from pipelex.libraries.library_crate import LibraryCrate
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_apply
from pipelex.pipe_signature.exceptions import PipeSignatureNotExecutableError
from pipelex.pipeline.job_metadata import JobMetadata, OtelContext
from pipelex.pipeline.pipeline_factory import PipelineFactory
@@ -43,6 +49,41 @@ if TYPE_CHECKING:
PipeAbstractType = type["PipeAbstract"]
+class AbsentInput(NamedTuple):
+ """One needed input that is absent from working memory, with its ledger record when one exists."""
+
+ named_stuff_spec: NamedStuffSpec
+ absence_record: AbsenceRecord
+
+
+class CompanionSlot(NamedTuple):
+ """One extra slot a pipe would have written into working memory besides its main output
+ (e.g. an `add_each_output` parallel's branch result slots). When the pipe is lifted, each
+ companion slot must be resolved too — a recorded absence for a singular slot, an empty
+ list for a plural one (D4) — or downstream consumers would hit a hard neither-value-nor-record miss.
+ """
+
+ slot_name: str
+ concept: Concept
+ is_plural: bool
+ producing_pipe_code: str
+
+
+class InputPresenceScan(NamedTuple):
+ """The runtime trichotomy (D3) applied to a pipe's needed inputs against working memory.
+
+ - ``missing_names``: absent with NO absence record and not optional — never produced, a hard error.
+ - ``forced_absent``: `!` inputs fed a recorded absence — typed failure with provenance.
+ - ``liftable``: plain inputs fed a recorded absence — the pipe is lifted (skipped).
+
+ Absent optional (`?`) inputs appear nowhere: the pipe runs and handles absence itself.
+ """
+
+ missing_names: list[str]
+ forced_absent: list[AbsentInput]
+ liftable: list[AbsentInput]
+
+
class PipeAbstract(ABC, BaseModel):
model_config = ConfigDict(strict=True, extra="forbid")
@@ -261,10 +302,13 @@ class PipeAbstract(ABC, BaseModel):
declared_stuff_spec = self.inputs.root[var_name]
needed_stuff_spec = the_needed_inputs.root[named_stuff_spec.requirement_expression or var_name]
- # Allow mismatch if the needed stuff_spec is a flexible type (Dynamic or Anything)
- if (
- needed_stuff_spec.concept.code not in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}
- and declared_stuff_spec != needed_stuff_spec
+ # Allow mismatch if the needed stuff_spec is a flexible type (Dynamic or Anything).
+ # Presence markers are deliberately NOT compared: a controller's boundary marker may
+ # legitimately differ from a child's need (e.g. a sequence declares `X?` while a step
+ # needs `X` plain — that is exactly the lift-skip case, D3). Only concept and
+ # multiplicity define the spec contract here.
+ if needed_stuff_spec.concept.code not in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING} and (
+ declared_stuff_spec.concept != needed_stuff_spec.concept or declared_stuff_spec.multiplicity != needed_stuff_spec.multiplicity
):
# Identify the specific mismatched field(s)
mismatch_details: list[str] = []
@@ -292,7 +336,7 @@ class PipeAbstract(ABC, BaseModel):
# Check that all declared inputs are actually needed
for input_name in self.inputs.variables:
- if input_name not in the_needed_inputs.required_names:
+ if input_name not in the_needed_inputs.declared_names:
msg = f"Extraneous input '{input_name}' found in the inputs of pipe {self.code}"
raise PipeValidationError(
message=msg,
@@ -324,6 +368,38 @@ class PipeAbstract(ABC, BaseModel):
def validate_output_static(self):
pass
+ def _scan_input_presence(self, *, working_memory: WorkingMemory) -> InputPresenceScan:
+ """Apply the runtime trichotomy (D3) to this pipe's needed inputs.
+
+ The presence marker is read from this pipe's OWN input declaration when it exists —
+ that is the boundary contract (D5: boundaries explicit). A controller's aggregated
+ ``needed_inputs()`` carries the children's markers, and e.g. a sequence declaring
+ ``X?`` over a step that needs ``X`` plain must run (its steps lift), not lift wholesale.
+ """
+ missing_names: list[str] = []
+ forced_absent: list[AbsentInput] = []
+ liftable: list[AbsentInput] = []
+ for named_stuff_spec in self.needed_inputs().named_stuff_specs:
+ if working_memory.get_optional_stuff(named_stuff_spec.variable_name):
+ continue
+ absence_record = working_memory.get_optional_absence(named_stuff_spec.variable_name)
+ declared_stuff_spec = self.inputs.root.get(named_stuff_spec.variable_name)
+ presence = declared_stuff_spec.presence if declared_stuff_spec else named_stuff_spec.presence
+ match presence:
+ case PresenceMarker.OPTIONAL:
+ continue
+ case PresenceMarker.FORCE:
+ if absence_record:
+ forced_absent.append(AbsentInput(named_stuff_spec=named_stuff_spec, absence_record=absence_record))
+ else:
+ missing_names.append(named_stuff_spec.variable_name)
+ case PresenceMarker.PLAIN:
+ if absence_record:
+ liftable.append(AbsentInput(named_stuff_spec=named_stuff_spec, absence_record=absence_record))
+ else:
+ missing_names.append(named_stuff_spec.variable_name)
+ return InputPresenceScan(missing_names=missing_names, forced_absent=forced_absent, liftable=liftable)
+
@final
async def validate_before_run(
self,
@@ -332,28 +408,46 @@ class PipeAbstract(ABC, BaseModel):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
- ):
+ ) -> InputPresenceScan:
# A PipeSignature has no implementation: reject live execution before the input
# checks below, so callers get the actionable error, not a misleading "missing inputs".
if self.is_signature and pipe_run_params.run_mode.is_live:
raise PipeSignatureNotExecutableError(pipe_ref=self.pipe_ref)
- # Check that all the needed inputs are actually in the working memory
- missing_input_names: list[str] = []
+ # The runtime trichotomy (D3): a recorded absence on a `!` input is a typed failure with
+ # provenance; absent with no record (and not optional) is a hard miss — the bug case.
+ presence_scan = self._scan_input_presence(working_memory=working_memory)
- for named_stuff_spec in self.needed_inputs().named_stuff_specs:
- if not working_memory.get_optional_stuff(named_stuff_spec.variable_name):
- missing_input_names.append(named_stuff_spec.variable_name)
+ if presence_scan.forced_absent:
+ forced = presence_scan.forced_absent[0]
+ raise OptionalValueAbsentError.make(
+ run_mode=pipe_run_params.run_mode,
+ pipe_code=self.code,
+ variable_name=forced.named_stuff_spec.variable_name,
+ concept_ref=forced.named_stuff_spec.concept.concept_ref,
+ absence_record=forced.absence_record,
+ )
- if missing_input_names:
- msg = f"Dry run of {self.type} '{self.code}': missing required inputs: {', '.join(missing_input_names)}"
+ if presence_scan.missing_names:
+ run_label = "Dry run" if pipe_run_params.run_mode.is_dry else "Live run"
+ msg = f"{run_label} of {self.type} '{self.code}': missing required inputs: {', '.join(presence_scan.missing_names)}."
+ optional_input_names = [
+ named_stuff_spec.variable_name for named_stuff_spec in self.needed_inputs().named_stuff_specs if named_stuff_spec.presence.is_optional
+ ]
+ if optional_input_names:
+ msg += f" These optional inputs may be omitted: {', '.join(optional_input_names)}."
raise PipeRunInputsError(
message=msg,
run_mode=pipe_run_params.run_mode,
pipe_code=self.code,
- missing_inputs=missing_input_names,
+ missing_inputs=presence_scan.missing_names,
)
+ # The pipe is about to be lifted (skipped): per-pipe validation and resource checks are
+ # moot — they would inspect inputs the pipe will never consume.
+ if presence_scan.liftable:
+ return presence_scan
+
# Validate external resources (URLs, file paths) referenced by input contents.
# Skipped in dry-run mode because inputs are mock-generated with fake URLs.
if not pipe_run_params.run_mode.is_dry:
@@ -376,6 +470,7 @@ class PipeAbstract(ABC, BaseModel):
await self._validate_before_run(
job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
)
+ return presence_scan
@abstractmethod
async def _validate_before_run(
@@ -504,7 +599,7 @@ class PipeAbstract(ABC, BaseModel):
# Capture input specs from working memory for data flow tracking
input_specs: list[IOSpec] = []
- for var_name in self.needed_inputs().required_names:
+ for var_name in self.needed_inputs().declared_names:
stuff = working_memory.get_optional_stuff(var_name)
if stuff is not None:
# E1: gate the expensive payload serialization on emit_graph_events too — in
@@ -556,29 +651,40 @@ class PipeAbstract(ABC, BaseModel):
trace_context=child_trace_context,
)
try:
- await self.validate_before_run(
- job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
- )
- match pipe_run_params.run_mode:
- case PipeRunMode.LIVE:
- pipe_output = await self.live_run_pipe(
- job_metadata=job_metadata,
- working_memory=working_memory,
- pipe_run_params=pipe_run_params,
- output_name=output_name,
- library_crate=library_crate,
- )
- case PipeRunMode.DRY:
- pipe_output = await self.dry_run_pipe(
- job_metadata=job_metadata,
- working_memory=working_memory,
- pipe_run_params=pipe_run_params,
- output_name=output_name,
- library_crate=library_crate,
- )
- await self.validate_after_run(
+ presence_scan = await self.validate_before_run(
job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
)
+ if presence_scan.liftable:
+ # Implicit lifting (D3): a plain input fed a recorded absence skips the pipe;
+ # its output is recorded absent with provenance chaining to the input's record.
+ pipe_output = self._make_lifted_output(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ liftable=presence_scan.liftable,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ )
+ else:
+ match pipe_run_params.run_mode:
+ case PipeRunMode.LIVE:
+ pipe_output = await self.live_run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
+ )
+ case PipeRunMode.DRY:
+ pipe_output = await self.dry_run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
+ )
+ await self.validate_after_run(
+ job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ )
except Exception as exc:
# Broad catch is intentional: graph tracing must record EVERY failure mode,
# including unexpected ones — an untraced failure is an observability hole.
@@ -600,41 +706,163 @@ class PipeAbstract(ABC, BaseModel):
)
raise
- # Record graph tracing success
+ # Record graph tracing completion — a lifted (skipped) pipe ends its node in the distinct
+ # `skipped` state with the skip reason; every other completion is a success.
if tracer_manager is not None and parent_trace_context is not None:
- # Capture output spec for data flow tracking — a completed pipe run always delivers a main stuff.
- main_stuff = pipe_output.working_memory.get_main_stuff()
- # E1: same gating as the input block — skip the discarded payload dumps in costs-only mode.
- include_graph_data = parent_trace_context.emit_graph_events
- output_spec = IOSpec(
- name=output_name or main_stuff.stuff_name or "main_stuff",
- concept=main_stuff.concept.code,
- content_type=main_stuff.content.content_type,
- digest=main_stuff.stuff_code,
- data=main_stuff.content.smart_dump() if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content) else None,
- data_text=main_stuff.content.rendered_pretty_text()
- if (include_graph_data and parent_trace_context.data_inclusion.stuff_text_content)
- else None,
- data_html=main_stuff.content.rendered_pretty_html()
- if (include_graph_data and parent_trace_context.data_inclusion.stuff_html_content)
- else None,
- )
-
- # Serialize output concept for registry if enabled (E1: also gated on emit_graph_events).
+ # Capture output spec for data flow tracking — a completed pipe run always resolves its
+ # declared output: a value or a recorded absence. An absent output has no payload to
+ # capture. A LIFTED pipe with a PLURAL output still wrote a real empty-list Stuff (D4)
+ # that downstream pipes consume, so its spec must register in the producer map exactly
+ # like a produced value — otherwise the consumers' DATA edges silently drop.
+ main_resolved = pipe_output.working_memory.resolve_main_stuff()
+ output_spec: IOSpec | None = None
output_concept_data: dict[str, Any] | None = None
- if parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry:
- output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept, library_crate=library_crate)
-
- tracer_manager.on_pipe_end_success(
- lookup_key=parent_trace_context.lookup_key,
- node_id=graph_node_id,
- ended_at=datetime.now(UTC),
- output_spec=output_spec,
- output_concept_data=output_concept_data,
- )
+ if isinstance(main_resolved, Stuff):
+ main_stuff = main_resolved
+ # E1: same gating as the input block — skip the discarded payload dumps in costs-only mode.
+ include_graph_data = parent_trace_context.emit_graph_events
+ output_spec = IOSpec(
+ name=output_name or main_stuff.stuff_name or "main_stuff",
+ concept=main_stuff.concept.code,
+ content_type=main_stuff.content.content_type,
+ digest=main_stuff.stuff_code,
+ data=main_stuff.content.smart_dump() if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content) else None,
+ data_text=main_stuff.content.rendered_pretty_text()
+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_text_content)
+ else None,
+ data_html=main_stuff.content.rendered_pretty_html()
+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_html_content)
+ else None,
+ # The optional-edge marker (D8): a data edge fed by this output reports that the
+ # value may be absent in other runs.
+ extra={"optional": True} if self.output.presence.is_optional else {},
+ )
+
+ # Serialize output concept for registry if enabled (E1: also gated on emit_graph_events).
+ if parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry:
+ output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept, library_crate=library_crate)
+
+ if presence_scan.liftable:
+ tracer_manager.on_pipe_end_skipped(
+ lookup_key=parent_trace_context.lookup_key,
+ node_id=graph_node_id,
+ ended_at=datetime.now(UTC),
+ skip_reason=self._make_skip_reason(liftable=presence_scan.liftable),
+ output_spec=output_spec,
+ output_concept_data=output_concept_data,
+ )
+ # Plural companion slots also wrote real empty-list Stuffs on the lift — register
+ # them as this node's outputs so their downstream DATA edges resolve too.
+ # (graph_node_id is set whenever tracing recorded the start; guard for the type.)
+ if graph_node_id is not None:
+ for companion_slot in self.lifted_companion_slots():
+ if not companion_slot.is_plural:
+ continue
+ companion_stuff = pipe_output.working_memory.get_optional_stuff(companion_slot.slot_name)
+ if companion_stuff is None:
+ continue
+ tracer_manager.register_controller_output(
+ lookup_key=parent_trace_context.lookup_key,
+ node_id=graph_node_id,
+ output_spec=IOSpec(
+ name=companion_slot.slot_name,
+ concept=companion_stuff.concept.code,
+ content_type=companion_stuff.content.content_type,
+ digest=companion_stuff.stuff_code,
+ ),
+ )
+ else:
+ tracer_manager.on_pipe_end_success(
+ lookup_key=parent_trace_context.lookup_key,
+ node_id=graph_node_id,
+ ended_at=datetime.now(UTC),
+ output_spec=output_spec,
+ output_concept_data=output_concept_data,
+ )
return pipe_output
+ def lifted_companion_slots(self) -> list[CompanionSlot]:
+ """Extra slots this pipe would have written besides its main output, to resolve when it
+ is lifted. Default: none; an `add_each_output` PipeParallel reports its branch slots.
+ """
+ return []
+
+ @classmethod
+ def _make_skip_reason(cls, *, liftable: list[AbsentInput]) -> str:
+ """The one skip-reason wording, shared by the absence record and the graph node."""
+ return f"skipped because input '{liftable[0].named_stuff_spec.variable_name}' is absent"
+
+ def _make_lifted_output(
+ self,
+ job_metadata: JobMetadata,
+ *,
+ working_memory: WorkingMemory,
+ liftable: list[AbsentInput],
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ ) -> PipeOutput:
+ """Skip this pipe (implicit lifting, D3) and record its output as absent with provenance.
+
+ A plural output does not go absent: it normalizes to an empty list (D4) — `[]`-emptiness
+ is the absence story for plurals — with a ledger note kept for observability; taint stops.
+ Plurality is resolved like the run path resolves it (declared multiplicity + the
+ invocation-level override), matching what the static taint pass promised downstream.
+ Companion slots the pipe would also have written (`lifted_companion_slots`) are resolved
+ the same way, so no downstream consumer meets a neither-value-nor-record hard miss.
+ """
+ lifted_input = liftable[0]
+ absent_names = ", ".join(absent.named_stuff_spec.variable_name for absent in liftable)
+ output_slot_name = output_name or MAIN_STUFF_NAME
+ log.info(f"Skipping {self.type} '{self.code}': absent input(s): {absent_names}")
+
+ skip_reason = self._make_skip_reason(liftable=liftable)
+ skip_record = AbsenceRecord(
+ variable_name=output_slot_name,
+ kind=AbsenceKind.SKIPPED,
+ reason=skip_reason,
+ producing_pipe=self.code,
+ upstream=lifted_input.absence_record,
+ )
+ multiplicity_resolution = output_multiplicity_to_apply(
+ base_multiplicity=self.output.multiplicity,
+ override_multiplicity=pipe_run_params.output_multiplicity,
+ )
+ if multiplicity_resolution.is_multiple_outputs_enabled:
+ empty_list_stuff = Stuff(
+ concept=self.output.concept,
+ content=ListContent[StuffContent](items=[]),
+ stuff_name=output_slot_name,
+ stuff_code=shortuuid.uuid()[:5],
+ )
+ working_memory.set_new_main_stuff(empty_list_stuff, name=output_name)
+ # Observability note only — the empty list is the value consumers see.
+ working_memory.record_absence(skip_record)
+ else:
+ working_memory.record_new_main_absence(skip_record)
+
+ for companion_slot in self.lifted_companion_slots():
+ companion_record = AbsenceRecord(
+ variable_name=companion_slot.slot_name,
+ kind=AbsenceKind.SKIPPED,
+ reason=skip_reason,
+ producing_pipe=companion_slot.producing_pipe_code,
+ upstream=lifted_input.absence_record,
+ )
+ if companion_slot.is_plural:
+ empty_companion_stuff = Stuff(
+ concept=companion_slot.concept,
+ content=ListContent[StuffContent](items=[]),
+ stuff_name=companion_slot.slot_name,
+ stuff_code=shortuuid.uuid()[:5],
+ )
+ working_memory.set_stuff(name=companion_slot.slot_name, stuff=empty_companion_stuff)
+ working_memory.record_absence(companion_record)
+ else:
+ working_memory.record_resolved_absence(companion_record)
+
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
+
@final
async def live_run_pipe(
self,
@@ -819,7 +1047,7 @@ class PipeAbstract(ABC, BaseModel):
span_attributes[LangfuseSpanAttr.OBSERVATION_DESCRIPTION] = self.description
# Capture full input content for Langfuse
- needed_input_names = set(self.needed_inputs().required_names)
+ needed_input_names = set(self.needed_inputs().declared_names)
inputs_json = OtelFactory.make_inputs_json(
working_memory=working_memory,
needed_input_names=needed_input_names,
diff --git a/pipelex/core/pipes/pipe_blueprint.py b/pipelex/core/pipes/pipe_blueprint.py
index f2cc4a23d..705661c14 100644
--- a/pipelex/core/pipes/pipe_blueprint.py
+++ b/pipelex/core/pipes/pipe_blueprint.py
@@ -7,8 +7,14 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator, model_valida
from pipelex.core.concepts.exceptions import ConceptStringError
from pipelex.core.concepts.validation import validate_concept_ref_or_code
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
from pipelex.core.pipes.validation import validate_input_name
-from pipelex.core.pipes.variable_multiplicity import MUTLIPLICITY_PATTERN, PipeVariableMultiplicityError, parse_concept_with_multiplicity
+from pipelex.core.pipes.variable_multiplicity import (
+ MULTIPLICITY_PATTERN,
+ PipeVariableMultiplicityError,
+ PresenceMarker,
+ parse_concept_with_multiplicity,
+)
# A signature is NOT an executable pipe kind: it is deliberately absent from `PipeType` and
# `PipeCategory`. This is the one extra `type` tag the parse-time allowlists admit beyond the
@@ -277,17 +283,35 @@ class PipeBlueprint(ABC, BaseModel):
for input_name, concept_spec in self.inputs.items():
validate_input_name(input_name)
- # Validate the concept spec format with optional multiplicity brackets
- # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]
- match = re.match(MUTLIPLICITY_PATTERN, concept_spec)
+ # Validate the concept spec format: optional multiplicity brackets then optional
+ # presence marker. Pattern allows: ConceptName, domain.ConceptName, ConceptName[],
+ # ConceptName[N], ConceptName?, ConceptName!
+ match = re.match(MULTIPLICITY_PATTERN, concept_spec)
if not match:
msg = (
f"Invalid input syntax for '{input_name}': '{concept_spec}'. "
- f"Expected format: 'ConceptName', 'ConceptName[]', or 'ConceptName[N]' where N is an integer."
+ f"Expected format: 'ConceptName', 'ConceptName[]', 'ConceptName[N]' where N is an integer, "
+ f"with an optional presence marker '?' or '!' on singular forms (e.g. 'ConceptName?')."
)
raise ValueError(msg)
- # Extract the concept part (without multiplicity) and validate it
+ # D1/D4: presence markers are mutually exclusive with multiplicity — a plural slot's
+ # "nothing" is the empty list, so there is nothing for `?` or `!` to say.
+ bracket_content = match.group(2)
+ presence = PresenceMarker.from_symbol(match.group(3))
+ if not presence.is_plain and bracket_content is not None:
+ msg = (
+ f"Invalid input '{input_name}': '{concept_spec}'. "
+ f"The presence marker '{presence.symbol}' cannot be combined with multiplicity: "
+ f"a plural slot is never absent — when nothing is found, it is the empty list."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_MARKER_INVALID,
+ variable_names=[input_name],
+ )
+
+ # Extract the concept part (without markers) and validate it
concept_ref_or_code = match.group(1)
try:
validate_concept_ref_or_code(concept_ref_or_code=concept_ref_or_code)
@@ -299,12 +323,34 @@ class PipeBlueprint(ABC, BaseModel):
@final
def generic_validate_output(self):
- # Strip multiplicity brackets before validating
+ # Strip multiplicity brackets and presence marker before validating
try:
output_parse_result = parse_concept_with_multiplicity(self.output)
except PipeVariableMultiplicityError as exc:
msg = f"Invalid concept specification syntax: '{self.output}'. {exc}"
raise ValueError(msg) from exc
+
+ # D1: `!` is a use-site assertion — it is meaningless on outputs (a producer doesn't unwrap).
+ if output_parse_result.presence.is_force:
+ msg = (
+ f"Invalid output: '{self.output}'. The force marker '!' is not allowed on outputs — "
+ f"it is a use-site assertion for inputs. To declare that this pipe may produce no value, use '?'."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_MARKER_INVALID,
+ )
+ # D4: `?` is mutually exclusive with multiplicity — an absent plural normalizes to the empty list.
+ if output_parse_result.presence.is_optional and output_parse_result.multiplicity is not None:
+ msg = (
+ f"Invalid output: '{self.output}'. The optional marker '?' cannot be combined with multiplicity: "
+ f"a plural slot is never absent — when nothing is found, it is the empty list."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_MARKER_INVALID,
+ )
+
try:
validate_concept_ref_or_code(concept_ref_or_code=output_parse_result.concept_ref_or_code)
except ConceptStringError as exc:
diff --git a/pipelex/core/pipes/pipe_factory.py b/pipelex/core/pipes/pipe_factory.py
index 28e9083dc..91236061c 100644
--- a/pipelex/core/pipes/pipe_factory.py
+++ b/pipelex/core/pipes/pipe_factory.py
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any, Generic, Protocol, TypeVar
from kajson.exceptions import ClassRegistryInheritanceError, ClassRegistryNotFoundError
from typing_extensions import runtime_checkable
-from pipelex.core.concepts.helpers import strip_multiplicity_from_concept_ref_or_code
+from pipelex.core.concepts.helpers import strip_markers_from_concept_ref_or_code
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.pipes.exceptions import PipeFactoryError, PipeFactoryErrorType
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
@@ -59,7 +59,7 @@ class PipeFactory(Generic[PipeAbstractType]):
# Validate that the specified concepts are declared in the bundle, or are natives concepts.
if blueprint.inputs is not None:
for input_name, input_concept_ref_or_code in blueprint.inputs.items():
- stripped_input_concept_ref_or_code = strip_multiplicity_from_concept_ref_or_code(concept_ref_or_code=input_concept_ref_or_code)
+ stripped_input_concept_ref_or_code = strip_markers_from_concept_ref_or_code(concept_ref_or_code=input_concept_ref_or_code)
if "." not in stripped_input_concept_ref_or_code:
if (
not NativeConceptCode.is_native_concept_ref_or_code(concept_ref_or_code=stripped_input_concept_ref_or_code)
@@ -74,7 +74,7 @@ class PipeFactory(Generic[PipeAbstractType]):
raise PipeFactoryError(msg)
if "." not in blueprint.output:
- stripped_output_concept_ref_or_code = strip_multiplicity_from_concept_ref_or_code(concept_ref_or_code=blueprint.output)
+ stripped_output_concept_ref_or_code = strip_markers_from_concept_ref_or_code(concept_ref_or_code=blueprint.output)
if (
not NativeConceptCode.is_native_concept_ref_or_code(concept_ref_or_code=stripped_output_concept_ref_or_code)
and stripped_output_concept_ref_or_code not in concept_codes_from_the_same_domain
diff --git a/pipelex/core/pipes/stuff_spec/stuff_spec.py b/pipelex/core/pipes/stuff_spec/stuff_spec.py
index 9ee4b96fb..2f036273a 100644
--- a/pipelex/core/pipes/stuff_spec/stuff_spec.py
+++ b/pipelex/core/pipes/stuff_spec/stuff_spec.py
@@ -4,12 +4,13 @@ from pydantic import BaseModel
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_representation_generator import ConceptRepresentationFormat
-from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker, VariableMultiplicity, format_concept_with_multiplicity
class StuffSpec(BaseModel):
concept: Concept
multiplicity: VariableMultiplicity | None = None
+ presence: PresenceMarker = PresenceMarker.PLAIN
def is_multiple(self) -> bool:
if self.multiplicity is None:
@@ -19,11 +20,11 @@ class StuffSpec(BaseModel):
return self.multiplicity > 1
def to_bundle_representation(self) -> str:
- if self.multiplicity is None:
- return self.concept.concept_ref
- if isinstance(self.multiplicity, bool):
- return f"{self.concept.concept_ref}[]"
- return f"{self.concept.concept_ref}[{self.multiplicity}]"
+ return format_concept_with_multiplicity(
+ self.concept.concept_ref,
+ multiplicity=self.multiplicity,
+ presence=self.presence,
+ )
def render_stuff_spec(self, output_format: ConceptRepresentationFormat) -> dict[str, Any]:
"""Render a representation of this stuff spec.
diff --git a/pipelex/core/pipes/stuff_spec/stuff_spec_factory.py b/pipelex/core/pipes/stuff_spec/stuff_spec_factory.py
index 0e5f48a47..f30a88176 100644
--- a/pipelex/core/pipes/stuff_spec/stuff_spec_factory.py
+++ b/pipelex/core/pipes/stuff_spec/stuff_spec_factory.py
@@ -52,4 +52,4 @@ class StuffSpecFactory:
),
)
- return StuffSpec(concept=concept, multiplicity=parse_result.multiplicity)
+ return StuffSpec(concept=concept, multiplicity=parse_result.multiplicity, presence=parse_result.presence)
diff --git a/pipelex/core/pipes/template_guard_lint.py b/pipelex/core/pipes/template_guard_lint.py
new file mode 100644
index 000000000..c52300726
--- /dev/null
+++ b/pipelex/core/pipes/template_guard_lint.py
@@ -0,0 +1,73 @@
+"""Shared template guard-lint (optionals design D7), applied by every pipe that renders
+authored templates over its inputs (PipeLLM prompts, PipeCompose templates, PipeCondition
+expressions): each reference to a declared-optional (`?`) input must be guarded, otherwise
+validation fails with `OPTIONAL_INPUT_UNGUARDED` and the precise fix.
+"""
+
+from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_preprocessor import rewrite_template_sigils
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
+from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError
+from pipelex.tools.jinja2.jinja2_optional_guards import detect_unguarded_optional_references
+
+
+def lint_optional_input_guards(
+ *,
+ pipe_code: str,
+ domain_code: str | None,
+ inputs: InputStuffSpecs,
+ template_source: str,
+ template_category: TemplateCategory,
+ template_label: str,
+) -> None:
+ """Raise `OPTIONAL_INPUT_UNGUARDED` when the template references a declared-optional input
+ without a guard.
+
+ The template is given in authored form: sigils (`@var`, `@?var`, `$var`) are rewritten to
+ their Jinja2 form before the walk, so `@?var` counts as a guard and a plain `@var` on an
+ optional input counts as an unguarded reference (the `tag` filter raises on an undefined
+ value at render time).
+
+ Args:
+ pipe_code: The pipe being validated (for the error).
+ domain_code: The pipe's domain (for the error).
+ inputs: The pipe's declared inputs — optional (`?`) ones drive the lint.
+ template_source: The authored template source.
+ template_category: The template's category (LLM_PROMPT, EXPRESSION, ...).
+ template_label: Which template this is, for the message ("prompt", "system_prompt",
+ "template", "expression").
+ """
+ optional_variable_names = {input_name for input_name, stuff_spec in inputs.root.items() if stuff_spec.presence.is_optional}
+ if not optional_variable_names:
+ return
+ try:
+ findings = detect_unguarded_optional_references(
+ template_category=template_category,
+ template_source=rewrite_template_sigils(template_source),
+ optional_variable_names=optional_variable_names,
+ )
+ except Jinja2DetectVariablesError:
+ # An unparseable template is not this lint's concern: the template-parsing gates
+ # (blueprint validation, the dry-run variable detection) report syntax errors through
+ # their own typed channels. Raising here — often inside a pydantic model validator —
+ # would leak an internal exception type past callers expecting ValidationError.
+ return
+ if not findings:
+ return
+ finding = findings[0]
+ msg = (
+ f"In the {template_label} of pipe '{pipe_code}', the reference '{{{{ {finding.path} }}}}' uses the optional input "
+ f"'{finding.variable_name}' unguarded — when the value is absent, the variable is undefined in the template. "
+ f"Guard every use: write `@?{finding.variable_name}` instead of `@{finding.variable_name}`, wrap the block in "
+ f"`{{% if {finding.variable_name} %}}...{{% endif %}}`, or use an inline conditional "
+ f"(`... if {finding.variable_name} is defined else ...`)."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED,
+ domain_code=domain_code,
+ pipe_code=pipe_code,
+ variable_names=[finding.variable_name],
+ explanation=f"Unguarded reference '{finding.path}' to optional input '{finding.variable_name}' in the {template_label}.",
+ )
diff --git a/pipelex/core/pipes/variable_multiplicity.py b/pipelex/core/pipes/variable_multiplicity.py
index 7cf16d3e0..4d7cb6388 100644
--- a/pipelex/core/pipes/variable_multiplicity.py
+++ b/pipelex/core/pipes/variable_multiplicity.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import re
+from enum import StrEnum
from pydantic import BaseModel, Field
@@ -8,7 +9,68 @@ from pipelex.core.pipes.exceptions import PipeVariableMultiplicityError
VariableMultiplicity = bool | int
-MUTLIPLICITY_PATTERN = r"^([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*)(?:\[(\d*)\])?$"
+# The io-ref suffix grammar: an optional multiplicity suffix ("[]" or "[N]") followed by an
+# optional presence marker ("?" optional / "!" force). Order is fixed: multiplicity then presence.
+# Group 1: concept ref or code; group 2: bracket content (None / "" / digits); group 3: presence
+# marker symbol (None / "?" / "!").
+MULTIPLICITY_PATTERN = r"^([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*)(?:\[(\d*)\])?([?!])?$"
+
+
+class PresenceMarker(StrEnum):
+ """Presence of a value in a declared slot: plain (always present), optional (`?` — the slot may
+ legitimately hold no value), or force (`!` — a use-site assertion that the value must be present).
+ """
+
+ PLAIN = "plain"
+ OPTIONAL = "optional"
+ FORCE = "force"
+
+ @classmethod
+ def from_symbol(cls, symbol: str | None) -> PresenceMarker:
+ match symbol:
+ case None | "":
+ return PresenceMarker.PLAIN
+ case "?":
+ return PresenceMarker.OPTIONAL
+ case "!":
+ return PresenceMarker.FORCE
+ case _:
+ msg = f"Invalid presence marker symbol: '{symbol}'. Expected '?', '!', or nothing."
+ raise PipeVariableMultiplicityError(msg)
+
+ @property
+ def symbol(self) -> str:
+ match self:
+ case PresenceMarker.PLAIN:
+ return ""
+ case PresenceMarker.OPTIONAL:
+ return "?"
+ case PresenceMarker.FORCE:
+ return "!"
+
+ @property
+ def is_optional(self) -> bool:
+ match self:
+ case PresenceMarker.OPTIONAL:
+ return True
+ case PresenceMarker.PLAIN | PresenceMarker.FORCE:
+ return False
+
+ @property
+ def is_force(self) -> bool:
+ match self:
+ case PresenceMarker.FORCE:
+ return True
+ case PresenceMarker.PLAIN | PresenceMarker.OPTIONAL:
+ return False
+
+ @property
+ def is_plain(self) -> bool:
+ match self:
+ case PresenceMarker.PLAIN:
+ return True
+ case PresenceMarker.OPTIONAL | PresenceMarker.FORCE:
+ return False
class VariableMultiplicityResolution(BaseModel):
@@ -53,45 +115,49 @@ def make_variable_multiplicity(*, nb_items: int | None, multiple_items: bool | N
class MultiplicityParseResult(BaseModel):
concept_ref_or_code: str
multiplicity: int | bool | None
+ presence: PresenceMarker = PresenceMarker.PLAIN
def parse_concept_with_multiplicity(concept_ref_or_code: str) -> MultiplicityParseResult:
- """Parse a concept specification string to extract concept and multiplicity.
+ """Parse a concept specification string to extract concept, multiplicity, and presence.
Supported formats:
- - "ConceptName" -> (ConceptName, None)
- - "ConceptName[]" -> (ConceptName, True)
- - "ConceptName[5]" -> (ConceptName, 5)
- - "domain.ConceptName" -> (domain.ConceptName, None)
- - "domain.ConceptName[]" -> (domain.ConceptName, True)
- - "domain.ConceptName[5]" -> (domain.ConceptName, 5)
+ - "ConceptName" -> (ConceptName, None, plain)
+ - "ConceptName[]" -> (ConceptName, True, plain)
+ - "ConceptName[5]" -> (ConceptName, 5, plain)
+ - "ConceptName?" -> (ConceptName, None, optional)
+ - "ConceptName!" -> (ConceptName, None, force)
+ - "domain.ConceptName" and any of the suffixes above on a qualified ref
+
+ The parser accepts multiplicity + presence combinations ("ConceptName[]?") so that callers with
+ context (blueprint input vs output validation) can reject them with a precise, typed error; the
+ D1 v1 rule that markers are mutually exclusive with multiplicity is NOT enforced here.
Args:
concept_ref_or_code: Concept specification string with optional multiplicity brackets
+ and optional presence marker (order fixed: multiplicity then presence)
Returns:
- MultiplicityParseResult with concept (without brackets) and multiplicity value
+ MultiplicityParseResult with concept (without suffixes), multiplicity, and presence
Raises:
PipeVariableMultiplicityError: If the concept specification has invalid syntax
or if multiplicity is zero or negative (a pipe must produce at least one output)
"""
- # Use strict pattern to validate identifier syntax
- # Concept must start with letter/underscore, with zero or more dotted domain segments, optional brackets
- pattern = r"^([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*)(?:\[(\d*)\])?$"
- match = re.match(pattern, concept_ref_or_code)
+ match = re.match(MULTIPLICITY_PATTERN, concept_ref_or_code)
if not match:
msg = (
f"Invalid concept specification syntax: '{concept_ref_or_code}'. "
- f"Expected format: 'ConceptName', 'ConceptName[]', 'ConceptName[N]', "
- f"'domain.ConceptName', 'domain.ConceptName[]', or 'domain.ConceptName[N]' "
+ f"Expected format: 'ConceptName' or 'domain.ConceptName' with an optional multiplicity "
+ f"suffix ('[]' or '[N]') and/or an optional presence marker ('?' or '!') after it, "
f"where concept and domain names must start with a letter or underscore."
)
raise PipeVariableMultiplicityError(msg)
extracted_concept = match.group(1)
bracket_content = match.group(2)
+ marker_symbol = match.group(3)
multiplicity: int | bool | None
if bracket_content is None:
@@ -107,7 +173,11 @@ def parse_concept_with_multiplicity(concept_ref_or_code: str) -> MultiplicityPar
msg = f"Invalid multiplicity value in '{concept_ref_or_code}': multiplicity must be at least 1. A pipe must produce at least one output."
raise PipeVariableMultiplicityError(msg)
- return MultiplicityParseResult(concept_ref_or_code=extracted_concept, multiplicity=multiplicity)
+ return MultiplicityParseResult(
+ concept_ref_or_code=extracted_concept,
+ multiplicity=multiplicity,
+ presence=PresenceMarker.from_symbol(marker_symbol),
+ )
def is_multiplicity_compatible(*, source_multiplicity: VariableMultiplicity | None, target_multiplicity: VariableMultiplicity | None) -> bool:
@@ -167,8 +237,13 @@ def is_multiplicity_compatible(*, source_multiplicity: VariableMultiplicity | No
return source_multiplicity == target_multiplicity
-def format_concept_with_multiplicity(concept_code_or_string: str, *, multiplicity: VariableMultiplicity | None) -> str:
- """Format a concept code or string with multiplicity notation.
+def format_concept_with_multiplicity(
+ concept_code_or_string: str,
+ *,
+ multiplicity: VariableMultiplicity | None,
+ presence: PresenceMarker = PresenceMarker.PLAIN,
+) -> str:
+ """Format a concept code or string with multiplicity and presence notation.
This is the reverse operation of parse_concept_with_multiplicity.
@@ -178,30 +253,31 @@ def format_concept_with_multiplicity(concept_code_or_string: str, *, multiplicit
- None: single item (no brackets)
- True: variable-length list (empty brackets [])
- int: fixed-length list (brackets with number [N])
+ presence: The presence marker, rendered after the multiplicity suffix ("" / "?" / "!")
Returns:
- Formatted concept specification string with multiplicity notation
+ Formatted concept specification string with multiplicity and presence notation
Examples:
- >>> format_concept_with_multiplicity("Text", None)
+ >>> format_concept_with_multiplicity("Text", multiplicity=None)
"Text"
- >>> format_concept_with_multiplicity("Text", True)
+ >>> format_concept_with_multiplicity("Text", multiplicity=True)
"Text[]"
- >>> format_concept_with_multiplicity("Text", 3)
+ >>> format_concept_with_multiplicity("Text", multiplicity=3)
"Text[3]"
- >>> format_concept_with_multiplicity("domain.Text", None)
- "domain.Text"
- >>> format_concept_with_multiplicity("domain.Text", True)
- "domain.Text[]"
- >>> format_concept_with_multiplicity("domain.Text", 5)
- "domain.Text[5]"
+ >>> format_concept_with_multiplicity("Text", multiplicity=None, presence=PresenceMarker.OPTIONAL)
+ "Text?"
+ >>> format_concept_with_multiplicity("domain.Text", multiplicity=None, presence=PresenceMarker.FORCE)
+ "domain.Text!"
"""
+ multiplicity_suffix: str
if multiplicity is None:
# Single item - no brackets
- return concept_code_or_string
+ multiplicity_suffix = ""
elif multiplicity is True:
# Variable-length list - empty brackets
- return f"{concept_code_or_string}[]"
+ multiplicity_suffix = "[]"
else:
# Fixed-length list - brackets with number
- return f"{concept_code_or_string}[{multiplicity}]"
+ multiplicity_suffix = f"[{multiplicity}]"
+ return f"{concept_code_or_string}{multiplicity_suffix}{presence.symbol}"
diff --git a/pipelex/core/stuffs/stuff_artefact.py b/pipelex/core/stuffs/stuff_artefact.py
index 30ac778ec..5dd787762 100644
--- a/pipelex/core/stuffs/stuff_artefact.py
+++ b/pipelex/core/stuffs/stuff_artefact.py
@@ -262,6 +262,19 @@ class StuffArtefact:
msg = f"'{content_type}' content does not support len()."
raise TypeError(msg)
+ def __bool__(self) -> bool:
+ """A present artefact is truthy; a ListContent artefact follows list emptiness.
+
+ Load-bearing for the optionals guard idiom: without `__bool__`, Jinja2's truth test
+ (`{% if var %}`, `@?var`'s expansion) falls through to `__len__`, which raises for
+ non-list content — the D7-blessed guard would crash on a PRESENT singular value.
+ An empty list is falsy on purpose (D4: `[]`-emptiness is the absence story for plurals).
+ """
+ content = self._stuff.content
+ if isinstance(content, ListContent):
+ return len(content) > 0
+ return True
+
# -------------------------------------------------------------------------
# Dict-like iteration (for template compatibility)
# Named with 'iter_' prefix to avoid conflicts with content fields
diff --git a/pipelex/graph/graph_tracer.py b/pipelex/graph/graph_tracer.py
index 1845b9461..37e96b228 100644
--- a/pipelex/graph/graph_tracer.py
+++ b/pipelex/graph/graph_tracer.py
@@ -19,6 +19,7 @@ from pipelex.graph.graphspec import (
NodeStatus,
PipelineRef,
TimingSpec,
+ output_digest_is_optional,
)
from pipelex.graph.trace_context import TraceContext
from pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in __init__ annotations
@@ -30,6 +31,7 @@ from pipelex.tracing.trace_events import (
ExecutionDataEvent,
ParallelCombineEvent,
PipeEndErrorEvent,
+ PipeEndSkippedEvent,
PipeEndSuccessEvent,
PipeStartEvent,
TraceEvent,
@@ -61,6 +63,7 @@ class _MutableNodeData:
self.domain_code = domain_code
self.ended_at: datetime | None = None
self.status: NodeStatus = NodeStatus.RUNNING
+ self.skip_reason: str | None = None
self.output_preview: str | None = None
self.metrics: dict[str, float] = {}
self.error: ErrorSpec | None = None
@@ -93,6 +96,7 @@ class _MutableNodeData:
description=self.description,
domain_code=self.domain_code,
status=self.status,
+ skip_reason=self.skip_reason,
timing=timing,
node_io=node_io,
error=self.error,
@@ -354,14 +358,27 @@ class GraphTracer(GraphTracerProtocol):
if producer_node_id == consumer_node_id:
# Don't create self-loops
continue
- # Create DATA edge: producer → consumer, labeled with the stuff name
+ # Create DATA edge: producer → consumer, labeled with the stuff name. The edge
+ # carries the optional marker when the producer's output was declared `?` (the
+ # value may be absent in other runs) — read off the producer's output spec.
self.add_edge(
source_node_id=producer_node_id,
target_node_id=consumer_node_id,
edge_kind=EdgeKind.DATA,
label=input_spec.name,
+ optional=self._is_optional_output_digest(producer_node_id=producer_node_id, digest=input_spec.digest),
)
+ def _is_optional_output_digest(self, *, producer_node_id: str, digest: str) -> bool:
+ """Whether the producer registered this digest as a declared-optional (`?`) output.
+
+ Node lookup here; the marker semantics live in the shared `output_digest_is_optional`.
+ """
+ producer_data = self._nodes.get(producer_node_id)
+ if producer_data is None:
+ return False
+ return output_digest_is_optional(producer_data.output_specs, digest=digest)
+
def _generate_batch_item_edges(self) -> None:
"""Generate BATCH_ITEM edges for batch fan-out.
@@ -857,6 +874,65 @@ class GraphTracer(GraphTracerProtocol):
)
)
+ @override
+ def on_pipe_end_skipped(
+ self,
+ node_id: str,
+ *,
+ ended_at: datetime,
+ skip_reason: str,
+ output_spec: IOSpec | None = None,
+ output_concept_data: dict[str, Any] | None = None,
+ ) -> None:
+ """Record that a pipe was lifted (skipped): its own node state, with the reason.
+
+ A lifted pipe with a PLURAL output still writes a real empty-list Stuff (D4) that
+ downstream pipes consume — its spec registers in the producer map exactly like a
+ success output, so the DATA edge to those consumers resolves.
+ """
+ if not self._is_active:
+ return
+
+ node_data = self._nodes.get(node_id)
+ if node_data is None:
+ return
+
+ node_data.ended_at = ended_at
+ node_data.status = NodeStatus.SKIPPED
+ node_data.skip_reason = skip_reason
+
+ # Accumulate output concept data (deduplicated) — mirrors on_pipe_end_success
+ if output_concept_data is not None:
+ concept_ref = f"{output_concept_data.get('domain_code', '')}.{output_concept_data.get('code', '')}"
+ if concept_ref not in self._concept_registry:
+ self._concept_registry[concept_ref] = output_concept_data
+
+ # Store output spec and register in producer map — mirrors on_pipe_end_success
+ # (including the pass-through check, vacuous here since the empty list is fresh)
+ if output_spec is not None:
+ input_digests = {spec.digest for spec in node_data.input_specs if spec.digest is not None}
+ if output_spec.digest not in input_digests:
+ node_data.output_specs.append(output_spec)
+ if output_spec.digest:
+ self._stuff_producer_map[output_spec.digest] = node_id
+
+ # Emit PipeEndSkippedEvent
+ if self._event_log is not None:
+ self._emit_event(
+ PipeEndSkippedEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ writer_id=self._event_writer_id(),
+ workflow_id=self._workflow_id,
+ timestamp=ended_at,
+ sequence=self._next_event_sequence(),
+ node_id=node_id,
+ ended_at=ended_at,
+ skip_reason=skip_reason,
+ output_spec=output_spec,
+ output_concept_data=output_concept_data or {},
+ )
+ )
+
@override
def add_edge(
self,
@@ -867,6 +943,7 @@ class GraphTracer(GraphTracerProtocol):
label: str | None = None,
source_stuff_digest: str | None = None,
target_stuff_digest: str | None = None,
+ optional: bool = False,
) -> None:
"""Add an edge between two nodes."""
if not self._is_active:
@@ -879,6 +956,7 @@ class GraphTracer(GraphTracerProtocol):
source=source_node_id,
target=target_node_id,
kind=edge_kind,
+ optional=optional,
label=label,
source_stuff_digest=source_stuff_digest,
target_stuff_digest=target_stuff_digest,
@@ -898,6 +976,7 @@ class GraphTracer(GraphTracerProtocol):
source_node_id=source_node_id,
target_node_id=target_node_id,
edge_kind=edge_kind,
+ optional=optional,
label=label,
source_stuff_digest=source_stuff_digest,
target_stuff_digest=target_stuff_digest,
diff --git a/pipelex/graph/graph_tracer_manager.py b/pipelex/graph/graph_tracer_manager.py
index 1b45d88ce..4431370b8 100644
--- a/pipelex/graph/graph_tracer_manager.py
+++ b/pipelex/graph/graph_tracer_manager.py
@@ -7,7 +7,7 @@ from pipelex import log
from pipelex.graph.graph_config import DataInclusionConfig
from pipelex.graph.graph_tracer import GraphTracer
from pipelex.graph.graph_tracer_protocol import GraphTracerProtocol
-from pipelex.graph.graphspec import EdgeKind, GraphSpec, IOSpec, NodeKind
+from pipelex.graph.graphspec import GraphSpec, IOSpec, NodeKind
from pipelex.graph.trace_context import TraceContext
from pipelex.system.registries.singleton import ABCSingletonMeta, MetaSingleton
from pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in open_tracer signature
@@ -322,25 +322,26 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
return
tracer.register_execution_data(node_id=node_id, execution_data=execution_data)
- def on_pipe_end_error(
+ def on_pipe_end_skipped(
self,
*,
lookup_key: str,
node_id: str | None,
ended_at: datetime,
- error_type: str,
- error_message: str,
- error_stack: str | None = None,
+ skip_reason: str,
+ output_spec: IOSpec | None = None,
+ output_concept_data: dict[str, Any] | None = None,
) -> None:
- """Record failed completion of a pipe execution.
+ """Record that a pipe was lifted (skipped) because a plain input resolved absent.
Args:
lookup_key: The tracer lookup key.
node_id: The node ID returned from on_pipe_start.
- ended_at: When the pipe failed.
- error_type: The exception type name.
- error_message: The error message.
- error_stack: Optional stack trace.
+ ended_at: When the skip was decided.
+ skip_reason: Human-readable reason (names the absent input).
+ output_spec: The real output a lifted pipe still wrote (PLURAL outputs normalize
+ to an empty list, D4) — registered in the producer map for DATA edges.
+ output_concept_data: Optional serialized concept dict for that output's concept.
"""
if node_id is None:
return
@@ -349,41 +350,47 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
if tracer is None:
return
- tracer.on_pipe_end_error(
+ tracer.on_pipe_end_skipped(
node_id=node_id,
ended_at=ended_at,
- error_type=error_type,
- error_message=error_message,
- error_stack=error_stack,
+ skip_reason=skip_reason,
+ output_spec=output_spec,
+ output_concept_data=output_concept_data,
)
- def add_edge(
+ def on_pipe_end_error(
self,
*,
lookup_key: str,
- source_node_id: str,
- target_node_id: str,
- edge_kind: EdgeKind,
- label: str | None = None,
+ node_id: str | None,
+ ended_at: datetime,
+ error_type: str,
+ error_message: str,
+ error_stack: str | None = None,
) -> None:
- """Add an edge between two nodes.
+ """Record failed completion of a pipe execution.
Args:
lookup_key: The tracer lookup key.
- source_node_id: The source node ID.
- target_node_id: The target node ID.
- edge_kind: The type of edge.
- label: Optional label for the edge.
+ node_id: The node ID returned from on_pipe_start.
+ ended_at: When the pipe failed.
+ error_type: The exception type name.
+ error_message: The error message.
+ error_stack: Optional stack trace.
"""
+ if node_id is None:
+ return
+
tracer = self._get_tracer(lookup_key)
if tracer is None:
return
- tracer.add_edge(
- source_node_id=source_node_id,
- target_node_id=target_node_id,
- edge_kind=edge_kind,
- label=label,
+ tracer.on_pipe_end_error(
+ node_id=node_id,
+ ended_at=ended_at,
+ error_type=error_type,
+ error_message=error_message,
+ error_stack=error_stack,
)
def register_controller_output(
diff --git a/pipelex/graph/graph_tracer_protocol.py b/pipelex/graph/graph_tracer_protocol.py
index 0b754bd1c..d4dd6c097 100644
--- a/pipelex/graph/graph_tracer_protocol.py
+++ b/pipelex/graph/graph_tracer_protocol.py
@@ -132,6 +132,33 @@ class GraphTracerProtocol(Protocol):
"""
...
+ def on_pipe_end_skipped(
+ self,
+ node_id: str,
+ *,
+ ended_at: datetime,
+ skip_reason: str,
+ output_spec: IOSpec | None = None,
+ output_concept_data: dict[str, Any] | None = None,
+ ) -> None:
+ """Record that a pipe was lifted (skipped) because a plain input resolved absent (D3).
+
+ A skip is a successful outcome (the run continues; the pipe's output is a recorded
+ absence), rendered as its own node state so "why did my workflow produce nothing?"
+ is answerable from the graph.
+
+ Args:
+ node_id: The node ID returned from on_pipe_start.
+ ended_at: When the skip was decided.
+ skip_reason: Human-readable reason (names the absent input).
+ output_spec: The real output a lifted pipe still wrote, when there is one — a
+ PLURAL output normalizes to an empty list (D4) that downstream pipes consume,
+ so it must register in the producer map for DATA edges. None for a singular
+ output (a recorded absence has no payload).
+ output_concept_data: Optional serialized concept dict for that output's concept.
+ """
+ ...
+
def add_edge(
self,
*,
@@ -141,6 +168,7 @@ class GraphTracerProtocol(Protocol):
label: str | None = None,
source_stuff_digest: str | None = None,
target_stuff_digest: str | None = None,
+ optional: bool = False,
) -> None:
"""Add an edge between two nodes.
@@ -151,6 +179,7 @@ class GraphTracerProtocol(Protocol):
label: Optional label for the edge.
source_stuff_digest: Optional stuff digest for the source (for batch edges).
target_stuff_digest: Optional stuff digest for the target (for batch edges).
+ optional: Marker for a data edge fed by a declared-optional (`?`) output.
"""
...
@@ -328,6 +357,18 @@ class GraphTracerNoOp(GraphTracerProtocol):
) -> None:
pass
+ @override
+ def on_pipe_end_skipped(
+ self,
+ node_id: str,
+ *,
+ ended_at: datetime,
+ skip_reason: str,
+ output_spec: IOSpec | None = None,
+ output_concept_data: dict[str, Any] | None = None,
+ ) -> None:
+ pass
+
@override
def add_edge(
self,
@@ -338,6 +379,7 @@ class GraphTracerNoOp(GraphTracerProtocol):
label: str | None = None,
source_stuff_digest: str | None = None,
target_stuff_digest: str | None = None,
+ optional: bool = False,
) -> None:
pass
diff --git a/pipelex/graph/graphspec.py b/pipelex/graph/graphspec.py
index e176645d8..eedaea58d 100644
--- a/pipelex/graph/graphspec.py
+++ b/pipelex/graph/graphspec.py
@@ -4,6 +4,7 @@ This module defines the canonical, versioned data model for Pipelex run graphs.
GraphSpec is renderer-agnostic and designed for JSON serialization.
"""
+from collections.abc import Sequence
from datetime import datetime
from enum import StrEnum
from typing import Any, Self
@@ -198,6 +199,20 @@ class IOSpec(BaseModel):
return _truncate_string(value, max_length=MAX_PREVIEW_LENGTH)
+def output_digest_is_optional(output_specs: Sequence[IOSpec], *, digest: str) -> bool:
+ """Whether a producer registered ``digest`` as a declared-optional (`?`) output.
+
+ The optional marker rides the output IOSpec's ``extra`` dict (set at the pipe-run
+ epilogue from the pipe's declared output presence). One helper for both graph builders
+ (in-process GraphTracer and the event-replay assembler) so the optional-edge computation
+ cannot drift between them.
+ """
+ for output_spec in output_specs:
+ if output_spec.digest == digest:
+ return bool(output_spec.extra.get("optional"))
+ return False
+
+
class NodeIOSpec(BaseModel):
"""Input/output specification for a node."""
@@ -241,6 +256,8 @@ class NodeSpec(BaseModel):
description: str | None = None
domain_code: str | None = None
status: NodeStatus = Field(strict=False)
+ # Why a `skipped` node was lifted (names the absent input); None for every other status.
+ skip_reason: str | None = None
timing: TimingSpec | None = None
node_io: NodeIOSpec = Field(
default_factory=NodeIOSpec,
@@ -262,6 +279,8 @@ class EdgeSpec(BaseModel):
source: str
target: str
kind: EdgeKind = Field(strict=False)
+ # A data edge fed by a declared-optional (`?`) output: the value may be absent in other runs.
+ optional: bool = False
label: str | None = None
# For batch edges, specify the stuff digests for renderers to connect stuff nodes directly
source_stuff_digest: str | None = None
diff --git a/pipelex/libraries/contract_match.py b/pipelex/libraries/contract_match.py
index 6ae491302..2b1df312e 100644
--- a/pipelex/libraries/contract_match.py
+++ b/pipelex/libraries/contract_match.py
@@ -17,7 +17,7 @@ import re
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
-from pipelex.core.pipes.variable_multiplicity import MUTLIPLICITY_PATTERN
+from pipelex.core.pipes.variable_multiplicity import MULTIPLICITY_PATTERN
from pipelex.core.qualified_ref import QualifiedRef
@@ -30,6 +30,10 @@ def _canonical_concept_spec(spec: str, *, domain_code: str) -> str:
``[]`` to ``True`` and ``[1]`` to ``int 1``, and Python evaluates ``True == 1`` as true. Keeping
the suffix as text also means a ``[0]`` spec compares as a literal rather than raising.
+ The presence marker (``"?"`` / ``"!"``) is likewise preserved verbatim, so ``Brief`` and
+ ``Brief?`` denote different contracts (D5: method-boundary signatures are explicit about
+ optionality, so a header and its definition must agree on the marker).
+
The concept part is canonicalized so equivalent spellings collapse to one identity:
- native (``Text`` or ``native.Text``) -> the bare native code (``Text``);
@@ -41,12 +45,13 @@ def _canonical_concept_spec(spec: str, *, domain_code: str) -> str:
cannot appear in a blueprint's ``inputs``/``output`` but is handled defensively) is returned
verbatim — it is already unambiguous.
"""
- match = re.match(MUTLIPLICITY_PATTERN, spec)
+ match = re.match(MULTIPLICITY_PATTERN, spec)
if match is None:
return spec
concept_ref_or_code = match.group(1)
bracket_content = match.group(2) # None -> no brackets; "" -> "[]"; digits -> "[N]"
- suffix = "" if bracket_content is None else f"[{bracket_content}]"
+ marker_symbol = match.group(3) or "" # None -> no marker; "?" / "!" kept verbatim
+ suffix = ("" if bracket_content is None else f"[{bracket_content}]") + marker_symbol
if NativeConceptCode.is_native_concept_ref_or_code(concept_ref_or_code):
return f"{QualifiedRef.parse(concept_ref_or_code).local_code}{suffix}"
diff --git a/pipelex/pipe_controllers/absence_taint.py b/pipelex/pipe_controllers/absence_taint.py
new file mode 100644
index 000000000..c658276a5
--- /dev/null
+++ b/pipelex/pipe_controllers/absence_taint.py
@@ -0,0 +1,157 @@
+"""Carriers and helpers for the static absence-taint pass (optionals design D6).
+
+The taint pass runs over a controller's dataflow at validation time, computing per-slot presence:
+`guaranteed` or `maybe-absent`. A slot is maybe-absent (tainted) when it is fed by an optional
+(`?`) input of the controller, by a pipe declaring an optional output, or by a pipe that would be
+lifted (skipped) because one of its plain inputs is fed a maybe-absent slot. Presence markers on
+a consuming pipe's own input declarations decide how taint propagates (D3 trichotomy): plain →
+the pipe lifts and its output is tainted; `?` → absorbed, taint terminates; `!` → asserted at
+run time, taint terminates.
+
+Plural slots are never tainted (D4): a skipped plural output normalizes to an empty list and a
+batched step compacts absent branch results, so a list slot is always guaranteed (possibly empty).
+
+The walk logic itself lives on the controllers (`PipeSequence.analyze_taint`,
+`PipeParallel.analyze_branch_taint`); this module holds the shared value objects and the
+presence-resolution helper so the pipeline layer (liftable-pipe inventory) can consume the
+same shapes without importing controller internals.
+"""
+
+from typing import NamedTuple
+
+from pydantic.dataclasses import dataclass
+
+from pipelex.core.pipes.inputs.input_stuff_specs import NamedStuffSpec
+from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker, VariableMultiplicity
+
+
+@dataclass(frozen=True)
+class SlotTaint:
+ """A maybe-absent slot: where the absence originates and how it propagated here."""
+
+ source: str
+ origin_slot_name: str
+ chain: tuple[str, ...] = ()
+
+ def describe(self) -> str:
+ description = f"Absence origin: {self.source}."
+ if self.chain:
+ description += f" Propagation: {' → '.join(self.chain)}."
+ return description
+
+
+@dataclass(frozen=True)
+class LiftableStepInfo:
+ """One pipe that may be lifted (skipped) within a controller's flow."""
+
+ within_pipe_ref: str
+ pipe_ref: str
+ trigger_variable_names: tuple[str, ...]
+ absence_source: str
+
+
+@dataclass(frozen=True)
+class ForceConsumptionInfo:
+ """One `!` (force) consumption observed during a controller walk.
+
+ ``is_asserting`` is True when the consumed slot was maybe-absent at that point of the flow
+ (the `!` can fire — meaningful) and False when the slot was guaranteed present (the `!` can
+ never fire in this flow — a candidate for the useless-`!` lint, subject to cross-flow
+ aggregation: one asserting flow silences the redundant observations).
+ """
+
+ within_pipe_ref: str
+ pipe_ref: str
+ variable_name: str
+ is_asserting: bool
+
+
+@dataclass(frozen=True)
+class SequenceTaintAnalysis:
+ """Result of the taint walk over a PipeSequence's steps."""
+
+ liftable_steps: tuple[LiftableStepInfo, ...]
+ output_taint: SlotTaint | None
+ force_consumptions: tuple[ForceConsumptionInfo, ...] = ()
+
+
+@dataclass(frozen=True)
+class ParallelTaintAnalysis:
+ """Result of the taint walk over a PipeParallel's branches."""
+
+ branch_taints: dict[str, SlotTaint]
+ liftable_steps: tuple[LiftableStepInfo, ...]
+ force_consumptions: tuple[ForceConsumptionInfo, ...] = ()
+
+
+class TaintTriggerScan(NamedTuple):
+ """How a pipe consumes the currently tainted slots: the plain-consumed (lift-trigger)
+ variable names, the first trigger's taint (for provenance chaining), and the `!`
+ consumptions split by whether the slot was maybe-absent (asserting) or guaranteed
+ (redundant candidate) at this point of the flow.
+ """
+
+ trigger_names: tuple[str, ...]
+ trigger_taint: SlotTaint | None
+ asserting_force_names: tuple[str, ...] = ()
+ redundant_force_names: tuple[str, ...] = ()
+
+
+def scan_taint_triggers(pipe: PipeAbstract, *, slot_taints: dict[str, SlotTaint]) -> TaintTriggerScan:
+ """Apply the D3 trichotomy to a pipe's needed inputs against the given taint map.
+
+ A tainted slot consumed PLAIN is a lift trigger; consumed `?` (absorb) or `!` (assert),
+ the taint terminates at that consumption. `!` consumptions are also classified for the
+ useless-`!` lint: asserting when the slot was tainted, redundant when it was guaranteed.
+ """
+ trigger_names: list[str] = []
+ trigger_taint: SlotTaint | None = None
+ asserting_force_names: list[str] = []
+ redundant_force_names: list[str] = []
+ for named_stuff_spec in pipe.needed_inputs().named_stuff_specs:
+ incoming_taint = slot_taints.get(named_stuff_spec.variable_name)
+ match effective_consumption_presence(pipe, named_stuff_spec=named_stuff_spec):
+ case PresenceMarker.PLAIN:
+ if incoming_taint is not None:
+ trigger_names.append(named_stuff_spec.variable_name)
+ if trigger_taint is None:
+ trigger_taint = incoming_taint
+ case PresenceMarker.FORCE:
+ if incoming_taint is not None:
+ asserting_force_names.append(named_stuff_spec.variable_name)
+ else:
+ redundant_force_names.append(named_stuff_spec.variable_name)
+ case PresenceMarker.OPTIONAL:
+ continue
+ return TaintTriggerScan(
+ trigger_names=tuple(trigger_names),
+ trigger_taint=trigger_taint,
+ asserting_force_names=tuple(asserting_force_names),
+ redundant_force_names=tuple(redundant_force_names),
+ )
+
+
+def is_plural_step_result(
+ pipe: PipeAbstract,
+ *,
+ step_output_multiplicity: VariableMultiplicity | None,
+ has_batch_params: bool,
+) -> bool:
+ """Whether a step's result slot is plural — a plural result is never tainted (D4:
+ a lifted plural output is the empty list; a batched step compacts).
+ """
+ return has_batch_params or step_output_multiplicity is not None or pipe.output.multiplicity is not None
+
+
+def effective_consumption_presence(pipe: PipeAbstract, *, named_stuff_spec: NamedStuffSpec) -> PresenceMarker:
+ """The presence marker governing how `pipe` consumes the given needed input.
+
+ The pipe's OWN input declaration wins when it exists — that is the boundary contract
+ (D5: boundaries explicit); a controller's aggregated needs carry its children's markers,
+ which must not override the declared boundary. Mirrors the runtime `_scan_input_presence`.
+ """
+ declared_stuff_spec = pipe.inputs.root.get(named_stuff_spec.variable_name)
+ if declared_stuff_spec is not None:
+ return declared_stuff_spec.presence
+ return named_stuff_spec.presence
diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.py
index 0e8a6be56..029df2bf5 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch.py
@@ -6,10 +6,12 @@ from typing_extensions import override
from pipelex import log
from pipelex.cogt.content_generation.dry_mock import stamp_mock_main_coordination
from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.graph.graph_tracer_manager import GraphTracerManager
from pipelex.hub import get_pipe_router, get_required_pipe
@@ -24,7 +26,6 @@ from pipelex.urls import URLs
if TYPE_CHECKING:
from collections.abc import Awaitable, Callable
- from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.libraries.library_crate import LibraryCrate
@@ -212,9 +213,14 @@ class PipeBatch(PipeController):
branch_output_stuff_codes: list[str] = []
for pipe_output in pipe_outputs:
- branch_output_stuff = pipe_output.main_stuff
- output_items.append(branch_output_stuff.content)
- branch_output_stuff_codes.append(branch_output_stuff.stuff_code)
+ branch_resolved = pipe_output.working_memory.resolve_main_stuff()
+ if isinstance(branch_resolved, AbsenceRecord):
+ # Compaction (D4): an absent branch result cannot occupy a list slot — it is
+ # dropped, so the aggregated output holds only the found items (compactMap).
+ log.verbose(f"PipeBatch '{self.code}': dropping absent branch result ({branch_resolved.reason})")
+ continue
+ output_items.append(branch_resolved.content)
+ branch_output_stuff_codes.append(branch_resolved.stuff_code)
list_content: ListContent[StuffContent] = ListContent(items=output_items)
output_stuff = StuffFactory.make_stuff(
@@ -275,11 +281,14 @@ class PipeBatch(PipeController):
library_crate=library_crate,
)
# Dry-run coordination: see stamp_mock_main_coordination's docstring (single home, D3).
- main_stuff = pipe_output.main_stuff
- content = main_stuff.content
- if isinstance(content, ListContent):
- list_content = cast("ListContent[StuffContent]", content)
- stamp_mock_main_coordination(list_content.items)
+ # Tri-state read for robustness: the batch's own aggregated output is always stamped by the
+ # live arm today, but this site must never crash on a resolved-as-absent main.
+ main_resolved = pipe_output.working_memory.resolve_main_stuff()
+ if isinstance(main_resolved, Stuff):
+ content = main_resolved.content
+ if isinstance(content, ListContent):
+ list_content = cast("ListContent[StuffContent]", content)
+ stamp_mock_main_coordination(list_content.items)
return pipe_output
@override
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index 1fbd2f513..f6cee19c4 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -6,11 +6,13 @@ from pipelex import log
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.template_rendering import render_template
from pipelex.core.concepts.native.concept_native import NativeConceptCode
-from pipelex.core.memory.working_memory import WorkingMemory, WorkingMemoryStuffNotFoundError
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory
from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.pipes.template_guard_lint import lint_optional_input_guards
from pipelex.hub import get_optional_pipe, get_pipe_router, get_required_pipe
from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
from pipelex.pipe_controllers.pipe_controller import PipeController
@@ -88,6 +90,7 @@ class PipeCondition(PipeController):
variable_name=var_name,
concept=stuff_spec.concept,
multiplicity=stuff_spec.multiplicity,
+ presence=stuff_spec.presence,
)
# Add the inputs needed by all possible target pipes
@@ -97,21 +100,52 @@ class PipeCondition(PipeController):
pipe_needed_inputs = pipe.needed_inputs(visited_pipes_with_current)
for input_name, stuff_spec in pipe_needed_inputs.items:
- needed_inputs.add_stuff_spec(variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
return needed_inputs
@override
def validate_inputs_static(self):
- pass
+ # Guard-lint (D7): a declared-optional input referenced unguarded in the expression
+ # would evaluate over an undefined variable when the value is absent.
+ lint_optional_input_guards(
+ pipe_code=self.code,
+ domain_code=self.domain_code,
+ inputs=self.inputs,
+ template_source=self.expression,
+ template_category=TemplateCategory.EXPRESSION,
+ template_label="expression",
+ )
@override
def validate_inputs_with_library(self):
pass
+ @property
+ def _continue_reachable(self) -> bool:
+ return SpecialOutcome.is_continue(self.default_outcome) or any(SpecialOutcome.is_continue(outcome) for outcome in self.outcome_map.values())
+
@override
def validate_output_static(self):
- pass
+ # OPTIONAL_OUTPUT_REQUIRED (D5/D6): `continue` resolves the declared output as ABSENT
+ # (design §14), so a `continue`-reachable condition must declare its output optional —
+ # otherwise the no-output path would be invisible to the type system, which is exactly
+ # the invisible-optional wart this feature removes.
+ if self._continue_reachable and not self.output.presence.is_optional:
+ msg = (
+ f"PipeCondition '{self.code}' can resolve to 'continue', which resolves the declared output as absent, "
+ f"but its output '{self.output.concept.concept_ref}' is not declared optional. "
+ f"Declare the output optional ('{self.output.concept.concept_ref}?'), or remove the 'continue' outcome."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_OUTPUT_REQUIRED,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ provided_concept_code=self.output.concept.concept_ref,
+ )
@override
def validate_output_with_library(self):
@@ -129,6 +163,27 @@ class PipeCondition(PipeController):
# No actual pipes to validate against (all special outcomes)
return
+ # Boundary taint (D6): a mapped pipe with an optional output makes this condition's own
+ # output maybe-absent, so the condition must declare `?` too — otherwise the taint would
+ # silently escape through the condition boundary.
+ if not self.output.presence.is_optional:
+ for pipe_code in sorted(mapped_pipe_codes):
+ mapped_pipe = get_required_pipe(pipe_code=pipe_code)
+ if mapped_pipe.output.presence.is_optional:
+ msg = (
+ f"PipeCondition '{self.code}' maps outcome pipe '{pipe_code}' whose output "
+ f"'{mapped_pipe.output.concept.concept_ref}' is declared optional, but the condition's own output "
+ f"'{self.output.concept.concept_ref}' is not. Declare the condition's output optional "
+ f"('{self.output.concept.concept_ref}?') so the maybe-absent result stays visible downstream."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_NOT_HANDLED,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ provided_concept_code=self.output.concept.concept_ref,
+ )
+
# Collect all unique output concept refs from mapped pipes
mapped_output_refs: set[str] = set()
for pipe_code in mapped_pipe_codes:
@@ -241,20 +296,18 @@ class PipeCondition(PipeController):
"selected_outcome": str(outcome),
}
- # Handle continue case: `continue` delivers the current main stuff by passing it through.
- # With nothing to pass through, the run would deliver no main stuff — a contract violation
- # ("a pipe run always delivers a main stuff") failed loudly here, at the semantic site,
- # instead of crashing downstream (graph tracer, delivery, telemetry).
+ # Handle continue case (design §14, phase 1): `continue` resolves the declared output as
+ # ABSENT — a declared-absent record with provenance, memory otherwise unchanged. A previous
+ # main stuff stays under its own name (the migration idiom: consume it explicitly
+ # downstream); it no longer passes through as this pipe's output.
if SpecialOutcome.is_continue(outcome):
log.dev(f"PipeCondition '{self.code}' continued with outcome: {outcome}. Evaluated expression: {evaluated_expression}")
self._register_execution_data(job_metadata, execution_data=execution_data_dict)
- if working_memory.get_optional_main_stuff() is None:
- msg = (
- f"PipeCondition '{self.code}' resolved to the 'continue' outcome but the working memory has "
- "no main stuff to pass through — a pipe run must deliver a main stuff. Map the outcome to a "
- "pipe, or place this condition after a step that produces the output to pass through."
- )
- raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
+ self._record_declared_absent_output(
+ working_memory=working_memory,
+ output_name=output_name,
+ reason=f"PipeCondition '{self.code}' resolved to 'continue' for evaluated expression '{evaluated_expression}'",
+ )
return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
if SpecialOutcome.is_fail(outcome):
@@ -269,14 +322,22 @@ class PipeCondition(PipeController):
required_variables = chosen_pipe.required_variables()
# TODO: Merge `needed_inputs` and `required_variables` methods for cleaner code.
required_stuff_names = {get_root_from_dotted_path(req_var) for req_var in required_variables if not req_var.startswith("_")}
- try:
- working_memory.get_stuffs(names=required_stuff_names)
- except WorkingMemoryStuffNotFoundError as exc:
+ # A variable declared optional (`?`) on the chosen pipe is never presence-required (the
+ # `@?` fix, D7): the pipe runs with the slot absent and its guarded templates handle it.
+ required_stuff_names = {
+ name
+ for name in required_stuff_names
+ if not ((declared_spec := chosen_pipe.inputs.root.get(name)) is not None and declared_spec.presence.is_optional)
+ }
+ # A recorded absence is not a miss: the chosen pipe's own gate applies the trichotomy
+ # (skip / run / force). Only a name with neither a value nor a record is a hard miss.
+ missing_names = working_memory.list_missing_names(names=required_stuff_names)
+ if missing_names:
pipe_condition_path = [*pipe_run_params.pipe_layers, self.code]
pipe_condition_path_str = ".".join(pipe_condition_path)
- error_details = f"PipeCondition '{pipe_condition_path_str}', required_variables: {required_variables}, missing: '{exc.variable_name}'"
+ error_details = f"PipeCondition '{pipe_condition_path_str}', required_variables: {required_variables}, missing: '{missing_names[0]}'"
msg = f"Some required stuff(s) not found: {error_details}"
- raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code) from exc
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
pipe_output = await get_pipe_router().run(
pipe_job=PipeJobFactory.make_pipe_job(
@@ -347,13 +408,32 @@ class PipeCondition(PipeController):
"selected_outcome": "all_outcomes",
}
self._register_execution_data(job_metadata, execution_data=execution_data_dict)
- # Dry-run parity with the live `continue` guard: the mapped pipes just dry-ran into this
- # memory and stamped their outputs; if none did (all-special-outcomes condition) and no
- # main stuff pre-existed, the run cannot deliver one — fail loudly at the pipe level.
- if working_memory.get_optional_main_stuff() is None:
- msg = (
- f"Dry run of PipeCondition '{self.code}' delivered no main stuff — with only special outcomes "
- "mapped and no pre-existing main stuff, the run cannot deliver an output."
+ # Dry-run parity with the live `continue` arm: with only special outcomes mapped, no pipe
+ # dry-ran into this memory. When `continue` is reachable the declared output resolves
+ # absent, memory otherwise unchanged (the static rule "continue-reachable ⇒ `?` output" is
+ # Step D's OPTIONAL_OUTPUT_REQUIRED). A fail-only condition has no absent arm — every live
+ # path raises — so fabricating an absence would let dry-run bless a method that can only
+ # fail at runtime.
+ if not self.pipe_dependencies():
+ if not self._continue_reachable:
+ msg = (
+ f"PipeCondition '{self.code}' maps every outcome (and the default) to 'fail': "
+ f"every live run of this pipe raises. Map at least one outcome to a pipe or to 'continue'."
+ )
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
+ self._record_declared_absent_output(
+ working_memory=working_memory,
+ output_name=output_name,
+ reason=f"dry run of PipeCondition '{self.code}': all outcomes are special, the declared output resolves absent",
)
- raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
+
+ def _record_declared_absent_output(self, *, working_memory: WorkingMemory, output_name: str | None, reason: str) -> None:
+ """Resolve this pipe's declared output as a declared-absent record (the `continue` arm)."""
+ record = AbsenceRecord(
+ variable_name=output_name or MAIN_STUFF_NAME,
+ kind=AbsenceKind.DECLARED_ABSENT,
+ reason=reason,
+ producing_pipe=self.code,
+ )
+ working_memory.record_new_main_absence(record)
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index d99954dd9..cd18522c7 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -9,19 +9,29 @@ from pipelex import log
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.exceptions import ConceptValueError
from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.absence import AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
from pipelex.core.pipes.inputs.exceptions import InputStuffSpecNotFoundError
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactory
+from pipelex.core.pipes.pipe_abstract import CompanionSlot
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.composite_content import CompositeContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.graph.graph_tracer_manager import GraphTracerManager
from pipelex.graph.graphspec import IOSpec
-from pipelex.hub import get_required_pipe
+from pipelex.hub import get_optional_pipe, get_required_pipe
from pipelex.libraries.pipe.exceptions import PipeNotFoundError
+from pipelex.pipe_controllers.absence_taint import (
+ ForceConsumptionInfo,
+ LiftableStepInfo,
+ ParallelTaintAnalysis,
+ SlotTaint,
+ is_plural_step_result,
+ scan_taint_triggers,
+)
from pipelex.pipe_controllers.pipe_controller import PipeController
from pipelex.pipe_controllers.sub_pipe import SubPipe
from pipelex.pipe_run.exceptions import PipeRunError
@@ -98,10 +108,14 @@ class PipeParallel(PipeController):
)
for input_name, stuff_spec in pipe_needed_inputs.items:
if input_name != sub_pipe.batch_params.input_item_stuff_name:
- needed_inputs.add_stuff_spec(input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
else:
for input_name, stuff_spec in pipe_needed_inputs.items:
- needed_inputs.add_stuff_spec(input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
return needed_inputs
@override
@@ -209,6 +223,139 @@ class PipeParallel(PipeController):
self._validate_branch_output_types(structure_class=structure_class)
+ # D11 static rule: a REQUIRED field cannot be fed by a maybe-absent branch — absorption
+ # (field-level None) only exists for non-required fields. The runtime guard in
+ # `_run_branches_and_combine` is the belt-and-suspenders for this check.
+ branch_taints = self.analyze_branch_taint().branch_taints
+ for sub_pipe in self.parallel_sub_pipes:
+ result_name = sub_pipe.output_name
+ if not result_name:
+ continue
+ branch_taint = branch_taints.get(result_name)
+ if branch_taint is None:
+ continue
+ field_info = structure_class.model_fields.get(result_name)
+ if field_info is not None and field_info.is_required():
+ msg = (
+ f"PipeParallel '{self.code}': branch '{sub_pipe.pipe_code}' may resolve absent "
+ f"({branch_taint.describe()}), but field '{result_name}' of output "
+ f"'{self.output.concept.concept_ref}' is required. Make the field optional "
+ f"(required = false) or sink the absence upstream (a `?` input on the branch path)."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_BRANCH_REQUIRED_FIELD,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ provided_concept_code=self.output.concept.concept_ref,
+ variable_names=[result_name],
+ )
+
+ def analyze_branch_taint(self) -> ParallelTaintAnalysis:
+ """Static taint over the branches (D6): which branch results are maybe-absent.
+
+ Within the parallel's frame, the maybe-absent slots are exactly its own `?`-declared
+ inputs (a tainted slot declared plain at the parallel's boundary lifts the whole
+ parallel — the enclosing controller's concern, not this walk's). A branch is
+ maybe-absent when it is liftable (a plain input fed by an optional input of the
+ parallel) or when its pipe declares an optional output; a plural branch result is
+ never maybe-absent (D4: compaction / empty list).
+ """
+ optional_input_taints: dict[str, SlotTaint] = {}
+ for input_name, stuff_spec in self.inputs.root.items():
+ if stuff_spec.presence.is_optional and not stuff_spec.is_multiple():
+ optional_input_taints[input_name] = SlotTaint(
+ source=f"optional input '{input_name}' of pipe '{self.code}'",
+ origin_slot_name=input_name,
+ )
+
+ branch_taints: dict[str, SlotTaint] = {}
+ liftable_steps: list[LiftableStepInfo] = []
+ force_consumptions: list[ForceConsumptionInfo] = []
+ for sub_pipe in self.parallel_sub_pipes:
+ branch_pipe = get_optional_pipe(pipe_code=sub_pipe.pipe_code)
+ if branch_pipe is None:
+ continue
+ trigger_scan = scan_taint_triggers(branch_pipe, slot_taints=optional_input_taints)
+ for asserting_name in trigger_scan.asserting_force_names:
+ force_consumptions.append(
+ ForceConsumptionInfo(
+ within_pipe_ref=self.pipe_ref, pipe_ref=branch_pipe.pipe_ref, variable_name=asserting_name, is_asserting=True
+ )
+ )
+ for redundant_name in trigger_scan.redundant_force_names:
+ force_consumptions.append(
+ ForceConsumptionInfo(
+ within_pipe_ref=self.pipe_ref, pipe_ref=branch_pipe.pipe_ref, variable_name=redundant_name, is_asserting=False
+ )
+ )
+ if trigger_scan.trigger_names and trigger_scan.trigger_taint is not None:
+ liftable_steps.append(
+ LiftableStepInfo(
+ within_pipe_ref=self.pipe_ref,
+ pipe_ref=branch_pipe.pipe_ref,
+ trigger_variable_names=trigger_scan.trigger_names,
+ absence_source=trigger_scan.trigger_taint.source,
+ ),
+ )
+ if not sub_pipe.output_name:
+ continue
+ if is_plural_step_result(
+ branch_pipe, step_output_multiplicity=sub_pipe.output_multiplicity, has_batch_params=sub_pipe.batch_params is not None
+ ):
+ continue
+ if trigger_scan.trigger_names and trigger_scan.trigger_taint is not None:
+ branch_taints[sub_pipe.output_name] = SlotTaint(
+ source=trigger_scan.trigger_taint.source,
+ origin_slot_name=trigger_scan.trigger_taint.origin_slot_name,
+ chain=(
+ *trigger_scan.trigger_taint.chain,
+ (
+ f"branch '{branch_pipe.code}' may be skipped when '{trigger_scan.trigger_names[0]}' is absent"
+ f" → result '{sub_pipe.output_name}'"
+ ),
+ ),
+ )
+ elif branch_pipe.output.presence.is_optional:
+ branch_taints[sub_pipe.output_name] = SlotTaint(
+ source=f"optional output of branch pipe '{branch_pipe.code}' (result '{sub_pipe.output_name}')",
+ origin_slot_name=sub_pipe.output_name,
+ )
+ return ParallelTaintAnalysis(
+ branch_taints=branch_taints,
+ liftable_steps=tuple(liftable_steps),
+ force_consumptions=tuple(force_consumptions),
+ )
+
+ @override
+ def lifted_companion_slots(self) -> list[CompanionSlot]:
+ """When an `add_each_output` parallel is lifted, every branch result slot it would have
+ written must be resolved too: a recorded absence for singular results, an empty list for
+ plural ones (D4).
+ """
+ if not self.add_each_output:
+ return []
+ companion_slots: list[CompanionSlot] = []
+ for sub_pipe in self.parallel_sub_pipes:
+ if not sub_pipe.output_name:
+ continue
+ branch_pipe = get_optional_pipe(pipe_code=sub_pipe.pipe_code)
+ if branch_pipe is None:
+ continue
+ companion_slots.append(
+ CompanionSlot(
+ slot_name=sub_pipe.output_name,
+ concept=branch_pipe.output.concept,
+ is_plural=is_plural_step_result(
+ branch_pipe,
+ step_output_multiplicity=sub_pipe.output_multiplicity,
+ has_batch_params=sub_pipe.batch_params is not None,
+ ),
+ producing_pipe_code=branch_pipe.code,
+ ),
+ )
+ return companion_slots
+
# Note: builtins.type because the `type: Literal["PipeParallel"]` field shadows the
# builtin in this class body, where signature annotations are evaluated.
def _validate_branch_output_types(self, *, structure_class: builtins.type[StuffContent]) -> None:
@@ -310,21 +457,54 @@ class PipeParallel(PipeController):
pipe_outputs = await asyncio.gather(*tasks)
+ structure_class = self.output.concept.get_structure_class()
+ seen_output_names: set[str] = set()
output_stuffs: dict[str, Stuff] = {}
output_stuff_contents: dict[str, StuffContent] = {}
for output_index, pipe_output in enumerate(pipe_outputs):
- output_stuff = pipe_output.main_stuff
sub_pipe_output_name = self.parallel_sub_pipes[output_index].output_name
if not sub_pipe_output_name:
sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code
msg = f"PipeParallel '{self.code}': sub-pipe '{sub_pipe_code}' output name not specified"
raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
- if sub_pipe_output_name in output_stuffs:
+ if sub_pipe_output_name in seen_output_names:
# TODO: check that at the blueprint / factory level
sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code
msg = f"PipeParallel '{self.code}': sub-pipe '{sub_pipe_code}' duplicate output name '{sub_pipe_output_name}'"
raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
+ seen_output_names.add(sub_pipe_output_name)
+ branch_resolved = pipe_output.working_memory.resolve_main_stuff()
+ if isinstance(branch_resolved, AbsenceRecord):
+ # Combine under absence (D11): a required structured field cannot be fed a hole —
+ # typed error (statically unreachable once Step D's taint pass lands). Otherwise the
+ # absent component is omitted from the combine — a non-required structured field
+ # absorbs it as field-level None — and a ledger note keeps observability.
+ if not issubclass(structure_class, CompositeContent):
+ field_info = structure_class.model_fields.get(sub_pipe_output_name)
+ if field_info is not None and field_info.is_required():
+ sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code
+ msg = (
+ f"PipeParallel '{self.code}': branch '{sub_pipe_code}' resolved absent "
+ f"({branch_resolved.reason}), but field '{sub_pipe_output_name}' of output "
+ f"'{self.output.concept.concept_ref}' is required. Make the field optional "
+ f"(required = false) or sink the absence upstream (a `?` input on the branch path)."
+ )
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)
+ if self.add_each_output:
+ # Branch results are parent slots only under add_each_output — same gate as the
+ # present arm below and as lifted_companion_slots(). Writing the record without
+ # the gate would clobber an unrelated same-named parent value.
+ branch_record = (
+ branch_resolved
+ if branch_resolved.variable_name == sub_pipe_output_name
+ else branch_resolved.model_copy(update={"variable_name": sub_pipe_output_name})
+ )
+ # Resolution, not a note: a stale value under the result name must not outrank it.
+ working_memory.record_resolved_absence(branch_record)
+ log.verbose(f"PipeParallel '{self.code}': branch result '{sub_pipe_output_name}' is absent, omitted from the combine")
+ continue
+ output_stuff = branch_resolved
if self.add_each_output:
working_memory.add_new_stuff(name=sub_pipe_output_name, stuff=output_stuff)
output_stuffs[sub_pipe_output_name] = output_stuff
@@ -332,7 +512,7 @@ class PipeParallel(PipeController):
log.verbose(f"PipeParallel '{self.code}': output_stuff_contents[{sub_pipe_output_name}]: {output_stuff_contents[sub_pipe_output_name]}")
# Always combine the branch outputs into the declared output concept and stamp it as main stuff:
- # a pipe run always delivers a main stuff.
+ # a pipe run always resolves its declared output — the combine is the parallel's value arm.
combined_output_stuff = StuffFactory.combine_stuffs(
concept=self.output.concept,
stuff_contents=output_stuff_contents,
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index 7a859aa4b..d3ce9786d 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -12,6 +12,14 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.pipes.variable_multiplicity import is_multiplicity_compatible
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.hub import get_concept_library, get_optional_pipe, get_required_pipe
+from pipelex.pipe_controllers.absence_taint import (
+ ForceConsumptionInfo,
+ LiftableStepInfo,
+ SequenceTaintAnalysis,
+ SlotTaint,
+ is_plural_step_result,
+ scan_taint_triggers,
+)
from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
from pipelex.pipe_controllers.pipe_controller import PipeController
from pipelex.pipe_controllers.sequence.exceptions import PipeSequenceValueError
@@ -106,6 +114,150 @@ class PipeSequence(PipeController):
required_concept_codes=[self.output.concept.concept_ref],
)
+ # The absence-taint boundary check (D6): a maybe-absent slot ending the sequence must be
+ # matched by an optional (`?`) declared output, or the taint silently escapes the boundary.
+ taint_analysis = self.analyze_taint()
+ if taint_analysis.output_taint is not None and not self.output.presence.is_optional:
+ msg = (
+ f"PipeSequence '{self.code}' output '{self.output.concept.concept_ref}' may resolve absent at run time, "
+ f"but the output is not declared optional. {taint_analysis.output_taint.describe()} "
+ f"Fix: declare the sequence output optional ('{self.output.concept.concept_ref}?'), absorb the absence "
+ f"with an optional input ('X?') on a downstream step and guard its use, or assert presence with a "
+ f"force input ('X!')."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.OPTIONAL_NOT_HANDLED,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ provided_concept_code=self.output.concept.concept_ref,
+ variable_names=[taint_analysis.output_taint.origin_slot_name],
+ )
+
+ def analyze_taint(self) -> SequenceTaintAnalysis:
+ """Static absence-taint walk over the steps (D6), computing per-slot presence.
+
+ Taint enters through the sequence's own `?` inputs and through steps producing optional
+ outputs; it propagates through lifted steps (plain input fed a tainted slot), terminates
+ at `?` (absorb) and `!` (assert) inputs, and never touches plural slots (D4). A step that
+ rewrites a tainted slot with a guaranteed value clears it — the static mirror of the
+ runtime value-supersedes-record invariant.
+ """
+ slot_taints: dict[str, SlotTaint] = {}
+ for input_name, stuff_spec in self.inputs.root.items():
+ if stuff_spec.presence.is_optional and not stuff_spec.is_multiple():
+ slot_taints[input_name] = SlotTaint(
+ source=f"optional input '{input_name}' of pipe '{self.code}'",
+ origin_slot_name=input_name,
+ )
+
+ liftable_steps: list[LiftableStepInfo] = []
+ force_consumptions: list[ForceConsumptionInfo] = []
+ last_step_taint: SlotTaint | None = None
+
+ for sequential_sub_pipe in self.sequential_sub_pipes:
+ sub_pipe = get_optional_pipe(pipe_code=sequential_sub_pipe.pipe_code)
+ if sub_pipe is None:
+ # Unresolved cross-package ref: assume the step delivers (conservative-permissive,
+ # mirroring needed_inputs), so a partially-loaded library never false-errors.
+ last_step_taint = None
+ if sequential_sub_pipe.output_name:
+ slot_taints.pop(sequential_sub_pipe.output_name, None)
+ continue
+
+ # How does this step consume the currently tainted slots?
+ trigger_scan = scan_taint_triggers(sub_pipe, slot_taints=slot_taints)
+ for asserting_name in trigger_scan.asserting_force_names:
+ force_consumptions.append(
+ ForceConsumptionInfo(within_pipe_ref=self.pipe_ref, pipe_ref=sub_pipe.pipe_ref, variable_name=asserting_name, is_asserting=True)
+ )
+ for redundant_name in trigger_scan.redundant_force_names:
+ force_consumptions.append(
+ ForceConsumptionInfo(within_pipe_ref=self.pipe_ref, pipe_ref=sub_pipe.pipe_ref, variable_name=redundant_name, is_asserting=False)
+ )
+ trigger_taint = trigger_scan.trigger_taint
+ step_lifted = bool(trigger_scan.trigger_names) and trigger_taint is not None
+ if step_lifted and trigger_taint is not None:
+ liftable_steps.append(
+ LiftableStepInfo(
+ within_pipe_ref=self.pipe_ref,
+ pipe_ref=sub_pipe.pipe_ref,
+ trigger_variable_names=trigger_scan.trigger_names,
+ absence_source=trigger_taint.source,
+ ),
+ )
+
+ # The step's own output presence. A plural result is never tainted (D4): a lifted
+ # plural output normalizes to an empty list and a batched step compacts.
+ output_slot_name = sequential_sub_pipe.output_name
+ is_plural_result = is_plural_step_result(
+ sub_pipe,
+ step_output_multiplicity=sequential_sub_pipe.output_multiplicity,
+ has_batch_params=sequential_sub_pipe.batch_params is not None,
+ )
+ step_output_taint: SlotTaint | None = None
+ if not is_plural_result:
+ if step_lifted and trigger_taint is not None:
+ step_output_taint = SlotTaint(
+ source=trigger_taint.source,
+ origin_slot_name=trigger_taint.origin_slot_name,
+ chain=(
+ *trigger_taint.chain,
+ f"pipe '{sub_pipe.code}' may be skipped when '{trigger_scan.trigger_names[0]}' is absent"
+ + (f" → slot '{output_slot_name}'" if output_slot_name else ""),
+ ),
+ )
+ elif sub_pipe.output.presence.is_optional:
+ step_output_taint = SlotTaint(
+ source=f"optional output of pipe '{sub_pipe.code}'",
+ origin_slot_name=output_slot_name or sub_pipe.code,
+ )
+
+ # An add_each_output parallel also writes each branch's result slot into this flow.
+ if isinstance(sub_pipe, PipeParallel) and sub_pipe.add_each_output:
+ if step_lifted and trigger_taint is not None:
+ # The whole parallel lifts: companion (branch) slots resolve exactly like the
+ # runtime `_make_lifted_output` does — singular slots go absent, plural slots
+ # become guaranteed empty lists (D4).
+ for companion_slot in sub_pipe.lifted_companion_slots():
+ if companion_slot.is_plural:
+ slot_taints.pop(companion_slot.slot_name, None)
+ else:
+ slot_taints[companion_slot.slot_name] = SlotTaint(
+ source=trigger_taint.source,
+ origin_slot_name=trigger_taint.origin_slot_name,
+ chain=(
+ *trigger_taint.chain,
+ (
+ f"pipe '{sub_pipe.code}' may be skipped when '{trigger_scan.trigger_names[0]}' is absent"
+ f" → branch slot '{companion_slot.slot_name}'"
+ ),
+ ),
+ )
+ else:
+ branch_taints = sub_pipe.analyze_branch_taint().branch_taints
+ for parallel_sub_pipe in sub_pipe.parallel_sub_pipes:
+ if not parallel_sub_pipe.output_name:
+ continue
+ branch_taint = branch_taints.get(parallel_sub_pipe.output_name)
+ if branch_taint is None:
+ slot_taints.pop(parallel_sub_pipe.output_name, None)
+ else:
+ slot_taints[parallel_sub_pipe.output_name] = branch_taint
+
+ if output_slot_name:
+ if step_output_taint is None:
+ slot_taints.pop(output_slot_name, None)
+ else:
+ slot_taints[output_slot_name] = step_output_taint
+ last_step_taint = step_output_taint
+
+ return SequenceTaintAnalysis(
+ liftable_steps=tuple(liftable_steps),
+ output_taint=last_step_taint,
+ force_consumptions=tuple(force_consumptions),
+ )
+
@override
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputStuffSpecs:
if visited_pipes is None:
@@ -158,11 +310,15 @@ class PipeSequence(PipeController):
)
for input_name, stuff_spec in sub_pipe_needed_inputs.items:
if input_name != sequential_sub_pipe.batch_params.input_item_stuff_name and input_name not in generated_outputs:
- needed_inputs.add_stuff_spec(input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
else:
for input_name, stuff_spec in sub_pipe_needed_inputs.items:
if input_name not in generated_outputs:
- needed_inputs.add_stuff_spec(input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
# Add this step's output to generated outputs
if sequential_sub_pipe.output_name:
diff --git a/pipelex/pipe_controllers/sub_pipe.py b/pipelex/pipe_controllers/sub_pipe.py
index 108323e2f..3661e7919 100644
--- a/pipelex/pipe_controllers/sub_pipe.py
+++ b/pipelex/pipe_controllers/sub_pipe.py
@@ -210,23 +210,32 @@ class SubPipe(BaseModel):
# TODO: Merge `needed_inputs` and `required_variables` methods for cleaner code.
required_stuff_names: set[str] = set()
for req_var in required_variables:
- if not req_var.startswith("_"):
- required_stuff_names.add(get_root_from_dotted_path(req_var))
- try:
- required_stuffs = working_memory.get_stuffs(names=required_stuff_names)
- except WorkingMemoryStuffNotFoundError as exc:
+ if req_var.startswith("_"):
+ continue
+ root_name = get_root_from_dotted_path(req_var)
+ # A variable declared optional (`?`) on the sub-pipe is never presence-required
+ # (the `@?` fix, D7): the pipe runs with the slot absent and its guarded
+ # templates handle it.
+ declared_spec = sub_pipe.inputs.root.get(root_name)
+ if declared_spec is not None and declared_spec.presence.is_optional:
+ continue
+ required_stuff_names.add(root_name)
+ # A recorded absence is not a miss: the sub-pipe's own gate applies the trichotomy
+ # (skip / run / force). Only a name with neither a value nor a record is a hard miss.
+ missing_names = working_memory.list_missing_names(names=required_stuff_names)
+ if missing_names:
sub_pipe_path = [*sub_pipe_run_params.pipe_stack, self.pipe_code]
sub_pipe_path_str = ".".join(sub_pipe_path)
- error_details = f"SubPipe '{sub_pipe_path_str}', required_variables: {required_variables}, missing: '{exc.variable_name}'"
+ error_details = f"SubPipe '{sub_pipe_path_str}', required_variables: {required_variables}, missing: '{missing_names[0]}'"
msg = f"Some required stuff(s) not found: {error_details}"
raise PipeRunInputsError(
message=msg,
run_mode=sub_pipe_run_params.run_mode,
pipe_code=self.pipe_code,
- variable_name=exc.variable_name,
+ variable_name=missing_names[0],
concept_code=None,
- ) from exc
- log.verbose(required_stuffs, title=f"Required stuffs for {self.pipe_code}")
+ )
+ log.verbose(working_memory.get_existing_stuffs(names=required_stuff_names), title=f"Required stuffs for {self.pipe_code}")
pipe_output = await get_pipe_router().run(
pipe_job=PipeJobFactory.make_pipe_job(
pipe=sub_pipe,
diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.py
index cdb8c33e8..c431aec58 100644
--- a/pipelex/pipe_operators/compose/pipe_compose.py
+++ b/pipelex/pipe_operators/compose/pipe_compose.py
@@ -15,6 +15,7 @@ from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErr
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.pipes.template_guard_lint import lint_optional_input_guards
from pipelex.core.stuffs.html_content import HtmlContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
@@ -90,12 +91,24 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputStuffSpecs:
needed_inputs = InputStuffSpecsFactory.make_empty()
for input_name, stuff_spec in self.inputs.root.items():
- needed_inputs.add_stuff_spec(variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
return needed_inputs
@override
def validate_inputs_static(self):
- pass
+ # Guard-lint (D7): every template reference to a declared-optional input must be guarded.
+ # Construct mode composes structured fields, not authored templates — nothing to lint there.
+ if self.template is not None:
+ lint_optional_input_guards(
+ pipe_code=self.code,
+ domain_code=self.domain_code,
+ inputs=self.inputs,
+ template_source=self.template,
+ template_category=self.category,
+ template_label="template",
+ )
@override
def validate_inputs_with_library(self):
diff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.py
index f60cc0566..55615184e 100644
--- a/pipelex/pipe_operators/extract/pipe_extract.py
+++ b/pipelex/pipe_operators/extract/pipe_extract.py
@@ -52,7 +52,7 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
@override
def required_variables(self) -> set[str]:
- return set(self.inputs.required_names)
+ return set(self.inputs.declared_names)
@model_validator(mode="after")
def validate_fields(self) -> Self:
diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen.py b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
index 6350a8444..18255eb87 100644
--- a/pipelex/pipe_operators/img_gen/pipe_img_gen.py
+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
@@ -17,6 +17,7 @@ from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.pipes.template_guard_lint import lint_optional_input_guards
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.exceptions import StuffContentTypeError
from pipelex.core.stuffs.image_content import ImageContent
@@ -81,6 +82,22 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):
check_img_gen_choice_with_deck(img_gen_choice=self.img_gen_choice)
self._validate_param_support_against_model_rules()
+ # Guard-lint (D7): every reference to a declared-optional input must be guarded.
+ for template_blueprint, template_label in [
+ (self.img_gen_prompt_blueprint.prompt_blueprint, "prompt"),
+ (self.img_gen_prompt_blueprint.negative_prompt_blueprint, "negative_prompt"),
+ ]:
+ if template_blueprint is None:
+ continue
+ lint_optional_input_guards(
+ pipe_code=self.code,
+ domain_code=self.domain_code,
+ inputs=self.inputs,
+ template_source=template_blueprint.template,
+ template_category=template_blueprint.category,
+ template_label=template_label,
+ )
+
def _validate_param_support_against_model_rules(self) -> None:
"""If `img_gen_choice` resolves to a concrete inference model with rules,
validate that explicitly-set blueprint params are accepted by those rules.
diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.py
index 8760c89a6..b56cf4363 100644
--- a/pipelex/pipe_operators/llm/pipe_llm.py
+++ b/pipelex/pipe_operators/llm/pipe_llm.py
@@ -22,6 +22,7 @@ from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.core.pipes.template_guard_lint import lint_optional_input_guards
from pipelex.core.pipes.validation import is_input_used_by_variables, is_variable_satisfied_by_inputs
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.list_content import ListContent
@@ -92,6 +93,22 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
explanation=f"Variable '{variable_path}' is used in prompt/system_prompt but not declared in inputs.",
)
+ # Guard-lint (D7): every reference to a declared-optional input must be guarded.
+ for template_blueprint, template_label in [
+ (self.llm_prompt_spec.prompt_blueprint, "prompt"),
+ (self.llm_prompt_spec.system_prompt_blueprint, "system_prompt"),
+ ]:
+ if template_blueprint is None:
+ continue
+ lint_optional_input_guards(
+ pipe_code=self.code,
+ domain_code=self.domain_code,
+ inputs=self.inputs,
+ template_source=template_blueprint.template,
+ template_category=template_blueprint.category,
+ template_label=template_label,
+ )
+
@override
def validate_inputs_with_library(self):
pass
@@ -130,7 +147,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
needed_inputs = InputStuffSpecsFactory.make_empty()
for input_name, stuff_spec in self.inputs.items:
- needed_inputs.add_stuff_spec(variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
return needed_inputs
diff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.py
index 2a55aa4c7..add5bf72e 100644
--- a/pipelex/pipe_operators/search/pipe_search.py
+++ b/pipelex/pipe_operators/search/pipe_search.py
@@ -13,6 +13,7 @@ from pipelex.cogt.templating.template_rendering import render_template
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.pipes.template_guard_lint import lint_optional_input_guards
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_content_generator, get_model_deck
from pipelex.pipe_operators.pipe_operator import PipeOperator
@@ -58,6 +59,16 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
msg = f"Search choice '{self.search_choice}' was not found in the model deck"
raise ValueError(msg) from exc
+ # Guard-lint (D7): every reference to a declared-optional input must be guarded.
+ lint_optional_input_guards(
+ pipe_code=self.code,
+ domain_code=self.domain_code,
+ inputs=self.inputs,
+ template_source=self.prompt_blueprint.template,
+ template_category=self.prompt_blueprint.category,
+ template_label="prompt",
+ )
+
@override
def validate_inputs_with_library(self):
pass
diff --git a/pipelex/pipe_operators/structure/pipe_structure.py b/pipelex/pipe_operators/structure/pipe_structure.py
index 7122e1d46..d054ffff7 100644
--- a/pipelex/pipe_operators/structure/pipe_structure.py
+++ b/pipelex/pipe_operators/structure/pipe_structure.py
@@ -48,7 +48,9 @@ class PipeStructure(PipeOperator[PipeStructureOutput]):
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputStuffSpecs:
needed_inputs = InputStuffSpecsFactory.make_empty()
for input_name, stuff_spec in self.inputs.items:
- needed_inputs.add_stuff_spec(variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)
+ needed_inputs.add_stuff_spec(
+ variable_name=input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity, presence=stuff_spec.presence
+ )
return needed_inputs
@override
diff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.py
index 71af5b68a..e35e19a5d 100644
--- a/pipelex/pipe_run/delivery_executor.py
+++ b/pipelex/pipe_run/delivery_executor.py
@@ -11,6 +11,8 @@ from pipelex import log
from pipelex.base_exceptions import DisclosureMode, ErrorReport
from pipelex.config import get_config
from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.absence import AbsenceRecord
+from pipelex.core.memory.absence_render import build_absence_html, build_absence_json, build_absence_markdown
from pipelex.core.memory.working_memory import MAIN_STUFF_NAME
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -89,10 +91,13 @@ class DeliveryExecutor:
render so built-in content types still get typed rendering and
dynamic concepts produce a readable JSON dump.
- A completed run always delivers a main stuff, so the main_stuff.*
- artifact files are always produced — a working memory without one is
- a contract violation that fails the delivery loudly, never an "empty
- envelope" silently missing its result files.
+ A completed run always resolves its declared output: a value or a
+ recorded absence. The main_stuff.* artifact files are always produced —
+ for an absent output they are an explicit absence artifact (never an
+ error: an absent result is a first-class success). A working memory
+ with neither a value nor a recorded absence is a contract violation
+ that fails the delivery loudly, never an "empty envelope" silently
+ missing its result files.
"""
files: dict[str, ResultFile] = {}
@@ -103,20 +108,30 @@ class DeliveryExecutor:
)
raw_main_stuff = self._get_raw_main_stuff_dict(pipe_output.working_memory_raw)
if raw_main_stuff is None:
- msg = "Delivery of a completed run found no main stuff in the raw working memory — a completed run always delivers a main stuff."
- raise PipeJobError(msg)
- hydrated_main_stuff = self.try_local_hydrate_stuff(raw_main_stuff)
- if hydrated_main_stuff is not None:
- await self._generate_main_stuff_files(hydrated_main_stuff, files=files)
+ main_absence = self._get_raw_main_absence(pipe_output.working_memory_raw)
+ if main_absence is None:
+ msg = (
+ "Delivery of a completed run found neither a main stuff nor a recorded absence in the raw "
+ "working memory — a completed run always resolves its declared output."
+ )
+ raise PipeJobError(msg)
+ self._generate_absence_files(main_absence, files=files)
else:
- self._generate_main_stuff_files_from_raw(raw_main_stuff, files=files)
+ hydrated_main_stuff = self.try_local_hydrate_stuff(raw_main_stuff)
+ if hydrated_main_stuff is not None:
+ await self._generate_main_stuff_files(hydrated_main_stuff, files=files)
+ else:
+ self._generate_main_stuff_files_from_raw(raw_main_stuff, files=files)
else:
files["working_memory.json"] = ResultFile(
data=clean_json_dumps(pipe_output.working_memory.smart_dump(), indent=2).encode("utf-8"),
content_type="application/json",
)
- main_stuff = pipe_output.working_memory.get_main_stuff()
- await self._generate_main_stuff_files(main_stuff, files=files)
+ main_resolved = pipe_output.working_memory.resolve_main_stuff()
+ if isinstance(main_resolved, AbsenceRecord):
+ self._generate_absence_files(main_resolved, files=files)
+ else:
+ await self._generate_main_stuff_files(main_resolved, files=files)
graph_spec = pipe_output.graph_spec
if graph_spec:
@@ -135,6 +150,34 @@ class DeliveryExecutor:
return cast("dict[str, Any]", candidate)
return None
+ @classmethod
+ def _get_raw_main_absence(cls, working_memory_raw: dict[str, Any]) -> AbsenceRecord | None:
+ """Extract the recorded main-output absence from a raw working_memory's ledger, if any.
+
+ A malformed record is treated as missing (the caller then fails the delivery loudly as a
+ contract violation) rather than half-rendered.
+ """
+ absences: dict[str, Any] = working_memory_raw.get("absences", {})
+ candidate = absences.get(MAIN_STUFF_NAME)
+ if not isinstance(candidate, dict):
+ return None
+ try:
+ return AbsenceRecord.model_validate(candidate)
+ except ValidationError as exc:
+ log.warning(f"Malformed main-output absence record in raw working memory, treating as missing: {exc}")
+ return None
+
+ @classmethod
+ def _generate_absence_files(cls, absence_record: AbsenceRecord, *, files: dict[str, ResultFile]) -> None:
+ """Render the explicit absence artifact for an absent main output.
+
+ The JSON carries an explicit ``"absent": true`` discriminator beside the record fields so
+ a consumer polling ``main_stuff.json`` can tell an absence document from a value dump.
+ """
+ files["main_stuff.json"] = ResultFile(data=build_absence_json(absence_record).encode("utf-8"), content_type="application/json")
+ files["main_stuff.md"] = ResultFile(data=build_absence_markdown(absence_record).encode("utf-8"), content_type="text/markdown")
+ files["main_stuff.html"] = ResultFile(data=build_absence_html(absence_record).encode("utf-8"), content_type="text/html")
+
@classmethod
def try_local_hydrate_stuff(cls, stuff_raw: dict[str, Any]) -> Stuff | None:
"""Attempt to hydrate a single Stuff dict using only globally-registered classes.
diff --git a/pipelex/pipeline/controller_taint.py b/pipelex/pipeline/controller_taint.py
new file mode 100644
index 000000000..68413577b
--- /dev/null
+++ b/pipelex/pipeline/controller_taint.py
@@ -0,0 +1,39 @@
+"""Single-pass controller taint collection shared by the validate-report projections.
+
+The absence-taint walk (`PipeSequence.analyze_taint`, `PipeParallel.analyze_branch_taint`)
+hub-resolves every sub-pipe and recurses through nested controllers, so it must run exactly
+once per validate pass. Both report projections — the `liftable_pipes` inventory and the
+`warnings` optionality lints — consume the same analyses; callers collect them here and feed
+the list to both builders instead of each builder re-walking the pipes.
+
+Must run while the validation library is loaded — the analyses resolve child pipes through
+the hub.
+"""
+
+from collections.abc import Sequence
+from typing import TypeAlias
+
+from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.pipe_controllers.absence_taint import ParallelTaintAnalysis, SequenceTaintAnalysis
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+
+ControllerTaintAnalysis: TypeAlias = SequenceTaintAnalysis | ParallelTaintAnalysis
+
+
+def collect_controller_taint_analyses(pipes: Sequence[PipeAbstract]) -> list[ControllerTaintAnalysis]:
+ """Run each controller's absence-taint analysis once, in deterministic order (by pipe_ref).
+
+ Args:
+ pipes: The loaded pipes to analyze (typically `ValidateBundleResult.pipes`).
+
+ Returns:
+ One analysis per controller pipe, ordered by the controller's `pipe_ref`.
+ """
+ analyses: list[ControllerTaintAnalysis] = []
+ for pipe in sorted(pipes, key=lambda loaded_pipe: loaded_pipe.pipe_ref):
+ if isinstance(pipe, PipeSequence):
+ analyses.append(pipe.analyze_taint())
+ elif isinstance(pipe, PipeParallel):
+ analyses.append(pipe.analyze_branch_taint())
+ return analyses
diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.py
index 5ee28e333..2d1a400d0 100644
--- a/pipelex/pipeline/execution_seams.py
+++ b/pipelex/pipeline/execution_seams.py
@@ -26,6 +26,7 @@ from mthds.protocol.pipeline_inputs import PipelineInputs
from pipelex import log
from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.pipes.pipe_abstract import PipeAbstract
@@ -217,6 +218,37 @@ async def prepare_pipe_job(
for name, stuff in mock_memory.root.items():
working_memory.add_new_stuff(name=name, stuff=stuff)
+ # Optional method inputs (D5): each declared-optional input the caller omitted starts as a
+ # recorded not-provided absence instead of a bare missing key. Runs after mock filling so the
+ # dry-run sweep stays all-present (D6) — a mocked optional slot gets no record. The marker is
+ # read from the pipe's OWN declared inputs — the method-boundary contract — not from
+ # ``needed_inputs()``, whose aggregation carries a controller's children markers.
+ omitted_optional_specs = [
+ named_stuff_spec
+ for named_stuff_spec in pipe.inputs.named_stuff_specs
+ if named_stuff_spec.presence.is_optional
+ and (
+ working_memory is None
+ or (
+ working_memory.get_optional_stuff(named_stuff_spec.variable_name) is None
+ # A slot already resolved as a recorded absence was NOT omitted — re-recording
+ # would downgrade its provenance to a fresh not-provided.
+ and working_memory.get_optional_absence(named_stuff_spec.variable_name) is None
+ )
+ )
+ ]
+ if omitted_optional_specs:
+ if working_memory is None:
+ working_memory = WorkingMemoryFactory.make_empty()
+ for named_stuff_spec in omitted_optional_specs:
+ working_memory.record_absence(
+ AbsenceRecord(
+ variable_name=named_stuff_spec.variable_name,
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason=f"optional input '{named_stuff_spec.variable_name}' was not provided by the caller",
+ )
+ )
+
# Normalize data URLs to pipelex-storage:// URIs if configured.
if working_memory and execution_config.is_normalize_data_urls_to_storage and not execution_config.is_mock_inputs:
working_memory = await normalize_data_urls_to_storage(working_memory)
diff --git a/pipelex/pipeline/liftable_pipes.py b/pipelex/pipeline/liftable_pipes.py
new file mode 100644
index 000000000..bed70fccd
--- /dev/null
+++ b/pipelex/pipeline/liftable_pipes.py
@@ -0,0 +1,58 @@
+"""Liftable-pipe inventory (`liftable_pipes`) for the validate surfaces.
+
+Implicit lifting (optionals design D3) means a pipe with a plain input fed by a maybe-absent
+slot is silently skipped at run time. The phase-1 commitment that makes this acceptable is
+build-time visibility: the valid report lists every pipe that may be lifted, with the slots
+whose absence triggers the skip and the absence source — structured data beside
+`pipe_io_contracts`, keyed the same way (namespaced `pipe_ref`).
+"""
+
+from collections.abc import Sequence
+
+from pydantic import BaseModel, Field
+
+from pipelex.pipeline.controller_taint import ControllerTaintAnalysis
+
+
+class LiftablePipeEntry(BaseModel):
+ """One pipe that may be skipped (lifted) when an optional slot resolves absent."""
+
+ pipe_ref: str
+ """Namespaced ref of the liftable pipe."""
+
+ within_pipe_ref: str
+ """Namespaced ref of the controller in whose flow the lift happens."""
+
+ skipped_when_absent: list[str] = Field(default_factory=list)
+ """The slot names whose absence lifts the pipe."""
+
+ absence_source: str
+ """Where the possible absence originates (human-readable)."""
+
+
+def build_liftable_pipes(taint_analyses: Sequence[ControllerTaintAnalysis]) -> list[LiftablePipeEntry]:
+ """Project the controllers' taint analyses into `liftable_pipes` entries.
+
+ Sequences contribute their lifted steps; parallels contribute their liftable branches
+ (a parallel nested in a sequence reports its branches under itself, so there is no
+ double-reporting).
+
+ Args:
+ taint_analyses: The controllers' analyses from `collect_controller_taint_analyses`
+ (one taint walk per validate pass, shared with `build_optionality_warnings`).
+
+ Returns:
+ Entries in deterministic order (by controller ref, then flow order).
+ """
+ entries: list[LiftablePipeEntry] = []
+ for taint_analysis in taint_analyses:
+ for liftable_step in taint_analysis.liftable_steps:
+ entries.append(
+ LiftablePipeEntry(
+ pipe_ref=liftable_step.pipe_ref,
+ within_pipe_ref=liftable_step.within_pipe_ref,
+ skipped_when_absent=list(liftable_step.trigger_variable_names),
+ absence_source=liftable_step.absence_source,
+ ),
+ )
+ return entries
diff --git a/pipelex/pipeline/optionality_warnings.py b/pipelex/pipeline/optionality_warnings.py
new file mode 100644
index 000000000..e75d19569
--- /dev/null
+++ b/pipelex/pipeline/optionality_warnings.py
@@ -0,0 +1,78 @@
+"""Advisory optionality lints (`warnings`) for the validate surfaces.
+
+The validation report's `warnings` array carries findings that are worth surfacing on a VALID
+bundle but must never flip `is_valid`. The first occupant is the useless-`!` lint
+(`optional_force_redundant`): a `!` (force) input asserts at run time that a maybe-absent slot
+is present — but when every analyzed flow guarantees the slot, the assertion can never fire and
+the marker is dead weight.
+
+The presence data comes from the same controller taint analyses that feed the liftable-pipe
+inventory (`PipeSequence.analyze_taint`, `PipeParallel.analyze_branch_taint`). Observations are
+aggregated per (pipe, variable) ACROSS flows before warning: a pipe whose `!` is meaningful in
+one flow is never told to remove it just because another flow guarantees the slot. A `!` input
+on a pipe that appears in no analyzed flow yields no observation and no warning (conservative —
+there is no flow context to judge it in).
+
+The over-claimed-`?` lint (a `?` output that can never produce absence) is deliberately NOT
+emitted in phase 1: D5 blesses over-claiming as signature-evolution room.
+"""
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from pipelex.base_exceptions import ValidationErrorCategory, ValidationErrorItem
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
+from pipelex.pipeline.controller_taint import ControllerTaintAnalysis
+
+if TYPE_CHECKING:
+ from pipelex.pipe_controllers.absence_taint import ForceConsumptionInfo
+
+
+def build_optionality_warnings(taint_analyses: Sequence[ControllerTaintAnalysis]) -> list[ValidationErrorItem]:
+ """Project the controllers' taint analyses into advisory `warnings` items.
+
+ Args:
+ taint_analyses: The controllers' analyses from `collect_controller_taint_analyses`
+ (one taint walk per validate pass, shared with `build_liftable_pipes`).
+
+ Returns:
+ One `optional_force_redundant` item per (pipe, `!`-input) whose slot is guaranteed
+ present in every analyzed flow, in deterministic order (by pipe_ref, then variable).
+ """
+ observations: list[ForceConsumptionInfo] = []
+ for taint_analysis in taint_analyses:
+ observations.extend(taint_analysis.force_consumptions)
+
+ asserted: set[tuple[str, str]] = set()
+ redundant: dict[tuple[str, str], list[ForceConsumptionInfo]] = {}
+ for observation in observations:
+ consumption_key = (observation.pipe_ref, observation.variable_name)
+ if observation.is_asserting:
+ asserted.add(consumption_key)
+ else:
+ redundant.setdefault(consumption_key, []).append(observation)
+
+ warnings: list[ValidationErrorItem] = []
+ for consumption_key in sorted(redundant):
+ if consumption_key in asserted:
+ continue
+ pipe_ref, variable_name = consumption_key
+ domain_code, _, pipe_code = pipe_ref.partition(".")
+ flow_refs = sorted({observation.within_pipe_ref for observation in redundant[consumption_key]})
+ flows_desc = ", ".join(f"'{flow_ref}'" for flow_ref in flow_refs)
+ message = (
+ f"Input '{variable_name}' of pipe '{pipe_ref}' is declared with a force marker ('!'), but the slot is "
+ f"guaranteed present in every analyzed flow ({flows_desc}) — the assertion can never fire there. "
+ f"Remove the '!' (plain input), unless a flow that feeds it a maybe-absent value is coming."
+ )
+ warnings.append(
+ ValidationErrorItem(
+ category=ValidationErrorCategory.PIPE_VALIDATION,
+ error_type=PipeValidationErrorType.OPTIONAL_FORCE_REDUNDANT,
+ pipe_code=pipe_code,
+ domain_code=domain_code,
+ variable_names=[variable_name],
+ message=message,
+ )
+ )
+ return warnings
diff --git a/pipelex/pipeline/pipe_io_contracts.py b/pipelex/pipeline/pipe_io_contracts.py
index bb75bd9b1..de47c75de 100644
--- a/pipelex/pipeline/pipe_io_contracts.py
+++ b/pipelex/pipeline/pipe_io_contracts.py
@@ -43,6 +43,10 @@ class PipeInputContract(BaseModel):
"""One declared input: the concept it expects and the JSON Schema of its content."""
concept_ref: str
+ optional: bool = False
+ """`True` when the input is declared optional (`?`): the caller may omit it and the pipe
+ handles absence itself. A plain or force (`!`) input reports `False` — it must be provided."""
+
json_schema: dict[str, Any] = Field(default_factory=dict)
@@ -51,6 +55,9 @@ class PipeOutputContract(BaseModel):
concept_ref: str
multiplicity: IOMultiplicity
+ optional: bool = False
+ """`True` when the output is declared optional (`?`): the pipe may resolve it as a recorded
+ absence instead of a value — a successful run with an absent result."""
class PipeIOContract(BaseModel):
@@ -107,11 +114,13 @@ def build_pipe_io_contracts(pipes: Sequence[PipeAbstract]) -> dict[str, PipeIOCo
schema_memo[memo_key] = json_schema
pipe_inputs[var_name] = PipeInputContract(
concept_ref=stuff_spec.concept.concept_ref,
+ optional=stuff_spec.presence.is_optional,
json_schema=json_schema,
)
pipe_output = PipeOutputContract(
concept_ref=pipe.output.concept.concept_ref,
multiplicity=IOMultiplicity.VARIABLE if pipe.output.is_multiple() else IOMultiplicity.SINGLE,
+ optional=pipe.output.presence.is_optional,
)
io_contracts[pipe.pipe_ref] = PipeIOContract(inputs=pipe_inputs, output=pipe_output)
return io_contracts
diff --git a/pipelex/pipeline/pipeline_response.py b/pipelex/pipeline/pipeline_response.py
index 3dd1e8e87..53ebc9a1c 100644
--- a/pipelex/pipeline/pipeline_response.py
+++ b/pipelex/pipeline/pipeline_response.py
@@ -31,7 +31,9 @@ class PipelexRunResultExecute(RunResultExecute[PipeOutput]):
created_at: str
state: RunState
finished_at: str | None = None
- # A completed run always delivers a main stuff; this is the actual working-memory root key it lives under.
+ # A completed run always resolves its declared output: a value or a recorded absence. This names
+ # the working-memory root key the value lives under — or, when the output resolved absent, the
+ # declared slot the absence record is keyed by (consumers branch on the record).
main_stuff_name: str
@classmethod
diff --git a/pipelex/pipeline/validate_in_process.py b/pipelex/pipeline/validate_in_process.py
index 27e5fa41c..e7b4fdc8c 100644
--- a/pipelex/pipeline/validate_in_process.py
+++ b/pipelex/pipeline/validate_in_process.py
@@ -24,7 +24,7 @@ from pathlib import Path
from typing import TYPE_CHECKING
from pipelex import log
-from pipelex.base_exceptions import PipelexError
+from pipelex.base_exceptions import PipelexError, ValidationErrorItem
from pipelex.hub import (
clear_current_library,
get_current_library_id_or_none,
@@ -33,6 +33,9 @@ from pipelex.hub import (
)
from pipelex.pipe_run.dry_run_in_process import best_effort_graph_spec
from pipelex.pipeline.blueprint_selection import select_primary_blueprint
+from pipelex.pipeline.controller_taint import collect_controller_taint_analyses
+from pipelex.pipeline.liftable_pipes import LiftablePipeEntry, build_liftable_pipes
+from pipelex.pipeline.optionality_warnings import build_optionality_warnings
from pipelex.pipeline.pipe_io_contracts import PipeIOContract, build_pipe_io_contracts
from pipelex.pipeline.validate_bundle import validate_bundle
from pipelex.pipeline.validation_report import PipelexValidationReport, build_validation_report
@@ -107,6 +110,10 @@ async def validate_bundles_in_process(
# something inside the window later moves the contextvar.
validation_library_id = get_current_library_id_or_none()
pipe_io_contracts: dict[str, PipeIOContract] = build_pipe_io_contracts(result.pipes)
+ # One taint walk per validate pass — both report projections read the same analyses.
+ taint_analyses = collect_controller_taint_analyses(result.pipes)
+ liftable_pipes: list[LiftablePipeEntry] = build_liftable_pipes(taint_analyses)
+ warnings: list[ValidationErrorItem] = build_optionality_warnings(taint_analyses)
graph_target_ref = graph_pipe_code if graph_pipe_code is not None else select_primary_blueprint(result.blueprints).main_pipe_ref
graph_spec: GraphSpec | None = await best_effort_graph_spec(
pipe_ref=graph_target_ref,
@@ -138,7 +145,9 @@ async def validate_bundles_in_process(
return build_validation_report(
blueprints=result.blueprints,
pipe_io_contracts=pipe_io_contracts,
+ liftable_pipes=liftable_pipes,
dry_run_result=result.dry_run_result,
pending_signatures=result.pending_signatures,
graph_spec=graph_spec,
+ warnings=warnings,
)
diff --git a/pipelex/pipeline/validation_render.py b/pipelex/pipeline/validation_render.py
index 62d312f6c..4efe17253 100644
--- a/pipelex/pipeline/validation_render.py
+++ b/pipelex/pipeline/validation_render.py
@@ -84,6 +84,13 @@ def format_validate_markdown(result: dict[str, Any]) -> str:
),
]
+ # Advisory lints (e.g. the useless-`!` lint) — informational: the bundle IS valid.
+ warnings: list[dict[str, Any]] = result.get("warnings") or []
+ if warnings:
+ lines += ["", f"## Warnings ({len(warnings)})", ""]
+ for warning in warnings:
+ lines.append(f"- **{warning.get('error_type') or warning.get('category')}** — {warning.get('message')}")
+
graph_files = result.get("graph_files")
if isinstance(graph_files, dict):
lines += ["", "## Graph files", ""]
diff --git a/pipelex/pipeline/validation_report.py b/pipelex/pipeline/validation_report.py
index ebab5a9ea..ca7f5218a 100644
--- a/pipelex/pipeline/validation_report.py
+++ b/pipelex/pipeline/validation_report.py
@@ -17,10 +17,12 @@ from typing import Literal
from mthds.protocol.models import ValidationReport
from pydantic import Field
+from pipelex.base_exceptions import ValidationErrorItem
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.graph.graphspec import GraphSpec
from pipelex.pipeline.blueprint_selection import select_primary_blueprint
from pipelex.pipeline.bundle_validator import DryRunOutput
+from pipelex.pipeline.liftable_pipes import LiftablePipeEntry
from pipelex.pipeline.pipe_io_contracts import PipeIOContract
from pipelex.pipeline.validate_bundle import ValidatedPipeEntry, build_validated_pipes
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
@@ -46,6 +48,10 @@ class PipelexValidationReport(ValidationReport):
pipe_io_contracts: dict[str, PipeIOContract] = Field(default_factory=dict)
"""Per-pipe input/output contracts, keyed by namespaced `pipe_ref` (`domain.code`)."""
+ liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry))
+ """Pipes that may be skipped (lifted) when an optional slot resolves absent — the
+ build-time visibility that the implicit-lifting design (D3) commits to."""
+
graph_spec: GraphSpec | None = None
"""Best-effort execution graph of the declared main pipe; `None` when the batch
declares no `main_pipe` or the graph dry-run degrades."""
@@ -53,6 +59,11 @@ class PipelexValidationReport(ValidationReport):
validated_pipes: list[ValidatedPipeEntry] = Field(default_factory=empty_list_factory_of(ValidatedPipeEntry))
"""Per-pipe sweep outcomes (`{pipe_ref, status}`) — the same entries as the agent-CLI envelope."""
+ warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem))
+ """Advisory findings (lints) on a VALID bundle — same item shape as `validation_errors`,
+ but they never flip `is_valid`. First occupant: the useless-`!` lint
+ (`optional_force_redundant`)."""
+
pending_signatures: list[str] = Field(default_factory=list)
"""Qualified refs of pipes still declared as `PipeSignature` in the assembled library."""
@@ -67,6 +78,8 @@ def build_validation_report(
dry_run_result: dict[str, DryRunOutput],
pending_signatures: list[str],
graph_spec: GraphSpec | None = None,
+ liftable_pipes: list[LiftablePipeEntry] | None = None,
+ warnings: list[ValidationErrorItem] | None = None,
) -> PipelexValidationReport:
"""Assemble the canonical `PipelexValidationReport` from its ingredients.
@@ -80,6 +93,8 @@ def build_validation_report(
dry_run_result: The sweep's per-pipe status map (`ValidateBundleResult.dry_run_result`).
pending_signatures: Library-wide unsatisfied signature refs.
graph_spec: Best-effort graph of the declared main pipe, when one was produced.
+ liftable_pipes: Entries from `build_liftable_pipes`, when the caller computed them.
+ warnings: Advisory items from `build_optionality_warnings`, when the caller computed them.
Returns:
The canonical report.
@@ -87,8 +102,10 @@ def build_validation_report(
return PipelexValidationReport(
bundle_blueprint=select_primary_blueprint(blueprints).blueprint,
pipe_io_contracts=pipe_io_contracts,
+ liftable_pipes=liftable_pipes or [],
graph_spec=graph_spec,
validated_pipes=build_validated_pipes(dry_run_result),
pending_signatures=pending_signatures,
is_runnable=not pending_signatures,
+ warnings=warnings or [],
)
diff --git a/pipelex/runtime_bridge/payloads.py b/pipelex/runtime_bridge/payloads.py
index ec46fbd1e..fb14329ce 100644
--- a/pipelex/runtime_bridge/payloads.py
+++ b/pipelex/runtime_bridge/payloads.py
@@ -48,7 +48,9 @@ class PipelexPipeRunOutput(BaseModel):
model_config = ConfigDict(extra="forbid")
output_dict: dict[str, Any]
- # A completed run always delivers a main stuff; this is the actual `root` key it lives under.
+ # A completed run always resolves its declared output: a value or a recorded absence. This names
+ # the `root` key the value lives under — or, when the output resolved absent, the declared slot
+ # that keys the record in ``output_dict["absences"]`` (consumers branch on the record).
main_stuff_name: str
pipeline_run_id: str
workflow_id: str | None = None
diff --git a/pipelex/runtime_bridge/primitives/hydration.py b/pipelex/runtime_bridge/primitives/hydration.py
index 2e4590c3f..adf9a31d2 100644
--- a/pipelex/runtime_bridge/primitives/hydration.py
+++ b/pipelex/runtime_bridge/primitives/hydration.py
@@ -4,6 +4,7 @@ from kajson.exceptions import KajsonException
from pydantic import ValidationError
from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.absence import AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.composite_content import CompositeContent
from pipelex.core.stuffs.list_content import ListContent
@@ -143,6 +144,9 @@ def hydrate_working_memory(working_memory_raw: dict[str, Any]) -> WorkingMemory:
Must be called AFTER load_from_crate() has registered dynamic classes
in the scoped ClassRegistry. Uses concept.structure_class_name to look up
the correct StuffContent subclass from the registry.
+
+ The absence ledger round-trips too: a recorded absence must survive cross-process
+ transit, or a resolved-as-absent slot would degrade to a hard miss on the other side.
"""
working_memory = WorkingMemory()
@@ -166,4 +170,13 @@ def hydrate_working_memory(working_memory_raw: dict[str, Any]) -> WorkingMemory:
working_memory.root[stuff_name] = stuff
working_memory.aliases = working_memory_raw.get("aliases", {})
+
+ raw_absences: dict[str, Any] = working_memory_raw.get("absences", {})
+ for absence_name, absence_raw in raw_absences.items():
+ try:
+ working_memory.absences[absence_name] = AbsenceRecord.model_validate(absence_raw)
+ except ValidationError as exc:
+ msg = f"Failed to hydrate absence record '{absence_name}': {exc}"
+ raise PipeJobError(msg) from exc
+
return working_memory
diff --git a/pipelex/runtime_bridge/serialization.py b/pipelex/runtime_bridge/serialization.py
index 001e3e7f4..94eddd467 100644
--- a/pipelex/runtime_bridge/serialization.py
+++ b/pipelex/runtime_bridge/serialization.py
@@ -77,15 +77,21 @@ def serialize_completed_output(
def resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str:
- """Return the actual ``root`` dict key under which the main stuff lives.
+ """Return the key under which the run's resolved main result lives — always the declared slot.
The main stuff can either sit directly at ``root[MAIN_STUFF_NAME]`` or be
referenced via ``aliases[MAIN_STUFF_NAME]`` pointing at its real name.
Callers indexing the output_dict need the actual root key, not the
stuff's display ``stuff_name``.
- A completed run always delivers a main stuff — a working memory without
- one at this boundary is a contract violation and raises ``PipeJobError``.
+ When the main output resolved as a recorded absence (an optional output that produced
+ nothing), the returned key names the declared output slot and indexes the serialized
+ ``absences`` ledger instead of ``root`` — consumers branch on the absence record, and
+ the run stays a success.
+
+ A completed run always resolves its declared output: a value or a recorded absence. A
+ working memory with neither at this boundary is a contract violation and raises
+ ``PipeJobError``.
"""
working_memory = pipe_output.working_memory
if MAIN_STUFF_NAME in working_memory.root:
@@ -93,7 +99,11 @@ def resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str:
aliased_target = working_memory.aliases.get(MAIN_STUFF_NAME)
if aliased_target is not None and aliased_target in working_memory.root:
return aliased_target
+ main_absence = working_memory.get_optional_absence(MAIN_STUFF_NAME)
+ if main_absence is not None:
+ return main_absence.variable_name
msg = (
- f"Completed run '{pipe_output.pipeline_run_id}' delivered no main stuff in its working memory — a completed run always delivers a main stuff."
+ f"Completed run '{pipe_output.pipeline_run_id}' resolved neither a main stuff nor a recorded absence in its "
+ f"working memory — a completed run always resolves its declared output: a value or a recorded absence."
)
raise PipeJobError(msg)
diff --git a/pipelex/system/telemetry/otel_factory.py b/pipelex/system/telemetry/otel_factory.py
index df5fe511b..8487fb5bc 100644
--- a/pipelex/system/telemetry/otel_factory.py
+++ b/pipelex/system/telemetry/otel_factory.py
@@ -7,6 +7,8 @@ import base64
import hashlib
from typing import TYPE_CHECKING, Any
+from pipelex.core.memory.absence import AbsenceRecord
+from pipelex.core.memory.absence_render import build_absence_payload
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.system.environment import get_optional_env
@@ -77,7 +79,16 @@ class OtelFactory:
"""
inputs_dict: dict[str, Any] = {}
for input_name in needed_input_names:
- stuff = working_memory.get_stuff(name=input_name)
+ # An input slot resolved as a recorded absence (an omitted optional, a lifted
+ # producer) captures as the explicit absence document. A slot with neither a
+ # value nor a record stays the loud hard-miss raise below.
+ optional_stuff = working_memory.get_optional_stuff(name=input_name)
+ if optional_stuff is None:
+ absence_record = working_memory.get_optional_absence(input_name)
+ if absence_record is not None:
+ inputs_dict[input_name] = build_absence_payload(absence_record)
+ continue
+ stuff = optional_stuff or working_memory.get_stuff(name=input_name)
inputs_dict[input_name] = {
"concept": stuff.concept.simple_concept_ref,
"content": stuff.content.smart_dump(),
@@ -96,19 +107,25 @@ class OtelFactory:
"""Serialize pipe output to JSON for telemetry.
Args:
- pipe_output: The pipe output containing the main stuff.
+ pipe_output: The pipe output containing the resolved main result.
max_length: Maximum allowed length for the JSON string, or None for no limit.
Returns:
JSON string representing the output, potentially truncated.
"""
- # Called on the success path only — a completed pipe run always delivers a main stuff.
- main_stuff = pipe_output.working_memory.get_main_stuff()
-
- output_dict: dict[str, Any] = {
- "concept": main_stuff.concept.simple_concept_ref,
- "content": main_stuff.content.smart_dump(),
- }
+ # Called on the success path only — a completed pipe run always resolves its declared
+ # output: a value or a recorded absence. An absence serializes to the explicit absence
+ # document (the span still ends successfully).
+ main_resolved = pipe_output.working_memory.resolve_main_stuff()
+
+ output_dict: dict[str, Any]
+ if isinstance(main_resolved, AbsenceRecord):
+ output_dict = build_absence_payload(main_resolved)
+ else:
+ output_dict = {
+ "concept": main_resolved.concept.simple_concept_ref,
+ "content": main_resolved.content.smart_dump(),
+ }
json_str = cls.stringify_json(json_conent=output_dict)
return cls.make_truncated_content(content=json_str, max_length=max_length)
diff --git a/pipelex/tools/jinja2/jinja2_optional_guards.py b/pipelex/tools/jinja2/jinja2_optional_guards.py
new file mode 100644
index 000000000..3e73e650d
--- /dev/null
+++ b/pipelex/tools/jinja2/jinja2_optional_guards.py
@@ -0,0 +1,248 @@
+"""Guard classification for declared-optional template variables (optionals design D7).
+
+An absent optional input is simply undefined in the Jinja context: `{% if var %}` is falsy and
+`@?var` renders nothing, but a bare `{{ var }}` silently renders empty and a deep `{{ var.x }}`
+raises at render time. The static companion that makes this design safe is the guard-lint: every
+template reference to a declared-optional input must be *guarded* — reachable only inside a
+`{% if var %}`-style block, an inline presence conditional (`... if var is defined else ...`),
+or via `@?var` (whose rewritten form is a `{% if var %}` block). This module classifies the
+references; validation turns the unguarded ones into `OPTIONAL_INPUT_UNGUARDED` errors.
+
+Recognized guard shapes (kept deliberately narrow — a conservative lint with a precise fix beats
+a clever one that blesses subtly unsafe templates):
+
+- an `{% if %}` / inline-conditional test that guards the variable: the bare variable name, a
+ `var is defined` test, or an `and` combination containing one of those;
+- inside a test position, a bare name or a presence test (`defined` / `undefined` / `none`) is
+ itself a safe reference (truthiness of an undefined name is a legal presence probe); any other
+ shape rooted at an unguarded optional (deep access, filters, other tests) is unguarded.
+
+The `{% else %}` arm of a guard is NOT guarded — it is exactly the branch that runs when the
+variable is absent.
+"""
+
+from jinja2 import nodes
+from jinja2.exceptions import TemplateSyntaxError
+from pydantic.dataclasses import dataclass
+
+from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.tools.jinja2.exceptions import Jinja2DetectVariablesError
+from pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loader
+from pipelex.tools.misc.string_utils import get_root_from_dotted_path
+
+# Test names that safely probe presence on an undefined variable.
+_PRESENCE_TEST_NAMES = {"defined", "undefined", "none"}
+
+
+@dataclass(frozen=True)
+class UnguardedOptionalReference:
+ """One unguarded template reference to a declared-optional variable."""
+
+ variable_name: str
+ path: str
+
+
+def _build_full_path(node: nodes.Node) -> str | None:
+ """Build the dotted path of a Name / Getattr chain, or None for unsupported shapes."""
+ if isinstance(node, nodes.Name):
+ return node.name
+ if isinstance(node, nodes.Getattr):
+ parent_path = _build_full_path(node.node)
+ if parent_path is not None:
+ return f"{parent_path}.{node.attr}"
+ return None
+
+
+class _GuardWalker:
+ """Recursive AST walk tracking which optional variables are currently guarded and which
+ names are locally declared (loop targets, macro params, `{% set %}` assignments).
+ """
+
+ def __init__(self, *, optional_variable_names: set[str]) -> None:
+ self.optional_variable_names = optional_variable_names
+ self.findings: list[UnguardedOptionalReference] = []
+ self._seen_paths: set[str] = set()
+
+ def _record(self, full_path: str) -> None:
+ root_name = get_root_from_dotted_path(full_path)
+ if full_path in self._seen_paths:
+ return
+ self._seen_paths.add(full_path)
+ self.findings.append(UnguardedOptionalReference(variable_name=root_name, path=full_path))
+
+ def _is_unguarded_optional(self, full_path: str, *, guarded: frozenset[str], declared: frozenset[str]) -> bool:
+ """Whether the dotted path references an optional variable that is neither guarded
+ nor shadowed by a local declaration.
+ """
+ root_name = get_root_from_dotted_path(full_path)
+ return root_name in self.optional_variable_names and root_name not in guarded and root_name not in declared
+
+ def _guard_vars(self, test_node: nodes.Node) -> frozenset[str]:
+ """Variables positively guaranteed present inside the body guarded by `test_node`."""
+ if isinstance(test_node, nodes.Name):
+ return frozenset({test_node.name})
+ if isinstance(test_node, nodes.Test) and test_node.name == "defined" and isinstance(test_node.node, nodes.Name):
+ return frozenset({test_node.node.name})
+ if isinstance(test_node, nodes.And):
+ return self._guard_vars(test_node.left) | self._guard_vars(test_node.right)
+ return frozenset()
+
+ def _walk_test(self, test_node: nodes.Node, *, guarded: frozenset[str], declared: frozenset[str]) -> None:
+ """Walk a test position: bare names and presence tests are safe references there."""
+ if isinstance(test_node, nodes.Name):
+ return
+ if isinstance(test_node, nodes.Test) and test_node.name in _PRESENCE_TEST_NAMES and isinstance(test_node.node, nodes.Name):
+ return
+ if isinstance(test_node, nodes.And):
+ # `and` short-circuits: the right operand only evaluates when the left is truthy,
+ # so the left operand's guard extends over the right (`{% if var and var.attr %}`).
+ self._walk_test(test_node.left, guarded=guarded, declared=declared)
+ self._walk_test(test_node.right, guarded=guarded | self._guard_vars(test_node.left), declared=declared)
+ return
+ if isinstance(test_node, nodes.Or):
+ self._walk_test(test_node.left, guarded=guarded, declared=declared)
+ self._walk_test(test_node.right, guarded=guarded, declared=declared)
+ return
+ if isinstance(test_node, nodes.Not):
+ self._walk_test(test_node.node, guarded=guarded, declared=declared)
+ return
+ self.walk(test_node, guarded=guarded, declared=declared)
+
+ def _walk_body(self, body_nodes: list[nodes.Node], *, guarded: frozenset[str], declared: frozenset[str]) -> None:
+ """Walk a statement body sequentially: a `{% set %}` or `{% macro %}` declares its name
+ for SUBSEQUENT statements only — a read occurring before the assignment still refers to
+ the (possibly undefined) context value and must be classified against it.
+ """
+ for body_node in body_nodes:
+ if isinstance(body_node, nodes.Assign):
+ # The assignment's right-hand side is evaluated against the current scope.
+ self.walk(body_node.node, guarded=guarded, declared=declared)
+ declared |= self._assign_target_names(body_node.target)
+ continue
+ if isinstance(body_node, nodes.AssignBlock):
+ # `{% set x %}...{% endset %}`: the body (and filter) evaluate against the
+ # current scope; the target declares for subsequent statements only — and the
+ # target itself is a store, never a read.
+ self._walk_body(body_node.body, guarded=guarded, declared=declared)
+ if body_node.filter is not None:
+ self.walk(body_node.filter, guarded=guarded, declared=declared)
+ declared |= self._assign_target_names(body_node.target)
+ continue
+ if isinstance(body_node, nodes.Macro):
+ declared |= {body_node.name}
+ self.walk(body_node, guarded=guarded, declared=declared)
+
+ @classmethod
+ def _assign_target_names(cls, target: nodes.Node) -> frozenset[str]:
+ """The names a `{% set %}` target declares — a plain Name or a Tuple of Names."""
+ if isinstance(target, nodes.Name):
+ return frozenset({target.name})
+ if isinstance(target, nodes.Tuple):
+ return frozenset(item.name for item in target.items if isinstance(item, nodes.Name))
+ return frozenset()
+
+ def walk(self, node: nodes.Node, *, guarded: frozenset[str], declared: frozenset[str]) -> None:
+ if isinstance(node, nodes.Template):
+ self._walk_body(node.body, guarded=guarded, declared=declared)
+ return
+
+ if isinstance(node, nodes.If):
+ body_guarded = guarded | self._guard_vars(node.test)
+ self._walk_test(node.test, guarded=guarded, declared=declared)
+ self._walk_body(node.body, guarded=body_guarded, declared=declared)
+ for elif_node in node.elif_:
+ self.walk(elif_node, guarded=guarded, declared=declared)
+ self._walk_body(node.else_, guarded=guarded, declared=declared)
+ return
+
+ if isinstance(node, nodes.For):
+ # The iterable is evaluated before the loop targets bind.
+ self.walk(node.iter, guarded=guarded, declared=declared)
+ loop_declared: set[str] = {"loop"}
+ if isinstance(node.target, nodes.Name):
+ loop_declared.add(node.target.name)
+ elif isinstance(node.target, nodes.Tuple):
+ for item in node.target.items:
+ if isinstance(item, nodes.Name):
+ loop_declared.add(item.name)
+ if node.test is not None:
+ # The loop filter evaluates after the target binds per item, so a shadowing
+ # target stays a local while an optional read in the filter gets classified.
+ # Its guard vars deliberately do NOT bless the body (same conservative stance
+ # as else-arms and inverted guards).
+ self._walk_test(node.test, guarded=guarded, declared=declared | loop_declared)
+ self._walk_body(node.body, guarded=guarded, declared=declared | loop_declared)
+ self._walk_body(node.else_, guarded=guarded, declared=declared)
+ return
+
+ if isinstance(node, nodes.With):
+ # The bound values are evaluated before the targets bind; the body then reads the
+ # locals, never the shadowed optionals. Walking the targets as children would
+ # miscount the store-context Names as reads.
+ for value in node.values:
+ self.walk(value, guarded=guarded, declared=declared)
+ with_declared = frozenset(target.name for target in node.targets if isinstance(target, nodes.Name))
+ self._walk_body(node.body, guarded=guarded, declared=declared | with_declared)
+ return
+
+ if isinstance(node, nodes.Macro):
+ macro_declared = frozenset(arg.name for arg in node.args)
+ self._walk_body(node.body, guarded=guarded, declared=declared | macro_declared)
+ return
+
+ if isinstance(node, nodes.CondExpr):
+ expr1_guarded = guarded | self._guard_vars(node.test)
+ self._walk_test(node.test, guarded=guarded, declared=declared)
+ self.walk(node.expr1, guarded=expr1_guarded, declared=declared)
+ if node.expr2 is not None:
+ self.walk(node.expr2, guarded=guarded, declared=declared)
+ return
+
+ if isinstance(node, (nodes.Name, nodes.Getattr)):
+ full_path = _build_full_path(node)
+ if full_path is None:
+ # Not a plain Name/Getattr chain (e.g. an attribute on a subscript or call
+ # result): keep walking inward so inner references still get classified.
+ for child in node.iter_child_nodes():
+ self.walk(child, guarded=guarded, declared=declared)
+ return
+ if self._is_unguarded_optional(full_path, guarded=guarded, declared=declared):
+ self._record(full_path)
+ # Never recurse into a resolvable Name/Getattr chain: the full path is the reference.
+ return
+
+ for child in node.iter_child_nodes():
+ self.walk(child, guarded=guarded, declared=declared)
+
+
+def detect_unguarded_optional_references(
+ *,
+ template_category: TemplateCategory,
+ template_source: str,
+ optional_variable_names: set[str],
+) -> list[UnguardedOptionalReference]:
+ """Return every unguarded reference to a declared-optional variable in the template.
+
+ Args:
+ template_category: Category of the template (LLM_PROMPT, EXPRESSION, etc.)
+ template_source: Jinja2 template source (sigils already rewritten to Jinja2).
+ optional_variable_names: Root names of the pipe's declared-optional (`?`) inputs.
+
+ Returns:
+ One entry per distinct unguarded dotted path, in template order.
+
+ Raises:
+ Jinja2DetectVariablesError: If the template cannot be parsed.
+ """
+ if not optional_variable_names:
+ return []
+ jinja2_env = make_jinja2_env_without_loader(template_category=template_category)
+ try:
+ parsed_ast = jinja2_env.parse(template_source)
+ except TemplateSyntaxError as syntax_error:
+ msg = f"Jinja2 guard lint — syntax error: '{syntax_error}', template_category: {template_category}, template_source:\n{template_source}"
+ raise Jinja2DetectVariablesError(msg) from syntax_error
+
+ walker = _GuardWalker(optional_variable_names=optional_variable_names)
+ walker.walk(parsed_ast, guarded=frozenset(), declared=frozenset())
+ return walker.findings
diff --git a/pipelex/tracing/graphspec_assembler.py b/pipelex/tracing/graphspec_assembler.py
index 438cbe521..1cf619d54 100644
--- a/pipelex/tracing/graphspec_assembler.py
+++ b/pipelex/tracing/graphspec_assembler.py
@@ -23,6 +23,7 @@ from pipelex.graph.graphspec import (
NodeStatus,
PipelineRef,
TimingSpec,
+ output_digest_is_optional,
)
from pipelex.tracing.trace_events import (
BatchAggregateEvent,
@@ -32,6 +33,7 @@ from pipelex.tracing.trace_events import (
ExecutionDataEvent,
ParallelCombineEvent,
PipeEndErrorEvent,
+ PipeEndSkippedEvent,
PipeEndSuccessEvent,
PipeStartEvent,
TraceEvent,
@@ -67,6 +69,7 @@ class _AssemblerNodeData:
self.domain_code = domain_code
self.ended_at: datetime | None = None
self.status: NodeStatus = NodeStatus.RUNNING
+ self.skip_reason: str | None = None
self.metrics: dict[str, float] = {}
self.error: ErrorSpec | None = None
self.input_specs: list[IOSpec] = input_specs or []
@@ -95,6 +98,7 @@ class _AssemblerNodeData:
description=self.description,
domain_code=self.domain_code,
status=self.status,
+ skip_reason=self.skip_reason,
timing=timing,
node_io=node_io,
error=self.error,
@@ -171,6 +175,8 @@ class _AssemblerState:
self._handle_pipe_end_success(event)
elif isinstance(event, PipeEndErrorEvent):
self._handle_pipe_end_error(event)
+ elif isinstance(event, PipeEndSkippedEvent):
+ self._handle_pipe_end_skipped(event)
elif isinstance(event, EdgeEvent):
self._handle_edge_event(event)
elif isinstance(event, ControllerOutputEvent):
@@ -283,6 +289,29 @@ class _AssemblerState:
node_data.status = NodeStatus.FAILED
node_data.error = event.error
+ def _handle_pipe_end_skipped(self, event: PipeEndSkippedEvent) -> None:
+ node_data = self._nodes.get(event.node_id)
+ if node_data is None:
+ log.warning(f"PipeEndSkippedEvent for unknown node: {event.node_id}")
+ return
+
+ node_data.ended_at = event.ended_at
+ node_data.status = NodeStatus.SKIPPED
+ node_data.skip_reason = event.skip_reason
+
+ # A lifted pipe with a PLURAL output still wrote a real empty-list Stuff (D4) — register
+ # it so downstream DATA edges resolve (mirrors GraphTracer.on_pipe_end_skipped).
+ if event.output_concept_data:
+ concept_ref = f"{event.output_concept_data.get('domain_code', '')}.{event.output_concept_data.get('code', '')}"
+ if concept_ref not in self._concept_registry:
+ self._concept_registry[concept_ref] = event.output_concept_data
+ if event.output_spec is not None:
+ input_digests = {spec.digest for spec in node_data.input_specs if spec.digest is not None}
+ if event.output_spec.digest not in input_digests:
+ node_data.output_specs.append(event.output_spec)
+ if event.output_spec.digest:
+ self._stuff_producer_map[event.output_spec.digest] = event.node_id
+
def _handle_edge_event(self, event: EdgeEvent) -> None:
# DATA, BATCH_ITEM, BATCH_AGGREGATE, PARALLEL_COMBINE are regenerated
# in pass 2 with full cross-worker visibility — skip to avoid duplicates.
@@ -293,6 +322,7 @@ class _AssemblerState:
source=event.source_node_id,
target=event.target_node_id,
kind=event.edge_kind,
+ optional=event.optional,
label=event.label,
source_stuff_digest=event.source_stuff_digest,
target_stuff_digest=event.target_stuff_digest,
@@ -366,18 +396,30 @@ class _AssemblerState:
label: str | None = None,
source_stuff_digest: str | None = None,
target_stuff_digest: str | None = None,
+ optional: bool = False,
) -> None:
edge = EdgeSpec(
edge_id=self._make_edge_id(),
source=source_node_id,
target=target_node_id,
kind=edge_kind,
+ optional=optional,
label=label,
source_stuff_digest=source_stuff_digest,
target_stuff_digest=target_stuff_digest,
)
self._generated_edges.append(edge)
+ def _is_optional_output_digest(self, *, producer_node_id: str, digest: str) -> bool:
+ """Whether the producer registered this digest as a declared-optional (`?`) output.
+
+ Node lookup here; the marker semantics live in the shared `output_digest_is_optional`.
+ """
+ producer_data = self._nodes.get(producer_node_id)
+ if producer_data is None:
+ return False
+ return output_digest_is_optional(producer_data.output_specs, digest=digest)
+
def _generate_data_edges(self) -> None:
"""Generate DATA edges by correlating input digests with producer nodes.
@@ -397,6 +439,7 @@ class _AssemblerState:
target_node_id=consumer_node_id,
edge_kind=EdgeKind.DATA,
label=input_spec.name,
+ optional=self._is_optional_output_digest(producer_node_id=producer_node_id, digest=input_spec.digest),
)
def _generate_batch_item_edges(self) -> None:
diff --git a/pipelex/tracing/trace_events.py b/pipelex/tracing/trace_events.py
index 0e646d988..521f24700 100644
--- a/pipelex/tracing/trace_events.py
+++ b/pipelex/tracing/trace_events.py
@@ -22,6 +22,7 @@ class TraceEventKind(StrEnum):
PIPE_START = "pipe_start"
PIPE_END_SUCCESS = "pipe_end_success"
PIPE_END_ERROR = "pipe_end_error"
+ PIPE_END_SKIPPED = "pipe_end_skipped"
EDGE = "edge"
CONTROLLER_OUTPUT = "controller_output"
BATCH_ITEM = "batch_item"
@@ -96,6 +97,24 @@ class PipeEndErrorEvent(TraceEvent):
error: ErrorSpec
+class PipeEndSkippedEvent(TraceEvent):
+ """Emitted when a pipe is lifted (skipped) because a plain input resolved absent (D3).
+
+ A skip is a successful outcome — the run continues and the pipe's output is a recorded
+ absence — but it gets its own node state so graph consumers can render it distinctly.
+ """
+
+ event_kind: Literal[TraceEventKind.PIPE_END_SKIPPED] = TraceEventKind.PIPE_END_SKIPPED
+ node_id: str
+ ended_at: datetime
+ skip_reason: str
+ output_spec: IOSpec | None = None
+ """A lifted pipe with a PLURAL output still writes a real empty-list Stuff (D4) — its spec
+ rides the skip event so the digest registers in the producer map and downstream DATA edges
+ resolve. None for a singular output (a recorded absence has no payload)."""
+ output_concept_data: dict[str, Any] = Field(default_factory=dict)
+
+
class EdgeEvent(TraceEvent):
"""Emitted when an edge is added (CONTAINS, SELECTED_OUTCOME, etc.)."""
@@ -104,6 +123,7 @@ class EdgeEvent(TraceEvent):
source_node_id: str
target_node_id: str
edge_kind: EdgeKind
+ optional: bool = False
label: str | None = None
source_stuff_digest: str | None = None
target_stuff_digest: str | None = None
@@ -175,6 +195,7 @@ AnyTraceEvent = Annotated[
PipeStartEvent
| PipeEndSuccessEvent
| PipeEndErrorEvent
+ | PipeEndSkippedEvent
| EdgeEvent
| ControllerOutputEvent
| BatchItemEvent
diff --git a/tests/e2e/pipelex/pipes/optionals/guarded_condition/guarded_condition.mthds b/tests/e2e/pipelex/pipes/optionals/guarded_condition/guarded_condition.mthds
new file mode 100644
index 000000000..b39d90a00
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/guarded_condition/guarded_condition.mthds
@@ -0,0 +1,27 @@
+domain = "opt_guarded_condition"
+description = "PipeCondition whose expression guards a declared-optional input"
+main_pipe = "ogc_route_on_flag"
+
+[pipe.ogc_make_flagged_note]
+type = "PipeFunc"
+description = "Branch taken when the flag is present"
+output = "Text"
+function_name = "ogc_make_flagged_note"
+
+[pipe.ogc_make_plain_note]
+type = "PipeFunc"
+description = "Branch taken when the flag is absent"
+output = "Text"
+function_name = "ogc_make_plain_note"
+
+[pipe.ogc_route_on_flag]
+type = "PipeCondition"
+description = "Routes on presence of the optional flag (guarded expression)"
+inputs = { flag = "Text?" }
+output = "Text"
+expression_template = "{{ 'with_flag' if flag is defined else 'no_flag' }}"
+default_outcome = "ogc_make_plain_note"
+
+[pipe.ogc_route_on_flag.outcomes]
+with_flag = "ogc_make_flagged_note"
+no_flag = "ogc_make_plain_note"
diff --git a/tests/e2e/pipelex/pipes/optionals/guarded_condition/guarded_condition_funcs.py b/tests/e2e/pipelex/pipes/optionals/guarded_condition/guarded_condition_funcs.py
new file mode 100644
index 000000000..903ab83de
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/guarded_condition/guarded_condition_funcs.py
@@ -0,0 +1,17 @@
+"""PipeFunc implementations for the guarded-condition bundle, auto-registered at library load via
+the @pipe_func decorator (FuncRegistryUtils.register_funcs_in_folder).
+"""
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.system.registries.func_registry import pipe_func
+
+
+@pipe_func()
+async def ogc_make_flagged_note(working_memory: WorkingMemory) -> TextContent: # noqa: ARG001,RUF029
+ return TextContent(text="flagged-branch")
+
+
+@pipe_func()
+async def ogc_make_plain_note(working_memory: WorkingMemory) -> TextContent: # noqa: ARG001,RUF029
+ return TextContent(text="plain-branch")
diff --git a/tests/e2e/pipelex/pipes/optionals/lift_parity/lift_parity.mthds b/tests/e2e/pipelex/pipes/optionals/lift_parity/lift_parity.mthds
new file mode 100644
index 000000000..f0431cbb2
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/lift_parity/lift_parity.mthds
@@ -0,0 +1,27 @@
+domain = "opt_lift_parity"
+description = "Pure lift chain used to pin dry/live absence-ledger and graph parity"
+main_pipe = "opar_flow"
+
+[pipe.opar_make_analysis]
+type = "PipeFunc"
+description = "Analyzes the source (plain input: lifts when source is absent)"
+inputs = { source = "Text" }
+output = "Text"
+function_name = "opar_make_analysis"
+
+[pipe.opar_summarize]
+type = "PipeFunc"
+description = "Summarizes the analysis (plain input: chained lift)"
+inputs = { analysis = "Text" }
+output = "Text"
+function_name = "opar_summarize"
+
+[pipe.opar_flow]
+type = "PipeSequence"
+description = "Whole chain lifts when source is absent"
+inputs = { source = "Text?" }
+output = "Text?"
+steps = [
+ { pipe = "opar_make_analysis", result = "analysis" },
+ { pipe = "opar_summarize", result = "summary" },
+]
diff --git a/tests/e2e/pipelex/pipes/optionals/lift_parity/lift_parity_funcs.py b/tests/e2e/pipelex/pipes/optionals/lift_parity/lift_parity_funcs.py
new file mode 100644
index 000000000..bc7382d7a
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/lift_parity/lift_parity_funcs.py
@@ -0,0 +1,17 @@
+"""PipeFunc implementations for the lift-parity bundle, auto-registered at library load via the
+@pipe_func decorator (FuncRegistryUtils.register_funcs_in_folder).
+"""
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.system.registries.func_registry import pipe_func
+
+
+@pipe_func()
+async def opar_make_analysis(working_memory: WorkingMemory) -> TextContent: # noqa: RUF029
+ return TextContent(text=f"analysis of {working_memory.get_stuff_as_str(name='source')}")
+
+
+@pipe_func()
+async def opar_summarize(working_memory: WorkingMemory) -> TextContent: # noqa: RUF029
+ return TextContent(text=f"summary: {working_memory.get_stuff_as_str(name='analysis')}")
diff --git a/tests/e2e/pipelex/pipes/optionals/optionals_showcase/optionals_showcase.mthds b/tests/e2e/pipelex/pipes/optionals/optionals_showcase/optionals_showcase.mthds
new file mode 100644
index 000000000..864213ef9
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/optionals_showcase/optionals_showcase.mthds
@@ -0,0 +1,84 @@
+domain = "opt_showcase"
+description = "Optionals showcase: lift chain, continue condition, absorbing sink, force flow"
+main_pipe = "oshow_main_flow"
+
+[pipe.oshow_analyze_clause]
+type = "PipeFunc"
+description = "Analyzes the clause (plain input: lifts when clause is absent)"
+inputs = { clause = "Text" }
+output = "Text"
+function_name = "oshow_analyze_clause"
+
+[pipe.oshow_summarize_analysis]
+type = "PipeFunc"
+description = "Summarizes the analysis (plain input: chained lift)"
+inputs = { analysis = "Text" }
+output = "Text"
+function_name = "oshow_summarize_analysis"
+
+[pipe.oshow_flag_risk]
+type = "PipeFunc"
+description = "Produces the risk note for a flagged contract"
+inputs = { contract = "Text" }
+output = "Text"
+function_name = "oshow_flag_risk"
+
+[pipe.oshow_assess_risk]
+type = "PipeCondition"
+description = "Flags a risk when the contract mentions a penalty; otherwise continues"
+inputs = { contract = "Text" }
+output = "Text?"
+expression_template = "{{ 'flagged' if 'penalty' in contract.text else 'clean' }}"
+default_outcome = "continue"
+
+[pipe.oshow_assess_risk.outcomes]
+flagged = "oshow_flag_risk"
+
+[pipe.oshow_compose_report]
+type = "PipeCompose"
+description = "Absorbing sink: composes the report whether or not the optional slots resolved"
+inputs = { contract = "Text", clause_summary = "Text?", risk_note = "Text?" }
+output = "Text"
+template = """
+Report for: $contract
+@?clause_summary
+{% if risk_note is defined %}Risk: {{ risk_note.text }}{% else %}No risk assessment.{% endif %}
+"""
+
+[pipe.oshow_main_flow]
+type = "PipeSequence"
+description = "Lift chain + continue condition + absorbing sink delivering a guaranteed report"
+inputs = { contract = "Text", clause = "Text?" }
+output = "Text"
+steps = [
+ { pipe = "oshow_analyze_clause", result = "analysis" },
+ { pipe = "oshow_summarize_analysis", result = "clause_summary" },
+ { pipe = "oshow_assess_risk", result = "risk_note" },
+ { pipe = "oshow_compose_report", result = "report" },
+]
+
+# The force flow is nested on purpose: the method boundary seeds the NOT_PROVIDED record for the
+# `?` input, then the inner step's `!` marker asserts presence and raises the typed error.
+[pipe.oshow_force_extract]
+type = "PipeFunc"
+description = "Asserts presence of the clause with a force marker"
+inputs = { clause = "Text!" }
+output = "Text"
+function_name = "oshow_force_extract"
+
+[pipe.oshow_force_flow]
+type = "PipeSequence"
+description = "Force flow: optional at the boundary, asserted inside"
+inputs = { clause = "Text?" }
+output = "Text"
+steps = [{ pipe = "oshow_force_extract", result = "extracted" }]
+
+[pipe.oshow_lift_only_flow]
+type = "PipeSequence"
+description = "Whole chain lifts when clause is absent; output declared optional"
+inputs = { clause = "Text?" }
+output = "Text?"
+steps = [
+ { pipe = "oshow_analyze_clause", result = "analysis" },
+ { pipe = "oshow_summarize_analysis", result = "clause_summary" },
+]
diff --git a/tests/e2e/pipelex/pipes/optionals/optionals_showcase/optionals_showcase_funcs.py b/tests/e2e/pipelex/pipes/optionals/optionals_showcase/optionals_showcase_funcs.py
new file mode 100644
index 000000000..0df7c5f90
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/optionals_showcase/optionals_showcase_funcs.py
@@ -0,0 +1,27 @@
+"""PipeFunc implementations for the optionals showcase bundle, auto-registered at library load
+via the @pipe_func decorator (FuncRegistryUtils.register_funcs_in_folder).
+"""
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.system.registries.func_registry import pipe_func
+
+
+@pipe_func()
+async def oshow_analyze_clause(working_memory: WorkingMemory) -> TextContent: # noqa: RUF029
+ return TextContent(text=f"analysis of {working_memory.get_stuff_as_str(name='clause')}")
+
+
+@pipe_func()
+async def oshow_summarize_analysis(working_memory: WorkingMemory) -> TextContent: # noqa: RUF029
+ return TextContent(text=f"summary: {working_memory.get_stuff_as_str(name='analysis')}")
+
+
+@pipe_func()
+async def oshow_flag_risk(working_memory: WorkingMemory) -> TextContent: # noqa: ARG001,RUF029
+ return TextContent(text="penalty risk detected")
+
+
+@pipe_func()
+async def oshow_force_extract(working_memory: WorkingMemory) -> TextContent: # noqa: RUF029
+ return TextContent(text=f"extracted: {working_memory.get_stuff_as_str(name='clause')}")
diff --git a/tests/e2e/pipelex/pipes/optionals/prompt_guard/prompt_guard.mthds b/tests/e2e/pipelex/pipes/optionals/prompt_guard/prompt_guard.mthds
new file mode 100644
index 000000000..652cc0528
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/prompt_guard/prompt_guard.mthds
@@ -0,0 +1,17 @@
+domain = "opt_prompt_guard"
+description = "PipeLLM whose prompt guards a declared-optional input (dry-run keyless)"
+main_pipe = "opg_draft_note"
+
+[pipe.opg_draft_note]
+type = "PipeLLM"
+description = "Drafts a note; the optional style hint is guarded via the optional sigil and an if-defined block"
+inputs = { topic = "Text", style_hint = "Text?" }
+output = "Text"
+prompt = """
+Write one short sentence about $topic.
+
+@?style_hint
+{% if style_hint is defined %}
+Follow the style hint above.
+{% endif %}
+"""
diff --git a/tests/e2e/pipelex/pipes/optionals/test_guarded_condition_mthds.py b/tests/e2e/pipelex/pipes/optionals/test_guarded_condition_mthds.py
new file mode 100644
index 000000000..826a51cd2
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/test_guarded_condition_mthds.py
@@ -0,0 +1,74 @@
+"""Runtime branch selection by a PipeCondition whose expression guards a declared-optional input,
+from a `.mthds` file: present selects the flagged branch, absent evaluates the guard over an
+undefined variable and selects the fallback — plus a negative control proving the load gate
+rejects the unguarded variant.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from pipelex.base_exceptions import PipelexError
+from pipelex.core.memory.absence import AbsenceKind
+from pipelex.pipeline.pipeline_response import RunState
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.registries.func_registry_utils import FuncRegistryUtils
+
+_FIXTURE_DIR = Path(__file__).parent / "guarded_condition"
+_BUNDLE_PATH = _FIXTURE_DIR / "guarded_condition.mthds"
+
+
[email protected](loop_scope="class")
+class TestGuardedConditionExpression:
+ async def test_flag_present_selects_with_flag_branch(self):
+ """With the optional flag provided, the guarded expression evaluates truthy and routes to
+ the flagged branch.
+ """
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)])
+
+ response = await runner.execute(pipe_code="ogc_route_on_flag", inputs={"flag": "urgent"})
+
+ assert response.state == RunState.COMPLETED
+ assert response.pipe_output.main_stuff.as_text.text == "flagged-branch"
+ assert response.pipe_output.working_memory.absences == {}
+
+ async def test_flag_absent_selects_no_flag_branch_and_keeps_record(self):
+ """With the flag omitted, the expression evaluates the guard over an undefined variable,
+ routes to the fallback branch, and the not-provided record persists.
+ """
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)])
+
+ response = await runner.execute(pipe_code="ogc_route_on_flag", inputs={})
+
+ assert response.state == RunState.COMPLETED
+ assert response.pipe_output.main_stuff.as_text.text == "plain-branch"
+ flag_record = response.pipe_output.working_memory.get_optional_absence("flag")
+ assert flag_record is not None
+ assert flag_record.kind == AbsenceKind.NOT_PROVIDED
+
+ async def test_unguarded_variant_fails_at_load(self):
+ """Negative control: the same bundle with an unguarded expression over the optional input
+ is rejected by the load-time guard lint — the passing bundle passes because it guards.
+ """
+ guarded_text = _BUNDLE_PATH.read_text(encoding="utf-8")
+ unguarded_text = guarded_text.replace(
+ """expression_template = "{{ 'with_flag' if flag is defined else 'no_flag' }}\"""",
+ """expression_template = "{{ flag.text }}\"""",
+ )
+ assert unguarded_text != guarded_text
+
+ # Register the bundle's @pipe_func impls so PipeFunc construction succeeds and the load-time
+ # guard lint — not a missing-function error — is what rejects the unguarded variant. Without
+ # this, the raw-content load path never walks a folder to register funcs, so construction
+ # fails first and the failure "passes" for the wrong reason (and only on some xdist workers).
+ FuncRegistryUtils.register_funcs_in_folder(folder_path=_FIXTURE_DIR)
+
+ runner = PipelexMTHDSProtocol()
+ with pytest.raises(PipelexError) as exc_info:
+ await runner.execute(mthds_contents=[unguarded_text], inputs={})
+
+ # The load-time guard lint (OPTIONAL_INPUT_UNGUARDED) is raised inside a pydantic validator,
+ # so it surfaces wrapped in a PipeExecutionError → pydantic ValidationError rather than on the
+ # __cause__ chain. Match its distinctive verdict text: the missing-function error (the wrong
+ # reason this control used to pass for) never contains "unguarded".
+ assert "'flag' unguarded" in str(exc_info.value)
diff --git a/tests/e2e/pipelex/pipes/optionals/test_lift_parity_dry_live.py b/tests/e2e/pipelex/pipes/optionals/test_lift_parity_dry_live.py
new file mode 100644
index 000000000..ee667198e
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/test_lift_parity_dry_live.py
@@ -0,0 +1,73 @@
+"""Dry/live parity for a lifted chain loaded from a `.mthds` file: the lift gate is run-mode
+independent, so a dry run (without mock inputs) must produce the same absence ledger and the same
+skipped graph nodes as the live run.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceRecord
+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME
+from pipelex.graph.graphspec import GraphSpec, NodeStatus
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipeline.pipeline_response import RunState
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+
+_FIXTURE_DIR = Path(__file__).parent / "lift_parity"
+
+
+def _skipped_nodes(graph_spec: GraphSpec) -> dict[str | None, str | None]:
+ return {node.pipe_code: node.skip_reason for node in graph_spec.nodes if node.status == NodeStatus.SKIPPED}
+
+
[email protected](loop_scope="class")
+class TestLiftedChainDryLiveParity:
+ async def test_dry_and_live_absence_ledgers_and_skips_match(self):
+ """Running the lift chain with the optional source omitted, live and dry (no mock
+ seeding), yields identical ledgers (keys, kinds, producing pipes) and identical skipped
+ graph nodes with the same skip reasons.
+ """
+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True)
+
+ live_runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)], execution_config=execution_config)
+ live_response = await live_runner.execute(pipe_code="opar_flow", inputs={})
+
+ dry_runner = PipelexMTHDSProtocol(
+ library_dirs=[str(_FIXTURE_DIR)],
+ execution_config=execution_config,
+ pipe_run_mode=PipeRunMode.DRY,
+ )
+ dry_response = await dry_runner.execute(pipe_code="opar_flow", inputs={})
+
+ for response in [live_response, dry_response]:
+ assert response.state == RunState.COMPLETED
+ assert response.main_stuff_name == "summary"
+ assert isinstance(response.pipe_output.working_memory.resolve_main_stuff(), AbsenceRecord)
+
+ live_memory = live_response.pipe_output.working_memory
+ dry_memory = dry_response.pipe_output.working_memory
+ # The absent main output records under its slot name AND the positional main-stuff key.
+ assert set(live_memory.absences.keys()) == set(dry_memory.absences.keys()) == {"source", "analysis", "summary", MAIN_STUFF_NAME}
+ for name in live_memory.absences:
+ live_record = live_memory.absences[name]
+ dry_record = dry_memory.absences[name]
+ assert live_record.kind == dry_record.kind
+ assert live_record.producing_pipe == dry_record.producing_pipe
+
+ live_graph = live_response.pipe_output.graph_spec
+ dry_graph = dry_response.pipe_output.graph_spec
+ assert live_graph is not None
+ assert dry_graph is not None
+ live_skips = _skipped_nodes(live_graph)
+ dry_skips = _skipped_nodes(dry_graph)
+ assert set(live_skips.keys()) == set(dry_skips.keys()) == {"opar_make_analysis", "opar_summarize"}
+ for pipe_code, live_skip_reason in live_skips.items():
+ assert live_skip_reason is not None
+ assert live_skip_reason == dry_skips[pipe_code]
+
+ live_controller_statuses = {node.pipe_code: node.status for node in live_graph.nodes}
+ dry_controller_statuses = {node.pipe_code: node.status for node in dry_graph.nodes}
+ assert live_controller_statuses["opar_flow"] == NodeStatus.SUCCEEDED
+ assert dry_controller_statuses["opar_flow"] == NodeStatus.SUCCEEDED
diff --git a/tests/e2e/pipelex/pipes/optionals/test_optionals_showcase_mthds.py b/tests/e2e/pipelex/pipes/optionals/test_optionals_showcase_mthds.py
new file mode 100644
index 000000000..e913c915a
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/test_optionals_showcase_mthds.py
@@ -0,0 +1,159 @@
+"""End-to-end optionals showcase from a real `.mthds` bundle file on disk: the lift chain, the
+absorbing `?` sink, the `continue` condition, and the `!` force flow all behave per the language
+promise when loaded through the normal library-loading path — including the plain-CLI run core
+writing the explicit absence artifact.
+"""
+
+import json
+from pathlib import Path
+
+import pytest
+
+from pipelex.cli.commands.run._run_core import _execute_run # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceKind
+from pipelex.core.pipes.inputs.exceptions import OptionalValueAbsentError
+from pipelex.graph.graphspec import NodeStatus
+from pipelex.pipeline.exceptions import PipelineExecutionError
+from pipelex.pipeline.pipeline_response import RunState
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+
+_FIXTURE_DIR = Path(__file__).parent / "optionals_showcase"
+
+
+def _find_cause(exc: BaseException, *, cause_type: type[BaseException]) -> BaseException | None:
+ """Walk the __cause__ chain to find the first exception of the given type."""
+ current: BaseException | None = exc
+ while current is not None:
+ if isinstance(current, cause_type):
+ return current
+ current = current.__cause__
+ return None
+
+
[email protected](loop_scope="class")
+class TestOptionalsShowcaseE2E:
+ async def test_optional_provided_runs_everything_ledger_empty(self):
+ """With the optional clause provided and a penalty-bearing contract, nothing lifts: the
+ summary rides into the report, the condition takes the flagged branch, and the ledger
+ stays empty.
+ """
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)])
+
+ response = await runner.execute(
+ pipe_code="oshow_main_flow",
+ inputs={"contract": "contains a penalty clause", "clause": "clause 12"},
+ )
+
+ assert response.state == RunState.COMPLETED
+ assert response.pipe_output.working_memory.absences == {}
+ report_text = response.pipe_output.main_stuff.as_text.text
+ assert "summary: analysis of clause 12" in report_text
+ assert "Risk: penalty risk detected" in report_text
+
+ async def test_optional_absent_lifts_absorbs_and_records(self):
+ """Omitting the optional clause lifts the chain with provenance, the clean contract takes
+ the `continue` arm, and the absorbing sink still delivers the main output — with the graph
+ marking the lifted pipes as skipped.
+ """
+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True)
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)], execution_config=execution_config)
+
+ response = await runner.execute(pipe_code="oshow_main_flow", inputs={"contract": "a clean contract"})
+
+ assert response.state == RunState.COMPLETED
+ report_text = response.pipe_output.main_stuff.as_text.text
+ assert "No risk assessment." in report_text
+ assert "summary: analysis" not in report_text
+
+ memory = response.pipe_output.working_memory
+ assert set(memory.absences.keys()) == {"clause", "analysis", "clause_summary", "risk_note"}
+
+ analysis_record = memory.get_optional_absence("analysis")
+ assert analysis_record is not None
+ assert analysis_record.kind == AbsenceKind.SKIPPED
+ assert analysis_record.producing_pipe == "oshow_analyze_clause"
+ assert analysis_record.origin().variable_name == "clause"
+ assert analysis_record.origin().kind == AbsenceKind.NOT_PROVIDED
+
+ summary_record = memory.get_optional_absence("clause_summary")
+ assert summary_record is not None
+ assert summary_record.kind == AbsenceKind.SKIPPED
+ assert summary_record.producing_pipe == "oshow_summarize_analysis"
+ assert summary_record.origin().variable_name == "clause"
+
+ risk_record = memory.get_optional_absence("risk_note")
+ assert risk_record is not None
+ assert risk_record.kind == AbsenceKind.DECLARED_ABSENT
+ assert risk_record.producing_pipe == "oshow_assess_risk"
+
+ graph_spec = response.pipe_output.graph_spec
+ assert graph_spec is not None
+ nodes_by_code = {node.pipe_code: node for node in graph_spec.nodes}
+ assert nodes_by_code["oshow_analyze_clause"].status == NodeStatus.SKIPPED
+ analyze_skip_reason = nodes_by_code["oshow_analyze_clause"].skip_reason
+ assert analyze_skip_reason is not None
+ assert "clause" in analyze_skip_reason
+ assert nodes_by_code["oshow_summarize_analysis"].status == NodeStatus.SKIPPED
+ assert nodes_by_code["oshow_assess_risk"].status == NodeStatus.SUCCEEDED
+ assert nodes_by_code["oshow_compose_report"].status == NodeStatus.SUCCEEDED
+
+ async def test_force_flow_absent_raises_typed_error(self):
+ """The `!` marker on an absent slot surfaces the typed force error with provenance
+ through the pipeline front door.
+ """
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)])
+
+ with pytest.raises(PipelineExecutionError) as exc_info:
+ await runner.execute(pipe_code="oshow_force_flow", inputs={})
+
+ force_error = _find_cause(exc_info.value, cause_type=OptionalValueAbsentError)
+ assert isinstance(force_error, OptionalValueAbsentError)
+ assert force_error.variable_name == "clause"
+ assert force_error.pipe_code == "oshow_force_extract"
+ assert force_error.absence_record.origin().kind == AbsenceKind.NOT_PROVIDED
+
+ async def test_force_flow_present_runs(self):
+ """With the clause provided, the force marker is satisfied and the flow delivers."""
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)])
+
+ response = await runner.execute(pipe_code="oshow_force_flow", inputs={"clause": "clause 12"})
+
+ assert response.state == RunState.COMPLETED
+ assert response.pipe_output.main_stuff.as_text.text == "extracted: clause 12"
+
+ async def test_cli_run_core_writes_absence_artifact(self, tmp_path: Path):
+ """The unmocked plain-CLI run core (`run pipe <code> -L <dir>` shape), with the optional
+ omitted, writes the explicit absence artifact (json + md, no interactive viewer) and the
+ working memory dump carrying the ledger. Loading via the library dir (not bundle_path)
+ also auto-registers the co-located @pipe_func implementations.
+ """
+ await _execute_run(
+ "oshow_lift_only_flow",
+ bundle_path=None,
+ inputs=None,
+ save_working_memory=True,
+ working_memory_path=None,
+ save_main_stuff=True,
+ no_pretty_print=True,
+ graph=None,
+ graph_full_data=None,
+ output_dir=str(tmp_path),
+ dry_run=False,
+ mock_usage=False,
+ mock_inputs=False,
+ library_dir=[str(_FIXTURE_DIR)],
+ )
+
+ output_dirs = list(tmp_path.glob("oshow_lift_only_flow_output*"))
+ assert len(output_dirs) == 1
+ output_path = output_dirs[0]
+
+ absence_payload = json.loads((output_path / "main_stuff.json").read_text(encoding="utf-8"))
+ assert absence_payload["absent"] is True
+ assert absence_payload["variable_name"] == "clause_summary"
+ assert (output_path / "main_stuff.md").exists()
+ assert not (output_path / "main_stuff_viewer.html").exists()
+
+ memory_dump = json.loads((output_path / "working_memory.json").read_text(encoding="utf-8"))
+ assert "clause" in memory_dump["absences"]
diff --git a/tests/e2e/pipelex/pipes/optionals/test_prompt_guard_dry_run.py b/tests/e2e/pipelex/pipes/optionals/test_prompt_guard_dry_run.py
new file mode 100644
index 000000000..30c9d2497
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/optionals/test_prompt_guard_dry_run.py
@@ -0,0 +1,49 @@
+"""Dry-run PipeLLM with a guarded optional prompt variable, from a `.mthds` file: dry mode rides
+the live render path (only the cogt leaf mocks), so completing the run proves the guarded template
+rendered correctly with the optional input genuinely absent — no API key involved.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from pipelex.core.memory.absence import AbsenceKind
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipeline.pipeline_response import RunState
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+
+_FIXTURE_DIR = Path(__file__).parent / "prompt_guard"
+
+
[email protected](loop_scope="class")
+class TestPromptGuardDryRun:
+ async def test_dry_run_with_absent_optional_completes_and_keeps_record(self):
+ """Omitting the guarded optional (without mock seeding) leaves it undefined in the prompt
+ render context: the guarded arms render empty, the mock output is produced, and the
+ not-provided record persists in the ledger.
+ """
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)], pipe_run_mode=PipeRunMode.DRY)
+
+ response = await runner.execute(pipe_code="opg_draft_note", inputs={"topic": "quantum computing"})
+
+ assert response.state == RunState.COMPLETED
+ assert response.pipe_output.main_stuff.as_text.text
+ memory = response.pipe_output.working_memory
+ style_hint_record = memory.get_optional_absence("style_hint")
+ assert style_hint_record is not None
+ assert style_hint_record.kind == AbsenceKind.NOT_PROVIDED
+ # The output itself is a delivered mock value, not an absence.
+ assert memory.get_optional_absence(response.main_stuff_name or "") is None
+
+ async def test_dry_run_with_present_optional_renders_guarded_arm(self):
+ """With the optional provided, the value rides into the render context and the ledger
+ stays empty.
+ """
+ runner = PipelexMTHDSProtocol(library_dirs=[str(_FIXTURE_DIR)], pipe_run_mode=PipeRunMode.DRY)
+
+ response = await runner.execute(pipe_code="opg_draft_note", inputs={"topic": "quantum computing", "style_hint": "keep it formal"})
+
+ assert response.state == RunState.COMPLETED
+ memory = response.pipe_output.working_memory
+ assert memory.absences == {}
+ assert memory.get_stuff("style_hint").as_text.text == "keep it formal"
diff --git a/tests/integration/pipelex/pipeline/test_optional_method_inputs.py b/tests/integration/pipelex/pipeline/test_optional_method_inputs.py
new file mode 100644
index 000000000..37385790f
--- /dev/null
+++ b/tests/integration/pipelex/pipeline/test_optional_method_inputs.py
@@ -0,0 +1,188 @@
+"""Optional method inputs (D5): omitting a `?` input from the pipeline inputs yields a
+not-provided absence record instead of an error; the required-input error message tells the
+caller which optional inputs may legitimately be omitted; mock seeding stays all-present (D6).
+"""
+
+from typing import Callable
+
+import pytest
+from mthds.protocol.pipeline_inputs import PipelineInputs
+
+from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.execution_seams import prepare_pipe_job
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.registries.func_registry import func_registry
+
+
+def optionals_method_report(working_memory: WorkingMemory) -> TextContent:
+ topic = working_memory.get_stuff_as_str(name="topic")
+ source_stuff = working_memory.get_optional_stuff(name="source")
+ if source_stuff is None:
+ return TextContent(text=f"report about {topic} (no source)")
+ return TextContent(text=f"report about {topic} from {source_stuff.as_text.text}")
+
+
+def _make_method_pipe() -> PipeFunc:
+ return PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code="test_optionals_method",
+ pipe_code="opt_method_report",
+ blueprint=PipeFuncBlueprint(
+ description="Method with a required topic and an optional source",
+ inputs={"topic": "Text", "source": "Text?"},
+ output="Text",
+ function_name="optionals_method_report",
+ ),
+ )
+
+
[email protected](loop_scope="class")
+class TestOptionalMethodInputs:
+ @classmethod
+ def setup_class(cls):
+ func_registry.register_function(optionals_method_report)
+
+ @classmethod
+ def teardown_class(cls):
+ if func_registry.has_function("optionals_method_report"):
+ func_registry.unregister_function_by_name("optionals_method_report")
+
+ async def test_omitted_optional_input_records_not_provided(self, load_empty_library: Callable[[], str]):
+ """prepare_pipe_job seeds a not-provided absence record for each omitted `?` input."""
+ library_id = load_empty_library()
+ pipe = _make_method_pipe()
+ execution_config = get_config().pipelex.pipeline_execution_config
+
+ pipe_job = await prepare_pipe_job(
+ pipe=pipe,
+ library_id=library_id,
+ execution_config=execution_config,
+ pipe_run_mode=PipeRunMode.LIVE,
+ pipeline_run_id="test-optional-method-inputs",
+ user_id="pytest",
+ inputs=PipelineInputs({"topic": "penalties"}),
+ )
+
+ job_memory = pipe_job.working_memory
+ assert job_memory is not None
+ record = job_memory.get_optional_absence("source")
+ assert record is not None
+ assert record.kind == AbsenceKind.NOT_PROVIDED
+ assert record.producing_pipe is None
+ assert "source" in record.reason
+ # The pipe then runs on the absent arm without error.
+ pipe_output = await pipe.run_pipe(
+ job_metadata=pipe_job.job_metadata,
+ working_memory=job_memory,
+ pipe_run_params=pipe_job.pipe_run_params,
+ )
+ assert pipe_output.main_stuff.as_text.text == "report about penalties (no source)"
+
+ async def test_provided_optional_input_leaves_no_record(self, load_empty_library: Callable[[], str]):
+ library_id = load_empty_library()
+ pipe = _make_method_pipe()
+ execution_config = get_config().pipelex.pipeline_execution_config
+
+ pipe_job = await prepare_pipe_job(
+ pipe=pipe,
+ library_id=library_id,
+ execution_config=execution_config,
+ pipe_run_mode=PipeRunMode.LIVE,
+ pipeline_run_id="test-optional-method-inputs-provided",
+ user_id="pytest",
+ inputs=PipelineInputs({"topic": "penalties", "source": "clause 12"}),
+ )
+
+ job_memory = pipe_job.working_memory
+ assert job_memory is not None
+ assert job_memory.get_optional_absence("source") is None
+ assert job_memory.get_optional_stuff("source") is not None
+
+ async def test_preexisting_absence_record_keeps_its_provenance(self, load_empty_library: Callable[[], str]):
+ """A caller-provided WorkingMemory whose `?` slot already resolved as a recorded absence
+ (e.g. a previous run's output chained in) keeps that record — prepare_pipe_job must not
+ downgrade it to a fresh not-provided.
+ """
+ library_id = load_empty_library()
+ pipe = _make_method_pipe()
+ execution_config = get_config().pipelex.pipeline_execution_config
+
+ chained_memory = WorkingMemoryFactory.make_from_single_stuff(StuffFactory.make_from_str("penalties", name="topic"))
+ upstream_record = AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'clause' is absent",
+ producing_pipe="extract_source",
+ )
+ chained_memory.record_absence(upstream_record)
+
+ pipe_job = await prepare_pipe_job(
+ pipe=pipe,
+ library_id=library_id,
+ execution_config=execution_config,
+ pipe_run_mode=PipeRunMode.LIVE,
+ pipeline_run_id="test-optional-method-inputs-chained",
+ user_id="pytest",
+ inputs=chained_memory,
+ )
+
+ job_memory = pipe_job.working_memory
+ assert job_memory is not None
+ record = job_memory.get_optional_absence("source")
+ assert record is not None
+ assert record.kind == AbsenceKind.SKIPPED
+ assert record.producing_pipe == "extract_source"
+ assert record.reason == "skipped because input 'clause' is absent"
+
+ async def test_missing_required_error_hints_omittable_optionals(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Omitting a required input still fails, but the message now names the optional inputs
+ that may legitimately be omitted.
+ """
+ load_empty_library()
+ pipe = _make_method_pipe()
+ run_params = PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+ with pytest.raises(PipeRunInputsError) as exc_info:
+ await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipe_run_params=run_params,
+ )
+ message = str(exc_info.value)
+ assert "topic" in message
+ assert "optional" in message.lower()
+ assert "source" in message
+
+ async def test_mock_inputs_seed_optional_slots_all_present(self, load_empty_library: Callable[[], str]):
+ """Dry-run mock seeding stays all-present (D6): optional inputs get mocks, no records."""
+ library_id = load_empty_library()
+ pipe = _make_method_pipe()
+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(
+ generate_graph=False,
+ mock_inputs=True,
+ )
+
+ pipe_job = await prepare_pipe_job(
+ pipe=pipe,
+ library_id=library_id,
+ execution_config=execution_config,
+ pipe_run_mode=PipeRunMode.DRY,
+ pipeline_run_id="test-optional-method-inputs-mocked",
+ user_id="pytest",
+ )
+
+ job_memory = pipe_job.working_memory
+ assert job_memory is not None
+ assert job_memory.get_optional_stuff("source") is not None
+ assert job_memory.get_optional_stuff("topic") is not None
+ assert job_memory.get_optional_absence("source") is None
diff --git a/tests/integration/pipelex/pipeline/test_pipe_io_contracts.py b/tests/integration/pipelex/pipeline/test_pipe_io_contracts.py
index fd2e826cc..8a3924709 100644
--- a/tests/integration/pipelex/pipeline/test_pipe_io_contracts.py
+++ b/tests/integration/pipelex/pipeline/test_pipe_io_contracts.py
@@ -66,8 +66,62 @@ prompt = "Make items from:\\n@docs"
"""
+_OPTIONAL_MTHDS = """
+domain = "optional_contracts_test"
+description = "Bundle exercising presence markers on IO contracts"
+
+[concept.Verdict]
+description = "A verdict"
+
+[pipe.assess]
+type = "PipeLLM"
+description = "Assess with an optional hint"
+inputs = { doc = "Text", hint = "Text?" }
+output = "Verdict"
+prompt = '''
+Assess $doc.
+@?hint
+'''
+
+[pipe.check]
+type = "PipeLLM"
+description = "Check the doc"
+inputs = { doc = "Text" }
+output = "Verdict"
+prompt = "Check $doc"
+
+[pipe.gate]
+type = "PipeCondition"
+description = "Gate that may continue with no output"
+inputs = { doc = "Text" }
+output = "Verdict?"
+expression = "doc"
+default_outcome = "continue"
+
+[pipe.gate.outcomes]
+go = "check"
+"""
+
+
@pytest.mark.asyncio(loop_scope="class")
class TestBuildPipeIOContracts:
+ async def test_optional_markers_reported_on_contracts(self, load_empty_library: Callable[[], str]) -> None:
+ """A `?`-declared input/output reports optional=True on its contract; plain ones False."""
+ outer_library_id = load_empty_library()
+ try:
+ result = await validate_bundle(mthds_contents=[_OPTIONAL_MTHDS])
+ io_contracts = build_pipe_io_contracts(result.pipes)
+ finally:
+ _teardown_validation_library(outer_library_id)
+
+ assess = io_contracts["optional_contracts_test.assess"]
+ assert assess.inputs["doc"].optional is False
+ assert assess.inputs["hint"].optional is True
+ assert assess.output.optional is False
+
+ gate = io_contracts["optional_contracts_test.gate"]
+ assert gate.output.optional is True
+
async def test_multiplicity_entry_shapes(self, load_empty_library: Callable[[], str]) -> None:
"""Entries are keyed by namespaced pipe_ref; output multiplicity is single vs variable."""
outer_library_id = load_empty_library()
diff --git a/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_compaction.py b/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_compaction.py
new file mode 100644
index 000000000..654b62252
--- /dev/null
+++ b/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_compaction.py
@@ -0,0 +1,149 @@
+"""PipeBatch compaction (D4 bonus): a branch that resolves absent is dropped from the aggregated
+list — the batch output contains only found items (compactMap), instead of crashing on the raising
+main-stuff accessor.
+
+The absence source is the phase-1 `continue` semantics: the branch is a PipeCondition that routes
+matching items to a processing pipe and non-matching items to `continue` (declared-absent output).
+"""
+
+from typing import Callable, cast
+
+import pytest
+
+from pipelex.config import get_config
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.batch.pipe_batch import PipeBatch
+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_batch"
+
+
+def optionals_batch_shout_item(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=working_memory.get_stuff_as_str(name="item").upper())
+
+
+_TEST_FUNCS = [optionals_batch_shout_item]
+
+
+def _make_live_run_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+
+def _build_compacting_batch() -> PipeBatch:
+ """Batch over Text items whose branch keeps 'good *' items (shouted) and continues on the rest."""
+ shout_pipe = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_batch_shout",
+ blueprint=PipeFuncBlueprint(
+ description="Uppercase the batch item",
+ inputs={"item": "Text"},
+ output="Text",
+ function_name="optionals_batch_shout_item",
+ ),
+ )
+ gate_pipe = PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_batch_gate",
+ blueprint=PipeConditionBlueprint(
+ description="Keep items starting with 'good', continue on the rest",
+ inputs={"item": "Text"},
+ output="Text?",
+ expression_template="{% if item.text.startswith('good') %}keep{% else %}reject{% endif %}",
+ outcomes={"keep": "opt_batch_shout"},
+ default_outcome=SpecialOutcome.CONTINUE,
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [shout_pipe, gate_pipe]:
+ pipe_library.add_new_pipe(pipe=pipe)
+
+ batch = PipeFactory[PipeBatch].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_batch_compact",
+ blueprint=PipeBatchBlueprint(
+ description="Batch the gate over the items",
+ branch_pipe_code="opt_batch_gate",
+ output="Text",
+ input_list_name="items",
+ input_item_name="item",
+ inputs={"items": "Text"},
+ ),
+ )
+ pipe_library.add_new_pipe(pipe=batch)
+ return batch
+
+
[email protected](loop_scope="class")
+class TestPipeBatchCompaction:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ async def test_absent_branch_results_are_dropped(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Mixed items: rejected items resolve absent (continue) and are dropped from the list."""
+ load_empty_library()
+ batch = _build_compacting_batch()
+
+ items_stuff = StuffFactory.make_stuff(
+ concept=batch.inputs.get_required_stuff_spec("items").concept,
+ content=ListContent[TextContent](
+ items=[TextContent(text="good morning"), TextContent(text="bad day"), TextContent(text="good night")],
+ ),
+ name="items",
+ )
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(items_stuff)
+
+ pipe_output = await batch.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ list_content = pipe_output.main_stuff.content
+ assert isinstance(list_content, ListContent)
+ texts = [item.text for item in cast("ListContent[TextContent]", list_content).items]
+ assert texts == ["GOOD MORNING", "GOOD NIGHT"]
+
+ async def test_all_branches_absent_yields_empty_list(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Every item rejected: the batch still delivers a real (empty) list output."""
+ load_empty_library()
+ batch = _build_compacting_batch()
+
+ items_stuff = StuffFactory.make_stuff(
+ concept=batch.inputs.get_required_stuff_spec("items").concept,
+ content=ListContent[TextContent](items=[TextContent(text="bad day"), TextContent(text="worse day")]),
+ name="items",
+ )
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(items_stuff)
+
+ pipe_output = await batch.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ list_content = pipe_output.main_stuff.content
+ assert isinstance(list_content, ListContent)
+ assert cast("ListContent[TextContent]", list_content).items == []
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_1.mthds b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_1.mthds
index 63d1441c0..f3ba79d85 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_1.mthds
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_1.mthds
@@ -9,7 +9,7 @@ CategoryInput = "Input with a category field"
type = "PipeCondition"
description = "Route based on category field"
inputs = { input_data = "CategoryInput" }
-output = "native.Text"
+output = "native.Text?"
expression_template = "{{ input_data.category }}"
default_outcome = "continue"
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_2.mthds b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_2.mthds
index c9bb96475..351cd2788 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_2.mthds
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_2.mthds
@@ -9,7 +9,7 @@ CategoryInput = "Input with a category field"
type = "PipeCondition"
description = "Route based on category field using expression"
inputs = { input_data = "CategoryInput" }
-output = "native.Text"
+output = "native.Text?"
expression = "input_data.category"
default_outcome = "continue"
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.mthds b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.mthds
index 5a8037ec0..bfc8c7714 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.mthds
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.mthds
@@ -12,7 +12,7 @@ ProcessingContext = "Combined processing context"
type = "PipeCondition"
description = "Primary routing by document type"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
-output = "native.Text"
+output = "native.Text?"
expression_template = "{{ doc_request.document_type }}"
default_outcome = "continue"
@@ -26,7 +26,7 @@ legal = "legal_document_router"
type = "PipeCondition"
description = "Route technical documents by priority and user level"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
-output = "native.Text"
+output = "native.Text?"
expression_template = "{% if doc_request.priority == 'urgent' %}urgent_tech{% elif user_profile.user_level == 'expert' and doc_request.complexity == 'high' %}expert_tech{% else %}standard_tech{% endif %}"
default_outcome = "continue"
@@ -40,7 +40,7 @@ standard_tech = "standard_technical_processor"
type = "PipeCondition"
description = "Route business documents by department and priority"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
-output = "native.Text"
+output = "native.Text?"
expression_template = "{% if doc_request.priority == 'urgent' %}urgent_business{% elif user_profile.department == 'finance' %}finance_business{% elif user_profile.department == 'marketing' %}marketing_business{% else %}general_business{% endif %}"
default_outcome = "continue"
@@ -55,7 +55,7 @@ general_business = "general_business_processor"
type = "PipeCondition"
description = "Route legal documents by complexity and user level"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
-output = "native.Text"
+output = "native.Text?"
expression_template = "{% if doc_request.complexity == 'high' and user_profile.user_level != 'beginner' %}complex_legal{% elif doc_request.language != 'english' %}international_legal{% else %}standard_legal{% endif %}"
default_outcome = "continue"
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_continue_output_type.mthds b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_continue_output_type.mthds
index 0cfe6849c..610498333 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_continue_output_type.mthds
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_continue_output_type.mthds
@@ -19,7 +19,7 @@ branch_pipe_code = "build_or_skip"
type = "PipeCondition"
description = "Routes approved links to builder, rejected links to skip (continue)."
inputs = { verified_link = "VerifiedLink" }
-output = "Constraint"
+output = "Constraint?"
expression_template = "{{ verified_link.verdict }}"
default_outcome = "continue"
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py
index 77cf750e3..e6fbca5b2 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py
@@ -1,10 +1,9 @@
-"""PipeCondition's `continue` outcome delivers the current main stuff by passing it through.
+"""PipeCondition's `continue` outcome resolves the declared output as ABSENT (design §14, breaking).
-A pipe run always delivers a main stuff. For `continue`, the delivered main stuff is the one
-already in the working memory (pass-through). When there is nothing to pass through (e.g. the
-condition is the entry pipe and memory holds only named inputs), the run must fail loudly at the
-pipe level with an actionable error — not crash downstream (graph tracer, delivery, telemetry)
-with a bare WorkingMemoryStuffNotFoundError.
+Phase-1 semantics: `continue` records a declared-absent AbsenceRecord for the declared output and
+returns success — memory otherwise unchanged (a previous main stuff stays under its own name but no
+longer masquerades as this pipe's output). The old pass-through-or-error behavior is gone; the
+ergonomic replacement for pass-through is coalescing (phase 2).
"""
from pathlib import Path
@@ -16,6 +15,7 @@ from pipelex.config import get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.core.stuffs.stuff_factory import StuffFactory
@@ -30,11 +30,11 @@ from pipelex.pipeline.job_metadata import JobMetadata
def _make_continue_only_condition() -> PipeCondition:
- """A condition whose every outcome is `continue` (a pure pass-through gate)."""
+ """A condition whose every outcome is `continue` (a pure absence gate)."""
blueprint = PipeConditionBlueprint(
- description="Pass-through gate for continue-delivery tests",
+ description="Absence gate for continue-delivery tests",
inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},
- output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",
+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}?",
expression_template="{% if input_text.text %}skip{% else %}skip{% endif %}",
outcomes={"skip": SpecialOutcome.CONTINUE},
default_outcome=SpecialOutcome.CONTINUE,
@@ -62,66 +62,124 @@ def _make_run_params(run_mode: PipeRunMode) -> PipeRunParams:
@pytest.mark.asyncio(loop_scope="class")
class TestPipeConditionContinueDelivery:
- async def test_continue_with_main_stuff_passes_it_through(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
- """With a main stuff in memory, `continue` delivers it unchanged."""
+ async def test_continue_records_declared_absent_output(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
+ """`continue` succeeds and resolves the declared output as a declared-absent record."""
load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
pipe_condition = _make_continue_only_condition()
- input_text_stuff = _make_input_text_stuff()
- working_memory = WorkingMemoryFactory.make_from_single_stuff(input_text_stuff)
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.set_stuff(name="input_text", stuff=_make_input_text_stuff())
pipe_output = await pipe_condition.run_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
pipe_run_params=_make_run_params(PipeRunMode.LIVE),
+ output_name="gate_result",
)
- assert pipe_output.main_stuff.stuff_code == input_text_stuff.stuff_code
- # Run identity must survive the pass-through: the continue path builds its own PipeOutput,
+ result_memory = pipe_output.working_memory
+ resolved_main = result_memory.resolve_main_stuff()
+ assert isinstance(resolved_main, AbsenceRecord)
+ assert resolved_main.kind == AbsenceKind.DECLARED_ABSENT
+ assert resolved_main.producing_pipe == "continue_gate"
+ # The record names the declared output slot and carries the evaluated expression.
+ assert resolved_main.variable_name == "gate_result"
+ assert "skip" in resolved_main.reason
+ assert result_memory.get_optional_absence("gate_result") == resolved_main
+ # Memory otherwise unchanged: the input is still there, untainted.
+ assert result_memory.get_optional_stuff("input_text") is not None
+ # Run identity must survive the absence path: the continue arm builds its own PipeOutput,
# so it must stamp pipeline_run_id like every other controller — otherwise the blocking
# bridge serializes SpecialPipelineId.UNTITLED and callers track the wrong run.
assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
- async def test_continue_without_main_stuff_fails_loud(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
- """With nothing to pass through, `continue` is a clear PipeRunError at the pipe level."""
+ async def test_continue_supersedes_previous_main_stuff(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
+ """A previous main stuff no longer passes through: it stays under its own name while the
+ condition's output resolves absent (the migration idiom — consume it explicitly downstream).
+ """
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
+ pipe_condition = _make_continue_only_condition()
+ input_text_stuff = _make_input_text_stuff()
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(input_text_stuff)
+ assert working_memory.get_optional_main_stuff() is not None
+
+ pipe_output = await pipe_condition.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_run_params(PipeRunMode.LIVE),
+ )
+
+ result_memory = pipe_output.working_memory
+ resolved_main = result_memory.resolve_main_stuff()
+ assert isinstance(resolved_main, AbsenceRecord)
+ assert resolved_main.kind == AbsenceKind.DECLARED_ABSENT
+ # The previous value remains addressable under its own name.
+ previous = result_memory.get_optional_stuff("input_text")
+ assert previous is not None
+ assert previous.stuff_code == input_text_stuff.stuff_code
+ assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+
+ async def test_dry_run_all_special_outcomes_records_absence(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
+ """Dry-run parity: an all-special-outcomes condition records the declared output absent
+ instead of raising — same semantics as the live continue arm, run id stamped alike.
+ """
load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
pipe_condition = _make_continue_only_condition()
working_memory = WorkingMemoryFactory.make_empty()
working_memory.set_stuff(name="input_text", stuff=_make_input_text_stuff())
- with pytest.raises(PipeRunError) as exc_info:
- await pipe_condition.run_pipe(
- job_metadata=job_metadata,
- working_memory=working_memory,
- pipe_run_params=_make_run_params(PipeRunMode.LIVE),
- )
- assert "continue_gate" in str(exc_info.value)
- assert "main stuff" in str(exc_info.value)
+ pipe_output = await pipe_condition.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_run_params(PipeRunMode.DRY),
+ )
- async def test_dry_run_of_special_outcomes_only_condition_without_main_stuff_fails_loud(
- self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]
- ):
- """Dry-run parity: an all-special-outcomes condition with no pre-existing main stuff cannot deliver."""
+ resolved_main = pipe_output.working_memory.resolve_main_stuff()
+ assert isinstance(resolved_main, AbsenceRecord)
+ assert resolved_main.kind == AbsenceKind.DECLARED_ABSENT
+ assert resolved_main.producing_pipe == "continue_gate"
+ assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+
+ async def test_dry_run_fail_only_condition_raises(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
+ """A fail-only condition has no absent arm — every live path raises — so dry-run must
+ raise too instead of fabricating an absence that would bless an always-failing method.
+ """
load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
- pipe_condition = _make_continue_only_condition()
+ blueprint = PipeConditionBlueprint(
+ description="Fail-only condition (assert-style misauthoring)",
+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},
+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",
+ expression_template="{{ input_text.text }}",
+ outcomes={"bad": SpecialOutcome.FAIL},
+ default_outcome=SpecialOutcome.FAIL,
+ )
+ pipe_condition = PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code="test_integration",
+ pipe_code="fail_only_gate",
+ blueprint=blueprint,
+ )
working_memory = WorkingMemoryFactory.make_empty()
working_memory.set_stuff(name="input_text", stuff=_make_input_text_stuff())
- with pytest.raises(PipeRunError):
+ with pytest.raises(PipeRunError) as exc_info:
await pipe_condition.run_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
pipe_run_params=_make_run_params(PipeRunMode.DRY),
)
+ message = str(exc_info.value)
+ assert "fail_only_gate" in message
+ assert "fail" in message
- async def test_dry_run_continue_with_main_stuff_preserves_run_id(
+ async def test_dry_run_all_special_outcomes_supersedes_previous_main_stuff(
self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]
):
- """Dry-run parity path (all-special-outcomes condition with a pre-existing main stuff) must also
- stamp pipeline_run_id, like the live continue path — the dry-run PipeOutput is built directly too.
+ """Dry-run parity with the live supersede rule: a pre-existing main stuff stays under its
+ own name while the condition's output resolves absent.
"""
load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
pipe_condition = _make_continue_only_condition()
- working_memory = WorkingMemoryFactory.make_from_single_stuff(_make_input_text_stuff())
+ input_text_stuff = _make_input_text_stuff()
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(input_text_stuff)
pipe_output = await pipe_condition.run_pipe(
job_metadata=job_metadata,
@@ -129,4 +187,7 @@ class TestPipeConditionContinueDelivery:
pipe_run_params=_make_run_params(PipeRunMode.DRY),
)
+ result_memory = pipe_output.working_memory
+ assert isinstance(result_memory.resolve_main_stuff(), AbsenceRecord)
+ assert result_memory.get_optional_stuff("input_text") is not None
assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_output_type.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_output_type.py
index eb0b1e835..46a426d84 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_output_type.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_output_type.py
@@ -2,7 +2,11 @@
Tests that when batching over VerifiedLinks, the PipeCondition correctly:
- Creates Constraints for approved links
-- Skips (continues) for rejected links
+- Resolves absent (continue) for rejected links, which the batch compacts away (D4)
+
+Live: the rejected link's branch resolves absent and is dropped from the aggregated list.
+Dry: the dry sweep runs every mapped outcome (the builder pipe), so every item yields a mock
+constraint — the continue arm is modeled statically (Step D), not by the dry run.
"""
from collections.abc import Callable
@@ -28,7 +32,6 @@ from tests.integration.pipelex.pipes.controller.pipe_condition.pipe_condition_co
@pytest.mark.llm
@pytest.mark.inference
[email protected](reason="This test identifies a bug. it should be fixed")
@pytest.mark.dry_runnable
@pytest.mark.asyncio(loop_scope="class")
class TestPipeConditionContinueOutputType:
@@ -43,10 +46,8 @@ class TestPipeConditionContinueOutputType:
"""Test batching over VerifiedLinks with mixed verdicts.
Given 2 VerifiedLinks:
- - One approved (should create a Constraint)
- - One rejected (should be skipped via 'continue')
-
- The output should be a list with only 1 Constraint (from the approved link).
+ - One approved (creates a Constraint)
+ - One rejected (resolves absent via 'continue' and is compacted away — live only)
"""
load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
@@ -100,7 +101,11 @@ class TestPipeConditionContinueOutputType:
pretty_print(constraint_list, title="Constraint list")
- # Should have only 1 Constraint (from the approved link)
- # The rejected link should have been skipped via 'continue'
- if pipe_run_mode.is_dry:
+ if pipe_run_mode.is_live:
+ # Compaction: only the approved link yields a Constraint; the rejected link's branch
+ # resolved absent and was dropped from the aggregated list.
assert len(constraint_list.items) == 1, f"Expected 1 constraint, got {len(constraint_list.items)}"
+ else:
+ # The dry sweep runs every mapped outcome for every item, so both items yield a mock
+ # constraint — the continue arm is not selected by expression evaluation in dry mode.
+ assert len(constraint_list.items) == 2, f"Expected 2 mock constraints, got {len(constraint_list.items)}"
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py
index a72d1cbd4..1ee19d743 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py
@@ -36,7 +36,7 @@ class TestPipeConditionSimple:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Text length condition for testing",
inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},
- output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",
+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}?",
expression_template="{% if input_text.text|length > 5 %}long{% else %}short{% endif %}",
outcomes={"long": "capitalize_long_text", "short": "add_prefix_short_text"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -112,7 +112,7 @@ class TestPipeConditionSimple:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Text length condition for short text testing",
inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},
- output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",
+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}?",
expression_template="{% if input_text.text|length > 5 %}long{% else %}short{% endif %}",
outcomes={"long": "capitalize_long_text", "short": "add_prefix_short_text"},
default_outcome=SpecialOutcome.CONTINUE,
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py
index be1c28fc8..5d4d08cc5 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py
@@ -73,7 +73,7 @@ class TestPipeConditionValidation:
pipe_condition_template_blueprint = PipeConditionBlueprint(
description="Test condition with expression template",
inputs={"var": concept_1.concept_ref},
- output=concept_2.concept_ref,
+ output=f"{concept_2.concept_ref}?",
expression_template="{{ var }}",
outcomes={"value": "target_pipe"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -89,7 +89,7 @@ class TestPipeConditionValidation:
pipe_condition_expr_blueprint = PipeConditionBlueprint(
description="Test condition with expression",
inputs={"var": concept_1.concept_ref},
- output=concept_2.concept_ref,
+ output=f"{concept_2.concept_ref}?",
expression="var",
outcomes={"value": "target_pipe"},
default_outcome=SpecialOutcome.CONTINUE,
diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py
new file mode 100644
index 000000000..bb60342a3
--- /dev/null
+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py
@@ -0,0 +1,276 @@
+"""PipeParallel combine under absence (D11): an absent branch result is absorbed, omitted, or a
+typed runtime error — never a crash on the raising main-stuff accessor.
+
+- `Composite` output: the absent component is omitted from the composite, with a ledger note.
+- Structured output, non-required field: the absence converts to field-level `None` at the combine
+ boundary (taint terminates there).
+- Structured output, required field: a typed PipeRunError naming the branch and the field (this is
+ statically unreachable once Step D's taint pass lands, but the runtime must not feed
+ `combine_stuffs` a hole).
+
+Branches are PipeFuncs; the lifted branch consumes a plain input fed a recorded absence (D3).
+"""
+
+from typing import Callable
+
+import pytest
+from pydantic import Field
+
+from pipelex.config import get_config
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.composite_content import CompositeContent
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry, get_concept_library, get_pipe_library
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipe_run.exceptions import PipeRunError
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_par"
+
+
+class ParAbsReport(StructuredContent):
+ """Combination target with one required and one absorbable (non-required) field."""
+
+ base_result: TextContent = Field(description="Result of the always-running branch")
+ found_result: TextContent | None = Field(default=None, description="Result of the maybe-lifted branch")
+
+
+class ParAbsStrictReport(StructuredContent):
+ """Combination target whose fields are all required — no absorption possible."""
+
+ base_result: TextContent = Field(description="Result of the always-running branch")
+ found_result: TextContent = Field(description="Result of the maybe-lifted branch")
+
+
+def optionals_par_echo_source(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"found:{working_memory.get_stuff_as_str(name='source')}")
+
+
+def optionals_par_echo_topic(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"base:{working_memory.get_stuff_as_str(name='topic')}")
+
+
+_TEST_FUNCS = [optionals_par_echo_source, optionals_par_echo_topic]
+
+
+def _make_live_run_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+
+def _build_parallel(*, output_ref: str, structure_class_names: list[str], add_each_output: bool = False) -> PipeParallel:
+ """Register the two branch PipeFuncs and build the parallel over them.
+
+ Branch 'found_result' consumes `source` plain — lifted when source is absent-with-record.
+ Branch 'base_result' consumes `topic` — always runs. The parallel declares `source` optional
+ at its own boundary so it runs (its branch lifts) instead of lifting wholesale.
+ """
+ branch_found = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_par_find",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes the maybe-absent source (plain input): lifted when source is absent",
+ inputs={"source": "Text"},
+ output="Text",
+ function_name="optionals_par_echo_source",
+ ),
+ )
+ branch_base = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_par_base",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes the always-present topic",
+ inputs={"topic": "Text"},
+ output="Text",
+ function_name="optionals_par_echo_topic",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [branch_found, branch_base]:
+ pipe_library.add_new_pipe(pipe=pipe)
+
+ for structure_class_name in structure_class_names:
+ concept = ConceptFactory.make(
+ concept_code=structure_class_name,
+ domain_code=_DOMAIN_CODE,
+ description=f"Test combination concept {structure_class_name}",
+ structure_class_name=structure_class_name,
+ )
+ get_concept_library().add_new_concept(concept=concept)
+
+ parallel = PipeFactory[PipeParallel].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_par_combine",
+ blueprint=PipeParallelBlueprint(
+ description="Parallel over a maybe-lifted branch and an always-running branch",
+ inputs={"source": "Text?", "topic": "Text"},
+ output=output_ref,
+ branches=[
+ SubPipeBlueprint(pipe="opt_par_find", result="found_result"),
+ SubPipeBlueprint(pipe="opt_par_base", result="base_result"),
+ ],
+ add_each_output=add_each_output,
+ ),
+ concept_codes_from_the_same_domain=structure_class_names,
+ )
+ pipe_library.add_new_pipe(pipe=parallel)
+ return parallel
+
+
+def _make_absent_source_memory() -> WorkingMemory:
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(StuffFactory.make_from_str("penalties", name="topic"))
+ working_memory.record_absence(
+ AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.DECLARED_ABSENT,
+ reason="no penalty clause found in this contract",
+ producing_pipe="extract_penalty_clause",
+ ),
+ )
+ return working_memory
+
+
[email protected](loop_scope="class")
+class TestPipeParallelAbsence:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+ get_class_registry().register_class(ParAbsReport)
+ get_class_registry().register_class(ParAbsStrictReport)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ async def test_composite_output_omits_absent_component_with_ledger_note(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Composite output: the absent component is omitted; a ledger note keeps observability."""
+ load_empty_library()
+ parallel = _build_parallel(output_ref="Composite", structure_class_names=[], add_each_output=True)
+ working_memory = _make_absent_source_memory()
+
+ pipe_output = await parallel.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ main_stuff = pipe_output.main_stuff
+ assert isinstance(main_stuff.content, CompositeContent)
+ assert set(main_stuff.content.components.keys()) == {"base_result"}
+
+ result_memory = pipe_output.working_memory
+ # add_each_output: the present branch is exposed by name; the absent one is a ledger note.
+ base_stuff = result_memory.get_optional_stuff("base_result")
+ assert base_stuff is not None
+ assert base_stuff.as_text.text == "base:penalties"
+ assert result_memory.get_optional_stuff("found_result") is None
+ note = result_memory.get_optional_absence("found_result")
+ assert note is not None
+ assert note.kind == AbsenceKind.SKIPPED
+ assert note.producing_pipe == "opt_par_find"
+ assert note.origin().producing_pipe == "extract_penalty_clause"
+ assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+
+ async def test_structured_output_optional_field_absorbs_as_none(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Structured output: a non-required field absorbs the absent branch as field-level None."""
+ load_empty_library()
+ parallel = _build_parallel(output_ref=f"{_DOMAIN_CODE}.ParAbsReport", structure_class_names=["ParAbsReport", "ParAbsStrictReport"])
+ working_memory = _make_absent_source_memory()
+
+ pipe_output = await parallel.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ main_stuff = pipe_output.main_stuff
+ combined = main_stuff.content
+ assert isinstance(combined, ParAbsReport)
+ assert combined.found_result is None
+ assert combined.base_result.text == "base:penalties"
+ # The taint terminates at the combine boundary: the combined output is a real value.
+ assert isinstance(pipe_output.working_memory.resolve_main_stuff(), Stuff)
+
+ async def test_structured_output_required_field_fed_absent_branch_raises(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Structured output: a required field fed an absent branch is a typed runtime error that
+ names the branch and the field — never a hole handed to combine_stuffs.
+ """
+ load_empty_library()
+ parallel = _build_parallel(output_ref=f"{_DOMAIN_CODE}.ParAbsStrictReport", structure_class_names=["ParAbsReport", "ParAbsStrictReport"])
+ working_memory = _make_absent_source_memory()
+
+ with pytest.raises(PipeRunError) as exc_info:
+ await parallel.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+ message = str(exc_info.value)
+ assert "opt_par_find" in message
+ assert "found_result" in message
+
+ async def test_absent_branch_without_add_each_output_leaves_parent_slot_untouched(
+ self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]
+ ):
+ """Without add_each_output, branch result names are not parent slots: an absent branch must
+ not clobber an unrelated same-named value in the parent memory (present branches don't).
+ """
+ load_empty_library()
+ parallel = _build_parallel(output_ref="Composite", structure_class_names=[], add_each_output=False)
+ working_memory = _make_absent_source_memory()
+ working_memory.add_new_stuff(name="found_result", stuff=StuffFactory.make_from_str("pre-existing unrelated value", name="found_result"))
+
+ pipe_output = await parallel.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ main_stuff = pipe_output.main_stuff
+ assert isinstance(main_stuff.content, CompositeContent)
+ assert set(main_stuff.content.components.keys()) == {"base_result"}
+
+ result_memory = pipe_output.working_memory
+ surviving = result_memory.get_optional_stuff("found_result")
+ assert surviving is not None
+ assert surviving.as_text.text == "pre-existing unrelated value"
+ assert result_memory.get_optional_absence("found_result") is None
+
+ async def test_all_branches_present_combines_as_before(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """With every branch present, the combine is byte-for-byte the pre-optionals behavior."""
+ load_empty_library()
+ parallel = _build_parallel(output_ref=f"{_DOMAIN_CODE}.ParAbsReport", structure_class_names=["ParAbsReport", "ParAbsStrictReport"])
+ working_memory = WorkingMemoryFactory.make_from_multiple_stuffs(
+ [
+ StuffFactory.make_from_str("clause 12", name="source"),
+ StuffFactory.make_from_str("penalties", name="topic"),
+ ],
+ )
+
+ pipe_output = await parallel.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ combined = pipe_output.main_stuff.content
+ assert isinstance(combined, ParAbsReport)
+ assert combined.found_result is not None
+ assert combined.found_result.text == "found:clause 12"
+ assert combined.base_result.text == "base:penalties"
diff --git a/tests/integration/pipelex/pipes/optionals/test_absent_main_output_surfaces.py b/tests/integration/pipelex/pipes/optionals/test_absent_main_output_surfaces.py
new file mode 100644
index 000000000..f4ecb05b7
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_absent_main_output_surfaces.py
@@ -0,0 +1,125 @@
+"""End-to-end absent main output (Step E): a run whose declared `?` output resolves absent is a
+first-class success on every in-repo surface — the execute response names the declared slot, the
+working memory carries the provenance ledger, and the execution graph shows the lifted pipes as
+`skipped` with their skip reasons.
+"""
+
+import json
+
+import pytest
+
+from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.graph.graphspec import NodeStatus
+from pipelex.pipeline.pipeline_response import RunState
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.registries.func_registry import func_registry
+
+_BUNDLE = """
+domain = "opt_surfaces"
+description = "Optional flow whose whole chain lifts when the optional input is omitted"
+main_pipe = "opt_flow"
+
+[pipe.opt_surfaces_make_analysis]
+type = "PipeFunc"
+description = "Analyzes the source (plain input: lifts when source is absent)"
+inputs = { source = "Text" }
+output = "Text"
+function_name = "opt_surfaces_make_analysis"
+
+[pipe.opt_surfaces_summarize]
+type = "PipeFunc"
+description = "Summarizes the analysis (plain input: lifts in chain)"
+inputs = { analysis = "Text" }
+output = "Text"
+function_name = "opt_surfaces_summarize"
+
+[pipe.opt_flow]
+type = "PipeSequence"
+description = "Whole chain lifts when source is absent; output declared optional"
+inputs = { source = "Text?" }
+output = "Text?"
+steps = [
+ { pipe = "opt_surfaces_make_analysis", result = "analysis" },
+ { pipe = "opt_surfaces_summarize", result = "summary" },
+]
+"""
+
+
+def opt_surfaces_make_analysis(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"analysis of {working_memory.get_stuff_as_str(name='source')}")
+
+
+def opt_surfaces_summarize(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"summary: {working_memory.get_stuff_as_str(name='analysis')}")
+
+
+_TEST_FUNCS = [opt_surfaces_make_analysis, opt_surfaces_summarize]
+
+
[email protected](loop_scope="class")
+class TestAbsentMainOutputSurfaces:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ async def test_execute_with_absent_main_output_succeeds_across_surfaces(self):
+ """Omitting the optional input lifts the whole chain: the execute response is a success
+ naming the declared slot, the ledger chains provenance, and the graph marks skips.
+ """
+ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True)
+ runner = PipelexMTHDSProtocol(execution_config=execution_config)
+
+ response = await runner.execute(mthds_contents=[_BUNDLE], inputs={})
+
+ # Execute response: success with an absent result — main_stuff_name names the declared slot.
+ assert response.state == RunState.COMPLETED
+ assert response.main_stuff_name == "summary"
+
+ memory = response.pipe_output.working_memory
+ main_resolved = memory.resolve_main_stuff()
+ assert isinstance(main_resolved, AbsenceRecord)
+ assert main_resolved.kind == AbsenceKind.SKIPPED
+ assert main_resolved.producing_pipe == "opt_surfaces_summarize"
+ # Provenance chains back to the omitted method input.
+ origin = main_resolved.origin()
+ assert origin.variable_name == "source"
+ assert origin.kind == AbsenceKind.NOT_PROVIDED
+
+ # Graph: the lifted pipes are skipped with reasons; the controller succeeded.
+ graph_spec = response.pipe_output.graph_spec
+ assert graph_spec is not None
+ nodes_by_code = {node.pipe_code: node for node in graph_spec.nodes}
+ analysis_node = nodes_by_code["opt_surfaces_make_analysis"]
+ assert analysis_node.status == NodeStatus.SKIPPED
+ assert analysis_node.skip_reason is not None
+ assert "source" in analysis_node.skip_reason
+ summary_node = nodes_by_code["opt_surfaces_summarize"]
+ assert summary_node.status == NodeStatus.SKIPPED
+ assert summary_node.skip_reason is not None
+ assert "analysis" in summary_node.skip_reason
+ assert nodes_by_code["opt_flow"].status == NodeStatus.SUCCEEDED
+
+ # The skipped state serializes on the wire.
+ graph_dump = json.loads(graph_spec.to_json())
+ statuses = {node["pipe_code"]: node["status"] for node in graph_dump["nodes"]}
+ assert statuses["opt_surfaces_make_analysis"] == "skipped"
+
+ async def test_execute_with_source_runs_the_full_chain(self):
+ """With the optional input provided, nothing lifts and the chain delivers a value."""
+ runner = PipelexMTHDSProtocol()
+
+ response = await runner.execute(mthds_contents=[_BUNDLE], inputs={"source": "clause 12"})
+
+ assert response.state == RunState.COMPLETED
+ assert response.pipe_output.main_stuff.as_text.text == "summary: analysis of clause 12"
+ assert response.pipe_output.working_memory.absences == {}
diff --git a/tests/integration/pipelex/pipes/optionals/test_condition_optional_output_validation.py b/tests/integration/pipelex/pipes/optionals/test_condition_optional_output_validation.py
new file mode 100644
index 000000000..a48fa2779
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_condition_optional_output_validation.py
@@ -0,0 +1,131 @@
+"""PipeCondition static optionality rules (D5/D6):
+
+- `OPTIONAL_OUTPUT_REQUIRED`: a `continue`-reachable condition (any outcome mapped to `continue`,
+ or `default_outcome = "continue"`) MUST declare its output optional — `continue` resolves the
+ declared output absent (Step C), so a plain output would hide the no-output path.
+- Condition boundary taint: a mapped pipe with a `?` output makes the condition's own output
+ maybe-absent, so the condition must declare `?` too (`OPTIONAL_NOT_HANDLED`).
+"""
+
+from typing import Callable
+
+import pytest
+from pydantic import ValidationError
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.handle_pipe_errors import extract_wrapped_pipe_validation_error
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_cond"
+
+
+def optionals_cond_echo_topic(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"echo:{working_memory.get_stuff_as_str(name='topic')}")
+
+
+_TEST_FUNCS = [optionals_cond_echo_topic]
+
+
+def _make_condition(*, output_ref: str, outcomes: dict[str, str], default_outcome: str) -> PipeCondition:
+ return PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_condition",
+ blueprint=PipeConditionBlueprint(
+ description="Condition under optionality rules",
+ inputs={"topic": "Text"},
+ output=output_ref,
+ expression="topic",
+ outcomes=outcomes,
+ default_outcome=default_outcome,
+ ),
+ )
+
+
+def _register_target_pipe(*, pipe_code: str, output_ref: str) -> None:
+ target = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code=pipe_code,
+ blueprint=PipeFuncBlueprint(
+ description="Mapped target pipe",
+ inputs={"topic": "Text"},
+ output=output_ref,
+ function_name="optionals_cond_echo_topic",
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=target)
+
+
+class TestConditionOptionalOutputValidation:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ # ---- OPTIONAL_OUTPUT_REQUIRED (static, at pipe construction) ----
+
+ def test_continue_outcome_with_plain_output_is_rejected(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ _make_condition(output_ref="Text", outcomes={"skip": "continue", "go": "target_pipe"}, default_outcome="fail")
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_OUTPUT_REQUIRED
+
+ def test_continue_default_outcome_with_plain_output_is_rejected(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ _make_condition(output_ref="Text", outcomes={"go": "target_pipe"}, default_outcome="continue")
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_OUTPUT_REQUIRED
+
+ def test_continue_with_optional_output_is_accepted(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ condition = _make_condition(output_ref="Text?", outcomes={"skip": "continue", "go": "target_pipe"}, default_outcome="fail")
+ assert condition.output.presence.is_optional
+
+ def test_no_continue_with_plain_output_is_accepted(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ condition = _make_condition(output_ref="Text", outcomes={"go": "target_pipe"}, default_outcome="fail")
+ assert condition.output.presence.is_plain
+
+ # ---- Condition boundary taint (with library) ----
+
+ def test_mapped_pipe_with_optional_output_requires_optional_condition_output(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_target_pipe(pipe_code="maybe_target", output_ref="Text?")
+ condition = _make_condition(output_ref="Text", outcomes={"go": "maybe_target"}, default_outcome="fail")
+ get_pipe_library().add_new_pipe(pipe=condition)
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ condition.validate_with_libraries()
+ assert exc_info.value.error_type == PipeValidationErrorType.OPTIONAL_NOT_HANDLED
+ assert "maybe_target" in str(exc_info.value)
+
+ def test_mapped_optional_output_accepted_when_condition_output_is_optional(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_target_pipe(pipe_code="maybe_target", output_ref="Text?")
+ condition = _make_condition(output_ref="Text?", outcomes={"go": "maybe_target"}, default_outcome="fail")
+ get_pipe_library().add_new_pipe(pipe=condition)
+ condition.validate_with_libraries()
+
+ def test_mapped_plain_outputs_keep_todays_rules(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_target_pipe(pipe_code="plain_target", output_ref="Text")
+ condition = _make_condition(output_ref="Text", outcomes={"go": "plain_target"}, default_outcome="fail")
+ get_pipe_library().add_new_pipe(pipe=condition)
+ condition.validate_with_libraries()
diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py b/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py
new file mode 100644
index 000000000..8f98223ca
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_lift_sequence.py
@@ -0,0 +1,271 @@
+"""A lift-skip chain inside a PipeSequence: SubPipe's miss-gate tolerates recorded absences so
+each step's own gate decides (skip / run / force). The absorbing `?` step sinks the taint before
+the method boundary, so the sequence delivers a real main stuff (Step B scope).
+"""
+
+from typing import Callable, cast
+
+import pytest
+from mthds.protocol.pipeline_inputs import PipelineInputs
+
+from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.execution_seams import prepare_pipe_job
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_seq"
+
+
+def optionals_seq_echo_source(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"a:{working_memory.get_stuff_as_str(name='source')}")
+
+
+def optionals_seq_echo_a_out(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"b:{working_memory.get_stuff_as_str(name='a_out')}")
+
+
+def optionals_seq_sink(working_memory: WorkingMemory) -> TextContent:
+ topic = working_memory.get_stuff_as_str(name="topic")
+ b_out_stuff = working_memory.get_optional_stuff(name="b_out")
+ if b_out_stuff is None:
+ return TextContent(text=f"report about {topic} (no analysis available)")
+ return TextContent(text=f"report about {topic}: {b_out_stuff.as_text.text}")
+
+
+_TEST_FUNCS = [optionals_seq_echo_source, optionals_seq_echo_a_out, optionals_seq_sink]
+
+
+def _make_live_run_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+
+def _build_lift_sequence() -> PipeSequence:
+ """Register the A -> B -> C(sink) step pipes in the library and build the sequence over them.
+
+ The sequence declares `source = "Text?"` at its boundary while step A needs it plain — the
+ declared-vs-needed comparison must not treat the presence marker as a spec mismatch.
+ """
+ step_a = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_seq_step_a",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes the maybe-absent source (plain input): lifted when source is absent",
+ inputs={"source": "Text"},
+ output="Text",
+ function_name="optionals_seq_echo_source",
+ ),
+ )
+ step_b = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_seq_step_b",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes step A's output (plain input): lifted in chain",
+ inputs={"a_out": "Text"},
+ output="Text",
+ function_name="optionals_seq_echo_a_out",
+ ),
+ )
+ sink_c = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_seq_sink_c",
+ blueprint=PipeFuncBlueprint(
+ description="Absorbing sink: declares b_out optional and handles both arms",
+ inputs={"topic": "Text", "b_out": "Text?"},
+ output="Text",
+ function_name="optionals_seq_sink",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [step_a, step_b, sink_c]:
+ pipe_library.add_new_pipe(pipe=pipe)
+
+ sequence = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_lift_sequence",
+ blueprint=PipeSequenceBlueprint(
+ description="Lift chain: A and B skip when source is absent; C absorbs",
+ inputs={"source": "Text?", "topic": "Text"},
+ output="Text",
+ steps=[
+ SubPipeBlueprint(pipe="opt_seq_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="opt_seq_step_b", result="b_out"),
+ SubPipeBlueprint(pipe="opt_seq_sink_c", result="final_report"),
+ ],
+ ),
+ )
+ pipe_library.add_new_pipe(pipe=sequence)
+ # The declared-vs-needed input check must accept the `?` boundary over a plain child need.
+ sequence.validate_with_libraries()
+ return sequence
+
+
[email protected](loop_scope="class")
+class TestLiftSequence:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ async def test_sequence_lifts_steps_and_sinks_absence(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """With source absent (recorded), steps A and B skip with chained provenance and the
+ absorbing sink C still delivers the sequence's main output.
+ """
+ load_empty_library()
+ sequence = _build_lift_sequence()
+
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ StuffFactory.make_from_str("penalties", name="topic"),
+ )
+ source_record = AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason="optional input 'source' was not provided by the caller",
+ )
+ working_memory.record_absence(source_record)
+
+ pipe_output = await sequence.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ # The sink ran on the absent arm and delivered the sequence's output.
+ assert pipe_output.main_stuff.as_text.text == "report about penalties (no analysis available)"
+
+ result_memory = pipe_output.working_memory
+ a_record = result_memory.get_optional_absence("a_out")
+ assert a_record is not None
+ assert a_record.kind == AbsenceKind.SKIPPED
+ assert a_record.producing_pipe == "opt_seq_step_a"
+ assert a_record.upstream == source_record
+
+ b_record = result_memory.get_optional_absence("b_out")
+ assert b_record is not None
+ assert b_record.kind == AbsenceKind.SKIPPED
+ assert b_record.producing_pipe == "opt_seq_step_b"
+ assert b_record.upstream == a_record
+
+ # Run-report enumeration: the ledger lists exactly the genuinely absent slots — the
+ # positional main-stuff record was superseded when the sink delivered a real output.
+ assert set(result_memory.absences.keys()) == {"source", "a_out", "b_out"}
+
+ async def test_lifted_step_honors_invocation_multiplicity_override(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """A singular-output pipe invoked with a plural override (`multiple_output = true` on the
+ sub-pipe) normalizes its lifted output to the empty list — matching what the static taint
+ pass promised downstream list consumers — instead of recording a singular absence.
+ """
+ load_empty_library()
+ step_a = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="opt_seq_step_a",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes the maybe-absent source (plain input): lifted when source is absent",
+ inputs={"source": "Text"},
+ output="Text",
+ function_name="optionals_seq_echo_source",
+ ),
+ )
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ StuffFactory.make_from_str("penalties", name="topic"),
+ )
+ working_memory.record_absence(
+ AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason="optional input 'source' was not provided by the caller",
+ ),
+ )
+ run_params = _make_live_run_params()
+ # The invocation-level override a SubPipe applies (`multiple_output = true`).
+ run_params.output_multiplicity = True
+
+ pipe_output = await step_a.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=run_params,
+ output_name="a_out",
+ )
+
+ result_memory = pipe_output.working_memory
+ lifted_stuff = result_memory.get_optional_stuff("a_out")
+ assert lifted_stuff is not None
+ lifted_content = lifted_stuff.content
+ assert isinstance(lifted_content, ListContent)
+ assert cast("ListContent[TextContent]", lifted_content).items == []
+ # The ledger keeps the observability note beside the empty-list value.
+ note = result_memory.get_optional_absence("a_out")
+ assert note is not None
+ assert note.kind == AbsenceKind.SKIPPED
+
+ async def test_sequence_runs_fully_when_source_provided(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """With source present, nothing is lifted and the sink gets the analysis arm."""
+ load_empty_library()
+ sequence = _build_lift_sequence()
+
+ working_memory = WorkingMemoryFactory.make_from_multiple_stuffs(
+ [
+ StuffFactory.make_from_str("clause 12", name="source"),
+ StuffFactory.make_from_str("penalties", name="topic"),
+ ],
+ )
+
+ pipe_output = await sequence.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ assert pipe_output.main_stuff.as_text.text == "report about penalties: b:a:clause 12"
+ assert pipe_output.working_memory.absences == {}
+
+ async def test_prepare_pipe_job_seeds_boundary_optional_from_declared_marker(self, load_empty_library: Callable[[], str]):
+ """The not-provided record is seeded from the sequence's DECLARED `source = "Text?"`
+ boundary, even though the aggregated needed-inputs carry step A's plain marker —
+ and the whole method then runs end-to-end on the absent arm.
+ """
+ library_id = load_empty_library()
+ sequence = _build_lift_sequence()
+
+ pipe_job = await prepare_pipe_job(
+ pipe=sequence,
+ library_id=library_id,
+ execution_config=get_config().pipelex.pipeline_execution_config,
+ pipe_run_mode=PipeRunMode.LIVE,
+ pipeline_run_id="test-optional-boundary-seeding",
+ user_id="pytest",
+ inputs=PipelineInputs({"topic": "penalties"}),
+ )
+
+ job_memory = pipe_job.working_memory
+ assert job_memory is not None
+ source_record = job_memory.get_optional_absence("source")
+ assert source_record is not None
+ assert source_record.kind == AbsenceKind.NOT_PROVIDED
+
+ pipe_output = await sequence.run_pipe(
+ job_metadata=pipe_job.job_metadata,
+ working_memory=job_memory,
+ pipe_run_params=pipe_job.pipe_run_params,
+ )
+ assert pipe_output.main_stuff.as_text.text == "report about penalties (no analysis available)"
diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
new file mode 100644
index 000000000..6c66132b3
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
@@ -0,0 +1,316 @@
+"""Runtime trichotomy on absent inputs (D3): a plain input fed a recorded absence lifts (skips)
+the pipe with provenance; a `?` input runs; a `!` input raises a typed error carrying the chain.
+A missing input with no absence record stays a hard PipeRunInputsError — that is the bug case.
+
+All tests run PipeFunc pipes live (no inference) and sink absence before any method boundary.
+"""
+
+from typing import Callable, cast
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.config import get_config
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.inputs.exceptions import OptionalValueAbsentError, PipeRunInputsError
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.registries.func_registry import func_registry
+from tests.unit.pipelex.graph.conftest import make_trace_context
+
+
+def optionals_echo_source(working_memory: WorkingMemory) -> TextContent:
+ source_text = working_memory.get_stuff_as_str(name="source")
+ return TextContent(text=f"echo:{source_text}")
+
+
+def optionals_absorbing_sink(working_memory: WorkingMemory) -> TextContent:
+ source_stuff = working_memory.get_optional_stuff(name="source")
+ if source_stuff is None:
+ return TextContent(text="report:source-absent")
+ return TextContent(text=f"report:{source_stuff.as_text.text}")
+
+
+def optionals_list_from_source(working_memory: WorkingMemory) -> ListContent[TextContent]:
+ source_text = working_memory.get_stuff_as_str(name="source")
+ return ListContent[TextContent](items=[TextContent(text=source_text)])
+
+
+_TEST_FUNCS = [optionals_echo_source, optionals_absorbing_sink, optionals_list_from_source]
+
+
+def _make_pipe_func(pipe_code: str, *, source_ref: str, function_name: str = "optionals_echo_source", output: str = "Text") -> PipeFunc:
+ blueprint = PipeFuncBlueprint(
+ description=f"Optionals trichotomy test pipe {pipe_code}",
+ inputs={"source": source_ref},
+ output=output,
+ function_name=function_name,
+ )
+ return PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code="test_optionals",
+ pipe_code=pipe_code,
+ blueprint=blueprint,
+ )
+
+
+def _make_source_stuff(text: str = "hello") -> Stuff:
+ return StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ content=TextContent(text=text),
+ name="source",
+ )
+
+
+def _make_seeded_absence() -> AbsenceRecord:
+ return AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.DECLARED_ABSENT,
+ reason="no penalty clause found in this contract",
+ producing_pipe="extract_penalty_clause",
+ )
+
+
+def _make_live_run_params() -> PipeRunParams:
+ # Constructed directly (not via the factory) so a keyless boot's forced-DRY coercion cannot
+ # silently swap which code path (live vs dry) the test exercises.
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+
[email protected](loop_scope="class")
+class TestLiftSkipTrichotomy:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ async def test_plain_input_absent_with_record_lifts_pipe(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """A plain input fed a recorded absence skips the pipe and records a chained skipped-absence."""
+ load_empty_library()
+ pipe = _make_pipe_func("opt_plain_consumer", source_ref="Text")
+ seeded = _make_seeded_absence()
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.record_absence(seeded)
+
+ pipe_output = await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ output_name="echoed",
+ )
+
+ result_memory = pipe_output.working_memory
+ assert result_memory.get_optional_stuff("echoed") is None
+ skipped_record = result_memory.get_optional_absence("echoed")
+ assert skipped_record is not None
+ assert skipped_record.kind == AbsenceKind.SKIPPED
+ assert skipped_record.producing_pipe == "opt_plain_consumer"
+ assert "source" in skipped_record.reason
+ assert skipped_record.upstream == seeded
+ # The skipped pipe's output is the resolved-as-absent main result.
+ resolved_main = result_memory.resolve_main_stuff()
+ assert isinstance(resolved_main, AbsenceRecord)
+ assert result_memory.get_optional_main_stuff() is None
+ # Run identity is stamped like on every other path.
+ assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+
+ async def test_lift_chain_provenance(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Two lifted pipes in a row chain their skipped-absence records back to the origin."""
+ load_empty_library()
+ pipe_first = _make_pipe_func("opt_chain_first", source_ref="Text")
+ blueprint_second = PipeFuncBlueprint(
+ description="Second chained consumer",
+ inputs={"echoed": "Text"},
+ output="Text",
+ function_name="optionals_absorbing_sink",
+ )
+ pipe_second = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code="test_optionals",
+ pipe_code="opt_chain_second",
+ blueprint=blueprint_second,
+ )
+ seeded = _make_seeded_absence()
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.record_absence(seeded)
+
+ first_output = await pipe_first.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ output_name="echoed",
+ )
+ second_output = await pipe_second.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=first_output.working_memory,
+ pipe_run_params=_make_live_run_params(),
+ output_name="final",
+ )
+
+ final_record = second_output.working_memory.get_optional_absence("final")
+ assert final_record is not None
+ assert final_record.kind == AbsenceKind.SKIPPED
+ chain = final_record.provenance_chain()
+ assert [record.variable_name for record in chain] == ["final", "echoed", "source"]
+ assert final_record.origin() == seeded
+
+ async def test_optional_input_absent_runs_pipe(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """A `?` input absorbs absence: the pipe runs and the taint terminates."""
+ load_empty_library()
+ pipe = _make_pipe_func("opt_absorber", source_ref="Text?", function_name="optionals_absorbing_sink")
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.record_absence(_make_seeded_absence())
+
+ pipe_output = await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ output_name="report",
+ )
+
+ assert pipe_output.main_stuff.as_text.text == "report:source-absent"
+ assert pipe_output.working_memory.get_optional_absence("report") is None
+
+ async def test_force_input_absent_raises_typed_error(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """A `!` input on a recorded absence raises OptionalValueAbsentError naming the variable,
+ the consuming pipe, the producing pipe, and the original reason (D9 failure UX).
+ """
+ load_empty_library()
+ pipe = _make_pipe_func("opt_forcer", source_ref="Text!")
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.record_absence(_make_seeded_absence())
+
+ with pytest.raises(OptionalValueAbsentError) as exc_info:
+ await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ error = exc_info.value
+ message = str(error)
+ assert "source" in message
+ assert "opt_forcer" in message
+ assert "extract_penalty_clause" in message
+ assert "no penalty clause found in this contract" in message
+ assert error.variable_name == "source"
+ assert error.pipe_code == "opt_forcer"
+ assert error.absence_record.origin().producing_pipe == "extract_penalty_clause"
+
+ async def test_force_input_present_runs(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """`!` on a present value is a no-op assertion: the pipe runs normally."""
+ load_empty_library()
+ pipe = _make_pipe_func("opt_forcer_fed", source_ref="Text!")
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(_make_source_stuff("hello"))
+
+ pipe_output = await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+ assert pipe_output.main_stuff.as_text.text == "echo:hello"
+
+ async def test_missing_without_record_still_raises_inputs_error(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Absent with no record is a genuine miss: PipeRunInputsError, whose live-path message
+ no longer claims to be a dry run (pre-existing message bug fixed in Step B).
+ """
+ load_empty_library()
+ pipe = _make_pipe_func("opt_no_record", source_ref="Text")
+ working_memory = WorkingMemoryFactory.make_empty()
+
+ with pytest.raises(PipeRunInputsError) as exc_info:
+ await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+ message = str(exc_info.value)
+ assert "source" in message
+ assert "Dry run" not in message
+
+ async def test_skipped_plural_output_normalizes_to_empty_list(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """A lifted pipe with plural output writes an empty ListContent (plus a ledger note),
+ not an absent slot — taint stops there (D4).
+ """
+ load_empty_library()
+ pipe = _make_pipe_func(
+ "opt_plural_consumer",
+ source_ref="Text",
+ function_name="optionals_list_from_source",
+ output="Text[]",
+ )
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.record_absence(_make_seeded_absence())
+
+ pipe_output = await pipe.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ output_name="items",
+ )
+
+ result_memory = pipe_output.working_memory
+ items_stuff = result_memory.get_optional_stuff("items")
+ assert items_stuff is not None
+ list_content = items_stuff.content
+ assert isinstance(list_content, ListContent)
+ assert cast("ListContent[TextContent]", list_content).items == []
+ # The value wins for consumers; the ledger note stays for observability.
+ assert isinstance(result_memory.resolve_stuff("items"), Stuff)
+ note = result_memory.get_optional_absence("items")
+ assert note is not None
+ assert note.kind == AbsenceKind.SKIPPED
+ # The empty list is a real output: it is the main stuff.
+ assert pipe_output.main_stuff.stuff_name == "items"
+
+ async def test_lifted_pipe_under_tracer_ends_node_skipped(
+ self, job_metadata: JobMetadata, load_empty_library: Callable[[], str], mocker: MockerFixture
+ ):
+ """The graph-tracer epilogue fires on every pipe run: a lifted pipe ends its node in the
+ distinct `skipped` state with the skip reason (Step E) — never a crash, never an error node.
+ """
+ load_empty_library()
+ pipe = _make_pipe_func("opt_lift_traced", source_ref="Text")
+ working_memory = WorkingMemoryFactory.make_empty()
+ working_memory.record_absence(_make_seeded_absence())
+
+ mock_manager = mocker.MagicMock()
+ mock_manager.on_pipe_start = mocker.MagicMock(return_value=("node-1", None))
+ mocker.patch(
+ "pipelex.core.pipes.pipe_abstract.GraphTracerManager.get_instance",
+ return_value=mock_manager,
+ )
+ traced_metadata = JobMetadata(
+ user_id="pytest",
+ pipeline_run_id=job_metadata.pipeline_run_id,
+ trace_context=make_trace_context(graph_id=job_metadata.pipeline_run_id),
+ )
+
+ pipe_output = await pipe.run_pipe(
+ job_metadata=traced_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ output_name="echoed",
+ )
+
+ assert isinstance(pipe_output.working_memory.resolve_main_stuff(), AbsenceRecord)
+ mock_manager.on_pipe_end_skipped.assert_called_once()
+ assert "source" in mock_manager.on_pipe_end_skipped.call_args.kwargs["skip_reason"]
+ mock_manager.on_pipe_end_success.assert_not_called()
+ mock_manager.on_pipe_end_error.assert_not_called()
diff --git a/tests/integration/pipelex/pipes/optionals/test_liftable_pipes_inventory.py b/tests/integration/pipelex/pipes/optionals/test_liftable_pipes_inventory.py
new file mode 100644
index 000000000..2affcd638
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_liftable_pipes_inventory.py
@@ -0,0 +1,181 @@
+"""Liftable-pipe inventory (D3 phase-1 commitment): the valid report carries structured data
+listing every pipe that may be skipped (lifted) when an optional slot is absent — the build-time
+visibility that makes implicit lifting acceptable.
+"""
+
+from typing import Callable
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipeline.controller_taint import collect_controller_taint_analyses
+from pipelex.pipeline.liftable_pipes import build_liftable_pipes
+from pipelex.pipeline.validation_report import PipelexValidationReport
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_inventory"
+
+
+def optionals_inv_echo_source(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"a:{working_memory.get_stuff_as_str(name='source')}")
+
+
+def optionals_inv_echo_a_out(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"b:{working_memory.get_stuff_as_str(name='a_out')}")
+
+
+def optionals_inv_sink(working_memory: WorkingMemory) -> TextContent:
+ b_out_stuff = working_memory.get_optional_stuff(name="b_out")
+ return TextContent(text="ok" if b_out_stuff is None else b_out_stuff.as_text.text)
+
+
+def optionals_inv_echo_topic(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"base:{working_memory.get_stuff_as_str(name='topic')}")
+
+
+_TEST_FUNCS = [optionals_inv_echo_source, optionals_inv_echo_a_out, optionals_inv_sink, optionals_inv_echo_topic]
+
+
+def _build_lift_chain_pipes() -> list[str]:
+ """Register the lift-chain sequence (A and B liftable, C absorbs) and return the pipe codes."""
+ step_a = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_step_a",
+ blueprint=PipeFuncBlueprint(
+ description="Liftable when source is absent",
+ inputs={"source": "Text"},
+ output="Text",
+ function_name="optionals_inv_echo_source",
+ ),
+ )
+ step_b = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_step_b",
+ blueprint=PipeFuncBlueprint(
+ description="Liftable in chain",
+ inputs={"a_out": "Text"},
+ output="Text",
+ function_name="optionals_inv_echo_a_out",
+ ),
+ )
+ sink_c = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_sink_c",
+ blueprint=PipeFuncBlueprint(
+ description="Absorbing sink",
+ inputs={"b_out": "Text?"},
+ output="Text",
+ function_name="optionals_inv_sink",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [step_a, step_b, sink_c]:
+ pipe_library.add_new_pipe(pipe=pipe)
+ sequence = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_sequence",
+ blueprint=PipeSequenceBlueprint(
+ description="Lift chain",
+ inputs={"source": "Text?"},
+ output="Text",
+ steps=[
+ SubPipeBlueprint(pipe="inv_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="inv_step_b", result="b_out"),
+ SubPipeBlueprint(pipe="inv_sink_c", result="final_report"),
+ ],
+ ),
+ )
+ pipe_library.add_new_pipe(pipe=sequence)
+ return ["inv_sequence", "inv_step_a", "inv_step_b", "inv_sink_c"]
+
+
+class TestLiftablePipesInventory:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ def test_sequence_lift_chain_is_inventoried(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ pipe_codes = _build_lift_chain_pipes()
+ pipes = [get_pipe_library().get_required_pipe(pipe_code=pipe_code) for pipe_code in pipe_codes]
+
+ entries = build_liftable_pipes(collect_controller_taint_analyses(pipes))
+
+ by_pipe_ref = {entry.pipe_ref: entry for entry in entries}
+ assert set(by_pipe_ref.keys()) == {f"{_DOMAIN_CODE}.inv_step_a", f"{_DOMAIN_CODE}.inv_step_b"}
+
+ step_a_entry = by_pipe_ref[f"{_DOMAIN_CODE}.inv_step_a"]
+ assert step_a_entry.within_pipe_ref == f"{_DOMAIN_CODE}.inv_sequence"
+ assert step_a_entry.skipped_when_absent == ["source"]
+ assert "source" in step_a_entry.absence_source
+
+ step_b_entry = by_pipe_ref[f"{_DOMAIN_CODE}.inv_step_b"]
+ assert step_b_entry.skipped_when_absent == ["a_out"]
+
+ def test_parallel_liftable_branch_is_inventoried(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ branch_found = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_par_find",
+ blueprint=PipeFuncBlueprint(
+ description="Liftable branch",
+ inputs={"source": "Text"},
+ output="Text",
+ function_name="optionals_inv_echo_source",
+ ),
+ )
+ branch_base = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_par_base",
+ blueprint=PipeFuncBlueprint(
+ description="Guaranteed branch",
+ inputs={"topic": "Text"},
+ output="Text",
+ function_name="optionals_inv_echo_topic",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [branch_found, branch_base]:
+ pipe_library.add_new_pipe(pipe=pipe)
+ parallel = PipeFactory[PipeParallel].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="inv_parallel",
+ blueprint=PipeParallelBlueprint(
+ description="One liftable branch",
+ inputs={"source": "Text?", "topic": "Text"},
+ output="Composite",
+ branches=[
+ SubPipeBlueprint(pipe="inv_par_find", result="found_result"),
+ SubPipeBlueprint(pipe="inv_par_base", result="base_result"),
+ ],
+ add_each_output=False,
+ ),
+ )
+ pipe_library.add_new_pipe(pipe=parallel)
+
+ entries = build_liftable_pipes(collect_controller_taint_analyses([parallel, branch_found, branch_base]))
+
+ assert len(entries) == 1
+ entry = entries[0]
+ assert entry.pipe_ref == f"{_DOMAIN_CODE}.inv_par_find"
+ assert entry.within_pipe_ref == f"{_DOMAIN_CODE}.inv_parallel"
+ assert entry.skipped_when_absent == ["source"]
+
+ def test_validation_report_carries_the_inventory_field(self):
+ """The valid report exposes `liftable_pipes` beside `pipe_io_contracts` (empty by default)."""
+ assert "liftable_pipes" in PipelexValidationReport.model_fields
diff --git a/tests/integration/pipelex/pipes/optionals/test_lifted_parallel_companion_slots.py b/tests/integration/pipelex/pipes/optionals/test_lifted_parallel_companion_slots.py
new file mode 100644
index 000000000..4e58ad0e3
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_lifted_parallel_companion_slots.py
@@ -0,0 +1,177 @@
+"""When a whole `add_each_output` PipeParallel is lifted, every branch result slot it would
+have written must be RESOLVED, not just the combined output: a recorded absence for singular
+branch slots, an empty list for plural ones (D4). Otherwise a downstream consumer that the
+static taint pass blessed meets a neither-value-nor-record hard miss at its gate.
+"""
+
+from typing import Callable, cast
+
+import pytest
+
+from pipelex.config import get_config
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_companion"
+
+
+def optionals_comp_find(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"found:{working_memory.get_stuff_as_str(name='source')}")
+
+
+def optionals_comp_find_many(working_memory: WorkingMemory) -> ListContent[TextContent]:
+ return ListContent[TextContent](items=[TextContent(text=f"item:{working_memory.get_stuff_as_str(name='source')}")])
+
+
+def optionals_comp_sink(working_memory: WorkingMemory) -> TextContent:
+ a_out_stuff = working_memory.get_optional_stuff(name="a_out")
+ if a_out_stuff is None:
+ return TextContent(text="no findings")
+ return TextContent(text=f"findings: {a_out_stuff.as_text.text}")
+
+
+_TEST_FUNCS = [optionals_comp_find, optionals_comp_find_many, optionals_comp_sink]
+
+
+def _make_live_run_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+
+def _build_sequence_with_liftable_parallel() -> PipeSequence:
+ """The parallel declares `source` PLAIN, so an absent `source` lifts it wholesale; its
+ branch slots (`a_out` singular, `many_out` plural) land in the sequence flow via
+ add_each_output; the sink absorbs `a_out` with a `?` input.
+ """
+ branch_single = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="comp_find",
+ blueprint=PipeFuncBlueprint(
+ description="Singular branch over source",
+ inputs={"source": "Text"},
+ output="Text",
+ function_name="optionals_comp_find",
+ ),
+ )
+ branch_plural = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="comp_find_many",
+ blueprint=PipeFuncBlueprint(
+ description="Plural branch over source",
+ inputs={"source": "Text"},
+ output="Text[]",
+ function_name="optionals_comp_find_many",
+ ),
+ )
+ sink = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="comp_sink",
+ blueprint=PipeFuncBlueprint(
+ description="Absorbs the singular branch slot",
+ inputs={"a_out": "Text?"},
+ output="Text",
+ function_name="optionals_comp_sink",
+ ),
+ )
+ parallel = PipeFactory[PipeParallel].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="comp_parallel",
+ blueprint=PipeParallelBlueprint(
+ description="Lifted wholesale when source is absent",
+ inputs={"source": "Text"},
+ output="Composite",
+ branches=[
+ SubPipeBlueprint(pipe="comp_find", result="a_out"),
+ SubPipeBlueprint(pipe="comp_find_many", result="many_out"),
+ ],
+ add_each_output=True,
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [branch_single, branch_plural, sink, parallel]:
+ pipe_library.add_new_pipe(pipe=pipe)
+ sequence = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="comp_sequence",
+ blueprint=PipeSequenceBlueprint(
+ description="Lifted parallel then absorbing sink",
+ inputs={"source": "Text?"},
+ output="Text",
+ steps=[
+ SubPipeBlueprint(pipe="comp_parallel", result="combined"),
+ SubPipeBlueprint(pipe="comp_sink", result="final_report"),
+ ],
+ ),
+ )
+ pipe_library.add_new_pipe(pipe=sequence)
+ return sequence
+
+
[email protected](loop_scope="class")
+class TestLiftedParallelCompanionSlots:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ async def test_lifted_parallel_resolves_branch_slots(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """Static validation blesses the flow AND the runtime resolves every branch slot on lift:
+ singular → SKIPPED record with provenance; plural → empty list. The absorbing sink runs.
+ """
+ load_empty_library()
+ sequence = _build_sequence_with_liftable_parallel()
+ # The static pass must accept exactly what the runtime produces.
+ sequence.validate_with_libraries()
+
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(StuffFactory.make_from_str("penalties", name="topic"))
+ source_record = AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason="optional input 'source' was not provided by the caller",
+ )
+ working_memory.record_absence(source_record)
+
+ pipe_output = await sequence.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+
+ assert pipe_output.main_stuff.as_text.text == "no findings"
+
+ result_memory = pipe_output.working_memory
+ # Singular branch slot: recorded absent with provenance chaining to the source record.
+ a_out_record = result_memory.get_optional_absence("a_out")
+ assert a_out_record is not None
+ assert a_out_record.kind == AbsenceKind.SKIPPED
+ assert a_out_record.producing_pipe == "comp_find"
+ assert a_out_record.upstream == source_record
+ assert result_memory.get_optional_stuff("a_out") is None
+ # Plural branch slot: guaranteed empty list (D4), with an observability note.
+ many_out_stuff = result_memory.get_optional_stuff("many_out")
+ assert many_out_stuff is not None
+ many_out_content = many_out_stuff.content
+ assert isinstance(many_out_content, ListContent)
+ assert cast("ListContent[TextContent]", many_out_content).items == []
diff --git a/tests/integration/pipelex/pipes/optionals/test_optional_gate_tolerates_missing.py b/tests/integration/pipelex/pipes/optionals/test_optional_gate_tolerates_missing.py
new file mode 100644
index 000000000..067adf27b
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_optional_gate_tolerates_missing.py
@@ -0,0 +1,108 @@
+"""The `@?` fix, observable at the runtime miss-gates: a variable that a pipe's templates
+reference but whose input is declared optional (`?`) must NOT be presence-required by the
+SubPipe / PipeCondition gates. With neither a value nor an absence record for the slot, the
+pipe still runs and its guarded template takes the absent arm (matching the pipe's own
+`validate_before_run` trichotomy, which already tolerates a missing optional).
+"""
+
+from typing import Callable
+
+import pytest
+
+from pipelex.config import get_config
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.compose.pipe_compose import PipeCompose
+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import PipeRunParams
+from pipelex.pipeline.job_metadata import JobMetadata
+
+_DOMAIN_CODE = "test_optionals_gate"
+
+
+def _make_live_run_params() -> PipeRunParams:
+ return PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)
+
+
+def _register_guarded_compose() -> None:
+ """A PipeCompose whose template references the optional `maybe_note` (guarded), so its
+ `required_variables()` include the optional name — exactly what trips a naive gate.
+ """
+ compose = PipeFactory[PipeCompose].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="gate_compose_sink",
+ blueprint=PipeComposeBlueprint(
+ description="Guarded compose over an optional slot",
+ inputs={"topic": "Text", "maybe_note": "Text?"},
+ output="Text",
+ template="report about {{ topic }}{% if maybe_note %} (note: {{ maybe_note }}){% endif %}",
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=compose)
+
+
[email protected](loop_scope="class")
+class TestOptionalGateToleratesMissing:
+ async def test_sub_pipe_gate_tolerates_missing_optional(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """`maybe_note` has neither a value nor a record: the SubPipe gate must not hard-fail;
+ the compose runs on the absent arm.
+ """
+ load_empty_library()
+ _register_guarded_compose()
+ sequence = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="gate_sequence",
+ blueprint=PipeSequenceBlueprint(
+ description="Single guarded step",
+ inputs={"topic": "Text", "maybe_note": "Text?"},
+ output="Text",
+ steps=[SubPipeBlueprint(pipe="gate_compose_sink", result="final_report")],
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=sequence)
+
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(StuffFactory.make_from_str("penalties", name="topic"))
+
+ pipe_output = await sequence.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+ assert pipe_output.main_stuff.as_text.text == "report about penalties"
+
+ async def test_condition_gate_tolerates_missing_optional(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
+ """The PipeCondition chosen-pipe gate applies the same rule for the chosen pipe's
+ declared-optional inputs.
+ """
+ load_empty_library()
+ _register_guarded_compose()
+ condition = PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="gate_condition",
+ blueprint=PipeConditionBlueprint(
+ description="Routes to the guarded compose",
+ inputs={"topic": "Text", "maybe_note": "Text?"},
+ output="Text",
+ expression="'go'",
+ outcomes={"go": "gate_compose_sink"},
+ default_outcome="fail",
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=condition)
+
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(StuffFactory.make_from_str("penalties", name="topic"))
+
+ pipe_output = await condition.run_pipe(
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=_make_live_run_params(),
+ )
+ assert pipe_output.main_stuff.as_text.text == "report about penalties"
diff --git a/tests/integration/pipelex/pipes/optionals/test_optional_guard_lint.py b/tests/integration/pipelex/pipes/optionals/test_optional_guard_lint.py
new file mode 100644
index 000000000..46c7b9885
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_optional_guard_lint.py
@@ -0,0 +1,225 @@
+"""Template guard-lint (D7): every template reference to a declared-optional input must be
+guarded (`{% if var %}` block, inline `is defined` conditional, or `@?var`). An unguarded
+reference is a typed validation error (`OPTIONAL_INPUT_UNGUARDED`) with the precise fix.
+Covers PipeLLM prompts/system prompts, PipeCondition expressions, and PipeCompose templates.
+"""
+
+from typing import Callable
+
+import pytest
+from pydantic import ValidationError
+
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
+from pipelex.core.pipes.handle_pipe_errors import extract_wrapped_pipe_validation_error
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_operators.compose.pipe_compose import PipeCompose
+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint
+from pipelex.pipe_operators.img_gen.pipe_img_gen import PipeImgGen
+from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+from pipelex.pipe_operators.search.pipe_search import PipeSearch
+from pipelex.pipe_operators.search.pipe_search_blueprint import PipeSearchBlueprint
+
+_DOMAIN_CODE = "test_optionals_guard_lint"
+
+
+def _make_pipe_llm(*, prompt: str, system_prompt: str | None = None) -> PipeLLM:
+ return PipeFactory[PipeLLM].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_llm",
+ blueprint=PipeLLMBlueprint(
+ description="Guard-lint test pipe",
+ inputs={"contract": "Text", "assessment": "Text?"},
+ output="Text",
+ prompt=prompt,
+ system_prompt=system_prompt,
+ ),
+ )
+
+
+class TestOptionalGuardLint:
+ # ---- PipeLLM prompts ----
+
+ @pytest.mark.parametrize(
+ ("topic", "prompt"),
+ [
+ ("at_optional_sigil", "Write a report.\n@contract\n@?assessment"),
+ ("if_block", "Write a report.\n@contract\n{% if assessment %}Consider: {{ assessment }}{% endif %}"),
+ ("if_defined_block", "Write a report.\n@contract\n{% if assessment is defined %}{{ assessment }}{% endif %}"),
+ ("inline_cond", "Report ({{ 'with assessment' if assessment is defined else 'no assessment' }}).\n@contract"),
+ ],
+ )
+ def test_guarded_prompts_are_accepted(self, topic: str, prompt: str, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ pipe_llm = _make_pipe_llm(prompt=prompt)
+ assert pipe_llm.code == "guard_lint_llm", f"{topic}: construction should pass the lint"
+
+ @pytest.mark.parametrize(
+ ("topic", "prompt"),
+ [
+ ("bare_interpolation", "Write a report.\n@contract\n{{ assessment }}"),
+ ("deep_access", "Write a report.\n@contract\n{{ assessment.amount }}"),
+ ("plain_at_sigil_on_optional", "Write a report.\n@contract\n@assessment"),
+ ("else_branch_use", "@contract\n{% if assessment %}x{% else %}{{ assessment }}{% endif %}"),
+ ],
+ )
+ def test_unguarded_prompts_are_rejected(self, topic: str, prompt: str, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ _make_pipe_llm(prompt=prompt)
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None, f"{topic}: expected a wrapped PipeValidationError"
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED
+ assert "assessment" in str(wrapped)
+ # The fix is named precisely.
+ assert "@?assessment" in str(wrapped) or "{% if assessment %}" in str(wrapped)
+
+ def test_unguarded_system_prompt_is_rejected(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ _make_pipe_llm(prompt="Write a report.\n@contract\n@?assessment", system_prompt="You know: {{ assessment }}")
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED
+
+ def test_unguarded_reference_to_plain_input_is_not_linted(self, load_empty_library: Callable[[], None]):
+ """The lint only applies to declared-optional inputs."""
+ load_empty_library()
+ pipe_llm = _make_pipe_llm(prompt="Write a report.\n$contract\n@?assessment")
+ assert pipe_llm.code == "guard_lint_llm"
+
+ # ---- PipeCondition expressions ----
+
+ def test_unguarded_optional_in_condition_expression_is_rejected(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_condition",
+ blueprint=PipeConditionBlueprint(
+ description="Condition on a maybe-absent slot, unguarded",
+ inputs={"maybe_var": "Text?"},
+ output="Text?",
+ expression="maybe_var",
+ outcomes={"yes": "continue"},
+ default_outcome="continue",
+ ),
+ )
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED
+
+ def test_presence_branching_idiom_is_accepted(self, load_empty_library: Callable[[], None]):
+ """The design §15 idiom: branch on presence with an `is defined` inline conditional."""
+ load_empty_library()
+ condition = PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_condition_ok",
+ blueprint=PipeConditionBlueprint(
+ description="Condition branching on presence",
+ inputs={"maybe_var": "Text?"},
+ output="Text?",
+ expression_template="{{ 'present' if maybe_var is defined else 'absent' }}",
+ outcomes={"present": "continue", "absent": "continue"},
+ default_outcome="continue",
+ ),
+ )
+ assert condition.code == "guard_lint_condition_ok"
+
+ # ---- PipeCompose templates ----
+
+ def test_unguarded_optional_in_compose_template_is_rejected(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ PipeFactory[PipeCompose].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_compose",
+ blueprint=PipeComposeBlueprint(
+ description="Compose over a maybe-absent slot, unguarded",
+ inputs={"maybe_note": "Text?"},
+ output="Text",
+ template="note: {{ maybe_note }}",
+ ),
+ )
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED
+
+ # ---- PipeSearch / PipeImgGen templates ----
+
+ def test_unguarded_optional_in_search_prompt_is_rejected(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ PipeFactory[PipeSearch].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_search",
+ blueprint=PipeSearchBlueprint(
+ description="Search over a maybe-absent slot, unguarded",
+ inputs={"maybe_topic": "Text?"},
+ output="SearchResult[]",
+ prompt="latest news about {{ maybe_topic }}",
+ ),
+ )
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED
+
+ def test_unguarded_optional_in_img_gen_prompt_is_rejected(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ with pytest.raises(ValidationError) as exc_info:
+ PipeFactory[PipeImgGen].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_img_gen",
+ blueprint=PipeImgGenBlueprint(
+ description="Image prompt over a maybe-absent slot, unguarded",
+ inputs={"maybe_style": "Text?"},
+ output="Image",
+ prompt="a landscape in the style of {{ maybe_style }}",
+ ),
+ )
+ wrapped = extract_wrapped_pipe_validation_error(exc_info.value.errors()[0])
+ assert wrapped is not None
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED
+
+ # ---- Unparseable templates stay the parsing gates' concern ----
+
+ def test_broken_expression_does_not_leak_internal_error(self, load_empty_library: Callable[[], None]):
+ """A syntactically broken expression on a pipe with optional inputs must not surface the
+ internal Jinja2DetectVariablesError through the guard-lint at construction time — syntax
+ errors belong to the template-parsing gates and their typed channels.
+ """
+ load_empty_library()
+ try:
+ PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_broken_expression",
+ blueprint=PipeConditionBlueprint(
+ description="Condition with a broken expression over an optional input",
+ inputs={"maybe_var": "Text?"},
+ output="Text?",
+ expression_template="{{ maybe_var.",
+ outcomes={"yes": "continue"},
+ default_outcome="continue",
+ ),
+ )
+ except ValidationError:
+ # A wrapped pydantic error from a parsing gate is fine — a raw
+ # Jinja2DetectVariablesError escaping construction is what this test forbids.
+ pass
+
+ def test_guarded_compose_template_is_accepted(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
+ compose = PipeFactory[PipeCompose].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="guard_lint_compose_ok",
+ blueprint=PipeComposeBlueprint(
+ description="Compose handling both arms",
+ inputs={"maybe_note": "Text?"},
+ output="Text",
+ template="{% if maybe_note %}note: {{ maybe_note }}{% else %}no note{% endif %}",
+ ),
+ )
+ assert compose.code == "guard_lint_compose_ok"
diff --git a/tests/integration/pipelex/pipes/optionals/test_parallel_optional_combine_validation.py b/tests/integration/pipelex/pipes/optionals/test_parallel_optional_combine_validation.py
new file mode 100644
index 000000000..0f9637df0
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_parallel_optional_combine_validation.py
@@ -0,0 +1,182 @@
+"""D11 static rule: a REQUIRED structure field fed by a maybe-absent branch is a validation error
+(extends `PipeParallel.validate_output_with_library`). A branch is maybe-absent when its pipe is
+liftable inside the parallel (plain input fed by the parallel's `?` input) or when it declares a
+`?` output. Non-required fields absorb (field-level None); `Composite` omits the component.
+"""
+
+from typing import Callable
+
+import pytest
+from pydantic import Field
+
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry, get_concept_library, get_pipe_library
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_par_static"
+
+
+class ParTaintReport(StructuredContent):
+ """Combination target with one required and one absorbable (non-required) field."""
+
+ base_result: TextContent = Field(description="Result of the always-guaranteed branch")
+ found_result: TextContent | None = Field(default=None, description="Result of the maybe-absent branch")
+
+
+class ParTaintStrictReport(StructuredContent):
+ """Combination target whose fields are all required — no absorption possible."""
+
+ base_result: TextContent = Field(description="Result of the always-guaranteed branch")
+ found_result: TextContent = Field(description="Result of the maybe-absent branch")
+
+
+def optionals_par_static_find(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"found:{working_memory.get_stuff_as_str(name='source')}")
+
+
+def optionals_par_static_base(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"base:{working_memory.get_stuff_as_str(name='topic')}")
+
+
+_TEST_FUNCS = [optionals_par_static_find, optionals_par_static_base]
+
+
+def _register_branch_pipes(*, find_output_ref: str = "Text") -> None:
+ branch_found = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="par_static_find",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes source plain: liftable when the parallel's source is optional",
+ inputs={"source": "Text"},
+ output=find_output_ref,
+ function_name="optionals_par_static_find",
+ ),
+ )
+ branch_base = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="par_static_base",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes the always-present topic",
+ inputs={"topic": "Text"},
+ output="Text",
+ function_name="optionals_par_static_base",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [branch_found, branch_base]:
+ pipe_library.add_new_pipe(pipe=pipe)
+
+
+def _build_parallel(*, output_ref: str, structure_class_names: list[str], inputs: dict[str, str]) -> PipeParallel:
+ for structure_class_name in structure_class_names:
+ concept = ConceptFactory.make(
+ concept_code=structure_class_name,
+ domain_code=_DOMAIN_CODE,
+ description=f"Test combination concept {structure_class_name}",
+ structure_class_name=structure_class_name,
+ )
+ get_concept_library().add_new_concept(concept=concept)
+
+ parallel = PipeFactory[PipeParallel].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="par_static_combine",
+ blueprint=PipeParallelBlueprint(
+ description="Parallel over a maybe-absent branch and a guaranteed branch",
+ inputs=inputs,
+ output=output_ref,
+ branches=[
+ SubPipeBlueprint(pipe="par_static_find", result="found_result"),
+ SubPipeBlueprint(pipe="par_static_base", result="base_result"),
+ ],
+ add_each_output=False,
+ ),
+ concept_codes_from_the_same_domain=structure_class_names,
+ )
+ get_pipe_library().add_new_pipe(pipe=parallel)
+ return parallel
+
+
+class TestParallelOptionalCombineValidation:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+ get_class_registry().register_class(ParTaintReport)
+ get_class_registry().register_class(ParTaintStrictReport)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ def test_required_field_fed_by_liftable_branch_is_rejected(self, load_empty_library: Callable[[], str]):
+ """The parallel's `?` input makes the find branch liftable, and 'found_result' is required."""
+ load_empty_library()
+ _register_branch_pipes()
+ parallel = _build_parallel(
+ output_ref="ParTaintStrictReport",
+ structure_class_names=["ParTaintStrictReport"],
+ inputs={"source": "Text?", "topic": "Text"},
+ )
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ parallel.validate_with_libraries()
+ error = exc_info.value
+ assert error.error_type == PipeValidationErrorType.OPTIONAL_BRANCH_REQUIRED_FIELD
+ assert "found_result" in str(error)
+ assert "par_static_find" in str(error)
+
+ def test_required_field_fed_by_optional_output_branch_is_rejected(self, load_empty_library: Callable[[], str]):
+ """A branch declaring a `?` output is maybe-absent even with all-plain parallel inputs."""
+ load_empty_library()
+ _register_branch_pipes(find_output_ref="Text?")
+ parallel = _build_parallel(
+ output_ref="ParTaintStrictReport",
+ structure_class_names=["ParTaintStrictReport"],
+ inputs={"source": "Text", "topic": "Text"},
+ )
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ parallel.validate_with_libraries()
+ assert exc_info.value.error_type == PipeValidationErrorType.OPTIONAL_BRANCH_REQUIRED_FIELD
+
+ def test_non_required_field_absorbs_the_maybe_absent_branch(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_branch_pipes()
+ parallel = _build_parallel(
+ output_ref="ParTaintReport",
+ structure_class_names=["ParTaintReport"],
+ inputs={"source": "Text?", "topic": "Text"},
+ )
+ parallel.validate_with_libraries()
+
+ def test_composite_output_accepts_maybe_absent_branches(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_branch_pipes()
+ parallel = _build_parallel(
+ output_ref="Composite",
+ structure_class_names=[],
+ inputs={"source": "Text?", "topic": "Text"},
+ )
+ parallel.validate_with_libraries()
+
+ def test_all_plain_inputs_keep_todays_rules(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_branch_pipes()
+ parallel = _build_parallel(
+ output_ref="ParTaintStrictReport",
+ structure_class_names=["ParTaintStrictReport"],
+ inputs={"source": "Text", "topic": "Text"},
+ )
+ parallel.validate_with_libraries()
diff --git a/tests/integration/pipelex/pipes/optionals/test_redundant_force_warning.py b/tests/integration/pipelex/pipes/optionals/test_redundant_force_warning.py
new file mode 100644
index 000000000..ddc662d34
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_redundant_force_warning.py
@@ -0,0 +1,143 @@
+"""The useless-`!` lint (Step E): a `!` input whose slot is guaranteed present in every analyzed
+flow can never fire — it surfaces as a `warnings` item on the validation report (advisory channel,
+never flips `is_valid`). A `!` that asserts a maybe-absent slot in at least one flow is meaningful
+and must NOT be warned about, even if another flow guarantees the slot.
+"""
+
+from typing import Callable
+
+from pipelex.base_exceptions import ValidationErrorCategory
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.pipeline.controller_taint import collect_controller_taint_analyses
+from pipelex.pipeline.optionality_warnings import build_optionality_warnings
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_redundant_force"
+
+
+def redundant_force_make_a(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"a:{working_memory.get_stuff_as_str(name='topic')}")
+
+
+def redundant_force_consume_a(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"b:{working_memory.get_stuff_as_str(name='a_out')}")
+
+
+_TEST_FUNCS = [redundant_force_make_a, redundant_force_consume_a]
+
+
+def _build_force_pipes() -> tuple[PipeSequence, PipeSequence]:
+ """Register the step pipes and build the two flows over the same `!`-consuming pipe.
+
+ - `rf_seq_guaranteed`: a plain-input step produces `a_out` before the `!` step consumes it —
+ the slot is guaranteed, the `!` is redundant in this flow.
+ - `rf_seq_asserting`: the sequence declares `a_out = "Text?"` at its boundary, so the same
+ `!` step asserts a maybe-absent slot — meaningful.
+ """
+ step_make = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="rf_step_make",
+ blueprint=PipeFuncBlueprint(
+ description="Always produces a_out",
+ inputs={"topic": "Text"},
+ output="Text",
+ function_name="redundant_force_make_a",
+ ),
+ )
+ step_force = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="rf_step_force",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes a_out with a force marker",
+ inputs={"a_out": "Text!"},
+ output="Text",
+ function_name="redundant_force_consume_a",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [step_make, step_force]:
+ pipe_library.add_new_pipe(pipe=pipe)
+
+ seq_guaranteed = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="rf_seq_guaranteed",
+ blueprint=PipeSequenceBlueprint(
+ description="a_out is produced by a guaranteed step before the force consumption",
+ inputs={"topic": "Text"},
+ output="Text",
+ steps=[
+ SubPipeBlueprint(pipe="rf_step_make", result="a_out"),
+ SubPipeBlueprint(pipe="rf_step_force", result="final"),
+ ],
+ ),
+ )
+ seq_asserting = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="rf_seq_asserting",
+ blueprint=PipeSequenceBlueprint(
+ description="a_out is maybe-absent at the boundary; the force consumption asserts it",
+ inputs={"a_out": "Text?"},
+ output="Text",
+ steps=[
+ SubPipeBlueprint(pipe="rf_step_force", result="final"),
+ ],
+ ),
+ )
+ for sequence in [seq_guaranteed, seq_asserting]:
+ pipe_library.add_new_pipe(pipe=sequence)
+ return seq_guaranteed, seq_asserting
+
+
+class TestRedundantForceWarning:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ def test_force_on_guaranteed_slot_warns(self, load_empty_library: Callable[[], str]):
+ """A `!` consumption whose slot is guaranteed in its only analyzed flow yields one warning."""
+ load_empty_library()
+ seq_guaranteed, _ = _build_force_pipes()
+
+ warnings = build_optionality_warnings(collect_controller_taint_analyses([seq_guaranteed]))
+
+ assert len(warnings) == 1
+ warning = warnings[0]
+ assert warning.category == ValidationErrorCategory.PIPE_VALIDATION
+ assert warning.error_type == PipeValidationErrorType.OPTIONAL_FORCE_REDUNDANT
+ assert warning.pipe_code == "rf_step_force"
+ assert warning.domain_code == _DOMAIN_CODE
+ assert warning.variable_names == ["a_out"]
+ assert "a_out" in warning.message
+ assert "!" in warning.message
+
+ def test_force_asserting_a_tainted_slot_does_not_warn(self, load_empty_library: Callable[[], str]):
+ """A `!` that asserts a maybe-absent slot is meaningful — no warning."""
+ load_empty_library()
+ _, seq_asserting = _build_force_pipes()
+
+ assert build_optionality_warnings(collect_controller_taint_analyses([seq_asserting])) == []
+
+ def test_meaningful_in_one_flow_silences_the_redundant_one(self, load_empty_library: Callable[[], str]):
+ """The lint aggregates across flows: one asserting flow silences the redundant observation,
+ so authors are never told to remove a `!` another flow relies on.
+ """
+ load_empty_library()
+ seq_guaranteed, seq_asserting = _build_force_pipes()
+
+ assert build_optionality_warnings(collect_controller_taint_analyses([seq_guaranteed, seq_asserting])) == []
diff --git a/tests/integration/pipelex/pipes/optionals/test_taint_sequence_validation.py b/tests/integration/pipelex/pipes/optionals/test_taint_sequence_validation.py
new file mode 100644
index 000000000..ea03088ba
--- /dev/null
+++ b/tests/integration/pipelex/pipes/optionals/test_taint_sequence_validation.py
@@ -0,0 +1,326 @@
+"""Static absence-taint pass over a PipeSequence's dataflow (D6): a maybe-absent slot must reach
+an explicit sink (`?` input, `!` input) before the sequence boundary, or the sequence must declare
+its output optional — otherwise validation fails with `OPTIONAL_NOT_HANDLED` naming the source,
+the propagation path, and the fixes.
+"""
+
+from typing import Callable
+
+import pytest
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_pipe_library
+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
+from pipelex.system.registries.func_registry import func_registry
+
+_DOMAIN_CODE = "test_optionals_taint_seq"
+
+
+def optionals_taint_echo_source(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"a:{working_memory.get_stuff_as_str(name='source')}")
+
+
+def optionals_taint_echo_a_out(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text=f"b:{working_memory.get_stuff_as_str(name='a_out')}")
+
+
+def optionals_taint_sink(working_memory: WorkingMemory) -> TextContent:
+ b_out_stuff = working_memory.get_optional_stuff(name="b_out")
+ if b_out_stuff is None:
+ return TextContent(text="no analysis")
+ return TextContent(text=f"analysis: {b_out_stuff.as_text.text}")
+
+
+_TEST_FUNCS = [optionals_taint_echo_source, optionals_taint_echo_a_out, optionals_taint_sink]
+
+
+def _register_step_pipes(*, source_input_ref: str = "Text") -> None:
+ """Register step A (consumes `source`), step B (consumes `a_out`), and the absorbing sink."""
+ step_a = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_step_a",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes source",
+ inputs={"source": source_input_ref},
+ output="Text",
+ function_name="optionals_taint_echo_source",
+ ),
+ )
+ step_b = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_step_b",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes step A's output",
+ inputs={"a_out": "Text"},
+ output="Text",
+ function_name="optionals_taint_echo_a_out",
+ ),
+ )
+ sink_c = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_sink_c",
+ blueprint=PipeFuncBlueprint(
+ description="Absorbing sink: declares b_out optional",
+ inputs={"b_out": "Text?"},
+ output="Text",
+ function_name="optionals_taint_sink",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ for pipe in [step_a, step_b, sink_c]:
+ pipe_library.add_new_pipe(pipe=pipe)
+
+
+def _build_sequence(*, output_ref: str, steps: list[SubPipeBlueprint], inputs: dict[str, str]) -> PipeSequence:
+ sequence = PipeFactory[PipeSequence].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_sequence",
+ blueprint=PipeSequenceBlueprint(
+ description="Taint-pass test sequence",
+ inputs=inputs,
+ output=output_ref,
+ steps=steps,
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=sequence)
+ return sequence
+
+
+class TestSequenceTaintValidation:
+ @classmethod
+ def setup_class(cls):
+ for func in _TEST_FUNCS:
+ func_registry.register_function(func)
+
+ @classmethod
+ def teardown_class(cls):
+ for func in _TEST_FUNCS:
+ if func_registry.has_function(func.__name__):
+ func_registry.unregister_function_by_name(func.__name__)
+
+ def test_taint_reaching_plain_sequence_output_is_rejected(self, load_empty_library: Callable[[], str]):
+ """Optional sequence input feeds a lift chain that ends the sequence: plain output → error
+ naming the source, the tainted variable, and the three fixes.
+ """
+ load_empty_library()
+ _register_step_pipes()
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"source": "Text?"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ ],
+ )
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ sequence.validate_with_libraries()
+ error = exc_info.value
+ assert error.error_type == PipeValidationErrorType.OPTIONAL_NOT_HANDLED
+ assert error.pipe_code == "taint_sequence"
+ # The structured field carries the origin variable identifier, not prose.
+ assert error.variable_names == ["source"]
+ # The message names the absence source and the fixes.
+ assert "source" in str(error)
+ assert "?" in str(error)
+
+ def test_optional_sequence_output_accepts_the_taint(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_step_pipes()
+ sequence = _build_sequence(
+ output_ref="Text?",
+ inputs={"source": "Text?"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ ],
+ )
+ sequence.validate_with_libraries()
+
+ def test_absorbing_sink_terminates_the_taint(self, load_empty_library: Callable[[], str]):
+ """The `?` input on the final step sinks the chain: plain sequence output is fine."""
+ load_empty_library()
+ _register_step_pipes()
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"source": "Text?"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ SubPipeBlueprint(pipe="taint_sink_c", result="final_report"),
+ ],
+ )
+ sequence.validate_with_libraries()
+
+ def test_force_input_terminates_the_taint(self, load_empty_library: Callable[[], str]):
+ """A `!` input asserts presence at run time, so statically the taint stops there."""
+ load_empty_library()
+ _register_step_pipes(source_input_ref="Text!")
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"source": "Text?"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ ],
+ )
+ sequence.validate_with_libraries()
+
+ def test_plain_inputs_produce_no_taint(self, load_empty_library: Callable[[], str]):
+ load_empty_library()
+ _register_step_pipes()
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"source": "Text"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ ],
+ )
+ sequence.validate_with_libraries()
+
+ def test_continue_chain_taints_the_condition_result(self, load_empty_library: Callable[[], str]):
+ """A `continue`-able condition (declared `?` output per OPTIONAL_OUTPUT_REQUIRED) taints
+ its result slot; a plain consumer ending the sequence leaks it.
+ """
+ load_empty_library()
+ _register_step_pipes()
+ gate = PipeFactory[PipeCondition].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_gate",
+ blueprint=PipeConditionBlueprint(
+ description="All-special-outcomes gate: output resolves absent on continue",
+ inputs={"topic": "Text"},
+ output="Text?",
+ expression="topic",
+ outcomes={"skip": "continue"},
+ default_outcome="continue",
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=gate)
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"topic": "Text"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_gate", result="source"),
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ ],
+ )
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ sequence.validate_with_libraries()
+ assert exc_info.value.error_type == PipeValidationErrorType.OPTIONAL_NOT_HANDLED
+
+ def test_batch_compaction_stops_the_taint(self, load_empty_library: Callable[[], str]):
+ """A batched step over a plain list whose inner pipe declares a `?` output compacts to a
+ guaranteed list (D4): no taint escapes, the plain sequence output is fine.
+ """
+ load_empty_library()
+ finder = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_item_finder",
+ blueprint=PipeFuncBlueprint(
+ description="Per-item finder that may find nothing",
+ inputs={"item": "Text"},
+ output="Text?",
+ function_name="optionals_taint_echo_source",
+ ),
+ )
+ list_sink = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_list_sink",
+ blueprint=PipeFuncBlueprint(
+ description="Consumes the compacted list — a guaranteed slot",
+ inputs={"found_items": "Text[]"},
+ output="Text",
+ function_name="optionals_taint_sink",
+ ),
+ )
+ pipe_library = get_pipe_library()
+ pipe_library.add_new_pipe(pipe=finder)
+ pipe_library.add_new_pipe(pipe=list_sink)
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"items": "Text[]"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_item_finder", result="found_items", batch_over="items", batch_as="item"),
+ SubPipeBlueprint(pipe="taint_list_sink", result="summary"),
+ ],
+ )
+ sequence.validate_with_libraries()
+
+ def test_lifted_parallel_branch_slot_is_tainted(self, load_empty_library: Callable[[], str]):
+ """An add_each_output parallel writes branch result slots into the sequence flow; a
+ liftable branch's slot is maybe-absent and a plain consumer ending the sequence leaks it.
+ """
+ load_empty_library()
+ _register_step_pipes()
+ branch_base = PipeFactory[PipeFunc].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_par_base",
+ blueprint=PipeFuncBlueprint(
+ description="Guaranteed branch",
+ inputs={"topic": "Text"},
+ output="Text",
+ function_name="optionals_taint_echo_source",
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=branch_base)
+ parallel = PipeFactory[PipeParallel].make_from_blueprint(
+ domain_code=_DOMAIN_CODE,
+ pipe_code="taint_parallel",
+ blueprint=PipeParallelBlueprint(
+ description="Parallel with a liftable branch, each output added to the flow",
+ inputs={"source": "Text?", "topic": "Text"},
+ output="Composite",
+ branches=[
+ SubPipeBlueprint(pipe="taint_step_a", result="a_out"),
+ SubPipeBlueprint(pipe="taint_par_base", result="base_out"),
+ ],
+ add_each_output=True,
+ ),
+ )
+ get_pipe_library().add_new_pipe(pipe=parallel)
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"source": "Text?", "topic": "Text"},
+ steps=[
+ SubPipeBlueprint(pipe="taint_parallel", result="combined"),
+ # Consumes the liftable branch's slot plain, and ends the sequence.
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ ],
+ )
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ sequence.validate_with_libraries()
+ assert exc_info.value.error_type == PipeValidationErrorType.OPTIONAL_NOT_HANDLED
+
+ def test_rewritten_slot_supersedes_the_taint(self, load_empty_library: Callable[[], str]):
+ """A later step writing a guaranteed value under a tainted name clears the taint
+ (the static mirror of the runtime value-supersedes-record invariant).
+ """
+ load_empty_library()
+ _register_step_pipes()
+ sequence = _build_sequence(
+ output_ref="Text",
+ inputs={"source": "Text?", "a_out": "Text"},
+ steps=[
+ # Step A consumes the optional source and writes b_out (tainted)...
+ SubPipeBlueprint(pipe="taint_step_a", result="b_out"),
+ # ...then step B (consuming the guaranteed a_out) overwrites b_out with a
+ # guaranteed value, so the sequence output is guaranteed.
+ SubPipeBlueprint(pipe="taint_step_b", result="b_out"),
+ ],
+ )
+ sequence.validate_with_libraries()
diff --git a/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py b/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py
index dee8c2082..a4c18d1f3 100644
--- a/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py
+++ b/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py
@@ -76,7 +76,7 @@ class TestBracketNotationInControllers:
blueprint = PipeConditionBlueprint(
description="Route based on category",
inputs={"items": "Category[]"},
- output="Result",
+ output="Result?",
expression="items",
outcomes={"A": "pipe_a"},
default_outcome=SpecialOutcome.CONTINUE,
diff --git a/tests/unit/pipelex/builder/pipe/test_pipe_spec_presence_markers.py b/tests/unit/pipelex/builder/pipe/test_pipe_spec_presence_markers.py
new file mode 100644
index 000000000..6e17a9921
--- /dev/null
+++ b/tests/unit/pipelex/builder/pipe/test_pipe_spec_presence_markers.py
@@ -0,0 +1,55 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.builder.pipe.pipe_spec import PipeSpec
+
+
+def _make_spec(*, inputs: dict[str, str] | None = None, output: str = "Text") -> PipeSpec:
+ return PipeSpec(
+ pipe_code="marker_test_pipe",
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ description="presence marker test pipe",
+ inputs=inputs or {},
+ output=output,
+ )
+
+
+class TestPipeSpecPresenceMarkers:
+ """Builder specs mirror the blueprint grammar rules for `?` and `!` (D5)."""
+
+ @pytest.mark.parametrize("input_spec", ["Text?", "Text!", "domain.Concept?", "domain.Concept!"])
+ def test_singular_input_markers_accepted(self, input_spec: str):
+ spec = _make_spec(inputs={"the_var": input_spec})
+ assert spec.inputs == {"the_var": input_spec}
+
+ @pytest.mark.parametrize("output_spec", ["Text?", "domain.Concept?"])
+ def test_optional_output_accepted(self, output_spec: str):
+ spec = _make_spec(output=output_spec)
+ assert spec.output == output_spec
+
+ def test_to_blueprint_passes_markers_through(self):
+ spec = _make_spec(inputs={"maybe_var": "Text?", "must_var": "Text!"}, output="Text?")
+ blueprint = spec.to_blueprint()
+ assert blueprint.inputs == {"maybe_var": "Text?", "must_var": "Text!"}
+ assert blueprint.output == "Text?"
+
+ @pytest.mark.parametrize("input_spec", ["Text[]?", "Text[3]?", "Text[]!"])
+ def test_marker_on_plural_input_rejected(self, input_spec: str):
+ with pytest.raises(ValidationError, match="cannot be combined with multiplicity"):
+ _make_spec(inputs={"the_var": input_spec})
+
+ @pytest.mark.parametrize("output_spec", ["Text[]?", "Text[3]?"])
+ def test_optional_marker_on_plural_output_rejected(self, output_spec: str):
+ with pytest.raises(ValidationError, match="cannot be combined with multiplicity"):
+ _make_spec(output=output_spec)
+
+ @pytest.mark.parametrize("output_spec", ["Text!", "domain.Concept!"])
+ def test_force_marker_on_output_rejected(self, output_spec: str):
+ with pytest.raises(ValidationError, match="not allowed on outputs"):
+ _make_spec(output=output_spec)
+
+ @pytest.mark.parametrize("input_spec", ["Text??", "Text?[]"])
+ def test_malformed_marker_input_syntax_rejected(self, input_spec: str):
+ with pytest.raises(ValidationError, match="Invalid input syntax"):
+ _make_spec(inputs={"the_var": input_spec})
diff --git a/tests/unit/pipelex/cli/agent_cli/test_run_core_absent_output.py b/tests/unit/pipelex/cli/agent_cli/test_run_core_absent_output.py
new file mode 100644
index 000000000..b6cee7ee3
--- /dev/null
+++ b/tests/unit/pipelex/cli/agent_cli/test_run_core_absent_output.py
@@ -0,0 +1,109 @@
+"""The agent CLI run core on an absent main output (Step E): the run is a success and the JSON
+envelope carries an explicit absence payload (`"absent": true` + the record fields) instead of a
+rendered value — never a crash on the missing stuff.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from types import SimpleNamespace
+from typing import TYPE_CHECKING, Any
+
+from pipelex.cli.agent_cli.commands.run._output_helpers import format_run_markdown # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+from pipelex.cli.agent_cli.commands.run._run_core import run_pipeline_core # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.working_memory import WorkingMemory
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from pytest_mock import MockerFixture
+
+
+class TestAgentRunCoreAbsentOutput:
+ def test_absent_main_output_returns_absence_payload(
+ self,
+ mocker: MockerFixture,
+ tmp_path: Path,
+ monkeypatch: Any,
+ ) -> None:
+ """An absent main output flows through the compact envelope as an explicit absence doc."""
+ monkeypatch.chdir(tmp_path)
+ memory = WorkingMemory()
+ memory.record_new_main_absence(
+ AbsenceRecord(
+ variable_name="summary",
+ kind=AbsenceKind.DECLARED_ABSENT,
+ reason="condition 'gate' evaluated to continue",
+ producing_pipe="gate",
+ )
+ )
+ pipe_output = SimpleNamespace(
+ working_memory=memory,
+ graph_spec=None,
+ tokens_usages=None,
+ )
+ runner_mock = mocker.MagicMock()
+ runner_mock.execute = mocker.AsyncMock(return_value=SimpleNamespace(pipe_output=pipe_output))
+ mocker.patch("pipelex.cli.agent_cli.commands.run._run_core.PipelexMTHDSProtocol", return_value=runner_mock)
+
+ result = asyncio.run(run_pipeline_core("gate", costs=False))
+
+ # Compact mode returns the payload directly — here, the explicit absence doc.
+ assert result["absent"] is True
+ assert result["variable_name"] == "summary"
+ assert result["reason"] == "condition 'gate' evaluated to continue"
+
+ # The on-disk output mirrors the envelope.
+ disk_payload = json.loads((tmp_path / "mthds-wip" / "live_run.json").read_text(encoding="utf-8"))
+ assert disk_payload["absent"] is True
+
+ def test_absent_main_output_with_memory_renders_markdown_absence_doc(
+ self,
+ mocker: MockerFixture,
+ tmp_path: Path,
+ monkeypatch: Any,
+ ) -> None:
+ """With --with-memory, the markdown arm carries the prose absence doc — headings demoted
+ so the doc composes under the run markdown's `## Result` without a second H1.
+ """
+ monkeypatch.chdir(tmp_path)
+ memory = WorkingMemory()
+ memory.record_new_main_absence(
+ AbsenceRecord(
+ variable_name="summary",
+ kind=AbsenceKind.SKIPPED,
+ reason="pipe 'enrich' was lifted: input 'hint' is absent",
+ producing_pipe="enrich",
+ upstream=AbsenceRecord(
+ variable_name="hint",
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason="optional method input 'hint' was omitted by the caller",
+ ),
+ )
+ )
+ pipe_output = SimpleNamespace(
+ working_memory=memory,
+ graph_spec=None,
+ tokens_usages=None,
+ )
+ runner_mock = mocker.MagicMock()
+ runner_mock.execute = mocker.AsyncMock(return_value=SimpleNamespace(pipe_output=pipe_output))
+ mocker.patch("pipelex.cli.agent_cli.commands.run._run_core.PipelexMTHDSProtocol", return_value=runner_mock)
+
+ result = asyncio.run(run_pipeline_core("enrich", with_memory=True, costs=False))
+
+ absence_markdown = result["main_stuff"]["markdown"]
+ assert "## Output absent" in absence_markdown
+ assert "- **Slot:** `summary`" in absence_markdown
+ assert "- **Kind:** skipped" in absence_markdown
+ assert "### Provenance" in absence_markdown
+ assert "`hint` — not_provided" in absence_markdown
+
+ composed = format_run_markdown(result, with_memory=True)
+ assert "## Result" in composed
+ assert "## Output absent" in composed
+ # Well-formed composition: the run title is the document's only H1.
+ h1_lines = [line for line in composed.splitlines() if line.startswith("# ")]
+ assert h1_lines == ["# Pipeline run complete"]
diff --git a/tests/unit/pipelex/cli/test_run_core_execution.py b/tests/unit/pipelex/cli/test_run_core_execution.py
index 17973478d..806ce1a44 100644
--- a/tests/unit/pipelex/cli/test_run_core_execution.py
+++ b/tests/unit/pipelex/cli/test_run_core_execution.py
@@ -83,6 +83,7 @@ class TestRunCoreExecution:
) -> Any:
working_memory = mocker.MagicMock()
working_memory.get_main_stuff.return_value = main_stuff
+ working_memory.resolve_main_stuff.return_value = main_stuff
working_memory.smart_dump.return_value = {"stuff": "dump"}
return SimpleNamespace(
main_stuff=main_stuff,
@@ -90,6 +91,53 @@ class TestRunCoreExecution:
working_memory=working_memory,
)
+ def _make_absent_main_pipe_output(self) -> Any:
+ """A pipe output whose main output resolved absent (real WorkingMemory, recorded absence)."""
+ from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord # noqa: PLC0415
+ from pipelex.core.memory.working_memory import WorkingMemory # noqa: PLC0415
+
+ memory = WorkingMemory()
+ memory.record_new_main_absence(
+ AbsenceRecord(
+ variable_name="summary",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'analysis' is absent",
+ producing_pipe="summarize",
+ )
+ )
+ return SimpleNamespace(graph_spec=None, working_memory=memory)
+
+ @pytest.mark.usefixtures("config_mock")
+ def test_absent_main_output_prints_absence_and_saves_artifact(self, mocker: MockerFixture, console: Console, tmp_path: Path) -> None:
+ """An absent main output is a success: the recap prints the absence (no crash) and
+ --save-main-stuff writes an explicit absence artifact, not value files.
+ """
+ pipe_output = self._make_absent_main_pipe_output()
+ self._mock_runner(mocker, pipe_output)
+
+ _run_async(
+ _call_execute_run(
+ no_pretty_print=False,
+ save_main_stuff=True,
+ output_dir=str(tmp_path),
+ )
+ )
+
+ output = console.export_text()
+ assert "Pipeline execution completed successfully" in output
+ assert "resolved absent" in output
+ assert "skipped because input 'analysis' is absent" in output
+
+ output_dirs = list(tmp_path.glob("test_pipe_output*"))
+ assert len(output_dirs) == 1
+ absence_json = json.loads((output_dirs[0] / "main_stuff.json").read_text(encoding="utf-8"))
+ assert absence_json["absent"] is True
+ assert absence_json["variable_name"] == "summary"
+ assert absence_json["reason"] == "skipped because input 'analysis' is absent"
+ assert (output_dirs[0] / "main_stuff.md").exists()
+ # Nothing to view: the interactive viewer is not produced for an absence.
+ assert not (output_dirs[0] / "main_stuff_viewer.html").exists()
+
@pytest.mark.usefixtures("config_mock")
def test_happy_path_prints_recap(self, mocker: MockerFixture, console: Console) -> None:
"""A successful run prints the completion recap and passes the pipe code to the runner."""
diff --git a/tests/unit/pipelex/core/concepts/test_concept_side_rejects_presence_markers.py b/tests/unit/pipelex/core/concepts/test_concept_side_rejects_presence_markers.py
new file mode 100644
index 000000000..88758f8ac
--- /dev/null
+++ b/tests/unit/pipelex/core/concepts/test_concept_side_rejects_presence_markers.py
@@ -0,0 +1,39 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
+from pipelex.core.concepts.validation import is_concept_ref_or_code_valid
+
+
+class TestConceptSideRejectsPresenceMarkers:
+ """`?` and `!` are slot markers: they are never legal on concept definitions, `refines`,
+ structure-field refs, or package refs (D1). Presence is a property of the flow, not of concepts.
+ """
+
+ @pytest.mark.parametrize("ref", ["Text?", "Text!", "domain.Concept?", "domain.Concept!", "Text[]?", "alias->domain.Concept?"])
+ def test_concept_ref_or_code_with_marker_is_invalid(self, ref: str):
+ assert not is_concept_ref_or_code_valid(ref)
+
+ @pytest.mark.parametrize("refines", ["Text?", "Text!", "domain.Base?"])
+ def test_refines_with_marker_rejected(self, refines: str):
+ with pytest.raises(ValidationError, match="must be a valid concept ref"):
+ ConceptBlueprint(description="a concept", refines=refines)
+
+ @pytest.mark.parametrize("concept_ref", ["Clause?", "Clause!", "legal.Clause?"])
+ def test_structure_field_concept_ref_with_marker_rejected(self, concept_ref: str):
+ with pytest.raises(ValidationError):
+ ConceptStructureBlueprint(
+ description="a field",
+ type=ConceptStructureBlueprintFieldType.CONCEPT,
+ concept_ref=concept_ref,
+ )
+
+ @pytest.mark.parametrize("item_concept_ref", ["Clause?", "Clause!"])
+ def test_structure_field_item_concept_ref_with_marker_rejected(self, item_concept_ref: str):
+ with pytest.raises(ValidationError):
+ ConceptStructureBlueprint(
+ description="a list field",
+ type=ConceptStructureBlueprintFieldType.LIST,
+ item_concept_ref=item_concept_ref,
+ )
diff --git a/tests/unit/pipelex/core/concepts/test_helpers.py b/tests/unit/pipelex/core/concepts/test_helpers.py
index 26113965b..9a10348e1 100644
--- a/tests/unit/pipelex/core/concepts/test_helpers.py
+++ b/tests/unit/pipelex/core/concepts/test_helpers.py
@@ -1,15 +1,15 @@
import pytest
-from pipelex.core.concepts.helpers import strip_multiplicity_from_concept_ref_or_code
+from pipelex.core.concepts.helpers import strip_markers_from_concept_ref_or_code
-class TestStripMultiplicityFromConceptStringOrCode:
- """Test the strip_multiplicity_from_concept_ref_or_code function."""
+class TestStripMarkersFromConceptStringOrCode:
+ """Test the strip_markers_from_concept_ref_or_code function."""
@pytest.mark.parametrize(
("input_string", "expected_output"),
[
- # No multiplicity (PascalCase concepts)
+ # No markers (PascalCase concepts)
("ConceptCode", "ConceptCode"),
("MyConceptCode", "MyConceptCode"),
("ComplexConceptName", "ComplexConceptName"),
@@ -43,17 +43,26 @@ class TestStripMultiplicityFromConceptStringOrCode:
("some_domain.MyConceptCode[42]", "some_domain.MyConceptCode"),
("another_domain.SomeOtherConcept[]", "another_domain.SomeOtherConcept"),
("complex_domain_name.ComplexConceptName[123]", "complex_domain_name.ComplexConceptName"),
+ # Presence markers
+ ("ConceptCode?", "ConceptCode"),
+ ("ConceptCode!", "ConceptCode"),
+ ("my_domain.ConceptCode?", "my_domain.ConceptCode"),
+ ("my_domain.ConceptCode!", "my_domain.ConceptCode"),
+ # Multiplicity and presence combined (multiplicity then presence)
+ ("ConceptCode[]?", "ConceptCode"),
+ ("ConceptCode[3]!", "ConceptCode"),
+ ("my_domain.ConceptCode[]?", "my_domain.ConceptCode"),
],
)
- def test_strip_multiplicity(self, input_string: str, expected_output: str):
- """Test stripping multiplicity from concept strings with various formats.
+ def test_strip_markers(self, input_string: str, expected_output: str):
+ """Test stripping multiplicity and presence markers from concept strings.
This tests:
- - Concepts without multiplicity (should return unchanged)
+ - Concepts without markers (should return unchanged)
- Concepts with empty multiplicity []
- - Concepts with single-digit multiplicity [1-9]
- - Concepts with multi-digit multiplicity [10-999]
- - Concepts with large multiplicity [1000+]
+ - Concepts with numeric multiplicity [N]
+ - Concepts with presence markers ? and !
+ - Concepts with both multiplicity and presence markers
"""
- result = strip_multiplicity_from_concept_ref_or_code(input_string)
+ result = strip_markers_from_concept_ref_or_code(input_string)
assert result == expected_output
diff --git a/tests/unit/pipelex/core/memory/test_absence_ledger.py b/tests/unit/pipelex/core/memory/test_absence_ledger.py
new file mode 100644
index 000000000..452805604
--- /dev/null
+++ b/tests/unit/pipelex/core/memory/test_absence_ledger.py
@@ -0,0 +1,201 @@
+import pytest
+
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+
+
+def _make_text_stuff(name: str, *, text: str = "hello") -> Stuff:
+ return StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ content=TextContent(text=text),
+ name=name,
+ )
+
+
+def _make_record(variable_name: str, *, upstream: AbsenceRecord | None = None) -> AbsenceRecord:
+ return AbsenceRecord(
+ variable_name=variable_name,
+ kind=AbsenceKind.DECLARED_ABSENT,
+ reason=f"no value found for {variable_name}",
+ producing_pipe="some_producer",
+ upstream=upstream,
+ )
+
+
+class TestAbsenceLedger:
+ def test_record_and_lookup_absence(self):
+ """A recorded absence is retrievable by variable name; the slot holds no stuff."""
+ working_memory = WorkingMemory()
+ record = _make_record("penalty_clause")
+ working_memory.record_absence(record)
+
+ assert working_memory.get_optional_stuff("penalty_clause") is None
+ found = working_memory.get_optional_absence("penalty_clause")
+ assert found is not None
+ assert found.variable_name == "penalty_clause"
+ assert found.kind == AbsenceKind.DECLARED_ABSENT
+ assert found.reason == "no value found for penalty_clause"
+ assert found.producing_pipe == "some_producer"
+
+ def test_resolve_stuff_returns_absence_record_when_absent(self):
+ """The tri-state resolved accessor returns the AbsenceRecord for a recorded-absent slot."""
+ working_memory = WorkingMemory()
+ record = _make_record("penalty_clause")
+ working_memory.record_absence(record)
+
+ resolved = working_memory.resolve_stuff("penalty_clause")
+ assert isinstance(resolved, AbsenceRecord)
+ assert resolved.variable_name == "penalty_clause"
+
+ def test_resolve_stuff_prefers_present_value(self):
+ """When a slot has both a stuff and a stale ledger note, the value wins."""
+ working_memory = WorkingMemory()
+ stuff = _make_text_stuff("items")
+ working_memory.set_stuff(name="items", stuff=stuff)
+ # A ledger note can coexist with a present value (D4 plural normalization).
+ working_memory.absences["items"] = _make_record("items")
+
+ resolved = working_memory.resolve_stuff("items")
+ assert isinstance(resolved, Stuff)
+ assert resolved.stuff_name == "items"
+
+ def test_resolve_stuff_raises_when_neither_value_nor_record(self):
+ """A slot with no value and no record is still a hard miss — that is the bug case."""
+ working_memory = WorkingMemory()
+ with pytest.raises(WorkingMemoryStuffNotFoundError):
+ working_memory.resolve_stuff("never_produced")
+
+ def test_set_stuff_supersedes_absence_record(self):
+ """Writing a value under a name clears that name's absence record."""
+ working_memory = WorkingMemory()
+ working_memory.record_absence(_make_record("assessment"))
+ working_memory.set_stuff(name="assessment", stuff=_make_text_stuff("assessment"))
+
+ assert working_memory.get_optional_absence("assessment") is None
+ assert isinstance(working_memory.resolve_stuff("assessment"), Stuff)
+
+ def test_record_new_main_absence_mirrors_to_main_stuff(self):
+ """Recording a pipe's output absence marks both the slot and the main-stuff position,
+ and removes any stale main stuff so the previous value cannot masquerade as the output.
+ """
+ working_memory = WorkingMemory()
+ previous = _make_text_stuff("previous_result")
+ working_memory.set_new_main_stuff(previous, name="previous_result")
+
+ record = _make_record("assessment")
+ working_memory.record_new_main_absence(record)
+
+ assert working_memory.get_optional_main_stuff() is None
+ assert working_memory.get_optional_absence("assessment") == record
+ main_resolved = working_memory.resolve_main_stuff()
+ assert isinstance(main_resolved, AbsenceRecord)
+ assert main_resolved.variable_name == "assessment"
+ # The previous value stays in memory under its own name.
+ assert working_memory.get_optional_stuff("previous_result") is not None
+
+ def test_set_new_main_stuff_clears_stale_main_absence(self):
+ """A later pipe setting a real main stuff supersedes the positional main-stuff record."""
+ working_memory = WorkingMemory()
+ working_memory.record_new_main_absence(_make_record("assessment"))
+ working_memory.set_new_main_stuff(_make_text_stuff("report"), name="report")
+
+ main_resolved = working_memory.resolve_main_stuff()
+ assert isinstance(main_resolved, Stuff)
+ # The named-slot record stays: 'assessment' is still genuinely absent.
+ assert working_memory.get_optional_absence("assessment") is not None
+ assert working_memory.get_optional_absence(MAIN_STUFF_NAME) is None
+
+ def test_record_resolved_absence_supersedes_stale_value_under_same_name(self):
+ """A resolution absence clears a stale value (and alias) under the same name — the mirror
+ of set_stuff's value-supersedes-record — so old data cannot outrank the fresh absence.
+ """
+ working_memory = WorkingMemory()
+ working_memory.set_stuff(name="draft", stuff=_make_text_stuff("draft", text="old draft"))
+ record = _make_record("draft")
+ working_memory.record_resolved_absence(record)
+
+ assert working_memory.get_optional_stuff("draft") is None
+ assert working_memory.resolve_stuff("draft") == record
+
+ def test_record_new_main_absence_supersedes_stale_value_under_output_name(self):
+ """When a pipe's output name collides with an earlier value, recording the output absence
+ supersedes the stale value under that name too, not just the main-stuff position.
+ """
+ working_memory = WorkingMemory()
+ working_memory.set_new_main_stuff(_make_text_stuff("draft", text="old draft"), name="draft")
+ record = _make_record("draft")
+ working_memory.record_new_main_absence(record)
+
+ assert working_memory.get_optional_stuff("draft") is None
+ assert working_memory.resolve_stuff("draft") == record
+ assert isinstance(working_memory.resolve_main_stuff(), AbsenceRecord)
+
+ def test_get_optional_absence_follows_aliases(self):
+ """An alias to a resolved-absent slot surfaces the record — mirroring get_optional_stuff —
+ instead of degrading the alias name to a neither-value-nor-record hard miss.
+ """
+ working_memory = WorkingMemory()
+ working_memory.set_stuff(name="draft", stuff=_make_text_stuff("draft"))
+ working_memory.add_alias("draft_alias", target="draft")
+ record = _make_record("draft")
+ working_memory.record_resolved_absence(record)
+
+ assert working_memory.get_optional_absence("draft_alias") == record
+ assert working_memory.resolve_stuff("draft_alias") == record
+ assert working_memory.list_missing_names({"draft_alias"}) == []
+
+ def test_record_absence_is_a_note_beside_a_value(self):
+ """The plain record_absence stays a ledger NOTE: a value under the same name is kept and
+ wins in resolve_stuff (the D4 plural empty-list note depends on this).
+ """
+ working_memory = WorkingMemory()
+ working_memory.set_stuff(name="items", stuff=_make_text_stuff("items"))
+ working_memory.record_absence(_make_record("items"))
+
+ assert working_memory.get_optional_stuff("items") is not None
+ assert isinstance(working_memory.resolve_stuff("items"), Stuff)
+
+ def test_provenance_chain_walks_to_origin(self):
+ """provenance_chain() lists the record then its upstream chain; origin is the first absence."""
+ origin = _make_record("penalty_clause")
+ middle = AbsenceRecord(
+ variable_name="assessment",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'penalty_clause' is absent",
+ producing_pipe="assess_penalty",
+ upstream=origin,
+ )
+ tip = AbsenceRecord(
+ variable_name="summary",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'assessment' is absent",
+ producing_pipe="summarize",
+ upstream=middle,
+ )
+
+ chain = tip.provenance_chain()
+ assert chain == [tip, middle, origin]
+ assert tip.origin() == origin
+ assert origin.origin() == origin
+
+ def test_deep_copy_preserves_absences(self):
+ working_memory = WorkingMemory()
+ working_memory.record_absence(_make_record("penalty_clause"))
+ copy = working_memory.make_deep_copy()
+ copied_record = copy.get_optional_absence("penalty_clause")
+ assert copied_record is not None
+ assert copied_record.reason == "no value found for penalty_clause"
+
+ def test_model_dump_includes_absences(self):
+ working_memory = WorkingMemory()
+ working_memory.record_absence(_make_record("penalty_clause"))
+ dumped = working_memory.model_dump()
+ assert "absences" in dumped
+ assert dumped["absences"]["penalty_clause"]["reason"] == "no value found for penalty_clause"
+ assert dumped["absences"]["penalty_clause"]["kind"] == "declared_absent"
diff --git a/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py b/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py
index 4f79230c2..b17a3a108 100644
--- a/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py
+++ b/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py
@@ -2,7 +2,7 @@
import pytest
-from pipelex.core.pipes.variable_multiplicity import format_concept_with_multiplicity
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker, format_concept_with_multiplicity, parse_concept_with_multiplicity
class TestFormatConceptWithMultiplicity:
@@ -41,3 +41,39 @@ class TestFormatConceptWithMultiplicity:
def test_format_concept_with_multiplicity(self, concept_code: str, multiplicity: int | bool | None, expected: str):
"""Test formatting concept codes with all multiplicity variations."""
assert format_concept_with_multiplicity(concept_code, multiplicity=multiplicity) == expected
+
+ @pytest.mark.parametrize(
+ ("concept_code", "multiplicity", "presence", "expected"),
+ [
+ ("Text", None, PresenceMarker.PLAIN, "Text"),
+ ("Text", None, PresenceMarker.OPTIONAL, "Text?"),
+ ("Text", None, PresenceMarker.FORCE, "Text!"),
+ ("domain.Concept", None, PresenceMarker.OPTIONAL, "domain.Concept?"),
+ ("domain.Concept", None, PresenceMarker.FORCE, "domain.Concept!"),
+ # Fixed order: multiplicity then presence
+ ("Text", True, PresenceMarker.OPTIONAL, "Text[]?"),
+ ("Text", 3, PresenceMarker.FORCE, "Text[3]!"),
+ ],
+ )
+ def test_format_concept_with_presence(
+ self,
+ concept_code: str,
+ multiplicity: int | bool | None,
+ presence: PresenceMarker,
+ expected: str,
+ ):
+ """Formatting renders the presence marker after the multiplicity suffix."""
+ assert format_concept_with_multiplicity(concept_code, multiplicity=multiplicity, presence=presence) == expected
+
+ @pytest.mark.parametrize("spec", ["Text", "Text[]", "Text[3]", "Text?", "Text!", "domain.Concept?"])
+ def test_parse_format_round_trip(self, spec: str):
+ """Parse then format reproduces the original spec string."""
+ parsed = parse_concept_with_multiplicity(spec)
+ assert (
+ format_concept_with_multiplicity(
+ parsed.concept_ref_or_code,
+ multiplicity=parsed.multiplicity,
+ presence=parsed.presence,
+ )
+ == spec
+ )
diff --git a/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py b/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py
index bddea91bc..f94c57f21 100644
--- a/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py
+++ b/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py
@@ -5,6 +5,7 @@ import pytest
from pipelex.core.concepts.exceptions import ConceptStringError
from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactory, InputStuffSpecsFactoryError
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker
from pipelex.libraries.concept.exceptions import ConceptLibraryError
from tests.unit.pipelex.core.pipes.data import (
CONCEPT_CODE_RESOLUTION_TEST_CASES,
@@ -103,6 +104,28 @@ class TestMakeInputRequirementsFromString:
assert result.concept.concept_ref == "native.Text"
assert result.multiplicity == 3
+ @pytest.mark.parametrize(
+ ("stuff_spec_str", "expected_presence", "expected_multiplicity"),
+ [
+ ("native.Text", PresenceMarker.PLAIN, None),
+ ("native.Text?", PresenceMarker.OPTIONAL, None),
+ ("native.Text!", PresenceMarker.FORCE, None),
+ ],
+ )
+ def test_presence_markers(
+ self,
+ stuff_spec_str: str,
+ expected_presence: PresenceMarker,
+ expected_multiplicity: int | bool | None,
+ load_empty_library: Callable[[], None],
+ ):
+ """Test parsing presence markers on input stuff spec strings."""
+ load_empty_library()
+ result = InputStuffSpecsFactory.make_from_string(domain_code="native", stuff_spec_str=stuff_spec_str)
+ assert result.concept.concept_ref == "native.Text"
+ assert result.multiplicity == expected_multiplicity
+ assert result.presence == expected_presence
+
def test_concept_not_found_raises_error(self, load_empty_library: Callable[[], None]):
load_empty_library()
"""Test that an invalid concept code raises ConceptLibraryConceptNotFoundError."""
diff --git a/tests/unit/pipelex/core/pipes/test_input_stuff_specs_presence.py b/tests/unit/pipelex/core/pipes/test_input_stuff_specs_presence.py
new file mode 100644
index 000000000..93d442d71
--- /dev/null
+++ b/tests/unit/pipelex/core/pipes/test_input_stuff_specs_presence.py
@@ -0,0 +1,57 @@
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
+from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker
+
+
+def _text_concept() -> Concept:
+ return ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)
+
+
+class TestInputStuffSpecsPresence:
+ """The required vs declared split: declared_names lists every input, required_names excludes optional ones."""
+
+ def _make_specs(self) -> InputStuffSpecs:
+ concept = _text_concept()
+ return InputStuffSpecs(
+ root={
+ "plain_var": StuffSpec(concept=concept),
+ "optional_var": StuffSpec(concept=concept, presence=PresenceMarker.OPTIONAL),
+ "forced_var": StuffSpec(concept=concept, presence=PresenceMarker.FORCE),
+ },
+ )
+
+ def test_declared_names_lists_all_inputs(self):
+ specs = self._make_specs()
+ assert specs.declared_names == ["plain_var", "optional_var", "forced_var"]
+
+ def test_required_names_excludes_optional_inputs(self):
+ specs = self._make_specs()
+ assert specs.required_names == ["plain_var", "forced_var"]
+
+ def test_required_equals_declared_without_markers(self):
+ concept = _text_concept()
+ specs = InputStuffSpecs(root={"var_a": StuffSpec(concept=concept), "var_b": StuffSpec(concept=concept, multiplicity=True)})
+ assert specs.required_names == specs.declared_names == ["var_a", "var_b"]
+
+ def test_named_stuff_specs_carry_presence(self):
+ specs = self._make_specs()
+ presence_by_name = {named.variable_name: named.presence for named in specs.named_stuff_specs}
+ assert presence_by_name == {
+ "plain_var": PresenceMarker.PLAIN,
+ "optional_var": PresenceMarker.OPTIONAL,
+ "forced_var": PresenceMarker.FORCE,
+ }
+
+ def test_add_stuff_spec_carries_presence(self):
+ specs = InputStuffSpecs(root={})
+ specs.add_stuff_spec("maybe_var", concept=_text_concept(), presence=PresenceMarker.OPTIONAL)
+ assert specs.root["maybe_var"].presence == PresenceMarker.OPTIONAL
+
+ def test_root_validator_preserves_presence(self):
+ """The wrap validator rebuilds StuffSpecs (dotted-path rooting) and must not drop presence."""
+ concept = _text_concept()
+ specs = InputStuffSpecs(root={"doc.summary": StuffSpec(concept=concept, presence=PresenceMarker.OPTIONAL)})
+ assert specs.root["doc"].presence == PresenceMarker.OPTIONAL
diff --git a/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.py b/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.py
index 21878ea3d..d4d3a56fb 100644
--- a/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.py
+++ b/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.py
@@ -1,6 +1,6 @@
import pytest
-from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker, parse_concept_with_multiplicity
class TestParseConceptWithMultiplicity:
@@ -116,3 +116,62 @@ class TestParseConceptWithMultiplicity:
result = parse_concept_with_multiplicity("a.b.c.d.Entity[]")
assert result.concept_ref_or_code == "a.b.c.d.Entity"
assert result.multiplicity is True
+
+ # ========== Presence marker tests ==========
+
+ def test_no_marker_means_plain_presence(self):
+ """A spec without a presence marker parses as plain presence."""
+ for spec in ["Text", "Text[]", "Text[5]", "domain.Concept"]:
+ result = parse_concept_with_multiplicity(spec)
+ assert result.presence == PresenceMarker.PLAIN
+
+ @pytest.mark.parametrize(
+ ("spec", "expected_concept", "expected_multiplicity", "expected_presence"),
+ [
+ ("Text?", "Text", None, PresenceMarker.OPTIONAL),
+ ("Text!", "Text", None, PresenceMarker.FORCE),
+ ("domain.Concept?", "domain.Concept", None, PresenceMarker.OPTIONAL),
+ ("domain.Concept!", "domain.Concept", None, PresenceMarker.FORCE),
+ ("legal.contracts.PenaltyClause?", "legal.contracts.PenaltyClause", None, PresenceMarker.OPTIONAL),
+ ("_PrivateConcept?", "_PrivateConcept", None, PresenceMarker.OPTIONAL),
+ # The parser accepts marker + multiplicity combinations (multiplicity then presence,
+ # e.g. "X[]?"); the D1 mutual-exclusion rule is enforced at the blueprint/spec layer
+ # where the context (input vs output) is known.
+ ("Text[]?", "Text", True, PresenceMarker.OPTIONAL),
+ ("Text[3]?", "Text", 3, PresenceMarker.OPTIONAL),
+ ("Text[]!", "Text", True, PresenceMarker.FORCE),
+ ],
+ )
+ def test_presence_marker_parsing(
+ self,
+ spec: str,
+ expected_concept: str,
+ expected_multiplicity: int | bool | None,
+ expected_presence: PresenceMarker,
+ ):
+ """Presence markers parse alongside multiplicity."""
+ result = parse_concept_with_multiplicity(spec)
+ assert result.concept_ref_or_code == expected_concept
+ assert result.multiplicity == expected_multiplicity
+ assert result.presence == expected_presence
+
+ @pytest.mark.parametrize(
+ "spec",
+ [
+ "Text??",
+ "Text?!",
+ "Text!?",
+ "Text!!",
+ # Fixed order: multiplicity then presence — marker before brackets is a syntax error
+ "Text?[]",
+ "Text![3]",
+ "?Text",
+ "!Text",
+ "?",
+ "!",
+ ],
+ )
+ def test_invalid_presence_marker_syntax(self, spec: str):
+ """Doubled markers, marker-before-brackets, and bare markers are syntax errors."""
+ with pytest.raises(ValueError, match="Invalid concept specification syntax"):
+ parse_concept_with_multiplicity(spec)
diff --git a/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py b/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py
index f996aa56e..7d7515b61 100644
--- a/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py
+++ b/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py
@@ -92,7 +92,9 @@ class TestPipeBlueprintValidation:
type="PipeLLM",
pipe_category="PipeOperator",
description="lorem ipsum",
- inputs={"text": "InvalidFormat!"},
+ # NOTE: a trailing "!" is now valid grammar (force marker); use a genuinely
+ # malformed spec to exercise the syntax rejection
+ inputs={"text": "Invalid@Format"},
output="Text",
)
assert "Invalid input syntax" in str(exc_info.value)
@@ -113,7 +115,9 @@ class TestPipeBlueprintValidation:
pipe_category="PipeOperator",
description="lorem ipsum",
inputs={"text": "Text"},
- output="InvalidConcept!",
+ # NOTE: a trailing "!" is now valid grammar but rejected on outputs (force marker
+ # is a use-site assertion); a genuinely malformed spec still hits the syntax error
+ output="Invalid@Concept",
)
assert "Invalid concept specification syntax" in str(exc_info.value)
diff --git a/tests/unit/pipelex/core/pipes/test_pipe_blueprint_presence_markers.py b/tests/unit/pipelex/core/pipes/test_pipe_blueprint_presence_markers.py
new file mode 100644
index 000000000..cec4bafa5
--- /dev/null
+++ b/tests/unit/pipelex/core/pipes/test_pipe_blueprint_presence_markers.py
@@ -0,0 +1,83 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.handle_pipe_errors import extract_wrapped_pipe_validation_error
+from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
+
+
+class ConcretePipeBlueprint(PipeBlueprint):
+ pass
+
+
+def _make_blueprint(*, inputs: dict[str, str] | None = None, output: str = "Text") -> ConcretePipeBlueprint:
+ return ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ description="presence marker test pipe",
+ inputs=inputs,
+ output=output,
+ )
+
+
+def _extract_marker_error(exc: ValidationError) -> PipeValidationError:
+ """The blueprint must raise a typed PipeValidationError that survives pydantic wrapping."""
+ for error_details in exc.errors():
+ wrapped = extract_wrapped_pipe_validation_error(error_details)
+ if wrapped is not None:
+ return wrapped
+ msg = f"No wrapped PipeValidationError found in: {exc}"
+ raise AssertionError(msg)
+
+
+class TestPipeBlueprintPresenceMarkers:
+ """Blueprint-parse-time grammar rules for `?` and `!` (D1, D4)."""
+
+ # ---- accepted forms ----
+
+ @pytest.mark.parametrize("input_spec", ["Text?", "Text!", "domain.Concept?", "domain.Concept!"])
+ def test_singular_input_markers_accepted(self, input_spec: str):
+ blueprint = _make_blueprint(inputs={"the_var": input_spec})
+ assert blueprint.inputs == {"the_var": input_spec}
+
+ @pytest.mark.parametrize("output_spec", ["Text?", "domain.Concept?"])
+ def test_optional_output_accepted(self, output_spec: str):
+ blueprint = _make_blueprint(output=output_spec)
+ assert blueprint.output == output_spec
+
+ # ---- rejected forms: typed OPTIONAL_MARKER_INVALID ----
+
+ @pytest.mark.parametrize("input_spec", ["Text[]?", "Text[3]?", "Text[]!", "Text[3]!"])
+ def test_marker_on_plural_input_rejected(self, input_spec: str):
+ with pytest.raises(ValidationError) as exc_info:
+ _make_blueprint(inputs={"the_var": input_spec})
+ wrapped = _extract_marker_error(exc_info.value)
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_MARKER_INVALID
+ assert wrapped.variable_names == ["the_var"]
+
+ @pytest.mark.parametrize("output_spec", ["Text[]?", "Text[3]?"])
+ def test_optional_marker_on_plural_output_rejected(self, output_spec: str):
+ with pytest.raises(ValidationError) as exc_info:
+ _make_blueprint(output=output_spec)
+ wrapped = _extract_marker_error(exc_info.value)
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_MARKER_INVALID
+
+ @pytest.mark.parametrize("output_spec", ["Text!", "domain.Concept!", "Text[]!"])
+ def test_force_marker_on_output_rejected(self, output_spec: str):
+ """`!` is a use-site assertion: it is meaningless on outputs."""
+ with pytest.raises(ValidationError) as exc_info:
+ _make_blueprint(output=output_spec)
+ wrapped = _extract_marker_error(exc_info.value)
+ assert wrapped.error_type == PipeValidationErrorType.OPTIONAL_MARKER_INVALID
+
+ # ---- rejected forms: plain syntax errors ----
+
+ @pytest.mark.parametrize("input_spec", ["Text??", "Text?[]", "?Text"])
+ def test_malformed_marker_input_syntax_rejected(self, input_spec: str):
+ with pytest.raises(ValidationError, match="Invalid input syntax"):
+ _make_blueprint(inputs={"the_var": input_spec})
+
+ @pytest.mark.parametrize("output_spec", ["Text??", "Text?[]", "!Text"])
+ def test_malformed_marker_output_syntax_rejected(self, output_spec: str):
+ with pytest.raises(ValidationError, match="Invalid concept specification syntax"):
+ _make_blueprint(output=output_spec)
diff --git a/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py b/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py
index 897601bc2..fb22ac3bb 100644
--- a/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py
+++ b/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py
@@ -7,6 +7,7 @@ from pytest_mock import MockerFixture
from pipelex.core.concepts.concept import Concept
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_abstract import InputPresenceScan
from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.text_content import TextContent
@@ -50,8 +51,10 @@ class TestRunPipeForwardsTracerMetadata:
)
expected_output = mocker.MagicMock()
expected_output.working_memory.get_main_stuff.return_value = main_stuff
+ expected_output.working_memory.resolve_main_stuff.return_value = main_stuff
mocker.patch.object(PipeLLM, "_live_run_operator_pipe", mocker.AsyncMock(return_value=expected_output))
- mocker.patch.object(PipeLLM, "validate_before_run", mocker.AsyncMock(return_value=None))
+ empty_presence_scan = InputPresenceScan(missing_names=[], forced_absent=[], liftable=[])
+ mocker.patch.object(PipeLLM, "validate_before_run", mocker.AsyncMock(return_value=empty_presence_scan))
mocker.patch.object(PipeLLM, "validate_after_run", mocker.AsyncMock(return_value=None))
on_pipe_start_mock = mocker.MagicMock(return_value=(None, None))
diff --git a/tests/unit/pipelex/core/pipes/test_stuff_spec.py b/tests/unit/pipelex/core/pipes/test_stuff_spec.py
index 979432fc7..ab33e1a59 100644
--- a/tests/unit/pipelex/core/pipes/test_stuff_spec.py
+++ b/tests/unit/pipelex/core/pipes/test_stuff_spec.py
@@ -3,7 +3,7 @@ import pytest
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
-from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
+from pipelex.core.pipes.variable_multiplicity import PresenceMarker, VariableMultiplicity
class TestStuffSpecToBundleRepresentation:
@@ -83,3 +83,23 @@ class TestStuffSpecToBundleRepresentation:
assert stuff_spec_single.to_bundle_representation() == "my_domain.MyCustomConcept"
assert stuff_spec_list.to_bundle_representation() == "my_domain.MyCustomConcept[]"
assert stuff_spec_fixed.to_bundle_representation() == "my_domain.MyCustomConcept[7]"
+
+ def test_presence_defaults_to_plain(self):
+ """A StuffSpec built without presence is plain (required, always produced)."""
+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)
+ stuff_spec = StuffSpec(concept=concept)
+ assert stuff_spec.presence == PresenceMarker.PLAIN
+
+ @pytest.mark.parametrize(
+ ("presence", "expected"),
+ [
+ (PresenceMarker.PLAIN, "native.Text"),
+ (PresenceMarker.OPTIONAL, "native.Text?"),
+ (PresenceMarker.FORCE, "native.Text!"),
+ ],
+ )
+ def test_to_bundle_representation_with_presence(self, presence: PresenceMarker, expected: str):
+ """to_bundle_representation renders the presence marker suffix."""
+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)
+ stuff_spec = StuffSpec(concept=concept, presence=presence)
+ assert stuff_spec.to_bundle_representation() == expected
diff --git a/tests/unit/pipelex/core/stuffs/test_stuff_artefact_iteration.py b/tests/unit/pipelex/core/stuffs/test_stuff_artefact_iteration.py
index b352d059a..c1eec6309 100644
--- a/tests/unit/pipelex/core/stuffs/test_stuff_artefact_iteration.py
+++ b/tests/unit/pipelex/core/stuffs/test_stuff_artefact_iteration.py
@@ -171,3 +171,16 @@ class TestStuffArtefactIteration:
assert artefact["_stuff_name"] == "test_list"
assert artefact["_stuff_code"] == "test_list"
assert artefact["_content_class"] == "ListContent"
+
+ def test_truthiness_of_present_singular_value(self) -> None:
+ """A present non-list artefact is truthy — Jinja2's `{% if var %}` guard (the optionals
+ guard idiom) must never crash on a present value via the __len__ fallback.
+ """
+ artefact = StuffArtefact(_make_text_stuff("Plain text"))
+
+ assert bool(artefact) is True
+
+ def test_truthiness_of_list_follows_emptiness(self) -> None:
+ """A ListContent artefact follows list emptiness (D4: [] is the absence story for plurals)."""
+ assert bool(StuffArtefact(_make_list_stuff([TextContent(text="Item")]))) is True
+ assert bool(StuffArtefact(_make_list_stuff([]))) is False
diff --git a/tests/unit/pipelex/graph/test_graph_tracer.py b/tests/unit/pipelex/graph/test_graph_tracer.py
index c81857c48..db5a6e8a3 100644
--- a/tests/unit/pipelex/graph/test_graph_tracer.py
+++ b/tests/unit/pipelex/graph/test_graph_tracer.py
@@ -178,6 +178,117 @@ class TestGraphTracer:
assert node.error.message == "Something went wrong"
assert node.error.stack == "Traceback..."
+ def test_skipped_tracking(self) -> None:
+ """A lifted (skipped) pipe ends its node in the SKIPPED state with the skip reason."""
+ tracer = GraphTracer()
+ context = tracer.setup(graph_id="skip-test", data_inclusion=make_defaulted_data_inclusion_config())
+
+ started_at = datetime.now(UTC)
+ node_id, _ = tracer.on_pipe_start(
+ trace_context=context,
+ pipe_code="lifted_pipe",
+ pipe_type="PipeFunc",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ tracer.on_pipe_end_skipped(
+ node_id=node_id,
+ ended_at=started_at + timedelta(milliseconds=5),
+ skip_reason="skipped because input 'source' is absent",
+ )
+
+ graph_spec = tracer.teardown()
+
+ assert graph_spec is not None
+ node = graph_spec.nodes[0]
+ assert node.status == NodeStatus.SKIPPED
+ assert node.skip_reason == "skipped because input 'source' is absent"
+ assert node.error is None
+ assert node.timing is not None
+
+ def test_skipped_plural_output_still_produces_data_edge(self) -> None:
+ """A lifted pipe with a PLURAL output wrote a real empty-list Stuff (D4) — its output spec
+ rides the skip and registers in the producer map, so the downstream DATA edge resolves.
+ """
+ tracer = GraphTracer()
+ context = tracer.setup(graph_id="skip-plural-edge-test", data_inclusion=make_defaulted_data_inclusion_config())
+
+ started_at = datetime.now(UTC)
+ producer_id, _ = tracer.on_pipe_start(
+ trace_context=context,
+ pipe_code="lifted_plural_pipe",
+ pipe_type="PipeFunc",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+ tracer.on_pipe_end_skipped(
+ node_id=producer_id,
+ ended_at=started_at + timedelta(milliseconds=5),
+ skip_reason="skipped because input 'source' is absent",
+ output_spec=IOSpec(name="items", concept="Text", digest="stuff-empty-list"),
+ )
+
+ consumer_id, _ = tracer.on_pipe_start(
+ trace_context=context,
+ pipe_code="items_consumer",
+ pipe_type="PipeFunc",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(milliseconds=10),
+ input_specs=[IOSpec(name="items", digest="stuff-empty-list")],
+ )
+ tracer.on_pipe_end_success(node_id=consumer_id, ended_at=started_at + timedelta(milliseconds=20))
+
+ graph_spec = tracer.teardown()
+
+ assert graph_spec is not None
+ skipped_node = next(node for node in graph_spec.nodes if node.node_id == producer_id)
+ assert skipped_node.status == NodeStatus.SKIPPED
+ assert [output.digest for output in skipped_node.node_io.outputs] == ["stuff-empty-list"]
+ data_edges = [edge for edge in graph_spec.edges if edge.kind.is_data]
+ assert len(data_edges) == 1
+ assert data_edges[0].source == producer_id
+ assert data_edges[0].target == consumer_id
+ assert data_edges[0].optional is False
+
+ def test_data_edge_from_optional_output_carries_marker(self) -> None:
+ """A DATA edge whose producer output is declared optional (`?`) reports optional=True."""
+ tracer = GraphTracer()
+ context = tracer.setup(graph_id="optional-edge-test", data_inclusion=make_defaulted_data_inclusion_config())
+
+ started_at = datetime.now(UTC)
+ producer_id, _ = tracer.on_pipe_start(
+ trace_context=context,
+ pipe_code="maybe_producer",
+ pipe_type="PipeCondition",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+ tracer.on_pipe_end_success(
+ node_id=producer_id,
+ ended_at=started_at + timedelta(milliseconds=10),
+ output_spec=IOSpec(name="verdict", digest="stuff-verdict", extra={"optional": True}),
+ )
+
+ consumer_id, _ = tracer.on_pipe_start(
+ trace_context=context,
+ pipe_code="verdict_consumer",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(milliseconds=20),
+ input_specs=[IOSpec(name="verdict", digest="stuff-verdict")],
+ )
+ tracer.on_pipe_end_success(node_id=consumer_id, ended_at=started_at + timedelta(milliseconds=30))
+
+ graph_spec = tracer.teardown()
+
+ assert graph_spec is not None
+ data_edges = [edge for edge in graph_spec.edges if edge.kind.is_data]
+ assert len(data_edges) == 1
+ assert data_edges[0].source == producer_id
+ assert data_edges[0].target == consumer_id
+ assert data_edges[0].optional is True
+
def test_running_nodes_marked_canceled_on_teardown(self) -> None:
"""Test that running nodes are marked as canceled on teardown."""
tracer = GraphTracer()
diff --git a/tests/unit/pipelex/libraries/test_contract_match.py b/tests/unit/pipelex/libraries/test_contract_match.py
new file mode 100644
index 000000000..6f2c379c5
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_contract_match.py
@@ -0,0 +1,52 @@
+import pytest
+
+from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
+from pipelex.libraries.contract_match import contracts_match
+
+
+class ConcretePipeBlueprint(PipeBlueprint):
+ pass
+
+
+def _make_blueprint(*, inputs: dict[str, str] | None = None, output: str = "Text") -> ConcretePipeBlueprint:
+ return ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ description="contract match test pipe",
+ inputs=inputs,
+ output=output,
+ )
+
+
+class TestContractsMatchPresenceMarkers:
+ """Contract canonicalization compares presence markers (D5) on top of concept identity + multiplicity."""
+
+ def test_equivalent_spellings_with_same_marker_match(self):
+ existing = _make_blueprint(inputs={"brief": "Brief?"}, output="native.Text?")
+ incoming = _make_blueprint(inputs={"brief": "thisdomain.Brief?"}, output="Text?")
+ assert contracts_match(existing=existing, incoming=incoming, domain_code="thisdomain")
+
+ @pytest.mark.parametrize(
+ ("existing_spec", "incoming_spec"),
+ [
+ ("Brief", "Brief?"),
+ ("Brief?", "Brief"),
+ ("Brief!", "Brief?"),
+ ("Brief!", "Brief"),
+ ],
+ )
+ def test_differing_input_markers_do_not_match(self, existing_spec: str, incoming_spec: str):
+ existing = _make_blueprint(inputs={"brief": existing_spec})
+ incoming = _make_blueprint(inputs={"brief": incoming_spec})
+ assert not contracts_match(existing=existing, incoming=incoming, domain_code="thisdomain")
+
+ def test_differing_output_markers_do_not_match(self):
+ existing = _make_blueprint(output="Brief?")
+ incoming = _make_blueprint(output="Brief")
+ assert not contracts_match(existing=existing, incoming=incoming, domain_code="thisdomain")
+
+ def test_multiplicity_suffixes_stay_distinct(self):
+ """Pre-existing behavior: [] and [1] compare as text, never conflated."""
+ existing = _make_blueprint(output="Brief[]")
+ incoming = _make_blueprint(output="Brief[1]")
+ assert not contracts_match(existing=existing, incoming=incoming, domain_code="thisdomain")
diff --git a/tests/unit/pipelex/pipe_controllers/condition/data.py b/tests/unit/pipelex/pipe_controllers/condition/data.py
index 7d3f0ddc0..86cb0513e 100644
--- a/tests/unit/pipelex/pipe_controllers/condition/data.py
+++ b/tests/unit/pipelex/pipe_controllers/condition/data.py
@@ -53,7 +53,7 @@ class PipeConditionInputTestCases:
PipeConditionBlueprint(
description="Test case: valid_with_special_outcome",
inputs={"should_process": "native.Text"},
- output="native.Text",
+ output="native.Text?",
expression="should_process",
outcomes={"yes": "process_pipe"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -78,7 +78,7 @@ class PipeConditionInputTestCases:
PipeConditionBlueprint(
description="Test case: valid_single_outcome",
inputs={"flag": "native.Text"},
- output="native.Text",
+ output="native.Text?",
expression="flag",
outcomes={"true": "process_pipe"},
default_outcome=SpecialOutcome.CONTINUE,
diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_needed_inputs.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_needed_inputs.py
index 932e6e17e..105d27a0b 100644
--- a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_needed_inputs.py
+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_needed_inputs.py
@@ -37,7 +37,7 @@ class TestPipeConditionNeededInputs:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with expression variables",
inputs={"status": "native.Text", "priority": "native.Text"},
- output="native.Text",
+ output="native.Text?",
expression="status", # Uses 'status' variable (gets wrapped in {{ }} internally)
outcomes={"active": SpecialOutcome.CONTINUE, "inactive": SpecialOutcome.CONTINUE},
default_outcome=SpecialOutcome.CONTINUE,
@@ -115,7 +115,7 @@ class TestPipeConditionNeededInputs:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition mapping to pipe_a and pipe_b",
inputs={"selector": "native.Text"},
- output=analysis_concept.concept_ref,
+ output=f"{analysis_concept.concept_ref}?",
expression="selector",
outcomes={"option_a": "pipe_a", "option_b": "pipe_b"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -182,7 +182,7 @@ class TestPipeConditionNeededInputs:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with list input pipe",
inputs={"mode": "native.Text"},
- output=result_concept.concept_ref,
+ output=f"{result_concept.concept_ref}?",
expression="mode",
outcomes={"batch": "pipe_with_list"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -267,7 +267,7 @@ class TestPipeConditionOutputValidation:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with matching outputs",
inputs={"selector": "native.Text"},
- output=output_concept.concept_ref,
+ output=f"{output_concept.concept_ref}?",
expression="selector",
outcomes={"a": "pipe_a", "b": "pipe_b"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -335,7 +335,7 @@ class TestPipeConditionOutputValidation:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with Dynamic output when not needed",
inputs={"selector": "native.Text"},
- output=NativeConceptCode.DYNAMIC.concept_ref,
+ output=f"{NativeConceptCode.DYNAMIC.concept_ref}?",
expression="selector",
outcomes={"a": "pipe_a", "b": "pipe_b"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -394,7 +394,7 @@ class TestPipeConditionOutputValidation:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with Anything output when not needed",
inputs={"selector": "native.Text"},
- output=NativeConceptCode.ANYTHING.concept_ref,
+ output=f"{NativeConceptCode.ANYTHING.concept_ref}?",
expression="selector",
outcomes={"a": "pipe_a"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -470,7 +470,7 @@ class TestPipeConditionOutputValidation:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with Dynamic output",
inputs={"selector": "native.Text"},
- output=NativeConceptCode.ANYTHING.concept_ref,
+ output=f"{NativeConceptCode.ANYTHING.concept_ref}?",
expression="selector",
outcomes={"a": "pipe_a", "b": "pipe_b"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -543,7 +543,7 @@ class TestPipeConditionOutputValidation:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with specific output when Dynamic is required",
inputs={"selector": "native.Text"},
- output=output_a.concept_ref,
+ output=f"{output_a.concept_ref}?",
expression="selector",
outcomes={"a": "pipe_a", "b": "pipe_b"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -620,7 +620,7 @@ class TestPipeConditionOutputValidation:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with Anything output when Dynamic is required",
inputs={"selector": "native.Text"},
- output=NativeConceptCode.TEXT.concept_ref,
+ output=f"{NativeConceptCode.TEXT.concept_ref}?",
expression="selector",
outcomes={"a": "pipe_a", "b": "pipe_b"},
default_outcome=SpecialOutcome.CONTINUE,
@@ -654,7 +654,7 @@ class TestPipeConditionSpecialOutcomes:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with only CONTINUE outcomes",
inputs={"status": "native.Text"},
- output="native.Text",
+ output="native.Text?",
expression="status",
outcomes={"active": SpecialOutcome.CONTINUE, "inactive": SpecialOutcome.CONTINUE},
default_outcome=SpecialOutcome.CONTINUE,
@@ -743,7 +743,7 @@ class TestPipeConditionSpecialOutcomes:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with mixed pipe and CONTINUE",
inputs={"action": "native.Text"},
- output=output_concept.concept_ref,
+ output=f"{output_concept.concept_ref}?",
expression="action",
outcomes={
"process": "process_pipe", # Actual pipe
@@ -839,7 +839,7 @@ class TestPipeConditionSpecialOutcomes:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with only special outcomes",
inputs={"flag": "native.Text"},
- output="native.Text",
+ output="native.Text?",
expression="flag",
outcomes={
"yes": SpecialOutcome.CONTINUE,
@@ -890,7 +890,7 @@ class TestPipeConditionSpecialOutcomes:
pipe_condition_blueprint = PipeConditionBlueprint(
description="Condition with mixed outcomes",
inputs={"mode": "native.Text"},
- output=output_concept.concept_ref,
+ output=f"{output_concept.concept_ref}?",
expression="mode",
outcomes={
"run": "real_pipe",
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
index 9ab0a6708..4e8ff3fc4 100644
--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py
+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
@@ -7,6 +7,7 @@ from pipelex.base_exceptions import ErrorDomain, ErrorReport
from pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction, UserActionKind
from pipelex.cogt.inference.provider_name import ProviderName
from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.stuff import Stuff
@@ -47,7 +48,7 @@ class TestDeliveryExecutor:
mock_output = mocker.MagicMock()
mock_output.working_memory_raw = None
mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
- mock_output.working_memory.get_main_stuff.return_value = _make_main_stuff()
+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
mock_output.graph_spec = None
executor = DeliveryExecutor()
@@ -165,7 +166,7 @@ class TestDeliveryExecutor:
mock_output = mocker.MagicMock()
mock_output.working_memory_raw = None
mock_output.working_memory.smart_dump.return_value = {}
- mock_output.working_memory.get_main_stuff.return_value = _make_main_stuff()
+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
mock_output.graph_spec = None
executor = DeliveryExecutor()
@@ -181,7 +182,9 @@ class TestDeliveryExecutor:
)
async def test_generate_result_files_raises_without_main_stuff_typed(self, mocker: MockerFixture) -> None:
- """A completed run always delivers a main stuff — a typed working memory without one is a contract violation, not an empty envelope."""
+ """A completed run always resolves its declared output — a typed working memory with neither
+ a main stuff nor a recorded absence is a contract violation, not an empty envelope.
+ """
mock_output = mocker.MagicMock()
mock_output.working_memory_raw = None
mock_output.working_memory = WorkingMemory()
@@ -191,7 +194,7 @@ class TestDeliveryExecutor:
await DeliveryExecutor().generate_result_files(mock_output)
async def test_generate_result_files_raises_without_main_stuff_raw(self, mocker: MockerFixture) -> None:
- """Same contract on the raw (cross-process) path: a raw working memory without a main stuff fails loudly."""
+ """Same contract on the raw (cross-process) path: neither a raw main stuff nor a recorded absence fails loudly."""
mock_output = mocker.MagicMock()
mock_output.working_memory_raw = {"root": {}, "aliases": {}}
mock_output.graph_spec = None
@@ -199,6 +202,65 @@ class TestDeliveryExecutor:
with pytest.raises(PipeJobError):
await DeliveryExecutor().generate_result_files(mock_output)
+ async def test_generate_result_files_absent_main_typed(self, mocker: MockerFixture) -> None:
+ """An absent main output is a first-class success: the typed arm delivers an explicit
+ absence artifact (main_stuff.json/md/html) instead of raising.
+ """
+ working_memory = WorkingMemory()
+ working_memory.record_new_main_absence(
+ AbsenceRecord(
+ variable_name="penalty_summary",
+ kind=AbsenceKind.DECLARED_ABSENT,
+ reason="no penalty clause found in this contract",
+ producing_pipe="check_penalty_clause",
+ ),
+ )
+ mock_output = mocker.MagicMock()
+ mock_output.working_memory_raw = None
+ mock_output.working_memory = working_memory
+ mock_output.graph_spec = None
+
+ files = await DeliveryExecutor().generate_result_files(mock_output)
+
+ assert "working_memory.json" in files
+ json_text = files["main_stuff.json"].data.decode("utf-8")
+ assert '"absent": true' in json_text
+ assert "no penalty clause found in this contract" in json_text
+ assert "check_penalty_clause" in json_text
+ md_text = files["main_stuff.md"].data.decode("utf-8")
+ assert "absent" in md_text.lower()
+ assert "no penalty clause found in this contract" in md_text
+ assert "main_stuff.html" in files
+
+ async def test_generate_result_files_absent_main_raw(self, mocker: MockerFixture) -> None:
+ """Same on the raw (cross-process) path: a recorded main absence in the raw ledger delivers
+ the absence artifact instead of the contract-violation error.
+ """
+ mock_output = mocker.MagicMock()
+ mock_output.working_memory_raw = {
+ "root": {},
+ "aliases": {},
+ "absences": {
+ "main_stuff": {
+ "variable_name": "penalty_summary",
+ "kind": "skipped",
+ "reason": "skipped because input 'penalty_clause' is absent",
+ "producing_pipe": "summarize_penalty",
+ "upstream": None,
+ },
+ },
+ }
+ mock_output.graph_spec = None
+
+ files = await DeliveryExecutor().generate_result_files(mock_output)
+
+ json_text = files["main_stuff.json"].data.decode("utf-8")
+ assert '"absent": true' in json_text
+ assert "skipped because input 'penalty_clause' is absent" in json_text
+ md_text = files["main_stuff.md"].data.decode("utf-8")
+ assert "summarize_penalty" in md_text
+ assert "main_stuff.html" in files
+
async def test_try_local_hydrate_stuff_returns_typed_for_builtin(self) -> None:
from pipelex.core.stuffs.text_content import TextContent # noqa: PLC0415
from pipelex.hub import get_class_registry # noqa: PLC0415
@@ -468,7 +530,7 @@ class TestDeliveryExecutor:
mock_output = mocker.MagicMock()
mock_output.working_memory_raw = None
mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
- mock_output.working_memory.get_main_stuff.return_value = _make_main_stuff()
+ mock_output.working_memory.resolve_main_stuff.return_value = _make_main_stuff()
mock_output.graph_spec = None
executor = DeliveryExecutor()
diff --git a/tests/unit/pipelex/pipeline/test_validation_report.py b/tests/unit/pipelex/pipeline/test_validation_report.py
index 0eede27cd..4fe8616a4 100644
--- a/tests/unit/pipelex/pipeline/test_validation_report.py
+++ b/tests/unit/pipelex/pipeline/test_validation_report.py
@@ -9,6 +9,7 @@ projection from the dry-run status map, and ``is_runnable = not pending_signatur
Pure unit test (no Pipelex boot): blueprints from the interpreter, hand-built status map.
"""
+from pipelex.base_exceptions import ValidationErrorCategory, ValidationErrorItem
from pipelex.core.interpreter.interpreter import PipelexInterpreter
from pipelex.pipeline.bundle_validator import DryRunOutput, DryRunStatus
from pipelex.pipeline.pipe_io_contracts import IOMultiplicity, PipeIOContract, PipeOutputContract
@@ -82,3 +83,28 @@ class TestBuildValidationReport:
assert report.bundle_blueprint is blueprints[0]
assert report.is_runnable is True
assert report.validated_pipes == []
+ # The advisory channel defaults empty and never affects the verdict.
+ assert report.warnings == []
+
+ def test_warnings_ride_the_report_without_flipping_the_verdict(self) -> None:
+ """Warnings share the error item shape but the report stays the valid arm (is_valid True)."""
+ blueprints = [PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=_MAIN_PIPE_MTHDS)]
+ warning_item = ValidationErrorItem(
+ category=ValidationErrorCategory.PIPE_VALIDATION,
+ error_type="optional_force_redundant",
+ pipe_code="do_it",
+ domain_code="beta",
+ variable_names=["doc"],
+ message="Input 'doc' of pipe 'beta.do_it' is declared '!' but is guaranteed present in every analyzed flow.",
+ )
+
+ report = build_validation_report(
+ blueprints=blueprints,
+ pipe_io_contracts={},
+ dry_run_result={},
+ pending_signatures=[],
+ warnings=[warning_item],
+ )
+
+ assert report.is_valid is True
+ assert report.warnings == [warning_item]
diff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py b/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py
index 089f7c018..0416f3525 100644
--- a/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py
+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py
@@ -4,6 +4,7 @@ import pytest
from pipelex.core.concepts.concept import Concept
from pipelex.core.domains.domain import SpecialDomain
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.number_content import NumberContent
@@ -216,3 +217,42 @@ class TestHydrateWorkingMemory:
with pytest.raises(PipeJobError, match="invalid_stuff"):
hydrate_working_memory(raw)
+
+ def test_hydrate_round_trips_absence_ledger(self) -> None:
+ """The absence ledger survives the dump_for_transport → hydrate round-trip: a recorded
+ absence must not degrade to a hard miss after cross-process transit.
+ """
+ working_memory = WorkingMemory()
+ working_memory.root["kept"] = _make_text_stuff("kept", "still here")
+ origin = AbsenceRecord(
+ variable_name="source",
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason="optional input 'source' was not provided by the caller",
+ )
+ chained = AbsenceRecord(
+ variable_name="analysis",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'source' is absent",
+ producing_pipe="analyze",
+ upstream=origin,
+ )
+ working_memory.record_absence(origin)
+ working_memory.record_resolved_absence(chained)
+
+ hydrated = hydrate_working_memory(working_memory.dump_for_transport())
+
+ assert hydrated.absences["source"] == origin
+ assert hydrated.absences["analysis"] == chained
+ assert hydrated.absences["analysis"].upstream == origin
+ assert hydrated.root["kept"].content == TextContent(text="still here")
+
+ def test_hydrate_raises_on_malformed_absence_record(self) -> None:
+ """A malformed ledger entry fails hydration loudly — never a silently dropped record."""
+ raw: dict[str, Any] = {
+ "root": {},
+ "aliases": {},
+ "absences": {"broken": {"kind": "not_a_kind"}},
+ }
+
+ with pytest.raises(PipeJobError, match="broken"):
+ hydrate_working_memory(raw)
diff --git a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py b/tests/unit/pipelex/runtime_bridge/test_output_serialization.py
index d0f35ff5a..affb5f933 100644
--- a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py
+++ b/tests/unit/pipelex/runtime_bridge/test_output_serialization.py
@@ -1,6 +1,7 @@
import pytest
from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.stuff import Stuff
@@ -88,7 +89,9 @@ class TestSerializeCompletedOutput:
assert resolve_main_stuff_root_key(pipe_output=pipe_output) == MAIN_STUFF_NAME
def test_missing_main_stuff_raises(self) -> None:
- """A completed run always delivers a main stuff — serializing one without it is a contract violation."""
+ """A completed run always resolves its declared output — a working memory with neither a
+ value nor a recorded absence at this boundary is a contract violation.
+ """
pipe_output = PipeOutput(
working_memory=WorkingMemory(),
pipeline_run_id="run-broken",
@@ -96,3 +99,28 @@ class TestSerializeCompletedOutput:
with pytest.raises(PipeJobError):
serialize_completed_output(pipe_output=pipe_output, workflow_id=None)
+
+ def test_absent_main_output_resolves_declared_slot_name(self) -> None:
+ """A run whose main output resolved absent is a success: main_stuff_name names the
+ declared output slot and the serialized output_dict carries the absence records —
+ consumers branch on the record, never on transport.
+ """
+ memory = WorkingMemory()
+ record = AbsenceRecord(
+ variable_name="summary",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'analysis' is absent",
+ producing_pipe="summarize",
+ )
+ memory.record_new_main_absence(record)
+ pipe_output = PipeOutput(working_memory=memory, pipeline_run_id="run-absent")
+
+ assert resolve_main_stuff_root_key(pipe_output=pipe_output) == "summary"
+
+ dto = serialize_completed_output(pipe_output=pipe_output, workflow_id=None)
+ assert dto.main_stuff_name == "summary"
+ serialized_record = dto.output_dict["absences"]["summary"]
+ assert serialized_record["kind"] == "skipped"
+ assert serialized_record["producing_pipe"] == "summarize"
+ # The positional main-stuff key marks the absence as the run's resolved main result.
+ assert dto.output_dict["absences"][MAIN_STUFF_NAME]["variable_name"] == "summary"
diff --git a/tests/unit/pipelex/system/telemetry/test_otel_factory.py b/tests/unit/pipelex/system/telemetry/test_otel_factory.py
index cfc939f2b..5fdb2a349 100644
--- a/tests/unit/pipelex/system/telemetry/test_otel_factory.py
+++ b/tests/unit/pipelex/system/telemetry/test_otel_factory.py
@@ -1,13 +1,77 @@
"""Unit tests for OTel utility functions."""
+import json
+
import pytest
+from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
+from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.system.telemetry.otel_factory import OtelFactory
class TestOtelFactory:
"""Test OTel factory utilities."""
+ # --- make_output_json tests ---
+
+ def test_make_output_json_absent_main_output(self) -> None:
+ """An absent main output serializes to an explicit absence payload — the span ends
+ successfully, never raises (a completed run always resolves its declared output).
+ """
+ memory = WorkingMemory()
+ memory.record_new_main_absence(
+ AbsenceRecord(
+ variable_name="summary",
+ kind=AbsenceKind.SKIPPED,
+ reason="skipped because input 'analysis' is absent",
+ producing_pipe="summarize",
+ )
+ )
+ pipe_output = PipeOutput(working_memory=memory, pipeline_run_id="run-absent")
+
+ output_json = OtelFactory.make_output_json(pipe_output=pipe_output, max_length=None)
+
+ payload = json.loads(output_json)
+ assert payload["absent"] is True
+ assert payload["variable_name"] == "summary"
+ assert payload["kind"] == "skipped"
+ assert payload["reason"] == "skipped because input 'analysis' is absent"
+
+ # --- make_inputs_json tests ---
+
+ def test_make_inputs_json_captures_recorded_absence(self) -> None:
+ """An input slot resolved as a recorded absence captures as the explicit absence
+ payload — a legitimately omitted optional never breaks span input capture.
+ """
+ memory = WorkingMemoryFactory.make_from_single_stuff(StuffFactory.make_from_str("penalties", name="topic"))
+ memory.record_absence(
+ AbsenceRecord(
+ variable_name="clause",
+ kind=AbsenceKind.NOT_PROVIDED,
+ reason="optional input 'clause' was not provided by the caller",
+ )
+ )
+
+ inputs_json = OtelFactory.make_inputs_json(memory, needed_input_names={"topic", "clause"}, max_length=None)
+
+ payload = json.loads(inputs_json)
+ assert payload["topic"]["content"] == {"text": "penalties"}
+ assert payload["clause"]["absent"] is True
+ assert payload["clause"]["kind"] == "not_provided"
+
+ def test_make_inputs_json_neither_value_nor_record_raises(self) -> None:
+ """A slot with neither a value nor a recorded absence is the bug case — the capture
+ raises loudly instead of silently omitting the input.
+ """
+ memory = WorkingMemory()
+
+ with pytest.raises(WorkingMemoryStuffNotFoundError):
+ OtelFactory.make_inputs_json(memory, needed_input_names={"ghost"}, max_length=None)
+
# --- make_truncated_content tests ---
def test_make_truncated_content_no_truncation_when_within_limit(self) -> None:
diff --git a/tests/unit/pipelex/tools/test_jinja2_optional_guards.py b/tests/unit/pipelex/tools/test_jinja2_optional_guards.py
new file mode 100644
index 000000000..cc46df286
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_jinja2_optional_guards.py
@@ -0,0 +1,106 @@
+"""Guard classification for declared-optional template variables (D7): every reference to an
+optional input must be guarded — reachable only inside `{% if var %}`-style blocks, inline
+`is defined` conditionals, or via `@?var` (whose rewritten form is a `{% if %}` guard). The
+walker reports each unguarded reference so validation can name the precise fix.
+"""
+
+import pytest
+
+from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.tools.jinja2.jinja2_optional_guards import detect_unguarded_optional_references
+
+
+class TestDetectUnguardedOptionalReferences:
+ @pytest.mark.parametrize(
+ ("topic", "template_source"),
+ [
+ ("if_block_guard", "{% if assessment %}{{ assessment.amount }}{% endif %}"),
+ ("if_defined_guard", "{% if assessment is defined %}{{ assessment }}{% endif %}"),
+ ("at_optional_rewritten_form", '{% if assessment %}{{ assessment|tag("assessment") }}{% endif %}'),
+ ("inline_cond_defined", "{{ 'has assessment' if assessment is defined else 'none' }}"),
+ ("inline_cond_truthy", "{{ assessment.amount if assessment else 'none' }}"),
+ ("and_guard", "{% if assessment and topic %}{{ assessment.amount }}{% endif %}"),
+ ("bare_presence_test_only", "{% if assessment %}present{% endif %}"),
+ ("nested_guard", "{% if assessment %}{% for item in assessment.items %}{{ item }}{% endfor %}{% endif %}"),
+ ("non_optional_vars_untouched", "{{ topic }} and {{ topic.detail }}"),
+ ("no_references_at_all", "static text only"),
+ ("and_short_circuit_deep_test", "{% if assessment and assessment.flag %}x{% endif %}"),
+ ("and_short_circuit_guards_body", "{% if assessment and assessment.flag %}{{ assessment.detail }}{% endif %}"),
+ ("set_before_read", "{% set assessment = 'x' %}{{ assessment }}"),
+ ("nested_set_shadows", "{% if topic %}{% set assessment = 'fallback' %}{{ assessment }}{% endif %}"),
+ ("set_block_before_read", "{% set assessment %}fallback{% endset %}{{ assessment }}"),
+ ("set_block_target_is_a_store_not_a_read", "{% set assessment %}fallback{% endset %}"),
+ ("set_block_body_guarded", "{% set summary %}{% if assessment %}{{ assessment.amount }}{% endif %}{% endset %}{{ summary }}"),
+ ("tuple_set_shadows", "{% set assessment, other = 'a', 'b' %}{{ assessment }}"),
+ ("with_shadow_local", "{% with assessment = 'x' %}{{ assessment }}{% endwith %}"),
+ ("for_filter_bare_presence_probe", "{% for item in topic_items if assessment %}{{ item }}{% endfor %}"),
+ ("for_filter_target_shadows", "{% for assessment in topic_items if assessment.flag %}{{ assessment }}{% endfor %}"),
+ ],
+ )
+ def test_guarded_references_produce_no_findings(self, topic: str, template_source: str):
+ findings = detect_unguarded_optional_references(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source=template_source,
+ optional_variable_names={"assessment"},
+ )
+ assert findings == [], f"{topic}: expected no findings, got {findings}"
+
+ @pytest.mark.parametrize(
+ ("topic", "template_source", "expected_path"),
+ [
+ ("bare_interpolation", "{{ assessment }}", "assessment"),
+ ("deep_access", "{{ assessment.amount }}", "assessment.amount"),
+ ("guard_on_other_var", "{% if topic %}{{ assessment }}{% endif %}", "assessment"),
+ ("else_branch_use", "{% if assessment %}x{% else %}{{ assessment.amount }}{% endif %}", "assessment.amount"),
+ ("for_iteration", "{% for item in assessment %}{{ item }}{% endfor %}", "assessment"),
+ ("deep_access_in_test", "{% if assessment.flag %}x{% endif %}", "assessment.flag"),
+ ("filtered_unguarded", "{{ assessment|length }}", "assessment"),
+ ("inline_cond_else_side", "{{ 'x' if topic else assessment.amount }}", "assessment.amount"),
+ ("tag_filter_from_plain_at_sigil", '{{ assessment|tag("assessment") }}', "assessment"),
+ ("attr_on_subscript_result", "{{ assessment[0].field }}", "assessment"),
+ ("read_before_set", "Value: {{ assessment }} {% set assessment = 'x' %}", "assessment"),
+ ("set_block_body_reads_optional", "{% set summary %}{{ assessment }}{% endset %}{{ summary }}", "assessment"),
+ ("read_before_set_block", "{{ assessment }}{% set assessment %}x{% endset %}", "assessment"),
+ ("with_value_reads_optional", "{% with local = assessment %}{{ local }}{% endwith %}", "assessment"),
+ ("with_body_reads_other_optional", "{% with other = 'x' %}{{ assessment.amount }}{% endwith %}", "assessment.amount"),
+ ("for_filter_deep_access", "{% for item in topic_items if assessment.flag %}{{ item }}{% endfor %}", "assessment.flag"),
+ ],
+ )
+ def test_unguarded_references_are_reported(self, topic: str, template_source: str, expected_path: str):
+ findings = detect_unguarded_optional_references(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source=template_source,
+ optional_variable_names={"assessment"},
+ )
+ assert len(findings) == 1, f"{topic}: expected one finding, got {findings}"
+ assert findings[0].variable_name == "assessment"
+ assert findings[0].path == expected_path
+
+ def test_multiple_optional_variables_reported_independently(self):
+ findings = detect_unguarded_optional_references(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source="{% if extra %}{{ extra }}{% endif %} {{ assessment }} {{ note.detail }}",
+ optional_variable_names={"assessment", "note", "extra"},
+ )
+ assert {(finding.variable_name, finding.path) for finding in findings} == {
+ ("assessment", "assessment"),
+ ("note", "note.detail"),
+ }
+
+ def test_guard_does_not_leak_outside_its_block(self):
+ findings = detect_unguarded_optional_references(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source="{% if assessment %}{{ assessment }}{% endif %}{{ assessment }}",
+ optional_variable_names={"assessment"},
+ )
+ assert len(findings) == 1
+ assert findings[0].path == "assessment"
+
+ def test_expression_category_supported(self):
+ """The PipeCondition presence-branching idiom (design §15) lints clean."""
+ findings = detect_unguarded_optional_references(
+ template_category=TemplateCategory.EXPRESSION,
+ template_source="{{ 'present' if penalty_clause is defined else 'absent' }}",
+ optional_variable_names={"penalty_clause"},
+ )
+ assert findings == []
diff --git a/tests/unit/pipelex/tracing/test_graphspec_assembler.py b/tests/unit/pipelex/tracing/test_graphspec_assembler.py
index 1e5a29654..b0a6d7e37 100644
--- a/tests/unit/pipelex/tracing/test_graphspec_assembler.py
+++ b/tests/unit/pipelex/tracing/test_graphspec_assembler.py
@@ -12,6 +12,7 @@ from pipelex.tracing.trace_events import (
EdgeEvent,
ParallelCombineEvent,
PipeEndErrorEvent,
+ PipeEndSkippedEvent,
PipeEndSuccessEvent,
PipeStartEvent,
)
@@ -117,6 +118,56 @@ class TestGraphSpecAssembler:
assert result.concept_registry["sourceful.InputConcept"]["source"] == input_concept_source
assert result.concept_registry["sourceful.OutputConcept"]["source"] == output_concept_source
+ def test_skipped_plural_output_still_produces_data_edge(self) -> None:
+ """A PipeEndSkippedEvent carrying the lifted pipe's real empty-list output spec (D4)
+ registers the digest in the producer map, so the downstream DATA edge resolves —
+ mirroring GraphTracer.on_pipe_end_skipped.
+ """
+ producer_node = _node_id(_WF_A, 0)
+ consumer_node = _node_id(_WF_A, 1)
+
+ result = GraphSpecAssembler.assemble(
+ events=[
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=producer_node,
+ pipe_code="lifted_plural_pipe",
+ pipe_type="PipeFunc",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ PipeEndSkippedEvent(
+ **_base(_WF_A, 1),
+ node_id=producer_node,
+ ended_at=_time_at(1),
+ skip_reason="skipped because input 'source' is absent",
+ output_spec=_io_spec("items", "stuff-empty-list"),
+ ),
+ PipeStartEvent(
+ **_base(_WF_A, 2),
+ node_id=consumer_node,
+ pipe_code="items_consumer",
+ pipe_type="PipeFunc",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("items", "stuff-empty-list")],
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 3),
+ node_id=consumer_node,
+ ended_at=_time_at(3),
+ ),
+ ],
+ graph_id=_GRAPH_ID,
+ )
+
+ skipped_node = next(node for node in result.nodes if node.node_id == producer_node)
+ assert skipped_node.status == NodeStatus.SKIPPED
+ assert skipped_node.skip_reason == "skipped because input 'source' is absent"
+ assert [output.digest for output in skipped_node.node_io.outputs] == ["stuff-empty-list"]
+ data_edges = [edge for edge in result.edges if edge.kind == EdgeKind.DATA]
+ assert len(data_edges) == 1
+ assert data_edges[0].source == producer_node
+ assert data_edges[0].target == consumer_node
+
def test_simple_sequence(self) -> None:
"""3 child pipes in sequence under a parent.
diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.py
index 5cab64675..34644258a 100644
--- a/tests/unit/pipelex/tracing/test_trace_events.py
+++ b/tests/unit/pipelex/tracing/test_trace_events.py
@@ -20,6 +20,7 @@ from pipelex.tracing.trace_events import (
ExecutionDataEvent,
ParallelCombineEvent,
PipeEndErrorEvent,
+ PipeEndSkippedEvent,
PipeEndSuccessEvent,
PipeStartEvent,
TraceEvent,
@@ -125,6 +126,15 @@ class TestTraceEvents:
"error": _Shared.make_error_spec(),
},
),
+ (
+ "pipe_end_skipped",
+ TraceEventKind.PIPE_END_SKIPPED,
+ {
+ "node_id": _Shared.NODE_ID,
+ "ended_at": _Shared.TIMESTAMP,
+ "skip_reason": "skipped because input 'source' is absent",
+ },
+ ),
(
"edge",
TraceEventKind.EDGE,
@@ -198,6 +208,7 @@ class TestTraceEvents:
TraceEventKind.PIPE_START: PipeStartEvent,
TraceEventKind.PIPE_END_SUCCESS: PipeEndSuccessEvent,
TraceEventKind.PIPE_END_ERROR: PipeEndErrorEvent,
+ TraceEventKind.PIPE_END_SKIPPED: PipeEndSkippedEvent,
TraceEventKind.EDGE: EdgeEvent,
TraceEventKind.CONTROLLER_OUTPUT: ControllerOutputEvent,
TraceEventKind.BATCH_ITEM: BatchItemEvent,
diff --git a/wip/optionals/README.md b/wip/optionals/README.md
new file mode 100644
index 000000000..9c124721f
--- /dev/null
+++ b/wip/optionals/README.md
@@ -0,0 +1,24 @@
+# Optionals (`?` / `!`) — track folder
+
+## Where things stand
+
+**Phase 1 (the language core) is COMPLETE** — all steps A–F landed, final checkpoint cleared. The authoritative record is [optionals-phase1-tracker.md](optionals-phase1-tracker.md) (archived from the worktree-root `TODOS.md`; checkpoint log, decisions/deviations, hand-off notes). PR #1021 → dev is merge-ready; Louis merges.
+
+**Decided:** `??` coalescing is pulled forward from phase 3 into phase 2 — it is the ergonomic replacement for the `continue` pass-through that phase 1 broke (design doc §13 migration note, §17 phasing).
+
+## What comes next — three streams
+
+1. **Release + conformance de-gate — no new plan needed.** Merge PR #1021, cut the pipelex release carrying optionals, pipelex-api re-pins to it; then run the gate-removal checklist in [deferred-step-f-notes.md](deferred-step-f-notes.md) (remove the `OPTIONALS_PENDING_FEATURE` gates, `make validate-error-qa`). Mechanical once the release exists.
+2. **Cross-repo optionals wave — plan to write.** Scope: mthds spec pages, mthds-python `absences` ledger on `DictWorkingMemoryAbstract` (un-gates the `_run_core_api.py` absent arm back here), mthds-js type mirrors, mthds-ui (skipped nodes, optional edges, `skip_reason`, the `conceptRefs.ts` parse-to-null bug), vscode-pipelex (grammar, semantic tokens, the Rust `strip_concept_qualifiers` bug), pipelex-api response models (`warnings` + contract `optional` flags), mthds-plugins skills. **Gated:** lands after the Required-main-stuff Phase 3 sweep (same mthds spec pages) and the release — except the two syntax-highlighting handoffs, which are ungated and already written up (workspace root `../wip/optionals/`).
+3. **Phase 2 — plan to write, in-repo and ungated.** The PipeLLM/`Text?` maybe-wrapper with reasons + `??` coalescing (per the decision above). Candidate rider: the PipeBatch ledger-parity observability question from the Step F notes.
+
+## Documents
+
+- [optionals-design.md](optionals-design.md) — the design reference: decisions D1–D11, phasing (§17), decision summary. Still authoritative for future phases.
+- [optionals-plan.md](optionals-plan.md) — the phase-1 narrative plan (steps A–F). CLOSED; kept as the record the tracker implements.
+- [optionals-phase1-tracker.md](optionals-phase1-tracker.md) — the phase-1 tracker (checkpoint log, decisions/deviations log, cold-start context). CLOSED; archived here from the worktree-root `TODOS.md` now that phase 1 is done.
+- [deferred-step-f-notes.md](deferred-step-f-notes.md) — **live follow-ups**: the conformance gate-removal checklist for the release, the hosted-wire absence-records pin, PipeBatch ledger-parity question, a pre-existing template-variable detection quirk.
+- [deferred-absence-handling-dedup.md](deferred-absence-handling-dedup.md), [deferred-step-d-review-tradeoffs.md](deferred-step-d-review-tradeoffs.md), [deferred-step-e-review-tradeoffs.md](deferred-step-e-review-tradeoffs.md) — design-tradeoff findings from the checkpoint cold reviews, deliberately not applied; triage on demand.
+- [optionals-phase1-highlights.html](optionals-phase1-highlights.html) — presentation artifact summarizing phase 1.
+
+Cross-repo hand-off docs (vscode-pipelex + mthds-ui marker highlighting) live at the workspace root: `../wip/optionals/`.
diff --git a/wip/optionals/deferred-absence-handling-dedup.md b/wip/optionals/deferred-absence-handling-dedup.md
new file mode 100644
index 000000000..d67054dac
--- /dev/null
+++ b/wip/optionals/deferred-absence-handling-dedup.md
@@ -0,0 +1,19 @@
+# Deferred design-tradeoff items from the Checkpoint C cold review
+
+Review findings on the Step C diff (`7d2ed6bf4..0df759bd6`) that are deliberate tradeoffs, not bugs. Recorded here per the checkpoint protocol instead of reflexively applied; revisit when the shapes stabilize (after Step E at the earliest, since Step E touches the same boundaries).
+
+## 1. The tri-state branch-collect pattern is hand-rolled per controller
+
+`resolve_main_stuff()` → `isinstance AbsenceRecord` → policy appears in `pipe_parallel.py` (D11 combine policy), `pipe_batch.py` collect loop (compaction), and `pipe_batch.py` dry stamping (guard). The three policies are genuinely different (omit-or-error vs drop vs skip-stamping), so a shared helper today would abstract only the two-line resolve+isinstance prelude while forcing the policies through a callback. Decision: keep the pattern inline until a fourth site appears (e.g. a future controller); then extract a `resolve_branch_output()` helper with the re-keying rule from `pipe_parallel.py` as part of it.
+
+## 2. `_generate_absence_files` shares the JSON + escaped-`<pre>` triplet with `_generate_main_stuff_files_from_raw`
+
+Both render `clean_json_dumps` + `html.escape` inside `<pre>`. The overlap is a few lines and the markdown bodies are entirely different. Decision: not worth a shared helper for two call sites; if a third "render this dict as json/md/html artifact triplet" site appears in Step E (execute response? CLI?), extract then.
+
+## 3. Nesting depth in `PipeParallel._run_branches_and_combine`
+
+The per-branch loop nests name bookkeeping, the D11 required-field guard, and accumulation. An extraction like `_resolve_branch_or_record_absence(...) -> Stuff | None` would flatten it. Decision: cosmetic; fold into item 1's helper if/when it happens.
+
+## 4. Non-`None` field defaults absorb the absence as the default (documented, kept)
+
+The D11 absorption check is `field_info.is_required()`, so a non-required field with a non-`None` default absorbs an absent branch as that default. This is deliberate: it matches pydantic semantics, and an author-declared default on an optional field is exactly a per-field coalescing declaration (a preview of the phase-2 `??`). Changelog and docs now state "the field's default (`None` unless the author declared another default)" rather than promising `None`.
diff --git a/wip/optionals/deferred-step-d-review-tradeoffs.md b/wip/optionals/deferred-step-d-review-tradeoffs.md
new file mode 100644
index 000000000..8c6b81d4d
--- /dev/null
+++ b/wip/optionals/deferred-step-d-review-tradeoffs.md
@@ -0,0 +1,38 @@
+# Deferred design tradeoffs from the Checkpoint D cold review
+
+Findings from the Checkpoint D cold review that are deliberate scoping choices or efficiency polish, not silent bugs. Each entry records the finding, why it is deferred, and what would trigger picking it up. (Correctness findings from the same review were fixed in the checkpoint follow-up commit: the lifted `add_each_output` companion-slot gap, the guard-walker false negatives/positives, the leaking `Jinja2DetectVariablesError`, the PipeSearch/PipeImgGen lint coverage, the prose-in-`variable_names`, and the duplicated taint bookkeeping.)
+
+## 1. `liftable_pipes` only covers pipes in the validated bundle
+
+`build_liftable_pipes` iterates `result.pipes` — the pipes loaded from the validated bundle batch. A cross-package dependency controller (resolved through the hub during a taint walk) is not itself iterated, so ITS liftable branches don't appear as separate entries.
+
+**Why deferred:** whether the valid report should enumerate internals of *other* packages is a real design question — the inventory is keyed to the bundle under validation, and a dependency package's flows are its own validate's concern. The D3 commitment ("every liftable pipe is visible at build time") is satisfied within the bundle; cross-package visibility overlaps with how the protocol wants to scope reports per package.
+
+**Pick up when:** the Step E/F protocol work formalizes `liftable_pipes` on the wire — decide the scoping there (per-bundle vs per-closure) and document it in the spec either way.
+
+## 2. Taint analyses recomputed per validate pass
+
+A controller's taint analysis can run up to three times during one validation: its own `validate_output_with_library`, an enclosing sequence's `analyze_taint` (branch-taint sub-analysis), and `build_liftable_pipes`. Similarly, PipeLLM templates are Jinja2-parsed twice in `validate_inputs_static` (variable detection + guard-lint).
+
+**Why deferred:** validation-time only, small ASTs/flows, no measured pain. Memoizing per-pipe analyses or sharing a parsed AST adds state/invalidation complexity to save milliseconds at author time.
+
+**Pick up when:** validation latency on large libraries becomes a measurable complaint (profile first).
+
+## 3. `_build_full_path` duplicated between jinja2 walkers
+
+`jinja2_optional_guards._build_full_path` mirrors the private helper in `jinja2_required_variables`. Sharing means exporting a private helper or minting a tiny common module.
+
+**Why deferred:** the helper is small and stable; a cross-module private import or a one-function module is more structure than the duplication costs today.
+
+**Pick up when:** a third walker needs it, or either copy needs a behavior change (then unify first).
+
+## 4. Guard shapes deliberately not recognized
+
+The guard walker recognizes `{% if var %}` bodies (including `and`-combos with short-circuit awareness), `is defined` tests, and CondExpr true-arms. Still linted as unguarded, though runtime-safe in principle:
+
+- inverted guards: `{% if not var %}fallback{% else %}{{ var.attr }}{% endif %}` (the else-arm of a negative presence test);
+- the false-arm of a negated inline conditional: `{{ 'x' if not var else var.attr }}`.
+
+**Why deferred:** the walker is documented as deliberately narrow — a conservative lint with a precise fix message beats a clever one; the positive-guard rewrite is always available and arguably clearer.
+
+**Pick up when:** real bundles hit these shapes and the rewrite is genuinely awkward (collect examples first).
diff --git a/wip/optionals/deferred-step-e-review-tradeoffs.md b/wip/optionals/deferred-step-e-review-tradeoffs.md
new file mode 100644
index 000000000..7b50aeced
--- /dev/null
+++ b/wip/optionals/deferred-step-e-review-tradeoffs.md
@@ -0,0 +1,47 @@
+# Deferred design tradeoffs from the Checkpoint E cold review
+
+Findings from the Checkpoint E cold review that are deliberate scoping choices, cross-repo-gated work, or pre-existing sharp edges the diff newly exposed — not silent bugs introduced by Step E. Each entry records the finding, why it is deferred, and what would trigger picking it up. (Correctness and consistency findings from the same review were fixed in the checkpoint follow-up commit: the lifted-plural producer registration in both graph builders, the useless-`!` lint wired into the bare CLI and the builder validate_ops twins, the single shared taint walk per validate pass, the shared `output_digest_is_optional` helper, the shared absence JSON/HTML renderers, and the docs for the new graph and validate surfaces.)
+
+## 1. API-runner run path errors on a designed absence (cross-repo gated)
+
+`pipelex-agent run --runner api` raises a "runner contract violation" `PipeExecutionError` when the main output legitimately resolved absent, because the mthds SDK wire model (`DictWorkingMemoryAbstract`, `extra="forbid"`, root + aliases only) cannot carry the absences ledger — so the CLI cannot distinguish a designed absence from a genuinely missing output. The local runner renders a first-class absence document for the same run.
+
+**Why deferred:** fixing it requires the mthds SDK (and the API response models) to carry `absences` on the wire — a cross-repo protocol bump already listed in the tracker's cross-repo hand-off notes. Guessing "absent" from a missing key without the ledger would erase provenance and mask real contract bugs.
+
+**Pick up when:** the mthds SDK working-memory model gains the `absences` field; then implement the absent arm in `_run_core_api.py` (mirror `_run_core.py`'s absence document) and add the API-arm test.
+
+## 2. Builder-generated runner code crashes on an absent optional output (pre-existing, newly exposed)
+
+`pipelex/builder/runner_code.py` scaffolds example scripts that read `pipe_output.main_stuff` / `main_stuff_as(...)` unconditionally. For a pipe with a declared-optional output whose run resolves absent, the generated script dies in `get_main_stuff` (the ledger is not consulted) while every in-repo surface renders an absence document.
+
+**Why deferred:** builder codegen is the spec/authoring layer — making the generated snippet absence-aware (branch on `resolve_main_stuff()`, print the absence summary) is an authoring-UX design choice that belongs with the builder-layer optionals pass, not a one-line patch inside a checkpoint triage.
+
+**Pick up when:** the optionals track reaches the builder/spec layer (specs learn `?`/`!`), or the first user report of a scaffolded script crashing on a designed absence.
+
+## 3. `{{ var | length }}` on singular content still raises (pre-existing sharp edge)
+
+`StuffArtefact.__bool__` fixed the `{% if var %}` truth test, but `__len__` still raises `TypeError` for non-list content, and Jinja2's `length` filter is the `len()` builtin — so `{{ var | length }}` (or `{% if var|length > 0 %}`) on a PRESENT singular value raises the very error the `__bool__` fix targets.
+
+**Why deferred:** giving singular content a `__len__` invents a semantic (1? character count?) that would silently change template behavior far beyond optionals; D4 keeps length a list concept. The truthiness idiom (`{% if var %}`) is the documented guard shape and works on both arms.
+
+**Pick up when:** real bundles hit `|length` on singular optionals — then extend the template guard-lint to flag `|length` applied to a singular slot and point at the truthiness idiom instead (lint, not semantics).
+
+## 4. `GraphTracerManager.add_edge` facade does not expose `optional` (latent)
+
+The tracer-level `add_edge` carries `optional`, but no current caller routes explicit edges through the manager facade with that flag — reviewer-confirmed latent, no behavior impact today.
+
+**Why deferred:** dead parameter-plumbing until a controller actually emits an explicit optional edge; the DATA-edge path (where `optional` is computed) does not go through the facade.
+
+**Pick up when:** a controller needs to emit an explicit edge with the optional marker — extend the facade signature then.
+
+## 5. No OTel span for lifted leaf pipes (pre-existing)
+
+A lifted pipe returns from `run_pipe` before the operator-level OTel span factory runs, so a skipped leaf emits no span at all (the graph, ledger, and parent spans still record the skip). Pre-existing behavior untouched by Step E; the new absence-aware OTel code covers live-run pipes with absent optional inputs/outputs.
+
+**Why deferred:** emitting a dedicated "skipped" span changes the telemetry shape and belongs with a deliberate observability design pass, not a triage patch.
+
+**Pick up when:** operators report needing per-skip spans for tracing dashboards (the graph's `SKIPPED` node + `skip_reason` covers today's "why did this produce nothing?" question).
+
+## 6. Deeper taint-walk memoization still open (carried from Step D)
+
+Step E's triage collapsed the two validate-report projections onto one `collect_controller_taint_analyses` pass, but a controller's analysis can still run again inside `validate_output_with_library` and inside an enclosing controller's walk (Step D deferred item 2). Same reasoning, unchanged: validation-time only, no measured pain — profile before memoizing.
diff --git a/wip/optionals/deferred-step-f-notes.md b/wip/optionals/deferred-step-f-notes.md
new file mode 100644
index 000000000..cdacc5dd7
--- /dev/null
+++ b/wip/optionals/deferred-step-f-notes.md
@@ -0,0 +1,23 @@
+# Step F deferred notes
+
+Deferred items surfaced while writing the Step F specs/conformance/docs. Same policy as the other deferred docs in this folder: none of these block phase 1; triage them with the cross-repo wave or on demand.
+
+## Pre-existing: inline `$var` followed by ` (` escapes template variable detection
+
+Found while ground-truthing the `optional_input_unguarded` conformance fixture. A prompt like `"Greet $name (nickname: $nickname)."` reports `extraneous_input_variable` for `name` — the inline `$name` reference is not detected as a variable use when it is immediately followed by a space and an opening parenthesis. `"Greet $name today, nickname: $nickname."` detects both fine. Most likely the preprocessed Jinja2 parses `name (…)` as a call expression, and the variable walker misses the callee (or the detection regex swallows the call shape). Pre-existing behavior, unrelated to optionals — reproduced with plain (non-optional) inputs too. Worth a small fix in the variable-detection walk (`jinja2` dissection utilities); the workaround is any other punctuation/wording between the variable and the parenthesis.
+
+## Conformance optionals coverage is pinned ahead of the runtime — remove the gates at release
+
+All new conformance coverage (the `CATEGORIZED_VALIDATION_ITEMS` rows, `tests/pipelex_api/test_validate_optionals.py`, `tests/pipelex_api/test_validate_warnings.py`, `tests/pipelex_agent/test_validate_optionals.py`, and the QA-corpus cases) is gated on `OPTIONALS_PENDING_FEATURE` because the sibling venvs (`pipelex/.venv`, `pipelex-api/.venv`) run the released pipelex without optionals. The gated CLI-arm coverage has been validated LIVE against this branch via `CONFORMANCE_CLI_PIPELEX_AGENT=$workspace/_optionals/.venv/bin/pipelex-agent` (all categorized rows + both warnings tests pass, zero skips), so the gates are known non-vacuous. Once a pipelex release with optionals ships **and** pipelex-api re-pins to it:
+
+1. remove `pending_runtime_feature=OPTIONALS_PENDING_FEATURE` from the table rows and QA cases (the documented marker-removal policy),
+2. run `make validate-error-qa` in `conformance/` to generate the committed artifacts for the six new cases,
+3. drop the probe-skips if desired (they self-deactivate, but the gate comments say to remove them so regressions become visible).
+
+## PipeBatch compaction does not ledger the dropped branches (PipeParallel does)
+
+Surfaced by the Step F cold review. When PipeBatch compacts an absent branch result out of the aggregated list (`pipe_batch.py`, the `AbsenceRecord` arm of the aggregation loop), it only emits a `log.verbose` line — unlike PipeParallel, which calls `working_memory.record_resolved_absence` for an omitted composite component. So a run's absence ledger does not show which batch items were dropped. Design question, not a bug: N dropped branches would add N records (noisy for large batches), and the compacted list itself is a present value at the boundary. If ledger parity is wanted (e.g. for the route-or-skip auditing story), record one absence per dropped branch (or one summarizing record) into the parent working memory during aggregation. Revisit with phase 2 observability work.
+
+## Absence records on the hosted wire — conformance pin awaits the mthds bump
+
+The spec section "Absence records on run artifacts" is `<!-- unverified -->` by design: the hosted `/v1/execute` cannot return the `absences` ledger until the mthds SDK wire models can carry it (cross-repo hand-off item 1 in the tracker). When that lands, add an HTTP-arm conformance test (a PipeFunc/PipeCondition-only bundle whose optional output resolves absent) and flip the spec section to `> Verified by:`.
diff --git a/wip/optionals/optionals-design.md b/wip/optionals/optionals-design.md
new file mode 100644
index 000000000..a4d2dd860
--- /dev/null
+++ b/wip/optionals/optionals-design.md
@@ -0,0 +1,283 @@
+# Optionals in the MTHDS conceptual typing system — design choices
+
+Status: **design approved — all decisions D1–D11 decided.** Branch `feature/Optionals`; implementation plan in `wip/optionals/optionals-plan.md`. This document records the design space for adding optionality (`?`) to concept refs: each decision, the options considered, the chosen option, and its consequences. Decisions are numbered D1–D11 and summarized at the end.
+
+## 1. The problem
+
+MTHDS can express *how many* values flow through a slot (`contracts.PenaltyClause[]`) but not *whether* a value flows at all. Every declared input is unconditionally required and every declared output is unconditionally produced. Real methods constantly need "maybe there is no value", and today authors work around it in four ways, all bad:
+
+- **The empty-string hack.** A required structured field whose description says "empty string if full match" (seen verbatim in the cookbook's `rfp_qualifier`: `gap_description = { ..., description = "... empty string if partial or no match, empty string if full match", required = true }`). The type system says "always a value"; the prompt smuggles in the absence convention. Downstream pipes must know the convention to test for it.
+- **Hallucination pressure.** "Extract the penalty clause" with `output = "PenaltyClause"` gives the model no way to say "there isn't one" — the schema demands a value, so the model invents one. This is the single worst failure mode: it is silent and it is wrong.
+- **The invisible optional in `PipeCondition`.** An outcome mapped to `continue` returns working memory unchanged and produces *no* output — a de-facto optional that the type system cannot see. Validation runs all branches during dry run and never models the "produced nothing" path, so nothing downstream is checked against it.
+- **The cosmetic `@?`.** The prompt sigil `@?var` (→ `{% if var %}{{ var|tag("var") }}{% endif %}`) says "render only if present", but the variable detection still counts it as a required input, and the runtime gate (`validate_before_run`) still hard-fails when it is missing. The author can express optional *rendering* but not optional *presence*.
+
+And when a value genuinely is missing at run time, the user gets one of: a generic `PipeRunInputsError` (whose message says "Dry run of ..." even on live runs — pre-existing bug in `PipeAbstract.validate_before_run`), a `WorkingMemoryStuffNotFoundError` listing valid keys, or a Jinja `UndefinedError` from deep inside template rendering. None of them can say *why* the value is absent, because absence is not a modeled fact — it's just a missing dict key.
+
+Swift's optional model is the guide: declare with `?`, chain with `?`, force with `!`, all checked at compile time so `nil` never surprises you at run time. The translation problem is that MTHDS is declarative and high-level: there is no expression context in which to write `a?.b`, no `if let` statement. The unwrapping constructs must live where MTHDS authors already work — input/output declarations, prompts, and controllers — and the "compile time" is our validation engine.
+
+## 2. The shape at a glance
+
+Before the decision grind, here is the full design on the running example, so the taste is visible end-to-end:
+
+```toml
+[pipe.extract_penalty_clause]
+type = "PipeLLM"
+inputs = { contract = "contracts.Contract" }
+output = "contracts.PenaltyClause?" # NEW: this pipe may legitimately find nothing
+prompt = """Extract the penalty clause from this contract, if there is one.
+@contract"""
+
+[pipe.assess_penalty]
+type = "PipeLLM"
+inputs = { penalty_clause = "contracts.PenaltyClause" } # plain required input
+output = "contracts.PenaltyAssessment"
+prompt = """Assess the severity of this penalty clause. @penalty_clause"""
+
+[pipe.write_report]
+type = "PipeLLM"
+inputs = { contract = "contracts.Contract", assessment = "contracts.PenaltyAssessment?" } # NEW: absorbs absence
+output = "contracts.ContractReport"
+prompt = """Write a contract report.
+@contract
+@?assessment"""
+```
+
+Semantics, in order:
+
+- `extract_penalty_clause` declares an **optional output**. The runtime gives the LLM an explicit way to answer "no penalty clause" (a maybe-wrapper schema carrying a reason), and when it does, the slot is recorded as **absent with provenance** ("no penalty clause found in this contract") instead of holding a hallucinated value.
+- `assess_penalty` declares a plain required input fed by a maybe-absent slot. It is **lifted**: when the clause is absent, the pipe is *skipped* and its own output is recorded absent, provenance chaining back to the extraction. This is Swift's `a?.b` — write `?` once where absence enters; the chain short-circuits.
+- `write_report` declares an **optional input** (`?`): it runs regardless, and its prompt handles both arms via `@?assessment`. This is where the absence chain terminates — the explicit sink.
+- Validation proves the whole thing: every slot that may be absent must terminate in an explicit sink (an optional input, a forced input `!`, or an optional method output), or the bundle is invalid. Absence can never silently reach a consumer that doesn't expect it.
+- If the author instead writes `penalty_clause = "contracts.PenaltyClause!"`, absence becomes a **typed runtime failure** whose report names the variable, the consuming pipe, the producing pipe, and the model's stated reason for the absence.
+
+## 3. Three kinds of "nothing" — and which one this is
+
+The design must not blur three distinct notions, two of which already exist:
+
+| Notion | Where it lives | Existing? | Example |
+|---|---|---|---|
+| Field-level null | Inside a structured content: `required = false` → generated `Optional[T] = None` | Yes | `notes` field of an Invoice is `None` |
+| Plural emptiness | A list slot with zero items: `ListContent(items=[])` | Yes | Extraction found no requirements → `RFPRequirement[]` is empty |
+| Slot absence | A named slot in working memory has **no Stuff at all** | **New** | No penalty clause → no `PenaltyClause` stuff exists |
+
+`?` is about the third notion only. Concepts themselves stay total and singular — the existing multiplicity philosophy ("concepts are always singular; multiplicity is a property of the flow") extends naturally: **concepts are always present; presence is a property of the slot.** So `?`, like `[]`, is legal only where slots are declared (pipe `inputs` and `output`), never on concept definitions, `refines`, `concept_ref`/`item_concept_ref` structure fields, or package export lists.
+
+## 4. The Swift dictionary
+
+| Swift | MTHDS analog | Phase |
+|---|---|---|
+| `T?` declaration | `output = "X?"` / `inputs = { v = "X?" }` | 1 |
+| `nil` | Absent slot, recorded with provenance in working memory | 1 |
+| `a?.b.c` optional chaining | Lifting: a pipe with a plain required input fed by a maybe-absent slot is skipped, its output becomes absent (D3) | 1 |
+| `x!` force unwrap | `inputs = { v = "X!" }` → typed runtime error if absent | 1 |
+| Function taking `T?` | Optional input `X?`: pipe runs, handles absence itself (prompt guards) | 1 |
+| `if let` / `guard let` | `@?var` and `{% if var %}` in prompts; PipeCondition branching on presence | 1 (idiom), 2 (sugar) |
+| `?? default` | Input defaults / fallback values | 2 |
+| `try?` (error → nil) | `on_error = "absent"` on a pipe with optional output | 2 |
+| `compactMap` | PipeBatch whose inner pipe has optional output → absences dropped from result list | 1 or 2 |
+
+## 5. D1 — Grammar: where `?` and `!` may appear
+
+The suffix grammar becomes:
+
+```
+io_ref := concept_ref_or_code marker?
+marker := "[]" | "[" digits "]" | "?" | "!" # v1: markers are mutually exclusive
+```
+
+- `?` is legal on pipe **inputs and outputs** (blueprints and specs), and therefore on method signatures (a method is a pipe).
+- `!` is legal on **inputs only** — it is a use-site assertion, not a type. `!` on an output is meaningless (a producer doesn't unwrap anything) and is a grammar error.
+- In v1, `?` may not combine with `[]`/`[N]` — see D4. If we ever allow it, the order is fixed: multiplicity then presence (`X[]?`), read "the list slot may be absent".
+- `?` and `!` never appear in concept definitions, `refines`, structure-field `concept_ref`s, or `METHODS.toml` refs. Package-layer parsers (`qualified_ref` in pipelex, mthds-js, mthds-python) never see them.
+
+Consequences: the change concentrates in the multiplicity parser (`variable_multiplicity.py` — one regex plus its inline twin), the looser input-factory regex, and the naive `[`-splitter in `concepts/helpers.py` which today would silently pass a `?` through. While touching this file, fix the `MUTLIPLICITY_PATTERN` typo (rename to `MULTIPLICITY_PATTERN`).
+
+An alternative considered and rejected: expressing optionality as an expanded input table (`penalty = { concept = "PenaltyClause", optional = true }`) instead of a suffix. It avoids grammar changes but breaks symmetry with `[]`, is far more verbose, requires inventing an expanded input form that doesn't exist today, and gives outputs no way to be optional at all. The suffix is the right design; the expanded form can arrive later for defaults (D10) without conflict.
+
+Naming: `?` is the **optional marker**, pronounced "optional" (consistent with the `@?` sigil); `!` is the **force marker** (Swift-familiar). These are the terms the spec, docs, error messages, and skills use.
+
+## 6. D2 — What absence *is* at runtime
+
+Options:
+
+- **A. Absent key + absence ledger (chosen).** Absence stays what it mechanically is today — no Stuff under that name — but becomes a *recorded fact*: working memory gains a small side-table of `AbsenceRecord`s (`variable_name`, `producing_pipe`, `kind` = declared-absent | skipped | not-provided, `reason`, and the upstream record it chains to). A pipe that produces absence writes a record instead of a Stuff. `get_optional_stuff` and the run report can then distinguish "declared absent, here's why" from "never produced — that's a bug".
+- **B. Sentinel Stuff (present key, null content).** Make `Stuff.content` nullable or introduce an `AbsentContent`. Rejected: it breaks the strongest invariant in the content model (`Stuff.content` is never None, every accessor assumes it), leaks nulls into templates and the wire, and forces every operator to check for the sentinel. All cost, no benefit over A.
+
+Consequence of A: the wire format for run results needs an explicit representation of absence (D8), because "key missing from the output dict" must now be readable as "absent by design, reason attached" rather than ambiguous. The ledger is also what makes the failure UX (D9) possible — provenance is captured at the moment absence is produced, not reconstructed at failure time.
+
+## 7. D3 — Consuming an optional: the central decision
+
+A slot is **maybe-absent** if it is fed by an optional output, by a liftable (skippable) pipe, or is an optional method input. What may a downstream pipe's input declaration say about such a slot?
+
+The model is a trichotomy on the input marker:
+
+| Input form | Static rule when fed maybe-absent | Runtime behavior when actually absent | Swift analog |
+|---|---|---|---|
+| `v = "X"` (plain) | Allowed; the pipe becomes *liftable* and its output maybe-absent (taint propagates) | Pipe is **skipped**; absence recorded with provenance "skipped because v absent ← ..." | `a?.b` chaining |
+| `v = "X?"` (optional) | Allowed; taint **terminates** here | Pipe **runs**; templates see the var as absent and must guard (D7) | function taking `T?` |
+| `v = "X!"` (forced) | Allowed; taint terminates here | Pipe would run, but absence raises a typed error carrying the provenance chain | `x!` |
+
+And the safety theorem validation enforces: **every absence source must reach an explicit sink** (`?` input, `!` input, or a `?` on the enclosing method's declared output). If taint reaches a non-optional method output or otherwise escapes unhandled, validation fails with a dedicated error naming the source, the path, and the three ways to fix it.
+
+**Decision: implicit lifting**, for user-friendliness. Consequence: the mitigations listed below (liftable-pipe visibility in the validation report, skipped-node rendering in the graph, run-report skip reasons) are commitments of phase 1, not optional extras.
+
+Why implicit lifting for plain inputs (rather than Swift's "consuming an optional without unwrapping is a compile error"):
+
+- It matches Swift more closely than it first appears: in `a?.b.c.d`, you write `?` exactly once — at the point absence enters — and `.b.c.d` are *implicitly* conditional. The MTHDS equivalent of that single `?` is the optional output declaration upstream; the pipes in between are the `.b.c.d`.
+- It is what a high-level declarative language should do: "no penalty clause → no severity assessment" is the obviously-intended semantics, and demanding a marker on every intermediate pipe turns a one-decision design into annotation churn.
+- It keeps pipe signatures reusable. `assess_penalty : PenaltyClause → PenaltyAssessment` stays a total, non-optional pipe; the *flow context* lifts it. The same pipe used in a flow where the clause is guaranteed present is not polluted by optional markers it doesn't need. (This is precisely functorial lifting: the chain applies `(T → U)` to a `T?` and gets a `U?`.)
+- The escape hatches are exactly where Swift puts them: `!` to assert, `?` to absorb.
+
+The honest counter-argument: implicit skipping is action-at-a-distance. An author who *didn't intend* their pipe to be skippable might be surprised a step didn't run. Mitigations, in order of importance: (1) the taint can never appear out of nowhere — some upstream author wrote an explicit `?`, and the sink is also explicit, so the skip zone is bracketed by visible markers; (2) the validation report lists every liftable pipe ("may be skipped when X is absent"), so it is visible at build time, not just run time; (3) the run report and graph render skipped pipes distinctly (D8/D9). If we still find this too implicit in practice, the strict alternative below is a compatible tightening (it turns silently-allowed cases into errors), so starting permissive does not paint us into a corner — whereas starting strict and loosening later would churn every method written in between.
+
+Rejected alternatives:
+
+- **Strict-explicit (Swift-literal).** Plain `X` fed by maybe-absent = validation error; author must write `?`, `!`, or a new chain sigil on every consuming input. Consequence: mid-chain noise, and the "chain sigil" would have to be invented (a third marker meaning "skip me", distinct from `?` which means "run me") — three markers where the trichotomy needs two.
+- **Error-only (no skip at all).** `?` inputs and `!` are the only consumers; feeding maybe-absent into plain `X` is an error. Consequence: every pipe downstream of an extraction must guard its prompt for a value that, when absent, makes the pipe pointless. Authors would immediately ask for skip; PipeCondition + `continue` boilerplate would proliferate.
+
+One more rule: an input may declare `?` even when fed by a guaranteed-present slot (over-tolerance is allowed, exactly as Swift lets you pass a `T` where `T?` is expected). This matters for signature stability — a reusable pipe can declare tolerant inputs without caring what feeds it. `!` on a guaranteed slot is statically useless; it stays legal and is flagged through the `warnings` channel (D6).
+
+## 8. D4 — Optionals and plurals
+
+Should `X[]?` exist? The distinction it would encode is "the list step was skipped" vs "the list step ran and found nothing". Options:
+
+- **A. Forbid `?` on plural refs; absence of a plural normalizes to the empty list (chosen).** `[]` already has a perfectly good "nothing": empty. When a pipe with plural output is skipped by lifting, the runtime writes an **empty ListContent** (plus an absence-flavored note in the ledger for observability) rather than an absent slot. Taint stops there: downstream batch over it runs zero branches (already today's clean behavior), downstream prompts see an empty list.
+- **B. Allow `X[]?` as a distinct state.** Two kinds of nothing for every plural slot; every consumer must consider both; PipeBatch needs an absent-vs-empty policy; templates need to distinguish undefined from empty. High complexity, and the "skipped vs empty" signal is preserved anyway in the ledger and the graph without needing a type-level distinction.
+
+**Decision: option A** — `?` forbidden on plurals; the empty list does the job. It yields a pleasing symmetry: `?` is the absence story for singulars, `[]`-emptiness is the absence story for plurals, and the two never stack. Corollary: `!` on plural inputs is also forbidden (there is nothing to force).
+
+Bonus primitive that falls out: **PipeBatch whose inner pipe has optional output performs compaction** — absent branch results are dropped, so the batch output is `X[]` containing only found items (Swift's `compactMap`). The existing `build_or_skip`-inside-batch test pattern (condition routing rejects to `continue`) is exactly this, done by hand today; optionals make it typed. Whether compaction ships in phase 1 or 2 is a scoping call — the semantics should be fixed now.
+
+## 9. D5 — Producing absence: which pipes can declare `output = "X?"`
+
+Grammar-wise, allow `?` on any pipe's output (uniformity; over-claiming optionality is harmless and preserves signature evolution room — a method may declare `X?` to reserve the right to return absence later). Semantically, absence is actually *produced* in these ways:
+
+- **PipeCondition with special outcomes.** If any outcome maps to `continue` (or `default_outcome = "continue"` is reachable), the declared output MUST be `X?` — this makes today's invisible optional explicit, and validation finally models the no-output path. This is the cleanest immediate win of the whole feature. (Interaction with the required-main-stuff invariant: see §14.)
+- **PipeLLM / structured generation with a maybe-wrapper.** When `output = "X?"`, the response schema becomes a wrapper the model can decline through: conceptually `{ found: bool, reason_if_absent: str, value: X | null }` (instructor's `Maybe` pattern). `found = false` → absence record carrying the model's stated reason. This kills the hallucination pressure *and* harvests the reason that makes downstream error messages excellent. For non-structured text outputs (`Text?`), the same wrapper applies. The exact wrapper shape and prompt scaffolding is an implementation detail of the operator, invisible in the `.mthds` surface.
+- **Lifted (skipped) pipes** — absence propagates per D3, no declaration needed on the intermediate pipe itself (its declared output stays total; effective optionality is computed).
+- **Optional method inputs** — a caller may omit `v = "X?"` from `inputs.json` / the execute request; the slot starts as recorded-absent ("not provided by caller") instead of raising `PipeRunInputsError`.
+
+Note the asymmetry with intermediate pipes: *method-boundary* signatures (the top-level pipe's inputs/output, anything exported/signature-matched) must be explicit about optionality — that's the API contract, and `contract_match` canonicalization must compare presence markers. *Internal* step-to-step effective optionality is inferred. Explicit at boundaries, inferred locally — same balance Swift strikes between API signatures and local type inference.
+
+## 10. D6 — Static validation: the taint pass
+
+The validator gains an absence-propagation pass over each controller's dataflow, computing per-slot presence: `guaranteed` or `maybe-absent`, using exactly the bookkeeping that already exists (`PipeSequence.needed_inputs`'s `generated_outputs` set is where "produced upstream" is tracked; it gains a presence dimension). Checks, all static:
+
+- Optional output feeding the checks in D3's matrix (nothing to check — every arm is legal; the *pipe-level* results are recorded for reporting).
+- Taint reaching a non-optional method output, sequence output, or other boundary → **error** (`OPTIONAL_NOT_HANDLED`, a new `PipeValidationErrorType`), message naming the absence source, the propagation path, and the three fixes (`?` the output, `!` an input, absorb with a `?` input).
+- `continue`-reachable PipeCondition without `?` output → error (`OPTIONAL_OUTPUT_REQUIRED`).
+- `?`/`!` grammar misuse (on plurals, on outputs for `!`, on `refines`...) → error at blueprint parse (`OPTIONAL_MARKER_INVALID`).
+- Optional input referenced unguarded in a template → see D7.
+
+New error types flow automatically through the existing machinery (`build_validation_error_items` → agent CLI JSON/markdown, API 422, conformance) — the plumbing is already unified; we add enum values, fixtures, and conformance rows, not new channels.
+
+Two sub-decisions:
+
+- **Dry-run modeling.** v1: keep the dry run all-present (mock every input, including optional ones) and rely on the static taint pass for the absent arm. Alternative: dual-arm dry run (a second pass with all optionals absent) — better coverage of skip paths and template guards, but doubles dry-run cost and requires the skip machinery to run under DRY. Decision: static-only in v1, dual-arm as a follow-up if the static pass proves insufficient.
+- **Advisory channel.** Some findings are lints, not errors: `!` on a guaranteed slot, `?` output on a pipe that can never produce absence. The validation report is binary today; the only advisory precedent is `pending_signatures`/`is_runnable`. Options: (a) mirror that pattern with a dedicated `optionality_notes` list; (b) add a general `warnings` array to the report (protocol change, conformance impact, but the right long-term shape); (c) stay silent on lints in v1. **Decision: option (b)** — a general `warnings` array (same item shape as errors, never flips `is_valid`), bundled into the same protocol bump that adds the `optional` flag to IO contracts, so one bump covers both. Note the split with the *factual* liftable-pipe inventory ("pipe X may be skipped when Y is absent"), which is dataflow information, not a lint — it ships as structured data on the valid report (beside `pipe_io_contracts`), per the D3 implicit-lifting commitment.
+
+## 11. D7 — Prompts and templates: what the author sees
+
+When an optional input is absent, what does the Jinja context contain? Options:
+
+- **A. Nothing — the variable is undefined (chosen).** `{% if var %}` and `@?var` work (falsy), `{{ var }}` renders empty, and a *deep* unguarded use (`{{ var.amount }}`) raises — loud, which is what we want, but today it surfaces as a raw `Jinja2TemplateRenderError`. Pair with the static guard-lint below so authors rarely meet that error.
+- **B. A well-behaved `Absent` object** (falsy, chains safely, renders empty — a ChainableUndefined flavor). Everything renders silently; nothing ever fails. Rejected: silence hides bugs — an unguarded `{{ var.amount }}` producing empty prose *changes the prompt's meaning* without anyone noticing. Swift taste says absence misuse should be loud.
+
+The static companion (this is what makes A safe): validation lints that every template reference to a declared-optional input is **guarded** — reachable only inside `{% if var %}`-style guards, inline presence conditionals (`{{ 'present' if var is defined else 'absent' }}` — the §15 idiom, so the lint must recognize `is defined` tests as guards), or via `@?var`. The AST walk that already extracts required variables (`detect_jinja2_required_variables`) can classify guarded vs unguarded references. Unguarded use of an optional → validation error with a precise fix ("wrap in `{% if assessment %}` or use `@?assessment`").
+
+Two existing warts this design must fix in passing:
+
+- `@?var` currently still *registers the variable as required* (the detection walks both the `{% if %}` condition and body). Under optionals, `@?` on a declared-optional input must not mark it required — this finally makes `@?` mean what it says. `@?` on a *non*-optional input stays as today (cosmetic conditional rendering of a guaranteed value — still occasionally useful for falsy-but-present values, e.g. empty text).
+- The `None`-renders-as-`"None"` gotcha (field-level `None` produces the literal string `None` in prompts, and `|default()` doesn't catch `None`) is adjacent, field-level territory — out of scope for slot optionality, but worth a follow-up lint of its own since users will conflate the two kinds of nothing when debugging prompts.
+
+## 12. D8 — Wire, API, and graph surfaces
+
+- **IO contracts.** `PipeInputContract`/`PipeOutputContract` (and the protocol spec + mthds-js/mthds-python mirrors + `StuffSpecInfo` in mthds-ui) gain `optional: bool`. Neutral, MTHDS-brand naming — this is language surface, not runtime-specific, so no `pipelex_` prefixing anywhere on the wire.
+- **Run results.** An absent output must be *explicitly* absent on the wire, not just a missing key: the result carries the absence records (variable, producing pipe, kind, reason, provenance chain). A run whose main output is absent is a **success with an absent result**, not an error — clients branch on the structured field (presence), never on transport (same presentation-vs-contract rule as `/validate`'s `is_valid`).
+- **Execute requests.** Omitting an optional method input is legal; omitting a required one keeps today's failure but the error can now say "this input is required — the method also has optional inputs X, Y you may omit".
+- **Graph.** `GraphSpec` gains a `skipped` node state (alongside failed) and edges from optional outputs get a marker; mthds-ui renders skipped nodes/optional edges distinctly (dashed is the obvious treatment) and shows the absence reason in the detail panel. This is where "why did my workflow produce nothing?" gets answered visually.
+
+## 13. D9 — Failure UX and the error taxonomy
+
+New error classes (in the relevant `exceptions.py` modules, so docs pages, `type_uri`s, and RFC 7807 rendering come free):
+
+- `OptionalValueAbsentError` — the `!` failure. Carries: variable name, consuming pipe, the full provenance chain from the ledger, and the original reason. `error_domain = RUNTIME` (it is data-dependent — the method's assumption failed on this input data, not the caller's request shape), with a `user_action` explaining that the method force-unwraps (`!`) a value this input data doesn't contain.
+- The validation-side errors from D6 (`OPTIONAL_NOT_HANDLED` etc.) ride the existing `PipeValidationError` machinery.
+
+The concrete failure a user sees when a force-unwrap trips (markdown arm; the JSON/problem+json arm carries the same fields structured):
+
+> **Pipe `assess_penalty` requires `penalty_clause` to be present (declared `contracts.PenaltyClause!`), but it is absent.**
+> Absence origin: pipe `extract_penalty_clause` produced no value — "The contract contains no penalty or liquidated-damages clause."
+> Fix in the method: declare the input `contracts.PenaltyClause?` and guard the prompt, or let the step skip by removing `!`. Fix in the data: provide a contract that contains a penalty clause.
+
+Compare with today's equivalent (`Stuff 'penalty_clause' not found in working memory, valid keys are: [...]`). The delta is the entire point of the feature: absence with provenance turns "mysterious missing key" into "the model looked and found nothing, here is where and why".
+
+Success-path observability matters equally: the run report enumerates recorded absences and skipped pipes even when the run succeeds, so "my report doesn't mention penalties" is answerable without debugging.
+
+## 14. Interactions with merged and in-flight work
+
+- **Required-main-stuff invariant (PR #1014).** The invariant "a pipe run always delivers a main stuff" is enforced and test-pinned at every post-run boundary. Reconciliation: an `AbsenceRecord` satisfies the invariant as the terminal state of `main_stuff` when (and only when) the method output is declared `?` — the invariant generalizes to "main stuff is always *resolved*: a value or a recorded absence". Concrete consequences:
+ - **`continue` semantics change (breaking).** Current behavior: `continue` = pass-through-or-error (deliver the current main stuff; `PipeRunError` if there is none, live and dry-run alike), pinned by `test_pipe_condition_continue_delivery.py` including run-id stamping. Phase 1 replaces it with **`continue` = declared output absent, memory otherwise unchanged**: the runtime records an absence for the declared output instead of passing through or raising, the pinned tests are rewritten (not extended), and the dry-run all-special-outcomes guard becomes "legal iff the output is declared `?` → record absence". The pass-through runtime error becomes unreachable for validated bundles once `continue`-reachable ⇒ `?` output is statically enforced (D6's `OPTIONAL_OUTPUT_REQUIRED`).
+ - **Migration note — the pass-through capability disappears.** "If the value is already fine, `continue` and deliver it as-is" is expressible today. Under optionals the condition's output is absent instead; the previous value remains in memory under its own name, so downstream consumes it explicitly (`?` input + `@?` guard) — the ergonomic replacement is coalescing (`??`, D10), pulled forward into phase 2 for exactly this reason. The phase-1 docs and changelog must state the migration idiom explicitly (breaking).
+ - **The tightened boundaries are the phase-1 absence-arm checklist.** These sites now read the main stuff directly and raise when it is missing; each must learn the "resolved as absent" arm: the graph-tracer epilogue in `pipe_abstract.py` (`get_main_stuff()` direct), the delivery executor (typed and raw-transport arms both raise), the CLI run cores (bare, agent, agent-API), `otel_factory.py`, and `resolve_main_stuff_root_key` in `runtime_bridge/serialization.py` (raises `PipeJobError`).
+ - **Wire constraint: `main_stuff_name` stays required.** `PipelexPipeRunOutput` and `PipelexRunResultExecute` require `main_stuff_name: str` (fire-and-forget is a separate `PipelexPipeDispatchAck`). Do not re-optionalize it — that would undo #1014 and re-spread null guards. D8's shape: `main_stuff_name` keeps naming the declared output slot; the run result's absence records mark that slot absent; consumers branch on the structured absence record (presentation-vs-contract, as with `/validate`'s `is_valid`).
+ - **No resurrection of `PipeOutput.optional_main_stuff`** (deleted in #1014). The new accessor is tri-state *resolved* (Stuff | AbsenceRecord) — never `None` for a completed run.
+ - **Prose sweep:** the invariant wording ("a pipe run always delivers a main stuff") lives in docstrings and error messages across several files; phase 1 rewrites it to "always resolves its declared output: a value or a recorded absence".
+- **D11 — Absent branch results in PipeParallel's always-combine. Decision: absorption requires an optional attribute.** Since #1014, `PipeParallel` unconditionally combines its branch results into the declared `output` concept (`combined_output` deleted; untyped vehicle = `native.Composite`; static field/result-name/type compatibility checks in `validate_output_with_library`). Under optionals a branch may resolve absent (its pipe lifted, or a `?` output produced absence), so the combine needs a policy:
+ - **Structured output:** a structure field with `required = false` **absorbs** the absent branch as field-level `None` — slot absence converts to field-level null (§3's first kind of nothing) at the combine boundary, and taint terminates there. A **required** field fed by a maybe-absent branch is a **static validation error** (extends #1014's `validate_output_with_library` checks; part of the D6 taint pass), naming the branch, the field, and the two fixes (make the field optional, or sink the absence upstream).
+ - **`Composite` output:** absent components are **omitted** from the composite, with a ledger note for observability (compaction-flavored, mirroring D4's story for plurals). The component-wise transport encoding already tolerates omitted components, so no wire change.
+ - Tainting the whole parallel output on any absent branch was rejected: it discards exactly the partial results fan-out exists to collect.
+ - Whole-parallel lifting (the parallel's own *input* absent) is plain D3 and needs no new rule.
+- **`@?` required-variable detection** (D7) — the fix lands as part of this feature.
+- **Pre-existing wart to fix while in the area:** structure-field shorthand strings default to `required=True` while the expanded `ConceptStructureBlueprint` defaults `required=False` — opposite defaults for the same concept. Unrelated to slot optionality but adjacent enough to confuse; fix or at least document deliberately.
+- **Not in scope:** structure-field `?` sugar (e.g. `concept_ref = "X?"` meaning `required=false`). It would create two spellings for one thing and blur the slot-vs-field distinction §3 works hard to establish. Revisit only if field/slot unification ever becomes a goal.
+
+## 15. Later-phase candidates (design now, ship later)
+
+- **`try?` analog — `on_error = "absent"`.** A pipe attribute, legal only when the output is `?`: a failing pipe degrades to a recorded absence (kind = `failed`, reason = the error) instead of failing the run. Powerful resilience primitive (flaky enrichment step → absent, not dead run). Two hard requirements inherited from our error-handling principles: the underlying error must still be fully captured in the run report/trace (degraded success, never a swallowed failure), and terminal-failure webhooks/notifications semantics must be worked out so "degraded" is distinguishable from "clean". Scoping to *which* errors degrade (inference errors yes, config errors no?) is the main open design question.
+- **Coalescing — the `??`.** **Committed to phase 2** (it replaces the `continue` pass-through phase 1 broke — §13 migration note). Input-level defaults. For native concepts (Text, Number...) a literal default is easy (`v = { concept = "Text?", default = "N/A" }` — note this finally motivates an expanded input table form). For structured concepts, a default *pipe* (`fallback = "make_default_x"`) is the more MTHDS-native shape. Nothing in phase 1 blocks either form.
+- **Presence-branching sugar for PipeCondition.** The v1 idiom is `expression_template = "{{ 'present' if penalty_clause is defined else 'absent' }}"` with outcomes mapped on `present`/`absent`. If it proves common, add first-class sugar (e.g. `switch_on_presence = "penalty_clause"`). Idiom first, sugar on evidence.
+
+## 16. Blast radius
+
+Inside `pipelex/` (the bulk): grammar (`variable_multiplicity.py` + the input-factory regex + the `[`-splitter in `concepts/helpers.py`), `StuffSpec`/`NamedStuffSpec` gain the presence field, `InputStuffSpecs.required_names` splits required/declared, the three runtime miss-gates (`validate_before_run`, `SubPipe`, PipeCondition's branch check) learn the trichotomy, `PipeSequence.needed_inputs`/`generated_outputs` gain the taint dimension, WorkingMemory gains the absence ledger, PipeLLM gains the maybe-wrapper, mock seeding and the dry-run sweep learn optional slots, validation gains the new error types + template guard-lint, `contract_match` canonicalization compares markers, builder specs mirror blueprint validation, error classes + generated error pages, graph tracer `skipped` state, and docs.
+
+From the #1014 surfaces (§14): PipeCondition's `continue` arms (live + dry) and their pinned tests, PipeParallel's unconditional combine + its static compatibility validation (D11), and the tightened post-run boundary sites that must learn the resolved-as-absent arm — `pipe_abstract.py` tracer epilogue, `delivery_executor.py` (typed + raw), the CLI run cores, `otel_factory.py`, `resolve_main_stuff_root_key`/`payloads.py`/`pipeline_response.py`.
+
+Cross-repo (from the workspace survey):
+
+| Repo | Surfaces | Size |
+|---|---|---|
+| `mthds/` (spec) | Normative suffix table + concept-reference-syntax sections in `mthds-format.md`, per-pipe-type allow/forbid rules, `validation-rules.md`, a new optionality language guide beside `multiplicity.md`, reconcile `@?` docs, regenerate JSON schema | L |
+| `vscode-pipelex/` | `strip_concept_qualifiers` in the LSP (strips `[...]` but not `?` — a trailing `?` breaks resolution today) + tests, TextMate grammar regexes, bundled schema sync, `plxt` format preservation | M |
+| `mthds-ui/` | `StuffSpecInfo.optional`, Shiki grammar copy, skipped-node + optional-edge rendering, detail panels | M |
+| `mthds-plugins/` | Skill templates that teach the ref syntax (`shared/mthds-reference.md.j2`, build/inputs/fix skills) + regeneration; teach the trichotomy and `@?` pairing | M |
+| `conformance/` + `docs/specs/` | `PipeIOContract.optional` in the protocol spec, new validation-error categories in the locked contract, valid/invalid fixtures (`optional_not_handled.mthds` etc.), error-QA corpus entries | M |
+| `mthds-js` / `mthds-python` | Type mirrors (IO contracts, ValidationErrorItem); package-layer ref parsers untouched by design (D1) | S |
+| `pipelex-api/` | Pass-through; new absence fields in run results, docs | S |
+| JSON schemas | Regenerate only — `inputs`/`output` are unconstrained strings, already `?`-permissive | S |
+
+Sequencing note: per workspace rules, `docs/specs/` + `conformance/` move in the same change as the pipelex surface they verify; the mthds spec, tooling, and UI repos can follow the runtime release. The Required-main-stuff Phase 3 cross-repo sweep (release-gated: `combined_output` removal + `Composite` in the spec/schema/skills) touches the same mthds spec pages — land it before the optionals cross-repo wave to avoid colliding edits.
+
+## 17. Phasing
+
+- **Phase 1 — the language core (pipelex + spec/conformance).** D1 grammar, D2 ledger, D3 trichotomy + lifting, D4 plural rules, PipeCondition `continue` integration, optional method inputs, D6 taint validation + new error types, D7 template guard-lint + `@?` fix, D9 error classes + failure UX, IO-contract `optional` flag, run-result absence records, graph `skipped` state. This is a complete, useful feature on its own — even before LLM maybe-outputs, it legitimizes conditional flows and optional method inputs.
+- **Phase 2 — producing absence from models + coalescing.** PipeLLM/`Text?` maybe-wrapper with reasons, and `??` coalescing/defaults (pulled forward from phase 3: it is the ergonomic replacement for the `continue` pass-through that phase 1 broke — see the §13 migration note). Batch compaction shipped in phase 1 (Step C).
+- **Phase 3 — ergonomics.** `on_error = "absent"`, presence sugar, dual-arm dry run if warranted.
+- **Cross-repo wave** after the pipelex release: mthds spec pages, vscode/plxt, mthds-ui, skills.
+
+## 18. Decision summary
+
+| # | Decision | Chosen option |
+|---|---|---|
+| D1 | Grammar | `?` suffix on inputs+outputs; `!` inputs only; mutually exclusive with `[]` in v1; never on concept definitions/refines/fields |
+| D2 | Runtime absence | Absent key + provenance ledger; no null Stuff |
+| D3 | Consumption | Trichotomy `X` (lift/skip) / `X?` (absorb) / `X!` (force); every absence source must reach an explicit sink |
+| D4 | Plurals | `?` forbidden on plurals; skipped plural output normalizes to empty list |
+| D5 | Producers | `?` output allowed anywhere; `continue`-reachable conditions MUST declare it; LLM maybe-wrapper (phase 2); boundaries explicit, internals inferred |
+| D6 | Validation | Static taint pass, new `PipeValidationErrorType`s; dry run stays all-present in v1; general `warnings` array on the validation report |
+| D7 | Templates | Absent = undefined (loud on deep use) + static guard-lint; `@?` finally means optional |
+| D8 | Wire | `optional` on IO contracts; absence records in run results; absent main output = success, not error |
+| D9 | Errors | `OptionalValueAbsentError` with provenance chain; RFC 7807 + error pages via existing machinery |
+| D10 | Deferred | Coalescing (`??`) — phase 2 (the `continue` migration's ergonomic replacement); `on_error = "absent"` (`try?`), presence sugar — phase 3; semantics sketched now |
+| D11 | Parallel combine | Absent branch → optional structure field absorbs as field-`None` / `Composite` omits the component + ledger note; required field fed maybe-absent = static error |
diff --git a/wip/optionals/optionals-phase1-highlights.html b/wip/optionals/optionals-phase1-highlights.html
new file mode 100644
index 000000000..e0aea2dec
--- /dev/null
+++ b/wip/optionals/optionals-phase1-highlights.html
@@ -0,0 +1,455 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Optionals phase 1 — what we built and why it's solid</title>
+<style>
+ :root {
+ --bg: #ffffff;
+ --fg: #1c2230;
+ --muted: #5b6474;
+ --card: #f6f8fb;
+ --border: #dfe4ec;
+ --accent: #3556c7;
+ --accent-soft: #e8edfb;
+ --ok: #1e7d46;
+ --ok-soft: #e2f3e9;
+ --warn: #a3670a;
+ --warn-soft: #fdf1dc;
+ --err: #b3372f;
+ --err-soft: #fbe7e5;
+ --code-bg: #f0f2f7;
+ --diff-add: #e2f3e9;
+ --diff-del: #fbe7e5;
+ --diff-add-fg: #145c33;
+ --diff-del-fg: #8a2a24;
+ }
+ @media (prefers-color-scheme: dark) {
+ :root {
+ --bg: #14181f;
+ --fg: #e4e8ef;
+ --muted: #9aa4b5;
+ --card: #1c222c;
+ --border: #313a48;
+ --accent: #7d9bf0;
+ --accent-soft: #232d45;
+ --ok: #52c07e;
+ --ok-soft: #17301f;
+ --warn: #dfa447;
+ --warn-soft: #33270f;
+ --err: #e5766c;
+ --err-soft: #381c19;
+ --code-bg: #10141a;
+ --diff-add: #17301f;
+ --diff-del: #381c19;
+ --diff-add-fg: #7fd4a2;
+ --diff-del-fg: #eb9d96;
+ }
+ }
+ * { box-sizing: border-box; }
+ body {
+ margin: 0; padding: 0 1.25rem 4rem;
+ background: var(--bg); color: var(--fg);
+ font: 16px/1.6 -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+ }
+ main { max-width: 60rem; margin: 0 auto; }
+ h1 { font-size: 1.9rem; line-height: 1.25; margin: 2.5rem 0 0.4rem; }
+ h2 { font-size: 1.35rem; margin: 3rem 0 0.8rem; padding-top: 0.6rem; border-top: 1px solid var(--border); }
+ h3 { font-size: 1.05rem; margin: 1.6rem 0 0.5rem; }
+ p { margin: 0.6rem 0; }
+ .sub { color: var(--muted); margin-top: 0; }
+ a { color: var(--accent); }
+ code, pre { font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace; font-size: 0.85em; }
+ code { background: var(--code-bg); padding: 0.1em 0.35em; border-radius: 4px; }
+ pre {
+ background: var(--code-bg); border: 1px solid var(--border); border-radius: 8px;
+ padding: 0.8rem 1rem; overflow-x: auto; line-height: 1.5;
+ }
+ pre code { background: none; padding: 0; }
+ .tldr {
+ background: var(--accent-soft); border: 1px solid var(--accent);
+ border-radius: 10px; padding: 1rem 1.3rem; margin: 1.5rem 0;
+ }
+ .tldr h2 { border: none; margin: 0 0 0.5rem; padding: 0; font-size: 1.1rem; color: var(--accent); }
+ .tldr ul { margin: 0.5rem 0 0; padding-left: 1.2rem; }
+ .tldr li { margin: 0.25rem 0; }
+ .card {
+ background: var(--card); border: 1px solid var(--border);
+ border-radius: 10px; padding: 1rem 1.3rem; margin: 1rem 0;
+ }
+ .pill {
+ display: inline-block; font-size: 0.72rem; font-weight: 600; letter-spacing: 0.04em;
+ padding: 0.15em 0.6em; border-radius: 999px; vertical-align: 0.12em; text-transform: uppercase;
+ }
+ .pill.ok { background: var(--ok-soft); color: var(--ok); }
+ .pill.warn { background: var(--warn-soft); color: var(--warn); }
+ .pill.err { background: var(--err-soft); color: var(--err); }
+ .pill.info { background: var(--accent-soft); color: var(--accent); }
+ table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: 0.92rem; }
+ th, td { text-align: left; padding: 0.5rem 0.7rem; border: 1px solid var(--border); vertical-align: top; }
+ th { background: var(--card); }
+ .tablewrap { overflow-x: auto; }
+ figure { margin: 1.4rem 0; text-align: center; }
+ figure svg { max-width: 100%; height: auto; }
+ figcaption { color: var(--muted); font-size: 0.85rem; margin-top: 0.4rem; }
+ details {
+ border: 1px solid var(--border); border-radius: 8px;
+ padding: 0.6rem 1rem; margin: 0.8rem 0; background: var(--card);
+ }
+ details > summary { cursor: pointer; font-weight: 600; }
+ details[open] > summary { margin-bottom: 0.6rem; }
+ .diff { white-space: pre; }
+ .diff .add { display: block; background: var(--diff-add); color: var(--diff-add-fg); }
+ .diff .del { display: block; background: var(--diff-del); color: var(--diff-del-fg); }
+ .diff .ctx { display: block; color: var(--muted); }
+ /* CSS tabs */
+ .tabs { margin: 1.2rem 0; }
+ .tabs input[type="radio"] { display: none; }
+ .tabs label {
+ display: inline-block; padding: 0.45rem 1rem; cursor: pointer;
+ border: 1px solid var(--border); border-bottom: none;
+ border-radius: 8px 8px 0 0; background: var(--card); color: var(--muted);
+ font-weight: 600; font-size: 0.92rem; margin-right: 0.25rem;
+ }
+ .tabs .panel {
+ display: none; border: 1px solid var(--border); border-radius: 0 8px 8px 8px;
+ padding: 0.4rem 1.2rem 1rem; background: var(--bg);
+ }
+ #tab-cond:checked ~ label[for="tab-cond"],
+ #tab-par:checked ~ label[for="tab-par"],
+ #tab-batch:checked ~ label[for="tab-batch"] { background: var(--bg); color: var(--fg); border-bottom: 1px solid var(--bg); position: relative; top: 1px; }
+ #tab-cond:checked ~ .panel-cond,
+ #tab-par:checked ~ .panel-par,
+ #tab-batch:checked ~ .panel-batch { display: block; }
+ .footer { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid var(--border); color: var(--muted); font-size: 0.85rem; }
+ .kbd { font-weight: 700; }
+</style>
+</head>
+<body>
+<main>
+
+<h1>Optionals phase 1 — <code>?</code> and <code>!</code> land in MTHDS</h1>
+<p class="sub">Presence markers, an absence ledger with provenance, implicit lifting, and a static safety net — for co-developers reviewing <a href="https://github.com/Pipelex/pipelex/pull/1021">PR&nbsp;#1021</a>.</p>
+
+<div class="tldr">
+ <h2>TL;DR</h2>
+ <p>MTHDS methods can now say <em>"this slot may hold nothing"</em> — and both the validator and the runtime take that seriously. One PR (<a href="https://github.com/Pipelex/pipelex/pull/1021">Pipelex/pipelex#1021</a> → <code>dev</code>) delivers the whole of phase 1:</p>
+ <ul>
+ <li><strong>Grammar:</strong> <code>"PenaltyClause?"</code> (optional) and <code>"PenaltyClause!"</code> (force) on pipe inputs/outputs, beside the existing <code>[]</code> multiplicity.</li>
+ <li><strong>Runtime trichotomy:</strong> a pipe fed a recorded absence on a <em>plain</em> input is <strong>skipped</strong> (Swift-style <code>a?.b.c</code> chaining); a <code>?</code> input <strong>absorbs</strong> (the pipe runs both arms); a <code>!</code> input <strong>fails loudly</strong> with the full provenance chain.</li>
+ <li><strong>Absence is data, not a crash:</strong> a ledger records <em>why</em> each slot is empty (who skipped, what was never provided, which <code>continue</code> fired), and that record survives serialization, the API wire, CLI artifacts, and graph rendering.</li>
+ <li><strong>Static safety theorem:</strong> validation proves at author time that <em>every possible absence reaches an explicit sink</em> — no more "my workflow silently produced nothing".</li>
+ <li><strong>Breaking, on purpose:</strong> <code>PipeCondition</code>'s <code>continue</code> no longer passes a value through — it resolves the declared output as absent, killing the language's last invisible optional.</li>
+ </ul>
+ <p>Status: dev is merged in, CI is green, all review-bot findings are triaged (one real P1 caught &amp; fixed), and the PR awaits merge. Conformance coverage and protocol-spec sections are written but ride separate branches in the <code>conformance</code> and workspace-root repos (no PRs yet — they de-gate only once both conformance arms run an optionals-speaking pipelex).</p>
+</div>
+
+<h2>Why we did this</h2>
+<p>MTHDS had no way to say "no value here, and that's fine". That gap leaked everywhere:</p>
+<ul>
+ <li><strong>Empty-string hacks.</strong> Authors returned <code>""</code> or fabricated placeholder objects when an extraction found nothing — downstream pipes couldn't tell "empty" from "absent".</li>
+ <li><strong>Hallucination pressure.</strong> An LLM pipe <em>required</em> to produce a <code>PenaltyClause</code> from a contract that has none will invent one. The type system was actively encouraging hallucination.</li>
+ <li><strong>The invisible optional.</strong> <code>PipeCondition</code>'s <code>continue</code> outcome quietly passed the <em>previous</em> main stuff through — a no-output path the type system couldn't see, which crashed delivery, tracing, and telemetry in creative ways.</li>
+</ul>
+
+<h2>The language: two markers, strict rules</h2>
+<pre><code>[pipe.extract_penalty]
+type = "PipeLLM"
+description = "Find the penalty clause, if any"
+inputs = { contract = "Contract" }
+output = "PenaltyClause?" # ← may resolve absent: no clause found is a SUCCESS
+
+[pipe.assess_penalty]
+type = "PipeLLM"
+description = "Assess severity — handles both arms"
+inputs = { clause = "PenaltyClause?" } # ← absorbs: runs even when clause is absent
+prompt = """
+Assess the contract risk.
+@?clause # ← guarded block: rendered only when present
+"""
+
+[pipe.compute_fine]
+type = "PipeLLM"
+description = "Force-unwrap: we KNOW it's there by now"
+inputs = { clause = "PenaltyClause!" } # ← asserts presence; absence = typed error
+</code></pre>
+<p>Grammar misuse is rejected at parse time with a typed error (<code>optional_marker_invalid</code>): markers never combine with multiplicity (<code>X[]?</code> is illegal — an absent plural is just the empty list <code>[]</code>), <code>!</code> is inputs-only, and markers are banned on concept definitions, <code>refines</code>, structure fields, and package refs.</p>
+
+<h2>The runtime: one gate, three arms</h2>
+<figure>
+<svg viewBox="0 0 760 300" role="img" aria-label="Trichotomy diagram: an input slot fed a recorded absence takes one of three arms depending on the declared marker">
+ <defs>
+ <marker id="arr" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+ <path d="M0,0 L10,5 L0,10 z" fill="currentColor"/>
+ </marker>
+ </defs>
+ <g font-family="ui-monospace, Menlo, monospace" font-size="13">
+ <!-- source -->
+ <rect x="20" y="115" width="215" height="70" rx="10" fill="var(--card)" stroke="var(--border)"/>
+ <text x="127" y="143" text-anchor="middle" fill="var(--fg)" font-weight="bold">input slot holds a</text>
+ <text x="127" y="163" text-anchor="middle" fill="var(--fg)" font-weight="bold">recorded ABSENCE</text>
+ <!-- arrows -->
+ <g stroke="var(--muted)" fill="none" marker-end="url(#arr)" color="var(--muted)">
+ <path d="M235 135 C 300 90, 330 60, 390 55"/>
+ <path d="M235 150 C 300 150, 330 150, 390 150"/>
+ <path d="M235 165 C 300 210, 330 240, 390 245"/>
+ </g>
+ <text x="305" y="80" fill="var(--muted)">declared plain</text>
+ <text x="305" y="142" fill="var(--muted)">declared ?</text>
+ <text x="305" y="235" fill="var(--muted)">declared !</text>
+ <!-- SKIP arm -->
+ <rect x="395" y="25" width="340" height="62" rx="10" fill="var(--warn-soft)" stroke="var(--warn)"/>
+ <text x="415" y="50" fill="var(--warn)" font-weight="bold">SKIP (implicit lifting)</text>
+ <text x="415" y="70" fill="var(--fg)">pipe doesn't run; its output records absent,</text>
+ <text x="415" y="84" fill="var(--fg)" font-size="12">chaining provenance back to the origin</text>
+ <!-- RUN arm -->
+ <rect x="395" y="120" width="340" height="62" rx="10" fill="var(--ok-soft)" stroke="var(--ok)"/>
+ <text x="415" y="145" fill="var(--ok)" font-weight="bold">RUN (absorb)</text>
+ <text x="415" y="165" fill="var(--fg)">pipe runs; guarded templates take the</text>
+ <text x="415" y="179" fill="var(--fg)" font-size="12">absent arm (@?var, {% if var %})</text>
+ <!-- FAIL arm -->
+ <rect x="395" y="215" width="340" height="62" rx="10" fill="var(--err-soft)" stroke="var(--err)"/>
+ <text x="415" y="240" fill="var(--err)" font-weight="bold">FAIL (force)</text>
+ <text x="415" y="260" fill="var(--fg)">typed OptionalValueAbsentError naming the</text>
+ <text x="415" y="274" fill="var(--fg)" font-size="12">variable, both pipes, and the original reason</text>
+ </g>
+</svg>
+<figcaption>The trichotomy. Write <code>?</code> once where absence enters; every plain consumer downstream short-circuits automatically.</figcaption>
+</figure>
+
+<p>The skip arm is the headline: it makes absence <em>compose</em>. You don't annotate the whole chain — the chain lifts itself, and every skipped pipe's output records an absence that points back at the origin:</p>
+
+<figure>
+<svg viewBox="0 0 780 150" role="img" aria-label="Lift chain: source absent, steps A and B skip, step C declares its input optional and absorbs">
+ <defs>
+ <marker id="arr2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+ <path d="M0,0 L10,5 L0,10 z" fill="currentColor"/>
+ </marker>
+ </defs>
+ <g font-family="ui-monospace, Menlo, monospace" font-size="13">
+ <rect x="10" y="45" width="130" height="60" rx="10" fill="var(--err-soft)" stroke="var(--err)"/>
+ <text x="75" y="70" text-anchor="middle" fill="var(--fg)" font-weight="bold">source?</text>
+ <text x="75" y="90" text-anchor="middle" fill="var(--err)" font-size="12">not provided</text>
+
+ <g stroke="var(--muted)" fill="none" marker-end="url(#arr2)" color="var(--muted)">
+ <line x1="140" y1="75" x2="185" y2="75"/>
+ <line x1="330" y1="75" x2="375" y2="75"/>
+ <line x1="520" y1="75" x2="565" y2="75"/>
+ </g>
+
+ <rect x="190" y="45" width="140" height="60" rx="10" fill="var(--warn-soft)" stroke="var(--warn)" stroke-dasharray="6 4"/>
+ <text x="260" y="70" text-anchor="middle" fill="var(--fg)" font-weight="bold">pipe A</text>
+ <text x="260" y="90" text-anchor="middle" fill="var(--warn)" font-size="12">SKIPPED</text>
+
+ <rect x="380" y="45" width="140" height="60" rx="10" fill="var(--warn-soft)" stroke="var(--warn)" stroke-dasharray="6 4"/>
+ <text x="450" y="70" text-anchor="middle" fill="var(--fg)" font-weight="bold">pipe B</text>
+ <text x="450" y="90" text-anchor="middle" fill="var(--warn)" font-size="12">SKIPPED</text>
+
+ <rect x="570" y="45" width="195" height="60" rx="10" fill="var(--ok-soft)" stroke="var(--ok)"/>
+ <text x="667" y="70" text-anchor="middle" fill="var(--fg)" font-weight="bold">pipe C (b_out?)</text>
+ <text x="667" y="90" text-anchor="middle" fill="var(--ok)" font-size="12">RUNS — absorbs, delivers</text>
+
+ <text x="390" y="132" text-anchor="middle" fill="var(--muted)" font-size="12">each skip records an AbsenceRecord whose `upstream` chains to the previous one — provenance for free</text>
+ </g>
+</svg>
+<figcaption>A lift chain: like Swift's <code>a?.b.c</code>, the <code>?</code> at the boundary is written once; C is the explicit sink.</figcaption>
+</figure>
+
+<h3>Absence is a first-class record, never a null</h3>
+<p>There is no null <code>Stuff</code>. A slot with no value has an <code>AbsenceRecord</code> in the working memory's ledger: variable name, producing pipe, kind (<span class="pill warn">skipped</span> <span class="pill info">declared-absent</span> <span class="pill err">not-provided</span>), a human reason, and an <code>upstream</code> chain. Post-run reads use a tri-state accessor (<code>resolve_stuff</code> → value <em>or</em> record). A slot with <strong>neither</strong> a value <strong>nor</strong> a record is still a hard error: never-produced is a bug, not an absence.</p>
+
+<h2>The static safety net</h2>
+<p>The runtime story would be scary without the compile-time story. Validation now proves the <strong>absence safety theorem</strong>: a taint pass walks every controller's dataflow, computes per-slot presence (<em>guaranteed</em> vs <em>maybe-absent</em>), and rejects any flow where an absence could escape without an explicit sink.</p>
+
+<div class="tablewrap">
+<table>
+ <tr><th>Check</th><th>Severity</th><th>What it catches</th></tr>
+ <tr><td><code>optional_marker_invalid</code></td><td><span class="pill err">error</span></td><td>Grammar misuse: <code>X[]?</code>, <code>!</code> on outputs, markers on concepts/refines/fields.</td></tr>
+ <tr><td><code>optional_not_handled</code></td><td><span class="pill err">error</span></td><td>A maybe-absent slot ends a sequence (or feeds a condition) without the controller declaring a <code>?</code> output. Message names the origin, the propagation path, and three fixes.</td></tr>
+ <tr><td><code>optional_output_required</code></td><td><span class="pill err">error</span></td><td>A <code>continue</code>-reachable <code>PipeCondition</code> with a plain output — <code>continue</code> resolves the output absent, so a plain output would hide the no-output path. <strong>Breaking.</strong></td></tr>
+ <tr><td><code>optional_branch_required_field</code></td><td><span class="pill err">error</span></td><td>A required structure field of a <code>PipeParallel</code> output fed by a maybe-absent branch.</td></tr>
+ <tr><td><code>optional_input_unguarded</code></td><td><span class="pill err">error</span></td><td>A template reference to a <code>?</code> input without a guard (<code>@?var</code>, <code>{% if var %}</code>, inline <code>is defined</code>). Covers PipeLLM, PipeCompose, PipeCondition, PipeSearch, PipeImgGen templates.</td></tr>
+ <tr><td><code>optional_force_redundant</code></td><td><span class="pill warn">warning</span></td><td>A <code>!</code> whose slot is guaranteed present in <em>every</em> analyzed flow — dead assertion. Rides the new advisory <code>warnings</code> array (never flips <code>is_valid</code>), aggregated across flows so a <code>!</code> that matters anywhere is never flagged.</td></tr>
+</table>
+</div>
+
+<p>The valid report also gains a <code>liftable_pipes</code> inventory: every pipe that <em>may</em> be skipped, with its trigger slots and absence source. That's what makes implicit lifting acceptable — authors see "may be skipped when X is absent" at <code>/validate</code> time, not by surprise in production.</p>
+
+<h2>Controllers under absence</h2>
+<div class="tabs">
+ <input type="radio" name="ctrl" id="tab-cond" checked>
+ <input type="radio" name="ctrl" id="tab-par">
+ <input type="radio" name="ctrl" id="tab-batch">
+ <label for="tab-cond">PipeCondition</label>
+ <label for="tab-par">PipeParallel</label>
+ <label for="tab-batch">PipeBatch</label>
+
+ <div class="panel panel-cond">
+ <h3><code>continue</code> now means "no output" — honestly <span class="pill err">breaking</span></h3>
+ <p>Before: <code>continue</code> passed the current main stuff through, or errored when there was none — an untyped, invisible optional. Now: it records a <em>declared-absent</em> output (reason = the evaluated expression) and returns success, memory otherwise unchanged. The static rule forces such a condition to declare <code>output = "X?"</code>, so downstream code knows.</p>
+ <p><strong>Migration:</strong> the previous value stays in memory under its own name — consume it explicitly by that name (declared <code>?</code> where it may be absent). The ergonomic replacement for pass-through is the phase-2 coalescing operator (<code>??</code>).</p>
+ <p>Dry-run parity: an all-special-outcomes condition records the same absence — <em>except</em> a fail-only condition (every outcome maps to <code>fail</code>), which raises in dry run too, since every live path can only raise.</p>
+ </div>
+
+ <div class="panel panel-par">
+ <h3>Combine absorbs absence — by declared shape</h3>
+ <ul>
+ <li><strong><code>Composite</code> output:</strong> the absent component is simply omitted (a ledger note keeps observability).</li>
+ <li><strong>Structured output, non-required field:</strong> the absence becomes the <em>field default</em> (<code>None</code> unless the author declared another default — per-field coalescing avant la lettre).</li>
+ <li><strong>Structured output, required field:</strong> a typed runtime error naming the branch, the field, and the fixes — but this shape is already rejected statically by <code>optional_branch_required_field</code>, so the runtime guard is a belt-and-suspenders backstop.</li>
+ </ul>
+ </div>
+
+ <div class="panel panel-batch">
+ <h3>Batch compacts</h3>
+ <p>Absent branch results are dropped from the aggregated list. Batching a "keep or skip" condition over N items yields only the kept results — a list cannot hold a hole, so compaction is the only lossless behavior. The empty list <code>[]</code> is the whole absence story for plurals: a lifted pipe with a plural output writes a real empty <code>ListContent</code> (plus a ledger note), and taint stops there.</p>
+ </div>
+</div>
+
+<h2>First-class on every surface</h2>
+<p>A run whose declared <code>?</code> output resolves absent is a <strong>success</strong> everywhere — no surface treats it as an error or crashes on it:</p>
+<ul>
+ <li><strong>Wire:</strong> <code>main_stuff_name</code> still names the declared slot; consumers branch on the absence record carried in the serialized memory (<code>absences</code>), never on transport. The ledger round-trips cross-process — a recorded absence no longer degrades to a hard miss after an API/worker hop.</li>
+ <li><strong>Contracts:</strong> <code>PipeInputContract</code>/<code>PipeOutputContract</code> gain <code>optional: bool</code>, so tooling can see optionality without parsing the grammar.</li>
+ <li><strong>CLI:</strong> <code>pipelex run</code> prints an explicit absence document; <code>--save-main-stuff</code> writes a <code>main_stuff.json</code> carrying <code>"absent": true</code> plus the provenance chain (and a markdown twin). The agent CLI returns the absence document as its result payload.</li>
+ <li><strong>Graphs:</strong> nodes gain a <span class="pill warn">skipped</span> status with a <code>skip_reason</code>, and data edges fed by <code>?</code> outputs carry <code>optional: true</code> — "why did my workflow produce nothing?" is now answerable from the execution graph.</li>
+ <li><strong>Telemetry:</strong> OTel/Langfuse capture absence documents for both inputs and outputs.</li>
+</ul>
+
+<figure>
+<svg viewBox="0 0 700 170" role="img" aria-label="Execution graph fragment showing a skipped node and an optional data edge">
+ <defs>
+ <marker id="arr3" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+ <path d="M0,0 L10,5 L0,10 z" fill="currentColor"/>
+ </marker>
+ </defs>
+ <g font-family="ui-monospace, Menlo, monospace" font-size="13">
+ <rect x="15" y="55" width="150" height="60" rx="10" fill="var(--card)" stroke="var(--ok)"/>
+ <text x="90" y="80" text-anchor="middle" fill="var(--fg)" font-weight="bold">extract_penalty</text>
+ <text x="90" y="100" text-anchor="middle" fill="var(--ok)" font-size="12">success</text>
+
+ <g color="var(--muted)">
+ <line x1="165" y1="85" x2="255" y2="85" stroke="var(--muted)" stroke-dasharray="5 5" marker-end="url(#arr3)"/>
+ <text x="210" y="72" text-anchor="middle" fill="var(--muted)" font-size="11">optional: true</text>
+ </g>
+
+ <rect x="260" y="55" width="150" height="60" rx="10" fill="var(--warn-soft)" stroke="var(--warn)" stroke-dasharray="6 4"/>
+ <text x="335" y="80" text-anchor="middle" fill="var(--fg)" font-weight="bold">compute_fine</text>
+ <text x="335" y="100" text-anchor="middle" fill="var(--warn)" font-size="12">skipped</text>
+ <text x="335" y="135" text-anchor="middle" fill="var(--muted)" font-size="11">skip_reason: "input 'clause' is absent"</text>
+
+ <g color="var(--muted)">
+ <line x1="410" y1="85" x2="500" y2="85" stroke="var(--muted)" marker-end="url(#arr3)"/>
+ </g>
+
+ <rect x="505" y="55" width="175" height="60" rx="10" fill="var(--card)" stroke="var(--ok)"/>
+ <text x="592" y="80" text-anchor="middle" fill="var(--fg)" font-weight="bold">summarize (b_out?)</text>
+ <text x="592" y="100" text-anchor="middle" fill="var(--ok)" font-size="12">success</text>
+ </g>
+</svg>
+<figcaption>Graph rendering of the same story: dashed optional edge, dashed skipped node with its reason.</figcaption>
+</figure>
+
+<h2>The critical changes, as minimal diffs</h2>
+
+<details open>
+<summary>1 — The lift gate (<code>pipe_abstract.py</code>): the trichotomy in four lines</summary>
+<pre class="diff"><code><span class="ctx"> presence_scan = await self.validate_before_run(...) # raises for `!` (force) and for hard misses</span>
+<span class="add">+ if presence_scan.liftable:</span>
+<span class="add">+ # Implicit lifting (D3): a plain input fed a recorded absence skips the pipe;</span>
+<span class="add">+ # its output is recorded absent with provenance chaining to the input's record.</span>
+<span class="add">+ pipe_output = self._make_lifted_output(..., pipe_run_params=pipe_run_params)</span>
+<span class="ctx"> else:</span>
+<span class="ctx"> pipe_output = await self.live_run_pipe(...) # or dry — unchanged</span></code></pre>
+<p>Inside <code>_make_lifted_output</code>, plurality is resolved with the exact rule the run path uses (declared multiplicity + invocation-level override), so a plural lift writes the real empty list downstream consumers were promised — singulars record the absence:</p>
+<pre class="diff"><code><span class="add">+ multiplicity_resolution = output_multiplicity_to_apply(</span>
+<span class="add">+ base_multiplicity=self.output.multiplicity,</span>
+<span class="add">+ override_multiplicity=pipe_run_params.output_multiplicity,</span>
+<span class="add">+ )</span>
+<span class="add">+ if multiplicity_resolution.is_multiple_outputs_enabled:</span>
+<span class="add">+ working_memory.set_new_main_stuff(empty_list_stuff, name=output_name) # [] IS the plural absence</span>
+<span class="add">+ working_memory.record_absence(skip_record) # observability note beside the value</span>
+<span class="add">+ else:</span>
+<span class="add">+ working_memory.record_new_main_absence(skip_record) # the slot's resolution</span></code></pre>
+</details>
+
+<details>
+<summary>2 — The ledger's two write modes (<code>working_memory.py</code>): NOTE vs RESOLUTION</summary>
+<p>This invariant pair is what keeps values and records from fighting. <code>set_stuff</code> pops any record under the written name (value supersedes record); the resolution write is its exact mirror.</p>
+<pre class="diff"><code><span class="add">+ def record_absence(self, record): # NOTE: a value under the name is kept and wins</span>
+<span class="add">+ self.absences[record.variable_name] = record</span>
+<span class="add">+</span>
+<span class="add">+ def record_resolved_absence(self, record): # RESOLUTION: mirror of set_stuff's invariant —</span>
+<span class="add">+ self.remove_stuff(name=record.variable_name) # a stale value must not outrank</span>
+<span class="add">+ self.remove_alias(alias=record.variable_name) # the fresh absence</span>
+<span class="add">+ self.record_absence(record)</span></code></pre>
+</details>
+
+<details>
+<summary>3 — <code>continue</code> resolves the output absent (<code>pipe_condition.py</code>) <span class="pill err">breaking</span></summary>
+<pre class="diff"><code><span class="del">- # 'continue': deliver the current main stuff by passing it through</span>
+<span class="del">- # (raise PipeRunError when there is no main stuff to pass through)</span>
+<span class="add">+ # 'continue': the declared output resolves absent — memory otherwise unchanged</span>
+<span class="add">+ record = AbsenceRecord(</span>
+<span class="add">+ variable_name=output_name or MAIN_STUFF_NAME,</span>
+<span class="add">+ kind=AbsenceKind.DECLARED_ABSENT,</span>
+<span class="add">+ reason=f"expression '{evaluated_expression}' resolved to 'continue'",</span>
+<span class="add">+ producing_pipe=self.code,</span>
+<span class="add">+ )</span>
+<span class="add">+ working_memory.record_new_main_absence(record)</span></code></pre>
+<p>Paired with the static rule: a <code>continue</code>-reachable condition <em>must</em> declare <code>output = "X?"</code> (<code>optional_output_required</code>), so the no-output path is visible in the types.</p>
+</details>
+
+<details>
+<summary>4 — Parallel combine under absence (<code>pipe_parallel.py</code>), incl. the P1 the bots caught</summary>
+<pre class="diff"><code><span class="ctx"> branch_resolved = pipe_output.working_memory.resolve_main_stuff() # tri-state, never raises on absence</span>
+<span class="add">+ if isinstance(branch_resolved, AbsenceRecord):</span>
+<span class="add">+ if not issubclass(structure_class, CompositeContent):</span>
+<span class="add">+ field_info = structure_class.model_fields.get(sub_pipe_output_name)</span>
+<span class="add">+ if field_info is not None and field_info.is_required():</span>
+<span class="add">+ raise PipeRunError(...) # required field fed a hole — statically unreachable backstop</span>
+<span class="add">+ if self.add_each_output: # ← review round 1: parent slots exist ONLY under</span>
+<span class="add">+ working_memory.record_resolved_absence(branch_record) # add_each_output, same as present branches</span>
+<span class="add">+ continue # omitted from the combine; non-required field → field default</span></code></pre>
+<p>The <code>if self.add_each_output</code> gate is the fix for the one P1 of the review round: without it, an absent branch wrote a resolved absence into the <em>parent</em> memory even when branch results aren't parent slots — deleting an unrelated same-named value. Present branches always had the gate; now both arms publish under the same rule.</p>
+</details>
+
+<details>
+<summary>Bonus — the load-bearing one-liner: <code>StuffArtefact.__bool__</code></summary>
+<p>The guard idiom the lint recommends (<code>{% if var %}</code>, which <code>@?var</code> expands to) crashed on a <em>present</em> singular value: Jinja2's truth test fell through to the artefact's list-only <code>__len__</code> and raised <code>TypeError</code>. Found by our own new contract fixture failing its dry run.</p>
+<pre class="diff"><code><span class="add">+ def __bool__(self) -&gt; bool:</span>
+<span class="add">+ if isinstance(self._content, ListContent):</span>
+<span class="add">+ return bool(self._content.items) # empty list is falsy (D4)</span>
+<span class="add">+ return True # a present singular is truthy</span></code></pre>
+</details>
+
+<h2>Why you can trust it</h2>
+<ul>
+ <li><strong>One commit per checkpoint, six checkpoints (A–F).</strong> Each ran the full gate set (<code>make agent-check</code> + full <code>make agent-test</code>) plus a <em>context-free</em> cold code review of the step diff by a fresh reviewer with no access to the plan or rationale. Findings were triaged every time: correctness fixed in-step, design tradeoffs recorded as deferred docs, over-engineering simplified.</li>
+ <li><strong>TDD throughout:</strong> red tests first for every step — lift chains with provenance assertions, force-failure UX (the message must name the variable, both pipes, and the reason), plural normalization, hydration round-trips, end-to-end absent-main-output runs over every surface.</li>
+ <li><strong>Review round 1 on the PR:</strong> 13 bot findings (greptile, cubic, Codex), each adversarially verified before acting — 9 confirmed and fixed (headline: the parallel parent-memory clobber P1), 1 dead-code deletion, 1 refuted with evidence, 1 already-deliberate design decision. All threads answered with commit citations; CI fully green.</li>
+ <li><strong>Ground-truth fixtures:</strong> every conformance fixture's expectations were generated by running it against the real build before being written down — no guessed error categories.</li>
+</ul>
+
+<h2>What's deliberately NOT in phase 1</h2>
+<div class="card">
+<ul>
+ <li><strong>Coalescing <code>??</code></strong> — the ergonomic replacement for <code>continue</code> pass-through. Phase 2. (Non-required parallel fields absorbing as their <em>declared default</em> is a small preview of it.)</li>
+ <li><strong>The LLM maybe-wrapper</strong> — making <code>?</code> on operator outputs change the LLM contract ("return nothing rather than invent"). Phase 2; over-claiming <code>?</code> today is deliberately legal as evolution room, and deliberately not linted.</li>
+ <li><strong>Cross-repo wave</strong> — mthds wire models carrying the <code>absences</code> ledger (which unblocks the hosted-API absent arm), mthds-ui rendering skipped nodes/optional edges, pipelex-api surfacing <code>warnings</code> + <code>optional</code> contract flags. Gated behind other in-flight work; no PRs yet.</li>
+ <li><strong>Conformance de-gating</strong> — the suite's optionals coverage is written and validated live against this branch, but stays probe-skipped (<code>OPTIONALS_PENDING_FEATURE</code>) until <em>both</em> conformance arms run an optionals-speaking pipelex (the HTTP arm pins the released <code>pipelex==0.37.0</code>).</li>
+</ul>
+</div>
+
+<h2>Links</h2>
+<ul>
+ <li><a href="https://github.com/Pipelex/pipelex/pull/1021">Pipelex/pipelex#1021</a> — the phase-1 PR (feature/Optionals → dev). Everything above lands there.</li>
+ <li>Conformance coverage and protocol-spec sections are committed on side branches of the <code>conformance</code> and workspace-root repos — PRs will follow once their merge vehicles are chosen.</li>
+</ul>
+
+<p class="footer">Optionals phase 1 · feature/Optionals · generated 2026-07-05 · lives in <code>wip/optionals/</code> (work-in-progress notes, not release docs)</p>
+
+</main>
+</body>
+</html>
diff --git a/wip/optionals/optionals-phase1-tracker.md b/wip/optionals/optionals-phase1-tracker.md
new file mode 100644
index 000000000..9f5f78be7
--- /dev/null
+++ b/wip/optionals/optionals-phase1-tracker.md
@@ -0,0 +1,144 @@
+# Optionals phase 1 — tracker (CLOSED — phase 1 complete)
+
+> Archived from `TODOS.md` at the worktree root into `wip/optionals/` now that phase 1 is done. See [README.md](README.md) for current track status and what comes next.
+
+**Phase 1 is COMPLETE** (all steps A–F done, FINAL CHECKPOINT cleared — see the checkpoint log). This tracker is kept as the feature's record; the only live follow-ups are the hand-off notes in "Cold-start context" and `wip/optionals/deferred-step-f-notes.md`.
+
+Branch `feature/Optionals` (worktree `_optionals`). This file is the single live tracker: progress checkboxes, checkpoint log, decisions/deviations. Reference docs: `wip/optionals/optionals-design.md` (decisions D1–D11, all decided) and `wip/optionals/optionals-plan.md` (the narrative plan this tracker implements — steps A–F map one-to-one).
+
+Working rules: TDD (red tests first per step), `make agent-check` after every code change, full `make agent-test` at each checkpoint, **one commit per checkpoint**, no backward compatibility (breaking changes go in the changelog under `[Unreleased]`).
+
+## Cold-start context (update at every checkpoint)
+
+- **Status: PHASE 1 COMPLETE — Step F DONE, FINAL CHECKPOINT cleared.** This tracker is CLOSED; nothing remains in-repo for phase 1. Step F landed across three repos: (1) `_optionals` `b66bd751a` — the "Understanding Optionality" docs guide (both mkdocs nav lists), PipeBatch compaction section, PipeLLM `@?` sigil docs, run-CLI absence-artifact docs, changelog Documentation entry, `wip/optionals/deferred-step-f-notes.md`; schema + error pages regenerated with zero diff (already current). (2) `conformance` `9acd10e` + triage `39c5b4e` (on `feature/config-dialect-fixture`) — invalid fixtures + `CATEGORIZED_VALIDATION_ITEMS` rows for all five optionals error types, `test_validate_optionals.py` on both arms (`optional` contract flags, `liftable_pipes`, `warnings`), six QA-corpus cases; ALL gated via `OPTIONALS_PENDING_FEATURE` / `pending_feature_skip_reason` because the sibling venvs run released pipelex without optionals — gate-removal checklist in the Step-F deferred notes. (3) workspace root `c28380d` + triage `caea84e` (on `feature/SDK-and-starter`) — protocol-spec sections (optional IO contracts + liftable pipes, advisory warnings, optionality validation errors, absence records on run artifacts) with bidirectional Verified-by links; `make check-spec-links` green. Nothing pushed in any repo.
+- **Previous status (Step E):** Step E DONE (Checkpoint E cleared — SHAs in the checkpoint log). An absent result is a first-class success everywhere in-repo. Protocol bump landed in one motion: `optional: bool` on `PipeInputContract`/`PipeOutputContract` (from `presence.is_optional`), `warnings: list[ValidationErrorItem]` on `PipelexValidationReport` (assembled in `build_validation_report`, populated by the new `pipeline/optionality_warnings.py::build_optionality_warnings` in `validate_in_process`, the agent-CLI whole-bundle validate cores, the bare CLI (`_echo_optionality_warnings`, bundle arm + `--all`), and the three bundle-context builder `validate_ops` twins — all through the single-walk collector `pipeline/controller_taint.py::collect_controller_taint_analyses`, which also feeds `build_liftable_pipes`; markdown gets a "Warnings" section in `validation_render.py`). The useless-`!` lint (`OPTIONAL_FORCE_REDUNDANT`, advisory-only enum value): the taint walks classify every `!` consumption (asserting = slot tainted, redundant = slot guaranteed) via `scan_taint_triggers` + new `ForceConsumptionInfo` carriers on both analyses; the builder aggregates per (pipe_ref, variable) ACROSS flows — one asserting flow silences redundant observations, and a `!` never observed in any flow is never flagged. Boundaries: `resolve_main_stuff_root_key` returns the declared slot name for a recorded main absence (message swept to the new invariant), `hydrate_working_memory` round-trips the `absences` ledger (malformed → loud `PipeJobError`), OTel captures absence documents on inputs and outputs, bare CLI prints/saves the absence artifact (no viewer), agent CLI returns the absence document as the compact result. Shared artifact renderers: `core/memory/absence_render.py::build_absence_payload/build_absence_json/build_absence_markdown/build_absence_html` (delivery executor + both CLI run cores render through them). Graph: `NodeSpec.skip_reason` + `skipped` status via the new `on_pipe_end_skipped` tracer hook (protocol + noop + manager + tracer + `pipe_end_skipped` event + assembler), skip reason from the shared `PipeAbstract._make_skip_reason`; a skipped pipe with a PLURAL output registers its real empty-list output spec (and plural companion slots) so downstream DATA edges resolve; `EdgeSpec.optional` on data edges whose producer registered `extra={"optional": True}` on its output IOSpec (set in the run epilogue from `output.presence.is_optional`; both builders share `graphspec.py::output_digest_is_optional`). **Bonus fix (load-bearing):** `StuffArtefact.__bool__` — Jinja2 truth tests (`{% if var %}`, `@?`) on a PRESENT singular value crashed via the list-only `__len__` fallback; now truthy, with empty lists falsy per D4. End-to-end pin: `test_absent_main_output_surfaces.py` (execute → response slot name, provenance chain to NOT_PROVIDED origin, graph skipped nodes + serialized).
+- **Next action:** none in-repo — hand off. (a) **Release**: cut the pipelex release carrying optionals, then run the conformance gate-removal checklist (`wip/optionals/deferred-step-f-notes.md`) and `make validate-error-qa`. (b) **Cross-repo wave** (after the Required-main-stuff Phase 3 sweep): the three hand-off notes below. (c) **Phase 2**: the LLM maybe-wrapper + `??` coalescing (design doc §phase 2).
+- **Cross-repo hand-off notes (accumulating for the wave):** (1) mthds-python `DictWorkingMemoryAbstract` must gain the `absences` ledger (extra=forbid blocks it today) — then `_run_core_api.py`'s gated absent arm can land (comment in code marks the spot); (2) mthds-ui renders `skipped` nodes + `optional` edges + `skip_reason` detail; (3) pipelex-api response models surface `warnings` + contract `optional` flags; (4) syntax highlighting of `?`/`!` markers is broken in both grammar copies — ungated, can land before the release; handoffs live at the workspace root: `../wip/optionals/handoff-vscode-pipelex-optional-markers.md` (TextMate grammar source + semantic tokens + Rust `strip_concept_qualifiers` goto/hover bug) and `../wip/optionals/handoff-mthds-ui-optional-markers.md` (shiki grammar + themes + FUNCTIONAL `conceptRefs.ts` parse-to-null bug that drops marked refs from static graphs).
+- **Open questions for Louis:** none blocking. Standing flagged compromise: `continue` pass-through breaks in phase 1 while its ergonomic replacement (`??` coalescing) is phase 2 — restructure only if Louis asks.
+
+## Checkpoint protocol — MANDATORY STOP at every CHECKPOINT
+
+At each checkpoint the agent MUST stop forward progress and run this sequence, in order. No rolling into the next step with the checkpoint half-done.
+
+1. **Verify:** full `make agent-check` + `make agent-test` green (`make tb` too if config files were touched). Fix failures before anything else.
+2. **Commit:** one commit for the step (changelog entry included when the step is user-visible/breaking).
+3. **Update this file:** tick the boxes, fill the checkpoint log row (SHA), and refresh "Cold-start context" — decisions taken, deviations from the plan, open questions, exact next action — so a brand-new session can resume efficiently from this file alone. If a design decision changed, update `wip/optionals/optionals-design.md`/`wip/optionals/optionals-plan.md` too.
+4. **Fan out a context-free code review:** spawn a **Sonnet-5 sub-agent with NO inherited context** (general-purpose agent, `model: sonnet` — never a fork) whose prompt is ONLY a pointer to the changes under review plus the instruction to run the `/code-review` skill. Example prompt: *"In /Users/lchoquel/repos/Pipelex/_optionals, run the /code-review skill on `git diff <previous-checkpoint-SHA>..HEAD`. Report findings."* Do **not** pass the plan, the design doc, the rationale, or your own conclusions — the reviewer judges the code cold. Review bar: clean solid software, not over-engineering.
+5. **Triage findings:** correctness bugs → fix now (amend or follow-up commit, re-run gates). Design-tradeoff findings → do not reflexively apply; record in a deferred-items doc under `wip/optionals/` and move on. Over-engineering findings → simplify now.
+6. Only then proceed to the next step.
+
+## Prep
+
+- [x] Commit this tracker (+ staged old-TODOS deletion). Record the SHA as the Step-A review base in the checkpoint log. *(Turned out already done: the tracker landed in `eace1d607`.)*
+
+## Step A — grammar and carriers (parse `?` / `!`, no behavior change)
+
+- [x] Red tests first: parser unit tests for every marker × multiplicity combination; blueprint + spec accept/reject fixtures.
+- [x] Extend the marker grammar (D1) in `pipelex/core/pipes/variable_multiplicity.py` (+ its inline twin), the looser regex in `input_stuff_specs_factory.py`, and the naive `[`-splitter in `concepts/helpers.py`. Fix the `MUTLIPLICITY_PATTERN` typo while there.
+- [x] `StuffSpec`/`NamedStuffSpec` gain the presence marker (a `PresenceMarker` StrEnum: plain / optional / force); `InputStuffSpecs.required_names` splits into required vs declared.
+- [x] Blueprint-parse-time grammar errors (`OPTIONAL_MARKER_INVALID`): `?` on plurals, `!` on outputs or plurals, markers on concept definitions / `refines` / structure-field refs / package refs (D1, D4).
+- [x] Builder specs mirror the blueprint rules (`to_blueprint()` passes markers through); `contract_match` canonicalization compares presence markers (D5).
+- [x] **CHECKPOINT A** — markers parse, validate, and round-trip; zero runtime behavior change; run the full checkpoint protocol above.
+
+## Step B — absence at runtime (ledger + trichotomy + lifting)
+
+- [x] Red tests first: lift-skip chain with provenance, absorb, force-failure UX (assert the message names variable, pipes, and reason), plural normalization, optional method inputs, run-report absence enumeration. All tests sink absence before the method boundary (a downstream `?` input) — end-to-end absent-main-output coverage is Step E's.
+- [x] WorkingMemory gains the absence ledger (D2): `AbsenceRecord` (`variable_name`, `producing_pipe`, `kind` = declared-absent | skipped | not-provided, `reason`, upstream chain). New tri-state resolved accessor (Stuff | AbsenceRecord) for post-run reads — do **not** resurrect `optional_main_stuff` (design §14).
+- [x] The three runtime miss-gates learn the trichotomy (D3): `PipeAbstract.validate_before_run` (fix the "Dry run of ..." message bug on the live path), `SubPipe`, PipeCondition's branch check. Plain input fed absence → **skip** (lift) + record skipped-absence for the pipe's output with provenance; `?` input → run; `!` input absent → `OptionalValueAbsentError` (D9: new error class in the relevant `exceptions.py`, provenance chain, RFC 7807 + error page via `pipelex-dev generate-error-pages`).
+- [x] The per-pipe graph-tracer epilogue in `pipe_abstract.py` (success path reads `working_memory.get_main_stuff()` directly) learns the resolved-as-absent arm **here, not in Step E** — it fires on every pipe run, so the first skipped pipe would crash it.
+- [x] Skipped plural output normalizes to empty `ListContent` + ledger note (D4); taint stops.
+- [x] Optional method inputs (D5): omitting a `?` input in `inputs.json` / the execute request → not-provided record instead of `PipeRunInputsError`; the required-input error message gains the "these optional inputs may be omitted" hint.
+- [x] Mock seeding / dry-run sweep learn optional slots (dry run stays all-present per D6): mock seeding already enumerates every declared input including `?` ones (all-present); pinned by test, and not-provided records are only written for slots the mock pass left empty.
+- [x] **CHECKPOINT B** — absence exists, propagates, and fails loudly only through `!`; run the full checkpoint protocol.
+
+## Step C — controllers: `continue` replacement + parallel combine (D11)
+
+- [x] Red tests first: continue-absent (live + dry), parallel with absorbed/omitted/required-absent branches, batch compaction if shipped.
+- [x] **PipeCondition `continue`** (design §14, breaking): replace pass-through-or-error with "declared output absent, memory otherwise unchanged" — record the absence (kind = declared-absent, reason = the evaluated expression/outcome), stamp no main stuff, return success. Same in dry run (the all-special-outcomes guard becomes: legal iff output declared `?`). Rewrite `test_pipe_condition_continue_delivery.py` to pin the new semantics (keep the run-id stamping assertions).
+- [x] These tests exercise the delivery boundary, so `delivery_executor.py` (typed + raw arms: absent main output delivers an explicit absence artifact, not an error) lands **here, not in Step E**.
+- [x] Document the migration idiom (previous value stays in memory under its own name; coalescing is the phase-2 answer) in docs + changelog (breaking).
+- [x] **PipeParallel combine under absence** (D11): absent branch result → structured output field with `required = false` absorbs as field-`None`; `Composite` output omits the component + ledger note. Runtime guard raises a typed error if a required field's branch is absent (statically unreachable after Step D, but the runtime must not feed `combine_stuffs` a hole). **Checkpoint-B review pinned the entry points:** `pipe_parallel.py` `_run_branches_and_combine` (`pipe_output.main_stuff` — the raising accessor) crashes today on a lifted branch (`WorkingMemoryStuffNotFoundError`, reproduced); `pipe_batch.py` has the identical shape at its branch-collect loop and in `_dry_run_controller_pipe`. Both must move to the tri-state `resolve_main_stuff()` with the D11 policy.
+- [x] PipeBatch compaction (D4 bonus): absent branch results dropped from the aggregated list — SHIPPED here (it fell out of the tri-state collect loop naturally; no `?`-output gating needed at runtime since a list cannot hold a hole).
+- [x] **CHECKPOINT C** — controllers speak absence; the #1014 `continue` tests replaced; run the full checkpoint protocol.
+
+## Step D — static validation: the taint pass + template lint
+
+- [x] Red tests first: taint fixtures (valid + each error type), guard-lint accept/reject, `@?` detection.
+- [x] Absence-propagation pass over controller dataflow (D6): per-slot presence computed (`guaranteed` / `maybe-absent`) by walk methods on the controllers (`PipeSequence.analyze_taint`, `PipeParallel.analyze_branch_taint`); shared carriers + presence-resolution helper in `pipe_controllers/absence_taint.py`.
+- [x] New `PipeValidationErrorType`s: `OPTIONAL_NOT_HANDLED` (taint reaches a non-optional boundary — message names source, path, three fixes), `OPTIONAL_OUTPUT_REQUIRED` (`continue`-reachable condition without `?` output), `OPTIONAL_BRANCH_REQUIRED_FIELD` (the D11 static error, extending `PipeParallel.validate_output_with_library`).
+- [x] Template guard-lint (D7): `OPTIONAL_INPUT_UNGUARDED` via the new guard walker (`tools/jinja2/jinja2_optional_guards.py`) + shared `lint_optional_input_guards` helper, wired into PipeLLM prompts/system prompts, PipeCompose templates, and PipeCondition expressions. `@?` fix: declared-optional inputs are no longer presence-required by the SubPipe / PipeCondition miss-gates.
+- [x] Liftable-pipe inventory as structured data on the valid report (`liftable_pipes` beside `pipe_io_contracts`), built by `pipeline/liftable_pipes.py` from the same taint analyses.
+- [x] **CHECKPOINT D** — validation proves the safety theorem ("every absence source reaches an explicit sink"); full checkpoint protocol run (see the checkpoint log).
+
+## Step E — wire, boundaries, and graph
+
+- [x] Red tests first: end-to-end absent-main-output run over each surface (execute response, CLI output cores, telemetry, hydration round-trip), graph skipped-state serialization, contracts/warnings, redundant-`!` lint.
+- [x] Protocol bump, one motion (D6 + D8): `optional: bool` on `PipeInputContract`/`PipeOutputContract`, the general `warnings` array on the validation report (same item shape as errors, never flips `is_valid`), absence records on run results. Neutral MTHDS naming, no `pipelex_` prefixes.
+- [x] Populate `warnings` with the useless-`!` lint (`optional_force_redundant`). The over-claimed-`?` lint is deliberately **not** emitted in phase 1 (D5 blesses over-claiming as evolution room; phase 2 makes `?` on operator outputs meaningful).
+- [x] The remaining post-run boundaries learn the resolved-as-absent arm: `resolve_main_stuff_root_key` (absent arm returns the declared slot name), `hydrate_working_memory` (absences ledger round-trips, malformed record fails loud), `otel_factory` (inputs AND outputs capture), bare CLI run core (pretty print + `--save-main-stuff` absence artifact + `--save-csv` message), agent CLI run core (absence document as result payload). Shared renderers in `core/memory/absence_render.py`; delivery executor refactored onto them. **Agent-API run core (`_run_core_api.py`) stays gated on the cross-repo mthds bump** — the SDK wire models (`DictWorkingMemoryAbstract`, extra=forbid, root+aliases only) cannot carry absences yet; noted in code, hand-off item for the cross-repo wave.
+- [x] Graph (D8): `NodeSpec` gains `skip_reason`, lifted pipes end their node `skipped` (new tracer hook `on_pipe_end_skipped` + `pipe_end_skipped` trace event + assembler handler), `EdgeSpec.optional` marks data edges fed by `?`-declared outputs (producer output IOSpec `extra["optional"]`). (mthds-ui rendering is the cross-repo wave.)
+- [x] Sweep the invariant prose to "a pipe run always resolves its declared output: a value or a recorded absence".
+- [x] **CHECKPOINT E** — an absent result is a first-class success everywhere in-repo; full checkpoint protocol run (see the checkpoint log).
+
+## Step F — specs, conformance, docs, release prep
+
+- [x] `docs/specs/` protocol spec: `optional` flag, `warnings` array, absence records, new validation-error categories; matching `conformance/` fixtures (`optional_not_handled.mthds` etc.) + error-QA corpus rows; `make check-spec-links` green (run in `conformance/`).
+- [x] Regenerate the MTHDS JSON schema (`pipelex-dev generate-mthds-schema`) and error pages; docs: an optionality guide beside `multiplicity.md`'s pipelex counterpart, pipe-page updates (PipeCondition, PipeParallel, PipeBatch), `@?` docs reconciled. *(Both regenerators produced zero diff — already current since Steps B/D; the schema carries no concept-ref patterns, so the marker grammar needs no schema change.)*
+- [x] Changelog under `[Unreleased]`: breaking — `continue` semantics, `?`/`!` grammar, protocol additions. *(Already complete from Steps A–E; Step F added the Documentation entry.)*
+- [x] Full `make agent-test` + `make tb`.
+- [x] **FINAL CHECKPOINT** — phase 1 complete; run the full checkpoint protocol. Hand off: cross-repo wave (after the Required-main-stuff Phase 3 sweep) and phase 2 (LLM maybe-wrapper). Fold or close this tracker and `wip/optionals/optionals-plan.md`. *(Closed — completion banners on this tracker and the plan doc; design doc stays as reference.)*
+
+## Checkpoint log
+
+| Checkpoint | Commit SHA | Review verdict | Notes |
+|---|---|---|---|
+| Prep | `eace1d607` | n/a | Tracker was committed alongside the plan update; serves as the Step-A review base. |
+| A | `ae4ace79e` | Clean — no findings at reporting confidence | Cold Sonnet review of `eace1d607..HEAD`. Two sub-threshold doc-in-code notes applied in the follow-up commit (PipeSpec authoring-surface descriptions teach `?`/`!`; enum comment covers all three trigger cases); third note (factory doesn't re-enforce mutual exclusion) = deliberate layering, see deviations log. |
+| B | `6c258e2ba` | 4 findings, all triaged | Cold Sonnet review of `80c4e56f6..HEAD`. (1)+(2) PipeParallel/PipeBatch crash (`get_main_stuff` raise) on a lifted branch — CONFIRMED reachable, but it is exactly the already-planned Step C work (D11 combine-under-absence + batch compaction); pinned explicitly in the Step C bullets. (3) delivery executor generic error on an unsunk lift chain — the plan's explicit Step C boundary (delivery arms land there). (4) duplicated miss-gate check → hoisted to `WorkingMemory.list_missing_names` in the checkpoint commit. |
+| C | `0df759bd6` | 10 findings: 1 bug fixed, rest triaged | Cold Sonnet review of `7d2ed6bf4..HEAD`. (1) **Stale-value masking — CONFIRMED bug, fixed in the checkpoint commit:** recording an absence under a name that already held a `Stuff` left the stale value winning `resolve_stuff`; fix = new `record_resolved_absence` (removes value + alias under the name — the mirror of `set_stuff`'s value-supersedes-record), used by `record_new_main_absence` and the parallel branch arm; `record_absence` stays the pure NOTE write (D4 plural note depends on it). Regression tests added. (2)–(4) OTel + three CLI run cores raise on absent main — already-planned Step E work, now pinned as REACHABLE in the Step E bullet (Checkpoint-B precedent). (5) `hydrate_working_memory` drops the absences ledger — added to the Step E bullet (was missing from the checklist). (6) non-`None` field default absorbs as the default — deliberate (proto-coalescing); changelog + docs wording corrected to say "field default". (7)–(9) dedup/nesting tradeoffs → `wip/optionals/deferred-absence-handling-dedup.md`. (10) informational. |
+| D | `f3d58d104` (+ triage follow-up commit) | 10 findings: 8 fixed, 2 deferred | Cold Sonnet review of `2d4211420..HEAD`. **Fixed in the triage commit:** (1) lifted `add_each_output` parallel left branch slots neither-value-nor-record at runtime while the static pass blessed the flow — new `PipeAbstract.lifted_companion_slots()` hook + `_make_lifted_output` resolves each companion (singular → SKIPPED record with provenance, plural → empty list per D4); the sequence walk's lifted-parallel arm now keys on the same hook (one source of truth); end-to-end static+runtime agreement test added. (2) walker skipped subtrees under unbuildable paths (`{{ x[0].f }}`) → now recurses inward. (3)+(7) `{% set %}` handling rebuilt as sequential body-walking: read-before-set is flagged, nested sets shadow correctly. (4) `Jinja2DetectVariablesError` leaked raw from construction → lint skips unparseable templates (parsing gates own syntax errors). (5) PipeSearch + PipeImgGen (prompt + negative_prompt) wired into the guard-lint. (6) `and` short-circuit recognized (`{% if var and var.attr %}` clean). (8) `variable_names` now carries `SlotTaint.origin_slot_name`, not prose. (10) duplicated taint bookkeeping factored into `scan_taint_triggers` + `is_plural_step_result` in `absence_taint.py`. **Deferred → `wip/optionals/deferred-step-d-review-tradeoffs.md`:** (9) cross-package liftable inventory scoping (decide at the Step E/F protocol bump), taint-analysis recompute/double-parse efficiency, `_build_full_path` duplication, inverted-guard shapes. |
+| E | `2f55da612` (+ checkpoint commit) | 10 findings: 1 bug + 5 consistency/cleanup fixed, 4 deferred | Cold Sonnet review of `26738d440..HEAD`. **Fixed in the checkpoint commit:** (1) **lifted-plural producer registration — CONFIRMED regression, fixed:** the skipped early-return never registered the lifted pipe's output IOSpec, but a lifted PLURAL output writes a real empty-list Stuff (D4) that downstream pipes consume — its digest never entered either graph builder's producer map, silently dropping the DATA edge. Fix: `on_pipe_end_skipped` (protocol/tracer/manager/event/assembler) now carries `output_spec` + `output_concept_data` and registers them like a success output; the epilogue computes the spec on both arms; plural companion slots register via `register_controller_output`. Unit tests on both builders. (3) useless-`!` lint was unreachable from the human `pipelex validate` → `_echo_optionality_warnings` on the bundle arm + `--all` sweep. (4) builder `validate_ops` twins drifted → `warnings` added to the three bundle-context ops. (7) taint walk ran twice per validate → shared `collect_controller_taint_analyses` pass feeds both projections (`pipelex/pipeline/controller_taint.py`); both builders now take analyses. (8) `_is_optional_output_digest` duplicated → shared `output_digest_is_optional` in `graphspec.py`. (10) absence-artifact serializer divergence → `build_absence_json`/`build_absence_html` in `absence_render.py`, all three call sites routed through them. (9) missing docs → graph-tracing page (SKIPPED + skip_reason + optional edges) and agent-cli page (validate `warnings` shape vs init/doctor warnings shape). **Deferred → `wip/optionals/deferred-step-e-review-tradeoffs.md`:** (2) API-runner absent arm (cross-repo gated, already in hand-off notes), (5) builder runner_code codegen crashes on absent optional output (pre-existing, builder-layer pass), (6) `\|length` on singular still raises (D4 keeps length a list concept; candidate guard-lint extension), + reviewer's below-cut latent facade gap and pre-existing no-span-on-lifted-leaf. |
+| F (final) | `b66bd751a` (+ triage commits: conformance `39c5b4e`, workspace root `caea84e`, `_optionals` checkpoint commit) | docs review: 5 findings, all doc-accuracy — fixed/deferred; conformance+spec review: 8 findings — 2 blockers fixed, rest fixed/mitigated | Two cold Sonnet reviews (docs diff `65e50f6a3..HEAD`; conformance `436d037..HEAD` + workspace-root spec diff). **Docs review:** run.md "Absent main output" over-claimed vs the Step-E code (no `main_stuff.html` on the absent arm; `--save-csv` exits 1 with an explicit error, consistent with its non-flat-list arm; console prints the reason only, provenance chain lives in `main_stuff.md`) → paragraph rewritten to match reality; PipeBatch.md claimed dropped branches ledger an absence record — they only log → doc corrected + ledger-parity question deferred (`deferred-step-f-notes.md`); EOF newline added; the new optionality guide verified fully clean. **Conformance review:** (1)+(2) CONFIRMED blockers — the `pending_feature_skip_reason` gate ran AFTER each arm's hard assertions, so `optional_output_required` (new semantics on OLD syntax → a pre-optionals runtime says VALID) hard-failed both arms → gate now decided first, with a lenient `_observed_validation_errors` probe on the CLI arm; (3) gating logic had zero unit coverage (it live-skips everywhere until release) → `runtime_supports_case` valid-probe + fallback tests and a new `test_validation_contract_gate.py` harness module; (4) spec hardcoded counts → count-free wording; (5) probe-skips greppability — already mitigated (both modules import + cite `OPTIONALS_PENDING_FEATURE`; removal checklist names them); (6) duplicated hand-rolled warning assertions → shared `REDUNDANT_FORCE_WARNING_ITEM` through `assert_categorized_validation_item` on both arms; (7) stale `pending_runtime_feature` field docstring → documents both probe paths; (8) two TestClasses in one module → HTTP-arm valid coverage split per spec section (`test_validate_warnings.py`), probe shared via conftest. **Bonus:** `CONFORMANCE_CLI_<NAME>` env override added (precedent `CONFORMANCE_PIPELEX_PYTHON`) and used to validate ALL gated CLI-arm coverage LIVE against this worktree — every row asserts and passes, zero skips. Also unblocked a local-env drift: the main checkout moved to `release/v0.37.0` mid-session (release cut) and stale untracked `conformance/.pipelex/pipelex.toml` carried `temporal_dynamodb`, breaking the pinned CLI's boot — section stripped (backup kept), mirroring the June ~/.pipelex strip. All gates re-run green after triage: conformance agent-check + FULL agent-test (exit 0, gates skip correctly) + check-spec-links; pipelex docs-check. |
+
+## Decisions & deviations log
+
+(fill at checkpoints — anything done differently from `wip/optionals/optionals-plan.md`, with why)
+
+- **Step A — parser is combo-permissive; validators own the mutual-exclusion rule.** `parse_concept_with_multiplicity` accepts `X[]?` (multiplicity + marker) and returns both; the D1 v1 rule (markers mutually exclusive with multiplicity) is enforced at the blueprint/spec layer where the input-vs-output context is known, raising the typed `OPTIONAL_MARKER_INVALID`. Rejecting in the parser would have degraded those cases to untyped syntax errors.
+- **Step A — `OPTIONAL_MARKER_INVALID` is a `PipeValidationErrorType` raised as `PipeValidationError` inside blueprint validators**, riding the existing pydantic-unwrap machinery (`extract_wrapped_pipe_validation_error`) exactly like `BATCH_ITEM_NAME_COLLISION`. No categorizer change needed.
+- **Step A — the naive `[`-splitter was renamed** `strip_multiplicity_from_concept_ref_or_code` → `strip_markers_from_concept_ref_or_code` (it now strips presence markers too; the old name would have lied). Internal helper, single caller (`pipe_factory`).
+- **Step A — runtime gates pinned to `declared_names`.** The three `pipe_abstract` enumeration sites + `pipe_extract.required_variables` now use `declared_names` (all inputs), so an input declared `?` is still enforced-present at run time in Step A — exact current behavior preserved; Step B flips the gates to the trichotomy.
+- **Step A — derived batch-list needs stay presence-plain.** In `pipe_parallel`/`pipe_sequence`, the computed list-input specs that override `multiplicity=True` (batch `input_list_name` derivation) do not carry the inner spec's presence (a plural + marker combo is exactly what D4 forbids); all pass-through `add_stuff_spec` sites carry `presence=stuff_spec.presence`.
+- **Step A — two stale fixtures updated**, since a trailing `!` is now legal grammar: `test_pipe_blueprint.py` invalid-syntax cases switched to genuinely malformed specs (`Invalid@Format`, `Invalid@Concept`).
+- **Step B — the lift gate reads the pipe's OWN declared presence, not the needed-aggregation.** A controller's `needed_inputs()` carries its children's markers (a sequence over a step needing `X` plain aggregates `X` plain even when the sequence declares `X?`). Lifting on the aggregated marker would skip the whole controller instead of letting its steps lift. `_scan_input_presence` therefore prefers `self.inputs.root[name].presence` (the boundary contract, D5 "boundaries explicit") and falls back to the needed spec's marker. Consequence: whole-controller lifting still happens when the author declares the input plain at the controller boundary — which is the intended semantics.
+- **Step B — the declared-vs-needed controller input check no longer compares presence markers.** With presence on `StuffSpec`, the old full-model `!=` would have rejected exactly the legal D3 shapes (controller `X?` over child-needed `X` plain) with an empty mismatch summary. The check now compares concept + multiplicity explicitly; presence relationships across the boundary are all legal per the trichotomy (Step D's taint pass reasons about them).
+- **Step B — `validate_before_run` returns the presence scan** (`InputPresenceScan`: missing / forced-absent / liftable) instead of `None`; it owns all three raise cases (force → `OptionalValueAbsentError`, missing-no-record → `PipeRunInputsError` with the run-mode-correct label + optional-inputs hint) and skips resource checks + per-pipe `_validate_before_run` when the pipe is about to be lifted (pipe-specific validation would inspect inputs the pipe never consumes — e.g. PipeCondition's expression render would crash on the absent var).
+- **Step B — value-supersedes-record invariant:** `set_stuff` pops the absence record under the written name; `set_new_main_stuff` also pops the positional `main_stuff` record. The D4 plural note is written AFTER the empty list so it survives as an observability note (stuff wins in `resolve_stuff`).
+- **Step B — lifting applies in dry mode too** (uniform gate): records never exist in today's dry sweeps (mock seeding is all-present per D6), so this is unreachable until Step C's continue-under-dry work, which will pin the dry semantics.
+- **Step B — sequence test builds pipes in code** (`get_pipe_library().add_new_pipe`) instead of an `.mthds` fixture: a bundle declaring `PipeFunc`s can't validate standalone (the functions aren't registered in the validation hook's process).
+- **Step C — dry-run `continue` parity keys on "no mapped pipes", not "no main stuff".** The dry sweep runs every mapped outcome, so when `pipe_dependencies()` is non-empty the mapped pipes' stamped outputs stand (the continue arm is Step D's static concern). Only an all-special-outcomes condition records the declared-absent output in dry — including over a pre-existing main stuff, mirroring the live supersede rule.
+- **Step C — batch compaction shipped (D4 bonus), ungated.** The tri-state collect loop drops any absent branch result — no check that the inner output is declared `?`. Rationale: a `ListContent` cannot hold a hole, so compaction is the only lossless runtime behavior; whether the author *may* rely on it is Step D's static story. The old xfail `test_pipe_condition_continue_output_type` ("identifies a bug") is un-xfailed and pins the fixed semantics: live = compacted list, dry = every mapped outcome modeled (one mock item per batch item).
+- **Step C — parallel ledger note is re-keyed to the branch result name.** `_make_lifted_output` records the branch absence under the sub-pipe's `output_name` already; the combine defensively re-keys via `model_copy(update={"variable_name": ...})` if a record ever arrives under another name, so the parent ledger always speaks in result names.
+- **Step C — delivery absence artifact shape:** `main_stuff.json` = `{"absent": true, ...AbsenceRecord dump...}` (explicit discriminator so consumers can tell an absence doc from a value dump), `main_stuff.md` = human-readable summary + provenance chain, `main_stuff.html` = escaped `<pre>` of the JSON. `main_stuff_viewer.html` is NOT produced for absences (nothing to view). Raw arm reads the serialized `absences` ledger; a malformed raw record degrades to the loud contract-violation error.
+- **Step C — PipeCondition docs page was stale** (`pipe_map` / `default_pipe_code` — fields that don't exist in the language; it's `outcomes` / `default_outcome`, and `default_outcome` is required). Fixed while documenting the new `continue` semantics + migration idiom on that page.
+- **Step C (checkpoint triage) — the ledger API now distinguishes NOTE from RESOLUTION.** `record_absence` = pure ledger note beside a possible value (D4 plural note); `record_resolved_absence` = the slot's resolution, removing any stale value/alias under the name (mirror of `set_stuff`'s value-supersedes-record). `record_new_main_absence` and the parallel branch arm go through the resolution form. Found by the Checkpoint C cold review (stale value outranked a fresh absence under a colliding name).
+- **Step C (checkpoint triage) — non-required structured fields absorb absence as their DEFAULT, not hardcoded `None`.** Deliberate: an author-declared default on an optional field is per-field coalescing avant la lettre (phase-2 `??` preview). Docs + changelog say "field default (`None` unless the author declared another default)".
+- **Step D — the taint pass is a separate walk, not a presence dimension inside `needed_inputs`.** The plan wording ("`needed_inputs`/`generated_outputs` gain the presence dimension") would have entangled taint state with the input-aggregation used by unrelated validation. Instead: `PipeSequence.analyze_taint()` / `PipeParallel.analyze_branch_taint()` re-use `needed_inputs()` per step and track presence in their own `slot_taints` map — same bookkeeping, isolated concern, and the liftable-pipe inventory reuses the identical analyses.
+- **Step D — `OPTIONAL_OUTPUT_REQUIRED` fires at pipe construction** (`validate_output_static`, inside the pydantic model validator, wrapped like `OPTIONAL_MARKER_INVALID`), not at library validation: outcome map and output declaration are both local to the pipe, so the earliest gate wins.
+- **Step D — guard-lint coverage extended beyond the plan's "prompts":** PipeLLM (prompt + system_prompt), PipeCompose (template mode), PipeCondition (expression), PipeSearch (prompt), PipeImgGen (prompt + negative_prompt) — every pipe rendering authored templates over its inputs. PipeCompose construct mode composes structured fields, not authored templates: not linted.
+- **Step E — the agent-API run core's absent arm is GATED on the cross-repo mthds bump.** The mthds SDK wire models (`DictWorkingMemoryAbstract`: root + aliases, `extra="forbid"`) cannot carry the absences ledger, so `_run_core_api.py` keeps its contract-violation raise (message swept to the new invariant wording) with a comment marking the gate. Everything else on the §14 checklist landed.
+- **Step E — `warnings` also ride the agent-CLI whole-bundle validate surfaces**, not just the protocol report: `validate_bundle_core` and `validate_pipe_in_bundle_core` attach `warnings` (dumped `exclude_none`, mirroring the error-items projection); `validate_pipe_core` (no bundle context) does not. Rationale: the lint's cross-flow aggregation needs the whole loaded bundle, which those surfaces have.
+- **Step E — redundant-`!` aggregation is deliberately conservative.** A warning fires only when the (pipe, variable) has ≥1 redundant observation and 0 asserting observations across ALL analyzed flows; a `!` input on a pipe referenced by no controller yields no observation and no warning (no flow context to judge it in). This guarantees the lint never tells an author to remove a `!` some flow relies on.
+- **Step E — otel INPUTS capture also learned the absence arm** (site not on the §14 checklist): `make_inputs_json` hard-read every needed input and crashed on a running controller whose `?` input was absent (found by the new end-to-end test). Value → as before; recorded absence → absence payload; neither → still the loud hard-miss raise.
+- **Step E — pre-existing truthiness bug fixed (load-bearing for D7):** `StuffArtefact` had no `__bool__`, so Jinja2's truth test (`{% if var %}` — exactly the guard idiom the Step D lint recommends) fell through to the list-only `__len__` and raised `TypeError` on a PRESENT singular value. Found when the new `@?hint` contract fixture failed its dry run. Fix: present non-list artefact is truthy; list artefacts follow emptiness (`[]` falsy, D4). Changelog entry under Fixed.
+- **Step E — the optional-edge marker is producer-side data:** the run epilogue stamps `extra={"optional": True}` on the output IOSpec when the pipe declares a `?` output; edge generation (tracer + assembler) reads the producer's registered output spec for the digest (shared `output_digest_is_optional` helper since the checkpoint triage). A skipped node with a SINGULAR output registers no output spec (a recorded absence has no payload), so no data edge leaves it; a skipped node with a PLURAL output registers its real empty-list output (see the triage entry below), so its data edges resolve like any produced value.
+- **Step E (checkpoint triage) — skipped nodes register real outputs.** `on_pipe_end_skipped` (protocol, NoOp, tracer, manager, `PipeEndSkippedEvent`, assembler) gained optional `output_spec`/`output_concept_data`, registered in the producer map exactly like a success output. The run epilogue now computes the output spec on BOTH arms (a lifted plural output is a real empty-list Stuff per D4) and, on the lifted arm, also registers plural companion slots via `register_controller_output` (the companion-slot registration is exercised only under an `add_each_output` lift with tracing on — covered by the code path, no dedicated test; the main-output arm is unit-tested on both builders).
+- **Step E (checkpoint triage) — one taint walk per validate pass.** `collect_controller_taint_analyses` (`pipelex/pipeline/controller_taint.py`) runs each controller's analysis once; `build_liftable_pipes` and `build_optionality_warnings` now take the analyses, not pipes. Every warnings call site (protocol report, agent-CLI cores, bare CLI, builder validate_ops) goes through the collector. Deeper memoization (recompute inside `validate_output_with_library` / enclosing walks) stays deferred (Step D item 2, carried into the Step E deferred doc).
+- **Step E (checkpoint triage) — the lint now reaches all four validate surfaces.** Bare `pipelex validate` prints warnings in yellow (bundle arm + `--all` sweep; advisory, never the exit code); the three bundle-context builder `validate_ops` twins carry the same `warnings` key as their agent-CLI cores. Single-pipe surfaces stay lint-less on every CLI (no flow context).
+- **Step E (checkpoint triage) — absence artifact renderers centralized.** `build_absence_json` (clean_json_dumps, indent=2) and `build_absence_html` (escaped `<pre>`) joined `absence_render.py`; the bare CLI (`--save-main-stuff`), the agent CLI envelope, and the delivery executor all render through them — no per-surface serializer drift.
+- **Step F — all conformance coverage is capability-gated, not green-against-HEAD.** The conformance suite never imports pipelex: its CLI arm resolves `pipelex/.venv/bin/pipelex-agent` and its HTTP arm boots `pipelex-api/.venv` — both running the released pipelex (main checkout on `release/v0.37.0`, pre-optionals). So the new table rows carry `pending_runtime_feature` (new field + `pending_feature_skip_reason` helper, the row analog of the QA corpus's `runtime_supports_case`), the new test modules probe-skip (POST/run the marker bundle; not-valid ⇒ runtime lacks optionals ⇒ skip), and the QA cases are gated — `runtime_supports_case` extended to probe a gated VALID case's `expected_is_valid` (a feature bundle an older grammar rejects must be preserved-not-clobbered too). Gate-removal checklist for the release: `wip/optionals/deferred-step-f-notes.md`.
+- **Step F — spec sections match what conformance can pin; absence records stay unverified-marked.** The four new spec sections carry Verified-by links to the gated modules (spec-links validates linkage, not execution). "Absence records on run artifacts" is `<!-- unverified -->` by design: the hosted execute wire cannot carry the ledger until the cross-repo mthds bump (hand-off item 1), and conformance has no execute arm yet — the section documents the canonical pipelex shapes and names the in-repo pins.
+- **Step F — fixture ground truth taken from this branch, not guessed.** Every new fixture was run against `_optionals/.venv/bin/pipelex-agent` (and the local protocol runner for the report artifacts) before its expectations were written down: categories confirmed (`optional_marker_invalid` → `blueprint_validation`; the four taint/lint errors → `pipe_validation`), locators recorded (e.g. `optional_output_required` carries no `variable_names`).
+- **Step F — flagged pre-existing quirk:** inline `$var` immediately followed by ` (` escapes template variable detection (`"Greet $name (nickname: ...)"` reports `name` as unused — likely Jinja2 parsing `name (…)` as a call). Unrelated to optionals; noted in `wip/optionals/deferred-step-f-notes.md`.
+- **Step F — cross-repo commits ride the repos' current working branches** (conformance `feature/config-dialect-fixture`, workspace root `feature/SDK-and-starter`) — the same branches the previous spec/conformance commits landed on; only the optionals files were staged. Flag for Louis: both carry unrelated in-flight work, so pick the merge vehicle at push time.
+- **Step D (checkpoint triage) — lifting learned companion slots.** `_make_lifted_output` (a Step B site) only resolved the pipe's own output slot; a lifted `add_each_output` parallel left its branch slots neither-value-nor-record — a hard miss downstream on flows the new static pass declares safe. New `PipeAbstract.lifted_companion_slots()` hook (overridden by PipeParallel) resolves each branch slot on lift (singular → SKIPPED record, plural → empty list), and the sequence taint walk keys its lifted-parallel arm on the same hook so static and runtime cannot diverge.
diff --git a/wip/optionals/optionals-plan.md b/wip/optionals/optionals-plan.md
new file mode 100644
index 000000000..a76d247b5
--- /dev/null
+++ b/wip/optionals/optionals-plan.md
@@ -0,0 +1,70 @@
+# Optionals — phase-1 implementation plan (CLOSED — phase 1 complete)
+
+> **Status: COMPLETE.** All steps A–F landed and the FINAL CHECKPOINT cleared — see [optionals-phase1-tracker.md](optionals-phase1-tracker.md) (checkpoint log + hand-off notes, archived from the worktree-root `TODOS.md`). Live follow-ups: the release-time conformance gate removal and cross-repo wave items in `wip/optionals/deferred-step-f-notes.md`. This document is kept as the narrative record; `wip/optionals/optionals-design.md` remains the design reference (incl. phases 2–3).
+
+Branch `feature/Optionals` (worktree `_optionals`). Design: `wip/optionals/optionals-design.md` — all decisions D1–D11 decided. This plan covers **phase 1 of §17**: the complete language core in pipelex, with `docs/specs/` + `conformance/` moving in the same change. The LLM maybe-wrapper (phase 2) and ergonomics (phase 3) are out of scope; the cross-repo wave (mthds spec, vscode/plxt, mthds-ui, skills) follows the pipelex release and must land **after** the Required-main-stuff Phase 3 sweep.
+
+Working rules: TDD (red tests first per step), `make agent-check` after every code change, full `make agent-test` at each checkpoint, one commit per checkpoint, no backward compatibility (breaking changes go in the changelog under `[Unreleased]`).
+
+Live progress tracking (checkboxes, checkpoint log, decisions/deviations) was in `TODOS.md` at the worktree root, now archived as [optionals-phase1-tracker.md](optionals-phase1-tracker.md); this document stays the narrative plan.
+
+## Step A — grammar and carriers (parse `?` / `!`, no behavior change)
+
+- Extend the marker grammar (D1) in `pipelex/core/pipes/variable_multiplicity.py` (+ its inline twin), the looser regex in `input_stuff_specs_factory.py`, and the naive `[`-splitter in `concepts/helpers.py`. Fix the `MUTLIPLICITY_PATTERN` typo while there.
+- `StuffSpec`/`NamedStuffSpec` gain the presence marker (a `PresenceMarker` StrEnum: plain / optional / force); `InputStuffSpecs.required_names` splits into required vs declared.
+- Blueprint-parse-time grammar errors (`OPTIONAL_MARKER_INVALID`): `?` on plurals, `!` on outputs or plurals, markers on concept definitions / `refines` / structure-field refs / package refs (D1, D4).
+- Builder specs mirror the blueprint rules (`to_blueprint()` passes markers through); `contract_match` canonicalization compares presence markers (D5).
+- Tests: parser unit tests for every marker × multiplicity combination, blueprint + spec accept/reject fixtures.
+
+**CHECKPOINT A** — markers parse, validate, and round-trip; zero runtime behavior change; all gates green; run the checkpoint protocol in [optionals-phase1-tracker.md](optionals-phase1-tracker.md).
+
+## Step B — absence at runtime (ledger + trichotomy + lifting)
+
+- WorkingMemory gains the absence ledger (D2): `AbsenceRecord` (`variable_name`, `producing_pipe`, `kind` = declared-absent | skipped | not-provided, `reason`, upstream chain). New tri-state resolved accessor (Stuff | AbsenceRecord) for post-run reads — do **not** resurrect `optional_main_stuff` (§14).
+- The three runtime miss-gates learn the trichotomy (D3): `PipeAbstract.validate_before_run` (fix the "Dry run of ..." message bug on the live path), `SubPipe`, PipeCondition's branch check. Plain input fed absence → **skip** (lift) + record skipped-absence for the pipe's output with provenance; `?` input → run; `!` input absent → `OptionalValueAbsentError` (D9: new error class in the relevant `exceptions.py`, provenance chain, RFC 7807 + error page via `pipelex-dev generate-error-pages`).
+- The per-pipe graph-tracer epilogue in `pipe_abstract.py` (the success path reads `working_memory.get_main_stuff()` directly) learns the resolved-as-absent arm **here, not in Step E** — it fires on every pipe run, so the first skipped pipe would crash it. The method-level boundaries (delivery, CLI, telemetry, serialization) still land in Steps C/E, so Step B tests sink every absence before the method boundary (a downstream `?` input); end-to-end absent-main-output coverage belongs to Step E.
+- Skipped plural output normalizes to empty `ListContent` + ledger note (D4); taint stops.
+- Optional method inputs (D5): omitting a `?` input in `inputs.json` / the execute request → not-provided record instead of `PipeRunInputsError`; the required-input error message gains the "these optional inputs may be omitted" hint.
+- Mock seeding / dry-run sweep learn optional slots (dry run stays all-present per D6).
+- Tests: lift-skip chain with provenance, absorb, force-failure UX (assert the message names variable, pipes, and reason), plural normalization, optional method inputs, run-report absence enumeration.
+
+**CHECKPOINT B** — absence exists, propagates, and fails loudly only through `!`; all gates green; run the checkpoint protocol in [optionals-phase1-tracker.md](optionals-phase1-tracker.md).
+
+## Step C — controllers: `continue` replacement + parallel combine (D11)
+
+- **PipeCondition `continue`** (§14): replace pass-through-or-error with "declared output absent, memory otherwise unchanged" — record the absence (kind = declared-absent, reason = the evaluated expression/outcome), stamp no main stuff, return success. Same in dry run (the all-special-outcomes guard becomes: legal iff output declared `?`). Rewrite `test_pipe_condition_continue_delivery.py` to pin the new semantics (keep the run-id stamping assertions) — these tests exercise the delivery boundary, so `delivery_executor.py` (typed + raw arms: absent main output delivers an explicit absence artifact, not an error) moves up from Step E into this step. Document the migration idiom (previous value stays in memory under its own name; coalescing is the phase-2 answer).
+- **PipeParallel combine under absence** (D11): absent branch result → structured output field with `required = false` absorbs as field-`None`; `Composite` output omits the component + ledger note. Runtime guard raises a typed error if a required field's branch is absent (should be statically unreachable after Step D, but the runtime must not feed `combine_stuffs` a hole).
+- PipeBatch compaction (D4 bonus): inner `?` output → absent branch results dropped from the aggregated list. Ship it here if it falls out of the lifting machinery naturally; otherwise defer to phase 2 with the semantics recorded.
+- Tests: continue-absent (live + dry), parallel with absorbed/omitted/required-absent branches, batch compaction if shipped.
+
+**CHECKPOINT C** — controllers speak absence; the #1014 `continue` tests replaced; all gates green; run the checkpoint protocol in [optionals-phase1-tracker.md](optionals-phase1-tracker.md).
+
+## Step D — static validation: the taint pass + template lint
+
+- Absence-propagation pass over controller dataflow (D6): `PipeSequence.needed_inputs` / `generated_outputs` gain the presence dimension; per-slot presence computed (`guaranteed` / `maybe-absent`).
+- New `PipeValidationErrorType`s: `OPTIONAL_NOT_HANDLED` (taint reaches a non-optional boundary — message names source, path, three fixes), `OPTIONAL_OUTPUT_REQUIRED` (`continue`-reachable condition without `?` output), the D11 static error (required structure field fed by a maybe-absent branch, extending #1014's `validate_output_with_library` checks).
+- Template guard-lint (D7): classify guarded vs unguarded references to declared-optional inputs in the Jinja AST walk; unguarded → validation error with the precise fix. Fix `@?` so it no longer registers a declared-optional variable as required.
+- Liftable-pipe inventory as structured data on the valid report (D3 phase-1 commitment), beside `pipe_io_contracts`.
+- Tests: taint fixtures (valid + each error type), guard-lint accept/reject, `@?` detection.
+
+**CHECKPOINT D** — validation proves the safety theorem ("every absence source reaches an explicit sink"); all gates green; run the checkpoint protocol in [optionals-phase1-tracker.md](optionals-phase1-tracker.md).
+
+## Step E — wire, boundaries, and graph
+
+- Protocol bump, one motion (D6 + D8): `optional: bool` on `PipeInputContract`/`PipeOutputContract`, the general `warnings` array on the validation report (same item shape as errors, never flips `is_valid`), absence records on run results. Neutral MTHDS naming, no `pipelex_` prefixes.
+- Populate `warnings` with the useless-`!` lint (`!` on a guaranteed slot — presence data comes from Step D's taint pass). The over-claimed-`?` lint (a `?` output that can never produce absence) is deliberately **not** emitted in phase 1: D5 blesses over-claiming as signature-evolution room, and phase 2 makes `?` on operator outputs meaningful — linting it now would warn on exactly the declarations phase 2 rewards.
+- The remaining post-run boundaries learn the resolved-as-absent arm (§14 checklist; the tracer epilogue is handled in Step B, the delivery executor in Step C): CLI run cores (bare, agent, agent-API), `otel_factory.py`, `resolve_main_stuff_root_key` / `payloads.py` / `pipeline_response.py`. `main_stuff_name: str` **stays required** — it names the declared output slot; consumers branch on the absence record. Absent main output = success with an absent result.
+- Graph (D8): `GraphSpec` gains the `skipped` node state and optional-edge marker; tracer registers skip reasons. (mthds-ui rendering is the cross-repo wave.)
+- Sweep the invariant prose to "a pipe run always resolves its declared output: a value or a recorded absence".
+- Tests: end-to-end absent-main-output run over each surface (delivery artifacts, execute response, CLI output, telemetry), graph skipped-state serialization.
+
+**CHECKPOINT E** — an absent result is a first-class success everywhere in-repo; all gates green; run the checkpoint protocol in [optionals-phase1-tracker.md](optionals-phase1-tracker.md).
+
+## Step F — specs, conformance, docs, release prep
+
+- `docs/specs/` protocol spec: `optional` flag, `warnings` array, absence records, new validation-error categories; matching `conformance/` fixtures (`optional_not_handled.mthds` etc.) + error-QA corpus rows; `make check-spec-links` green.
+- Regenerate the MTHDS JSON schema and error pages; docs: an optionality guide beside `multiplicity.md`'s pipelex counterpart, pipe-page updates (PipeCondition, PipeParallel, PipeBatch), `@?` docs reconciled.
+- Changelog under `[Unreleased]`: breaking — `continue` semantics, `?`/`!` grammar, protocol additions.
+- Full `make agent-test` + `make tb`.
+
+**FINAL CHECKPOINT** — phase 1 complete; hand off to the cross-repo wave (after RMS Phase 3) and phase 2 (LLM maybe-wrapper). Close or fold this doc.