Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix: report unknown for a chain ending at a cross-package base

Louis Choquel2f078374ef9f
diff --git a/.badges/tests.json b/.badges/tests.json
index fd3605c70..90d18bcf9 100644
--- a/.badges/tests.json
+++ b/.badges/tests.json
@@ -1,7 +1,7 @@
{
"schemaVersion": 1,
"label": "tests",
- "message": "10894",
+ "message": "10895",
"color": "blue",
"cacheSeconds": 300
}
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bf71bec34..1ccbf52ff 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,7 +11,7 @@
- **MTHDS intent hints**: A non-normative `hints` table can now be authored on a concept, on a structure field, and on a pipe input slot, with one defined key — `intent`, over the closed vocabulary `prose` / `label` / `rating` / `quantity`. Hints travel through the library crate (a concept's effective hints merge along its refinement chain, the nearer declaration winning) and surface on the input-form descriptor, while execution, validation verdicts and pipe contracts never read them. Validation emits advisory warnings (`hint_unknown_key`, `hint_unknown_intent`, `hint_inapplicable_intent`) rather than failing, and unknown-but-well-formed content is preserved. See [Intent hints](building-methods/concepts/intent-hints.md).
- **Input-form descriptors on the validation report (Breaking)**: `PipelexValidationReport.input_form` maps every `pipe_ref` to an ordered list of field descriptors, derived by `build_input_form` from the loaded pipes and blueprints — never from the emitted JSON Schema — so a UI renderer gets the namespaced `concept_ref` and its `refines` chain, the three-valued presence, a stated `gating` fact, fixed counts, authored defaults beside `required`, and one-member `choices` lists, with `unknown` as the honest escape hatch. Class-backed concepts are reflected through the new `reflect_structure_class`. Breaking: `build_validation_report` now requires `input_form`, so every backend assembles it or fails loudly. See [Input-form descriptor](under-the-hood/input-form-descriptor.md).
- **Richer emitted JSON Schemas**: A schema now carries the concept ref as its top-level `title` — replacing the mangled generated-class name — and the concept's authored description as its top-level `description`, uniformly for generated, class-backed and native concepts. A fixed-count slot (`Concept[N]`) emits `minItems`/`maxItems` on its array wrap while a variable-length one (`Concept[]`) stays unbounded, and the native content classes (`TextContent`, `ImageContent`, `DocumentContent`, `PageContent`, `NumberContent`, `HtmlContent`, `YesNoContent`) gained docstrings matching their pinned native descriptions, so provider-side schemas built straight from the classes carry them too.
-- **`pipelex-dev trace-input-semantics`**: A capture harness for the chain that turns authored structure syntax into the `json_schema` on `pipe_io_contracts`. Given one or more `.mthds` bundles it dumps one artifact per hop — parse blueprint, generated class source, raw pydantic schema, SCHEMA render, wire contract — plus a manifest of each input's wire framing, so a lost or mangled authored fact is localized to exactly one hop. A committed probe bundle exercises every construct the language accepts. See [Trace input semantics](contribute/trace-input-semantics.md).
+- **`pipelex-dev trace-input-semantics`**: A capture harness for the chain that turns authored structure syntax into the `json_schema` on `pipe_io_contracts`. Given one or more `.mthds` bundles it dumps per-hop captures — parse blueprint, generated class source, raw pydantic schema, SCHEMA render, wire contract — plus a manifest of each input's wire framing, so a lost or mangled authored fact is localized to exactly one hop. A committed probe bundle exercises every construct the language accepts. See [Trace input semantics](contribute/trace-input-semantics.md).
### Changed
diff --git a/docs/building-methods/pipes/understanding-optionality.md b/docs/building-methods/pipes/understanding-optionality.md
index 275403733..f61378732 100644
--- a/docs/building-methods/pipes/understanding-optionality.md
+++ b/docs/building-methods/pipes/understanding-optionality.md
@@ -120,4 +120,4 @@ Two more optionality facts surface on the **valid** report (`pipelex validate`,
- [PipeParallel](pipe-controllers/PipeParallel.md) — combining under absence
- [PipeBatch](pipe-controllers/PipeBatch.md) — compaction of absent branch results
- [Execution Graph Tracing](../../under-the-hood/execution-graph-tracing.md) — skipped nodes, `skip_reason`, and optional data edges
-- [Per-Pipe IO Contracts](../../under-the-hood/pipe-io-contracts.md) — how the marker is reported on `pipe_io_contracts` as the three-valued `presence`
+- [Per-Pipe IO Contracts](../../under-the-hood/pipe-io-contracts.md) — how an input's marker is reported on `pipe_io_contracts` as the three-valued `presence`, and an output's as the two-valued `optional`
diff --git a/docs/contribute/trace-input-semantics.md b/docs/contribute/trace-input-semantics.md
index eff406edf..a46d64ba4 100644
--- a/docs/contribute/trace-input-semantics.md
+++ b/docs/contribute/trace-input-semantics.md
@@ -1,6 +1,6 @@
# Tracing Input Semantics
-`pipelex-dev trace-input-semantics` is an internal capture harness that answers one question with evidence instead of code-reading: **what does each hop of the input-schema emission chain do to every fact a method author writes?** Given one or more `.mthds` bundle files, it loads them through the validation library and dumps one artifact per hop of the chain that turns authored structure syntax into the `json_schema` emitted on `pipe_io_contracts`.
+`pipelex-dev trace-input-semantics` is an internal capture harness that answers one question with evidence instead of code-reading: **what does each hop of the input-schema emission chain do to every fact a method author writes?** Given one or more `.mthds` bundle files, it loads them through the validation library and dumps per-hop captures of the chain that turns authored structure syntax into the `json_schema` emitted on `pipe_io_contracts`.
```bash
.venv/bin/pipelex-dev trace-input-semantics path/to/bundle.mthds -o output_dir
@@ -8,7 +8,7 @@
## What it captures
-The chain has five hops, and the output directory holds one artifact per hop, so a lost or mangled fact is localized to exactly one:
+Each hop of the chain writes its captures into the output directory, so a lost or mangled fact is localized to exactly one hop:
- `hop1_bundle_blueprints.json` — **parse**: the `PipelexBundleBlueprint` dumps, i.e. what survived TOML.
- `hop2_generated_sources/` — **resolve + generate**: the structure-class source the runtime generates per concept, re-derived with the same `StructureGenerator` calls the concept factory issues at load time. Saved as `.py.txt` so captures stay inert to linters.
diff --git a/docs/under-the-hood/input-form-descriptor.md b/docs/under-the-hood/input-form-descriptor.md
index 18c391917..71ee925d8 100644
--- a/docs/under-the-hood/input-form-descriptor.md
+++ b/docs/under-the-hood/input-form-descriptor.md
@@ -17,7 +17,7 @@ The report always carries the field. Whether it travels on the HTTP wire is the
The derivation reads two things and nothing else:
-- **Slot facts** from the loaded pipes' `StuffSpec`s: the authored input order, the three-valued presence marker (`plain` / `optional` / `force`), and the multiplicity including a fixed `[N]` count. Iterating the same loaded pipes as `build_pipe_io_contracts` — `PipeSignature` placeholders and controller-inferred inputs included — is what makes the two key sets equal by construction.
+- **Slot facts** from the loaded pipes' `StuffSpec`s: the authored input order, the three-valued presence marker (`plain` / `optional` / `force`), and the multiplicity including a fixed `[N]` count. Iterating the same loaded pipes as `build_pipe_io_contracts` — `PipeSignature` placeholders included — is what makes the two key sets equal by construction.
- **Concept facts** from the *qualified* library crate built from the parsed blueprints: descriptions, refinement links, structure fields with their defaults, choices, required-ness and nested concept refs. Qualified, not normalized: normalization flattens in-crate refinement, and the descriptor must report the `refines` chain as a list. Native concepts contribute their pinned blueprints; class-backed concepts (`structure = "ClassName"`) are reflected from the class registry, which is why the derivation must run while the validation library is still loaded.
The derivation is total. A node it cannot map honestly reports `kind: "unknown"`, the renderer's raw escape hatch against the sibling `json_schema`; nothing raises.
diff --git a/docs/under-the-hood/pipe-io-contracts.md b/docs/under-the-hood/pipe-io-contracts.md
index ec8d282a1..122eed0bb 100644
--- a/docs/under-the-hood/pipe-io-contracts.md
+++ b/docs/under-the-hood/pipe-io-contracts.md
@@ -1,6 +1,6 @@
# Per-Pipe IO Contracts
-Every valid `validate` report carries a **`pipe_io_contracts`** map: for each pipe, the typed contract of what it takes in and what it produces. It sits beside the [input-form descriptor](input-form-descriptor.md) on the same report, and the two are keyed identically — by the namespaced `pipe_ref` (`domain.code`) — over exactly the same set of pipes, `PipeSignature` placeholders and controller-inferred inputs included.
+Every valid `validate` report carries a **`pipe_io_contracts`** map: for each pipe, the typed contract of what it takes in and what it produces. It sits beside the [input-form descriptor](input-form-descriptor.md) on the same report, and the two are keyed identically — by the namespaced `pipe_ref` (`domain.code`) — over exactly the same set of pipes, `PipeSignature` placeholders included.
Reach for `pipe_io_contracts` when you need the schema and the declared contract; reach for `input_form` when you are building a form and want field kinds, defaults, and choices without reading schemas.
diff --git a/pipelex/core/concepts/concept_factory.py b/pipelex/core/concepts/concept_factory.py
index 9b25fd096..d9c072730 100644
--- a/pipelex/core/concepts/concept_factory.py
+++ b/pipelex/core/concepts/concept_factory.py
@@ -438,9 +438,11 @@ class ConceptFactory:
qualified_class_name = make_qualified_structure_class_name(domain_code=domain_code, concept_code=concept_code)
- # Cross-package refines: base class isn't available locally, so generate
- # a standalone TextContent subclass. The refinement relationship is tracked
- # in the concept model's refines field for runtime compatibility checks.
+ # Cross-package refines: base class isn't available locally, so generate a standalone,
+ # field-less StructuredContent subclass (no base_class_name is passed). The refinement
+ # relationship is tracked in the concept model's refines field for runtime compatibility
+ # checks. Because no local fact describes the base's shape, the input-form descriptor
+ # reports such a concept as `unknown` rather than promoting it to prose.
if QualifiedRef.has_cross_package_prefix(current_refine):
try:
_, the_generated_class = StructureGenerator().generate_from_structure_blueprint(
diff --git a/pipelex/pipeline/input_form.py b/pipelex/pipeline/input_form.py
index 24f99eec1..15c8df222 100644
--- a/pipelex/pipeline/input_form.py
+++ b/pipelex/pipeline/input_form.py
@@ -352,6 +352,12 @@ class InputFormDeriver:
refines=refines,
seen=seen,
)
+ if chain and self._concepts.get(chain[-1]) is None:
+ # The chain ends at a base this crate does not hold — a cross-package `alias->…` refines,
+ # whose concepts live in an isolated child library and never enter the crate. The engine
+ # backs such a concept with a field-less `StructuredContent` subclass, so its shape is
+ # genuinely unknown: the same answer `_concept_node` gives for an absent concept.
+ return _unknown_node(name=name, concept_ref=concept_ref, description=blueprint.description, refines=refines)
return _prose_promoted_node(name=name, concept_ref=concept_ref, description=blueprint.description, chain=chain)
def _native_node(
diff --git a/tests/unit/pipelex/pipeline/test_input_form_deriver.py b/tests/unit/pipelex/pipeline/test_input_form_deriver.py
index 0207bdc97..ce2f8e1f2 100644
--- a/tests/unit/pipelex/pipeline/test_input_form_deriver.py
+++ b/tests/unit/pipelex/pipeline/test_input_form_deriver.py
@@ -65,3 +65,14 @@ class TestInputFormDeriverEscapeHatches:
assert missing.kind == FieldKind.UNKNOWN
assert missing.concept_ref == "elsewhere.Missing"
+
+ def test_chain_ending_at_a_base_absent_from_the_crate_is_unknown(self) -> None:
+ """A cross-package `alias->…` base never enters the crate, so its shape is unknown — never prose."""
+ concepts: dict[str, ConceptBlueprint | str] = {
+ "demo.RefinedScore": ConceptBlueprint(description="Refines a dependency's score", refines="dep->other.Score"),
+ }
+ node = InputFormDeriver(concepts=concepts).derive_concept(name="score", concept_ref="demo.RefinedScore")
+
+ assert node.kind == FieldKind.UNKNOWN
+ assert node.refines == ["dep->other.Score"]
+ assert node.description == "Refines a dependency's score"
diff --git a/wip/input-semantics/deferred-cross-package-refines-runtime-shape.md b/wip/input-semantics/deferred-cross-package-refines-runtime-shape.md
new file mode 100644
index 000000000..ac582936e
--- /dev/null
+++ b/wip/input-semantics/deferred-cross-package-refines-runtime-shape.md
@@ -0,0 +1,33 @@
+# Deferred: what should a cross-package refinement generate at runtime?
+
+**Raised during** the review-bot triage of PR #1151 (release/v0.52.0), while verifying a Codex finding on `input_form`. Not itself bot-reported — it surfaced from the code the finding pointed at.
+
+## The open question
+
+When a concept refines a cross-package base (`alias->domain.Code`), the base class is not available locally. `ConceptFactory._handle_refines` (`pipelex/core/concepts/concept_factory.py`, the `QualifiedRef.has_cross_package_prefix(current_refine)` branch) handles this by generating a standalone structure class with an empty structure blueprint and **no** `base_class_name`. `StructureGenerator` defaults that to `StructuredContent`, so the result is a **field-less `StructuredContent` subclass**. The refinement itself survives only as the `refine_string` on the concept model, for runtime compatibility checks.
+
+The comment above that branch used to say it generated a `TextContent` subclass — which the code has never done. That comment has been corrected to describe the actual behaviour, but correcting it does not settle the question it was hiding: **is a field-less `StructuredContent` the right thing to generate here at all?**
+
+Three candidate answers, none obviously correct:
+
+1. **Keep `StructuredContent`** — honest about knowing nothing, and structurally compatible with whatever the base turns out to be. But a field-less structured object is not usable shape information for anything downstream.
+2. **`TextContent`, as the stale comment claimed** — makes the concept usable as prose, but invents a shape the base may not have. If the dependency's concept is structured, this is a lie with consequences at execution time, not just in a descriptor.
+3. **Resolve the base from the dependency's child library** — the only answer that is actually informed. Costly: dependency concepts load into an isolated `child_library` and are deliberately not in the consuming crate, so this would mean crossing a boundary the loader draws on purpose.
+
+## Why it is deferred rather than fixed
+
+The descriptor half of this is already fixed and does not depend on the answer. `build_input_form` now reports such a concept as `unknown` (with its real `refines` chain preserved) instead of promoting it to `prose` with a fabricated `native.Text` link — see `_blueprint_node_for_chain` in `pipelex/pipeline/input_form.py` and the pinning test `test_chain_ending_at_a_base_absent_from_the_crate_is_unknown` in `tests/unit/pipelex/pipeline/test_input_form_deriver.py`. That is correct **under all three answers above**: `input_form` cannot know the base's shape in any of them, so `unknown` is the honest report either way.
+
+What remains is a runtime design decision about what the engine should back these concepts with, and that is not a call a PR-review pass should make unilaterally — particularly not on a release branch.
+
+## Where to look
+
+- `pipelex/core/concepts/concept_factory.py` — `_handle_refines`, the cross-package branch.
+- `pipelex/libraries/concept/concept_library.py` — `validation_static`, which hard-rejects every dangling `refines` *except* a cross-package one. That carve-out is what makes this path reachable at all.
+- `pipelex/libraries/library_manager.py` — `_load_single_dependency`, which builds the isolated child library. This is why the base never appears in the consuming crate, loaded or not.
+- `tests/data/packages/refining_consumer/refining.mthds` — a fixture already shipping this shape (`PkgTestRefinedScore` refines a dependency's concept).
+
+## Related
+
+- [[deferred-descriptor-reflection-and-roundtrip]] — the other descriptor-side follow-up from this workstream.
+- PR thread: <https://github.com/Pipelex/pipelex/pull/1151#discussion_r> (Codex, `pipelex/pipeline/input_form.py`) — the descriptor fix that thread asked for has landed; this note is the runtime question left behind it.