Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Address SWE-agent review findings on PR #1020

Louis Choquel9efc3dc3eaf6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 120fff96b..956ebded1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,7 +24,7 @@
- **Type preservation across transport:** `Composite` components (and their nested lists) now retain strict types across transport boundaries (`dump_for_transport` / hydration) via private class markers.
- **`PipeParallel` honors `final_stuff_code`:** A requested final stuff code (e.g. from a `PipeBatch`) is now correctly stamped on the parallel's combined output.
- **Docs — inference plugins:** Rewrote `using-inference-plugins.md` to demonstrate a genuine `pipelex.plugins` entry-point package instead of a legacy config workaround.
-- **Linting pipeline:** `plxt lint` now validates `.mthds` files against the locally generated schema (`derived/mthds_schema.json`) rather than the released schema bundled with `plxt`.
+- **Linting pipeline (repo-local):** in the pipelex repo itself, `plxt lint` now validates `.mthds` files against the locally generated schema (`derived/mthds_schema.json`) rather than the released schema bundled with `plxt`. This is a `.pipelex/plxt.toml` override for this repo only — the `plxt.toml` template distributed by `pipelex init config` is unchanged.
- **Dry-run mocks:** Reduced the default dry-run mock list generation from 3 items to 2 to cut dry-run processing time.
- Bumped `mthds` dependency from `>=0.6.0` to `>=0.7.0`.
@@ -164,7 +164,7 @@
- **Mistral chat requests now send the system message before the user message**: `make_simple_messages` previously appended it after the user message, contradicting its own docstring and the OpenAI-typed sibling.
- **String concept values in bundle specs are constructible again**: a bare string in the `ConceptSpec | str` union now validates (no longer crashing the `mode="before"` validator) and passes through to the blueprint as the concept's description instead of into `structure`, where the loader rejected it.
- **LocalObserver JSONL records always carry the true lifecycle event name**: a payload's own `event_type` key can no longer overwrite the event name in the written record, which had broken event-type filtering for JSONL consumers.
-- **Tests under `tests/**/build/`are no longer silently skipped**: pytest's default`norecursedirs`includes`build`, so the whole `tests/unit/pipelex/cli/commands/build/`tree was never collected; the config now overrides`norecursedirs`and sets`testpaths = ["tests"]`.
+- **Tests under `tests/**/build/` are no longer silently skipped**: pytest's default `norecursedirs` includes `build`, so the whole `tests/unit/pipelex/cli/commands/build/` tree was never collected; the config now overrides `norecursedirs` and sets `testpaths = ["tests"]`.
- **`Anything`-output option numbering is now deterministic**: the output renderer sorts a `PipeCondition`'s possible outputs by pipe code instead of iterating a set, so `output_option_N` / `schema_option_N` numbering is stable across runs.
- **Documentation examples**: Updated code snippets in `docs/under-the-hood/` to match the new keyword-only signatures.
- **Test mock assertions**: Migrated mock assertions from `.call_args.args` to `.call_args.kwargs` to reflect keyword-only calls.
diff --git a/pipelex/core/stuffs/html_rendering.py b/pipelex/core/stuffs/html_rendering.py
index 0210f7ab5..1e2531875 100644
--- a/pipelex/core/stuffs/html_rendering.py
+++ b/pipelex/core/stuffs/html_rendering.py
@@ -16,7 +16,7 @@ def render_value_html(value: Any) -> str:
Uses match/case to handle all common Pydantic field types:
- None: renders as <em>None</em>
- StuffContent subclasses: calls rendered_html() recursively
- - str: escapes HTML (or passes through if already HTML)
+ - str: always escaped — a value that merely looks like HTML is untrusted input
- bool: renders as "True" or "False" (must be before int, as bool is subclass of int)
- int: renders as string
- float: renders as string
@@ -34,10 +34,6 @@ def render_value_html(value: Any) -> str:
return value.rendered_html()
case str():
- # If it looks like HTML (starts with <), render as-is
- stripped = value.strip()
- if stripped.startswith("<") and (">" in stripped):
- return value
return html_module.escape(value)
case bool():
diff --git a/pipelex/kit/configs/plxt.toml b/pipelex/kit/configs/plxt.toml
index a843f8e7e..df34dafcd 100644
--- a/pipelex/kit/configs/plxt.toml
+++ b/pipelex/kit/configs/plxt.toml
@@ -99,13 +99,6 @@ include = ["**/*.toml"]
# Which files this rule applies to (relative globs).
include = ["**/*.mthds", "**/*.plx"]
-# This repo is the source of truth for the MTHDS language: validate .mthds files
-# against the locally generated schema, not the released one bundled with plxt
-# (which lags behind whenever a language surface changes here).
-# Generate it with: .venv/bin/pipelex-dev generate-mthds-schema
-[rule.schema]
-path = "derived/mthds_schema.json"
-
# Per-rule formatting overrides — all at defaults so you can tweak them
# independently of .toml files.
[rule.formatting]
diff --git a/pipelex/kit/paths.py b/pipelex/kit/paths.py
index 5295565ce..09e649692 100644
--- a/pipelex/kit/paths.py
+++ b/pipelex/kit/paths.py
@@ -26,10 +26,14 @@ GIT_IGNORED_CONFIG_FILES: frozenset[str] = frozenset(
)
# Files excluded from config sync checks but still copied during `pipelex init config`.
-# Extends GIT_IGNORED_CONFIG_FILES with `telemetry.toml`: the kit's `telemetry.toml`
-# holds the active global template, while the pipelex repo's `.pipelex/telemetry.toml`
-# dogfoods the commented-out project template. The two files intentionally differ.
-CONFIG_SYNC_EXCLUDED_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES | {"telemetry.toml"}
+# Extends GIT_IGNORED_CONFIG_FILES with files that intentionally differ between the two sides:
+# - telemetry.toml: the kit's holds the active global template, while the pipelex repo's
+# `.pipelex/telemetry.toml` dogfoods the commented-out project template.
+# - plxt.toml: the repo's `.pipelex/plxt.toml` adds a `[rule.schema]` override pointing at the
+# locally generated `derived/mthds_schema.json` (this repo is the source of truth for the MTHDS
+# language); the kit template must not reference that repo-internal artifact — plxt hard-fails
+# on every .mthds file when the configured schema path does not exist.
+CONFIG_SYNC_EXCLUDED_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES | {"telemetry.toml", "plxt.toml"}
# Directories that should not be synced between .pipelex and kit/configs.
# These are runtime directories created locally:
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index af46e756a..d99954dd9 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -7,6 +7,7 @@ from typing_extensions import override
from pipelex import log
from pipelex.core.concepts.concept import Concept
+from pipelex.core.concepts.exceptions import ConceptValueError
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
@@ -150,7 +151,22 @@ class PipeParallel(PipeController):
names: required fields ⊆ result names, result names ⊆ declared fields. This turns the
runtime combine failure into an author-time error surfaced by `/validate`.
"""
- structure_class = self.output.concept.get_structure_class()
+ try:
+ structure_class = self.output.concept.get_structure_class()
+ except ConceptValueError as exc:
+ # A plain ValueError would escape the /validate sweep and abort the whole bundle;
+ # convert it so this pipe alone is reported as failed.
+ msg = (
+ f"PipeParallel '{self.code}' output '{self.output.concept.concept_ref}' has no registered "
+ f"structure class ('{self.output.concept.structure_class_name}'): {exc}"
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ provided_concept_code=self.output.concept.concept_ref,
+ ) from exc
if issubclass(structure_class, CompositeContent):
return
@@ -220,7 +236,23 @@ class PipeParallel(PipeController):
if branch_pipe.output.multiplicity is not None:
# A list-producing branch combines as a ListContent, not as the item class.
continue
- branch_structure_class = branch_pipe.output.concept.get_structure_class()
+ try:
+ branch_structure_class = branch_pipe.output.concept.get_structure_class()
+ except ConceptValueError as exc:
+ # Same conversion as the output lookup above: keep the failure per-pipe.
+ msg = (
+ f"PipeParallel '{self.code}' branch '{sub_pipe.pipe_code}' output "
+ f"'{branch_pipe.output.concept.concept_ref}' has no registered structure class "
+ f"('{branch_pipe.output.concept.structure_class_name}'): {exc}"
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ provided_concept_code=branch_pipe.output.concept.concept_ref,
+ variable_names=[sub_pipe.output_name],
+ ) from exc
if not issubclass(branch_structure_class, field_annotation):
msg = (
f"PipeParallel '{self.code}' branch '{sub_pipe.pipe_code}' produces "
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py b/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py
index c4de9517a..02886a97f 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py
@@ -3,6 +3,7 @@
import io
from pathlib import Path
from urllib.parse import urlparse
+from urllib.request import url2pathname
import pytest
from PIL import Image
@@ -123,8 +124,11 @@ class TestPipeImgGen:
parsed_url = urlparse(image_content.public_url)
if parsed_url.scheme in {"http", "https"}:
image_bytes = await fetch_file_from_url_httpx(url=image_content.public_url)
+ elif parsed_url.scheme == "file":
+ # file:// paths are URI-formatted (percent-encoded) — decode before touching the filesystem
+ image_bytes = Path(url2pathname(parsed_url.path)).read_bytes()
else:
- image_bytes = Path(parsed_url.path).read_bytes()
+ image_bytes = Path(image_content.public_url).read_bytes()
with Image.open(io.BytesIO(image_bytes)) as generated_image:
assert generated_image.size == (expected_width, expected_height), (
f"{topic}: expected {expected_width}x{expected_height}, got {generated_image.size[0]}x{generated_image.size[1]}"
diff --git a/tests/integration/pipelex/cogt/test_img2img.py b/tests/integration/pipelex/cogt/test_img2img.py
index f29c830be..e4595c052 100644
--- a/tests/integration/pipelex/cogt/test_img2img.py
+++ b/tests/integration/pipelex/cogt/test_img2img.py
@@ -32,7 +32,7 @@ class Img2ImgTestCases:
# Single input image test cases: (topic, image_uri)
SINGLE_INPUT_IMAGES: ClassVar[list[tuple[str, str]]] = [
("PNG local file", ImageTestCases.IMAGE_FILE_PATH_PNG_1),
- # ("JPG local file", ImageTestCases.IMAGE_FILE_PATH_JPG_3),
+ ("JPG local file", ImageTestCases.IMAGE_FILE_PATH_JPG_3),
]
# Multiple input images test cases: (topic, list of image_uris)
@@ -41,10 +41,10 @@ class Img2ImgTestCases:
"Two JPG files",
[ImageTestCases.IMAGE_FILE_PATH_JPG_1, ImageTestCases.IMAGE_FILE_PATH_JPG_3],
),
- # (
- # "Mixed PNG and JPG",
- # [ImageTestCases.IMAGE_FILE_PATH_PNG_1, ImageTestCases.IMAGE_FILE_PATH_JPG_1],
- # ),
+ (
+ "Mixed PNG and JPG",
+ [ImageTestCases.IMAGE_FILE_PATH_PNG_1, ImageTestCases.IMAGE_FILE_PATH_JPG_1],
+ ),
]
# Remote URL input image test cases: (topic, image_url)
diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_unresolvable_structure_class.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_unresolvable_structure_class.py
new file mode 100644
index 000000000..80b6cf4ed
--- /dev/null
+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_unresolvable_structure_class.py
@@ -0,0 +1,93 @@
+from typing import Callable
+
+import pytest
+from pydantic import Field
+
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.pipes.exceptions import PipeValidationError
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry, get_concept_library, get_pipe_library
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+
+DOMAIN_CODE = "test_usc"
+
+
+class UscRegisteredCombined(StructuredContent):
+ """Registered combination target, used to reach the per-branch structure-class lookup."""
+
+ summary: TextContent = Field(description="Summary produced by the branch")
+
+
+class TestPipeParallelUnresolvableStructureClass:
+ """A missing structure class must surface as PipeValidationError, not a raw ConceptValueError.
+
+ The `/validate` sweep only catches Pipelex-typed errors around per-pipe validation, so a
+ bare ConceptValueError (a plain ValueError) escaping `validate_output_with_library` would
+ abort the whole bundle instead of failing just this pipe.
+ """
+
+ def _setup_library(self, *, output_concept_code: str, branch_output_concept_code: str) -> PipeParallel:
+ """Build a parallel whose output (or branch output) concept has an unregistered structure class."""
+ concept_library = get_concept_library()
+ for concept_code in {output_concept_code, branch_output_concept_code}:
+ structure_class_name = "UscRegisteredCombined" if concept_code == "UscRegisteredCombined" else f"{concept_code}UnregisteredStructure"
+ concept = ConceptFactory.make(
+ concept_code=concept_code,
+ domain_code=DOMAIN_CODE,
+ description=f"Concept {concept_code}",
+ structure_class_name=structure_class_name,
+ )
+ concept_library.add_concepts(concepts=[concept])
+
+ pipe_library = get_pipe_library()
+ branch_pipe = PipeFactory[PipeLLM].make_from_blueprint(
+ domain_code=DOMAIN_CODE,
+ pipe_code="usc_branch",
+ blueprint=PipeLLMBlueprint(
+ description="Branch producing the summary",
+ inputs={"input_text": "Text"},
+ output=f"{DOMAIN_CODE}.{branch_output_concept_code}" if branch_output_concept_code != "Text" else "Text",
+ prompt="Summarize: $input_text",
+ ),
+ concept_codes_from_the_same_domain=[output_concept_code, branch_output_concept_code],
+ )
+ pipe_library.add_new_pipe(pipe=branch_pipe)
+
+ pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(
+ domain_code=DOMAIN_CODE,
+ pipe_code="usc_parallel",
+ blueprint=PipeParallelBlueprint(
+ description="Parallel with an unresolvable output structure class",
+ inputs={"input_text": "Text"},
+ output=f"{DOMAIN_CODE}.{output_concept_code}",
+ branches=[
+ SubPipeBlueprint(pipe="usc_branch", result="summary"),
+ ],
+ ),
+ concept_codes_from_the_same_domain=[output_concept_code, branch_output_concept_code],
+ )
+ pipe_library.add_new_pipe(pipe=pipe_parallel)
+ return pipe_parallel
+
+ def test_unregistered_output_structure_class_raises_pipe_validation_error(self, load_empty_library: Callable[[], str]):
+ """The parallel's own output concept resolves to no registered class: must be a PipeValidationError."""
+ load_empty_library()
+ pipe_parallel = self._setup_library(output_concept_code="UscCombined", branch_output_concept_code="UscCombined")
+
+ with pytest.raises(PipeValidationError, match="UscCombined"):
+ pipe_parallel.validate_output_with_library()
+
+ def test_unregistered_branch_structure_class_raises_pipe_validation_error(self, load_empty_library: Callable[[], str]):
+ """A branch output concept resolving to no registered class: must be a PipeValidationError too."""
+ load_empty_library()
+ get_class_registry().register_class(UscRegisteredCombined)
+ pipe_parallel = self._setup_library(output_concept_code="UscRegisteredCombined", branch_output_concept_code="UscBranchOnly")
+
+ with pytest.raises(PipeValidationError, match="UscBranchOnly"):
+ pipe_parallel.validate_output_with_library()
diff --git a/tests/unit/pipelex/core/stuffs/test_html_rendering.py b/tests/unit/pipelex/core/stuffs/test_html_rendering.py
new file mode 100644
index 000000000..86be9a635
--- /dev/null
+++ b/tests/unit/pipelex/core/stuffs/test_html_rendering.py
@@ -0,0 +1,22 @@
+from pipelex.core.stuffs.html_rendering import render_value_html
+
+
+class TestRenderValueHtml:
+ """Plain strings must ALWAYS be escaped — a value that merely looks like HTML is untrusted input."""
+
+ def test_html_looking_string_is_escaped(self):
+ """A field value shaped like an HTML tag is an XSS vector and must not pass through raw."""
+ result = render_value_html("<img src=x onerror=alert(1)>")
+ assert result == "&lt;img src=x onerror=alert(1)&gt;"
+
+ def test_plain_string_is_escaped(self):
+ result = render_value_html("Tom & Jerry <3")
+ assert result == "Tom &amp; Jerry &lt;3"
+
+ def test_html_looking_string_inside_list_is_escaped(self):
+ result = render_value_html(["<script>alert(1)</script>"])
+ assert result == "<ul><li>&lt;script&gt;alert(1)&lt;/script&gt;</li></ul>"
+
+ def test_html_looking_string_inside_dict_is_escaped(self):
+ result = render_value_html({"payload": "<b>bold</b>"})
+ assert result == "<dl><dt>payload</dt><dd>&lt;b&gt;bold&lt;/b&gt;</dd></dl>"
diff --git a/wip/llm-max-tokens-fallback-dead-code.md b/wip/llm-max-tokens-fallback-dead-code.md
new file mode 100644
index 000000000..07a700df3
--- /dev/null
+++ b/wip/llm-max-tokens-fallback-dead-code.md
@@ -0,0 +1,34 @@
+# LLM `max_tokens` model fallback in `_apply_constraints` is (almost) dead code
+
+**Status:** Deferred design tradeoff (not a regression — pre-existing on `main`, moved byte-equivalently by the plugins refactor). Flagged by cubic on PR #1020.
+
+## Current behavior
+
+`LLMWorkerAbstract._apply_constraints` (`pipelex/cogt/llm/llm_worker_abstract.py`; previously identical in `llm_worker_internal_abstract.py` on `main`) computes a max-tokens fallback but only ships it when a *temperature* rule fires:
+
+```python
+max_tokens = original_params.max_tokens or self.inference_model.max_tokens
+new_max_tokens = max_tokens
+has_changes = False
+# ... only temperature constraints set has_changes = True ...
+if not has_changes:
+ return None # applied_job_params stays None → workers use the original params
+return original_params.model_copy(update={"temperature": new_temperature, "max_tokens": new_max_tokens})
+```
+
+Every worker consumes `llm_job.applied_job_params or llm_job.job_params`, then sends `job_params.max_tokens or omit`. So when a job omits `max_tokens`:
+
+- **No temperature constraint on the model** → no cap is sent; the provider default applies.
+- **A temperature constraint fires** → the model's `max_tokens` from the spec IS sent as the request cap.
+
+The cap a provider receives depends on an unrelated temperature rule — an inconsistency, though each individual outcome is defensible.
+
+## Options
+
+1. **Mark the fallback as a change** (cubic's suggestion: `has_changes = new_max_tokens != original_params.max_tokens`): the model spec's `max_tokens` is always sent explicitly when the job omits it. Consistent, "model constraints actually enforced" — but changes wire behavior for every provider/backend on every capless request.
+2. **Remove the fallback**: `new_max_tokens = original_params.max_tokens` unchanged; when a temperature rule fires, the applied params keep the job's own (possibly absent) cap. Provider default semantics become uniform — also a wire behavior change, in the opposite direction, limited to constraint-firing models.
+3. **Status quo**: keep the inconsistency documented here.
+
+## Decision needed
+
+Pick 1 or 2 in a normal dev cycle (not a release branch) with a targeted live-inference sanity pass, since either option changes what gets sent on the wire. Whichever way it goes, add a unit test on `_apply_constraints` pinning the chosen `max_tokens` semantics.
diff --git a/wip/nested-composite-hydration-not-recursive.md b/wip/nested-composite-hydration-not-recursive.md
new file mode 100644
index 000000000..52cb735ef
--- /dev/null
+++ b/wip/nested-composite-hydration-not-recursive.md
@@ -0,0 +1,22 @@
+# Nested CompositeContent hydration is not recursive across the runtime bridge
+
+**Status:** Deferred follow-up (confirmed defect, deliberately not fixed in the v0.37.0 release). Flagged by codex and cubic on PR #1020.
+
+## Problem
+
+The transport encode side recurses, the hydration side doesn't:
+
+- **Encode** (`pipelex/core/memory/working_memory.py`): `_encode_composite_for_transport` sends every composite component through `_encode_content_with_class_markers`, which recurses back into `_encode_composite_for_transport` when the component is itself a `CompositeContent`. Inner components are correctly stamped with `__pipelex_class__` / `__pipelex_module__` markers at every depth.
+- **Hydrate** (`pipelex/runtime_bridge/primitives/hydration.py`): `_hydrate_list_item`'s marker branch strips the markers only at the top level and calls `item_class.model_validate(clean_item)`. When `item_class` is a `CompositeContent` subclass (extra="allow" — no annotations to drive validation), the inner components are left as raw marker-bearing dicts instead of typed `StuffContent`.
+
+Consequence: a `CompositeContent` nested inside another `CompositeContent` (e.g. a `PipeParallel` result feeding another parallel's combine, crossing the Temporal/runtime-bridge boundary via `dump_for_transport()` → `hydrate_working_memory()`) round-trips with corrupted typed access — downstream `rendered_*` calls and type checks see dicts with `__pipelex_*` keys.
+
+One level of composite (the common case, top-level composite stuff) hydrates correctly via `hydrate_content`'s composite branch; only *nested* composites are affected.
+
+## Proposed fix
+
+In `_hydrate_list_item` (hydration.py), in the marker branch: after resolving `item_class` and building `clean_item`, when `issubclass(item_class, CompositeContent)`, map each value of `clean_item` through the existing `_hydrate_composite_component` before `model_validate`. The mutual recursion `_hydrate_composite_component` ↔ `_hydrate_list_item` then handles arbitrary nesting depth.
+
+## Test shape (TDD)
+
+In `tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py`: build a `WorkingMemory` whose main stuff is a `CompositeContent` containing (a) a nested `CompositeContent` component and (b) a list-of-composites component; run `dump_for_transport()` → `json.dumps`/`json.loads` → `hydrate_working_memory()`; assert every nested component is a typed `StuffContent` instance and no `__pipelex_class__` / `__pipelex_module__` key survives anywhere in the tree.