Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Address SWE-agent review findings on PR #1020
Louis Choquel9efc3dc3eaf6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 120fff96b..956ebded1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,7 +24,7 @@- **Type preservation across transport:** `Composite` components (and their nested lists) now retain strict types across transport boundaries (`dump_for_transport` / hydration) via private class markers.- **`PipeParallel` honors `final_stuff_code`:** A requested final stuff code (e.g. from a `PipeBatch`) is now correctly stamped on the parallel's combined output.- **Docs — inference plugins:** Rewrote `using-inference-plugins.md` to demonstrate a genuine `pipelex.plugins` entry-point package instead of a legacy config workaround.-- **Linting pipeline:** `plxt lint` now validates `.mthds` files against the locally generated schema (`derived/mthds_schema.json`) rather than the released schema bundled with `plxt`.+- **Linting pipeline (repo-local):** in the pipelex repo itself, `plxt lint` now validates `.mthds` files against the locally generated schema (`derived/mthds_schema.json`) rather than the released schema bundled with `plxt`. This is a `.pipelex/plxt.toml` override for this repo only — the `plxt.toml` template distributed by `pipelex init config` is unchanged.- **Dry-run mocks:** Reduced the default dry-run mock list generation from 3 items to 2 to cut dry-run processing time.- Bumped `mthds` dependency from `>=0.6.0` to `>=0.7.0`.@@ -164,7 +164,7 @@- **Mistral chat requests now send the system message before the user message**: `make_simple_messages` previously appended it after the user message, contradicting its own docstring and the OpenAI-typed sibling.- **String concept values in bundle specs are constructible again**: a bare string in the `ConceptSpec | str` union now validates (no longer crashing the `mode="before"` validator) and passes through to the blueprint as the concept's description instead of into `structure`, where the loader rejected it.- **LocalObserver JSONL records always carry the true lifecycle event name**: a payload's own `event_type` key can no longer overwrite the event name in the written record, which had broken event-type filtering for JSONL consumers.-- **Tests under `tests/**/build/`are no longer silently skipped**: pytest's default`norecursedirs`includes`build`, so the whole `tests/unit/pipelex/cli/commands/build/`tree was never collected; the config now overrides`norecursedirs`and sets`testpaths = ["tests"]`.+- **Tests under `tests/**/build/` are no longer silently skipped**: pytest's default `norecursedirs` includes `build`, so the whole `tests/unit/pipelex/cli/commands/build/` tree was never collected; the config now overrides `norecursedirs` and sets `testpaths = ["tests"]`.- **`Anything`-output option numbering is now deterministic**: the output renderer sorts a `PipeCondition`'s possible outputs by pipe code instead of iterating a set, so `output_option_N` / `schema_option_N` numbering is stable across runs.- **Documentation examples**: Updated code snippets in `docs/under-the-hood/` to match the new keyword-only signatures.- **Test mock assertions**: Migrated mock assertions from `.call_args.args` to `.call_args.kwargs` to reflect keyword-only calls.diff --git a/pipelex/core/stuffs/html_rendering.py b/pipelex/core/stuffs/html_rendering.pyindex 0210f7ab5..1e2531875 100644--- a/pipelex/core/stuffs/html_rendering.py+++ b/pipelex/core/stuffs/html_rendering.py@@ -16,7 +16,7 @@ def render_value_html(value: Any) -> str:Uses match/case to handle all common Pydantic field types:- None: renders as <em>None</em>- StuffContent subclasses: calls rendered_html() recursively- - str: escapes HTML (or passes through if already HTML)+ - str: always escaped — a value that merely looks like HTML is untrusted input- bool: renders as "True" or "False" (must be before int, as bool is subclass of int)- int: renders as string- float: renders as string@@ -34,10 +34,6 @@ def render_value_html(value: Any) -> str:return value.rendered_html()case str():- # If it looks like HTML (starts with <), render as-is- stripped = value.strip()- if stripped.startswith("<") and (">" in stripped):- return valuereturn html_module.escape(value)case bool():diff --git a/pipelex/kit/configs/plxt.toml b/pipelex/kit/configs/plxt.tomlindex a843f8e7e..df34dafcd 100644--- a/pipelex/kit/configs/plxt.toml+++ b/pipelex/kit/configs/plxt.toml@@ -99,13 +99,6 @@ include = ["**/*.toml"]# Which files this rule applies to (relative globs).include = ["**/*.mthds", "**/*.plx"]-# This repo is the source of truth for the MTHDS language: validate .mthds files-# against the locally generated schema, not the released one bundled with plxt-# (which lags behind whenever a language surface changes here).-# Generate it with: .venv/bin/pipelex-dev generate-mthds-schema-[rule.schema]-path = "derived/mthds_schema.json"-# Per-rule formatting overrides — all at defaults so you can tweak them# independently of .toml files.[rule.formatting]diff --git a/pipelex/kit/paths.py b/pipelex/kit/paths.pyindex 5295565ce..09e649692 100644--- a/pipelex/kit/paths.py+++ b/pipelex/kit/paths.py@@ -26,10 +26,14 @@ GIT_IGNORED_CONFIG_FILES: frozenset[str] = frozenset()# Files excluded from config sync checks but still copied during `pipelex init config`.-# Extends GIT_IGNORED_CONFIG_FILES with `telemetry.toml`: the kit's `telemetry.toml`-# holds the active global template, while the pipelex repo's `.pipelex/telemetry.toml`-# dogfoods the commented-out project template. The two files intentionally differ.-CONFIG_SYNC_EXCLUDED_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES | {"telemetry.toml"}+# Extends GIT_IGNORED_CONFIG_FILES with files that intentionally differ between the two sides:+# - telemetry.toml: the kit's holds the active global template, while the pipelex repo's+# `.pipelex/telemetry.toml` dogfoods the commented-out project template.+# - plxt.toml: the repo's `.pipelex/plxt.toml` adds a `[rule.schema]` override pointing at the+# locally generated `derived/mthds_schema.json` (this repo is the source of truth for the MTHDS+# language); the kit template must not reference that repo-internal artifact — plxt hard-fails+# on every .mthds file when the configured schema path does not exist.+CONFIG_SYNC_EXCLUDED_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES | {"telemetry.toml", "plxt.toml"}# Directories that should not be synced between .pipelex and kit/configs.# These are runtime directories created locally:diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex af46e756a..d99954dd9 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -7,6 +7,7 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.core.concepts.concept import Concept+from pipelex.core.concepts.exceptions import ConceptValueErrorfrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType@@ -150,7 +151,22 @@ class PipeParallel(PipeController):names: required fields ⊆ result names, result names ⊆ declared fields. This turns theruntime combine failure into an author-time error surfaced by `/validate`."""- structure_class = self.output.concept.get_structure_class()+ try:+ structure_class = self.output.concept.get_structure_class()+ except ConceptValueError as exc:+ # A plain ValueError would escape the /validate sweep and abort the whole bundle;+ # convert it so this pipe alone is reported as failed.+ msg = (+ f"PipeParallel '{self.code}' output '{self.output.concept.concept_ref}' has no registered "+ f"structure class ('{self.output.concept.structure_class_name}'): {exc}"+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,+ domain_code=self.domain_code,+ pipe_code=self.code,+ provided_concept_code=self.output.concept.concept_ref,+ ) from excif issubclass(structure_class, CompositeContent):return@@ -220,7 +236,23 @@ class PipeParallel(PipeController):if branch_pipe.output.multiplicity is not None:# A list-producing branch combines as a ListContent, not as the item class.continue- branch_structure_class = branch_pipe.output.concept.get_structure_class()+ try:+ branch_structure_class = branch_pipe.output.concept.get_structure_class()+ except ConceptValueError as exc:+ # Same conversion as the output lookup above: keep the failure per-pipe.+ msg = (+ f"PipeParallel '{self.code}' branch '{sub_pipe.pipe_code}' output "+ f"'{branch_pipe.output.concept.concept_ref}' has no registered structure class "+ f"('{branch_pipe.output.concept.structure_class_name}'): {exc}"+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,+ domain_code=self.domain_code,+ pipe_code=self.code,+ provided_concept_code=branch_pipe.output.concept.concept_ref,+ variable_names=[sub_pipe.output_name],+ ) from excif not issubclass(branch_structure_class, field_annotation):msg = (f"PipeParallel '{self.code}' branch '{sub_pipe.pipe_code}' produces "diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py b/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.pyindex c4de9517a..02886a97f 100644--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_img_gen/test_pipe_img_gen.py@@ -3,6 +3,7 @@import iofrom pathlib import Pathfrom urllib.parse import urlparse+from urllib.request import url2pathnameimport pytestfrom PIL import Image@@ -123,8 +124,11 @@ class TestPipeImgGen:parsed_url = urlparse(image_content.public_url)if parsed_url.scheme in {"http", "https"}:image_bytes = await fetch_file_from_url_httpx(url=image_content.public_url)+ elif parsed_url.scheme == "file":+ # file:// paths are URI-formatted (percent-encoded) — decode before touching the filesystem+ image_bytes = Path(url2pathname(parsed_url.path)).read_bytes()else:- image_bytes = Path(parsed_url.path).read_bytes()+ image_bytes = Path(image_content.public_url).read_bytes()with Image.open(io.BytesIO(image_bytes)) as generated_image:assert generated_image.size == (expected_width, expected_height), (f"{topic}: expected {expected_width}x{expected_height}, got {generated_image.size[0]}x{generated_image.size[1]}"diff --git a/tests/integration/pipelex/cogt/test_img2img.py b/tests/integration/pipelex/cogt/test_img2img.pyindex f29c830be..e4595c052 100644--- a/tests/integration/pipelex/cogt/test_img2img.py+++ b/tests/integration/pipelex/cogt/test_img2img.py@@ -32,7 +32,7 @@ class Img2ImgTestCases:# Single input image test cases: (topic, image_uri)SINGLE_INPUT_IMAGES: ClassVar[list[tuple[str, str]]] = [("PNG local file", ImageTestCases.IMAGE_FILE_PATH_PNG_1),- # ("JPG local file", ImageTestCases.IMAGE_FILE_PATH_JPG_3),+ ("JPG local file", ImageTestCases.IMAGE_FILE_PATH_JPG_3),]# Multiple input images test cases: (topic, list of image_uris)@@ -41,10 +41,10 @@ class Img2ImgTestCases:"Two JPG files",[ImageTestCases.IMAGE_FILE_PATH_JPG_1, ImageTestCases.IMAGE_FILE_PATH_JPG_3],),- # (- # "Mixed PNG and JPG",- # [ImageTestCases.IMAGE_FILE_PATH_PNG_1, ImageTestCases.IMAGE_FILE_PATH_JPG_1],- # ),+ (+ "Mixed PNG and JPG",+ [ImageTestCases.IMAGE_FILE_PATH_PNG_1, ImageTestCases.IMAGE_FILE_PATH_JPG_1],+ ),]# Remote URL input image test cases: (topic, image_url)diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_unresolvable_structure_class.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_unresolvable_structure_class.pynew file mode 100644index 000000000..80b6cf4ed--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_unresolvable_structure_class.py@@ -0,0 +1,93 @@+from typing import Callable++import pytest+from pydantic import Field++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.pipes.exceptions import PipeValidationError+from pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.hub import get_class_registry, get_concept_library, get_pipe_library+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint++DOMAIN_CODE = "test_usc"+++class UscRegisteredCombined(StructuredContent):+ """Registered combination target, used to reach the per-branch structure-class lookup."""++ summary: TextContent = Field(description="Summary produced by the branch")+++class TestPipeParallelUnresolvableStructureClass:+ """A missing structure class must surface as PipeValidationError, not a raw ConceptValueError.++ The `/validate` sweep only catches Pipelex-typed errors around per-pipe validation, so a+ bare ConceptValueError (a plain ValueError) escaping `validate_output_with_library` would+ abort the whole bundle instead of failing just this pipe.+ """++ def _setup_library(self, *, output_concept_code: str, branch_output_concept_code: str) -> PipeParallel:+ """Build a parallel whose output (or branch output) concept has an unregistered structure class."""+ concept_library = get_concept_library()+ for concept_code in {output_concept_code, branch_output_concept_code}:+ structure_class_name = "UscRegisteredCombined" if concept_code == "UscRegisteredCombined" else f"{concept_code}UnregisteredStructure"+ concept = ConceptFactory.make(+ concept_code=concept_code,+ domain_code=DOMAIN_CODE,+ description=f"Concept {concept_code}",+ structure_class_name=structure_class_name,+ )+ concept_library.add_concepts(concepts=[concept])++ pipe_library = get_pipe_library()+ branch_pipe = PipeFactory[PipeLLM].make_from_blueprint(+ domain_code=DOMAIN_CODE,+ pipe_code="usc_branch",+ blueprint=PipeLLMBlueprint(+ description="Branch producing the summary",+ inputs={"input_text": "Text"},+ output=f"{DOMAIN_CODE}.{branch_output_concept_code}" if branch_output_concept_code != "Text" else "Text",+ prompt="Summarize: $input_text",+ ),+ concept_codes_from_the_same_domain=[output_concept_code, branch_output_concept_code],+ )+ pipe_library.add_new_pipe(pipe=branch_pipe)++ pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(+ domain_code=DOMAIN_CODE,+ pipe_code="usc_parallel",+ blueprint=PipeParallelBlueprint(+ description="Parallel with an unresolvable output structure class",+ inputs={"input_text": "Text"},+ output=f"{DOMAIN_CODE}.{output_concept_code}",+ branches=[+ SubPipeBlueprint(pipe="usc_branch", result="summary"),+ ],+ ),+ concept_codes_from_the_same_domain=[output_concept_code, branch_output_concept_code],+ )+ pipe_library.add_new_pipe(pipe=pipe_parallel)+ return pipe_parallel++ def test_unregistered_output_structure_class_raises_pipe_validation_error(self, load_empty_library: Callable[[], str]):+ """The parallel's own output concept resolves to no registered class: must be a PipeValidationError."""+ load_empty_library()+ pipe_parallel = self._setup_library(output_concept_code="UscCombined", branch_output_concept_code="UscCombined")++ with pytest.raises(PipeValidationError, match="UscCombined"):+ pipe_parallel.validate_output_with_library()++ def test_unregistered_branch_structure_class_raises_pipe_validation_error(self, load_empty_library: Callable[[], str]):+ """A branch output concept resolving to no registered class: must be a PipeValidationError too."""+ load_empty_library()+ get_class_registry().register_class(UscRegisteredCombined)+ pipe_parallel = self._setup_library(output_concept_code="UscRegisteredCombined", branch_output_concept_code="UscBranchOnly")++ with pytest.raises(PipeValidationError, match="UscBranchOnly"):+ pipe_parallel.validate_output_with_library()diff --git a/tests/unit/pipelex/core/stuffs/test_html_rendering.py b/tests/unit/pipelex/core/stuffs/test_html_rendering.pynew file mode 100644index 000000000..86be9a635--- /dev/null+++ b/tests/unit/pipelex/core/stuffs/test_html_rendering.py@@ -0,0 +1,22 @@+from pipelex.core.stuffs.html_rendering import render_value_html+++class TestRenderValueHtml:+ """Plain strings must ALWAYS be escaped — a value that merely looks like HTML is untrusted input."""++ def test_html_looking_string_is_escaped(self):+ """A field value shaped like an HTML tag is an XSS vector and must not pass through raw."""+ result = render_value_html("<img src=x onerror=alert(1)>")+ assert result == "<img src=x onerror=alert(1)>"++ def test_plain_string_is_escaped(self):+ result = render_value_html("Tom & Jerry <3")+ assert result == "Tom & Jerry <3"++ def test_html_looking_string_inside_list_is_escaped(self):+ result = render_value_html(["<script>alert(1)</script>"])+ assert result == "<ul><li><script>alert(1)</script></li></ul>"++ def test_html_looking_string_inside_dict_is_escaped(self):+ result = render_value_html({"payload": "<b>bold</b>"})+ assert result == "<dl><dt>payload</dt><dd><b>bold</b></dd></dl>"diff --git a/wip/llm-max-tokens-fallback-dead-code.md b/wip/llm-max-tokens-fallback-dead-code.mdnew file mode 100644index 000000000..07a700df3--- /dev/null+++ b/wip/llm-max-tokens-fallback-dead-code.md@@ -0,0 +1,34 @@+# LLM `max_tokens` model fallback in `_apply_constraints` is (almost) dead code++**Status:** Deferred design tradeoff (not a regression — pre-existing on `main`, moved byte-equivalently by the plugins refactor). Flagged by cubic on PR #1020.++## Current behavior++`LLMWorkerAbstract._apply_constraints` (`pipelex/cogt/llm/llm_worker_abstract.py`; previously identical in `llm_worker_internal_abstract.py` on `main`) computes a max-tokens fallback but only ships it when a *temperature* rule fires:++```python+max_tokens = original_params.max_tokens or self.inference_model.max_tokens+new_max_tokens = max_tokens+has_changes = False+# ... only temperature constraints set has_changes = True ...+if not has_changes:+ return None # applied_job_params stays None → workers use the original params+return original_params.model_copy(update={"temperature": new_temperature, "max_tokens": new_max_tokens})+```++Every worker consumes `llm_job.applied_job_params or llm_job.job_params`, then sends `job_params.max_tokens or omit`. So when a job omits `max_tokens`:++- **No temperature constraint on the model** → no cap is sent; the provider default applies.+- **A temperature constraint fires** → the model's `max_tokens` from the spec IS sent as the request cap.++The cap a provider receives depends on an unrelated temperature rule — an inconsistency, though each individual outcome is defensible.++## Options++1. **Mark the fallback as a change** (cubic's suggestion: `has_changes = new_max_tokens != original_params.max_tokens`): the model spec's `max_tokens` is always sent explicitly when the job omits it. Consistent, "model constraints actually enforced" — but changes wire behavior for every provider/backend on every capless request.+2. **Remove the fallback**: `new_max_tokens = original_params.max_tokens` unchanged; when a temperature rule fires, the applied params keep the job's own (possibly absent) cap. Provider default semantics become uniform — also a wire behavior change, in the opposite direction, limited to constraint-firing models.+3. **Status quo**: keep the inconsistency documented here.++## Decision needed++Pick 1 or 2 in a normal dev cycle (not a release branch) with a targeted live-inference sanity pass, since either option changes what gets sent on the wire. Whichever way it goes, add a unit test on `_apply_constraints` pinning the chosen `max_tokens` semantics.diff --git a/wip/nested-composite-hydration-not-recursive.md b/wip/nested-composite-hydration-not-recursive.mdnew file mode 100644index 000000000..52cb735ef--- /dev/null+++ b/wip/nested-composite-hydration-not-recursive.md@@ -0,0 +1,22 @@+# Nested CompositeContent hydration is not recursive across the runtime bridge++**Status:** Deferred follow-up (confirmed defect, deliberately not fixed in the v0.37.0 release). Flagged by codex and cubic on PR #1020.++## Problem++The transport encode side recurses, the hydration side doesn't:++- **Encode** (`pipelex/core/memory/working_memory.py`): `_encode_composite_for_transport` sends every composite component through `_encode_content_with_class_markers`, which recurses back into `_encode_composite_for_transport` when the component is itself a `CompositeContent`. Inner components are correctly stamped with `__pipelex_class__` / `__pipelex_module__` markers at every depth.+- **Hydrate** (`pipelex/runtime_bridge/primitives/hydration.py`): `_hydrate_list_item`'s marker branch strips the markers only at the top level and calls `item_class.model_validate(clean_item)`. When `item_class` is a `CompositeContent` subclass (extra="allow" — no annotations to drive validation), the inner components are left as raw marker-bearing dicts instead of typed `StuffContent`.++Consequence: a `CompositeContent` nested inside another `CompositeContent` (e.g. a `PipeParallel` result feeding another parallel's combine, crossing the Temporal/runtime-bridge boundary via `dump_for_transport()` → `hydrate_working_memory()`) round-trips with corrupted typed access — downstream `rendered_*` calls and type checks see dicts with `__pipelex_*` keys.++One level of composite (the common case, top-level composite stuff) hydrates correctly via `hydrate_content`'s composite branch; only *nested* composites are affected.++## Proposed fix++In `_hydrate_list_item` (hydration.py), in the marker branch: after resolving `item_class` and building `clean_item`, when `issubclass(item_class, CompositeContent)`, map each value of `clean_item` through the existing `_hydrate_composite_component` before `model_validate`. The mutual recursion `_hydrate_composite_component` ↔ `_hydrate_list_item` then handles arbitrary nesting depth.++## Test shape (TDD)++In `tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py`: build a `WorkingMemory` whose main stuff is a `CompositeContent` containing (a) a nested `CompositeContent` component and (b) a list-of-composites component; run `dump_for_transport()` → `json.dumps`/`json.loads` → `hydrate_working_memory()`; assert every nested component is a typed `StuffContent` instance and no `__pipelex_class__` / `__pipelex_module__` key survives anywhere in the tree.