Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Step-multiplicity-corpus-entry · L-261006-db2674 (#1357)
Louis Choquel46305dea266a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d9e01f72a..e35f40251 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,7 @@### Added- **A run request carries the method packages its bundle calls (Breaking)**: a `files` or `bundle_b64` request to `/v1/execute` or `/v1/start` may ship each package its bundle calls by address under `.mthds/methods/<name>/` at its root, with its `METHODS.toml`, and the server searches them before its installed copies and fetch-on-miss, for that run alone, so a shipped package wins over an installed copy and runs with fetching off; a package fetched by `method_ref` that ships its own `.mthds/methods/` is handled the same way. Those files no longer join the bundle's own `.mthds` content or its library directory, any other `.mthds` path in a bundle is refused with a `422 InvalidBundle` naming the entry, as are two shipped packages declaring the same address and name without regard to case, and on a sandbox-hosted deployment a shipped package declaring a Python structure class is refused with `403 MethodStructuresRefusedError`. The engine takes these directories as the new keyword-only `methods_dirs` of `PipelexMTHDSProtocol`, `pipeline_run_setup`, `acquire_library` and `LibraryManagerAbstract.load_from_blueprints`, which every implementation of the abstract must accept.+- **The MTHDS Test Corpus covers a sequence step's multiplicity**: the invalid entry `invalid_input_stuff_spec_mismatch_step_multiplicity` has a step read as a single `Page` the pages an earlier extraction stored as `Page[]`, and fails with `input_stuff_spec_mismatch`; the valid offline entry `controller_sequence_house_rules` declares `house_rules = "Markdown"` for a sequence whose first step reads it as `Markdown` and whose second reads it as `Text`, and runs it.- **The MTHDS Test Corpus covers the `markdown` template filter**: the new hand-maintained `feature.markdown_filter` tag names the filter of HTML templates that turns Markdown held in a plain text field into HTML, and the offline-tier entry `feature_markdown_filter_cake_order_slip` covers it with a PipeCompose run that pins its output, raw HTML in the source shown as text and only a URL with a scheme made a link.### Changed@@ -15,6 +16,7 @@### Fixed+- **A validation message no longer names one concept as both required and provided**: the `(required: …, provided: …)` suffix of a validation item is left off when the provided concept is among the required ones, since the disagreement then lies elsewhere, so a step reading one `Page` where an earlier step stored `Page[]` no longer ends its message with `(required: native.Page, provided: native.Page)`.- **A provenance sidecar that is not UTF-8 no longer breaks method discovery**: an installed or shipped method package whose `.provenance.json` is not valid UTF-8 is discovered with no recorded provenance, as one that does not parse already was, where discovery raised `UnicodeDecodeError` and a run calling the package answered a `500`.- **A `files` entry holding a lone surrogate is refused as an invalid bundle**: a name or a text in a run request's `files` map holding an escaped lone surrogate such as `"\ud800"`, which JSON carries but UTF-8 cannot encode, is refused with a `422 InvalidBundle` naming the entry, where the server answered a `500`.- **A bundle naming one path twice is refused**: a run request whose bundle holds two entries for one file, spelled differently in a `files` map (`a/./b` beside `a/b`) or twice in a zip, or a file that is also the directory of another entry, is refused with a `422 InvalidBundle` naming the path. The shipped packages' manifest check read the first copy while the engine read the last, so two packages of one address could get past it, and the second shape answered a `500`.diff --git a/pipelex/mthds_parsing/handle_pipe_errors.py b/pipelex/mthds_parsing/handle_pipe_errors.pyindex f3b389e5b..a308a44f0 100644--- a/pipelex/mthds_parsing/handle_pipe_errors.py+++ b/pipelex/mthds_parsing/handle_pipe_errors.py@@ -191,11 +191,18 @@ def categorize_pipe_validation_with_libraries_error("""message = pipe_error.explanation or str(pipe_error)error_type = pipe_error.error_type or PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR- # The required/provided concept refs are a debugging aid appended to the message. Suppress them- # for multiplicity errors (identical concept on both sides — see is_inadequate_output_multiplicity)- # and otherwise render the required refs as joined author-syntax refs, never a Python list repr+ # The required/provided concept refs are a debugging aid appended to the message, for a disagreement on+ # the concept. Suppress them for multiplicity errors (see is_inadequate_output_multiplicity) and whenever+ # the provided concept is among the required ones, since the disagreement then lies elsewhere — a step+ # reading one `Page` where a list of them is stored — and the suffix would print the same ref on both+ # sides. Otherwise render the required refs as joined author-syntax refs, never a Python list repr# (whose `['x']` brackets read like MTHDS `[]` multiplicity syntax).- if not error_type.is_inadequate_output_multiplicity and pipe_error.required_concept_codes and pipe_error.provided_concept_code:+ if (+ not error_type.is_inadequate_output_multiplicity+ and pipe_error.required_concept_codes+ and pipe_error.provided_concept_code+ and pipe_error.provided_concept_code not in pipe_error.required_concept_codes+ ):required_refs = ", ".join(pipe_error.required_concept_codes)message += f" (required: {required_refs}, provided: {pipe_error.provided_concept_code})"return PipesAndConceptValidationErrorData(diff --git a/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/bundle.mthdsnew file mode 100644index 000000000..2ea450d47--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/bundle.mthds@@ -0,0 +1,28 @@+domain = "holiday_lets"+description = "Passing a holiday let's house rules on to arriving guests, pinned in the hall and in a text message"+main_pipe = "pass_on_house_rules"++[pipe.pass_on_house_rules]+type = "PipeSequence"+description = "Lays the house rules out as the notice pinned in the hall, then writes the text message sent to arriving guests"+# The first step reads `house_rules` as Markdown and the second as Text: the sequence declares the Markdown, which both accept.+inputs = { house_rules = "Markdown" }+output = "Text"+steps = [+ { pipe = "lay_out_hall_notice", result = "hall_notice" },+ { pipe = "write_arrival_message", result = "arrival_message" },+]++[pipe.lay_out_hall_notice]+type = "PipeCompose"+description = "Lays the house rules out as the notice pinned in the hall"+inputs = { house_rules = "Markdown" }+output = "Markdown"+template = "# House rules\n\n$house_rules"++[pipe.write_arrival_message]+type = "PipeCompose"+description = "Writes the text message sent to guests on the day they arrive"+inputs = { house_rules = "Text" }+output = "Text"+template = "Welcome! A few house rules before you settle in: $house_rules"diff --git a/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/entry.toml b/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/entry.tomlnew file mode 100644index 000000000..bd13535b2--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/entry.toml@@ -0,0 +1,10 @@+name = "controller_sequence_house_rules"+description = "A holiday let's house rules declared as Markdown, laid out as a hall notice and then read as plain text for the arrival message"++validity = "valid"+tier = "offline"+granularity = "focused"++covers = [+ "controller.sequence",+]diff --git a/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/inputs.json b/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/inputs.jsonnew file mode 100644index 000000000..fb4e439da--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/controller_sequence_house_rules/inputs.json@@ -0,0 +1,8 @@+{+ "house_rules": {+ "concept": "native.Markdown",+ "content": {+ "text": "- Quiet after ten in the evening\n- No smoking indoors\n- Bins out on Tuesday evening"+ }+ }+}diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_step_multiplicity/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_step_multiplicity/bundle.mthdsnew file mode 100644index 000000000..62e45c11f--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_step_multiplicity/bundle.mthds@@ -0,0 +1,27 @@+domain = "job_applications"+description = "Sizing up a job applicant from the scan of their CV"+main_pipe = "size_up_applicant"++[pipe.size_up_applicant]+type = "PipeSequence"+description = "Reads the pages out of the scanned CV, then sums up the applicant's work experience"+inputs = { cv_scan = "Document" }+output = "Text"+# Wrong on purpose: `read_cv_pages` stores a list of pages under `cv_pages`, which `sum_up_experience` reads as a single page.+steps = [+ { pipe = "read_cv_pages", result = "cv_pages" },+ { pipe = "sum_up_experience", result = "experience" },+]++[pipe.read_cv_pages]+type = "PipeExtract"+description = "Reads each page out of the scanned CV"+inputs = { cv_scan = "Document" }+output = "Page[]"++[pipe.sum_up_experience]+type = "PipeLLM"+description = "Sums up the work experience a CV lists"+inputs = { cv_pages = "Page" }+output = "Text"+prompt = "Sum up in a few lines the work experience this CV lists: $cv_pages"diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_step_multiplicity/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_step_multiplicity/entry.tomlnew file mode 100644index 000000000..511906d87--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_step_multiplicity/entry.toml@@ -0,0 +1,12 @@+name = "invalid_input_stuff_spec_mismatch_step_multiplicity"+description = "The pages read out of a scanned CV, stored as a list by one sequence step and read as a single page by the next"++validity = "invalid"+tier = "static"+granularity = "focused"++covers = [+ "error.input_stuff_spec_mismatch",+]++expected_error = "input_stuff_spec_mismatch"diff --git a/tests/integration/pipelex/test_extras/test_data.py b/tests/integration/pipelex/test_extras/test_data.pyindex b556b327d..8f67083a5 100644--- a/tests/integration/pipelex/test_extras/test_data.py+++ b/tests/integration/pipelex/test_extras/test_data.py@@ -189,4 +189,23 @@ class OfflineRunExpectations:),"loading_list": Bound(concept_ref="native.Text", content={"text": "- QP-1001\n- QP-1002\n- BR-2001\n"}),},+ "controller_sequence_house_rules": {+ "house_rules": Bound(+ concept_ref="native.Markdown",+ content={"text": "- Quiet after ten in the evening\n- No smoking indoors\n- Bins out on Tuesday evening"},+ ),+ "hall_notice": Bound(+ concept_ref="native.Markdown",+ content={"text": "# House rules\n\n- Quiet after ten in the evening\n- No smoking indoors\n- Bins out on Tuesday evening"},+ ),+ "arrival_message": Bound(+ concept_ref="native.Text",+ content={+ "text": (+ "Welcome! A few house rules before you settle in: "+ "- Quiet after ten in the evening\n- No smoking indoors\n- Bins out on Tuesday evening"+ )+ },+ ),+ },}diff --git a/tests/unit/pipelex/mthds_parsing/test_handle_pipe_errors_suffix.py b/tests/unit/pipelex/mthds_parsing/test_handle_pipe_errors_suffix.pyindex 740094391..0a47df552 100644--- a/tests/unit/pipelex/mthds_parsing/test_handle_pipe_errors_suffix.py+++ b/tests/unit/pipelex/mthds_parsing/test_handle_pipe_errors_suffix.py@@ -1,3 +1,5 @@+import pytest+from pipelex.core.pipes.exceptions import PipeValidationErrorfrom pipelex.mthds_parsing.handle_pipe_errors import categorize_pipe_validation_with_libraries_errorfrom pipelex.validation_error_types import PipeValidationErrorType@@ -7,10 +9,37 @@ class TestRequiredProvidedSuffix:"""The ``(required: …, provided: …)`` suffix the pipe categorizer appends to the message.Two rules: it is suppressed entirely for multiplicity errors (identical concept on both sides- by definition), and for the cases where it stays the required refs render as joined author-syntax- refs — never a Python ``['x']`` list repr whose brackets masquerade as ``[]`` multiplicity syntax.+ by definition) and whenever the provided concept is among the required ones, and for the cases+ where it stays the required refs render as joined author-syntax refs — never a Python ``['x']``+ list repr whose brackets masquerade as ``[]`` multiplicity syntax."""+ @pytest.mark.parametrize(+ ("required_concept_codes", "provided_concept_code"),+ [+ pytest.param(["native.Page"], "native.Page", id="the-same-concept-on-both-sides"),+ pytest.param(["native.Text", "native.Html"], "native.Html", id="the-provided-concept-among-several-required"),+ ],+ )+ def test_provided_concept_among_the_required_suppresses_the_suffix(self, required_concept_codes: list[str], provided_concept_code: str) -> None:+ # A sequence step reading one `Page` where an earlier step stored `Page[]`: the concepts agree, the multiplicity does not.+ base_message = (+ "In pipe 'size_up_applicant', step 2 (pipe 'sum_up_experience') reads 'cv_pages' as 'Page', but step 1 (pipe 'read_cv_pages') "+ "stores it as 'Page[]'."+ )+ pipe_error = PipeValidationError(+ message=base_message,+ error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,+ domain_code="job_applications",+ pipe_code="size_up_applicant",+ required_concept_codes=required_concept_codes,+ provided_concept_code=provided_concept_code,+ )++ error_data = categorize_pipe_validation_with_libraries_error(pipe_error)++ assert error_data.message == base_message+def test_multiplicity_error_suppresses_the_suffix(self) -> None:base_message = "output multiplicity mismatch: 'brainstorm' declares 'StoryIdea' but yields 'StoryIdea[]'."pipe_error = PipeValidationError(