Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Io-multiplicity · L-260926-7565a9 (#1278)
Louis Choqueld93a24767fcc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 15fbf3382..7aa79d1c5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,7 @@- **An unknown model is a located `unknown_model` validation item**: a pipe whose model field names a handle, alias, preset or waterfall the model deck does not define now validates to an invalid verdict with one `pipe_validation` item of the new closed `error_type` `unknown_model`, carrying the `pipe_code`, `domain_code`, `source`, `field_name`, the `field_path` of the reference (`pipe.<code>.model`, or `pipe.<code>.model_to_structure`), and the new optional item fields `model_reference` (the reference as written), `model_type` and `suggestions` (the deck's close matches of the same kind, also kept in the message). Every pipe type that names a model gives the same item, where `PipeExtract` and `PipeSearch` used to give an `unknown_validation_error` whose message was a Python repr, and with exactly one suggestion the item carries an unsafe `rename-model` fix, which `pipelex fix bundle` never applies on its own because a fuzzy match can name a different model. The MTHDS Test Corpus gains the `invalid_unknown_model` entry covering the new `error.unknown_model` tag.- **An unsafe suggested fix says so wherever it is printed**: the per-error line of `pipelex validate`, of `pipelex fix`'s remaining errors and of every `pipelex-agent` Markdown rendering, the `rendered_markdown` shared with `/validate` included, reads `💡 Suggested fix (unsafe, confirm before applying):` for a fix whose `safety` is `unsafe`, while a safe fix keeps the plain `💡 Suggested fix:` line, so a person or an agent can tell which fixes to check before making them by hand.+- **A run graph marks a list-valued stuff as a list**: every io item of a GraphSpec produced by a dry or live run now carries `multiplicity`, `true` when the stuff is a list and `null` otherwise, so a renderer shows an input declared `Document[]`, a `Record[]` output, a batch's aggregate, a step run with `nb_output` or a lifted plural output as a list, on both the in-process graph and the one assembled from trace events. A trace event carrying the field cannot be read by an earlier pipelex, so the process that assembles a graph must run a pipelex at least as new as the workers that wrote its events. The Mermaid renderer labels such a stuff `Record[]`, and its labels now keep square brackets instead of turning them into parentheses.- **Validation items carry the positions and declared concepts the parser knew**: a TOML syntax error's item carries the 1-based `line` and `column` the parser stopped at, as new optional item fields, and an `unresolved_concept` item carries `declared_concepts`, the bare codes of the concepts the validated bundle declares in its domain, so an editor can anchor the diagnostic and an author sees what the reference could have named.### Changeddiff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.mdindex 051cf3781..57c428370 100644--- a/docs/under-the-hood/execution-graph-tracing.md+++ b/docs/under-the-hood/execution-graph-tracing.md@@ -479,9 +479,12 @@ class IOSpec(BaseModel):size: int | None # Content sizedigest: str | None # Unique identifier for data flowdata: str | dict[str, Any] | list[str] | list[dict[str, Any]] | None # Full serialized content+ multiplicity: bool | int | None # True for a list, None for a single valueextra: dict[str, Any] # Extra markers, e.g. `optional` set on a declared-optional (`?`) output```+Every site that records a stuff on a node builds its IOSpec with `make_stuff_io_spec` (`pipelex/graph/stuff_io_spec.py`): a pipe's declared inputs at its start, its main output at its end, a lifted pipe's plural companion slots, and a `PipeParallel`'s branch outputs. `concept` is always the bare concept code, and `multiplicity` says whether the stuff is a list, in the encoding the pipe registry uses for a stuff spec: `true` for a variable-length list, a positive integer for a fixed count, `null` for a single value. A run reads it off the value it holds, so a list is `true` whatever its length, an empty list from a lifted plural output included, and a batch's item is single while its aggregate is a list. It never writes the item count, which would make a one-item list read as single and an empty one write `0`. Because the trace events carry the IOSpec whole, both graph builders show the same marker, and a stuff reads the same on its producer's io item and on its consumers'. The Mermaid renderer labels such a stuff `Record[]`.+!!! warning "Preview Truncation"Previews are automatically truncated to 200 characters. Stack traces are truncated to 2000 characters. Use `--graph-full-data` to capture complete content.diff --git a/pipelex/graph/graph_analysis.py b/pipelex/graph/graph_analysis.pyindex 46febc968..99cb15efc 100644--- a/pipelex/graph/graph_analysis.py+++ b/pipelex/graph/graph_analysis.py@@ -13,7 +13,7 @@ from typing import Anyfrom pydantic import BaseModel, ConfigDict, Field-from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeSpec+from pipelex.graph.graphspec import EdgeKind, GraphSpec, IOMultiplicity, NodeSpec, make_io_concept_labelfrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of@@ -28,8 +28,14 @@ class StuffInfo(BaseModel):name: strconcept: str | None = None+ multiplicity: IOMultiplicity | None = Nonedata: str | dict[str, Any] | list[str] | list[dict[str, Any]] | None = None+ @property+ def concept_label(self) -> str | None:+ """The concept as a renderer labels this stuff, with its multiplicity marker (`Record[]`)."""+ return make_io_concept_label(concept=self.concept, multiplicity=self.multiplicity)+class GraphAnalysis(BaseModel):"""Pre-computed analysis of a GraphSpec for rendering.@@ -136,6 +142,7 @@ class GraphAnalysis(BaseModel):stuff_registry[output_spec.digest] = StuffInfo(name=output_spec.name,concept=output_spec.concept,+ multiplicity=output_spec.multiplicity,data=output_spec.data,)stuff_producers[output_spec.digest] = node.node_id@@ -148,6 +155,7 @@ class GraphAnalysis(BaseModel):stuff_registry[input_spec.digest] = StuffInfo(name=input_spec.name,concept=input_spec.concept,+ multiplicity=input_spec.multiplicity,data=input_spec.data,)stuff_consumers[input_spec.digest].append(node.node_id)diff --git a/pipelex/graph/graphspec.py b/pipelex/graph/graphspec.pyindex 30c71e040..b82a92325 100644--- a/pipelex/graph/graphspec.py+++ b/pipelex/graph/graphspec.py@@ -7,7 +7,7 @@ GraphSpec is renderer-agnostic and designed for JSON serialization.from collections.abc import Sequencefrom datetime import datetimefrom enum import StrEnum-from typing import Any, Self+from typing import Annotated, Any, Selffrom pydantic import BaseModel, ConfigDict, Field, computed_field, field_validator, model_validator@@ -185,6 +185,11 @@ class TimingSpec(BaseModel):return data+# How many values a stuff holds on a GraphSpec io item: `True` for a variable-length list, a+# positive integer for a fixed count, `False` for a single value (as is `None` where it is optional).+IOMultiplicity = bool | Annotated[int, Field(gt=0)]++class IOSpec(BaseModel):"""Specification for an input or output variable.@@ -193,6 +198,11 @@ class IOSpec(BaseModel):The optional `data` field can hold the full serialized content whenfull data capture is enabled (via --graph-full-data CLI option).++ `concept` is always the bare concept code; whether the stuff is a list is `multiplicity`,+ in the encoding the pipe registry uses for a stuff spec's multiplicity: `True` for a+ variable-length list, a positive integer for a fixed count, `None` (or `False`) for a single+ value, a count of one reading as single. A reader takes absence as single."""model_config = ConfigDict(extra="forbid", strict=True)@@ -204,6 +214,7 @@ class IOSpec(BaseModel):size: int | None = Nonedigest: str | None = Nonedata: str | dict[str, Any] | list[str] | list[dict[str, Any]] | None = None+ multiplicity: IOMultiplicity | None = Noneextra: dict[str, Any] = Field(default_factory=dict)@field_validator("preview", mode="after")@@ -212,6 +223,25 @@ class IOSpec(BaseModel):"""Truncate preview to MAX_PREVIEW_LENGTH."""return _truncate_string(value, max_length=MAX_PREVIEW_LENGTH)+ @property+ def concept_label(self) -> str | None:+ """The concept as a renderer labels this stuff, with its multiplicity marker (`Record[]`)."""+ return make_io_concept_label(concept=self.concept, multiplicity=self.multiplicity)+++def make_io_concept_label(*, concept: str | None, multiplicity: IOMultiplicity | None) -> str | None:+ """A concept with the marker an author would write for the multiplicity: `[]`, `[N]` or nothing.++ A count of one, `False` and `None` all read as single and get no marker. `None` when there is no concept.+ """+ if concept is None:+ return None+ if multiplicity is True:+ return f"{concept}[]"+ if isinstance(multiplicity, int) and not isinstance(multiplicity, bool) and multiplicity > 1:+ return f"{concept}[{multiplicity}]"+ return concept+def output_digest_is_optional(*, output_specs: Sequence[IOSpec], digest: str) -> bool:"""Whether a producer registered ``digest`` as a declared-optional (`?`) output.diff --git a/pipelex/graph/mermaidflow/mermaidflow_factory.py b/pipelex/graph/mermaidflow/mermaidflow_factory.pyindex 4c887db92..41c2dbd88 100644--- a/pipelex/graph/mermaidflow/mermaidflow_factory.py+++ b/pipelex/graph/mermaidflow/mermaidflow_factory.py@@ -103,7 +103,7 @@ class MermaidflowFactory:# Build stuff registry as tuple format for compatibility with rendering codestuff_registry: dict[str, tuple[str, str | None]] = {}for digest, stuff_info in analysis.stuff_registry.items():- stuff_registry[digest] = (stuff_info.name, stuff_info.concept)+ stuff_registry[digest] = (stuff_info.name, stuff_info.concept_label)# Will be populated during renderingstuff_id_mapping: dict[str, str] = {}@@ -139,7 +139,7 @@ class MermaidflowFactory:if controller_node:for output_spec in controller_node.node_io.outputs:if output_spec.digest and output_spec.digest in digest_map:- digest_map[output_spec.digest] = (output_spec.name, output_spec.concept)+ digest_map[output_spec.digest] = (output_spec.name, output_spec.concept_label)# Render pipe nodes and their produced stuff within controller subgraphslines.append("")@@ -223,7 +223,7 @@ class MermaidflowFactory:for node in graph.nodes:for output_spec in node.node_io.outputs:if output_spec.digest and output_spec.digest not in all_stuff_info:- all_stuff_info[output_spec.digest] = (output_spec.name, output_spec.concept)+ all_stuff_info[output_spec.digest] = (output_spec.name, output_spec.concept_label)# Render edges: producer -> stufflines.append("")diff --git a/pipelex/graph/mermaidflow/stuff_collector.py b/pipelex/graph/mermaidflow/stuff_collector.pyindex 8f1e6afe0..185966b07 100644--- a/pipelex/graph/mermaidflow/stuff_collector.py+++ b/pipelex/graph/mermaidflow/stuff_collector.py@@ -100,8 +100,9 @@ def collect_stuff_metadata(graph: GraphSpec) -> dict[str, dict[str, str]]:def extract_metadata(io_spec: IOSpec) -> dict[str, str]:meta: dict[str, str] = {"name": io_spec.name}- if io_spec.concept:- meta["concept"] = io_spec.concept+ concept_label = io_spec.concept_label+ if concept_label:+ meta["concept"] = concept_labelreturn metareturn _collect_stuff_field(graph, extractor=extract_metadata)diff --git a/pipelex/graph/stuff_io_spec.py b/pipelex/graph/stuff_io_spec.pynew file mode 100644index 000000000..b860fa6ed--- /dev/null+++ b/pipelex/graph/stuff_io_spec.py@@ -0,0 +1,49 @@+"""The GraphSpec io item a pipe run writes for a stuff it holds.++Every site that records a stuff on a graph node (a pipe's declared inputs at its start, its main+output at its end, a lifted pipe's plural companion slots, a parallel's branch outputs) builds the+item here, so what the item says about the stuff, its multiplicity above all, cannot drift between+them. Both graph builders read these items as they were written: the in-process `GraphTracer`+directly, and the event-replay `GraphSpecAssembler` through the trace events that carry them whole.+"""++from typing import TYPE_CHECKING, Any++from pipelex.graph.graphspec import IOSpec++if TYPE_CHECKING:+ from pipelex.core.stuffs.stuff import Stuff+++def make_stuff_io_spec(*, name: str, stuff: "Stuff", include_data: bool, extra: dict[str, Any] | None = None) -> IOSpec:+ """Build the io item for `stuff`, recorded on its node under `name`.++ Args:+ name: The variable name the node knows the stuff by.+ stuff: The stuff itself, as the run holds it.+ include_data: Whether to embed the stuff's full serialized content (full data capture).+ extra: Extra markers for the item, such as the optional-output marker.+ """+ return IOSpec(+ name=name,+ concept=stuff.concept.code,+ content_type=stuff.content.content_type,+ digest=stuff.stuff_code,+ data=stuff.content.smart_dump() if include_data else None,+ multiplicity=stuff_io_multiplicity(stuff=stuff),+ extra=extra or {},+ )+++def stuff_io_multiplicity(*, stuff: "Stuff") -> bool | None:+ """The multiplicity an io item records for `stuff`: `True` for a list, `None` for a single value.++ It is read off the value, not off any declaration: by the time a stuff exists the runtime has+ applied every rule that decides whether it is a list (a batch's item is single and its+ aggregate a list, a lifted plural output an empty list, a step run with `nb_output` a list), so+ the value is the one reading that the producer and every consumer of a stuff agree on.++ Never the item count. A variable-length list that produced one item would read as single, and+ an empty one would write `0`, which a GraphSpec reader refuses. `True` is truthful for every list.+ """+ return True if stuff.is_list else Nonediff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex 0be3d86e6..1727f8b3f 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -25,7 +25,7 @@ from pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.graph.graph_tracer_manager import GraphTracerManager-from pipelex.graph.graphspec import IOSpec+from pipelex.graph.stuff_io_spec import make_stuff_io_specfrom pipelex.interpreter_hub import get_concept_library, get_optional_pipe, get_required_pipefrom pipelex.libraries.pipe.exceptions import PipeNotFoundErrorfrom pipelex.pipe_controllers.absence_taint import (@@ -774,12 +774,10 @@ class PipeParallel(PipeController):if tracer_manager is None or trace_context.parent_node_id is None:returnfor output_name_key, output_stuff in output_stuffs.items():- output_spec = IOSpec(+ output_spec = make_stuff_io_spec(name=output_name_key,- concept=output_stuff.concept.code,- content_type=output_stuff.content.content_type,- digest=output_stuff.stuff_code,- data=output_stuff.content.smart_dump() if trace_context.data_inclusion.stuff_json_content else None,+ stuff=output_stuff,+ include_data=trace_context.data_inclusion.stuff_json_content,)tracer_manager.register_controller_output(lookup_key=trace_context.lookup_key,diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.pyindex dae189dfa..675f4e524 100644--- a/pipelex/pipe_machinery/pipe_abstract.py+++ b/pipelex/pipe_machinery/pipe_abstract.py@@ -24,6 +24,7 @@ from pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff import Stufffrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.graph.graph_tracer_manager import GraphTracerManager, IOSpec, NodeKind+from pipelex.graph.stuff_io_spec import make_stuff_io_specfrom pipelex.libraries.library_crate import LibraryCratefrom pipelex.pipe_machinery.pipe_blueprint import PipeCategory, PipeType, valid_pipe_type_tagsfrom pipelex.pipe_machinery.validation import is_variable_satisfied_by_inputs@@ -682,14 +683,10 @@ class PipeAbstract(ABC, BaseModel):# then discarded. The lightweight IOSpec (name/concept/content_type/digest) is kept# so node ids and usage-event correlation are unaffected.include_graph_data = parent_trace_context.emit_graph_events- input_spec = IOSpec(+ input_spec = make_stuff_io_spec(name=var_name,- concept=stuff.concept.code,- content_type=stuff.content.content_type,- digest=stuff.stuff_code,- data=stuff.content.smart_dump()- if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content)- else None,+ stuff=stuff,+ include_data=include_graph_data and parent_trace_context.data_inclusion.stuff_json_content,)input_specs.append(input_spec)@@ -790,15 +787,13 @@ class PipeAbstract(ABC, BaseModel):main_stuff = main_resolved# E1: same gating as the input block — skip the discarded payload dumps in costs-only mode.include_graph_data = parent_trace_context.emit_graph_events- output_spec = IOSpec(+ output_spec = make_stuff_io_spec(name=output_name or main_stuff.stuff_name or "main_stuff",- concept=main_stuff.concept.code,- content_type=main_stuff.content.content_type,- digest=main_stuff.stuff_code,- data=main_stuff.content.smart_dump() if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content) else None,+ stuff=main_stuff,+ include_data=include_graph_data and parent_trace_context.data_inclusion.stuff_json_content,# The optional-edge marker (D8): a data edge fed by this output reports that the# value may be absent in other runs.- extra={"optional": True} if self.output.presence.is_optional else {},+ extra={"optional": True} if self.output.presence.is_optional else None,)# Serialize output concept for registry if enabled (E1: also gated on emit_graph_events).@@ -827,11 +822,10 @@ class PipeAbstract(ABC, BaseModel):tracer_manager.register_controller_output(lookup_key=parent_trace_context.lookup_key,node_id=graph_node_id,- output_spec=IOSpec(+ output_spec=make_stuff_io_spec(name=companion_slot.slot_name,- concept=companion_stuff.concept.code,- content_type=companion_stuff.content.content_type,- digest=companion_stuff.stuff_code,+ stuff=companion_stuff,+ include_data=False,),)else:diff --git a/pipelex/tools/mermaid/mermaid_utils.py b/pipelex/tools/mermaid/mermaid_utils.pyindex 4afde5442..fe00c3ac8 100644--- a/pipelex/tools/mermaid/mermaid_utils.py+++ b/pipelex/tools/mermaid/mermaid_utils.py@@ -114,7 +114,9 @@ def sanitize_mermaid_id(node_id: str) -> str:def escape_mermaid_label(label: str) -> str:"""Escape special characters in Mermaid labels.- Handles characters that could break Mermaid syntax or inject directives.+ Handles characters that could break Mermaid syntax or inject directives. Square brackets+ become Mermaid entity codes, which render as the brackets themselves, so a multiplicity+ marker such as `Record[]` reads as written.Args:label: The label text to escape.@@ -125,8 +127,8 @@ def escape_mermaid_label(label: str) -> str:result = labelresult = result.replace("\\", "\\\\") # Escape backslashes firstresult = result.replace('"', "'")- result = result.replace("[", "(")- result = result.replace("]", ")")+ result = result.replace("[", "#91;")+ result = result.replace("]", "#93;")result = result.replace("{", "(")result = result.replace("}", ")")result = result.replace("<", "<")diff --git a/tests/integration/pipelex/pipeline/test_run_graph_io_multiplicity.py b/tests/integration/pipelex/pipeline/test_run_graph_io_multiplicity.pynew file mode 100644index 000000000..10e9bf97f--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_run_graph_io_multiplicity.py@@ -0,0 +1,303 @@+"""A run-produced graph marks every list-valued stuff as a list, on both graph builders.++The io items a run writes carry `multiplicity: True` exactly when the stuff is a list, read off the+value the runtime holds: a method input declared `Text[]`, an output declared `Record[]`, a batch's+aggregate, a step run with `nb_output`, a parallel's list-valued branch output, and a lifted plural+output's empty list. A batch's item and every single value carry no multiplicity. A stuff reads the+same on its producer's io item and on its consumers'.++Each run is read through both builders at once: the GraphSpec the run returns is the event-replay+`GraphSpecAssembler`'s, and the in-process `GraphTracer`'s is captured from its teardown.+"""++from typing import Any++import pytest+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.graph.graph_tracer import GraphTracer+from pipelex.graph.graphspec import GraphSpec, IOSpec, NodeSpec, NodeStatus+from pipelex.pipeline.dry_run_pipeline import dry_run_pipeline+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.runtime_hub import scoped_event_log+from pipelex.system.registries.func_registry import func_registry+from pipelex.tracing.in_memory_event_log import InMemoryEventLog++_DRY_BUNDLE = '''+domain = "io_multiplicity_dry"+description = "Every way a run makes a list, and the single values around them"+main_pipe = "process_documents"++[concept.Record]+description = "A record extracted from the documents"++[concept.Record.structure]+title = { type = "text", description = "The record's title" }++[pipe.process_documents]+type = "PipeSequence"+description = "Extract, batch, brainstorm, fan out, conclude"+inputs = { documents = "Text[]" }+output = "Text"+steps = [+ { pipe = "extract_records", result = "records" },+ { pipe = "summarize_record", batch_over = "records", batch_as = "record", result = "summaries" },+ { pipe = "brainstorm", result = "ideas", nb_output = 3 },+ { pipe = "assess", result = "assessment" },+ { pipe = "conclude", result = "conclusion" },+]++[pipe.extract_records]+type = "PipeLLM"+description = "A declared list output"+inputs = { documents = "Text[]" }+output = "Record[]"+prompt = """+Extract the records from these documents:++@documents+"""++[pipe.summarize_record]+type = "PipeLLM"+description = "The batch's body, run once per record"+inputs = { record = "Record" }+output = "Text"+prompt = "Summarize $record"++[pipe.brainstorm]+type = "PipeLLM"+description = "A single output made a list by the step's nb_output"+inputs = { summaries = "Text[]" }+output = "Text"+prompt = """+Brainstorm an idea from these summaries:++@summaries+"""++[pipe.assess]+type = "PipeParallel"+description = "One list-valued branch and one single-valued branch"+inputs = { ideas = "Text[]" }+output = "Composite"+add_each_output = true+branches = [+ { pipe = "list_risks", result = "risks" },+ { pipe = "pick_idea", result = "pick" },+]++[pipe.list_risks]+type = "PipeLLM"+description = "A list-valued branch"+inputs = { ideas = "Text[]" }+output = "Text[]"+prompt = """+List the risks of these ideas:++@ideas+"""++[pipe.pick_idea]+type = "PipeLLM"+description = "A single-valued branch"+inputs = { ideas = "Text[]" }+output = "Text"+prompt = """+Pick the best of these ideas:++@ideas+"""++[pipe.conclude]+type = "PipeLLM"+description = "Consumes the branch outputs"+inputs = { risks = "Text[]", pick = "Text" }+output = "Text"+prompt = """+Conclude on $pick given these risks:++@risks+"""+'''++_LIFT_BUNDLE = """+domain = "io_multiplicity_lift"+description = "Plural outputs lifted to empty lists when the optional source is absent"+main_pipe = "lift_flow"++[pipe.lift_flow]+type = "PipeSequence"+description = "Lifts its first two steps, then counts what they left"+inputs = { source = "Text?" }+output = "Text"+steps = [+ { pipe = "find_many", result = "findings" },+ { pipe = "branch_out", result = "combined" },+ { pipe = "count_findings", result = "report" },+]++[pipe.find_many]+type = "PipeFunc"+description = "A plural output: lifted to an empty list"+inputs = { source = "Text" }+output = "Text[]"+function_name = "io_multiplicity_find_many"++[pipe.branch_out]+type = "PipeParallel"+description = "Lifted wholesale: its plural branch slot becomes an empty list"+inputs = { source = "Text" }+output = "Composite"+add_each_output = true+branches = [+ { pipe = "find_one", result = "one_out" },+ { pipe = "find_many_again", result = "many_out" },+]++[pipe.find_one]+type = "PipeFunc"+description = "A singular branch"+inputs = { source = "Text" }+output = "Text"+function_name = "io_multiplicity_find_one"++[pipe.find_many_again]+type = "PipeFunc"+description = "A plural branch"+inputs = { source = "Text" }+output = "Text[]"+function_name = "io_multiplicity_find_many"++[pipe.count_findings]+type = "PipeFunc"+description = "Consumes the lifted lists"+inputs = { findings = "Text[]", many_out = "Text[]", one_out = "Text?" }+output = "Text"+function_name = "io_multiplicity_count_findings"+"""+++def io_multiplicity_find_many(working_memory: WorkingMemory) -> ListContent[TextContent]:+ return ListContent[TextContent](items=[TextContent(text=f"found:{working_memory.get_stuff_as_str(name='source')}")])+++def io_multiplicity_find_one(working_memory: WorkingMemory) -> TextContent:+ return TextContent(text=f"found:{working_memory.get_stuff_as_str(name='source')}")+++def io_multiplicity_count_findings(working_memory: WorkingMemory) -> TextContent:+ findings = working_memory.get_stuff(name="findings").content+ many_out = working_memory.get_stuff(name="many_out").content+ assert isinstance(findings, ListContent)+ assert isinstance(many_out, ListContent)+ return TextContent(text=f"{findings.nb_items} findings, {many_out.nb_items} more")+++_TEST_FUNCS = [io_multiplicity_find_many, io_multiplicity_find_one, io_multiplicity_count_findings]+++def _io_items(graph: GraphSpec) -> list[tuple[NodeSpec, IOSpec]]:+ return [(node, io_spec) for node in graph.nodes for io_spec in [*node.node_io.inputs, *node.node_io.outputs]]+++def _assert_one_reading_per_stuff(graph: GraphSpec) -> None:+ """A stuff reads the same on every io item that names it, and only ever `True` or `None`."""+ readings: dict[str, set[Any]] = {}+ for _, io_spec in _io_items(graph):+ assert io_spec.multiplicity in {True, None}, f"{io_spec.name}: a run never writes a count, got {io_spec.multiplicity!r}"+ if io_spec.digest is not None:+ readings.setdefault(io_spec.digest, set()).add(io_spec.multiplicity)+ disagreeing = {digest: values for digest, values in readings.items() if len(values) > 1}+ assert not disagreeing, f"io items disagree on a stuff's multiplicity: {disagreeing}"+++def _multiplicities(graph: GraphSpec, *, pipe_code: str, direction: str, name: str) -> set[Any]:+ """Every multiplicity written for the io item `name` on the nodes of `pipe_code`."""+ found: set[Any] = set()+ for node in graph.nodes:+ if node.pipe_code != pipe_code:+ continue+ io_specs = node.node_io.inputs if direction == "input" else node.node_io.outputs+ found.update(io_spec.multiplicity for io_spec in io_specs if io_spec.name == name)+ assert found, f"no {direction} named '{name}' on '{pipe_code}'"+ return found+++def _in_process_graph(teardown_spy: Any) -> GraphSpec:+ """The one GraphSpec the in-process tracer built during the run."""+ graphs = [graph for graph in teardown_spy.spy_return_list if isinstance(graph, GraphSpec)]+ assert len(graphs) == 1+ return graphs[0]++[email protected](loop_scope="class")+class TestRunGraphIoMultiplicity:+ @classmethod+ def setup_class(cls):+ for func in _TEST_FUNCS:+ func_registry.register_function(func)++ @classmethod+ def teardown_class(cls):+ for func in _TEST_FUNCS:+ if func_registry.has_function(func.__name__):+ func_registry.unregister_function_by_name(func.__name__)++ async def test_dry_run_marks_every_list(self, mocker: MockerFixture) -> None:+ teardown_spy = mocker.spy(GraphTracer, "teardown")++ assembled_graph, _ = await dry_run_pipeline(mthds_contents=[_DRY_BUNDLE])+ in_process_graph = _in_process_graph(teardown_spy)++ for graph in (assembled_graph, in_process_graph):+ _assert_one_reading_per_stuff(graph)+ # A method input declared as a list, on the method and on its first consumer.+ assert _multiplicities(graph, pipe_code="process_documents", direction="input", name="documents") == {True}+ assert _multiplicities(graph, pipe_code="extract_records", direction="input", name="documents") == {True}+ # A declared list output, with the concept left bare.+ assert _multiplicities(graph, pipe_code="extract_records", direction="output", name="records") == {True}+ records_concepts = {+ io_spec.concept for node, io_spec in _io_items(graph) if node.pipe_code == "extract_records" and io_spec.name == "records"+ }+ assert records_concepts == {"Record"}+ # A batch: each item single, each item's result single, the aggregate a list.+ assert _multiplicities(graph, pipe_code="summarize_record", direction="input", name="record") == {None}+ assert _multiplicities(graph, pipe_code="brainstorm", direction="input", name="summaries") == {True}+ # A single output made a list by the step's nb_output.+ assert _multiplicities(graph, pipe_code="brainstorm", direction="output", name="ideas") == {True}+ # A parallel's branch outputs, registered on the parallel itself.+ assert _multiplicities(graph, pipe_code="assess", direction="output", name="risks") == {True}+ assert _multiplicities(graph, pipe_code="assess", direction="output", name="pick") == {None}+ assert _multiplicities(graph, pipe_code="conclude", direction="input", name="risks") == {True}+ assert _multiplicities(graph, pipe_code="conclude", direction="input", name="pick") == {None}+ assert _multiplicities(graph, pipe_code="conclude", direction="output", name="conclusion") == {None}++ async def test_lifted_plural_outputs_read_as_lists(self, mocker: MockerFixture) -> None:+ """An empty list from a lift is still a list: `True`, never `0` and never single."""+ teardown_spy = mocker.spy(GraphTracer, "teardown")+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=True)+ runner = PipelexMTHDSProtocol(execution_config=execution_config)++ with scoped_event_log(InMemoryEventLog()):+ response = await runner.execute(mthds_contents=[_LIFT_BUNDLE], inputs={})++ assert response.pipe_output.main_stuff.as_text.text == "0 findings, 0 more"+ assembled_graph = response.pipe_output.graph_spec+ assert assembled_graph is not None+ in_process_graph = _in_process_graph(teardown_spy)++ for graph in (assembled_graph, in_process_graph):+ _assert_one_reading_per_stuff(graph)+ lifted_codes = {node.pipe_code for node in graph.nodes if node.status == NodeStatus.SKIPPED}+ assert {"find_many", "branch_out"} <= lifted_codes+ # A lifted pipe's plural main output.+ assert _multiplicities(graph, pipe_code="find_many", direction="output", name="findings") == {True}+ assert _multiplicities(graph, pipe_code="count_findings", direction="input", name="findings") == {True}+ # A lifted parallel's plural companion slot.+ assert _multiplicities(graph, pipe_code="branch_out", direction="output", name="many_out") == {True}+ assert _multiplicities(graph, pipe_code="count_findings", direction="input", name="many_out") == {True}diff --git a/tests/unit/pipelex/graph/test_escape_mermaid_label.py b/tests/unit/pipelex/graph/test_escape_mermaid_label.pyindex d1c7aabf7..a4e16f794 100644--- a/tests/unit/pipelex/graph/test_escape_mermaid_label.py+++ b/tests/unit/pipelex/graph/test_escape_mermaid_label.py@@ -13,12 +13,14 @@ class TestEscapeMermaidLabel:assert "'" in resultdef test_escape_brackets(self) -> None:- """Test escaping square brackets."""+ """Square brackets become Mermaid entity codes, which render as the brackets themselves."""result = escape_mermaid_label("Label [with] brackets")- assert "[" not in result- assert "]" not in result- assert "(" in result- assert ")" in result+ assert result == "Label #91;with#93; brackets"++ def test_multiplicity_marker_survives(self) -> None:+ """A multiplicity marker keeps its brackets once rendered, rather than turning into `Record()`."""+ assert escape_mermaid_label("Record[]") == "Record#91;#93;"+ assert escape_mermaid_label("Record[3]") == "Record#91;3#93;"def test_no_escape_needed(self) -> None:"""Test label that doesn't need escaping."""diff --git a/tests/unit/pipelex/graph/test_mermaidflow.py b/tests/unit/pipelex/graph/test_mermaidflow.pyindex aace899c4..67bf47edf 100644--- a/tests/unit/pipelex/graph/test_mermaidflow.py+++ b/tests/unit/pipelex/graph/test_mermaidflow.py@@ -13,6 +13,8 @@ from pipelex.graph.graphspec import (PipelineRef,)from pipelex.graph.mermaidflow.mermaidflow_factory import MermaidflowFactory+from pipelex.graph.mermaidflow.mermaidflow_utils import make_stuff_id+from pipelex.graph.mermaidflow.stuff_collector import collect_stuff_metadatafrom .conftest import make_graph_config@@ -472,3 +474,36 @@ class TestMermaidflow:subgraph_content = "\n".join(lines[subgraph_start_idx : subgraph_end_idx + 1])assert "combined_result" in subgraph_content, "Combined output stuff should be inside the controller subgraph"assert ":::stuff" in subgraph_content, "Combined output stuff should have :::stuff class styling"++ def test_stuff_labels_carry_the_multiplicity_marker(self) -> None:+ """A list stuff reads `Record[]` on its Mermaid node and in the viewer's metadata; a single one stays bare."""+ pipeline_input_consumer = {+ "node_id": "extract_1",+ "kind": NodeKind.OPERATOR,+ "pipe_code": "extract_records",+ "status": NodeStatus.SUCCEEDED,+ "node_io": NodeIOSpec(+ inputs=[IOSpec(name="documents", concept="Document", digest="docs_digest", multiplicity=True)],+ outputs=[IOSpec(name="records", concept="Record", digest="records_digest", multiplicity=True)],+ ),+ }+ single_consumer = {+ "node_id": "summarize_1",+ "kind": NodeKind.OPERATOR,+ "pipe_code": "summarize",+ "status": NodeStatus.SUCCEEDED,+ "node_io": NodeIOSpec(+ inputs=[IOSpec(name="records", concept="Record", digest="records_digest", multiplicity=True)],+ outputs=[IOSpec(name="summary", concept="Text", digest="summary_digest")],+ ),+ }+ graph = self._make_graph(nodes=[pipeline_input_consumer, single_consumer])++ result = MermaidflowFactory.make_from_graphspec(graph, graph_config=make_graph_config())++ assert "records<br/>Record#91;#93;" in result.mermaid_code+ assert "documents<br/>Document#91;#93;" in result.mermaid_code+ assert 'summary<br/>Text"' in result.mermaid_code+ metadata = collect_stuff_metadata(graph)+ assert metadata[make_stuff_id("records_digest")]["concept"] == "Record[]"+ assert metadata[make_stuff_id("summary_digest")]["concept"] == "Text"diff --git a/tests/unit/pipelex/graph/test_stuff_io_spec.py b/tests/unit/pipelex/graph/test_stuff_io_spec.pynew file mode 100644index 000000000..6fa0886ee--- /dev/null+++ b/tests/unit/pipelex/graph/test_stuff_io_spec.py@@ -0,0 +1,125 @@+"""The GraphSpec io item a run writes for a stuff: its multiplicity is read off the value.++A list reads `True` whatever its length, so an empty list never writes `0` (which a GraphSpec+reader refuses) and a one-item list never reads as single. Anything else reads `None`.+"""++import pytest+from pydantic import ValidationError++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.stuff import Stuff+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_content import TextContent+from pipelex.graph.graphspec import IOSpec, make_io_concept_label+from pipelex.graph.stuff_io_spec import make_stuff_io_spec+++def _make_text_stuff(*, name: str) -> Stuff:+ return StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ content=TextContent(text="hello"),+ name=name,+ )+++def _make_text_list_stuff(*, name: str, texts: list[str]) -> Stuff:+ return StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ content=ListContent[TextContent](items=[TextContent(text=text) for text in texts]),+ name=name,+ )+++class TestMakeStuffIoSpec:+ def test_single_stuff_reads_single(self) -> None:+ stuff = _make_text_stuff(name="summary")++ io_spec = make_stuff_io_spec(name="summary", stuff=stuff, include_data=False)++ assert io_spec.multiplicity is None+ assert io_spec.concept == "Text"+ assert io_spec.digest == stuff.stuff_code++ @pytest.mark.parametrize(+ "texts",+ [+ pytest.param(["a", "b", "c"], id="several_items"),+ pytest.param(["a"], id="one_item_is_still_a_list"),+ pytest.param([], id="empty_list_never_writes_zero"),+ ],+ )+ def test_list_stuff_reads_variable_length_whatever_its_length(self, texts: list[str]) -> None:+ stuff = _make_text_list_stuff(name="records", texts=texts)++ io_spec = make_stuff_io_spec(name="records", stuff=stuff, include_data=False)++ assert io_spec.multiplicity is True+ # The concept stays bare: the marker is the multiplicity field's job.+ assert io_spec.concept == "Text"++ def test_data_is_embedded_only_when_asked(self) -> None:+ stuff = _make_text_list_stuff(name="records", texts=["a"])++ assert make_stuff_io_spec(name="records", stuff=stuff, include_data=False).data is None+ assert make_stuff_io_spec(name="records", stuff=stuff, include_data=True).data is not None++ def test_extra_rides_through_and_defaults_empty(self) -> None:+ stuff = _make_text_stuff(name="verdict")++ assert make_stuff_io_spec(name="verdict", stuff=stuff, include_data=False).extra == {}+ assert make_stuff_io_spec(name="verdict", stuff=stuff, include_data=False, extra={"optional": True}).extra == {"optional": True}+++class TestIoSpecMultiplicityField:+ @pytest.mark.parametrize(+ "multiplicity",+ [+ pytest.param(True, id="variable_length"),+ pytest.param(3, id="fixed_count"),+ pytest.param(1, id="count_of_one"),+ pytest.param(False, id="false_single"),+ pytest.param(None, id="absent_single"),+ ],+ )+ def test_accepts_what_a_graphspec_reader_accepts(self, multiplicity: bool | int | None) -> None:+ assert IOSpec(name="stuff", multiplicity=multiplicity).multiplicity == multiplicity++ @pytest.mark.parametrize(+ "multiplicity",+ [+ pytest.param(0, id="zero"),+ pytest.param(-2, id="negative"),+ pytest.param("[]", id="authored_suffix_string"),+ pytest.param(2.0, id="float"),+ ],+ )+ def test_refuses_what_a_graphspec_reader_refuses(self, multiplicity: object) -> None:+ with pytest.raises(ValidationError):+ IOSpec.model_validate({"name": "stuff", "multiplicity": multiplicity})++ def test_survives_a_json_round_trip(self) -> None:+ for multiplicity in (True, 3, None):+ io_spec = IOSpec(name="stuff", concept="Record", multiplicity=multiplicity)+ assert IOSpec.model_validate_json(io_spec.model_dump_json()) == io_spec+++class TestIoConceptLabel:+ @pytest.mark.parametrize(+ ("multiplicity", "expected"),+ [+ pytest.param(True, "Record[]", id="variable_length"),+ pytest.param(3, "Record[3]", id="fixed_count"),+ pytest.param(1, "Record", id="count_of_one_reads_single"),+ pytest.param(False, "Record", id="false_reads_single"),+ pytest.param(None, "Record", id="absent_reads_single"),+ ],+ )+ def test_marks_the_concept_as_an_author_writes_it(self, multiplicity: bool | int | None, expected: str) -> None:+ assert make_io_concept_label(concept="Record", multiplicity=multiplicity) == expected+ assert IOSpec(name="records", concept="Record", multiplicity=multiplicity).concept_label == expected++ def test_no_concept_has_no_label(self) -> None:+ assert make_io_concept_label(concept=None, multiplicity=True) is Nonediff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.pyindex 2a225b8a2..406d30a0e 100644--- a/tests/unit/pipelex/tracing/test_assembler_equivalence.py+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py@@ -60,13 +60,14 @@ def _normalize_node(node: NodeSpec) -> dict[str, Any]:"error_type": node.error.error_type if node.error else None,"error_message": node.error.message if node.error else None,"metrics": node.metrics,+ # Multiplicity rides along so both builders are held to carry each io item's marker."inputs": sorted(- [(spec.name, spec.digest) for spec in node.node_io.inputs],- key=lambda pair: (pair[0] or "", pair[1] or ""),+ [(spec.name, spec.digest, spec.multiplicity) for spec in node.node_io.inputs],+ key=lambda triple: (triple[0] or "", triple[1] or ""),),"outputs": sorted(- [(spec.name, spec.digest) for spec in node.node_io.outputs],- key=lambda pair: (pair[0] or "", pair[1] or ""),+ [(spec.name, spec.digest, spec.multiplicity) for spec in node.node_io.outputs],+ key=lambda triple: (triple[0] or "", triple[1] or ""),),}@@ -295,7 +296,7 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: TraceContext) -pipe_type="PipeBatch",node_kind=NodeKind.CONTROLLER,started_at=started_at,- input_specs=[IOSpec(name="input_list", digest="digest_list")],+ input_specs=[IOSpec(name="input_list", digest="digest_list", multiplicity=True)],)# Register batch item extractions@@ -360,7 +361,7 @@ def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: TraceContext) -tracer.on_pipe_end_success(node_id=ctrl_id,ended_at=started_at + timedelta(seconds=4),- output_spec=IOSpec(name="output_list", digest="digest_output_list"),+ output_spec=IOSpec(name="output_list", digest="digest_output_list", multiplicity=True),)@@ -548,6 +549,20 @@ class TestAssemblerEquivalence:direct_spec, assembled_spec = _run_both_paths(scenario_fn)_assert_graphs_equivalent(direct_spec, assembled_spec)+ def test_multiplicity_survives_both_builders(self) -> None:+ """Agreement alone would pass if both builders dropped the marker, so pin its value too:+ the batch's list io items stay plural and its items stay single, on both routes.+ """+ direct_spec, assembled_spec = _run_both_paths(_scenario_batch_fan_out_fan_in)+ for spec in (direct_spec, assembled_spec):+ multiplicity_by_digest = {+ io_spec.digest: io_spec.multiplicity for node in spec.nodes for io_spec in [*node.node_io.inputs, *node.node_io.outputs]+ }+ assert multiplicity_by_digest["digest_list"] is True+ assert multiplicity_by_digest["digest_output_list"] is True+ assert multiplicity_by_digest["digest_item_0"] is None+ assert multiplicity_by_digest["digest_result_0"] is None+def test_usage_is_the_one_intentional_divergence(self) -> None:"""The two builders stay structurally equivalent, and diverge on `usage` on purpose.diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.pyindex cc06d852e..a0196398c 100644--- a/tests/unit/pipelex/tracing/test_trace_events.py+++ b/tests/unit/pipelex/tracing/test_trace_events.py@@ -280,7 +280,7 @@ class TestTraceEvents:assert restored.metrics == {}def test_io_spec_survives_round_trip_within_event(self) -> None:- """IOSpec fields (digest, preview, concept) are preserved through event serialization."""+ """IOSpec fields (digest, preview, concept, multiplicity) are preserved through event serialization."""io_spec = IOSpec(name="complex_stuff",concept="StructuredData",@@ -288,6 +288,7 @@ class TestTraceEvents:preview='{"key": "value"}',size=16,digest="sha256_abc",+ multiplicity=True,extra={"custom_field": "custom_value"},)event = PipeStartEvent(@@ -306,6 +307,7 @@ class TestTraceEvents:assert restored_io.name == "complex_stuff"assert restored_io.concept == "StructuredData"assert restored_io.digest == "sha256_abc"+ assert restored_io.multiplicity is Trueassert restored_io.extra == {"custom_field": "custom_value"}def test_error_spec_survives_round_trip_within_event(self) -> None: