Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Refactor/merge pipe signature category (#380)
Louis Choquelfe266cb30bce
diff --git a/.cursor/rules/llms.mdc b/.cursor/rules/llms.mdcindex a21831a30..8babc7660 100644--- a/.cursor/rules/llms.mdc+++ b/.cursor/rules/llms.mdc@@ -55,7 +55,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/.cursor/rules/python_standards.mdc b/.cursor/rules/python_standards.mdcindex 6e5615be1..2e2e0a9b2 100644--- a/.cursor/rules/python_standards.mdc+++ b/.cursor/rules/python_standards.mdc@@ -40,7 +40,7 @@ This document outlines the core coding standards, best practices, and quality co- Both have a title arg which is handy when logging/printing objects:```python- log.debug("Hello, world!", title="Your first Pipelex log")+ log.verbose("Hello, world!", title="Your first Pipelex log")pretty_print(output_object, title="Your first Pipelex output")```- Both handle formatting json using Rich, pretty_print makes it prettier.diff --git a/.cursor/rules/write_pipelex.mdc b/.cursor/rules/write_pipelex.mdcindex a4f75c153..841da9765 100644--- a/.cursor/rules/write_pipelex.mdc+++ b/.cursor/rules/write_pipelex.mdc@@ -91,23 +91,23 @@ inputs = {### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-# Single item (default) - use standard syntax+# Single item (default)inputs = { document = "Text" }# Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }# Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)## Structuring Models@@ -375,22 +375,22 @@ prompt = "Analyze this data"### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```### Vision@@ -641,8 +641,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -669,7 +668,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -823,7 +821,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.mdindex 95f9c3b70..5a54470dd 100644--- a/.github/copilot-instructions.md+++ b/.github/copilot-instructions.md@@ -1,4 +1,6 @@<!-- BEGIN_PIPELEX_RULES -->+# Pipelex Rules+## Guide to write or edit pipelines using the Pipelex language in .plx files- Always first write your "plan" in natural language, then transcribe it in pipelex.@@ -85,23 +87,23 @@ inputs = {#### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-## Single item (default) - use standard syntax+## Single item (default)inputs = { document = "Text" }## Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }## Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)### Structuring Models@@ -369,22 +371,22 @@ prompt = "Analyze this data"#### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```#### Vision@@ -635,8 +637,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -663,7 +664,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -817,7 +817,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```@@ -1125,7 +1125,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/.pipelex/inference/deck/base_deck.toml b/.pipelex/inference/deck/base_deck.tomlindex 1b6fbcff1..7cfc55299 100644--- a/.pipelex/inference/deck/base_deck.toml+++ b/.pipelex/inference/deck/base_deck.toml@@ -67,15 +67,17 @@ llm_cheap = { model = "base-gemini", temperature = 0.1 }# Generation skillsllm_for_factual_writing = { model = "base-gpt", temperature = 0.1 }llm_for_creative_writing = { model = "claude-4.5-sonnet", temperature = 0.9 }+llm_to_write_questions = { model = "gpt-4o-mini", temperature = 0.3 }# Reasoning skills-llm_to_reason_short = { model = "base-claude", temperature = 0.5, max_tokens = 500 }-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_reason_on_diagram = { model = "base-claude", temperature = 0.5 }+llm_to_analyze_data = { model = "base-claude", temperature = 0.5 }# Search and answer skills-llm_to_answer = { model = "base-claude", temperature = 0.1 }-llm_to_retrieve = { model = "base-gemini", temperature = 0.1 }+llm_to_answer_easy_questions = { model = "gpt-4o-mini", temperature = 0.3 }+llm_to_answer_hard_questions = { model = "base-claude", temperature = 0.3 }+llm_to_retrieve = { model = "base-claude", temperature = 0.1 }llm_for_enrichment = { model = "gpt-4o", temperature = 0.1 }llm_to_enrich = { model = "base-claude", temperature = 0.1 }llm_for_question_and_excerpt_reformulation = { model = "gpt-4o", temperature = 0.9 }@@ -84,10 +86,15 @@ llm_for_question_and_excerpt_reformulation = { model = "gpt-4o", temperature = 0llm_to_engineer = { model = "smart_llm", temperature = 0.2 }llm_to_write = { model = "base-gemini", temperature = 0.2 }llm_to_pipelex = { model = "base-claude", temperature = 0.2 }+llm_to_code = { model = "base-claude", temperature = 0.1 }+llm_to_analyze_large_codebase = { model = "base-claude", temperature = 0.1 }# Image skills+llm_for_basic_vision = { model = "gpt-4o-mini", temperature = 0.1 }llm_to_write_img_gen_prompt = { model = "base-claude", temperature = 0.2 }llm_to_describe_img = { model = "base-claude", temperature = 0.5 }+llm_for_visual_analysis = { model = "base-claude", temperature = 0.5 }+llm_for_visual_design = { model = "base-claude", temperature = 0.5 }llm_to_design_fashion = { model = "base-claude", temperature = 0.7 }llm_for_img_to_text = { model = "base-claude", temperature = 0.1 }@@ -107,28 +114,30 @@ for_text = "cheap_llm_for_text"for_object = "cheap_llm_for_object"####################################################################################################-# OCR Presets+# Extract Presets####################################################################################################[extract]-choice_default = "base_ocr_mistral"+choice_default = "extract_text_from_visuals"[extract.presets]-base_ocr_mistral = { model = "mistral-ocr", max_nb_images = 100, image_min_size = 50 }-base_extract_pypdfium2 = { model = "pypdfium2-extract-text", max_nb_images = 100, image_min_size = 50 }+extract_text_from_visuals = { model = "mistral-ocr", max_nb_images = 100, image_min_size = 50 }+extract_text_from_pdf = { model = "pypdfium2-extract-text", max_nb_images = 100, image_min_size = 50 }##################################################################################################### Image Generation Presets####################################################################################################[img_gen]-choice_default = "base_img_gen"+choice_default = "gen_image_basic"[img_gen.presets]++# General purpose-base_img_gen = { model = "base-img-gen", quality = "medium", guidance_scale = 7.5, is_moderated = true, safety_tolerance = 3 }-fast_img_gen = { model = "fast-img-gen", nb_steps = 4, guidance_scale = 5.0, is_moderated = true, safety_tolerance = 3 }-high_quality_img_gen = { model = "best-img-gen", quality = "high", guidance_scale = 8.0, is_moderated = true, safety_tolerance = 3 }+gen_image_basic = { model = "base-img-gen", quality = "medium", guidance_scale = 7.5, is_moderated = true, safety_tolerance = 3 }+gen_image_fast = { model = "fast-img-gen", nb_steps = 4, guidance_scale = 5.0, is_moderated = true, safety_tolerance = 3 }+gen_image_high_quality = { model = "best-img-gen", quality = "high", guidance_scale = 8.0, is_moderated = true, safety_tolerance = 3 }# Specific skillsimg_gen_for_art = { model = "best-img-gen", quality = "high", guidance_scale = 9.0, is_moderated = false, safety_tolerance = 5 }diff --git a/.windsurfrules.md b/.windsurfrules.mdindex 95f9c3b70..5a54470dd 100644--- a/.windsurfrules.md+++ b/.windsurfrules.md@@ -1,4 +1,6 @@<!-- BEGIN_PIPELEX_RULES -->+# Pipelex Rules+## Guide to write or edit pipelines using the Pipelex language in .plx files- Always first write your "plan" in natural language, then transcribe it in pipelex.@@ -85,23 +87,23 @@ inputs = {#### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-## Single item (default) - use standard syntax+## Single item (default)inputs = { document = "Text" }## Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }## Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)### Structuring Models@@ -369,22 +371,22 @@ prompt = "Analyze this data"#### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```#### Vision@@ -635,8 +637,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -663,7 +664,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -817,7 +817,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```@@ -1125,7 +1125,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/AGENTS.md b/AGENTS.mdindex 95f9c3b70..5a54470dd 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -1,4 +1,6 @@<!-- BEGIN_PIPELEX_RULES -->+# Pipelex Rules+## Guide to write or edit pipelines using the Pipelex language in .plx files- Always first write your "plan" in natural language, then transcribe it in pipelex.@@ -85,23 +87,23 @@ inputs = {#### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-## Single item (default) - use standard syntax+## Single item (default)inputs = { document = "Text" }## Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }## Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)### Structuring Models@@ -369,22 +371,22 @@ prompt = "Analyze this data"#### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```#### Vision@@ -635,8 +637,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -663,7 +664,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -817,7 +817,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```@@ -1125,7 +1125,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/BLACKBOX_RULES.md b/BLACKBOX_RULES.mdindex 95f9c3b70..5a54470dd 100644--- a/BLACKBOX_RULES.md+++ b/BLACKBOX_RULES.md@@ -1,4 +1,6 @@<!-- BEGIN_PIPELEX_RULES -->+# Pipelex Rules+## Guide to write or edit pipelines using the Pipelex language in .plx files- Always first write your "plan" in natural language, then transcribe it in pipelex.@@ -85,23 +87,23 @@ inputs = {#### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-## Single item (default) - use standard syntax+## Single item (default)inputs = { document = "Text" }## Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }## Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)### Structuring Models@@ -369,22 +371,22 @@ prompt = "Analyze this data"#### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```#### Vision@@ -635,8 +637,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -663,7 +664,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -817,7 +817,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```@@ -1125,7 +1125,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f77ba883..fd5683b92 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -34,7 +34,7 @@ We tried to group all the renamings we wanted to do which impact our language, sThis is all in the spirit of making Pipelex a declarative language, where you express what you want to do, and the system will figure out how to do it. So our focus inwas to make the Pipelex language easier to understand and use for non-technical users, and at the same time use more consistent and obvious words that developers are used to.-**💡 Pro tip:** To make migration easier, pass the [migration guide](https://github.com/PipelexLab/pipelex/blob/main/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md) to your favorite SWE agent (Cursor, Claude Code, github copilot, etc.) and let it handle the bulk of the changes!+**💡 Pro tip:** To make migration easier, pass the [migration guide](https://github.com/Pipelex/pipelex/blob/main/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md) to your favorite SWE agent (Cursor, Claude Code, github copilot, etc.) and let it handle the bulk of the changes!- **Removed centralized `pipelex_libraries` folder system**- Pipelines are now auto-discovered from anywhere in your project—no special directory requireddiff --git a/CLAUDE.md b/CLAUDE.mdindex 95f9c3b70..5a54470dd 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1,4 +1,6 @@<!-- BEGIN_PIPELEX_RULES -->+# Pipelex Rules+## Guide to write or edit pipelines using the Pipelex language in .plx files- Always first write your "plan" in natural language, then transcribe it in pipelex.@@ -85,23 +87,23 @@ inputs = {#### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-## Single item (default) - use standard syntax+## Single item (default)inputs = { document = "Text" }## Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }## Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)### Structuring Models@@ -369,22 +371,22 @@ prompt = "Analyze this data"#### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```#### Vision@@ -635,8 +637,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -663,7 +664,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -817,7 +817,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```@@ -1125,7 +1125,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.mdindex b19ac037a..c17b8f53b 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md@@ -44,7 +44,7 @@ An LLM Preset is simply a name for a LLM Settings that you have predefined in or```toml[llm_presets]-llm_to_reason = {+llm_for_complex_reasoning = {llm_handle = "gpt-4-turbo",temperature = 0.7,max_tokens = "auto"diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.mdindex 5d2cb6b6b..60f708bc4 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md@@ -74,8 +74,7 @@ Tagline = "A catchy marketing tagline"type = "PipeLLM"description = "Extract keywords from a product description"inputs = { description = "ProductDescription" }-output = "Keyword"-multiple_output = true+output = "Keyword[]"prompt = """Please extract the most relevant keywords from the following product description:@@ -86,7 +85,7 @@ Focus on features, benefits, and unique selling points.```!!! note "Multiple Outputs"- The `multiple_output = true` setting allows the LLM to generate as many keywords as it finds relevant. For a comprehensive guide on controlling how many items pipes produce or accept, see [Understanding Multiplicity](understanding-multiplicity.md).+ The `[]` bracket notation in `output = "Keyword[]"` allows the LLM to generate as many keywords as it finds relevant. For a comprehensive guide on controlling how many items pipes produce or accept, see [Understanding Multiplicity](understanding-multiplicity.md).```plx[pipe.generate_tagline_from_keywords]diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeExtract.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeExtract.mdindex 48b40e377..1466c0fca 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeExtract.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeExtract.md@@ -48,7 +48,7 @@ OCR presets are defined in your model deck configuration and can include paramet| `page_views` | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`. | No || `page_views_dpi` | integer | The resolution (in Dots Per Inch) for the generated page views when processing a PDF. Defaults to `150`. | No || `page_image_captions` | boolean | If `true`, the OCR service may attempt to generate captions for the images found. *Note: This feature depends on the OCR provider.* | No |-| `ocr` | string | The OCR model handle, preset, or setting to use (e.g., `"mistral-ocr"`, `"base_ocr_mistral"`). Defaults to the model specified in the global config. | No |+| `model` | string | The Extract model choice by name, setting, or preset to use (e.g., `"mistral-ocr"`, `"extract_text_from_visuals"`). Defaults to the model specified in the global config. | No |### Example: Processing a PDFdiff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.mdindex 8ce99bf60..c498b51f4 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md@@ -39,15 +39,11 @@ Image generation presets are defined in your model deck configuration and can in| `type` | string | The type of the pipe: `PipeImgGen` | Yes || `description` | string | A description of the image generation operation. | Yes || `inputs` | dictionary | The input concept(s) for the image generation operation, as a dictionary mapping input names to concept codes. | Yes |-| `output` | string | The output concept produced by the image generation operation. | Yes |+| `output` | string | The output concept produced by the image generation operation. Use bracket notation for multiple images: `"Image[3]"` generates exactly 3 images. | Yes || `img_gen_prompt` | string | A static text prompt for image generation. Use this *or* `input`. | No |-| `nb_output` | integer | The number of images to generate. If omitted, a single image is generated. | No |-| `img_gen` | string | The choice of image generation model handle preset or setting to use (e.g., `"gpt-image-1"`). Defaults to the model specified in the global config. | No |+| `model` | string | The choice of image generation model name, setting, or preset to use (e.g., `"gpt-image-1"`). Defaults to the model specified in the global config. | No || `aspect_ratio` | string | The desired aspect ratio of the image (e.g., `"16:9"`, `"1:1"`). | No |-| `quality` | string | The quality of the generated image (e.g., `"standard"`, `"hd"`). | No || `seed` | integer or "auto" | A seed for the random number generator to ensure reproducibility. `"auto"` uses a random seed. | No |-| `nb_steps` | integer | For diffusion models, the number of steps to run. More steps can increase detail but take longer. | No |-| `guidance_scale` | float | How strictly the model should adhere to the prompt. Higher values mean closer adherence. | No |### Example: Generating a single image from a static prompt@@ -60,14 +56,14 @@ type = "PipeImgGen"description = "Generate a futuristic car image"output = "Image"img_gen_prompt = "A sleek, futuristic sports car driving on a neon-lit highway at night."-model = "base_img_gen"+model = "gen_image_basic"aspect_ratio = "16:9"quality = "hd"```### Example: Generating multiple images from a dynamic prompt-This pipe takes a text prompt as input and generates three variations of the image.+This pipe takes a text prompt as input and generates three variations of the image using bracket notation.```plx[concept]@@ -77,9 +73,8 @@ ImagePrompt = "A text prompt for generating an image"type = "PipeImgGen"description = "Generate three logo variations from a prompt"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3-model = "base_img_gen"+output = "Image[3]"+model = "gen_image_basic"aspect_ratio = "1:1"```diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.mdindex 880f5636a..a608ccf65 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md@@ -119,25 +119,33 @@ Analyze the document page shown in the image and explain how it relates to the p| `type` | string | The type of the pipe: `PipeLLM` | Yes || `description` | string | A description of the LLM operation. | Yes || `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image_argurment"`)-| `output` | string | The output concept produced by the LLM operation. | Yes |-| `llm` | string or table | Specifies the LLM preset(s) to use. Can be a single preset or a table mapping different presets for different generation modes (e.g., `main`, `object_direct`). | No |+| `output` | string | The output concept produced by the LLM operation with multiplicity notation using brackets (e.g., `"Text"`, `"Text[]"`, `"Text[3]"`). | Yes |+| `model` | string or table | Specifies the LLM choice by name, setting, or preset to use. | No |+| `model_to_structure` | string or table | Specifies the LLM choice by name, setting, or preset to use for structuring after preliminary text generation. | No || `system_prompt` | string | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Can be inline text or a reference to a template file (`"file:path/to/prompt.md"`). | No |-| `prompt` | string | A simple, static user prompt. Use this when you don't need to inject any variables. | No || `prompt_template` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`). **Note**: Do not use `@` or `$` for image variables. | No |-| `images` | list of strings | **Deprecated**: Use the `inputs` section to declare image inputs instead. | No || `structuring_method` | string | The method for generating structured output. Can be `direct` or `preliminary_text`. Defaults to the global configuration. | No |-| `nb_output` | integer | Specifies exactly how many outputs to generate (e.g., `nb_output = 3` for exactly 3 outputs). Use when you need a fixed number of results. Mutually exclusive with `multiple_output`. | No |-| `multiple_output` | boolean | Controls output generation mode. Default is `false` (single output). Set to `true` for variable-length list generation when you need an indeterminate number of outputs. Mutually exclusive with `nb_output`. | No |-### Output Generation Modes+### Output Multiplicity-`PipeLLM` supports three different output generation modes:+Specify output multiplicity using bracket notation in the `output` field:-1. **Single Output** (default): Don't specify `nb_output` or `multiple_output`, or set `multiple_output = false`. The LLM generates exactly one result.+- **Single output** (default): `output = "Text"` - generates exactly one item+- **Variable output**: `output = "Text[]"` - lets the LLM decide how many items to generate+- **Fixed output**: `output = "Text[5]"` - generates exactly 5 items-2. **Fixed Multiple Outputs**: Use `nb_output = N` (where N is a positive integer) when you need exactly N outputs. For example, `nb_output = 3` will try to generate 3 results. The parameter `_nb_output` will be available in the prompt template, e.g. "Give me the names of $_nb_output flowers".+Examples:-3. **Variable Multiple Outputs**: Use `multiple_output = true` when you need a variable-length list where the LLM determines how many outputs to generate based on the content and context.+```plx+# Single output (default)+output = "Summary"++# Variable - extract all keywords found+output = "Keyword[]"++# Fixed - generate exactly 3 alternatives+output = "Headline[3]"+```!!! info "Learn More About Multiplicity"For a comprehensive guide on output multiplicity, input multiplicity, and the philosophy behind how Pipelex handles single items versus collections, see [Understanding Multiplicity](../understanding-multiplicity.md).diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.mdindex 56b64c95d..0581ab2f6 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md@@ -27,7 +27,7 @@ pipelex build pipe "Take a photo as input, and render the opposite of the photo"**Options:**-- `--output, -o`: Output path for generated PLX file (default: `./results/generated_pipeline.plx`)+- `--output, -o`: Output path for generated PLX file- `--no-output`: Skip saving the pipeline to file### Build One-Shot (Fast)@@ -80,7 +80,7 @@ This will:2. Generate a complete pipeline with appropriate concepts and pipes3. Validate the pipeline for correctness4. Fix any deterministic issues automatically-5. Save the working pipeline to `./results/generated_pipeline.plx`+5. Save the working pipeline## Best Practicesdiff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.mdindex 1f336fdbc..af6dc22fb 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md@@ -33,51 +33,35 @@ By keeping concepts singular and using multiplicity to express quantity, Pipelex## Output Multiplicity-Output multiplicity controls how many items a pipe produces. This is most commonly used with `PipeLLM` to generate multiple outputs from a single execution.+Output multiplicity controls how many items a pipe produces. You can specify this using bracket notation in the `output` field.-### The Two Modes of Output Multiplicity+### The Three Output Multiplicity Modes-Pipelex supports two ways to specify output multiplicity:+**1. Single output (default)**-1. **Fixed multiplicity** (`nb_output`): Generate an exact number of items-2. **Variable multiplicity** (`multiple_output`): Let the LLM decide how many items to generate--### Fixed Output Multiplicity--Use `nb_output` when you need a specific, predetermined number of outputs:+When no brackets are used in the output, the pipe produces a single item:```plx[concept]-Headline = "A catchy title for content"+Summary = "A concise overview of content"-[pipe.generate_headline_options]+[pipe.summarize]type = "PipeLLM"-description = "Generate headline alternatives"-inputs = { article_text = "Text" }-output = "Headline"-nb_output = 5+description = "Create a summary of the document"+inputs = { document = "Text" }+output = "Summary"prompt = """-Read this article:--@article_text+Summarize this document concisely:-Generate $_nb_output different headline options for this article.-Make each one unique and compelling.+@document"""```-In this example, the pipe will always produce exactly 5 headlines. The variable `$_nb_output` is automatically available in your prompt, making it easy to communicate the requirement to the LLM.--**Common use cases for fixed multiplicity:**--- Generate N alternative versions for A/B testing-- Create a fixed set of options for user selection-- Produce a specific number of variations for comparison-- Match external requirements (e.g., "always provide 3 recommendations")+This is the default behavior and represents the most common case.-### Variable Output Multiplicity+**2. Variable output (bracket notation `[]`)**-Use `multiple_output = true` when the number of outputs should depend on the content:+Use empty brackets in the output to let the LLM decide how many items to generate:```plx[concept]@@ -93,8 +77,7 @@ total = { type = "number", description = "Total price for this line" }type = "PipeLLM"description = "Extract all line items from an invoice"inputs = { invoice_text = "Text" }-output = "LineItem"-multiple_output = true+output = "LineItem[]"prompt = """Extract all line items from this invoice:@@ -104,36 +87,46 @@ For each line item, extract the description, quantity, unit price, and total amo"""```-Here, the pipe will extract however many line items appear in the invoice. A simple invoice might have 2 line items, while a detailed purchase order might have 50.+The pipe will extract however many line items appear in the invoice. A simple invoice might have 2 line items, while a detailed purchase order might have 50.-**Common use cases for variable multiplicity:**+**Common use cases for variable output:**- Extract entities from text (unknown count in advance)- Generate as many alternatives as needed- List all items that match criteria- Identify all occurrences of a pattern-### Single Output (Default)+**3. Fixed output (bracket notation `[N]`)**-When you don't specify `nb_output` or `multiple_output`, a pipe produces a single output:+Use a number in brackets to generate an exact number of items:```plx[concept]-Summary = "A concise overview of content"+Headline = "A catchy title for content"-[pipe.summarize]+[pipe.generate_headline_options]type = "PipeLLM"-description = "Create a summary of the document"-inputs = { document = "Text" }-output = "Summary"+description = "Generate headline alternatives"+inputs = { article_text = "Text" }+output = "Headline[5]"prompt = """-Summarize this document concisely:+Read this article:-@document+@article_text++Generate 5 different headline options for this article.+Make each one unique and compelling."""```-This is the default behavior and represents the most common case.+The pipe will always produce exactly 5 headlines.++**Common use cases for fixed output:**++- Generate N alternative versions for A/B testing+- Create a fixed set of options for user selection+- Produce a specific number of variations for comparison+- Match external requirements (e.g., "always provide 3 recommendations")## Input Multiplicity@@ -141,20 +134,17 @@ Input multiplicity specifies whether a pipe expects a single item or multiple it### Syntax for Input Multiplicity-Input multiplicity is specified using an expanded syntax in the `inputs` dictionary:+Input multiplicity is specified using bracket notation in the `inputs` dictionary:```plx# Standard syntax (single item, the default)inputs = { document = "Text" }-# Equivalent with explicit multiplicity = false (not needed, but shown for clarity)-inputs = { document = { concept = "Text", multiplicity = false } }--# Variable list (indeterminate number)-inputs = { documents = { concept = "Text", multiplicity = true } }+# Variable list (indeterminate number of items)+inputs = { documents = "Text[]" }# Fixed count (exactly N items)-inputs = { comparison_items = { concept = "Text", multiplicity = 2 } }+inputs = { comparison_items = "Text[2]" }```### The Three Input Multiplicity Modes@@ -179,12 +169,12 @@ Analyze this report in detail:"""```-!!! note "No Need for `multiplicity = false`"- You don't need to specify `multiplicity = false` explicitly, it's the default. Only use the expanded syntax when you need `multiplicity = true` or a fixed integer count.+!!! note "Default Single Item Behavior"+ When no brackets are used, the input expects a single item. Only use brackets when you need multiple items (`[]`) or a specific count (`[N]`).-**2. Variable list (`multiplicity = true`)**+**2. Variable list (bracket notation `[]`)**-When you set `multiplicity = true`, the pipe expects a list with an indeterminate number of items:+Use empty brackets `[]` to specify that the pipe expects a list with an indeterminate number of items:```plx[concept]@@ -194,7 +184,7 @@ Summary = "A concise overview of multiple documents"[pipe.summarize_all_documents]type = "PipeLLM"description = "Create a unified summary of multiple documents"-inputs = { documents = { concept = "Document", multiplicity = true } }+inputs = { documents = "Document[]" }output = "Summary"prompt = """Analyze all of these documents:@@ -205,9 +195,17 @@ Create a single unified summary that captures the key points across all document"""```-**3. Fixed count (`multiplicity = N`)**+!!! info "When You Actually Need `[]` Notation"+ The `[]` notation is only required for **advanced use cases**:++ 1. **Batching over items**: When using `batch_over` in a `PipeSequence` to process each item separately+ 2. **Looping in templates**: When you need to iterate over items using Jinja2 syntax (`{% for item in items %}`) in `PipeLLM`, `PipeCompose`, or `PipeCondition` prompts++ For most cases where you simply pass multiple items to a pipe that processes them all together, you don't need to declare the input with `[]`. The pipe will receive the list and process it as a whole.++**3. Fixed count (bracket notation `[N]`)**-When you set `multiplicity` to a specific integer, the pipe expects exactly that many items:+Use a number in brackets `[N]` to specify that the pipe expects exactly that many items:```plx[concept]@@ -217,7 +215,7 @@ Comparison = "A detailed comparison analysis"[pipe.compare_two_images]type = "PipeLLM"description = "Compare exactly two images side by side"-inputs = { images = { concept = "Image", multiplicity = 2 } }+inputs = { images = "Image[2]" }output = "Comparison"prompt = """Compare these two images in detail:@@ -253,7 +251,7 @@ Extract all fields from this invoice:[pipe.process_invoice_batch]type = "PipeSequence"description = "Process multiple invoices"-inputs = { invoice_images = { concept = "InvoiceImage", multiplicity = true } }+inputs = { invoice_images = "InvoiceImage[]" }output = "InvoiceData"steps = [{ pipe = "extract_single_invoice", batch_over = "invoice_images", batch_as = "invoice_image", result = "all_invoice_data" }@@ -296,7 +294,7 @@ Comparison = "A comparative analysis of products"[pipe.compare_products]type = "PipeLLM"description = "Compare two products"-inputs = { products = { concept = "ProductDescription", multiplicity = 2 } }+inputs = { products = "ProductDescription[2]" }output = "Comparison"prompt = """Compare these two products:@@ -320,8 +318,7 @@ CompanyName = "The name of a company or organization"type = "PipeLLM"description = "Extract all company names from an article"inputs = { article = "Article" }-output = "CompanyName"-multiple_output = true+output = "CompanyName[]"prompt = """Read this article:@@ -334,7 +331,7 @@ Only include entities that are explicitly named.## Best Practices-### When to Use Variable Output (`multiple_output = true`)+### When to Use Variable Output (`[]`)Use variable output multiplicity when:@@ -343,7 +340,7 @@ Use variable output multiplicity when:- The count isn't known until after processing- You want the LLM to use its judgment about completeness-### When to Use Fixed Output (`nb_output = N`)+### When to Use Fixed Output (`[N]`)Use fixed output multiplicity when:@@ -352,7 +349,7 @@ Use fixed output multiplicity when:- External requirements dictate a fixed count- You want consistent batch sizes for processing-### When to Use Variable Input (`multiplicity = true`)+### When to Use Variable Input (Empty Brackets `[]`)Use variable input multiplicity when:@@ -361,7 +358,7 @@ Use variable input multiplicity when:- The workflow involves collecting items before processing- You want maximum flexibility in how the pipe is called-### When to Use Fixed Input (`multiplicity = N`)+### When to Use Fixed Input (Brackets with Number `[N]`)Use fixed input multiplicity when:@@ -414,10 +411,10 @@ pipe_output = await execute_pipeline(Multiplicity in Pipelex gives you precise control over how many items flow through your pipelines:- **Concepts stay singular** to maintain clean semantics-- **Output multiplicity** (`nb_output`, `multiple_output`) controls generation-- **Input multiplicity** (in the `inputs` definition) enforces expectations-- **Variable multiplicity** (`true`) for unknown counts-- **Fixed multiplicity** (integer) for exact requirements+- **Output multiplicity** (bracket notation in `output` field) controls generation+- **Input multiplicity** (bracket notation in `inputs` definition) enforces expectations+- **Variable multiplicity** (`[]`) for unknown counts+- **Fixed multiplicity** (`[N]`) for exact requirementsBy understanding and using multiplicity effectively, you can build pipelines that handle both single items and collections with clarity and type safety.diff --git a/docs/pages/configuration/config-technical/inference-backend-config.md b/docs/pages/configuration/config-technical/inference-backend-config.mdindex 7c091808c..099d282e7 100644--- a/docs/pages/configuration/config-technical/inference-backend-config.md+++ b/docs/pages/configuration/config-technical/inference-backend-config.md@@ -385,7 +385,7 @@ cheap_llm_for_object = { model = "cheap_llm_for_object", temperature = 0.5 }# Task-specific presetsllm_for_creative_writing = { model = "claude-4.5-sonnet", temperature = 0.9 }llm_to_extract_invoice = { model = "claude-4.5-sonnet", temperature = 0.1 }-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }### OCR Presets@@ -394,8 +394,8 @@ OCR presets combine OCR model selection with optimized parameters:```toml[extract.presets]# General purpose OCR-base_ocr_mistral = { ocr_handle = "mistral-ocr", max_nb_images = 100, image_min_size = 50 }-base_extract_pypdfium2 = { model = "pypdfium2-extract-text", max_nb_images = 100, image_min_size = 50 }+extract_text_from_visuals = { ocr_handle = "mistral-ocr", max_nb_images = 100, image_min_size = 50 }+extract_text_from_pdf = { model = "pypdfium2-extract-text", max_nb_images = 100, image_min_size = 50 }```### Image Generation Presets@@ -405,9 +405,9 @@ Image generation presets combine model selection with generation parameters:```toml[img_gen.presets]# General purpose image generation-base_img_gen = { model = "base-img-gen", quality = "medium", guidance_scale = 7.5, is_moderated = true, safety_tolerance = 3 }-fast_img_gen = { model = "fast-img-gen", nb_steps = 4, guidance_scale = 5.0, is_moderated = true, safety_tolerance = 3 }-high_quality_img_gen = { model = "best-img-gen", quality = "high", guidance_scale = 8.0, is_moderated = true, safety_tolerance = 3 }+gen_image_basic = { model = "base-img-gen", quality = "medium", guidance_scale = 7.5, is_moderated = true, safety_tolerance = 3 }+gen_image_fast = { model = "fast-img-gen", nb_steps = 4, guidance_scale = 5.0, is_moderated = true, safety_tolerance = 3 }+gen_image_high_quality = { model = "best-img-gen", quality = "high", guidance_scale = 8.0, is_moderated = true, safety_tolerance = 3 }```### Default Choices@@ -420,10 +420,10 @@ for_text = "cheap_llm_for_text"for_object = "cheap_llm_for_object"[extract]-choice_default = "base_ocr_mistral"+choice_default = "extract_text_from_visuals"[img_gen]-choice_default = "base_img_gen"+choice_default = "gen_image_basic"```## Customizationdiff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.mdindex 95669e2f9..c4baf1e70 100644--- a/docs/pages/tools/cli.md+++ b/docs/pages/tools/cli.md@@ -80,24 +80,87 @@ pipelex validate all### Validate Single Pipe```bash-pipelex validate pipe PIPE_CODE+pipelex validate PIPE_CODE+pipelex validate --pipe PIPE_CODE```-Validates and dry-runs a specific pipe, useful for iterative development.+Validates and dry-runs a specific pipe from your imported packages, useful for iterative development.**Arguments:**-- `PIPE_CODE` - The pipe code to validate (e.g., `analyze_cv_matching` or `write_weekly_report`)+- `PIPE_CODE` - The pipe code to validate as a positional argument, or use `--pipe` option++**Options:**++- `--pipe PIPE_CODE` - Explicitly specify the pipe code to validate (alternative to positional argument)**Examples:**```bash-# Validate a specific pipe-pipelex validate pipe analyze_cv_matching-pipelex validate pipe write_weekly_report+# Validate a specific pipe (positional argument)+pipelex validate analyze_cv_matching+pipelex validate write_weekly_report++# Validate a specific pipe (explicit option)+pipelex validate --pipe analyze_cv_matching+```++### Validate Bundle++```bash+pipelex validate BUNDLE_FILE.plx+pipelex validate --bundle BUNDLE_FILE.plx```-**What validation checks:**+Validates all pipes defined in a bundle file. The command automatically detects `.plx` files as bundles.++**Arguments:**++- `BUNDLE_FILE.plx` - Path to the bundle file (auto-detected by `.plx` extension)++**Options:**++- `--bundle BUNDLE_FILE.plx` - Explicitly specify the bundle file path++**Examples:**++```bash+# Validate a bundle (auto-detected)+pipelex validate my_pipeline.plx+pipelex validate pipelines/invoice_processor.plx++# Validate a bundle (explicit option)+pipelex validate --bundle my_pipeline.plx+```++**Note:** When validating a bundle, ALL pipes in that bundle are validated, not just the main pipe.++### Validate Specific Pipe in Bundle++```bash+pipelex validate --bundle BUNDLE_FILE.plx --pipe PIPE_CODE+```++Validates all pipes in a bundle, while ensuring a specific pipe exists in that bundle. The entire bundle is validated, not just the specified pipe.++**Options:**++- `--bundle BUNDLE_FILE.plx` - Path to the bundle file+- `--pipe PIPE_CODE` - Pipe code that must exist in the bundle++**Examples:**++```bash+# Validate bundle and ensure specific pipe exists in it+pipelex validate --bundle my_pipeline.plx --pipe extract_invoice+pipelex validate --bundle invoice_processor.plx --pipe validate_amounts+```++**Important:** The specified pipe must be defined in the bundle. This is useful when you want to validate a bundle and confirm a specific pipe is present and valid within it. However, the entire bundle will be validated regardless.++### What Validation Checks++All validation commands check:- Syntax correctness of `.plx` files- Concept and pipe definitions are valid@@ -374,8 +437,8 @@ For complete documentation including all options and examples, see [Kit Commands2. **Development Workflow**- Write or generate pipelines in `.plx` files- - Validate with `pipelex validate pipe your_pipe_code` during development- - Run `pipelex validate all` before committing changes, or pass a pipe code for faster iteration on a specific pipe+ - Validate with `pipelex validate your_pipe_code` or `pipelex validate your_bundle.plx` during development+ - Run `pipelex validate all` before committing changes3. **Running Pipelines**- Use `pipelex show pipes` to see available pipesdiff --git a/docs/pages/tools/logging.md b/docs/pages/tools/logging.mdindex fff75603b..bd39b04a8 100644--- a/docs/pages/tools/logging.md+++ b/docs/pages/tools/logging.md@@ -82,7 +82,7 @@ log.info("Quick update", inline="Status")# Logging structured datadata = {"key": "value", "nested": {"data": True}}-log.debug(data, title="Configuration")+log.verbose(data, title="Configuration")# Warning with problem IDlog.warning("API rate limit approaching", problem_id="rate_limit_warning")@@ -91,7 +91,7 @@ log.warning("API rate limit approaching", problem_id="rate_limit_warning")log.error("Failed to process", include_exception=True)# Development logging-log.dev("Testing new feature")+log.verbose("Testing new feature")# Verbose logginglog.verbose("Detailed debug information")diff --git a/docs/pages/tools/pipe-builder.md b/docs/pages/tools/pipe-builder.mdindex 9ab278ae5..bb94a10e1 100644--- a/docs/pages/tools/pipe-builder.md+++ b/docs/pages/tools/pipe-builder.md@@ -39,7 +39,7 @@ This command:The build command supports the following options:-- `--output`, `-o`: Path to save the generated file (default: `./results/generated_pipeline.plx`)+- `--output`, `-o`: Path to save the generated file- `--no-output`: Skip saving the file (useful for testing)## How It Worksdiff --git a/pipelex/builder/builder.plx b/pipelex/builder/builder.plxindex 44f44a44e..b2bece9a0 100644--- a/pipelex/builder/builder.plx+++ b/pipelex/builder/builder.plx@@ -77,21 +77,34 @@ Available pipe controllers:When describing the task of a pipe controller, be concise, don't detail all the sub-pipes.Available pipe operators:-- PipeLLM: A pipe that uses an LLM to generate a text, or a structured object. It is a vision LLM so it can also use images.- CRITICAL: When extracting MULTIPLE items (articles, employees, products), use multiple_output = true with SINGULAR concepts!+- PipeLLM: A pipe that uses an LLM to generate text or structured objects. It is a vision LLM so it can also use images.+ CRITICAL: When extracting MULTIPLE items (articles, employees, products), use bracket notation in output with SINGULAR concepts!- Create concept "Article" (not "Articles") with fields "item_name", "quantity" (not "item_names", "quantities")- - Then set multiple_output = true to get a list of Article objects-- PipeImgGen: A pipe that uses an AI model to generate an image.+ - Then set output = "Article[]" to get a variable list, or output = "Article[5]" for exactly 5 items+ - Examples: output = "Text[]" (multiple texts), output = "Image[3]" (exactly 3 images), output = "Employee[]" (list of employees)+- PipeImgGen: A pipe that uses an AI model to generate images.+ - Use bracket notation for multiple images: output = "Image[3]" generates exactly 3 imagesVERY IMPORTANT: IF YOU DECIDE TO CREATE A PipeImgGen, YOU ALSO HAVE TO CREATE A PIPELLM THAT WILL WRITE THE PROMPT, AND THAT NEEDS TO PRECEED THE PIPEIMGEN, based on the necessary elements.That means that in the MAIN pipeline, the prompt MUST NOT be considered as an input. It should be the output of a step that generates the prompt.-- PipeExtract: A pipe that uses an OCR technology to extract text from an image or a pdf.- VERY IMPORTANT: THE INPUT OF THE PIPEOCR MUST BE either an image or a pdf or a concept which refines one of them.+- PipeExtract: A pipe that uses OCR technology to extract text from an image or a pdf.+ - Always outputs a list of pages: output = "Page[]"+ VERY IMPORTANT: THE INPUT OF THE PIPEEXTRACT MUST BE either an image or a pdf or a concept which refines one of them.+Be smart about multiplicity: when a step is to generate a variable or fixed number of items, use the multiplicity notation to specify it.+Don't create numbered variables like `idea_1`, `idea_2`, etc: trust that the pipes will handle the multiplicity for you.+You can name the variables with a plural name like "ideas" and make them use a concept with a singular name and a multiplicity like "Idea[]" or "Idea[5]".Be smart about splitting the workflow into steps (sequence or parallel):- You can use an LLM to extract or analyze several things at the same time, they can be output as a single concept which will be structured with attributes etc.- But don't ask the LLM for many things which are unrelated, it would lose reliability.+Apply the DRY principle: don't repeat yourself. if you have a task to apply several times, make it a dedicated pipe.+If you're in a sequence and you are to apply that pipe to a previous output which is multiple, plan to batch over it.++You must never include nore than one batch step in the same pipe sequence.+Instead, you must create a pipe sequence specifically for the process to apply to each batched element+and that sub pipe will have the second batch step.+Keep your style concise, no need to write tags such as "Description:", just write what you need to write.Do not write any intro or outro, just write the plan.@@ -147,8 +160,7 @@ List the concept drafts in Markdown format with a heading 3 for each, e.g. `###type = "PipeLLM"description = "Structure the concept definitions."inputs = { concept_drafts = "ConceptDrafts", brief = "UserBrief" }-output = "concept.ConceptSpec"-multiple_output = true+output = "concept.ConceptSpec[]"model = "llm_to_engineer"system_prompt = """You are an expert at data extraction and json formatting.@@ -166,8 +178,7 @@ Your task here is to extract a list of ConceptSpec from these concept drafts:type = "PipeLLM"description = "Write the pipe signatures for the plan."inputs = { plan_draft = "PlanDraft", brief = "UserBrief", concept_specs = "concept.ConceptSpec" }-output = "pipe_design.PipeSignature"-multiple_output = true+output = "pipe_design.PipeSignature[]"model = "llm_to_engineer"system_prompt = """You are a Senior engineer, very well versed in creating pipelines.@@ -204,18 +215,32 @@ Available pipe controllers:When describing the task of a pipe controller, be concise, don't detail all the sub-pipes.Available pipe operators:-- PipeLLM: A pipe that uses an LLM to generate a text, or a structured object. It is a vision LLM so it can also use images.- CRITICAL: When extracting MULTIPLE items (articles, employees, products), use multiple_output = true with SINGULAR concepts!+- PipeLLM: A pipe that uses an LLM to generate text or structured objects. It is a vision LLM so it can also use images.+ CRITICAL: When extracting MULTIPLE items (articles, employees, products), use bracket notation in output with SINGULAR concepts!- Create concept "Article" (not "Articles") with fields "item_name", "quantity" (not "item_names", "quantities")- - Then set multiple_output = true to get a list of Article objects-- PipeImgGen: A pipe that uses an AI model to generate an image.+ - Then set output = "Article[]" to get a variable list, or output = "Article[5]" for exactly 5 items+ - Examples: output = "Text[]" (multiple texts), output = "Image[3]" (exactly 3 images), output = "Employee[]" (list of employees)+- PipeImgGen: A pipe that uses an AI model to generate images.+ - Use bracket notation for multiple images: output = "Image[3]" generates exactly 3 imagesVERY IMPORTANT: IF YOU DECIDE TO CREATE A PipeImgGen, YOU ALSO HAVE TO CREATE A PIPELLM THAT WILL WRITE THE PROMPT, AND THAT NEEDS TO PRECEED THE PIPEIMGEN, based on the necessary elements.That means that in the MAIN pipeline, the prompt MUST NOT be considered as an input. It should be the output of a step that generates the prompt.-- PipeExtract: A pipe that extracts text from an image or a pdf. PipeExtract must have a exactly one input which must be either an `Image` or a `PDF`.+- PipeExtract: A pipe that extracts text from an image or a pdf. PipeExtract must have exactly one input which must be either an `Image` or a `PDF`.+ - Always outputs a list of pages: output = "Page[]"++Be smart about multiplicity: when a step is to generate a variable or fixed number of items, use the multiplicity notation to specify it.+Don't create numbered variables like `idea_1`, `idea_2`, etc: trust that the pipes will handle the multiplicity for you.+You can name the variables with a plural name like "ideas" and make them use a concept with a singular name and a multiplicity like "Idea[]" or "Idea[5]".Be smart about splitting the workflow into steps (sequence or parallel):- You can use an LLM to extract or analyze several things at the same time, they can be output as a single concept which will be structured with attributes etc.- But don't ask the LLM for many things which are unrelated, it would lose reliability.+- Apply the DRY principle: don't repeat yourself. if you have a task to apply several times, make it a dedicated pipe.+- If you're in a sequence and you are to apply that pipe to a previous output which is multiple, use batch_over/batch_as attributes in that step.+++You must never include nore than one batch step in the same pipe sequence.+Instead, you must create a pipe sequence specifically for the process to apply to each batched element+and that sub pipe will have the second batch step."""[pipe.assemble_pipelex_bundle_spec]diff --git a/pipelex/builder/builder.py b/pipelex/builder/builder.pyindex a6972df94..09f82b133 100644--- a/pipelex/builder/builder.py+++ b/pipelex/builder/builder.py@@ -1,4 +1,3 @@-from pathlib import Pathfrom typing import TYPE_CHECKING, castfrom pydantic import ValidationError@@ -8,21 +7,35 @@ from pipelex.builder.builder_errors import (PipelexBundleError,PipelexBundleUnexpectedError,)-from pipelex.builder.builder_validation import document_pipe_failures_from_dry_run_blueprint, dry_run_bundle_blueprint+from pipelex.builder.builder_validation import (+ validate_dry_run_bundle_blueprint,+)from pipelex.builder.bundle_header_spec import BundleHeaderSpecfrom pipelex.builder.bundle_spec import PipelexBundleSpecfrom pipelex.builder.concept.concept_spec import ConceptSpec-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec+from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec+from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec+from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpec+from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpec+from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec+from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec+from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.builder.pipe.pipe_spec_map import pipe_type_to_spec_classfrom pipelex.builder.pipe.pipe_spec_union import PipeSpecUnion+from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.core.interpreter import PipelexInterpreterfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.structured_content import StructuredContentfrom pipelex.system.registries.func_registry import pipe_funcfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorif TYPE_CHECKING:from pipelex.core.stuffs.list_content import ListContent+ from pipelex.core.stuffs.stuff_content import StuffContent# # TODO: Put this in a factory. Investigate why it is necessary.@@ -55,7 +68,29 @@ async def assemble_pipelex_bundle_spec(working_memory: WorkingMemory) -> Pipelexitem_type=ConceptSpec,)- pipe_specs: list[PipeSpecUnion] = cast("ListContent[PipeSpecUnion]", working_memory.get_stuff(name="pipe_specs").content).items+ # pipe_specs: list[PipeSpecUnion] = cast("ListContent[PipeSpecUnion]", working_memory.get_stuff(name="pipe_specs").content).items+ pipe_specs_list: ListContent[StuffContent] = working_memory.get_stuff_as_list(name="pipe_specs", item_type=StructuredContent)+ pipe_specs_list_items: list[StuffContent] = pipe_specs_list.items+ pipe_specs: list[PipeSpecUnion] = []+ for pipe_spec_item in pipe_specs_list_items:+ if not isinstance(+ pipe_spec_item,+ (+ PipeFuncSpec,+ PipeImgGenSpec,+ PipeComposeSpec,+ PipeLLMSpec,+ PipeExtractSpec,+ PipeBatchSpec,+ PipeConditionSpec,+ PipeParallelSpec,+ PipeSequenceSpec,+ ),+ ):+ msg = f"Pipe spec item '{pipe_spec_item}' is not any type of PipeSpecUnion, it's a {type(pipe_spec_item)}"+ raise PipeBuilderError(msg)+ pipe_specs.append(pipe_spec_item)+bundle_header_spec = working_memory.get_stuff_as(name="bundle_header_spec", content_type=BundleHeaderSpec)# Properly validate and reconstruct concept specs to ensure proper Pydantic validation@@ -64,7 +99,7 @@ async def assemble_pipelex_bundle_spec(working_memory: WorkingMemory) -> Pipelextry:# Re-create the ConceptSpec to ensure proper Pydantic validation# This handles any serialization/deserialization issues from working memory- validated_concept = ConceptSpec(**concept_spec.model_dump(serialize_as_any=True))+ validated_concept = ConceptSpec.model_validate(concept_spec.model_dump(serialize_as_any=True))validated_concepts[validated_concept.the_concept_code] = validated_conceptexcept ValidationError as exc:msg = f"Failed to validate concept spec {concept_spec.the_concept_code}: {format_pydantic_validation_error(exc)}"@@ -125,36 +160,24 @@ async def reconstruct_bundle_with_all_fixes(working_memory: WorkingMemory) -> Pireturn pipelex_bundle_spec-async def load_pipe_from_bundle(bundle_path: str) -> str:+async def load_and_validate_bundle(bundle_path: str) -> PipelexBundleBlueprint:"""Load a bundle file and extract its main_pipe.Args:bundle_path: Path to the .plx bundle file.Returns:- The pipe_code from the bundle's main_pipe.+ PipelexBundleBlueprint: The blueprint of the bundle.Raises:- FileNotFoundError: If the bundle file does not exist.- PipelexBundleError: If no main_pipe is declared or if pipes fail during dry run.- PipeInputError: If there are input errors during dry run validation.+ PipelexBundleError: If any pipe failed during dry run."""- bundle_path_obj = Path(bundle_path)- if not bundle_path_obj.exists():- msg = f"Bundle file not found: {bundle_path}"- raise FileNotFoundError(msg)-- interpreter = PipelexInterpreter(file_path=bundle_path_obj)- bundle_blueprint = interpreter.make_pipelex_bundle_blueprint()-- if not bundle_blueprint.main_pipe:- msg = f"Bundle '{bundle_path}' does not declare a main_pipe"- raise PipelexBundleError(message=msg)+ bundle_blueprint = PipelexInterpreter.load_bundle_blueprint(bundle_path=bundle_path)- dry_run_result = await dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)- pipe_failures = document_pipe_failures_from_dry_run_blueprint(bundle_blueprint=bundle_blueprint, dry_run_result=dry_run_result)- if pipe_failures:- msg = f"Pipes failed during dry run in bundle '{bundle_path}'"- raise PipelexBundleError(message=msg, pipe_failures=pipe_failures)+ try:+ await validate_dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)+ except PipelexBundleError as exc:+ msg = f"Bundle at '{bundle_path}' failed to validate"+ raise PipelexBundleError(message=msg, pipe_failures=exc.pipe_failures) from exc- return bundle_blueprint.main_pipe+ return bundle_blueprintdiff --git a/pipelex/builder/builder_errors.py b/pipelex/builder/builder_errors.pyindex c30895a3d..5dd74dbf0 100644--- a/pipelex/builder/builder_errors.py+++ b/pipelex/builder/builder_errors.py@@ -72,6 +72,15 @@ class PipeDefinitionErrorData(StructuredContent):source: str | None = Field(None, description="Source of the error")+class PipeInputErrorData(StructuredContent):+ """Structured data for PipeInputError."""++ message: str = Field(description="The error message")+ pipe_code: str | None = Field(None, description="The pipe code")+ variable_name: str | None = Field(None, description="The variable name")+ concept_code: str | None = Field(None, description="The concept code")++class DomainFailure(StructuredContent):"""Details of a single domain failure during dry run."""@@ -109,6 +118,7 @@ class PipelexBundleErrorData(StructuredContent):message: str = Field(description="The main error message")static_validation_error: StaticValidationErrorData | None = Field(None, description="Static validation error if present")domain_failures: list[DomainFailure] | None = Field(None, description="List of domain failures")+ pipe_input_errors: list[PipeInputErrorData] | None = Field(None, description="List of pipe input errors")pipe_failures: list[PipeFailure] | None = Field(None, description="List of pipe failures")concept_failures: list[ConceptFailure] | None = Field(None, description="List of concept failures")concept_definition_errors: list[ConceptDefinitionErrorData] | None = Field(None, description="List of concept definition errors")@@ -164,9 +174,11 @@ class PipelexBundleError(PipelexException):concept_failures: list[ConceptFailure] | None = None,concept_definition_errors: list[ConceptDefinitionErrorData] | None = None,pipe_definition_errors: list[PipeDefinitionErrorData] | None = None,+ pipe_input_errors: list[PipeInputErrorData] | None = None,) -> None:self.static_validation_error = static_validation_errorself.domain_failures = domain_failures+ self.pipe_input_errors = pipe_input_errorsself.pipe_failures = pipe_failuresself.concept_failures = concept_failuresself.concept_definition_errors = concept_definition_errors@@ -179,6 +191,7 @@ class PipelexBundleError(PipelexException):message=str(self),static_validation_error=self.static_validation_error,domain_failures=self.domain_failures,+ pipe_input_errors=self.pipe_input_errors,pipe_failures=self.pipe_failures,concept_failures=self.concept_failures,concept_definition_errors=self.concept_definition_errors,diff --git a/pipelex/builder/builder_loop.py b/pipelex/builder/builder_loop.pyindex b57243ac0..a44998714 100644--- a/pipelex/builder/builder_loop.py+++ b/pipelex/builder/builder_loop.py@@ -16,11 +16,17 @@ from pipelex.exceptions import StaticValidationErrorTypefrom pipelex.hub import get_required_pipefrom pipelex.language.plx_factory import PlxFactoryfrom pipelex.pipeline.execute import execute_pipeline-from pipelex.tools.misc.file_utils import save_text_to_path+from pipelex.tools.misc.file_utils import get_incremental_file_path, save_text_to_pathclass BuilderLoop:- async def build_and_fix(self, pipe_code: str, input_memory: PipelineInputs | None = None) -> PipelexBundleSpec:+ async def build_and_fix(+ self,+ pipe_code: str,+ input_memory: PipelineInputs | None = None,+ is_save_first_iteration_enabled: bool = True,+ is_save_second_iteration_enabled: bool = True,+ ) -> PipelexBundleSpec:pretty_print(f"Building and fixing with {pipe_code}")pipe_output = await execute_pipeline(pipe_code=pipe_code,@@ -31,17 +37,33 @@ class BuilderLoop:pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)pretty_print(pipelex_bundle_spec, title="Pipelex Bundle Spec • 1st iteration")plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())- save_text_to_path(text=plx_content, path="results/generated_pipeline_1st_iteration.plx")++ if is_save_first_iteration_enabled:+ first_iteration_path = get_incremental_file_path(+ base_path="results",+ base_name="generated_pipeline_1st_iteration",+ extension="plx",+ )+ save_text_to_path(text=plx_content, path=first_iteration_path)try:await validate_bundle_spec(bundle_spec=pipelex_bundle_spec)except PipelexBundleError as bundle_error:pretty_print(bundle_error.as_structured_content(), title="Pipelex Bundle Error")- pipelex_bundle_spec = self._fix_bundle_error(bundle_error=bundle_error, pipelex_bundle_spec=pipelex_bundle_spec)+ pipelex_bundle_spec = self._fix_bundle_error(+ bundle_error=bundle_error,+ pipelex_bundle_spec=pipelex_bundle_spec,+ is_save_second_iteration_enabled=is_save_second_iteration_enabled,+ )return pipelex_bundle_spec- def _fix_bundle_error(self, bundle_error: PipelexBundleError, pipelex_bundle_spec: PipelexBundleSpec) -> PipelexBundleSpec:+ def _fix_bundle_error(+ self,+ bundle_error: PipelexBundleError,+ pipelex_bundle_spec: PipelexBundleSpec,+ is_save_second_iteration_enabled: bool,+ ) -> PipelexBundleSpec:fixed_pipes: list[PipeSpecUnion] = []# Fix static validation errors for PipeController inputs@@ -84,10 +106,15 @@ class BuilderLoop:msg = "Static validation error had too many candidate inputs. We don't support fixing this error yet."raise PipelexBundleNoFixForError(message=msg) from bundle_error- if fixed_pipes:+ if fixed_pipes and is_save_second_iteration_enabled:pipelex_bundle_spec = reconstruct_bundle_with_pipe_fixes(pipelex_bundle_spec=pipelex_bundle_spec, fixed_pipes=fixed_pipes)pretty_print(pipelex_bundle_spec, title="Pipelex Bundle Spec • 2nd iteration")plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())- save_text_to_path(text=plx_content, path="results/generated_pipeline_2nd_iteration.plx")+ second_iteration_path = get_incremental_file_path(+ base_path="results",+ base_name="generated_pipeline_2nd_iteration",+ extension="plx",+ )+ save_text_to_path(text=plx_content, path=second_iteration_path)return pipelex_bundle_specdiff --git a/pipelex/builder/builder_validation.py b/pipelex/builder/builder_validation.pyindex 50f712491..2b7405120 100644--- a/pipelex/builder/builder_validation.py+++ b/pipelex/builder/builder_validation.py@@ -4,6 +4,7 @@ from pipelex.builder.builder_errors import (DomainFailure,PipeDefinitionErrorData,PipeFailure,+ PipeInputErrorData,PipelexBundleError,PipelexBundleUnexpectedError,PipeSpecError,@@ -17,6 +18,7 @@ from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.exceptions import (ConceptLoadingError,DomainLoadingError,+ PipeInputError,PipeLoadingError,StaticValidationError,)@@ -145,5 +147,20 @@ async def dry_run_bundle_blueprint(bundle_blueprint: PipelexBundleBlueprint) ->source=pipe_def_error.source,)raise PipelexBundleError(message=pipe_loading_error.message, pipe_definition_errors=[pipe_definition_error_data]) from pipe_loading_error-+ except PipeInputError as pipe_input_error:+ pipe_input_error_data = PipeInputErrorData(+ message=str(pipe_input_error),+ pipe_code=pipe_input_error.pipe_code,+ variable_name=pipe_input_error.variable_name,+ concept_code=pipe_input_error.concept_code,+ )+ raise PipelexBundleError(message=pipe_input_error.message, pipe_input_errors=[pipe_input_error_data]) from pipe_input_errorreturn dry_run_result+++async def validate_dry_run_bundle_blueprint(bundle_blueprint: PipelexBundleBlueprint):+ dry_run_result = await dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)+ pipe_failures = document_pipe_failures_from_dry_run_blueprint(bundle_blueprint=bundle_blueprint, dry_run_result=dry_run_result)+ if pipe_failures:+ msg = "Dry run failed for bundle"+ raise PipelexBundleError(message=msg, pipe_failures=pipe_failures)diff --git a/pipelex/builder/concept/concept.plx b/pipelex/builder/concept/concept.plxindex e9b6f96ef..2a11af9c4 100644--- a/pipelex/builder/concept/concept.plx+++ b/pipelex/builder/concept/concept.plx@@ -20,17 +20,15 @@ steps = [type = "PipeLLM"description = "Convert the ConceptSpec (with its structure draft) into a proper ConceptStructureSpec."inputs = { concept_spec_draft = "ConceptSpecDraft" }-output = "ConceptStructureSpec"-multiple_output = true+output = "ConceptStructureSpec[]"model = "llm_to_engineer"prompt = """Create a ConceptStructureSpec from the ConceptSpecDraft.Please focus only on the structure.-The field "definition" IS NOT a structure. It is a general definition of the concept.+The field "description" IS NOT a structure. It is a general description of the concept.If the field "structure" is empty, return an empty list.-ConceptSpecDraft:@concept_spec_draft"""diff --git a/pipelex/builder/concept/concept_spec.py b/pipelex/builder/concept/concept_spec.pyindex 107a08b27..c94e101c6 100644--- a/pipelex/builder/concept/concept_spec.py+++ b/pipelex/builder/concept/concept_spec.py@@ -1,3 +1,4 @@+import refrom datetime import datetimefrom typing import Any@@ -223,21 +224,31 @@ class ConceptSpec(StructuredContent):@classmethoddef validate_concept_string_or_code(cls, concept_string_or_code: str) -> None:- if concept_string_or_code.count(".") > 1:+ # Strip multiplicity brackets if present (e.g., 'Text[]' or 'Text[2]' -> 'Text')++ multiplicity_pattern = r"^(.+?)(?:\[\d*\])?$"+ match = re.match(multiplicity_pattern, concept_string_or_code)+ if not match:+ msg = f"Invalid concept string format: '{concept_string_or_code}'"+ raise ConceptStringOrConceptCodeError(msg)++ concept_without_multiplicity = match.group(1)++ if concept_without_multiplicity.count(".") > 1:msg = (- f"concept_string_or_code '{concept_string_or_code}' is invalid. "+ f"concept_string_or_code '{concept_without_multiplicity}' is invalid. ""It should either contain a domain in snake_case and a concept code in PascalCase separated by one dot, ""or be a concept code in PascalCase.")raise ConceptStringOrConceptCodeError(msg)- if concept_string_or_code.count(".") == 1:- domain, concept_code = concept_string_or_code.split(".")+ if concept_without_multiplicity.count(".") == 1:+ domain, concept_code = concept_without_multiplicity.split(".")# Validate domain codeDomainBlueprint.validate_domain_code(code=domain)cls._post_validate_concept_code(concept_code=concept_code)else:- cls._post_validate_concept_code(concept_code=concept_string_or_code)+ cls._post_validate_concept_code(concept_code=concept_without_multiplicity)def to_blueprint(self) -> ConceptBlueprint:"""Convert this ConceptBlueprint to the original core ConceptBlueprint."""diff --git a/pipelex/builder/flow_factory.py b/pipelex/builder/flow_factory.pyindex 5625dfb73..ceea24421 100644--- a/pipelex/builder/flow_factory.py+++ b/pipelex/builder/flow_factory.py@@ -62,7 +62,7 @@ class FlowFactory:PipeBatchBlueprint | PipeConditionBlueprint | PipeParallelBlueprint | PipeSequenceBlueprint,): # pyright: ignore[reportUnnecessaryIsInstance]flow_elements[pipe_code] = FlowElement(controller_blueprint=pipe_blueprint)- log.debug(+ log.verbose(f"Adding controller {pipe_code} to flow: category is '{pipe_blueprint.pipe_category}' and type is '{pipe_blueprint.type}'")else:@@ -72,10 +72,10 @@ class FlowFactory:# Convert operators to signaturessignature_from_blueprint = FlowFactory._convert_blueprint_to_signature(pipe_code=pipe_code, pipe_blueprint=pipe_blueprint)flow_elements[pipe_code] = FlowElement(operator_signature=signature_from_blueprint)- log.debug(+ log.verbose(f"Adding operator {pipe_code} to flow: category is '{pipe_blueprint.pipe_category}' and type is '{pipe_blueprint.type}'")- log.debug(signature_from_blueprint, title="Signature from blueprint")+ log.verbose(signature_from_blueprint, title="Signature from blueprint")return Flow(domain=bundle_blueprint.domain,description=bundle_blueprint.description,diff --git a/pipelex/builder/pipe/pipe_batch_spec.py b/pipelex/builder/pipe/pipe_batch_spec.pyindex e1332b50c..7eb6605d2 100644--- a/pipelex/builder/pipe/pipe_batch_spec.py+++ b/pipelex/builder/pipe/pipe_batch_spec.py@@ -3,7 +3,7 @@ from typing import Literalfrom pydantic import Fieldfrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprintdiff --git a/pipelex/builder/pipe/pipe_compose_spec.py b/pipelex/builder/pipe/pipe_compose_spec.pyindex 2650043b8..e341b66db 100644--- a/pipelex/builder/pipe/pipe_compose_spec.py+++ b/pipelex/builder/pipe/pipe_compose_spec.py@@ -4,7 +4,7 @@ from pydantic import Field, field_validatorfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprintfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormatdiff --git a/pipelex/builder/pipe/pipe_condition_spec.py b/pipelex/builder/pipe/pipe_condition_spec.pyindex 23b763f13..e48e4dcea 100644--- a/pipelex/builder/pipe/pipe_condition_spec.py+++ b/pipelex/builder/pipe/pipe_condition_spec.py@@ -4,7 +4,7 @@ from pydantic import Fieldfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprintfrom pipelex.pipe_controllers.condition.special_outcome import SpecialOutcomediff --git a/pipelex/builder/pipe/pipe_design.plx b/pipelex/builder/pipe/pipe_design.plxindex fb47efdf5..103740422 100644--- a/pipelex/builder/pipe/pipe_design.plx+++ b/pipelex/builder/pipe/pipe_design.plx@@ -56,6 +56,8 @@ You will specifically generate the PipeSequence related to this signature:Here are the concepts you can use for inputs/outputs:@concept_specs++- If you are to apply a pipe step to a previous output which is multiple, use batch_over/batch_as attributes in that step."""[pipe.detail_pipe_parallel]diff --git a/pipelex/builder/pipe/pipe_extract_spec.py b/pipelex/builder/pipe/pipe_extract_spec.pyindex 286e0df5f..a8919da2e 100644--- a/pipelex/builder/pipe/pipe_extract_spec.py+++ b/pipelex/builder/pipe/pipe_extract_spec.py@@ -4,7 +4,7 @@ from pydantic import Field, field_validatorfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.exceptions import PipeDefinitionErrorfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprintfrom pipelex.types import StrEnum@@ -13,23 +13,9 @@ if TYPE_CHECKING:from pipelex.cogt.extract.extract_setting import ExtractModelChoice-class AvailableExtractModel(StrEnum):- BASE_OCR_MISTRAL = "base_ocr_mistral"- # BASE_EXTRACT_PYPDFIUM2 = "base_extract_pypdfium2"--class ExtractSkill(StrEnum):EXTRACT_TEXT_FROM_VISUALS = "extract_text_from_visuals"- EXTARCT_TEXT_FROM_PDF = "extract_text_from_pdf"-- @property- def model_recommendation(self) -> AvailableExtractModel:- match self:- case ExtractSkill.EXTRACT_TEXT_FROM_VISUALS:- return AvailableExtractModel.BASE_OCR_MISTRAL- case ExtractSkill.EXTARCT_TEXT_FROM_PDF:- # TODO: Debug the BaseOcrPypdfium2- return AvailableExtractModel.BASE_OCR_MISTRAL+ EXTRACT_TEXT_FROM_PDF = "extract_text_from_pdf"class PipeExtractSpec(PipeSpec):@@ -51,10 +37,11 @@ class PipeExtractSpec(PipeSpec):page_image_captions: bool | None = Field(default=None, description="Whether to generate captions for detected images using AI.")page_views: bool | None = Field(default=None, description="Whether to include rendered page views in the output.")+ @override@field_validator("output", mode="before")@classmethod- def validate_output(cls, _: str) -> str:- return "Page"+ def validate_output(cls, output: str) -> str:+ return "Page[]"@field_validator("extract_skill", mode="before")@classmethod@@ -65,10 +52,10 @@ class PipeExtractSpec(PipeSpec):@classmethoddef validate_extract_inputs(cls, inputs_value: dict[str, str] | None) -> dict[str, str] | None:if inputs_value is None:- msg = "PipeExtract must have exactly one input which must be either`Image` or `PDF`."+ msg = "PipeExtract must have exactly one input which must be either `Image` or `PDF`."raise PipeDefinitionError(msg)if len(inputs_value) != 1:- msg = "PipeExtract must have exactly one input which must be either`Image` or `PDF`."+ msg = "PipeExtract must have exactly one input which must be either `Image` or `PDF`."raise PipeDefinitionError(msg)return inputs_value@@ -77,11 +64,7 @@ class PipeExtractSpec(PipeSpec):base_blueprint = super().to_blueprint()# create extract choice as a str- extract_model_choice: ExtractModelChoice- if isinstance(self.extract_skill, ExtractSkill):- extract_model_choice = self.extract_skill.model_recommendation.value- else:- extract_model_choice = ExtractSkill(self.extract_skill).model_recommendation.value+ extract_model_choice: ExtractModelChoice = self.extract_skillreturn PipeExtractBlueprint(source=None,diff --git a/pipelex/builder/pipe/pipe_func_spec.py b/pipelex/builder/pipe/pipe_func_spec.pyindex 20c8b8731..0cb0da548 100644--- a/pipelex/builder/pipe/pipe_func_spec.py+++ b/pipelex/builder/pipe/pipe_func_spec.py@@ -4,7 +4,7 @@ from pydantic import Fieldfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprintdiff --git a/pipelex/builder/pipe/pipe_img_gen_spec.py b/pipelex/builder/pipe/pipe_img_gen_spec.pyindex 55924a085..9f3a0d2b4 100644--- a/pipelex/builder/pipe/pipe_img_gen_spec.py+++ b/pipelex/builder/pipe/pipe_img_gen_spec.py@@ -1,34 +1,21 @@-from typing import Literal+from typing import TYPE_CHECKING, Literalfrom pydantic import Field, field_validatorfrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprintfrom pipelex.types import StrEnum--class AvailableImgGen(StrEnum):- BASE_IMG_GEN = "base_img_gen"- FAST_IMG_GEN = "fast_img_gen"- HIGH_QUALITY_IMG_GEN = "high_quality_img_gen"+if TYPE_CHECKING:+ from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoiceclass ImgGenSkill(StrEnum):- GEN_IMAGE = "gen_image"+ GEN_IMAGE_BASIC = "gen_image_basic"GEN_IMAGE_FAST = "gen_image_fast"GEN_IMAGE_HIGH_QUALITY = "gen_image_high_quality"- @property- def model_recommendation(self) -> AvailableImgGen:- match self:- case ImgGenSkill.GEN_IMAGE:- return AvailableImgGen.BASE_IMG_GEN- case ImgGenSkill.GEN_IMAGE_FAST:- return AvailableImgGen.FAST_IMG_GEN- case ImgGenSkill.GEN_IMAGE_HIGH_QUALITY:- return AvailableImgGen.HIGH_QUALITY_IMG_GEN-class PipeImgGenSpec(PipeSpec):"""Specs for image generation pipe operations in the Pipelex framework.@@ -36,12 +23,16 @@ class PipeImgGenSpec(PipeSpec):PipeImgGen enables AI-powered image generation using various models like DALL-E ordiffusion models. Supports static and dynamic prompts with configurable generationparameters.++ Output Multiplicity:+ Specify using bracket notation in output field:+ - output = "Image" - single image (default)+ - output = "Image[3]" - exactly 3 images"""type: Literal["PipeImgGen"] = "PipeImgGen"pipe_category: Literal["PipeOperator"] = "PipeOperator"- img_gen_skill: ImgGenSkill | None = None- nb_output: int | None = Field(default=None, ge=1)+ img_gen_skill: ImgGenSkill | str = Field(description="Select the most adequate image generation skill according to the task to be performed.")@field_validator("img_gen_skill", mode="before")@classmethod@@ -55,6 +46,9 @@ class PipeImgGenSpec(PipeSpec):def to_blueprint(self) -> PipeImgGenBlueprint:"""Convert this PipeImgGenBlueprint to the core PipeImgGenBlueprint."""base_blueprint = super().to_blueprint()++ img_gen_choice: ImgGenModelChoice = self.img_gen_skill+return PipeImgGenBlueprint(description=base_blueprint.description,inputs=base_blueprint.inputs,@@ -63,11 +57,10 @@ class PipeImgGenSpec(PipeSpec):pipe_category=self.pipe_category,img_gen_prompt=None,img_gen_prompt_var_name=None,- model=self.img_gen_skill.model_recommendation if self.img_gen_skill else None,+ model=img_gen_choice,aspect_ratio=None,background=None,output_format=None,is_raw=None,seed=None,- nb_output=self.nb_output,)diff --git a/pipelex/builder/pipe/pipe_llm_spec.py b/pipelex/builder/pipe/pipe_llm_spec.pyindex 3c93c1ea0..d0d30ffba 100644--- a/pipelex/builder/pipe/pipe_llm_spec.py+++ b/pipelex/builder/pipe/pipe_llm_spec.py@@ -1,71 +1,32 @@from typing import TYPE_CHECKING, Literal-from pydantic import Field, field_validator, model_validator+from pydantic import Field, field_validatorfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec-from pipelex.cogt.llm.llm_setting import LLMSetting-from pipelex.exceptions import PipeDefinitionError+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint-from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_list-from pipelex.types import Self, StrEnum+from pipelex.types import StrEnumif TYPE_CHECKING:from pipelex.cogt.llm.llm_setting import LLMModelChoice-class AvailableLLM(StrEnum):- CLAUDE_4_SONNET = "claude-4.5-sonnet"- CLAUDE_4_1_OPUS = "claude-4.1-opus"- GPT_5 = "gpt-5"- GEMINI_2_5_PRO = "gemini-2.5-pro"- GEMINI_2_5_FLASH = "gemini-2.5-flash"- GEMINI_2_5_FLASH_LITE = "gemini-2.5-flash-lite"--class LLMSkill(StrEnum):LLM_TO_RETRIEVE = "llm_to_retrieve"- LLM_CHEAP_FOR_EASY_QUESTIONS = "llm_cheap_for_easy_questions"+ LLM_TO_ANSWER_EASY_QUESTIONS = "llm_to_answer_easy_questions"LLM_TO_ANSWER_HARD_QUESTIONS = "llm_to_answer_hard_questions"- LLM_CHEAP_FOR_VISION = "llm_cheap_for_vision"+ LLM_TO_WRITE_QUESTIONS = "llm_to_write_questions"+ LLM_FOR_BASIC_VISION = "llm_for_basic_vision"LLM_FOR_VISUAL_ANALYSIS = "llm_for_visual_analysis"LLM_FOR_VISUAL_DESIGN = "llm_for_visual_design"LLM_FOR_CREATIVE_WRITING = "llm_for_creative_writing"- LLM_TO_REASON = "llm_to_reason"+ LLM_FOR_COMPLEX_REASONING = "llm_for_complex_reasoning"LLM_TO_REASON_ON_DIAGRAM = "llm_to_reason_on_diagram"LLM_TO_ANALYZE_DATA = "llm_to_analyze_data"LLM_TO_CODE = "llm_to_code"LLM_TO_ANALYZE_LARGE_CODEBASE = "llm_to_analyze_large_codebase"- @property- def llm_recommendation(self) -> AvailableLLM:- match self:- case LLMSkill.LLM_TO_RETRIEVE:- return AvailableLLM.GEMINI_2_5_FLASH- case LLMSkill.LLM_CHEAP_FOR_EASY_QUESTIONS:- return AvailableLLM.CLAUDE_4_SONNET- case LLMSkill.LLM_TO_ANSWER_HARD_QUESTIONS:- return AvailableLLM.GPT_5- case LLMSkill.LLM_CHEAP_FOR_VISION:- return AvailableLLM.GEMINI_2_5_FLASH_LITE- case LLMSkill.LLM_FOR_VISUAL_ANALYSIS:- return AvailableLLM.GEMINI_2_5_FLASH- case LLMSkill.LLM_FOR_VISUAL_DESIGN:- return AvailableLLM.GEMINI_2_5_FLASH- case LLMSkill.LLM_FOR_CREATIVE_WRITING:- return AvailableLLM.CLAUDE_4_1_OPUS- case LLMSkill.LLM_TO_REASON:- return AvailableLLM.CLAUDE_4_SONNET- case LLMSkill.LLM_TO_REASON_ON_DIAGRAM:- return AvailableLLM.GPT_5- case LLMSkill.LLM_TO_ANALYZE_DATA:- return AvailableLLM.CLAUDE_4_SONNET- case LLMSkill.LLM_TO_CODE:- return AvailableLLM.CLAUDE_4_SONNET- case LLMSkill.LLM_TO_ANALYZE_LARGE_CODEBASE:- return AvailableLLM.GEMINI_2_5_PRO-class PipeLLMSpec(PipeSpec):"""Spec for LLM-based pipe operations in the Pipelex framework.@@ -73,18 +34,17 @@ class PipeLLMSpec(PipeSpec):PipeLLM enables Large Language Model processing to generate text or structured output.Supports text, structured data, and image inputs.- Validation Rules:- nb_output: Fixed number of outputs to generate (e.g., 3 for exactly 3 outputs).- Must be > 1. Mutually exclusive with multiple_output.- multiple_output: Enables variable-length list generation. Default is false (single output).- Set to true for indeterminate number of outputs. Mutually exclusive with nb_output.+ Output Multiplicity:+ Specify using bracket notation in output field:+ - output = "Text" - single item (default)+ - output = "Text[]" - variable list+ - output = "Text[3]" - exactly 3 items"""type: SkipJsonSchema[Literal["PipeLLM"]] = "PipeLLM"pipe_category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"- llm: LLMSkill | str = Field(description="Select the most adequate LLM model skill according to the task to be performed.")- temperature: float | None = Field(default=None, ge=0, le=1)+ llm_skill: LLMSkill | str = Field(description="Select the simplest LLM skill corresponding to the task to be performed.")system_prompt: str | None = Field(default=None, description="A system prompt to guide the LLM's behavior, style and skills. Can be a template.")prompt: str | None = Field(description="""A template for the user prompt:@@ -105,50 +65,17 @@ So, don't have to write a bullet-list of all the attributes definitions yourself""")- nb_output: int | None = Field(- default=None,- description=(- "Specifies exactly how many outputs to generate (e.g., `nb_output = 3` for exactly 3 outputs). "- "Set it if we need a fixed number of results."- ),- gt=1,- )- multiple_output: bool | None = Field(- default=None,- description=(- "Controls output generation mode. Set to `true` for variable-length list generation when we need an indeterminate number of outputs."- ),- )-- @field_validator("llm", mode="before")+ @field_validator("llm_skill", mode="before")@classmethoddef validate_llm(cls, llm_value: str) -> LLMSkill:return LLMSkill(llm_value)- @model_validator(mode="after")- def validate_multiple_output(self) -> Self:- if excess_attributes_list := has_more_than_one_among_attributes_from_list(- self,- attributes_list=["nb_output", "multiple_output"],- ):- msg = f"PipeLLMSpec must have no more than one of {excess_attributes_list}"- raise PipeDefinitionError(msg)- return self-@overridedef to_blueprint(self) -> PipeLLMBlueprint:base_blueprint = super().to_blueprint()# create llm choice as a str- llm_choice: LLMModelChoice- if isinstance(self.llm, LLMSkill):- llm_choice = self.llm.llm_recommendation.value- else:- llm_choice = LLMSkill(self.llm).llm_recommendation.value-- # Make it a LLMSetting if temperature is provided- if self.temperature:- llm_choice = LLMSetting(model=llm_choice, temperature=self.temperature)+ llm_choice: LLMModelChoice = self.llm_skillreturn PipeLLMBlueprint(type="PipeLLM",@@ -159,6 +86,4 @@ So, don't have to write a bullet-list of all the attributes definitions yourselfsystem_prompt=self.system_prompt,prompt=self.prompt,model=llm_choice,- nb_output=self.nb_output,- multiple_output=self.multiple_output,)diff --git a/pipelex/builder/pipe/pipe_parallel_spec.py b/pipelex/builder/pipe/pipe_parallel_spec.pyindex e468b6aad..629651ff3 100644--- a/pipelex/builder/pipe/pipe_parallel_spec.py+++ b/pipelex/builder/pipe/pipe_parallel_spec.py@@ -4,7 +4,7 @@ from pydantic import Field, field_validator, model_validatorfrom typing_extensions import overridefrom pipelex.builder.concept.concept_spec import ConceptSpec-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.exceptions import PipeDefinitionErrorfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprintdiff --git a/pipelex/builder/pipe/pipe_sequence_spec.py b/pipelex/builder/pipe/pipe_sequence_spec.pyindex 1ff721629..cc137d522 100644--- a/pipelex/builder/pipe/pipe_sequence_spec.py+++ b/pipelex/builder/pipe/pipe_sequence_spec.py@@ -4,7 +4,7 @@ from pydantic import Fieldfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprintdiff --git a/pipelex/builder/pipe/pipe_signature.py b/pipelex/builder/pipe/pipe_signature.pyindex 7a19bccbf..39dfecc6f 100644--- a/pipelex/builder/pipe/pipe_signature.py+++ b/pipelex/builder/pipe/pipe_signature.py@@ -3,23 +3,29 @@ from typing import Anyfrom pydantic import Field, field_validator, model_validatorfrom pydantic.json_schema import SkipJsonSchema-from pipelex import log-from pipelex.builder.concept.concept_spec import ConceptSpecfrom pipelex.core.pipes.exceptions import PipeBlueprintError-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories, AllowedPipeTypes, PipeBlueprint+from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories, AllowedPipeTypesfrom pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.tools.misc.string_utils import is_snake_case, normalize_to_asciiclass PipeSignature(StructuredContent):"""PipeSignature is a contract for a pipe.- It defines the inputs, outputs, and the purpose of the pipe.- It doesn't go into the details of how it does it.++ It defines the inputs, outputs, and the purpose of the pipe without implementation details.++ Multiplicity Notation:+ Use bracket notation to specify how many items are produced, but only for the outputs:+ - No brackets: single item (default)+ - []: variable-length list+ - [N]: exactly N items (where N is a positive integer)++ Examples:+ - output = "Text[]" - produces multiple text items+ - output = "Image[3]" - produces exactly 3 images"""code: str = Field(description="Pipe code identifying the pipe. Must be snake_case.")- type: AllowedPipeTypes = Field(description="Pipe type.")+ type: AllowedPipeTypes | str = Field(description="Pipe type.")pipe_category: SkipJsonSchema[AllowedPipeCategories] = Field(description="Pipe category set according to its type.")description: str = Field(description="What the pipe does")inputs: dict[str, str] = Field(@@ -28,7 +34,7 @@ class PipeSignature(StructuredContent):result: str = Field(description="variable_name for the result of the pipe. Must be snake_case. It could be referenced as input in a following pipe.")- output: str = Field(description="Just the output ConceptCode in PascalCase")+ output: str = Field(description="The output ConceptCode in PascalCase, possibly with multiplicity brackets")pipe_dependencies: list[str] = Field(description="List of pipe codes that this pipe depends on. This is for the PipeControllers")@model_validator(mode="before")@@ -44,81 +50,7 @@ class PipeSignature(StructuredContent):values["pipe_category"] = the_type.categoryreturn values--class PipeSpec(StructuredContent):- """Spec defining a pipe: an executable component with a clear contract defined by its inputs and output.- There are two categories of pipes: controllers and operators.- Controllers are used to control the flow of the pipeline, and operators are used to perform specific tasks.- """-- pipe_code: str = Field(description="Pipe code. Must be snake_case.")- type: Any = Field(description=f"Pipe type. It is defined with type `Any` but validated at runtime and it must be one of: {AllowedPipeTypes}")- pipe_category: Any = Field(- description=f"Pipe category. It is defined with type `Any` but validated at runtime and it must be one of: {AllowedPipeCategories}"- )- description: str | None = Field(description="Natural language description of what the pipe does.")- inputs: dict[str, str] = Field(- description=("Input concept specifications. The keys are input names in snake_case. Each value must be a ConceptCode in PascalCase"),- )- output: str = Field(description="Output concept code in PascalCase format!! Very important")-- @field_validator("pipe_code", mode="before")- @classmethod- def validate_pipe_code(cls, value: str) -> str:- return cls.validate_pipe_code_syntax(value)-- @field_validator("type", mode="after")- @classmethod- def validate_pipe_type(cls, value: Any) -> Any:- if value not in AllowedPipeTypes.value_list():- msg = f"Invalid pipe type '{value}'. Must be one of: {AllowedPipeTypes.value_list()}"- raise PipeBlueprintError(msg)- return value-- @field_validator("output", mode="after")+ @field_validator("type", mode="before")@classmethod- def validate_concept_string_or_code(cls, output: str) -> str:- ConceptSpec.validate_concept_string_or_code(concept_string_or_code=output)- return output-- @field_validator("inputs", mode="after")- @classmethod- def validate_inputs(cls, inputs: dict[str, str] | None) -> dict[str, str] | None:- if inputs is None:- return None- for input_name, concept_code in inputs.items():- if not is_snake_case(input_name):- msg = f"Invalid input name syntax '{input_name}'. Must be in snake_case."- raise PipeBlueprintError(msg)- ConceptSpec.validate_concept_string_or_code(concept_string_or_code=concept_code)- return inputs-- @classmethod- def validate_pipe_code_syntax(cls, pipe_code: str) -> str:- # First, normalize Unicode to ASCII to prevent homograph attacks- normalized_pipe_code = normalize_to_ascii(pipe_code)-- if normalized_pipe_code != pipe_code:- log.warning(f"Pipe code '{pipe_code}' contained non-ASCII characters, normalized to '{normalized_pipe_code}'")-- if not is_snake_case(normalized_pipe_code):- msg = f"Invalid pipe code syntax '{normalized_pipe_code}'. Must be in snake_case."- raise PipeBlueprintError(msg)- return normalized_pipe_code-- def to_blueprint(self) -> PipeBlueprint:- converted_inputs: dict[str, str | InputRequirementBlueprint] | None- if not self.inputs:- converted_inputs = None- else:- converted_inputs = {}- for input_name, concept_code in self.inputs.items():- converted_inputs[input_name] = InputRequirementBlueprint(concept=concept_code)-- return PipeBlueprint(- description=self.description,- inputs=converted_inputs,- output=self.output,- type=self.type,- pipe_category=self.pipe_category,- )+ def validate_type(cls, type_value: str) -> AllowedPipeTypes:+ return AllowedPipeTypes(type_value)diff --git a/pipelex/builder/pipe/pipe_spec.py b/pipelex/builder/pipe/pipe_spec.pynew file mode 100644index 000000000..1f65409fe--- /dev/null+++ b/pipelex/builder/pipe/pipe_spec.py@@ -0,0 +1,131 @@+import re+from typing import Any++from pydantic import Field, field_validator++from pipelex import log+from pipelex.builder.concept.concept_spec import ConceptSpec+from pipelex.core.pipes.exceptions import PipeBlueprintError+from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories, AllowedPipeTypes, PipeBlueprint+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.tools.misc.string_utils import is_snake_case, normalize_to_ascii+++class PipeSpec(StructuredContent):+ """Spec defining a pipe: an executable component with a clear contract defined by its inputs and output.++ Pipes are the building blocks of a Pipelex pipeline. There are two categories:+ - Controllers: Manage execution flow (PipeSequence, PipeParallel, PipeCondition, PipeBatch)+ - Operators: Perform specific tasks (PipeLLM, PipeImgGen, PipeExtract, PipeFunc, PipeCompose)++ Multiplicity Notation:+ Both inputs and outputs use bracket notation to specify item counts:+ - No brackets: single item (default)+ - []: variable-length list (e.g., "Text[]")+ - [N]: exactly N items (e.g., "Image[3]" for 3 images)++ Examples:+ inputs = {"document": "PDF", "queries": "Text[]"} # single PDF, multiple texts+ output = "Article[]" # produces a list of articles+ output = "Image[5]" # produces exactly 5 images+ """++ pipe_code: str = Field(description="Unique pipe identifier. Must be snake_case.")+ type: Any = Field(+ description=(+ f"Pipe type. Validated at runtime, must be one of: {AllowedPipeTypes}. "+ "Examples: PipeLLM, PipeImgGen, PipeExtract, PipeSequence, PipeParallel."+ )+ )+ pipe_category: Any = Field(+ description=(f"Pipe category. Validated at runtime, must be one of: {AllowedPipeCategories}. Either 'PipeController' or 'PipeOperator'.")+ )+ description: str | None = Field(description="Natural language description of the pipe's purpose and functionality.")+ inputs: dict[str, str] = Field(+ description=(+ "Input specifications mapping variable names to concept codes with optional multiplicity. "+ "Keys: input names in snake_case. "+ "Values: ConceptCodes in PascalCase with optional brackets. "+ "Examples: 'Text' (single), 'Text[]' (variable list), 'Image[2]' (exactly 2 images), 'domain.Concept[]' (domain-qualified list)."+ )+ )+ output: str = Field(+ description=(+ "Output concept code in PascalCase with optional multiplicity brackets. "+ "Examples: 'Text' (single text), 'Article[]' (list of articles), 'Image[3]' (exactly 3 images). "+ "IMPORTANT: Always use PascalCase for the concept name."+ )+ )++ @field_validator("pipe_code", mode="before")+ @classmethod+ def validate_pipe_code(cls, value: str) -> str:+ return cls.validate_pipe_code_syntax(value)++ @field_validator("type", mode="after")+ @classmethod+ def validate_pipe_type(cls, value: Any) -> Any:+ if value not in AllowedPipeTypes.value_list():+ msg = f"Invalid pipe type '{value}'. Must be one of: {AllowedPipeTypes.value_list()}"+ raise PipeBlueprintError(msg)+ return value++ @field_validator("output", mode="after")+ @classmethod+ def validate_output(cls, output: str) -> str:+ # Extract concept without multiplicity for validation+ parse_result = parse_concept_with_multiplicity(output)+ ConceptSpec.validate_concept_string_or_code(concept_string_or_code=parse_result.concept)+ return output # Return original with brackets intact++ @field_validator("inputs", mode="after")+ @classmethod+ def validate_inputs(cls, inputs: dict[str, str] | None) -> dict[str, str] | None:+ if inputs is None:+ return None++ # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]+ multiplicity_pattern = r"^([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)?)(?:\[(\d*)\])?$"++ for input_name, concept_spec in inputs.items():+ if not is_snake_case(input_name):+ msg = f"Invalid input name syntax '{input_name}'. Must be in snake_case."+ raise PipeBlueprintError(msg)++ # Validate the concept spec format with optional multiplicity brackets+ match = re.match(multiplicity_pattern, concept_spec)+ if not match:+ msg = (+ f"Invalid input syntax for '{input_name}': '{concept_spec}'. "+ f"Expected format: 'ConceptName', 'ConceptName[]', or 'ConceptName[N]' where N is an integer."+ )+ raise PipeBlueprintError(msg)++ # Extract the concept part (without multiplicity) and validate it+ concept_string_or_code = match.group(1)+ ConceptSpec.validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)++ return inputs++ @classmethod+ def validate_pipe_code_syntax(cls, pipe_code: str) -> str:+ # First, normalize Unicode to ASCII to prevent homograph attacks+ normalized_pipe_code = normalize_to_ascii(pipe_code)++ if normalized_pipe_code != pipe_code:+ log.warning(f"Pipe code '{pipe_code}' contained non-ASCII characters, normalized to '{normalized_pipe_code}'")++ if not is_snake_case(normalized_pipe_code):+ msg = f"Invalid pipe code syntax '{normalized_pipe_code}'. Must be in snake_case."+ raise PipeBlueprintError(msg)+ return normalized_pipe_code++ def to_blueprint(self) -> PipeBlueprint:+ return PipeBlueprint(+ description=self.description,+ inputs=self.inputs if self.inputs else None,+ output=self.output,+ type=self.type,+ pipe_category=self.pipe_category,+ )diff --git a/pipelex/builder/pipe/sub_pipe_spec.py b/pipelex/builder/pipe/sub_pipe_spec.pyindex 978983a34..1fde3b673 100644--- a/pipelex/builder/pipe/sub_pipe_spec.py+++ b/pipelex/builder/pipe/sub_pipe_spec.py@@ -3,7 +3,6 @@ from pydantic import ConfigDict, model_validatorfrom pipelex.core.stuffs.structured_content import StructuredContentfrom pipelex.exceptions import PipeDefinitionErrorfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint-from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_listfrom pipelex.types import Self@@ -17,10 +16,6 @@ class SubPipeSpec(StructuredContent):Attributes:pipe_code: The pipe code to execute. Must reference an existing pipe in the pipeline.result: Name to assign to the pipe's output in the context.- nb_output: Fixed number of outputs to generate. Mutually exclusive with- multiple_output.- multiple_output: When true, allows LLM to determine the number of outputs.- Mutually exclusive with nb_output.batch_over: Name of the list in context to iterate over for batch processing.When false (default), no batching occurs. When specified as string,references a list in context. Requires batch_as when set.@@ -28,7 +23,6 @@ class SubPipeSpec(StructuredContent):Required when batch_over is specified.Validation Rules:- 1. nb_output and multiple_output are mutually exclusive.2. batch_over and batch_as must be specified together (both or neither).3. pipe must reference a valid pipe code.4. result, when specified, should follow naming conventions.@@ -39,18 +33,9 @@ class SubPipeSpec(StructuredContent):pipe_code: strresult: str- nb_output: int | None = None- multiple_output: bool | None = Nonebatch_over: str | None = Nonebatch_as: str | None = None- @model_validator(mode="after")- def validate_multiple_output(self) -> Self:- if has_more_than_one_among_attributes_from_list(self, attributes_list=["nb_output", "multiple_output"]):- msg = "PipeStepBlueprint should have no more than '1' of nb_output or multiple_output"- raise PipeDefinitionError(msg)- return self-@model_validator(mode="after")def validate_batch_params(self) -> Self:if self.batch_over and not self.batch_as:@@ -67,8 +52,6 @@ class SubPipeSpec(StructuredContent):return SubPipeBlueprint(pipe=self.pipe_code,result=self.result,- nb_output=self.nb_output,- multiple_output=self.multiple_output,batch_over=self.batch_over,batch_as=self.batch_as,)diff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.pyindex 3a992bbf6..cc7c0cddb 100644--- a/pipelex/cli/_cli.py+++ b/pipelex/cli/_cli.py@@ -8,7 +8,7 @@ from pipelex.cli.commands.init_cmd import init_appfrom pipelex.cli.commands.kit_cmd import kit_appfrom pipelex.cli.commands.run_cmd import run_cmdfrom pipelex.cli.commands.show_cmd import show_app-from pipelex.cli.commands.validate_cmd import validate_app+from pipelex.cli.commands.validate_cmd import validate_cmdclass PipelexCLI(TyperGroup):@@ -45,10 +45,8 @@ app.add_typer(kit_app, name="kit", help="Manage kit assets: agent rules, migratiapp.add_typer(build_app, name="build", help="Generate AI workflows from natural language requirements: pipelines in .plx format and python code to run them")-app.add_typer(- validate_app,- name="validate",- help="Validate pipes: static validation for syntax and dependencies, dry-run execution for logic and consistency",+app.command(name="validate", help="Validate pipes: static validation for syntax and dependencies, dry-run execution for logic and consistency")(+ validate_cmd)app.command(name="run", help="Run a pipe, optionally providing a specific bundle file (.plx)")(run_cmd)app.add_typer(show_app, name="show", help="Show configuration, pipes, and list AI models")diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.pyindex b8f7213a2..12720b340 100644--- a/pipelex/cli/commands/build_cmd.py+++ b/pipelex/cli/commands/build_cmd.py@@ -6,7 +6,7 @@ import clickimport typerfrom pipelex import pretty_print-from pipelex.builder.builder import PipelexBundleSpec, load_pipe_from_bundle+from pipelex.builder.builder import PipelexBundleSpec, load_and_validate_bundlefrom pipelex.builder.builder_errors import PipelexBundleErrorfrom pipelex.builder.builder_loop import BuilderLoopfrom pipelex.builder.runner_code import generate_runner_code@@ -23,8 +23,12 @@ build_app = typer.Typer(help="Build working pipelines from natural language requ"""Today's example:pipelex build pipe "Imagine a cute animal mascot for a startup based on its elevator pitch"+pipelex build pipe "Imagine a cute animal mascot for a startup based on its elevator pitch and some brand guidelines"+pipelex build pipe "Imagine a cute animal mascot for a startup based on its elevator pitch and some brand guidelines, \+ include 3 variants of the ideas and 2 variants of each prompt"pipelex build pipe "Given an expense report, apply company rules"pipelex build pipe "Take a CV in a PDF file, a Job offer text, and analyze if they match"+pipelex build pipe "Take a CV in a PDF file and a Job offer text, analyze if they match and generate 5 questions for the interview"pipelex build partial "Given an expense report, apply company rules" -o results/generated.jsonpipelex build flow "Given an expense report, apply company rules" -o results/flow.json@@ -170,8 +174,12 @@ def prepare_runner_cmd(if bundle_path:try:- main_pipe_code = await load_pipe_from_bundle(bundle_path)+ bundle_blueprint = await load_and_validate_bundle(bundle_path)if not pipe_code:+ main_pipe_code = bundle_blueprint.main_pipe+ if not main_pipe_code:+ typer.secho(f"Bundle '{bundle_path}' does not declare a main_pipe", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)pipe_code = main_pipe_codetyper.echo(f"Using main pipe '{pipe_code}' from bundle '{bundle_path}'")else:diff --git a/pipelex/cli/commands/run_cmd.py b/pipelex/cli/commands/run_cmd.pyindex 477ab712f..878b50bb6 100644--- a/pipelex/cli/commands/run_cmd.py+++ b/pipelex/cli/commands/run_cmd.py@@ -7,7 +7,7 @@ import clickimport typerfrom pipelex import log, pretty_print_md-from pipelex.builder.builder import load_pipe_from_bundle+from pipelex.builder.builder import load_and_validate_bundlefrom pipelex.builder.builder_errors import PipelexBundleErrorfrom pipelex.exceptions import PipeInputErrorfrom pipelex.pipelex import Pipelex@@ -108,8 +108,12 @@ def run_cmd(source_description: strif bundle_path:try:- main_pipe_code = await load_pipe_from_bundle(bundle_path)+ bundle_blueprint = await load_and_validate_bundle(bundle_path)if not pipe_code:+ main_pipe_code = bundle_blueprint.main_pipe+ if not main_pipe_code:+ typer.secho(f"Bundle '{bundle_path}' does not declare a main_pipe", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)pipe_code = main_pipe_codesource_description = f"bundle '{bundle_path}' • main pipe: '{pipe_code}'"else:@@ -164,7 +168,7 @@ def run_cmd(base_name=f"run_{pipe_code}",extension="json",)- working_memory_dict = pipe_output.working_memory.model_dump()+ working_memory_dict = pipe_output.working_memory.smart_dump()save_as_json_to_path(object_to_save=working_memory_dict, path=output_path)typer.secho(f"✅ Working memory saved to: {output_path}", fg=typer.colors.GREEN)diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.pyindex d96cce0dc..a786113c2 100644--- a/pipelex/cli/commands/validate_cmd.py+++ b/pipelex/cli/commands/validate_cmd.py@@ -3,10 +3,14 @@ from __future__ import annotationsimport asynciofrom typing import Annotated+import clickimport typerfrom pipelex import log-from pipelex.hub import get_pipeline_tracker, get_pipes, get_required_pipe+from pipelex.builder.builder import load_and_validate_bundle+from pipelex.builder.builder_errors import PipelexBundleError+from pipelex.exceptions import PipeInputError+from pipelex.hub import get_pipes, get_required_pipefrom pipelex.pipe_run.dry_run import dry_run_pipe, dry_run_pipesfrom pipelex.pipelex import Pipelex@@ -19,31 +23,110 @@ def do_validate_all_libraries_and_dry_run() -> None:log.info("Setup sequence passed OK, config and pipelines are validated.")-def do_dry_run_pipe(pipe_code: str) -> None:- """Dry run a single pipe."""- pipelex_instance = Pipelex.make()- pipelex_instance.validate_libraries()+def validate_cmd(+ target: Annotated[+ str | None,+ typer.Argument(help="Pipe code, bundle file path (auto-detected based on .plx extension), or 'all' to validate all pipes"),+ ] = None,+ pipe: Annotated[+ str | None,+ typer.Option("--pipe", help="Pipe code to validate (optional when using --bundle)"),+ ] = None,+ bundle: Annotated[+ str | None,+ typer.Option("--bundle", help="Bundle file path (.plx) - validates all pipes in the bundle"),+ ] = None,+) -> None:+ """Validate and dry run a pipe or a bundle or all pipes.- asyncio.run(- dry_run_pipe(- get_required_pipe(pipe_code=pipe_code),- raise_on_failure=True,- ),- )- get_pipeline_tracker().output_flowchart()+ Examples:+ pipelex validate my_pipe+ pipelex validate my_bundle.plx+ pipelex validate --bundle my_bundle.plx+ pipelex validate --bundle my_bundle.plx --pipe my_pipe+ pipelex validate all+ """+ # Check for "all" keyword+ if target == "all" and not pipe and not bundle:+ do_validate_all_libraries_and_dry_run()+ return+ # Validate mutual exclusivity+ provided_options = sum([target is not None, pipe is not None, bundle is not None])+ if provided_options == 0:+ ctx: click.Context = click.get_current_context()+ typer.echo(ctx.get_help())+ raise typer.Exit(0)-# Typer group for validation commands-validate_app = typer.Typer(no_args_is_help=True)+ # Let's analyze the options and determine what pipe code to use and if we need to load a bundle+ pipe_code: str | None = None+ bundle_path: str | None = None+ # Determine source:+ if target:+ if target.endswith(".plx"):+ bundle_path = target+ if bundle:+ typer.secho(+ "Failed to validate: cannot use option --bundle if you're already passing a bundle file (.plx) as positional argument",+ fg=typer.colors.RED,+ err=True,+ )+ raise typer.Exit(1)+ else:+ pipe_code = target+ if pipe:+ typer.secho(+ "Failed to validate: cannot use option --pipe if you're already passing a pipe code as positional argument",+ fg=typer.colors.RED,+ err=True,+ )+ raise typer.Exit(1)-@validate_app.command("pipe", help="Validate and dry run a single pipe")-def dry_run_pipe_cmd(- pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],-) -> None:- do_dry_run_pipe(pipe_code=pipe_code)+ if bundle:+ assert not bundle_path, "bundle_path should be None at this stage if --bundle is provided"+ bundle_path = bundle++ if pipe:+ assert not pipe_code, "pipe_code should be None at this stage if --pipe is provided"+ pipe_code = pipe++ if not pipe_code and not bundle_path:+ typer.secho("Failed to validate: no pipe code or bundle file specified", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)++ async def validate_pipe(pipe_code: str | None = None, bundle_path: str | None = None):+ # Initialize Pipelex+ pipelex_instance = Pipelex.make()+ if bundle_path:+ # When validating a bundle, load_pipe_from_bundle validates ALL pipes in the bundle+ try:+ await load_and_validate_bundle(bundle_path)+ if not pipe_code:+ typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}'", fg=typer.colors.GREEN)+ else:+ typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}' (including '{pipe_code}')", fg=typer.colors.GREEN)+ except FileNotFoundError as exc:+ typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)+ raise typer.Exit(1) from exc+ except PipelexBundleError as exc:+ typer.secho(f"Failed to validate bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)+ raise typer.Exit(1) from exc+ except PipeInputError as exc:+ typer.secho(f"Failed to validate bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)+ raise typer.Exit(1) from exc+ elif pipe_code:+ # Validate a single pipe by code+ typer.echo(f"Validating pipe '{pipe_code}'...")+ pipelex_instance.validate_libraries()+ await dry_run_pipe(+ get_required_pipe(pipe_code=pipe_code),+ raise_on_failure=True,+ )+ typer.secho(f"✅ Successfully validated pipe '{pipe_code}'", fg=typer.colors.GREEN)+ else:+ typer.secho("Failed to validate: no pipe code or bundle specified", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)-@validate_app.command("all", help="Validate and dry run all pipes")-def validate_all_cmd() -> None:- do_validate_all_libraries_and_dry_run()+ asyncio.run(validate_pipe(pipe_code=pipe_code, bundle_path=bundle_path))diff --git a/pipelex/client/client.py b/pipelex/client/client.pyindex dd507e092..2edceb323 100644--- a/pipelex/client/client.py+++ b/pipelex/client/client.py@@ -8,8 +8,8 @@ from pipelex.client.pipeline_response_factory import PipelineResponseFactoryfrom pipelex.client.protocol import PipelexProtocol, PipelineInputs, PipelineResponsefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.exceptions import ClientAuthenticationError-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicityfrom pipelex.system.environment import get_required_env@@ -84,7 +84,7 @@ class PipelexClient(PipelexProtocol):working_memory: WorkingMemory | None = None,inputs: PipelineInputs | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,) -> PipelineResponse:if working_memory and inputs:@@ -111,7 +111,7 @@ class PipelexClient(PipelexProtocol):working_memory: WorkingMemory | None = None,inputs: PipelineInputs | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,) -> PipelineResponse:if working_memory and inputs:diff --git a/pipelex/client/pipeline_request_factory.py b/pipelex/client/pipeline_request_factory.pyindex fab7073d1..301486f08 100644--- a/pipelex/client/pipeline_request_factory.py+++ b/pipelex/client/pipeline_request_factory.py@@ -3,7 +3,7 @@ from typing import Any, castfrom pipelex.client.api_serializer import ApiSerializerfrom pipelex.client.protocol import PipelineInputs, PipelineRequestfrom pipelex.core.memory.working_memory import WorkingMemory-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityclass PipelineRequestFactory:@@ -13,7 +13,7 @@ class PipelineRequestFactory:def make_from_working_memory(working_memory: WorkingMemory | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,plx_content: str | None = None,) -> PipelineRequest:diff --git a/pipelex/client/protocol.py b/pipelex/client/protocol.pyindex fbde8c6a0..429b03398 100644--- a/pipelex/client/protocol.py+++ b/pipelex/client/protocol.py@@ -7,9 +7,9 @@ from typing_extensions import Annotated, runtime_checkablefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.pipe_output import DictPipeOutput+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.stuff import DictStufffrom pipelex.core.stuffs.stuff_content import StuffContent-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicityfrom pipelex.types import StrEnum# StuffContentOrData represents all possible formats for pipeline inputs input:@@ -79,7 +79,7 @@ class PipelineRequest(BaseModel):inputs: Annotated[PipelineInputs | None, SkipValidation] = Noneoutput_name: str | None = None- output_multiplicity: PipeOutputMultiplicity | None = None+ output_multiplicity: VariableMultiplicity | None = Nonedynamic_output_concept_code: str | None = Noneplx_content: str | None = None@@ -130,7 +130,7 @@ class PipelexProtocol(Protocol):working_memory: WorkingMemory | None = None,inputs: PipelineInputs | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,) -> PipelineResponse:"""Execute a pipeline synchronously and wait for its completion.@@ -159,7 +159,7 @@ class PipelexProtocol(Protocol):working_memory: WorkingMemory | None = None,inputs: PipelineInputs | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,) -> PipelineResponse:"""Start a pipeline execution asynchronously without waiting for completion.diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.pyindex 90a4f555e..1f61aef26 100644--- a/pipelex/cogt/content_generation/content_generator.py+++ b/pipelex/cogt/content_generation/content_generator.py@@ -179,7 +179,7 @@ class ContentGenerator(ContentGeneratorProtocol):preliminary_text = await llm_gen_text(llm_assignment=llm_assignment_for_text)- log.dev(f"preliminary_text: {preliminary_text}")+ log.verbose(f"preliminary_text: {preliminary_text}")fup_llm_assignment = await workflow_arg.llm_assignment_factory_to_object.make_llm_assignment(preliminary_text=preliminary_text,@@ -214,7 +214,7 @@ class ContentGenerator(ContentGeneratorProtocol):nb_images=1,)generated_image = await img_gen_single_image(img_gen_assignment=img_gen_assignment)- log.dev(f"{self.__class__.__name__} generated image: {generated_image}")+ log.verbose(f"{self.__class__.__name__} generated image: {generated_image}")return generated_image@override@@ -238,7 +238,7 @@ class ContentGenerator(ContentGeneratorProtocol):nb_images=nb_images,)generated_image_list = await img_gen_image_list(img_gen_assignment=img_gen_assignment)- log.dev(f"{self.__class__.__name__} generated image list: {generated_image_list}")+ log.verbose(f"{self.__class__.__name__} generated image list: {generated_image_list}")return generated_image_list@overridediff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.pyindex f1ed386b7..8fa9673f5 100644--- a/pipelex/cogt/content_generation/content_generator_dry.py+++ b/pipelex/cogt/content_generation/content_generator_dry.py@@ -42,7 +42,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):llm_prompt_for_text: LLMPrompt,) -> str:func_name = "make_llm_text"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")prompt_truncated = llm_prompt_for_text.desc(truncate_text_length=self._text_gen_truncate_length)return f"DRY RUN: {func_name} • llm_setting={llm_setting_main.desc()} • prompt={prompt_truncated}"@@ -55,9 +55,6 @@ class ContentGeneratorDry(ContentGeneratorProtocol):llm_setting_for_object: LLMSetting,llm_prompt_for_object: LLMPrompt,) -> BaseModelTypeVar:- func_name = "make_object_direct"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")-class ObjectFactory(ModelFactory[object_class]): # type: ignore[valid-type]__model__ = object_class__check_model__ = True@@ -82,7 +79,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,) -> BaseModelTypeVar:func_name = "make_text_then_object"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")return await self.make_object_direct(job_metadata=job_metadata,@@ -102,7 +99,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):nb_items: int | None = None,) -> list[BaseModelTypeVar]:func_name = "make_object_list_direct"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")nb_list_items = nb_items or get_config().pipelex.dry_run_config.nb_list_itemsreturn [await self.make_object_direct(@@ -127,7 +124,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):nb_items: int | None = None,) -> list[BaseModelTypeVar]:func_name = "make_text_then_object_list"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")return await self.make_object_list_direct(job_metadata=job_metadata,object_class=object_class,@@ -147,7 +144,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):img_gen_job_config: ImgGenJobConfig | None = None,) -> GeneratedImage:func_name = "make_single_image"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")image_urls = get_config().pipelex.dry_run_config.image_urlsimage_url = image_urls[0]return GeneratedImage(@@ -168,7 +165,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):img_gen_job_config: ImgGenJobConfig | None = None,) -> list[GeneratedImage]:func_name = "make_image_list"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")image_urls = get_config().pipelex.dry_run_config.image_urlsreturn [GeneratedImage(@@ -189,7 +186,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):) -> str:check_jinja2_parsing(template_source=template, template_category=template_category or TemplateCategory.BASIC)func_name = "make_templated_text"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")jinja2_truncated = template[: self._text_gen_truncate_length]return (f"DRY RUN: {func_name} • context={context} • "@@ -206,7 +203,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):extract_job_config: ExtractJobConfig | None = None,) -> ExtractOutput:func_name = "make_extract_pages"- log.dev(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")if extract_input.image_uri:image_as_page = Page(text="DRY RUN: OCR text",diff --git a/pipelex/cogt/extract/extract_output.py b/pipelex/cogt/extract/extract_output.pyindex 11daeafa6..732b3475a 100644--- a/pipelex/cogt/extract/extract_output.py+++ b/pipelex/cogt/extract/extract_output.py@@ -13,7 +13,7 @@ class ExtractedImage(CustomBaseModel):def save_to_directory(self, directory: str):ensure_directory_exists(directory)- log.debug(f"Saving image to directory: {directory}")+ log.verbose(f"Saving image to directory: {directory}")if base_64 := self.base_64:filename = self.image_idfile_path = f"{directory}/{filename}"@@ -34,7 +34,7 @@ class Page(CustomBaseModel):def save_to_directory(self, directory: str, page_text_file_name: str):ensure_directory_exists(directory)- log.debug(f"Saving page to directory: {directory}")+ log.verbose(f"Saving page to directory: {directory}")if text := self.text:filename = page_text_file_namesave_text_to_path(text=text, path=f"{directory}/{filename}")diff --git a/pipelex/cogt/extract/extract_worker_abstract.py b/pipelex/cogt/extract/extract_worker_abstract.pyindex b9aafb3ed..028daafdb 100644--- a/pipelex/cogt/extract/extract_worker_abstract.py+++ b/pipelex/cogt/extract/extract_worker_abstract.py@@ -40,7 +40,7 @@ class ExtractWorkerAbstract(InferenceWorkerAbstract):self,extract_job: ExtractJob,) -> ExtractOutput:- log.debug(f"Extract Worker extract_pages:\n{self.inference_model.desc}")+ log.verbose(f"Extract Worker extract_pages:\n{self.inference_model.desc}")# Verify that the job is validextract_job.validate_before_execution()diff --git a/pipelex/cogt/img_gen/img_gen_worker_abstract.py b/pipelex/cogt/img_gen/img_gen_worker_abstract.pyindex d55630df5..6807f3521 100644--- a/pipelex/cogt/img_gen/img_gen_worker_abstract.py+++ b/pipelex/cogt/img_gen/img_gen_worker_abstract.py@@ -37,7 +37,7 @@ class ImgGenWorkerAbstract(InferenceWorkerAbstract):self,img_gen_job: ImgGenJob,) -> GeneratedImage:- log.debug(f"Image gen worker gen_image using {self.desc}")+ log.verbose(f"Image gen worker gen_image using {self.desc}")# Verify that the job is validimg_gen_job.validate_before_execution()@@ -73,7 +73,7 @@ class ImgGenWorkerAbstract(InferenceWorkerAbstract):img_gen_job: ImgGenJob,nb_images: int,) -> list[GeneratedImage]:- log.debug(f"Image gen worker gen_image_list using {self.desc}")+ log.verbose(f"Image gen worker gen_image_list using {self.desc}")# Verify that the job is validimg_gen_job.validate_before_execution()diff --git a/pipelex/cogt/inference/inference_manager.py b/pipelex/cogt/inference/inference_manager.pyindex 89c98136c..c2cf6a9c0 100644--- a/pipelex/cogt/inference/inference_manager.py+++ b/pipelex/cogt/inference/inference_manager.py@@ -40,18 +40,18 @@ class InferenceManager(InferenceManagerProtocol):log.verbose("InferenceManager teardown done")def print_workers(self):- log.debug("LLM Workers:")+ log.verbose("LLM Workers:")for handle, llm_worker in self.llm_workers.items():- log.debug(f" {handle}:")- log.debug(llm_worker.desc)- log.debug("Image Workers:")+ log.verbose(f" {handle}:")+ log.verbose(llm_worker.desc)+ log.verbose("Image Workers:")for handle, img_gen_worker_async in self.img_gen_workers.items():- log.debug(f" {handle}:")- log.debug(img_gen_worker_async.desc)- log.debug("OCR Workers:")+ log.verbose(f" {handle}:")+ log.verbose(img_gen_worker_async.desc)+ log.verbose("OCR Workers:")for handle, extract_worker_async in self.extract_workers.items():- log.debug(f" {handle}:")- log.debug(extract_worker_async.desc)+ log.verbose(f" {handle}:")+ log.verbose(extract_worker_async.desc)##################################################################################################### Setup LLM Workersdiff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.pyindex 4b77ad942..05b2b47ab 100644--- a/pipelex/cogt/llm/llm_worker_abstract.py+++ b/pipelex/cogt/llm/llm_worker_abstract.py@@ -70,7 +70,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):self,llm_job: LLMJob,) -> str:- log.debug("LLM Worker gen_text")+ log.verbose("LLM Worker gen_text")log.verbose(llm_job.llm_prompt.desc(), title="llm_prompt")# metadata@@ -96,7 +96,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):llm_job: LLMJob,schema: type[BaseModelTypeVar],) -> BaseModelTypeVar:- log.debug(f"LLM Worker gen_object using {self.desc}")+ log.verbose(f"LLM Worker gen_object using {self.desc}")log.verbose(llm_job.llm_prompt.desc(), title="llm_prompt")# metadatadiff --git a/pipelex/cogt/model_routing/routing_profile_library.py b/pipelex/cogt/model_routing/routing_profile_library.pyindex c2b27d257..2186796a5 100644--- a/pipelex/cogt/model_routing/routing_profile_library.py+++ b/pipelex/cogt/model_routing/routing_profile_library.py@@ -68,8 +68,8 @@ class RoutingProfileLibrary(RootModel[RoutingProfileLibraryRoot]):)self._active_config = catalog_blueprint.active- log.debug(f"Loaded routing profile library with active profile: '{self._active_config}'")- log.debug(f"Available profiles: {list(self.root.keys())}")+ log.verbose(f"Loaded routing profile library with active profile: '{self._active_config}'")+ log.verbose(f"Available profiles: {list(self.root.keys())}")def get_backend_match_for_model_from_active_routing_profile(self, model_name: str) -> BackendMatchForModel | None:"""Get the backend name for a given model.diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.pyindex 6a068d439..8eba68656 100644--- a/pipelex/cogt/models/model_deck.py+++ b/pipelex/cogt/models/model_deck.py@@ -193,7 +193,7 @@ class ModelDeck(ConfigModel):if inference_model := self.inference_models.get(model_handle):return inference_modelif redirection := self.aliases.get(model_handle):- log.debug(f"Redirection for '{model_handle}': {redirection}")+ log.verbose(f"Redirection for '{model_handle}': {redirection}")if isinstance(redirection, str):alias_list = [redirection]else:@@ -213,5 +213,5 @@ class ModelDeck(ConfigModel):msg = f"Model handle '{model_handle}' not found in deck"raise ModelNotFoundError(msg)if model_handle not in self.inference_models:- log.dev(f"Model handle '{model_handle}' is an alias which resolves to '{inference_model.name}'")+ log.verbose(f"Model handle '{model_handle}' is an alias which resolves to '{inference_model.name}'")return inference_modeldiff --git a/pipelex/core/concepts/concept.py b/pipelex/core/concepts/concept.pyindex 3b9889e0b..b3e968752 100644--- a/pipelex/core/concepts/concept.py+++ b/pipelex/core/concepts/concept.py@@ -65,6 +65,8 @@ class Concept(BaseModel):@classmethoddef are_concept_compatible(cls, concept_1: "Concept", concept_2: "Concept", strict: bool = False) -> bool:+ if NativeConceptCode.is_dynamic_concept(concept_code=concept_2.code):+ return Trueif concept_1.concept_string == concept_2.concept_string:return Trueif concept_1.structure_class_name == concept_2.structure_class_name:diff --git a/pipelex/core/concepts/concept_native.py b/pipelex/core/concepts/concept_native.pyindex df4c44b7d..b099db8ce 100644--- a/pipelex/core/concepts/concept_native.py+++ b/pipelex/core/concepts/concept_native.py@@ -18,6 +18,10 @@ class NativeConceptCode(StrEnum):PAGE = "Page"ANYTHING = "Anything"+ @property+ def as_output_multiple_indeterminate(self) -> str:+ return f"{self.value}[]"+@propertydef concept_string(self) -> str:return f"{SpecialDomain.NATIVE}.{self.value}"@@ -49,6 +53,29 @@ class NativeConceptCode(StrEnum):):return False+ @classmethod+ def is_dynamic_concept(cls, concept_code: str) -> bool:+ try:+ enum_value = NativeConceptCode(concept_code)+ except ValueError:+ return False++ match enum_value:+ case (+ NativeConceptCode.TEXT+ | NativeConceptCode.IMAGE+ | NativeConceptCode.PDF+ | NativeConceptCode.TEXT_AND_IMAGES+ | NativeConceptCode.NUMBER+ | NativeConceptCode.LLM_PROMPT+ | NativeConceptCode.IMG_GEN_PROMPT+ | NativeConceptCode.PAGE+ | NativeConceptCode.ANYTHING+ ):+ return False+ case NativeConceptCode.DYNAMIC:+ return True+@classmethoddef values_list(cls) -> list["NativeConceptCode"]:return list(cls)diff --git a/pipelex/core/interpreter.py b/pipelex/core/interpreter.pyindex 725349d58..31cb0116b 100644--- a/pipelex/core/interpreter.py+++ b/pipelex/core/interpreter.py@@ -79,3 +79,14 @@ class PipelexInterpreter(BaseModel):except TomlError as exc:raise PLXDecodeError(message=exc.message, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno) from excreturn PipelexBundleBlueprint.model_validate(blueprint_dict)++ @classmethod+ def load_bundle_blueprint(cls, bundle_path: str) -> PipelexBundleBlueprint:+ """Load a bundle file and return its blueprint."""+ bundle_path_obj = Path(bundle_path)+ if not bundle_path_obj.exists():+ msg = f"Bundle file not found: {bundle_path}"+ raise FileNotFoundError(msg)++ interpreter = cls(file_path=bundle_path_obj)+ return interpreter.make_pipelex_bundle_blueprint()diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.pyindex ad9294611..897ce02a0 100644--- a/pipelex/core/memory/working_memory.py+++ b/pipelex/core/memory/working_memory.py@@ -207,7 +207,7 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):return artefact_dict@override- def get_typed_object_or_attribute(self, name: str, wanted_type: type[Any] | None = None) -> Any:+ def get_typed_object_or_attribute(self, name: str, wanted_type: type[Any] | None = None, accept_list: bool = False) -> Any:# TODO: Refactor this method. In the python paradigm, we should not have those ".", but arrays with field names.if "." in name:parts = name.split(".", 1) # Split only at the first dot@@ -215,7 +215,43 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):attr_path_str = parts[1] # Keep the rest as a dot-separated stringbase_stuff = self.get_stuff(base_name)-+ if isinstance(base_stuff.content, ListContent):+ if not accept_list:+ raise WorkingMemoryTypeError(+ variable_name=name,+ message=f"Content of '{base_name}' is ListContent, but accept_list is False",+ )+ the_items: list[Any] = []+ list_content: ListContent[Any] = base_stuff.content # pyright: ignore[reportUnknownVariableType, reportUnknownMemberType]+ for item in list_content.items:+ item_content = attrgetter(attr_path_str)(item)+ # check type+ if isinstance(item_content, list):+ for item_content_item in item_content: # pyright: ignore[reportUnknownVariableType]+ if item_content_item is None:+ continue+ if wanted_type and not isinstance(item_content_item, wanted_type):+ raise WorkingMemoryTypeError(+ variable_name=name,+ message=(+ f"Item of '{base_name}' is of type '{type(item_content_item).__name__}', " # pyright: ignore[reportUnknownArgumentType]+ f"it should be '{wanted_type.__name__}'"+ ),+ )+ the_items.append(item_content_item)+ else:+ if item_content is None:+ continue+ if wanted_type and not isinstance(item_content, wanted_type):+ raise WorkingMemoryTypeError(+ variable_name=name,+ message=(+ f"Item of item_content from '{base_name} [item] {attr_path_str}' is of type '{type(item_content).__name__}', "+ f"it should be '{wanted_type.__name__}'"+ ),+ )+ the_items.append(item_content)+ return the_itemstry:stuff_content = attrgetter(attr_path_str)(base_stuff.content)except AttributeError as exc:@@ -339,3 +375,11 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):def main_stuff_as_mermaid(self) -> MermaidContent:"""Get main stuff content as MermaidContent if applicable."""return self.get_stuff_as_mermaid(name=MAIN_STUFF_NAME)++ ################################################################################################+ # Serialization+ ################################################################################################++ def smart_dump(self) -> dict[str, Any]:+ """Serialize the working memory as a dictionary."""+ return self.model_dump(serialize_as_any=True)diff --git a/pipelex/core/pipes/input_requirement_blueprint.py b/pipelex/core/pipes/input_requirement_blueprint.pydeleted file mode 100644index 7fbf1e237..000000000--- a/pipelex/core/pipes/input_requirement_blueprint.py+++ /dev/null@@ -1,15 +0,0 @@-from pydantic import BaseModel, field_validator--from pipelex.core.concepts.concept_blueprint import ConceptBlueprint-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity---class InputRequirementBlueprint(BaseModel):- concept: str- multiplicity: PipeOutputMultiplicity | None = None-- @field_validator("concept", mode="before")- @classmethod- def validate_concept_string(cls, concept_string: str) -> str:- ConceptBlueprint.validate_concept_string_or_code(concept_string_or_code=concept_string)- return concept_stringdiff --git a/pipelex/core/pipes/input_requirements.py b/pipelex/core/pipes/input_requirements.pyindex fd202969f..044fc0d8f 100644--- a/pipelex/core/pipes/input_requirements.py+++ b/pipelex/core/pipes/input_requirements.py@@ -4,14 +4,14 @@ from pydantic import BaseModel, Field, RootModel, field_validatorfrom pipelex import logfrom pipelex.core.concepts.concept import Concept+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.exceptions import PipeInputNotFoundError-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicityclass InputRequirement(BaseModel):concept: Concept- multiplicity: PipeOutputMultiplicity | None = None+ multiplicity: VariableMultiplicity | None = Noneclass NamedInputRequirement(InputRequirement):@@ -77,7 +77,7 @@ class InputRequirements(RootModel[InputRequirementsRoot]):raise PipeInputNotFoundError(msg)return requirement- def add_requirement(self, variable_name: str, concept: Concept, multiplicity: PipeOutputMultiplicity | None = None):+ def add_requirement(self, variable_name: str, concept: Concept, multiplicity: VariableMultiplicity | None = None):self.root[variable_name] = InputRequirement(concept=concept, multiplicity=multiplicity)@propertydiff --git a/pipelex/core/pipes/input_requirements_factory.py b/pipelex/core/pipes/input_requirements_factory.pyindex 1d275e60b..b8bc7dfb6 100644--- a/pipelex/core/pipes/input_requirements_factory.py+++ b/pipelex/core/pipes/input_requirements_factory.py@@ -1,9 +1,19 @@+import re+from typing import TYPE_CHECKING+from pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.input_requirements import InputRequirement, InputRequirements+from pipelex.exceptions import PipelexExceptionfrom pipelex.hub import get_required_concept+if TYPE_CHECKING:+ from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity+++class InputRequirementsFactorySyntaxError(PipelexException):+ pass+class InputRequirementsFactory:@classmethod@@ -14,24 +24,74 @@ class InputRequirementsFactory:def make_from_blueprint(cls,domain: str,- blueprint: dict[str, str | InputRequirementBlueprint],+ blueprint: dict[str, str],concept_codes_from_the_same_domain: list[str] | None = None,) -> InputRequirements:input_requirements_dict: dict[str, InputRequirement] = {}- for var_name, input_requirement_blueprint in blueprint.items():- if isinstance(input_requirement_blueprint, str):- input_requirement_blueprint = InputRequirementBlueprint(concept=input_requirement_blueprint)-- concept_string_or_code = input_requirement_blueprint.concept- ConceptBlueprint.validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)- concept_string_with_domain = ConceptFactory.make_concept_string_with_domain_from_concept_string_or_code(+ for var_name, requirement_str in blueprint.items():+ input_requirement = InputRequirementsFactory.make_from_string(domain=domain,- concept_sring_or_code=concept_string_or_code,+ requirement_str=requirement_str,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)-- input_requirements_dict[var_name] = InputRequirement(- concept=get_required_concept(concept_string=concept_string_with_domain),- multiplicity=input_requirement_blueprint.multiplicity,- )+ input_requirements_dict[var_name] = input_requirementreturn InputRequirements(root=input_requirements_dict)++ @classmethod+ def make_from_string(+ cls,+ domain: str,+ requirement_str: str,+ concept_codes_from_the_same_domain: list[str] | None = None,+ ) -> InputRequirement:+ """Parse an input requirement string and return an InputRequirement.++ Interprets multiplicity from a string in the form:+ - "domain.ConceptCode[5]" -> multiplicity = 5 (int)+ - "domain.ConceptCode[]" -> multiplicity = True+ - "domain.ConceptCode" -> multiplicity = None (single item, default)+ - "ConceptCode[5]" -> multiplicity = 5 (resolved with domain)++ Args:+ domain: The domain to use for resolving concept codes without domain prefix+ requirement_str: String in the format "domain.ConceptCode" or "ConceptCode" with optional "[multiplicity]"+ concept_codes_from_the_same_domain: List of concept codes from the same domain for resolution++ Returns:+ InputRequirement with the parsed concept and multiplicity++ Raises:+ InputRequirementsFactorySyntaxError: If the requirement string format is invalid+ """+ # Pattern to match concept string and optional multiplicity brackets+ # Group 1: concept string (everything before brackets)+ # Group 2: content inside brackets (empty string for [], digits for [5])+ pattern = r"^(.+?)(?:\[(\d*)\])?$"+ match = re.match(pattern, requirement_str)++ if not match:+ msg = f"Invalid input requirement string: {requirement_str}"+ raise InputRequirementsFactorySyntaxError(msg)++ concept_string_or_code = match.group(1)+ multiplicity_str = match.group(2)++ # Validate and resolve concept string with domain+ ConceptBlueprint.validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)+ concept_string_with_domain = ConceptFactory.make_concept_string_with_domain_from_concept_string_or_code(+ domain=domain,+ concept_sring_or_code=concept_string_or_code,+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,+ )++ # Determine multiplicity+ multiplicity: VariableMultiplicity | None = None+ if multiplicity_str is not None: # Brackets were present+ if multiplicity_str == "": # Empty brackets []+ multiplicity = True+ else: # Number in brackets [5]+ multiplicity = int(multiplicity_str)+ # else: No brackets, multiplicity stays None++ concept = get_required_concept(concept_string=concept_string_with_domain)+ return InputRequirement(concept=concept, multiplicity=multiplicity)diff --git a/pipelex/core/pipes/pipe_blueprint.py b/pipelex/core/pipes/pipe_blueprint.pyindex 68652ec6c..1028f3237 100644--- a/pipelex/core/pipes/pipe_blueprint.py+++ b/pipelex/core/pipes/pipe_blueprint.py@@ -4,7 +4,7 @@ from pydantic import BaseModel, ConfigDict, Field, field_validatorfrom pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.pipes.exceptions import PipeBlueprintError-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.tools.misc.string_utils import is_snake_casefrom pipelex.types import StrEnum@@ -80,7 +80,7 @@ class PipeBlueprint(BaseModel):pipe_category: Any = Field(exclude=True) # Technical field for Union discrimination, not user-facingtype: Any # TODO: Find a better way to handle this.description: str | None = None- inputs: dict[str, str | InputRequirementBlueprint] | None = None+ inputs: dict[str, str] | None = Noneoutput: str@property@@ -108,8 +108,8 @@ class PipeBlueprint(BaseModel):return None@field_validator("type", mode="after")- @staticmethod- def validate_pipe_type(value: Any) -> Any:+ @classmethod+ def validate_pipe_type(cls, value: Any) -> Any:"""Validate that the pipe type is one of the allowed values."""if value not in AllowedPipeTypes.value_list():msg = f"Invalid pipe type '{value}'. Must be one of: {AllowedPipeTypes.value_list()}"@@ -117,8 +117,8 @@ class PipeBlueprint(BaseModel):return value@field_validator("pipe_category", mode="after")- @staticmethod- def validate_pipe_category(value: Any) -> Any:+ @classmethod+ def validate_pipe_category(cls, value: Any) -> Any:"""Validate that the pipe category is one of the allowed values."""if value not in AllowedPipeCategories.value_list():msg = f"Invalid pipe category '{value}'. Must be one of: {AllowedPipeCategories.value_list()}"@@ -126,10 +126,12 @@ class PipeBlueprint(BaseModel):return value@field_validator("output", mode="before")- @staticmethod- def validate_concept_string_or_code(output: str) -> str:- ConceptBlueprint.validate_concept_string_or_code(concept_string_or_code=output)- return output+ @classmethod+ def validate_output(cls, output: str) -> str:+ # Strip multiplicity brackets before validating+ output_parse_result = parse_concept_with_multiplicity(output)+ ConceptBlueprint.validate_concept_string_or_code(concept_string_or_code=output_parse_result.concept)+ return output # Return with brackets intact@classmethoddef validate_pipe_code_syntax(cls, pipe_code: str) -> str:diff --git a/pipelex/core/pipes/variable_multiplicity.py b/pipelex/core/pipes/variable_multiplicity.pynew file mode 100644index 000000000..5df9d2693--- /dev/null+++ b/pipelex/core/pipes/variable_multiplicity.py@@ -0,0 +1,105 @@+from __future__ import annotations++import re++from pydantic import BaseModel, Field++VariableMultiplicity = bool | int+++class VariableMultiplicityResolution(BaseModel):+ """Result of resolving output multiplicity settings between base and override values."""++ resolved_multiplicity: VariableMultiplicity | None = Field(description="The final multiplicity value to use after resolution")+ is_multiple_outputs_enabled: bool = Field(description="Whether multiple values should be expected/generated")+ specific_output_count: int | None = Field(default=None, description="Exact number of items to expect/generate, if specified")+++def make_variable_multiplicity(nb_items: int | None, multiple_items: bool | None) -> VariableMultiplicity | None:+ """This function takes two mutually exclusive parameters that control how many items a variable can have+ and converts them into a single VariableMultiplicity type.++ Args:+ nb_items: Specific number of outputs to generate. If provided and truthy,+ takes precedence over multiple_output.+ multiple_items: Boolean flag indicating whether to generate multiple outputs.+ If True, lets the LLM decide how many outputs to generate.++ Examples:+ >>> make_variable_multiplicity(nb_items=3, multiple_items=None)+ 3+ >>> make_variable_multiplicity(nb_items=None, multiple_items=True)+ True+ >>> make_variable_multiplicity(nb_items=None, multiple_items=False)+ None+ >>> make_variable_multiplicity(nb_items=0, multiple_items=True)+ True++ """+ variable_multiplicity: VariableMultiplicity | None+ if nb_items:+ variable_multiplicity = nb_items+ elif multiple_items:+ variable_multiplicity = True+ else:+ variable_multiplicity = None+ return variable_multiplicity+++class MultiplicityParseResult:+ """Result of parsing a concept string with multiplicity notation."""++ def __init__(self, concept: str, multiplicity: int | bool | None):+ self.concept: str = concept+ self.multiplicity: int | bool | None = multiplicity+++def parse_concept_with_multiplicity(concept_spec: str) -> MultiplicityParseResult:+ """Parse a concept specification string to extract concept and multiplicity.++ Supported formats:+ - "ConceptName" -> (ConceptName, None)+ - "ConceptName[]" -> (ConceptName, True)+ - "ConceptName[5]" -> (ConceptName, 5)+ - "domain.ConceptName" -> (domain.ConceptName, None)+ - "domain.ConceptName[]" -> (domain.ConceptName, True)+ - "domain.ConceptName[5]" -> (domain.ConceptName, 5)++ Args:+ concept_spec: Concept specification string with optional multiplicity brackets++ Returns:+ MultiplicityParseResult with concept (without brackets) and multiplicity value++ Raises:+ ValueError: If the concept specification has invalid syntax+ """+ # Use strict pattern to validate identifier syntax+ # Concept must start with letter/underscore, optional domain prefix, optional brackets+ pattern = r"^([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)?)(?:\[(\d*)\])?$"+ match = re.match(pattern, concept_spec)++ if not match:+ msg = (+ f"Invalid concept specification syntax: '{concept_spec}'. "+ f"Expected format: 'ConceptName', 'ConceptName[]', 'ConceptName[N]', "+ f"'domain.ConceptName', 'domain.ConceptName[]', or 'domain.ConceptName[N]' "+ f"where concept and domain names must start with a letter or underscore."+ )+ raise ValueError(msg)++ concept = match.group(1)+ bracket_content = match.group(2)++ multiplicity: int | bool | None+ if bracket_content is None:+ # No brackets - single item+ multiplicity = None+ elif bracket_content == "":+ # Empty brackets [] - variable list+ multiplicity = True+ else:+ # Number in brackets [N] - fixed count+ multiplicity = int(bracket_content)++ return MultiplicityParseResult(concept=concept, multiplicity=multiplicity)diff --git a/pipelex/core/stuffs/stuff.py b/pipelex/core/stuffs/stuff.pyindex f32ccaa71..3139e58ce 100644--- a/pipelex/core/stuffs/stuff.py+++ b/pipelex/core/stuffs/stuff.py@@ -15,7 +15,7 @@ from pipelex.core.stuffs.stuff_artefact import StuffArtefactfrom pipelex.core.stuffs.stuff_content import StuffContent, StuffContentTypefrom pipelex.core.stuffs.text_and_images_content import TextAndImagesContentfrom pipelex.core.stuffs.text_content import TextContent-from pipelex.exceptions import StuffContentValidationError, StuffError+from pipelex.exceptions import StuffArtefactReservedFieldError, StuffContentTypeError, StuffContentValidationErrorfrom pipelex.tools.misc.string_utils import pascal_case_to_snake_casefrom pipelex.tools.typing.pydantic_utils import CustomBaseModel, format_pydantic_validation_error@@ -39,7 +39,7 @@ class Stuff(CustomBaseModel):msg = f"""Cannot create stuff artefact for stuff '{stuff_name}' of concept '{self.concept.code}' because reserved field '{key}'in the structured output '{self.content.__class__.__name__}' already exists in the stuff content.Forbidden fields are: 'stuff_name', 'content_class', 'concept_code', 'stuff_code', 'content'."""- raise StuffError(msg)+ raise StuffArtefactReservedFieldError(message=msg)artefact_dict[key] = valueset_artefact_field("stuff_name", self.stuff_name)@@ -114,7 +114,7 @@ Forbidden fields are: 'stuff_name', 'content_class', 'concept_code', 'stuff_codeif type(content).__name__ == content_type.__name__:content_dict = content.smart_dump()validated_content = content_type.model_validate(content_dict)- log.debug(f"Model validation passed: converted {type(content).__name__} to {content_type.__name__}")+ log.verbose(f"Model validation passed: converted {type(content).__name__} to {content_type.__name__}")return validated_contentexcept ValidationError as exc:formatted_error = format_pydantic_validation_error(exc)@@ -124,8 +124,9 @@ Forbidden fields are: 'stuff_name', 'content_class', 'concept_code', 'stuff_codevalidation_error=formatted_error,) from exc- msg = f"Content is of type '{type(content)}', instead of the expected '{content_type}'"- raise TypeError(msg)+ actual_type = type(content)+ msg = f"Content is of type '{actual_type}', instead of the expected '{content_type}'"+ raise StuffContentTypeError(message=msg, expected_type=content_type.__name__, actual_type=actual_type.__name__)def as_list_content(self) -> ListContent: # pyright: ignore[reportMissingTypeArgument, reportUnknownParameterType]"""Get content as ListContent with items of any type."""@@ -155,7 +156,7 @@ Forbidden fields are: 'stuff_name', 'content_class', 'concept_code', 'stuff_code@propertydef as_text(self) -> TextContent:"""Get content as TextContent if applicable."""- return self.content_as(TextContent)+ return self.content_as(content_type=TextContent)@propertydef as_str(self) -> str:@@ -165,27 +166,27 @@ Forbidden fields are: 'stuff_name', 'content_class', 'concept_code', 'stuff_code@propertydef as_image(self) -> ImageContent:"""Get content as ImageContent if applicable."""- return self.content_as(ImageContent)+ return self.content_as(content_type=ImageContent)@propertydef as_pdf(self) -> PDFContent:"""Get content as PDFContent if applicable."""- return self.content_as(PDFContent)+ return self.content_as(content_type=PDFContent)@propertydef as_text_and_image(self) -> TextAndImagesContent:"""Get content as TextAndImageContent if applicable."""- return self.content_as(TextAndImagesContent)+ return self.content_as(content_type=TextAndImagesContent)@propertydef as_number(self) -> NumberContent:"""Get content as NumberContent if applicable."""- return self.content_as(NumberContent)+ return self.content_as(content_type=NumberContent)@propertydef as_html(self) -> HtmlContent:"""Get content as HtmlContent if applicable."""- return self.content_as(HtmlContent)+ return self.content_as(content_type=HtmlContent)@propertydef as_mermaid(self) -> MermaidContent:diff --git a/pipelex/core/validation.py b/pipelex/core/validation.pyindex b43cf4a35..8070c9f38 100644--- a/pipelex/core/validation.py+++ b/pipelex/core/validation.py@@ -13,7 +13,7 @@ def report_validation_error(category: str, validation_error: ValidationError) ->migration_reports: list[str] = []# Build field-to-renamings mapping for missing fields- log.debug(validation_error_analysis.missing_fields, title="Missing fields")+ log.verbose(validation_error_analysis.missing_fields, title="Missing fields")missing_field_renamings: dict[tuple[tuple[str, str], ...], list[str]] = {}for missing_field in validation_error_analysis.missing_fields:text = missing_field.split(".")[-1]@@ -25,7 +25,7 @@ def report_validation_error(category: str, validation_error: ValidationError) ->missing_field_renamings[renamings_key].append(missing_field)# Build field-to-renamings mapping for extra fields- log.debug(validation_error_analysis.extra_fields, title="Extra fields")+ log.verbose(validation_error_analysis.extra_fields, title="Extra fields")extra_field_renamings: dict[tuple[tuple[str, str], ...], list[str]] = {}for extra_field in validation_error_analysis.extra_fields:# Extract field path before the colon (extra fields include ": value")diff --git a/pipelex/exceptions.py b/pipelex/exceptions.pyindex e80470f33..7f66817e6 100644--- a/pipelex/exceptions.py+++ b/pipelex/exceptions.py@@ -208,6 +208,14 @@ class PipeDefinitionError(PipelexException):return details+class PipeInputError(PipelexException):+ def __init__(self, message: str, pipe_code: str, variable_name: str, concept_code: str | None = None):+ self.pipe_code = pipe_code+ self.variable_name = variable_name+ self.concept_code = concept_code+ super().__init__(message)++class DomainLoadingError(LibraryLoadingError):def __init__(self, message: str, domain_code: str, description: str, source: str | None = None):self.domain_code = domain_code@@ -244,6 +252,13 @@ class StuffError(PipelexException):pass+class StuffContentTypeError(StuffError):+ def __init__(self, message: str, expected_type: str, actual_type: str):+ self.expected_type = expected_type+ self.actual_type = actual_type+ super().__init__(message)++class StuffContentValidationError(StuffError):"""Raised when content validation fails during type conversion."""@@ -288,6 +303,17 @@ class StructureClassError(PipelexException):pass+class PipeControllerError(PipelexException):+ pass+++class PipeControllerOutputConceptMismatchError(PipeControllerError):+ def __init__(self, message: str, tested_concept: str, wanted_concept: str):+ self.tested_concept = tested_concept+ self.wanted_concept = wanted_concept+ super().__init__(message)++class PipeRunParamsError(PipelexException):pass@@ -316,11 +342,11 @@ class JobHistoryError(PipelexException):pass-class PipeInputError(PipelexException):+class StuffArtefactError(PipelexException):pass-class StuffArtefactError(PipelexException):+class StuffArtefactReservedFieldError(StuffArtefactError):passdiff --git a/pipelex/kit/agent_rules/llms.md b/pipelex/kit/agent_rules/llms.mdindex f79c234bf..1c3b01521 100644--- a/pipelex/kit/agent_rules/llms.md+++ b/pipelex/kit/agent_rules/llms.md@@ -48,7 +48,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.mdindex bbbcd80ae..06db00371 100644--- a/pipelex/kit/agent_rules/python_standards.md+++ b/pipelex/kit/agent_rules/python_standards.md@@ -34,7 +34,7 @@ This document outlines the core coding standards, best practices, and quality co- Both have a title arg which is handy when logging/printing objects:```python- log.debug("Hello, world!", title="Your first Pipelex log")+ log.verbose("Hello, world!", title="Your first Pipelex log")pretty_print(output_object, title="Your first Pipelex output")```- Both handle formatting json using Rich, pretty_print makes it prettier.diff --git a/pipelex/kit/agent_rules/write_pipelex.md b/pipelex/kit/agent_rules/write_pipelex.mdindex dfb1ea6f5..edbda188f 100644--- a/pipelex/kit/agent_rules/write_pipelex.md+++ b/pipelex/kit/agent_rules/write_pipelex.md@@ -84,23 +84,23 @@ inputs = {### Input Multiplicity-By default, inputs expect a single item. Use expanded syntax to specify multiple items:+By default, inputs expect a single item. Use bracket notation to specify multiple items:```plx-# Single item (default) - use standard syntax+# Single item (default)inputs = { document = "Text" }# Variable list - indeterminate number of items-inputs = { documents = { concept = "Text", multiplicity = true } }+inputs = { documents = "Text[]" }# Fixed count - exactly N items-inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }+inputs = { comparison_items = "Image[2]" }```**Key points:**-- Don't use `multiplicity = false` - it's unnecessary (default behavior)-- Use `multiplicity = true` for lists of unknown length-- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)+- No brackets = single item (default behavior)+- Use `[]` for lists of unknown length+- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)## Structuring Models@@ -368,22 +368,22 @@ prompt = "Analyze this data"### Multiple Outputs-Generate multiple outputs (fixed number):+Generate multiple outputs (fixed number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas+output = "Idea[3]" # Generate exactly 3 ideas+prompt = "Generate 3 ideas"```-Generate multiple outputs (variable number):+Generate multiple outputs (variable number) - use bracket notation:```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"-output = "Idea"-multiple_output = true # Let the LLM decide how many to generate+output = "Idea[]" # Let the LLM decide how many to generate+prompt = "Generate ideas"```### Vision@@ -634,8 +634,7 @@ Multiple Image Generation:type = "PipeImgGen"description = "Generate multiple image variations"inputs = { prompt = "ImgGenPrompt" }-output = "Image"-nb_output = 3+output = "Image[3]"seed = "auto"```@@ -662,7 +661,6 @@ safety_tolerance = 3- `quality`: Image quality ("standard", "hd")**Output Configuration:**-- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")@@ -816,7 +814,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (tempExamples:```toml-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```diff --git a/pipelex/kit/configs/inference/deck/base_deck.toml b/pipelex/kit/configs/inference/deck/base_deck.tomlindex 1b6fbcff1..7cfc55299 100644--- a/pipelex/kit/configs/inference/deck/base_deck.toml+++ b/pipelex/kit/configs/inference/deck/base_deck.toml@@ -67,15 +67,17 @@ llm_cheap = { model = "base-gemini", temperature = 0.1 }# Generation skillsllm_for_factual_writing = { model = "base-gpt", temperature = 0.1 }llm_for_creative_writing = { model = "claude-4.5-sonnet", temperature = 0.9 }+llm_to_write_questions = { model = "gpt-4o-mini", temperature = 0.3 }# Reasoning skills-llm_to_reason_short = { model = "base-claude", temperature = 0.5, max_tokens = 500 }-llm_to_reason = { model = "base-claude", temperature = 1 }+llm_for_complex_reasoning = { model = "base-claude", temperature = 1 }llm_to_reason_on_diagram = { model = "base-claude", temperature = 0.5 }+llm_to_analyze_data = { model = "base-claude", temperature = 0.5 }# Search and answer skills-llm_to_answer = { model = "base-claude", temperature = 0.1 }-llm_to_retrieve = { model = "base-gemini", temperature = 0.1 }+llm_to_answer_easy_questions = { model = "gpt-4o-mini", temperature = 0.3 }+llm_to_answer_hard_questions = { model = "base-claude", temperature = 0.3 }+llm_to_retrieve = { model = "base-claude", temperature = 0.1 }llm_for_enrichment = { model = "gpt-4o", temperature = 0.1 }llm_to_enrich = { model = "base-claude", temperature = 0.1 }llm_for_question_and_excerpt_reformulation = { model = "gpt-4o", temperature = 0.9 }@@ -84,10 +86,15 @@ llm_for_question_and_excerpt_reformulation = { model = "gpt-4o", temperature = 0llm_to_engineer = { model = "smart_llm", temperature = 0.2 }llm_to_write = { model = "base-gemini", temperature = 0.2 }llm_to_pipelex = { model = "base-claude", temperature = 0.2 }+llm_to_code = { model = "base-claude", temperature = 0.1 }+llm_to_analyze_large_codebase = { model = "base-claude", temperature = 0.1 }# Image skills+llm_for_basic_vision = { model = "gpt-4o-mini", temperature = 0.1 }llm_to_write_img_gen_prompt = { model = "base-claude", temperature = 0.2 }llm_to_describe_img = { model = "base-claude", temperature = 0.5 }+llm_for_visual_analysis = { model = "base-claude", temperature = 0.5 }+llm_for_visual_design = { model = "base-claude", temperature = 0.5 }llm_to_design_fashion = { model = "base-claude", temperature = 0.7 }llm_for_img_to_text = { model = "base-claude", temperature = 0.1 }@@ -107,28 +114,30 @@ for_text = "cheap_llm_for_text"for_object = "cheap_llm_for_object"####################################################################################################-# OCR Presets+# Extract Presets####################################################################################################[extract]-choice_default = "base_ocr_mistral"+choice_default = "extract_text_from_visuals"[extract.presets]-base_ocr_mistral = { model = "mistral-ocr", max_nb_images = 100, image_min_size = 50 }-base_extract_pypdfium2 = { model = "pypdfium2-extract-text", max_nb_images = 100, image_min_size = 50 }+extract_text_from_visuals = { model = "mistral-ocr", max_nb_images = 100, image_min_size = 50 }+extract_text_from_pdf = { model = "pypdfium2-extract-text", max_nb_images = 100, image_min_size = 50 }##################################################################################################### Image Generation Presets####################################################################################################[img_gen]-choice_default = "base_img_gen"+choice_default = "gen_image_basic"[img_gen.presets]++# General purpose-base_img_gen = { model = "base-img-gen", quality = "medium", guidance_scale = 7.5, is_moderated = true, safety_tolerance = 3 }-fast_img_gen = { model = "fast-img-gen", nb_steps = 4, guidance_scale = 5.0, is_moderated = true, safety_tolerance = 3 }-high_quality_img_gen = { model = "best-img-gen", quality = "high", guidance_scale = 8.0, is_moderated = true, safety_tolerance = 3 }+gen_image_basic = { model = "base-img-gen", quality = "medium", guidance_scale = 7.5, is_moderated = true, safety_tolerance = 3 }+gen_image_fast = { model = "fast-img-gen", nb_steps = 4, guidance_scale = 5.0, is_moderated = true, safety_tolerance = 3 }+gen_image_high_quality = { model = "best-img-gen", quality = "high", guidance_scale = 8.0, is_moderated = true, safety_tolerance = 3 }# Specific skillsimg_gen_for_art = { model = "best-img-gen", quality = "high", guidance_scale = 9.0, is_moderated = false, safety_tolerance = 5 }diff --git a/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md b/pipelex/kit/migrations/migrate_0.11.0_0.12.x.mdsimilarity index 89%rename from pipelex/kit/migrations/migrate_0.11.0_0.12.0.mdrename to pipelex/kit/migrations/migrate_0.11.0_0.12.x.mdindex 3bbc88aed..198a79b07 100644--- a/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md+++ b/pipelex/kit/migrations/migrate_0.11.0_0.12.x.md@@ -18,6 +18,8 @@ This release introduces several breaking changes to make the Pipelex language mo- [ ] Move .plx files to appropriate locations in your project- [ ] Update Pipelex.make() calls (remove config path parameters)- [ ] Add @pipe_func() decorators to custom functions used in PipeFunc operators+- [ ] **Update input multiplicity syntax to use bracket notation**+- [ ] **Update output multiplicity to use bracket notation (remove nb_output/multiple_output)**- [ ] Update PipeCompose (formerly PipeJinja2)- [ ] Update PipeExtract (formerly PipeOCR)- [ ] Update PipeLLM prompts and fields@@ -270,7 +272,108 @@ Solution:2. Ensure function is `async` and accepts `working_memory`3. Verify function is in a discoverable location-## 2. General Changes+## 2. Input Multiplicity Syntax Simplification++Input multiplicity now uses concise bracket notation instead of verbose dictionary syntax.++### Migration Patterns++- `{ concept = "ConceptName" }` → `"ConceptName"`+- `{ concept = "ConceptName", multiplicity = true }` → `"ConceptName[]"`+- `{ concept = "ConceptName", multiplicity = N }` → `"ConceptName[N]"`++### Examples++**Variable list:**++Before: `inputs = { documents = { concept = "Document", multiplicity = true } }`++After: `inputs = { documents = "Document[]" }`++**Fixed count:**++Before: `inputs = { images = { concept = "Image", multiplicity = 2 } }`++After: `inputs = { images = "Image[2]" }`++## 3. Output Multiplicity - Bracket Notation++Output multiplicity now uses bracket notation in the output field instead of separate parameters.++### Migration Patterns++**Variable output:**+- Before: `output = "Concept"` + `multiple_output = true`+- After: `output = "Concept[]"`++**Fixed output:**+- Before: `output = "Concept"` + `nb_output = 5`+- After: `output = "Concept[5]"`++### Examples++**PipeLLM variable output:**++Before:+```plx+[pipe.extract_companies]+type = "PipeLLM"+inputs = { article = "Article" }+output = "CompanyName"+multiple_output = true+```++After:+```plx+[pipe.extract_companies]+type = "PipeLLM"+inputs = { article = "Article" }+output = "CompanyName[]"+```++**PipeLLM fixed output:**++Before:+```plx+[pipe.generate_ideas]+type = "PipeLLM"+output = "Idea"+nb_output = 3+```++After:+```plx+[pipe.generate_ideas]+type = "PipeLLM"+output = "Idea[3]"+```++**PipeImgGen fixed output:**++Before:+```plx+[pipe.generate_logos]+type = "PipeImgGen"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+nb_output = 3+```++After:+```plx+[pipe.generate_logos]+type = "PipeImgGen"+inputs = { prompt = "ImgGenPrompt" }+output = "Image[3]"+```++### Find and Replace++In `.plx` files, remove these fields when updating to bracket notation:+- Remove `multiple_output = true` → Add `[]` to output+- Remove `nb_output = N` → Add `[N]` to output++## 4. General Changes### Rename `definition` to `description`@@ -293,7 +396,7 @@ type = "PipeLLM"description = "Process data"```-## 3. PipeCompose (formerly PipeJinja2)+## 5. PipeCompose (formerly PipeJinja2)### Rename pipe type@@ -355,7 +458,7 @@ category = "html"templating_style = { tag_style = "square_brackets", text_format = "html" }```-## 4. PipeExtract (formerly PipeOCR)+## 6. PipeExtract (formerly PipeOCR)### Rename pipe type@@ -401,7 +504,7 @@ If you're using these functions in Python code:**Find:** `ocr_page_contents_and_views_from_pdf`**Replace with:** `extract_page_contents_and_views_from_pdf`-## 5. PipeLLM Changes+## 7. PipeLLM Changes### Rename prompt field@@ -479,7 +582,7 @@ Extract person information from this text:"""```-## 6. PipeImgGen Changes+## 8. PipeImgGen Changes### Rename model field@@ -519,7 +622,7 @@ Or use a preset:model = "img_gen_preset_name"```-## 7. PipeCondition Changes+## 9. PipeCondition Changes### Rename outcome fields@@ -569,7 +672,7 @@ To fail when no match:default_outcome = "fail"```-## 8. Configuration Files (.pipelex/ directory)+## 10. Configuration Files (.pipelex/ directory)### LLM presets in deck files@@ -579,13 +682,13 @@ default_outcome = "fail"**Before (.pipelex/inference/deck/base_deck.toml):**```toml[presets.llm]-llm_to_reason = { llm_handle = "claude-3-5-sonnet", temperature = 1 }+llm_for_complex_reasoning = { llm_handle = "claude-3-5-sonnet", temperature = 1 }```**After:**```toml[presets.llm]-llm_to_reason = { model = "claude-3-5-sonnet", temperature = 1 }+llm_for_complex_reasoning = { model = "claude-3-5-sonnet", temperature = 1 }```### Image generation presets@@ -614,15 +717,15 @@ fast_gen = { model = "fast-img-gen", quality = "standard" }**Replace with:** `[presets.extract]`**Find:** `base_ocr_pypdfium2`-**Replace with:** `base_extract_pypdfium2`+**Replace with:** `extract_text_from_pdf`-**Find:** `base_ocr_mistral`+**Find:** `extract_text_from_visuals`**Replace with:** `base_extract_mistral`**Before:**```toml[presets.ocr]-base_ocr_mistral = { ocr_handle = "mistral-ocr" }+extract_text_from_visuals = { ocr_handle = "mistral-ocr" }```**After:**@@ -656,7 +759,7 @@ is_auto_setup_preset_extract = truenb_extract_pages = 10```-## 9. Test Markers+## 11. Test Markers### Update pytest markers@@ -691,7 +794,7 @@ class TestExtractPipeline:**Find:** `make test-ocr` or `make to`**Replace with:** `make test-extract` or `make te`-## 10. Validation+## 12. ValidationAfter making changes, thoroughly test your migration:@@ -735,7 +838,7 @@ source .venv/bin/activate # Unix/macOS- Verify structure classes are discovered- Confirm custom functions are registered-## 11. Python API Changes for Client Projects+## 13. Python API Changes for Client ProjectsThese changes affect Python code that imports from or uses pipelex.@@ -979,7 +1082,7 @@ get_inference_manager().set_llm_worker_from_external_plugin()```-## 12. File Cleanup+## 14. File Cleanup### Remove Deprecated Files@@ -1004,7 +1107,7 @@ If your project has `AGENTS.md` or `CLAUDE.md` files with Pipelex examples:- `llm = ` → `model = `- `prompt_template = ` → `prompt = `-## 13. Common Issues+## 15. Common Issues### Issue: Pipeline validation fails with "unknown field"@@ -1070,7 +1173,7 @@ from pipelex.core.stuffs.text_content import TextContent- `pipelex.pipe_operators.ocr` → `pipelex.pipe_operators.extract`- All class names: `Ocr*` → `Extract*`-## 14. Automation Tools+## 16. Automation ToolsYou can automate many of these text replacements using standard tools available on your platform:@@ -1092,6 +1195,12 @@ The following replacements can be done with find/replace tools:**In `.plx` files:**- `definition = "` → `description = "`+- `{ concept = "ConceptName", multiplicity = false }` → `"ConceptName"`+- `{ concept = "ConceptName", multiplicity = true }` → `"ConceptName[]"`+- `{ concept = "ConceptName", multiplicity = N }` → `"ConceptName[N]"`+- `{ concept = "ConceptName" }` → `"ConceptName"`+- Remove `multiple_output = true` lines and add `[]` to output field+- Remove `nb_output = N` lines and add `[N]` to output field- `type = "PipeJinja2"` → `type = "PipeCompose"`- `type = "PipeOCR"` → `type = "PipeExtract"`- `prompt_template = ` → `prompt = `@@ -1101,6 +1210,10 @@ The following replacements can be done with find/replace tools:- `default_pipe_code = ` → `default_outcome = `**In `.py` files:**+- Remove `from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint`+- `InputRequirementBlueprint(concept="ConceptName", multiplicity=True)` → `"ConceptName[]"`+- `InputRequirementBlueprint(concept="ConceptName", multiplicity=N)` → `"ConceptName[N]"`+- `InputRequirementBlueprint(concept="ConceptName")` → `"ConceptName"`- `ocr_page_contents_from_pdf` → `extract_page_contents_from_pdf`- Remove `relative_config_folder_path` parameters from `Pipelex.make()`- Remove `config_folder_path` parameters from `Pipelex.make()`@@ -1145,7 +1258,7 @@ These require manual intervention:3. **Activate your virtual environment** before running Pipelex commands4. **Validate after each change** (see Validation section)-## 15. Additional Resources+## 17. Additional Resources- See AGENTS.md for complete documentation of the current syntax- Run `make validate` frequently to catch issues earlydiff --git a/pipelex/language/plx_factory.py b/pipelex/language/plx_factory.pyindex 6494a979d..f5929e0ce 100644--- a/pipelex/language/plx_factory.py+++ b/pipelex/language/plx_factory.py@@ -227,15 +227,15 @@ class PlxFactory:@classmethoddef make_table_obj_for_pipe(cls, section_value: Mapping[str, Any]) -> Any:"""Make a table object for a pipe section."""- log.debug("******** Making table object for pipe section ********")+ log.verbose("******** Making table object for pipe section ********")table_obj = table()for field_key, field_value in section_value.items():- log.debug(f"------ Field {field_key} is a {type(field_value)}")+ log.verbose(f"------ Field {field_key} is a {type(field_value)}")if not isinstance(field_value, Mapping):- log.debug(f"Field is not a mapping: key = {field_key}, value = {field_value}")+ log.verbose(f"Field is not a mapping: key = {field_key}, value = {field_value}")table_obj.add(field_key, cls.convert_dicts_to_inline_tables(field_value))continue- log.debug(f"Field is a mapping: key = {field_key}, value = {field_value}")+ log.verbose(f"Field is a mapping: key = {field_key}, value = {field_value}")field_value = cast("Mapping[str, Any]", field_value)# Convert pipe configuration to table (handles template field specially)table_obj.add(field_key, cls.convert_mapping_to_table(field_value, field_ordering=cls._plx_config().pipes.field_ordering))@@ -244,34 +244,34 @@ class PlxFactory:@classmethoddef make_table_obj_for_concept(cls, section_value: Mapping[str, Any]) -> Any:"""Make a table object for a concept section."""- log.debug("******** Making table object for concept section ********")+ log.verbose("******** Making table object for concept section ********")table_obj = table()for concept_key, concept_value in section_value.items():if isinstance(concept_value, str):- log.debug(f"Concept '{concept_key}' is a string: {concept_value}")+ log.verbose(f"Concept '{concept_key}' is a string: {concept_value}")table_obj.add(concept_key, concept_value)continueif not isinstance(concept_value, Mapping):msg = f"Concept field value is not a mapping: key = {concept_key}, value = {concept_value}"raise TypeError(msg)- log.debug(f"Concept '{concept_key}' is a mapping: {concept_value}")+ log.verbose(f"Concept '{concept_key}' is a mapping: {concept_value}")concept_value = cast("Mapping[str, Any]", concept_value)concept_table_obj = table()for concept_field_key, concept_field_value in concept_value.items():if concept_field_key == CONCEPT_STRUCTURE_FIELD_KEY:if isinstance(concept_field_value, str):- log.debug(f"Structure for concept '{concept_key}' is a string: {concept_field_value}")+ log.verbose(f"Structure for concept '{concept_key}' is a string: {concept_field_value}")concept_table_obj.add("structure", concept_field_value)continueif not isinstance(concept_field_value, Mapping):msg = f"Structure field value is not a mapping: key = {concept_field_key}, value = {concept_field_value}"raise TypeError(msg)- log.debug(f"Structure for concept '{concept_key}' is a mapping: {concept_field_value}")+ log.verbose(f"Structure for concept '{concept_key}' is a mapping: {concept_field_value}")structure_value = cast("Mapping[str, Any]", concept_field_value)structure_table_obj = table()for structure_field_key, structure_field_value in structure_value.items():if isinstance(structure_field_value, str):- log.debug(f"Structure '{structure_field_key}' is a string: {structure_field_value}")+ log.verbose(f"Structure '{structure_field_key}' is a string: {structure_field_value}")structure_table_obj.add(structure_field_key, structure_field_value)continueif not isinstance(structure_field_value, Mapping):@@ -280,7 +280,7 @@ class PlxFactory:f"key = {structure_field_key}, value = {structure_field_value}")raise TypeError(msg)- log.debug(f"Structure for '{concept_key}' is a mapping: {structure_field_value}")+ log.verbose(f"Structure for '{concept_key}' is a mapping: {structure_field_value}")structure_table_obj.add(structure_field_key,cls.convert_dicts_to_inline_tables(@@ -290,7 +290,7 @@ class PlxFactory:concept_table_obj.add("structure", structure_table_obj)else:# sub_table = _convert_mapping_to_table(concept_field_value)- log.debug(f"{concept_key}/'{concept_field_key}' is inline: {concept_field_value}")+ log.verbose(f"{concept_key}/'{concept_field_key}' is inline: {concept_field_value}")concept_table_obj.add(concept_field_key, cls.convert_dicts_to_inline_tables(concept_field_value))table_obj.add(concept_key, concept_table_obj)return table_obj@@ -298,23 +298,23 @@ class PlxFactory:@classmethoddef dict_to_plx_styled_toml(cls, data: Mapping[str, Any]) -> str:"""Top-level keys become tables; second-level mappings become tables; inline tables start at third level."""- log.debug("=" * 100)+ log.verbose("=" * 100)data = remove_none_values_from_dict(data=data)document_root = document()for root_key, root_value in data.items():if not isinstance(root_value, Mapping):- log.debug(f"Root root_key is not a mapping: key = {root_key}, value = {root_value}")+ log.verbose(f"Root root_key is not a mapping: key = {root_key}, value = {root_value}")document_root.add(root_key, cls.convert_dicts_to_inline_tables(root_value))continue# It's a mapping, therefore it's a section- log.debug(f"Root {root_key} is a section -------------------")+ log.verbose(f"Root {root_key} is a section -------------------")section_key = SectionKey(root_key)section_value = cast("Mapping[str, Any]", root_value)# Skip empty mappings (empty concept and pipe sections)if not section_value:- log.debug(f"Section {section_key} is empty, skipping")+ log.verbose(f"Section {section_key} is empty, skipping")continuematch section_key:case SectionKey.PIPE:diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex e69b39ecc..cb0761519 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -75,7 +75,7 @@ class LibraryManager(LibraryManagerAbstract):@overridedef validate_libraries(self):- log.debug("LibraryManager validating libraries")+ log.verbose("LibraryManager validating libraries")self.concept_library.validate_with_libraries()self.pipe_library.validate_with_libraries()@@ -103,7 +103,7 @@ class LibraryManager(LibraryManagerAbstract):for dir_path in dirs:if not dir_path.exists():- log.debug(f"Directory does not exist, skipping: {dir_path}")+ log.verbose(f"Directory does not exist, skipping: {dir_path}")continue# Find all .plx files in the directory, excluding problematic directories@@ -119,14 +119,14 @@ class LibraryManager(LibraryManagerAbstract):# Skip if already seenif absolute_path in seen_files:- log.debug(f"Skipping duplicate PLX file: {plx_file}")+ log.verbose(f"Skipping duplicate PLX file: {plx_file}")continueif PipelexInterpreter.is_pipelex_file(plx_file):all_plx_paths.append(plx_file)seen_files.add(absolute_path)else:- log.debug(f"Skipping non-Pipelex PLX file: {plx_file}")+ log.verbose(f"Skipping non-Pipelex PLX file: {plx_file}")return all_plx_paths@@ -299,7 +299,7 @@ class LibraryManager(LibraryManagerAbstract):# Then try filesystem-based scanning if package is accessible (for completeness)pipelex_pkg_dir = get_pipelex_package_dir_for_imports()if pipelex_pkg_dir:- log.debug(f"Additionally scanning pipelex package filesystem: {pipelex_pkg_dir}")+ log.verbose(f"Additionally scanning pipelex package filesystem: {pipelex_pkg_dir}")ClassRegistryUtils.import_modules_in_folder(folder_path=str(pipelex_pkg_dir),base_class_names=[StructuredContent.__name__],@@ -310,7 +310,7 @@ class LibraryManager(LibraryManagerAbstract):# Auto-discover and register all StructuredContent classes from sys.modulesnum_registered = ClassRegistryUtils.auto_register_all_subclasses(base_class=StructuredContent)- log.debug(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")+ log.verbose(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")# Parse all blueprints firstblueprints: list[PipelexBundleBlueprint] = []diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.pyindex 04830ab14..cedd97ea3 100644--- a/pipelex/pipe_controllers/batch/pipe_batch.py+++ b/pipelex/pipe_controllers/batch/pipe_batch.py@@ -5,7 +5,6 @@ import shortuuidfrom pydantic import model_validatorfrom typing_extensions import override-from pipelex import logfrom pipelex.config import get_configfrom pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemoryfrom pipelex.core.pipes.input_requirements import InputRequirements@@ -59,7 +58,7 @@ class PipeBatch(PipeController):for variable_name in required_variables:if variable_name not in self.inputs.root:msg = f"Input '{variable_name}' of pipe '{self.code}' is not in the inputs of the pipe '{self.branch_pipe_code}'"- raise PipeInputError(msg)+ raise PipeInputError(message=msg, pipe_code=self.code, variable_name=variable_name, concept_code=None)return self@override@@ -92,11 +91,10 @@ class PipeBatch(PipeController):input_item_concept_code = self.inputs.get_required_input_requirement(input_item_stuff_name)except PipeInputNotFoundError as exc:msg = f"Batch input item stuff named '{input_item_stuff_name}' is not in this PipeBatch '{self.code}' input spec: {self.inputs}"- raise PipeInputError(msg) from exc+ raise PipeInputError(message=msg, pipe_code=self.code, variable_name=input_item_stuff_name, concept_code=None) from excif pipe_run_params.final_stuff_code:method_name = "dry_run_pipe" if pipe_run_params.run_mode == PipeRunMode.DRY else "_run_controller_pipe"- log.debug(f"PipeBatch.{method_name}() final_stuff_code: {pipe_run_params.final_stuff_code}")pipe_run_params.final_stuff_code = Nonepipe_run_params.push_pipe_layer(pipe_code=self.branch_pipe_code)@@ -106,14 +104,13 @@ class PipeBatch(PipeController):msg = (f"Input list stuff '{batch_params.input_list_stuff_name}' required by this PipeBatch '{self.code}' not found in working memory: {exc}")- raise PipeInputError(msg) from exc+ raise PipeInputError(message=msg, pipe_code=self.code, variable_name=batch_params.input_list_stuff_name, concept_code=None) from excinput_stuff_code = input_stuff.stuff_codeinput_content = input_stuff.content-if not isinstance(input_content, ListContent):msg = f"Input of PipeBatch must be ListContent, got {input_stuff.stuff_name or 'unnamed'} = {type(input_content)}. stuff: {input_stuff}"- raise PipeInputError(msg)+ raise PipeInputError(message=msg, pipe_code=self.code, variable_name=batch_params.input_list_stuff_name, concept_code=None)input_content = cast("ListContent[StuffContent]", input_content)# TODO: Make commented code work when inputing images named "a.b.c"diff --git a/pipelex/pipe_controllers/batch/pipe_batch_factory.py b/pipelex/pipe_controllers/batch/pipe_batch_factory.pyindex 93c0347df..a58b6ef48 100644--- a/pipelex/pipe_controllers/batch/pipe_batch_factory.py+++ b/pipelex/pipe_controllers/batch/pipe_batch_factory.py@@ -3,6 +3,7 @@ from typing_extensions import overridefrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_controllers.batch.pipe_batch import PipeBatchfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint@@ -19,9 +20,12 @@ class PipeBatchFactory(PipeFactoryProtocol[PipeBatchBlueprint, PipeBatch]):blueprint: PipeBatchBlueprint,concept_codes_from_the_same_domain: list[str] | None = None,) -> PipeBatch:+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeBatch(diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.pyindex bfe5536a5..50d4ba84a 100644--- a/pipelex/pipe_controllers/condition/pipe_condition.py+++ b/pipelex/pipe_controllers/condition/pipe_condition.py@@ -258,10 +258,10 @@ class PipeCondition(PipeController):error_msg = f"Conditional expression returned no result in pipe {self.code}:"error_msg += f"\n\nExpression: {self.applied_expression_template}"raise PipeConditionError(error_msg)- log.debug(f"evaluated_expression: '{evaluated_expression}'")+ log.verbose(f"evaluated_expression: '{evaluated_expression}'")# Add alias if configured- log.debug(f"add_alias: {evaluated_expression} -> {self.add_alias_from_expression_to}")+ log.verbose(f"add_alias: {evaluated_expression} -> {self.add_alias_from_expression_to}")if self.add_alias_from_expression_to:working_memory.add_alias(alias=evaluated_expression,@@ -278,7 +278,7 @@ class PipeCondition(PipeController):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeOutput:- log.dev(f"{self.class_name} generating a '{self.output.code}'")+ log.verbose(f"{self.class_name} generating a '{self.output.code}'")# TODO: restore pipe_layer feature# pipe_run_params.push_pipe_code(pipe_code=pipe_code)@@ -310,7 +310,7 @@ class PipeCondition(PipeController):pipe_condition_path_str = ".".join(pipe_condition_path)error_details = f"PipeCondition '{pipe_condition_path_str}', required_variables: {required_variables}, missing: '{exc.variable_name}'"msg = f"Some required stuff(s) not found: {error_details}"- raise PipeInputError(msg) from exc+ raise PipeInputError(message=msg, pipe_code=self.code, variable_name=exc.variable_name, concept_code=None) from exc# Track condition stepsfor required_stuff in required_stuffs:@@ -323,7 +323,7 @@ class PipeCondition(PipeController):)# Execute the chosen pipe- log.debug(f"Chosen pipe: {chosen_pipe.code}")+ log.verbose(f"Chosen pipe: {chosen_pipe.code}")pipe_output = await get_pipe_router().run(pipe_job=PipeJobFactory.make_pipe_job(pipe=chosen_pipe,@@ -354,7 +354,7 @@ class PipeCondition(PipeController):"""Dry run implementation for PipeCondition.Validates that all required inputs are present, expression is valid, and target pipes exist."""- log.debug(f"PipeCondition: dry run controller pipe: {self.code}")+ log.verbose(f"PipeCondition: dry run controller pipe: {self.code}")# 1. Validate that all required inputs are present in the working memoryneeded_inputs = self.needed_inputs()@@ -378,7 +378,7 @@ class PipeCondition(PipeController):template_category=TemplateCategory.EXPRESSION,template_source=self.applied_expression_template,)- log.debug(f"Expression template is valid, requires variables: {required_variables}")+ log.verbose(f"Expression template is valid, requires variables: {required_variables}")except Exception as exc:log.error(f"Dry run failed: invalid expression template: {exc}")error_msg = (diff --git a/pipelex/pipe_controllers/condition/pipe_condition_factory.py b/pipelex/pipe_controllers/condition/pipe_condition_factory.pyindex f3be562e1..05c5c7fcf 100644--- a/pipelex/pipe_controllers/condition/pipe_condition_factory.py+++ b/pipelex/pipe_controllers/condition/pipe_condition_factory.py@@ -3,6 +3,7 @@ from typing_extensions import overridefrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_controllers.condition.pipe_condition import PipeConditionfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint@@ -18,9 +19,12 @@ class PipeConditionFactory(PipeFactoryProtocol[PipeConditionBlueprint, PipeCondiblueprint: PipeConditionBlueprint,concept_codes_from_the_same_domain: list[str] | None = None,) -> PipeCondition:+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeCondition(diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex 013504459..c6173a0b1 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -149,7 +149,7 @@ class PipeParallel(PipeController):msg = "PipeParallel requires either add_each_output or combined_output to be set"raise PipeDefinitionError(msg)if pipe_run_params.final_stuff_code:- log.debug(f"PipeBatch.run_pipe() final_stuff_code: {pipe_run_params.final_stuff_code}")+ log.verbose(f"PipeBatch.run_pipe() final_stuff_code: {pipe_run_params.final_stuff_code}")pipe_run_params.final_stuff_code = Nonetasks: list[Coroutine[Any, Any, PipeOutput]] = []@@ -190,7 +190,7 @@ class PipeParallel(PipeController):msg = f"PipeParallel requires unique output names for each parallel sub pipe, but {sub_pipe_output_name} is already used"raise PipeDefinitionError(msg)output_stuff_contents[sub_pipe_output_name] = output_stuff.content- log.debug(f"PipeParallel '{self.code}': output_stuff_contents[{sub_pipe_output_name}]: {output_stuff_contents[sub_pipe_output_name]}")+ log.verbose(f"PipeParallel '{self.code}': output_stuff_contents[{sub_pipe_output_name}]: {output_stuff_contents[sub_pipe_output_name]}")if self.combined_output:combined_output_stuff = StuffFactory.combine_stuffs(@@ -225,7 +225,7 @@ class PipeParallel(PipeController):"""Dry run implementation for PipeParallel.Validates that all required inputs are present and that all parallel sub-pipes can be dry run."""- log.debug(f"PipeParallel: dry run controller pipe: {self.code}")+ log.verbose(f"PipeParallel: dry run controller pipe: {self.code}")if pipe_run_params.run_mode != PipeRunMode.DRY:msg = f"PipeSequence._dry_run_controller_pipe() called with run_mode = {pipe_run_params.run_mode} in pipe {self.code}"raise PipeRunParamsError(msg)diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.pyindex bad2ac441..ba48e68c1 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py@@ -5,6 +5,7 @@ from typing_extensions import overridefrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.exceptions import PipeDefinitionErrorfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel@@ -43,10 +44,13 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParallemessage=msg, domain_code=domain, pipe_code=pipe_code, description=blueprint.description, source=blueprint.source)+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+if blueprint.combined_output:combined_output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)combined_output = get_required_concept(@@ -60,7 +64,7 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParalleoutput_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeParallel(diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.pyindex a3ef87cdb..4916b181d 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py@@ -10,14 +10,14 @@ from pipelex.core.pipes.input_requirements import InputRequirementsfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.exceptions import (+ PipeControllerOutputConceptMismatchError,PipeRunParamsError,StaticValidationError,StaticValidationErrorType,)-from pipelex.hub import get_required_pipe+from pipelex.hub import get_concept_library, get_required_pipefrom pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallelfrom pipelex.pipe_controllers.pipe_controller import PipeController-from pipelex.pipe_controllers.sequence.exceptions import PipeSequenceErrorfrom pipelex.pipe_controllers.sub_pipe import SubPipefrom pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadata@@ -85,13 +85,15 @@ class PipeSequence(PipeController):The output of the pipe sequence should match the output of the last step."""last_step_output_pipe = get_required_pipe(pipe_code=self.sequential_sub_pipes[-1].pipe_code)- if self.output.concept_string != last_step_output_pipe.output.concept_string:- msg = (- f"The output concept code '{self.output.concept_string}' of the pipe '{self.code}' is "- f"not matching the output concept code '{last_step_output_pipe.output.concept_string}' "- f"of the last step '{self.sequential_sub_pipes[-1].pipe_code}'",+ concept_of_last_step = last_step_output_pipe.output+ if not get_concept_library().is_compatible(tested_concept=concept_of_last_step, wanted_concept=self.output):+ msg = f"""PipeSequence concept mismatch:+the output concept '{concept_of_last_step.concept_string}' of the last step '{self.sequential_sub_pipes[-1].pipe_code}'+of sequence pipe '{self.code}' is not compatible with the output concept '{self.output.concept_string}' of the sequence.+"""+ raise PipeControllerOutputConceptMismatchError(+ message=msg, tested_concept=concept_of_last_step.concept_string, wanted_concept=self.output.concept_string)- raise PipeSequenceError(msg)@model_validator(mode="after")def validate_inputs(self) -> Self:@@ -135,7 +137,7 @@ class PipeSequence(PipeController):# Check that all declared inputs are actually neededfor input_name in self.inputs.variables:if input_name not in the_needed_inputs.required_names:- log.debug(f"the_needed_inputs.required_names: {the_needed_inputs.required_names}")+ log.verbose(f"the_needed_inputs.required_names: {the_needed_inputs.required_names}")extraneous_input_var_error = StaticValidationError(error_type=StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,domain=self.domain,@@ -207,13 +209,16 @@ class PipeSequence(PipeController):msg = f"PipeSequence._dry_run_controller_pipe() called with run_mode = {pipe_run_params.run_mode} in pipe {self.code}"raise PipeRunParamsError(msg)# Verify the output of this pipe is matching the output of the last step.- concept_string_of_last_step = get_required_pipe(pipe_code=self.sequential_sub_pipes[-1].pipe_code).output.concept_string- if self.output.concept_string != concept_string_of_last_step:- msg = (- f"The output concept code '{self.output.concept_string}' of the pipe '{self.code}' is "- f"not matching the output concept code '{concept_string_of_last_step}' of the last step '{self.sequential_sub_pipes[-1].pipe_code}'"+ concept_of_last_step = get_required_pipe(pipe_code=self.sequential_sub_pipes[-1].pipe_code).output+ # if self.output.concept_string != concept_string_of_last_step:+ if not get_concept_library().is_compatible(tested_concept=concept_of_last_step, wanted_concept=self.output):+ msg = f"""PipeSequence concept mismatch:+the output concept '{concept_of_last_step.concept_string}' of the last step '{self.sequential_sub_pipes[-1].pipe_code}'+of sequence pipe '{self.code}' is not compatible with the output concept '{self.output.concept_string}' of the sequence.+"""+ raise PipeControllerOutputConceptMismatchError(+ message=msg, tested_concept=concept_of_last_step.concept_string, wanted_concept=self.output.concept_string)- raise ValueError(msg)return await self._run_controller_pipe(job_metadata=job_metadata,working_memory=working_memory,diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py b/pipelex/pipe_controllers/sequence/pipe_sequence_factory.pyindex a56a4225c..bbde1cc13 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py@@ -3,6 +3,7 @@ from typing_extensions import overridefrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequencefrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint@@ -19,9 +20,12 @@ class PipeSequenceFactory(PipeFactoryProtocol[PipeSequenceBlueprint, PipeSequencblueprint: PipeSequenceBlueprint,concept_codes_from_the_same_domain: list[str] | None = None,) -> PipeSequence:+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeSequence(diff --git a/pipelex/pipe_controllers/sub_pipe.py b/pipelex/pipe_controllers/sub_pipe.pyindex 79d0c9c37..14bc4a2cd 100644--- a/pipelex/pipe_controllers/sub_pipe.py+++ b/pipelex/pipe_controllers/sub_pipe.py@@ -2,8 +2,8 @@ from pydantic import BaseModelfrom pipelex import logfrom pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.exceptions import PipeInputError, WorkingMemoryStuffNotFoundErrorfrom pipelex.hub import get_pipe_router, get_pipeline_tracker, get_required_pipe@@ -11,14 +11,14 @@ from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprifrom pipelex.pipe_controllers.batch.pipe_batch_factory import PipeBatchFactoryfrom pipelex.pipe_controllers.condition.pipe_condition import PipeConditionfrom pipelex.pipe_run.pipe_job_factory import PipeJobFactory-from pipelex.pipe_run.pipe_run_params import BatchParams, PipeOutputMultiplicity, PipeRunMode, PipeRunParams+from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunMode, PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadataclass SubPipe(BaseModel):pipe_code: stroutput_name: str | None = None- output_multiplicity: PipeOutputMultiplicity | None = None+ output_multiplicity: VariableMultiplicity | None = Nonebatch_params: BatchParams | None = Noneconcept_codes_from_the_same_domain: list[str] | None = None@@ -45,7 +45,9 @@ class SubPipe(BaseModel):f"Input list stuff named '{batch_params.input_list_stuff_name}' required by sub_pipe '{self.pipe_code}' "f"of pipe '{calling_pipe_code}' not found in working memory: {exc}")- raise PipeInputError(msg) from exc+ raise PipeInputError(+ message=msg, pipe_code=self.pipe_code, variable_name=batch_params.input_list_stuff_name, concept_code=None+ ) from excitem_stuff_requirement = sub_pipe.inputs.get_required_input_requirement(variable_name=batch_params.input_item_stuff_name)pipe_batch_blueprint = PipeBatchBlueprint(@@ -55,9 +57,7 @@ class SubPipe(BaseModel):input_list_name=batch_params.input_list_stuff_name,input_item_name=batch_params.input_item_stuff_name,inputs={- batch_params.input_item_stuff_name: InputRequirementBlueprint(- concept=item_stuff_requirement.concept.concept_string,- ),+ batch_params.input_item_stuff_name: item_stuff_requirement.concept.concept_string,},)@@ -117,7 +117,7 @@ class SubPipe(BaseModel):sub_pipe_path_str = ".".join(sub_pipe_path)error_details = f"SubPipe '{sub_pipe_path_str}', required_variables: {required_variables}, missing: '{exc.variable_name}'"msg = f"Some required stuff(s) not found: {error_details}"- raise PipeInputError(msg) from exc+ raise PipeInputError(message=msg, pipe_code=self.pipe_code, variable_name=exc.variable_name, concept_code=None) from exclog.verbose(required_stuffs, title=f"Required stuffs for {self.pipe_code}")# This is the only line that changes between run and dry_rundiff --git a/pipelex/pipe_controllers/sub_pipe_factory.py b/pipelex/pipe_controllers/sub_pipe_factory.pyindex f465a3815..023d63160 100644--- a/pipelex/pipe_controllers/sub_pipe_factory.py+++ b/pipelex/pipe_controllers/sub_pipe_factory.py@@ -1,6 +1,7 @@+from pipelex.core.pipes.variable_multiplicity import make_variable_multiplicityfrom pipelex.pipe_controllers.sub_pipe import SubPipefrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint-from pipelex.pipe_run.pipe_run_params import BatchParams, make_output_multiplicity+from pipelex.pipe_run.pipe_run_params import BatchParamsclass SubPipeFactory:@@ -11,9 +12,9 @@ class SubPipeFactory:concept_codes_from_the_same_domain: list[str] | None = None,) -> SubPipe:"""Create a SubPipe from a SubPipeBlueprint."""- output_multiplicity = make_output_multiplicity(- nb_output=blueprint.nb_output,- multiple_output=blueprint.multiple_output,+ output_multiplicity = make_variable_multiplicity(+ nb_items=blueprint.nb_output,+ multiple_items=blueprint.multiple_output,)batch_params = BatchParams.make_optional_batch_params(input_list_name=blueprint.batch_over,diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.pyindex 6f2ac546c..4d5553fee 100644--- a/pipelex/pipe_operators/compose/pipe_compose.py+++ b/pipelex/pipe_operators/compose/pipe_compose.py@@ -153,10 +153,10 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):) -> PipeComposeOutput:content_generator_used: ContentGeneratorProtocolif get_config().pipelex.dry_run_config.apply_to_jinja2_rendering:- log.debug(f"PipeCompose: using dry run operator pipe for jinja2 rendering: {self.code}")+ log.verbose(f"PipeCompose: using dry run operator pipe for jinja2 rendering: {self.code}")content_generator_used = ContentGeneratorDry()else:- log.debug(f"PipeCompose: using regular operator pipe for jinja2 rendering (dry run not applied to jinja2): {self.code}")+ log.verbose(f"PipeCompose: using regular operator pipe for jinja2 rendering (dry run not applied to jinja2): {self.code}")content_generator_used = get_content_generator()return await self._run_operator_pipe(diff --git a/pipelex/pipe_operators/compose/pipe_compose_factory.py b/pipelex/pipe_operators/compose/pipe_compose_factory.pyindex 5a54dcac4..435772e26 100644--- a/pipelex/pipe_operators/compose/pipe_compose_factory.py+++ b/pipelex/pipe_operators/compose/pipe_compose_factory.py@@ -6,6 +6,7 @@ from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements import InputRequirementsfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_operators.compose.pipe_compose import PipeComposefrom pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint@@ -28,9 +29,12 @@ class PipeComposeFactory(PipeFactoryProtocol[PipeComposeBlueprint, PipeCompose])template_category=blueprint.template_category,)+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeCompose(diff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.pyindex 16759fcbc..19dcc454a 100644--- a/pipelex/pipe_operators/extract/pipe_extract.py+++ b/pipelex/pipe_operators/extract/pipe_extract.py@@ -116,7 +116,7 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):# We have confirmed right above that we have exactly one input# get input_name, requirement from single item in inputsinput_name, requirement = self.inputs.items[0]- log.debug(f"Validating input '{input_name}' with concept code '{requirement.concept.code}'")+ log.verbose(f"Validating input '{input_name}' with concept code '{requirement.concept.code}'")if concept_library.is_compatible(tested_concept=requirement.concept,wanted_concept=get_native_concept(native_concept=NativeConceptCode.IMAGE),@@ -199,21 +199,21 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):# Build the output stuff, which is a list of page contentspage_view_contents: list[ImageContent] = []if self.should_include_page_views:- log.debug(f"should_include_page_views: {self.should_include_page_views}, pdf_uri: {pdf_uri}, image_uri: {image_uri}")+ log.verbose(f"should_include_page_views: {self.should_include_page_views}, pdf_uri: {pdf_uri}, image_uri: {image_uri}")if pdf_uri:page_view_contents.extend(ImageContent.make_from_extracted_image(extracted_image=page.page_view) for page in extract_output.pages.values() if page.page_view)- log.debug(f"page_view_contents: {page_view_contents}")+ log.verbose(f"page_view_contents: {page_view_contents}")needs_to_generate_page_views: boolif len(page_view_contents) == 0:- log.debug("No page views found in the OCR output")+ log.verbose("No page views found in the OCR output")needs_to_generate_page_views = Trueelif len(page_view_contents) < len(extract_output.pages):log.warning(f"Only {len(page_view_contents)} page found in the OCR output, but {len(extract_output.pages)} pages")needs_to_generate_page_views = Trueelse:- log.debug("All page views found in the OCR output")+ log.verbose("All page views found in the OCR output")needs_to_generate_page_views = Falseif needs_to_generate_page_views:@@ -225,7 +225,7 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):page_contents: list[PageContent] = []for page_index, page in extract_output.pages.items():images = [ImageContent.make_from_extracted_image(extracted_image=img) for img in page.extracted_images]- log.debug(f"images: {images}, page_view_contents: {page_view_contents}, index: {page_index}")+ log.verbose(f"images: {images}, page_view_contents: {page_view_contents}, index: {page_index}")page_view = page_view_contents[page_index - 1] if self.should_include_page_views else Nonepage_contents.append(PageContent(@@ -263,7 +263,7 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeExtractOutput:- log.debug(f"PipeExtract: dry run operator pipe: {self.code}")+ log.verbose(f"PipeExtract: dry run operator pipe: {self.code}")if pipe_run_params.run_mode != PipeRunMode.DRY:msg = f"Running pipe '{self.code}' (PipeExtract) _dry_run_operator_pipe() in non-dry mode is not allowed."raise PipeDefinitionError(msg)diff --git a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py b/pipelex/pipe_operators/extract/pipe_extract_blueprint.pyindex 518b56296..752dde075 100644--- a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py+++ b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py@@ -1,5 +1,8 @@from typing import Literal+from pydantic import field_validator+from typing_extensions import override+from pipelex.cogt.extract.extract_setting import ExtractModelChoicefrom pipelex.core.pipes.pipe_blueprint import PipeBlueprint@@ -12,3 +15,9 @@ class PipeExtractBlueprint(PipeBlueprint):page_image_captions: bool | None = Nonepage_views: bool | None = Nonepage_views_dpi: int | None = None++ @override+ @field_validator("output", mode="before")+ @classmethod+ def validate_output(cls, output: str) -> str:+ return "Page[]"diff --git a/pipelex/pipe_operators/extract/pipe_extract_factory.py b/pipelex/pipe_operators/extract/pipe_extract_factory.pyindex f27d56516..6a3c9fad3 100644--- a/pipelex/pipe_operators/extract/pipe_extract_factory.py+++ b/pipelex/pipe_operators/extract/pipe_extract_factory.py@@ -4,6 +4,7 @@ from pipelex.config import get_configfrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_operators.extract.pipe_extract import PipeExtractfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint@@ -19,9 +20,12 @@ class PipeExtractFactory(PipeFactoryProtocol[PipeExtractBlueprint, PipeExtract])blueprint: PipeExtractBlueprint,concept_codes_from_the_same_domain: list[str] | None = None,) -> PipeExtract:+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)diff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.pyindex 56a0f89d2..c9e7a5350 100644--- a/pipelex/pipe_operators/func/pipe_func.py+++ b/pipelex/pipe_operators/func/pipe_func.py@@ -48,7 +48,7 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeFuncOutput:- log.debug(f"Applying function '{self.function_name}'")+ log.verbose(f"Applying function '{self.function_name}'")function = func_registry.get_required_function(self.function_name)@@ -93,7 +93,7 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeFuncOutput:- log.debug(f"Dry run for PipeFunc '{self.function_name}'")+ log.verbose(f"Dry run for PipeFunc '{self.function_name}'")function = func_registry.get_required_function(self.function_name)diff --git a/pipelex/pipe_operators/func/pipe_func_factory.py b/pipelex/pipe_operators/func/pipe_func_factory.pyindex 1b8e90a6d..3a848b332 100644--- a/pipelex/pipe_operators/func/pipe_func_factory.py+++ b/pipelex/pipe_operators/func/pipe_func_factory.py@@ -3,6 +3,7 @@ from typing_extensions import overridefrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_operators.func.pipe_func import PipeFuncfrom pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint@@ -18,9 +19,12 @@ class PipeFuncFactory(PipeFactoryProtocol[PipeFuncBlueprint, PipeFunc]):blueprint: PipeFuncBlueprint,concept_codes_from_the_same_domain: list[str] | None = None,) -> PipeFunc:+ # Parse output to strip multiplicity brackets+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)+output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeFunc(diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen.py b/pipelex/pipe_operators/img_gen/pipe_img_gen.pyindex 0cd6b8d43..f94d693c3 100644--- a/pipelex/pipe_operators/img_gen/pipe_img_gen.py+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen.py@@ -6,7 +6,7 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDryfrom pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol-from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobParams, OutputFormat, Quality+from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobParams, OutputFormatfrom pipelex.cogt.img_gen.img_gen_prompt import ImgGenPromptfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSettingfrom pipelex.cogt.models.model_deck_check import check_img_gen_choice_with_deck@@ -17,6 +17,7 @@ from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.input_requirements import InputRequirementsfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff_factory import StuffFactory@@ -26,12 +27,13 @@ from pipelex.exceptions import (PipeRunParamsError,StaticValidationError,StaticValidationErrorType,+ StuffContentTypeError,UnexpectedPipeDefinitionError,WorkingMemoryStuffNotFoundError,)from pipelex.hub import get_concept_library, get_content_generator, get_model_deck, get_native_conceptfrom pipelex.pipe_operators.pipe_operator import PipeOperator-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity, PipeRunMode, PipeRunParams, output_multiplicity_to_apply+from pipelex.pipe_run.pipe_run_params import PipeRunMode, PipeRunParams, output_multiplicity_to_applyfrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.types import Self@@ -64,16 +66,15 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):img_gen_prompt: str | None = Noneimg_gen_prompt_var_name: str | None = None- # New ImgGenChoice pattern (like LLM)img_gen: ImgGenModelChoice | None = None# Legacy individual settings (for backwards compatibility)- img_gen_handle: str | None = None- quality: Quality | None = Field(default=None, strict=False)- nb_steps: int | None = Field(default=None, gt=0)- guidance_scale: float | None = Field(default=None, gt=0)- is_moderated: bool | None = None- safety_tolerance: int | None = Field(default=None, ge=1, le=6)+ # img_gen_handle: str | None = None+ # quality: Quality | None = Field(default=None, strict=False)+ # nb_steps: int | None = Field(default=None, gt=0)+ # guidance_scale: float | None = Field(default=None, gt=0)+ # is_moderated: bool | None = None+ # safety_tolerance: int | None = Field(default=None, ge=1, le=6)# One-time settings (not in ImgGenSetting)aspect_ratio: AspectRatio | None = Field(default=None, strict=False)@@ -81,7 +82,7 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):seed: int | Literal["auto"] | None = Nonebackground: Background | None = Field(default=None, strict=False)output_format: OutputFormat | None = Field(default=None, strict=False)- output_multiplicity: PipeOutputMultiplicity+ output_multiplicity: VariableMultiplicity@field_validator("img_gen_prompt_var_name")@classmethod@@ -222,15 +223,23 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):)applied_output_multiplicity = multiplicity_resolution.resolved_multiplicity- log.debug("Getting image generation prompt from context")+ log.verbose("Getting image generation prompt from context")if self.img_gen_prompt:img_gen_prompt_text = self.img_gen_promptelif stuff_name := self.img_gen_prompt_var_name:try:img_gen_prompt_text = working_memory.get_stuff_as_str(stuff_name)- except WorkingMemoryStuffNotFoundError as exc:- msg = f"Could not find a valid user image named '{stuff_name}' in the working_memory: {exc}"- raise PipeInputError(msg) from exc+ except WorkingMemoryStuffNotFoundError as stuff_not_found_error:+ msg = f"Could not find a valid image generation prompt named '{stuff_name}' in the working_memory: {stuff_not_found_error}"+ raise PipeInputError(+ message=msg,+ pipe_code=self.code,+ variable_name=stuff_name,+ concept_code=None,+ ) from stuff_not_found_error+ except StuffContentTypeError as stuff_content_type_error:+ msg = f"The image generation prompt named '{stuff_name}' is not of the right type: {stuff_content_type_error}"+ raise PipeInputError(message=msg, pipe_code=self.code, variable_name=stuff_name, concept_code=None) from stuff_content_type_errorelse:msg = "You must provide an image gen prompt either as attribute of the pipe or as a single text input"raise UnexpectedPipeDefinitionError(msg)@@ -244,16 +253,16 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):if self.img_gen is not None:# New pattern: use img_gen choice (preset or inline setting)img_gen_setting = model_deck.get_img_gen_setting(self.img_gen)- elif self.img_gen_handle is not None:- # Legacy pattern: create ImgGenSetting from individual settings- img_gen_setting = ImgGenSetting(- model=self.img_gen_handle,- quality=self.quality,- nb_steps=self.nb_steps,- guidance_scale=self.guidance_scale or img_gen_param_defaults.guidance_scale,- is_moderated=self.is_moderated if self.is_moderated is not None else img_gen_param_defaults.is_moderated,- safety_tolerance=self.safety_tolerance or img_gen_param_defaults.safety_tolerance,- )+ # elif self.img_gen_handle is not None:+ # # Legacy pattern: create ImgGenSetting from individual settings+ # img_gen_setting = ImgGenSetting(+ # model=self.img_gen_handle,+ # quality=self.quality,+ # nb_steps=self.nb_steps,+ # guidance_scale=self.guidance_scale or img_gen_param_defaults.guidance_scale,+ # is_moderated=self.is_moderated if self.is_moderated is not None else img_gen_param_defaults.is_moderated,+ # safety_tolerance=self.safety_tolerance or img_gen_param_defaults.safety_tolerance,+ # )else:# Use default from model deckimg_gen_setting = model_deck.get_img_gen_setting(model_deck.img_gen_choice_default)@@ -281,7 +290,7 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):)# Get the image generation handleimg_gen_handle = img_gen_setting.model- log.debug(f"Using img_gen handle: {img_gen_handle}")+ log.verbose(f"Using img_gen handle: {img_gen_handle}")the_content: StuffContentimage_urls: list[str] = []@@ -367,7 +376,6 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeImgGenOutput:- log.debug(f"PipeImgGen: dry run operator pipe: {self.code}")content_generator_dry = ContentGeneratorDry()return await self._run_operator_pipe(job_metadata=job_metadata,diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py b/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.pyindex 734e45bd4..402bc7e2e 100644--- a/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py@@ -19,6 +19,5 @@ class PipeImgGenBlueprint(PipeBlueprint):aspect_ratio: AspectRatio | None = Field(default=None, strict=False)is_raw: bool | None = Noneseed: int | Literal["auto"] | None = None- nb_output: int | None = Field(default=None, ge=1)background: Background | None = Field(default=None, strict=False)output_format: OutputFormat | None = Field(default=None, strict=False)diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py b/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.pyindex 72f62f0e6..a25574245 100644--- a/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py@@ -3,6 +3,7 @@ from typing_extensions import overridefrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.hub import get_required_conceptfrom pipelex.pipe_operators.img_gen.pipe_img_gen import PipeImgGenfrom pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint@@ -18,9 +19,16 @@ class PipeImgGenFactory(PipeFactoryProtocol[PipeImgGenBlueprint, PipeImgGen]):blueprint: PipeImgGenBlueprint,concept_codes_from_the_same_domain: list[str] | None = None,) -> PipeImgGen:+ # Parse output for multiplicity (may have brackets like "Image[]" or "Image[3]")+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)++ # Convert bracket notation to output_multiplicity (default to 1 if no brackets)+ final_multiplicity = output_parse_result.multiplicity if isinstance(output_parse_result.multiplicity, int) else 1++ # Use concept without brackets for output concept resolutionoutput_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)return PipeImgGen(@@ -38,7 +46,7 @@ class PipeImgGenFactory(PipeFactoryProtocol[PipeImgGenBlueprint, PipeImgGen]):concept_code=output_domain_and_code.concept_code,),),- output_multiplicity=blueprint.nb_output or 1,+ output_multiplicity=final_multiplicity,img_gen_prompt=blueprint.img_gen_prompt,img_gen_prompt_var_name=blueprint.img_gen_prompt_var_name,img_gen=blueprint.model,diff --git a/pipelex/pipe_operators/llm/llm_prompt_blueprint.py b/pipelex/pipe_operators/llm/llm_prompt_blueprint.pyindex 6d5a031ed..80ce1b08a 100644--- a/pipelex/pipe_operators/llm/llm_prompt_blueprint.py+++ b/pipelex/pipe_operators/llm/llm_prompt_blueprint.py@@ -73,13 +73,30 @@ class LLMPromptBlueprint(BaseModel):prompt_user_images: dict[str, PromptImage] = {}if self.user_images:for user_image_name in self.user_images:- log.debug(f"Getting user image '{user_image_name}' from context")+ log.verbose(f"Getting user image '{user_image_name}' from context")# Try to get as a single ImageContent firsttry:- prompt_image_content = context_provider.get_typed_object_or_attribute(name=user_image_name, wanted_type=ImageContent)+ prompt_image_content = context_provider.get_typed_object_or_attribute(+ name=user_image_name,+ wanted_type=ImageContent,+ accept_list=True,+ )if isinstance(prompt_image_content, ImageContent):user_image = PromptImageFactory.make_prompt_image(url=prompt_image_content.url, base_64_str=prompt_image_content.base_64)prompt_user_images[user_image_name] = user_image+ elif isinstance(prompt_image_content, list):+ for image_item in prompt_image_content: # pyright: ignore[reportUnknownVariableType]+ if not isinstance(image_item, ImageContent):+ msg = f"Item of '{user_image_name}' is of type '{type(image_item).__name__}', it should be ImageContent" # pyright: ignore[reportUnknownArgumentType]+ raise LLMPromptSpecError(msg)+ user_image = PromptImageFactory.make_prompt_image(url=image_item.url, base_64_str=image_item.base_64)+ prompt_user_images[user_image_name] = user_image+ else:+ msg = (+ f"User image '{user_image_name}' is of type '{type(prompt_image_content).__name__}', "+ "it should be ImageContent or list of ImageContent"+ )+ raise LLMPromptSpecError(msg)except ContextProviderException:# If single image failed, try to get as a collection (list or tuple)try:diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.pyindex cf36e5a69..1b2f93a75 100644--- a/pipelex/pipe_operators/llm/pipe_llm.py+++ b/pipelex/pipe_operators/llm/pipe_llm.py@@ -20,6 +20,7 @@ from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.input_requirements import InputRequirementsfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.stuff_factory import StuffFactory@@ -41,7 +42,6 @@ from pipelex.pipe_operators.llm.llm_prompt_blueprint import LLMPromptBlueprintfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import StructuringMethodfrom pipelex.pipe_operators.pipe_operator import PipeOperatorfrom pipelex.pipe_run.pipe_run_params import (- PipeOutputMultiplicity,PipeRunParamKey,PipeRunParams,output_multiplicity_to_apply,@@ -60,7 +60,7 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):llm_prompt_spec: LLMPromptBlueprintllm_choices: LLMSettingChoices | None = Nonestructuring_method: StructuringMethod | None = None- output_multiplicity: PipeOutputMultiplicity | None = None+ output_multiplicity: VariableMultiplicity | None = None@model_validator(mode="after")def _validate_inputs(self) -> Self:@@ -173,7 +173,6 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):llm_config = get_config().cogt.llm_configcontent_generator = content_generator or get_content_generator()# interpret / unwrap the arguments- log.debug(f"PipeLLM pipe_code = {self.code}")output_concept = self.outputif self.output.code == SpecialDomain.NATIVE + "." + NativeConceptCode.DYNAMIC:# TODO: This DYNAMIC_OUTPUT_CONCEPT should not be a field in the params attribute of PipeRunParams.@@ -264,16 +263,16 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):)else:if is_multiple_output:- log.debug(f"PipeLLM generating {fixed_nb_output} output(s)" if fixed_nb_output else "PipeLLM generating a list of output(s)")+ log.verbose(f"PipeLLM generating {fixed_nb_output} output(s)" if fixed_nb_output else "PipeLLM generating a list of output(s)")else:- log.debug(f"PipeLLM generating a single object output, class name: '{output_concept.structure_class_name}'")+ log.verbose(f"PipeLLM generating a single object output, class name: '{output_concept.structure_class_name}'")# TODO: we need a better solution for structuring_method (text then object), meanwhile,# we acknowledge the code here with llm_prompt_1 and llm_prompt_2 is overly complex and should be refactored.llm_prompt_2_factory: LLMPromptFactoryAbstract | Noneif self.structuring_method:structuring_method = cast("StructuringMethod", self.structuring_method)- log.debug(f"PipeLLM pipe_code is '{self.code}' and structuring_method is '{structuring_method}'")+ log.verbose(f"PipeLLM pipe_code is '{self.code}' and structuring_method is '{structuring_method}'")match structuring_method:case StructuringMethod.DIRECT:llm_prompt_2_factory = None@@ -281,9 +280,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):log.verbose(f"Creating llm_prompt_2_factory for pipe {self.code} with structuring_method {structuring_method}")llm_prompt_2_factory = LLMPromptTemplate.make_for_structuring_from_preliminary_text()elif get_config().pipelex.structure_config.is_default_text_then_structure:- log.debug(f"PipeLLM pipe_code is '{self.code}' and is_default_text_then_structure")+ log.verbose(f"PipeLLM pipe_code is '{self.code}' and is_default_text_then_structure")llm_prompt_2_factory = LLMPromptTemplate.make_for_structuring_from_preliminary_text()- log.debug(llm_prompt_2_factory, title="llm_prompt_2_factory")+ log.verbose(llm_prompt_2_factory, title="llm_prompt_2_factory")else:llm_prompt_2_factory = None@@ -349,12 +348,12 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):task_desc = f"{self.__class__.__name__}_gen_{fixed_nb_output}x{content_class.__class__.__name__}"else:task_desc = f"{self.__class__.__name__}_gen_list_{content_class.__class__.__name__}"- log.dev(task_desc)+ log.verbose(task_desc)generated_objects: list[StuffContent]if llm_prompt_2_factory is not None:# We're generating a list of objects using preliminary textmethod_desc = "text_then_object"- log.dev(f"{task_desc} by {method_desc}")+ log.verbose(f"{task_desc} by {method_desc}")log.verbose(f"llm_prompt_2_factory: {llm_prompt_2_factory}")generated_objects = await content_generator.make_text_then_object_list(@@ -369,7 +368,7 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):else:# We're generating a list of objects directlymethod_desc = "object_direct"- log.dev(f"{task_desc} by {method_desc}, content_class={content_class.__name__}")+ log.verbose(f"{task_desc} by {method_desc}, content_class={content_class.__name__}")generated_objects = await content_generator.make_object_list_direct(job_metadata=job_metadata,object_class=content_class,@@ -431,8 +430,6 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):async def get_output_structure_prompt(concept_string: str, is_with_preliminary_text: bool) -> str | None:concept = get_required_concept(concept_string=concept_string)output_class = get_class_registry().get_class(concept.structure_class_name)- log.debug(f"get_output_structure_prompt for {concept_string} with {is_with_preliminary_text=}")- log.debug(f"output_class: {output_class}")if not output_class:return Nonediff --git a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py b/pipelex/pipe_operators/llm/pipe_llm_blueprint.pyindex 32369e84e..78868772a 100644--- a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py+++ b/pipelex/pipe_operators/llm/pipe_llm_blueprint.py@@ -1,12 +1,8 @@from typing import Literal-from pydantic import field_validator, model_validator-from pipelex.cogt.llm.llm_setting import LLMModelChoicefrom pipelex.core.pipes.pipe_blueprint import PipeBlueprint-from pipelex.exceptions import PipeDefinitionError-from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_list-from pipelex.types import Self, StrEnum+from pipelex.types import StrEnumclass StructuringMethod(StrEnum):@@ -25,24 +21,3 @@ class PipeLLMBlueprint(PipeBlueprint):prompt: str | None = Nonestructuring_method: StructuringMethod | None = None-- nb_output: int | None = None- multiple_output: bool | None = None-- @field_validator("nb_output", mode="after")- @classmethod- def validate_nb_output(cls, value: int | None = None) -> int | None:- if value and value < 2:- msg = "PipeLLMBlueprint nb_output must be at least 2"- raise PipeDefinitionError(message=msg)- return value-- @model_validator(mode="after")- def validate_multiple_output(self) -> Self:- if excess_attributes_list := has_more_than_one_among_attributes_from_list(- self,- attributes_list=["nb_output", "multiple_output"],- ):- msg = f"PipeLLMBlueprint should have no more than one of {excess_attributes_list} among them"- raise PipeDefinitionError(msg)- return selfdiff --git a/pipelex/pipe_operators/llm/pipe_llm_factory.py b/pipelex/pipe_operators/llm/pipe_llm_factory.pyindex 7392071ab..c97d82970 100644--- a/pipelex/pipe_operators/llm/pipe_llm_factory.py+++ b/pipelex/pipe_operators/llm/pipe_llm_factory.py@@ -6,15 +6,14 @@ from pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCode-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.input_requirements_factory import InputRequirementsFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.pipes.variable_multiplicity import make_variable_multiplicity, parse_concept_with_multiplicityfrom pipelex.exceptions import PipeDefinitionErrorfrom pipelex.hub import get_native_concept, get_optional_domain, get_required_conceptfrom pipelex.pipe_operators.llm.llm_prompt_blueprint import LLMPromptBlueprintfrom pipelex.pipe_operators.llm.pipe_llm import PipeLLMfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint-from pipelex.pipe_run.pipe_run_params import make_output_multiplicityfrom pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError@@ -61,17 +60,13 @@ class PipeLLMFactory(PipeFactoryProtocol[PipeLLMBlueprint, PipeLLM]):user_images: list[str] = []if blueprint.inputs:- for stuff_name, requirement in blueprint.inputs.items():- if isinstance(requirement, str):- # if it's just a concept string, it's a concept, we make a basic input requirement from it- input_requirement_blueprint = InputRequirementBlueprint(concept=requirement)- else:- input_requirement_blueprint = requirement+ for stuff_name, requirement_str in blueprint.inputs.items():+ # Parse to strip multiplicity brackets+ input_parse_result = parse_concept_with_multiplicity(requirement_str)- concept_string = input_requirement_blueprint.conceptdomain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=concept_string,+ concept_string_or_code=input_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)concept = get_required_concept(@@ -101,16 +96,19 @@ class PipeLLMFactory(PipeFactoryProtocol[PipeLLMBlueprint, PipeLLM]):for_object=blueprint.model_to_structure,)- # output_multiplicity defaults to False for PipeLLM so unless it's run with explicit demand for multiple outputs,- # we'll generate only one output- output_multiplicity = make_output_multiplicity(- nb_output=blueprint.nb_output,- multiple_output=blueprint.multiple_output,+ # Parse output for multiplicity (may have brackets like "Text[]" or "Text[3]")+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)++ # Convert bracket notation to output_multiplicity+ output_multiplicity = make_variable_multiplicity(+ nb_items=output_parse_result.multiplicity if isinstance(output_parse_result.multiplicity, int) else None,+ multiple_items=output_parse_result.multiplicity if isinstance(output_parse_result.multiplicity, bool) else None,)+ # Use concept without brackets for output concept resolutionoutput_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(domain=domain,- concept_string_or_code=blueprint.output,+ concept_string_or_code=output_parse_result.concept,concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,)output_concept_domain, output_concept_code = output_domain_and_code.domain, output_domain_and_code.concept_codediff --git a/pipelex/pipe_run/dry_run.py b/pipelex/pipe_run/dry_run.pyindex c2ca25716..d9459627b 100644--- a/pipelex/pipe_run/dry_run.py+++ b/pipelex/pipe_run/dry_run.py@@ -66,7 +66,7 @@ async def dry_run_pipe(pipe: PipeAbstract, raise_on_failure: bool = False) -> Drerror_message = f"Dry run failed for pipe '{pipe.code}': {exc}"return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.FAILURE, error_message=error_message)- log.info(f"Pipe '{pipe.code}' dry run completed successfully")+ log.info(f"✅ Pipe '{pipe.code}' dry run completed successfully")return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.SUCCESS)@@ -123,8 +123,8 @@ async def dry_run_pipes(pipes: list[PipeAbstract], run_in_parallel: bool = True,unexpected_failures = {pipe_code: results[pipe_code] for pipe_code in failed_pipes if pipe_code not in allowed_to_fail_pipes}log.info(- f"Dry run completed: '{len(successful_pipes)}' successful, '{len(failed_pipes)}' failed, "- f"'{len(allowed_to_fail_pipes)}' allowed to fail, in '{time.time() - start_time:.2f}' seconds",+ f"Dry run completed: {len(successful_pipes)} successful, {len(failed_pipes)} failed, "+ f"{len(allowed_to_fail_pipes)} allowed to fail, in {time.time() - start_time:.2f} seconds",)if unexpected_failures:unexpected_failures_details = "\n".join([f"'{pipe_code}': {results[pipe_code]}" for pipe_code in unexpected_failures])@@ -168,7 +168,7 @@ def _convert_to_working_memory_format(needed_inputs_spec: InputRequirements) ->needed_inputs_for_factory.append(typed_named_input_requirement)else:# Fallback to TextContent if we can't get the proper class- log.warning(+ log.verbose(f"Could not get structure class '{structure_class_name}' for "f"concept '{named_input_requirement.concept.code}', falling back to TextContent",)diff --git a/pipelex/pipe_run/pipe_run_params.py b/pipelex/pipe_run/pipe_run_params.pyindex d487c83f6..a5588edbf 100644--- a/pipelex/pipe_run/pipe_run_params.py+++ b/pipelex/pipe_run/pipe_run_params.py@@ -1,11 +1,12 @@from __future__ import annotations-from typing import Any, Union+from typing import Anyfrom pydantic import BaseModel, Field, field_validatorfrom pipelex import logfrom pipelex.core.memory.working_memory import BATCH_ITEM_STUFF_NAME, MAIN_STUFF_NAME+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity, VariableMultiplicityResolutionfrom pipelex.exceptions import BatchParamsErrorfrom pipelex.types import Self, StrEnum@@ -30,58 +31,11 @@ class PipeRunMode(StrEnum):FORCE_DRY_RUN_MODE_ENV_KEY = "PIPELEX_FORCE_DRY_RUN_MODE"-PipeOutputMultiplicity = Union[bool, int]---class OutputMultiplicityResolution(BaseModel):- """Result of resolving output multiplicity settings between base and override values.-- This model provides a clear, structured representation of how output multiplicity- should be applied, replacing the previous tuple-based return format.- """-- resolved_multiplicity: PipeOutputMultiplicity | None = Field(description="The final multiplicity value to use after resolution")-- is_multiple_outputs_enabled: bool = Field(description="Whether multiple outputs should be generated")-- specific_output_count: int | None = Field(default=None, description="Exact number of outputs to generate, if specified")---def make_output_multiplicity(nb_output: int | None, multiple_output: bool | None) -> PipeOutputMultiplicity | None:- """This function takes two mutually exclusive parameters that control how many outputs- a pipe should generate and converts them into a single PipeOutputMultiplicity type.-- Args:- nb_output: Specific number of outputs to generate. If provided and truthy,- takes precedence over multiple_output.- multiple_output: Boolean flag indicating whether to generate multiple outputs.- If True, lets the LLM decide how many outputs to generate.-- Examples:- >>> make_output_multiplicity(nb_output=3, multiple_output=None)- 3- >>> make_output_multiplicity(nb_output=None, multiple_output=True)- True- >>> make_output_multiplicity(nb_output=None, multiple_output=False)- None- >>> make_output_multiplicity(nb_output=0, multiple_output=True)- True-- """- output_multiplicity: PipeOutputMultiplicity | None- if nb_output:- output_multiplicity = nb_output- elif multiple_output:- output_multiplicity = True- else:- output_multiplicity = None- return output_multiplicity-def output_multiplicity_to_apply(- base_multiplicity: PipeOutputMultiplicity | None,- override_multiplicity: PipeOutputMultiplicity | None,-) -> OutputMultiplicityResolution:+ base_multiplicity: VariableMultiplicity | None,+ override_multiplicity: VariableMultiplicity | None,+) -> VariableMultiplicityResolution:"""Resolve output multiplicity settings by combining base configuration with override.This function implements a priority system where override values take precedence over@@ -126,19 +80,19 @@ def output_multiplicity_to_apply(# Case 1: No override provided - use base value as-isif override_multiplicity is None:if isinstance(base_multiplicity, bool):- return OutputMultiplicityResolution(+ return VariableMultiplicityResolution(resolved_multiplicity=base_multiplicity,is_multiple_outputs_enabled=base_multiplicity,specific_output_count=None,)elif isinstance(base_multiplicity, int):- return OutputMultiplicityResolution(+ return VariableMultiplicityResolution(resolved_multiplicity=base_multiplicity,is_multiple_outputs_enabled=True,specific_output_count=base_multiplicity,)else:- return OutputMultiplicityResolution(+ return VariableMultiplicityResolution(resolved_multiplicity=base_multiplicity, is_multiple_outputs_enabled=False, specific_output_count=None)@@ -146,19 +100,19 @@ def output_multiplicity_to_apply(elif isinstance(override_multiplicity, bool):if override_multiplicity:if isinstance(base_multiplicity, bool):- return OutputMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None)+ return VariableMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None)else:- return OutputMultiplicityResolution(+ return VariableMultiplicityResolution(resolved_multiplicity=base_multiplicity,is_multiple_outputs_enabled=True,specific_output_count=base_multiplicity if isinstance(base_multiplicity, int) else None,)else:- return OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None)+ return VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None)else:# Case 3: Override is an integer- return OutputMultiplicityResolution(+ return VariableMultiplicityResolution(resolved_multiplicity=override_multiplicity,is_multiple_outputs_enabled=True,specific_output_count=override_multiplicity,@@ -199,7 +153,7 @@ class PipeRunParams(BaseModel):run_mode: PipeRunMode = PipeRunMode.LIVEfinal_stuff_code: str | None = Noneis_with_preliminary_text: bool | None = None- output_multiplicity: PipeOutputMultiplicity | None = None+ output_multiplicity: VariableMultiplicity | None = Nonedynamic_output_concept_code: str | None = Nonebatch_params: BatchParams | None = Noneparams: dict[str, Any] = Field(default_factory=dict)@@ -230,7 +184,7 @@ class PipeRunParams(BaseModel):def copy_by_injecting_multiplicity(cls,pipe_run_params: Self,- applied_output_multiplicity: PipeOutputMultiplicity | None,+ applied_output_multiplicity: VariableMultiplicity | None,is_with_preliminary_text: bool | None = None,) -> Self:"""Copy the run params the nb_output into the params, and remove the attribute.diff --git a/pipelex/pipe_run/pipe_run_params_factory.py b/pipelex/pipe_run/pipe_run_params_factory.pyindex a1cfe4b03..674c5b0e1 100644--- a/pipelex/pipe_run/pipe_run_params_factory.py+++ b/pipelex/pipe_run/pipe_run_params_factory.py@@ -1,7 +1,8 @@from typing import Anyfrom pipelex.config import get_config-from pipelex.pipe_run.pipe_run_params import BatchParams, PipeOutputMultiplicity, PipeRunMode, PipeRunParams+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity+from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunMode, PipeRunParamsclass PipeRunParamsFactory:@@ -10,7 +11,7 @@ class PipeRunParamsFactory:cls,pipe_run_mode: PipeRunMode = PipeRunMode.LIVE,pipe_stack_limit: int | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,batch_params: BatchParams | None = None,params: dict[str, Any] | None = None,diff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex 277169e41..802515698 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -93,7 +93,7 @@ class Pipelex(metaclass=MetaSingleton):raise PipelexConfigError(msg) from validation_errorlog.configure(log_config=get_config().pipelex.log_config)- log.debug("Logs are configured")+ log.verbose("Logs are configured")# toolsself.class_registry = class_registry or ClassRegistry()@@ -153,7 +153,7 @@ class Pipelex(metaclass=MetaSingleton):self.activity_manager = ActivityManagerNoOp()self.pipelex_hub.set_activity_manager(activity_manager=self.activity_manager)- log.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} init done")+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} init done")@staticmethoddef _get_config_not_found_error_msg(component_name: str) -> str:@@ -241,7 +241,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.self.reporting_delegate.setup()self.class_registry.register_classes(CoreRegistryModels.get_all_models())if runtime_manager.is_unit_testing:- log.debug("Registering test models for unit testing")+ log.verbose("Registering test models for unit testing")self.class_registry.register_classes(TestRegistryModels.get_all_models())self.activity_manager.setup()@@ -254,12 +254,12 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.self.pipeline_tracker.setup()self.pipeline_manager.setup()- log.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup done")+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup done")def setup_libraries(self):self.library_manager.setup()self.library_manager.load_libraries()- log.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup libraries done")+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup libraries done")def validate_libraries(self):try:@@ -268,7 +268,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)msg = f"Could not validate libraries because of: {validation_error_msg}"raise PipelexSetupError(msg) from validation_error- log.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} validate libraries done")+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} validate libraries done")def teardown(self):# pipelex@@ -287,7 +287,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.self.class_registry.teardown()func_registry.teardown()- log.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} teardown done (except config & logs)")+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} teardown done (except config & logs)")self.pipelex_hub.reset_config()# Clear the singleton instance from metaclassif self.__class__ in MetaSingleton.instances:@@ -312,6 +312,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.pipelex_instance = cls()pipelex_instance.setup()pipelex_instance.setup_libraries()+ log.info(f"{PACKAGE_NAME} version {PACKAGE_VERSION} ready")return pipelex_instance@classmethoddiff --git a/pipelex/pipeline/activity/activity_manager.py b/pipelex/pipeline/activity/activity_manager.pyindex 494d7a188..1a0a2a044 100644--- a/pipelex/pipeline/activity/activity_manager.py+++ b/pipelex/pipeline/activity/activity_manager.py@@ -22,7 +22,7 @@ class ActivityManager(ActivityManagerProtocol):@overridedef teardown(self) -> None:self._reset()- log.debug("ActivityManager teardown done")+ log.verbose("ActivityManager teardown done")def _reset(self):self.activity_callbacks = {}@@ -40,5 +40,5 @@ class ActivityManager(ActivityManagerProtocol):@overridedef dispatch_activity(self, activity_report: ActivityReport):for key, callback in self.activity_callbacks.items():- log.dev(f"Dispatching activity to callback '{key}'")+ log.verbose(f"Dispatching activity to callback '{key}'")callback(activity_report)diff --git a/pipelex/pipeline/execute.py b/pipelex/pipeline/execute.pyindex 5a241f60b..2e6b86e33 100644--- a/pipelex/pipeline/execute.py+++ b/pipelex/pipeline/execute.py@@ -11,8 +11,8 @@ from pipelex.hub import (from pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import (FORCE_DRY_RUN_MODE_ENV_KEY,- PipeOutputMultiplicity,PipeRunMode,+ VariableMultiplicity,)from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadata@@ -24,7 +24,7 @@ async def execute_pipeline(inputs: PipelineInputs | WorkingMemory | None = None,search_domains: list[str] | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,pipe_run_mode: PipeRunMode | None = None,) -> PipeOutput:diff --git a/pipelex/pipeline/start.py b/pipelex/pipeline/start.pyindex 0438e51fc..18ba885d5 100644--- a/pipelex/pipeline/start.py+++ b/pipelex/pipeline/start.py@@ -6,7 +6,7 @@ from pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.hub import get_pipe_router, get_pipeline_manager, get_report_delegate, get_required_pipefrom pipelex.pipe_run.pipe_job_factory import PipeJobFactory-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity, PipeRunMode+from pipelex.pipe_run.pipe_run_params import PipeRunMode, VariableMultiplicityfrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadata@@ -15,7 +15,7 @@ async def start_pipeline(pipe_code: str,inputs: PipelineInputs | WorkingMemory | None = None,output_name: str | None = None,- output_multiplicity: PipeOutputMultiplicity | None = None,+ output_multiplicity: VariableMultiplicity | None = None,dynamic_output_concept_code: str | None = None,pipe_run_mode: PipeRunMode = PipeRunMode.LIVE,) -> tuple[str, asyncio.Task[PipeOutput]]:diff --git a/pipelex/pipeline/track/flow_chart.py b/pipelex/pipeline/track/flow_chart.pyindex 77e145114..e34c25295 100644--- a/pipelex/pipeline/track/flow_chart.py+++ b/pipelex/pipeline/track/flow_chart.py@@ -52,7 +52,7 @@ class PipelineFlowChart:if nb_nodes == 0:msg = "Graph has no nodes"raise JobHistoryError(msg)- log.debug(f"Generating mermaid flowchart for the whole graph which holds {nb_nodes} nodes")+ log.verbose(f"Generating mermaid flowchart for the whole graph which holds {nb_nodes} nodes")mermaid_settings: dict[str, Any] = {}if title:mermaid_settings["title"] = title@@ -158,7 +158,7 @@ class PipelineFlowChart:for cycle, (subgraph_name, nodes) in enumerate(graph_tree.nodes_by_subgraph.items()):node_lines: list[str] = []for node in nodes:- # log.debug(f"generate_subgraph_lines for node '{node}'")+ # log.verbose(f"generate_subgraph_lines for node '{node}'")node_attributes = self.nx_graph.nodes[node]if not node_attributes:msg = f"Node attributes are empty for node '{node}'"diff --git a/pipelex/pipeline/track/pipeline_tracker.py b/pipelex/pipeline/track/pipeline_tracker.pyindex 9ae145f36..20f908d5d 100644--- a/pipelex/pipeline/track/pipeline_tracker.py+++ b/pipelex/pipeline/track/pipeline_tracker.py@@ -77,7 +77,7 @@ class PipelineTracker(PipelineTrackerProtocol):as_item_index: int | None = None,) -> str:concept_display = Concept.sentence_from_concept(concept=stuff.concept)- log.debug(f"Concept display: {stuff.concept.code} -> {concept_display}")+ log.verbose(f"Concept display: {stuff.concept.code} -> {concept_display}")if stuff.is_list:concept_display = f"List of [{concept_display}]"if as_item_index is not None:@@ -328,7 +328,7 @@ class PipelineTracker(PipelineTrackerProtocol):def _print_mermaid_flowchart_code_and_url(self, title: str | None = None, subtitle: str | None = None):if not self.nx_graph.nodes:- log.debug("No nodes in the pipeline tracker")+ log.verbose("No nodes in the pipeline tracker")returnif self.start_node is None:msg = "Start node is not set"@@ -343,7 +343,7 @@ class PipelineTracker(PipelineTrackerProtocol):def _print_mermaid_flowchart_url(self, title: str | None = None, subtitle: str | None = None) -> str | None:if not self.nx_graph.nodes:- log.debug("No nodes in the pipeline tracker")+ log.verbose("No nodes in the pipeline tracker")return Noneif self.start_node is None:msg = "Start node is not set"diff --git a/pipelex/plugins/anthropic/anthropic_factory.py b/pipelex/plugins/anthropic/anthropic_factory.pyindex a59d76f56..5875b9f60 100644--- a/pipelex/plugins/anthropic/anthropic_factory.py+++ b/pipelex/plugins/anthropic/anthropic_factory.py@@ -130,7 +130,7 @@ class AnthropicFactory:text_block_param: TextBlockParam = {"type": "text", "text": user_content_txt}message: MessageParamif prepped_user_images is not None:- log.debug(prepped_user_images)+ log.verbose(prepped_user_images)images_block_params: list[ImageBlockParam] = []for prepped_image in prepped_user_images:image_block_param_in_loop: ImageBlockParamdiff --git a/pipelex/plugins/anthropic/anthropic_llm_worker.py b/pipelex/plugins/anthropic/anthropic_llm_worker.pyindex 760bda121..4d7b2e6ea 100644--- a/pipelex/plugins/anthropic/anthropic_llm_worker.py+++ b/pipelex/plugins/anthropic/anthropic_llm_worker.py@@ -81,7 +81,7 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):self.anthropic_async_client = sdk_instanceif structure_method:instructor_mode = structure_method.as_instructor_mode()- log.debug(f"Anthropic structure mode: {structure_method} --> {instructor_mode}")+ log.verbose(f"Anthropic structure mode: {structure_method} --> {instructor_mode}")self.instructor_for_objects = instructor.from_anthropic(client=sdk_instance, mode=instructor_mode)else:self.instructor_for_objects = instructor.from_anthropic(client=sdk_instance)diff --git a/pipelex/plugins/bedrock/bedrock_client_boto3.py b/pipelex/plugins/bedrock/bedrock_client_boto3.pyindex b47c62a9a..a97cd892f 100644--- a/pipelex/plugins/bedrock/bedrock_client_boto3.py+++ b/pipelex/plugins/bedrock/bedrock_client_boto3.py@@ -12,7 +12,7 @@ from pipelex.plugins.bedrock.bedrock_message import BedrockMessageDictListclass BedrockClientBoto3(BedrockClientProtocol):def __init__(self, aws_region: str):- log.debug(f"Initializing BedrockClientBoto3 with region '{aws_region}'")+ log.verbose(f"Initializing BedrockClientBoto3 with region '{aws_region}'")self.boto3_client = boto3.client(service_name="bedrock-runtime", region_name=aws_region) # pyright: ignore[reportUnknownMemberType]@overridediff --git a/pipelex/plugins/bedrock/bedrock_llm_worker.py b/pipelex/plugins/bedrock/bedrock_llm_worker.pyindex 5098dbee7..ba4dd80ed 100644--- a/pipelex/plugins/bedrock/bedrock_llm_worker.py+++ b/pipelex/plugins/bedrock/bedrock_llm_worker.py@@ -51,7 +51,7 @@ class BedrockLLMWorker(LLMWorkerInternalAbstract):) -> str:message = BedrockFactory.make_simple_message(llm_job=llm_job)- log.debug(self.inference_model.model_id)+ log.verbose(self.inference_model.model_id)bedrock_response_text, nb_tokens_by_category = await self.bedrock_client_for_text.chat(messages=message.to_dict_list(),diff --git a/pipelex/plugins/fal/fal_img_gen_worker.py b/pipelex/plugins/fal/fal_img_gen_worker.pyindex cf023d174..b8a1db9ef 100644--- a/pipelex/plugins/fal/fal_img_gen_worker.py+++ b/pipelex/plugins/fal/fal_img_gen_worker.py@@ -84,7 +84,7 @@ class FalImgGenWorker(ImgGenWorkerAbstract):continuenew_fal_logs = event.logs[log_index:]for fal_log in new_fal_logs:- log.debug(fal_log["message"], title="FAL Log")+ log.verbose(fal_log["message"], title="FAL Log")log_index = len(event.logs)fal_result = await handler.get()diff --git a/pipelex/plugins/google/google_llm_worker.py b/pipelex/plugins/google/google_llm_worker.pyindex 30adf36b0..4623b4e04 100644--- a/pipelex/plugins/google/google_llm_worker.py+++ b/pipelex/plugins/google/google_llm_worker.py@@ -43,7 +43,7 @@ class GoogleLLMWorker(LLMWorkerInternalAbstract):self.genai_async_client = genai_client.aioif structure_method:instructor_mode = structure_method.as_instructor_mode()- log.debug(f"Google structure mode: {structure_method} --> {instructor_mode}")+ log.verbose(f"Google structure mode: {structure_method} --> {instructor_mode}")self.instructor_for_objects = instructor.from_genai(client=sdk_instance, mode=instructor_mode, use_async=True)else:self.instructor_for_objects = instructor.from_genai(client=sdk_instance, use_async=True)diff --git a/pipelex/plugins/mistral/mistral_extract_worker.py b/pipelex/plugins/mistral/mistral_extract_worker.pyindex f3bf088f2..2aabe2d05 100644--- a/pipelex/plugins/mistral/mistral_extract_worker.py+++ b/pipelex/plugins/mistral/mistral_extract_worker.py@@ -90,7 +90,7 @@ class MistralExtractWorker(ExtractWorkerAbstract):msg = "Captioning is not implemented for Mistral OCR."raise ExtractCapabilityError(msg)if should_include_page_views:- log.debug("Page views are not implemented for Mistral OCR.")+ log.verbose("Page views are not implemented for Mistral OCR.")# TODO: use a model capability flag to check possibility before asking for it# it it's asked and not available, raise# the caller will be responsible to get the page views using other solution if neededdiff --git a/pipelex/plugins/mistral/mistral_llm_worker.py b/pipelex/plugins/mistral/mistral_llm_worker.pyindex ef4160c3f..a21e9d0a2 100644--- a/pipelex/plugins/mistral/mistral_llm_worker.py+++ b/pipelex/plugins/mistral/mistral_llm_worker.py@@ -47,7 +47,7 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):if structure_method:instructor_mode = structure_method.as_instructor_mode()- log.debug(f"Mistral structure mode: {structure_method} --> {instructor_mode}")+ log.verbose(f"Mistral structure mode: {structure_method} --> {instructor_mode}")self.instructor_for_objects = instructor.from_mistral(client=sdk_instance, mode=instructor_mode, use_async=True)else:self.instructor_for_objects = instructor.from_mistral(client=sdk_instance, use_async=True)diff --git a/pipelex/plugins/openai/openai_factory.py b/pipelex/plugins/openai/openai_factory.pyindex fe28215e8..c40ffcab0 100644--- a/pipelex/plugins/openai/openai_factory.py+++ b/pipelex/plugins/openai/openai_factory.py@@ -50,7 +50,7 @@ class OpenAIFactory:the_client: openai.AsyncOpenAImatch sdk_variant:case OpenAISdkVariant.AZURE_OPENAI:- log.debug(f"Making AsyncOpenAI client with endpoint: {backend.endpoint}")+ log.verbose(f"Making AsyncOpenAI client with endpoint: {backend.endpoint}")if backend.endpoint is None:msg = "Azure OpenAI endpoint is not set"raise OpenAIFactoryError(msg)@@ -61,7 +61,7 @@ class OpenAIFactory:)case OpenAISdkVariant.OPENAI:- log.debug(f"Making AsyncOpenAI client with endpoint: {backend.endpoint}")+ log.verbose(f"Making AsyncOpenAI client with endpoint: {backend.endpoint}")the_client = openai.AsyncOpenAI(api_key=backend.api_key,base_url=backend.endpoint,@@ -132,7 +132,7 @@ class OpenAIFactory:@staticmethoddef make_logit_bias(nb_items: int, weight: int = 100) -> dict[str, int]:logit_bias = {str(item): weight for item in range(15, 15 + nb_items + 1)}- log.debug(f"logit_bias: {logit_bias}")+ log.verbose(f"logit_bias: {logit_bias}")return logit_bias@staticmethoddiff --git a/pipelex/plugins/openai/openai_img_gen_worker.py b/pipelex/plugins/openai/openai_img_gen_worker.pyindex 66c7f627d..118733aee 100644--- a/pipelex/plugins/openai/openai_img_gen_worker.py+++ b/pipelex/plugins/openai/openai_img_gen_worker.py@@ -81,7 +81,7 @@ class OpenAIImgGenWorker(ImgGenWorkerAbstract):ensure_path(folder_path)img_path = f"{folder_path}/{image_id}_{image_index}.png"save_base_64_str_to_binary_file(base_64_str=image_base64, file_path=img_path)- log.debug(f"Saved image to {img_path}")+ log.verbose(f"Saved image to {img_path}")generated_image_list.append(GeneratedImage(url=img_path,diff --git a/pipelex/plugins/openai/openai_llm_worker.py b/pipelex/plugins/openai/openai_llm_worker.pyindex 706919790..f662192ba 100644--- a/pipelex/plugins/openai/openai_llm_worker.py+++ b/pipelex/plugins/openai/openai_llm_worker.py@@ -45,7 +45,7 @@ class OpenAILLMWorker(LLMWorkerInternalAbstract):self.openai_client_for_text: openai.AsyncOpenAI = sdk_instanceif structure_method:instructor_mode = structure_method.as_instructor_mode()- log.debug(f"OpenAI structure mode: {structure_method} --> {instructor_mode}")+ log.verbose(f"OpenAI structure mode: {structure_method} --> {instructor_mode}")self.instructor_for_objects = instructor.from_openai(client=sdk_instance, mode=instructor_mode)else:self.instructor_for_objects = instructor.from_openai(client=sdk_instance)diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 35eab2a47..0ed64582f 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -85,7 +85,7 @@ class ReportingManager(ReportingProtocol):@overridedef report_inference_job(self, inference_job: InferenceJobAbstract):- log.debug(f"Inference job '{inference_job.job_metadata.unit_job_id}' completed in {inference_job.job_metadata.duration:.2f} seconds")+ log.verbose(f"Inference job '{inference_job.job_metadata.unit_job_id}' completed in {inference_job.job_metadata.duration:.2f} seconds")if not isinstance(inference_job, LLMJob):# ReportingManager does not support reporting for other types of inference jobs yet# TODO: add support for other types of inference jobsdiff --git a/pipelex/system/registries/class_registry_utils.py b/pipelex/system/registries/class_registry_utils.pyindex c81a118a5..37c889fa1 100644--- a/pipelex/system/registries/class_registry_utils.py+++ b/pipelex/system/registries/class_registry_utils.py@@ -139,11 +139,11 @@ class ClassRegistryUtils:import_module_from_file(str(python_file))except ModuleFileError:# Expected: file validation issues (directories with .py extension, etc.)- # log.debug(f"Skipping file {python_file}: {e}")+ # log.verbose(f"Skipping file {python_file}: {e}")passexcept ImportError:# Common: missing dependencies, circular imports, relative imports- # log.debug(f"Could not import {python_file}: {e}"+ # log.verbose(f"Could not import {python_file}: {e}"passexcept SyntaxError as exc:# Potentially problematic: invalid Python syntax may indicate broken codediff --git a/pipelex/tools/aws/aws_config.py b/pipelex/tools/aws/aws_config.pyindex cf591cc12..9b61e0017 100644--- a/pipelex/tools/aws/aws_config.py+++ b/pipelex/tools/aws/aws_config.py@@ -32,7 +32,7 @@ class AwsConfig(ConfigModel):def get_aws_access_keys_with_method(self, api_key_method: AwsKeyMethod) -> tuple[str, str, str]:match api_key_method:case AwsKeyMethod.ENV:- log.debug("Getting AWS access keys from environment (key id and secret access key).")+ log.verbose("Getting AWS access keys from environment (key id and secret access key).")try:aws_access_key_id = get_required_env(AWS_ACCESS_KEY_ID_VAR_NAME)aws_secret_access_key = get_required_env(AWS_SECRET_ACCESS_KEY_VAR_NAME)@@ -40,10 +40,10 @@ class AwsConfig(ConfigModel):except EnvVarNotFoundError as exc:msg = f"Error getting AWS access keys from environment: {exc}"raise AwsCredentialsError(msg) from exc- log.debug("Getting AWS region from environment (priority override) or from aws_config.")+ log.verbose("Getting AWS region from environment (priority override) or from aws_config.")case AwsKeyMethod.SECRET_PROVIDER:- log.debug("Getting AWS secret access key from secrets provider (key id and secret access key).")+ log.verbose("Getting AWS secret access key from secrets provider (key id and secret access key).")try:aws_access_key_id = get_secret(AWS_ACCESS_KEY_ID_VAR_NAME)aws_secret_access_key = get_secret(AWS_SECRET_ACCESS_KEY_VAR_NAME)@@ -51,6 +51,6 @@ class AwsConfig(ConfigModel):except SecretNotFoundError as exc:msg = "Error getting AWS access keys from secrets provider."raise AwsCredentialsError(msg) from exc- log.debug("Getting AWS region from environment (priority override) or from aws_config.")+ log.verbose("Getting AWS region from environment (priority override) or from aws_config.")return aws_access_key_id, aws_secret_access_key, aws_regiondiff --git a/pipelex/tools/misc/context_provider_abstract.py b/pipelex/tools/misc/context_provider_abstract.pyindex 84e8a8608..259e66567 100644--- a/pipelex/tools/misc/context_provider_abstract.py+++ b/pipelex/tools/misc/context_provider_abstract.py@@ -16,7 +16,7 @@ class ContextProviderAbstract(ABC):"""@abstractmethod- def get_typed_object_or_attribute(self, name: str, wanted_type: type[Any] | None = None) -> Any:+ def get_typed_object_or_attribute(self, name: str, wanted_type: type[Any] | None = None, accept_list: bool = False) -> Any:pass@abstractmethoddiff --git a/pipelex/tools/misc/filetype_utils.py b/pipelex/tools/misc/filetype_utils.pyindex a1659e4b9..7597bc459 100644--- a/pipelex/tools/misc/filetype_utils.py+++ b/pipelex/tools/misc/filetype_utils.py@@ -77,13 +77,13 @@ def detect_file_type_from_base64(b64: str | bytes) -> FileType:"""# Normalise to bytes holding only the Base-64 alphabetif isinstance(b64, bytes):- log.debug(f"b64 is already bytes: {b64[:100]!r}")+ log.verbose(f"b64 is already bytes: {b64[:100]!r}")b64_bytes = b64else: # str → handle optional data-URL header- log.debug(f"b64 is a string: {b64[:100]!r}")+ log.verbose(f"b64 is a string: {b64[:100]!r}")if b64.lstrip().startswith("data:") and "," in b64:b64 = b64.split(",", 1)[1]- log.debug(f"b64 after split: {b64[:100]!r}")+ log.verbose(f"b64 after split: {b64[:100]!r}")b64_bytes = b64.encode("ascii") # Base-64 is pure ASCIItry:diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.pyindex 8be49dba4..e24ba958e 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py@@ -7,7 +7,6 @@ from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.domains.domain import SpecialDomainfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.input_requirements import TypedNamedInputRequirementfrom pipelex.exceptions import DryRunErrorfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint@@ -27,7 +26,7 @@ class TestPipeConditionSimple:# Create a PipeCondition directly in Python that requires an inputpipe_condition_blueprint = PipeConditionBlueprint(description="Test condition that should fail",- inputs={"user_category": InputRequirementBlueprint(concept="test_pipe_condition.CategoryInput")},+ inputs={"user_category": "test_pipe_condition.CategoryInput"},output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",expression_template="{{ user_category.category }}",outcomes={"small": "process_small", "medium": "process_medium", "large": "process_large"},@@ -61,7 +60,7 @@ class TestPipeConditionSimple:# Create a PipeCondition directly in Pythonpipe_condition_blueprint = PipeConditionBlueprint(description="Test condition that should succeed",- inputs={"user_status": InputRequirementBlueprint(concept="test_pipe_condition.CategoryInput")},+ inputs={"user_status": "test_pipe_condition.CategoryInput"},output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",expression_template="{{ user_status.category }}",outcomes={"active": "process_small", "inactive": "process_medium", "pending": "process_large"},diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.pyindex 579f92ab6..3c8a6d066 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py@@ -76,7 +76,7 @@ async def test_review_analysis_sequence_with_batching(pipe_run_mode: PipeRunModeassert pipe_output.main_stuff.concept.domain == "customer_feedback"# Log the working memory for debugging- log.debug("Final working memory after pipeline execution:")+ log.verbose("Final working memory after pipeline execution:")pipe_output.working_memory.pretty_print_summary()# Verify final product ratingdiff --git a/tests/integration/pipelex/conftest.py b/tests/integration/pipelex/conftest.pyindex dea6ff5ce..2d2aaf225 100644--- a/tests/integration/pipelex/conftest.py+++ b/tests/integration/pipelex/conftest.py@@ -227,7 +227,7 @@ def extract_handle_from_image(request: pytest.FixtureRequest) -> str:@pytest.fixture(params=[- "base_ocr_mistral",+ "extract_text_from_visuals",],)def extract_choice_for_pdf(request: pytest.FixtureRequest) -> str:@@ -237,7 +237,7 @@ def extract_choice_for_pdf(request: pytest.FixtureRequest) -> str:@pytest.fixture(params=[- "base_ocr_mistral",+ "extract_text_from_visuals",],)def extract_choice_for_image(request: pytest.FixtureRequest) -> str:diff --git a/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py b/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.pyindex 137d430ec..5888f70d3 100644--- a/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py+++ b/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py@@ -9,7 +9,6 @@ from pipelex import pretty_printfrom pipelex.core.concepts.concept_factory import ConceptBlueprint, ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.stuff_factory import StuffFactory@@ -51,8 +50,8 @@ class TestPipeBatchSimple:description="Simple batch processing test",branch_pipe_code="uppercase_transformer", # This exists in the PLX fileinputs={- "text_list": InputRequirementBlueprint(concept=concept_1.concept_string),- "text_item": InputRequirementBlueprint(concept=concept_2.concept_string),+ "text_list": concept_1.concept_string,+ "text_item": concept_2.concept_string,},output=concept_2.concept_string,input_list_name="text_list",diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.pyindex 52a54ac56..307118e67 100644--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py@@ -10,7 +10,6 @@ from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.domains.domain import SpecialDomainfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.exceptions import DryRunError@@ -33,7 +32,7 @@ class TestPipeConditionSimple:"""Test PipeCondition with long text that should trigger capitalize_long_text pipe."""pipe_condition_blueprint = PipeConditionBlueprint(description="Text length condition for testing",- inputs={"input_text": InputRequirementBlueprint(concept=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}")},+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",expression_template="{% if input_text.text|length > 5 %}long{% else %}short{% endif %}",outcomes={"long": "capitalize_long_text", "short": "add_prefix_short_text"},@@ -103,7 +102,7 @@ class TestPipeConditionSimple:# Create PipeCondition instance - pipes are loaded from PLX filespipe_condition_blueprint = PipeConditionBlueprint(description="Text length condition for short text testing",- inputs={"input_text": InputRequirementBlueprint(concept=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}")},+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",expression_template="{% if input_text.text|length > 5 %}long{% else %}short{% endif %}",outcomes={"long": "capitalize_long_text", "short": "add_prefix_short_text"},diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.pyindex d73e169ae..ec5f9a9fb 100644--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py@@ -1,5 +1,4 @@from pipelex.core.concepts.concept_factory import ConceptBlueprint, ConceptFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.hub import get_concept_libraryfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprintfrom pipelex.pipe_controllers.condition.pipe_condition_factory import PipeConditionFactory@@ -29,7 +28,7 @@ class TestPipeConditionValidation:pipe_condition_blueprint = PipeConditionBlueprint(description="Test condition for validation",- inputs={"input_var": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"input_var": concept_1.concept_string},output=concept_2.concept_string,expression="input_var",outcomes={"value1": "pipe_a", "value2": "pipe_b"},@@ -71,7 +70,7 @@ class TestPipeConditionValidation:pipe_condition_template_blueprint = PipeConditionBlueprint(description="Test condition with expression template",- inputs={"var": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"var": concept_1.concept_string},output=concept_2.concept_string,expression_template="{{ var }}",outcomes={"value": "target_pipe"},@@ -87,7 +86,7 @@ class TestPipeConditionValidation:# Test with expressionpipe_condition_expr_blueprint = PipeConditionBlueprint(description="Test condition with expression",- inputs={"var": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"var": concept_1.concept_string},output=concept_2.concept_string,expression="var",outcomes={"value": "target_pipe"},diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.pyindex 26dda49ad..407af4bb5 100644--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py@@ -7,7 +7,6 @@ from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.domains.domain import SpecialDomainfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint@@ -27,7 +26,7 @@ class TestPipeParallelSimple:# Create PipeParallel instance - pipes are loaded from PLX filespipe_parallel_blueprint = PipeParallelBlueprint(description="Parallel text analysis pipeline",- inputs={"input_text": InputRequirementBlueprint(concept=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}")},+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",parallels=[SubPipeBlueprint(pipe="analyze_sentiment", result="sentiment_result"),@@ -143,7 +142,7 @@ class TestPipeParallelSimple:# Create PipeParallel instancepipe_parallel_blueprint = PipeParallelBlueprint(description="Parallel text analysis pipeline for short text",- inputs={"input_text": InputRequirementBlueprint(concept=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}")},+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",parallels=[SubPipeBlueprint(pipe="analyze_sentiment", result="sentiment_result"),diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.pyindex e7c07751d..ab098d8ee 100644--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py@@ -1,6 +1,5 @@from pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.input_requirements import InputRequirementsfrom pipelex.hub import get_concept_library, get_pipe_libraryfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint@@ -41,8 +40,8 @@ class TestPipeParallelValidation:pipe_llm_blueprint = PipeLLMBlueprint(inputs={- "document": InputRequirementBlueprint(concept=concept_1.concept_string),- "context": InputRequirementBlueprint(concept=concept_2.concept_string),+ "document": concept_1.concept_string,+ "context": concept_2.concept_string,},description="Analysis pipe for document processing",output=ConceptFactory.make_concept_string_with_domain(domain=domain, concept_code=concept_3.code),@@ -67,8 +66,8 @@ class TestPipeParallelValidation:pipe_parallel_blueprint = PipeParallelBlueprint(description="Parallel document processor for testing",inputs={- "document": InputRequirementBlueprint(concept=concept_1.code),- "context": InputRequirementBlueprint(concept=concept_2.code),+ "document": concept_1.code,+ "context": concept_2.code,},output=ConceptFactory.make_concept_string_with_domain(domain=domain, concept_code=concept_3.code),parallels=[SubPipeBlueprint(pipe=real_pipe.code, result="analysis_result")],@@ -122,7 +121,7 @@ class TestPipeParallelValidation:pipe_parallel_blueprint = PipeParallelBlueprint(description="Basic parallel pipe for testing",- inputs={"input_var": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"input_var": concept_1.concept_string},output=ConceptFactory.make_concept_string_with_domain(domain=domain, concept_code=concept_3.code),parallels=[SubPipeBlueprint(pipe="test_pipe_1", result="result_1")],add_each_output=True,@@ -176,8 +175,8 @@ class TestPipeParallelValidation:pipe_parallel_blueprint = PipeParallelBlueprint(description="Parallel processor for testing inputs structure",inputs={- "document": InputRequirementBlueprint(concept=concept_1.concept_string),- "context": InputRequirementBlueprint(concept=concept_2.concept_string),+ "document": concept_1.concept_string,+ "context": concept_2.concept_string,},output=ConceptFactory.make_concept_string_with_domain(domain=domain, concept_code=concept_3.code),parallels=[], # No sub-pipes to avoid dependency issuesdiff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.pyindex 0b6e8cb95..17cb30c39 100644--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py@@ -10,7 +10,6 @@ from pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.hub import get_concept_library, get_native_concept@@ -44,7 +43,7 @@ class TestPipeSequenceSimple:# Create PipeSequence instance - pipes are loaded from PLX filespipe_sequence_blueprint = PipeSequenceBlueprint(description="Simple sequence for text processing",- inputs={"input_text": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"input_text": concept_1.concept_string},output=concept_2.concept_string,steps=[SubPipeBlueprint(pipe="capitalize_text", result="capitalized_text"),diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.pyindex cedb8af63..4314130f0 100644--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py@@ -1,5 +1,4 @@from pipelex.core.concepts.concept_factory import ConceptBlueprint, ConceptFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.hub import get_concept_libraryfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprintfrom pipelex.pipe_controllers.sequence.pipe_sequence_factory import PipeSequenceFactory@@ -29,7 +28,7 @@ class TestPipeSequenceValidation:pipe_sequence_blueprint = PipeSequenceBlueprint(description="Test sequence for validation",- inputs={"text": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"text": concept_1.concept_string},output=concept_2.concept_string,steps=[SubPipeBlueprint(pipe="test_pipe_1", result="intermediate_result")],)@@ -68,7 +67,7 @@ class TestPipeSequenceValidation:pipe_sequence_blueprint = PipeSequenceBlueprint(description="Test sequence with multiple steps",- inputs={"initial_input": InputRequirementBlueprint(concept=concept_1.concept_string)},+ inputs={"initial_input": concept_1.concept_string},output=concept_2.concept_string,steps=[SubPipeBlueprint(pipe="step_1", result="intermediate"), SubPipeBlueprint(pipe="step_2", result="final_output")],)diff --git a/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py b/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.pyindex 4f1c0cf61..e616ad175 100644--- a/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py+++ b/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py@@ -5,7 +5,6 @@ from pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.stuffs.page_content import PageContentfrom pipelex.hub import get_concept_library, get_pipe_routerfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint@@ -46,7 +45,7 @@ class TestPipeExtract:):pipe_extract_blueprint = PipeExtractBlueprint(description="OCR test for image processing",- inputs={"page_scan": InputRequirementBlueprint(concept=NativeConceptCode.IMAGE)},+ inputs={"page_scan": NativeConceptCode.IMAGE},output=NativeConceptCode.TEXT_AND_IMAGES,page_images=True,page_image_captions=False,@@ -84,7 +83,7 @@ class TestPipeExtract:input_name = "arbitrary_name"blueprint = PipeExtractBlueprint(description="OCR test for PDF processing",- inputs={input_name: InputRequirementBlueprint(concept=NativeConceptCode.PDF)},+ inputs={input_name: NativeConceptCode.PDF},output=NativeConceptCode.TEXT_AND_IMAGES,model=extract_choice_for_pdf,page_images=True,diff --git a/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen.py b/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen.pyindex 7e70a6cfd..2ff69e129 100644--- a/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen.py+++ b/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen.py@@ -33,7 +33,6 @@ class TestPipeImgGen:description="Image generation test",img_gen_prompt=image_desc,output=NativeConceptCode.IMAGE,- nb_output=1,)pipe_job = PipeJobFactory.make_pipe_job(diff --git a/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py b/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.pyindex ee6d5019f..2c7c40ef2 100644--- a/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py+++ b/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py@@ -3,7 +3,6 @@ import pytestfrom pipelex import log, pretty_printfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.stuffs.stuff import Stufffrom pipelex.hub import get_class_registry, get_pipe_library, get_pipe_routerfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint, StructuringMethod@@ -145,7 +144,7 @@ class TestPipeLLM:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)pipe_llm_blueprint = PipeLLMBlueprint(description="LLM test for image processing with attributes",- inputs={stuff_name: InputRequirementBlueprint(concept=stuff.concept.concept_string)},+ inputs={stuff_name: stuff.concept.concept_string},output=NativeConceptCode.TEXT,system_prompt=PipeTestCases.SYSTEM_PROMPT,prompt=PipeTestCases.MULTI_IMG_DESC_PROMPT,diff --git a/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py b/tests/integration/pipelex/pipes/test_bracket_notation_controllers.pynew file mode 100644index 000000000..62a42a27f--- /dev/null+++ b/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py@@ -0,0 +1,120 @@+"""Integration tests for bracket notation in controller pipe factories."""++from pipelex.core.concepts.concept_blueprint import ConceptBlueprint+from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.hub import get_concept_library+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint+from pipelex.pipe_controllers.batch.pipe_batch_factory import PipeBatchFactory+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint+from pipelex.pipe_controllers.condition.pipe_condition_factory import PipeConditionFactory+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.parallel.pipe_parallel_factory import PipeParallelFactory+++class TestBracketNotationInControllers:+ """Test that controller factories correctly handle bracket notation in inputs and outputs."""++ def test_pipe_parallel_with_bracket_notation(self):+ """Test PipeParallel factory with bracket notation."""+ domain = "test"+ concept_library = get_concept_library()++ concept_1 = ConceptFactory.make_from_blueprint(+ concept_code="DataItem",+ domain=domain,+ blueprint=ConceptBlueprint(description="Data item"),+ concept_codes_from_the_same_domain=["DataItem"],+ )+ concept_library.add_concepts([concept_1])++ blueprint = PipeParallelBlueprint(+ description="Process items in parallel",+ inputs={"data": "DataItem[2]"},+ output="ProcessedData",+ parallels=[],+ add_each_output=True,+ )++ pipe = PipeParallelFactory.make_from_blueprint(+ domain=domain,+ pipe_code="test_parallel",+ blueprint=blueprint,+ )++ assert pipe.inputs.root["data"].multiplicity == 2+ assert pipe.output.code == "ProcessedData"++ concept_library.teardown()++ def test_pipe_condition_with_bracket_notation(self):+ """Test PipeCondition factory with bracket notation."""+ domain = "test"+ concept_library = get_concept_library()++ concept_1 = ConceptFactory.make_from_blueprint(+ concept_code="Category",+ domain=domain,+ blueprint=ConceptBlueprint(description="Category"),+ concept_codes_from_the_same_domain=["Category"],+ )+ concept_2 = ConceptFactory.make_from_blueprint(+ concept_code="Result",+ domain=domain,+ blueprint=ConceptBlueprint(description="Result"),+ concept_codes_from_the_same_domain=["Result"],+ )+ concept_library.add_concepts([concept_1, concept_2])++ blueprint = PipeConditionBlueprint(+ description="Route based on category",+ inputs={"items": "Category[]"},+ output="Result",+ expression="items",+ outcomes={"A": "pipe_a"},+ default_outcome=SpecialOutcome.CONTINUE,+ )++ pipe = PipeConditionFactory.make_from_blueprint(+ domain=domain,+ pipe_code="test_condition",+ blueprint=blueprint,+ )++ assert pipe.inputs.root["items"].multiplicity is True+ assert pipe.output.code == "Result"++ concept_library.teardown()++ def test_pipe_batch_with_bracket_notation(self):+ """Test PipeBatch factory with bracket notation."""+ domain = "test"+ concept_library = get_concept_library()++ concept_1 = ConceptFactory.make_from_blueprint(+ concept_code="Item",+ domain=domain,+ blueprint=ConceptBlueprint(description="Item"),+ concept_codes_from_the_same_domain=["Item"],+ )+ concept_library.add_concepts([concept_1])++ blueprint = PipeBatchBlueprint(+ description="Batch process items",+ inputs={"items": "Item[]"},+ output="ProcessedItem[]",+ branch_pipe_code="process_single",+ input_list_name="items",+ input_item_name="item",+ )++ pipe = PipeBatchFactory.make_from_blueprint(+ domain=domain,+ pipe_code="test_batch",+ blueprint=blueprint,+ )++ assert pipe.inputs.root["items"].multiplicity is True+ assert pipe.output.code == "ProcessedItem"++ concept_library.teardown()diff --git a/tests/integration/pipelex/pipes/test_bracket_notation_operators.py b/tests/integration/pipelex/pipes/test_bracket_notation_operators.pynew file mode 100644index 000000000..29616378e--- /dev/null+++ b/tests/integration/pipelex/pipes/test_bracket_notation_operators.py@@ -0,0 +1,142 @@+"""Integration tests for bracket notation in operator pipe factories."""++from pipelex.core.concepts.concept_native import NativeConceptCode+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint+from pipelex.pipe_operators.compose.pipe_compose_factory import PipeComposeFactory+from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint+from pipelex.pipe_operators.extract.pipe_extract_factory import PipeExtractFactory+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint+from pipelex.pipe_operators.func.pipe_func_factory import PipeFuncFactory+from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint+from pipelex.pipe_operators.img_gen.pipe_img_gen_factory import PipeImgGenFactory+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint+from pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory+++class TestBracketNotationInOperators:+ """Test that operator factories correctly handle bracket notation in inputs and outputs."""++ def test_pipe_llm_with_bracket_output_variable_list(self):+ """Test PipeLLM factory with variable list output (Text[])."""+ blueprint = PipeLLMBlueprint(+ description="Generate multiple items",+ inputs={"topic": NativeConceptCode.TEXT},+ output=f"{NativeConceptCode.TEXT}[]",+ prompt="Generate items about $topic",+ )++ pipe = PipeLLMFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_llm",+ blueprint=blueprint,+ )++ assert pipe.output.code == "Text"+ assert pipe.output_multiplicity is True++ def test_pipe_llm_with_bracket_output_fixed_count(self):+ """Test PipeLLM factory with fixed count output (Text[5])."""+ blueprint = PipeLLMBlueprint(+ description="Generate exactly 5 items",+ inputs={},+ output=f"{NativeConceptCode.TEXT}[5]",+ prompt="Generate 5 items",+ )++ pipe = PipeLLMFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_llm",+ blueprint=blueprint,+ )++ assert pipe.output.code == "Text"+ assert pipe.output_multiplicity == 5++ def test_pipe_llm_with_bracket_inputs(self):+ """Test PipeLLM factory with bracket notation in inputs."""+ blueprint = PipeLLMBlueprint(+ description="Process multiple documents",+ inputs={"documents": f"{NativeConceptCode.TEXT}[]", "query": NativeConceptCode.TEXT},+ output=NativeConceptCode.TEXT,+ prompt="Summarize @documents based on $query",+ )++ pipe = PipeLLMFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_llm",+ blueprint=blueprint,+ )++ assert "documents" in pipe.inputs.root+ assert "query" in pipe.inputs.root+ assert pipe.inputs.root["documents"].multiplicity is True+ assert pipe.inputs.root["query"].multiplicity is None++ def test_pipe_img_gen_with_bracket_output(self):+ """Test PipeImgGen factory with fixed count output (Image[3])."""+ blueprint = PipeImgGenBlueprint(+ description="Generate 3 images",+ inputs={"prompt": NativeConceptCode.TEXT},+ output=f"{NativeConceptCode.IMAGE}[3]",+ )++ pipe = PipeImgGenFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_img_gen",+ blueprint=blueprint,+ )++ assert pipe.output.code == "Image"+ assert pipe.output_multiplicity == 3++ def test_pipe_func_with_bracket_input_and_output(self):+ """Test PipeFunc factory with bracket notation."""+ blueprint = PipeFuncBlueprint(+ description="Process items",+ inputs={"items": f"{NativeConceptCode.TEXT}[2]"},+ output=f"{NativeConceptCode.TEXT}[]",+ function_name="process_function",+ )++ pipe = PipeFuncFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_func",+ blueprint=blueprint,+ )++ assert pipe.inputs.root["items"].multiplicity == 2+ assert pipe.output.code == "Text"++ def test_pipe_compose_with_bracket_notation(self):+ """Test PipeCompose factory with bracket notation."""+ blueprint = PipeComposeBlueprint(+ description="Compose multiple items",+ inputs={"items": f"{NativeConceptCode.TEXT}[]"},+ output=NativeConceptCode.TEXT,+ template="<ul>{% for item in items %}<li>{{ item }}</li>{% endfor %}</ul>",+ )++ pipe = PipeComposeFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_compose",+ blueprint=blueprint,+ )++ assert pipe.inputs.root["items"].multiplicity is True+ assert pipe.output.code == "Text"++ def test_pipe_extract_with_bracket_output(self):+ """Test PipeExtract factory with bracket notation in output."""+ blueprint = PipeExtractBlueprint(+ description="Extract pages",+ inputs={"document": NativeConceptCode.PDF},+ output=f"{NativeConceptCode.PAGE}[]", # Extract returns list of pages+ )++ pipe = PipeExtractFactory.make_from_blueprint(+ domain="test",+ pipe_code="test_extract",+ blueprint=blueprint,+ )++ assert pipe.output.code == "Page"diff --git a/tests/integration/pipelex/pipes/test_pipe_running_variants.py b/tests/integration/pipelex/pipes/test_pipe_running_variants.pyindex b97a9e599..8e29823e5 100644--- a/tests/integration/pipelex/pipes/test_pipe_running_variants.py+++ b/tests/integration/pipelex/pipes/test_pipe_running_variants.py@@ -5,10 +5,11 @@ from pytest import FixtureRequestfrom pipelex import log, pretty_printfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.stuff import Stufffrom pipelex.hub import get_pipe_router, get_required_pipefrom pipelex.pipe_run.pipe_job_factory import PipeJobFactory-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity, PipeRunMode+from pipelex.pipe_run.pipe_run_params import PipeRunModefrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.activity.activity_handler import ActivityHandlerForResultFilesfrom pipelex.pipeline.job_metadata import JobMetadata@@ -85,7 +86,7 @@ class TestPipeRunningVariants:save_working_memory: Any,topic: str,pipe_code: str,- output_multiplicity: PipeOutputMultiplicity | None,+ output_multiplicity: VariableMultiplicity | None,):log.verbose(f"{topic}: just run pipe '{pipe_code}'")pipe_output = await get_pipe_router().run(diff --git a/tests/integration/pipelex/test_data.py b/tests/integration/pipelex/test_data.pyindex 3f36a8b8b..b1189ed1e 100644--- a/tests/integration/pipelex/test_data.py+++ b/tests/integration/pipelex/test_data.py@@ -2,6 +2,7 @@ from typing import ClassVarfrom pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCode+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.pdf_content import PDFContent@@ -10,7 +11,6 @@ from pipelex.core.stuffs.stuff import Stufffrom pipelex.core.stuffs.stuff_factory import StuffBlueprint, StuffFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.exceptions import PipeStackOverflowError-from pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicityfrom tests.cases import ImageTestCases, PDFTestCases@@ -127,7 +127,7 @@ class PipeTestCases:"test_no_input_that_could_be_long",),]- NO_INPUT_PARALLEL1: ClassVar[list[tuple[str, str, PipeOutputMultiplicity | None]]] = [ # topic, pipe, multiplicity+ NO_INPUT_PARALLEL1: ClassVar[list[tuple[str, str, VariableMultiplicity | None]]] = [ # topic, pipe, multiplicity("Nature colors painting","choose_colors",diff --git a/tests/test_pipelines/misc_tests/multiplicity.plx b/tests/test_pipelines/misc_tests/multiplicity.plxindex 3d0576808..de161e6da 100644--- a/tests/test_pipelines/misc_tests/multiplicity.plx+++ b/tests/test_pipelines/misc_tests/multiplicity.plx@@ -11,12 +11,11 @@ FantasyScene = "A fantasy scene"[pipe.original_power_ranger_colors]type = "PipeLLM"description = "Recall the colors of the original power rangers"-output = "Color"+output = "Color[5]"prompt = """-Remind me of the colors of the {{ _nb_output }} original power rangers.+Remind me of the colors of the 5 original power rangers."""model = "llm_for_creative_writing"-nb_output = 5[pipe.alltime_power_ranger_colors]type = "PipeLLM"@@ -66,7 +65,7 @@ type = "PipeSequence"description = "Imagine nature scenes of Power Rangers colors"output = "ImgGenPrompt"steps = [- { pipe = "alltime_power_ranger_colors", result = "color", multiple_output = true },+ { pipe = "alltime_power_ranger_colors", result = "color" },{ pipe = "imagine_nature_product", result = "product_of_nature" },{ pipe = "imagine_fantasy_scene_including_products_of_nature" },]diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.plx b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.plxindex 37ed5a102..4dfb7c620 100644--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.plx+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.plx@@ -24,8 +24,7 @@ steps = [type = "PipeLLM"description = "Extract individual reviews from document"inputs = { document = "Document" }-output = "CustomerReview"-multiple_output = true+output = "CustomerReview[]"model = "llm_for_testing_gen_text"prompt = """Extract each individual customer review from this document as separate items:diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.plx b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.plxindex 5d9dfb63f..ccd27c0f1 100644--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.plx+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.plx@@ -26,11 +26,10 @@ steps = [type = "PipeLLM"description = "Generate exactly 5 creative ideas for a topic"inputs = { topic = "CreativeTopic" }-output = "CreativeIdea"-nb_output = 5+output = "CreativeIdea[5]"model = "llm_for_creative_writing"prompt = """-Generate $_nb_output creative and innovative ideas for the following topic:+Generate 5 creative and innovative ideas for the following topic:@topic@@ -46,8 +45,7 @@ Ideas:type = "PipeLLM"description = "Brainstorm variable number of concepts"inputs = { topic = "CreativeTopic" }-output = "CreativeIdea"-multiple_output = true+output = "CreativeIdea[]"model = "llm_for_creative_writing"prompt = """Brainstorm creative concepts related to this topic. Generate as many good ideas as come to mind:diff --git a/tests/test_pipelines/tricky_questions.plx b/tests/test_pipelines/tricky_questions.plxindex 393f118ae..70a0c931c 100644--- a/tests/test_pipelines/tricky_questions.plx+++ b/tests/test_pipelines/tricky_questions.plx@@ -16,7 +16,7 @@ type = "PipeLLM"description = "Analyze a question to determine whether it's straightforward or tricky"inputs = { question = "Question" }output = "QuestionAnalysis"-model = "llm_to_reason"+model = "llm_for_complex_reasoning"prompt = """Here is a question for an LLM:@question@@ -32,7 +32,7 @@ type = "PipeLLM"description = "Answer knowingly after analyzing a question"inputs = { question = "Question", question_analysis = "QuestionAnalysis" }output = "ThoughtfulAnswer"-model = "llm_to_reason"+model = "llm_for_complex_reasoning"prompt = """A question was asked:@questiondiff --git a/tests/unit/core/stuffs/test_stuff_factory_implicit_memory.py b/tests/unit/core/stuffs/test_stuff_factory_implicit_memory.pyindex 9383f27d8..8a2bd1791 100644--- a/tests/unit/core/stuffs/test_stuff_factory_implicit_memory.py+++ b/tests/unit/core/stuffs/test_stuff_factory_implicit_memory.py@@ -98,7 +98,7 @@ class TestStuffFactoryImplicitMemory:expected_stuff: Stuff,):log.info(f"Testing case: {test_name}")- log.debug(f"setup_test_concept: {setup_test_concept}")+ log.verbose(f"setup_test_concept: {setup_test_concept}")result = StuffFactory.make_stuff_from_stuff_content_or_data(name=stuff_name,@@ -132,7 +132,7 @@ class TestStuffFactoryImplicitMemoryWithSearchDomains:expected_stuff: Stuff,):log.info(f"Testing search domain case: {test_name}")- log.debug(f"setup_test_concept: {setup_test_concept}")+ log.verbose(f"setup_test_concept: {setup_test_concept}")result = StuffFactory.make_stuff_from_stuff_content_or_data(name=stuff_name,code=stuff_code,@@ -168,7 +168,7 @@ class TestStuffFactoryImplicitMemoryErrors:error_match: str,):log.info(f"Testing error case: {test_name}")- log.debug(f"setup_test_concept: {setup_test_concept}")+ log.verbose(f"setup_test_concept: {setup_test_concept}")with pytest.raises(expected_exception, match=error_match):StuffFactory.make_stuff_from_stuff_content_or_data(diff --git a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.pyindex e51f941e2..34f2cf5ed 100644--- a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.py@@ -1,7 +1,6 @@from typing import ClassVarfrom pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint@@ -17,7 +16,7 @@ class PipeBatchTestCases:),PipeBatchBlueprint(description="Process items in batch",- inputs={"items": InputRequirementBlueprint(concept="ItemList")},+ inputs={"items": "ItemList"},output="ProcessedItems",type="PipeBatch",pipe_category="PipeController",@@ -40,7 +39,7 @@ class PipeBatchTestCases:),PipeBatchBlueprint(description="Batch with custom names",- inputs={"data": InputRequirementBlueprint(concept="DataList")},+ inputs={"data": "DataList"},output="Results",type="PipeBatch",pipe_category="PipeController",diff --git a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.pyindex b9394b22c..579c7cdf4 100644--- a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py@@ -1,7 +1,6 @@from typing import ClassVarfrom pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint@@ -23,7 +22,7 @@ class PipeConditionTestCases:),PipeConditionBlueprint(description="Conditional with template",- inputs={"item": InputRequirementBlueprint(concept="Item")},+ inputs={"item": "Item"},output="ProcessedItem",type="PipeCondition",pipe_category="PipeController",diff --git a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.pyindex 8d260a49e..d616eed43 100644--- a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py@@ -2,7 +2,6 @@ from typing import ClassVarfrom pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpecfrom pipelex.builder.pipe.sub_pipe_spec import SubPipeSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprintfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint@@ -24,7 +23,7 @@ class PipeParallelTestCases:),PipeParallelBlueprint(description="Run pipes in parallel",- inputs={"data": InputRequirementBlueprint(concept="Data")},+ inputs={"data": "Data"},output="Results",type="PipeParallel",pipe_category="PipeController",@@ -55,7 +54,7 @@ class PipeParallelTestCases:),PipeParallelBlueprint(description="Parallel with combined output",- inputs={"input": InputRequirementBlueprint(concept="Input")},+ inputs={"input": "Input"},output="CombinedResult",type="PipeParallel",pipe_category="PipeController",@@ -85,7 +84,7 @@ class PipeParallelTestCases:),PipeParallelBlueprint(description="Parallel with combined output",- inputs={"input": InputRequirementBlueprint(concept="Input")},+ inputs={"input": "Input"},output="CombinedResult",type="PipeParallel",pipe_category="PipeController",diff --git a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.pyindex 321999083..b7d69220f 100644--- a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.py@@ -2,7 +2,6 @@ from typing import ClassVarfrom pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpecfrom pipelex.builder.pipe.sub_pipe_spec import SubPipeSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprintfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint@@ -23,7 +22,7 @@ class PipeSequenceTestCases:),PipeSequenceBlueprint(description="A sequence of operations",- inputs={"input_data": InputRequirementBlueprint(concept="Text")},+ inputs={"input_data": "Text"},output="ProcessedData",type="PipeSequence",pipe_category="PipeController",@@ -54,7 +53,7 @@ class PipeSequenceTestCases:),PipeSequenceBlueprint(description="Sequence with batch",- inputs={"items": InputRequirementBlueprint(concept="ItemList")},+ inputs={"items": "ItemList"},output="ProcessedItems",type="PipeSequence",pipe_category="PipeController",diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.pyindex 18d72ec75..8ade49b9f 100644--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.py@@ -4,7 +4,6 @@ from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpecfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprintfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormat-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint@@ -21,7 +20,7 @@ class PipeComposeTestCases:),PipeComposeBlueprint(description="Render a template",- inputs={"data": InputRequirementBlueprint(concept="Data")},+ inputs={"data": "Data"},output="RenderedText",type="PipeCompose",pipe_category="PipeOperator",diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.pyindex 21c0c5979..583b8cd6c 100644--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py@@ -1,7 +1,6 @@from typing import ClassVar-from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint+from pipelex.builder.pipe.pipe_extract_spec import ExtractSkill, PipeExtractSpecfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint@@ -13,16 +12,16 @@ class PipeExtractTestCases:description="Extract text from image",inputs={"image": "Image"},output="Page",- extract_skill="extract_text_from_visuals",+ extract_skill=ExtractSkill.EXTRACT_TEXT_FROM_VISUALS,),PipeExtractBlueprint(source=None,description="Extract text from image",- inputs={"image": InputRequirementBlueprint(concept="Image")},+ inputs={"image": "Image"},output="Page",type="PipeExtract",pipe_category="PipeOperator",- model="base_ocr_mistral",+ model=ExtractSkill.EXTRACT_TEXT_FROM_VISUALS,),)@@ -33,7 +32,7 @@ class PipeExtractTestCases:description="Extract with page options",inputs={"document": "PDF"},output="Page",- extract_skill="extract_text_from_pdf",+ extract_skill=ExtractSkill.EXTRACT_TEXT_FROM_PDF,page_images=True,page_image_captions=True,page_views=True,@@ -41,11 +40,11 @@ class PipeExtractTestCases:PipeExtractBlueprint(source=None,description="Extract with page options",- inputs={"document": InputRequirementBlueprint(concept="PDF")},+ inputs={"document": "PDF"},output="Page",type="PipeExtract",pipe_category="PipeOperator",- model="base_ocr_mistral",+ model=ExtractSkill.EXTRACT_TEXT_FROM_PDF,page_images=True,page_image_captions=True,page_views=True,diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.pyindex 1a0af343b..2583fcaae 100644--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.py@@ -1,7 +1,6 @@from typing import ClassVarfrom pipelex.builder.pipe.pipe_func_spec import PipeFuncSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint@@ -17,7 +16,7 @@ class PipeFuncTestCases:),PipeFuncBlueprint(description="Execute a function",- inputs={"data": InputRequirementBlueprint(concept="Data")},+ inputs={"data": "Data"},output="ProcessedData",type="PipeFunc",pipe_category="PipeOperator",diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.pyindex 93802c05a..33b023e51 100644--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py@@ -1,7 +1,6 @@from typing import ClassVarfrom pipelex.builder.pipe.pipe_img_gen_spec import ImgGenSkill, PipeImgGenSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint@@ -13,6 +12,7 @@ class PipeImgGenTestCases:description="Generate an image",inputs={},output="GeneratedImage",+ img_gen_skill=ImgGenSkill.GEN_IMAGE_BASIC,),PipeImgGenBlueprint(description="Generate an image",@@ -22,11 +22,10 @@ class PipeImgGenTestCases:pipe_category="PipeOperator",img_gen_prompt=None,img_gen_prompt_var_name=None,- model=None,+ model=ImgGenSkill.GEN_IMAGE_BASIC,aspect_ratio=None,background=None,output_format=None,- nb_output=None,is_raw=None,seed=None,),@@ -38,24 +37,22 @@ class PipeImgGenTestCases:pipe_code="advanced_img_gen",description="Generate image with options",inputs={"description": "Text"},- output="Image",- img_gen_skill=ImgGenSkill.GEN_IMAGE,- nb_output=3,+ output="Image[3]",+ img_gen_skill=ImgGenSkill.GEN_IMAGE_FAST,),PipeImgGenBlueprint(description="Generate image with options",- inputs={"description": InputRequirementBlueprint(concept="Text")},- output="Image",+ inputs={"description": "Text"},+ output="Image[3]",type="PipeImgGen",pipe_category="PipeOperator",img_gen_prompt=None,- model=ImgGenSkill.GEN_IMAGE.model_recommendation,+ model=ImgGenSkill.GEN_IMAGE_FAST,aspect_ratio=None,background=None,output_format=None,is_raw=None,seed=None,- nb_output=3,img_gen_prompt_var_name=None,),)diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.pyindex 523bd3361..a037eee51 100644--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.py@@ -1,8 +1,6 @@from typing import ClassVar-from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec-from pipelex.cogt.llm.llm_setting import LLMSetting-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint+from pipelex.builder.pipe.pipe_llm_spec import LLMSkill, PipeLLMSpecfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint@@ -14,17 +12,17 @@ class PipeLLMTestCases:description="Generate text",inputs={"topic": "Text"},output="Text",- llm="llm_for_creative_writing",+ llm_skill=LLMSkill.LLM_FOR_CREATIVE_WRITING,prompt="Write about $topic",),PipeLLMBlueprint(source=None,type="PipeLLM",description="Generate text",- inputs={"topic": InputRequirementBlueprint(concept="Text")},+ inputs={"topic": "Text"},output="Text",prompt="Write about $topic",- model="claude-4.1-opus",+ model=LLMSkill.LLM_FOR_CREATIVE_WRITING,),)@@ -35,7 +33,7 @@ class PipeLLMTestCases:description="Generate without inputs",inputs={},output="Text",- llm="llm_for_creative_writing",+ llm_skill=LLMSkill.LLM_FOR_CREATIVE_WRITING,prompt="Generate something interesting",),PipeLLMBlueprint(@@ -44,7 +42,7 @@ class PipeLLMTestCases:description="Generate without inputs",output="Text",prompt="Generate something interesting",- model="claude-4.1-opus",+ model=LLMSkill.LLM_FOR_CREATIVE_WRITING,),)@@ -56,16 +54,16 @@ class PipeLLMTestCases:inputs={"image": "Image"},output="Text",prompt="Analyze the image: $image",- llm="llm_cheap_for_vision",+ llm_skill=LLMSkill.LLM_FOR_BASIC_VISION,),PipeLLMBlueprint(source=None,type="PipeLLM",description="Analyze image",- inputs={"image": InputRequirementBlueprint(concept="Image")},+ inputs={"image": "Image"},output="Text",prompt="Analyze the image: $image",- model="gemini-2.5-flash-lite",+ model=LLMSkill.LLM_FOR_BASIC_VISION,),)@@ -77,13 +75,13 @@ class PipeLLMTestCases:inputs={},output="Text",prompt="Generate text",- llm="llm_to_reason",+ llm_skill=LLMSkill.LLM_FOR_COMPLEX_REASONING,),PipeLLMBlueprint(source=None,type="PipeLLM",description="Generate with preset",- model="claude-4.5-sonnet",+ model=LLMSkill.LLM_FOR_COMPLEX_REASONING,output="Text",prompt="Generate text",),@@ -97,18 +95,13 @@ class PipeLLMTestCases:inputs={},output="Text",prompt="Generate text",- llm="llm_cheap_for_easy_questions",- temperature=0.7,+ llm_skill=LLMSkill.LLM_TO_ANSWER_EASY_QUESTIONS,),PipeLLMBlueprint(source=None,type="PipeLLM",description="Generate with settings",- model=LLMSetting(- model="claude-4.5-sonnet",- temperature=0.7,- max_tokens=None, # "auto" is handled at conversion to core- ),+ model=LLMSkill.LLM_TO_ANSWER_EASY_QUESTIONS,output="Text",prompt="Generate text",),@@ -123,17 +116,17 @@ class PipeLLMTestCases:output="Analysis",system_prompt="You are a data analyst",prompt="Analyze: @data",- llm="llm_to_analyze_data",+ llm_skill=LLMSkill.LLM_TO_ANALYZE_DATA,),PipeLLMBlueprint(source=None,type="PipeLLM",description="Generate with system prompt",- inputs={"data": InputRequirementBlueprint(concept="Data")},+ inputs={"data": "Data"},system_prompt="You are a data analyst",prompt="Analyze: @data",output="Analysis",- model="claude-4.5-sonnet",+ model=LLMSkill.LLM_TO_ANALYZE_DATA,),)@@ -143,20 +136,17 @@ class PipeLLMTestCases:pipe_code="generate_items",description="Generate multiple items",inputs={},- output="Item",+ output="Item[]",prompt="Generate items",- multiple_output=True,- llm="llm_cheap_for_easy_questions",+ llm_skill=LLMSkill.LLM_TO_ANSWER_EASY_QUESTIONS,),PipeLLMBlueprint(source=None,type="PipeLLM",description="Generate multiple items",- multiple_output=True,- nb_output=None,- output="Item",+ output="Item[]",prompt="Generate items",- model="claude-4.5-sonnet",+ model=LLMSkill.LLM_TO_ANSWER_EASY_QUESTIONS,),)@@ -166,20 +156,17 @@ class PipeLLMTestCases:pipe_code="generate_items",description="Generate exactly 5 items",inputs={},- output="Item",+ output="Item[5]",prompt="Generate items",- nb_output=5,- llm="llm_cheap_for_easy_questions",+ llm_skill=LLMSkill.LLM_TO_ANSWER_EASY_QUESTIONS,),PipeLLMBlueprint(source=None,type="PipeLLM",description="Generate exactly 5 items",- nb_output=5,- multiple_output=None,- output="Item",+ output="Item[5]",prompt="Generate items",- model="claude-4.5-sonnet",+ model=LLMSkill.LLM_TO_ANSWER_EASY_QUESTIONS,),)diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.pyindex fa02b5156..4f7b9b9e0 100644--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py@@ -21,5 +21,4 @@ class TestPipeLLMBlueprintConversion:pipe_code=f"test_pipe_{test_name}",blueprint=blueprint,)-pretty_print(pipe_llm_from_blueprint, title="PipeLLM from blueprint")diff --git a/tests/unit/pipelex/builder/pipe/test_data_inputs.py b/tests/unit/pipelex/builder/pipe/test_data_inputs.pyindex 797be5c86..de8d19a60 100644--- a/tests/unit/pipelex/builder/pipe/test_data_inputs.py+++ b/tests/unit/pipelex/builder/pipe/test_data_inputs.py@@ -1,34 +1,32 @@from typing import ClassVar-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-class InputRequirementTestCases:SIMPLE_TEXT_INPUT = ("simple_text_input","Text",- InputRequirementBlueprint(concept="Text"),+ "Text",)IMAGE_INPUT = ("image_input","Image",- InputRequirementBlueprint(concept="Image"),+ "Image",)CUSTOM_CONCEPT_INPUT = ("custom_concept_input","CustomConcept",- InputRequirementBlueprint(concept="CustomConcept"),+ "CustomConcept",)DOMAIN_CONCEPT_INPUT = ("domain_concept_input","domain.ConceptName",- InputRequirementBlueprint(concept="domain.ConceptName"),+ "domain.ConceptName",)- TEST_CASES: ClassVar[list[tuple[str, str, InputRequirementBlueprint]]] = [+ TEST_CASES: ClassVar[list[tuple[str, str, str]]] = [SIMPLE_TEXT_INPUT,IMAGE_INPUT,CUSTOM_CONCEPT_INPUT,diff --git a/tests/unit/pipelex/builder/pipe/test_data_pipe.py b/tests/unit/pipelex/builder/pipe/test_data_pipe.pyindex 2c0be069c..cbbfed2c4 100644--- a/tests/unit/pipelex/builder/pipe/test_data_pipe.py+++ b/tests/unit/pipelex/builder/pipe/test_data_pipe.py@@ -1,7 +1,6 @@from typing import ClassVar-from pipelex.builder.pipe.pipe_signature import PipeSpec-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.core.pipes.pipe_blueprint import PipeBlueprint@@ -20,7 +19,7 @@ class PipeBlueprintTestCases:type="PipeLLM",pipe_category="PipeOperator",description="A simple pipe",- inputs={"input": InputRequirementBlueprint(concept="Text")},+ inputs={"input": "Text"},output="ProcessedText",),)@@ -43,8 +42,8 @@ class PipeBlueprintTestCases:pipe_category="PipeOperator",description="Pipe with input requirements",inputs={- "data": InputRequirementBlueprint(concept="Data"),- "config": InputRequirementBlueprint(concept="Config"),+ "data": "Data",+ "config": "Config",},output="Result",),diff --git a/tests/unit/pipelex/builder/pipe/test_data_sub_pipe.py b/tests/unit/pipelex/builder/pipe/test_data_sub_pipe.pyindex be01916c3..297bded4e 100644--- a/tests/unit/pipelex/builder/pipe/test_data_sub_pipe.py+++ b/tests/unit/pipelex/builder/pipe/test_data_sub_pipe.py@@ -11,18 +11,6 @@ class SubPipeTestCases:SubPipeBlueprint(pipe="process_data", result="processed_data"),)- SUB_PIPE_WITH_MULTIPLE_OUTPUT = (- "sub_pipe_with_multiple_output",- SubPipeSpec(pipe_code="generate_items", result="items", multiple_output=True),- SubPipeBlueprint(pipe="generate_items", result="items", multiple_output=True),- )-- SUB_PIPE_WITH_FIXED_OUTPUT = (- "sub_pipe_with_fixed_output",- SubPipeSpec(pipe_code="generate_ideas", result="ideas", nb_output=3),- SubPipeBlueprint(pipe="generate_ideas", result="ideas", nb_output=3),- )-SUB_PIPE_WITH_BATCH = ("sub_pipe_with_batch",SubPipeSpec(pipe_code="process_item", result="processed_items", batch_over="input_list", batch_as="current_item"),@@ -31,7 +19,5 @@ class SubPipeTestCases:TEST_CASES: ClassVar[list[tuple[str, SubPipeSpec, SubPipeBlueprint]]] = [SIMPLE_SUB_PIPE,- SUB_PIPE_WITH_MULTIPLE_OUTPUT,- SUB_PIPE_WITH_FIXED_OUTPUT,SUB_PIPE_WITH_BATCH,]diff --git a/tests/unit/pipelex/builder/pipe/test_inputs_blueprint.py b/tests/unit/pipelex/builder/pipe/test_inputs_blueprint.pyindex 521e9e7e2..57fcba7a7 100644--- a/tests/unit/pipelex/builder/pipe/test_inputs_blueprint.py+++ b/tests/unit/pipelex/builder/pipe/test_inputs_blueprint.py@@ -1,19 +1,18 @@import pytest-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom tests.unit.pipelex.builder.pipe.test_data_inputs import InputRequirementTestCasesclass TestInputRequirementBlueprintConversion:@pytest.mark.parametrize(- ("test_name", "input_spec", "expected_blueprint"),+ ("test_name", "input_spec", "expected_string"),InputRequirementTestCases.TEST_CASES,)- def test_input_requirement_spec_to_blueprint(+ def test_input_requirement_spec_to_string(self,test_name: str, # noqa: ARG002input_spec: str,- expected_blueprint: InputRequirementBlueprint,+ expected_string: str,):- result = InputRequirementBlueprint(concept=input_spec)- assert result == expected_blueprint+ # Now inputs are just strings, so we test that they match+ assert input_spec == expected_stringdiff --git a/tests/unit/pipelex/builder/pipe/test_pipe_blueprint.py b/tests/unit/pipelex/builder/pipe/test_pipe_blueprint.pyindex 24748240a..d1f1ccf6c 100644--- a/tests/unit/pipelex/builder/pipe/test_pipe_blueprint.py+++ b/tests/unit/pipelex/builder/pipe/test_pipe_blueprint.py@@ -1,6 +1,6 @@import pytest-from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.core.pipes.pipe_blueprint import PipeBlueprintfrom tests.unit.pipelex.builder.pipe.test_data_pipe import PipeBlueprintTestCasesdiff --git a/tests/unit/tools/test_runner_generator.py b/tests/unit/pipelex/builder/test_runner_generator.pysimilarity index 100%rename from tests/unit/tools/test_runner_generator.pyrename to tests/unit/pipelex/builder/test_runner_generator.pydiff --git a/tests/unit/pipelex/cogt/models/test_model_deck.py b/tests/unit/pipelex/cogt/models/test_model_deck.pyindex 06d044333..6b928f8e6 100644--- a/tests/unit/pipelex/cogt/models/test_model_deck.py+++ b/tests/unit/pipelex/cogt/models/test_model_deck.py@@ -1,5 +1,4 @@import pytest-from pytest_mock import MockerFixturefrom pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaultsfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec@@ -35,9 +34,9 @@ class TestModelDeckGetOptionalInferenceModel:for_object=LLMSetting(model="default_object", temperature=0.1, max_tokens=1000),),extract_presets={},- extract_choice_default="base_ocr_mistral",+ extract_choice_default="extract_text_from_visuals",img_gen_presets={},- img_gen_choice_default="base_img_gen",+ img_gen_choice_default="gen_image_basic",)def test_direct_model_lookup_success(self):@@ -51,145 +50,114 @@ class TestModelDeckGetOptionalInferenceModel:# Assertassert result == model_spec- def test_direct_model_lookup_not_found(self, mocker: MockerFixture):+ def test_direct_model_lookup_not_found(self):# Arrangemodel_deck = self._create_test_model_deck()- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("nonexistent-model")# Assertassert result is None- mock_log.warning.assert_called_once_with(- "Skipping model handle 'nonexistent-model' because it's not found in deck, it could be an external plugin."- )- def test_simple_string_alias_resolution_success(self, mocker: MockerFixture):+ def test_simple_string_alias_resolution_success(self):# Arrangemodel_spec = self._create_test_model_spec("gpt-4")model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, aliases={"best-gpt": "gpt-4"})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-gpt")# Assertassert result == model_spec- mock_log.debug.assert_called_once_with("Redirection for 'best-gpt': gpt-4")- def test_simple_string_alias_resolution_not_found(self, mocker: MockerFixture):+ def test_simple_string_alias_resolution_not_found(self):# Arrangemodel_deck = self._create_test_model_deck(aliases={"best-gpt": "nonexistent-model"})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-gpt")# Assertassert result is None- mock_log.debug.assert_called_once_with("Redirection for 'best-gpt': nonexistent-model")- # The final warning is about the original alias, not the target- mock_log.warning.assert_called_with("Skipping model handle 'best-gpt' because it's not found in deck, it could be an external plugin.")- def test_list_alias_resolution_first_success(self, mocker: MockerFixture):+ def test_list_alias_resolution_first_success(self):# Arrangemodel_spec = self._create_test_model_spec("gpt-4")model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, aliases={"best-model": ["gpt-4", "claude-3"]})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-model")# Assertassert result == model_spec- mock_log.debug.assert_called_once_with("Redirection for 'best-model': ['gpt-4', 'claude-3']")- def test_list_alias_resolution_second_success(self, mocker: MockerFixture):+ def test_list_alias_resolution_second_success(self):# Arrangemodel_spec = self._create_test_model_spec("claude-3")model_deck = self._create_test_model_deck(inference_models={"claude-3": model_spec},aliases={"best-model": ["nonexistent-model", "claude-3"]},)- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-model")# Assertassert result == model_spec- mock_log.debug.assert_called_once_with("Redirection for 'best-model': ['nonexistent-model', 'claude-3']")- def test_list_alias_resolution_none_found(self, mocker: MockerFixture):+ def test_list_alias_resolution_none_found(self):# Arrangemodel_deck = self._create_test_model_deck(aliases={"best-model": ["nonexistent-1", "nonexistent-2"]})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-model")# Assertassert result is None- mock_log.debug.assert_called_once_with("Redirection for 'best-model': ['nonexistent-1', 'nonexistent-2']")- # Should have warning calls: one for each alias in the list + one for the original alias- assert mock_log.warning.call_count == 3- def test_recursive_alias_resolution_success(self, mocker: MockerFixture):+ def test_recursive_alias_resolution_success(self):# Arrangemodel_spec = self._create_test_model_spec("gpt-4")model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, aliases={"best-model": "best-gpt", "best-gpt": "gpt-4"})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-model")# Assertassert result == model_spec- # Should have debug calls for both redirections- assert mock_log.debug.call_count == 2- def test_recursive_alias_resolution_with_list(self, mocker: MockerFixture):+ def test_recursive_alias_resolution_with_list(self):# Arrangemodel_spec = self._create_test_model_spec("gpt-4")model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec},aliases={"best-model": ["nonexistent", "best-gpt"], "best-gpt": "gpt-4"},)- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-model")# Assertassert result == model_spec- # Should have debug calls for both redirections- assert mock_log.debug.call_count >= 2- def test_empty_alias_list(self, mocker: MockerFixture):+ def test_empty_alias_list(self):# Arrangemodel_deck = self._create_test_model_deck(aliases={"empty-alias": []})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("empty-alias")# Assertassert result is None- # Empty list evaluates to False, so no debug log is called- mock_log.debug.assert_not_called()- mock_log.warning.assert_called_once_with(- "Skipping model handle 'empty-alias' because it's not found in deck, it could be an external plugin."- )- def test_circular_alias_prevention(self, mocker: MockerFixture):+ def test_circular_alias_prevention(self):# Note: The current implementation doesn't have explicit circular reference detection,# but Python's recursion limit should prevent infinite loops.# This test ensures the method handles it gracefully.# Arrangemodel_deck = self._create_test_model_deck(aliases={"alias-a": "alias-b", "alias-b": "alias-a"})- mocker.patch("pipelex.cogt.models.model_deck.log")# Act & Assert# This should either return None or raise RecursionError (which would be caught by pytest)@@ -199,24 +167,21 @@ class TestModelDeckGetOptionalInferenceModel:# If no exception, result should be Noneassert result is None- def test_complex_waterfall_scenario(self, mocker: MockerFixture):+ def test_complex_waterfall_scenario(self):# Arrangemodel_spec = self._create_test_model_spec("claude-3")model_deck = self._create_test_model_deck(inference_models={"claude-3": model_spec},aliases={"best-model": ["premium-gpt", "premium-claude"], "premium-gpt": ["gpt-4-turbo", "gpt-4"], "premium-claude": "claude-3"},)- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model("best-model")# Assertassert result == model_spec- # Should have multiple debug calls for the waterfall resolution- assert mock_log.debug.call_count >= 2- def test_mixed_string_and_list_aliases(self, mocker: MockerFixture):+ def test_mixed_string_and_list_aliases(self):# Arrangemodel_spec1 = self._create_test_model_spec("gpt-4")model_spec2 = self._create_test_model_spec("claude-3")@@ -228,7 +193,6 @@ class TestModelDeckGetOptionalInferenceModel:"backup-model": ["claude-4", "claude-3"], # list alias},)- mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult1 = model_deck.get_optional_inference_model("ai-model")@@ -247,31 +211,12 @@ class TestModelDeckGetOptionalInferenceModel:"UPPERCASE-MODEL", # uppercase],)- def test_edge_case_llm_handles(self, llm_handle: str, mocker: MockerFixture):+ def test_edge_case_llm_handles(self, llm_handle: str):# Arrangemodel_deck = self._create_test_model_deck()- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")# Actresult = model_deck.get_optional_inference_model(llm_handle)# Assertassert result is None- mock_log.warning.assert_called_once_with(- f"Skipping model handle '{llm_handle}' because it's not found in deck, it could be an external plugin."- )-- def test_logging_behavior(self, mocker: MockerFixture):- # Arrange- model_deck = self._create_test_model_deck(aliases={"test-alias": "target-model"})- mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")-- # Act- model_deck.get_optional_inference_model("test-alias")-- # Assert- # Verify debug log for redirection- mock_log.debug.assert_called_with("Redirection for 'test-alias': target-model")-- # The final warning is about the original alias, not the target- mock_log.warning.assert_called_with("Skipping model handle 'test-alias' because it's not found in deck, it could be an external plugin.")diff --git a/tests/unit/pipelex/core/concepts/concept/test_concept.py b/tests/unit/pipelex/core/concepts/test_concept.pysimilarity index 100%rename from tests/unit/pipelex/core/concepts/concept/test_concept.pyrename to tests/unit/pipelex/core/concepts/test_concept.pydiff --git a/tests/unit/core/concepts/test_concept_compact_memory.py b/tests/unit/pipelex/core/concepts/test_concept_compact_memory.pysimilarity index 100%rename from tests/unit/core/concepts/test_concept_compact_memory.pyrename to tests/unit/pipelex/core/concepts/test_concept_compact_memory.pydiff --git a/tests/unit/pipelex/core/pipes/data.py b/tests/unit/pipelex/core/pipes/data.pynew file mode 100644index 000000000..61974e0c6--- /dev/null+++ b/tests/unit/pipelex/core/pipes/data.py@@ -0,0 +1,114 @@+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity++# Test cases format: (nb_output, multiple_output, expected_result, test_description)+MAKE_VARIABLE_MULTIPLICITY_TEST_CASES: list[tuple[int | None, bool | None, VariableMultiplicity | None, str]] = [+ # nb_output, multiple_output, expected_result, description+ # Test cases where nb_output takes precedence+ (3, None, 3, "nb_output=3, multiple_output=None -> returns 3"),+ (5, False, 5, "nb_output=5, multiple_output=False -> returns 5 (nb_output takes precedence)"),+ (1, True, 1, "nb_output=1, multiple_output=True -> returns 1 (nb_output takes precedence)"),+ (10, None, 10, "nb_output=10, multiple_output=None -> returns 10"),+ # Test cases where multiple_output=True is used+ (None, True, True, "nb_output=None, multiple_output=True -> returns True"),+ (0, True, True, "nb_output=0 (falsy), multiple_output=True -> returns True"),+ # Test cases where default (None) is returned+ (None, None, None, "nb_output=None, multiple_output=None -> returns None (default)"),+ (None, False, None, "nb_output=None, multiple_output=False -> returns None"),+ (0, None, None, "nb_output=0 (falsy), multiple_output=None -> returns None"),+ (0, False, None, "nb_output=0 (falsy), multiple_output=False -> returns None"),+ # Edge cases with negative numbers (should still be truthy)+ (-1, None, -1, "nb_output=-1 (truthy), multiple_output=None -> returns -1"),+ (-5, True, -5, "nb_output=-5 (truthy), multiple_output=True -> returns -5 (nb_output takes precedence)"),+ # Edge cases with large numbers+ (1000, None, 1000, "nb_output=1000, multiple_output=None -> returns 1000"),+ (999999, False, 999999, "nb_output=999999, multiple_output=False -> returns 999999"),+]++# Test cases for InputRequirementsFactory.make_from_string+# Format: (domain, requirement_str, expected_concept_string, expected_multiplicity)+SINGLE_ITEM_NO_BRACKETS_TEST_CASES: list[tuple[str, str, str, int | bool | None]] = [+ # Native domain tests with full concept strings+ ("native", "native.Text", "native.Text", None),+ ("native", "native.Image", "native.Image", None),+ ("native", "native.PDF", "native.PDF", None),+ # Implicit concept creation (unknown concept codes become implicit)+ ("my_domain", "MyConcept", "implicit.MyConcept", None),+ ("accounting", "Invoice", "implicit.Invoice", None),+]++# Format: (domain, requirement_str, expected_concept_string)+MULTIPLE_ITEMS_EMPTY_BRACKETS_TEST_CASES: list[tuple[str, str, str]] = [+ # Native domain tests+ ("native", "native.Text[]", "native.Text"),+ ("native", "native.Image[]", "native.Image"),+ # Implicit concept creation with empty brackets+ ("my_domain", "MyConcept[]", "implicit.MyConcept"),+ ("accounting", "Invoice[]", "implicit.Invoice"),+]++# Format: (domain, requirement_str, expected_concept_string, expected_multiplicity)+FIXED_COUNT_TEST_CASES: list[tuple[str, str, str, int]] = [+ # Native domain tests+ ("native", "native.Text[5]", "native.Text", 5),+ ("native", "native.Image[3]", "native.Image", 3),+ ("native", "native.PDF[10]", "native.PDF", 10),+ # Implicit concept creation with fixed counts+ ("my_domain", "MyConcept[7]", "implicit.MyConcept", 7),+ ("accounting", "Invoice[2]", "implicit.Invoice", 2),+ ("hr", "Employee[10]", "implicit.Employee", 10),+]++# Format: (domain, requirement_str, concept_codes_from_same_domain, expected_concept_string, expected_multiplicity, description)+CONCEPT_CODE_RESOLUTION_TEST_CASES: list[tuple[str, str, list[str] | None, str, int | bool | None, str]] = [+ # Native concepts resolved from concept code only (no multiplicity)+ ("native", "Text", None, "native.Text", None, "Native Text without brackets"),+ ("native", "Image[3]", None, "native.Image", 3, "Native Image with fixed count"),+ ("native", "PDF[]", None, "native.PDF", True, "Native PDF with empty brackets"),+ # Native concepts with concept_codes_from_same_domain (native always takes precedence)+ ("native", "Text[5]", ["Text", "Image", "PDF"], "native.Text", 5, "Native Text with concept codes list"),+ # Native concepts take precedence even when used with custom domain parameter+ ("my_domain", "Text", ["Text", "Document"], "native.Text", None, "Native Text takes precedence over custom domain"),+ ("my_domain", "Image[2]", ["Image", "Photo"], "native.Image", 2, "Native Image takes precedence"),+ ("accounting", "Text[3]", ["Invoice", "Receipt"], "native.Text", 3, "Native always wins"),+ # Unknown concepts (not native, not in concept_codes_from_same_domain) create implicit concepts+ ("my_domain", "UnknownConcept[5]", ["KnownConcept"], "implicit.UnknownConcept", 5, "Unknown concept with known list"),+ ("my_domain", "UnknownConcept", None, "implicit.UnknownConcept", None, "Unknown concept without list"),+ ("accounting", "UnknownInvoice[]", ["Invoice"], "implicit.UnknownInvoice", True, "Unknown with empty brackets"),+]++# Format: (domain, requirement_str, expected_concept_string, expected_multiplicity)+EXPLICIT_DOMAIN_IN_STRING_TEST_CASES: list[tuple[str, str, str, int | bool | None]] = [+ # Explicitly specifying native domain in string+ ("my_domain", "native.Text", "native.Text", None),+ ("accounting", "native.Image[3]", "native.Image", 3),+ ("test_domain", "native.PDF[]", "native.PDF", True),+ # Explicitly using implicit domain creates implicit concepts+ ("my_domain", "implicit.OtherConcept[2]", "implicit.OtherConcept", 2),+ ("test_domain", "implicit.LineItem", "implicit.LineItem", None),+]++# Format: (domain, requirement_str, expected_concept_string, expected_multiplicity)+VARIOUS_FIXED_COUNTS_TEST_CASES: list[tuple[str, str, str, int]] = [+ # Native domain with various counts+ ("native", "native.Image[1]", "native.Image", 1),+ ("native", "native.Image[2]", "native.Image", 2),+ ("native", "native.Image[10]", "native.Image", 10),+ ("native", "native.Image[100]", "native.Image", 100),+ ("native", "native.Image[999]", "native.Image", 999),+ # Implicit concepts with various counts (unknown concept codes)+ ("my_domain", "CustomConcept[1]", "implicit.CustomConcept", 1),+ ("accounting", "Transaction[50]", "implicit.Transaction", 50),+]++# Format: (domain, requirement_str, expected_concept_string)+DIFFERENT_CONCEPT_CODES_TEST_CASES: list[tuple[str, str, str]] = [+ # Native concepts+ ("native", "native.Text", "native.Text"),+ ("native", "native.Image", "native.Image"),+ ("native", "native.PDF", "native.PDF"),+ ("native", "native.Number", "native.Number"),+ ("native", "native.Page", "native.Page"),+ # Implicit concepts (unknown concept codes)+ ("my_domain", "CustomConcept", "implicit.CustomConcept"),+ ("accounting", "Invoice", "implicit.Invoice"),+]diff --git a/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py b/tests/unit/pipelex/core/pipes/test_input_requirements_factory.pynew file mode 100644index 000000000..7058b5d5a--- /dev/null+++ b/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py@@ -0,0 +1,207 @@+import pytest++from pipelex.core.concepts.exceptions import ConceptCodeError+from pipelex.core.domains.exceptions import DomainError+from pipelex.core.pipes.input_requirements import InputRequirement+from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory, InputRequirementsFactorySyntaxError+from pipelex.exceptions import ConceptLibraryConceptNotFoundError+from tests.unit.pipelex.core.pipes.data import (+ CONCEPT_CODE_RESOLUTION_TEST_CASES,+ DIFFERENT_CONCEPT_CODES_TEST_CASES,+ EXPLICIT_DOMAIN_IN_STRING_TEST_CASES,+ FIXED_COUNT_TEST_CASES,+ MULTIPLE_ITEMS_EMPTY_BRACKETS_TEST_CASES,+ SINGLE_ITEM_NO_BRACKETS_TEST_CASES,+ VARIOUS_FIXED_COUNTS_TEST_CASES,+)+++class TestMakeInputRequirementsFromString:+ """Test the InputRequirementsFactory.make_from_str method."""++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "expected_concept_string", "expected_multiplicity"),+ SINGLE_ITEM_NO_BRACKETS_TEST_CASES,+ )+ def test_single_item_default_no_brackets(+ self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int | bool | None+ ):+ """Test parsing a concept string without brackets (single item, default)."""+ result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)++ assert isinstance(result, InputRequirement)+ assert result.concept.concept_string == expected_concept_string+ assert result.multiplicity == expected_multiplicity++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "expected_concept_string"),+ MULTIPLE_ITEMS_EMPTY_BRACKETS_TEST_CASES,+ )+ def test_multiple_items_with_empty_brackets(self, domain: str, requirement_str: str, expected_concept_string: str):+ """Test parsing a concept string with empty brackets (multiple items)."""+ result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)++ assert isinstance(result, InputRequirement)+ assert result.concept.concept_string == expected_concept_string+ assert result.multiplicity is True++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "expected_concept_string", "expected_multiplicity"),+ FIXED_COUNT_TEST_CASES,+ )+ def test_fixed_count_with_number_in_brackets(self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int):+ """Test parsing a concept string with a number in brackets (fixed count)."""+ result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)++ assert isinstance(result, InputRequirement)+ assert result.concept.concept_string == expected_concept_string+ assert result.multiplicity == expected_multiplicity++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "expected_concept_string", "expected_multiplicity"),+ VARIOUS_FIXED_COUNTS_TEST_CASES,+ )+ def test_various_fixed_counts(self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int):+ """Test parsing concept strings with various numbers in brackets."""+ result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)+ assert result.multiplicity == expected_multiplicity, f"Failed for {requirement_str}"+ assert result.concept.concept_string == expected_concept_string++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "expected_concept_string"),+ DIFFERENT_CONCEPT_CODES_TEST_CASES,+ )+ def test_different_concept_codes(self, domain: str, requirement_str: str, expected_concept_string: str):+ """Test parsing various concept codes without multiplicity."""+ result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)+ assert result.concept.concept_string == expected_concept_string+ assert result.multiplicity is None++ def test_custom_domain_concepts(self):+ """Test parsing concept codes from custom domains."""+ # Note: This test will only work if these concepts exist in the system+ # For now, we'll test with native concepts, but the pattern should work for any domain+ result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[3]")+ assert result.concept.concept_string == "native.Text"+ assert result.multiplicity == 3++ def test_concept_not_found_raises_error(self):+ """Test that an invalid concept code raises ConceptLibraryConceptNotFoundError."""+ with pytest.raises(ConceptLibraryConceptNotFoundError):+ InputRequirementsFactory.make_from_string(domain="nonexistent", requirement_str="nonexistent.InvalidConcept")++ def test_concept_not_found_with_multiplicity_raises_error(self):+ """Test that an invalid concept code with multiplicity raises ConceptLibraryConceptNotFoundError."""+ with pytest.raises(ConceptLibraryConceptNotFoundError):+ InputRequirementsFactory.make_from_string(domain="nonexistent", requirement_str="nonexistent.InvalidConcept[5]")++ def test_empty_string_raises_value_error(self):+ """Test that an empty string raises InputRequirementsFactorySyntaxError."""+ with pytest.raises(InputRequirementsFactorySyntaxError, match="Invalid input requirement string"):+ InputRequirementsFactory.make_from_string(domain="native", requirement_str="")++ def test_malformed_brackets_with_non_digit(self):+ """Test that brackets with non-digit content are treated as part of concept string."""+ # The regex will match "native.Text[abc]" as concept="native.Text[abc]", multiplicity=None+ # This will then fail during concept validation with ConceptCodeError+ with pytest.raises(ConceptCodeError):+ InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[abc]")++ def test_multiplicity_zero_in_brackets(self):+ """Test parsing a concept string with 0 in brackets."""+ result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[0]")++ assert isinstance(result, InputRequirement)+ assert result.concept.concept_string == "native.Text"+ assert result.multiplicity == 0++ def test_return_type(self):+ """Test that the method returns an InputRequirement instance."""+ result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text")+ assert isinstance(result, InputRequirement)++ def test_concept_attribute_access(self):+ """Test that the returned InputRequirement has proper concept attributes."""+ result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[5]")++ assert hasattr(result, "concept")+ assert hasattr(result, "multiplicity")+ assert result.concept.concept_string == "native.Text"+ assert result.concept.code == "Text"++ def test_edge_case_very_long_number(self):+ """Test parsing with a very long number in brackets."""+ result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[999999]")++ assert result.multiplicity == 999999+ assert result.concept.concept_string == "native.Text"++ def test_whitespace_not_trimmed(self):+ """Test that whitespace is not automatically trimmed."""+ # Whitespace should cause domain validation to fail+ with pytest.raises(DomainError):+ InputRequirementsFactory.make_from_string(domain="native", requirement_str=" native.Text")++ # Trailing whitespace should cause concept code validation to fail+ with pytest.raises(ConceptCodeError):+ InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text ")++ def test_multiple_brackets_treated_as_concept_name(self):+ """Test that multiple brackets are treated as part of the concept name."""+ # "native.Text[5][10]" should match as concept="native.Text[5]", multiplicity=10+ # This will fail during concept code validation+ with pytest.raises(ConceptCodeError):+ InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[5][10]")++ def test_brackets_at_start_treated_as_concept_name(self):+ """Test that brackets at the start are part of the concept name."""+ # This will fail during domain validation+ with pytest.raises(DomainError):+ InputRequirementsFactory.make_from_string(domain="native", requirement_str="[5]native.Text")++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "concept_codes_from_same_domain", "expected_concept_string", "expected_multiplicity", "description"),+ CONCEPT_CODE_RESOLUTION_TEST_CASES,+ )+ def test_concept_code_resolution(+ self,+ domain: str,+ requirement_str: str,+ concept_codes_from_same_domain: list[str] | None,+ expected_concept_string: str,+ expected_multiplicity: int | bool | None,+ description: str,+ ):+ """Test that concept codes are resolved correctly with domain and concept_codes_from_same_domain.++ This tests:+ 1. Native concepts are always recognized regardless of domain parameter+ 2. Unknown concepts (not native, not in the domain) become implicit concepts+ 3. concept_codes_from_same_domain helps resolve ambiguous concept codes+ """+ result = InputRequirementsFactory.make_from_string(+ domain=domain,+ requirement_str=requirement_str,+ concept_codes_from_the_same_domain=concept_codes_from_same_domain,+ )++ assert isinstance(result, InputRequirement)+ assert result.concept.concept_string == expected_concept_string, f"Failed: {description}"+ assert result.multiplicity == expected_multiplicity, f"Failed: {description}"++ @pytest.mark.parametrize(+ ("domain", "requirement_str", "expected_concept_string", "expected_multiplicity"),+ EXPLICIT_DOMAIN_IN_STRING_TEST_CASES,+ )+ def test_explicit_domain_in_string(+ self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int | bool | None+ ):+ """Test that explicitly specifying a domain in the requirement string works correctly."""+ result = InputRequirementsFactory.make_from_string(+ domain=domain,+ requirement_str=requirement_str,+ )++ assert isinstance(result, InputRequirement)+ assert result.concept.concept_string == expected_concept_string+ assert result.multiplicity == expected_multiplicitydiff --git a/tests/unit/pipelex/core/pipes/test_make_variable_multiplicity.py b/tests/unit/pipelex/core/pipes/test_make_variable_multiplicity.pynew file mode 100644index 000000000..a1dc65b65--- /dev/null+++ b/tests/unit/pipelex/core/pipes/test_make_variable_multiplicity.py@@ -0,0 +1,101 @@+import pytest++from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity, make_variable_multiplicity+from tests.unit.pipelex.core.pipes.data import MAKE_VARIABLE_MULTIPLICITY_TEST_CASES+++class TestMakeVariableMultiplicity:+ @pytest.mark.parametrize(+ ("nb_items", "multiple_items", "expected_result", "description"),+ MAKE_VARIABLE_MULTIPLICITY_TEST_CASES,+ ids=[case[3] for case in MAKE_VARIABLE_MULTIPLICITY_TEST_CASES],+ )+ def test_make_variable_multiplicity_all_cases(+ self,+ nb_items: int | None,+ multiple_items: bool | None,+ expected_result: VariableMultiplicity | None,+ description: str,+ ):+ """Test make_variable_multiplicity with all parameter combinations."""+ result = make_variable_multiplicity(nb_items=nb_items, multiple_items=multiple_items)+ assert result == expected_result, f"Failed case: {description}"++ def test_nb_items_precedence_over_multiple_items(self):+ """Test that nb_items takes precedence when both parameters are provided."""+ # When both are provided and nb_items is truthy, it should take precedence+ result = make_variable_multiplicity(nb_items=3, multiple_items=True)+ assert result == 3++ result = make_variable_multiplicity(nb_items=1, multiple_items=False)+ assert result == 1++ # When nb_items is falsy (0), multiple_items should be used+ result = make_variable_multiplicity(nb_items=0, multiple_items=True)+ assert result is True++ def test_return_types(self):+ """Test that the function returns the correct types."""+ # Should return int when nb_items is provided+ result = make_variable_multiplicity(nb_items=5, multiple_items=None)+ assert isinstance(result, int)+ assert result == 5++ # Should return bool when multiple_items=True and nb_items is falsy/None+ result = make_variable_multiplicity(nb_items=None, multiple_items=True)+ assert isinstance(result, bool)+ assert result is True++ # Should return None when both are falsy/None+ result = make_variable_multiplicity(nb_items=None, multiple_items=None)+ assert result is None++ result = make_variable_multiplicity(nb_items=0, multiple_items=False)+ assert result is None++ def test_edge_cases(self):+ """Test edge cases and boundary conditions."""+ # Negative numbers should still be truthy+ result = make_variable_multiplicity(nb_items=-1, multiple_items=True)+ assert result == -1++ # Large numbers+ result = make_variable_multiplicity(nb_items=999999, multiple_items=None)+ assert result == 999999++ # Zero is falsy, so multiple_items should be used+ result = make_variable_multiplicity(nb_items=0, multiple_items=True)+ assert result is True++ # Both falsy should return None+ result = make_variable_multiplicity(nb_items=0, multiple_items=False)+ assert result is None++ def test_mutually_exclusive_logic(self):+ """Test the mutually exclusive behavior of the parameters."""+ # nb_items takes precedence when truthy+ assert make_variable_multiplicity(nb_items=2, multiple_items=True) == 2+ assert make_variable_multiplicity(nb_items=2, multiple_items=False) == 2++ # multiple_items is used when nb_items is falsy+ assert make_variable_multiplicity(nb_items=0, multiple_items=True) is True+ assert make_variable_multiplicity(nb_items=None, multiple_items=True) is True++ # Default case when both are falsy/None+ assert make_variable_multiplicity(nb_items=0, multiple_items=False) is None+ assert make_variable_multiplicity(nb_items=None, multiple_items=False) is None+ assert make_variable_multiplicity(nb_items=None, multiple_items=None) is None++ def test_function_signature_and_return_annotation(self):+ """Test that the function can be called with the expected signature."""+ # Test with keyword arguments+ result = make_variable_multiplicity(nb_items=3, multiple_items=None)+ assert result == 3++ # Test with positional arguments+ result = make_variable_multiplicity(5, False)+ assert result == 5++ # Test with mixed arguments+ result = make_variable_multiplicity(nb_items=2, multiple_items=True)+ assert result == 2diff --git a/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.py b/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.pynew file mode 100644index 000000000..cbbeb49fa--- /dev/null+++ b/tests/unit/pipelex/core/pipes/test_parse_concept_with_multiplicity.py@@ -0,0 +1,76 @@+import pytest++from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity+++class TestParseConceptWithMultiplicity:+ """Test parsing of concept specifications with bracket notation."""++ def test_valid_simple_concept(self):+ """Test parsing simple concept names without brackets."""+ result = parse_concept_with_multiplicity("Text")+ assert result.concept == "Text"+ assert result.multiplicity is None++ def test_valid_concept_with_variable_list(self):+ """Test parsing concept with empty brackets []."""+ result = parse_concept_with_multiplicity("Text[]")+ assert result.concept == "Text"+ assert result.multiplicity is True++ def test_valid_concept_with_fixed_count(self):+ """Test parsing concept with fixed count [N]."""+ result = parse_concept_with_multiplicity("Text[5]")+ assert result.concept == "Text"+ assert result.multiplicity == 5++ def test_valid_domain_qualified_concept(self):+ """Test parsing domain-qualified concepts."""+ result = parse_concept_with_multiplicity("domain.Concept")+ assert result.concept == "domain.Concept"+ assert result.multiplicity is None++ def test_valid_domain_qualified_with_brackets(self):+ """Test parsing domain-qualified concepts with brackets."""+ result = parse_concept_with_multiplicity("native.Text[]")+ assert result.concept == "native.Text"+ assert result.multiplicity is True++ result = parse_concept_with_multiplicity("custom.Item[3]")+ assert result.concept == "custom.Item"+ assert result.multiplicity == 3++ def test_invalid_syntax_starting_with_number(self):+ """Test that concepts starting with numbers are rejected."""+ with pytest.raises(ValueError, match="Invalid concept specification syntax"):+ parse_concept_with_multiplicity("123Invalid[]")++ def test_invalid_syntax_with_hyphen(self):+ """Test that concepts with hyphens are rejected."""+ with pytest.raises(ValueError, match="Invalid concept specification syntax"):+ parse_concept_with_multiplicity("foo-bar[2]")++ def test_invalid_syntax_with_special_chars(self):+ """Test that concepts with special characters are rejected."""+ with pytest.raises(ValueError, match="Invalid concept specification syntax"):+ parse_concept_with_multiplicity("!@#$%")++ def test_invalid_syntax_empty_string(self):+ """Test that empty string is rejected."""+ with pytest.raises(ValueError, match="Invalid concept specification syntax"):+ parse_concept_with_multiplicity("")++ def test_invalid_syntax_only_brackets(self):+ """Test that only brackets without concept is rejected."""+ with pytest.raises(ValueError, match="Invalid concept specification syntax"):+ parse_concept_with_multiplicity("[]")++ def test_concepts_starting_with_underscore(self):+ """Test that concepts starting with underscore are valid."""+ result = parse_concept_with_multiplicity("_PrivateConcept")+ assert result.concept == "_PrivateConcept"+ assert result.multiplicity is None++ result = parse_concept_with_multiplicity("_internal.Data[]")+ assert result.concept == "_internal.Data"+ assert result.multiplicity is Truediff --git a/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py b/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.pyindex 52e989960..5d86d0d11 100644--- a/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py+++ b/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py@@ -10,8 +10,8 @@ description = "Domain with extract pipe"[pipe.extract_text]type = "PipeExtract"description = "Extract text from document"-output = "Page"-model = "base_extract_pypdfium2"+output = "Page[]"+model = "extract_text_from_pdf"""",PipelexBundleBlueprint(domain="test_pipes",@@ -20,8 +20,8 @@ model = "base_extract_pypdfium2""extract_text": PipeExtractBlueprint(type="PipeExtract",description="Extract text from document",- output=NativeConceptCode.PAGE,- model="base_extract_pypdfium2",+ output=NativeConceptCode.PAGE.as_output_multiple_indeterminate,+ model="extract_text_from_pdf",),},),diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py b/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.pyindex d1e28ff1d..5479a7b39 100644--- a/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py+++ b/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py@@ -1,7 +1,6 @@# ruff: noqa: E501from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.core.concepts.concept_native import NativeConceptCode-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint# Basic PipeLLM with prompt_template@@ -129,9 +128,8 @@ description = "Domain with pipe definitions"[pipe.generate_ideas]type = "PipeLLM"description = "Generate multiple ideas"-output = "Text"+output = "Text[3]"prompt = "Generate creative ideas for a mobile app"-nb_output = 3""",PipelexBundleBlueprint(domain="test_pipes",@@ -140,8 +138,7 @@ nb_output = 3"generate_ideas": PipeLLMBlueprint(type="PipeLLM",description="Generate multiple ideas",- output=NativeConceptCode.TEXT,- nb_output=3,+ output="Text[3]",prompt="Generate creative ideas for a mobile app",),},@@ -157,10 +154,9 @@ description = "Domain with pipe definitions"[pipe.brainstorm_solutions]type = "PipeLLM"description = "Brainstorm multiple solutions"-inputs = { problem = { concept = "Text" } }-output = "Text"+inputs = { problem = "Text" }+output = "Text[]"prompt = "Brainstorm solutions for this problem: $problem"-multiple_output = true""",PipelexBundleBlueprint(domain="test_pipes",@@ -169,9 +165,8 @@ multiple_output = true"brainstorm_solutions": PipeLLMBlueprint(type="PipeLLM",description="Brainstorm multiple solutions",- inputs={"problem": InputRequirementBlueprint(concept="Text")},- output=NativeConceptCode.TEXT,- multiple_output=True,+ inputs={"problem": "Text"},+ output="Text[]",prompt="Brainstorm solutions for this problem: $problem",),},@@ -274,7 +269,7 @@ PersonInfo = "Information about a person"[pipe.extract_person_info]type = "PipeLLM"description = "Extract structured person information"-inputs = { text = { concept = "Text", multiplicity = 1 } }+inputs = { text = "Text[1]" }output = "PersonInfo"prompt = "Extract person information from this text: @text"""",@@ -286,7 +281,7 @@ prompt = "Extract person information from this text: @text""extract_person_info": PipeLLMBlueprint(type="PipeLLM",description="Extract structured person information",- inputs={"text": InputRequirementBlueprint(concept="Text", multiplicity=1)},+ inputs={"text": "Text[1]"},output="PersonInfo",prompt="Extract person information from this text: @text",),@@ -306,7 +301,7 @@ DocumentSummary = "Summary of a document"[pipe.analyze_documents]type = "PipeLLM"description = "Analyze multiple documents and single query"-inputs = { documents = { concept = "Text", multiplicity = true }, query = { concept = "Text", multiplicity = false } }+inputs = { documents = "Text[]", query = "Text" }output = "DocumentSummary"prompt = """Analyze these documents based on the query: $query@@ -323,8 +318,8 @@ Documents: @documentstype="PipeLLM",description="Analyze multiple documents and single query",inputs={- "documents": InputRequirementBlueprint(concept="Text", multiplicity=True),- "query": InputRequirementBlueprint(concept="Text", multiplicity=False),+ "documents": "Text[]",+ "query": "Text",},output="DocumentSummary",prompt="""Analyze these documents based on the query: $querydiff --git a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.pyindex 2eef16fa6..653cfe705 100644--- a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py+++ b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py@@ -16,7 +16,7 @@ class TestPipeBatchValidateInputs:test_id: str,blueprint: PipeBatchBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")# Validation happens automatically during instantiation via model_validatorpipe_batch = PipeBatchFactory.make_from_blueprint(diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.pyindex fb810bf2e..4f3b2857f 100644--- a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py@@ -19,7 +19,7 @@ class TestPipeConditionValidateInputs:test_id: str,blueprint: PipeConditionBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")# Validation happens automatically during instantiation via model_validatorpipe_condition = PipeConditionFactory.make_from_blueprint(@@ -42,7 +42,7 @@ class TestPipeConditionValidateInputs:blueprint_dict: dict[str, Any],expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012# Construct blueprint from dict at test time to trigger validationdiff --git a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.pyindex 08497daf5..83e54b6ee 100644--- a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py+++ b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py@@ -19,7 +19,7 @@ class TestPipeParallelValidateInputs:test_id: str,blueprint: PipeParallelBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")# Validation happens automatically during instantiation via model_validatorpipe_parallel = PipeParallelFactory.make_from_blueprint(@@ -42,7 +42,7 @@ class TestPipeParallelValidateInputs:blueprint_dict: dict[str, Any],expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012# Construct blueprint from dict at test time to trigger validationdiff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.pyindex 9043917ae..f577ab588 100644--- a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py@@ -19,7 +19,7 @@ class TestPipeSequenceValidateInputs:test_id: str,blueprint: PipeSequenceBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")# Validation happens automatically during instantiation via model_validatorpipe_sequence = PipeSequenceFactory.make_from_blueprint(@@ -42,7 +42,7 @@ class TestPipeSequenceValidateInputs:blueprint_dict: dict[str, Any],expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012# Construct blueprint from dict at test time to trigger validationdiff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.pyindex dbf8aa90c..be758053a 100644--- a/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py@@ -16,7 +16,7 @@ class TestPipeComposeValidateInputs:test_id: str,blueprint: PipeComposeBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")# Validation happens automatically during instantiation via model_validatorpipe_compose = PipeComposeFactory.make_from_blueprint(@@ -40,7 +40,7 @@ class TestPipeComposeValidateInputs:expected_error_type: type[Exception],expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises(expected_error_type) as exc_info:PipeComposeFactory.make_from_blueprint(diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.pyindex 359460d9b..39540d5f7 100644--- a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py+++ b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py@@ -17,7 +17,7 @@ class TestPipeExtractValidateInputs:test_id: str,blueprint: PipeExtractBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")# Validation happens automatically during instantiation via model_validatorpipe_extract = PipeExtractFactory.make_from_blueprint(@@ -40,7 +40,7 @@ class TestPipeExtractValidateInputs:blueprint: PipeExtractBlueprint,expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises(StaticValidationError) as exc_info:PipeExtractFactory.make_from_blueprint(diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.pyindex cffd07d99..6c799c372 100644--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py@@ -16,7 +16,7 @@ class TestPipeFuncValidateInputs:test_id: str,blueprint: PipeFuncBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")pipe_func = PipeFuncFactory.make_from_blueprint(domain="test_domain",diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.pyindex 5ab79f8a5..8e6de6897 100644--- a/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py@@ -42,8 +42,7 @@ class PipeImgGenInputTestCases:PipeImgGenBlueprint(description="Test case: valid_with_nb_output",inputs={"prompt": "native.Text"},- output="native.Image",- nb_output=3,+ output="native.Image[3]",),)diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.pyindex ca07c75c5..c35cc719e 100644--- a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py@@ -17,7 +17,7 @@ class TestPipeImgGenValidateInputs:test_id: str,blueprint: PipeImgGenBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")pipe_img_gen = PipeImgGenFactory.make_from_blueprint(domain="test_domain",@@ -39,7 +39,7 @@ class TestPipeImgGenValidateInputs:blueprint: PipeImgGenBlueprint,expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises((StaticValidationError, PipeDefinitionError)) as exc_info:PipeImgGenFactory.make_from_blueprint(diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.pyindex ecfa7ebe7..466dd9dbf 100644--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py@@ -17,7 +17,7 @@ class TestPipeLLMValidateInputs:test_id: str,blueprint: PipeLLMBlueprint,):- log.debug(f"Testing valid case: {test_id}")+ log.verbose(f"Testing valid case: {test_id}")pipe_llm = PipeLLMFactory.make_from_blueprint(domain="test_domain",@@ -37,7 +37,7 @@ class TestPipeLLMValidateInputs:blueprint: PipeLLMBlueprint,expected_error_message_fragment: str,):- log.debug(f"Testing error case: {test_id}")+ log.verbose(f"Testing error case: {test_id}")with pytest.raises(StaticValidationError) as exc_info:PipeLLMFactory.make_from_blueprint(diff --git a/tests/unit/pipelex/pipe_run/data.py b/tests/unit/pipelex/pipe_run/data.pyindex 8c41ac690..aa7f406b7 100644--- a/tests/unit/pipelex/pipe_run/data.py+++ b/tests/unit/pipelex/pipe_run/data.py@@ -1,196 +1,171 @@-from pipelex.pipe_run.pipe_run_params import OutputMultiplicityResolution, PipeOutputMultiplicity--# Test cases format: (nb_output, multiple_output, expected_result, test_description)-MAKE_OUTPUT_MULTIPLICITY_TEST_CASES: list[tuple[int | None, bool | None, PipeOutputMultiplicity | None, str]] = [- # nb_output, multiple_output, expected_result, description- # Test cases where nb_output takes precedence- (3, None, 3, "nb_output=3, multiple_output=None -> returns 3"),- (5, False, 5, "nb_output=5, multiple_output=False -> returns 5 (nb_output takes precedence)"),- (1, True, 1, "nb_output=1, multiple_output=True -> returns 1 (nb_output takes precedence)"),- (10, None, 10, "nb_output=10, multiple_output=None -> returns 10"),- # Test cases where multiple_output=True is used- (None, True, True, "nb_output=None, multiple_output=True -> returns True"),- (0, True, True, "nb_output=0 (falsy), multiple_output=True -> returns True"),- # Test cases where default (None) is returned- (None, None, None, "nb_output=None, multiple_output=None -> returns None (default)"),- (None, False, None, "nb_output=None, multiple_output=False -> returns None"),- (0, None, None, "nb_output=0 (falsy), multiple_output=None -> returns None"),- (0, False, None, "nb_output=0 (falsy), multiple_output=False -> returns None"),- # Edge cases with negative numbers (should still be truthy)- (-1, None, -1, "nb_output=-1 (truthy), multiple_output=None -> returns -1"),- (-5, True, -5, "nb_output=-5 (truthy), multiple_output=True -> returns -5 (nb_output takes precedence)"),- # Edge cases with large numbers- (1000, None, 1000, "nb_output=1000, multiple_output=None -> returns 1000"),- (999999, False, 999999, "nb_output=999999, multiple_output=False -> returns 999999"),-]-+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity, VariableMultiplicityResolution# Test cases format: (base, override, expected_result, description)OUTPUT_MULTIPLICITY_TO_APPLY_TEST_CASES: list[- tuple[PipeOutputMultiplicity | None, PipeOutputMultiplicity | None, OutputMultiplicityResolution, str]+ tuple[VariableMultiplicity | None, VariableMultiplicity | None, VariableMultiplicityResolution, str]] = [# base, override, expected_result, description# Override is None - use base value as-is(None,None,- OutputMultiplicityResolution(resolved_multiplicity=None, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=None, is_multiple_outputs_enabled=False, specific_output_count=None),"base=None, override=None",),(True,None,- OutputMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None),"base=True, override=None",),(False,None,- OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),"base=False, override=None",),(3,None,- OutputMultiplicityResolution(resolved_multiplicity=3, is_multiple_outputs_enabled=True, specific_output_count=3),+ VariableMultiplicityResolution(resolved_multiplicity=3, is_multiple_outputs_enabled=True, specific_output_count=3),"base=3, override=None",),(5,None,- OutputMultiplicityResolution(resolved_multiplicity=5, is_multiple_outputs_enabled=True, specific_output_count=5),+ VariableMultiplicityResolution(resolved_multiplicity=5, is_multiple_outputs_enabled=True, specific_output_count=5),"base=5, override=None",),# Override is False - force single output regardless of base(None,False,- OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),"base=None, override=False",),(True,False,- OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),"base=True, override=False",),(False,False,- OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),"base=False, override=False",),(3,False,- OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),"base=3, override=False",),(10,False,- OutputMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=False, is_multiple_outputs_enabled=False, specific_output_count=None),"base=10, override=False",),# Override is True - enable multiple outputs(None,True,- OutputMultiplicityResolution(resolved_multiplicity=None, is_multiple_outputs_enabled=True, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=None, is_multiple_outputs_enabled=True, specific_output_count=None),"base=None, override=True",),(True,True,- OutputMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None),"base=True, override=True",),(False,True,- OutputMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None),+ VariableMultiplicityResolution(resolved_multiplicity=True, is_multiple_outputs_enabled=True, specific_output_count=None),"base=False, override=True",),(3,True,- OutputMultiplicityResolution(resolved_multiplicity=3, is_multiple_outputs_enabled=True, specific_output_count=3),+ VariableMultiplicityResolution(resolved_multiplicity=3, is_multiple_outputs_enabled=True, specific_output_count=3),"base=3, override=True - preserve base count",),(7,True,- OutputMultiplicityResolution(resolved_multiplicity=7, is_multiple_outputs_enabled=True, specific_output_count=7),+ VariableMultiplicityResolution(resolved_multiplicity=7, is_multiple_outputs_enabled=True, specific_output_count=7),"base=7, override=True - preserve base count",),# Override is int - use override count, enable multiple outputs(None,2,- OutputMultiplicityResolution(resolved_multiplicity=2, is_multiple_outputs_enabled=True, specific_output_count=2),+ VariableMultiplicityResolution(resolved_multiplicity=2, is_multiple_outputs_enabled=True, specific_output_count=2),"base=None, override=2",),(True,4,- OutputMultiplicityResolution(resolved_multiplicity=4, is_multiple_outputs_enabled=True, specific_output_count=4),+ VariableMultiplicityResolution(resolved_multiplicity=4, is_multiple_outputs_enabled=True, specific_output_count=4),"base=True, override=4",),(False,6,- OutputMultiplicityResolution(resolved_multiplicity=6, is_multiple_outputs_enabled=True, specific_output_count=6),+ VariableMultiplicityResolution(resolved_multiplicity=6, is_multiple_outputs_enabled=True, specific_output_count=6),"base=False, override=6",),(3,8,- OutputMultiplicityResolution(resolved_multiplicity=8, is_multiple_outputs_enabled=True, specific_output_count=8),+ VariableMultiplicityResolution(resolved_multiplicity=8, is_multiple_outputs_enabled=True, specific_output_count=8),"base=3, override=8 - override takes precedence",),(5,1,- OutputMultiplicityResolution(resolved_multiplicity=1, is_multiple_outputs_enabled=True, specific_output_count=1),+ VariableMultiplicityResolution(resolved_multiplicity=1, is_multiple_outputs_enabled=True, specific_output_count=1),"base=5, override=1 - override takes precedence",),# Edge cases with negative numbers(-1,None,- OutputMultiplicityResolution(resolved_multiplicity=-1, is_multiple_outputs_enabled=True, specific_output_count=-1),+ VariableMultiplicityResolution(resolved_multiplicity=-1, is_multiple_outputs_enabled=True, specific_output_count=-1),"base=-1, override=None",),(3,-2,- OutputMultiplicityResolution(resolved_multiplicity=-2, is_multiple_outputs_enabled=True, specific_output_count=-2),+ VariableMultiplicityResolution(resolved_multiplicity=-2, is_multiple_outputs_enabled=True, specific_output_count=-2),"base=3, override=-2",),# Edge cases with zero(0,None,- OutputMultiplicityResolution(resolved_multiplicity=0, is_multiple_outputs_enabled=True, specific_output_count=0),+ VariableMultiplicityResolution(resolved_multiplicity=0, is_multiple_outputs_enabled=True, specific_output_count=0),"base=0, override=None",),(True,0,- OutputMultiplicityResolution(resolved_multiplicity=0, is_multiple_outputs_enabled=True, specific_output_count=0),+ VariableMultiplicityResolution(resolved_multiplicity=0, is_multiple_outputs_enabled=True, specific_output_count=0),"base=True, override=0",),# Large numbers(1000,None,- OutputMultiplicityResolution(resolved_multiplicity=1000, is_multiple_outputs_enabled=True, specific_output_count=1000),+ VariableMultiplicityResolution(resolved_multiplicity=1000, is_multiple_outputs_enabled=True, specific_output_count=1000),"base=1000, override=None",),(100,999,- OutputMultiplicityResolution(resolved_multiplicity=999, is_multiple_outputs_enabled=True, specific_output_count=999),+ VariableMultiplicityResolution(resolved_multiplicity=999, is_multiple_outputs_enabled=True, specific_output_count=999),"base=100, override=999",),]diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run_params.py b/tests/unit/pipelex/pipe_run/test_pipe_run_params.pyindex a927f61cf..f3839a84e 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_run_params.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_run_params.py@@ -1,109 +1,10 @@import pytest+from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity, VariableMultiplicityResolutionfrom pipelex.pipe_run.pipe_run_params import (- OutputMultiplicityResolution,- PipeOutputMultiplicity,- make_output_multiplicity,output_multiplicity_to_apply,)-from tests.unit.pipelex.pipe_run.data import MAKE_OUTPUT_MULTIPLICITY_TEST_CASES, OUTPUT_MULTIPLICITY_TO_APPLY_TEST_CASES---class TestMakeOutputMultiplicity:- @pytest.mark.parametrize(- ("nb_output", "multiple_output", "expected_result", "description"),- MAKE_OUTPUT_MULTIPLICITY_TEST_CASES,- ids=[case[3] for case in MAKE_OUTPUT_MULTIPLICITY_TEST_CASES],- )- def test_make_output_multiplicity_all_cases(- self,- nb_output: int | None,- multiple_output: bool | None,- expected_result: PipeOutputMultiplicity | None,- description: str,- ):- """Test make_output_multiplicity with all parameter combinations."""- result = make_output_multiplicity(nb_output=nb_output, multiple_output=multiple_output)- assert result == expected_result, f"Failed case: {description}"-- def test_nb_output_precedence_over_multiple_output(self):- """Test that nb_output takes precedence when both parameters are provided."""- # When both are provided and nb_output is truthy, it should take precedence- result = make_output_multiplicity(nb_output=3, multiple_output=True)- assert result == 3-- result = make_output_multiplicity(nb_output=1, multiple_output=False)- assert result == 1-- # When nb_output is falsy (0), multiple_output should be used- result = make_output_multiplicity(nb_output=0, multiple_output=True)- assert result is True-- def test_return_types(self):- """Test that the function returns the correct types."""- # Should return int when nb_output is provided- result = make_output_multiplicity(nb_output=5, multiple_output=None)- assert isinstance(result, int)- assert result == 5-- # Should return bool when multiple_output=True and nb_output is falsy/None- result = make_output_multiplicity(nb_output=None, multiple_output=True)- assert isinstance(result, bool)- assert result is True-- # Should return None when both are falsy/None- result = make_output_multiplicity(nb_output=None, multiple_output=None)- assert result is None-- result = make_output_multiplicity(nb_output=0, multiple_output=False)- assert result is None-- def test_edge_cases(self):- """Test edge cases and boundary conditions."""- # Negative numbers should still be truthy- result = make_output_multiplicity(nb_output=-1, multiple_output=True)- assert result == -1-- # Large numbers- result = make_output_multiplicity(nb_output=999999, multiple_output=None)- assert result == 999999-- # Zero is falsy, so multiple_output should be used- result = make_output_multiplicity(nb_output=0, multiple_output=True)- assert result is True-- # Both falsy should return None- result = make_output_multiplicity(nb_output=0, multiple_output=False)- assert result is None-- def test_mutually_exclusive_logic(self):- """Test the mutually exclusive behavior of the parameters."""- # nb_output takes precedence when truthy- assert make_output_multiplicity(nb_output=2, multiple_output=True) == 2- assert make_output_multiplicity(nb_output=2, multiple_output=False) == 2-- # multiple_output is used when nb_output is falsy- assert make_output_multiplicity(nb_output=0, multiple_output=True) is True- assert make_output_multiplicity(nb_output=None, multiple_output=True) is True-- # Default case when both are falsy/None- assert make_output_multiplicity(nb_output=0, multiple_output=False) is None- assert make_output_multiplicity(nb_output=None, multiple_output=False) is None- assert make_output_multiplicity(nb_output=None, multiple_output=None) is None-- def test_function_signature_and_return_annotation(self):- """Test that the function can be called with the expected signature."""- # Test with keyword arguments- result = make_output_multiplicity(nb_output=3, multiple_output=None)- assert result == 3-- # Test with positional arguments- result = make_output_multiplicity(5, False)- assert result == 5-- # Test with mixed arguments- result = make_output_multiplicity(nb_output=2, multiple_output=True)- assert result == 2+from tests.unit.pipelex.pipe_run.data import OUTPUT_MULTIPLICITY_TO_APPLY_TEST_CASESclass TestOutputMultiplicityToApply:@@ -116,9 +17,9 @@ class TestOutputMultiplicityToApply:)def test_output_multiplicity_to_apply_all_cases(self,- base: PipeOutputMultiplicity | None,- override: PipeOutputMultiplicity | None,- expected_result: OutputMultiplicityResolution,+ base: VariableMultiplicity | None,+ override: VariableMultiplicity | None,+ expected_result: VariableMultiplicityResolution,description: str,):"""Test output_multiplicity_to_apply with all parameter combinations."""@@ -207,7 +108,7 @@ class TestOutputMultiplicityToApply:def test_return_basemodel_structure(self):"""Test that the function always returns an OutputMultiplicityResolution with correct types."""result = output_multiplicity_to_apply(base_multiplicity=None, override_multiplicity=None)- assert isinstance(result, OutputMultiplicityResolution)+ assert isinstance(result, VariableMultiplicityResolution)# Test field typesassert result.resolved_multiplicity is None or isinstance(result.resolved_multiplicity, (bool, int))